Media Use

media-use

The media OS for HyperFrames: resolve · generate · operate · remember, every media type, one skill, zero context noise.

Setup — install heygen first (free-usage path)

curl -fsSL https://static.heygen.ai/cli/install.sh | bash
heygen update             # free usage needs the OAuth-capable CLI (v0.3.0+)
heygen auth login --oauth # OAuth = free subscription credits; --api-key bills API credits

This unlocks the FREE path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar videos. Sign in with --oauth — the free allowance rides on the OAuth session (an API key bills API credits instead). media-use requires heygen >= v0.3.0 uniformly (the OAuth free-usage path needs it), so --doctor nudges older CLIs to update even for API-key-only use. Before resolving anything, verify setup with:

node <SKILL_DIR>/scripts/resolve.mjs --doctor

What it owns (the gaps HyperFrames leaves)

HyperFrames owns media playback; media-use owns everything else. Each row is enforced by scripts/lib/coverage.test.mjs so the claim can't rot.

When to use

Call resolve whenever a composition needs media: background music, sound effects, images, icons, brand logos, voice, a color grade, or a LUT. For voiceover / TTS, music, SFX, and caption timing, use the audio engine (below); background removal is delegated to the hyperframes CLI; transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via scripts/transcribe.mjs, with whisper.cpp auto-fallback (see references/operations.md). For cutting / reframing / transforming existing media, see references/operations.md. media-use searches the HeyGen catalog first for media files, resolves official logos through the logo cascade, uses local deterministic color grading for grade/lut, freezes the best match locally when a file is needed, registers it in a manifest, and hands the agent one line; all search noise stays on disk.

Be proactive — run a media opportunity pass

The human usually can't tell which media would lift the piece. You can. When you build or review a composition, do one grounded scan and then ask once — don't silently add, and don't nag per asset.

Surface an opportunity only when a concrete signal is present:

Rules that keep this a help, not nagware:

• Grounded, not generic. No signal → no suggestion. Never open with "want better images?".
• Opinionated + concrete. Propose the specific fix ("add a VO from your script, swap 3 emoji for real icons, replace the 400×400 hero, whooshes on the 4 cuts"), with defaults chosen — the human just approves all / some / none.
• Once per project. One consolidated ask, top few highest-value items. Respect "leave it" and don't re-raise.
• Surface, never silently mutate. Color grades especially: propose and preview, never auto-apply — a gray-world "correction" ruins an intentional sunset or neon look.

Resolve

node <SKILL_DIR>/scripts/resolve.mjs --type <type> --intent "<description>" --project <dir>

Returns one line: resolved <id> → <path> (<type>, <metadata>)

Types

Examples

# Background music
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --project .
# → resolved bgm_001 → .media/audio/bgm/bgm_001.mp3 (bgm, 25s)

# Sound effect
node <SKILL_DIR>/scripts/resolve.mjs --type sfx --intent "whoosh" --project .
# → resolved sfx_001 → .media/audio/sfx/sfx_001.mp3 (sfx, 0.57s)

# Image
node <SKILL_DIR>/scripts/resolve.mjs --type image --intent "gradient tech background" --project .
# → resolved image_001 → .media/images/image_001.jpg (image)

# Icon
node <SKILL_DIR>/scripts/resolve.mjs --type icon --intent "rocket" --project .
# → resolved icon_001 → .media/images/icon_001.png (icon, transparent)

# Brand logo (official mark — never redrawn by hand)
node <SKILL_DIR>/scripts/resolve.mjs --type logo --entity linkedin --intent "LinkedIn logo" --project .
# → resolved logo_001 → .media/images/logo_001.svg (logo, official mark)

# Color grade block
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json
# → {"ok":true,"preset":"warm-daylight","grading":{"preset":"warm-daylight","intensity":1},...}

# LUT file
node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project .
# → resolved lut_001 → .media/luts/lut_001.cube (lut)

Flags

Reuse before you resolve

Before resolving bgm/sfx/image/icon/logo/grade/lut, check what already exists and reuse it when it fits. media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide.

node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --candidates --project .
#   [project] upbeat tech launch (25s, heygen.audio.sounds)
#           .media/audio/bgm/bgm_001.wav
#   [global]  energetic tech intro (22s, heygen.audio.sounds)
#           --reuse 06e052c075fd2b80

Read the list and judge semantic fit yourself — "upbeat tech launch" ≈ "energetic tech intro" is a call only you can make from the descriptions. Then:

• A project candidate fits → just reference its path in your composition. Nothing else to run.
• A global candidate fits → resolve --type bgm --reuse <sha> copies it into this project (self-contained render) and records it.
• Nothing fits → resolve fresh (--type ... --intent ...).

Trust guardrail — when unsure, resolve fresh. A redundant download is cheap; shipping the wrong asset is not. Judge fit from description + prompt + type + duration/dims. For brand/entity assets, reuse a global candidate only when the entity matches exactly — the global cache aggregates every project you have worked on, so a --candidates list can surface another client's brand mark and its prompt text. Never reuse a cross-project brand asset on a loose match.

The deterministic floor still runs automatically: an identical (case/whitespace-insensitive) repeat auto-reuses with no --candidates step. --candidates is only for the semantic layer above that floor — and a fuzzy match is never auto-applied; reuse is always your explicit call. On a resolve that misses the floor and is about to fetch, media-use prints a one-line stderr hint when similar cached assets exist, pointing you back here.

Color grading

Use grade when you need the actual HyperFrames data-color-grading value to paste onto an <img> or <video>. Core presets and params-backed library looks resolve locally; future CDN-backed library looks require network unless already frozen:

Never cat/read a .cube file into context. A 3D LUT is ~size^3 lines of raw numbers (33^3 ≈ 36k lines at the default size). It bloats context and carries zero human/agent-legible signal. To understand or choose a LUT, use hyperframes grade-compare to see it rendered, or cube-validate.mjs for a one-line {ok,size} check. Read .media/index.md or luts/index.json for the description. Never read the LUT body itself.

node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json

Preset-first output uses the core runtime vocabulary and does not freeze a file:

{
  "preset": "warm-daylight",
  "intensity": 1
}

Paste it as an attribute value after JSON string escaping:

<video
  class="clip"
  src="./media/scene.mp4"
  data-color-grading='{"preset":"warm-daylight","intensity":1}'
></video>

Looks beyond the preset vocabulary freeze a validated .cube under .media/luts/ and return a block that references it:

node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "teal orange blockbuster" --project . --json

{
  "intensity": 1,
  "lut": { "src": ".media/luts/grade_001.cube", "intensity": 0.85 }
}

Use lut when you only need the reusable .cube file:

node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project .

For a describable technical look, author an explicit parametric LUT with --params:

node <SKILL_DIR>/scripts/resolve.mjs --type lut --params '{"contrast":0.2,"temperature":-0.3}' --project .
node <SKILL_DIR>/scripts/resolve.mjs --type grade --params '{"exposure":0.2}' --project . --json

For a LUT generated by your own script, ingest it with --from; media-use validates it before registration and rejects invalid or oversized cubes:

node <SKILL_DIR>/scripts/resolve.mjs --type lut --from custom.cube --project .

Parametric math (buildCube) cannot reproduce real film stocks or emulsion looks. Use a CDN-backed scanned .cube entry or ingest a real scanned .cube for those.

For visual selection, list reusable looks with resolve --type grade --candidates, write the promising entries to a grades.json, run hyperframes grade-compare --for <frame> --grades grades.json, then commit the winner with resolve -t grade as the final data-color-grading block.

Smart grade is grade --for <media>. It runs local ffmpeg/ffprobe signalstats, merges a bounded adjust suggestion into the returned block, and prints the measured evidence to stderr. Stdout remains valid JSON under --json; the suggestion is a starting point for the agent to tune, not an automatic neutralization of intentional color.

node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm cinematic" --for ./frame.png --project . --json

Library looks live in luts/index.json. Each entry keeps id, description, tags, and intensity, then supplies either compact params for on-demand buildCube(params) generation or a direct CDN url for future scanned .cube files. Do not commit generated .cube bodies; resolve validates generated or downloaded cubes as it freezes them under .media/luts/.

node skills/media-use/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project . --json
node skills/media-use/scripts/lib/cube-validate.mjs .media/luts/lut_001.cube

Providers

media-use holds no keys; every external tool owns its auth. Generation is centered on the HeyGen CLI free-usage path. Install and authenticate heygen before resolving bgm/sfx/image/icon/voice/avatar-video. Local tools are opt-in alternatives where they exist: mflux for image, Kokoro for voice, Parakeet for transcription, and LTX for local video generation. resolve spec-checks AVAILABLE RAM for those local ladders (describeModelLadder); the agent can see the ladder and override.

Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free, private, offline once cached). The codex CLI remains the ChatGPT-sub image upsell. Cost rule (X4): the agent confirms before an agent-initiated paid call; a user-requested one just runs.

To force a specific generator (e.g. a user says "make this image with codex"), pass --provider codex: it pins resolution to that provider and skips the free-usage default. See references/operations.md for the RAM ladders and provider recipes.

--local-only skips every network provider, including the free HeyGen ones, leaving the project + global cache and any installed local provider. For HeyGen-only types, that means no fresh resolve.

How it works

resolve runs an automatic floor, then falls through to fetching:

1. Check project .media/manifest.jsonl for a prompt match (case- and whitespace-insensitive) — auto-reuse
2. Scan existing assets/ directory for unregistered files that share a word with the need
3. Check global cache ~/.media/ for a reusable asset matched on the same normalized prompt — auto-reuse
4. Search via provider (HeyGen audio catalog, HeyGen asset search), or resolve color locally
5. Freeze file to .media/<type>/, register in manifest, regenerate index.md, auto-promote to ~/.media/

Steps 1 and 3 are the deterministic floor: they only auto-reuse an exact-normalized match, never a fuzzy one. Semantic reuse ("close enough") is the agent's explicit call via Reuse before you resolve — it never happens automatically. The agent gets back one line; candidates, scores, provenance stay on disk.

Adopt existing projects

Most HyperFrames projects already have assets in assets/. media-use adopts them:

node <SKILL_DIR>/scripts/resolve.mjs --adopt --project .
# → adopted 9 assets from assets/
#   bgm_001 → assets/bgm/mango-fizz.mp3 (bgm, 146.6s)
#   image_001 → assets/images/avatar.jpg (image, 400×400)

ffprobe extracts real duration and dimensions. During resolve, unregistered files in assets/ matching the intent are adopted on the fly.

Reading the inventory

After resolve or adopt, read .media/index.md for the full inventory:

# .media · 4 assets

id         type   dur   dims       path                          description
bgm_001    bgm    25s   -          .media/audio/bgm/bgm_001.mp3  upbeat tech launch
sfx_001    sfx    0.6s  -          .media/audio/sfx/sfx_001.mp3  whoosh
image_001  image  -     1920×1080  .media/images/image_001.jpg   gradient tech background
icon_001   icon   -     200×200    .media/images/icon_001.png    rocket

Cross-project reuse

Assets are cached automatically on resolve. Every resolved/ingested asset is auto-promoted to the global cache at ~/.media/, so subsequent resolves for the same (or near-identical) prompt, in any project, hit the cache with no re-download and no provider call.

For a semantically similar (not identical) need in another project, the exact-match floor won't fire — use Reuse before you resolve: --candidates lists the global assets, and --reuse <sha> imports the one you pick. This is how a track resolved in one project gets reused in the next when the wording differs.

Usage stats

Use resolve --stats for a local, shareable report over the current project's .media/ manifest, the global ~/.media/ cache, and local resolve misses. Human output is compact; add --json for a single machine-readable object, and --days N to window timestamped records.

node <SKILL_DIR>/scripts/resolve.mjs --stats --project . --days 7
# media-use stats
# total resolves: 12
# misses: 2
# hit rate: 86%

Files

• .media/manifest.jsonl: machine SSOT, one JSON record per line
• .media/index.md: agent-readable table (id, type, dur, dims, path, description)
• ~/.media/: global cross-project reuse cache (content-addressed, SHA-256)
• ~/.media/misses.jsonl: local-only resolve misses, including intent text for --stats

Audio engine: voiceover, music, SFX, captions, transcription

For a full audio pass (TTS voiceover + background music + sound effects in one shot), use the shared engine at audio/scripts/audio.mjs. It takes a neutral audio_request.json and writes audio_meta.json plus assets under .media/audio/{voice,bgm,sfx}:

node <SKILL_DIR>/audio/scripts/audio.mjs --request ./audio_request.json --out ./audio_meta.json

• Request { provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }: id joins each line back to your model; bgm.mode = retrieve | generate | none (omit for auto). --only tts,bgm,sfx runs a subset and merges into an existing --out.
• Output audio_meta.json (id-keyed): voices[].{path,duration_s,words[]} (word timestamps for captions), sfx[], bgm, total_duration_s.
• HeyGen free-usage path: HeyGen CLI auth unlocks TTS plus music/SFX retrieval. Local/provider-specific generators are explicit alternatives where installed; run node <SKILL_DIR>/scripts/resolve.mjs --doctor before assuming retrieval or TTS will work.
• If BGM took the generate path (bgm_pending: true), run audio/scripts/wait-bgm.mjs before final render.

Single-shot helpers: audio/scripts/heygen-tts.mjs (one voice file). Transcription / background removal / captions use the hyperframes CLI (transcribe, remove-background), see the per-topic guides in audio/references/ (tts.md, bgm.md, sfx.md, transcribe.md, remove-background.md, captions/).

Operating on media (cut, reframe, transform)

media-use resolves + remembers; for operating on assets see references/operations.md: local-tool recipes (ffmpeg trim/reframe/montage, auto-editor, scenedetect) and the local-vs-HeyGen transform table (background removal, upscale, lipsync, translate). Run the tool, then register the output with resolve --from <output> --type <type> so it joins the ledger + global cache.

CLI tools used (what to run, and how to enable each)

resolve auto-cascades; each provider shells one CLI. HeyGen is the free-usage path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar video, so those capabilities need heygen installed and authenticated. Local tools are OPT-IN alternatives where they exist; install one to unlock its free, private, on-device path instead of or ahead of HeyGen for that type. Only ffmpeg/ffprobe are strictly required for the tool to run at all.

The RAM-graded local-model shortlist + exact per-tier install/invoke lives in scripts/lib/local-models.mjs (the agent can read describeModelLadder(cap, specs) to see which model fits this machine). Without a tool on PATH, its provider prints a one-line diagnostic to stderr and resolve falls through where another provider exists (e.g. no mflux -> codex image upsell; no parakeet-mlx -> whisper.cpp).

heygen asset search is a pre-launch command hidden from heygen --help, but it runs; providers tag requests with the allowlisted X-HeyGen-Client-Source header (v0.3.0+).

Telemetry

resolve and the edit tools (transcribe / transcript-cut / audio-duck) send an anonymous usage event to PostHog (scripts/lib/telemetry.mjs), so we can see which capabilities are actually used. It records only the media TYPE, the resolution SOURCE, and the winning PROVIDER: never the intent text, file names, or paths, and $ip:null so no IP is stored. Best-effort and non-blocking (a resolve never waits on or fails from telemetry).

Opt out with DO_NOT_TRACK=1 or HYPERFRAMES_NO_TELEMETRY=1 (also off in CI and dev). Same public PostHog project key and opt-outs as the hyperframes CLI.

Privacy

media-use uses the same shared install id as the hyperframes CLI/studio (~/.hyperframes/config.json). When you are signed in to HeyGen, usage is linked to your account email, or username when email is unavailable, matching the CLI behavior. The events stay coarse: media type, source, provider, and small counts only; intent text and paths stay local. Disable telemetry with HYPERFRAMES_NO_TELEMETRY=1 or DO_NOT_TRACK=1.