Ffmpeg Skill

ffmpeg-skill

Scripts live in scripts/ next to this file; run them with python3 <skill-dir>/scripts/<name>.py. Every script has --help, and all of them accept --dry-run, --json (structured result with a probe of the output), --fast (preview quality) and --progress. Writing tools run nothing under --dry-run; probe/check/sync/multicam/scenes/cropdetect/report may still run ffmpeg/ffprobe to measure or analyse — they just don't write their final artifact; verify accepts the flag but ignores it. Exact per-tool semantics: contract --json's dry_run field (or docs/contract.md). Details for every flag: references/scripts.md. Device-specific behaviour (iPhone HDR, GoPro, DJI, screen recordings, Zoom): references/devices.md.

Workflow (always follow this order)

0. Check the environment once per session, if unfamiliar. On a machine you haven't confirmed capability on this session, run doctor --json once: check ok and the target tool's usable before relying on it. If usable isn't yes, don't run that tool — report the missing capability instead of discovering it via a runtime failure (a missing libass, zscale, or encoder is the common case, e.g. caption.py). Don't re-run doctor per job — it queries ffmpeg -filters/-encoders, not free, and once per session/unfamiliar machine is enough. contract --json's full tool schema is for a planning agent deciding which tool/params to use from an abstract goal — not part of this per-job workflow.
1. Probe first. Run probe.py on every input before touching it. Read the duration, fps, resolution, codecs, audio channels and the variable_frame_rate_suspected flag. Plan the edit from real numbers, never from assumptions about the file.
2. Prefer lossless. If the request can be satisfied without re-encoding (plain cuts on keyframes, remuxing, audio-only changes), do not re-encode. cut.py and loudness.py stream-copy video by default; only pass --accurate to cut.py when the user needs frame-exact cuts.
3. Plan with --dry-run --json, then execute. Every script accepts --dry-run (prints the ffmpeg commands that would run) and --json (structured result: output path, probe of the output, commands run). For writing tools this means nothing is written; probe/check still run ffprobe/loudness-measurement passes (they're read-only, so --dry-run changes nothing for probe, and only skips the loudness pass for check), sync/multicam/scenes/cropdetect/report still run ffmpeg/ffprobe to measure or analyse, and verify accepts the flag but ignores it entirely (its steps run regardless) — see contract --json's dry_run field per tool for exact semantics. Trust --json, not a dry-run's human-readable summary line, for any number after the plan (dimensions in that line can be a placeholder, not a computed preview — see docs/contract.md). Use them to confirm a plan before long encodes and to report exact facts. --fast gives a quick preview-quality render (x264 veryfast), --progress prints percent and ETA on stderr for long encodes.
4. Chain operations in a sensible order. Colour (HDR→SDR / LUT) → cut → join → silence → fit → caption/overlay → sync → audio → loudness → export. Do frame changes (fit/crop) before captions and overlays so text is sized for the final frame. Re-encode as few times as possible: keep intermediates at CRF 18 (the default) and only use export.py for the last step; for anything with more than two steps use render.py with a project.json.
5. Check the deliverable. Before reporting, run check.py OUTPUT --platform X for the destination the user named. Each row is marked format or judgement. Format rows (codec, pixel format, size, true peak, colour tags, VFR) are safe to fix mechanically. Judgement rows change the content: duration (cut loses material, speed changes motion), aspect (crop loses edges), fps (drops motion), loudness (ambience must not be boosted). Fix those only when the user's request already implies the answer, otherwise state the choice and its cost in one line. Mention WARNs; do not chase them.
6. Verify the output. Run probe.py on each result and confirm duration, resolution, fps and audio match what was requested. Report those numbers to the user (e.g. "final.mp4: 59.98 s, 1080x1920, 30 fps, AAC stereo"). A step is done only when the script exited 0 and the output probes as expected. Writing the command is not doing the job; a non-zero exit, a missing or empty file, or a probe that contradicts the request is a failure, and the report says so with the script's error message.
7. Keep the user's originals. Never overwrite the source file. Write new files next to the input or where the user asked.
8. Look at the picture. Whenever the picture changed (captions, overlays, graphics, crop/pad, resize, colour, transitions) run look.py OUTPUT (contact sheet) or look.py OUTPUT --at T, view the PNG. The job is not finished until the report's Look: line names that PNG; a probe alone cannot see a caption sitting on someone's face. Audio-only jobs (sync, loudness, silence, or any job whose input is an audio file) write Look: not needed; there is no picture to inspect. What to look for splits the same way check.py's rows do in step 5:
   • Mechanical (verify and report as this skill's own job): the specified text/logo is present at the specified position, subtitles/text appear at the specified timestamps, resolution has even dimensions. Letterboxing/pillarboxing from fit.py --fit pad is the correct result of that mode, not a defect — never flag it.
   • Judgement (report to the calling agent/user, don't silently pass or fail): whether a subject or face is cut off, whether text sits over a face, whether colours look washed out, whether a transition "lands" well or the edit feels cinematic. These require deciding what the subject is, which belongs to the calling agent (see "What this skill does and does not decide") — state what you see in one line and let the calling agent or user judge it, don't decide it here. If the execution environment cannot actually view images (no vision capability), write Look: PATH (pixels not inspected; agent has no image view) — never claim a picture was inspected when it wasn't, and don't stall indefinitely waiting for a capability that isn't there.

Before you run anything: what to ask, what to assume

Ask one short question only when the answer changes the output materially and the request does not imply it:

• Destination decides aspect, length limit, loudness and codec. "For Reels" answers all four. If no destination is named and the edit is a plain cut/caption, keep the source format and say so; if the user asks to "export", "post" or "deliver", ask where.
• Duration ("make it 60 s") without a method: speed up for ≤1.5× changes, trim otherwise, and state which you chose. Ask if the content is a talk (trimming loses words) and the change is large.
• Captions without a text source: use --transcribe if a local whisper exists, otherwise ask for the text or a timed file; never invent dialogue.
• Fonts and brand: if the user mentions a brand, colours or "our font", ask for or create brand.json once and reuse it.
• CJK / non-Latin text: check that a font exists before rendering (fc-list :lang=ja file / :lang=ko / :lang=zh); pass it with --font "Name" or --font-file /path.ttf. Tofu boxes are a failed job, not a style.
• Crop position for --fit crop: default to centre, but if the request or the source names an off-centre subject ("keep the product on the right", "don't cut off my hands", a logo/person visibly off-centre in look.py's sheet) use --crop-x/--crop-y (0=left/top, 1=right/bottom) instead of the silent centre guess. Ask which edge to keep when the sheet shows the subject near an edge and the request doesn't say.
• Anything else (transition type, caption style): pick the conventional default, say what you picked, and offer the alternative in one line.

Do not ask for things probe.py can tell you.

What this skill does and does not decide

This skill cuts, joins, measures, syncs, exports and checks files — it executes an edit, it does not decide one. Some things that sound like part of the job but belong to the human, the calling agent, or another skill instead:

• Which cut is the right one, or whether a deliverable is approvable for release — this skill measures and reports (check.py's PASS/WARN/FAIL, cut.py's measured duration error); a production agent or the user decides whether that's good enough to ship.
• What makes a highlight interesting — scenes.py --highlights ranks by a measured proxy (audio energy or scene duration, see its own docs), never by understanding the content; treat its output as candidates, not a verdict.
• Thumbnail or cover-image composition — that's a design decision, not a measurement; a thumbnail-generation skill or the user makes it.
• Understanding what a video is about — this skill has no transcription or vision beyond look.py's contact sheets, which exist for the calling agent's own eyes, not for this skill to interpret on its own.
• Judging what looks good — "apply this LUT" or "correct exposure by +0.3 stops" (color.py) is mechanical, parameter-determined execution and belongs here; "grade this scene to look cinematic" is a subjective judgement about what looks right and belongs in a colour-grading skill (color-grading-skill, see README's "Standalone, and in an ecosystem") that decides the parameters and then calls color.py to apply them.
• Picking a subject or region without being told one — "crop to this exact box" or "crop to 9:16 keeping x=200,y=0" (crop.py/fit.py --fit crop --crop-x/-y) is mechanical once the box is known; "crop to keep the speaker in frame" requires deciding what the speaker is, which is a vision/composition judgement for the calling agent (from a look.py contact sheet) or a motion-graphics skill, not this one.

The line in general: if the same input and the same explicit parameters always produce the same, verifiable output, it belongs here. If the "right" answer depends on taste, content understanding, or what looks or sounds good, it belongs to whichever skill or agent makes that judgement — this skill only ever executes parameters it's given, never infers them from what something looks or sounds like.

If a request needs an FFmpeg feature none of the 28 scripts expose, say so and name the closest built-in option (--dry-run to show what would run, or a documented limitation) — never fall back to guessing a raw ffmpeg/ffprobe invocation or a hand-built filter graph outside scripts/*.py. A raw command bypasses every guarantee this skill makes (no shell, typed arguments, verification afterwards); it is exactly the failure mode this skill exists to prevent, so it is never the fallback when a script's flag doesn't cover something.

Request → script

Audio-only files

Audio files are a first-class input, not a special case. probe.py, cut.py, silence.py, loudness.py, audio.py, sync.py and check.py --platform podcast all accept WAV, FLAC, MP3, M4A/AAC, OGG and Opus (any container ffmpeg can read) and write the codec that fits the output extension, so the same commands work with talk.wav in place of talk.mp4. What changes:

• The output extension picks the format: -o out.mp3 converts, -o out.wav keeps PCM, -o out.m4a writes AAC. audio.py in.wav -o out.mp3 with no other flag is a plain conversion.
• cut.py stream-copies audio too, so trims land on a packet boundary (precision: packet, a few ms; the JSON reports duration_error_ms). Pass --accurate for a sample-exact trim: precision: sample when the output is PCM or FLAC, codec_frame when a lossy codec (AAC, MP3, Opus) frames it again. A .wav output is always PCM, never AAC packets inside a WAV.
• join.py joins audio-only clips as audio (acrossfade or a butt join) at one sample rate and channel layout; the output must have an audio extension. Video and audio clips cannot be mixed in one join.
• An audio extension on a video input (audio.py talk.mp4 -o talk.wav, cut.py talk.mp4 --start 1:00 --end 2:00 -o part.wav) extracts the audio; the output has no video stream. audio.py --audio-stream N picks a track when probe lists several under audio_streams.
• Look: not needed in the report; Check: still applies for loudness (check.py file.wav --platform podcast measures LUFS and true peak).
• Scripts that need a picture (fit, caption, overlay, graphics, color, export, scenes, look) refuse an audio file with "input has no video stream". Say so instead of forcing a video wrapper.

Report format

Reply in the language the user wrote their request in — a Japanese request gets a Japanese report, English gets English, Chinese gets Chinese, and so on for any other language. Keep the shape below and the field labels (Done:, Steps:, Check:, Look:, Notes:) in English (they read like log fields, not prose, and stay recognisable across languages); the sentences around them, any question asked, and any explanation of a judgement call are in the user's language. Never default to English because the tool names and flags happen to be English. A mid-conversation language switch follows the user's latest message, not the first one.

Finish every job with this shape (numbers from probe.py/check.py, not memory):

Done: final.mp4 — 59.98 s, 1080x1920, 30 fps, H.264, AAC stereo, -14.1 LUFS
Steps: cut 0:12-1:12 (lossless) -> fit 9:16 crop -> captions (pop, karaoke) -> loudness -14 -> export reels
Check: reels — all 12 checks pass
Look: final_sheet.png (captions inside the safe area, logo top-right)
Notes: source was VFR, conformed to 30 fps; audio was mono, made stereo

Keep it to those five lines plus anything the user must decide. Attach the contact sheet when the edit touched the picture. Never report success without the probe of the output; never describe a fix you did not run.

When a step fails, replace Done: with Failed: and keep the rest honest:

Failed: color.py --lut grade.cube exited 1 — ffmpeg: "Unable to parse LUT file" (the .cube is not a valid LUT)
Steps: probe -> color (failed); nothing written
Notes: send a valid .cube, or say if you want the clip left as is

Every script prints {"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool, "message": ...}} with --json and exits non-zero; quote the message, do not paraphrase it into a success.

Things that look right but are wrong

• Re-encoding an HDR (iPhone, HDR10) source through the SDR path: colours go flat. The scripts keep HDR; if you hand-write ffmpeg, do not tag BT.709 on BT.2020 pixels.
• Lossless -c copy cuts on VFR or non-keyframe boundaries: the file "works" but starts on a frozen or wrong frame. cut.py re-encodes automatically when the snap exceeds 0.5 s; respect that.
• A sync or multicam alignment with confidence under 0.3, or an offset larger than 60 % of the analysis window: probably wrong; enlarge --analyze-seconds or find a clap. multicam.py reports one confidence per camera — check all of them, not just that the command succeeded, before trusting the cut.
• sync.py/multicam.py align audio tracks to each other, never lip sync (mouth movement vs. audio) — there is no face or mouth detection anywhere in this skill. A high confidence means the audio matched well, not that the picture looks right; if the user asks whether lip sync is correct, that needs a look at the actual video, not just the reported offset.
• "Normalised" audio that still clips: check true peak, not just LUFS (check.py does both).
• Normalising ambience or near-silence to a speech target: a clip measured at -40 LUFS or below is room tone, wind or nothing; raising it 25 dB raises the noise, not the content. Leave the level, say so, and offer music or narration.
• Captions burned before a crop/resize: text lands off-frame. Frame changes first, then text.
• Anything chained by hand through three re-encodes: use render.py so the plan is one file and the user can change one number.
• --fit crop to reach 9:16 from 16:9 throws away 70 % of the width: a wide shot loses people at the edges. Check the sheet; pad (bars), --crop-x/--crop-y toward the subject, or a reframe is often the honest answer — a silent centre crop is a guess, not a decision.
• Conforming 60 fps to 30 halves the motion samples: fine for a talking head, visibly choppy for sports, gaming, drone pans. Keep 60 when the platform allows it.
• "Make it 60 seconds" on a 3-minute talk by speed change is unwatchable (3×); by trim it drops two thirds of the words. Ask which, or propose a highlight cut with scenes.py.
• scenes.py --highlights defaults to the loudest scenes (--rank-by audio): a quiet but important moment (a confession, a punchline landing in silence) is skipped, and pure crowd noise or a mic bump can outrank it. --rank-by duration picks the longest unbroken scenes instead. Neither is "the best parts" — check the contact sheet (--sheet) before treating the picks as final.

Gotchas

• Variable frame rate (phone/screen recordings). probe.py sets variable_frame_rate_suspected when r_frame_rate and avg_frame_rate disagree. Every re-encoding script then adds -fps_mode cfr at the source's average rate, and cut.py switches itself to --accurate (copy-cuts on VFR are unreliable). Pick the rate explicitly with fit.py --fps 30|60 when the average is odd (e.g. 23.4 fps from dropped frames).
• Audio drift / sync. Don't mix files with different frame rates or sample rates in one cut.py --segments join without re-encoding (--accurate). After sync.py, verify by running it again on the output: offset (and drift ppm with --fix-drift) should be ~0. Recordings longer than ~10 minutes from separate devices: always use --fix-drift.
• Colour. SDR outputs are H.264 tagged BT.709 yuv420p. When probe.py reports hdr: true (HDR10/PQ, HLG, Dolby Vision, BT.2020), every editing script keeps the output HDR (HEVC Main10, source colour tags) so nothing is silently flattened. Decide with the user: keep HDR (fine for YouTube/phones) or run color.py --to-sdr first for SDR-only destinations, LUT work or H.264 deliverables. export.py platform presets are SDR and warn on HDR input. iPhone .mov files also carry timecode/metadata tracks; scripts map only the first audio track, so extra tracks are dropped on re-encode. For Log footage (S-Log, V-Log, C-Log: looks grey and low-contrast but is tagged SDR) run probe.py --analyze; looks_like_log: true means apply the manufacturer's .cube with color.py --lut before anything else. Keep ProRes masters at source colour: export.py --preset prores does not retag.
• CJK and other non-Latin text. libass and drawtext need a font that has the glyphs. Check with fc-list | grep -i cjk. Then either name it (caption.py --font "Noto Sans CJK JP") or point at the file (overlay.py --font-file /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc, caption.py --fonts-dir ./fonts --font "Noto Sans CJK JP"). Without a matching font you get boxes, not an error. Install: apt install fonts-noto-cjk, brew install --cask font-noto-sans-cjk.
• Windows drawtext crashes on some real builds. On certain Windows ffmpeg builds (e.g. winget's gyan.dev), drawtext crashes with an access violation whenever it resolves a font by family name through fontconfig, even with a valid fonts.conf (#100). look.py, scenes.py --sheet, overlay.py --text and graphics.py all resolve a concrete --font-file by default when one is available (fontfile= skips fontconfig entirely and is the form confirmed not to crash), so this should already be handled automatically. If a drawtext tool still crashes, pass --font-file explicitly rather than relying on --font/font= resolution; doctor also runs a real one-frame drawtext probe and reports filter:drawtext missing (with the crash detail in errors[]) rather than a false "available" from the -filters listing alone.
• Keyframe cuts. A lossless cut.py result may start up to one GOP (often 1–10 s) earlier than requested; the script re-encodes automatically when the deviation exceeds 0.5 s. If the user insists on lossless output, pass --tolerance -1 and tell them the cut lands on the nearest earlier keyframe.
• Rotation metadata. Phone footage often has a rotation tag; probe.py reports it and fit.py accounts for it when computing the output frame.
• Odd dimensions. yuv420p needs even width/height; fit.py and export.py round to even values automatically.
• Speed. Re-encodes use x264 medium. For long files add --preset veryfast to intermediates and keep the default for the final export.