` (no `.en` suffix) 2. Language known and English → `--model small.en` 3. Language unknown → `--model small` (no `.en`, no `--language`) — whisper auto-detects **Default model is `small`, not `small.en`.** ### Model Sizes | Model | Size | Speed | When to use | | ---------- | ------ | -------- | ------------------------------------- | | `tiny` | 75 MB | Fastest | Quick previews, testing pipeline | | `base` | 142 MB | Fast | Short clips, clear audio | | `small` | 466 MB | Moderate | **Default** — most content | | `medium` | 1.5 GB | Slow | Important content, noisy audio, music | | `large-v3` | 3.1 GB | Slowest | Production quality | Music with vocals: start at `medium` minimum; produced tracks often need manual SRT/VTT import. For caption-quality checks (mandatory after every transcription), the cleaning JS, retry rules, and the OpenAI/Groq API import path, see [framevideo/references/transcript-guide.md](../framevideo/references/transcript-guide.md). ### Output Shape Compositions consume a flat array of word objects. The `id` field (`w0`, `w1`, ...) is added during normalization for stable references in caption overrides; it's optional for backwards compatibility. ```json [ { "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 }, { "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 } ] ``` ## Background Removal (`remove-background`) Remove the background from a video or image so the subject (typically a person — avatar, presenter, talking head) sits as a transparent overlay in a composition. ```bash npx framevideo remove-background subject.mp4 -o transparent.webm # default: VP9 alpha WebM npx framevideo remove-background subject.mp4 -o transparent.mov # ProRes 4444 (editing) npx framevideo remove-background portrait.jpg -o cutout.png # single-image cutout npx framevideo remove-background subject.mp4 -o subject.webm \ --background-output plate.webm # both layers in one pass npx framevideo remove-background subject.mp4 -o transparent.webm --device cpu npx framevideo remove-background --info # detected providers ``` Uses `u2net_human_seg` (MIT). First run downloads ~168 MB of weights to `~/.cache/framevideo/background-removal/models/`. ### Layer separation (`--background-output`) Pass `--background-output` (or `-b`) to emit a **second** transparent video alongside the cutout: same source RGB, alpha is `255 − mask` instead of `mask`. The cutout is the subject with a transparent background; the plate is the original surroundings with a transparent hole where the subject was. | File | Alpha is… | Use it for | | -------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------- | | `-o subject.webm` | The mask — subject opaque, background transparent | Foreground layer, place on top | | `--background-output plate.webm` | Inverse — surroundings opaque, subject region transparent | Bottom layer; put text or graphics between this and the subject | Both outputs share the same `--quality` preset and run from a single inference pass — encode cost roughly doubles, segmentation cost stays the same. Only valid for video inputs and `.webm`/`.mov` outputs. **Hole-cut plate, not an inpainted clean plate.** The subject region in `plate.webm` is fully transparent — composite something opaque under it to fill the hole. The single test for whether `--background-output` is the right tool: _will anything ever be visible through the subject's silhouette where the subject used to be?_ | Use case | Right tool | | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | Text/graphics between the cutout and the plate (this command's reason for existing) | **Hole-cut** (`--background-output`) | | Subject onto an unrelated scene | Just `subject.webm`; ignore the plate | | Show the room _without_ the person, alone over no other content | **Clean plate** — needs an inpainter (LaMa, ProPainter, E2FGVI). Not this command. | | Replace the subject with a different subject | **Clean plate** — same as above | If a user asks for "the room with the person removed" and intends to display it standalone, do **not** reach for `--background-output`. Tell them they need an inpainter. Typical layered composition (the canonical hole-cut use case): ```html MAKE IT IN FRAMEVIDEO ``` This is functionally equivalent to the text-behind-subject pattern below, but you don't need the original `presenter.mp4` in the project — the plate replaces it. Useful when you want to ship just the two transparent layers and let the user drop arbitrary content between them. ### Output Format | Format | When | | --------------------- | ------------------------------------------------------------- | | `.webm` (VP9 + alpha) | Default. Compositions play this directly via ``. | | `.mov` (ProRes 4444) | Editing in DaVinci/Premiere/FCP. Large files. | | `.png` | Single-image cutout (still subject, layered over a backdrop). | Chrome decodes VP9 alpha natively, so the `.webm` plugs into a composition like any other muted-autoplay video — see the `framevideo` skill for the `` track conventions. ### Quality presets `--quality fast|balanced|best` controls only the VP9 encoder's CRF — segmentation quality is fixed. | Preset | CRF | When | | ---------- | --- | ----------------------------------------------------- | | `fast` | 30 | Iterating, smaller file, looser color match | | `balanced` | 18 | Default. Visually identical for most uses | | `best` | 12 | Master / final delivery. Largest file, tightest match | ### Compositing patterns — pick the right one The cutout webm is a **re-encoded copy** of the source mp4's RGB. That choice has consequences depending on what you put behind it: | Pattern | What's behind the cutout | Result | | -------------------------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Cutout over a different scene** (most common) | Static image, gradient, or unrelated video | Looks great. The cutout's RGB is the only source of the subject — no doubling, no edge halo. This is what `remove-background` is built for. | | **Cutout over its own source mp4** (text-behind-subject) | Same mp4 the cutout was generated from | Two RGB sources for the same person. At default `--quality balanced` (crf 18) the doubling is barely visible; at `--quality fast` (crf 30) you'll see a faint color shift / edge halo. Use `--quality best` (crf 12) for masters. | | **Cutout over a _different_ take of the same person** | Footage of the same subject | Will look like two separate people overlapping. Don't do this. | **Text-behind-subject** (headline behind a presenter): ```html MAKE IT IN FRAMEVIDEO ``` Two key rules: 1. **Wrap the cutout video in a non-timed ``** and animate the wrapper's opacity, not the video element's. The framework forces opacity:1 on active clips (any element with `data-start`/`data-duration`), so animating the video's opacity directly is silently overridden. The wrapper has no `data-*` attributes, so it's owned by your CSS/GSAP. 2. **Both videos use `data-start="0"` and `data-media-start="0"`** so the framework decodes them in sync from t=0. Late-mounting the cutout (`data-start=3.3`) introduces a seek + warm-up that lands a frame off the base mp4 — visible as one frame of misalignment at the cut. Then GSAP-flip the wrapper opacity at the cut: `tl.set(cutoutWrap, { opacity: 1 }, 3.3)`. ## TTS → Transcribe → Captions When there's no pre-recorded voiceover, generate one and transcribe it back to get word-level timestamps for captions: ```bash npx framevideo tts script.txt --voice af_heart --output narration.wav npx framevideo transcribe narration.wav # → transcript.json ``` Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. --- ## Validation After preprocessing assets: ```bash # Verify audio files npx framevideo inspect # Check audio levels and duration # Test in preview npx framevideo preview # Verify audio plays correctly in composition ``` **Manual checks:** 1. **File paths** — assets saved to correct directory (`assets/music/`, `assets/sfx/`, etc.) 2. **Audio levels** — BGM at 0.12-0.22, SFX at 0.6-1.0, narration at default 3. **Transcription accuracy** — review `transcript.json` word boundaries 4. **Background removal quality** — check transparent video for edge artifacts 5. **Integration** — verify asset referenced correctly in composition HTML --- ## Integration This skill produces assets consumed by: - **framevideo** — reference audio/video in composition HTML - **framevideo-voiceover-ssml** — apply SSML markup before TTS generation - **chanjing-digital-human** — alternative to local TTS for voiceover - **website-to-framevideo** — Step 4 uses this skill for audio generation --- ## Credits And References - Kokoro TTS: https://github.com/hexgrad/kokoro - Whisper transcription: https://github.com/openai/whisper - u2net background removal: https://github.com/xuebinqin/U-2-Net - Chanjing platform: https://www.chanjing.cc