# Subtitle Voiceover

> Runs a scripted movie dubbing pipeline that turns a video or audio file into a Burmese voice-over — AssemblyAI transcription with speaker diarization into a speaker-tagged SRT, agent-only translation that preserves the original timestamps, then edge-tts voices rendered per speaker (a female voice for female characters, a male voice for male characters), an optional agent-written opening hook, an editor-ready subtitle file, and a music mix long enough to cover the whole film; working files are cleaned up automatically. Use when the user wants a Burmese voice-over or dub from a film, a subtitle file to import into a video editor, subtitles with speaker labels, SRT translation, or a hook or cold-open line over a movie. Only the translation is done by the agent; every other stage is scripted Python.

- Skill: `aungminthu1722/subtitle-voiceover` (Agent Skill, multi-file: 88 files)
- Install (CLI): `npx skillmds@latest add aungminthu1722/subtitle-voiceover`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aungminthu1722/subtitle-voiceover/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: AungMinThu1722 (https://skillmd.com/u/aungminthu1722)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/aungminthu1722/subtitle-voiceover

---


# Subtitle Voice-over

Turn a movie's audio track into a Burmese voice-over that stays locked to the picture.

**The division of labour is fixed and must not be renegotiated:**

| Stage | Who runs it | Why |
| --- | --- | --- |
| 1. Media in + transcription + diarization (AssemblyAI) | `scripts/transcribe.py` | deterministic API work; pulls the speech track out of a video |
| 2. Speaker-tagged SRT | `scripts/srt_from_transcript.py` | deterministic formatting |
| 2b. Which voice each speaker needs | `scripts/detect_voices.py` | measured from the audio, never guessed from text |
| 3. **Translation into the target language**, optional character names and the **opening hook** | **the agent** | the only judgement step |
| 4. Voice-over synthesis (edge-tts) | `scripts/render_voiceover.py` | deterministic per-speaker TTS |
| 5. Voice-over track + hand-off | `scripts/render_voiceover.py`, `scripts/check_voiceover.py`, `scripts/export_timeline.py` | deterministic audio work |
| 5c. Subtitle readiness for the editor | `scripts/check_subtitles.py` | deterministic file checks |
| 6. Working-file cleanup | `scripts/cleanup.py` | deterministic housekeeping |

Never hand-transcribe, hand-write SRT, hand-edit timestamps, guess a speaker's gender from the text,
or synthesize audio yourself when a script in this skill already does it. The one exception is the hook
(Stage 3): that line is written by the agent, in the film's own voice. Voices are **measured** from
the audio by `scripts/detect_voices.py`; the agent's job at Stage 3 is the Burmese text (and, optionally, the
character names). Run the scripts for everything else and report what they printed.

## Setup

```bash
python3 -m pip install --user edge-tts pyyaml      # PyYAML is used to read the config
sudo apt-get update && sudo apt-get install -y ffmpeg   # ffmpeg + ffprobe, required for audio
```

Put the AssemblyAI key in a `.env` file next to this skill — `scripts/transcribe.py` loads it automatically
(still works with `export` too):

```
ASSEMBLYAI_API_KEY=your_key_here
```

`.env` is listed in `.gitignore`; never commit it or paste the key into a script.

Voice-over defaults (see `voiceover.config.yaml`): female = `my-MM-NilarNeural`,
male = `my-MM-ThihaNeural`. The Multilingual family has no Burmese voice yet, so these two remain
the working pair — swapping later means editing only the `voices:` block.

Measured on real Burmese output (2026-09): these voices speak at **~10-20 characters per second**
depending on the text, and the female voice runs slower than the male one, so `voice_settings:`
in the config carries a small per-voice `rate` (`+8%` on `female` by default). Two measured facts
shape the whole timing model, both handled by the scripts:

- **AssemblyAI `utterances[].start` includes the silence before a line** (measured: ~1.1 s early on
  the fixture), which would make the voice lead the picture. Cues are therefore built from
  **word-level** timings (`--timing words`, the default).
- **edge-tts pads every clip with up to ~1 s of digital silence** (measured: a 0.7 s line came back
  as a 1.87 s file). `scripts/render_voiceover.py` trims that silence before fitting, which cut "needs
  shortening" cues from 7/17 to 1/17 on the fixture.

**No music is mixed into the deliverable.** The pipeline ends with the voice-over stem
(`work/voiceover.mp3`, starting at 00:00:00 and padded to the film's length) plus the subtitle
files; the editor lays the score under the stem and owns the ducking. Do not add a music stage.

That is a workflow decision, not a limitation of Stage 1: **diarization is unaffected by background
music** — the fixture was transcribed clean and with a music bed at -14/-8/-4 dB, and every run found
both speakers with 100% assignment accuracy and 1.2% word error rate. A film's own score needs no
special handling before transcription.

## Workflow

### Stage 1 — transcribe with speaker detection

```bash
python3 scripts/transcribe.py movie.mp3 --work-dir work --language auto
```

- Uploads the file, requests `speaker_labels: true`, polls to completion, writes
  `work/transcript.json`, `work/transcript.raw.srt` and `work/transcribe.state.json`.
- If it times out or a run is interrupted, `--resume` continues the same job without re-uploading
  or re-billing.
- Tell the user to pass `--speakers-expected N` when the cast size is known, and `--language xx`
  when the film is not English; auto-detect handles the rest.

**Video input** (`mp4`, `mkv`, `mov`, ...): pass the film itself. The script probes it, extracts a
16 kHz mono speech track to `work/audio.16k.mp3` (a fraction of the film's size), uploads that, and
remembers both paths in `work/transcribe.state.json` (`audio` for the pitch analysis, `upload_source`
for what was sent). The film is never re-encoded. `--prepare-audio-only` does just the inspection and
extraction so you can see the plan before spending an API call; `--extract-audio always|never` forces
or forbids the extraction.

### Stage 2 — build the speaker-tagged SRT

```bash
python3 scripts/srt_from_transcript.py --work-dir work
```

Produces `work/movie.en.srt` with `[Speaker]` prefixes, splitting long utterances by sentence and
allocating time by character count, plus `work/speakers.json` and `work/speakers.csv`.

**Check the speaker stats with the user before translating.** Labels are `A`, `B`, `C` in order of
first appearance and are not consistent between films. See `references/speaker-mapping.md`.

### Stage 2b — decide the voice for each speaker (automatic)

```bash
python3 scripts/detect_voices.py --work-dir work
```

Isolates each speaker's own segments (using word timings), measures the fundamental frequency by
autocorrelation, and writes `work/voices.detected.yaml` — the higher voice gets the female voice, the
lower one the male voice, with a confidence rating and the measured pitch in the comments. Every later
stage loads this file automatically, so **no manual speaker mapping is needed**. A `speakers:` entry in
`voiceover.config.yaml` still wins whenever the user wants to pin one.

- Measured on this skill's fixture: speaker A 208 Hz → female, speaker B 160 Hz → male, both "high"
  confidence, matching the known ground truth.
- Without numpy it uses the same measurement in pure Python (`--method f0-pure`, ~1.5 s for a minute of
  audio). An earlier zero-crossing fallback was removed after it swapped the two fixture speakers.
- Report low-confidence results to the user and suggest `--limit 10` listening before the full render.
- Sets on a film the detector has not seen: 3+ speakers, or two speakers of the same gender, get a
  rate offset so they stay distinguishable.

### Stage 3 — translate (this is the agent's job)

```bash
python3 scripts/extract_for_translation.py --work-dir work --glossary glossary.txt
```

That writes `work/translation.my.json`: one entry per cue with `source`, `speaker`, read-only
`start`/`end`/`duration_ms`, and empty `text` / `tts_text` fields. Then:

1. Read the worksheet and `assets/translation-prompt.md` (the rules that apply to you).
2. Fill `text` for **every** cue with natural, speakable Burmese that fits the cue's duration.
   Use `tts_text` only when the readable form is hard to speak aloud (numbers, names, symbols).
3. Keep `index`, `start`, `end`, `duration_ms`, `speaker`, `source` exactly as given — the merge
   step re-attaches the original timestamps and discards any edit, but do not rely on that.
4. Optional but useful: fill `speaker_names` with the character behind each label (the worksheet
   provides `speaker_profiles` with sample lines and the already-detected voice role). Names only label
   the subtitles — they never decide a voice.
5. Merge back:

```bash
python3 scripts/apply_translation.py --input work/translation.my.json --work-dir work
```

Long films: `extract_for_translation.py --batch-size 120` splits the worksheet into
`work/batches/` and `apply_translation.py --from-dir work/batches` merges it back. Translate every
cue in every batch; the merge refuses to finish on missing cues unless `--allow-missing` is given,
and those slots end up silent.

### Stage 3b — the opening hook (optional, also the agent's job)

The hook replaces the film's first 5-10 seconds with one line the agent writes, so the voice-over
opens on a strong beat instead of a cold start. It is undetectable as long as it sounds like the
film: same character voice, same register, and it carries the information the cut opening carried.

```bash
python3 scripts/extract_for_translation.py --work-dir work --hook --hook-window-ms 8000
```

The worksheet then carries a `hook` block (schema `subtitle-voiceover/translation@2`) holding the
window (`window_start_ms`/`window_end_ms`), the `dropped_cues` and their `source_lines`, the
`next_dialogue_text` the hook must lead into, and its own `text` field for the agent to fill. The
worksheet's `cues` contain only the lines after the window.

Rules for the hook, in order of importance:

1. **Stay inside the character.** Write it as the character who speaks around it — never a narrator,
   never marketing, never a synopsis. If it could not be a line of this film's dialogue, rewrite it.
2. **Do not spoil anything** that has not happened yet.
3. **Carry the cut information, do not contradict it.** The dropped lines are shown to you; when the
   next surviving line refers back to them ("30 seconds is all we have"), the hook has to keep that
   beat alive — see `references/hook-guide.md` for a worked example.
4. **Fit the window**: 5-8 characters per second (the script warns above 13 cps), one sentence or two
   short ones, and no repetition of the line that follows it.
5. Dialogue may only resume after `window_end_ms`; the merge drops every cue that starts earlier.

`hook.voice` in the config (or the default, the voice of whoever speaks first after the window)
decides which voice speaks the hook; the hook itself is written into the SRT **without** a speaker
label, so it reads as the film's own opening line.

Details on length, register and subtitle-versus-speech choices: `references/translation-guide.md`.

### Stage 4 — render the voice-over

```bash
python3 scripts/render_voiceover.py --work-dir work --jobs 4
```

For each cue the renderer picks the voice mapped to that cue's speaker, synthesizes with edge-tts,
and fits the clip into the cue's slot without moving it: audio longer than the slot is sped up with
`atempo` (pitch preserved) up to `tts.max_tempo`, then trimmed with a short fade if it still
overruns. Outputs:

| File | Purpose |
| --- | --- |
| `work/voiceover.mp3` | the full track, starting at 00:00:00, movie-length |
| `work/clips/cue-NNNN.mp3` | per-cue clips (plus `.fit`/`.final` variants when a cue was adjusted) |
| `work/voiceover.cues.json` | per-cue report: voice, duration, speed-up, trims, failures |

Render a sample first on a new film (`--limit 20`, or `--cues 1,5,9`) and let the user listen before
committing to the whole feature. Re-run with `--reuse-clips` to keep finished clips.

### Stage 5 — verify and hand off

```bash
python3 scripts/check_voiceover.py --work-dir work
python3 scripts/export_timeline.py --work-dir work --format all --fps 25
```

The track is **the deliverable**: it starts at 00:00:00, is padded with silence to the film's own
length (from `work/transcribe.state.json`; `--dialogue-length` ends it at the last spoken line,
`--length-ms` sets it explicitly), and carries no music. State its length when you report to the
user — that is what they drop on the timeline. The checks catch missing audio, a track that is too
short for the dialogue, unmapped speakers and lines that are too dense to read or speak. The export writes `work/voiceover.fcpxml` (Premiere Pro,
Final Cut Pro, DaVinci Resolve), `work/voiceover.edl` and `work/voiceover.cue-list.csv`, each clip
placed at its original timecode.

### Stage 6 — clean the working files

```bash
python3 scripts/cleanup.py --work-dir work            # after the deliverables exist
python3 scripts/cleanup.py --work-dir work --dry-run  # see what would go first
```

Removes regenerable material — `work/clips/`, the extracted speech track, `*.fit.*`, `*.speech.*` and
`work/batches/` — and keeps every deliverable (the voice-over stem and its report, both subtitle
tracks, the hand-off files, the transcript, the translations, the detection and the logs). It never touches the film itself. `--keep-clips` keeps the per-cue clips,
`--all` also removes reports and the worksheet, `--dry-run` prints the plan. The report lands in
`work/cleanup.report.json`.

### One command for the whole run

```bash
python3 scripts/run_pipeline.py movie.mp4 --work-dir work --config voiceover.config.yaml \
    --hook --stages python        # transcribe -> SRT -> detect -> worksheet
#   ... the agent fills work/translation.my.json, then ...
python3 scripts/run_pipeline.py --work-dir work --config voiceover.config.yaml \
    --stages post                 # apply -> render -> checks -> hand-off -> cleanup
```

`--stages post` refuses to render a worksheet that still has empty cues, or a hook block with no
text, so a half-finished translation cannot reach the renderer. `--dry-run` prints the plan,
`--skip stepname` resumes past a step that already ran, and every step's output is kept in
`work/pipeline.log`. The run ends with a summary of the deliverables and their measured numbers.

## Rules that keep the dub in sync

- **Timing is owned by the scripts.** Cue start/end come from AssemblyAI and are never recomputed
  from translated text. If a line does not fit, the fix is a shorter translation, not a moved cue.
- **Voice choice is measured, not inferred.** `scripts/detect_voices.py` decides from the audio; the config
  overrides it; nothing ever guesses gender from a character's name or from the Burmese text.
- **One pass, one language.** `output.target_language` drives the filenames (`movie.my.srt`,
  `movie.th.srt`, ...), so several language versions can coexist in one `work/` directory.
- **No music stage.** The score, the ducking and the loudness of the finished programme belong to the
  editor's timeline; the pipeline delivers the voice-over stem and stops there. If a user asks for a
  mixed file, explain what they get instead and hand them `references/editor-handoff.md`.
- **Never re-run Stage 1 to fix a Stage 3/4 problem.** Re-transcribing costs money and changes the
  speaker labels; fix the mapping, the translation, or the render instead.

## Reporting to the user

After each stage, state the artefact path and the one number that matters: cues and speakers from
Stage 2, cues translated and any dense lines from Stage 3, cue count plus sped-up/trimmed/failed
counts from Stage 4, then the checks result and the hand-off files from Stage 5. Do not paste a
whole SRT or the track listing unless asked.

## Bundled resources

- `scripts/vo_common.py` — SRT parsing, config, ffmpeg and edge-tts helpers (imported by the others).
- `scripts/transcribe.py` — Stage 1, AssemblyAI with diarization and resume support.
- `scripts/srt_from_transcript.py` — Stage 2, speaker-tagged SRT and speaker statistics.
- `scripts/detect_voices.py` — Stage 2b, measures each speaker's pitch and assigns the voice.
- `scripts/extract_for_translation.py` — Stage 3a, the worksheet the agent fills in.
- `scripts/apply_translation.py` — Stage 3b, merges translations onto the original timeline.
- `scripts/render_voiceover.py` — Stage 4, per-speaker edge-tts and the film-length stem.
- `scripts/check_voiceover.py` — Stage 5a, quality checks.
- `scripts/export_timeline.py` — Stage 5b, FCPXML / EDL / CSV hand-off.
- `scripts/check_subtitles.py` — Stage 5c, editor-readiness checks on the delivered SRT.
- `scripts/cleanup.py` — Stage 7, removes working files and keeps the deliverables.
- `scripts/run_pipeline.py` — runs the whole pipeline in order, with a plan, a log and a summary.
- `voiceover.config.yaml` — voices, speaker map, timing tolerances; copy per project.
- `references/speaker-mapping.md` — how detection works, unusual casts, pinning by hand, diarization problems.
- `references/translation-guide.md` — length budgets, register, subtitles versus speech.
- `references/edge-tts-voices.md` — Burmese and Multilingual voice list, how to swap the pair.
- `references/troubleshooting.md` — setup, pipeline and quality failures with fixes.
- `references/hook-guide.md` — how to write the opening hook so nobody notices it was added.
- `references/editor-handoff.md` — importing the audio and subtitles into Premiere, Resolve or FCP.
- `assets/translation-prompt.md` — the translation contract given to the agent.
- `assets/glossary.example.txt` — glossary format for names and terminology.
- `tests/run_tests.sh`, `tests/make_fixture.py` — offline smoke tests (fixture transcript, real TTS).
- `tests/make_dialogue_audio.py` — builds a two-speaker English dialogue + music bed + ground truth.
- `tests/score_vs_ground_truth.py` — scores a transcript against that ground truth (speakers, WER, timing).
- `demo-film/` — two complete real-API runs of the 53 s fixture: the audio run, and the video run with
  an opening hook, an editor-ready SRT and the cleanup, both with measurements.

## Example: five minutes of film

```bash
python3 -m pip install --user edge-tts pyyaml && export ASSEMBLYAI_API_KEY=sk-...
python3 scripts/transcribe.py scene.mp3 --work-dir work --speakers-expected 2
python3 scripts/srt_from_transcript.py --work-dir work        # 84 cues, speakers A and B
python3 scripts/detect_voices.py --work-dir work              # A -> female, B -> male (measured)
python3 scripts/extract_for_translation.py --work-dir work    # worksheet for translation
# agent translates every cue, then:
python3 scripts/apply_translation.py --input work/translation.my.json --work-dir work
python3 scripts/render_voiceover.py --work-dir work --limit 20   # sample, let the user listen
python3 scripts/render_voiceover.py --work-dir work              # full track
python3 scripts/check_voiceover.py --work-dir work
python3 scripts/export_timeline.py --work-dir work --format all
python3 scripts/cleanup.py --work-dir work          # after the stem and subtitles exist
```

## Example: a video, with a hook, ending in a clean work directory

```bash
python3 scripts/run_pipeline.py night-shift.mp4 --work-dir work --config voiceover.config.yaml \
    --language en --speakers-expected 2 --hook --hook-window-ms 8000 --stages python
# agent: read work/translation.my.json, fill every cue's text AND hook.text, keep the timestamps
python3 scripts/run_pipeline.py --work-dir work --config voiceover.config.yaml --stages post
```

Result, measured on this skill's own 53-second demo film: hook 0.72-8.72 s spoken by the male lead
(62 characters, 7.8 cps, the four cut opening cues replaced), dialogue resuming at 11.48 s with the
line that answers the hook, 14 cues rendered with no failures, a music-free stem of 53.26 s (the
film's length, not the 51.48 s of dialogue), subtitles `READY`, and 30 working files removed
afterwards. The full record is in `demo-film/work-video/` and
`demo-film/README.md`.