Subtitle Voiceover

Subtitle Voice-over

Turn a movie's audio track into a Burmese voice-over that stays locked to the picture.

The division of labour is fixed and must not be renegotiated:

Never hand-transcribe, hand-write SRT, hand-edit timestamps, guess a speaker's gender from the text, or synthesize audio yourself when a script in this skill already does it. The one exception is the hook (Stage 3): that line is written by the agent, in the film's own voice. Voices are measured from the audio by scripts/detect_voices.py; the agent's job at Stage 3 is the Burmese text (and, optionally, the character names). Run the scripts for everything else and report what they printed.

Setup

python3 -m pip install --user edge-tts pyyaml      # PyYAML is used to read the config
sudo apt-get update && sudo apt-get install -y ffmpeg   # ffmpeg + ffprobe, required for audio

Put the AssemblyAI key in a .env file next to this skill — scripts/transcribe.py loads it automatically (still works with export too):

ASSEMBLYAI_API_KEY=your_key_here

.env is listed in .gitignore; never commit it or paste the key into a script.

Voice-over defaults (see voiceover.config.yaml): female = my-MM-NilarNeural, male = my-MM-ThihaNeural. The Multilingual family has no Burmese voice yet, so these two remain the working pair — swapping later means editing only the voices: block.

Measured on real Burmese output (2026-09): these voices speak at ~10-20 characters per second depending on the text, and the female voice runs slower than the male one, so voice_settings: in the config carries a small per-voice rate (+8% on female by default). Two measured facts shape the whole timing model, both handled by the scripts:

• AssemblyAI utterances[].start includes the silence before a line (measured: ~1.1 s early on the fixture), which would make the voice lead the picture. Cues are therefore built from word-level timings (--timing words, the default).
• edge-tts pads every clip with up to ~1 s of digital silence (measured: a 0.7 s line came back as a 1.87 s file). scripts/render_voiceover.py trims that silence before fitting, which cut "needs shortening" cues from 7/17 to 1/17 on the fixture.

No music is mixed into the deliverable. The pipeline ends with the voice-over stem (work/voiceover.mp3, starting at 00:00:00 and padded to the film's length) plus the subtitle files; the editor lays the score under the stem and owns the ducking. Do not add a music stage.

That is a workflow decision, not a limitation of Stage 1: diarization is unaffected by background music — the fixture was transcribed clean and with a music bed at -14/-8/-4 dB, and every run found both speakers with 100% assignment accuracy and 1.2% word error rate. A film's own score needs no special handling before transcription.

Workflow

Stage 1 — transcribe with speaker detection

python3 scripts/transcribe.py movie.mp3 --work-dir work --language auto

• Uploads the file, requests speaker_labels: true, polls to completion, writes work/transcript.json, work/transcript.raw.srt and work/transcribe.state.json.
• If it times out or a run is interrupted, --resume continues the same job without re-uploading or re-billing.
• Tell the user to pass --speakers-expected N when the cast size is known, and --language xx when the film is not English; auto-detect handles the rest.

Video input (mp4, mkv, mov, ...): pass the film itself. The script probes it, extracts a 16 kHz mono speech track to work/audio.16k.mp3 (a fraction of the film's size), uploads that, and remembers both paths in work/transcribe.state.json (audio for the pitch analysis, upload_source for what was sent). The film is never re-encoded. --prepare-audio-only does just the inspection and extraction so you can see the plan before spending an API call; --extract-audio always|never forces or forbids the extraction.

Stage 2 — build the speaker-tagged SRT

python3 scripts/srt_from_transcript.py --work-dir work

Produces work/movie.en.srt with [Speaker] prefixes, splitting long utterances by sentence and allocating time by character count, plus work/speakers.json and work/speakers.csv.

Check the speaker stats with the user before translating. Labels are A, B, C in order of first appearance and are not consistent between films. See references/speaker-mapping.md.

Stage 2b — decide the voice for each speaker (automatic)

python3 scripts/detect_voices.py --work-dir work

Isolates each speaker's own segments (using word timings), measures the fundamental frequency by autocorrelation, and writes work/voices.detected.yaml — the higher voice gets the female voice, the lower one the male voice, with a confidence rating and the measured pitch in the comments. Every later stage loads this file automatically, so no manual speaker mapping is needed. A speakers: entry in voiceover.config.yaml still wins whenever the user wants to pin one.

• Measured on this skill's fixture: speaker A 208 Hz → female, speaker B 160 Hz → male, both "high" confidence, matching the known ground truth.
• Without numpy it uses the same measurement in pure Python (--method f0-pure, ~1.5 s for a minute of audio). An earlier zero-crossing fallback was removed after it swapped the two fixture speakers.
• Report low-confidence results to the user and suggest --limit 10 listening before the full render.
• Sets on a film the detector has not seen: 3+ speakers, or two speakers of the same gender, get a rate offset so they stay distinguishable.

Stage 3 — translate (this is the agent's job)

python3 scripts/extract_for_translation.py --work-dir work --glossary glossary.txt

That writes work/translation.my.json: one entry per cue with source, speaker, read-only start/end/duration_ms, and empty text / tts_text fields. Then:

1. Read the worksheet and assets/translation-prompt.md (the rules that apply to you).
2. Fill text for every cue with natural, speakable Burmese that fits the cue's duration. Use tts_text only when the readable form is hard to speak aloud (numbers, names, symbols).
3. Keep index, start, end, duration_ms, speaker, source exactly as given — the merge step re-attaches the original timestamps and discards any edit, but do not rely on that.
4. Optional but useful: fill speaker_names with the character behind each label (the worksheet provides speaker_profiles with sample lines and the already-detected voice role). Names only label the subtitles — they never decide a voice.
5. Merge back:

python3 scripts/apply_translation.py --input work/translation.my.json --work-dir work

Long films: extract_for_translation.py --batch-size 120 splits the worksheet into work/batches/ and apply_translation.py --from-dir work/batches merges it back. Translate every cue in every batch; the merge refuses to finish on missing cues unless --allow-missing is given, and those slots end up silent.

Stage 3b — the opening hook (optional, also the agent's job)

The hook replaces the film's first 5-10 seconds with one line the agent writes, so the voice-over opens on a strong beat instead of a cold start. It is undetectable as long as it sounds like the film: same character voice, same register, and it carries the information the cut opening carried.

python3 scripts/extract_for_translation.py --work-dir work --hook --hook-window-ms 8000

The worksheet then carries a hook block (schema subtitle-voiceover/translation@2) holding the window (window_start_ms/window_end_ms), the dropped_cues and their source_lines, the next_dialogue_text the hook must lead into, and its own text field for the agent to fill. The worksheet's cues contain only the lines after the window.

Rules for the hook, in order of importance:

1. Stay inside the character. Write it as the character who speaks around it — never a narrator, never marketing, never a synopsis. If it could not be a line of this film's dialogue, rewrite it.
2. Do not spoil anything that has not happened yet.
3. Carry the cut information, do not contradict it. The dropped lines are shown to you; when the next surviving line refers back to them ("30 seconds is all we have"), the hook has to keep that beat alive — see references/hook-guide.md for a worked example.
4. Fit the window: 5-8 characters per second (the script warns above 13 cps), one sentence or two short ones, and no repetition of the line that follows it.
5. Dialogue may only resume after window_end_ms; the merge drops every cue that starts earlier.

hook.voice in the config (or the default, the voice of whoever speaks first after the window) decides which voice speaks the hook; the hook itself is written into the SRT without a speaker label, so it reads as the film's own opening line.

Details on length, register and subtitle-versus-speech choices: references/translation-guide.md.

Stage 4 — render the voice-over

python3 scripts/render_voiceover.py --work-dir work --jobs 4

For each cue the renderer picks the voice mapped to that cue's speaker, synthesizes with edge-tts, and fits the clip into the cue's slot without moving it: audio longer than the slot is sped up with atempo (pitch preserved) up to tts.max_tempo, then trimmed with a short fade if it still overruns. Outputs:

Render a sample first on a new film (--limit 20, or --cues 1,5,9) and let the user listen before committing to the whole feature. Re-run with --reuse-clips to keep finished clips.

Stage 5 — verify and hand off

python3 scripts/check_voiceover.py --work-dir work
python3 scripts/export_timeline.py --work-dir work --format all --fps 25

The track is the deliverable: it starts at 00:00:00, is padded with silence to the film's own length (from work/transcribe.state.json; --dialogue-length ends it at the last spoken line, --length-ms sets it explicitly), and carries no music. State its length when you report to the user — that is what they drop on the timeline. The checks catch missing audio, a track that is too short for the dialogue, unmapped speakers and lines that are too dense to read or speak. The export writes work/voiceover.fcpxml (Premiere Pro, Final Cut Pro, DaVinci Resolve), work/voiceover.edl and work/voiceover.cue-list.csv, each clip placed at its original timecode.

Stage 6 — clean the working files

python3 scripts/cleanup.py --work-dir work            # after the deliverables exist
python3 scripts/cleanup.py --work-dir work --dry-run  # see what would go first

Removes regenerable material — work/clips/, the extracted speech track, *.fit.*, *.speech.* and work/batches/ — and keeps every deliverable (the voice-over stem and its report, both subtitle tracks, the hand-off files, the transcript, the translations, the detection and the logs). It never touches the film itself. --keep-clips keeps the per-cue clips, --all also removes reports and the worksheet, --dry-run prints the plan. The report lands in work/cleanup.report.json.

One command for the whole run

python3 scripts/run_pipeline.py movie.mp4 --work-dir work --config voiceover.config.yaml \
    --hook --stages python        # transcribe -> SRT -> detect -> worksheet
#   ... the agent fills work/translation.my.json, then ...
python3 scripts/run_pipeline.py --work-dir work --config voiceover.config.yaml \
    --stages post                 # apply -> render -> checks -> hand-off -> cleanup

--stages post refuses to render a worksheet that still has empty cues, or a hook block with no text, so a half-finished translation cannot reach the renderer. --dry-run prints the plan, --skip stepname resumes past a step that already ran, and every step's output is kept in work/pipeline.log. The run ends with a summary of the deliverables and their measured numbers.

Rules that keep the dub in sync

• Timing is owned by the scripts. Cue start/end come from AssemblyAI and are never recomputed from translated text. If a line does not fit, the fix is a shorter translation, not a moved cue.
• Voice choice is measured, not inferred. scripts/detect_voices.py decides from the audio; the config overrides it; nothing ever guesses gender from a character's name or from the Burmese text.
• One pass, one language. output.target_language drives the filenames (movie.my.srt, movie.th.srt, ...), so several language versions can coexist in one work/ directory.
• No music stage. The score, the ducking and the loudness of the finished programme belong to the editor's timeline; the pipeline delivers the voice-over stem and stops there. If a user asks for a mixed file, explain what they get instead and hand them references/editor-handoff.md.
• Never re-run Stage 1 to fix a Stage 3/4 problem. Re-transcribing costs money and changes the speaker labels; fix the mapping, the translation, or the render instead.

Reporting to the user

After each stage, state the artefact path and the one number that matters: cues and speakers from Stage 2, cues translated and any dense lines from Stage 3, cue count plus sped-up/trimmed/failed counts from Stage 4, then the checks result and the hand-off files from Stage 5. Do not paste a whole SRT or the track listing unless asked.

Bundled resources

• scripts/vo_common.py — SRT parsing, config, ffmpeg and edge-tts helpers (imported by the others).
• scripts/transcribe.py — Stage 1, AssemblyAI with diarization and resume support.
• scripts/srt_from_transcript.py — Stage 2, speaker-tagged SRT and speaker statistics.
• scripts/detect_voices.py — Stage 2b, measures each speaker's pitch and assigns the voice.
• scripts/extract_for_translation.py — Stage 3a, the worksheet the agent fills in.
• scripts/apply_translation.py — Stage 3b, merges translations onto the original timeline.
• scripts/render_voiceover.py — Stage 4, per-speaker edge-tts and the film-length stem.
• scripts/check_voiceover.py — Stage 5a, quality checks.
• scripts/export_timeline.py — Stage 5b, FCPXML / EDL / CSV hand-off.
• scripts/check_subtitles.py — Stage 5c, editor-readiness checks on the delivered SRT.
• scripts/cleanup.py — Stage 7, removes working files and keeps the deliverables.
• scripts/run_pipeline.py — runs the whole pipeline in order, with a plan, a log and a summary.
• voiceover.config.yaml — voices, speaker map, timing tolerances; copy per project.
• references/speaker-mapping.md — how detection works, unusual casts, pinning by hand, diarization problems.
• references/translation-guide.md — length budgets, register, subtitles versus speech.
• references/edge-tts-voices.md — Burmese and Multilingual voice list, how to swap the pair.
• references/troubleshooting.md — setup, pipeline and quality failures with fixes.
• references/hook-guide.md — how to write the opening hook so nobody notices it was added.
• references/editor-handoff.md — importing the audio and subtitles into Premiere, Resolve or FCP.
• assets/translation-prompt.md — the translation contract given to the agent.
• assets/glossary.example.txt — glossary format for names and terminology.
• tests/run_tests.sh, tests/make_fixture.py — offline smoke tests (fixture transcript, real TTS).
• tests/make_dialogue_audio.py — builds a two-speaker English dialogue + music bed + ground truth.
• tests/score_vs_ground_truth.py — scores a transcript against that ground truth (speakers, WER, timing).
• demo-film/ — two complete real-API runs of the 53 s fixture: the audio run, and the video run with an opening hook, an editor-ready SRT and the cleanup, both with measurements.

Example: five minutes of film

python3 -m pip install --user edge-tts pyyaml && export ASSEMBLYAI_API_KEY=sk-...
python3 scripts/transcribe.py scene.mp3 --work-dir work --speakers-expected 2
python3 scripts/srt_from_transcript.py --work-dir work        # 84 cues, speakers A and B
python3 scripts/detect_voices.py --work-dir work              # A -> female, B -> male (measured)
python3 scripts/extract_for_translation.py --work-dir work    # worksheet for translation
# agent translates every cue, then:
python3 scripts/apply_translation.py --input work/translation.my.json --work-dir work
python3 scripts/render_voiceover.py --work-dir work --limit 20   # sample, let the user listen
python3 scripts/render_voiceover.py --work-dir work              # full track
python3 scripts/check_voiceover.py --work-dir work
python3 scripts/export_timeline.py --work-dir work --format all
python3 scripts/cleanup.py --work-dir work          # after the stem and subtitles exist

Example: a video, with a hook, ending in a clean work directory

python3 scripts/run_pipeline.py night-shift.mp4 --work-dir work --config voiceover.config.yaml \
    --language en --speakers-expected 2 --hook --hook-window-ms 8000 --stages python
# agent: read work/translation.my.json, fill every cue's text AND hook.text, keep the timestamps
python3 scripts/run_pipeline.py --work-dir work --config voiceover.config.yaml --stages post

Result, measured on this skill's own 53-second demo film: hook 0.72-8.72 s spoken by the male lead (62 characters, 7.8 cps, the four cut opening cues replaced), dialogue resuming at 11.48 s with the line that answers the hook, 14 cues rendered with no failures, a music-free stem of 53.26 s (the film's length, not the 51.48 s of dialogue), subtitles READY, and 30 working files removed afterwards. The full record is in demo-film/work-video/ and demo-film/README.md.