renky1025 Video Dubbing & Subtitle Sync
Complete workflow: ASR → translate → TTS → merge audio → burn subtitles
Pipeline
视频 → 抽音轨(16kHz) → ASR(whisper) → AI翻译 → TTS(dub_segments) → 合并音轨 → 烧录硬字幕
Workflow
0. Prerequisite Check
Ensure whisper + moviepy + mlx-audio are installed:
pip install openai-whisper moviepy 2>/dev/null
# mlx_audio for TTS:
pip install mlx-audio 2>/dev/null # Apple Silicon
1. Extract Audio & Transcribe (ASR)
Extract 16kHz mono audio, then run whisper for accurate per-segment timestamps:
ffmpeg -y -i input.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio16k.wav
whisper audio16k.wav \
--model large-v3-turbo \
--language <source_lang> \
--output_format srt \
--output_dir .
Output: audio16k.srt with native timestamps matching the video.
Use
large-v3-turbofor best accuracy. If you want even better Chinese recognition, install Qwen3-ASR (pip install qwen-asr) and usescripts/qwen3_asr.pyinstead.
2. Translate Subtitles (AI does this)
Read the SRT, translate each segment's text into the target language. Output one line per SRT segment, preserving order exactly.
- Count SRT segments first (e.g. 27 segments → 27 translated lines)
- Save as
translated.txtin the working directory - Verify line count matches SRT segment count before proceeding
3. Generate Dubbed Audio
python3 scripts/dub_segments.py audio16k.srt translated.txt dubbing.wav subtitle_synced.srt --lang <target_lang>
What it does:
- Generates TTS per segment with voice consistency (first segment = voice reference)
- Preserves original timestamps — subtitles stay synced with video
- Smart speed adjustment: only adjusts segments that overflow their slot (atempo 0.88–1.20)
- Bridges adjacent gaps <1s for natural flow
4. Merge Dubbed Audio into Video
ffmpeg -y \
-i input.mp4 \
-i dubbing.wav \
-map 0:v:0 -map 1:a:0 \
-c:v copy -c:a aac -b:a 192k \
-shortest \
output_temp.mp4
Verify audio replaced:
ffprobe -v error -show_entries stream=codec_type -of csv=p=0 output_temp.mp4
# Should show: video, audio
5. Burn Hard Subtitles into Video
python3 scripts/burn_subtitles.py output_temp.mp4 subtitle_synced.srt output_final.mp4
This renders translated subtitles permanently into the video frame (hardcoded).
6. Cleanup (Optional)
rm -f audio16k.wav audio16k.srt translated.txt dubbing.wav subtitle_synced.srt output_temp.mp4
Output Files
| File | Description |
|---|---|
output_final.mp4 |
Final video with dubbed audio + hardcoded subtitles |
subtitle_synced.srt |
Synced subtitles (original timestamps + translated text) |
dubbing.wav |
Per-segment aligned dubbing track |
Key Design Decisions
| Principle | Why |
|---|---|
| Preserve original ASR timestamps | They match the video's visual cues natively. Re-ASR on dubbing drifts. |
| Per-segment speed adjustment (0.88–1.20x) | Global atempo on the whole dubbing sounds robotic. Per-segment is natural. |
| Hard subtitles (burned in) | Soft subtitles don't work on all platforms. Burn them so they always show. |
| 16kHz mono for ASR | Whisper expects this. Higher sample rate wastes compute without improving accuracy. |
| Voice consistency via first segment reference | mlx-tts supports --ref_audio. First segment sets the voice for all others. |
| No Demucs by default | For typical narration/speech videos, background music removal is unnecessary overhead. |
| large-v3-turbo model | Best accuracy/speed tradeoff. Significantly better than base for proper nouns and fast speech. |
Dependencies
| Component | Install | Purpose |
|---|---|---|
ffmpeg |
brew install ffmpeg |
Audio extraction, merging |
openai-whisper |
pip install openai-whisper |
Speech-to-text (ASR) |
moviepy |
pip install moviepy |
Burn subtitles into video |
mlx-audio |
pip install mlx-audio |
TTS on Apple Silicon |
qwen-asr |
pip install qwen-asr (optional) |
Better Chinese ASR |
Common Mistakes to Avoid
| Mistake | Why It Fails | Correct Approach |
|---|---|---|
ffmpeg without -map |
Picks original audio (video source's audio stream), dubbing ignored | Always use -map 0:v:0 -map 1:a:0 |
| Global atempo on entire dubbing | All speech slows uniformly (EN→ZH ~0.8x), sounds robotic | Per-segment alignment (dub_segments.py) |
| Re-ASR on dubbed audio for timestamps | Timestamps drift from visual cues | Reuse original Step 1 timestamps |
| Translation line count ≠ SRT segments | dub_segments.py requires strict 1:1 mapping |
Count SRT segments first, match exactly |
Use whisper base/tiny model |
Proper nouns wrong, fast speech missed | Use large-v3-turbo or turbo |
| Burn subtitles without checking FFmpeg libass | subtitles= filter silently fails |
Use scripts/burn_subtitles.py (moviepy) |
| Translate segments out of order | Subtitles play at wrong times | Keep segment order: 1 translated line per SRT segment |
| Skip verification | Audio may not have been replaced | Run ffprobe to check both audio + video streams exist |