Captions and Transcription
Machine transcription is a first draft. Captions ship only after a human-quality pass for accuracy, timing, and readability.
When to Use
- Any video with speech that will be published (captions are a WCAG requirement for prerecorded synchronized media)
- Sound-off social video where captions carry the message
- Creating word-timed "kinetic" captions for Remotion or HyperFrames compositions
- Auditing existing captions
Workflow
Step 1: Extract Clean Audio
# 16 kHz mono 16-bit WAV: the input format the whisper.cpp README asks for; fine for the others too
ffmpeg -i video.mp4 -ar 16000 -ac 1 -c:a pcm_s16le speech.wav
If the mix has loud music, transcribe the VO stem rather than the final mix.
Step 2: Choose a Transcription Tool
All four run locally; audio does not leave the machine. Pick by environment, not by hype.
| Tool | Install | Strength | Licence |
|---|---|---|---|
| openai-whisper | pip install -U openai-whisper (needs FFmpeg) |
Reference implementation; CLI writes SRT/VTT directly | MIT |
| whisper.cpp | Build with CMake; download a ggml model | C/C++, no Python; CLI writes SRT/VTT | MIT |
| faster-whisper | pip install faster-whisper |
CTranslate2 reimplementation; Python API with word timestamps and VAD filter | MIT |
| WhisperX | pip install whisperx |
Word-level timestamps via forced alignment; optional speaker diarization | BSD-2-Clause |
Hosted speech-to-text APIs are also an option when local compute is unavailable; check the provider's data-handling terms before uploading user audio.
openai-whisper CLI:
whisper speech.wav --model turbo --language English --output_format srt --output_dir captions/
# Word-level timing and line control (these flags require --word_timestamps True)
whisper speech.wav --model turbo --word_timestamps True --max_line_width 42 --max_line_count 2 --output_format srt
Models: tiny, base, small, medium, large, turbo (English-only .en variants exist except for large and turbo). --task translate translates to English.
whisper.cpp:
cmake -B build && cmake --build build -j --config Release
sh ./models/download-ggml-model.sh base.en
./build/bin/whisper-cli -m models/ggml-base.en.bin -f speech.wav -osrt -ovtt -of captions/speech
-ml N sets a maximum segment length in characters; -sow splits on words.
faster-whisper (Python):
from faster_whisper import WhisperModel
model = WhisperModel("small", device="cpu", compute_type="int8")
segments, info = model.transcribe("speech.wav", word_timestamps=True, vad_filter=True)
def ts(t: float) -> str:
h, rem = divmod(t, 3600); m, s = divmod(rem, 60)
return f"{int(h):02}:{int(m):02}:{int(s):02},{int((s % 1) * 1000):03}"
with open("speech.srt", "w") as f:
for i, seg in enumerate(segments, 1): # segments is a generator; iterate once
f.write(f"{i}\n{ts(seg.start)} --> {ts(seg.end)}\n{seg.text.strip()}\n\n")
WhisperX (word alignment; diarization needs a Hugging Face token and accepting the pyannote model terms):
whisperx speech.wav --highlight_words True
whisperx speech.wav --diarize --hf_token "$HF_TOKEN"
Step 3: Correct the Transcript
Machine output will contain errors. Fix, in order:
- Proper nouns and product names. Keep a glossary from the brief and search-replace.
- Numbers, units, and code identifiers. Speech models normalize these inconsistently.
- Hallucinated text in silence or music. Delete captions over non-speech sections that contain invented words.
- Non-speech information. Add speaker labels when the speaker is not obvious, and meaningful sounds:
[door slams],[upbeat music]. WCAG defines captions as covering both speech and non-speech audio needed to understand the content.
Step 4: Fix Timing and Line Breaks
Reference limits from the Netflix Timed Text Style Guide (English, USA), a widely used published spec:
| Rule | Limit |
|---|---|
| Characters per line | 42 |
| Lines per caption | 2 maximum; prefer 1 |
| Reading speed | Up to 20 characters/second (adult), 17 (children's) |
| Minimum duration | 5/6 second per event |
| Maximum duration | 7 seconds per event |
Line breaks: break after punctuation and before conjunctions or prepositions; do not split an article from its noun, an adjective from its noun, or a name across lines.
For short vertical social clips, shorter lines (a few words, one line) read better on phones; keep the same reading-speed ceiling.
Step 5: Pick the Format
| Format | Use for | Notes |
|---|---|---|
| SRT | Uploading to platforms, simple sidecars | HH:MM:SS,mmm timestamps (comma) |
| WebVTT | HTML <track> on the web |
File starts with WEBVTT; . before milliseconds |
| ASS | Styled burn-in (fonts, colours, positions, karaoke) | Rendered by libass via FFmpeg subtitles/ass filters |
Convert with FFmpeg: ffmpeg -i captions.srt captions.vtt or ffmpeg -i captions.srt captions.ass.
Step 6: Style Burned-In Captions
- High contrast: white or yellow text with an outline or a semi-opaque box behind it.
- Keep captions inside the safe area and clear of platform UI in vertical feeds (see
motion-design-principles). - Do not cover faces or the UI element being demonstrated; move captions to the top for those shots.
- Word-by-word highlight is fine for social clips; never let the effect slow reading or flash rapidly.
- FFmpeg's
force_styleFontSizeis relative to the ASS script'sPlayResY, not output pixels. An SRT converted by FFmpeg gets a small default script resolution (288 lines in our test with FFmpeg 8.1), so check a still before rendering the whole video.
Burn-in commands live in ffmpeg-post-production, Step 7.
In Remotion or HyperFrames, render captions as components from the word-timed JSON instead of burning with FFmpeg. Remotion documents @remotion/captions (import SRT/VTT, transcription options, display, export); HyperFrames ships caption guidance in its /hyperframes skill. Follow those official skills for the code.
Step 7: Deliver
- Upload the sidecar (SRT/VTT) to platforms that accept it, even if captions are also burned in. Sidecars are searchable, resizable, and translatable.
- On the web, attach captions with
<track kind="captions" srclang="en" label="English" src="captions.vtt" default>. - Keep the corrected transcript alongside the video; it doubles as alt content and a script for translations.
Caption QA Checklist
- Every spoken line captioned; nothing captioned during silence
- Names, brand terms, and numbers match the glossary
- Speaker identification and meaningful sounds included
- No caption over 2 lines or 42 characters per line; reading speed within limits
- Captions appear and disappear in sync with speech
- Burned-in captions readable on a phone and clear of platform UI
- Sidecar file attached where the platform supports it
Best Practices
- Transcribe the cleanest audio available (VO stem, 16 kHz mono)
- Treat model output as a draft; always do a correction pass
- Use word-level timestamps for kinetic captions, segment-level for standard captions
- Ship a sidecar caption file even when burning in
- Keep a project glossary of names and terms for search-replace
- Never present an uncorrected machine transcript as final captions
References
- WCAG 2.2 SC 1.2.2 Captions (Prerecorded): https://www.w3.org/WAI/WCAG22/Understanding/captions-prerecorded.html
- openai/whisper (install, CLI, models, licence): https://github.com/openai/whisper and CLI flags in https://github.com/openai/whisper/blob/main/whisper/transcribe.py
- whisper.cpp (build, models,
whisper-cliflags): https://github.com/ggml-org/whisper.cpp - faster-whisper: https://github.com/SYSTRAN/faster-whisper
- WhisperX: https://github.com/m-bain/whisperX
- Netflix Timed Text Style Guide, General Requirements: https://partnerhelp.netflixstudios.com/hc/en-us/articles/215758617-Timed-Text-Style-Guide-General-Requirements
- Netflix English (USA) Timed Text Style Guide: https://partnerhelp.netflixstudios.com/hc/en-us/articles/217350977-English-USA-Timed-Text-Style-Guide
- MDN, WebVTT format: https://developer.mozilla.org/en-US/docs/Web/API/WebVTT_API/Web_Video_Text_Tracks_Format
- FFmpeg
subtitlesandassfilters: https://ffmpeg.org/ffmpeg-filters.html - Remotion captions: https://www.remotion.dev/docs/captions
- HyperFrames skills: https://github.com/heygen-com/hyperframes