Storyboard Tts

Storyboard TTS

Take a storyboard deliverable and batch-synthesize Chinese + English voice-over with IndexTTS2 (shared setup with ai-text-to-speech).

Shot id from headers (### Shot 01 — … → 01.wav).

Subtitles: one SRT per language. Shots are laid end-to-end on the VO timeline (shot N starts when N−1 ends). Inside a shot, text is split on sentence punctuation (.!?;… and CJK equivalents) into multiple cues; cue lengths share that shot’s WAV duration by non-whitespace character weight. Skip (no VO) / missing audio.

Rules

When this skill applies, read and follow skill-dependency-manager — run scripts as documented, install missing tools into .dependency/.

1. Batch synthesis only via .ai/storyboard-tts/synthesize.py with the index-tts interpreter. Do not hand-write IndexTTS loops, temporary batch drivers, or N× single tts.py calls for a full storyboard.
2. Trial / single-line checks may use ai-text-to-speech tts.py, or synthesize.py --limit 1.
3. Parse-only / report-only / subtitle-only steps use stdlib python (.dependency/python/python).
4. Never overwrite the storyboard source. Write only under <audio-dir>/.
5. Skip (no VO) / empty lines — no empty WAVs or empty subtitle cues.
6. Confirm voice reference (and output dir if unclear) before a full batch.
7. The top-level output directory uses the reference voice filename stem. Subdirectory names stay fixed. Default <audio-dir> is <storyboard-dir>/<voice-stem>/. Do not use <storyboard-stem>-speech.

Inputs

Layout

<storyboard-dir>/<voice-stem>/   # e.g. narrator-self-fast/ from narrator-self-fast.wav
  Chinese/
    01.wav
    …
  English/
    01.wav
    …
  shots.json
  speech-timeline.md
  Chinese.srt            # all Chinese cues, continuous timeline
  English.srt            # all English cues, continuous timeline
  _text/                 # only with --write-text

Quick Start

From project root:

.dependency/index-tts/.venv/Scripts/python.exe .ai/storyboard-tts/synthesize.py --storyboard path/to/storyboard.md --voice path/to/ref.wav --fp16 --report

Trial run (first line only):

.dependency/index-tts/.venv/Scripts/python.exe .ai/storyboard-tts/synthesize.py --storyboard path/to/storyboard.md --voice path/to/ref.wav --fp16 --limit 1

Writes under <storyboard-dir>/<voice-stem>/ (override with --audio-dir only when needed).

This will:

1. Parse the storyboard → <audio-dir>/shots.json
2. Load IndexTTS2 once
3. Write Chinese/<id>.wav and English/<id>.wav (skip existing unless --force)
4. Pad each WAV in place to 0.4 s leading/trailing silence (--no-pad to skip)
5. Write speech-timeline.md and Chinese.srt / English.srt when --report

Separate voices / language

.dependency/index-tts/.venv/Scripts/python.exe .ai/storyboard-tts/synthesize.py --storyboard path/to/storyboard.md --voice-zh path/to/zh_ref.wav --voice-en path/to/en_ref.wav --fp16 --report

.dependency/index-tts/.venv/Scripts/python.exe .ai/storyboard-tts/synthesize.py --storyboard path/to/storyboard.md --voice path/to/ref.wav --lang chinese --fp16 --report

Parse, report, or subtitles alone (stdlib python)

.dependency/python/python .ai/storyboard-tts/parse_storyboard.py path/to/storyboard.md -o path/to/<audio-dir>/shots.json

.dependency/python/python .ai/storyboard-tts/duration_report.py --storyboard path/to/storyboard.md --audio-dir path/to/<audio-dir> --shots path/to/<audio-dir>/shots.json -o path/to/<audio-dir>/speech-timeline.md

.dependency/python/python .ai/storyboard-tts/write_subtitles.py --audio-dir path/to/<audio-dir> --shots path/to/<audio-dir>/shots.json

Resume from an existing shots.json:

.dependency/index-tts/.venv/Scripts/python.exe .ai/storyboard-tts/synthesize.py --shots path/to/<audio-dir>/shots.json --voice path/to/ref.wav --fp16 --report

Common Flags (synthesize.py)

On partial failure: script continues remaining jobs, prints Failed jobs: …, exit code 1. Fix install/voice per ai-text-to-speech troubleshooting, re-run (existing OK files are skipped).

Agent Notes

1. Do not invent narration — use storyboard Chinese/English fields as-is (audio and subtitles).
2. Prefer one synthesize.py invocation for a full board; model reload cost is the reason.
3. Chat summary: audio-dir (<storyboard-dir>/<voice-stem>/), counts, path to speech-timeline.md, Chinese.srt / English.srt, Chinese/English total seconds — no full transcripts unless asked. Omit --audio-dir unless overriding.
4. IndexTTS install lives in ai-text-to-speech; do not duplicate Setup here beyond “populate index-tts if missing”.
5. Edge padding is built in (default 0.4 s, in place via pad.py). Use --no-pad only when they want raw TTS with no extra silence; --pad-duration if they want a different length.
6. Loudnorm / OGG / trim remain separate skills after this one.
7. If audio already exists and only subtitles are needed, run write_subtitles.py alone (stdlib python). Re-running synthesize.py --report without --force still pads existing WAVs, then rewrites the timeline/SRT.

Tests

Stdlib scripts (from repo root):

.dependency/python/python .ai/storyboard-tts/test_parse_storyboard.py
.dependency/python/python .ai/storyboard-tts/test_write_subtitles.py

IndexTTS batch driver (requires populated index-tts; see cli/storyboard-tts.md):

.dependency/index-tts/.venv/Scripts/python.exe .ai/storyboard-tts/synthesize.py --storyboard path/to/storyboard.md --voice .ai/test/audio/han.wav --fp16 --limit 1 --report

Related

• storyboard — source markdown
• ai-text-to-speech — single-line TTS + IndexTTS setup
• storyboard-av-mix — mux VO with per-shot video
• Optional after: audio-loudness-normalization