Carocut Media Audio

Audio Assets

Generates voiceovers via TTS, retrieves background music and sound effects. Creates durations.json for timing synchronization in Remotion.

Audio Types

Voiceover Generation

Generate speech audio from script using Edge TTS.

TTS Batch Processing Workflow

Processes entire script.md and generates all voiceover files:

1. Parse {project_dir}/manifests/script.md for all [VO_XXX] markers and their text content
2. For each VO segment, invoke Edge TTS with the configured voice and speed
3. Save each result as VO_XXX.wav in {project_dir}/raws/audio/vo/
4. After all segments complete, extract durations to create {project_dir}/manifests/durations.json

Voice Selection

完整 voice 列表和选择建议见 references/edge-tts.md。

Speed Adjustment Rules

• The default TTS voice may sound slow. Consider using speed 1.2 for more natural pacing.
• Speed range: 0.5 (half speed) to 2.0 (double speed)
• Recommended: 1.0 - 1.3 for narration, 0.8 - 1.0 for emotional/dramatic content

TTS Speed and Storyboard Pacing (语速与节奏对应)

When generating TTS, read the corresponding shot's pacing field from the storyboard to determine speed:

Rules:

• If the storyboard does not specify pacing for a shot, default to medium (speed 1.2)
• For pause pacing shots, skip TTS generation entirely — these are breathing shots with no voiceover
• The pacing value is per-shot; different VO segments in the same video may have different speeds
• After applying pacing-based speed, duration extraction must reflect the actual playback length at that speed

Duration Extraction

Extract audio durations for timing synchronization. This step is mandatory.

Output Format (durations.json)

{
  "VO_001": 3450,
  "VO_002": 2800,
  "VO_003": 4200,
  "VO_004": 3100
}

All durations are in milliseconds. This file is critical for shot timing in Remotion.

Duration Rules

• All values MUST be in milliseconds (integer)
• Every [VO_XXX] marker in script.md must have a corresponding entry
• Duration reflects the actual audio length at the configured speed (e.g., 1.2x)
• Always re-extract durations after any audio modification

durations.json and Audio-Visual Relation

The durations.json format itself remains unchanged (simple VO_XXX: milliseconds mapping). However, the Builder's timing.ts module uses the storyboard's audio_visual_relation field in combination with durations to calculate actual audio start frames:

The audioOffsetFrames function in timing.ts reads these values to apply frame-level offsets. The Audio agent does not need to modify durations.json for this — it is handled downstream by the Builder.

重要依赖说明：虽然 durations.json 格式简单（VO_XXX: 毫秒数），但 Builder 在计算实际音频起始帧时，会同时读取 storyboard.yaml 中对应 shot 的 audio_visual_relation 字段。Audio agent 只需确保 durations.json 中的时长准确；偏移量由 Builder 的 timing.ts → audioOffsetFrames() 函数自动计算。

BGM and SFX Retrieval

Download royalty-free audio from Freesound.

BGM Search Strategy

• Search for loops or long-form tracks matching the video mood
• Minimum duration: 60 seconds (to cover full video length)
• Keywords should describe mood, genre, and instrumentation

BGM Rhythm Matching (BGM 节奏匹配)

Select BGM tempo based on the dominant pacing of each chapter/section in the storyboard:

Rules:

• Determine a chapter's dominant pacing by the most frequent pacing value among its shots
• If a video has distinct chapters with different pacing, consider using multiple BGM tracks with crossfade transitions
• For breathing shots between chapters, use ambient/atmospheric sounds or fade the BGM to bridge the transition
• When searching Freesound, include BPM range in keywords (e.g., "ambient piano 70bpm", "upbeat electronic 130bpm")
• BGM should complement, not compete with, the voiceover. Lower BPM tracks naturally sit better under narration.

SFX Search Strategy

• Search for short, punchy sound effects for transitions and UI feedback
• Maximum duration: typically 3-5 seconds
• Keywords should describe the exact sound needed

Search Keywords Tips

License Information

For CC-BY sounds, record attribution in raws/audio/LICENSES.txt.

Output Structure

raws/
  audio/
    vo/
      VO_001.wav
      VO_002.wav
      ...
      durations.json       # Critical timing data
    bgm/
      ambient_loop.mp3
    sfx/
      whoosh.mp3
      notification.mp3
    LICENSES.txt           # Attribution for CC-BY sounds

Workflow

1. Generate Voiceovers

   • Process manifests/script.md for all [VO_XXX] markers
   • All [VO_XXX] markers become VO_XXX.wav files
   • Extract durations to create durations.json

2. Retrieve BGM

   • Search for appropriate background music
   • Download to raws/audio/bgm/

3. Retrieve SFX

   • Search for transition sounds, UI sounds
   • Download to raws/audio/sfx/

4. Verify and Report

   • Check all files generated correctly
   • Report results to user

Incremental Mode

当 dispatch context 中 mode: "incremental" 时：

输入

• amendment.affected_resources 中指定的 VO 段落 ID 列表（需要重新生成的语音片段）

执行规则

1. 读取 script.md，仅定位指定的 VO 段落
2. 仅对指定的 VO 段落重新生成 TTS 音频
3. 使用与原始生成相同的 character、tone、speed 配置
4. 替换 raws/audio/vo/ 中对应的 WAV 文件
5. 必须重新提取完整的 durations.json（因为任何音频变化都会影响时序）
6. 返回处理摘要（重新生成了哪些 VO、新旧时长对比）

不执行

• 不重新生成未指定的 VO 音频
• 不重新检索 BGM/SFX（除非 amendment 中明确要求）

关键注意

• 即使只修改了一个 VO 片段，也必须重新提取整个 durations.json
• 时长变化可能影响下游的 Remotion 时序计算

Audio Timing Rules

These rules are critical for correct Remotion synchronization:

Network Requirements

Edge TTS requires outbound internet access and the Python package edge-tts.

User Communication

Progress Report

音频素材生成进行中...

TTS 生成:
  进度: 15/45 条语音
  当前: VO_016
  音色: zh-CN-XiaoxiaoNeural

Completion Report

音频素材生成完成。

语音配音:
  - 45 条语音文件
  - 总时长: 4分23秒
  - 音色: zh-CN-XiaoxiaoNeural
  - 时长数据: raws/audio/vo/durations.json

背景音乐:
  - 1 个循环音频 (2分30秒)

音效:
  - 3 个过渡音效
  - 2 个提示音

输出目录: raws/audio/
请确认后继续下一步。

Error Handling

Critical Notes

Duration File

durations.json is mandatory for Remotion implementation. It contains timing data in milliseconds that drives:

• Shot duration calculations
• Audio layer positioning
• Voiceover synchronization

Always re-extract durations after any audio modification.

Default Voice Speed

The default TTS voice may sound slow. Consider using speed 1.2 for more natural pacing. When the storyboard provides a pacing field, use the TTS Speed and Storyboard Pacing table above to determine the appropriate speed for each VO segment.