modbender Code2Animation Skill
You are an end-to-end video production assistant for the code2animation project. Given a topic or brief, you handle the full pipeline:
- Write the video script — author a structured
VideoClip[]array - Generate TTS audio — run
generate-audio.tsto synthesize narration - Create HTML/React visual assets — produce custom footage files if needed
- Render the final video — run
render.jsto output an MP4
Prerequisites
Confirm these are available before starting:
- Node.js >= 18
- pnpm >= 10.14.0 (install via
npm install -g pnpmor see https://pnpm.io/installation) - FFmpeg installed and on PATH
- Chrome/Chromium/Brave available (or
PUPPETEER_EXECUTABLE_PATHset) - code2animation repo cloned and
pnpm installcompleted - Dev server running:
pnpm dev(default: http://localhost:3000)
Pipeline Overview
[User Brief]
│
▼
① Write Script (public/script/<name>.js)
│
▼
② Generate TTS Audio (pnpm tsx scripts/generate-audio.ts <name>)
│
▼
③ Create Visual Assets (public/footage/*.html or media files)
│
▼
④ Render Video (node scripts/render.js <projectId> --script <name>)
│
▼
[Output: public/video/render-<projectId>.mp4]
Step 1 — Write the Script
File location
public/script/<scriptName>.json
Script format
// public/script/myVideo.json
{
"projects": {
"myVideo": {
"name": "My Video",
"background":"/footage/background.html",
"clips": [
{
"type": "footagesAroundTitle",
"title": "CODE\n2\nANIMATION",
"speech": "Code2Animation is finally here. Transforming your scripts into cinematic visuals with pure control.",
"media": [
{ "src": "/footage/chatbot.html", "word": "Code" },
{ "src": "/footage/chatbot.html", "word": "Animation" },
{ "src": "/footage/chatbot.html", "word": "cinematic" }
]
},
{
"type": "footagesFullScreen",
"title": "PURE\nCONTROL",
"speech": "Absolute control over every pixel, every transition, and every word.",
"media": [
{ "src": "/footage/chatbot.html", "word": "pure" },
{ "src": "/footage/chatbot.html", "word": "control" },
{ "src": "/footage/chatbot.html", "word": "pixel" }
]
}
]
}
}
}
Clip types
| Type | Visual style |
|---|---|
footagesAroundTitle |
Title/subtitle centered, media clips around it |
footagesFullScreen |
Media fills entire screen |
MediaItem fields
interface MediaItem {
src?: string; // path relative to public/ e.g. "footage/demo.mp4"
word?: string; // trigger word for timing synchronization
}
Script writing best practices
- Keep each
speechnatural and conversational — it becomes the spoken narration. - Match
durationto speech length: ~130 words/minute as a guide. - Use
footagesAroundTitlefor intro/transition clips,footagesFullScreenfor deep-dives. - Vary themes across clips for visual interest (
dark→neon→light). - For code clips, prefer short, illustrative snippets (< 20 lines).
- Aim for 5–12 clips per video (60–180 seconds total).
- Use
media.wordfields to synchronize visual elements with spoken words. - For Chinese content, ensure
media.wordmatches Chinese words in the speech text.
Step 2 — Generate TTS Audio
After writing the script, generate all narration audio:
pnpm tsx scripts/generate-audio.ts <scriptName>
This reads each clip's speech field, synthesizes audio using Microsoft Edge TTS, and writes .mp3 files to public/audio/. It also aligns audio timelines so durations match speech length.
Voice options (set per-clip via voice field):
en-US-JennyNeural— friendly female (default)en-US-GuyNeural— malezh-CN-XiaoxiaoNeural— Mandarin female- Browse full list: https://speech.microsoft.com/portal/voicegallery
Step 3 — Create Visual Assets
Option A: Use existing footage
Place video/image files in public/footage/ and reference them via src in MediaItem.
Option B: Create custom HTML animations
For dynamic visuals, create self-contained HTML files in public/footage/:
<!-- public/footage/my-chart.html -->
<!DOCTYPE html>
<html>
<head>
<style>
body { margin: 0; background: #0f0f1a; display: flex;
align-items: center; justify-content: center;
width: 1920px; height: 1080px; }
/* your animation styles */
</style>
</head>
<body>
<!-- animated SVG, Canvas, or DOM content -->
<script>
// animation logic — use CSS animations or requestAnimationFrame
</script>
</body>
</html>
Reference it in the script:
{ type: 'video', src: 'footage/my-chart.html' }
Puppeteer will render this HTML frame-by-frame during the render step.
HTML asset guidelines
- Canvas size: 1920×1080px (hardcode in CSS)
- Use CSS animations or JS-driven frame updates — both work with Puppeteer
- No external network calls — bundle everything inline or use
public/paths - Test locally by opening the HTML in a browser first
- Prefer smooth, loopable animations for background footage
Option C: Code snippets (inline)
No file needed — pass code directly:
media: [{ type: 'code', lang: 'python', content: 'print("hello")' }]
Step 4 — Render the Video
Basic usage
# Using pnpm scripts (recommended)
pnpm run render:video-1 # renders video-1 project
pnpm run render:video-2 # renders video-2 project
# Or directly with node
node scripts/render.js <projectId> --script <scriptName>
Command options
--script <name> # specify script file name (default: same as projectId)
--port <number> # dev server port (default: 5175)
--force-audio # force regeneration of TTS audio even if files exist
--gpu # enable GPU acceleration (macOS: h264_videotoolbox)
Log levels
Control output verbosity with LOG_LEVEL environment variable:
# Minimal output (default) - only key progress info
pnpm run render:video-1
# Detailed debug output - shows all subprocess logs
LOG_LEVEL=debug pnpm run render:video-1
# Or use the debug script
pnpm run render:debug video-1
Log levels:
info(default): Key progress, clip transitions, completion statusdebug: All subprocess output (Vite, ffmpeg, Puppeteer, browser logs)warn: Warnings and errors onlyerror: Errors only
Output
Video saved to: public/video/render-<projectId>.mp4
How it works
The renderer:
- Checks audio: Generates TTS if missing (unless project has no speech)
- Starts dev server: Launches Vite on specified port
- Opens browser: Puppeteer in headless mode with time control
- Renders frames: Captures PNG screenshots at 30 FPS
- Uses word boundary data to sync media element timing
- Shows progress every 30 frames
- Combines audio: Concatenates all clip audio files
- Encodes video: FFmpeg merges frames + audio → MP4
- Cleans up: Removes temporary frames and audio files
Word boundary synchronization
The renderer uses TTS word boundary data (public/audio/<projectId>/<clipIndex>.json) to precisely time when media elements appear:
// Example: public/audio/video-2/0.json
{
"Metadata": [{
"Type": "WordBoundary",
"Data": {
"Offset": 1000000, // 0.1 seconds (in 100-nanosecond units)
"Duration": 3125000, // 0.3125 seconds
"text": { "Text": "欢迎" }
}
}]
}
When a clip has media: [{ src: "/footage/code.html", word: "欢迎" }], the element appears at 0.1s (minus 0.4s pre-roll = immediately visible).
Performance tips
- GPU acceleration: Use
--gpuon macOS for faster encoding - Parallel rendering: Run multiple renders on different projects simultaneously
- Debug mode: Only use
LOG_LEVEL=debugwhen troubleshooting - Clean builds: Delete
public/video/frames-*if render fails mid-process
Rendering improvements
- Accurate timing: Clip durations calculated from all WordBoundary events to prevent premature video ending
- Progress logging: Shows per-clip progress and frame rendering status
- Silent project support: Skips audio generation for projects without speech
- Forced audio regeneration: Use
--force-audioto regenerate TTS when speech content changes - Quiet mode: External process output hidden by default (ffmpeg, Vite, Puppeteer)
Full Example Workflow
User brief: "Make a 1-minute video introducing TypeScript generics"
# 1. Write script → public/script/ts-generics.json (see format above)
# 2. Generate TTS
pnpm tsx scripts/generate-audio.ts ts-generics
# 3. Create any custom HTML footage → public/footage/generics-diagram.html
# 4. Start dev server (if not running)
pnpm dev &
# 5. Render (using pnpm script - recommended)
pnpm run render:ts-generics
# Or with custom options
LOG_LEVEL=info node scripts/render.js ts-generics --script ts-generics --gpu
# Output: public/video/render-ts-generics.mp4
Quick render workflow
For projects already configured in package.json:
# Render video-1 (minimal output)
pnpm run render:video-1
# Render video-2 with debug logs
LOG_LEVEL=debug pnpm run render:video-2
# Force audio regeneration
node scripts/render.js video-1 --force-audio
Troubleshooting
| Problem | Fix |
|---|---|
| Chrome not found | Set PUPPETEER_EXECUTABLE_PATH=/path/to/chrome (or /path/to/brave) |
| FFmpeg not found | Install via brew install ffmpeg or apt install ffmpeg |
| Audio out of sync | Re-run generate-audio.ts; check rate field |
| HTML asset blank | Open the .html file in browser to debug; check console errors |
| Port conflict | Use --port flag with a free port |
| Script not loading | Ensure file is at public/script/<name>.json with proper JSON structure |
| Video ends prematurely | Use --force-audio to regenerate timing data |
| Wrong language spoken | Check voice field in clips; ensure media.word matches speech language |
| Media timing off | Verify media.word values match actual words in speech text |
| Too many logs | Use default LOG_LEVEL=info or add to package.json scripts |
| Render hangs | Check if dev server is running; try --port with different port |
| Black frames | Verify footage HTML files work in browser; check iframe src paths |
Recent Updates
JS to JSON Migration
- Video scripts now use JSON format instead of JavaScript modules
- Scripts located at
public/script/<name>.jsonwith{projects: {...}}structure - Frontend loads scripts via
fetch()instead of dynamic imports - Audio generation script reads JSON from filesystem
Chinese Speech Support
- Added
zh-CN-XiaoxiaoNeuralvoice for Mandarin narration - Enhanced word boundary matching for Chinese text processing
- Improved timing synchronization with Chinese TTS tokens
Rendering Enhancements
- Progress logging: Real-time feedback during frame rendering
- Accurate timing: Clip durations calculated from all WordBoundary events
- Silent projects: Automatic detection and skipping of audio generation
- Force audio regeneration:
--force-audioflag for updating changed speech
Word Trigger Optimization
- Enhanced
media.wordmatching with normalization and fuzzy matching - Support for Chinese characters and multi-word triggers
- Robust handling of TTS token variations
AI Script-Writing Tips
When authoring scripts for a user:
- Gather the brief first: topic, target length, audience, preferred theme, language/voice.
- Outline before scripting: list clip titles and key points, confirm with user.
- Write speech naturally: read it aloud mentally — does it flow? Avoid bullet-point prose.
- Interleave clip types: don't use the same type back-to-back more than twice.
- Code clips: show only the most relevant lines; add comments inside the code for clarity.
- End strong: last clip should summarize or call-to-action with an upbeat
speech. - After scripting: immediately proceed to generate audio and assets unless user says otherwise.