Wordmotion

Wordmotion

Turn a blog post or essay (.md) into a Remotion video. This skill covers pipeline-specific decisions and hard-won lessons. For Remotion API details, consult the official docs or the demo sources in any installed card library (see Phase 4).

Pipeline Phases (follow in order)

0. Video Type Detection → 1. Audio Generation → 2. Data Generation → 3. Infrastructure → 4. Slide Design → 5. Assembly → 6. Render

Phase 0: Video Type Detection

Before generating anything, classify the video by its source text and the user's intent. Record the type in shots.json (top-level videoType field). The type drives card-retrieval preferences in Phase 4 — it never excludes a card, it only re-weights the default search.

Ask the user only when the signals genuinely split between two types. Otherwise detect and proceed.

Phase 1: Audio Generation

Tool

edge-tts CLI (Python). Ask the user which voice and playback speed they want. If they haven't decided, suggest they browse available voices with edge-tts --list-voices and pick one that matches the text's language and the tone they're going for.

Sentence splitting

Split source .md at sentence-ending punctuation (., !, ?, 。, ！, ？ — adapt to the text's language). Filter out empty strings and lone punctuation artifacts from quote splitting.

Generate script

Write scripts/generate-audio.mjs. For each sentence, shell out to edge-tts with the user's chosen voice and rate:

edge-tts --voice "<voice>" --rate="<rate>" --text "sentence text" --write-media "public/audio/s_001.mp3"

Use the template in scripts/generate-audio.mjs — it accepts the source file path and voice/rate as arguments.

• Output: public/audio/s_XXX.mp3 (zero-padded, sequential)
• Generate public/audio/manifest.json: [{ index, id, filename, text, charCount }]

Handle failures

• Empty-text sentences produce 0-byte files — delete them
• After cleanup, rename remaining files sequentially so filenames match manifest indices

Phase 2: Data Generation

Measure audio durations

Use afinfo (macOS) or ffprobe (Linux) on each file.

Generate src/data/captions.json

Walk sentences in order, accumulating currentMs += durationMs:

[{ "text": "...", "startMs": 0, "endMs": 1340, "sentenceIndex": 1, "audioFile": "s_001.mp3", "charCount": 7 }]

All startMs/endMs are absolute (relative to video start, not per-shot).

Generate src/data/shots.json

Group sentences into 15-25 shots. Each shot maps to one slide. Attributes: id, title, sentences (1-based), durationFrames, captionIndices (0-based), audioFiles. Also record videoType (from Phase 0) at the top level and, per shot, a one-line semanticSignature (content type / emotion / pace) — Phase 4 card retrieval uses both.

Shot grouping principle: sentences that share a theme → one slide. The slide summarizes the audio's key point — it does NOT repeat the transcript verbatim.

Phase 3: Infrastructure

Project setup

• Resolution and fps: ask the user. Common defaults are 1920×1080 at 30fps for landscape (YouTube/Bilibili) or 1080×1920 for vertical (TikTok/Shorts).
• Root.tsx: import shots.json, sum durationFrames for total, pass to Composition (no calculateMetadata needed)
• tsconfig.json: ensure resolveJsonModule: true
• Dependencies: @remotion/media, @remotion/captions
• Font: a serif font matching the text's language (Google Fonts)

Shared components (template code in assets/templates/)

Copy these into src/components/shared/:

Phase 4: Slide Design Rules

Visual philosophy

Each slide = a dynamic presentation page by default, not a cinematic scene. Typography, charts, tables and diagrams carry the content. This is the baseline — the video type may justify heavier treatment (e.g. a product video can borrow camera moves and 2.5D cards from an external card library), but information clarity always outranks spectacle.

Layout rules (non-negotiable)

1. Fill the canvas. No maxWidth constraints on content containers. Minimum font: 22px body, 28px headings, 36px+ for key messages. Use flex: 1 to stretch content vertically.
2. Dark theme. Background #0a0a0f, text #e5e7eb, accent #f59e0b, danger #ef4444.
3. Bottom safety. SlideFrame gives 160px bottom padding. Never use position: absolute; bottom: <small_value>. Subtitles occupy the bottom ~140px.
4. Every slide MUST have at least one element that animates continuously from start to finish. Not just entrance — sustained motion. The particle background counts, but also add one visible element animation:
   • Pulse: Math.sin(frame * 0.08) * 0.5 + 0.5
   • Float: Math.sin(frame * 0.04 + i) * 6 (px)
   • Blink: Math.floor(frame / 15) % 2 === 0
   • Rotate: Math.sin(frame * 0.05) * 3 (degrees)
   • Shimmer: Math.sin(frame * 0.07) * 0.15 + 0.15

Baseline content patterns (fallback library)

Pick one per shot based on the message when no external card matches (see the card retrieval protocol below):

External card retrieval protocol (card libraries)

The baseline table is the fallback, not the ceiling. If a card library is installed, search it before settling for a baseline pattern.

Supported libraries

A library is "present" when its card root exists. Never hard-fail when a library is missing — the pipeline runs on baseline patterns alone.

Retrieval per shot

For every shot in shots.json:

1. Semantic signature. Write one line: content type (statement / data / contrast / process / question / conclusion), emotional intent, and pace need. This is the search key.
2. Category lookup. Shotcraft's directories are the index: typography, data, opening, outro, transition, effects, rhythm, camera, ui-entrance, interaction. Map the signature to one or two categories.
3. Default weights by video type. These only re-order the search — override freely when a shot's semantics demand it:

(● = preferred, ○ = allowed. Nothing is forbidden.)

4. Read, don't guess. When a card matches, read the recipe card .md in full and the exact demo source its "参考实现" points to. The demo holds the tuned truth — easing, timing, pitfalls. Re-writing from the card name alone throws away all of that tuning. Cards mark "已知坑/命门" parameters: never downgrade them.
5. Adapt, don't copy verbatim. Port the card through templates/card-adaptation.md before it enters the slide.
6. Record usage. Append every card used to src/data/cards-used.json: { shotId, library, category, card, demoSource, attributionNote }. Keep the library's ATTRIBUTION notice intact.
7. No match → fallback. If no card fits, use a baseline pattern. Never force a card that fights the audio.

Card adaptation rules (summary — full checklist in templates/card-adaptation.md)

• Re-skin to the wordmotion dark tokens; carry the card's motion grammar and tuned parameters, not its colors.
• Keep the 160px subtitle safety zone and the canvas-filling layout rules.
• Ensure at least one continuously animating element per slide; entrance-only cards get a sustained accent (pulse / float / shimmer).
• Map card durations onto absolute frames inside the shot's <Sequence>; never use TransitionSeries.
• Deterministic rendering: no Date.now() / Math.random(); seed all randomness.

Phase 5: MainVideo Assembly (Critical)

Audio-visual sync — DO NOT use TransitionSeries

TransitionSeries overlaps cause audio/subtitle timing drift. Use plain <Sequence> with absolute frame positions:

// Calculate each shot's absolute global start frame (NO overlap)
const shotPositions = useMemo(() => {
  let frame = 0;
  return shots.map(s => { const start = frame; frame += s.durationFrames; return { ...s, start }; });
}, []);

Audio per shot

Each shot plays its sentence audio files in order, offset by their relative start times within the shot:

// Inside each shot's Sequence:
audioFiles.map((file, i) => (
  <Sequence from={Math.round((captions[i].startMs - captions[0].startMs) / 1000 * fps)}>
    <Audio src={staticFile(`audio/${file}`)} />
  </Sequence>
))

Subtitles at root level

Render <Subtitles captions={allCaptions} /> OUTSIDE all shot Sequences. The Subtitles component:

• Uses useCurrentFrame() at root level → gets absolute global frame
• Finds active caption by currentTimeMs >= startMs && currentTimeMs < endMs
• Splits text into characters, highlights progress * length characters in gold
• Font: 44px serif, bottom 72px, z-index 100

Full structure:

<AbsoluteFill style={{ backgroundColor: "#0a0a0f" }}>
  {shotPositions.map(shot => (
    <Sequence key={shot.id} from={shot.startFrame} durationInFrames={shot.durationFrames}>
      <SlideComponent />
      <ShotAudio ... />
    </Sequence>
  ))}
  <Subtitles captions={allCaptions} />
</AbsoluteFill>

Phase 6: Render

npx remotion render <CompositionId> out/output.mp4

Common Pitfalls