Remotion Video

Remotion Video — Encode the Actual Frames

You are the encoder and renderer. You take assets — a recording, a voiceover, b-roll, a transcript — plus React code, and you emit a real file: out/video.mp4. Your siblings write words and plans; you are the only one that produces pixels. The rigor here is reproducible frames: same input, same deterministic output, verified by a render that ffprobe can read.

You own the Remotion project scaffold, the <Composition> / <Sequence> / <TransitionSeries> graph, the captions pipeline (Whisper.cpp → toCaptions → createTikTokStyleCaptions), the silence-removal pass, and the npx remotion render invocation with its codec and concurrency flags.

The one decision: frames or words?

If the ask is to produce a file, you are in the right place. If it is to produce text or a plan, route out before writing a single .tsx.

Boundary in one line: video-shorts decides the cuts and writes the caption text; you execute the cuts in code and burn the captions into frames.

Scaffold the project

npx create-video@latest --yes --blank my-video
cd my-video
npm i
npm run dev        # opens Remotion Studio in the browser

Remotion's current stable line is 4.0.471; it needs Node 16+ (or Bun 1.0.3+), and local rendering targets macOS 15 (Sequoia)+. Since January 2026 Remotion ships Agent Skills — npx skills add remotion-dev/skills wires Remotion-aware guidance into Claude Code. Run it inside a Remotion project when you want the framework's own skill loaded alongside this one.

Pin fps and dimensions on <Composition> first, and never change them mid-project. Why: every duration downstream is measured in frames, and frames = seconds * fps. Change fps after you have written durations and every timing silently shifts. Vertical shorts are 1080×1920 @ 30; landscape is 1920×1080 @ 30.

The composition graph

// src/Root.tsx
import { Composition } from "remotion";
import { MyVideo } from "./MyVideo";

export const RemotionRoot: React.FC = () => {
  return (
    <Composition
      id="MyVideo"            // the id you pass to `remotion render`
      component={MyVideo}
      durationInFrames={300}  // 10s at 30fps
      fps={30}
      width={1080}
      height={1920}
    />
  );
};

// src/MyVideo.tsx
import { AbsoluteFill, Sequence, useCurrentFrame, interpolate, spring, useVideoConfig } from "remotion";

export const MyVideo: React.FC = () => {
  const frame = useCurrentFrame();
  const { fps } = useVideoConfig();
  const opacity = interpolate(frame, [0, 30], [0, 1], { extrapolateRight: "clamp" });
  const scale = spring({ frame, fps, config: { damping: 200 } });
  return (
    <AbsoluteFill style={{ backgroundColor: "black" }}>
      <Sequence from={0} durationInFrames={90}>
        <AbsoluteFill style={{ opacity, transform: `scale(${scale})` }}>{/* scene 1 */}</AbsoluteFill>
      </Sequence>
      <Sequence from={90} durationInFrames={210}>{/* scene 2 */}</Sequence>
    </AbsoluteFill>
  );
};

Animate off useCurrentFrame() with interpolate() and spring() only. Never read wall-clock time (Date.now()) or call Math.random() unseeded inside a composition. Why: rendering is parallel and frame-addressable — each frame is computed independently, so any non-frame input produces a different pixel on re-render and breaks the "same input, same output" guarantee. If you need randomness, use Remotion's random(seed).

Transitions

Use @remotion/transitions (available since v4.0.53). <TransitionSeries> interleaves .Sequence (a clip, with durationInFrames) and .Transition (a presentation + a timing). The transition duration is subtracted from the total, so adjacent sequences overlap during the wipe.

import { TransitionSeries, linearTiming, springTiming } from "@remotion/transitions";
import { slide } from "@remotion/transitions/slide";
import { fade } from "@remotion/transitions/fade";
import { Easing } from "remotion";

<TransitionSeries>
  <TransitionSeries.Sequence durationInFrames={90}>{/* scene A */}</TransitionSeries.Sequence>
  <TransitionSeries.Transition
    presentation={slide({ direction: "from-left" })}
    timing={springTiming({ config: { damping: 200 }, durationInFrames: 30, durationRestThreshold: 0.001 })}
  />
  <TransitionSeries.Sequence durationInFrames={120}>{/* scene B */}</TransitionSeries.Sequence>
  <TransitionSeries.Transition
    presentation={fade()}
    timing={linearTiming({ durationInFrames: 15, easing: Easing.inOut(Easing.ease) })}
  />
  <TransitionSeries.Sequence durationInFrames={90}>{/* scene C */}</TransitionSeries.Sequence>
</TransitionSeries>

Each presentation is a sub-import (@remotion/transitions/slide, /fade, /wipe, /flip, /clockWipe, /none).

linearTiming for predictable, frame-exact cuts; springTiming for organic motion. Why: linear is deterministic in duration so you can budget frames exactly; spring overshoots and settles, which reads as natural but needs durationRestThreshold so the render knows when it has finished.

Animated burned-in captions

The native @remotion/captions package shipped in v4.0.216 (the same release that deprecated the old convertToCaptions() helper). The pipeline runs once on a Node server, then the composition reads the captions:

1. Transcribe. @remotion/install-whisper-cpp downloads Whisper.cpp and a model (medium.en is ~1.5 GB) and transcribes the audio on a Node server to Whisper JSON.
2. Convert. toCaptions() from @remotion/install-whisper-cpp turns that JSON into a Caption[] with per-token timestamps. (convertToCaptions() is the legacy alias — deprecated as of v4.0.216; use toCaptions().)
3. Segment into pages. @remotion/captions createTikTokStyleCaptions({ captions, combineTokensWithinMilliseconds }) groups tokens into "pages" that appear together.

The combineTokensWithinMilliseconds value is the page-size dial. Why: a low value (200ms) keeps each word on its own page → word-by-word pop animation; a high value (1200ms) packs a phrase per page. Low ms = TikTok word-by-word energy; high ms = readable phrases. Pick by the format, not by default.

Full Whisper.cpp install, the transcribe server, and the token-highlight caption renderer component (with safe-zone styling) live in references/captions-pipeline.md — read it before building the captions layer.

Automatic silence removal

The silence pass runs on the source audio/video BEFORE it enters Remotion, not inside a composition. Why: Remotion renders frames you give it; trimming dead air is an upstream edit on the asset, and doing it first means every downstream frame number already reflects the tightened timeline.

Use auto-editor (a Python + ffmpeg engine) for a first pass that cuts dead space by audio loudness:

auto-editor input.mp4 --margin 0.2s --edit audio:threshold=4% -o tightened.mp4

• --margin pads each kept region so cuts do not clip speech.
• --edit audio:threshold=4% sets the loudness floor below which a region is "silence".
• --export premiere emits an EDL/XML instead of a file, to re-import into an NLE.

pip distribution is stale/discontinued — install via the official binary or pipx, not pip install. Why: the PyPI package lags behind and may not match the documented flags. When auto-editor is unavailable, the low-level fallback is ffmpeg's silencedetect / silenceremove filters. Both, with the full flag matrix, are in references/render-and-pipeline.md.

B-roll overlays

Stack the overlay above the main video by layering <OffthreadVideo> (or <Img>) inside an <AbsoluteFill>, gated by a <Sequence from>:

import { AbsoluteFill, Sequence, OffthreadVideo, staticFile } from "remotion";

const fps = 30;
const broll = { start: 4.0, duration: 3.0 }; // seconds
<AbsoluteFill>
  <OffthreadVideo src={staticFile("main.mp4")} />        {/* base layer */}
  <Sequence from={Math.round(broll.start * fps)} durationInFrames={Math.round(broll.duration * fps)}>
    <AbsoluteFill style={{ /* e.g. inset for picture-in-picture */ }}>
      <OffthreadVideo src={staticFile("broll.mp4")} />    {/* overlay layer */}
    </AbsoluteFill>
  </Sequence>
</AbsoluteFill>

Convert every timecode to frames with Math.round(seconds * fps), once, at the edge. Why: a b-roll cue at 4.0s is frame 120 at 30fps but frame 240 at 60fps — keep seconds in your data and multiply by fps from useVideoConfig() so changing fps never desyncs overlays. Use <OffthreadVideo> (not the DOM <video> or <Video>) for frame-accurate decoding during render.

Render

# region test first: 1-2 seconds, validates the pipeline cheaply
npx remotion render MyVideo out/test.mp4 --frames=0-45

# then the full render
npx remotion render MyVideo out/video.mp4

Omit the composition id to get an interactive picker. Configure via @remotion/cli/config in remotion.config.ts, or pass flags on the CLI (flags win):

// remotion.config.ts
import { Config } from "@remotion/cli/config";
Config.setConcurrency(8);
Config.setCodec("h264");

Always render a 1–2s region (--frames=0-45) before the full render. Why: a full render of a minutes-long composition costs real time and CPU; a region test surfaces a broken caption layer or missing asset in seconds. For headless/CI renders, deterministic-output rules, and the codec/quality matrix in full, see references/render-and-pipeline.md.

Anti-patterns

Verify & references

• bash scripts/verify.sh <project-dir> — checks the Remotion project is well-formed and renders. With Node + ffmpeg present it runs npx remotion compositions to confirm a composition id exists, does a short region render to a temp MP4, and uses ffprobe to confirm a video stream with the expected dimensions/fps. With neither, it falls back to a static check: at least one composition .tsx exists and the render script references a real composition id and an output path. Read-only by default; exits 0 on an empty/clean target.
• references/captions-pipeline.md — full Whisper.cpp install + transcribe server, toCaptions, the token-highlight caption renderer component, and caption styling/safe-zone notes.
• references/render-and-pipeline.md — auto-editor + ffmpeg silence commands, the full render flag matrix, headless/CI render, the codec table, and ffprobe verification.