Video Editing

Video Editing

When to use

Reach for this skill any time you modify how recordings are edited: new edit operations, the timeline UI, transcript-driven cuts ("remove the ums"), the preview overlay, or the export flow. Editing must be non-destructive — the recording blob is immutable after upload.

Data model touched

• recordings.edits_json — a JSON column with the edit document. Shape:

  {
    "trims": [{ "startMs": 0, "endMs": 3200, "excluded": true }],
    "cuts":  [{ "startMs": 12000, "endMs": 13500 }],
    "speed": [{ "startMs": 0, "endMs": 60000, "rate": 1.5 }],
    "blurs": [{ "startMs": 0, "endMs": 90000, "box": { "x": 10, "y": 10, "w": 200, "h": 80 } }]
  }
  

• recordings.chapters_json — JSON array of { startMs, title }.

• application_state.editor-draft — in-progress editor state the user is previewing (recording id, playhead, preview playback speed, zoom, and edits JSON). Persisted edit operations still write directly to edits_json through actions.

Rules

1. Non-destructive. Never re-encode on edit. The original webm/mp4 stays intact at recordings.video_url. Edits only change the JSON.
2. Single source of truth. The player renders edits at playback time — read edits_json, compute the virtual timeline, and skip excluded ranges via HTMLVideoElement.currentTime seeks. Do not fork the edit model for the editor vs the player.
3. Export is explicit. The user must click Export to render a new file. That call goes through export-video and kicks off ffmpeg.wasm (or server-side ffmpeg if the recording is long).
4. Append, don't rewrite. Prefer pushing a new entry into edits_json over editing an existing one, so undo/redo can reverse a single edit without ambiguity.

Operations

All of these append to edits_json. The action validates non-overlapping ranges per type (except speed, which compounds) and throws on bad input.

Transcript-based editing

Users love editing by text — click a filler word in the transcript, remove it. The editor uses recording_transcripts.segments_json to map words to { startMs, endMs } ranges; clicking a segment creates a cut edit for that range.

"Remove the filler words" is not solved inline — it delegates to the agent. See the ai-video-tools skill. The agent analyzes the transcript, proposes a list of cuts, writes them to editor-draft for review, and the user one-click approves.

Export

// actions/export-video.ts
export default defineAction({
  schema: z.object({ id: z.string(), format: z.enum(["mp4", "webm"]).default("mp4") }),
  run: async ({ id, format }) => {
    await assertAccess("recording", id, "editor");
    const rec = await getRecordingOrThrow(id);
    const edits = JSON.parse(rec.editsJson || "{}");
    // Short clips (< 2 min) render client-side via ffmpeg.wasm.
    // Longer ones are queued for server-side ffmpeg.
    return { exportId: enqueueExport({ recordingId: id, edits, format }) };
  },
});

For short recordings, load ffmpeg.wasm in a web worker, feed it the source blob + the edit list, and return the rendered file. For long recordings, enqueue a background job and have the UI poll export-status every 2s.

Player integration

The player is the canonical renderer for edits. Given edits_json, compute a VirtualTimeline:

interface VirtualTimeline {
  // Maps virtual ms -> source ms, skipping cuts/trims.
  toSource(virtualMs: number): number;
  // Total virtual duration after edits applied.
  durationMs: number;
  // Ranges to skip on playback: the player listens for timeupdate and seeks past them.
  excludedRanges: { startMs: number; endMs: number }[];
}

Never call video.currentTime with a raw segment index — always go through toSource.

Rules

• The edit UI writes to editor-draft on every change, and only writes to edits_json on Save (so Cmd+Z is cheap and the DB stays clean).
• Never mutate edits_json from db-exec. Use apply-edit or reset-edits.
• Speed edits compound (1.5 × 2 = 3x) — validate the result is in [0.25, 4].
• Blur coordinates are in source resolution, not display pixels. Always normalize against recordings.width / recordings.height.

Related skills

• ai-video-tools — filler-word removal and chapter generation propose edits via the agent.
• real-time-sync — when the agent writes edits_json, the player must reflect it; edits_json is part of the recordings row which is already on the sync list.
• video-sharing — exports honor the share's enableDownloads flag.
• storing-data — why edits live in a JSON column rather than a separate table.

Editor implementation reference

Concrete editsJson shape (editor team)

The editor writes and reads this exact shape. Any new edit operation MUST preserve this shape so the player, editor, and export pipeline stay in sync.

interface EditsJson {
  version: 1;
  // Ripple-style trim ranges. `excluded:true` ranges are skipped during
  // playback and collapsed during export. `excluded:false` entries where
  // `startMs === endMs` are SPLIT MARKERS used by the editor UI — they
  // never affect playback.
  trims: Array<{ startMs: number; endMs: number; excluded: boolean }>;
  blurs: Array<{
    id: string;
    startMs: number;
    endMs: number;
    x: number; y: number; w: number; h: number; // normalized 0..1
    intensity: number;
  }>;
  thumbnail?:
    | { kind: "url"; value: string /* absolute URL */ }
    | { kind: "frame"; value: string /* timeMs as string */ }
    | { kind: "gif"; value: string /* JSON: { url, startMs, durationMs } */ }
    | null;
  // Provenance for stitched recordings — set by `stitch-recordings`.
  stitchedFrom?: string[];
}

Comment videoTimestampMs, reaction videoTimestampMs, and recording_transcripts.segmentsJson timestamps all refer to original video time. The player converts to edited time; never assume the two are equal.

Timestamp helpers

app/lib/timestamp-mapping.ts is the single source of truth:

The player team may have its own copy of some of these. If so, consolidate on app/lib/timestamp-mapping.ts — never let the two drift.

Editor actions

Every mutation ends with writeAppState("refresh-signal", { ts: Date.now() }).

ffmpeg.wasm usage

app/lib/ffmpeg-export.ts lazy-loads @ffmpeg/ffmpeg (only fetched on first Export click; core wasm is ~30MB). Three entry points:

exportMp4(recording, edits, onProgress)  // kept-range concat → H.264+AAC MP4
exportGif(recording, startMs, durationMs, onProgress)  // animated thumbnail
exportConcat(sources, onProgress)  // stitching N recordings into one MP4

Assumed limits (tested in practice — update if you hit new ceilings):

• Single-threaded WASM, ~2GB memory ceiling per tab.
• Roughly 10 minutes of 1080p WebM→MP4 is the practical upper bound before tabs run out of memory. The editor surfaces a confirm dialog when effectiveDuration(...) > LONG_EXPORT_THRESHOLD_MS (10 min) and offers "Download original" as an escape hatch.
• Both the trim/concat paths and the exportConcat (for stitching) re-encode to H.264+AAC so the output plays everywhere.

Stitching

Decision: client-side ffmpeg concat rather than a virtual playlist that plays N sources sequentially. Rationale:

• A real combined MP4 plays in share links, embeds, iframes, and any mobile browser without special-case player logic.
• The virtual-playlist approach couples the player to the stitched structure and breaks sharing / thumbnails / comments (which are keyed off recording_id).

Flow from the UI:

1. stitch-manager.tsx collects the ordered list of source recordings.
2. exportConcat() fetches each videoUrl and concatenates with ffmpeg.wasm.
3. We upload the resulting blob via uploadFile() (or fall back to a data URL when no provider is configured — dev mode).
4. We call stitch-recordings with the uploaded URL + total duration; the action inserts a new recordings row with status: "ready" and editsJson.stitchedFrom set to the source IDs for provenance.

Waveform peak caching

computePeaks() decodes the video's audio track via the Web Audio API and downsamples to a 2000-point peaks array. Peaks are cached in application_state under waveform-<recordingId> so remounts don't recompute. The editor reads the cache first and falls back to computing only when the key is missing or corrupted.

Timeline filmstrip

The trim track shows video frames behind the waveform. Two paths, and the order matters:

1. Sprite (preferred). generate-filmstrip runs one ffmpeg pass (fps → scale → pad → tile) into a single JPEG grid, uploads it, and stores the URL plus grid geometry on the recording. The editor renders cells with CSS background-position — one cached image request, no video decoding in the browser.
2. Browser extraction (fallback). extractFilmstripThumbnails() seeks a detached <video> once per frame. Only used when there is no sprite, which means hosts without ffmpeg and local/dev media the server cannot fetch.

Three traps this design exists to avoid:

• Pass the proxied URL. Frame extraction must use getWaveformMediaUrl(), never recording.videoUrl — reading pixels back out of a cross-origin video taints the canvas and toDataURL throws, so provider media silently yields no filmstrip at all.
• Both paths sample cell midpoints, not 0 … duration endpoints. The strip renders N equal cells, so cell i must show the middle of the slot it occupies or every thumbnail sits up to half a cell from the time beneath it.
• Cell count comes from the geometry, trackWidth / (height × aspect), not from the sprite's frame count. Rendering all 40 frames across an unzoomed track makes each cell portrait, and object-cover then shows a narrow centre slice of each frame instead of a recognisable thumbnail.

The filmstrip never replaces the waveform — peaks still draw on top of a scrim, because frames say nothing about the audio.

Keyboard shortcuts (editor scope)

• Space — play / pause (even while focused in the editor area)
• Cmd/Ctrl+Z — undo the last trim (no redo stack)
• I / O — mark in / out of the current trim selection
• S — split at playhead (via split-button.tsx)
• Delete / Backspace — while the transcript editor has a selection: trim the selected text's timestamp range

File map

actions/
  trim-recording.ts   split-recording.ts   set-thumbnail.ts
  set-chapters.ts     stitch-recordings.ts
  undo-edit.ts        clear-edits.ts
app/lib/
  timestamp-mapping.ts   waveform-peaks.ts   ffmpeg-export.ts
app/components/editor/
  editor-layout.tsx       editor-toolbar.tsx
  waveform.tsx            trim-handles.tsx      timeline.tsx
  transcript-editor.tsx   chapters-editor.tsx
  thumbnail-picker.tsx    stitch-manager.tsx    split-button.tsx

When the recording route enters edit mode, render <EditorLayout recordingId={id} /> — it wires the toolbar, video preview, transcript editor, waveform, trim handles, timeline, and chapters sidebar. The toolbar includes a Loom-style preview-speed dropdown next to the playhead time; it changes video.playbackRate for trimming/review only and writes playbackSpeed into editor-draft so the agent can see how the user is previewing. The dialogs for thumbnail picking and stitching are mounted inside the layout and toggled by the toolbar.