# Multicam Skill

> Multicam

- Skill: `jianshuo/multicam-skill` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add jianshuo/multicam-skill`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jianshuo/multicam-skill/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: jianshuo (https://skillmd.com/u/jianshuo)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jianshuo/multicam-skill

---


# Multicam

Two phases. Each works standalone — sync without editing is fine; editing assumes a synced set.

## Phase 1 — Audio Sync

Goal: find a single time offset for each camera so the timelines line up, then trim every file to the common overlap window. Output files are byte-clean playable MOVs/MP4s with matching durations.

### Algorithm (proven, robust under noise)

1. **Extract mono PCM at low SR** (8 kHz, 16-bit) from each input. Don't use raw waveform cross-correlation — at low SNR (typical for the second/third camera with a worse mic) it gives weak peaks and false matches.
2. **Build a log-energy envelope** at 100 Hz frame rate (10 ms hop, 50 ms window). High-pass it (cutoff 0.05 Hz, 2nd-order Butterworth, filtfilt) to remove slow drift / gain differences. The envelope captures dialogue/music dynamics — the same event hits both mics regardless of room response.
3. **FFT cross-correlate the envelopes** end-to-end. The peak gives a coarse offset to ~10 ms.
4. **Refine at sample level** with a 60 s probe from B near the coarse-aligned position in A, using a ±2 s search window. Parabolic peak interpolation gets sub-sample precision.
5. **Multi-probe drift check**: run step 4 every ~3 minutes across the timeline. Linear fit `delta(t) = slope·t + intercept` reveals real clock drift between cameras (typical: 5–50 ppm). Use the **midpoint canonical** offset (`slope · midpoint + intercept`) so residual error is symmetric around zero across the duration.
6. **Compute overlap window** in each camera's local timeline using the chosen delta:
    - `delta = tB0 − tA0` (B's start in wall-clock minus A's start). If delta > 0, A leads.
    - In A: overlap is `[max(0, delta), min(A_dur, delta + B_dur)]`.
    - In B: same span shifted by `−delta`.
7. **Trim each file** to its overlap window with re-encode (stream-copy can't cut mid-GOP; the trim must be frame-accurate). On macOS use `hevc_videotoolbox` (5–10× realtime on Apple Silicon). For shorter clips x264/x265 software encode is fine.

### Partial-coverage clips (one source covers only part of the session)

A common case: the main cameras run for the whole 75-min session, but a third recorder (Riverside, separate audio recorder, late-arriving phone) only covers the middle 30 min. Sync that to the existing common timeline without touching the main cams.

`scripts/sync_partial.py REF_SYNCED.mov NEW_INPUT.mp4` does this:

1. Cross-correlates the new input's audio against the reference's audio.
2. Finds the start offset (where the new clip's t=0 sits in the reference's timeline).
3. Produces a file with the same duration as the reference, with leading black + silence padding so it's drop-in usable as another timeline-aligned input.
4. Writes a `.sync.json` sidecar describing the offset/trim plan.

Use `--audio-only` for audio-only sources (saves the cost of encoding 30+ minutes of black video).

The padded approach trades disk for ergonomics — the file plays naturally alongside the main cams in any NLE without per-track time offsets. If disk matters more than drop-in ergonomics, skip the script's encode step and use the sidecar JSON's `delta_seconds` as an `-itsoffset` argument when adding the input to ffmpeg.

### When to skip drift correction

For camera-cut editing (the common case), ±25 ms residual across an hour is below human perception. Just use the static midpoint offset.

For sync-sound or lip-sync at very long durations (>30 min, drift >40 ms), apply `atempo` to the slower file: `atempo = 1 + slope`. Re-encode is required.

### Reference script

`scripts/sync.py` is the end-to-end implementation: discovers inputs, extracts envelopes, finds the offset, runs multi-probe drift analysis, prints the trim plan, and emits an `ffmpeg` command per file. Read it before adapting.

### Output naming

`<stem>_synced.MOV` next to each input. If A doesn't need trimming (its window is fully inside B's), still produce `A_synced.MOV` so the user has a clean pair with matching names and durations. Stream-copy A — no re-encode needed.

### Verification (always run)

After producing the synced pair, re-extract audio and run the multi-probe correlation on the **outputs**. Median residual should be a few ms; the spread tells you the residual drift. If the spread exceeds 1 frame at the target frame rate, retry with drift correction. `scripts/verify.py` does this.

## Phase 2 — Director-Style Auto-Edit

Goal: produce a single render using the cleanest audio and switching among cameras + virtual crops at moments a human director would pick.

This is open enough that you should brainstorm with the user before executing — confirm style (talking-heads, performance, interview, vlog), pacing preference (fast cuts vs. long takes), and whether they want PiP or only hard cuts.

### Audio selection

The "main" audio track is one of:
- The cleanest single mic (use SNR + spectral flatness — pick the track with highest median SNR over voice-active frames).
- A multi-mic mix with per-speaker gating (only if you have speaker diarization and per-mic energy maps; usually overkill for two-camera shoots).

Compute SNR per file by VAD-segmenting the envelope: the 90th percentile / 10th percentile ratio in dB. Pick the track with the largest ratio. Verify by listening to a 30 s sample.

### Camera switching — "Director HMM"

Model the edit as a state machine where each state is a candidate shot (each physical camera plus virtual crops/PiP variants). Transitions are scored from a few signals:

| Signal | What it favors |
| --- | --- |
| **Active-speaker energy** | Camera whose mic is loudest right now (proxy: closest mic = closest to speaker). |
| **Face / framing score** | Cameras where a face is well-framed (rule-of-thirds, eyes upper third). Use `face_recognition` or MediaPipe at 1 fps. |
| **Composition variety** | Penalize the current shot the longer it stays on (boredom term: rises linearly after 8 s, sharp climb after 20 s). |
| **Cut cost** | Penalize switching too soon (min cut ~3 s for talking heads, ~1.5 s for action). Hard floor: 1.5 s. |
| **Beat alignment** | Prefer cuts on silence boundaries or at the start of a new utterance — detect via VAD edge transitions. |
| **Reaction-shot bonus** | When speaker A talks, occasional cut to listener B for ≤ 4 s. |

Combine via Viterbi: per-second scores, transition penalty for cut cost, decode the lowest-cost path. This produces the EDL.

**Rules of thumb the algorithm should encode:**
- Open on the master/wide shot for 5–10 s before the first cut.
- Hold a shot ≥ 3 s. Cut on the first natural beat after that.
- Don't ping-pong: same pair can't repeat within 8 s.
- After 25 s on one shot, force a cut at the next beat regardless.
- Reaction shots ≤ 4 s, returning to the active speaker.

### Virtual cameras (crop-zoom from a wide shot)

A wide camera with high resolution (1080p+) yields multiple "virtual cameras" by cropping. Useful when the second physical camera is unavailable or its angle isn't right.

Implementation: pre-compute a face track (one box per frame at 1 fps, interpolated). Define virtual crops:
- **Wide**: full frame.
- **MCU (medium close-up)**: crop centered on the active speaker's face, 2× face height.
- **CU**: crop, 1.2× face height.
- **2-shot**: bounding box of two faces with 20% padding.

Crops are smoothed with a low-pass filter (0.5 Hz cutoff) to avoid jitter. Output via `ffmpeg crop` filter, scaled back to delivery resolution (e.g. 1920×1080).

### Picture-in-picture

Treat the inset as a separate "shot" in the HMM. Use sparingly (≤ 15% of total runtime). Common patterns:
- Speaker on main, listener PiP bottom-right (Q&A, interview).
- Wide on main, CU PiP top-right (performance / demo).

Transition: 0.3 s opacity fade in, hard cut out (or vice versa). Avoid sliding animations — they look amateur.

### Render

Two paths, pick by complexity:

**Pure ffmpeg** (no PiP, no fancy transitions, just hard cuts + crops): write a `concat` filter graph from the EDL. Each segment is `ss=...:t=...,crop=...,scale=...` from one input. Fastest, no extra deps.

**HyperFrames** (PiP, transitions, captions, audio-reactive elements): generate `index.html` with `<hf-clip>` elements per EDL row, each `<hf-clip>` containing a `<video>` whose `data-clip-time` controls playback. Use `hyperframes-cli render` to produce the final MP4. Best when the user wants overlays, transitions, or branding.

For pure cuts, ffmpeg is faster and simpler. For anything with PiP or motion graphics, prefer HyperFrames — that's what it's built for, and the `hyperframes` skill covers the composition patterns.

### Reference script

`scripts/autoedit.py` skeleton (not full implementation): runs VAD per camera audio, computes per-camera scores, runs Viterbi, emits an EDL JSON. Stop short of full face/composition scoring on a first pass — get hard cuts working with active-speaker scoring alone, listen to results, then add complexity.

## File layout convention

Inputs in a working directory; outputs alongside.

```
working_dir/
  cam_a.MOV                  # original
  cam_b.MOV
  cam_a_synced.MOV           # Phase 1 outputs
  cam_b_synced.MOV
  edl.json                   # Phase 2 EDL (camera id, start, end, crop, audio_source)
  multicam_render.mp4        # Phase 2 render
```

## Common pitfalls

- **Raw waveform cross-correlation gives weak/false peaks** when the two mics have different gain or room response. Always use envelope first.
- **Stream-copy trim is not frame-accurate** — it cuts at the previous keyframe (often 2 s off). Re-encode the trim.
- **Display Matrix metadata is dropped during re-encode** unless you preserve it. iPhone videos often have an identity display matrix on visually-horizontal content stored as 1080×1920 — extract a frame to confirm orientation before delivering.
- **Audio-only sync misses long-period drift**. Always run the multi-probe verification on the outputs.
- **Don't pick "best" per-second cam without hysteresis** — the result ping-pongs. Min-cut-length and cut-cost terms in the HMM are non-optional.