Video Bg Replace

Video BG Replace

Put a talking-head speaker on a new background. The speaker is matted with Robust Video Matting (RVM) on Replicate — a recurrent net with temporal memory, so the matte stays stable frame-to-frame instead of flickering like per-image background removers — then composited over a new background video/image with ffmpeg.

This is the compositing counterpart to broll-generator: that skill generates a silent animated background, this skill marries the matted speaker to it.

The key idea

Both "mask the speaker and drop them on a new bg" and "put an animated video behind the front video" reduce to the same layer stack:

[ new background video/image ]   ← bottom layer
[ matted speaker (with alpha) ]   ← top layer

There's no way to see "behind" the front clip without transparency, so the speaker must be matted. Talking-head clips from avatar-talking-video / p-video-avatar / seedance are rendered full-frame with their own background, so matting is always required (you can't skip it by generating on green).

Setup

Shares the Replicate token with the other Replicate-based skills, so if any of them is configured it's found automatically.

pip3 install -r .cursor/skills/video-bg-replace/scripts/requirements.txt
# Only if no sibling skill has a token yet:
python3 .cursor/skills/video-bg-replace/scripts/setup_key.py r8_YOUR_TOKEN

ffmpeg/ffprobe must be on PATH (libx264).

Quick start

SCRIPTS=.cursor/skills/video-bg-replace/scripts

# Route A (recommended): alpha matte + original RGB, composited over a b-roll bg
python3 $SCRIPTS/replace_bg.py lolo/generated-videos/scene01.mp4 \
  --bg lolo/broll/003_calle-de-noche.mp4 --shadow --grade

# See the planned RVM call + ffmpeg command without spending anything
python3 $SCRIPTS/replace_bg.py scene01.mp4 --bg bg.mp4 --dry-run

Output: <avatar>/generated-videos/bg-replaced/<speaker>__bg-<bg>.mp4 (when the speaker lives under an avatar folder) plus a manifest.json entry. A JSON summary is printed to stdout for an orchestrating skill.

How it works

1. Matte (RVM, Replicate) — arielreplicate/robust_video_matting returns a single video per run; output_type selects what it renders:
   • --matte alpha → output_type=alpha-mask → grayscale matte (default)
   • --matte green → output_type=green-screen → speaker on green
2. Composite (ffmpeg):
   • Route A (alpha) — alphamerge the original speaker RGB with the matte (true colors, no green spill), then overlay on the background.
   • Route B (green) — chromakey + despill the green clip, then overlay.
3. Background prep — the bg is cover-scaled + cropped to the target frame and looped (videos via -stream_loop, images via -loop 1) so a short clip covers the full speaker duration; output is trimmed to the speaker's length.
4. Audio — the speaker's original audio is preserved (the RVM output has none); in the green route the original clip is re-attached just for audio.
5. Manifest — every render is recorded (paths, route, model, dims, options).

Route A vs Route B

Prefer A. Use B only if the alpha matte disappoints on a given clip. If RVM edges aren't good enough at all, the same graph accepts a matte from a higher-quality video matter (MatAnyone / BEN2) via --reuse-matte.

Making the composite believable

The difference between "pasted on" and "shot there":

• --feather 1.5 — soften the matte edge to kill the halo.
• --shadow — soft grounding drop-shadow under the subject (alpha route).
• --grade — subtle contrast/saturation + vignette over the whole composite so subject and background share a color temperature.
• Motion match — a locked-still subject over a moving bg reads as fake. Keep the bg motion subtle, or give the generated bg a gentle camera move (see broll-generator --camera).

Iterating cheaply

RVM costs an API call per run, so once you have a matte, iterate the composite for free:

# 1) Render once, keeping the matte
python3 $SCRIPTS/replace_bg.py scene01.mp4 --bg bgA.mp4 --keep-matte
#    -> writes ...scene01__bg-bgA.mp4 and ...scene01__bg-bgA.matte.mp4

# 2) Reuse the matte against other backgrounds / settings — no API spend
python3 $SCRIPTS/replace_bg.py scene01.mp4 --bg bgB.mp4 \
  --reuse-matte .../bg-replaced/scene01__bg-bgA.matte.mp4 --shadow --grade

Key options

Fits the avatar pipeline

Use as a leaf skill the way broll-generator is: generate a background with broll-generator (or seedance), render the talking head with avatar-talking-video, then call replace_bg.py per talking-head scene. The composited clip then flows through the normal avatar-reel-composer finishing pass (captions, music, transitions). Mattes and composites are cached under the avatar's generated-videos/ like the rest of the generated media.

Notes

• RVM output has no audio — Route A keeps the original RGB clip (audio intact); Route B re-attaches the original clip just for its audio track.
• Output defaults to the speaker's own resolution to keep the face crisp; the background conforms to it (cover-fit + crop).
• Matting is a long-running Replicate op (~seconds to a minute depending on clip length); the composite is local ffmpeg.