Local Lipsync Pipeline

Local Lip-Sync Pipeline for Stylized Characters

Overview

Off-the-shelf lip-sync models were trained on real human faces and degrade in specific, fixable ways on stylized input. The fixes live in the order you run things, the swap points between stages, and a handful of compositing details. This skill captures that pipeline end-to-end.

Key terms used throughout:

• Stylized character — any non-photoreal speaking face. Sub-styles: cel-shaded 2D (anime, classic Western cartoons — flat fills, hard line art), 3D rendered (Pixar/Disney/Arcane-style CG with soft shading), semi-realistic CG (stylized human faces with photoreal-leaning shading), illustrated/painted (textured 2D, brushwork visible), puppet/mascot (physical foam or rigid-shell heads, often hinged or fixed mouths).
• Real-face detector — trained on photographs of human faces (RetinaFace, S3FD, MTCNN, OpenCV Haar "face" cascades). Opposite: anime/illustration detectors trained on drawn faces.
• Locked-off shot — static camera with no pan, tilt, zoom, or roll. Face stays in roughly the same image-space region; manual bbox is trivially viable.
• Double-hop — audio→cartoon technique: generate a driver video from audio using SadTalker, then feed that driver into LivePortrait. Two model passes.
• CFR / VFR — constant-frame-rate vs. variable-frame-rate source. All sync models here assume CFR; VFR sources must be conformed first.

Decision-tree, compositing, licensing, and orchestrator details are in this SKILL.md (Step 0, Steps 4–6, Orchestration, Licensing). This pack does not ship separate companion files.

When to Use

Use this skill when:

• Lip-syncing a stylized or non-photoreal character to a voice track on local GPU hardware.
• A real-face detector is rejecting a stylized face (cel-shaded, anime, puppet, mascot).
• You see jaw or chin clipping after Wav2Lip paste-back.
• You see blurry hallucinated mouth texture or visible mask seams.
• Audio drifts, or lips fire before/after sound.
• Ghosting appears on already-animated source frames.
• "Real-time" claims fall below realtime on the user's hardware.
• VRAM exhaustion occurs when chaining models in one process.
• torch/onnxruntime pin conflicts break a shared environment.
• MPS/Apple Silicon limitations surface (MuseTalk ONNX paths fall back to CPU).
• Multi-character dialogue needs per-speaker sync passes.
• Occluded mouths (hand over mouth, head turn) produce ghost mouth shapes.
• HDR or alpha-channel source needs pre-processing before sync.
• License questions arise around Wav2Lip/GFPGAN/CodeFormer weights for commercial delivery.

Skip when the question is a generic ffmpeg/audio question that happens to mention a stylized character but doesn't involve sync — answer those directly.

Prerequisites

GPU / Platform

• CUDA on Linux/Windows is the smooth path.
• MPS (Apple Silicon): works partially for LivePortrait; MuseTalk's ONNX-accelerated paths are CUDA-only and fall back to CPU on Mac.
• ROCm (AMD) and Intel Arc: treat as CUDA-equivalent in API but expect rough edges. Install torch ROCm wheels and onnxruntime-rocm (AMD) or onnxruntime-openvino (Intel). Some custom CUDA kernels in MuseTalk have no ROCm/OneAPI port and fall back to slow paths.
• CPU-only: Wav2Lip will run (slowly); MuseTalk and LivePortrait are impractical without a GPU.

VRAM

• Under ~12 GB free: chain stages as separate subprocesses so VRAM is released between them. Loading all three models in one Python process is the most common OOM pattern.
• Over 12 GB: chaining is safe but still optional.
• The orchestrator probes VRAM up front and picks chained vs. serialized from the result.

Environment Isolation

Wav2Lip's reference code expects torch 1.13 with the old FP16 API; MuseTalk wants torch 2.0+; LivePortrait wants 2.3+. Pinning all three into one env works briefly, then breaks on the next upgrade.

Default: one Python environment per model, orchestrated by a top-level script that shells out.

Escape hatches: Docker, uv workspaces, or pip-tools constraints — don't tear down a working multi-env or container setup just to match a convention. A single env with careful pinning is possible but brittle across upgrades — only attempt if explicitly requested.

Suggested layout (on Windows substitute Scripts\python.exe for bin/python):

lipsync/
├── envs/
│   ├── wav2lip/         # python 3.8, torch 1.13
│   ├── musetalk/        # python 3.10, torch 2.0+
│   ├── liveportrait/    # python 3.10, torch 2.3+
│   └── enhance/         # gfpgan + codeformer + realesrgan
├── models/              # shared checkpoints (symlink into each env)
├── work/                # scratch frames, masks, alignments
└── orchestrator.py      # runs on its own python (3.10+), outside any env above

The orchestrator runs on its own Python interpreter (3.10+ — it uses list[str] and pathlib idioms that the Wav2Lip env's 3.8 wouldn't accept) outside the per-model envs — it never imports torch, it just shells out. Symlinking the shared models/ directory into each env keeps a single source of truth.

Checkpoint Sourcing

• Pin checkpoint files by SHA-256 in a small models/manifest.toml and verify on download. The orchestrator refuses to run with mismatched hashes — a silently corrupted MuseTalk weight produces plausible-looking sync that's subtly wrong.
• Maintain a fallback mirror list per checkpoint (HF mirror, IPFS pin, internal artifact storage). When the primary 404s, fall through.
• Cache checkpoints in models/ and never inside the per-env site-packages — re-installing an env should not re-download a 2 GB weight.

Model Selection

A common winning combination: MuseTalk for the mouth + LivePortrait for everything else + an enhancer + a feathered composite mask.

Two newcomer surprises to surface up front: LivePortrait needs a driver video, and Wav2Lip's 96×96 ceiling is a property of the architecture, not a parameter to tune.

Procedure

Step 0: Decision Tree (do this first)

Answer these before any rendering — they pick the model and gate later choices, and re-litigating after a 20-minute render wastes time:

1. Sub-style → enhancer. Cel-shaded 2D / anime → RealESRGAN (no face prior, so it won't fight the line art). 3D rendered → CodeFormer (handles soft shading well). Semi-realistic CG → GFPGAN (lightweight face prior, identity-preserving, roughly 2× the inference speed of CodeFormer on equivalent hardware in our measurements). Puppet / mascot / illustrated → start with RealESRGAN; GFPGAN and CodeFormer tend to push features toward photoreal proportions on these inputs, which is rarely what you want.
2. Driver video available? Yes → LivePortrait is viable. Audio only → Wav2Lip or MuseTalk, or the double-hop (§3c) that generates a driver from audio first.
3. Camera / head motion. Locked-off shot → a fixed manual bbox is the fastest and most stable path. Moving shot → tracker or per-frame detector; flickery detection is a major cause of jitter complaints.
4. Delivery fps (24, 23.976, 25, 29.97, 30). Wav2Lip, MuseTalk, and LivePortrait checkpoints were all trained at 25 fps. Sync internally at 25 and conform to the delivery fps in the final remux. Running these models on 30 fps source without conforming isn't fatal, but mouth motion drifts at the ratio difference (~17% on 30 → 25).
5. Aspect ratio. 16:9 and 9:16 (vertical/reels) both work, but the mask heuristics in §5 (ellipse at 70% of bbox height, etc.) were measured against 16:9 with roughly square face crops. On vertical phone-shot source, faces are usually taller-than-wide — shift the ellipse center upward by ~5–10% of crop height to stay anchored on the mouth. Anamorphic or letterboxed source: crop to the live image area before sync; bbox math doesn't account for black bars.
6. GPU / platform. See Prerequisites above.
7. VRAM. Under ~12 GB, chain stages as separate subprocesses. The orchestrator probes VRAM up front and picks chained vs. serialized from the result.
8. Audio composition. Dialogue-only audio works as-is. Dialogue mixed with music or SFX in the same track degrades Wav2Lip's mel features and MuseTalk's audio encoder — run stem separation first (e.g. Demucs htdemucs, vocals stem) and feed only the vocal stem to the sync model. Keep the full mix for the final remux.

Step 1: Probe and Prep Input

ffprobe -v error -show_entries stream=codec_type,r_frame_rate,nb_frames,sample_rate,channels,duration -of json INPUT.mp4

The most common silent failure is a VFR source: a nominal r_frame_rate doesn't prove constant frame rate — phone recordings and screen captures routinely advertise "30 fps" while dropping packets. Spot-check with ffprobe -select_streams v -show_entries packet=pts_time on a few frames; if the deltas vary, it's VFR. When in doubt, re-encode to CFR:

ffmpeg -i INPUT.mp4 -vf fps=25 -fps_mode cfr -c:v libx264 -crf 16 \
  -c:a pcm_s16le -ar 16000 -ac 1 INPUT_cfr.mov

Key notes about that command:

• fps=25 -fps_mode cfr (older ffmpeg: -vsync cfr) actually forces CFR by duplicating or dropping frames. -r 25 alone does not on VFR input — it just re-tags the container, which is a foot-gun.
• 16 kHz mono is what Wav2Lip / MuseTalk audio encoders expect for model input. Keep the original audio file separately — 16 kHz mono is below typical broadcast/streaming spec (usually 48 kHz stereo); remux the original back at Step 6. For low-bar deliverables (social-media mascot loops, internal demos) the 16 kHz can be acceptable; surface the choice to the user rather than assume.
• .mov accepts pcm_s16le; .mp4 does not reliably. Use .mov or .mkv here, or transcode the working audio to AAC if you must stay in .mp4.
• For 24 / 23.976 / 29.97 delivery, set fps= accordingly and plan a pulldown step at remux.
• HDR / log inputs: if ffprobe reports bt2020, smpte2084, or arib-std-b67, tone-map before sync — libx264 with default flags crushes HDR colorimetry and you'll deliver an SDR clip with a faint green cast. Example: -vf zscale=t=linear,...,zscale=t=bt709:m=bt709:r=tv,format=yuv420p.
• BT.601 ↔ BT.709 matrix mismatch is more common than HDR and easier to miss. SD-era content (480p/576p) is BT.601; HD is BT.709. If the source is SD or labeled BT.601 and you don't convert with -vf zscale=matrix=709:matrixin=601 (or colormatrix filter), the sync model writes its output as BT.709 and reds shift greenish on remux. Check ffprobe's color_space field and force-convert when it disagrees with the working space.
• Alpha channels (mascot/puppet on a comp): extract the alpha matte first, sync the RGB only, then re-mux the original alpha back. Sync models will destroy or ignore alpha if left in.
• Embedded subtitle / caption streams: the CFR re-encode preserves subtitle streams only if you pass -c:s copy. If you later apply -itsoffset at remux (Step 6), embedded subtitle PTS does not shift with -itsoffset on the video — captions desync. Either burn captions in after final remux, or re-time them with ffmpeg -itsoffset on the subtitle input separately.
• Long clips: anything over ~2 minutes should be chunked, especially with MuseTalk (VRAM use grows with batch size; long renders compound). Split at silence boundaries (ffmpeg -af silencedetect), sync each chunk independently, then concat. Overlap each chunk by ~0.5 s and crossfade the overlap on concat to avoid audible/visible seams at chunk boundaries — a known MuseTalk failure mode.

Step 2: Face Detect / Align

Real-face detectors frequently reject stylized faces — their training data shares almost no low-level features with cel-shaded art. Fallback ladder, in order of preference:

1. Bundled detector — try it first; sometimes works on semi-realistic CG.
2. YOLOv8-face (ultralytics) — more forgiving on illustrated faces because YOLO's training data is broader.
3. anime-face-detector (LFD/SSD-based) for anime specifically. Preferred over lbpcascade_animeface.xml (Haar), which has poor recall on modern styles — keep Haar only as a last-resort fallback.
4. Manual bbox — ask the user for a bounding box on frame 1 and propagate with a moving-average smoother over ~5–9 frames. On locked-off shots and slow camera moves, this is often more stable than a flickery detector.
5. No face at all (faceless mascot, abstract character, mouth-only render, plush with a fixed jaw hinge): skip face detection entirely and define the mouth region directly as the ROI. Ask the user to mark the mouth bbox on frame 1; sync models accept a manually-specified mouth crop without a surrounding "face." Wav2Lip's --box flag is the relevant entry point; MuseTalk needs a stub bbox in the per-shot YAML treated as a mouth-only region with bbox_shift=0.

Multi-face frames: detectors return all faces, so pick by who's speaking. Two practical signals: force-aligned phonemes vs. lip motion in the source, or explicit per-shot selection (largest face, leftmost, named track). Two-character dialogue typically wants two passes with two bboxes and two audio tracks, composited at the end.

Speaker change mid-shot (off-screen narrator hands off to an on-screen character, or two characters trade lines without a cut): time-slice the audio per speaker, run sync only over the on-screen speaker's segments, and pass off-screen segments through unmodified. A force-aligner like whisperx gives word-level timestamps that map cleanly to "sync this range, skip that range."

Off-screen / occluded frames (hand over mouth, head turn, prop in the way): skip them, passing the original frame through unchanged. Syncing through an occluder produces ghost mouth shapes drawn over the obstructing object, which looks worse than no sync. Detection methods, in increasing reliability:

• Face-confidence threshold from the detector — fast but unreliable on stylized art, where the detector is already shaky.
• IoU between the current bbox and the moving-average expected bbox; large drops flag occlusion or sudden head turn.
• A separate hand/object segmenter (e.g. SAM 2 with a "hand" prompt) on suspect frames — slow but accurate, suitable for a final pass before delivery.

Step 3: Run the Sync Model

3a. Wav2Lip

python inference.py \
  --checkpoint_path checkpoints/wav2lip.pth \
  --face INPUT_cfr.mov \
  --audio AUDIO.wav \
  --pads 0 20 0 0 \
  --resize_factor 1 \
  --nosmooth \
  --outfile work/wav2lip_raw.mp4

• --pads 0 20 0 0 adds bottom padding so the chin is included. The default tight crop clips the chin on stylized faces, producing a visible jaw-step when the crop is pasted back.
• --nosmooth disables Wav2Lip's frame-averaging smoother. The smoother helps on live-action but ghosts on already-animated source. Rule of thumb: use --nosmooth on animated input, omit on live-action.
• wav2lip_gan.pth is sharper but produces photoreal skin texture under stylized line art, which most users describe as off-putting. Use it only when followed by an enhancer pass.

Caveats: Wav2Lip is biased toward English visemes (training data skew). Non-English audio works but expect weaker alignment on phonemes absent from English (clicks, trills, tonal contour). Singing, screaming, whispered audio is out of distribution for all three models — sync will degrade and no parameter fixes it.

3b. MuseTalk

MuseTalk takes a bbox_shift parameter to position the mouth correctly. Units are pixels in the original frame's coordinate space. Sign convention: positive bbox_shift moves the inferred mouth region downward (toward the chin); negative moves it upward. The default is tuned for real-face proportions, so on stylized characters — particularly large heads with small mouths — it needs per-character tuning, typically in the range −20 to +20 px. Symptom-to-direction map:

• Mouth appears too high (closure happens above the actual lips) → increase bbox_shift (more positive, pushes the inferred region downward to meet the real mouth).
• Mouth appears too low (closure happens on the chin) → decrease bbox_shift (more negative).

Iterate cheaply: render 2 seconds, check, adjust, re-render.

python -m scripts.inference \
  --inference_config work/per_shot.yaml \
  --result_dir work/musetalk \
  --bbox_shift 5

The orchestrator writes per_shot.yaml programmatically — example shape:

# work/per_shot.yaml — generated by orchestrator.py, do not hand-edit
shot_001:
  video_path: work/shot_001_cfr.mov
  audio_path: work/shot_001_vocals.wav
  bbox_shift: 5
shot_002:
  video_path: work/shot_002_cfr.mov
  audio_path: work/shot_002_vocals.wav
  bbox_shift: -8

Editing the bundled configs/inference/test.yaml by hand causes merge conflicts on every repo upgrade and breaks reproducibility — generate the per-shot file fresh each run.

MuseTalk's "real-time" claim is hardware- and resolution-dependent: faster-than-realtime at 256–512 px on an RTX 4090; ~1× on a 3060; well below realtime on Apple MPS, where ONNX-accelerated paths fall back to CPU. Set expectations against the user's hardware, not the README headline.

3c. LivePortrait (and the Double-Hop)

LivePortrait needs a source image (the stylized character) and a driver video (a face performing the motion you want). For audio-only inputs, generate the driver first with an audio-driven talking-head model — SadTalker is the usual choice (it's audio-driven, not TTS — feed it your audio plus a generic portrait), then feed that output as LivePortrait's driver. This is the double-hop.

python inference.py \
  -s assets/source_character.png \
  -d work/driver.mp4 \
  --flag_relative_motion \
  --flag_pasteback \
  -o work/liveportrait_out.mp4

--flag_relative_motion transfers changes in pose rather than absolute pose. This matters because cartoon and human head proportions differ — often dramatically — and absolute-pose transfer pulls the cartoon's features toward the human's resting position.

Identity-leakage caveats — both technical and legal:

• Technical: a SadTalker driver generated from an actor's portrait carries that actor's idiosyncratic mouth shape, asymmetries, and smile-lines into the cartoon. Two ways to dampen: pick a neutral, frontal, symmetric portrait for the driver, or pre-stylize the driver portrait before SadTalker sees it.
• Legal: using a recognizable person's likeness as the driver portrait may carry rights-of-publicity or likeness-rights issues, separate from model licensing. Most acute for celebrities and stock photos with restrictive licenses. Use a portrait you have rights to use as a driving signal, even though the final delivery doesn't show that face.

Step 4: Enhancement Pass

Sync models output a low-resolution mouth crop. Enhancement restores detail. The default approach is: crop to mouth, enhance, paste back. Face priors can introduce small color shifts (warmer skin tones) and minor geometric warping (eye position, jaw width) when applied full-frame; cropping to the mouth contains both.

• For locked-off shots with no background motion: full-frame enhancement is fine and simpler.
• For moving shots or detailed backgrounds: crop-and-paste avoids drift outside the face region.

Enhancer selection (from the decision tree):

Step 5: Composite Mask

The sync model returns a rectangular crop; pasting it raw leaves a visible box. Use the feathered_mouth_mask helper below. Feather and shape:

import cv2, numpy as np

def feathered_mouth_mask(crop_h, crop_w, feather=15):
    mask = np.zeros((crop_h, crop_w), dtype=np.float32)
    cv2.ellipse(
        mask,
        (crop_w // 2, int(crop_h * 0.70)),
        (int(crop_w * 0.45), int(crop_h * 0.28)),
        0, 0, 360, 1.0, -1,
    )
    return cv2.GaussianBlur(mask, (0, 0), feather)

Composite: out = orig * (1 - mask) + synced * mask.

Ellipse parameters per character: anchor to the eye-line and chin position. Rule of thumb — center the ellipse at ~70% of bbox height for human-proportioned faces; raise to ~55–60% for chibi or large-forehead designs (their mouths sit higher relative to the bbox); horizontal radius covers the widest "open mouth" pose without crossing the cheek line. On vertical/portrait-aspect source, shift center upward by ~5–10% of crop height. Render a 1-second test on a vowel-heavy phrase ("amazing apples"), eyeball it, adjust, re-render — faster than guessing the math.

Tracking the mask on moving shots: anchor to the same bbox produced in Step 2 (detector output or manual + smoother) and re-derive crop coordinates per frame from that bbox. The bbox is the single source of truth for mouth-region location; a second tracker on the mask gives you two truths to reconcile. The legitimate exception is when the detector itself is wrong on a specific stretch (manual review flags it) — hand-paint the bbox for those frames rather than introduce a second tracker.

Step 6: Audio-Video Remux and Sync Verification

Put the original audio (native sample rate / channels, not the 16 kHz mono used for model input) back over the synced video:

ffmpeg -i work/composited.mp4 -i ORIGINAL_AUDIO.wav \
  -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -shortest \
  OUTPUT.mp4

Verify on phoneme-rich segments — p, b, m should be fully closed on every occurrence. If off by a constant offset, the model's audio preprocessing dropped or padded samples. Typical offsets are 1–3 frames (40–120 ms at 25 fps); larger usually means an fps mismatch and is worth revisiting Step 1.

-itsoffset sign convention (classic foot-gun): -itsoffset T applied before an input delays that input by T seconds. If the lips fire before the sound, the video needs to be delayed relative to the audio — apply -itsoffset to the video input. Equivalent: advance the audio. Confirm direction on a 2-second test clip before rendering the whole timeline.

# Lips leading audio by ~40 ms — delay video relative to audio:
ffmpeg -itsoffset 0.04 -i work/composited.mp4 -i ORIGINAL_AUDIO.wav \
  -map 0:v -map 1:a -c:v copy -c:a aac OUTPUT.mp4

Caption tracks: if the deliverable has burned or embedded subtitles, -itsoffset on the video does not shift embedded subtitle PTS. Either burn captions after this step, or apply the same -itsoffset to the subtitle input.

Orchestration

When setting up the pipeline or debugging stage handoffs, keep the orchestrator dumb: shell out to each env, pass file paths, check return codes. Two non-obvious requirements: skip-if-exists (so a mid-pipeline failure doesn't redo a 20-minute MuseTalk pass) and argument lists rather than shell strings (paths with spaces work, no shell-injection surface). The orchestrator runs on its own Python (3.10+) outside any per-model env:

# orchestrator.py — runs on its own python (3.10+), never inside a per-model env.
# Uses list[str] and pathlib idioms that the Wav2Lip env (3.8) wouldn't accept.
import subprocess
import sys
from pathlib import Path

IS_WIN = sys.platform.startswith("win")
# Windows envs put python in Scripts\python.exe; POSIX envs use bin/python.
PY_REL = "Scripts/python.exe" if IS_WIN else "bin/python"

ENVS = {
    "wav2lip":      Path.home() / "lipsync/envs/wav2lip"      / PY_REL,
    "musetalk":     Path.home() / "lipsync/envs/musetalk"     / PY_REL,
    "liveportrait": Path.home() / "lipsync/envs/liveportrait" / PY_REL,
    "enhance":      Path.home() / "lipsync/envs/enhance"      / PY_REL,
}

def run(env: str, args: list[str], cwd: Path, out: Path) -> None:
    if out.exists() and out.stat().st_size > 0:
        print(f"[{env}] skip (exists): {out}")
        return
    cmd = [str(ENVS[env]), *args]
    print(f"[{env}] {' '.join(cmd)}")
    subprocess.run(cmd, cwd=cwd, check=True)

The orchestrator must probe available VRAM up front (nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits on CUDA; rocm-smi on ROCm; platform-equivalent elsewhere) and pick chained vs. serialized execution from the result. Under 12 GB free → serialize (one model at a time, releasing VRAM between subprocess calls). Over 12 GB → chaining is safe but still optional.

Pitfalls

1. VFR source masquerading as CFR. A nominal r_frame_rate from ffprobe doesn't prove constant frame rate. Phone recordings and screen captures routinely advertise "30 fps" while dropping packets. Spot-check with ffprobe -select_streams v -show_entries packet=pts_time. When in doubt, re-encode to CFR — re-encoding is cheap compared to debugging "why does the mouth drift after 10 seconds."

2. -r 25 alone does NOT force CFR on VFR input. It just re-tags the container. Use fps=25 -fps_mode cfr (or -vsync cfr on older ffmpeg) to actually force CFR by duplicating or dropping frames.

3. Chin clipping from tight Wav2Lip crop. The default tight crop clips the chin on stylized faces, producing a visible jaw-step on paste-back. Fix: --pads 0 20 0 0.

4. Ghosting on animated source from Wav2Lip smoother. The frame-averaging smoother helps on live-action but ghosts on already-animated source. Fix: --nosmooth on animated input.

5. Wav2Lip 96×96 resolution ceiling. This is a property of the architecture, not a parameter to tune. Enhancement can sharpen the output but cannot add detail that was never generated.

6. Loading all three models in one Python process. The most common OOM pattern. Fix: chain stages as separate subprocesses so VRAM is released between them, especially under ~12 GB VRAM.

7. Editing bundled configs/inference/test.yaml by hand for MuseTalk. Causes merge conflicts on every repo upgrade and breaks reproducibility. Fix: generate the per-shot YAML fresh each run via the orchestrator.

8. MuseTalk bbox_shift wrong direction. Positive moves the inferred mouth region downward (toward chin); negative moves it upward. If the mouth appears too high, increase bbox_shift. If too low, decrease it.

9. Visible rectangular mask seam on paste-back. The sync model returns a rectangular crop. Fix: apply a feathered elliptical mask (see Step 5).

10. Audio drift from 16 kHz mono remux. 16 kHz mono is below broadcast/streaming spec. Always keep the original audio and remux it back at Step 6.

11. -itsoffset sign confusion. -itsoffset T applied before an input delays that input by T seconds. If lips fire before sound, delay the video (apply -itsoffset to the video input). Confirm on a 2-second test clip before rendering the whole timeline.

12. Embedded subtitle desync after -itsoffset. -itsoffset on the video does not shift embedded subtitle PTS. Either burn captions after the remux step, or apply the same -itsoffset to the subtitle input separately.

13. HDR color crush. If ffprobe reports bt2020, smpte2084, or arib-std-b67, tone-map before sync — libx264 with default flags crushes HDR colorimetry, delivering an SDR clip with a faint green cast.

14. BT.601 ↔ BT.709 matrix mismatch. SD-era content (480p/576p) is BT.601; HD is BT.709. If the source is BT.601 and you don't convert, reds shift greenish on remux. Check ffprobe's color_space field and force-convert with -vf zscale=matrix=709:matrixin=601.

15. Alpha channel destruction. Sync models will destroy or ignore alpha if left in. Extract the alpha matte first, sync the RGB only, then re-mux the original alpha back.

16. MuseTalk VRAM growth on long clips. VRAM use grows with batch size; long renders compound. Chunk anything over ~2 minutes at silence boundaries, sync each chunk independently, then concat with ~0.5 s overlap crossfaded.

17. Ghost mouth shapes over occluders. Syncing through an occluder (hand over mouth, head turn, prop) produces ghost mouth shapes drawn over the obstructing object. Skip occluded frames, passing the original through unchanged.

18. Identity leakage in double-hop. A SadTalker driver generated from an actor's portrait carries that actor's mouth shape, asymmetries, and smile-lines into the cartoon. Dampen with a neutral, frontal, symmetric driver portrait, or pre-stylize the driver before SadTalker sees it.

19. Music/SFX degrading mel features. Dialogue mixed with music or SFX in the same track degrades Wav2Lip's mel features and MuseTalk's audio encoder. Run stem separation first (e.g. Demucs htdemucs, vocals stem) and feed only the vocal stem to the sync model.

20. Singing/screaming/whispered audio. Out of distribution for all three models — sync will degrade and no parameter fixes it.

21. MuseTalk "real-time" expectations. Faster-than-realtime at 256–512 px on an RTX 4090; ~1× on a 3060; well below realtime on Apple MPS. Set expectations against the user's hardware, not the README headline.

22. Checkpoint corruption from mirror outages. A silently corrupted MuseTalk weight produces plausible-looking sync that's subtly wrong. Pin checkpoints by SHA-256 in models/manifest.toml and verify on download. The orchestrator refuses to run with mismatched hashes.

23. Caching checkpoints inside per-env site-packages. Re-installing an env should not re-download a 2 GB weight. Cache in models/ and symlink into each env.

24. torch/onnxruntime pin conflicts in shared env. Wav2Lip wants torch 1.13; MuseTalk wants 2.0+; LivePortrait wants 2.3+. One env per model is the default. A single env with careful pinning is brittle across upgrades.

Verification

Input conformance

# Confirm CFR after re-encode — deltas should be uniform
ffprobe -select_streams v -show_entries packet=pts_time -of csv INPUT_cfr.mov | head -20

All pts_time deltas should be identical (0.04s at 25 fps). If they vary, the source is still VFR — re-encode again.

Audio alignment

# Check audio stream properties of the final output
ffprobe -v error -show_entries stream=codec_type,sample_rate,channels,duration -of json OUTPUT.mp4

Confirm the audio sample rate matches the original (e.g. 48000 Hz, stereo), not the 16 kHz mono used for model input.

Lip sync spot-check

Verify on phoneme-rich segments — p, b, m should be fully closed on every occurrence. If off by a constant offset, the model's audio preprocessing dropped or padded samples. Typical offsets are 1–3 frames (40–120 ms at 25 fps); larger usually means an fps mismatch — revisit Step 1.

VRAM probe

# CUDA
nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits

# ROCm
rocm-smi --showmeminfo vram

Confirm free VRAM before launching the orchestrator. Under 12 GB → expect serialized execution.

Checkpoint integrity

# Verify SHA-256 against models/manifest.toml
sha256sum models/wav2lip.pth
sha256sum models/musetalk/*.pth

The orchestrator refuses to run with mismatched hashes. If a hash fails, re-download from a fallback mirror.

Mask seam check

Render a 1-second test on a vowel-heavy phrase ("amazing apples") and eyeball the composite boundary. If a rectangular seam is visible, increase the feather radius in feathered_mouth_mask or adjust the ellipse parameters.

Licensing

These checkpoints come with non-trivial license terms. A local pipeline is not automatically a licensed pipeline. Surface this section before any commercial delivery discussion:

• Wav2Lip: pretrained weights are research-only in the original release; commercial use of the released checkpoints is not permitted under that license.
• GFPGAN, CodeFormer: each ship with their own license; review the current terms at the version pulled.
• MuseTalk, LivePortrait: check the current repo license at the commit hash pulled, not the historical one.

License terms above verified against upstream repos as of 2026-06. Re-check at the version actually pulled, since these change. If the deliverable is commercial, surface the licensing question explicitly and let the user decide rather than assume "it ran on my machine" implies "I can ship it."

Defaults (single canonical table)

Reasoning is consolidated here so the inline pipeline stays terse. Deviate when the reason in the right column doesn't apply.