Media Lipsync

Media Lipsync (open-source talking-head + lip-sync)

Context: $ARGUMENTS

Quick start

• Animate a photo from audio (head pose + expressions + mouth): scripts/lipsync.py animate --model liveportrait --source face.jpg --driving speech.wav --out talking.mp4 → Step 3.
• Animate a photo from a driving video (retarget expressions): scripts/lipsync.py animate --model liveportrait --source face.jpg --driving actor.mp4 --out retarget.mp4 → Step 3.
• Re-sync an existing video's lips to new audio: scripts/lipsync.py sync --model latentsync --video original.mp4 --audio new.wav --out resynced.mp4 → Step 4.
• Dub a foreign-language clip with lip correction: scripts/lipsync.py dub --video en.mp4 --new-audio es.wav --out es_dubbed.mp4 → Step 5.
• Install a model: scripts/lipsync.py install liveportrait → Step 2.

When to use

• Drive a still portrait photo with an audio track to produce a talking-head video (full facial motion, not just the mouth).
• Re-sync the lips of an existing video to a new audio track (dubbing, ADR, language swap) without regenerating the whole face.
• Need commercial-safe, open-source weights. Do NOT use Wav2Lip, SadTalker, HeyGen, D-ID, Synthesia, Runway Act-One — all excluded per references/LICENSES.md.

Which model does what (read this first)

Rule of thumb:

• Start from a still photo -> LivePortrait. It invents head motion + expressions.
• Start from an existing video -> LatentSync. It preserves everything except the lips, which are re-synced to the new audio.
• For dubbing a real actor on existing footage, LatentSync is the correct pick (keeps the actor's real face / performance, just fixes the lip-audio sync).
• LivePortrait has a --driving flag that accepts a driving video of ANOTHER face — it retargets that face's expressions onto your still. This is the "deepfake expression transfer" mode; use ethically.

Step 1 — Install

Neither model is on PyPI as a drop-in wheel; both are installed by cloning the reference repo and running their own inference.py. This skill drives those scripts via subprocess.

uv run .claude/skills/media-lipsync/scripts/lipsync.py install liveportrait
# prints:
#   git clone https://github.com/KwaiVGI/LivePortrait ~/LivePortrait
#   cd ~/LivePortrait && pip install -r requirements.txt
#   (follow README.md section "Download pretrained weights")

uv run .claude/skills/media-lipsync/scripts/lipsync.py install latentsync
# prints:
#   git clone https://github.com/bytedance/LatentSync ~/LatentSync
#   cd ~/LatentSync && pip install -r requirements.txt
#   huggingface-cli download ByteDance/LatentSync --local-dir checkpoints

Check what's present:

uv run .claude/skills/media-lipsync/scripts/lipsync.py check

Set LIVEPORTRAIT_ROOT / LATENTSYNC_ROOT env vars if you installed outside ~/LivePortrait / ~/LatentSync. The script finds them automatically if they are in the standard location.

Step 2 — Prep inputs

All models want a clean face in frame. Preprocess for best results:

• Source image (LivePortrait): square, face centered, >=512x512, neutral expression, good lighting. If the face is small in the frame, pre-crop to the face.
• Driving video (LivePortrait expression transfer): single face in frame throughout, minimal head turns past ~30°.
• Source video (LatentSync): face visible the whole clip, no heavy occlusion on the mouth (hand-in-front, microphone over lips => artifacts).
• Audio: 16 kHz mono WAV is safest. Both models internally resample but MP3/M4A sometimes decode differently across backends.

ffmpeg -i driver_raw.wav -ar 16000 -ac 1 -c:a pcm_s16le driver.wav
ffmpeg -i source_raw.mp4 -vf "scale=-2:512" -r 25 source.mp4   # LatentSync prefers 25 fps

Step 3 — LivePortrait: animate a still

From audio

uv run .../lipsync.py animate \
    --model liveportrait \
    --source face.jpg \
    --driving speech.wav \
    --out talking.mp4 \
    --fps 25

LivePortrait's audio-driven mode uses an inferred pose/expression sequence from the audio. Movement is subtle — plausible talking-head, not a wild performance.

From a driving video (expression retargeting)

uv run .../lipsync.py animate \
    --model liveportrait \
    --source face.jpg \
    --driving actor.mp4 \
    --out retargeted.mp4

The output preserves the SOURCE's identity, with the DRIVING video's head pose + expression. Ethical note: do not use for impersonation without consent.

Step 4 — LatentSync: re-sync an existing video

uv run .../lipsync.py sync \
    --model latentsync \
    --video original.mp4 \
    --audio new_voice.wav \
    --out resynced.mp4 \
    --steps 25 --guidance 1.5

LatentSync runs 25-step latent diffusion per lip region. On an H100 it's ~10 seconds of processing per 1 second of video; on a 4090 ~30 seconds per second. CPU-only is not practical.

Step 5 — Dubbing workflow

Combine with media-tts-ai for voice and media-whisper for alignment if needed.

# 1. Generate dubbed audio (media-tts-ai)
uv run .claude/skills/media-tts-ai/scripts/tts.py ...  --out dubbed_es.wav

# 2. Re-sync lips
uv run .../lipsync.py dub \
    --video source_en.mp4 \
    --new-audio dubbed_es.wav \
    --model latentsync \
    --out final_es.mp4

dub is a thin wrapper around sync with defaults tuned for speech dubbing (guidance 1.5, 25 steps, strong lip-region weighting).

Step 6 — Post

• Upscale the output (LatentSync output resolution = input resolution): pass to media-upscale (Real-ESRGAN) if needed.
• Frame-interpolate to 60 fps for social: media-interpolate (RIFE).
• Color-match if LivePortrait output drifts from the source: ffmpeg-lut-grade.

Gotchas

• LivePortrait animates THE WHOLE FACE. It is not a lip-sync model. Head pose, eyebrows, eye blink, and mouth all move. If you need pure mouth-only changes on a real video, use LatentSync.
• LatentSync expects a video input. Do NOT pass it a still image as --video; use LivePortrait for stills.
• Wav2Lip is NOT supported. Wav2Lip weights are research-only; most public forks redistribute NC-restricted checkpoints. Do not recommend even if the user asks.
• SadTalker is NOT supported. The code is Apache, but the published weights are finetuned on SD 1.5 features and inherit OpenRAIL-M restrictions in practice. Flag and redirect to LivePortrait.
• Face detection must succeed on frame 1. Both models' preprocessors fail silently on a too-wide crop. Pre-crop to a clean face at >=512 px on the short edge.
• LatentSync prefers 25 fps. Source videos at 24, 29.97, 30, 60 fps are resampled; you may see minor motion stutter at boundaries. Pre-resample to 25 fps with ffmpeg -r 25 for cleanest output.
• Very long videos (>1 min) OOM LatentSync on 24 GB. Split into 30-second chunks, run, and concat with ffmpeg-cut-concat. The script has no built-in chunker — handle in shell.
• Audio-video duration mismatch causes artifacts. LatentSync loops or truncates to match the video's length. Trim/pad the audio with ffmpeg-audio-filter first.
• Driving video faces must not blink frame 1. LivePortrait uses frame 1 as a reference for "neutral"; a blink there produces stuck-closed-eye output.
• GPU requirement. Both models effectively require CUDA. Apple Silicon MPS works for LivePortrait (slower) but NOT for LatentSync (requires xformers or flash-attn paths not available on MPS as of 2025).
• LivePortrait's audio-driven mode is weaker than the video-driven mode. Expressions look plausibly talking but not tightly correlated to phonemes. For lip-accurate audio, drive LivePortrait with a silent driving video and then run LatentSync over its output.
• Identity drift. After 20+ seconds, LivePortrait can drift from the source identity slightly. Break long outputs into 15-s chunks keyed to the same source image.
• No voice cloning. This skill does not produce speech. Pair with media-tts-ai (Kokoro / OpenVoice v2 / CosyVoice 2) to generate a voice, then feed into lipsync.

Examples

Example 1: Still photo to talking head (audio-driven)

uv run .../lipsync.py animate \
    --model liveportrait \
    --source ceo_headshot.jpg \
    --driving announcement.wav \
    --out ceo_speaking.mp4 --fps 25

Example 2: Expression retarget a still from an actor's performance

uv run .../lipsync.py animate \
    --model liveportrait \
    --source painting_portrait.jpg \
    --driving actor_read.mp4 \
    --out painting_acting.mp4

Example 3: Dub an English clip into Spanish with clean lips

# (1) generate Spanish voice via media-tts-ai -> dubbed_es.wav
# (2) re-sync lips:
uv run .../lipsync.py dub \
    --video interview_en.mp4 \
    --new-audio dubbed_es.wav \
    --out interview_es.mp4

Example 4: Check env before running

uv run .../lipsync.py check
# reports: cuda ok, LivePortrait repo at ~/LivePortrait, LatentSync at ~/LatentSync

Example 5: Dry-run to see the exact inference.py command

uv run .../lipsync.py animate --model liveportrait \
    --source face.jpg --driving speech.wav --out out.mp4 \
    --dry-run --verbose
# prints the exact `python inference.py ...` command without running it.

Troubleshooting

Error: LivePortrait repo not found

Cause: repo isn't at ~/LivePortrait and LIVEPORTRAIT_ROOT isn't set. Solution: git clone https://github.com/KwaiVGI/LivePortrait ~/LivePortrait, OR export LIVEPORTRAIT_ROOT=/path/to/LivePortrait.

Error: No module named 'torch' inside inference.py

Cause: the LivePortrait / LatentSync venv doesn't have the deps installed. Solution: cd ~/LivePortrait && pip install -r requirements.txt. The driver does not auto-install.

Output has eyes closed / blinking forever

Cause: LivePortrait reference frame had closed eyes. Solution: pick a source image with eyes open, neutral expression. Do NOT laugh / grimace.

LatentSync output is blurry around the mouth

Cause: source video resolution too low, or heavy JPEG compression on the video. Solution: source at >=512 px short edge, re-encode lossless first (ffmpeg -c:v libx264 -crf 12). Consider upscaling source with media-upscale before LatentSync.

Lip sync drifts after 20 seconds

Cause: audio and video clocks drift (VFR source, or audio sample rate mismatch). Solution: force CFR + 16 kHz audio upfront. ffmpeg -r 25 -i in.mp4 -c:v libx264 -c:a copy fixed.mp4 && ffmpeg -i new.aif -ar 16000 -ac 1 -c:a pcm_s16le new.wav.

CUDA out of memory on a long video (LatentSync)

Cause: model holds whole video in VRAM. Solution: split the video into 30-s chunks (ffmpeg-cut-concat segment mode), run LatentSync per chunk, concat the outputs. No built-in chunker in the driver.

"module xformers has no attribute ..." on Apple Silicon (LatentSync)

Cause: xformers / flash-attn have no MPS path. Solution: run LatentSync on a CUDA machine. On Apple Silicon, only LivePortrait works (audio-driven mode is usable; video-driven mode runs but slow).