Media Pipeline Triage

Media Pipeline Triage

Media pipeline failures hide behind vague symptoms like "it's out of sync" or "the lip-sync is off." Triage localizes which stage introduced the drift — audio extraction, clip generation, lip-sync retargeting, subtitle timing, or final compilation — because each points to a completely different fix.

When to Use

Use when compiled music videos or shorts have audio-video misalignment, when lip-sync looks wrong, when subtitles appear at the wrong time, when FFmpeg produces silent output, or when AI-generated clip speech conflicts with the target song audio. This is the media-specific complement to network-and-api-failure-triage and extends the investigation methods of hypothesis-driven-investigation into the AV domain.

Trigger keywords: out of sync, lip-sync, mouth not moving, silent audio, subtitle drift, VFR, ffmpeg merge, veo3 speech, pipeline no change, no visible change, audio drift, face not detected.

Prerequisites

• FFmpeg and FFprobe installed and on PATH (Windows PowerShell: ffmpeg -version must succeed).
• Python 3.8+ with numpy, scipy, opencv-python if using automated offset detection or coordinate diagnostics.
• SyncNet (pip install syncnet-python) for quantitative lip-sync scoring.
• InsightFace with buffalo_l model and det_10g.onnx detector for face detection on stylized content.
• Whisper (or equivalent transcription tool) for subtitle timing validation.
• Access to all intermediate pipeline artifacts (raw clips, Wav2Lip outputs, LivePortrait outputs, audio slices). If intermediates are missing, the pipeline must be re-run with intermediate output enabled.

Procedure

Core Principle

Never trust intermediate claims — visually verify every stage's output. A log line that says "lip-sync complete, 133/133 frames processed" is not verification. An extracted frame showing open mouths during speech is verification.

Phase 1: Identify the Failure Mode

Match the symptom to one of the seven canonical failure modes below before diving into debugging.

Failure Mode 1: Veo 3 Speech-vs-Song Conflict (Visual Mouth ≠ Audio Words)

Symptom: Character's mouth visibly says one thing while the audio track plays different words.

Root Cause: Veo 3 generates its own speech in clips. When the original song is overlaid, the visual speech doesn't match.

Diagnosis:

1. Extract audio from the raw clip (before song overlay):

   ffmpeg -y -i shot_01.mp4 -vn -acodec pcm_s16le raw_clip_audio.wav
   

2. Transcribe the extracted audio with Whisper.
3. Compare the transcription against the target song lyrics at 3+ timestamps.

Fix: Regenerate clips using speech-suppressed prompts, or run the lip-sync re-sync pipeline to overwrite the visual mouth shapes with shapes driven by the target song audio.

Failure Mode 2: Silent Audio After FFmpeg Merge

Symptom: Final video plays but has no sound, or plays the wrong (silent AI-generated) audio track.

Root Cause: FFmpeg defaults to the first audio stream (often the silent AI-generated track from input 0) instead of the provided music file.

Diagnosis:

ffprobe -v error -select_streams a -show_entries stream=codec_name,channels,bit_rate -of default=noprint_wrappers=1 output.mp4

If bit_rate=N/A or channels=0, the audio stream is empty.

Fix: Add explicit stream mapping to force video from input 0 and audio from the music file input:

ffmpeg -y -i video_input.mp4 -i music.wav -map 0:v:0 -map 1:a:0 -c:v copy -c:a aac -shortest output.mp4

Failure Mode 3: Progressive Audio Drift (VFR Clips)

Symptom: Audio and video start aligned but gradually drift apart over the duration.

Root Cause: AI-generated clips often have variable frame rate (VFR). Concatenating VFR clips causes cumulative timing errors.

Diagnosis — use the vfrdet filter (ffprobe metadata is unreliable; it reports nominal rate, not actual):

ffmpeg -i input.mp4 -vf vfrdet -an -f null -

Parse stderr for VFR:X.XXX — any value > 0 indicates variable frame rate.

Alternatively check frame rates:

ffprobe -v error -select_streams v -show_entries stream=r_frame_rate,avg_frame_rate -of default=noprint_wrappers=1 clip.mp4

If r_frame_rate ≠ avg_frame_rate, it's VFR.

Fix: Convert all clips to CFR before concatenation:

ffmpeg -y -i input_vfr.mp4 -vsync cfr -r 30 output_cfr.mp4

Failure Mode 4: Lip-Sync Face Detection Failure on Cartoons

Symptom: Lip-sync pipeline produces unchanged output or crashes with "no face detected."

Root Cause: Face detection models (trained on photorealistic faces) fail on heavily stylized Pixar/anime characters.

Diagnosis: Run the face detector in isolation and check if bounding boxes are returned. Print all detected faces and their bounding boxes:

import cv2
# Load frame from the intermediate video at the resolution it's processed at
frame = cv2.imread("frame.png")
# ... run face detector ...
for face in detected_faces:
    x1, y1, x2, y2 = face['bbox']
    x_center = (x1 + x2) / 2.0
    y_center = (y1 + y2) / 2.0
    width = x2 - x1
    height = y2 - y1
    print(f"Face at center=({x_center:.0f}, {y_center:.0f}), size=({width:.0f}x{height:.0f})")
    cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 2)
cv2.imwrite("debug_faces.png", frame)

Fix: Use InsightFace with the buffalo_l model and det_10g.onnx detector. For extreme stylization, use the LivePortrait two-stage pipeline which has better cartoon face handling.

Failure Mode 5: Lip-Sync Pipeline Produces No Visible Change (Silent Failure)

Symptom: Lip-sync pipeline runs to completion without errors, but the output video shows identical closed-mouth frames as the input. No crash, no warning — just zero lip movement.

Root Cause: One or more stages in a multi-stage pipeline silently fail, producing pass-through output. Common culprits:

• A function signature mismatch (method doesn't accept **kwargs but caller passes keyword args → TypeError or NameError silently swallowed).
• Face detection bounding box coordinates are valid at one resolution but wrong after crop/scale transforms between pipeline stages.
• The ML model (e.g., Wav2Lip) cannot generate meaningful output for the input domain (cartoon/3D vs photorealistic faces) but doesn't raise an error — it just outputs the original face unchanged.

Diagnosis: Follow the Multi-Stage Pipeline Debugging Protocol below.

Fix: Depends on which stage fails. If the ML model can't handle the input domain, switch to an alternative architecture (e.g., direct LivePortrait retargeting with a human driving video instead of Wav2Lip → LivePortrait chain).

Failure Mode 6: Subtitle Timing Drift

Symptom: Subtitles appear too early or too late relative to the vocals.

Root Cause: Subtitle timestamps were guessed or derived from storyboard estimates instead of actual audio transcription.

Diagnosis: Compare SRT timestamps against Whisper transcription timestamps of the audio file.

Fix: Re-generate subtitles from the actual audio using the auto_subtitle_generator.py script.

Failure Mode 7: Subtitle Oversized / Covering Character

Symptom: Text dominates the frame, covering the character's face or occupying >15% of screen height.

Root Cause: FontSize set too high (above 26 for libass, above 36 for MoviePy TextClip).

Diagnosis:

ffmpeg -y -i video.mp4 -ss 00:00:15 -vframes 1 check.png

Visually inspect the extracted frame.

Fix: Reduce FontSize to 20–24 (libass) or 28–36 (MoviePy). Limit subtitle lines to 3–5 words per line.

Phase 2: Multi-Stage Pipeline Debugging Protocol

Use this systematic method when the failure mode is "no visible change" or when you need to localize which stage in a multi-stage pipeline (e.g., audio extraction → Wav2Lip → LivePortrait → FFmpeg compile) introduced the defect.

Step 1: Verify the Final Output First (Reverse Trace)

Start from the end product — the compiled video — and confirm the symptom visually:

ffmpeg -y -i final_output.mp4 -ss 00:00:02 -vframes 1 -update 1 verify_2s.png
ffmpeg -y -i final_output.mp4 -ss 00:00:12 -vframes 1 -update 1 verify_12s.png

View the extracted frames. Do NOT skip this step. Do NOT rely on logs or metadata claiming "lip-sync successful." The frame is the ground truth.

If the frame shows closed mouths during speech, the lip-sync failed. If mouths are open but wrong words, it's Failure Mode 1 (Veo 3 Speech Conflict). If the frame looks correct, the compilation step is the likely culprit.

Step 2: Trace Backwards Through Every Intermediate Artifact

For each pipeline stage, locate and visually inspect its output:

Stage N (final compile)  →  final_output.mp4      → extract frame, view it
Stage N-1 (lip-sync)     →  synced_01.mp4          → extract frame, view it
Stage N-2 (driving gen)  →  wav2lip_output.mp4     → extract frame, view it
Stage N-3 (audio slice)  →  vocals_01.wav          → check duration, loudness
Stage 0 (raw clip)       →  shot_01.mp4            → extract frame, view it (baseline)

The stage where the defect first appears is the culprit. If the raw clip has closed mouths (expected) and the Wav2Lip output ALSO has closed mouths, Wav2Lip is the broken stage. If Wav2Lip output has open mouths but the LivePortrait output has closed mouths, LivePortrait is the broken stage.

For each intermediate file:

# Extract a frame
ffmpeg -y -i intermediate.mp4 -ss 00:00:02 -vframes 1 -update 1 stage_check.png
# Check duration matches expectations
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 intermediate.mp4
# Check resolution
ffprobe -v error -select_streams v -show_entries stream=width,height -of csv=p=0 intermediate.mp4

Step 3: Read Code Signatures and Trace the Call Graph

Once you've localized the failing stage, read the actual source code — do not guess about how functions are called:

1. Read the entry point (the script that invokes the pipeline).
2. Read every function signature that handles the failing stage — check whether the method accepts **kwargs, what parameters it actually uses, and how they are passed.
3. Trace keyword arguments end-to-end: if the caller passes source_path=args.source, verify that every function in the chain actually receives and uses this parameter.
4. Look for kwargs in non-kwargs methods — a function that references kwargs.get(...) but has no **kwargs in its signature will raise a NameError at runtime.

Common code-level failures in media pipelines:

• Method signature doesn't accept **kwargs but caller passes keyword args.
• Scene-specific coordinate ranges hardcoded for one resolution but applied at a different resolution after crop/scale.
• Face detection bounding box filter ranges (x_min, x_max, y_max_thresh) tuned for widescreen but applied to vertical video.
• Exception swallowed in a try/except block, causing the pipeline to silently fall back to the unmodified input.

Step 4: Isolate Each Stage Independently

Run each pipeline stage in isolation to confirm it works or fails on its own:

# 1. Extract the audio segment
ffmpeg -y -i vocals.wav -ss 0.07 -t 5.57 vocals_scene1.wav

# 2. Run Wav2Lip directly (NOT through the wrapper pipeline)
python Wav2Lip\inference.py --checkpoint_path Wav2Lip\checkpoints\wav2lip_gan.pth --face shot_01.mp4 --audio vocals_scene1.wav --outfile test_wav2lip_scene1.mp4

# 3. Extract frames and visually compare
ffmpeg -y -i test_wav2lip_scene1.mp4 -ss 00:00:02 -vframes 1 -update 1 wav2lip_check.png
ffmpeg -y -i shot_01.mp4 -ss 00:00:02 -vframes 1 -update 1 raw_check.png

View both frames side-by-side. If the Wav2Lip frame looks identical to the raw frame, Wav2Lip is not generating lip shapes for this face type.

Step 5: Check Coordinate Systems Across Resolution Transforms

Multi-stage pipelines often transform video resolution between stages. Face detection coordinates that work at one resolution may be completely wrong at another. Use the diagnostic Python snippet from Failure Mode 4 to print all detected faces and draw bounding boxes.

Common coordinate failures:

• Widescreen (1920×1080) coordinates applied to vertical (720×1280) after crop.
• Face center ranges [300, 480] tuned for a specific character position but the character is at x=643 after re-composition.
• Multiple faces detected; the filter selects the wrong one (e.g., steering wheel scored higher than the actual face because the size threshold was too permissive).

Step 6: Verify Audio Input Quality

A lip-sync pipeline that receives silence or near-silence will correctly produce no lip movement:

# Check duration
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 vocals_scene1.wav
# Check loudness
ffmpeg -i vocals_scene1.wav -af "volumedetect" -f null - 2>&1 | Select-String "mean_volume"

If mean_volume < -40 dB, the segment is effectively silent.

Step 7: Differential Test — Swap One Component

If still stuck after Steps 1–6, perform a differential test:

• Replace the cartoon face with a photorealistic human face and run the same pipeline → if lips move, the model can't handle cartoon faces.
• Replace the audio with a known-good speech clip → if lips move, the audio input was the problem.
• Replace the driving video with a pre-made driving video with visible lip motion → if the source video still shows closed mouths, the motion transfer stage is broken.

Only change one variable at a time. Each test should confirm or eliminate exactly one hypothesis.

Phase 3: Automated Quality Verification

Apply these quantitative checks after applying a fix to confirm the defect is resolved.

SyncNet Lip-Sync Scoring

Quantify lip-sync quality without ground-truth video:

• LSE-D (Lip Sync Error - Distance): Lower = better. Pass threshold: < 7.0
• LSE-C (Lip Sync Error - Confidence): Higher = better. Pass threshold: > 5.0
• Install: pip install syncnet-python
• Run on 10–100 frame sliding windows for stability.

VMAF Video Quality

Measure visual quality regression after re-encoding:

ffmpeg -i distorted.mp4 -i reference.mp4 -lavfi libvmaf -f null -

Flag segments where VMAF drops below 80. Use VMAF-CUDA for GPU-accelerated monitoring.

VFR Detection

The most reliable VFR detection method:

ffmpeg -i input.mp4 -vf vfrdet -an -f null -

Parse stderr for VFR:X.XXX — any value > 0 indicates variable frame rate. ffprobe metadata is unreliable (reports nominal rate, not actual).

Cross-Correlation Offset Detection

Programmatically detect audio-video offset:

import numpy as np
from scipy.signal import correlate

# Extract audio from both tracks, compute cross-correlation
correlation = correlate(audio_track_1, audio_track_2, mode='full')
lag = np.argmax(correlation) - len(audio_track_1) + 1
offset_ms = lag / sample_rate * 1000
print(f"Detected offset: {offset_ms:.1f}ms")
# ITU-R BT.1359: ±45ms acceptable, ±80ms annoying

Subtitle Timing Validation

• Auto-align with ffsubsync or alass.
• Validate: CPS (characters per second) ≤ 25, duration ≥ 0.7s, no overlapping cues.
• Match against platform specs (Netflix, YouTube).

Pitfalls

• "Verified" without transcription — claiming the video is correct without transcribing and comparing spoken words against the audio track.
• Retrying the same pipeline — re-running an identical command expecting different results instead of diagnosing the root cause.
• Trusting AI-claimed sync — accepting lip-sync or subtitle alignment based on metadata alone without visual and auditory human verification.
• Ignoring VFR — concatenating variable-frame-rate clips and wondering why audio drifts. Always use vfrdet filter, not ffprobe nominal rate.
• Oversized text defense — "the text looks fine to me" without extracting a frame and measuring screen percentage.
• Trusting logs over frames — reading a log line that says "lip-sync complete, 133/133 frames processed" and claiming success without extracting a single frame from the output and visually confirming mouths are open.
• Skipping intermediate outputs — jumping from "the pipeline ran" to "the final video is wrong" without checking which stage's output first shows the defect.
• Assuming code matches docs — the walkthrough says the face coordinates were fixed, but the actual code may have a different bug (e.g., kwargs referenced in a non-kwargs method) that prevents the fix from ever executing.
• Resolution-blind coordinate hardcoding — using face detection coordinates tuned at one resolution across pipeline stages that operate at different resolutions.
• Silent fallback acceptance — not noticing that a pipeline stage caught an exception and silently returned the unmodified input instead of the processed result.
• FFmpeg default stream selection — relying on FFmpeg's automatic stream selection instead of explicit -map flags when inputs have multiple audio streams.
• ffprobe VFR false negatives — ffprobe reports the nominal frame rate, not the actual frame rate. Use the vfrdet filter for reliable VFR detection.

Verification

Diagnostic Checklist

Confirm each item before declaring the pipeline fixed:

• Failure stage localized (clip generation / lip-sync / subtitle timing / FFmpeg compilation).
• Raw clips inspected individually (before assembly) to isolate the faulty clip.
• Audio stream verified via FFprobe (codec, channels, bitrate all non-zero).
• Stream mapping confirmed in FFmpeg command (-map 0:v:0 -map 1:a:0).
• VFR checked using vfrdet filter and converted to CFR if detected.
• Spoken words transcribed and compared against target lyrics at 3+ timestamps.
• Subtitle timestamps compared against Whisper transcription of actual audio.
• Frame extracted and visually inspected for subtitle sizing and lip movement.
• Multi-stage pipeline: frames extracted and visually inspected at EVERY intermediate stage, not just the final output.
• Multi-stage pipeline: function signatures read and kwargs traced end-to-end through the call chain.
• Multi-stage pipeline: each stage tested in isolation to confirm it can produce the expected output independently.
• Multi-stage pipeline: coordinate systems verified across resolution transforms (widescreen ↔ vertical, crop offsets, scale factors).
• Multi-stage pipeline: audio segment loudness checked (mean_volume > -40 dB) to confirm non-silent input.

Automated Quality Checklist

• VFR detected and converted to CFR using vfrdet filter.
• Audio loudness normalized to -16 LUFS (streaming) or -23 LUFS (broadcast).
• SyncNet LSE-D < 7.0 and LSE-C > 5.0 on lip-synced content.
• VMAF > 80 after any re-encode step.
• Audio offset < ±45ms (ITU-R BT.1359 guideline).
• Subtitle CPS ≤ 25 with no overlapping cues.
• Stream mapping confirmed (-map 0:v:0 -map 1:a:0).
• Frame extracted and visually inspected for layout.

Quick Verification Commands

# Verify audio stream is non-empty in final output
ffprobe -v error -select_streams a -show_entries stream=codec_name,channels,bit_rate -of default=noprint_wrappers=1 output.mp4

# Verify no VFR in final output
ffmpeg -i output.mp4 -vf vfrdet -an -f null - 2>&1 | Select-String "VFR"

# Extract verification frames
ffmpeg -y -i output.mp4 -ss 00:00:02 -vframes 1 -update 1 final_check_2s.png
ffmpeg -y -i output.mp4 -ss 00:00:12 -vframes 1 -update 1 final_check_12s.png

# Check audio loudness
ffmpeg -i output.mp4 -af "volumedetect" -f null - 2>&1 | Select-String "mean_volume"

Related Skills

• hypothesis-driven-investigation — Localize the clip source using structured hypothesis testing.
• network-and-api-failure-triage — Check the API calls (Veo 3, model inference endpoints) when pipeline stages fail to fetch or generate.
• video-maker — Verify the fix with the Mandatory Media Verification Protocol (Section 10).
• audio-reactive-music-video — Reference for auto_subtitle_generator.py and subtitle generation workflow (Section 7B).
• lip-sync-agent — Reference for lip-sync re-sync pipeline (Section 4).
• google-flow-agent — Reference for Veo 3 speech-suppressed prompts (Section 5).