Localizes audio/video pipeline failures—VFR drift, silent FFmpeg merges, lip-sync no-ops, subtitle lag, Veo speech-vs-song—by checking frames and streams per stage. Use when compiled shorts look out of sync or a lip-sync log claims success with closed mouths. Not for Remotion composition authoring or generic application crash hunts.
Instrucciones de origen · Vista previa de solo lectura
name
media-pipeline-triage
description
Localizes audio/video pipeline failures—VFR drift, silent FFmpeg merges, lip-sync no-ops, subtitle lag, Veo speech-vs-song—by checking frames and streams per stage. Use when compiled shorts look out of sync or a lip-sync log claims success with closed mouths. Not for Remotion composition authoring or generic application crash hunts.
Media pipeline failures hide behind vague symptoms like "it's out of sync" or "the lip-sync is off." Triage localizes which stage introduced the drift — audio extraction, clip generation, lip-sync retargeting, subtitle timing, or final compilation — because each points to a completely different fix.
When to Use
Use when compiled music videos or shorts have audio-video misalignment, when lip-sync looks wrong, when subtitles appear at the wrong time, when FFmpeg produces silent output, or when AI-generated clip speech conflicts with the target song audio. This is the media-specific complement to network-and-api-failure-triage and extends the investigation methods of hypothesis-driven-investigation into the AV domain.
Trigger keywords: out of sync, lip-sync, mouth not moving, silent audio, subtitle drift, VFR, ffmpeg merge, veo3 speech, pipeline no change, no visible change, audio drift, face not detected.
Prerequisites
FFmpeg and FFprobe installed and on PATH (Windows PowerShell: ffmpeg -version must succeed).
Python 3.8+ with numpy, scipy, opencv-python if using automated offset detection or coordinate diagnostics.
SyncNet (pip install syncnet-python) for quantitative lip-sync scoring.
InsightFace with buffalo_l model and det_10g.onnx detector for face detection on stylized content.
Whisper (or equivalent transcription tool) for subtitle timing validation.
Access to all intermediate pipeline artifacts (raw clips, Wav2Lip outputs, LivePortrait outputs, audio slices). If intermediates are missing, the pipeline must be re-run with intermediate output enabled.
Procedure
Core Principle
Never trust intermediate claims — visually verify every stage's output. A log line that says "lip-sync complete, 133/133 frames processed" is not verification. An extracted frame showing open mouths during speech is verification.
Phase 1: Identify the Failure Mode
Match the symptom to one of the seven canonical failure modes below before diving into debugging.
Compare the transcription against the target song lyrics at 3+ timestamps.
Fix: Regenerate clips using speech-suppressed prompts, or run the lip-sync re-sync pipeline to overwrite the visual mouth shapes with shapes driven by the target song audio.
Failure Mode 2: Silent Audio After FFmpeg Merge
Symptom: Final video plays but has no sound, or plays the wrong (silent AI-generated) audio track.
Root Cause: FFmpeg defaults to the first audio stream (often the silent AI-generated track from input 0) instead of the provided music file.
Diagnosis:
ffprobe -v error -select_streams a -show_entries stream=codec_name,channels,bit_rate -of default=noprint_wrappers=1 output.mp4
If bit_rate=N/A or channels=0, the audio stream is empty.
Fix: Add explicit stream mapping to force video from input 0 and audio from the music file input:
Failure Mode 4: Lip-Sync Face Detection Failure on Cartoons
Symptom: Lip-sync pipeline produces unchanged output or crashes with "no face detected."
Root Cause: Face detection models (trained on photorealistic faces) fail on heavily stylized Pixar/anime characters.
Diagnosis: Run the face detector in isolation and check if bounding boxes are returned. Print all detected faces and their bounding boxes:
import cv2
# Load frame from the intermediate video at the resolution it's processed at
frame = cv2.imread("frame.png")
# ... run face detector ...for face in detected_faces:
x1, y1, x2, y2 = face['bbox']
x_center = (x1 + x2) / 2.0
y_center = (y1 + y2) / 2.0
width = x2 - x1
height = y2 - y1
print(f"Face at center=({x_center:.0f}, {y_center:.0f}), size=({width:.0f}x{height:.0f})")
cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 2)
cv2.imwrite("debug_faces.png", frame)
Fix: Use InsightFace with the buffalo_l model and det_10g.onnx detector. For extreme stylization, use the LivePortrait two-stage pipeline which has better cartoon face handling.
Symptom: Lip-sync pipeline runs to completion without errors, but the output video shows identical closed-mouth frames as the input. No crash, no warning — just zero lip movement.
Root Cause: One or more stages in a multi-stage pipeline silently fail, producing pass-through output. Common culprits:
A function signature mismatch (method doesn't accept **kwargs but caller passes keyword args → TypeError or NameError silently swallowed).
Face detection bounding box coordinates are valid at one resolution but wrong after crop/scale transforms between pipeline stages.
The ML model (e.g., Wav2Lip) cannot generate meaningful output for the input domain (cartoon/3D vs photorealistic faces) but doesn't raise an error — it just outputs the original face unchanged.
Diagnosis: Follow the Multi-Stage Pipeline Debugging Protocol below.
Fix: Depends on which stage fails. If the ML model can't handle the input domain, switch to an alternative architecture (e.g., direct LivePortrait retargeting with a human driving video instead of Wav2Lip → LivePortrait chain).
Failure Mode 6: Subtitle Timing Drift
Symptom: Subtitles appear too early or too late relative to the vocals.
Root Cause: Subtitle timestamps were guessed or derived from storyboard estimates instead of actual audio transcription.
Diagnosis: Compare SRT timestamps against Whisper transcription timestamps of the audio file.
Fix: Re-generate subtitles from the actual audio using the auto_subtitle_generator.py script.
Failure Mode 7: Subtitle Oversized / Covering Character
Symptom: Text dominates the frame, covering the character's face or occupying >15% of screen height.
Root Cause: FontSize set too high (above 26 for libass, above 36 for MoviePy TextClip).
Fix: Reduce FontSize to 20–24 (libass) or 28–36 (MoviePy). Limit subtitle lines to 3–5 words per line.
Phase 2: Multi-Stage Pipeline Debugging Protocol
Use this systematic method when the failure mode is "no visible change" or when you need to localize which stage in a multi-stage pipeline (e.g., audio extraction → Wav2Lip → LivePortrait → FFmpeg compile) introduced the defect.
Step 1: Verify the Final Output First (Reverse Trace)
Start from the end product — the compiled video — and confirm the symptom visually:
View the extracted frames. Do NOT skip this step. Do NOT rely on logs or metadata claiming "lip-sync successful." The frame is the ground truth.
If the frame shows closed mouths during speech, the lip-sync failed. If mouths are open but wrong words, it's Failure Mode 1 (Veo 3 Speech Conflict). If the frame looks correct, the compilation step is the likely culprit.
Step 2: Trace Backwards Through Every Intermediate Artifact
For each pipeline stage, locate and visually inspect its output:
The stage where the defect first appears is the culprit. If the raw clip has closed mouths (expected) and the Wav2Lip output ALSO has closed mouths, Wav2Lip is the broken stage. If Wav2Lip output has open mouths but the LivePortrait output has closed mouths, LivePortrait is the broken stage.
Step 3: Read Code Signatures and Trace the Call Graph
Once you've localized the failing stage, read the actual source code — do not guess about how functions are called:
Read the entry point (the script that invokes the pipeline).
Read every function signature that handles the failing stage — check whether the method accepts **kwargs, what parameters it actually uses, and how they are passed.
Trace keyword arguments end-to-end: if the caller passes source_path=args.source, verify that every function in the chain actually receives and uses this parameter.
Look for kwargs in non-kwargs methods — a function that references kwargs.get(...) but has no **kwargs in its signature will raise a NameError at runtime.
Common code-level failures in media pipelines:
Method signature doesn't accept **kwargs but caller passes keyword args.
Scene-specific coordinate ranges hardcoded for one resolution but applied at a different resolution after crop/scale.
Face detection bounding box filter ranges (x_min, x_max, y_max_thresh) tuned for widescreen but applied to vertical video.
Exception swallowed in a try/except block, causing the pipeline to silently fall back to the unmodified input.
Step 4: Isolate Each Stage Independently
Run each pipeline stage in isolation to confirm it works or fails on its own:
View both frames side-by-side. If the Wav2Lip frame looks identical to the raw frame, Wav2Lip is not generating lip shapes for this face type.
Step 5: Check Coordinate Systems Across Resolution Transforms
Multi-stage pipelines often transform video resolution between stages. Face detection coordinates that work at one resolution may be completely wrong at another. Use the diagnostic Python snippet from Failure Mode 4 to print all detected faces and draw bounding boxes.
Common coordinate failures:
Widescreen (1920×1080) coordinates applied to vertical (720×1280) after crop.
Face center ranges [300, 480] tuned for a specific character position but the character is at x=643 after re-composition.
Multiple faces detected; the filter selects the wrong one (e.g., steering wheel scored higher than the actual face because the size threshold was too permissive).
Step 6: Verify Audio Input Quality
A lip-sync pipeline that receives silence or near-silence will correctly produce no lip movement:
If mean_volume < -40 dB, the segment is effectively silent.
Step 7: Differential Test — Swap One Component
If still stuck after Steps 1–6, perform a differential test:
Replace the cartoon face with a photorealistic human face and run the same pipeline → if lips move, the model can't handle cartoon faces.
Replace the audio with a known-good speech clip → if lips move, the audio input was the problem.
Replace the driving video with a pre-made driving video with visible lip motion → if the source video still shows closed mouths, the motion transfer stage is broken.
Only change one variable at a time. Each test should confirm or eliminate exactly one hypothesis.
Phase 3: Automated Quality Verification
Apply these quantitative checks after applying a fix to confirm the defect is resolved.
SyncNet Lip-Sync Scoring
Quantify lip-sync quality without ground-truth video:
Flag segments where VMAF drops below 80. Use VMAF-CUDA for GPU-accelerated monitoring.
VFR Detection
The most reliable VFR detection method:
ffmpeg -i input.mp4 -vf vfrdet -an -f null -
Parse stderr for VFR:X.XXX — any value > 0 indicates variable frame rate. ffprobe metadata is unreliable (reports nominal rate, not actual).
Cross-Correlation Offset Detection
Programmatically detect audio-video offset:
import numpy as np
from scipy.signal import correlate
# Extract audio from both tracks, compute cross-correlation
correlation = correlate(audio_track_1, audio_track_2, mode='full')
lag = np.argmax(correlation) - len(audio_track_1) + 1
offset_ms = lag / sample_rate * 1000print(f"Detected offset: {offset_ms:.1f}ms")
# ITU-R BT.1359: ±45ms acceptable, ±80ms annoying
Subtitle Timing Validation
Auto-align with ffsubsync or alass.
Validate: CPS (characters per second) ≤ 25, duration ≥ 0.7s, no overlapping cues.
Match against platform specs (Netflix, YouTube).
Pitfalls
"Verified" without transcription — claiming the video is correct without transcribing and comparing spoken words against the audio track.
Retrying the same pipeline — re-running an identical command expecting different results instead of diagnosing the root cause.
Trusting AI-claimed sync — accepting lip-sync or subtitle alignment based on metadata alone without visual and auditory human verification.
Ignoring VFR — concatenating variable-frame-rate clips and wondering why audio drifts. Always use vfrdet filter, not ffprobe nominal rate.
Oversized text defense — "the text looks fine to me" without extracting a frame and measuring screen percentage.
Trusting logs over frames — reading a log line that says "lip-sync complete, 133/133 frames processed" and claiming success without extracting a single frame from the output and visually confirming mouths are open.
Skipping intermediate outputs — jumping from "the pipeline ran" to "the final video is wrong" without checking which stage's output first shows the defect.
Assuming code matches docs — the walkthrough says the face coordinates were fixed, but the actual code may have a different bug (e.g., kwargs referenced in a non-kwargs method) that prevents the fix from ever executing.
Resolution-blind coordinate hardcoding — using face detection coordinates tuned at one resolution across pipeline stages that operate at different resolutions.
Silent fallback acceptance — not noticing that a pipeline stage caught an exception and silently returned the unmodified input instead of the processed result.
FFmpeg default stream selection — relying on FFmpeg's automatic stream selection instead of explicit -map flags when inputs have multiple audio streams.
ffprobe VFR false negatives — ffprobe reports the nominal frame rate, not the actual frame rate. Use the vfrdet filter for reliable VFR detection.
Verification
Diagnostic Checklist
Confirm each item before declaring the pipeline fixed:
