Ffmpeg

FFmpeg Media Engineering AI Skill Guide (Claude)

Overview & Engine Architecture

FFmpeg is the universal open-source command-line framework for video/audio decoding, transcoding, streaming, muxing, and complex filtergraph processing. Claude operates as a Principal Video Streaming and Codec Engineer, specializing in codec rate control (CRF, CBR, VBR, CQP), hardware acceleration (NVIDIA NVENC, Intel QuickSync/QSV, Apple VideoToolbox, VAAPI), adaptive bitrate HLS/DASH packaging, and Python asyncio batch automation.

FFmpeg Core Subsystems & Pipeline Architecture

┌─────────────────────────────────────────────────────────────┐
│                 FFmpeg Transcoding Pipeline                 │
│                                                             │
│  Demuxing & Decoding Layer                                  │
│  ├── `libavformat` (Container Demuxer: MP4, MKV, MOV, TS)   │
│  ├── `libavcodec` (Decoders: H.264, HEVC, AV1, ProRes, AAC) │
│  └── Hardware Decoders (`cuvid`, `qsv`, `videotoolbox`)     │
│                                                             │
│  Processing & Encoding Layer                                │
│  ├── `libavfilter` (Complex Filtergraphs: scale, pad, fps)  │
│  ├── `libswscale` & `libswresample` (Color & Audio Resample)│
│  └── Hardware Encoders (`h264_nvenc`, `hevc_qsv`, `libsvtav1`)│
└─────────────────────────────────────────────────────────────┘

Operational Capabilities & Agent Directives

1. Hardware-Accelerated Codec Optimization: Configure optimal encoding flags for target hardware platforms (-hwaccel cuda -c:v h264_nvenc -preset p7 -tune hq -rc vbr -cq 19).
2. Deterministic Filtergraph Authoring: Author multi-input/multi-output -filter_complex graphs for watermark overlay, side-by-side video stitching, loudness normalization (loudnorm), and subtitle burn-in.
3. Adaptive Bitrate (ABR) HLS Streaming: Construct multi-rendition HLS pipelines (1080p, 720p, 480p) with keyframe interval alignment (-g 60 -keyint_min 60 -sc_threshold 0).
4. Automated Stream Health Diagnostics: Analyze ffprobe JSON outputs to detect variable framerates (VFR), corrupted audio PTS/DTS timestamps, and pixel format incompatibilities.

Production Python Automation: Adaptive Multi-Rendition HLS Packager

Save this script as hls_packager.py and run with Python 3 to generate a production-ready Master HLS playlist with 1080p, 720p, and 480p streams:

"""
FFmpeg Automated Adaptive Bitrate (ABR) HLS Packager
Generates aligned keyframe HLS renditions and a Master Playlist.
"""

import sys
import os
import subprocess
import json

def get_video_info(input_file: str) -> dict:
    cmd = [
        "ffprobe", "-v", "quiet", "-print_format", "json",
        "-show_format", "-show_streams", input_file
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"FFprobe failed: {result.stderr}")
    return json.loads(result.stdout)

def encode_hls_stream(input_file: str, output_dir: str):
    if not os.path.exists(input_file):
        print(f"Error: Input file '{input_file}' not found.")
        sys.exit(1)

    os.makedirs(output_dir, exist_ok=True)
    
    # FFmpeg Command for Multi-Rendition HLS
    ffmpeg_cmd = [
        "ffmpeg", "-y", "-i", input_file,
        # Common encoding parameters: 2s GOP at 30fps = 60 frames
        "-filter_complex",
        "[0:v]split=3[v1][v2][v3]; "
        "[v1]scale=w=1920:h=1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2[v1out]; "
        "[v2]scale=w=1280:h=720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2[v2out]; "
        "[v3]scale=w=854:h=480:force_original_aspect_ratio=decrease,pad=854:480:(ow-iw)/2:(oh-ih)/2[v3out]",
        
        # Rendition 1 (1080p)
        "-map", "[v1out]", "-c:v:0", "libx264", "-b:v:0", "5000k", "-maxrate:v:0", "5350k", "-bufsize:v:0", "7500k",
        # Rendition 2 (720p)
        "-map", "[v2out]", "-c:v:1", "libx264", "-b:v:1", "2800k", "-maxrate:v:1", "2996k", "-bufsize:v:1", "4200k",
        # Rendition 3 (480p)
        "-map", "[v3out]", "-c:v:2", "libx264", "-b:v:2", "1400k", "-maxrate:v:2", "1498k", "-bufsize:v:2", "2100k",
        
        # Map Audio
        "-map", "0:a?", "-c:a", "aac", "-b:a", "128k", "-ac", "2",
        
        # Strict GOP alignment for HLS
        "-g", "60", "-keyint_min", "60", "-sc_threshold", "0",
        
        # HLS Packaging Flags
        "-f", "hls",
        "-hls_time", "4",
        "-hls_playlist_type", "vod",
        "-hls_flags", "independent_segments",
        "-hls_segment_type", "mpegts",
        "-hls_segment_filename", os.path.join(output_dir, "stream_%v_data%03d.ts"),
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0 v:1,a:0 v:2,a:0",
        os.path.join(output_dir, "stream_%v.m3u8")
    ]

    print(f"Starting Multi-Rendition HLS encoding for {input_file}...")
    proc = subprocess.run(ffmpeg_cmd, capture_output=True, text=True)
    
    if proc.returncode == 0:
        print(f"HLS package generated successfully at: {os.path.join(output_dir, 'master.m3u8')}")
    else:
        print(f"Encoding failed:\n{proc.stderr}")

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: python hls_packager.py <input_video> <output_dir>")
        sys.exit(1)
    encode_hls_stream(sys.argv[1], sys.argv[2])

Technical Troubleshooting Matrix

Command Line Syntax & Production Recipes

# 1. High-Quality Web Transcode (Software CRF 23, Faststart for Web Streaming)
ffmpeg -i input.mov -c:v libx264 -preset medium -crf 23 -pix_fmt yuv420p -movflags +faststart -c:a aac -b:a 160k output.mp4

# 2. Ultra-Fast GPU Transcode (NVIDIA NVENC Hardware Encoding)
ffmpeg -hwaccel cuda -i input.mkv -c:v h264_nvenc -preset p6 -cq 20 -c:a copy output.mp4

# 3. Lossless Stream Remux (Instant Container Conversion without Re-encoding)
ffmpeg -i input.mkv -c copy -movflags +faststart output.mp4

# 4. Two-Pass Audio Loudness Normalization (EBU R128 Broadcast Standard)
ffmpeg -i input.wav -af loudnorm=I=-16:TP=-1.5:LRA=11 -c:a pcm_s24le output.wav

Agent Operational Directive

MANDATORY: For web-distributed MP4 files, always include -pix_fmt yuv420p for broad browser compatibility and -movflags +faststart to move the MOOV atom to the beginning of the file for instant streaming playback.