Rust Ffmpeg

Rust FFmpeg

Guide for implementing FFmpeg functionality in Rust: library selection, code generation, and problem solving.

Selection Framework

Decision Logic

Layer 1: Integration Method

Default: Library Integration (ez-ffmpeg/ffmpeg-next/ffmpeg-sys-next)

• Better performance, type-safe API, direct Frame access

Alternative: Binary Approach (ffmpeg-sidecar) - consider when:

• Cannot install FFmpeg development libraries
• Restricted CI/CD environment without admin access
• Pure CLI batch processing (no real-time needs)

If installation constrained → Load ffmpeg_sidecar.md

Layer 2: Scenario Detection

Multiple rows can match one request — load all applicable references, then follow the most specific row's target first.

Layer 3: Library Selection

CLI Migration: Both ez-ffmpeg and ffmpeg-sidecar support CLI-style APIs. Choose based on constraints below.

• ez-ffmpeg: CLI Migration Guide — library integration, native performance
• ffmpeg-sidecar: CLI to Rust Migration — process wrapper, no FFmpeg libs needed

1. Need async/await? → ez-ffmpeg (only library with native async)
2. Need custom Rust frame processing? → ez-ffmpeg FrameFilter (safe, simple API)
3. Need custom GPU shaders (WGSL)? → ez-ffmpeg wgpu feature (WgpuFrameFilter, headless, no display server) — check the built-in wgpu_filter::effects catalog first (13 typed effects, 0.13+); for native hardware filters (scale_cuda/scale_vaapi/libplacebo) probe with get_gpu_filter_backends()
4. Need to burn in subtitles without libass? → ez-ffmpeg subtitle feature (SubtitleFilter, pure-Rust ASS/SRT renderer, no --enable-libass)
5. Need typed detection/measurement (black/silence/scene/crop/EBU R128)? → ez-ffmpeg analysis module (Analysis one-shot or MetadataEventFilter streaming — no feature flag)
6. Need a one-shot thumbnail / sprite sheet / GIF / HLS ABR ladder? → ez-ffmpeg recipes module (no feature flag)
7. Need frame-level codec control? → ffmpeg-next for codec internals
8. Need custom I/O from memory? → ez-ffmpeg (read/write/seek callbacks), or ffmpeg-next safe stream I/O (input_from_stream/output_to_stream, any Read/Write/Seek); ffmpeg-sys-next for lowest-level control
9. Need zero-copy or max performance? → ffmpeg-sys-next (requires unsafe code)
10. Cannot install FFmpeg libs? → ffmpeg-sidecar (process-based, stdin/stdout only, no custom I/O)
11. Need decoded frames/audio in memory for AI/CV/ASR? → ez-ffmpeg frame_export (FrameExtractor → packed RGB, SampleExtractor::for_whisper → 16 kHz mono f32; no feature flag; experimental, 0.14) — see frame_io.md
12. Need to generate/push video from Rust-rendered frames? → ez-ffmpeg VideoWriter (frame push into encode/mux/stream, no demuxer; experimental, 0.14) — see frame_io.md
13. Need to feed a WebCodecs decoder, RTP/SRT packetizer, or hand-rolled fMP4 segmenter straight from the encoder? → ez-ffmpeg packet_sink (PacketSink, strict H.264-only/AAC tier, no container; no feature flag; experimental, 0.15) — see packet_sink.md
14. Need to run or auto-translate an existing ffmpeg CLI command? → ez-ffmpeg cli feature (from_cli_args/emit_rust_code, narrow golden-tested subset — 6 verified shapes; 0.15) — see cli_compat.md; anything outside that subset still needs the manual cli_migration.md tables

Note: FrameFilter (custom Rust frame processing) is separate from custom I/O callbacks (custom data sources/sinks). ez-ffmpeg supports both.

Quick Start

New to Rust FFmpeg? See quick_start.md for 5-minute setup.

Library References

• ez-ffmpeg - High-level API (sync and async)
• ffmpeg-next - Medium-level API for advanced control
• ffmpeg-sys-next - Low-level unsafe FFI
• ffmpeg-sidecar - CLI wrapper

Version Compatibility

Source: crates.io

FFmpeg 7 vs 8: all four libraries build against both majors with the current crate versions above — no crate-major-to-system-major matching. Guard: when pulling ffmpeg-next alongside ez-ffmpeg, keep it at 8.1.0 — a 7.1.0 mixed in collides via the links = "ffmpeg" key. Bindgen mechanics, the legacy 7.1-pin rationale, and the rust-ffmpeg #246 history: see installation.md.

Installation Issues: installation.md — includes the Windows vcpkg static-link unresolved external symbol fix (extra system libs in your app's build.rs; VCPKG_ROOT must be set in the shell, not via std::env::set_var in build.rs)

Guidelines for Claude

When this skill activates, follow this workflow:

For implementation tasks:

1. Identify task — Determine: video/audio/streaming/inspection? Simple or complex?
2. Select library — Apply Layer 3 decision logic. State which library and why in one sentence
3. Load references — Follow Layer 2 scenario detection to load the right reference files
4. Generate code — Production-ready: proper error handling, Result<> return types, no unwrap() in library code
5. Explain briefly — One-line summary of approach before the code block. No lengthy tutorials
6. Suggest next steps — If applicable: performance optimization, hardware acceleration, or testing

For evaluation/migration/selection tasks:

1. Load selection guide — Load library_selection.md and relevant scenario references
2. Assess requirements — Identify constraints: async needs, frame-level access, install limitations, existing code patterns
3. Compare options — Apply Layer 3 decision logic against the specific use case
4. Give verdict — Clear feasibility conclusion with rationale. If migration is possible, show key API differences. If not, explain why and recommend the alternative

Rules:

• Follow Layer 3 decision logic to select the library
• Always add Cargo.toml dependencies when introducing a new library
• Use async when the user's context is async (tokio/actix/axum)
• If the user's need is ambiguous between libraries, ask — don't guess
• For complex pipelines, break into steps with comments, not monolithic blocks

Interaction Examples

User: "I need frame-level access to decode H.264 and apply custom processing" Claude: Identify → frame-level + custom processing → ez-ffmpeg FrameFilter (safe, simple) or ffmpeg-next (codec internals) → ask user preference if unclear → load ffmpeg_next.md or ez_ffmpeg/filters.md → generate decode loop with proper EAGAIN handling

User: "Can this video trimming code be converted to ez-ffmpeg? Fall back to ffmpeg-next if not" Claude: Identify → library migration feasibility → load library_selection.md + video_transcoding.md → review existing code against ez-ffmpeg API → assess feasibility → provide migration path or explain why ffmpeg-next is needed

User: "Can you just run this ffmpeg command from Rust instead of me hand-writing the builder chain?" Claude: Identify → automatic CLI translation → load cli_compat.md → if the command matches one of the 6 verified shapes, use cli::from_cli_args/emit_rust_code, flagging the FFmpeg-7.1-only runtime gate on execution → otherwise fall back to the manual cli_migration.md tables

Best Practices

• Codec copy first: use -c copy / stream copy only when no re-encoding is needed — 10x faster, zero quality loss
• Keyframe alignment: for HLS/DASH segmentation use fixed-GOP codec opts (g/keyint_min/sc_threshold); for keyframes at known absolute times use Output::set_force_key_frames("0,5,10.5") — it is not an encoder AVOption, so set_video_codec_opt is silently ignored
• Error propagation: return Result<T, Box<dyn Error>>, never panic in library code
• Resource cleanup: rely on RAII — FfmpegContext/Decoder/Encoder drop automatically; don't add manual cleanup
• Hardware acceleration: probe availability at runtime before enabling (get_gpu_filter_backends() for GPU filters); opengl is deprecated — use wgpu (see hardware_acceleration.md)
• Built-in GPU effects before custom WGSL: check the typed wgpu_filter::effects catalog before writing shaders (see filters.md)
• Prefer recipes for one-shot jobs: recipes::{thumbnail, sprite_sheet, animated_gif, HlsLadder} bake in the correctness details (see image_sequences.md, gif_creation.md, streaming_rtmp_hls.md)
• Measure, don't scrape logs: typed analysis API for black/silence/scene/crop/EBU R128 (see detection_analysis.md)
• Frames in/out of memory: frame_export::{FrameExtractor, SampleExtractor} for AI/CV/ASR ingest, VideoWriter for frame push — experimental, default build. Guard: the default conversion precision changed in 0.15 — byte-identical 0.14 output needs .conversion_precision(ConversionPrecision::High) (see frame_io.md)
• Encoded packets (PacketSink): strict tier is H.264 libx264-only + AAC, and backpressure blocks by design — a slow consumer stalls the encoders; drain concurrently, never only after wait() (see packet_sink.md)
• Automatic CLI translation: from_cli_args/emit_rust_code cover only 6 golden-tested shapes, and from_cli_args execution refuses any unverified runtime — FFmpeg 7.1 only as of 0.16 (8.x refused even for verified shapes; emit_rust_code unaffected). Indexed per-stream options (-c:v:0) are permanently outside the subset — port them with StreamMap (see cli_compat.md, fallback cli_migration.md)
• Mixed stream lengths: Output::set_shortest(true) (FFmpeg -shortest parity) ends the output when the shortest stream ends — without it, video + longer music keeps running to the longer stream
• Typed progress: use progress_handle() snapshots, not hand-rolled FrameFilters. Guard: every metric is Option — None means unknowable (HLS/null muxers, packet-sink outputs), never fabricate zero; percentage needs your own total via percent_of(total_us) (see advanced.md)
• Per-stream encoders: same-type streams in one output need StreamMap per-map codecs (-c:v:0 parity); per-map settings override per-type setters key by key, and copy×re-encode conflicts fail typed at build() (see video.md)
• HDR→SDR needs tone mapping, not scaling: naive scale,format=yuv420p yields washed-out gray; route on StreamInfo::Video color_transfer (PQ=16/HLG=18 — transfer, not primaries) and use the cookbook chains (see modern_codecs.md)
• Graceful shutdown: stop() flushes and returns Result (check it — output valid on Ok); abort() is the hard cancel with no output guarantee. The embedded RTMP server's stop() joins its threads — never call it from inside a logger (see streaming.md)
• In-place frame edits: call make_frame_writable before mutating frame data in a FrameFilter — decoder frames are refcounted and shared
• Testing: prefer lavfi-generated media (testsrc/sine) over shipping binary fixtures where suitable (see testing.md)