FFmpeg + OpenCV Integration Guide
Use this skill when FFmpeg handles video I/O and OpenCV handles image processing. This SKILL is a lean orchestrator; full pipe patterns, library examples, and Modal recipes are preserved in references/opencv-pipelines-and-libraries.md.
When to Use
- Decode with FFmpeg and process frames with OpenCV
- Encode OpenCV-generated/processed frames with FFmpeg
- Compare
cv2.VideoCapture, subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord
- Fix RGB/BGR color bugs or
(height, width) dimension-order bugs
- Preserve audio while replacing processed video frames
- Build GPU-assisted video I/O around CPU OpenCV processing
Library Selection
| Need |
Best option |
Why |
| Simple local file |
cv2.VideoCapture |
Built-in and simple |
| Full FFmpeg format/protocol support |
subprocess pipe |
Exact CLI behavior |
| GPU video I/O |
ffmpegcv |
NVDEC/NVENC with OpenCV-like API |
| Network/RTSP streaming |
VidGear |
Threaded capture and stream helpers |
| ML batch loading |
Decord |
Fast random/batch frame access |
| Frame-level libav control |
PyAV |
Direct FFmpeg library access |
Critical Gotchas
- OpenCV is BGR. FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
- NumPy dimensions are
(height, width, channels). Pixel access is img[y, x], not img[x, y].
- Video filters can drop audio. Preserve or remux original audio intentionally.
- Release resources. Always close
VideoCapture, pipes, writers, and PyAV containers.
- Rawvideo pipes require exact frame size.
width * height * channels must match -pix_fmt.
Minimal Pipe Patterns
FFmpeg to OpenCV using BGR frames:
cmd = [
"ffmpeg", "-i", input_path,
"-f", "rawvideo", "-pix_fmt", "bgr24", "-"
]
OpenCV to FFmpeg using BGR frames:
cmd = [
"ffmpeg", "-y",
"-f", "rawvideo", "-vcodec", "rawvideo",
"-s", f"{width}x{height}", "-pix_fmt", "bgr24",
"-r", str(fps), "-i", "-",
"-c:v", "libx264", "-preset", "fast", "-crf", "23",
"-pix_fmt", "yuv420p", output_path
]
Core Workflow
- Probe input dimensions, fps, duration, and audio streams.
- Pick the I/O library based on the selection table.
- Lock pixel format at boundaries (
bgr24 for OpenCV pipes; yuv420p for final H.264 output).
- Process frames in generators/batches; avoid loading full videos unless they are small.
- Preserve or re-encode audio explicitly.
- Verify output duration, fps, resolution, codec, and A/V sync.
Reference Map
references/opencv-pipelines-and-libraries.md - Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.
Related Skills
ffmpeg-python-integration-reference - Type-safe parameters, colors, time unitsffmpeg-pyav-integration - PyAV API detailsffmpeg-hardware-acceleration - GPU decode/encode and filter pipelines
1---2name: ffmpeg-opencv-integration3description: Complete FFmpeg + OpenCV + Python integration guide for video processing pipelines. PROACTIVELY activate for: (1) FFmpeg to OpenCV frame handoff, (2) cv2.VideoCapture vs ffmpeg subprocess, (3) BGR/RGB color format conversion gotchas, (4) Frame dimension order img[y,x] vs img[x,y], (5) ffmpegcv GPU-accelerated video I/O, (6) VidGear multi-threaded streaming, (7) Decord batch video loading for ML, (8) PyAV frame-level processing, (9) Audio stream preservation with video filters, (10) Memory-efficient frame generators, (11) OpenCV + FFmpeg + Modal parallel processing, (12) Pipe frames between FFmpeg and OpenCV. Provides: Color format conversion patterns, coordinate system gotchas, library selection guide, memory management, subprocess pipe patterns, GPU-accelerated alternatives to cv2.VideoCapture. Ensures: Correct integration between FFmpeg and OpenCV without color/coordinate bugs. See also: ffmpeg-python-integration-reference for type-safe parameter mappings.4---5
6# FFmpeg + OpenCV Integration Guide
7
8Use this skill when FFmpeg handles video I/O and OpenCV handles image processing. This SKILL is a lean orchestrator; full pipe patterns, library examples, and Modal recipes are preserved in `references/opencv-pipelines-and-libraries.md`.
9
10## When to Use
11
12- Decode with FFmpeg and process frames with OpenCV
13- Encode OpenCV-generated/processed frames with FFmpeg
14- Compare `cv2.VideoCapture`, subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord
15- Fix RGB/BGR color bugs or `(height, width)` dimension-order bugs
16- Preserve audio while replacing processed video frames
17- Build GPU-assisted video I/O around CPU OpenCV processing
18
19## Library Selection
20
21| Need | Best option | Why |
22|---|---|---|
23| Simple local file | `cv2.VideoCapture` | Built-in and simple |
24| Full FFmpeg format/protocol support | subprocess pipe | Exact CLI behavior |
25| GPU video I/O | ffmpegcv | NVDEC/NVENC with OpenCV-like API |
26| Network/RTSP streaming | VidGear | Threaded capture and stream helpers |
27| ML batch loading | Decord | Fast random/batch frame access |
28| Frame-level libav control | PyAV | Direct FFmpeg library access |
29
30## Critical Gotchas
31
321. **OpenCV is BGR.** FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
332. **NumPy dimensions are `(height, width, channels)`.** Pixel access is `img[y, x]`, not `img[x, y]`.
343. **Video filters can drop audio.** Preserve or remux original audio intentionally.
354. **Release resources.** Always close `VideoCapture`, pipes, writers, and PyAV containers.
365. **Rawvideo pipes require exact frame size.** `width * height * channels` must match `-pix_fmt`.
37
38## Minimal Pipe Patterns
39
40FFmpeg to OpenCV using BGR frames:
41
42```python
43cmd = [
44 "ffmpeg", "-i", input_path,
45 "-f", "rawvideo", "-pix_fmt", "bgr24", "-"
46]
47```
48
49OpenCV to FFmpeg using BGR frames:
50
51```python
52cmd = [
53 "ffmpeg", "-y",
54 "-f", "rawvideo", "-vcodec", "rawvideo",
55 "-s", f"{width}x{height}", "-pix_fmt", "bgr24",
56 "-r", str(fps), "-i", "-",
57 "-c:v", "libx264", "-preset", "fast", "-crf", "23",
58 "-pix_fmt", "yuv420p", output_path
59]
60```
61
62## Core Workflow
63
641. Probe input dimensions, fps, duration, and audio streams.
652. Pick the I/O library based on the selection table.
663. Lock pixel format at boundaries (`bgr24` for OpenCV pipes; `yuv420p` for final H.264 output).
674. Process frames in generators/batches; avoid loading full videos unless they are small.
685. Preserve or re-encode audio explicitly.
696. Verify output duration, fps, resolution, codec, and A/V sync.
70
71## Reference Map
72
73- `references/opencv-pipelines-and-libraries.md` - Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.
74
75## Related Skills
76
77- `ffmpeg-python-integration-reference` - Type-safe parameters, colors, time units
78- `ffmpeg-pyav-integration` - PyAV API details
79- `ffmpeg-hardware-acceleration` - GPU decode/encode and filter pipelines
josiahsiegel