FFmpeg + OpenCV Integration Guide
Use this skill when FFmpeg handles video I/O and OpenCV handles image processing. This SKILL is a lean orchestrator; full pipe patterns, library examples, and Modal recipes are preserved in references/opencv-pipelines-and-libraries.md.
When to Use
- Decode with FFmpeg and process frames with OpenCV
- Encode OpenCV-generated/processed frames with FFmpeg
- Compare
cv2.VideoCapture, subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord
- Fix RGB/BGR color bugs or
(height, width) dimension-order bugs
- Preserve audio while replacing processed video frames
- Build GPU-assisted video I/O around CPU OpenCV processing
Library Selection
| Need |
Best option |
Why |
| Simple local file |
cv2.VideoCapture |
Built-in and simple |
| Full FFmpeg format/protocol support |
subprocess pipe |
Exact CLI behavior |
| GPU video I/O |
ffmpegcv |
NVDEC/NVENC with OpenCV-like API |
| Network/RTSP streaming |
VidGear |
Threaded capture and stream helpers |
| ML batch loading |
Decord |
Fast random/batch frame access |
| Frame-level libav control |
PyAV |
Direct FFmpeg library access |
Critical Gotchas
- OpenCV is BGR. FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
- NumPy dimensions are
(height, width, channels). Pixel access is img[y, x], not img[x, y].
- Video filters can drop audio. Preserve or remux original audio intentionally.
- Release resources. Always close
VideoCapture, pipes, writers, and PyAV containers.
- Rawvideo pipes require exact frame size.
width * height * channels must match -pix_fmt.
Minimal Pipe Patterns
FFmpeg to OpenCV using BGR frames:
cmd = [
"ffmpeg", "-i", input_path,
"-f", "rawvideo", "-pix_fmt", "bgr24", "-"
]
OpenCV to FFmpeg using BGR frames:
cmd = [
"ffmpeg", "-y",
"-f", "rawvideo", "-vcodec", "rawvideo",
"-s", f"{width}x{height}", "-pix_fmt", "bgr24",
"-r", str(fps), "-i", "-",
"-c:v", "libx264", "-preset", "fast", "-crf", "23",
"-pix_fmt", "yuv420p", output_path
]
Core Workflow
- Probe input dimensions, fps, duration, and audio streams.
- Pick the I/O library based on the selection table.
- Lock pixel format at boundaries (
bgr24 for OpenCV pipes; yuv420p for final H.264 output).
- Process frames in generators/batches; avoid loading full videos unless they are small.
- Preserve or re-encode audio explicitly.
- Verify output duration, fps, resolution, codec, and A/V sync.
Reference Map
references/opencv-pipelines-and-libraries.md - Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.
Related Skills
ffmpeg-python-integration-reference - Type-safe parameters, colors, time units
ffmpeg-pyav-integration - PyAV API details
ffmpeg-hardware-acceleration - GPU decode/encode and filter pipelines
1---
2name: ffmpeg-opencv-integration
3description: |
4 Complete FFmpeg + OpenCV + Python integration guide for video processing pipelines.
5 PROACTIVELY activate for: (1) FFmpeg to OpenCV frame handoff, (2) cv2.VideoCapture vs ffmpeg subprocess, (3) BGR/RGB color format conversion gotchas, (4) Frame dimension order img[y,x] vs img[x,y], (5) ffmpegcv GPU-accelerated video I/O, (6) VidGear multi-threaded streaming, (7) Decord batch video loading for ML, (8) PyAV frame-level processing, (9) Audio stream preservation with video filters, (10) Memory-efficient frame generators, (11) OpenCV + FFmpeg + Modal parallel processing, (12) Pipe frames between FFmpeg and OpenCV.
6 Provides: Color format conversion patterns, coordinate system gotchas, library selection guide, memory management, subprocess pipe patterns, GPU-accelerated alternatives to cv2.VideoCapture.
7 Ensures: Correct integration between FFmpeg and OpenCV without color/coordinate bugs.
8 See also: ffmpeg-python-integration-reference for type-safe parameter mappings.
9metadata:
10 source: https://skillmd.com/skills/josiahsiegel/ffmpeg-opencv-integration
11---
12
13# FFmpeg + OpenCV Integration Guide
14
15Use this skill when FFmpeg handles video I/O and OpenCV handles image processing. This SKILL is a lean orchestrator; full pipe patterns, library examples, and Modal recipes are preserved in `references/opencv-pipelines-and-libraries.md`.
16
17## When to Use
18
19- Decode with FFmpeg and process frames with OpenCV
20- Encode OpenCV-generated/processed frames with FFmpeg
21- Compare `cv2.VideoCapture`, subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord
22- Fix RGB/BGR color bugs or `(height, width)` dimension-order bugs
23- Preserve audio while replacing processed video frames
24- Build GPU-assisted video I/O around CPU OpenCV processing
25
26## Library Selection
27
28| Need | Best option | Why |
29|---|---|---|
30| Simple local file | `cv2.VideoCapture` | Built-in and simple |
31| Full FFmpeg format/protocol support | subprocess pipe | Exact CLI behavior |
32| GPU video I/O | ffmpegcv | NVDEC/NVENC with OpenCV-like API |
33| Network/RTSP streaming | VidGear | Threaded capture and stream helpers |
34| ML batch loading | Decord | Fast random/batch frame access |
35| Frame-level libav control | PyAV | Direct FFmpeg library access |
36
37## Critical Gotchas
38
391. **OpenCV is BGR.** FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
402. **NumPy dimensions are `(height, width, channels)`.** Pixel access is `img[y, x]`, not `img[x, y]`.
413. **Video filters can drop audio.** Preserve or remux original audio intentionally.
424. **Release resources.** Always close `VideoCapture`, pipes, writers, and PyAV containers.
435. **Rawvideo pipes require exact frame size.** `width * height * channels` must match `-pix_fmt`.
44
45## Minimal Pipe Patterns
46
47FFmpeg to OpenCV using BGR frames:
48
49```python
50cmd = [
51 "ffmpeg", "-i", input_path,
52 "-f", "rawvideo", "-pix_fmt", "bgr24", "-"
53]
54```
55
56OpenCV to FFmpeg using BGR frames:
57
58```python
59cmd = [
60 "ffmpeg", "-y",
61 "-f", "rawvideo", "-vcodec", "rawvideo",
62 "-s", f"{width}x{height}", "-pix_fmt", "bgr24",
63 "-r", str(fps), "-i", "-",
64 "-c:v", "libx264", "-preset", "fast", "-crf", "23",
65 "-pix_fmt", "yuv420p", output_path
66]
67```
68
69## Core Workflow
70
711. Probe input dimensions, fps, duration, and audio streams.
722. Pick the I/O library based on the selection table.
733. Lock pixel format at boundaries (`bgr24` for OpenCV pipes; `yuv420p` for final H.264 output).
744. Process frames in generators/batches; avoid loading full videos unless they are small.
755. Preserve or re-encode audio explicitly.
766. Verify output duration, fps, resolution, codec, and A/V sync.
77
78## Reference Map
79
80- `references/opencv-pipelines-and-libraries.md` - Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.
81
82## Related Skills
83
84- `ffmpeg-python-integration-reference` - Type-safe parameters, colors, time units
85- `ffmpeg-pyav-integration` - PyAV API details
86- `ffmpeg-hardware-acceleration` - GPU decode/encode and filter pipelines