video-understand
Understand video content locally using ffmpeg for frame extraction and Whisper for transcription. Fully offline, no API keys required.
Prerequisites
ffmpeg + ffprobe (required): brew install ffmpeg
openai-whisper (optional, for transcription): pip install openai-whisper
Commands
# Scene detection + transcribe (default)
python3 skills/video-understand/scripts/understand_video.py video.mp4
# Keyframe extraction
python3 skills/video-understand/scripts/understand_video.py video.mp4 -m keyframe
# Regular interval extraction
python3 skills/video-understand/scripts/understand_video.py video.mp4 -m interval
# Limit frames extracted
python3 skills/video-understand/scripts/understand_video.py video.mp4 --max-frames 10
# Use a larger Whisper model
python3 skills/video-understand/scripts/understand_video.py video.mp4 --whisper-model small
# Frames only, skip transcription
python3 skills/video-understand/scripts/understand_video.py video.mp4 --no-transcribe
# Quiet mode (JSON only, no progress)
python3 skills/video-understand/scripts/understand_video.py video.mp4 -q
# Output to file
python3 skills/video-understand/scripts/understand_video.py video.mp4 -o result.json
CLI Options
| Flag |
Description |
video |
Input video file (positional, required) |
-m, --mode |
Extraction mode: scene (default), keyframe, interval |
--max-frames |
Maximum frames to keep (default: 20) |
--whisper-model |
Whisper model size: tiny, base, small, medium, large (default: base) |
--no-transcribe |
Skip audio transcription, extract frames only |
-o, --output |
Write result JSON to file instead of stdout |
-q, --quiet |
Suppress progress messages, output only JSON |
Extraction Modes
| Mode |
How it works |
Best for |
scene |
Detects scene changes via ffmpeg select='gt(scene,0.3)' |
Most videos, varied content |
keyframe |
Extracts I-frames (codec keyframes) |
Encoded video with natural keyframe placement |
interval |
Evenly spaced frames based on duration and max-frames |
Fixed sampling, predictable output |
If scene mode detects no scene changes, it automatically falls back to interval mode.
Output
The script outputs JSON to stdout (or file with -o). See references/output-format.md for the full schema.
{
"video": "video.mp4",
"duration": 18.076,
"resolution": {"width": 1224, "height": 1080},
"mode": "scene",
"frames": [
{"path": "/abs/path/frame_0001.jpg", "timestamp": 0.0, "timestamp_formatted": "00:00"}
],
"frame_count": 12,
"transcript": [
{"start": 0.0, "end": 2.5, "text": "Hello and welcome..."}
],
"text": "Full transcript...",
"note": "Use the Read tool to view frame images for visual understanding."
}
Use the Read tool on frame image paths to visually inspect extracted frames.
References
references/output-format.md -- Full JSON output schema documentation
1---
2name: video-understand
3description: |
4 Understand video content locally using ffmpeg frame extraction and Whisper transcription. No API keys needed.
5 Use when: (1) Understanding what a video contains, (2) Transcribing video audio locally,
6 (3) Extracting key frames for visual analysis, (4) Getting video content without API keys.
7metadata:
8 source: https://skillmd.com/skills/calesthio/video-understand
9---
10
11# video-understand
12
13Understand video content locally using ffmpeg for frame extraction and Whisper for transcription. Fully offline, no API keys required.
14
15## Prerequisites
16
17- `ffmpeg` + `ffprobe` (required): `brew install ffmpeg`
18- `openai-whisper` (optional, for transcription): `pip install openai-whisper`
19
20## Commands
21
22```bash
23# Scene detection + transcribe (default)
24python3 skills/video-understand/scripts/understand_video.py video.mp4
25
26# Keyframe extraction
27python3 skills/video-understand/scripts/understand_video.py video.mp4 -m keyframe
28
29# Regular interval extraction
30python3 skills/video-understand/scripts/understand_video.py video.mp4 -m interval
31
32# Limit frames extracted
33python3 skills/video-understand/scripts/understand_video.py video.mp4 --max-frames 10
34
35# Use a larger Whisper model
36python3 skills/video-understand/scripts/understand_video.py video.mp4 --whisper-model small
37
38# Frames only, skip transcription
39python3 skills/video-understand/scripts/understand_video.py video.mp4 --no-transcribe
40
41# Quiet mode (JSON only, no progress)
42python3 skills/video-understand/scripts/understand_video.py video.mp4 -q
43
44# Output to file
45python3 skills/video-understand/scripts/understand_video.py video.mp4 -o result.json
46```
47
48## CLI Options
49
50| Flag | Description |
51|------|-------------|
52| `video` | Input video file (positional, required) |
53| `-m, --mode` | Extraction mode: `scene` (default), `keyframe`, `interval` |
54| `--max-frames` | Maximum frames to keep (default: 20) |
55| `--whisper-model` | Whisper model size: tiny, base, small, medium, large (default: base) |
56| `--no-transcribe` | Skip audio transcription, extract frames only |
57| `-o, --output` | Write result JSON to file instead of stdout |
58| `-q, --quiet` | Suppress progress messages, output only JSON |
59
60## Extraction Modes
61
62| Mode | How it works | Best for |
63|------|-------------|----------|
64| `scene` | Detects scene changes via ffmpeg `select='gt(scene,0.3)'` | Most videos, varied content |
65| `keyframe` | Extracts I-frames (codec keyframes) | Encoded video with natural keyframe placement |
66| `interval` | Evenly spaced frames based on duration and max-frames | Fixed sampling, predictable output |
67
68If `scene` mode detects no scene changes, it automatically falls back to `interval` mode.
69
70## Output
71
72The script outputs JSON to stdout (or file with `-o`). See `references/output-format.md` for the full schema.
73
74```json
75{
76 "video": "video.mp4",
77 "duration": 18.076,
78 "resolution": {"width": 1224, "height": 1080},
79 "mode": "scene",
80 "frames": [
81 {"path": "/abs/path/frame_0001.jpg", "timestamp": 0.0, "timestamp_formatted": "00:00"}
82 ],
83 "frame_count": 12,
84 "transcript": [
85 {"start": 0.0, "end": 2.5, "text": "Hello and welcome..."}
86 ],
87 "text": "Full transcript...",
88 "note": "Use the Read tool to view frame images for visual understanding."
89}
90```
91
92Use the Read tool on frame image paths to visually inspect extracted frames.
93
94## References
95
96- `references/output-format.md` -- Full JSON output schema documentation