Video Benchmarks

Same video with same fps, max_frames, and prompt will hit cache. Changing any parameter causes a miss.

tools-only Updated Sep 29, 2026 7 repo stars

Coding & Dev Tools

Install

Press C to copy

$ npx skillmds@latest add tools-only/video-benchmarks

Installs into ~/.claude/skills (or .claude/skills inside a project). Restart Claude Code to pick it up.

＋ Add to plugin

README badge

[![SkillMD](https://skillmd.com/badge/tools-only/video-benchmarks.svg)](https://skillmd.com/skills/tools-only/video-benchmarks)

Trust & safety

2 of 3 audits passed

Last audited Sep 23, 2026 · commit cc6236e

Works with: Claude Code Claude.ai OpenAI Codex +

File contents

Video Benchmarks

Running Video Benchmarks

# Full benchmark (10 configurations, 2-64 frames)
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video

# Quick benchmark (3 frame counts)
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick

# Custom video
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4

Results - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB)

Configuration	Frames	Time	Tokens	Speed	Memory
2 frames @ 0.5fps	2	4.48s	256	57.1 tok/s	6.4 GB
4 frames @ 1fps	4	4.65s	256	55.0 tok/s	6.4 GB
6 frames @ 1fps	6	5.15s	197	38.2 tok/s	6.6 GB
8 frames @ 2fps	8	6.45s	240	37.2 tok/s	6.8 GB
12 frames @ 2fps	12	8.73s	256	29.3 tok/s	7.1 GB
16 frames @ 2fps	16	10.96s	256	23.4 tok/s	7.6 GB
24 frames @ 4fps	24	14.95s	226	15.1 tok/s	8.4 GB
32 frames @ 4fps	32	20.00s	256	12.8 tok/s	9.2 GB
48 frames @ 8fps	48	31.11s	246	7.9 tok/s	11.1 GB
64 frames @ 8fps	64	59.81s	256	4.3 tok/s	12.9 GB

Summary: Fastest at 2 frames (57.1 tok/s), slowest at 64 frames (4.3 tok/s). Memory scales from 6.4 GB to 12.9 GB.

Note: 96+ frames causes GPU timeout on most hardware due to memory/compute limits

Results - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB)

Configuration	Frames	FPS	Time	Tokens	Speed
4 frames @ 1fps	4	1.0	8.84s	256	29.0 tok/s
8 frames @ 2fps	8	2.0	13.05s	256	19.6 tok/s
16 frames @ 2fps	16	2.0	21.60s	256	11.9 tok/s

Configs	Total Time (s)	Total Tokens	Aggregate Tok/s
3	43.48	768	17.7

Results - Qwen3-VL-4B-Instruct-3bit (M1 Max, 64GB)

Configuration	Frames	FPS	Time	Tokens	Speed
4 frames @ 1fps	4	1.0	5.09s	150	29.5 tok/s
8 frames @ 2fps	8	2.0	8.36s	150	17.9 tok/s
16 frames @ 2fps	16	2.0	15.21s	150	9.9 tok/s

Configs	Total Time (s)	Total Tokens	Aggregate Tok/s
3	28.66	450	15.7

Video Cache Results

----------------------------------------------------------------------
  TEST 4: Video Cache - fps/max_frames in Cache Key
----------------------------------------------------------------------
    Config: fps=2.0, max_frames=16

    Results:
    Step   | Description               | Expected | Actual | Time   | Status
    -------+---------------------------+----------+--------+--------+-------
    4a     | Video first request       | MISS     | MISS   | 0.03ms | ✓
    4b     | Same video+params         | HIT      | HIT    | 0.14ms | ✓
    4c     | Different fps (4.0)       | MISS     | MISS   | 0.01ms | ✓
    4d     | Different max_frames (32) | MISS     | MISS   | 0.01ms | ✓
    4.0.5a | fps=0.5 first             | MISS     | MISS   | 0.01ms | ✓
    4.0.5b | fps=0.5 cached            | HIT      | HIT    | 0.14ms | ✓
    4.1.0a | fps=1.0 first             | MISS     | MISS   | 0.01ms | ✓
    4.1.0b | fps=1.0 cached            | HIT      | HIT    | 0.14ms | ✓
    4.2.0a | fps=2.0 first             | MISS     | MISS   | 0.01ms | ✓
    4.2.0b | fps=2.0 cached            | HIT      | HIT    | 0.14ms | ✓
    4.4.0a | fps=4.0 first             | MISS     | MISS   | 0.01ms | ✓
    4.4.0b | fps=4.0 cached            | HIT      | HIT    | 0.14ms | ✓

----------------------------------------------------------------------
  TEST 5: Additional Videos
----------------------------------------------------------------------
    Results:
    Step   | Description               | Expected | Actual | Time   | Status
    -------+---------------------------+----------+--------+--------+-------
    5a     | Video 1 first             | MISS     | MISS   | 0.01ms | ✓
    5b     | Video 2 first             | MISS     | MISS   | 0.01ms | ✓
    5c     | Video 1 cached            | HIT      | HIT    | 0.13ms | ✓
    5d     | Video 2 cached            | HIT      | HIT    | 0.13ms | ✓

Cache Key Strategy

Videos: hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)

Same video with same fps, max_frames, and prompt will hit cache. Changing any parameter causes a miss.

Performance Tips

Lower FPS = faster processing
Fewer frames = less memory usage
64 frames is practical maximum
96+ frames causes GPU timeout

Frame Extraction

FPS	10s Video	30s Video	60s Video
0.5	5 frames	15 frames	30 frames
1.0	10 frames	30 frames	60 frames
2.0	20 frames	60 frames	120 frames*
4.0	40 frames	120 frames*	240 frames*

*May hit max_frames limit

Metrics Reference

Metric	Description
Configuration	FPS and max frames settings
Frames	Actual frames extracted
Time	Total generation time
Tokens	Output tokens generated
Speed	Tokens per second (tok/s)
Memory	GPU memory usage

---name: video-benchmarksdescription: Same video with same fps, max_frames, and prompt will hit cache. Changing any parameter causes a miss.---# Video Benchmarks## Running Video Benchmarks```bash# Full benchmark (10 configurations, 2-64 frames)vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video# Quick benchmark (3 frame counts)vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick# Custom videovllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4```## Results - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB)| Configuration | Frames | Time | Tokens | Speed | Memory ||---------------|--------|------|--------|-------|--------|| 2 frames @ 0.5fps | 2 | 4.48s | 256 | 57.1 tok/s | 6.4 GB || 4 frames @ 1fps | 4 | 4.65s | 256 | 55.0 tok/s | 6.4 GB || 6 frames @ 1fps | 6 | 5.15s | 197 | 38.2 tok/s | 6.6 GB || 8 frames @ 2fps | 8 | 6.45s | 240 | 37.2 tok/s | 6.8 GB || 12 frames @ 2fps | 12 | 8.73s | 256 | 29.3 tok/s | 7.1 GB || 16 frames @ 2fps | 16 | 10.96s | 256 | 23.4 tok/s | 7.6 GB || 24 frames @ 4fps | 24 | 14.95s | 226 | 15.1 tok/s | 8.4 GB || 32 frames @ 4fps | 32 | 20.00s | 256 | 12.8 tok/s | 9.2 GB || 48 frames @ 8fps | 48 | 31.11s | 246 | 7.9 tok/s | 11.1 GB || 64 frames @ 8fps | 64 | 59.81s | 256 | 4.3 tok/s | 12.9 GB |**Summary:** Fastest at 2 frames (57.1 tok/s), slowest at 64 frames (4.3 tok/s). Memory scales from 6.4 GB to 12.9 GB.> **Note:** 96+ frames causes GPU timeout on most hardware due to memory/compute limits## Results - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB)| Configuration | Frames | FPS | Time | Tokens | Speed ||---------------|--------|-----|------|--------|-------|| 4 frames @ 1fps | 4 | 1.0 | 8.84s | 256 | 29.0 tok/s || 8 frames @ 2fps | 8 | 2.0 | 13.05s | 256 | 19.6 tok/s || 16 frames @ 2fps | 16 | 2.0 | 21.60s | 256 | 11.9 tok/s || Configs | Total Time (s) | Total Tokens | Aggregate Tok/s ||---------|-----------------|--------------|-----------------|| 3 | 43.48 | 768 | 17.7 |## Results - Qwen3-VL-4B-Instruct-3bit (M1 Max, 64GB)| Configuration | Frames | FPS | Time | Tokens | Speed ||---------------|--------|-----|------|--------|-------|| 4 frames @ 1fps | 4 | 1.0 | 5.09s | 150 | 29.5 tok/s || 8 frames @ 2fps | 8 | 2.0 | 8.36s | 150 | 17.9 tok/s || 16 frames @ 2fps | 16 | 2.0 | 15.21s | 150 | 9.9 tok/s || Configs | Total Time (s) | Total Tokens | Aggregate Tok/s ||---------|-----------------|--------------|-----------------|| 3 | 28.66 | 450 | 15.7 |## Video Cache Results```---------------------------------------------------------------------- TEST 4: Video Cache - fps/max_frames in Cache Key---------------------------------------------------------------------- Config: fps=2.0, max_frames=16 Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 4a | Video first request | MISS | MISS | 0.03ms | ✓ 4b | Same video+params | HIT | HIT | 0.14ms | ✓ 4c | Different fps (4.0) | MISS | MISS | 0.01ms | ✓ 4d | Different max_frames (32) | MISS | MISS | 0.01ms | ✓ 4.0.5a | fps=0.5 first | MISS | MISS | 0.01ms | ✓ 4.0.5b | fps=0.5 cached | HIT | HIT | 0.14ms | ✓ 4.1.0a | fps=1.0 first | MISS | MISS | 0.01ms | ✓ 4.1.0b | fps=1.0 cached | HIT | HIT | 0.14ms | ✓ 4.2.0a | fps=2.0 first | MISS | MISS | 0.01ms | ✓ 4.2.0b | fps=2.0 cached | HIT | HIT | 0.14ms | ✓ 4.4.0a | fps=4.0 first | MISS | MISS | 0.01ms | ✓ 4.4.0b | fps=4.0 cached | HIT | HIT | 0.14ms | ✓---------------------------------------------------------------------- TEST 5: Additional Videos---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 5a | Video 1 first | MISS | MISS | 0.01ms | ✓ 5b | Video 2 first | MISS | MISS | 0.01ms | ✓ 5c | Video 1 cached | HIT | HIT | 0.13ms | ✓ 5d | Video 2 cached | HIT | HIT | 0.13ms | ✓```## Cache Key Strategy- **Videos**: `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)`Same video with same fps, max_frames, and prompt will hit cache. Changing any parameter causes a miss.## Performance Tips- Lower FPS = faster processing- Fewer frames = less memory usage- 64 frames is practical maximum- 96+ frames causes GPU timeout## Frame Extraction| FPS | 10s Video | 30s Video | 60s Video ||-----|-----------|-----------|-----------|| 0.5 | 5 frames | 15 frames | 30 frames || 1.0 | 10 frames | 30 frames | 60 frames || 2.0 | 20 frames | 60 frames | 120 frames* || 4.0 | 40 frames | 120 frames* | 240 frames* |*May hit `max_frames` limit## Metrics Reference| Metric | Description ||--------|-------------|| Configuration | FPS and max frames settings || Frames | Actual frames extracted || Time | Total generation time || Tokens | Output tokens generated || Speed | Tokens per second (tok/s) || Memory | GPU memory usage |

tools-only/X-Skills/tree/main/daily-assistant/2477-video_26f5b640 commit cc6236e18e

Frequently asked questions

npx skillmds@latest add tools-only/video-benchmarks