What this does
Turns a source video into an understanding index an agent (or a downstream stage) can read:
- Scene detection —
scenes.json (cut points, durations) + junk-scene filtering.
- Frame extraction — sampled frames for the visual analysis.
- ASR —
asr_result.json (timestamped dialogue) via MiMo mimo-v2.5-asr.
- Silence detection —
silence_periods.json (quiet windows, has_speech flag).
- VLM analysis —
vlm_analysis.json (per-scene description, depth analysis, frame_facts).
- Timeline fusion + brief —
timeline_fusion.json, asr_writing_chunks.json, agent_narration_brief.md.
Stateless: reusable stages are skipped only when their output and provenance sidecar match
the current source video plus output-affecting settings. --force recomputes.
Requirements
# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=*** # one key drives ASR (mimo-v2.5-asr) + VLM (mimo-v2.5)
ASR uses MiMo mimo-v2.5-asr; pass --skip-asr to skip dialogue transcription. The full understanding run still requires MIMO_API_KEY for VLM scene analysis.
Optional MiMo scene-chunk video understanding: --mimo-video-overview.
If work_dir/background_research.json exists (story research the agent did first, see
references/research-guide.md), its synopsis and named characters are folded into the VLM
context, so scene descriptions can name people and read scenes with plot knowledge. Combine with
--context for a quick inline hint.
Running the scripts below — the scripts/… paths are relative to this skill's own directory (the folder containing this SKILL.md). Claude Code runs commands from there, so they work as written. If your harness runs commands from the project root instead (opencode / Codex / OpenClaw commonly do), prefix this skill's absolute directory — e.g. <skill-dir>/scripts/…, using the directory your harness reports when it loads the skill. The scripts self-locate from their own path, so once started by the correct path they resolve their sibling skills and assets regardless of the working directory.
Run
python3 scripts/understand.py <video> --work-dir <work_dir> \
[--context "节目名/角色名"] [--scene-threshold 0.1] [--skip-asr] [--mimo-video-overview] [--force]
Output contract
| File |
Content |
scenes.json |
scene cut list (start/end/duration) |
asr_result.json |
[{start, end, text}] timestamped transcript |
vlm_analysis.json |
per-scene description / depth / frame_facts |
silence_periods.json |
[{start, end, duration, has_speech}] quiet windows |
timeline_fusion.json |
VLM + ASR + silence overlap, unified timeline |
asr_writing_chunks.json |
ASR split at sentence boundaries, scene-aligned |
agent_narration_brief.md |
the human/agent-facing writing brief (read this first) |
Downstream, video-script reads the brief + index to write narration.json.
References
- Background research before writing:
references/research-guide.md (writes background_research.json).
- Output JSON shapes:
references/data-schema.md.
What this skill does NOT do
- Does NOT write narration / 解说词 or score it — that is video-script.
- Does NOT cut, edit, voice, or render video.
- Does NOT invent plot the signal doesn't support — it emits a substrate warning when ASR/VLM are thin, rather than fabricating.
- Does NOT publish or schedule anything; it writes artifacts to work_dir and stops.
1---2name: video-understanding3description: Analyze a video into a structured understanding index: scene detection, ASR transcript, per-scene visual (VLM) analysis, silence windows, a fused timeline, and a narration-writing brief. Use to understand / index / summarize what happens in a video, or as the first stage of the video-recap bundle before writing narration. Input: a video file. Output: scenes.json, asr_result.json, vlm_analysis.json, silence_periods.json, timeline_fusion.json, agent_narration_brief.md. 触发词: 视频理解, 视频分析, 视频索引, video understanding, analyze video, 看懂视频.4---5
6## What this does
7
8Turns a source video into an **understanding index** an agent (or a downstream stage) can read:
91. **Scene detection** — `scenes.json` (cut points, durations) + junk-scene filtering.
102. **Frame extraction** — sampled frames for the visual analysis.
113. **ASR** — `asr_result.json` (timestamped dialogue) via MiMo `mimo-v2.5-asr`.
124. **Silence detection** — `silence_periods.json` (quiet windows, `has_speech` flag).
135. **VLM analysis** — `vlm_analysis.json` (per-scene description, depth analysis, `frame_facts`).
146. **Timeline fusion + brief** — `timeline_fusion.json`, `asr_writing_chunks.json`, `agent_narration_brief.md`.
15
16Stateless: reusable stages are skipped only when their output and provenance sidecar match
17the current source video plus output-affecting settings. `--force` recomputes.
18
19## Requirements
20
21```bash
22# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
23export MIMO_API_KEY=*** # one key drives ASR (mimo-v2.5-asr) + VLM (mimo-v2.5)
24```
25
26ASR uses MiMo `mimo-v2.5-asr`; pass `--skip-asr` to skip dialogue transcription. The full understanding run still requires `MIMO_API_KEY` for VLM scene analysis.
27Optional MiMo scene-chunk video understanding: `--mimo-video-overview`.
28
29If `work_dir/background_research.json` exists (story research the agent did first, see
30`references/research-guide.md`), its synopsis and named characters are folded into the VLM
31context, so scene descriptions can name people and read scenes with plot knowledge. Combine with
32`--context` for a quick inline hint.
33
34> **Running the scripts below** — the `scripts/…` paths are relative to this skill's own directory (the folder containing this `SKILL.md`). Claude Code runs commands from there, so they work as written. If your harness runs commands from the project root instead (opencode / Codex / OpenClaw commonly do), prefix this skill's absolute directory — e.g. `<skill-dir>/scripts/…`, using the directory your harness reports when it loads the skill. The scripts self-locate from their own path, so once started by the correct path they resolve their sibling skills and assets regardless of the working directory.
35
36## Run
37
38```bash
39python3 scripts/understand.py <video> --work-dir <work_dir> \
40 [--context "节目名/角色名"] [--scene-threshold 0.1] [--skip-asr] [--mimo-video-overview] [--force]
41```
42
43## Output contract
44
45| File | Content |
46|------|---------|
47| `scenes.json` | scene cut list (start/end/duration) |
48| `asr_result.json` | `[{start, end, text}]` timestamped transcript |
49| `vlm_analysis.json` | per-scene description / depth / `frame_facts` |
50| `silence_periods.json` | `[{start, end, duration, has_speech}]` quiet windows |
51| `timeline_fusion.json` | VLM + ASR + silence overlap, unified timeline |
52| `asr_writing_chunks.json` | ASR split at sentence boundaries, scene-aligned |
53| `agent_narration_brief.md` | the human/agent-facing writing brief (read this first) |
54
55Downstream, **video-script** reads the brief + index to write `narration.json`.
56
57## References
58- Background research before writing: `references/research-guide.md` (writes `background_research.json`).
59- Output JSON shapes: `references/data-schema.md`.
60
61## What this skill does NOT do
62- Does NOT write narration / 解说词 or score it — that is video-script.
63- Does NOT cut, edit, voice, or render video.
64- Does NOT invent plot the signal doesn't support — it emits a substrate warning when ASR/VLM are thin, rather than fabricating.
65- Does NOT publish or schedule anything; it writes artifacts to work_dir and stops.
majiayu000