Graph Memory — Long Video QA Retrieval
Check the qwen-mm-plugins-video-memory tools in your tool list for full schemas and parameters.
ROUTING RULE — Read This First
When a user asks about a video that is longer than 30 minutes, you MUST use this skill instead of read_video. read_video cannot meaningfully cover a multi-hour video — it samples too few frames once, which is ineffective and always misses important content. You MUST build memory first, then you can fast explore the whole long video with the help of your pre-built memory.
Workflow:
- Check if
<video_path>.memory/ directory exists
- Memory exists → query it using the
qwen-mm-plugins-video-memory tools
- Memory does not exist → build it first (see "Building Graph Memory" section), then query
- After locating a specific segment via memory tools, you MUST use
read_video with a narrow start_time/end_time for frame-level visual inspection, because memory is always coarse and maybe inaccurate.
Hierarchical Graph Memory
The memory is a 4-level tree. You start broad (Root/SuperEvent) and drill down to find specifics:
Root (1 per video)
└─ SuperEvent (10-20 story arcs)
└─ MacroEvent (3-8 per SuperEvent, each ~3-8 min)
└─ Subgraph (entities, events, OCR text, relations)
Level Details
| Level |
Contains |
IDs |
| Root |
Title, description, themes, key entities, emotional tone |
— |
| SuperEvent |
High-level narrative arc, time range, key entities |
super_01, super_02, ... |
| MacroEvent |
Specific event segment with time range, summary, key entities |
macro_0001, macro_0002, ... |
| Subgraph |
Entity nodes (PERSON/OBJECT/LOCATION), Event nodes (timestamped actions), OCR text (on-screen text), and Edges (SEMANTIC/CAUSAL/TEMPORAL/HIERARCHICAL/SPATIAL/IDENTITY relations) |
macro_0001:ent_001, ... |
Building Graph Memory
If graph_memory.json doesn't exist for a video, build it:
# Build memory for a single video (output: <video_path>.memory/)
bash script/build_memory/build_memory.sh /path/to/video.mp4
# Build with custom options
bash script/build_memory/build_memory.sh /path/to/video.mp4 \
--output-dir /path/to/output --model qwen3.7-plus --api-key "$DASHSCOPE_API_KEY"
Retrieval Workflow
Most questions follow a 2-step pattern:
Step 1: Locate (find the right macro_id)
Choose ONE entry point based on the question type:
| Question Type |
Entry Tool |
Example |
| References a specific time |
search_by_time(start_sec, end_sec) |
"What happens at 45:30?" |
| Asks about a person/action/event |
search_nodes(query) |
"What did the chef cook?" |
| Asks about on-screen text/scores |
search_ocr_text(query) |
"What was the final score?" |
| References a known story segment |
get_macro_events(super_id) |
"In the first quarter..." |
| Asks about event causality |
search_nodes(query) → get_subgraph |
"What caused the explosion?" |
Step 2: Drill Down (get full detail)
Call get_subgraph(macro_id) on the 1-2 most relevant macro_ids from Step 1. This returns ALL detail: entities, events, OCR text, and relations. Answer from this data.
Decision Flow
Question arrives
├─ Does the SuperEvent list already narrow it? → get_macro_events(super_id) → get_subgraph
├─ Time reference? → search_by_time → get_subgraph
├─ On-screen text/score? → search_ocr_text → get_subgraph (if needed)
├─ Person/action/event? → search_nodes → get_subgraph
└─ Causal/relational? → search_nodes → get_subgraph (relations are in the subgraph)
Tools Reference
Navigation (hierarchy drill-down)
get_summary — Video-level root summary.
- Params:
video_path (optional)
- Returns: title, description, themes, key entities, emotional tone
- Use when: you need a quick overview before drilling down
get_super_events — List all SuperEvents (high-level narrative arcs).
- Params:
video_path (optional)
- Returns: super_id, label, time_range, key_entities, description, num_macros
- Use when: you need the overall structure or to pick the relevant story arc
get_macro_events — List MacroEvents under a SuperEvent.
- Params:
super_id (optional), macro_ids (optional list)
- Returns: macro_id, time_range, label, key_entity_names, description
- Use when: initial context already narrows the segment
get_subgraph — Full detail for one MacroEvent. Most information-rich tool.
- Params:
macro_id (required)
- Returns: Entities, Events, OCR texts, Relations
- Use when: you have a macro_id and need specifics
Search (direct access)
search_nodes — Semantic search over Entity and Event nodes (excludes OCR).
- Params:
query (required), top_k, node_types (["event"], ["entity"], or both)
- Query guide: write a DESCRIPTIVE STATEMENT, not a question
- Good: "A player scores with an alley-oop dunk"
- Bad: "Which player scored the alley-oop?"
- For multi-choice: extract SHARED info across options, exclude differing details
search_ocr_text — Semantic search over on-screen text only.
- Params:
query (required), top_k
- Use for: scores, stats, jersey numbers, broadcast graphics
- Do NOT use search_nodes for OCR content — they are separate indexes
search_asr_text — Semantic search over ASR transcript text (speech, dialogue, narration).
- Params:
query (required), top_k
- Use for: spoken content, dialogue, narration, or other audio/verbal information
search_by_time — Find MacroEvents by time range.
- Params:
start_sec/end_sec (seconds), or start_time/end_time (DAY{N} HH:MM:SS, for EgoLife)
- Returns summaries only — follow up with get_subgraph for detail
enumerate_events — Enumerate ALL matching event instances in time order — built for COUNTING / "how many times / list every occurrence" questions.
- Params:
query (required; a descriptive statement of the target event), min_cosine, max_results, node_types (default ["event"]; use ["entity"] for objects/people)
- Unlike search_nodes (top-k best matches), returns every match above a similarity threshold, deduped and sorted by start time
- If undercounting, retry with a lower
min_cosine and alternative phrasings
Video Frame Extraction
read_video — Extract frames from the original video after narrowing the time range via memory tools. video_path is in the initial context.
Anti-Patterns (common mistakes)
- Answering from SuperEvent descriptions — these are summaries. Always get_subgraph for specific details.
- Repeating similar searches — if search_nodes found macro_0022, call get_subgraph(macro_0022) instead of searching with a slightly different query.
- Ignoring tool types — if search_nodes gives poor results, try search_ocr_text or search_by_time. Different indexes cover different content.
- Guessing counts — for "how many" questions, get_subgraph and count from the explicit node list. Never estimate from words like "several" or "multiple".
- Skipping tool calls — never answer without querying the memory graph first, even if the question seems simple.
Storage
Graph memory is stored next to the video, in <video_path>.memory/ (e.g.
/data/clip.mp4 → /data/clip.mp4.memory/). The server auto-loads it from there
when you pass video_path, or set GRAPH_MEMORY_PATH to point at one explicitly.
Contents: graph_memory.json, embeddings.npz, 01_macros.json, subgraphs/
Efficiency Rules
- Maximum 10 tool call rounds — use them wisely
- Most questions need: 1 search + 1-2 get_subgraph calls
- Always drill down before giving up — the answer is in the subgraph, not the summary
1---
2name: qwen-mm-plugins-video-memory
3description: "Triggered for long videos (30+ minutes), whether a single file or a directory of multiple videos. Vision-language MCP tools designed for efficient reading and semantic analysis of long videos (30+ minutes), supporting memory construction and semantic search."
4metadata:
5 source: https://skillmd.com/skills/theheavenlyd3mon/qwen-mm-plugins-video-memory
6---
7
8# Graph Memory — Long Video QA Retrieval
9
10Check the `qwen-mm-plugins-video-memory` tools in your tool list for full schemas and parameters.
11
12
13## ROUTING RULE — Read This First
14
15When a user asks about a video that is **longer than 30 minutes**, you MUST use this skill instead of `read_video`. `read_video` cannot meaningfully cover a multi-hour video — it samples too few frames once, which is ineffective and always misses important content. You MUST build memory first, then you can fast explore the whole long video with the help of your pre-built memory.
16
17**Workflow:**
181. Check if `<video_path>.memory/` directory exists
192. **Memory exists** → query it using the `qwen-mm-plugins-video-memory` tools
203. **Memory does not exist** → build it first (see "Building Graph Memory" section), then query
214. After locating a specific segment via memory tools, you MUST use `read_video` with a narrow `start_time`/`end_time` for frame-level visual inspection, because memory is always coarse and maybe inaccurate.
22
23## Hierarchical Graph Memory
24
25The memory is a 4-level tree. You start broad (Root/SuperEvent) and drill down to find specifics:
26
27```
28Root (1 per video)
29 └─ SuperEvent (10-20 story arcs)
30 └─ MacroEvent (3-8 per SuperEvent, each ~3-8 min)
31 └─ Subgraph (entities, events, OCR text, relations)
32```
33
34### Level Details
35
36| Level | Contains | IDs |
37|-------|----------|-----|
38| **Root** | Title, description, themes, key entities, emotional tone | — |
39| **SuperEvent** | High-level narrative arc, time range, key entities | `super_01`, `super_02`, ... |
40| **MacroEvent** | Specific event segment with time range, summary, key entities | `macro_0001`, `macro_0002`, ... |
41| **Subgraph** | Entity nodes (PERSON/OBJECT/LOCATION), Event nodes (timestamped actions), OCR text (on-screen text), and Edges (SEMANTIC/CAUSAL/TEMPORAL/HIERARCHICAL/SPATIAL/IDENTITY relations) | `macro_0001:ent_001`, ... |
42
43
44## Building Graph Memory
45
46If `graph_memory.json` doesn't exist for a video, build it:
47
48```bash
49# Build memory for a single video (output: <video_path>.memory/)
50bash script/build_memory/build_memory.sh /path/to/video.mp4
51
52# Build with custom options
53bash script/build_memory/build_memory.sh /path/to/video.mp4 \
54 --output-dir /path/to/output --model qwen3.7-plus --api-key "$DASHSCOPE_API_KEY"
55```
56
57
58## Retrieval Workflow
59
60Most questions follow a 2-step pattern:
61
62### Step 1: Locate (find the right macro_id)
63
64Choose ONE entry point based on the question type:
65
66| Question Type | Entry Tool | Example |
67|---|---|---|
68| References a specific time | `search_by_time(start_sec, end_sec)` | "What happens at 45:30?" |
69| Asks about a person/action/event | `search_nodes(query)` | "What did the chef cook?" |
70| Asks about on-screen text/scores | `search_ocr_text(query)` | "What was the final score?" |
71| References a known story segment | `get_macro_events(super_id)` | "In the first quarter..." |
72| Asks about event causality | `search_nodes(query)` → `get_subgraph` | "What caused the explosion?" |
73
74### Step 2: Drill Down (get full detail)
75
76Call `get_subgraph(macro_id)` on the 1-2 most relevant macro_ids from Step 1. This returns ALL detail: entities, events, OCR text, and relations. Answer from this data.
77
78### Decision Flow
79
80```
81Question arrives
82 ├─ Does the SuperEvent list already narrow it? → get_macro_events(super_id) → get_subgraph
83 ├─ Time reference? → search_by_time → get_subgraph
84 ├─ On-screen text/score? → search_ocr_text → get_subgraph (if needed)
85 ├─ Person/action/event? → search_nodes → get_subgraph
86 └─ Causal/relational? → search_nodes → get_subgraph (relations are in the subgraph)
87```
88
89## Tools Reference
90
91### Navigation (hierarchy drill-down)
92
93**get_summary** — Video-level root summary.
94- Params: `video_path` (optional)
95- Returns: title, description, themes, key entities, emotional tone
96- Use when: you need a quick overview before drilling down
97
98**get_super_events** — List all SuperEvents (high-level narrative arcs).
99- Params: `video_path` (optional)
100- Returns: super_id, label, time_range, key_entities, description, num_macros
101- Use when: you need the overall structure or to pick the relevant story arc
102
103**get_macro_events** — List MacroEvents under a SuperEvent.
104- Params: `super_id` (optional), `macro_ids` (optional list)
105- Returns: macro_id, time_range, label, key_entity_names, description
106- Use when: initial context already narrows the segment
107
108**get_subgraph** — Full detail for one MacroEvent. **Most information-rich tool.**
109- Params: `macro_id` (required)
110- Returns: Entities, Events, OCR texts, Relations
111- Use when: you have a macro_id and need specifics
112
113### Search (direct access)
114
115**search_nodes** — Semantic search over Entity and Event nodes (excludes OCR).
116- Params: `query` (required), `top_k`, `node_types` (`["event"]`, `["entity"]`, or both)
117- Query guide: write a DESCRIPTIVE STATEMENT, not a question
118 - Good: "A player scores with an alley-oop dunk"
119 - Bad: "Which player scored the alley-oop?"
120 - For multi-choice: extract SHARED info across options, exclude differing details
121
122**search_ocr_text** — Semantic search over on-screen text only.
123- Params: `query` (required), `top_k`
124- Use for: scores, stats, jersey numbers, broadcast graphics
125- Do NOT use search_nodes for OCR content — they are separate indexes
126
127**search_asr_text** — Semantic search over ASR transcript text (speech, dialogue, narration).
128- Params: `query` (required), `top_k`
129- Use for: spoken content, dialogue, narration, or other audio/verbal information
130
131**search_by_time** — Find MacroEvents by time range.
132- Params: `start_sec`/`end_sec` (seconds), or `start_time`/`end_time` (`DAY{N} HH:MM:SS`, for EgoLife)
133- Returns summaries only — follow up with get_subgraph for detail
134
135**enumerate_events** — Enumerate ALL matching event instances in time order — built for COUNTING / "how many times / list every occurrence" questions.
136- Params: `query` (required; a descriptive statement of the target event), `min_cosine`, `max_results`, `node_types` (default `["event"]`; use `["entity"]` for objects/people)
137- Unlike search_nodes (top-k best matches), returns every match above a similarity threshold, deduped and sorted by start time
138- If undercounting, retry with a lower `min_cosine` and alternative phrasings
139
140### Video Frame Extraction
141
142**read_video** — Extract frames from the original video after narrowing the time range via memory tools. `video_path` is in the initial context.
143
144## Anti-Patterns (common mistakes)
145
1461. **Answering from SuperEvent descriptions** — these are summaries. Always get_subgraph for specific details.
1472. **Repeating similar searches** — if search_nodes found macro_0022, call get_subgraph(macro_0022) instead of searching with a slightly different query.
1483. **Ignoring tool types** — if search_nodes gives poor results, try search_ocr_text or search_by_time. Different indexes cover different content.
1494. **Guessing counts** — for "how many" questions, get_subgraph and count from the explicit node list. Never estimate from words like "several" or "multiple".
1505. **Skipping tool calls** — never answer without querying the memory graph first, even if the question seems simple.
151
152## Storage
153
154Graph memory is stored next to the video, in `<video_path>.memory/` (e.g.
155`/data/clip.mp4` → `/data/clip.mp4.memory/`). The server auto-loads it from there
156when you pass `video_path`, or set `GRAPH_MEMORY_PATH` to point at one explicitly.
157
158Contents: `graph_memory.json`, `embeddings.npz`, `01_macros.json`, `subgraphs/`
159
160## Efficiency Rules
161
162- Maximum 10 tool call rounds — use them wisely
163- Most questions need: 1 search + 1-2 get_subgraph calls
164- Always drill down before giving up — the answer is in the subgraph, not the summary