# Video2knowledge > Convert videos into timestamped subtitles and structured knowledge artifacts. Two ingestion paths: (1) a native multimodal VLM (at most 4B params, via Ollama) reads sampled video frames into timestamped captions; (2) faster-whisper ASR transcribes the audio into timestamped subtitles. Then refine into a structured knowledge doc (custom template supported), a self-contained HTML page, a knowledge-card CSV, and an Anki apkg deck. Fully local — no video, audio, or output ever leaves the host; the repo tracks only code and config changes. Use when transcribing or summarizing a video, building study cards from a lecture or recording, turning a silent or screen-recording video into notes, or producing reviewable knowledge from any video file. - Skill: `caciniep/video2knowledge` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add caciniep/video2knowledge` - Raw SKILL.md: https://api.skillmd.com/api/skills/caciniep/video2knowledge/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Web & Frontend - Author: CacinieP (https://skillmd.com/u/caciniep) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/caciniep/video2knowledge --- # Video2Knowledge Turn a video into: **timestamped subtitles → knowledge doc → HTML / Anki cards**. Two local ingestion paths, all-local inference (no cloud API). **Processing stays fully local** — videos, subtitles, and outputs are never uploaded; they live in a local `runs/` folder (gitignored). The repo tracks only code/config changes. ## Prerequisites & First-Time Setup Required on the host: `ollama`, `ffmpeg`, `python3` (or `uv`). One-time: ```bash bash scripts/setup_models.sh ``` This (idempotently) **auto-detects your machine's hardware profile** (RAM / GPU / Apple Silicon / NVIDIA) via `scripts/hardware_profile.py`, then pulls the recommended VLM, creates a venv at `.venv/`, and installs `faster-whisper` + `genanki`. See what it picked: ```bash python3 scripts/hardware_profile.py ``` Profiles range from `tiny` (4 GB machines → whisper-tiny + qwen3.5:0.8b) through `high-gpu` (NVIDIA ≥8 GB → whisper-large-v3 + qwen3.5:9b on CUDA). Full table and tuning in `references/hardware-profiles.md`. Override any choice with env vars (`VLM_MODEL=`, `ASR_DEFAULT_MODEL=`, ...) or CLI flags. Activate the venv before running any python step: ```bash source .venv/bin/activate ``` ## Choose an Ingestion Path (Step 1) Decision tree: - **Clear speech track** (lecture, talk, interview, narration) → **Path 2 (ASR)**. Faster, more accurate text, fine-grained timestamps. → `references/path2-asr.md` - **Silent / slide-only / screen recording**, or you need on-screen text & diagrams → **Path 1 (multimodal)**. → `references/path1-multimodal.md` - **Slide/PPT video with a speech track** (the common online-course case): the core content is on screen (tables, formulas, examples) AND the speaker narrates → **Path 3 (dual-path fusion)**. Runs ASR for narration + VLM OCR for slides and fuses them by timestamp. → `references/path3-fusion.md` Paths 1 & 2 emit the same segment schema (`{start,end,text}`), so Step 2 is path-agnostic. Path 3 emits a fused `merged.json` consumed by Step 2's `--merged`. ### Path 1 — Multimodal captions ```bash python3 scripts/mm_caption.py \ --video VIDEO --out-dir OUT \ --model openbmb/minicpm-v4.6:latest --interval 2.0 ``` Outputs: `OUT/captions.srt`, `OUT/captions.json`, `OUT/frames/`. For slide/PPT videos, use dedup sampling (general-purpose, no threshold tuning) plus the table/formula OCR prompt: ```bash python3 scripts/mm_caption.py \ --video VIDEO --out-dir OUT \ --mode dedup --prompt-ocr ``` ### Path 2 — ASR transcription ```bash python3 scripts/asr_caption.py \ --video VIDEO --out-dir OUT \ --model small --language zh \ --hotwords "术语一, 术语二" # optional jargon biasing via initial_prompt ``` Outputs: `OUT/subtitles.{srt,vtt,json}`. ### Path 3 — Dual-path fusion (ASR × VLM) Run Path 2 then Path 1 (OCR), feed OCR terms back into ASR hotwords, fuse, and build from the merged file: ```bash # 1a. ASR python3 scripts/asr_caption.py --video VIDEO --out-dir OUT --language zh # 1b/1c. dedup frames + VLM OCR python3 scripts/mm_caption.py --video VIDEO --out-dir OUT --mode dedup --prompt-ocr # 1d. OCR terms -> hotwords + coverage check (re-run ASR with # --hotwords @OUT/ocr_hotwords.txt when coverage is poor) python3 scripts/hotwords_from_ocr.py --captions OUT/captions.json \ --subtitles OUT/subtitles.json --out OUT/ocr_hotwords.txt # 2. fuse by timestamp + semantic alignment check python3 scripts/merge_visual.py --subtitles OUT/subtitles.json --visual OUT/captions.json --out OUT/merged.json # 3. build (Step 2 with --merged) python3 scripts/build_knowledge.py --subtitles OUT/subtitles.json --merged OUT/merged.json --out-dir OUT --format all ``` The fusion loop has three cross-path pieces: (1) `hotwords_from_ocr.py` extracts on-screen terms (recurring CJK n-grams, slide titles, Latin acronyms — heuristic, no LLM) and reports which ones the ASR never heard, the likely mis-heard jargon; (2) `merge_visual.py` attaches visuals by timestamp, then conservatively re-binds a narration line to an ADJACENT slide when its word overlap with the timestamp-attached slide is near zero and the neighbour matches clearly better (speaker lag / timestamp drift), flagging uncertain cases as `weak` instead of guessing — except on SETTLED slides (≥60 s on screen), where verbal elaboration is the norm and no flag is raised (`--no-semantic` restores pure timestamp matching); (3) `build_knowledge.py --merged` feeds the LLM interleaved audio+visual text (chunk boundaries never cut a slide table; prompt-echo lines are fingerprint-filtered from every list field; field caps scale with video duration, 3× at 3 h) with ⚠️ markers on swap/weak notes, and re-ranks map-reduced list fields in one global pass with the summary as context. Chunk-level LLM responses are cached in `build_cache.json`, so an interrupted 3-hour build resumes instead of restarting. In batches, `batch_run.py` accumulates OCR terms into `course_hotwords.txt` so later videos transcribe better, and `--asr-verify` re-transcribes a video whose OCR-term coverage fell below 50 %. The default text model is `qwen3.5:4b` (unified vision+text — on `high` machines the same pull serves Path 1 and Step 2; override with `--model openbmb/minicpm5:Q4_K_M` for low-RAM/fast runs). Quality measurement: `scripts/recall_check.py --run-dir --draft` emits a golden must-have list (terms / both-channel numbers / timeline coverage) that a human prunes in minutes; `--golden` then scores produced artifacts against it (baselines in `tests/golden/BASELINE.md`) — "missing content" becomes a number you can track across parameter changes. ## Refine into Knowledge Artifacts (Step 2) Point `build_knowledge.py` at either path's `.json` output: ```bash python3 scripts/build_knowledge.py \ --subtitles OUT/subtitles.json \ --out-dir OUT --format all ``` Produces, in `OUT/`: | Artifact | File | Section | |---|---|---| | 2.1 Knowledge doc (templated) | `knowledge.md` | uses `assets/default-template.md` or `--template ` | | 2.2 Self-contained HTML | `knowledge.html` | clickable `[mm:ss]` timeline | | 2.2b Office/print export | `knowledge.docx` / `knowledge.pdf` | `--format docx` / `--format pdf` (auto in `--format all` when python-docx/fpdf2 installed) | | 2.3 Knowledge cards | `cards.csv` | `question,answer,tags,timestamp,source` | | 2.4 Illustrated notes (图文笔记) | `notes.md` + `notes.html` | key frames × narration, see below | | 2.4b Notes office/print | `notes.docx` / `notes.pdf` | `build_notes.py --docx --pdf` (frames embedded) | Then convert cards to Anki (2.3 final): ```bash python3 scripts/gen_apkg.py --csv OUT/cards.csv --out OUT/cards.apkg --deck "视频知识卡" ``` ### Illustrated notes (2.4, 图文笔记) A scrollable note that interleaves deduped KEY FRAMES with the narration around each timestamp — the "watch it back as a reading" artifact. Needs `frames.json` (from `extract_frames.py`, usually via Path 1's dedup mode) plus either path's subtitles: ```bash python3 scripts/build_notes.py \ --subtitles OUT/subtitles.json \ --frames OUT/frames/frames.json \ --out-dir OUT --max-frames 12 --describe-frames ``` Per key frame: LLM section title → frame image → optional VLM 画面 description (`--describe-frames`) → 1-2 sentence note condensed from the narration window → verbatim 原声 excerpt. Emits `notes.md` (relative image refs, renders anywhere the frames dir travels) and `notes.html` (frames embedded as base64 — a single shareable file). Degrades to frames + raw excerpts when no model is reachable. See `references/outputs.md` for schema, single-format runs, and degraded mode. ## Custom Templates (2.1) Default template: `assets/default-template.md`. Override with `--template `: ```bash python3 scripts/build_knowledge.py --subtitles s.json --out-dir o \ --template ./my-lecture-template.md --format knowledge ``` Templates are plain Markdown using `{{placeholders}}` (`{{title}}`, `{{summary}}`, `{{bullets}}` (音画合并要点速览), `{{timeline}}`, `{{key_points}}`, `{{qa}}`, `{{glossary}}`, `{{source}}`, `{{duration}}`, `{{date}}`, `{{meta}}`). Only placeholders you include are filled; everything else stays verbatim. Full spec + 3 example templates (lecture / meeting / tutorial) in `references/templates.md`. ## Local-Only Processing & Change Tracking **All video processing stays on your machine.** Videos, extracted frames, subtitles, knowledge docs, and cards are written to a local `runs/` folder that is **gitignored** — nothing about your media is ever uploaded or committed. The GitHub repo tracks **only code and config changes** (scripts, references, templates, README). This gives a clear history of how the skill evolved, without exposing any user's media. Use descriptive `feat:`/`fix:`/`docs:` commit messages. If you want a local record of a specific run, write a `manifest.json` into that run folder (path taken, models, args, output list) — but keep it local: ```bash cat > "$RUN/manifest.json" < "$RUN/manifest.json" <