Video2knowledge

Video2Knowledge

Turn a video into: timestamped subtitles → knowledge doc → HTML / Anki cards. Two local ingestion paths, all-local inference (no cloud API). Processing stays fully local — videos, subtitles, and outputs are never uploaded; they live in a local runs/ folder (gitignored). The repo tracks only code/config changes.

Prerequisites & First-Time Setup

Required on the host: ollama, ffmpeg, python3 (or uv). One-time:

bash scripts/setup_models.sh

This (idempotently) auto-detects your machine's hardware profile (RAM / GPU / Apple Silicon / NVIDIA) via scripts/hardware_profile.py, then pulls the recommended VLM, creates a venv at .venv/, and installs faster-whisper + genanki. See what it picked:

python3 scripts/hardware_profile.py

Profiles range from tiny (4 GB machines → whisper-tiny + qwen3.5:0.8b) through high-gpu (NVIDIA ≥8 GB → whisper-large-v3 + qwen3.5:9b on CUDA). Full table and tuning in references/hardware-profiles.md. Override any choice with env vars (VLM_MODEL=, ASR_DEFAULT_MODEL=, ...) or CLI flags.

Activate the venv before running any python step:

source .venv/bin/activate

Choose an Ingestion Path (Step 1)

Decision tree:

• Clear speech track (lecture, talk, interview, narration) → Path 2 (ASR). Faster, more accurate text, fine-grained timestamps. → references/path2-asr.md
• Silent / slide-only / screen recording, or you need on-screen text & diagrams → Path 1 (multimodal). → references/path1-multimodal.md
• Slide/PPT video with a speech track (the common online-course case): the core content is on screen (tables, formulas, examples) AND the speaker narrates → Path 3 (dual-path fusion). Runs ASR for narration + VLM OCR for slides and fuses them by timestamp. → references/path3-fusion.md

Paths 1 & 2 emit the same segment schema ({start,end,text}), so Step 2 is path-agnostic. Path 3 emits a fused merged.json consumed by Step 2's --merged.

Path 1 — Multimodal captions

python3 scripts/mm_caption.py \
  --video VIDEO --out-dir OUT \
  --model openbmb/minicpm-v4.6:latest --interval 2.0

Outputs: OUT/captions.srt, OUT/captions.json, OUT/frames/.

For slide/PPT videos, use dedup sampling (general-purpose, no threshold tuning) plus the table/formula OCR prompt:

python3 scripts/mm_caption.py \
  --video VIDEO --out-dir OUT \
  --mode dedup --prompt-ocr

Path 2 — ASR transcription

python3 scripts/asr_caption.py \
  --video VIDEO --out-dir OUT \
  --model small --language zh \
  --hotwords "术语一, 术语二"   # optional jargon biasing via initial_prompt

Outputs: OUT/subtitles.{srt,vtt,json}.

Path 3 — Dual-path fusion (ASR × VLM)

Run Path 2 then Path 1 (OCR), feed OCR terms back into ASR hotwords, fuse, and build from the merged file:

# 1a. ASR
python3 scripts/asr_caption.py --video VIDEO --out-dir OUT --language zh
# 1b/1c. dedup frames + VLM OCR
python3 scripts/mm_caption.py --video VIDEO --out-dir OUT --mode dedup --prompt-ocr
# 1d. OCR terms -> hotwords + coverage check (re-run ASR with
#     --hotwords @OUT/ocr_hotwords.txt when coverage is poor)
python3 scripts/hotwords_from_ocr.py --captions OUT/captions.json \
  --subtitles OUT/subtitles.json --out OUT/ocr_hotwords.txt
# 2. fuse by timestamp + semantic alignment check
python3 scripts/merge_visual.py --subtitles OUT/subtitles.json --visual OUT/captions.json --out OUT/merged.json
# 3. build (Step 2 with --merged)
python3 scripts/build_knowledge.py --subtitles OUT/subtitles.json --merged OUT/merged.json --out-dir OUT --format all

The fusion loop has three cross-path pieces: (1) hotwords_from_ocr.py extracts on-screen terms (recurring CJK n-grams, slide titles, Latin acronyms — heuristic, no LLM) and reports which ones the ASR never heard, the likely mis-heard jargon; (2) merge_visual.py attaches visuals by timestamp, then conservatively re-binds a narration line to an ADJACENT slide when its word overlap with the timestamp-attached slide is near zero and the neighbour matches clearly better (speaker lag / timestamp drift), flagging uncertain cases as weak instead of guessing — except on SETTLED slides (≥60 s on screen), where verbal elaboration is the norm and no flag is raised (--no-semantic restores pure timestamp matching); (3) build_knowledge.py --merged feeds the LLM interleaved audio+visual text (chunk boundaries never cut a slide table; prompt-echo lines are fingerprint-filtered from every list field; field caps scale with video duration, 3× at 3 h) with ⚠️ markers on swap/weak notes, and re-ranks map-reduced list fields in one global pass with the summary as context. Chunk-level LLM responses are cached in build_cache.json, so an interrupted 3-hour build resumes instead of restarting. In batches, batch_run.py accumulates OCR terms into course_hotwords.txt so later videos transcribe better, and --asr-verify re-transcribes a video whose OCR-term coverage fell below 50 %. The default text model is qwen3.5:4b (unified vision+text — on high machines the same pull serves Path 1 and Step 2; override with --model openbmb/minicpm5:Q4_K_M for low-RAM/fast runs). Quality measurement: scripts/recall_check.py --run-dir <dir> --draft emits a golden must-have list (terms / both-channel numbers / timeline coverage) that a human prunes in minutes; --golden then scores produced artifacts against it (baselines in tests/golden/BASELINE.md) — "missing content" becomes a number you can track across parameter changes.

Refine into Knowledge Artifacts (Step 2)

Point build_knowledge.py at either path's .json output:

python3 scripts/build_knowledge.py \
  --subtitles OUT/subtitles.json \
  --out-dir OUT --format all

Produces, in OUT/:

Then convert cards to Anki (2.3 final):

python3 scripts/gen_apkg.py --csv OUT/cards.csv --out OUT/cards.apkg --deck "视频知识卡"

Illustrated notes (2.4, 图文笔记)

A scrollable note that interleaves deduped KEY FRAMES with the narration around each timestamp — the "watch it back as a reading" artifact. Needs frames.json (from extract_frames.py, usually via Path 1's dedup mode) plus either path's subtitles:

python3 scripts/build_notes.py \
  --subtitles OUT/subtitles.json \
  --frames OUT/frames/frames.json \
  --out-dir OUT --max-frames 12 --describe-frames

Per key frame: LLM section title → frame image → optional VLM 画面 description (--describe-frames) → 1-2 sentence note condensed from the narration window → verbatim 原声 excerpt. Emits notes.md (relative image refs, renders anywhere the frames dir travels) and notes.html (frames embedded as base64 — a single shareable file). Degrades to frames + raw excerpts when no model is reachable.

See references/outputs.md for schema, single-format runs, and degraded mode.

Custom Templates (2.1)

Default template: assets/default-template.md. Override with --template <file>:

python3 scripts/build_knowledge.py --subtitles s.json --out-dir o \
  --template ./my-lecture-template.md --format knowledge

Templates are plain Markdown using {{placeholders}} ({{title}}, {{summary}}, {{bullets}} (音画合并要点速览), {{timeline}}, {{key_points}}, {{qa}}, {{glossary}}, {{source}}, {{duration}}, {{date}}, {{meta}}). Only placeholders you include are filled; everything else stays verbatim. Full spec + 3 example templates (lecture / meeting / tutorial) in references/templates.md.

Local-Only Processing & Change Tracking

All video processing stays on your machine. Videos, extracted frames, subtitles, knowledge docs, and cards are written to a local runs/ folder that is gitignored — nothing about your media is ever uploaded or committed.

The GitHub repo tracks only code and config changes (scripts, references, templates, README). This gives a clear history of how the skill evolved, without exposing any user's media. Use descriptive feat:/fix:/docs: commit messages.

If you want a local record of a specific run, write a manifest.json into that run folder (path taken, models, args, output list) — but keep it local:

cat > "$RUN/manifest.json" <<EOF
{"video":"~/Movies/lecture.mp4","path":"2","model":"small",
 "outputs":["subtitles.srt","knowledge.md","knowledge.html","cards.csv","cards.apkg"]}
EOF

End-to-End Example

ASR path on a lecture, full pipeline (everything stays local):

source .venv/bin/activate
RUN=runs/$(date +%Y%m%d-%HMMSS)-lecture
mkdir -p "$RUN"

# Step 1 — subtitles
python3 scripts/asr_caption.py \
  --video ~/Movies/lecture.mp4 \
  --out-dir "$RUN" --language zh

# Step 2 — knowledge doc / HTML / CSV
python3 scripts/build_knowledge.py \
  --subtitles "$RUN/subtitles.json" \
  --out-dir "$RUN" --format all

# 2.3 — Anki deck
python3 scripts/gen_apkg.py \
  --csv "$RUN/cards.csv" --out "$RUN/cards.apkg"

# Optional local record (stays on your machine; runs/ is gitignored)
cat > "$RUN/manifest.json" <<EOF
{"video":"~/Movies/lecture.mp4","path":"2","model":"small","outputs":["subtitles.srt","knowledge.md","knowledge.html","cards.csv","cards.apkg"]}
EOF

Scripts Reference

References (load as needed)

• references/hardware-profiles.md — profile table, sizing rationale, tuning
• references/path1-multimodal.md — VLM details, API format, sampling strategy
• references/path2-asr.md — model sizing, device/compute, language options
• references/path3-fusion.md — dual-path fusion (ASR × VLM OCR), dedup sampling, --merged build
• references/templates.md — placeholder spec + custom template examples
• references/outputs.md — HTML/CSV/APKG schemas, single-format runs, degraded mode