# Video Analyzer Dev > Architecture, conventions, testing rules, verification workflow and release process for the mcp-video-analyzer repository. Use this skill whenever you read, write, review or debug any file in this repo — sources under src/, tests, the Dockerfile, package.json, CI workflows or the docs — and before running checks, cutting a release or publishing. It carries hard-won invariants recovered from real production bugs (yt-dlp output templates, ffmpeg crash handling, OCR frame sizing, analysis cache keys, graceful degradation) that are not derivable from reading the code, so consult it before proposing a change rather than after. - Skill: `guimatheus92/video-analyzer-dev` (Agent Skill) - Install (CLI): `npx skillmds@latest add guimatheus92/video-analyzer-dev` - Raw SKILL.md: https://api.skillmd.com/api/skills/guimatheus92/video-analyzer-dev/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: guimatheus92 (https://skillmd.com/u/guimatheus92) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/guimatheus92/video-analyzer-dev --- # mcp-video-analyzer — development guide Project-internal. This is the single source of truth for contributing to this repository; `AGENTS.md` points here rather than repeating it. ## Project MCP server for video analysis — extracts transcripts, key frames, metadata, OCR text, and annotated timelines from video URLs (Loom, YouTube and other yt-dlp platforms, direct links) and local video files. The same engine is also exposed as a one-shot CLI (`mcp-video-analyzer analyze `) and as the portable `/video` agent skill (`skills/video/SKILL.md` + Claude Code plugin). ## Commands - `npm run check` — run ALL checks (format, lint, typecheck, knip, tests). Always run before committing. - `npm run build` — compile TypeScript to dist/ - `npm run test` — run unit tests (vitest) - `npm run test:watch` — run tests in watch mode - `npm run test:smoke` — build + verify MCP server starts and responds to initialize - `npm run test:formats` — just the video-format matrix (`test/e2e/video-formats.e2e.test.ts`); ~15s on a warm cache. Clips are all generated locally, but `detail: 'standard'` always runs OCR, so tesseract.js fetches eng+por traineddata (~7MB) once on a cold `cachePath`. - `npm run security` — `npm audit` on shipped deps only (`--omit=dev`, fails at moderate+). The blocking tier. - `npm run security:all` — `npm audit` on the whole tree (fails at high+); dev-only moderates deliberately don't gate. - `npm run verify-package` — build + pack tarball + install in temp dir + verify startup (pre-publish) - `npm run lint:fix` — auto-fix lint issues (**`security` is NOT part of `check`** — `check` is what `prepublishOnly` runs and must stay offline and deterministic. `npm audit` reads a live advisory database, so folding it in means an advisory published overnight fails `npm publish` for code that never changed. It runs as its own CI job, plus a weekly cron so a new advisory turns CI red without waiting for a PR.) - `npm run format` — auto-format with Prettier - `npm run inspect` — open FastMCP inspector for manual testing - `node dist/index.js analyze [flags]` — run the one-shot CLI against the local build (after `npm run build`) - `npx tsx examples/generate.ts` — regenerate example outputs (run after changing tool output format, processors, or adapters) ## Architecture - **Adapters** (`src/adapters/`) — platform-specific logic (Loom GraphQL, yt-dlp platforms [YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook], direct URL download, TwelveLabs, local files). Each implements `IVideoAdapter`. Registered most-specific-first in `server.ts`: Loom → LocalFile → YtDlp → TwelveLabs → Direct. - **Processors** (`src/processors/`) — shared processing: frame extraction (ffmpeg + browser fallback), image optimization + OCR preprocessing (sharp), frame dedup (dHash, visual + OCR-text-aware), OCR (tesseract.js), audio transcription (whisper), annotated timeline. - **Tools** (`src/tools/`) — MCP tool definitions registered on the FastMCP server. `analyze-core.ts` holds the shared cache + pipeline (`getAnalysis`) + content builder reused by both `analyze_video` and the batch `analyze_videos`. - **Utils** (`src/utils/`) — URL detection, VTT parsing, temp files, in-memory + on-disk cache (`cache.ts`, `analysis-sidecar.ts`), bounded concurrency (`concurrency.ts`), env-flag parsing (`env.ts`). - **CLI** (`src/cli.ts`) — one-shot `analyze` subcommand (`mcp-video-analyzer analyze `) reusing the same `getAnalysis` pipeline: single JSON document on stdout, progress/errors on stderr, frame JPEGs copied to `--out` (default `/mcp-video-analyzer//` via `persistentCacheDir()`) *before* `handle.cleanup()`. `src/index.ts` dispatches on `argv[2]` — no args = MCP stdio server (Docker/smithery/MCP configs rely on this). Adapter registration is shared via `registerAllAdapters()` in `server.ts`. Version literal lives in `src/version.ts`. - **Skill + plugin** (`skills/video/SKILL.md`, `.claude-plugin/`, root `.mcp.json`) — the `/video` agent skill (Route A: MCP tools; Route B: the CLI via npx) and Claude Code plugin/marketplace manifests; the root `.mcp.json` is the plugin's bundled server config (auto-registered on `/plugin install`). Installed from GitHub, never shipped in the npm tarball (`files: ["dist"]`). ## Conventions - TypeScript strict mode. No `any` unless explicitly necessary (use `// eslint-disable-next-line`). - All exports must be used — knip enforces zero unused exports. - Unit tests live next to source files: `foo.ts` → `foo.test.ts`. - Shared test infrastructure lives in `test/`: helpers (`test/helpers/`), fixtures (`test/fixtures/`), smoke tests (`test/smoke/`), e2e tests (`test/e2e/`). - Use `createTestImage()` from `test/helpers/images.ts` and `FIXTURES_DIR` from `test/helpers/fixtures.ts` — don't redefine in each test file. - Use `vitest` with `pool: 'forks'` (required on Windows). - Graceful degradation: never throw when partial results are available. Use `warnings[]` array. This includes the frame-only tools (`get_frames`/`get_frame_at`/`get_frame_burst`) **and** `analyze_moment`: a zero-frame outcome — extraction failure, or every extracted frame filtered out as black — returns `frameCount: 0` with the accumulated warnings, never a thrown `UserError` (issue #26). (Dedup can't empty a non-empty set — it always keeps `frame[0]` — so a filtered-to-zero result is always black-frame filtering.) Keep the input-validation throws — `getAdapter`, an invalid timestamp, or `to <= from` all still throw a `UserError`; **validate timestamps up front, outside the extraction `try`**, so both the download and browser strategies see an already-valid value and only the extraction outcome degrades. Wrap only the `extractFrameAt`/`extractFrameBurst` call in `try/catch` (they throw a raw ffmpeg `Error` that leaks the command line, unlike `extractKeyFrames` which degrades to `[]`); surface a fixed, path-free reason, never the caught `e.message`. Both tools' success and degraded paths emit the same JSON `{ frameCount, ..., warnings }` text block so a client can parse either uniformly. - **`warnings[]` is public output, and every entry goes through `warningReason()`** (`src/utils/warnings.ts`). It is returned to the MCP client, read into the agent's context, and written next to the user's video under `MCP_WRITE_SIDECARS=1`, so an entry must be one line, human-readable, and free of filesystem paths and command lines. A spawned binary's `e.message` is none of those — for `execFile` it is `Command failed: ` plus the entire stderr — which is how issue #46 answered "this clip has no audio" with ~300 lines of ffmpeg banner and absolute temp paths. `ffmpegCrashReason()` (frames, from #26) and `extractYtDlpError()` (yt-dlp, which also redacts the cookie path) remain the specific translators on their own paths; `warningReason()` covers everything else and returns an already-clean single-line message untouched, so a crafted hint is never truncated. `src/utils/warning-sources.test.ts` scans all of `src/` at authoring time and fails the build on the next raw `${e.message}` — it is what found 7 of the 25 sites, the multi-line calls a line-based grep walks straight past. - Three-strategy video download: yt-dlp (primary) → direct HTTP via Loom CDN API (fallback) → headless Chrome screenshots (last resort). **`downloadViaYtDlp()` in `src/utils/ytdlp.ts` is the only yt-dlp download implementation** — both `YtDlpAdapter.downloadVideo` and `LoomAdapter` strategy 1 call it, and it must stay that way (issue #24 was a second, divergent copy that hardcoded `.mp4`). Adapters are siblings behind `IVideoAdapter` and must not depend on each other; shared yt-dlp behaviour belongs in the util. - **yt-dlp `-o` templates must never hardcode a container extension.** Always `-o .%(ext)s` plus a `readdir` glob for `.*`; when yt-dlp merges separate DASH video+audio streams it appends the REAL container to whatever template you gave it, so `-o x.mp4` writes `x.mp4.webm` and every `existsSync('x.mp4')` after it silently fails on a download that actually succeeded. Pair it with `--ffmpeg-location ffmpegPath` — without that, yt-dlp can't merge at all in our published image (no system ffmpeg) and leaves the streams separate, so the glob can return the audio-only file. The `%(ext)s` rule is enforced repo-wide by `src/adapters/ytdlp-output-template.test.ts` (which also unit-tests its own detector against the original #24 code); the `--ffmpeg-location` argument is asserted in `src/adapters/ytdlp.adapter.test.ts`. - yt-dlp platform URLs (single-video pages only; playlists/channels rejected) route through `YtDlpAdapter`. `src/utils/ytdlp.ts` owns everything yt-dlp: `findYtDlp()` (positive probe cached per process), `runYtDlp()`, `ytdlpCookieArgs()`, `commonArgs()`, `extractYtDlpError()`, `YTDLP_MISSING` and `downloadViaYtDlp()` — always spawn through `runYtDlp` so the bin/prefix pairing can't be forgotten, and add shared flags to `commonArgs()` rather than inlining them at one call site. The `findYtDlp()` probe uses `YTDLP_PROBE_TIMEOUT` (20s, not 5s): the standalone binary cold-starts in >7s under load, and a shorter timeout made a slow-but-present binary look absent (issue #26). Raising it is nearly free — a genuinely-missing binary rejects via ENOENT in a few ms, never waiting for the timeout. Missing yt-dlp surfaces as install-hint warnings: adapter `getTranscript`/`getMetadata` throw `YTDLP_MISSING` and every tool handler catches adapter rejections into `warnings[]`; `downloadVideo` must return `null`, never reject (the pipeline calls it without catch) and reports its failure reason via the optional `onWarning` sink. Native captions preferred (uploaded > auto-generated with rolling-window collapse); `[]` from `getTranscript` strictly means "no captions exist" (fetch failures throw) → Whisper fallback. - Standard-detail `maxFrames` default is duration-adaptive via `resolveMaxFrames()` in `detail-levels.ts` (~12 for ≤30s up to 60 for >10min). An explicit `maxFrames` always wins and keys the cache separately (`undefined` drops out of the cache key). `get_frames` keeps its fixed default of 20. - Silent-audio gate: `transcribeAudio()` probes the track with ffmpeg `volumedetect` (first 2 min) before any Whisper strategy; mean volume ≤ −55dB skips transcription with a warning — an empty transcript on a mute track is content, not a bug. - Frame extraction uses bundled `ffmpeg-static` — no system ffmpeg needed. - **The bundled `ffmpeg-static` 7.0.2 LINUX build segfaults on the MPEG-TS demuxer.** Any `.mts`/`.m2ts` input (the standard AVCHD camcorder format) kills ffmpeg on probe, extract, and even remux — in the published Docker image and for every Linux user. The byte-identical file parses fine on the Windows build, so it is the binary, not the file; `ffmpeg-static@5.3.0` is the latest release, so there is no version to upgrade to. Found by `test/e2e/video-formats.e2e.test.ts`, which is exactly the blind spot it was written for. A signal-terminated ffmpeg is NOT "no frames found": `ffmpegCrashReason()` in `frame-extractor.ts` turns it into one actionable, path-free warning (never `e.message`, which is the full argv + banner), and every rewrap site in that file resolves the reason while the original error still carries `.signal`. The matrix probes the binary's actual capability rather than branching on `process.platform` — a platform check is a guess about a binary, not a fact about it — and asserts the degraded contract where the demuxer is broken, so a silently-empty result stays a failure on every platform. - Black frame detection filters out DRM-protected/blank frames automatically. - Scene detection threshold default: 0.1 (optimized for screencasts/demos). Use `extractKeyFrames()` (not raw `extractSceneFrames`) so static clips with no scene cuts fall back to uniform temporal sampling — critical for talking-head Reels/Stories. - OCR runs on every frame *before* dedup; when OCR is enabled, dedup uses `dedupeKeepingTextChanges()` (visual + on-screen-text aware) so frames whose only change is the text overlay survive. Plain `deduplicateFrames()` (visual only) is used when OCR is off. - OCR frames are preprocessed (grayscale + 2× upscale + contrast normalization + sharpen) by default; `MCP_OCR_PREPROCESS=0` disables. - Transcription strategy order: HF transformers (opt-in) → whisper CLI → OpenAI API. HF only runs when `WHISPER_HF_MODEL` is set, so otherwise the CLI wins and its `WHISPER_MODEL`/`WHISPER_LANGUAGE` settings are never silently overridden. `model`/`language`/`initialPrompt` are overridable per call on `analyze_video`/`analyze_videos`/`get_transcript`. - The whisper CLI is run directly (no `--help` probe — it double-imports torch and crashes on Windows on non-ASCII help text); `ENOENT` distinguishes "not installed" (try next candidate) from "installed but crashed" (warn). Spawned with `PYTHONUTF8=1`/`PYTHONIOENCODING=utf-8` so multilingual transcripts don't crash the Python stdout codec. When NO backend is configured at all, `transcribeAudio` emits an actionable "No speech-to-text backend available" warning instead of a bare `[]`. - yt-dlp errors that look auth-related (login/cookies/private/age-restricted/empty-media/rate-limit) get a cookie hint appended by `extractYtDlpError` naming this server's env vars (`YTDLP_COOKIES` / `YTDLP_COOKIES_FROM_BROWSER`), not yt-dlp's raw CLI flags. - Persistent sidecars (`MCP_WRITE_SIDECARS=1`) write `.vtt` (Whisper transcripts only, never clobbering an existing one) + `.analysis.json` + `.frames/` next to local videos for resumable bulk processing; reads validate `mtime:size` + params. - CLI mode: stdout is reserved for the single JSON result document — progress, warnings-in-flight, and errors go to stderr. CLI flags validate through the shared `AnalyzeOptionsSchema` (no hand-rolled validation). Partial failures ride in `warnings[]` with exit 0; only hard failures exit 1. - `skills/video/SKILL.md` is a public contract: any change to MCP tool names, CLI flags, or the CLI JSON shape must update `skills/video/SKILL.md` + `README.md` + the "Using this project as a tool" section of `AGENTS.md` in the same PR. - Tesseract `.traineddata` downloads are cached in `/mcp-video-analyzer/tessdata` via `cachePath` (frame-ocr.ts) — never let them land in the process cwd (pollutes the agent's project dir under npx). - **No persistent on-disk location may use a fixed name under `os.tmpdir()`.** `persistentCacheDir()` (`temp-files.ts`) resolves the OS per-user cache dir (`MCP_CACHE_DIR` overrides it; CI pins that so its `actions/cache` paths match what the code writes), and the golden-clip cache (`test/helpers/golden-clips.ts`) lives in the gitignored repo-local `.cache/golden-clips` — **not** `node_modules/.cache`, which `npm ci` wipes after the CI cache step restores it. Both were `/mcp-video-analyzer/...` and both are gone on purpose. The one remaining `tmpdir()` fallback (no usable home, no override) is uid-keyed so two users cannot collide, and is enforced by `src/utils/tmpdir-usage.test.ts`, which scans `src/` and `test/` for any other fixed-name use. The shared temp dir is world-traversable, so a predictable path there is pre-creatable by any other local user: they read the frames the CLI copies out and can plant a `.traineddata` for our OCR to load. CodeQL flagged the test-side one (`js/insecure-temporary-file`), following the taint from the golden clips through the e2e tests into the sidecar `writeFile`s — the production one it could not see, because those sinks are `copyFile` and tesseract's own write. Per-call scratch space is different and stays in tmp: `createTempDir()` uses `mkdtemp`, which is randomized and mode-0700. - **`detectPlatform` (`url-detector.ts`) is the network trust boundary, and `assertPublicUrl` (`ssrf-guard.ts`) is the only gate at the sinks.** All 8 MCP tools and the CLI validate through `.refine(isVideoSource)`, which is `detectPlatform(...) !== null` — so a destination refused there is refused at every entry point at once. That is why GHSA-hpmc-4g74-v53v was fixed there rather than at each caller. `detectPlatform` must stay **sync** (adapters call it from `canHandle`, and it keys the analysis cache), so it can only do literal checks: scheme, IP literals, `localhost`/`.local`, UNC. Anything needing DNS belongs in `assertPublicUrl`, which every outbound sink must call — today `downloadDirectVideo` and `extractBrowserFrames`. **Adding a new sink that takes a client URL means adding that call**; the browser fallback was an unreported second SSRF sink precisely because it was never treated as one, and it is the worse of the two (it renders the response and returns it as a JPEG). Inside it, the pre-flight call covers the first hop only — Chrome follows redirects itself and every subresource is its own request — so the `page.on('request')` interceptor is a second sink in its own right and calls `assertPublicUrl` too, asynchronously (puppeteer holds the request until continue/abort settles). A literal-only check there is not a cheaper version of the gate, it is a hole: it cannot see a hostname that merely *resolves* internal. Two things that path must keep: `data:`/`blob:`/`about:` continue unvalidated (an MSE player hands `