# Video Perception > Analyze what a video clip's pixels show and reframe horizontal footage for vertical output. Use when working with perception.json, shot type, scene type, camera motion, exposure, on-screen text, saliency, subject tracking, auto-reframe, 9:16 conversion, or when configuring a vision model for Video Review OS. Also use when the user asks why a clip was flagged as needing reframe, or wants to plug a different vision model into the pipeline. - Skill: `cgallic/video-perception` (Agent Skill) - Install (CLI): `npx skillmds@latest add cgallic/video-perception` - Raw SKILL.md: https://api.skillmd.com/api/skills/cgallic/video-perception/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: cgallic (https://skillmd.com/u/cgallic) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/cgallic/video-perception --- # Visual perception and reframing `perception.json` is what the *pixels* say, kept separate from what the words say. It is optional everywhere: a project that has never been perceived behaves exactly as it did before the stage existed. ```bash video-review-os perceive # writes perception.json video-review-os perceive --dry-run # plan without decoding or calling out ``` ## What a reading contains Per clip, under `visual`: | Field | Values | | --- | --- | | `shot_type` | wide, medium, close_up, extreme_close_up, insert, screen, unknown | | `scene_type` | talking_head, demo, performance, crowd, product, environment, b_roll_abstract, unknown | | `setting` / `area` | indoor, outdoor, unknown / a label from `[perception] areas` | | `people_count` | 0, 1, 2, group, unknown | | `camera_motion` | static, handheld, pan, zoom, moving, unknown | | `motion_energy` | still, low, medium, high | | `exposure`, `contrast`, `saturation` | quality readings from the luma distribution | | `text_on_screen` | true/false, or `null` when the signal is ambiguous | | `subject_region`, `subject_x_pct` | where the salient content sits horizontally | | `evidence` | which signals produced this reading | Plus a `reframe` block: whether the clip needs reframing for the target aspect, and the crop window (`x_pct`/`y_pct`/`w_pct`/`h_pct`) that keeps the subject in frame. ## Read `unknown` as information The default `fallback` provider is deterministic, offline, and dependency-free: it decodes a small RGB grid and derives motion, exposure, contrast, saturation, letterboxing, cut density, and a saliency-weighted crop in pure Python. It reports `unknown` for shot type, people count, and setting **on purpose** — those are not knowable from a luma grid, and a plausible guess would poison every downstream filter that trusts them. `unknown` means "nobody has looked yet", not "the analysis failed". Configure a vision provider to fill them. Likewise `camera_motion` is only ever `static` or `unknown` from the fallback: a whole-frame delta cannot tell a panning camera from a subject moving in front of a locked-off one. ## Plugging in a vision model Any vision model behind a chat-completions endpoint works — local or hosted, no code change: ```toml [perception] provider = "openai-compatible" model = "" base_url_env = "VIDEO_REVIEW_VISION_BASE_URL" # e.g. http://localhost:11434/v1 api_key_env = "VIDEO_REVIEW_VISION_API_KEY" max_frames_per_clip = 3 ``` The provider sends the clip's `scenes/` stills and the allowed vocabulary, then sanitizes the reply against that vocabulary. A model cannot invent an enum value, cannot return an out-of-frame crop, and cannot break the run by failing — it degrades to the local reading. To write your own provider instead, see the `video-review-os-plugins` skill. ## Turning a reframe into a render The window is advisory until you ask for it: ```toml [assembly] fit_mode = "subject" # blur | crop | pad | subject ``` Under `subject`, each segment is cropped to its perception window and then scaled to fill the vertical frame. Segments without a window fall back to the blur fill rather than guessing a crop. The window is part of `assembly_signature`, so changing the framing correctly invalidates a prior approval. ## What the window is not One static window per segment, not a per-frame tracked crop. A subject that moves further across the frame than the window is wide cannot stay fully inside it — the window is placed on the subject's centre of mass over the clip, so it holds the average framing rather than following the movement. If a clip needs the camera to actually follow, cut it shorter so each segment gets its own window, or reframe that one by hand. The window is also biased back toward centre in proportion to how uncertain the reading is: a clearly localized subject is trusted almost completely, while diffuse energy (no subject to find) produces a centred window rather than a confident wrong one. ## Cost and honesty The deterministic pass costs one ffmpeg decode per clip at ~48×27 and no network. A vision provider costs one request per clip. `perception.json` always records which provider produced the reading and, when a hosted one partially failed, which clips fell back — check `provider`, `status`, and `errors` before trusting a batch.