cgallic Visual perception and reframing
perception.json is what the pixels say, kept separate from what the words say. It is
optional everywhere: a project that has never been perceived behaves exactly as it did
before the stage existed.
video-review-os perceive <project> # writes perception.json
video-review-os perceive <project> --dry-run # plan without decoding or calling out
What a reading contains
Per clip, under visual:
| Field | Values |
|---|---|
shot_type |
wide, medium, close_up, extreme_close_up, insert, screen, unknown |
scene_type |
talking_head, demo, performance, crowd, product, environment, b_roll_abstract, unknown |
setting / area |
indoor, outdoor, unknown / a label from [perception] areas |
people_count |
0, 1, 2, group, unknown |
camera_motion |
static, handheld, pan, zoom, moving, unknown |
motion_energy |
still, low, medium, high |
exposure, contrast, saturation |
quality readings from the luma distribution |
text_on_screen |
true/false, or null when the signal is ambiguous |
subject_region, subject_x_pct |
where the salient content sits horizontally |
evidence |
which signals produced this reading |
Plus a reframe block: whether the clip needs reframing for the target aspect, and the
crop window (x_pct/y_pct/w_pct/h_pct) that keeps the subject in frame.
Read unknown as information
The default fallback provider is deterministic, offline, and dependency-free: it decodes
a small RGB grid and derives motion, exposure, contrast, saturation, letterboxing, cut
density, and a saliency-weighted crop in pure Python. It reports unknown for shot type,
people count, and setting on purpose — those are not knowable from a luma grid, and a
plausible guess would poison every downstream filter that trusts them. unknown means
"nobody has looked yet", not "the analysis failed". Configure a vision provider to fill them.
Likewise camera_motion is only ever static or unknown from the fallback: a whole-frame
delta cannot tell a panning camera from a subject moving in front of a locked-off one.
Plugging in a vision model
Any vision model behind a chat-completions endpoint works — local or hosted, no code change:
[perception]
provider = "openai-compatible"
model = "<whatever your endpoint serves>"
base_url_env = "VIDEO_REVIEW_VISION_BASE_URL" # e.g. http://localhost:11434/v1
api_key_env = "VIDEO_REVIEW_VISION_API_KEY"
max_frames_per_clip = 3
The provider sends the clip's scenes/ stills and the allowed vocabulary, then sanitizes
the reply against that vocabulary. A model cannot invent an enum value, cannot return an
out-of-frame crop, and cannot break the run by failing — it degrades to the local reading.
To write your own provider instead, see the video-review-os-plugins skill.
Turning a reframe into a render
The window is advisory until you ask for it:
[assembly]
fit_mode = "subject" # blur | crop | pad | subject
Under subject, each segment is cropped to its perception window and then scaled to fill
the vertical frame. Segments without a window fall back to the blur fill rather than
guessing a crop. The window is part of assembly_signature, so changing the framing
correctly invalidates a prior approval.
What the window is not
One static window per segment, not a per-frame tracked crop. A subject that moves further across the frame than the window is wide cannot stay fully inside it — the window is placed on the subject's centre of mass over the clip, so it holds the average framing rather than following the movement. If a clip needs the camera to actually follow, cut it shorter so each segment gets its own window, or reframe that one by hand.
The window is also biased back toward centre in proportion to how uncertain the reading is: a clearly localized subject is trusted almost completely, while diffuse energy (no subject to find) produces a centred window rather than a confident wrong one.
Cost and honesty
The deterministic pass costs one ffmpeg decode per clip at ~48×27 and no network. A vision
provider costs one request per clip. perception.json always records which provider
produced the reading and, when a hosted one partially failed, which clips fell back —
check provider, status, and errors before trusting a batch.