video-perception

name	video-perception
description	Analyze what a video clip's pixels show and reframe horizontal footage for vertical output. Use when working with perception.json, shot type, scene type, camera motion, exposure, on-screen text, saliency, subject tracking, auto-reframe, 9:16 conversion, or when configuring a vision model for Video Review OS. Also use when the user asks why a clip was flagged as needing reframe, or wants to plug a different vision model into the pipeline.
license	MIT

Visual perception and reframing

perception.json is what the pixels say, kept separate from what the words say. It is optional everywhere: a project that has never been perceived behaves exactly as it did before the stage existed.

video-review-os perceive <project>            # writes perception.json
video-review-os perceive <project> --dry-run  # plan without decoding or calling out

What a reading contains

Per clip, under visual:

Field	Values
shot_type	wide, medium, close_up, extreme_close_up, insert, screen, unknown
scene_type	talking_head, demo, performance, crowd, product, environment, b_roll_abstract, unknown
setting / area	indoor, outdoor, unknown / a label from [perception] areas
people_count	0, 1, 2, group, unknown
camera_motion	static, handheld, pan, zoom, moving, unknown
motion_energy	still, low, medium, high
exposure, contrast, saturation	quality readings from the luma distribution
text_on_screen	true/false, or null when the signal is ambiguous
subject_region, subject_x_pct	where the salient content sits horizontally
evidence	which signals produced this reading

Plus a reframe block: whether the clip needs reframing for the target aspect, and the crop window (x_pct/y_pct/w_pct/h_pct) that keeps the subject in frame.

Read unknown as information

The default fallback provider is deterministic, offline, and dependency-free: it decodes a small RGB grid and derives motion, exposure, contrast, saturation, letterboxing, cut density, and a saliency-weighted crop in pure Python. It reports unknown for shot type, people count, and setting on purpose — those are not knowable from a luma grid, and a plausible guess would poison every downstream filter that trusts them. unknown means "nobody has looked yet", not "the analysis failed". Configure a vision provider to fill them.

Likewise camera_motion is only ever static or unknown from the fallback: a whole-frame delta cannot tell a panning camera from a subject moving in front of a locked-off one.

Plugging in a vision model

Any vision model behind a chat-completions endpoint works — local or hosted, no code change:

[perception]
provider = "openai-compatible"
model = "<whatever your endpoint serves>"
base_url_env = "VIDEO_REVIEW_VISION_BASE_URL"   # e.g. http://localhost:11434/v1
api_key_env  = "VIDEO_REVIEW_VISION_API_KEY"
max_frames_per_clip = 3

The provider sends the clip's scenes/ stills and the allowed vocabulary, then sanitizes the reply against that vocabulary. A model cannot invent an enum value, cannot return an out-of-frame crop, and cannot break the run by failing — it degrades to the local reading.

To write your own provider instead, see the video-review-os-plugins skill.

Turning a reframe into a render

The window is advisory until you ask for it:

[assembly]
fit_mode = "subject"   # blur | crop | pad | subject

Under subject, each segment is cropped to its perception window and then scaled to fill the vertical frame. Segments without a window fall back to the blur fill rather than guessing a crop. The window is part of assembly_signature, so changing the framing correctly invalidates a prior approval.

What the window is not

One static window per segment, not a per-frame tracked crop. A subject that moves further across the frame than the window is wide cannot stay fully inside it — the window is placed on the subject's centre of mass over the clip, so it holds the average framing rather than following the movement. If a clip needs the camera to actually follow, cut it shorter so each segment gets its own window, or reframe that one by hand.

The window is also biased back toward centre in proportion to how uncertain the reading is: a clearly localized subject is trusted almost completely, while diffuse energy (no subject to find) produces a centred window rather than a confident wrong one.

Cost and honesty

The deterministic pass costs one ffmpeg decode per clip at ~48×27 and no network. A vision provider costs one request per clip. perception.json always records which provider produced the reading and, when a hosted one partially failed, which clips fell back — check provider, status, and errors before trusting a batch.