Video Scene Analysis

Video Scene Analysis

Analyze a local video into a structured scene sequence: cuts, visual type, zoom transitions, transcript with timecodes, and per-scene focus/emotion.

Setup (one-time)

pip3 install -r ~/.cursor/skills/video-scene-analysis/scripts/requirements.txt
bash ~/.cursor/skills/video-scene-analysis/scripts/setup_models.sh

Requires ffmpeg and ffprobe on PATH.

Quick start

SCRIPTS=~/.cursor/skills/video-scene-analysis/scripts

python3 $SCRIPTS/analyze_video.py video.mp4 -o .
python3 $SCRIPTS/analyze_video.py lolo/videos/clip.mp4 -o ./analysis --language es

Outputs in -o directory (default: cwd):

• {stem}.analysis.json — machine-readable sequence
• {stem}.analysis.md — human-readable report
• {stem}_frames/scene_XX.jpg — one sharp representative frame per scene

Agent workflow (mandatory)

The script handles steps 1–4 (visual + audio + frames + a conservative split-screen layout.hint). You (the active session LLM) must complete steps 5–10 (camera, composition, summaries, mannerisms, avatar profile) by viewing the frames before delivering results. Do not call Gemini or any external LLM API.

1. Confirm video path and output directory.
2. Run setup if models or deps are missing.
3. Run analyze_video.py → JSON with "summary": null and "camera": null per scene, plus {stem}_frames/.
4. Read each representative_frame image (use the Read tool on every scene_XX.jpg). Classify camera for each scene and write scenes[].camera:

   {
     "angle": "eye_level | low_angle | low_angle_v2 | high_angle | three_quarter | dutch_tilt | negative_space | pull_out | zoom_in | none",
     "framing": "extreme_close_up | close_up | medium_close_up | medium_shot | medium_wide | wide_shot | unknown",
     "description": "Nota breve en español (encuadre vertical, selfie, etc.)"
   }
   

   angle must be an English pipeline slug (snake_case), aligned with avatar prompts in lolo/angles/prompts/ when applicable. Use eye_level for baseline frontal talking head; none for B-roll that does not map to the pipeline. Cross-check zoom_from_previous (zoom_in / zoom_out → consider zoom_in / pull_out slugs on presenter shots).
5. Composition (every scene — while you have the frame open). From the same scene_XX.jpg, fill the agent-written composition fields. The script pre-fills scenes[].layout.hint (fullscreen / possible_split_horizontal / possible_split_vertical) as a conservative guess you must confirm or correct.
   • scenes[].layout — screen composition (see Layout taxonomy):

     {
       "type": "fullscreen | split_horizontal | split_vertical | pip | overlay_graphics",
       "regions": [
         { "position": "top|bottom|left|right|inset", "content": "broll|main_character|screen|graphics",
           "description": "qué se ve en esa región" }
       ],
       "notes": "breve, en español"
     }
     

     Always inspect for split scenes: a single scene that shows B-roll in one band (top or bottom) and the main character talking in the other is split_horizontal (side-by-side is split_vertical; a small inset is pip). List one entry per region. For a normal single shot use "fullscreen" with empty regions.
   • scenes[].broll_kind (B-roll / supplementary scenes, AND any B-roll region of a split — see B-roll kind taxonomy). Distinguish pre-recorded archival footage of a recognizable person (archival_known_person) from generic complementary material (stock_generic). null for pure talking-head.
   • scenes[].known_people — array of recognizable real people shown in pre-recorded footage (names if you recognize them, else short descriptions like "older male chef, 2000s TV interview"). []/null when none or unsure.
   • scenes[].background (presenter / talking-head scenes — see Background taxonomy): is the person's backdrop a real set/location or animated (drawings, cartoons, motion graphics)?

     { "type": "real_set | animated | mixed | plain | virtual | unknown",
       "elements": "qué hay detrás (p.ej. 'dibujos animados de nubes', 'oficina real')",
       "notes": "opcional" }
     

     null for B-roll / non-presenter scenes.
6. Read the JSON metadata. For every scene, write scenes[].summary using:
   • transcript, scene_type, layout, zoom_from_previous, visual, camera, audio
7. Facial mannerisms (talking-head only). For each main_character_solo scene (including the presenter band of a split), while you have the frame open, write scenes[].mannerisms: a brief (one sentence) note of how the face/head moves — eyebrow activity, head nods/tilts, eye contact, mouth/expression, lean, gesture restraint. Leave null for B-roll / non-presenter scenes.
8. Avatar profile (talking head). Synthesize the talking-head mannerisms into a single reusable avatar_profile (top-level), consistent across the video, with:

   {
     "mannerisms_summary": "1-2 sentence description of the recurring facial behavior",
     "video_prompt": "Concise p-video-avatar prompt describing how this person naturally speaks to camera (identity-consistent, present tense)",
     "negative_prompt": "very brief, comma-separated failure modes to avoid (e.g. exaggerated gestures, big toothy grin, looking away, jittery head movement, subtitles, watermark)"
   }
   

   Keep video_prompt short and behavior-focused; keep negative_prompt brevísimo. If the video has no talking-head scenes, leave avatar_profile null.
9. Optionally rewrite overview (1–2 sentences, Spanish) with the narrative arc.
10. Re-render markdown:

python3 $SCRIPTS/render_report.py path/to/{stem}.analysis.json

11. Present the final .analysis.md to the user.

Never skip steps 4–10. Heuristic or API-based summaries/classification are intentionally not used.

Export the talking profile (for avatar-talking-video)

Once avatar_profile is written, export it to the avatar folder so the avatar-talking-video skill auto-loads it for every generated talking-head clip:

python3 $SCRIPTS/export_talking_profile.py path/to/{stem}.analysis.json
# → writes <avatar>/talking_profile.json (avatar dir auto-inferred; override with --avatar-dir)

Pass several analyses to pick the first with a profile; preview with --dry-run.

Batch

Run the script for each video, then enrich each JSON before re-rendering:

SCRIPTS=~/.cursor/skills/video-scene-analysis/scripts
OUT=./analysis
mkdir -p "$OUT"
for f in lolo/videos/*.mp4; do
  python3 $SCRIPTS/analyze_video.py "$f" -o "$OUT" --language es
done
# → enrich each $OUT/*.analysis.json, then render_report.py on each

Script options

Camera taxonomy (agent-written)

camera.angle — English pipeline slug (primary)

Prompts de referencia: lolo/angles/prompts/{slug}.txt

camera.framing — shot size

Layout taxonomy (scene.layout.type — agent-written)

A single scene can combine B-roll and the presenter. Capture that here (the script only pre-fills layout.hint).

For splits/pip list one regions[] entry per band: position (top/bottom/left/right/inset) + content (broll/main_character/screen/graphics) + a short description.

B-roll kind taxonomy (scene.broll_kind — agent-written)

For B-roll / supplementary scenes (and the B-roll region of a split), say what kind of footage it is — pre-recorded archival of a known person vs generic.

Record any recognizable people in scene.known_people (array of names or short descriptions). When a reel leans on archival_known_person/archival_footage, sourcing it for a new reel is the job of the broll-finder skill (real YouTube footage), not broll-generator (synthetic).

Background taxonomy (scene.background.type — agent-written, presenter scenes)

Is the main character's backdrop a real place or animated?

Put a short description of what's behind the person in background.elements.

Audio profiles (per scene)

Heuristic: Whisper masks speech intervals; transients in non-speech audio → SFX; sustained energy → music bed. For mixes complejos, usar Demucs (youtube-audio-toolkit) como complemento.

What the script detects vs what you write

Scene types

Tips

• Reels with hard cuts: --scene-mode detect (default in auto).
• Uniform 6s windows: --scene-mode interval --interval 6.
• Quick smoke test on transcript only: --whisper-model tiny.
• Long videos (>3 min): --whisper-model small.

Troubleshooting

• Face model not found → run setup_models.sh.
• scenedetect / faster-whisper missing → reinstall requirements.
• No transcript → check audio track; try --language es.
• MD shows "Pendiente" → you skipped agent enrichment (summary or camera).

Reference

JSON schema: REFERENCE.md