Clip To Plan

clip-to-plan

Convert a review clip into a plan the way a careful engineer would: watch it, transcribe it, then prove each complaint against the code before writing tasks. Never draft fixes straight from the transcript — the transcript says what looks wrong, the code says why.

When to use

A stakeholder/reviewer/tester sends a recording (screen capture with narration, a Loom, a voice memo) describing bugs or feedback, and the goal is a concrete plan. Works for any spoken language — Whisper translates to English by default.

Prerequisites (check once)

• ffmpeg + ffprobe on PATH (ffmpeg -version).
• openai-whisper for the python that runs the script: pip install -U openai-whisper.
• If either is missing, tell the user the one-line install and stop.

Step 1 — Extract evidence (deterministic script)

Run the bundled script; put output in a scratchpad dir, not the repo:

python3 <skill-dir>/scripts/extract.py "<path/to/clip.mp4>" \
  -o "<scratchpad>/clip-report" [--task translate|transcribe] [--model small] [--language ro]

• --task translate (default) → any language to English. --task transcribe keeps the original language. Ask/choose based on what language the plan is written in.
• --model default small (fast, supports translate). Bump to medium or large-v3 if the audio is noisy or the translation reads rough. turbo is transcribe-only — the script refuses it for translate.
• Output: transcript.md (timestamped lines), audio.json (raw Whisper), frames/NN_<t>.jpg (one per segment, grabbed 1s past each start), frame_times.txt, and findings.md (the skeleton table you fill in next).

Step 2 — Watch and read, together

• Read transcript.md in full. Then Read the frames — actually view the images, at least around every line that sounds like a complaint or request. The frame is what the reviewer was looking at when they spoke; the pairing is the whole point. Frame NN matches transcript segment NN.
• Distinguish complaints from filler. Reactions ("ah, this is nice", thinking out loud) are context, not tasks. Only turn problems, requests, and confusions into findings.
• Sanity-check transcription artifacts. Whisper mangles proper nouns and foreign words — a common failure is a short filler word in the source language getting rendered as a plausible English name, which can invent a nonexistent person. Cross-read suspicious names/nouns against the frame and surrounding context before trusting them.

Step 3 — Cross-reference each finding against the codebase (the load-bearing step)

For every real complaint, find the root cause in the code, don't guess:

• Use the frame to identify the exact screen/component, then grep/read the relevant source. Trace to a specific file:line and a concrete mechanism.
• A vague complaint often hides a different, verifiable bug once you look (e.g. "I can't see the funnels" → the click-through works, but Delete never refreshes the list). Report what the code actually shows, and note when the surface complaint and the real defect differ.
• Fill findings.md: each row gets a root cause (file:line + why) and a fix idea. If something can't be reproduced in code, say so explicitly rather than inventing a cause.

Step 4 — Draft the plan

Write a plan document matching the repo's existing planning conventions (look at how prior plans in the project are structured and mirror them). If there's no house style, use this shape:

• Header — source clip (name, duration), how it was processed (Whisper task/model), and where the transcript+frames live.
• Goal — one paragraph: the end state when every finding is resolved.
• Complaints → root causes table — reviewer's words (with timestamp) | verified root cause (file:line). This is the spine; it makes each task traceable to a moment in the clip.
• Decisions — anything ambiguous, surface as explicit questions for the user before finalizing tasks (auth approach, scope calls, privacy trade-offs…). Use AskUserQuestion; fold answers back in.
• Tasks — sliced into small, independently shippable units (respect any project task-size rule; default ≤3–4h each), each with a clear Exit check. Number them, note dependencies/chains.
• Verification — how each fix is proven (local tests + live checks).
• Risks — landmines, physics-not-bugs, follow-ups to flag.

Step 5 — Close out

• Keep audio.wav and frames/ in the scratchpad; only the plan doc lands in the repo (ask before adding new files, per project rules).
• Offer a short status blurb summarizing findings and the plan, in whatever format/language the team uses for updates.

Reference

A representative run: a ~4.5-minute screen-recording with spoken narration → ~11 findings, each traced to a concrete root cause in the code, sliced into small tasked units with dependencies and a verification matrix. Keep your own worked example plan alongside it as the house template for future runs.