# Video To Shorts > Use when a Project Protocol video needs short-form candidate planning, approval, horizontal extraction, or reviewed 9:16 delivery before or after the shared main render. - Skill: `whitetowerai/video-to-shorts` (Agent Skill, multi-file: 22 files) - Install (CLI): `npx skillmds@latest add whitetowerai/video-to-shorts` - Raw SKILL.md: https://api.skillmd.com/api/skills/whitetowerai/video-to-shorts/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: WhiteTowerAI (https://skillmd.com/u/whitetowerai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/whitetowerai/video-to-shorts --- # Video To Shorts ## Dependencies `/video-understand` is a prerequisite. Run it first so shorts use the validated source transcript and canonical timeline. Before starting, verify that it is installed. If it is not, warn the user that this prerequisite is missing and stop before processing media. Require `ffmpeg`/`ffprobe` on PATH and Python with `Pillow`. Check them before processing media. ## Scope This skill has two project phases: 1. **Plan before render:** map the understood transcript to program time, author and approve candidates, and write `work/shorts/shorts-plan.json`. This phase writes no final video. 2. **Finalize after render:** after every selected main-sequence edit is approved and the shared main delivery is verified, rerun `plan.py` to bind the same approved ranges to that delivery, then extract horizontal and optional vertical outputs. It owns short candidate judgment, deterministic qualification, horizontal extraction, short-relative transcripts, and optional deterministic vertical delivery. It does not transcribe in project mode, change the main sequence, add captions/graphics, grade, publish, or perform continuous subject tracking. Delivered Shorts are derivatives of `final/final-video.mp4`, but candidate planning precedes that file. Record the operation in the project DAG, but never add it to `sequences.main.operations`; the main delivery and its original-vs-final comparison remain unchanged. ## Project Inputs - `work/project.json` - the project source recorded by `project.source.path` - `work/understand/transcript.json` in source time - `work/timeline.json` - active main-operation revisions at each phase Planning does not require `final/final-video.mp4`. Finalization and extraction require that exact project render with `render.status: verified`. `prepare_transcript.py --timeline` calls the shared `projectlib.map_transcript_to_timeline`. Candidate/extraction times are therefore program seconds on the actual final video. Words removed by the main edit are not available to shorts. Each retained word keeps its `source_range`, `program_range`, and `clip_id`. Project transcript metadata binds the project source, source transcript, and timeline by absolute path, size, modified time, and SHA-256. Candidate review frames map their program timestamps back through `timeline.json` to the source video. Candidate validation refuses a changed binding. ## Output Layout ```text work/shorts/ |-- transcript.json |-- transcript_metadata.json |-- candidates.json |-- shorts-plan.json |-- review/ | |-- candidate_review.json | `-- approved_candidates.json `-- short-001/ |-- transcript.json |-- extraction-report.json `-- vertical-agent/ |-- agent_vertical_plan.json |-- vertical_plan.json `-- review/vertical_review.json work/cache/shorts/ # disposable frames/sheets/intermediate media review/06-shorts/ |-- candidates-.html # immutable authoritative page |-- candidates--question.md |-- assets/candidates-/ # immutable start/middle/end frames |-- candidates.html # non-authoritative latest alias |-- candidates-summary.md |-- short-001-horizontal.jpg |-- short-001-vertical-preview.mp4 |-- short-001-vertical-contact-sheet.jpg |-- short-001-vertical-preview-summary.json |-- short-001-vertical-preview-probe.json |-- -vertical-review-.html # immutable authoritative page |-- -vertical-review--question.md |-- -vertical-review.html # non-authoritative latest alias |-- -vertical-review-assets/ | `-- / # immutable authoritative evidence | |-- preview.mp4 # absent for REVIEW_REQUIRED | |-- contact-sheet.jpg # absent for REVIEW_REQUIRED | |-- preview-summary.json | `-- media-probe.json `-- shorts-summary.md final/shorts/ |-- short-001-horizontal.mp4 `-- short-001-vertical.mp4 # only when selected and approved ``` ## Program Transcript Run from the repository root: ```powershell $RepoRoot = (Resolve-Path ".").Path $SkillRoot = Join-Path $RepoRoot "skills\video-to-shorts" $ProjectRoot = (Resolve-Path "").Path $Work = Join-Path $ProjectRoot "work" $ShortsWork = Join-Path $Work "shorts" $ProjectSource = (Resolve-Path "").Path $FinalVideo = Join-Path $ProjectRoot "final\final-video.mp4" $ShortsReview = Join-Path $ProjectRoot "review\06-shorts" New-Item -ItemType Directory -Force $ShortsWork,$ShortsReview | Out-Null python "$SkillRoot\scripts\prepare_transcript.py" $ProjectSource ` --transcript "$Work\understand\transcript.json" ` --timeline "$Work\timeline.json" ` --out $ShortsWork ``` Project timeline mode requires the canonical supplied transcript. Standalone mode may omit `--timeline`; it keeps input-relative time and may use the temporary `video-understand/scripts/transcribe.py` fallback. No transcript is auto-discovered. ## Candidate Authoring Generate long-video visual context into cache, inspect every sheet needed for a decision, and hand-author candidate JSON. Deterministic code validates editorial judgment; it does not invent candidates or scores. ```powershell python "$SkillRoot\scripts\prepare_visual_context.py" $ProjectSource ` --timeline "$Work\timeline.json" ` --out "$Work\cache\shorts\candidate-visual" --interval 15 ``` Candidate input remains `shorts-candidates.v2` for compatibility. Set `selection.evidence_mode` and every candidate `evidence_mode` to `text_visual`. Each candidate requires: - stable `candidate_id`, title, and scene type; - complete `start_time`/`end_time` in program time; - `transcript_excerpt` exactly equal to overlapping transcript words, not whole overlapping segments; - all six score entries and non-empty reasons; - non-empty `metadata.editorial_reason`; - optional reviewed `filler_drop_spans` and visual evidence fields. Do not supply total `score`; the validator adds the six dimensions: | Dimension | Maximum | |---|---:| | `hook` | 20 | | `completeness` | 20 | | `audience_value` | 20 | | `emotion_tension` | 15 | | `quotability` | 15 | | `pace_editability` | 10 | Prefer a complete, independently understandable thought over a shorter fragment. Keep a dependent interviewer question, premise, claim support, reversal, or payoff when it is necessary. Boundaries must land on whole words and thoughts. Validate the authored file: ```powershell $CandidateDir = Join-Path $ShortsWork "preview\text_visual" python "$SkillRoot\scripts\candidates.py" ` --out $CandidateDir ` --candidates "$CandidateDir\candidate-input.json" ` --transcript "$ShortsWork\transcript.json" ` --project-root $ProjectRoot ``` This writes the compatibility preview files, durable `work/shorts/candidates.json`, and `review/06-shorts/candidates.{html,summary.md}`. ## Review Modes Candidate selection and delivery mode are one hash-bound gate. Delivery is exactly `horizontal_only` or `horizontal_and_vertical`. Open the bound interactive page in human mode: ```powershell python "$SkillRoot\scripts\interaction.py" candidate-open --out $ShortsWork --review-out $ShortsReview ``` The command prints an authoritative review-ID-scoped page. The receipt binds that page, its candidate JSON, transcript, source media, fixed question, and every extracted frame by SHA-256. `candidates.html` is only a non-authoritative convenience alias to the latest page. Open the printed authoritative path with the current OS native command: ```text # Windows PowerShell Start-Process (Resolve-Path -LiteralPath "") # macOS open "" # Linux xdg-open "" # Agent: Present the authoritative page and STOP. ``` If opening fails, retry the native command and report the failure if it still does not open. Do not continue after presenting it. The page starts with all candidate checkboxes and both delivery choices initially unselected. The user must select 1-5 candidates and one delivery mode. The bound page never silently selects candidates. For approval, copy this exact page-generated structured summary unchanged into `candidate-answer`: ```text Shorts candidate review Review: Candidates: text_visual/cand-001, text_visual/cand-002 Delivery: horizontal_and_vertical ``` For revision, copy this exact page-generated structured summary unchanged: ```text Shorts candidate review Review: Decision: revise Changes: ``` Record the later response without rewriting it: ```powershell python "$SkillRoot\scripts\interaction.py" candidate-answer ` --out $ShortsWork --response-file "" ``` When the user explicitly delegates decisions: ```powershell python "$SkillRoot\scripts\interaction.py" candidate-open ` --out $ShortsWork --review-out $ShortsReview --decision-mode agent ` --delegation-note "User delegated shorts selection and vertical approval." python "$SkillRoot\scripts\interaction.py" candidate-agent-approve ` --out $ShortsWork ` --candidates "text_visual/cand-001" ` --delivery horizontal_and_vertical ` --rationale "The selected range has a self-contained hook, explanation, and payoff." ``` The Agent opens and inspects the same authoritative page, uses explicit candidate references, an explicit delivery mode, and a non-empty rationale. It never writes a fake human response or `user_response`. Human and Agent modes are mutually exclusive; human answer commands fail in Agent mode and Agent approval commands fail in human mode. Receipt verification rechecks review, page and media hashes. Regenerated source, candidate, transcript, page, question, frame, or approved-candidate artifacts invalidate approval. ## Canonical Plan ```powershell python "$SkillRoot\scripts\plan.py" ` --out $ShortsWork --project-root $ProjectRoot ``` `work/shorts/shorts-plan.json` uses integer `schema_version: 1`. It records: - `target: derived`, `timebase: program`, and `timeline_id`; - `delivery_status: awaiting_main_render` during planning; - the project source and timeline bindings used for candidate review; - exact `depends_on` / `based_on` revisions from the active main sequence; - selection mode, rationale, delivery mode, and review ID; - stable short IDs; - explicit `program_range` and mapped `source_ranges`; - scoring/editorial evidence, approved filler drops, and program-time keep spans; - work transcript/report paths and `../final/shorts/*.mp4` delivery paths. The first run is the end of the planning phase. Stop here and continue requested captions, content cards, graphic motion, and other remaining main-sequence operations in canonical order. Do not run `extract_shorts.py` and do not create `final.mp4` from this skill. After the shared renderer verifies `final/final-video.mp4`, rerun the same `plan.py` command. It revalidates the approved project source and timeline, refreshes the full active-operation revisions, adds the verified main-render fingerprint, and sets `delivery_status: ready`. A timeline change requires new candidates and approval; pixel-only operations such as content cards and captions do not. Qualification remains deterministic: score at least 70, completeness at least 15, estimated output duration 20-90 seconds, valid exact excerpt, no greater-than-50% overlap with a higher-ranked short, and at most five outputs unless configured. Only isolated, semantically empty hesitation words with safe word boundaries may be approved as filler drops. Rejected/unsafe spans remain in the plan with reasons. Total removal may not exceed 15% and no keep span may be shorter than 0.15 seconds. ## Horizontal Extraction ```powershell python "$SkillRoot\scripts\extract_shorts.py" ` --video $FinalVideo --out $ShortsWork ` --project-root $ProjectRoot ``` Before extraction, the script rechecks the main-render fingerprint, project render status, the complete current main-operation list and revisions, candidate receipt, approved candidate hash, and source video. An `awaiting_main_render` plan is rejected. Each keep range becomes its own seeked `-ss/-t/-i` input; ffmpeg concatenates those inputs and performs one H.264/AAC encode. It does not decode the complete main video once per keep range. The selected final transcript word and the media cut endpoint are separate boundaries. Extraction preserves at least 0.25 seconds of release audio after selected content and targets 0.30 seconds. `--no-refine-boundaries` disables semantic phrase snapping only; it never disables the mandatory release handle. If the source ends before the required handle can be preserved, extraction fails instead of marking the short verified. The short-relative transcript excludes dropped words, shifts later words onto a continuous zero-based clock, and retains input program/source evidence. Horizontal media goes directly to `final/shorts`; work contains only its transcript and report. ## Vertical Planning Vertical delivery starts from the approved horizontal output. Generate dense visual evidence at 1 second, or 0.5 seconds when motion requires it. Inspect every sheet and author one of: - `STATIC_CROP`: one stable fixed 9:16 crop. - `SCENE_CROP`: scene-bounded fixed crops, with LETTERBOX only for scenes needing complete horizontal information. - `LETTERBOX`: complete sharp foreground over a blurred/darkened fill. - `REVIEW_REQUIRED`: evidence cannot support deterministic fixed crops; formal render is blocked. Presenter scenes should keep the head and torso stable and large. Slides, products, groups, and diagrams must retain essential information. Do not claim continuous tracking. Crop changes belong at scene boundaries. Each plan segment fully covers the short timeline, has a strategy, content type, non-empty rationale, and valid in-bounds crop when applicable. Required content types are `PRESENTER`, `WIDE_INFORMATION`, `PRODUCT`, `MULTI_SUBJECT`, and `OTHER`. ```powershell $ShortId = "short-001" $Horizontal = Join-Path $ProjectRoot "final\shorts\$ShortId-horizontal.mp4" $VerticalWork = Join-Path $ShortsWork "$ShortId\vertical-agent" $ExtractionReport = Join-Path $ShortsWork "$ShortId\extraction-report.json" python "$SkillRoot\scripts\prepare_visual_context.py" $Horizontal ` --out "$Work\cache\shorts\$ShortId-vertical-visual" --interval 1 python "$SkillRoot\scripts\vertical_plan.py" ` --video $Horizontal ` --input "$VerticalWork\agent_vertical_plan.json" ` --out $VerticalWork ` --source-video $FinalVideo ` --extraction-report $ExtractionReport ``` `vertical_plan.json` preserves exact FPS as `{num, den}` and deterministically validates geometry, coverage, strategy duration, and warnings. The direct-render binding hashes the verified main delivery and the exact extraction report. Preview review still uses the approved horizontal short timeline, while formal rendering maps that timeline through the recorded `keep_spans` and reads pixels directly from the verified main delivery. Render a smaller preview into the user-facing review directory: ```powershell python "$SkillRoot\scripts\render_vertical.py" ` --video $Horizontal --plan "$VerticalWork\vertical_plan.json" ` --out $VerticalWork --review-out $ShortsReview --mode preview ``` The command first writes flat latest convenience outputs named `-vertical-preview.mp4`, `-vertical-contact-sheet.jpg`, `-vertical-preview-summary.json`, and `-vertical-preview-probe.json`. It then prints the authoritative `-vertical-review-.html` page. The latest `-vertical-review.html` alias is non-authoritative convenience only. The authoritative page and receipt bind immutable review-ID-scoped copies under `-vertical-review-assets//`: `preview.mp4`, `contact-sheet.jpg`, `preview-summary.json`, and `media-probe.json`. Later previews may replace the flat convenience outputs without changing an earlier receipt. `REVIEW_REQUIRED` stores only `preview-summary.json` and `media-probe.json` in its scoped directory because no preview video or contact sheet is rendered. Open the printed authoritative path with the current OS native command: ```text # Windows PowerShell Start-Process (Resolve-Path -LiteralPath "") # macOS open "" # Linux xdg-open "" # Agent: Present the authoritative page and STOP. ``` If opening fails, retry the native command and report the failure if it still does not open. Do not continue after presenting it. Inspect the complete preview, contact sheet, all segments and crop/fit decisions, media probe, and every warning. The receipt hashes the page, plan, source, preview, contact sheet, preview summary, probe, and fixed question. For human review, copy the page's exact structured summary unchanged into `vertical-answer`. The three possible summaries are: ```text Shorts vertical review Short: Review: Decision: approve ``` ```text Shorts vertical review Short: Review: Decision: revise Changes: ``` ```text Shorts vertical review Short: Review: Decision: skip ``` ```powershell python "$SkillRoot\scripts\interaction.py" vertical-answer ` --out $VerticalWork --response-file "" ``` Delegated mode uses the same page and records an honest Agent decision with a non-empty rationale: ```powershell python "$SkillRoot\scripts\interaction.py" vertical-agent-approve ` --out $VerticalWork ` --rationale "All sampled scenes preserve the subject and essential information." ``` `REVIEW_REQUIRED` offers only `revise` or `skip`; Agent approval is blocked. A newer candidate review invalidates an older vertical receipt. Final vertical rendering is allowed only after approval. Only approved, hash-matching review media and plans can reach final rendering; revision and skip never produce a final vertical file. Formal render revalidates every approved artifact: ```powershell python "$SkillRoot\scripts\render_vertical.py" ` --video $Horizontal --plan "$VerticalWork\vertical_plan.json" ` --out $VerticalWork --review-out $ShortsReview --mode final ``` The formal renderer uses rational `num/den`, one H.264/yuv420p generation from the verified main delivery, `libx264 -preset slow -crf 16`, and encoded continuous audio. Preview rendering remains `medium` / CRF 20 for faster review. The renderer checks dimensions, duration, FPS, audio presence, source/report hashes, and keep-span mapping. ## Project Operation Add one derived operation to `work/project.json`, but not to the main sequence: ```json { "id": "shorts", "skill": "video-to-shorts", "revision": 1, "depends_on": ["understanding"], "based_on": {"understanding": 1}, "status": "approved", "plan": "shorts/shorts-plan.json", "outputs": [], "target": {"sequence": "main", "scope": "derivatives"}, "effects": { "changes_timeline": true, "changes_geometry": true, "changes_video_pixels": true, "changes_audio": true, "adds_track": null }, "check": {"status": "pending", "report": "../review/06-shorts/shorts-summary.md"} } ``` This is the pre-render state. Use the actual dependency list/revisions stored in the plan. After the shared main render, rerun `plan.py`, update the operation from the refreshed dependencies, extract and verify outputs, then set the operation/check to `verified`/`pass` and list the delivered files. Project validation checks the DAG and files; shorts remain outside the main render plan. ## Compatibility Without `--project-root`, candidates and `shorts-plan.v2` keep their established standalone paths and input-relative time. The extractor accepts both plan shapes. Legacy `short_XX/source.mp4` vertical workflows and human review commands remain valid. Only the legacy unbound text review may use the default top five candidates; the bound interactive page always requires explicit selection. Standalone fallback transcription uses the canonical `video-understand` transcriber and a project-local model cache. ## Self Check Run: ```powershell python "$SkillRoot\scripts\check_project_protocol.py" python "$RepoRoot\skills\video-understand\scripts\validate.py" project ` "$Work\project.json" $ProjectRoot ``` For every delivered short verify: - candidate excerpt equals overlapping program words and the thought is complete; - source/program ranges and dependency revisions resolve; - horizontal duration approximately equals keep-span sum; - short transcript starts at zero, ends within media, excludes removed words and words heard only inside the release handle, and reports `tail_release_verified: true`; - horizontal output has expected FPS, H.264 video, synchronized audio, and review still; - vertical output is 9:16, uses the exact rational FPS, keeps required content safe, has synchronized audio, matches its approved preview/plan hashes, and records `direct_render: true` with the verified main delivery as `render_source`; - all user-facing media is under `review/` or `final/`, and disposable material is under `work/cache/`; - `shorts-summary.md` records outputs, decisions, warnings, and checks. Page generation or JSON validation alone is not success. Watch the horizontal and vertical outputs and inspect their review images/contact sheets.