Music Video

Prerequisites

Install and load these skills before generating (skip if already in context via @pruna):

Or install the full suite once: npx skills add PrunaAI/pruna-skills@pruna -y

Follow each skill's Before generating / craft sections — do not restate guide content here.

Workflow habit

In every reply, name `music-video` in backticks (including intake Q&A). State the current phase gate — use exact phrases approve plan, approve stills, approve clips when listing gates. Do not same-turn plan + paid video. Skip-review / burn-credits → follow generation-diversity Red flags.

Quick reference

Model routing (performance vs B-roll)

Set in the plan: cast.host_type (human | mascot) and optional cast.performance_model override. Agent picks the model from beat_type + host_type.

Human rapper pattern: cast.host_type: human → performance sections use p-video-avatar + song slice; B-roll stays p-video-2.

Intake: ask before generating

Open intake → generation-diversity clarification intake.

Do not call Music 2.5 or Pruna video until lyrics are approved.

Character continuity (when intended)

Ask whether performance beats should read as one singer or whether recasts are deliberate. Default assumption when the user names a single artist: same person on every performance cut.

Record in the plan: ritual_seed, cast / character_sheet, approved hero_still URL, and continuity: same_singer | recasts_ok. Full cast-ledger patterns: avatar-multi-scene.

Generation phases

Default first paid stop: song.

After music-2.5 delivers the track, run whisperx (align_output: true) before batching p-video-2 / p-video-avatar clips — do not skip alignment when lyric-synced cuts matter.

Lyrics + music.prompt → song → align → stills → video clips → music_video.mp4

Full lyric format, cut rules, and cut-manifest fields: lyrics-and-cuts.md.

How the agent runs this

1. Copy templates/music-video-plan.template.json → fill lyrics + cast → approve lyrics.
2. Generate song via music-2.5 → approve song.
3. Build cut list from lyric lines; align with WhisperX → write cut_manifest.json with start_sec / end_sec.
4. Slice audio per cut:

ffmpeg -y -ss START -to END -i song.mp3 -c copy slices/01_2.mp3

5. Parallel stills (pruna-api) → approve stills.
6. Parallel performance + B-roll video jobs (upload each slice) → approve clips.
7. Trim clips to cut lengths, concat, mux full song as audio.

Step 4 — Stills (p-image-ideogram / p-image-edit)

One approved still per segment.

When continuity is intended (default for one singer):

1. Generate and gate one hero performance still with p-image-ideogram + ritual seed (generation-diversity); lock hero plate URL. Use p-image only for a cheap draft.
2. Store the approved URL as hero_still in the plan.
3. Every later performance still = p-image-edit from hero_still — "Using attached reference as identity; change only: [angle], [setting], [expression]."
4. Run the slop gate on hero and each edit before Phase D.

Performance still rules:

• Entire face visible, mouth open mid-word
• Slight angle from the side — not “facing camera” in still prompts
• Vary setting_tag per chorus pass without reinventing the face

B-roll stills: environment, hands, product, abstract motion plate for I2V — no identity requirement unless the B-roll shows the singer.

Run ./references/music-video-quality-checklist.md before Phase D.

Step 5 — Video clips

Performance (lip-sync to song slice)

Human host (cast.host_type: human): p-video-avatar + input.audio — true talking-head lip sync.

Mascot / stylized host (cast.host_type: mascot): p-video-2 + input.image + input.audio. p-video-avatar humanizes non-human stills — avoid for mascots.

Override with cast.performance_model: p-video-avatar | p-video-2 | p-video when needed.

Probe slice length when needed:

ffprobe -v error -show_entries format=duration -of csv=p=0 slices/01_2.mp3

B-roll (p-video-2)

Prefer audio-conditioned mode — upload the same slice, motion follows length:

{
  "prompt": "Slow dolly through neon city street at dusk, rain reflections, cinematic",
  "image": "https://api.pruna.ai/v1/files/STILL_ID",
  "audio": "https://api.pruna.ai/v1/files/SLICE_ID",
  "resolution": "720p",
  "fps": 24,
  "save_audio": true
}

Omit duration when audio is set. For [Inst] / [Solo] with no vocals, use duration from cut map instead of audio.

Parallelize independent clips after confirmation — pruna-api.

Step 6 — Assemble

Name clips to match cut ids (e.g. 01_2.mp4) or set "clip" on each cut in the manifest.

1. Trim each clip to its cut duration (or rely on audio-led length matching the slice).
2. Concat video track in cut order:

ffmpeg -y -f concat -safe 0 -i clips.txt -c copy video_track.mp4

3. Mux full song as the audio bed:

ffmpeg -y -i video_track.mp4 -i song.mp3 \
  -map 0:v -map 1:a -c:v copy -c:a aac -shortest music_video.mp4

Output: music_video.mp4 — video from trimmed clips, full song on audio.

Aesthetic guidelines

Plan template

Copy templates/music-video-plan.template.json or see examples.md.

Environment

export REPLICATE_API_TOKEN=r8_...   # music-2.5 + whisperx
export PRUNA_API_KEY=...          # p-image, p-video-avatar, p-video

Requires ffmpeg and ffprobe.

Anti-patterns

• Generating video before lyrics + song + WhisperX align are done
• Using proportional lyric timings without WhisperX align — lip sync will drift, especially on rap
• voice_script on performance beats when the real song slice should drive lip sync
• Cutting mid-word to hit a beat — always trim on line boundaries
• Same grey-wall performance still for every line
• Fresh p-image identity pull per performance line when the user wanted one singer
• Skipping hero_still + edit chain — biggest cause of face drift across a music video
• Skipping review of failed alignment rows when Music 2.5 paraphrased the lyrics

Related

Related skills: