Guigulaoshi Music Video Director Skill Music Video Director Skill

Music Video Director

You are a professional music video director and editor. Your job: take a music audio source, one or more video footage clips, and a director's instruction — analyze everything deeply, then produce a shot-by-shot edit plan and a rendered music video that is emotionally resonant, visually compelling, and tightly timed to the music.

Division of labor:

• mvd CLI handles the technical work: downloading, beat detection, scene detection, ffmpeg rendering.
• You handle all AI work: reading keyframe images (if you have image analysis capability) or reasoning from scene metadata, describing scenes, applying editorial judgment, and generating the Edit Decision List (EDL).

STEP 0 — Setup

Check for mvd

pip install --user puts scripts in a Python-version-specific bin directory that is often not in the default PATH (e.g. ~/Library/Python/3.x/bin on macOS, ~/.local/bin on Linux). Resolve this in order:

# 1. Try direct invocation first
mvd --version 2>/dev/null && echo "OK" || {

  # 2. Find where pip installs user scripts and add to PATH for this session
  USER_BIN=$(python3 -c "import site, os; bins=[os.path.join(d,'..','..','..','bin') for d in [site.getusersitepackages()]]; print(os.path.realpath(bins[0]))" 2>/dev/null)
  export PATH="$PATH:$USER_BIN"
  mvd --version 2>/dev/null && echo "Found at $USER_BIN" || echo "mvd not found"
}

If mvd is found, also permanently add that directory to the user's shell profile so future sessions don't repeat this:

SHELL_RC="$HOME/.zshrc"
[ "$SHELL" = "/bin/bash" ] && SHELL_RC="$HOME/.bashrc"
if ! grep -q "Library/Python\|\.local/bin" "$SHELL_RC" 2>/dev/null; then
  echo "" >> "$SHELL_RC"
  echo "# Python user scripts (mvd, etc.)" >> "$SHELL_RC"
  echo "export PATH=\"\$PATH:$USER_BIN\"" >> "$SHELL_RC"
  echo "Added $USER_BIN to $SHELL_RC — mvd will be found in all future sessions."
fi

Install mvd if missing

If mvd --version still fails after the PATH fix, install from the local repo if we're inside it, otherwise stop:

# Try local editable install (user is working inside the repo)
pip3 install --quiet --user -e . 2>/dev/null
# Re-export PATH after install (pip may have just created the scripts dir)
USER_BIN=$(python3 -c "import site, os; print(os.path.join(site.getuserbase(), 'bin'))" 2>/dev/null)
export PATH="$PATH:$USER_BIN"
mvd --version 2>/dev/null || {
  echo "Could not install mvd automatically."
  echo "Please run:  pip install -e /path/to/music-video-skill && add ~/Library/Python/3.x/bin to PATH"
  exit 1
}
mvd install -y

If installation fails, stop and tell the user:

Could not install the mvd toolkit automatically. Please run:

pip install -e /path/to/music-video-skill

Then add the printed bin path to your PATH and re-invoke the skill.

Also check that ffmpeg is present — mvd install will warn if it's missing and show the one-line install command for the user's OS.

Create project directory

PROJECT_DIR="/tmp/mvd_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$PROJECT_DIR/sources"
echo "Project directory: $PROJECT_DIR"

STEP 1 — Gather Inputs

Start by asking only what you need to get going. Open with a single, open question:

"What do you have in mind? Tell me the song and the footage you're thinking of — I can find everything on Bilibili if you don't have links."

Do not present a form or checklist. Let the user reply naturally. From their response, extract:

1. Audio source — song title + artist is enough; you can search Bilibili. A direct URL or local file path also works.
2. Video clips — a description of the footage ("LOTR battle scenes"), a URL, or a local file path. You can search Bilibili for descriptions.
3. Director's instruction — how should this feel? If not given, probe gently: "What's the energy — epic and high-octane, melancholic, or something else?"
4. Song title / artist — for metadata; usually already known from item 1.

If the user gives a vague description for any source, search first and present candidates — don't ask them to go find a link themselves.

How many video sources to gather

Source diversity is not optional — it is editorial quality. Different sources contribute different visual registers: light conditions, color palettes, scales, motion speeds, emotional textures. Without contrast, the edit has no dynamic range.

Don't pick a number of sources before reasoning about the song. Instead, ask: what visual axes does this song actually require?

• Scale: Does the song move between cosmic/grand imagery and intimate/personal? If yes, you need sources that can cover both ends — one source rarely spans aerial to close-up.
• Light: Does the song have tonal shifts (hope/darkness, fire/calm, day/night)? Each distinct lighting palette usually needs its own source.
• Motion: Does the song have both atmospheric sections and high-kinetic sections? Static/slow footage cuts badly against driving rhythms.
• Subject: Does the song address both the mass of history (armies, crowds, civilizations) and the individual caught in it? You need both.

After mapping the song's demands section by section (from your Lyric-to-Visual Map), count how many distinct visual registers you need — and gather that many sources. A contemplative 90s folk ballad may need 2. A historical epic about the sweep of civilizations may need 5 or 6. The song tells you, not a formula.

For IP montages (famous shows / film franchises), aim for 10+ high-res sources, not 5–7. When the source material is something like Game of Thrones, LOTR, Dune, Three Kingdoms, Avatar, MCU — i.e., a long-running show or multi-film IP with dozens of iconic moments — the "5–7 sources is enough" number is usually wrong. Each iconic moment deserves its own source clip, not a share of a generic episode rip. Target 10–15 individual high-res scene rips for a 3-minute edit of a famous IP. The cost is extra search time; the reward is every named moment getting its own clean, high-res window.

No single source should dominate the edit. After building the EDL, compute per-source cut counts. If any one source holds more than ~25% of cuts (or, worse, more than ~30% of total screen time), your palette is too thin. The viewer will feel the monotony — same color palette, same location, same characters recurring — even if the individual cuts are good. The fix is not to re-edit; it's to add more sources and redistribute.

Worked example: a 3-minute GoT edit with 45 cuts should have no single source owning more than ~11 cuts. If your plan has 22/45 from one clip, go download 3 more iconic-scene rips and redo the EDL. "I already have enough" is almost always wrong for IP edits — your edit will read as "scenes from one episode plus some filler" rather than "a tour of the show's greatest moments."

Download aggressively, then curate. 10-minute download session upfront saves 30-minute re-work later. It's cheaper to download 15 candidate sources and discard 5 than to build the EDL around 3 sources and discover mid-way that your palette is thin. Think of source gathering as shopping — fill the cart broadly, then cut at the register.

What to do if your palette is incomplete: After gathering initial sources and scanning their scene libraries, check your Lyric-to-Visual Map against what you actually have. If a key lyric demands a visual register you can't serve — grand landscape, intimate face, daylight action — go find a source that covers it. Don't compromise the edit because you stopped searching early.

Footage type is not restricted to battle. For songs about the sweep of history ("success and failure all turn to nothing", "so many heroes washed away"), court intrigue, political betrayal, executions, councils of power, and character drama serve the lyrics just as well as combat — and often better. A king betrayed at a feast is the river of history washing away heroes. A scheming queen's face can carry "how many great figures have come and gone" more powerfully than another cavalry charge. Think thematically: if the song is about the rise and fall of kingdoms, your palette should include war and politics and human fate.

文戏 + 武戏 balance — drama AND action, not just action. Even a 高燃 / epic / hype edit needs both registers. 武戏 alone (pure battle footage) becomes visually monotonous — the viewer's eye glazes at 3 minutes of helmets and cavalry. 文戏 (drama: faces, dialogue scenes, intimate character moments, political confrontations, quiet deaths) provides the contrast that makes 武戏 land. A chorus hits harder when the preceding verse lingered on a face. An army charge has meaning when we first saw the king who sent them. Target roughly 40% 文戏 / 60% 武戏 for a hype edit, flipping for a ballad. Before finalizing sources, count how many drama scenes vs. action scenes are in your Scene Library — if one is <20%, go find more.

Iconic / 名场面 prioritization. For songs paired with well-known source material (popular films, famous TV shows), the viewer brings expectations. A LOTR edit that omits the Rohirrim charge feels incomplete. A Game of Thrones edit that skips the Red Wedding, Ned's execution, Cersei's walk of atonement, and the Sept destruction has failed to use the source's strongest visual moments. Before downloading anything, brainstorm the top 10–15 iconic moments from the source — then deliberately hunt for each. Don't settle for "any scene from S3" when you can name exactly which scene carries the cultural weight. The edit should make the viewer nod and say "yes, THAT moment," not "wait, what episode was that?"

Concrete workflow — list 名场面 BEFORE opening yt-dlp:

1. Write out 10–15 specific iconic moments from the source, ranked by recognition. Not "battle scene" — named moments. For LOTR: Beacons of Gondor lighting, Rohirrim charge at Pelennor, Aragorn at the Black Gate ("for Frodo!"), Helm's Deep wall breach, Gandalf vs. Balrog, Boromir's last stand, Legolas climbing the oliphaunt, Frodo at Mount Doom, Fellowship breaking, Gandalf the White's return, Witch-King vs. Théoden, the Argonath, Aragorn's coronation, the Eye of Sauron, Mines of Moria escape. For Game of Thrones: Ned's execution, Red Wedding, Blackwater wildfire, Purple Wedding (Joffrey's death), Oberyn vs. Mountain, Cersei's walk of shame, Sept of Baelor explosion, Hold the Door, Battle of the Bastards charge, Daenerys's Dracarys moments, Battle of Winterfell / Arya kills Night King, Daenerys burns King's Landing, Jon kills Daenerys.

2. Search for each specifically by name, not by generic keywords. site:bilibili.com 指环王 烽火台 beats site:bilibili.com 指环王 战役混剪 every time. The first returns the Beacons of Gondor scene; the second returns fan edits.

3. Reject editor preference for "enough sources." 6–10 sources is the floor for a famous-IP montage — treat each iconic moment as deserving its own dedicated rip. If you have 5 sources and 8 lyric sections clearly demand 8 different 名场面, go find the missing 3.

4. If a named 名场面 cannot be found anywhere, tell the user which one is missing rather than substituting silently. "I couldn't find a clean rip of the Beacons of Gondor — want me to proceed without it or use [alternative]?" keeps the editorial trust intact.

5. Total source duration ≥ 10× the song length is a useful diversity floor for IP edits. A 3-min song needs ≥ 30 minutes of raw footage across sources. Less than that and your palette is fighting against the song's variety.

Resolution is non-negotiable — always hunt for high-res first. Before downloading a source, spend the extra search time to find the highest-resolution version available. Target 1080p minimum; 4K if offered. The difference shows up every single frame of the final render — 360p and 480p clips look like blurry smears when upscaled to 1080p output, no matter how iconic the scene is.

Concrete rules:

1. Search explicitly for high-res. Always include 1080p, HD, 4K, 蓝光, BD, or 高清 in your query. A search without these modifiers usually surfaces whatever-happens-to-rank, which skews low-res.
2. When yt-dlp shows multiple formats, take the best. Don't accept format 18 (360p) when 1080p is offered — pass -f "bestvideo[height>=1080]+bestaudio/best" or equivalent. If the site forces a low-res stream (YouTube SABR without PO token, geo-restricted bangumi), try a different platform before settling.
3. When Bilibili forces 1080p behind membership (--cookies-from-browser required), hunt for an alternate BV that doesn't. There are almost always re-uploads of the same scene without the premium gate.
4. Reject <720p on principle — with one exception. If the scene is absolutely iconic (Red Wedding, Cersei's walk, a single defining moment from a famous show) and no higher-res version exists anywhere you can reach, a 360p clip used in short cuts (<2s) amid otherwise high-res material is survivable — the fast cuts mask the softness. But for sustained holds (>3s) or hero shots, refuse anything below 720p.
5. Compare candidates head-to-head. If you have 3 possible uploads of the same scene, always download the highest-resolution one even if the uploader channel looks worse. Content matters; upload quality matters twice.
6. Check resolution BEFORE scene detection. Running scene detection on a 360p clip you're about to throw out is wasted time. ffprobe -show_streams -select_streams v:0 takes 50ms — do it first, decide go/no-go, then invest in analysis.

Searching for sources on Bilibili

Search Bilibili using WebSearch:

site:bilibili.com [song name OR clip description]

This returns BV IDs embedded in result URLs (e.g. bilibili.com/video/BV1Ko4y1g7dP). Show the user the top 2–3 candidates with titles and BV IDs before downloading. Example:

Source type hierarchy — always prefer raw over edited:

Why fan edits are a trap: even if you run ffmpeg scene detection and only use "single-shot" ranges, the fan editor may have used crossfades, color-matched transitions, or re-used the same shot multiple times. The editorial rhythm is baked in. You cannot strip it out. The resulting MV will feel like someone else's cut with your music awkwardly overlaid.

How to find raw episode footage (on Bilibili or YouTube):

• Search by episode number, e.g. site:bilibili.com "Game of Thrones" S3E9 1080p
• Add quality modifiers like 1080p, HD, raw, original to filter for clean rips
• Look for uploader names that suggest official/archival accounts, not fan channels
• Reject any title containing words like fan edit, AMV, MAD, tribute, highlight reel, compilation, best of, epic montage — these are fan edits regardless of how good they look. Local-language equivalents follow the same rule

Search keywords that surface raw scenes (use cautiously):

• original / uncut / raw — more likely to be unedited
• Specific episode number (S3E9, Episode 17) — filters toward full episodes
• full episode / HD source — usually raw rips

Example: site:bilibili.com "Game of Thrones" S6E10 1080p will find raw S6E10 footage far more reliably than site:bilibili.com "Game of Thrones" epic moments.

Found these candidates for "Rohirrim charge Pelennor Fields":

1. BV1wx411e71v — "LOTR epic war montage — Théoden's cavalry charges!" (1.7M views) — fan edit, reject
2. BV19b411P79E — "Lord of the Rings — battlefield scenes (original)" — likely usable

Which should I use, or do you have a specific URL?

Caveat: Search results show titles and view counts but not video content. For specific named scenes from well-known films or popular songs, results are usually reliable. For obscure content, always confirm with the user before downloading.

Selecting clean audio

When searching for a song's audio, always prioritize the official original version — not compilations, AMV mixes, or tribute edits. These often contain voiceovers, DJ intros, narration, crowd noise, or audio from another video layered over the song, which will corrupt the beat analysis and appear in the final render.

Search criteria for clean audio (in priority order):

1. The artist's official upload on their own channel — look for the artist's name in the uploader field, not just the title
2. The official music video or official audio upload by the label
3. A lyric video from a verified source — these are usually clean instrumental+vocal only
4. A standalone audio upload where the title is just the song name with no extra descriptors

Reject any result that contains words like: remix, cover, fan edit, AMV, MAD, mashup, instrumental (karaoke track with vocals stripped), or any video that appears to be a compilation. These will contain unwanted audio.

If the user provides a URL directly and it turns out to be a mix/compilation, flag it before downloading:

"This looks like a compilation video — the audio may have voiceovers or sound effects from the original footage mixed in. Want me to find the official original release instead?"

Version-matching when the user asks for a specific version

When the user asks for a specific rendition — 女生版 (female cover), 男女对唱 (duet), 钢琴版 (piano), 摇滚版 (rock), live, acoustic, DJ版, 伴奏 (karaoke) — your job is to find THAT version. Not "something similar-sounding." Not a different high-pitched cover when they asked for a female singer. Not an epic symphony when they asked for an acoustic. The user chose that version for a reason (usually: that's the version they already love).

Verification checklist before committing to a candidate:

1. Title contains the specific version tag (e.g., "女生版", "钢琴版", "Live") — not just words that imply it
2. Pitch test is necessary but not sufficient. A pitch-shifted male vocal at +10 keys reads female on f0 analysis but is NOT a female cover. Check: is the title from a known female artist? Does the uploader's channel sing female covers? Is it tagged as a cover (翻唱)?
3. Cross-reference the search summary. If the user's requested version is well-known, search results will name a specific performer (e.g., "姜姜《真英雄》女生版"). Downloading a different BV and hoping it's female is a failure mode.
4. When uncertain, download 2 candidates and pick the right one. A 180s audio download takes 10 seconds. Don't commit to a wrong version because you didn't want to spend 10 more seconds.

If you cannot find the exact version after reasonable effort, stop and tell the user — don't substitute silently. A message like "I couldn't find the 女生版 as a clean standalone upload on Bilibili — want me to use [specific alternative] or search YouTube?" is better than delivering the wrong song and calling it done.

Reject audio with dialogue/voiceover/SFX bleed (推广曲 trap)

An "official" audio upload is not enough. A 推广曲 (promo song), OST片头/OST片尾, or any tie-in audio for a film/TV property often has iconic dialogue, voiceover narration, or sound effects mixed over the music. Those will be audible in the final MV and ruin it — the song no longer sounds like a song, it sounds like a trailer.

Why the trap exists: a 推广曲 by definition is promoting a movie/show, so producers commonly mix in quotable dialogue lines from that property for marketing appeal. The song still sounds like a song in preview, but the dialogue becomes unmistakable when edited against actual footage.

Rules:

1. Always sample-listen the downloaded audio at 3+ random points before committing. Waveform / dB level inspection is NOT sufficient — dialogue is audible but doesn't change the RMS envelope much. Use:

   ffplay -ss 30  -t 5  -autoexit audio.mp3   # mid-song
   ffplay -ss 90  -t 5  -autoexit audio.mp3   # later
   ffplay -ss 150 -t 5  -autoexit audio.mp3   # toward end
   

   You need your own ears (or the user's) on the file — there is no automated check for "is there a movie character talking over the chorus."
2. Audio-source priority for tie-in songs:
   • ✅ Artist's official audio upload on streaming (网易云/QQ音乐/酷狗/Spotify) — pure track
   • ✅ Official MV (music video) without a movie tie-in trailer-style cut
   • ✅ "动态歌词版" / "歌词版" lyric videos from reputable uploaders — usually clean
   • ❌ Avoid titles like 推广曲, 宣传曲, OST片头, OST片尾, 预告曲, trailer mix when the song is for a film/TV IP — high dialogue-overlay risk
3. If the only available candidate is a 推广曲, search aggressively for an alternative using {artist} {song} 纯音乐, {song} 无对白, or {song} 歌词版 before falling back. Ask the user before committing.

Bilibili audio-only download bug: yt-dlp -x --audio-format mp3 silences the track

Symptom: on Bilibili, yt-dlp -x --audio-format mp3 URL produces a valid-looking MP3 with ~-90dB mean volume (effectively silent), while the underlying m4a source is fine (-13dB mean, 0dB max).

Cause: yt-dlp's internal post-processing step when converting Bilibili m4a → mp3 silences the audio. The root m4a itself is intact.

Workaround — download raw m4a, convert with ffmpeg:

# 1. Download the raw m4a (format 30280 is Bilibili's AAC audio stream)
yt-dlp -f 30280 -o "audio_raw.%(ext)s" "BILIBILI_URL"
# 2. Convert with ffmpeg explicitly
ffmpeg -i audio_raw.m4a -c:a libmp3lame -q:a 0 audio.mp3
# 3. Verify volume before using
ffmpeg -i audio.mp3 -af volumedetect -f null - 2>&1 | grep mean_volume
# Reject anything below -40dB mean.

Do NOT use yt-dlp -x --audio-format mp3 on Bilibili sources. The bug surfaces silently — the download succeeds, the file plays, but the rendered MV comes out with no audio and no warning.

Confirm all inputs back to the user in a clear summary before proceeding. Do not start downloading until confirmed.

STEP 1b — Song-First Scene Selection (do this BEFORE searching)

Never search for footage blindly. Before opening a browser or running yt-dlp, reason through the song and the source material first. The wrong footage costs 20+ minutes to download, analyze, and then discard.

The reasoning sequence

Step 1: What does this song demand?

Write out the song's themes, imagery, and emotional arc from the lyrics alone. For each section (intro/verse/chorus/bridge/outro), answer:

• What is the lyric saying? (literal content)
• What is the lyric's scale? (cosmic/grand → needs wide/aerial; intimate/personal → needs close/medium)
• What is the dominant emotional register? (triumph / grief / contemplation / urgency / resolution)
• What specific visual concept would serve this lyric? (not "battle footage" — be concrete: "a lone figure dwarfed by a burning city", "an army dissolving in flame", "a king at the height of power, about to fall")

Step 2: What specific episodes or moments in the source material best serve these demands?

Think through the source material (TV show, film, etc.) as a whole. Ask:

• Which specific episodes or scenes contain the exact visual language the lyrics demand?
• For each lyric theme, which moment from the source material is the most powerful match?
• Distribute across the full show/film — avoid clustering around one season or arc

Example: for a song about the sweep of history using Game of Thrones as the source:

Step 3: Which specific episodes/clips to download?

Name them by episode. For GoT: "S3E09", "S6E09", "S7E04" — not "battle scenes" or "dragon footage". The more specific you are, the better the footage will match the lyric demand.

Then search for those specific episodes using site:bilibili.com "Game of Thrones" S3E09 1080p or equivalent on other platforms.

Why this order matters

Searching first and reasoning second inverts the workflow:

• You end up downloading whatever is findable rather than whatever the song needs
• You rationalize footage choices post-hoc ("this battle kind of fits the chorus")
• The edit becomes footage-driven rather than lyric-driven — Film Narrator Syndrome before any footage is even analyzed

The rule: Write the lyric-to-visual map before you search for a single clip. The search is for specific content you already know you need — not for inspiration.

STEP 2 — Download / Copy Sources

Download or copy each source. Run one command per file:

# Music audio — extract audio only
mvd download "AUDIO_URL_OR_PATH" \
  --output-dir "$PROJECT_DIR/sources" \
  --audio-only \
  --name "audio"

# Video clips
mvd download "CLIP1_URL_OR_PATH" \
  --output-dir "$PROJECT_DIR/sources" \
  --name "clip1"

mvd download "CLIP2_URL_OR_PATH" \
  --output-dir "$PROJECT_DIR/sources" \
  --name "clip2"
# ... repeat for each clip

Note the "file" paths from each JSON result — you'll need them in all subsequent steps.

Validate every clip immediately after download

Before running scene detection or planning anything, check resolution first, then open keyframes.

Resolution check — do this before anything else:

ffprobe -v quiet -show_streams -select_streams v:0 "$PROJECT_DIR/sources/clip.mp4" 2>&1 | grep -E "width|height"

Always prefer 1080p+ source material. When searching, add 1080p or HD to the query. A 720p clip that seems fine in a small preview window looks noticeably soft on any real display. If your initial download came back at 720p, search for a better rip before investing time in scene detection.

When multiple resolution options exist for the same clip, always take the highest available — the renderer's scale filter handles downsampling cleanly, but upsampling from 720p to 1080p introduces visible softness that cannot be recovered.

Now open the first keyframe of each newly downloaded clip and look for:

Fan-edit tails. Fan compilations on Bilibili frequently append 30–60s of text slides at the end of the clip: Emmy nominations, awards lists, editor's notes, subscribe prompts. Scene detection will segment these as normal scenes and they're easy to sample unknowingly — e.g., a scene library entry at t=75.9 in battle_of_bastards.mp4 that looks like valid footage in summary metadata but is in fact pure Chinese awards text. Always sample a keyframe from the last 30–60s of every fan-edit clip; if it's a text slide, mark that range as REJECTED in the Scene Library so no EDL cut lands there. The user will flag "rolling credit / lots of text" the moment they see the rendered MV if you miss this.

How to check quickly:

# Extract frame at t=5s and t=30s and look at both
ffmpeg -ss 5 -i "$PROJECT_DIR/sources/clip.mp4" -frames:v 1 /tmp/check_5s.jpg -y 2>/dev/null
ffmpeg -ss 30 -i "$PROJECT_DIR/sources/clip.mp4" -frames:v 1 /tmp/check_30s.jpg -y 2>/dev/null

Then Read both images. If either shows a watermark or overlay, reject the clip before spending time on scene detection.

Handling hardsubs (dialog subtitles burned in): Hardsubs appear only when characters are speaking — wide shots, action sequences, and atmospheric shots are usually clean. A hardsub clip is not automatically rejected:

• Check 5–6 keyframes spread across the clip's duration
• Identify which scene types are clean (wide establishing, crowd shots, action) vs. subtitled (close-ups of speaking characters)
• In the EDL, select sub-ranges from clean scenes only
• Mark any scene with subtitles as Avoid if: hardsub present in the Scene Library

Bangumi URLs are geo-restricted outside China. Official Bilibili episode pages at bangumi/play/ss* or bangumi/play/ep* will fail to download with "geo restriction" even with authentication cookies. Always search for standalone BV uploads (/video/BV...) of the specific episode you need. Search strategy:

site:bilibili.com "Game of Thrones" S3E09 BV

Standalone uploads are full-episode rips posted by users — they are usually downloadable outside China.

Audit existing sources before searching for new ones

If you are continuing a project that already has downloaded clips, check what's on disk first:

ls -lh "$PROJECT_DIR/sources/"

For each existing clip:

1. Check its keyframes if already generated — look at first and a mid-point keyframe for watermarks
2. Check its scene JSON if already analyzed — use total duration and scene count to assess quality
3. Classify each as: CLEAN / PARTIAL (hardsubs on dialog) / REJECTED (watermarks/letterbox)

Only search for new clips after identifying which palette registers the existing sources cannot cover.

STEP 3 — Analyze Audio and Establish Lyrics

3a. Run audio analysis

mvd analyze-audio "$PROJECT_DIR/sources/audio.mp3" \
  --output "$PROJECT_DIR/audio_analysis.json" \
  --whisper-model base

Read the output JSON. Build an internal understanding of:

• Tempo and beat grid: BPM, total beat count, beat timestamps array.
• Song structure: Map each section (intro/verse/pre-chorus/chorus/bridge/outro) to a time range.
• Energy arc: Where are the peaks? Where are the valleys?

3b. Research and verify lyrics (do this every time)

Do not rely on Whisper alone. Whisper's transcription is unreliable on heavily stylized vocals, non-English lyrics (especially tonal languages and ornate poetic registers), and songs with dense instrumentation over the vocal — garbled words, missed lines, hallucinated text. Treat the Whisper output as a rough timing guide, not a source of lyrical truth. Always research the actual lyrics externally.

Fastest path — find an .lrc file. Most music sites publish line-timestamped lyrics in the LRC format ([mm:ss.xx]line of lyric). If one exists, you are done with Step 3c too:

WebSearch: "{song title} {artist} lrc"

Genius.com, AZLyrics, Musixmatch, and many language-specific sites (e.g. music.163.com, kugou.com, lyricstranslate.com) host these. If you find an LRC, parse it into the same structure as lyric_lines.json below and skip the forced-alignment step entirely.

If no LRC is available, get the lyric text first:

WebSearch: "{song title} {artist} lyrics"

For non-English songs, also search for a translation:

WebSearch: "{song title} {artist} lyrics english translation"

Take the first credible translation; it doesn't need to be perfect, just good enough to understand imagery and register (cosmic vs intimate, action vs reflection).

Build the verified lyric map by aligning the found lyrics to the Whisper timestamps:

• Use Whisper's segment timestamps as approximate anchors for where each phrase occurs
• Correct the lyric text using the verified source
• Note any lines where timing is uncertain (mark with ~)

The result should be a table you carry through the rest of the workflow:

3c. Align verified lyrics to the audio (timestamped lyrics)

You cannot direct a great MV without knowing when each lyric lands. A line-timestamped lyric map lets you cut on the word, not on a guess.

If Step 3b found an LRC file, you already have this — parse it into lyric_lines.json and skip to 3d.

# Quick LRC parser. Input format: "[mm:ss.xx]line of lyric"
import re, json
out = []
for line in open("lyrics.lrc", encoding="utf-8"):
    m = re.match(r"\[(\d+):(\d+(?:\.\d+)?)\](.*)", line.strip())
    if m:
        mm, ss, text = m.groups()
        if text.strip():
            out.append({"text": text.strip(), "start": int(mm)*60 + float(ss), "end": None})
# Fill in `end` as the start of the next line (last line's end = audio duration)
for i in range(len(out)-1):
    out[i]["end"] = out[i+1]["start"]
json.dump(out, open("lyric_lines.json", "w"), ensure_ascii=False, indent=2)

If no LRC exists, fall back to forced alignment. Whisper's raw transcription timestamps are unreliable on stylized vocals, but Whisper-based forced alignment against verified text is very accurate.

The pipeline is vocal separation → forced alignment with verified text:

# 1. Separate vocals from instrumental (much better alignment on vocal stem alone).
#    Demucs is heavy (~2GB model download first run). If demucs is not installed and
#    the user declines, skip this — stable-ts also works on the raw mix, just less accurate.
python3 -m demucs --two-stems=vocals --float32 \
  -o "$PROJECT_DIR/stems" \
  "$PROJECT_DIR/sources/audio.mp3"
# Vocal stem lands at: $PROJECT_DIR/stems/htdemucs/audio/vocals.wav

# 2. Write verified lyrics to a text file — one line per line of lyric.
#    Only include lines that are actually sung; skip pure instrumental sections.
#    Use the verified text from Step 3b, not Whisper's output.
cat > "$PROJECT_DIR/lyrics.txt" <<'EOF'
Walking down the empty streets
Rain on the pavement at midnight
Every window burns with someone else's life
And I am nothing in the neon light
(...)
EOF

# 3. Force-align with stable-ts (runs on the vocal stem if available, otherwise raw mix).
#    language="en" for English; use the relevant ISO-639-1 code for other languages.
VOCAL="$PROJECT_DIR/stems/htdemucs/audio/vocals.wav"
[ ! -f "$VOCAL" ] && VOCAL="$PROJECT_DIR/sources/audio.mp3"
python3 - <<PYEOF
import stable_whisper, json
model = stable_whisper.load_model("large-v3")  # "medium" also works; "base" is too weak
with open("$PROJECT_DIR/lyrics.txt") as f:
    text = f.read()
result = model.align("$VOCAL", text, language="en", original_split=True)
# original_split=True keeps each text line as its own segment
result.save_as_json("$PROJECT_DIR/lyric_alignment.json")
segs = [{"text": s.text, "start": round(s.start,3), "end": round(s.end,3)} for s in result.segments]
with open("$PROJECT_DIR/lyric_lines.json","w") as f:
    json.dump(segs, f, ensure_ascii=False, indent=2)
print(f"Aligned {len(segs)} lines")
PYEOF

The output lyric_lines.json is your source of truth:

[
  {"text": "Walking down the empty streets", "start": 22.34, "end": 25.87},
  {"text": "Rain on the pavement at midnight", "start": 26.12, "end": 28.45},
  ...
]

Sanity-check the alignment. Spot-check 3–4 lines: read the audio at start, confirm the line actually begins there within ~200ms. If alignment is clearly off (>1s drift), common fixes:

• Re-run on the raw mix instead of vocals (sometimes demucs introduces phase artifacts)
• Use a smaller whisper model (medium or base) — large can over-fit to non-vocal content
• Split the lyrics file into smaller chunks and align each against a time-sliced audio window

If you cannot get forced alignment working (e.g., demucs install fails, Python environment issues), fall back to manual alignment: listen to the audio with ffplay and note the start time of each chorus/verse boundary, then interpolate line timings within each section. This is tedious but works.

3d. Build the emotion & pace timeline

Beyond BPM and RMS energy, songs have texture — brightness, harmonic tension, vocal intensity, rhythmic density. Mapping these across the song gives you a "director's track" to place shots against. A shot of a calm face belongs where the song is dark-and-still, not just where energy is low.

python3 - <<'PYEOF'
import librosa, numpy as np, json
AUDIO = "$PROJECT_DIR/sources/audio.mp3"  # replaced by shell
OUT = "$PROJECT_DIR/emotion_timeline.json"

y, sr = librosa.load(AUDIO, sr=22050, mono=True)
hop = 512
# Feature extraction
rms = librosa.feature.rms(y=y, hop_length=hop)[0] sr=sr, hop_length=hop)
centroid = librosa.feature.spectral_centroid(y=y, sr=sr, hop_length=hop)[0]
flatness = librosa.feature.spectral_flatness(y=y, hop_length=hop)[0]
y_h, y_p = librosa.effects.hpss(y)
perc_rms = librosa.feature.rms(y=y_p, hop_length=hop)[0]
harm_rms = librosa.feature.rms(y=y_h, hop_length=hop)[0]
perc_ratio = perc_rms / (harm_rms + perc_rms + 1e-9)  # 0=pure tonal, 1=pure percussive

# Window into 2s slices (aligns well with 4-beat phrases at typical BPMs)
WIN_S = 2.0
fpw = int(WIN_S * sr / hop)
n = len(rms) // fpw

def norm(x):
    x = np.asarray(x)
    lo, hi = np.percentile(x, 5), np.percentile(x, 95)
    return float(np.clip((x.mean() - lo) / (hi - lo + 1e-9), 0, 1))

timeline = []
for w in range(n):
    s, e = w*fpw, (w+1)*fpw
    energy    = norm(rms[s:e])          # 0-1: quiet → loud
    pace      = norm(onset_env[s:e])    # 0-1: sparse → dense onsets
    brightness= norm(centroid[s:e])     # 0-1: dark/low → bright/high
    tension   = norm(flatness[s:e])     # 0-1: tonal/resolved → noisy/tense
    percussive= float(perc_ratio[s:e].mean())  # 0-1: melodic → rhythmic

    # Derived mood label (heuristic — treat as a hint, not gospel)
    if energy > 0.6 and brightness > 0.5 and pace > 0.5:
        mood = "driving"        # chorus / hook
    elif energy > 0.5 and brightness < 0.4:
        mood = "heavy"          # menacing / epic-dark
    elif energy < 0.35 and brightness > 0.5:
        mood = "ethereal"       # reflective / shimmering
    elif energy < 0.35:
        mood = "still"          # intro / outro / breath
    elif pace > 0.6 and energy < 0.55:
        mood = "tense"          # pre-chorus build
    else:
        mood 

…(truncated)