Multimodal Extraction

Multimodal Extraction

Overview

This skill composes the existing video workflows into one artifact:

• download-video for URL inputs
• thumbnail-extraction for slide frames and key screenshots
• transcribe-anything for transcript strategy

The implementation is intentionally speed-first:

1. Download only when the input is a URL
2. Reuse the fast slide/key-frame heuristics from thumbnail-extraction
3. Use local whisper JSON output for timestamped transcript segments
4. Merge everything into one Markdown timeline with relative image links

When To Use

• "Turn this talk into multimodal notes"
• "Make me a markdown transcript with screenshots"
• "Extract slides and transcript together"
• "Build a recap doc from this video"
• "Given this YouTube URL, produce a slide-synced transcript"

Requirements

brew install ffmpeg yt-dlp
pip3 install --break-system-packages openai-whisper

The following existing local script is reused:

• ../thumbnail-extraction/thumbnail_extractor.py

Command

python3 multimodal_extract.py <video_or_url> [output_dir] [--language en] [--whisper-model turbo] [--top-n 4]

What It Does

Step 1: Resolve the Source

• If the input is a local file, use it directly
• If the input starts with http:// or https://, download it first with yt-dlp
• For YouTube URLs, direct yt-dlp is usually enough
• For trickier hosted pages, this skill follows the same practical intent as download-video: get a usable local file first

Step 2: Extract Visual Anchors

Run:

python3 ../thumbnail-extraction/thumbnail_extractor.py "$VIDEO" "$OUTPUT/visuals" 4 --extract-slides

This produces:

• top thumbnail candidates in the root of visuals/
• slide images in visuals/slides/
• manifests with timestamps

Step 3: Transcribe

Extract normalized mono 16k audio:

ffmpeg -y -i "$VIDEO" -vn -ac 1 -ar 16000 -acodec pcm_s16le \
  -af "highpass=f=80,lowpass=f=8000,loudnorm=I=-16:TP=-1.5:LRA=11" \
  "$OUTPUT/audio/source_preprocessed.wav"

Then transcribe with Whisper:

whisper "$OUTPUT/audio/source_preprocessed.wav" \
  --model turbo \
  --language en \
  --word_timestamps True \
  --condition_on_previous_text False \
  --output_format json \
  --output_dir "$OUTPUT/transcript"

Step 4: Merge into Markdown

The script:

• reads slide and thumbnail manifests
• reads Whisper transcript segments
• sorts all visual anchors by timestamp
• groups transcript text between successive visual anchors
• writes multimodal_timeline.md with:
  • section timestamp
  • associated image(s)
  • transcript span for that interval

Output

output_dir/
  source/
  visuals/
  audio/
  transcript/
  multimodal_timeline.md

Design Principle

The goal is total end-to-end extraction speed.

That means:

• heuristics first
• local transcript by default
• no VLM in the common path
• only enough structure to make the Markdown artifact useful immediately

Future Extensions

• add backend switching for transcribe-anything
• add deck-aware slide labeling when a source deck exists
• add speaker diarization sections
• add chaptering or summary generation on top of the Markdown timeline