Video Toolkit

Video Toolkit

Comprehensive video processing skill combining editing capabilities with multi-modal analysis (visual frames + audio transcription).

When to Use This Skill

Activate this skill when:

• User shares a video file (.mov, .mp4, .avi, .mkv, .webm, etc.)
• User asks to analyze, summarize, or understand video content
• User requests video editing operations (clip, merge, split)
• User asks questions about what's in a video
• You encounter "cannot read binary files" errors when trying to read video files
• Guides agents to use proper video analysis workflow

What this means for users:

When you share a video file, Claude will automatically recognize it and offer to analyze it properly using the video-toolkit, rather than attempting to read the binary file directly.

Prerequisites

Required Dependencies:

1. FFmpeg - Video processing and frame extraction

   • Installation handled by scripts/install_dependencies.sh
   • Verify: ffmpeg -version

2. Python 3.8+ with virtual environment

   • Installation handled by scripts/install_dependencies.sh

3. OpenAI Whisper - Speech transcription (local, no API key required)

   • Installed via pip in .venv
   • Model: base (good balance of speed/accuracy)
   • Other models available: tiny.en, small, medium, large

4. Google Gemini API - Audio analysis and music detection

   • Installed via pip in .venv
   • Requires API key (see setup below)
   • Model: gemini-2.5-flash

5. Shazam API - Music identification

   • Installed via pip in .venv (shazamio)
   • No API key required (uses public endpoint)

Setup:

Step 1: Install Dependencies

Run the installation script on first use:

bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/install_dependencies.sh

This creates a Python virtual environment at ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/ and installs the required packages (ffmpeg-python, openai-whisper, google-genai, shazamio).

Step 2: Configure API Keys

Set up Gemini API key for audio analysis:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py gemini YOUR_API_KEY

Get your Gemini API key from: https://aistudio.google.com/app/apikey

Optional: If you want to use RapidAPI's Shazam endpoint instead of the public one:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py shazam YOUR_RAPIDAPI_KEY

Note: Shazam/music identification works without an API key using shazamio's public endpoint.

How to Use This Skill

IMPORTANT: Check API Key Setup First

Before analyzing videos with audio, verify that the Gemini API key is configured:

• Config file location: /Users/emdash/Dev/claude-code-plugins/emdashcodes/.video-toolkit-config.json
• If the file doesn't exist or doesn't contain gemini.apiKey, run the setup first (see Prerequisites above)
• The scripts will error with clear setup instructions if the API key is missing

Multi-Modal Video Analysis Workflow

When analyzing a video file, follow this comprehensive workflow:

1. Frame Extraction

Extract visual frames using either interval mode or scene detection mode:

Interval Mode (time-based):

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py <video_path> <interval_seconds> <output_dir>

• interval_seconds: Time between frames (e.g., 2 for every 2 seconds)
• output_dir: Temporary directory for frames
• Best for: Consistent sampling, slideshows, time-based analysis

Scene Detection Mode (change-based):

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py <video_path> --scene-detect <output_dir> [threshold]

• threshold: Sensitivity (0.0-1.0, optional, default: 0.02)
• Automatically detects visual changes
• Best for: Screen recordings, presentations, variable content

Choosing the Right Mode:

Agent Decision Making:

• For UI/screen recordings: Use interval mode with 2-3s (captures static periods reliably)
• For presentations/slides: Use scene detection with 0.05 (good for slide transitions)
• For movies with hard cuts: Use scene detection with 0.20
• For talking heads/static: Use interval mode with 3-5s
• For action/movement: Use interval mode with 1-2s
• DEFAULT: Interval mode with 2s (safest for most videos)

IMPORTANT: Evaluate and Re-run if Needed

• After extraction, check the frame count - does it seem reasonable for the video duration?
• If you get too few frames (e.g., <5 frames for a 2-min screen recording), SWITCH TO INTERVAL MODE
• If you get too many frames (e.g., >100 frames), the threshold may be TOO LOW
• Think critically about the results, check some frames, and RE-RUN with different settings if necessary
• Don't proceed with analysis if frame extraction clearly failed or produced poor results

Scene Detection Limitation: Scene detection only captures frames when visual content changes significantly. If the video has long static periods (e.g., same screen for 30+ seconds), scene detection will miss that content entirely. For videos with static content, USE INTERVAL MODE instead with --mode interval --interval 2 or --interval 3.

2. Audio Extraction

Extract audio track using scripts/extract_audio.py:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_audio.py <video_path> <output_wav_path>

• Converts to WAV format (Whisper-compatible)
• Preserves original audio quality
• Returns: Path to extracted WAV file

3. Audio Analysis (Sequential Workflow)

The audio analysis follows a sequential workflow to maximize accuracy and efficiency:

3a. Speech Transcription (Whisper - Local)

Transcribe speech using Whisper (runs locally, no API key required):

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/transcribe_audio.py <wav_path> [model_name]

• model_name: Whisper model (default: base)
  • tiny.en: Fastest, English only
  • base: Good balance (recommended)
  • small: Better accuracy
  • medium: High accuracy
  • large: Best accuracy, slowest
• Returns: Timestamped transcript with speaker diarization info
• Purpose: Extract speech/dialogue from the audio

3b. Audio Understanding (Gemini Audio API)

Analyze audio comprehensively using Gemini Audio API:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_audio_gemini.py <wav_path> [output_markdown]

• Analyzes full audio content (not just speech)
• Detects music with precise timestamps (MM:SS to MM:SS format)
• Identifies non-speech sounds (applause, ambient noise, sound effects)
• Translates foreign languages: If Whisper detects non-English speech, Gemini provides both original and English translation
• Returns:
  • Markdown analysis saved to [output_markdown]
  • JSON data automatically saved to gemini_audio.json in same directory
  • JSON contains has_music flag and music_segments array for downstream processing
• Purpose: Understand what's in the audio beyond speech, detect if music is present, translate foreign languages

3c. Music Identification (Shazam - Conditional)

If Gemini detects music, identify songs using Shazam:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/identify_music.py <wav_path> <gemini_json> [output_markdown]

• <gemini_json>: Path to the gemini_audio.json file created by analyze_audio_gemini.py
• Only runs if Gemini detected music (checks has_music flag in JSON)
• Extracts music segments based on Gemini's timestamps using FFmpeg
• Identifies each segment with Shazam API
• Returns: Song title, artist, album, genre, year
• Purpose: Identify specific songs in the video

Sequential Workflow Summary:

1. Whisper (local)     → Speech transcription
2. Gemini Audio (API)  → Detect music + timestamps, analyze non-speech audio
3. FFmpeg              → Extract music segments (if music detected)
4. Shazam (API)        → Identify songs (if music detected)

This approach ensures:

• Whisper focuses on speech (what it does best, runs locally)
• Gemini provides overall audio understanding and music detection
• Shazam only processes clean music segments (better accuracy)
• No unnecessary API calls if no music is present

Gemini vs Shazam - Complementary Capabilities:

• Gemini Audio API: AI-powered audio understanding

  • Recognizes music characteristics: genre, mood, instrumentation, tempo
  • Example: "Rock/Alternative song with electric guitar and male vocals, energetic"
  • May recognize well-known classical pieces (especially if mentioned in speech)
  • Provides timestamps and musical context
  • Does NOT identify specific tracks, artists, or albums

• Shazam: Audio fingerprint matching

  • Identifies specific recordings: track title, artist, album, year
  • Example: "Substitution - Silversun Pickups, Swoon (2009)"
  • Requires at least 3 seconds of audio for reliable identification
  • Provides Shazam URLs and metadata

Both work together: Gemini describes what the music sounds like, Shazam identifies which specific track it is.

3.5. Context Clarification with Intelligent Question Generation

CRITICAL STEP: After gathering initial data (frames, audio, music identification), ask the user context questions to understand the video's perspective and avoid misidentification.

Why this step is critical: Visual evidence alone can be ambiguous. For example, seeing streaming platform with a BRB screen could mean either:

• The user is streaming and stepped away (creator perspective)
• The user is watching someone else's stream (viewer perspective)

Only the user can clarify their relationship to the content.

Workflow: Analyze First, Then Ask

Step 1: Complete Initial Data Gathering

Before asking any questions, run the full analysis pipeline through Step 3c:

• ✓ Extract frames (Step 1)
• ✓ Extract and transcribe audio (Steps 2, 3a)
• ✓ Analyze audio with Gemini (Step 3b)
• ✓ Identify music with Shazam if detected (Step 3c)

Step 2: Build Contextual Evidence

Review what was gathered:

• Visual evidence: Sample 3-5 frames - what's visible? (UI, people, environments, screen content)
• Audio evidence: What's the audio content? (speech, music, ambient sounds)
• Music evidence: What songs were identified? With what timestamps?
• Conversation context: What has the user said about this video?
• User knowledge: What do you know about this user from conversation history and preferences?

Step 3: Ask Context Question (Q1) with Evidence

Present Q1 with context from your gathered evidence to help the user answer:

# Build context string from evidence
# Adapt summary based on what was found in the video

# Example 1: Stream Viewer
evidence_summary = """I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57
"""

# Example 2: Tutorial/How-to
evidence_summary = """I've analyzed the video and found:
- Visual: Person's hands visible working with materials, step-by-step demonstrations
- Audio: Instructional narration explaining process, background music
- Duration: 8:34
"""

# Example 3: Screen Recording (Software Demo)
evidence_summary = """I've analyzed the video and found:
- Visual: VS Code editor with code visible, cursor movements, terminal commands
- Audio: Voiceover explaining code changes, keyboard typing sounds
- Duration: 5:12
"""

# Example 4: Screen Recording (Gameplay)
evidence_summary = """I've analyzed the video and found:
- Visual: Game interface with HUD elements, character movement, gameplay
- Audio: Game sounds, background music, occasional commentary
- Duration: 12:45
"""

# Example 5: Personal/Casual
evidence_summary = """I've analyzed the video and found:
- Visual: Person on camera in home setting, handheld/phone camera movement
- Audio: Person speaking directly to camera, ambient room sounds
- Duration: 2:18
"""

# Example 6: Personal/Casual (Phone Clip)
evidence_summary = """I've analyzed the video and found:
- Visual: Quick clip of outdoor scene, vertical/portrait orientation, casual framing
- Audio: Ambient sounds, brief speech, wind noise
- Duration: 0:43
"""

# Example 7: Event/Performance
evidence_summary = """I've analyzed the video and found:
- Visual: Stage with performers, audience visible, concert venue setting
- Audio: Live music performance, crowd noise, applause
- Duration: 3:56
"""

# Example 8: Presentation/Slides
evidence_summary = """I've analyzed the video and found:
- Visual: Slide deck with bullet points and diagrams, presenter occasionally visible
- Audio: Presenter speaking, slide transition sounds
- Duration: 18:24
"""

# Example 9: Casual Documentation (Workspace)
evidence_summary = """I've analyzed the video and found:
- Visual: Desk setup with monitors and equipment, informal camera angles
- Audio: Background music, ambient office sounds, no narration
- Duration: 1:34
"""

# Example 10: Behind-the-Scenes
evidence_summary = """I've analyzed the video and found:
- Visual: Production equipment, camera setup, people working on set
- Audio: Technical discussions, equipment sounds, music playing
- Duration: 4:27
"""

# Template for implementation:
# Select the most relevant description based on gathered evidence
def build_evidence_summary(frames_analysis, audio_analysis, music_data, duration):
    """
    Generate evidence summary based on analyzed content

    Returns contextual summary that helps user identify video type
    """
    visual_desc = describe_visual_content(frames_analysis)
    audio_desc = describe_audio_content(audio_analysis, music_data)

    return f"""I've analyzed the video and found:
- Visual: {visual_desc}
- Audio: {audio_desc}
- Duration: {format_duration(duration)}
"""

# Present Q1 with evidence context
AskUserQuestion(
    questions=[
        {
            "question": f"{evidence_summary}\n\nWhat type of video is this?",
            "header": "Video Type",
            "multiSelect": false,
            "options": [
                {
                    "label": "Tutorial/How-to",
                    "description": "Teaching or demonstrating something step-by-step"
                },
                {
                    "label": "Personal",
                    "description": "Personal video, daily life content, a moment or activity, clip from phone"
                },
                {
                    "label": "Screen recording",
                    "description": "Recording your own screen (software demo, debugging video, gameplay, etc.)"
                },
                {
                    "label": "Watching something",
                    "description": "Recording yourself viewing someone else's content"
                },
                {
                    "label": "Event/Performance",
                    "description": "Concert, presentation, ceremony, or live event"
                },
                {
                    "label": "Other/Not sure",
                    "description": "Type a custom description of what this video shows"
                }
            ]
        }
    ]
)

Key Patterns in Evidence Summaries:

1. Be specific about what's visible: Not just "screen content" but "VS Code editor with code" or "Streaming platform interface with BRB screen"

2. Mention identifying audio: "Music playing (song identified)" vs. "instructional narration" vs. "live performance audio"

3. Note video characteristics: Portrait orientation, handheld movement, production quality

4. Keep it concise: 2-3 bullet points max, user can read it quickly

5. Help user self-identify: Evidence should make the correct option obvious

Step 4: Generate Smart Follow-up (Q2) Based on Evidence + Q1 Answer

CRITICAL: Generate Q2 dynamically based on:

• ✓ User's Q1 answer
• ✓ Visual evidence from frames
• ✓ Audio evidence from transcription/Gemini
• ✓ Music identification results
• ✓ Conversation context
• ✓ User preferences and history

Skip Q2 entirely if:

• The evidence makes the context obvious
• Q1 answer is self-explanatory (Tutorial, Event/Performance)
• You're highly confident about the analysis direction

Generate custom Q2 if needed:

If Q1 = "Watching something" AND you identified what they were watching:

# Example: You saw streaming platform + identified music
AskUserQuestion(
    questions=[
        {
            "question": "I can see streaming platform with a BRB screen and identified 'Song Title' by Artist Name playing in the background. Is this what was happening?",
            "header": "Confirm Context",
            "multiSelect": false,
            "options": [
                {
                    "label": "Yes, that's correct",
                    "description": "Watching a stream with music playing in my space"
                },
                {
                    "label": "Partially correct",
                    "description": "Some of that is right, but let me clarify"
                },
                {
                    "label": "No, different context",
                    "description": "That's not quite what was happening"
                }
            ]
        }
    ]
)

If Q1 = "Watching something" AND you're uncertain what:

AskUserQuestion(
    questions=[
        {
            "question": "What were you watching?",
            "header": "Content Source",
            "multiSelect": true,  # Allow multiple selections
            "options": [
                {
                    "label": "Live stream",
                    "description": "Streaming platforms, Twitch, YouTube live, etc."
                },
                {
                    "label": "Video content",
                    "description": "YouTube video, movie, TV show, etc."
                },
                {
                    "label": "Music playing",
                    "description": "Music was playing in your space"
                },
                {
                    "label": "Other media",
                    "description": "Game, presentation, or other content"
                }
            ]
        }
    ]
)

If Q1 = "Personal" AND video shows workspace/music:

# You already know music was playing and workspace is visible
# Generate confirmation question instead of generic tags
AskUserQuestion(
    questions=[
        {
            "question": "I can see your workspace and 'Song Title' playing. What tags describe this video?",
            "header": "Context Tags",
            "multiSelect": true,
            "options": [
                {
                    "label": "Music playing",
                    "description": "Background music in your space"
                },
                {
                    "label": "Workspace/setup",
                    "description": "Showing desk, equipment, or environment"
                },
                {
                    "label": "Activity in progress",
                    "description": "Doing something while recording"
                },
                {
                    "label": "Artistic/experimental",
                    "description": "Creative or artistic intent"
                }
            ]
        }
    ]
)

If Q1 = "Screen recording":

# Only ask if genuinely uncertain from frames
# You likely already saw what's on screen
AskUserQuestion(
    questions=[
        {
            "question": "I can see {specific_app_or_content}. What were you recording?",
            "header": "Screen Content",
            "multiSelect": false,
            "options": [
                {
                    "label": "Software/application",
                    "description": "Demonstrating a program or workflow"
                },
                {
                    "label": "Gameplay",
                    "description": "Playing a game"
                },
                {
                    "label": "Design/creative work",
                    "description": "Working in design tools, editors, etc."
                },
                {
                    "label": "General computer use",
                    "description": "Browsing, chatting, or mixed activities"
                }
            ]
        }
    ]
)

If Q1 = "Tutorial/How-to" or "Event/Performance": → Skip Q2 - these are self-explanatory, proceed to analysis

If Q1 = "Other/Not sure" (custom text provided): → Use the custom text to inform analysis, skip Q2 unless truly necessary

Store and Use Metadata

After Q1 (and Q2 if asked), store structured metadata:

video_metadata = {
    "video_type": user_q1_answer,           # "Watching something"
    "perspective": derived_perspective,      # "First-Person Viewer"
    "content_tags": user_q2_answers,        # ["live stream", "music playing"]
    "evidence": {
        "visual": visual_summary,            # From frames
        "audio": audio_summary,              # From Gemini
        "music": music_identification,       # From Shazam
        "duration": video_duration
    }
}

Use this metadata throughout remaining analysis:

• Guide visual analysis focus (Step 5)
• Inform subagent prompts
• Set appropriate terminology in final summary
• Avoid misidentifying perspective or intent

Example: Context-Aware Question Flow

Scenario: User shares video showing streaming platform with BRB screen, person visible, music playing

Data Gathered (Steps 1-3d):

• Frames: Streaming platform UI, BRB overlay, person, workspace visible
• Audio: Music "Song Title" by Artist Name (0:32-1:57), minimal speech
• Duration: 1:57
• Context: User information from conversation history

Q1 with Context:

I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57

What type of video is this?
→ User selects: "Watching something"

Smart Q2 Generation:

# You already know:
# - Streaming platform with BRB = watching a stream
# - Music identified = "Song Title" playing in their space
# - Person visible = user on camera

# Instead of generic "What were you watching?", generate specific confirmation:
"I can see streaming platform with a BRB screen and identified 'Song Title' by Artist Name
playing in the background. Were you watching someone's stream while music played in your space?"

Options:
- Yes, that's correct
- Partially - let me clarify
- No, different context

Result: Only 2 questions needed, and Q2 is informed and specific rather than generic.

Alternative - High Confidence, Skip Q2: If extremely confident from evidence, skip Q2 entirely:

Q1 answer: "Watching something"

→ Skip Q2, proceed with metadata:
{
  video_type: "Watching something",
  perspective: "First-Person Viewer",
  content_tags: ["live stream", "music playing", "workspace visible"]
}

Benefits of This Approach

1. Evidence-informed questions: Q1 includes what you found, helping user answer
2. Smart Q2 generation: Tailored to what you already know + user's Q1 answer
3. Skip when confident: Don't ask unnecessary questions if evidence is clear
4. Confirmation over discovery: Ask "Is this correct?" when you have strong evidence
5. Context-aware: Uses conversation history, user preferences, and gathered data
6. Prevents misidentification: Clarifies creator vs. viewer perspective early
7. Efficient: Maximum 2 questions, often just 1

Integration with Remaining Workflow

After completing Step 3.5 (Context Clarification), proceed to:

Step 4: Video Type Detection and Analysis Strategy

• Use video_metadata.perspective to select appropriate analysis focus
• Apply type-specific guidance based on clarified context

Step 5: Visual Analysis with video-frame-analyzer Subagent(s)

• Pass video_metadata to subagent prompts
• Ensure analysis matches user's stated relationship to content
• Use correct terminology (viewer/creator/observer language)

The metadata from context clarification informs all downstream analysis.

4. Video Perspective and Type Detection (UPDATED)

After context clarification (Step 3.5), use the user's answers and gathered evidence to determine the appropriate analysis approach.

Two-Dimensional Classification System

Dimension 1: Perspective (Who is filming and why?) Dimension 2: Content Type (What's in the video?)

Perspective Classification

Derive perspective from user's Q1 answer and evidence:

Expanded Content Type Taxonomy

Based on perspective, select the appropriate content type analysis:

First-Person Viewer Types

The user is watching/consuming content created by others

Stream Viewer

Indicators:

• Visual: Streaming platform UI (Discord, Twitch, YouTube), others' stream content, BRB screens, chat visible
• Audio: Music in viewer's space, ambient sounds, minimal commentary
• Evidence: Streaming platform visible, user not the presenter/streamer

Analysis Focus:

• Primary: What stream/content is being watched? (streamer name, stream title, platform)
• Track: BRB screens, waiting periods, stream states, chat activity
• Note: Viewer's environment (workspace, room, objects visible)
• Look For:
  • Parallel activities (what is viewer doing while watching?)
  • Music or ambient audio in viewer's space (distinct from stream audio)
  • Environmental context and mood
  • Correlation between stream content and viewer behavior

Reaction Video

Indicators:

• Visual: Split screen (content + reactor) or switches between content and reactions
• Audio: Commentary over content, reactions, pauses for analysis
• Evidence: User visible reacting to visible content

Analysis Focus:

• Primary: What content is being reacted to? User's reactions and commentary
• Track: Reaction timing, pauses, emotional responses
• Note: User's analysis, critiques, or engagement with content
• Look For: Key moments that trigger reactions, commentary themes

Viewing Documentation

Indicators:

• Visual: Browser/player UI, content consumption in progress, informal framing
• Audio: Background music, ambient sounds, casual environment
• Evidence: Passive viewing being documented

Analysis Focus:

• Primary: What's being consumed, why is it being documented
• Track: Viewing progression, pauses, environmental changes
• Note: Mood, atmosphere, context for documentation
• Look For: Artistic intent, thematic connections

First-Person Creator Types

The user is creating/presenting content

Tutorial/How-to

Indicators:

• Visual: Step-by-step demonstrations, hands visible, process shown
• Audio: Instructional narration, explanations, process sounds
• Evidence: Deliberate teaching structure

Analysis Focus:

• Primary: What's being taught, step-by-step process
• Track: Instructions, materials/tools, technique demonstrations
• Note: Key concepts, warnings, tips
• Look For: Learning objectives, common mistakes addressed

Screen Recording (Creator)

Indicators:

• Visual: User's own screen with deliberate navigation, cursor movements, UI interactions
• Audio: Voiceover tutorial, keyboard clicks, narration
• Evidence: User demonstrating their own workflow

Analysis Focus:

• Primary: UI elements, workflow steps, applications used
• Track: Mouse/cursor position, clicked elements, opened applications
• Note: Workflow steps, error messages, command sequences
• Look For: Transitions between applications, file operations, settings changes

Personal (Vlog/Life Content)

Indicators:

• Visual: Person on camera, home/outdoor settings, handheld or mounted camera
• Audio: Direct-to-camera speech, personal narrative, ambient sounds
• Evidence: User sharing personal moments or experiences

Analysis Focus:

• Primary: Setting/location, what person is saying/doing
• Track: Emotional tone, activities, locations visited
• Note: Personal details, narrative arc, objects/items shown
• Look For: Story progression, mood shifts, environment changes

Presentation/Slides

Indicators:

• Visual: Slide layouts, bullet points, diagrams, presenter view
• Audio: Presenter speaking, slide advance sounds
• Evidence: Formal presentation structure

Analysis Focus:

• Primary: Slide content (headings, bullet points, data)
• Track: Slide numbers, transitions, presenter appearance
• Note: Key concepts, visualizations, quotes, formulas
• Look For: Slide themes, speaker gestures, main arguments

Meta/Hybrid Types

The user is documenting the process of creating/consuming

Casual Documentation

Indicators:

• Visual: Informal framing, switches between activities, workspace visible, no clear structure
• Audio: Ambient audio, music, casual speech, environmental sounds
• Evidence: Informal recording of moments/activities

Analysis Focus:

• Primary: Activities happening, personal environment, objects/items
• Track: Environmental changes, mood shifts, items introduced
• Note: Personal details that provide context, atmosphere
• Look For:
  • Documentary-style observations
  • Authentic, unscripted moments
  • Connections between visual, audio, and thematic elements
  • Artistic or experimental intent

Behind-the-Scenes

Indicators:

• Visual: Production equipment visible, setup shots, workspace, process in action
• Audio: Technical discussions, process sounds, tools being used
• Evidence: Documenting creation workflow

Analysis Focus:

• Primary: Creation process, tools/equipment, methodology
• Track: Setup changes, workflow steps, technical decisions
• Note: Challenges encountered, solutions implemented
• Look For: Insights into creative/production process

Process Video

Indicators:

• Visual: Work-in-progress visible, tools/software, iterative changes
• Audio: Narration of process, thinking out loud, work sounds
• Evidence: Documenting work as it happens

Analysis Focus:

• Primary: Steps taken, decision-making, progress
• Track: Iterations, changes, refinements
• Note: Reasoning, challenges, solutions
• Look For: Learning moments, key decisions

Third-Person Observer Types

The user is filming someone/something else

Event/Performance

Indicators:

• Visual: Stage/venue, audience, performers, event setting
• Audio: Live audio, applause, ambient event noise
• Evidence: User filming an event they're attending

Analysis Focus:

• Primary: Performers, performance content, venue
• Track: Key performance moments, crowd reactions, technical elements
• Note: Event highlights, special effects, atmosphere
• Look For: Memorable moments, mistakes, energy shifts

Interview/Podcast

Indicators:

• Visual: Two+ people, static camera, simple background
• Audio: Conversational dialogue, minimal music
• Evidence: Discussion/interview being filmed

Analysis Focus:

5. Visual Analysis with video-frame-analyzer Subagent(s)

After frames are extracted AND video type is assessed, delegate to the specialized video-frame-analyzer subagent(s) for systematic visual analysis:

Single Subagent (< 30 frames): For videos with fewer than 30 frames, use a single subagent:

IMPORTANT: Use the full qualified subagent name: video-toolkit:video-frame-analyzer

Step 1: Assess Video Type (sample first 3-5 frames)

Before full analysis, quickly review a few early frames to determine video type and appropriate analysis focus:

Use the video-toolkit:video-frame-analyzer subagent to assess video type from /tmp/video-toolkit-[timestamp]/frames

Sample frames 1-5 and determine:
1. Video type (screen recording, vlog, presentation, music video, tutorial, etc.)
2. Recommended analysis focus based on type (see Type-Specific Analysis Focus section)
3. Key elements to track throughout the video

Save assessment to: /tmp/video-toolkit-[timestamp]/video-type-assessment.md

Step 2: Full Frame Analysis (with type-adaptive prompts)

Use the video-toolkit:video-frame-analyzer subagent to analyze the frames in /tmp/video-toolkit-[timestamp]/frames

**Video Type Detected:** [Insert detected type from assessment]

The following files are available:
- Frames: /tmp/video-toolkit-[timestamp]/frames/*.png (X frames total)
- Metadata: /tmp/video-toolkit-[timestamp]/frames/frames_metadata.json (contains frame timestamps)
- Transcript: /tmp/video-toolkit-[timestamp]/transcript.md (if video has audio)
- Audio Analysis: /tmp/video-toolkit-[timestamp]/audio_analysis.md (Gemini audio understanding, if available)
- Music Identification: /tmp/video-toolkit-[timestamp]/music_identification.md (Shazam results, if music detected)
- Video Type Assessment: /tmp/video-toolkit-[timestamp]/video-type-assessment.md

Based on the detected video type, apply appropriate analysis focus:
[Insert type-specific focus points from Type-Specific Analysis Focus section]

Analyze all frames, correlate visual content with transcript/audio using timestamps, and pay special attention to elements relevant for this video type.

Save your complete analysis to: /tmp/video-toolkit-[timestamp]/video-analysis.md

Parallel Subagents (≥ 30 frames): For videos with 30+ frames, spawn multiple video-toolkit:video-frame-analyzer subagents in parallel, each handling a subset of frames:

IMPORTANT:

• Use the full qualified subagent name: video-toolkit:video-frame-analyzer
• Use a single message with multiple Task tool calls to run agents in parallel

Example for 50 frames:

# Spawn 5 subagents in parallel, each analyzing 10 frames
# Agent 1: frames 1-10
# Agent 2: frames 11-20
# Agent 3: frames 21-30
# Agent 4: frames 31-40
# Agent 5: frames 41-50

How to split frames:

• 30-50 frames: 3-5 subagents (10 frames each) - launch all in parallel
• 50-100 frames: 5-10 subagents (10 frames each) - launch in waves of 5
• 100+ frames: Consider using interval mode instead, or max 10 subagents in waves

Staged Execution for Long Videos: For videos requiring more than 5 subagents, launch them in waves to avoid overwhelming the system:

Example: 100 frames = 10 subagents

• Wave 1: Launch subagents 1-5 in parallel (frames 1-50)
• Wait for Wave 1 to complete
• Wave 2: Launch subagents 6-10 in parallel (frames 51-100)
• Wait for Wave 2 to complete
• Synthesize all results

Why staged execution?

• Prevents too many concurrent subagents
• Better resource management
• Easier to debug if something fails
• More predictable performance

Prompt for parallel subagents:

IMPORTANT:

1. First, assess video type with a quick subagent analyzing frames 1-5
2. Then create the partial-analyses/ directory: mkdir -p /tmp/video-toolkit-[timestamp]/partial-analyses
3. Launch parallel subagents with type-aware prompts

Step 1: Quick Video Type Assessment

Use the video-toolkit:video-frame-analyzer subagent to assess video type from frames 1-5 in /tmp/video-toolkit-[timestamp]/frames

Sample the first 5 frames and determine:
1. Video type (screen recording, vlog, presentation, music video, tutorial, interview, documentary, gameplay, event, animation)
2. Recommended analysis focus based on type
3. Key elements to track throughout

Review audio analysis if available to help determine type.

Save assessment to: /tmp/video-toolkit-[timestamp]/video-type-assessment.md

Step 2: Type-Adaptive Parallel Analysis

After video type is assessed, use this prompt template for each parallel subagent:

Use the video-toolkit:video-frame-analyzer subagent to analyze frames [start]-[end] from /tmp/video-toolkit-[timestamp]/frames

**Video Type:** [Insert type from assessment]
**Analysis Focus:** [Insert relevant focus points for this type]

Files available:
- Frames: frame_00[start].png through frame_00[end].png
- Metadata: /tmp/video-toolkit-[timestamp]/frames/frames_metadata.json
- Transcript: /tmp/video-toolkit-[timestamp]/transcript.md (if available)
- Audio Analysis: /tmp/video-toolkit-[timestamp]/audio_analysis.md (if available)
- Video Type Assessment: /tmp/video-toolkit-[timestamp]/video-type-assessment.md

Focus on frames [start] through [end] only. Based on video type, provide:
1. Timestamp range covered
2. Visual content description (emphasize type-specific elements)
3. Key elements for this video type (UI/text for screen recordings, actions/setting for vlogs, etc.)
4. Scene changes within your range
5. Correlation with transcript/audio (if applicable)

Save your analysis to: /tmp/video-toolkit-[timestamp]/partial-analyses/part[N].md

Example for Vlog/Personal Video:

Use the video-toolkit:video-frame-analyzer subagent to analyze frames 1-12 from /tmp/video-toolkit-[timestamp]/frames

**Video Type:** Vlog/Personal
**Analysis Focus:**
- Primary: Setting/location, people present, activities happening
- Track: Emotional tone, interactions, objects/items shown
- Note: What person is doing, wearing, holding
- Look For: Personal details (smoking, eating, gestures), environment changes

Files available:
- Frames: frame_0001.png through frame_0012.png
- Metadata: /tmp/video-toolkit-[timestamp]/frames/frames_metadata.json
- Transcript: /tmp/video-toolkit-[timestamp]/transcript.md
- Audio Analysis: /tmp/video-toolkit-[timestamp]/audio_analysis.md
- Video Type Assessment: /tmp/video-toolkit-[timestamp]/video-type-assessment.md

Focus on frames 1-12. Provide:
1. Timestamp range: 0:00 - 0:22
2. Visual content: Describe setting, people, activities with attention to personal details
3. Personal elements: Note smoking, gestures, objects held, clothing, emotional state
4. Scene/environment changes within this range
5. Correlation with what's being said (transcript) and music/ambient audio

Save your analysis to: /tmp/video-toolkit-[timestamp]/partial-analyses/p

…(truncated)