Media MCP

Media MCP

Principle

1. Fetch structured data, not raw HTML. Every tool calls a purpose-built API (TwitterAPI.io, optional Xquik for overlapping read tools, Cobalt, YouTube captions) and returns structured, LLM-ready text. No scraping, no DOM parsing, no fragile selectors.
2. Transcribe locally, always. Audio never leaves the machine. Whisper runs on local hardware against a local model file. The only network calls are to download the media itself.
3. Captions first, Whisper second. For YouTube, try platform captions (instant, free, accurate). Fall back to download + Whisper only when captions don't exist. Don't burn API time or compute when the platform already did the work.
4. Download then process. Media is downloaded to a temp file, processed (transcribed, frame-extracted), and cleaned up. No streaming pipelines, no partial results. The user gets complete output or a clear error.
5. One tool, one job. Each tool does exactly one thing. get_tweet fetches a tweet. get_youtube_transcript gets a transcript. extract_video_frames pulls frames. No multi-purpose tools, no mode flags that change behavior.
6. Return file paths for visual content. When downloading images or extracting frames, return absolute local paths so the LLM can read them directly with its vision capabilities. Don't describe images — let the model see them.
7. Confidence-driven accuracy. Transcription always carries per-token probabilities and surfaces Uncertainty zones — the exact time spans where Whisper was guessing. When accuracy matters (proper nouns, URLs, install commands, on-screen text), Claude calls get_video_frames_at on those specific timestamps and reads the resulting frames with its own vision. No OCR — Claude's vision is the single model that does all multimodal reasoning. Minimum frames, maximum accuracy.

Architecture

                    ┌──────────────────────────────────────────┐
                    │              MCP Client                   │
                    │   (Claude Desktop / Claude Code / etc.)   │
                    └──────────────┬───────────────────────────┘
                                   │ stdio
                    ┌──────────────▼───────────────────────────┐
                    │           media-mcp server                │
                    │         (Node.js, MCP SDK)                │
                    └──┬────────┬────────┬────────┬────────────┘
                       │        │        │        │
              ┌────────▼──┐ ┌──▼────┐ ┌─▼─────┐ ┌▼──────────┐
              │ Twitter/X  │ │YouTube│ │Insta- │ │  Video     │
              │ 26 tools   │ │1 tool │ │gram   │ │  Frames    │
              │            │ │       │ │1 tool │ │  1 tool    │
              └────┬───────┘ └──┬────┘ └──┬────┘ └─────┬──────┘
                   │            │         │            │
          ┌────────▼──┐   ┌────▼───┐  ┌──▼─────┐  ┌───▼─────┐
          │Twitter API │   │yt-dlp  │  │ Cobalt │  │ yt-dlp  │
          │   .io      │   │caption │  │  API   │  │ direct  │
          │  (REST)    │   │  API   │  │(self-  │  │  fetch  │
          └────────────┘   └───┬────┘  │hosted) │  └────┬────┘
                               │       └───┬────┘       │
                               ▼           ▼            ▼
                         ┌─────────────────────────────────┐
                         │     Local Processing Pipeline    │
                         │                                  │
                         │  ffmpeg ──► audio extraction     │
                         │  whisper-cli ──► transcription   │
                         │  ffmpeg ──► frame extraction     │
                         │                                  │
                         │  All temp files cleaned up       │
                         └─────────────────────────────────┘

Pipelines

Each tool follows a specific pipeline. Understanding what happens at each step helps you predict latency and failure modes.

Tweet Fetch Pipeline

URL ──► TwitterAPI.io or Xquik REST ──► parse tweet JSON ──► extract media URLs
                                                         │
                                          ┌──────────────┤
                                          ▼              ▼
                                       [photo]        [video]
                                     return URL    download ──► ffmpeg ──► whisper-cli
                                                                              │
                                                                     return text + metrics

What it does: Fetches tweet text, author info, metrics (likes/RT/views), media URLs, threads, quoted tweets, and articles via REST API. If the tweet contains video and transcribe: true, downloads the video to a temp file, extracts audio with ffmpeg (16kHz mono WAV), transcribes with whisper-cli (segment-level timestamps included), then cleans up.

Latency: ~1s for text-only tweets. ~10-30s when video transcription is involved (download + ffmpeg + Whisper).

YouTube Transcript Pipeline

URL ──► extract video ID
             │
             ├──► try captions API (instant, ~200ms)
             │         │
             │    success? ──► return timestamped segments
             │         │
             │    fail? ──► yt-dlp download ──► ffmpeg extract audio ──► whisper-cli
             │                                                              │
             │                                                     return full transcript
             ▼
        return text + segments + source indicator

What it does: First tries YouTube's built-in caption system (free, instant, accurate — already timestamped). If no captions exist, falls back to downloading audio with yt-dlp, extracting with ffmpeg, and transcribing with whisper-cli (segment-level timestamps included).

Latency: ~200ms with captions. ~20-60s with Whisper fallback (depends on video length).

Instagram Download Pipeline

URL ──► Cobalt API ──► get download URLs + metadata
                            │
                ┌───────────┼───────────┐
                ▼           ▼           ▼
           [single]    [carousel]   [video/reel]
          download     download     download ──► ffmpeg ──► whisper-cli
          to folder    all items                                │
                       to folder              return paths + transcription
                │           │
           return local file paths (absolute)

What it does: Sends the Instagram URL to your self-hosted Cobalt instance. Cobalt returns download URLs for all media items. Each item is downloaded to a local folder with a unique ID. Videos are optionally transcribed with Whisper (segment-level timestamps included). Returns absolute file paths so the LLM can view images directly.

Requires: Self-hosted Cobalt instance (COBALT_API_URL).

Video Frame Extraction Pipeline

URL ──► detect source type
             │
             ├──► [direct URL] ──► HTTP download to temp
             ├──► [YouTube/TikTok/etc.] ──► yt-dlp download to temp
             │
             ▼
        ffmpeg -i <video> -vf fps=<rate> -ss <start> -to <end> frame_%04d.jpg
             │
             ▼
        return folder path + frame paths + timestamps

What it does: Downloads video from any supported URL, then uses ffmpeg to extract frames at a configurable rate (default: 1 fps). Supports time range extraction with start/end parameters. Returns absolute paths to frame images so the LLM can analyze them visually.

Latency: ~5-30s depending on video length and frame rate.

Tool Reference

Twitter/X — 26 tools

TwitterAPI.io is the default backend for all Twitter tools and requires TWITTER_API_KEY. Set TWITTER_BACKEND=xquik with XQUIK_API_KEY to send the overlapping read tools to Xquik instead. Both backends return the same tool output.

These 15 tools run on either backend: get_tweet, get_user_profile, get_user_about, get_user_tweets, get_user_followers, get_user_following, get_verified_followers, get_user_mentions, get_tweet_replies, get_tweet_quotes, get_tweet_retweeters, search_tweets, search_users, check_follow_relationship, get_trends.

These stay on TwitterAPI.io and still need TWITTER_API_KEY: get_tweet_replies_v2, get_list_timeline, get_community_tweets, get_space_detail, get_bookmarks, the 3 monitor tools, and the 3 filter-rule tools.

Fetching tweets

Fetching profiles

Real-time monitoring

YouTube — 1 tool

Instagram — 1 tool

Requires self-hosted Cobalt instance (COBALT_API_URL).

Video — 2 tools

Dependencies

How Transcription Works

1. Video is downloaded to a temp file (via direct HTTP, yt-dlp, or Cobalt)
2. ffmpeg extracts audio as 16kHz mono WAV: ffmpeg -i video.mp4 -ar 16000 -ac 1 -f wav audio.wav
3. whisper-cli transcribes locally with segment-level timestamps AND per-token confidence: whisper-cli -m <model> -f audio.wav -l <lang> -ojf. The language comes from the tool's language param (ISO 639-1 or auto, default English) and the model from its model param (tiny → large-v3-turbo, auto-downloaded to ~/.media-mcp/models, or a custom .bin path). Per-token probabilities are parsed; tokens with p < 0.5 are flagged. Output is rendered as: [HH:MM:SS.mmm --> HH:MM:SS.mmm] text with ⟨uncertain-token p=0.XX⟩ markers followed by an Uncertainty zones summary and Demonstrative phrases block.
4. Temp files (video + audio) are cleaned up automatically

All transcription happens on your machine. No audio is sent to external services.

Accuracy-critical workflow — confidence-driven frame lookup

Transcription is fast but not infallible. Small Whisper models (base, ~140MB) will mishear unusual proper nouns, rare URLs, and written-only content. Rather than ship a separate OCR pipeline, media-mcp exposes Whisper's own uncertainty and lets Claude decide when to look at frames.

The two-tool pattern:

1. Any transcription tool (get_tweet, get_instagram_post, get_youtube_transcript) returns:

   • Segment-level timestamped transcript
   • Inline ⟨token p=0.XX⟩ markers where Whisper was below confidence threshold (default 0.5)
   • An Uncertainty zones summary with midpoint_s timestamps
   • A Demonstrative phrases block (phrases like "visit our", "this command", "in the bio" — strong signals that on-screen content is being referenced)

2. get_video_frames_at(url, timestamps[]) — precision frame extraction. Pass the midpoint_s values from the uncertainty zones or demonstrative phrases. Returns one JPG per timestamp. Claude reads the JPGs directly with its vision.

When to trigger frame lookup (Claude's decision):

• The user's question depends on an uncertain span (typically: proper nouns, install commands, URLs, handles, prices, code snippets, brand spellings).
• A demonstrative phrase appears near content the user is asking about.
• The user asks "what exactly did they show" / "what's the link" / "what's the command".

When NOT to trigger frame lookup:

• The transcript is entirely high-confidence for content the user cares about.
• The user asked for a summary / opinion / vibe — audio is sufficient.
• Uncertainty zones fall in filler phrases ("um", "you know") that don't matter to the answer.

Video cache: every transcription tool caches the downloaded video under a hash of the URL at ~/.media-mcp/cache/videos/ (24h TTL). When get_video_frames_at is called on the same URL, it reuses the cached video — no re-download.

Example — the canonical test case. User shares an Instagram reel about three Claude Code skills. Whisper (base) hears Emil Koval skill (p=0.33 on "val") — that's wrong; the screen shows npx skills add emilkowalski/skill. Transcript flags the uncertainty zone at ~5.58s. Claude calls get_video_frames_at(url, [5.58, 22.27, 33.59]), reads three JPGs with its vision, and reports the correct install slugs: emilkowalski/skill, pbakaus/impeccable, tasteskill.dev.

Anti-patterns

• Calling get_tweet in a loop for bulk data. Use get_user_tweets or search_tweets instead — they return 20 results per call.
• Using extract_video_frames at high FPS for long videos. 1 fps on a 10-minute video = 600 frames. Use time ranges (start/end) to focus on the relevant section.
• Assuming Instagram tools work without Cobalt. They don't. The tool will return a clear error, but check COBALT_API_URL is set before attempting.
• Re-transcribing the same video. Transcription is the slowest operation (~10-30s). If you need the same transcript again, cache the result in your conversation.
• Fetching profiles to get tweet content. get_user_profile returns bio data, not tweets. Use get_user_tweets for tweet content.
• Using search_tweets without operators. Raw keyword search returns noisy results. Use from:user, min_faves:100, date ranges, etc. for precision.