Youtube Batch Extraction

YouTube Batch Extraction Pipeline

When to Use

User wants to extract content from multiple YouTube playlists or a large number of videos (10+). This covers the full pipeline: URL discovery, rate-limited extraction, checkpoint/resume, cron scheduling, and retry logic.

Overview

Two extraction methods, chosen by need:

1. yt-dlp transcripts — fast (3-5s/video), only works if creator enabled captions
2. web_extract summaries — slower (10-15s/video), works on all videos, produces richer structured content

For game dev tutorials, web_extract is preferred — it produces step-by-step implementation details, code snippets, and parameter values rather than raw timestamped text.

Preferred: Unified pipeline (scripts/batch_unified.py) — tries transcript API first, falls back to page-scraped summary. Handles both paths automatically. Use this for new batch work.

Prerequisites

brew install yt-dlp   # required for URL discovery and subtitle extraction

yt-dlp is NOT installed by default on macOS. The execute_code sandbox uses system Python (3.9) which doesn't have yt-dlp in PATH. Install via Homebrew — pip's --user flag is unsupported on modern macOS without a venv.

Unified Pipeline (transcript-first, summary-fallback)

The preferred approach for batch processing: try youtube-transcript-api for raw transcripts first, fall back to page-scraped summaries for videos without captions. Use scripts/batch_unified.py (in this skill's scripts/ directory):

# Single playlist (use ABSOLUTE paths — ~ expands to sandbox home in cron/agent context)
python3 ~/.hermes/profiles/senna/skills/media/youtube-batch-extraction/scripts/batch_unified.py \
  /tmp/PLAYLIST_NAME_urls.txt \
  --outdir ~/Documents/YouTube-Transcripts/PLAYLIST_NAME \
  --name PLAYLIST_NAME \
  --delay 15 --summary-delay 8

# Process a subset (lines 41-43 from a 1-indexed file → --start 40 --limit 3)
python3 .../batch_unified.py /tmp/urls.txt --start 40 --limit 3 --outdir /absolute/path --name NAME

# All playlists at once
python3 SKILL_DIR/scripts/batch_unified.py --all --delay 15 --summary-delay 5

Critical --outdir behavior: The script ALWAYS creates a nested directory: outdir/PLAYLIST_NAME/. It does NOT write directly to outdir. After the script completes, files are at outdir/PLAYLIST_NAME/*.md and the checkpoint is at outdir/PLAYLIST_NAME/.checkpoint.json. You must move/copy them to the intended final location.

Critical --start indexing: --start is 0-based (first line = 0). URL files are 1-indexed (line 1 = first video). To process lines 41-43: use --start 40 --limit 3. Using --start 41 skips the first video and processes lines 42-44.

Delay settings:

• --delay 15 — seconds between transcript API calls (15s minimum to avoid IP bans on 50+ video playlists; 30s for 80+ videos)
• --summary-delay 5 — seconds between page-scrape summaries (safer, no API ban risk)

Checkpoint format: Dict with video_id -> "transcript"|"summary". Handles migration from old list format automatically.

Output: Transcripts are 20-60KB with timestamps. Summaries are 1-2KB with title, channel, duration, description, and chapters extracted from page source.

Pipeline Steps

Step 0: Check Existing Output (ALWAYS FIRST)

Before running ANY pipeline step, check whether prior runs already completed the work:

from hermes_tools import terminal

for name in playlists:
    outdir = f"~/Documents/YouTube-Transcripts/{name}"
    total = terminal(f"ls '{outdir}'/*.md 2>/dev/null | wc -l")["output"].strip()
    good = terminal(f"find '{outdir}' -name '*.md' -size +200c | wc -l")["output"].strip()
    stubs = terminal(f"find '{outdir}' -name '*.md' ! -size +200c | wc -l")["output"].strip()
    print(f"{name}: {total} files, {good} good (>200b), {stubs} stubs")

If good count meets or exceeds expected video count, that playlist is DONE — skip it. Don't rebuild URL lists, don't install tools, don't re-process. Only work on playlists with missing or stub files.

Step 1: Discover URLs

YTDLP="~/.hermes/hermes-agent/venv/bin/yt-dlp"
"$YTDLP" --flat-playlist --dump-single-json --no-warnings "PLAYLIST_URL" | \
  python3 -c "
import json,sys
d=json.load(sys.stdin)
for e in d.get('entries',[]):
    vid=e.get('id') or e.get('url','').split('v=')[-1].split('&')[0]
    print(f'{vid}|{e.get(\"title\",\"Untitled\")}')
" > /tmp/PLAYLIST_NAME_urls.txt

Save to /tmp/PLAYLIST_NAME_urls.txt — one video_id|title per line. Use --flat-playlist to avoid fetching full metadata (very slow on 50+ video playlists).

Step 2: Extract via web_extract (agent-driven)

• Call web_extract in batches of 5 URLs max (API limit)
• 8-10 second delay between batches
• Save each summary as .md with YAML frontmatter:

---
title: "Video Title"
source: "https://www.youtube.com/watch?v=VIDEO_ID"
video_id: "VIDEO_ID"
type: "youtube-summary"
tags: [ue5, tutorial, topic, gamedev]
---

• Write files via Python script in terminal with explicit absolute paths (most reliable method across all session types); see Pitfalls section for write_file and heredoc gotchas
• Output dir: ~/Documents/YouTube-Transcripts/PLAYLIST_NAME/

Step 3: Checkpoint tracking

Save ~/Documents/YouTube-Transcripts/PLAYLIST_NAME/.checkpoint.json.

Current format (dict) — tracks content type:

{"done": {"vid1": "transcript", "vid2": "summary"}, "failed": ["vid3"]}

Legacy format (list) — just video IDs:

{"done": ["vid1", "vid2"], "failed": ["vid3"]}

Always handle migration on load:

done = data.get("done", {})
if isinstance(done, list):
    done = {vid: "transcript" for vid in done}
    data["done"] = done

The dict format lets you know which files are real transcripts (>5KB) vs summaries (<2KB) for future re-processing.

Step 4: Cron scheduling for large batches

• One agent-driven cron job per 20-video batch
• 30 minutes between jobs (generous spacing prevents overlap)
• Each job reads /tmp/PLAYLIST_NAME_urls.txt, skips already-done videos
• Retry job scheduled 30 min after last batch
• Schedule at least 15 min in the future (past times get auto-cleaned silently)

Step 5: Retry failed videos

• Check .checkpoint.json for "failed" list
• Retry with 10s delay between attempts
• Some videos fail on first pass due to transient YouTube errors

Output Format

---
title: "Video Title"
source: "https://www.youtube.com/watch?v=VIDEO_ID"
video_id: "VIDEO_ID"
type: "youtube-summary"
tags: [ue5, tutorial, topic, gamedev]
---

# Video Title

**URL:** ...
**Creator:** ...
**Length:** ...

## Overview
[2-3 sentence summary]

## Key Implementation Details
[Structured content with code snippets, node names, parameter values]

## Timestamps
| Time | Topic |
|------|-------|
| 0:00 | ... |

Pitfalls

• File writing reliability (critical): Neither write_file nor terminal heredocs are fully reliable for writing markdown summaries:
  • write_file may report success (bytes_written, resolved_path) but files don't actually persist to the real filesystem. In sandboxed/cron environments, it may write to a virtual overlay that doesn't match the real path.
  • Terminal heredocs (cat > file << 'EOF') fail when markdown content contains shell-interpreted characters (&, backticks, $, * in certain contexts). Error: "Foreground command uses '&' backgrounding."
  • Terminal ~ resolves to the sandbox home (e.g., ~/.hermes/profiles/senna/home/...) while write_file resolves to the real home (~/...). Files split across both locations.
  • Reliable method: Use Python via terminal with explicit absolute paths:

    python3 -c "
    import os
    dest = '~/Documents/YouTube-Transcripts/PLAYLIST_NAME'
    for name, content in files.items():
        with open(os.path.join(dest, name), 'w') as f:
            f.write(content)
    "
    

  • Always verify with terminal ls/find after writing, regardless of method used.
• hermes_tools not importable: web_extract etc. are agent-internal. Background scripts CANNOT use them. Must use agent-driven cron jobs.
• Cron auto-cleanup: One-shot jobs scheduled in the past disappear silently. Schedule 15+ min in the future.
• Non-English extraction results: web_extract sometimes returns summaries in the video's spoken language rather than English (e.g., Chinese for a Chinese-dubbed tutorial). When this happens, translate and rewrite the summary in the target language when saving the .md file — don't save non-English content to an English knowledge base.
• Rate limiting: After rapid-fire YouTube requests, you may get IP-blocked for 15-30 min. Use 8-10s delays. After a block, wait 30+ min.
• 403 errors are often transient — retry immediately. When web_extract returns 403 Forbidden on YouTube URLs, retry the same URLs in the next batch cycle. In a 20-video batch processed in 4 groups of 5, 5 videos initially returned 403 — all 5 succeeded on immediate retry (no 30-min cooldown needed). This is distinct from IP-level rate blocking. The 403 appears to be YouTube's per-request scraping defense, not an IP ban. Pattern: collect failures during batch processing → retry all failures in a single final batch → only mark truly failed after retry.
• Sibling subagent file conflicts (harmless). When multiple Hermes sessions or subagents write to the same output directory concurrently, write_file may return warnings like "was modified by sibling subagent... but this agent never read it." This is a safety check, not an error — the writes still succeed. The warning appears when two agents write the same filename within a short window. If processing a playlist with multiple concurrent cron jobs, expect these warnings. They don't affect file content or integrity. No action needed.
• wc -l returns 0 for heredoc-written files. The cat > file << 'EOF' heredoc pattern strips trailing newlines. wc -l counts newline characters, so a file with content but no final \n reports 0 lines. This caused a false "0 URLs" report when the files actually had 43 and 85 entries. Fix: Use wc -c (byte count) or ls -la to verify file size, or python3 -c "print(len(open(f).readlines()))" which counts lines regardless of trailing newline.
• execute_code quoting hell with shell-in-Python. When execute_code scripts need to run shell commands with complex quoting (nested single/double/escaped quotes), the string escaping becomes fragile and breaks. Fix: Write the helper script to disk first via terminal() with a heredoc, then call it by path:

  terminal(f"cat > /tmp/helper.py << 'EOF'\n{script_content}\nEOF")
  terminal(f"python3 /tmp/helper.py '{arg1}' '{arg2}'")
  

  This avoids all quoting issues since the script content is in a quoted heredoc and arguments are passed separately.
• yt-dlp --flat-playlist: Always use for playlist metadata. Without it, yt-dlp fetches full video info for ALL videos (extremely slow on large playlists).
• Python type annotations: Use from __future__ import annotations for | syntax on Python <3.10. Always handle None titles from yt-dlp — skip lines where title is "None" or empty when reading URL files.
• Cron/transcript conflict: If cron jobs are running web_extract summaries to the same directory where you're fetching raw transcripts, they'll create files with different names and sizes. Summaries use short filenames (060-title.md, 1-2KB) while transcripts use full titles (Playlist_Name_-_#60_Title.md, 20-60KB). Rule: Never run both approaches to the same output directory simultaneously. If switching from summaries to transcripts, delete old summary files first (filter by size: <5KB = summary, >5KB = transcript). Check for re-created summaries periodically during long transcript batches.
• Most videos in a playlist may lack captions. Don't assume all videos have transcripts. In a typical tutorial playlist, only 10-30% may have captions enabled. The unified pipeline handles this gracefully — transcript API returns None, summary fallback kicks in. Report the final breakdown (📝 X transcripts, 📋 Y summaries, ❌ Z failed) so the user knows the coverage.
• Don't present options when you can execute. If the pipeline breaks, fix it and deliver. Don't lay out 4 options and ask which direction. The user said "run it" meaning "stop talking and deliver results."
• YouTube page-source summary extraction: When youtube-transcript-api and web_extract both fail or are rate-limited, you can still extract useful metadata directly from the YouTube page via curl + regex. The batch_unified.py script's fetch_summary_via_curl() function does this — extracts title, channel, duration, description, and chapters from the page HTML. This never gets rate-limited (same as loading a webpage) but produces less structured content than web_extract.
• Cron job spacing: 30 min between jobs. Each batch of 20 takes ~5-10 min. Tighter spacing causes overlap failures.
• Post-batch file relocation required: Because --outdir always nests (outdir/PLAYLIST_NAME/), files end up in a subdirectory. After the script completes: (1) copy/move *.md and .checkpoint.json from the nested dir to the intended output dir, (2) verify with ls -la on the target, (3) clean up the nested dir. Do this immediately after each batch run — don't leave files in the wrong location.
• YouTube IP block = expected fallback, not an error: After processing a batch of transcripts (typically 3-5 videos), YouTube will IP-block the transcript API. The batch_unified.py script automatically falls back to page-scraped summaries for remaining videos. This is normal — report the breakdown (📝 N transcripts, 📋 M summaries) and note that transcripts can be retried after 30+ min cooldown. Don't retry transcripts in the same session.
• Merge with previous-run transcripts: When a directory already contains transcript files from prior runs (>2KB, with timestamps) and a new batch produces summaries (<2KB, description+chapters only) for the same videos, KEEP the transcripts. They contain more detailed content. After a batch run: (1) detect duplicates by video_id in frontmatter, (2) keep the largest file, (3) move smaller duplicates to _duplicates/, (4) update checkpoint to reflect actual content types.

Post-Extraction Organization

After extraction completes, the output directory typically needs cleanup: duplicate files from multiple runs, inconsistent naming, and a mix of educational vs non-educational content. See references/post-extraction-organization.md for the full workflow covering:

• Content quality triage: Classify files as educational (steps, >200 words, structured) vs non-educational (YouTube descriptions, chapter lists, links-only). Script pattern included.
• Folder deduplication: When the same series exists across multiple directories, compare file sizes, keep the largest version, move dupes to _duplicates/.
• File naming normalization: Consistent NN_Topic_Name.md format, strip special chars, remove long prefixes. Use os.rename() (not shell) for files with #, !, , in names.
• _duplicates/ protection: Never run broad find -delete or xargs rm against the extraction root — scope cleanup commands to specific directories or use Python with explicit path checks.

Obsidian Vault Preparation

After organization, convert the directory into an Obsidian vault. See references/obsidian-vault-preparation.md for the full workflow covering:

• Frontmatter standardization: Consistent YAML with title, source, video_id, type, series, episode, tags. Folder→series mapping for batch processing.
• Wikilink generation: ## Related section with ← Previous, → Next, 📚 Series: [[_MOC_Folder]] links. Cross-folder references for Step-by-Step guides → source transcripts.
• MOC index files: One _MOC_FolderName.md per folder linking to all files. Entry point for browsing the vault.
• Verification: 100% frontmatter, 99%+ wikilinks, all folders have MOC.