# Youtube Batch Extraction > Batch extraction pipeline for multiple YouTube playlists — URL discovery, rate-limited web_extract, checkpoint/resume, cron scheduling, retry logic. - Skill: `theheavenlyd3mon/youtube-batch-extraction` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds add theheavenlyd3mon/youtube-batch-extraction` - Raw SKILL.md: https://api.skillmd.com/api/skills/theheavenlyd3mon/youtube-batch-extraction/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Product & Planning - Author: theheavenlyd3mon (https://skillmd.com/u/theheavenlyd3mon) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/theheavenlyd3mon/youtube-batch-extraction --- # YouTube Batch Extraction Pipeline ## When to Use User wants to extract content from multiple YouTube playlists or a large number of videos (10+). This covers the full pipeline: URL discovery, rate-limited extraction, checkpoint/resume, cron scheduling, and retry logic. ## Overview Two extraction methods, chosen by need: 1. **yt-dlp transcripts** — fast (3-5s/video), only works if creator enabled captions 2. **web_extract summaries** — slower (10-15s/video), works on all videos, produces richer structured content For game dev tutorials, web_extract is preferred — it produces step-by-step implementation details, code snippets, and parameter values rather than raw timestamped text. **Preferred: Unified pipeline** (`scripts/batch_unified.py`) — tries transcript API first, falls back to page-scraped summary. Handles both paths automatically. Use this for new batch work. ## Prerequisites ```bash brew install yt-dlp # required for URL discovery and subtitle extraction ``` yt-dlp is NOT installed by default on macOS. The `execute_code` sandbox uses system Python (3.9) which doesn't have yt-dlp in PATH. Install via Homebrew — pip's `--user` flag is unsupported on modern macOS without a venv. ## Unified Pipeline (transcript-first, summary-fallback) The preferred approach for batch processing: try `youtube-transcript-api` for raw transcripts first, fall back to page-scraped summaries for videos without captions. Use `scripts/batch_unified.py` (in this skill's scripts/ directory): ```bash # Single playlist (use ABSOLUTE paths — ~ expands to sandbox home in cron/agent context) python3 ~/.hermes/profiles/senna/skills/media/youtube-batch-extraction/scripts/batch_unified.py \ /tmp/PLAYLIST_NAME_urls.txt \ --outdir ~/Documents/YouTube-Transcripts/PLAYLIST_NAME \ --name PLAYLIST_NAME \ --delay 15 --summary-delay 8 # Process a subset (lines 41-43 from a 1-indexed file → --start 40 --limit 3) python3 .../batch_unified.py /tmp/urls.txt --start 40 --limit 3 --outdir /absolute/path --name NAME # All playlists at once python3 SKILL_DIR/scripts/batch_unified.py --all --delay 15 --summary-delay 5 ``` **Critical `--outdir` behavior:** The script ALWAYS creates a nested directory: `outdir/PLAYLIST_NAME/`. It does NOT write directly to `outdir`. After the script completes, files are at `outdir/PLAYLIST_NAME/*.md` and the checkpoint is at `outdir/PLAYLIST_NAME/.checkpoint.json`. You must move/copy them to the intended final location. **Critical `--start` indexing:** `--start` is 0-based (first line = 0). URL files are 1-indexed (line 1 = first video). To process lines 41-43: use `--start 40 --limit 3`. Using `--start 41` skips the first video and processes lines 42-44. **Delay settings:** - `--delay 15` — seconds between transcript API calls (15s minimum to avoid IP bans on 50+ video playlists; 30s for 80+ videos) - `--summary-delay 5` — seconds between page-scrape summaries (safer, no API ban risk) **Checkpoint format:** Dict with `video_id -> "transcript"|"summary"`. Handles migration from old list format automatically. **Output:** Transcripts are 20-60KB with timestamps. Summaries are 1-2KB with title, channel, duration, description, and chapters extracted from page source. ## Pipeline Steps ### Step 0: Check Existing Output (ALWAYS FIRST) Before running ANY pipeline step, check whether prior runs already completed the work: ```python from hermes_tools import terminal for name in playlists: outdir = f"~/Documents/YouTube-Transcripts/{name}" total = terminal(f"ls '{outdir}'/*.md 2>/dev/null | wc -l")["output"].strip() good = terminal(f"find '{outdir}' -name '*.md' -size +200c | wc -l")["output"].strip() stubs = terminal(f"find '{outdir}' -name '*.md' ! -size +200c | wc -l")["output"].strip() print(f"{name}: {total} files, {good} good (>200b), {stubs} stubs") ``` If `good` count meets or exceeds expected video count, that playlist is DONE — skip it. Don't rebuild URL lists, don't install tools, don't re-process. Only work on playlists with missing or stub files. ### Step 1: Discover URLs ```bash YTDLP="~/.hermes/hermes-agent/venv/bin/yt-dlp" "$YTDLP" --flat-playlist --dump-single-json --no-warnings "PLAYLIST_URL" | \ python3 -c " import json,sys d=json.load(sys.stdin) for e in d.get('entries',[]): vid=e.get('id') or e.get('url','').split('v=')[-1].split('&')[0] print(f'{vid}|{e.get(\"title\",\"Untitled\")}') " > /tmp/PLAYLIST_NAME_urls.txt ``` Save to `/tmp/PLAYLIST_NAME_urls.txt` — one `video_id|title` per line. Use `--flat-playlist` to avoid fetching full metadata (very slow on 50+ video playlists). ### Step 2: Extract via web_extract (agent-driven) - Call `web_extract` in batches of 5 URLs max (API limit) - 8-10 second delay between batches - Save each summary as `.md` with YAML frontmatter: ```yaml --- title: "Video Title" source: "https://www.youtube.com/watch?v=VIDEO_ID" video_id: "VIDEO_ID" type: "youtube-summary" tags: [ue5, tutorial, topic, gamedev] --- ``` - Write files via Python script in terminal with explicit absolute paths (most reliable method across all session types); see Pitfalls section for write_file and heredoc gotchas - Output dir: `~/Documents/YouTube-Transcripts/PLAYLIST_NAME/` ### Step 3: Checkpoint tracking Save `~/Documents/YouTube-Transcripts/PLAYLIST_NAME/.checkpoint.json`. **Current format (dict)** — tracks content type: ```json {"done": {"vid1": "transcript", "vid2": "summary"}, "failed": ["vid3"]} ``` **Legacy format (list)** — just video IDs: ```json {"done": ["vid1", "vid2"], "failed": ["vid3"]} ``` Always handle migration on load: ```python done = data.get("done", {}) if isinstance(done, list): done = {vid: "transcript" for vid in done} data["done"] = done ``` The dict format lets you know which files are real transcripts (>5KB) vs summaries (<2KB) for future re-processing. ### Step 4: Cron scheduling for large batches - One agent-driven cron job per 20-video batch - **30 minutes between jobs** (generous spacing prevents overlap) - Each job reads `/tmp/PLAYLIST_NAME_urls.txt`, skips already-done videos - Retry job scheduled 30 min after last batch - Schedule at least 15 min in the future (past times get auto-cleaned silently) ### Step 5: Retry failed videos - Check `.checkpoint.json` for "failed" list - Retry with 10s delay between attempts - Some videos fail on first pass due to transient YouTube errors ## Output Format ```markdown --- title: "Video Title" source: "https://www.youtube.com/watch?v=VIDEO_ID" video_id: "VIDEO_ID" type: "youtube-summary" tags: [ue5, tutorial, topic, gamedev] --- # Video Title **URL:** ... **Creator:** ... **Length:** ... ## Overview [2-3 sentence summary] ## Key Implementation Details [Structured content with code snippets, node names, parameter values] ## Timestamps | Time | Topic | |------|-------| | 0:00 | ... | ``` ## Pitfalls - **File writing reliability (critical)**: Neither `write_file` nor terminal heredocs are fully reliable for writing markdown summaries: - `write_file` may report success (bytes_written, resolved_path) but files don't actually persist to the real filesystem. In sandboxed/cron environments, it may write to a virtual overlay that doesn't match the real path. - Terminal heredocs (`cat > file << 'EOF'`) fail when markdown content contains shell-interpreted characters (`&`, backticks, `$`, `*` in certain contexts). Error: "Foreground command uses '&' backgrounding." - Terminal `~` resolves to the sandbox home (e.g., `~/.hermes/profiles/senna/home/...`) while `write_file` resolves to the real home (`~/...`). Files split across both locations. - **Reliable method**: Use Python via terminal with explicit absolute paths: ```python python3 -c " import os dest = '~/Documents/YouTube-Transcripts/PLAYLIST_NAME' for name, content in files.items(): with open(os.path.join(dest, name), 'w') as f: f.write(content) " ``` - Always verify with `terminal` ls/find after writing, regardless of method used. - **hermes_tools not importable**: `web_extract` etc. are agent-internal. Background scripts CANNOT use them. Must use agent-driven cron jobs. - **Cron auto-cleanup**: One-shot jobs scheduled in the past disappear silently. Schedule 15+ min in the future. - **Non-English extraction results**: web_extract sometimes returns summaries in the video's spoken language rather than English (e.g., Chinese for a Chinese-dubbed tutorial). When this happens, translate and rewrite the summary in the target language when saving the .md file — don't save non-English content to an English knowledge base. - **Rate limiting**: After rapid-fire YouTube requests, you may get IP-blocked for 15-30 min. Use 8-10s delays. After a block, wait 30+ min. - **403 errors are often transient — retry immediately.** When `web_extract` returns 403 Forbidden on YouTube URLs, retry the same URLs in the next batch cycle. In a 20-video batch processed in 4 groups of 5, 5 videos initially returned 403 — all 5 succeeded on immediate retry (no 30-min cooldown needed). This is distinct from IP-level rate blocking. The 403 appears to be YouTube's per-request scraping defense, not an IP ban. Pattern: collect failures during batch processing → retry all failures in a single final batch → only mark truly failed after retry. - **Sibling subagent file conflicts (harmless).** When multiple Hermes sessions or subagents write to the same output directory concurrently, `write_file` may return warnings like "was modified by sibling subagent... but this agent never read it." This is a safety check, not an error — the writes still succeed. The warning appears when two agents write the same filename within a short window. If processing a playlist with multiple concurrent cron jobs, expect these warnings. They don't affect file content or integrity. No action needed. - **`wc -l` returns 0 for heredoc-written files.** The `cat > file << 'EOF'` heredoc pattern strips trailing newlines. `wc -l` counts newline characters, so a file with content but no final `\n` reports 0 lines. This caused a false "0 URLs" report when the files actually had 43 and 85 entries. **Fix:** Use `wc -c` (byte count) or `ls -la` to verify file size, or `python3 -c "print(len(open(f).readlines()))"` which counts lines regardless of trailing newline. - **`execute_code` quoting hell with shell-in-Python.** When `execute_code` scripts need to run shell commands with complex quoting (nested single/double/escaped quotes), the string escaping becomes fragile and breaks. **Fix:** Write the helper script to disk first via `terminal()` with a heredoc, then call it by path: ```python terminal(f"cat > /tmp/helper.py << 'EOF'\n{script_content}\nEOF") terminal(f"python3 /tmp/helper.py '{arg1}' '{arg2}'") ``` This avoids all quoting issues since the script content is in a quoted heredoc and arguments are passed separately. - **`yt-dlp --flat-playlist`: Always use for playlist metadata.** Without it, yt-dlp fetches full video info for ALL videos (extremely slow on large playlists). - **Python type annotations**: Use `from __future__ import annotations` for `|` syntax on Python <3.10. Always handle None titles from yt-dlp — skip lines where title is "None" or empty when reading URL files. - **Cron/transcript conflict**: If cron jobs are running `web_extract` summaries to the same directory where you're fetching raw transcripts, they'll create files with different names and sizes. Summaries use short filenames (`060-title.md`, 1-2KB) while transcripts use full titles (`Playlist_Name_-_#60_Title.md`, 20-60KB). **Rule:** Never run both approaches to the same output directory simultaneously. If switching from summaries to transcripts, delete old summary files first (filter by size: <5KB = summary, >5KB = transcript). Check for re-created summaries periodically during long transcript batches. - **Most videos in a playlist may lack captions.** Don't assume all videos have transcripts. In a typical tutorial playlist, only 10-30% may have captions enabled. The unified pipeline handles this gracefully — transcript API returns `None`, summary fallback kicks in. Report the final breakdown (📝 X transcripts, 📋 Y summaries, ❌ Z failed) so the user knows the coverage. - **Don't present options when you can execute.** If the pipeline breaks, fix it and deliver. Don't lay out 4 options and ask which direction. The user said "run it" meaning "stop talking and deliver results." - **YouTube page-source summary extraction**: When `youtube-transcript-api` and `web_extract` both fail or are rate-limited, you can still extract useful metadata directly from the YouTube page via curl + regex. The `batch_unified.py` script's `fetch_summary_via_curl()` function does this — extracts title, channel, duration, description, and chapters from the page HTML. This never gets rate-limited (same as loading a webpage) but produces less structured content than `web_extract`. - **Cron job spacing**: 30 min between jobs. Each batch of 20 takes ~5-10 min. Tighter spacing causes overlap failures. - **Post-batch file relocation required**: Because `--outdir` always nests (`outdir/PLAYLIST_NAME/`), files end up in a subdirectory. After the script completes: (1) copy/move `*.md` and `.checkpoint.json` from the nested dir to the intended output dir, (2) verify with `ls -la` on the target, (3) clean up the nested dir. Do this immediately after each batch run — don't leave files in the wrong location. - **YouTube IP block = expected fallback, not an error**: After processing a batch of transcripts (typically 3-5 videos), YouTube will IP-block the transcript API. The `batch_unified.py` script automatically falls back to page-scraped summaries for remaining videos. This is normal — report the breakdown (📝 N transcripts, 📋 M summaries) and note that transcripts can be retried after 30+ min cooldown. Don't retry transcripts in the same session. - **Merge with previous-run transcripts**: When a directory already contains transcript files from prior runs (>2KB, with timestamps) and a new batch produces summaries (<2KB, description+chapters only) for the same videos, KEEP the transcripts. They contain more detailed content. After a batch run: (1) detect duplicates by video_id in frontmatter, (2) keep the largest file, (3) move smaller duplicates to `_duplicates/`, (4) update checkpoint to reflect actual content types. ## Post-Extraction Organization After extraction completes, the output directory typically needs cleanup: duplicate files from multiple runs, inconsistent naming, and a mix of educational vs non-educational content. See `references/post-extraction-organization.md` for the full workflow covering: - **Content quality triage**: Classify files as educational (steps, >200 words, structured) vs non-educational (YouTube descriptions, chapter lists, links-only). Script pattern included. - **Folder deduplication**: When the same series exists across multiple directories, compare file sizes, keep the largest version, move dupes to `_duplicates/`. - **File naming normalization**: Consistent `NN_Topic_Name.md` format, strip special chars, remove long prefixes. Use `os.rename()` (not shell) for files with `#`, `!`, `,` in names. - **`_duplicates/` protection**: Never run broad `find -delete` or `xargs rm` against the extraction root — scope cleanup commands to specific directories or use Python with explicit path checks. ## Obsidian Vault Preparation After organization, convert the directory into an Obsidian vault. See `references/obsidian-vault-preparation.md` for the full workflow covering: - **Frontmatter standardization**: Consistent YAML with title, source, video_id, type, series, episode, tags. Folder→series mapping for batch processing. - **Wikilink generation**: `## Related` section with `← Previous`, `→ Next`, `📚 Series: [[_MOC_Folder]]` links. Cross-folder references for Step-by-Step guides → source transcripts. - **MOC index files**: One `_MOC_FolderName.md` per folder linking to all files. Entry point for browsing the vault. - **Verification**: 100% frontmatter, 99%+ wikilinks, all folders have MOC.