# Youtube Bulk Ingest

> Bulk ingest YouTube transcripts from the local workspace into the insight tracking system. Scans the youtube/ directory for all videos with transcripts, skips already-indexed ones, and extracts insights with timestamps. Reusable for future batches. Triggers on "youtube bulk ingest", "ingest all videos", "process youtube transcripts", "bulk ingest".

- Skill: `harperaa/youtube-bulk-ingest` (Agent Skill)
- Install (CLI): `npx skillmds@latest add harperaa/youtube-bulk-ingest`
- Raw SKILL.md: https://api.skillmd.com/api/skills/harperaa/youtube-bulk-ingest/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: harperaa (https://skillmd.com/u/harperaa)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/harperaa/youtube-bulk-ingest

---


# Bulk YouTube Transcript Ingestion

Process all local YouTube transcripts in the workspace and extract insights with timestamp-linked sources. Designed to be run repeatedly — it skips already-indexed videos.

## Directory Structure

Transcripts live at `/workspace/group/youtube/{date}/{channel-handle}/{video-slug}/`:
- `transcript.json` — structured transcript with timing: `{"video_id", "language", "transcript": [{"text", "start", "duration"}]}`
- `transcript.txt` — plain text with `[seconds]` prefix per line
- `metadata/*.json` — video metadata: `{"video_id", "title", "author_name", "published", "viewCount", "link"}`

## Process

### 1. Discover All Videos

Scan `/workspace/group/youtube/` recursively for directories containing `transcript.json`. Build a manifest:

```bash
find /workspace/group/youtube -name "transcript.json" -type f
```

For each transcript.json found, read the corresponding `metadata/*.json` (first .json file in the metadata/ subdirectory) to get the video URL, title, and author.

### 2. Filter to Unindexed Videos

For each video, call `check_source` with the YouTube URL (`https://www.youtube.com/watch?v={video_id}`). Skip any that return `exists: true`.

Report: "Found X total videos, Y already indexed, Z to process."

### 3. Process Each Video

For each unindexed video, follow the same insight extraction process as the `/ingest` skill:

#### a. Read the transcript
Read `transcript.json`. The transcript array contains segments with `text`, `start` (seconds), and `duration`.

Concatenate segments into readable chunks (aim for ~2000-3000 word chunks for long videos). Preserve the `start` timestamp for each chunk boundary so you can reference it later.

#### b. Extract insights
For each video, extract **5-15 insights**. Each insight has TWO parts:

**text** (bold thesis) — SHORT, GENERALIZABLE principle (10-20 words). This is the dedup key. If two different videos from different creators express the same idea, their `text` should be nearly identical. Think headline, not paragraph.

Good: "AI commoditizes execution, making taste and curation the new scarce skills"
Bad: "Greg Isenberg's guest James says you need taste to stand out from AI"

**detail** — 2-3 sentences expanding on the thesis with specific context from this video.

Also provide:
- **category**: `strategy`, `technical`, `trend`, `principle`, `observation`, `tactic`, or custom
- **context**: A direct quote from the transcript (1-3 sentences)
- **timestamp_ref**: The `MM:SS` or `HH:MM:SS` timestamp where this insight is discussed

**Computing timestamp_ref from transcript.json:**
Find the transcript segment(s) where the insight is discussed. Use the `start` field:
- `start: 754.2` → `"12:34"`
- `start: 3661.0` → `"1:01:01"`
- Format: `Math.floor(start/60)` for minutes, `Math.floor(start%60)` for seconds

#### c. Deduplicate
For each insight, call `search_insights` to check for semantic matches. If a similar insight exists (same core thesis), call `link_insight_source` instead of `add_insight`. The abstract `text` field is what you compare — two insights match if they express the same generalizable principle.

#### d. Store with rich metadata
When calling `add_insight` or `link_insight_source`:
- `source_url`: `https://www.youtube.com/watch?v={video_id}`
- `source_title`: From metadata `title`
- `source_type`: `youtube`
- `source_metadata`: JSON string: `{"author_name": "...", "video_id": "...", "published": "...", "viewCount": "...", "channel": "..."}`  (author_name from metadata JSON)
- `context`: The supporting quote
- `timestamp_ref`: The `MM:SS` string

#### e. CRITICAL: Refresh index after each video
After processing all insights for ONE video, call `refresh_memory_index` and wait for it to complete before moving to the next video. This ensures `search_insights` can find insights from previously-processed videos, enabling dedup across the batch.

**Process videos SERIALLY, one at a time.** Do NOT process multiple videos in parallel. The serial + refresh pattern maximizes dedup cache hits.

### 4. Progress Reporting

After every 5 videos processed, send a progress update via `send_message`:
```
Bulk ingest progress: X/Z videos processed (Y new insights, W linked to existing)
```

### 5. Final Summary

After all videos are processed, send a completion message:
```
Bulk YouTube Ingest Complete
Videos processed: X
New insights: Y
Linked to existing: Z
Total insights now: N
Top corroborated insight: "{text}" (K sources)
```

### 6. Refresh Index

Call `refresh_memory_index` at the end so all new insight markdown files get indexed for semantic search.

## Performance Notes

- For very long videos (60+ min), focus on the **10-15 most impactful insights** rather than exhaustively covering everything.
- Process videos from newest to oldest (sort by date directory) so the most recent content is prioritized.
- If a video has no metadata JSON, construct the URL from the `video_id` field in transcript.json.
- Skip directories named `recommended` or `recommended-videos` (these contain original content drafts, not transcripts).

## Running

This skill can be triggered via:
- Telegram: "@Kia bulk ingest youtube transcripts"
- WebUI chat: "bulk ingest youtube"
- Scheduled task: schedule with `context_mode: 'group'` for access to conversation history

It's safe to run repeatedly — already-indexed videos are skipped automatically.