Youtube Bulk Ingest

Bulk YouTube Transcript Ingestion

Process all local YouTube transcripts in the workspace and extract insights with timestamp-linked sources. Designed to be run repeatedly — it skips already-indexed videos.

Directory Structure

Transcripts live at /workspace/group/youtube/{date}/{channel-handle}/{video-slug}/:

• transcript.json — structured transcript with timing: {"video_id", "language", "transcript": [{"text", "start", "duration"}]}
• transcript.txt — plain text with [seconds] prefix per line
• metadata/*.json — video metadata: {"video_id", "title", "author_name", "published", "viewCount", "link"}

Process

1. Discover All Videos

Scan /workspace/group/youtube/ recursively for directories containing transcript.json. Build a manifest:

find /workspace/group/youtube -name "transcript.json" -type f

For each transcript.json found, read the corresponding metadata/*.json (first .json file in the metadata/ subdirectory) to get the video URL, title, and author.

2. Filter to Unindexed Videos

For each video, call check_source with the YouTube URL (https://www.youtube.com/watch?v={video_id}). Skip any that return exists: true.

Report: "Found X total videos, Y already indexed, Z to process."

3. Process Each Video

For each unindexed video, follow the same insight extraction process as the /ingest skill:

a. Read the transcript

Read transcript.json. The transcript array contains segments with text, start (seconds), and duration.

Concatenate segments into readable chunks (aim for ~2000-3000 word chunks for long videos). Preserve the start timestamp for each chunk boundary so you can reference it later.

b. Extract insights

For each video, extract 5-15 insights. Each insight has TWO parts:

text (bold thesis) — SHORT, GENERALIZABLE principle (10-20 words). This is the dedup key. If two different videos from different creators express the same idea, their text should be nearly identical. Think headline, not paragraph.

Good: "AI commoditizes execution, making taste and curation the new scarce skills" Bad: "Greg Isenberg's guest James says you need taste to stand out from AI"

detail — 2-3 sentences expanding on the thesis with specific context from this video.

Also provide:

• category: strategy, technical, trend, principle, observation, tactic, or custom
• context: A direct quote from the transcript (1-3 sentences)
• timestamp_ref: The MM:SS or HH:MM:SS timestamp where this insight is discussed

Computing timestamp_ref from transcript.json: Find the transcript segment(s) where the insight is discussed. Use the start field:

• start: 754.2 → "12:34"
• start: 3661.0 → "1:01:01"
• Format: Math.floor(start/60) for minutes, Math.floor(start%60) for seconds

c. Deduplicate

For each insight, call search_insights to check for semantic matches. If a similar insight exists (same core thesis), call link_insight_source instead of add_insight. The abstract text field is what you compare — two insights match if they express the same generalizable principle.

d. Store with rich metadata

When calling add_insight or link_insight_source:

• source_url: https://www.youtube.com/watch?v={video_id}
• source_title: From metadata title
• source_type: youtube
• source_metadata: JSON string: {"author_name": "...", "video_id": "...", "published": "...", "viewCount": "...", "channel": "..."} (author_name from metadata JSON)
• context: The supporting quote
• timestamp_ref: The MM:SS string

e. CRITICAL: Refresh index after each video

After processing all insights for ONE video, call refresh_memory_index and wait for it to complete before moving to the next video. This ensures search_insights can find insights from previously-processed videos, enabling dedup across the batch.

Process videos SERIALLY, one at a time. Do NOT process multiple videos in parallel. The serial + refresh pattern maximizes dedup cache hits.

4. Progress Reporting

After every 5 videos processed, send a progress update via send_message:

Bulk ingest progress: X/Z videos processed (Y new insights, W linked to existing)

5. Final Summary

After all videos are processed, send a completion message:

Bulk YouTube Ingest Complete
Videos processed: X
New insights: Y
Linked to existing: Z
Total insights now: N
Top corroborated insight: "{text}" (K sources)

6. Refresh Index

Call refresh_memory_index at the end so all new insight markdown files get indexed for semantic search.

Performance Notes

• For very long videos (60+ min), focus on the 10-15 most impactful insights rather than exhaustively covering everything.
• Process videos from newest to oldest (sort by date directory) so the most recent content is prioritized.
• If a video has no metadata JSON, construct the URL from the video_id field in transcript.json.
• Skip directories named recommended or recommended-videos (these contain original content drafts, not transcripts).

Running

This skill can be triggered via:

• Telegram: "@Kia bulk ingest youtube transcripts"
• WebUI chat: "bulk ingest youtube"
• Scheduled task: schedule with context_mode: 'group' for access to conversation history

It's safe to run repeatedly — already-indexed videos are skipped automatically.