Audio Scrape

Scrape audio sources: $ARGUMENTS

$ARGUMENTS should be one of:

• A person/topic name to search (e.g. "Alan Hirsch") — discovers podcasts via iTunes
• --rss https://feed.url/rss to scrape a specific RSS feed
• --episode-url https://example.com/episode.mp3 --title "Episode Title" to scrape a single audio file
• --refresh to re-transcribe episodes that don't have transcripts yet
• Empty — ask the user for the search term

Before Starting

1. Confirm OPENAI_API_KEY is set in .env.local — needed for Whisper transcription
2. Read src/lib/database/schema.ts for the podcastEpisodes, podcastSeries table definitions
3. Read scripts/audio-scrape.ts to understand the existing scrape script
4. Optionally confirm ffmpeg is installed (which ffmpeg) — needed for splitting large audio files (>25MB)

Pipeline Stages

Stage 1 — Discover Podcasts

1. iTunes Search API (free, no key needed): Search https://itunes.apple.com/search?term={query}&media=podcast
2. Multiple query variations: "{name}", "{name} podcast", "{name} interview", "{name} sermon"
3. Deduplicate feeds: Track by feedUrl to avoid duplicate processing
4. Delta check: Compare fetched episodes against existing external_id values in podcast_episodes

Stage 2 — Parse RSS Feeds

For each discovered feed:

1. Fetch the RSS/XML feed with a User-Agent header
2. Parse <item> elements extracting:
   • <title> → episode title
   • <enclosure url="..."> → audio URL (required, skip items without it)
   • <itunes:duration> → duration in seconds
   • <itunes:episode>, <itunes:season> → numbering
   • <pubDate> → published date (convert to ISO 8601)
   • <itunes:summary> or <description> → episode description
   • <itunes:image href="..."> → thumbnail
3. Filter relevance: Only keep episodes where the search term appears in title, description, or series name

Stage 3 — Download & Transcribe

For each new episode (not already in DB with a transcript):

1. Download audio via curl to data/audio/{external_id}.mp3 with caching
2. Transcribe via OpenAI Whisper API (whisper-1 model):
   • Files ≤ 25MB: single API call
   • Files > 25MB: split into 20-minute chunks via ffmpeg, transcribe each, concatenate
3. If transcription fails, set transcript to null and continue

Stage 4 — Chunk Transcripts for Search

Split each transcript into chunks for embedding/search:

• Target: 500 words per chunk
• Overlap: 75 words
• Stride: 425 words (500 - 75)
• Each chunk gets: chunk_index, text, start_offset
• Absorb tiny final chunks (< 75 words) into the previous chunk

Stage 5 — Database Upsert

Upsert into podcast_episodes and podcast_series tables.

Database Mapping

Map RSS/audio fields to the podcast_episodes table:

Key Design Rules

• Idempotent — Use upserts keyed on external_id (hash of audio URL). Safe to re-run.
• Resume-safe — Skip audio download if file already exists in data/audio/
• Incremental — Only download/transcribe new episodes
• Graceful degradation — Per-episode failures don't crash the pipeline
• All timestamps UTC — ISO 8601 throughout
• Pace API calls — 1-second delay between Whisper calls, 200ms between RSS fetches

Running

pnpm audio:scrape -- --search "Alan Hirsch"
pnpm audio:scrape -- --rss https://feed.example.com/rss
pnpm audio:scrape -- --episode-url https://example.com/episode.mp3 --title "My Episode"
pnpm audio:scrape -- --refresh
pnpm audio:scrape -- --search "Alan Hirsch" --max 10
pnpm audio:scrape -- --search "Alan Hirsch" --skip-transcribe
pnpm audio:scrape -- --search "Alan Hirsch" --source-type sermon

Output Format

## Audio Scrape Report

### Search: [query]
### Podcasts found: N feeds, M relevant episodes

### Stage 1 — Discovery: OK
- Feeds found: N
- Episodes parsed: M

### Stage 2 — Relevance filter: OK
- Relevant episodes: N

### Stage 3 — Transcripts: OK
- Transcribed: X
- Skipped (already exists): Y
- Failed: Z

### Stage 4 — Chunks: OK
- Total chunks: N across M episodes

### Stage 5 — Database:
- Series upserted: N
- Episodes inserted: X, updated: Y

### Warnings:
- [any non-blocking issues]

Cost Notes

• iTunes Search API: Free, no key needed
• RSS feeds: Free, public
• OpenAI Whisper API: ~$0.006/minute of audio
  • A 1-hour episode costs ~$0.36
  • 100 episodes averaging 45 min each = ~$27
• Audio downloads: bandwidth only; files cached in data/audio/
• Large files (>25MB) require ffmpeg for splitting

Error Handling

• Missing OPENAI_API_KEY → stop immediately with instructions
• Individual episode download failure → log warning, continue pipeline
• Whisper API error → log warning, set transcript null, continue
• RSS parse failure → log warning, skip feed, continue
• Large file without ffmpeg → attempt direct transcription (may fail for >25MB)