Video Ingest

Video Ingest — via Inngest Pipeline

Videos are ingested through the Inngest event bus. Do not run yt-dlp, mlx-whisper, scp, or create vault notes manually. The pipeline handles everything: download → NAS transfer → transcription → vault note → summary enrichment.

Quick Start

joelclaw send pipeline/video.requested -d '{"url":"URL_HERE"}'

# legacy alias still accepted
joelclaw send pipeline/video.download -d '{"url":"URL_HERE"}'

That's it. The event chain handles the rest.

Alternative (raw curl):

curl -s -X POST "http://localhost:8288/e/$INNGEST_EVENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"pipeline/video.requested","data":{"url":"URL_HERE"}}'

Pipeline Flow

pipeline/video.requested       — you send this (legacy alias: pipeline/video.download)
    ↓
video-download function        — yt-dlp → /tmp → NAS transfer
    ↓ emits
pipeline/video.downloaded      — logged by system-logger
pipeline/transcript.requested  — auto-triggered (legacy alias: pipeline/transcript.process)
    ↓
transcript-process function    — mlx-whisper (M4 Pro, ~5min/hr of video)
    ↓ emits
pipeline/transcript.processed  — logged
content/summarize.requested    — auto-triggered (legacy alias: content/summarize)
    ↓
content-summarize function     — pi enrichment → vault note with summary
    ↓ emits
content/summarized             — logged, done

Before Sending: Health Check

Always verify the pipeline is healthy before sending events:

# Inngest server
curl -s http://localhost:8288/health

# Worker (should show functions including video-download)
curl -s http://localhost:3111/ | python3 -c "
import json,sys
d=json.load(sys.stdin)
fns = [f.get('id','?') for f in d.get('functions',[])]
print(f'Worker OK — {len(fns)} functions: {', '.join(fns)}')
"

# Docker container running
docker ps --filter ancestor=inngest/inngest --format "table {{.Status}}\t{{.Ports}}"

If the worker is down:

kubectl -n joelclaw rollout restart deployment/system-bus-worker
kubectl -n joelclaw rollout status deployment/system-bus-worker --timeout=180s
joelclaw refresh

Monitoring a Run

Watch progress in real-time

# Worker logs — shows step execution + failures
kubectl logs -n joelclaw deploy/system-bus-worker -f

# Docker logs — shows event dispatch
docker logs -f $(docker ps -q --filter ancestor=inngest/inngest) 2>&1 | grep -v DEBUG

Check if events fired

# Look for the video's events in Docker logs
docker logs $(docker ps -q --filter ancestor=inngest/inngest) 2>&1 | grep -i "video\|transcript\|summarize" | tail -20

Dashboard

Open http://localhost:8288 in browser — shows functions, events, runs with per-step traces.

Verify completion

# Check if vault note was created
ls -la ~/Vault/Resources/videos/*SLUG*

# Check canonical telemetry for pipeline entries
joelclaw otel search "video|transcript|summarize" --hours 24

Batch Ingest

Send multiple events. Inngest queues and processes them with concurrency control:

joelclaw send pipeline/video.requested -d '{"url":"https://youtube.com/watch?v=XXXX"}'
joelclaw send pipeline/video.requested -d '{"url":"https://youtube.com/watch?v=YYYY"}'
joelclaw send pipeline/video.requested -d '{"url":"https://youtube.com/watch?v=ZZZZ"}'

Manual Transcript (Non-YouTube)

For audio files already on disk, or raw text from Granola/Fathom:

# From audio file
joelclaw send pipeline/transcript.requested -d '{"source":"manual","audioPath":"/path/to/audio.mp4","title":"Recording Title","slug":"recording-title"}'

# From raw text (Granola, Fathom, etc.)
joelclaw send pipeline/transcript.requested -d '{"source":"granola","text":"transcript text...","title":"Meeting Title","slug":"meeting-title"}'

Re-run Summary Only

If the vault note exists but needs a better summary:

joelclaw send content/summarize.requested -d '{"vaultPath":"/Users/joel/Vault/Resources/videos/SLUG.md"}'

Options

Where Things End Up

Troubleshooting

If events are accepted (200 OK) but nothing happens:

1. Check Docker→worker connectivity — the most common issue:

   docker logs $(docker ps -q --filter ancestor=inngest/inngest) 2>&1 | grep -E "ERROR|Unable" | tail -5
   

   If you see "Unable to reach SDK URL" → see the inngest skill's serveHost gotcha.

2. Check worker is actually running:

   kubectl get deploy -n joelclaw system-bus-worker
   kubectl get pods -n joelclaw -l app=system-bus-worker
   

3. Check worker errors for the specific function:

   kubectl logs -n joelclaw deploy/system-bus-worker --tail=80
   

4. Use inngest-debug skill for deep inspection of specific run IDs via GraphQL.

What NOT to Do

• ❌ Don't run yt-dlp directly — the pipeline handles download + NAS transfer
• ❌ Don't run mlx_whisper directly — the pipeline handles transcription
• ❌ Don't scp to NAS manually — the pipeline handles transfer
• ❌ Don't create vault notes manually — the pipeline creates them with proper frontmatter
• ❌ Don't use codex/background tasks for video processing — Inngest is durable and has retries