Kp Source

Instructions

When to activate

Activate when the user invokes /kp-source add, /kp-source status, or /kp-source check-updates, or asks to add a document or source file to the project.

Sub-commands

add <path-or-url>

If <path> is a directory:

1. Run <skill-dir>/scripts/ingest.py --list-dir <path> to discover files. The script walks recursively (skipping hidden files and dirs), hashes each file, and checks against existing sources for duplicates. All file types are included - no extension filtering.
2. Count files where duplicate_of is null (new files to ingest).
3. If that count exceeds the threshold (default 50, override with --threshold N):

   Found N files. This will create N sources. Continue? [y/N]
   

   If the user answers N (or gives no answer), abort with zero sources created. Pass --yes to skip this prompt for scripted use (--threshold 0 also bypasses it).
4. For each non-duplicate file, run the single-file ingestion steps below (derive slug, create directory, copy, run ingest.py, handle --sensitive).
5. After all files are processed, print a summary:

   Added:      N
   Duplicate:  N  (already ingested as <source-id>)
   Failed:     N
   

If <path> is a single file or a URL:

1. Assign a source-id slug:
   • Derive a short kebab-case slug from the filename or URL (e.g. cairn-annual-report, privacy-policy-2024).
   • For YouTube URLs, derive the slug from the video title if available, or fall back to youtube-<video-id> (e.g. youtube-dQw4w9WgXcQ).
   • Keep it concise: 2-4 meaningful words, lowercase, hyphens only.
   • Check uniqueness against existing directory names in sources/.
   • If the slug already exists, append -2, -3, etc. until unique.
2. Create sources/<source-id>/.
3. Fetch content into sources/<source-id>/:
   • YouTube URL (youtube.com/watch?v= or youtu.be/): run uv run <skill-dir>/scripts/fetch_youtube.py <url> sources/<source-id>/ (add --language <code> or --language <code1,code2> if the user requested a specific subtitle language or priority order, e.g. --language fr or --language en,fr). This writes transcript.txt (timestamped plain text, one line per caption) and attempts to write <title>.info.json via yt-dlp (best-effort). The primary content file for extraction is transcript.txt. Without --language, the first available transcript is used (manually-created tracks are preferred over auto-generated ones), so non-English videos work with no extra flags. If none of the requested languages are available, it falls back to any available transcript. The script's JSON stdout includes language and is_generated for the transcript it selected - capture these to pass to ingest.py in the next step.
   • Other URL: download with curl -L -o <filename>.
   • Local file: copy into the directory.
4. Run <skill-dir>/scripts/ingest.py --source-id <source-id> --origin <path-or-url> (for YouTube sources, add --language <code> --is-generated <true|false> using the values captured from fetch_youtube.py's JSON stdout):
   • Computes SHA-256 hash.
   • Detects type (pdf, csv, url, db-dump, markdown, image, other).
   • Reads page count for PDFs.
   • Writes sources/<source-id>/.meta.json, including language / is_generated when provided.
5. If --sensitive: append sources/<source-id>/ to .gitignore.
6. Print the assigned source-id and confirm.

.meta.json schema:

{
  "source_id": "generated-source-slug",
  "origin": "<path-or-url>",
  "type": "pdf",
  "ingested_at": "<ISO datetime>",
  "hash": "sha256:<hex>",
  "page_count": 42,
  "sensitive": false,
  "extraction": {"status": "pending"},
  "stale": false,
  "language": "fr",
  "is_generated": false
}

language and is_generated are only present for sources where the content language was explicitly selected (currently YouTube transcripts).

status

For each directory in sources/, read .meta.json and print a table:

check-updates

For each source in sources/:

1. Run <skill-dir>/scripts/ingest.py --check-update --source-id <source-id>.
2. The script recomputes the hash (or re-fetches for URLs) and compares to the stored hash.
3. If changed: set stale: true in .meta.json and print a warning.

Flags

Edge cases

• URL download fails: report the error, do not create a partial sources/<source-id>/.
• YouTube transcript unavailable (private video, no captions at all, requested --language not found and no other transcript exists): fetch_youtube.py exits non-zero; report the error and do not create a partial source directory.
• File not found: report clearly, suggest checking the path.
• Project not initialized (.knowledge-project missing): prompt the user to run /kp-init first.
• Empty directory: report zero files found.
• Duplicate file in directory input: log it in the summary (Duplicate: N) with the existing source-id; do not re-ingest.