LLM Code Docs

LLM Code Docs

Overview

Centralized repository of AI-optimized documentation for 245+ frameworks, libraries, and developer tools. Three-tier structure prioritizing llms.txt standard sites.

Location: ~/github/llm-code-docs

grep -i "fastapi" ~/github/llm-code-docs/index.yaml
ls ~/github/llm-code-docs/docs/github-scraped/fastapi/
cat ~/github/llm-code-docs/docs/github-scraped/fastapi/docs/advanced/security.md

Output: FastAPI documentation available in github-scraped folder with complete API security patterns and examples.

grep -r "middleware" ~/github/llm-code-docs/docs/llms-txt/*/llms-full.txt | head -10
grep -r "middleware" ~/github/llm-code-docs/docs/github-scraped/ --include="*.md" | head -10

Output: Multiple implementations found across FastAPI, Express, Django, and other frameworks in the repository.

# Check if llms.txt exists
./scripts/find-llms-txt.sh example.com

# Add to config
echo "  - name: example-framework
    base_url: https://docs.example.com/
    description: Example framework documentation" >> scripts/llms-sites.yaml

# Download documentation
python3 scripts/llms-txt-scraper.py --site example-framework

# Update index
python3 scripts/update-index.py

Output: New framework documentation indexed and available at docs/llms-txt/example-framework/

cd ~/github/llm-code-docs
./scripts/update.sh

Output: All 228+ llms.txt sites, 14 Git repositories, and web-scraped sources refreshed with latest content.

ls ~/github/llm-code-docs/docs/llms-txt/ | grep -i playwright
head -20 ~/github/llm-code-docs/docs/llms-txt/playwright/llms-full.txt

Output: Playwright documentation found and verified to contain expected content.

First-Time Setup

Initial Configuration

# Navigate to the repository
cd ~/github/llm-code-docs

# Install Python dependencies
pip install -r requirements.txt

# Verify documentation is up-to-date
./scripts/update.sh

Verify Installation

# Check available documentation
ls docs/llms-txt/ | wc -l        # Should show ~228
ls docs/github-scraped/           # Should show ~14 repos
ls docs/web-scraped/              # Should show custom scrapers

# Verify index
cat index.yaml | head -20

Repository Structure

llm-code-docs/
├── docs/
│   ├── llms-txt/           # 228 sites (HIGHEST PRIORITY)
│   ├── github-scraped/     # 14 Git repo extractions
│   └── web-scraped/        # Custom scrapers (ntfy, claude-code-sdk, etc.)
├── scripts/                # All extraction and update tools
├── index.yaml              # Index of all documentation sources
└── AGENTS.md               # Guide for AI agents

Quick Start

Find Documentation

# Search by topic
ls ~/github/llm-code-docs/docs/llms-txt/ | grep -i react

# Check if a library exists
grep -i "fastapi" ~/github/llm-code-docs/index.yaml

# Read documentation
cat ~/github/llm-code-docs/docs/llms-txt/anthropic/llms-full.txt

Update All Documentation

cd ~/github/llm-code-docs
./scripts/update.sh

Update Specific Sources

# Update single llms.txt site
python3 scripts/llms-txt-scraper.py --site anthropic

# Force re-download (ignore cache)
python3 scripts/llms-txt-scraper.py --site vercel-ai-sdk --force

# Update Git repository extractions
python3 scripts/extract_docs.py

# Update specific web-scraped docs
python3 scripts/ntfy-docs.py
python3 scripts/claude-code-sdk-docs.py

Adding New Documentation

Priority Order

1. llms.txt - Check if site has llms.txt first (highest quality)
2. Git repos - For comprehensive docs from source
3. Web scraping - Last resort for critical docs

Add llms.txt Site

# 1. Check if llms.txt exists (probes common subdomain/path combinations)
./scripts/find-llms-txt.sh example.com
# Or manually check:
curl -sL https://docs.example.com/llms.txt | head -20

# 2. Add to config (alphabetical order)
# Edit scripts/llms-sites.yaml:
#   - name: example-site
#     base_url: https://docs.example.com/
#     description: Example documentation

# 3. Download
python3 scripts/llms-txt-scraper.py --site example-site

# 4. Verify
ls -la docs/llms-txt/example-site/

# 5. Update index
python3 scripts/update-index.py

Add Web Scraper (No llms.txt)

# 1. Create scraper script based on existing pattern
cp scripts/ntfy-docs.py scripts/newsite-docs.py

# 2. Edit script:
#    - Update BASE_URL
#    - Update DOC_PAGES list (from sitemap or manual)
#    - Update output path to docs/web-scraped/newsite/

# 3. Run scraper
python3 scripts/newsite-docs.py

# 4. Update index
python3 scripts/update-index.py

Add Git Repository

# Edit scripts/repo_config.yaml:
repositories:
  - name: example
    repo_url: https://github.com/owner/repo
    source_folder: docs/
    target_folder: docs/github-scraped/example
    branch: main

# Run extraction
python3 scripts/extract_docs.py

Key Scripts

Configuration Files

llms-sites.yaml

sites:
- name: anthropic
  base_url: https://docs.anthropic.com/
  description: Claude AI documentation
- name: vercel-ai-sdk
  base_url: https://ai-sdk.dev/
  description: AI SDK for building AI-powered apps

repo_config.yaml

repositories:
  - name: fastapi
    repo_url: https://github.com/fastapi/fastapi
    source_folder: docs/en/docs/
    target_folder: docs/github-scraped/fastapi
    branch: master

Caching System

• Cache duration: 23 hours
• Files downloaded within cache window are skipped
• Use --force to override cache
• Parallel downloads with 15 concurrent workers

Common Tasks

Check What's Available

# List all llms.txt sites
ls docs/llms-txt/ | wc -l

# List web-scraped docs
ls docs/web-scraped/

# Search for specific topic
grep -r "authentication" docs/llms-txt/*/llms-full.txt | head -20

Verify Documentation Quality

# Check file sizes (small = possibly incomplete)
ls -lh docs/llms-txt/*/llms-full.txt | sort -k5 -h | tail -10

# Check for code examples
grep -l '```' docs/llms-txt/anthropic/*.md | wc -l

# Verify source headers present
head -5 docs/web-scraped/ntfy/publish.md

Troubleshooting

Site returns 404

# Probe all common locations automatically
./scripts/find-llms-txt.sh example.com

# Or check manually
curl -sI https://docs.example.com/llms.txt
curl -sI https://example.com/llms.txt
curl -sI https://docs.example.com/llms-full.txt

Push blocked by secrets

# Find and redact test credentials
grep -n "sk_test\|AC[0-9a-f]\{32\}" docs/web-scraped/*/

# Replace with placeholders
sed -i '' 's/AC12345.../AC_EXAMPLE_SID/g' file.md

Statistics

• 228 llms.txt sites fetched
• 14 Git repositories extracted
• 3 web-scraped documentation sets
• 12,000+ markdown files
• 300MB+ total documentation

See Also

• reference.md - Complete script reference
• examples.md - Real-world usage patterns