LLM

/llm — Universal LLM Router

Route prompts to any LLM model across all providers. CLI-first for Codex/Kimi/Claude (zero cost), Google API for Gemini, OpenRouter for everything else. Auto-discovers new models.

When to Use

• You need to call a specific LLM model (not the current Claude Code session)
• You want to compare outputs across different models
• A task benefits from a specific model's strengths (e.g., GPT-5.3-Codex for code generation, Gemini for grounded search)
• You need to route through a specific provider (e.g., force OpenRouter for a model that usually goes through CLI)

When NOT to Use

• The current Claude Code session can handle the task directly (don't call yourself via CLI)
• You need Aristotle formal verification — use /prove instead
• You need multi-model debate — use /debate instead
• You need parallel Codex swarm — use /CodexCode instead

Quick Start

# Call a model
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Say hello in 3 words"

# With system prompt
python3 ~/.claude/skills/llm/scripts/llm_route.py --model chatgpt-5.4 --prompt "Write fizzbuzz" --system "You are a Python expert"

# From files
python3 ~/.claude/skills/llm/scripts/llm_route.py --model gemini-3.1-pro --prompt-file prompt.txt --system-file system.txt

# Force a different provider
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" --route openrouter

# JSON output with metadata
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" --json

# Custom parameters
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" --temperature 0.3 --max-tokens 8192 --timeout 300

# Pipe prompt via stdin
echo "Explain quantum computing" | python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus

# List models
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models --all
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models --tier 2

# List providers
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-providers

Routing Hierarchy

Models are routed to providers in this priority order:

Use --route <provider> to override the default route for any model. For example, --route openrouter forces a CLI model through OpenRouter instead.

Important: CLI tools must be installed on the machine running Claude Code. The CLI-first routing for GPT models (Codex CLI) and Kimi models (Kimi CLI) requires those CLIs to be installed and authenticated locally. If you're running Claude Code on a remote server, CI runner, or any machine without these CLIs, those routes will silently fail. Use --route openrouter to force API-based routing instead, or install the CLIs:

npm install -g @openai/codex && codex login     # GPT models
npm install -g kimi-cli && kimi login            # Kimi models

Claude CLI is always available since you're already running inside Claude Code.

Model Tiers

Tier 1 Models (Curated)

Auto-Discovery

Discover new models from OpenRouter and Google APIs:

# Dry run — show what's new
python3 ~/.claude/skills/llm/scripts/discover_models.py

# Apply — update the registry
python3 ~/.claude/skills/llm/scripts/discover_models.py --apply

# Query only one source
python3 ~/.claude/skills/llm/scripts/discover_models.py --source openrouter
python3 ~/.claude/skills/llm/scripts/discover_models.py --source google

Discovery never overwrites tier 1 models. New models from major providers with context >= 32k become tier 2; everything else is tier 3.

Prompting Overrides

Per-model temperature and system prompt wrapping is applied automatically from settings/prompting-overrides.json:

• Claude (opus): XML format preferred, no temperature override
• GPT (gpt-5.2/chatgpt-5.4): CTCO framework preamble, temperature not overridden (reasoning model)
• Gemini (gemini-3.1-pro/gemini-3-flash): Forced temperature=1.0, concise preamble
• Kimi (kimi-2.5): temperature=1.0
• MiniMax (minimax-m2.5): temperature=1.0

Credential Resolution

API keys are resolved in this order:

1. Environment variable (e.g., GOOGLE_API_KEY)
2. Debate agent's shared keys at ~/.claude/skills/convolutional-debate-agent/api-keys/provider-keys.env
3. Error with setup instructions

CLI tools (claude, codex, kimi) use their own stored logins — no API keys needed.

File Structure

~/.claude/skills/llm/
├── SKILL.md                              # This file
├── scripts/
│   ├── llm_route.py                      # Core router (provider calls + CLI)
│   ├── discover_models.py                # Auto-discovery from OpenRouter/Google
│   ├── fetch_benchmarks.py              # Fetch benchmarks from public leaderboards
│   ├── scrape_vals.py                  # Headless scraper for vals.ai benchmarks
│   └── .venv/                          # Python venv for Playwright (scrape_vals.py)
├── settings/
│   ├── model-registry.json               # All known models + routes + tiers
│   ├── routing-rules.json                # Regex patterns for auto-routing new models
│   ├── prompting-overrides.json          # Per-model temperature + system preambles
│   └── benchmark-quality.json            # BetterBench quality metadata per benchmark
├── benchmarks/
│   ├── rankings.csv                      # Unified rankings — THE file other skills read
│   ├── vals.json                         # Scraped vals.ai data (all 23 benchmarks)
│   └── _meta.json                        # Fetch timestamps and source status
└── references/
    ├── provider-setup.md                 # Auth setup per provider
    └── betterbench-notes.md              # BetterBench paper findings + methodology

Benchmarks

Fetch and cache LLM benchmark rankings from public leaderboards. The unified CSV at benchmarks/rankings.csv is the canonical source other skills should read for model comparisons.

Data Sources

Evaluated but skipped: LiveBench (already in Epoch AI data), LM Council (aggregator of our sources), LLM Stats/ZeroEval (aggregator of our sources).

Quality methodology: Informed by BetterBench (NeurIPS 2024 Spotlight). Each model row gets a benchmark_quality tier (high/medium/low) based on which high-quality data sources contributed scores. See references/betterbench-notes.md for details.

Fetch Commands

# Fetch all sources, update rankings.csv
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py

# Fetch only one source
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source arena
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source epoch
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source openrouter
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source aa
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source vals

# Scrape vals.ai (requires Playwright venv, ~90s for all 23 benchmarks)
~/.claude/skills/llm/scripts/.venv/bin/python3 ~/.claude/skills/llm/scripts/scrape_vals.py

# View local rankings (no network calls)
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --list
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --list --top 50

# Look up a specific model
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --model opus
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --model gemini-3.1-pro --json

CSV Schema (benchmarks/rankings.csv)

Usage from Other Skills

Other skills can read the local CSV for model selection decisions:

import csv
from pathlib import Path

rankings_path = Path.home() / ".claude/skills/llm/benchmarks/rankings.csv"
with open(rankings_path) as f:
    for row in csv.DictReader(f):
        if row["registry_name"] == "opus":
            print(f"Opus GPQA: {row['gpqa']}%, Arena Elo: {row['arena_elo']}")

Or via CLI for quick lookups:

# Get opus benchmarks as JSON
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --model opus --json

Error Handling

Usage from Claude Code (Programmatic)

When invoked as /llm from Claude Code, the skill works as a reference for how to call models. Claude Code should:

1. Read this SKILL.md for routing decisions
2. Call llm_route.py via Bash tool for external model calls
3. Use --json flag when parsing the response programmatically
4. Use --route to force a specific provider when needed

Example from Claude Code:

# Get a response from GPT-5.3 and parse it
response=$(python3 ~/.claude/skills/llm/scripts/llm_route.py --model chatgpt-5.4 --prompt "Your prompt" --json)

Setup

1. Install CLIs (for zero-cost routing):

   npm install -g @anthropic-ai/claude-code   # claude login
   npm install -g @openai/codex               # codex login
   npm install -g kimi-cli                     # kimi login
   

2. Set API keys (for Google/OpenRouter):

   export GOOGLE_API_KEY="your-key"
   export OPENROUTER_API_KEY="your-key"
   

3. Verify:

   python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models
   python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello"
   

See references/provider-setup.md for detailed per-provider instructions.