Local LLM Claude Code

Local LLM → Claude Code

Connect Claude Code to a locally-served open model. Since Ollama v0.14+ natively speaks the Anthropic Messages API (/v1/messages), this needs no proxy — just three env vars.

The one thing to say up front (set expectations honestly)

Making this work and making it usable are different. Two independent thresholds must BOTH be cleared, and cheap hardware usually misses both:

1. Capability — the model must emit valid structured tool calls, turn after turn. Small models (≤1.7B) malform them (args object vs string, skipped tools). Fixed by a bigger model (≥14B, ideally a 30B-class MoE), NOT by faster hardware.
2. Throughput — the box must chew Claude Code's ~18,000-token system prompt every turn fast enough to beat its request timeout. CPU-only = minutes/turn. Fixed by a GPU, NOT by a smarter model.

Say this to the user before installing, then read references/hardware-sizing.md and run scripts/check-hardware.sh to give them a specific, honest verdict for their box.

Workflow

Run these in order. Each script is idempotent and needs no root (user-space install).

1. Reality-check the hardware

bash scripts/check-hardware.sh

Reports CPU/RAM/GPU/disk and recommends a model tier (or warns it'll only be a learning rig). Details + the sizing table: references/hardware-sizing.md.

2. Install Ollama (lean, user-space)

bash scripts/install-ollama.sh

Streams the release tarball through zstd and excludes the CUDA/ROCm/MLX GPU libraries, so a CPU box installs in ~112 MB instead of ~3 GB. Starts ollama serve on :11434. (On a GPU box, drop the --exclude flags — see the script's header.)

3. Pull + tune a model

bash scripts/make-model.sh <base-model> <derived-name>
# e.g. bash scripts/make-model.sh qwen3:14b qwen3-cc

Pulls the base model and builds a derived model with a Modelfile that sets num_ctx large enough for Claude Code's ~18K prompt (default 32768). This is the #1 cause of "local Claude Code is broken" — the Ollama default of 4096 silently truncates the prompt.

4. Launch Claude Code against it

bash scripts/launch.sh <derived-name>     # e.g. qwen3-cc

Exports the three env vars and runs claude --model <name>. The wiring:

export ANTHROPIC_BASE_URL="http://localhost:11434"
export ANTHROPIC_AUTH_TOKEN="ollama"   # required but ignored locally
export ANTHROPIC_API_KEY=""            # MUST be empty so the token is used

On slow (CPU) boxes the launcher also sets API_TIMEOUT_MS=900000 so turns don't 500 mid-prefill.

5. Verify end-to-end

bash scripts/verify.sh <derived-name>

Tests, in order: native /v1/messages chat → a real tool call → count_tokens (a 404 here is FINE on v0.14+, not a hang) → measures prompt-eval tok/s so you can predict per-turn latency.

Hard-won gotchas (all verified the hard way)

• No proxy needed. Ollama v0.14+ serves the Anthropic API natively. Do NOT install LiteLLM or claude-code-router for Ollama. (vLLM/LM Studio DO need a proxy — see references/other-backends.md.)
• num_ctx ≥ 32768. Claude Code's system prompt is ~18K tokens. The 4096 default breaks it.
• Reasoning models (Qwen3, DeepSeek-R1) think by default and it's slow. You CANNOT disable thinking through the Anthropic /v1/messages path (no think field passes; there's no PARAMETER think in Modelfiles; template edits don't change server-side parse). For Claude Code, either accept the overhead or choose a non-reasoning model. For YOUR OWN agents, call Ollama's native /api/chat with "think": false — that DOES disable it (big speed win).
• count_tokens returns 404 on v0.14+ — harmless; Claude Code falls back to local estimation. A hang (not a 404) means your Ollama is too old — upgrade.
• Prompt cache helps a lot. Turn 2+ reuses the cached ~18K system-prompt prefix, so only the first turn pays full prefill. Any context growth past the cached prefix re-triggers eval.
• Install lean or run out of disk. The full tarball unpacks to ~3 GB (GPU libs); stripping them → ~112 MB. Critical on small VPSs.

When the user's box can't do it

Be direct: it's a great learning rig (the wiring, API, and agent-loop mechanics all work), but not a usable coding agent. Point them to a 14B–30B model on a GPU (even a modest rented one) — same Ollama, same three env vars, the setup transfers 1:1. See references/hardware-sizing.md.

References

• references/hardware-sizing.md — model-by-RAM/VRAM table, the two-threshold model, GPU guidance
• references/troubleshooting.md — every failure mode seen and its fix (timeouts, 404s, truncation, malformed tool calls)
• references/other-backends.md — vLLM / LM Studio / llama.cpp (need a proxy) and how to wire them