LLM Backends

llm_backends — Local LLM discovery, audit & routing

Turn idle local hardware into a token-saving tier. Every task served by a local model is a task not billed to the cloud.

When to use

• The user mentions LM Studio, Ollama, LocalAI, vLLM, llama.cpp, "local model", "run it locally", or "use my GPU".
• You need to know what models are reachable (this machine or the network).
• A task is cheap/local-suitable (classification, extraction, short summary, routing, spell-check, simple Q&A) and could skip the cloud entirely.
• The user has no local model yet and wants step-by-step, hardware-aware setup.

Quick commands

# Discover + register backends (writes configs/llm-endpoints.json)
python -m skills.llm_backends.cli scan                 # localhost only
python -m skills.llm_backends.cli scan --subnet        # sweep local /24
python -m skills.llm_backends.cli scan 192.168.1.47    # specific host(s)

# What's registered?
python -m skills.llm_backends.cli list

# Audit: are local models used? what can this machine run? next steps?
python -m skills.llm_backends.cli audit --fresh

# Run a prompt locally (0 cloud tokens)
python -m skills.llm_backends.cli chat "classify: bug or feature?" --max-tokens 128

# Suggest a local model for a project (adaptive per project type)
python -m skills.llm_backends.cli profile ~/my-project

Programmatic use

from skills.llm_backends import registry, quick_chat, audit

registry.refresh()                       # discover + persist
best = registry.best_chat_backend()      # lowest-latency chat backend
model = registry.preferred_model(best)   # coder/instruct over voice/reasoning

res = quick_chat("summarize in 1 line: ...", max_tokens=200)
print(res.text, res.total_tokens)        # all local — no cloud cost

Supported backends

How it saves tokens

1. Audit finds reachable local backends and profiles RAM/VRAM/GPU.
2. Route sends local-suitable tasks (see tiered_router L0/L1) to a local model.
3. Call runs them via the OpenAI-compatible client — zero cloud tokens.
4. Onboard users with no local model, recommending the largest model their hardware can run and the right server to install.

Related: [[llm_mcp]] (MCP tools for agents), tiered_router (cost tiers), local_router (task→backend mapping), response_cache (skip repeated calls).