theheavenlyd3mon Local Model Agent Evaluation
How to benchmark, evaluate, and tool up local models for agent/coding tasks. Covers BenchLoop harnesses, coding agent selection, and the "harness mismatch" problem.
The Harness Mismatch Problem
Many "this model can't tool-call" claims are actually harness mismatches — wrong tool format for the model family. BenchLoop has observed +15 point overall score swings just from switching harnesses.
BenchLoop Harnesses
BenchLoop (pipx install benchloop-cli) is the standard local-First benchmark suite. Seven fixed task suites (speed, toolcall, coding, dataextract, instructfollow, reasonmath, agent). Four harnesses control how tools are presented and parsed:
| Harness | Format | Best For |
|---|---|---|
raw |
OpenAI tools=[...] param |
Frontier models, generic endpoints |
hermes |
<tools> in system prompt, <tool_call>{...}</tool_call> output |
Nous models, Hermes fleet |
qwen |
<function_call>{...}</function_call> XML tags |
Qwen3-Coder, Qwen-Agent |
pi |
<think>...</think> + Hermes tags (reasoning stripped before scoring) |
Reasoning models (R1-style) |
Quick A/B Test
benchloop run --model qwen3:8b --harness raw --suites toolcall
benchloop run --model qwen3:8b --harness qwen --suites toolcall
benchloop run --model qwen3:8b --harness hermes --suites toolcall
Run all four on a new model to find its best format. Results at ~/.bench-loop/runs/.
Web Dashboard
benchloop dashboard # → http://127.0.0.1:8877
Auto-discovers models on localhost:11434 (Ollama), :1234 (LM Studio), :8000 (MLX/vLLM). Also supports cloud models via --remote.
Coding Agent Selection
Swival (swival.dev)
Frank Denis (jedisct1 / libsodium). Purpose-built for local/small models. MIT, no telemetry.
Why it's different: Assumes the model WILL fail tool calls. Engineers around it:
- 3-pass forgiving JSON parser — exact → line-trimmed → unicode-normalized
- Graduated 4-level context compaction — shrink → drop low-value → nuclear → shed tools
- Durable state —
think/todo/snapshotsurvive context resets - Loop protection — warns after 2 repeats, stops after 3
- Bounded output — 50KB reads, 100 grep matches, 10KB commands
Install: brew install swival/tap/swival (macOS) or uv tool install swival.
OpenCode
The dominant open-source option (~185K stars). More "plug and play," larger community, more integrations. Better default choice if the model handles tool calling well natively.
Qwen-Code
Qwen-specific CLI. Best if exclusively using Qwen models. Less general.
Claude Code / Codex
Frontier-model agents. Not designed for local models. Users report Claude Code is "heavily subsidized" but quality-regressed at times. Codex stronger for large codebases and following plans.
Harness-to-Model Mapping (Community-Tested)
| Model | Best Harness | Notes |
|---|---|---|
| Qwen3-Coder / Qwen3.6 | qwen or pi |
Native <function_call> format; pi for reasoning variants |
| Hermes models | hermes |
Native format |
| Generic OpenAI endpoints | raw |
Safe default, may underperform |
| Reasoning models (R1-style) | pi |
Strips think blocks from scoring |
Pitfalls
- Don't assume
rawis best — it's the default but rarely optimal for local models - BenchLoop requires model already pulled — it doesn't pull models, just tests endpoints
- Swival: very new (208 stars, 1 primary dev) — active development but bus factor of 1. OpenCode is the safer community choice.
- Provider edge cases — Swival had issues with Gemini/DeepSeek (now resolved in latest releases). Test with your specific provider.
- 12GB VRAM constraint — Qwen3-Coder-Next GGUF (Q4) runs on 12GB. Qwen3.6-35B-A3B at Q4 also fits.
References
references/swival-community-sentiment.md— Community reviews, HN discussions, user experiences- BenchLoop docs: https://bench-loop.com/docs
- Swival docs: https://swival.dev/pages/