# Benchmark

> Benchmark a local LLM model with MetriLLM. Measures performance (tok/s, TTFT, memory) and quality (reasoning, math, coding, instruction following, structured output, multilingual). Use when the user wants to test, compare, or evaluate a local model.

- Skill: `metrillm/benchmark` (Agent Skill)
- Install (CLI): `npx skillmds@latest add metrillm/benchmark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/metrillm/benchmark/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: MetriLLM (https://skillmd.com/u/metrillm)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/metrillm/benchmark

---


# Benchmark a Local LLM Model

Run an MetriLLM benchmark on a locally available model. This skill handles setup verification, model selection, benchmark execution, and result interpretation.

## Prerequisites Check

Before running any benchmark, verify the environment:

1. **Runtime reachable**: Run `metrillm list` to check the active backend. If needed, use `metrillm list --backend <name>` for a specific runtime and tell the user to start that runtime first.
2. **MetriLLM available**: If the MetriLLM MCP server is connected, use the MCP tools directly. Otherwise, check if `metrillm` works.

## Workflow

### Step 1 — Select the model

If `$ARGUMENTS` is provided, use it as the model name. Otherwise:

- **With MCP**: Use the `list_models` tool to list available models.
- **Without MCP**: Run `metrillm list` (or `metrillm list --backend <name>`).

Help the user pick a model if they're unsure. Smaller models (1-3B) benchmark faster (~30s). Larger models (7B+) take 2-5 minutes.

### Step 2 — Run the benchmark

**With MCP** (preferred):
Use the `run_benchmark` tool with the model name. Set `perfOnly: true` for performance-only (faster).

**Without MCP**:
```bash
metrillm bench --model <model-name> --json
```

For performance-only (skip quality tests, much faster):
```bash
metrillm bench --model <model-name> --perf-only --json
```

### Step 3 — Interpret results

The benchmark produces a JSON result with:

| Metric | What it means |
|---|---|
| `performance.tokensPerSecond` | Generation speed. >30 tok/s = good for interactive use |
| `performance.ttft` | Time to first token in ms. <500ms = responsive |
| `performance.memoryUsedGB` | RAM/VRAM consumed during inference |
| `fitness.verdict` | EXCELLENT / GOOD / MARGINAL / NOT RECOMMENDED |
| `fitness.globalScore` | 0-100 composite score (30% perf + 70% quality) |
| `fitness.interpretation` | Human-readable summary of the verdict |

Present the results in a clear, concise format. Highlight the verdict prominently.

### Step 4 — Compare (optional)

If the user wants to compare models:

- **With MCP**: Use the `get_results` tool to retrieve previous results.
- **Without MCP**: Read JSON files from `~/.metrillm/results/`.

Compare side-by-side on key metrics.

### Step 5 — Share (optional)

If the user wants to share results to the public leaderboard:

- **With MCP**: Use the `share_result` tool with the result file path.
- **Without MCP**: `metrillm bench --model <model-name> --share`

Uses official upload defaults by default. For self-hosted/staging deployments, set `METRILLM_SUPABASE_URL` and `METRILLM_SUPABASE_ANON_KEY`.

## Tips

- Use `--perf-only` / `perfOnly: true` for quick tests when quality scoring isn't needed
- Qwen3 and other "thinking" models generate many tokens and take significantly longer
- Close other GPU-intensive applications before benchmarking for accurate results
- Run the same model twice to verify consistency — first run may be slower due to model loading