How do I install the Benchmark skill?

Run npx skillmds@latest add metrillm/benchmark in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Benchmark safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Benchmark?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Benchmark free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Benchmark?

MetriLLM (@metrillm) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Benchmark by metrillm — Agent Skill

Name: Benchmark
Rating: 4.2 (5 reviews)
Author: MetriLLM

Benchmark

Benchmark a local LLM model with MetriLLM. Measures performance (tok/s, TTFT, memory) and quality (reasoning, math, coding, instruction following, structured output, multilingual). Use when the user wants to test, compare, or evaluate a local model.

MetriLLM Updated Sep 10, 2026

File contents

Benchmark a Local LLM Model

Run an MetriLLM benchmark on a locally available model. This skill handles setup verification, model selection, benchmark execution, and result interpretation.

Prerequisites Check

Before running any benchmark, verify the environment:

Runtime reachable: Run metrillm list to check the active backend. If needed, use metrillm list --backend <name> for a specific runtime and tell the user to start that runtime first.
MetriLLM available: If the MetriLLM MCP server is connected, use the MCP tools directly. Otherwise, check if metrillm works.

Workflow

Step 1 — Select the model

If $ARGUMENTS is provided, use it as the model name. Otherwise:

With MCP: Use the list_models tool to list available models.
Without MCP: Run metrillm list (or metrillm list --backend <name>).

Help the user pick a model if they're unsure. Smaller models (1-3B) benchmark faster (~30s). Larger models (7B+) take 2-5 minutes.

Step 2 — Run the benchmark

With MCP (preferred): Use the run_benchmark tool with the model name. Set perfOnly: true for performance-only (faster).

Without MCP:

metrillm bench --model <model-name> --json

For performance-only (skip quality tests, much faster):

metrillm bench --model <model-name> --perf-only --json

Step 3 — Interpret results

The benchmark produces a JSON result with:

Metric	What it means
`performance.tokensPerSecond`	Generation speed. >30 tok/s = good for interactive use
`performance.ttft`	Time to first token in ms. <500ms = responsive
`performance.memoryUsedGB`	RAM/VRAM consumed during inference
`fitness.verdict`	EXCELLENT / GOOD / MARGINAL / NOT RECOMMENDED
`fitness.globalScore`	0-100 composite score (30% perf + 70% quality)
`fitness.interpretation`	Human-readable summary of the verdict

Present the results in a clear, concise format. Highlight the verdict prominently.

Step 4 — Compare (optional)

If the user wants to compare models:

With MCP: Use the get_results tool to retrieve previous results.
Without MCP: Read JSON files from ~/.metrillm/results/.

Compare side-by-side on key metrics.

Step 5 — Share (optional)

If the user wants to share results to the public leaderboard:

With MCP: Use the share_result tool with the result file path.
Without MCP: metrillm bench --model <model-name> --share

Uses official upload defaults by default. For self-hosted/staging deployments, set METRILLM_SUPABASE_URL and METRILLM_SUPABASE_ANON_KEY.

Tips

Use --perf-only / perfOnly: true for quick tests when quality scoring isn't needed
Qwen3 and other "thinking" models generate many tokens and take significantly longer
Close other GPU-intensive applications before benchmarking for accurate results
Run the same model twice to verify consistency — first run may be slower due to model loading

---name: benchmarkdescription: Benchmark a local LLM model with MetriLLM. Measures performance (tok/s, TTFT, memory) and quality (reasoning, math, coding, instruction following, structured output, multilingual). Use when the user wants to test, compare, or evaluate a local model.---# Benchmark a Local LLM ModelRun an MetriLLM benchmark on a locally available model. This skill handles setup verification, model selection, benchmark execution, and result interpretation.## Prerequisites CheckBefore running any benchmark, verify the environment:1. **Runtime reachable**: Run `metrillm list` to check the active backend. If needed, use `metrillm list --backend <name>` for a specific runtime and tell the user to start that runtime first.2. **MetriLLM available**: If the MetriLLM MCP server is connected, use the MCP tools directly. Otherwise, check if `metrillm` works.## Workflow### Step 1 — Select the modelIf `$ARGUMENTS` is provided, use it as the model name. Otherwise:- **With MCP**: Use the `list_models` tool to list available models.- **Without MCP**: Run `metrillm list` (or `metrillm list --backend <name>`).Help the user pick a model if they're unsure. Smaller models (1-3B) benchmark faster (~30s). Larger models (7B+) take 2-5 minutes.### Step 2 — Run the benchmark**With MCP** (preferred):Use the `run_benchmark` tool with the model name. Set `perfOnly: true` for performance-only (faster).**Without MCP**:```bashmetrillm bench --model <model-name> --json```For performance-only (skip quality tests, much faster):```bashmetrillm bench --model <model-name> --perf-only --json```### Step 3 — Interpret resultsThe benchmark produces a JSON result with:| Metric | What it means ||---|---|| `performance.tokensPerSecond` | Generation speed. >30 tok/s = good for interactive use || `performance.ttft` | Time to first token in ms. <500ms = responsive || `performance.memoryUsedGB` | RAM/VRAM consumed during inference || `fitness.verdict` | EXCELLENT / GOOD / MARGINAL / NOT RECOMMENDED || `fitness.globalScore` | 0-100 composite score (30% perf + 70% quality) || `fitness.interpretation` | Human-readable summary of the verdict |Present the results in a clear, concise format. Highlight the verdict prominently.### Step 4 — Compare (optional)If the user wants to compare models:- **With MCP**: Use the `get_results` tool to retrieve previous results.- **Without MCP**: Read JSON files from `~/.metrillm/results/`.Compare side-by-side on key metrics.### Step 5 — Share (optional)If the user wants to share results to the public leaderboard:- **With MCP**: Use the `share_result` tool with the result file path.- **Without MCP**: `metrillm bench --model <model-name> --share`Uses official upload defaults by default. For self-hosted/staging deployments, set `METRILLM_SUPABASE_URL` and `METRILLM_SUPABASE_ANON_KEY`.## Tips- Use `--perf-only` / `perfOnly: true` for quick tests when quality scoring isn't needed- Qwen3 and other "thinking" models generate many tokens and take significantly longer- Close other GPU-intensive applications before benchmarking for accurate results- Run the same model twice to verify consistency — first run may be slower due to model loading

Frequently asked questions