How do I install the Model Benchmarks skill?

Run npx skillmds@latest add theopenco/model-benchmarks in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Model Benchmarks safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Model Benchmarks?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Model Benchmarks free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Model Benchmarks?

theopenco (@theopenco) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Model Benchmarks by theopenco — Agent Skill

Name: Model Benchmarks
Author: theopenco

Model Benchmarks

Run and report repository model or provider-mapping benchmarks. Use when asked to benchmark a model, compare all mappings or regions, collect latency or throughput metrics, run the smoke, standard, load, capability, quality, or performance suites, or inspect an existing benchmark JSON result.

theopenco 058ec4a 3 files · 10.1 KB Updated Sep 17, 2026

File contents

Model benchmarks

Run paid benchmarks only when the user explicitly asks. Use the wrapper so raw results survive and failed evaluations still retain their timing data:

node .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> [benchmark options]

The wrapper builds @llmgateway/benchmarks, creates an absolute output path, and writes results under .context/benchmarks. It defaults to every active mapping, the smoke profile, and a 120-second per-target budget. Override those defaults with ordinary CLI options, for example:

node .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> \
  --mapping <provider> --profile load --budget 300000

Do not pass --output or --format; the wrapper owns them. Set BENCHMARK_OUTPUT_DIR when results belong elsewhere. The benchmark CLI reads LLM_GATEWAY_API_KEY, falls back to LLMGATEWAY_API_KEY, and loads the root .env file.

A budget is a maximum per target, not total run time. A request already in flight may finish after its target reaches the budget. Keep cache and fallback disabled unless the user explicitly asks otherwise, so results stay pinned to the selected mapping.

The wrapper preserves:

.json: raw trials, responses, usage, cost, and stream events.
.md and .html: built-in benchmark reports.
-timings.md: timings grouped by target and case, including evaluator failures.
-timings.csv: every trial's timing and usage fields.

If rendering fails after JSON is saved, render it without rerunning paid requests:

node .agents/skills/model-benchmarks/scripts/render-results.mjs <absolute-json-path>

If a run makes requests but saves no JSON, report the loss and ask before rerunning. After completion, link the reports and summarize quality, reliability, TTFT, total latency, throughput, buffering, truncation, cost, and provider errors.

---name: model-benchmarksdescription: Run and report repository model or provider-mapping benchmarks. Use when asked to benchmark a model, compare all mappings or regions, collect latency or throughput metrics, run the smoke, standard, load, capability, quality, or performance suites, or inspect an existing benchmark JSON result.---# Model benchmarksRun paid benchmarks only when the user explicitly asks. Use the wrapper so rawresults survive and failed evaluations still retain their timing data:```bashnode .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> [benchmark options]```The wrapper builds `@llmgateway/benchmarks`, creates an absolute output path,and writes results under `.context/benchmarks`. It defaults to every activemapping, the `smoke` profile, and a 120-second per-target budget. Override thosedefaults with ordinary CLI options, for example:```bashnode .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> \ --mapping <provider> --profile load --budget 300000```Do not pass `--output` or `--format`; the wrapper owns them. Set`BENCHMARK_OUTPUT_DIR` when results belong elsewhere. The benchmark CLI reads`LLM_GATEWAY_API_KEY`, falls back to `LLMGATEWAY_API_KEY`, and loads the root`.env` file.A budget is a maximum per target, not total run time. A request already inflight may finish after its target reaches the budget. Keep cache and fallbackdisabled unless the user explicitly asks otherwise, so results stay pinned tothe selected mapping.The wrapper preserves:- `.json`: raw trials, responses, usage, cost, and stream events.- `.md` and `.html`: built-in benchmark reports.- `-timings.md`: timings grouped by target and case, including evaluator failures.- `-timings.csv`: every trial's timing and usage fields.If rendering fails after JSON is saved, render it without rerunning paidrequests:```bashnode .agents/skills/model-benchmarks/scripts/render-results.mjs <absolute-json-path>```If a run makes requests but saves no JSON, report the loss and ask beforererunning. After completion, link the reports and summarize quality,reliability, TTFT, total latency, throughput, buffering, truncation, cost, andprovider errors.

Frequently asked questions