Benchmark Assistant

Benchmark Assistant

Generated from config/activation-policy.yaml. Do not edit this block.

Activation mode: auto. This Skill is eligible under its authored domain boundaries. Cross-Skill routing remains owned by thinking-router.

Purpose

benchmark-assistant helps the user actually use the benchmark runner.

It is a meta skill. It does not replace scripts/run-benchmark.js, and it does not define domain quality by itself. It chooses useful benchmark commands, runs or prepares them, explains results, and hands failure diagnosis to skill-evaluator when needed.

When to Use

Use this skill when the user says things like:

• "Run benchmark."
• "Test content-creator."
• "Check learning-coach regression."
• "Generate benchmark prompts."
• "Score these benchmark responses."
• "Update benchmark dashboard."
• "Compare the latest benchmark runs."
• "Add this failure to benchmark."
• "What failed in benchmark?"
• "I changed a skill, help me test it."
• "帮我测一下这个 skill."
• "跑一下 benchmark."

When Not to Use

• Use skill-evaluator directly when the user already has a specific failed response and wants a failure diagnosis.
• Use the relevant domain skill when the user wants the substantive task answered, not benchmarked.
• Do not create a new benchmark framework when the existing runner is enough.

Required Context

Read these files when needed:

• docs/benchmark.md
• docs/benchmark-dashboard.md when comparing runs
• scripts/run-benchmark.js
• scripts/update-benchmark-dashboard.js
• benchmark-runs/ when the user asks for historical comparison
• Relevant benchmarks/<skill>/ cases
• Relevant evals/<skill>-cases.md
• skills/skill-evaluator/SKILL.md when failures need diagnosis

Core Workflow

1. Identify the target:
   • All benchmarks
   • One skill directory, such as benchmarks/content-creator
   • A specific failure or proposed new case
2. Choose the lightest useful command.
3. Run the command if it does not require external credentials or unknown agent setup.
4. If no external agent command is configured, generate prompts or ask for saved responses instead of pretending the benchmark was run.
5. Do not write a not_run report to benchmark-runs/ or update the dashboard from it unless the user explicitly asks for a coverage-only record.
6. If the user wants trend comparison, update docs/benchmark-dashboard.md from benchmark-runs/.
7. Summarize results in plain language.
8. If failures appear, classify whether the likely source is:
   • Router
   • Domain skill
   • Eval or benchmark wording
   • Missing response data
   • External agent setup
9. Recommend the smallest next action.

Common Commands

List all cases:

node scripts/run-benchmark.js --list

List cases for one skill:

node scripts/run-benchmark.js --cases benchmarks/content-creator --list

Generate prompts:

node scripts/run-benchmark.js --prompts

Generate prompts for one skill:

node scripts/run-benchmark.js --cases benchmarks/content-creator --prompts

Score saved responses:

node scripts/run-benchmark.js --responses benchmark-responses.json --out benchmark-runs/my-run.json

Run an external agent command:

node scripts/run-benchmark.js --command "your-agent-command"

Update dashboard:

node scripts/update-benchmark-dashboard.js

Output Format

For normal benchmark runs, respond with:

## Benchmark Result

Command:

Summary:

Key failures:

Next action:

For prompt generation, respond with:

## Benchmark Prompts

Command:

Cases:

How to use:

For failures, respond with:

## Failure Triage

Failed case:

Likely source:

Existing coverage:

Smallest useful patch:

Should this become a failure case?

For dashboard updates, respond with:

## Benchmark Dashboard

Command:

Updated file:

Latest score:

Largest changes:

Next action:

Safety and Scope

• Do not claim a live benchmark ran when only prompts were generated.
• Do not claim a skill passed if the run returned not_run.
• Do not present not_run as a quality regression; it means no response was scored.
• Do not automatically commit benchmark results or patches.
• Do not include private raw conversations in benchmark cases; abstract them first.
• Keep benchmark cases small and realistic.
• Preserve local drafts such as articles/ and docs/superpowers/ unless the user explicitly asks to commit them.

Common Mistakes

• Treating --prompts output as a completed benchmark.
• Running all benchmarks when the user only changed one skill.
• Patching a skill when the benchmark case wording is the real issue.
• Adding a new case without expected, must_not, and quality boundaries.
• Making benchmark work heavier than the actual skill change.