# Sk Create Benchmark

> Author MCP-promotion and behavior artifacts.

- Skill: `michelkerkmeester/sk-create-benchmark` (Agent Skill, multi-file: 31 files)
- Install (CLI): `npx get-skill add michelkerkmeester/sk-create-benchmark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/michelkerkmeester/sk-create-benchmark/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: MichelKerkmeester (https://skillmd.com/u/michelkerkmeester)
- Updated: 2026-11-15
- Page: https://skillmd.com/skills/michelkerkmeester/sk-create-benchmark

---


# create-benchmark

`create-benchmark` is the `sk-doc` benchmark-authoring packet. It covers:

- **MCP promotion** — promote curated evidence from a shipped spec packet into a consuming skill's `mcp-server/benchmarks/benchmark-<YYYY-MM-DD>/`, including winner, fixture, caveats, replay commands, and source pointer (§3-8).
- **Behavior** — author a `<mode>/behavior-benchmark/` index, scenario contracts, and Claude baseline under the shared measurement framework (§9).

The skill-local surface is the look-here-first entry point, not the archive.

---

## 1. WHEN TO USE

Use this packet to author completed benchmark evidence or benchmark inputs into the skill tree. Route through the family table in `ROUTER.md` first; families are distinct.

### Activation Triggers

- **MCP promotion** (on-disk `shared`; §3-8) — promote a completed MCP benchmark from a spec packet: author the ten-section `benchmark-report.md` and `source.md`, copy `results.csv`, applicable `per-probe.jsonl` and runtime sidecars into a dated folder, then update `benchmarks/README.md`.
- **Behavior benchmark** (§9) — author or extend a deep-loop mode's index, `<PREFIX>-NNN-<slug>.md` scenario contracts, baseline, and entry-surface/clarity matrix. Fixed prefixes are `research` (RSB) and `review` (RVB); declare a new mode's prefix in the index OVERVIEW.

Keyword triggers: `benchmark-report.md`, `source.md`, `mcp-server/benchmarks`; `behavior benchmark`, `behavior-benchmark.md`, `behavior_benchmark`, `scenario contract`, `benchmark/README.md`, `run-label folder`, `benchmark package`

### Adoption Gate (MCP promotion)

Create a skill-local MCP-promotion folder only when all apply:

- The skill houses an MCP server under `mcp-server/`.
- It produces a measurable retrieval, quality, runtime, throughput, recall, hit-rate, latency, RAM, dimension, or similar numeric outcome.
- A benchmark run has already completed inside a spec packet.
- The headline belongs where MCP operators read code.
- A stable fixture, replay commands, and expected outcome make the run reproducible.
- The source packet has accepted ADRs or an accepted decision record.

Decision rule:

```text
Measurable retrieval surface + shipped spec packet with accepted ADRs + stable fixture?
  YES -> Create a benchmark folder
  NO  -> Keep results in the spec packet's evidence/ directory
```

### Trigger Signals

Signals that warrant MCP promotion:

- An accepted decision promoted a non-trivial default such as an embedder, reranker, retrieval pipeline, or runtime setting, and operators will ask why.
- Readers already need a deep spec path to find the numbers.
- The same comparison table would otherwise be duplicated, or a sibling skill has an analogous benchmark surface.

### When NOT to Use

Use another `sk-doc` packet when:

- The benchmark is still in progress or lacks an accepted decision record.
- The result is unreplayable: no stable fixture or replay commands.
- The skill lacks an MCP server or measurable retrieval, quality, runtime, or throughput surface.
- The user only needs a release note or changelog row. Use `create-changelog`.
- The user wants to audit, validate, score, or optimize existing benchmark markdown without authoring a benchmark-family artifact. Use `create-quality-control`.
- The task is a general benchmark design exercise rather than promotion of an already-curated run.
- A re-run confirms the same headline; update the existing `benchmark-report.md` with a re-run note instead.
- The result mixes data from different MCP stacks and asks for a single comparative verdict.
- The task defines scoring (rubric, evaluator or reviewer verdict). Scoring stays lane-local; this packet authors inputs, indexes, and reports.
- A one-off experiment fits none of the families, such as a one-off image-audit benchmark kept beside the skill that ran it. Keep it lane-local; there is no "misc" family.

If unsure, default to "not yet." Promote after rigor.

---

## 2. SMART ROUTING

The `sk-doc` hub routes benchmark authoring here: the request names a benchmark report, a benchmark package or a behavior benchmark, and the hub's `sk-create-benchmark` mode signal selects this packet.
Two families exist, MCP promotion and behavior, and the family table decides which one applies before any file is written.

`references/shared/README.md` is the route map and the fallback load; on a resolved family the sorted keyed set loads the `assets/` templates before it.

Read [`ROUTER.md`](./ROUTER.md) before selecting skill-local resources; open only the matched files.

---

## 3. REQUIRED PACKAGE SHAPE

Use this shape in the consuming skill:

```text
mcp-server/benchmarks/
├── README.md
└── benchmark-<YYYY-MM-DD>/
    ├── benchmark-report.md
    ├── results.csv
    ├── per-probe.jsonl
    ├── runtime-measurements.md
    └── source.md
```

Required and optional files:

| File | Required | Purpose |
| --- | --- | --- |
| `README.md` | Yes | Index of all benchmark folders in this skill |
| `benchmark-<YYYY-MM-DD>/` | One per promoted run | Dated subfolder using the benchmark execution date |
| `benchmark-report.md` | Yes | Ten-section operator-facing report |
| `results.csv` | Yes | Primary aggregate metrics, one row per candidate |
| `per-probe.jsonl` | When applicable | Per-query or per-probe rows |
| `runtime-measurements.md` | Optional | RAM, GPU, latency, cold-load, or runtime profile worth promoting |
| `source.md` | Yes | Pointer back to the authoritative spec packet |

`source.md` is navigation, not an audit duplicate. Include the packet path, purpose, question-to-file and evidence maps, follow-on and rename/renumber notes, and last-updated date.

Use `assets/shared/benchmark-report-template.md` for `benchmark-report.md` and `assets/shared/source-template.md` for `source.md`.

For case studies, the worked example, and pitfalls, use `references/shared/README.md`.

---

## 4. HOW IT WORKS: AUTHORING WORKFLOW

Complete these steps in order after the spec packet ships.

Before naming a folder, run label, fixture, or report, validate its resolved path or slug. Do not pass machine keys such as `behavior_benchmark` as filesystem names.

```bash
python3 .skilled/skills/sk-doc/shared/scripts/check_authored_name_kebab.py <artifact-path-or-slug>
```

1. **Confirm the promotion gate.** Read `decision-record.md`, `implementation-summary.md`, and benchmark evidence. Require an accepted decision, stable headline and fixture, replay commands, and a defensible winner or explicit provisional status.
2. **Classify the task.** Decide whether this is a true promotion, a re-run update, or a retirement update.
3. **Confirm the target skill.** Verify the consuming skill has `mcp-server/` and an appropriate measurable MCP surface.
4. **Create or update the index.** Ensure `mcp-server/benchmarks/README.md` can hold date, folder link, winner/status, headline metric, and source path.
5. **Choose the dated folder.** Name it `benchmark-YYYY-MM-DD/` using the benchmark execution date, not the document authoring date.
6. **Copy source artifacts.** Copy aggregate CSV, applicable `per-probe.jsonl`, then only decision-relevant runtime or risk sidecars.
7. **Write `benchmark-report.md`.** Use the ten-section template. Generalize the headline; separate a non-obvious load-bearing insight from the winner.
8. **Write `source.md`.** Use the template with packet path, navigation and evidence maps, follow-on notes, and last-updated date.
9. **Update the README index row.** Add or update the row with date, folder link, winner or status, headline metric, and source packet path.
10. **Validate markdown.** Run `python3 .skilled/skills/sk-doc/shared/scripts/validate_document.py --type readme <file>` on `benchmark-report.md` and `benchmarks/README.md`; fix blockers.

### Authoring Order Rules

- Stabilize the headline and section structure before polishing prose.
- Write the load-bearing insight and caveats before recommendations.
- Validate before adding or finalizing the README index row.
- Keep raw evidence out of the report; curate and link through `source.md`.

### Artifact Sources

| Skill-local file | Source in spec packet |
| --- | --- |
| `benchmark-report.md` | Curated rewrite of `benchmark-results.md` or equivalent benchmark summary |
| `results.csv` | Copy of `evidence/*comparison*.csv` or topic equivalent |
| `per-probe.jsonl` | Copy of `evidence/*.jsonl` per-probe output when applicable |
| `runtime-measurements.md` | Curated rewrite or copy of runtime evidence when the profile affects the decision |
| `source.md` | Authored fresh; points back to the spec packet |

---

## 5. REPORT CONTRACT

Keep this fixed ten-section `benchmark-report.md` structure; never merge, reorder, or omit:

1. **HEADLINE / OVERVIEW**: one-line winner or status plus key metric.
2. **AGGREGATE RESULTS**: one row per candidate, headline metrics, and verdict.
3. **METHODOLOGY**: fixture, sample size, pipeline, environment, and replay context.
4. **PER-CANDIDATE PROFILES**: RAM, disk, dim, release, strengths, and weaknesses.
5. **PROCESS NOTES**: what was tried, what failed, and why.
6. **FINDINGS**: unique wins, universal floor and ceiling, mismatch analysis, and load-bearing insight.
7. **CAVEATS**: single-run signal, fixture limits, stack-level confounds, schema migration cost, reranker/runtime confounds.
8. **RECOMMENDATIONS**: Tier 1 apply now, Tier 2 validate first, Tier 3 future.
9. **REPRODUCIBILITY**: exact replay commands plus expected wall-clock.
10. **CROSS-LINKS**: sibling MCP benchmarks, authoritative spec packet, follow-on packets.

For an empty section, retain its header and write: `Not applicable to this bench. Reason: ...`

### Report Style

- Match sk-doc reference quality and frontmatter.
- Use H2 numbered headers with ALL CAPS section names.
- Use tables for data and fenced code blocks for verbatim commands.
- Keep slugs and anchors stable across revisions.
- Do not paste the full spec packet decision trail into the report.
- Do not compare numeric results across different MCP stacks as if equivalent.

### Headline Pattern

Use this shape near the top of the report:

```markdown
# <topic> bake-off: <date>

> Headline: `<winner>` + <pipeline-config> is the production default for `<MCP-stack>`.
> <primary-metric> <numeric-result>, <secondary-metric> <numeric-result>. Closes packet <packet-id>.
```

If the load-bearing insight differs from the winner, state it in Sections 1 and 6.

---

## 6. DATE AND NAMING CONVENTION

### Folder Names

One grammar governs every benchmark family. A run folder is named:

```text
<YYYY-MM-DD>--<subject>--<variant>/
```

- Fields are separated by a double hyphen. Words inside a field use a single hyphen.
- Lowercase ASCII only. No underscores, no dots, no capitals.
- The date is the **execution** date, not the authoring or publication date. Publication date belongs in
  the report body, where a reader can see both.
- `<subject>` names the corpus the run measured: `manual-testing-playbook`, `mcp-retrieval`,
  `model-eval`, `behavior`.
- `<variant>` names what most distinguishes the run. For a **feature-scoped** run — one validating a
  single feature or scenario group rather than comparing executors — name the feature (e.g. `goal-hook`),
  not the model; the model/executor is recorded inside the report (`model`/`providerModel` fields, the
  byline, the command shape), so the folder stays legible when the same feature is later re-validated on
  another model. For an **executor-scoped** run — one whose point is which model and effort produced it —
  name the executor identity as `<runtime>-<model>-<effort>`; use `model-comparison` when it compares
  several, or a short topic slug when no model applies.
- Model versions flatten their dots: `glm-5.2` becomes `glm-5-2`, `kimi-2.7` becomes `kimi-2-7`. A dot
  is rejected by the run-label validator, so this is a hard requirement rather than a style preference.
- Two runs of the same subject and variant on one day disambiguate with a trailing topic field.

Examples:

```text
2026-07-29--manual-testing-playbook--goal-hook/                 # feature-scoped: names the feature, model recorded inside
2026-07-26--manual-testing-playbook--cursor-grok-4-5-high-fast/ # executor-scoped: names the model+effort that ran it
2026-07-26--manual-testing-playbook--model-comparison/
2026-05-18--mcp-retrieval--bge-confirmation/
```

**One exception: `baseline/`.** The frozen comparison anchor keeps its name. It is not a run, it is the
before-snapshot every run is measured against, and the archiver refuses it as a run label for exactly
that reason. Dating it would make it look like one run among many.

Authored scenario contracts are not run folders. Behavior families keep their
`<PREFIX>-NNN-<slug>.md` scenario identifiers, which name a scenario rather than a directory. Any run
those families produce follows the grammar above.

### In-Document Dates

Use readable long-form dates in prose, such as `May 18, 2026`.

### Re-Runs

For a re-run confirming the same headline, update the existing report with `Re-run YYYY-MM-DD`; do not create a folder.

### Retirement

Keep retired benchmarks. Mark `README.md` as `RETIRED` with date; add report date, reason, and replacement if any.

### Renamed or Renumbered Spec Packets

Do not rename the dated folder. Update `source.md`, report and README cross-links, then record old slug, new slug, and date in `source.md`.

---

## 7. RULES: AUTHORITY AND GATES

### Authority Hierarchy

When documents disagree:

1. Source spec packet `decision-record.md` and `implementation-summary.md` are authoritative.
2. Skill-local `benchmark-report.md` is the curated operator-facing summary.
3. Copied CSV and JSONL files preserve the source packet evidence.
4. `source.md` is navigation, not a duplicate audit trail.

### ✅ ALWAYS

1. Read the source packet decision record, implementation summary, and benchmark evidence before writing.
2. Use the benchmark execution date for `benchmark-<YYYY-MM-DD>/`.
3. Keep `source.md` lean and navigational.
4. Include caveats for single-run signal, fixture limits, stack mismatch, schema migration cost, and reranker/runtime confounds when present.
5. Preserve retired benchmark folders.
6. Validate authored markdown before delivery.

### ⛔ NEVER

1. Never promote an in-flight benchmark as a final skill-local record.
2. Never compare numeric results across different MCP stacks as if equivalent.
3. Never paste the full spec packet audit trail into `benchmark-report.md`.
4. Never create a new dated folder for a confirming re-run.
5. Never name the folder by authoring date, source packet slug, or candidate name.
6. Never leave template placeholders in shipped benchmark files.
7. Never add packet-local `graph-metadata.json`.

### ⚠️ ESCALATE IF

1. The source packet has no accepted decision record or stable benchmark headline.
2. The target skill lacks `mcp-server/` or an appropriate measurable MCP surface.
3. Source artifacts are missing, non-replayable, or internally contradictory.
4. The benchmark spans multiple stacks and the user wants a single comparative verdict.
5. Validation fails on required markdown structure after local fixes.

---

## 8. SUCCESS CRITERIA

- The consuming skill has a dated `mcp-server/benchmarks/benchmark-<YYYY-MM-DD>/` folder.
- `benchmark-report.md` uses the ten-section structure and includes winner or status, aggregate table, methodology, candidate profiles, findings, caveats, recommendations, replay commands, and cross-links.
- `source.md` points to the authoritative spec packet and maps reader questions to source files.
- Raw artifacts are copied or intentionally omitted with a documented reason.
- The benchmarks README index links the new folder and source packet.
- Shared sk-doc validation passes for authored markdown, or any remaining issue is escalated with exact command output.

---

## 9. BEHAVIOR BENCHMARK PACKAGES

Behavior benchmarks are run contracts, not numeric MCP bake-offs. They specify
whether a realistic prompt makes an executor model dispatch the mode's LEAF agent,
ask one consolidated setup question when under-specified, respect invariants, and
finish relative to a Claude reference leg.

The authoring path, matrix rules, and naming live in
[`references/behavior-benchmark/behavior-benchmark-guide.md`](references/behavior-benchmark/behavior-benchmark-guide.md).
The normative five-dimension rubric, terminal buckets, budget formula, entry-surface
and clarity enums, and ID prefixes live only in
[`../../system-deep-loop/shared/behavior-benchmark/framework.md`](../../system-deep-loop/shared/behavior-benchmark/framework.md).
Instantiate that framework; do not redefine it.

### Package Shape

Author this shape inside the owning deep-loop mode-packet:

```text
<mode>/behavior-benchmark/
├── behavior-benchmark.md          # package index: scenario table + axis coverage
├── scenarios/
│   └── <PREFIX>-NNN-<slug>.md     # one machine-contract file per scenario
└── baselines/
    └── claude-baseline.md         # per-scenario Claude-leg reference checkpoints
```

Do not ship fixtures, lane configs, transcripts, result JSON, or scorecards here.
The executing packet provisions fixtures and holds proof; this package is the contract.

### Templates, Workflow, and Naming

Load the [behavior-benchmark guide](references/behavior-benchmark/behavior-benchmark-guide.md)
and shared framework before authoring. The guide owns templates, sequence, matrix,
and naming; execution and evidence stay in the executing packet.

### ✅ ALWAYS / ⛔ NEVER (behavior benchmark)

- **ALWAYS** keep the shared `framework.md` as the single source for rubric, buckets, budget formula, and enums; the package instantiates, it does not redefine.
- **ALWAYS** keep the index SCENARIO TABLE and the scenario files in exact sync.
- **ALWAYS** ship uncaptured baseline cells as `pending` / `not_captured` rather than inventing values.
- **ALWAYS** count an expected marker, target or postcondition only after a run that skips the behavior has failed it (a negative control), and name that run in the scenario's Failure modes.
- **NEVER** ship fixtures, transcripts, or result JSONs inside the package — they belong to the executing spec-packet phase.
- **NEVER** give a shipped scenario file frontmatter or a `## OVERVIEW` heading; it opens at the `# <PREFIX>-NNN` H1 (the scenario template's own frontmatter, usage comment, and Overview are stripped on copy).
- **NEVER** add a scenario whose `id` disagrees with its filename or its index-table row.

### Success Criteria (behavior benchmark)

- The owning mode-packet has a `behavior-benchmark/` package with an index, one scenario file per table row, and a baseline.
- Every scenario's first json block parses, its `id` matches its filename and index row, and its axis values agree across the table, scenario, and baseline.
- The index AXIS COVERAGE section reports per-surface and per-clarity counts and names any axis intentionally left out with its reason.
- The baseline ships with real values or with `pending` / `not_captured` cells, never invented ones.
- Shared sk-doc validation passes for the index and baseline, or any remaining issue is escalated with exact command output.

---

## 10. INTEGRATION POINTS

This packet authors inputs and indexes.

Filesystem outputs, template directories, and filenames use lowercase kebab-case. Router and JSON family keys remain snake_case machine identifiers; the router converts them to resource paths.

`/create:benchmark` drives `mcp_promotion` (§3-8) end-to-end. Author behavior directly from its section; no command scaffolds it.

---

## 11. REFERENCES AND RELATED RESOURCES

**Within this packet** — family guides and the overflow route-map; the fillable templates are mapped in each family section above:

- [`references/shared/README.md`](references/shared/README.md) — overflow route-map (case studies, worked example, pitfalls).
- [`references/behavior-benchmark/behavior-benchmark-guide.md`](references/behavior-benchmark/behavior-benchmark-guide.md) — behavior package authoring path (§9).

**Lane-owned contracts** — cross-link, never restate:

- [`behavior-benchmark/framework.md`](../../system-deep-loop/shared/behavior-benchmark/framework.md) — behavior rubric, buckets, budget formula, runner.

**Shared sk-doc backbone**: [`../shared/scripts/validate_document.py`](../shared/scripts/validate_document.py) — every authored `.md` must pass with 0 issues; [`../shared/references/`](../shared/references/) — cross-document standards.

