create-benchmark
create-benchmark is the sk-doc benchmark-authoring packet. It covers:
- MCP promotion — promote curated evidence from a shipped spec packet into a consuming skill's
mcp-server/benchmarks/benchmark-<YYYY-MM-DD>/, including winner, fixture, caveats, replay commands, and source pointer (§3-8).
- Behavior — author a
<mode>/behavior-benchmark/ index, scenario contracts, and Claude baseline under the shared measurement framework (§9).
The skill-local surface is the look-here-first entry point, not the archive.
1. WHEN TO USE
Use this packet to author completed benchmark evidence or benchmark inputs into the skill tree. Route through the family table in ROUTER.md first; families are distinct.
Activation Triggers
- MCP promotion (on-disk
shared; §3-8) — promote a completed MCP benchmark from a spec packet: author the ten-section benchmark-report.md and source.md, copy results.csv, applicable per-probe.jsonl and runtime sidecars into a dated folder, then update benchmarks/README.md.
- Behavior benchmark (§9) — author or extend a deep-loop mode's index,
<PREFIX>-NNN-<slug>.md scenario contracts, baseline, and entry-surface/clarity matrix. Fixed prefixes are research (RSB) and review (RVB); declare a new mode's prefix in the index OVERVIEW.
Keyword triggers: benchmark-report.md, source.md, mcp-server/benchmarks; behavior benchmark, behavior-benchmark.md, behavior_benchmark, scenario contract, benchmark/README.md, run-label folder, benchmark package
Adoption Gate (MCP promotion)
Create a skill-local MCP-promotion folder only when all apply:
- The skill houses an MCP server under
mcp-server/.
- It produces a measurable retrieval, quality, runtime, throughput, recall, hit-rate, latency, RAM, dimension, or similar numeric outcome.
- A benchmark run has already completed inside a spec packet.
- The headline belongs where MCP operators read code.
- A stable fixture, replay commands, and expected outcome make the run reproducible.
- The source packet has accepted ADRs or an accepted decision record.
Decision rule:
Measurable retrieval surface + shipped spec packet with accepted ADRs + stable fixture?
YES -> Create a benchmark folder
NO -> Keep results in the spec packet's evidence/ directory
Trigger Signals
Signals that warrant MCP promotion:
- An accepted decision promoted a non-trivial default such as an embedder, reranker, retrieval pipeline, or runtime setting, and operators will ask why.
- Readers already need a deep spec path to find the numbers.
- The same comparison table would otherwise be duplicated, or a sibling skill has an analogous benchmark surface.
When NOT to Use
Use another sk-doc packet when:
- The benchmark is still in progress or lacks an accepted decision record.
- The result is unreplayable: no stable fixture or replay commands.
- The skill lacks an MCP server or measurable retrieval, quality, runtime, or throughput surface.
- The user only needs a release note or changelog row. Use
create-changelog.
- The user wants to audit, validate, score, or optimize existing benchmark markdown without authoring a benchmark-family artifact. Use
create-quality-control.
- The task is a general benchmark design exercise rather than promotion of an already-curated run.
- A re-run confirms the same headline; update the existing
benchmark-report.md with a re-run note instead.
- The result mixes data from different MCP stacks and asks for a single comparative verdict.
- The task defines scoring (rubric, evaluator or reviewer verdict). Scoring stays lane-local; this packet authors inputs, indexes, and reports.
- A one-off experiment fits none of the families, such as a one-off image-audit benchmark kept beside the skill that ran it. Keep it lane-local; there is no "misc" family.
If unsure, default to "not yet." Promote after rigor.
2. SMART ROUTING
The sk-doc hub routes benchmark authoring here: the request names a benchmark report, a benchmark package or a behavior benchmark, and the hub's sk-create-benchmark mode signal selects this packet.
Two families exist, MCP promotion and behavior, and the family table decides which one applies before any file is written.
references/shared/README.md is the route map and the fallback load; on a resolved family the sorted keyed set loads the assets/ templates before it.
Read ROUTER.md before selecting skill-local resources; open only the matched files.
3. REQUIRED PACKAGE SHAPE
Use this shape in the consuming skill:
mcp-server/benchmarks/
├── README.md
└── benchmark-<YYYY-MM-DD>/
├── benchmark-report.md
├── results.csv
├── per-probe.jsonl
├── runtime-measurements.md
└── source.md
Required and optional files:
| File |
Required |
Purpose |
README.md |
Yes |
Index of all benchmark folders in this skill |
benchmark-<YYYY-MM-DD>/ |
One per promoted run |
Dated subfolder using the benchmark execution date |
benchmark-report.md |
Yes |
Ten-section operator-facing report |
results.csv |
Yes |
Primary aggregate metrics, one row per candidate |
per-probe.jsonl |
When applicable |
Per-query or per-probe rows |
runtime-measurements.md |
Optional |
RAM, GPU, latency, cold-load, or runtime profile worth promoting |
source.md |
Yes |
Pointer back to the authoritative spec packet |
source.md is navigation, not an audit duplicate. Include the packet path, purpose, question-to-file and evidence maps, follow-on and rename/renumber notes, and last-updated date.
Use assets/shared/benchmark-report-template.md for benchmark-report.md and assets/shared/source-template.md for source.md.
For case studies, the worked example, and pitfalls, use references/shared/README.md.
4. HOW IT WORKS: AUTHORING WORKFLOW
Complete these steps in order after the spec packet ships.
Before naming a folder, run label, fixture, or report, validate its resolved path or slug. Do not pass machine keys such as behavior_benchmark as filesystem names.
python3 .skilled/skills/sk-doc/shared/scripts/check_authored_name_kebab.py <artifact-path-or-slug>
- Confirm the promotion gate. Read
decision-record.md, implementation-summary.md, and benchmark evidence. Require an accepted decision, stable headline and fixture, replay commands, and a defensible winner or explicit provisional status.
- Classify the task. Decide whether this is a true promotion, a re-run update, or a retirement update.
- Confirm the target skill. Verify the consuming skill has
mcp-server/ and an appropriate measurable MCP surface.
- Create or update the index. Ensure
mcp-server/benchmarks/README.md can hold date, folder link, winner/status, headline metric, and source path.
- Choose the dated folder. Name it
benchmark-YYYY-MM-DD/ using the benchmark execution date, not the document authoring date.
- Copy source artifacts. Copy aggregate CSV, applicable
per-probe.jsonl, then only decision-relevant runtime or risk sidecars.
- Write
benchmark-report.md. Use the ten-section template. Generalize the headline; separate a non-obvious load-bearing insight from the winner.
- Write
source.md. Use the template with packet path, navigation and evidence maps, follow-on notes, and last-updated date.
- Update the README index row. Add or update the row with date, folder link, winner or status, headline metric, and source packet path.
- Validate markdown. Run
python3 .skilled/skills/sk-doc/shared/scripts/validate_document.py --type readme <file> on benchmark-report.md and benchmarks/README.md; fix blockers.
Authoring Order Rules
- Stabilize the headline and section structure before polishing prose.
- Write the load-bearing insight and caveats before recommendations.
- Validate before adding or finalizing the README index row.
- Keep raw evidence out of the report; curate and link through
source.md.
Artifact Sources
| Skill-local file |
Source in spec packet |
benchmark-report.md |
Curated rewrite of benchmark-results.md or equivalent benchmark summary |
results.csv |
Copy of evidence/*comparison*.csv or topic equivalent |
per-probe.jsonl |
Copy of evidence/*.jsonl per-probe output when applicable |
runtime-measurements.md |
Curated rewrite or copy of runtime evidence when the profile affects the decision |
source.md |
Authored fresh; points back to the spec packet |
5. REPORT CONTRACT
Keep this fixed ten-section benchmark-report.md structure; never merge, reorder, or omit:
- HEADLINE / OVERVIEW: one-line winner or status plus key metric.
- AGGREGATE RESULTS: one row per candidate, headline metrics, and verdict.
- METHODOLOGY: fixture, sample size, pipeline, environment, and replay context.
- PER-CANDIDATE PROFILES: RAM, disk, dim, release, strengths, and weaknesses.
- PROCESS NOTES: what was tried, what failed, and why.
- FINDINGS: unique wins, universal floor and ceiling, mismatch analysis, and load-bearing insight.
- CAVEATS: single-run signal, fixture limits, stack-level confounds, schema migration cost, reranker/runtime confounds.
- RECOMMENDATIONS: Tier 1 apply now, Tier 2 validate first, Tier 3 future.
- REPRODUCIBILITY: exact replay commands plus expected wall-clock.
- CROSS-LINKS: sibling MCP benchmarks, authoritative spec packet, follow-on packets.
For an empty section, retain its header and write: Not applicable to this bench. Reason: ...
Report Style
- Match sk-doc reference quality and frontmatter.
- Use H2 numbered headers with ALL CAPS section names.
- Use tables for data and fenced code blocks for verbatim commands.
- Keep slugs and anchors stable across revisions.
- Do not paste the full spec packet decision trail into the report.
- Do not compare numeric results across different MCP stacks as if equivalent.
Headline Pattern
Use this shape near the top of the report:
# <topic> bake-off: <date>
> Headline: `<winner>` + <pipeline-config> is the production default for `<MCP-stack>`.
> <primary-metric> <numeric-result>, <secondary-metric> <numeric-result>. Closes packet <packet-id>.
If the load-bearing insight differs from the winner, state it in Sections 1 and 6.
6. DATE AND NAMING CONVENTION
Folder Names
One grammar governs every benchmark family. A run folder is named:
<YYYY-MM-DD>--<subject>--<variant>/
- Fields are separated by a double hyphen. Words inside a field use a single hyphen.
- Lowercase ASCII only. No underscores, no dots, no capitals.
- The date is the execution date, not the authoring or publication date. Publication date belongs in
the report body, where a reader can see both.
<subject> names the corpus the run measured: manual-testing-playbook, mcp-retrieval,
model-eval, behavior.
<variant> names what most distinguishes the run. For a feature-scoped run — one validating a
single feature or scenario group rather than comparing executors — name the feature (e.g. goal-hook),
not the model; the model/executor is recorded inside the report (model/providerModel fields, the
byline, the command shape), so the folder stays legible when the same feature is later re-validated on
another model. For an executor-scoped run — one whose point is which model and effort produced it —
name the executor identity as <runtime>-<model>-<effort>; use model-comparison when it compares
several, or a short topic slug when no model applies.
- Model versions flatten their dots:
glm-5.2 becomes glm-5-2, kimi-2.7 becomes kimi-2-7. A dot
is rejected by the run-label validator, so this is a hard requirement rather than a style preference.
- Two runs of the same subject and variant on one day disambiguate with a trailing topic field.
Examples:
2026-07-29--manual-testing-playbook--goal-hook/ # feature-scoped: names the feature, model recorded inside
2026-07-26--manual-testing-playbook--cursor-grok-4-5-high-fast/ # executor-scoped: names the model+effort that ran it
2026-07-26--manual-testing-playbook--model-comparison/
2026-05-18--mcp-retrieval--bge-confirmation/
One exception: baseline/. The frozen comparison anchor keeps its name. It is not a run, it is the
before-snapshot every run is measured against, and the archiver refuses it as a run label for exactly
that reason. Dating it would make it look like one run among many.
Authored scenario contracts are not run folders. Behavior families keep their
<PREFIX>-NNN-<slug>.md scenario identifiers, which name a scenario rather than a directory. Any run
those families produce follows the grammar above.
In-Document Dates
Use readable long-form dates in prose, such as May 18, 2026.
Re-Runs
For a re-run confirming the same headline, update the existing report with Re-run YYYY-MM-DD; do not create a folder.
Retirement
Keep retired benchmarks. Mark README.md as RETIRED with date; add report date, reason, and replacement if any.
Renamed or Renumbered Spec Packets
Do not rename the dated folder. Update source.md, report and README cross-links, then record old slug, new slug, and date in source.md.
7. RULES: AUTHORITY AND GATES
Authority Hierarchy
When documents disagree:
- Source spec packet
decision-record.md and implementation-summary.md are authoritative.
- Skill-local
benchmark-report.md is the curated operator-facing summary.
- Copied CSV and JSONL files preserve the source packet evidence.
source.md is navigation, not a duplicate audit trail.
✅ ALWAYS
- Read the source packet decision record, implementation summary, and benchmark evidence before writing.
- Use the benchmark execution date for
benchmark-<YYYY-MM-DD>/.
- Keep
source.md lean and navigational.
- Include caveats for single-run signal, fixture limits, stack mismatch, schema migration cost, and reranker/runtime confounds when present.
- Preserve retired benchmark folders.
- Validate authored markdown before delivery.
⛔ NEVER
- Never promote an in-flight benchmark as a final skill-local record.
- Never compare numeric results across different MCP stacks as if equivalent.
- Never paste the full spec packet audit trail into
benchmark-report.md.
- Never create a new dated folder for a confirming re-run.
- Never name the folder by authoring date, source packet slug, or candidate name.
- Never leave template placeholders in shipped benchmark files.
- Never add packet-local
graph-metadata.json.
⚠️ ESCALATE IF
- The source packet has no accepted decision record or stable benchmark headline.
- The target skill lacks
mcp-server/ or an appropriate measurable MCP surface.
- Source artifacts are missing, non-replayable, or internally contradictory.
- The benchmark spans multiple stacks and the user wants a single comparative verdict.
- Validation fails on required markdown structure after local fixes.
8. SUCCESS CRITERIA
- The consuming skill has a dated
mcp-server/benchmarks/benchmark-<YYYY-MM-DD>/ folder.
benchmark-report.md uses the ten-section structure and includes winner or status, aggregate table, methodology, candidate profiles, findings, caveats, recommendations, replay commands, and cross-links.
source.md points to the authoritative spec packet and maps reader questions to source files.
- Raw artifacts are copied or intentionally omitted with a documented reason.
- The benchmarks README index links the new folder and source packet.
- Shared sk-doc validation passes for authored markdown, or any remaining issue is escalated with exact command output.
9. BEHAVIOR BENCHMARK PACKAGES
Behavior benchmarks are run contracts, not numeric MCP bake-offs. They specify
whether a realistic prompt makes an executor model dispatch the mode's LEAF agent,
ask one consolidated setup question when under-specified, respect invariants, and
finish relative to a Claude reference leg.
The authoring path, matrix rules, and naming live in
references/behavior-benchmark/behavior-benchmark-guide.md.
The normative five-dimension rubric, terminal buckets, budget formula, entry-surface
and clarity enums, and ID prefixes live only in
../../system-deep-loop/shared/behavior-benchmark/framework.md.
Instantiate that framework; do not redefine it.
Package Shape
Author this shape inside the owning deep-loop mode-packet:
<mode>/behavior-benchmark/
├── behavior-benchmark.md # package index: scenario table + axis coverage
├── scenarios/
│ └── <PREFIX>-NNN-<slug>.md # one machine-contract file per scenario
└── baselines/
└── claude-baseline.md # per-scenario Claude-leg reference checkpoints
Do not ship fixtures, lane configs, transcripts, result JSON, or scorecards here.
The executing packet provisions fixtures and holds proof; this package is the contract.
Templates, Workflow, and Naming
Load the behavior-benchmark guide
and shared framework before authoring. The guide owns templates, sequence, matrix,
and naming; execution and evidence stay in the executing packet.
✅ ALWAYS / ⛔ NEVER (behavior benchmark)
- ALWAYS keep the shared
framework.md as the single source for rubric, buckets, budget formula, and enums; the package instantiates, it does not redefine.
- ALWAYS keep the index SCENARIO TABLE and the scenario files in exact sync.
- ALWAYS ship uncaptured baseline cells as
pending / not_captured rather than inventing values.
- ALWAYS count an expected marker, target or postcondition only after a run that skips the behavior has failed it (a negative control), and name that run in the scenario's Failure modes.
- NEVER ship fixtures, transcripts, or result JSONs inside the package — they belong to the executing spec-packet phase.
- NEVER give a shipped scenario file frontmatter or a
## OVERVIEW heading; it opens at the # <PREFIX>-NNN H1 (the scenario template's own frontmatter, usage comment, and Overview are stripped on copy).
- NEVER add a scenario whose
id disagrees with its filename or its index-table row.
Success Criteria (behavior benchmark)
- The owning mode-packet has a
behavior-benchmark/ package with an index, one scenario file per table row, and a baseline.
- Every scenario's first json block parses, its
id matches its filename and index row, and its axis values agree across the table, scenario, and baseline.
- The index AXIS COVERAGE section reports per-surface and per-clarity counts and names any axis intentionally left out with its reason.
- The baseline ships with real values or with
pending / not_captured cells, never invented ones.
- Shared sk-doc validation passes for the index and baseline, or any remaining issue is escalated with exact command output.
10. INTEGRATION POINTS
This packet authors inputs and indexes.
Filesystem outputs, template directories, and filenames use lowercase kebab-case. Router and JSON family keys remain snake_case machine identifiers; the router converts them to resource paths.
/create:benchmark drives mcp_promotion (§3-8) end-to-end. Author behavior directly from its section; no command scaffolds it.
11. REFERENCES AND RELATED RESOURCES
Within this packet — family guides and the overflow route-map; the fillable templates are mapped in each family section above:
references/shared/README.md — overflow route-map (case studies, worked example, pitfalls).
references/behavior-benchmark/behavior-benchmark-guide.md — behavior package authoring path (§9).
Lane-owned contracts — cross-link, never restate:
Shared sk-doc backbone: ../shared/scripts/validate_document.py — every authored .md must pass with 0 issues; ../shared/references/ — cross-document standards.
1---2name: sk-create-benchmark3description: Author MCP-promotion and behavior artifacts.4allowed-tools: [Read, Write, Edit, Bash, Grep, Glob]5version: 1.6.7.06metadata:7 source: https://skillmd.com/skills/michelkerkmeester/sk-create-benchmark8---910# create-benchmark1112`create-benchmark` is the `sk-doc` benchmark-authoring packet. It covers:1314- **MCP promotion** — promote curated evidence from a shipped spec packet into a consuming skill's `mcp-server/benchmarks/benchmark-<YYYY-MM-DD>/`, including winner, fixture, caveats, replay commands, and source pointer (§3-8).15- **Behavior** — author a `<mode>/behavior-benchmark/` index, scenario contracts, and Claude baseline under the shared measurement framework (§9).1617The skill-local surface is the look-here-first entry point, not the archive.1819---2021## 1. WHEN TO USE2223Use this packet to author completed benchmark evidence or benchmark inputs into the skill tree. Route through the family table in `ROUTER.md` first; families are distinct.2425### Activation Triggers2627- **MCP promotion** (on-disk `shared`; §3-8) — promote a completed MCP benchmark from a spec packet: author the ten-section `benchmark-report.md` and `source.md`, copy `results.csv`, applicable `per-probe.jsonl` and runtime sidecars into a dated folder, then update `benchmarks/README.md`.28- **Behavior benchmark** (§9) — author or extend a deep-loop mode's index, `<PREFIX>-NNN-<slug>.md` scenario contracts, baseline, and entry-surface/clarity matrix. Fixed prefixes are `research` (RSB) and `review` (RVB); declare a new mode's prefix in the index OVERVIEW.2930Keyword triggers: `benchmark-report.md`, `source.md`, `mcp-server/benchmarks`; `behavior benchmark`, `behavior-benchmark.md`, `behavior_benchmark`, `scenario contract`, `benchmark/README.md`, `run-label folder`, `benchmark package`3132### Adoption Gate (MCP promotion)3334Create a skill-local MCP-promotion folder only when all apply:3536- The skill houses an MCP server under `mcp-server/`.37- It produces a measurable retrieval, quality, runtime, throughput, recall, hit-rate, latency, RAM, dimension, or similar numeric outcome.38- A benchmark run has already completed inside a spec packet.39- The headline belongs where MCP operators read code.40- A stable fixture, replay commands, and expected outcome make the run reproducible.41- The source packet has accepted ADRs or an accepted decision record.4243Decision rule:4445```text46Measurable retrieval surface + shipped spec packet with accepted ADRs + stable fixture?47 YES -> Create a benchmark folder48 NO -> Keep results in the spec packet's evidence/ directory49```5051### Trigger Signals5253Signals that warrant MCP promotion:5455- An accepted decision promoted a non-trivial default such as an embedder, reranker, retrieval pipeline, or runtime setting, and operators will ask why.56- Readers already need a deep spec path to find the numbers.57- The same comparison table would otherwise be duplicated, or a sibling skill has an analogous benchmark surface.5859### When NOT to Use6061Use another `sk-doc` packet when:6263- The benchmark is still in progress or lacks an accepted decision record.64- The result is unreplayable: no stable fixture or replay commands.65- The skill lacks an MCP server or measurable retrieval, quality, runtime, or throughput surface.66- The user only needs a release note or changelog row. Use `create-changelog`.67- The user wants to audit, validate, score, or optimize existing benchmark markdown without authoring a benchmark-family artifact. Use `create-quality-control`.68- The task is a general benchmark design exercise rather than promotion of an already-curated run.69- A re-run confirms the same headline; update the existing `benchmark-report.md` with a re-run note instead.70- The result mixes data from different MCP stacks and asks for a single comparative verdict.71- The task defines scoring (rubric, evaluator or reviewer verdict). Scoring stays lane-local; this packet authors inputs, indexes, and reports.72- A one-off experiment fits none of the families, such as a one-off image-audit benchmark kept beside the skill that ran it. Keep it lane-local; there is no "misc" family.7374If unsure, default to "not yet." Promote after rigor.7576---7778## 2. SMART ROUTING7980The `sk-doc` hub routes benchmark authoring here: the request names a benchmark report, a benchmark package or a behavior benchmark, and the hub's `sk-create-benchmark` mode signal selects this packet.81Two families exist, MCP promotion and behavior, and the family table decides which one applies before any file is written.8283`references/shared/README.md` is the route map and the fallback load; on a resolved family the sorted keyed set loads the `assets/` templates before it.8485Read [`ROUTER.md`](./ROUTER.md) before selecting skill-local resources; open only the matched files.8687---8889## 3. REQUIRED PACKAGE SHAPE9091Use this shape in the consuming skill:9293```text94mcp-server/benchmarks/95├── README.md96└── benchmark-<YYYY-MM-DD>/97 ├── benchmark-report.md98 ├── results.csv99 ├── per-probe.jsonl100 ├── runtime-measurements.md101 └── source.md102```103104Required and optional files:105106| File | Required | Purpose |107| --- | --- | --- |108| `README.md` | Yes | Index of all benchmark folders in this skill |109| `benchmark-<YYYY-MM-DD>/` | One per promoted run | Dated subfolder using the benchmark execution date |110| `benchmark-report.md` | Yes | Ten-section operator-facing report |111| `results.csv` | Yes | Primary aggregate metrics, one row per candidate |112| `per-probe.jsonl` | When applicable | Per-query or per-probe rows |113| `runtime-measurements.md` | Optional | RAM, GPU, latency, cold-load, or runtime profile worth promoting |114| `source.md` | Yes | Pointer back to the authoritative spec packet |115116`source.md` is navigation, not an audit duplicate. Include the packet path, purpose, question-to-file and evidence maps, follow-on and rename/renumber notes, and last-updated date.117118Use `assets/shared/benchmark-report-template.md` for `benchmark-report.md` and `assets/shared/source-template.md` for `source.md`.119120For case studies, the worked example, and pitfalls, use `references/shared/README.md`.121122---123124## 4. HOW IT WORKS: AUTHORING WORKFLOW125126Complete these steps in order after the spec packet ships.127128Before naming a folder, run label, fixture, or report, validate its resolved path or slug. Do not pass machine keys such as `behavior_benchmark` as filesystem names.129130```bash131python3 .skilled/skills/sk-doc/shared/scripts/check_authored_name_kebab.py <artifact-path-or-slug>132```1331341. **Confirm the promotion gate.** Read `decision-record.md`, `implementation-summary.md`, and benchmark evidence. Require an accepted decision, stable headline and fixture, replay commands, and a defensible winner or explicit provisional status.1352. **Classify the task.** Decide whether this is a true promotion, a re-run update, or a retirement update.1363. **Confirm the target skill.** Verify the consuming skill has `mcp-server/` and an appropriate measurable MCP surface.1374. **Create or update the index.** Ensure `mcp-server/benchmarks/README.md` can hold date, folder link, winner/status, headline metric, and source path.1385. **Choose the dated folder.** Name it `benchmark-YYYY-MM-DD/` using the benchmark execution date, not the document authoring date.1396. **Copy source artifacts.** Copy aggregate CSV, applicable `per-probe.jsonl`, then only decision-relevant runtime or risk sidecars.1407. **Write `benchmark-report.md`.** Use the ten-section template. Generalize the headline; separate a non-obvious load-bearing insight from the winner.1418. **Write `source.md`.** Use the template with packet path, navigation and evidence maps, follow-on notes, and last-updated date.1429. **Update the README index row.** Add or update the row with date, folder link, winner or status, headline metric, and source packet path.14310. **Validate markdown.** Run `python3 .skilled/skills/sk-doc/shared/scripts/validate_document.py --type readme <file>` on `benchmark-report.md` and `benchmarks/README.md`; fix blockers.144145### Authoring Order Rules146147- Stabilize the headline and section structure before polishing prose.148- Write the load-bearing insight and caveats before recommendations.149- Validate before adding or finalizing the README index row.150- Keep raw evidence out of the report; curate and link through `source.md`.151152### Artifact Sources153154| Skill-local file | Source in spec packet |155| --- | --- |156| `benchmark-report.md` | Curated rewrite of `benchmark-results.md` or equivalent benchmark summary |157| `results.csv` | Copy of `evidence/*comparison*.csv` or topic equivalent |158| `per-probe.jsonl` | Copy of `evidence/*.jsonl` per-probe output when applicable |159| `runtime-measurements.md` | Curated rewrite or copy of runtime evidence when the profile affects the decision |160| `source.md` | Authored fresh; points back to the spec packet |161162---163164## 5. REPORT CONTRACT165166Keep this fixed ten-section `benchmark-report.md` structure; never merge, reorder, or omit:1671681. **HEADLINE / OVERVIEW**: one-line winner or status plus key metric.1692. **AGGREGATE RESULTS**: one row per candidate, headline metrics, and verdict.1703. **METHODOLOGY**: fixture, sample size, pipeline, environment, and replay context.1714. **PER-CANDIDATE PROFILES**: RAM, disk, dim, release, strengths, and weaknesses.1725. **PROCESS NOTES**: what was tried, what failed, and why.1736. **FINDINGS**: unique wins, universal floor and ceiling, mismatch analysis, and load-bearing insight.1747. **CAVEATS**: single-run signal, fixture limits, stack-level confounds, schema migration cost, reranker/runtime confounds.1758. **RECOMMENDATIONS**: Tier 1 apply now, Tier 2 validate first, Tier 3 future.1769. **REPRODUCIBILITY**: exact replay commands plus expected wall-clock.17710. **CROSS-LINKS**: sibling MCP benchmarks, authoritative spec packet, follow-on packets.178179For an empty section, retain its header and write: `Not applicable to this bench. Reason: ...`180181### Report Style182183- Match sk-doc reference quality and frontmatter.184- Use H2 numbered headers with ALL CAPS section names.185- Use tables for data and fenced code blocks for verbatim commands.186- Keep slugs and anchors stable across revisions.187- Do not paste the full spec packet decision trail into the report.188- Do not compare numeric results across different MCP stacks as if equivalent.189190### Headline Pattern191192Use this shape near the top of the report:193194```markdown195# <topic> bake-off: <date>196197> Headline: `<winner>` + <pipeline-config> is the production default for `<MCP-stack>`.198> <primary-metric> <numeric-result>, <secondary-metric> <numeric-result>. Closes packet <packet-id>.199```200201If the load-bearing insight differs from the winner, state it in Sections 1 and 6.202203---204205## 6. DATE AND NAMING CONVENTION206207### Folder Names208209One grammar governs every benchmark family. A run folder is named:210211```text212<YYYY-MM-DD>--<subject>--<variant>/213```214215- Fields are separated by a double hyphen. Words inside a field use a single hyphen.216- Lowercase ASCII only. No underscores, no dots, no capitals.217- The date is the **execution** date, not the authoring or publication date. Publication date belongs in218 the report body, where a reader can see both.219- `<subject>` names the corpus the run measured: `manual-testing-playbook`, `mcp-retrieval`,220 `model-eval`, `behavior`.221- `<variant>` names what most distinguishes the run. For a **feature-scoped** run — one validating a222 single feature or scenario group rather than comparing executors — name the feature (e.g. `goal-hook`),223 not the model; the model/executor is recorded inside the report (`model`/`providerModel` fields, the224 byline, the command shape), so the folder stays legible when the same feature is later re-validated on225 another model. For an **executor-scoped** run — one whose point is which model and effort produced it —226 name the executor identity as `<runtime>-<model>-<effort>`; use `model-comparison` when it compares227 several, or a short topic slug when no model applies.228- Model versions flatten their dots: `glm-5.2` becomes `glm-5-2`, `kimi-2.7` becomes `kimi-2-7`. A dot229 is rejected by the run-label validator, so this is a hard requirement rather than a style preference.230- Two runs of the same subject and variant on one day disambiguate with a trailing topic field.231232Examples:233234```text2352026-07-29--manual-testing-playbook--goal-hook/ # feature-scoped: names the feature, model recorded inside2362026-07-26--manual-testing-playbook--cursor-grok-4-5-high-fast/ # executor-scoped: names the model+effort that ran it2372026-07-26--manual-testing-playbook--model-comparison/2382026-05-18--mcp-retrieval--bge-confirmation/239```240241**One exception: `baseline/`.** The frozen comparison anchor keeps its name. It is not a run, it is the242before-snapshot every run is measured against, and the archiver refuses it as a run label for exactly243that reason. Dating it would make it look like one run among many.244245Authored scenario contracts are not run folders. Behavior families keep their246`<PREFIX>-NNN-<slug>.md` scenario identifiers, which name a scenario rather than a directory. Any run247those families produce follows the grammar above.248249### In-Document Dates250251Use readable long-form dates in prose, such as `May 18, 2026`.252253### Re-Runs254255For a re-run confirming the same headline, update the existing report with `Re-run YYYY-MM-DD`; do not create a folder.256257### Retirement258259Keep retired benchmarks. Mark `README.md` as `RETIRED` with date; add report date, reason, and replacement if any.260261### Renamed or Renumbered Spec Packets262263Do not rename the dated folder. Update `source.md`, report and README cross-links, then record old slug, new slug, and date in `source.md`.264265---266267## 7. RULES: AUTHORITY AND GATES268269### Authority Hierarchy270271When documents disagree:2722731. Source spec packet `decision-record.md` and `implementation-summary.md` are authoritative.2742. Skill-local `benchmark-report.md` is the curated operator-facing summary.2753. Copied CSV and JSONL files preserve the source packet evidence.2764. `source.md` is navigation, not a duplicate audit trail.277278### ✅ ALWAYS2792801. Read the source packet decision record, implementation summary, and benchmark evidence before writing.2812. Use the benchmark execution date for `benchmark-<YYYY-MM-DD>/`.2823. Keep `source.md` lean and navigational.2834. Include caveats for single-run signal, fixture limits, stack mismatch, schema migration cost, and reranker/runtime confounds when present.2845. Preserve retired benchmark folders.2856. Validate authored markdown before delivery.286287### ⛔ NEVER2882891. Never promote an in-flight benchmark as a final skill-local record.2902. Never compare numeric results across different MCP stacks as if equivalent.2913. Never paste the full spec packet audit trail into `benchmark-report.md`.2924. Never create a new dated folder for a confirming re-run.2935. Never name the folder by authoring date, source packet slug, or candidate name.2946. Never leave template placeholders in shipped benchmark files.2957. Never add packet-local `graph-metadata.json`.296297### ⚠️ ESCALATE IF2982991. The source packet has no accepted decision record or stable benchmark headline.3002. The target skill lacks `mcp-server/` or an appropriate measurable MCP surface.3013. Source artifacts are missing, non-replayable, or internally contradictory.3024. The benchmark spans multiple stacks and the user wants a single comparative verdict.3035. Validation fails on required markdown structure after local fixes.304305---306307## 8. SUCCESS CRITERIA308309- The consuming skill has a dated `mcp-server/benchmarks/benchmark-<YYYY-MM-DD>/` folder.310- `benchmark-report.md` uses the ten-section structure and includes winner or status, aggregate table, methodology, candidate profiles, findings, caveats, recommendations, replay commands, and cross-links.311- `source.md` points to the authoritative spec packet and maps reader questions to source files.312- Raw artifacts are copied or intentionally omitted with a documented reason.313- The benchmarks README index links the new folder and source packet.314- Shared sk-doc validation passes for authored markdown, or any remaining issue is escalated with exact command output.315316---317318## 9. BEHAVIOR BENCHMARK PACKAGES319320Behavior benchmarks are run contracts, not numeric MCP bake-offs. They specify321whether a realistic prompt makes an executor model dispatch the mode's LEAF agent,322ask one consolidated setup question when under-specified, respect invariants, and323finish relative to a Claude reference leg.324325The authoring path, matrix rules, and naming live in326[`references/behavior-benchmark/behavior-benchmark-guide.md`](references/behavior-benchmark/behavior-benchmark-guide.md).327The normative five-dimension rubric, terminal buckets, budget formula, entry-surface328and clarity enums, and ID prefixes live only in329[`../../system-deep-loop/shared/behavior-benchmark/framework.md`](../../system-deep-loop/shared/behavior-benchmark/framework.md).330Instantiate that framework; do not redefine it.331332### Package Shape333334Author this shape inside the owning deep-loop mode-packet:335336```text337<mode>/behavior-benchmark/338├── behavior-benchmark.md # package index: scenario table + axis coverage339├── scenarios/340│ └── <PREFIX>-NNN-<slug>.md # one machine-contract file per scenario341└── baselines/342 └── claude-baseline.md # per-scenario Claude-leg reference checkpoints343```344345Do not ship fixtures, lane configs, transcripts, result JSON, or scorecards here.346The executing packet provisions fixtures and holds proof; this package is the contract.347348### Templates, Workflow, and Naming349350Load the [behavior-benchmark guide](references/behavior-benchmark/behavior-benchmark-guide.md)351and shared framework before authoring. The guide owns templates, sequence, matrix,352and naming; execution and evidence stay in the executing packet.353354### ✅ ALWAYS / ⛔ NEVER (behavior benchmark)355356- **ALWAYS** keep the shared `framework.md` as the single source for rubric, buckets, budget formula, and enums; the package instantiates, it does not redefine.357- **ALWAYS** keep the index SCENARIO TABLE and the scenario files in exact sync.358- **ALWAYS** ship uncaptured baseline cells as `pending` / `not_captured` rather than inventing values.359- **ALWAYS** count an expected marker, target or postcondition only after a run that skips the behavior has failed it (a negative control), and name that run in the scenario's Failure modes.360- **NEVER** ship fixtures, transcripts, or result JSONs inside the package — they belong to the executing spec-packet phase.361- **NEVER** give a shipped scenario file frontmatter or a `## OVERVIEW` heading; it opens at the `# <PREFIX>-NNN` H1 (the scenario template's own frontmatter, usage comment, and Overview are stripped on copy).362- **NEVER** add a scenario whose `id` disagrees with its filename or its index-table row.363364### Success Criteria (behavior benchmark)365366- The owning mode-packet has a `behavior-benchmark/` package with an index, one scenario file per table row, and a baseline.367- Every scenario's first json block parses, its `id` matches its filename and index row, and its axis values agree across the table, scenario, and baseline.368- The index AXIS COVERAGE section reports per-surface and per-clarity counts and names any axis intentionally left out with its reason.369- The baseline ships with real values or with `pending` / `not_captured` cells, never invented ones.370- Shared sk-doc validation passes for the index and baseline, or any remaining issue is escalated with exact command output.371372---373374## 10. INTEGRATION POINTS375376This packet authors inputs and indexes.377378Filesystem outputs, template directories, and filenames use lowercase kebab-case. Router and JSON family keys remain snake_case machine identifiers; the router converts them to resource paths.379380`/create:benchmark` drives `mcp_promotion` (§3-8) end-to-end. Author behavior directly from its section; no command scaffolds it.381382---383384## 11. REFERENCES AND RELATED RESOURCES385386**Within this packet** — family guides and the overflow route-map; the fillable templates are mapped in each family section above:387388- [`references/shared/README.md`](references/shared/README.md) — overflow route-map (case studies, worked example, pitfalls).389- [`references/behavior-benchmark/behavior-benchmark-guide.md`](references/behavior-benchmark/behavior-benchmark-guide.md) — behavior package authoring path (§9).390391**Lane-owned contracts** — cross-link, never restate:392393- [`behavior-benchmark/framework.md`](../../system-deep-loop/shared/behavior-benchmark/framework.md) — behavior rubric, buckets, budget formula, runner.394395**Shared sk-doc backbone**: [`../shared/scripts/validate_document.py`](../shared/scripts/validate_document.py) — every authored `.md` must pass with 0 issues; [`../shared/references/`](../shared/references/) — cross-document standards.