# Report

> Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for "did this optimization actually work, and by how much".

- Skill: `skillberry-ai/report` (Agent Skill, multi-file: 9 files)
- Install (CLI): `npx skillmds@latest add skillberry-ai/report`
- Raw SKILL.md: https://api.skillmd.com/api/skills/skillberry-ai/report/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: skillberry-ai (https://skillmd.com/u/skillberry-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/skillberry-ai/report

---


# report — did it work, and by how much?

The result of a run is not "we made edits" — it is a defensible answer to *did this
actually work, and by how much.* report lays three numbers side by side: where the
seed started (val), where the best candidate landed (val), and the single **held-out
test** number that counts. It is what a human reads to decide whether to ship.

Runs standalone as `/cap-evolve:report`, or headlessly as the last step of
`cap-evolve run` — same `scripts/run.py` either way. report is a phase SCRIPT, not a
`cap-evolve` subcommand; invoke the script.

## How to read the three numbers
The honest reading is always **test vs baseline**, with val as a sanity check in
between. `scripts/run.py` produces the numbers; this is the judgment you add on top:

- **test ≈ baseline** → no real gain. The val improvement was overfitting or noise
  the gate let through. Do not ship; tighten `gate_k_se` or add trials.
- **test ≫ baseline** → genuine improvement on data the optimizer never saw. Ship.
- **val ≫ test** → the classic overfit signature: the optimizer learned the val set,
  not the capability. The reported val→test gap *is* the overfitting, quantified.
- **pass^k far below pass^1** → the gain is *fragile* across trials; the agent
  sometimes succeeds but not reliably. A high mean with low pass^k is not a
  dependable win (τ-bench's point).

Every number is rendered with its stderr when one was measured, because "0.71" and
"0.71 ± 0.08" support very different decisions. A gain smaller than the noise floor
is not a result — say so plainly rather than quoting the point estimate alone.

## Output contract
`scripts/run.py` owns both artifacts and writes them deterministically from the run
dir — do not hand-write or paraphrase them, or two runs stop being comparable.

`report.md` is exactly this skeleton (bracketed lines appear only when they apply):

```
# cap-evolve run report — <run_id>

[> **NOT FINALIZED** — no held-out test number. Run the finalize phase first; …]
[> **No holdout** (train == val == test). The test number below is a *fit* metric, …]

- Best candidate: `<best_id>`
- Baseline val: <r> ± <se>
- Best val: <r> ± <se>
- **Held-out test (optimized skills): <r> ± <se>**  (pass^1=…, pass^k=…)
[- Held-out test (baseline `<baseline_id>` skills): <r> ± <se>]
[- **Test improvement (optimized − baseline): <+Δ>**]
[- Val→test gap: <+Δ> — selection optimism on val; this gap IS the overfitting]
- Iterations: <n>
[- Optimized for: <consuming model> (tier <t>)]

[<sealed note — omitted entirely when the run was never finalized>]
```

stdout is **exactly one** JSON object — `cap-evolve run` echoes it as its own result, so
it is the machine contract for everything downstream. Keys (null for whatever the run
dir does not carry; an unfinalized run is `finalized: false` with null test numbers):
`run_dir`, `best_id`, `finalized` bool, `no_holdout` bool, `baseline_val`,
`baseline_val_stderr`, `best_val`, `test_reward`, `test_stderr`,
`test_baseline_reward`, `test_baseline_stderr`, `test_delta`, `test_pass_k` (k→float),
`val_test_gap`, `iterations`, `target_profile` (`{model,tier,resolution_note}`), plus
`dashboard` / `dashboard_server` / `*_error` on the paths that produce them.

## How to run
```
python scripts/run.py --run-dir .capevolve/run_XXXX            # JSON + report.md + dashboard.html
python scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report
python scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard
```
`--dashboard-mode` / `--dashboard-port` / `--dashboard-url` are orchestrator-supplied.
Re-reporting by hand after `cap-evolve run` needs `--dashboard-url <the URL run printed>`
or `--no-dashboard` — launching is deliberately not idempotent, so a bare re-run spawns
a second server on a second port and reports that one instead.

## The dashboard (`dashboard.html`)
One self-contained static file (inline CSS/JS/SVG, no CDN, no server, no network — opens
from `file://`); the single shareable artifact. Eight panels reduced from the event log
plus baseline/final, rollouts and the git store; every value passes a recursive secret
redactor so a shared dashboard leaks no API keys; optional panels degrade silently when
per-task data, diffs, or finalize are missing. `--terminal` renders the same reduction
as an ANSI chart for in-chat progress.

## References
- `references/concepts.md` — why the val→test gap measures overfitting, pass^k
  fragility, reporting uncertainty, with sources. Load when writing the human
  interpretation and you want the reasoning or a citation.
- `references/dashboard.md` — the reduced graph + summary schema, per-panel field
  sources, `--terminal`, redaction, degradation matrix. Load when changing or
  debugging the dashboard; not needed to run the phase.