GhostlyGawd Eval Capture
The corpus is the only proof that harness vN+1 beats vN. Feed it.
When to capture (any one suffices)
- The task shape recurs (third "write a migration" = the migration eval)
- The user corrected you before accepting — the correction IS the rubric
- The result encodes taste the user articulated ("never bury the lede")
Skip: one-offs, secrets/proprietary data (sanitize or skip), pure lookups.
Case anatomy: evals/corpus//
task.md— the request, verbatim enough to be re-runnable, with any needed fixtures alongside.- exactly one grader:
check.py— objective: gets the output dir as argv[1], exits 0/1. Prefer this whenever checkable; it's free to run forever.rubric.md— subjective: 3-6 falsifiable criteria for the critic agent. "Subject line under 60 chars" yes; "feels professional" no.
meta.json—{"date", "category", "source_session", "origin": "..."}
Rules
- A correction-born case MUST encode the corrected behavior as the expected one — that's the regression you're guarding against.
- Keep cases hermetic: no network, no machine-specific paths, fixtures < 50KB.
- Tag
categoryto match prediction categories so calibration stats and eval results join up in /meta-retro. - Leakage check (behavioral evals). An eval that claims to test "the agent
used system X" is confounded if X's answer is pre-taught. Before finalizing,
grep the loaded
skills/,hooks/, andmemory/decisions/for the expected answer; if an artifact that loads for this task already teaches it, the eval passes WITHOUT exercising the capability — choose a fixture whose solution is NOT pre-taught. Writing a fidelity constraint is not satisfying it: verify the chosen fixture against it. (session 0d0fe086, 2026-06-22: a cp1252 recall eval used the single most pre-taught fix in the repo.)
Verify in-session before committing: run /run-evals (fresh subagent performs the task, no headless — ADR 0003) and confirm it passes today. A case that fails on day one is either a real finding (file it) or a bad rubric (fix it).