Eval Calibration

Eval Calibration

Regrade the anchors. A rising pass rate is ambiguous until you have done this, and softening rubrics are invisible from the inside.

When to use

• Any time a spec is edited, including a wording change
• Pass rates have moved and nobody knows why
• The banned-vocabulary or voice reference changed, which changes every spec importing it
• Quarterly, as a standing check

Inputs

• Reads: workspace/evals/specs/<type>.md, the frozen calibration set, prior calibration runs
• Needs from user: nothing, if the frozen set exists. If it does not, go back to eval-spec-authoring

Workflow

1. Understand the failure this catches

A spec is edited a few times over some months. Each edit is small and reasonable, usually softening language that felt too harsh. Pass rates climb from about half to nearly nine in ten.

Everybody reads that as the generator improving. It is not necessarily. The ruler may have stretched, and the only way to tell is to regrade artifacts whose correct scores you already know.

2. Regrade the frozen set

Each anchor was assigned an expected band when the spec was written. Re-judge every one with the current spec, in a fresh context, and compare.

example-01   expected 85-95   scored 91   ✓ in band
example-02   expected 70-79   scored 84   ✗ +5 above band
example-03   expected 40-55   scored 62   ✗ +7 above band
...
Mean drift: +4.8 points
Direction:  softening

3. Read the drift

Direction matters as much as magnitude. Softening drift passes bad work. Hardening drift fails good work and burns revision cycles. Softening is far more common, because edits are usually made in response to something being failed that somebody wanted to pass.

4. Find the dimension responsible

Drift is rarely spread evenly. Compare per-dimension scores against the original grading and find the one or two that moved.

The usual culprits are a dimension whose anchors were softened, and a gate that was downgraded to a dimension. The second is the more damaging, because a non-negotiable became something a good average can absorb.

5. Decide explicitly, and re-freeze if you accept

Two legitimate outcomes.

Revert. The drift was accidental. Restore the prior anchor language, bump the version, and re-run to confirm.

Accept and re-freeze. The quality bar genuinely moved, on purpose. Re-assign the expected bands, note the reason and the date in the spec, and bump the version.

The unacceptable outcome is doing nothing and letting the drift stand undocumented, because then no future run has a reference point.

6. Separate the two explanations for a rising pass rate

Now you can answer the original question.

Output

• Writes: workspace/evals/calibration/<type>-<date>.md
• Prints: the per-anchor table with expected against actual, mean drift and direction, the dimension responsible, and the decision with its reason

Rules & quality bar

• Every spec edit triggers a calibration run. Including wording-only edits
• A change to a shared reference recalibrates every spec that imports it
• The frozen set stays frozen. Changing anchors to fit the spec defeats the entire exercise
• Report direction, not just magnitude
• Name the dimension responsible. "The spec drifted" is not actionable
• Revert or re-freeze deliberately. Undocumented drift is the failure mode
• Never read a rising pass rate as improvement without a calibration run

Related skills

• Requires: eval-spec-authoring for the frozen set
• Pairs with: eval-loop
• See also: docs/field-notes.md story 8