Eval Spec Authoring

Eval Spec Authoring

Write the rubric before the thing it judges. A spec encodes one team's judgment, which is why it cannot be borrowed from anybody else.

When to use

• Before building an agent that produces judged output
• Output quality is inconsistent and nobody can say why
• Adding a new artifact type to an existing eval system
• Reviewers disagree with each other

Inputs

• Needs from user: 5 to 10 real examples of the artifact spanning excellent to bad, and roughly an hour with whoever holds the quality bar
• Reads: workspace/agents/context/voice.md and the rest of the portfolio

The real examples are required. A spec written without a corpus scores against imagination, and it will pass things a reader would reject.

Workflow

1. Structure it in three stages

Order is load-bearing. Gates first, because a numeric average will happily pass work that trips a non-negotiable.

STAGE 0   Hard gates          pass / fail, any trip = FAIL regardless of score
STAGE 1   Weighted dimensions 1 to 5 on anchors, weighted, normalized to 0 to 100
STAGE 2   Verdict             PASS / REVISE / FAIL, plus revision guidance

2. Write the hard gates

Non-negotiables. Things that make the artifact unusable no matter how good the rest is.

GATE 1  <banned vocabulary or structure present>
GATE 2  <a factual claim with no source>
GATE 3  <wrong audience for the stated segment>
GATE 4  <artifact-specific killer, e.g. a feature list in a cold email>

Keep it to three to six. A long gate list becomes a scoring dimension by accident, and gates should be rare and decisive.

Every gate must be checkable by reading the artifact. A gate requiring outside knowledge cannot be applied consistently.

3. Write the dimensions with real anchors

Six to ten dimensions, each scored 1 to 5, each weighted. Weights sum to 100.

The anchors are the entire value of the spec. Without them, a 3 means whatever the reader felt.

DIMENSION: <name>              Weight: n%
  5  <what a 5 looks like, concretely, referencing a real example>
  4  <the common near-miss>
  3  <acceptable but unremarkable, and what makes it so>
  2  <the specific flaw>
  1  <the failure>

Write 3 first. It is the hardest and the most used. Then 5 and 1 as the poles, then fill 4 and 2 as the near-misses. Anchors written top-down tend to describe an ideal and a strawman with nothing usable in between.

4. Set the verdict thresholds

PASS     ≥ 80, no dimension below 3, no gate tripped
REVISE   65 to 79, or any single dimension below 3
FAIL     < 65, or any gate tripped

Calibrate against the real distribution. If your first-pass rate lands above 85%, the spec is too easy. If it lands under 25%, it is too harsh or the generator is not reading it. The healthy band is 40 to 65%.

5. Freeze a calibration set

Pick 5 to 8 of the real examples, assign each an expected score band by hand, and freeze them.

example-01.md   expected band: 85 to 95   (strong)
example-02.md   expected band: 70 to 79   (revise)
example-03.md   expected band: 40 to 55   (fail)

This set is what makes drift detectable. Any future edit to the spec gets regraded against it. Without a frozen set, a spec can soften over months and everybody reads the rising pass rate as improvement. See eval-calibration.

6. Require revision guidance, not just a score

A verdict with no guidance produces a reviser that guesses. The judge must return, for each dimension under 4, what specifically would raise it.

Output

• Writes: workspace/evals/specs/<type>.md, plus the frozen calibration set
• Uses: templates/eval-spec-template.md
• Prints: the gates, the weighted dimensions with anchors, the thresholds, and the calibration set with expected bands

Rules & quality bar

• Gates before scores. A gate trip fails regardless of the number
• Anchors are mandatory, and 3 gets written first
• Real corpus required. No spec from imagination
• Three to six gates. More than that and they are dimensions
• Every gate checkable from the artifact alone
• Weights sum to 100 and reflect what actually matters, not what is easy to score
• A frozen calibration set, or drift is undetectable
• Revision guidance per weak dimension, not just a score
• Target a 40 to 65% first-pass rate. Higher means the ruler is soft

Related skills

• Hands off to: eval-loop to run it, eval-calibration to maintain it
• Required by: agent-scoping before building anything judged
• See also: docs/metrics.md on quality-plane metrics