# Eval Spec Authoring

> Use before building anything whose output needs a quality bar, or when the user says "write a rubric", "eval spec", "how do we score this", "define what good looks like", "quality bar", "grading criteria", "our AI output is inconsistent". Encodes judgment as hard gates plus weighted dimensions with 1-to-5 anchors and a frozen calibration set. Writes workspace/evals/specs/<type>.md. Writes the rubric. Not for running it against an artifact (`eval-loop`) or retesting it against its anchors (`eval-calibration`).

- Skill: `guerrilla2799/eval-spec-authoring` (Agent Skill)
- Install (CLI): `npx skillmds@latest add guerrilla2799/eval-spec-authoring`
- Raw SKILL.md: https://api.skillmd.com/api/skills/guerrilla2799/eval-spec-authoring/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: guerrilla2799 (https://skillmd.com/u/guerrilla2799)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/guerrilla2799/eval-spec-authoring

---


# Eval Spec Authoring

Write the rubric before the thing it judges. A spec encodes one team's judgment, which is why it cannot be borrowed from anybody else.

## When to use
- Before building an agent that produces judged output
- Output quality is inconsistent and nobody can say why
- Adding a new artifact type to an existing eval system
- Reviewers disagree with each other

## Inputs
- Needs from user: 5 to 10 real examples of the artifact spanning excellent to bad, and roughly an hour with whoever holds the quality bar
- Reads: `workspace/agents/context/voice.md` and the rest of the portfolio

**The real examples are required.** A spec written without a corpus scores against imagination, and it will pass things a reader would reject.

## Workflow

### 1. Structure it in three stages

Order is load-bearing. Gates first, because a numeric average will happily pass work that trips a non-negotiable.

```
STAGE 0   Hard gates          pass / fail, any trip = FAIL regardless of score
STAGE 1   Weighted dimensions 1 to 5 on anchors, weighted, normalized to 0 to 100
STAGE 2   Verdict             PASS / REVISE / FAIL, plus revision guidance
```

### 2. Write the hard gates

Non-negotiables. Things that make the artifact unusable no matter how good the rest is.

```
GATE 1  <banned vocabulary or structure present>
GATE 2  <a factual claim with no source>
GATE 3  <wrong audience for the stated segment>
GATE 4  <artifact-specific killer, e.g. a feature list in a cold email>
```

Keep it to three to six. A long gate list becomes a scoring dimension by accident, and gates should be rare and decisive.

**Every gate must be checkable by reading the artifact.** A gate requiring outside knowledge cannot be applied consistently.

### 3. Write the dimensions with real anchors

Six to ten dimensions, each scored 1 to 5, each weighted. Weights sum to 100.

The anchors are the entire value of the spec. Without them, a 3 means whatever the reader felt.

```
DIMENSION: <name>              Weight: n%
  5  <what a 5 looks like, concretely, referencing a real example>
  4  <the common near-miss>
  3  <acceptable but unremarkable, and what makes it so>
  2  <the specific flaw>
  1  <the failure>
```

**Write 3 first.** It is the hardest and the most used. Then 5 and 1 as the poles, then fill 4 and 2 as the near-misses. Anchors written top-down tend to describe an ideal and a strawman with nothing usable in between.

### 4. Set the verdict thresholds

```
PASS     ≥ 80, no dimension below 3, no gate tripped
REVISE   65 to 79, or any single dimension below 3
FAIL     < 65, or any gate tripped
```

**Calibrate against the real distribution.** If your first-pass rate lands above 85%, the spec is too easy. If it lands under 25%, it is too harsh or the generator is not reading it. The healthy band is 40 to 65%.

### 5. Freeze a calibration set

Pick 5 to 8 of the real examples, assign each an expected score band by hand, and freeze them.

```
example-01.md   expected band: 85 to 95   (strong)
example-02.md   expected band: 70 to 79   (revise)
example-03.md   expected band: 40 to 55   (fail)
```

This set is what makes drift detectable. Any future edit to the spec gets regraded against it. Without a frozen set, a spec can soften over months and everybody reads the rising pass rate as improvement. See `eval-calibration`.

### 6. Require revision guidance, not just a score

A verdict with no guidance produces a reviser that guesses. The judge must return, for each dimension under 4, what specifically would raise it.

## Output
- Writes: `workspace/evals/specs/<type>.md`, plus the frozen calibration set
- Uses: `templates/eval-spec-template.md`
- Prints: the gates, the weighted dimensions with anchors, the thresholds, and the calibration set with expected bands

## Rules & quality bar
- **Gates before scores.** A gate trip fails regardless of the number
- **Anchors are mandatory,** and 3 gets written first
- **Real corpus required.** No spec from imagination
- **Three to six gates.** More than that and they are dimensions
- **Every gate checkable from the artifact alone**
- **Weights sum to 100** and reflect what actually matters, not what is easy to score
- **A frozen calibration set,** or drift is undetectable
- **Revision guidance per weak dimension,** not just a score
- **Target a 40 to 65% first-pass rate.** Higher means the ruler is soft

## Related skills
- Hands off to: `eval-loop` to run it, `eval-calibration` to maintain it
- Required by: `agent-scoping` before building anything judged
- See also: `docs/metrics.md` on quality-plane metrics