Eval Loop

Eval Loop

Generate, judge independently, revise, re-judge, approve. The separation between generator and judge is structural, and it is the highest-leverage architectural choice in this repo.

When to use

• Any artifact with an eval spec is produced
• Before anything reaches an external surface
• Spot-checking a batch of agent output
• A quality complaint needs diagnosing

Inputs

• Reads: workspace/evals/specs/<type>.md, the draft
• Needs from user: the artifact type, and a title for the log entry

Workflow

1. Understand why separation is not optional

A context that produced a draft knows what it meant. Asked to grade it, it reads its own intent rather than the text, and passes almost everything.

Moving the same rubric into a genuinely separate context that receives only the artifact and the spec drops pass rates substantially, and what it starts catching is what a reader would have caught.

A different prompt in the same context is not independence. Independence is a separate context window that never saw the reasoning.

2. Run the loop

GENERATE   the draft, in its own context
     ↓
JUDGE      separate context. Receives ONLY the artifact + the spec.
           Runs gates, then dimensions, then verdict.
           Never revises. Returns verdict + guidance + a log line.
     ↓
     ├── PASS   → approve, return with the receipt
     ├── REVISE → REVISE (separate context again)
     │            Receives draft + verdict. Fixes the named weaknesses
     │            surgically. Never rewrites what already works.
     │            Never approves itself. → back to JUDGE
     └── FAIL   → gate trip is not a revision problem. Back to GENERATE
                  with the gate named, or escalate

Cap at three cycles. A loop that has not passed in three is not converging, and the usual cause is that the generator is not reading the spec or the spec is wrong.

3. Keep the roles strictly separated

The reviser rule matters more than it looks. A reviser allowed to declare itself finished will, and the loop collapses back into self-grading.

4. Revise surgically

The reviser fixes what the verdict named and leaves the rest alone. A full rewrite discards whatever was already working and usually introduces new problems in dimensions that were fine, which shows up as a score that moves sideways across cycles.

5. Log every cycle, especially the failures

{"ts":"<iso>", "type":"<artifact>", "title":"<title>", "cycle":n,
 "gates":"pass|<gate tripped>", "score":n, "dims":{...},
 "verdict":"PASS|REVISE|FAIL", "spec_version":"<v>"}

Append-only. One line per cycle, including the ones that failed.

The failed cycles are the trend data. A log containing only passes is a marketing document. What tells you where the system is weak is which gates trip and which dimensions repeatedly score low.

6. Return the receipt

eval-clean: <type> v<version>, score <n>, cycle <n>

If a genuinely independent judge was unavailable, say so and label it honestly:

eval-clean (inline, no independent judge)

Never claim a fuller eval than the one you ran. A missing receipt means the gate did not run, and it should never be read as a silent pass.

Output

• Writes: appends to workspace/evals/log/eval-log.jsonl, plus the approved artifact
• Uses: templates/eval-log-schema.csv
• Prints: the verdict per cycle, the final score, the receipt

Rules & quality bar

• The judge runs in a separate context and receives only the artifact plus the spec
• The judge never edits. The reviser never approves. Both directions
• Gates before scores, and a gate trip returns to the generator rather than to the reviser
• Revise surgically. No full rewrites
• Cap at three cycles, then escalate with the score trajectory
• Log every cycle including failures. 100% log completeness or the trend is fiction
• The receipt names the spec version
• Label an inline check as inline. Never overclaim

Related skills

• Requires: eval-spec-authoring
• Pairs with: eval-calibration to keep the spec honest
• Feeds: agent-observability
• See also: docs/field-notes.md story 9