Generator/Evaluator Pattern
A reusable design pattern for honest evaluation of agent-produced artifacts. Separates the agent that creates work (generator) from the agent that evaluates it (evaluator), because agents consistently praise their own work.
"When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre."
— Anthropic, "Harness Design for Long-Running Application Development"
Dual-use: consumer skills read this skill's references/ files — the canonical homes for scoring, contracts, protocol, recovery, and findings — while invoking /aep-gen-eval directly runs a full gen/eval loop on any artifact.
How Other Skills Use This
| Skill |
What it uses |
Reference files |
/aep-build Phase 5 |
Scoring framework + eval protocol |
scoring-framework.md, eval-protocol.md, recovery-ladder.md |
/aep-launch |
Dimension presets for brainstorming |
scoring-framework.md (presets section) |
/aep-validate |
Agent prompts + findings format |
agent-contracts.md, findings-format.md, scoring-framework.md |
/aep-build, /aep-dispatch, /aep-wrap, /aep-autopilot |
Failure taxonomy, verification tiers/recipes, accounting |
verification-economics.md |
The Core Principle
Generator and evaluator are separate agents. Self-evaluation produces inflated scores and rationalized problems (the finding quoted above); an independent evaluator catches what the generator is blind to.
Scaling up: generator/evaluator is the canonical instance of adversarial
verification. When one task produces many findings/claims that each need an
independent check, /aep-workflow generalizes this to a fan-out of N
verifiers/refuters — reusing this skill's scoring framework and findings format
per finding.
Reference Files
These files are the canonical homes for the gen/eval contracts every consumer (/aep-build, /aep-validate, /aep-launch) points at. Read the one the branch needs; each file is self-contained.
| File |
Contents |
When to read |
references/scoring-framework.md |
Dimension definitions (1-5 scale), hard failure thresholds, dimension presets (UI, API, security, data, mixed), few-shot examples, anti-patterns |
Setting up evaluation criteria, scoring work, calibrating evaluators |
references/agent-contracts.md |
Generator/evaluator role separation, prompt templates (generator, evaluator, protocol checker), context assembly rules |
Spawning evaluation agents, assembling prompts |
references/eval-protocol.md |
Eval request/response format, verification JSON schema, the eval loop (request → response → fix → re-evaluate), execution contexts (Task subagent, codex exec, tmux, workflow), the needs-human gate record |
Running the evaluation loop, tracking verification state |
references/recovery-ladder.md |
Two-round recovery ladder (same fix → re-ground → human gate proposing fresh generator / decompose) for a blocking finding that survives round 1 |
A blocking finding is still open after round 1 |
references/findings-format.md |
Severity categorization (blocking/important/minor), deduplication protocol, presentation format, changelog entry format |
Consolidating findings from multiple agents, presenting results |
references/verification-economics.md |
Validator placement matrix, failure taxonomy + classification authority, environment preflight, verification tiers + two-point derivation, verification recipe, accounting sensors, tamper-evident evidence classes |
Routing a FAIL, deriving verification depth, pricing/recording verification spend |
Standalone Usage
When invoked directly, this skill runs a gen/eval loop on any artifact.
Step 1: Identify the artifact
What is being evaluated? Options:
- A document (product context, architecture, design doc)
- Code changes (implementation, PR diff)
- An OpenSpec change (proposal, design, specs, tasks)
- A structured file (YAML, JSON config, migration plan)
Step 2: Choose execution mode
| Mode |
Agents |
When to use |
| Parallel |
Generator + Evaluator spawned simultaneously |
Documents, designs, product context — agents work independently |
| Sequential |
Generator first, then Evaluator reads generator's work |
Code review — evaluator needs to see the implementation |
| Loop |
Generator → Evaluator → fix → re-evaluate (max 2 rounds) |
Active development — generator can fix issues between rounds |
Step 3: Configure dimensions
Read references/scoring-framework.md and select the preset that matches the artifact (UI-heavy, API-only, security-sensitive, data pipeline, mixed/full-stack, product/design, or document), or define custom dimensions. The preset tables, hard-failure thresholds, and few-shot calibration all live in that file.
Step 4: Spawn agents
Read references/agent-contracts.md for prompt templates. Customize the templates with:
- The artifact content
- The technical constraints
- The verification checklist (what the evaluator should check against the codebase)
Step 5: Process results
Read references/findings-format.md to consolidate, categorize, and present findings, then converge to one of two checkable end states:
- Fixes applied and re-scored: the generator applies the fixes and the artifact passes a fresh evaluation round with no blocking findings remaining. The cap is two rounds; a
blocking finding still open after round 2 escalates per references/recovery-ladder.md.
- Findings handed off: a consolidated findings file is written to a named path (e.g.
<artifact-dir>/eval-findings.md) for a downstream owner to act on.
Design Decisions
Gen/eval is packaged as its own invocable skill that doubles as a reference library, rather than folded into /aep-validate or /aep-launch. Rationale: docs/decisions/gen-eval-rationale.md.
Next Step
After running gen/eval, proceed based on what was evaluated:
- Product context →
/aep-dispatch
- Design artifacts →
/aep-launch
- Code → create PR or continue
/aep-build
- Documents → publish or share
1---2name: aep-gen-eval3description: Defines the reusable generator/evaluator pattern: scoring, agent contracts, eval protocol. Validating one specific artifact is /aep-validate.4---56# Generator/Evaluator Pattern78A reusable design pattern for honest evaluation of agent-produced artifacts. Separates the agent that creates work (generator) from the agent that evaluates it (evaluator), because agents consistently praise their own work.910> "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre."11> — Anthropic, ["Harness Design for Long-Running Application Development"](https://www.anthropic.com/engineering/harness-design-long-running-apps)1213Dual-use: consumer skills read this skill's `references/` files — the canonical homes for scoring, contracts, protocol, recovery, and findings — while invoking `/aep-gen-eval` directly runs a full gen/eval loop on any artifact.1415---1617## How Other Skills Use This1819| Skill | What it uses | Reference files |20| ------------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------ |21| `/aep-build` Phase 5 | Scoring framework + eval protocol | `scoring-framework.md`, `eval-protocol.md`, `recovery-ladder.md` |22| `/aep-launch` | Dimension presets for brainstorming | `scoring-framework.md` (presets section) |23| `/aep-validate` | Agent prompts + findings format | `agent-contracts.md`, `findings-format.md`, `scoring-framework.md` |24| `/aep-build`, `/aep-dispatch`, `/aep-wrap`, `/aep-autopilot` | Failure taxonomy, verification tiers/recipes, accounting | `verification-economics.md` |2526---2728## The Core Principle2930**Generator and evaluator are separate agents.** Self-evaluation produces inflated scores and rationalized problems (the finding quoted above); an independent evaluator catches what the generator is blind to.3132> **Scaling up:** generator/evaluator is the canonical instance of _adversarial33> verification_. When one task produces many findings/claims that each need an34> independent check, `/aep-workflow` generalizes this to a fan-out of N35> verifiers/refuters — reusing this skill's scoring framework and findings format36> per finding.3738---3940## Reference Files4142These files are the canonical homes for the gen/eval contracts every consumer (`/aep-build`, `/aep-validate`, `/aep-launch`) points at. Read the one the branch needs; each file is self-contained.4344| File | Contents | When to read |45| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |46| [`references/scoring-framework.md`](references/scoring-framework.md) | Dimension definitions (1-5 scale), hard failure thresholds, dimension presets (UI, API, security, data, mixed), few-shot examples, anti-patterns | Setting up evaluation criteria, scoring work, calibrating evaluators |47| [`references/agent-contracts.md`](references/agent-contracts.md) | Generator/evaluator role separation, prompt templates (generator, evaluator, protocol checker), context assembly rules | Spawning evaluation agents, assembling prompts |48| [`references/eval-protocol.md`](references/eval-protocol.md) | Eval request/response format, verification JSON schema, the eval loop (request → response → fix → re-evaluate), execution contexts (Task subagent, codex exec, tmux, workflow), the needs-human gate record | Running the evaluation loop, tracking verification state |49| [`references/recovery-ladder.md`](references/recovery-ladder.md) | Two-round recovery ladder (same fix → re-ground → human gate proposing fresh generator / decompose) for a `blocking` finding that survives round 1 | A `blocking` finding is still open after round 1 |50| [`references/findings-format.md`](references/findings-format.md) | Severity categorization (blocking/important/minor), deduplication protocol, presentation format, changelog entry format | Consolidating findings from multiple agents, presenting results |51| [`references/verification-economics.md`](references/verification-economics.md) | Validator placement matrix, failure taxonomy + classification authority, environment preflight, verification tiers + two-point derivation, verification recipe, accounting sensors, tamper-evident evidence classes | Routing a FAIL, deriving verification depth, pricing/recording verification spend |5253---5455## Standalone Usage5657When invoked directly, this skill runs a gen/eval loop on any artifact.5859### Step 1: Identify the artifact6061What is being evaluated? Options:6263- A document (product context, architecture, design doc)64- Code changes (implementation, PR diff)65- An OpenSpec change (proposal, design, specs, tasks)66- A structured file (YAML, JSON config, migration plan)6768### Step 2: Choose execution mode6970| Mode | Agents | When to use |71| -------------- | ------------------------------------------------------ | --------------------------------------------------------------- |72| **Parallel** | Generator + Evaluator spawned simultaneously | Documents, designs, product context — agents work independently |73| **Sequential** | Generator first, then Evaluator reads generator's work | Code review — evaluator needs to see the implementation |74| **Loop** | Generator → Evaluator → fix → re-evaluate (max 2 rounds) | Active development — generator can fix issues between rounds |7576### Step 3: Configure dimensions7778Read `references/scoring-framework.md` and select the preset that matches the artifact (UI-heavy, API-only, security-sensitive, data pipeline, mixed/full-stack, product/design, or document), or define custom dimensions. The preset tables, hard-failure thresholds, and few-shot calibration all live in that file.7980### Step 4: Spawn agents8182Read `references/agent-contracts.md` for prompt templates. Customize the templates with:8384- The artifact content85- The technical constraints86- The verification checklist (what the evaluator should check against the codebase)8788### Step 5: Process results8990Read `references/findings-format.md` to consolidate, categorize, and present findings, then converge to one of two checkable end states:9192- **Fixes applied and re-scored:** the generator applies the fixes and the artifact passes a fresh evaluation round with no blocking findings remaining. The cap is two rounds; a `blocking` finding still open after round 2 escalates per `references/recovery-ladder.md`.93- **Findings handed off:** a consolidated findings file is written to a named path (e.g. `<artifact-dir>/eval-findings.md`) for a downstream owner to act on.9495---9697## Design Decisions9899Gen/eval is packaged as its own invocable skill that doubles as a reference library, rather than folded into `/aep-validate` or `/aep-launch`. Rationale: [`docs/decisions/gen-eval-rationale.md`](https://github.com/memorysaver/agentic-engineering-patterns/blob/main/docs/decisions/gen-eval-rationale.md).100101---102103## Next Step104105After running gen/eval, proceed based on what was evaluated:106107- Product context → `/aep-dispatch`108- Design artifacts → `/aep-launch`109- Code → create PR or continue `/aep-build`110- Documents → publish or share
memorysaver