Agentic Evaluation
Use this skill when you are designing or implementing an evaluation loop that lets an agent assess and improve its own outputs through iteration — not when you are running a pre-existing test suite or doing a one-off review with no refinement cycle.
The core pattern is: Generate → Evaluate → Critique → Refine → Output, looping until a convergence condition is met or a max-iteration budget is exhausted.
Use this skill when
- Implementing a self-critique or reflection loop that feeds output quality back into generation.
- Building an evaluator-optimizer pipeline that separates generation from evaluation responsibilities.
- Designing LLM-as-judge scoring to compare or rank multiple candidate outputs.
- Adding rubric-based scoring with weighted dimensions to iterative generation.
- Setting iteration limits, convergence checks, or structured evaluation output contracts.
- The task requires measurable improvement across runs, not just a single-shot best effort.
Do not use this skill when
- You are running an existing test suite to verify code — use
verification-before-completion.
- You are diagnosing a specific failure or bug, not evaluating output quality — use
systematic-debugging.
- The goal is writing test coverage (unit tests, integration tests) — use
test-driven-development.
- You are reviewing a completed artifact once without a refinement loop (a single code review, an editorial pass, a PR check).
Routing boundary
| Situation |
Use this skill? |
Route instead |
| Designing a reflection loop with a score threshold and max iterations |
Yes |
— |
| Implementing LLM-as-judge comparison of two candidate outputs |
Yes |
— |
Running npm test to confirm a fix works |
No |
verification-before-completion |
| Tracing why a specific assertion fails |
No |
systematic-debugging |
| Writing Jest or pytest test coverage for a module |
No |
test-driven-development |
| Reviewing a PR diff once, no iteration |
No |
implementation-review |
Inputs to gather
Required before starting
- The skill or agent behavior to evaluate.
- The evaluation objective and target metric for the application's outputs.
Helpful if present
- Existing evaluation fixtures or application quality criteria to extend.
First move
- Identify the application behavior to evaluate and the desired outcome.
- Define the evaluation criteria and target metric.
- Open the relevant reference file based on the evaluation type.
Navigation
The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) and full Python examples are in references/patterns.md.
The implementation checklist — criteria, threshold, loop wiring, convergence, logging — is in assets/eval-checklist.md.
For a new implementation, start with the checklist to confirm your setup is complete, then use the patterns reference to choose and adapt an evaluation strategy.
Outputs
- Evaluation loop design with defined criteria, convergence check, and max iteration budget.
- Structured evaluation scores per iteration with input, output, and critique logged.
- Convergence or budget-exhaustion result confirming the loop terminated cleanly.
Workflow
- Name the application behavior under evaluation and the success criteria.
- Choose a strategy from
references/patterns.md: outcome-based, LLM-as-judge, or rubric-based.
- Complete the setup checklist in
assets/eval-checklist.md (criteria, threshold, max iterations, logging).
- Isolate generate → evaluate → optimize steps; keep the evaluator replaceable.
- Run the loop with a hard
max_iterations bound and a convergence check.
- Log full trajectories (input, output, score, critique) for every iteration.
- Stop on threshold met, no improvement, or budget exhaustion; report structured scores.
- Feed failures back into evaluator criteria, prompts, or optimization steps, then re-run.
Guardrails
- Always set a
max_iterations bound (3–5 is a safe default) before wiring up a refinement loop. Unbounded loops stall agents.
- Require structured output (JSON) from the evaluation step so the optimize step has a reliable signal to act on. Free-text critique is fragile.
- Add a convergence check: if the score does not improve between iterations, stop early. Oscillating loops that never converge waste budget.
- Log the full iteration trajectory. Evaluation loops are hard to debug post-hoc without a history of inputs, outputs, scores, and critiques.
- Define evaluation criteria before generating any output. Criteria added mid-loop drift and make scores incomparable across iterations.
- Keep the evaluate step isolated from the generate step. Blending them makes it hard to replace the evaluator or diagnose score instability.
- Handle evaluation parse failures gracefully — if the LLM judge returns malformed JSON, fall back to a safe default (treat as failing) rather than crashing the loop.
Validation
After implementing an evaluation loop, confirm:
Examples
- "Add a self-critique loop to my report-generation agent that retries up to three times if the rubric score is below 0.8."
- "Implement an evaluator-optimizer where a separate LLM judge scores code clarity and the generator rewrites until it passes."
- "Build a rubric-based evaluator with accuracy, completeness, and style dimensions that returns a weighted score as JSON."
Reference files
references/patterns.md — The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) with annotated Python examples and a best-practices table.
assets/eval-checklist.md — Implementation checklist: setup, loop wiring, convergence, logging, and safety items to confirm before shipping.
1---2name: agentic-eval3description: "Use when designing or implementing an evaluation loop for AI agent outputs — reflection loops, evaluator-optimizer pipelines, LLM-as-judge scoring, or rubric-based iteration. Not when running an existing test suite or reviewing a completed artifact without iterating."4license: GNU GPL v35metadata:6 source: https://skillmd.com/skills/matt-riley/agentic-eval7 version: 2.0.1 # x-release-please-version8 category: workflow9 audience: general-coding-agent10 maturity: draft11 kind: task12---1314# Agentic Evaluation1516Use this skill when you are designing or implementing an evaluation loop that lets an agent assess and improve its own outputs through iteration — not when you are running a pre-existing test suite or doing a one-off review with no refinement cycle.1718The core pattern is: **Generate → Evaluate → Critique → Refine → Output**, looping until a convergence condition is met or a max-iteration budget is exhausted.1920## Use this skill when2122- Implementing a self-critique or reflection loop that feeds output quality back into generation.23- Building an evaluator-optimizer pipeline that separates generation from evaluation responsibilities.24- Designing LLM-as-judge scoring to compare or rank multiple candidate outputs.25- Adding rubric-based scoring with weighted dimensions to iterative generation.26- Setting iteration limits, convergence checks, or structured evaluation output contracts.27- The task requires measurable improvement across runs, not just a single-shot best effort.2829## Do not use this skill when3031- You are running an existing test suite to verify code — use `verification-before-completion`.32- You are diagnosing a specific failure or bug, not evaluating output quality — use `systematic-debugging`.33- The goal is writing test coverage (unit tests, integration tests) — use `test-driven-development`.34- You are reviewing a completed artifact once without a refinement loop (a single code review, an editorial pass, a PR check).3536## Routing boundary3738| Situation | Use this skill? | Route instead |39| --- | --- | --- |40| Designing a reflection loop with a score threshold and max iterations | Yes | — |41| Implementing LLM-as-judge comparison of two candidate outputs | Yes | — |42| Running `npm test` to confirm a fix works | No | `verification-before-completion` |43| Tracing why a specific assertion fails | No | `systematic-debugging` |44| Writing Jest or pytest test coverage for a module | No | `test-driven-development` |45| Reviewing a PR diff once, no iteration | No | [`implementation-review`](../implementation-review/SKILL.md) |4647## Inputs to gather4849**Required before starting**5051- The skill or agent behavior to evaluate.52- The evaluation objective and target metric for the application's outputs.5354**Helpful if present**5556- Existing evaluation fixtures or application quality criteria to extend.5758## First move59601. Identify the application behavior to evaluate and the desired outcome.612. Define the evaluation criteria and target metric.623. Open the relevant reference file based on the evaluation type.6364## Navigation6566The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) and full Python examples are in `references/patterns.md`.6768The implementation checklist — criteria, threshold, loop wiring, convergence, logging — is in `assets/eval-checklist.md`.6970For a new implementation, start with the checklist to confirm your setup is complete, then use the patterns reference to choose and adapt an evaluation strategy.7172## Outputs7374- Evaluation loop design with defined criteria, convergence check, and max iteration budget.75- Structured evaluation scores per iteration with input, output, and critique logged.76- Convergence or budget-exhaustion result confirming the loop terminated cleanly.7778## Workflow79801. Name the application behavior under evaluation and the success criteria.812. Choose a strategy from [`references/patterns.md`](references/patterns.md): outcome-based, LLM-as-judge, or rubric-based.823. Complete the setup checklist in [`assets/eval-checklist.md`](assets/eval-checklist.md) (criteria, threshold, max iterations, logging).834. Isolate generate → evaluate → optimize steps; keep the evaluator replaceable.845. Run the loop with a hard `max_iterations` bound and a convergence check.856. Log full trajectories (input, output, score, critique) for every iteration.867. Stop on threshold met, no improvement, or budget exhaustion; report structured scores.878. Feed failures back into evaluator criteria, prompts, or optimization steps, then re-run.8889## Guardrails9091- Always set a `max_iterations` bound (3–5 is a safe default) before wiring up a refinement loop. Unbounded loops stall agents.92- Require structured output (JSON) from the evaluation step so the optimize step has a reliable signal to act on. Free-text critique is fragile.93- Add a convergence check: if the score does not improve between iterations, stop early. Oscillating loops that never converge waste budget.94- Log the full iteration trajectory. Evaluation loops are hard to debug post-hoc without a history of inputs, outputs, scores, and critiques.95- Define evaluation criteria before generating any output. Criteria added mid-loop drift and make scores incomparable across iterations.96- Keep the evaluate step isolated from the generate step. Blending them makes it hard to replace the evaluator or diagnose score instability.97- Handle evaluation parse failures gracefully — if the LLM judge returns malformed JSON, fall back to a safe default (treat as failing) rather than crashing the loop.9899## Validation100101After implementing an evaluation loop, confirm:102103- [ ] `max_iterations` is set and respected by the loop104- [ ] Evaluate step returns structured output (JSON or equivalent)105- [ ] Convergence check exits early when score does not improve106- [ ] All iterations are logged with input, output, score, and critique107- [ ] Parse-failure fallback is present on the evaluate step108- [ ] Criteria are defined before any generation begins109110## Examples111112- "Add a self-critique loop to my report-generation agent that retries up to three times if the rubric score is below 0.8."113- "Implement an evaluator-optimizer where a separate LLM judge scores code clarity and the generator rewrites until it passes."114- "Build a rubric-based evaluator with accuracy, completeness, and style dimensions that returns a weighted score as JSON."115116## Reference files117118- `references/patterns.md` — The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) with annotated Python examples and a best-practices table.119- `assets/eval-checklist.md` — Implementation checklist: setup, loop wiring, convergence, logging, and safety items to confirm before shipping.