Agent Evaluation Design
Turn an AI-reliability question into a decision-ready evaluation contract. Read the evaluation contract reference when defining the case set, graders, or acceptance rule.
Preconditions
Identify the system version, decision owner, intended users, operating environment, available evidence, and decision the evaluation must support. If the target behavior or decision is not defined, return the missing decisions before inventing metrics.
Workflow
- State the evaluation question and the change, configuration, or baseline being compared.
- Define the population of real tasks without embedding private inputs. Partition representative, boundary, adversarial, abstention, recovery, and known-failure cases.
- Pin the system configuration: model or agent version, instructions, tools, context policy, sampling settings, dependencies, and environment.
- Define a baseline that makes improvement or regression interpretable. Use the current system, a simpler workflow, human performance, or a justified no-system baseline.
- Assign an oracle or grader to each behavior. Prefer deterministic checks for objective facts; define calibrated human review or rubric-based grading where judgment is unavoidable.
- Create an error taxonomy that separates wrong answers, missing required content, unsupported claims, unsafe actions, routing failures, refusals, tool failures, and evidence gaps.
- Define metrics, units, aggregation, confidence or uncertainty treatment, cost, latency, and subgroup slices. Prevent averages from hiding severe or safety-critical failures.
- Set thresholds and stopping rules before observing candidate results. State which failures are release blockers and which require human review.
- Document sampling limits, grader limitations, contamination risks, nonindependence, and what the evaluation cannot establish.
- Produce the evaluation contract. Do not run a paid or external evaluation without separate authority for the provider, data, budget, and network action.
For usage measurements, distinguish user task requests, delegated work, and
client-managed inference. Record the observation window, request categories,
models, input/output and cache usage where available, and gaps in collection.
Do not equate session counts with API calls or estimated token cost with billed
spend or subscription allowance. Compare task outcomes and elapsed time alongside
usage; fewer requests alone do not establish improvement. Instrumentation is a
separate configuration and data-export decision, not implicit evaluation authority.
Routing Boundaries
- Use
plan-tests to select software test layers for deterministic application behavior.
- Use
review-tests to assess the defect sensitivity and quality of existing tests.
- Use
track-acceptance when the user wants a durable completion contract across substantial work.
- A product-specific model architecture or provider review belongs to its product catalog unless the request specifically asks for a comparative evaluation design.
Output
Return a concise evaluation contract covering the decision, system, population,
cases, baseline, configurations, graders, error classes, metrics, thresholds,
human review, cost, latency, execution authority, and limitations.
Declare planned case, arm, and repetition coverage before collection. Bind each
result to a unique study, case, arm, repetition, client, source, instructions,
and execution envelope. Reject duplicate or mismatched identities rather than
overwriting. Check coverage and evidence integrity before scoring; keep failed
execution, missing evidence, deterministic diagnostics, and pending human review
distinct. Isolate treatment setup from grading and protect the test contract.
Read the original acceptance example when checking this behavior.
Example
Synthetic example; not an observed client result.
Input: One of twelve planned cases has a favorable answer.
Output: The comparison is incomplete. One response cannot establish the agent's success rate or justify adoption.
1---2name: design-agent-evaluations3description: "Designs AI evaluations when asked how to test an agent, compare prompts, or measure answer quality; requires representative cases and a baseline rather than one favorable response."4license: MIT5metadata:6 source: https://skillmd.com/skills/jovanipink/design-agent-evaluations7 author: "Measured Studios"8 version: "0.19.0"9 plugin: "measured-quality"10 invocation: "implicit"11 provenance: "original"12 risk_class: "read-only"13---1415# Agent Evaluation Design1617Turn an AI-reliability question into a decision-ready evaluation contract. Read [the evaluation contract reference](references/evaluation-contract.md) when defining the case set, graders, or acceptance rule.1819## Preconditions2021Identify the system version, decision owner, intended users, operating environment, available evidence, and decision the evaluation must support. If the target behavior or decision is not defined, return the missing decisions before inventing metrics.2223## Workflow24251. State the evaluation question and the change, configuration, or baseline being compared.262. Define the population of real tasks without embedding private inputs. Partition representative, boundary, adversarial, abstention, recovery, and known-failure cases.273. Pin the system configuration: model or agent version, instructions, tools, context policy, sampling settings, dependencies, and environment.284. Define a baseline that makes improvement or regression interpretable. Use the current system, a simpler workflow, human performance, or a justified no-system baseline.295. Assign an oracle or grader to each behavior. Prefer deterministic checks for objective facts; define calibrated human review or rubric-based grading where judgment is unavoidable.306. Create an error taxonomy that separates wrong answers, missing required content, unsupported claims, unsafe actions, routing failures, refusals, tool failures, and evidence gaps.317. Define metrics, units, aggregation, confidence or uncertainty treatment, cost, latency, and subgroup slices. Prevent averages from hiding severe or safety-critical failures.328. Set thresholds and stopping rules before observing candidate results. State which failures are release blockers and which require human review.339. Document sampling limits, grader limitations, contamination risks, nonindependence, and what the evaluation cannot establish.3410. Produce the evaluation contract. Do not run a paid or external evaluation without separate authority for the provider, data, budget, and network action.3536For usage measurements, distinguish user task requests, delegated work, and37client-managed inference. Record the observation window, request categories,38models, input/output and cache usage where available, and gaps in collection.39Do not equate session counts with API calls or estimated token cost with billed40spend or subscription allowance. Compare task outcomes and elapsed time alongside41usage; fewer requests alone do not establish improvement. Instrumentation is a42separate configuration and data-export decision, not implicit evaluation authority.4344## Routing Boundaries4546- Use `plan-tests` to select software test layers for deterministic application behavior.47- Use `review-tests` to assess the defect sensitivity and quality of existing tests.48- Use `track-acceptance` when the user wants a durable completion contract across substantial work.49- A product-specific model architecture or provider review belongs to its product catalog unless the request specifically asks for a comparative evaluation design.5051## Output5253Return a concise evaluation contract covering the decision, system, population,54cases, baseline, configurations, graders, error classes, metrics, thresholds,55human review, cost, latency, execution authority, and limitations.5657Declare planned case, arm, and repetition coverage before collection. Bind each58result to a unique study, case, arm, repetition, client, source, instructions,59and execution envelope. Reject duplicate or mismatched identities rather than60overwriting. Check coverage and evidence integrity before scoring; keep failed61execution, missing evidence, deterministic diagnostics, and pending human review62distinct. Isolate treatment setup from grading and protect the test contract.6364Read [the original acceptance example](references/continuity-example.md) when checking this behavior.6566## Example6768Synthetic example; not an observed client result.6970Input: One of twelve planned cases has a favorable answer.7172Output: The comparison is incomplete. One response cannot establish the agent's success rate or justify adoption.