Agent Eval Contract

Agent Eval Contract

Evaluate a decision-relevant claim, not a demo. Make the tested system, task distribution, budget, evidence, and limits explicit so a higher score means something.

Lock the claim before the run

Define:

• Claim: the narrow behavior or improvement the evaluation can support
• Decision: what result would cause a ship, rollback, investigation, or redesign
• System boundary: model and version, instructions or skill, tool access, permissions, context, retry policy, and harness
• Task distribution: the real requests and environments to which the result may generalize
• Budget: turns, tokens, wall time, tool calls, retries, cost, and any human assistance

Do not generalize from a narrow benchmark to a broader capability. If the harness or budget differs from production, record that as a limitation.

Build a scenario matrix

Cover behavior that changes the release decision, not every wording variation:

• representative success paths from real usage
• boundary and ambiguous requests
• partial tool failures, timeouts, and recovery paths
• authorization and must-not-change invariants
• long-horizon or context-heavy cases when the workflow claims to support them
• known regressions and plausible shortcuts that could game the score

Derive cases from actual failures and user workflows when available. Keep a development set for iteration and a holdout set for the release decision. Version the tasks, fixtures, environment, and expected outcomes; prevent answers or close variants from leaking into the tested context.

Match evidence to the claim

Use the strongest observable evidence available:

• Outcome: authoritative final state, artifact behavior, or task completion
• Trajectory: tool selection, arguments, ordering, retries, and verification steps
• Invariants: forbidden mutations, privacy boundaries, and preserved user state
• Efficiency: accepted outcomes per unit of latency, cost, tokens, or attempts
• Human judgment: qualities that cannot be reduced to deterministic state

Prefer state and behavioral assertions over exact-string matching unless exact text is the requirement. Mark catastrophic safety, privacy, or authorization violations as critical failures rather than averaging them away.

Establish a fair baseline

• Compare against the current shipped system, no-skill condition, or other relevant baseline.
• Hold the harness, tools, task data, and resource budget constant; change one tested factor at a time.
• Use the same retry and best-of-N policy on every condition. Never cherry-pick the best attempt unless that selection process is part of the deployed system.
• Repeat stochastic cases enough to reveal meaningful variance, and report the distribution rather than only the best run.
• Record every model, prompt, skill, tool, dependency, dataset, and harness revision needed to reproduce the comparison.

Grade without fooling yourself

Use deterministic checks for objective state and rule compliance. Use a model judge only when the criterion requires interpretation.

For model-graded criteria:

• write a criterion-specific rubric with positive, negative, and borderline anchors
• blind the judge to treatment labels and desired conclusions
• calibrate against a human-reviewed sample and inspect disagreements
• give the judge the artifacts needed to decide, but do not require hidden private reasoning
• keep critical safety checks independent of a single model judge

Inspect individual failures. An aggregate score can hide a repeated tool misuse, a broken task, or one unacceptable side effect.

Audit the evaluation

Before trusting the result, run validity checks:

1. Confirm a clearly correct solution can pass.
2. Use negative controls to confirm plausible but wrong behavior fails.
3. Check that criteria describe outcomes rather than one preferred implementation.
4. Review tasks for contradictions, missing fixtures, stale external state, and impossible requirements.
5. Look for contamination, reward hacking, grader bias, and evaluation-aware shortcuts.
6. Set thresholds before viewing the treatment result.

Evaluation tasks that invoke tools should use isolated or synthetic environments. Do not spend credits, contact people, change production, or mutate external systems without authorization specific to the evaluation.

Write the contract

Claim and release decision:
Tested system and comparison:
Task distribution and exclusions:
Budget and retry policy:
Scenario matrix and dataset version:
Evidence and graders:
Critical failures:
Baseline and variance plan:
Ship / no-ship thresholds:
Validity checks:
Artifacts needed to reproduce:

Classify failures by mechanism—task understanding, planning, tool choice, tool arguments, environment interaction, recovery, verification, or authorization—so the evaluation guides the next change. If validity checks fail or results are inconclusive, report that the evaluation cannot support the release claim instead of forcing a verdict.