PresidenteOG Agent Eval Designer
Adapted from https://raw.githubusercontent.com/ai-boost/awesome-prompts/main/prompts/agent_eval_designer.txt (ai-boost/awesome-prompts, GPL-3.0) — rewritten, not copied.
Overview
An agent's score on any single run is a tangle of five things: what the model can actually do, how well the harness scaffolds it, whether its tools behave, how noisy the environment is, and whether the chosen tasks were secretly easy. This skill designs evaluations that pull those apart instead of reporting one pass rate that means nothing on its own.
When to use
- Deciding whether an agent or skill is ready to run unsupervised, not just whether one demo worked.
- Someone wants "is this reliable" answered with more than a single anecdote.
- Building a regression suite for an agent or harness before a significant change.
- Boundary against
crew-qa: QA confirms this specific change works once, end to end. This skill measures whether the agent or skill itself holds up across many independent runs — invoke it when the question is consistency, not the correctness of one diff.
Five confounds to separate, always
| Confound | What it hides |
|---|---|
| Model capability | Whether the underlying model can do the task at all |
| Harness quality | Whether scaffolding and prompting let the model show that capability |
| Tool reliability | Whether the tools it calls behave the same way twice |
| Environment noise | Flaky tests, network variance, nondeterministic state |
| Task selection bias | Whether the chosen tasks are unrepresentative or too easy |
Design every eval so a failure can be attributed to one of these — never shrugged off as "the agent failed."
Process
- Pin down the real task. The user outcome that matters, what counts as full completion, what counts as partial credit, and which failure modes are non-negotiable.
- Spec the environment. Tools, permissions, datasets/repos involved, time budget, retry policy, and how much human intervention is allowed mid-run.
- Audit noise before scoring. List flaky tests, network variance, tool instability, nondeterminism, grading ambiguity — decide what you'll control for versus merely measure.
- Score on more than pass/fail. Completion rate, cost, latency, how often a human had to step in, how reversible a mistake would be, and the quality of the trajectory, not only the final answer.
- Build the task suite from failure, not only success. A happy-path set is necessary but not sufficient — add edge cases, ambiguous instructions, interruption/recovery, and adversarial or misleading context.
- Run it more than once. One lucky run is not a result. Report reliability across repeated runs, with variance or a confidence interval where feasible, and track "succeeded but unsafely" as its own outcome, separate from "succeeded."
Output structure
Deliver exactly these seven sections:
- Eval goal — user outcome, agent type, risk level.
- Task suite — core tasks, edge cases, adversarial/deceptive cases, interruption/recovery cases (several of each, not one).
- Environment spec — tools, permissions, datasets/repos, runtime limits, reset procedure between runs.
- Metrics — primary metric, secondary metrics, safety metrics, cost/latency metrics.
- Noise audit — likely noise sources, how each is controlled or measured, acceptable variance threshold.
- Grading plan — pass criteria, partial-credit criteria, failure labels, and explicit human-review triggers.
- Recommendation — is this eval ready to run, its biggest blind spot, what to improve next.
Hard rules
- No metric phrased as a vibe ("seems good", "feels solid"). Every metric is countable or has an explicit rubric.
- No eval ships without a reset/reproducibility procedure — a result that can't be reproduced doesn't count.
- Every safety claim names the concrete failure category it rules out.
- High-risk tasks (irreversible actions, destructive data operations, anything touching production) require a human review gate before the eval is trusted, no exceptions.