Agent Eval

Agent Eval

Turns "does this actually work" into a repeatable, evidence-based answer instead of a gut feeling.

How this works

1. Figure out what "good" means first. Before writing any eval, get a concrete definition of success from the user, or infer it from context and confirm it back to them. What does a correct/good output look like? What does a clearly bad one look like? Is there a reference answer, or is this judgment-based?

2. Pick the eval type — don't default to one without considering the fit:

   • Reference-based: there's a known correct answer (exact or fuzzy/semantic match). Cheapest and most reliable, but only works when "correct" is well-defined.
   • Rubric-based (LLM-as-judge): quality is graded against explicit criteria (e.g. "factually grounded," "follows the required format," "appropriately concise"). Use references/llm-judge-prompt.md as the starting template — don't write a judge prompt from scratch each time.
   • Pairwise comparison: judging which of two outputs is better, not scoring each in isolation. More reliable than absolute scoring for subjective quality, but watch for position bias — always run both orderings and average. Use scripts/run_pairwise.py for this — it runs both orderings itself and reconciles them (see references/pairwise-comparison.md), rather than leaving "remember to run it twice" as a step to repeat by hand each time.
   • Programmatic/structural: format compliance, schema validation, code that actually executes, tool calls matching an expected sequence. Cheapest and most deterministic when applicable — prefer this over LLM-as-judge whenever the criterion is mechanically checkable.
   • Trajectory evaluation (agents specifically): don't just grade the final output — check whether the agent picked the right tool, passed the right arguments, and used a reasonable sequence of steps. A correct final answer reached via a broken process is still a signal of an unreliable agent. Use references/trajectory-eval.md for the case shape, the judge-rubric addendum, and the flattening convention — don't invent one from scratch each time.
   • Adversarial: the correct outcome is refusal, hedging, or graceful degradation — the question is "did the agent handle this appropriately?" not "was the answer correct." Never blend adversarial scores with correctness scores; run them as a separate eval set, in their own results file. category should be the specific failure mode under test (prompt-injection, jailbreak, over-refusal, ...), not a single flat adversarial bucket — that's what lets score_eval.py's per-category breakdown show pass rate per attack type, not just one undifferentiated adversarial score. See skills/agent-redteam/examples/adversarial_results.jsonl for the real convention, and skills/agent-redteam/ for a dedicated case-generation skill.
   • Multi-turn: not a distinct scoring type so much as a case-shape variant of the ones above — use when the conversation leading up to a response is itself part of what makes it right or wrong (a frame established earlier, an incremental escalation), not just the final message in isolation. Use references/multi-turn-eval.md for the case shape (a turns field, rendered into the judge prompt as {transcript} by scripts/run_judge.py) — don't collapse a real conversation into one flattened string.

3. Build a small, reusable eval set, not a one-off. Even 10-20 representative cases beats eyeballing a handful of outputs. Include a mix of: clear-pass cases, known-hard edge cases, and at least a few cases the current system is expected to fail — a sanity check that the eval can actually detect failure, not just confirm success.

4. Score it. For rubric/LLM-as-judge evals, use the judge prompt template and request structured JSON output (a score plus a one-line rationale per criterion) — never a vibe-based pass/fail. For programmatic evals, write the check directly.

   For rubric/LLM-as-judge evals specifically, use scripts/run_judge.py rather than calling the judge and flattening its response by hand each time — it fills the template, calls the judge, parses the structured JSON, and writes already-flattened rows in the exact schema score_eval.py reads:

   export ANTHROPIC_API_KEY=...
   python scripts/run_judge.py cases.jsonl --template references/llm-judge-prompt.md --out results.jsonl --category accuracy
   
   # Or judge with Gemini instead of Claude — same output, same schema:
   export GEMINI_API_KEY=...
   python scripts/run_judge.py cases.jsonl --template references/llm-judge-prompt.md --out results.jsonl --provider gemini
   

   --provider (default: anthropic; gemini also supported) picks which judge API gets called — scripts/run_pairwise.py takes the same flag. --model defaults to the chosen provider's own default model if not given. See call_judge()'s own docstring in run_judge.py for one honest caveat: Gemini's token-usage field (which cost_usd depends on) isn't independently live-verified in this repo the way Anthropic's is, so treat cost_usd from a Gemini-judged run as lower-confidence than from an Anthropic one.

   cases.jsonl is one row per case (id, plus whatever fields the template's {placeholder} tokens need — typically input/output, or input/trajectory/final_output for a trajectory case). It's generic over criterion names, so it handles both the plain rubric shape and references/trajectory-eval.md's shape without any mode flag. A per-case failure (judge call error, unparseable response, or a malformed case that fails template-filling) is reported to stderr and that case is skipped — never given a fabricated score. cost_usd is only included when both --input-price-per-mtok/--output-price-per-mtok are given, computed from the API's own real token counts.

   The judge prompt returns one nested object per case (per-criterion scores plus an overall_score) — that's a different shape from what scripts/score_eval.py reads. If scoring by some other means than run_judge.py (a different judge model/provider, a notebook), flatten each case before saving, to this schema (one JSON object per line):

   {"id": "case_001", "score": 0.83, "category": "accuracy", "rationale": "...", "cost_usd": 0.003, "latency_ms": 1240}
   

   • id — a stable identifier for the eval case, assigned when you build the eval set (not produced by the judge).
   • score — the judge's overall_score for LLM-as-judge evals, or 1.0/0.0 for a programmatic pass/fail check.
   • category — the criterion group or failure mode you're tracking (e.g. accuracy, format, tool_use), assigned by you, not read from the judge's per-criterion keys — this is what score_eval.py breaks results down by.
   • rationale — a one-line reason for the score. For a multi-criterion judge response, use the rationale from the lowest-scoring criterion, since that's the one explaining the failure.
   • cost_usd (optional) — API cost for generating the output under test.
   • latency_ms (optional) — wall-clock time to generate the output, in milliseconds.

   Save the flattened lines to a JSON/JSONL file, not just a summary — the failure examples are what make this actionable.

5. Calibrate the judge periodically. Every 25-50 judge calls (or whenever you revise the judge prompt), hand-score 5-10 cases yourself and compare to the judge's scores. Use scripts/calibrate_judge.py rather than eyeballing the delta — it computes the mean delta, flags whether it exceeds the threshold (default 0.2), and can log the result for you:

   python scripts/calibrate_judge.py judge_results.jsonl human_scores.jsonl --update-log references/llm-judge-prompt.md
   

   human_scores.jsonl only needs the 5-10 IDs you actually hand-scored (same JSONL schema as any score_eval.py results file) — it compares whichever IDs appear in both files. See examples/README.md's calibration example for a worked case where this catches a judge fooled by confident, verbose wrong answers. --update-log appends a row to references/llm-judge-prompt.md's calibration table automatically, replacing the "not yet calibrated" placeholder on the first real run — don't edit that table by hand.

6. Aggregate and report using scripts/score_eval.py. Don't manually tally pass rates — run the script against the results file:

   python scripts/score_eval.py results.jsonl
   python scripts/score_eval.py results.jsonl --baseline previous_results.jsonl
   

   It computes pass rate, mean score (overall and per-category), surfaces the lowest-scoring cases for review, and reports mean cost and latency per category when those fields are present. With --baseline, it flags regressions — cases that passed before and fail now.

   As a CI gate, add --fail-under and/or --fail-on-regression so the script exits non-zero (failing the build) when quality drops:

   python scripts/score_eval.py results.jsonl --fail-under 0.85
   python scripts/score_eval.py results.jsonl --baseline eval_set_v2.jsonl --fail-on-regression
   

   This is what makes an eval a gate rather than a report — the same run that scores your change also blocks it if it regressed. See examples/ for a worked before/after where a change looks like a win on cost, latency, and format but the gate catches three silent accuracy regressions.

   The reverse gap has a gate too: a change that holds accuracy perfectly steady while cost or latency quietly balloons was previously invisible to every flag above. --fail-on-cost-regression/--fail-on-latency-regression (needs --baseline; tolerance via --cost-regression-tolerance/--latency-regression-tolerance, default 20%) catch that directly; --fail-if-mean-cost-above/--fail-if-mean-latency-above set an absolute ceiling with no baseline needed:

   python scripts/score_eval.py results.jsonl --baseline previous_results.jsonl --fail-on-cost-regression --fail-on-latency-regression
   python scripts/score_eval.py results.jsonl --fail-if-mean-cost-above 0.01 --fail-if-mean-latency-above 2000
   

   See examples/README.md's cost/latency regression example for a worked case where accuracy is unchanged (same 90% pass rate) but cost/latency both roughly triple, and the gate catches it.

7. Be honest about sample size. With under ~20 cases, a 2-3 case swing can look like a large percentage shift. Say so explicitly: "3/10 passed (30%) — too small a sample to call this a real regression yet" rather than presenting a precise-looking percentage as statistically solid.

8. When re-evaluating after a change (new prompt, new model, new tool definition), always run the same eval set as before and diff against the saved baseline. That's what catches regressions — a fresh set of cases each time doesn't.

   When the case set itself changes (new cases added, old ones removed), save it as a new versioned file (eval_set_v2.jsonl, eval_set_v3.jsonl) rather than overwriting. Before reporting a regression, confirm both runs used the identical case set — same filename, same line count, same case IDs. A score drop may just be a case-set change, not a model regression.

Output discipline

• Never let an LLM-as-judge grade its own output unflagged — if the system under test and the judge share a model or prompt, say so as a caveat. Self-grading is a known source of inflated scores.
• Don't report an aggregate score without 2-3 concrete failure examples alongside it. A percentage with no examples isn't actionable.
• If the user hasn't defined success criteria and won't, don't silently invent a rubric and present results as objective — flag that the rubric is your best guess at their intent.
• Keep judge prompts in references/llm-judge-prompt.md versioned alongside the eval set, not rewritten ad hoc each run. Consistency between runs is what makes before/after comparisons valid.
• Report adversarial pass rates separately from correctness pass rates — never blend the two into a single aggregate.