QA Agent Testing

QA Agent Testing

Design and run reliable evaluation suites for LLM agents, including tool-using, multi-turn, and multi-agent systems.

Default QA Workflow

1. Define the Agent Under Test (AUT): scope, tools, approval boundaries, out-of-scope requests, and safety rules.
2. Build a starter suite from real work:
   • Smoke suite: 5-8 highest-signal checks for PR gates
   • Regression suite: 15-25 tasks from real failures, tickets, or production traces
   • Refusal/security pack: unsafe requests, prompt injection, tool-output poisoning, and exfiltration attempts
   • Iterative coding trajectory: evolving specifications applied to the agent's own carried workspace, when extension quality matters
3. Define objective graders first: schema checks, golden traces, deterministic mocks, policy oracles, and tool side-effect checks.
4. Add model-based graders only where objective checks are insufficient; calibrate them and log judge versions.
5. Run offline evals with deterministic controls and trace logging.
6. Add optional online evals or canary comparisons for live traffic.
7. Gate changes on one consistent status model and log regressions.

Use the starter templates in assets/ for day-0 setup. The template keeps 10 tasks + 5 refusals as a starter scaffold, not a best-practice cap.

Determinism and Flake Control

• Pin prompts, configs, fixtures, and tool mocks where possible.
• Freeze time, timezone, and locale for tests that depend on them.
• Log model, judge, and tool versions for every run.
• Record traces: prompt or message history, tool name, args, outputs, latency, errors, retries, approvals, and side effects.

Minimal instrumentation: Instrument agents at three points only — LLM call entry/exit (with span IDs), tool invocations (input, output, duration), and branching decision points (which path was chosen and why). Avoid instrumenting every intermediate computation; each additional trace dimension increases latency and storage cost, and the exact overhead depends on SDK, sampling, export path, and backend. Start minimal, expand only when a category of failure is consistently hard to diagnose without it.

Evaluation Model

Use two layers and one rubric:

Canonical Per-Task Rubric (0-3 each, 6 dimensions)

Track these separately at suite level, not as per-task rubric rows: latency, cost, stability, bias or fairness, and debuggability.

Trace grading in practice: When grading the Trace layer, evaluate four properties independently:

1. Tool selection accuracy — did the agent call the right tool for the step? Use code-based graders (compare tool name to expected set).
2. Argument correctness — were the tool args valid and well-formed? Schema graders handle this.
3. Call ordering — did the agent sequence tool calls in a logical, dependency-respecting order? LLM-as-judge works well here.
4. Recovery behavior — when a tool failed or returned unexpected data, did the agent handle it safely (retry, escalate, or degrade gracefully)? Use fault injection to test this explicitly.

Prefer code-based graders for (1) and (2); reserve LLM judges for (3) and (4) where rubrics are harder to express as code.

CI Economics

• PR gate: smoke suite plus critical refusal and security checks.
• After a fix, rerun the smallest affected pack first; only then expand to the full regression or canary comparison.
• Nightly or scheduled: full regression, adversarial pack, latency and cost tracking, and optional online eval comparisons.
• High-risk changes: add targeted reruns for affected tools, prompts, or judge models.

Expert Judgment: Sizing, Cost, and Drift

Judgment calls a checklist alone will not surface:

• Sizing an eval sample. Do not default to a round number. If you need to detect a specific effect size (e.g. "did this change drop accuracy by 5 points"), run a power calculation (references/eval-dataset-design.md has a worked example) — the answer is often larger than intuition suggests. A release-gate golden set answering "did anything obviously break" needs far fewer cases (15-25) than a statistical A/B claim needs (often 300-900+ per arm, depending on baseline and effect size). Match the sample size to the claim you intend to make, not to a fixed convention.
• Human review vs. automated judging. Route to a human, not a judge model, when the decision is high-stakes/hard-to-reverse, when the task falls outside your judge's calibration coverage, when repeated judge runs disagree with themselves on the same case, or when the batch is small enough that human review is cheap relative to building and validating a judge prompt. See references/llm-judge-limitations.md for the full decision guide.
• Cost and latency budgeting. Eval cost scales with (suite size) x (judge calls per case) x (judge model cost) x (run frequency). A PR-gate smoke suite (5-8 cases, mostly code-based graders) should run in seconds to low minutes and cost near-zero; reserve LLM-judge-heavy grading and multi-trial pass^k reruns for nightly or pre-release runs, not every commit. If a suite's per-PR cost or wall-clock time creates pressure to skip it, that is a signal to split it (fast code-based gate on every PR, expensive judged/adversarial pack on a schedule) rather than to weaken the gate.
• Overfitting to your own eval suite. A static suite that a team iterates against for months stops measuring what it was built to measure, even with zero contamination. Keep a held-out slice untouched by prompt iteration, and treat a long-stable score with mild suspicion rather than pure satisfaction. Full treatment in references/eval-dataset-design.md.
• Judge-model drift. Judges drift for three separate reasons — the provider updates the model behind a stable-looking name, the grader prompt gets edited for one fix and quietly changes other scores, or the agent's real failure modes shift — and each has a different fix. Re-check judge-vs-human agreement on a fixed cadence, not only when something looks wrong.

Security and Robustness Tests (Required for Tool Agents)

• Prompt injection: retrieved text, tool outputs, and user files must be treated as untrusted.
• Tool-output poisoning: tool returns malicious instructions; the agent must ignore them.
• Tool argument smuggling: unsafe parameters must be blocked by validation or approval layers.
• Secret exfiltration: verify the agent refuses and does not leak environment or file secrets.
• Tool faults: timeouts, partial data, retries, malformed payloads, and permission failures.
• Approval-boundary checks: verify the agent does not silently cross sandbox or approval limits.
• Differential tests: compare model or config changes on the same suite for regressions.

Regression Prevention for Coding Agents

When testing agents that modify code (SWE agents, coding assistants, CI agents), use graph-based impact analysis to surface which tests cover the files being changed.

TDAD (arXiv:2603.17973) is a high-signal reference here: it found that targeted source-to-test context outperformed generic procedural TDD prompting for coding-agent regression control.

Working rule:

• give the agent targeted test context, not only test instructions
• require affected-test evidence before marking a task complete
• track regression rate separately from task completion

Use references/coding-agent-regression-testing.md for the benchmark details, protocol, and how to combine this with classic TDD.

Iterative Extension Evaluation

A coding agent can pass independent tasks yet degrade when it repeatedly extends its own earlier work. For edit, refactor, and migration agents, add an evolving-spec trajectory when this compounding risk is in scope:

• start from an empty workspace, then carry the agent-produced workspace from checkpoint to checkpoint
• start each checkpoint with fresh conversation and runtime state so only the workspace preserves earlier decisions
• grade observable behavior through an external black-box contract, including held-out tests
• report strict, isolated, core, and regression correctness separately at every checkpoint
• track structural erosion, verbosity, cost, and duration as trajectory signals; do not treat the quality signals as correctness predictors
• aggregate unequal-length trajectories into Start, Early, Mid, Late, and Final phases

Benchmark-mode hidden tests and production-mode targeted test context serve different goals. Keep held-out tests hidden from the agent when estimating unbiased benchmark performance. In production regression control, expose the relevant TDAD source-to-test map and targeted test context so the agent can protect known behavior; retain a separate held-out evaluation slice for measurement.

Use references/iterative-coding-agent-evals.md for the protocol, formulas, interpretation limits, and trajectory record.

Do / Avoid

Do:

• Prefer code-based or schema-based graders over model judges.
• Keep task cases tied to real failures and live usage patterns.
• Calibrate judge models on a small human-labeled set before trusting them.
• Quarantine flaky evals with an owner and expiry date.
• Include regression rate as a metric for coding agents, not just task success.
• Provide targeted test context (source→test maps) instead of generic "write tests" instructions.

Avoid:

• Treating happy-path prompt checks as sufficient coverage.
• Letting one generic rubric stand in for tool traces, approvals, or side effects.
• Using unsourced numeric claims in guidance or thresholds.
• Treating LLM-as-judge as the sole source of truth for high-stakes tasks.
• Adding procedural TDD instructions without pairing them with targeted test context — this can increase regressions.

Quick Reference

Eval Tooling Patterns

Key tools mapped to QA jobs (see references/eval-tooling-patterns.md for the full table and references/eval-platform-selection.md for platform comparison with code examples):

• Promptfoo for config-driven regression suites, refusal packs, and red-team attack sets. Supports trajectory assertions and major agent SDKs. Use it when the team needs fast iteration and diffable eval configs.
• DeepEval (v4.0+) for pytest-style unit evals with agent-native metrics (Task Completion, Tool Correctness, Step Efficiency, Plan Quality). Use when evaluation should live next to CI tests.
• lmnr for trace-native evaluation and execution-graph visibility. Use when regressions are about workflow shape, latency, or tool sequencing.
• Langfuse for production tracing and online evals on live traffic. Supports native OpenTelemetry, code evaluators, and MCP in 2026.
• Agent Governance Toolkit when policy boundaries, approvals, and authorization rules need explicit middleware-level tests instead of prompt-only checks.

Platform deprecation note (June 2026): The OpenAI Platform Evals UI (hosted at platform.openai.com) is being shut down — read-only October 31 2026, full shutdown November 30 2026. The open-source openai/evals package and API remain available. Teams using the Platform UI should migrate to Promptfoo (OpenAI-recommended), DeepEval, or Braintrust.

Rule of thumb:

• Start with your eval design and grader model first.
• Then choose tooling based on the failure mode you need to observe.
• Do not let the tool pick the eval rubric for you.

Decision Tree

Testing an agent?
  - New agent?
    - Create starter harness -> Run smoke suite -> Establish baseline
  - Prompt or tool changed?
    - Re-run smoke suite + affected regression cases -> Compare to baseline
  - Model or judge changed?
    - Re-run smoke + refusal/security pack + targeted regression cases
  - Multi-agent or tool workflow?
    - Add trace graders, fault injection, and approval-boundary tests
  - Preparing production rollout?
    - Add optional online evals or canary comparisons

Scoring and Gates

• Score each task with the canonical 6-dimension agent rubric (0-3 each, max 18).
• Score refusals separately on a 0-3 refusal rubric.
• Use one consistent status model everywhere:
  • FAIL: any task <9, any refusal =0, or any objective policy hard fail
  • PASS: all tasks >=12 and all refusals >=2
  • CONDITIONAL: everything else
• If you also track normalized score bands, treat them as informational quality bands unless your suite explicitly adopts them as gate criteria.

ASCII Flow

Agent QA request
  -> Define AUT scope, tools, approvals, and forbidden behavior
  -> Build smoke + regression + refusal/security packs from real work
  -> Prefer objective graders: schema, policy, golden traces, side effects
  -> Add calibrated model judges only for hard-to-code judgments
  -> Run offline evals with traces and deterministic controls
  -> Compare against baseline and classify FAIL/PASS/CONDITIONAL
  -> Expand to canary or online evals only for production rollout evidence

Navigation

Resources

• references/scoring-rubric.md - canonical scoring model, thresholds, and variance notes
• references/regression-protocol.md - rerun scopes, baselines, online evals, and recovery
• references/tool-sandboxing.md - sandbox tiers, MCP and tool hardening, approval checks
• references/eval-tooling-patterns.md - Promptfoo, trace tooling, and governance-tool selection rules
• references/eval-platform-selection.md - platform comparison and decision tree for DeepEval, Inspect AI, Braintrust, Ragas, Promptfoo, OpenAI Evals, and Langfuse
• references/multi-agent-testing.md - coordination testing patterns and handoff checks
• references/llm-judge-limitations.md - judge biases, calibration, and escalation rules
• references/agentic-benchmarks.md - τ²-bench usage, ABC checklist, and anti-patterns for reading benchmark results
• references/iterative-coding-agent-evals.md - evolving-spec, carried-workspace protocol and longitudinal quality signals

Templates

• assets/qa-harness-template.md - starter harness
• assets/scoring-sheet.md - per-run scoring tracker
• assets/regression-log.md - versioned regression log

External Resources

See data/sources.json for current primary sources, including OpenAI eval and grader docs (note: Platform Evals UI deprecated Nov 2026), Anthropic agent eval guidance, LangSmith evaluation docs, Promptfoo, DeepEval v4.0, Langfuse, Agent Governance Toolkit, OWASP LLM Top 10, OWASP Agentic Top 10 (2026), and UK AISI Inspect sandboxing docs.

Related Skills

Quick Start

1. Copy assets/qa-harness-template.md
2. Fill in AUT scope, tools, and approval boundaries
3. Define the starter 10 tasks + 5 refusals
4. Add smoke, regression, and security packs from real work
5. Set objective graders and refusal oracles
6. Run baseline tests and record traces
7. Log results in assets/regression-log.md

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.