Eval Guide

Eval Guide

Iron Law

NEVER generate eval tool code from memory. ALWAYS query Context7 MCP for the official API before writing any eval code. Every metric class name, constructor signature, and YAML provider ID must be verified against current official docs — these APIs change between minor versions.

Prefer LangChain-free eval paths. Ragas, Giskard, and Promptfoo all have LangChain-free paths — prefer them to reduce dependency surface and avoid version conflicts.

Consistent LLM judge. Best practice: use the same LLM provider as your main stack for LLM-as-judge to reduce vendor sprawl. For Gemini-based stacks: GeminiModel("gemini-2.5-flash") for DeepEval, Generator(model="google/gemini-3.1-flash") for Giskard, google:gemini-2.5-pro for Promptfoo.

Dispatch eval-reviewer agent after writing any eval code — same mandate as dispatching adk-reviewer after ADK agent code.

Documentation Sources — Query Context7 BEFORE Writing Any Tool Code

Reference Files

Process — Before Writing Any Eval Code

1. Identify which tool(s) are needed
2. Query Context7 for that tool's current API — the reference files are starting points, NOT the final authority on API signatures
3. Read the relevant reference file for patterns and gotchas
4. Check reference/per-agent-thresholds.md for the target agent's required thresholds
5. Check reference/golden-dataset.md for dataset structure and minimum case counts
6. Mark every test with @pytest.mark.r1 (PR gate) or @pytest.mark.r2 (nightly) — never unmarked
7. After implementation: dispatch eval-reviewer agent

Make Targets Quick Reference

Golden Dataset Minimum Requirements (Day-1)

Related Skills

• adk-eval-guide — ADK-native eval only (8 ADK criteria, evalset schema, user simulation)
• google-adk — ADK agent construction patterns
• adk-observability-guide — Phoenix OTel integration, span inspection