ADK Evaluation Guide
Iron Law
NEVER evaluate agents against the live Gemini API in unit tests. Use adk eval with evalsets.
Hitting the real API in eval makes results non-deterministic, incurs cost, and breaks CI. Always run evaluations through the ADK eval CLI with a defined evalset and config file.
Reference Files
| File |
Contents |
reference/criteria-guide.md |
Complete metrics reference — all 8 criteria, match types, custom metrics, judge model config |
reference/user-simulation.md |
Dynamic conversation testing — ConversationScenario, user simulator config, compatible criteria |
reference/builtin-tools-eval.md |
google_search and model-internal tools — trajectory behavior, metric compatibility |
reference/multimodal-eval.md |
Multimodal inputs — evalset schema, built-in metric limitations, custom evaluator pattern |
Process
- Define criteria — choose metrics based on your goal (see Choosing the Right Criteria below); read
reference/criteria-guide.md for all 8 options
- Write evalset — create
evalset.json with eval cases; include intermediate_data.tool_uses for every turn that expects tool calls
- Write eval config — create
eval_config.json with criteria thresholds; set match_type to IN_ORDER unless you need strict EXACT regression tests
- Run eval —
adk eval ./app <path_to_evalset.json> --config_file_path=<path_to_config.json> --print_detailed_results
- Interpret results — identify which criteria failed; match failure symptom to cause (see Common Eval Failure Causes below)
- Fix the agent — adjust prompts, tool descriptions, or agent instructions; do NOT widen thresholds to hide failures
- Fix the evalset if needed — if expected trajectory doesn't match real agent behavior due to model variance, switch match type or use rubric-based criteria
- Rerun eval — verify the fix resolves the specific failure without regressing other cases
- Expand coverage — once current cases pass, add more eval cases incrementally
- Repeat — expect 5–10 iterations per feature; each iteration makes the agent more reliable
Choosing the Right Criteria
| Goal |
Recommended Metric |
| Regression testing / CI (fast, deterministic) |
tool_trajectory_avg_score + response_match_score |
| Semantic response correctness (flexible phrasing OK) |
final_response_match_v2 |
| Response quality without reference answer |
rubric_based_final_response_quality_v1 |
| Validate tool usage reasoning |
rubric_based_tool_use_quality_v1 |
| Detect hallucinated claims |
hallucinations_v1 |
| Safety compliance |
safety_v1 |
| Dynamic multi-turn conversations |
User simulation + hallucinations_v1 / safety_v1 |
| Multimodal input (image, audio, file) |
tool_trajectory_avg_score + custom metric for response quality |
For complete metrics reference with config examples, see reference/criteria-guide.md.
Running Evaluations
# Run eval directly via ADK CLI:
adk eval ./app <path_to_evalset.json> --config_file_path=<path_to_config.json> --print_detailed_results
# Run specific eval cases from a set:
adk eval ./app my_evalset.json:eval_1,eval_2
# With GCS storage:
adk eval ./app my_evalset.json --eval_storage_uri gs://my-bucket/evals
# Manage eval sets:
adk eval_set create <agent_path> <eval_set_id>
adk eval_set add_eval_case <agent_path> <eval_set_id> --scenarios_file <path> --session_input_file <path>
CLI options: --config_file_path, --print_detailed_results, --eval_storage_uri, --log_level
Key Patterns
| Pattern |
What to Do |
| Trajectory always 0 |
Check if agent uses google_search — see reference/builtin-tools-eval.md |
| Extra tool calls failing EXACT match |
Switch to IN_ORDER or ANY_ORDER match type |
| Non-deterministic scores |
Set temperature=0 or switch to rubric-based eval |
| Multi-turn trajectory failures |
Ensure tool_uses is defined for ALL turns, not just the final one |
| App name mismatch error |
App(name=...) must match the directory name exactly |
| State type mismatch |
session_input.state types must match Python initialization types |
| Image/audio not visible to judge |
Built-in judge strips non-text; use custom metric — see reference/multimodal-eval.md |
| User simulation criteria mismatch |
Only rubric/hallucination/safety criteria work with conversation_scenario |
Common Eval Failure Causes
| Symptom |
Cause |
Fix |
tool_trajectory_avg_score always 0 |
Agent uses google_search (model-internal) |
Remove trajectory metric; see reference/builtin-tools-eval.md |
Missing tool_uses in intermediate turns |
Trajectory expects match per invocation |
Add expected tool calls to all turns |
| "Session not found" error |
App name mismatch |
Ensure App(name=...) matches directory name |
| Score fluctuates between runs |
Non-deterministic model |
Set temperature=0 or use rubric-based eval |
| Trajectory fails but tools are correct |
Extra tools called |
Switch to IN_ORDER/ANY_ORDER match type |
| Agent mentions data not in tool output |
Hallucination |
Tighten instructions; add hallucinations_v1 |
| LLM judge ignores image/audio |
get_text_from_content() skips non-text |
Use custom metric with vision-capable judge |
Documentation Sources
| Source |
URL |
Purpose |
| Evaluation overview |
https://google.github.io/adk-docs/evaluate/index.md |
Getting started, CLI reference |
| Criteria reference |
https://google.github.io/adk-docs/evaluate/criteria/index.md |
All metric configs, custom metrics API |
| User simulation |
https://google.github.io/adk-docs/evaluate/user-sim/index.md |
ConversationScenario schema, simulator config |
1---2name: adk-eval-guide3description: Use when evaluating ADK agents, running adk eval, writing evalsets, configuring eval metrics, or debugging evaluation results. Covers all 8 criteria (tool_trajectory_avg_score, response_match_score, final_response_match_v2, rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1, hallucinations_v1, safety_v1, per_turn_user_simulator_quality_v1), eval schema, LLM-as-judge configuration, user simulation, multimodal evaluation, and common eval failure causes.4---56# ADK Evaluation Guide78## Iron Law910**NEVER evaluate agents against the live Gemini API in unit tests. Use `adk eval` with evalsets.**11Hitting the real API in eval makes results non-deterministic, incurs cost, and breaks CI. Always run evaluations through the ADK eval CLI with a defined evalset and config file.1213## Reference Files1415| File | Contents |16|------|----------|17| `reference/criteria-guide.md` | Complete metrics reference — all 8 criteria, match types, custom metrics, judge model config |18| `reference/user-simulation.md` | Dynamic conversation testing — ConversationScenario, user simulator config, compatible criteria |19| `reference/builtin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |20| `reference/multimodal-eval.md` | Multimodal inputs — evalset schema, built-in metric limitations, custom evaluator pattern |2122---2324## Process25261. **Define criteria** — choose metrics based on your goal (see Choosing the Right Criteria below); read `reference/criteria-guide.md` for all 8 options272. **Write evalset** — create `evalset.json` with eval cases; include `intermediate_data.tool_uses` for every turn that expects tool calls283. **Write eval config** — create `eval_config.json` with criteria thresholds; set `match_type` to `IN_ORDER` unless you need strict `EXACT` regression tests294. **Run eval** — `adk eval ./app <path_to_evalset.json> --config_file_path=<path_to_config.json> --print_detailed_results`305. **Interpret results** — identify which criteria failed; match failure symptom to cause (see Common Eval Failure Causes below)316. **Fix the agent** — adjust prompts, tool descriptions, or agent instructions; do NOT widen thresholds to hide failures327. **Fix the evalset if needed** — if expected trajectory doesn't match real agent behavior due to model variance, switch match type or use rubric-based criteria338. **Rerun eval** — verify the fix resolves the specific failure without regressing other cases349. **Expand coverage** — once current cases pass, add more eval cases incrementally3510. **Repeat** — expect 5–10 iterations per feature; each iteration makes the agent more reliable3637---3839## Choosing the Right Criteria4041| Goal | Recommended Metric |42|------|--------------------|43| Regression testing / CI (fast, deterministic) | `tool_trajectory_avg_score` + `response_match_score` |44| Semantic response correctness (flexible phrasing OK) | `final_response_match_v2` |45| Response quality without reference answer | `rubric_based_final_response_quality_v1` |46| Validate tool usage reasoning | `rubric_based_tool_use_quality_v1` |47| Detect hallucinated claims | `hallucinations_v1` |48| Safety compliance | `safety_v1` |49| Dynamic multi-turn conversations | User simulation + `hallucinations_v1` / `safety_v1` |50| Multimodal input (image, audio, file) | `tool_trajectory_avg_score` + custom metric for response quality |5152For complete metrics reference with config examples, see `reference/criteria-guide.md`.5354---5556## Running Evaluations5758```bash59# Run eval directly via ADK CLI:60adk eval ./app <path_to_evalset.json> --config_file_path=<path_to_config.json> --print_detailed_results6162# Run specific eval cases from a set:63adk eval ./app my_evalset.json:eval_1,eval_26465# With GCS storage:66adk eval ./app my_evalset.json --eval_storage_uri gs://my-bucket/evals6768# Manage eval sets:69adk eval_set create <agent_path> <eval_set_id>70adk eval_set add_eval_case <agent_path> <eval_set_id> --scenarios_file <path> --session_input_file <path>71```7273**CLI options:** `--config_file_path`, `--print_detailed_results`, `--eval_storage_uri`, `--log_level`7475---7677## Key Patterns7879| Pattern | What to Do |80|---------|-----------|81| Trajectory always 0 | Check if agent uses `google_search` — see `reference/builtin-tools-eval.md` |82| Extra tool calls failing EXACT match | Switch to `IN_ORDER` or `ANY_ORDER` match type |83| Non-deterministic scores | Set `temperature=0` or switch to rubric-based eval |84| Multi-turn trajectory failures | Ensure `tool_uses` is defined for ALL turns, not just the final one |85| App name mismatch error | `App(name=...)` must match the directory name exactly |86| State type mismatch | `session_input.state` types must match Python initialization types |87| Image/audio not visible to judge | Built-in judge strips non-text; use custom metric — see `reference/multimodal-eval.md` |88| User simulation criteria mismatch | Only rubric/hallucination/safety criteria work with `conversation_scenario` |8990---9192## Common Eval Failure Causes9394| Symptom | Cause | Fix |95|---------|-------|-----|96| `tool_trajectory_avg_score` always 0 | Agent uses `google_search` (model-internal) | Remove trajectory metric; see `reference/builtin-tools-eval.md` |97| Missing `tool_uses` in intermediate turns | Trajectory expects match per invocation | Add expected tool calls to all turns |98| "Session not found" error | App name mismatch | Ensure `App(name=...)` matches directory name |99| Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval |100| Trajectory fails but tools are correct | Extra tools called | Switch to `IN_ORDER`/`ANY_ORDER` match type |101| Agent mentions data not in tool output | Hallucination | Tighten instructions; add `hallucinations_v1` |102| LLM judge ignores image/audio | `get_text_from_content()` skips non-text | Use custom metric with vision-capable judge |103104---105106## Documentation Sources107108| Source | URL | Purpose |109|--------|-----|---------|110| Evaluation overview | `https://google.github.io/adk-docs/evaluate/index.md` | Getting started, CLI reference |111| Criteria reference | `https://google.github.io/adk-docs/evaluate/criteria/index.md` | All metric configs, custom metrics API |112| User simulation | `https://google.github.io/adk-docs/evaluate/user-sim/index.md` | ConversationScenario schema, simulator config |
kumaran-is