Google Adk Eval

Google ADK — Evaluation

CLI Command

adk eval <agent_module_path> <eval_set_path>

Eval Set Format (JSON)

[
  {
    "name": "weather_query_success",
    "data": [
      {
        "query": "What's the weather in New York?",
        "expected_tool_use": [
          {
            "tool_name": "get_weather",
            "tool_input": {"city": "New York"}
          }
        ],
        "reference": "The weather in New York is sunny with a temperature of 25 degrees Celsius."
      }
    ]
  },
  {
    "name": "unknown_city_handling",
    "data": [
      {
        "query": "What's the weather in Atlantis?",
        "expected_tool_use": [
          {
            "tool_name": "get_weather",
            "tool_input": {"city": "Atlantis"}
          }
        ],
        "reference": "Weather information for 'Atlantis' is not available."
      }
    ]
  }
]

Eval Set Fields

Multi-Turn Eval

[
  {
    "name": "multi_turn_conversation",
    "data": [
      {
        "query": "My name is Alice",
        "reference": "Hello Alice"
      },
      {
        "query": "What's my name?",
        "reference": "Alice"
      }
    ]
  }
]

Running Evaluations

# Basic eval
adk eval my_agent/ my_agent/eval_set.json

# With specific model for evaluation
adk eval my_agent/ my_agent/eval_set.json --eval_model gemini-2.5-pro

# Output results to file
adk eval my_agent/ my_agent/eval_set.json --output results.json

File Naming Convention

my_agent/
├── __init__.py
├── agent.py
├── my_agent_eval_set_001.evalset.json    # Convention: *_eval_set_*.evalset.json
└── my_agent_eval_set_002.evalset.json

Evaluation Criteria

The eval framework checks:

1. Tool Use: Did the agent call the expected tools with expected inputs?
2. Response Quality: Does the response match the reference (semantic similarity)?
3. Conversation Flow: In multi-turn, does the agent maintain context?

Expected Tool Use Schema

{
  "expected_tool_use": [
    {
      "tool_name": "search_database",
      "tool_input": {
        "query": "user preferences",
        "limit": 10
      }
    }
  ]
}

Writing Good Eval Sets

1. Cover happy paths: Normal successful interactions
2. Cover edge cases: Invalid inputs, empty results, errors
3. Cover multi-turn: Context retention, state management
4. Be specific in references: Clear expected outcomes for scoring
5. Test tool selection: Verify the agent picks the right tool

Programmatic Evaluation

from google.adk.evaluation.eval_set import EvalSet

# Load eval set from file
eval_set = EvalSet.model_validate_json(open("eval_set.json").read())

# Programmatic evaluation is primarily done via the CLI:
# adk eval my_agent/ eval_set.json
# For custom evaluation logic, use the Runner directly with test inputs.

Key Rules

• Eval sets are JSON arrays of test cases
• Each test case has a name and data (list of turns)
• reference is the ground truth for scoring
• expected_tool_use validates the agent calls the right tools
• Use adk eval CLI for quick evaluation runs
• Multi-turn evals test conversation continuity
• Keep eval sets alongside agent code for co-evolution

Related Skills

• google-adk-scaffold — Project structure (eval file placement)
• google-adk-deploy — Production deployment after passing evals