adk-eval-guide

name	adk-eval-guide
description	Use when evaluating ADK agents, running adk eval, writing evalsets, configuring eval metrics, or debugging evaluation results. Covers all 8 criteria (tool_trajectory_avg_score, response_match_score, final_response_match_v2, rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1, hallucinations_v1, safety_v1, per_turn_user_simulator_quality_v1), eval schema, LLM-as-judge configuration, user simulation, multimodal evaluation, and common eval failure causes.
allowed-tools	Bash, Read, Write, Edit
metadata	{"triggers":"adk eval, adk evaluation, evalset, tool_trajectory_avg_score, response_match_score, final_response_match_v2, rubric_based, hallucinations_v1, safety_v1, LLM-as-judge, eval criteria, eval config","related-skills":"google-adk, agentic-ai-dev, python-dev","domain":"agentic-ai","role":"specialist","scope":"evaluation","output-format":"code"}
last-reviewed	2026-03-15

ADK Evaluation Guide

Iron Law

NEVER evaluate agents against the live Gemini API in unit tests. Use adk eval with evalsets. Hitting the real API in eval makes results non-deterministic, incurs cost, and breaks CI. Always run evaluations through the ADK eval CLI with a defined evalset and config file.

Reference Files

File	Contents
reference/criteria-guide.md	Complete metrics reference — all 8 criteria, match types, custom metrics, judge model config
reference/user-simulation.md	Dynamic conversation testing — ConversationScenario, user simulator config, compatible criteria
reference/builtin-tools-eval.md	google_search and model-internal tools — trajectory behavior, metric compatibility
reference/multimodal-eval.md	Multimodal inputs — evalset schema, built-in metric limitations, custom evaluator pattern

---

Process

1. Define criteria — choose metrics based on your goal (see Choosing the Right Criteria below); read reference/criteria-guide.md for all 8 options
2. Write evalset — create evalset.json with eval cases; include intermediate_data.tool_uses for every turn that expects tool calls
3. Write eval config — create eval_config.json with criteria thresholds; set match_type to IN_ORDER unless you need strict EXACT regression tests
4. Run eval — adk eval ./app <path_to_evalset.json> --config_file_path=<path_to_config.json> --print_detailed_results
5. Interpret results — identify which criteria failed; match failure symptom to cause (see Common Eval Failure Causes below)
6. Fix the agent — adjust prompts, tool descriptions, or agent instructions; do NOT widen thresholds to hide failures
7. Fix the evalset if needed — if expected trajectory doesn't match real agent behavior due to model variance, switch match type or use rubric-based criteria
8. Rerun eval — verify the fix resolves the specific failure without regressing other cases
9. Expand coverage — once current cases pass, add more eval cases incrementally
10. Repeat — expect 5–10 iterations per feature; each iteration makes the agent more reliable

---

Choosing the Right Criteria

Goal	Recommended Metric
Regression testing / CI (fast, deterministic)	tool_trajectory_avg_score + response_match_score
Semantic response correctness (flexible phrasing OK)	final_response_match_v2
Response quality without reference answer	rubric_based_final_response_quality_v1
Validate tool usage reasoning	rubric_based_tool_use_quality_v1
Detect hallucinated claims	hallucinations_v1
Safety compliance	safety_v1
Dynamic multi-turn conversations	User simulation + hallucinations_v1 / safety_v1
Multimodal input (image, audio, file)	tool_trajectory_avg_score + custom metric for response quality

For complete metrics reference with config examples, see reference/criteria-guide.md.

---

Running Evaluations

# Run eval directly via ADK CLI:
adk eval ./app <path_to_evalset.json> --config_file_path=<path_to_config.json> --print_detailed_results

# Run specific eval cases from a set:
adk eval ./app my_evalset.json:eval_1,eval_2

# With GCS storage:
adk eval ./app my_evalset.json --eval_storage_uri gs://my-bucket/evals

# Manage eval sets:
adk eval_set create <agent_path> <eval_set_id>
adk eval_set add_eval_case <agent_path> <eval_set_id> --scenarios_file <path> --session_input_file <path>

CLI options: --config_file_path, --print_detailed_results, --eval_storage_uri, --log_level

---

Key Patterns

Pattern	What to Do
Trajectory always 0	Check if agent uses google_search — see reference/builtin-tools-eval.md
Extra tool calls failing EXACT match	Switch to IN_ORDER or ANY_ORDER match type
Non-deterministic scores	Set temperature=0 or switch to rubric-based eval
Multi-turn trajectory failures	Ensure tool_uses is defined for ALL turns, not just the final one
App name mismatch error	App(name=...) must match the directory name exactly
State type mismatch	session_input.state types must match Python initialization types
Image/audio not visible to judge	Built-in judge strips non-text; use custom metric — see reference/multimodal-eval.md
User simulation criteria mismatch	Only rubric/hallucination/safety criteria work with conversation_scenario

---

Common Eval Failure Causes

Symptom	Cause	Fix
tool_trajectory_avg_score always 0	Agent uses google_search (model-internal)	Remove trajectory metric; see reference/builtin-tools-eval.md
Missing tool_uses in intermediate turns	Trajectory expects match per invocation	Add expected tool calls to all turns
"Session not found" error	App name mismatch	Ensure App(name=...) matches directory name
Score fluctuates between runs	Non-deterministic model	Set temperature=0 or use rubric-based eval
Trajectory fails but tools are correct	Extra tools called	Switch to IN_ORDER/ANY_ORDER match type
Agent mentions data not in tool output	Hallucination	Tighten instructions; add hallucinations_v1
LLM judge ignores image/audio	get_text_from_content() skips non-text	Use custom metric with vision-capable judge

---

Documentation Sources

Source	URL	Purpose
Evaluation overview	https://google.github.io/adk-docs/evaluate/index.md	Getting started, CLI reference
Criteria reference	https://google.github.io/adk-docs/evaluate/criteria/index.md	All metric configs, custom metrics API
User simulation	https://google.github.io/adk-docs/evaluate/user-sim/index.md	ConversationScenario schema, simulator config