| name | agent-eval-flywheel |
| description | Use this skill to evaluate an ADK agent with Google's Agent Quality Flywheel, run an evalset, choose the right metric, and read the results. Covers the ADK metric menu and which metrics run locally versus which call an AI judge, how the LLM judge scores meaning rather than words, how a rubric judge writes a plain-English reason for each verdict graded against trusted evidence, and how to close the eval fix loop. SDKs and tools used, agents-cli, adk eval, google-adk[eval], the Gemini API for the agent and the judge.
|
Agent Eval Flywheel Skill
How Google grades AI agents, and how you drive the loop. You pick a metric that matches what you care
about, run the eval, read what the judge says, and refine. The payoff is a judge that scores an
answer's meaning and explains its verdict in plain English, a score you can act on.
[!IMPORTANT]
Two skills ship this workflow. google-agents-cli-eval drives ADK agents through the agents-cli
toolchain (in agents-cli 0.1.1 eval run shells out to adk eval, in 1.0.0 it chains
eval generate and eval grade). agent-platform-eval-flywheel is the framework agnostic sibling
for the cloud Evaluation SDK, published in google/skills under skills/cloud/. This skill covers the
local ADK path, which needs nothing but the Gemini API key.
[!IMPORTANT]
The metric menu is the whole mental model. Two metrics run LOCALLY and free, tool_trajectory_avg_score
(did it call the right tools) and response_match_score (does the text match a reference). The rest
are LLM as judge, final_response_match_v2, rubric_based_final_response_quality_v1,
rubric_based_tool_use_quality_v1, hallucinations_v1, and safety_v1. First decision every time,
do you care about the exact words or the meaning. Words, use a local metric. Meaning, use a judge.
Quick Start
The full eval fix loop, all verified against google-adk 2.4.0 on Python 3.12 with the Gemini API key.
export GOOGLE_GENAI_USE_VERTEXAI=0
export GOOGLE_API_KEY=YOUR_AI_STUDIO_KEY
adk eval ./episode_finder tests/eval/evalsets/episode_finder.evalset.json \
--config_file_path tests/eval/eval_config.json --print_detailed_results
adk eval ./episode_finder tests/eval/evalsets/episode_finder.evalset.json \
--config_file_path tests/eval/eval_config_judge.json --print_detailed_results
adk eval ./episode_finder tests/eval/evalsets/episode_finder.evalset.json \
--config_file_path tests/eval/eval_config_fixed.json --print_detailed_results
[!NOTE]
response_match_score is ROUGE, it compares words. A correct answer worded differently from your
reference scores low (measured 0.2 to 0.35 on this agent, always under the 0.5 threshold). That is
not a broken agent, it is the signal to use a judge when you care about meaning, not exact wording.
[!WARNING]
adk eval needs the eval extra. A bare pip install google-adk gives Error: Eval module is not installed, please install via pip install "google-adk[eval]". Install google-adk[eval].
[!WARNING]
The system macOS Python is 3.9, google-adk needs 3.10 or newer. Stand up a current interpreter,
uv venv --python 3.12, before installing.
The judge that explains itself
The best part of the toolkit. A rubric judge grades against plain-English rules you write, and writes
a reason for each one, against the trusted answer. adk eval prints scores but drops the rationale
into a history JSON under AGENT_DIR/.adk/eval_history/, so eval_report.py surfaces it.
python eval_report.py ./episode_finder \
tests/eval/evalsets/episode_finder.evalset.json \
--config tests/eval/eval_config_rubric.json
Real output, the judge grading against the trusted evidence.
Q Which Stage Studio episode covered background agents?
A Episode e02, titled "Managed Agents background and remote MCP", covered background agents.
rubric_based_final_response_quality_v1: PASSED score=1.0
why [names_episode_id] 1.0: states "Episode e02", consistent with the trusted evidence.
why [names_title] 1.0: includes the title "Managed Agents background and remote MCP".
why [no_guessing] 1.0: only the episode id and title from the find_episode tool, nothing invented.
[!IMPORTANT]
The judge is decoupled by design, Google's rule. Nothing grades its own work, the agent does not
score its answers and the optimizer does not score its fixes. A separate judge model call does the
grading against the reference and writes the reason. Keeping those two roles apart is the point of
the flywheel.
Optimize and iterate, catching a hallucination
This is the payoff, and the failure it catches is the one everyone worries about, the agent making
something up. The judge tells you WHY a case failed, you apply one targeted fix, and rerun to prove it
helped. Google's own framing, it proposes, you approve, human in the loop, not hands off. The fix is
almost always an edit to the agent's instruction, not the model.
Start from a well meaning but ungrounded agent (optimize/finder). Its instruction pushes it to
always sound confident and never leave a viewer empty handed. Ask it about topics the show has not
covered (optimize/grounding.evalset.json), and grade it with a grounding rubric
(optimize/eval_config_grounding.json). It invents episodes that do not exist, and the judge catches
every one.
BEFORE, ungrounded agent, asked about a topic the tool did not find
Q Which episode covered quantum computing?
A We covered quantum computing in Episode 77: "Quantum Leaps and Bounds: The Future of Computing."
rubric_based_final_response_quality_v1: FAILED score=0.0
why [grounded] 0.0: the tool returned found=false, but the answer invented "Episode 77", contradicting the evidence.
Now the proposing part. propose_fix.py reads the judge's failing reason and the agent's current
instruction, and asks a model to propose a revised instruction. The proposer is deliberately NOT the
judge, same decoupling rule.
python propose_fix.py ./optimize/finder optimize/grounding.evalset.json \
--config optimize/eval_config_grounding.json
=== Proposed instruction (a model wrote this, NOT the judge) ===
You are the Stage Studio episode helper. When a viewer asks which episode covered a topic,
answer with the episode number and title. If the topic has not been covered, clearly state
that it has not and do not invent an episode.
Approve it, paste it into the agent's instruction (the grounded version ships in finder_fixed/),
and rerun the same eval. The loop closes.
AFTER, grounded agent, same question
Q Which episode covered quantum computing?
A We have not covered quantum computing yet.
rubric_based_final_response_quality_v1: PASSED score=1.0
why [grounded] 1.0: says the topic has not been covered, invents no episode. Grounded.
That is one turn of the flywheel, read the reason, apply a targeted fix, rerun, compare to the
baseline. Repeat until the case passes, expect several turns on a real agent.
[!NOTE]
An automated version of this turn exists. google-adk 2.4.0 ships adk optimize, which refines the
root agent instructions with the GEPA framework against a target metric, and agents-cli 1.0.0
exposes it as the experimental eval optimize. Both are confirmed from the tools' own help output,
not yet exercised by this skill's verified runs, treat them as experimental.
Workflow
When asked to evaluate an ADK agent, follow this loop.
- Analyze the goal. Care about MEANING, reach for an LLM judge. Care about the exact tool sequence,
use
tool_trajectory_avg_score, local and free.
- Run
eval_report.py AGENT_DIR EVALSET --config CONFIG so you see the judge's reasons, not just
red or green numbers.
- Read the reason, make one change, rerun. Expect several iterations on a real agent, that is normal,
each pass makes it better.
Dependencies and Prerequisites
- Python 3.10 or newer (
uv venv --python 3.12 if the system Python is 3.9).
google-adk[eval] >= 2.4.0, install uv pip install "google-adk[eval]".agents-cli optional, uv tool install google-agents-cli. Verified at 0.1.1 where eval run
shells out to adk eval. The 1.0.0 release restructures the group, eval run chains
eval generate and eval grade, and adds experimental eval analyze (loss clusters, needs a
GCP project) and eval optimize (runs adk optimize, GEPA prompt refinement).- A Gemini API key for both the agent and the judge, or Vertex via
GOOGLE_GENAI_USE_VERTEXAI=1 and a project.
[!NOTE]
Automatic Loss Analysis (failure clustering) and adaptive AutoRaters that write a feedback paragraph
are Gemini Enterprise Agent Platform features, the cloud product, not the local adk eval path.
Locally the per rubric rationale above is the closest thing to that, and it is most of the value.
Supporting files
- scripts/eval_report.py, run any agent and evalset and print the judge's
per rubric reasons.
python eval_report.py AGENT_DIR EVALSET --config CONFIG.
- scripts/episode_finder/, the tiny verified demo agent, a helper with one
find_episode tool.
- scripts/tests/eval/evalsets/episode_finder.evalset.json, a two case evalset.
- scripts/tests/eval/eval_config.json, local metrics only.
- scripts/tests/eval/eval_config_judge.json, adds the AI judge
final_response_match_v2.
- scripts/tests/eval/eval_config_fixed.json, tool trajectory plus the judge, all green.
- scripts/tests/eval/eval_config_rubric.json, rubric judge with three rules, the reasons come from here.
- scripts/optimize/, the optimize and iterate demo.
finder/ is the ungrounded agent that hallucinates, finder_fixed/ is the grounded fix, grounding.evalset.json asks about uncovered topics, eval_config_grounding.json is the grounding rubric that catches the made-up episodes, and propose_fix.py reads the judge's reason and proposes the instruction fix (a model, not the judge, writes the proposal). python optimize/propose_fix.py AGENT_DIR EVALSET --config CONFIG.
Documentation Pages
You MUST fetch the matching page below before writing eval code. These hosted docs are the source of
truth for the metric names, config schema, and thresholds, do not rely solely on the examples above.
From the episode
Video, link once it is up.
