evaluation-debugging

# Evaluation Debugging Use this sub-skill when the user needs to create or run ADK evaluation cases, replay JSON tests, inspect session events, summarize traces, or choose safe native verification targets for an ADK agent. ## Route Common Requests - Convert a manual interaction into an `adk test` fixture or replay existing `tests/*.json` event files with deterministic mocks: use [Evaluation Workflows](references/evaluation-workflows.md). - Run metric-based EvalSet checks with `adk eval`, `adk eval_set`, or `google.adk.evaluation.AgentEvaluator`: use [Evaluation Workflows](references/evaluation-workflows.md). - Debug missing tool calls, unexpected output, callback order, branch/node routing, or event/session state: use [Debugging](references/debugging.md). - Diagnose schema validation, credentials, flaky LLM metrics, missing trace attributes, and server session lookup failures: use [Troubleshooting](references/troubleshooting.md). - Summarize raw ADK session/event JSON without exposing long values: run [summarize_adk_events.py](scripts/summarize_adk_events.py). ## Boundaries - Stay in this sub-skill for `adk test`, `adk eval`, `adk eval_set`, JSON test/eval fixtures, `AgentEvaluator`, event printing, session/trace inspection, and selecting safe native eval/debug verification candidates. - Route general CLI command discovery, agent app layout, server startup, deployment, and YAML configuration mechanics to `cli-configuration-deployment`. - Route code changes to agent constructors, tools, callbacks, instructions, schemas, or model binding to `agent-construction` or `tools-and-integrations`. - Route repository maintainer style, broad pytest policy, docs/sample update policy, and PR review gates to `repo-development`. ## Safe Defaults - Prefer read-only checks first: `adk test --help`, `adk eval --help`, `adk eval_set --help`, fixture validation, event summaries, and focused native candidates. - Do not start a long-running `adk web` or `api_server` process unless the user asks or confirms; if one is already running, inspect it before starting another. - Treat model credentials, cloud storage, Vertex AI scenario generation, and LLM-as-judge metrics as environment-dependent; separate these from deterministic local replay. - Never print full tool arguments, session state, trace payloads, or model requests when they may contain secrets; truncate and redact before sharing. ## Quick Decision Guide | Goal | Best path | Why | | --- | --- | --- | | Regression-test exact event flow | `adk test AGENTS_DIR` with `tests/*.json` | Replays stored event JSON, mocks model outputs/randomness, and normalizes volatile IDs. | | Score behavior against references/tools | `adk eval AGENT_INIT EVAL_SET` or `AgentEvaluator.evaluate(...)` | Uses EvalSet metrics such as `tool_trajectory_avg_score` and `response_match_score`. | | Inspect why a web session failed | Fetch session JSON, summarize events, then inspect `/dev/apps/{app}/debug/trace/session/{session}` | Correlates events, node paths, tool calls, and LLM/tool spans. | | Choose repo-native verification | Start with focused evaluation unit coverage, selected eval fixture integration coverage, and CLI help | Avoids broad, credential-heavy, or server-dependent runs unless needed. | ## Bundled Reference Map - [Evaluation Workflows](references/evaluation-workflows.md) — EvalSet schemas, legacy JSON, CLI/API routes, fixture design, assertions, and native candidates. - [Debugging](references/debugging.md) — `adk run`/`web` debugging, session and trace inspection, event fields, callback order, and `print_event` usage. - [Troubleshooting](references/troubleshooting.md) — Concrete failure signatures for schema mismatches, credentials, flaky metrics, missing tools, sessions, and trace attributes. - [summarize_adk_events.py](scripts/summarize_adk_events.py) — Safe stdin/file event JSON summarizer for sessions, event arrays, and trace-like dumps.