agent-platform-eval-flywheel

# Agent Platform Eval Flywheel Skill Help users evaluate and iteratively improve GenAI models and agents using the Agent Platform GenAI Evaluation SDK (`google.genai` / `agentplatform`). ## When to use this skill - Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation SDK (`client.evals.evaluate()`). - Creating evaluation datasets from session traces, pandas DataFrames, or synthetic generation. - Selecting, configuring, or writing custom evaluation metrics. - Analyzing rubric verdicts, loss patterns, and clustering failures. - Suggesting concrete code/prompt improvements based on eval results. - Evaluating a model served on an Agent Platform **endpoint** (BYOM) or a **Model-as-a-Service (MaaS)** model by ID — including deploying the model first if needed. For this case, follow [references/deployment.md](references/deployment.md) and use the `endpoint_evaluation.py` / `maas_evaluation.py` scripts. ## Safety & Confirmation Tiers (CRITICAL) Before executing any commands or scripts on behalf of the user, you MUST adhere to the following safety tiers based on the action requested: 1. **Tier R**: Read-only (`inspect_results.py`, `compare_results.py`, `validate_dataset.py`, `parse_adk_traces.py`, `render_html_report.py`) * **Rule**: No confirmation needed. You may execute these helper scripts immediately to inspect data, validate schemas, parse traces, or compare evaluation results. 2. **Tier M: Read-only with Compute Costs (`client.evals.run_inference`, `client.evals.evaluate`, `client.evals.generate_conversation_scenarios`, `client.evals.generate_loss_clusters`)** * **Rule**: These operations invoke LLMs or remote evaluation services that consume compute resources and incur costs. This requires **interactive confirmation** with 'Yes'/'No' options. * **Confirmation for EVERY evaluation run**: Every evaluation, re-evaluation, metric update, parameter change, or synthetic scenario generation requires its own dry-run preview and interactive confirmation. Never execute a second evaluation, comparison pass, or modified evaluation without presenting a new confirmation preview and obtaining user approval. * **Same-turn restriction**: Do not run the evaluation in the same turn as presenting the confirmation prompt. End your turn after asking and wait for the user's reply; only execute after explicit 'Yes' / approval. Printing a preview and then calling the tool before the user can answer does not count as obtaining confirmation. * **No Pre-Execution of Remote Evaluation**: NEVER execute `client.evals.evaluate()`, `client.evals.run_inference()`, `client.evals.generate_conversation_scenarios()`, or run any script invoking these remote operations before user confirmation. In the initial turn, you may prepare local data structures and compose the script, but you MUST present the dry-run preview card and obtain explicit user confirmation before running any remote evaluation or scenario generation call. * **Immediate Execution Upon Approval**: Once the user explicitly approves (e.g., 'Yes', 'Approved', 'Go ahead', 'Proceed'), proceed directly to executing the previewed evaluation script via `run_command` and report the results. Do not conclude the turn without executing the approved action. ## Setup The scripts need `vertexai` (from `google-cloud-aiplatform[evaluation]`), `google-genai`, `pandas`, and `requests`. Do **not** create a virtual environment — it starts empty and hides packages the environment already provides, forcing a redundant install. Probe, and install only what is missing: ```bash python3 -c "import vertexai, google.genai, pandas, requests" \ || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0' ``` The version specifiers must stay quoted: unquoted, bash reads `>=1.154.0` as a redirect and silently writes an empty file instead of constraining the install. Need `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`. - **Preserve User Project and Location**: Always prioritize the user's explicitly provided project and location (e.g. `project='<PROJECT_NUMBER>'`, `location='us-central1'`). Never change or override the user's requested location to 'global' unless the user explicitly requested 'global'. - **Missing Parameters**: If the user's request omits the project or location, you MUST pause in your response and ask the user for the missing location/project before preparing or running the evaluation. ### Correct SDK entrypoints ```python import agentplatform client = agentplatform.Client(project=PROJECT, location=LOCATION) client.evals.run_inference(model=..., src=...) client.evals.evaluate(dataset=..., metrics=...) client.evals.generate_conversation_scenarios(...) ``` Two imports that look plausible and are not: - `from agentplatform.types import evals` -- `ModuleNotFoundError`. `types` is a module, not a package; use `from agentplatform import types`. - `from vertexai.evaluation import PointwiseMetric, EvalTask` -- the superseded SDK. Its classes take different arguments (`PointwiseMetric` has no `system_instruction`), so code written against it fails with `TypeError` rather than an import error. Use `agentplatform` throughout. ## The Quality Flywheel Five stages, run in order on the first pass, then loop 2 → 5 until quality targets are met. ### Shortcuts that waste time | Shortcut | Why it fails | | ------------------------------------ | ------------------------------------ | | "I'll tune the metric threshold down | Hides real failures. Fix the agent, | : so it passes." : not the bar. : | "This case is flaky, I'll skip it." | Flakiness reveals non-determinism in | : : the agent. Fix with `temperature=0` : : : or stricter instructions. : | "I just need to fix the eval | If expected outputs keep moving, the | : dataset, not the agent." : agent has a behavior problem. : | "I can tell from the trace it works | Self-grading doesn't generalize. | : — skip Stage 3." : Always run `evaluate()` and read : : : scores. : | "One iteration is enough." | Expect 5–10+ iterations. Stopping | : : early leaves regressions on other : : : metrics undetected. : ### 1. Prepare Data Produce an `EvaluationDataset`. There are three input shapes, pick the one that matches the data the user already has: - **`EvalCase` list (single-turn or multi-turn):** ```python from agentplatform import types from google.genai import types as genai_types # prompt/reference/response values are Content, not str. UserContent and # ModelContent wrap a plain string and set the right role. dataset = types.EvaluationDataset(eval_cases=[ types.EvalCase( prompt=genai_types.UserContent("What is 2+2?"), responses=[types.ResponseCandidate( response=genai_types.ModelContent("4"))], reference=types.ResponseCandidate( response=genai_types.ModelContent("4")), ), # For multi-turn agent traces, set agent_data instead of prompt/responses. ]) ``` Multi-turn agent traces wrap each conversation in `AgentData` → `ConversationTurn` → `AgentEvent`. See [references/dataset_schema.md](references/dataset_schema.md) for the full type hierarchy. - **Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets):** ```python import pandas as pd from agentplatform import types df = pd.DataFrame({ "prompt": ["What is 2+2?", "Capital of France?"], "response": ["4", "Paris"], "reference": ["4", "Paris"], }) dataset = types.EvaluationDataset(eval_dataset_df=df) ``` Column names must match the fields the chosen metrics expect (see [references/dataset_schema.md](references/dataset_schema.md) for the per-metric requirements table). - **Cold start (no data at all):** synthesize scenarios server-side with `client.evals.generate_conversation_scenarios(agent=..., config=...)` -- the parameter is `agent` or `agent_info`, not `agents`, and `config` is required. The config class is `types.evals.UserScenarioGenerationConfig`, not `types.UserScenarioGenerationConfig`. Set its `user_scenario_count` (1-100): it defaults to None, the client accepts that, and the server rejects the call with `400 INVALID_ARGUMENT`. `count` is a separate field and does not substitute for it. Stage 2 plays the scenarios out. * **CRITICAL - Multi-turn user-simulation scenarios must use this API**: When the user asks to synthesize multi-turn **user scenarios** to simulate customers interacting with an agent (as opposed to writing a local single-turn test dataset in the sandbox), never author the conversation scenarios yourself in prose or a handwritten file. Calling `client.evals.generate_conversation_scenarios` is a server-side Tier M API call (using the user's project, location, model, and environment data) and requires a confirmation card; local single-turn dataset files written in the sandbox without calling Agent Platform APIs are Tier R. * **CRITICAL - Underspecified Requests**: When asked to synthesize scenarios, if the request omits required parameters (such as `location`, `environment_data`, `simulation_instruction`, or `model_name`), do NOT assume defaults or guess values. You MUST pause in your first turn and explicitly ask the user for the missing information (e.g., "Please provide the missing simulation instructions, environment data, model name, and location"). Only proceed with the dry-run preview after the user provides them. `location` has no safe default: neither `us-central1` nor a location you assumed on your own for another call is a substitute for a location the user has not given. * **Friction & Parameter Changes**: When asked to generate synthetic user scenarios, if the user modifies requested parameters (such as scenario count, model, or instructions) or pushes back, you MUST present a revised dry-run confirmation card with the updated parameters and wait for explicit user approval before executing generation code via `run_command`. - **Managed Agents (Gemini Agents API):** evaluate agents created with the [Managed Agents API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/managed-agents). Use `generate_conversation_scenarios` to create test scenarios from the agent's configuration, `run_inference` to execute the agent, and `evaluate` to score the traces. These functions now accept managed agents and interaction ids as input. You can also evaluate existing interactions recorded via the Interactions API using `InteractionsDataSource`. See [references/sdk_patterns.md](references/sdk_patterns.md) Pattern 8 for the full code pattern. For ADK session dumps, use `scripts/parse_adk_traces.py` instead of writing the conversion by hand. ### 2. Run Inference Populate responses/traces on the dataset. **Skip this stage** if traces are already complete (e.g., production logs or replay). ```python # Agent eval — pass a callable wrapping the user's ADK Agent/App. client.evals.run_inference(model=agent_callable, src=dataset) # Model eval — pass a model ID directly. client.evals.run_inference(model="gemini-2.5-flash", src=dataset) # Synthesized scenarios — let the simulator drive. client.evals.run_inference( model=agent_callable, src=dataset, user_simulator_config=UserSimulatorConfig(max_turn=10), ) # DataFrame also works as src= — no EvalCase wrapping needed. client.evals.run_inference(model="gemini-2.5-flash", src=df) # Managed Agent — pass an agent resource name. AGENT_RESOURCE = f"projects/{PROJECT_ID}/locations/global/agents/{AGENT_ID}" client.evals.run_inference( agent=AGENT_RESOURCE, src=scenarios, config={"user_simulator_config": {"max_turn": 3}}, ) ``` ### 3. Grade (always run) ```python result = client.evals.evaluate(dataset=dataset, metrics=[...]) result.show() # Interactive HTML report with scores, rubrics, and traces. ``` **Pick metrics by what you want to measure.** Full catalog in [references/metric_registry.md](references/metric_registry.md). **If the user names a metric, use it directly.** Every identifier in the tables below (`general_quality`, `text_quality`, `instruction_following`, `hallucination`, `grounding`, `safety`, `multi_turn_*`, `final_response_*`, `tool_use_quality`) is a `types.RubricMetric.<UPPERCASE_NAME>` accessor — pass it straight into `metrics=[types.RubricMetric.GENERAL_QUALITY, ...]`. Do not scaffold a custom `LLMMetric` for a name that appears here, and do not reach for `vertexai.evaluation.EvalTask` / `PointwiseMetric` / `MetricPromptTemplateExamples` — that SDK is superseded (see Setup). **`judge_model` format and handling user-named models:** * **Full resource name required for `judge_model`**: Both `LLMMetric` and `RubricMetric` send `judge_model` as `autorater_config.autorater_model` to the backend (even though `agentplatform` logs a client-side "ignored" warning on `RubricMetric`). Short names like `"gemini-2.5-flash"` or `"gemini-2.5-pro"` fail with `400 INVALID_ARGUMENT ("Invalid autorater model resource name")`. Always pass the full publisher model resource name: `f"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_NAME}"` (prefer `gemini-2.5-flash` as autorater; `gemini-2.5-pro` can intermittently fail server-side with `"Rubric results could not be reliably computed"`). * **Datasets without vs. with responses**: * If the dataset contains **prompts only** (no `response` column), call `run_inference(model="<the named model>", src=dataset)` to generate the candidate model's responses first, then grade them with `evaluate()`. * If the user **already supplies `response`s** in the dataset, **do NOT call `run_inference`** — `run_inference` overwrites the existing `response` column with the model's generated output and discards the user's responses! Instead, grade the user's supplied `response`s directly with `evaluate(dataset=dataset, ...)`, passing the named model as the full resource path in `judge_model` (or if a predefined `RubricMetric` cannot reliably compute rubric results with that autorater model, fall back to the default autorater and explain why). **Agent metrics (multi-turn, adaptive rubrics)** — start here for agent eval. Goal | Metric --------------------------------------------- | ------------------------------- Did the agent achieve the user's goal? | `multi_turn_task_success` Was the reasoning path logical and efficient? | `multi_turn_trajectory_quality` Tool/function calling quality across turns | `multi_turn_tool_use_quality` Overall conversational quality | `multi_turn_general_quality` Final response quality (no reference needed) | `final_response_quality` Final response vs. a golden reference | `final_response_match` Single-turn tool use | `tool_use_quality` **General quality metrics (single-turn, adaptive rubrics)** — for model eval. Goal | Metric