quality-flywheel

Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK. Creates eval datasets (from session traces or synthetic generation), selects and configures metrics (RubricMetric, LLMMetric, CodeExecutionMetric), executes evals via client.evals.evaluate(), and analyzes results to suggest concrete fixes. Supports both single-turn model evaluation and multi-turn agent trajectory evaluation. Use when asked to "evaluate my agent", "evaluate my model", "create eval dataset", "run evals", "analyze eval results", "which metrics should I use", "generate test data", or "improve quality".

Jump to install

Repository: GoogleCloudPlatform/vertex-ai-samples
Last source activity: April 23, 2026 at 16:37
Detected SKILL.md language: English
Stars: 783
Forks: 307

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

Back to search Back to installation

Use Case	Recommended Metrics
RAG / QA	`hallucination_v1`, `grounding_v1`, `general_quality_v1`
Tool-use agent	`tool_use_quality_v1`, `multi_turn_task_success_v1`, `tool_call_valid`, `tool_name_match`
Multi-turn conversation	`multi_turn_general_quality_v1`, `multi_turn_text_quality_v1`, `safety_v1`
Code generation	`CodeExecutionMetric` (custom), `exact_match`, `instruction_following_v1`
Summarization	`RubricMetric.SUMMARIZATION_QUALITY`, `rouge_l_sum`
Single-turn model eval	`general_quality_v1`, `text_quality_v1`, `instruction_following_v1`

import vertexai
from vertexai import Client, types
from google.genai import types as genai_types

# Initialize client
client = vertexai.Client(project="PROJECT_ID", location="LOCATION")

# --- SINGLE-TURN EVAL ---
dataset = types.EvaluationDataset(eval_cases=[
    types.EvalCase(prompt="Query here", response="Model response here"),
])

# --- MULTI-TURN AGENT EVAL ---
agent_data = types.evals.AgentData(
    agents={"my_agent": types.evals.AgentConfig(
        agent_id="my_agent", instruction="You are helpful.")},
    turns=[types.evals.ConversationTurn(turn_index=0, events=[
        types.evals.AgentEvent(author="user",
            content=genai_types.Content(role="user",
                parts=[genai_types.Part(text="Hello")])),
        types.evals.AgentEvent(author="my_agent",
            content=genai_types.Content(role="model",
                parts=[genai_types.Part(text="Hi! How can I help?")])),
    ])],
)
dataset = types.EvaluationDataset(
    eval_cases=[types.EvalCase(agent_data=agent_data)])

# --- METRICS ---
predefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY
custom_llm = types.LLMMetric(name="tone",
    prompt_template="Is this polite? Response: {response}")
custom_code = types.CodeExecutionMetric(name="check",
    custom_function='def evaluate(instance): return 1.0')

# --- EVALUATE ---
result = client.evals.evaluate(dataset=dataset, metrics=[predefined])

# --- RESULTS ---
for s in result.summary_metrics:
    print(f"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}")
for case in result.eval_case_results:
    for cand in case.response_candidate_results:
        for name, r in cand.metric_results.items():
            print(f"  {name}: score={r.score}, explanation={r.explanation}")