eval-guide

name	eval-guide
description	Use when writing eval code, configuring eval infrastructure, creating golden datasets, setting up PromptRegistry, authoring CI eval gates, or working with any eval tool: DeepEval, Ragas, Giskard OSS v3, Promptfoo, Langfuse, Arize Phoenix, adk eval, ADK User Simulation, Vertex GenAI Eval. Covers per-agent accuracy thresholds, CI tier structure (R1-R4), MCP eval suites, golden dataset structure, and PromptRegistry architecture. Also covers pytest harness configuration (asyncio_mode, InMemoryRunner, parametrize-over-golden).
allowed-tools	Bash, Read, Write, Edit
metadata	{"triggers":"deepeval, ragas, giskard, promptfoo, langfuse, phoenix, PromptRegistry, golden dataset, evalset, eval-smoke, eval-adk, redteam, MCPUseMetric, Faithfulness, ToolCallAccuracy, HallucinationsV1, tests/golden, dataset_manifest, seed-prompts, R1 gate, R2 nightly, R3 pre-release, R4 canary, ci tier, eval threshold, ArenaGEval, GeminiModel judge, set_default_generator, promptfooconfig.yaml, Scenario, Suite, make eval-all-local, make redteam-promptfoo, make giskard-scan, make mcp-eval-all, make seed-prompts-local, eval-reviewer, per-agent threshold, golden case, asyncio_mode, InMemoryRunner, pytest-asyncio, dataset_manifest.yaml, diff_langfuse_prompts, emergency pack, prompt registry, prompt lifecycle","related-skills":"google-adk, adk-eval-guide, python-dev, adk-observability-guide","domain":"agentic-ai","role":"specialist","scope":"evaluation","output-format":"code"}
last-reviewed	2026-04-28

Eval Guide

Iron Law

NEVER generate eval tool code from memory. ALWAYS query Context7 MCP for the official API before writing any eval code. Every metric class name, constructor signature, and YAML provider ID must be verified against current official docs — these APIs change between minor versions.

Prefer LangChain-free eval paths. Ragas, Giskard, and Promptfoo all have LangChain-free paths — prefer them to reduce dependency surface and avoid version conflicts.

Consistent LLM judge. Best practice: use the same LLM provider as your main stack for LLM-as-judge to reduce vendor sprawl. For Gemini-based stacks: GeminiModel("gemini-2.5-flash") for DeepEval, Generator(model="google/gemini-3.1-flash") for Giskard, google:gemini-2.5-pro for Promptfoo.

Dispatch eval-reviewer agent after writing any eval code — same mandate as dispatching adk-reviewer after ADK agent code.

Documentation Sources — Query Context7 BEFORE Writing Any Tool Code

Tool	Context7 query	Fallback
DeepEval	deepeval	https://docs.confident-ai.com/docs
Ragas	ragas	https://docs.ragas.io/en/latest
Giskard OSS v3	giskard	https://docs.giskard.ai/en/latest
Promptfoo	promptfoo	https://www.promptfoo.dev/docs
Langfuse	langfuse	https://langfuse.com/docs
Arize Phoenix	arize-phoenix	https://docs.arize.com/phoenix
ADK Eval	google-adk (adk-docs MCP)	https://google.github.io/adk-docs/evaluate
Vertex GenAI Eval	google-cloud-aiplatform