06-evaluation

# Track A Step 6: Evaluate the Agent Verify your agent produces relevant, grounded, and safe responses before deploying it. The template includes evaluation infrastructure — this step teaches you to run it, interpret results, and customize the dataset. ## When to Use - Your agent is running locally with tools, auth, and (optionally) memory configured from Steps 1-5. - You want to validate response quality before deploying to Databricks Apps. - You need to establish a baseline for ongoing monitoring after deployment. ## Relationship to SDLC Evaluation This step runs the template's built-in evaluation as a **smoke test**. The full SDLC evaluation pipeline ([S2](../../../sdlc/02-evaluation-datasets/SKILL.md)–[S4](../../../sdlc/04-evaluation-runs/SKILL.md)) adds custom benchmark datasets, domain-specific scorers with threshold gates, and human feedback labeling. Think of A6 as "does my agent work?" and S4 as "does my agent meet production quality bars?" | Aspect | A6 (this step) | S4 (SDLC) | |--------|----------------|-----------| | **Scope** | Template smoke test | Comprehensive quality gate | | **Dataset** | `tests/eval_dataset.json` (5-10 cases) | Custom benchmark (50-500+ cases) | | **Scorers** | Built-in judges only | Custom + built-in judges with thresholds | | **Trigger** | Manual `uv run agent-evaluate` | CI/CD pipeline on every PR | | **Output** | Pass/fail summary | Detailed metrics, regression detection | --- ## Step 6a: Run the Built-In Evaluation Smoke evaluation uses the configured model route from `docs/agent_tool_plan.yaml.runtime_config.llm`. The evaluation runner must not substitute a hardcoded model endpoint. If the route is the default Databricks provider, this resolves to `llm_endpoint` in `config.yml`. The agent template includes evaluation code in `agent_server/evaluate_agent.py` (or `tests/evaluate_agent.py` depending on your template version). Run it: ```bash uv run agent-evaluate ``` **What this does:** 1. Loads the evaluation dataset (typically `tests/eval_dataset.json`) 2. Sends each test case to your agent via `predict()` 3. Runs built-in judges against the responses 4. Outputs a summary report with pass/fail and scores **Expected output:** ``` Running evaluation... ✓ relevance: 4/5 passed ✓ groundedness: 5/5 passed ✓ safety: 5/5 passed Evaluation complete. Results saved to MLflow experiment. ``` If the command fails with auth errors, see the auth-in-evaluation section below. --- ## Step 6b: Understand the Evaluation Dataset The template ships a sample dataset in `tests/eval_dataset.json`: ```json [ { "request": { "input": [ {"role": "user", "content": "What is MLflow?"} ] }, "expected_response": "MLflow is an open source platform for managing the ML lifecycle.", "expected_retrieved_context": [ {"doc_uri": "https://mlflow.org/docs/latest/index.html"} ] } ] ``` | Field | Required | Purpose | |---|---|---| | `request` | Yes | The input sent to the agent (`input` messages, optional `custom_inputs`) | | `expected_response` | No | Ground truth for relevance comparison | | `expected_retrieved_context` | No | Expected `doc_uri` values for retrieval judges | The `expected_retrieved_context` field connects to `set_retriever_schema()` from Step 3 — the `doc_uri` column is how judges match retrieved vs. expected documents. --- ## Step 6c: Understand the Built-In Judges Agent Evaluation runs multiple judges automatically: | Judge | What It Checks | Requires | |---|---|---| | **Relevance** | Is the response relevant to the question? | `expected_response` (optional — can judge without) | | **Groundedness** | Is the response grounded in retrieved context? | RETRIEVER spans in traces | | **Safety** | Does the response avoid harmful content? | Nothing extra | | **Chunk relevance** | Are retrieved chunks relevant to the query? | RETRIEVER spans | Judges are LLM-based — they use a Databricks model serving endpoint to evaluate quality. The evaluation harness handles this automatically. --- ## Step 6d: Interpret Results After running evaluation, results appear in two places: ### 1. Terminal output Summary pass/fail counts and aggregate scores. ### 2. MLflow Experiment UI Navigate to your experiment → **Evaluation** tab for: - Per-request scores and judge rationale - Trace visualization for each test case - Aggregate metrics across the dataset Look for: - **Low relevance scores** → Agent may need better instructions or tools - **Low groundedness** → Agent may be hallucinating beyond retrieved context - **Safety failures** → Agent needs guardrails or instruction updates --- ## Step 6e: Customize the Evaluation Dataset Replace the sample dataset with test cases for your agent's domain: ```json [ { "request": { "input": [ {"role": "user", "content": "What were Q3 2025 total sales?"} ] }, "expected_response": "Q3 2025 total sales were $42.3M.", "expected_retrieved_context": [ {"doc_uri": "catalog.schema.sales_summary"} ] }, { "request": { "input": [ {"role": "user", "content": "Who is the VP of Engineering?"} ] }, "expected_response": null } ] ``` **Guidelines for good eval datasets:** - Include 10-50 test cases covering your agent's core capabilities - Mix questions that require tools with general knowledge questions - Include edge cases (ambiguous queries, out-of-scope questions) - Set `expected_response` to `null` when you only want safety/groundedness checks (no relevance baseline) - Use real `doc_uri` values from your Vector Search indexes --- ## Step 6f: Run Evaluation Programmatically For more control, call `mlflow.evaluate()` directly: ```python import mlflow import json eval_dataset = json.load(open("tests/eval_dataset.json")) with mlflow.start_run(): results = mlflow.evaluate( model="agent.py", data=eval_dataset, model_type="databricks-agent", ) print(f"Metrics: {results.metrics}") print(f"Results table:\n{results.tables['eval_results']}") ``` This gives you access to: - `results.metrics` — aggregate scores (relevance, groundedness, safety) - `results.tables["eval_results"]` — per-row results with judge rationale --- ## Authentication in Evaluation Context Evaluation runs use **different auth** than Model Serving or Databricks Apps. Key differences: | Context | Auth Method | |---|---| | Model Serving | OBO (user token passthrough) | | Databricks Apps | SP (app service principal) | | **Evaluation** | **Your credentials** (notebook/CLI user) or `SystemAuthPolicy` SP | If your agent uses OBO (`IS_IN_DB_MODEL_SERVING_ENV`), evaluation will fail because OBO environment variables are absent. Your agent's environment detection from Step 4 handles this — it falls back to default `WorkspaceClient()` auth outside Model Serving. If evaluation still fails with permission errors: 1. Verify your agent declares resources in `SystemAuthPolicy` (see Step 4, `references/model-serving-auth-passthrough.md`) 2. Ensure the evaluation user has access to all resources the agent needs 3. Check that Genie Spaces and Vector Search indexes are granted to the system SP --- ## DO / DON'T ### DO — Run evaluation before every deployment ```bash uv run agent-evaluate # Review results, then deploy ``` ### DON'T — Deploy without evaluating Changes to instructions, tools, or models can cause regressions that are invisible without evaluation. ### DO — Keep the eval dataset in version control ``` tests/ ├── eval_dataset.json # Your test cases └── eval_dataset_sample.json # Template's original (for reference) ``` ### DON'T — Use only the template's sample dataset The sample covers generic cases. Replace it with your domain-specific test cases for meaningful quality assurance. ### DO — Treat low scores as blockers ``` # If relevance < 80% or safety has failures: # 1. Review failing test cases in MLflow UI # 2. Fix agent instructions or tools # 3. Re-run evaluation ``` --- ## Common Issues | Issue | Fix | |-------|-----| | `uv run agent-evaluate` not found | Check `pyproject.toml` for the `agent-evaluate` script entry | | Auth errors during evaluation | Agent's env detection should fall back to default auth outside serving | | All relevance scores are low | Check `expected_response` values — they may not match your agent's style | | Groundedness judge returns N/A | Agent has no RETRIEVER spans — add `set_retriever_schema()` (Step 3) | | Safety judge flags benign content | Review the flagged responses; may need instruction tuning | --- ## Validation Gate All must pass before proceeding: - [ ] `uv run agent-evaluate` runs to completion without errors - [ ] Evaluation results visible in MLflow Experiment UI - [ ] Relevance scores meet your threshold (recommended: ≥80%) - [ ] Zero safety failures - [ ] Evaluation dataset customized with domain-specific test cases - [ ] Auth works in evaluation context (no permission errors) ## Next Step After passing this gate, **load and execute** `tracks/A-custom-agent-apps/07-deploy-and-query/SKILL.md` (A7: Deploy and Query) for the first manual deployment. ## Going Further: Full SDLC Evaluation The smoke test above validates basic functionality. After completing Track A (A7), you will enter the SDLC pipeline to build the comprehensive evaluation with custom scorers and threshold gates. When you reach that point, load these skills in order: 1. **Load and execute** `sdlc/02-evaluation-datasets/SKILL.md` (S2) — build your benchmark dataset (50-500+ rows) 2. **Load and execute** `sdlc/03-scorers-and-judges/SKILL.md` (S3) — create domain-specific scorers with pass/fail thresholds 3. **Load and execute** `sdlc/04-evaluation-runs/SKILL.md` (S4) — run `mlflow.genai.evaluate()` with your `predict_fn` from A5 S4 uses the same `predict_fn` you produced in A5 — no additional wrapping needed. The SDLC evaluation replaces this smoke test as the production quality gate. ## Notes to Carry Forward | Key | Value | |-----|-------| | `eval_dataset` | Path to your evaluation dataset | | `eval_results` | Baseline metrics (relevance, groundedness, safety scores) | | `mlflow_experiment` | MLflow experiment name/ID with evaluation runs | --- ## Related Skills | Skill | Relationship | |-------|-------------| | [A5: Lakebase Memory](../05-lakebase-memory/SKILL.md) | Previous step — produces `predict_fn` | | [A7: Deploy and Query](../07-deploy-and-query/SKILL.md) | Next step | | [S2: Evaluation Datasets](../../../sdlc/02-evaluation-datasets/SKILL.md) | SDLC: custom benchmark datasets | | [S3: Scorers and Judges](../../../sdlc/03-scorers-and-judges/SKILL.md) | SDLC: custom scorers with threshold gates | | [S4: Evaluation Runs](../../../sdlc/04-evaluation-runs/SKILL.md) | SDLC: comprehensive evaluation with `mlflow.genai.evaluate()` | ## References - [Author an agent — Evaluate the agent](https://docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent#step-6-evaluate-the-agent) - [Agent Evaluation](https://docs.databricks.com/aws/en/generative-ai/agent-evaluation/) - [Evaluation sets](https://docs.databricks.com/aws/en/generative-ai/agent-evaluation/evaluation-set) - [MLflow evaluate()](https://mlflow.org/docs/latest/genai/evaluation/) - [Agent Evaluation judges](https://docs.databricks.com/aws/en/generative-ai/agent-evaluation/evaluation-judges) ## Version History | Version | Date | Changes | |---------|------|---------| | 1.0.0 | 2026-04-11 | Initial skill: agent-evaluate, eval datasets, judges, result interpretation, auth context |