agent-evaluation

# Agent Evaluation with MLflow Comprehensive guide for evaluating GenAI agents with MLflow. Use this skill for the complete evaluation workflow or individual components - tracing setup, environment configuration, dataset creation, scorer definition, or evaluation execution. Each section can be used independently based on your needs. ## ⛔ CRITICAL: Must Use MLflow APIs **DO NOT create custom evaluation frameworks.** You MUST use MLflow's native APIs: - **Datasets**: Use `mlflow.genai.datasets.create_dataset()` - NOT custom test case files - **Scorers**: Use `mlflow.genai.scorers` and `mlflow.genai.judges.make_judge()` - NOT custom scorer functions - **Evaluation**: Use `mlflow.genai.evaluate()` - NOT custom evaluation loops - **Scripts**: Use the provided `scripts/` directory templates - NOT custom `evaluation/` directories **Why?** MLflow tracks everything (datasets, scorers, traces, results) in the experiment. Custom frameworks bypass this and lose all observability. If you're tempted to create `evaluation/eval_dataset.py` or similar custom files, STOP. Use `scripts/create_dataset_template.py` instead. ## Table of Contents 1. [Quick Start](#quick-start) 2. [Documentation Access Protocol](#documentation-access-protocol) 3. [Setup Overview](#setup-overview) 4. [Evaluation Workflow](#evaluation-workflow) 5. [References](#references) ## Quick Start **⚠️ REMINDER: Use MLflow APIs from this skill. Do not create custom evaluation frameworks.** **Setup (prerequisite)**: Install MLflow 3.8+, configure environment, integrate tracing **Evaluation workflow in 5 steps** (each uses MLflow APIs): 1. **Understand**: Run agent, inspect traces, understand purpose 2. **Scorers**: Select and register scorers for quality criteria 3. **Dataset**: ALWAYS discover existing datasets first, only create new if needed 3.5. **Dry Run**: Run 3 questions first — catch broken tools and misconfigured scorers before full eval 4. **Evaluate**: Run agent on dataset, apply scorers, analyze results ## Command Conventions **Always use `uv run` for MLflow and Python commands:** ```bash uv run mlflow --version # MLflow CLI commands uv run python scripts/xxx.py # Python script execution uv run python -c "..." # Python one-liners ``` This ensures commands run in the correct environment with proper dependencies. **CRITICAL: Separate stderr from stdout when capturing CLI output:** When saving CLI command output to files for parsing (JSON, CSV, etc.), always redirect stderr separately to avoid mixing logs with structured data: ```bash # Save both separately for debugging uv run mlflow traces evaluate ... --output json > results.json 2> evaluation.log ``` ## Documentation Access Protocol **All MLflow documentation must be accessed through llms.txt:** 1. Start at: `https://mlflow.org/docs/latest/llms.txt` 2. Query llms.txt for your topic with specific prompt 3. If llms.txt references another doc, use WebFetch with that URL 4. Do not use WebSearch - use WebFetch with llms.txt first **This applies to all steps**, especially: - Dataset creation (read GenAI dataset docs from llms.txt) - Scorer registration (check MLflow docs for scorer APIs) - Evaluation execution (understand mlflow.genai.evaluate API) ## Discovering Agent Structure **Each project has unique structure.** Use dynamic exploration instead of assumptions: ### Find Agent Entry Points ```bash # Search for main agent functions grep -r "def.*agent" . --include="*.py" grep -r "def (run|stream|handle|process)" . --include="*.py" # Check common locations ls main.py app.py src/*/agent.py 2>/dev/null # Look for API routes grep -r "@app\.(get|post)" . --include="*.py" # FastAPI/Flask grep -r "def.*route" . --include="*.py" ``` ### Understand Project Structure ```bash # Check entry points in package config cat pyproject.toml setup.py 2>/dev/null | grep -A 5 "scripts\|entry_points" # Read project documentation cat README.md docs/*.md 2>/dev/null | head -100 # Explore main directories ls -la src/ app/ agent/ 2>/dev/null ``` ## Setup Overview ### Pre-check: Use Existing Environment **Before doing ANY setup, check if `MLFLOW_TRACKING_URI` and `MLFLOW_EXPERIMENT_ID` are already set:** ```bash echo "MLFLOW_TRACKING_URI=$MLFLOW_TRACKING_URI" echo "MLFLOW_EXPERIMENT_ID=$MLFLOW_EXPERIMENT_ID" ``` **If BOTH are already set, skip Steps 1-2 entirely.** The environment is pre-configured. Do NOT run `setup_mlflow.py`, do NOT create a `.env` file, do NOT override these values. Go directly to Step 3 (tracing integration) and the evaluation workflow. ### Setup Steps (only if environment is NOT pre-configured) 1. **Install MLflow** (version >=3.8.0) 2. **Configure environment** (tracking URI and experiment) - **Guide**: Follow `references/setup-guide.md` Steps 1-2 3. **Integrate tracing** (autolog and @mlflow.trace decorators) - ⚠️ **MANDATORY**: Use the `instrumenting-with-mlflow-tracing` skill for tracing setup - ✓ **VERIFY**: Run `scripts/validate_tracing_runtime.py` after implementing ⚠️ **Tracing must work before evaluation.** If tracing fails, stop and troubleshoot. **Checkpoint - verify before proceeding:** - [ ] MLflow >=3.8.0 installed - [ ] MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID set - [ ] Autolog enabled and @mlflow.trace decorators added - [ ] Test run creates a trace (verify trace ID is not None) **Validation scripts:** ```bash uv run python scripts/validate_environment.py # Check MLflow install, env vars, connectivity uv run python scripts/validate_auth.py # Test authentication before expensive operations ``` ## Evaluation Workflow ### Step 1: Agent Interview (REQUIRED — do not skip) Before doing anything else, ask the user these questions. Do NOT proceed until you have answers. **Required:** 1. "What does your agent do? Describe its purpose in 1-2 sentences." 2. "What are the 2-3 most important things it needs to get right?" 3. "Are there common failure modes you've already noticed?" **Use answers to:** - Derive scorer names and criteria (do not invent them) - Write the `agent_description` parameter for `generate_evals_df` - Set evaluation priorities **If running in automated mode:** Read agent purpose from the codebase (SKILL.md, README, or main entry point docstring). Still surface what you found and confirm before proceeding. ### Step 2: Define Quality Scorers 1. **Check registered scorers in your experiment:** ```bash uv run mlflow scorers list -x $MLFLOW_EXPERIMENT_ID ``` **IMPORTANT: if there are registered scorers in the experiment then they must be used for evaluation.** 2. **Select additional built-in scorers that apply to the agent** See `references/scorers.md` for the built-in scorers. Select any that are useful for assessing the agent's quality and that are not already registered. 3. **Create additional custom scorers as needed** If needed, create additional scorers using the `make_judge()` API. See `references/scorers.md` on how to create custom scorers and `references/scorers-constraints.md` for best practices. > ⚠️ **CRITICAL — Scorer Return Values:** Scorers MUST instruct the LLM judge to return `"yes"` or `"no"` (or booleans/numerics). Return values of `"pass"` or `"fail"` are **silently cast to `None`** by `_cast_assessment_value_to_float` and **excluded from `results.metrics`** with no error or warning — results simply disappear. See `references/scorers-constraints.md` Constraint 2 for the full list of safe vs. broken return values. 4. **REQUIRED: Register new scorers before evaluation** using Python API: ```python from mlflow.genai.judges import make_judge from mlflow.genai.scorers import BuiltinScorerName import os scorer = make_judge(...) # Or, scorer = BuiltinScorerName() scorer.register() ``` ** IMPORTANT: See `references/scorers.md` → "Model Selection for Scorers" to configure the `model` parameter of scorers before registration. ⚠️ **Scorers MUST be registered before evaluation.** Inline scorers that aren't registered won't appear in `mlflow scorers list` and won't be reusable. 5. **Verify registration:** ```bash uv run mlflow scorers list -x $MLFLOW_EXPERIMENT_ID # Should show your scorers ``` ### Step 3: Prepare Evaluation Dataset **ALWAYS discover existing datasets first** to prevent duplicate work: 1. **Run dataset discovery** (mandatory): ```bash uv run python scripts/list_datasets.py # Lists, compares, recommends datasets uv run python scripts/list_datasets.py --format json # Machine-readable output uv run python scripts/list_datasets.py --help # All options ``` 2. **Present findings to user**: - Show all discovered datasets with their characteristics (size, topics covered) - If datasets found, highlight most relevant options based on agent type 3. **Ask user about existing datasets**: - "I found [N] existing evaluation dataset(s). Do you want to use one of these? (y/n)" - If yes: Ask which dataset to use and record the dataset name — skip to Step 3.5 - If no: Proceed to Phase A below **If creating a new dataset, use the two-phase approach below.** --- #### Phase A: Sanity Check (5 questions — always run first) Create a minimal 5-question dataset manually from the Step 1 interview answers. The goal is to confirm the pipeline works end-to-end before investing in large-scale generation. ```python import mlflow from mlflow.genai.datasets import create_dataset # Derive 5 representative questions directly from the agent's stated purpose # and known failure modes identified in Step 1 sanity_records = [ {"inputs": {"query": "<question 1 from interview>"}, "expected_response": "<expected answer>"}, {"inputs": {"query": "<question 2 from interview>"}, "expected_response": "<expected answer>"}, # ... 5 total ] sanity_dataset = create_dataset( records=sanity_records, name="sanity-check-5q", ) ``` Run evaluation on this dataset (see Step 4), then **present results to the user** with this framing: > "This is a sanity check — 5 questions confirm the pipeline works but aren't statistically meaningful. Proceeding to Phase B to generate a proper evaluation set." Only proceed to Phase B once Phase A completes without errors. --- #### Phase B: Proper Evaluation Dataset (100+ questions — run after Phase A passes) Generate questions from the agent's actual corpus rather than inventing them from scratch. The approach depends on whether the project uses Databricks or OSS MLflow. **On Databricks** — use `generate_evals_df` to synthesize questions from the agent's document corpus: ```python from databricks.agents.evals import generate_evals_df, estimate_synthetic_num_evals import mlflow # agent_description comes from Step 1 interview answers agent_description = "<agent purpose from interview>" # docs_df: a Spark or pandas DataFrame with a "content" column containing # the documents/chunks the agent retrieves from (e.g., your Vector Search index) evals = generate_evals_df( docs=docs_df, num_evals=100, agent_description=agent_description, ) # Merge into MLflow dataset — don't create a separate dataset dataset = mlflow.genai.datasets.create_dataset(name="generated-evals-100q") dataset.merge_records(evals) ``` To estimate the right `num_evals` before generating: ```python recommended = estimate_synthetic_num_evals(docs_df) print(f"Recommended num_evals: {recommended}") ``` **Dataset size guidance:** - **<30 questions**: not statistically meaningful — avoid drawing conclusions - **50–100 questions**: adequate for catching regressions, suitable for most agents - **200+ questions**: recommended when comparing model variants or scoring multiple dimensions **On OSS MLflow** — use RAGAS `TestsetGenerator` to generate from your document corpus: ```python from ragas.testset import TestsetGenerator from ragas.llms import LangchainLLMWrapper from ragas.embeddings import LangchainEmbeddingsWrapper generator = TestsetGenerator( llm=LangchainLLMWrapper(your_llm), embedding_model=LangchainEmbeddingsWrapper(your_embeddings), ) testset = generator.generate_with_langchain_docs(docs, testset_size=100) evals_df = testset.to_pandas() # Convert to MLflow dataset schema and merge import mlflow records = [ {"inputs": {"query": row["user_input"]}, "expected_response": row["reference"]} for _, row in evals_df.iterrows() ] dataset = mlflow.genai.datasets.create_dataset(name="generated-evals-100q") dataset.merge_records(records) ``` **If no document corpus is available** — ask the user to provide 50–100 representative queries from production logs or usage history. These are more realistic than synthetic questions and are preferable when available. --- **IMPORTANT**: Do not skip dataset discovery. Always run `list_datasets.py` first, even if you plan to create a new dataset. This prevents duplicate work and ensures users are aware of existing evaluation datasets. **For complete dataset guide:** See `references/dataset-preparation.md` **Checkpoint - verify before proceeding:** - [ ] Scorers have been registered - [ ] Phase A sanity check passed (pipeline runs end-to-end) - [ ] Phase B dataset created with 50+ questions (or existing dataset selected) ### Step 3.5: Dry Run (REQUIRED before full eval) Run evaluation on **3 questions** from the dataset before committing to the full run. This catches broken tools, misconfigured scorers, and auth failures early — before they silently corrupt 100-question results. If you completed Phase A above, the pipeline is already validated — focus the dry run on scorer output only. ```python import mlflow dataset = mlflow.genai.datasets.get_dataset(name="<your-dataset-name>") dry_run_records = dataset.df.head(3) ``` Run `mlflow.genai.evaluate()` on these 3 records using the same wrapper and scorers as the full eval. **For each response, check:** 1. **Tool calls** — Did the agent call any tools? If it called zero tools on questions that require retrieval, tools are likely broken (403s, rate limits, missing credentials). 2. **Response quality** — Are responses empty or generic ("I don't know", "I can't help with that")? Empty responses score as irrelevant and will skew the full eval. 3. **Scorer output** — Did all 3 scores come back as `0` or `None`? If so, the scorer is misconfigured (check return values — `"pass"`/`"fail"` are silently cast to `None`; use `"yes"`/`"no"` instead). **Decision gate:** - **If dry run shows tool failures or empty responses:** Stop. Fix the underlying issue (auth, tool config, retrieval) before proceeding. Do not run the full eval on broken infrastructure. - **If all 3 scorer outputs are 0 or None:** Stop. Debug scorer return values and re-register before proceeding. - **If dry run passes:** Report to the user: *"Dry run passed (3/3 responses non-empty, tools called, scores non-zero). Proceeding to full eval."* Then continue to Step 4. > **Why this matters:** Tool failures (403s from docs scraping, GitHub API rate limits) produce empty agent responses that score as 0. Running a 100-question eval only to discover all tools were failing wastes time and produces misleading results. The dry run catches this in under a minute. --- ### Step 4: Run Evaluation > **Large datasets (50+ questions)?** See `references/throughput-guide.md` for throughput optimization — covers parallelism env vars, async predict functions, and dataset splitting for 200+ question evals. #### 4a. Estimate Runtime Before Starting Before launching evaluation, tell the user how long it will take: 1. **Count the dataset questions:** ```python import mlflow dataset = mlflow.genai.datasets.get_dataset(name="<your-dataset-name>") print(f"Dataset size: {len(dataset.df)} questions") ```