04-evaluation-runs

# Evaluation runs (MLflow GenAI) Canonical reference for **evaluation execution**, **threshold gating**, **human feedback**, and **repeatability** when using `mlflow.genai.evaluate()` on Databricks. Grounded in the official MLflow 3 GenAI eval harness, evaluation runs, conversation evaluation, and human feedback documentation. ## Upstream Lineage This skill extends Databricks Agent Skills' `databricks-mlflow-evaluation` skill for `mlflow.genai.evaluate()` execution, regression detection, production trace re-scoring, human labeling loops, and evaluation result analysis. If the eval harness contract or result object behavior is ambiguous, consult the upstream skill first, then preserve this skill's SDLC telemetry and gate-capture requirements. ## When to Use - Measure agent quality on a fixed benchmark with `mlflow.genai.evaluate()`. - Wire `predict_fn`, scorers, and dataset—or use **answer sheet** mode with pre-computed `outputs`. - Gate promote/deploy decisions on judge scores vs thresholds. - Add **retries** for transient harness or infrastructure failures. - **Re-score** existing outputs with new scorers or evaluate pre-collected production traces. - Run **labeling sessions** and sync human annotations into metrics. ## Core Evaluation Flow ```python import mlflow results = mlflow.genai.evaluate( data=eval_dataset, predict_fn=agent_predict_fn, scorers=scorer_list, ) ``` The eval harness runs your predictor (if provided), attaches traces, applies scorers, and returns structured results. See [Evaluation runs](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/evaluation-runs) and [Eval harness](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/eval-harness). ## Trace destination (Unity Catalog) Configure UC trace storage **once** before `evaluate()` so traces persist for labeling and dashboards. See **Skill 07 (Production Monitoring) → Trace Destination** for the full `set_experiment(trace_location=UnityCatalog(...))` pattern and UC permissions. ## predict_fn Contract MLflow passes **one row’s `inputs`** into `predict_fn` (as a dict). The provided evaluation harness (`run_evaluation.py`) accepts both `str` and `dict` returns — it normalizes dicts to strings automatically via the `_out()` wrapper. Built-in scorers (Safety, RelevanceToQuery) work with traces, so a simple `-> str` return is sufficient. **`predict_fn` is optional** when `data` already includes an **`outputs`** column (see Answer sheet mode below). ### Using the provided harness (recommended) Your track's `predict_fn(inputs: dict) -> str` works as-is with `run_evaluation.py`: ```bash # --experiment-path MUST be the user-and-use-case-pinned eval experiment, e.g. # /Users/<user_email>/mlflow/<APP_NAME>-eval # Read it from .vibecoding-state.md (mlflow_experiment_path with -eval leaf swap) # instead of using a literal /Shared/my-agent/traces. uv run run_evaluation.py --predict-module predict_fn.py \ --experiment-path /Users/<user_email>/mlflow/<APP_NAME>-eval \ --dataset-table catalog.schema.benchmarks \ --thresholds '{"safety/mean": 0.7, "relevance_to_query/mean": 0.7}' ``` ### Returning dicts for richer scorer input (advanced) If calling `mlflow.genai.evaluate()` directly (without the harness), return a `dict` to pass richer data to custom scorers: ```python def make_eval_predict_fn(track_fn): """Adapts a track callable for direct mlflow.genai.evaluate() use with dict return.""" def predict_fn(inputs: dict) -> dict: question = inputs["question"] response = track_fn(question) return {"response": response} return predict_fn ``` ### Three common dict shapes (for direct evaluate() use) 1. **Simple Q&A** — `(inputs) -> {"response": str}` ```python def predict_fn(inputs: dict) -> dict: q = inputs["question"] return {"response": my_agent.answer(q)} ``` 2. **RAG** — include retrieval for context-aware scorers: ```python def predict_fn(inputs: dict) -> dict: chunks = retriever.search(inputs["question"]) answer = my_agent.answer(inputs["question"], chunks) return {"response": answer, "retrieved_context": chunks} ``` 3. **Conversation** — `inputs` includes message history; same return shape: ```python def predict_fn(inputs: dict) -> dict: messages = inputs["messages"] # multi-turn history return {"response": my_agent.chat(messages)} ``` For multi-turn evaluation patterns, see [Evaluate conversations](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/evaluate-conversations). ## `Correctness` consumes `expected_response` The built-in `Correctness` scorer reads `expectations["expected_response"]` from the dataset row — **not** `expected_signal`, `expected_answer`, or any other field name. If your benchmark stores the gold answer under a different key, either rename the column to `expected_response` or pass `targets="expectations/<your_field>"` explicitly when constructing the scorer: ```python from mlflow.genai.scorers import Correctness # Default: reads expectations["expected_response"] correctness = Correctness() # Explicit target if your dataset uses a different field name correctness = Correctness(targets="expectations/expected_response") ``` Mismatched field names produce silent `None` rows (the scorer skips, no error) and break threshold gates. See **Skill 02 (Evaluation Datasets)** for the canonical `expected_response` field on dataset rows and **Skill 03 (Scorers and Judges)** for the matching scorer contract. ## Answer sheet evaluation mode When **`data` includes both `outputs` and `expectations`** (and optionally other columns the scorers need), you can call **`evaluate()` without `predict_fn`**. The harness scores existing outputs—useful to **re-score with new scorers** or **evaluate pre-collected production traces**. ```python import mlflow eval_df = ... # columns: inputs, outputs, expectations (and any scorer inputs) eval_result = mlflow.genai.evaluate( data=eval_df, scorers=scorer_list, # predict_fn omitted — outputs column supplies model outputs ) ``` This matches the harness behavior described in [Eval harness](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/eval-harness): pre-computed predictions are scored directly. ## Retry wrapper Wrap `mlflow.genai.evaluate()` with **retry and exponential backoff** for transient failures (timeouts, rate limits, intermittent worker errors). ```python import time TRANSIENT_MARKERS = ("timeout", "temporarily unavailable", "rate limit", "503", "504") def is_retryable_error(exc: BaseException) -> bool: msg = str(exc).lower() return any(m in msg for m in TRANSIENT_MARKERS) def evaluate_with_retry(data, scorers, predict_fn=None, max_retries=4, base_sleep_s=10): for attempt in range(max_retries): try: kwargs = {"data": data, "scorers": scorers} if predict_fn is not None: kwargs["predict_fn"] = predict_fn return mlflow.genai.evaluate(**kwargs) except Exception as e: if attempt >= max_retries - 1: raise if not is_retryable_error(e): raise time.sleep(base_sleep_s * (attempt + 1)) ``` Tune `TRANSIENT_MARKERS`, worker env vars, and max workers per your environment. Optionally fall back to **sequential row-by-row** evaluation if batch mode keeps failing. ## Handling None traces If **`predict_fn` raises** for a row, the harness may record **`None`** for that row’s trace. Do not assume every index has a valid trace. ```python for i, tr in enumerate(eval_result.traces or []): if tr is None: # log row id, skip tagging, or collect for retry continue # use tr.trace_id, etc. ``` ## Extracting results After `evaluate()`, use the returned object’s fields (names align with [Evaluation runs](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/evaluation-runs)): ```python eval_result = mlflow.genai.evaluate(...) metrics = eval_result.metrics # aggregate scorer metrics traces = eval_result.traces # per-row traces; entries may be None table = eval_result.eval_table # tabular join of inputs, outputs, scores ``` Prefer **`eval_table`** for threshold checks on per-row or aggregated columns. ## Score normalization Two scales often appear in the same pipeline: | Role | Typical scale | Example | |------|----------------|---------| | Harness / MLflow metric columns | 0–1 means (`metric/mean`) | `relevance/mean` | | Product thresholds in “points” | 0–100 per judge | dashboard gates | Normalize before comparing: multiply 0–1 by 100 when your gates expect 0–100, or divide thresholds by 100 when comparing to 0–1 columns. Keep **one mental model per gate** so you never compare raw 0–1 scores to 0–100 thresholds without conversion. ## Threshold gate checks See **Skill 03 (Scorers and Judges) → Threshold Checking** for the `all_thresholds_met()` pattern and normalization helpers. Log pass/fail and persist `thresholds_met` on the MLflow run for auditability. ## Failure shape router (normative) When a scored evaluation **fails its gate**, the next iteration step depends on **what kind of failure** it was — not just which scorer regressed. Emit a `failure_shape_classification` payload alongside threshold results, and route iteration based on `primary_shape`: ```yaml failure_shape_classification: primary_shape: enum # one of: instruction | tool_call_empty | retrieval | scorer_calibration | safety_classifier failing_scorers_if_regressed: [string] l1_failures: [string] # L1 = architecture-level (system prompt, role binding, refusal) failing_trace_ids: - trace_id: string failing_scorers: [string] predict_fn_status: string # ok | exception | sentinel ``` ### Routing rules | `primary_shape` | Route to | Pre-condition | |-----------------|----------|---------------| | `instruction` | **Skill 08b (prompt hand-authoring)** | Only if `l1_failures` is empty. If L1 failures exist, route to architecture / system-prompt redesign instead — do **not** paper over an L1 failure with prompt iteration. | | `tool_call_empty` | **Skill 06 direct trace debug** | Symptoms: `UNRESOLVED_COLUMN.WITH_SUGGESTION`, `TABLE_OR_VIEW_NOT_FOUND`, permission-denied, or empty tool output. Fix the data/grant/SQL-grounding issue before re-running eval. | | `retrieval` | **Retrieval tuning** (chunking, reranker, top-k, embeddings) | Failing scorers are retrieval-shaped (`groundedness`, `retrieval_relevance`, `context_precision`). Do not iterate the system prompt. | | `scorer_calibration` | **Skill 03 (Scorers and Judges)** | Judge disagrees with human labels at >X%. Fix the scorer prompt, aggregation, or `feedback_value_type` before treating the eval signal as ground truth. | | `safety_classifier` | **Endpoint audit and role re-binding** | Safety scorer regressed because the scoring endpoint is the wrong model or hit a guardrail. Audit `llm_role_endpoints.llm_judge_safety` binding before iterating the agent. | **Hard rule:** never route an L1 failure to Skill 08b. L1 means architecture/role-binding/refusal — instruction iteration cannot fix it. Mis-routing here is the single most expensive failure mode in the SDLC. ## Eval telemetry contract (normative) Every scored evaluation run must capture and persist the following fields on the MLflow run (as tags, params, or artifact JSON — pick one and stay consistent): | Field | Type | Meaning | |-------|------|---------| | `failing_trace_ids` | `[{trace_id, failing_scorers, predict_fn_status}]` | Per-row failure detail. Required for routing and for re-running iteration on a focused subset. | | `safety_buffer` | `{<scorer_name>: float}` | Margin between observed metric and gate threshold (positive = passing with headroom; negative = failing). Lets the next iteration step know how close to the cliff each scorer is. | | `predict_fn_exception_count` | `int` | Total rows where `predict_fn` raised. Non-zero values mean some scorer means are computed over a smaller denominator than the dataset row count. | | `predict_fn_sentinel_count_per_run` | `int` | Rows that returned a sentinel string (e.g. `"INPUT_GUARDRAIL_BLOCKED"`, `"LAKEBASE_COLD_START_FAILED"`). Not the same as exceptions — sentinels successfully return but represent productized debt. See `debt: predict_fn_input_guardrail_sentinel`. | | `judges_with_silent_aggregation_dropouts` | `[string]` | Judges where `<scorer>/mean` is missing from `eval_result.metrics` because aggregation defaulted to `[]`. Must be empty before promoting. See **Skill 03 → make_judge aggregation contract**. | | `mlflow_eval_predict_fn_signature` | `string` | The exact signature the harness saw, e.g. `(inputs: dict) -> str` or `(inputs: dict) -> dict`. Captured because mismatched signatures are the most common cause of empty / `None` traces. | | `mlflow_eval_known_quality_issues` | `[{issue_id, owner_prompt_role, target_prompt_role, status}]` | Open quality issues against this evaluation run. If any item has `target_prompt_role: first_scored_eval` and `status != closed`, the gate **must fail closed** until the item is resolved or explicitly waived. | Capture in a single payload per run so dashboards and the iteration router can read it as one unit: ```python import json import mlflow eval_telemetry = { "failing_trace_ids": [...], "safety_buffer": {"safety/mean": 0.04, "correctness/mean": -0.12}, "predict_fn_exception_count": 0, "predict_fn_sentinel_count_per_run": 2, "judges_with_silent_aggregation_dropouts": [], "mlflow_eval_predict_fn_signature": "(inputs: dict) -> str", "mlflow_eval_known_quality_issues": [], }