04-evaluation-runs

name	04-evaluation-runs
description	>

Evaluation runs (MLflow GenAI)

Canonical reference for evaluation execution, threshold gating, human feedback, and repeatability when using mlflow.genai.evaluate() on Databricks. Grounded in the official MLflow 3 GenAI eval harness, evaluation runs, conversation evaluation, and human feedback documentation.

Upstream Lineage

This skill extends Databricks Agent Skills' databricks-mlflow-evaluation skill for mlflow.genai.evaluate() execution, regression detection, production trace re-scoring, human labeling loops, and evaluation result analysis. If the eval harness contract or result object behavior is ambiguous, consult the upstream skill first, then preserve this skill's SDLC telemetry and gate-capture requirements.

When to Use

• Measure agent quality on a fixed benchmark with mlflow.genai.evaluate().
• Wire predict_fn, scorers, and dataset—or use answer sheet mode with pre-computed outputs.
• Gate promote/deploy decisions on judge scores vs thresholds.
• Add retries for transient harness or infrastructure failures.
• Re-score existing outputs with new scorers or evaluate pre-collected production traces.
• Run labeling sessions and sync human annotations into metrics.

Core Evaluation Flow

import mlflow

results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=agent_predict_fn,
    scorers=scorer_list,
)

The eval harness runs your predictor (if provided), attaches traces, applies scorers, and returns structured results. See Evaluation runs and Eval harness.

Trace destination (Unity Catalog)

Configure UC trace storage once before evaluate() so traces persist for labeling and dashboards. See Skill 07 (Production Monitoring) → Trace Destination for the full set_experiment(trace_location=UnityCatalog(...)) pattern and UC permissions.

predict_fn Contract

MLflow passes one row’s inputs into predict_fn (as a dict). The provided evaluation harness (run_evaluation.py) accepts both str and dict returns — it normalizes dicts to strings automatically via the _out() wrapper. Built-in scorers (Safety, RelevanceToQuery) work with traces, so a simple -> str return is sufficient.

predict_fn is optional when data already includes an outputs column (see Answer sheet mode below).

Using the provided harness (recommended)

Your track's predict_fn(inputs: dict) -> str works as-is with run_evaluation.py:

# --experiment-path MUST be the user-and-use-case-pinned eval experiment, e.g.
#   /Users/<user_email>/mlflow/<APP_NAME>-eval
# Read it from .vibecoding-state.md (mlflow_experiment_path with -eval leaf swap)
# instead of using a literal /Shared/my-agent/traces.
uv run run_evaluation.py --predict-module predict_fn.py \
  --experiment-path /Users/<user_email>/mlflow/<APP_NAME>-eval \
  --dataset-table catalog.schema.benchmarks \
  --thresholds '{"safety/mean": 0.7, "relevance_to_query/mean": 0.7}'

Returning dicts for richer scorer input (advanced)

If calling mlflow.genai.evaluate() directly (without the harness), return a dict to pass richer data to custom scorers:

def make_eval_predict_fn(track_fn):
    """Adapts a track callable for direct mlflow.genai.evaluate() use with dict return."""

    def predict_fn(inputs: dict) -> dict:
        question = inputs["question"]
        response = track_fn(question)
        return {"response": response}

    return predict_fn

Three common dict shapes (for direct evaluate() use)

1. Simple Q&A — (inputs) -> {"response": str}

def predict_fn(inputs: dict) -> dict:
    q = inputs["question"]
    return {"response": my_agent.answer(q)}

2. RAG — include retrieval for context-aware scorers:

def predict_fn(inputs: dict) -> dict:
    chunks = retriever.search(inputs["question"])
    answer = my_agent.answer(inputs["question"], chunks)
    return {"response": answer, "retrieved_context": chunks}

3. Conversation — inputs includes message history; same return shape:

def predict_fn(inputs: dict) -> dict:
    messages = inputs["messages"]  # multi-turn history
    return {"response": my_agent.chat(messages)}

For multi-turn evaluation patterns, see Evaluate conversations.

Correctness consumes expected_response

The built-in Correctness scorer reads expectations["expected_response"] from the dataset row — not expected_signal, expected_answer, or any other field name. If your benchmark stores the gold answer under a different key, either rename the column to expected_response or pass targets="expectations/<your_field>" explicitly when constructing the scorer:

from mlflow.genai.scorers import Correctness

# Default: reads expectations["expected_response"]
correctness = Correctness()

# Explicit target if your dataset uses a different field name
correctness = Correctness(targets="expectations/expected_response")

Mismatched field names produce silent None rows (the scorer skips, no error) and break threshold gates. See Skill 02 (Evaluation Datasets) for the canonical expected_response field on dataset rows and Skill 03 (Scorers and Judges) for the matching scorer contract.

Answer sheet evaluation mode

When data includes both outputs and expectations (and optionally other columns the scorers need), you can call evaluate() without predict_fn. The harness scores existing outputs—useful to re-score with new scorers or evaluate pre-collected production traces.

import mlflow

eval_df = ...  # columns: inputs, outputs, expectations (and any scorer inputs)

eval_result = mlflow.genai.evaluate(
    data=eval_df,
    scorers=scorer_list,
    # predict_fn omitted — outputs column supplies model outputs
)

This matches the harness behavior described in Eval harness: pre-computed predictions are scored directly.

Retry wrapper

Wrap mlflow.genai.evaluate() with retry and exponential backoff for transient failures (timeouts, rate limits, intermittent worker errors).

import time

TRANSIENT_MARKERS = ("timeout", "temporarily unavailable", "rate limit", "503", "504")


def is_retryable_error(exc: BaseException) -> bool:
    msg = str(exc).lower()
    return any(m in msg for m in TRANSIENT_MARKERS)


def evaluate_with_retry(data, scorers, predict_fn=None, max_retries=4, base_sleep_s=10):
    for attempt in range(max_retries):
        try:
            kwargs = {"data": data, "scorers": scorers}
            if predict_fn is not None:
                kwargs["predict_fn"] = predict_fn
            return mlflow.genai.evaluate(**kwargs)
        except Exception as e:
            if attempt >= max_retries - 1:
                raise
            if not is_retryable_error(e):
                raise
            time.sleep(base_sleep_s * (attempt + 1))

Tune TRANSIENT_MARKERS, worker env vars, and max workers per your environment. Optionally fall back to sequential row-by-row evaluation if batch mode keeps failing.

Handling None traces

If predict_fn raises for a row, the harness may record None for that row’s trace. Do not assume every index has a valid trace.

for i, tr in enumerate(eval_result.traces or []):
    if tr is None:
        # log row id, skip tagging, or collect for retry
        continue
    # use tr.trace_id, etc.

Extracting results

After evaluate(), use the returned object’s fields (names align with Evaluation runs):

eval_result = mlflow.genai.evaluate(...)

metrics = eval_result.metrics  # aggregate scorer metrics
traces = eval_result.traces  # per-row traces; entries may be None
table = eval_result.eval_table  # tabular join of inputs, outputs, scores

Prefer eval_table for threshold checks on per-row or aggregated columns.

Score normalization

Two scales often appear in the same pipeline:

Role	Typical scale	Example
Harness / MLflow metric columns	0–1 means (metric/mean)	relevance/mean
Product thresholds in “points”	0–100 per judge	dashboard gates

Normalize before comparing: multiply 0–1 by 100 when your gates expect 0–100, or divide thresholds by 100 when comparing to 0–1 columns. Keep one mental model per gate so you never compare raw 0–1 scores to 0–100 thresholds without conversion.

Threshold gate checks

See Skill 03 (Scorers and Judges) → Threshold Checking for the all_thresholds_met() pattern and normalization helpers. Log pass/fail and persist thresholds_met on the MLflow run for auditability.

Failure shape router (normative)

When a scored evaluation fails its gate, the next iteration step depends on what kind of failure it was — not just which scorer regressed. Emit a failure_shape_classification payload alongside threshold results, and route iteration based on primary_shape:

failure_shape_classification:
  primary_shape: enum  # one of: instruction | tool_call_empty | retrieval | scorer_calibration | safety_classifier
  failing_scorers_if_regressed: [string]
  l1_failures: [string]              # L1 = architecture-level (system prompt, role binding, refusal)
  failing_trace_ids:
    - trace_id: string
      failing_scorers: [string]
      predict_fn_status: string      # ok | exception | sentinel

Routing rules

primary_shape	Route to	Pre-condition
instruction	Skill 08b (prompt hand-authoring)	Only if l1_failures is empty. If L1 failures exist, route to architecture / system-prompt redesign instead — do not paper over an L1 failure with prompt iteration.
tool_call_empty	Skill 06 direct trace debug	Symptoms: UNRESOLVED_COLUMN.WITH_SUGGESTION, TABLE_OR_VIEW_NOT_FOUND, permission-denied, or empty tool output. Fix the data/grant/SQL-grounding issue before re-running eval.
retrieval	Retrieval tuning (chunking, reranker, top-k, embeddings)	Failing scorers are retrieval-shaped (groundedness, retrieval_relevance, context_precision). Do not iterate the system prompt.
scorer_calibration	Skill 03 (Scorers and Judges)	Judge disagrees with human labels at >X%. Fix the scorer prompt, aggregation, or feedback_value_type before treating the eval signal as ground truth.
safety_classifier	Endpoint audit and role re-binding	Safety scorer regressed because the scoring endpoint is the wrong model or hit a guardrail. Audit llm_role_endpoints.llm_judge_safety binding before iterating the agent.

Hard rule: never route an L1 failure to Skill 08b. L1 means architecture/role-binding/refusal — instruction iteration cannot fix it. Mis-routing here is the single most expensive failure mode in the SDLC.

Eval telemetry contract (normative)

Every scored evaluation run must capture and persist the following fields on the MLflow run (as tags, params, or artifact JSON — pick one and stay consistent):

Field	Type	Meaning
failing_trace_ids	[{trace_id, failing_scorers, predict_fn_status}]	Per-row failure detail. Required for routing and for re-running iteration on a focused subset.
safety_buffer	{<scorer_name>: float}	Margin between observed metric and gate threshold (positive = passing with headroom; negative = failing). Lets the next iteration step know how close to the cliff each scorer is.
predict_fn_exception_count	int	Total rows where predict_fn raised. Non-zero values mean some scorer means are computed over a smaller denominator than the dataset row count.
predict_fn_sentinel_count_per_run	int	Rows that returned a sentinel string (e.g. "INPUT_GUARDRAIL_BLOCKED", "LAKEBASE_COLD_START_FAILED"). Not the same as exceptions — sentinels successfully return but represent productized debt. See debt: predict_fn_input_guardrail_sentinel.
judges_with_silent_aggregation_dropouts	[string]	Judges where <scorer>/mean is missing from eval_result.metrics because aggregation defaulted to []. Must be empty before promoting. See Skill 03 → make_judge aggregation contract.
mlflow_eval_predict_fn_signature	string	The exact signature the harness saw, e.g. (inputs: dict) -> str or (inputs: dict) -> dict. Captured because mismatched signatures are the most common cause of empty / None traces.
mlflow_eval_known_quality_issues	[{issue_id, owner_prompt_role, target_prompt_role, status}]	Open quality issues against this evaluation run. If any item has target_prompt_role: first_scored_eval and status != closed, the gate must fail closed until the item is resolved or explicitly waived.

Capture in a single payload per run so dashboards and the iteration router can read it as one unit:

import json
import mlflow

eval_telemetry = {
    "failing_trace_ids": [...],
    "safety_buffer": {"safety/mean": 0.04, "correctness/mean": -0.12},
    "predict_fn_exception_count": 0,
    "predict_fn_sentinel_count_per_run": 2,
    "judges_with_silent_aggregation_dropouts": [],
    "mlflow_eval_predict_fn_signature": "(inputs: dict) -> str",
    "mlflow_eval_known_quality_issues": [],
}