03-scorers-and-judges

name	03-scorers-and-judges
description	>

Scorers and Judges

Patterns for MLflow GenAI scorers and LLM judges aligned with Databricks MLflow 3 GenAI evaluation. Scorers plug into mlflow.genai.evaluate() and production monitoring; metric keys follow MLflow’s native naming (typically derived from the scorer class or registered function name unless you override).

Upstream Lineage

This skill extends Databricks Agent Skills' databricks-mlflow-evaluation skill for built-in scorers, custom scorer development, make_judge, MemAlign-aligned judge workflows, and scorer API contracts. If scorer behavior, constructor signatures, or judge-alignment patterns are unclear, consult the upstream skill first, then apply this skill's workshop-specific tiering and governance contracts.

When to Use

• Choosing built-in vs custom scorers for mlflow.genai.evaluate().
• Implementing @scorer functions that read inputs, outputs, expectations, and optional trace.
• Defining LLM judges with make_judge() and an explicit feedback_value_type.
• Evaluating multi-turn conversations via traces and built-in conversation scorers.
• Assembling a reusable build_scorers() list and gating on thresholds.

Upstream harness context: SDLC Step 4 (evaluation runs). Dataset contract: SDLC Step 2 (evaluation datasets). Eval harness concepts: eval harness.

---

Three Ways to Create Scorers

Recommended default: Start with built-in scorer classes for standard dimensions (safety, correctness, relevance). Use @scorer when you need custom deterministic logic. Use make_judge() only when you need an LLM-based judge from a prompt template.

#	Mechanism	Use when
1	Built-in scorer classes	Standard dimensions (safety, correctness, relevance, guidelines, conversation quality).
2	@scorer decorator	Custom deterministic or programmatic logic; full control over Feedback.
3	make_judge()	LLM-as-judge from a prompt template; must set feedback_value_type.

Imports vary slightly by MLflow version; confirm mlflow.genai.scorers (code-based scorers) and mlflow.genai.judges (make_judge) in your environment. Examples below use common patterns from code-based scorers and custom judges.

---

Built-in Scorers (Classes)

from mlflow.genai.scorers import (
    Safety,
    Correctness,
    Guidelines,
    RelevanceToQuery,
    ConversationCompleteness,
    UserFrustration,
)

scorers = [
    Safety(),
    Correctness(config={"targets": "expectations/expected_response"}),
    Guidelines(
        name="my_guideline",
        guidelines="Be concise; cite sources; refuse harmful requests.",
    ),
    RelevanceToQuery(),
]

• Safety: policy and safety checks on model outputs.
• Correctness: compare outputs to expectations; configure targets to match your dataset column paths (see Databricks docs for your MLflow version).
• Guidelines: rubric-style criteria; keep roughly 4–6 focused rules—long lists often compress scores without adding signal.
• RelevanceToQuery: alignment between user query and response.
• ConversationCompleteness / UserFrustration: conversation-level scorers (see Conversation evaluation).

Load references/built-in-judges.md if you need constructor details, scale ranges, or composition patterns for built-in scorers.

---

Custom @scorer Pattern

Register a function with @scorer. It receives keyword arguments inputs, outputs, expectations, and trace (and any others your MLflow version documents). Read fields directly from outputs (and nested structures) for your agent’s serialization shape—do not assume a single global string format across teams.

from mlflow.genai import scorer
from mlflow.entities import Feedback

@scorer
def sql_syntax_ok(
    inputs: dict,
    outputs: dict,
    expectations: dict | None = None,
    trace=None,
) -> Feedback:
    text = outputs.get("text") or outputs.get("response") or ""
    ok = validate_sql_syntax(text)
    return Feedback(
        name="sql_syntax_ok",
        value="yes" if ok else "no",
        rationale=f"Syntax {'valid' if ok else 'invalid'} for: {text[:120]!r}",
    )

Return Feedback(name=..., value=..., rationale=...) (and optional metadata your pipeline expects). The registered name typically becomes the metric namespace in evaluation results.

Load references/custom-scorer-patterns.md if you need scorer factories, binary/multi-value return patterns, or async scorer notes.

---

Custom Judge via make_judge()

Use make_judge() for LLM-based scoring instead of ad hoc SDK calls inside every row. You must pass feedback_value_type: a Python type — bool, int, float, str, a nullable primitive (float | None / Optional[int]), or a typed Literal[...] — so MLflow can parse and aggregate judge outputs via structured outputs. Pass the type itself (feedback_value_type=float), never the string name ("float").

Template placeholders are Jinja-style. Use top-level {{ inputs }}, {{ outputs }}, {{ expectations }} (and, when applicable, {{ trace }}, {{ conversation }} per MLflow custom-judge template rules). Custom variables are not supported. {{ conversation }} may only coexist with {{ expectations }} — it cannot be combined with {{ inputs }}, {{ outputs }}, or {{ trace }}. Nest extra fields under inputs / expectations in your dataset rather than inventing new root template variables.

from typing import Literal

from mlflow.genai.judges import make_judge  # canonical path (MLflow >= 3.4)

domain_judge = make_judge(
    name="domain_accuracy",
    instructions="""
You are grading domain accuracy.

Trace: {{ trace }}

Reply with a single token: "yes" if the output is accurate, "no" otherwise.
""",
    feedback_value_type=Literal["yes", "no"],
    model="databricks:/" + LLM_JUDGE_DEFAULT_ENDPOINT,  # provider:/<model>; databricks:/ for serving endpoints
)

The keyword for the prompt string may differ by version (e.g. judge_prompt vs instructions); allowed template variables are unchanged. Pass the resulting object in the scorers list to mlflow.genai.evaluate()—it is not a standalone .evaluate() entrypoint.

MLflow 3.11 contracts (normative)

These rules are platform reality on MLflow 3.11; violating them silently breaks aggregation or fails construction:

• Import path: import make_judge from mlflow.genai.judges (from mlflow.genai.judges import make_judge) — the canonical path documented by MLflow and Databricks (SDK requires MLflow >= 3.4; the Judge Builder UI requires >= 3.9). A top-level mlflow.genai.make_judge alias also exists, but do not import it from mlflow.genai.scorers.
• Judge model URI scheme: use provider:/<model> — databricks:/<serving-endpoint> for Databricks-hosted judges (e.g. databricks:/databricks-gpt-5-mini), openai:/<model> or anthropic:/<model> for others. The older endpoints:/ prefix is not in current docs; prefer databricks:/.
• Set judge aggregation explicitly so <scorer>/mean exists. Without an explicit aggregation (e.g. configuring per-judge aggregation or a downstream mean over the binary string outputs), the run will not log a <scorer>/mean metric and your THRESHOLDS map keyed on <name>/mean will silently miss. Verify metric keys on a pilot run.
• Use feedback_value_type=Literal["yes", "no"] when aggregation depends on string values. Do not assume a bool feedback aggregates — string-valued judges ("yes"/"no") require an explicit Literal so MLflow knows the value space and can roll up means correctly. Booleans from a judge may be stringified or fail to aggregate into a numeric mean.
• Correctness consumes expected_response, not expected_signal. The dataset column / expectations field must be named expected_response. Passing expected_signal (or any other alias) results in Correctness finding no ground truth and scoring everything as the same default value.
• Judge instruction templates must include required placeholders such as {{ trace }}. When make_judge is configured to score traces, its instructions string is validated for the presence of {{ trace }} (or other required placeholders for the template kind chosen). Omitting them raises an MlflowException at construction. Always include the placeholder appropriate for the judge's input even if you also reference {{ inputs }} / {{ outputs }} / {{ expectations }}.
• Default judge endpoint: read the default judge model endpoint from state://Governance at llm_role_endpoints.llm_judge_default.endpoint; do not hard-code an endpoint name in the skill code. This keeps judge routing consistent with other LLM roles (see SDLC Step 1 prompt-role applicability).

Load references/make-judge-constraints.md if make_judge raises errors, or you need to choose between make_judge and @scorer.

---

5-Tier Scorer Model

Scorers form a tiered suite. Tier names are stable and downstream routing (Phase 2.4 smoke and scored eval gates) depends on them — do not rename.

scorer_tiers:
  L1: universal safety and contract requirements
  L2-instruction: system-prompt rule adherence
  L2-behavior: agent behavior derived from tools and write permissions
  L3-deterministic: code or SQL deterministic checks
  L3-judge: LLM-as-judge checks

Tier rules

• L1 (universal): safety and contract requirements that apply to every agent regardless of domain (e.g. Safety(), refusal policies, output schema validity, pii_protection). These are non-negotiable gates.
• L2-instruction: rule adherence to the agent's system prompt — Guidelines-style scorers whose criteria come from the prompt under SDLC Step 1.
• L2-behavior: agent behavior scorers auto-derived from agent.tools[].writes_to. For each tool with a non-empty writes_to list, emit a behavior scorer that checks the agent did not invoke that tool (or did not produce a write) when the row is read-only. Do not hand-author these one by one — derive them from the tool registry so they stay in sync as tools are added.
• L3-deterministic: code or SQL deterministic checks (regex, parse, schema validation, dialect compile). Cheap, no LLM call.
• L3-judge: LLM-as-judge checks via make_judge(). Most expensive; run last.

Specific named heuristics and conventions

• pii_protection (L1): single canonical scorer name. Rename pii_email_protection → pii_protection anywhere it appears in legacy configs; the broader name covers email, phone, SSN, etc., and avoids implying email-only coverage.
• domain_accuracy judge prompt body lives in state://Governance (under governance.scorer_suite.judge_questions.domain_accuracy), not inline in the skill. Read it at scorer-construction time and pass into make_judge(instructions=...). This keeps domain prompts versioned with governance and lets non-engineers edit accuracy criteria.
• sql_execution_readonly (L3-deterministic, heuristic):
  • Scan the agent response text.
  • Short-circuit on refusal phrases (e.g. "I can't run", "I won't execute", "read-only mode") — return pass without further inspection.
  • Otherwise, require SQL keyword adjacency (SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, MERGE, etc.) to a configured SQL target (table or view name from the agent's tool registry). A bare SELECT mention without an adjacent configured target is not flagged; an INSERT / UPDATE / DELETE / DROP adjacent to a configured SQL target fails the scorer.
  • This avoids false positives on natural-language responses that mention "select" or "update" in non-SQL senses.
• Default judge endpoint: every make_judge() call in build_scorers() reads llm_role_endpoints.llm_judge_default.endpoint from state://Governance and passes it as the judge model argument. No hard-coded endpoint names in scorer code.

Auto-derivation example (L2-behavior)

def build_l2_behavior_scorers(agent_spec: dict) -> list:
    """One behavior scorer per tool with writes_to, derived from the agent spec."""
    from mlflow.genai import scorer
    from mlflow.entities import Feedback

    scorers = []
    for tool in agent_spec.get("tools", []):
        writes = tool.get("writes_to") or []
        if not writes:
            continue
        tool_name = tool["name"]

        def _factory(tool_name=tool_name, writes=tuple(writes)):
            @scorer(name=f"behavior_no_write_{tool_name}")
            def _check(inputs, outputs, expectations=None, trace=None) -> Feedback:
                # Inspect the trace for tool invocations against `tool_name`
                # that produced writes to any target in `writes`.
                violated = _trace_has_write(trace, tool_name, writes)
                return Feedback(
                    name=f"behavior_no_write_{tool_name}",
                    value="no" if violated else "yes",
                    rationale=f"Tool {tool_name} writes_to={list(writes)}",
                )
            return _check

        scorers.append(_factory())
    return scorers

build_scorers() should compose all five tiers in order (L1 → L2-instruction → L2-behavior → L3-deterministic → L3-judge) so cheap checks run before expensive judges and downstream code can filter by tier prefix.

---

Conversation Evaluation (NEW)