llm-judge

name	llm-judge
description	Use an LLM as an automated judge to score responses against a rubric. Use when you need nuanced quality evaluation that can't be done with exact match or regex. Triggers include "score with AI", "llm-judge", "rubric-based scoring", "quality evaluation", or any need to rate free-form responses.

llm-judge

Use an LLM model to score responses against a natural language rubric. Returns a 0.0-1.0 score with reasoning.

How it works

The judge model receives:

1. The rubric describing what a good response looks like
2. The actual response to evaluate

The judge returns a JSON object: {"score": 0.0-1.0, "reason": "brief explanation"}

Configuration in YAML

scoring:
  - method: llm-judge
    rubric: "The answer clearly explains the return policy and includes the timeframe"
    threshold: 0.8       # score >= 0.8 = pass (default: 0.7)
    weight: 2            # double weight vs other scorers (default: 1)
    endpoint: openai-prod  # override default judge endpoint
    model: gpt-4o-mini    # override default judge model

Rubric Writing Tips

Write rubrics as positive statements of what a good response contains:

Good rubric:

"The response mentions the 30-day return window and explains the condition requirement"

Bad rubric (vague):

"The response is good"

Good rubric for tone:

"The response is empathetic, acknowledges the customer's frustration, and provides a concrete next step"

Default Configuration

Set globally in Settings or environment:

• Default judge endpoint: openai-prod (configurable)
• Default judge model: gpt-4o-mini (configurable - cheaper models work well for judging)

Prompt Injection Protection

Case input is wrapped in XML delimiters before insertion into the judge prompt to prevent injection attacks:

<case_input>
{user's case input}
</case_input>

Cost Considerations

llm-judge makes one additional LLM call per scored case. For 100 test cases, each with one judge scorer, you incur 100 extra LLM calls. Use gpt-4o-mini or claude-3-haiku as the judge to minimize cost.