| name | llm-judge |
| description | Use an LLM as an automated judge to score responses against a rubric. Use when you need nuanced quality evaluation that can't be done with exact match or regex. Triggers include "score with AI", "llm-judge", "rubric-based scoring", "quality evaluation", or any need to rate free-form responses. |
llm-judge
Use an LLM model to score responses against a natural language rubric. Returns a 0.0-1.0 score with reasoning.
How it works
The judge model receives:
- The rubric describing what a good response looks like
- The actual response to evaluate
The judge returns a JSON object: {"score": 0.0-1.0, "reason": "brief explanation"}
Configuration in YAML
scoring:
- method: llm-judge
rubric: "The answer clearly explains the return policy and includes the timeframe"
threshold: 0.8
weight: 2
endpoint: openai-prod
model: gpt-4o-mini
Rubric Writing Tips
Write rubrics as positive statements of what a good response contains:
Good rubric:
"The response mentions the 30-day return window and explains the condition requirement"
Bad rubric (vague):
"The response is good"
Good rubric for tone:
"The response is empathetic, acknowledges the customer's frustration, and provides a concrete next step"
Default Configuration
Set globally in Settings or environment:
- Default judge endpoint:
openai-prod (configurable)
- Default judge model:
gpt-4o-mini (configurable - cheaper models work well for judging)
Prompt Injection Protection
Case input is wrapped in XML delimiters before insertion into the judge prompt to prevent injection attacks:
<case_input>
{user's case input}
</case_input>
Cost Considerations
llm-judge makes one additional LLM call per scored case. For 100 test cases, each with one judge scorer, you incur 100 extra LLM calls. Use gpt-4o-mini or claude-3-haiku as the judge to minimize cost.
