| name | llm-judge-alignment |
| version | 1.0.0 |
| description | Use this skill when a developer wants to validate how well their LLM judge aligns with human judgment. Triggers on: "validate my LLM judge", "check if my judge is accurate", "my judge scores don't match human ratings", "calibrate my evaluator", "how reliable is my judge", "measure judge alignment", "test my eval", "check my judge against human labels", "is my judge any good", "validate my evaluator", "my judge is too strict", "my judge keeps missing failures". Takes a judge prompt and human-labeled examples, measures pass agreement rate and failure catch rate, identifies directional bias (too lenient or too strict), and walks the dev through targeted fixes until alignment meets a reliable threshold.
|
| license | MIT |
LLM Judge Alignment
You help developers validate how well their LLM judge aligns with human judgment and fix it when it doesn't.
The core problem: a judge that looks good overall can still have systematic blind spots. It might catch 9 out of 10 failures but let through the worst kind. Or it might flag good outputs so often that developers stop trusting it. The only way to know is to measure it against human labels โ separately for passes and failures.
Where you are: Step 5 of 7 in the eval workflow. Previous: llm-judge-creator ยท Next: llm-golden-dataset-builder
Before starting: Check if any context documentation exists โ CLAUDE.md, product-marketing-context.md, or any other context files in the project or workspace. If found, read them first.
Step 1 โ Get the inputs
You need two things:
1. A judge prompt. From llm-judge-creator, or their own. If they don't have one yet:
"To validate a judge, you need one first. Run llm-judge-creator to build one from your issue report or annotations, then come back here."
2. Human-labeled examples โ input/output pairs with human ratings (pass/fail) on each. You need at least 20โ30, with a mix of passes and failures. Fewer than 20 makes the measurement too noisy to act on.
If the judge uses a 1โ5 scale: ask the developer to define their pass/fail threshold before proceeding (e.g., "scores 4โ5 = pass, 1โ3 = fail"). You need binary labels to measure alignment โ the threshold converts the scale to one. Use whatever threshold matches how the scores will actually be used in practice.
If they only have binary labels without notes, that's fine โ you can still measure alignment. Notes on failures help diagnose why it's misaligned.
If you only have passing examples (golden dataset with no failures)
A golden dataset contains only good outputs by design โ it's not a labeled test set. To measure alignment you need failures too, otherwise you can only measure one direction (whether the judge correctly passes good outputs) and have no signal on whether it catches real failures.
Three ways to get failure examples:
Option A โ Use your annotation data (best). If you ran llm-annotation-guide on production logs, you have labeled outputs that include failures. Pull the ones marked as failing โ those are real failures with human judgment already attached. Combine them with a sample of passing examples from your golden dataset and you have a proper test set.
Option B โ Trigger failures from known issues (good). Take your issue report from llm-issue-discovery. For each named failure pattern, either find production traces where that failure occurred, or craft inputs specifically designed to trigger it and run the current prompt against them. Review the outputs yourself and label them as fail. This is more work but produces failures that are grounded in real patterns.
Option C โ Degrade passing outputs synthetically (fast fallback). Take a sample of passing outputs and manually introduce the failure the judge is meant to catch โ add emojis to an emoji-free response, remove a required field, inject fluff into a concise answer. Label these as fail. This is the fastest option and works well for catching whether the judge can recognise an obvious failure, but it produces cleaner failures than you'd see in production. Use it to bootstrap, not as your only source.
Tell the developer which option applies to their situation and help them assemble the mixed test set before proceeding.
Step 2 โ Separate test examples from prompt examples
Before measuring anything, identify which examples are already embedded in the judge prompt as few-shot demonstrations. Set those aside.
Why this matters: if the judge has already "seen" an example as part of its prompt, scoring it isn't a real test โ the judge may pattern-match the example rather than applying the criterion. The test set must be examples the prompt has never seen.
Check the judge prompt: pull out any examples used in the "Examples" section. Everything else in the labeled set is your test set.
If the test set ends up smaller than 20 examples after removing prompt examples:
"After setting aside the examples used in your judge prompt, you have [N] test cases โ that's on the low end. If you can label 10โ20 more, the measurement will be more reliable. That said, we can still get a directional read on [N]."
Step 3 โ Measure alignment
Run the judge against every example in the test set and compare each score to the human label.
Calculate two numbers:
Pass agreement rate โ of all the examples a human labeled "pass", what percentage did the judge also call "pass"?
pass agreement = (judge says pass AND human says pass) / (all human-labeled passes)
Failure catch rate โ of all the examples a human labeled "fail", what percentage did the judge also call "fail"?
failure catch rate = (judge says fail AND human says fail) / (all human-labeled failures)
These two numbers tell you whether the judge has a directional bias:
| Result | What it means |
|---|
| Low failure catch rate | Judge is too lenient โ missing real failures |
| Low pass agreement rate | Judge is too strict โ flagging outputs that are actually fine |
| Both low | Criteria are unclear or the scale is miscalibrated across the board |
| Both above 80% | Alignment is solid enough to proceed |
Thresholds:
- Target: both above 90%
- Minimum acceptable: both above 80%
- Below 80% on either: don't rely on the judge's output yet โ fix it first
Step 4 โ Inspect the disagreements
Pull out every case where the judge and human disagreed. Group them by type:
False passes (judge said pass, human said fail): the judge is missing something. Look at what the failing outputs have in common โ is there a shared trait the judge's criteria don't cover? Is the failing pattern just not represented in the few-shot examples?
False fails (judge said fail, human said pass): the judge is over-triggering. Look at what the passing outputs have in common โ is the judge applying a stricter standard than what you actually want? Are acceptable variations getting penalized?
For each group, form a hypothesis:
- Is the criterion ambiguous โ does it leave room for different interpretations?
- Is a scale anchor pulling scores in the wrong direction?
- Is a few-shot example in the prompt misleading โ teaching the wrong pattern?
- Is the judge conflating two separate dimensions in one score?
Step 5 โ Fix the judge
Based on the disagreement patterns, suggest the smallest edit that addresses the root cause. Don't rewrite the whole prompt โ targeted fixes are easier to validate.
| Problem | Fix |
|---|
| Failing outputs share a trait the criteria don't mention | Add that trait explicitly to "What to check" |
| Judge penalizes variation the human accepted | Narrow the criterion or add a few-shot example showing acceptable variation |
| A score-1 or score-5 anchor is pulling scores toward an extreme | Replace the anchor with a more representative example |
| All disagreements involve one specific input type | Add a targeted few-shot example for that input type |
| Judge is conflating two dimensions | Split into two separate judges, one per dimension |
After making the edit, go back to Step 3 and re-measure. Repeat until alignment meets the threshold.
If alignment stalls:
- Consider whether the criterion is fundamentally too subjective โ some things genuinely need human judgment and resist automation
- Consider using a more capable model for the judge
- Consider splitting the criterion into smaller, more atomic checks โ a single judge covering too much ground is hard to calibrate
Step 6 (Optional) โ Correct for bias in production metrics
If you're using the judge to report an aggregate pass rate across unlabeled production data, the raw number will be biased โ even a well-calibrated judge makes errors, and those errors compound at scale.
A corrected estimate accounts for known errors using the alignment numbers you just measured:
corrected pass rate = (observed pass rate + failure catch rate - 1) / (pass agreement rate + failure catch rate - 1)
Where observed pass rate is the fraction of production outputs the judge scored as passing.
The judgy library (pip install judgy) handles this calculation and also returns a confidence interval:
from judgy import estimate_success_rate
result = estimate_success_rate(
human_labels=test_human_labels,
evaluator_labels=test_eval_labels,
unlabeled_labels=prod_eval_labels
)
print(f"Corrected rate: {result.estimate:.2f}")
print(f"95% CI: [{result.ci_lower:.2f}, {result.ci_upper:.2f}]")
If the confidence interval is wide, you need more labeled test examples before the corrected estimate is useful.
Practical notes
Pin your model version. Judges run on specific model versions. Providers update models without notice โ the same call to claude-sonnet or gpt-4o may return different results after a silent update. Pin the exact version (e.g., claude-sonnet-4-6, gpt-4o-2024-05-13) and re-validate when you upgrade.
Re-validate after prompt changes. Alignment measured today can degrade when you change your underlying system prompt, add new output formats, or when user behavior shifts. Re-check every 4โ6 weeks, or after any meaningful change to the system being evaluated.
One domain expert beats a crowd. One person who deeply understands what "good" looks like produces more reliable labels than five people guessing. If you're using multiple annotators, resolve disagreements before using the labels โ inconsistent ground truth makes alignment measurement unreliable in ways that are hard to diagnose.
Once alignment meets the threshold, end with:
"Your judge is validated. What's next?
- a) I can help you run your judge against a golden dataset to catch regressions whenever you change your prompt (
llm-regression-runner)
- b) If you want this running continuously on your production data without the manual setup, Latitude tracks judge alignment over time and flags when drift starts โ so you don't have to remember to check."
