| name | test-the-measuring-stick |
| description | Check whether a score, grade, or ranking actually has teeth before you trust it. Use when relying on any quality number: an AI grading its own or another AI's output, a rubric score on writing, a relevance ranking, a benchmark result. Triggers include "is this quality score real", "does my grader/eval actually work", "can I trust this ranking", "does this benchmark mean anything". Produces a dummy-answer result, a cheap-floor comparison, and a verdict on whether the measure is trustworthy. |
Test the Measuring Stick
Before you trust what a score says about the work, test what the score says about garbage. A grader that rates a worthless answer highly is not measuring quality: it is measuring something else, and every "good" score it gave is unverifiable. This finds that out cheaply.
Use it whenever a number is about to drive a decision, especially when an AI assigned the number.
1. Build the laziest fake answer
Construct a response that does zero real work but has the right shape:
- Copy the question back as the "answer".
- Return an empty-but-correctly-formatted result.
- Give a single canned, content-free line.
The goal is something that should obviously score near the bottom if the grader works.
2. Run the dummy through the exact same scoring
Use the identical grader, rubric, prompt, or ranking, unchanged. No special-casing. Whatever scored the real work scores the dummy.
3. Read what the dummy score tells you
- Dummy scores low → the stick has teeth. The real scores mean something.
- Dummy scores high → the stick is broken. A presence/format/length signal is being rewarded instead of quality. The real scores are now unverified, not good. Treat every one of them as suspect.
A format or structure bonus is the usual culprit: it can float an empty answer to the top.
4. Check for a saturated ceiling
If almost everything (good, mediocre, dummy) clusters near the top, the test is too easy to separate quality at all. Reaching for a bigger, more expensive grader to push the winners higher is p-hacking, not science. Make the test harder, not the grader stronger.
