calibration-audit

name	calibration-audit
description	Use this to compare what was expected against what actually happened, and find where confidence systematically outruns outcomes.

Calibration Audit

The Big Idea

Calibration is a measurable behavior, not a tone of voice. Assertive language proves nothing. Graded predictions against outcomes prove everything.

Uncertainty Handling Inventory

When the corpus shows the subject did not know something, classify the response: asked for evidence, researched, made assumptions, requested confirmation, ran a small experiment, tested a hypothesis, acted immediately, escalated to another model.

Count responses per task type. The default move under uncertainty is a fingerprint.

Expected vs Actual

Harvest expectations implicit in plans and statements: "this should take an hour," "this approach will work," "the bug is in X."

Grade them:

• expected difficulty vs actual difficulty
• expected solution path vs path taken
• predicted failure points vs actual failure points
• estimated time vs elapsed time, where timestamps allow

Output

Two lists, stated plainly:

1. Domains where perceived certainty systematically exceeds outcomes.
2. Domains where the subject undersells themselves - outcomes beat their own expectations.

When It Backfires

• Reading confident tone as confident prediction. Separate linguistic style from calibration explicitly.
• Grading predictions the model made. Grade only USER INITIATED expectations.
• Demanding numeric precision the data cannot give. State uncertainty ranges instead of inventing decimals.

One-Line Memory

Grade expectations, not grammar.