| name | chart-review-calibrate |
| description | Calibrates a draft chart-review guideline before locking by computing per-criterion inter-rater reliability (Cohen's kappa) from blind dual-reviewer samples. Use when the user says "calibrate this draft", "is this guideline ready to lock", "compute kappa for this protocol", "check inter-rater agreement", "validate the rubric", "run a calibration sample", or asks to gate locking on agreement metrics. Reports per-criterion kappa with Landis and Koch buckets, surfaces disagreements with confusion matrices, and writes a calibration report. Acts as the release gate between draft and locked; composes with chart-review-improve to act on criteria that fail the kappa threshold.
|
| metadata | {"author":"chart-review-platform","version":"0.2.0"} |
Guideline Calibration
Pre-lock validation for a draft guideline. Two or more reviewers do blind
chart reviews on the same sample of patients; this skill computes per-criterion
Cohen's κ, surfaces low-agreement criteria for tightening, and writes a
calibration report. It is the release gate between draft and locked — the
methodologist typically requires κ ≥ 0.70 across all primary criteria before
locking.
When to use
- User says "calibrate this draft", "is this guideline ready to lock", "compute
kappa", "check inter-rater agreement", "validate the rubric", or asks to gate
locking on agreement metrics
- After
chart-review-author or chart-review-build produces a draft and the
team has collected blind dual-reviewer samples
- After
chart-review-improve revises low-kappa criteria and a re-calibration
pass is needed
Inputs
- guideline_path: draft (
.claude/skills/chart-review-<id>/, with status: draft in meta.yaml) or locked (for refresh)
- patient_ids: stratified sample of N patients (typical: 10-30)
- reviewer_ids: 2+ reviewers who independently completed the same sample
- Review records: each reviewer's answers captured separately at
reviews/<patient>/<guideline-id>/<reviewer>.json OR via the platform's
blinded-review machinery
Procedure
-
Verify blind-review preconditions. For each patient in the sample, check
that 2+ reviewers' answers exist. Flag and exclude patients with fewer.
-
Read the guideline. Enumerate criteria from
<guideline_path>/criteria/*.yaml. Skip derived fields — derivation is
mechanical and has no inter-rater variance.
-
Per criterion, compute κ:
- Gather
{patient_id, reviewer_a_answer, reviewer_b_answer} triples.
- Use the platform's
kappa MCP tool if available; otherwise compute
manually using the formula in references/kappa-thresholds.md.
- Build the confusion matrix (categories × categories).
-
Bucket criteria by κ using Landis & Koch thresholds — see
references/kappa-thresholds.md. The locking gate is κ ≥ 0.70 on all
primary criteria.
-
For each criterion with κ < 0.70, surface actual disagreements: list
every (patient, reviewer-A, reviewer-B) triple where they disagreed.
Quote the criterion's guidance_prose.examples and note what's ambiguous.
These observations become direct input to chart-review-improve.
-
Write the calibration report at
calibration/<guideline-id>/<run-id>/report.md. Contents:
- Sample metadata (N patients, reviewer IDs, timestamp)
- Summary (criteria calibrated, how many pass / need work, lock recommendation)
- Per-criterion κ table with bucket labels
- For each κ < 0.70 criterion: confusion matrix, list of all disagreements
with quotes, pattern observations, and recommended next step
-
Write calibration/<guideline-id>/<run-id>/raw.json with the structured
data so downstream tools (chart-review-improve, platform QA dashboard) can
consume it.
-
Summarize for the user in 3-5 sentences: per-criterion bucket counts,
blockers (κ < 0.40 criteria), and recommended next step (lock /
improve first / re-sample).
Universal references
- See
skills/chart-review/references/reliability-metrics.md for guidance on
which metric to use per criterion type (weighted κ for ordinal, ICC for
continuous, Jaccard for sets).
- See
skills/chart-review/references/lifecycle.md for the phase transition
that this skill gates (draft → calibrated → locked).
Skill-specific references
references/kappa-thresholds.md — Landis & Koch bucket definitions, the κ
formula with edge cases, minimum sample requirements, and metric selection
by criterion type
Hard rules (with reasons)
- Output goes ONLY to
calibration/<guideline-id>/<run-id>/. Calibration
reports are audit artifacts; writing to other locations breaks the audit trail.
- Skip derived fields. Computing κ on derived fields is meaningless because
their values are deterministic from leaf inputs — including them would
artificially inflate the overall agreement numbers.
- Require ≥ 2 reviewers per patient; exclude patients with fewer. κ is
undefined with a single reviewer; including such patients would corrupt the
computation.
- Flag κ as unreliable when n_shared < 10. Small samples produce wildly
unstable κ estimates; flagging protects the team from making locking decisions
on noise. See
references/kappa-thresholds.md for the threshold.
- Never infer or make up missing answers. If a reviewer didn't touch a
criterion for a patient, exclude that pair from that criterion's computation —
imputing values would misrepresent agreement.
Troubleshooting
Only 1 reviewer's data exists: Can't compute κ with one reviewer. Either
wait for the second reviewer to complete the sample, or run a degraded
"agent vs reviewer" comparison using chart-review-improve instead.
Criterion has only 1 unique answer value in the sample: κ collapses when
Pe = 1. Report κ but flag it: "n unique answers = 1; reliability not
discriminating. Either the criterion is fine or the sample lacks variance.
Suggest stratified resampling to include cases where the criterion's answer
varies."
Reviewers disagree on an applicability gate (one marks not_applicable,
the other gives an answer): Flag these as gate-disagreement cases separately
from answer disagreements. The fix is usually to tighten the upstream
criterion's prose, not the gated one. Surface this in the report under the
upstream criterion.
