summarize-eval

name	summarize-eval
description	Summarize a CXR eval result directory (or compare two) as a Markdown table with metrics as columns and confidence bins as rows. Invoke when the user points at one or two result-dirs and asks for a summary, calibration analysis, or comparison.

summarize-eval

Single result-dir

uv run python .github/skills/summarize-eval/eval_summary.py --result-dir <result-dir>

Required output, in this order:

1. The script's stdout, copied verbatim (the Markdown tables — do not edit, re-format, drop, or re-render them).
2. The analysis bullets below.

Analysis. One bold headline sentence, then 2–5 free-form bullets. Classification leads when present.

• Headline (bold, 1 sentence). What's true about this run? Lead with the classification verdict (or cxr_metric verdict if classification is absent). Cite the macro-F1 / sensitivity / specificity numbers (or the strongest cxr_metric movement) that support it. No hedging in the headline.
• Bullets. Cover, in whatever order makes the story clearest:
  • Calibration — high vs medium (and low if n ≥ 10) on macro-F1 and sensitivity. Well-calibrated, flat, or inverted? Cite the gap.
  • Agreement — do cxr_metric numbers point the same direction as the classification verdict? Name the metric that agrees most, or call out a disagreement explicitly.
  • Caveats — any bucket with n < 10, or suspicious values (specificity == sensitivity, RadCliQ-v1 negative, SEMB near zero).
• Write bullets as direct observations, not labeled headers ("SEMB dropped -0.076…" not "SEMB: dropped …"). Skip any bullet that has nothing to say. Don't write a "summary" or "overall" bullet — the headline does that. Don't re-hedge magnitudes already in the headline.

Two result-dirs (comparison)

uv run python .github/skills/summarize-eval/eval_summary.py \
    --baseline <baseline-dir> --candidate <candidate-dir>