model-evaluation

name	model-evaluation
description	Compute and report task-correct held-out metrics for a trained medical-imaging model — segmentation (Dice plus a boundary metric such as HD95 or NSD, per structure), classification (AUROC plus AUPRC and sensitivity/specificity with bootstrap CIs at the deployment prevalence), detection (FROC or mAP with a stated IoU criterion), interactive/promptable segmentation (the interaction-count, convergence, and per-case-time axes a static Dice omits), or generative/synthesis image evaluation (similarity plus the downstream-task efficacy similarity alone cannot establish) — plus calibration and subgroup slices. Emits a per-case results table that analyze-stats turns into publication tables, and gates the metric choice against Metrics Reloaded, CLAIM 2024, and Park et al. 2024 (no pixel accuracy for segmentation, no bare accuracy under imbalance, no static Dice for an interactive method, no similarity-only claim for a generative model). Numbers come only from executed code, never hand-typed.
triggers	model evaluation, held-out metrics, test set metrics, Dice, HD95, NSD, surface distance, Metrics Reloaded, AUROC, AUPRC, bootstrap CI, calibration, ECE, reliability diagram, subgroup analysis, slice metrics, mAP, FROC, segmentation metrics, detection metrics, evaluate predictions, interactive segmentation, promptable segmentation, SAM2, MedSAM2, nnInteractive, number of clicks, NoC, interactions-to-threshold, click budget, generative metrics, image synthesis, SSIM, PSNR, SNR, CNR, downstream task, multiclass classification, Obuchowski index, Harrell's C, time-dependent ROC
tools	Read, Write, Edit, Bash, Grep, Glob
model	inherit

Model-Evaluation Skill

Purpose

This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It emits a per-case metric table that the publication statistics build on, and gates the metric choice against Metrics Reloaded (Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024.

It sits between /model-validation (which audits the split / design) and /analyze-stats (which owns the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC, ECE of a softmax head); /analyze-stats owns DeLong / NRI / IDI / decision curves / MRMC. Like /analyze-stats, it generates and executes code on your predictions — numbers are never hand-typed.

When to use

• You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and subgroup slices, plus a per-case table for the manuscript statistics.

When NOT to use

• Auditing the validation design / leakage → /model-validation.
• DeLong / NRI / IDI / decision curves / MRMC reader study → /analyze-stats.
• Building / training the model → /model-scaffold; LLM / MLLM → /mllm-eval.
• Figure rendering → /make-figures.

Workflow

Phase 1 — Fix the analysis unit and the task

State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as per-patient.

Phase 2 — Compute task-correct metrics

Generate evaluation code that computes, on the held-out predictions:

• segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global mean, with bootstrap 95% CIs.
• classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV at the deployment prevalence (not a balanced set).
• detection: FROC / mAP with the IoU match criterion stated.
• interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation metrics above the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time (see the metric guide; the study design is in + ).