| name | evaluation |
| description | Evaluate biomedical ML models with appropriate metrics and confidence intervals. Use when: (1) Computing classification metrics (AUC-ROC, balanced accuracy, sensitivity, specificity, F1) with confidence intervals, (2) Evaluating segmentation models (Dice, IoU, Hausdorff, surface Dice), (3) Survival analysis (C-index, Kaplan-Meier, Cox PH, time-dependent AUC), (4) Statistical comparison between models (Wilcoxon, paired t-test), (5) Calibration assessment (Brier score, ECE, reliability diagrams), (6) Regression metrics (MAE, RMSE, R-squared, Bland-Altman), (7) Multi-label classification metrics.
|
Evaluation Metrics
Workflow
Evaluating a biomedical model involves these steps:
- Identify the task type -- classification, segmentation, survival, regression, or multi-label
- Select primary and secondary metrics -- based on clinical relevance
- Compute metrics with confidence intervals -- t-distribution for k-fold, bootstrap for held-out
- Assess calibration -- if probability thresholds guide clinical decisions
- Compare models statistically -- Wilcoxon or paired t-test with correction
Decision Tree
What is the task type?
- Binary/multi-class classification → AUC-ROC, balanced accuracy, sensitivity, specificity, F1. See evaluation-metrics.md
- Multi-label classification → Per-label AUC, macro/micro AUC, average precision. See evaluation-metrics.md
- Segmentation → Dice, IoU, HD95, surface Dice. See segmentation-metrics.md
- Survival / time-to-event → C-index, time-dependent AUC, Brier score, Kaplan-Meier. See survival-metrics.md
- Regression / biomarker prediction → MAE, RMSE, R-squared, Bland-Altman. See regression-metrics.md
How to compute confidence intervals?
- K-fold cross-validation → t-distribution CI (
compute_ci() in evaluation-metrics.md)
- Single held-out test set → Bootstrap CI (
bootstrap_ci() in evaluation-metrics.md)
- ASK the user which evaluation setup they have
Is calibration important?
- Model outputs risk scores / probabilities → Yes, assess ECE + reliability diagram. See calibration-metrics.md
- Ranking or comparison only → Calibration optional
Comparing multiple models?
- 2 models on same folds → Wilcoxon signed-rank test (non-parametric)
- Multiple models → Apply Bonferroni correction
- See evaluation-metrics.md statistical comparison section
ASK the user before starting:
- What is the task type?
- Cross-validation or held-out test set?
- Is model calibration clinically relevant?
References
| File | Read When |
|---|
| references/evaluation-metrics.md | Classification metrics (AUC, sensitivity, specificity, F1), CIs (t-distribution + bootstrap), multi-label, statistical model comparison |
| references/segmentation-metrics.md | Dice, IoU, Hausdorff (HD95), surface Dice, multi-class segmentation, per-subject aggregation |
| references/survival-metrics.md | C-index, Kaplan-Meier, log-rank, Cox PH, time-dependent AUC, bootstrap CIs for survival |
| references/calibration-metrics.md | ECE, Brier score, reliability diagrams, temperature scaling for post-hoc calibration |
| references/regression-metrics.md | MAE, RMSE, R-squared, Bland-Altman analysis and plots for quantitative biomarker prediction |
Note: segmentation-metrics.md references compute_ci() from evaluation-metrics.md.
