model-validation

# Model Validation Skill ## Triggers - "validate model performance" - "external validation" - "statistical analysis" - "clinical validation" - "model comparison" - "regulatory submission" - "performance benchmarking" - "fairness audit" ## Parameters - `validation_type` (required): Type of validation needed - `internal` - Retrospective internal dataset - `external` - Prospective/out-of-distribution testing - `prospective` - Clinical deployment study - `regulatory` - FDA/EMA submission prep - `fairness` - Subgroup disparity analysis - `comparison` - Head-to-head model comparison - `model_task` (required): Model's intended use - `detection` - Sensitivity, specificity, PPV, NPV - `segmentation` - Dice, IoU, Hausdorff distance - `classification` - Accuracy, AUC, F1 score - `regression` - MAE, RMSE, correlation - `modality` (optional): Imaging modality - `regulatory_path` (optional): Target clearance pathway ## Validation Framework ### Performance Metrics | Task | Primary Metrics | Secondary | |------|-----------------|-----------| | Detection | Sensitivity, Specificity, AUC | PPV, NPV, FROC | | Segmentation | Dice, IoU | Hausdorff, ASD | | Classification | Accuracy, AUC, F1 | Sensitivity, Specificity | | Regression | MAE, RMSE | Correlation, Bland-Altman | ### Statistical Methods - Confidence intervals (bootstrap, binominal) - Significance testing (McNemar, DeLong for AUC) - Power analysis for sample sizing - Multiple comparison correction - Subgroup interaction testing ### Regulatory Standards - FDA 510(k) predicate comparison - FDA De Novo requirements - EU MDR clinical evaluation - IMDRF clinical evidence framework - ACR-SIIM AI performance standards ## Output Format Returns structured JSON with: - Validation protocol and methodology - Required sample size with power analysis - Statistical test selection and rationale - Results template with standard metrics - Interpretation guidelines - Regulatory compliance checklist ## Usage Examples ``` validation_type: external model_task: detection modality: CT validation_type: regulatory model_task: classification regulatory_path: 510k ```