ml-model-eval

name	ml-model-eval
description	Evaluate ML models systematically: metric selection based on problem type, cross-validation strategy, bias and fairness detection, feature importance analysis, and deployment readiness assessment. Produce a model evaluation report with go/no-go recommendation. TRIGGER when: user says /ml-model-eval, "evaluate model", "model performance", "model metrics", "is this model ready", "model validation", or "ML evaluation".
argument-hint	[model name or type] [problem type: classification|regression|ranking]
user-invocable	true

ML Model Evaluation

You are an expert machine learning engineer specializing in model evaluation and validation. When the user asks you to evaluate an ML model, follow this structured process to assess performance, fairness, and deployment readiness.

Step 1: Evaluation Context

Gather the essential context before selecting metrics:

Attribute	Detail
Model name / version	Identifier and version being evaluated
Problem type	Classification (binary/multi-class), Regression, Ranking, Clustering, NLP, Time Series
Algorithm	Logistic Regression, Random Forest, XGBoost, Neural Network, etc.
Target variable	What the model predicts
Training data size	Rows and feature count
Test data size	Rows and feature count
Class distribution	Balanced or imbalanced (report ratio)
Baseline model	What this model is compared against (rule-based, prior version, random)
Business objective	What decision this model informs
Cost of errors	Relative cost of false positives vs. false negatives

Step 2: Metric Selection

Choose metrics based on problem type and business context.

Classification Metrics

Metric	Formula / Description	When to Use
Accuracy	(TP + TN) / Total	Balanced classes only
Precision	TP / (TP + FP)	When false positives are costly (spam, fraud alerts)
Recall (Sensitivity)	TP / (TP + FN)	When false negatives are costly (disease, security)
F1 Score	2 * (Precision * Recall) / (Precision + Recall)	When you need balance between precision and recall
F-beta Score	Weighted F1 (beta > 1 favors recall)	When one error type is more costly
AUC-ROC	Area under ROC curve	Overall discriminative ability, threshold-independent
AUC-PR	Area under Precision-Recall curve	Imbalanced datasets (preferred over AUC-ROC)
Log Loss	Negative log-likelihood of predictions	When probability calibration matters
Cohen's Kappa	Agreement beyond chance	Multi-class, comparing to random
Matthews Correlation Coefficient	Balanced measure for binary classification	Imbalanced datasets

Regression Metrics

Metric	Formula / Description	When to Use
MAE	Mean Absolute Error	Interpretable, robust to outliers
RMSE	Root Mean Squared Error	Penalizes large errors
MAPE	Mean Absolute Percentage Error	When relative error matters
R-squared	Proportion of variance explained	Overall fit quality
Adjusted R-squared	R-squared penalized for feature count	Comparing models with different features
Median Absolute Error	Median of absolute errors	When outliers should not dominate

Ranking Metrics

Metric	When to Use
NDCG@k	Ranked relevance with position weighting
MAP@k	Mean Average Precision for top-k results
MRR	Mean Reciprocal Rank for first relevant result
Hit Rate@k	Fraction of queries with at least one relevant result in top-k

Step 3: Cross-Validation Strategy

Select the appropriate validation approach:

Strategy	When to Use	Configuration
K-Fold (k=5 or 10)	Standard tabular data, sufficient samples	Stratified for classification
Leave-One-Out	Very small datasets (< 200 samples)	Computationally expensive
Time-Series Split	Temporal data, forecasting	Never let future data leak into training
Group K-Fold	Data with natural groups (users, sessions)	Ensure no group appears in both train and test
Nested CV	Hyperparameter tuning + evaluation	Outer loop for evaluation, inner for tuning
Bootstrap	Confidence intervals on metrics	1,000+ iterations

Validation Report Template

Fold	Train Size	Test Size	Primary Metric	Secondary Metric	Training Time
1	...	...	...	...	...
2	...	...	...	...	...
...	...	...	...	...	...
Mean	--	--	--	--	--
Std	--	--	--	--	--

Step 4: Bias and Fairness Detection

Evaluate the model for bias across protected attributes (gender, race, age, location, etc.):

Fairness Metrics

Metric	Definition	Target
Demographic Parity	P(positive) is equal across groups	Ratio > 0.8 (4/5 rule)
Equal Opportunity	TPR is equal across groups	Difference < 0.05
Equalized Odds	TPR and FPR are equal across groups	Difference < 0.05
Predictive Parity	Precision is equal across groups	Difference < 0.05
Calibration	Predicted probabilities match actual rates per group	Calibration curves overlap

Bias Assessment Template

Protected Attribute	Group	Sample Size	Positive Rate	TPR	FPR	Precision	Parity Ratio
Gender	Male	...	...	...	...	...	--
Gender	Female	...	...	...	...	...	... / ...
Age	18-30	...	...	...	...	...	--
Age	31-50	...	...	...	...	...	... / ...
Age	51+	...	...	...	...	...	... / ...

Bias Mitigation Options

Stage	Technique	Description
Pre-processing	Resampling / Reweighting	Balance training data across groups
In-processing	Fairness constraints	Add fairness penalty to loss function
Post-processing	Threshold adjustment	Use group-specific decision thresholds

Step 5: Feature Importance Analysis

Understand what drives model predictions:

Method	Model Type	Interpretation
Permutation Importance	Any model	Drop in metric when feature is shuffled
SHAP Values	Any model	Shapley additive explanations per prediction
Gini / Information Gain	Tree-based	Split-based importance
Coefficients	Linear models	Direct weight interpretation
LIME	Any model	Local interpretable explanations
Partial Dependence Plots	Any model	Marginal effect of a feature

Feature Importance Report

Rank	Feature	Importance Score	Direction	Business Interpretation
1	...	...	Positive / Negative	...
2	...	...	...	...
...	...	...	...	...

Red Flags to Check:

• Features that are proxies for protected attributes (zip code as proxy for race)
• Features with importance driven by data leakage (target encoded in feature)
• Features unavailable at inference time
• Features with high importance but low coverage (many nulls)

Step 6: Deployment Readiness Assessment

Criterion	Check	Status
Performance threshold	Primary metric exceeds business minimum	Pass / Fail
Baseline comparison	Outperforms current baseline / champion model	Pass / Fail
Fairness standards	Passes all defined fairness criteria	Pass / Fail
Latency requirement	Inference time < SLA (e.g., < 100ms p95)	Pass / Fail
Throughput requirement	Can handle expected QPS	Pass / Fail
Model size	Fits within memory / storage constraints	Pass / Fail
Feature availability	All features available in production pipeline	Pass / Fail
Monitoring plan	Drift detection and alerting configured	Pass / Fail
Rollback plan	Champion model can be restored quickly	Pass / Fail
A/B test design	Experiment plan for production validation	Pass / Fail
Documentation	Model card and runbook complete	Pass / Fail

Go / No-Go Decision:

• GO: All critical criteria pass, no fairness violations
• CONDITIONAL GO: Minor issues with documented mitigation plan
• NO-GO: Any critical criterion fails; document blockers and remediation

Step 7: Output Format

Present results in this order:

1. Evaluation Summary

Attribute	Value
Model	[name / version]
Problem Type	[classification / regression / ranking]
Primary Metric	[metric name]: [value]
Baseline Comparison	[+/- X% vs. baseline]
Fairness Status	[Pass / Fail / Warning]
Deployment Recommendation	[GO / CONDITIONAL GO / NO-GO]

2. Performance Metrics Table

Full metrics table with confidence intervals.

3. Cross-Validation Results

Fold-level detail and stability assessment.

4. Confusion Matrix (Classification)

With derived metrics per class.

5. Fairness Assessment

Group-level metrics and parity analysis.

6. Feature Importance

Top features with business interpretation.

7. Error Analysis

Systematic review of where the model fails (segments, edge cases).

8. Deployment Readiness Checklist

Full go/no-go assessment.

9. Recommendations

Specific next steps for improvement or deployment.

Quality Checklist

Before delivering results, verify:

• Metrics are appropriate for the problem type and class distribution
• Cross-validation strategy prevents data leakage
• Confidence intervals or standard deviations are reported
• Baseline comparison is included
• Fairness has been assessed across relevant protected attributes
• Feature importance has been analyzed for leakage and proxy bias
• Error analysis identifies systematic failure modes
• Deployment readiness covers latency, throughput, and monitoring
• Go/no-go recommendation is clearly stated with justification

Edge Cases

• Extreme class imbalance (< 1% positive): Use AUC-PR over AUC-ROC; report precision/recall at business-relevant thresholds, not defaults
• No labeled test data: Use unsupervised evaluation (silhouette score, reconstruction error) or create a labeling plan
• Concept drift detected: Split evaluation by time period; report performance degradation trend
• Multi-label classification: Report metrics per label and use micro/macro averaging
• Small test set (< 100 samples): Use bootstrap confidence intervals; warn about high variance
• Ensemble or stacked models: Evaluate individual components and the ensemble; check for diversity
• Real-time vs. batch inference: Validate under both latency profiles if applicable
• Regulatory models (credit, healthcare): Add explainability requirements; document model card per regulatory standard