model-evaluation

name	model-evaluation
description	Evaluate ML model performance with appropriate metrics, validation strategies, and statistical rigor. Outputs evaluation reports, confusion matrices, calibration curves, fairness analysis, and go/no-go recommendations.
argument-hint	["task type","business metric","class imbalance","fairness requirements"]
allowed-tools	Read, Write, Bash

Model Evaluation

Rigorous evaluation separates models that work in notebooks from models that work in production. Choose metrics that align with business goals, validate on held-out data that mirrors production, and quantify uncertainty.

Process

1. Define business objective first — what decision does this model inform?
2. Choose primary metric — aligned with business cost of errors.
3. Choose secondary metrics — for monitoring, fairness, calibration.
4. Design evaluation split — temporal split for time-series, stratified for imbalanced classes.
5. Evaluate on held-out test set — never tune on test data.
6. Compute confidence intervals — single-point estimates are not enough.
7. Analyze error cases — where does the model fail and why?
8. Fairness analysis — performance across subgroups.
9. Calibration check — predicted probabilities vs. actual rates.
10. Make go/no-go recommendation with evidence.

Output Format

Metric Selection Guide

Task	Primary Metric	When to Use
Binary classification (balanced)	F1, AUC-ROC	Equal cost of FP/FN
Binary classification (imbalanced)	AUC-PR, F1	Rare positive class
Binary classification (cost-sensitive)	Custom: FP cost × FPR + FN cost × FNR	Fraud detection, medical
Multi-class	Macro F1, per-class F1	When all classes matter equally
Regression	MAE, RMSE, MAPE	Depends on outlier sensitivity
Ranking	NDCG@K, MAP	Search, recommendations
Survival	C-index, Brier score	Time-to-event

Evaluation Framework

# evaluation/evaluator.py
import numpy as np
import pandas as pd
from sklearn.metrics import (
    roc_auc_score, average_precision_score,
    f1_score, precision_score, recall_score,
    confusion_matrix, classification_report,
    mean_absolute_error, mean_squared_error,
    calibration_curve, brier_score_loss,
    RocCurveDisplay, PrecisionRecallDisplay
)
from sklearn.utils import resample
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from dataclasses import dataclass
from typing import Optional
import warnings

@dataclass
class EvaluationConfig:
    task_type: str                    # "binary_classification", "regression", "ranking"
    positive_label: int = 1
    decision_threshold: float = 0.5
    n_bootstrap: int = 1000
    ci_level: float = 0.95
    cost_fp: float = 1.0             # False positive cost (for cost-sensitive)
    cost_fn: float = 1.0             # False negative cost
    fairness_columns: list = None    # Columns to check for fairness
    top_k: list = None               # [1, 5, 10] for ranking tasks


 :
     ():
        .config = config
    
     () -> :
        y_pred = (y_prob >= .config.decision_threshold).astype()
        
        
        metrics = {
            : (y_true),
            : y_true.mean(),
            : y_pred.mean(),
            : roc_auc_score(y_true, y_prob),
            : average_precision_score(y_true, y_prob),
            : f1_score(y_true, y_pred),
            : precision_score(y_true, y_pred, zero_division=),
            : recall_score(y_true, y_pred),
            : brier_score_loss(y_true, y_prob),
            : .config.decision_threshold,
        }
        
        
        cm = confusion_matrix(y_true, y_pred)
        tn, fp, fn, tp = cm.ravel()
        metrics[] = (fp * .config.cost_fp + fn * .config.cost_fn)
        metrics[] = {: (tn), : (fp), : (fn), : (tp)}
        
        
        ci = ._bootstrap_ci(y_true, y_prob, [, , , ])
        metrics[] = ci
        
        
        metrics[] = ._evaluate_calibration(y_true, y_prob)
        
        
         .config.fairness_columns  df   :
            metrics[] = ._evaluate_fairness(y_true, y_prob, df)
        
        
        metrics[] = ._analyze_thresholds(y_true, y_prob)
        
         metrics
    
     () -> :
        
        bootstrap_scores = {m: []  m  metric_names}
        
         _  (.config.n_bootstrap):
            
            indices = resample(((y_true)), replace=)
            y_true_boot = y_true[indices]
            y_prob_boot = y_prob[indices]
            y_pred_boot = (y_prob_boot >= .config.decision_threshold).astype()
            
             y_true_boot.() ==   y_true_boot.() == (y_true_boot):
                  
            
            :
                bootstrap_scores[].append(roc_auc_score(y_true_boot, y_prob_boot))
                bootstrap_scores[].append(f1_score(y_true_boot, y_pred_boot, zero_division=))
                bootstrap_scores[].append(precision_score(y_true_boot, y_pred_boot, zero_division=))
                bootstrap_scores[].append(recall_score(y_true_boot, y_pred_boot))
             Exception:
                
        
        alpha =  - .config.ci_level
        ci = {}
         metric, scores  bootstrap_scores.items():
             scores:
                ci[metric] = {
                    : np.mean(scores),
                    : np.percentile(scores, alpha /  * ),
                    : np.percentile(scores, ( - alpha / ) * ),
                    : np.std(scores),
                }
         ci
    
     () -> :
        
        fraction_positives, mean_predicted = calibration_curve(y_true, y_prob, n_bins=)
        
        
        bin_sizes = []
        ece = 
         fp, mp  (fraction_positives, mean_predicted):
            ece += (fp - mp)
        ece /= (fraction_positives)
        
         {
            : ece,               
            : fraction_positives.tolist(),
            : mean_predicted.tolist(),
            : ece < ,
        }
    
     () -> :
        
        y_pred = (y_prob >= .config.decision_threshold).astype()
        fairness_report = {}
        
         col  .config.fairness_columns:
             col   df.columns:
                
            
            subgroup_metrics = {}
             group_val  df[col].unique():
                mask = df[col] == group_val
                 mask.() < :  
                    
                
                g_true = y_true[mask]
                g_prob = y_prob[mask]
                g_pred = y_pred[mask]
                
                 g_true.() == :
                    
                
                subgroup_metrics[(group_val)] = {
                    : (mask.()),
                    : (g_true.mean()),
                    : (roc_auc_score(g_true, g_prob)),
                    : (f1_score(g_true, g_pred, zero_division=)),
                    : (((g_pred == ) & (g_true == )).() / (g_true == ).()),
                    : (((g_pred == ) & (g_true == )).() / (g_true == ).()),
                }
            
             subgroup_metrics:
                aucs = [v[]  v  subgroup_metrics.values()]
                fnrs = [v[]  v  subgroup_metrics.values()]
                fairness_report[col] = {
                    : subgroup_metrics,
                    : (aucs) - (aucs),
                    : (fnrs) - (fnrs),
                    : ((aucs) - (aucs)) >   ((fnrs) - (fnrs)) > ,
                }
        
         fairness_report
    
     () -> :
        
        thresholds = np.linspace(, , n_thresholds)
        results = []
        
         t  thresholds:
            y_pred = (y_prob >= t).astype()
            results.append({
                : ((t), ),
                : (precision_score(y_true, y_pred, zero_division=)),
                : (recall_score(y_true, y_pred)),
                : (f1_score(y_true, y_pred, zero_division=)),
                : (y_pred.mean()),
                : (
                    ((y_pred == ) & (y_true == )).() * .config.cost_fp +
                    ((y_pred == ) & (y_true == )).() * .config.cost_fn
                ),
            })
        
         results
    
     ():
        
        fig = plt.figure(figsize=(, ))
        gs = gridspec.GridSpec(, , figure=fig)
        
        
        ax1 = fig.add_subplot(gs[, ])
        ax1.plot([, ], [, ], , alpha=)
        ax1.set_xlabel()
        ax1.set_ylabel()
        ax1.set_title()
        
        
        ax2 = fig.add_subplot(gs[, ])
        ax2.axhline(y=metrics[], color=, linestyle=, alpha=)
        ax2.set_title()
        
        
        ax3 = fig.add_subplot(gs[, ])
        cal = metrics[]
        ax3.plot([, ], [, ], , alpha=, label=)
        ax3.plot(cal[], cal[], , label=)
        ax3.set_xlabel()
        ax3.set_ylabel()
        ax3.set_title()
        ax3.legend()
        
        
        ax4 = fig.add_subplot(gs[, :])
        ta = pd.DataFrame(metrics[])
        ax4.plot(ta[], ta[], label=)
        ax4.plot(ta[], ta[], label=)
        ax4.plot(ta[], ta[], label=)
        ax4.axvline(x=.config.decision_threshold, color=, linestyle=, label=)
        ax4.set_xlabel()
        ax4.set_ylabel()
        ax4.set_title()
        ax4.legend()
        
        plt.tight_layout()
        plt.savefig(, dpi=, bbox_inches=)
        plt.close()
    
     () -> :
        
        issues = []
        
        ci = metrics.get(, {})
        
         metric, threshold  requirements.items():
            current = metrics.get(metric)
             current  :
                
            
             current < threshold:
                ci_lower = ci.get(metric, {}).get(, current)
                issues.append({
                    : metric,
                    : threshold,
                    : current,
                    : ci_lower,
                    : threshold - current,
                })
        
        
        fairness = metrics.get(, {})
        fairness_issues = []
         col, report  fairness.items():
             report.get():
                fairness_issues.append()
        
        go = (issues) ==   (fairness_issues) == 
        
         {
            :   go  ,
            : issues,
            : fairness_issues,
            : (
                
                
            )
        }

Usage Example

# evaluate.py
import numpy as np
import pandas as pd
import mlflow

# Load test data (NEVER use val data for final evaluation)
test_df = pd.read_parquet("data/test_set.parquet")
X_test = test_df.drop(columns=["label", "user_id", "timestamp"])
y_test = test_df["label"].values

# Load production candidate model
model = mlflow.pyfunc.load_model("models:/order-propensity/Staging")
y_prob = model.predict(X_test)

# Configure evaluator
config = EvaluationConfig(
    task_type="binary_classification",
    decision_threshold=0.4,   # Tuned on validation set
    n_bootstrap=1000,
    fairness_columns=["age_group", "country"],
    cost_fp=1.0,   # Cost of false positive (irrelevant alert)
    cost_fn=10.0,  # Cost of false negative (missed purchase)
)

evaluator = ModelEvaluator(config)
metrics = evaluator.evaluate_binary_classification(
    y_true=y_test,
    y_prob=y_prob,
    df=test_df,
    model_name="order-propensity-v2"
)

# Generate visual report
evaluator.generate_report(metrics, "reports/evaluation")

# Check requirements
requirements = {
    "auc_roc": 0.78,
    "recall": 0.65,
    "precision": 0.50,
}

recommendation = evaluator.make_recommendation(metrics, requirements)
print(recommendation["recommendation"])
print(recommendation["summary"])

# Log to MLflow
 mlflow.start_run(tags={: }):
    mlflow.log_metrics({k: v  k, v  metrics.items()  (v, )})
    mlflow.log_artifact()
    mlflow.log_dict(recommendation, )

Rules

• Test set is sacred — touch it exactly once, after all tuning is done.
• Use temporal split for time-series data — random split leaks future information.
• Report confidence intervals — single-point metrics hide variance.
• Calibration matters for probability outputs — a model that says 80% should be right 80% of the time.
• Always check fairness across subgroups — aggregate metrics hide disparate impact.
• Match metric to cost structure — recall vs. precision trade-off depends on FP/FN costs.
• Stratify splits for imbalanced classes — random split can create label-free test folds.
• Compare to a simple baseline — a majority-class classifier or mean predictor is your floor.
• Document the decision threshold — changing threshold at deployment changes all metrics.
• Evaluate on production-like data — a test set that doesn't match production is misleading.