train-evaluate-ml-model

name	train-evaluate-ml-model
description	Trains and evaluates a classic (non-LLM) ML model — business-aligned metric selection, leakage-safe train/validation/test splits, Pipeline-scoped feature engineering, baseline-first model selection, cross-validated hyperparameter tuning, bias/variance diagnosis, and experiment tracking — guarding against data leakage and overfitting.
when_to_use	Fitting and validating a classification, regression, ranking, forecasting, or clustering model on tabular/feature data. Distinct from profile-dataset (EDA only), wrangle-tabular-data (cleaning/shaping the feature table), serve-deploy-ml-model (deployment), monitor-ml-drift (post-deploy), and rag-pipeline/prompt-engineering (LLM work).

When to Use

Reach for this skill when the request is to fit and validate a model that predicts, not to explore or clean data:

• "Train a classifier to predict churn / fraud / default and tell me if it's good"
• "Build a regression/forecasting model for demand/price and pick the best one"
• "My model gets 99% accuracy — is that real or leakage?"
• "Tune hyperparameters / cross-validate / compare XGBoost vs logistic regression"
• "Cluster these customers into segments"

NOT this skill:

• Summary stats, distributions, correlations, missingness before modeling → profile-dataset
• Cleaning, type coercion, joins, dedup, resampling to build the feature table → wrangle-tabular-data
• Asserting schema/range/null contracts on the data before training → validate-data-quality
• Packaging the trained model behind an API / batch job → serve-deploy-ml-model
• Watching a live model's inputs/outputs degrade over time → monitor-ml-drift
• Scoring or grading an LLM's outputs against a rubric/golden set → llm-eval-harness
• Getting an LLM to answer over a corpus, or designing prompts → rag-pipeline / prompt-engineering

Steps

1. Choose the metric from the business cost FIRST — never optimize bare accuracy on imbalanced data. A 1%-positive fraud set scores 99% accuracy by predicting all-negative and catches zero fraud. Pick before you train:

   Task / situation	Metric	Why
   Imbalanced classification, cost of missing a positive high	Recall @ fixed precision, or PR-AUC	accuracy & ROC-AUC look great while recall is ~0
   Imbalanced, ranking/threshold-free comparison	PR-AUC (not ROC-AUC)	ROC-AUC is optimistic under heavy imbalance
   Balanced classification	F1 or ROC-AUC	symmetric cost
   Asymmetric FP vs FN cost	Expected cost = cFP·FP + cFN·FN, tune threshold	maps directly to money
   Regression, outliers matter	RMSE	penalizes large errors
   Regression, robust to outliers	MAE / MAPE	business reads "off by X"
   Ranking / recsys	NDCG@k, MAP@k	position-aware
   Forecasting	MASE / sMAPE vs naive	scale-free, must beat seasonal-naive
   Clustering (no labels)	silhouette + downstream business check	inertia alone is meaningless

   Fix the decision threshold from the cost matrix later — don't ship the default 0.5.

2. Split BEFORE any feature engineering or fitting — this is the #1 leakage source. Three sets: train / validation (or CV folds) / held-out test touched once at the very end. The split strategy is not optional — pick by data structure:

   • Random stratified for i.i.d. rows: train_test_split(X, y, stratify=y, test_size=0.2, random_state=42).
   • Time-based for any temporal data — train on past, test on future, never shuffle. Use TimeSeriesSplit for CV. A random split on time-series leaks the future and inflates every metric.
   • Group split when rows share an entity (same user/patient/device across rows): GroupKFold / StratifiedGroupKFold so the same group never appears in both train and test.

Common Errors

• Splitting after feature engineering / scaling on the full dataset. Test statistics bleed into train; metrics inflate, prod collapses. Split first, fit transforms on train only (use a Pipeline).
• Random split on temporal data. The model trains on future rows and "predicts" the past. Use a time-based split / TimeSeriesSplit.
• Reporting accuracy on an imbalanced problem. 99% accuracy with 0% recall is useless. Pick PR-AUC / recall-at-precision from the cost (step 1).
• A feature that's too good. One column driving a near-perfect score is almost always leakage (post-outcome field, ID proxy, target-derived). Audit and drop it.
• Target encoding / imputation / feature selection computed before the split or outside the CV fold. Subtle leakage that survives CV but not production. Fit them inside the pipeline, per fold.
• SMOTE/oversampling applied before splitting. Synthetic copies of test rows land in train. Resample inside the training fold only.
• Tuning against the test set / peeking repeatedly. You overfit to it manually. Tune on CV/validation; touch test once.
• No baseline. Without DummyClassifier/linear you can't tell if your model learned anything. Always beat the dumb baseline first.
• scoring= left at default (accuracy/R²) during search. The search optimizes the wrong thing. Pass your business metric to cross_val_score / *SearchCV.
• Train/serve feature skew. Features computed differently (or with different code/library versions) at training vs inference. Reuse the exact fitted pipeline artifact for serving.
• Unpinned seeds / untracked runs. Results aren't reproducible and "best model" can't be recovered. Pin random_state, log params+metrics+data+code version.

Verify

1. Beats baseline: held-out test metric (step-1 metric) > the DummyClassifier/DummyRegressor and the linear baseline by a margin larger than the bootstrap CI. If it doesn't, there's no model.
2. Leakage check passes: drop each top-importance feature individually — no single feature should collapse the score to chance-level; remove any post-outcome/target-derived/ID-proxy column; confirm all learned transforms were fit on train only.
3. Split integrity: for temporal data, every test timestamp is strictly after every train timestamp; for grouped data, no group ID appears in two sets (assert programmatically).
4. Generalization gap sane: train metric − validation metric is small and explained by your bias/variance call; test ≈ validation (a big test drop means tuning overfit to validation).
5. Metric matches business cost: the reported metric and the chosen decision threshold come from step 1, not the library default.
6. No train/serve skew: running the saved pipeline on a held-out row reproduces the exact training-time prediction (same features, same library versions).
7. Reproducible: re-running with the logged seed + data snapshot + code commit yields the same metric (within float tolerance); the experiment tracker has params, CV metrics with std, data hash, and git commit.

Done = the held-out test metric (chosen for business cost, evaluated once) beats both baselines beyond its CI, the leakage and split-integrity checks pass, the generalization gap is explained, and the saved pipeline reproduces predictions with no train/serve skew under a pinned, tracked run.