| name | ml-modeling |
| description | Best-practice suggestions for predictive ML modelling with scikit-learn — Pipelines, ColumnTransformer, cross-validation, hyperparameter search, honest train/test evaluation, model leaderboards, and threshold/loss-based decisions. Use when building, comparing or reviewing supervised ML models (regression or classification) in Python notebooks or scripts. |
ML Modelling Best Practices (scikit-learn flavoured)
These are suggestions, not absolute rules — the user is an industry-experienced developer and prefers pragmatic patterns that resonated across several CEU MSBA course repos. Adapt freely to context.
The patterns below are inspired by these course and open-source repositories:
When to Use
Auto-apply when the task involves:
- Building or refactoring sklearn / xgboost / lightgbm / catboost models.
- Cross-validation, hyperparameter tuning, train/test splitting.
- Comparing multiple candidate models on a held-out set.
- Choosing a classification threshold for a real business decision.
- Reviewing a colleague's modelling notebook.
For pure inference / explanation work (OLS coefficient interpretation, confidence intervals, hypothesis tests), reach for the statistical-modeling skill instead — they're complementary.
Default Stack (industry-standard, what the inspiration repos use)
scikit-learn for everything that fits (Pipeline, ColumnTransformer, model_selection, metrics).xgboost / lightgbm for tabular boosting baselines.pandas + numpy for data wrangling.matplotlib + seaborn for plots.optuna only when sklearn's GridSearchCV / RandomizedSearchCV is genuinely too slow.- For experiment tracking: a plain DataFrame leaderboard is often enough (see
ResultCollector below). Reach for MLflow / W&B when you have many runs across sessions.
shap for model interpretation and feature importance (standard in production analytics projects).ydata-profiling (formerly pandas-profiling) for automated EDA reports.missingno for missing-data visualisation.joblib for model serialisation (prefer over pickle — handles large NumPy arrays better).
Keep the stack boring. Sometimes a print(score) in a notebook cell beats wiring a tracker.
Core Principles
- Always have a baseline. Predict the mean / majority class first — every model must beat it.
- Pipeline everything. No fitting transformers on the full dataset before the split. Fit-transform leaks are silent and devastating.
- One held-out test set, touched once. All model selection happens in CV on the train set.
- Same split, same metric, same test set for every candidate — fair comparison only.
- Track results in a leaderboard as you go. The
ResultCollector pattern below makes this a one-liner.
- Set a
random_state. Reproducibility is free.
- Optimize the metric the business cares about, not just the one easiest to compute. For classification, that often means a custom loss + tuned threshold, not raw accuracy.
- Sanity-check generalisation. When in doubt, run a paired statistical test on per-fold CV scores before declaring "Model B beats Model A" (see reference/significance.md).
The Workflow
When asked to build a predictive model, follow this skeleton (aligned with Géron's ML project checklist):
1. Frame: What are we predicting? What metric matters? (→ designing-analytics-projects skill)
2. Get data: Automate ingestion; sample a test set, put it aside, never look at it.
3. Explore: EDA in a dedicated notebook — distributions, correlations, target leakage checks.
4. Prepare: Write transform functions, not ad-hoc cells. Treat prep choices as hyperparams.
5. Baseline: Mean / majority. Beat this or stop.
6. Pipeline: ColumnTransformer + estimator. No leakage possible.
7. Short-list: Cross-validate 3–5 model families on the train set, log to leaderboard.
8. Tune: GridSearchCV / RandomizedSearchCV on the best 1–2.
9. Threshold: (Classification) Pick a cutoff using a business loss function.
10. Evaluate: Final fit on full train, evaluate ONCE on held-out test.
11. Present: Leaderboard, key plots, confusion matrix, limitations. Highlight the big picture.
12. Monitor: In production, track input quality and model drift (PSI) over time.
For details on each step see reference/workflow.md. For ready-to-paste code see snippets/.
The Pipeline Pattern (no leakage, no exceptions)
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
numeric = ["age", "income", "tenure"]
categorical = ["region", "plan"]
preprocess = ColumnTransformer([
("num", Pipeline([
("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler()),
]), numeric),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
pipe = Pipeline([
("prep", preprocess),
("model", LogisticRegression(max_iter=1000, random_state=42)),
])
pipe.fit(X_train, y_train)
Why this matters: when you cross_val_score(pipe, X_train, y_train, ...), sklearn re-fits the preprocessor inside each fold automatically. No data leakage from the validation fold into scaling statistics. This is non-negotiable.
Model Comparison: the ResultCollector Pattern
Adapted from class5_bike_share_demand.ipynb. Drop into any modelling notebook to keep a running leaderboard. The full class lives at snippets/result_collector.py.
from snippets.result_collector import ResultCollector
results = ResultCollector(metric_name="RMSLE", lower_is_better=True)
results.add("baseline (mean)", train=baseline_train, test=baseline_test)
results.add("linear (FE)", train=lin_train, test=lin_test)
results.add("xgboost", train=xgb_train, test=xgb_test)
results.show()
A simple print(pd.DataFrame(rows)) is often enough — don't over-engineer. The point is: every model lands in the same table, with the same metric, sorted the same way.
Cross-Validation Cheatsheet
| Situation | Use |
|---|
| iid regression / classification | KFold(shuffle=True, random_state=42) |
| imbalanced classification | StratifiedKFold(...) (standard in ch17 firm-exit case study) |
| grouped data (same user across rows) | GroupKFold |
| time-series | TimeSeriesSplit — never shuffle time! |
| nested model selection | inner CV for tuning, outer CV for honest score |
Default to n_splits=5 (good bias-variance trade-off, fast). Use n_splits=10 only if folds are very small.
Hyperparameter Search
Order of preference:
- Manual sweep over 3–5 sensible values in a loop. Often enough, fully transparent.
GridSearchCV when the grid is small (< 100 combinations).RandomizedSearchCV when the search space is wide.optuna when the above are too slow or you need conditional spaces.
Always wire the search into the pipeline (pipe__model__C, pipe__prep__num__scale__with_mean) so preprocessing is tuned along with the estimator.
Classification Thresholds & Business Loss
Don't default to 0.5 if the costs are asymmetric. Pattern from Békés–Kézdi ch17 predicting firm exit:
def expected_loss(y_true, proba, threshold, fn_cost=4, fp_cost=1):
pred = (proba >= threshold).astype(int)
fn = ((pred == 0) & (y_true == 1)).sum()
fp = ((pred == 1) & (y_true == 0)).sum()
return fn * fn_cost + fp * fp_cost
See snippets/threshold_optimization.py for the full per-fold-then-average pattern.
Are the Differences Significant?
When comparing two models, don't trust a single test-set number if the gap is small relative to noise. Use:
- Paired t-test or Wilcoxon signed-rank on per-fold CV scores (regression or classification).
- McNemar's test on the disagreements between two classifiers on the same test set.
- Bootstrap CI on the held-out metric for a single model.
Code and decision rules in reference/significance.md. For the full statistical-inference toolkit (OLS coefficients, confidence intervals on coefficients, prediction intervals), see the statistical-modeling skill.
Data Drift Monitoring (PSI)
In production, monitor whether feature distributions have shifted since training using the Population Stability Index (PSI):
| PSI | Interpretation | Action |
|---|
| < 0.1 | Stable | No action |
| 0.1 – 0.2 | Moderate drift | Investigate |
| > 0.2 | Significant shift | Retrain |
Compute PSI by binning the training (expected) and production (actual) distributions and comparing proportions. Implementations for both continuous and categorical features are in snippets/datadrift_psi.py. This is a standard industry technique — also emphasised in Géron's "Launch!" checklist step: "monitor your inputs' quality".
Data Preparation Best Practices
- Outlier handling: Use Tukey's IQR method or z-score filtering, but always document and justify removals — explain why values are invalid, not just extreme.
- Feature selection: For high-dimensional datasets, a quick correlation-with-target filter (
corrwith().abs() > threshold) is a useful coarse screen before building pipelines. Not a substitute for proper feature importance.
- Write transform functions, not ad-hoc notebook cells — so you can reuse them on test data, new data, and treat prep choices as hyperparameters (Géron ch. 2).
Model Serialisation
Prefer joblib over pickle — it handles large NumPy arrays better:
import joblib
joblib.dump(pipe, 'models/model_v2.pkl')
pipe = joblib.load('models/model_v2.pkl')
For project structure (dev/models/ → prod/models/ split), see the analytics-project-setup skill.
Notebook Hygiene
- Set seeds at the top:
np.random.seed(42), random_state=42 everywhere.
- Print shapes after every split / transform — catches silent bugs.
- One
Pipeline per model, named clearly (pipe_xgb, pipe_logit_l2).
- Plot calibration curves and confusion matrices in classification work — accuracy alone hides a lot.
- "Restart & Run All" must succeed before you commit.
- Number notebooks by execution order (
01_features.ipynb, 02_exploration.ipynb, …) — see the analytics-project-setup skill.
- Use
%load_ext autoreload / %autoreload 2 to hot-reload dev/src/ modules during development.
- Keep reusable utilities (EDA, I/O, drift checks) in
dev/src/ and import them — don't copy-paste between notebooks.
- Clear notebook outputs before committing — set up the pre-commit hook from
analytics-project-setup.
Anti-Patterns to Flag in Reviews
StandardScaler().fit(X) before train_test_split — leakage.- Imputing with the full-data mean before CV — leakage.
- Comparing models with different CV splits or different metrics.
- Picking a threshold on the test set after looking at the test scores.
- Reporting a single test-set number without any uncertainty estimate when models are close.
accuracy_score on a 95/5 imbalanced dataset.- One mega-cell that does load + preprocess + fit + plot — split it.
- Re-using the test set for "one more tweak". The test set is sacred.
- No data drift monitoring between training data and production inference data.
- Hardcoded file paths instead of shared config/constants — see
analytics-project-setup skill.
- Model saved as
.pkl with no versioning or naming convention.
- No
dev/prod separation — exploratory and production code in the same folder.
Code Snippets
In snippets/:
result_collector.py — the leaderboard helper from ceu-ml.pipeline_template.py — full ColumnTransformer + Pipeline + GridSearchCV scaffold for a tabular problem.cv_compare_models.py — fair cross_val_score comparison loop with mean ± std per model.threshold_optimization.py — find the loss-minimising threshold per CV fold and average.paired_cv_test.py — paired t-test / Wilcoxon on CV scores; wraps the "is the difference significant?" decision.
Data Sources Used in the Inspiration Courses (good for ML practice)
The inspiration repos hit a few canonical datasets repeatedly. They're solid defaults when
prototyping a new modelling pattern or wanting a reproducible benchmark:
Reusable scikit-learn / OpenML defaults for quick experiments without downloading anything:
from sklearn.datasets import fetch_california_housing, fetch_openml, load_breast_cancer
X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
adult = fetch_openml("adult", version=2, as_frame=True)
For larger benchmark suites, see OpenML CC18 (72 curated tabular tasks)
or Kaggle Datasets filtered by tabular + competition.
When pulling data for a project, pin the download URL, version/snapshot date, and licence
in a load-data cell — the ceu-ml notebooks consistently load straight from a versioned GitHub URL
(e.g. https://raw.githubusercontent.com/divenyijanos/ceu-ml/2026/data/bike_sharing_demand/train.csv),
which is a clean pattern.
Further Reference
Inspiration repos (check these for full worked examples):
Companion skills:
analytics-project-setup — folder structure, branching, AGENTS.md, environment management.statistical-modeling — for inferential/explanatory modelling (OLS coefficients, confidence intervals, significance tests).data-warehousing — for bronze/silver/gold data pipeline patterns feeding into ML models.
External:
Suggestions, not gospel. When in doubt, prefer simpler models + honest evaluation over exotic techniques.
