Builds diagnostic and prognostic classifiers on omics feature matrices with regularized logistic regression, random forest, and gradient-boosted trees, handling the p>>n regime, batch shortcut learning, class imbalance, and probability calibration. Use when building a classifier from expression, methylation, or variant data, choosing an algorithm for high-dimensional small-n data, or diagnosing a suspiciously perfect AUC. For unbiased evaluation see machine-learning/model-validation; for feature selection see machine-learning/biomarker-discovery; for time-to-event outcomes see machine-learning/survival-analysis.
Builds diagnostic and prognostic classifiers on omics feature matrices with regularized logistic regression, random forest, and gradient-boosted trees, handling the p>>n regime, batch shortcut learning, class imbalance, and probability calibration. Use when building a classifier from expression, methylation, or variant data, choosing an algorithm for high-dimensional small-n data, or diagnosing a suspiciously perfect AUC. For unbiased evaluation see machine-learning/model-validation; for feature selection see machine-learning/biomarker-discovery; for time-to-event outcomes see machine-learning/survival-analysis.
Before using code patterns, verify installed versions match. If versions differ:
Python: pip show <package> then help(module.function) to check signatures
Two high-risk drifts: XGBoost moved early_stopping_rounds from fit() to the constructor (deprecated 1.6, removed from fit() in 2.1); scikit-learn deprecated LogisticRegression(penalty=) in 1.8 (use l1_ratio+C) and CalibratedClassifierCV(cv='prefit') in 1.6 (use FrozenEstimator). If code throws TypeError/FutureWarning, switch to the constructor / / form.
l1_ratio
FrozenEstimator
Classification Models for Omics Data
"Build a classifier from my expression data" -> Start with a regularized linear model (often the ceiling in p>>n), check for batch shortcuts, and treat the probability -- not the label -- as the product.
Linear (often best): LogisticRegression(penalty='elasticnet', solver='saga')
Trees when nonlinear/interaction signal: RandomForestClassifier, xgboost.XGBClassifier
Imbalance: class_weight='balanced' or threshold tuning, NOT SMOTE for risk models
The Single Most Important Modern Insight -- In p>>n, Simple Often Wins and the Probability, Not the Label, Is the Product
Omics classification almost always lives in p>>n (thousands of features, tens-to-hundreds of samples). Two counterintuitive consequences follow. First, more flexible is not better: with n in the dozens the variance of a flexible learner dominates, the full covariance is singular so QDA/full-LDA are undefined, and simple diagonal/linear methods match or beat elaborate ones (Dudoit 2002). "Random forest is the obvious choice for expression data" is a myth -- SVM/regularized logistic frequently win on microarray-style problems (Statnikov 2008), and gradient-boosted trees beat deep nets on tabular/omics data (Grinsztajn 2022). Regularization is the load-bearing wall, not a tuning nicety.
Second, in diagnostic/prognostic use the probability is the product, not the label -- which makes calibration, not accuracy, the thing that breaks silently (Van Calster 2019). A model can rank perfectly (AUC 0.9) and still output dishonest risks. And the most common cause of a beautiful AUC is not skill but a batch artifact: if batch correlates with the outcome, the classifier learns the cleaner technical signal and the performance collapses on any independent cohort.
Algorithm Choice for p>>n
Model
Wins when
Overfits / fails when
Calibration
Scaling
L1 logistic (lasso)
Sparse signal, want a small signature
Correlated features -> unstable selection; >n true signals
Good (proper loss); shrinks toward base rate
Standardize
L2 / elastic-net logistic
Many small correlated effects; omics default
Needs C (and l1_ratio) tuning
Good; preferred when calibration matters
Standardize
DLDA / nearest-centroid
Tiny n, roughly linear (Dudoit 2002)
Strong interactions; non-Gaussian
Crude; recalibrate
Variance-scaled
Linear SVM
High-dim linear separability (Statnikov 2008)
Heavy overlap; needs C
No native probabilities -- Platt-scale decision_function
Critical
Random forest
Nonlinear/interaction signal; robust baseline
Sparse-linear signal; tiny n; OOB-as-test leakage
Bagged votes bounded away from 0 and 1
Scale-invariant
GBDT (XGBoost/LightGBM)
Best general tabular performer
Tiny n + deep/many rounds; needs early stopping
Log-loss overfitting tends to overconfident extremes
Scale-invariant
Tabular deep nets
Very large n; multimodal/transfer
Typical omics n -> loses to GBDT
Variable; often needs temperature scaling
Standardize
Tree ensembles are often miscalibrated and the direction depends on the learner and loss: classic boosted ensembles push probabilities toward 0.5 (sigmoid distortion; Niculescu-Mizil 2005), bagged forests are comparatively well-calibrated but bound their votes away from 0 and 1, and modern gradient boosting trained to log-loss for many rounds tends to overfit toward overconfident extremes. Check a reliability curve and recalibrate rather than assuming a direction.
Decision Tree by Scenario
Scenario
Recommended approach
Why
Default omics classifier, want a signature
Elastic-net logistic
Often the ceiling in p>>n; sparse + grouping; well-calibrated
GBDT beats deep on engineered omics matrices (Grinsztajn 2022)
Need unbiased performance / nested CV / calibration metrics
-> machine-learning/model-validation
Evaluation is its own discipline
Time-to-event outcome
-> machine-learning/survival-analysis
Censoring needs survival models, not classifiers
Core Workflow: Regularized Logistic First
Goal: A calibrated, interpretable baseline that is often the best omics classifier.
Approach: Standardize inside a Pipeline and fit elastic-net logistic with cross-validated penalty; the L2 component keeps correlated genes together, the L1 component yields a sparse signature.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegressionCV
# saga supports elasticnet; C = 1/lambda (small C = strong shrinkage). Standardize: the penalty is scale-sensitive.
clf = LogisticRegressionCV(penalty='elasticnet', solver='saga', l1_ratios=[0.1, 0.5, 0.9],
Cs=20, cv=5, max_iter=10000, class_weight='balanced')
pipe = Pipeline([('scaler', StandardScaler()), ('clf', clf)])
pipe.fit(X_train, y_train)
Tree Ensembles
Goal: Capture nonlinear and interaction structure when the linear baseline leaves signal on the table.
Approach: Random forest needs no scaling and is a robust baseline; XGBoost needs a low learning rate, shallow depth, and early stopping (set in the constructor in 2.x) to avoid overfitting tiny n.
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
rf = RandomForestClassifier(n_estimators=500, max_features='sqrt', min_samples_leaf=3,
class_weight='balanced', n_jobs=-1, random_state=0)
# XGBoost 2.x: early_stopping_rounds and eval_metric go in the CONSTRUCTOR, not fit().# scale_pos_weight is omitted on purpose: like resampling, it reweights the prior and# distorts calibration -- use it only for hard-label problems, not risk models (see Class Imbalance).
xgb = XGBClassifier(n_estimators=2000, learning_rate=0.03, max_depth=4, subsample=0.8,
colsample_bytree=0.5, reg_lambda=1.0,
early_stopping_rounds=50, eval_metric='aucpr', n_jobs=-1, random_state=0)
xgb.fit(X_train, y_train, eval_set=[(X_val, y_val)]) # NaN handled natively (missing=np.nan)
Detecting Batch Shortcut Learning
Goal: Rule out that a high AUC is a batch artifact rather than biology.
Approach: Try to predict the batch from the features and use batch-aware splits; if batch is confounded with the outcome, no correction rescues the design (Soneson 2014) -- fix it at the design stage.
from sklearn.model_selection import cross_val_score, StratifiedGroupKFold
from scipy.stats import chi2_contingency
import pandas as pd
# 1. Can the classifier predict the BATCH? If yes, batch is a strong axis and the label model is suspect.
batch_auc = cross_val_score(pipe, X, batch_labels, cv=5, scoring='roc_auc')
print(f'Batch predictability AUC: {batch_auc.mean():.2f} (high = shortcut risk)')
# 2. Is the outcome associated with batch by design?print('label vs batch p:', chi2_contingency(pd.crosstab(y, batch_labels))[1])
# 3. Leave-one-batch-out is the honest generalization estimate (usually << random-split CV).
gcv = StratifiedGroupKFold(n_splits=5)
honest = cross_val_score(pipe, X, y, cv=gcv, groups=batch_labels, scoring='roc_auc')
print(f'Batch-aware AUC: {honest.mean():.2f}')
Class Imbalance: What Works and What Fails
Goal: Handle a rare positive class without destroying the probabilities.
Approach: For a risk model, do not resample -- class-weight cautiously or tune the threshold on a validation fold; SMOTE/oversampling change the training prior, inflate minority probabilities, give no AUC gain, and the same sensitivity is recoverable by moving the threshold (van den Goorbergh 2022). When resampling is unavoidable (a hard-label problem), use an imblearn Pipeline so only training folds are resampled.
from imblearn.pipeline import Pipeline as ImbPipeline # NOT sklearn's Pipelinefrom imblearn.over_sampling import SMOTE
from sklearn.linear_model import LogisticRegression
# Correct placement: SMOTE's fit_resample runs only during fit on the train fold, no-op on transform.
imb = ImbPipeline([('smote', SMOTE(random_state=0)),
('clf', LogisticRegression(max_iter=5000))])
# Prefer for risk models: no resampling, then pick the operating threshold by cost on a validation fold.
Probability Calibration
Goal: Ensure a "0.9" means a 90% risk, not just a high rank.
Approach: Tree ensembles are often miscalibrated -- bagged forests bound votes away from 0 and 1, classic boosting is sigmoid-distorted toward 0.5 (Niculescu-Mizil 2005), and log-loss GBDT can overfit to overconfident extremes -- so check a reliability curve and recalibrate on a disjoint fold. Logistic regression optimizes a proper scoring rule and is usually best-calibrated out of the box. See machine-learning/model-validation for reliability curves, Brier, and the full protocol.
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator # sklearn >=1.6; cv='prefit' deprecated
calibrated = CalibratedClassifierCV(FrozenEstimator(rf.fit(X_tr, y_tr)), method='isotonic')
calibrated.fit(X_cal, y_cal) # X_cal disjoint from train and test
Preprocessing, Scaling, Encoding, Missing Data
Inside the CV fold (refit per fold via Pipeline): feature selection, scaling, log/VST/quantile normalization, ComBat batch correction, imputation, PCA. Fitting any of these on the full matrix is leakage (machine-learning/model-validation).
Variant features: additive 0/1/2 ordinal for trees/additive logistic; one-hot for non-additive (dominant/recessive) effects; CatBoost-style encoding for high-cardinality (HLA).
Missing data: XGBoost/LightGBM learn a default split direction for NaN; in omics missingness is often MNAR (below detection limit) and a missingness indicator can carry signal -- naive zero-fill conflates "absent" with "not measured."
Hyperparameters That Matter
Model
Tune these
Leave default
Logistic
C (log-spaced), l1_ratio, class_weight
solver (saga for elasticnet)
Random forest
max_features, min_samples_leaf, max_depth (cap for tiny n)
Trigger: Cases and controls processed in different batches/sites/times.
Mechanism: The classifier exploits the cleaner technical signal; even nested CV is optimistic, and ComBat cannot rescue a confounded design (Soneson 2014).
Symptom: Near-perfect CV AUC; collapse on an independent cohort; the model predicts batch easily.
Fix: Detect with the batch-prediction check; use leave-one-batch-out; fix at design (balance batches across outcome).
RF/boosting probabilities trusted as risks
Trigger: Reading predict_proba from RF or XGBoost as a calibrated risk.
Mechanism: RF bounds votes away from 0 and 1; classic boosting is sigmoid-distorted (Niculescu-Mizil 2005) and log-loss GBDT can overfit to overconfident extremes.
Symptom: Good AUC, reliability curve far from diagonal.
Fix: Recalibrate on a disjoint fold; or prefer logistic when the probability matters.
SMOTE-before-split / SMOTE for a risk model
Trigger: Resampling before the CV split, or to "fix" imbalance for a probability model.
Mechanism: Synthetic points derived from test samples leak; resampling inflates minority risk and wrecks calibration (van den Goorbergh 2022; Carriero 2025).
Symptom: Inflated CV performance; predicted risks systematically too high.
Fix:imblearn Pipeline (train-fold only); for risk models, do not resample -- tune the threshold.
OOB error read as an unbiased test estimate
Trigger: Reporting RF out-of-bag error after selecting features on the full data.
Mechanism: Selection leaked; OOB then reflects the contaminated feature set.
Symptom: Optimistic OOB; external collapse.
Fix: Selection inside CV; estimate performance by nested CV.
Quantitative Thresholds
Threshold
Source
Rationale
Try a regularized linear model first
Dudoit 2002; Statnikov 2008
Simple often beats complex in p>>n
GBDT over deep nets for tabular omics
Grinsztajn 2022
Trees handle uninformative features and non-rotational data
Do not resample for risk models
van den Goorbergh 2022; Carriero 2025
Resampling destroys calibration for no AUC gain
XGBoost: low LR + many rounds + early stopping
field standard
Prevents overfitting tiny n
Report AUPRC + MCC under imbalance
Saito 2015; Chicco 2020
Accuracy and ROC-AUC mislead when positives are rare
Common Errors
Error / symptom
Cause
Solution
XGBoost early_stopping_rounds TypeError in fit()
Moved to constructor in 2.x
Pass it (and eval_metric) in XGBClassifier(...)
penalty='l1' FutureWarning
Deprecated in sklearn 1.8
Use l1_ratio=1 + C (1.8+) or keep penalty on 1.4-1.7
elasticnet solver error
Only saga supports it
solver='saga' + l1_ratio
CalibratedClassifierCV(cv='prefit') deprecated
sklearn 1.6
Wrap in FrozenEstimator
95% accuracy but useless model
Imbalance + accuracy metric
Report AUPRC/MCC; check the confusion matrix
References
Dudoit S, Fridlyand J, Speed TP. 2002. Comparison of discrimination methods for the classification of tumors using gene expression data. J Am Stat Assoc 97:77-87.
Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. 2002. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Intell Res 16:321-357.
Zou H, Hastie T. 2005. Regularization and variable selection via the elastic net. J R Stat Soc B 67:301-320.
Niculescu-Mizil A, Caruana R. 2005. Predicting good probabilities with supervised learning. Proc 22nd ICML 625-632.
Diaz-Uriarte R, Alvarez de Andres S. 2006. Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7:3.
Statnikov A, Wang L, Aliferis CF. 2008. A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification. BMC Bioinformatics 9:319.
Soneson C, Gerster S, Delorenzi M. 2014. Batch effect confounding leads to strong bias in performance estimates obtained by cross-validation. PLoS ONE 9:e100335.
Saito T, Rehmsmeier M. 2015. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10:e0118432.
Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. 2019. Calibration: the Achilles heel of predictive analytics. BMC Med 17:230.
Chicco D, Jurman G. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21:6.
Shwartz-Ziv R, Armon A. 2022. Tabular data: deep learning is not all you need. Inf Fusion 81:84-90.
Grinsztajn L, Oyallon E, Varoquaux G. 2022. Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS Datasets and Benchmarks.
van den Goorbergh R, van Smeden M, Timmerman D, Van Calster B. 2022. The harm of class imbalance corrections for risk prediction models. J Am Med Inform Assoc 29:1525-1534.
Carriero A, Luijken K, de Hond A, Moons KGM, van Calster B, van Smeden M. 2025. The harms of class imbalance corrections for machine learning based prediction models. Stat Med 44:e10320.
Related Skills
machine-learning/model-validation - Nested CV, calibration, and net benefit for the trained classifier
machine-learning/biomarker-discovery - Select features before modeling (inside the CV fold)
machine-learning/prediction-explanation - Interpret the classifier and detect shortcuts with SHAP
machine-learning/survival-analysis - Time-to-event outcomes that classifiers cannot handle
differential-expression/batch-correction - Batch correction done design-aware, not across the split
expression-matrix/normalization - Per-sample normalization that is safe outside the CV fold
