sklearn-advanced

# scikit-learn - Advanced Architecture To move beyond simple scripts, you must master the Pipeline API. This allows you to treat your entire preprocessing and modeling sequence as a single object, ensuring that your training logic is identical to your production inference logic. ## When to Use - Building complex feature engineering flows for heterogeneous data. - Creating reusable, custom preprocessing steps (e.g., domain-specific cleaning). - Performing rigorous hyperparameter tuning without data leakage. - Implementing ensemble methods beyond standard Random Forest. - Monitoring and interpreting model decisions (Partial Dependence, Permutation Importance). - Exporting models for high-performance production environments. ## Reference Documentation - **Pipeline Guide**: https://scikit-learn.org/stable/modules/compose.html - **Custom Estimators**: https://scikit-learn.org/stable/developers/develop.html - **Model Evaluation**: https://scikit-learn.org/stable/modules/model_evaluation.html - **Search patterns**: `sklearn.base.BaseEstimator`, `sklearn.compose.make_column_selector`, `sklearn.model_selection.GridSearchCV` ## Core Principles ### Everything is an Object Every step in your workflow should be an estimator. If you find yourself doing manual pandas operations between training and testing, you are risking Data Leakage. ### The Pipeline Contract A Pipeline ensures that `.fit()` is only called on training data and `.transform()` is applied consistently to both train and test sets. ### Heterogeneous Data handling Use `ColumnTransformer` to apply different logic to numerical, categorical, and text data in parallel, then merge the results automatically. ## Quick Reference ### Standard Imports ```python import numpy as np import pandas as pd from sklearn.base import BaseEstimator, TransformerMixin from sklearn.pipeline import Pipeline, FeatureUnion from sklearn.compose import ColumnTransformer, make_column_selector from sklearn.preprocessing import FunctionTransformer, StandardScaler, OneHotEncoder from sklearn.model_selection import cross_validate, StratifiedKFold ``` ### Basic Pattern - Professional Pipeline ```python # 1. Define Preprocessor preprocessor = ColumnTransformer( transformers=[ ('num', StandardScaler(), make_column_selector(dtype_include=np.number)), ('cat', OneHotEncoder(handle_unknown='ignore'), make_column_selector(dtype_include=object)) ]) # 2. Create the Full Pipeline clf = Pipeline(steps=[ ('preprocessor', preprocessor), ('classifier', RandomForestClassifier()) ]) # 3. Fit and Tune (The entire pipeline is tuned together) # Use 'classifier__' prefix to access parameters inside the pipeline param_grid = {'classifier__n_estimators': [100, 200]} grid = GridSearchCV(clf, param_grid, cv=5).fit(X_train, y_train) ``` ## Critical Rules ### ✅ DO - **Inherit from BaseEstimator and TransformerMixin** - This gives you `.fit_transform()` and `.get_params()` for free. - **Use check_is_fitted** - In custom transformers, always verify the model is trained before allowing `.transform()`. - **Set handle_unknown='ignore'** - In OneHotEncoder, this prevents crashes if a new category appears in production. - **Use TransformedTargetRegressor** - If you need to log-transform the target variable (Y), use this to automate the inverse transformation for predictions. - **Prefer cross_validate over cross_val_score** - It allows multiple metrics and returns training scores to detect overfitting. - **Set n_jobs=-1** - Maximize CPU usage during GridSearch and Cross-validation. ### ❌ DON'T - **Don't use fit_transform on Test Data** - This is the #1 cause of over-optimistic results. - **Don't implement fit if it's not needed** - For stateless transformations (like log-transform), use `FunctionTransformer`. - **Don't hardcode Column Names** - Use `make_column_selector` to make your pipelines resilient to new columns. - **Don't ignore the Pipeline index** - If a pipeline fails, use `pipe.named_steps['step_name']` to inspect internal state. ## Custom Estimator Development ### Creating a Custom Feature Selector ```python from sklearn.base import BaseEstimator, TransformerMixin from sklearn.utils.validation import check_is_fitted class VarianceSelector(BaseEstimator, TransformerMixin): def __init__(self, threshold=0.01): self.threshold = threshold def fit(self, X, y=None): X = pd.DataFrame(X) self.variances_ = X.var() self.columns_to_keep_ = self.variances_[self.variances_ > self.threshold].index self.n_features_in_ = X.shape[1] return self def transform(self, X): check_is_fitted(self) X = pd.DataFrame(X) return X[self.columns_to_keep_] ``` ## Advanced Preprocessing ### Target Encoding (Handling high-cardinality categories) ```python from sklearn.preprocessing import TargetEncoder # Efficiently encodes categories like 'City' or 'ZipCode' # based on the average target value, with internal cross-validation encoder = TargetEncoder(smooth="auto") X_encoded = encoder.fit_transform(X_cat, y) ``` ### Stacking and Voting Ensembles ```python from sklearn.ensemble import StackingClassifier from sklearn.linear_model import LogisticRegression from sklearn.svm import SVC estimators = [ ('rf', RandomForestClassifier()), ('svc', Pipeline([('scaler', StandardScaler()), ('svr', SVC())])) ] # Use a meta-learner (LogisticRegression) to combine base model predictions stack_clf = StackingClassifier( estimators=estimators, final_estimator=LogisticRegression() ) ``` ## Model Evaluation & Diagnostics ### Rigorous Cross-Validation ```python from sklearn.model_selection import cross_validate scoring = ['accuracy', 'precision_macro', 'recall_macro', 'f1_macro'] results = cross_validate(clf, X, y, cv=5, scoring=scoring, return_train_score=True) print(f"Test F1: {results['test_f1_macro'].mean():.4f}") print(f"Train F1: {results['train_f1_macro'].mean():.4f}") # Check for gap (overfitting) ``` ### Calibration Curves (Ensuring probabilities are real) ```python from sklearn.calibration import CalibrationDisplay # A well-calibrated model's predicted probability matches the actual frequency CalibrationDisplay.from_estimator(clf, X_test, y_test, n_bins=10) ``` ## Production & Persistence ### Using Joblib for large models ```python import joblib # Save model joblib.dump(clf, 'final_model.joblib', compress=3) # Load model loaded_model = joblib.load('final_model.joblib') ``` ### Exporting to ONNX (High-speed inference) ```python # requires skl2onnx from skl2onnx import convert_sklearn from skl2onnx.common.data_types import FloatTensorType initial_type = [('float_input', FloatTensorType([None, X.shape[1]]))] onx = convert_sklearn(clf, initial_types=initial_type) with open("model.onnx", "wb") as f: f.write(onx.SerializeToString()) ``` ## Practical Workflows ### 1. Handling Missing Data and Outliers automatically ```python from sklearn.impute import KNNImputer from sklearn.ensemble import IsolationForest def build_robust_pipe(): return Pipeline([ ('imputer', KNNImputer(n_neighbors=5)), # FunctionTransformer for outlier removal is tricky because # it changes row count. IsolationForest is better used for filtering. ('scaler', StandardScaler()), ('model', GradientBoostingClassifier()) ]) ``` ### 2. Time-Series Split (Avoid future leakage) ```python from sklearn.model_selection import TimeSeriesSplit tscv = TimeSeriesSplit(n_splits=5) # Use this cv object in GridSearchCV grid = GridSearchCV(model, params, cv=tscv) ``` ### 3. Feature Union (Parallel Feature Extraction) ```python from sklearn.pipeline import FeatureUnion from sklearn.decomposition import PCA from sklearn.feature_selection import SelectKBest # Extract PCA features AND SelectKBest features in parallel combined_features = FeatureUnion([ ("pca", PCA(n_components=2)), ("univ_select", SelectKBest(k=5)) ]) pipe = Pipeline([ ("features", combined_features), ("clf", RandomForestClassifier()) ]) ``` ## Performance Optimization ### Cache Pipeline Results If your preprocessing (like KNNImputer) is slow and you are doing GridSearch, use memory to cache the transformer output. ```python from tempfile import mkdtemp from shutil import rmtree cachedir = mkdtemp() pipe = Pipeline(steps=[...], memory=cachedir) # Clean up after # rmtree(cachedir) ``` ## Common Pitfalls and Solutions ### The "LabelEncoder for X" Error LabelEncoder is only for labels (y). For features (X), always use OrdinalEncoder or OneHotEncoder. ### Column Mismatch in Production The Pipeline stores the training column order. If you pass a DataFrame with different column order in production, it might fail or give wrong results. ```python # ✅ Solution: Ensure your pipeline is the first point of entry # for raw data, or use a custom transformer that sorts columns. ``` ### Leakage during Hyperparameter Tuning Standard Cross-validation inside a Pipeline is safe. But if you perform feature selection (like SelectKBest) before the Pipeline, you have leaked information from the whole dataset into your model. ```python # ✅ Solution: Always include feature selection AS A STEP in the Pipeline. ``` Advanced scikit-learn is about discipline. By forcing all data transformations into the Pipeline/Transformer architecture, you create models that are not only accurate but also robust, maintainable, and ready for real-world deployment.