Skip to main content Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/xjtulyc/awesome-rosetta-skills --skill ml-for-researchEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Más de este repositorio Astronomical data analysis with astropy and astroquery — FITS I/O, WCS transforms, catalog cross-matching, aperture photometry, and CMB power spectra.
Seismological data analysis with ObsPy — FDSN waveform download, response removal, phase picking, moment tensor inversion, and seismicity mapping.
Download and analyze oceanographic data from Copernicus Marine Service and Argo floats using copernicusmarine, gsw, and xarray.
Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
name ml-for-research description Use this Skill to apply machine learning in research: scikit-learn pipelines, FLAML AutoML, SHAP explainability, nested cross-validation, and model cards.
tags ["universal","machine-learning","scikit-learn","SHAP","AutoML","explainability"] version 1.0.0 authors [{"name":"awesome-rosetta-skills contributors","github":"@xjtulyc"}] license MIT platforms ["claude-code","codex","gemini-cli","cursor"] dependencies {"python":["scikit-learn>=1.2","flaml>=2.0","shap>=0.42","optuna>=3.0","pandas>=1.5","matplotlib>=3.6"]} last_updated 2026-03-18 status stable
Machine Learning for Research — Pipelines, AutoML, and Explainability
TL;DR — Build reproducible ML workflows for research: scikit-learn
ColumnTransformer pipelines, nested cross-validation, FLAML AutoML with
baseline comparison, Optuna hyperparameter search, SHAP TreeExplainer
summaries, calibration curves, and model cards for transparent reporting.
When to Use This Skill
Use this Skill whenever you need to:
Apply supervised ML to tabular research data (clinical, social science, omics)
Construct a preprocessing pipeline handling mixed numeric and categorical features
Evaluate models with nested cross-validation to obtain unbiased performance estimates
Run AutoML (FLAML) with a time budget and compare it to a simple baseline
Tune hyperparameters with Optuna TPE search
Explain model predictions globally (SHAP summary) and locally (waterfall plots)
Assess classifier calibration and apply probability calibration
Document model properties in a structured model card for reproducible reporting
Task When to apply ColumnTransformer pipeline Any dataset with mixed feature types Nested CV Primary model evaluation for a research paper FLAML AutoML Quick strong baseline; feature selection exploration SHAP TreeExplainer Tree-based models (RF, XGBoost, LightGBM) CalibratedClassifierCV Any model used for probabilistic risk prediction Model card Methods section reporting; supplementary material
Background & Key Concepts
scikit-learn Pipeline and ColumnTransformer
A Pipeline chains preprocessing steps and a final estimator so that
cross-validation splits are applied correctly — the scaler and imputer are fit
only on the training fold, preventing data leakage.
ColumnTransformer applies different transformers to numeric and categorical
columns in parallel, then concatenates the results:
Numeric : SimpleImputer(strategy='median') → StandardScaler()Categorical : SimpleImputer(strategy='most_frequent') → OneHotEncoder(handle_unknown='ignore')
Train / Validation / Test Split Split Fraction Purpose Train 60 % Model fitting and cross-validation Validation 20 % Hyperparameter selection and early stopping Test 20 % Final unbiased performance estimate (touch once)
Use stratify=y for classification to preserve class balance across splits.
Nested Cross-Validation Nested CV prevents optimistic bias when tuning and evaluating on the same data:
Outer loop : RepeatedStratifiedKFold(n_splits=5, n_repeats=3) — provides
15 performance estimates that are averaged for the paper.Inner loop : GridSearchCV or RandomizedSearchCV — tunes hyperparameters
on each outer training fold independently.
FLAML AutoML FLAML (Fast and Lightweight AutoML) explores model families and hyperparameters
within a user-defined time_budget (seconds). Key parameters:
Parameter Recommendation time_budget60–300 s for initial exploration; 3600 s for final run task'classification' or 'regression'metric'roc_auc', 'accuracy', 'r2', 'rmse'estimator_list['lgbm', 'rf', 'xgboost', 'linear'] or 'auto'
SHAP TreeExplainer SHAP (SHapley Additive exPlanations) provides feature attributions that satisfy
desirable properties (efficiency, symmetry, dummy). TreeExplainer computes
exact SHAP values for tree-based models in polynomial time.
summary_plot : global feature importance ranked by mean |SHAP|waterfall_plot : single-instance explanation showing additive contributionsdependence_plot : interaction between one feature's value and its SHAP value
Model Calibration A well-calibrated model produces probabilities that match empirical frequencies:
predicted probability 0.7 should correspond to ~70 % event rate. Use
CalibratedClassifierCV(method='isotonic') for post-hoc calibration and
CalibrationDisplay for the reliability diagram.
Environment Setup
python -m venv .venv
source .venv/bin/activate
pip install "scikit-learn>=1.2" "flaml>=2.0" "shap>=0.42" \
"optuna>=3.0" "pandas>=1.5" "matplotlib>=3.6"
python -c "import sklearn, flaml, shap, optuna, pandas, matplotlib; print('Setup OK')"
Optional: suppress verbose Optuna and FLAML logs during development:
export OPTUNA_VERBOSITY=WARNING
export FLAML_VERBOSITY=0
Core Workflow
Step 1 — Nested CV Pipeline with SHAP Summary Plot import os
import warnings
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import shap
from sklearn.datasets import make_classification
from sklearn.model_selection import (
RepeatedStratifiedKFold,
GridSearchCV,
train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
warnings.filterwarnings("ignore" )
def build_preprocessing_pipeline (
numeric_cols: list [str ],
categorical_cols: list [str ],
) -> ColumnTransformer:
"""
Build a ColumnTransformer for mixed numeric and categorical features.
Args:
numeric_cols: List of numeric feature column names.
categorical_cols: List of categorical feature column names.
Returns:
Fitted ColumnTransformer (fit inside a Pipeline to avoid leakage).
"""
numeric_transformer = Pipeline([
("imputer" , SimpleImputer(strategy="median" )),
("scaler" , StandardScaler()),
])
categorical_transformer = Pipeline([
("imputer" , SimpleImputer(strategy="most_frequent" )),
("encoder" , OneHotEncoder(handle_unknown="ignore" , sparse_output=False )),
])
return ColumnTransformer(transformers=[
("num" , numeric_transformer, numeric_cols),
("cat" , categorical_transformer, categorical_cols),
], remainder="drop" )
def run_nested_cv (
X: pd.DataFrame,
y: pd.Series,
numeric_cols: list [str ],
categorical_cols: list [str ],
outer_n_splits: int = 5 ,
outer_n_repeats: int = 3 ,
inner_cv_folds: int = 5 ,
param_grid: dict | None = None ,
random_state: int = 42 ,
) -> dict :
"""
Run nested cross-validation for a RandomForestClassifier.
Args:
X: Feature DataFrame.
y: Binary target Series.
numeric_cols: Numeric feature column names.
categorical_cols: Categorical feature column names.
outer_n_splits: Number of outer CV folds.
outer_n_repeats: Number of outer CV repeats.
inner_cv_folds: Number of inner CV folds for hyperparameter tuning.
param_grid: Hyperparameter grid for GridSearchCV. Uses default if None.
random_state: Random seed.
Returns:
Dictionary with outer AUC scores list, mean AUC, and std AUC.
"""
if param_grid is None :
param_grid = {
"clf__n_estimators" : [100 , 200 ],
"clf__max_depth" : [None , 5 , 10 ],
"clf__min_samples_leaf" : [1 , 5 ],
}
preproc = build_preprocessing_pipeline(numeric_cols, categorical_cols)
clf = RandomForestClassifier(random_state=random_state, n_jobs=-1 )
pipeline = Pipeline([("preproc" , preproc), ("clf" , clf)])
outer_cv = RepeatedStratifiedKFold(
n_splits=outer_n_splits, n_repeats=outer_n_repeats, random_state=random_state
)
inner_cv = RepeatedStratifiedKFold(n_splits=inner_cv_folds, n_repeats=1 ,
random_state=random_state)
outer_scores = []
for fold_idx, (train_idx, test_idx) in enumerate (outer_cv.split(X, y)):
X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
gs = GridSearchCV(
pipeline, param_grid, cv=inner_cv,
scoring="roc_auc" , n_jobs=-1 , refit=True ,
)
gs.fit(X_train, y_train)
y_prob = gs.predict_proba(X_test)[:, 1 ]
auc = roc_auc_score(y_test, y_prob)
outer_scores.append(auc)
if fold_idx % 5 == 0 :
print (f" Outer fold {fold_idx+1 } /{outer_n_splits*outer_n_repeats} : AUC = {auc:.4 f} " )
mean_auc = float (np.mean(outer_scores))
std_auc = float (np.std(outer_scores, ddof=1 ))
print (f"\nNested CV result: AUC = {mean_auc:.4 f} ± {std_auc:.4 f} " )
return {"outer_scores" : outer_scores, "mean_auc" : mean_auc, "std_auc" : std_auc}
def compute_shap_values_and_plot (
X_train: pd.DataFrame,
X_test: pd.DataFrame,
y_train: pd.Series,
numeric_cols: list [str ],
categorical_cols: list [str ],
output_path: str = "shap_summary.png" ,
max_display: int = 15 ,
) -> None :
"""
Fit a RandomForest on X_train, compute SHAP values on X_test, and save summary plot.
Args:
X_train: Training features (DataFrame).
X_test: Test features (DataFrame).
y_train: Training labels.
numeric_cols: Numeric feature column names.
categorical_cols: Categorical feature column names.
output_path: Path to save the SHAP summary plot.
max_display: Maximum features to display in the summary plot.
"""
preproc = build_preprocessing_pipeline(numeric_cols, categorical_cols)
clf = RandomForestClassifier(n_estimators=200 , random_state=42 , n_jobs=-1 )
pipeline = Pipeline([("preproc" , preproc), ("clf" , clf)])
pipeline.fit(X_train, y_train)
X_test_transformed = pipeline.named_steps["preproc" ].transform(X_test)
feature_names = (
numeric_cols
+ list (pipeline.named_steps["preproc" ]
.named_transformers_["cat" ]
.named_steps["encoder" ]
.get_feature_names_out(categorical_cols))
)
explainer = shap.TreeExplainer(pipeline.named_steps["clf" ])
shap_values = explainer.shap_values(X_test_transformed)
sv = shap_values[1 ] if isinstance (shap_values, list ) else shap_values
plt.figure(figsize=(10 , 6 ))
shap.summary_plot(
sv,
X_test_transformed,
feature_names=feature_names,
max_display=max_display,
show=False ,
plot_type="dot" ,
)
plt.tight_layout()
plt.savefig(output_path, dpi=150 , bbox_inches="tight" )
plt.close()
print (f"SHAP summary plot saved to {output_path} " )
if __name__ == "__main__" :
X_raw, y_raw = make_classification(
n_samples=500 , n_features=20 , n_informative=10 ,
n_redundant=4 , random_state=0 ,
)
feature_names_num = [f"lab_{i} " for i in range (15 )]
feature_names_cat = [f"group_{i} " for i in range (5 )]
df = pd.DataFrame(X_raw, columns=feature_names_num + feature_names_cat)
for col in feature_names_cat:
df[col] = pd.cut(df[col], bins=3 , labels=["low" , "med" , "high" ])
target = pd.Series(y_raw, name="outcome" )
X_train_full, X_test, y_train_full, y_test = train_test_split(
df, target, test_size=0.20 , stratify=target, random_state=42
)
print ("Running nested cross-validation..." )
results = run_nested_cv(
X_train_full, y_train_full,
numeric_cols=feature_names_num,
categorical_cols=feature_names_cat,
)
print (f"\nFinal nested CV AUC: {results['mean_auc' ]:.3 f} ± {results['std_auc' ]:.3 f} " )
print ("\nComputing SHAP values..." )
compute_shap_values_and_plot(
X_train_full, X_test, y_train_full,
numeric_cols=feature_names_num,
categorical_cols=feature_names_cat,
output_path="shap_summary.png" ,
)
Step 2 — FLAML AutoML and Comparison with Baseline import time
import pandas as pd
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from flaml import AutoML
def run_flaml_automl (
X_train: np.ndarray | pd.DataFrame,
y_train: np.ndarray | pd.Series,
X_test: np.ndarray | pd.DataFrame,
y_test: np.ndarray | pd.Series,
time_budget: int = 60 ,
metric: str = "roc_auc" ,
task: str = "classification" ,
estimator_list: list [str ] | None = None ,
seed: int = 42 ,
) -> dict :
"""
Run FLAML AutoML and compare with a logistic regression baseline.
Args:
X_train: Training features.
y_train: Training labels.
X_test: Test features.
y_test: Test labels.
time_budget: Maximum seconds for FLAML search.
metric: Optimization metric (e.g., 'roc_auc', 'accuracy').
task: ML task type: 'classification' or 'regression'.
estimator_list: Model families to consider; None = auto.
seed: Random seed.
Returns:
Dictionary with AutoML and baseline AUC, best model name, and elapsed time.
"""
if estimator_list is None :
estimator_list = ["lgbm" , "rf" , "xgboost" , "lrl2" ]
dummy = DummyClassifier(strategy="stratified" , random_state=seed)
dummy.fit(X_train, y_train)
dummy_auc = roc_auc_score(y_test, dummy.predict_proba(X_test)[:, 1 ])
lr = LogisticRegression(max_iter=1000 , random_state=seed)
lr.fit(X_train, y_train)
lr_auc = roc_auc_score(y_test, lr.predict_proba(X_test)[:, 1 ])
automl = AutoML()
settings = {
"time_budget" : time_budget,
"metric" : metric,
"task" : task,
"estimator_list" : estimator_list,
"seed" : seed,
"verbose" : 0 ,
}
t0 = time.time()
automl.fit(X_train, y_train, **settings)
elapsed = round (time.time() - t0, 1 )
automl_prob = automl.predict_proba(X_test)[:, 1 ]
automl_auc = roc_auc_score(y_test, automl_prob)
print (f"Baseline (Dummy) AUC: {dummy_auc:.4 f} " )
print (f"Baseline (LogReg) AUC: {lr_auc:.4 f} " )
print (f"FLAML AutoML AUC: {automl_auc:.4 f} "
f"[best model: {automl.best_estimator} , elapsed: {elapsed} s]" )
return {
"dummy_auc" : round (dummy_auc, 4 ),
"logreg_auc" : round (lr_auc, 4 ),
"automl_auc" : round (automl_auc, 4 ),
"best_estimator" : automl.best_estimator,
"best_config" : automl.best_config,
"elapsed_sec" : elapsed,
"model" : automl,
}
if __name__ == "__main__" :
X, y = make_classification(n_samples=800 , n_features=25 , n_informative=12 ,
random_state=1 )
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2 ,
stratify=y, random_state=1 )
result = run_flaml_automl(X_tr, y_tr, X_te, y_te, time_budget=60 )
print (f"\nAUC lift over LogReg: "
f"{(result['automl_auc' ] - result['logreg_auc' ])*100 :+.2 f} pp" )
Step 3 — Model Card Generation import json
import datetime
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.calibration import CalibratedClassifierCV, CalibrationDisplay
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
from sklearn.metrics import (
roc_auc_score, average_precision_score, brier_score_loss,
classification_report,
)
MODEL_CARD_TEMPLATE = {
"model_name" : "" ,
"version" : "1.0.0" ,
"date_trained" : "" ,
"intended_use" : {
"primary_use" : "" ,
"out_of_scope" : [],
},
"training_data" : {
"source" : "" ,
"n_samples" : 0 ,
"n_features" : 0 ,
"target_prevalence" : 0.0 ,
"preprocessing" : [],
},
"evaluation_data" : {
"source" : "" ,
"n_samples" : 0 ,
},
"performance" : {
"roc_auc" : 0.0 ,
"average_precision" : 0.0 ,
"brier_score" : 0.0 ,
"calibrated" : False ,
},
"limitations" : [],
"ethical_considerations" : [],
"caveats_and_recommendations" : [],
}
def generate_model_card (
model_name: str ,
y_test: np.ndarray,
y_prob: np.ndarray,
training_info: dict ,
limitations: list [str ],
ethical_considerations: list [str ],
output_path: str = "model_card.json" ,
) -> dict :
"""
Populate and save a model card dictionary with performance metrics.
Args:
model_name: Short name for the model.
y_test: True binary labels on the held-out test set.
y_prob: Predicted probabilities for the positive class.
training_info: Dict with keys: source, n_train, n_features,
prevalence, preprocessing.
limitations: List of known model limitations.
ethical_considerations: List of ethical considerations.
output_path: File path to save the JSON model card.
Returns:
Populated model card dictionary.
"""
import copy
card = copy.deepcopy(MODEL_CARD_TEMPLATE)
card["model_name" ] = model_name
card["version" ] = "1.0.0"
card["date_trained" ] = datetime.date.today().isoformat()
card["training_data" ]["source" ] = training_info.get("source" , "" )
card["training_data" ]["n_samples" ] = training_info.get("n_train" , 0 )
card["training_data" ]["n_features" ] = training_info.get("n_features" , 0 )
card["training_data" ]["target_prevalence" ] = round (float (np.mean(y_test)), 4 )
card["training_data" ]["preprocessing" ] = training_info.get("preprocessing" , [])
card["evaluation_data" ]["n_samples" ] = len (y_test)
card["performance" ]["roc_auc" ] = round (roc_auc_score(y_test, y_prob), 4 )
card["performance" ]["average_precision" ] = round (
average_precision_score(y_test, y_prob), 4
)
card["performance" ]["brier_score" ] = round (brier_score_loss(y_test, y_prob), 4 )
card["limitations" ] = limitations
card["ethical_considerations" ] = ethical_considerations
with open (output_path, "w" , encoding="utf-8" ) as fh:
json.dump(card, fh, indent=2 )
print (f"Model card saved to {output_path} " )
return card
def plot_calibration_curve (
y_test: np.ndarray,
y_prob_uncal: np.ndarray,
y_prob_cal: np.ndarray,
output_path: str = "calibration_curve.png" ,
) -> None :
"""Plot reliability diagram comparing uncalibrated and calibrated probabilities."""
fig, ax = plt.subplots(figsize=(7 , 5 ))
CalibrationDisplay.from_predictions(
y_test, y_prob_uncal, n_bins=10 , ax=ax, name="Uncalibrated"
)
CalibrationDisplay.from_predictions(
y_test, y_prob_cal, n_bins=10 , ax=ax, name="Calibrated (isotonic)"
)
ax.set_title("Calibration Curve (Reliability Diagram)" )
plt.tight_layout()
plt.savefig(output_path, dpi=150 )
print (f"Calibration curve saved to {output_path} " )
if __name__ == "__main__" :
X, y = make_classification(n_samples=600 , n_features=20 ,
n_informative=10 , random_state=5 )
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2 ,
stratify=y, random_state=5 )
rf = RandomForestClassifier(n_estimators=100 , random_state=5 )
rf.fit(X_tr, y_tr)
y_prob_uncal = rf.predict_proba(X_te)[:, 1 ]
cal_rf = CalibratedClassifierCV(rf, method="isotonic" , cv=5 )
cal_rf.fit(X_tr, y_tr)
y_prob_cal = cal_rf.predict_proba(X_te)[:, 1 ]
plot_calibration_curve(y_te, y_prob_uncal, y_prob_cal)
card = generate_model_card(
model_name="RandomForest-AD-Classifier" ,
y_test=y_te,
y_prob=y_prob_cal,
training_info={
"source" : "Synthetic research dataset" ,
"n_train" : len (X_tr),
"n_features" : X_tr.shape[1 ],
"prevalence" : float (y_tr.mean()),
"preprocessing" : ["StandardScaler" , "SimpleImputer(median)" ],
},
limitations=[
"Trained on synthetic data; external validation required." ,
"Does not account for temporal drift in feature distributions." ,
],
ethical_considerations=[
"Model should not be used for clinical decision-making without "
"prospective validation." ,
"Performance has not been assessed in subgroups defined by race, "
"sex, or socioeconomic status." ,
],
)
print ("\nModel Card:" )
import json
print (json.dumps(card["performance" ], indent=2 ))
Advanced Usage
Optuna Hyperparameter Search import optuna
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification
optuna.logging.set_verbosity(optuna.logging.WARNING)
def optuna_objective (
trial: optuna.Trial,
X: np.ndarray,
y: np.ndarray,
cv_folds: int = 5 ,
) -> float :
"""
Optuna objective for RandomForest hyperparameter tuning.
Suggests n_estimators and max_depth via the trial object, fits a
RandomForest with cross-validation, and returns mean AUC.
Args:
trial: Optuna trial object.
X: Feature matrix.
y: Binary labels.
cv_folds: Number of stratified CV folds.
Returns:
Mean ROC-AUC across CV folds (to be maximized).
"""
n_estimators = trial.suggest_int("n_estimators" , 50 , 500 , step=50 )
max_depth = trial.suggest_int("max_depth" , 3 , 15 )
min_samples_leaf = trial.suggest_int("min_samples_leaf" , 1 , 20 )
max_features = trial.suggest_float("max_features" , 0.3 , 1.0 )
clf = RandomForestClassifier(
n_estimators=n_estimators,
max_depth=max_depth,
min_samples_leaf=min_samples_leaf,
max_features=max_features,
random_state=42 ,
n_jobs=-1 ,
)
scores = cross_val_score(clf, X, y, cv=cv_folds, scoring="roc_auc" , n_jobs=-1 )
return float (scores.mean())
X, y = make_classification(n_samples=500 , n_features=20 , random_state=0 )
study = optuna.create_study(direction="maximize" ,
sampler=optuna.samplers.TPESampler(seed=42 ))
study.optimize(lambda trial: optuna_objective(trial, X, y), n_trials=50 , show_progress_bar=False )
print (f"Best AUC: {study.best_value:.4 f} " )
print (f"Best params: {study.best_params} " )
Permutation Feature Importance import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.inspection import permutation_importance
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400 , n_features=15 , n_informative=8 , random_state=3 )
feature_names = [f"feature_{i:02d} " for i in range (X.shape[1 ])]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2 , random_state=3 )
clf = RandomForestClassifier(n_estimators=100 , random_state=3 )
clf.fit(X_tr, y_tr)
perm = permutation_importance(clf, X_te, y_te, n_repeats=20 ,
scoring="roc_auc" , random_state=3 , n_jobs=-1 )
perm_df = pd.DataFrame({
"feature" : feature_names,
"importance_mean" : perm.importances_mean,
"importance_std" : perm.importances_std,
}).sort_values("importance_mean" , ascending=False )
print (perm_df.head(10 ).to_string(index=False ))
Troubleshooting Problem Likely cause Fix ValueError: could not convert string to floatCategorical columns passed to scaler Ensure ColumnTransformer routes categoricals to OneHotEncoder SHAP TypeError: unhashable type Non-numeric transformed features Use get_feature_names_out() after transformation FLAML timeout reached immediately time_budget too smallSet time_budget ≥ 30 s; check system clock Nested CV very slow Too many grid combinations Use RandomizedSearchCV instead of GridSearchCV Calibration curve not flat Overfit uncalibrated probabilities Apply CalibratedClassifierCV(method='isotonic', cv=5) Optuna TrialPruned errors Intermediate value reporting missing Remove pruning callbacks if not using early stopping Memory error in SHAP Dataset too large for TreeExplainer Subsample X_test to 500–1000 rows for SHAP computation
External Resources
Examples
Example 1 — Full Nested CV on a Clinical Dataset import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
import numpy as np
X_raw, y_raw = make_classification(n_samples=600 , n_features=18 ,
n_informative=9 , random_state=10 )
num_cols = [f"biomarker_{i} " for i in range (13 )]
cat_cols = [f"category_{i} " for i in range (5 )]
df = pd.DataFrame(X_raw, columns=num_cols + cat_cols)
for col in cat_cols:
df[col] = pd.cut(df[col], bins=3 , labels=["A" , "B" , "C" ])
y = pd.Series(y_raw, name="outcome" )
X_tr, X_te, y_tr, y_te = train_test_split(df, y, test_size=0.2 ,
stratify=y, random_state=42 )
nested_results = run_nested_cv(
X_tr, y_tr, numeric_cols=num_cols, categorical_cols=cat_cols,
outer_n_splits=5 , outer_n_repeats=3 ,
)
print (f"Nested CV AUC: {nested_results['mean_auc' ]:.3 f} ± {nested_results['std_auc' ]:.3 f} " )
Example 2 — FLAML AutoML vs Logistic Regression Baseline from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=1000 , n_features=30 ,
n_informative=15 , random_state=99 )
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2 ,
stratify=y, random_state=99 )
result = run_flaml_automl(X_tr, y_tr, X_te, y_te, time_budget=60 )
print (f"\nBest model: {result['best_estimator' ]} " )
print (f"AUC improvement over LogReg: "
f"{(result['automl_auc' ] - result['logreg_auc' ])*100 :+.2 f} pp" )
Example 3 — Model Card with Calibration from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV
X, y = make_classification(n_samples=700 , n_features=20 , random_state=7 )
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2 ,
stratify=y, random_state=7 )
base = RandomForestClassifier(n_estimators=100 , random_state=7 )
cal = CalibratedClassifierCV(base, method="isotonic" , cv=5 )
cal.fit(X_tr, y_tr)
y_prob = cal.predict_proba(X_te)[:, 1 ]
card = generate_model_card(
model_name="CalibratedRF-v1" ,
y_test=y_te,
y_prob=y_prob,
training_info={
"source" : "Demo dataset" ,
"n_train" : len (X_tr),
"n_features" : 20 ,
"preprocessing" : ["StandardScaler" ],
},
limitations=["Synthetic data only; requires real-world validation." ],
ethical_considerations=["No subgroup fairness analysis performed." ],
output_path="model_card_demo.json" ,
)
import json
print (json.dumps(card["performance" ], indent=2 ))
Changelog Version Date Change 1.0.0 2026-03-18 Initial release — nested CV, FLAML AutoML, SHAP, calibration, Optuna, model cards
