ml-best-practices

# ML Best Practices I want to read a story about the data, not just run code. Ensure every code cell is followed by a markdown cell analyzing the results. End the notebook with a summary comprehensively answering the prompt. If there is a good match between the user's request and a corresponding example plan, then adapt the example plan to fully answer the user's request: ## Clustering: Identify distinct groups based on their features. - Understand the schema and field descriptions. - Visualize features referenced in the prompt (e.g., with histograms, scatterplots). - Transform dates into timestamps. - Before applying encoders, check if the dataset already contains pre-encoded features and prefer existing numerical representations. - Prefer to keep data instead of dropping it when possible. - Transform ordinal data with an ordinal encoder. - Transform nominal data with a one hot encoder. - Standardize numerical features. - Perform clustering with a range of values, and collect the silhouette score. - Choose the optimal number of clusters based on the silhouette score. - Use dimensionality reduction (e.g., PCA) to project the data into two dimensions. - Scatterplot the samples in two dimensions with cluster labels as the hue. - Scatterplot the samples in two dimensions with a discrete feature as the hue. - Describe the clusters in text by feature distributions or typical feature values. - Conclusion: comprehensively answer the prompt in a final markdown cell. ## Time Series Forecasting: Develop a predictive model to estimate future values based on historical trends. How might different modeling approaches impact the prediction accuracy? - Understand the schema and field descriptions. - Visualize the target feature over time at a reasonable granularity. - Always perform a chronological split on the data to create training, validation, and test sets. - Are there seasonal trends? - Test for stationarity. - Discuss possible modeling approaches. How might different modeling approaches impact the prediction accuracy? - Train two time series forecasting models to predict the target feature. Use previous seasonality and stationarity information as model hyperparameters. - Predict the target feature for the training and validation sets. - Optionally, hypertune models with the validation set. - Visualize the actual and predicted target feature vs time for each model on the training and validation sets. - Evaluate the validation performance with error metrics. - Select a model. - Retrain the selected model on the test and validation sets. - Predict the test values with the selected model. - Visualize the average target feature and the predicted test values. - Conclusion: comprehensively answer the prompt in a final markdown cell. ## Exploratory Data Analysis / Anomaly Detection: Identify and describe any outliers, unusual patterns, or significant trends observed in the data. Provide visualizations to support your findings. - Understand the schema and field descriptions. - Visualize the target feature distribution in a way that shows outliers. - Identify and describe any outliers in the target feature. - Visualize relationships between the target feature and other features. - Identify and describe unusual patterns or significant trends. - Visualize patterns and trends. - Conclusion: comprehensively answer the prompt in a final markdown cell. ## Classification: Given the data, can we classify by the target feature? - Understand the schema and field descriptions. - Identify rows that don't make sense. How many are there and what do they contain? - Identify rows without a target value. How many are there and what do they contain? - Drop rows that don't match the schema or don't have the target value (if it is reasonable to do so). - Split data into training, validation, and test sets. - Create features to represent when data are missing, if this is meaningful. - Handle missing data. Prefer to keep data instead of dropping it when possible. - Before applying encoders, check if the dataset already contains pre-encoded features and prefer existing numerical representations. - Transform ordinal data with an ordinal encoder. - Transform nominal data with a one hot encoder. - Standardize numerical features. - Train multiple models. - If there is evidence of overfitting, regularize and retrain the model. - If there is evidence of underfitting, consider adding or engineering features. - Evaluate the models. - Create confusion matrices. - Conclusion: comprehensively answer the prompt in a final markdown cell. ## Regression: Predict the continuous valued target feature. - Understand the schema and field descriptions. - Identify rows that don't make sense. How many are there and what do they contain? - Identify rows without a target value. How many are there and what do they contain? - Develop an understanding of the data and determine how to handle missing values. This should make sense in the business context. - Identify any potential sources of group leakage. Aggregate where appropriate to prevent this. - Visualize target feature. - Split data into training, validation, and test sets. - Handle missing data. Prefer to keep data instead of dropping it when possible. - Before applying encoders, check if the dataset already contains pre-encoded features and prefer existing numerical representations. - Transform ordinal data with an ordinal encoder. - Transform nominal data with a one hot encoder. Restrict high cardinality categorical features to a tractable size. - Standardize numerical features. - Train multiple models. - Visualize the actual vs predicted values on training and validation data. - If there is evidence of overfitting, regularize and retrain the model. - If there is evidence of underfitting, consider adding or engineering features. - Evaluate the model error. - Conclusion: comprehensively answer the prompt in a final markdown cell. ## Comparing ML Models: Evaluate and compare multiple models to determine which is most suitable for production based on predictive power, robustness, and viability. - Understand the schema and align metrics with business goals (e.g., cost of false positives vs. false negatives). - Establish baselines: define a naive baseline (majority class/mean) and a simple ML baseline (e.g., Logistic/Linear Regression). - Ensure rigorous validation: use identical, fixed data splits for all models and perform $k$-fold cross-validation. - If data is temporal, use chronological splits for validation. - Select and report metrics beyond accuracy (e.g., F1-Score, PR-AUC, MAE, RMSE) that reflect business impact. - Use bootstrapping to calculate 95% confidence intervals for key metrics to determine statistical significance. - Perform slice-based error analysis: evaluate model performance across key subpopulations and demographics to identify bias or specific failure modes. - Inspect and compare confusion matrices, residual plots, and calibration curves. - Evaluate operational trade-offs: consider inference latency, training time, compute cost, and model size. - Assess interpretability using tools like SHAP or LIME where transparency is required. - Conclusion: Recommend the optimal model for the specific use case, justifying the choice with both performance and production viability. ## No match: - Understand the schema and field descriptions. - Identify rows that don't make sense. How many are there and what do they contain? - Identify rows without a target value. How many are there and what do they contain? - Drop rows that don't match the schema or don't have the target value (if it is reasonable to do so). - Create features to represent when data are missing, if this is meaningful. - Handle missing data. Prefer to keep data instead of dropping it when possible. - Before applying encoders, check if the dataset already contains pre-encoded features and prefer existing numerical representations. - Transform ordinal data with an ordinal encoder. - Transform nominal data with a one hot encoder. - Standardize numerical features. - Conclusion: comprehensively answer the prompt in a final markdown cell. ## Essential ML Practices [!IMPORTANT] ALWAYS follow these ML practices - **Strict Featurization Ordering**: For supervised learning **ALWAYS** split the dataset into training and test data **BEFORE** fitting preprocessing pipelines (e.g. scaling, encoding). Fit the pipelines on the training data and test data independently. - **Handling Missing or NULL Values**: **ALWAYS** check for and handle missing and NULL values. First, analyze their frequency. Then, decide whether to keep them, drop them or impute them with a contextually appropriate value, and explain your reasoning.