feature-engineering

# Feature Engineering ## What I do I provide techniques for creating, transforming, and selecting features that improve machine learning model performance. I help you derive meaningful predictors from raw data, handle categorical variables, create interaction features, reduce dimensionality, and select the most informative features. Effective feature engineering is often more impactful than model choice. ## When to use me - Converting raw data into meaningful predictors - Handling categorical variables (encoding) - Creating date/time features - Generating interaction and polynomial features - Scaling and normalizing features - Reducing feature dimensionality - Selecting most important features - Handling missing values strategically ## Core Concepts ### Feature Types - **Numerical**: Continuous, discrete, counts - **Categorical**: Nominal (no order), ordinal (has order) - **Temporal**: Dates, times, durations - **Text**: Words, n-grams, embeddings - **Derived**: Computed from other features ### Encoding Methods - **One-Hot**: Binary columns for each category - **Label Encoding**: Integer mapping for categories - **Target Encoding**: Category mapped to target statistics - **Frequency Encoding**: Category mapped to frequency - **Embedding**: Dense vector representation ### Feature Transformations - **Scaling**: Standardization, normalization, robust scaling - **Log/Box-Cox**: Handle skewed distributions - **Binning**: Convert continuous to categorical - **Polynomial**: Capture non-linear relationships ### Feature Selection - **Filter Methods**: Statistical tests, correlation - **Wrapper Methods**: Forward/backward selection - **Embedded Methods**: L1 regularization, tree importance - **Permutation Importance**: Model-agnostic importance ### Dimensionality Reduction - **PCA**: Principal Component Analysis - **t-SNE/UMAP**: Non-linear manifold learning - **Factor Analysis**: Latent factor discovery ## Code Examples (Python) ```python import pandas as pd import numpy as np from sklearn.preprocessing import ( StandardScaler, MinMaxScaler, RobustScaler, OneHotEncoder, LabelEncoder, OrdinalEncoder, PolynomialFeatures, KBinsDiscretizer ) from sklearn.feature_selection import ( SelectKBest, f_classif, mutual_info_classif, RFE, SelectFromModel, VarianceThreshold ) from sklearn.decomposition import PCA from sklearn.manifold import TSNE from scipy import stats # Categorical encoding # One-Hot Encoding df = pd.DataFrame({'category': ['A', 'B', 'C', 'A', 'B']}) ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore') encoded = ohe.fit_transform(df[['category']]) ohe_df = pd.DataFrame(encoded, columns=ohe.get_feature_names_out()) # Label Encoding (for ordinal or tree-based models) le = LabelEncoder() df['category_encoded'] = le.fit_transform(df['category']) # Frequency Encoding freq = df['category'].value_counts(normalize=True) df['category_freq'] = df['category'].map(freq) # Target Encoding (with regularization to prevent leakage) def target_encode(train_df, test_df, col, target, smoothing=10): global_mean = train_df[target].mean() agg = train_df.groupby(col)[target].agg(['mean', 'count']) smooth = (agg['count'] * agg['mean'] + smoothing * global_mean) / (agg['count'] + smoothing) train_encoded = train_df[col].map(smooth) test_encoded = test_df[col].map(smooth).fillna(global_mean) return train_encoded, test_encoded # Ordinal Encoding ordinal_map = {'low': 0, 'medium': 1, 'high': 2} df['ordinal'] = df['category'].map(ordinal_map) # Numerical transformations # Standardization (z-score) scaler = StandardScaler() df_scaled = pd.DataFrame( scaler.fit_transform(df[['numeric_col']]), columns=['numeric_scaled'] ) # Min-Max Scaling (0-1) scaler = MinMaxScaler() df_normalized = pd.DataFrame( scaler.fit_transform(df[['numeric_col']]), columns=['numeric_normalized'] ) # Robust scaling (using IQR) scaler = RobustScaler() df_robust = pd.DataFrame( scaler.fit_transform(df[['numeric_col']]), columns=['numeric_robust'] ) # Log transformation (handle skewness) df['log_value'] = np.log1p(df['value']) # log(1+x) for non-negative values df['log_value'] = np.log(df['value'].clip(lower=1)) # Box-Cox transformation df['boxcox_value'], lambda_param = stats.boxcox(df['value'].clip(lower=1)) # Square root transformation df['sqrt_value'] = np.sqrt(df['value']) # Binning/Discretization # Equal-width bins df['binned_equal_width'] = pd.cut(df['value'], bins=5, labels=['VL', 'L', 'M', 'H', 'VH']) # Quantile-based bins df['binned_quantile'] = pd.qcut(df['value'], q=5, labels=False) # K-bins with KBinsDiscretizer discretizer = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='quantile') df['binned_kbins'] = discretizer.fit_transform(df[['value']]).astype(int) # Polynomial features poly = PolynomialFeatures(degree=2, include_bias=False) poly_features = poly.fit_transform(df[['x1', 'x2']]) poly_df = pd.DataFrame(poly_features, columns=poly.get_feature_names_out()) # Date/Time features df['date'] = pd.to_datetime(df['date']) df['year'] = df['date'].dt.year df['month'] = df['date'].dt.month df['day'] = df['date'].dt.day df['day_of_week'] = df['date'].dt.dayofweek df['quarter'] = df['date'].dt.quarter df['is_weekend'] = df['date'].dt.dayofweek.isin([5, 6]).astype(int) df['hour'] = df['date'].dt.hour df['is_business_hours'] = df['hour'].between(9, 17).astype(int) df['days_since_reference'] = (df['date'] - reference_date).dt.days df['month_sin'] = np.sin(2 * np.pi * df['month'] / 12) df['month_cos'] = np.cos(2 * np.pi * df['month'] / 12) # Text features from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer # TF-IDF tfidf = TfidfVectorizer(max_features=100, ngram_range=(1, 2)) tfidf_features = tfidf.fit_transform(df['text_column']) tfidf_df = pd.DataFrame(tfidf_features.toarray(), columns=tfidf.get_feature_names_out()) # Count Vectorizer (Bag of Words) count_vec = CountVectorizer(max_features=100) count_features = count_vec.fit_transform(df['text_column']) # Rolling/Aggregation features df = df.sort_values('date') df['rolling_mean_7d'] = df['value'].rolling(window=7, min_periods=1).mean() df['rolling_std_7d'] = df['value'].rolling(window=7, min_periods=1).std() df['rolling_max_7d'] = df['value'].rolling(window=7, min_periods=1).max() df['expanding_mean'] = df['value'].expanding().mean() df['diff_1d'] = df['value'].diff() df['pct_change_1d'] = df['value'].pct_change() # Grouped aggregation features df['group_mean'] = df.groupby('category')['value'].transform('mean') df['group_std'] = df.groupby('category')['value'].transform('std') df['group_min'] = df.groupby('category')['value'].transform('min') df['group_max'] = df.groupby('category')['value'].transform('max') df['group_count'] = df.groupby('category')['value'].transform('count') df['group_rank'] = df.groupby('category')['value'].rank(ascending=False) # Interaction features df['interaction'] = df['feature1'] * df['feature2'] df['ratio'] = df['feature1'] / (df['feature2'] + 1) # Avoid division by zero # Feature selection # Variance threshold var_selector = VarianceThreshold(threshold=0.1) X_selected = var_selector.fit_transform(X) # Correlation-based removal def remove_correlated_features(df, threshold=0.95): corr_matrix = df.corr().abs() upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)) to_drop = [column for column in upper.columns if any(upper[column] > threshold)] return df.drop(columns=to_drop) # Select K Best selector = SelectKBest(score_func=f_classif, k=10) X_selected = selector.fit_transform(X, y) selected_features = X.columns[selector.get_support()] # Mutual information mi_scores = mutual_info_classif(X, y) mi_df = pd.DataFrame({'feature': X.columns, 'mi_score': mi_scores}) mi_df = mi_df.sort_values('mi_score', ascending=False) # Recursive Feature Elimination from sklearn.ensemble import RandomForestClassifier rfe = RFE(estimator=RandomForestClassifier(n_estimators=100), n_features_to_select=10) rfe.fit(X, y) selected = X.columns[rfe.support_] # Tree-based feature importance rf = RandomForestClassifier(n_estimators=100) rf.fit(X, y) importance_df = pd.DataFrame({ 'feature': X.columns, 'importance': rf.feature_importances_ }).sort_values('importance', ascending=False) # Select from model (threshold) selector = SelectFromModel(rf, threshold='mean') X_selected = selector.fit_transform(X, y) # Dimensionality reduction # PCA pca = PCA(n_components=0.95) # Keep 95% variance X_pca = pca.fit_transform(X_scaled) print(f"Reduced from {X.shape[1]} to {pca.n_components_} features") # t-SNE for visualization tsne = TSNE(n_components=2, random_state=42) X_tsne = tsne.fit_transform(X_scaled[:1000]) # Sample for speed # Feature importance from multiple models def get_feature_importances(models, X, y): importances = {} for name, model in models.items(): if hasattr(model, 'feature_importances_'): importances[name] = model.fit(X, y).feature_importances_ return pd.DataFrame(importances, index=X.columns).mean(axis=1).sort_values(ascending=False) ``` ## Best Practices 1. **Feature engineering over complex models**: Simple models with good features outperform complex models with poor features. 2. **Document transformations**: Keep track of all feature engineering steps for reproducibility. 3. **Use domain knowledge**: Create features that capture domain-specific patterns. 4. **Handle target leakage**: Never use information from the target when creating features. 5. **Create interpretable features**: Understandable features are easier to debug and maintain. 6. **Test features individually**: Evaluate feature usefulness before combining. 7. **Handle edge cases**: Division by zero, infinite values, extreme outliers. 8. **Automate feature engineering**: Use feature tools or AutoML for large-scale feature creation. ## Common Patterns ### Pattern 1: Automated Feature Engineering ```python class FeatureEngineer: def __init__(self, df, target_col=None): self.df = df self.target_col = target_col self.features = [] def add_numeric_features(self, cols): for col in cols: self.features.extend([ f'{col}_log', f'{col}_sqrt', f'{col}_square', f'{col}_reciprocal' ]) return self def add_interaction_features(self, cols): for i, c1 in enumerate(cols): for c2 in cols[i+1:]: self.features.extend([