feature-engineering

name	feature-engineering
description	Techniques for creating, transforming, and selecting features to improve machine learning model performance, including encoding, scaling, dimensionality reduction, and automated feature generation.
category	data-science
keywords	["feature-engineering","feature-selection","feature-creation","encoding","scaling","dimensionality-reduction","feature-extraction","feature-importance"]
difficulty	intermediate
related_skills	["pandas","numpy","scikit-learn","exploratory-data-analysis"]

Feature Engineering

What I do

I provide techniques for creating, transforming, and selecting features that improve machine learning model performance. I help you derive meaningful predictors from raw data, handle categorical variables, create interaction features, reduce dimensionality, and select the most informative features. Effective feature engineering is often more impactful than model choice.

When to use me

• Converting raw data into meaningful predictors
• Handling categorical variables (encoding)
• Creating date/time features
• Generating interaction and polynomial features
• Scaling and normalizing features
• Reducing feature dimensionality
• Selecting most important features
• Handling missing values strategically

Core Concepts

Feature Types

• Numerical: Continuous, discrete, counts
• Categorical: Nominal (no order), ordinal (has order)
• Temporal: Dates, times, durations
• Text: Words, n-grams, embeddings
• Derived: Computed from other features

Encoding Methods

• One-Hot: Binary columns for each category
• Label Encoding: Integer mapping for categories
• Target Encoding: Category mapped to target statistics
• Frequency Encoding: Category mapped to frequency
• Embedding: Dense vector representation

Feature Transformations

• Scaling: Standardization, normalization, robust scaling
• Log/Box-Cox: Handle skewed distributions
• Binning: Convert continuous to categorical
• Polynomial: Capture non-linear relationships

Feature Selection

• Filter Methods: Statistical tests, correlation
• Wrapper Methods: Forward/backward selection
• Embedded Methods: L1 regularization, tree importance
• Permutation Importance: Model-agnostic importance

Dimensionality Reduction

• PCA: Principal Component Analysis
• t-SNE/UMAP: Non-linear manifold learning
• Factor Analysis: Latent factor discovery

Code Examples (Python)

import pandas as pd
import numpy as np
from sklearn.preprocessing import (
    StandardScaler, MinMaxScaler, RobustScaler,
    OneHotEncoder, LabelEncoder, OrdinalEncoder,
    PolynomialFeatures, KBinsDiscretizer
)
from sklearn.feature_selection import (
    SelectKBest, f_classif, mutual_info_classif,
    RFE, SelectFromModel, VarianceThreshold
)
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from scipy import stats

# Categorical encoding
# One-Hot Encoding
df = pd.DataFrame({'category': ['A', 'B', 'C', 'A', 'B']})
ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
encoded = ohe.fit_transform(df[['category']])
ohe_df = pd.DataFrame(encoded, columns=ohe.get_feature_names_out())

# Label Encoding (for ordinal or tree-based models)
le = LabelEncoder()
df['category_encoded'] = le.fit_transform(df['category'])

# Frequency Encoding
freq = df['category'].value_counts(normalize=True)
df['category_freq'] = df['category'].map(freq)

# Target Encoding (with regularization to prevent leakage)
def target_encode(train_df, test_df, col, target, smoothing=10):
    global_mean = train_df[target].mean()
    agg = train_df.groupby(col)[target].agg(['mean', 'count'])
    smooth = (agg[] * agg[] + smoothing * global_mean) / (agg[] + smoothing)
    train_encoded = train_df[col].(smooth)
    test_encoded = test_df[col].(smooth).fillna(global_mean)
     train_encoded, test_encoded


ordinal_map = {: , : , : }
df[] = df[].(ordinal_map)



scaler = StandardScaler()
df_scaled = pd.DataFrame(
    scaler.fit_transform(df[[]]),
    columns=[]
)


scaler = MinMaxScaler()
df_normalized = pd.DataFrame(
    scaler.fit_transform(df[[]]),
    columns=[]
)


scaler = RobustScaler()
df_robust = pd.DataFrame(
    scaler.fit_transform(df[[]]),
    columns=[]
)


df[] = np.log1p(df[])  
df[] = np.log(df[].clip(lower=))


df[], lambda_param = stats.boxcox(df[].clip(lower=))


df[] = np.sqrt(df[])



df[] = pd.cut(df[], bins=, labels=[, , , , ])


df[] = pd.qcut(df[], q=, labels=)


discretizer = KBinsDiscretizer(n_bins=, encode=, strategy=)
df[] = discretizer.fit_transform(df[[]]).astype()


poly = PolynomialFeatures(degree=, include_bias=)
poly_features = poly.fit_transform(df[[, ]])
poly_df = pd.DataFrame(poly_features, columns=poly.get_feature_names_out())


df[] = pd.to_datetime(df[])
df[] = df[].dt.year
df[] = df[].dt.month
df[] = df[].dt.day
df[] = df[].dt.dayofweek
df[] = df[].dt.quarter
df[] = df[].dt.dayofweek.isin([, ]).astype()
df[] = df[].dt.hour
df[] = df[].between(, ).astype()
df[] = (df[] - reference_date).dt.days
df[] = np.sin( * np.pi * df[] / )
df[] = np.cos( * np.pi * df[] / )


 sklearn.feature_extraction.text  TfidfVectorizer, CountVectorizer


tfidf = TfidfVectorizer(max_features=, ngram_range=(, ))
tfidf_features = tfidf.fit_transform(df[])
tfidf_df = pd.DataFrame(tfidf_features.toarray(), columns=tfidf.get_feature_names_out())


count_vec = CountVectorizer(max_features=)
count_features = count_vec.fit_transform(df[])


df = df.sort_values()
df[] = df[].rolling(window=, min_periods=).mean()
df[] = df[].rolling(window=, min_periods=).std()
df[] = df[].rolling(window=, min_periods=).()
df[] = df[].expanding().mean()
df[] = df[].diff()
df[] = df[].pct_change()


df[] = df.groupby()[].transform()
df[] = df.groupby()[].transform()
df[] = df.groupby()[].transform()
df[] = df.groupby()[].transform()
df[] = df.groupby()[].transform()
df[] = df.groupby()[].rank(ascending=)


df[] = df[] * df[]
df[] = df[] / (df[] + )  



var_selector = VarianceThreshold(threshold=)
X_selected = var_selector.fit_transform(X)


 ():
    corr_matrix = df.corr().()
    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=).astype())
    to_drop = [column  column  upper.columns  (upper[column] > threshold)]
     df.drop(columns=to_drop)


selector = SelectKBest(score_func=f_classif, k=)
X_selected = selector.fit_transform(X, y)
selected_features = X.columns[selector.get_support()]


mi_scores = mutual_info_classif(X, y)
mi_df = pd.DataFrame({: X.columns, : mi_scores})
mi_df = mi_df.sort_values(, ascending=)


 sklearn.ensemble  RandomForestClassifier
rfe = RFE(estimator=RandomForestClassifier(n_estimators=), n_features_to_select=)
rfe.fit(X, y)
selected = X.columns[rfe.support_]


rf = RandomForestClassifier(n_estimators=)
rf.fit(X, y)
importance_df = pd.DataFrame({
    : X.columns,
    : rf.feature_importances_
}).sort_values(, ascending=)


selector = SelectFromModel(rf, threshold=)
X_selected = selector.fit_transform(X, y)



pca = PCA(n_components=)  
X_pca = pca.fit_transform(X_scaled)
()


tsne = TSNE(n_components=, random_state=)
X_tsne = tsne.fit_transform(X_scaled[:])  


 ():
    importances = {}
     name, model  models.items():
         (model, ):
            importances[name] = model.fit(X, y).feature_importances_
     pd.DataFrame(importances, index=X.columns).mean(axis=).sort_values(ascending=)

Best Practices

1. Feature engineering over complex models: Simple models with good features outperform complex models with poor features.

2. Document transformations: Keep track of all feature engineering steps for reproducibility.

3. Use domain knowledge: Create features that capture domain-specific patterns.

4. Handle target leakage: Never use information from the target when creating features.

5. Create interpretable features: Understandable features are easier to debug and maintain.

6. Test features individually: Evaluate feature usefulness before combining.

7. Handle edge cases: Division by zero, infinite values, extreme outliers.

8. Automate feature engineering: Use feature tools or AutoML for large-scale feature creation.

Common Patterns

Pattern 1: Automated Feature Engineering

class FeatureEngineer:
    def __init__(self, df, target_col=None):
        self.df = df
        self.target_col = target_col
        self.features = []
    
    def add_numeric_features(self, cols):
        for col in cols:
            self.features.extend([
                f'{col}_log', 
                f'{col}_sqrt',
                f'{col}_square',
                f'{col}_reciprocal'
            ])
        return self
    
    def add_interaction_features(self, cols):
        for i, c1 in enumerate(cols):
            for c2 in cols[i+1:]:
                self.features.extend([
                    f'{c1}_x_{c2}',
                    f'{c1}_plus_{c2}',
                    f'{c1}_minus_{c2}',
                    f'{c1}_ratio_{c2}'
                ])
        return self
    
     ():
         agg  [, , , , ]:
            .features.append()
         
    
     ():
         window  windows:
            .features.extend([
                ,
                ,
                ,
                
            ])
         
    
     ():
        
         feat  .features:
            .df[feat] = ._compute_feature(feat)
         .df
    
     ():
        
        
        

Pattern 2: Feature Selection Pipeline

def select_features(X, y, n_features=20):
    """Comprehensive feature selection."""
    # Step 1: Remove constant features
    var_selector = VarianceThreshold(threshold=0)
    X_var = var_selector.fit_transform(X)
    var_features = X.columns[var_selector.get_support()]
    
    # Step 2: Remove highly correlated
    X_corr = remove_correlated_features(X[var_features], threshold=0.95)
    
    # Step 3: Select K Best
    selector = SelectKBest(score_func=f_classif, k=min(n_features, len(X_corr.columns)))
    selector.fit(X_corr, y)
    kbest_features = X_corr.columns[selector.get_support()]
    
    # Step 4: Recursive Feature Elimination
    from sklearn.linear_model import LogisticRegression
    rfe = RFE(estimator=LogisticRegression(max_iter=1000), n_features_to_select=n_features)
    rfe.fit(X_corr[kbest_features], y)
    final_features = kbest_features[rfe.support_]
    
    return final_features

Pattern 3: Categorical Feature Encoding Strategy

def smart_encode(train_df, test_df, cat_cols, target_col, cardinality_threshold=10):
    """Choose encoding strategy based on cardinality."""
    encoded_train = train_df.copy()
    encoded_test = test_df.copy()
    
    for col in cat_cols:
        n_unique = train_df[col].nunique()
        
        if n_unique == 2:
            # Binary: Label encode
            le = LabelEncoder()
            combined = pd.concat([train_df[col], test_df[col]])
            le.fit(combined)
            encoded_train[col] = le.transform(train_df[col])
            encoded_test[col] = le.transform(test_df[col])
        
        elif n_unique <= cardinality_threshold:
            # Low cardinality: One-hot encode
            ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
            train_encoded = ohe.fit_transform(encoded_train[[col]])
            test_encoded = ohe.transform(encoded_test[[col]])
            
            train_encoded_df = pd.DataFrame(
                train_encoded, 
                columns=[f'{col}_{cat}' for cat in ohe.categories_[0]],
                index=encoded_train.index
            )
            test_encoded_df = pd.DataFrame(
                test_encoded,
                columns=[f'{col}_{cat}' for cat in ohe.categories_[0]],
                index=encoded_test.index
            )
            
            encoded_train = pd.concat([encoded_train, train_encoded_df], axis=1)
            encoded_test = pd.concat([encoded_test, test_encoded_df], axis=1)
            encoded_train = encoded_train.drop(col, axis=1)
            encoded_test = encoded_test.drop(col, axis=1)
        
        else:
            # High cardinality: Target encode
            train_encoded, test_encoded = target_encode(
                encoded_train, encoded_test, col, target_col, smoothing=
            )
            encoded_train[] = train_encoded
            encoded_test[] = test_encoded
            encoded_train = encoded_train.drop(col, axis=)
            encoded_test = encoded_test.drop(col, axis=)
    
     encoded_train, encoded_test