feature-engineer

name

feature-engineer

description

Comprehensive feature engineering for ML pipelines: data quality assessment, feature creation, selection, transformation, and encoding. Activates for "feature engineering", "create features", "feature selection", "data preprocessing", "handle missing values", "encode categorical", "scale features", "feature importance". Ensures features are production-ready with automated validation, documentation, and integration with SpecWeave increments.

Feature Engineer

Overview

Feature engineering often makes the difference between mediocre and excellent ML models. This skill transforms raw data into model-ready features through systematic data quality assessment, feature creation, selection, and transformation—all integrated with SpecWeave's increment workflow.

The Feature Engineering Pipeline

Phase 1: Data Quality Assessment

Before creating features, understand your data:

from specweave import DataQualityReport

# Automated data quality check
report = DataQualityReport(df, increment="0042")

# Generates:
# - Missing value analysis
# - Outlier detection
# - Data type validation
# - Distribution analysis
# - Correlation matrix
# - Duplicate detection

Quality Report Output:

# Data Quality Report

## Dataset Overview
- Rows: 100,000
- Columns: 45
- Memory: 34.2 MB

## Missing Values
| Column          | Missing | Percentage |
|-----------------|---------|------------|
| email           | 15,234  | 15.2%      |
| phone           | 8,901   | 8.9%       |
| purchase_date   | 0       | 0.0%       |

## Outliers Detected
- transaction_amount: 234 outliers (>3 std dev)
- user_age: 12 outliers (<18 or >100)

## Data Type Issues
- user_id: Stored as float, should be int
- date_joined: Stored as string, should be datetime

## Recommendations
1. Impute email/phone or create "missing" indicator features
2. Cap/remove outliers in transaction_amount
3. Convert data types for efficiency

Phase 2: Feature Creation

Create features from domain knowledge:

from specweave import FeatureCreator

creator = FeatureCreator(df, increment="0042")

# Temporal features (from datetime)
creator.add_temporal_features(
    date_column="purchase_date",
    features=["hour", "day_of_week", "month", "is_weekend", "is_holiday"]
)

# Aggregation features (user behavior)
creator.add_aggregation_features(
    group_by="user_id",
    target="purchase_amount",
    aggs=["mean", "std", "count", "min", "max"]
)
# Creates: user_purchase_amount_mean, user_purchase_amount_std, etc.

# Interaction features
creator.add_interaction_features(
    features=[("age", "income"), ("clicks", "impressions")],
    operations=["multiply", "divide", "subtract"]
)
# Creates: age_x_income, clicks_per_impression, etc.

# Ratio features
creator.add_ratio_features([
    ("revenue", "cost"),
    ("conversions", "visits")
])
# Creates: revenue_to_cost_ratio, conversion_rate

# Binning (discretization)
creator.add_binned_features(
    column="age",
    bins=[0, 18, 25, 35, 50, 65, 100],
    labels=["child", "young_adult", , , , ]
)


creator.add_text_features(
    column=,
    features=[, , , ]
)


df_enriched = creator.generate()


creator.save_feature_definitions(
    path=
)

Feature Definitions (auto-generated):

# .specweave/increments/0042.../features/feature_definitions.yaml

features:
  - name: purchase_hour
    type: temporal
    source: purchase_date
    description: Hour of purchase (0-23)
    
  - name: user_purchase_amount_mean
    type: aggregation
    source: purchase_amount
    group_by: user_id
    description: Average purchase amount per user
    
  - name: age_x_income
    type: interaction
    sources: [age, income]
    operation: multiply
    description: Product of age and income
    
  - name: conversion_rate
    type: ratio
    sources: [conversions, visits]
    description: Conversion    

Phase 3: Feature Selection

Reduce dimensionality, improve performance:

from specweave import FeatureSelector

selector = FeatureSelector(X_train, y_train, increment="0042")

# Method 1: Correlation-based (remove redundant features)
selector.remove_correlated_features(threshold=0.95)
# Removes features with >95% correlation

# Method 2: Variance-based (remove constant features)
selector.remove_low_variance_features(threshold=0.01)
# Removes features with <1% variance

# Method 3: Statistical tests
selector.select_by_statistical_test(k=50)
# SelectKBest with chi2/f_classif

# Method 4: Model-based (tree importance)
selector.select_by_model_importance(
    model=RandomForestClassifier(),
    threshold=0.01
)
# Removes features with <1% importance

# Method 5: Recursive Feature Elimination
selector.select_by_rfe(
    model=LogisticRegression(),
    n_features=30
)

# Get selected features
selected_features = selector.get_selected_features()

# Generate selection report
selector.generate_report()

Feature Selection Report:

# Feature Selection Report

## Original Features: 125
## Selected Features: 35 (72% reduction)

## Selection Process
1. Removed 12 correlated features (>95% correlation)
2. Removed 8 low-variance features
3. Statistical test: Selected top 50 (chi-squared)
4. Model importance: Removed 15 low-importance features (<1%)

## Top 10 Features (by importance)
1. user_purchase_amount_mean (0.18)
2. days_since_last_purchase (0.12)
3. total_purchases (0.10)
4. age_x_income (0.08)
5. conversion_rate (0.07)
...

## Removed Features
- user_id_hash (constant)
- temp_feature_1 (99% correlated with temp_feature_2)
- random_noise (0% importance)
...

Phase 4: Feature Transformation

Scale, normalize, encode for model compatibility:

from specweave import FeatureTransformer

transformer = FeatureTransformer(increment="0042")

# Numerical transformations
transformer.add_numerical_transformer(
    columns=["age", "income", "purchase_amount"],
    method="standard_scaler"  # Or: min_max, robust, quantile
)

# Categorical encoding
transformer.add_categorical_encoder(
    columns=["country", "device_type", "product_category"],
    method="onehot",  # Or: label, target, binary
    handle_unknown="ignore"
)

# Ordinal encoding (for ordered categories)
transformer.add_ordinal_encoder(
    column="education",
    order=["high_school", "bachelors", "masters", "phd"]
)

# Log transformation (for skewed distributions)
transformer.add_log_transform(
    columns=["transaction_amount", "page_views"],
    method="log1p"  # log(1 + x) to handle zeros
)

# Box-Cox transformation (for normalization)
transformer.add_power_transform(
    columns=["revenue", "engagement_score"],
    method="box-cox"
)

# Custom transformation
def clip_outliers(x):
    return np.clip(x, x.quantile(0.01), x.quantile(0.99))

transformer.add_custom_transformer(
    columns=["outlier_prone_feature"],
    func=clip_outliers
)

# Fit and transform
X_train_transformed = transformer.fit_transform(X_train)
X_test_transformed = transformer.transform(X_test)


transformer.save(
    path=
)

Phase 5: Feature Validation

Ensure features are production-ready:

from specweave import FeatureValidator

validator = FeatureValidator(
    X_train, X_test,
    increment="0042"
)

# Check for data leakage
leakage_report = validator.check_data_leakage()
# Detects: perfectly correlated features, future data in training

# Check for distribution drift
drift_report = validator.check_distribution_drift()
# Compares train vs test distributions

# Check for missing values after transformation
missing_report = validator.check_missing_values()

# Check for infinite/NaN values
invalid_report = validator.check_invalid_values()

# Generate validation report
validator.generate_report()

Validation Report:

# Feature Validation Report

## Data Leakage: ✅ PASS
No perfect correlations detected between train and test.

## Distribution Drift: ⚠️  WARNING
Features with significant drift (KS test p < 0.05):
- user_age: p=0.023 (minor drift)
- device_type: p=0.001 (major drift)

Recommendation: Check if test data is from different time period.

## Missing Values: ✅ PASS
No missing values after transformation.

## Invalid Values: ✅ PASS
No infinite or NaN values detected.

## Overall: READY FOR TRAINING
2 warnings, 0 critical issues.

Integration with SpecWeave

Automatic Feature Documentation

# All feature engineering steps logged to increment
with track_experiment("feature-engineering-v1", increment="0042") as exp:
    # Create features
    df_enriched = creator.generate()
    
    # Select features
    selected = selector.select()
    
    # Transform features
    X_transformed = transformer.fit_transform(X)
    
    # Validate
    validation = validator.validate()
    
    # Auto-logs:
    exp.log_param("original_features", 125)
    exp.log_param("created_features", 45)
    exp.log_param("selected_features", 35)
    exp.log_metric("feature_reduction", 0.72)
    exp.save_artifact("feature_definitions.yaml")
    exp.save_artifact("transformer.pkl")
    exp.save_artifact("validation_report.md")

Living Docs Integration

After completing feature engineering:

/sw:sync-docs update

Updates:

<!-- .specweave/docs/internal/architecture/feature-engineering.md -->

## Recommendation Model Features (Increment 0042)

### Feature Engineering Pipeline
1. Data Quality: 100K rows, 45 columns
2. Created: 45 new features (temporal, aggregation, interaction)
3. Selected: 35 features (72% reduction via importance + RFE)
4. Transformed: StandardScaler for numerical, OneHot for categorical

### Key Features
- user_purchase_amount_mean: Average user spend (top feature, 18% importance)
- days_since_last_purchase: Recency indicator (12% importance)
- age_x_income: Interaction feature (8% importance)

### Feature Store
All features documented in: `.specweave/increments/0042.../features/`
- feature_definitions.yaml: Feature catalog
- transformer.pkl: Production transformation pipeline
- validation_report.md: Quality checks

Best Practices

1. Document Feature Rationale

# Bad: Create features without explanation
df["feature_1"] = df["col_a"] * df["col_b"]

# Good: Document why features were created
creator.add_interaction_feature(
    sources=["age", "income"],
    operation="multiply",
    rationale="High-income older users have different behavior patterns"
)

2. Handle Missing Values Systematically

# Options for missing values:
# 1. Imputation (mean, median, mode)
creator.impute_missing(column="age", strategy="median")

# 2. Indicator features (flag missing as signal)
creator.add_missing_indicator(column="email")
# Creates: email_missing (0/1)

# 3. Forward/backward fill (for time series)
creator.fill_missing(column="sensor_reading", method="ffill")

# 4. Model-based imputation
creator.impute_with_model(column="income", model=RandomForestRegressor())

3. Avoid Data Leakage

# ❌ WRONG: Fit on all data (includes test set!)
scaler.fit(X)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

# ✅ CORRECT: Fit only on train, transform both
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

# SpecWeave's transformer enforces this pattern
transformer.fit_transform(X_train)  # Fits
transformer.transform(X_test)        # Only transforms

4. Version Feature Engineering Pipeline

# Version features with increment
transformer.save(
    path=".specweave/increments/0042.../features/transformer-v1.pkl",
    metadata={
        "version": "v1",
        "features": selected_features,
        "transformations": ["standard_scaler", "onehot"]
    }
)

# Load specific version for reproducibility
transformer_v1 = FeatureTransformer.load(
    ".specweave/increments/0042.../features/transformer-v1.pkl"
)

5. Test Feature Engineering on New Data

# Before deploying, test on held-out data
X_production_sample = load_production_data()

try:
    X_transformed = transformer.transform(X_production_sample)
except Exception as e:
    raise FeatureEngineeringError(f"Failed on production data: {e}")
    
# Check for unexpected values
validator = FeatureValidator(X_train, X_production_sample)
validation_report = validator.validate()

if validation_report["status"] == "CRITICAL":
    raise FeatureEngineeringError("Feature engineering failed validation")

Common Feature Engineering Patterns

Pattern 1: RFM (Recency, Frequency, Monetary)

# For e-commerce / customer analytics
creator.add_rfm_features(
    user_id="user_id",
    transaction_date="purchase_date",
    transaction_amount="purchase_amount"
)
# Creates:
# - recency: days since last purchase
# - frequency: total purchases
# - monetary: total spend

Pattern 2: Rolling Window Aggregations

# For time series
creator.add_rolling_features(
    column="daily_sales",
    windows=[7, 14, 30],
    aggs=["mean", "std", "min", "max"]
)
# Creates: daily_sales_7day_mean, daily_sales_7day_std, etc.

Pattern 3: Target Encoding (Categorical → Numerical)

# Encode categorical as target mean (careful: can leak!)
creator.add_target_encoding(
    column="product_category",
    target="purchase_amount",
    cv_folds=5  # Cross-validation to prevent leakage
)
# Creates: product_category_target_encoded

Pattern 4: Polynomial Features

# For non-linear relationships
creator.add_polynomial_features(
    columns=["age", "income"],
    degree=2,
    interaction_only=True
)
# Creates: age^2, income^2, age*income

Commands

# Generate feature engineering pipeline for increment
/ml:engineer-features 0042

# Validate features before training
/ml:validate-features 0042

# Generate feature importance report
/ml:feature-importance 0042

Integration with Other Skills

• ml-pipeline-orchestrator: Task 2 is "Feature Engineering" (uses this skill)
• experiment-tracker: Logs all feature engineering experiments
• model-evaluator: Uses feature importance from models
• ml-deployment-helper: Packages feature transformer for production

Summary

Feature engineering is 70% of ML success. This skill ensures:

• ✅ Systematic approach (quality → create → select → transform → validate)
• ✅ No data leakage (train/test separation enforced)
• ✅ Production-ready (versioned, validated, documented)
• ✅ Reproducible (all steps tracked in increment)
• ✅ Traceable (feature definitions in living docs)

Good features make mediocre models great. Great features make mediocre models excellent.