Comprehensive feature engineering for ML pipelines: data quality assessment, feature creation, selection, transformation, and encoding. Activates for "feature engineering", "create features", "feature selection", "data preprocessing", "handle missing values", "encode categorical", "scale features", "feature importance". Ensures features are production-ready with automated validation, documentation, and integration with SpecWeave increments.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Comprehensive feature engineering for ML pipelines: data quality assessment, feature creation, selection, transformation, and encoding. Activates for "feature engineering", "create features", "feature selection", "data preprocessing", "handle missing values", "encode categorical", "scale features", "feature importance". Ensures features are production-ready with automated validation, documentation, and integration with SpecWeave increments.
Feature Engineer
Overview
Feature engineering often makes the difference between mediocre and excellent ML models. This skill transforms raw data into model-ready features through systematic data quality assessment, feature creation, selection, and transformation—all integrated with SpecWeave's increment workflow.
The Feature Engineering Pipeline
Phase 1: Data Quality Assessment
Before creating features, understand your data:
from specweave import DataQualityReport
# Automated data quality check
report = DataQualityReport(df, increment="0042")
# Generates:# - Missing value analysis# - Outlier detection# - Data type validation# - Distribution analysis# - Correlation matrix# - Duplicate detection
Quality Report Output:
# Data Quality Report## Dataset Overview- Rows: 100,000
- Columns: 45
- Memory: 34.2 MB
| Column | Missing | Percentage |
|-----------------|---------|------------|
| email | 15,234 | 15.2% |
| phone | 8,901 | 8.9% |
| purchaseamount: 234 outliers (>3 std dev)
userid: Stored as float, should be int
dateamount
Convert data types for efficiency
_age: 12 outliers (<18 or >100)
## Data Type Issues
- user_
-
_joined: Stored as string, should be datetime
## Recommendations
1. Impute email/phone or create "missing" indicator features
2. Cap/remove outliers in transaction_
3.
Phase 2: Feature Creation
Create features from domain knowledge:
from specweave import FeatureCreator
creator = FeatureCreator(df, increment="0042")
# Temporal features (from datetime)
creator.add_temporal_features(
date_column="purchase_date",
features=["hour", "day_of_week", "month", "is_weekend", "is_holiday"]
)
# Aggregation features (user behavior)
creator.add_aggregation_features(
group_by="user_id",
target="purchase_amount",
aggs=["mean", "std", "count", "min", "max"]
)
# Creates: user_purchase_amount_mean, user_purchase_amount_std, etc.# Interaction features
creator.add_interaction_features(
features=[("age", "income"), ("clicks", "impressions")],
operations=["multiply", "divide", "subtract"]
)
# Creates: age_x_income, clicks_per_impression, etc.# Ratio features
creator.add_ratio_features([
("revenue", "cost"),
("conversions", "visits")
])
# Creates: revenue_to_cost_ratio, conversion_rate# Binning (discretization)
creator.add_binned_features(
column="age",
bins=[0, 18, 25, 35, 50, 65, 100],
labels=["child", "young_adult", "adult", "middle_aged", "senior", "elderly"]
)
# Text features (from text columns)
creator.add_text_features(
column="product_description",
features=["length", "word_count", "unique_words", "sentiment"]
)
# Generate all features
df_enriched = creator.generate()
# Auto-documents in increment folder
creator.save_feature_definitions(
path=".specweave/increments/0042.../features/feature_definitions.yaml"
)
from specweave import FeatureValidator
validator = FeatureValidator(
X_train, X_test,
increment="0042"
)
# Check for data leakage
leakage_report = validator.check_data_leakage()
# Detects: perfectly correlated features, future data in training# Check for distribution drift
drift_report = validator.check_distribution_drift()
# Compares train vs test distributions# Check for missing values after transformation
missing_report = validator.check_missing_values()
# Check for infinite/NaN values
invalid_report = validator.check_invalid_values()
# Generate validation report
validator.generate_report()
Validation Report:
# Feature Validation Report## Data Leakage: ✅ PASS
No perfect correlations detected between train and test.
## Distribution Drift: ⚠️ WARNING
Features with significant drift (KS test p < 0.05):
- user_age: p=0.023 (minor drift)
- device_type: p=0.001 (major drift)
Recommendation: Check if test data is from different time period.
## Missing Values: ✅ PASS
No missing values after transformation.
## Invalid Values: ✅ PASS
No infinite or NaN values detected.
## Overall: READY FOR TRAINING
2 warnings, 0 critical issues.
Integration with SpecWeave
Automatic Feature Documentation
# All feature engineering steps logged to incrementwith track_experiment("feature-engineering-v1", increment="0042") as exp:
# Create features
df_enriched = creator.generate()
# Select features
selected = selector.select()
# Transform features
X_transformed = transformer.fit_transform(X)
# Validate
validation = validator.validate()
# Auto-logs:
exp.log_param("original_features", 125)
exp.log_param("created_features", 45)
exp.log_param("selected_features", 35)
exp.log_metric("feature_reduction", 0.72)
exp.save_artifact("feature_definitions.yaml")
exp.save_artifact("transformer.pkl")
exp.save_artifact("validation_report.md")
Living Docs Integration
After completing feature engineering:
/sw:sync-docs update
Updates:
<!-- .specweave/docs/internal/architecture/feature-engineering.md -->
## Recommendation Model Features (Increment 0042)### Feature Engineering Pipeline1. Data Quality: 100K rows, 45 columns
2. Created: 45 new features (temporal, aggregation, interaction)
3. Selected: 35 features (72% reduction via importance + RFE)
4. Transformed: StandardScaler for numerical, OneHot for categorical
### Key Features- user_purchase_amount_mean: Average user spend (top feature, 18% importance)
- days_since_last_purchase: Recency indicator (12% importance)
- age_x_income: Interaction feature (8% importance)
### Feature Store
All features documented in: `.specweave/increments/0042.../features/`- feature_definitions.yaml: Feature catalog
- transformer.pkl: Production transformation pipeline
- validation_report.md: Quality checks
Best Practices
1. Document Feature Rationale
# Bad: Create features without explanation
df["feature_1"] = df["col_a"] * df["col_b"]
# Good: Document why features were created
creator.add_interaction_feature(
sources=["age", "income"],
operation="multiply",
rationale="High-income older users have different behavior patterns"
)
2. Handle Missing Values Systematically
# Options for missing values:# 1. Imputation (mean, median, mode)
creator.impute_missing(column="age", strategy="median")
# 2. Indicator features (flag missing as signal)
creator.add_missing_indicator(column="email")
# Creates: email_missing (0/1)# 3. Forward/backward fill (for time series)
creator.fill_missing(column="sensor_reading", method="ffill")
# 4. Model-based imputation
creator.impute_with_model(column="income", model=RandomForestRegressor())
3. Avoid Data Leakage
# ❌ WRONG: Fit on all data (includes test set!)
scaler.fit(X)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# ✅ CORRECT: Fit only on train, transform both
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# SpecWeave's transformer enforces this pattern
transformer.fit_transform(X_train) # Fits
transformer.transform(X_test) # Only transforms
4. Version Feature Engineering Pipeline
# Version features with increment
transformer.save(
path=".specweave/increments/0042.../features/transformer-v1.pkl",
metadata={
"version": "v1",
"features": selected_features,
"transformations": ["standard_scaler", "onehot"]
}
)
# Load specific version for reproducibility
transformer_v1 = FeatureTransformer.load(
".specweave/increments/0042.../features/transformer-v1.pkl"
)
5. Test Feature Engineering on New Data
# Before deploying, test on held-out data
X_production_sample = load_production_data()
try:
X_transformed = transformer.transform(X_production_sample)
except Exception as e:
raise FeatureEngineeringError(f"Failed on production data: {e}")
# Check for unexpected values
validator = FeatureValidator(X_train, X_production_sample)
validation_report = validator.validate()
if validation_report["status"] == "CRITICAL":
raise FeatureEngineeringError("Feature engineering failed validation")
Common Feature Engineering Patterns
Pattern 1: RFM (Recency, Frequency, Monetary)
# For e-commerce / customer analytics
creator.add_rfm_features(
user_id="user_id",
transaction_date="purchase_date",
transaction_amount="purchase_amount"
)
# Creates:# - recency: days since last purchase# - frequency: total purchases# - monetary: total spend
Pattern 2: Rolling Window Aggregations
# For time series
creator.add_rolling_features(
column="daily_sales",
windows=[7, 14, 30],
aggs=["mean", "std", "min", "max"]
)
# Creates: daily_sales_7day_mean, daily_sales_7day_std, etc.
