Explore, clean, and engineer datasets end-to-end: statistical profiling, distribution checks, missing value analysis, duplicate detection, outlier removal, type fixing, encoding, create features, encode categories, transform columns, add rolling windows, build interaction terms, and feature engineering. Supports pandas, polars, and PySpark. Use when the user wants to explore data, profile columns, understand a dataset, clean data, handle missing values, remove duplicates, fix data types, preprocess a dataset before modeling, create features, encode categories, transform columns, add rolling windows, build interaction terms, or do feature engineering.
Explore, clean, and engineer datasets end-to-end: statistical profiling, distribution checks, missing value analysis, duplicate detection, outlier removal, type fixing, encoding, create features, encode categories, transform columns, add rolling windows, build interaction terms, and feature engineering. Supports pandas, polars, and PySpark. Use when the user wants to explore data, profile columns, understand a dataset, clean data, handle missing values, remove duplicates, fix data types, preprocess a dataset before modeling, create features, encode categories, transform columns, add rolling windows, build interaction terms, or do feature engineering.
allowed-tools
Bash(uv run * scripts/eda.py *) Bash(uv run * scripts/clean.py *) Bash(uv run * scripts/engineer_features.py *) Read Write Glob Grep
Default value dominance (one value has suspiciously high frequency)
Round number bias (all values multiples of 5 or 10 — suggests estimation)
Stale data (updated_at shows no recent changes in an active system)
Phase 2: Data Cleaning
Quick start
# Full cleaning pipeline
uv run ${CLAUDE_SKILL_DIR}/scripts/clean.py data.csv -o clean.csv
# Clean without outlier removal
uv run ${CLAUDE_SKILL_DIR}/scripts/clean.py data.csv --no-outliers -o clean.csv
# Save cleaning report as JSON
uv run ${CLAUDE_SKILL_DIR}/scripts/clean.py data.csv -o clean.csv --report report.json
# Quality check only (no cleaning)
uv run ${CLAUDE_SKILL_DIR}/scripts/clean.py data.csv --check-only
The clean.py script runs the full cleaning pipeline: deduplication, type fixing, missing value handling, outlier removal, and validation. It prints a report to stderr and outputs the cleaned CSV.
Cleaning order (always follow this sequence)
Step
Operation
Strategy
1
Remove duplicates
drop_duplicates(subset=key_cols)
2
Fix data types
Auto-detect dates, cast numerics, strip strings
3
Handle missing
Numeric: median. Categorical: mode/"unknown". Critical: drop
4
Remove outliers
IQR (1.5x) or Z-score (threshold=3)
5
Normalize text
Lowercase, strip whitespace
6
Encode categoricals
Label (ordinal) or one-hot (nominal)
7
Validate ranges
Domain constraints (age>0, price>=0)
8
Generate report
Before/after stats
Alternative frameworks
Polars (large datasets, faster)
import polars as pl
defprepare(df: pl.DataFrame) -> pl.DataFrame:
return (
df.unique()
.with_columns([
pl.col(c).fill_null(pl.col(c).median()) for c in df.select(pl.col(pl.Float64)).columns
])
.with_columns([
pl.col(c).fill_null("unknown") for c in df.select(pl.col(pl.Utf8)).columns
])
)
PySpark (distributed)
from pyspark.sql import DataFrame
from pyspark.sql.functions import col, mean
defprepare(df: DataFrame) -> DataFrame:
df = df.dropDuplicates()
for field in df.schema.fields:
if field.dataType.simpleString() in ("double", "float", "int"):
avg = df.select(mean(col(field.name))).first()[0]
df = df.fillna({field.name: avg or0})
elif field.dataType.simpleString() == "string":
df = df.fillna({field.name: "unknown"})
return df
Transforms clean data into model-ready features. Run the script for automated engineering, or use the recipes below for custom transforms.
Decision guide
Data type
Transform
Skewed numeric
Log, sqrt
High cardinality categorical
Target/frequency encoding
Low cardinality categorical
One-hot
Datetime
Year/month/day + cyclical
Free text
Length, word count
Multiple numeric
Interactions, ratios
Time series
Rolling stats, lags, diffs
Grouped data
Aggregations, deviation from mean
Quick start
# Auto-engineer all columns
uv run ${CLAUDE_SKILL_DIR}/scripts/engineer_features.py data/clean.csv -o data/features.csv
# Engineer specific columns with target encoding
uv run ${CLAUDE_SKILL_DIR}/scripts/engineer_features.py data/clean.csv --cols age income category --target price -o features.csv
# Generate interaction features
uv run ${CLAUDE_SKILL_DIR}/scripts/engineer_features.py data/clean.csv --interactions -o features.csv
# Time series features
uv run ${CLAUDE_SKILL_DIR}/scripts/engineer_features.py data/clean.csv --cols revenue --types timeseries -o features.csv
# Group aggregations
uv run ${CLAUDE_SKILL_DIR}/scripts/engineer_features.py data/clean.csv --group segment revenue -o features.csv
# Summary as JSON
uv run ${CLAUDE_SKILL_DIR}/scripts/engineer_features.py data/clean.csv --json
from sklearn.feature_selection import mutual_info_classif
mi = mutual_info_classif(X.fillna(0), y, random_state=42)
top = pd.Series(mi, index=X.columns).sort_values(ascending=False).head(20)
print(top)
