| name | dataset-cleaning |
| description | Clean, normalize, and prepare raw datasets for analysis or ML. Covers missing value handling, deduplication, outlier treatment, type normalization, categorical encoding, and transformation logging. |
| tags | ["data-cleaning","data-preparation","data-quality","imputation","deduplication","normalization","dataset-curation"] |
Dataset Cleaning
Overview
Dataset cleaning transforms raw, messy data into a consistent, analysis-ready form. Every operation must be documented: what was changed, why, and how many rows/columns were affected. This is not a one-shot script — it's a reproducible pipeline.
When to Use
Use this skill when:
- Raw data has missing values, duplicates, inconsistent types, or outliers.
- Data comes from multiple sources with conflicting formats or encodings.
- You need a documented cleaning pipeline, not silent
dropna().
- The dataset will be used for modeling, analysis, or sharing.
Do not use for:
- Exploratory analysis without transformation — use
exploratory-data-analysis first.
- Splitting strategies — use
dataset-splitting.
- Version tracking — use
dataset-versioning.
Cleaning Workflow
1. Missing Value Strategy
Choose and document ONE strategy per column:
| Strategy | When to use | Risk |
||-------------|------|
| Drop rows | < 5% missing, rows are independent | Loss of rare cases |
| Drop column | > 40% missing and not critical | Loss of signal |
| Mean/median imputation | Continuous, symmetric distribution | Underestimates variance |
| | Categorical, dominant class clear | Over-represents majority |
| | Domain-knowledge default exists | May introduce bias |
| | High missingness, strong predictors | Leakage if not careful |
| | Missingness itself is informative | Adds dimensionality |
