| name | data-wrangling |
| description | Data cleaning, transformation, reshaping, joins, missing data handling, and tidy data principles. Covers the full pipeline from raw ingestion to analysis-ready datasets -- type coercion, deduplication, outlier detection, normalization, melting/pivoting, regex extraction, and reproducible transformation chains. Use when preparing, cleaning, or transforming data for analysis. |
| type | skill |
| category | data-science |
| status | stable |
| origin | tibsfox |
| modified | false |
| first_seen | "2026-04-12T00:00:00.000Z" |
| first_path | examples/skills/data-science/data-wrangling/SKILL.md |
| superseded_by | null |
Data Wrangling
Data wrangling is the work that sits between raw data and analysis -- the unglamorous, indispensable practice of making data trustworthy. Estimates vary, but practitioners consistently report that 60-80% of analysis time is spent wrangling. This skill covers the principles and techniques of data cleaning, transformation, reshaping, and integration, grounded in Hadley Wickham's tidy data framework and extended to the realities of messy real-world datasets.
Agent affinity: tukey (EDA-driven cleaning), nightingale (routing wrangling tasks)
Concept IDs: data-data-sources, data-data-quality, data-sampling-bias
The Wrangling Pipeline
| Stage | Goal | Key operations |
|---|
| 1. Ingestion | Get data into a working environment | Read CSV/JSON/Parquet/SQL, handle encodings, parse dates |
| 2. Profiling | Understand what you have | Shape, dtypes, nulls, distributions, cardinality |
| 3. Cleaning | Fix structural problems | Dedup, type coercion, standardize categories, fix encodings |
| 4. Missing data | Handle gaps | Detect patterns (MCAR/MAR/MNAR), impute or flag |
| 5. Transformation | Derive analysis-ready features | Normalize, bin, log-transform, create indicators |
| 6. Reshaping | Match the analysis structure | Melt, pivot, tidy form, denormalize |
| 7. Integration | Combine sources | Joins (inner/left/right/full/cross), concatenation, dedup post-join |
| 8. Validation | Confirm readiness | Schema checks, assertion tests, row-count reconciliation |
Tidy Data Principles
Hadley Wickham (2014) formalized "tidy data" as three rules:
- Each variable forms a column. A single column should contain values of exactly one variable.
- Each observation forms a row. A single row should contain all values for exactly one observational unit.
- Each type of observational unit forms a table. Mixing patient demographics and lab results in one table violates this rule.
Most messy datasets violate one or more of these rules in predictable ways:
