data-wrangling

Data cleaning, transformation, reshaping, joins, missing data handling, and tidy data principles. Covers the full pipeline from raw ingestion to analysis-ready datasets -- type coercion, deduplication, outlier detection, normalization, melting/pivoting, regex extraction, and reproducible transformation chains. Use when preparing, cleaning, or transforming data for analysis.

Jump to install

Repository: Tibsfox/gsd-skill-creator
Last source activity: April 14, 2026 at 07:57
Detected SKILL.md language: English
Stars: 69
Forks: 9

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/Tibsfox/gsd-skill-creator --skill data-wrangling

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

SKILL.md

Source instructions · Read-only preview

Back to search Back to installation

goal-ambiguity-gate

Use the moment a request is underspecified and you are about to act on an ASSUMED goal. Enumerate the candidate goals the request could mean, project the action or artifact each one implies, and check whether they diverge: if two or more plausible goals lead to materially different actions or artifacts, ask exactly ONE targeted clarifying question; otherwise proceed on the most-likely goal and state the assumption in one line. Clarification is an evidence-producing action, not a delay — but one question, never an interrogation. Distinct from gsd-spec-phase (GSD-phase-bound, emits a heavy SPEC.md) and intent-router (fetch strategy, not goal disambiguation). Backed by agent goal-state inference (arxiv 2606.16813v1). Triggers on acting under an assumed goal when the request admits more than one materially different reading.

2026-07-19

memory-hubness-gate

Use at write time to vector/embedding memory — Grove content-addressed insertion, chroma/pgvector upserts, memory-consolidation promoting session traces to MEMORY.md, or embedding externally-ingested content. Scores each candidate record against a fixed panel of sentinel queries and quarantines any record that would become the nearest neighbor of too many unrelated queries — a hub — whether from adversarial poisoning or accidental over-generality. This is the memory-record-side sibling of skill-injection-guardian (file-side) and the write-side complement of memory-use-warrant (read-side). Quarantine, never silently drop; a human reviews. Backed by the admission-time hubness gate (arxiv 2606.19692v1). Triggers on inserting into vector memory, consolidating memory, and embedding stranger content.

2026-07-19

memory-use-warrant

Run this appropriateness check the moment you are about to integrate a retrieved long-term memory — a Grove content-addressed hit, a chroma/pgvector neighbour, a memory-consolidation digest, or a MEMORY.md line — into a response, especially anything touching private origins, Fox Companies IP, credentials, or Center Camp / consent-governed content. It answers a question intent-router never asks: not WHAT to fetch or HOW, but WHETHER a correctly-retrieved item should reach output. Relevance is not appropriateness — a perfect similarity match can still be a boundary violation. Default is FAIL-CLOSED: if in-context authorization is uncertain, the memory may inform behaviour but must not be surfaced. Backed by RBI-Eval (arxiv 2606.06055v1). Triggers on surfacing recalled sensitive memory into a response.

2026-07-19

View all from this repository

name	data-wrangling
description	Data cleaning, transformation, reshaping, joins, missing data handling, and tidy data principles. Covers the full pipeline from raw ingestion to analysis-ready datasets -- type coercion, deduplication, outlier detection, normalization, melting/pivoting, regex extraction, and reproducible transformation chains. Use when preparing, cleaning, or transforming data for analysis.
type	skill
category	data-science
status	stable
origin	tibsfox
modified	false
first_seen	"2026-04-12T00:00:00.000Z"
first_path	examples/skills/data-science/data-wrangling/SKILL.md
superseded_by	null

Data Wrangling

Data wrangling is the work that sits between raw data and analysis -- the unglamorous, indispensable practice of making data trustworthy. Estimates vary, but practitioners consistently report that 60-80% of analysis time is spent wrangling. This skill covers the principles and techniques of data cleaning, transformation, reshaping, and integration, grounded in Hadley Wickham's tidy data framework and extended to the realities of messy real-world datasets.

Agent affinity: tukey (EDA-driven cleaning), nightingale (routing wrangling tasks)

Concept IDs: data-data-sources, data-data-quality, data-sampling-bias

The Wrangling Pipeline

Stage	Goal	Key operations
1. Ingestion	Get data into a working environment	Read CSV/JSON/Parquet/SQL, handle encodings, parse dates
2. Profiling	Understand what you have	Shape, dtypes, nulls, distributions, cardinality
3. Cleaning	Fix structural problems	Dedup, type coercion, standardize categories, fix encodings
4. Missing data	Handle gaps	Detect patterns (MCAR/MAR/MNAR), impute or flag
5. Transformation	Derive analysis-ready features	Normalize, bin, log-transform, create indicators
6. Reshaping	Match the analysis structure	Melt, pivot, tidy form, denormalize
7. Integration	Combine sources	Joins (inner/left/right/full/cross), concatenation, dedup post-join
8. Validation	Confirm readiness	Schema checks, assertion tests, row-count reconciliation

Tidy Data Principles

Hadley Wickham (2014) formalized "tidy data" as three rules:

Each variable forms a column. A single column should contain values of exactly one variable.
Each observation forms a row. A single row should contain all values for exactly one observational unit.
Each type of observational unit forms a table. Mixing patient demographics and lab results in one table violates this rule.

Most messy datasets violate one or more of these rules in predictable ways:

Violation	Example

Column headers are values, not variable names	Columns: `income_2020`, `income_2021`, `income_2022`	Melt to columns: `year`, `income`
Multiple variables stored in one column	`"M-25"` encodes both sex and age	Split into `sex` and `age` columns
Variables stored in both rows and columns	Pivot table with row headers as categories	Melt and re-pivot to tidy form
Multiple types in one table	Patient info mixed with visit records	Normalize into two related tables
One type spread across multiple tables	Monthly CSV files with identical schema	Concatenate with a `month` column

Mechanism	Definition	Example	Implication
MCAR	Missingness is unrelated to any variable	Lab sample randomly dropped	Safe to drop or impute; no bias
MAR	Missingness depends on observed variables	High-income respondents skip income question less often	Imputation using observed predictors is valid
MNAR	Missingness depends on the missing value itself	People with depression less likely to report depression severity	No imputation is fully valid; requires sensitivity analysis

Strategy	When to use	Trade-off
Listwise deletion	MCAR, small fraction missing (<5%)	Simple but loses observations
Pairwise deletion	MCAR, different analyses need different subsets	Keeps more data but correlation matrices may not be positive-definite
Mean/median imputation	Quick exploration only	Reduces variance, biases correlations toward zero
Regression imputation	MAR, continuous variables	Better than mean but inflates R-squared
Multiple imputation	MAR, formal inference	Gold standard; accounts for imputation uncertainty
Indicator method	Any mechanism, tree-based models	Add a binary "was_missing" column; let the model learn missingness patterns
Domain-specific fill	Known defaults	"No response" for surveys, 0 for counts that should exist

Join	Keeps	Use when
Inner	Rows matching in both tables	You only want complete matches
Left	All rows from left, matching from right	Left table is the primary; right is enrichment
Right	All rows from right, matching from left	Symmetric to left join
Full outer	All rows from both	You need the complete picture of both sources
Cross	Every combination of left and right rows	Generating all pairs (e.g., all product-store combinations)
Anti	Left rows with NO match in right	Finding orphans or gaps

Method	Formula	Use when
Min-max	(x - min) / (max - min)	Need values in [0, 1]; distribution shape preserved
Z-score	(x - mean) / std	Need zero-centered data; assumes roughly normal
Robust scaling	(x - median) / IQR	Outliers present; median/IQR resistant to extremes
Log transform	log(x + 1)	Right-skewed data; multiplicative relationships

data-wrangling

Install options

Review the source files

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Cleaning Techniques

Type Coercion

Deduplication

Outlier Detection

Missing Data

Missing Data Mechanisms

Handling Strategies

Joins and Integration

Join Types

Join Hazards

Transformation Techniques

Normalization and Scaling

Binning and Discretization

Feature Engineering

Reproducibility

Common Mistakes

Cross-References

References

Mistake	Why it fails	Fix
Imputing before exploring	You do not know the missingness mechanism	Profile first, then choose strategy
Joining without checking cardinality	Many-to-many join silently explodes row count	Check key uniqueness before joining
Dropping outliers without investigation	Outliers may be the most important observations	Investigate, then decide
Mixing cleaning and analysis	Impossible to reproduce or audit	Separate wrangling script from analysis
Ignoring encoding issues	Garbled text, silent data loss	Specify encoding explicitly at ingestion
Normalizing before train/test split	Data leakage -- test set statistics influence training	Fit scaler on training data only

data-wrangling

Source facts

Install options

Review the source files

More from this repository

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Cleaning Techniques

Type Coercion

Deduplication

Outlier Detection

Missing Data

Missing Data Mechanisms

Handling Strategies

Joins and Integration

Join Types

Join Hazards

Transformation Techniques

Normalization and Scaling

Binning and Discretization

Feature Engineering

Reproducibility

Common Mistakes

Cross-References

References

More from this repository