data-wrangling

Data cleaning, transformation, reshaping, joins, missing data handling, and tidy data principles. Covers the full pipeline from raw ingestion to analysis-ready datasets -- type coercion, deduplication, outlier detection, normalization, melting/pivoting, regex extraction, and reproducible transformation chains. Use when preparing, cleaning, or transforming data for analysis.

설치로 이동

저장소: Tibsfox/gsd-skill-creator
최근 소스 활동: 2026년 4월 14일 07:57
감지된 SKILL.md 언어: 영어
스타: 69
포크: 9

설치 방법

기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.

npx skills add https://github.com/Tibsfox/gsd-skill-creator --skill data-wrangling

명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.

로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.

소스 파일 검토

설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.

SKILL.md

소스 지침 · 읽기 전용 미리보기

검색으로 돌아가기 설치로 돌아가기

goal-ambiguity-gate

Use the moment a request is underspecified and you are about to act on an ASSUMED goal. Enumerate the candidate goals the request could mean, project the action or artifact each one implies, and check whether they diverge: if two or more plausible goals lead to materially different actions or artifacts, ask exactly ONE targeted clarifying question; otherwise proceed on the most-likely goal and state the assumption in one line. Clarification is an evidence-producing action, not a delay — but one question, never an interrogation. Distinct from gsd-spec-phase (GSD-phase-bound, emits a heavy SPEC.md) and intent-router (fetch strategy, not goal disambiguation). Backed by agent goal-state inference (arxiv 2606.16813v1). Triggers on acting under an assumed goal when the request admits more than one materially different reading.

2026-07-19

memory-hubness-gate

Use at write time to vector/embedding memory — Grove content-addressed insertion, chroma/pgvector upserts, memory-consolidation promoting session traces to MEMORY.md, or embedding externally-ingested content. Scores each candidate record against a fixed panel of sentinel queries and quarantines any record that would become the nearest neighbor of too many unrelated queries — a hub — whether from adversarial poisoning or accidental over-generality. This is the memory-record-side sibling of skill-injection-guardian (file-side) and the write-side complement of memory-use-warrant (read-side). Quarantine, never silently drop; a human reviews. Backed by the admission-time hubness gate (arxiv 2606.19692v1). Triggers on inserting into vector memory, consolidating memory, and embedding stranger content.

2026-07-19

memory-use-warrant

Run this appropriateness check the moment you are about to integrate a retrieved long-term memory — a Grove content-addressed hit, a chroma/pgvector neighbour, a memory-consolidation digest, or a MEMORY.md line — into a response, especially anything touching private origins, Fox Companies IP, credentials, or Center Camp / consent-governed content. It answers a question intent-router never asks: not WHAT to fetch or HOW, but WHETHER a correctly-retrieved item should reach output. Relevance is not appropriateness — a perfect similarity match can still be a boundary violation. Default is FAIL-CLOSED: if in-context authorization is uncertain, the memory may inform behaviour but must not be surfaced. Backed by RBI-Eval (arxiv 2606.06055v1). Triggers on surfacing recalled sensitive memory into a response.

2026-07-19

이 저장소의 모든 Skills 보기

name	data-wrangling
description	Data cleaning, transformation, reshaping, joins, missing data handling, and tidy data principles. Covers the full pipeline from raw ingestion to analysis-ready datasets -- type coercion, deduplication, outlier detection, normalization, melting/pivoting, regex extraction, and reproducible transformation chains. Use when preparing, cleaning, or transforming data for analysis.
type	skill
category	data-science
status	stable
origin	tibsfox
modified	false
first_seen	"2026-04-12T00:00:00.000Z"
first_path	examples/skills/data-science/data-wrangling/SKILL.md
superseded_by	null

Data Wrangling

Data wrangling is the work that sits between raw data and analysis -- the unglamorous, indispensable practice of making data trustworthy. Estimates vary, but practitioners consistently report that 60-80% of analysis time is spent wrangling. This skill covers the principles and techniques of data cleaning, transformation, reshaping, and integration, grounded in Hadley Wickham's tidy data framework and extended to the realities of messy real-world datasets.

Agent affinity: tukey (EDA-driven cleaning), nightingale (routing wrangling tasks)

Concept IDs: data-data-sources, data-data-quality, data-sampling-bias

The Wrangling Pipeline

Stage	Goal	Key operations
1. Ingestion	Get data into a working environment	Read CSV/JSON/Parquet/SQL, handle encodings, parse dates
2. Profiling	Understand what you have	Shape, dtypes, nulls, distributions, cardinality
3. Cleaning	Fix structural problems	Dedup, type coercion, standardize categories, fix encodings
4. Missing data	Handle gaps	Detect patterns (MCAR/MAR/MNAR), impute or flag
5. Transformation	Derive analysis-ready features	Normalize, bin, log-transform, create indicators
6. Reshaping	Match the analysis structure	Melt, pivot, tidy form, denormalize
7. Integration	Combine sources	Joins (inner/left/right/full/cross), concatenation, dedup post-join
8. Validation	Confirm readiness	Schema checks, assertion tests, row-count reconciliation

Tidy Data Principles

Hadley Wickham (2014) formalized "tidy data" as three rules:

Each variable forms a column. A single column should contain values of exactly one variable.
Each observation forms a row. A single row should contain all values for exactly one observational unit.
Each type of observational unit forms a table. Mixing patient demographics and lab results in one table violates this rule.

Most messy datasets violate one or more of these rules in predictable ways:

Violation	Example

Column headers are values, not variable names	Columns: `income_2020`, `income_2021`, `income_2022`	Melt to columns: `year`, `income`
Multiple variables stored in one column	`"M-25"` encodes both sex and age	Split into `sex` and `age` columns
Variables stored in both rows and columns	Pivot table with row headers as categories	Melt and re-pivot to tidy form
Multiple types in one table	Patient info mixed with visit records	Normalize into two related tables
One type spread across multiple tables	Monthly CSV files with identical schema	Concatenate with a `month` column

Mechanism	Definition	Example	Implication
MCAR	Missingness is unrelated to any variable	Lab sample randomly dropped	Safe to drop or impute; no bias
MAR	Missingness depends on observed variables	High-income respondents skip income question less often	Imputation using observed predictors is valid
MNAR	Missingness depends on the missing value itself	People with depression less likely to report depression severity	No imputation is fully valid; requires sensitivity analysis

Strategy	When to use	Trade-off
Listwise deletion	MCAR, small fraction missing (<5%)	Simple but loses observations
Pairwise deletion	MCAR, different analyses need different subsets	Keeps more data but correlation matrices may not be positive-definite
Mean/median imputation	Quick exploration only	Reduces variance, biases correlations toward zero
Regression imputation	MAR, continuous variables	Better than mean but inflates R-squared
Multiple imputation	MAR, formal inference	Gold standard; accounts for imputation uncertainty
Indicator method	Any mechanism, tree-based models	Add a binary "was_missing" column; let the model learn missingness patterns
Domain-specific fill	Known defaults	"No response" for surveys, 0 for counts that should exist

Join	Keeps	Use when
Inner	Rows matching in both tables	You only want complete matches
Left	All rows from left, matching from right	Left table is the primary; right is enrichment
Right	All rows from right, matching from left	Symmetric to left join
Full outer	All rows from both	You need the complete picture of both sources
Cross	Every combination of left and right rows	Generating all pairs (e.g., all product-store combinations)
Anti	Left rows with NO match in right	Finding orphans or gaps

Method	Formula	Use when
Min-max	(x - min) / (max - min)	Need values in [0, 1]; distribution shape preserved
Z-score	(x - mean) / std	Need zero-centered data; assumes roughly normal
Robust scaling	(x - median) / IQR	Outliers present; median/IQR resistant to extremes
Log transform	log(x + 1)	Right-skewed data; multiplicative relationships

data-wrangling

설치 방법

소스 파일 검토

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Cleaning Techniques

Type Coercion

Deduplication

Outlier Detection

Missing Data

Missing Data Mechanisms

Handling Strategies

Joins and Integration

Join Types

Join Hazards

Transformation Techniques

Normalization and Scaling

Binning and Discretization

Feature Engineering

Reproducibility

Common Mistakes

Cross-References

References

Mistake	Why it fails	Fix
Imputing before exploring	You do not know the missingness mechanism	Profile first, then choose strategy
Joining without checking cardinality	Many-to-many join silently explodes row count	Check key uniqueness before joining
Dropping outliers without investigation	Outliers may be the most important observations	Investigate, then decide
Mixing cleaning and analysis	Impossible to reproduce or audit	Separate wrangling script from analysis
Ignoring encoding issues	Garbled text, silent data loss	Specify encoding explicitly at ingestion
Normalizing before train/test split	Data leakage -- test set statistics influence training	Fit scaler on training data only

data-wrangling

소스 정보

설치 방법

소스 파일 검토

이 저장소의 다른 Skills

이 저장소의 다른 Skills

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Data Wrangling

The Wrangling Pipeline

Tidy Data Principles

Cleaning Techniques

Type Coercion

Deduplication

Outlier Detection

Missing Data

Missing Data Mechanisms

Handling Strategies

Joins and Integration

Join Types

Join Hazards

Transformation Techniques

Normalization and Scaling

Binning and Discretization

Feature Engineering

Reproducibility

Common Mistakes

Cross-References

References