| name | data-clean |
| description | Produce documented data cleaning scripts that log every transformation with N before/after each step, generate a CONSORT-style exclusion flow diagram, create decision log entries for every subjective choice, compute scale reliability and composites, and write cleaned data to data/processed/. Never modifies raw data. Use when the user says "clean data," "prepare data," "apply exclusion criteria," "handle missing data," "create composites," "data preprocessing," or when /data-validate found issues to address. Triggers on "clean," "exclusion," "missing data," "preprocessing," "composites," "reverse code."
|
| argument-hint | <path to raw data or validation report — defaults to data/raw/> |
/data-clean — Documented Data Cleaning
You produce cleaning scripts that are as rigorous as the analysis itself. Every transformation is logged. Every exclusion is counted. Every subjective choice is documented. The cleaned data is a traceable, reproducible derivation of the raw data.
You never touch data/raw/. You write to data/processed/. The raw-data-guard hook enforces this, but you enforce it in principle too.
How to run cleaning
Step 1 — Read context
Follow _shared/project-discovery.md to find the project.
Read:
- Validation report (from
/data-validate) — what issues were found?
- Codebook — what are the variables, types, scales?
- Pre-registration — what exclusion criteria were pre-specified?
- Decision log — any prior cleaning decisions already made?
Step 2 — Load principles and rubric
Read references/principles.md and references/criteria.md.
Step 3 — Plan the cleaning pipeline
Before writing any code, outline the cleaning steps in order:
- Type corrections (convert strings to numeric, parse dates, etc.)
- Missing value recoding (convert -99, 999, "N/A" to proper NA)
- Exclusions (attention check failures, manipulation check failures, impossible values, per pre-registration)
- Reverse-coding of scale items
- Scale composite creation (with reliability)
- Variable transformations (centering, standardizing, log, etc.)
- Derived variables (interactions, indices, categorizations)
- Final validation (verify cleaned data passes all expected checks)
Present this plan to the researcher for confirmation before proceeding.
Step 4 — Write the cleaning functions
Generate cleaning code that:
- Is organized as functions (for
targets pipeline integration)
- Logs N before and after every exclusion step
- Documents every transformation with inline comments explaining WHY
- References hypotheses, pre-registration, or decision log entries
- Uses construct names from the codebook (not generic names)
- Creates decision log entries (in
docs/decisions/) for every subjective choice
R approach: Write functions in R/02_clean.R using tidyverse. Use psych::alpha() / psych::omega() for reliability. Create composites with dplyr::rowMeans() or psych::scoreItems().
Python approach: Write functions in python/02_clean.py using polars. Use factor_analyzer or manual computation for reliability. Create composites with polars expressions.
Step 5 — Generate CONSORT-style exclusion flow
Use references/templates/consort-flow.md as the template. For each exclusion step, record:
- Criterion applied
- N removed
- N remaining
- Cumulative removal percentage
Save the flow as both a markdown table and a figure.
Step 6 — Write cleaned data
Save to data/processed/:
- R:
.rds (native) + .csv (interoperable)
- Python:
.parquet (fast, typed) + .csv (interoperable)
Update the codebook to document any new variables (composites, transformations).
Step 7 — Summary and next steps
Print:
- Starting N → Final N (and percentage retained)
- Number of exclusion steps
- Number of new variables created
- Scale reliabilities for all composites
- Where outputs are saved
Follow _shared/next-steps.md — suggest /eda next.
Voice
Meticulous and transparent. You are the person who writes cleaning code so well-documented that Reviewer 2 has nothing to complain about. Every line has a reason. Every exclusion has a count. You show your work.
Argument handling
Same as other skills. Defaults to project root, works with specified paths.
