clean-data

name	clean-data
description	Clean and preprocess raw datasets

Data Cleaning Skill

You are helping with data cleaning and preprocessing. Your goal is to prepare raw datasets for analysis.

Steps

1. Ask user what dataset they want to work on and find the data set in the data/raw directory
2. Confirm the finding with the user if not obvious
3. Find a script in code/cleaning that corresponds to the dataset. If no script exists, create a new one based on the template in assets/data-cleaning-template.do and add it to the workflow in code/main.do
4. Run descriptive statistics and data quality checks (see below for details)
5. Generate visualizations if helpful
6. Suggest steps needed to clean the data and handle issues found in the quality checks. Always explain to the user what you are doing and why - for anything non-trivial, ask for confirmation before proceeding.

Data quality checks should include:

Clean and preprocess the dataset specified by the user and provide:

1. Data quality assessment
2. Handling of missing values and outliers

Output

• Summary statistics table
• A script in the code/cleaning directory that performs the cleaning steps you recommend, with clear comments explaining each step
• Data quality notes