| name | Data Analysis |
| description | Analyze datasets by profiling, finding distributions and correlations, detecting anomalies, and translating statistics into actionable business insights. |
Purpose
Turn raw data into actionable insight statements. Statistics are not insights. "Mean revenue is $4.2M" is a statistic. "Revenue grew 34% YoY but growth is decelerating" is an insight.
Steps
- Profile the dataset: row count, column count, null percentages, type inference, duplicates
- Compute descriptive statistics for every numeric column: mean, median, standard deviation, min/max, percentiles, skewness
- Analyze distributions: identify normal, skewed, bimodal, or power-law shapes
- Check correlations: flag strong (|r| > 0.7) and moderate (0.4-0.7) relationships
- Segment by key categorical variables: compare group means, check for Simpson's Paradox
- Detect anomalies: use IQR fences for skewed data, z-scores for normal data. Flag but never auto-remove.
- Translate every finding into an insight statement using the DIKW framework
- Create an Excel workbook with a summary sheet and supporting data sheets
- Create a Word document with the top 5 insights, each following the insight template
Data Profiling Checklist
| Metric | Red Flag |
|---|
| Null percentage | > 20% in key column = unreliable |
| Unique count = row count | Likely an ID column |
| Mixed types in column | Needs cleaning before analysis |
| > 1% duplicate rows | Dedup decision required |
| Mean >> median | Right-skewed, report median as "typical" |
Insight Statement Template
For each finding, write:
- WHAT: [metric] is [value/behavior]
- SO WHAT: This means [business implication]
- NOW WHAT: Consider [action or follow-up question]
Example:
- WHAT: California's population (39M) is 5x the median state (7.5M)
- SO WHAT: Resource allocation using state averages will dramatically under-serve CA
- NOW WHAT: Segment by population tier, not just state count
Correlation Rules
| Strength | Range | Action |
|---|
| Strong | 0.7-1.0 | Investigate causation, check for confounders |
| Moderate | 0.4-0.7 | Worth reporting, segment and re-check |
| Weak | 0.1-0.4 | Mention only if business-relevant |
| None | < 0.1 | Do not report |
Always check: does the correlation reverse when split by a categorical variable? (Simpson's Paradox)
Time-Series Decomposition
Separate trend, seasonality, and residual. Use rolling averages:
- Daily data: 7 or 30 day window
- Weekly: 4 or 13 week window
- Monthly: 3 or 12 month window
Output Format
- Excel workbook: Summary sheet with top insights + data sheets with full analysis
- Word document: Top 5 insight statements with supporting charts
- Each insight includes: what, so what, now what
Guidelines
- Always add "so what?" to every statistic before reporting it
- Report both mean and median; the gap itself is an insight
- Never claim "X causes Y" from correlation alone; say "correlates with"
- Show distributions, not just averages (averages hide variation)
- Complete the full EDA before recommending action
- Flag data quality issues at the start, not buried in footnotes
