hypothesis-testing

name	hypothesis-testing
description	Rigorous hypothesis testing process for data analysis - formulate hypotheses before looking at data, design tests, analyze systematically, interpret with skepticism

Hypothesis Testing Process

Overview

This skill guides you through rigorous hypothesis testing in data analysis. The core principle is scientific rigor: formulate your hypotheses BEFORE looking at the data to avoid p-hacking and confirmation bias.

Hypothesis testing is appropriate when:

• You have a specific claim or belief to test
• You want to avoid confirmation bias
• The question can be framed as "Does X affect Y?"
• You need defensible, reproducible conclusions

Prerequisites

Before using this skill, you MUST:

1. Have data imported into SQLite database using the importing-data skill
2. Have data quality validated and cleaned using the cleaning-data skill (MANDATORY - never skip)
3. Have created an analysis workspace (just start-analysis hypothesis-testing <name>)
4. Understand the basic structure of your data tables
5. Be familiar with the component skills:
   • understanding-data - for data profiling
   • writing-queries - for SQL query construction
   • interpreting-results - for result analysis
   • creating-visualizations - for text-based visualizations

Mandatory Process Structure

You MUST use TodoWrite to track progress through all 5 phases. Create todos at the start:

- Phase 1: Hypothesis Formulation (H0/H1) - pending
- Phase 2: Test Design - pending
- Phase 3: Data Analysis - pending
- Phase 4: Result Interpretation - pending
- Phase 5: Conclusion and Follow-up - pending

Update status as you progress. Mark phases complete ONLY after checkpoint verification.

---

Phase 1: Hypothesis Formulation

CHECKPOINT: Before proceeding, you MUST have:

• Written null hypothesis (H0) - specific, testable statement
• Written alternative hypothesis (H1) - specific, directional or non-directional
• Documented WHY you're testing this (user goal, business context)
• Saved to 01 - hypothesis-formulation.md

Instructions

1. Ask clarifying questions about the user's analytical goal

   • What belief or claim needs testing?
   • What would constitute evidence for/against this belief?
   • What's the practical significance threshold?

2. Write hypotheses in 01 - hypothesis-formulation.md with: ./templates/phase-1.md

3. STOP and get user confirmation

   • Read the hypotheses back to the user
   • Confirm this is what they want to test
   • Do NOT proceed to Phase 2 until confirmed

Common Rationalization: "I'll just peek at the data to refine the hypothesis" Reality: This creates confirmation bias. Formulate hypothesis FIRST, always.

---

Phase 2: Test Design

CHECKPOINT: Before proceeding, you MUST have:

• Defined exact metrics to calculate
• Specified comparison method (segments, time periods, etc.)
• Listed required data checks (missing values, outliers, sample sizes)
• Written test plan WITHOUT executing any queries yet
• Saved to 02 - test-design.md

Instructions

1. Design your test in 02 - test-design.md with: ./templates/phase-2.md

2. STOP and verify test design

   • Does this test actually answer the hypothesis?
   • Are there simpler/better ways to test it?
   • Have you planned for data quality issues?
   • Get user confirmation before proceeding

Common Rationalization: "I'll just run one quick query to see if the data exists" Reality: That's looking at data before test design is complete. Finish design FIRST.

---

Phase 3: Data Analysis

CHECKPOINT: Before proceeding, you MUST have:

• Executed schema/data quality checks
• Documented any data issues found
• Executed main analysis query
• Executed supporting analysis queries
• Saved all queries and results to numbered files (03-, 04-, etc.)

Instructions

1. Execute queries in order, one file per query

Create separate numbered files:

• 03 - schema-check.md
• 04 - data-quality-check.md
• 05 - main-analysis.md
• 06 - supporting-analysis.md (if needed)

2. For each query file, use this structure: ./templates/phase-3-query.md

3. Execute queries using appropriate tool

   • Use SQLite CLI or database tool
   • Copy EXACT results (don't summarize or round)
   • If query fails, document error and revise

4. Handle data quality issues

   • If quality checks reveal problems, document them
   • Decide: exclude bad data, transform it, or note as limitation
   • Update test design if needed (document why)

Common Rationalization: "I'll skip data quality checks since the data looks fine" Reality: ALWAYS check data quality. Surprises happen. Document what you checked.

Common Rationalization: "I'll combine all queries into one file to save time" Reality: Separate files create clear audit trail. One query, one file.

---

Phase 4: Result Interpretation

CHECKPOINT: Before proceeding, you MUST have:

• Described what the data shows (facts only)
• Calculated relevant comparisons (% differences, ratios)
• Considered alternative explanations
• Assessed confounding factors identified in Phase 1
• Stated whether results support/reject H0
• Saved to 07 - interpretation.md (or next number)

Instructions

1. Create interpretation file: XX - interpretation.md with: ./templates/phase-4.md

2. Be intellectually honest

   • State limitations clearly
   • Don't overstate conclusions
   • Acknowledge uncertainty
   • Suggest what additional data would help

Common Rationalization: "The pattern is obvious, I don't need to consider alternatives" Reality: Always consider alternatives. Obvious patterns often have surprising explanations.

Common Rationalization: "I'll downplay limitations so the conclusion looks stronger" Reality: Stating limitations INCREASES credibility. Be honest about uncertainty.

---

Phase 5: Conclusion and Follow-up

CHECKPOINT: Before proceeding, you MUST have:

• Written clear, actionable conclusion
• Listed 2-3 specific follow-up questions
• Suggested concrete next steps
• Saved to 08 - conclusion.md (or next number)
• Updated 00 - overview.md with summary

Instructions

1. Create conclusion file: XX - conclusion.md with: ./templates/phase-5.md

2. Update overview file: 00 - overview.md

Add summary section with: ./templates/overview-summary.md

3. Final checklist before marking complete
   • All phases documented in numbered files
   • Queries and results included
   • Limitations acknowledged
   • Follow-up questions specified
   • Overview updated
   • User informed of conclusions

Common Rationalization: "I found the answer, I'm done" Reality: Good analysis always identifies the NEXT question. List follow-ups.

Common Rationalization: "I'll skip updating the overview since it's all in the detailed files" Reality: Overview provides navigational summary. Always update it.

---

Complete Example: Day-of-Week Analysis

Example Scenario

User wants to test: "Do we get more sales on weekends?"

Phase 1: Hypothesis Formulation (01 - hypothesis-formulation.md)

# Hypothesis Formulation

## Analytical Goal
Determine if weekend days (Saturday, Sunday) have higher sales than weekday days (Monday-Friday)

## Context
Business wants to optimize staffing and inventory. If weekends are significantly busier, we should staff up. If not, we can balance resources across the week.

## Hypotheses

### Null Hypothesis (H0)
Weekend days and weekday days have equal average sales. Any observed differences are due to random variation.

### Alternative Hypothesis (H1)
Weekend days have significantly different average sales compared to weekday days.

## Success Criteria
- Difference >25% would be practically meaningful (enough to justify staffing changes)
- Pattern should be consistent across multiple weeks

## Potential Confounds
- Holidays: Holiday Monday might inflate weekday averages
- Promotions: Weekend promotions might inflate weekend sales
- Store hours: Different hours on weekends might affect opportunity
- Seasonality: Analysis period might not be representative of full year

Phase 2: Test Design (02 - test-design.md)

# Test Design

## Metrics

### Primary Metric
Average daily sales amount: SUM(amount) / COUNT(DISTINCT date) for weekend vs weekday groups

### Supporting Metrics
- Total transaction count per day-of-week (to assess sample size)
- Median sales per transaction (to check if average is representative)
- Week-over-week consistency (to see if pattern is stable)

## Comparison Structure
Group days into two categories:
- Weekend: Saturday (day 6), Sunday (day 0)
- Weekday: Monday-Friday (days 1-5)

Calculate average daily sales for each group, compare the ratio.

## Data Requirements

### Required Tables/Columns
- Table: `sales` (or similar)
  - `transaction_date` (DATE or TEXT in ISO format)
  - `amount` (NUMERIC)

### Data Quality Checks
1. Check for NULL dates or amounts
2. Verify date range covers complete weeks
3. Confirm adequate sample size (>100 transactions per day-of-week)
4. Identify any outlier days (holidays, system outages)

### Queries Needed
1. Schema check: PRAGMA table_info(sales)
2. Data quality: NULL checks, date range, outlier detection
3. Main analysis: Daily sales by day-of-week
4. Supporting: Transaction counts, weekly pattern consistency

## Statistical Considerations
Look for differences >25% between weekend and weekday averages. Check if pattern is consistent across multiple weeks (not just one unusual weekend).

Phase 3: Data Analysis (03-06 files)

03 - schema-check.md:

# Schema Check

## Rationale
Verify the sales table exists and has the required columns for date and amount

## Query
```sql
PRAGMA table_info(sales);

Results

cid | name              | type    | notnull | dflt_value | pk
0   | transaction_id    | INTEGER | 0       | NULL       | 1
1   | transaction_date  | TEXT    | 0       | NULL       | 0
2   | amount            | REAL    | 0       | NULL       | 0
3   | product_id        | INTEGER | 0       | NULL       | 0

Initial Observations

• Table sales exists
• Has transaction_date column (TEXT type - will need STRFTIME to extract day-of-week)
• Has amount column (REAL type - good for calculations)
• Columns allow NULLs - need to check data quality

**04 - data-quality-check.md:**
```markdown
# Data Quality Check

## Rationale
Verify data completeness and identify any issues before main analysis

## Query
```sql
SELECT
  COUNT(*) as total_rows,
  COUNT(CASE WHEN transaction_date IS NULL THEN 1 END) as null_dates,
  COUNT(CASE WHEN amount IS NULL THEN 1 END) as null_amounts,
  MIN(transaction_date) as earliest_date,
  MAX(transaction_date) as latest_date,
  MIN(amount) as min_amount,
  MAX(amount) as max_amount
FROM sales;

Results

total_rows | null_dates | null_amounts | earliest_date | latest_date | min_amount | max_amount
15847      | 0          | 0            | 2024-01-01    | 2024-03-31  | 5.00       | 899.99

Initial Observations

• 15,847 total transactions
• No NULL dates or amounts (clean data)
• 3 months of data (Jan-Mar 2024)
• 13 complete weeks (91 days / 7)
• Amount range: $5 to $899.99 (no obvious outliers)

**05 - main-analysis.md:**
```markdown
# Main Analysis: Sales by Day of Week

## Rationale
Calculate average sales by day of week to test if weekend days differ from weekdays

## Query
```sql
SELECT
  CAST(STRFTIME('%w', transaction_date) AS INTEGER) as day_of_week,
  CASE
    WHEN CAST(STRFTIME('%w', transaction_date) AS INTEGER) IN (0, 6) THEN 'Weekend'
    ELSE 'Weekday'
  END as day_type,
  COUNT(DISTINCT transaction_date) as days_count,
  COUNT(*) as transaction_count,
  SUM(amount) as total_sales,
  ROUND(SUM(amount) / COUNT(DISTINCT transaction_date), 2) as avg_daily_sales,
  ROUND(AVG(amount), 2) as avg_transaction_amount
FROM sales
GROUP BY day_of_week, day_type
ORDER BY day_of_week;

Results

day_of_week | day_type | days_count | transaction_count | total_sales | avg_daily_sales | avg_transaction_amount
0           | Weekend  | 13         | 1456              | 52389.44    | 4030.00        | 35.98
1           | Weekday  | 13         | 2234              | 102345.67   | 7872.90        | 45.81
2           | Weekday  | 13         | 2198              | 98234.55    | 7556.50        | 44.70
3           | Weekday  | 13         | 2301              | 105678.90   | 8129.15        | 45.93
4           | Weekday  | 13         | 2345              | 108901.23   | 8377.02        | 46.44
5           | Weekday  | 13         | 2456              | 115432.11   | 8879.39        | 47.00
6           | Weekend  | 13         | 2857              | 98765.43    | 7597.34        | 34.56

Initial Observations

• Sunday (0): $4,030 avg daily sales, 1,456 transactions, $35.98 avg transaction
• Monday-Friday (1-5): $7,800-$8,800 avg daily sales, 2,200-2,500 transactions, $44-47 avg transaction
• Saturday (6): $7,597 avg daily sales, 2,857 transactions, $34.56 avg transaction
• Saturday has HIGH transaction count but LOW per-transaction amount
• Sunday has both low transaction count AND low per-transaction amount

**06 - supporting-analysis.md:**
```markdown
# Supporting Analysis: Weekend vs Weekday Comparison

## Rationale
Aggregate weekend and weekday categories to test the hypothesis directly

## Query
```sql
SELECT
  CASE
    WHEN CAST(STRFTIME('%w', transaction_date) AS INTEGER) IN (0, 6) THEN 'Weekend'
    ELSE 'Weekday'
  END as day_type,
  COUNT(DISTINCT transaction_date) as days_count,
  COUNT(*) as transaction_count,
  SUM(amount) as total_sales,
  ROUND(SUM(amount) / COUNT(DISTINCT transaction_date), 2) as avg_daily_sales,
  ROUND(AVG(amount), 2) as avg_transaction_amount
FROM sales
GROUP BY day_type;

Results

day_type | days_count | transaction_count | total_sales | avg_daily_sales | avg_transaction_amount
Weekday  | 65         | 11534             | 530592.46   | 8163.00         | 46.01
Weekend  | 26         | 4313              | 151154.87   | 5813.65         | 35.04

Initial Observations

• Weekday average: $8,163 per day
• Weekend average: $5,814 per day
• Difference: $2,349 (29% lower on weekends)
• Weekend has 62% fewer transactions per day
• Weekend transactions are 24% smaller on average

### Phase 4: Interpretation (07 - interpretation.md)

```markdown
# Result Interpretation

## Summary of Findings

### Primary Metric Results
Weekends have significantly LOWER sales than weekdays:
- Weekday avg: $8,163 per day
- Weekend avg: $5,814 per day
- Difference: 29% lower on weekends

This contradicts the original hypothesis that weekends would have higher sales.

### Statistical Assessment
- Magnitude: 29% difference is both statistically meaningful (large sample) and practically significant (>25% threshold)
- Consistency: Pattern holds across 13 weeks of data
- Practical importance: This magnitude justifies different staffing/inventory approaches

## Alternative Explanations

1. **Store hours:** Maybe stores close earlier on weekends
   - Evidence: Transaction count is 62% lower on weekends, suggesting operational constraint
   - Impact: Could fully explain lower daily totals if hours are reduced

2. **Customer behavior:** Maybe weekend shoppers buy smaller items
   - Evidence: Average transaction is 24% smaller on weekends ($35 vs $46)
   - Impact: Explains some but not all of the difference

3. **Product mix:** Maybe high-value products aren't available on weekends
   - Evidence: Would need product category data to verify
   - Impact: Unknown

4. **Sunday effect:** Maybe Sunday drags down weekend average
   - Evidence: Sunday has only $4,030 avg (47% lower than Saturday's $7,597)
   - Impact: If we exclude Sunday, Saturday is closer to weekdays but still 7% lower

## Hypothesis Test Result

### Null Hypothesis (H0)
Weekend days and weekday days have equal average sales.

### Decision
**REJECT H0**

### Rationale
The 29% difference between weekend and weekday average sales is large, consistent across 13 weeks, and exceeds our 25% practical significance threshold. The pattern is clear: weekends have lower sales than weekdays.

HOWEVER: This is the OPPOSITE of what we hypothesized. The alternative hypothesis stated "weekend days have significantly different sales" which is true, but we expected higher, not lower.

## Limitations

1. **Cannot determine causation:** Data shows weekends are lower but doesn't explain why
2. **Store operational factors unknown:** Don't have data on hours, staffing, inventory availability
3. **Sunday is dramatically lower:** Weekend average is heavily influenced by very low Sunday sales
4. **Seasonal effects:** 3 months (Jan-Mar) may not represent full year (could be post-holiday slump)
5. **No control for promotions/holidays:** Any holidays in the period could skew results

Phase 5: Conclusion (08 - conclusion.md)

# Conclusion and Follow-up

## Main Conclusion

Weekends have 29% lower average daily sales than weekdays, contradicting the initial hypothesis that weekends would be busier. This pattern is consistent across 13 weeks and appears driven by both fewer transactions (62% lower) and smaller average purchases (24% lower).

## Actionable Insights

**DO NOT increase weekend staffing based on this data.** The hypothesis that weekends are busier is not supported.

Instead:

1. **Investigate Sunday specifically:** With only $4,030 avg daily sales (vs $7,600 on Saturday), Sunday operations may not be profitable. Consider:
   - Reduced hours on Sunday
   - Sunday-specific promotions to drive traffic
   - Or closing on Sundays if fixed costs exceed contribution margin

2. **Understand operational constraints:** Before making staffing decisions, determine:
   - Are weekend hours reduced? (This might explain lower transaction counts)
   - Is weekend inventory limited? (This might explain smaller purchases)
   - Are certain high-value products unavailable on weekends?

3. **Focus weekday resources:** If pattern holds year-round, optimize for Monday-Friday peak demand

## Follow-up Questions

1. **What are actual store operating hours by day?**
   - Data needed: Store hours table, or time-of-day in transaction timestamps
   - Approach: Calculate sales per operating hour rather than per day
   - Why: Would distinguish "fewer hours" from "fewer customers per hour"

2. **Does the weekend pattern hold across all seasons?**
   - Data needed: Full year of transaction data
   - Approach: Repeat this analysis for each quarter
   - Why: Jan-Mar might be post-holiday slump; need to verify pattern is year-round

3. **Are there product category differences on weekends?**
   - Data needed: Product category or department in transaction records
   - Approach: Analyze category mix and average prices by day-of-week
   - Why: Would explain the 24% smaller average transaction on weekends

4. **How do weekends compare if we exclude Sunday?**
   - Data needed: Current dataset (already have this)
   - Approach: Re-run analysis treating Sunday separately
   - Why: Sunday is so dramatically lower it may be distorting the weekend average

## Confidence Level

**Medium confidence** in the finding that weekends are lower, but **low confidence** in the explanation.

Reasoning:
- Pattern is clear and consistent across 13 weeks (strengthens confidence)
- Sample sizes are adequate (4,313 weekend transactions is plenty)
- BUT: Cannot distinguish operational constraints from customer behavior (weakens confidence)
- AND: Only 3 months of data may not represent full year (weakens confidence)

**Next step:** Investigate store hours and expand to full year data before making operational changes.

---

Common Rationalizations

"I'll just look at the data first to understand it better"

Why this is wrong: Looking at data before formulating hypotheses creates confirmation bias. You'll unconsciously form hypotheses that match what you see, then "test" them. This isn't science.

Do instead: Formulate hypothesis from domain knowledge, business context, or theory. Then look at data.

"The hypothesis is obvious from the results, I don't need to write it down"

Why this is wrong: Hindsight bias makes everything seem obvious after you see the answer. Writing hypothesis first creates accountability.

Do instead: Always write H0 and H1 in Phase 1 before any query execution.

"I'll skip data quality checks since the data looks clean"

Why this is wrong: You can't know data is clean until you check. Surprises happen often.

Do instead: ALWAYS run data quality queries. Document what you checked and what you found (even if it's "no issues").

"The pattern is clear, I don't need to consider alternative explanations"

Why this is wrong: Obvious patterns often have non-obvious causes. Confounding factors are common.

Do instead: Always list 2-3 alternative explanations in Phase 4, even if you think they're unlikely.

"I'll combine multiple queries into one file for efficiency"

Why this is wrong: One file per query creates clear audit trail and makes analysis reproducible.

Do instead: One query, one file. Use consistent numbering (03-, 04-, etc.).

"I found the answer, analysis is complete"

Why this is wrong: Good analysis always identifies the next question. Every conclusion should raise new questions.

Do instead: Always list 2-3 follow-up questions in Phase 5.

"I'll downplay limitations so the conclusion looks stronger"

Why this is wrong: Acknowledging limitations increases credibility. Readers trust honest uncertainty more than false certainty.

Do instead: State limitations clearly. Be honest about what you don't know.

"I'll skip updating the overview since everything is in detailed files"

Why this is wrong: Overview provides navigation and quick summary. Future readers (including you) will thank you.

Do instead: Always update 00 - overview.md with results summary in Phase 5.

---

Summary

This skill ensures rigorous, reproducible hypothesis testing by:

1. Preventing confirmation bias: Formulate hypothesis BEFORE looking at data
2. Ensuring thoughtful design: Plan your test before executing queries
3. Creating audit trail: One query per file, with rationale and results
4. Demanding intellectual honesty: Consider alternatives, state limitations
5. Identifying next questions: Every analysis should suggest follow-up investigations

Follow this process and you'll produce defensible, credible analysis that stands up to scrutiny.