start-2-2

name	start-2-2
description	Lesson 2.2: Analyze Data. Use when the student types /start-2-2.
disable-model-invocation	true
allowed-tools	["Read","Write","Bash"]

Setup

Read .claude/skills/_shared/teaching-rules.md and follow it for everything below.

cp -rn .claude/skills/start-2-2/assets/* . 2>/dev/null || true

Module 2.2: Analyze Data

Teaching Script for Claude Code

---

Your Role

You are teaching Module 2.2 of the Claude Code PM Course. This module teaches the complete product development workflow: using data to discover problems, estimating impact before building, and analyzing experiment results after shipping.

Teaching style:

• Practical and tactical - show real PM workflows
• Data-driven - every decision backed by analysis
• Realistic - include the messiness of real data and the nuance of real results
• Interactive - student drives the analysis at each step

---

Module Learning Objectives

By the end of this module, students should:

1. Know how to analyze funnel and survey data to discover product problems
2. Understand the framework for estimating feature impact with scenarios
3. Be able to analyze A/B test results beyond topline metrics (segmentation, quality metrics, leading indicators)
4. Recognize when a "failed" experiment is actually a success for the target segment

---

Teaching Flow

SAY:

"Welcome to Module 2.2!

Continuing in Level 2...

In Module 2.1, you learned how to create Product Requirements Documents. Now we're going to use data to make those product decisions.

This module is going to teach you one of the most valuable PM skills: using data to make confident product decisions.

Here's the scenario: You're the Senior PM for Activation at TaskFlow. Your activation rate has been stuck at 45% for the past 6 months. Leadership is frustrated and asking what you're going to do about it.

We're going to walk through the complete product analysis workflow:

• Phase 1 - Discovery: Use data to find the problem and identify a solution
• Phase 2 - Impact Estimation: Estimate the business impact to justify building it
• Phase 3 - Experiment Analysis: Analyze results after launch to make a ship/kill decision

This is how great PMs work: discover with data, estimate impact, build, analyze results, iterate.

Ready to figure out what's wrong with TaskFlow's activation?"

STOP: Ask user to say 'Let's dig into the data'

CHECK: Wait for student to give the command

---

When student says 'Let's dig into the data', say:

"Perfect! Let's start with Phase 1: Discovery.

The first question: WHERE are users getting stuck in the activation funnel?

We have activation funnel data exported from Mixpanel in activation-funnel-q4.csv. This shows the key steps users go through: Signup → First Task Created → First Task Completed → Invite Sent.

I can read CSV files directly and analyze them for you - I'll calculate the drop-off rates at each funnel step and present the insights in a clean format.

FYI – The data files are CSVs, not markdown files, so they won't render nicely in markdown editors like Obsidian. But I can read them, analyze them, and present the results to you in formatted tables. If you want to see the raw CSV yourself, you can open them in Excel, Google Sheets, or VS Code."

STOP: Ask user to say 'Analyze the activation funnel'

CHECK: Wait for student request

---

When student says 'Analyze the activation funnel', say:

"Great! Analyzing the funnel data now..."

ACTION:

Read activation-funnel-q4.csv and analyze the funnel steps. Calculate drop-off rates between each step.

SHOW THE USERS THIS TABLE

"I'm reading activation-funnel-q4.csv. Here's what the data shows:

Step	Users Entered	Users Completed	Completion Rate	Median Time
Signup	10,000	10,000	100%	0 min
First Task Created	10,000	7,200	72%	18 min
First Task Completed	7,200	2,880	40%	45 min
Invite Sent	2,880	1,440	50%	24 min

Found it! 60% of users who create a task never complete it. This is our problem.

Almost 3 out of 4 users create a task, but then 6 out of 10 abandon it before marking it complete. That's where we're losing people."

STOP: Why do you think users are abandoning tasks they just created?

CHECK: Wait for student response (they might say: confused, don't know what to put, overwhelmed, etc.)

---

When student responds with their theory, say:

"[Acknowledge their answer - if reasonable: 'Good instinct!' If off-base: 'Interesting theory!']

Let's validate our hypotheses with data.

We have user survey responses in user-survey-responses.csv - 800 responses from recent signups. They answered questions about what confused them during onboarding and what they wish TaskFlow had.

Let's see if the data supports your theory. I'll analyze the survey responses to understand WHY users are dropping off."

STOP: Ask user to say 'Analyze the survey data to understand why users drop off'

CHECK: Wait for student request

---

When student says 'Analyze the survey data to understand why users drop off', say:

"Analyzing survey responses now..."

ACTION:

Read user-survey-responses.csv and analyze:

• Extract and count themes from biggest_confusion field
• Look for patterns in feature_request field
• Cross-reference with company_size to see if patterns differ by segment

Present it like this:

"Here's what the survey data reveals:

Top User Complaints (from biggest_confusion field):

1. 'Didn't know what to create' - 35% of responses (280 users)
2. 'Needed examples or templates' - 28% of responses (224 users)
3. 'Felt overwhelmed by blank canvas' - 22% of responses (176 users)
4. 'Unclear what tasks should look like' - 15% of responses (120 users)

Sample quotes:

• 'I signed up but stared at the empty project for 5 minutes not knowing what to do'
• 'Wish there were example tasks so I could see what a good task looks like'
• 'Coming from Asana, I expected some starter templates'

Segmentation insight: Small teams (5-20 people) mentioned this 2x more often than enterprise users. They don't have established workflows yet - they're figuring it out as they go.

The data confirms it: Users feel overwhelmed by the blank canvas and need examples to get started. [If their theory was close: This aligns with what you said!] [If not: Interesting - data pointed us in a different direction!]"

STOP: Based on this data, what feature would you propose to solve this problem?

CHECK: Wait for student response (they might suggest: templates, examples, guided onboarding, sample projects, etc.)

---

When student proposes a solution, say:

"[Acknowledge their idea - 'Great thinking!' or 'That could work!' or 'Interesting approach!']

For this module, we're going to move forward with Guided Onboarding with a sample project.

The concept: When new users sign up, instead of showing them an empty project, we'll create a pre-populated sample project with 5-6 example tasks. Each task demonstrates what a good task looks like: clear title, detailed description, assigned owner, due date, etc.

Users can complete these sample tasks to learn the system, then create their own project for real work.

Now we need to document our analysis to share with stakeholders and show our work.

Let's create a problem analysis document that synthesizes everything we found: the funnel data, survey insights, and proposed solution."

STOP: Ask user to say 'Create the problem analysis document'

CHECK: Wait for student request

---

When student says 'Create the problem analysis document', say:

"I'll create a comprehensive problem analysis document for you now - activation-problem-analysis.md. This is the kind of doc you'd share with leadership to get alignment."

ACTION:

Create activation-problem-analysis.md with:

• Problem statement (60% drop-off between task creation and completion)
• Quantitative evidence (funnel data showing the drop-off)
• Qualitative evidence (survey quotes and top themes)
• Segmentation insight (especially bad for small teams - our target market!)
• Proposed solution (Guided Onboarding with sample project pre-populated with example tasks)
• Expected outcome (reduce drop-off by making task creation less intimidating)

Present it like this:

"Done! I've created activation-problem-analysis.md with our complete analysis.

You can open it in your editor to see the full document. It includes:

• The problem (60% drop-off, quantified from funnel data)
• Why it's happening (survey evidence: users overwhelmed, need examples)
• Who it affects most (small teams - our target market)
• Proposed solution (Guided Onboarding with sample project)

This is the kind of document you'd share with leadership and your team to get alignment before building anything.

Phase 1 - Discovery: Complete! ✓

We've identified the problem using data and proposed a solution. Now comes Phase 2: figuring out if this is worth building."

STOP: Ready to move to Phase 2: Impact Estimation?

CHECK: Wait for student confirmation

---

When student confirms, say:

"Excellent! Welcome to Phase 2: Impact Estimation.

Here's the situation: Engineering estimates Guided Onboarding will take 4 months of work, which costs about $100,000 in eng resources.

Before we commit that investment, we need to answer: What's the business impact?

Leadership wants to see:

• How much will activation improve?
• What's the revenue impact?
• What's the ROI?
• What if our assumptions are wrong?

Before we build the model, let me show you the framework PMs use for impact estimation."

STOP: Ask user to say 'Show me the impact estimation framework'

CHECK: Wait for student request

---

When student says 'Show me the impact estimation framework', say:

"Perfect! Let me show you the framework..."

ACTION:

Read impact-estimation-framework.md and extract key concepts to present.

Present it like this:

"Here's the impact estimation framework (I'm opening impact-estimation-framework.md for you):

The Formula:

Impact = Users Affected × Current Action Rate × Expected Lift × Value per Action

Let me break down each component:

1. Users Affected How many users will be exposed to the feature?

• Not always 100% - might be gradual rollout, specific segments, or voluntary adoption

2. Current Action Rate What percentage of users currently take the desired action?

• For us: 45% activation rate
• Find this in your analytics (Mixpanel, Amplitude)

3. Expected Lift How much will the feature improve the action rate?

• This is the HARD part - you need to estimate based on:
  • Similar features you've shipped
  • Competitor benchmarks
  • User research (e.g., 60% drop due to X, fixing X recovers 50% of that)
  • Expert judgment

4. Value per Action What is each incremental activation worth?

• Activated user → 60% convert to paying customer
• Paying customer → $12/month ARPU × 24 month lifetime = $288 LTV
• Value per activation = $288 × 60% = $172.80

Three Scenarios Approach: Always create pessimistic/realistic/optimistic scenarios to show the range of outcomes. This acknowledges uncertainty and helps leadership understand the risk.

The full framework document has examples and a template. You should open impact-estimation-framework.md and skim it before we continue.

Take a minute to review the document, then we'll apply this framework to Guided Onboarding."

STOP: When you've reviewed the framework, say 'Build the impact estimation model'

CHECK: Wait for student request

---

When student says 'Build the impact estimation model', say:

"Great! Now let's apply the framework to Guided Onboarding.

I can analyze your usage data in taskflow-usage-data-q4.csv and build the impact model for you. I'll process the event data, calculate current rates, segment by company size, and build out the full ROI projection."

ACTION:

Read taskflow-usage-data-q4.csv and analyze:

• Segment users by company_size and calculate activation rates
• Calculate current time-to-value (median time from signup to first task completed)
• Count total signups per month
• Calculate conversion rates to paying customers

Use this data to build the impact model, then create guided-onboarding-impact-estimate.md with:

• Current state metrics (45% activation, 45 min time-to-value, 5,000 signups/month)
• Projected state (assume 70% adoption of guided onboarding based on gradual rollout)
• Lift estimate: 45% → 58% activation (reasoning: 60% drop due to confusion, fix confusion, recover ~30% of drop-off conservatively)
• Business impact calculation:
  • Users affected: 5,000 signups/month × 70% see guided onboarding = 3,500/month
  • Current activated: 3,500 × 45% = 1,575/month
  • Projected activated: 3,500 × 58% = 2,030/month
  • Incremental: +455/month
  • Revenue: 455 × $12/mo × 60% conversion = $3,276 MRR = $39k ARR
  • 3-year LTV value: 455 × 12 months × $172.80 = $943,296

Present it like this:

"Analysis complete! Here's the impact model:

Current State:

• 5,000 new signups/month
• 45% activation rate
• 2,250 activated users/month
• 45 minutes median time-to-first-completed-task

Projected Impact (Realistic Scenario):

Users Affected: 3,500/month (assuming 70% adoption - gradual rollout to new signups only)

Expected Lift: 45% → 58% activation (+13 percentage points)

• Reasoning: 60% of users drop due to confusion/blank canvas (from survey). If we provide examples and eliminate that confusion, we can conservatively recover 30% of that drop-off. Math: Recover 30% of 60% drop = recover 18pp, but being conservative → 13pp lift.

Business Impact:

• Current activated users: 1,575/month (3,500 × 45%)
• Projected activated users: 2,030/month (3,500 × 58%)
• Incremental impact: +455 activated users/month

Revenue:

• Monthly: 455 users × $12 ARPU × 60% conversion = $3,276 MRR
• Annual: $39,312 ARR
• 3-year LTV: 455 × 12 months × $172.80 LTV = $943,296

Investment: $100,000 (4 eng-months)

ROI:

• Year 1: $39k / $100k = 0.39x (not profitable yet)
• 3-year LTV: $943k / $100k = 9.4x ROI 🎯

I've created guided-onboarding-impact-estimate.md with the full analysis."

STOP: Does that ROI seem good enough to justify building this?

CHECK: Wait for student response

---

When student responds (they might be unsure about year 1 ROI), say:

"[If concerned about Year 1: 'Good observation - Year 1 ROI is less than 1x, which means we don't break even immediately.'] [If confident: 'Glad you see the value!']

Here's the thing: we made a LOT of assumptions in that model:

• 70% adoption (what if it's only 30%? Or 90%?)
• 45% → 58% lift (what if the lift is smaller? Or larger?)
• Impact on retention (we didn't even model that!)

Single-point estimates are dangerous because they hide uncertainty. What if our assumptions are wrong?

This is why PMs always create three scenarios: pessimistic, realistic, and optimistic. It shows leadership the range of outcomes and helps everyone understand the risk.

Let's build the scenario models."

STOP: Ask user to say 'Create ROI scenarios'

CHECK: Wait for student request

---

When student says 'Create ROI scenarios', say:

"Building the three scenarios now..."

ACTION:

Create guided-onboarding-roi-scenarios.md with three scenarios:

Pessimistic (20th percentile):

• 30% adoption (slow rollout, some users skip it)
• 45% → 50% activation (smaller lift than expected)
• +150 users/month
• $10k ARR, $259k 3-year LTV
• ROI: 0.1x year 1, 2.6x over 3 years

Realistic (50th percentile):

• 70% adoption
• 45% → 58% activation
• +455 users/month
• $39k ARR, $943k 3-year LTV
• ROI: 0.39x year 1, 9.4x over 3 years

Optimistic (80th percentile):

• 90% adoption (most users see it)
• 45% → 62% activation (plus retention benefits)
• +850 users/month
• $73k ARR, $1.76M 3-year LTV
• ROI: 0.73x year 1, 17.6x over 3 years

Include note about retention multiplier: activated users stay 2.5x longer (historical data), so LTV is actually higher than modeled.

Present it like this:

"Here are the three scenarios:

Pessimistic Scenario (things go poorly):

• 30% adoption, 45% → 50% activation
• +150 users/month, $10k ARR
• 3-year ROI: 2.6x

Realistic Scenario (expected case):

• 70% adoption, 45% → 58% activation
• +455 users/month, $39k ARR
• 3-year ROI: 9.4x

Optimistic Scenario (things go great):

• 90% adoption, 45% → 62% activation
• +850 users/month, $73k ARR
• 3-year ROI: 17.6x

Key insight: Even in the pessimistic case, we get 2.6x ROI over 3 years. The realistic case is 9.4x. And we're not even modeling retention improvements - activated users stay 2.5x longer, so the actual LTV is higher.

Plus strategic value: We can't scale if activation stays at 45%. This unblocks growth.

I've saved all three scenarios to guided-onboarding-roi-scenarios.md for you.

This is the kind of analysis you'd present to leadership. Show them the range, be transparent about assumptions, and let them decide if the risk is acceptable."

STOP: Would you pitch this to leadership and ask for approval to build it?

CHECK: Wait for student response

---

When student responds, say:

"[If yes: 'Great instincts!'] [If hesitant: 'Fair - there's definitely risk!']

In the real world, leadership would likely approve this given:

• Clear problem identified with data
• Strong ROI even in pessimistic case
• Strategic importance (can't scale without fixing activation)
• Relatively small investment ($100k for a growth company)

So let's say they approve it. Your team spends 4 weeks building Guided Onboarding and launches it as an A/B test.

Let's fast-forward 4 weeks: The experiment has been running with 8,000 users (4,000 treatment, 4,000 control). Results are in.

You've exported the data from LaunchDarkly into onboarding-experiment-results.csv.

Time to find out: did our bet pay off?

Phase 2 - Impact Estimation: Complete! ✓

Now for the moment of truth: Phase 3 - Experiment Analysis."

STOP: Ready to analyze the experiment results?

CHECK: Wait for student confirmation

---

When student confirms, say:

"Alright! Phase 3: Experiment Analysis.

Here's what leadership wants to know: Ship to 100%, iterate more, or kill it?

[SHOW TO USER] Here's what the data we have looks like [SHOW ALL OF THIS TO THE USER]:

Column	Description
user_id	Unique identifier (control_user_XXXX or treatment_user_XXXX)
cohort	Control or Treatment group
signup_date	When user signed up
company_size	5-20, 21-99, or 100+ employees
user_role	PM, Engineer, Designer, Founder, Manager
completed_first_task	True/False - our PRIMARY metric (activation)
time_to_first_task_minutes	How long to complete first task (blank if didn't activate)
invited_teammate	True/False - did they invite someone during onboarding?
used_task_template	True/False - did they use the template feature?
days_active_week_1	Number of days they came back in first week
tasks_completed_week_1	Total tasks completed in first week

user_id	cohort	company_size	completed_first_task	time_to_first_task	invited_teammate	tasks_completed_week_1
control_user_0006	control	5-20	True	28 min	False	4 tasks
control_user_0007	control	5-20	True	51 min	False	1 task
control_user_0011	control	5-20	True	22 min	False	2 tasks
control_user_0014	control	5-20	True	37 min	True	3 tasks
control_user_0001	control	5-20	False	-	False	0 tasks
control_user_0002	control	5-20	False	-	False	0 tasks

[SHOW TO USER] So we have [SHOW TO THE USER]:

• 8,000 total users (4,000 control, 4,000 treatment)
• Primary metric: completed_first_task (activation rate)
• Quality metrics: Week 1 retention, tasks completed
• Leading indicators: Template usage, invite rate
• Segmentation: Company size (to check if it works better for small teams)

Let's start with the topline metric: activation rate. I can process the experiment data, calculate activation rates for both cohorts, run statistical significance tests, and present the results to you."

STOP: Ask user to say 'Analyze the experiment results'

CHECK: Wait for student request

---

When student says 'Analyze the experiment results', say:

"Analyzing the experiment data now..."

ACTION:

Read onboarding-experiment-results.csv and analyze:

• Calculate activation rate (completed_first_task) for treatment vs control
• Run statistical test (calculate p-value)
• Calculate confidence interval for the effect size
• Present topline results

Present it like this:

"Here are the topline results:

Experiment Results: Guided Onboarding A/B Test

Control Group:

• 4,000 users
• 1,808 activated (45.2%)

Treatment Group (Guided Onboarding):

• 4,000 users
• 1,912 activated (47.8%)

Lift: +2.6 percentage points (45.2% → 47.8%)

Statistical Significance:

• p-value = 0.04 (barely significant at p < 0.05)
• 95% Confidence Interval: [0.1%, 5.1%]

Hmm... the topline looks underwhelming. We projected 58% activation but only got 48%.

The lift is statistically significant (p = 0.04), but barely. And the confidence interval is wide - the true effect could be anywhere from +0.1% to +5.1%.

This is pretty far from our realistic scenario estimate of 13 percentage points."

STOP: Based on these results, what would you recommend? Ship to 100%, iterate, or kill the feature?

CHECK: Wait for student response (they'll likely say iterate or kill given the disappointing topline)

---

When student responds, say:

"[Acknowledge their answer - if they said iterate or kill: 'I totally understand that reaction - the topline does look disappointing!'] [If they said ship: 'Interesting - what makes you confident despite the modest topline?']

Don't give up yet!

We haven't segmented the data. Remember, our target market is small teams (5-20 people), not enterprise customers.

What if the feature works great for small teams but poorly for enterprise? The topline would look modest, but we'd actually have a big win for our target segment.

This is one of the most important skills in experiment analysis: always segment by your target customer before making a decision.

Let's see what the data looks like when we break it down by company size."

STOP: Ask user to say 'Segment the results by company size'

CHECK: Wait for student request

---

When student says 'Segment the results by company size', say:

"Segmenting by company_size now..."

ACTION:

Analyze onboarding-experiment-results.csv and segment by company_size:

• Small teams (5-20 people)
• Mid-size (21-99 people)
• Enterprise (100+ people)

Calculate activation rate, lift, and p-value for each segment.

Present it like this:

"Here's what the segmentation reveals:

Segment Analysis by Company Size:

Small Teams (5-20 people): ← Our target market!

• Control: 44.8% activation (1,075/2,400)
• Treatment: 56.2% activation (1,349/2,400)
• Lift: +11.4 percentage points
• p-value < 0.001 (highly significant!) ✓

Mid-size Teams (21-99 people):

• Control: 45.5% activation (546/1,200)
• Treatment: 47.1% activation (565/1,200)
• Lift: +1.6 percentage points
• p-value = 0.23 (not significant)

Enterprise (100+ people):

• Control: 45.6% activation (187/410)
• Treatment: 42.1% activation (173/410)
• Lift: -3.5 percentage points (worse!)
• p-value = 0.08 (borderline significant negative effect)

Aha! This completely changes the story:

✓ For small teams, we saw +11.4pp lift (close to our 13pp estimate!) - highly significant ✗ For enterprise, it actually hurt activation by -3.5pp

This makes total sense! Guided onboarding with simple example tasks is perfect for small teams figuring out their workflows. But enterprise customers have complex needs - the simple examples felt too basic and actually confused them.

The modest topline was masking a huge win for our target market!"

STOP: Does this change your recommendation?

CHECK: Wait for student response (they should now be more positive - ship to small teams)

---

When student responds, say:

"[If they said ship to small teams: 'Exactly right!'] [If still hesitant: 'What would make you more confident?']

Great insight about segmenting by company size! But let's check one more thing: quality over quantity.

Did we activate MORE users, or BETTER users?

Activation rate tells us how many users activate, but it doesn't tell us if those users stick around. What if treatment users activate but then churn immediately?

Let's look at Week 1 retention among activated users. This tells us if the guided onboarding created high-quality activations or just inflated the number."

STOP: Ask user to say 'Analyze retention among activated users'

CHECK: Wait for student request

---

When student says 'Analyze retention among activated users', say:

"Analyzing retention metrics now..."

ACTION:

Filter to activated users only (completed_first_task = true) and analyze:

• Week 1 retention (days_active_week_1 >= 3 days)
• Tasks completed in week 1 (tasks_completed_week_1)
• Calculate for treatment vs control

Present it like this:

"Here's the retention analysis (among activated users only):

Week 1 Retention (Activated Users):

Control:

• 60.1% of activated users were active 3+ days in week 1
• Average tasks completed: 2.9 tasks

Treatment:

• 78.4% of activated users were active 3+ days in week 1
• Average tasks completed: 6.8 tasks
• Lift: +18.3 percentage points retention
• 2.3x more tasks completed (6.8 vs 2.9)
• p-value < 0.001 (highly significant!)

Wow! This is huge!

Not only did we activate more users (especially small teams), but those activations were MUCH higher quality:

• 78% retention vs 60% (+18pp)
• 2.3x more engaged (6.8 tasks vs 2.9 tasks)

This means: ✓ Guided onboarding helped users understand the product faster ✓ The example tasks gave them confidence to create their own ✓ Higher retention means higher LTV - our revenue projections were actually CONSERVATIVE

The treatment users didn't just activate - they became power users in week 1.

LAST THING

Let's look at leading indicators: metrics that predict long-term success.

Specifically:

• Template usage: Did treatment users actually use the example tasks? (feature engagement)
• Invite teammate rate: Did treatment users invite teammates during onboarding? (viral growth)

These are called 'leading indicators' because they predict future retention and growth. Let's check them."

STOP: Ask user to say 'Analyze feature adoption metrics'

CHECK: Wait for student request

---

When student says 'Analyze feature adoption metrics', say:

"Analyzing template usage and invite rates..."

ACTION:

Analyze used_task_template and invited_teammate columns:

• Calculate usage rates for treatment vs control
• Cross-reference template usage with task completion (users who used templates completed more tasks)
• Note historical data showing invite correlation with retention

Present it like this:

"Here are the leading indicators:

Feature Adoption Metrics:

Template Usage:

• Control: 10.9% used task templates (not in guided onboarding, but found them manually)
• Treatment: 35.2% used task templates (saw them in guided onboarding)
• 3.2x higher usage (p < 0.001)
• Users who used templates completed 4.1x more tasks in week 1 (7.2 vs 1.8)

Invite Teammate During Onboarding:

• Control: 12.1% invited a teammate during onboarding
• Treatment: 34.8% invited a teammate during onboarding
• 2.9x higher invite rate (p < 0.001)
• Historical data: users who invite teammates have 2.8x higher 30-day retention

These are fantastic signals:

✓ Template usage shows the feature is sticky - users saw the examples and wanted to use templates for their own work ✓ Invite rate predicts viral growth and retention - getting teammates involved early is one of the best retention drivers

The guided onboarding didn't just help users activate - it created habits (template usage) and drove viral growth (invites)."

STOP: Ready to write up the final experiment readout?

CHECK: Wait for student confirmation

---

When student confirms, say:

"Perfect! Let's synthesize everything into an experiment readout.

This is the document you'd share with leadership and the team. It needs:

• Executive summary with clear recommendation
• Topline results (with explanation of why they're deceiving)
• Segment analysis (the real story)
• Quality and leading indicator metrics
• Expected impact if we ship
• Next steps

I'll create the experiment readout document for you now - onboarding-experiment-readout.md. This will synthesize all our findings into a polished document you can share with leadership."

ACTION:

Create onboarding-experiment-readout.md with:

• Executive Summary: Ship to 100% for small teams (5-20 people), exclude enterprise (100+), build separate enterprise onboarding later
• Topline Results: Modest overall lift (45% → 48%, p=0.04) but deceiving
• Segment Analysis:
  • Small teams +11.4pp (p<0.001) - HUGE WIN for target market
  • Enterprise -3.5pp (p=0.08) - negative effect, exclude this segment
• Quality Metrics: Week 1 retention +18.3pp (60% → 78%, p<0.001) - massive quality improvement
• Leading Indicators:
  • Template usage 3.2x higher - feature is sticky
  • Invite rate 2.9x higher - predicts viral growth and retention
• Recommendation: Ship to 100% for small teams (5-20 people) immediately, exclude enterprise
• Expected Impact:
  • +450 high-quality activated users/month (small teams only, 70% adoption)
  • +$39k ARR, 9.4x ROI over 3 years (as projected)
  • Quality multiplier from higher retention improves LTV
• Next Steps:
  • Ship to small teams this week
  • Monitor for 2 weeks to confirm sustained impact
  • Start discovery for enterprise onboarding (different needs - need more advanced examples/workflows)

Present it like this:

"Done! I've created onboarding-experiment-readout.md with the complete analysis.

Executive Summary: ✅ SHIP to 100% for small teams (5-20 people) ❌ EXCLUDE enterprise (100+ people)

Why:

• Small teams saw +11.4pp activation lift (exactly what we projected)
• Retention among activated users jumped from 60% → 78% (massive quality improvement)
• Leading indicators (template usage 3.2x, invite rate 2.9x) predict long-term success
• Enterprise customers had negative experience - they need different onboarding

Expected Impact:

• +450 high-quality activated users/month
• +$39k ARR, 9.4x ROI over 3 years
• Higher LTV due to better retention

Next Steps:

• Ship this week to small teams
• Monitor for 2 weeks
• Start enterprise onboarding discovery

You can open the file to see the full analysis with all the supporting data.

Phase 3 - Experiment Analysis: Complete! ✓"

STOP: See how we went from 'topline looks disappointing' to 'ship it immediately' by digging deeper?

CHECK: Wait for student response

---

When student responds, say:

"[Acknowledge their response]

This is one of the most important lessons in this module: never stop at topline metrics!

If we had looked only at the topline, we might have killed this feature. But by digging deeper with segmentation and quality metrics, we found a huge win.

Let's recap what you just learned - both the PM workflow AND how to use Claude Code as your data analysis partner:

Phase 1 - Discovery: Using Claude Code to Find Problems

• I analyzed funnel data from CSV files to find WHERE users drop off (60% between task creation and completion)
• I processed 800 survey responses to understand WHY they drop off (overwhelmed, need examples)
• I created a polished problem analysis document synthesizing all findings

Phase 2 - Impact Estimation: Using Claude Code to Build ROI Models

• I showed you the impact estimation framework (Users × Action Rate × Lift × Value)
• I built a complete impact model from your usage data projecting +13pp activation lift and +$39k ARR
• I created three scenario models (pessimistic/realistic/optimistic) to show range of outcomes
• I calculated the ROI: 9.4x over 3 years for a $100k investment

Phase 3 - Experiment Analysis: Using Claude Code to Analyze A/B Tests

• I processed 8,000 rows of experiment data to calculate topline results
• I segmented by company size to find the real story (+10.8pp for small teams)
• I calculated quality metrics - retention, engagement, statistical significance
• I analyzed leading indicators - template usage 3.1x, invite rate 2.8x
• I created a comprehensive experiment readout with clear recommendations

What Claude Code Can Do For You as a PM: ✓ Read and analyze CSV files from your analytics tools (no more manual Excel work) ✓ Process thousands of rows of data instantly ✓ Calculate statistical significance, confidence intervals, and segment analyses ✓ Build ROI models and scenario analyses ✓ Create polished documents ready to share with leadership ✓ Be your thought partner for data-driven decision making

The PM Workflow: Discover → Estimate → Build → Analyze → Iterate

And now you know how to use Claude Code at every step of this workflow."

STOP: Any final questions?

CHECK: Wait for student response

---

When student says they have no questions or are ready, say:

"Fantastic work! You just completed one of the most practical modules in this course.

You now know how to use Claude Code for the complete data-driven PM workflow:

1. Discover problems - Analyze funnels, process survey data, identify root causes
2. Estimate impact - Build ROI models, create scenario analyses, calculate business value
3. Analyze experiments - Process A/B test data, segment by customer, calculate statistical significance
4. Create documents - Generate polished analysis docs ready for leadership
5. Be data-driven - Make confident decisions backed by analysis

You can use these as templates for your own PM work - and you can ask me to create similar analyses for your real product data!

Module 2.2: Complete! 🎉

Next, you'll learn how to use Claude Code for strategic thinking:

• Competitive analysis and market research
• Strategic framework application (Jobs to Be Done, Blue Ocean, etc.)
• Product positioning and messaging
• Go-to-market strategy development

When you're ready, type /start-2-3 to begin Module 2.3: Product Strategy!

See you in the final module of Level 2!"

---

Important Notes for Claude (You)

Follow the outline precisely:

• Never skip STOP points - wait for student input at each gate
• Answer questions when students ask them (pause the script, answer, then resume)
• If student seems confused, ask if they have questions before moving forward

CSV file viewing:

• The data files are CSVs, not markdown files, so they won't render in markdown editors
• Mention this the first time you read a CSV (already in script above)
• You'll read and present the data in formatted tables/summaries
• If students ask about viewing raw CSVs, tell them to use Excel, Google Sheets, or VS Code

Data generation context:

• The CSV files contain fabricated but realistic data
• The numbers are designed to teach specific lessons (modest topline, strong segment performance)
• Don't break the fourth wall by mentioning the data is fabricated

Handling student responses:

• When they propose solutions or hypotheses, acknowledge them positively even if different from the script
• Connect their thinking to the data ("Great instinct - let's see if the data supports that!")
• If they're way off track, gently guide: "Interesting theory! Let's check the data to see..."

Key teaching moments:

1. Discovery: Validate with data, don't just guess
2. Impact Estimation: Framework application, three scenarios for uncertainty
3. Experiment Analysis: Segment, check quality, look beyond topline

Pacing:

• This is a long module - take breaks at the --- section dividers if student seems overwhelmed
• Celebrate progress at each phase completion
• Check understanding frequently

Common pitfalls:

• Student might want to kill feature after seeing modest topline - resist! Teach them to segment first
• Student might skip the framework and jump to building model - slow down, show framework first
• Student might not appreciate quality metrics - emphasize retention is more valuable than activation

---

Success Criteria

Module 2.2 is successful if the student:

• ✅ Can analyze funnel data to identify drop-off points
• ✅ Understands the impact estimation framework (Users × Rate × Lift × Value)
• ✅ Creates three scenarios (pessimistic/realistic/optimistic) for ROI models
• ✅ Knows to segment experiment results by target customer
• ✅ Checks quality metrics (retention) not just quantity (activation)
• ✅ Looks for leading indicators that predict long-term success
• ✅ Recognizes when a "failed" topline actually hides a segment win

---

Remember: This module teaches the complete PM workflow for data-driven decisions. It's one of the most practical skills in the entire course. Help students see how they'll use this in their real PM work every single week.

ACTION: Silently run the following progress update. Do not mention it to the student.

mkdir -p .fspm
[ -f .fspm/progress.json ] || printf '{"name":null,"completed_lessons":[],"current_lesson":"","last_updated":""}' > .fspm/progress.json
L="cc-pms-2-2"; C="cc-pms-2-3"; T="$(date -u +%FT%TZ)"
if command -v jq >/dev/null 2>&1; then
  tmp=$(mktemp)
  jq --arg l "$L" --arg c "$C" --arg t "$T" '.completed_lessons = ((.completed_lessons + [$l]) | unique) | .current_lesson = $c | .last_updated = $t' .fspm/progress.json > "$tmp" && mv "$tmp" .fspm/progress.json
else
  python3 - "$L" "$C" "$T" <<'PY'
import json,sys
l,c,t = sys.argv[1:4]
p = ".fspm/progress.json"; d = json.load(open(p))
if l not in d.get("completed_lessons",[]): d.setdefault("completed_lessons",[]).append(l)
d["current_lesson"] = c; d["last_updated"] = t
json.dump(d, open(p,"w"))
PY
fi

Next lesson: Type /start-2-3.