playtest-design

name

playtest-design

description

Designing and reading playtests: what to observe versus what to ask, question generation, metrics to track, sample sizes, confirmation-bias protocol, and solo-developer validation when you have no testers. Use when planning a playtest or usability session, writing a feedback survey, deciding what to watch for, or when you have playtest data and need to decide what it means. Reach for this when a tester says 'it was fine' and you learn nothing, when you catch yourself explaining the game to the person playing it, or when you need to know whether five testers is enough. Observation-first: self-report is the least reliable data a playtest produces. For capturing one felt reaction verbatim as it happens use moment-capture; for post-ship quantitative measurement use data-driven-design.

Playtest Design

Purpose: Get useful signal from playtests. Most playtest sessions are wasted — observers confirm what they already believe, ask leading questions, and draw conclusions from noise. This skill provides structured methods to avoid those traps.

Sources: Observation-over-self-report and the developer-blindness counter-measures follow Celia Hodent, The Gamer's Brain (2017). Confirmation-bias protocol draws on standard experimental-design practice. Small-sample usability guidance derives from Nielsen & Landauer (1993), whose "five users find ~85% of usability problems" applies to usability testing specifically and does not transfer to balance or preference questions — a distinction this skill is careful about below.

---

Core Principle: Observe, Then Ask

Players are reliable reporters of their experience (what they felt) but unreliable reporters of causes (why they felt it). Design your process accordingly.

Most Reliable ←———————————————→ Least Reliable
  What they did    What they felt    Why they think
  (behavior)       (experience)      they felt it
                                     (attribution)

Hierarchy of evidence:

1. Behavioral data — what players actually did (metrics, video, observation)
2. Experience reports — what players say they felt ("I was frustrated," "that was exciting")
3. Causal attribution — what players think caused their experience ("the controls are bad")

Players attributing frustration to "bad controls" might actually be experiencing a perception failure (they couldn't see the indicator) or a pacing problem (too many new concepts at once). Use behavior to diagnose; use self-report to locate.

---

Question Generation Framework

The Three-Pillar Method

Generate questions along the perception → attention → memory pipeline:

Perception Questions (Did they see it?)

• Did the player notice [critical UI element / feedback / environmental cue]?
• How long before they noticed?
• Did they look at it before acting or after?
• Did they confuse it with something else?

Attention Questions (Did they focus on the right thing?)

• Where was the player looking during [critical moment]?
• Did they engage with [intended system] or get distracted by [ancillary system]?
• Did they understand what was important vs. optional?
• Was there a moment where they seemed overwhelmed?

Memory Questions (Will they retain it?)

• After a break, can they recall how to [key mechanic]?
• Did they apply a lesson learned earlier to a later challenge?
• Did they remember the goal after a distraction?
• Can they explain the core rules to someone else?

Stage-Specific Questions

Dev Stage	Focus	Key Questions
Prototype	Core loop viability	Is the core action inherently interesting? Do they want to do it again?
Alpha	System comprehension	Do they understand the rules? Can they make intentional decisions?
Beta	Pacing and polish	Does the session arc feel right? Where do they get bored or frustrated?
Pre-launch	Edge cases and balance	What breaks? What's exploitable? What did we miss?

---

Observation Protocol

What to Watch (Not Ask)

Observable	What It Tells You
First action	What the UI communicates as "start here"
Hesitation points	Where clarity fails or cognitive load spikes
Repeated failures	Where difficulty exceeds skill (or UI is misleading)
Where they look	What's grabbing attention (intended or not)
Body language	Leaning in = engaged; leaning back = disengaged; fidgeting = frustrated
Utterances	Unprompted comments ("what?", "oh!", "come on") are gold
Where they quit	The most valuable data point you'll collect
What they skip	Content they ignore reveals priority mismatches

The Silent Observer Protocol

1. Say nothing unless they're about to break the test setup
2. Don't explain — if they're confused, that's data
3. Don't reassure — "you're doing great" biases the session
4. Note timestamps — when you feel the urge to help, write down the time and why
5. Record everything — your memory of the session will be biased toward your expectations

---

Metrics to Track

Core Metrics (Track Always)

Metric	What It Measures	Warning Signal
Session length	Engagement	Bimodal distribution (some quit fast, some stay long)
Quit points	Pain points	Cluster of quits at same location/moment
Completion rate	Difficulty/clarity	< 70% on intended-critical-path content
Time per section	Pacing	Sections taking 2x+ longer than designed
Death/failure rate	Difficulty curve	Spike = wall; zero = too easy

Balance Metrics (When Tuning Systems)

Metric	What It Measures	Warning Signal
Pick rate by option	Strategy diversity	One option > 50% pick rate
Win rate by strategy	Balance	Any strategy > 55% win rate at comparable skill
Average game/match length	Pacing	Games consistently shorter or longer than intended
Resource accumulation rate	Economy health	Exponential growth = inflation incoming
Strategy churn	Meta health	If dominant strategy shifts too fast, balance is noisy

UX Metrics (When Testing Comprehension)

Metric	What It Measures	Warning Signal
Time to first meaningful action	Onboarding quality	> 60 seconds before the player does something
Tutorial completion rate	Tutorial design	< 90% = tutorial is the problem, not the player
Hint/help usage	Clarity	High usage = UI isn't communicating; zero usage = help system is invisible
Error rate on intended actions	Usability	Player tries to do the right thing but fails due to UI

---

Avoiding Confirmation Bias

The biggest threat to useful playtest data is your own expectations.

Pre-Test Protocol

Before the session:

1. Write down your predictions — what do you expect to happen?
2. Define "surprising" outcomes — what would change your mind?
3. Assign a skeptic — one team member whose job is to challenge interpretations
4. Pre-commit to sample size — decide how many sessions before drawing conclusions (minimum 5 for qualitative, 30+ for quantitative)

Post-Test Protocol

After the session:

1. Review predictions vs. reality — where were you wrong? Those are the insights.
2. Separate observation from interpretation — "Player hesitated for 8 seconds at the door" (observation) vs. "Player didn't understand the door mechanic" (interpretation)
3. Look for disconfirming evidence — actively search for data that contradicts your preferred narrative
4. Quantify before concluding — "it felt like everyone struggled" vs. "3 of 7 players failed this section"
5. Delay solutions — understand the problem fully before proposing fixes

Common Bias Traps

Trap	Mechanism	Counter
Anchoring	First session dominates your impression	Review all sessions before concluding
Availability	Dramatic moments overshadow quiet ones	Use metrics, not memory
Projection	Attributing your own experience to players	Watch what they do, not what you'd do
Sunk cost	Defending features you spent time on	Ask "would we add this today?" not "should we cut this?"
Survivorship	Only hearing from players who stayed	Track quit points with equal priority

---

Survey Design

Good Questions (Experience-Focused)

• "How would you describe the experience in one word?"
• "What moment stands out most?" (Then probe: "What made it stand out?")
• "Was there a point where you wanted to stop? What was happening?"
• "What would you do differently on a second playthrough?"
• "Rate how [specific emotion] you felt during [specific moment]" (1-5 scale)

Bad Questions (Leading or Attributive)

• "Did you find the controls intuitive?" (Leading — assumes controls are the issue)
• "What would you change?" (Too broad — gets surface-level answers)
• "Did you like it?" (Binary, social pressure toward "yes")
• "Was it too hard?" (Leading — frames difficulty as the variable)
• "What features would you add?" (Players aren't designers; this generates noise)

The One-Question Shortcut

If you can only ask one question: "Tell me about a moment that stood out — good or bad."

Then follow up with: "What were you trying to do?" and "What happened next?"

---

Interpreting Data

Decision Framework

Signal	Confidence	Action
Metrics + observation + self-report all agree	High	Act on it
Metrics show it, observation confirms, self-report disagrees	Moderate-High	Trust behavior over self-report
Self-report says it, but metrics/observation don't show it	Low	Investigate further — the report may point to a different real problem
Single session shows it, others don't	Very Low	Note it but don't act — one data point isn't a pattern

Sample Size Guidance

• 5-8 sessions — finds ~85% of major usability problems
• 15-20 — identifies behavioral patterns
• 30+ — minimum for quantitative conclusions (win rates, balance)
• A/B tests — require statistical power calculation; varies by effect size

---

Solo Developer Validation

When you're building alone, you can't run traditional playtests during development. These techniques bridge the gap:

Self-Testing Techniques

Technique	How	What It Catches
The 2-week break	Play your own game after not touching it for 2 weeks	UX failures, forgotten controls, unclear objectives
The mute test	Play with sound off	Audio-dependent information, missing visual feedback
The squint test	Squint at the screen or reduce resolution	Visual clarity, contrast, UI readability
The record-and-review	Record gameplay, watch it the next day	Pacing problems, dead time, repetitive patterns
The explain test	Explain what you're doing out loud while playing	Logic gaps, unjustified assumptions, unclear goals
The wrong-hand test	Play with your non-dominant hand	Input complexity, timing windows, control accessibility

Recruiting First Testers

When you're ready for external eyes (earlier than you think):

1. Friends/family who DON'T play games — best for UX/clarity testing
2. Friends who play games in your genre — best for feel/depth testing
3. Online communities (itch.io, indie forums, Discord) — best for unbiased feedback
4. Start with 3 testers — even 3 external players reveal more than 100 hours of self-play

Solo Metrics

If you're a solo developer shipping updates:

• Track your own play session length (are YOU getting bored?)
• Count deaths/failures per section (is difficulty spiking where you don't intend?)
• Time each section (is pacing matching your design?)
• Screenshot every moment of confusion or frustration — these are your UX bugs

---

Cross-References

Three tight links. Everything else routes through the map, so adding a skill touches one file rather than twenty: references/routing-map.md (in game-design).

• moment-capture — Capture felt reactions verbatim as they happen
• data-driven-design — The quantitative half of the same question
• player-ux — What to look for, and why self-report misses it