How to run a playtest that tells you something

What a playtest is, and what it is not

That last row is the one that trips teams up. A player saying "the map should have a filter" is data about frustration, not a specification. What they were actually doing when they said it is the useful part.

How many players

The widely cited finding from usability research is that around five participants surface the large majority of usability problems in a given design, because the same obstacles recur quickly. That number is right for the question "where do people get stuck", which is what most game playtests are for.

• Five, for usability. Where they get lost, what they misread, what they cannot find. Problems that appear in three of five sessions are real.
• More, and structured, for preference and balance. "Is this fun" and "is this too hard" are different questions that need larger numbers and comparisons.
• Five again, after the fix. Small and frequent beats large and occasional. Two rounds of five will improve a game more than one round of twenty.
• One is not nothing. A single session watched properly is worth more than none, and it is often enough to find something embarrassing.

Who to recruit

• People who match your actual audience, particularly in genre familiarity. A strategy veteran and someone who has never played one will fail at completely different points, and both are useful, but not interchangeable.
• Nobody who has seen the game. Not the team, not their partners, not the friend who has been following the devlog. First contact is the thing you are measuring, and it can only be spent once per person.
• Not your friends, unless you are certain they will tell you the game is boring while you are sitting next to them.
• Mix experience levels across the round. Five people who are all experts will tell you nothing about your onboarding.
• Say what it costs them. An hour, a screen recording, and honesty. Pay them if you can, even a small amount, because it changes the relationship from favour to work.

The setup

1. A build that survives an hour. Not the newest one, the most stable one. A crash costs you a session.
2. Save states or a shortcut to the section you are testing, so you are not spending forty minutes reaching the screen you care about.
3. Record the screen and the audio. Face optional and useful. You will miss things live, every time.
4. Sit slightly behind them, not opposite. Watching a face across a table changes how people behave.
5. Have a notes template, so you write behaviour rather than conclusions during the session.
6. Test remotely if you must, with screen share and think-aloud. It works, it loses some body language, and it widens who you can recruit.

The script

Same structure every time, so sessions can be compared. It takes about an hour.

1. 01

   Set expectations, two minutes

   Tell them you are testing the game, not them, that anything confusing is the game's fault, and that you will mostly stay quiet. Ask them to say what they are thinking out loud, and warn them you may not answer questions.

2. 02

   Warm-up, three minutes

   Ask what they play, on what, and how much. It relaxes them, it gives you context for everything that follows, and it tells you which kind of participant you have.

3. 03

   First contact, unassisted

   Hand them the game with one sentence and nothing else. This is the highest-value ten minutes of the whole session, and every word you say during it destroys some of the data.

4. 04

   Free play, twenty to thirty minutes

   Let them go where they want. Note hesitations, wrong turns, repeated attempts, and anything they say. Resist the urge to steer them towards the part you wanted feedback on.

5. 05

   Directed tasks, ten minutes

   Now ask for specifics: "upgrade something", "change the controls", "find out what that item does". Phrase the goal, never the route.

6. 06

   Debrief, ten minutes

   Ask what the game is about, what they liked, what confused them, and whether they would play again. Ask about specific moments you saw, not about design opinions.

What to say, and what never to say

Watch behaviour, discount opinions

What people do is evidence. What people say about what they would do is a prediction, and people are poor at predicting their own behaviour. Weight your notes accordingly.

• Hesitation. Any pause over about three seconds in front of a screen is a signal. Note what they were looking at.
• Wrong first attempts. Where did they click before they clicked the right thing? That is where they expected it to be.
• Repeats. Doing the same thing twice usually means the first attempt gave no feedback, which is a feedback problem rather than a comprehension one.
• Wrong vocabulary. If they call it a shield and you call it a barrier, your text is fighting their mental model. That is a UX writing problem.
• Where their eyes go on a new screen. The first thing they look at should be the most important thing.
• The gap between what they say and what they did. "That was clear" from someone who took ninety seconds to find the button is a politeness reflex, not a verdict.

Turning notes into a fix list

After five sessions you will have a hundred observations and no time to act on a hundred things. Sort them the same way every round.

1. Group by what happened, not by who said it. Five people struggling with one screen is one problem, not five.
2. Count frequency. Three or more out of five is a real problem. One of five is a note to watch.
3. Rate severity. Did it stop them, slow them, or annoy them? Stopping beats everything.
4. Multiply. Frequent and stopping goes to the top. Rare and annoying goes to the bottom, and that is fine.
5. Write the problem, not the solution. "Four of five could not find the upgrade screen" leaves the design open. "Add an upgrade button to the HUD" closes it before anyone thought.
6. Decide what you will not fix, explicitly, and say why. A list where everything is a priority is a list nobody uses.

What ruins a playtest

• Helping. The single biggest destroyer of playtest value, and the hardest habit to break.
• Testing with the team, who know where everything is and cannot un-know it.
• Leading questions, which reliably produce the answer the asker wanted.
• Testing too late. A playtest three weeks before launch produces a list of things you cannot afford to fix.
• Only testing the opening. The mid-game is where the systems interact and where long-term players are lost.
• Treating one loud opinion as a mandate, particularly if that opinion came from the most confident participant.
• Skipping the write-up. A session nobody summarised is a session that did not happen. An hour of notes turns into a page anyone on the team can act on.

Remote, unmoderated and telemetry

The pairing that works is telemetry to find where, and moderated sessions to find why. Neither one alone is enough, and the funnel instrumentation to make the first half work is in the first ten minutes.

Every playtest tells you something. A badly run one tells you what the team already believed.

A one-day plan

1. Morning: write one sentence describing what you want to learn. Everything else follows from it.
2. Pick the build, make save states, test the recording setup on yourself.
3. Recruit five people, spaced ninety minutes apart, and warn them it is an unfinished game.
4. Run the sessions with the script above. Take behavioural notes only.
5. Between sessions, spend five minutes writing the three things that stood out.
6. End of day: group the observations, count frequency, rate severity.
7. Write one page: the top five problems, with a timestamp for each so anyone can watch it happen.
8. Share the clips, not just the summary. Watching a player fail changes minds that a bullet point cannot.
9. Fix the top three.
10. Book the next five people.