What should I measure in a playtest?
Measure starts, session length, the point where testers quit, errors and crashes, and returns. Each one answers a different question, and each one comes from a small set of events. Surveys and interviews tell you why. Events tell you how many and where.
- Playtest survey questions: what to ask your playtesters →
- Playtest survey builder →
- Best playtest and feedback tools for indie games →
| Measure | Event | Question it answers |
|---|---|---|
| Starts | game_open with is_first_open | How many testers launched the build? |
| Session length | session_end with duration_s | How long does a session last? |
| Progress | level_start, level_complete, level_fail | Where do testers stop or fail? |
| Errors and crashes | error, app_unclean_exit | What share of sessions crash? |
| Returns | session_start on a later day | Do testers come back the next day? |
| Intent | A click on your wishlist or feedback button | Do testers want more? |
Sources: Manalens app events, metrics and error tracking documentation.
How does Steam Playtest work?
Steam Playtest is a separate child app of your game. It has no store page of its own and shows as a section on the store page of the base game. Because it is a separate app, it does not affect the wishlists, reviews, refunds or achievements of your main game.
Players request access on your store page. With limited sign-up, the default, you grant access in batches, and Steam selects players at random from the pool. With open sign-up, Steam adds players as they ask. You can also target countries, for example to manage server latency.
Playtests must be free. An experimental option lets each player invite friends: you choose 1, 2 or 3 invites per player, and players must have been friends for 30 days. The Steamworks example: 5,000 players with three invites each can send up to 15,000 invites.
Steam emails a player when you add them to the playtest. It does not email anyone when you set a build to playable, so Valve recommends an announcement for that. A playtest is not confidential: Valve says testers are not under a nondisclosure agreement and should not expect secrecy.
The Steamworks Playtest documentation does not describe a metrics report. Its dashboard shows the users who have access. For feedback, Valve says that collecting it is up to you, and notes that a number of developers put a large feedback button on the main menu. Plan your own measurement before the first batch of testers.
| Topic | What Valve documents |
|---|---|
| Price | Free for players and developers. You cannot charge for access or sell playtest keys. |
| Store page | None of its own. The sign-up shows on the store page of the base game. |
| Sign-up | Limited (the default, you add players in batches) or open (players are added as they ask). |
| Who gets in | Steam picks at random from the players who asked. You can limit by country. |
| Friend invites | Experimental. 1, 2 or 3 invites per player, only to friends of at least 30 days. |
| Keys | Standard release keys only. Above 50,000 keys, Valve treats the test as an open beta. |
| Main game | Playtime, wishlists, reviews and refunds are separate. A tester cannot review the main game. |
| End | Set the playtest to Not Playable. A reset removes all testers and cannot be undone. |
Sources: Steamworks Steam Playtest documentation.
Should I use Steam Playtest, keys or a demo?
Use Steam Playtest for most closed tests, release override keys for a confidential test, and a demo when you want everyone to play. Valve calls Steam Playtest the preferred way to run a closed beta. It names two common ways to test with people outside the team: Steam Playtest, and release override keys on the App ID of the main game.
Release override keys are limited to 2,500 per app, and they keep access to the base game until you ban them. Valve points to them when you need a nondisclosure agreement, because you run the sign-up yourself. Playtest keys are easier: you can ask for larger batches, and they work even before the store page is live.
A demo is public and has no gate. Steam Next Fest needs one. A playtest does not change your Next Fest eligibility, but Valve does not recommend a live playtest during the fest, because it splits your players between two builds. If you want to charge for an unfinished game, Valve points to Early Access, which is for games in a playable alpha or beta state.
- Steam Playtest or demo: which one? →
- Glossary: Steam demo →
- Demo analytics: playtime and demo to wishlist benchmarks →
| Option | Who can play | Good for | Limit |
|---|---|---|---|
| Steam Playtest | Players you add, or anyone with open sign-up | Closed and open tests before release | Must be free; not confidential |
| Release override keys | Players you send a key to | A confidential test with your own sign-up | 2,500 keys per app |
| Demo | Everyone | Steam Next Fest and wishlists | Public; no control over who plays |
Sources: Steamworks Steam Playtest, Testing On Steam, Steam Next Fest and Early Access documentation.
How many testers do I need?
About 5 per round to find problems, and 20 or more to trust a number. Both figures come from usability research on websites by Jakob Nielsen, not from games, so read them as a starting point.
Nielsen found that the first 5 test users find about 85% of the usability problems, because a single user finds 31% on average. He advises three small rounds of 5 with a fix after each one, not one round of 15. For a game, that means: watch 5 people play the first 30 minutes, fix what stops them, then invite the next 5.
Numbers need more people. Nielsen writes that 20 users give a reasonably tight confidence interval for a usability metric. With 20 testers, one tester moves a rate by 5 percentage points, by our arithmetic. So a funnel from a 20 person test can show a wall, but it cannot show a small change between two builds.
This is why limited sign-up fits a playtest well: add a small batch, fix, then add the next batch on a new build.
Sources: Nielsen Norman Group (18 March 2000; 25 June 2006); Steamworks Steam Playtest documentation.
How do I set up analytics for a playtest build?
Install the SDK, give the playtest build its own build value, turn the SDK off in the editor, and add a few progress events. The Godot and Unity SDKs send game_open, session_start, session_end, errors and crashes for you.
- Install the Godot addon or the Unity package, and copy the Publishable key from your project settings. Never ship a Secret key in a build.
- Call init with build set to the playtest build, for example 571, and storefront set to steam.
- Set enabled to false in the editor, so your own test runs do not mix into the results.
- Track level_start, level_complete and level_fail with level and attempt, or one event per checkpoint.
- Run the build once and check that game_open and session_start appear in the activity feed.
- Give each new playtest build a new build value before you send it to testers.
GDScript
Manalens.init({
"publishable_key": "YOUR_PUBLISHABLE_KEY",
"endpoint": "https://YOUR_COLLECTOR_ENDPOINT",
"app_version": "0.9.0",
"build": "571",
"storefront": "steam",
"enabled": not OS.has_feature("editor"),
})
Manalens.track("level_start", {"level": 2, "attempt": 1})Sources: Manalens Godot SDK and Unity SDK documentation.
How do I find where testers quit?
Build a funnel from the first launch to your later checkpoints and read the largest drop. A funnel in Manalens has two to five steps and counts people, not events. The default conversion window is 24 hours, which suits a first session.
The Levels table in the Game KPIs dashboard adds the fail rate per level: fails divided by completes plus fails. A level with a high fail rate and a large funnel drop is a strong lead. User paths then show what testers did before they left.
Conversion insights under a funnel list the events that go with completion. They show correlation, not cause, and a row needs at least 10 people. Small playtests often show no rows.
Sources: Manalens funnels, user paths and metrics documentation.
How do I compare two playtest builds?
Filter or break down every report by build. The App filter group has Build, so a filter such as Build is 571 limits a report to one build. Funnels, top events and retention take a breakdown by app_build.
The Game health dashboard template shows crash-free sessions, crash-free players, errored sessions and errors per session by build. A crash counts when the next launch sends app_unclean_exit in the session that crashed. Manalens does not store native crash dumps, so keep a crash reporter such as Sentry or Backtrace for stack traces.
Compare builds over date ranges of the same length. A longer range has more sessions for each player, so the rates change.
Sources: Manalens app events and error tracking documentation.
What does a playtest report look like?
One page: the build, the question, the funnel, stability, returns, the top survey answers and the decisions. The table below is a sample report for a made-up game. Every number in it is an example, not a benchmark.
Read the sample from the top. 148 of 200 invited players launched the build. The largest loss is at the first boss: 121 testers finished the tutorial, but only 77 beat the boss, and the level has a 64% fail rate. Crashes are a second problem, but a smaller one. So the decision is about the boss, and the next build tests that one change.
Write the decision in the report, with the build that will test it. A report without a decision is only a list of numbers.
- Write the question before the test starts, and one number that would answer it.
- After the test, filter every report by the playtest build.
- Fill the funnel lines first, then stability, then returns.
- Add the three most common survey answers in the words of the testers.
- End with at most three decisions, and name the build that will test each one.
- Playtest survey questions: what to ask your playtesters →
- Playtest survey builder →
- Glossary: D1 retention →
- Glossary: Crash-free rate →
| Line | Example value | Where it comes from |
|---|---|---|
| Build and dates | Build 571, 7 days | The build value in the SDK |
| Question | Can new players beat the first boss? | Written before the test |
| Players given access | 200 | Steam Playtest dashboard |
| Launched the build | 148 (74% of 200) | game_open with is_first_open |
| Finished the tutorial | 121 (82% of 148) | Funnel step: your tutorial_complete event |
| Beat the first boss | 77 (52% of 148) | Funnel step: level_complete, level 2 |
| Fail rate at the first boss | 64% | Levels table: fails / (completes + fails) |
| Crash-free sessions | 96.5% | Game health dashboard, build 571 |
| Came back the next day | 53 (36% of 148) | Daily retention, D1 |
| Survey answers | 61 (41% of 148) | Your survey tool |
| Top survey answer | The boss gives no hint before its big attack | Open question in the survey |
| Decision | Add a warning animation; test again in build 572 | The team |
Sources: sample report with made-up numbers; our own method.
Will playtest numbers match my launch?
No. Testers chose to sign up, and friend invites bring in people who know each other. Use playtest numbers to compare one build with the next, not to forecast sales.
Retention in a short playtest is also limited. A daily retention chart in Manalens shows days 0 to 30, and periods that are not complete show a dash, not a low number. A one-week playtest can show D1, but not D30. Rolling retention counts a tester who returns on that day or later, so it suits a short test with few testers.
Session length needs the same care. The Game KPIs dashboard shows the average duration_s of session_end events, and a few very long sessions pull an average up. Read it next to the funnel, and compare one build with the next, not with a number from another tool.
Sources: Steamworks Steam Playtest documentation; Manalens retention and metrics documentation.
What should I not collect from testers?
Do not put personal data in event names, properties or error messages. The SDKs send a random anonymous ID, the session ID and the app context. They send no device ID, no hardware details and no IP, and the collector derives the country and does not store the IP.
If your playtest needs consent before tracking, start the SDK with consent_required. Then it records no events and writes no ID until set_consent(true).
Sources: Manalens Godot SDK and app events documentation.
How do I start measuring my next playtest?
Manalens is the analytics and growth engine built for games. Install the Godot or Unity SDK, tag the playtest build, and read the first session funnel, Game health and returns by build in one project.
Sources: Manalens product documentation.
Frequently asked questions
Does a Steam Playtest affect my main game reviews?
No. Steam Playtest is a separate child app of your game. According to the Steamworks documentation, it does not affect the wishlists, reviews, refunds or achievements of the main game. It has no store page of its own and shows as a section on the base game store page.
What is the first metric to check after a playtest?
Check how far testers got in the first session. Build a funnel from game_open to your first few checkpoints, read the largest drop, then check the fail rate of that level and the crash-free sessions of the build. Fix that point before you add content.
How do I keep my own test runs out of the playtest data?
Set enabled to false in the editor. In Godot, use enabled: not OS.has_feature("editor"). In Unity, use Enabled = !Application.isEditor. With enabled set to false, the SDK sends nothing, writes no files and keeps no ID, so only builds that testers run send events.
Can I measure D7 retention in a playtest?
Only if the playtest stays open for at least seven days after testers start. Manalens shows a dash for periods that are not complete. For a short test, read D1 and use rolling retention, which counts a tester who returns on that day or any later day.
Sources and implementation
- Steamworks documentation: Steam Playtest (Valve, read 19 November 2026) ↗
- Steamworks documentation: Testing On Steam (Valve, read 19 November 2026) ↗
- Steamworks documentation: Steam Next Fest (Valve, read 19 November 2026) ↗
- Steamworks documentation: Early Access (Valve, read 19 November 2026) ↗
- Nielsen Norman Group: Why You Only Need to Test with 5 Users (Jakob Nielsen, 18 March 2000) ↗
- Nielsen Norman Group: Quantitative Studies: How Many Users to Test? (Jakob Nielsen, 25 June 2006) ↗