✦ Now with returning players and D1, D7, D30 retention
☰
All featuresAttributionRetentionAnalytics OverviewIntegrationsPricingDocumentationComparisonsFree toolsBlogGlossary
FREE GAME CALCULATOR

A/B test significance calculator for games

A difference between two variants is significant at 95% confidence when the two-sided p-value is below 0.05. Enter visits and wishlists, or players and retained players, for a capsule, trailer thumbnail, ad creative or tutorial test. The calculator gives a two-proportion z-test p-value, a confidence interval for the difference and the sample size you need per variant.

Rate A and rate B8%, 9.5%
Lift of B over A18.75%
p-value (two-sided)0.0176
95% interval for B - A (points)0.26 to 2.74
ResultSignificant at 95%
Visits needed per variant18,873

z = 2.374. Each group has at least 10 conversions and 10 non-conversions, so the normal method is fine. The sample size is for a lift of 10% over the rate of A, with 80% power.

How to use it

  1. Pick "What you measure": visits and wishlists, or players and retained players.
  2. Pick the "Confidence level": 90%, 95% or 99%.
  3. Enter the visits or players and the wishlists or retained players for A and for B.
  4. Enter the "Smallest lift you care about (%)" and pick the "Power" for the sample size.
  5. Read the p-value, the interval and the result, and check the sample size needed per variant before you stop the test. The page address keeps your inputs.

How it works

Each rate is conversions divided by visitors. The test is the pooled two-proportion z-test from the NIST/SEMATECH handbook: the pooled rate is all conversions divided by all visitors, and z is the difference in rates divided by the pooled standard error. The p-value is two-sided, and the result is significant when it is below 1 minus the confidence level.

The interval for B minus A uses the unpooled standard error from Penn State STAT 200. The normal method is reliable when each group has at least 10 conversions and 10 non-conversions, and the tool warns you when a group has fewer.

The sample size per variant is the Penn State STAT 509 formula with equal groups. p1 is the rate of A, and p2 is the rate of A raised by the smallest lift you care about, as a relative lift.

rate = conversions ÷ visitors
pooled = (conversions A + conversions B) ÷ (visitors A + visitors B)
z = (rate B - rate A) ÷ √(pooled × (1 - pooled) × (1/visitors A + 1/visitors B))
interval = (rate B - rate A) ± z* × √(rate A (1 - rate A)/visitors A + rate B (1 - rate B)/visitors B)
n per variant = 2 × (z(1 - α/2) + z(power))² × p̄ (1 - p̄) ÷ (p1 - p2)², p̄ = (p1 + p2) ÷ 2

Worked example

Example with made-up inputs: two capsule ad creatives, 5,000 store visits each

Creative A brought 300 wishlists from 5,000 tagged store visits, 6%. Creative B brought 360, 7.2%. The lift of B over A is 20%, and z = 2.417.

The two-sided p-value is 0.0157, below 0.05, so the result is significant at 95%. The 95% interval for B minus A runs from 0.23 to 2.17 percentage points, so the real gain could be small. A test planned to detect a 10% lift over 6% with 80% power would need 25,741 visits per variant. This test reached significance because the lift was larger, but with fewer visits its interval stays wide.

How to read the result

Significant means a difference this large would be unlikely if A and B were the same. It does not mean the lift is as large as measured. Read the interval: its low end is the smallest gain that fits the data at that confidence.

Decide the sample size before you start, and do not stop the test the first time it looks significant. Checking often and stopping early makes false wins more likely. Our practice: run a store page or ad test for at least one full week, so weekdays and the weekend are both in it.

The p-value uses the pooled error and the interval uses the unpooled error, so near the line they can disagree: a p-value just under the cutoff with an interval that touches 0, or the reverse. When that happens, treat the result as unclear and collect more data.

How to split traffic for a game test

This calculator does not split traffic, so you split it yourself. To compare capsule art or a trailer thumbnail, run them as two ad creatives or two posts that link to your Steam page with different utm_content, at the same time and to similar audiences. For a store page change, compare the same link before and after only with care, because traffic changes week to week. For a tutorial or onboarding change, give each new player a variant in the game and count who comes back on day 1 or day 7.

Sample size and test length

Small lifts on low rates need many visitors. At a 6% wishlist rate, detecting a 10% relative lift at 95% confidence and 80% power needs 25,741 visits per variant. A 50% lift on the same rate needs 1,211. Divide the visits you need by the visits each variant gets per day to see how many days the test must run. If that is months, test a bigger change.

Limits

The calculator compares two variants. With three or more, each extra comparison raises the chance of a false win, and this tool does not correct for that.

It uses a fixed sample, not a sequential test. Its p-value holds only if you stop at the sample size you planned.

It does not split traffic or collect data. The Steamworks UTM report counts only visits that load the store page with your tags and leaves out rows with few visits, so your counts can be lower than your ad clicks.

The smallest lift is relative: 10% on a 6% rate means 6.6%, not 16%.

Frequently asked questions

How do I know if my A/B test result is significant?

Compare the p-value with 1 minus your confidence level. At 95% confidence, a p-value below 0.05 is significant. Check the interval too: when it stays above zero, B likely beats A. Make sure each group has at least 10 conversions and 10 non-conversions, and that you reached the sample size you planned before you started.

What does the p-value mean?

It is the chance of seeing a difference at least this large if the two variants were really the same. A small p-value means the difference is unlikely to be luck alone. It does not tell you how large the real difference is, or the chance that B is better. The confidence interval shows the likely size of the difference.

How do I A/B test a Steam capsule or store page?

Split the traffic yourself. Make two tagged Steam links that differ only in utm_content, and use each one with a different capsule creative or post, shown to similar audiences at the same time. Read visits and wishlists for each link in the Steamworks UTM report, then enter them here as A and B.

Why do I need so many players?

Small lifts on low rates need large samples. At a 6% wishlist rate, detecting a 10% relative lift at 95% confidence and 80% power needs about 25,741 visits per variant. A 50% lift on the same rate needs about 1,211. Test big changes first, such as a new capsule style, and save small tweaks for when you have the traffic.

Can I use this for retention?

Yes. Choose "Players and retained players". Enter the players who started in each variant, and the players who came back on day 1 or day 7 as retained players. Count the same day for both groups, and give the newest players time to reach that day before you compare, or the newer group will look worse.

Is my data sent anywhere?

No. The math runs in your browser, so your visits, wishlists and player counts stay on your device. The tool is free and needs no account, and nothing you type is stored on our servers. The page address keeps your inputs so you can share a result. Anyone who opens that link sees the same numbers, so share it with care.

Sources

Last reviewed . All math runs in your browser. We do not send your inputs anywhere.

Next step

Send the variant name as an event property, and Manalens can filter funnels and retention by it, so each variant gets its own conversion and return rate. See funnels →

Explore more game tools →