# A/B test significance calculator for games

> Check if a new capsule, trailer thumbnail, ad creative or tutorial made a real difference. Get a p-value, an interval and the sample size per variant. Free.

Canonical URL: https://manalens.dev/tools/ab-test-significance-calculator

A difference between two variants is significant at 95% confidence when the two-sided p-value is below 0.05. Enter visits and wishlists, or players and retained players, for a capsule, trailer thumbnail, ad creative or tutorial test. The calculator gives a two-proportion z-test p-value, a confidence interval for the difference and the sample size you need per variant.

## How to use this tool

The interactive tool runs in the browser at https://manalens.dev/tools/ab-test-significance-calculator. This section describes it for readers who cannot run it.

### Inputs

- What you measure
- Confidence level
- Visits in A
- Wishlists in A
- Visits in B
- Wishlists in B
- Smallest lift you care about (%)
- Power

### Outputs

- Rate A and rate B
- Lift of B over A
- p-value (two-sided)
- 95% interval for B - A (points)
- Result
- Visits needed per variant

### Worked example

These are the default inputs of the tool. The result below was computed by the same code that runs on the page.

| Input | Value |
| --- | --- |
| What you measure | Visits and wishlists |
| Confidence level | 95% |
| Visits in A | 4,000 |
| Wishlists in A | 320 |
| Visits in B | 4,000 |
| Wishlists in B | 380 |
| Smallest lift you care about (%) | 10 |
| Power | 80% |

| Result | Value |
| --- | --- |
| Rate A and rate B | 8%, 9.5% |
| Lift of B over A | 18.75% |
| p-value (two-sided) | 0.0176 |
| 95% interval for B - A (points) | 0.26 to 2.74 |
| Result | Significant at 95% |
| Visits needed per variant | 18,873 |

z = 2.374.

Open the tool with these inputs: https://manalens.dev/tools/ab-test-significance-calculator

## How to use it

1. Pick "What you measure": visits and wishlists, or players and retained players.
2. Pick the "Confidence level": 90%, 95% or 99%.
3. Enter the visits or players and the wishlists or retained players for A and for B.
4. Enter the "Smallest lift you care about (%)" and pick the "Power" for the sample size.
5. Read the p-value, the interval and the result, and check the sample size needed per variant before you stop the test. The page address keeps your inputs.

## How it works

Each rate is conversions divided by visitors. The test is the pooled two-proportion z-test from the NIST/SEMATECH handbook: the pooled rate is all conversions divided by all visitors, and z is the difference in rates divided by the pooled standard error. The p-value is two-sided, and the result is significant when it is below 1 minus the confidence level.

The interval for B minus A uses the unpooled standard error from Penn State STAT 200. The normal method is reliable when each group has at least 10 conversions and 10 non-conversions, and the tool warns you when a group has fewer.

The sample size per variant is the Penn State STAT 509 formula with equal groups. p1 is the rate of A, and p2 is the rate of A raised by the smallest lift you care about, as a relative lift.

```
rate = conversions ÷ visitors
pooled = (conversions A + conversions B) ÷ (visitors A + visitors B)
z = (rate B - rate A) ÷ √(pooled × (1 - pooled) × (1/visitors A + 1/visitors B))
interval = (rate B - rate A) ± z* × √(rate A (1 - rate A)/visitors A + rate B (1 - rate B)/visitors B)
n per variant = 2 × (z(1 - α/2) + z(power))² × p̄ (1 - p̄) ÷ (p1 - p2)², p̄ = (p1 + p2) ÷ 2
```

## Worked example

### Example with made-up inputs: two capsule ad creatives, 5,000 store visits each

Creative A brought 300 wishlists from 5,000 tagged store visits, 6%. Creative B brought 360, 7.2%. The lift of B over A is 20%, and z = 2.417.

The two-sided p-value is 0.0157, below 0.05, so the result is significant at 95%. The 95% interval for B minus A runs from 0.23 to 2.17 percentage points, so the real gain could be small. A test planned to detect a 10% lift over 6% with 80% power would need 25,741 visits per variant. This test reached significance because the lift was larger, but with fewer visits its interval stays wide.

## How to read the result

Significant means a difference this large would be unlikely if A and B were the same. It does not mean the lift is as large as measured. Read the interval: its low end is the smallest gain that fits the data at that confidence.

Decide the sample size before you start, and do not stop the test the first time it looks significant. Checking often and stopping early makes false wins more likely. Our practice: run a store page or ad test for at least one full week, so weekdays and the weekend are both in it.

The p-value uses the pooled error and the interval uses the unpooled error, so near the line they can disagree: a p-value just under the cutoff with an interval that touches 0, or the reverse. When that happens, treat the result as unclear and collect more data.

## How to split traffic for a game test

This calculator does not split traffic, so you split it yourself. To compare capsule art or a trailer thumbnail, run them as two ad creatives or two posts that link to your Steam page with different utm_content, at the same time and to similar audiences. For a store page change, compare the same link before and after only with care, because traffic changes week to week. For a tutorial or onboarding change, give each new player a variant in the game and count who comes back on day 1 or day 7.

## Sample size and test length

Small lifts on low rates need many visitors. At a 6% wishlist rate, detecting a 10% relative lift at 95% confidence and 80% power needs 25,741 visits per variant. A 50% lift on the same rate needs 1,211. Divide the visits you need by the visits each variant gets per day to see how many days the test must run. If that is months, test a bigger change.

## Limits

The calculator compares two variants. With three or more, each extra comparison raises the chance of a false win, and this tool does not correct for that.

It uses a fixed sample, not a sequential test. Its p-value holds only if you stop at the sample size you planned.

It does not split traffic or collect data. The Steamworks UTM report counts only visits that load the store page with your tags and leaves out rows with few visits, so your counts can be lower than your ad clicks.

The smallest lift is relative: 10% on a 6% rate means 6.6%, not 16%.

## Which tool should I use?

Use this calculator to compare two variants. Use the conversion rate calculator for a single rate, and the retention calculator for a full retention curve.

- [Conversion rate calculator](https://manalens.dev/tools/conversion-rate-calculator)
- [D1, D7, D30 retention calculator](https://manalens.dev/tools/retention-calculator)

## Frequently asked questions

**How do I know if my A/B test result is significant?**

Compare the p-value with 1 minus your confidence level. At 95% confidence, a p-value below 0.05 is significant. Check the interval too: when it stays above zero, B likely beats A. Make sure each group has at least 10 conversions and 10 non-conversions, and that you reached the sample size you planned before you started.

**What does the p-value mean?**

It is the chance of seeing a difference at least this large if the two variants were really the same. A small p-value means the difference is unlikely to be luck alone. It does not tell you how large the real difference is, or the chance that B is better. The confidence interval shows the likely size of the difference.

**How do I A/B test a Steam capsule or store page?**

Split the traffic yourself. Make two tagged Steam links that differ only in utm_content, and use each one with a different capsule creative or post, shown to similar audiences at the same time. Read visits and wishlists for each link in the Steamworks UTM report, then enter them here as A and B.

**Why do I need so many players?**

Small lifts on low rates need large samples. At a 6% wishlist rate, detecting a 10% relative lift at 95% confidence and 80% power needs about 25,741 visits per variant. A 50% lift on the same rate needs about 1,211. Test big changes first, such as a new capsule style, and save small tweaks for when you have the traffic.

**Can I use this for retention?**

Yes. Choose "Players and retained players". Enter the players who started in each variant, and the players who came back on day 1 or day 7 as retained players. Count the same day for both groups, and give the newest players time to reach that day before you compare, or the newer group will look worse.

**Is my data sent anywhere?**

No. The math runs in your browser, so your visits, wishlists and player counts stay on your device. The tool is free and needs no account, and nothing you type is stored on our servers. The page address keeps your inputs so you can share a result. Anyone who opens that link sees the same numbers, so share it with care.

## Sources

- [NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3: comparing two proportions](https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm), read 2026-11-19
- [Penn State STAT 200, 9.1: two independent proportions (interval and the 10 successes rule)](https://online.stat.psu.edu/stat200/book/export/html/193), read 2026-11-19
- [Penn State STAT 509, 6a.7: sample size for two proportions](https://online.stat.psu.edu/stat509/book/export/html/667), read 2026-11-19
- [Valve, Steamworks documentation: UTM analytics. Checked 18 November 2026.](https://partner.steamgames.com/doc/marketing/utm_analytics), read 2026-11-19

Last reviewed 2026-11-19. All math runs in your browser. We do not send your inputs anywhere.

## Related guides, tools and terms

### Guides

- [A/B testing in games: when is a result real?](https://manalens.dev/blog/ab-testing-games-statistical-significance)
- [How to A/B test a Steam store page with UTM links](https://manalens.dev/blog/ab-test-steam-store-page)
- [Steam capsule art guide: sizes, rules and tests](https://manalens.dev/blog/steam-capsule-art-guide)

### Tools

- [Conversion rate calculator](https://manalens.dev/tools/conversion-rate-calculator)
- [D1, D7, D30 retention calculator](https://manalens.dev/tools/retention-calculator)
- [Steam capsule preview](https://manalens.dev/tools/steam-capsule-preview)

### Glossary

- [What is conversion rate in game analytics?](https://manalens.dev/glossary/conversion-rate)
- [What is CTR (click-through rate)?](https://manalens.dev/glossary/ctr)
- [What is a cohort in game analytics?](https://manalens.dev/glossary/cohort)

## Next step

Send the variant name as an event property, and Manalens can filter funnels and retention by it, so each variant gets its own conversion and return rate. [See funnels](https://manalens.dev/features/funnels)

## Guides that use this tool

- [Cohort analysis for games](https://manalens.dev/blog/cohort-analysis-for-games)
- [Measure the impact of updates and live ops on players](https://manalens.dev/blog/measure-game-update-impact)
- [Wishlist capture page: does an email or Discord step cost you wishlists?](https://manalens.dev/blog/wishlist-landing-page-conversion)
- [In-game feedback surveys: join survey answers to player data](https://manalens.dev/blog/in-game-feedback-survey-player-data)
- [Test your game's hook before you build: ads and pages](https://manalens.dev/blog/test-game-hook-before-you-build)
- [Cozy game sound and music: what players notice](https://manalens.dev/blog/cozy-game-sound-design)
- [A weekly metrics review for your game after launch](https://manalens.dev/blog/weekly-game-metrics-review)
- [How to describe your game: the pitch, the tagline and the one-sentence description](https://manalens.dev/blog/how-to-describe-your-game)

[Explore more game tools](https://manalens.dev/tools)

Interactive tool: https://manalens.dev/tools/ab-test-significance-calculator
