What can a small studio A/B test?

Test changes inside the game first, because there you control who sees what. Good candidates are the tutorial, the first level, difficulty, the order of early rewards and the main menu. Measure a clear rate for each, such as tutorial completion or D1 retention.

Store tests are harder. You can send two tagged links to two audiences and compare wishlist rates in the Steam UTM report, but the audiences are not random. A difference may come from the audience, not from the capsule.

Test big changes. Small teams rarely have the players to detect a 1 point change, as the sample sizes below show.

Write the plan down before you start: the metric, the smallest lift you care about, the players per variant and the end date.

Sources: Manalens A/B test significance calculator.

How do I know if a result is significant?

Run a two-proportion z-test and read the p-value and the confidence interval together. The p-value is the chance of a difference this large if the two variants were the same. The interval shows how big the real difference could be. If the interval includes zero, the test has not shown a difference.

Penn State notes that this normal method needs at least 10 successes and 10 failures in each group. With fewer, use an exact method.

Here is our own example. Variant A has 1,000 new players and 120 return on day 1 (12%). Variant B has 1,000 players and 150 return (15%). The z score is 1.96 and the two-sided p-value is about 0.0496. The 95% interval for the lift runs from 0.01 to 6.0 points. That is barely significant, and the true lift could be tiny.

Sources: Penn State STAT 200, 9.1; Manalens A/B test significance calculator.

How many players do I need?

Decide the smallest lift that would change your decision, then compute the players per variant before you start. The formula from Penn State STAT 509 uses the two rates, the confidence level and the power.

The table is our own arithmetic at 95% confidence and 80% power. Low rates and small lifts need far more players. The 6.5% row is close to the median PC D1 of 6.54% in the GameAnalytics Benchmarks Hub.

Rate in ARate in B you want to detectPlayers per variant (our arithmetic)
2%2.2%80,683
6.5%8%4,692
10%12%3,843
10%15%687
20%25%1,095

Sources: Penn State STAT 509, 6a.7; GameAnalytics Benchmarks Hub (October 2026).

Why should I not stop a test early?

Because each look at the data is another chance for luck to cross the line. Evan Miller calls this repeated significance testing. The significance numbers assume that the sample size was fixed in advance.

In his worst case, with a check after every observation, a test of two identical variants found a significant result 26.1% of the time instead of 5%. If you peek ten times, what looks like 1% significance is really 5%. If you must peek, use a stricter threshold, as in his table below.

Times you peekedReported significance you need for 5% actual
12.9%
22.2%
31.8%
51.4%

Sources: Evan Miller, How Not To Run an A/B Test.

What else makes a result look real when it is not?

Bad splits and too many metrics. Check the items below before you ship a winner.

  • Many metrics: if you test 20 metrics at 0.05, expect about one false winner by chance alone (our arithmetic).
  • Uneven groups: if the split is not close to 50/50, the assignment code may have a bug.
  • Players who switch groups: store the variant in the save file so a player keeps it.
  • Test runs: turn the SDK off in the editor so your own runs do not enter a group.
  • Mixed builds or platforms: compare variants inside one build and one storefront.
  • Novelty: a new menu can lift day 1 and fade by day 7, so read the later days too.

Sources: Evan Miller, How Not To Run an A/B Test; Manalens Godot and Unity SDK documentation.

How do I run a test with Manalens?

Assign the variant in your own code, send it as an event property, then filter or break down your funnel by it. Manalens has no feature flags or experiment assignment, so the split lives in your game.

Send an exposure event when the player first meets the change, and add the variant to the events that measure the result. In a funnel, use the event property key variant as the breakdown. For retention, save a segment with the filter variant is b and pick it in the retention chart.

GDScript

# Godot: pick a variant once and keep it in the save file
if not save.has("tutorial_variant"):
    save["tutorial_variant"] = "b" if randi() % 2 == 1 else "a"
var variant: String = save["tutorial_variant"]
Manalens.track("experiment_exposure", {"experiment": "tutorial_v2", "variant": variant})

# Later, on the result event
Manalens.track("tutorial_complete", {"variant": variant})

Sources: Manalens funnels, retention and filters documentation; Godot SDK documentation.

How do I compare two Steam store assets?

Use tagged links and the Steam UTM report, and treat the result as a hint, not a clean test. Send link A and link B to two audiences that are as similar as you can make them, for example two posts on the same channel a week apart. Compare wishlists per tracked visit for each link.

Steam counts wishlists and purchases within 72 hours of a tracked visit, and only visitors who are logged in to Steam in that browser can convert. Steam also hides UTM combinations with few visits. So you need more visits than you think, and the two groups are never randomly split.

If one asset wins clearly on several audiences in a row, the result is more likely to be real than one close win.

Sources: Manalens Steam documentation; Steamworks UTM Analytics documentation.

How do I check my own test?

Manalens is the analytics and growth engine built for games. Send the variant with your events, read the funnel or retention for each group, and paste the counts into the significance calculator before you decide.

Sources: Manalens product documentation.

Frequently asked questions

What p-value should a game test use?

Use 0.05 for most decisions, which is 95% confidence. Use a stricter value, such as 0.01, when a wrong call is expensive or when you plan to look at the data several times. Choose the threshold before the test starts, not after you see the numbers.

Can I A/B test with a few hundred players?

Only for large changes. In our arithmetic, a lift from 10% to 15% needs about 687 players per variant at 95% confidence and 80% power. A lift from 10% to 12% needs about 3,843. With a few hundred players, test bold changes or use playtests.

Does Manalens split players for me?

No. It has no feature flags or experiment assignment. You pick the variant in your game code, store it, and send it as an event property. Then funnels, breakdowns and segments show the result for each variant, and the calculator checks if the difference is real.

Is a significant result always worth shipping?

No. A significant result can still be a tiny lift. Read the confidence interval: if it runs from almost zero to a few points, the gain may not pay for the work. Also check later metrics, such as D7, so a short novelty effect does not fool you.

Can I use the calculator for retention tests?

Yes. Use the players who started in each variant as the visitors, and the players who came back on day 1 or day 7 as the conversions. Make sure each group had the same number of days to return, or the newer group will look worse for no real reason.

Sources and implementation