A/B Test Significance Calculator

Which test this runs

A two-proportion z-test with a pooled estimate of the common rate, two-tailed. The null hypothesis is that both variants convert at the same underlying rate. Under that hypothesis the best estimate of the shared rate is the pooled figure, total conversions over total visitors, and the standard error of the difference is the square root of p-pooled times (1 minus p-pooled) times the sum of the reciprocals of the two group sizes. The z statistic is the observed difference divided by that standard error, and the p-value is the two-tailed normal tail area beyond it.

Two-tailed matters. It asks whether the variants differ in either direction, which is the honest question when you did not know in advance which way the change would go. A one-tailed test asks only whether B beats A and returns a p-value half the size on identical data. Switching to one-tailed after seeing which variant came out ahead is not a legitimate move, and it is a quiet way that borderline results get pushed over the line. Everything on this page is two-tailed.

The z-test is a normal approximation to a binomial and it needs a reasonable number of successes and failures in each group — a common rule of thumb is at least five of each, preferably more. Below that a Fisher exact test is the correct choice, and this page will say so.

What a p-value is not

It is the probability of data at least this extreme assuming the variants are identical. Read it backwards and you get three claims that are all false and all common:

• p = 0.03, so there is a 97 percent chance B is better. No. The p-value is conditioned on the null being true; it cannot turn around and tell you how likely the null is. Getting to a probability that B is better requires a prior over how large the effect might be, which is a Bayesian calculation this page does not do.
• p = 0.2, so the variants are the same. No. Failing to detect a difference is not the same as establishing there is none. A test with 200 visitors per group fails to detect almost everything. Absence of evidence is weak evidence of absence when the test had little power to begin with.
• p = 0.049 is a result and p = 0.051 is not. These are the same evidence. 0.05 is a convention inherited from a 1920s textbook, not a property of the world. A result at 0.06 with a large effect and a plausible mechanism is more interesting than one at 0.04 with a trivial effect found while looking at forty metrics.

Peeking, and why it wrecks tests

The 5 percent false positive rate is a promise about a test whose sample size was fixed in advance and examined once. Watch a live dashboard and stop as soon as the p-value dips under 0.05, and the false positive rate climbs steeply — with frequent checks over a long run it can pass 20 or 30 percent. The reason is simple: the p-value wanders as data accumulates, and if you take as many looks as you like, you are waiting for a random walk to touch a line it will eventually touch by chance.

This is not a subtle statistical nicety. It is the single most common reason organisations ship changes that do nothing, then find the conversion rate has not moved after a year of wins. The fix is to decide the smallest effect worth acting on, compute the sample size that detects it, run to that number, and evaluate once. If continuous monitoring is genuinely required — and sometimes it is — the honest route is a design built for it, such as an alpha-spending boundary or a sequential Bayesian rule, not a fixed-sample test being watched.

Many variants and many metrics

Test twenty things against one control at the 0.05 level and, if none of them does anything at all, you should expect roughly one to come up significant. That is what the threshold means. The same arithmetic applies to metrics: comparing two variants on conversion, revenue per visitor, bounce rate, time on page, sign-ups and returns is six chances for luck to hand you a headline. Nominate the one primary metric before the test starts, treat everything else as exploratory, and if you genuinely need several comparisons, apply a correction — Bonferroni is crude but it beats pretending the problem is not there.

Small differences need enormous tests

The per-group requirement is roughly 16 times p-pooled(1 minus p-pooled) divided by the squared difference, for 5 percent significance and 80 percent power. Because the difference is squared, the cost of resolving small effects explodes. These are the same figures the calculator reports.

Read that table before designing a test rather than after running one. If your site sees 400 visitors a week and the change might plausibly move conversion by half a point, the test would take a year, and the answer is not to run it — it is to make a bigger change, or to decide on grounds other than a measurement you cannot afford.

Things that break a test before the statistics get involved

• Splitting by time instead of at random. Running A this week and B next week confounds the variant with everything else that changed: the weather, a campaign, the day of the week. Randomise each visitor at the moment of arrival.
• Counting sessions where you mean people. One enthusiastic user across nine sessions is not nine independent observations, and treating them as such shrinks your standard errors dishonestly.
• Novelty, and its opposite. Regular users react to a change because it is a change. An effect that is large in week one and gone by week three was a reaction to novelty, not an improvement.
• Leakage between arms. If the two variants can affect each other through shared inventory, a social feed or a referral loop, the arms are not independent and the test is measuring something other than what you think.

None of these show up in the p-value. Every one of them will produce a clean, confident, wrong number, and they deserve more of your attention than the arithmetic does.

Questions people ask