Maaz Zindani

Loading the index…

↑↓ move↵ openesc close3993 pages

Free tool · Business and sellers

AB test calculator

Enter visitors and conversions for versions A and B to see whether the gap is statistically significant, with the p-value, the relative uplift and a confidence interval. Switch to the sample size mode to work out how many visitors a test needs before it starts.

  • Free
  • No sign-up
  • Runs in your browser

By Maaz Zindani · Updated

What do you want to know?
Version A (control)
Version B (variant)
Confidence level
A/B test result

Not significant at 95%p-value 0.0583 · two-sided z-test · 95% confidence

  • A 5.00%
  • B 5.60%
  • Relative change +12.00%

The gap is not statistically significant at 95% confidence (p = 0.0583). The data cannot rule out chance, which is not the same as showing that A and B are equal: a larger sample could still reveal a real difference.

Conversion rate A
5.00%
Conversion rate B
5.60%
Absolute difference
+0.60 pp
Relative change
+12.00%
z statistic
1.89
p-value
0.0583

95% interval for the difference (B − A): -0.02 to +1.22 percentage points

0-0.021.22

Your numbers: pooled rate = (500 + 560) ÷ (10,000 + 10,000) = 5.3000%; z = (5.600% − 5.000%) ÷ 0.3168% = 1.89; p = 0.0583

Method: fixed-horizon two-sided z-test with pooled variance. Decide the sample size before the test starts, do not stop at the first significant look, and treat several variants as separate tests.

How to use the A/B Test Significance Calculator

  1. Under What do you want to know? choose Is my result significant? to test a finished result, or How many visitors do I need? to plan a test.
  2. For a result, type the Visitors and Conversions of Version A and Version B, then pick a Confidence level.
  3. For a plan, type your Current conversion rate, the Change to detect (relative or in percentage points) and the power. Add your daily traffic to see the number of days.
  4. Read the verdict, the p-value and the interval, then use Your numbers to check each step of the formula.

How do I know if my A/B test result is statistically significant?

A result is statistically significant when its p-value is below the threshold set by the confidence level: 0.05 for 95% confidence. The p-value is the chance of seeing a gap at least this large if A and B truly converted at the same rate. This calculator gets it from a two-sided z-test on two proportions, with the pooled rate used for the standard error.

Take version A with 500 conversions from 10,000 visitors (5.00%) and version B with 560 from 10,000 (5.60%). B is 0.60 percentage points higher, a relative change of +12.0%. The pooled rate is (500 + 560) ÷ 20,000 = 5.30%. The standard error is the square root of 0.053 × 0.947 × (1 ÷ 10,000 + 1 ÷ 10,000), which is 0.3168 percentage points. So z = 0.60 ÷ 0.3168 = 1.89, and the two-sided p-value is 0.0583.

That is above 0.05, so the result is not significant at 95% confidence. It is significant at 90%, because 0.0583 is below 0.10. The calculator words the verdict this way on purpose: "not significant" means the data cannot rule out chance, not that the two versions are equal.

The table shows how the same test reads as B converts more. Version A stays at 500 of 10,000 in every row.

Version B conversions out of 10,000 against A at 500 of 10,000 (95% confidence)
B conversionsRate BRelative changezp-valueVerdict
5205.20%+4.0%0.640.5203Not significant
5405.40%+8.0%1.270.2027Not significant
5605.60%+12.0%1.890.0583Not significant
5805.80%+16.0%2.500.0123Significant
6006.00%+20.0%3.100.0019Significant
6206.20%+24.0%3.690.0002Significant

What does 95% confidence mean in an A/B test?

It means the test is set to call a difference real only when chance alone would produce a gap that large less than 5% of the time. It does not mean there is a 95% chance that B is better. That second reading is a common mix-up: a p-value describes the data under the assumption of no difference, not the probability of either version being the better one.

The confidence interval gives the size of the gap, not just its existence. For the example above, the 95% interval for B minus A runs from −0.02 to +1.22 percentage points. It includes zero, which matches the p-value above 0.05. If the interval had stayed above zero, the result would be significant, and the interval would show how large the gain could plausibly be. A narrow interval around a small gain can be significant and still not worth shipping.

The confidence level is a choice made before looking. 90% accepts a false alarm in one test out of ten, 95% in one out of twenty and 99% in one out of a hundred. A higher level needs a larger gap or more visitors to reach significance, and the sample size mode shows what that costs.

What sample size do I need for an A/B test?

The visitors needed per version depend on four things: the current conversion rate, the smallest change worth detecting, the confidence level and the power. Power is the chance of detecting a change of that size when it is real. This page uses 80% power by default and offers 90%.

The formula is n = (zα/2 × √(2 p̄ q̄) + zpower × √(p₁q₁ + p₂q₂))² ÷ (p₁ − p₂)², rounded up. Here p₁ is the current rate, p₂ the rate after the change, p̄ their mean and q = 1 − p. At 95% confidence zα/2 is 1.960, and at 80% power zpower is 0.842. For a 5% rate and a 20% relative lift (5% to 6%), the result is 8,158 visitors per version, or 16,316 in total. With 1,000 visitors a day split across both versions, that is about 17 days.

Smaller lifts cost far more. Halving the change to detect roughly quadruples the sample, because the gap is squared in the denominator. A lower baseline rate also needs more visitors, since fewer people convert and the rate is noisier. The table lists the visitors per version at 95% confidence and 80% power.

Visitors needed per version, by current rate and relative lift
Current rate10% lift20% lift30% lift
2%80,68221,1099,798
5%31,2348,1583,780
10%14,7513,8411,774
20%6,5101,683772

A relative lift is a share of the current rate: 20% on 5% is 1 percentage point. An absolute change is in percentage points. The sample size mode accepts either, and 5% with a 1-point change gives the same 8,158 as 5% with a 20% relative lift, because both describe 5% to 6%.

Can I stop an A/B test early when it looks significant?

Stopping the moment the p-value dips below 0.05 raises the false-positive rate well above 5%. The calculator assumes a fixed-horizon test: the sample size is chosen in advance, the data are read once when that size is reached, and the verdict applies to that single look. Checking a running test every day and stopping at the first significant reading gives chance many chances to cross the line.

A workable routine is to fix the sample size from the planner, run the test until both versions reach it, and read the result once. Many teams also run for whole weeks, so weekday and weekend visitors are both represented, though the arithmetic does not require it. Methods built for continuous monitoring exist, but they use different formulas from the ones here.

Testing several variants against one control multiplies the chances of a false alarm. Each extra comparison is another roll of the dice, so a single "significant" variant out of five deserves less trust than the same p-value from a two-version test. The calculator treats each pair as a separate test and does not adjust for that.

Which statistical test does this calculator use, and when is it unreliable?

It uses the large-sample z-test for two proportions described in section 7.3.3 of the NIST/SEMATECH e-Handbook of Statistical Methods, with a pooled standard error for the test and an unpooled one for the confidence interval. The p-value and critical values come from the normal distribution functions coded in this page and checked against published table values, so they match standard tables to many decimals.

The normal approximation needs enough events in every cell. When any of the four counts (converted and not converted, in each version) is below 10, the calculator shows a warning, because the p-value becomes rough and an exact test such as Fisher's is the better tool. The test also assumes each visitor appears once, visitors are split at random and the two groups are independent. A test with returning visitors counted twice or an uneven split by device breaks those assumptions.

Statistical significance is not the same as business value. Tie the result to the money it moves: the conversion rate calculator turns rates into visitors and revenue, and the ROI calculator compares the gain with the cost of building the change. Results are estimates from the numbers entered, not advice or a guarantee.

Use it with

Questions

How do I know if my A/B test result is statistically significant?

Compare the p-value with 1 minus the confidence level. With 500 of 10,000 against 560 of 10,000, z is 1.89 and p is 0.0583, so the result is not significant at 95% but is at 90%.

What sample size do I need for an A/B test?

At 95% confidence and 80% power, a 5% conversion rate and a 20% relative lift (5% to 6%) need 8,158 visitors per version, 16,316 in total. A 10% lift on the same rate needs 31,234 per version.

What does 95% confidence mean?

The test calls a difference real only when chance would produce a gap that large less than 5% of the time. It does not mean there is a 95% chance that B is better than A.

How long should I run an A/B test?

Until both versions reach the planned sample size. Divide the total visitors needed by your daily traffic: 16,316 visitors at 1,000 a day is about 17 days.

Can I stop the test early when it looks significant?

Not if you want the stated confidence level to hold. Stopping at the first significant reading raises the false-positive rate above 5%, so fix the sample size first and read the result once.

What is the difference between relative uplift and absolute difference?

Absolute difference is B minus A in percentage points: 5.60% minus 5.00% is 0.60 points. Relative uplift divides that by A: 0.60 ÷ 5.00 = +12.0%.

How this page was made: the tool's results are computed in your browser by the code on this page, using the sources listed above; the explanations were drafted with AI help and checked against the tool's own output.