Convert Lane

Free tool

A/B test significance calculator

Work out how long a test needs to run, then check whether the result you got is real. Both answers in plain English, no statistics degree required.

Control (A)

4.00% conversion rate

Variant B

4.60% conversion rate

Variant B vs Control

Significant
Control rate4.00%
Variant rate4.60%
Relative lift+15.0%
p-value0.0220
Confidence97.8%
95% interval on difference0.09% to 1.11%

Variant B is beating control by 15.0% relative. At 95.0% confidence this is a real difference, not noise. Ship it.

Stay sharp on ecommerce CRO

3 short examples, 2 CRO tips, 1 teardown. Every Tuesday.

What this calculator does

Two jobs, on two tabs. Plan takes your current conversion rate, the lift you want to be able to detect, and your daily traffic, and returns the visitors per arm and the number of days you need. Analyze takes the finished numbers and tells you whether the difference between them is real.

It is for anyone running tests on a store: founders deciding whether a test is worth starting, developers being asked to ship a variant, and agencies who need to show their working. If you have ever been handed a screenshot of a testing tool showing "94% confidence" and not known what to do with it, this is the page that answers it.

The maths, in the open

Nothing here is proprietary. It is the standard two-proportion test, the standard sample size formula, and a Beta-Binomial posterior. All of it runs in your browser; nothing you type is sent anywhere.

Sample size (Plan tab)

p2 = p1 × (1 + MDE)
n per arm = (z_confidence + z_power)² × [ p1 × (1 − p1) + p2 × (1 − p2) ] ÷ (p2 − p1)²

p1 is your current conversion rate, MDE is the relative lift you want to catch, and the two z values come from the confidence and power you pick. At 95% confidence z_confidence is 1.960; at 80% power z_power is 0.842. Days to run is total visitors needed divided by your daily traffic, rounded up.

Significance (Analyze tab)

pooled = (conversions_A + conversions_B) ÷ (visitors_A + visitors_B)
SE = sqrt( pooled × (1 − pooled) × (1 ÷ visitors_A + 1 ÷ visitors_B) )
z = (rate_B − rate_A) ÷ SE
p-value = 2 × (1 − Φ(|z|))

Φ is the standard normal cumulative distribution. The confidence interval on the difference uses the unpooled standard error instead; pooled is the right choice for the test, unpooled for estimating the size of the gap.

The Bayesian read

posterior = Beta(1 + conversions, 1 + visitors − conversions)
P(B beats A) = share of 20,000 paired draws where B's draw is higher

The Beta(1,1) prior means no opinion before the data, which is the honest default when nobody agreed a prior in advance. "Expected loss" is the average amount of conversion rate you give up in the draws where B turned out to be worse: the cost of being wrong.

A worked example

These are the numbers the Analyze tab loads with, so you can check every figure below against the tool itself.

Arm Visitors Conversions Rate
Control (A) 12,000 480 4.00%
Variant (B) 12,000 552 4.60%

Pooled rate is 4.30%, so the standard error is 0.262 percentage points and z is 2.29. That gives a two-tailed p-value of 0.0220 and a confidence of 97.8%, past a 95% bar. The relative lift is +15.0%, and the 95% interval on the absolute difference runs from +0.09 to +1.11 percentage points.

On the Plan tab: a 4% baseline, a 10% relative lift, 95% confidence and 80% power needs 39,473 visitors per arm, or 78,946 in total. At 1,500 visitors a day that is 53 days, so you book eight full weeks.

How to read the result

Significant is not the same as big. In the example the true effect could be anywhere from +0.09 to +1.11 percentage points, somewhere between a rounding error and a genuinely good quarter. Plan the next decision around the bottom of the interval, not the headline lift.

"Not yet" means keep going, not "it lost." An inconclusive test at the planned sample size means the effect, if any, is smaller than the one you sized for. That is a result. Write it down and move on.

Check the direction before the maths. If the variant is behind, no amount of extra traffic makes it a winner. Kill it and free the traffic for the next test.

What it will not do

Non-binary metrics. Revenue per visitor, average order value and items per order are not conversion rates. Their long tails break the assumptions here and need a different test.

Very small counts. The normal approximation behind the z-test needs a reasonable number of conversions in each arm. The usual rule is at least ten before you trust the p-value.

Sequential testing. The sample size assumes a fixed horizon: you decide the number, then look once. If you intend to peek and stop early on purpose, you need a sequential method, not this one.

Validity problems. A broken split, overlapping tests, bot traffic and novelty effects all produce numbers this calculator will happily process. It checks the arithmetic, not the experiment.

The mistakes this calculator is trying to prevent

Most failed test programmes do not fail because the maths was wrong. They fail for three reasons, and all three are visible before you start.

1. Calling it early

Significance is not a finish line you can watch approach. Every extra peek is another chance to catch a random high point. Set the sample size first (the Plan tab does this) and hold yourself to it.

2. Testing something too small to see

A button colour on a store doing 40 orders a week is unmeasurable. The Plan tab will tell you it needs six months. Believe it, and go test the offer instead.

3. Measuring the wrong step

If your change affects the product page, measure add-to-cart, not purchase. Every step between your change and your metric adds noise you have to pay for in traffic.

The full version of the reasoning behind this tool, from sample size and peeking to frequentist against Bayesian and the six ways stores accidentally report a win, is Chapter 4 of our CRO course. It is free and there is no gate. If the Plan tab says your test cannot conclude, Chapter 3 covers what to do instead, and the CRO ROI calculator puts a number on it.

All three calculators, live on one page → Significance, programme ROI and the free-shipping threshold, with what to do about each.

What founders ask before they book.

What does statistical significance actually mean here?

It is the chance that the difference you are seeing is not just random noise. At 95% confidence, roughly 1 in 20 "winners" will be a fluke. That is an acceptable rate for most ecommerce decisions. It is not acceptable for anything you cannot reverse.

Frequentist or Bayesian: which should I use?

Frequentist answers "how surprising would this result be if the change did nothing?" Bayesian answers "what is the chance this variant is genuinely better?" The second is the question you actually have. Use Bayesian for shipping decisions, frequentist when a client or a stakeholder expects a p-value.

Can I stop the test as soon as it hits 95%?

No, and this is the most expensive mistake in CRO. If you check every day and stop the first time it crosses the line, your real false-positive rate is far above 5%. Decide the sample size before you start, then run it out.

Why does it tell me to run in whole weeks?

Traffic on a Tuesday does not behave like traffic on a Saturday. Stopping mid-week over-weights whichever days you happened to catch. Whole weeks keep the mix honest.

My test needs 200,000 visitors. What now?

That is the calculator telling you the truth: the effect you are chasing is too small to measure at your traffic. Test something bolder, move the test higher in the funnel where volume is greater, or pick a metric closer to the change (add-to-cart rather than purchase).

Does this handle revenue or AOV, not just conversion rate?

Not yet. This calculator is built for binary outcomes (converted or did not). Revenue per visitor has a long tail and needs a different test. Ask us and we will run it for you.

Book A Call

Want someone to run the tests?

We build the hypothesis, ship the variant, and call the result honestly, including when it lost. Free 20-minute call.

  • 437+ stores built
  • 2,000+ tests shipped
  • No call needed to start

Claim your free audit of one page

Send us one page of your store and we’ll review it on video: what’s leaking, what to fix first, and what to leave alone. Changes you can ship this week to lift CVR, AOV and revenue per visitor.

Get My Free Audit