Chapter 4 of 11
A/B Testing for Ecommerce: Mechanics and Statistics
How A/B tests actually work, how to size one before you start, why peeking at results lies to you, and how to tell a real win from a story.
Contents
- How an A/B test actually works
- Sample size, before you start
- Peeking, and why it lies to you
- Frequentist and Bayesian, in plain English
- Flicker, and why client-side tests lose
- Reading the result
- How stores accidentally fake a win
- More than two variants
- Bandits, and when they are wrong
- What to do with a winner
This is the chapter where the statistics show up. They are less complicated than they are usually made to sound, and getting them wrong is the most common reason a testing programme produces numbers that do not turn into money.
You do not need to derive anything. You need to understand four ideas well enough to not fool yourself.
How an A/B test actually works
You split incoming traffic randomly between the current page (control) and a variant. You track a primary metric for each group. At the end, you ask a specific question:
If the variant were genuinely no different from control, how surprising is the difference I observed?
That is the whole idea. Not “which number is bigger.” Numbers are always different, because random variation exists. Flip two fair coins a thousand times each and you will not get 500 and 500. The question is whether the gap you see is bigger than the gap randomness would produce on its own.
Three requirements make the answer trustworthy:
Random assignment. Every visitor has the same chance of landing in either group, and stays in that group across sessions. Assignment by day of week, or by device, or by anything correlated with intent, invalidates the test completely.
Simultaneous exposure. Both variants run in the same window. Running control in March and variant in April is not a test, it is a comparison of March and April, which differ for a hundred reasons.
One meaningful change. You can change several elements if they serve one hypothesis. What you cannot do is change several unrelated things and expect to attribute the result. That is a redesign wearing a lab coat.
Sample size, before you start
Calculating sample size after the test is running is like deciding the finish line after the race. The number has to exist beforehand, because it is what stops you from stopping early.
Four inputs:
- Baseline conversion rate. What control currently does. Use the last 30 days on the same segment you will test.
- Minimum detectable effect (MDE). The smallest relative lift you would actually act on. This is a business decision, not a statistical one.
- Confidence level. Usually 95%. How willing you are to call a non-effect a win.
- Statistical power. Usually 80%. How willing you are to miss a real effect.
The relationship people underestimate: halving your MDE roughly quadruples the traffic you need. It is not linear. Detecting a 10% lift on a 3% baseline needs around 13,000 visitors per variant. Detecting a 5% lift on the same baseline needs around 50,000.
| Baseline CR | Detect 20% lift | Detect 10% lift | Detect 5% lift |
|---|---|---|---|
| 1% | ~10,700 | ~41,000 | ~160,000 |
| 3% | ~3,400 | ~13,000 | ~51,000 |
| 5% | ~2,000 | ~7,700 | ~30,000 |
Per variant, 95% confidence, 80% power. Round numbers; use a calculator for the real thing.
Look at the 1% row. A store with a 1% conversion rate that wants to detect a 5% lift needs 320,000 visitors across both arms. Most stores reading this do not have that in a quarter. This is the arithmetic behind the “do not test” advice in Chapter 3, and it is not negotiable by wanting it more.
Run your own numbers in our statistical significance calculator. The Plan tab does exactly this: baseline, MDE, confidence, and it returns visitors per variant and an estimated duration from your traffic. If the duration is over six weeks, redesign the test to target a bigger effect, or do not run it.
Peeking, and why it lies to you
Here is the single most expensive mistake in commercial A/B testing.
You start a test. On day three you check. The variant is up 14% and the tool says 91% confident. Exciting. You check again on day five. Up 9%, 88% confident. Day nine: 96% confident. You call it, ship the winner, and move on.
That result is not trustworthy, and the reason is subtle enough that very good operators get it wrong.
A 95% confidence threshold means: if there is genuinely no difference, you will wrongly cross 95% about 5% of the time. That guarantee holds for one look, at a predetermined sample size. Every additional look is another chance to cross the line by accident. Check daily for two weeks and your real false-positive rate is not 5%. Depending on the test it can be 20–30%.
Put plainly: if you watch a null test for long enough, it will eventually show you a significant result, and you will ship it.
Worse, stopping early biases the size of the effect. Tests get stopped when they look good, which means the moment you stop is disproportionately likely to be a random high point. This is why a winner that measured +14% ships and produces +3%, or nothing. The direction was often real. The magnitude was inflated by when you looked.
What to do instead:
- Set sample size and end date before launch. Write them down.
- Do not look at the significance number before then. Look at health metrics only: traffic split holding near 50/50, no console errors, both variants rendering.
- If you genuinely need to monitor continuously, use a tool built for it. Sequential testing and Bayesian methods with proper stopping rules exist precisely for this. What you cannot do is use a fixed-horizon frequentist test and peek at it like it is a live scoreboard.
And run a minimum of two full weeks even if you hit sample size early. Weekday buyers and weekend buyers are different people.
Frequentist and Bayesian, in plain English
Both are in the calculator. Here is what they actually tell you.
Frequentist answers: assuming there is no real difference, how likely is a result at least this extreme? That is the p-value. p = 0.03 means a gap this big would show up 3% of the time by chance alone. It is a statement about the data given a hypothesis.
The output is a yes/no against a threshold, and it does not tell you the probability that your variant is better. It genuinely does not, no matter how often people describe it that way.
Bayesian answers the question people actually want: given what I observed, what is the probability the variant beats control? You get “93% probability B is better,” which is a statement about your belief, and you can also get expected loss, which is the average amount you give up if you pick wrong.
Practical guidance:
- Bayesian’s “probability to beat baseline” is easier to communicate honestly to people who are not going to read a p-value correctly.
- Bayesian does not exempt you from sample size discipline. A Bayesian result on 400 visitors is a confident-sounding number computed from almost no information.
- Expected loss is the genuinely useful output. “94% likely to win, and if we are wrong the average cost is 0.1% of conversion rate” is a decision. A p-value is not.
Pick one before the test, not after you see which one gives the friendlier answer.
Flicker, and why client-side tests lose
A practical problem that invalidates more ecommerce tests than any statistical subtlety.
Most testing tools work client-side: the page loads as control, then JavaScript rewrites it into the variant. If that script runs late, the visitor sees the original page for a moment before it changes. That is flicker, sometimes called flash of original content.
Two consequences, and the second is worse than the first:
It hurts the variant. A visible flash reads as a broken page. Some of those people leave, and they leave only from the variant, which biases the test against whatever you are trying to prove.
It slows both arms. Most tools handle flicker by hiding the page with an anti-flicker snippet until the script resolves. Now every visitor, control included, waits on a blocking script before seeing anything. You have made the site slower to run a test about making it better.
What to do:
- Load the testing script synchronously in the
<head>, before render, and keep it small. - Keep variant changes simple enough to apply instantly. Rewriting a whole page layout client-side is where flicker gets bad.
- For anything structural, use server-side testing or your framework’s own rendering rather than client-side DOM rewriting.
- Measure it. Record the variant on a throttled connection and watch frame by frame. If you can see the change happen, your visitors can.
Check your page speed metrics before and during a test. If largest contentful paint moves materially when the test goes live, the test is measuring your testing tool.
Reading the result
Three outcomes, and the middle one is the one people mishandle.
Win. Reached sample size, crossed the threshold, guardrail metric held. Ship it. Then check it again in four weeks, because some winners are novelty effects that decay once returning visitors stop noticing the change.
Loss. Reached sample size, variant is worse or flat. Keep control. Write down what the hypothesis predicted and mark that branch eliminated. This is a finding, and it is most of your results.
Inconclusive. Reached sample size, no significant difference. This means your test could not detect an effect of the size you specified. It does not mean the variant is equivalent, and it absolutely does not mean “the variant was up 3% so let’s ship it.” If you ship on a non-significant 3%, you have shipped a coin flip and added a change you now have to maintain.
The correct response to inconclusive is to ask whether the MDE was realistic. If you powered for a 5% lift and got nothing, the honest conclusion is usually that the change does not produce a 5% lift, which is a real answer.
Put the final numbers back into the significance calculator before you write up any result. The Analyze tab will give you the confidence interval alongside the verdict, and the interval is the more honest object: “+4%, somewhere between −1% and +9%” tells the story that “+4%, not significant” hides.
Check every result by segment before you celebrate. A win that is driven entirely by desktop while mobile is flat or down is not a win, it is two results averaged together into one misleading number. Always split by device, and by new against returning.
How stores accidentally fake a win
Six ways we see regularly. Every one of them produces a number somebody reports to a board.
1. Stopping when it looks good. Covered above. The most common by a distance.
2. Changing the primary metric after the fact. Add-to-cart was flat but checkout starts were up, so the test is reported on checkout starts. If the metric was not named before launch, it is not a result.
3. Ignoring the guardrail. Conversion up 8%, average order value down 11%, reported as a win because nobody looked at revenue per session. The opposite is also possible and worth having: Erth Wellness moved both upward, which you only know because both were being watched.
4. Running until it wins. Extending the test past the planned end date because it is “nearly there.” This is peeking with extra steps.
5. Testing during an abnormal period. A test spanning Black Friday measures Black Friday. So does a test spanning a stockout, a PR spike, or a two-day tracking outage. Note anomalies during the run and discard tests that overlap them.
6. Sample ratio mismatch. The split should be near 50/50. If you end at 54/46, something is wrong with assignment or with which sessions got counted, and the result is not usable. Every serious tool reports this. Check it before you read anything else.
The uncomfortable summary: most published ecommerce case studies fail at least one of these six, which is why the lifts in case studies are so much larger than the lifts real programmes produce.
More than two variants
Two related things get confused, and they have very different costs.
A/B/n testing is one change with several versions. Three headlines, four hero images. Each additional arm splits your traffic further, so three arms need roughly 50% more total traffic than two, and the multiple-comparisons problem means your effective false-positive rate rises with each arm unless the tool corrects for it. Most do not by default.
Practical rule: two arms unless you have a specific reason and the traffic to pay for it. If you must run three, apply a correction and accept the longer run.
Multivariate testing is several changes tested simultaneously to find interaction effects. Two headlines by two images by two button labels is eight combinations. Traffic requirements become absurd very quickly, and the results are hard to act on.
We have run multivariate tests roughly twice in the last several years, both on stores with eight-figure traffic, and in both cases the interaction effects were smaller than the main effects, which is the usual finding. For nearly every store reading this, multivariate testing is a way of not concluding anything for a quarter.
If you want to change several things at once and you do not have the traffic to separate them, that is fine. Change them all, ship them, and accept that you have made a business decision rather than run an experiment. Just do not describe it afterwards as a test.
Bandits, and when they are wrong
Multi-armed bandit algorithms shift traffic toward the better-performing variant while the test runs, instead of holding a fixed split. They are sold as strictly better than A/B tests. They are not; they optimise for a different thing.
A bandit maximises revenue during the experiment. An A/B test maximises what you learn from it. Those goals conflict, because learning requires continuing to send traffic to the arm that currently looks worse.
Bandits genuinely fit short-lived decisions where learning has no future value: which of five creatives to show during a two-week promotion, which homepage banner during a sale. The campaign ends, the knowledge expires, so exploitation is the right call.
Bandits fit badly where you want a durable answer, which is nearly all of the work in this course. If you want to know whether fit guidance matters to your customers, you want a clean estimate, not a system that stopped sending traffic to the loser on day three.
What to do with a winner
Ship it to 100%. Remove the testing code rather than leaving a permanently 100%-allocated experiment running, because those accumulate and slow the site.
Record it: hypothesis, dates, sample size, result with confidence interval, guardrail outcome, segments. Six months from now this record is the most valuable thing your programme owns.
Then look at what the win implies. A win on size guidance for one product category is a hypothesis about every other category. Winners generate the next round of hypotheses more reliably than brainstorms do.
Next: the product page, where most of these tests will end up pointing. If you want the raw ideas first, the product page test list is organised by impact and difficulty, and Chapter 5 explains which of them matter and why.
If you want this run properly on your store, with the sample size maths done before anything gets built, that is what we do.
What founders ask before they book.
How long should an ecommerce A/B test run?
Long enough to reach your pre-calculated sample size, and never fewer than two full weeks regardless of what the sample size says. Two weeks covers both weekend and weekday behaviour and a full pay cycle for most audiences. If you hit sample size in four days, keep running to fourteen. If fourteen days is not enough to reach sample size, the test was underpowered before you started.
What sample size do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you would act on. As a rough anchor, detecting a 5% relative lift on a 3% baseline at 95% confidence needs roughly 50,000 visitors per variant. Detecting a 10% relative lift on the same baseline needs roughly 13,000 per variant. Calculate it before you build the variant, not after.
Is 95% confidence necessary, or can I use 90%?
90% is defensible for cheap, reversible changes where being wrong costs little. Keep 95% for anything expensive to build or hard to roll back. What you cannot do is start at 95%, fail to reach it, and then decide 90% was fine all along. Pick the threshold before the test runs.
Why did my winning test not produce the revenue it predicted?
Three common reasons. The test measured a step metric like add-to-cart rather than revenue per session. The observed effect was inflated by stopping early, so the true effect is smaller than the measured one. Or the win was real but small, and normal week-to-week variance in the business is larger than the lift, which makes it invisible in the top-line number even though it is genuinely there.
Can I run multiple A/B tests at the same time?
Yes, if they are on different pages or different funnel steps, and you accept some interaction risk. Do not run two tests on the same element or the same page simultaneously. If traffic is tight enough that you are tempted to overlap tests on one page to save time, that is a signal you do not have the traffic to test that page at all.