Chapter 11 of 11
Experimentation Strategy
How to run a testing programme rather than a pile of tests: backlog design, why ICE and PIE scores are opinions wearing numbers, and what to do with a loss.
Contents
- Why a programme beats a pile of tests
- Building a backlog that compounds
- ICE, PIE, and scores that are opinions wearing numbers
- Sizing the programme to the traffic you actually have
- What to do with a loss
- What to do with an inconclusive result
- Documenting learnings so year two is faster than year one
- Reporting that a CFO can read
- Scaling a winner across the store
- How testing programmes die
- When to stop testing an area
Most stores that “do CRO” are running a pile of tests. A pile is not a programme, and the difference shows up in the second year, when the pile has produced a folder of screenshots and the programme has produced a map of what your customers actually respond to.
This chapter is about the second thing. How to build a backlog that gets more valuable as it grows, how to prioritize without pretending your guesses are data, how to size the whole operation to the traffic you have rather than the traffic in a case study, and what to do on the four days out of five when the test does not win.
It assumes you have already read how to turn research findings into hypotheses and the mechanics and statistics of running a single test. This is the layer above both: the operating system rather than the individual test.
Why a programme beats a pile of tests
A pile of tests answers one question at a time and forgets each answer as soon as the next test starts.
A programme answers a question, writes down what the answer eliminated, and uses that to make the next question better. The compounding is not in the winners. It is in the eliminations.
Here is the practical difference. Two stores each run twenty tests in a year and each get four winners, which is a normal outcome. The pile store has four shipped changes and sixteen forgotten failures, and next January somebody proposes a test they already ran. The programme store has four shipped changes and a documented answer to sixteen questions about its customers, which makes its year-two backlog sharper than its year-one backlog was.
Same traffic. Same win rate. Wildly different asset at the end.
Three things separate them, and none of them are tooling:
A written record of what you believed before each test. Without it, a loss is noise. With it, a loss removes a branch.
A prioritization rule that survives contact with the loudest person in the room. Otherwise the backlog is just a queue of whoever asked most recently.
An honest ceiling on throughput. A programme that runs more tests than its traffic can conclude produces a stream of inconclusive results and calls them wins.
Building a backlog that compounds
Most teams treat the backlog as a to-do list that gets shorter. It should get longer, and it should get better as it gets longer.
Every entry carries seven fields. Each one stops a specific failure you will otherwise hit.
The hypothesis, in full. The four-part version from the hypotheses chapter. Not a summary. The summary is the version that turns a loss into nothing.
The evidence, with its source. “340 reviews mention fit” is evidence. “The founder thinks the page feels cluttered” is a hypothesis about the founder, not about the customer. Both can go in the backlog. Only one of them gets to claim evidence.
The affected step and its size. What percentage of sessions reach the thing you are changing. This is the number that does most of your prioritization work, and it comes free from your conversion research.
The primary metric and its guardrail. Named before the test exists, so nobody can pick the metric after seeing the data.
The required sample size and the weeks it implies. Calculated at the moment the idea enters the backlog, not the week you plan to build it. Half your ideas die here, which is the cheapest place for them to die.
Cost to build, and cost to be wrong. A headline change and a checkout re-architecture are not the same bet even when they score the same.
After it runs: the result, and what the result eliminated. That last clause is the whole asset. A backlog with results attached is the only defence against re-running a test somebody ran two years ago.
One more field is worth adding once you have run a dozen tests: the branch this belongs to. Trust signals. Mobile information hierarchy. Cost transparency. Delivery certainty. When four tests in a branch all come back flat, you learn something about the branch that no individual test told you, and you can retire the whole thing.
ICE, PIE, and scores that are opinions wearing numbers
You will be told to score every idea and sort by the product. ICE gives you Impact, Confidence and Ease. PIE gives you Potential, Importance and Ease. There are a dozen others and they all work the same way.
The mechanic is fine. The inputs are the problem.
Look at what you are actually doing. You assign an 8 to Impact for something whose impact is unknowable; if you knew the impact, you would not need the test. You assign a 6 to Confidence, which is a number describing how confident you feel, scored by the person who wants to run it. Then you multiply two invented numbers by a third that is at least real (Ease, which an engineer can estimate) and treat the product as a decision.
The output looks like arithmetic. It is a vote with decimal places.
This matters more than it sounds, because the format launders the bias. Nobody in a meeting argues with 8 × 6 × 7 = 336. They argue with “I think the hero matters more than the size chart”, which is the same claim, honestly stated, and a much better conversation.
Two real failure modes follow:
Confidence inflates for the ideas people like. The score does not neutralise politics, it hides it. The person scoring their own idea gives it a 9, and now the politics is inside a spreadsheet where it looks objective.
Ease dominates the ranking. Ease is the only input anyone can estimate, so it is the only one with real variance, so cheap tests float to the top. You end up with a year of button and headline tests, all shipped fast, all inside the noise floor.
Here is what we sort on instead, in order:
1. Size of the affected step. A change on a page 68% of sessions see beats a brilliant change on a page 4% see. Not a guess. Does most of the work.
2. Strength of evidence. Rank it honestly. Multiple independent sources pointing at the same thing outranks one recording that stuck in your memory.
3. Cost to build and cost to be wrong. When two candidates look similar, run the reversible one first.
4. Whether the test can conclude at all. Which disqualifies more candidates than the other three combined, and is the next section.
Use ICE if it helps two people surface a disagreement. Then throw the number away and make the decision out loud.
Sizing the programme to the traffic you actually have
This is arithmetic, and it is the part that decides whether you have a testing programme or a testing hobby.
Start from one number: how many concluded tests per year can this store support on the surface that matters?
Take monthly sessions on that surface. Take the sample size each test needs at a minimum detectable effect you would actually act on. Divide. That is your ceiling, and you cannot negotiate with it.
Work an illustrative example. Northmoor Supply is a made-up store, used here only to show the shape of the calculation. The numbers are arithmetic, not a benchmark. It does 60,000 monthly sessions, 70% of which touch a product page, at a 2.5% conversion rate. To detect a 10% relative lift the arithmetic lands in the region of five figures of visitors per variant, which on 42,000 monthly product-page sessions means roughly four to six weeks per test, one test at a time on that surface.
Call it eight to ten concluded tests a year. At a one-in-five win rate, that is one or two winners a year on the product page.
That is the honest number, and it changes the strategy completely. With eight slots a year you do not spend one on a button colour. You spend them on the largest, best-evidenced, most reversible bets you have, and you ship everything else on judgement without a control group.
Run your own version before you plan anything. Our statistical significance calculator has a planning mode that turns a baseline rate and a minimum detectable effect into visitors and weeks. Do that for your top five backlog items in one sitting. It is the fastest hour in this discipline.
Three consequences worth stating plainly:
Below roughly 40,000 monthly sessions, the programme is mostly research plus decisive shipping. Tests are reserved for the two or three changes a year that are big, expensive and reversible. That is the correct version of the work at that size, not a lesser one.
Never run two tests on the same metric on the same page at once. Traffic scarcity makes people overlap tests to feel productive. The result is two uninterpretable answers instead of one clean one.
Raise the minimum detectable effect rather than lowering the confidence threshold. If a test needs fourteen weeks at a 5% MDE, look for a bigger swing rather than accepting worse evidence. Bigger changes are also more informative when they lose.
What to do with a loss
Four out of five tests do not beat control. That is the baseline you plan around, and a loss is a normal working outcome rather than a failure of the person who proposed it.
The response has three steps.
Read what it eliminated, out loud, in one sentence. “We believed size hesitation limited mobile add-to-cart. At a 10% MDE it does not.” That sentence goes in the backlog and retires everything else in that branch until new evidence appears.
Check the segments before you accept the headline. A flat overall result sometimes hides a real effect in one segment: mobile up, desktop down, netting to nothing. That is a finding, not a win. Treat it as a new hypothesis to test cleanly, never as a reason to ship the variant to everyone. Slicing a finished test until something looks significant is how you generate false winners on demand.
Kill the variant and move on the same week. The most expensive thing about a loss is the fortnight of arguing that follows it. Losses that get re-run with minor modifications until one finally wins are not tests, they are a slow way of shipping an opinion.
One more thing, and it is cultural rather than technical. Report losses in the same format, in the same meeting, with the same energy as wins. The moment they get reported quietly, people start proposing tests they expect to win.
What to do with an inconclusive result
Inconclusive is the most common outcome in ecommerce testing, and it is the one nearly everyone reads wrong.
Inconclusive does not mean “no difference”. It means your test could not see a difference of the size you were looking for. Those are different claims and only one of them is true.
The first question is not “which variant was ahead”. It is: was the minimum detectable effect realistic? If you powered the test to detect a 15% lift, you have learned nothing about whether a 4% lift exists, and a 4% lift on a large step is real money. If you powered it for 5% and got nothing, you have learned considerably more.
What to do next, in order of how often it applies:
Accept it and move the traffic. Usually correct. The test asked a fair question, the answer was “no effect that size”, and the slot is worth more elsewhere.
Do not extend the test. Deciding to run four more weeks after seeing the interim numbers reintroduces exactly the peeking problem the A/B testing chapter describes. If you want the option to extend, you set the longer duration before launch or you do not have it.
Do not ship the variant because it was up 2%. It was up 2% inside a range that also contained down 3%. Shipping on that is a coin flip with a story attached.
Consider a bigger version of the same idea. If the mechanism is well evidenced and the change was timid, the honest read is often that you tested a whisper. Rebuild it as a real change and spend another slot, once.
The trap to name: a store with three inconclusive results in a row will start describing them internally as “slight positives” and then as “wins” by the time they reach a slide. Nothing about the data changed. The language did.
Documenting learnings so year two is faster than year one
The record is the product. Everything else is traffic you spent to produce it.
Keep it in one place your whole team can search, and give every concluded test the same short entry: hypothesis, what shipped, primary and guardrail results, the segment reads you looked at, the one-sentence elimination, and the date.
Then add two artefacts on top, reviewed quarterly.
A branch map. Group every result by mechanism rather than by page. Trust, cost transparency, delivery certainty, fit confidence, information hierarchy, urgency. Each branch gets a status: untested, live winner, exhausted, or eliminated. This is the document that stops the fifth trust-badge test.
A principles list. Five to fifteen sentences that only your store knows. “Our customers do not respond to social proof volume, they respond to photos from people like them.” “Every cost-transparency change we have run has won.” These come from patterns across tests, never from a single result, and they are what makes a new hire useful in a fortnight instead of a quarter.
The test for whether your documentation works: hand it to somebody who has never worked on the account and ask them to propose next quarter’s three tests. If what they propose is sensible, the record is doing its job. If they propose things you already ran, it is not.
Reporting that a CFO can read
Most CRO reporting is designed to look busy. A finance audience reads it in ten seconds and correctly concludes it is marketing.
Four things belong in the report, and very little else.
Throughput. Tests concluded this period, tests won, tests eliminated a branch. Three integers. No commentary needed.
Revenue effect of the winners, with the assumptions in the open. State it as: measured relative lift on the primary metric, applied to the affected traffic, annualised, with the confidence interval shown. Then say plainly that the low end of that interval is the number to plan on. A range you defend beats a point estimate you cannot.
What was eliminated. Give the learnings a line each. This is the part that reads as competence to a sceptical audience, because nobody making numbers up volunteers the failures.
What is queued and why. Three items, with the affected step size next to each. It shows the prioritization is a rule rather than a mood.
What does not belong: a site-wide conversion-rate line chart. Conversion rate moves with traffic mix, promotions and seasonality, so attributing its movement to your tests is a claim you cannot support and will eventually be caught making. If you must show a trend line, show revenue per session, and still say out loud that the tests are measured in the tests and not in the trend.
One habit makes the whole report credible: publish the losses at the same size as the wins. And when a shipped winner does not show up in the top line, say why: usually because normal week-to-week variance is larger than the lift, which does not make the lift imaginary but does make it invisible.
Scaling a winner across the store
A winner on one template is not a winner on the store. Rolling it everywhere on the strength of one result is the most common way a good test turns into a quiet loss.
Roll out in this order.
Ship it exactly where it was tested, first. Same template, same audience, same viewport. Then confirm the change is actually live and rendering correctly, which fails more often than anyone admits.
Ask whether the mechanism travels. If fit guidance above the add-to-cart button won on apparel, the mechanism is “resolve the dominant purchase question before the button”, not “put a size chart there”. On a hardware category the dominant question might be compatibility. Port the mechanism, not the markup.
Re-test on a genuinely different context, once, if the traffic allows. Different category, different price band, or a different traffic source. One confirmation on the most different context you have is usually enough to justify the full roll-out.
Watch the guardrails after roll-out, not just during the test. Average order value and revenue per session, for a full cycle. A change that lifted add-to-cart across the catalogue can suppress average order value in a way a single-template test never saw.
Write the roll-out into the record. Where it is live, where it was deliberately not applied, and why. In eighteen months somebody will “clean up” the theme and remove it, and this entry is the only thing that will bring it back.
Resist the version where a winner becomes a brand rule applied to every surface without evidence. That is how a store ends up with the same intervention in nine places, four of which are quietly costing money.
How testing programmes die
Programmes rarely die from bad statistics. They die from organisational causes, and every one of these is visible months before the funeral.
They become an argument-settling device. Once tests are used to resolve internal disputes, somebody has a stake in every outcome and results stop being read honestly. The tell: tests get proposed with a preferred result already attached.
They optimise for volume. Somebody starts reporting tests launched rather than questions answered. Launch count is easy to move and means nothing. It reliably produces underpowered tests on small surfaces.
They run out of research. The backlog empties of evidence-backed items and fills with ideas from competitor screenshots. The win rate falls, and the falling win rate gets blamed on testing rather than on the input.
They promise a number to leadership. Someone commits to a percentage lift for the quarter. Now every inconclusive result has to become a win to protect the forecast, and within two quarters the reporting is fiction. Nobody can guarantee a testing outcome; a programme that has promised one has already stopped being a programme.
They lose the record. The person who held the context leaves, the spreadsheet is in their drive, and year three starts with year one’s backlog.
They test only what is easy to change. Copy, colours, badge positions. All cheap, all inside the noise floor, all avoiding the pricing, range, delivery and positioning questions where the real constraint usually sits.
If you see two of these at once, stop launching tests for a month and fix the programme instead. Running more experiments through a broken operating system produces more broken results, faster.
When to stop testing an area
Knowing when to walk away from a surface is a strategic skill and almost nobody teaches it.
Stop when three or four consecutive tests against the same mechanism come back flat. That is not variance, that is the surface telling you its constraint is elsewhere. Retire the branch, write the elimination down, and move the traffic.
Stop when the remaining ideas are all below your minimum detectable effect. If everything left is a change you would expect to move things 1-2% and your tests can only see 8%, you are not testing, you are burning weeks to produce inconclusive results.
Stop when the surface is not the constraint. A checkout converting at the top of its plausible range does not have much left in it. The traffic is better spent where the funnel drop is largest, which you already know from your funnel and research work.
Stop when the honest answer is upstream. Some conversion problems are pricing, range, delivery promise or audience problems, and no amount of page work reaches them. Saying so is more valuable than another quarter of tests that measure the problem more precisely.
Come back when conditions change: a redesign, a new category, a new primary traffic source, or new research that contradicts what you concluded. An eliminated branch is eliminated under the conditions you tested it in, not permanently.
That is the strategy layer. Fewer tests than you want, chosen better, sized honestly, documented so the next one is cheaper, reported in a way that survives scrutiny.
We run this for a living, for a small number of ecommerce brands at a time, and roughly four out of five things we ship do not beat control. That is what our testing engagement actually looks like, stated the same way we would state it on a call.
What founders ask before they book.
How do I prioritize CRO experiments?
Sort on the size of the affected step first, because it is the only input that is not a guess. Then strength of evidence, then cost to build and cost to be wrong, then whether your traffic can conclude the test at all. Scoring frameworks like ICE and PIE are useful for forcing a conversation between two people who disagree. They are not useful as an output you sort on, because two of the three inputs are opinions you invented.
How many A/B tests should we run per month?
However many your traffic can conclude, which is usually fewer than you want. Take your monthly sessions on the surface you are testing, divide by the sample size each test needs, and that is your ceiling. Most stores in the $1-10M range land somewhere between one and four concluded tests a month across the whole site. Running more than that means running underpowered tests, and an underpowered test is a slower way of guessing.
What is a realistic win rate for an ecommerce testing programme?
Plan for roughly one in five, which is what a typical programme with honest stopping rules produces. Our own published engagements have run better than that (39 of 71 beat control), but planning off somebody else's best record is how programmes get overcommitted. Anyone quoting a much higher number without a tests-run count beside it is either counting inconclusive results as wins or stopping tests when they are ahead. Assume most of the programme returns nothing directly, because the losses are where most of the learning is.
What should a CRO report contain?
Tests concluded, tests won, what each result eliminated, and the annualised revenue effect of the winners with the assumptions stated in the open. It should not contain a conversion-rate line chart with an arrow on it, because conversion rate moves for reasons that have nothing to do with your work. If the report cannot survive someone asking "how do you know", it is a presentation, not a report.
When should we stop testing a page?
When three or four consecutive tests against the same mechanism all come back flat. That is not bad luck, that is the page telling you its constraint is somewhere else. Move the traffic to a surface you have not exhausted and revisit the area only when new research, a new product line, or a redesign changes the underlying conditions.