Convert Lane

Chapter 3 of 11

Hypotheses That Deserve Traffic

How to turn research findings into hypotheses worth spending traffic on, prioritize without scoring theater, and recognise the tests you should not run at all.

Contents
  1. Anatomy of a testable hypothesis
  2. From finding to hypothesis, worked
  3. Prioritization without the scoring theater
  4. Writing the test spec
  5. When not to test
  6. The backlog is an asset, not a queue
  7. A worked backlog
  8. Losing gracefully

You have a problem list from Chapter 2. Now you have to decide what to do about it, and the decision is harder than it looks, because the obvious move is usually wrong.

The obvious move is: take the top finding, think of a fix, test the fix. That skips the step that does most of the work. Between a finding and a test sits a hypothesis, and a hypothesis is a claim about why something is happening. Skip it and you are testing a guess, which means you cannot learn anything when it loses.

Anatomy of a testable hypothesis

A usable hypothesis has four parts. Miss any one and it degenerates into an idea.

Because we observed [evidence], we believe changing [element] for [audience] will cause [effect on metric], which we will measure by [primary metric, with a guardrail].

Concretely:

Because 22% of our three-star reviews mention the product running small, and 31% of mobile sessions never reach the size chart, we believe moving fit guidance above the add-to-cart button on mobile will reduce size-related hesitation and increase add-to-cart rate, measured by mobile add-to-cart rate with revenue per session as a guardrail.

Compare that to what usually gets written on a whiteboard: “Test moving the size chart up.”

The difference matters for a specific, practical reason. When the whiteboard version loses, you have learned that one layout did not beat another layout. When the four-part version loses, you have learned that size hesitation is probably not what is limiting your mobile add-to-cart rate, which eliminates a whole branch of your backlog. A well-formed hypothesis pays out on losses. That is the entire point of writing it down.

The guardrail is not optional. Every hypothesis names a second metric that must not get worse. Add-to-cart with revenue per session as guardrail. Checkout completion with average order value as guardrail. Without it you will eventually ship a change that lifts the metric you watched and quietly costs you money, and you will not find out for two quarters.

From finding to hypothesis, worked

Take the four findings from the end of Chapter 2 and run them through.

Finding: shipping cost first appears at step 2 of checkout, 41% drop at that step.

This one barely needs a hypothesis, because the mechanism is not in doubt. Surprise costs at checkout are the single most-cited abandonment reason in every study ever run, and you have a 41% drop sitting exactly where the surprise happens. The hypothesis writes itself, and honestly, so does the decision: show shipping cost or the free-shipping threshold before checkout. That is not a test candidate, it is a bug fix. Ship it.

Finding: review block below the fold on mobile, 31% reach it.

Two competing hypotheses, and they lead to different tests:

  1. People are not reaching reviews because the page is too long, so shortening the page will increase review exposure and add-to-cart.
  2. People are not reaching reviews because they do not need them, and the ones who do already scroll. Moving reviews up will do nothing.

Hypothesis 2 is the one nobody writes down, and it is right more often than people expect. Test the move, but write both down first, so the loss means something.

Finding: non-brand paid social converts at 0.4% against a 2.2% site average.

This is not a product page hypothesis at all. It is a message-match problem or an audience problem, and no amount of button-colour work will fix it. This finding belongs to Chapter 7, not to your PDP backlog. Recognising which surface a finding belongs to saves entire quarters.

Prioritization without the scoring theater

You will be told to score every idea on Impact, Confidence, and Ease, multiply, and sort. ICE. Or PIE, or PXL, or whichever acronym is current.

Here is the problem with all of them: the inputs are made up. You are assigning a 7 to “impact” for something whose impact you cannot know, and a 4 to “confidence” which is a number describing how confident you are, scored by the person who wants to run the test. Then you multiply three fabricated numbers together and treat the product as a decision.

We use scoring frameworks for exactly one thing: forcing a conversation when two people disagree about ordering. The number is a discussion prompt, not an output.

What we actually sort on, in order:

1. Size of the affected step. A change on a page 68% of sessions see beats a brilliant change on a page 4% see. This is the only input that is not a guess, so it does most of the work.

2. Strength of the evidence. A finding backed by 340 reviews and a funnel drop outranks a finding backed by one recording. Rank your own evidence honestly. “The CEO does not like it” is evidence of something, but not of a conversion problem.

3. Cost to build and cost to be wrong. A copy change is cheap to build and free to revert. A checkout re-architecture is neither. When two candidates look similar, run the reversible one first.

4. Whether the store can afford the test at all. Which is the next section, and it disqualifies more candidates than the other three combined.

Notice what is not on the list: how excited anyone is about the idea. Enthusiasm is not a prioritization input, and the loudest idea in the room is usually the one with the least evidence behind it.

If you need raw material for the backlog itself, we keep a running list of product page test ideas tagged by impact and difficulty. Use it as a prompt against your findings, not as a to-do list. An idea from a list is not a hypothesis until you have attached your own evidence to it.

Writing the test spec

A hypothesis says what you believe. A spec says what will be built and how you will know. Without one, three things reliably go wrong: the variant that gets built is not the variant you designed, the result gets read against a metric nobody agreed on, and nobody can reconstruct what happened six months later.

Everything below fits on one page. If it does not, the test is too big.

Hypothesis. The full four-part version. Copied, not summarised.

What changes, exactly. Element by element, with the copy written out. “Move the size chart” is not a spec. “Insert the fit guidance block, containing [this copy], directly above the add-to-cart button, on mobile viewports under 768px, on all products in the Apparel collection” is.

What does not change. Explicitly. This is how you stop a developer helpfully improving three other things while they are in there, which is the most common way a clean test becomes uninterpretable.

Audience and traffic allocation. Which visitors are in the test. Mobile only? New visitors only? A specific collection? 50/50 split unless you have a reason.

Primary metric. One. Named before launch, never changed after.

Guardrail metrics. Usually revenue per session and average order value. The test does not ship as a winner if these move against you, even if the primary metric wins.

Sample size and planned end date. Both calculated in advance. Written down where other people can see them, which is what stops the conversation where somebody wants to call it on day four.

What we will do with each outcome. Write the three sentences now: if it wins we do X, if it loses we do Y, if it is inconclusive we do Z. Doing this before the result exists is the only reliable defence against reading the result you want.

That last field is worth dwelling on. A surprising number of tests get run where a loss and an inconclusive result would lead to exactly the same action, which means the test was never going to change anything and should not have consumed six weeks of traffic.

When not to test

This is the section that gets cut from most CRO content, because “you probably cannot test this” does not sell testing tools.

Do not test if you do not have the traffic. This is arithmetic, not opinion. To detect a 5% relative lift on a 3% baseline conversion rate at 95% confidence, you need roughly 50,000 visitors per variant. A store doing 30,000 sessions a month cannot run that test in under three months, and in three months your traffic mix, your ad creative, and your product range will all have changed. The test is not just slow, it is invalid.

Run your own numbers before you commit. Our statistical significance calculator has a planning mode: put in your baseline conversion rate and the smallest lift you would care about, and it tells you the visitors and the weeks. Do that before you build anything. If the answer is fourteen weeks, you are not running that test.

Do not test the obviously broken. If your mobile checkout has a field that fails on iOS, you do not need a control group to prove that fixing it is good. Fix it, note the date, watch the trend. Testing a bug fix is a way of spending six weeks proving something you already knew.

Do not test tiny changes on a small store. A 1% relative improvement is real and worth having on a store doing $40M. On a store doing $2M it is undetectable and unprofitable to chase. Small stores should be making big, evidence-backed changes and shipping them, not running underpowered tests that return “inconclusive” and cost a quarter.

Do not test what you will not act on. If leadership will not remove the hero carousel regardless of result, do not test the hero carousel.

The honest position for a store under roughly 40,000 monthly sessions: your programme is research plus decisive shipping, with tests reserved for the two or three changes a year that are big, risky, and reversible. That is not a lesser version of CRO. It is the correct version at your size, and pretending otherwise is how agencies bill for tests that could never have concluded.

The backlog is an asset, not a queue

Most teams treat the backlog as a to-do list that gets shorter. Ours gets longer, and that is the point.

Every entry carries: the hypothesis in full, the evidence behind it, the affected step and its size, the required sample size, and after it runs, the result and what the result eliminated.

The value compounds because of that last field. Six months in, you can look at your own record and see that four separate tests aimed at trust signals all came back flat, which tells you trust is not your constraint and stops a fifth. That knowledge is worth more than any individual winner, and it exists only if you wrote down what you believed before you ran the test.

A backlog with results attached is also the only defence against the most expensive failure mode in this work: re-running a test somebody already ran two years ago because nobody wrote it down.

A worked backlog

Four findings from Chapter 2, run through everything above, and sorted. This is what the output actually looks like.

1. Shipping cost appears at checkout step 2. Ship, do not test. Affected step: 41% drop at checkout. Evidence: manual walkthrough plus funnel. Mechanism not in doubt. Action: surface the free-shipping threshold and an estimated total in the cart. Reversible, cheap, no control group needed. First, because it is the largest drop and the least ambiguous.

2. Non-brand paid social converts at 0.4% against 2.2%. Investigate before building anything. Affected step: whole funnel, one channel. This is not a PDP item. Before any page work, check whether the ad creative and the landing experience are talking about the same thing. Half of these turn out to be a targeting problem, in which case there is no site work to do at all.

3. Fit guidance missing. Test, if traffic allows. Affected step: PDP to add-to-cart, 68% of sessions see the page. Evidence: 340 reviews, plus scroll data. Sample size needed at a 10% MDE on mobile add-to-cart: roughly six weeks at current traffic. Marginal. Run it, and accept it may come back inconclusive.

4. Review block below the fold on mobile. Hold. Affected step: same as above. But two competing hypotheses, and it targets the same metric on the same page as item 3, so it cannot run concurrently. It waits. If item 3 wins, this becomes more interesting; if item 3 loses, the whole “mobile PDP information hierarchy” branch gets downgraded and this may never run at all.

Four findings, and only one is actually an A/B test. One is a bug fix, one is a channel investigation, one is queued behind another test.

That ratio is normal, and it is the thing most testing programmes get wrong. They convert every finding into a test, run them all at once on the same page, and end the quarter with four uninterpretable results.

Losing gracefully

Plan for roughly one test in five. Sometimes one in six. That is what people who report honestly tend to publish, and it is the number to build a backlog around. Our own written-up engagements have run better, at 39 of 71, but you should not plan your programme off our best stores.

That means most of your work returns nothing directly. Which is fine, and here is how to make it not fine but actually useful:

A loss with a hypothesis is a finding. You believed size hesitation limited mobile add-to-cart. It did not. Cross off that branch. You now know something about your customers that you did not know last month, and it cost you three weeks instead of a redesign.

A loss without a hypothesis is nothing. You moved a thing, the number did not move, you learned that the thing does not matter in that one position. You will move it again in eight months.

An inconclusive result is not a loss. It means you did not have enough traffic to see an effect of the size you were looking for. The correct response is to look at whether your minimum detectable effect was realistic, not to declare the variant a winner because it was up 2%. Chapter 4 deals with exactly this, because reading results wrong is the most common way stores convince themselves they are winning.

Keep a public record of your losses inside the company. It is the fastest way to kill the culture where only successful tests get reported, which is the culture where the numbers stop meaning anything.

There is an organisational version of this worth naming. When testing is used to settle internal arguments rather than to learn about customers, the results stop being read honestly, because somebody now has a stake in each outcome. The tell is easy to spot: tests get proposed with a preferred result already attached, and losing tests get re-run with small modifications until one wins. If you notice that happening, the problem is not statistical and no methodology will fix it.

We do this for a living and about 80% of what we ship does not beat control. That is what the engagement actually looks like, and any agency describing it differently is describing something else.

Next: the mechanics of running the test, and the statistics that decide what you are allowed to conclude.

Have findings and no idea which deserve traffic?

That is exactly what the first month with us looks like. We turn the findings into hypotheses, price the tests against your traffic, and drop the ones that will not conclude before they eat a quarter.

  • 437+ stores built
  • 2,000+ tests shipped
  • No call needed to start

Claim your free audit of one page

Send us one page of your store and we’ll review it on video: what’s leaking, what to fix first, and what to leave alone. Changes you can ship this week to lift CVR, AOV and revenue per visitor.

Get My Free Audit