Convert Lane

Rules written before the result exists.

How we run CRO tests

What we look at, how a finding becomes a test, how it is sized before anyone opens a code editor, and the rules that decide whether it won. Rules written afterwards are not rules.

20 minutes · your numbers · no deck

39 of 71 beat control · 32 didn’t Every written-up test. Filled beat control; empty did not, and is published anyway.

The process, end to end

Eight steps. Steps one to four run once at the start of an engagement and then top up continuously; steps five to eight repeat per test.

  1. Pull the numbers

    Funnel by step and by device, landing page performance split by source, internal search terms, checkout drop-off, refund and return rates. This stage produces no opinions. It produces a list of places where money leaves.

    The research chapter
  2. Read what buyers actually say

    Your reviews and your competitors, support tickets, live chat transcripts, post-purchase survey answers. Quantitative data says where the leak is. This says why, in the customer’s own words, which is also the wording that ends up in the variation.

  3. Watch the sessions that explain a number

    Recordings and heatmaps are used to answer a question we already have, never as browsing. Fifty recordings watched without a question produce anecdotes and unearned confidence.

  4. Rank the problems, not the ideas

    Every problem gets scored on how much traffic it touches, how much money sits behind it, and how strong the evidence is. Ideas are ranked later and only against problems that survived this step.

    Prioritising without the theatre
  5. Write a hypothesis with the evidence attached

    One sentence naming the observation, the change, the expected direction, and the metric it should move. If the evidence line is empty, the idea goes back to the backlog rather than onto the roadmap.

    Hypotheses that deserve traffic
  6. Size it before anyone builds it

    Baseline rate and the smallest lift worth detecting give the sample size, and the sample size gives a run length. We agree the primary metric, the guardrails, the confidence threshold and the stop date at this point, in writing, while nobody knows the answer yet.

    Size a test
  7. Build it, break it, then run it

    QA across the browsers and devices your traffic actually uses, an A/A sanity check where the platform is new to us, and confirmation that the variation is being recorded before real traffic sees it. Then it runs to the planned sample. Nobody stops it early because it looks good on day three.

  8. Call it, publish it, decide what happens next

    Win, loss or inconclusive against the rules agreed in step six. A win ships and goes on a watch list. A loss is written up. An inconclusive result closes the idea rather than quietly becoming a second attempt with a new coat of paint.

    How we call a result

The research inputs we actually use

There is no proprietary framework here. There is a fixed list of sources, and the discipline is in using all of them before forming a view rather than picking the one that agrees with the redesign someone already wants.

  • Analytics

    Funnel by step and device, landing pages by source and by intent, internal search queries, checkout abandonment by stage. Split by device before anyone speaks.

  • Voice of customer

    Reviews on your store and on three competitors, support tickets, chat transcripts, and post-purchase survey answers. The cheapest research available, the most skipped.

  • Behavioural

    Session recordings and scroll or click maps, sampled against a specific question. A recording explains a number; it is not a substitute for having one.

  • Technical

    Real-user page timings on the pages that carry revenue, third-party script inventory, and the store rebuilt on the devices your traffic actually arrives on.

  • Commercial

    Margin by product, shipping and duty policy, return rate, repeat purchase interval. A conversion win that shrinks contribution margin is not a win.

  • Comparative

    What direct competitors do, used strictly as a source of questions. Nobody knows whether their version tested, including them.

How a finding becomes a hypothesis

A finding is an observation. A hypothesis is a bet with its reasoning attached. We write every one in the same four parts, and an idea that cannot fill all four does not get traffic.

Observation
What we saw, and in which source. Named, not summarised.
Change
The specific thing that will be different for the variation group.
Expected effect
Which metric moves, in which direction, and roughly why the mechanism works.
What would falsify it
The result that would make us abandon the idea rather than reshape it. Written before the test runs, because afterwards everyone has an explanation.

A hypothesis that reads a cleaner hero will convert better is a preference. One that reads because the reviews and the support tickets both keep asking about sizing, a fit guide in the gallery will raise add-to-cart is a hypothesis. Only the second one can be wrong, which is the entire point.

Tests are sized before they are built

Sizing is the step that decides whether an idea is testable at all, so it happens before design, not after. Baseline conversion rate and the smallest lift worth detecting produce a required sample; the store’s traffic turns that sample into a number of weeks.

Weeks to detect a 10% lift on a 2.5% baseline

  • 20k sessions / month 28 wks
  • 40k sessions / month 14 wks
  • 80k sessions / month 7 wks
  • 160k sessions / month 4 wks
The line is the eight-week cut-off. Past it, seasonality and traffic drift contaminate the comparison faster than the sample arrives, so the test is not built. Illustrative inputs; run your own numbers →

If the required sample will not arrive within about eight weeks, the test does not get built. Longer than that and seasonality, price changes and traffic-mix drift contaminate the comparison faster than the sample accumulates. That leaves three honest options: target a bigger change with a bigger expected effect, move the test to a higher-traffic surface, or ship the change on judgement and say plainly that it was judgement.

Everything below gets written down and agreed before the build starts: primary metric, guardrail metrics and their limits, required sample, confidence threshold, planned stop date, and what each outcome will mean. Fixing the rules while the answer is still unknown is the single largest difference between a programme that measures and one that narrates.

Model what a programme is worth on your traffic

Guardrail metrics

The primary metric is revenue per session, because conversion rate on its own can be bought with a discount, a shipping subsidy or a smaller basket. Alongside it, every test carries guardrails: metrics that are not the goal but that veto a win if they move the wrong way.

  • Average order value. Catches lifts bought by shrinking the basket.
  • Contribution margin per order. Catches lifts bought with shipping or discount subsidy.
  • Refund and return rate. Catches persuasion that oversold the product. This one needs a full return window before it can be read.
  • Support contact rate. Catches changes that moved confusion downstream rather than removing it.
  • Page timing on the affected template. Catches the variation that won on a fast connection and lost everywhere else.

A test that lifts conversion 6% while cutting order value 9% is a loss. Reporting it as a win is how agencies keep contracts they have not earned.

How we call a result: win, loss, or inconclusive

Three outcomes, defined in advance, with the same evidence bar applied to each. The result label always travels with a word, never only a colour.

  1. Rules written down
  2. Sample sized
  3. Runs in whole weeks
  4. Read once, at the planned stop
  5. Win · Loss · Inconclusive
  • Win

    The primary metric moved in the predicted direction, cleared the confidence threshold at the planned sample, and no guardrail moved against us.

    Ships to 100% of traffic. It stays on a watch list for a full purchase cycle, because a lift that disappears in four weeks was never a lift.

  • Loss

    The primary metric moved against us at the planned sample, or a guardrail metric breached its limit even though the headline number looked good.

    The variation is switched off and written up with the same detail a winner gets. The finding usually kills three neighbouring ideas as well, which is most of its value.

  • Inconclusive

    At the planned sample, the result cannot distinguish the variation from the control. This is the most common outcome of an honestly run programme.

    The idea is closed, not re-run. Re-running an inconclusive test until it crosses a line is how a programme fabricates wins without anyone deciding to.

Two rules sit behind all three. We do not look at a running test and stop it because it looks good; every extra look is another chance for noise to cross the line. And we do not go hunting for a segment where the result was positive. If a segment analysis was not declared before the test ran, it is a hypothesis for next time, not a finding from this one.

Most tests do not win. A typical programme sees roughly one in five, and our own published engagements run at 39 of 71. Losing tests get written up in the same format winners get (same detail, same document) and where we publish an engagement, the tests-run count appears next to the tests-won count. An agency reporting a 90% win rate is not better at this than everyone else; it is reporting differently.

What each kind of evidence can and cannot prove

Most disagreements about CRO are really disagreements about what counts as proof. This is the version we work from.

Criterion Controlled A/B test Before-and-after Observational
What it compares Two versions running at the same time, on randomly split live traffic.The same page in two different time periods.People who did one thing against people who did another, with no split at all.
What it can prove That the change caused the difference, within a stated margin of error.That the number changed. Nothing about why.That two things happen together.
What it cannot prove That the effect holds at a different season, price or traffic mix.Anything about cause. Season, ad spend, promotions and PR all moved too.Cause, in either direction. Buyers who use search convert better; search did not make them buyers.
What contaminates it Stopping early, changing the variation mid-flight, uneven bot traffic, a broken tracking tag.Literally everything else that happened that month.The reason someone self-selected into the group in the first place.
Traffic needed Enough conversions to detect the effect you care about. Usually the binding constraint.None, which is exactly why it is popular.Enough to segment without the segments becoming meaningless.
Where we use it Any change big enough to size, on a store with the traffic to support it.Never as proof. Occasionally as a prompt to go and look at something.Research only. It generates hypotheses; it never closes them.

A controlled test is the only one of the three that can support a causal claim, and even then only about the traffic, season and price it ran under.

When your traffic is too low to test

Most Shopify stores cannot run the tests they read about. At a few hundred orders a month, detecting a small relative lift takes longer than the season the change was designed for. Pretending otherwise sells a retainer and delivers noise.

What we do instead, in this order:

  1. Fix what is unambiguously broken. Errors on real devices, a checkout step that fails on one browser, a page that takes eleven seconds on mobile. None of this needs a test, and testing it would be theatre.
  2. Make the changes best practice already settles. Where the evidence base is large and the mechanism is obvious, the expected value of shipping beats the expected value of an underpowered test.
  3. Test fewer, larger things. A restructured landing page or a rebuilt product page swings a bigger effect than a button colour, and a bigger effect needs a smaller sample.
  4. Test on a surface with more traffic. Paid landing pages and email flows often accumulate a sample faster than the sitewide funnel does.
  5. Measure what you can measure. Micro-conversions, on-page surveys and qualitative work still generate real learning, as long as nobody calls the result a lift.

And sometimes the answer is that the site is not the problem. Pricing, offer, product or acquisition costs sit outside what a testing programme can reach, and we say so on the first call rather than four months in.

How results get reported

One document per test, written to the same shape whatever the outcome, and readable by somebody who was not in the room.

  • The hypothesis as it was written before the test, unedited.
  • Screenshots of control and variation, on desktop and mobile.
  • Dates, sample reached, and the sample that was planned.
  • Primary metric with its interval: the range the true effect plausibly sits in, not a single flattering number.
  • Every guardrail, including the ones that did not move.
  • The call, in a word, and the decision that follows from it.
  • What we learned that changes the backlog, which is often more valuable than the result.

Monthly, that rolls up into tests concluded, tests won, revenue-per-session change on the affected surfaces, and what is queued next with its sizing already done. No vanity charts, and no percentage lift quoted without the sample it came from.

What this method will not do

  • It will not guarantee a lift. Nobody can, and the ones who do are describing a sales process rather than a testing one.
  • It will not make a low-traffic store testable. It will tell you that early, in writing, before you have paid for a quarter of it.
  • It will not fix a pricing, offer or product problem. Those show up in the research, and we hand them back rather than testing around them.
  • It will not produce a result every month. Some tests need a full return window before the guardrails can be read at all.

If that reads as underselling: it is the same method whether you hire us or run it yourself, and the whole of it is free in the guide.

Want this run on your store?

Twenty minutes on your numbers. We tell you where the store is leaking, what we would test first, and whether you have the traffic to test at all.

  • 437+ stores built
  • 2,000+ tests shipped
  • No call needed to start

Claim your free audit of one page

Send us one page of your store and we’ll review it on video: what’s leaking, what to fix first, and what to leave alone. Changes you can ship this week to lift CVR, AOV and revenue per visitor.

Get My Free Audit