Convert Lane

Chapter 2 of 11

Conversion Research: Find What's Actually Broken

Six research methods that tell you what is broken on your store, in the order we run them. Analytics first, then reviews, support tickets, recordings, and surveys.

Contents
  1. Trust your analytics before you trust your analytics
  2. Funnel and device analysis
  3. Review and support-ticket mining
  4. The walkthrough, on real devices
  5. Competitor research, done usefully
  6. Session recordings without wasting a week
  7. Surveys that get real answers
  8. The method that keeps producing nothing
  9. When research finds something that is not a conversion problem
  10. How long this takes
  11. Turning findings into a problem list

Every bad testing program starts the same way: somebody has an idea.

Every good one starts with a problem list. The list is not opinions. It is findings, each one traceable to something you observed, each one specific enough that a fix would be obvious if you knew the cause.

This chapter is how you build that list. Six methods, in the order we run them, with what each is actually good for and where each one lies to you.

Trust your analytics before you trust your analytics

Before any analysis, verify the data. This step gets skipped roughly always, and it invalidates roughly everything downstream.

Four checks, in order:

1. Is the purchase event firing once per order? Compare orders in your ecommerce platform against purchase events in your analytics for the same 30-day window, same timezone. Off by more than 3–5%? Stop. Fix that first. Double-firing on a thank-you page refresh is the most common cause, and it inflates every conversion rate you look at.

2. Is revenue matching? Same window. If events match but revenue does not, you are probably counting tax or shipping in one system and not the other. Not fatal, but you need to know which.

3. Are bots and internal traffic excluded? Your own team, your developer, your uptime monitor, and a surprising volume of automated traffic all sit in the denominator. Filter internal IPs. Turn on known-bot filtering. On smaller stores this alone can move measured conversion rate by a tenth of a point.

4. Is cross-domain tracking intact? If checkout sits on a different domain, and the session breaks there, every checkout number you have is fiction. Run one test purchase and watch the session in real time.

We have walked into stores where the funnel showed a catastrophic checkout drop-off that turned out to be a tracking break. Two weeks of planned testing pointed at a problem that did not exist.

Funnel and device analysis

Now the actual work. You are looking for the biggest relative drop, not the biggest absolute number.

Build the step chain from Chapter 1: sessions, product views, add to cart, begin checkout, order. Then cut it three ways.

By device. Mobile and desktop are different stores. On most accounts mobile converts at 40–60% of desktop, and the gap is usually not inevitable, it is a specific broken thing. Sticky buttons that cover content. Images that need pinch-zoom. Forms that trigger the wrong keyboard. If your mobile-to-desktop ratio is below 0.4, you have found your quarter’s work.

By traffic source. Branded search, non-brand search, paid social, email, direct. A conversion rate that looks fine in aggregate often hides one channel converting at a fifth of the others. That is either a targeting problem or a landing experience problem, and Chapter 7 covers the second.

By new versus returning. If returning customers convert at 4x new, your acquisition experience is the constraint, not your checkout. If they convert about the same, something is wrong with how you treat people who already trust you.

The output of this step is one sentence: the biggest relative drop is at [step], concentrated in [segment]. If you cannot write that sentence, you have not finished.

Review and support-ticket mining

This is the highest-yield research method available to an ecommerce store, and almost nobody runs it systematically.

Your customers have already written down every objection, confusion, and unmet expectation. They did it in reviews, in support emails, and in the chat widget. It is free, it is specific, and it is in your own words rather than a consultant’s framework.

How we run it:

Export the last 200–400 reviews, including your competitors’ reviews on the same product category. Export the last 300 support conversations. Then tag every one against a fixed set of themes. Sizing, shipping speed, shipping cost, product performance, comparison to alternative, durability, taste or smell, setup difficulty, returns.

Count the tags. The top three themes on negative and neutral reviews are your product page’s job description. If 22% of your three-star reviews mention that the product ran small, your size guidance is not a nice-to-have, it is the conversion problem.

Support tickets are even better, because a support ticket is a purchase that almost did not happen. Pre-purchase questions especially. Every pre-purchase question in your inbox is a question your product page failed to answer, and it arrived with the exact wording customers use. That wording belongs on the page.

The thing this method gets wrong: reviews over-represent the extremes and under-represent the silent majority who bounced. It tells you what buyers struggled with. It cannot tell you what non-buyers needed. Pair it with the next two methods.

The walkthrough, on real devices

Before any tool, buy something from your own store. Properly, with your own money, on a phone you have not cleared.

This sounds trivial and it consistently produces findings that no analytics package will surface, because analytics tells you where people left and never tells you what it felt like to be there.

Do it on the devices your traffic actually uses. Pull your top five device and browser combinations from analytics. On most stores this means a mid-range Android on Chrome, an older iPhone on Safari, and at least one in-app browser, because traffic arriving from Instagram or TikTok renders inside their own webview and behaves differently. In-app browsers are where we find the most unreported breakage: fonts failing, sticky elements misbehaving, payment sheets not launching.

Do it on a throttled connection. Not office wifi. Chrome’s device toolbar has a “Fast 3G” preset; use it, because a share of your customers are effectively on it.

Write down every moment of friction as it happens, not afterwards. The value is in the reaction you have in the moment and forget within a minute. Where did you hesitate? What did you have to hunt for? What made you tap twice?

Then repeat it as a customer with a problem: find the return policy, find whether it ships to your country, find the size guide, contact support. Each of those is a real customer journey, and each one is usually much worse than the buying flow because nobody has ever tested it.

Competitor research, done usefully

Most competitor research is a screenshot deck that changes nothing. There are two versions that pay.

Mine their reviews, not their design. Their one and two star reviews tell you what their customers are unhappy about, and that is a list of things you could say on your own product page. If 15% of their negative reviews complain about slow shipping and you ship in two days, that is a fact your page should carry.

Buy from them once. Their post-purchase flow, their emails, their packaging, their returns process. This is the part of a competitor you cannot see from the outside, and it is where most of the interesting decisions are.

What not to do: copy their layout because they are bigger than you. Their page is the output of their traffic, their price point, and their catalogue, and their tests. You do not have any of those. Copying a big competitor’s product page is the single most common way stores import a solution to a problem they do not have.

Session recordings without wasting a week

Recordings are seductive and enormously wasteful if you watch them wrong. Watching 40 random sessions is entertainment.

Watch them with a filter and a question.

Useful filters, roughly in order:

  • Sessions that reached checkout and did not convert
  • Sessions with a rage click or repeated dead click
  • Sessions on the one page your funnel analysis flagged
  • Mobile sessions above the 90th percentile for duration on a product page (long is usually confusion, not interest)

Cap it. Twenty filtered sessions, one page, one question, notes as you go. When you see the same behaviour three times, that is a finding. Once is an anecdote.

What recordings are genuinely good at: seeing that people scroll past your key content, seeing them hunt for something that is not there, seeing them try to tap something that is not a link, and seeing forms fail on real devices.

What they are bad at: telling you why. You are watching a cursor, then inventing a motive. Keep the finding descriptive (“nine of twenty scrolled past the size chart without opening it”) and leave the why to a hypothesis in Chapter 3.

Surveys that get real answers

Two surveys are worth running. Ignore the rest.

On-site exit survey, one question, product page only. “What is stopping you from buying today?” Free text. Show it after 25 seconds and only to people who have not added to cart. You will get a low response rate and it does not matter, because the responses are specific and unprompted.

Post-purchase survey, two questions, on the thank-you page. “What almost stopped you from buying?” and “What were you using before this?” The first gives you objections from people who overcame them, which is the closest thing to a list of what your page needs to fix. The second gives you the alternative you are actually competing against, which is usually not the competitor you think.

What not to do: do not ask people to rate things on a scale, do not ask what features they want, and do not ask hypotheticals. People are unreliable narrators of their own future behaviour and reliable reporters of their recent past. Ask about the past.

The method that keeps producing nothing

Heatmaps. We keep them in the toolkit and they keep earning their place about a tenth as often as they get cited.

Scroll maps have one genuine use: confirming whether people reach a section at all. If 30% of mobile sessions reach your review block, the reviews are not underperforming, they are unseen, and that is a layout finding.

Click maps are mostly noise. Aggregate click density on a page with a sticky button tells you people click the sticky button. Thanks. The one exception is clicks on non-interactive elements, which reliably surfaces things people expect to be clickable and are not. Product images that do not open a gallery. Text that looks like a link. Fix those and move on.

We have run engagements where heatmaps produced exactly zero findings that the funnel analysis had not already produced. Budget your time accordingly.

When research finds something that is not a conversion problem

A good research pass regularly produces findings that no page change can fix. Recognising them early saves a quarter of pointless work.

Pricing. If your exit survey fills with “too expensive” and your post-purchase survey shows people comparing you against something half the price, you have a positioning problem. You can sometimes solve it on the page, by making the value legible or by introducing a lower entry product. You cannot solve it with a better button.

Product. If negative reviews cluster on the product failing to do the thing, the conversion rate is not the problem. It is the symptom, and improving the page will increase returns rather than revenue.

Operations. Slow shipping, stockouts, and bad support show up in research as conversion findings because customers experience them that way. A page can set expectations honestly, which helps. It cannot make a fourteen-day delivery into a three-day one.

Traffic quality. If one channel converts at a fifth of the others and the landing experience is fine, the problem is upstream. That belongs to whoever buys the media.

Category. Some things are researched for weeks and bought elsewhere. If your product is one of them, a low conversion rate is a property of the category rather than a failure of your store.

Write these down separately from your testing backlog, and give them to whoever owns them. Two useful things follow. The people who can act on them find out, and your backlog stops containing items that will never work.

The failure mode here is quiet and expensive: research surfaces a pricing problem, it gets filed as a conversion finding because that is whose report it is, and six months of page tests get run against something no page can move.

How long this takes

Two to three weeks of one person’s attention, and it is front-loaded: you do it once properly and the output stays useful for a year.

A realistic split:

  • Days 1–2: analytics verification. Do not skip and do not shorten.
  • Days 3–5: funnel and segment analysis. Output is the one sentence.
  • Days 6–8: review and support mining. The tagging is tedious and it is the highest-yield step, so protect the time.
  • Day 9: device walkthrough and competitor purchase.
  • Days 10–12: filtered recordings and scroll checks, capped.
  • Days 13–15: survey setup, then wait. Surveys run in the background from here.
  • End: write the problem list.

The most common failure is doing all of it at 20% depth in three days and producing a list of impressions. Better to run three of these methods properly than six of them badly.

The second most common failure is doing the research, producing an excellent list, and then never converting it into anything. Book the next step before you start the first one.

Turning findings into a problem list

Research output is not a slide deck. It is a table, and every row has to survive one question: what did you observe, and where?

FindingSourceSegmentSize of the affected step
Size guidance missing; 22% of 3-star reviews mention fitReview mining, n=340AllPDP → ATC, 68% of sessions
Shipping cost first appears at step 2 of checkoutManual walkthroughAllCheckout → order, 41% drop
Review block below the fold on mobile, 31% reach itScroll mapMobile, 74% of trafficPDP → ATC
Non-brand paid social converts at 0.4% vs 2.2% site averageGA4 funnelPaid socialSession → order

Four rows beat forty. A problem list you can hold in your head is a problem list that gets worked.

Sort by size of affected step, not by how interesting the finding is. The unglamorous shipping-cost row is worth more than the clever one, every time.

One discipline worth enforcing: if you cannot name the source and the sample behind a row, it does not go on the list. “The page feels cluttered” is an opinion, and opinions are how testing backlogs fill with things nobody can evaluate afterwards.

If you want a longer walkthrough of the audit itself, with the eleven steps we run on a client account, we wrote that up separately in the ecommerce CRO audit checklist. It is the same method at more length. You can also read a worked teardown where this research shows up as specific annotated fixes on a real page.

Doing this properly on a store takes us two to three weeks. If you would rather hand it over, that is the engagement we sell.

Next: turning this list into hypotheses that deserve traffic.

Want us to run the research instead?

Same six methods, run against your analytics, reviews and support inbox. You get the problem list ranked by size of affected step, and the honest read on which findings are conversion problems at all.

  • 437+ stores built
  • 2,000+ tests shipped
  • No call needed to start

Claim your free audit of one page

Send us one page of your store and we’ll review it on video: what’s leaking, what to fix first, and what to leave alone. Changes you can ship this week to lift CVR, AOV and revenue per visitor.

Get My Free Audit