You Ran the CRO Playbook and the Number Didn’t Move

You did everything the playbook told you to do.

You ran the A/B tests. You bought the tool with the heatmaps and the session recordings. Maybe you hired the agency with the case studies. You shipped a cleaner checkout, a stronger button, better product-page copy, a sticky add-to-cart on mobile. Twenty tests, thirty tests, a whole quarter of them.

And the conversion rate is roughly where it started.

So now you’re doing the thing every operator does at this point: quietly wondering whether you’re bad at this. Whether the agency oversold you. Whether CRO is just snake oil dressed up in statistics.

It’s none of those. What’s actually happening is more useful to know, and once you see it, you stop wasting money on the wrong half of the problem.

Most tests don’t win. That’s the design, not your failure.

Here’s the number nobody puts in the sales deck. When Ronny Kohavi ran experimentation at Microsoft and Bing — one of the largest, most sophisticated testing operations on the planet — he found that roughly a third of ideas produced a statistically significant improvement. A third did nothing. And a third actively made the metric worse.

Read that again, because it reframes your whole quarter. The best testers in the world, with unlimited traffic and full-time data scientists, get a win on about one idea in three. The other two-thirds are flat or negative.

So if you ran thirty tests and a handful moved the needle a little and the rest went nowhere — you didn’t fail. You produced a completely normal hit rate. The problem was never your execution. The problem is that you were running the playbook at the wrong altitude.

CRO, as it’s usually sold, is a tactics library: button colors, urgency badges, form fields, trust seals. Those are real, and occasionally one of them is worth a point or two. But a tactics library assumes the strategy underneath is already right. Testing the checkout on a page that’s selling the wrong thing, to the wrong visitor, at the wrong moment of intent, doesn’t produce a lift. It produces a rounding error and a very organized-looking report.

The conversion rate is a symptom. Read it in this order.

A stuck conversion rate is telling you something. It’s just not telling you “test more buttons.” It’s telling you to look, in a specific order, at three things underneath the number.

1. The offer. Not the product — the offer. Is what you’re asking someone to buy, at the price you’re asking, an obvious yes for the person landing on the page? Most “conversion problems” are offer problems wearing a CRO costume. If the value isn’t clear in five seconds, no headline test fixes that. When the offer is right, conversion tends to be forgiving of a clumsy checkout. When the offer is wrong, a perfect checkout still doesn’t save it.

2. The traffic. A conversion rate is a fraction, and you’ve been staring at the numerator. Look at the denominator. If you’re buying more visitors to hold the same sales, your “conversion problem” is partly a traffic-quality problem. Colder, broader, cheaper traffic converts worse almost by definition. Testing the page harder won’t fix a page that’s being shown to people who were never close to buying.

3. The trust. This is the one the tools can’t see. Does the store feel like somewhere a stranger hands over a card without a second thought? Reviews that read like real humans, shipping and returns that don’t require a scavenger hunt, a price that doesn’t jump at the last step, a brand that looks like it’ll still be here in six months. Trust is where most DTC conversion quietly leaks, and it never shows up as a single testable element — which is exactly why the playbook skips it.

Offer, then traffic, then trust. In that order. Because a fix at the top makes the fixes below it work harder, and a test at the bottom can’t rescue a miss at the top.

A five-minute diagnostic before you approve another test

Next time someone brings you a test roadmap, don’t ask “what should we test?” Ask these four questions first.

• Would a stranger understand the offer — and want it — in five seconds? If you’re not sure, that’s your quarter. Not the button.
• Has traffic quality changed while conversion stalled? Pull the trend. If sessions are up and the rate is down, you have a sourcing problem masquerading as a page problem.
• Where does your own trust break? Walk your checkout as a first-time buyer on your phone, cold. The place you hesitate is the place they leave.
• Is this test big enough to matter even if it wins? A one-percent lift on a page that 4% of visitors reach is a rounding error. Test where the volume and the money actually are.

If a test can’t survive those four questions, it’s not a priority — it’s motion. And motion is what a stalled CRO program is usually made of.

Where this actually goes

Sometimes you run this diagnostic and the answer is “the offer is right, the traffic is qualified, the trust is solid — go test the checkout.” Great. Now testing will actually pay, because everything underneath it is carrying its weight.

But far more often, the stalled number is pointing at a decision no A/B test was ever going to make for you: a repositioning, a pricing change, a different buyer, a tighter promise. That’s not a CRO project. That’s a marketing-strategy call — the kind that has a conversion rate attached but doesn’t live inside the testing tool.

You’re not bad at this. You’ve just been optimizing the one layer that was never the problem.

Sources: Ronny Kohavi, Online Controlled Experiments and A/B Testing (Microsoft / exp-platform) — roughly one-third of tested ideas produce a statistically significant positive result, one-third are flat, and one-third are negative.