A new take on testing in the digital world

Testing in the world of digital marketing is like so many things in life — eating right, flossing, driving the speed limit, getting enough sleep.

Of course you should do it, but in fact, doing it isn’t always as easy as it sounds. I’ll save the healthy living lecture for some other post on some other blog. But let’s talk about testing — testing the performance of organic search, for instance.


You’re no doubt worrying and working hard on organic search. You’re focused on SEO. You’ve probably hired an agency. You’re probably working with various technologies to improve discoverability.

Key takeaways

  • Testing the effect of one change in digital marketing is extremely difficult because of ever-changing variables.
  • A control test can be the gold standard, but they are costly and can’t be sustained for long.
  • Think about identifying other metrics that point to the health of your site in order to monitor the ongoing value of established improvements.

You want to know what’s working. And you should want to know. The reason everyone talks endlessly about testing is because it’s vital. It’s a best practice. But when it comes to organic search, like so much in digital marketing, it’s not a one-size-fits-all proposition.

“The challenging thing in organic search is understanding and disambiguating what is causing growth or decline,” says Nikhil Gupta, who works on BloomReach’s product and engagement team.

You know doubt know the drill: Launch a site improvement and monitor the go-to indicators — things like keyword ranking or traffic on a page or bounce rate. Maybe you look at new leads or the number of conversions.

Static metrics in a changing world

The problem, of course, is that you’re looking at changing metrics in isolation. Any number of factors could be behind a boost or drop in any of the barometers you choose.

Maybe your keyword ranking improved. Great. But was that because of your latest initiative or because a new page is linking to your page? Maybe it was because one of your vendors gave you a new product description with more and better keywords, leading to a better ranking.

And what if your keyword ranking goes up and so does your bounce rate? Chances are you’ve made improvements to your content that search engines are really into, but actual people are not.

There is simply too much going on in the wide world of the Web for you to isolate cause-and-effect. That’s why companies that deal in copious amounts of data, companies like Pinterest and BloomReach, have turned to a different strategy — the control test.

On one hand, a control test is a thing of beauty. The idea is to isolate the one change you want to evaluate. You fire up a new search marketing technology or decide to double down on improving your content. Then you apply the new initiative to half your pages and leave the other half as a control, without the benefit of the latest change.

All things being equal, you can get a very good idea of the difference the change is making.

The problem is: In the real world of search, things don’t remain equal for long. If you’re a retailer selling on the Web, are seasonal factors affecting the traffic coming to your site? What about changes to your catalog or products that go out of stock? What if you add or remove a category from you site? Or that email promotion you launched: Any chance that brought a spike in traffic? What about that big movie star showing off his camp socks on Jimmy Kimmel? Could that have anything to do with the bump in traffic to your socks category?

What else has changed?

What about the major search engines? Have they changed the way they operate or display search results? What about your own site? Have you instituted strategies or tactics to improve your SEO?

“All these factors affect organic search,” Gupta says. “It’s hard to understand the effect of anything you or your agency has done, unless you can separate these factors.”


Beyond the issue of fighting through the noise of real life on the Web, there is the question of cost. Control tests require both the website owner and its technology partner to dedicate resources (people for instance) to the test for the duration. And by definition, if your initiative is improving your site’s performance, only half your pages are going to benefit from that improvement during the test itself.

“The control only works if you hold things steady. That becomes harder and harder as you extend the time,” Gupta says. “You can’t continue a control indefinitely,” because of the challenges in maintaining a constant state among frequently changing variables.

So, what do you do, once you run your initial control test and confirm that your move was the right one in the first place? As everything changes around you, how do you remain confident that your efforts are continuing to pay off?

Other meaningful metrics 

Why not focus on metrics that don’t require the world to stop dead in its tracks? Why not considering measuring your performance in some lighter-touch and cheaper way?

When it comes to organic search, for instance, you want to make sure that your products can be found and that, when consumers search for products, the products on your site are returned as relevant results.

So, rather than run an entire control test on your site again, why not turn to a metric that can measure the discoverability of your offerings by focusing on the number of clicks it takes a consumer to get to your products and whether that number is rising or falling? Or why not keep track of the relevance of your site by tracking the number of pages on your site that receive external traffic, a key indicator of the effectiveness of your organic search?

Alternatively, take some time to consider the bottom-line metrics that are important to you and figure out what more accessible metrics might serve as proxies for those milestones. Again, look for signs that chart the general direction of your site’s performance.

Coming up with creative and effective ways to measure the progress you’re making toward your goals could save you a lot of money and a lot of aggravation.

And it might even free you up to concentrate on any number of other things on your to-do list — even those that have a lot more to do with your personal well-being than the efficacy of your search strategy.

Photo of test sign by Andy Rennie and test-pattern art by D.Billy published under Creative Commons license.

Mike Cassidy is BloomReach’s storyteller. Contact him at mike.cassidy@bloomreach.com; follow him on Twitter at @mikecassidy.