Stephen Pratley – a CRO consultant at The Conversion Co – kindly agreed to answer our questions about A/B testing in marketing and why it is important to do it before launching new features.
Let me ask you some general things thirst. What is A/B Test & What can you A/B test?
An A/B test (also known as a split test) is a way of trying out changes on a website, ad campaign, email, or any other marketing asset.
I’ll focus on testing web pages here for simplicity.
Importantly, it “splits” the traffic to each version of your page. Version A, then Version B, A, B, A, B, hence the name.
What this does is make sure that any other influences, like the weather, the day of the week, big news events, are the same for both versions.
Why is it essential to do A/B test before launching a product/ new feature?
Every new feature has a cost.
It can mean a slower site, a more complex decision for the prospect, or another feature being bumped down the page.
Testing before launching means you know if the benefits of the new feature outweigh it’s costs.
Where do you start when doing A/B test? What are the first steps?
There are two big things that will give you ideas for what to test.
For an existing site, try to uncover the little hurdles and friction in your site that you can remove. We use tools like Hotjar a lot to watch what people are doing and we do lots of user tests where we can get feedback as users go from page to page.
The other end of the equation is to find out what your customers REALLY want. Uncovering their real motivations can help you highlight features, and change little things like headlines, which can make a massive difference.
Unstructured research like that should be shared with your copywriters and as much of your marketing team as possible. Most companies don’t do nearly enough of it.
When you have some ideas for what to test, you need to prioritize.
Tests need traffic, so for most smaller businesses they’ll focus on where they have the most traffic – at the top of their funnel.
That means emails, advert creatives and landing pages.
If you have more traffic, then tests at the end of the funnel, can have a bigger effect. Tests on pricing will have the biggest impact on your profits, but they’re the hardest to do.
What was the most unexpected result after doing A/B test? Any unpredictable results? Outstanding cases?
Pricing always surprises us a lot. I’ve seen loads of cases where increasing pricing increases the volume of sales made, not just profits.
Something about the perception of value changes when you play with price.
Photos can also have a huge effect when you hit on something that works. I put one in of a client giving a TedX talk and we’ve not beaten that in 2 years of testing. That’s very rare in online.
Something about the authority it gave us was incredibly powerful.
Would you recommend any articles, webinars, books to learn more about A/B test?
The place I started from was when I read “Don’t make me think” by Steve Krug.
It really taught me how much you can improve a site by making it simpler, not by adding more to it.
Another one which will give you some great ideas is “108 Split Tests” by Russell Brunson. A lot of people write him off as a bit spammy, but Russell is one of the best Direct Response marketers working today.
When it comes to implementing the tests, the websites of the tools that you use are the best place.
Google Optimize has a great resource centre:
Lastly, follow @conversioncohq on Twitter where we share insights and a few of the test results that aren’t kept secret by our clients.
We’ll be launching some short courses in the new year that will get beginners from an idea to their first test results.
Thanks Stephen for the interview.