Digital Ballgame: Why A/B Testing All Day Long Works!

Buzzfeed, Amazon, and Netflix know this: The more you test, the better you make a connection to your customers’ wishes. A/B tests are a great way to get these insights.

In his recent book Hacking Growth, startup gurus Sean Ellis and Morgan Brown tell an instructive story. In 2007 Art Briles took over as head coach of the Baylor University football team. Soon, the Baylor Bears were averaging an impressive 64 points a game and heading to bowl games every year as one of the nation’s top-ranked teams.

What was the secret to the Baylor Bears’ success? “By running plays faster and shortening the time between plays, Baylor was able to run about 13 more offensive plays per game than their competitors,” Ellis and Brown wrote. This translated into much more experiential learning  about which plays work best and in what circumstances.

What Briles and his team did is the core of A/B testing. Also known as a “split-run test,” an A/B test is a controlled experiment in which 2 versions of a single variable are compared by testing a subject’s response to variable A against variable B. This will determine which of the 2 variables is more effective.

In marketing, A/B testing has proven to be very effective. Because of the scientific background of this type of testing, this task may seem a little intimidating. You don’t have to be a developer to run these tests, though. So, don’t get cold feet, start with split-run testing today! As I describe below, you can optimize each stage of your process through A/B testing.

Everything begins with a product or service you want to sell. Through A/B testing, you are able to keep improving your product.

This means that you make little variations in the product and test which version has the best market-fit. But this process never stops. Depending on your product or market, new tests will be run after longer or shorter timespans.

Case Study: Netflix

Netflix is the poster boy for data-driven creativity. Almost all their decisions are based on data analytics and A/B testing.

The Netflix streaming experience with minimal playback interruptions and high-quality video is a case in point. Through split-run testing, Netflix selects the best artwork for videos too, sometimes resulting in 20% to 30% more viewing for that title.

Your product needs to have a price. Or does it? It can have different prices too.

Through split-runs, you test the effectiveness of your price by showing different prices to different groups of website visitors (segmented in customer type). For each group, a price is calculated which ensures the highest profit. Each group will now see the price that is calculated for them. Variations in prices based on time of day are also possible.

Case Study: Amazon

In 2000, dynamic and personalized pricing was a revolutionary approach. An outcry followed after an Amazon customer noticed that when he deleted his cookies, he could buy a DVD he wanted for $4 less. Amazon CEO Jeff Bezos apologized, and thousands of customers were refunded.

The pricing was part of a company test offering random prices to determine an optimal price for products, Bezos said at the time. No customer demographic information had been used to determine the prices being offered, though.

What was controversial 18 years ago is a good practice among etailers everywhere today. Based on customer data and Internet behavior over different devices, web retailers such as Amazon know which prices are most effective for which customers at which moment. The giant corporation changes prices up to 2.5 million times per day.

The next step is to design and maintain a website for your product. You can optimize this platform through A/B testing, too.

This holds true for every aspect of your website. Copy, photos, font, and — yes — the cliché example of a button’s color. There is nothing on a website you cannot test.

Case Study: Buzzfeed

The news site Buzzfeed shows multiple versions of articles to website visitors. “We A/B test pretty much everything,” BuzzFeed data infrastructure engineer Walter Menendez said during a talk. “Not just the headline, but also the number in the headline, the thumbnail that’s rendered on Facebook or Twitter.”

After a couple of hours, the website visitors’ clicks decide the winning combination of copy, image, and headline. It really works: ranking at 193 in estimated, global popularity, many Buzzfeed articles go viral.

After optimizing your website, you want to convince Internet users to come to visit it. A/B testing will offer a helping hand again.

Advertising online is the main method you can use to drive traffic to your website. This gives you a nice opportunity to A/B test this part of your work process too.

For example, Facebook offers the option of A/B tests with the goal to help you target your audience. You can test different versions of your ads to see what works best and what can be improved in future campaigns.

The best advertising set is determined by comparing the cost per result of each ad set. Facebook’s attribution system calculates which advertising set has the lowest cost per result, such as the cost per purchase on the website.

Case Study: AdEspresso

The makers of Facebook optimization tool AdEspresso noticed that one of their sidebar ads was underperforming. Because the title and the image didn’t seem to be the problem, they decided to A/B test the copy of the ad.

These were the 2 variants:

• LIKE us for daily tips on how to improve your Facebook ad campaigns & increase your ROI! (old version)
• LIKE us for pro tips on how to improve your Facebook ad campaigns today! (new version)

The old version didn’t get a single like; the updated copy drove 70 new likes in just one week. There are a couple of reasons why the new copy was better, but the beautiful thing about A/B testing is, it doesn’t matter. Let the numbers speak for themselves.

A/B testing is a continuous process; start today and become a little better every day! Do you want to know more about split-run testing? Watch this space, because I will return to this subject in due time.