In a lead generation system, you're gonna work with a lot of numbers. It’s inevitable. And if you're performing some landing page tests, you’re gonna have to tell other people what they mean! “I’m in marketing!” you may exclaim. “I didn’t sign up for this because I’m good at math!” Well the truth is, today’s marketers have access to so many tools that statistical analysis is becoming a major part of the job.
Handing off the A/B test results to an interpreter just doesn’t happen, and now you’ve got to be the one crunching the numbers for your boss. As someone who took one statistics course in college and feels way overqualified because of it, let’s get down and dirty with these numbers and talk about how to interpret your landing page tests to make your boss happy.
Sample size
Working with an appropriate sample size is the first key to running good A/B tests. If your sample size, the amount of times you collect data, is too small, your data will be unreliable. If you're only asking two people whether they prefer red or blue, you're either gonna find that 0%, 50%, or 100% of people prefer either. Not good. Working with a huge sample size, however, takes a lot of time and a lot of energy, and for some lead generation systems, waiting for 10,000 responses could take months. So how can you figure out how many samples you need? Try a helpful tool like Evan Miller's Sample Size Calculator, which does the math for you to get reliable data. Some of the numbers and fields on there may look scary. Basically, put in your current data for what you’re measuring under Baseline Conversion Rate, then under Minimum Desirable Effect put in your...
Margin of Error
So your landing page tests have been running, and you've gotten an appropriate sample size to work with the data. The data comes back and looks a little funny. Any good statistical data will show up as a percentage ± (plus or minus) another percentage. What does this mean? Well, the margin of error is basically how confident your data is. If your data shows a clickthrough rate of 15% ± 2%, that means anything from 13% (15-2) to 17% (15+2) could be correct. This is extremely common with any statistical analysis, so don’t worry too much.
So let’s say that you started with a conversion rate of 10% ± 2%, and your test page showed a conversion rate of 15% ± 2%. This isn’t necessarily a 5% increase. Your improvement could show it went from 8% (the low end of our original conversion rate) to 17% (the high end of our test rate), which is a pretty huge jump. It could also be interpreted as going from 12% (the high end of our original rate) to 13% (the low end of our test rate): a pretty small jump. However, as long as these intervals don't overlap, it's safe to say that the data is significant. If they do overlap, the difference between our original page and our test page may be negligible or impossible to define, and is thus interpreted as an insignificant find. Basically, nothing happened, start over.
Understanding results
We've all seen advertisements that claim conversion rate increases in the thousands of percentage points. But how is that possible? 100% is the most any conversion rate could be, right? Well, yes and no. If 100% of people convert, everyone that goes to the page is converting, and congratulations! That’s amazing. But if your original page had a 25% conversion rate, and you’ve gone up to 100% conversion, you might think it’s a 75% increase, but let’s do the math. By taking the test conversion percentage, 100%, and dividing it by your original page’s percentage, 25%, it turns out you’re actually getting 400% of your original conversions (a 300% increase, since your original page gets 100% of its own conversions)! Looking at our data from before, an 8-17% jump would mean you’re getting (.17/.08=2.13) 213% of your original conversions (a 113% increase). Not bad! But that same data could be interpreted as (.13/.12=1.08) 108% of your original conversions, or only an 8% increase. So what do you tell your boss? It’s easy to fudge the numbers, but as long as your margins and raw numbers are understood, you can present it any way you like. Just don’t lie to your boss. That never ends well.
We hope this guide to understanding A/B test results is helpful for you to start working up your own landing page tests and building up your conversion rates. Though we are committed to helping your lead generation however we can, boberdoo.com is primarily a lead distribution service. If you are in need of leads, we would be happy to help connect you to our clients with a quick phone call. And if you’d like to generate your own, straight from your boberdoo system… well, keep an eye out in the coming weeks. We’re working on a nice surprise!