Digital Marketers Need to Understand Statistical Significance

Marketers are relying more and more on data, which is fantastic. There is an important concept that is often skipped by people new to data analysis - statistical significance.

A common area for this error is in reviewing PPC ads and landing pages: ad copies, designs, sizes, conversion rates, CTR, lead quality etc.

Statistical significance (very basically) is whether the results of your analysis are valid or simply a matter of chance. If more people converted on landing page B than A, is it actually because B is better? Or if you ran the test again maybe A would outperform?

Most of the time the error comes from sample sizes that are too small, or are skewed. For example Ad size 1 has high impressions and Ad Size 2 has very low impressions, but the CTR alone might be presented to show that Ad Size 2 is much better/worse.

Or it could be that Ad size 1 ran on different targeting parameters to Ad size 2 so you are comparing the results of different audiences.

While you can do actual statistical confidence measurement (and you should if there are major decisions being made on the basis of the data), you can also use a calculator like this one: https://neilpatel.com/ab-testing-calculator/

Most of the cases that I see though are basic enough to recognise on sight or gut feel. If the numbers are small enough that just a couple more clicks or conversions would change your results, then don’t rely on the data. You should be confident that if you ran the test again, the results would not change.