## A technical walk through Hypothesis Testing

When facing the decision on whether to deploy a marketing activity to an entire customer base, from a company’s strategic point of view: measuring and testing the level of significance of the incremental effect that could be achieved by implementing the campaign via a “sample” of the base, could make the whole difference between succeed (“high” return on investment) and/or failure (monetary and even reputation loss). In this arena, statistics becomes the best tool to go to.

If you start reading this post, you might be wondering if this is related to A/B Test. Yes, it is. However, the focus I am given to this post is more on the statistical evaluation of the results obtained from an A/B Test rather than the detail explanation behind the methodology (i.e. design, planning and running of the test). By doing this, I am able to keep the decision-making side at the core of the analysis.

Some background information first. An A/B test is a methodology that can be used to run an experiment on two variants, A and B. These might be known by different names, most commonly a Control and Treated group where one is seeking to identify any difference between these two. As such, statistical analysis is used to test the significance of such difference. Among the different elements to consider when setting up an A/B test are: 1) a randomized selection process where any individual composing the population has an equal probability to be selected and be part of the “samples” 2) the random “samples” are representative of the different groups within the population hence not biased 3) and that the “samples” are of considerable size to run any test and analysis on. Let us think of the big population out there that because of lack of resources and time we are unable to measure it all.

In order to be able to conduct any investigation under a given timescale samples from the population are drawn based on the above criteria so inference about the properties of the population can be made and questions under the investigation can be answered. As such, one wants to make sure the sample size that will be taken from such population is large enough from which a given level of confidence can be placed on when evaluating the results. How large is large? might be a question at this point. As mentioned above, this post does not expand on the methodology behind an A/B test and the…