Testing the Hypothesis that Proportions Are Equal

A typical task in statistics is to test hypotheses about the equality of proportions of populations. If one part of a statistical population has a certain feature while another part doesn't, we can make conclusions about the size of that proportion based on a sample taken from the population. As with the mean, sample proportions will be normally distributed around the actual one. Python doesn't have a standard test for this, so we'll write one ourselves.

The difference between the proportions we observe in our samples will be our statistic. That's what you call a variable whose values can be found only from sample data. You can prove that it's distributed normally:

$$ Z \cong \frac{(P_{1} - P_{2}) - (\pi_{1} - \pi_{2})}{\sqrt{P(1-P)(1/n_{1} - 1/n_{2})}} \sim N(0,1) $$

Z is the standard value for a criterion with a standard normal distribution, where the mean is 0 and the standard deviation is 1. All this is stated in the right part of the formula after the '~' sign, which says the expression is distributed as N(0,1).

n₁ and n₂ represent the sizes of the two samples being compared (the number of observations they contain). P₁ and P₂ are the proportions observed in the samples, and P is the proportion in the sample made up of P₁ and P₂. π₁ and π₂ are the actual proportions in the populations we're comparing.

With A/B testing, one usually tests the hypothesis that π₁ = π₂. Then, if the null hypothesis is true, the expression (π₁ - π₂) in the nominator will equal 0 and the criterion can be calculated using only the sample data.

The statistic thus obtained will be normally distributed, making it possible to carry out two-sided and one-sided (bilateral and unilateral) tests. Using the same null hypothesis that two populations' proportions are equal, we can test the alternative hypotheses that 1) the proportions simply aren't equal, or that 2) one proportion is larger or smaller than the other.

from scipy import stats as st
import numpy as np
import math as mth

alpha = .05 # significance level

successes = np.array([78, 120])
trials = np.array([830, 909])

# success proportion in the first group:
p1 = successes[0]/trials[0]

# success proportion in the second group:
p2 = successes[1]/trials[1]

# success proportion in the combined dataset:
p_combined = (successes[0] + successes[1]) / (trials[0] + trials[1])

# the difference between the datasets' proportions
difference = p1 - p2

Let's calculate the statistic in terms of standard deviations of the standard normal distribution:

# calculating the statistic in standard deviations of the standard normal distribution
z_value = difference / mth.sqrt(p_combined * (1 - p_combined) * (1/trials[0] + 1/trials[1]))

# setting up the standard normal distribution (mean 0, standard deviation 1)
distr = st.norm(0, 1)  

If the proportions were equal, the difference between them would be 0. Let's calculate how far from 0 our statistic turned out to be. What is the probability of getting such a difference or a greater one? Since the statistic's distribution is normal, we'll call the cdf() method. We'll get the absolute value of the statistic using the abs() method. This will ensure we get the right result no matter what sign the statistic has. (It could be negative since the test is two-sided.) For the same reason, we double the result:

# calculating the statistic in standard deviations of the standard normal distribution
z_value = difference / mth.sqrt(p_combined * (1 - p_combined) * (1/trials[0] + 1/trials[1]))

# setting up the standard normal distribution (mean 0, standard deviation 1)
distr = st.norm(0, 1) 

p_value = (1 - distr.cdf(abs(z_value))) * 2

print('p-value: ', p_value)

if (p_value < alpha):
    print("Rejecting the null hypothesis: there is a significant difference between the proportions")
else:
    print("Failed to reject the null hypothesis: there is no reason to consider the proportions different")

Normality Tests. The Shapiro–Wilk Test

In real life, many variables diverge from the normal distribution: they contain outliers that cannot be ignored. To test whether datasets are accurately modeled by the normal distribution, we use normality tests.

According to the central limit theorem, sample means are normally distributed around the population's true mean (and sample proportions around the true proportion). This is even true for distributions containing major outliers. This approach is good if you're drawing conclusions based on dozens of samples. Each individual sample could contain outliers that will shift the results and affects the results.

First you need to learn to test the hypothesis that a sample was taken from a normally distributed population.

One simple criterion is χ² (chi-squared). The sum of squared differences between observed and expected values is divided by the expected values:

$$ \frac{\sum (O_{i} - E_{i})^{2}}{E_{i}} $$