AA Testing

Before starting an A/B test, we run an A/A test to make sure that:

A/A tests are similar to A/B tests, but in this case each group is shown the same version of a page. If the traffic and the tool for carrying out the A/A test work as they should, there won't be a (significant) difference in the results. A/A tests also help determine how long the A/B test should last and the method for analyzing the results.

Here are the criteria for a successful A/A test:

The extent to which the key metrics differ among groups depends on how sensitive the experiments needs to be.

Type I and Type II Errors in Testing Hypotheses. Power and Significance