AA Testing
Before starting an A/B test, we run an A/A test to make sure that:
- The results are not affected by anomalies or outliers in the statistical population
- The tool for splitting traffic works correctly
- Data is sent to analytical systems correctly
A/A tests are similar to A/B tests, but in this case each group is shown the same version of a page. If the traffic and the tool for carrying out the A/A test work as they should, there won't be a (significant) difference in the results. A/A tests also help determine how long the A/B test should last and the method for analyzing the results.
Here are the criteria for a successful A/A test:
- The number of users in different groups doesn't vary by more than 1%
- For all groups, data on the same event is recorded and sent to analytical systems
- None of the key metrics varies by a statistically significant amount—typically by no more than 1%
- Users remain within their groups until the end of the test. If they see different versions of the page during the study, it won't be clear which version influenced their decisions, so the reliability of the results will be compromised
The extent to which the key metrics differ among groups depends on how sensitive the experiments needs to be.
Type I and Type II Errors in Testing Hypotheses. Power and Significance