A statistically significant difference between the expected and actual ratios of treatment and control groups in an experiment is a sample ratio mismatch (SRM). This imbalance often occurs in online controlled experiments due to failures in randomization and instrumentation. SRM can be detected using a chi-squared test, helping non-experts avoid decisions based on biased data. Even minor discrepancies in group sizes can invalidate experiment results if the sample size is large enough.
Causes of SRM
SRM occurs when the actual ratio of users assigned to different variants in an experiment does not match the intended allocation. For example, if an A/B test is designed to split traffic evenly (50/50), but one variant receives significantly more or fewer users, it indicates an SRM issue. Common causes include bugs in the traffic allocation logic, incorrect randomization, or tracking problems.
Impact of SRM
SRM is a serious problem because it can skew the experiment's results, making it difficult to trust the data. It often suggests technical issues that can lead to misleading conclusions about which variant performs better. Left unchecked, SRM can compromise the validity of the experiment, leading to incorrect decisions based on biased data.
Detection and Resolution
To detect SRM, experimenters use statistical tests to compare the expected and actual traffic ratios. When SRM is identified, it's essential to pause the test, investigate the root cause, and ensure proper traffic distribution before continuing or restarting the experiment. Addressing SRM promptly helps maintain the integrity of the experiment and ensures reliable results.
