SRM

Sample Ratio Mismatch

SRM occurs when the actual ratio of users assigned to different variants in an experiment does not match the intended allocation. For example, if an A/B test is supposed to split traffic evenly (50/50), but one variant receives significantly more or fewer users, it indicates an SRM issue.

SRM is a serious problem because it can skew the experiment’s results, making it difficult to trust the data. It often suggests technical issues, such as bugs in the traffic allocation logic, incorrect randomization, or tracking problems. Left unchecked, SRM can lead to misleading conclusions about which variant performs better.

To detect SRM, experimenters use statistical tests to compare the expected and actual traffic ratios. When SRM is identified, it’s essential to pause the test, investigate the root cause, and ensure proper traffic distribution before continuing or restarting the experiment.

• Abbreviation: SRM