The Peeking Problem: A/B Test Significance (2D) |…

What is an A/B Test?

An A/B test, also known as a split test or bucket test, is a method used to compare two versions of a webpage or app feature to determine which one performs better. The goal is to identify the most effective design or content by measuring user behavior and preferences.

In an A/B test for conversion rates, users are randomly assigned to either the control group (A) or the variant group (B), with each group experiencing a different version of the page or feature.

The Peeking Problem

Peeking at the results of an A/B test before it has reached its conclusion can lead to misleading conclusions. This is because repeated checks can increase the likelihood of detecting a statistically significant difference even when there isn't one, leading to false positives.

In the context of this simulation, repeatedly peeking at the significance level (p-value) can artificially inflate the false positive rate above the nominal alpha level, which is typically set at 0.05.

live demo · related simulation● LIVE

Why It Matters

The consequences of peeking in A/B testing are significant because it undermines the validity and reliability of the test results. This can lead to incorrect decisions about which version of a webpage or app feature is better, potentially resulting in wasted resources and missed opportunities.

Understanding the dangers of peeking helps ensure that tests are conducted properly, leading to more accurate and trustworthy conclusions.

Real-World Examples

For instance, a company might run an A/B test for a new landing page design. If they repeatedly check the results every day, they might incorrectly conclude that the new design is better than the old one, even if there's no real difference in user behavior.

In another scenario, a software developer might implement a new feature and conduct an A/B test to see if users prefer it over the existing version. Repeatedly peeking at the results could lead them to prematurely abandon the current feature for the new one, only to find out later that there was no significant difference in user satisfaction.

Frequently asked questions

What is a p-value?

A p-value is a measure of the probability that an observed result or more extreme could have occurred by random chance. In A/B testing, it helps determine if the difference between two groups is statistically significant.

How does repeatedly checking for significance affect the test results?

Repeatedly checking can lead to a higher false positive rate because each check increases the likelihood of detecting a significant result even when there isn't one, due to random fluctuations in the data.

What is alpha (α) in A/B testing?

Alpha (α) represents the threshold for statistical significance. It's the probability of rejecting the null hypothesis when it is actually true, also known as a Type I error. Commonly set at 0.05.

Why should we avoid peeking during A/B testing?

Peeking can lead to overestimating the significance of results and making incorrect decisions based on false positives, which can be costly in terms of resources and missed opportunities for improvement.

Try it live

Everything above runs in your browser — open The Peeking Problem: A/B Test Significance (2D) and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open The Peeking Problem: A/B Test Significance (2D) simulation