Multiple Comparisons: p-Hacking vs Bonferroni |…

What Multiple Comparisons Are

Multiple comparisons refer to the statistical problem that arises when conducting numerous hypothesis tests on a single dataset. Each test has its own chance of producing a false positive, and as more tests are performed, this probability increases significantly.

This phenomenon is particularly problematic in fields like genetics or neuroscience where researchers might test thousands of genes or brain regions for associations with a trait.

Why It Happens

The issue stems from the basic properties of p-values. A p-value represents the probability of observing data at least as extreme as the one observed, assuming that the null hypothesis is true. When conducting multiple tests on a single dataset, the chance of obtaining a p-value below the significance threshold (often 0.05) increases.

Without proper adjustments, this leads to an inflated family-wise error rate, where more than the intended proportion of hypotheses are incorrectly rejected as significant.

live demo · related simulation● LIVE

p-Hacking and Bonferroni Correction

p-Hacking is a term used to describe the practice of manipulating data or testing multiple hypotheses until a desired result (a statistically significant finding) is achieved. This can lead to false discoveries and misleading conclusions.

The Bonferroni correction, on the other hand, adjusts the significance threshold by dividing it by the number of tests performed. For example, if you are performing 20 tests, each test should be considered significant only at a p-value of 0.05/20 = 0.0025 to maintain an overall error rate of 5%. This method is conservative but effective in controlling for the family-wise error rate.

Real-World Applications

In clinical trials, multiple comparisons can lead to false positives if not properly controlled. For instance, testing various doses of a drug might result in some showing significant effects purely by chance without any real therapeutic benefit.

In financial markets, traders might perform numerous statistical tests on historical data to find patterns that do not exist, leading to poor investment decisions.

Frequently asked questions

What is p-hacking and why should I avoid it?

p-Hacking involves manipulating data or testing multiple hypotheses until a desired result is achieved. It can lead to false discoveries and misleading conclusions, undermining the integrity of research findings.

How does Bonferroni correction work in practice?

Bonferroni correction adjusts the significance threshold by dividing it by the number of tests performed. For example, if you are performing 20 tests, each test should be considered significant only at a p-value of 0.05/20 = 0.0025 to maintain an overall error rate of 5%. This method is conservative but effective in controlling for the family-wise error rate.

Can Bonferroni correction be too strict?

Yes, Bonferroni correction can be overly conservative, potentially leading to a high rate of Type II errors (false negatives) where true effects are missed due to the stringent threshold for significance.

Are there alternatives to Bonferroni correction?

Yes, other methods like the False Discovery Rate (FDR) control procedures or permutation tests can be used as alternatives. These methods offer a balance between controlling false positives and maintaining statistical power.

Try it live

Everything above runs in your browser — open Multiple Comparisons: p-Hacking vs Bonferroni and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Multiple Comparisons: p-Hacking vs Bonferroni simulation