Multiple comparisons problem

Each test carries its own chance of a false positive, and those chances accumulate. Run enough comparisons and something will cross the line by luck alone. Genomics, neuroimaging, and studies with many outcomes or subgroups are especially exposed, since they may run hundreds or thousands of tests at once.

Corrections trade sensitivity for specificity. The Bonferroni method divides the threshold by the number of tests and is simple but conservative. Controlling the false discovery rate, for instance with the Benjamini-Hochberg procedure, is less stringent and better suited to large-scale screening. The right choice depends on whether the cost of a single false positive or many missed effects is greater.

Reviewers should count how many comparisons a paper actually performed, including the ones mentioned only in passing, and check whether any correction was applied. A single significant subgroup effect buried among a dozen unadjusted tests deserves heavy skepticism.