Other method for adjustment of multiple testing exists

Editor—Perneger’s paper on Bonferroni adjustments consists almost entirely of errors.¹ He states that the Bonferroni adjustments are concerned with the wrong hypothesis and that the two groups are identical on all 20 variables (the universal null hypothesis). This misses the main point of multiple test adjustments.

Similarly he says, “If one or more of the 20 P values is less than 0.00256 ... we can say that the two groups are not equal for all 20 variables, but we cannot say which, or even how many, variables differ.” Researchers who adjust P values almost always present them for their individual hypotheses. With n hypotheses each tested at level α, Perneger claims that “the formula for the error rate across the study is 1−(1−α)ⁿ.” This formula assumes independence of the test statistics; the actual bound on the error probability is nα.

Perneger sees multiple adjustment as a violation of common sense, as a given comparison will be interpreted differently according to how many other tests were performed. In other words, it’s all right to dredge your data and not tell anyone.

Perneger queries whether adjustment should take place for each investigator—”taking the number of tests he or she has done in their lifetime into consideration.” None but opponents of multiple adjusting have ever suggested this absurd idea.

“The integration of prior beliefs with evidence is best achieved by Bayesian methods, not by Bonferroni adjustments.” Bayesians compute probabilities for simultaneous statements about multiple variables—which is just their way of adjusting. There is nothing new, and no solution here.

Perneger takes it for granted that the Bonferroni method should be used for multiple testing adjustments, whereas it has been known for almost 20 years that there is another procedure, the Holm method, that is uniformly superior to the Bonferroni method and applies in every case that the Bonferroni method does.² This has led the American Journal of Public Health to declare this alternative as the method of choice.

If we used hypothesis testing sensibly, computing benefits and costs of right and wrong decisions, and using the resulting optimal decision making procedure, then arguments about multiple adjustment would be unnecessary and we could concentrate on the real question—whether a given study should be statistically analysed at all.