Don't Test for Baseline Imbalances Unless They Are Known To Be Present?

Editor: Fayers and King [1] are correct to “not suggest that one should never carry out significance tests on baseline characteristics” but they also state that “significance tests are pointless in a conventional (individual patient) randomized trial that has an effective randomization procedure [and] are usually [worth doing only] if potential violation of the randomization is suspected”. This advice is analogous to drivers wearing a seat belt only when they expect to be in an accident. The flaw in the argument is the circularity in suggesting that one can know the results of an analysis without actually conducting this analysis. That is, “the basis for suspicion of flawed randomization can be the very tests of baseline balance that would not be performed, under this approach, without prior suspicion” [2, page 125]. Moreover, it is not true that in individually randomized trials:

at the time the patient enters into the trial neither the patient nor the clinical team know which treatment will be allocated; the randomized allocation is concealed until after the patient has been registered into the trial. This avoids all possibility of selection bias.

The reality is that highly restricted randomization schemes, such as permuted blocks, allow for the prediction of upcoming allocations. In fact Chapter 3 of [2] enumerates 30 actual trials, almost all of them individually randomized, in which selection bias was at least suspected. Hence, there is always a reason for suspicion, at least until this suspicion has been quelled by a combination of baseline tests and the more sensitive Berger-Exner test [2, page 132].

Fayers and King [1] caution that “Significance tests of imbalance in baseline characteristics are [informative only] if the p-value is extremely small – we expect on average that one in 20 baseline characteristics will be ‘significant’ with p<0.05, and p-values of 0.01 or smaller are not unusual, purely by chance”. One has to wonder if these authors are equally quick to dismiss significant efficacy p-values. Probability is a unitless quantity; one event with probability 0.05 is exactly as likely to occur as any other. Finally, note that selection bias need not condemn a trial to be “abandoned and deemed unworthy of publication”, as special methods [2, Section 7.3.1] can ensure a valid treatment comparison even in the presence of selection bias.