Table 2.
Approach to multiplicity due to multiple outcomes
| a) Review: multiplicity approach taken | ||||
| Formal adjustment | Hierarchical testing | Other approach | None | |
| PRIMARY OUTCOMES | ||||
| All outcomes to be declared effective | 0/8 | 2/8a | 0/8 | 6/8 |
| One or more outcomes to be declared effectiveb | 4/20c | 1/20d | 2/20e | 13/20 |
| SECONDARY OUTCOMES | ||||
| Secondary outcomes | 14/136f (10%) | 1/136g (1%) | 2/136h (1%) | 119/136 (88%) |
| b) Survey: responses to posed scenarios | ||||
| Yes | No | Unsure | ||
| PRIMARY OUTCOMES | ||||
| Consider a parallel group trial with two primary outcomes. Would you adjust for multiplicity in the following scenarios? | ||||
| The trial hypotheses require both null hypotheses to be rejected? | 9/27 (33%) | 16/27 (59%) | 2/27 (7%) | |
| The trial hypotheses require either null hypothesis to be rejected? | 16/27 (59%) | 8/27 (30%) | 3/27 (11%) | |
| SECONDARY OUTCOMES | ||||
| Would you adjust for multiplicity arising from multiple secondary outcomes? | 3/27 (11%) | 20/27 (74%) | 4/27 (15%) | |
| Would the type of outcomes (efficacy, safety, cost-effectiveness) have an impact on your response to the above question? | 9/27 (33%) | 17/27 (63%) | 1/27 (4%) | |
Notes: aBoth trials sequentially tested two outcomes
b Includes 13 trials that stated multiple primary outcomes in the methods section, and seven that stated only one outcome but made multiple comparisons
c Two trials performed a Holm correction, and two implemented a graphical multiple testing procedure
d Trial sequentially tested non-inferiority then superiority
e One trial recommended that p-values between 0.025 and 0.05 were considered to have borderline significance; one trial performed post-hoc analysis of the primary outcomes with one-sided 97.5% confidence intervals
f Three trials performed a Bonferroni correction, five a Holm correction, one a Hochberg correction, three used a 1% threshold for significance and two used a graphical method
g Formal hypothesis testing was only performed for secondary outcomes if the primary efficacy outcome was statistically significant
h Formal hypothesis testing was only performed for a small number of key secondary outcomes, other secondary outcomes were just presented descriptively