We read with interest the article by Aberegg and colleagues1 on reporting and interpretation of noninferiority trials. Notably, the authors found that only 117 out of 182 (64%) noninferiority comparisons reported both intention-to-treat (ITT) and per-protocol (PP) analyses. Of the five comparisons where results from these analyses differed, the ITT analysis was more conservative in four (80%). In this letter, we extend these results and comment on the findings.
We identified 231 noninferiority randomized controlled trials (NI-RCTs) in the same five journals reviewed from 2005 to 2014. Just 103 (45%) NI-RCTs reported both ITT and PP analyses in 131 comparisons. Eighteen reports omitted ITT analyses and 110 omitted PP analyses. Discrepancies between analyses occurred in 8 of 131 (6%) comparisons. The kappa coefficient for agreement between NI analyses was 0.715, representing good but imperfect agreement. The ITT analysis was more conservative in four of eight (50%) discrepancies.
Discrepancies between ITT and PP analyses in NI-RCTs are common, occurring in approximately 1 in 17 comparisons where both are reported. This is particularly concerning given that one of these analyses is omitted in 55% of NI-RCT publications. The Food and Drug Administration (FDA) and other experts recommend reporting both ITT and PP analyses in NI-RCTs to avoid false noninferiority conclusions based on only one of these analyses.2,3 These recommendations stem from the complementary nature of ITT and PP analyses. ITT analysis includes all randomized patients, therefore preserving randomization and minimizing selection and attrition biases. However, ITT analysis may also attenuate treatment differences in trials with differential dropouts or crossover between groups, which could bias the result towards noninferiority.4,5 Conversely, PP analysis aims to isolate the effect of the intervention by generally excluding dropouts and treatment crossover. In many instances, however, dropouts and crossovers are due to intervention inefficacy or intolerance and associated with patient prognosis, introducing bias and confounding in PP analyses, which produce invalid estimates of effect.3 In most cases, discrepancies between ITT and PP analyses suggest that bias has been introduced into the trial, which requires further analysis and interpretation. Omission of either analysis in the published trial report may therefore conceal study limitations that should preclude a conclusion of noninferiority.
Authors of NI-RCTs need to perform and report both ITT and PP analyses, and conclude that an intervention is noninferior only when both analyses are consistent with this conclusion. Authors should be skeptical of any NI-RCT that omits either analysis.
Contributors
RT conceived the idea. All authors participated in study selection, data extraction, analysis, and manuscript drafting. RT performed all statistical analyses. All authors read and approved the final manuscript.
Compliance with Ethical Standards
The authors declare that they do not have a conflict of interest.
None.
References
- 1.Aberegg SK, Hersh AM, Samore MH. Empirical consequences of current recommendations for the design and interpretation of noninferiority trials. J Gen Intern Med 10.1007/s11606-017-4161-4. [DOI] [PMC free article] [PubMed]
- 2.Guidance for industry non-inferiority clinical trials: draft guidance [Internet]. Silver Spring (MD): U.S. Department of Health and Human Services, Food and Drug Administration. c2010. F. Potential biases in an NI study. p. 33. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM202140.pdf (Accessed 2015 Aug 7).
- 3.Mulla SM, Scott IA, Jackevicius CA, You JJ, Guyatt GH. How to use a noninferiority trial: users’ guides to the medical literature. JAMA. 2012;308(24):2605–2611. doi: 10.1001/2012.jama.11235. [DOI] [PubMed] [Google Scholar]
- 4.Garrett AD. Therapeutic equivalence: fallacies and falsification. Stat Med. 2003;22(5):741–762. doi: 10.1002/sim.1360. [DOI] [PubMed] [Google Scholar]
- 5.Abraha I, Cherubini A, Cozzolino F, et al. Deviation from intention to treat analysis in randomized trials and treatment effect estimates: meta-epidemiological study. BMJ. 2015;350:h2445. doi: 10.1136/bmj.h2445. [DOI] [PMC free article] [PubMed] [Google Scholar]