Abstract
This cross-sectional analysis of 71 articles discussing the ACOSOG Z6051 and ALaCaRT trials of laparoscopic vs open resection for rectal cancer evaluates the frequency of misinterpretation of these inconclusive trials and determines whether interpretations were concordant with the findings of the primary studies.
Noninferiority clinical trials are designed to determine whether an intervention is not worse than a comparator by a prespecified difference, known as the noninferiority margin (ΔNI). These trials are useful when comparing standard therapies with novel treatments that may be easier to use, are less costly, or have fewer adverse effects.1 The number of noninferiority trials is increasing; however, particularly compared with superiority trials, the interpretation of noninferiority trials is not straightforward. Superiority trials can be classified as positive or negative based on whether the null hypothesis can be rejected (ie, P < .05). In contrast, noninferiority trials can be interpreted as noninferior, inconclusive, or not noninferior according to the location of the 95% CIs in relation to the prespecified margin (Figure). Inconclusive and not noninferior results are often both presented as “negative” results (eg, failure to meet noninferiority criteria). This can lead readers to erroneously interpret an inconclusive trial as showing evidence that the novel treatment is worse, while in reality the findings are indeterminate and further research is required to determine noninferiority.
Figure. Consolidated Standards of Reporting Trials (CONSORT)–Recommended Interpretation of Noninferiority Findings and Results of the ACOSOG Z6051 and ALaCaRT Trials2,3.
Error bars indicate 95% CIs. A, Shown are examples of noninferiority (NI) trial results and their interpretation according to the CONSORT recommendations. In example 1, if the 95% CI lies wholly to the left of zero, the new intervention is superior. In examples 2 and 3, if the 95% CI lies to the left of ΔNI and includes zero, the new treatment is noninferior but not shown to be superior. In example 4, if the 95% CI lies wholly to the left of ΔNI and wholly to the right of zero, the new treatment is noninferior in the sense already defined, but it is also inferior in the sense that a null treatment difference is excluded. In examples 5 and 6, if the 95% CI includes ΔNI and zero, the difference is not significant, but the result regarding noninferiority is inconclusive. In example 7, if the 95% CI includes ΔNI and is wholly to the right of zero, the difference is statistically significant, but the result is inconclusive regarding possible inferiority. Last, in example 8, if the 95% CI is wholly above ΔNI, the new treatment is not noninferior (looking-glass opposite of noninferior as inferior is also used to refer to the opposite to superior). The vertical dotted lines indicate the noninferiority margin (∆NI). B, Results of the ACOSOG Z6051 and ALaCaRT trials are presented as risk differences and 95% CIs with their respective ΔNI. The ACOSOG Z6051 noninferiority margin is risk difference 6.0 (Δ2); the ALaCaRT noninferiority margin is risk difference 8.0 (Δ1). As seen, the results of the ACOSOG Z6051 trial correspond to example 6, and the results of the ALaCaRT trial correspond to example 7. The vertical dotted lines indicate the noninferiority margin from the ACOSOG Z6051 trial (∆1) and the ALaCaRT trial (∆2).
To better understand how often inconclusive noninferiority trials are misinterpreted, we studied the interpretation of 2 recent noninferiority trials (American College of Surgeons Oncology Group [ACOSOG2] Z6051 and Australasian Laparoscopic Cancer of the Rectum [ALaCaRT3]) evaluating the surgical treatment of rectal cancer that had inconclusive results regarding the noninferiority of laparoscopy in terms of quality of surgical resection (Figure). Both trials used correct yet ambiguous wording (ie, failed to meet the criteria for noninferiority) to report their findings.2,3 Publication of these 2 randomized clinical trials (RCTs) reignited concerns regarding the oncologic safety of the laparoscopic approach for rectal cancer.2,3 To evaluate the frequency of misinterpretation of these inconclusive trials, we examined publications citing the ACOSOG Z6051 and ALaCaRT trials and determined whether interpretations were concordant with the findings of the primary studies.
Methods
We used Scopus to identify articles citing the ACOSOG Z6051 and ALaCaRT trials2,3 from their publication (October 6, 2015) to February 15, 2018. We excluded articles not published in English, book chapters, and news pieces. Full texts were reviewed to identify articles that discussed the main findings of the trials. Sentences discussing the RCT results were extracted and independently reviewed by 2 of us (S.A.A. and F.D.). We recorded whether articles complied with the Consolidated Standards of Reporting Trials (CONSORT) recommendations4 for the interpretation of noninferiority trials (Figure) and whether the authors acknowledged the inconclusive nature of the results. No institutional review board approval of the study was required.
Results
We identified 150 unique publications citing the ACOSOG Z6051 and ALaCaRT trials,2,3 with 130 articles having available full text in English. Seventy-one of 130 articles (54.6%) discussed the main findings of at least 1 of the trials. A substantial proportion of articles (20 of 71 [28.2%]) incorrectly interpreted the RCT findings as suggestive of inferiority of laparoscopic surgery (Table). Although the remainder of the articles included interpretations compatible with the CONSORT recommendations, most (38 of 51 [74.5%]) used ambiguous wording akin to that used in the trials (ie, failed to meet the criteria for noninferiority). Less than one-quarter of the publications (12 of 51 [23.5%]) acknowledged that the results were inconclusive.
Table. Categorization of the Interpretation of the ACOSOG Z6051 and ALaCaRT Trials2,3 and Examples.
| Category | No. of Studies | Type of Studies | Example | Explanation |
|---|---|---|---|---|
| Incorrect interpretation | 20 | 6 Overviews (30.0%), 8 observational studies (40.0%), 6 commentaries/editorials (30.0%) | “…these findings confirm concerns that laparoscopic surgery may lead to more cancer recurrences and shorter survival. …ACOSOG Z6051 currently providing data that laparoscopy does not provide the same quality of resection as open surgery for rectal cancer”5 | Suggests that laparoscopy met criteria for inferiority (ie, not noninferior) |
| Ambiguous wording | 39 | 12 Overviews (30.8%), 16 observational studies (41.0%), 2 commentaries/editorials (5.1%), 3 systematic reviews and meta-analyses (7.7%), 2 clinical practice guidelines (5.1%), 4 other (10.3%) | “Some have concerns that oncologic outcomes may be compromised with the laparoscopic approach, especially for rectal cancer. Two recent randomized clinical trials failed to show that laparoscopy was noninferior to open surgery in a composite score of immediate oncologic outcomes”6 | Does not convey to the reader whether laparoscopy was not noninferior or whether results were inconclusive |
| Acknowledges inconclusive nature of results | 12 | 2 Overviews (16.6%), 3 observational studies (25.0%), 4 commentaries/editorials (33.3%), 1 systematic review and meta-analysis (8.3%), 1 clinical practice guideline (8.3%), 1 trial (8.3%) | “Overall, both trials were unable to establish noninferiority for pathological outcomes in comparison of laparoscopic and open resection for rectal cancer…these results have been used by some commentators to suggest that laparoscopy has had its day when it comes to treating rectal cancer. However, it is important to realize that failure to show non-inferiority cannot be used to imply inferiority…”7 | Correctly acknowledges that trial results are inconclusive and thus conclusions regarding the noninferiority of laparoscopic to open surgery cannot be drawn |
Discussion
This cross-sectional analysis of 71 articles discussing the ACOSOG Z6051 and ALaCaRT trials2,3 demonstrates that noninferiority trials with inconclusive results are often misinterpreted or described using correct yet ambiguous language that may obscure the true trial results. Most previous analyses of noninferiority trials have focused on quality and completeness of reporting of design variables and results.
Limitations
This study has some limitations. These include the use of Scopus, which may not have accurately identified all articles citing the 2 trials studied. Moreover, our findings might not be generalizable to other noninferiority trials.
Conclusions
Our analysis is the first that we know of to explore the interpretation of inconclusive noninferiority trials among knowledge users. Appropriate interpretation of these studies is important because decisions about the use of new therapies increasingly rely on data from noninferiority trials. Explicit statements in publications of noninferiority trials when the findings are inconclusive would aid interpretation and avoid erroneous conclusions that novel treatments are inferior.
References
- 1.Kaul S, Diamond GA. Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med. 2006;145(1):62-69. doi: 10.7326/0003-4819-145-1-200607040-00011 [DOI] [PubMed] [Google Scholar]
- 2.Fleshman J, Branda M, Sargent DJ, et al. Effect of laparoscopic-assisted resection vs open resection of stage II or III rectal cancer on pathologic outcomes: the ACOSOG Z6051 randomized clinical trial. JAMA. 2015;314(13):1346-1355. doi: 10.1001/jama.2015.10529 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Stevenson AR, Solomon MJ, Lumley JW, et al. ; ALaCaRT Investigators . Effect of laparoscopic-assisted resection vs open resection on pathological outcomes in rectal cancer: the ALaCaRT randomized clinical trial. JAMA. 2015;314(13):1356-1363. doi: 10.1001/jama.2015.12009 [DOI] [PubMed] [Google Scholar]
- 4.Piaggio G, Elbourne DR, Pocock SJ, Evans SJ, Altman DG; CONSORT Group . Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. JAMA. 2012;308(24):2594-2604. doi: 10.1001/jama.2012.87802 [DOI] [PubMed] [Google Scholar]
- 5.Burstein HJ, Krilov L, Aragon-Ching JB, et al. Clinical cancer advances 2017: annual report on progress against cancer from the American Society of Clinical Oncology. J Clin Oncol. 2017;35(12):1341-1367. doi: 10.1200/JCO.2016.71.5292 [DOI] [PubMed] [Google Scholar]
- 6.Carmichael JC, Keller DS, Baldini G, et al. Clinical practice guidelines for enhanced recovery after colon and rectal surgery from the American Society of Colon and Rectal Surgeons and Society of American Gastrointestinal and Endoscopic Surgeons. Dis Colon Rectum. 2017;60(8):761-784. doi: 10.1097/DCR.0000000000000883 [DOI] [PubMed] [Google Scholar]
- 7.Stevenson AR. The future for laparoscopic rectal cancer surgery. Br J Surg. 2017;104(6):643-645. doi: 10.1002/bjs.10503 [DOI] [PubMed] [Google Scholar]