Communication of Randomized Controlled Trial Results Must Match the Study Focus

Dear Editor:

We read with interest the recent article by Hernández-Cordero et al. (1) that adds another data point to a growing body of randomized, controlled evidence demonstrating that sugar-sweetened–beverage-reduction initiatives attempted, to date, do not have large or reliable effects on obesity (2). We express concern about how this article may be interpreted, however, because the article’s title and focus revolves around findings related to a secondary analysis of the primary outcome and a tertiary analysis of a tertiary outcome. By focusing the title and abstract on secondary and tertiary analyses, the article may distract readers from the statistically nonsignificant primary findings in favor of statistically significant exploratory findings.

The choice of which outcomes to analyze and in what ways can influence type I error rates (i.e., the probability of obtaining a statistically significant finding when a true difference does not exist). To put the issue into perspective in this particular case, the authors prespecify the following outcomes in the trial registry (NCT01245010): a single primary outcome of blood TGs and at least 13 secondary outcomes including body weight, fasting insulin, fasting glucose, homeostasis model assessment, HDL cholesterol, LDL cholesterol, total cholesterol, systolic and diastolic blood pressures, waist circumference, glycated hemoglobin, hydration status (assumed to mean serum and urine osmolality), and 24 h-dietary recalls (each of which includes numerous data points to analyze). However, insulin and homeostasis model assessment are not mentioned in the published article, but the authors add metabolic syndrome, BMI, and physical activity to their measurements listed in the article that were not in the trial registry document. The authors declare a primary intent-to-treat analysis and a secondary analysis stratified by overweight and obese status in the article, but not in the trial registry. Therefore, there are at least 12 outcomes planned or added that were compared in at least 2 ways for a total of 24 declared comparisons.

With a type I error rate set to 0.05 across all comparisons, and assuming all comparisons were independent, one would expect to see an average of 1.2 significant comparisons across all similar studies if there were no true differences between treatments. Indeed, they reported 1 nominally significant result from their secondary analysis of the primary outcome (TGs stratified by weight status; P = 0.02), which would become nominally insignificant if corrected for multiple comparisons (the threshold for nominal significance at α = 0.05 when Bonferroni corrected for 24 comparisons is 0.05/24 = 0.002).

The Bonferroni correction mentioned previously, however, may not be correct because it is unclear how many comparisons were actually calculated. The article also includes a tertiary analysis of a tertiary outcome that was not registered with clinicaltrials.gov. The use of change-in-physical-activity as a postrandomization covariate, which has been discouraged (3), in the logistic regression for metabolic syndrome prevalence implies that more comparisons were conducted in producing this publication than we are aware [so-called “researcher degrees of freedom” (4)]. To make the concern more clear, consider that conducting 59 uncorrected, independent tests gives a 95% probability of obtaining at least 1 significant P value under the null (5). Reporting the total number of comparisons investigated provides the reader with information to evaluate the probability of the study producing significant findings by chance, especially when no multiple comparison corrections are made.

Besides the title, other aspects of reporting in this publication are consistent with a bias toward nominally significant results: the nominally significant results garnered the only results-based figure in the article (Figure 2) and were afforded more text in the Results section (25 lines) compared with the 12 primary analysis comparisons (13 lines).

Exploratory data analyses can lead to interesting hypotheses and important investigative avenues, but even the strongest proponents of exploratory data analysis suggest such analyses “should be described as such, and conveyed with a strong note of caution” (6). We commend the authors for what appears to be a well-conducted study and note that we would not have been able to critique these finer points of reporting and analysis had the authors not reported their results as thoroughly as they have. However, this study should have been appropriately titled and communicated to recognize the design and main findings as planned, rather than using “specific reporting strategies… to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome” (7).