Undisclosed outcome switching, undisclosed analysis switching, inappropriate rounding, and selective reporting render paper “Effectiveness of a minimally processed food-based nutritional counselling intervention on weight gain in overweight pregnant women: a randomized controlled trial” unreliable

The paper by Sartorelli et al.¹ addresses a topic of great current interest. Indeed, the question of the effects of foods classified as ultra-processed on body weight and other aspects of health and behavior have been the subject of much recent writing and spirited scholarly debates.² Although some of the authors here have been on opposites sides of those debates, we all stand together for good science. In estimating and testing the effects of assignment to treatments, there is no substitute for rigorous randomized controlled trials (RCTs).³ Therefore, we commend Sartorelli et al. for conducting an RCT testing the effects of assigning people to receive diets involving more or less food classified as ultra-processed. At the conclusion of their abstract, Sartorelli et al. state, “The present study was unprecedented in demonstrating that nutritional counselling based on the NOVA food classification system, together with encouraging the practice of physical activity, is effective in preventing excessive weight gain in overweight pregnant women”. This is a strong statement. Do the results warrant the claim that effectiveness has been demonstrated?

Unfortunately, multiple concerns about distortions in reporting lead to considerable skepticism that this result meaningfully reflects the true causal effect and even more so that the reporting of the results of this study are commensurate with current standards of good reporting practices.

We note the following.

1. Undisclosed Outcome Switching.

In the published study protocol,⁴ the authors say “the primary outcome of the study is adequacy of weight gain,” which is consistent with the trial registration⁵ stating that the primary outcome is:

“Mean difference of 20% in the proportion of pregnant women with adequate weight gain according to the criteria of the Institute of Medicine (IoM)”.

Interestingly, the manuscript reported that there were no significant differences in adequacy of gestational weight gain (GWG) between the intervention and control groups. But rather than utilize the pre-specified primary outcome, the authors shifted their focus from adequacy of weight gain to excessive weight gain when they stated in the manuscript:

“The primary outcome was the proportion of women whose weekly GWG exceeded the Institute of Medicine (IoM) guidelines.”

Furthermore, the sample size calculations in the manuscript were claimed to be based on “a 20% difference in the proportion of excessive weight gain between the treatment groups”.

While the revised primary outcome was reported to be marginally significantly different between intervention and control groups, its significance would not survive adjustment for even the single additional statistical comparison to the pre-specified primary outcome. Of course, the lack of a significant effect on any measure of GWG might not be surprising given that the intervention failed to demonstrate any significant difference in either ultra-processed food intake or physical activity.

Undisclosed outcome switching is inconsistent with ethical and reporting standards and biases the research record towards unreliable findings of illusory statistical significance. As noted elsewhere: “Importantly, we emphasise that changing outcomes is not always a bad thing. There can be many sound reasons for so doing. It is covert changing of outcomes without full and transparent reporting of the changes (or a statement of rationale) that is unacceptable. Failure to declare such changes means that those trial publications are dishonest, misleading, and potentially harmful to patients; it contravenes the Declaration of Helsinki”.⁶ When taken together, the statements in the manuscript, the clinical trials registration, and the published protocol are consistent with research misconduct involving “changing or omitting data or results such that the research is not accurately represented in the research record”.⁷

2. Inappropriate Rounding.

In the body of the manuscript, the p-value for the modified intent to treat analysis is listed as 0.049. In contrast, in the abstract, the p-value was listed as 0.04. This is inconsistent with standards about numerical rounding,⁸ and is a second example of misleading reporting that exaggerates results.

3. Switching from non-significant preplanned analysis to statistically significant alternative.

The protocol paper for the study⁴ states, “The analyses of this study will follow the intention to treat principles.” No mention of a modified intent-to-treat (ITT) analysis is mentioned in the protocol paper, and the published grant abstract⁹ has no information on analysis. There is no statistically significant effect for an ITT analysis. The statistically significant effect is only present for the modified ITT analysis. However, the primary analysis is described as following “the intention to treat principles”. Yet this is not disclosed in the manuscript. This again seems misleading and fails to meet best reporting practices of parallel group randomized controlled trials.¹⁰

When these items are collectively taken into account, it seems that this study is being reported in a way that misleads readers and renders the current manuscript unreliable. The contrast is especially stark between an objective analysis of these results and the stated conclusion that the results provide an “unprecedented” demonstration of effectiveness. We recommend that this paper be corrected or retracted.