The Need for Greater Rigor in Childhood Nutrition and Obesity Research

Why Should Pediatric Obesity Science Be Rigorous?

The stakes are high with pediatric obesity research, which guides policy, individual choices, and clinical care. Only the most rigorous science achievable and accurate reporting of the results will enable the soundest decisions. Therefore, under the caveat that all empirical knowledge is provisional, methods must be used that increase the probability of generating accurate knowledge about obesity in childhood and offer successively ever-better approximations to the truth.

Is There a Rigor, Reproducibility, or Replicability Crisis in the Science of Pediatric Obesity?

The rigor literature is rife with the word crisis. If crisis means a previously functional system facing imminent collapse, given the ongoing creation of new knowledge (eg, the identification of novel childhood obesity genes¹) and debunking of false notions (eg, refuting that regularly consuming breakfast increases weight loss¹), we see no crisis. Alternatively, if crisis is defined as a time of intense difficulty, as in the Oxford Dictionary, then the multiple calls for retractions for articles on obesity witnessed recently^2–4 aptly demonstrate difficult times. Coupled with a growing recognition that we should be making more progress toward solving the problems we know exist and that the methods used to ensure scientific rigor in the past are now judged insufficient,⁵ the word crisis seems appropriate to us.

Is There a Special Problem in Pediatric Obesity Research?

We are not aware of any formal evaluation of rigor across fields. Mindful that our opinions reflect our own experiences and expertise, we have observed at least 3 challenges to rigor particularly relevant to the study of childhood obesity.

First, there are inherent methodological difficulties that challenge rigor. For example, by the nature of children’s daily lives, interventions often need to be delivered within clusters (such as the family, classroom, or school district). Clustering reduces power, and there are instances in which misanalysis of clustered childhood obesity interventions has led to unsubstantiated conclusions.^2,4 The low stability of adiposity in childhood increases regression to the mean, which, if unaccounted for, can lead to erroneous inferences. Energy intake and expenditure are difficult to accurately self-report, and if parent reports are needed, then these can add further biases.

Second, we speculate that the emotional climate surrounding childhood obesity can compromise rigor. Gallup polls, the World Health Organization, and the US Centers for Disease Control and Prevention each cited obesity as a top health concern in the past decade. We have witnessed statements regarding a war on childhood obesity and calls for urgent action and perceive a zeal and sense of “moral panic”⁶ around pediatric obesity more so than for other diseases and populations. Context influences the acceptability of breaches in rigor.⁷ In The Righteous Mind: Why Good People Are Divided by Politics and Religion, Jonathan Haidt argues that moral investment leads us to construct world views for which we aggressively pursue proof and avoid falsifying evidence. Karl Popper argued that assessing scientific theories hinges on falsifiability; rigor in pediatric obesity research would be compromised if we sought to preserve our beliefs and reject their falsification. Although this is conjecture, others have questioned whether emotion “alters thoughts and behavior of [obesity] scientists.”⁵

Third, the pediatric obesity literature contains multiple examples of reduced rigor, including from top-ranking research institutions.³ Editorials call for scientists to “determine those [assumptions about obesity] which are truly evidence based, and improve the rigor of [obesity] science,”⁵ and the August 2018 issue of the Internal Journal of Obesity contained 4 articles describing previous research errors. Collectively, this set of observations does not prove but does suggest that increasing rigor in pediatric obesity research merits particular attention.

So What Can We Do?

The national conversation on improving scientific rigor makes it clear that there is no single solution. We do not profess to have all the answers, and we recognize the need to constantly examine our own research practices for rigor. However, our first suggestion is to emphasize separating the issue of transparency and rigor in research reporting from the extent of rigor in other elements of the research. The former should be uncompromisable; sufficient detail on experimental design, procedures, and analysis must be given to allow readers to judge the extent to which the research is likely to produce veridical knowledge. Rigor in the methods is sometimes compromisable (eg, with lower priority or more exploratory research) when communicated transparently.

Second, we must apply sound inductive reasoning and not overreach our data when drawing conclusions and communicating results. We are aware of multiple articles in which child obesity interventions have not been found to be effective (which is not to say they have been found to be ineffective), yet the articles declare that effectiveness has been demonstrated based on subset analyses, secondary end points, or incorrect analyses.^2,4 There are numerous factors that could lead to such exaggeration of results, such as the aforementioned passion, a priori expectations for the results,⁸ or the challenges inherent in navigating a career through academia. However, weak evidence remains weak evidence. We may choose to act on weak evidence, but we must acknowledge it truthfully as such. There is neither need nor place for spin in science.

Third, while calls for more rigorous reporting have been around for some time, approximately 10% to 15% of articles in Medline have incorrect assessments of statistical significance,⁹ and we have begun to consider the appropriate level of statistical expertise needed for rigor. The senior author’s (D.B.A.) research team has started having all statistical analyses verified by someone with formal training in and whose primary activity is the practice of statistics or biostatistics. We cannot currently quantify the effect of this. Nevertheless, we have observed that when the first data analyst is a professional statistician, the errors detected (and errors have been detected in every case) are usually minor. Contrarily, when the first analyst is not a professional statistician, the errors are sometimes profound. Alongside the frequency with which severe statistical errors, which account for many retractions,¹⁰ seem to occur,⁹ we believe an approach to data analysis that incorporates professional statisticians is warranted.

Finally, we call for a cultural change at all levels in the scientific community. Prevent, detect, admit, and correct is a focusing slogan among our group. In the pursuit of truth, investigators must take personal responsibility for the accurate collection, interpretation, and reporting of results, even when this requires actions, such as correcting their own scientific error. Journal editors can facilitate this by considering the implementation of newer techniques, such as retract and replace and loss of confidence statements, which may reduce stigma associated with correcting the scientific record. Data suggest that journal editors also need to remove retracted papers from websites or label them as retracted more clearly.¹⁰ This will ensure that the information needed to assess the strength of data supporting a hypothesis or idea is readily available.

We have shared some of our opinions on approaches that may increase rigor in pediatric obesity research. The health of children and the public’s trust in the scientific community is on the line; this is something in which we all have a stake.