Conditioning on “study” is essential for valid inference when combining individual data from multiple randomized controlled trials: a comment on Reesor et al’s School-based weight management program curbs summer weight gain among low-income Hispanic middle school students. J Sch Health. 2019;89(1):59-67

Considering the need for examining summertime versus school year weight gain among children, we read with interest the paper, “School-based weight management program curbs summer weight gain among low-income Hispanic middle school students.”¹ We were intrigued by the conclusion that “a school-based weight management program protected overweight/obese students against potentially greater summer weight gain.”¹ We commend the authors for recognizing the importance of aggregating individual participant data (IPD) from several randomized controlled trials (RCTs) to allow for stronger causal inferences and obtain an adequate sample size. We note, however, 2 critical concerns that raise doubts about the veracity of the results: the authors appear to have ignored “study” as a factor in their analysis, and the sample sizes do not appear to match those in the original studies.

The authors present 2-way repeated measures analyses of variance (ANOVA) for zBMI changes as a series of comparisons between conditions (weight management program and control) across time points (baseline, spring, and fall), separately for children with over-weight/obesity and normal weight. The conclusions, however, are derived from analyses that are inappropriate for aggregation of IPD from multiple RCTs. Compared to meta-analyses and systematic reviews that are based on results obtained from empirical publications, the process of combining and analyzing IPD from multiple RCTs involves different complexities. In the latter, participants were recruited according to diverse RCT protocols and they were subjected to different intervention and control conditions;consequently, combining IPD across RCTs as though they come from a single large RCT could lead to inappropriately precise effect estimates and biased program comparisons.² Therefore, if IPD from multiple RCTs are combined, a term for “study” should be included in the analysis. In the absence of a term for “study,” the analysis is, at best, inefficient due to the inability to reduce residual variance; at worst, it is inaccurate due to confounding, if group conditions (intervention and control) or group assignment probabilities are not identical across all RCTs.

Reesor et al.¹ reported similarities across program conditions of original RCTs. For example, all weight management programs met for 35–40 minutes. There are, however, substantial differences among programs in terms of frequency of meetings per week, eg, 3 days (N = 319) versus 5 days (N = 106), and duration of the program, eg, 3 months (N = 319) versus 6 months (N = 106). In the methods section, the authors justified their approach to combining RCTs by stating that students in different weight management programs “exhibited similar changes in zBMI during the summer”¹ and “similar results were observed among the control participants.”¹ It is important to recognize that similarities in zBMI change do not necessarily indicate similarities in sampled populations, group conditions, prevailing historical circumstances, recruitment protocols, interventions, and other extraneous factors. Furthermore, using the results of the primary outcome as the criterion to decide whether it is acceptable to combine RCTs is inappropriate, because the results depend on the intervention. Such a selection method could inappropriately be used to justify combining pharmaceutical, environmental, and lifestyle interventions together, or create an unrealistically homogeneous effect estimate. Instead, an appropriate a priori analysis method should be used to stratify or adjust for clustering of IPD within original RCTs, while preserving trial membership of individual participants.^3,4

Secondarily, the number of original RCTs and their sample sizes do not seem compatible with the numbers provided in the paper. Reesor et al.¹ state that their references 3638 contain the data from 5 RCTs pooled in their paper. Within references 36–38, we were unable to find 5 RCTs with similar designs from 2005 to 2010. Moreover, the sample sizes reported in references 36–38 do not add up to totals reported in Reesor et al.¹ Next, apart from mentioning 5 RCTs, they indicate that “[i]t was necessary to aggregate the participants across multiple waves of data collection in order to obtain an adequate sample size to evaluate summer weight gain.”¹ Thus, we hope the authors will clarify whether they used multiple waves of one or more RCTs or 5 separate RCTs for aggregating IPD. Providing unambiguous and accurate details about original RCTs is also helpful to those who may perform similar studies in the future.

Misestimates of effects in metaanalyses are not uncommon.^5,6 Therefore, we request that the aforementioned issues be addressed by the authors. Specifically, the results should be reanalyzed including a term for “study” in the model, more details about the origins of the data used from the original RCTs should be provided, and analytical code (and, better still, the anonymized raw data) should be made publically available to contribute to the rigor, reproducibility, and transparency of these analyses.⁷ If the authors wish, we offer our assistance with the analysis, as some of us have done previously.⁸