Reanalysis Accounting for Clustering and Nesting Overturns Conclusions in: “Watching TV Cooking Programs: Effects on Actual Food Intake Among Children”

Folkvord et al¹ addressed the interesting and important topic of the effects of watching TV cooking programs on food choice (the selection of pictures of food) in children. In the study, 8 classes (ie, clusters) from 5 different schools were randomized to 3 study conditions: healthy cooking program, unhealthy cooking program, and control. We (the research team at Indiana University) observed invalidating errors² in the statistical analysis by not accounting for clustering and nesting effects and using a 1-sided P value to deduce conclusions about the effects of the intervention. The original authors collegially shared their data. We reanalyzed the data using a valid statistical approach. Our correction to the statistical analysis invalidates the original conclusions about TV cooking program effectiveness. The 2 research groups collegially discussed the issue and decided to publish the reanalysis results jointly.

In the article, randomization occurred at the classroom level and not at the individual child level. However, the authors analyzed data and reported program effects at the individual child level. Such an experimental design necessitates accounting for clustering and nesting effects at the classroom level, as well established and described elsewhere.^3-5 Not doing so can lead to unduly small SE estimates; therefore, smaller than valid P values and inflated type 1 error rates.^6,7 To account for clustering and nesting effects, one of the suggested valid approaches is deploying multilevel hierarchical modeling in which individual children are nested within classes, and classes are nested within schools. The authors acknowledge that they did not account for clustering in their analyses because they were not aware this statistical method should have been used to analyze the data:

Finally, because the current study did not take into account clustering effects, significance may be overstated, although separate analyses with class or schools included as factors did not lead to different results.

The inclusion of treatment unit ID as a factor does not appropriately account for clustering and nesting effects.

Note that we could reproduce the original results per the methods described in the article that ignored clustering and nesting. After addressing the clustering and nesting effects, published conclusions about the effect of TV cooking programs no longer hold (Table 1). We applied multilevel mixed-effect modeling and reported results in columns 3 and 4 in the Table. We show that the corrected results are no longer statistically significant. Another serious issue is that a 1-sided P value was used for determining statistical significance even though the effects of the intervention could conceivably occur in either direction,⁸ in which the authors hypothesized a positive effect of the TV cooking program.

Table 1.

Reproduced and Corrected Results

	(1) Odds Ratio	(2) Odds Ratio	(3) Odds Ratio	(4) Odds Ratio
	Reproduced Results of Folkvord et al		Corrected Results (Accounting for Clustering and Nesting)
Variables	Table 2 of Folkvord et al1	Table 3 of Folkvord et al1	Multilevel Mixed-Effect Model
Healthy cooking program (reference is control group)	2.45*	2.72*	3.39	3.05
SE	1.24	1.41	2.68	2.44
P	0.08	0.05	0.12	0.16
Unhealthy cooking program (reference is the control group)	0.90		1.11
SE	0.53		0.78
P	0.86		0.88
Age	0.96	0.96	0.98	0.98
SE	0.31	0.31	0.36	0.36
P	0.91	0.91	0.96	0.96
Control group (reference is the unhealthy group)		1.11		0.90
SE		0.65		0.63
P		0.86		0.88
var(_cons[class])			1.31	1.30
SE			43.90	43.82
var(_cons[class > school])			1.29	1.30
SE			43.55	43.52
Constant	0.44	0.39	0.31	0.34
SE	1.54	1.46	1.23	1.41
Observations	124	124	124	124
No. of groups			9	9

^*P < 0.1.

Note: Reported effects of the TV program are in odds ratio. The dependent variable is healthy food photograph choice (apple or cucumber) is equal to 1, otherwise (chips or pretzel) zero. We report 2-sided P values, whereas the authors report 1-sided P values.

Nonetheless, it is possible that watching cooking programs could lead to a greater selection of unhealthy food pictures,⁹ which would warrant the use of a 2-sided P value. In this case, the 1-sided P value is half of the 2-sided P value, which is anticonservative. That said, regardless of whether one uses a 2-sided or 1-sided P value at the authors’ designation of 0.05, the Table shows that after correcting for clustering and nesting, none of the results are statistically significant. We conducted this analysis using Stata 15.

We have 2 additional concerns related to the reporting of results. When we reanalyzed the authors’ data per the methods they describe (ie, incorrectly without accounting for clustering and nesting), we noticed that they reported the odds ratio, but the SE was from the coefficient estimates, not from the odds ratio. However, we reported SE from the odds ratio; therefore, our reported SE does not match the reported SE by the original authors. In addition, the number of observations we obtained is 124, whereas they reported 125. Similarly, we find 9 classes, although the authors reported 8 classes.

We again thank the original authors for sharing their raw data so that we could perform this reanalysis. When correcting the results to account for clustering and nesting, we show that, contrary to the conclusions in the original report, no results are statistically significant. Per Committee on Publication Ethics guidelines,

Editors should consider retracting a publication if… They have clear evidence that findings are unreliable, either a result of major error (eg, mis-calculation or experimental error), or as a result of fabrication (eg, of data) or falsification…¹⁰

To uphold the self-correcting ideal of science, the errors in the methods, reporting, and conclusions should be publicly corrected.