School-based obesity treatment and prevention programs: all in all, just another brick in the wall?

Abstract

In this issue of the International Journal of Obesity, Katz et al. report on a meta-analysis of school-based interventions (SBIs) for lessening overweight and obesity in children. A hard look at SBIs is certainly warranted and we applaud Katz et al. for undertaking the task. SBIs have been and remain among the most advocated of anti-obesity approaches and, at least as research projects, they have tended to be large expensive projects. Not surprisingly, evidence of the efficacy/ effectiveness of these SBIs has been variable, presumably as a result of the heterogeneity of approaches.

What did Katz et al. do and report?

Katz et al. initially identified 64 trials published in or before 2004 for potential inclusion in a meta-analysis. They subsequently retained 8 of the 64 trials in their analysis. On the basis of their analysis of these eight trials, they reported that:

• Nutrition and physical activity interventions resulted in significantly reduced weight compared with control conditions (standardized mean difference, SMD = −0.29, 95% CI = −0.45 to −0.14, random effects model).

• Reduction in TV viewing, on the basis of one study, as a treatment also showed equivalent efficacy (SMD = −0.35, 95% CI = −0.63 to −0.06).

• Weight reduction was also induced in trials that included parental or family involvement (SMD = −0.20, 95% CI = −0.37 to −0.04).

• Combined nutrition and physical activity interventions were not, on average, significantly more effective than the one nutrition intervention that did not include physical activity (SMD = −0.39, 95% CI =−0.56 to −0.23).

• Interventions aimed at increasing physical activity without any effort toward dietary intervention did not significantly reduce body weight.

• The robustness of these findings is limited since there is a high degree of heterogeneity.’

Are the studies meta-analyzed likely to represent the available data?

This is not clear. The authors searched electronic databases such as MEDLINE, HealthStar, Psych Info and Embase for trials conducted during the period from 1966 to February 2000 for a previous review.¹ Additional electronic database searches for studies published between February 2000 and October 2004 were conducted to add to data obtained previously. The authors further reviewed the Cochrane Library and seminal work by ‘prominent authors in the field of obesity’ to locate additional relevant citations. The authors, however, report no effort to identify unpublished literature or attempt to assess the effects of publication bias on results.

Approximately half of the initially identified trials were excluded due to ‘poor methodological quality’, but we are not told the nature of these methodological problems, whether standardized exclusion criteria existed, and if so, what these exclusion criteria were. Of 19 studies identified as appropriate for potential inclusion in the meta-analysis that were not excluded because of methodological problems, 2 were subsequently excluded because they lacked control groups. Lack of a control group, to us, represents a substantial methodological flaw.

Moreover, 5 more of the 19 studies were not included in the meta-analysis because Katz et al. were unable to obtain or calculate an effect size from the published report. There is no description of an attempt to contact the authors for the requisite information, to place plausible boundaries on the sample effect size from incomplete data or to back-calculate effect sizes from test statistics as can typically be done in such situations (cf., Allison et al.²). Several of the excluded studies were relatively large and strong studies, so their inclusion may make a difference. One excluded study was that of Sallis et al.,³ and yet they wrote ‘Effect sizes were large for boys’ …BMI (d = 0.83).’ So, at least for boys, the exclusion was not necessary. No calculation was needed, the original authors had provided the effect size. Similarly, Table 2 of Caballero et al.,⁴ a study listed in Katz et al.’s Table 1 as one for which an effect size could not be calculated, seems to have all the requisite information for calculating an effect size.

Thus, we are unable to place full confidence in the process that led to studies being included in the meta-analysis as being one that fully represented the available data.

Interpreting the results

If we put our reservations above aside and take the results at face value, how can we interpret and place them in context?

Magnitude of effect

Katz et al. wrote ‘In these analyses, we observed a pooled effect size (SMD) of −0.29 in clusters of studies using combination nutrition and physical activity interventions.’ This effect size corresponds to 21.3% nonoverlap between the intervention and the control conditions according to Cohen’s interpretation.’ This latter statistic, the percent nonoverlap, is based on the assumption of a normal distribution⁵ and can be difficult to interpret (note that under the null hypothesis of no effect the expected value of this statistic is 0%). Hence, let us try a few other interpretive devices.

First, an SMD can be converted to a proportion of variance by the formula r² = d²/(d² +4), where d denotes the SMD (cf. Wolf,⁶ Table 8). Thus, the SMD of −0.29 corresponds to treatment assignment explaining about 2.0% of the variance in child weight. Graphically, what would this look like? If we adopt the implicit assumption Katz et al. used in calculating the percentage of nonoverlap, we can plot two distributions that have an SMD of −0.29. As can be seen in Figure 1, the difference is discernable but not dramatic. Alternatively, we can calculate the probability that a randomly selected student from a school implementing an SBI with an effect size (SMD) of −0.29 will have a lower body mass index (BMI) than a randomly selected student from a school not implementing the SBI. The answer is roughly 58%. But recall that with no effect the probability would be 50%, which yields a net increase of only about 8%.

Figure 1.

Two distributions that have SMD of −0.29.

Finally, even though the meta-analysis may show some effectiveness overall of SBIs, evidence of effectiveness for body weight outcomes has not been found in several ‘top drawer’ studies led by world class investigators. As Katz et al. noted, this leads to questions about the robustness and reproducibility of SBI effects. Substantial federal and private funding (sometimes on the order of tens of millions of dollars in a single study) has been invested in these studies, some of which have not produced statistically significant reductions in BMI, improved nutrition and/or increases in physical activity. Thus, we must proceed with caution before advising large-scale implementation of school-based interventions as designed previously. Specifically, school superintendents and principals with limited operating budgets have a right to ask whether they can be reasonably assured that interventions proposed for their use will result in improved outcomes when delivered as intended. Efficacy studies typically include substantial resources to insure successful implementation (for example, expert interventionists, adequate training and supervision, financial incentives to schools and/or students). However, local schools will implement these interventions with few or none of these resources and as such will likely fall short of the impact of the initial efficacy study. If such modest confidence regarding efficacy of the SBI can be offered, given the aforementioned challenges local schools will likely encounter, can we expect school officials to be motivated to proceed?

Is the cost justified?

Katz et al. do not address cost effectiveness of school-based interventions to control obesity. Finkelstein and Trogdon⁷ considered the issue of whether financial return on investment could justify the use of current public health childhood obesity programs. They concluded that ‘under current best practices, it is unlikely that interventions will be able to meet the level of effectiveness required at a low enough implementation cost to show positive ROI (return on investment). The merits of childhood obesity interventions should be based on their ability to efficiently control weight and improve health compared with alternative uses for available resources. They should not be based on the potential for short-term financial savings.’ In that light, Planet Health and the Coordinated Approach to Child Health are two combination nutrition–physical activity school-based interventions reported to be cost effective on the basis of actual observed weight loss and estimates of avoided future costs associated with overweight in adulthood.^8,9 However, as pointed out in a recent commentary, cost-effectiveness analyses in such interventions rely on two assumptions.¹⁰ First, one must assume retention of weight loss after the intervention into adulthood. Second, one must assume childhood weight status is correlated with medical outcomes of obesity in adults. As most studies lack long-term follow-up data to assess maintenance of intervention effects into adulthood, there is considerable debate regarding the incorporation of avoided adult overweight morbidity, mortality and medical costs into cost-effectiveness models of school-based interventions.^10,11 Consequently, data on the cost effectiveness of school-based interventions are mixed and an area in need of future prospective study.

The value of family involvement

Many authors seem to suggest schools as the essential intervention environment to address childhood obesity, particularly when augmented with parental involvement. However, findings from prior reviews are inconsistent. For example, a recent meta-analysis of family-based behavioral weight loss treatments for children suggests large and reliable effects beyond other treatments or control;¹² however, another recent meta-analysis of obesity-prevention programs suggest that parent involvement is not associated with larger effects.¹³ One potential explanation for the discrepant findings may be that Young et al. investigated the effects of child obesity treatment, whereas Stice and colleagues focused on obesity prevention. Katz et al. combine treatment and prevention programs in their meta-analysis, but the result adds to the inconsistency. In their review, Katz et al. report (in the text of the review) improved body weight in children who participated in combination interventions with a parent/family component (SMD = −0.20, 95% CI = −0.37 to −0.04) but note a decreased effect when combination interventions include both a family component and an environmental intervention (SMD = −0.16, 95% CI = −0.32–0.00, random effects model). They do not offer a suggestion as why the effect may be diminished with the addition of the environmental component. Moreover, they do not seemingly address the fact that the effect sizes for the two types of family interventions are both lower than the effect sizes of the combination interventions (SMD = −0.29), the nutrition-only intervention (SMD = −0.39) and the TV reduction intervention (SMD = −0.35).

We recognize and appreciate that families shape the eating and physical activity patterns of children in myriad ways (for example, role modeling, availability of food/activities, cultural values). The effective use of family interventions to maximize prevention and/or treatment effects has been understudied. For example, to our knowledge, an overlooked research question is whether family-based treatment interventions are enhanced by the addition of an SBI. Given the personnel, supplies and instruction time needed to deliver SBIs to reach a modest effect, a more efficacious and efficient first line of defense may be family-based interventions for weight loss. Such interventions may also be more amenable than SBIs to address the sociodemographic and cultural circumstances of children targeted for intervention.¹⁴

Is action justified?

In wide-scale implementation

In 2004, The Institute of Medicine Committee on Prevention of Obesity in Children and Youth suggested that we do not wait on the best research, but rather take action on the available research to address the escalating childhood obesity epidemic¹⁵ and we have addressed the quandary of making decisions in the face of uncertainty elsewhere.¹⁶ On the basis of what we know from extant SBIs, there is limited empirical support for full-scale implementation of these programs as currently designed. Thus we concur with Stevens et al.,¹⁷ who wrote ‘We believe that creative and potent interventions must be developed and shown to be efficacious. After this demonstration, investigators can address the issue of how to implement the intervention in the real world. Thus, we support a progression of studies from evidentiary, to efficacy, to effectiveness. We believe this approach would yield more progress in the challenging area of obesity prevention research.’ We recognize, however, the appeal of such implementation efforts as they follow a public health approach in which a low-efficacy intervention with high reach (about 95% of US children aged 5–17 attend school)¹⁸ can still produce a moderate population impact. Also, we acknowledge that such a public health intervention has substantial appeal as an initial step because it allows access to all children regardless of socioeconomic status.

In further research

Schools can conceivably play an important role in reducing obesity levels. However, as pointed out in two recent systematic reviews of SBI, the ability to draw conclusions regarding the efficacy of SBIs continues to be limited given the small number of published studies and issues with methodology.^19,20 Thus, some types of additional research on SBIs are warranted. That being said, which research studies seem most appropriate? First, we concur with Stevens et al.¹⁷ that efficacy studies should precede large-scale effectiveness studies. In this regard, previous SBI studies have largely put the cart before the horse to date and we believe the trend should be reversed. More modest-size and methodologically rigorous studies to determine what can work if precisely implemented are needed before large-scale implementation work begins. Second, investigators proposing SBIs should ask themselves ‘What is markedly different about this proposed SBI than the more-than-score of SBI studies already done, many by pinnacle investigators, that have often yielded minimal or even no effects?’ In our view, only those proposals for which a compelling answer can be given merit implementation. As Stevens et al.¹⁷ wrote, ‘Time, effort, and money should not be wasted on poorly designed trials or on weak interventions that differ little from those that have been previously shown to produce null effects.’

We are aware of one recent school-centered efficacy trial targeting childhood obesity on multiple levels including community, family, policy and environmental components that may increase the probability of success. The Shape Up Somerville study was a 3-year school-system-targeted efficacy trial that resulted in a significant reduction in BMI z-score of elementary school children in the intervention compared with the control group.²¹ The more global approach of this intervention makes it unique in contrast to previous SBIs targeting obesity.

Do we need to expose all children?

This article presents evidence that programs combining nutritional and physical activity interventions do lead to weight reduction among school-aged children (pooled effect size of −0.29 in combination interventions). An intriguing question is whether such programs would be beneficial among all youth or only those who are overweight or obese. In a randomized controlled trial among 103 adolescents, Ebbeling et al.²² found that replacement of sugar-sweetened beverages with non-caloric beverages appeared to result in decreasing body weight compared with usual consumption only among those above the upper baseline BMI tertile. This interaction between weight change and baseline BMI suggests that reduction in consumption of sugar-sweetened beverages might be most beneficial in adolescents with higher BMIs. To infer the effect of physical education instruction on BMI, Datar et al.²³ followed a national sample of 9751 US kindergartners over 2 years and similarly observed a reduction of BMI only among girls who were overweight or at risk for overweight at baseline. Again, the effectiveness of the intervention may increase with baseline weight. These two studies suggest the notion that physical activity or nutritional interventions may be most effective among children and adolescents at the highest risk for overweight or obesity. The corollary to this is that it may not be necessary to expose all children to these interventions and that trying to do so may therefore unnecessarily expose children not at risk to restrictions or distractions and may unnecessarily expend resources. If it is true that only children who are above some BMI percentile benefit most from the SBIs, are there realistic ways to deliver the SBIs only to them and thereby save resources? However, we must acknowledge that stratifying such interventions in school settings may introduce substantial logistical and procedural issues regarding identification of participants that could negate budgetary savings. Additionally, such targeting may introduce psychosocial issues including heightening of existing teasing, discrimination and social isolation suffered by overweight and obese children.^24,25

Conclusion

We conclude that beneficial effects of existing SBIs are rather small and not reliably reproducible even by highly qualified investigators. Again, we encourage investigators proposing SBIs to ask themselves, ‘what is markedly different about this proposed SBI’ before proceeding with large-scale implementation. We do share the common a priori intuition that schools can play an important role in promoting healthy body weights, but new out-of-the-box ideas are needed to try and develop approaches whose results will support that intuition. Thus, for future SBIs, we advocate novel approaches and use of smaller efficacy studies rather than larger effectiveness trials.