Childhood Obesity Policy Research and Practice Evidence for Policy and Environmental Strategies

Abstract

Investigators developed a review system to evaluate the growing literature on policy and environmental strategies to prevent childhood obesity. Over 2000 documents published between January 2000 and May 2009 in the scientific and grey literature were identified (2008–2009) and systematically analyzed (2009–2012). These focused on policy or environmental strategies to reduce obesity/overweight, increase physical activity, and/or improve nutrition/diet among youth (aged 3–18 years). Guided by the RE-AIM framework, investigators abstracted studies of 24 intervention strategies and assessed evidence for their effectiveness (i.e., study design, intervention duration, and outcomes) and population impact (i.e., effectiveness and reach – participation or exposure, and representativeness) in 142 evaluation study groupings and 254 associational study groupings (n=396 groupings of 600 peer-reviewed studies).

The 24 strategies yielded 25 classifications (school wellness policies yielded nutrition and physical activity classifications): 1st-tier effective (n=5); 2nd-tier effective (n=6); “promising” (n=5); or “emerging” (n=9). Evidence for intervention effectiveness was reported in 56% of the evaluation, and 77% of the associational, study groupings. Among the evaluation study groupings, only 49% reported sufficient data for population impact ratings, and only 22% qualified for a rating of high population impact. Effectiveness and impact ratings were summarized in graphic evidence maps, displaying effects/associations with behavioral and obesity/overweight outcomes. This paper describes the results and products of the review, with recommendations for policy research and practice.

Introduction

Childhood obesity has become one of the nation’s most serious health problems with several lifelong co-morbidities, including cardiovascular disease, type 2 diabetes, hypertension, osteoarthritis, cancer, and psychosocial burdens.^1–4 Over the past 3 decades, obesity rates have increased threefold among U.S. children and adolescents aged 2 to 19 years,^5,6 rising to 17% by 2009–2010.⁷ Prevalence is greatest in lower-income and racial/ethnic minority populations as well as in communities with limited access to healthy, affordable foods or safe places to walk, bike, and play.⁸ It’s proposed that this epidemic resulted from small, cumulative environmental changes that altered children’s physical activity and dietary patterns creating an accrual of small increases in children’s daily energy gap—the excess of calories consumed over calories expended.⁹ To reverse this epidemic, numerous policy and environmental changes have been implemented to create healthier environments to promote energy balance.^1,10,11

Since 2000, major U.S. research funders, public and private, have devoted increased funds and resources to research to discovering effective, feasible, sustainable, and cost-effective policy and environmental interventions (nccor.org).¹² Policymakers, public health practitioners, and community leaders increasingly turn to this research and trustworthy systematic reviews to discover what works. However, existing authoritative and systematic review groups, such as the Institute of Medicine, Community Preventive Services Task Force (www.thecommunityguide.org), and Cochrane Collaborative (www.cochrane.org), have not been able to keep up with the rapidly growing research literature, have focused on searchable scientific articles (e.g., PubMed), and thus have only partially reviewed the many intervention strategies that have been studied.^{10, 11, 13–19} As a result, recent recommendations for addressing childhood obesity have often had to rely on research that has not been systematically reviewed, and has focused much more on assessing the internal validity of study results than on evaluating the external validity, feasibility, or sustainability of intervention effects.^20–22

To address these past review and evidence gaps, and accelerate the discovery and application of replicable, evidence-based policy and environmental strategies for childhood obesity prevention, the current review system was designed to:²³

1. develop and apply replicable methods—modeled after respected formal systematic evidence review systems (e.g., Community Guide¹⁹) —to assess the scientific and grey literature addressing policy and environmental strategies for reducing obesity levels, improving healthy eating, and/or increasing physical activity among youth aged 3–18 years;

2. summarize these findings using easy-to-read evidence maps that identify effects/associations related to obesity/overweight, physical activity, and nutrition/diet outcomes; and

3. classify intervention strategies, based on their effectiveness and population impact using ratings ranging from “effective” (recommended for use) to “promising” and “emerging” (recommended for further testing).

Review System Process and Methods

As described elsewhere,²³ a team of investigators, aided by trained research assistants, conducted a systematic review of the grey and published literature appearing from January 2000 to May 2009, analyzing 600 peer-reviewed studies (online list at www.transtria.com/evidence) from over 2000 evidence documents (e.g., journal articles, online evaluation reports) identified. These studies were limited to those published in the English language or translated to the English language; inclusion criteria, key words, and related search information have been published²³ and are available online (www.transtria.com/evidence). Figure 1 illustrates the screening process from the search to the analysis and the primary steps in the analysis.

Figure 1.

Search, inventory, abstraction, and analysis flow chart

Conceptual Framework

The framework and guiding principles undergirding this review, summarized in Appendix A (available online at www.ajpmonline.org), assume that:

1. physical, economic, social, and communication policies and environmental changes can act alone and in concert to influence behaviors, health outcomes, and quality of life²⁴;

2. policy and environmental interventions can modify social norms and attitudes, behaviors, and health across multiple socio-ecologic levels (i.e., national, state, local, organizational, and household)^25–27; and

3. policy and environmental changes can reduce and eliminate disparities and increase benefits to underserved and marginalized populations through the equitable distribution of access, resources, and supports across populations in implementation and enforcement.

Creating an Inventory and Abstracting the Evidence

Tools and protocols were adapted from respected evidence review models^{19, 28, 29} and investigative team members trained research assistants. Evidence documents (n=1288) were abstracted by two research assistants to ensure accuracy through inter-rater uniformity. Discrepancies between abstractors were addressed with the entire investigative team to develop consensus on abstraction methods and enhance training of research assistants, and the investigative team performed regular quality-assurance checks. When distinct articles presented mutually exclusive findings from the same intervention evaluation or associational study, they were combined to form a single independent study grouping (Figure 1). This resulted in a total of 396 independent study groupings: 142 that included at least one formal intervention evaluation (i.e., systematic review, narrative review, or peer-reviewed intervention evaluation based on an experimental, quasi-experimental, prospective cross-sectional, or natural experimental study design), and 254 that included only indicator-outcome associations (i.e., systematic review, narrative review, or peer-reviewed study based on cross-sectional data linking policy or environmental indicators to health or behavioral outcomes).

Defining Policy and Environmental Strategies

The operational definitions used to classify policy and environmental strategies into 24 intervention categories were based on those defined by other review groups (Appendix B, available online at www.ajpmonline.org).^{28, 30–32} Policy changes included laws, regulations, ordinances, organizational policies, resolutions, formal and informal rules, institutional practices or guidelines, advocacy and agenda-setting, policy development, funding and resource allocation, policy implementation, or policy enforcement. Changes to the physical environment included enhanced access to new or improved facilities, amenities, and cultural or artistic enhancements. Changes to the social, economic, and communication environments included increasing equitable access to resources and services; positive media and events; and incorporation of existing or new social networks. Interventions employing only programmatic or promotional strategies, without policy or environmental change components, were excluded.

Intervention components and outcomes for each of the 24 strategies are presented in detailed online intervention tables (www.transtra.com/evidence).

Identifying and Assigning Effectiveness and Impact Ratings

The RE-AIM framework (i.e., Reach, Effectiveness, Adoption, Implementation, and Maintenance)³³ was used both to assess internal and external validity, and to derive standard, objective ratings of intervention effectiveness and impact. Search results, intervention category assignments, and evidence from the abstracted studies were presented, reviewed, repeatedly discussed, and rated by the investigative team and an international expert panel of over 40 outside reviewers until majority agreement (>50%) was reached. The investigative team and expert review panel collectively represented the knowledge and perspectives of researchers, evaluators, evidence review bodies, practitioners, and policy- or decision-makers working in multiple areas (e.g., public health, behavioral medicine, transportation, health economics).

Standard tables were developed to summarize effectiveness and impact ratings for each of the 396 independent study groupings organized by intervention strategy. Major categories for study ratings were derived from the RE-AIM framework³³ and from the criteria used by leading established systematic evidence review systems, including the Community Preventive Services Task Force¹⁹ and Cochrane Collaborative,³⁴ and by newer, similarly rigorous, systems or frameworks.^{22, 33, 35–43} Detailed criteria and operational definitions were provided to help reviewers assess those aspects of research design and intervention reach and implementation most critical for assessing population-level effectiveness and impact (see Brennan et al.²³; Appendix C [available online at www.ajpmonline.org]; Table 1; and visit www.transtra.com/evidence for more information).

Table 1.

RE-AIM framework indicators and operational definitions

RE-AIM	Indicators	Operational definitions
Reach	Participation	High ≥ 75% participation rate Low < 75% participation rate
	Exposure	High = Entire population with daily/weekly exposure Low = A portion of the population and/or periodic exposure
	High-risk populations	High ≥ 40% participation/ exposure from racial/ethnic or lower income populations Low < 40% participation/exposure
	Representativeness	High = No sociodemographic differences in the intervention and target population Low = Significant sociodemographic differences in these populations
	Population reach	High = High participation or exposure + high representativeness Low = One low rating for participation, exposure, or representativeness
	High-risk population reach	High = High rating for high-risk populations + high representativeness Low = One low rating for high-risk populations or representativeness
Effectiveness	Study design	Intervention evaluations: [used Community Guide criteria, Table 2] Associational and descriptive studies: [used Community Guide criteria, Table 2]
	Intervention duration	High ≥ 12 months OR Moderate = 6–12 months Low < 6 months
	Outcomes	Number of outcomes for nutrition, physical activity, and overweight/obesity
	Effects	Number of effects coded as positive, neutral, or negative OR Number of associations coded as positive, negative, or no association
	Effectiveness	Effective = Intervention evaluation + high/moderate duration + >50% positive effects Somewhat effective = Intervention evaluation + low duration + >50% positive effects Not effective = Intervention evaluation + majority neutral or negative effects Positive association = Associational study + majority positive associations No association = Associational study + majority no associations Negative association = Associational study + majority negative associations
	Sampling/Representativeness*	High = No sociodemographic differences in study sample and exposed population or oversampling of high-risk populations in the study sample Low = Significant sociodemographic differences in these populations
Adoption & Implementation	Intervention design components*	Multi-component = Multiple intervention strategies Complex = Single strategy with multiple activities Simple = Single activity
	Intervention feasibility*	High = Simple, complex, or few components + low cost + low resources Low = Other
	Policy feasibility*	High = Simple policy + low cost + low resources + small scale Low = Other
	Implementation complexity*	High = Multi-component or complex + low intervention or low policy feasibility Low = Other
Maintenance	Maintenance*	Nutrition, physical activity, or overweight/obesity effects maintained after intervention
	Potential sustainability*	Yes = Sustainability plans or activities No = No stated sustainability plans or activities
IMPACT	Population impact	High = Effective + high population reach Low = Effective + low population reach or somewhat effective No impact = Not effective
	High-risk population impact	High = Effective (high-risk populations) + high for high-risk population reach Low = Effective (high-risk populations) + low for high-risk population reach or somewhat effective (high-risk populations) No impact = Not effective (high-risk populations)

^*Indicators with insufficient data to be incorporated into the effectiveness or impact algorithms.

Specifically, these criteria and definitions assessed major attributes of study design (e.g., controlled evaluation, cross-sectional), population exposure and reach (overall and for high-risk populations), and intervention complexity and feasibility, in order to evaluate overall intervention effectiveness and impact (Table 1). Operational definitions of indicators of effectiveness and impact were developed using clear, well-defined cut-points to enable the investigative team and review panel to reach high levels of agreement or consensus on strategy ratings. Criteria ratings were completed by the investigative team, and reviewed by assigned members of the expert review panel with expertise related to the intervention strategy.

Developing Evidence Maps

For each of the 396 independent study groupings, the investigative team classified pertinent outcome measures into three domains: obesity/ overweight, physical activity, and/or nutrition/diet. Evidence maps were created specifically for this review to condense and visually illustrate intervention effects or associations, linking interventions and short-term outcomes (proxy or surrogate measures of behavior), intermediate outcomes (behaviors), and long-term outcomes (obesity/overweight) for each of the 24 intervention strategies. The purpose of these maps was to identify evidence strengths and gaps to help inform future research and practice.

Rating Intervention Strategies

Finally, based on the totality of evidence reviews and ratings described above, the investigative team and assigned expert reviewers independently rated the evidence for each of the 24 interventions. Intervention strategies were classified as “1st tier effective,” “2nd tier effective,” “promising,” or “emerging” using criteria set forth earlier.^{23, 44} Ratings for 1st tier effective intervention strategies were operationally defined to include any interventions which had been found to be effective through rigorous, published systematic evidence reviews. Strategies classified as “2nd tier effective” were judged to have sufficient evidence to merit a formal systematic review. “Promising” strategies were those judged to merit further investigation through systematic evaluation studies, and “emerging” strategies were recommended for pilot studies or evaluability assessments prior to investments in systematic evaluation efforts. Expert panel reviewers judged a sample of two to four strategies each, and convened in small groups (four or five reviewers) to develop agreement or consensus. Reviewers reached consensus or majority agreement for 21 of the 24 interventions, and assigned the most conservative group rating in the remaining three cases.

Review System Findings

Rating Intervention Effectiveness

Intervention effectiveness ratings reflect the significance and direction of the relationship between independent and dependent variables (i.e., interventions and outcomes) reported in the study groupings (Table 1). For the 142 intervention evaluation study groupings, effectiveness ratings were based on study design, intervention duration, and the proportion of positive, neutral, or negative intervention effects (i.e., the direction/significance of intervention effects). For the 254 study groupings employing associational or descriptive designs, effectiveness ratings were based on the proportion of positive, neutral, or negative associations (i.e., direction and significance of the association between the intervention —or independent variable—and the dependent variable). Across all 396 study groupings, dependent or outcome variables fell into one or more of three domains: obesity/overweight, physical activity, and nutrition/diet. Effects and associations were categorized as positive, neutral (no association), or negative based on the majority of effects or associations for each independent grouping of intervention evaluations or associational studies. Table 2 presents a breakdown of the effectiveness indicators.

Table 2.

Effectiveness indicators, definitions, and proportions

EFFECTIVENESS (n=600 independent studies, 396 independent interventions or observations – study groupings)
Indicator	Operational definition		Proportion, %
	Types	Subtypes
Study design (n=396 study groupings with independent interventions or observations)	Intervention evaluation	TOTAL	36%
		Group randomized trial	14
		Nonrandomized trial	9
		Before and after study	9
		Randomized trial	<1
		Prospective cohort study	1
		Retrospective cohort study	<1
		Time series study	2
	Associational study	TOTAL	62
		Cross-sectional study	61
		Prospective cross-sectional	<1
		Retrospective cross-sectional	1
	Descriptive study	TOTAL	2
Intervention duration (n=396 study groupings)	High = intervention > 1 year		15
	Moderate = intervention 6–12 months		10
	Low = intervention < 6 months		9
	Intervention duration not reported		2
	No intervention		64
Outcomes (n=525 across study groupings)	Overweight and obesity		25
	Physical activity		45
	Nutrition		18
	Sedentary behaviors		2
	Short-term proxies (e.g., purchasing behavior, bikeway use, behavioral intention)		10
Effectiveness ratings (n=263 across study groupings)*	Effective = intervention evaluation + intervention ≥ 6 months + majority of positive outcomes	TOTAL	56
		Overweight and obesity	18
		Physical activity	21
		Nutrition	16
	Somewhat effective = intervention evaluation + intervention < 6 months + majority positive outcomes	TOTAL	14
		Overweight and obesity	2
		Physical activity	6
		Nutrition	5
	Not effective = intervention evaluation + majority neutral or negative outcomes	TOTAL	30
		Overweight and obesity	16
		Physical activity	5
		Nutrition	9
Association ratings (n=598 across study groupings)*	Positive association = associational study + a majority of positive outcomes	TOTAL	77
		Overweight and obesity	17
		Physical activity	54
		Nutrition	4
	No association = associational study + neutral outcomes	TOTAL	16
		Overweight and obesity	4
		Physical activity	10
		Nutrition	<2
	Negative association = associational study + a majority of negative outcomes	TOTAL	7
		Overweight and obesity	2
		Physical activity	5
		Nutrition	<1

^*Sedentary behavior corresponded to: <2% effective, <1% not effective, 2% positive association, and <1% no association ratings.

Across the 396 study groupings, results were mostly favorable with positive, significant outcome effects reported for 56% of the 142 evaluation study groupings, and positive, significant indicator-outcome associations reported for 77% of the 254 associational study groupings. Considering only the 142 evaluation study groupings, 69% reported an intervention duration categorized as moderate or high (i.e., in effect for 6 months or longer prior to evaluation). Several studies were unable to be rated for the following reasons: 32 studies did not report intermediate or long-term outcomes; nine studies examined only policy and environmental changes—with neither obesity/overweight, physical activity, or nutrition/diet outcomes, nor associations; eight studies did not report intervention duration; and two studies did not report effects or associations.

For all 396 study groupings, a total of 525 different outcomes were examined: 25% assessed changes in overweight and/or obesity prevalence, 47% assessed changes in physical activity or sedentary behaviors, and 18% assessed changes in nutrition/dietary behaviors. Ten percent of the outcomes represented intermediate intervention impacts, such as measures of actual policy and environment changes or behavioral proxies (e.g., purchasing behaviors, transit use).

As noted in Table 1, insufficient data were reported for sampling/representativeness or maintenance for all study groupings, so these two factors could not be rated using the effectiveness or impact algorithms created for this review (i.e., they are not reflected in Table 2A; see www.transtra.com/evidence for more information).

Rating Intervention Impact

Intervention impact ratings combine intervention effectiveness ratings with the proportion of the intended population reached, reflecting the level of participation by the population or the proportion of the population exposed to the intervention as well as the overall representativeness of those participating or exposed. Impact ratings were derived for the 142 evaluation study groupings. Where feasible, ratings for high-risk population reach and high-risk population impact similarly were derived. Table 1 presents operational definitions for each of the impact indicators and Table 3 summarizes the proportion of study groupings receiving different ratings. Participation rates were not reported for 91% of the 142 evaluation study groupings. Because participation is often difficult to assess for policy and environmental interventions (e.g., proportional use of a park or farmers market by residents within a one mile perimeter), the investigative team and expert advisors developed and applied an algorithm for exposure ratings (Table 1). The majority of the 142 evaluation study groupings (67%) warranted a “high exposure” rating, but fewer than half reported exposure data for high-risk populations (46%), and only 19% warranted a rating of “high” high-risk population reach. Many study groupings (43%) did not provide sufficient representativeness data (i.e., did not report differences in the socio-demographic composition of the exposed population in comparison to the intended population). However, 50% of the remaining evaluation study groupings merited a “high” rating for representativeness. As Table 3 shows, 43% of the 142 study groupings met the criteria for a “high” rating for overall population reach, while only 19% met the criteria for a “high” rating for high-risk population reach.

Table 3.

Impact indicators, definitions, and proportions

IMPACT (n=142 independent interventions – study groupings)
Indicator	Operational definition		Percent
	Types	Subtypes
Participation (n=142 study groupings with independent interventions)	High ≥ 75% of the intervention population		6
	Low < 75% of the intervention population		3
	Participation not reported		91
Exposure (n=142 study groupings)	High = entire intervention population with daily/weekly exposure		67
	Low = portion of the population and/or less than daily/weekly exposure		18
	Exposure unable to be rated due to insufficient reporting of data		15
High-risk population (n=142 study groupings)	High ≥ 40% racial/ethnic or lower-income populations in the intervention population		37
	Low < 40% racial/ethnic or lower-income populations in the intervention population		9
	High-risk population data not reported		54
Representativeness (n=142 study groupings)	High = no significant differences between the intervention (exposed) population and the target (intended) population		50
	Low = significant differences between the intervention (exposed) population and the target (intended) population		7
	Representativeness unable to be rated due to insufficient reporting of data		43
Population reach (n=142 study groupings)	High = high participation or exposure and high representativeness		43
	Low = low participation and exposure or low representativeness		11
	Population reach unable to be rated due to insufficient reporting of data		46
High-risk population reach (n=142 study groupings)	High = high for high-risk population and high representativeness		19
	Low = low for high-risk population or low representativeness		6
	High-risk population reach unable to be rated due to insufficient reporting of data		75
Population impact ratings (n=155 across study groupings)*	High = effective rating and high population reach	TOTAL	45
		Overweight and obesity	14
		Physical activity	16
		Nutrition	14
	Low = effective rating and low population reach or somewhat effective rating	TOTAL	23
		Overweight and obesity	6
		Physical activity	12
		Nutrition	5
	No impact = not effective rating	TOTAL	32
		Overweight and obesity	20
		Physical activity	6
		Nutrition	6
High-risk population impact ratings (n=42 across study groupings)*	High = effective rating (specific to racial/ethnic or lower-income populations) and high for high-risk population reach	TOTAL	45
		Overweight and obesity	19
		Physical activity	14
		Nutrition	12
	Low = effective rating (high-risk populations) and low population reach or somewhat effective rating (high-risk populations)	TOTAL	14
		Overweight and obesity	2
		Physical activity	5
		Nutrition	7
	No impact = not effective rating (high-risk populations)	TOTAL	41
		Overweight and obesity	21
		Physical activity	7
		Nutrition	12

^*Sedentary behavior corresponded to: <1% high, 0% low, and 0% no population impact ratings, and there were no high-risk population impact ratings.

Summary population impact ratings were derived for 155 of the total 525 outcomes measured. Criteria for “high” population impact were satisfied for almost half (45%) of the 155 outcomes reported, and these were fairly evenly distributed across obesity/overweight, physical activity, and nutrition/diet outcomes. Similarly, 45% of the 42 outcomes for which a high-risk population impact rating could be derived, including obesity/overweight, physical activity, and nutrition/diet outcomes, met the criteria for a “high” rating. See Table 3 for these ratings.

Five additional impact-related indicators assessed intervention design components, feasibility (entire intervention and policies specifically), and implementation complexity, as well as potential sustainability (Table 1). Except for intervention design, most evaluation study groupings lacked sufficient implementation data to generate impact ratings (i.e., they are not reflected in Table 3, see www.transtra.com/evidence for more information).

Electronic versions of all of the effectiveness and impact ratings tables across the 24 intervention strategies may be accessed at www.transtra.com/evidence.

Evidence Maps

For each of the 24 intervention strategies, investigators generated two evidence maps using graphic displays to illustrate the intervention effects or associations reported for obesity/overweight, physical activity, and/or nutrition/diet outcomes. Figure 2 displays the associational map indicating positive, negative, or neutral associations corresponding to environmental and policy indicators on the left and their relationship to short-, intermediate-, and long-term outcomes on the right. For example, this figure focuses on childcare food and beverage policies and environments^45–67 and shows that no study grouping assessed associations between intervention indicators and obesity/overweight outcomes. Yet, one study grouping demonstrated that increased access to healthy foods, and reduced access to unhealthy foods, in childcare settings have positive or neutral associations with nutrition/diet outcomes. This figure also illustrates that each study grouping can have more than one outcome (obesity/overweight, physical activity, or nutrition/diet) and more than one association (or effect, as seen below) per outcome. (Strategies for improving childcare physical activity policies and environments were separately rated for their effects on physical activity outcomes.)

Figure 2.

Example evidence map for associational studies for childcare food and beverage policies and environments

Returning again to the example of childcare food and beverage policies,^45–67 the evidence map in Figure 3 provides a snapshot of the intervention effects, showing positive, neutral, or negative effect ratings for the nine evaluation study groupings reviewed. For this map, 28 of the effects were positive across short-, intermediate-, and long-term outcomes, with ten neutral and ten negative effects. More specifically, many effects were reported for reduced access to unhealthy foods, with the majority showing positive influences on the outcomes, whereas increased access to fruit and healthy afterschool snacks had fewer effects reported, although they included largely positive influences on obesity/overweight.

Figure 3.

Example evidence map for intervention studies for childcare food and beverage policies and environments

Electronic versions of all of the evidence maps for all 24 intervention strategies may be accessed at www.transtra.com/evidence.

Classifying Policy and Environmental Strategies along an Evidence-based Continuum

The investigative team and expert panel advisors reviewed the intervention tables as well as the effectiveness and impact ratings to classify each of the 24 intervention strategies based on readiness for application and needs for further research, particularly practice-based research. (Note: School wellness policies were reviewed by two independent advisory groups, one focused on physical activity and the other on nutrition/diet outcomes.)

Intervention strategies were classified as “effective” (first- or second-tier) and ready for practice and/or systematic evaluation, “promising,” or “emerging.” Five intervention strategies were classified as “1st tier effective,” six as “2nd tier effective,” five as “promising,” and nine as “emerging.” Table 4 aligns existing CDC Community Guide ratings (www.thecommunityguide.org) with the ratings from this review.

Table 4.

Policy and environmental strategy ratings

Nutrition Strategiesa	Community Guide Ratingb	1st Tier Effective	2nd Tier Effective	Promising	Emerging
Childcare food and beverage policies and environments			X
School food and beverage policies and environmentsc,d	Insufficient evidence (school-based programs promoting nutrition and physical activity)		X
Food pricing			X
Government nutrition assistance			X
School wellness policiese				X
School and community gardens				X
Menu labelingf					X
Neighborhood availability of food stores					X
Neighborhood availability of restaurants					X
Neighborhood availability of food stores and restaurants					X
Provision of free or subscription fruits and vegetables at school					X
Provision of drinking water at school					X
Point of purchase prompts					X
Physical activity strategiesa
Community designc	Recommended (Community-scale urban design and land use policies)	X
School physical activity policies and environmentsd	Recommended (Enhanced school-based physical education)	X
Street designc	Recommended (Street-scale urban design and land use policies)	X
Availability of parks and recreation facilitiesd	Recommended (Creation of or enhanced access to places for physical activity combined with informational outreach activities)	X
Point of decision promptse	Recommended (Point-of-decision prompts to encourage use of stairs)	X
Transportationc	Insufficient Evidence# (Transportation and travel policies and practices)		X
Childcare physical activity policies and environments			X
Safe Routes to School				X
Traffic Safetyf				X
Interpersonal safetyf				X
Screen time	Policy and environmental strategies are not reviewed				X
School wellness policiesg					X

^aSee Appendix A (available online at www.ajpmonline.org) for definitions of the strategies.

^bThe most widely used systematic review in the U.S.

^cThe Community Guide Rating was based on literature prior to the review in 2006 (while this review included literature through May 2009).

^dThe Community Guide Rating was based on literature prior to the review in 2001 (while this review included literature through May 2009).

^eThe Community Guide Rating was based on literature prior to the review in 2005 (while this review included literature through May 2009).

^fNo consensus was reached for this subcommittee; investigators provided a conservative rating based on the subcommittee feedback.

^gSchool Wellness Policies have ratings for nutrition and physical activity.

Review System Reflections and Implications

This review found the evidence base for policy and environmental childhood obesity prevention strategies difficult to describe and summarize largely due to the scarcity of formal intervention evaluations, particularly those with strong study designs. To increase understanding of these interventions and their direct or associated outcomes, this review system explored new ways to characterize and synthesize practice-based evidence using assessments of its internal validity and external validity.^{22, 35–43} Through this process, investigators identified a number of common methodologic and measurement gaps that should be addressed in future research to strengthen the value/utility of the evidence base for policymakers and practitioners.

Understanding Intervention Effectiveness: Research Strengths and Gaps

The preponderance of multi-component or complex interventions (i.e., single component interventions with multiple intervention activities) made it difficult to attribute the outcomes to specific intervention strategies or activities. For the most part, the study designs employed fell short of estimating the independent population effects of specific intervention strategies or activities on BMI and important weight-related outcomes. Rigorous systems science approaches will be needed to track the multiple pathways from comprehensive policy and environmental changes to outcomes and to assess the different local, state, or national contexts.^{68, 69} To better understand policy and environmental strategy effectiveness and impacts, and to guide efficient use of complex, multi-pronged interventions, greater attention also needs to be given to identification of the cumulative benefits and costs of these interventions (including cost-effectiveness analysis).⁷⁰

The evidence maps created to summarize review results show a mix of positive (i.e., 49% for obesity/overweight, 65% for physical activity, and 53% for nutrition) and negative (i.e., 45% for obesity/overweight, 16% for physical activity, and 30% for nutrition) intervention evaluation effects. In contrast, most of the associations reported were positive, including outcomes related to obesity/overweight (75%), physical activity (78%), and nutrition/diet (71%). Overall, these evidence maps help to identify possible causal or correlational pathways for the intervention strategies; yet, the evidence for nearly all of these pathways is based on a single study or small handful of studies. More rigorous intervention evaluation studies are needed to replicate and validate the pathways from the evidence maps.

Complicating things further, outcome measures for the three major domains (obesity/overweight, physical activity, nutrition/diet) varied widely across studies making cross-intervention comparisons for studies of the same strategy difficult, if not impossible. This variation greatly complicates or prevents study-to-study comparisons, as well as the conclusions that can be drawn about the effectiveness and impact of any particular intervention strategy with respect to youth obesity levels.

Understanding Intervention Impact: Building the Evidence Base on External Validity

Most of the studies reviewed failed to report on key elements required for assessing the external validity or generalizability of intervention effects, including those elements specified by the RE-AIM framework.³³ For example, fewer than 10% of the study groupings included any assessment of intervention participation, a measure of intervention reach. Moreover, fewer than 20% of the study groupings included sufficient socio-demographic data for judging the degree to which the evaluation sample was representative of the population exposed to the intervention or the population for which it was designed. These limitations reflect the challenges inherent in assessing intervention reach for population-level interventions in comparison to smaller-scale interventions with defined exposed or unexposed groups (e.g., attendance in a program with an established curriculum).

Data on high-risk-population reach are necessary for ascertaining effects of policy and environmental changes on health equity, defined as opportunities to attain full health potential without being disadvantaged due to social position or other socially determined circumstance.⁷¹ Three fourths of the intervention evaluation study groupings reported insufficient data to assess high-risk-population reach. Within this group, the majority (54%) did not report the proportion of high-risk racial and ethnic or lower-income populations exposed to the intervention, and 43% did not report representativeness of the population exposed to the intervention as it relates to the intended population.

Data on intervention adoption, implementation, and enforcement were rarely reported. The degree to which policies are adopted (e.g., based in part on political and community support), implemented as intended (e.g., full or partial implementation), and enforced (e.g., authority and resources for policy compliance) can all affect policy effectiveness and impact. For instance, the adoption process may include community organizing and capacity building activities that influence civic engagement, and, in turn, participation in programs resulting in lifestyle and behavioral changes.^{72, 73}

Measures of intervention or policy feasibility were inconsistently reported across studies (e.g., intervention activities, expertise, resources). These gaps severely limit: evaluation of the intervention duration needed for a policy and environmental change to have an impact on the population; development of standards to assess implementation feasibility and minimal elements needed for change; and creation of methods and measures to assess intervention fidelity and the implications of adaptation or customization of interventions.

Only 3% of the study groupings reporting intervention effects noted that they were maintained beyond the end of the intervention. Outcomes related to intervention sustainability, bearing on the actual or potential institutionalization of a policy or environmental change, were reported in only 21% of the study groupings and omitted in 78% of them, and 1% of study groupings reported having no plans for sustainability.

To provide an overall snapshot of intervention effectiveness and impact, the 24 intervention strategies (with 25 classifications because school wellness policies yielded nutrition and physical activity classifications) were rated as “1st tier effective” (already judged effective by a formal systematic review), “2nd tier effective” (meriting additional evaluation and formal systematic review), “promising” (meriting additional evaluation), or “emerging” (meriting pilot studies and evaluability assessments).¹⁷ Only 11 of the 25 strategy classifications warranted ratings of “1st tier effective” or “2nd tier effective,” leaving policymakers and practitioners with more questions than answers about what works to prevent childhood obesity. Review results and tools (e.g., standards for assessing intervention effectiveness and impact, summary evidence maps) can inform researchers, funders, practitioners, and policymakers about the major strengths and gaps of the current and evolving evidence base.

Limitations of the Review System

Evidence is a moving target; any review is outdated when the search process begins given that new interventions constantly emerge in the field and new studies are continuously developed and reported. Any comprehensive evidence rating system requires the systematic collection, analysis, and reporting of an evolving evidence base. Measures of intervention effectiveness are more often reported and valued in the peer-reviewed literature than are equally critical measures of impact, more often reported in the grey literature. This review system depends on successful reporting of effectiveness and impact; thus, the findings presented in this paper likely reflect some publication bias. Population impact and how it relates to intervention dose is a relatively new concept in public health and it has been used in a wide variety of ways that differ from its use as a summary measure for reach, effectiveness, adoption, implementation, and maintenance/sustainability in this review.^74,75 Moreover, the translation of research and evaluation findings into widespread practice depends on research that assesses both the internal and external validity of interventions—as applied in real-world settings—harvesting practice-based evidence.

Updates to and replication of this review system (i.e., other topics) can help to improve practices for documentation, collection, reporting, and review of evidence. Investigators anticipate that future implementation of the review system will be more efficient with criteria and a system in place. In turn, funders, researchers, evaluators, policy- and decision-makers, practitioners, and community members can improve population health impacts through greater insights into the internal (effectiveness) and external (reach, adoption, implementation, sustainability) validity of policy and environmental strategies and systems. This requires greater time and resources to develop and refine the ratings in response to increased reporting of indicators related to internal and external validity of policy and environmental strategies as well as to search, abstract, rate, and summarize the evidence base.

Conclusion

Twenty-four different policy and environmental interventions are reviewed, rated, and classified using tools that illuminate research strengths, gaps, and priorities for multiple audiences seeking to halt or reverse the nation’s childhood obesity epidemic. While there is encouraging evidence for the efficacy of policy and environmental intervention strategies, there is limited understanding of the pathways from promising policy and environmental changes to demonstrated reductions in the prevalence of childhood obesity and in the marked socio-demographic disparities in its prevalence and its health and economic tolls.^1,10,11 This comprehensive review system can be regularly updated to continually assess interventions taking place at every level of the socio-ecologic model—to accelerate progress toward a world in which the healthiest choices for physical activity and diet are the easiest and most abundant choices for children and their families.