The Childhood Obesity Declines Project: Implications for Research and Evaluation Approaches

Abstract

Background: Childhood obesity remains prevalent and is increasing in some disadvantaged populations. Numerous research, policy and community initiatives are undertaken to impact this pandemic. Understudied are natural experiments. The need to learn from these efforts is paramount. Resulting evidence may not be readily available to inform future research, community initiatives, and policy development/implementation.

Methods: We discuss the implications of using an adaptation of the Systematic Screening and Assessment (SSA) method to evaluate the Childhood Obesity Declines (COBD) project. The project examined successful initiatives, programs and policies in four diverse communities which were concurrent with significant declines in child obesity. In the context of other research designs and evaluation schemas, rationale for use of SSA is presented. Evidence generated by this method is highlighted and guidance suggested for evaluation of future studies of community-based childhood obesity prevention initiatives. Support for the role of stakeholder collaboratives, in particular the National Collaborative on Childhood Obesity Research, as a synergistic vehicle to accelerate research on childhood obesity is discussed.

Results/Discussion: SSA mapped active processes and provided contextual understanding of multi-level/component simultaneous efforts to reduce rates of childhood obesity in community settings. Initiatives, programs and policies were not necessarily coordinated. And although direct attribution of intervention/initiative/policy components could not be made, the what, by who, how, to whom was temporally associated with statistically significant reductions in childhood obesity.

Conclusions: SSA provides evidence for context and processes which are not often evaluated in other data analytic methods. SSA provides an additional tool to layer with other evaluation approaches.

Background

Recent Centers for Disease Control (CDC) data (2016) regarding prevalence of childhood obesity shows increases at all ages, at each level of overweight and obesity, for both males and females. Minority and underserved youth continue to see disproportionate increases.^1,2 Populations with the highest prevalence often do not have access to successful intervention strategies to prevent and treat obesity during childhood, before comorbid chronic diseases (e.g., diabetes, cardiopulmonary conditions, and orthopedic disability) develop.³ Thus, disadvantaged populations are in need of effective feasible community-based prevention and intervention programs that could help ameliorate this pandemic problem.

This article discusses the rationale for the use of the systematic screening and assessment (SSA) method by the Childhood Obesity Declines (COBD) project, a method that facilitates identification of programmatic and policy efforts to address rates of childhood obesity at the community level; highlights evidence generated by this evaluation strategy in contrast to other experimental designs; suggests guidance for future studies of community-based childhood obesity prevention; and supports the role of stakeholder collaboratives, in particular the National Collaborative on Childhood Obesity Research (NCCOR) (www.nccor.org) as a synergistic vehicle to accelerate research in childhood obesity.

Rationale for Use and Adaptation of the SSA Method

Although randomized clinical trials remain the gold standard for demonstrating the efficacy of behavior change strategies, translation of behavior change approaches has been difficult in free-living populations within their environmental context. Major impediments to adoption of behavior change strategies in real-world settings include uncontrolled influences such as social determinants of health,⁴ contextual characteristics,⁵ availability of resources,⁶ and prevailing behavior patterns and health beliefs between and among networks of individuals^7,8: elements that typically inform community-based initiatives.⁹ Thus, in addition to highly controlled behavioral clinical trials, alternative intervention approaches and evaluation strategies are needed for initiatives that are community based and multilevel, wherein randomization to intervention vs. control condition is not specified, desired, or considered feasible.

Modified SSA follows the Institute of Medicine L.E.A.D. framework that outlines a process by which evidence that is currently available can be identified, evaluated, and compiled to inform decisions regarding obesity prevention and treatment.¹⁰ For the purposes of this article, the term “natural experiment” has been adopted to describe the nonrandomized multilevel community strategies/interventions/programs that were examined post hoc in COBD. According to Medical Research Council guidance: “The common thread in most definitions (of natural experiments) is that exposure to the event or intervention of interest has not been manipulated by the researcher.” (pg. 1183).¹¹ The researchers are in this case the COBD evaluation group. The cluster of initiatives and policies was used intentionally by community and policy implementers, but not necessarily in an intentionally coordinated manner and then evaluated post hoc by the COBD research group.

The COBD project used an adaptation of the SSA^12,13 described in this issue,¹⁴ to demonstrate a retrospective systematic reproducible approach to identify and examine community-based initiatives intended to promote healthy eating and active living strategies to influence weight outcomes in children. Each case cited was community based and stakeholder driven, implemented locally, and tailored to the targeted population by the implementers.¹⁵

In contrast to controlled experiments that target behavior change on the individual level, randomize communities or entities to intervention or control condition, or observational studies that provide a “snapshot” of behavioral patterns at single or different points in time, COBD examined potential drivers of population-level change that occurred simultaneously, outside the framework of a controlled experiment, and appear to be synergistic. Specific attribution of causality to programmatic components cannot be made. Regardless, rates of childhood obesity declined, and there is a need to understand what, how, where, and to whom initiatives produced desired outcomes. The COBD project was designed to identify these critical elements.

Evidence Derived from Evaluation in COBD Using SSA vs. Observational and Clinical Trial Evaluation Schemas

The adapted version of the SSA methodology does not, by design, examine specific mechanisms and specificity of action in highly characterized or delimited groups of subjects. The COBD is grounded in the social ecologic model and hypothesizes that understanding individual behavior and behavior change processes must take into account multilevels of contextual influences,¹⁶ also making the method effective for evaluation of groups of people in situ, in free-living conditions. It “unpacks” successful efforts to identify strategies that could prove effective in community settings.

Use of the SSA schema for evaluation does not address dose, strength of the intervention components, systematic implementation, or mechanisms/pathways of action.¹⁷ Effect sizes of contextual influences cannot be determined in uncontrolled experiments and are often not addressed in randomized clinical trials. COBD examination of the implementation of prospective strategies with concurrent reduction of BMI status in children who were intended recipients suggests attribution of effectiveness (producing the desired result) by comparison of baseline and postimplementation rates of obesity.

After examination of published reports of decline, COBD took the further step of verifying statistical significance in each of its chosen sites, a traditional use of change in primary outcome to determine success or failure of a clinical trial. A matched control population that did not receive the same package of initiatives would have allowed direct attribution of intervention effects. However, community-based approaches and policy implementation with public health intent often do not a priori utilize this evaluation schema, nor is it generally feasible given community needs and resources.

Natural experiments, particularly when generated by community and policy stakeholders, compared with clinical or observational trials, do not preselect or randomize recipients to the intervention. Natural experiments (as defined here) do not control dose or systematic implementation. Based on community needs and local policy implementation to inform multiple intervention components, natural experiments have the capacity to address multilevel factors in context. Randomization in clinical trials aims to account for variance between the experimental and control condition due to unidentified inputs. Preselecting recipient groups in randomized clinical trials aims at controlling individual level and group factors that might impact response to intervention.

Randomization is difficult in free-living conditions because of contamination in exposure to the inputs, establishing equivalence of population characteristics that are geographically separated, and/or determining appropriate level of randomization—person, system, and/or community—in aggregate threats to validity.¹⁵ Furthermore, natural experiments often do not have a formal or systematic evaluation schema to enable association of rigor and dose of processes with outcomes.¹⁶ As an example, as regards policy implementation, the rigor, penetration, and available resources may not be uniform across settings and populations. Success of policy implementation and prototypic intervention strategies may not be manualized for fidelity in delivery.

Social determinants of health, health literacy, and reduced access to services and resources may or may not be addressed or evaluated in a community-based policy implementation initiative. Diverse nonselected populations, or in other words, the diverse community at large, cannot be assumed to benefit equally from nontailored initiatives.¹⁷ Rather, because natural experiments are most often embedded in complex systems and have a myriad of “inputs” reflecting free-living conditions and environmental context, the main focus becomes population-level change.¹⁸

The COBD project, using a retrospective adaptation of SSA, addresses the design characteristics of natural experiments: evaluation is grounded in the context in which initiatives occur; it is descriptive, process-focused, and temporally anchored; primary data collection occurs through narrative; identification of key components is informed by multiple stakeholders including implementers; convergent validity is utilized to establish a timeline of implementation; and implementation stories from multiple stakeholders identify leaders, champions, implementers, settings, and funders. This information serves to identify what, how, and for whom the initiatives occurred.

Contextual implementation assumes that each experiment is driven by community needs. Goals, resources, champions, context, and scope are determined by stakeholders rather than by an external research agenda.¹⁹ (See definition of natural experiment.) Thus, in the context of no overarching a priori evaluation schema, this adaptation of the SSA method provided insight into how community-tailored in situ components occurred during a time of reduced rates of childhood obesity. In contrast to controlled experimental behavioral manipulation, strategies were driven by public health goals.

Use of SSA to retrospectively evaluate such natural experiments points the policy maker, public health practitioner, community or organization program implementer, and/or researcher to feasible strategies, which can be used to drive change in child weight status through community-generated stakeholder engagement, policy, systems, and environmental changes, as well as program efforts. Initiatives in four different geographic sites identified different sets of contextual characteristics, resources, champions, policies, and intervention components. Thus, using SSA methodology identified not only the strategies that were occurring in these natural experiments but also the contextual influences, thereby identifying what, how, where, and to whom these efforts were intended.

Guidance for Development of Future Efforts to Reduce Population Rates of Childhood Obesity

To place COBD in context, a representative scan of the literature on childhood obesity interventions (for the past 5 years) was conducted using SCOPUS (www.elsevier.com/__data/assets/pdf_file/.../scopus-quick-reference-guide.pdf). The majority of projects identified (∼1240) were either clinical trials or observational cohort studies, designed and implemented by academic researchers. Also included were reports in the public domain, including the gray literature and community public health evaluation reports. Thus the total number of identified initiatives included those with varying degrees of design and evaluation rigor.

Exemplar studies from NCCOR stakeholders illustrate interventions using traditional study design. As an example of controlled experimental design, the NIH-funded ORBIT studies were small clinical trials designed to establish proof of concept and feasibility of novel behavioral approaches to impact mechanisms effecting obesity-related individual behavior of underserved minority individuals across the lifespan, including children.²⁰ Studies were developed to test a wide variety of mechanisms that could guide intervention trials, from individual-based factors to environmental manipulation, to reduce rates of obesity.

The Healthy Communities Study (HCS), mentioned in the introduction to this issue, is an exemplar observational cohort study. The NIH-funded HCS targeted jurisdictions with underserved child populations and high rates of obesity, mapping current local efforts in each community.²¹ The primary aim of this study was to determine the predictive validity of composite efforts that could be associated with health, including potential reductions in BMI, in the targeted child populations. HCS methods characterized the target populations, mapped efforts, but did not address community implementation processes.²²

The CDC Childhood Obesity Research Demonstration (CORD) studies were by design whole-of-community obesity prevention and obesity management initiatives embedded in community settings, including schools, early care and education, and healthcare. CORD programs were implemented in low-resourced communities using evidence-based interventions adapted for the community context. CORD evaluations used a variety of methods, including randomized clinical trials for obesity management programs, observational cohort studies, and pre-post intervention comparisons of primary outcomes. Studies were prospective and focused on underserved, minority, and at-risk child populations. CORD studies had common intervention obesity–behavior targets as well as some overlapping evaluation strategies, but intervention components at each site were tailored to specific high-risk ethnic and minority groups.

CORD intervention components included selected cultural elements, community health workers, and addressing population and community needs. Individual behavior change and weight status were primary outcomes.²³ In so far as randomization to condition took place, dosing and control of purported active intervention components were monitored. Community settings identified contextual programmatic elements, but these were not systematically mapped or experimentally manipulated inputs. Thus, the CORD studies could be considered hybrid design: community-based participatory plus clinical trial design. Evidence generated from each of the CORD studies was specific to the targeted population.

In contrast to the three studies already mentioned, the COBD study cites efforts were driven by community stakeholders and policy makers, without an intentional coordinated evaluation of the impact of all the potential factors contributing to the observed decline in obesity. COBD did identify programmatic and policy efforts that were simultaneously implemented but not randomized, or dosed as part of a standardized intervention protocol, or controlled. The stability of targeted COBD populations was not assessed, nor population characteristics defined, except for the location and age of the children and source of BMI data. Outcomes were thus based on a population statistic demonstrating significant declines in rates of obesity in children who were the recipients of the package of initiatives and policies. COBD sites did not aim to demonstrate correlational associations of specific programs, policies, or initiatives with weight outcomes. Despite a lack of a priori systematic evaluation approach, declines were demonstrated in subgroup population prevalence of obesity.

COBD methodology took a case study approach: COBD sites were chosen for effectiveness, that is, reductions in rates of child obesity. This adaptation of SSA mapped active processes and provided contextual understanding. Thus, COBD provides guidance for dissemination of potentially effective approaches in real-world settings. The benefit of using this methodology is that it allows for evaluation of “packaged” policy and programmatic components, generates information on policy implementation processes and contextual factors, and provides proof of concept. Given that efforts to reduce childhood obesity will likely need to occur on the population level, use of this methodology provides an added layer of evaluation and evidence beyond traditional experimental designs.

Limitations

As previously noted, evaluation of the COBD natural experiments was largely descriptive, and direct attribution of active intervention components to observed outcomes cannot be made. Furthermore, effectiveness regarding individual behavior change cannot be determined. Mechanisms that have previously been implicated to contribute to the development of and resistance to treatment of obesity such as poverty, stress, and early life adverse events²⁴ were not examined in the COBD project. However, use of modified SSA provides a post hoc method to assess the prevalence of such factors in the examined communities using population-based databases (such as National Health and Nutrition Examination Survey [NHANES]²⁵), thus enriching potential intervention targets.

Community-based programs and policy implementation efforts are well suited to address social determinants of health: the social and environmental systems in which people live. Thus, there is a confluence between natural experiments and the use of this adaptation of the SSA method. To better understand individual behavior change, examination of the underlying mechanisms of programmatic elements of natural experiments would need to occur. Although expensive and difficult to implement in community settings, this combination of experimental designs could help identify characteristics of responders and nonresponders, setting characteristics, needed doses of intervention components, penetration of policies, cost of implementation, and other factors that are well recognized as contributors to the understanding of individual- and population-based behavior change.²⁴

How Research Collaboratives Can Promote Novel Research and Evaluation Strategies

COBD occurred through the collaborative of NCCOR members. The working group members represent a unique partnership between the major funders of childhood obesity research in the United States: the Centers for Disease Control and Prevention, National Institutes of Health, Robert Wood Johnson Foundation, and the U.S. Department of Agriculture (www.NCCOR.org). Each organization views the issue of childhood obesity through an agency-specific lens and all differ in their research and evaluation agenda. All are focused on reducing rates of childhood obesity in the United States, particularly for underserved populations such as low-income and communities of color. This collaborative recognizes that traditional research and evaluation methods need to be augmented to address the complexities of the impact of community-based initiatives and policy implementation. The case is made here that proof-of-concept trials, randomized clinical trials, hybrid trials, and social ecologically driven natural experiments each adds layers of evidence to the identification of effective strategies to reduce rates of childhood obesity, and used synergistically, can accelerate efforts to find effective obesity reduction strategies.

NCCOR's focus on harmonizing evidence obtained from these different methods, and dissemination of efficacious methods to real-world settings, is a prototype for research collaboratives. NCCOR serves the purpose of accelerating the field of obesity prevention and treatment research and evaluation by promoting unique research opportunities.²⁵

Acknowledgments

The NCCOR is a private and public partnership between the CDC, NIH, RWJF, and USDA that provided technical assistance for this study. FHI360 serves as the Coordinating Center for NCCOR. ICF International served as the lead contractor for the study. The authors thank NCCOR members Melissa Abelev, Veronica Uzoebo, and Ruth Morgan of the Food and Nutrition Service of USDA for participation in the study advisory committee. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC, NIH, USDA, or any of the other project agencies. The Robert Wood Johnson Foundation funded this project (ID #71772—Analyzing the Signs of Progress in Childhood Obesity, Princeton, NJ).

Author Disclosure Statement

No author has any commercial associations that might create a conflict of interest.

The findings and conclusions of this article are those of the authors and do not represent the official position of the National Institutes of Health, the Centers for Disease Control and Prevention, ICF, or Robert Wood Johnson Foundation. The authors did not report any conflicts of interest or financial disclosures.