Evaluation Protocol To Assess an Integrated Framework for the Implementation of the Childhood Obesity Research Demonstration Project at the California (CA-CORD) and Massachusetts (MA-CORD) Sites

Abstract

Background: The long-term success of child obesity prevention and control efforts depends not only on the efficacy of the approaches selected, but also on the strategies through which they are implemented and sustained. This study introduces the Multilevel Implementation Framework (MIF), a conceptual model of factors affecting the implementation of multilevel, multisector interventions, and describes its application to the evaluation of two of three state sites (CA and MA) participating in the Childhood Obesity Research Demonstration (CORD) project.

Methods/Design: A convergent mixed-methods design is used to document intervention activities and identify determinants of implementation effectiveness at the CA-CORD and MA-CORD sites. Data will be collected from multiple sectors and at multiple levels of influence (e.g., delivery system, academic-community partnership, and coalition). Quantitative surveys will be administered to coalition members and staff in participating delivery systems. Qualitative, semistructured interviews will be conducted with project leaders and key informants at multiple levels (e.g., leaders and frontline staff) within each delivery system. Document analysis of project-related materials and in vivo observations of training sessions will occur on an ongoing basis. Specific constructs assessed will be informed by the MIF. Results will be shared with project leaders and key stakeholders for the purposes of improving processes and informing sustainability discussions and will be used to test and refine the MIF.

Conclusions: Study findings will contribute to knowledge about how to coordinate and implement change strategies within and across sectors in ways that effectively engage diverse stakeholders, minimize policy resistance, and maximize desired intervention outcomes.

Introduction

Despite national efforts to combat the obesity epidemic, almost one third of US children and adolescents 2–19 years of age remain overweight or obese (BMI ≥85th percentile).¹ Racially/ethnically diverse and low socioeconomic status groups in particular are disproportionately affected.² To more effectively prevent and control childhood obesity, federal agencies, such as the National Institutes of Health and the CDC, have begun to promote multilevel, multisector (e.g., family,³ school,⁴ healthcare,⁵ community,^6,7 and policy⁸) approaches that focus on changing not just individual behavior, but the broader sociophysical environment in which children live, learn, eat, and play.^9,10 Recent, large-scale initiatives, such as Shape Up Somerville¹¹ and the California Endowment's Healthy Eating, Active Communities,¹² provide preliminary evidence that social ecological approaches can promote healthy behaviors and prevent weight gain in children.

Developing effective approaches is a critical first step to addressing the obesity epidemic.¹³ However, subsequent dissemination and implementation of such approaches across communities play an equally important role in determining whether meaningful improvements in population health and well-being are achieved and sustained.¹⁴ Dissemination, defined as active and planned efforts to persuade target groups to adopt a new program, policy, or practice, influences the rate at which evidence-based approaches spread to new settings.¹⁵ Implementation, which refers to the processes through which a new program, policy, or practice is put into use,^16,17 is arguably even more critical given that it affects the extent to which desired outcomes are realized.^18,19

Implementation of multilevel, multisector approaches is challenging because they require significant buy-in and coordination of activities from diverse stakeholders, each of whom may vary in their readiness, capacity, and willingness to put in place the system, environmental, and policy changes involved.^20,21 The scale of proposed changes and the diversity of actors involved may also result in unintended consequences that adversely affect outcomes and sustainability (e.g., policy resistance).^21,22 Policy makers, practitioners, and health promotion researchers are increasingly aware of these challenges.⁹ However, in part because so few multilevel, multisector interventions have been developed, currently little is known about strategies for overcoming these challenges and effectively implementing childhood obesity prevention and control efforts within communities.

The current study contributes to the literature by introducing the Multilevel Implementation Framework (MIF), a conceptual framework, and describing its application to the evaluation of two of three state sites (MA and CA) implementing multilevel, multisector interventions as part of the Childhood Obesity Research Demonstration (CORD) project. By systematically examining the processes, activities, and resources affecting intervention uptake,^17,23 this study will contribute to knowledge about how to coordinate and implement change strategies in ways that effectively engage diverse stakeholders in implementing socioecological approaches to maximize desired intervention outcomes.

Initial Conceptualization of the Multilevel Implementation Framework

Over the last decade, a growing body of research has developed around the science of dissemination and implementation. Drawing upon research from multiple disciplines, such as healthcare,^21,24,25 public health,^26,27 and business,^16,28 a variety of models have been proposed to explain how different contextual factors and implementation activities influence organizational change efforts, consistency and quality of intervention use, and subsequent outcomes. As might be expected, these models are strongly influenced “by the service contexts chosen for emphasis and by the contextual levels that serve as primary organizing arenas”²⁴ and, consequently, tend to differ both in constructs examined and in the level of analysis on which they are focused.²⁹

The MIF is a conceptual model developed specifically for identifying factors affecting implementation of multilevel, multisector interventions, for example, obesity prevention and control initiatives that utilize a social ecological approach (see Fig. 1).^25,27,30 In developing the MIF, theoretical constructs from previous research particularly relevant to implementation of multilevel, multisector interventions were reviewed, adapted, and integrated into the framework. Specifically, the MIF is informed by constructs from the Interactive Systems Framework,²⁷ the organizational model of innovation implementation,^21,30 and the Consolidated Framework for Implementation Research.²⁵ A more detailed overview of how each contributed to the MIF is provided in Supplementary Appendix A (see online supplementary material at www.liebertpub.com/chi).

Figure 1.

Multilevel Implementation Framework. Adapted from the Consolidated Framework for Implementation Research,²⁵ the Interactive Systems Framework,²⁷ and the organizational model of innovation implementation.³⁰

Briefly, the MIF differs from other implementation models by describing how different factors interact not only within, but also across sectors to influence implementation effectiveness and subsequent intervention outcomes. Consistent with a social ecological perspective,³¹ the MIF does not view individual behavioral settings in isolation, but instead explicitly accounts for the broader community context in which behaviors are enacted.³² Selected constructs reflect multiple levels of influence (community, delivery system, and individual) and can be applied to identify factors affecting implementation in a wide range of contexts. For example, the MIF includes community-centered constructs that account for the roles of multiple stakeholders (funder, researcher, and practitioner) in the implementation process, as well as organization-specific constructs known to affect implementation within individual delivery systems. A complete list of MIF constructs, their definitions, and their hypothesized level of influence is provided in Table 1.^30,33–46

Table 1.

Key Constructs Affecting Implementation

	Definition
Level of analysis: all
Outer setting	Sociopolitical context, funding opportunities, and interorganizational networks that influence adoption and implementation24,33,34
Inner setting	Structural, political, and cultural context within a given entity responsible for implementation, including previous experience/history, resource availability, and leadership support for the intervention35
Characteristics of individuals involved	Knowledge, skills, beliefs, and personal characteristics of individuals responsible for implementing intervention activities and of intervention end users (e.g., children and their families).36–39 Characteristics of intervention end users play a particularly important role in implementation of public health and chronic care management interventions.38,39
Intervention characteristics	Attributes of the intervention that may influence the implementation, such as strength of evidence base, cost, extent to which intervention can be adapted to meet local needs, perceived advantages, and difficulty of implementing the intervention, compared to existing practice or an alternative solution15,40–43,88
Level of analysis: community (academic-community partnership and coalition)
Synthesis and translation system	Efforts by scientific leadership to distill and disseminate research evidence to other stakeholders (e.g., developing and adapting an intervention to fit local contexts)27,44
Support system	Stakeholder efforts to build capacity for implementing and/or sustaining the intervention (intervention-specific training and coaching, coalition and/or other partnership development, and so on)27,44
Delivery system	Inner context, implementation processes, and other factors influencing implementation in the organizations/sectors in which the intervention is taking place27,44
Level of analysis: delivery system
Readiness for change	Extent to which individuals responsible for implementation are psychologically and behaviorally prepared to make the changes necessary to implement the intervention.45,98 Strongly influenced by management support and resource availability
Implementation processes	Practices, policies, structures, and/or strategies used to put an intervention in place and support its use16
Implementation climate	Extent to which individuals responsible for implementation perceive that participation in intervention activities is expected, supported, and rewarded by the delivery system in which they are located67
Innovation-values fit	Extent to which the intervention is compatible with professional or organizational mission and values46
Innovation-task fit	Extent to which individuals responsible for implementation perceive intervention activities as compatible with local task demands (e.g., work processes and preferences)15
Intervention champion	An individual that promotes the intervention within the delivery system21

Description of the MA-CORD and CA-CORD Interventions

Funded by the CDC, the CORD project is designed to test the effectiveness of integrated clinical and public health evidence-based approaches to child obesity prevention and control.⁴⁷ Though specific activities vary across the CA-CORD and MA-CORD sites, the two interventions share a number of common features^48,49: Both CA-CORD and MA-CORD seek to effect change in obesity factors and outcomes among underserved children 2–12 years of age. Their interventions are based on the obesity chronic care model^50,51 and, as such, span multiple sectors (healthcare, school, early care, and education) and levels of influence (family, organization, and community).⁴⁷ CA-CORD and MA-CORD are each led by a project team that includes a public health department, an academic university, and at least one healthcare organization (among other partners). Both sites involve clinicians in promoting intervention activities in the clinic setting and have identified community coalitions as a primary mechanism for coordinating and sustaining intervention activities. Although the communities in which CA-CORD and MA-CORD are being implemented (Brawley and El Centro, CA and New Bedford and Fitchburg, MA, respectively) differ in urbanicity and racial/ethnic composition, they also share certain similarities: All of these communities are low income and have a high prevalence of child overweight and obesity, with specific rates ranging from 37% to 46%.^48,49

Table 2^52–59 provides a brief overview of CA-CORD and MA-CORD activities. More detailed descriptions of the overall CORD project, the CA-CORD and MA-CORD interventions, and the communities in which they are being implemented are available elsewhere.^48,49,60,61

Table 2.

Summary of Key Intervention Components^a

	MA-CORD Children ages 2–12 years in Fitchburg and New Bedford, MA	CA-CORD Children ages 2–10 years in Brawley and El Centro, CA
Community health centers (CHCs) Personnel: providers, nurse practitioner, medical assistants, registered dieticians, community health workers (CHWs)	• Two CHCs (one per community) • Practice change initiative based on High Five for Kids52 • Learning collaborative to improve obesity-related quality of care • Two healthy weight clinics: specialized unit for overweight/obesity referrals	• Two largest clinics within one CHC (one clinic per community) • Delivery system design including chronic care team and modifications to electronic health records • Practice team preparation including staff and provider training • CHW-led workshops based on several evidence-based interventions: Entre Familia,53,54Aventuras para Niños,55,56 Move/Me Muevo57
Schools Personnel: administrators, teachers, school nurses, food service staff, school wellness champion	Public elementary schools (N=23), middle schools (N=5), and after-school programs (N=15) in Fitchburg and New Bedford • Eat Well Keep Moving (grades 3–4) • Planet Health (grades 5–6) • Food & Fun (grades K–3, after-school sites) • Media campaign designed by schools	All public elementary schools (N=13) in Brawley and El Centro • School wellness policy change • SPARK Physical Education58 • Structural water promotion • Sleep curriculum and tip sheets • Parent outreach • Social marketing campaign
Early care and education centers Personnel: early care and education center directors, staff, nutritionist, health education specialists	Nine early care and education centers (for children ages 2–5 years) • I am Moving I Am Learning • NAP SACC59 • Media campaign	Twnety-six center-based and private early care and education centers • NAP SACC59 • Policy change • SPARK Physical Education58 • Quarterly trainings, toolkit, and technical assistance • Social marketing campaign
Community Personnel: coalition coordinator, public health officials, restaurant owners	• Efforts led by two municipal wellness coalitions • Safe Routes to School: planning only • Media campaign	• Parks and recreation and boys and girls clubs and gardens (N=4) • Restaurant intervention (N=3) • Social marketing campaign

^aMore detailed information regarding intervention design and evaluation of intervention effectiveness is available elsewhere.^48,49,60,61

NAP SACC, The Nutrition and Physical Activity Self-Assessment for Child Care.

Methods/Design

In applying the MIF to examine implementation of the CA-CORD and MA-CORD interventions, a convergent, mixed-methods study design⁶² involving concurrent qualitative and quantitative data collection at multiple time points and levels of influence will be used to (1) document intervention activities and the processes used to implement them and (2) identify factors affecting implementation effectiveness (i.e., consistency and quality of intervention use).¹⁶

Qualitative Procedures

Three inter-related methods of qualitative data collection will be used to identify intervention activities and implementation processes, examine facilitators and barriers to intervention uptake, clarify mechanisms used to coordinate and sustain intervention activities, and empirically verify the relevance of key constructs identified in Figure 1. These qualitative procedures include:

• • Semistructured interviews with site-specific CORD project leaders and key informants within each participating delivery system. Given the large number of individuals involved in implementation, a quota sampling approach will be used to ensure that key informants are selected at multiple levels within each delivery system (e.g., school district superintendent, principals, and teachers, clinic chief executive officer, providers, and frontline staff). Semistructured interview questions will gather information on constructs identified in the MIF (see Table 1). A common interview template will be used; however, specific questions will be tailored to reflect differences in respondent role (e.g., project leader, delivery system leader, and individuals responsible for implementing intervention activities) and level of influence as well as in the intervention activities being implemented at each site. Example interview questions are provided in Supplementary Appendix B (see online supplementary material at www.liebertpub.com/chi). Interviews are expected to last between 30 and 60 minutes and, with respondents' permission, will be recorded and transcribed verbatim. Key informant interviews will take place at two time points: shortly after initial implementation (baseline) and 12 months later. Interviews with project leaders will occur once, approximately 9–10 months after initial implementation.

• • In vivo observations and corresponding field notes from selected training sessions identified as critical to capacity-building efforts will be used to enhance the external validity of information obtained regarding dose delivered and received, as well as specific implementation strategies being employed.⁶³

• • Document analysis of project-related materials, such as investigator and coalition meeting minutes, which will be collected on an ongoing basis. Whereas these materials will primarily be used to document intervention activities and implementation processes, coalition meeting minutes will also be used to track changes in coalition size, diversity, stability, and outcomes (e.g., community changes, planning products, media products, and additional resources generated and/or leveraged) over time.^64,65

Quantitative Procedures

To examine dynamic, emergent effects of implemented changes on participating stakeholders,²² quantitative surveys will be administered at two levels of influence: community coalition and organization. These quantitative surveys will assess specific constructs hypothesized to affect implementation at two points of uptake: shortly after initial implementation (baseline) and 12 months later.

• • Coalition survey. The coalition survey will be administered to all coalition members in the four intervention communities. At baseline, the coalition survey will include the nine-item Organizational Readiness for Implementing Change (ORIC)⁶⁶ as well as additional measures assessing specific determinants of readiness for change (e.g., resource availability). At 12 months, the ORIC will be replaced with a six-item measure of implementation climate.⁶⁷ Both the baseline and 12-month surveys also include items assessing respondent demographic characteristics, 5-point Likert scale items regarding the perceived benefits of participating in the coalition,⁶⁸ and the Wilder Collaboration Factors Inventory (WCFI). The WCFI is comprised of 47 individual Likert scale items representing six categories of factors shown to affect the success of collaborative endeavors,^69,70 including: (1) the local environment in which coalition members are located (e.g., earlier history of collaboration); (2) member characteristics; (3) collaborative processes and structure; (4) quality of communication between coalition members; (5) clarity of coalition purpose; and (6) resources, specifically adequacy of coalition funds, staff, and leadership.

• • Organization-level survey. The organization-level survey will be administered to staff responsible for carrying out and/or coordinating intervention activities (e.g., teachers, early care and education center directors, community health workers, and providers). At baseline, survey items will include the nine-item ORIC,⁶⁶ items assessing respondent demographic characteristics, and four additional measures of innovation-values fit (extent to which intervention activities are compatible with professional or organizational mission and values), innovation-task fit (extent to which implementers perceive intervention activities as compatible with local task demands), management support, and resource availability. In the 12-month survey, the ORIC will be replaced with measures of implementation climate tailored to reflect unique intervention activities being implemented in each delivery system setting.^67,71 The 12-month survey also includes Likert scale items regarding perceived utility of specific implementation resources and capacity-building activities provided to respondents over the last year (e.g., training sessions). Additional, quantitative data on environmental and policy changes will also be collected at both time points as part of the overall CORD evaluation.^60,61

Measuring Implementation Effectiveness

Implementation effectiveness refers to the consistency and quality of intervention use.¹⁶ In the dissemination and implementation literature, conceptualizations of implementation effectiveness vary based on the nature of the intervention or organizational change being introduced.^72–75 In the CORD project, dimensions of implementation effectiveness identified as critical mediators of intervention outcomes include exposure (dose and reach), quality of delivery, fidelity, participant responsiveness, and differentiation. A detailed definition of each of these dimensions is provided in Table 3. Data on three dimensions of implementation effectiveness (dose delivered, reach, and fidelity) and participant-level intervention outcomes (e.g., child BMI, physical activity, and fruit and vegetable consumption) will be collected as part of the overall CORD evaluation and are described in more detail elsewhere.^60,61 Data on participant responsiveness and differentiation will be collected as part of the qualitative procedures outlined above.

Table 3.

Key Dimensions of Implementation Effectiveness

	Definition
Dose delivered	The amount of intervention delivered or provided by interventionists
Exposure (dose received)	The amount of intervention received by participating individuals and organizations
Exposure (reach)	Intervention coverage or reach (i.e., whether all participating individuals and organizations who should be exposed to the intervention are actually exposed)
Quality	The manner in which staff deliver the intervention (i.e., skill/preparedness, attitude toward participants)
Fidelity	Degree to which intervention components are delivered as intended
Responsiveness (participant engagement)	The extent to which participating individuals and organizations are engaged in intervention activities and content
Differentiation	Identification of core versus peripheral intervention components

Data Analysis

Qualitative data analysis will occur in multiple steps. First, the qualitative software program, NVivo (10.0; QSR International, Burlington, MA)⁷⁶ will be used to code all qualitative data files. The initial codebook will be informed by the MIF, but may be revised to include emergent constructs identified from the data. Potential coding areas include constructs identified in Table 1, discrete implementation strategies employed, unexpected outcomes experienced, and facilitators and barriers to implementation perceived by respondents. Codes will also be assigned to describe connections between constructs, including those at different levels of analysis (also known as axial coding).⁷⁷ All qualitative data files will be reviewed by at least two members of the research team. Any ambiguity or discrepancies in coding will be resolved through discussion and/or enhanced definition of codes. Once the codebook has been finalized, inter-rater reliability will be assessed by comparing level of agreement in the coding.⁷⁸ Next, we will generate reports of all text segments for each code. These reports will be analyzed to identify themes in the coded data for each construct as well as the degree to which each construct positively or negatively affected implementation. The relative importance of each construct will be examined within and across sectors and communities as well as by respondent role. Results will be shared with project leaders as an additional validity check and used to test and refine the MIF. Selected findings (e.g., summary of facilitators and barriers to effective implementation in each sector) will also be shared with stakeholders for improving processes and informing discussion of sustainability.

Quantitative survey data will also be analyzed in multiple steps. First, confirmatory factor analysis will be used to examine the underlying factor structure of quantitative survey measures. Assuming sample sizes are sufficient, measurement equivalence across sectors will also be assessed. Assessment of psychometric properties will be conducted using standard statistical packages, such as SAS (SAS Institute Inc., Cary, NC) or Stata (StataCorp LP, College Station, TX). Quantitative survey data will then be aggregated to the organization level and combined with qualitative data to empirically test pathways outlined in Figure 1, with a specific focus on establishing predictive validity (i.e., the extent to which constructs are associated with key dimensions of implementation effectiveness) and identifying best practices for implementation.

Analyses will be conducted in aggregate as well as by sector (clinic, school, early care, and education). Specific approaches used to analyze the data will vary by sector. For example, in the healthcare sector, where the number of clinics involved is small (n<5), a multiple case-study approach will be used. Case studies are well suited for studying nonlinear, context-sensitive processes, such as implementation,⁷⁹ and permit in-depth analysis of individual cases as well as systematic cross-case comparison. In the school and early care and education sectors, where the number of organizations involved is more moderate, case-study data will be analyzed using qualitative comparative analysis (QCA). QCA has been described as a bridge between quantitative and qualitative techniques because it utilizes principles from both case- and variable-oriented research to assess cross-case commonalities and differences that explain why an outcome (e.g., implementation effectiveness or behavior change) occurs.^80,81 As appropriate, qualitative and quantitative data will be transformed and calibrated for use in QCA. Analyses will be conducted and visualized using either the TOSMANA^81,82 or fsQCA⁸³ software, depending on the nature of the specific constructs being tested.

All study procedures have been approved by the institutional review boards at San Diego State University (San Diego, CA) and the Harvard School of Public Health (Boston, MA).

Discussion

In applying the MIF to evaluate implementation at the MA-CORD and CA-CORD sites, this study will provide both sites with information that can be used to facilitate process corrections or adjustments to intervention activities and/or implementation processes being employed.^84,85 In the long term, this study will also contribute to knowledge about how to effectively engage diverse stakeholder groups in multilevel, multisector interventions as well as coordinate and implement change strategies within and across behavioral settings in ways that minimize policy resistance, maximize desired intervention outcomes, and promote subsequent sustainment and scale-up of effective approaches.

As one of the first, large-scale initiatives to test combined public health and clinical approaches to child obesity prevention and control, the CORD project represents an important opportunity to better understand how specified intervention strategies and delivery systems may interact within communities to affect population outcomes over time. To date, few demonstration studies or controlled trials of social ecological approaches to health promotion have systematically collected practitioner- and/or system-level data on facilitators and barriers to implementation (i.e., the processes, activities, and resources affecting uptake).^17,23 Part of the challenge is that in their “ideal types,” effectiveness and implementation research do not share many design features, and thus many studies are not structured a priori to address both intervention and implementation aims.⁸⁶ In recent years, however, hybrid research designs that simultaneously assess program effects and gather information on implementation have been proposed as a way to maximize the relevance of research findings to practitioners and policy makers.^86,87

This type of hybrid effectiveness-implementation design may be particularly useful for evaluating interventions such as MA-CORD and CA-CORD, which include evidence-informed activities, but whose overall effectiveness in real-world settings is not as well established. All interventions are comprised of core components (elements essential to the internal logic of the intervention and considered directly responsible for intervention effects) as well as a soft, adaptable periphery (elements, structures, and systems that support implementation and sustainment, but whose necessity may vary across contexts).¹⁵ Adaptation at the periphery can facilitate delivery of core components in nonresearch settings and often reflects cultural and contextual “translations” critical to successful dissemination and implementation efforts.^88,89 However, adaptation can also result in drift, that is, abandonment of core components or introduction of counterproductive elements that negatively affect intervention effectiveness.^88,90 In complex interventions such as CA-CORD and MA-CORD, the distinction between core and peripheral intervention components is often difficult to determine and may be discerned only through trial and error over time as an intervention is “scaled up” and adapted for use in a variety of new contexts.^25,91 Careful monitoring of intervention activities and implementation processes can significantly speed up this process. Specific evaluation activities may vary based on available resources. In contexts where resource constraints and/or respondent burden are of particular concern, document analysis of meeting minutes and semistructured interviews with project leaders can capture ad-hoc adaptation and other system changes without contributing significantly to program costs. By providing information of use in evaluating a program's potential for translation and impact in other contexts, this type of approach can decrease the time lag between “discovery” and “routine use” of programs.^25,29

Limitations

As with all studies, a number of limitations should be considered. First, implementation is often a dynamic, nonlinear process characterized by considerable ambiguity.^92,93 Although data will be collected at multiple points in time, recall bias may limit the accuracy of information provided by respondents. The use of multiple data sources and informants will minimize the threat of this bias and increase the validity of study findings.^94,95 Second, all qualitative procedures are susceptible to subjective bias and preconceived ideas of investigators. To minimize such bias, multiple coders will be used to analyze qualitative data collected in this study. Project leaders and other key stakeholders involved with CA-CORD and MA-CORD will also have the opportunity to review and provide feedback on study findings.

Conclusion

Significant resources have been invested in the development of interventions to prevent and control childhood obesity. The long-term success of these efforts depends not only on the efficacy of the approaches selected for use, but also on the strategies through which they are implemented and sustained.^46,96,97 By systematically collecting data on processes, activities, and resources affecting intervention uptake in the CA-CORD and MA-CORD sites, this study will contribute to knowledge about how to coordinate and implement multilevel, multisector change strategies in ways that effectively engage diverse stakeholders, minimize policy resistance, and maximize desired intervention outcomes.

Acknowledgments

Preparation of this article was supported by the CDC (1U18DP003377-01 and 1U18DP003370-01) and by the Pilot Studies Core of the Johns Hopkins Global Obesity Prevention Center, which is funded by the National Institute of Child Health and Human Development (U54HD070725). The information and opinions expressed herein reflect solely the position of the authors.

Author Disclosure Statement

No competing financial interests exist.