The What, When, And Why of Implementation Frameworks for Evidence-Based Practices in Child Welfare and Child Mental Health Service Systems

Abstract

It is widely recognized that children in the child welfare system are particularly vulnerable to the adverse health and mental effects associated with exposure to abuse and neglect, making it imperative to have broad-based availability of evidence-based practices (EBPs) that can prevent child maltreatment and reduce the negative mental health outcomes for youth who are victims. A variety of EBPs exist for reducing child maltreatment risk and addressing the associated negative mental health outcomes, but the reach of these practices is limited. An emerging literature documents factors that can enhance or inhibit the success of EBP implementation in community service agencies, including how the selection of a theory-driven conceptual framework, or model, might facilitate implementation planning by providing guidance for best practices during implementation phases. However, limited research is available to guide decision makers in the selection of implementation frameworks that can boost implementation success for EBPs that focus on preventing child welfare recidivism and serving the mental health needs of maltreated youth. The aims of this conceptual paper are to (1) provide an overview of existing implementation frameworks, beginning with a discussion of definitional issues and the selection criteria for frameworks included in the review; and (2) offer recommendations for practice and policy as applicable for professionals and systems serving victims of child maltreatment and their families.

Introduction

Child maltreatment is a prevalent (e.g., Finkelhor, Turner, Shattuck, & Hamby, 2015), significant and costly public health problem in the United States (Corso, Edwards, Fang, & Mercy, 2008; Hammond et al., 2006; Fang, Brown, Florence, & Mercy, 2012) and worldwide (Butchart & Mikton, 2014), with extensive data indicating its association with myriad adverse short-term and long-term physical and mental health consequences (Anda et al., 2006; Hanson et al., 2008; McLaughlin et al., 2013; Walsh et al., 2012). There is a dire need for the widespread use of evidence-based practices (EBPs) that can prevent child maltreatment in families at-risk and improve the outcomes for youth who have experienced maltreatment. To date EBP access is limited (Costello, Egger, & Angold, 2005; Hurlburt et al., 2004; Merikangas et al., 2010), and the limited reach is particularly pronounced among traditionally underserved racial/ethnic minority children, who are overrepresented in the child welfare system (Smedley, Stith & Nelson, 2003; U.S. DHHS, 2013).

A review of the extant research literature identifies several factors that may enhance successful implementation of an EBP. These include increased attention to EBP selection, both in terms of innovation-organizational fit (Meyers, Durlak, & Wandersman, 2012), as well as systemic fit related to availability of funding and whether the EBP meets the needs of the identified community, participating agencies and targeted consumers (Aarons, Hurlburt, & Horwitz, 2011); thoughtful identification of agencies and providers for EBP training (Fixsen, Blasé, Naoom, & Wallace, 2009); extensive initial training and ongoing support for sustained delivery of an EBP with fidelity (e.g., Beidas, Edmunds, Marcus, & Kendall, 2012; Durlak & DuPre, 2008; Sholomskas et al., 2005); and an emphasis on senior leadership (e.g., Aarons & Sommerfeld, 2012; Fixsen et al., 2009) to promote a supportive organizational culture and climate (Aarons & Sawitzky, 2006; Glisson et al., 2010; Glisson & Green, 2011; Meyers et al., 2012). Most researchers and community stakeholders recognize the need for ongoing monitoring and evaluation throughout implementation to promote sustained use of EBPs with fidelity (e.g., Herschell, 2010; Meyers et al., 2012). Relatedly, studies highlight the importance of sustainability planning during the initial phases of implementation to increase the likelihood of implementation success (Beidas, Mehta, Atkins, Solomon, & Merz, 2013; Peterson et al., 2014; Pluye, Potvin, & Denis, 2004; Shediac-Rizkallah & Bone; 1998; Wiltsey-Stirman et al., 2012).

Perhaps one of the most overlooked factors is how a theory-driven conceptual framework, or model, might facilitate implementation planning by providing guidance for best practices during implementation phases (Aarons et al., 2012; Metz et al., 2014). Emerging research provides support that approaching implementation with a conceptual framework for EBPs targeting child maltreatment prevention (Aarons et al., 2012), child mental health interventions (Glisson et al., 2010) and other community-based EBPs (Brodowski, Counts, Gillam, Baker, & Collins, 2013; Gopalan et al., 2014; Metz et al., 2014) can enhance the process and ultimately increase overall implementation success. The purpose of this paper is to offer a structured review of the scientific literature documenting the application of existing implementation frameworks to EBP implementation in child welfare and child mental health, specifically with EBPs that can reduce child maltreatment perpetration or recidivism, and those that address the mental health needs of maltreated youth.

While other reviews have been conducted (Aarons, 2012, Damschroder et al., 2009; Proctor, Powell, & McMillen, 2013; Rabin et al., 2008; Tabak, Khoong, Chambers, & Brownson, 2012; Wandersman, Chien, & Katz, 2012), these primarily discussed the use of frameworks for dissemination and/or implementation research and did not specify an implementation context. The most inclusive review to date by Tabak et al. (2012) identified 61 models and provided guidance for implementation science researchers on how to select from and apply frameworks to research studies. In contrast, rather than provide an exhaustive review with a focus on research exclusively, we highlight frameworks most relevant for use in child welfare and child mental health, with an emphasis on service and policy implications for children and families impacted by abuse or trauma. Below, we explicate the terms utilized for purposes of this review, as well as the inclusion criteria for the selection of implementation frameworks.

Defining Terms

For the purposes of this review, it was critical to select terms for the treatments or interventions being implemented, as well as the frameworks and models to be discussed in relation to the implementation processes. The term evidence-based practice (EBP) was selected to refer to a treatment or intervention that has been demonstrated, typically through randomized controlled trials, to be safe and effective when delivered with fidelity (Chaffin & Friedrich, 2004; Rabin, Brownson, Haire-Joshu, Kreuter, & Weaver, 2008). Additionally, consistent with prior reviews (e.g. Tabak, Khoong, Chambers, & Brownson et al., 2012; Walsh, Reutz, & Williams, 2014), we elected to use the term framework, defined as “strategic or action-planning models that provide a systematic way to develop, manage, and evaluate interventions” (p. 337).

Guidance for Framework Selection

A variety of conceptual frameworks and models have been proposed as heuristics to guide successful dissemination/implementation (D/I) efforts (for reviews, see Beidas et al., 2013; Meyers et al., 2012; Tabak et al., 2012), with research indicating that longer-term, multilevel implementation strategies are needed to ensure penetration and sustained use of efficacious treatments across child welfare and community-based mental health settings (e.g., Damschroder et al., 2009; Fixsen et al., 2005). Conceptual frameworks are important to the extent that they help to guide implementation; however, decisions about which are most useful within a given setting can present significant challenges to stakeholders embarking on an implementation effort. To further compound these challenges, there is no single framework identified by extant research to guide implementation specifically within the child welfare or child mental health settings. This paper provides an overview of several models that have been applied to the implementation of EBPs within the context of child welfare and children’s mental health. While this focus on child welfare and children’s mental health initially may seem somewhat diffuse, given the overlap of children receiving services in these two service systems, it is important to include EBPs with relevance to both. Additionally, since maltreated youth have 2.5 times greater risk for developing mental health problems than children in the general population (Burns et al., 2004), access to EBPs that address the mental health needs of youth in child welfare is imperative.

Since there are no published works to date that have centered on an exclusive review of frameworks and applied examples used in EBP implementation in child welfare and child mental health service settings, this formed the basis for inclusion in this review. Additional criteria for inclusion included the following: evidence of a theoretical basis, guidance for the full realm of D/I activities, inclusion of practice and policy-relevant planning, training/consultation, evaluation, and/or sustainability activities, and at least one published peer-reviewed article describing application in child welfare or child mental health service settings. We began with frameworks included in prior reviews (Beidas et al., 2013; Meyers et al., 2012; Tabak et al., 2012) and distilled to ten frameworks meeting these criteria. We acknowledge that this is not an all-inclusive, exhaustive listing of conceptual frameworks, but instead a selection to inform efforts specifically in these two service settings. Following presentation of the ten frameworks below, we provide a discussion of implications and guidance on selecting a framework relevant to implementation decision-making and policy that can impact care, treatment, and service delivery for at-risk or maltreated children and their families. The ten frameworks are organized according to their distinguishing features and include those that: a) are structured by stages or phases (“Stages of Implementation Frameworks”); b) consolidate and/or integrate core components from multiple existing models or theories (“Consolidation and Core Component Frameworks”); c) emphasize organizational support (“Organizational Support Frameworks”); and d) target planning specifically (“Planning Frameworks”) (see Table 1).

Table 1.

Implementation Frameworks: Overview of Common Elements and Distinguishing Features

Implementation Framework	Common Elements					Distinguishing Features
	Stages	Domains	Inner/Outer Setting	Implementation Teams	Sustainability
Stages of Implementation Frameworks
Fixsen et al.	Y	Y	Y	Y	Y	Implementation Drivers Implementation Teams
EPIS	Y	N	Y	N	Y	Interaction of the Inner and Outer Context for promoting implementation success
AIF	Y	N	N	Y	Y	Implementation drivers Focus on sustainability throughout all initial phases of implementation
Learning Collaborative	Y	N	N	Y	Y	Agency-based implementation teams include front line providers, supervisors and senior leaders Emphasizes quality improvement and active learning strategies
Consolidation and Core Component Frameworks
PRISM	N	Y	Y	Y	Y	Reliance on implementation teams with senior leaders, middle managers, and frontline staff Emphasizes interaction between intervention and characteristics of organization and recipients
CFIR	N	Y	Y	N	N	Fuses common constructs from existing frameworks to guide evaluations of implementation
RE-AIM	N	Y	N	N	N	Used to measure public health impact of interventions
Systems and Organizational Support Frameworks
ISF	N	N	Y	N	N	Three interrelated systems: Prevention Synthesis and Translation, Prevention Support, and Prevention Delivery Innovation-specific capacity-building General capacity-building
ARC	N	Y	Y	N	N	Change agents
Planning Framework
PRECEDE-PROCEED	Y	Y	Y	N	N	Preparation focused Included participatory driven planning to identify most impactful EBPS

Framework Review

Stages of Implementation Frameworks

Stages of Implementation

Fixsen and colleagues identified six essential stages in implementation: exploration, installation, initial implementation, full implementation, innovation, and sustainability (Fixsen et al., 2005; Fixsen et al., 2009). They suggest that the stages do not progress linearly, but interact with each other throughout. Several core components, or ‘implementation drivers,’ have also been identified and are classified as Competency Drivers (e.g., staff selection, training, and coaching), Leadership Drivers, and Organization Drivers (e.g., facilitative administration, decision support data systems). The framework further specifies the critical importance of Implementation Teams for successful implementation, as these individuals are actively involved in the initiation, oversight, and facilitation of implementation (Fixsen et al., 2009, p. 532). Teams may comprise program developers and purveyors, intermediary organizations that assist other organizations in implementing EBPs, or on site staff who receive support from outside the organization. The Implementation Teams have considerable knowledge about EBPs, implementation science and best practices, and are responsible for assuring that interventions and implementation are effective and yield beneficial outcomes for children and families.

Application

The Stages of Implementation framework initially put forth by Fixsen and colleagues (2005) was derived from an extensive review of the implementation literature that included multiple disciplines (e.g., psychology, medicine, engineering, sociology) applied across a variety of settings (e.g., business and industry, human services, schools, community mental health, juvenile justice, child welfare). Subsequent applications focused more specifically on work in child mental health and child welfare (Fixsen, Blasé, Metz, & Van Dyke, 2013). One example is the implementation of Multidimensional Treatment Foster Care (MTFC), an intervention with demonstrated effectiveness in reducing behavioral problems of youth in foster care. When this framework was applied to implementation planning, a few key agency staff were selected from the target agency and trained by the program’s developers (Chamberlain, Price, Reid, & Landsverk, 2008). These individuals comprised the Implementation Team, who provided oversight and supervision of the local implementation efforts, as well as training and coaching of new staff in the model. This application allowed for a titrated approach to implementation, where in the early phases, the implementation team was extensively trained and supported by purveyors; but, over time as the expertise of the implementation team increased, they become more self-sufficient in leading all aspects of the implementation planning and training. Consequently, the implementation site did not have long-term reliance on expertise and consultation, allowing for a cost-effective and practical sustainability plan (Fixsen et al., 2009).

EPIS Framework

As a result of extensive work in child welfare and mental health, Aarons and colleagues define different stages or phases (Exploration, Adoption/Preparation, Implementation, and Sustainment) involved in implementation of an EBP and also highlight the importance of factors operating at the inner and outer contexts that influence successful implementation (Aarons, Hurlburt, & McCue Horowitz, 2011; Aarons et al., 2012). Inner context factors include those at the individual level, such as attitudes and experience of the individual provider; aspects of the participating agencies, such as organizational culture and climate that may hinder or promote implementation; and leadership characteristics; whereas the broader, ‘outer’ layer relevant factors include existing policies (at the local, state, regional and national levels), availability of funding, and political climate. Of note, these inner and outer contextual factors can operate differentially across the phases of implementation.

The Exploration and Adoption phases begin with awareness of an issue facing an organization, or interest in integrating a better method to address an existing organizational challenge. The outer context can either encourage or discourage an organization’s exploration and adoption of new interventions or practices. Inner contextual factors such as openness to innovation, organizational culture and climate are also influential in increasing likelihood of adoption.

The Implementation phase is heavily influenced by the outer context of funding, as fiscal resources are necessary for implementation activities (Aarons et al., 2011). The structure, goals, and extent to which an organization is ready for change are also important mechanisms that drive the implementation process forward. Less is known about the Sustainment phase, as the factors that facilitate or interfere with the maintenance of an innovation are only beginning to receive empirical attention (e.g., Chambers, Glasgow & Stange, 2013; Wiltsey-Stirman et al., 2012). However, strong leadership, policies that support sustainment, and continued funding appear necessary to sustain EBPs. Further, it is important that the staff trained in an EBP deliver it on a regular basis and with fidelity, which requires ongoing monitoring and support, either from external (e.g., intervention developers) or local experts.

Application

Several factors at each of the EPIS stages, as well as the attention to inner and outer contextual factors, have been identified as critical to implementation success. A study examining implementation of SafeCare, an evidence-based home visitation program targeting prevention and/or risk reduction for child maltreatment, found that an inner contextual factor, transformational leadership—or the extent to which leaders encourage staff adoption of similar goals and behaviors—was strongly related to innovation climate, which in turn, was associated with more positive provider attitudes toward EBPs (another inner contextual factor) (Aarons & Sommerfeld, 2012). Fidelity monitoring has been recognized as particularly beneficial for the sustainability of EBPs, as one study found that SafeCare providers who were monitored for fidelity exhibited less turnover than those not monitored (Aarons, Sommerfield, Hecht, Silovsky, & Chaffin, 2009). Accordingly, it has become increasingly evident that recruitment and retention of a trained and knowledgeable staff help to ensure sustained and effective service delivery (Aarons et al., 2011). An ongoing project is comparing SafeCare implementation as usual to the EPIS implementation framework on various outcomes such as fidelity, provider engagement, and client satisfaction (Aarons et al., 2012). Using a mixed methods design, this project aims to identify organizational and provider factors that influence outcomes in adaptation and implementation as well as examine the feasibility, acceptability, and utility of the EPIS framework in a child welfare setting.

Learning Collaborative

The Learning Collaborative, adapted from the Institute for Healthcare Improvement’s (IHI, 2003) Break-Through Series, is an implementation framework comprised of multiple methods to support change across different levels of an agency and thereby spread EBPs across health care settings. The Learning Collaborative includes a combination of initial exploration and preparatory activities (e.g., readiness assessment, agency selection), in-person training (i.e., learning sessions), coaching/consultation calls and an emphasis on quality improvement strategies (Langley, Moen, Nolan, Nolan, & Norman, 2009; Markiewicz, Ebert, Ling, Amaya-Jackson & Kisiel, 2006; McHugh & Barlow, 2010; Nadeem et al., 2014), such as Plan-Do-Study-Act (PDSA) cycles, to promote sustained change. Agency teams and the corresponding training tracks include front-line providers, supervisors and senior leaders to facilitate implementation and sustainability (Markiewicz et al., 2006).

Application

In 2000, SAMHSA initiated funding for the National Child Traumatic Stress Network (NCTSN) to increase availability and accessibility of EBPs for youth affected by abuse or trauma (Pynoos et al., 2008). To date, the NCTSN has extensively promulgated the Learning Collaborative as one of the primary implementation frameworks for a variety of trauma-focused EBPs (Cohen & Mannarino, 2008), and studies are just now beginning to examine the effectiveness of this methodology (Ebert, Amaya-Jackson, Markiewicz, Kisiel, & Fairbank, 2012a; Ebert et al., 2012b; Goldman-Fraser et al., 2014). In an observational study examining the use and utility of Learning Collaborative methodology in the implementation of TF-CBT, both clinicians and supervisors perceived the use of the methodology, such as learning sessions, collaborative intranet, and PDSA cycles, as useful for implementation efforts (Ebert et al., 2012b). Participants reported that the most helpful features of the Learning Collaborative were the opportunities to collaborate with teams outside their agency and to work with individuals in different roles and varying levels within their own agencies. Clinicians also identified the clinical training, TF-CBT consultations, and learning sessions as particularly helpful.

Active Implementation Framework (AIF)

The Active Implementation Framework (AIF) consists of four stages: Exploration, Installation, Initial Implementation, and Full Implementation, with sustainability embedded within each stage rather than viewed as an independent, final stage (Metz et al., 2014; Metz & Bartley, 2012). A key feature of AIF is its use of implementation drivers, which are utilized to support change at the practice, organizational, and systems levels (Metz & Bartley, 2012). AIF delineates three types of implementation drivers: competency drivers, or mechanisms needed to support and maintain practitioners’ abilities to implement a program; organization drivers, or mechanisms that help develop and support an organization to make its environment amenable to the implementation and sustainment of an innovation; and leadership drivers, or mechanisms that support organization leaders in addressing challenges that arise during implementation. In the Exploration stage, the goal is to examine the degree to which an EBP meets the community or organization’s needs and whether implementation is practical. Once the adoption decision has been made, during the Installation stage, some organizational changes must precede program adoption, such as developing referral pathways, ensuring financial resources, purchasing equipment, and establishing practitioner competence. The Initial Implementation stage involves putting the new program into practice and using data to assess implementation and address barriers so problems can be identified and resolved. At the Full Implementation stage, the program is part of the organization’s regular practice and the scaffolds to support the EBP are in place. Sustainability is a consideration throughout, such that financial and programmatic sustainability planning and activities (e.g., ensuring adequate funding streams, effective training, coaching, and performance assessments) are an active component during the course of implementation.

Application

The AIF has been successfully applied to the implementation of the Child Well Being Project in the North Carolina child welfare system, which targets permanently placed children and their families (Metz et al., 2014). In this application, multiple implementation drivers were installed to facilitate implementation of three EBPs: Parent-Child Interaction Therapy (Eyberg & Boggs, 1998), the Strengthening Families Program (Kumpfer, Molgaard, & Spoth, 1996), and Success Coach, a home visiting service with enhanced case management. Implementation teams were developed at the leadership, management, and practice levels to support and continually evaluate implementation efforts. When the implementation team found the fidelity for Success Coach to be low at 3 months, as measured by the level of agreement between family assessments and goal planning, other implementation drivers were evaluated. Thus, based on baseline data, it was determined that improved coaching, administrative support, and increased data use were needed to enhance fidelity as well as support the implementation infrastructure (Metz et al., 2014). This resulted in an increase in fidelity from 18% to 83% at 12 and 24 months, respectively.

Consolidation and Core Component Frameworks

RE-AIM

In contrast to other frameworks discussed below, RE-AIM (Glasgow, Vogt, & Boles, 1999) emphasizes public health impact, rather than the process of implementation, according to five dimensions: Reach, Efficacy, Adoption, Implementation, and Maintenance. Accordingly, an intervention’s impact is determined by its joint influence on these five dimensions. The first dimension is Reach, a patient-level measure of program participation, which is used to evaluate the extent to which a program reaches those it is intended to benefit (e.g., families involved with child welfare services). Second, Efficacy refers to the impact of an intervention on relevant physiological, behavioral, quality of life, and participant satisfaction outcomes. Adoption refers to the quantity and representativeness of organizations and staff that adopt and offer a given innovation. The Implementation factor is measured at both the individual- and program-levels, and indicates the degree to which the program is delivered as intended. At the individual level, there are measures of participant adherence to the program’s protocol to assess implementation, while at the program level, the interest is in the extent to which staff deliver the intervention in real world settings as intended. Finally, Maintenance, defined as the organization’s ongoing use of an intervention, is similar to the concept of ‘sustainability’ discussed in other frameworks. Maintenance is also measured at the individual- and agency-levels, as indicated by the extent to which individual behavior change is sustained and the extent to which the intervention becomes a continued presence in the agency’s repertoire.

Application

The RE-AIM framework has been applied to a variety of health and mental health problems, including the evaluation of an implementation of a school-based, cognitive-behavioral program targeting the prevention of adolescent depressive symptoms (Garmy, Jakobsson, Carlsson, Berg, & Clausson, 2015). EBPs delivered in the school setting may be especially important for child welfare youth, as this may lead to easier access for youth whose caretakers or foster parents may have excessive barriers to getting the youth to services in community settings. RE-AIM domains were used to measure program impact and implementation success. Very few students declined to participate in the program, indicating the program’s success in reaching a majority of the targeted population. Moreover, youth-reported depressive symptoms decreased from baseline to a one-year follow-up, suggesting that the intervention was effective and that the outcomes were maintained.

Practical, Robust Implementation and Sustainability Model (PRISM)

PRISM (Feldstein & Glasgow, 2008) synthesizes key elements from several other theoretical models, including the diffusion of innovations (Rogers, 1995), chronic care model (Wagner, 2015), model of improvement (Langley et al., 2009), and RE-AIM (Glasgow, Vogt, & Boles, 1999). The model identifies and addresses common elements of interventions, the environment, the organizational infrastructure, and intervention recipients (the organization and client) that interact to influence reach or penetration of the EBP to the targeted population, efficacy of the EBP, level of adoption by providers, and factors related to full EBP implementation and maintenance (i.e, RE-AIM domains). PRISM also emphasizes the perspective and characteristics of three levels of organizational workers that may impede or facilitate implementation: senior leaders, middle managers, and frontline staff (i.e., practitioners). For instance, while senior leaders are needed to champion an innovation, it is also necessary for middle managers and staff to buy-in to program adoption.

PRISM encourages exploration of several features of the client population (e.g., child welfare families, families of children with mental health diagnoses) to determine whether a program will meet idiosyncratic needs and produce intended benefits. For example, these include the extent to which the program is client-centered, provides choices, addresses barriers to participation, minimizes client burden, and provides feedback to clients. Further, characteristics of the external environment, such as reimbursement issues and community resources, are strong determinants of implementation and maintenance success, and require careful examination. An organization’s infrastructure for implementation and sustainability is also considered to be crucial to successful implementation. For example, assessing organizational readiness, potential barriers to program implementation, and program burden (i.e., cost and difficulty) are recommended to establish feasibility of implementation and strategies to counter organizational deficiencies. Based on the quality improvement literature, the PRISM framework suggests dedication of a set implementation team, regular measurements of performance, development of procedures adapted to the local level, and the provision of ongoing implementation training and support. Finally, PRISM emphasizes tools to measure the implementation process and outcomes early and often in order to make any needed changes.

Application

Commonly used in health care settings, the PRISM framework has also been applied to the implementation of a program (i.e., the 4 Rs and 2 Ss for Strengthening Families Program) that targeted urban, low-income children with disruptive behavior disorders and their families, families who have demographic and child risk factors for CM perpetration (Gopalan et al., 2014). Within the PRISM framework, this project sought to implement a program that was adaptable, embedded within the existing work flow, could meet local clinic needs, minimize staff burden, discontinue if necessary, and permit staff to see results. To evaluate implementation success, program facilitators and clinic directors were surveyed about their perspectives on 1) collaboration with other program facilitators, 2) experiences working with the 4 Rs Program, and 3) any challenges encountered. While qualitative results revealed that, in general, facilitators and directors reported a positive experience with the 4Rs Program implementation, there were some concerns related to agency readiness and workforce capacity. Learning Collaboratives and Quality Improvement Teams were established to address agency-level barriers to implementation. Moreover, several facilitators noted the need for increased training and concerns about the time consumed by program facilitation. Based on these recommendations, the adapted program was simplified to reduce barriers, decrease the number of materials, and better fit into clinic structures. Currently, New York State is engaged in a statewide implementation of the 4 R’s program with ongoing evaluation to assess the impact of these initial changes on outcomes and sustainability.

The Consolidated Framework for Implementation Research (CFIR)

The CFIR fuses common constructs from various existing implementation theories into a single, overarching framework that can guide formative evaluations as well as the interpretation of findings, regardless of framework (Damschroder et al., 2009). The CFIR consists of five main domains that mirror core components of many of the aforementioned frameworks: EBP characteristics, outer and inner settings, individual characteristics, and the overall implementation process. The characteristics of the EBP refer to both its essential components as well as those elements that are adaptable to the implementation site. The outer setting consists of the social, economic, and political landscape in which the organization exists, while the inner setting consists of the structural, political, and cultural context in which implementation will occur. The fourth domain refers to the individuals who will deliver the EBP, all of whom possess characteristics that will influence the implementation process (e.g., attitudes about the intervention). Finally, in order for implementation to be achieved, there needs to be a process in which individuals and organizations actively engage in activities that promote successful implementation, such as determining (and executing) a plan of action, engaging implementation champions, and obtaining feedback about the success of implementation efforts.

Application

The CFIR domains are currently being applied in a systematic review examining D/I strategies to improve child and adolescent mental health care with the purpose of identifying potential moderators of strategy effectiveness (AHRQ, 2014). Specifically, this research aims to identify the client and contextual variables (e.g., characteristics of the child, provider, organization, and EBP) that impact the effectiveness of implementing mental health services for children and adolescents.

Systems and Organizational Support Frameworks

The Interactive Systems Framework (ISF)

The ISF (Wandersman, Duffy, Flaspohler, Noonan, Lubell et al., 2008) focuses on the infrastructure and systems needed to facilitate D/I. ISF encompasses three interrelated systems of activities: the Prevention Synthesis and Translation System, the Prevention Support System, and the Prevention Delivery System. The goal of the Prevention Synthesis and Translation System is to condense empirically generated information about an EBP so that it is accessible and understandable to consumers (e.g., agencies and practitioners) and to prepare that information for utilization in practice. Providing information about an EBP alone does not seem sufficient to change practice. Thus, the Prevention Support System is a key component of ISF and consists of two main support functions, innovation-specific capacity building and general capacity building within an organization. Innovation-specific capacity-building refers to support related to implementing a specific EBP and can include distributing information before the decision to adopt, providing training before EBP implementation (with subsequent supervision and fidelity monitoring), and providing technical assistance throughout implementation. The goal of general capacity building is to improve the infrastructure, skills, and motivation of an organization, without regard to a specific EBP. These activities can include grant writing, developing partnerships, and enhancing leadership skills.

The final system, the Prevention Delivery System, includes activities necessary to implement an EBP within clinical settings. This system emphasizes general capacities, such as maintenance of a functional organization, as well as innovation-specific capacity activities, such as making decisions about which EBPs to implement, and taking action to implement and maintain a given EBP. The interaction of these three systems enhances the likelihood of successful implementation. Information about an EBP, gathered in the Prevention Synthesis and Translation System, cannot be applied in clinical settings by the Prevention Delivery System, without the necessary organizational infrastructure provided by the Prevention Support System.

Application

The importance of the Prevention Support System to successful implementation has been highlighted in recent policy and practice efforts targeting the prevention of child maltreatment. The government prioritization of funding for EBPs stimulated a need for national, state, and local entities to execute the activities needed to support the implementation of evidence-based prevention efforts, specifically (Brodowski et al., 2013). For example, the Office on Child Abuse and Neglect provides fiscal support to grantees to enhance their capacity to implement prevention-focused community-based EBPs. Similarly, the Children’s Bureau provides funding to various national resource centers that support local prevention efforts by offering information, training, technical assistance, and expertise for the implementation of programs that target child maltreatment. These federal capacity-building efforts and activities within the Prevention Support System have helped facilitate the translation of information and knowledge about child maltreatment prevention to organizations around the country that can implement that knowledge through evidence-based prevention efforts in real world practice settings.

ARC

ARC (Availability, Responsiveness, and Continuity) (Glisson & Schoenwald, 2005) employs implementation strategies at both the organization and inter-organizational domain levels. Specific intervention strategies aim to enhance organizational effectiveness by targeting the social context (e.g., organizational culture and climate) in which mental health services will be offered. Inter-organizational domain development strategies cultivate relationships among service providers, organizations, and key stakeholders with the goal of identifying problems and supporting interventions that address those problems. Another component of ARC is change agents, who act at multiple levels to establish a group of community stakeholders, assist in the implementation of EBPs, and develop relationships with community leaders.

Application

ARC strategies have been found to enhance organizational climate (e.g., promote low conflict, increased cooperation) and reduce case manager turnover rates when applied to child welfare and juvenile justice systems (Glisson, Dukes, & Green, 2006). Social context factors (e.g., organizational culture) addressed by ARC are powerful predictors of children’s service outcomes and quality of services (Glisson & Hemmelgarn, 1998). Using a randomized trial, Glisson and colleagues (Glisson et al., 2010) examined the effect of ARC and Multisystemic Therapy (MST) (Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009)—the gold standard EBP for reducing delinquency recidivism—on delinquent youths’ outcomes. ARC offered organizations tools to recognize and address barriers to service delivery, presented components of effective service systems, and targeted provider behaviors and attitudes that can undermine implementation efforts. Results demonstrated the effectiveness of ARC in combination with MST in reducing youth problem behaviors to non-clinical levels. Additionally, the combination of ARC and MST significantly reduced the amount of out-of-home placements.

Planning Framework

PRECEDE-PROCEED Model

The PRECEDE-PROCEED Model (PPM) has been widely used to apply behavioral change theories in public health settings (e.g., Howat, Jones, Hall, Cross & Stevenson, 1997), including child maltreatment prevention (Gielen, McDonald, Gary, & Bone, 2008; Chasan-Taber & Tabachnick, 1999). Instead of providing specific steps or stages for implementation, the PPM guides the application of theories to interventions and is best conceptualized as a planning model (Gielen et al., 2008). The PRECEDE (Predisposing, Reinforcing, and Enabling Constructs in Educational/Environmental Diagnosis) framework emphasizes that diagnosis should come before planning interventions, while the PROCEED (Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development) framework focuses on environmental determinants of behavior. The PPM consists of four planning phases, one implementation phase, and three evaluation phases, and emphasizes participation of the intervention’s target population in the planning process. During the first phase, a social assessment, participatory planning, and situation analysis take place; ultimately, the community in which the intervention will be implemented is assessed for fit, and stakeholders engage in focus groups to determine community needs and readiness for change. The second phase consists of epidemiological, behavioral, and environmental assessments, in which behavior or health problems are identified, as well as the behavioral and environmental factors that contribute to the problems’ sustainment. In the third phase, implementation planners identify predisposing, reinforcing, and enabling factors that impact the change process. The fourth phase entails administrative and policy assessment as well as intervention alignment, in which implementation planners align the intervention with those influential factors identified in previous phases. The final four phases involve EBP implementation and evaluation.

Application

In an evaluation of Stop-It-Now, an evidence-based program that uses social marketing strategies to reduce risk for child sexual abuse (Rice, Hafner, & Pollard, 2010), telephone interviews and focus groups were utilized to determine attitudes, awareness, and knowledge of sexual abuse to determine predisposing (e.g., perceived access to services), enabling (e.g., mandatory reporting), and reinforcing (e.g., support groups) factors (Chasan-Taber & Tabachnick, 1999). These factors were then translated into interventions to prevent child sexual abuse, including media and outreach campaigns. PPM has also been used to develop an intervention targeting child injury occurring in the home by identifying behavioral (e.g., use of stair gates) and environmental (e.g., quality of housing) factors that contribute to child injury (Gielen et al., 2008). In the resulting intervention called SAFE Home, participating parents received counseling on injury prevention, home visits to address childproofing practices, and access to a safety resource center to increase the availability of home safety materials and education.

Discussion

The purpose of this paper was to identify frameworks with existing published applications relevant to the implementation of EBPs in child welfare and child mental health settings. As noted, frameworks can provide guidance in some or all phases of implementation, starting with the decision about EBP selection, to the best ways to train and sustain delivery of an EBP with fidelity, to the importance of fostering strong leadership to provide a supportive agency infrastructure and broader system change that can sustain practice after active implementation efforts have concluded. Framework selection appears to be especially critical as many implementation efforts fail without a guide for the process (Aarons et al., 2011).

Common Elements and Distinguishing Features

While this is an emerging area of research, there were ten frameworks identified with published work describing applications in child welfare or child mental health. Table 1 summarizes the ten reviewed frameworks categorized by their common elements (i.e., stages, domains, inner/outer setting, implementation teams, sustainability) and distinguishing features. As noted, half of the frameworks refer to the stages of implementation and core implementation components, identified initially by Fixsen and colleagues. However, there is variation in the emphasis given to these stages and/or components across frameworks, with some attending to both implementation and sustainability at the onset (AIF), rather than waiting to address sustainability after active implementation has ended. Further, the majority of frameworks highlight the importance of planning and preparation for implementation. A second common theme across several frameworks is the importance of attending to both inner and outer contextual factors demonstrated, in the empirical literature, as critical for implementation success (Stages of Implementation, EPIS, PRISM, CFIR, ISF, ARC, PRECEDE-PROCEED). Attention to context is likely to be an important consideration for policy and decision makers, rendering those frameworks that highlight these as particularly useful (e.g., CFIR, EPIS, Stages of Implementation, PRISM).

In contrast, the frameworks differ in their attempts to integrate or highlight core components across multiple theoretical models (i.e., CFIR, PRISM, RE-AIM) and their attention to strategies for EBP training and consultation specifically (i.e., Stages of Implementation, EPIS, Learning Collaborative, PRISM, ISF). Another notable nuance among the frameworks is the phase on which each primarily focused. While most focus on active implementation activities (i.e., Stages of Implementation, EPIS, AIF, ISF, Learning Collaborative, PRISM, ISF, ARC), a few place greater emphasis on planning and preparation (i.e., PRECEDE-PROCEED) and evaluation (i.e., CFIR, RE-AIM). These differences highlight the need to consider the chosen EBP (e.g., its complexity) and its fit with the context in which it will be implemented, because these factors will influence the extent to which efforts need to emphasize initial planning vs. more extensive training and active implementation activities. The different emphases also point to the potential benefits of blending frameworks that emphasize planning and evaluation (e.g., PRECEDE-PROCEED, RE-AIM) with ones that focus on the interaction of inner and outer contextual factors at various stages of implementation (e.g., EPIS, CFIR). These common and differentiating elements as captured in Table 1, can contribute to final decisions regarding framework selection in implementation planning and beyond.

Framework Considerations for Implementation Policy Makers in Child Welfare and Child Mental Health Settings

In making policy and programmatic decisions for children and families involved with child welfare and mental health settings, there are numerous considerations when selecting a framework for implementation. Framework selection may be contingent on the type or focus of the problem(s) being targeted by the EBP as well as the scope of the intended target. Wide-spread state level implementation, for instance, which is often the focus for child welfare recidivism prevention efforts (i.e., Whitaker et al., 2012), necessitates a carefully reasoned planning process, such as described in the PRECEDE-PROCEED approach, because of the multiple agencies, providers, and children and families targeted. In addition, frameworks that include planning stages allow for trouble-shooting and strategy development to address the potentially variable reimbursement and organizational infrastructures that exist within a system in public-private partnerships. Further, state-level decision makers may benefit from consideration of the frameworks that explicitly consider the inner and outer contexts, such as EPIS and CFIR, as implementation success in large systems is contingent on the interactions between effective policy, appropriate service reimbursement infrastructure, referrals, and practitioner/agency buy-in. Lastly, frameworks, such as EPIS and Stages of Implementation, that provide a structured and potentially low cost approach to allow for widespread dissemination of EBP training and field support throughout the service systems, can help to ensure longevity.

Along with these higher level, contextual factors, there are several, more nuanced considerations to take into account for framework selection. Frameworks that highlight elements relevant to organizational culture, for example, may be especially useful for policy makers interested in implementing an EBP targeting families involved in multiple, overlapping service systems which may have different, and even competing priorities and responsibilities. Frameworks, such as EPIS and ARC, that encourage examination of the culture of varied agencies would enable policy makers to be more attuned to the potential barriers faced by these multiple providers, which undoubtedly contribute to the challenges experienced by the families being served. ISF also encourages agency-level policy makers to focus on identifying strategies to address agency-specific needs in the implementation and to take into account how the installation of a new program will advance the skills and job satisfaction of service providers.

Framework selection for decision makers can also vary according to the service system. Child welfare and child mental health services, while often serving similar families, function very differently in terms of private-public partnerships, funding and billing for services, the selection of interventions (family risk based versus diagnosis based), consumer focus (service targeting parent vs. child), and desired outcome (e.g., child well-being and placement permanency vs. improved mental health), all of which would benefit from frameworks highlighting these inner and outer contextual factors. Additionally, the targeted implementation outcome influences the framework selection. For instance, is the focus small, primarily targeting sustainability within a given agency? Or, is it multi-tiered with a focus on both scale-up and sustainability?

Collectively, the majority of frameworks provide a phased or stage approach that decision makers should not only consider, but also fully explore in the implementation of an EBP, as these frameworks will guide a more systematic implementation effort that can increase implementation success (Aarons et al, 2012; Glisson et al., 2010; Brodowski et al., 2013; Gopalan et al., 2014; Metz et al., 2014). For example, in pre-adoption phases, critical considerations include identifying (1) the target population (e.g., maltreated or at-risk children and families), (2) which EBP will best meet the needs, challenges, and problems for the target population (e.g., TF-CBT, SafeCare), and (3) whether there are referral and funding mechanisms in place to support the services. Once an EBP is selected, it is important to explore its fit within a system or agency, identify those players critical to implementation success, engage in capacity building that allows for planning and preparation for implementation drivers, and establish a plan for preparing and training the practitioners who will be the EBP implementers. During active implementation and sustainability, the focus shifts more to how well the EBP is meeting the needs of the consumers (i.e., are consumers being referred and are they enrolling and completing services?); how engaged practitioners are in the EBP, practitioners’ fidelity and competency skills; effectiveness of policy and reimbursement decisions; and planning for staff turnover.

One important theme that has emerged in the sparse literature examining applications of frameworks to EBP implementation in child welfare and child mental health is the critical importance of data collection in determining successes and needed changes in EBP implementation (Bickman, 2008). While some consider data collection to be important primarily for research purposes, it is increasingly evident that program evaluation and data feedback from practitioners and implementation teams can be used to facilitate adaptations that might inform next steps in the implementation process. Importantly, the frameworks can help define the approach policy makers utilize for establishing a feedback loop system. For example, the studies discussed above with the PRISM and AIF frameworks illustrated how data and feedback can assist decision makers in adapting plans and developing strategies to address challenges or problems that arise throughout the implementation process. The Learning Collaborative emphasizes the importance of ongoing metrics collection to monitor use and perceived competence in the EBP; this information is continuously fed back to participants as a further mechanism for supporting sustained delivery of the EBP after active implementation has ended.

Another critical consideration for decision makers is the importance and cost associated with sustainability. For example, the ISF framework does not specifically address sustainability as a core implementation component. This could be less helpful to a legislative policy maker who must be attuned to longer-term fiscal considerations. In this instance, AIF, RE-AIM, or PRISM, which address sustainability upfront, would emphasize the importance of conducting an initial assessment to address how implementation could be affected by shifts in money allocation—an issue which would be paramount in the decision making process. Furthermore, the historical success and failure rates related to fiscal management, for both the intervention and implementation site, should be considered. Consequently, sustainability is key in this respect. The AIF may be best suited for policy makers in this arena as it highlights a focus on sustainability throughout all phases of implementation.

The ability of an EBP implementation to be sustained across time, administrations, and through fiscal crises is always important. However, there are instances when this may not be the most integral component of a framework, such as when time-limited block grants are awarded to address a certain issue. For example, child maltreatment has been shown to increase in times of natural disasters (Self-Brown, Anderson, Edwards, & McGill, 2013). Areas most affected may be given a grant to address this need, with the awareness that funding is unlikely to extend past a specified period. In these cases, frameworks focusing on communal needs/assessment, such as PRECEDE-PROCEED, would likely better serve the policy maker’s purpose.

A final determining factor in framework selection is the complexity of the EBP. For example, when introducing an EBP such as SafeCare in a child welfare setting, the choice of framework guiding implementation will be strongly influenced by whether the practice will need to be adapted for that particular service context, whether there is funding for a home-based prevention intervention, and whether there is sufficient buy-in from the front-line staff to deliver this intervention. For TF-CBT, a complex, multi-component intervention that includes caregiver involvement, as well as gradual exposure to trauma reminders, the Learning Collaborative framework may be especially useful because of its emphasis on intensive training, regular consultation, monitoring, and fostering of collaboration among staff at multiple levels to assure that TF-CBT continues to be delivered as intended. Along similar lines, a prevention intervention, such as SafeCare, which requires intensive individual training and 1:1 coaching, as well as significant agency and system-level buy-in, might benefit from a framework emphasizing preparatory and planning phases, attention to outer contextual factors and sustainability planning (e.g., EPIS).

One noted limitation of existing frameworks is the sparse attention to implementation efforts that target multidisciplinary providers across multiple service settings. For example, while the Learning Collaborative emphasizes training of mental health providers in an EBP, its focus is on inclusion of teams from single agencies. While this does increase the supply of trained clinicians, it has limited impact on the overall child maltreatment service delivery system because it does not include a method to increase awareness and demand for the EBP, nor does it work specifically to increase the capacity of communities to deliver and sustain these services to children and their families. One innovative approach to address this is the Community Based Learning Collaborative (CBLC; Saunders & Hanson, 2014), which extends the Learning Collaborative model by including clinical and broker (i.e., those who identify, screen, refer, and monitor youth and families) professionals from all levels (front-line provider, supervisor, senior leader), and from multiple organizations within a targeted community. The purpose of the CBLC, based upon a social economic model of supply and demand, is to create demand for an EBP from brokers and link them to the clinicians that provide the supply. Thus, the overall goal of a CBLC is to build community capacity for the implementation and sustainability of EBPs for children and their families by providing multiple opportunities to build and sustain collaboration amongst the multiple professionals involved in coordinating and providing services for children and their families. There are several ongoing projects examining the CBLC approach, including Project BEST (www.musc.edu/projectbest) in South Carolina, funded by the Duke Endowment; PATS (www.musc.edu/PATS) funded by SAMSHA; and a recently awarded NIMH grant (PI Hanson) that is examining which specific strategies of the CBLC appear most critical to promote collaboration and sustained use of trauma-focused EBPs and determine the feasibility of the CBLC implementation model.

Conclusions

The intended purpose of this synthesis was to facilitate more effective decision making for those involved directly in the implementation decision process, specifically in child welfare and child mental health service settings. Given the diversity of service settings that target children impacted by maltreatment, any decisions regarding EBP implementation are complex. Extensive efforts are being made in the research and practice realms to help inform best practices for children and their families. The selection of a framework, based on the recommendations provided in this paper, can guide the implementation process, foster more targeted decision-making, more thoughtful and informative evaluation of implementation success, and ultimately, improved outcomes for youth involved in child welfare and child mental health systems as a result of child maltreatment.