Dissemination and Implementation of Evidence-Based Practices for Child and Adolescent Mental Health: A Systematic Review

Abstract

Objective

While there has been a dramatic increase in the number of evidence-based practices (EBPs) to improve child and adolescent mental health, the poor uptake of these EBPs has led to investigations of factors related to their successful dissemination and implementation. The purpose of this systematic review was to identify key findings from empirical studies examining the dissemination and implementation of EBPs for child and adolescent mental health.

Method

Out of 14,247 citations initially identified, 73 articles drawn from 44 studies met inclusion criteria. The articles were classified by implementation phase (exploration, preparation, implementation, and sustainment) and specific implementation factors examined. These factors were divided into outer (i.e., system level) and inner (i.e., organizational level) contexts.

Results

Few studies utilized true experimental designs; most were observational. Of the many inner context factors that were examined in these studies (e.g., provider characteristics, organizational resources, leadership), fidelity monitoring and supervision had the strongest empirical evidence. While the focus of fewer studies, implementation interventions focused on improving organizational climate and culture were associated with better intervention sustainment as well as child and adolescent outcomes. Outer contextual factors such as training and use of specific technologies to support intervention use were also important in facilitating the implementation process.

Conclusions

The further development and testing of dissemination and implementation strategies is needed in order to more efficiently move EBPs into usual care.

INTRODUCTION

While the last several decades have been marked by the development of a number of important preventive and clinical evidence-based practices (EBPs) for mental health problems among children and adolescents, the science of disseminating and implementing these practices into community and clinical settings has received considerably less attention.¹ Indeed, the poor uptake of EBPs in usual care settings remains one of the major barriers to providing safe, effective, and efficient mental health care.^1,2 The improvement of this process is the focus of dissemination and implementation research.

For the purposes of this review we define dissemination as the “targeted distribution of information and intervention materials to a specific public health or clinical practice audience. The intent is to spread knowledge and the associated evidence-based interventions.”³ We define implementation as “the use of strategies to introduce or change evidence-based health interventions within specific settings.”^3,4 Both dissemination and implementation are considered “active” strategies, standing in contrast to processes described as “diffusion,” which refer to the natural uptake of innovations.^5,6 Implementation research falls under the broad rubric of “translational science” and is considered “T3” where T1 is the translation of basic science discovery to a clinical intervention (e.g., translating a basic biological process into a medication, or a behavioral process to a psychosocial intervention), T2 expands basic findings to clinical practice, and T3 is the dissemination and/or implementation of a new intervention.⁷

We define EBPs as those health interventions that are supported by rigorous scientific research, allow for clinical judgment and expertise in their application, and provide for consumer choice, preference, and culture.^8,9 While the extent to which EBPs are being provided in usual care for children and adolescents is not clear, it is clear that there are critical gaps in the quality and effectiveness of mental health care currently being delivered to children.¹ Indeed, the ability to adopt, implement, and sustain EBPs is becoming increasingly important for mental health, schools, and other human service organizations and providers¹⁰ as well as for integration into primary health care settings.

Implementation research is informed by a range of theories including seminal work on the diffusion of innovation in agriculture.^6,11 This work generalized to diffusion of innovations in general and in social service settings in particular.⁵ In the 1970s work being developed in the United Kingdom addressed understanding intervention effectiveness in health care settings through systematic reviews of scientific literature¹² and systematically applying relevant research to practice¹³ leading to clearinghouses to support dissemination of information regarding EBPs.¹⁴ In the 1980–1990s there was developing interest in the study of the implementation of innovations in business^15,16 and an increasing impetus for quality improvement in health care that culminated in the Health Care Quality Improvement Act (1986)¹⁷ and later the Institute of Medicine’s Crossing the Quality Chasm report (2001).¹⁸ Specific calls from the NIH to support implementation research began in 1999,³ Centers for Disease Control funding began in 2009.¹⁹ National Institutes of Health (NIH)–sponsored conferences²⁰ and training programs²¹ focused on dissemination and implementation research have also helped to advance the field.

Moving from the development of interventions to implementation in usual care is often a lengthy process and that time lag compromises the well-being of children with mental health needs.²² A number of mechanisms have been developed to accelerate this process through support for research and practice in implementation science. For example, NIH supports active research and training programs focused squarely on the dissemination and implementation of EBPs.³ The W.T. Grant Foundation is funding studies to better understand how research evidence is accessed, shared, and interpreted by policymakers and practitioners,^4,23 and the Centers for Disease Control and Prevention also funds studies to examine translation of EBPs into usual care.²⁴

There are a number of published dissemination and implementation frameworks; most approach implementation as a complex, multiphasic process that involves multiple stakeholders in service systems, organizations, and practices.^11,25,26 One such framework developed specifically for public mental health and social service settings is the EPIS model. It divides the dissemination and implementation process into the following 4 phases: Exploration (consideration of new approaches to providing services), Preparation (planning for providing a new service), Implementation (provision of this new service), and Sustainment (maintaining this new service over time; EPIS).¹⁰ The EPIS model also emphasizes the importance of contextual factors, both inside the unit providing services (i.e., service organization, individuals providers) as well as those in the larger environment in which the service unit operates (e.g., policy and funding, relationships with intervention developers and technical assistance providers, certification and regulatory environment) Figure 1 shows the multiple phases and levels of the EPIS framework. Note that some factors (e.g., fidelity, provider attitudes, interorganizational networks) are relevant to multiple EPIS phases. In order to illuminate this complexity, we provide the following hypothetical example:

Figure 1.

Exploration, preparation, implementation, sustainment (EPIS) multilevel implementation framework: factors impacting evidence-based treatment (EBT) implementation in child and adolescent mental health. Note: This figure depicts some examples of factors in the outer and inner contexts to be considered in each phase of the EPIS framework phases found in this study and in the original article. From Aarons et al. Advancing a conceptual model of evidence-based practice implementation in public service sectors. 2011;28(1):4–23.¹⁰ Reprinted with permission from Springer.

In the exploration phase, a service system, organization, (e.g., hospital, clinic, community-based provider, etc.) or an individual considers what factors might be important in regard to implementing a practice. For a new medication, these might include regulatory and reimbursement issues (e.g., Food and Drug Administration [FDA] approval, health plan formularies) and the need for training and support for physicians and pharmacists in appropriate prescribing practices and potential drug interactions. In the preparation phase, changes in formularies would be made and electronic medical records would need to be amended to allow for documenting indications and prescribing the new medication. Plans would need to be made for physician/pharmacist training including scheduling, procuring space, and follow-up coaching and support, if needed. In the implementation phase training begins along with assuring that the medication is now available in formularies and for patients to obtain from pharmacies. In the sustainment phase, ongoing monitoring of appropriate prescribing practices, patient adherence, adverse events, and outcomes would be utilized to understand and increase the likelihood of positive outcomes. While this example is oversimplified, it illustrates that there are a number of issues to be considered in order to facilitate effective implementation of an EBP in each EPIS phase.

In addition to EPIS, we considered a number of frameworks and approaches for framing the results of this review. For example, Damschroder et al.²⁵ identified multiple models, theories, and frameworks based on previous synthesis of implementation literature as well as individual frameworks. They synthesized 19 of these to develop the Consolidated Framework for Implementation Research (CFIR). Powell et al.¹¹ examined the characteristics of 65 implementation strategies and documented “key” processes including planning, educating, financing, restructuring, managing quality, and attending to the policy context. Meyers et al.²⁷ reviewed 25 implementation theories and developed the Quality Implementation Framework (QIF) that has 4 phases to address implementation processes. Each of these approaches has overlap with the EPIS framework. For example, like the CFIR the EPIS framework has a strong emphasis on structures and process in the outer policy and inner organizational contexts. Like the QIF, the EPIS framework has phases that help guide thinking and problem solving regarding implementation.

While there is variability in the focus and contexts considered in extant approaches, the EPIS framework integrates other theories of systemic and organizational change that help to inform EBP implementation efforts. In addition, for the purposes of this systematic review the EPIS framework was deemed appropriate because of its emphasis on public sector service sectors where a majority of children and adolescents receive mental health services. These service sectors include mental health, child welfare, substance abuse treatment, and primary care settings and commonly are strongly impacted by both outer policy and inner organizational context factors. In point of fact, most of the studies that met inclusion criteria for this review, took place in such public sector service settings.

The purpose of this systematic review is to examine the current state of the science regarding the implementation of EBPs for the prevention and treatment of mental health problems among children and adolescents in community, primary care, and specialty mental health settings.

METHOD

Databases and Search Methodology

We conducted a systematic literature search for all of the available empirical studies examining dissemination and implementation of evidence based practices in child and adolescent mental health using PubMed, Embase, PsycInfo, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINHL), and Cochrane Library. Search terms relating to 1) dissemination and implementation, 2) mental health including DSM-IV diagnoses, and 3) child and adolescent populations were used to search each database (see Supplement 1, available online, materials for Boolean search parameters). A second search strategy involved contacting model developers of prominent child/adolescent focused EBPs and examining publication lists on EBP websites. See Table S1, available online, for the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Systematic Review Checklist.

Study Selection

Included were English-language empirical journal articles that examined the dissemination and implementation of EBPs in child and adolescent mental health between 1991 and December 2011. Studies were excluded if they did not examine dissemination or implementation issues in child and adolescent mental health or if they contained no empirical data (e.g., case descriptions of implementations or a summary of “lessons learned” from such efforts). Screening of search results followed the PRISMA statement recommendations for reporting of the systematic review process.²⁸

A team of 4 investigators (1 child/adolescent psychiatrists, 2 child/adolescent psychologists, 1 general psychiatry resident) completed the review of titles, abstracts, and full-text articles. Investigators independently reviewed and selected articles resulting from the database, developer, and website searches. Once the results of these searches were merged and duplicates eliminated, the investigators employed an iterative calibration procedure prior to reviewing article abstracts. During the calibration procedure, ten percent of the abstracts were reviewed by the 4 investigators until consensus was made to either reject or include each article for full text review. After the calibration procedure was conducted, the team divided the remaining abstracts for independent investigator review. The resulting full text articles were then reviewed by at least 2 investigators for final inclusion. Finally, these full-text articles were rated on methodological rigor and relevance to EBP dissemination and implementation issues in child and family mental health. Scores ranged from 1 (high rigor/high relevance) to 3 (low rigor/low relevance). Studies that received a score of 3 were discussed by the team of coauthors and eliminated if consensus was reached that the study did not meet methodological rigor for inclusion or was not relevant enough to EBP dissemination and implementation in children’s mental health. Each study included in this qualitative review was examined and agreed upon for inclusion by all 4 coauthors.

Coding of Studies

Included studies were then coded by setting (e.g., mental health, school, child welfare), type of EBP (i.e. prevention, clinical), design (e.g., experimental, descriptive quantitative, qualitative, mixed methods), EPIS phase, and EPIS contextual level factors. Terms utilized by authors to describe their work were examined; however, articles were classified according to the EPIS model based on results and data that were presented. For example, a number of studies purported to address sustainment; however, in some cases the intervention was not carried out long enough (i.e., a minimum of 1 year) to meet criteria for sustainment.

RESULTS

A total of 14,247 citations were identified from database searches based on title alone, with an additional 336 records identified through EBP websites and contacting EBP model developers. These citations were further distilled down to 1,092 after reviewing the abstracts and eliminating duplicates. The abstract review resulted in 203 potential articles for full review; 104 were kept after full article review. Finally, these studies were assessed for methodological rigor and relevance to dissemination and implementation, resulting in 73 articles which met all study criteria for inclusion (see Figure 1 for PRISMA Diagram). Table 1 provides a synopsis of those papers utilizing randomized experimental methods. Table S2, available online, lists the key characteristics of all papers included in this review.

Table 1.

Summary of Randomized Studies of Evidence-Based Practice (EBP) Dissemination and Implementation in Youth Mental Health

Paper	EPIS Stage	Prevention/Treatment	Treatment/Practice Studied	Treatment/Practice Setting	Contextual Focus	Intervention	Unit(s) of Analysis	N
Aarons et al., 200956	Implementation	Prevention	SafeCare	Child welfare	Inner (organizational; provider; fidelity)	SafeCare vs. care as usual x fidelity monitoring vs. no monitoring	Multiple-organization/provider (organization dropped from final analyses)	153/25
	Summary: In multivariate model analyses, SafeCare plus fidelity monitoring had higher staff retention rates than SafeCare plus no monitoring. Older age, greater perceived job autonomy, and lower expressed intention to leave their job also positively correlated with higher staff retention rates.
Aarons et al., 200937	Implementation, Sustainment	Prevention	SafeCare	Child welfare	Inner (organizational; provider; fidelity; coaching)	SafeCare vs. care as usual x fidelity monitoring vs. no monitoring	Multiple-agencies/providers (analyses Were conducted to address clustering of providers in programs)	21/99
	Summary: Providers in programs implementing SafeCare had lower levels of emotional exhaustion. Significant predictors of emotional exhaustion included age (older=less emotional exhaustion) and caseload (higher=greater emotional exhaustion).
Barwick et al., 200958	Implementation	Treatment	CAFAS	Mental health clinics	Inner (organizational)	Use of a community of practice approach for improving adherence	Single - provider (randomization was done by organization, which was not addressed in the analyses)	(6)/37
	Summary: Providers working in organizations using a Community of Practice implementation model had higher rates CAFAS use; their knowledge regarding the CAFAS increased over time while those in the Practice as Usual programs decreased.
Beidas et al., 201251	Implementation	Treatment	CBT for anxiety	Mental health clinicians in private practice	Outer (training)	3 different training workshops (routine, computerized, and routine augmented with active learning principles)	Single - provider	115
	Summary: Based on the Adherence and Skill Checklist, the three workshops did not show significant differences in adherence, skill, and knowledge; however, participants preferred the in-person over the computerized training. Follow-up played a critical role with adherence and skill but was not a part of the randomized design.
Di Noia et al. 200329	Exploration	Prevention	Multiple	Schools, community agencies	Outer (Information transmission)	3 different dissemination strategies (pamphlet, CD, internet)	Single-provider	188
	Summary: Exploration impacts (accessibility of materials, self-efficacy to select/recommend an intervention, and behavioral intentions to encourage adoption of interventions) was stronger in the internet group, intermediate in the CD-ROM group, and weakest in the pamphlet group.
Epstein et al., 200752	Preparation, Implementation	Treatment	Stimulant Medication Dosage Determination Support	Primary care clinics	Inner (provider; fidelity; coaching)	A pre-packaged and specialist-supported titration trial to determine optimal methylphenidate dose in children with ADHD	Multiple-patient, provider, practice (note patient and provider levels dropped from key analyses)	145/52/12
	Summary: Availability of the titration service increased the use of titration trials but not the monitoring of ADHD symptoms, and it did not impact ADHD symptomatology overall. However, patients who actually received this service received higher stimulant doses and experienced greater improvement in symptoms.
Foster and Stiffman, 200934	Preparation, Sustainment	Treatment	Decision Support System	Social services	Inner (organizational; provider)	Personal digital assistant (PDA) vs. PDA plus desktop computer	Single-provider	28
	Summary: Availability of the titration service increased the use of titration trials but not the monitoring of ADHD symptoms, and it did not impact ADHD symptomatology overall. However, patients who actually received this service received higher stimulant doses and experienced greater improvement in symptoms.
Garner et al., 201162	Implementation	Treatment	Adolescent Community Reinforcement Approach (A-CRA) and Assertive Continuing Care (ACC).	Substance abuse treatment clinics	Inner (organizational)	Pay for performance (payments for receiving a rating of at least minimal competency on an audio session recording and for each adolescent receiving a targeted threshold of treatment)	Multiple - providers and agency	95/29
	Summary: Therapists in the pay for performance condition reported significantly greater intentions to achieve each of the two quality targets (to achieve monthly intervention competence and to deliver the intervention at a targeted threshold level) during the first 3 months of the experiment.
Glisson et al. 201060	Implementation	Treatment	MST and ARC (availability, responsiveness, and continuity)	Mental health clinics	Inner (organizational- culture/climate)	Organizational intervention (ARC) that addresses services barriers, training in principles of effective service systems, and facilitates the development of flexibility, openness, and engagement among providers	Multiple - youth, providers, counties (number of providers not specified, but this level was included in analyses)	615/?/14
	Summary: At 6-month follow-up youth total problem behavior in the MST plus ARC condition was at a nonclinical level and significantly lower than in other conditions. At 18-month follow-up youth in the MST plus ARC condition entered out-of-home placements at a significantly lower rate than in the other conditions.
Henggeler et al., 199753	Implementation	Treatment	MST	Mental health clinics	Inner (provider; fidelity)	Implementation of MST without fidelity measures compared to care as usual (and MST with fidelity measures drawing on prior publications)	Multiple - clinics/therapists/youth	2/4/155
	Summary: Modest outcomes were achieved when not considering fidelity to MST, especially when compared to prior MST trials that included fidelity monitoring. Parent and adolescent ratings of treatment adherence predicted low rates of re-arrest; and therapist ratings of treatment adherence and treatment engagement predicted decreased self-reported index offenses and low probability of incarceration, respectively.
Henggeler et al., 200835	Implementation, Sustainment	Treatment	Contingency Management within the context of MST	Mental health and substance abuse clinics	Outer (training); Inner (organizational; provider; fidelity; coaching)	Workshop training + intensive quality assurance vs. workshop training only	Multiple-provider/caregiver- youth	30/70
	Summary: Workshop plus intensive quality assurance was more effective than workshop only at increasing practitioner implementation of contingency management techniques in the short-term based on youth and caregiver reports, and these increases were sustained based on youth reports.
Holth et al., 201195	Implementation, Sustainment	Treatment	Multiple (MST, contingency management)	Mental health and substance abuse clinics	Outer (training); Inner (organizational; provider; fidelity; coaching)	Workshop training + intensive quality assurance vs. workshop training only	Multiple-supervisor/provider	21/41
	Summary: Compared to workshop only, intensive quality assurance enhanced therapist adherence to CBT techniques but not to contingency management. Cannabis abstinence increased as a function of time in therapy and was more likely with stronger adherence to contingency management.
Kolko et al., 201259	Implementation	Treatment	AF-CBT	Mental health clinics, child welfare	Outer (training)	“Learning community”-based training versus training as usual	Single-agency (not addressed in analyses)/providers	(10)/182
	Summary: Providers who participated in the community condition rated higher on knowledge and use of skills included in the EBP. Practitioners in both groups reported significantly more negative perceptions of organizational climate during the intervention phase.
Lochman et al., 200955	Implementation	Prevention	Coping Power	Schools	Outer (training); Inner (provider; fidelity); Outcomes (youth)	2 different levels of training and a control condition	Multiple-school/counselor/student (school not included in modeling-some schools shared counselors)	(57)/49/531
	Summary: The higher intensity training condition resulted in better engagement with students and better child outcomes (reductions in children’s externalizing behavior problems and improvement in their social and academic skills.
Pentz et al., 199057	Implementation	Prevention	Midwestern Prevention Project	Schools	Inner (organizational; provider; fidelity), Outcomes (participant)	Implementation versus wait list control (by school)	Multiple-schools, teachers, students	42/65/5065
	Summary: All schools implemented the program. Exposure to the intervention was the strongest predictor of student outcome (minimizing increases in drug use behavior). Reinvention (adherence/adaptation) was not related to student outcome.
Rohrbach et al. 199333	Preparation, Implementation	Prevention	Adolescent Alcohol Prevention Trial	Schools	Inner (organizational- leadership; adopter)	Brief versus intensive training x principal versus no principal intervention	Multiple-teacher, school	60/25
	Summary: There was no difference in rates of implementation/fidelity between the brief and intensive teacher training. Principal training increased implementation/fidelity. Teachers who reported higher rates of implementation/fidelity also reported less teaching experience, stronger self-efficacy, greater enthusiasm, better preparedness, higher teaching method compatibility, and greater principal encouragement. Integrity of implementation was related to positive program outcomes.

Note: ADHD=attention-deficit/hyperactivity disorder; AF-CBT=Alternatives for Families: A Cognitive-Behavioral Therapy (AF-CBT); ARC=availability, responsiveness, and continuity; CAFAS=Child And Adolescent Functional Assessment Scale; CBT=cognitive behavioral therapy; CD=compact disc; MST=Multisystemic Therapy.

In the following sections we present an analysis of the findings of the included 73 articles organized by the EPIS framework constructs. Papers that address more than one EPIS phase (n=16) are counted and analyzed in all relevant EPIS phase sections; results from these studies are described in each relevant phase. Table 2 provides a summary of key paper characteristics by EPIS phase.

Table 2.

Summary of Key Characteristics of Studies by Exploration, Preparation, Implementation, Sustainment (EPIS) Framework Phase

	Exploration (n=2)	Preparation (n=19)	Implementation (n=60)	Sustainment (n=8)	TOTAL (N=73)a
					n	%
Intervention Type
Preventive	1	8	22	3	27	37.0
Clinical	1	11	38	5	46	63.0
Design and Analysis
Experimental	1	3	14	4	18	24.7
Multilevelb	0	6	20	3	23	31.5
Contexts Examined
Inner	1	18	56	8	67	91.8
Outer	2	11	24	3	27	37.0
Intervention	1	2	3	0	4	5.5
Outcomesc	0	0	12	1	12	16.4

Note:

^aTotal = nonduplicated count of studies

^bMultilevel = studies that examined results using hierarchical modeling (e.g., providers clustered within clinics)

^cOutcomes = parent, child, and/or family outcomes

Exploration

The Exploration phase involves awareness of a clinical or service issue and developing processes to identify an improved approach to address that issue. Two papers met our review criteria and addressed issues of exploration. One paper focused on preventive interventions²⁹, the other on a clinical intervention³⁰. The preventive intervention paper utilized a randomized design and found that using the internet to facilitate the exploration of preventive interventions by schools and community agencies was more effective than pamphlets or CDs (outer contexts of the exploration process [i.e., information transmission]).²⁹ The treatment paper utilized both qualitative and quantitative (i.e., mixed) descriptive methods to assess the pilot implementation of a program for providing services to children with attention-deficit/hyperactivity disorder in primary care settings. The authors found that the intervention itself required adaption to provide better guidance for addressing assessment of insurance coverage, managing complex clinical situations, and working with mental health specialists in both outer [interorganizational networks] and inner [patient, family] contexts.³⁰

Preparation

The Preparation phase involves planning and decision making regarding adoption and implementation of an innovation and addressing system, organizational, and individual readiness for change. Nineteen papers addressed issues of preparation with 8 focused on preventive interventions (with the Triple P Positive Parenting Program®³¹ [n=3] and the Arson Prevention Program for Children [n=2]³² being the most commonly studied). Eleven studies focused on clinical interventions with 2 focused specifically on medication guidelines. The remainder focused on different psychosocial/psychotherapeutic interventions. Almost all of the papers involved an explicit focus on the inner context of the preparation and implementation process (n=17) with organizational (e.g., staffing, resources, climate), individual provider, and leadership factors being the most commonly examined. Eleven papers included an explicit focus on outer contexts with interorganizational networks, sociopolitical contexts, and training factors being the most commonly examined. One paper focused explicitly on the intervention itself and ways it should be adapted to improve prospects for effective implementation. Only 3 papers utilized randomized experimental designs. The remainder utilized descriptive methods, including 3 that exclusively utilized qualitative methods.

The 3 randomized controlled trials examined different methods to increase likelihood of EBP implementation. Rohrbach et al.³³ found that the training of the school principal, but not the intensity of teacher training, increased the likelihood of adoption of a school-based intervention. Foster and Stiffman³⁴ found that the addition of a personal digital assistant version of a desktop decision support system for social service workers increased EBP use. Epstein et al. found that (1) providing pre-packaged medications to enable practitioners to conduct n of 1 double-blind placebo-controlled trials of different stimulant doses for use with individual patients, along with (2) specialist support for its interpretation to pediatric practices increased the use of such trials, though it did not increase the frequency of monitoring of ADHD symptoms over time. None of these papers examined the importance or impacts of outer contextual factors in preparation for implementation.

Descriptive studies also consistently supported the importance of inner contextual factors during the preparation process and suggest that outer contextual factors may be important as well. Inner context factors that emerged as important beyond those identified in the above experimental studies include the fit (or perceived fit) of the intervention with the organization,^32,35–37 provider self-efficacy,³⁸ the availability of appropriate training, ^38,39 adequate resources (including personnel and funding),^39–43 use of locally and nationally-generated data for the decision making process,^37,41,44 leadership (e.g., leadership skills, administrator support of quality improvement goals, attitudes towards EBP implementation),^{37,39,42,43,45} and organizational culture and climate.^43,46 Some studies suggest that providers earlier in their careers had more positive attitudes towards EBPs and were more likely to implement them,^42,46 while others suggested that more experienced providers were more likely to actually adopt an intervention.³⁵ Outer contextual factors associated with preparation included the quality of interorganizational networks,^38,47 leadership from policymakers at the state level,^41,48 and the availability of implementation materials from nationally recognized sources.⁴¹

Implementation

The Implementation phase addresses factors related to the active implementation and scale up of an innovation. Sixty papers (82% of those included in this review) addressed issues of implementation. Twenty-three of these papers focused on preventive interventions (with the SafeCare® child neglect intervention [n=4])⁴⁹ and the Triple P Positive Parenting Program³¹ [n=3] being the most commonly studied) and 37 focused on clinical interventions (with Multisystemic Therapy®⁵⁰ being the most commonly studied [n=12]). Almost all of these papers included an explicit focus on the inner context of the implementation process (n=56) with training/fidelity monitoring and support and individual provider characteristics being the most commonly examined. Twenty-four papers included an explicit focus on outer contexts with training and interorganizational networks (particularly EBP developer engagement in implementation) being the most commonly examined. The importance of adherence to EBP protocol was examined in 27 papers while the place of flexibility and adaption of EBPs for use with diverse populations and settings was examined in 8 papers. Twelve papers examined the relationship of implementation quality and related factors to child, parent, and/or family outcomes. Fourteen papers utilized randomized experimental designs; 2 utilized nonrandomized experimental designs, and 4 used quasiexperimental designs. The remainder used descriptive methods, including 14 that exclusively utilized qualitative methods.

Five of the 16 of studies with experimental designs focused on the level of provider support necessary to assure implementation with adherence to EBP protocols. Consistent across these studies, which involved multiple EBPs and both preventive and clinical services, is that ongoing supervision, fidelity monitoring, and support to providers resulted in higher levels of adherence/fidelity.^51–55 Two other experimental studies suggested that supervision, monitoring, and support also improve staff retention⁵⁶ and reduce staff emotional exhaustion,³⁷ One paper found that training of school principals, but not the teachers delivering the intervention, improved adherence/fidelity to the EBP protocol.³³ Two of the above papers extended their work to demonstrate that higher levels of adherence/fidelity resulted in better patient/family outcomes;^53,55 another study that examined the impact of fidelity/adherence on student outcomes in the experimental arm of their effectiveness study were unable to find such a relationship.⁵⁷ Studies with experimental designs also support the premise that attention to organizational factors results in more successful implementation of EBPs, including the formation of “communities of practice”⁵⁸ or “learning communities”⁵⁹ to support EBP implementation and the use of availability, responsiveness, and continuity (ARC), an organizational intervention designed to improve organizational culture and climate in mental health and social service organizations.⁶⁰ The latter is particularly notable as it tied improvement in organizational culture and climate to youth outcomes in organizations that received the ARC intervention.⁶⁰ However, this study did not find an impact of ARC on intermediary implementation outcomes as identified by Proctor et al.⁶¹ (i.e., fidelity). One experimental study suggested that incentives to providers increases the quality of EBP implementation.⁶² Finally, studies that examined different approaches to initial training (brief versus intensive,³³ didactic versus experiential,⁶³ in-person versus videoconference⁶⁴) were unable to demonstrate significant differences in their impacts on implementation success.

Descriptive studies were generally consistent with the findings of the experimental studies around the need for training and ongoing monitoring/support in order to achieve successful implementation^35,65–71 and that better adherence was related to better child outcomes.^67,72–74 Additional inner contextual factors that were related to more successful implementation in these studies included those related to providers characteristics,⁷⁵ including sociodemographics,³⁵ experience,^43,76 disciplinary background,³⁵ exposure to^32,38 and attitudes towards EBPs,^{35,69,71,77–80} sense of self-efficacy,^32,38,68,77 opportunities for reward/achievement,^67,80 concerns about competing demands [which reduced likelihood of implementation]^71,79 and ethnic matching of therapist and youth.^78,81 Supervisor factors associated with implementation included experience with EBPs and⁸² relationships with supervisees,⁷⁰ while organizational factors included leadership,^{65,66,79,80,83} organizational culture and climate,^36,73,84 perceived fit of an EBP with the organization’s mission, ^{21,25, 53,63,80} organizational support for implementation,^{43,65,67–69,79,83} and effectively addressing institutional barriers^68,79,85 such as financing,^65,75,80 resources,³⁶ readiness to change,³⁵ and governance [i.e., for-profit organizations were more likely to implement EBPs than nonprofits].⁷⁶ Within the context of fidelity to EBP protocols, some studies suggested allowing therapists flexibility in their implementation of the EBP (e.g., adapting cognitive behavioral therapy for anxiety to better meet specific child needs,⁸⁶ language translation⁶⁵) improved engagement, treatment retention, and child outcomes.⁸⁶ Other studies noted that adaptation was common when providers implemented EBPs^87–90 with another study developing a process in which EBP developers worked with providers and youth to identify and approve potential adaptions.⁹¹ Child and family characteristics,^69,78,81,92 and family engagement in the treatment process^35,69,79 were also reported to be related to implementation success.

Outer contextual factors identified in these studies that supported the implementation process include interorganizational/provider networks,^38,79,80 external ratings/report cards,⁹³ and interactions between organizations and/or individual providers and EBP developers.⁸⁹

Finally, these studies also suggest that additional attention should be paid to the interaction of different variables in the implementation process. For example, Asgary-Eden and Lee found that provider experience was a significant predictor of implementation only in those organizations that were perceived as less amenable to implementation (in those organizations that were perceived amenable to implementation, provider experience was not related to implementation).⁴³ Schoenwald et al. found that organizational factors were predictive of child outcomes only when EBP adherence was low among the organization’s providers.⁶⁷

Several papers raised questions about optimal methods for measuring implementation quality (which is generally defined as adherence to EBP protocols). Independent observation, provider self-report, and parent report have all been utilized in implementation studies. Studies that have compared at least 2 of these methods suggest that agreement between them is generally good⁹⁴ but that providers tend to rate their implementation quality as higher than independent observers,⁹⁴ and in one study independent observer ratings quality were related to child outcomes, but provider ratings were not.⁹⁴

Sustainment

The Sustainment phase focuses on the maintenance of an innovation beyond 1 year. Only 8 papers addressed issues of sustainment with 3 of these papers focused on preventive interventions (with SafeCare [n=2])⁴⁹ being the most commonly studied) and 5 focused on clinical interventions (with Contingency Management in combination with Multisystemic Therapy⁵⁰ being the most commonly studied [n=2]). All of these papers included an explicit focus on the inner context of the implementation process with training/fidelity monitoring and support being the most commonly examined. Three papers included an explicit focus on outer contexts with interorganizational networks being the most commonly examined. One paper examined the relationship of sustainment and factors related to child, parent, and/or family outcomes. Four papers utilized randomized experimental designs, 4 utilized descriptive methods, including 1 that exclusively utilized qualitative methods.

Three of the 4 randomized controlled trials examined the impacts of different intensities of training/fidelity monitoring and support. Both Henggeler et al.⁵⁴ and Holth et al.⁹⁵ found that ongoing fidelity monitoring and support was superior to an initial workshop only for sustaining adherence to EBP protocols (in this case Multisystemic Therapy and Contingency Management), although Holth et al.’s results suggest that the impacts may have been limited to the use of cognitive behavioral therapy techniques included in Multisystemic Therapy and not Contingency Management. Aarons et al.³⁷ found that fidelity monitoring resulted in lower levels of staff emotional exhaustion when such monitoring took place in the context of sustainment of an EBP (in this case SafeCare) compared to fidelity monitoring for providers providing usual care services. In their small clinical trial comparing desktop computer only versus desktop computer plus personal digital assistant access to the a decision making support system, Foster and Stiffman³⁴ found that use of the system declined during the sustainment phase of their trial, though use of the decision support system remained higher in the desktop computer plus personal digital assistant condition during the sustainment phase.

The descriptive studies reinforced the importance of access to ongoing supervision to assure the sustainment of an intervention with fidelity ^65,96 (and that connections with the EBP developers themselves can be particularly helpful in supporting sustainment)⁶⁵ while also suggesting that careful selection of the EBP (taking into account important organizational, staff, and children and families served),^65,69 addressing sustainability during the preparation and implementation phase,⁶⁵ and a supportive organizational culture increases the likelihood of sustainment.⁸⁴

DISCUSSION

Of the many inner contextual factors that have been examined in the studies included in this review (e.g., provider characteristics, organizational resources, leadership), fidelity monitoring and supervision were most frequently examined and have the strongest empirical evidence. Ongoing fidelity assessment, supervision, and support increases the likelihood that expected intervention effects will be realized and has important ancillary benefits including reduced staff burnout and improved staff retention. This is an important finding for the health system and organization leaders as a strong and stable workforce is critical for delivery of effective services and cost containment.⁹⁷ While the focus of fewer studies, it is also compelling that interventions specifically focused on improving organizational culture and climate were associated with better intervention sustainment as well as better child and adolescent outcomes. These studies suggest that there is more to dissemination and implementation than simply getting individual providers to deliver interventions with fidelity, but that we also must pay attention to characteristics of the workplace.^73,84

Our findings suggest that factors such as training strategies and technologies to support intervention use are important to the dissemination and implementation process, though the results are more clearly linked to the outer context in contrast to the inner context. Studies suggest that web-based clearinghouses that provide EBP descriptions, efficacy/effectiveness ratings, and relevance ratings for particular service sectors or disorders will be more accessible and have higher utility with easier, more flexible access than print or physical media.^98,99 Though there is currently a great deal of interest in the relative impacts of different dissemination approaches, we were largely unable to find studies that support specific sets of strategies in this regard. Other outer contextual factors that appear to support the dissemination and implementation process include connections with EBP developers and interorganizational networks that link key stakeholders and facilitate interaction and communication. Unfortunately, we found little to no focus on policy or larger system issues that may impact uptake of EBPs. This is notable given federal, state, and local legislation and policies often dictate or encourage EBP adoption and use, but such policies do not always provide appropriate infrastructure support for implementation (e.g., funded vs. unfunded mandates).¹⁰⁰

This review raises a number of important methodological issues in the field. First, while we found a large number (n=14,583) of citations that invoke the terms dissemination, implementation, sustainability, and sustainment, we identified only 73 well-conducted, empirical studies of implementation process and/or outcomes in children’s mental health. When we adjust this number for the fact that several papers were published drawing data from the same studies, the number of unique studies is even smaller (n=44). While this low yield is partially the result of our inclusion criteria, which excluded review articles and single case descriptions, most of the papers identified by our search terms did not address dissemination or implementation per se. As the relatively young field of implementation science matures and garners, for example, its own set of specific search terms in Medical Subject Headings (MeSH),¹⁰¹ conducting more efficient literature searches will be possible, but identifying relevant articles is currently more challenging than is typical in studies of children’s mental health.

Second, studies encompassed different research methods including quantitative, qualitative, and mixed methods, consistent with the needs of a developing field of study but this is also indicative of the complexity of studying the dissemination and implementation process, which often involves individual providers nested within organizations, larger interorganizational networks, and functioning in a complex policy environment.¹⁰² Only 11 papers (15%) utilized a cluster-randomized design necessary to address these complexities. While such studies are centrally important to advancing the field (and we hope a future systematic review will be able to identify many more), it is also important to acknowledge that some dissemination and implementation research questions do not easily lend themselves to experimental designs (e.g., changes in policy) and the other methodological approaches included in this review will continue to remain important even as the field advances. For example, where the number of organizational units may be limited thus decreasing quantitative statistical power, qualitative methods can be used for triangulation (to corroborate findings) and expansion (to add new information) of results.¹⁰³ Other alternatives include rollout designs, where multiple cohorts of providers or organizations are randomized in sequence, and each cohort serves as the control group for the previous cohort.^104,105 Other methods will be required to identify and explicate complex interactions and change mechanisms within social and organizational contexts. It may be particularly fruitful to draw on measurement and process assessment from other fields including time series analysis, regression discontinuity designs, system dynamics, network analysis, and qualitative methods.^105,106.

Third, that this literature was dominated by implementation phase studies addressing training and training support is most likely because most of this research was conducted by intervention developers working to determine the most efficient and effective ways to implement their programs with fidelity. There was also a disproportionate number studies examining implementation of Multisystemic Therapy (MST). MST, one of the most widely utilized and best-studied EBPs for adolescents with behavior problems, was the subject of one of the earliest NIH-funded implementation studies in child mental health.⁶⁷ However, as implementation science matures research studies should address a more diverse array of EBPs and provide better coverage of the exploration and sustainment phases of this complex process.

Limitations of this review should be noted. First, the heterogeneity of papers included in this systematic review, which range from cluster randomized trials to qualitative investigations, precluded the utilization of quantitative meta-analytic techniques. The field needs to advance further before such techniques can be employed. Second, this paper utilizes the EPIS Framework as an organizing heuristic for this systematic review; it does not however represent an empirical test of this framework. Third, this paper focuses on dissemination and implementation research in child and adolescent mental health. Many of the general principles that emerged from this systematic review do not appear to differ in major ways from those emerging from research in other settings.¹⁰⁷ However, children’s mental health prevention and clinical services have a number of distinguishing characteristics that could lead to a greater separation of this literature over time. For example, children typically live in a setting with parents/caregivers and many interventions involve the family. There is a preponderance of psychosocial interventions for children and adolescence and a much thinner evidence-base for psychopharmacologic interventions. Professionals working with children and adolescents receive what is often additional, specialty training that distinguishes them from those serving adults. Service organizations and providers often are dedicated specifically to child services. There are also often different insurance and funding infrastructures for child and adolescent-focused services. Indeed, some findings highlighted in this review, such as the differential impacts of training on provider fidelity in implementing two EBPs in a single study (i.e., Multisystemic Therapy and Contingency Management)⁹⁵ suggest that an exclusive focus on child and adolescent mental health is warranted.

The present review reflects the current state of the field of implementation research as applied to child and adolescent mental health, capturing its progress from an era focused largely intervention development, to one that places increasing importance on moving effective interventions into widespread use in order to have a positive impact on public health. Forces moving the field in this direction include the development of evidence-based medicine and its spread to mental health prevention and treatment, the development and increase in interest and funding for research focusing on mechanisms for effective EBP implementation, and calls from funding agencies to increase the use of EBPs. Because scholarly and applied interest in implementation is increasing, there are a burgeoning number of studies underway that will enrich this literature over the next several years. Indeed, there are a number of implementation research areas ripe for study. This include approaches such as effective public–academic collaborations and community partnered research,¹⁰⁸ engaging intervention developers to consider implementation during the intervention design,¹⁰⁹ including implementation research as part of efficacy and effectiveness studies,¹¹⁰ the use of technology in training,¹¹¹ coaching and fidelity assessment and feedback,²⁴ the use of organizational development strategies to improve implementation efficiency, ⁶⁰ and the use of a strategic and comprehensive approach move large scale implementation through EBP exploration, preparation, implementation, and sustainment,²⁴ and strategies to scale-up EBPs across entire service systems¹¹². These studies carry additional significance in the coming years as the Affordable Care Act places greater emphasis on mental health services as an essential benefit as well as greater accountability regarding the use of evidence-based treatments, integration of allied health and primary care, overall quality of care, cost efficiency, and patient outcomes.¹¹³

Implementation research is inherently complex. It involves the study of system, organizational, individual, and social change that takes place in complex systems involving patients, clinicians, organizations, policymakers, drawing on an increasingly sophisticated evidence base, and supported by exciting technological advances. Despite this complexity, the development and testing of dissemination and implementation strategies is needed in order to efficiently move effective interventions into usual care. Indeed, the results of dissemination implementation studies must be broadly applied so that the benefits of basic scientific discovery and clinical intervention can be effectively and efficiently translated to improving the public health status of individuals and the communities in which they reside.

Figure 2.

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) study selection flow diagram.