Disseminating Child Maltreatment Interventions: Research on Implementing Evidence-Based Programs

Evidence-based practices (EBPs) have the promise to reduce child maltreatment and improve the lives of countless families and children, but effective implementation entails many challenges. Efficacious interventions now exist for parents at risk for or who have perpetrated maltreatment (i.e., Chaffin et al., 2004; Chaffin, Hecht, Bard, Silovsky, & Beasley, in press; Kolko, Iselin, & Gully, 2011; Prinz, Sanders, Shapiro, Whitaker, & Lutzker 2009; Webster-Stratton, 2010) and for youth victimized by maltreatment (Cohen, Mannarino, Berliner, & Deblinger, 2000; Deblinger & Heflin, 1996). Despite the development of such programs, many families involved in the child welfare and foster care systems are not provided interventions or services with strong empirical support (Barth et al., 2005; Hurlburt, Barth, Leslie, Landsverk, & McCrae. 2007). For example, while parenting interventions are virtually de rigueur on service plans, extant parenting services often consist of didactic classroom-centered parent training or mix systems therapy and case management that bear little resemblance to the evidence-based parent behavior management programs that are proven effective (Barth et al., 2005; Casanueva, Martin, Runyan, Barth, & Bradley, 2007).

The child maltreatment field is in the nascent stage with regard to transporting EBPs into relevant public sector services systems. Fortunately, research is advancing our understanding of the critical contextual factors at the client, therapist, organization, training, and sociopolitical levels that can increase the likelihood of effective EBP implementation (e.g., Addis, 1997, 2002; Beidas & Kendall, 2010; Berwick, 2003; Chamberlain et al., 2008; Elliott & Mihalic, 2004; Herschell et al., 2003; Sanders & Turner, 2005; Saul et al., 2008; Schoenwald, 2003; Turner & Sanders, 2006; Wandersman et al., 2008; Weisz & Gray, 2008). The purpose of this special issue is to advance implementation science in the field of child maltreatment prevention and intervention by presenting data-based articles that highlight recent efforts to conduct rigorous research on EBP dissemination and implementation.

For purposes of this special issue, we use definitions of implementation and stages of implementation provided by National Implementation Research Network (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). Specifically, implementation is defined as “a specified set of activities designed to put into practice an activity or program of known dimensions.” As implementation is a process, and not an event, Fixsen and colleagues identified the following discernible stages as critical to effective implementation of EBPs: (1) Exploration/Adoption, (2) Program Installation, (3) Initial Implementation, (4) Full Operation, (5) Innovation, and (6) Sustainability (for further description, please refer to Fixsen et al., 2005). The articles published in this issue address all of the implementation stages, with the exception of the innovation stage. Implementation stages relevant to this issue are presented below, along with a brief review of the current state of the literature, and a discussion of how the special issue articles advance current knowledge.

Exploration/Adoption

In order to explore and potentially make decisions to adopt an EBP, one first must have an awareness of EBPs, and believe that the implementation of an EBP can meet a particular need or challenge that has been identified (Grol, Bosch, Hulscher, Eccles, & Wensing, 2007). This stage has been described as a major obstacle for systems serving maltreating families and victimized youth for two main reasons. First, research-based information on the appropriateness/effectiveness of EBPs with maltreating families is not always readily available to providers and administrators in child welfare and public mental health systems. Second, the organizational climate and culture in such systems often are not conducive to rapid programmatic innovation based on scientific evidence, including the installation of EBPs (Chaffin & Friedrich, 2004; Glisson & Schoenwald, 2005; Klein & Sorra, 1996; Simpson, 2002).

Exploration and adoption may be further complicated by the lack of agreed upon language and definitions for imperative implementation terms, even ones as basic as the term EBP (Sexton & Kelley, 2010). As highlighted in a recent article by Rosenblatt and Tseng (2010), when local leaders in child welfare, probation, and mental health agencies were asked about their definitions of EBP, varying definitions were provided, many of which did not contain the terms research or evidence. In this issue, researchers use a total of four different terms to refer to EBPs. Even among academics and leaders in implementation, consistent language has not been established. The definitional problem is further complicated by the fact that several different EBP rating systems exist, and each have varying requirements to qualify a program as evidence based (i.e., California Evidence-Based Clearinghouse, Promising Practices Network, office of juvenile justice and delinquency prevention (OJJDP), substance abuse and mental health services administration (SAMSA), and Department of Health and Human Services (DHHS) Criteria for Evidence-Based Program Models). Given such lack of consensus, it is understandable why decision makers and providers may feel at a loss when it comes to exploring and adopting such practices.

Relevant to this exploration stage, Allen, Gharagozloo, and Johnson (2012) provide data from a nationwide survey with a basic research question: Can clinicians serving maltreated youth appropriately identify EBPs? Not surprisingly, results indicated that clinicians who were provided a comprehensive list of programs were not able to distinguish EBPs from non-EBPs. However, clinician characteristics (e.g., favorable attitudes, training background) were significantly associated with the ability to successfully complete this task. As part of the survey, clinicians were also asked to select practices for which they desired further training, and results suggested no distinction in desire for training between EBPs and non-EBPs. These data strongly suggest that the dissemination of EBP knowledge is still quite limited among providers in our field. Researchers have suggested ways to improve diffusion, including requiring EBP courses during social work and clinical graduate training (Howard, McMillen, & Pollio, 2003; Rubin, 2011), promoting workshops on EBPs for continuing education credits, and on-the-job training efforts (Rakoshik & McManus, 2010; Williams & Martinez, 2008). While these efforts would be a good step, major shifts in practice are unlikely to happen without administrator and decision maker buy-in to the basic premise that EBPs are critical to improved outcomes. In general, much more work is necessary to further explicate how to boost exploration, interest, and commitment to adoption of EBPS in the child maltreatment field, as a necessary first step to ensuring that families served have access to the most effective services.

Program Installation

In this stage, adopters begin actively preparing for the initial implementation of the EBP by enhancing structural supports, soliciting referrals, solidifying funding streams, addressing human resource needs, and developing appropriate policy (Fixsen et al., 2005). Very little research has focused on this stage to date, yet without awareness of how to methodically address these critical factors, implementations are likely to fail. In an innovative article by Dorsey, Kerns, Trupin, Conover, and Berliner (2012), data are presented on a training and consultation model, Project Focus (Dorsey, Kerns, Trupin, Conover, & Berliner, 2012). It is designed to improve the capacity of EBP brokers, in this instance, child welfare caseworkers, for making appropriate referrals based on clinical characteristics of cases. This training model was tested across four child welfare settings, in a state where clinicians providing child welfare services were extensively trained in several EBPs. Results from this pilot project were promising for improving the referral agents knowledge of available EBPs, as well as the fit between an EBP and family needs; however, the behavioral change of caseworkers, as measured by referrals to EBPs, did not demonstrate significant improvements. Nonetheless, this type of work is an important initial effort to understand more about how to set up and enhance key program installation components, which are necessary to achieve successful implementation and sustainability.

Initial Implementation

The initial implementation stage involves EBP provider training, as well as related changes to field work with clients and families. Empirical work on this stage is the most advanced, although there is still much to be learned. Mounting research documents what constitutes high-quality EBP training, and how the posttraining context should be structured to promote positive provider behavior change relevant to the EBP (Sanders & Turner, 2005; Turner & Sanders, 2006). A recent review of studies examining therapist training for EBPs by Beidas and Kendall (2010), indicates that active learning strategies (modeling, practice opportunities, building self-efficacy, interaction among learners, and role-plays) are integral to therapist EBP adherence and skill. Technical assistance (i.e., ongoing supervision/consultation) with field work also appears to be a key factor for successful implementation, particularly with regard to maintaining model fidelity (Fixsen et al., 2005; Schoenwald & Hoagwood, 2001).

In this issue, articles by Kolko et al. (2012), Nelson, Shanley, Funderburk, and Bard (2012), and Whitaker et al. (2012) examine initial implementation of Alternatives for Families-Cognitive Behavioral Therapy (AF-CBT), Parent–Child Interaction Therapy (PCIT), and SafeCare, respectively. Training procedures described in all three articles follow best practice recommendations for provider workshop and field training, with workshop training including behavioral rehearsal and feedback and postworkshop support including frequent consultation with EBP experts to discuss implementation of EBP skills with families. The three studies ask three different and important questions about initial implementation. Kolko and colleagues examine whether EBP training and support affects knowledge, attitudes, and use of AF-CBT relative to no training. Nelson and colleagues examine how initial provider attitudes are related to their participation and satisfaction with varying methods of postworkshop training support and case enrollment for PCIT. Whitaker and colleagues focus on initial implementation indicators of a statewide rollout of SafeCare.

Findings from Kolko et al. (2012) suggest that practitioners randomly assigned to AF-CBT training, as compared to training as usual, reported knowledge gains and greater use of AF-CBT procedures with families. Interestingly, outcomes did not vary by individual, supervisor, program, or organizational characteristics. Whitaker and colleagues (2012) similarly found that providers who were trained in SafeCare could implement the protocol with high levels of fidelity (as rated on fidelity checklist by expert listening to audio recording). Furthermore, trainees who showed highest performance on role-plays in workshops were ultimately the most successful in achieving fidelity in the field, suggesting generalization of skills from an active training. Similar to Kolko et al. (2012), few correlates of implementation outcomes emerged. Nelson et al. (2012) found that therapist attitudes predicted therapist participation in two different forms of postworkshop consultation (remote real-time online consultation vs. phone). Interestingly, therapists who reported greater likelihood of diverging from EBPs were less likely to attend postworkshop phone consultation, and younger therapists, as well as those who reported more openness to EBPs, were more likely to attend online sessions.

Each of these articles demonstrate, albeit indirectly, the difficulties of achieving even initial implementation due to a variety of reasons, including provider turnover, poor provider participation in consultation, or lack of referrals or system infrastructure that support EBP implementation. Significant questions relevant to this stage of implementation remain. For instance, as noted by Chinman et al. (2005), few empirical studies have examined the effectiveness and cost-effectiveness of different training methods or models, such as train the trainer or cascading training model, or how technology can be utilized in training. Additionally, further clarification is needed with regard to the baseline individual and organizational characteristics that act as moderators or predictors of training, as well as potential assessments or interventions that could be administered prior to or during training to ensure best provider outcomes (i.e., more appropriate readiness screens, motivational interventions to enhance the initial appeal of EBP to providers/organizations). Finally, research is warranted that focuses on the explication of which training, organizational, and systemic factors involved with the initial implementation stage predict sustainability of EBPs with high in-field fidelity and, ultimately, improved family outcomes.

Full Operation

This stage of the implementation process occurs when EBP-trained providers are receiving referrals, carrying out the EBP with proficiency and skill, and there is full support at the organizational and sociopolitical levels, which facilitates the new practice. Two articles in this issue (Aarons, Fettes, Sommer-feld, & Palinkas, 2012; Damashek, Bard, & Hecht, 2012) provide a snapshot of what takes place at the provider level and family level when an EBP has become fully operational. Both articles examine data from a statewide implementation of the SafeCare.

Damashek and colleagues (2012) focus on how families perceive and respond to an EBP operating at the full operation stage. It is well established that family attrition rates for parenting services are extremely high, even for programs delivered in the home (Duggan et al., 2000; McGuigan, Katzev, & Pratt, 2003). Thus, in order for an EBP to be successful, the program must be engaging and relevant to the targeted clients. To date, there have been few implementation studies of EBPs in child welfare; thus, there is little information about how families receiving child welfare services will respond to an EBP. Damashek et al. present data collected from over 1,300 families participating in child welfare services, who were either assigned to services as usual or SafeCare. Results indicated that families who received SafeCare services reported higher levels of therapist respect for cultural differences, which directly related to client satisfaction, engagement in services, and attainment of service goals. These findings are very promising and can serve to counter provider fears that manualized treatments may be too rigid to engage and fit individual client needs. Ultimately, these findings suggest that if we learn how to methodically and effectively implement EBPs, such services will be well received and accepted by clients.

Aarons and colleagues focus on methodology and discuss the advantages of using mixed-methods designs in implementation research. As an application of this methodology, they present staff turnover data during a statewide implementation of SafeCare. In the public service, systems that offer child maltreatment interventions, employee turnover is a significant issue and decision makers rightfully have concerns for how this can impact EBP training costs and sustainability (Aarons & Sawitzky, 2006; Glisson, Dukes, & Green, 2006; Glisson & James, 2002; Knudsen, Johnson, & Roman, 2003). The mixed-methods approach enhanced the interpretation of quantitative findings through the use of qualitative methods. The quantitative results showed that providers assigned to an EBP with supportive consultation compared to providers assigned to EBP with limited consultation or services as usual, had greater job retention over a 3-year period. The additional qualitative data presented gives providers in this study a stronger “voice” via direct quotes about their experiences with Safe-Care and consultation and provides strong examples of the variation of experiences not reflected by quantitative data. Overall, this methodological approach is very valuable for studying EBP implementations in public sector service systems and offers advantages over quantitative data alone for communicating with stakeholders.

While these articles help elucidate the impact of EBPs on families and providers in the full operational stage of implementation, there remains much to learn about system, organizational, provider, and family factors that can enhance or minimize the likelihood that this stage is achieved. At the system and organizational level, we need to know more about the role of leadership and support at all pertinent levels (e.g., executive director, managers, clinical supervisor; Aarons, Hurlburt, & Horwitz, 2011, as well as how to maintain sociopolitical support in ever-changing public service sectors, including relevant policy and funding streams for EBPs. At the provider level and client level, factors such as acceptance of innovation, and issues pertaining to fidelity versus adaptation (Saul et al., 2008) should be further explored.

Sustainability

Sustainability has been defined as the extent to which an EBP can deliver the intended benefits over an extended period of time after external support is terminated (Rubin, 2008). The ultimate goal during this stage is continued effectiveness of the implementation by ensuring providers deliver high-quality services and engage and retain families to achieve the desired outcomes (Damschroder et al., 2009). Basically, all other stages of implementation must be achieved before sustainability can be considered. Consequently, very little is known about the key factors for sustaining effective programs targeting child maltreatment prevention and intervention.

Two articles in this special issue provide findings that advance our knowledge of this stage. Allen and Johnson (2012) present data from a nationwide survey of clinicians from child advocacy centers, which inquired about Trauma Focused-Cognitive Behavioral Therapy (TF-CBT) training and implementation. While little information on the type of TF-CBT training received was provided by participants, a significant majority of clinicians reported being trained in TF-CBT and using TF-CBT components in practice. However, when providers were asked more specifically about their use of the five core components of TF-CBT, a smaller portion of providers reported regularly implementing all components, with the two components least likely to be used being those most important to addressing child trauma (trauma narrative and cognitive restructuring). Similarly, Shapiro, Prinz, and Sanders (2012) interviewed providers who had been trained in the Positive Parenting Program (Triple P) over a year post training. Results suggested that organizational characteristics (e.g., supervision, other responsibilities) and provider characteristics (e.g., self-efficacy, perception of knowledge of skills, intervention producing change in families, EBP requirements) had significant impacts on reported program use. Neither study, however, examined objectively observed implementation to get a sense of whether models were implemented with fidelity, what adaptations had occurred, and other important questions about sustained implementation.

Taken together, these studies highlight the importance of posttraining environment and provider characteristics for sustaining the implementation of EBPs at the provider level. They also suggest the need for ongoing fidelity checks and perhaps booster training sessions in the sustainability stage to ensure that EBPs continue to be employed with integrity. There is much more to learn about the sustainability stage, above and beyond the provider level as well. For instance, future work should examine how organization and systems levels factors (e.g., program requirements, partners, and political alliances) affect sustainability. Longitudinal work that focuses on implementation over time and across stages would be the most valuable to a comprehensive understanding of sustainability in public sector social services (Greenhalgh et al., 2004). Additionally, examining the long-term role of EBP purveyors and treatment developers in this stage is relevant. For instance, to what degree does developing well-trained local expertise in an EBP versus retaining the training/fidelity support with EBP experts impact the EBP sustainability and effectiveness of EBP with client outcomes?

Conclusions

The articles presented in this special issue were intended to promote further development of implementation science in the child maltreatment field. Results from the selected articles include implementation efforts pertaining to five evidence-based programs relevant to child maltreatment prevention and intervention (AF-CBT, SafeCare, PCIT, Triple P, TF-CBT, respectively), and address five of the six implementation stages identified by the National Implementation Research Network. While the articles in this special issue advance the current knowledge of the field, there remains much to be learned in each implementation stage. Our hope is that the innovative research presented in this issue, conducted by pioneers of implementation science in our field, will facilitate future work that ultimately allows us to achieve a standard of practice for EBP delivery to the millions of families impacted by maltreatment each year.