Looking Forward: The Promise of Widespread Implementation of Parent Training Programs

Abstract

Over the past quarter century a body of parent training programs has been developed and validated as effective in reducing child behavior problems, but few of these have made their way into routine practice. This article describes the long and winding road of implementation as applied to children's mental health. Adopting Rogers' (1995) diffusion framework and Fixsen and colleagues' implementation framework (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005), we review more than a decade of research on the implementation of Parent Management Training – Oregon Model (PMTO^®). Data from US and international PMTO implementations are used to illustrate the payoffs and the challenges of making empirically supported interventions routine practice in the community. Technological advances that break down barriers to communication across distances, the availability of efficacious programs suitable for implementation, and the urgent need for high quality mental health care provide strong rationales for prioritizing attention to implementation. Over the next quarter of a century, the challenge is to reduce the prevalence of children's psychopathology by creating science-based delivery systems to reach families in need, everywhere.

Twenty-five years ago, parent training programs to address children's behavior problems were in the process of becoming established as empirically supported interventions¹. Now, in the second decade of the 21^st century, we have a substantial set of parent training empirically supported interventions for youth across developmental stages. With widespread implementation, these programs hold the promise of reducing the prevalence of child and adolescent behavior problems, maltreatment and related poor outcomes (Prinz, Sanders, Shapiro, Whitaker, & Lutzker, 2009). Unfortunately, these programs remain largely unavailable to families seeking help in community agencies. Learning how to install proven programs in community practice settings is the challenge of the next quarter century.

In this paper, our goal is to outline the implementation process for empirically supported interventions within community service systems and point to strategies to improve their uptake and sustained practice with model fidelity. We review the process of the implementation of one empirically supported parent training intervention and, by way of this case example, describe the first decade of work implementing Parent Management Training – Oregon Model (PMTO^®; Forgatch & Patterson, 2010; Patterson, 2005). PMTO is little more than a vehicle for discussing fundamental questions, such as “What are optimal conditions for, and barriers to, system change in the context of widespread implementation?” Implementation is a deliberate and dynamic process requiring extensive collaboration between two systems: the adopting community and the program developer/purveyor (Forgatch & DeGarmo, 2011; Herschell, McNeil, & McNeil, 2004; Proctor et al., 2009). The interaction between these systems unfolds over time in an orderly fashion involving many stages (Chamberlain et al., 2008; Fixsen et al., 2005, Rogers, 1995). Effective communication, interpersonal problem solving and positive relationships promote the success of the implementation as the systems move through the process of managing logistics, building infrastructure, and troubleshooting barriers. In its function as delivery system, the community selects, administers, and evaluates the program; the program developer serves as the purveyor of change. Together these systems adapt the intervention for the community's cultures and contexts while ensuring that the method is delivered with model fidelity. Levels of functioning within and across systems must be coordinated. The community system includes an executive function that initiates and administers the program, practitioners who deliver the program and families who receive it. Within the program system, levels include the implementation director who oversees the installation, trainers who teach community practitioners to deliver the program with fidelity, and staff and technology to support the transfer process. In successful implementations, all levels operate in harmony within and across systems.

In Figure 1, we present a two-system four-stage model for the installation of a psychosocial intervention in a wide-scale community system: preparation, early adoption, implementation, and sustainability. In Stage 1, before implementation begins, the community and program systems ready themselves for change. Rogers refers to this as the initiation phase; Fixsen and colleagues call it the preplanning and preparation phase (Fixsen et al., 2005; Rogers, 1995). Once the community selects a program, Stage 2 begins—early adoption. In Stage 3, the implementation is underway. In Stage 4, the emphasis is on sustainability.

Figure 1. A dynamic implementation process.

Implementing an Empirically Supported Intervention: A Case Study with PMTO

We employ three examples to illustrate the process of implementing PMTO. One is a nationwide implementation in Norway across two systems of care: child mental health and child welfare. One is a statewide program in the children's community mental health system in Michigan; the other is a statewide program in the child welfare system in Kansas. All three programs began with a failure of the system to provide effective treatment for families in need. In each case, the community selected PMTO to address the problem.

Stage 1: Preparation

For the adopting community, activities involve identifying a need, designing a plan, gathering resources, engaging leadership, and selecting a program that can address their goals. Before a program can be deemed empirically supported, the developer must establish program credentials by articulating and testing the intervention in well-controlled conditions, a process that requires years of work. Next, the strategies and tools to implement the program in real-world settings must be designed and set in operation.

The community system

Wide-scale change requires strong leadership. To undertake this effort, a person or cohesive group must step forward with a plan. The initiator must have a unique set of qualities that combines social/political capital, leadership skills, access to necessary resources, and commitment to see the project through. In other words, the charge is led by a 900-pound gorilla. Such leaders recognize the need for change and establish goals that generate support from all levels within the community. Not every implementation begins in the same way, but the PMTO experience has consistently been a top-down approach followed by bottom-up support that ultimately engages the middle level. Initially, the practitioners can be less than enthusiastic to meet the requirements of the implementation. It is when families communicate their excitement with the successes at home that practitioner attitudes change. These practitioners often report that they successfully use the PMTO strategies at home with their own families.

The call for change

In Norway in the late 1990s, problems were increasing for severely maladjusted youth, the media became involved, and an international expert conference was organized. As a result, the Ministry of Child and Family Affairs provided funding for a center tasked with implementing and evaluating evidence-based family treatment programs on a nationwide basis. PMTO was one of two programs selected for this purpose (Ogden, Forgatch, Askeland, Patterson, & Bullock, 2005).

In Michigan, the need for change was identified with a study evaluating youngsters treated in the community mental health system. Results indicated that although some benefits ensued from community treatment, youngsters with the most substantial problems at baseline remained in the clinical range at termination (Hodges, Xue, & Wotring, 2004). These families needed better treatment. Michigan, a state that has suffered more than most from the economic downturn, had few resources to instigate system change. The statewide director of community programs for severely emotionally disturbed children competed for and won funds from the National Institute of Mental Health to begin the implementation process with PMTO.

In Kansas, the problem concerned children with severe emotional disturbance (SED) in foster care. These youngsters make up approximately half the state's foster care population, and compared to non-SED children, they have more placements, fewer and slower exits to permanency, and are likelier to age-out of care. Reviews of Child and Family Services conducted in 2001 and 2007 indicated that the state was out of compliance with timely permanent reunification, timely adoption, and placement stability for SED youngsters in foster care. A particular concern specified the inadequate services provided for birth parents (Akin, Bryson, McDonald & Walker, 2012; Bryson, Akin, Blase, McDonald, & Walker, in press). A program was initiated through a five-year grant provided to the University of Kansas by the US Department of Health and Human Services, Administration for Children and Families (Children's Bureau Express, 2011). Kansas received one of six grants in a federal effort to test child-welfare plans and promote effective initiatives nationwide.

Thus, in each case, the initiation of wide-scale implementation of PMTO was fully or partially supported with government funds designed to increase the availability of empirically supported interventions for youngsters with serious behavior problems.

Selecting a program

The process through which communities select a program varies considerably. We describe the selection process using the Kansas Intensive Permanency Project (KIPP) as an example. The project's leadership comprised a management team based at Kansas University and a Steering Committee with representatives from the four independent agencies within the state's five regions that provide services for the foster care system. When experts from the funding team recommended that KIPP adopt an empirically supported parent training model, the KIPP team searched clearinghouse websites that chart the attributes and credentials of interventions with established support to facilitate a systematic comparison. The KIPP selection process required more than 200 hours over three months. The selection committee conducted telephone interviews with finalist programs and sought recommendations from outside experts. When they had narrowed the field down to two programs, the selection team held meetings to reach consensus. Their stated reasons for selecting PMTO were: fit with the needs of the target population, ability of the intervention to reduce long-term foster care, the system of fidelity measurement, and long term sustainability with anticipated systems changes (Bryson et al., in press). A deciding factor was the PMTO implementation goal to transfer the program to the community by supporting their development of a self-sustaining infrastructure to conduct continued program training and delivery with fidelity.

The program system

For an intervention to achieve status as empirically supported and be suitable for widespread implementation, it must follow a set of clearly defined steps that take years to accomplish (Chambless & Hollon, 1998; Flay et al., 2005;McHugh & Barlow, 2010). Requirements are based on rigorous experimental design, typically using randomized controlled trials with intent-to-treat analysis (i.e., once randomized always analyzed), valid and reliable assessment, lasting effects, and replication. Efficacious programs follow experimental testing under optimal conditions, careful supervision with select populations, conducted in academic or research settings. Programs advance to status as effective when tested experimentally under the “real world” conditions provided in community treatment centers. Successfully completing these steps can lead to credentials that enable a program to be listed by clearinghouses that identify empirically supported programs, describing the program's attributes and evaluating levels of excellence (e.g., Blueprints Project; National Registry of Evidence Based Programs and Practices/NREPP; California Evidence-Based Clearinghouse/CEBC, Top Tier). Such empirically supported intervention programs are suitable for wide-scale implementation. PMTO is prominent on such lists.

PMTO is theory based

The PMTO intervention is founded on social interaction learning theory, which emphasizes the influence of the social environment on behavioral outcomes (Forgatch, Patterson, DeGarmo, & Beldavs, 2009; Patterson, 2005). Although all parenting empirically supported interventions must follow the process of efficacy and effectiveness testing, not all such programs rest on a solid theoretical foundation. The PMTO intervention grew out of decades of work reflecting iterations among theory building, basic research, and intervention development focused on clearly specified social contexts and mechanisms presumed to account for etiology, growth and maintenance of youngsters' behavior problems (Forgatch & Patterson, 2010; Patterson, 1982; Patterson, Reid, & Dishion, 1992; Reid, Patterson, & Snyder, 2002). Two key mechanisms have been defined, both of which involve reinforcement theory. One mechanism, negative reinforcement, takes place primarily within the family context; the other mechanism, positive reinforcement, occurs in the peer context.

In families, coercive processes can begin as early as age two or three and generalize to a set of overt antisocial behaviors, including noncompliance, temper tantrums, and physical aggression. As development proceeds, increasing coercion and time spent outside the family may prompt youngsters' drift into deviant peer groups who advance the shaping process. Deviant peers provide positive reinforcement for covert behaviors, such as lying, stealing, and truancy (Dishion, Spracklen, Andrews, & Patterson, 1996; Patterson & Yoerger, 2002; Snyder et al., 2005). Thus, an effective intervention calls for changes within these two social contexts. So far, randomized controlled trials of PMTO support the intervention effect on parenting; PMTO has yet to develop an effective intervention to address the peer context other than to produce reductions in deviant peer association (Forgatch et al., 2009).

A unique aspect of PMTO relative to other parent training programs is an emphasis on clinical and teaching processes to reduce resistance to change. This focus grew out of research on therapy process with studies conducted in the mid-1980s using direct observation of therapy sessions (Patterson & Chamberlain, 1988; Patterson & Forgatch, 1985; Stoolmiller, Duncan, Bank, & Patterson, 1993). Findings from this body of work indicated that in addition to client characteristics (e.g., poverty, depression, antisocial qualities), therapist behavior can drive resistance. For example, when therapists either teach or confront, resistance increases, and when they combine confrontation with teaching, resistance increases sevenfold (Patterson & Forgatch, 1985). In response to these findings, we shaped the intervention to replace pedantic teaching with active teaching, such as role play and problem solving, and to strengthen supportive clinical processes. The PMTO fidelity measure integrates the extent to which therapists practice these processes as they deliver the core content of the program (Forgatch & DeGarmo, 2011; Forgatch, Patterson, & DeGarmo, 2005).

Testing PMTO

Parent training programs engage parents as the agents of change for their youngsters' behavior problems. In PMTO, parents learn strategies to increase positive practices and reduce coercion. Five parenting practices have been specified as core components: Positive reinforcement to promote prosocial behavior, effective limit setting to decrease deviant behavior, monitoring to ensure that behavior stays on track, family problem solving to provide skills to prevent and manage stress and conflict, and positive involvement to emphasize the importance of spending time together in pleasant activity.

To evaluate the efficacy of the program as well as the theoretical model underlying PMTO, experimental tests were conducted to assess outcomes and verify the hypothesized mechanisms (Forgatch & Patterson, 2010). In one key study, a randomized controlled trial with a sample of single mothers provided a clean test of the parenting model by offering PMTO intervention for the mothers but not the children (Forgatch, et al., 2009). Multiple agent and method assessments were conducted at regular intervals over the course of nine years; mediational modeling evaluated parenting practices and deviant peer association as mechanisms of change for the youngsters' outcomes. In the context of an experimental trial, mediational modeling can test a theoretical model by showing that intervention effects on targeted outcomes are brought about by intervention effects on the putative mediators (Baron & Kenny, 1986; Holmbeck, 1997). The data supported the model and yielded some unanticipated outcomes. Assignment to the experimental group produced the expected changes in the two social contexts: improvement in the five core parenting components and reduction in deviant peer association. These benefits to the hypothesized mechanisms of change mediated the intervention effects on the youth—significant reduction in teacher-reported delinquency, deviant peer association, police arrests, and delayed timing of first arrest. As parents learned to replace coercive practices with positive parenting, they effectively reduced their youngsters' deviant behavior while increasing their prosocial skills (Forgatch, Beldavs, Patterson, & DeGarmo, 2008; Forgatch et al., 2009; Patterson, Forgatch, & DeGarmo, 2010).

The study yielded four surprises: the order of change in parenting practices following intervention, direct effects on deviant peer association, cascading benefits to maternal adjustment, and increasing effect sizes. Although PMTO programs start by teaching positive parenting and follow that with limit setting to reduce coercion, the findings showed that changes took place in the reverse order. First there was a decline in coercion over 12 months; then growth in positive parenting followed in the subsequent two years. Incidentally, the reverse of this change pattern was not significant—that is, changes in positive parenting over 12 months did not predict later growth in coercive parenting. The second surprise was that the intervention had a direct effect on deviant peer association. Changes in this domain were not mediated by parenting effects. The third unforeseen outcome involved positive effects on maternal adjustment, which were mediated by improved parenting. Relative to control group mothers, those in the experimental condition showed a significant increase in standard of living (education, income, occupation) and fewer maternal police arrests over the nine-year follow up. Finally, intervention effects started small and increased in size over the nine years. In other words, the longer the time in follow-up, the greater the magnitude of the difference between experimental and control participants—intervention benefits were increasing. Such perturbations in testing intervention programs leave room for improving the theory, the basic research and the intervention itself. Nevertheless, it must be said that this experimental test supported the basic theory by showing the hypothesized mediators to serve as the engines of change in youngsters' outcomes.

For an intervention to be suitable for wide-scale implementation, it must be generalizable across multiple cultures, contexts and populations. Studies of adaptation for these purposes are underway and represent a new research area in which modifications are introduced systematically to address variation in families' cultural patterns and problems (Bernal, Jimenez-Chafey, & Domenech Rodríguez, 2009). For example, two PMTO studies, one in Detroit with non-English speaking Latinos (Parra-Cardona et al., 2012) and one in Norway with Pakistani and Somali refugee mothers (Bjørknes, Kjøbli, Manger, & Jakobsen, 2012), showed the intervention to be acceptable and effective. Other adaptations address the needs of single mothers residing in a domestic violence shelter (Gewirtz & Taylor, 2009), and military families experiencing deployment (Gewirtz, Pinna, Hanson, & Brockberg, in press). Given slightly more than a decade of experience, it has been possible to engage hard-to-reach families within a variety of cultures and contexts with high degrees of participation and satisfaction.

Stage 2: Early Adoption

The primary activities during this phase include developing a collaborative relationship and carrying out training. Because system change is vulnerable to external and internal forces, success is enhanced when the two systems cooperate to address a seemingly endless supply of practical challenges (Fixsen et al., 2005). Together the community and program systems must address the logistics of change with activities that include the following: negotiating a working contract, training practitioners, tailoring the program for relevant cultures and contexts, and addressing feasibility and acceptability issues. The two systems may struggle during this phase as they build and test their collaborative relationship, and many programs fail to make it beyond this point (Fixsen et al, 2005).

The program system

The activities for the program team are extensive during this phase as they transfer knowledge with training activities, technical assistance, and establishing fidelity.

Training

The PMTO training program, which requires approximately 12 to 18 months, is based on a set of manuals and materials, and includes workshops, practice with simulated and real cases, and extensive coaching (Forgatch & DeGarmo, 2011). Between workshops, trainees treat families referred to their agency and receive coaching based on video recordings of their practice, a procedure that significantly strengthens the transfer of learning from training to application (Salas & Cannon-Bowers, 2001). Individual and group coachingis provided live, by phone/video conferencing, or in written format. When trainees achieve a certain standard of proficiency, they are invited to treat two new certification cases, from which they submit four videos on required topics. Candidates are certified when they attain a specified passing score on each session.

The training team consists of mentors who themselves have achieved certification as PMTO specialists, coaches, and trainers. Mentors are also reliable fidelity raters who can score sessions to certify trainees. The majority of the current training team resides in Oregon, although mentors come from Norway, Iceland, The Netherlands, Mexico, Michigan, Utah, and Minnesota. Languages spoken include English, Norwegian, Danish, Icelandic, Spanish, and Dutch. Leader and parent materials are available in these languages as well.

Model fidelity

Fidelity to the content and process of an intervention must be carefully evaluated to ensure that the outcomes attained in controlled conditions can be replicated in the field (Fixsen et al., 2005). Method fidelity is achieved when the program is practiced in accordance with the theory and goals underlying the method (Dumas, Lynch, Laughlin, Smith, & Prinz, 2001; Perepletchikova, Treat, & Kazdin, 2007). The measures must provide reliable, unbiased, and valid assessment of method delivery. Two aspects of fidelity are particularly relevant: adherence and competent delivery (Dumas et al., 2001; McHugh, Murray, & Barlow, 2009; Perepletchikova et al., 2007; Waller, 2009; Waltz, Addis, Koerner, & Jacobson, 1993). Adherence assesses application of the program in terms of the content and methods spelled out in the treatment manuals. Ratings can be made by the practitioner, the intervention recipient, or objective observers who listen to or view session recordings. Competence, a more complex construct, is scored less frequently, perhaps because ratings must be made by skilled practitioners with inter-rater reliability. Currently, evidence for reliability and validity of fidelity measures is in scarce supply within the field of implementation research (Schoenwald & Garland, 2013).

Competent adherence to PMTO is assessed with the Fidelity of Implementation Rating System (FIMP: Knutson, Forgatch, Rains, & Sigmarsdóttir, 2009). FIMP ratings are made by reliable certified PMTO Specialists and assess the intervention as delivered with individual families in community or home settings and in parent groups. Ratings are based on direct observation of segments of sessions based on core parenting components and evaluate five theoretically-relevant categories (i.e., Knowledge, Structure, Teaching, Process Skills, and Overall Development). Procedures, definitions of core parenting practices, and the rating scale are described in a manual (Knutson et al., 2009).

FIMP was first validated within an efficacy trial with a prevention sample in the United States and tested again with a clinical sample in Norway. In both studies, FIMP scores predicted changes in observed parenting practices before and after intervention, as hypothesized (Forgatch et al., 2005; Forgatch & DeGarmo, 2011). Higher fidelity scores yielded significantly greater improvement in parenting. In each study, findings were collected from two settings (intervention sessions and parent-child interactions); fidelity was observed during intervention; and parenting practices were observed before and after intervention from parent-child interaction tasks. In one Norwegian study, high fidelity also predicted greater reports of satisfaction by parents (Ogden, Amlund Hagen, Askeland, & Christensen, 2009). In another Norwegian study, fidelity scores collected three times during intervention predicted significant reduction in parent reports of their children's behavior problems (Hukkelberg& Ogden, 2013). Most importantly, the fidelity findings generalized from an efficacy prevention trial to a large-scale clinical trial in two cultures and two systems of care. These studies indicate that the FIMP measure is a valid and generalizable tool that can be employed in wide-scale implementation research.

Implementation outcomes

An effective training program must produce community practitioners who demonstrate competent adherence to the method. An important measure of implementation evaluates growth in fidelity over the course of training: clinicians starting with low scores should increase; those with high scores at the start should not decline. This question was evaluated within the PMTO training program in Norway using the FIMP measure. Fidelity was assessed early, mid, and late in the training program with the hypothesis that competent adherence would increase steadily and variability in performance would decrease. As predicted, trainees' performance improved and became significantly more homogeneous at certification (Forgatch & DeGarmo, 2011). During the early and midpoints of training, there was considerable variation in performance across the full cohort. At certification, however, all trainees were performing within a relatively narrow range of excellence.

Another test of a training program assesses the percentage of candidates who complete training with certification and subsequently conduct community practice with method fidelity (Durlak & DuPre, 2008; Proctor et al., 2011) In Norway, 83% of the 36 trainees who began training in 1999 were certified; in Michigan, 89% of the 19 who began in 2006 achieved certification The question is: how long do practitioners maintain their credentials and continue to practice with certification? Eight years after certification, 91% of the Norwegians were still certified and providing treatment to families. In Michigan, 16 of the 17 (94%) completers were certified seven years later and still practicing in community service. It is too early to report on the Kansas project as they are still in training.

The community system

The community supports the early adoption phase of implementation at the executive level by selecting participant agencies and candidate providers to be trained. It provides training and treatment facilities, sufficient appropriate treatment cases, computers for trainees with high speed internet access, child care for parents during treatment, and time for trainees to learn the program.

At the practitioner level, trainee activities include attendance at workshops and coaching, studying manuals and other informational material, conducting simulated and actual family sessions, making video recordings, reviewing sessions and feedback from coaches, and data management activities (completing case and session information forms, entering information into the database, uploading video recordings to the secure website). Trainees are required to attend 85% of workshops, video all sessions, upload videos and data to the database, see a minimum of training and certification families, submit video recordings of their best work, and receive passing fidelity scores on their certification sessions. They must do this in a timely fashion.

Challenges and solutions during early adoption

Things can go wrong at any stage. The next section describes some issues that emerged during the Norwegian, Michigan, and Kansas implementations and strategies that were used to address them.

Leaders must promote enthusiasm for the new program throughout the community at every level—executive, practitioners and their supervisors, and the families who are seeking help. In Michigan, Jim Wotring (then the director of community mental health programs for SED children) effectively prepared the way by holding a series of stakeholder meetings to build buy-in from state and agency leaders and families. This resulted in one agency becoming an early adopter, using their own resources to initiate the PMTO training. Wisely, the state sent two clinicians from other agencies to participate in this early start, one of whom emerged to become the statewide PMTO coordinator. Wotring also organized a parent advocate committee with meetings to introduce the idea of improving family services by providing programs that work.

Challenges at the executive level often involved logistics. For example, when the standard approach to children's behavior problems is direct treatment with the child, parents sit in the waiting room and child care is not needed. In parent training approaches, clinicians work directly with parents to shape effective parenting skills. Now who watches the children? Childcare arrangements had to be made at the treatment centers. In Michigan, another barrier involved policies that required that the child receive direct services in order for agencies to receive third-party payments. This policy had to be changed, which required political capital and cooperation at the highest levels. At the agency level, when offered an opportunity to enroll clinicians in the intensive training program, leaders sometimes made poor decisions, sending problem employees on the verge of being fired, or staff close to retirement.

One of the serious potential problems involves ensuring sufficient referrals of appropriate cases for the trainees, a major problem that has disrupted other large-scale implementations (Asmussen, Matthews, Weizel, Bebiroglu, & Scott, 2012). Although there may be enough appropriate cases, referring clinicians have to be trained in the selection criteria and given practice in pitching the program to parents. Parents are less than enthusiastic when told: We have an experimental new program in which we treat parents instead of children to solve the children's problems. We don't know if it works. Would you like to try it? Oh by the way, they videotape everything.

It was the middle level stakeholders, the clinicians and their supervisors, who tended to present the greatest resistance to the change. Some questions reflecting the common attitude were raised in Michigan and in Norway: “What can a program from Oregon contribute to the problems we're having in [name the place]?” “Where is Oregon anyway—is that near California?” “Rewarding children for what they ought to do in the first place will weaken their character.” “We don't like behaviorism; our children are not monkeys!” “If you can measure it, it's not meaningful.” “Manualized programs are rigid.” “Norwegians don't believe in punishment—in fact we have outlawed it in this country!” We had to reframe their questions and apply the same principles that were effective in work with resistant parents.

Many professionals felt their skills were being called into question and some were concerned they would have trouble learning the new techniques. Most were uncomfortable having all their work video recorded and reviewed by others. Trainers created a supportive and safe environment for learning and practicing the new skills with an emphasis on recognizing the strengths already present in practitioners' repertoires. Teaching strategies emphasized role play and problem solving that engaged the group in designing ways to address the clinical and contextual issues in their community. Practitioners reported that they found training procedures to be active and even fun.

Stage 3: Implementation

During the full implementation stage, community agencies provide the program to its consumers, strengthen their infrastructure and the procedures needed to ensure effective delivery, expand their reach and services, and evaluate both treatment outcomes and the implementation itself (Chamberlain et al., 2008; Fixsen et al., 2005; Rogers, 1995). The program developer supports these activities by continuing to provide technical assistance to strengthen the expertise of community leaders and practitioners. By this stage, the collaborative relationship should be strong enough for systems to mentor each other to ensure that services can expand yet sustain model fidelity. Processes begun in Stage 2 continue, such as adapting the program to ensure acceptability, feasibility and suitability for a given community. Evaluation activities emphasize process and outcome variables in terms of both treatment and implementation.

During Stage 3, the separate roles of the two systems become more blended in their functions. The challenge for this collaborating team is to negotiate the tension between sustaining method fidelity and adjusting the program for specific demands. Each system must maintain effective communication with the other and within its own levels of operation. On the community side, it is important to strengthen infrastructure to maintain and grow the program. On the program side, staff continue to deliver technical assistance to help the community maintain motivation and commitment, improve skills, and support problem solving efforts (Durlak & DuPre, 2008). Careful monitoring systems enable both community and program provider to identify and troubleshoot barriers. Technology facilitates every aspect of this phase of implementation.

Challenges and antidotes

Evaluating the success of an implementation is distinct from assessing clinical treatment outcomes. One essential variable in the assessment of implementation outcomes is “penetration,” which is defined as the integration of a practice within a given system (Proctor, et al., 2011). Penetration is concerned with two questions: Once clinicians are trained in an empirically supported intervention, do they actually apply it with fidelity; and is the intervention widely available in the community? Unfortunately, there are some dramatic examples of failure with respect to penetration. For example, a recent study in the UK evaluated a large-scale program in which more than 3000 practitioners were trained in 10 evidence-based parent training programs (Asmussen, et al., 2012). Although the practitioners gave the training program high ratings, only 42% of them delivered even a single session of the intervention for which they were trained within the next 6 months. What happened? The project report identified barriers that included the following: lack of time; lack of confidence; lack of funding; change in jobs; too few trained practitioners in the agency; lack of support from agency managers and supervisors, and difficulty recruiting and retaining parents. Finally, funding priorities changed and resources dried up.

These problems were not barriers for the Norwegian implementation. What contributed to their success? In a recent report, several factors were identified:

(a) genuine interest and commitment at the political and administrative level for national implementation of evidence-based programs, (b) increased interest among practitioners for evidence-based practice, (c) establishing a self-sustaining national center for implementation and research, (d) ability of the program developers to support the Norwegian implementation and research efforts, and (e) positive feedback from families and positive media coverage (Ogden et al., 2009, p. 590).

Some have said that Norway's success is easy to explain—they have an abundance of resources and a well-developed social welfare system. If this is the answer, a financially strapped state like Michigan would have little chance to carry out and sustain a successful implementation. So far, however, Michigan is doing well. In 2013, there are 35 agencies throughout the state with PMTO practitioners, representing 76% of the agencies serving SED children in the state community mental health system. There are 83 certified PMTO practitioners and 97 in training who provide services for families on an individual basis. Parent groups are provided by 27 practitioners. Families are seen in community treatment centers or in their homes, depending on local priorities.

During Stage 3, preparation begins for conducting the implementation in dependently. With input from Michigan's department of community mental health and the program developer, a document was produced describing the roles and responsibilities for all participating in the Michigan PMTO system, including agencies, trainers, fidelity raters, coaches, clinicians, consultants, regional coordinators, and the state coordinator (Gray, Rains, & Forgatch, 2009). This document, which required months of negotiation to produce, incorporated lessons learned throughout the implementation and now serves as the template for other implementations as they enter the stage of full transfer. Currently in Michigan, there are 20 trainers, 30 coaches, and 13 reliable fidelity raters.

How does this state with few financial resources maintain a continued commitment to carry out this work? Looking at the contributors to success listed by the Norwegians, above, all factors are true for the state of Michigan with the exception of the establishment of a self-sustaining center for implementation and research. The commitment and enthusiasm of leadership, practitioners, and families continues to build momentum. It is not only the well-resourced who can follow this path.

Stage 4: Sustainability

During this phase, the community must integrate the program within its system, maintain buy-in and support from stakeholders and solidify its infrastructure to grow the program with increasing numbers of well-trained practitioners and satisfied consumers. The job for the program developer's team is to continue to advise, troubleshoot, and support the community's efforts. Evaluation becomes integral to operations to ensure that the program is delivered with model fidelity and positive treatment outcomes and the transfer of expertise continues. Fixsen and colleagues describe this phase as full operation, “with full staffing complements, full client loads, and all of the realities of ‘doing business’” (Fixsen et al., 2005, p. 16). According to Fixsen and colleagues, this level of implementation is seldom if ever studied because full operation is rarely accomplished and most research for psychosocial interventions takes place in the earlier phases. Rogers refers to this as the confirmation or routinizing phase, which involves integrating the program into standard practice. Activities include establishing a well-articulated infrastructure within the community to support the continuing practice with fidelity, and the training, coaching, and certification of new practitioners. Assessment of fidelity and treatment outcomes continues. Team work between the two systems facilitates the transfer of the program to community responsibility. In Rogers' framework, the program becomes an ongoing element in the organization and “loses its identity” (Rogers, 1995, p. 392).

The program system

Training, retraining, and extending reach to new populations is a never ending struggle. However, when training and evaluation is transferred into the hands of the community, it is possible to provide the necessary infrastructure to regenerate staffing and procedures within the adopting system. PMTO implementations are designed for full transfer to the community, a dimension appreciated by the Norway, Michigan and Kansas implementing communities. The program team trains a progenitor group in the community, who are then trained to carry the program forward to future generations. To provide long-term support and practice throughout the site, the community establishes an effective infrastructure. This approach involves more than training-the-trainer. The community gradually assumes full responsibility for all activities involved in practice: training, coaching, certification, and continuous monitoring of fidelity and outcomes (Forgatch & DeGarmo, 2011).

The full transfer approach begs the question: can fidelity be sustained after the program developer's team with draws? This question was addressed with data from two nationwide PMTO implementations: Norway and Iceland. Fidelity scores (assessed by FIMP) at certification for practitioners trained by the program team were compared with scores for those trained by community trainers. In each instance, FIMP scores were provided by reliable raters for three generations. Generation 1 (G1) was trained by the program developer; Generation 2 (G2) was trained by selected certified G1 PMTO Specialists; and Generation 3 (G3) was trained by G1 and G2 trainers. The hypothesis was that fidelity would decrease with each generation.

In Norway, a small yet significant decline in fidelity did indeed follow the transfer to local trainers in G2. However, G3 scores were equivalent to those attained by G1. Thus, the hypothesis was not fully supported (Forgatch & DeGarmo, 2011). Fidelity recovered by G3. These Norwegian findings were replicated in Iceland. Again, there was a small but significant decline in fidelity at certification following the transfer of training to local trainers in G2, but the G3 trainees attained scores equivalent to those for G1 (Sigmarsdóttir & Guðmundsdóttir, 2012). In Iceland as in Norway, there was recovery by G3.

The temporary slippage in certification scores with recovery was replicated in two studies. What this means remains a mystery. In each case, the investigators speculated that training may have faltered as the local team developed the additional skills required to lead workshops and conduct regular coaching sessions with clinicians. In each case, the materials and procedures had to be translated and adapted for the trainers, practitioners, and parents. By the third generation, this process was complete, and the material had been assimilated by the trainers. In recent implementations, the PMTO program purveyors provide specific training and support as the G1 trainers begin the process of training new generations.

The data from these two nationwide implementations indicate that PMTO can be transferred successfully with sustained fidelity and cross-cultural generalization. A next question asks whether community practice will produce the same positive outcomes attained within effectiveness trials. This question has been examined in randomized controlled trials in Norway and in Iceland and is underway in The Netherlands. In each country, community practitioners from G1, G2, and G3 provided treatment to families in service agencies throughout their respective countries and compared results with families randomly assigned to receive services as usual. Multiple-method assessment and intent-to-treat analysis tested change in pre/post child outcomes. Findings for families receiving PMTO were significantly superior to those for families receiving services as usual (Ogden & Amlund Hagen, 2008; Sigmarsdóttir, Thorlacius, Guðmundsdóttir, DeGarmo, & Forgatch, in press).

PMTO implementations have harnessed the power of strong collaborations between the ivory tower where programs are developed and adoption communities. In every case, the implementation has been driven by key leaders in the communities, with attention paid to increasing consensus from front-line workers, management, and families. While the developers held tight to the program's core principles and observational methodology, the community modified the program's topography, created and maintained the organizational infrastructures necessary for implementation, and designed the policy and procedural elements required for successful delivery (Sigmarsdóttir & Guðmundsdóttir, 2012). The result can lead to empirically supported programs that are deployed with success in the community, using methods (e.g., videotaping of sessions and fidelity assessment based on observation) that some have claimed are unlikely to be established in routine community practice.

Technology is Key to Widespread Implementation with Fidelity

Technology has revolutionized communication and made direct observation possible for routine clinical practice. In 1988, use of the internet was difficult and accessed by few; in 2013, seven-year-old children skillfully navigate the internet and over 80% of U.S. households have internet access (Morales, 2013). This technological leap has advanced efforts to span the chasm from efficacy to large-scale implementation, providing support for communication, monitoring, training, coaching, certification, and fidelity. In these last 25 years, our implementation toolkit for parent training programs has expanded to contain: an extensive portfolio with effective parent training programs; standardized measures of treatment outcomes and method fidelity; efficient websites, sophisticated software programs, and centralized databases (Glasgow, Lichtenstein, & Marcus, 2003; Kazdin, 2008; Kazdin & Blase, 2011; McHugh et al., 2009). The combinations and permutations with which these tools can be applied are limitless.

Fidelity Uses Technology

A common excuse for scrimping on rigorous assessment of fidelity is that it is too costly/effortful/impractical (Perepletchikova et al., 2007, Proctor et al., 2009). PMTO implementations require regular assessment of fidelity using the FIMP measure within and across sites to prevent drift in PMTO practice and in the reliability of the assessment. To synchronize this level of performance, a secure database was established to be used by all PMTO sites with the goal of sustained competent adherence to the method based on direct observation. Each site establishes a FIMP team leader who conducts training, retraining, and regular reliability checks of their raters. FIMP leaders from around the world upload, view, and score videos of intervention sessions with members of their own teams using FIMP Central, a secure website whose purpose is to monitor practice and prevent drift. An orchestra conductor can use a tuning fork to ensure that the instruments are in tune with each other. Similarly, the centralized PMTO website is used to tune fidelity to the model throughout and across implementation sites. Are all the practitioners playing in the same key and with a reasonable level of competence?

Given advances in technology, we are able to address a number of implementation questions in the second decade of the new millennium.

• Does fidelity predict process and outcomes as expected?

• Can fidelity be scored reliably by raters of differing cultures?

• Are trainees completing training requirements?

• Are trainees ready to apply for certification?

• Are coaches/supervisors performing at maximal levels?

• Do practitioners continue to perform at certification levels during regular practice?

• Within or across cohorts, is there fidelity decay?

• Are fidelity standards at certification and during practice maintained across implementations/settings/cultures?

Technology, Implementation, and the Next 25 Years

Large-scale implementations have quietly waited in the wings for solutions to the complex problems of moving empirically supported interventions from well-controlled settings into the community. Technological progress has enabled the worldwide web and its tools to support this effort. Computing programs promote effective systems for data management, data storage and retrieval, data reduction, and data analysis. Investigators now have tools to record and monitor detailed information, such as completion of training activities, engagement in practice, and competent adherence to treatment models. HIPAA-approved secure web-based systems support video streaming and allow program participants to record activities during training and following program installation. All that is required is access to computer hardware and related technology, high-speed internet connections and an effective database that connects activities and participants across time, distance and culture. Information stored on the database can include current case and session information for each trainee, video/audio recordings of intervention sessions, assessment information, and all other relevant data. The information, which is securely stored, can be accessed with proper credentials. PMTO implementations employ this approach in several countries (e.g., Iceland, The Netherlands, Denmark, and Mexico) and several locations within the United States (e.g., Michigan, New York City, Minnesota, and Kansas). They are part of the ongoing process of implementation and can be applied to study public health programs. These are not someday developments; they are here and now.

Technology may be a necessary but insufficient requirement for successful widespread implementation, however. Little knowledge exists regarding the strategies for partnerships between researchers/developers and community service settings that promote successful practice of parent training empirically supported interventions. What policy changes are required in order to successfully install an empirically supported intervention? What protocol and procedural changes are required to make delivery routine across multiple clinics and other service entities within a given jurisdiction? When the service system breaks down, or sustainability is set back by changes in policy, how can we continue to monitor the processes and outcomes of our efforts?

Conclusions

Our approach to service provision is too limited in scope. We must pool our information about the lessons learned from the evolution of parent training programs in small-scale research to large-scale implementation. There's an elephant in the room. The elephant represents the pressing need to make effective programs available to every family in need. Each developer has approached a different facet of the elephant and is diligently working to solve this puzzle. Can we combine the puzzle pieces, capture the key components, and build a complete picture? Can we create an environment of collaboration, rather than continuing our policy of setting up competitions? When we reach the point on the horizon where efficacy and implementation merge, we may be able to achieve a public health approach to reducing the prevalence of children's behavior problems with the empirically supported interventions already developed but too seldom applied in community settings. We have completed the horse race epoch in which we established programs as evidence-based through careful assessment, sophisticated modeling, and replicated randomized controlled trials. In the 21^st century, we must learn how to create science-based delivery systems that reach families in need – everywhere.