The Evolution of Evidence-based Practices

Over the past 30 years, behavioral scientists have identified numerous evidence-based treatment and prevention programs, policies, and other practices (e.g., Biglan et al., 2004). Societies have the potential to help achieve a significant reduction in the prevalence of the most common and costly behavioral and psychological problems that develop in childhood and adolescence—if they can implement evidence-based practices widely and effectively. Putting it more positively, we could assure that many more young people reach adulthood with the skills and interests needed to become productive and caring adults who contribute to the wellbeing of others.

However, there is no guarantee that existing knowledge will result in widespread benefit. We know far more about how to affect young people than we know about how to influence the social systems that will have to adopt evidence-based practices if the fruits of recent discoveries are going to be realized.

This paper addresses one facet of the effort to translate growing behavioral science knowledge into concrete benefits - the evidence-based practices movement. We describe recent developments in this effort and present as a case study the experience of Norway in implementing evidence-based practices.

Case studies have a venerable history in the development of effective treatments. One of the seminal events in the development of behavior therapy was Joseph Wolpe’s treatment of a series of phobic clients with systematic desensitization. Careful documentation of effects with individual cases lays the groundwork for more systematic experimental evaluation (e.g., Paul, 1967). Case studies are especially valuable when the units under study are not individuals, but treatment agencies or even whole countries and research is therefore very expensive.

The Evidence-based Practices Movement

A Growing Body of Efficacious and Effective Interventions

The evidence-based practice movement consists of a variety of efforts to bring into widespread use, programs, policies, and other practices already shown by experimental evaluation to affect psychological or behavioral functioning. Beginning in the 1960s, psychologists and educational researchers began to report treatment procedures that were demonstrably more effective than the procedures with which they were compared. Gordon Paul conducted the first randomized trial of Wolpe’s systematic desensitization. Wesley Becker (Becker, Madsen, & Arnold, 1967; Madsen, Becker, &Thomas, 1968) showed in a time-series design that teachers’ use of rules, praise, and ignoring led to significantly better classroom cooperation. In the ensuing years, researchers reported on interventions that were beneficial for virtually every major psychological and behavioral problem of humans, from bedwetting to cigarette smoking to depression to marital discord (Kazdin, 1978).

Space obviously precludes a thorough review of this evidence. Biglan et al. (2004) review programs and policies targeting children or adolescents and their families that have been shown by experimental evaluations to prevent or ameliorate multiple problems in adolescence, including antisocial behavior; tobacco, alcohol, and other drug use; risky sexual behavior; academic failure; and dropping out of school. Programs to prevent depression among adolescents have shown promise (Clarke, Rohde, Lewinsohn, Hops, & Seeley, 1999; Horowitz & Garber, 2006). In addition, there has been progress on the prevention of schizophrenic episodes (Pilling et al., 2002a; 2002b; Tsuang, Stone, & Faraone, 2002).

Although much attention has been paid to treatment and prevention programs, we note two other types of valuable interventions. Policies may affect the prevalence of a behavior without any direct intervention with individuals. For example, increasing the taxes on cigarettes and alcohol have been shown in numerous interrupted time series evaluations not only to reduce consumption of these substances in general but also to prevent young people from using them (Biglan et al., 2004). Studies of alcohol taxation show that taxation can reduce binge drinking, drinking and driving, and even developing cirrhosis of the liver (Markowitz, 2000; Saffer & Grossman, 1987a; 1987b).

In addition, numerous simple behavior-influence strategies, such as timeout, praise notes, and social commitment, have shown in experiments to affect targeted behaviors (Embry, 2004; Embry & Biglan, under review). These so-called “kernels” of behavior influence have the potential for others to disseminate them readily throughout society, thereby making most people more effective and less coercive in their efforts to influence other people’s behavior.

Meta-analyses

As the number of empirical studies has mounted, meta-analyses have begun to proliferate. Meta-analyses involve statistical analysis of all of the studies done on a particular topic. In the case of interventions, meta-analyses gather all of the studies done on a particular intervention and analyze the average size of the effect of the interventions, typically by averaging the difference between intervention and control conditions on outcomes of interest as a proportion of the pooled standard deviation of the two conditions (MacLeod & Nelson, 2000). There are meta-analyses of the impact of interventions for depression (Horowitz & Garber, 2006), substance use (Faggiano et al., 2005; Gottfredson & Wilson, 2003; Tobler et al., 2000; Tobler & Stratton, 1997); and antisocial behavior (Beelmann & Losel, 2006; Wilson, Lipsey, & Noser, 2007). There are also meta-analyses of interventions for diverse problem behaviors (Wilson, Gottfredson, & Najaka, 2001), parent training programs (Coren, Barlow, & Stewart-Brown, 2003; Serketich & Dumas, 1996), child abuse (Lundahl, Nimer, & Parsons, 2006); and ADHD (DuPaul & Eckert, 1997). Our list is far from exhaustive. Most report significant benefits of one or more types of interventions. Interventions involving behavioral procedures are most consistently reported to have benefits (e.g., Beelmann & Losel, 2006; Lundahl et al., 2006; Wilson et al., 2007).

Efforts to Identify and Disseminate Effective Interventions in the United States

Agencies Identifying and Disseminating Interventions

Recognition of the growing empirical literature prompted a number of federal agencies to identify and promote evidence-based practices. One such effort is the Blueprints Project, which the Center for the Study and Prevention of Violence at the University of Colorado initiated to identify interventions that could reduce violence, delinquency, or drug abuse. After its initial start, it received funding from the Office of Juvenile Justice and Delinquency Prevention in the U.S. Department of Justice (Elliot & Mihalic, 2004). The standards for designating a program as worthy of dissemination include, “…a strong research design with evidence of a deterrent effect on violence, delinquency, or substance abuse; sustained effects at least one year beyond treatment; and replication with demonstrated effects at more than one site” (Elliot & Mihalic, 2004, p. 47). The project defines “strong research designs” as those that employ random assignment of cases to conditions or studies employing quasi-experimental designs with matched control groups. The project also acknowledges that randomized trials are preferable to other types (Center for the Study and Prevention of Violence, 2008).

The Blueprints Project is the most extensive effort to date to disseminate evidence-based practices widely and effectively. It has assisted applicant communities in implementing one or more of the 11 Blueprints model programs, which have been effective in reducing adolescent violent crime, aggression, delinquency, and substance abuse. The Center has identified another 18 programs as promising. To date, the Center has reviewed more than 600 programs, and it continues to look for programs that meet its selection criteria. On its website, the Center’s current Matrix of Prevention Programs lists nearly 300 programs. Each program in the matrix receives one of several possible designations (effective, exemplary, favorable, model, promising, or reviewed only), based on criteria of the various reviewing agencies or researchers (see http://www.colorado.edu/cspv/blueprints/model/criteria.html).

The dissemination process involves selection of sites with the requisite resources and commitment to implement a program, training of implementation staff, technical assistance to those implementing the program, monitoring of implementation fidelity, and follow-up assessments to determine the degree to which programs continue as originally created. Elliott and Mihalic (2004) reported that, by the end of the second year of the effort, 74% of sites had implemented all components of the programs they adopted.

However, the implementations were less likely to deliver the programs at the dosage level their designs required. Thirty-five of 42 sites that implemented violence reduction programs were still implementing them six months after Blueprints funding ended.

Mihalic and Irwin (2003) analyzed predictors of implementation success. They used regression analyses to identify the most important of these factors. They found that the quality of technical assistance, ideal program characteristics, consistent staffing, and community support were important influences on one or more measures of implementation success.

The Substance Abuse and Mental Health Services Administration (SAMHSA) of the U.S. Department of Health and Human Services has also identified evidence-based practices. They have created the National Registry of Effective Prevention Programs (NREPP). Their searchable database (http://www.nrepp.samhsa.gov/index.htm) allows a user to specify the problems with which they are concerned, the age range and ethnicity of the target population, the setting, and the quality of the empirical evidence. Developers of programs can register their programs with NREPP by submitting evidence about them. NREPP does not restrict the programs it accepts to experimentally evaluated ones. Nor does its system ensure that the programs with the strongest experimental evidence get the highest ratings. For example, a search for programs targeting adolescents with multiple problems identified one program, Chestnut Health Systems—Bloomington Adolescent Outpatient (OP) and Intensive Outpatient (IOP) Treatment Model, which received a rating of 3.7 on a 4.0 scale on the quality of the study design, yet had only one quasi experimental evaluation. In contrast, the Family Behavior Therapy program, which underwent evaluation in multiple randomized trials, received a score of only 2.9.

The National Institute on Drug Abuse created the Redbook to summarize current knowledge on the prevention of drug abuse (Robertson, David, & Rao, 2003). The Redbook presents a set of principles of prevention, a guide to planning community adoption of drug-abuse prevention programs, and numerous examples of empirically evaluated prevention programs. To be included, a program does not need to undergo evaluation through a randomized trial.

In addition to funding the Blueprints Project, the Office of Juvenile Justice & Delinquency Prevention (OJJDP, 1999) collaborates with SAMHSA to identify “Effective Family Programs for Prevention of Delinquency.” The project continues today with its website identifying effective interventions (see http://www.strengtheningfamilies.org/html/index.html).

The Campbell Collaboration is the creation of a network of behavioral scientists who wanted to enable systematic reviews of the empirical evidence relevant to education, crime and justice, and social welfare. It collects studies for its database, organizes reviews of the evidence, and publishes this information online (http://www.campbellcollaboration.org). A mark of its reviews is the very careful attention to the methodological characteristics of the research. The Collaboration is a worldwide effort funded, in part, by the Danish and Norwegian governments and open to input and criticism from investigators and policymakers. It continues to update its reviews as new evidence becomes available.

Other efforts to identify evidence-based practices include the Guide for Community Preventive Services of the Centers for Disease Control and Prevention (Carande-Kulis et al., 2000), the U.S. Department of Education’s Safe and Drug-Free Schools Program (1998), and the What Works Clearinghouse of the Institute of Education Science (Coalition for Evidence-based Policy, 2003; U.S. Department of Education, 2003). In short, both public and private institutions are devoting considerable resources and time toward identifying evidence-based practices and disseminating them to relevant organizations.

Standards of Evidence

Many behavioral scientists have been concerned about the standards used to identify interventions as evidence based. For example, the most recently promulgated standards from SAMHSA include the following criterion:

Guideline 3: The intervention is judged by informed experts to be effective (i.e., reflects and documents consensus among informed experts based on their knowledge that combines theory, research, and practice experience). “Informed experts” may include key community prevention leaders, and elders or other respected leaders within indigenous cultures.

U.S. Department of Health & Human Services (USDHHS), 2001

This criterion effectively means that programs for which there is no experimental evidence would count as evidence-based. For example, as Zili Sloboda, the President of the Society for Prevention Research (SPR), pointed out in a letter she sent to SAMHSA, the Scared Straight program would be evidence-based:

Using this definition, the Scared Straight Program, a program initially endorsed and supported by well intentioned influential leaders which is based in deterrence theory which has, itself, been supported in some intervention studies, is an ‘evidence based’ program. In fact, in well-controlled experimental trials, the Scared Straight program has not shown desired effects in reducing delinquency among young offenders, but has, instead, been followed by increased delinquency among those exposed to the program compared to controls.

A review of government efforts to identify evidence-based programs shows that a variety of standards is in use. For this reason, the SPR created a committee to articulate scientific standards for designating an intervention as “efficacious” or “effective.” Flay et al. (2005) describe these standards in detail. Briefly, the standards designate two levels of validated interventions. Efficacious interventions are those shown by experimental evaluations to have a significant impact, where the investigators had tight control over the implementation of the intervention and may have had more than the usual level of resources for conducting the intervention. In contrast, effective interventions are ones shown to affect specified outcomes in experimental evaluations conducted under real world conditions. As an example of this distinction in school-based prevention research, an evaluation in which the research team taught a curriculum in the classroom would be classed as an efficacy trial, while if regular classroom teachers taught the program it would be labeled an effectiveness trial.

The standards require that statements about efficacy or effectiveness be restricted to specific measured outcomes, using reliable and valid measures. There must also be evidence that effects continued at least six months after the intervention.

Perhaps the most important standard involves the experimental design of the evaluation. The standards indicate there must be a control condition that does not receive the intervention under evaluation. This could be a condition receiving no intervention or receiving usual care or an alternative intervention. The most common form of experimental design is a randomized controlled trial in which cases (individuals, classrooms, schools, or communities) are assigned at random to two or more conditions. Such designs allow for inferences about the generalizability of the effects of the intervention across cases.

The committee also recognized interrupted time series designs as valid means of evaluating interventions. In an interrupted time series design, the control condition could be a period of time in which the group normally receiving the intervention does not get it. The standards specify that the strongest way to ensure that an evaluation is unbiased is to assign cases randomly to the intervention or control condition. The committee applied this standard to interrupted time series designs as well. In the case of an ABA design, a researcher should randomly choose the point in time at which the intervention was implemented or withdrawn. In the case of a multiple baseline design, the order in which cases receive the intervention would be determined at random.

However, the committee noted, “For some kinds of large-scale interventions (e.g., policy interventions, whole-state interventions) where randomization is not practical or possible, repeated time-series designs without randomization can be convincing—given large effects and long baselines (Biglan et al., 2000)” (Flay et al., 2005, p. 158).

The committee also provides standards for the statistical analysis of evaluations, assessment of the generalizability of results, the practical or public health significance of the results, and the duration of effects. In addition to providing unbiased standards for identifying interventions most likely to yield benefits when a group adopts and implements them with fidelity, the standards provide a helpful guide in the design and conduct of any further evaluations of interventions.

The American Psychological Association has also addressed the issues involved in evidence-based practices. The APA created its Presidential Task Force on Evidence-based Practice (APA, 2006). The report defines evidence-based practice in psychology as “the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (APA, p. 273). Unfortunately, their approach leaves enormous latitude for practitioners to argue that they have based a decision not to employ an evidence-based practice on their clinical expertise. Yet it provides no operational definition of clinical expertise. It is questionable whether the task force report will foster the dissemination and adoption of evidence-based practices.

The Need for Ongoing Surveillance of Effects

The evolution of evidence-based practices must include development of surveillance systems to monitor the wellbeing of populations over time. There are at least two reasons for this. First, most of the focus of the evidence-based practices movement has been on identifying empirically supported interventions. However, an implicit assumption of this focus is that, once implemented with fidelity, an evidence-based program’s benefits will continue on an ongoing basis. This is an unrealistic assumption. In any human system, one can expect drift in practice and changing conditions that may result in deterioration in implementation fidelity or loss of effectiveness due to changes in the environment or in the target population. Failure to continue to monitor the effects of our practices would be like an auto manufacturer abandoning quality control procedures once it achieved a high level of quality.

Second, the ultimate benefit of interventions is in their effect on the incidence and prevalence of problems in the targeted population. Even if practitioners implement programs with fidelity and maintain their effectiveness for the individuals receiving them, we cannot assume an effect on the incidence and prevalence of problems. The RE-AIM model of Glasgow, Vogt, and Boles (1999) illustrates this point. According to this model, the public health impact of an intervention is a function of the reach of the intervention times its efficacy and is dependent on its adoption, implementation, and maintenance. Reach is a matter of the proportion of the target population the intervention affects. Notice that a very efficacious intervention will have a small public health impact if it reaches few people. Moreover, an intervention with a tiny effect on most individuals can have a significant public health benefit if it reaches many people. A focus on reach x efficacy is an important antidote to the tendency of many psychologists to focus on the efficacy of interventions, but to pay no attention to how many people receive those interventions. That focus was appropriate as long as there were few efficacious interventions, but given the accumulating wealth of interventions, it is time to turn our attention to affecting the prevalence of problems in populations.

Both the expectation that intervention effects may diminish over time and the need to focus on population impact of interventions demand the development of surveillance systems for monitoring the prevalence of target problems in defined populations. Only then will we be able to show that evidence-based interventions benefit societies in the way that efficacy and effectiveness studies suggest that they can.

The use of such surveillance systems is growing. In the U.S., nationally representative systems, such as Monitoring the Future (Johnston, O’Malley, & Bachman, 2001) and the Youth Risk Behavior Survey (Kann et al., 2000); state-level assessment systems, such as the Oregon Healthy Teens survey (Boles, Biglan, & Smolkowski, 2006); and community-level systems, such as Communities that Care (Arthur, Hawkins, Pollard, Catalano, & Baglioni, 2002) increasingly monitor adolescent health behavior. Behavioral scientists wishing to translate existing knowledge into widespread effects would do well to foster the development of such systems. These systems not only enable evaluation of the population-based impact of interventions, they can also motivate policymakers to implement programs, if evidence from the system shows high levels of problems. For example, in the mid-1990s, adolescent smoking was increasing dramatically. Evidence of this trend was one reason that antitobacco advocates mounted a major effort to counter youthful smoking. Smoking prevalence among adolescents declined subsequently and, although the evidence is based on experimental evaluations, it suggests that antitobacco efforts (including lawsuits against the tobacco companies, increased taxation, and less access to tobacco) contributed to this change.

Behavioral scientists have learned something about the organizational context that fosters innovation and change in practice. Coch and French (1948) originally showed that the adoption of a new method of sewing pajamas was facilitated by a participatory process in which sewers helped to design the new system. A number of studies have shown that schools that are high in participation and have a supportive, collegial environment are (a) more likely to adopt and effectively implement innovations (Bryk, Lee, & Holland, 1993; Lee & Bryk, 1989; Lee, Dedrick, & Smith, 1991; Lee & Smith, 1996) and (b) more effective in teaching (e.g., Bryk & Driscoll, 1988). However, the evidence does not meet the standards we described above for calling something evidence based. Specifically, there are no experimental studies showing that increasing participation or collegiality leads to adoption and implementation of a practice.

Bond and Bunce (2000; 2003), have, however, provided some relevant experimental data. They found that a series of workshops helped workers become more accepting of their thoughts and feelings, and more willing to take those thoughts and beliefs as just that—thoughts and beliefs; not reality. That study, and others using the same strategy, showed (in randomized trials) that the approach reduced stress and burnout and increased willingness to innovate (Hayes, Luoma, Bond, Masuda, & Lillis, 2006; Hayes et al., 2004). Thus, a psychological intervention provided to all members of an organization seemed to “tilt” the culture toward innovation—and less stress.

However, the influences on organizations—as a whole—remain a problem. We cannot specify with any precision the variables that influence organizations to act or to adopt practices. However, recent analyses of the evolution of cultural practices provide a framework for analyzing organizational practices (Biglan, 1995; 2003; Wilson, 2003; 2007). In particular, Sigrid Glenn has proposed a concept that may help us improve our understanding (Glenn, 2004). She suggests the practices of groups and organizations are selected by the material consequences that result from the practice. Consider this concept as it may apply to organizations’ adoption and implementation of evidence-based practices. Perhaps if schools and family service agencies received some of their funds contingent upon the use of EBPs, they would be more likely to adopt them. Perhaps if their financial wellbeing improved with evidence of the impact of what practices they followed, they would be more likely to maintain those practices.

The problem then is for the behavioral science community to develop and empirically evaluate strategies for getting targeted organizations to adopt, implement, and maintain evidence-based practices. In that light, we describe the progress in Norway in the implementation of evidence-based practices and highlight the contingencies the country created to influence organizations to adopt and implement these programs.

Norway’s Adoption and Implementation of Evidence-based Practices

Impetus for This Effort and History of its Development

In 1968, a committee appointed by the Norwegian government reported on the schools for severely maladjusted children and youth. Its members concluded that there was a strong probability that “open treatment” in the community, or combinations of open treatment and treatment in institutions, would give equally good or even better results than residential treatment alone. Thirty years later, international researchers invited to the Norwegian Research Council’s expert conference on particularly troublesome children echoed these conclusions. Outstanding scientists, clinicians, and program developers from the U.S., Canada, and Norway, by invitation, presented their research and intervention programs to a Norwegian audience of practitioners, policymakers, and researchers.

Discouraging long-term outcomes had occurred in several Nordic studies of placement of antisocial and offending youth in social or treatment institutions (Kjelsberg & Dahl, 1998; Levin, 1998; Sarnecki, 1996). It also became clear that a substantial number of antisocial children and youth were not receiving any services or help for their problems (Storvoll, 1997). There was a strong demand for increased competence and interventions that were more effective in dealing with conduct problems in children and youth.

Following the conference, a governmental initiative came into being to strengthen services, competence, and research in relation to children and youth with serious behavior problems. The Ministry of Child and Family Affairs funded a small research unit and organized it as a five-year project at the University of Oslo, Institute of Psychology. The adoption and implementation of evidence-based family and community treatment programs received priority. Among the theory-driven programs with a strong research track were the Oregon model of “Parent Management Training” for children (Forgatch, 1994; Patterson, Reid, & Dishion, 1992) and Multisystemic Treatment for adolescents (Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 1998). The initiative quickly expanded into a comprehensive and well-organized national implementation project for the treatment of antisocial behavior in children and adolescents, unprecedented in Norway. The Ministry invited the 19 county health directors to participate in the implementation process. They gave a quick and positive response. Based on encouraging results and experiences, the Behavior project became a permanent national center in 2003. The Norwegian Center for Child Behavioral Development (The Behavior Center–Unirand) is part of Unirand, owned by the University of Oslo, and funded by several ministries.

Evidence-based Interventions in Norway

Adopting and Implementing the Oregon Parent Management Training

In May 1999, Marion Forgatch, Gerald Patterson, and colleagues from the Oregon Social Learning Center (OSLC) in Eugene, Oregon USA began to provide an extensive parent-training program, the Parent Management Training-Oregon Model (PMTO), to qualified Norwegian professionals. The first round of training ended after 18 months, with certification of 30 candidates from all four of Norway’s health regions. OSLC staff ensured adherence to the PMTO model and certified the trainees’ competence. The next round of training began in 2002, when 83 new candidates received training from the first generation of therapists, backed by a team of PMTO consultants at the national and regional levels. The program recruited candidates from the regular service system in order to ensure they would be in position to apply their new competencies after training. The local implementation organizations had accepted the conditions for program participation, which included providing equipment, trainee salaries, travel expenses, and time off to participate in training activities (Ogden, Forgatch, Askeland, Patterson, & Bullock, 2005). The ministries funded infrastructure development, training, and supervision through the national center. To receive certification, each therapist completed three full-scale therapies. The ministries based the certification on PMTO expert evaluations of videos from therapy sessions using the Fidelity of Implementation system (FIMP; Knutson, Forgatch, & Rains, 2003) to assess therapist competence and treatment adherence. The two generations of PMTO therapists applied the core PMTO principles in the training and consultation of practitioners working with children in the child welfare and child mental health services at the municipal level.

Eight years after introduction of the training program, 208 therapists have received training and certification. Moreover, 190 received recertification within three years, which is a requirement for continuing to practice PMTO. The permanent training and supervision network established by the Center for Child Behavioral Development (CCBD) consists of five PMTO specialists at the CCBD, six regional coordinators, and seven regional consultants. The sustainability of therapist and program effectiveness is supported by quality assurance procedures, including eight meetings per year in small supervisory groups, regular network meetings, and recertification of therapists every third year. The national center also provides technical support by producing and updating handbooks and training materials. The number of children and families receiving PMTO has increased over the years as more therapists have received certification. In 2006, approximately 1,500 families went through treatment.

A randomised controlled trial took place in order to investigate the treatment effectiveness of PMTO in Norway (Ogden & Amlund-Hagen, 2007). A group of 112 children with conduct problems and their families participated in the study. Using a multiagent/multimethod approach, results showed that PMTO was effective in reducing child externalizing problems, improving social competence, and enhancing parental discipline. In conclusion, the findings showed that practitioners in Norway could successfully transport and implement the PMTO treatment program developed in the U.S.

Adopting and Implementing Multisystemic Therapy (MST)

In spring 1999, the Ministry of Child and Family Affairs contracted with MST-Services (a university-affiliated training organization) to train and supervise Norwegian MST teams. While the PMTO targeted children aged 12 and under, the ministry offered MST for families with adolescents between the ages of 13 and 18. The ministry integrated MST into the decentralized organization and into operations of the county Child Welfare services. At the outset, most team leaders (who also were clinical supervisors) received an introductory training week in the U.S., while therapists began a weeklong training in Norway in groups of four teams. Initially, senior consultants and experienced therapists from the U.S. conducted the training, supervision, and consultation, but gradually transferred the tasks and responsibilities to MST consultants at the CCBD.

By 2007, MST was in place across Norway, with 23 teams (86 therapists and 25 team leaders). The national support from the CCBD consists of site assessments conducted on a regular basis, a five-day introductory training, weekly telephone consultations, quarterly booster sessions, monitoring of treatment adherence and outcomes, and clinical outcome studies.

A randomized trial of MST took place during the first year of implementation, with 100 adolescents randomly assigned to MST or regular Child Welfare Services. The results showed the possibility of replicating in Norway the positive clinical outcomes obtained in the U.S. (Ogden & Halliday-Boykins, 2004). MST was more effective than regular services were at reducing youth internalizing and externalizing behaviors and out-of-home placements, as well as increasing youth social competence. MST families were also more satisfied with treatment received compared to families receiving regular services. A follow-up study (Ogden & Amlund-Hagen, 2006) of treatment effectiveness conducted two years after intake to treatment showed that MST was more effective than regular services were in reducing out-of-home placement and internalizing and externalizing behavior problems at three of four implementation sites.

The sustainability of MST program effectiveness across participant groups underwent examination during the second year of operation (Ogden, Amlund-Hagen, & Andersen, 2007). The results indicated sustained program effectiveness at the two sites included in the study. The outcomes in the second year of operation matched and, for key indices of antisocial behavior, even surpassed the achievements of the first year.

A national MST monitoring system is now in place in which all cases are registered and for which parents provide information about their child’s place of living, and problems related to drugs, criminal offenses, and acting out at 6, 12, and 18 months after termination of treatment. The monitoring data indicate the sustainability of behavior change over a period of 1.5 years. Based on monitoring reports from 2006, both the dropout rate (7–8%) and the proportion of clients placed out of home (6–7%) were low.

Adopting and Implementing PALS, the Schoolwide Positive Behavior Support Model

Positive Behavior, Interactions, and Learning Environment in School (Norwegian acronym: PALS) is a schoolwide intervention program (Arnesen, Ogden & Sørlie, 2006) modeled after Schoolwide Positive Behavior Support (PBS), developed at the University of Oregon (Sprague & Walker, 2005). The aim of PALS is to strengthen the students’ capacity for school, adaptation, and coping with challenges of education. It works in combination with PMTO and includes modification of the social learning environment, direct teaching, and behavioral interventions by school staff. A central aspect of PALS is the principle of matching interventions to a student’s risk level. While universal interventions enhance protective factors to prevent minor problems and difficulties from escalating into increasingly serious behavioral problems, selected interventions target students who do not respond to these measures. Therefore, those students receive individually tailored and intensive interventions like behavioral and academic support or mentoring combined with social skills training.

Implementation teams are in place at each school, with participants from staff, administration, parents, and school psychological services. The teams plan and implement interventions, introduce PALS to parents and staff, adapt the PALS handbook (Arnesen & Askeland, 2006) to the school’s situation, monitor the process and outcomes, and coordinate the schoolwide assessment of risk and protective factors. PALS supervisors supply monthly training and consultation with the teams, while the team trains the school staff on a weekly basis.

The program remains in place at each school for three years. In the first year, universal school-wide and classroom planning occurs and the school staff receives training. During the second year, staff combines universal interventions targeting all students with selected interventions targeting at-risk students. In the third year, staff adds functional assessments and interventions targeting the high-risk students to the program. PMTO is an option for parents of the high-risk children; their teachers receive supervision based on PMTO principles.

The first implementation and evaluation of the model began in 2001 in four schools. In light of the encouraging outcomes from this pilot, the ministry began large-scale implementation, targeting 51 schools in 2006 and 91 in 2007. The CCBD is responsible for the training of PALS supervisors, each responsible for consulting and supervising four schools. The implementation staff consists of 2 national consultants, 4 regional coordinators, and 45 PALS school supervisors.

Researchers evaluated its effectiveness using a quasiexperimental design in four elementary schools two years after implementation (Sørlie & Ogden, 2007). All comparison schools had initiated some type of school improvement projects in order to promote positive student behavior and/or improve learning conditions. The outcome study concluded that PALS seemed to be a promising intervention model for school-wide prevention of behavior problems, and for the promotion of positive behavior (Sørlie & Ogden, 2007). At a more general level, the PALS model demonstrates how schoolwide behavior support with comprehensive interventions at all arenas of the school and with participation of all staff might contribute to the reduction and prevention of student problem behavior.

Final Thoughts on the National Implementation Strategy

The Ministry of Child and Family Affairs formulated the Norwegian national implementation strategy in an effort to strike a fair balance between centralized dissemination and local implementation. The ministry established a national center for development and research in order to support local services across Norway in the process of implementing several evidence-based programs. The implementation strategy has five main components. The first is long-term funding of program implementation in the local services. Second is establishment of a national center to coordinate policy, practice, and research. Third, a therapist and practitioner recruitment strategy begins throughout the regular service systems. Fourth are comprehensive therapist/practitioner training, supervision, and maintenance programs. Finally, the ministry funds research on child and adolescent behavioral change and the implementation quality of the treatments offered. The national strategy also includes an extensive system of quality assurance, including monitoring of treatment and program adherence, productivity, and outcomes. The overarching goal of the implementation is to reduce and prevent conduct problems effectively.

It would be of great interest to investigate the wider impact of these national efforts to reduce and prevent antisocial behavior in Norway. The large-scale implementation of evidence-based programs over time might ideally affect the prevalence figures among children and youth. National prevalence studies do not occur regularly, however, so it is currently not possible to track changes this way. However, teens whose families receive MST are in the official child welfare statistics, and plans are underway to analyze their future contacts with the child and adolescent service system or later criminal offending as registered by official statistics.

Conclusion

The evidence reviewed in this paper documents the evolving movement toward the use of evidence-based practices to address the most common and costly problems of child and adolescent development. The rate of experimental evaluations of prevention and treatment interventions is increasing. Schools and other agencies are dedicating more resources toward identifying and disseminating interventions already validated in carefully accomplished experimental evaluations. Moreover, as the Norwegian experience indicates, programs found to be efficacious, such as PMTO and MST, increasingly undergo testing in effectiveness trials in which “real world” providers and organizations implement the programs.

Norway’s approach to implementing EBPs provides a model that the U.S. and other nations can employ. It is possible for a central government agency to identify and disseminate EBPs in a way that supports local agency efforts. Providing funding for training and technical support but allowing agency employees to volunteer to participate means the approach is not coercive. Having clear standards for training and evaluation of competence ensures implementation fidelity. Further evaluating the interventions in effectiveness trials ensures that previously obtained benefits of the program will continue.

The ultimate value of all of these efforts will be a reduction in the incidence and prevalence of child and adolescent problems. An important next step in the evolution of the use of evidence-based practices will be the documentation of effects on populations. As monitoring systems become better developed and more widely used, it will be possible to demonstrate that implementing EBPs contributes to the decreasing incidence and reduction in prevalence. Such an outcome should accelerate the spread of evidence-based practices (and monitoring systems) as states and nations observe the benefits that accrue to countries such as Norway, which has led the way in systematically making use of the best available science to improve the lives of its young people.