Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jun 1.
Published in final edited form as: Eval Program Plann. 2012 Dec 5;38:19–27. doi: 10.1016/j.evalprogplan.2012.11.001

Going-to-Scale with the Early Risers Conduct Problems Prevention Program: Use of a Comprehensive Implementation Support (CIS) System to Optimize Fidelity, Participation and Child Outcomes

Michael L Bloomquist 1, Gerald J August 1, Susanne S Lee 1, Chih-Yuan S Lee 1,1, George M Realmuto 1, Bonnie Klimes-Dougan 1,2
PMCID: PMC3602328  NIHMSID: NIHMS427027  PMID: 23266400

Abstract

The present study is a descriptive report of a comprehensive implementation support (CIS) service that was used to promote high levels of program fidelity in a going-to-scale intervention trial of the Early Risers conduct problems prevention program. The program was delivered across 27 geographically-dispersed, elementary school sites over a two-year period. In this study we examined the level of fidelity achieved by program implementers across intervention components, the rate of child and parent participation across intervention components, and proximal child outcomes targeted by the intervention across two years of programming. Results showed that over the two-year intervention period the program was implemented with high fidelity, participation rates were acceptable, and children made positive gains on target outcomes similar to those found in previous randomized controlled trials. The results suggest that implementation support services may be advantageous in the wide-scale implementation of prevention programs that aim to achieve high implementation fidelity.

Keywords: prevention, going-to-scale, implementation support, implementation fidelity

1. Introduction

Although evidence for the efficacy and effectiveness of programs aimed at the prevention of violence, drug use, and conduct problems is substantial, there is little research to guide scaling-up of these programs to maximize their public health impact (Elliott & Mihalic, 2004). Kellam and Langevin (2003) have suggested that the focus in going-to-scale research should be on designing and testing methods of training and technical assistance (TA) that facilitate high quality program implementation. The norm in the field of intervention science, however, is to provide and/or document very little in the way of implementation support (Borrelli et al., 2005). That is unfortunate given that many interventions are complex and can place considerable burden on real-world implementers with a resultant risk of fidelity deterioration over time (McCormick, Steckler, & McLeroy, 1994; Rohrbach, Graham, & Hansen, 1993). It stands to reason that program implementers may need up front and sustained implementation support services to achieve optimal fidelity levels (Durlak & Dupre, 2008; Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005).

Evidence is beginning to emerge that attests to the benefits of rigorous implementation support services in large-scale dissemination of prevention programs. For example, evaluation of the Blueprints for Violence Prevention dissemination project found that the quality of TA was a critical influence on the successful implementation and sustainability of eight evidence-based programs implemented in 42 sites over a two-year period (Mihalic & Irwin, 2003). Ongoing TA, one form of implementation support, can include some combination of booster training for original providers, training of replacement staff, ongoing monitoring of implementation efforts with corrective feedback, and on-site coaching (Durlak & DuPre, 2008). Rohrbach, Gunning, Sun, & Sussman (2010) randomly assigned school-based implementers of the Towards No Drug Abuse (TND) prevention program to a comprehensive implementation support system, a regular workshop training condition, or a standard care control. The comprehensive support system included regular workshop training, TA including on-site coaching, and web-based support. The comprehensive system produced more exacting implementation fidelity than regular workshop training alone, although there were few significant differences on immediate outcome variables. While TA was useful for this time-limited 12-session intervention, there is no known documentation of how well comprehensive support would work with a complex and more demanding intervention.

1.1. The Early Risers program

The present study is a demonstration of a comprehensive implementation support (CIS) service that was used to promote high levels of program fidelity in a going-to-scale intervention trial of the Early Risers conduct problems prevention program. The Early Risers program targets early elementary school-aged children (grades K-3, see August, Bloomquist, Realmuto, & Hektner, 2007 for a review) who show early behavioral and social adjustment problems. The intervention is delivered by a school practitioner, referred to as the “family advocate,” who provides a coordinated package of child-focused skills training and school support interventions along with parent education/skills training and family support interventions over a two-year period. Early Risers has been validated in a program of research that included efficacy, effectiveness, and sustainability trials (see August et al., 2007). These studies led to the current effort to bring Early Risers to scale across 27 geographically-dispersed, elementary school sites over a two-year period. It was thought that Early Risers would present challenges for large-scale implementation because of its multifaceted intervention framework, its use of one primary implementation agent to coordinate and deliver all program services, and its extended period of delivery. The CIS was therefore designed to meet these challenges while being logistically feasible for large-scale administration, acceptable to school-based program implementers, and relatively inexpensive to administer.

1.2. Comprehensive implementation support service

The development of our CIS was guided by the numerous examples of implementation support methods in the prevention and implementation literature (Durlak & DuPre, 2008; Fagan & Mihalic, 2003; Lochman, Boxmeyer, Powell, Qu, Wells, & Windle, 2009; Mihalic, Fagan, & Argamaso, 2008). A typical implementation support protocol minimally involves the use of manuals, training, technical assistance, supervision, and continuous performance feedback (Beidas & Kendall, 2010; Durlak & DuPre, 2008). In addition, a community advisory board, which typically advises researchers on methods for developing, evaluating, and disseminating prevention and intervention methods in communities, is useful in providing a foundation upon which to launch other implementation support strategies (Caldwell, Brotman, Coard, Wallace, Stellabotte, & Calzada, 2005). These ingredients were incorporated into the CIS used in this study.

The CIS package included: (1) a community practitioner advisory board that met twice yearly to provide administrative direction and oversight of program implementation, (2) program manuals that provided step-by-step scripts for program delivery, (3) two intensive two-day training workshops administered prior to the start of the first year summer session and first year regular school year programming followed by a one-day training booster prior to the start of the second program year, (4) regularly scheduled TA and coaching teleconferences, and (5) weekly access to a web-based fidelity monitoring system that simultaneously served to document implementation and enhance implementation through continuous reminders of practice parameters upon which implementers were trained.

1.3. Previous findings

In a previous published report from this going-to-scale trial, we presented data on the feasibility, acceptability, and utility of the CIS assessed following one year of programming (Lee, August, Realmuto, Horowitz, Bloomquist, & Klimes-Dougan, 2008). There was high compliance with the web-based fidelity documentation system and TA teleconferences and user feedback indicated general satisfaction with the CIS components. A multidimensional construct of fidelity was used to assess the dosage of services provided (exposure), the degree to which program strategies conformed to the manual (adherence), as well as quality of program delivery (quality of delivery), and high levels of fidelity was achieved across three of the four intervention components (e.g., Child Skills, Parent Skills, and Child School Support components). In contrast, fidelity ratings for Family Support were modest, most likely because of difficulties in delivering this component which required scheduling home visits at nonconventional times with challenging families.

The present study extends Lee et al. (2008) by reporting on key fidelity variables after completion of the full two-year going-to-scale trial including rate of child and parent participation across intervention components and proximal child outcomes targeted by the intervention. This is not an experimental study but rather an observational descriptive study that reports on implementation parameters that may reflect how well the implementation support services worked in a going-to-scale trial.

2. Method

2.1. Recruitment of intervention sites

A full description of procedures that were employed to recruit schools into this study is provided in Lee et al. (2008) and briefly summarized here. Letters of invitation to participate in a state-wide implementation study of the Early Risers program were mailed to systems leaders of the 34 regional Children’s Mental Health Collaboratives throughout the state of Minnesota. Invitations were also sent to regional recipients of 21st Century grants (federally funded programs designed to establish or expand community learning centers that provide students from low-income families with academic enrichment opportunities). In order to participate in the study sites had to have K-5th grade schools in their local district and agree to dedicate at least a half-time staff member, participate in all intervention training, and cooperate in the research protocol. Ultimately, all sites who agreed to these criteria were selected. Of the eighteen Collaboratives and thirteen 21st Century grantees that participated in formal recruitment presentations, nine were enrolled in the study. Of these, six regions began the study in the first year (cohort 1), and three regions began in the second year (cohort 2). The number of individual implementation sites (i.e., schools) varied across regions. Three regions had only one site, one region had two sites, two regions had three sites, two regions had four sites, and one region had eight sites, for a grand total of 27 school sites.

2.2. Recruitment of participants

A detailed descriptions of the subject recruitment process and eligibly criteria are provided elsewhere (Bloomquist, Horowitz, August, Lee, Realmuto, & Klimes-Dougan, 2009) and are briefly summarized here. Each of the 27 sites enrolled children (grade 1 and 2) into the Early Risers program that was being offered under the auspices of their schools. For the most part, schools applied child selection criteria stipulated by Early Risers program developers. These criteria included: (1) the presence of early school adjustment difficulties consistent with subclinical forms of aggressive/disruptive or socially withdrawn behaviors, (2) enrollment in regular educational classes, and (3) absence of a pervasive development disorder. Once enrolled, parents of program children were invited to participate in the going-to-scale research study. Parents understood that continued participation in the Early Risers program was not contingent on research participation. Once research consent was given, grade 1 and 2 teachers were asked to complete the Behavioral Assessment System for Children-II (BASC-2; Reynolds & Kamphaus, 2004). Only children receiving a gender-specific T-score > 60 on the aggression and/or the withdrawal scale were formally entered into the research study. An attempt was made to recruit a minimum of 15 children (and families) from each school site. The final sample with data across all assessment waves equaled 262. The average age (years) of the children was 6.92 (SD =0.93). Sixty-six percent of the participants were male. The majority of the sample (81%) consisted of Caucasians and 46% of the sample had annual income less than $30,000.

2.3. Intervention

Detailed descriptions of the Early Risers program are provided in August et al. (2007) and Lee et al. (2008) and are briefly summarized here. Two child-focused and two parent/family-focused components of Early Risers were delivered continuously over a two-year period. The Child Skills component incorporated a social-emotional skills curriculum, a literature appreciation curriculum, creative activities, and behavioral and milieu strategies for children. The protocol called for 112 hours of Child Skills per year (224 hours over two years). A needs-adjusted Child School Support program involved the family advocate monitoring each child during the school year, communicating with his or her teacher, and providing periodic mentoring services (e.g., help with schoolwork, social goal setting, consultation with teacher, facilitation of home-school collaboration, etc.). The protocol called for each child receiving between 3 (minimum) and 18 contacts per school year, depending on level of need (6 – 36 over two years). There was no specified amount of time prescribed for Child School Support. The Parent Skills component was delivered to parents within the context of five “Family Nights” per school year (ten over two years) where parents received parent education and skills training in groups, while their children participated in recreational activities. The needs-adjusted Family Support component involved individual meetings between the family advocate and parents and included a blend of individualized case management and parenting skills training interventions with contacts occurring in the home, at school, or in public settings. The protocol called for each family receiving between 3 (minimum) and 18 contacts per school year, depending on level of need (6 – 36 over two years).

2.4. Intervention implementers

The primary implementer for Early Risers is a family advocate who provides coordination and direct service across all child and family components. Each school recruited its own family advocate, typically from the ranks of existing school staff. For the most part, these individuals were employed as school social workers, behavioral specialists, or family counselors. Twenty-seven family advocates were employed during the first year of deployment. Natural attrition of family advocates over the two-year trial was attributed to professional advancement within or outside the site, choosing full-time parenting after maternity leave, or poor site-advocate fit. In year one, four advocates were replaced. In year 2, an additional six resigned and were replaced, for total of 10 family advocate attritions over a two-year period. The average age of the family advocates was 36.22 (SD=7.80). Their average years of work experience with children and families was 6.37 (SD=4.28). They were all Caucasian and were all female except for one male. All but one held at least a college degree, with 22% of them also having a master’s degree. Ninety percent of the advocates had prior experience working in school settings. Approximately half were trained as social workers and the others represented a variety of orientations (e.g., elementary education, behavioral sciences).

2.5. Comprehensive implementation support service (CIS)

CIS was offered to family advocates and support staff at all 27 sites. This CIS package included the following five components.

Community Practitioner Advisory Board (CPAB)

CPABs were assembled in each of the nine geographical regions participating in this project (regions differed in the number of elementary school sites that were enrolled in the project). The membership of each CPAB included representation from the regional mental health Collaborative serving the school district, school district superintendant, school site principals, school site family advocates, designated school site Early Risers supervisors, and two members of the Early Risers program developers’ staff. One or more meetings were held prior to the start of the intervention trial to address various start-up procedures and to create organizational policies. Thereafter, two meetings were scheduled per year to provide administrative direction and project oversight. Duties of the CPAB included: (1) review informed consent procedures, participant selection and recruitment methods, and procedures for monitoring confidentiality (HIPPA), safety, and reporting of adverse events, (2) oversee human resources issues such as staff hiring, payment and benefits, supervision, morale, and evaluation, (3) institute and monitor quality assurance procedures, and (4) access community health service agencies to serve as referral options for the Early Risers program.

Training program

An initial 2-day training session was provided to family advocates and their supervisors prior to the start of the program (14–16 hours), and another 2-day training was provided three months later (another 14–16 hours). Training provided coverage of (a) the Early Risers intervention model and standards of practice, (b) procedures for recruitment and participant consent, (c) content and delivery methods for all intervention components, (d) operation of the web-based program monitoring documentation system (see below); (e) evaluation methods and data collection procedures, and (f) professional boundaries. A six-hour booster training experience was offered prior to the start of the second program year and included discussion and problem-solving pertaining to program implementation.

Manual

The Early Risers program manual was distributed to family advocates at each site. The manual includes five modules. Module 1 provides an overview of the theory and objectives of the program, along with various administrative issues, including participant selection and recruitment procedures, supervisory and fidelity procedures, and program evaluation methods. Module 2 covers Child Social-Emotional Skills Training, Child Literature Appreciation, and Child Creative Activities components that were delivered during the summer and school years. Modules 3, 4, and 5 cover the Child School Support, Parent Skills, and Family Support components.

TA teleconferences

TA was provided to family advocates by an Early Risers program manager with eight years of experience training, coaching, and supervising family advocates in various research projects. Teleconferences were scheduled once per month during the first four months of the trial, then once every other month for the next six months, and finally once every three months during second year. During these calls, the program manager administered a semi-structured interview that was designed to elicit responses from the family advocate pertaining to adherence to program content and quality of program delivery. Family advocates were afforded opportunities to discuss strengths and weaknesses of their implementation. Practical solutions were generated to solve problems. The family advocate and TA supervisor discussed deviations or alterations to the manualized protocol. The supervisor assisted the advocate in maintaining program fidelity.

Web-based fidelity monitoring system

An on-line monitoring system was developed to track program implementation and also serve as a method for collecting fidelity data pertaining to exposure and adherence (see below). Consultation with internet web designers allowed program developers to construct a weekly implementation monitoring system. A secure internet portal was created to assist family advocates to report their program efforts. The web survey included questions that addressed implementation performance for all components of the Early Risers program. Family advocates were required to log in at the end of each week of programming year-round. Individualized login names and passwords for the family advocate at each site were created to permit them to enter the system when documenting program implementation. After logging into the system, the family advocate checked off how much time was spent delivering each program component. This prompt was asked, “How many hours did you offer your students for social-emotional skills this week?” Next, the family advocate was cued to an adherence page. Here the family advocate checked what was delivered and how it was delivered. For example, adherence for the Child Literature Appreciation component was prompted as follows: “What strategies did you use this week? (Check all that apply),” followed by these optional responses: read aloud, vocabulary, comprehension, paired reading, individual reading, and fun sheet. Client participation was also documented with this system. The family advocate checked the students who participated and typed in the amount of time they participated in each program component. Responses were translated into scores that allowed tracking of the level of program exposure and adherence achieved by the family advocate during each week.

2.6. Fidelity measures

Measurement of implementation fidelity

A multi-dimensional construct of fidelity was applied as operationalized by Dane and Schneider (1998). Dimensions included (a) exposure (sometimes referred to as dosage), (b) adherence, and (c) quality of delivery. Frequency of client participation was not included as a fidelity dimension but was considered as a separate dosage variable. Exposure was defined as the number of sessions (hours) or contacts delivered by each family advocate in relation to the amount prescribed by the program protocol (participation pertaining to how much was actually attended of what was offered is discussed in section 3.2). The required sessions/contacts for each program component across two years are provided in Table 1 and more fully detailed in Lee et al. (2008). Adherence pertained to the extent to which recommended behavioral change strategies were used in each program component as dictated by the manual (# of strategies used / # of strategies prescribed by the manual). Quality of delivery reflected the family advocate’s quality of delivery as expressed in response to interview questions through TA teleconferences with each school site. A scripted set of questions related to each program component of the program was asked within the domains of (a) conformity to program goals, (b) response of participants, (c) implementer enthusiasm, (d) delivery techniques, and (e) implementer preparedness. Ratings for each family advocate were completed on a 6-point Likert-type scale (1= very poor; 6=excellent) and aggregated for a global impression score. Teleconferences were administered throughout the year and data were averaged over all ratings for each program component. Quality of delivery ratings were created by dividing the score for each program component by 6. Reliability of the ratings was established in Lee et al. (2008). A full description of the fidelity assessment procedures used to record exposure, adherence, and quality of implementation are also provided in Lee et al. (2008).

Table 1.

Family advocate’s fidelity scores in program components over time

Child Skills Parent Skills Child Support Family Support
EX AD QA EX AD QA EX AD QA EX AD QA
Year 1
M 79.1 .91 .79 4.2 .73 .77 .64 .69 .70 .33 .55 .67
SD 24.4 .08 .10 1.1 .11 .15 .45 .22 .13 .39 .24 .16
Min. 30.5 .71 .59 2 .44 .47 0 .25 .40 0 0 .36
Max. 145.5 1.0 .98 5 .90 1.0 1.0 1.0 .97 1.0 1.0 1.0
Manual required 112.0 1.0 1.0 5 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
Year 2
M 79.6 .83 .71 3.6 .66 .72 .83 .93 .66 .31 .90 .59
SD 25.1 .13 .15 1.1 .10 .17 .34 .15 .14 .38 .15 .19
Min. 29.0 .52 .30 1 .47 .38 0 0 .29 0 .33 .25
Max. 136.0 .98 .97 5 .83 1.0 1.0 1.0 .90 1.0 1.0 .94
Manual required 112.0 1.0 1.0 5 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
Two Years Combined
M 158.6 .87 .75 7.7 .70 .74 .74 .84 .67 .32 .70 .62
SD 42.7 .10 .10 1.7 .09 .15 .32 .15 .13 .33 .24 .17
Min. 59.5 .65 .46 3 .52 .47 0 0 .29 0 0 .25
Max. 258.0 .98 .93 10 .86 .97 1.0 1.0 .91 1.0 1.0 .95
Manual required 224.0 1.0 1.0 10 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
Open in a new tab

Note. EX=Exposure (hours), AD=Adherence (%), QA=Quality of delivery (%). It is possible for family advocates to offer services that exceed the amount required by the manual.

2.7. Child outcome measures

Teachers rated child participants at baseline, after one year of intervention (Yr1), and after two years of intervention (Yr2) in the domains of behavioral functioning, developmental strengths, and academic functioning.

Behavioral functioning

In order to evaluate reductions in behavioral problems, teachers rated participants using the Behavior Assessment System for Children, 2nd edition (BASC-2; Reynolds & Kamphaus, 2004). The BASC-2 is a multidimensional system used to assess broad domains of externalizing, internalizing and school problems (alphas = .85 – .89). The child version consists of 139 items, which are rated on a 4-point scale ranging from 0 = never to 3 = almost always. Gender-specific normative scores are provided in the form of T-scores with a mean of 50 and a standard deviation of 10. Scores on the aggression and withdrawal scales as well as the externalizing problems and internalizing problems composites were used to monitor changes in participants’ emotional and behavioral problems. Evidence for reliability and validity were reported by Reynolds and Kamphaus.

Developmental strengths

To evaluate promotion of developmental strengths, teachers rated participants using the Behavioral and Emotional Rating Scale, Second Edition (BERS-2; Epstein, 2004). The 62-item measure is designed to assess behavioral and emotional strengths of children. Each item is rated on a 4-point scale, ranging from 0 ‘not at all like the student’ to 3 ‘very much like the student’. The measure yields a composite strength index (alpha = .96), which represents five factors: interpersonal strength, family involvement, intrapersonal strength, school functioning, and affective strength. Gender specific T-scores are presented with a mean of 50 and a standard deviation of 10. The BERS-2 demonstrates excellent psychometric properties (Epstein).

Academic functioning

Academic functioning was rated by teachers using the Academic Competence Evaluation Scales (ACES; DiPerna & Elliott, 2000). The 73-item measure assesses student’s academic skills, attitudes and behaviors that contribute to academic success (alphas = .94 to .99). Items are rated on a 5-point scale, ranging from 1 = never to 5 = almost always. Scores are provided in gender- and grade-specific T-scores. The academic enablers scale used in this study provides a composite rating of the attitudes and behaviors of students that facilitate active participation in classroom instruction, and includes four subscales: interpersonal skills, motivation, study skills, and engagement. Evidence of good reliability and validity of the measure were reported by DiPerna and Elliott.

3. Results

3.1 Implementer utilization of implementation support

The amount of training the school-based implementers participated in across two years was tabulated. There was 100% compliance of all implementers for participating in the initial four-day training during year one and also the one day booster training in year two. Calculations were made as to the rate at which the implementers logged on to the internet program report website. The implementers were required to login and document services on a weekly basis over two years (excluding normal school year breaks) and there was 100% compliance during year one and 94% compliance during year two. Finally, the rate of participation of implementers in the technical assistance teleconference calls was determined. During year one the implementers complied with 81% of these calls and during year two this dropped to a compliance rate of 56%. It is noteworthy that implementers reported to the program manager that during the second year they felt knowledgeable and comfortable doing the program and didn't feel the need for as much "assistance." The program manager obliged implementers with less technical assistance if it was requested so as to not create a burden or barrier to overall implementation.

3.2. Implementation fidelity

Table 1 presents the scores of the three fidelity indices (exposure, adherence, and quality of delivery) for each of the four Early Risers program components delivered over a two-year period. Others have noted a benchmark of 60% fidelity as a minimum to achieve as an indicator of high fidelity (meaning high enough to suggest the program was delivered with integrity) (Botvin, Baker, Dusenbury, Botvin, & Diaz, 1995; Botvin, Baker, Dusenbury, Tortu, & Botvin, 1990; Fagan & Mihalic, 2003). Accordingly, as shown in Table 1, across 27 sites a majority of the program was implemented with high fidelity, particularly for the fixed-prescription (i.e., content the same for all participants) intervention components (Child Skills, Parent Skills). For these components, on average, program implementers achieved at least 70% of the level required by the manual for all three fidelity indices during the implementation period. For the variable-prescription (i.e., content tailored based on individual need) intervention components (Child School Support, Family Support) high fidelity was observed for Child School Support with low fidelity noted for Family Support. Also noteworthy was an absence of implementer drift in fidelity over the two-year implementation period. Implementers demonstrated remarkable stability in maintaining their fidelity levels over time. Interestingly, the adherence score for Family Support improved markedly from year 1 to year 2 (.55 to .90).

3.3. Child and parent participation

Table 2 presents the rate of child and parent participation in each of the program components over the two years of implementation. Participation differs from exposure in that exposure is a measure of what is offered and participation is a measure of what is received (i.e., attendance). For the fixed-prescription components, on average, children participated in about 50% of the Child Skills programming over the two years (102 hours), whereas families participated in about 33% of the Parent Skills programming in the same period (3.4 sessions). As anticipated in prevention trials, there was considerable variability in individual rates of participation within the Child Skills component ranging from 0 to 223.5 hours and the Parent Skills component ranging from 0 to 10 sessions. Participation in the variable-prescription components is based on assessed need and therefore participation rates are usually skewed to those at highest risk. As expected, considerable variability was observed for both Child School Support and Family Support, as these components were delivered on the basis of need. However, there is a minimum exposure threshold expected upon which fidelity was determined. For these components the exposure minimum (what was offered) could be the entire extent of participation (what was attended). Beyond the minimum exposure, standard participation could vary and be quite extensive consistent with child or family need. On average over the two years, children received approximately 18 hours of Child School Support, while families received about 5 hours of Family Support.

Table 2.

Student and family participation in program components over time

Child Skills1 Parent Skills Child Support2 Family Support2
Year 1
    M 46.3 1.7 7.7 2.4
    SD 29.1 1.5 8.6 5.1
    Min. 0 0 0 0
    Max. 112.0 5 60.3 62.3
    Manual required 112.0 5 Need-based Need-based
Year 2
    M 55.7 1.8 10.4 2.5
    SD 27.4 1.4 11.1 4.8
    Min. 0 0 0 0
    Max. 136.0 5 75.3 38.8
    Manual required 112.0 5 Need-based Need-based
Two Years Combined
    M 102.0 3.4 18.0 4.9
    SD 46.1 2.4 18.0 9.2
    Min. 0 0 0 0
    Max. 223.5 10 119.0 101.0
    Manual required 224.0 10 Need-based Need-based
Open in a new tab
1

It is possible for students or families to receive services that exceed the amount required by the manual, given that some family advocates offered more than what was required.

2

It denotes the amount of hours which student or family received for child or family support, respectively, based on needs.

3.4. Child outcomes

Table 3 presents means and standard deviations of key child outcome variables. Repeated-measure ANOVAs were applied to test whether there were significant changes from pre- to post-test on child outcomes measured at three time points. Results showed that there was statistically significant change in aggression [F(2, 438) = 35.85, p < .001], withdrawal [F(2, 438) = 26.94, p < .001], externalizing behavior problems [F(2, 438) = 40.92, p < .001], and internalizing behavior problems [F(2, 440) = 6.04, p =.003]. Statistically significant pre- to posttest difference was also found in behavioral and emotional strengths [F(2, 390) = 22.02, p < .001] and in academic enablers [F(2, 362) = 31.10, p < .001]. Inspection of the means indicated that there was significant reduction in problem behaviors from pre- to post-test as measured by the BASC2 scales, and enhancement in developmental strengths as measured by the BERS2-strength index and academic competence as measured by the ACES-academic enablers scale.

Table 3.

Means and standard deviations on child outcome variables over time

Baseline Yr1 Yr2
Variables M SD M SD M SD
BASC2
      Aggression 65.2 14.4 60.9 13.9 58.5 12.7
      Withdrawal 64.9 11.8 60.4 10.9 59.5 10.4
      Externalizing behavior 64.8 12.1 60.9 12.6 58.8 11.4
      Internalizing behavior 60.7 13.6 58.1 12.3 57.1 12.0
BERS2
      Strength Index 41.6 8.1 44.8 9.7 45.5 9.7
ACES
      Academic Enablers 38.6 6.5 42.0 7.3 42.8 7.9
Open in a new tab

BASC2 = Behavior Assessment System for Children, 2nd Ed. (Reynolds & Kamphaus, 2004); BERS2 = Behavioral and Emotional Rating Scale, 2nd Ed. (Epstein, 2004); ACES = Academic Competence Evaluation Scales (DiPerna & Elliott, 2000).

3.5. Relations between fidelity and child outcomes

Table 4 presents correlations among fidelity indices and child outcomes across program components. Fidelity measures were averaged over the 2-year intervention, except for Child Skills exposure and Parent Skills exposure, which were the sum of Years 1 and 2. For child outcomes, the difference scores between baseline and Yr2 were calculated on the externalizing behavior (BASC2) and the strength index (BERS2) by subtracting the baseline scores from the Yr2 scores. We focused on the two composite scores (a) because they were the targeted proximal behavioral outcomes that the intervention was developed for and (b) to avoid multiple tests of correlations between child outcomes and fidelity measures. Correlations showed that fidelity indices mostly correlated significantly with one another within program components (r = .12 to .53). The results also showed that program components correlated with each other for each fidelity index, particularly quality of delivery (r = .57 to .78). However, for the most part, fidelity was not found to be significantly correlated with the difference scores in outcome variables.

Table 4.

Correlations among fidelity dimensions across program components over two years and primary outcome variables

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
2. .53***
3. .18** .50***
4. .33*** .45*** .23***
5. .55*** .35*** .16** .29***
6. .14* .27*** .78*** .09 .12*
7. .03 .06 .13* .20*** .14* .01
8. −.04 .09 .04 .05 −.19** −.04 −.17**
9. .05 .18** .58*** .49*** .16** .57*** .27*** −.08
10. −.09 −.11 .02 .22*** −.03 −.06 .50*** −.11 .29***
11. .04 .16* .30*** .30*** −.18** .21** −.09 .18* .19** .29***
12. .12* .31*** .59*** .47*** −.02 .58*** .23*** .15* .72*** .20** .36***
13. .07 .02 −.01 .07 .05 −.13* −.05 .07 −.08 .13*** .14 −.04
14. −.07 −.08 −.01 −.09 −.03 .11 .11 −.12 .08 −.02 −.03 .09 .35***
Open in a new tab

Note. 1=Child skills exposure, 2=Child skills adherence, 3=Child skills quality, 4= Parent skills exposure, 5=Parent skills adherence, 6=Parent skills quality, 7=Child school support exposure, 8=Child school support adherence, 9=Child school support quality, 10=Family support exposure, 11=Family support adherence, 12=Family support quality, 13=Difference in externalizing behavior scores (Yr2 minus BL), 14=Difference in strength index scores (Yr2 minus BL).

*

p < .05.

**

p < .01.

***

p < .001.

4. Discussion

The present study described a comprehensive implementation support (CIS) service and provided data on key implementation parameters of the Early Risers conduct problems prevention program when taken to scale. It is noteworthy that family advocates actively participated in the components of CIS and that most components of the program were implemented with high fidelity over two years of delivery. Furthermore child and parent participation rates were acceptable and children made positive gains on targeted outcomes.

4.1. Comprehensive Implementation Support and Fidelity

To enhance implementation fidelity in this going-to-scale trial, Early Risers program developers designed a CIS service to facilitate implementation at the practitioner level. In accordance with previous research we designated a benchmark of 60% fidelity as a minimum to achieve as an indicator of high fidelity (Botvin et al., 1995; Botvin et al., 1990; Fagan & Mihalic, 2003). Implementer documentation revealed high levels of fidelity for the Child Skills, Parent Skills, and Child School Support intervention components. Lower fidelity was initially observed for the Family Support intervention component, but it is noteworthy that the adherence score for Family Support improved from year 1 to year 2. These findings are similar to previous Early Risers efficacy and effectiveness trials where high levels of fidelity were observed via independent observations by trained technicians (see August, Realmuto, Hektner, & Bloomquist, 2001; August, Lee, Bloomquist, et al., 2003; August, Bloomquist, Lee, et al., 2006). We think the multiple components of CIS enhanced implementation efforts by promoting the growth of implementer skill sets necessary to deliver the program, increasing implementer self-efficacy and confidence, requiring implementers to be accountable for their performance, and providing monitoring and corrective feedback to address protocol violations in a timely manner.

Although most of the intervention components were delivered with high fidelity throughout, the Family Support intervention component that was delivered with relatively poor fidelity in year 1 improved in year 2. Family Support is typically delivered via a home visit and includes personalized brief interventions and brokering of formal and informal community and health-related services. Substandard delivery of this component was largely reflected in low fidelity exposure, indicating that family advocates did not meet the required number of family contacts. In the present going-to-scale study, family advocates were recruited from the ranks of school staff and most were educators with varying experience in working with families in a healthcare role. They noted practical barriers (e.g., scheduling meetings, traveling long distances to family homes) and, in other cases, voiced discomfort in addressing personal issues in families that were not related to how the child was doing in school. We theorize that the CIS, coupled with accumulated experience among family advocates, likely accounted for the improvements in the fidelity of the Family Support component from year 1 to year 2.

4.2. Child and Parent Participation

It is of note that in this going-to-scale study the rate of attendance in Child Skills and Parent Skills program components were at comparable levels to those observed in the previous Early Risers efficacy trial (August et al., 2001). In the efficacy trial participation barriers were removed and incentives provided to encourage high rates of attendance. In the present study, the CIS may have had an impact on attendance, especially with the parents. We speculate that the CIS influenced participation through its effect on fidelity. This is supported by an earlier report from the current going-to-scale study after the first year that examined the relationship between fidelity and participation (Bloomquist et al., 2009). In that study higher quality of delivery predicted higher parent participation in the Parent Skills component. It may be that when program implementers are supported by program developers in their efforts to deliver the program in compliance with the program model, and are accountable for their performance, a high quality program is delivered. This, in turn, is perceived by the families who respond with higher rates of attendance and engagement.

4.3. Child Outcomes

Change in child outcome indicators of aggression, social withdrawal, externalizing behavior problems, internalizing behavior problems, developmental strengths, and academic competence were in the positive direction and statistically significant across assessment waves. These changes are similar to those found in previous randomized controlled trials with the Early Risers program (August, et al., 2001; August et al., 2003; August et al., 2006; Bernat et al., 2007). With the observational research design employed in the current study, however, we cannot directly discern that the quality of implementation support was causally related to child outcomes.

Previous research suggests that implementation fidelity influences program outcomes (Durlak and DuPre, 2008) but this was not observed in the current study. Fidelity indices (exposure, adherence, quality of delivery) failed to correlate with the differences in outcome variables over time. We believe this can be attributed to the fact that there was little variability in fidelity scores across sites. This is not surprising given that the CIS service was routinely available to family advocates and that our internet self-report documentation was designed to serve as both a fidelity measure and a fidelity enhancer (Bellg et al., 2004). Thus, all sites achieved high levels of fidelity, and the lack of variability does not provide sufficient power to detect significant relationships (Durlak & Dupre, 2008). Future studies that examine the relationship between fidelity and outcome should be experimental in design similar to Rorhbach et al. (2010) to determine how much and what kind of TA implementation support is necessary to produce high fidelity and replicate outcomes, as well as how to make training and TA services feasible and acceptable to program users.

4.4. Implications for Practice and Research

Multifaceted programs such as Early Risers place considerable demands on the implementers. The family advocate is required to provide education, skills training, mentoring and support services to both children and parents, and often delivers program services beyond the usual time span of the school day. Additionally, if prevention programs like Early Risers are to be institutionalized within schools, administrators need to find creative ways to support and sustain the role of program implementer. In the present study, school-based family advocates who came from within the school ranks were able to implement most components of the Early Risers program with high fidelity and there was corresponding improvement in salient child outcomes. We believe it is paramount for school administrators to invest in comprehensive implementation support strategies to assure that school-based implementers can competently deliver effective program activities.

A critical issue facing research on preventive intervention is gaining a better understanding of how to disseminate evidence-based prevention programs on a wide scale (La Greca, Silverman, & Lochman, 2009) and particularly in schools (Weist, Lindsey, Moore, & Slade, 2006). The results of the present going-to-scale trial suggest that implementation support services may be an important component of the design of prevention programs that aim to achieve high implementation fidelity, replicate positive outcomes, and build service capacity when widely disseminated. The present study is noteworthy in that a substantial amount of capacity was built and sustained for the two-year intervention trial across geographically-dispersed elementary school sites including site funding, staffing, and administrative support. That capacity was no doubt critical in the successful implementation of Early Risers. It is our belief that the CIS provided the infrastructure needed for school sites to make the initial “investment” in the program and instilled confidence in administrators and implementers to keep it going over two years.

Kellam and Langevin (2003) argued that programs need to be developed through phases of research, including efficacy, effectiveness, sustainability, going-to-scale, and sustaining programs system wide. The process of evaluating and maintaining fidelity must accompany each of them. We posit that fidelity to the model is important to all phases of program development. The current study demonstrates the utility of CIS in a going-to-scale context with Early Risers. Future studies focusing on sustaining system wide should demonstrate the feasibility of continuing CIS, and its costs, when schools assume ownership of the Early Risers program. This would also have potential to inform the broader practice of prevention programs in schools.

4.5. Limitations

To evaluate implementation fidelity we relied on implementer-completed self-report via the internet (similar to Hill, Maucione, & Hood, 2007) and implementer-completed interviews administered via the telephone. The use of self-report in particular could be influenced by the family advocate bias or social desirability effects (Lillehoj, Griffin, & Spoth, 2004). Although independent observation by trained fidelity auditors is the gold standard for fidelity monitoring (Dane & Schneider, 1998), it was not feasible in the present study given the funding and travel constraints imposed by the magnitude of this scaled-up effort across a large geographic area. Previous research suggests, however, that self-report of fidelity is generally highly correlated with independent observation (Pentz et al., 1990; Sobol et al., 1989). It is also worth mentioning that in the current study the implementers reported low fidelity in their delivery of the Family Support component suggesting that they were providing accurate assessments of their implementation performance even when at socially undesirable substandard levels. Given the above we are relatively confident in the accuracy of the self-reported fidelity ratings but cannot unequivocally attest to their validity.

Highlights.

  • A comprehensive implementation support service was used to promote program fidelity.

  • The program was delivered across 27 geographically-dispersed, elementary schools.

  • Over a two-year intervention period the program was implemented with high fidelity.

  • Children made positive gains on target outcome.

  • Implementation support service may be useful in achiving high implementation fidelity.

Acknowledgements

This study was supported by a grant to Gerald J. August from the National Institute on Drug Abuse (DA016067). The authors would like to thank Ms. Sarah Coleman, the project manager, for her major contribution to this effort.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. August GJ, Bloomquist ML, Lee SS, Realmuto GM, Hektner JM. Can evidence-based prevention programs be sustained in community systems-of-care? The Early Risers advanced-stage effectiveness trial. Prevention Science. 2006;7:151–165. doi: 10.1007/s11121-005-0024-z. [DOI] [PubMed] [Google Scholar]
  2. August GJ, Bloomquist ML, Realmuto GM, Hektner JM. The Early Risers “Skills for Success” Program: A targeted intervention for preventing conduct problems and substance abuse in aggressive elementary school children. In: Tolan PH, Szapocznick J, Sambrano S, editors. Preventing Youth Substance Abuse: Science-based Programs for Children and Adolescents. Washington, D.C.: American Psychological Association; 2007. pp. 137–158. [Google Scholar]
  3. August GJ, Lee SS, Bloomquist ML, Realmuto GM, Hektner JM. Dissemination of an evidence-based prevention innovation for aggressive children living in culturally diverse, urban neighborhoods: The Early Risers effectiveness study. Prevention Science. 2003;4:271–286. doi: 10.1023/a:1026072316380. [DOI] [PubMed] [Google Scholar]
  4. August GJ, Realmuto GM, Hektner JM, Bloomquist ML. An integrated components preventive intervention for aggressive elementary school children: The Early Risers program. Journal of Consulting and Clinical Psychology. 2001;69:614–626. [PubMed] [Google Scholar]
  5. Beidas RS, Kendall PC. Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice. 2010;17:1–30. doi: 10.1111/j.1468-2850.2009.01187.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bellg AJ, Borrelli B, Resnick B, Hecht J, Miniucci DS, Ory M, et al. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations form the NIH Behavioral Change Consortium. Health Psychology. 2004;23:443–451. doi: 10.1037/0278-6133.23.5.443. [DOI] [PubMed] [Google Scholar]
  7. Bernat D, August GJ, Hektner JM, Bloomquist ML. The Early Risers preventive intervention: Six year outcomes and mediational processes. Journal of Abnormal Child Psychology. 2007;35:605–617. doi: 10.1007/s10802-007-9116-5. [DOI] [PubMed] [Google Scholar]
  8. Bloomquist ML, Horowitz JL, August GJ, Lee C-YS, Realmuto GM, Klimes-Dougan B. Understanding parent participation in a going-to-scale implementation trial of the Early Risers conduct problems prevention program. Journal of Child and Family Studies. 2009;18:710–718. [Google Scholar]
  9. Borrelli B, Sepinwall D, Ernst D, Bellg AJ, Czajkowski S, Sharp DL, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology. 2005;73:852–860. doi: 10.1037/0022-006X.73.5.852. [DOI] [PubMed] [Google Scholar]
  10. Botvin GJ, Baker E, Dusenbury L, Botvin E, Diaz T. Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association. 1995;273:1106–1112. [PubMed] [Google Scholar]
  11. Botvin GJ, Baker E, Dusenbury L, Tortu S, Botvin E. Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology. 1990;58:437–446. doi: 10.1037//0022-006x.58.4.437. [DOI] [PubMed] [Google Scholar]
  12. Caldwell MB, Brotman LM, Coard SI, Wallace SA, Stellabotte DJ, Calzada EJ. Community involvement in adapting and testing a prevention program for preschoolers living in urban communities: ParentCorps. Journal of Child and Family Studies. 2005;14:373–386. [Google Scholar]
  13. Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: Are implementation efforts out of control? Clinical Psychology Review. 1998;18:23–45. doi: 10.1016/s0272-7358(97)00043-3. [DOI] [PubMed] [Google Scholar]
  14. DiPerna JC, Elliott SN. ACES: The academic competence evaluation scales. San Antonio, TX: Psychological Corporation; 2000. [Google Scholar]
  15. Durlak JA, DuPre EP. Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology. 2008;41:327–350. doi: 10.1007/s10464-008-9165-0. [DOI] [PubMed] [Google Scholar]
  16. Elliott DS, Mihalic S. Issues in dissemination and replicating effective prevention programs. Prevention Science. 2004;5:47–53. doi: 10.1023/b:prev.0000013981.28071.52. [DOI] [PubMed] [Google Scholar]
  17. Epstein MH. Behavioral and emotional rating scale. 2nd ed. Austin, TX: PRO-ED; 2004. [Google Scholar]
  18. Fagan AB, Mihalic S. Strategies for enhancing the adoption of school-based prevention programs: Lessons learned from the Blueprints for Violence Prevention replications of the Life Skills Training program. Journal of Community Psychology. 2003;31:235–253. [Google Scholar]
  19. Fixsen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F. Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005. (FMHI Publication #231). [Google Scholar]
  20. Hill LG, Maucione K, Hood BK. A focused approach to assessing program fidelity. Prevention Science. 2007;8:25–34. doi: 10.1007/s11121-006-0051-4. [DOI] [PubMed] [Google Scholar]
  21. Kellam SG, Langevin DJ. A framework for understanding “evidence” in prevention research and programs. Prevention Science. 2003;4:137–153. doi: 10.1023/a:1024693321963. [DOI] [PubMed] [Google Scholar]
  22. La Greca AM, Silverman WK, Lochman JE. Moving beyond efficacy and effectiveness in child and adolescent intervention research. Journal of Consulting and Clinical Psychology. 2009;77:373–382. doi: 10.1037/a0015954. [DOI] [PubMed] [Google Scholar]
  23. Lee C-YS, August GJ, Realmuto GM, Horowitz JL, Bloomquist ML, Klimes-Dougan B. Fidelity at a distance: Assessing implementation fidelity of the Early Risers Prevention Program in a going-to-scale intervention trial. Prevention Science. 2008;9:215–229. doi: 10.1007/s11121-008-0097-6. [DOI] [PubMed] [Google Scholar]
  24. Lillehoj CJ, Griffin KW, Spoth R. Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education and Behavior. 2004;31:242–257. doi: 10.1177/1090198103260514. [DOI] [PubMed] [Google Scholar]
  25. Lochman JE, Boxmeyer C, Powell N, Qu L, Wells K, Windle M. Dissemination of the Coping Power Program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology. 2009;77:397–409. doi: 10.1037/a0014514. [DOI] [PubMed] [Google Scholar]
  26. McCormick LK, Steckler A, McLeroy KR. Diffusion of innovations in schools: A study of adoption and implementation of school-based tobacco prevention curricula. American Journal of Health Promotion. 1994;9:210–219. doi: 10.4278/0890-1171-9.3.210. [DOI] [PubMed] [Google Scholar]
  27. Mihalic SF, Fagan A, Argamaso S. Implementing the Life Skills Training drug prevention program: Factors related to implementation fidelity. Implementation Science. 2008;3:1–16. doi: 10.1186/1748-5908-3-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mihalic SF, Irwin K. Blueprints for Violence Prevention: From research to real-world settings-factors influencing successful replication of model programs. Youth Violence and Juvenile Justice. 2003;1:307–329. [Google Scholar]
  29. Pentz MA, Trebow EA, Hansen WB, MacKinnon DP, Dwyer JH, Johnson CA, et al. Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP) Evaluation Review. 1990;14:264–289. [Google Scholar]
  30. Reynolds CR, Kamphaus RW. Behavior Assessment System for Children. 2nd ed. Circle Pines, MN: American Guidance Service; 2004. [Google Scholar]
  31. Rohrbach LA, Graham JG, Hansen WB. Diffusion of a school-based substance use prevention program: Predictors of program implementation. Preventive Medicine. 1993;22:237–260. doi: 10.1006/pmed.1993.1020. [DOI] [PubMed] [Google Scholar]
  32. Rohrbach LA, Gunning M, Sun P, Sussman S. The project towards no drug abuse (TNS) dissemination trial: Implementation fidelity and intermediate outcomes. Prevention Science. 2010;11:77–88. doi: 10.1007/s11121-009-0151-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Sobol D, Rohrbach L, Dent C, Gleason L, Brannon B, Johnson CA, et al. The integrity of smoking prevention curriculum delivery. Health Education Research. 1989;4:59–67. [Google Scholar]
  34. Weist MD, Lindsey M, Moore E, Slade E. Building capacity in school mental health. The international Journal of Mental Health Promotion. 2006;8:30–36. [Google Scholar]

RESOURCES