Observational Measures of Implementer Fidelity for a School-based Preventive Intervention: Development, Reliability and Validity

Abstract

Current measures of implementer fidelity often fail to adequately measure core constructs of adherence and competence, and their relationship to outcomes can be mixed. To address these limitations, we used observational methods to assess these constructs and their relationships to proximal outcomes in a randomized trial of a school-based preventive intervention (Rochester Resilience Project) designed to strengthen emotion self-regulation skills in 1st–3rd graders with elevated aggressive-disruptive behaviors. Within the intervention group (n = 203), a subsample (n = 76) of students was selected to reflect the overall sample. Implementers were 10 paraprofessionals. Videotaped observations of three lessons from Year 1 of the intervention (14 lessons) were coded for each implementer-child dyad on Adherence (content) and Competence (quality). Using multi-level modeling we examined how much of the variance in the fidelity measures was attributed to implementer and to the child within implementer. Both measures had large and significant variance accounted for by implementer (Competence, 68%; Adherence, 41%); child within implementer did not account for significant variance indicating that ratings reflected stable qualities of the implementer rather than the child. Raw Adherence and Competence scores shared 46% of variance (r = .68). Controlling for baseline differences and age, the amount (Adherence) and quality (Competence) of program delivered predicted children’s enhanced response to the intervention on both child and parent reports after six months, but not on teacher report of externalizing behavior. Our findings support the use of multiple observations for measuring fidelity and that adherence and competence are important components of fidelity which could be assessed by many programs using these methods.

There is general agreement that implementation quality has an impact on intervention outcomes (Aarons et al., 2010; Brownson et al., 2012; Durlak & DuPre, 2008; Fixsen et al., 2005; Glasgow et al., 2012; Proctor et al., 2011). Several frameworks have been proposed to describe the factors that contribute to the implementation process (e.g., Curran et al., 2008; Damschroder et al; 2009; Glasgow, & Boles, 1999; Stetler, et al., 2008). Implementation fidelity is fundamental to these models. Implementer fidelity focuses on program delivery by the individual interventionist. Currently, two primary constructs are considered to be central to the measurement of implementer fidelity: a) adherence to program content as specified in manuals, and, b) competence in program delivery which describes the quality of the implementation (Breitenstein, et al. 2010; Cross & West, 2011; Dumas et al., 2001; Durlak & DuPre, 2008; Forgatch & DeGarmo, 2011; Hepner et al., 2011; Hogue et al., 2008; Schoenwald, et al., 2010; Waltz et al., 1993). A third component, treatment differentiation, is the degree that treatments differ from one another on critical dimensions and is used to test the outcomes of two or more treatments (Schoenwald, 2010). Conceptually, it is important to consider both implementer adherence and competence because low levels of adherence or competence separately, or both together, may be related to poor outcomes. For example, low levels of adherence to the manual would compromise a test of program outcomes because the intervention content would not have been delivered and conclusions about effectiveness would be erroneous. On the other hand, an implementer may deliver a high degree of program content, and thus be rated as highly adherent, but in a way that is not consistent with the program model’s expectation for quality or competence. Low levels of competence, such as insensitivity to the individual/group, rote program delivery that compromises the transactional nature of the intervention, and poor intervention timing are all likely to have an impact on outcomes (Cross & West, 2011). Studies where high ratings of adherence were associated with poor outcomes may be clarified if competence had been measured and found to be low (e.g., James et al., 2001).

Although implementer fidelity is theoretically related to clinical outcomes, studies have shown mixed findings regarding this relationship (Byrnes et al., 2010; Hamre et al., 2010; Webb et al., 2010). What accounts for this? One factor may be related to the failure to measure both constructs, adherence to content and competence, in implementer fidelity (Liber et al., 2010). Measurement methods may also account for the apparent disconnect between implementer fidelity and program outcomes. For example, fidelity measures based on implementer self-reports of fidelity are likely to be biased and positively skewed (Carroll et al., 2000; Lillehoj et all, 2004; Moore et al., 2009). Consequently, videotaped or audiotaped interactions, coded by independent observers, are currently the optimal standard for fidelity assessment (Schoenwald & Garland, 2013; Schoenwald et al., 2011; Synder et al., 2006). In some cases, low measurement reliability and validity may also contribute to the mixed findings regarding the relationship between program fidelity and outcomes (Schoenwald, 2011).

The present study

The current study sought to address gaps in fidelity measurement and contribute to the understanding of the relationship between fidelity and outcomes in the context of a randomized trial of a school-based preventive intervention delivered by paraprofessionals (Rochester Resilience Project, RRP; Cross & West, 2011; Wyman et al., 2010; Wyman et al., 2013). Our first research aim was to develop reliable observational fidelity measures of implementer adherence and competence, and to examine the relationship between them. To do this, we first used rigorous observational methods to code 206 sessions on both adherence and competence and analyzed these data to assess variation across implementer, lesson, and child. This was used to conduct formal tests that the underlying constructs of adherence and competence were reasonably constant over lesson and child. We wanted assurance that the measures captured implementer behaviors rather than other factors such as child variables. The second aim was to conduct an initial test of criterion-related validity of these measures by using them to predict improvements in child behavior after one year of the two-year intervention. Although the two measures developed may be specific to our intervention (particularly for adherence), the methods are generalizable to other program developers and researchers.

Method

The intervention: The Rochester Resilience Project

This study of implementer fidelity occurred in the context of a randomized trial of the Rochester Resilience Project (RRP; Wyman et al., 2010; Wyman et al., 2013), a school-based indicated preventive intervention for 1^st–3^rd grade students with elevated aggressive-disruptive behaviors, which are associated with long-term problems (Kellam et al., 1998). The intervention has three targets: child and parent components for selected children, and a universal classroom component. The current study focused solely on implementer fidelity delivering the individual child component of the intervention, which teaches hierarchically-ordered skills from emotion self-monitoring to behavioral and cognitive strategies, to assist children in using these skills in real world settings at school and home. Analyses focus on proximal outcomes in the first year of the intervention during which implementers (known as Resilience “Mentors”) meet individually with children for fourteen, 25-minute weekly lessons. The parent and classroom components, which were not the foci of this study, are briefer and designed to encourage adults to support the child’s skills. (For a full description of RRP see Wyman et al., 2013).

Ten implementers (“Mentors”)¹ delivered the school-based intervention and all participated in this fidelity study. Previously they had positions as school-based paraprofessionals (e.g., classroom aide) in a variety of schools. These paraprofessionals participated in extensive training (approximately 200 hours). All implementers were employed full time with the study for at least one year, with three implementers not retained for all four years of the trial. Implementer demographics are provided in Table 1. Mentors delivered all three intervention components (child, parent, classroom) for their assigned children, with permission to videotape lessons with children. Videos were stored on a secure server.

Table 1.

Implementer and Participant Demographics

	Implementers
Sex	Male	10% (1)
	Female	90% (9)
Age (yrs)	Mean (SD)	42 (6.2)
	Range	27 – 49
Race/ethnicity	White	20% (2)
	African American	50% (5)
	Hispanic	30% (3)
Education	High school diploma	20% (2)
	Some college	60% (6)
	Bachelor’s or Equivalent degree	20% (2)

	Child Participants *
Sex	Male	64.5% (49)
	Female	35.5% (27)
Age (yrs)	Mean (SD)	7.3 (1.000)
	Range	6 – 9.2
Race/ethnicity	White	6.6% (5)
	African American	60.5% (46)
	Hispanic	30.3% (23)
	Other	2.6% (2)
Grade at enrollment	1st grade	50.0% (38)
	2nd grade	30.3% (23)
	3rd grade	19.7% (15)

^*Note. There is no difference on any demographic variable for child participants in the study (n = 76) compared with the overall RCT sample (N = 403).

Details about child recruitment procedures for the randomized control study are provided elsewhere (Wyman et al., 2012; Wyman, et al., 2010). Briefly, over three school years, 2213 kindergarten-3^rd grade students across five schools were screened for eligibility for the Resilience Project using a modified Teacher Observation of Classroom Adaptation – Revised (TOCA; Werthamer-Larsson et al., 1991), administered to all teachers for all students. A total of 37.7% (834) of the students scored in the top tercile on the Authority Acceptance (AA) subscale that measures aggressiveness. After exclusions (e.g., sibling participating), 651 students were eligible and 403 (62% of eligible students with these high aggressive scores) were enrolled and randomized into the intervention study (individual Mentoring n = 203; control group n = 200). Fully 97% (197) of the students in the intervention (“Mentored”) group received the entire Year 1 of the intervention. The current study focuses on implementer fidelity during the first year of the program with just over a third (n = 76) of the Mentored children. This subset was chosen so that there would be three measures per child in our 206 coded sessions for fidelity.

Measurement development

Development of implementer fidelity measures in the one-to-one child component occurred during a pre-implementation year of the program, before enrollment in the trial began. In order to capture two critical dimensions of implementer fidelity, measures were developed for the purpose of coding implementer adherence to the manual and competence in delivering the intervention material (Schoenwald et al., 2010). We assessed fidelity in one intervention; therefore, we did not assess treatment differentiation.

Adherence

Because each lesson has content specified in the manual, unique measures of adherence were developed for each lesson. Lessons were chosen for analyses because they addressed key intervention skills and reflected phases (early, middle, late) of the first year of the intervention, as well as a broad range of anticipated difficulty of implementation. Videotapes of five lessons were used to produce fidelity ratings. Each lesson from the program manual (Cross & Wyman, 2004) follows an identical structural format, and each adherence measure was designed to assess the degree to which implementers administered each lesson as described in the manual (see Supplemental Material for the Adherence measure and coding system for Lesson 8; other adherence measures may be obtained through the authors). Lesson segments are: (a) Feelings “check-in”, (b) Review (of last lesson), (c) Introduction to the skill, (d) Teaching the skill, (e) Actively rehearsing the skill, and (f) Reviewing and generalizing the skill for future use. Given that each lesson focuses on novel content, distinctive criteria were identified for adherence to the manual when delivering each segment. One or more items are rated on each of these six segments, with ratings indicating that specific lesson material was not at all observed (0), partially observed (1), or fully observed (2). In addition to segment ratings, one point is given if implementers adhere to the designated timeframe of 25 +/− 3 minutes. Scores for overall adherence for a given lesson are determined by transforming the total number of points earned into a “percent of content delivered” score. Raw scores may also be reported for the adherence measures. Given that adherence items code the presence or absence of events (i.e., causal indicators), measuring internal consistency using Cronbach’s alpha is not appropriate (Smith & McCarthy, 1995; Streiner, 2003).

Competence

Unlike implementer adherence, the criteria used to determine the degree to which lessons were implemented competently are consistently applied to every lesson. Therefore, a single measure was developed to rate implementer competence for each of the coded lessons (see Supplemental Materials for the Competence measure and scoring system). Based on our intervention theory (Wyman et al., 2010), literature review of core competencies for working with children, and our experience with paraprofessionals, seven elements of implementer competence were identified as necessary for high-quality intervention delivery: (a) Emotional responsiveness (ability to respond empathically); (b) Boundaries (ability to maintain appropriate psychological/physical boundaries that promote the child’s autonomy); (c) Language (use of developmentally appropriate language to convey concepts/skills and the implementer’s empathy); (d) Pacing (ability to strategically and sensitively adjust lesson pace to the needs of the child); (e) Active learning/practice (ability to effectively use interactive strategies, such as demonstrations and role-plays, to introduce, teach and reinforce concepts/skills); (f) Individualizing/tailoring (ability to flexibly tailor teaching of the concepts/skills for the child and his/her context); and, (g) Use of in vivo learning opportunities (ability to use spontaneous material such as the child’s presentation to teach or reinforce a skill). Based on previous work by Forgatch and colleagues (Forgatch et al., 2005), domains of competence were rated using a 9-point scale categorized as: Good work (7–9), Adequate work (4–6), and Needs work (1–3).

Coding procedures

Five coders (two developers and three trained coders) rated entire 25-minute lessons. Trainees reviewed the program manual for familiarity with the requirements of the intervention, and initial exposure to coding rules and activities occurred in a group format where trainees coded alongside the measure developers. Group codes and the rationale for each scoring decision were discussed. Once a coder reached an acceptable level of reliability (> .80) with gold standard tapes, s/he was approved for independent coding and coded the same sessions as experienced coders. Coder disagreement was discussed with the coding team to reach a consensus. Codes were tracked over time and periodically analyzed for inter-rater reliability using intra-class correlations (ICC; Cicchetti, 1994). Coders were assigned to rate adherence or competence (sometimes both) for a variety of lessons.

Mentors shared cameras for recording at times due to technical or scheduling difficulties. Thus, most but not all lessons were recorded and retained. We planned to code an early, middle and late session for each randomly selected child, blocked by gender, grade, and baseline teacher rating. Videotaped lessons were coded for five of 14 intervention lessons and randomly assigned to the coding team. We identified comparable lessons delivered close in time that focused on the same skill (i.e., 5 and 6; 8 and 9) to ensure that each dyad had three equivalent observations for coding. For each child participant in the study, three lessons from the five selected for coding from the first year of the program were rated (lesson 5 or 6, 8 or 9, and 11) except three children who had only two lessons coded due to technical omissions. Coders overlapped on 50% of observations for coding. Coder group drift was assessed by re-coding 10% of the rated lessons throughout the 2 years of coding. An average of 21 sessions per implementer were coded with a range of 9 to 35 observations (M = 20.6; SD = 7.5). We coded 206 videotaped sessions reflecting a representative sample of (76 of 203) implementer-child dyads.

Coders also rated the child on one item that reflects behavior that might be challenging to even the most skillful implementer using a 3-point scale from “not behaviorally challenging” to “very behaviorally challenging.” Because only 1.3% of observed sessions rated a child as “very behaviorally challenging”, this category was combined with “somewhat” challenging into one category (high challenge). Twenty-eight percent (28%) of our sample was dichotomously coded as “high” on child behavior challenge. The majority of children demonstrated challenging behavior on only one observation and a minority (n = 7) demonstrated challenging behaviors on all 3 observations. Gender was not related to child challenge scores (χ²(1) = 0.24, ns). The resulting dichotomous Child Challenge score was used in the multi-level analyses of fidelity.

Data collection

Child participants

Seventy-six of the 203 mentored students (37.4%) were selected for the current study with the following blocking criteria: (a) grade, gender, and baseline TOCA scores to reflect the total child sample, and (b) equal number of dyads coded across the duration of the trial per implementer. Demographics of the sub-sample are provided in Table 1. A comparison of students in the observed group to the overall sample showed they did not differ significantly on sex, age, race/ethnicity or grade at time of enrollment.

Outcome Measures

Six outcome measures from the RCT were examined at the end of the first intervention year to assess criterion-related validity of the fidelity measures. (See Wyman et al., 2013 for details about outcomes measures). Briefly, teacher-reported child aggression was assessed with a shortened version of the Teacher Observation of Classroom Adaptation – Revised (TOCA; Werthamer-Larsson et al., 1991), including the 7-item Authority Acceptance scale (TOCA:AA). The AA measure was completed before a child’s random assignment to condition (baseline) and 6 months after the intervention was initiated to measure proximal outcomes. We found no effect for gender on AA. Child-rated symptoms were measured by individual administration of the Dominic Interactive (DI; Valla et al., 2000), a computerized self-report interview for 6 – 11 year old children that assesses symptoms of seven DSM-IV (American Psychiatric Association, 1994) diagnostic categories that resulted in two subscale totals: Internalizing and Externalizing. Parent ratings on child behavior were three scales from the Youth Outcome Questionnaire 2.0 (YOQ; Wells et al., 1996) at baseline and 6 months: Intrapersonal Distress (Internalizing), Behavioral Dysfunction, and Conduct Problems. Parent and child measures were administered before the intervention and after Year 1. All scales had good internal reliability (Wyman et al., 2013).

Statistical Analyses

We first examined descriptive summaries for the Adherence and Competence measures of implementer fidelity, and inter-rater reliability using intra-class correlation (ICC; Cicchetti, 1994). An exploratory factor analysis (EFA; Floyd & Widaman, 1995) was conducted to assess the factor structure of the Competence measure across lessons. The factor structure of the Competence measure from EFA was validated using internal consistency coefficients. Because the items for Adherence measures are causal indicators, and not expected to be highly correlated with each other (Smith & McCarthy, 1995), internal consistency analysis was not conducted for these measures. Multi-level modeling (child, implementer) was conducted using the SAS MIXED procedure to examine how much of the variance in Competence and Adherence measures was attributed to the implementer. We used multilevel analyses to validate that our implementer fidelity measures reflect mentor’s performance and not other variables such as those accounted for by fixed effects or by the child within mentor. Implementer and child within Mentor were entered as random effects in the analyses. At the same time, we tested influence of fixed effects to partial out their effects and allow us to examine only implementers’ behavior with respect to fidelity measurement: lesson number, the implementer’s years in the study, child gender, baseline aggression measured by the TOCA:AA, child age, and observed Child Challenge scores. We did not include a third level (observation) as no variance was available after accounting for lesson and child challenge. The same analytic strategy was applied to both implementer fidelity measures. Structural equation models (SEM), implemented in MPlus (Version 6, Muthén & Muthén, 1998–2010), were used to test criterion-related validity of the measures by creating latent variables for Adherence and Competence within student to predict 6-month outcome measures from three sources of measurement (child, parent, teacher). Three indicators consisting of Adherence or Competence scores for each mentor-child dyad at lesson 5 or 6, 8 or 9 and 11 were used to construct the latent variables. We controlled for baseline of these measures and age in the models. We used the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI) to assess fit (Browne & Cudeck, 1993; Hu & Bentler, 1999; Tabachnick & Fidell, 2007) and standard Wald-type tests to measure the strengths of predictive relationships.

Results

Inter-relater reliability

Table 2 shows the ICC coefficients for inter-rater reliability on Adherence, Competence and Child Challenge within lesson among raters. Lesson 9 was used for testing new coders’ reliability; thus, a higher number were coded by two or more raters. ICCs were all within the good-excellent range (.66 – .92; Cicchetti, 1994). Inter-rater reliability was stable across lessons and measures. Overall, approximately 50% of observations were coded by more than one rater. Coder drift over time was assessed by surreptitiously reassigning 10% of observations to coders over the course of the study. The agreement between original and re-assigned ratings was high (intraclass r = .80 to .90 across Adherence, Competence and lesson numbers).

Table 2.

Sample Descriptives and ICC for Adherence and Competence Measures

		Adherence				Competence			Child Challenge
Lesson	n	M (SD)	range	% range	ICC	M (SD)	range	ICC	M (SD)	range	ICC
5 a	17	13.88 (3.64)	4–19	19–90%	0.92	39.35 (9.69)	21–52	0.84	1.11 (0.32)	1–2	-
6 b	57	13.04 (3.84)	5–20	21.7–87%	0.78	36.49 (10.16)	18–55	0.82	1.29 (0.46)	1–2	1.00
8 b	18	15.11 (2.63)	10–19	43.5–82.6%	0.68	34.67 (9.31)	20–49	0.79	1.44 (0.60)	1–3	0.94
9 a	60	14.58 (3.44)	5–21	23.8–100%	0.66	36.20 (10.52)	15–59	0.77	1.40 (0.52)	1–3	0.80
11 b	54	15.12 (3.73)	4–23	17.4–100%	0.81	33.28 (10.47)	16–51	0.76	1.33 (0.51)	1–3	0.84

Note.

^a= Adherence range = 0 – 21;

^bAdherence range = 0 – 23;

Competence range = 7–63. - Not enough separate ratings of Lesson 5 Child Challenge to calculate ICC.

Competence items across implementer were found to create one coherent factor with a strong eigenvalue of 1.577 (other eigenvalues < 1) accounting for 80% of the variance on average by lesson. Internal consistency for the Competence scale across the 5 lessons was high, with Cronbach’s alphas ranging from .92 to .95. The correlation between raw Adherence and Competence scores was relatively high, r = .68 (p < .001). Although the measures of these two constructs share 46% of the variance, there remains substantial unique variance (54%). We further explored the relationship between the two measures by examining the correlation within implementer. The correlation of Adherence and Competence ranged from a significant .747 (p < .001) to negative and non-significant (−.174), with six of 10 mentors having a positive and significant correlation (.458 – .747). This variability underscores the importance of examining the two constructs separately and using a multilevel approach.

Descriptive data

Descriptive statistics for Competence and Adherence by lesson are shown in Table 2. The possible range for the Competence measure is 7 to 63, based on 7 items and a 9-point Likert rating scale. Ranges for raw scores of the Adherence measures differ slightly across lessons (raw scores from 0 to 21 for lessons 5 and 9; 0 to 23 for lessons 6, 8 and 11), and were recoded for a scale of 0 to 100 to reflect percent of lesson content delivered. We did not similarly rescale the Competence scores because it is not a ratio scale and because the range was consistent across lessons. There were no outliers. For the Competence measure, observed ranges were consistent across lessons and overall fell in the “adequate work” category based on the means/standard deviations. For Adherence measures, ranges were also consistent across lessons. Lesson 8 appeared to differ from the other lessons in terms of range; however, the sample is comparatively small for this lesson and a significance test showed no difference across lessons on the measure. Adherence scores indicated that on average 64% of program content was delivered by implementers to child participants (range: 56.7% to 69.4%).

Multi-level analyses of fidelity measures

Results of the multi-level analyses examining the variance in Competence and Adherence measures attributed to the implementer, and to the child within implementer, are summarized in Table 3. These analyses were conducted to validate that our measures reflect implementer behavior and not other variables such as child baseline aggression or implementer years of experience. For Competence, 67.7% of the variance was explained by implementer, and child within implementer did not contribute a significant amount of variance (est = .12, 0.1% variance explained). These findings were unchanged with the addition of fixed effects. Child gender, age and teacher-rated aggressive behavior were not related to implementer competence. Across implementers, years of experience in delivering the intervention did not impact competence scores. Implementers tended to have lower competence on lesson 11 than on lessons 5, 6, 8, or 9, (β range 1.55 – 4.71, SE range 1.13 – 1.75, p range .008 – .35) and were more competent with less challenging students (β = 3.41, SE = 1.03, CI = 1.39, 5.43, p = .001).

Table 3.

Multilevel Analyses of Measures’ Psychometrics

	Competence	Adherence
Fixed part – Level 1	F (df= num, den)	F (df= num, den)
Lesson number	2.78 (4,159)*	5.04 (4,157)**
Implementer years experience	0.19 (2,70.5)	0.92 (2,74.9)
Child gender	0.05 (1,51.2)	6.42 (1,57.7)*
Authority Acceptance	0.03 (1,50.5)	0.19 (1,57.1)
Child age in months	0.85 (1,54.2)	0.17 (1,62)
Child Challenge	11.06 (1,151)**	13.21 (1,171)**
Random part – Level 2	Percent of Variance (coeff, SE)	Percent of Variance (coeff, SE)
Implementer	67.7% (71.32, 34.56)	41.1% (4.97, 2.59)
Child within implementer	0.1% (0.12, 3.08)	6.5% (0.78, 0.68)

^*p < .05;

^**p < .01;

Percent of Variance = ICC*100; Lesson number was dummy coded using a contrast statement.

For Adherence, 41.1% of the variance was explained by implementer, and child within implementer did not contribute a significant amount of variance (est = .78, 6.5% variance explained). These findings were unchanged with the addition of fixed effects. Adherence was not affected by child age or teacher-rated aggressive behavior. Across implementers, years of experience delivering the intervention did not impact Adherence scores. Implementers were significantly less adherent on lesson 6 than on lessons 5, 8, 9, or 11. Implementers were more adherent with girls than with boys (β = 1.18, SE = 0.47, CI = 0.26, 2.10, p = .014,), and with less challenging students (β = 1.72, SE = 0.47, C I = 0.80, 2.64, p <.001). We also examined implementer by lesson interactions for both Adherence and Competence and found the interaction was not significant for either construct (p =.13 and p =.65, respectively).

Measures’ criterion-related validity: Predicting proximal outcomes

We examined criterion validity using structural equation modeling (SEM) by creating latent variables for Adherence and Competence within student to predict 6-month outcome measures from three sources: child, parent, and teacher reports. Seventy-five children (99%) had Adherence and Competence score data for three lessons: lesson 5 or 6, lesson 8 or 9, and lesson 11. These scores were used in structural equation modeling as manifest indicators of the latent Adherence and Competence measures. We controlled for baseline differences as well as age in the models. Child challenge was not used in these analyses because very few children were challenging in more than one lesson. We therefore considered the Challenge score to reflect a transient state rather than an individual characteristic of the child. We did not include additional covariates because the small sample necessitated parsimonious, yet meaningful, models.² We also examined ICCs for implementer on outcomes: All were low (< .09) indicating that it is not the individual implementers themselves who directly affected outcomes but rather the implementers’ behavior with the child. Finally, given our small sample size and the moderate correlation between Competence and Adherence, these analyses were run separately.

In all models, lesson values for Adherence and Competence significantly loaded onto their respective latent constructs, with similar loading values across models. SEM results are shown in Table 4 (values in columns three to five are regression weights and values in columns six to nine are model fit statistics). Higher Adherence and Competence both predicted reduced child-reported externalizing behaviors with the effects being strongly significant. Neither predicted child-reported internalizing behaviors. Within the parent-reported measures, higher Adherence and higher Competence predicted reduced conduct problem behaviors, and higher Adherence predicted reduced internalizing and behavioral dysfunction problems. Therefore, the actual amount of content delivered (Adherence) as well as quality with which the program was delivered (Competence) predicted child participants’ response to the intervention at 6 months by both child and parent report. Neither Adherence nor Competence predicted teacher-rated externalizing behaviors (Authority Acceptance). Model fit varied; however, with the exception of parent-reported externalizing and Competence, at least one fit index was acceptable for all models with significant effects. Given the small sample size and the fact that we were using SEM for criterion-related validity and not for hypothesis testing, these lower acceptable or marginal fit statistics were expected and acceptable.

Table 4.

Criterion Related Validity Using SEM to Predict Outcomes From Fidelity Scores

Source	Outcome	Variables predicting outcome				Model fit
		Adherence	Baseline	Age	χ2	CFI	TLI	RMSEA
Child	Externalizing	−2.22***	.54***	.18***	29.57***	.91	.86	.10
	Internalizing	<.001	.03+	.11***	17.24*	.97	.96	.06
Teacher	Authority Acceptance	.12	.58***	.01	19.69*	.83	.73	.15
Parent	Conduct	−2.84***	.71***	−.06	18.17*	.89	.82	.14
	Internalizing	−3.56***	.76***	−.08	17.56*	.89	.83	.13
	Behavior Dysfunction	−2.88**	.65***	−.02	12.89	.95	.92	.08
		Competence	Baseline	age	χ2	CFI	TLI	RMSEA
Child	Externalizing	−1.49**	.57***	.16***	34.70***	.93	.89	.11
	Internalizing	<.001	.03+	.11***	17.24*	.97	.96	.06
Teacher	Authority Acceptance	−.01	.57***	.01	8.68	1.00	1.00	.00
Parent	Conduct	−1.82**	.67***	−.08	23.30**	.87	.80	.17
	Internalizing	−.92	.76***	−.13	19.26*	.90	.85	.15
	Behavior Dysfunction	−.73	.66***	−.06	14.75+	.95	.93	.09

Note. Child N = 75, Implementer N = 10; df for χ² = 9 for all analyses;

⁺p < .10;

^*p < .05;

^**p < .01;

^***p < .001.

Discussion

In this study, implementer fidelity was conceptualized as adherence to the manual and competence in program delivery, and measures were developed for each. Because we had not conducted prior fidelity research on the intervention, and had empirical questions about the association between the fidelity measures and the relationship of each with child outcomes, we chose observational methods to assess implementer behaviors with child participants. Observational measures of adherence have been used frequently in the literature; competence has been examined less often (Schoenwald & Garland, 2013). Although Adherence and Competence measures were correlated in this study, the importance of creating and studying the two components of fidelity for purposes of training, implementation monitoring, and for outcome studies remains relevant. In particular, we found that although most implementers were comparable on both measures, some were quite discrepant. Thus, examination of multiple observations per Mentor revealed important variability at the individual level. Moreover, in another study using similar methods and measures, adherence and competence shared less variance (25%, Cross et al., 2014; Gould et al., 2013).

A primary finding of our study is that rigorously developed implementer fidelity measures of Adherence and Competence were associated with enhanced outcomes at the end of the first intervention year, which demonstrated the measures’ criterion-related validity. As expected, higher Adherence and Competence scores were associated with lower levels of child-reported externalizing behaviors and lower parent-reported conduct problems. Higher Adherence was also associated with lower internalizing and behavioral dysfunction per parent report. That is, there were clear variations in how well implementers delivered the intervention, and higher fidelity was associated with more positive proximal outcomes reported by children and parents. Neither measure of fidelity was related to proximal child outcomes on the teacher measure of aggressive-disruptive behavior. Our primary outcome study assessing impact of the RRP intervention over three school years found intervention impact on reduced classroom behavior problems, measured by teacher report, only for older children in the sample and under conditions of high parent participation (Wyman et al., 2013). Reducing aggressive-disruptive behaviors that generalize to improved classroom behavior may require more intensive involvement of parents, which is not captured by our fidelity measures.

Our implementer fidelity measures had high inter-rater reliability and high internal consistency for the Competence measure. There were generally higher ratings and less variability in Adherence than Competence. While competence in this effectiveness trial was generally lower than fidelity reported in efficacy trials (e.g., 50–60% versus 70–80%), this may be due to a number of factors. First, fidelity in other studies focuses primarily on adherence, which was higher than competence scores. Secondly, our ordinal coding scheme may be more demanding than other rating systems. Thirdly, the implementers in this study, who were paraprofessionals, may not have delivered the intervention as well as those in some other prevention studies (e.g., teachers, professional staff).

We examined our measures to answer the question: Are we measuring implementer fidelity or are other variables contributing to the fidelity scores? The most important finding from multilevel models is that the implementer accounted for the greatest amount of variance in the scores on each of the measures. In fact, for Competence, implementer accounted for over two-thirds of the variance in scores, and, for Adherence, over forty percent of the variance was accounted for by implementer. Because child within implementer was not significant, we can conclude that fidelity scores reflect the quality of the delivery of the program content by the Resilience Mentor, not attributes of the child.

There were, however, some differences in implementer fidelity associated with lesson number, gender, and child behavioral challenges. Specifically, compared to the other four rated lessons, implementers were less competent in their delivery of a lesson (11) that involved abstract concepts and associated behavioral skills (i.e., developing self-soothing strategies to manage problems or situations that are not wholly in the child’s control). Increased training and clarity in the manual may be required for interventionists to deliver the lesson in a manner that is individualized to the child without sacrificing core competencies. We also found that implementers were less adherent with a different lesson (6), which focuses on engaging the child in learning and practicing labeling feelings using a “charades” type of game. It appears that the implementers’ familiarity with the general game of charades may have interfered with carrying out the activities as prescribed. In other words, implementers “stuck to what they knew” about charades and missed several important content points and activities as a result.

These findings point to the strength of multiple assessments of fidelity. Had we relied on a single lesson to assess fidelity, we would have risked making erroneous conclusions about variability in fidelity and, potentially, the association with child outcomes. Importantly, our analysis indicates the feasibility of making adjustments to which lesson is assessed, as we found no indication of important interaction effects of lesson by facilitator within implementer-child dyad. This finding also highlights how information about implementation fidelity can be used to inform and enhance training. When training new implementers, we now attend closely to these lessons since our data indicate that they are more challenging to implement fully with high quality.

Based on our multi-level analyses, we also found that implementer fidelity varied in terms of higher Adherence but not Competence with girls. That is, our implementers delivered more of the manualized content to girls but were equally competent in their delivery (i.e., emotionally responsive, used active learning strategies, tailored activities) with boys and girls. Another way to view this finding is that implementers went “off manual” more with boys than with girls and had lower Adherence. As others have noted (Aarons et al., 2012), adaptations of programs during implementation in real world settings are common and can reflect positive “edits” to the intervention. Our Adherence measure coded prescribed elements only and scores reflect only what occurred; we did not note ‘non-manualized’ but effective adaptations of the content. One interpretation of the finding that implementers demonstrated lower adherence with boys but with competence equal to their work with girls, is they changed some of the content to be competent as they individualized the program for boys. We plan to examine this hypothesis by rating ‘off manual’ content as consistent or inconsistent with the program and may integrate positive adaptations into the intervention and training program.

In terms of examining variations on fidelity across lessons, we found that when children demonstrated behaviors that coders deemed challenging, implementers demonstrated lower levels of Adherence and Competence. It is important to note that the challenges were not systematic or related to specific children – that is, over three coded lessons only seven students (9.2 %) demonstrated challenging behaviors across each observation. Thus, it was not the individual child but specific behaviors that challenged implementer fidelity. Because our measures rely on observational data, it is possible to examine child behavior at a more microlevel to understand what those specific behaviors are and modify our training and supervision processes to increase implementers’ skill in addressing these behaviors such that fidelity is not compromised. The finding that challenging behaviors in child participants negatively impacted implementer fidelity is likely to be generalizable to other similar interventions.

We recognize that measuring fidelity with observational measures in community-based, non-research settings, has numerous practical challenges (Schoenwald & Garland, 2013). Methodological rigor cannot trump relevance in real world contexts. We have begun to address the challenge by developing ‘user friendly’ supervision tools based on our research measures of adherence and competence. These measures, developed with extensive input by community-based supervisors, have been shown to correlate highly with fidelity measures used in research. Clinical supervisors are trained to observe and reliably rate Mentors delivering the intervention (either in vivo or via video recordings) to inform supervision. Reliability checks are conducted with the research team. Monitoring implementer fidelity is, therefore, integrated into the community-based program supervision model. The translation of research measures to clinical tools is one strategy that may be useful for implementation in real world community settings.

There are several limitations to this study. Although the number of individual observations is high and the number of students substantial, the number of implementers is smaller and therefore the overall N is small. To account for the nested nature of the data, we conducted multi-level modeling to uncover variations in implementer behaviors across lessons and in response to extreme challenges by child participants. Thus, our thorough examination of the measures and outcomes balances the limited number of implementers in the study. In addition, the generalizability of our findings may be limited. Ours is a school-based preventive intervention delivered by paraprofessionals; findings may be relevant to programs with similar foci, setting or implementers. Although adherence ratings of implementer fidelity are necessarily intervention-specific, the approaches presented here are widely applicable. Moreover, the competencies we identified and reliably measured may be applicable for other programs. In fact, we have received requests for our Competence measure for use in other interventions. The fit statistics for the criterion related validity structural equation models varied – some were acceptable but others were not as strong. Given the nature of our research question they are acceptable. Nevertheless, replication of these findings in a larger sample is an important next step. It is also possible that the correlation between adherence and competence in this study may be due in part to coder bias because raters sometimes completed both measures for a single observation. Relatedly, the correlation, coupled with our sample size, did not allow us to examine criterion-related validity in models including both measures.

Finally, we measured implementer fidelity in only one component of the intervention. Wyman et al.’s (2013) study of overall program outcomes found a very strong effect for parent involvement on child outcomes. We do not have recordings of the parent component, however, and cannot measure fidelity of program delivery in these meetings. It is possible that implementer success engaging parents in this component of the intervention is related to their competencies delivering the child component (i.e., emotional responsiveness and tailoring). Wyman et al. (2013) also found that age-related factors were associated with child outcomes indicating that younger children were not cognitively ready for participation in the intervention. In these cases, implementer fidelity would not likely be a factor in outcomes. We focused our study on implementer delivery of the child component on proximal outcomes of Year 1 of the intervention only. In a subsequent study, we will extend our analyses to include fidelity ratings for Year 2 of the intervention and will examine implementer fidelity in the context of long-term outcomes for those children who received the intervention. To complete these analyses, we will include estimates of implementer fidelity for children for whom we do not have observed ratings.