The Trajectory of Fidelity in a Multiyear Trial of the Family Check-Up Predicts Change in Child Problem Behavior

Abstract

Therapist fidelity to evidence-based family interventions has consistently been linked to child and family outcomes. However, few studies evaluate the potential ebb and flow of fidelity of therapists over time. We examined therapist drift in fidelity over four years in the context of a Family Check-Up prevention services in early childhood (age 2–5). At age 2, families engaging in Women, Infants, and Children Nutritional Supplement Program (WIC) services were randomized and offered annual Family Check-Ups. Seventy-nine families with a child in the clinical range of problem behaviors at age 2 were included in this analysis. Latent growth modeling revealed a significant linear decline in fidelity over time (M = −0.35, SD = 0.35) and steeper declines were related to less improvement in caregiver-reported problem behaviors assessed at ages 7.5/8.5 (b = −.69, p = .003; β = −.95, CI: −2.11 | −0.22). These findings add to the literature concerning the need to continually monitor therapist fidelity to an evidence-based practice over time to optimize family benefits. Limitations and directions for future research are discussed.

Establishing the fidelity with which an intervention is delivered is a critical dimension of understanding intervention effects in both efficacy and effectiveness trials (McHugh & Barlow, 2010). Fidelity is typically comprised of two key components: adherence (the extent to which specific procedures were delivered) and competence (the level of skill demonstrated). Parent training interventions are a well-established approach for the prevention and treatment of child and adolescent behavior problems (e.g., Lundahl, Risser, & Lovejoy, 2006) and community implementation is a high priority (National Research Council and Institute of Medicine, 2009). Although fidelity to parent training programs has been consistently associated with clinically meaningful child and parent-level outcomes (e.g., Forgatch, Patterson, & DeGarmo, 2005; Henggeler, Melton, Brondino, Scherer, & Hanley, 1997; Hogue et al., 2008; Ogden, Hagen, Askeland, & Christensen, 2009; Smith, Dishion, Shaw, & Wilson, 2013), few studies have evaluated fidelity to evidence-based parenting programs with the same families longitudinally.

The notable exception is a series of studies concerning implementation of Parent Management Training–Oregon Model (PMTO). Forgatch and DeGarmo (2011) evaluated ratings from three time points during training in PMTO and found that fidelity increased and became less variable over time, and that fidelity was sustained across three generations of therapists trained in PMTO as part of a large-scale implementation effort in Norway. Also, Forgatch and colleagues (2005) and Hukkelberg and Ogden (2013) found, in studies from two separate trials of PMTO, that higher fidelity, assessed at three time points, predicted significant reductions in parent-reported child behavior problems.

The Family Check-Up

The Family Check-Up (FCU) is a brief, individually-tailored intervention intended to motivate use of effective parenting skills to improve youth adjustment. The FCU comprises three meetings: initial contact, ecological assessment, and feedback. Feedback to caregivers emphasizes family strengths, yet draws attention to possible areas of change. Findings from the Early Steps multisite randomized trial indicate that families assigned to the FCU condition experience reductions in youth problem behaviors during preschool (e.g., Dishion et al., 2008) and at at school age (e.g., Dishion et al., 2014). The Early Steps trial was delivered using a health maintenance framework consisting of offering the FCU to families in the intervention condition each year beginning at age 2. The COACH rating system (Dishion, Knutson, Brauer, Gill, & Risso, 2010) was developed to assess competent adherence to the FCU feedback session and is based on Forgatch and colleagues’ work with the PMTO model. Smith and colleagues (2013) examined fidelity to the FCU in a subsample of families from this trial who rated the child in the clinical range for behavior problems at age 2. Greater fidelity to the FCU at age 2 was related to improvements in parent-reported problem behaviors two years later (age 4). This effect was mediated by ratings of observed in-session caregiver engagement in the FCU at age 2 and improvements in observed positive parenting behaviors, assessed at age 3.

The Current Study

An important component to understanding therapist fidelity to interventions administered to the same families over time is the evaluation of drift; that is, changes in the delivery of the intervention over time. The health maintenance framework of the Early Steps trial provides a unique opportunity to examine the effect of drift as related to outcomes at the family level. Our aims were twofold: (1) evaluate drift in fidelity to the FCU across the first four years of a multisite randomized efficacy trial (child ages 2–5) and (2) examine the effect of drift in fidelity on changes in child oppositional and aggressive behaviors at ages 7.5 and 8.5. We hypothesized that the trajectory of fidelity would be related to intervention outcomes for families with children rated in the clinical range for behavior problems at study entry. We followed the same 79-family clinical subsample from the study by Smith et al. (2013). We applied latent growth modeling with covariates that could account for drift (changing therapist, study site, caregiver depression, income, baseline child noncompliance).

Method

Participants and Procedures

This study examined a subsample of the original 731 mother–child dyads (49% female children) recruited from the WIC services in three geographically and culturally diverse U.S. regions near Charlottesville, VA (188 dyads), Eugene, OR (271), and Pittsburgh, PA (272), for a randomized trial of the FCU. See Dishion et al. (2008) for comprehensive information regarding the recruitment and randomization protocol and the participant demographics for the full sample of the main study. Inclusion in the current study was determined by either the primary or alternate caregiver reporting clinical range scores on the Externalizing scale of the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2000) at age 2, which assesses a broad range of problem behaviors. Seventy-nine families (VA, 23; OR, 28; PA, 28) met criteria (see Smith, Dishion, Shaw, et al., 2013 for subsample characteristics). Caregivers and children who agreed to participate in the study completed a 2.5-hour assessment conducted in the family’s home. During the assessment, the caregiver(s) completed questionnaires and the family participated in a series of developmentally appropriate interaction tasks lasting approximately 20 minutes. The home-based assessment was the first contact and occurred prior to randomization at age 2 and prior to families in the intervention condition being offered the FCU feedback. Participation in the feedback session was voluntary each year, resulting in variable rates of engagement: all 79 families in this subsample received feedback at age 2. At subsequent annual FCUs (age 3 to age 5), families received an average of 2.22 (SD = 0.98) feedback sessions (0 feedback sessions = 8%; 1 = 17%; 2 = 23%; 3 = 53%). Additional information about the intervention procedure appears in Dishion et al. (2008) and Dishion et al. (2014). In addition to within-site supervision, weekly meetings were held to maintain cross-site fidelity.

Measures

Fidelity

Fidelity to the FCU is assessed using the COACH rating system (Dishion et al., 2010), which jointly assesses adherence to the FCU model and the quality of the delivery (competent adherence) during the feedback session. Trained coders observationally rated videotaped sessions of the interventionist on 5 core components of the FCU model using a 9-point scale (needs work: 1–3, competent work: 4–6, excellent work: 7–9): Conceptual accuracy and adherence; Observant and responsive to client needs; Actively structures sessions; Careful and appropriate teaching; Hope and motivation are generated. Low scores indicate minimal knowledge and skill, while high scores indicate clear understanding of the principles and mastery of the skills in delivering the FCU. A mean score of the 5 COACH dimensions is calculated. The scores reported in Smith et al. (2013) were used for age 2 in this study. Among the 79 families receiving feedback at age 2, 53, 41 and 45 videotapes were rated at ages, 3, 4, and 5 respectively. Internal consistencies of COACH mean scores were ≥ .90 at each age. A minimum of 20 percent of the sessions each year were randomly selected for double coding to determine inter-rater reliability. Intra-class correlation coefficients were in the “good” range: age 2 = .74; age 3 = .73; age 4 = .77; age 5 = .71. Coders participated in weekly meetings to counter drift.

Oppositional and aggressive behaviors

We created a measure of child oppositional and aggressive (OPP/AGG) behaviors from the Child Behavior Checklist (CBCL) parent and teacher report versions of the school-age forms (Achenbach & Rescorla, 2001) to assess behavior problems in the home and classroom. Primary and alternate caregivers and the primary teacher of study participants completed the CBCL at the age 7.5 and age 8.5 assessments. Respondents rated the applicability of eight items that map onto DSM-IV (American Psychiatric Association, 2000) criteria for oppositional defiant disorder and conduct disorder using a 3-point Likert scale in which 0 = not true, 1 = somewhat, sometimes true, and 2 = very true, often true. We averaged these items to create a composite score. Either report was used as the outcome when only 1 year was available. A mean of the two scores was used when data were available at both time points. These three scales comprised the indicators of the OPP/AGG behavior latent construct. Internal consistencies for each version of the CBCL were greater than or equal to .82 each year.

Covariates

Variables that are conceptually and empirically related to fidelity and/or child problem behaviors were included in the analyses. Primary caregivers’ depressive symptomatology was assessed with the 20-item Center for Epidemiological Studies on Depression Scale (Radloff, 1977) at child age 2. Ratings were provided on a scale ranging from 0 (less than a day) to 3 (5–7 days) and are summed. Internal consistency was good (α = .74). Income was included in the model due to its relation with problem behaviors in childhood (Yoshikawa, Aber, & Beardslee, 2012). Seventy-six percent of families were below the national poverty line at study entry. We included intervention site (OR, PA, VA) to control for local adaptation (see Smith, Dishion, Moore, Shaw, & Wilson, 2013 for an example in this trial), and therapist transfer, which could affect the within-family trajectory of fidelity (72% of families in the sample received services from one therapist; 28% had 2 different therapists). This item was dummy coded (1 = transfer, 0 = no transfer) for use in the analyses. Last, we controlled for child behavior problems at baseline using a measure of observed noncompliance (see Appendix).

Results

First, we conducted an unconditional latent growth curve (LGC) analysis in Mplus 7.2 (Muthén & Muthén, 2014) to model the trajectory of fidelity from age 2 to 5. Little’s (1988) MCAR test was nonsignificant (χ²(49) = 60.58, p = .124), indicating that missing data did not introduce bias into the analyses. The Bayesian estimator was used because it does not rely on large-sample theory and the normal distribution assumption; thus, it produces more accurate estimates with smaller samples (Lee & Song, 2004). Bayesian fit statistics¹ indicated that a linear model provided good fit to the data: Posterior Predictive P-value (PPP) = .11, and Posterior Predictive Checking (PPC) = (95% CI: −4.86 | 32.04). The estimated intercept (M = 5.57, SD = .89) and slope (M = −0.35, SD = 0.35) parameters in the unconditional LGC model (Figure 1) indicate that fidelity had a significant gradual decline of about one third of a point each year. Next, we added OPP/AGG and the covariates. The OPP/AGG scores were added in the model in two steps. First, a one factor model was fitted. Standardized factor loadings were .78, .77 and .50 for primary caregiver, alternate caregiver and teacher reports respectively. Then the estimated factor scores for each individual were saved. Second, the estimated OPP/AGG factor score was regressed on the intercept and linear slope of fidelity and the covariates (Figure 2). Importantly, none of the covariates were related to the LGC or OPP/AGG. Results revealed successful model convergence and adequate fit the data: Potential Scale Reduction² = 1.04, PPP = .24, and PPC = (95% CI: −24.49 | 50.20). After controlling for depression, income, noncompliance, intervention site, and therapist transfer, a less steep decline in fidelity, compared to the mean trajectory, was associated with lower OPP/AGG scores (b = −.95, p = .003, 95% CI: −2.11 | −0.22).

Figure 1.

Observed and estimated trajectory of the fidelity ratings from age 2 to age 5.

Note. The observed COACH scores are connected with the solid line. The estimated trajectory is represented by the dashed line.

Figure 2.

The trajectory of fidelity is related to child oppositional/aggressive behaviors in the home and school settings.

Note: Covariates with significant paths are presented. See Table 2 for full path analysis results. Unstandardized coefficients are in parentheses.

Discussion

The findings suggest that therapist drift in fidelity to parent training interventions can compromise outcomes. We are unaware of any studies of fidelity to a parenting intervention delivered to the same family across multiple years. Prior longitudinal studies in this area evaluated fidelity across generations of implementers (Forgatch & DeGarmo, 2011) and within an intensively delivered intervention program (i.e., 3 sessions of a 20-session protocol) (Forgatch et al., 2005; Hukkelberg & Ogden, 2013). Our study determine that fidelity to decline over time and this change was associated with greater reductions in children’s problem behavior at 7.5/8.5 years. Possible explanations for the modest drift in fidelity deserve discussion. Although drift did not differ statistically by study site (OR, PA, VA), our research team has found site differences in the delivery of some aspects of the FCU model (e.g., video feedback procedures; Smith, Dishion, Moore, et al., 2013). We held weekly 1-hour meetings with therapists from the three sites focused on problem solving challenging cases, with the bulk of the supervision occurring within site. We hypothesize that site-specific supervision and clinical meetings throughout the trial likely contributed to the lack of site differences in drift in fidelity. During this trial, the COACH system was developed in reaction to an apparent drift in fidelity noted by the program developer. COACH ratings were then used to provide individualized feedback to therapists during supervision procedures at age 5.

Another potential influence on drift could be a byproduct of the transactional nature of the therapeutic process. The therapist’s ability to demonstrate fidelity to a protocol is likely to be at least in part shaped by the client’s in-session behaviors. Patterson and Chamberlain’s (1994) “struggle-and-working through” hypothesis, for example, suggests that client resistance occurs when the therapist places demands to apply parenting skills. Resistance increases over time if the parent has not properly learned or applied these parenting skills and continues to encounter demands from the therapist and decreases with indications of improvement. In a sample of families whose children were referred for antisocial behavior, Stoolmiller, Duncan, Bank, and Patterson (1993) found clinically significant change in families whose initial resistance increased from baseline to mid-therapy and then declined as services continued, supporting Patterson and Chamberlain’s hypothesis that successfully working through initial growth in resistance is beneficial for the family. Further, the struggle-and-working-through hypothesis suggests that variation in client success can drive change in fidelity (e.g., Patterson & Chamberlain, 1994; Stoolmiller et al., 1993). Evaluating drift in fidelity could potentially illuminate the provider-client processes that promote or inhibit intervention effectiveness via protocol delivery.

The FCU model is unique in that families are seen repeatedly for multiple years, often by the same therapists. It is possible that this feature of the FCU model demands ‘refreshing’ therapists’ attention and focus, as there may be a tendency to accommodate services in response to client demands and characteristics. Future research would be well served to account for the potentially transactional relationship between fidelity and in-session caregiver behaviors, such as resistance (e.g., Stoolmiller et al., 1993) and engagement (e.g., Smith et al. 2013). Comprehensive models of fidelity highlight the importance of the therapist-client interaction (e.g., Berkel, Mauricio, Schoenfelder, & Sandler, 2011). Empirically examining this relationship would inform program development and refinement to increase effectiveness.

Finally, these findings suggest the need for a systematic, continuous consultation model that could support therapists and reduce drift as they work with families over time, to avoid individualized accommodations that are not evidence based. Further, it appears crucial to develop fidelity of implementation monitoring systems involving the use of evidence-based fidelity assessment measures, such as the COACH. These steps are critical to successfully transfer evidence-based interventions to community service delivery systems. Unfortunately, our small sample size prohibits a more nuanced evaluation of contributors to drift in fidelity. Insignificant effects of site and therapist transfer are, however, encouraging.

Table 1.

Intercorrelations Between Variables Included in the Final Model

(N = 79)
Variable	1	2	3	4	5	6	7	8	9	10
1. COACH composite score (age 2)	—	.33*	.34*	.18	.07	−.23*	.11	−.00	−.15	−.19
2. COACH composite score (age 3)		—	.52**	.45*	.02	.12	−.15	−.07	.16	−.21
3. COACH composite score (age 4)			—	.35*	−.11	−.12	−.11	−.02	−.25	−.13
4. COACH composite score (age 5)				—	.10	−.03	−.20	−.35*	−.10	−.33*
5. OPP/AGG factor score (age 7.5/8.5)					—	−.23	−.02	.18	.15	.10
6. Caregiver depression (age 2)						—	−.07	−.04	.19	.17
7. Income (age 2)							—	.01	−.05	.10
8. Observed child noncompliance (age 2)								—	−.06	.15
9. Intervention site									—	−.05
10. Therapist transfer										—
Mean	5.59	5.41	3.91	4.85	.001	18.99	3.58	.33	2.06	.28
Standard deviation	1.28	1.21	1.02	1.36	.27	10.85	1.74	.79	.81	.45

Note. OPP/AGG factor score = Composite of eight items on the Child Behavior Checklist (CBCL) related to DSM-IV criteria for oppositional defiant disorder and conduct disorder. Parent and teacher report versions.

Table 2.

Results of Path Analysis

N = 79
Model path	Posterior			95% Credibility Interval
	B	SD	β
Caregiver depression (age 2) → Fidelity LGC intercept	−0.02	.015	−0.19	−.468 | .136
Pittsburgh site → Fidelity LGC intercept	−0.49	.361	−0.24	−.569 | .124
Charlottesville site → Fidelity LGC intercept	−0.26	.407	−0.12	−.540 | .260
Therapist transfer → Fidelity LGC intercept	−0.44	.332	−0.20	−.505 | .087
Observed noncompliance (age 2) → Fidelity LGC intercept	0.13	.190	0.11	−.229| .424
Income (age 2) → Fidelity LGC intercept	0.05	.086	0.09	−.215| .376
Caregiver depression (age 2) → Fidelity LGC slope	0.01	.007	0.21	−.204 | .516
Pittsburgh site → Fidelity LGC slope	−0.06	.175	−0.07	−.488 | .353
Charlottesville site → Fidelity LGC slope	−0.15	.193	−0.17	−.543 | .355
Therapist transfer → Fidelity LGC slope	0.003	.163	0.003	−.389 | .366
Observed noncompliance (age 2) → Fidelity LGC slope	−0.14	.117	−0.28	−.655 | .274
Income (age 2) → Fidelity LGC slope	−0.09*	.043	−0.37*	−.678 | .057
Fidelity LGC intercept → OPP/AGG (age 7.5/8.5)	−0.05	.144	−0.16	−1.415| −.675
Fidelity LGC slope → OPP/AGG (age 7.5/8.5)	−0.69**	.333	−0.95**	−2.109 | −.221
Caregiver depression (age 2) → OPP/AGG (age 7.5/8.5)	0.01*	.005	0.35*	−.021 | .742
Pittsburgh site→ OPP/AGG (age 7.5/8.5)	−0.02	.156	−0.03	−.599 | .458
Charlottesville site→ OPP/AGG (age 7.5/8.5)	−0.12	.174	−0.21	−.769 | .390
Therapist transfer → OPP/AGG (age 7.5/8.5)	−0.01	.140	−0.02	−.449 | .496
Observed noncompliance (age 2) → OPP/AGG (age 7.5/8.5)	−0.02	.091	−0.05	−.743 | .493
Income (age 2) → OPP/AGG (age 7.5/8.5)	−0.05*	.040	−0.33*	−.964 | .063

Note.

^*p < .05.

^**p < .01.

^***p < .001. p-values are one-tailed.

APPENDIX

Child noncompliance

Staff completed macro-level ratings of the child’s compliance with caregiver directives observed during home visits using three items from the Coder Impressions Inventory (Dishion, Hogansen, Winter, & Jabson, 2004): cooperation with the caregiver (reverse scored), dysregulation and difficulty controlling behavior and emotion, and overall compliance (reverse scored). Interrater agreement (88%) and internal consistency (α = .86) were high.