Therapist Adherence in Brief Strategic Family Therapy for Adolescent Drug Abusers

Abstract

Objective

Therapist adherence has been shown to predict clinical outcomes in family therapy. In prior studies, adherence has been represented broadly by core principles and a consistent family (vs. individual) focus. To date, these studies have not captured the range of clinical skills that are represented in complex family-based approaches or examined how variations in these skills predict different clinically relevant outcomes over the course of treatment. In this study, the authors examined the reliability and validity of an observational adherence measure and the relationship between adherence and outcome in a sample of drug-using adolescents who received brief strategic family therapy within a multisite effectiveness study.

Method

Participants were 480 adolescents (age 12–17) and their family members, who were randomized to the Brief Strategic Family Therapist treatment condition (J. Szapocznik, U. Hervis, & S. Schwartz, 2003) or treatment as usual. The adolescents were mostly male (377 vs. 103 female) and Hispanic (213), whereas 148 were White, and 110 were Black. Therapists were also randomly assigned to treatment condition within agencies.

Results

Results supported the proposed factor structure of the adherence measure, providing evidence that it is possible to capture and discriminate between distinct dimensions of family therapy. Analyses demonstrated that the mean levels of the factors varied over time in theoretically and clinically relevant ways and that therapist adherence was associated with engagement and retention in treatment, improvements in family functioning, and reductions in adolescent drug use.

Conclusions

Clinical implications and future research directions are discussed, including the relevance of these findings on training therapists and studies focusing on mechanisms of action in family therapy.

As priorities in drug abuse treatment research have shifted toward transporting evidence-based treatments from research to community practice settings (Miller, Sorensen, Selzer, & Brigham, 2006; National Institute on Drug Abuse [NIDA], 2004), establishing and maintaining fidelity to treatment has achieved a high degree of importance. Treatment fidelity is challenged in real-world settings in many ways, as investigators typically have less control over therapist selection and the formation of procedures to monitor clinical activities. Also, therapists in real-world settings often carry larger caseloads than therapists in clinical trials and have multiple roles within their agency. In contrast to efficacy trials in which supervision and clinical services are covered within the budget and scope of work of the trial, community agencies’ business models require that they recover a certain number of billable hours from each therapist. Despite these concerns, recent studies have suggested cause for optimism by demonstrating that treatments can be transported to the community with high fidelity to core treatment philosophies and techniques (Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002; Liddle et al., 2006).

The successful dissemination of evidence-based interventions is particularly salient for family-based interventions for adolescents with drug use, delinquency, and associated problem behaviors because hundreds of community agencies (and thousands of therapists)— both nationally and internationally— have been trained to provide empirically based family services such as brief strategic family therapy (Szapocznik, Hervis, & Schwartz, 2003), functional family therapy (Alexander, Pugh, Parsons, & Sexton, 2000), multidimensional family therapy (Liddle et al., 2002), multidimensional treatment foster care (Chamberlain & Mihalic, 1998), and multisystemic therapy (Henggeler & Borduin, 1990).

Family therapy research has examined the critical relationship between treatment fidelity (adherence and competence) and clinical outcomes. The term adherence is used to denote the extent to which therapists are implementing key aspects of a clinical model, whereas competence refers to the quality with which the therapist implements these aspects. In general, findings indicate that higher rates of adherence have predicted better clinical outcomes (Hogue et al., 2008; Huey, Henggeler, Brondino, & Pickrel, 2000). Additionally, the relationship between fidelity and outcome has also been established when transporting family-based treatments to community settings (Henggeler, Melton, Brondino, Scherer, & Hanley, 1997). Prior research on treatment fidelity has included different methods for identifying treatment adherence or competence. Henggeler and colleagues have used therapist and family member self-reports to identify adherence to nine key principles of multisystemic therapy (Henggeler et al., 1997), whereas Hogue et al. (2008) used observational methods to capture extensiveness of model-consistent therapist interventions in the four core domains of multidimensional family therapy. The strength of adherence–outcome and competence– outcome relationships may be influenced, in part, by the theoretical specificity of these measures and rigorous methods that were used to collect data in prior studies.

Fidelity to empirically based family interventions requires a range of therapeutic skills, including both relational (interpersonal) and structuring (directive) behaviors. Successfully implementing evidence-based family therapy models involves tailoring interventions during moment-to-moment transactions to meet phase-specific goals of treatment. In fact, across all of the empirically supported family therapy approaches, the phase-based nature of change is highlighted, with each phase consisting of specific therapist skills and goals. A major challenge in prior research on therapist fidelity has been to measure and analyze in-session behaviors adequately enough to capture this complexity. For example, although the Hogue et al. (2008) adherence construct is composed of therapist interventions consistent with the theoretical underpinnings and techniques of multidimensional family therapy (Liddle et al., 2002), previous adherence studies with this model have operationalized adherence as a global composite measure (with possible adolescent-focused and family-focused facets; Hogue, Dauber, Samuolis, & Liddle, 2006) rather than focusing on the more specific behaviors therapists apply on a moment-to-moment basis. Similarly, the measurement of adherence in multi-systemic therapy provides a general impression of the extent to which key principles were implemented but does not provide an independent perspective on the specific interventions that occur during sessions.

Taken together, the results of adherence studies in family therapy have been useful for establishing adherence– outcome links. However, the measures have not adequately captured the complexity of family-based interventions, nor have they adequately attended to the phase-based nature of adherence to family therapy (e.g., the specific behaviors to which therapist adherence should look different in early sessions compared with later sessions). Thus, the focus of this study was to examine adherence ratings from a multisite, prospective, randomized clinical trial to identify the presence of theoretically based clinical interventions in Brief Strategic Family Therapy (BSFT). The BSFT treatment (Szapocznik et al., 2003) is an evidence-based approach that systematically targets adolescent drug use and behavior by improving family functioning. Interventions are delivered in phases, starting with joining with individual family members and the family system and building to structured and directive interventions intended to facilitate new behaviors and family interactions.

The present study pursues five aims: (a) examine the factor structure of the theoretically derived BSFT Adherence Scale (i.e., with four types of behaviors: joining with individual family members and the family system, tracking and eliciting family interactions, creating a motivational context for change, and restructuring family interactions); (b) validate the dimensions of the BSFT Adherence Scale by examining convergence with clinical supervisors’ competence ratings; (c) use the distinct domains identified by this measure to differentiate therapeutic interventions delivered early, middle, and late in treatment; (d) examine differences in the rates of these domains in cases that failed to engage (attend fewer than two sessions), failed to retain (attend between two and seven sessions), and completed treatment (attend eight or more sessions); and (e) examine the relationship between levels and timing of therapist behaviors and reductions in adolescent drug use and improvements in family functioning.

Method

Overview of Design

In the present study, adherence to the BSFT treatment model was examined in a sample of therapists selected from eight community-based outpatient substance abuse agencies. Therapists were randomly assigned to the BSFT condition or treatment as usual (TAU).¹ Adolescent and family participants were recruited from the pool of cases referred to outpatient community substance abuse treatment programs and randomized to the BSFT treatment or TAU.

BSFT

The BSFT intervention is an integrative model that combines structural and strategic family therapy theory and intervention techniques to address systemic/relational (primarily family) interactions that are associated with adolescent substance use and related behavior problems. The structural components of the BSFT treatment draw on the work of Minuchin (Minuchin, 1974; Minuchin & Fishman, 1981), and the strategic aspects of BSFT treatment were influenced by Haley (1976) and Madanes (1981). The BSFT intervention is a problem-focused, directive, and practical approach, following a prescribed format.

Interventions are organized into four domains to facilitate change and are planned and delivered in treatment phases to achieve specific goals at different times during treatment. Early sessions are characterized by joining interventions that are intended to establish a therapeutic alliance with each family member and with the family as a whole. This requires that the therapist demonstrate acceptance of and respect toward each individual family member as well as the way in which the family as a whole is organized. Early sessions also include tracking and diagnostic enactment interventions that are designed to systematically identify family strengths and weaknesses and develop a treatment plan. A core feature of tracking and diagnostic enactment interventions are interventions that encourage the family to behave as they would usually behave if the counselor were not present. That is, encouraging family members to speak with each other about the concerns that bring them to therapy, rather than directing comments to the therapist. From these observations, the therapist is able to diagnose both family strengths and problematic relations. Reframing interventions are used to reduce family conflict and create a sense of hope or possibility for positive change.

Over the course of treatment, therapists are expected to continue to maintain an effective working relationship with family members (joining), facilitate within-family interactions (tracking and diagnostic enactment), and to directly address negative affect/beliefs and family interactions. However, the focus of treatment shifts to implementing restructuring strategies to transform family relations from problematic to effective and mutually supportive, and include (a) directing, redirecting, or blocking communication; (b) shifting family alliances; (c) helping families develop conflict resolution skills; (d) developing effective behavior management skills; and (e) fostering parenting and parental leadership skills.

Community Treatment Programs and Therapist

Participants

Randomization of therapists and family participants occurred at eight outpatient community treatment providers (providers): La Frontera (Tucson, AZ), The Crossroads Center (Cincinnati, OH); The Village (Miami, FL); Gateway Community Services (Jacksonville, FL); Administración de Servicios de Salud Mental y Contra la Adicción (ASSMCA: Bayamón, PR); Daymark Recovery Services (Salisbury, NC); Tarzana Treatment Centers (Tarzana, CA); and Arapahoe House (Denver, CO). Sites volunteered from within the NIDA Clinical Trials Network.

Adherence Raters

BSFT treatment adherence raters (n = 5) consisted of four graduate students in counseling psychology (three master’s-level Mental Health Counseling students and one doctoral-level Counseling student) and one Mental Health Counselor. Three raters were women, and all but one were Hispanic and fully bilingual in English and Spanish (proficiency in Spanish was necessary due to the high frequency of therapy sessions conducted in Spanish). The age of raters ranged from 23 years to 34 years, with an average of 27.4 years.

Family Participants

Four hundred eighty adolescents and their families were randomized to the BSFT treatment or TAU. To enroll in the study, adolescent participants had to have used illicit drugs (other than alcohol and tobacco) in the 30-day period that preceded the baseline assessment or were referred from an institution (e.g., detention, residential treatment, court, etc.) for substance abuse treatment. A more complete description of the inclusion/exclusion criteria, randomization procedures, and baseline demographics are presented in Robbins et al. (2009).

Procedure

Therapist selection

At each agency, therapists consisted of four (or more) providers that provided clinical services, including regular part- and full-time clinical staff as well as contract workers. Although a selection process was designed to evaluate elementary counseling skills (i.e., openness to learning a new intervention; willingness to provide home-based services; the ability to convey understanding, acceptance, and respect to all family members; videotape evaluation for general interpersonal skills; and directness and clarity of communication), only two therapists were excluded. In both instances, the therapists were excluded due to concerns about the therapists’ willingness to be trained in a new intervention.

Randomization and training of therapist participants

Randomization stratified therapists by level of academic training and years of clinical experience. At two sites, it was necessary to stratify by language (e.g., Spanish speaking) to ensure that Spanish monolingual participants could receive treatment in each condition. Therapists assigned to deliver the BSFT intervention received approximately 96 hr of training delivered in four 3-day workshops and attended group weekly supervision sessions over the entire course of the study.

Training of adherence raters

BSFT treatment adherence raters received approximately 80 hr of training in the use of the BSFT Adherence Form. Raters used videotaped family therapy sessions recorded during the pilot phase of the study to identify and rate BSFT-prescribed therapist interventions in the four clinical domains of the BSFT model: joining, tracking and diagnostic enactments, reframing, and restructuring. Training was conducted by an expert supervisor (first author), and Adherence Raters completed their training once they achieved a desirable level of agreement, which was defined as an average intraclass correlation (ICC) over .70 for three to five consecutive ratings in their observations with the BSFT Master Trainer and with each other.

Therapist adherence

Every week, randomly selected sessions from each therapist were rated by independent raters. Sessions were randomly assigned to raters, thus it was possible for a rater to complete multiple ratings for a family over time. Sessions were not assigned to raters on the basis of site or therapists. However, one rater did not complete ratings for sessions in Spanish because he was not Spanish speaking. Raters were blind to clinical outcomes of the cases but were aware of the site and session number. A random sample of sessions was rated by multiple raters to establish interrater reliability. These reliability ratings were completed over the course of data collection, and raters were not aware that the session was being rated for reliability purposes.

Clinical supervision

BSFT intervention supervisors delivered weekly supervision via a 3-hr conference call with all BSFT intervention therapists at each site. Prior to the conference call, the BSFT clinical supervisor reviewed (a) clinical forms that were completed by each therapist, (b) one randomly selected videotape from each therapist’s current caseload, and (c) adherence ratings from one randomly selected videotape of the therapist’s current caseload rated by an independent observer. Sampling for adherence and videotape supervision was performed for every therapist. During supervision, the clinical supervisor reviewed each therapist’s active caseload and approximately 30 min of the randomly selected videotapes/DVDs for each therapist.

Measures

Therapist demographics

A therapist demographic form was developed for the present study to identify basic demographic information, including age, gender, ethnicity, years of education, highest degree attained, and total years of clinical experience.

BSFT Therapist Adherence Form

The BSFT Therapist Adherence Form (see the Appendix) was completed by graduate student raters from the University of Miami. Ratings were generated each week per therapist over the course of the implementation phase of the study. The form included 20 items describing prescribed therapist behaviors over four sections corresponding to the basic clinical domain areas in the BSFT model: (a) joining, (b) tracking and diagnostic enactments, (c) reframing, and (d) restructuring. Ratings were recorded using a 5-point scale, ranging from 1 ( poor adherence) to 5 (excellent adherence), with 3 or more representing the minimally acceptable level of adherence. The average (interrater) ICC across therapists and intervention domains was .83, ranging from .81 for restructuring to .85 for tracking and diagnostic enactments.

Appendix.

Items From the Therapist Adherence Scale

BSFT™ Study Therapist Adherence Form (ADH)
Joining
ADH (01). Conveys understanding, acceptance, and respect to every family member and does not criticize, accuse, or label.
ADH (02). Greets every family member and regularly connects with and seeks point of view of each family member.
ADH (04). Reflects back to the family their comments without challenging them.
ADH (05). Mimics the family style.
ADH (06). Establishes his/her therapeutic leadership.a
ADH (07). Joins with children/adolescents.a
ADH (08). Joins with parents/adults.a
Tracking and Diagnostic Enactments
ADH (09). Asks clarifying, process-oriented questions.
ADH (11). Stimulates dialogue/directs enactments between family members when they do not occur spontaneously.a
ADH (12). Remains decentralized.a
Reframing
ADH (14). Offers and sells the family new and more positive views of the presenting problems, of each other, or of themselves.a
ADH (16). Reframing is appropriate and effective for children/adolescents.a
ADH (17). Reframing is appropriate and effective for parents/adults.a
Restructuring Skills
ADH (18). Focuses on process, not content.a
ADH (19). Focuses on present interactions.a
ADH (20). Actively directs and elicits new behaviors.a
ADH (23). Works with a subsystem within the overall context of the whole family for purposes of restructuring.a
ADH (24). Strengthens, realigns, or creates boundaries between family members.a
ADH (26). Offers positive feedback at the end of a successful transaction.
ADH (27). Assigns homework tasks.

^aQuestions on the Therapist Adherence Form (ADH) that correspond to identical questions on the Clinical Supervision Checklist.

Clinical supervision checklist

The Clinical Supervision Checklist was completed every week by a BSFT clinical supervisor from the University of Miami to identify therapist competence to prescribed BSFT interventions. Ratings were completed for 16 therapist behaviors that represented the four clinical domains of the BSFT intervention. Ratings were completed using the same 5-point scale as the BSFT Therapy Adherence Form, but captured therapist competence in implementing BSFT interventions, defined as the extent to which behaviors were present as well as the quality with which they were implemented by the therapist. Ratings ranged from 1 ( poor competence) to 5 (excellent competence), with 3 representing the minimum standard for competence to the model. Ratings were based on clinical discussions during group supervision sessions (telephone), review of progress notes, and observation of at least one videotape per therapist (which was typically the same session that was randomly selected for adherence ratings). Ratings on the Clinical Supervision Checklist are informed by multiple sources of clinical information and thus represent a more comprehensive assessment of therapists’ level of competence to the BSFT model than the ratings of therapist adherence completed by the graduate student raters.

Therapy dose: Engagement and retention

To reduce privacy concerns in the collection of clinical information, dose was tracked through monthly interviews with therapists rather than by directly accessing agency charts or electronic records. Using the participant’s clinical charts, therapists were asked to report on the recommended dose and the number of sessions that were delivered during the past month (or since the last interview). Total dose was constructed as the sum of all therapy sessions conducted by any therapist (not just the study therapist) at the agency. It should be noted that over 98% of the sessions in BSFT were family therapy sessions led by the assigned BSFT therapist.

Fail to engage and fail to retain in therapy were constructed as binary variables, based on attendance in therapy sessions of any type. The criteria for each variable were consistent across treatment conditions. On the basis of prior research (Coatsworth, Santisteban, McBride, & Szapocznik, 2001; Prado et al., 2002; Santisteban et al., 1996), a participant was classified as a fail to engage if there were zero or one session with the adolescent and or family members. A participant was classified as fail to retain if there were seven or fewer sessions with the adolescent and/or family members (Robbins et al., 2008; Santisteban et al., 1996).

Adolescent drug use

The time line follow-back (TLFB; Bry, Conboy & Bisgay, 1986; Bry & Krinsley, 1992) was used to assess adolescent drug use. This interview uses a calendar and other memory prompts to stimulate recall to obtain retrospective reports of daily drug use. The TLFB yields consistently high test–retest correlations (Carey, 1997; Mason, Cauce, Gonzales, Hiraga, & Grove, 1994) and has been shown to correlate with other self-reports and collateral reports (Sobell & Sobell, 1992). At baseline, the TLFB was used to identify drug use in the 28-day period that preceded the baseline assessment. At Time 1 [T1], the TLFB was used to assess daily use for all days between randomization and the T1 assessment. At Time 2 and through Time 12, the TLFB was used to collect data on daily use from the prior assessment to the current assessment. Thus, the TLFB was used to collect 364 continuous days of data on daily drug use after randomization. The drug use outcome measure was coded into 13 equal 28-day measures.

Urine drug screens were conducted at baseline and all follow-up assessments using the SureStep Drug Screen Card 10A (Health Research Systems, Inc., Dunbar, WV) and urine cups, which includes temperature-controlled monitoring and detection of adulterants. Urine drug screens were administered immediately prior to the administration of the TLFB to improve the chances of accurate reporting of days of drug use.

Family functioning

Family functioning measures were assessed at baseline, 4, 8, and 12 months postrandomization. Two measures were used to assess family functioning. The Parenting Practices Questionnaire from The Chicago Youth Development Study was used to identify parenting practices (Gorman-Smith, Tolan, Zelli, & Huesmann, 1996). Adolescent and parent responses to 47 items were used to identify positive and negative parenting behaviors. Factor analyses have identified four factors: (a) Positive Parenting, (b) Discipline Effectiveness, (c) Avoidance of Discipline, and (d) Monitoring. Estimates of positive parenting and extent of monitoring were gathered from both parent and child. Internal consistency reliabilities of each of the subscales in this study ranged from .68 to .81. Also, the Cohesion and Conflict scales from the Family Environmental Scale (FES; Moos & Moos, 1986) were used to measure family functioning. The FES is a widely used measure that was developed to assess social and environmental characteristics of families. Internal consistency reliability estimates for the subscales in this study ranged from 0.61 to 0.78. Conflict and Cohesion subscales were administered to both parents and adolescents.

The dependent measure in analyses of family functioning is a composite scale created from the Parenting Practices Questionnaire and FES. Separate composites were created for youth (α = .90) and parents (α = .91; described in Feaster et al., 2010).

Data analysis plan

A series of measurement models and statistical tests of measurement invariance were used to assess the validity and reliability of independent ratings of adherence. The analysis plan consisted of five distinct components: reliability of the adherence ratings, reliability of the clinical supervisor competence ratings, the convergent validity across these two measures, sensitivity to detect differences in levels of intervention domains over the course of therapy, and the relationship of adherence to clinical outcomes.

Reliability of the adherence measures

The sample of adherence ratings was randomly split into two subsamples. The first was used as a model-building sample and the second as a cross-validation sample. The adherence ratings were constructed to fit a theoretical template, and thus the initial model was the one supported by BSFT model clinical theory. Modification indices were examined to gauge the impact of particular changes to fit of the data to the hypothesized model. The modification indices were then reviewed for their overlap with theoretically justifiable modifications to the hypothesized structures, and a series of alternative models were examined. The impact of the changes was assessed by examining model fit statistics. A structural equation model with excellent fit to the data will have a comparative fit index (CFI) ≥ .95 and root-mean-square error of approximation (RMSEA) ≤ .06, whereas, an adequate fit has a CFI ≥ .90 and an RMSEA ≤ .08 (McDonald & Ho, 2002). Once a final model was determined, this model was replicated on the confirmatory subsample. Following confirmation on the subsample, this final model was reestimated on the full sample to obtain the most stable estimates of factor loadings, means, and variances. Internal reliability for each estimated factor is calculated using a formula proposed by Reuterberg and Gustafsson (1992) to calculate composite reliability in a factor model. Convergent validity was established by examining the standardized loadings on each item within a factor and the factor correlations (Kline, 2005).

Note that the data in this investigation are nested. Individual cases are nested within 20 therapists, and therapists are nested within eight sites. Statistically, these are relatively small numbers of upper level units. Multilevel measurement modeling decomposes the overall variance– covariance into within- and between-therapist matrices using the therapist nesting as an example; the number of Level 2 items (in this case, therapists) is the maximum number of parameters that can be fitted to the between-level covariance. Therefore, we were not able to fit our measurement model to the between-therapist matrix. However, the results were compared (described below), which ignored nesting within therapist, with the results of an analysis within therapist and confirmed that there were no differences. Because later hypotheses are based on mean comparisons, the within-therapist results are not reported.

Reliability of the clinical supervisor’s competence ratings

Clinical supervisor ratings were not split into two samples due to the smaller sample size of ratings. The hypothesized structure, which included the BSFT clinical supervisor’s ratings of the four clinical domains of the BSFT intervention, was fit in a confirmatory model. Modification indices were examined to find the most parsimonious changes to the model to achieve adequate fit, which was subsequently reestimated. Internal reliability and convergent validity were calculated using the same methods as the adherence ratings.

Convergent validity of adherence and clinical supervisor ratings

Most weekly clinical supervision ratings were based in part on the review of a therapist videotape, which were frequently also the tapes that were randomly selected for adherence ratings. The clinical supervisor’s ratings for a particular videotape were merged with the adherence rating for the same tape. A combined model in which the final models for both the adherence ratings and the clinical supervisor’s ratings was jointly estimated on the merged data set. Convergent validity was assessed by examining the stability of the standardized loadings.

Level of interventions over time

It was hypothesized that joining would be higher early rather than later in therapy and that restructuring would be higher later than earlier in therapy. No differences in the rate of tracking and diagnostic enactments and reframing was expected over the course of treatment. These hypotheses were examined by comparing the mean levels of each factor across three phases of therapy (early, middle, and late). The first and second session are considered the early phase of therapy. The third through seventh sessions are considered the middle phase. The eighth and any later sessions are considered the late phase of therapy. The designation of early, middle, and late sessions, and our definitions of engagement and retention, represent arbitrarily selected cutoffs. However, all are informed by prior research (Coatsworth et al., 2001; Robbins et al., 2008; Santisteban et al., 1996). Prior to testing mean differences across these three phases, a measurement invariance analysis was performed to ensure that the instrument had comparable factor structure over time (Burlew, Feaster, Brecht, & Hubbard, 2009; Vandenberg & Lance, 2000). The two hypothesized mean differences are tested jointly in one test, which constrains the means of each of the rating factors to be equal across time. The post hoc test of the difference in means across time is determined from testing the significance of the Wald statistic.

Therapist adherence and clinical outcomes

Levels of therapist adherence were examined on each adherence factor to determine whether they predicted the probability of engagement (vs. fail to engage) and completion of treatment (vs. fail to retain) and whether the levels of therapist adherence and change in levels of adherence over time predicted levels of family functioning and adolescent drug use.

Fail to engage and fail to retain

Separate logistic regressions were conducted to examine whether therapist adherence was associated with engagement and completion of treatment. The first logistic regression used ratings from Session 1 to examine the hypothesis that there would be higher adherence ratings, across intervention domains, in families who engaged (attended two or more sessions) versus those who failed to engage. The second logistic regression used ratings from Sessions 1–7 to examine the hypothesis that there would be higher adherence ratings, across intervention domains, in families who completed treatment versus those who failed to retain (engaged but attended fewer than eight sessions).

Logistic regression models were estimated in Mplus 5.21 (Muthén & Muthén, 1998–2007) using the complex option. The predictors in these analyses were composite scores of the therapist intervention domains based on the measurement models described above. The family was the unit of analysis, so for repeated measures within the family, the composites were averaged across the appropriate period to provide a single score per family. The Mplus 5.21 complex option corrected the standard errors for the nesting of families within therapist. In addition, each outcome was tested for differences across site, dummy variables were included for sites that differed from the overall sample.

Therapist adherence and clinical outcomes

It was hypothesized that higher mean levels of adherence, across intervention domains, would be associated with lower adolescent drug use and higher family functioning at the end of the study. With respect to changes in adherence over the course of therapy, it was hypothesized that higher levels of joining early in treatment would be associated with lower drug use and higher family functioning. In addition, it was expected that higher levels of joining over time would be associated with positive outcomes (drug use and family functioning). Taken together, these hypotheses imply that it is important not only to begin therapy with high levels of joining but also to maintain adequate joining over time, even as the focus of treatment shifts to restructuring family interactions.

With respect to restructuring, it was hypothesized that higher levels of restructuring later in therapy would be associated with lower drug use and higher family functioning. In addition, it was predicted that increases in restructuring over time will be associated with lower drug use and higher family functioning. Thus, both the level of restructuring in the later stages of therapy and the increase in restructuring from early to late in treatment are important in predicting outcomes. These hypotheses are tested using latent growth curves. For joining, the growth curve is centered at the early stage of therapy. For restructuring, the growth curve is centered at the late stage of therapy. Drug use is modeled as a negative binomial to account for the overdispersion in this count variable. Baseline levels of family functioning and adolescent drug use were controlled in the respective models. All models were estimated using the Complex option in Mplus 5.21 to estimate standard errors that account for nesting within therapist.

Results

Adherence Ratings: Measurement Models

There were a total of 905 adherence ratings, with an average of 3.7 per family in the BSFT condition. The first exploratory sample represented 468 randomly selected session ratings across all phases of therapy. The initial measurement model of the theoretically derived four factors fit the data relatively well, but with a CFI of .92 and an RMSEA of .087, the model could be improved. After examining the modification indices, we correlated four pairs of items that showed modification indices greater than 30. We also explored dropping several items with relatively low loadings or moving items that appeared to have potential for loading on different dimensions, but these changes did not affect model fit appreciably. The final model choice was the a priori hypothesized structure with the addition of the four pairs of correlated errors. This final model had adequate fit with a CFI = .94 and an RMSEA = .081 on the replication sample (the 437 remaining ratings), and a CFI = .94 and an RMSEA = .076 on the full sample. Factor loadings were good, ranging from .32 to .77 for joining, from .48 to .86 for tracking and diagnostic enactments, from .99 to 1.00 for reframing, and from .32 to .82 for restructuring. The composite reliability based on the factor loadings was .85 for joining, .78 for tracking and diagnostic enactments, .99 for reframing, and .83 restructuring. The correlations of the four factors ranged from .43 to .80, indicating that these factors are distinct but highly correlated (see Table 1).

Table 1.

Factor Correlations of Adherence Ratings

	Within-adherence ratings
	1	Rating Team		4
		2	3
Rating team
1. Join	—	.46	.43	.50
2. Track		—	.47	.80
3. Reframe			—	.59
4. Restructure				—

	Within-supervisor competence ratings
	1	Supervisor rating		4
		2	3
Supervisor rating
1. Join	—	.55	.60	.67
2. Track		—	.53	.88
3. Reframe			—	.66
4. Restructure				—

	Between-adherence raters and clinical supervisor ratings
	Supervisor rating
	1	2	3	4
Rating team
1. Join	.64	.34	.36	.39
2. Track	.41	.68	.34	.61
3. Reframe	.43	.32	.59	.37
4. Restructure	.46	.5	.33	.33

Note. Boldface values indicate the shared variance between adherence raters and clinical supervisors on the same scales.

Clinical Supervisor Ratings

There were 509 clinical supervisor ratings. Like the adherence ratings, the measurement model of the hypothesized structure of the clinical supervisor ratings showed reasonable fit, although the RMSEA was higher than desired (CFI = .92, RMSEA = .089). Again, we correlated the same four pairs of items that showed modification indices greater than 30. When these four pairs of residuals were correlated, the fit was substantially improved (CFI = .97, RMSEA = .055). Factor loadings were good, ranging from .59 to .78 for joining, from .80 to .89 for tracking and diagnostic enactments, from .86 to .91 for reframing, and from .64 to .83 for restructuring. The composite reliability based on the factor loadings was .71 for joining, .90 for tracking and diagnostic enactments, .92 for reframing, and .90 restructuring. The correlations of the four factors ranged from .53 to .88, indicating that these factors are distinct but highly correlated (see Table 1).

Relationship of Adherence Ratings to Clinical Supervisor Ratings

There were 415 matched adherence and clinical supervisor’s ratings (i.e., the supervisor ratings were based in part on a videotape that had also been rated for adherence). The model estimated on the matched data fit well (CFI = .93, RMSEA = .062), and the loadings closely replicated the loading of the separate analyses. The bottom part of Table 1 shows that correlations of the adherence rating and the clinical supervisor’s rating showed moderate to high levels of agreement. The correlations of joining (ρ = .64), tracking and diagnostic enactments (ρ = .68), and reframing (ρ = .59) indicate that the shared variance of the constructs ranged from 35% to 46% shared variance. This shows agreement, but distinct information. The final construct, restructuring, had a smaller correlation (ρ = .33) representing only 11% shared variance.

Level of Interventions Over Time

Adherence ratings were split into three samples corresponding to the phases of therapy that the particular rating represented. The 901 adherence ratings included 232 from the early phase (Session 1 or 2), 286 from the middle phase (Sessions 3–7), and 383 from the late phase (Session 8 plus). A multigroup measurement model showed metric invariance (equal loadings across phases of therapy) and scalar equivalence (equal item intercepts across phases of therapy). There was evidence that 12 of 40 residual item variances did differ across phases, and these were freed in the final model reported. Combined, these results imply that we had adequate but not strict equivalence (Meredith, 1993), implying that phase comparisons can be made because they are measuring roughly the same underlying construct.

Prior to testing differences in factor means across phase, we tested and rejected equality of factor variances across therapy phase, χ²(8, N = 900) = 19.47, p < .02. Examination of the variances showed that the difference was caused by two variances. First, the variance of joining was lower (.15) in the middle phase than it was at both early and late phases (.21). Second, the variance of tracking and diagnostic enactments was larger in the early phase (.16) than it was in the middle and late phases (.11). The test of differences in means, described in the next paragraph, was done allowing these two variances to differ. All other variances were constrained equal across phases.

There were statistically significant differences in the omnibus test of the mean levels of the four factors across the three phases of therapy, χ²(8, N = 900) = 53.61, p < .0001. The specific means by phase are shown in Table 2. Note that the means of the early phase are set equal to zero for model identification purposes. Therefore, the test of whether the means at the later phases of therapy are different from zero is equivalent to the test of whether they are different from the early phase. Both joining and restructuring show significant mean differences across phases of therapy. Table 2 shows that joining interventions are significantly less frequent in the late phase of therapy than they are in the early phase, t(368) = −3.22, p < .002. Table 2 also shows that restructuring interventions are much more frequent in both the middle and late phase of therapy relative to the early phase: middle, t(368) = 3.87, p < .0001; late, t(368) = 3.99, p < .0001.

Table 2.

Adherence Factor Means by Phase of Therapy

Factor	Factor means
	Early	Middle	Late
Join	0	−0.174	−0.290
SE	0	0.106	0.090
Z	0	−1.65	−3.22
p	0	0.10	0.001
Track	0	0.122	0.079
SE	0	0.106	0.101
Z	0	1.15	0.79
p	0	0.25	0.430
Reframe	0	0.028	0.082
SE	0	0.089	0.084
Z	0	0.31	0.98
p	0	0.76	0.326
Restructure	0	0.369	0.359
SE	0	0.096	0.090
Z	0	3.87	3.99
p	0	0.001	0.001

Adherence and Clinical Outcomes

Engagement and retention in treatment

In the first analysis, we examined how mean levels of factors predicted fail to engage (attend fewer than two sessions) versus engaged in treatment. Adherence ratings from Session 1 were included in this analysis. This was necessary because, by definition, no additional sessions were available for fail-to-engage cases. Adherence scores were nested at the family level. It should be noted that only 11% (28 of 245) of the cases assigned to the BSFT intervention in the trial failed to engage in treatment. Of these cases, only 10 of these 28 families attended just the first session. As such, the sample of fail-to-engage cases used in this analysis is small. To assess the relative strength of these cases, we calculated the implied odds ratio for a one standard deviation increase in each of the therapist behaviors and report them in order of effect size. Results indicated that higher levels of both restructuring (OR [1 SD] = 1.46, 95% CI = 1.03, 2.07) and reframing (OR [1 SD] = 1.76, 95% CI = 1.01, 3.07) increased the likelihood of engagement. There was a weak effect for joining to be associated with higher rates of engagement (OR [1 SD] =1.35, 95% CI = .97, 1.87). No differences in tracking diagnostic enactments were observed.

In the second analysis, we examined differences in adherence between cases that were retained in treatment (attended eight or more sessions) versus those that failed to retain (attended two to seven sessions). Adherence ratings from Sessions 1–7 were included in this analysis. This was necessary because, by definition, no later sessions were available for failed-to-retain cases. Adherence scores were nested at the family level. A family could have from one to seven scores depending on the number of adherence ratings available for that family. An average score was created to represent therapist adherence for each family. Results indicated that higher levels of all four therapist behaviors predicted higher rates of retention. A one standard deviation increase in reframing implied a case was 19% more likely to retain in treatment (OR [1 SD] = 1.19, 95% CI = 1.01, 1.40). A one standard deviation increase in joining implied a case was 22% more likely to retain (OR [1 SD] = 1.22, 95% CI = 1.05, 1.42). A one standard deviation increase in restructuring implied a case was 59% more likely to retain (OR [1 SD] =1.59, 95% CI = 1.27, 2.00), whereas a one standard deviation increase in tracking implied a 62% increase in the likelihood of retention (OR [1 SD] =1.62, 95% CI = 1.32, 2.00).

Family functioning

Mean levels of joining were related to family functioning at the final assessment (b = .053, SE = .019, p < .005). Mean levels of the other three therapist behaviors were not significantly related to the final level of family functioning. The standardized coefficient, γ, was .203. There were no significant relationships between intercepts or slopes of the four adherence scales and family functioning at the last assessment.

Adolescent drug use

Mean levels of joining also predicted level of drug use at the final assessment (b = .121, SE = .051, p < .018). This coefficient implies an incidence rate ratio of drug use associated with a one standard deviation increase in joining of .74 (95% CI = .58, .95). There were no significant relationships between mean levels of the other three therapist behaviors and final levels of adolescent drug use. The growth curve model of joining showed that the linear slope in Joining was negatively related to drug use (b = −3.84, SE = .888, p < .0001), implying that less steeply declining trajectories of joining (the mean linear trajectory was negative) were associated with significantly lower levels of adolescent drug use (IRR [1 SD] = .41, 95% CI = .28, .62). The intercept (baseline) level of joining was not statistically significant. Similarly, the growth curve model of restructuring showed that the linear slope in restructuring was negatively related to drug use (b = −.487, SE = .185, p < .008), implying that more steeply increasing trajectories of restructuring (the mean linear trajectory was positive) were associated with significantly lower levels of drug use (IRR [1 SD] = .54, 95% CI = .34, .85).

Discussion

Adherence to evidence-based family interventions is a complex process that requires a mix of clinical skills that are delivered in a planned and strategic manner based on the phase of treatment. As such, the measurement of adherence must be multifaceted and dynamic. Unfortunately, to date, prior studies have not adequately captured this complexity; instead global impressions of intervention domains (e.g., overall adherence) have been examined. The results of the present study indicate that not only is it possible to capture and distinguish between theoretically relevant BSFT treatment techniques, it is also possible to identify variations in the extent to which these techniques are used over the course of therapy and to use these ratings to discriminate therapist behaviors in cases that dropout or complete treatment. These results have broad implications for clinical research and therapist training. For example, with respect to training, the specificity in identifying theoretically relevant behavioral domains provides supervisors with valuable information about key areas to address in BSFT intervention training.

Like many behavioral interventions, components of the BSFT model include a mix of relational and structuring skills (e.g., directive or coaching interventions) that therapists are expected to use to manage the moment-to-moment interactions that occur in therapy sessions (Szapocznik et al., 2003). Moreover, the BSFT intervention also proceeds in a planned manner, starting with engagement and assessment and leading to behavior change (restructuring, or changes in family interaction). The present results indicate that within a trained sample of therapists from community agencies, therapist interventions follow a pattern that is consistent with the theory behind the BSFT intervention. For example, during the critical engagement process, therapist joining interventions are higher in this early stage than over the remainder of treatment, although joining continues throughout treatment. In contrast, therapist restructuring interventions are lowest early in treatment and increase dramatically later after family members have been successfully engaged in treatment and therapeutic focus moves to behavior change. Also, tracking and diagnostic enactments and reframing did not vary significantly over time, which was expected because both sets of interventions are considered to be important during all phases of treatment; for example, reframing is critical to creating a motivational context for engagement in treatment as well as in creating a context for changing family members’ interactions. As such, the present findings provide support for the argument that therapists from community agencies can be trained to deliver the techniques of empirically based family interventions in an adherent manner.

The results also suggest that this measure may be useful for understanding differences in relevant clinical outcomes. For instance, significant findings were observed in therapist adherence among cases that dropped out versus those that successfully completed treatment. With respect to therapist interventions in the early phases of treatment, the present findings suggest that the overall level of Joining may be a necessary component of the BSFT intervention, but not sufficient to yield better treatment outcomes. For example, only a weak effect was observed in the level of Joining interventions between cases that failed to engage and those that engaged in treatment. Therapist reframing and restructuring in the first session appears to be the most significant predictor of engagement in treatment. This finding is consistent with prior research that has shown the effectiveness of reframing in reducing family members’ negativity early in family therapy (Robbins, Alexander, Newell, & Turner, 1996; Robbins, Alexander, & Turner, 2000). This finding should be interpreted with caution, however, due to the very small sample of cases that failed to engage in treatment.

Moreover, during the middle phase of treatment, rates at which therapists join, track and diagnose, and reframe family interventions were significantly higher among cases that completed treatment than those that dropped out. Thus, in both groups, therapists were able to successfully engage; however, over time (Sessions 1–7), therapists who used more joining and tracking interventions, while maintaining a higher level of reframing, were more successful in keeping families in treatment.

With respect to predicting clinical outcomes, only the overall level of Joining interventions across all phases of treatment was associated with improvements in family functioning. However, the trajectories of change in Joining and Restructuring were related to improvements in adolescent drug use. Specifically, positive drug use outcomes were predicted by less sharp declines in Joining and a sharper increase in restructuring interventions over the course of treatment. It was not the overall level of joining (over time or early in treatment) that was critical, but rather the therapist’s ability to maintain their level of Joining. And, with respect to restructuring, it was the increase in directive, restructuring interventions over time that was critical. However, it is also possible that the maintenance of higher levels of joining interventions influenced other unobserved factors, such as family member alliances with the therapist, that created a context in which family members were able to benefit from the more directive, behavioral interventions that are characteristic of the restructuring process. It should be noted that although tracking and reframing were not directly related to clinical outcomes, these interventions were associated with engagement and retention in treatment. Hence, it is possible that all four sets of techniques are necessary to engage and retain and thereby achieve outcomes.

The present findings have important implications for future research studies on identifying mechanisms of change in the BSFT intervention with drug-using adolescents. In the BSFT model, outcome is considered to be dynamic and multifaceted. Outcome is a process that unfolds throughout treatment, beginning with micro changes tied to specific interventions, evolving and building throughout therapy, and eventually reflected in ultimate outcomes observed following the completion of treatment. Each stage of treatment includes specific intervention strategies and clinical outcomes. This view of outcome is consistent with the focus in the broader field of psychotherapy research, which is explicitly concerned with understanding the links between therapist and client activities at different stages of the treatment process (e.g., identifying therapist behaviors associated with the immediate [little os] and long-term [big Os] outcomes of therapy; Gurman, Kniskern, & Pinsof, 1986). Generally, the focus of prior family therapy research has been limited because adherence measures (or clinical process) have not had the specificity and complexity necessary to identify phase-specific process–outcome links. The measurement and validation of distinct domains of therapy (e.g., joining, tracking and diagnostic enactmen, reframing, restructuring) made it possible for us to link therapist interventions to little os (e.g., retention) and big Os (e.g., family functioning and drug use).

The results of this study build on previous research that has demonstrated that higher treatment adherence is associated with better treatment outcomes (Henggeler et al., 1997; Hogue et al., 2008) by examining how BSFT intervention techniques are associated with intermediate (engagement and retention) and long-term treatment outcomes (changes in family functioning and adolescent drug use). Such work sets the foundation for studying the mechanisms by which the BSFT intervention achieves its effects, and could be important in widening the dissemination of the BSFT model. With respect to the latter, for example, understanding the techniques most responsible for treatment success will help to sharpen the focus and efficiency of therapist training and supervision (Kazdin, 2008).

It should be noted that the findings observed in this study do not imply a simple, linear causal effect of therapist adherence on client outcomes (engagement and retention, family functioning, or drug use). Treatment is a dynamic and reciprocal process in which therapists are influenced by the behaviors of individual family members and the within-family interactions that occur in treatment sessions. As such, therapists may behave quite differently in the session when they perceive that a family is at high risk for dropout than when they are more certain that the family is engaged and committed to the process. And, in fact, the most difficult families might lead to lower adherence. A controversial area highlighted by recent studies (Barber et al., 2006; Hogue et al., 2008; see also Amrhein, Miller, Yahne, Palmer, & Fulcher, 2003) is whether, under some circumstances, greater adherence may be related to worse therapy outcome. These researchers have raised the possibility that “overadherence” may produce deleterious outcomes; however, it should also be stated that the same phenomena could be produced by client rather than therapist factors (i.e., more difficult clients may show worse outcomes in the face of high therapist adherence). Examining associations between phase-specific therapist behavior and client change in therapy may help disentangle these relationships and provide a more nuanced view of optimal levels of adherence.

Limitations

The present results should be interpreted in light of important limitations. First, these results can only be generalized to therapists from community agencies that participate in weekly group supervision sessions. It is possible that the domains identified may be less stable, either within or over time, when therapists are not exposed to a consistent supervision structure. Second, the measures included in this study were designed to tap various aspects of the BSFT intervention, and are thus specific to the BSFT model. Although the domains identified share conceptual and behavioral features that are consistent with other family therapy models, the measures and results may not be applicable to other models. Third, the behaviors assessed in this study reflect adherence to treatment strategies and are not considered to be sensitive to the overall quality with which these techniques are implemented. As such, the measure may be limited in identifying more complex and clinically sophisticated interventions that reflect overall competence. Future research that includes ratings that capture therapist fidelity (i.e., adherence plus competence) may be essential for enhancing researchers’ understanding of the change process in the BSFT intervention. However, with one exception, Restructuring, there was concurrent validity with supervisors’ ratings that did assess quality of implementation. Obtaining ratings from multiple perspectives was a significant strength of the design of the present study, but only the objective ratings that were completed by independent raters were examined in these analyses.

Future Directions and Conclusion

Although this measure was developed specifically for the BSFT intervention, it is possible that the domains identified may be present in other empirically based family interventions. As such, future studies may consider validating the use of this or similar measures with other clinical models (and populations). These studies would be useful to further understanding of potential similarities and differences in processes and outcomes across models of family therapy.

Future research is needed to identify the relationship between clinical supervisor competence ratings and clinical outcomes. In addition, research is needed to examine the extent to which the supervisor’s ratings generated in this study are consistent with ratings of competence that were generated by an independent research team (Shoham & Rohrbaugh, 2009).

Prior process research in family therapy has been limited in its focus, with the majority of studies only examining early sessions or a limited number of relevant variables. For example, there are numerous studies in which alliance and conflict in early sessions are examined, or changes in alliance (mainly during early sessions). In large part, this restricted focus has been due to the lack of adequate measures with the ability to discriminate between theoretically relevant and phase-based intervention domains or strategies. The results of this study provide a necessary first step in this process by validating a measure that can be used to capture the complexity of family therapy, and can be used to understand mechanisms of change in family therapy with drug-using youth.