Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Apr 26.
Published in final edited form as: Psychotherapy (Chic). 2014 May 26;52(1):93–102. doi: 10.1037/a0035975

Predicting Early Positive Change in Multisystemic Therapy with Youth Exhibiting Antisocial Behaviors

Kristine Tiernan 1, Sharon L Foster 2, Phillippe B Cunningham 3, Patricia Brennan 4, Elizabeth Whitmore 5
PMCID: PMC4845646  NIHMSID: NIHMS773561  PMID: 24866967

Abstract

This study examined individual and family characteristics that predicted early positive change in the context of Multisystemic Therapy (MST). Families (n=185; 65% male; average youth age 15 years) receiving MST in community settings completed assessments at the outset of treatment and 6-12 weeks into treatment. Early positive changes in youth antisocial behavior were assessed using the caregiver report on the CBCL Externalizing Behaviors subscale and youth report on the Self-Report Delinquency Scale. Overall, families showed significant positive changes by 6-12 weeks into treatment; these early changes were maintained into mid-treatment 6-12 weeks later. Families who exhibited clinically significant gains early in treatment were more likely to terminate treatment successfully compared to those who did not show these gains. Low youth internalizing behaviors and absence of youth drug use predicted early positive changes in MST. High levels of parental monitoring and low levels of affiliation with deviant peers (mechanisms known to be associated with MST success) were also associated with early positive change.

Keywords: Multisystemic Therapy (MST), Evidence Based Treatment (EBT), early response, clinical significance, delinquency, substance use

Serious antisocial behavior among adolescents is a leading cause of mental health referrals (Office of Juvenile Justice and Delinquency Prevention, 2010), primarily because of the dangerousness of such behaviors towards the community. Antisocial behaviors include actions that typically violate the rights of others, such as physical assaults and bullying, vandalism, fire-setting, noncompliance with social mores, lying, and delinquent activities (Frick, 1998). Adolescents who exhibit antisocial behaviors are at increased risk for physical health problems, substance abuse, low educational achievement and interpersonal difficulties (Schaeffer & Borduin, 2005). Thus, it is not surprising that a variety of interventions have been developed to reduce such behavior problems.

One of the most researched and disseminated treatments for antisocial behavior is Multisystemic Therapy (MST; Henggeler, Schoenwald, Borduin, Rowland & Cunningham, 2009). MST is an intensive, evidence- and family-based treatment approach based on Bronfenbrenner's (1979) theory of social ecology. MST posits that adolescent antisocial behavior problems are influenced by multiple systems and therefore treatment addresses each system and its impact on the youth's antisocial behaviors. MST is delivered in natural settings (i.e., the family's home and community) and varies in terms of session frequency and duration based upon youth and family needs. Therapists set goals with families and follow nine standardized treatment principles (Henggeler et al., 2009):

  1. Treatment is guided by the therapist's hypotheses regarding the fit between the individual's problems and the contexts in which they appear.

  2. Treatment focuses on strengths of the family and systems in which the family lives.

  3. Interventions focus on promoting responsibility and accountability.

  4. Interventions are present-focused, action-oriented and well defined.

  5. Interventions target sequences of behavior.

  6. Interventions are developmentally appropriate.

  7. Interventions require continuous effort by family members

  8. Intervention effectiveness is evaluated continuously by MST team members.

  9. Interventions include the caregiver and are delivered in a manner that the caregiver can continue post treatment.

Guided by these principles, therapists use a variety of evidence-based therapies (e.g., parent-training techniques) and individualized procedures (e.g., cognitive-behavioral therapy) to address specific family and youth problems that contribute to the youth's behavior (Henggeler et al., 2009). An additional facet of MST is that therapists carry caseloads of 3-5 families and receive ongoing supervision according to clearly-defined quality assurance protocols focused on providing caregivers with skills and resources to address youth behaviors.

MST clinical trials targeting serious juvenile offenders have consistently demonstrated, relative to comparison conditions, reduced rates of short- and long-term criminal offending (e.g., Timmons-Mitchell, Bender, Kishna & Mitchell, 2006); rates of out-of-home placements (e.g., Shaeffer & Borduin, 2005); substance use (Henggeler et al., 2009; Timmons-Mitchell et al., 2006); and behavioral and mental health problems (Borduin et al., 1995; Henggeler et al., 1999). Although these outcomes are generally consistent across studies, the overall duration and intensity of MST that is necessary to produce these outcomes can vary considerably across families. This variation in duration has been minimally researched, as has the pattern of change that characterizes successful and unsuccessful families.

On average, MST lasts 4-6 months, but evidence suggests that some families may terminate treatment and exhibit improvement with as few as three meetings (Halliday-Boykins, Schoenwald & Letourneau, 2005). Furthermore, antisocial youth and their families who drop out of MST studies exhibit more reduced incarceration rates, arrests and antisocial behaviors compared to youth who do not receive MST or who drop out of other types of treatments (Halliday-Boykins et al., 2005). These findings support a need to further examine what, if any, early positive changes are made during MST in order to maximize clinical gains made early in treatment.

To this end, the present study examined the extent to which youth showed early positive changes in antisocial behavior problems while receiving MST; whether these changes were maintained over time; and whether early positive change was associated with successful termination. We also examined factors that we expected to predict individual differences in youth improvement during the early stages of MST. Because the typical duration of MST is 4- 6 months, early positive change was conceptualized as a decrease in youth antisocial behavior within the first 6 - 12 weeks of treatment.

Overall, assessing early positive change relative to treatment success in MST and identifying influential variables would add to treatment process research as well as having important practical applications. First, little is known about actual treatment trajectories or expected rates of improvement among youth presenting serious clinical problems. Additionally, the ability to target characteristics associated with early positive change from the onset of treatment could substantially reduce the time and costs associated with treatment and may further enhance parents’ likelihood to participate and commit to treatment. Finally, identifying factors contributing to early treatment gains or barriers may ultimately affect therapy process and therefore increase overall opportunities for success. MST implementation practices emphasize the importance of family engagement and thus, therapy explicitly addresses the reduction of environmental barriers to treatment (e.g., treatment is offered in natural environment, therapists are available all hours of the day, Henggeler et al., 2009). Since MST is multifaceted, additional knowledge regarding youth and family characteristics that may affect early treatment progress (caregiver symptomatology, youth substance abuse, family relationships) would greatly help therapists to know which factors should be monitored and possibly targeted from the onset of treatment to maximize gains early on.

Although MST research has not focused on early response, research related to brief treatment interventions has identified that clinically significant change does occur within a variety of populations, including difficult-to-treat, substance-abusing adolescents (Bloom, 2002) and depressed adults and adolescents participating in Cognitive Behavioral Therapy (CBT; Brent et al., 1997; Kazdin, 1995). Results from these studies suggest several common factors associated with early positive change: lower severity of youth and family psychopathology at treatment onset, greater family support, less severe pathology of the caregiver and less comorbidity of youth symptoms (Brent et al., 1997; Kazdin, 1995). Similarly, in MST, research indicates that youth symptoms and family relationship factors are related to treatment outcome at termination (Huey, Henggeler, Brondino & Pickrel, 2000). Given the association of youth, family and caregiver factors with outcomes at termination for both brief treatments and MST, we hypothesized that the absence of caregiver symptomatology and youth comorbid behavior problems (substance use, internalizing symptomatology), as well as better-quality family relationships assessed early in treatment would predict early positive gains in MST.

An additional area of interest is the role of youth gender and family race/ethnicity during early treatment processes. Although results of previous MST studies have suggested that treatment outcomes are not affected by these variables (Henggeler, 1999; Schaeffer & Borduin, 20055), these studies failed to examine whether race/ethnicity or gender operated as treatment moderators. Additionally, demographic factors (e.g. age, gender, socioeconomic status) have been related to treatment process during brief treatment (Lambert, 2005). With respect to antisocial behavior, because males are often overrepresented in treatment samples, predictors of outcome uncovered in past research of mixed-gender samples may have been based primarily upon data collected from males (Lydeker, 2010). Clinically, patterns of antisocial behavior differ for boys and girls; therefore insight into whether different treatment processes are essential for overall success is important to examine (Putallaz & Bierman, 2004). Similarly, racial/ethnic background has been associated with different patterns of relationships between therapist behavior and caregiver engagement in MST (Foster et al., 2009; Ryan et al., 2013). This underscores the importance of considering whether different variables might be implicated in early treatment response for different racial/ethnic groups. Consequently, this study examined caregiver ethnicity and youth gender in an exploratory fashion as potential moderators of the relationships between predictor variables and early positive change in MST.

We examined early positive change in two ways. First, we examined whether significant changes in youth antisocial behavior occurred after 6-12 weeks in MST treatment, and whether these early positive changes were maintained after 6-8 weeks of additional treatment. Second, because the brief treatment literature primarily focuses on clinically significant change as the indicator of success, we also examined whether any youth made clinically significant gains early in treatment by determining the percentage of youths who moved from a score typical of a dysfunctional population to a score typical of the “normal” population (Jacobsen & Truax, 1991).

In sum, the purposes of the present study were to (a) examine whether youth in MST show statistically and clinically significant reductions in antisocial behavior early in MST, and whether these changes continue into later treatment and are associated with successful termination; and (b) identify factors that predict early positive changes during MST. Additionally, we (c) examined mechanisms of change typically associated with treatment outcome in MST: treatment adherence, delinquent peer association and parenting practices (Huey et al., 2000). These mechanisms were investigated early in treatment to examine the role treatment mechanisms play in early positive changes.

We hypothesized that lower caregiver symptomatology, the absence of youth drug and alcohol use, lower youth internalizing behaviors, and higher family cohesiveness and family adaptability would be related to decreased youth antisocial behavior after 6-12 weeks of treatment. Given the paucity of literature in the area, we made no specific predictions regarding youth gender and ethnicity, but instead explored their potential role as moderators. Furthermore, we anticipated that early response to treatment would be positively associated with therapist adherence to MST treatment principles and parental monitoring, and negatively associated with youth affiliation with deviant peers.

Method

Participants

The current study used data from a larger NIMH-funded (R01 MH068813) longitudinal evaluation of MST. Participants included 185 youth and their caregivers recruited from one of four licensed MST programs serving a large, urban, diverse city in the western United States. Youth were referred to participating agencies for a range of problem behaviors including school difficulties and family problems, with the majority of youth referred for exhibiting multiple antisocial behaviors. To participate in the study youth were between the ages of 12 and 17 years, living in the caregiver's home for at least one month prior to treatment, and had at least one caregiver willing to participate in treatment. The majority of the youth (65.4%) were male; average age was 15 years.

Generally consistent with MST studies, data indicated that these youth displayed serious antisocial behavior. Average number of arrests pretreatment was 2.16 (range = 0 -16), 92.4% reported having been suspended or expelled at least once from school and 67% reported receiving previous mental health or substance abuse treatment. Furthermore, in the month prior to treatment, 12% reported having carried a weapon, 7% reported having attacked someone with the intent to cause serious bodily harm, 23% reported drinking to get drunk, 5% reported abusing prescription medications, cocaine, heroin or PCP, 33% reported marijuana use and 17% reported running away from home.

Participating caregivers were mostly female (85.9%), averaged 43 years of age (range 25-73), and were typically the youth's sole caregiver (63.2%, n = 117). Fifty-three percent of the caregivers were Caucasian (n =98), 26% (n=48) Latino/a, 18% (n=33) Black, and 3% (n=6) “other.” Sixty percent (n=111) of caregivers reported having a high school education or less, and 41.6% (n = 77) reported receiving financial assistance.

For the present study, MST was provided by 49 therapists. Therapists had an average age of 31 years; 71% were female, and 80% were Caucasian, 9% Latino/a, 2% African American and 9% other. At the time of recruitment, therapists averaged 9.51 (SD = 17.35) months experience using MST and 2.62 (SD = 2.96) years post degree. Although the gender of the therapists was not similarly matched to the youth, research suggests that the relationship between the caregiver and therapist may be more indicative of outcome than the youth-therapist alliance (Tolan & Gorman-Smith, 2002). In addition to gender, 44% of caregivers were the same ethnicity as their therapist; 75 of the 79 similarities involved Caucasian therapist-caregiver pairings. Of those dyads who were not ethnically matched (n = 106), 80% were Caucasian therapists treating minority clients.

Design and Procedures

Assessments were conducted with each youth and their caregiver at five points in time, however; only data from the first four time points were analyzed during this study. Changes in youth antisocial behavior were evaluated during the early stages of treatment; between time 1 (T1; M = 3.1 weeks after intake) and time 2, (T2; M = 9.8, SD = 2.73 weeks). Maintenance of T2 changes were examined with time 3 data (T3; M = 15.8, SD = 3.30 weeks after intake). Finally, termination data (collected at T4) were used for analyses of whether improvement at T2 was related to successful termination and to evaluate clinical significance in relation to identified treatment success.

Because this was a nonexperimental, longitudinal evaluation of MST treatment process (all participants received MST as delivered in a real-world practice context), there was no control or comparison group. Each of the four participating agencies provided a variety of outpatient and day-treatment services to youth and families. Seventy-five percent of youth had received some sort of additional intervention while receiving MST: 68% of youth had received additional services beyond MST from the agency where they were treated (e.g., day treatment, residential school), and 33% had met with a mental health professional from an outside agency (e.g., psychiatrist, school counselor). Although receiving concurrent treatment may be considered a confound within other treatment modalities, multiple types of support and interactions with varying systems are common in MST. Indeed, therapists encourage additional services if they are deemed appropriate for the family as a means to reduce behaviors and risks associated with youth exhibiting serious antisocial behaviors (Henggeler et al., 2009).

Data on the number of sessions received were available for 97 of the 185 participants and indicated that on average, participants received 23.49 total sessions before termination (SD = 12.11, range = 4-62). Duration of treatment ranged from 3-43 weeks. (M = 17.6 weeks). At each timepoint, families were paid $75.00 and therapists paid $25.00 as compensation for completing the assessment battery.

Missing Data

Twelve caregivers and 15 youth did not complete T2 assessments because they declined, could not be scheduled within the appropriate time frame, or could not be located. Thirty-seven caregivers and 38 youth did not provide data at T3, for similar reasons or because they terminated treatment between T2 and T3. One youth at T1 lacked data due to a computer malfunction. One-way ANOVAs and chi-square analyses indicated that youth and caregivers who completed T2 assessments did not differ significantly from those with missing data at T2 in age, ethnicity, gender, socioeconomic status, and days in treatment prior to the T1 assessment. Similar analyses with missing data at T3 also showed no significant differences.

Predictor Variables

Caregiver Psychopathology

Caregivers completed the Brief Symptom Inventory (BSI), a 53-item psychometrically sound measure that assesses psychiatric and medical symptomatology (Derogratis, 1993). Correlations with relevant MMPI scales examining discriminant validity averaged .32 compared with convergent validity correlations that averaged .90 (Derogratis, 1993). In the present study, internal consistency for the GSI at T1 was .97.

Adolescent Drug and Alcohol Use

Adolescent substance use was assessed at T1 using 9 items adapted from Poly Drug Use subscale of the Personal Experience Inventory (Winters & Henly, 1989) and 3 additional alcohol use items, each rated on a 1 to 3 scale. Due to the infrequent endorsement of some items, the Drug Use and Alcohol Use scales were dichotomized as 0 (No use) or 1 (Self-reported use in the last 30 days). The T1 Alcohol Use internal consistency (KR-20) in the present study was .80 and the Drug Use was .65.

Adolescent Internalizing Symptoms

Youth internalizing behavior problems were assessed using the caregiver-reported Child Behavior Checklist (CBCL; Achenbach, 1991). The CBCL is a well-validated measure of youth functioning and has been found to be sensitive to MST treatment effects (Schoenwald, Halliday-Boykins & Henggeler, 2003). Raw Internalizing Behavior scores at T1 were utilized as a predictor of early positive change. In the current study, internal consistency for the Internalizing scale was .90 at T1.

Family Functioning

Family functioning was measured using the Family Adaptability and Cohesion Evaluation Scale – III (FACES-III); Olson, Portner & Lavee, 1985), a well-validated 20-item instrument measuring two key family systems constructs: Cohesion, which measures emotional bonding of family members, and Adaptability, which assesses the capacity of the family system to change in response to stress (Blaske, Borduin, Henggeler & Mann, 1989). Coefficient alphas for youth and caregiver versions at T1 for the present sample were: youth-reported Adaptability .65 and Cohesion .80; and caregiver-reported Adaptability .82 and Cohesion .90.

Potential Mechanism Variables

Affiliation with Deviant Peers

Youth reported delinquent peer associations at T2 on the Peer Delinquency Scale (PDS; Loeber, Farrington, Stouthamer-Loeber, & Van Kammen, 1998). The PDS assesses the extent to which the youth's friends engaged in delinquent acts and/or substance use during the past 30 days. Keenan, Loeber, Zhang, Stouthamer-Loeber, and Van Kammen (1995) reported that PDS scores relate to current and future delinquent behavior. Internal consistency in the present study was .87.

Parenting Skills

The Alabama Parenting Questionnaire (APQ; Shelton, Frick & Wootton, 1996) is a psychometrically-sound 42-item questionnaire designed to evaluate parental practices related to youth behavior problems. Caregiver reports on the Parental Monitoring subscale at T2 were used in this study, based on previous findings that monitoring is related to treatment outcome in MST as well as early treatment response and premature termination (Gaynor et al., 2003; Henggeler et al., 1997). Internal consistency in the present study was .79 for the Parental Monitoring subscale at T2.

Therapist Adherence

Caregivers completed the 26-item Therapist Adherence Measure (TAM; Henggeler & Borduin, 1990), which assesses therapist adherence to MST principles. Items were dichotomized in accord with Rasch-based scoring procedures and summed. TAM scores have been related to successful discharge and reductions in child behavior problems post-treatment and at one-year follow-up (Schoenwald et al., 2003). Internal consistency for the TAM at T2 was .92.

Outcome Measures

Youth Antisocial Behavior

The 47-item Self-Report Delinquency Scale (SRD; Elliot, Ageton, Huizinga, Knowles & Canter, 1983) assessed youth self-reported criminal behavior. The SRD is one of the best validated self-report delinquency measures and criterion validity has been supported with serious offenders (Elliott, Huizinga & Ageton, 1985) for the general delinquency score. In the present study, Rasch analyses indicated that dichotomous scoring was preferable to continuous scoring for the General Delinquency Scale (Chapman, 2010), so the total number of items endorsed (each item scored 1 or 0) was utilized to assess delinquency. In the current study, the SRD exhibited strong internal consistency at T1, .89, but low internal consistencies at T2 and T3, .50 and .38 respectively, likely due to the restriction of range as delinquency declined over time.

The 33-item Externalizing Behavior subscale score of the CBCL (Achenbach, 1991) assessed caregiver report of youth externalizing behavior problems. The Externalizing subscale has been found to be sensitive to change produced by MST (Henggeler et al., 2009; Schoenwald et al., 2003). In the current study, internal consistency for the Externalizing subscale was .94 at each time point (T1, T2, T3).

Total raw scores were analyzed to assess changes over time, and t-scores were used to assess clinically significant change from T1 to T2. Specifically, youth were designated as having made clinically significant change from T1 to T2 if they met two criteria. First, the Externalizing subscale t-score on the CBCL at T2 had to be in the normative range (t-score of 59 or below). Second, the amount of change each participant exhibited had to exceed the amount of variation in scores expected as a function of measurement error. This was assessed using the Guliksen-Lord-Novick (GLN) method, an application of the Reliable Change Index (Jacobsen & Truax, 1991). The RCI-GLN method accounts for regression to the mean.

Successful Termination

Therapists completed two questions developed and used by Schoenwald et al. (2003) to assess: (a) the reason for termination, and (b) who made the decision to terminate MST. Schoenwald et al. (2003) provided data indicating that scores based on these questions had acceptable construct validity. Families were considered “successful” if (a) the therapist indicated the family met or partially met treatment goals; and (b) the family and therapist made the termination decision collaboratively (Schoenwald et al., 2003).

Statistical Analyses

Therapists in the study saw more than one family, which raised the possible need to use multilevel analyses to deal with nonindependent data. Intraclass correlations (ICCs) (Cohen, Cohen, West, & Aiken, 2003) indicated that therapist differences accounted for meaningful variance in the CBCL Externalizing Behaviors scores at T2 (ICC = .12), so multilevel linear and logistic regressions using Hierarchical Linear Modeling (HLM version 7.0; Raudenbush & Bryk, 2002) were used to examine CBCL scores and the presence/absence of clinically significant change on the CBCL.1 For analyses of changes over time, three-level analyses were used, with repeated assessments (Level 1) nested within families (Level 2), nested within therapists (Level 3). Analyses of predictors of early change used two-level analyses, with families at Level 1 nested within therapists (Level 2). The ICC for the SRD score was low (.01), so multilevel analyses were not used with these data. Because the SRD utilized count data, negative binomial regression analyses were conducted using Mplus (v. 6.1) to examine changes over time and SPSS/PASW (v. 17.0; General Linear Models) to examine predictors of early change. Proportional Reduction in Variance (PRV; Peugh, 2010) statistics are provided as an estimate of local effect size for significant predictors of early change examined in HLM; odds ratios are provided for negative binomial regressions.

Caregiver, youth and therapist gender, ethnicity and age, caregiver-therapist ethnic similarity (ethnic match), caregiver receipt of financial assistance (yes/no; an indication of poverty), youth on probation (yes/no during treatment), number of days in treatment through T2 and T3, number of days in treatment prior to administration of the T1 assessment and number of sessions were examined as possible control variables for each outcome variable. Multilevel regression analyses of CBCL externalizing scores at T2, controlling for T1 CBCL externalizing score, indicated that caregiver and therapist ethnic similarity was a significant predictor of CBCL externalizing scores at T2, t (160) = 2.10, p = .05; β = .42; 95% CI [.84, .01], PRV = .030. Ethnically-similar caregiver and therapists were actually associated with higher scores (increased youth externalizing behaviors) on the CBCL. This variable was controlled in HLM analyses with CBCL Externalizing subscale scores as the outcome. No other demographic variables significantly predicted CBCL, SRD and Clinically Significant Change as outcomes.

With respect to interaction effects, caregiver ethnicity (dummy coded: Latino; African-American; Caucasian yes/no) and youth gender were analyzed as potential moderators for the relationships between predictors and outcomes. Continuous variables were grand-mean centered for HLM analyses prior to creating interaction terms. Interactions and main effects that were not significant were removed from models; the re-estimated models are presented. Because the sample of clinically improved families was small (n=19), gender and ethnicity were not evaluated as moderators for that particular outcome.

Finally, given the number of analyses used to test hypotheses, Benjamini-Hochberg (B-H; Benjamini & Hochberg, 1995) adjustments to alpha levels were utilized. This method involves sequencing a set of results from highest to lowest p values, then setting a critical value for the p required to infer statistically reliable results for each result in the sequence. B-H adjustments help control for number of false discoveries when multiple analyses are conducted, and balance reductions in experiment-wise error with reductions in power that result from alpha adjustments (Williams, Jones, & Tukey, 1999). B-H corrections were applied within families of related analyses (i.e., one set of corrections for changes over time, another for predictors of CBCL scores, another for SRD predictions, etc.). All results presented in the text met the B-H critical alpha levels established using this correction.

Results

Initial analyses examined whether youth in MST showed statistically and clinically significant reductions in deviant behavior early in MST, whether these changes continued into later treatment, and whether clinically significant improvement at T2 was associated with successful termination. Following these analyses, we tested whether youth internalizing problems and substance use, caregiver distress, and indicators of family relations at treatment onset predicted early positive changes, and whether any of these variables interacted with youth gender or caregiver ethnicity. Finally, analyses examined the relationship between indicators of early positive change and T2 therapist adherence, parental monitoring, and association with deviant peers (known mechanisms of change in MST). Table 1 contains means and standard deviations for the predictor, mechanism, and outcome variables utilized in the study.

Table 1.

Means and Standard Deviations of Predictor and Outcome Variables Evaluated in the Study

Mean (SD) or %
Time 1
Caregiver Symptomology Time 1 (BSI) 35.82 (34.96)
CBCL Internalizing Time 1 12.82 (8.96)
Total Drug Use Time 1 41%
Total Alcohol Use Time 1 27%
Family Cohesiveness - Caregiver Time 1 (FACES) 33.40 (6.38)
Family Adaptability - Caregiver Time 1 (FACES) 23.91 (5.21)
Family Cohesiveness - Youth Time 1 (FACES) 27.70 (8.55)
Family Adaptability - Youth Time 1 (FACES) 23.10 (7.58)
CBCL Externalizing Behaviors Square Root Time 1 4.47 (1.56)
SRD Total Score Time 1 21.38 (36.55)
Time 2
Therapist Adherence - Caregiver Time 2 (TAM) 109.62 (13.83)
Parental Monitoring - Caregiver Time 2 (APQ) 36.34 (6.57)
Total Score Delinquent Peers Time 2 (PDS) .46 (.47)
CBCL Externalizing Behaviors Square Root Time 2 3.77 (1.72)
SRD Total Score Time 2 10.05 (22.22)
Time 3
CBCL Externalizing Behaviors Square Root Time 3 3.52 (1.72)
SRD Total Score Time 3 9.09 (26.91)
Open in a new tab

Note. Time 1 N = 185, Time 2 n = 173, Time 3 n = 152.

Changes Over Time

The first aim of the study was to examine whether youth in MST showed statistically significant reductions in deviant behavior early in MST and whether these gains persisted into mid-treatment. Number of weeks in treatment was used as the predictor variable at Level 1. These analyses indicated significant improvements in youth antisocial behavior on both SRD and CBCL scores between T1 and T3. Weeks in treatment predicted decreases in youth SRD scores, z (306) = −3.67, p = .01, B-H critical value p = .05. Similarly, the slope of change over time on the CBCL was significant, t (313) = −8.71, p = .001, B-H critical value p = .025; β = −.07; 95% CI [−.69, .50], see Figure 1.

Figure 1.

Figure 1

Open in a new tab

Time 1 – Time 3 Changes in Youth Antisocial Behavior as Measured by SRD and CBCL – Externalizing Behavior Scores

Note. Vertical bars represent standard error of mean scores.

Clinically Significant Change

Another goal of the study was to identify whether any youth showed clinically-significant gains early in treatment. A second goal was to see whether these youth who showed clinically significant improvement at T2 also terminated treatment successfully. As noted earlier, youth had to score in the normative range on the CBCL and scores had to indicate reliable change over time. Of those youth exhibiting an initial Externalizing score in the clinical or borderline clinical range at T1 (t-score of 60 or greater, n = 119), 41 (34%) improved to the normative range at T2. Of these, 19 families met criteria for reliable change at T2 and would be considered to show clinically significant early change.2 Of these 19 families, four participants terminated treatment by T3 and the rest terminated by T4 (after 16 weeks of treatment).

Across the entire sample, 39.1% of families ultimately terminated successfully, according to therapists. In contrast, 14 of 19 (74%) families who exhibited clinically significant gains at T2 were identified successes at the time of termination. We compared this with the number of families who did not show clinically significant improvement at T2, but went on to terminate successfully (34.5%). This difference was significant, χ2 = 10.80, p < .001. Thus, early clinically significant change more than doubled the likelihood of successful termination.

Predictors of Change

Caregiver, family and youth characteristics at T1 were examined as predictors of decreased antisocial behavior at T2 using the SRD, CBCL Externalizing subscale scores and the presence or absence of Clinically Significant Change as outcome variables. Predictors also included T1 scores for SRD and CBCL respectively and ethnic match for the CBCL. No main effects or interactions involving gender were significant at p < .05. Some significant interactions (p < .05) involving caregiver ethnicity emerged; however, these interactions did not meet criteria for statistical significance when the Benjamini-Hochberg adjustment was applied.3 These models were estimated without the interactions included.

SRD

Youth drug use at T1 predicted more T2 self reported delinquent behaviors, χ2 (162, N = 165) = 6.78, p = .010; B-H critical p value = .013; β = .73; 95% CI [.18, 1.28], odds ratio = 2.08; see Table 2. Lower internalizing behaviors at T1 also significantly predicted lower SRD scores at T2, χ2 (162, N = 165) = 9.00, p =.003; B-H critical p value = .006; β = .04; 95% CI [.02, .07], odds ratio = 1.04; see Table 2.

Table 2.

Results of Negative Binomial Regression Analyses Examining Youth Drug Use and Youth CBCL Internalizing Scores at Time 1 Predicting Youth Antisocial Behavior at Time 2

B Std Error Chi Square
Drug Use at Time 1
SRD Time 1 .11 .03 11.38***
PEI Drug Use .73 .29 6.79**
CBCL Internalizing Score at Time 1
SRD Time 1 .12 .03 22.62***
CBCL Internalizing Score .04 .01 9.00**
Open in a new tab

Note. df = 162. Chi Square analyses used Wald Chi Square.

* p < .05.

**

p < .01.

***

p < .001.

Because CBCL T1 internalizing and externalizing scores correlated .69, p < .001, this effect might simply be a function of greater levels of problem behavior overall predicting worse progress. When both CBCL T1 internalizing and externalizing scores were entered as predictors, neither significantly and uniquely predicted SRD at T2. This suggests that overall levels of problem behavior rather than internalizing behaviors per se accounted for the noted effect. After controlling for CBCL externalizing scores at T1, drug use was still a significant unique predictor of T2 delinquency, χ2 (161, N = 165) = 5.85, p = .008, β = .67; 95% CI [.13, 1.27], odds ratio = 1.96.

CBCL

Multilevel logistic regressions using HLM examined whether each predictor contributed to predicting CBCL scores or clinically significant change at T2. None of these findings met the requirements for statistical significance once the B-H correction was applied.

Treatment Mechanisms

Three characteristics known to contribute to successful outcome within MST (treatment adherence [TAM score at T2], parenting [parental monitoring, APQ PMT at T2], and delinquent peer affiliation [T2 Peer Delinquency Scores]) were evaluated as possible contributors to T2 progress in the current study, with T1 scores for SRD and CBCL respectively and ethnic match for the CBCL also entered as predictors.

In accord with predictions, increased parental monitoring and decreased affiliation with deviant peers were associated with decreased antisocial behavior at T2. Parental monitoring at T2 was a significant predictor of T2 SRD scores, χ2 (160, N = 163) = 5.37, p = .020; B-H critical p value = .025; β = −.04; 95% CI [−.07, −.01], odds ratio = .96; and caregiver T2 CBCL Externalizing subscale, t (131) =−3.74, p = .001; B-H critical p value = .016; β = −.06; 95% CI [−.06, .06]; PRV = .11. In addition, affiliation with deviant peers at T2 also predicted T2 SRD scores; χ2 (161, N = 164) = 11.15, p = .001; B-H critical p value = .008; β = .05; 95% CI [.02, .08], odds ratio = 2.05.

Discussion

Although MST is a widely accepted treatment for antisocial youth (Henggeler, 1999), changes in youth antisocial behavior during the early stages of MST have not, to date, been evaluated. The purpose of the present study was to explore whether early positive changes occur in MST and predict better ultimate MST outcome, and to identify variables that contribute to these changes.

The present study contributes to growing brief treatment literature by demonstrating that families receiving a family-based EBT (MST) delivered in an effectiveness context can show significant reductions in negative behaviors within a short period of time. These findings were substantiated by both caregivers and youth. Thus, clinicians in real-world settings are able to target influential factors within a short time frame that are associated with significant changes in youth externalizing problems.

The results of the present study also indicated that clients who showed early clinically significant gains on the CBCL were likely to terminate treatment having met their treatment goals. Interestingly, despite rapid improvements, those exhibiting clinically significant drops in problem behavior continued in treatment. This result suggests that both therapists and families saw that there was more work to be done in treatment despite youths’ early improvement.

These findings are important as they suggest that therapists should assess early changes in antisocial behavior in treatment. Early positive change bodes well for later termination success and can serve as a crucial barometer of ultimate treatment outcome. On the flip side, youth who do not show early response to treatment are at greater risk for ultimately terminating without having met treatment goals. Therapists may need to devote particular scrutiny to these families and ensure that they adjust treatment strategies early on to address factors that are impeding progress.

The present study also suggests that clinicians should pay special attention to factors interfering with progress in cases with youth comorbid drug use and internalizing problems, particularly early in treatment: the caregivers of youth who reported more comorbid drug use and internalizing issues entering treatment reported more antisocial behavior at time 2. These findings are consistent with current research suggesting that comorbid problems are more difficult to treat and therefore require longer treatment duration to assist with behavior changes (Grimbos & Granic, 2009). It may be appropriate for supervisors to provide more intense consultation during the early stages of treatment for these populations, and to monitor these families more closely to ensure that therapists remain adherent and the families remain engaged in the treatment process.

Better parent monitoring and less association with deviant peers at time 2 were related to early change, suggesting that these factors may contribute to treatment outcome early in treatment. These may also be areas where it is hard to produce change in families with youth with comorbid difficulties. Changes in parenting may be particularly difficult during early treatment for parents whose youth exhibit multiple and severe antisocial behaviors; these youth may not respond quickly or may undermine the caregivers’ efforts. Clinically, therapists should monitor families for possible difficulties in these areas early in treatment and try to prevent or troubleshoot problems quickly if they arise.

Finally, an interesting finding was that caregiver and therapist ethnicity match was significantly related to higher CBCL Externalizing Scores at time 2, a finding in the opposite direction from that noted at termination in other studies (Halliday-Boykins et al., 2005). In this study, minority and Caucasian participants treated by therapists of dissimilar ethnicities fared better in MST at time 2 when compared to Caucasian participants receiving treatment from Caucasian therapists. One explanation for this finding may be that MST is highly sensitive to cultural issues and perhaps supervision is more intensive with ethnically mismatched pairs. Additionally, because Caucasian ethnicity, as a main effect, did not significantly predict treatment outcome at time 2, other explanatory factors not examined here may have influenced the relationship between caregiver and therapist ethnicity match and CBCL scores (e.g., SES). Exploratory analyses (not reported here; see Tiernan, 2011) also revealed that the majority of the matched ethnicity pairs were mismatched in gender, however, neither gender matches nor the gender of youth, caregiver and therapists alone significantly predicted outcome. Future research should evaluate the differences between family, youth and caregiver characteristics across ethnicities to further clarify the role of ethnic similarities during the early stages of treatment. Nonetheless, these findings highlight the fact that clinicians should not assume that same-ethnicity families will be easier to treat early on, compared to dissimilar-ethnicity families.

One important limitation of this study is the absence of a control or comparison group, which makes it impossible to definitively attribute current findings to MST. However, the same mechanisms implicated in pre-post change in MST (parental monitoring and delinquent peer affiliation) were associated with improved youth behavior early in treatment in the current study, which supports the contention that treatment may have contributed to these effects. These findings support the importance of the family and peer environment as contributors to youth antisocial behavior, along with their pivotal role in MST. They also support the need for therapists to address these factors early in treatment as possible mechanisms of early change.

Another limitation of this study was that the large number of analyses conducted required Benjamini-Hochberg alpha corrections, which reduced statistical power, particularly for detecting significant interactions. Previous research has mainly examined main effects of ethnicity, not whether the components contributing to treatment success may differ for different ethnic groups. In this study we did not find statistically reliable gender or ethnicity interactions. Two interactions that involved ethnicity (one involving African American caregivers, another involving Latinos) met the p < .05 criterion, but not the more stringent critical values required by the B-H corrections (see footnote 3). Although these interactions may in fact reflect spurious findings, greater statistical power is required to test interactions than to test main effects (Cohen et al., 2003). It is possible that these effects might have emerged in larger samples, or samples that were entirely African American or Latino. Regardless, this underscores the importance of more in depth consideration of ethnicity as a possible moderator of treatment process mechanisms, with future studies designed with sufficient power to detect these effects. In addition, clinicians should keep in mind that predictors of early response to treatment may vary by ethnicity in ways we were not able to detect in this study. The individualized nature of MST treatment may play a role in compensating for ethnic differences, but the importance of ethnicity and characteristics associated with treatment success should be evaluated further.

Additionally, several participants were in concurrent therapies including day treatment, outpatient treatment and residential treatment. Although, as noted earlier, receiving concurrent interventions is common in MST, and our data do not indicate the timing of additional services or which were used in early treatment. Nonetheless, any additional resources utilized by the youth and their families could have ultimately contributed to early positive changes. In addition, whether results generalize to other evidence-based family treatments is an empirical question.

Despite these limitations, the current findings have several important implications for understanding the process of family treatment for youth antisocial behavior. Although the efficacy and effectiveness of MST are well-established, our findings regarding the presence and impact of early positive gains add new information to this body of literature. The presence of early positive change may be a marker for participants who are likely to sustain long term success; characteristics contributing to short-term success may set processes in motion that contribute to overall treatment outcome in MST. Social service agencies are consistently searching for ways to maximize clients served and make the greatest impact in the briefest period of time. Results identified in this study aim to provide clinicians with additional insight into variables that may contribute to treatment response as well as pinpointing areas for attention in working with a complex and diverse treatment population.

Conversely, factors associated with less progress indicate that youth with comorbid problems may be the hardest group to treat early in treatment and may require more familial and clinical resources. It is also important for therapists to be sensitive to cultural differences and to cultural match and mismatch, and supervision and training strategies should encourage cultural awareness for therapists working with families from different cultural backgrounds without ignoring the needs to also focus on engagement and treatment process of ethnically-similar clients.

Overall, MST is an effective treatment intervention for a typically treatment-resistant population and has been successful with a variety of serious youth problems and in multiple countries, but is quite resource-intensive. Future research should continue to evaluate treatment approaches that are short-term, cost-effective, and still benefit both the participant and the community. The current study provides evidence that some antisocial youth do benefit quickly from short-term, intensive treatments even though they may not terminate immediately, that family and youth characteristics contribute to early change, and that mechanisms of early change in treatment align closely with those that predict youth status at termination.

Footnotes

1

For all HLM analyses, predictor variables were treated as fixed effects and nonrobust population-based estimates of effects were used. CBCL raw scores at all time points were square root-transformed to reduce heteroscedasticity.

2

Three of the 19 families were in the borderline clinical range (t = 60 – 62) at Time 1. We included these families in the sample for three reasons: (a) to maximize the sample size for these analyses, (b) the clinical cutoff used to determine change at T2 was the more conservative clinical cutoff of 60, and (c) families had to both meet the clinical cutoff for normative levels at T2 and show reliable change to be considered clinically improved. Interestingly, all three of these families changed substantially over time and met criteria for significant reliable change.

3

Caregiver BSI scores interacted significantly with African American ethnicity to predict SRD scores at T2, χ2 (160, N = 165) = 4.75, p =.03, which was not less than the B-H critical p value of .019; β = −.02; 95% CI [−.033, −.002]; odds ratio = .98. BSI scores significantly predicted SRD at T2 for non-African American families, χ2 (160, N = 165) = 5.02, p = .02; β = .01; 95% CI [−.001, .013], odds ratio = .83; but not for African American families, χ2 (160, N = 165) = 1.74, p = .16; β = −.01; 95% CI [−.025, .004], odds ratio = .99. A significant interaction emerged between Latino caregiver ethnicity and youth report of family cohesiveness at T1 in the prediction of CBCL Externalizing subscale scores at T2, t(159) = 2.10, p=.04, B-H critical p value = .006; β = .04; 95% CI .00, .08]; PRV = .070. Higher family cohesiveness scores, as reported by youth, significantly predicted lower CBCL Externalizing subscale scores at T2 for Latino families t(159) = −2.42, p = .02, β = −.04; 95% CI [−.07, .01], but not for non Latino families, t(159) = .40, p = .69, B = .40, 95% CI [−.03, .04].

Contributor Information

Kristine Tiernan, Alliant International University.

Sharon L. Foster, Alliant International University

Phillippe B. Cunningham, Medical University of South Carolina

Patricia Brennan, Emory University.

Elizabeth Whitmore, University of Colorado School of Medicine.

References

  1. Achenbach TM. Manual for the Child Behavior Checklist 4-18 and 1991 Profile. University of Vermont, Department of Psychiatry; Burlington, VT: 1991. [Google Scholar]
  2. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) 1995:289–300. [Google Scholar]
  3. Blaske DM, Borduin CM, Henggeler SW, Mann BJ. Individual, family and peer characteristics of adolescent sex offenders and assaultive offenders. Developmental Psychology. 1989;25:846–855. doi: 10.1037/0012-1649.25.5.846. [Google Scholar]
  4. Bloom BL. Brief psychotherapy with children and adolescents: Recent treatment outcome studies. Brief Treatment and Crisis Intervention. 2002;2:261–273. doi: 10.1093/brief-treatment/2.3.261. [Google Scholar]
  5. Borduin CM, Mann BJ, Cone LT, Henggeler SW, Fucci BR, Blaske DM, Williams RA. Multisystemic treatment of serious juvenile offenders: long-term prevention of criminality and violence. Journal of Consulting and Clinical Psychology. 1995;63:569–578. doi: 10.1037//0022-006x.63.4.569. [DOI] [PubMed] [Google Scholar]
  6. Brent DA, Holder D, Kolko D, Birmaher B, Baugher M, Roth C. A psychotherapy trial for adolescent depression comparing cognitive, family and supportive therapy. Archives of General Psychiatry. 1997;54:877–885. doi: 10.1001/archpsyc.1997.01830210125017. doi: 10.1037/0022- 006X.68.4.603. [DOI] [PubMed] [Google Scholar]
  7. Bronfenbrenner U. The ecology of human development: Experiences by nature and design. Harvard University Press; Cambridge, MA: 1979. [Google Scholar]
  8. Chapman J. [October 23, 2010];NIMH treatment response: Rasch model with dichotomous item SRD responses. 2010 Mar 24; http://fostermst.alliant.wikispaces.net/Technical+Reports+and+Analysis+Summaries.
  9. Cohen J, Cohen P, West SG, Aiken LS. Applied multiple regression/correlation analysis for the behavioral sciences. 3rd ed. Erlbaum; Hillsdale, NJ: 2003. [Google Scholar]
  10. Derogratis LR. The symptom checklist series. NSC Assessments; Minneapolis: 1993. [Google Scholar]
  11. Elliot DS, Ageton SS, Huizinga D, Knowles BA, Canter RJ. The prevalence and incidence of delinquent behavior: 1976-1980 (Tech. Rep. No. 26) Behavioral Research Institute; Boulder, CO: 1983. [Google Scholar]
  12. Elliot DS, Huizinga D, Ageton SS. Explaining delinquency and drug use. Sage; Beverly Hills, CA: 1985. [Google Scholar]
  13. Foster SL, Cunningham PB, Warner SE, McCoy DM, Barr TS, Henggeler SW. Therapist behavior as a predictor of Black and White caregiver responsiveness in Multisystemic Therapy. Journal of Family Psychology. 2009;23:626–635. doi: 10.1037/a0016228. [DOI] [PubMed] [Google Scholar]
  14. Frick PJ. Conduct disorders and severe antisocial behaviors. Springer; New York: 1998. [Google Scholar]
  15. Gaynor ST, Weersing VR, Kolko DJ, Birmaher B, Heo J, Brent DA. The prevalence and impact of large sudden impacts during adolescent therapy for depression: A comparison across cognitive-behavioral, family and supportive therapy. Journal of Consulting and Clinical Psychology. 2003;71:386–393. doi: 10.1037/0022-006x.71.2.386. doi: 10.1037/0022-006X.71.2.386. [DOI] [PubMed] [Google Scholar]
  16. Grimbos T, Granic I. Changes in maternal depression are associated with MST outcomes for adolescents with co-occurring externalizing and internalizing problems. Journal of Adolescence. 2009;32:1–9. doi: 10.1016/j.adolescence.2009.05.004. doi:10.1016/j.adolescence.2009.05.004. [DOI] [PubMed] [Google Scholar]
  17. Halliday-Boykins CA, Schoenwald SK, Letourneau EJ. Caregiver-therapist ethnic similarity predicts youth outcomes from an empirically based treatment. Journal of Consulting and Clinical Psychology. 2005;75:808–818. doi: 10.1037/0022-006X.73.5.808. doi: 10.1037/0022-006X.73.5.808. [DOI] [PubMed] [Google Scholar]
  18. Henggeler SW. Multisystemic Therapy: An overview of clinical procedures, outcomes, and policy implications. Child Psychology and Psychiatry Review. 1999;4:2–10. doi: 10.1017/S1360641798001786. [Google Scholar]
  19. Henggeler SW, Borduin CM. Family therapy and beyond: A Multisystemic approach to treating the behavior problems of children and adolescents. Brooks / Cole; Pacific Grove, CA: 1990. doi: 10.1177/0306624X9003400204. [Google Scholar]
  20. Henggeler SW, Rowland MD, Randall J, Ward DM, Pickrel SG, Cunningham PB, Miller SL, Zealberg JJ, Hand LD, Santos AB. Home based Multisystemic therapy as an alternative to the hospitalization of youth in psychiatric crisis: Clinical outcomes. Journal of the American Academy of Child and Adolescent Psychiatry. 1999;38:1331–1339. doi: 10.1097/00004583-199911000-00006. [DOI] [PubMed] [Google Scholar]
  21. Henggeler SW, Schoenwald SK, Borduin CM, Rowland MD, Cunningham PB. Multisystemic therapy for antisocial behavior in children and adolescents. 2nd ed. Guilford. Alexander Waldron Newberry; New York: 2009. 1990. [Google Scholar]
  22. Huey SJ, Henggeler SW, Brondino MJ, Pickrel SG. Mechanisms of change in Multisystemic Therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting and Clinical Psychology. 2000;68:451–467. doi:10.1037//0022-006X.68.3.451. [PubMed] [Google Scholar]
  23. Jacobsen NS, Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology. 1991;59:12–19. doi: 10.1037//0022-006x.59.1.12. doi: 0022-006X/91/S30Q. [DOI] [PubMed] [Google Scholar]
  24. Kazdin AE. Child, parent and family dysfunction as predictors of outcome in cognitive-behavioral treatment of antisocial children. Behavior Research and Therapy. 1995;33:271–281. doi: 10.1016/0005-7967(94)00053-m. Doi: 10.1016/0005-7967(94)00053-M. [DOI] [PubMed] [Google Scholar]
  25. Keenan K, Loeber R, Zhang Q, Stouthamer-Loeber, Van Kammen WB. The influence of deviant peers on the development of boys’ disruptive and delinquent behavior: A temporal analysis. Development and Psychopathology. 1995;7:715–726. doi: 10.1017/S0954579400006805. [Google Scholar]
  26. Lambert MJ. Early response in psychotherapy: Further evidence for the importance of common factors rather than “placebo effects.”. Journal of Clinical Psychology. 2005;61:855–869. doi: 10.1002/jclp.20130. [DOI] [PubMed] [Google Scholar]
  27. Loeber R, Farrington DP, Stouthamer-Loeber M, Van Kammen WB. Antisocial Behavior and Mental Health Problems: Explanatory Factors in Childhood and Adolescence. Lawrence Erlbaum; Mawhaw, NJ: 1998. doi: 10.1017/CBO9780511571138.005. [Google Scholar]
  28. Lydeker KP. Role of gender in the presentation and course of antisocial behavior for alcohol and drug-disordered youth. (Doctoral dissertation) University of California; 2010. Retrieved from eScholarship.org. [Google Scholar]
  29. Office of Juvenile Justice and Delinquency Prevention [December 15, 2012];Statistical Briefing Book. from http://www.ojjdp.ncjrs.gov.
  30. Olson DH, Portner J, Lavee Y. FACES-III. University of Minnesota, Department of Family Social Science; St. Paul: 1985. [Google Scholar]
  31. Peugh JL. A practical guide to multilevel modeling. Journal of School Psychology. 2010;48:85–112. doi: 10.1016/j.jsp.2009.09.002. doi: 10.1016/j.jsp.2009.09.002. [DOI] [PubMed] [Google Scholar]
  32. Putallaz M, Bierman KL. Duke Series in Child Development and Public Policy. Vol. 1. Guilford; NY: 2004. Aggression, antisocial behavior, and violence among girls: A developmental perspective. [Google Scholar]
  33. Raudenbush SW, Bryk AS. Hierarchical Linear Models. 2nd Ed. Sage Publications, Inc.; Newbury Park, CA: 2002. [Google Scholar]
  34. Ryan RS, Cunningham PB, Foster SF, Brennan PA, Brock LR, Whitmore E. Predictors of therapist adherence and emotional bond in Multisystemic Therapy: Testing ethnicity as a moderator. Journal of Child and Family Studies. 2013;22:122–136. doi: 10.1007/s10826-012-9638-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Schaeffer CM, Borduin CM. Long-term follow up to a randomized clinical trial of Multisystemic therapy with serious and violent juvenile offenders. Journal of Consulting and Clinical Psychology. 2005;73:445–453. doi: 10.1037/0022-006X.73.3.445. [DOI] [PubMed] [Google Scholar]
  36. Schoenwald SK, Halliday-Boykins CA, Henggeler SW. Client level predictors of adherence to MST in community service settings. Family Process. 2003;42:345–359. doi: 10.1111/j.1545-5300.2003.00345.x. doi: 10.1111/j.1545-5300.2003.00345.x. [DOI] [PubMed] [Google Scholar]
  37. Schoenwald SK, Sheidow AJ, Letourneau EJ, Liao JG. Transportability of Multisystemic therapy: Evidence for multi-level influences. Mental Health Services Research. 2003;5:223–239. doi: 10.1023/a:1026229102151. doi: 10.1023/A: 1026229102151. [DOI] [PubMed] [Google Scholar]
  38. Shelton KK, Frick PJ, Wootton JM. Assessment of parenting practices in families of elementary school-aged children. Journal of Clinical Child Psychology. 1996;25:317–329. doi: 10.1207/s15374424jccp2503_8. [Google Scholar]
  39. Tiernan KN. Predicting early positive change in Multisystemic therapy with youth exhibiting antisocial behaviors. (Doctoral dissertation) 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Timmons-Mitchell J, Bender M, Kishna M, Mitchell C. An independent effectiveness trial of Multisystemic therapy with juvenile justice youth. Journal of Clinical Child and Adolescent Psychology. 2006;35:227–236. doi: 10.1207/s15374424jccp3502_6. [DOI] [PubMed] [Google Scholar]
  41. Tolan PH, Gorman-Smith D. What violence prevention research can tell us about developmental psychopathology. Development and Psychopathology. 2002;14:713–729. doi: 10.1017/S0954579402004042. [PubMed] [Google Scholar]
  42. Williams VS, Jones LV, Tukey JW. Controlling error in multiple comparisons, with special attention to the National Assessment of Educational Progress. Journal of Educational and Behavioral Statistics. 1999;24:42–63. [Google Scholar]
  43. Winters KC, Henly GA. Personal Experience Inventory and Manual. Western Psychological Services; Los Angeles: 1989. [Google Scholar]

RESOURCES