Taking Effective Treatments to Scale: Organizational Effects on Outcomes of Multisystemic Therapy for Youth with Co-occurring Substance Use

Abstract

A prospective multi-site study examined organizational climate and structure effects on the behavior and functioning of delinquent youth with and without co-occurring substance treated with an evidence-based treatment for serious antisocial behavior (i.e., Multisystemic Therapy). Participants were 1979 youth treated by 429 therapists across 45 provider organizations in North America. Results of Mixed Effects Regression Models showed some aspects of climate and structure had no effects, some had similar effects, and some had slightly differential and sometimes counter-intuitive effects on the outcomes of these youth. Implications are considered for research to increase the array and availability of effective treatments for youth with co-occurring substance use across service sectors.

Increasing the availability of effective substance abuse and mental health treatments across usual care practice contexts is a public health priority (Compton et al., 2005). For youth with substance abuse problems, addressing this priority includes at least four challenges. (1) The majority of clinics continue to provide untested or ineffective treatments, likely for several reasons. (a) Limitations of the evidence base on effective substance abuse treatments (e.g., insufficient sample sizes, measurement of substance use, follow-up periods; Waldron & Turner, 2008) have only been remedied within the last decade. (b) Given these limitations, and practice context complexities likely to challenge implementation, treatment developers have been circumspect in launching large-scale transport efforts (Liddle et al., 2002). (2) The co-occurrence of adolescent substance abuse with mental health problems (Dennis et el., 2004; Kaminer & Bukstein, 2007) and criminal behavior is high (Chassin, 2008; McClelland, Elkington, Teplin, & Abrams, 2004), but evidence is mixed regarding the effects of empirically tested substance abuse treatments on youth with co-occurring problems (Waldron & Turner, 2008). (3) Weaknesses in the nation’s substance abuse service sector, including instability, high staff turnover, program disruption, and program discontinuation, seriously limit its capacity to serve those in need, suggesting other systems should be enlisted to do so (McClellan, Carise, & Kleber, 2003). And, (4) variables at multiple levels of the practice context (i.e., service system, organization, clinician, consumer) may affect the implementation and outcomes in practice settings of treatments whose efficacy was established in research contexts (Schoenwald & Hoagwood, 2001).

With respect to practice context effects on evidence-based treatment implementation and outcomes, theory and research on the diffusion of innovation (Rogers, 1995), transfer of technology (Backer, David, & Soucy, 1995), and innovation implementation (Klein & Knight, 2005) generated in other disciplines and industries have increasingly been harnessed to help bridge the science-practice gap in substance abuse and mental health services. The role of organizational variables, in particular, in the adoption and implementation of substance abuse services has been a focus of recent research (Simpson & Flynn, 2007). For example, survey research has identified organizational predictors of the intention to adopt a particular substance abuse treatment (Knudsen & Roman, 2004; Roman & Johnson, 2002), although results are mixed regarding actual adoption or implementation at either the clinician or organizational level (Heinrich & Fournier, 2005). Cross-sectional associations have been established among organizational climate (as well as other organizational characteristics) and client engagement, participation, and satisfaction with substance abuse services (Greener, Joe, Simpson, Rowan-Szal, & Lehman, 2007).

Neither client outcomes nor evidence-based treatments have been examined in these survey and cross-sectional studies, however. Moreover, most of them focus on adult substance use treatment. Fortunately, the inclusion of at least one empirically tested family treatment for adolescent substance abuse, Brief Strategic Family Therapy; (BSFT; Szapocznik, Kurtines, Foote, Perez-Vida, & Hervis, 1983) in the National Drug Abuse Treatment Clinical Trials Network will facilitate evaluation of the implementation and outcomes of an evidence-based substance abuse treatment for youth in usual care settings. In addition, evidence has begun to emerge regarding the adoption and initial implementation of Contingency Management (CM) for adolescent substance use in usual care settings. Findings from a prospective study of clinician interest in and initial implementation of CM for adolescents treated in public outpatient substance abuse and mental health clinics suggests the promise of both service sectors for providing evidence-based substance abuse treatment to adolescents (Henggeler et al., 2007; Henggeler et al., 2008), and highlights the role of organizational variables, regardless of service sector, in delivering on that promise.

Although prospective studies of the implementation and outcomes of evidence-based adolescent substance abuse treatments in usual care settings are underway, the limited capacity of both the substance abuse and mental health sectors to effectively meet the needs of adolescents with co-occurring mental health and substance abuse problems remains a public health concern. It thus seems reasonable to pursue another avenue for making effective treatments more quickly and widely available. That is, to examine the implementation and outcomes of a treatment designed for populations with high rates of comorbidity with substance use in adolescence, and whose clinical effectiveness and transport has already been evaluated. Specifically, Multisystemic Therapy (MST; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009) is an adolescent treatment with established clinical effectiveness and transport for youth with antisocial behavior, a population with particularly high rates of substance use disorders (e.g., McClelland et al., 2004). The National Institute on Drug Abuse (NIDA) supported the long-term follow-up of a 45-site transportability study of MST originally funded by the National Institute of Mental Health (NIMH). That study, known as the MST Transportability Study, evaluated a mediation model of treatment implementation in which organizational climate and structure were hypothesized to affect youth outcomes of MST through their effects on treatment adherence (see Schoenwald, Chapman, Sheidow, & Carter, 2009, Figure 1). The model was informed by findings on the effects of organizational climate and culture on youth served in child welfare agencies (Glisson & Hemmelgarn, 1998) and by theory and research on innovation implementation (Fixsen et al., 2005).

An aim of the NIDA-funded research was to examine the effects of organizational climate and structure on MST outcomes for delinquent youth in the MST Transportability Study with and without substance use problems. Results from the NIMH-funded study did not support the mediation model, but instead showed select organizational structure and climate variables directly predicted behavioral and criminal youth outcomes of MST; relative to the influence of treatment adherence on outcomes, however, the organizational influences were limited (Schoenwald, Carter, Chapman, & Sheidow, 2008; Schoenwald et al., 2009).

The current investigation examined the effects of organizational climate and structure on the behavioral and functional outcomes of youth with and without substance use problems through one-year post-treatment. The organizational scales typically indicating a positive climate and structure were hypothesized to generate improved outcomes, while scales typically indicating a negative climate and structure were hypothesized to generate poorer outcomes. Further, given the added complexity in treating co-occurring substance use, these organizational effects were expected to differ between youth with and without substance use problems.

Methods

Participants

Youth and caregivers

A total of 1979 youth and their caregivers participated in the MST Transportability study. The mean age for youth was 14.0 years (SD = 2.35), and most were male (65.2%, n = 1,290) and Caucasian (57.4%, n = 1135), with 18.5% (n = 366) of youth identified as African American, 5.7% (, n = 113) Asian or Pacific Islander, 4.2% (n = 83) Hispanic, 12.9% other (n = 257), and 1.3% unknown (n = 25). Almost half of youth resided with their mother or mother and a significant other (48.4%, n = 956), with remaining youth residing with both parents (15.3%, n = 303), their father or father and a significant other (6.9%, n = 138) or alternating between parents’ households (0.3%, n = 6), in special living arrangements (16.8%, n = 333), with a foster family (3.3%, n = 65), or in other non-institutional settings (8%, n = 158). The primary referral sources to treatment for youth were juvenile justice or corrections agencies (44.2%, n = 874), social services (22.9%, n = 453), mental health agencies (17.6%, n = 349), or other agencies (15.1%, n = 297). The most frequent referral reasons (multiple reasons could be endorsed for a given youth) included status offenses (47.4%, n = 939), criminal offenses (46.7%, n = 925), substance use problems (31.3%, n = 619), and school suspensions or expulsions (29.8%, n = 589).

Mean caregiver age was 40.9 years (SD = 8.54), and most were female (87.3%, n = 1,727). As with the youth, most caregivers were Caucasian (64.1%, n = 1269), and remaining caregivers were African American (18.6%, n = 368), Asian or Pacific Islander (6.2%, n = 123), Hispanic (4.9%, n = 96), or other (6.2%, n = 123). One-third of caregivers completed high school (32.1%, n = 635), and one-third completed one or more years of college (33.6%, n = 664). Half of caregivers (58.8%, n = 968) reported annual incomes of $20,000.00 or less per year. Thus, this community-based sample appears to resemble samples studied in randomized trials of MST involving youth with serious antisocial behavior (Henggeler, Melton, Smith, Schoenwald, & Hanley, 1993; Henggeler, Melton, Brondino, Scherer, & Hanley, 1997; Henggeler, Pickrel, & Brondino, 1999).

Substance using youth (n = 412) were identified if they met all three of the following criteria at pre-treatment: 1) Youth substance abuse was a primary reason for referral; 2) Caregivers endorsed youth substance use in the past 6 months on the Child Behavior Checklist (CBCL; Achenbach, 1991); and 3) Caregivers endorsed items on the Vanderbilt Functioning Inventory (VFI; Bickman, Lambert, Karver, & Andrade, 1998) indicating youth drug and alcohol use in the past 6 months. Although youth were not assessed directly for the presence of substance use problems, the triangulation of these three criteria provides a strong, multi-source indication that substance use problems were present. CBCL item #105 (i.e., youth use of drugs or alcohol for non-medical purposes) has been shown to correlate highly with youth-reported drug and alcohol use in MST trials. Although the VFI was not used in those trials, items 22 and 23 ask caregivers to report whether (in the past six months) the youth has gotten drunk or into difficulty for drinking, and whether the youth has taken illegal drugs. Substance use status was unknown for thirteen of the 1,979 youth.

Therapists

A total of 489 therapists consented to participate in the study, but not all therapists treated study cases, and some families were treated by more than one therapist due to therapist illness, vacation, or leave. Accordingly, primary therapists (n = 429) were identifiable for 1,888 of the 1,979 families in the study. “Primary therapist” signifies the therapist treating the family for the entire treatment episode or, for families treated by more than one therapist, the therapist providing treatment for the majority of the family’s treatment episode. A primary therapist could not be identified for 91 families, each of which was treated by more than one therapist for approximately equal lengths of time. Analyses of demographic, youth problem severity, and treatment data show there are no differences between families with and without a primary therapist (Schoenwald, Chapman, & Sheidow, 2006). The majority of therapists was female (64.3%, n = 276) and had a master's degree (53.8%, n = 231), while 31.5% (n = 135) had bachelor’s degrees, 1.9% (n = 8) had a doctoral degree, and 0.9% (n = 4) had an “Other” or unspecified degree.

Procedures

Study procedures have been detailed previously (see Schoenwald et al., 2003; Schoenwald, Sheidow & Letourneau, 2004) and are briefly described here.

Youth and families

All youth referred for MST treatment at the study sites were eligible for the Transportability study except youth with autism or severe mental retardation. Families were recruited for study participation by clinical supervisors or therapists at the provider organizations upon referral to MST, and (82% of those recruited consented for study participation). Research assistants administered pre-treatment, post-treatment, and follow-up assessments of youth behavioral and functioning problems by telephone, and caregivers were reimbursed $20.00 per each completed assessment. Participation in the study was voluntary and the Institutional Review Board of the university approved all procedures.

Therapists

All therapists in the MST programs were eligible to participate (n = 491), and all but two of them consented to do so. Informed consent from therapists was obtained during site visits. For therapists employed after the study began, a telephone procedure was used to obtain informed consent. Therapist demographic, educational, and professional experience data were obtained upon enrollment in the study, as was the baseline organizational assessment. Therapists completed the organizational assessment semi-annually during the 2.3- year clinical implementation phase of the study to ensure organizational assessment data could be linked with data from the treatment episodes of all youth treated during the study. All therapists within a site were scheduled to complete each semi-annual assessment at the same date and time immediately following their weekly supervision meeting. Therapists employed after the study began provided the initial organizational assessment upon enrollment in the study, and semi-annual assessments as scheduled for the organization. Therapists sealed the completed forms in separate stamped and addressed envelopes provided by the research staff and an administrative assistant collected these sealed envelopes and mailed them to the research staff.

Provider organizations

There was an average of 9.5 (SD = 7.9) therapists per provider organization (range 1 to 40) and an average of 42.0 (SD = 27.4) youth per provider organization (range 3 to108). The vast majority (93.7%) of the organizations were privately held and contracted by public agencies to operate MST programs.

Clinical intervention

Because details of the clinical intervention and multi-component implementation protocol have been described elsewhere (Henggeler et al., 2009), the description here will be brief. MST is an intensive, family-based treatment originally developed for delinquent youths at imminent risk of incarceration or other out-of-home placements and their families. MST specifically targets those factors in each youth’s social ecology (family, peers, school, neighborhood, and community) contributing to the youth’s antisocial behavior. MST treatment is informed by the social ecological theory of human behavior articulated by Bronfenbrenner (1979) and by prospective research identifying the multiple predictors of serious antisocial and related behavior in adolescents. Given the youths’ imminent risk of placement, overarching treatment goals often relate to keeping the youth in the home and reducing criminal behavior. Specific goals, and the interventions to achieve them, are designed collaboratively with the youth’s caregivers, who also implement the majority of the interventions (initially with the instrumental and social support of the therapist).

The combination of intervention techniques applied varies in accordance with the circumstances of each youth and family. Thus, session-by-session guides are not used to implement MST. Instead, nine principles are used to guide the MST assessment and intervention process, balancing specification of the treatment model with responsiveness to the needs and strengths of each youth and family. Ongoing assessment and intervention follows an analytic process that encourages clinicians to: generate specific hypotheses about the combination of factors that sustain a particular problem behavior; provide evidence to support the hypotheses; test the hypotheses by intervening; assess the impact of the intervention; and begin the assessment process again. Interventions typically include improving caregiver discipline practices, enhancing family affective relations, decreasing youth association with deviant peers, increasing youth association with prosocial peers and activities, improving youth school or vocational performance, and developing an indigenous support network (e.g., extended family, neighbors, and friends) to help caregivers achieve and maintain changes. Specific treatment techniques used to facilitate these gains are integrated from therapies with the most empirical support, including cognitive behavioral, behavioral, and the pragmatic family therapies.

A home-based model of service delivery is used to provide comprehensive and intensive clinical interventions when and where they are needed (i.e., clinicians are available 24 hours/day, 7 days/week to respond to crises), with duration and frequency of treatment sessions varying in accordance with changing circumstances, needs, and treatment progress. MST therapists operate in teams of two to four therapists (plus the clinical supervisor). Each therapist’s caseload ranges from four to six families so that therapists are able to provide sufficiently intensive and individualized services to families. The length of treatment in clinical trials with juvenile offenders has ranged from an average of 13 to 17.5 weeks (Henggeler et al., 1993; Henggeler et al., 1997; Henggeler et al., 1999). Average length of treatment for the current sample was 21.9 weeks (SD = 10.1, interquartile range of 15 to 27 weeks).

Implementation protocol

Described in detail elsewhere (Henggeler et al., 2009; Schoenwald, 2008), a quality assurance and improvement system designed to replicate procedures and resources provided to therapists in randomized trials of MST is used to support the implementation of MST in community settings. This system is provided by a university-organization, MST Services. The system includes an intensive 5-day orientation for therapists and supervisors to MST theory and practice; quarterly 1.5-day booster sessions for therapists and supervisors; at least weekly group supervision of therapists by an on-site clinical supervisor trained in the MST supervisory protocol (Henggeler & Schoenwald, 1998); weekly group (i.e., supervisor and therapists) phone consultation with an MST expert who follows a specified consultation protocol (Schoenwald, 1998); and feedback from validated measures of therapist adherence (Henggeler, Borduin, Schoenwald, Huey, & Chapman, 2006), supervisor adherence (Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002), and consultant adherence (Schoenwald et al., 2004). In addition, an organizational manual, ongoing organizational consultation, and semi-annual formal program reviews are provided. The semi-annual Program Implementation Review (PIR) is designed to enable organization implementing MST, key stakeholders (including referral and funding sources), and MST purveyor to examine together key program performance indicators derived from the Goals and Guidelines document established for the MST program during the pre-implementation phase. Web-based and telephone forums are available for peer learning among MST program directors. Finally, the MST consultant helps the team to address organizational and stakeholder barriers to treatment implementation in specific cases during weekly consultation and at booster training sessions.

Measures

Youth behavioral and functioning problems were assessed by caregiver reports at pre-treatment (T1), immediately post-treatment (T2), 6 months post-treatment (T3) and 12 months post-treatment (T4).

Child Behavior Checklist (CBCL; Achenbach, 1991)

The CBCL is one of the best-validated measures of child behavioral functioning and has been normed with various age and ethnic groups (Achenbach, 1991; Drotar, Stein, & Perrin, 1995). The measure describes 113 behavior problem items applicable to children aged 2 to 18 years. Caregivers are asked to rate the extent to which the description is true of their child during the previous 6 months on a scale that ranges from 0, “not true,” to 2, “very often or often true.” T-scores for the broadband Externalizing and Internalizing scales were analyzed.

Vanderbilt Functioning Inventory (VFI; Bickman, Lambert, Karver, & Andrade, 1998)

Problems in youth psychosocial functioning was assessed using the VFI. Content areas indexed by the 24-item VFI are antisocial behavior, problems at home, problems at school, problems with peers, and self-harm. Analyses of the reliability and validity of the VFI indicate adequate internal consistency (.71), concurrent validity (e.g., significant correlations with established measures in the expected directions), predictive validity (e.g., VFI scores predicted cost of treatment and use of residential care), and incremental validity (e.g., VFI scores accounted for a significant portion of variance of treatment cost and residential care after accounting for the variance accounted for by other measures) (Bickman et al., 1998). VFI probability scores are computed by summing raw item scores (0 or 1) and dividing by the number of completed items. Thus, scores can range from .00 to 1.00, and we observed a pre-treatment (T1) mean of .42 (SD = .20).

Organizational climate

Organizational climate was assessed as each organization began the study (baseline) and semi-annually during the treatment portion of the study, and for therapists and supervisors hired by the organization after the study began, upon their enrollment in the study and in accordance with the organization’s subsequent scheduled semi-annual assessment date. Climate was assessed using reports on ten well-known scales from the Psychological Climate Questionnaire originally assembled by James and Sells (1981) and used by Glisson and colleagues in research on child welfare and juvenile justice systems (Glisson & Hemmelgarn, 1998; Glisson & James, 2002).

Individual perceptions of the impact of the work environment on one’s own well-being and work comprise the construct known as psychological climate; when these perceptions are shared by individuals within a work unit, they are typically aggregated to index the construct known as organizational climate (for reviews pertaining to the conceptualization, measurement, and statistical modeling of organizational constructs including climate see Chan, 1998; Glisson & James, 2002; Klein & Kozlowski, 2000). Included in the MST Transportability Study were ten scales: Fairness, Role Clarity, Role Overload, Role Conflict, Cooperation, Growth and Advancement, Job Satisfaction, Emotional Exhaustion, Personal Accomplishment, and Depersonalization. Item responses on a 5-point Likert-type response scale vary by instrument scale and include a range from “Strongly disagree” to “Strongly agree;” or from “Practically never” to “Almost always.” In the current study, Cronbach’s alpha reliabilities at the first administration in this sample ranged from 0.62 (fairness) to 0.92 (job satisfaction) with only the fairness construct having reliability less than 0.70.

A mean of 2.4 reports (range 1 – 7) were obtained per respondent. For each climate scale, an individual’s reports were averaged across the baseline and semi-annual administrations. Examination of interrater agreement on the organizational constructs using r _WG (James, Demaree, & Wolf, 1993; LeBreton, James, & Lindell, 2005) showed agreement ranged from medium to high (r _WG 59 – .78). Likewise, although some intraclass correlation coefficients for the climate scales were small (e.g., Emotional Exhaustion = 0.01), eight of the thirteen were greater than 0.06, with a maximum of 0.38 (for Growth and Advancement). This indicates that there was substantial consensus in therapist ratings for many of the scales. At the same time, however, in each case the majority of the variance was attributable to the therapist providing the report. Thus, as described in the data analysis strategy section, statistical models were used to simultaneously evaluate the effects of organizational average scores and therapist perceptions relative to that average on youth behavior change through one year post-treatment.

Organizational structure

Organizational structure, specifically the degree of formalization (explicit rules and procedures governing employee behavior) and centralization (degree to which authority and decision-making are concentrated vs. dispersed) were assessed simultaneously with organizational climate using three brief scales frequently administered together s a single instrument. The scales are: (1) Participation in Decision-making (8 items; from Hage & Aiken, 1967); (2) Hierarchy of Authority (4 items; from Hall, 1963); and (3) Procedural and Rule Specification (3 items; from Hall, 1963). As in prior studies (see, e.g., Glisson, 1996; Glisson & Martin, 1980), all original items from the three scales were administered. Response options were on 5-point rating scales. In the current sample, Cronbach’s alphas for the three structure scores were 0.75 (procedural specification), 0.86 (hierarchy of authority), and 0.89 (decision making). As with the climate data, therapist reports on the structure scales were averaged across administrations, and the models simultaneously evaluated organizational average scores and scores of therapist perceptions relative to that average.

Data Analysis Strategy

Data structure

Three features of the present data structure have implications for the data analysis strategy. First, the data are nested such that up to 4 repeated measurements of youth behavior problem and functioning outcomes (level-1, n_t ≈ 6742) are nested within i youths/caregivers (level-2, n_i ≈ 1872) who are nested within j primary therapists (level-3, n_j ≈ 427) who are nested within k provider organizations (level-4, n_k ≈ 45). This implies a 4-level mixed-effects regression model with variance in repeated measurements of youth outcomes partitioned among measurement occasions, youths/caregivers, therapists, and provider organizations. Second, there was variability in the spacing of assessments across youths/caregivers, and given four measurement occasions, there was the possibility of non-linear change trajectories. As a result, we used a variation on the discontinuous regression model (Singer & Willett, 2004, p. 189–242) to separately model change during and after treatment. The linear term for time was computed as the number of months between the individual’s post-treatment assessment and each of the subsequent assessments. The pre-treatment score on the outcome was entered as a covariate. Consequently, intercept effects and differences in effects between drug users and non-drug-users could be interpreted as indices of change during treatment; and, linear slope effects, estimated with three waves of data, could be interpreted as post-treatment change effects.

Third, there are constructs of interest at each of the four levels of nesting. Specifically, there are measurement occasion covariates (i.e., polynomial terms), youth-level covariates (i.e., substance use status), therapist-level covariates (i.e., therapist perception of organizational climate), and provider-level covariates (i.e., 10 climate and 3 structure scales). Of note, models incorporating climate scores at both therapist and provider organization levels are known as frog-pond models (e.g., Enders & Tofighi, 2007). This strategy is common in the organizational literature when individual reports are provided with reference to a common entity (Raudenbush & Bryk, 2002; Klein & Kozlowski, 2000). Frog-pond modeling attempts to disentangle the effect of the provider organization (i.e., the environment of the pond) on the outcome of interest from the standing of an individual (i.e., the frog) within that organization (and how that standing relates to the outcome of interest. Although multiple strategies can be utilized for modeling such data (e.g., Chan, 1998), the frog-pond model permits simultaneous evaluation of the effects of the overall organizational level of the construct and the individual respondent’s deviation from the organization’s score. This approach, detailed further in the Statistical models section, was undertaken in the current study to retain both therapist and organizational information about the constructs of climate and structure in this unique and large sample of provider organizations and therapists implementing an evidence-based treatment for youth.

Statistical models

Mixed-effects regression models (MRMs) with full information maximum likelihood estimation were conducted using SAS PROC MIXED, version 9.1 (SAS Institute, 2002). Utilizing the model building approach described by Singer and Willett (2003), random effects were specified according to the likelihood ratio test and theoretical considerations. Two models were conducted for each combination of outcome and organizational climate score. The first model evaluated organizational effects as predictors of change in caregiver-reported youth behavior problems and functioning. The organizational effects were specified according to the frog-pond model (Shinn, 1990). Specifically, the provider organization score was computed as the mean of all therapist scores within a given provider, and the therapist’s perception was computed as the individual therapist’s deviation from the respective provider organization’s mean score. The provider organization mean score was centered around the grand provider organization mean prior to entry, and as noted previously, the provider organization effects and the therapist perception effects were independently interpreted.

The second model was identical to the first, but incorporated a dichotomous youth substance use status indicator as well as interactions between substance use status and the therapist and provider organizational scores, and their interactions with the time terms. The substance use status indicator and the corresponding interaction terms were not centered in the model. Evidence of moderation was provided by a statistically significant interaction between substance use status (0,1) and therapist and/or provider organizational climate scores.

Missing data

A total of 1,872 families (94.6%) had an identifiable primary therapist, a youth with known substance use status, and data from behavioral and functioning assessments at multiple time points, and these families were included in the analyses. Provider organizations were identifiable for each of the 427 primary therapists who treated these 1872 families. The average number of assessments completed by families was 3.7 (SD = 0.68), with 2.0%, 6.4%, 11.1%, and 80.5% of families completing 1, 2, 3, or 4 assessments, respectively. Of the families completing at least one assessment, all had data sufficient for determining substance use status. Similarly, of the 427 primary therapists, all provided at least one organizational climate report, and as a result, all of the provider organizations had useable organizational climate scores.

Results

Main Effects of Organizational Climate and Structure on Youth Outcomes

Table 1 reports, for each organizational climate and structure subscale, the effects of the provider organizational mean and individual (therapist) deviation scores on all youth in the sample at post-treatment controlling for pre-treatment, and over time through on average one-year post-treatment follow-up.

Table 1.

Effects of Climate and Structure on Youth CBCL Externalizing, CBCL Internalizing and VFI Functioning Problem Scores

	Post-treatment Scores				Follow-Up Scores
Organizational Scale	Organizational Mean		Individual Deviation		Organizational Mean		Individual Deviation
	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE
CBCL Internalizing score
Climate
Cooperation	−0.06	0.12	0.00	0.08	0.00	0.02	0.00	0.01
Depersonalization	−0.05	0.17	−0.06	0.07	−0.01	0.02	0.00	0.01
Emotional exhaustion	−0.01	0.08	−0.03	0.04	0.01	0.01	0.00	0.01
Fairness	0.02	0.08	−0.01	0.05	−0.01	0.01	0.00	0.01
Growth and advancement	0.10	0.05	0.03	0.05	0.01	0.01	0.00	0.01
Job satisfaction	0.01	0.05	−0.01	0.03	0.00	0.01	0.00	0.00
Personal accomplishment	0.07	0.11	−0.10	0.06	−0.01	0.01	−0.01	0.01
Role clarity	0.01	0.08	−0.04	0.06	0.00	0.01	−0.01	0.01
Role conflict	0.01	0.07	−0.01	0.04	0.01	0.01	0.00	0.00
Role overload	0.11	0.08	−0.02	0.04	0.02	0.01	0.00	0.01
Structure
Hierarchy of authority	−0.03	0.08	0.00	0.04	0.00	0.01	0.00	0.00
Participation in decision making	0.00	0.13	−0.10	0.08	0.02	0.02	0.00	0.01
Procedural specification	0.02	0.21	0.01	0.09	−0.02	0.03	−0.01	0.01
CBCL Externalizing score
Climate
Cooperation	0.07	0.13	0.02	0.00	0.03	0.02	0.00	0.01
Depersonalization	−0.05	0.17	0.02	−0.01	−0.01	0.02	0.01	0.01
Emotional exhaustion	−0.02	0.08	0.01	−0.03	0.01	0.01	−0.01	0.01
Fairness	0.02	0.08	0.01	0.00	−0.01	0.01	0.00	0.01
Growth and advancement	0.22**	0.05	0.01	0.02	0.03**	0.01	0.00	0.01
Job satisfaction	0.09	0.05	0.01	−0.01	0.02**	0.01	0.00	0.00
Personal accomplishment	0.08	0.11	0.01	0.00	0.00	0.01	0.00	0.01
Role clarity	−0.03	0.08	0.01	0.02	0.00	0.01	−0.01	0.01
Role conflict	−0.03	0.07	0.01	0.00	0.00	0.01	0.00	0.01
Role overload	0.06	0.08	0.01	−0.02	0.01	0.01	−0.01	0.01
Structure
Hierarchy of authority	−0.03	0.08	0.01	0.00	0.01	0.01	0.00	0.00
Participation in decision making	0.07	0.14	0.02	−0.02	0.04*	0.02	0.01	0.01
Procedural specification	−0.15	0.21	0.03	−0.06	−0.03	0.03	−0.01	0.01
VFI Functioning Problem scorea
Climate
Cooperation	−0.04	0.19	0.03	0.11	0.06*	0.03	−0.01	0.02
Depersonalization	0.19	0.26	0.03	0.11	0.02	0.04	0.01	0.02
Emotional exhaustion	−0.15	0.13	−0.14*	0.06	0.00	0.02	−0.03	0.01
Fairness	−0.18	0.13	0.09	0.08	−0.03*	0.02	0.00	0.01
Growth and advancement	0.16	0.09	0.05	0.08	0.08**	0.01	0.00	0.01
Job satisfaction	0.07	0.07	0.02	0.04	0.04**	0.01	0.00	0.01
Personal accomplishment	0.01	0.17	0.07	0.08	0.01	0.02	0.01	0.01
Role clarity	−0.22	0.12	0.09	0.08	−0.01	0.02	−0.02	0.01
Role conflict	−0.10	0.10	−0.07	0.05	−0.02	0.01	−0.01	0.01
Role overload	−0.21	0.13	−0.10	0.06	0.01	0.02	−0.02	0.01
Structure
Hierarchy of authority	0.05	0.12	−0.08	0.05	0.02	0.02	0.00	0.01
Participation in decision making	0.10	0.21	−0.05	0.12	0.03	0.03	0.02	0.02
Procedural specification	−0.70*	0.32	−0.20	0.13	−0.13**	0.05	−0.02	0.02

Note.

^*p < .05.

^**p < .01

^aFunctioning Score coefficients and standard errors are multiplied by 100 for ease in reporting.

Internalizing and Externalizing behavior

There were no main effects of climate variables on youth Internalizing scores. Higher organizational average levels of Growth and Advancement predicted increased Externalizing behavior post-treatment, β = 0.22, SE = 0.05, t(42) = 4.0, p <.01, and over time through follow-up, β = 0.03, SE = 0.01, t(6156) = 4.7, p < .01. Organizational average levels of Job Satisfaction predicted linear increases in Externalizing behavior during follow-up, β = 0.02, SE = 0.01, t(6156) = 2.6, p < .01. One organizational structure variable, Participation in Decision Making, predicted post-treatment increases in Externalizing behavior, β = 0.04, SE = 0.02, t(6138) = 2.3, p < .05. That these attributes of organizational climate and structure generally conceptualized as positive predicted increases rather than decreases in youth Externalizing behaviors during or after treatment countered expectations.

Functioning problems

One climate variable, Fairness, related to youth Functioning problems in the expected direction: Higher organizational average levels of Fairness predicted decreases in youth Functioning problems over time following treatment completion, β = −0.03, SE = 0.02, t(6478) = 1.99, p < .05. Relations among four climate variables (i.e., Cooperation, Growth and Advancement, Emotional Exhaustion, Job Satisfaction) and youth Functioning problems countered expectations. Higher average organizational levels of Cooperation, β = 0.06, SE = 0.03, t(6478) = 2.3, p < .05, Growth and Advancement, β = 0. 08, SE = 0.01, t(6478) = 6.8, p < .01, and Job Satisfaction, β = 0.04, SE = 0.01, t(6478) = 3.4, p < .01, predicted increases in youth Functioning problems over time following treatment completion. Therapist perceptions of greater Emotional Exhaustion relative to the organizational average level predicted lower levels of youth Functioning problems post-treatment, β = − 0.14, SE = 0.06, t(42) = −2.2, p < .05.

One organizational structure variable related to youth Functioning problems (in the expected direction). Higher organizational average levels of Procedural Specification predicted greater reductions in youth Functioning problems at the end of treatment, β = −0.70, SE = 0.32, t(42) = 2.2, p < .05, and decreasing levels of Functioning problems over time after treatment completion, β = −0.13, SE = 0.05, t(6449) = 2.8, p < .01. There were no significant effects on youth Functioning problems of individual therapist perceptions (i.e., individual deviations from organizational means) on any of the organizational structure subscales.

Moderation by Substance Use of Organizational Climate and Structure Effects on Outcomes

To assess moderation by youth substance use problems (pre-treatment) of the organizational climate and structure effects on outcomes, the aforementioned four-level models were expanded to include a binary indicator for the youth substance use problems at pre-treatment, interactions between youth pre-treatment substance use and the intercept, and interactions between youth substance use and two terms representing the organizational climate and structure scores (organizational average and individual therapist deviation from that average). Of particular interest was interpretation of the three-way interactions between the climate or structure variables, change in youth behavior or functioning problems over time (growth), and youth pre-treatment substance use. Table 2 presents the organizational effect estimates (both organizational average level and individual therapist deviation) on change during treatment and significance tests for youth with and without substance use problems. Table 3 presents these effect estimates on change over time after treatment and significance tests for youth with and without substance use problems.

Table 2.

Moderation by Substance use of Climate and Structure Effects on Youth Post-treatment CBCL Externalizing, CBCL Internalizing, and VFI Functioning Problem Scores

	Post-treatment Controlling for Pre-Treatment
	Non substance using youth				Substance using youth
Organizational Scale	Organizational Mean		Individual Perception		Organizational Mean		Individual Perception
	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE
CBCL Internalizing score
Climate
Cooperation	−0.14	0.15	−0.06	0.08	0.28	0.27	0.26	0.19
Depersonalization	−0.17	0.20	−0.13	0.08	0.37	0.36	0.34	0.18
Emotional exhaustion	−0.01	0.09	−0.01	0.05	0.02	0.17	−0.10	0.11
Fairness	−0.01	0.09	−0.04	0.06	0.16	0.20	0.16	0.14
Growth and advancement	0.13	0.06	0.01	0.06	−0.10	0.14	0.10	0.13
Job satisfaction	0.02	0.05	−0.04	0.03	−0.03	0.10	0.13	0.07
Personal accomplishment	0.00	0.14	−0.13*	0.06	0.17	0.21	0.17	0.13
Role clarity	−0.06	0.09	−0.07	0.06	0.27	0.17	0.16	0.15
Role conflict	0.01	0.08	0.02	0.04	−0.02	0.15	−0.14	0.09
Role overload	0.11	0.09	−0.01	0.05	0.01	0.20	−0.04	0.11
Structure
Hierarchy of authority	−0.10	0.16	−0.05	0.09	0.30	0.27	−0.22	0.19
Participation in decision making	−0.03	0.09	−0.02	0.04	0.01	0.16	0.06	0.08
Procedural specification	−0.18	0.25	0.04	0.10	0.63	0.42	−0.18	0.21
CBCL Externalizing score
Climate
Cooperation	−0.06	0.15	−0.02	0.08	0.24	0.28	0.15	0.18
Depersonalization	−0.18	0.20	−0.11	0.08	0.25	0.36	0.39*	0.18
Emotional exhaustion	−0.04	0.10	0.00	0.05	0.08	0.17	−0.14	0.11
Fairness	−0.01	0.09	−0.02	0.06	0.19	0.20	0.13	0.14
Growth and advancement	0.19	0.06	0.02	0.06	0.01	0.14	0.04	0.13
Job satisfaction	0.07	0.06	−0.03	0.03	0.00	0.10	0.09	0.07
Personal accomplishment	0.01	0.14	−0.04	0.06	0.06	0.21	0.17	0.13
Role clarity	−0.13	0.09	−0.01	0.06	0.24	0.17	0.17	0.15
Role conflict	0.00	0.08	0.04	0.04	0.02	0.15	−0.17	0.09
Role overload	0.03	0.09	0.00	0.05	0.07	0.21	−0.10	0.11
Structure
Hierarchy of authority	−0.03	0.09	−0.01	0.04	−0.06	0.16	0.06	0.08
Participation in decision Making	−0.10	0.17	0.01	0.09	0.35	0.27	−0.20	0.19
Procedural specification	−0.41	0.25	−0.08	0.09	0.65	0.42	0.04	0.21
VFI Problem Functioning scorea
Climate
Cooperation	−0.11	0.29	0.08	0.15	−0.25	0.51	−0.06	0.34
Depersonalization	0.26	0.39	0.01	0.15	−0.70	0.67	−0.29	0.33
Emotional exhaustion	−0.12	0.18	−0.02	0.08	0.42	0.31	0.02	0.20
Fairness	0.03	0.18	0.01	0.11	−0.38	0.36	0.03	0.26
Growth and advancement	−0.30*	0.13	0.20	0.11	−0.04	0.27	−0.33	0.24
Job satisfaction	−0.09	0.11	0.10	0.06	−0.09	0.19	−0.19	0.13
Personal accomplishment	−0.02	0.27	0.10	0.11	−0.32	0.40	−0.31	0.25
Role clarity	0.02	0.18	0.19	0.11	−0.24	0.31	−0.34	0.27
Role conflict	−0.04	0.15	−0.04	0.07	0.19	0.27	0.19	0.18
Role overload	−0.27	0.18	−0.02	0.08	0.29	0.36	0.03	0.21
Structure
Hierarchy of authority	−0.26	0.17	−0.17*	0.07	0.47	0.31	0.19	0.16
Participation in decision making	−0.05	0.32	−0.30	0.15	1.03*	0.52	0.45	0.36
Procedural specification	−0.06	0.47	−0.25	0.17	0.53	0.80	0.34	0.40

Note.

^*p < .05.

^**p < .01

^aFunctioning Score coefficients and standard errors are multiplied by 100 for ease in reporting.

Table 3.

Moderation by Substance use of Climate and Structure effects on Youth Follow-Up CBCL Externalizing, CBCL Internalizing, and VFI Functioning Problem Scores

	Post-treatment Follow-up Slope
	Non substance using youth				Substance using youth
Organizational Scale	Organizational Mean		Individual Deviation		Organizational Mean		Individual Deviation
	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE
CBCL Internalizing score
Climate
Cooperation	−0.02	0.02	−0.01	0.01	0.06	0.04	0.05	0.02
Depersonalization	−0.04	0.03	−0.02	0.01	0.09*	0.05	0.07*	0.02
Emotional exhaustion	0.02	0.01	0.00	0.01	−0.02	0.02	−0.03	0.01
Fairness	−0.02	0.01	0.00	0.01	0.05	0.03	0.02	0.02
Growth and advancement	0.01	0.01	0.00	0.01	−0.01	0.02	0.02	0.02
Job satisfaction	0.00	0.01	−0.01	0.00	0.01	0.01	0.02*	0.01
Personal accomplishment	−0.03	0.02	−0.02*	0.01	0.05*	0.03	0.03	0.02
Role clarity	−0.02	0.01	−0.01	0.01	0.06*	0.02	0.02	0.02
Role conflict	0.02	0.01	0.00	0.01	−0.03	0.02	−0.02	0.01
Role overload	0.02*	0.01	0.00	0.01	−0.04	0.03	−0.01	0.02
Structure
Hierarchy of authority	0.00	0.02	0.01	0.01	0.04	0.04	−0.06	0.03
Participation in decision making	0.00	0.01	0.00	0.01	0.00	0.02	−0.02*	0.01
Procedural specification	−0.05	0.03	−0.01	0.01	0.09	0.06	−0.04	0.03
CBCL Externalizing score
Climate
Cooperation	0.00	0.02	−0.01	0.01	0.04	0.04	0.04	0.03
Depersonalization	−0.05	0.03	−0.01	0.01	0.09	0.05	0.05*	0.02
Emotional exhaustion	0.02	0.01	0.00	0.01	−0.04	0.02	−0.03	0.02
Fairness	−0.01	0.01	−0.01	0.01	0.04	0.03	0.02	0.02
Growth and advancement	0.03**	0.01	−0.01	0.01	−0.01	0.02	0.02	0.02
Job satisfaction	0.01	0.01	−0.01	0.00	0.01	0.01	0.01	0.01
Personal accomplishment	−0.02	0.02	0.00	0.01	0.03	0.03	0.03	0.02
Role clarity	−0.02	0.01	−0.01	0.01	0.04	0.02	0.03	0.02
Role conflict	0.01	0.01	0.00	0.01	−0.03	0.02	−0.03*	0.01
Role overload	0.01	0.01	0.00	0.01	−0.05	0.03	−0.03	0.02
Structure
Hierarchy of authority	0.01	0.01	0.00	0.01	−0.01	0.02	0.00	0.01
Participation in decision making	0.02	0.02	0.02	0.01	0.02	0.04	−0.05	0.03
Procedural specification	−0.06	0.03	−0.02	0.01	0.04	0.06	0.00	0.03
VFI Problem Functioning scorea
Climate
Cooperation	0.01	0.03	−0.01	0.02	−0.03	0.07	0.05	0.05
Depersonalization	−0.01	0.05	0.00	0.02	−0.02	0.09	0.06	0.05
Emotional exhaustion	−0.01	0.02	−0.02*	0.01	0.02	0.04	−0.02	0.03
Fairness	−0.03	0.02	0.00	0.01	−0.02	0.05	0.03	0.04
Growth and advancement	0.06**	0.01	−0.02	0.01	−0.05	0.04	0.07*	0.03
Job satisfaction	0.03*	0.01	−0.01	0.01	−0.04	0.03	0.02	0.02
Personal accomplishment	0.00	0.03	0.01	0.02	−0.04	0.05	0.02	0.03
Role clarity	−0.04	0.02	−0.02	0.02	−0.01	0.04	0.08*	0.04
Role conflict	−0.01	0.02	−0.01	0.01	0.04	0.04	−0.04	0.02
Role overload	−0.01	0.02	−0.01	0.01	0.01	0.05	−0.01	0.03
Structure
Hierarchy of authority	0.02	0.02	0.00	0.01	−0.03	0.04	0.01	0.02
Participation in decision making	0.00	0.04	0.02	0.02	−0.06	0.07	−0.05	0.05
Procedural specification	−0.18**	0.06	−0.02	0.02	−0.01	0.10	0.02	0.05

Note.

^*p < .05.

^**p < .01

^aFunctioning Score coefficients and standard errors are multiplied by 100 for ease in reporting.

Internalizing behavior

Moderation by substance use of organizational effects on youth Internalizing behavior was evident for several climate variables. Consistent with expectations, increases in the Internalizing behavior of substance using youth over time following treatment completion were predicted by higher organizational average levels of Depersonalization, β = 0.09, SE = 0.05, t(6150) = 2.0, p < .05, and therapist perceptions of greater Depersonalization relative to that average, β = 0.07, SE = 0.02, t(6150) = 2.8, p < .01. In contrast, three aspects of climate typically considered to be positive also predicted increases in the Internalizing behavior of substance using youth, compared to non-substance-using youth through post-treatment follow-up. Therapist perceptions of greater Job Satisfaction relative to organizational average levels, β = 0.02, SE = 0.01, t(6150) = 2.3, p < .05, and greater organizational average Personal Accomplishment, β = 0.05, SE = 0.03, t(6150) = 2.0, p < .05, and Role Clarity, β = 0.06, SE = 0.02, t(6150) = 2.5, p < .05 predicted such increases. Also for non-using youth, greater organizational average levels of Role Overload predicted increases in Internalizing behavior through follow-up, β = 0.02, SE = 0.01, t(6150) = 2.0, p < .05.

Youth substance use problems also moderated the effects of one structure variable, Participation in Decision Making, on Internalizing behavior. Therapist perceptions of greater participation in decision making relative to the organizational average predicted decreases through follow-up in the Internalizing behavior of substance using youth, β = −0.02, SE = 0.01, t(6150) = 2.2, p < .05, but not of non-users.

Externalizing behavior

The effects of three climate variables (i.e., Depersonalization, Role Conflict, Growth and Advancement) on youth Externalizing behavior were moderated by youth substance use. The negative effects of Depersonalization on the Internalizing behavior of substance using youth were also observed for Externalizing behavior. Therapist perceptions of greater Depersonalization relative to organizational average levels predicted higher Externalizing scores at post-treatment, β = 0.39, SE = 0.18, t(6150) = 2.2, p < .05, and increases in such problems through follow-up, β = 0.05, SE = 0.02, t(6150) = 2.0, p < .05. Counter to expectations, however, therapist perceptions of greater Role Conflict relative to the organizational average predicted decreases over time in the Externalizing behavior of youth with substance use problems, β = −0.03, SE = 0.01, t (6150) =2.21, p < .05. Finally, the effects of Growth and Advancement on Externalizing behavior accrued only to non-users, with higher organizational average levels predicting increases in Externalizing behavior through post-treatment follow-up, β = 0.03, SE = 0.01, t(6150) = 3.4, p < .01. Youth substance use did not moderate the effects of any organizational structure scales on Externalizing behavior.

Functioning problems

Substance use moderated the effects on youth Functioning problems of four climate variables, Emotional Exhaustion, Job Satisfaction, Role Clarity, and Growth and Advancement. The effects of Emotional Exhaustion and Job Satisfaction accrued only to youth without substance use problems, and in counter-intuitive ways. For non-users, therapist perceptions of greater Emotional Exhaustion relative to organizational averages predicted lower levels of Functioning problems through follow-up, β = −0. 02, SE = 0.01, t(6598) = 2.1, p < .05; and, higher organizational average levels of Job Satisfaction predicted increases in the Functioning problems through follow-up, β = 0.03, SE = 0.01, t(6598) = 2.0, p < .05. Finally and counter to expectations, therapist perceptions of greater Role Clarity relative to the organizational average level predicted increases in the Functioning problems over time of substance using youth, β = 0.08, SE = 0.04, t(6598) = 2.2, p < .05.

Relations between Growth and Advancement and youth Functioning problems through post-treatment follow-up were similar for substance using and non-using youth. Relative to non-using youth, therapist perceptions of greater levels of Growth and Advancement predicted increases in the Functioning problems over time of substance using youth, β = 0.07, SE = 0.03, t(6598) = 2.2, p < .05; and greater organizational average levels of Growth and Advancement predicted such increases among non-using youth, β = 0.06, SE = 0.01, t(6598) = 3.7, p < .01. Immediately post-treatment, however, Growth and Advancement predicted lower levels of Functioning problems among non-using youth, β = −0.30, SE = 0.13, t(42) = 2.3, p < .05.

Substance use moderated the effects of all three organizational structure variables on youth Functioning problems. For substance using youth, higher organizational average levels of Participation in Decision making predicted higher levels of Functioning problems post-treatment, β = 0.1.03, SE = 0.52, t(6579) = 2.0, p < .05. In contrast, Hierarchy of Authority and Procedural Specification related only to changes in the Functioning problems of non-using youth. For such youth, therapist perceptions that the Hierarchy of Authority was lower relative to the organizational average predicted fewer post-treatment functioning problems, β = −0.17, SE = 0.07, t(6579) = 2.4, p < .05. And, higher organizational average levels of Procedural Specification predicted greater decreases in the Functioning problems over time, β = −0.18, SE = 0.06, t(6568) = 3.1, p < .01.

Discussion

The current investigation is the first to our knowledge to examine the effects of organizational variables on treatment outcomes for delinquent youth with and without co-occurring substance use. Notably, this investigation focused on an evidence-based treatment implemented in usual care settings, and included longer-term youth outcomes (i.e., one-year post-treatment). Because greater challenges may be associated with treating youth whose serious antisocial behavior co-occurs with substance use, greater levels of organizational support may be needed to achieve positive outcomes for such youth. It was thus hypothesized that youth substance use would moderate the effects of organizational climate and structure on changes in the behavioral and functioning problems of youth treated with MST.

Did Organizational Climate and Structure Affect Youth Behavior and Functioning Problems?

No relations were found between organizational climate and structure and the Internalizing behavior in this sample of delinquent youth treated with MST. Two of ten climate variables (i.e., Growth and Advancement and Job Satisfaction) predicted increases in the Externalizing behavior or Functioning problems of these youth, as did the structure variable, Participation in Decision Making. Thus, two ostensibly positive aspects of climate and structure appear related to somewhat poorer outcomes over time of an evidence-based treatment for delinquent youth. Potential explanations for these findings include a possible mismatch of criteria used by organizations to foster therapist advancement with those required to support adequate implementation of MST, and that greater organizational focus on job satisfaction may similarly undercut such implementation. Partial support for these hypotheses lies in findings from analyses of relations among Growth and Advancement, and Job Satisfaction, therapist adherence to MST and short-term youth outcomes (Schoenwald, Sheidow, Letourneau, & Liao, 2003). To summarize, higher levels of Growth and Advancement did not relate to youth behavioral or functional outcomes when therapist adherence was high, but predicted poorer outcomes when therapist adherence was low, suggesting organizational criteria for growth and advancement may not have included a focus on therapist adherence and youth outcomes.

In the current evaluation, greater decreases in youth functioning problems were observed in organizations with climates characterized by higher levels of Fairness and structures characterized by higher average levels of Procedural Specification, an aspect of organizational structure that indexes the extent to which rules and procedures are clearly defined and observed. One interpretation of this finding emanates from theory and research on innovation implementation in organizations, which suggests re-alignment of rules and procedures is needed to support the adequate implementation of a new technology (Fixsen et al., 2005; Klein & Knight, 2005).

Did Organizational Climate and Structure Effects Differ for Youth with and without Co-Occurring Substance Use?

The results of moderation analyses suggest select aspects of organizational climate and structure can differentially affect the outcomes of an evidence-based treatment for youth with and without co-occurring substance use, although the magnitude of the differential effect is quite small. Table 4 presents a summary of the significant moderation effects and illustrates that these can be characterized as either countering expectations about the effects of a particular organizational construct on the outcomes of substance using youth; or, as meeting expectations about the effects of a particular organizational construct on these outcomes. For substance using youth, most aspects of organizational climate considered to be positive (i.e., Job Satisfaction, Growth and Advancement, Personal Accomplishment, Role Clarity) predicted increases rather than decreases in youth Internalizing, Externalizing, or Functioning problems at post-treatment and/or over time following treatment completion. One interpretation of these findings is that a potential mismatch exists between organizational strategies to facilitate Growth and Advancement and Job Satisfaction, as suggested earlier, and that this mismatch has particularly negative effects on adequate therapist implementation of MST with delinquent youth who present substance use problems. Thus, it may be particularly important to align organizational incentives and clinical training and support strategies to support adequate implementation of MST when therapists have a case mix that includes some youth with and some without substance use problems (Henggeler, Sheidow, Cunningham, Donohue, & Ford, 2008).

Table 4.

Summary of Organizational Effects on Outcomes that Differed for Youth With and Without Co-Occurring Substance Use

Organizational Scale	CBCL Internalizing (Int)		CBCL Externalizing (Ext)		Functioning Problems (FP)
Climate	Direction Of Association	Expected? Yes/No	Direction of Association	Expected? Yes/No	Direction of Association	Expected? Yes/No
Depersonalization (D)	Users: +	yes	Users: +	yes	ns	ns
Emotional Exhaustion (EE)	ns	ns	ns	ns	Non-users: > EE = < FP	no
Growth and Advancement (GA)	ns	ns	Non-users: +	no	ns	ns
Job Satisfaction (JS)	Users: +	no	ns	ns	Non-users: +	no
Personal Accomplishment (PA)	Users: +	no	ns	ns	ns	ns
Role Conflict (RC)	ns	ns	Users: > RC = < Ext.	no
Role Clarity (RCl)	Users: +	no	ns	ns	Users: +	no
Role Overload (RO)	Non-users: +	yes	ns	ns	ns	ns
Structure
Participation in Decision Making (PD)	Users: > PD = < Int	yes	ns	ns	+	no
Hierarchy of Authority (HA)	ns	ns	ns	ns	Non-users: −	yes
Procedural and Rule Specification (PRS)	ns	ns	ns	ns	Non-users: −	yes

Note. This narrative summary table does not distinguish between post-treatment and long-term follow-up outcomes, nor does it distinguish between average organizational levels and therapist perceptions of the climate and structure constructs.

One aspect of climate generally considered negative (i.e., Role Conflict) predicted decreases in the Externalizing behavior of substance-using youth. As noted above, Role Clarity, (generally a positive aspect of climate) predicted increases in the Internalizing problems of such youth. The pair of findings that Role Clarity predicted worse, and Role Conflict predicted better outcomes for substance using youth is provocative. A possible explanation of these findings is that high levels of Role Clarity constrained therapist flexibility to implement the MST approach to the assessment and intervention of substance use problems, an approach that does not set substance use treatment aside as a separate treatment component, but weaves assessment and intervention of substance use into the ongoing MST treatment process. The finding that therapist perceptions of greater Role Conflict predicted improvements among substance using youth could be construed as consistent with this hypothesis, if one allows that effective implementation of the MST approach to substance use requires case-by-case modification of previously defined roles (Henggeler et al., 2008; Sheidow & Henggeler, 2008). That therapist perceptions of greater Emotional Exhaustion predicted decreases in the Functioning problems of substance-using youth, might suggest greater effort is being expended for these cases.

Some findings from the moderation analyses were consistent with expectations. For example, Greater Depersonalization predicted increases in the Internalizing and Externalizing problems of substance using youth. It is not surprising that therapists who feel somewhat detached from their work, and operate in an organizational climate similarly characterized, are less effective in treating delinquent youth experiencing the added complications of substance use. Also consistent with expectations, substance using youth experienced greater reductions in Internalizing problems following treatment completion when they were treated in organizations with higher average levels of Participation in Decision Making.

Clinical and Research Implications

There are several implications of the findings presented here. First, even among organizations that were early adopters of standard MST for delinquent youth, a case mix of youth with and without substance use problems was served, as occurs not infrequently in community-based mental health service organizations that treat children and adolescents (Schoenwald, Chapman et al., 2008). This finding indicates service organizations and stakeholders importing one or more evidence-based treatments for adolescent problems other than substance use may well use those treatments to treat youth with co-occurring substance use. Accordingly, testing in effectiveness trials or benchmarking studies the effects of evidence-based treatments for those mental health problems that frequently co-occur with substance use could speed the availability of effective treatments to youth with co-occurring problems treated across the mental health and substance abuse service sectors. Second, it appears select aspects of organizational climate and structure affected the longer-term behavioral and functional outcomes of MST somewhat differently, although only slightly so, for delinquent youth with and without substance use. Other aspects of climate and structure had either no effect, or the same effect, on youth with and without substance use problems. These findings suggest organizational effects on the outcomes of comprehensive evidence-based treatments such as MST may be (a) limited, and (b) more similar than different across youth with and without co-occurring substance use problems. Thus, MST and other evidence-based treatments being transported to community settings would seem ready platforms from which to attempt treatment with of youth with co-occurring disorders. That said, youth substance use did moderate a few organizational effects on youth outcomes, albeit slightly, and the pathways through which these effects accrued are as yet unknown. In addition, because research evaluating organizational effects on the implementation and outcomes of other evidence-based treatments for youth is in its infancy, the extent to which the current pattern of findings is unique to MST is unknown. As noted in the Introduction, however, previous findings provide little support for the hypothesis that organizational climate and structure affects youth outcomes through their effects on therapist implementation of MST (Schoenwald et al., 2008; Schoenwald et al., 2009). Third, the extent to which role-related issues affect the implementation and outcomes of MST with youth characterized by both serious antisocial behavior and substance use suggests greater attention may be needed in the training and ongoing clinical and organizational support provided to therapists and programs whose case mix is characterized by some youth with, and some without, co-occurring substance use. This also may be the case for successful deployment and implementation across both the substance abuse and mental health services sectors of other evidence-based treatments for youth with co-occurring substance use.

Limitations

Several limitations of the current evaluation suggest caution is warranted in generalizing results. First, the indicator of youth substance use problems was not informed by youth reports, owing to confidentiality concerns (it was not possible to ensure a caregiver would not overhear youth responses to a telephone interview). As previously described, a multi-source method was used to identify youth with substance use problems. Second, limited descriptive data about provider organizations were obtained at the time of the study, and organizational covariates were therefore not included in models. Other youth and therapist factors that might affect youth outcomes were not included directly, although youth age and gender are accounted for in the standardized CBCL scores, and other covariates were partially accounted for by the random terms included in the model. Third, the small coefficients that characterized the significant findings suggest caution is warranted in making inferences about the differential effects of organizational context on the outcomes of MST for delinquent youth with and without co-occurring substance use problems.

The remaining limitations pertain to the use of a statistical modeling approach that examined both therapist and organization effects on youth outcomes. This approach contrasts with models that capture either the individual level response, or aggregate individual responses to reflect an organizational score on the basis of statistical indicators of the appropriateness of such aggregation (see, e.g., Aarons & Savitsky, 2006; Glisson & Hemmelgarn, 1998; Morris & Bloom, 2002). As noted in the Measures section, the current sample was characterized by both considerable interrater agreement and noteworthy therapist variance on organizational scales. In addition, the substantive questions of the investigation were best served by modeling both therapist and organizational level data effects, although not by modeling the contextual effect (the difference between the organizational and individual level effects; Raudenbush & Bryk, 2002). In addition, rather than aggregating across subscales within climate, or structure, separate models examined relations among each of the distinct climate and structure subscales, and the outcomes of youth with and without substance use problems. The decision to model the effect of each scale on youth outcomes reflected our substantive research questions and the first time use of the organizational climate and structure measures with a sample of therapists and mental health organizations implementing an evidence-based treatment. The substantive question driving the investigation was: Which, if any, specific aspects of climate and structure would differentially impact the outcomes of MST for youth with and without substance use problems? We recognize, however, the possible inflation of Type 1 error is a limitation of this modeling strategy, and are cautious in our interpretations of significant organizational findings.

Conclusion

The current study is the first to our knowledge to examine the effects of organizational context variables on the outcomes of an evidence-based treatment not designed explicitly for substance using youth, but delivered to such youth outside of the substance abuse services sector. The findings showed some aspects of organizational climate and structure do not affect youth outcomes, some affect the outcomes of youth with and without co-occurring substance use similarly, and a few differentially affect the outcomes of youth with and without co-occurring substance use. The findings suggest evidence-based treatments already transported to community settings may provide a logical platform for extending care to youth with co-occurring substance use disorders to evaluate their effects on such youth. Further research is needed to test the boundary conditions of the effectiveness of extant evidence-based treatments with substance using youth; and to identify the interplay of specific aspects of the organizational context and the implementation and outcomes of evidence-based treatments for youth, both those designed specifically to treat substance use, and those designed to treat problems frequently co-occurring with substance use.