Long-Term Youth Criminal Outcomes in MST Transport: The Impact of Therapist Adherence and Organizational Climate and Structure

Abstract

The current study investigated relations among therapist adherence to an evidence-based treatment for youth with serious antisocial behavior (i.e., Multisystemic Therapy), organizational climate and structure, and youth criminal charges on average 4 years post-treatment. Participants were 1979 youth and families treated by 429 therapists across 45 provider organizations. Results showed therapist adherence predicted significantly lower rates of youth criminal charges independently and in the presence of organizational variables. Therapist perceptions of job satisfaction and opportunities for growth and advancement relative to the organizational average predicted youth criminal charges, as did organizational average levels of participation in decision-making. These associations washed out in the presence of adherence, despite the fact that job satisfaction and growth and advancement were associated with adherence.

Despite the increasing availability and demand for well-validated interventions, it is estimated that 90% of public systems do not deliver treatments or services that are evidence-based (Rones & Hoagwood, 2000). Accordingly, research on factors associated with the implementation and outcomes of evidence-based treatments in community settings is becoming a public health priority (National Institute of Mental Health [NIMH], 2006). Until recently, studying the factors affecting implementation and outcomes in usual care settings of treatments developed in university settings was not possible, in part due to limitations in sample sizes of therapists and service provider organizations attempting their use. The transport of evidence-based treatments for youth with serious antisocial behavior, in particular, has now been underway for just over a decade (e.g., Chamberlain, 2003; Schoenwald & Henggeler, 2003; Sexton & Alexander, 2002), and has sufficient use in usual care settings to facilitate adequately powered evaluation of such factors.

As suggested in recent reviews, challenges to the transport and implementation of evidence-based psychosocial treatments are likely to arise at multiple levels of the practice context, including the client population, clinician, provider organization, and service system (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005; Schoenwald & Hoagwood, 2001; Southam-Gerow, Marder, & Austin, in press; Stirman, Crits-Christoph, & DeRubeis, 2004), and in the dynamic interplay of variables within and across these levels of the practice context (Schoenwald, Kelleher, Weisz, and the Research Network on Youth Mental Health, 2008). The findings reported here pertain to linkages among youth outcomes and a subset of practice context and implementation factors examined in a prospective, 45-site, NIMH-funded study of the transportability of Multisystemic Therapy (MST; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 1998) to usual care settings. As illustrated in Figure 1, this report focuses on relations found among select attributes of service provider organizations, therapist adherence, and the long-term post-treatment criminal activity of youth treated with MST. Prior publications reported findings regarding youth behavioral and functional outcomes and relations among organizational attributes, therapist adherence, and such outcomes (Schoenwald, Sheidow, Letourneau, & Liao, 2003; Schoenwald, Sheidow, & Letourneau, 2004).

Figure 1.

Subset of MST Transportability Study Variables Examined in Current Report

Relationships among organizational factors, treatment implementation, and clinical outcomes in community-based organizations deploying evidence-based treatments are potentially important for two additional reasons. First, psychosocial treatments are ”soft technologies.” Relative to “hard technologies” such as a computer chip or medical equipment, soft technologies such as teaching methods, diagnostic screening interviews, and psychosocial treatments are somewhat indeterminate and variable (Glisson, 1992). These qualities may render psychosocial treatments particularly vulnerable to adaptations when transported to community-based settings. Such adaptations, however, may compromise the treatment’s effectiveness in obtaining the clinical outcomes achieved in controlled research settings (Brown, 1995, 2000; Glisson, 1992).

Second, organizations may attempt antidotes to the indeterminacy of human services. That is, organizations may try to bring clarity to loosely specified services by emphasizing rules, conformity, and subservience to organizational procedures and authority. Such attempts may be “a misguided effort to inject certainty into what is an inherently uncertain technology” (Glisson, 2002, p. 237). For example, an emphasis on organizational conformity and rules may be a poor match for the multifaceted and inherently flexible approaches deemed to be effective for complex adolescent problems. Thus, a mismatch between the demand characteristics of a well-specified treatment and the focus of an organization on rules, conformity, and subservience could engender a negative workplace climate and compromise implementation of the treatment. Conversely, organizations that retain the flexibility required to manage complex and somewhat variable tasks may better support a positive workplace climate and effective implementation of such technologies. Indeed, evidence linking positive organizational climate with service quality and reduced youth behavior problems for child welfare (Glisson & Hemmelgarn, 1998) and with increased access to mental health services for youth in child welfare and juvenile justice systems (Glisson & Green, 2006) are consistent with this proposition.

It is not known, however, whether organizational effects on services and outcomes found in usual care in other service sectors (e.g., child welfare) will extend to evidence-based mental health treatments. Potential differences between the services rendered in usual care and evidence-based treatments could amplify, attenuate, or neutralize organizational effects on services and outcomes. Among potential differences are that psychosocial treatments validated in randomized trials are more clearly specified than treatments typically deployed in community mental health settings and require specialized training and ongoing clinical support (see, e.g., Weisz, Donenberg, Han, & Kauneckis, 1995; Weisz, Donenberg, Han, & Weiss, 1995). In addition, differences in the mandate and mission of child welfare and mental health sectors could be expected to generate differences in services they provide and service provider organizations providing them (Chen, 1990).

In the mental health sector, the examination of organizational context in outpatient service organizations providing is just beginning. A recent study of public mental health organizations serving adults with persistent mental illness found mixed effects with respect to relations between organizational culture and climate and consumer outcomes indices (Morris, Bloom, & Kang, 2007). A study of climate and culture in public mental health organizations serving children found relations among culture, climate, and work attitudes; but did not examine youth outcomes (Aarons & Sawitzky, 2006). Neither of these studies focused on organizations and clinicians implementing evidence-based treatments, nor on the mix of private and public service provider organizations that appear to characterize community mental health services for children nationally (Schoenwald, Chapman et al., 2008). Thus, the interplay of (a) organizational characteristics, (b) implementation of empirically supported mental health treatments, and (c) clinical outcomes is largely uncharted territory.

The MST Transportability Study provided an opportunity to chart some of that territory using a prospective descriptive design. In the original study design funded by the National Institute of Mental Health, measures of youth symptoms and functioning were obtained upon intake for treatment, immediately post-treatment, and at six- and 12-month post-treatment follow-up. In addition, caregiver reports of therapist adherence were obtained monthly and therapist reports of organizational climate and structure were obtained at baseline and every six months during the 2.4-year clinical implementation period. Data on youth criminal activity through 12-month post-treatment follow-up were obtained from archival sources as part of the original study, with collection and analyses of criminal data extending the follow-up through approximately 4-years post-treatment made possible by a subsequently awarded National Institute of Drug Abuse grant. The current manuscript is the first to report results pertaining to the criminal activity of youth in the study.

The original mediation model guiding the Transportability Study hypothesized replication in usual care settings of relations observed among caregiver-reported therapist adherence and youth behavioral and criminal outcomes demonstrated in randomized trials of MST with juvenile offenders (see, e.g., Huey, Henggeler, Brondino, & Pickrel, 2000; Schoenwald, Henggeler, Brondino, & Rowland, 2000). Early results of the study supported hypotheses regarding positive associations between therapist adherence, successful discharge, and short-term behavioral outcomes (Schoenwald et al., 2003); and expert consultation, therapist adherence, and such outcomes (Schoenwald et al., 2004). The significant relations between adherence and behavioral outcomes extended through the one-year post-treatment follow-up (Schoenwald, Carter, Chapman, & Sheidow, 2008). As expected, few therapist, youth, and caregiver variables affected therapist adherence (Schoenwald, Letourneau, & Halliday-Boykins, 2005). However, hypothesized mediation of the effects on youth behavioral outcomes of organizational climate and structure by therapist adherence was not found. Instead, select organizational climate (e.g., opportunities for growth and advancement) and structure (e.g., hierarchy of authority, participation in decision making) scales exerted direct effects on short-term youth outcomes, and some of these effects were moderated by adherence (Schoenwald et al., 2003). For example, when therapist adherence was high, there was no association between organizational growth and advancement and improvement in youth behavior problems post-treatment; however, when adherence was low, higher levels of growth and advancement predicted less youth behavioral improvement. Subsequently, models evaluating relations among organizational climate and structure, therapist adherence, and longer-term youth behavioral outcomes showed one climate variable - growth and advancement – and two structure variables - participation in decision-making, and hierarchy of authority -- predicted changes in youth behavior at one-year post-treatment follow-up. Despite the absence of formal mediation, the results suggested some interplay among these variables, adherence, and such longer-term outcomes evidenced in changes of parameter estimates (Schoenwald, Carter, et al., 2008).

The current report is the first to extend evaluation of the effects of organizational climate and structure, and of therapist adherence, to long-term (4 years post-treatment) youth outcomes with significant familial, social, and economic costs, namely criminal behavior. Indeed, it was the promise of improvement in these outcomes – reductions in criminal recidivism and its economic costs at 2.4 and 4-year post-treatment follow-ups – as demonstrated in randomized clinical trials of MST with juvenile offenders that prompted service system demand for MST transport (see, e.g., Henggeler & Schoenwald, 1999; Schoenwald, 2008). And, although research has not yet demonstrated the effects of organizational climate and structure on outcomes of parallel import in the child welfare sector (e.g., out-of-home placement; incidence of abuse or neglect) findings do link organizational climate with access to mental health services for youth (Glisson & Green, 2006) and with behavioral improvements of children served by child welfare case management teams (Glisson & Hemmelgarn, 1998). The current paper examines the role of climate and structure in the longer-term and high stakes outcomes of an evidence-based treatment, MST, for delinquent youth.

Method

Participants

Youth and caregivers

A total of 1979 youth and caregivers participated in the MST Transportability Study. The mean age for youth was 14.0 (SD = 2.35), and most were male (65.0%) and Caucasian (59.5%), with 19.3% of youth identified as African American, 6.4% Asian or Pacific Islander, and 14.8% other. The majority of the youth were not of Hispanic ethnicity (92.7%). Almost half of youth resided with their mother or mother and a significant other (48.6%), with the majority of remaining youth living with at least one parent figure. The primary referral sources to treatment for youth were juvenile justice or corrections agencies (44.2%), social services (23.0%), mental health agencies (17.6%), or other agencies (15.1%). The most frequent referral reasons (multiple reasons could be endorsed for a given youth) included status offenses (47.4%), criminal offenses (46.7%), substance use problems (31.3%), and school suspensions or expulsions (29.8%).

Therapists

Primary therapists (n = 429) were identifiable for 1,888 of the 1,979 families in the study. “Primary therapist” signifies the therapist treating the family for the entire treatment episode or, for families treated by more than one therapist, the therapist providing treatment for the majority of the family’s treatment episode. A primary therapist could not be identified for 91 families, each of which was treated by more than one therapist for approximately equal lengths of time. There were no significant differences between families with and without a primary therapist (Schoenwald, Chapman, & Sheidow, 2006). The majority of therapists were female (74%) and most held master's degrees (61%) with 32.11% holding a bachelor’s degree, 3.05% holding a doctoral degree, and 3.05% with an “Other” or unspecified degree. Therapists treated an average of 4.4 families (1–22; Mdn = 3.0, SD = 3.6).

Provider organizations

There was an average of 9.5 therapists per provider organization (1–40; Mdn = 7.0, SD = 7.9) and an average of 42.0 youth per provider organization (3–108; Mdn = 34.0, SD = 27.4). The vast majority (93.7%) of the organizations were private providers contracted by public agencies to operate MST programs.

Procedures

Study procedures have been detailed previously (see Schoenwald et al., 2003; Schoenwald et al., 2004) and are briefly described here.

Youth and families

All youth referred for MST treatment at the study sites were eligible for the Transportability Study except youth with autism or severe mental retardation. Families were recruited for study participation by clinical supervisors or therapists at the provider organizations upon referral, prior to treatment, and the consent rate was 82%. Individuals obtaining informed consent from caregivers and assent from youth were trained in the informed consent procedures by the research staff. As part of the informed consent process, caregivers completed Release of Information forms granting the research staff permission to obtain youth records from courts and juvenile justice agencies. Research assistants administered pre, post, and follow-up assessment measures to youth, caregivers, and therapists by telephone, and caregivers were reimbursed for completed assessments. Participation in the study was voluntary and the Institutional Review Board of the Medical University of South Carolina approved all procedures.

Therapists

All therapists in the MST programs were eligible to participate and all but two of them consented to do so. Informed consent from therapists was obtained during site visits; for therapists employed after the study began, a telephone procedure was used to obtain informed consent. Therapist demographic, educational, and professional experience data were obtained upon enrollment in the study, as was the baseline organizational assessment. Therapists completed the organizational assessment semi-annually during the clinical implementation portion of the study.

Clinical intervention

Details of the clinical intervention and multi-component implementation protocol have been described elsewhere (Henggeler et al., 1998; Henggeler & Schoenwald, 1999). To summarize, MST is an intensive, family-based treatment originally developed for delinquent youths at imminent risk of incarceration or other out-of-home placements and their families that specifically targets those factors in each youth’s social ecology (family, peers, school, neighborhood, and community) contributing to his or her antisocial behavior. MST treatment is informed by the social ecological theory of human behavior articulated by Bronfenbrenner (1979) and by prospective research identifying the multiple predictors of serious antisocial and related behavior in adolescents. Given the youths’ imminent risk of placement, overarching treatment goals often relate to keeping the youth in the home and reducing criminal behavior. Specific goals and the interventions to achieve them are designed collaboratively with the youth’s caregivers, who also implement the majority of the interventions, initially with the instrumental and social support of the therapist.

The combination of intervention techniques applied and the expected impact of intervention procedures vary in accordance with the circumstances of each youth and family. Thus, step-by-step or session-by-session guides are not used to implement MST. Instead, to balance adequate specification of the model with responsiveness to the needs and strengths of each youth and family, nine principles are used to guide the MST assessment and intervention process. Ongoing assessment, and intervention proceeds in accordance with an analytic process that encourages clinicians to generate specific hypotheses about the combination of factors that sustain a particular problem behavior, provide evidence to support the hypotheses, test the hypotheses by intervening, assess the impact of the intervention, and begin the assessment process again. Interventions typically include improving specific caregiver discipline practices, enhancing family affective relations, decreasing youth association with deviant peers, increasing youth association with prosocial peers and activities, improving youth school or vocational performance, and developing an indigenous support network of extended family, neighbors, and friends to help caregivers achieve and maintain such changes. Specific treatment techniques used to facilitate these gains are integrated from those therapies that have the most empirical support, including cognitive behavioral, behavioral, and the pragmatic family therapies.

A home-based model of service delivery is used to provide comprehensive and intensive clinical interventions when and where they are needed (i.e., clinicians are available 24 hours/day, 7 days/week to respond to crises), with duration and frequency of treatment sessions varying in accordance with changing circumstances, needs, and treatment progress. MST therapists operate in teams of no fewer than two and no more than four therapists (plus the clinical supervisor). Each therapist’s caseload ranges between four to six families so that therapists are able to provide sufficiently intensive and individualized services to families. The length of treatment in clinical trials with juvenile offenders has ranged from an average of 13 – 17.5 weeks. Average length of treatment for the current sample was 21.9 weeks (SD = 10.1).

Implementation protocol

As described in detail elsewhere (Henggeler & Schoenwald, 1999; Schoenwald, 2008) a multi-component approach designed to replicate procedures and resources provided therapists in randomized trials is used to help therapists and clinical supervisors in community settings implement MST. Therapists are provided with multiple layers of clinical and programmatic support and ongoing feedback from several sources. The quality assurance system for MST programs is provided by a university-licensed organization, MST Services. The system includes an intensive 5-day orientation to MST theory and practice for clinical staff; quarterly booster sessions for clinical staff; at least weekly group supervision of therapists by an on-site clinical supervisor trained in the MST supervisory protocol (Henggeler & Schoenwald, 1998); weekly group (i.e., supervisor and therapists) phone consultation with an MST expert who follows a specified consultation protocol (Schoenwald, 1998); and feedback from measures of therapist adherence (Henggeler, Borduin, Schoenwald, Huey, & Chapman, 2006), supervisor adherence (Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002), and consultant adherence (Schoenwald et al., 2004). Ongoing organizational consultation and semi-annual formal program reviews are provided to assess and address organizational, service system, and purveyor (i.e. MST expert-related) barriers to achieving integrity of treatment implementation and favorable youth outcomes.

Measures

Criminal charges

Criminal charge data were obtained for 1,791 (91% of the entire sample) youth, across a mean post-treatment follow-up period of 49.3 months (SD = 8.9), with a range of 24.7 to 68.2 months. Of these youth, 1,713 (96%) had an identifiable primary therapist. For all youth participants, criminal charge data were obtained from county and state juvenile justice agencies and courts. For youth participants who had reached adulthood at the time of the follow-up request, adult charge data were obtained via public record searches available through the Internet, or from agencies housing adult criminal records. Raw data were obtained on the dates, types, and severity of lifetime pre-treatment charges and charges accrued throughout the follow-up period. These data were coded by research staff to reflect charge types (i.e., person, property, drug, public order, status or other offense) and charge severity levels (e.g., murder was rated as the most severe, other types of person offenses were rated as next most severe; status offenses were rated least severe, and within status offenses “incorrigible/ungovernable behavior” was rated the least severe). The coding scheme was based on coding systems used in early studies of MST (Hanson, Henggeler, Haefele, & Rodick, 1984); these systems, in turn, were based on the Uniform Crime Reports standards used by the Federal Bureau of Investigation. An ongoing study using the same coding scheme documents 98.6% agreement across blind raters on individual charges and inter-rater agreement at 96.4% (Letourneau, 2006).

Of the 1,713 youth for whom criminal justice records were obtained and for whom a primary therapist was identifiable, 1,267 (71%) had at least one known charge (which could have occurred pre-, during-, or post-treatment), and of these, 978 (77%) had at least one charge during the follow-up period. Information on criminal charges could not be obtained for 188 (9.0%) of the 1,979 participants in the entire sample. Most of these participants (n = 178) were treated in jurisdictions that ultimately were unable to provide any juvenile justice data, despite initial agreements to do so.

Therapist adherence

Therapist adherence was assessed monthly during treatment using caregiver reports on the MST Therapist Adherence Measure – Revised (TAM-R; Henggeler et al., 2006). The TAM-R is a 28-item scale developed by expert consensus to assess therapist adherence to the nine principles of MST. For example, MST Principle 1 states, "the primary purpose of assessment is to understand the fit between the identified problems and their broader systemic context" (Henggeler et al., 1998, p. 23). A corresponding item on the TAM reads “the therapist tried to understand how the family’s problems all fit together" (Henggeler et al., 1998, p. 23). Similarly, MST Principle 4 states “interventions should be present-focused and action-oriented, targeting specific and well-defined problems,” and corresponding TAM items read “the therapist recommended that family members do specific things to solve their problems,” and “the family knew exactly which problems were being worked on.” The 28-item TAM-R retains 19 of the 26 items of the original MST Therapist Adherence Measure (TAM; Henggeler & Borduin, 1992) validated in two randomized clinical trials of MST with juvenile offenders (Henggeler Melton, Brondino, Scherer, & Hanley, 1997; Henggeler, Pickrel, & Brondino, 1999). In these trials, the TAM predicted reductions in youth arrests, days incarcerated, soft drug use, aggression, and other antisocial behavior problems as well as improvements in family functioning (Huey, et al., 2000; Schoenwald, et al., 2000). Although caregiver, therapist, and youth reports on the measure were obtained in these trials, caregiver reports were the better predictors of youth outcomes.

Confirmatory factor analyses from the MST Transportability Study supported a single-factor solution that predicted youth behavioral outcomes (Schoenwald et al., 2003). In addition, therapist ratings of the adherence of MST experts’ adherence to the consultation protocol predicted caregiver ratings of therapist adherence on this single factor scale and youth outcomes (Schoenwald et al., 2004). Importantly, no associations have been found between caregiver therapist adherence ratings and the severity of youth problems, caregiver and youth demographic characteristics, and therapist demographic or professional experience characteristics, although gender and ethnic similarity in therapist-caregiver dyads have been associated with higher adherence ratings (Halliday-Boykins, Schoenwald, & Letourneau, 2005) but not with differential criminal outcomes (Schoenwald, Chapman, & Halliday-Boykins, 2008). The discriminant validity of the scale is supported by findings from a clinical trial testing an adaptation of MST for substance-abusing juvenile offenders in which caregiver ratings of therapist adherence to MST treatment differed from control treatments (Henggeler, Halliday-Boykins, Cunningham, Randall, Shapiro, & Chapman, 2006).

The Transportability Study also included 12 new items that indexed whether treatment sessions focused on important aspects of the youths’ school, peer, and neighborhood/social support systems, consistent with the MST model. A comprehensive evaluation of the original 26 items and the 12 new items conducted using a Rasch-based approach to scale development retained 19 of the original TAM items and 9 of the new items onto a single factor and a two- rather than five-point rating scale. Approximately 2% and 3% of the variance in the scores was attributable to the primary therapist and provider organization, respectively, with the remaining 91% being attributable to the family and the TAM-R predicted one-year post-treatment behavior problem reductions in youth (Schoenwald, Carter, et al., 2008) and was predicted by therapist ratings of supervisor adherence (Schoenwald, 2008).

Organizational climate

Organizational climate was assessed as each organization began the study (baseline) and semi-annually during the treatment portion of the study, and for therapists and supervisors hired by the organization after the study began, at each clinician’s baseline and in accordance with the organization’s subsequent semi-annual assessment date. Climate was assessed using reports on ten well-known scales from the Psychological Climate Questionnaire originally assembled by James and Sells (1981), and used by Glisson and colleagues in research on child welfare and juvenile justice systems (Glisson & Hemmelgarn, 1998; Glisson & James, 2002). Individual perceptions of the impact of the work environment on one’s own well-being and work comprise the construct known as psychological climate; when these perceptions are shared by individuals within a work unit, they are typically aggregated to index the construct known as organizational climate (for reviews pertaining to the conceptualization, measurement, and statistical modeling of organizational constructs including climate see Chan, 1998; Glisson & James, 2002; Klein & Kozlowski, 2000). Included in the MST Transportability Study, these ten scales were: fairness, role clarity, role overload, role conflict, cooperation, growth and advancement, job satisfaction, emotional exhaustion, personal accomplishment, and depersonalization. Item responses on a 5-point Likert-type response scale vary by instrument scale and include a range from “Strongly disagree” to “Strongly agree;” or from “Practically never” to “Almost always.” In the current study, Cronbach’s alpha reliabilities at the first administration in this sample ranged from 0.62 (fairness) to 0.92 (job satisfaction) with only the fairness construct having reliability less than 0.70.

As noted previously, therapists reported on organizational climate and structure at baseline and semi-annually, with a mean of 2.4 reports (range 1 – 7) per respondent. For each climate scale, an individual’s reports were averaged across administrations. Interrater agreement as indexed using r _WG (James, Demaree, & Wolf, 1993; LeBreton, James, & Lindell, 2005) ranged from medium to high (r _WG 59 – .78).for all but one scale. Evaluation of the intraclass correlation coefficients (ICCs) indicated there was generally a non-ignorable proportion of variance in therapist ratings attributable to the provider organization and that a substantial portion of the variance was also attributable to the therapist providing report. Specifically, for Emotional Exhaustion, Growth and Advancement, Job Satisfaction, and Role Conflict, the percentage of variance attributable to the provider organization was 6%, 36%, 10%, and 4%, respectively, with 94%, 64%, 90%, and 96% of the variance, respectively, attributable to the therapist providing the report.

Organizational structure

Organizational structure, specifically the degree of formalization (explicit rules and procedures governing employee behavior) and centralization (degree to which authority and decision-making are concentrated) were assessed simultaneously with organizational climate using therapist reports on three brief scales frequently administered together as a single instrument. The scales are: (1) Participation in Decision-making (8 items; from Hage & Aiken, 1967); (2) Hierarchy of Authority (4 items; from Hall, 1963); and (3) Procedural and Rule Specification (3 items; from Hall, 1963). Response options were on 5-point rating scales. In the current sample, Cronbach’s alphas for the three structure scores were 0.75 (procedural specification), 0.86 (hierarchy of authority), and 0.89 (decision making). An individual’s reports were averaged across administrations. As with the climate data, therapist reports on the structure scales were averaged across administrations. For Participation in Decision Making, Hierarchy of Authority, and Procedural and Rule Specifications, the percentage of variance attributable to the provider organization was 14%, 8%, and 5%, respectively, with 86%, 92%, and 95% of the variance, respectively, attributable to the therapist providing the report.

Data Analysis

Distribution of outcome variable

The outcome variable was the count of charges per youth incurred during the period between the end of treatment and retrieval date of each youth’s charge record. Inspection of the charge data revealed a considerably non-linear distribution. Specifically, the mean number of post-treatment charges per youth was 3.92 (SD = 6.35), with 43%, 11%, 7%, and 6% of the sample having 0, 1, 2, or 3 charges, respectively, and the remaining 33% of the sample having between 4 and 65 charges. This distribution of charge counts most closely resembles a Poisson distribution, necessitating the use of a Poisson regression model. Two other characteristics of the outcome distribution have implications for the data analysis strategy. There are two conceptually distinct types of zero values for the outcome (Hedeker & Gibbons, 2006). Specifically, for youths with no post-treatment charges, the distinction is between those who have no pre-treatment legal involvement and those who have pre-treatment legal involvement. The meaning of zero post-treatment charges for each of these groups is potentially quite distinct. Notably, MST was developed and validated (through multiple randomized clinical trials) for use with chronic and violent offenders, and youth in the trials had been arrested at the time of study recruitment, were at imminent risk of juvenile justice placement, and had a history of juvenile justice involvement. During the initial transport of MST, some community-based provider organizations “widened the net” of youth eligible for services to include those without a history of legal involvement. The resulting sample thus included some youth with no pre-treatment charges. To provide a sample comparable to existing MST clinical trial data, the 642 of 1,713 (37%) youth with an identifiable primary therapist with zero lifetime pre-treatment charges were excluded from the current analyses. As expected, the youth with zero lifetime pre-treatment charges were significantly younger (p < .001), by about 1 year on average and were significantly more likely to be Caucasian (p < .004) relative to youth with pre-treatment charges. There were no gender differences between groups. Finally, the distribution of post-treatment charges was influenced by each youth’s “time at risk” for being charged (i.e., the length of time between the youth’s end of treatment and the retrieval of the charge records). Rather than imposing a fixed length of exposure across all youth, thereby losing available data, the statistical model was adjusted for the amount of time each youth was at risk of being charged (i.e., variable exposure; M = 4.15 years; SD = 0.57, adjusting for variance due to therapists and provider organizations). As a result of the variable exposure term, the interpretation of all results is with respect to the annual rate of post-treatment charges.

Data structure

Two features of the present data structure have implications for the data analysis strategy. First, the data are nested such that i youths (level-1, n_ijk ≈ 1,071) are nested within j therapists (level-2, n_·jk ≈ 304) who are nested within k provider organizations (level-3, n_··k ≈ 40), implying a three-level random-effects regression model where variance in the rate of post-treatment charges is partitioned among youths, therapists, and provider organizations.

Second, there were constructs of interest at each of the three levels of nesting. Specifically, there are youth-level covariates (i.e., age, gender, ethnicity, number of lifetime pre-treatment charges, level of therapist adherence to MST during treatment), therapist-level covariates (i.e., therapist reports of climate), and provider-level covariates (i.e., organizational aggregate climate scores). Of note, it is common in organizational research for individual participants to provide reports that make reference to a common entity (Raudenbush & Bryk; Klein & Kozlowski, 2000). There are multiple statistical methods for evaluating such data (e.g., Chan, 1998), and the analytic strategy must be carefully selected on the basis of the substantive research questions (Enders & Tofighi, 2007; Hofmann & Gavin, 1998; Kreft, de Leeuw, & Aiken, 1995). Given the Transportability Study was the first to assess the climate and structure of organizations providing an evidence-based treatment for youth via reports of therapists implementing that EBT, our interest was in the simultaneous evaluation of the effect of (a) the therapist’s perception of the organizational construct (i.e., the therapist effect) and (b) the provider organization’s level of the construct (i.e., the organization effect). This model is similar in specification and interpretation to the “frog-pond” model described by Hofmann and Gavin and Kreft et al.

Test of indirect effects

The aims of the present investigation imply tests of mediated, indirect, or intervening variable effects. Specifically, the study aimed to examine the independent effects of therapist adherence and organizational climate and structure on the long-term post-treatment criminal activity of youth; and to evaluate the hypothesis that therapist adherence would mediate the effects of organizational climate and structure on such criminal activity. Statistical methods exist for formal tests of mediation in single-level (e.g., Baron & Kenny, 1986; MacKinnon, Lockwood, Hoffman, West, & Sheets, 2002) and two-level nested designs (e.g., Bauer, Preacher, & Gil, 2006; Kenny, Korchmaros, & Bolger, 2003; Krull & MacKinnon, 2001), including robust methods for the computation of the mediated effect and associated confidence intervals (e.g., MacKinnon, Lockwood, & Williams, 2004). Less work, however, has been completed on tests of indirect effects for three-level nested models, generally, and non-linear outcomes and effects for a given construct modeled at different levels, specifically. Because of this constraint, a less formal approach was necessary to examine the indirect effect of organizational climate and structure operating on the youth criminal activity through therapist adherence.

Statistical models

Random effects regression models (RRMs) were conducted using HLM software (version 6.02; Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2004). Four models were performed for each organizational climate and structure scale. The first model tested the therapist and organization effects on the rate of post-treatment charges. This was a three-level variable exposure Poisson RRM with Penalized Quasi-Likelihood (PQL) estimation and a log-link function. The outcome was the number of post-treatment charges per youth, Y_ijk, holding constant the effects of age, gender, ethnicity, and number of lifetime pre-treatment charges, and adjusting for each youth’s length of exposure in years, m_ijk, yielding an annual post-treatment event rate per youth, λ_ijk. Age and number of lifetime pre-treatment charges were grand mean centered, and gender and ethnicity were uncentered. The therapist perception score (i.e., the therapist effect) was computed as the deviation of each therapist’s score from the mean value for the respective provider organization (i.e., group mean centered), and the provider organization average (i.e., the organization effect) was computed as the mean value for all therapists within a provider organization centered around the grand mean provider organization score (i.e., grand mean centered) according to the methods detailed by Kreft et al. (1995).

The second model evaluated the therapist effect and organization effect on therapist adherence. This was a three-level continuous RRM using Restricted Maximum Likelihood estimation, with the average level of therapist adherence experienced by each family (i.e., at level-1) during the treatment episode entered as the outcome. The third model tested the direct effect of therapist adherence on the number of post-treatment youth charges. This model (with the nesting levels, covariates, and the outcome identical to the first model) included the average level of therapist adherence to MST experienced by each family during the treatment episode as a level-1 covariate. The fourth model tested the therapist effect and organization effect on charges in the presence of the putative intervening variable, therapist adherence. This model was identical to the first, with the inclusion of the average level of therapist adherence to MST experienced by each family during the treatment episode at youth-level (level-1) of the model.

Of note, neither additional therapist nor organizational covariates were included in the models. With two exceptions, therapist covariates did not relate to climate and structure ratings in the current analyses. Therapists with a Master’s degree or higher provided lower ratings of Job Satisfaction, and therapists who were older provided lower ratings of Emotional Exhaustion. However, because these effects translated into relatively trivial differences in ratings (2.18 points on a 38-point scale and .06 points per year of age on a 24-point scale, respectively), the modeling strategy detailed below explicitly models individual therapist deviations, and the therapists demographics are not associated with youth outcomes, the demographic variables were omitted from subsequent analyses. Data on organizational variables other than structure, climate, and public/private status (93.7% private) were not obtained in the study, thereby precluding inclusion in the models of organizational covariates.

Utilizing the model building approach described by Snijders and Bosker (1999), random effects were specified according to the likelihood ratio test and theoretical considerations. For non-linear models (i.e., models 1, 3, and 4), population-average results were interpreted rather than less robust unit specific results (Raudenbush & Bryk, 2002; Raudenbush et al., 2004). Additionally, for non-linear models, the event rate ratio, or the increase/decrease in the rate of post-treatment charges per year associated with a given predictor, was computed through the exponentiation of each parameter estimate, according to exp(γ_ijk) (Raudenbush & Bryk, 2002). For all models, due to the relatively small number of level-3 units (n_··k ≈ 40), asymptotic standard errors, rather than robust standard errors, were used for the computation of the T-ratio test statistic (Maas & Hox, 2005).

Results

Preliminary Models

Intraclass correlation coefficients

An unconditional three-level continuous RRM indicated that 89%, 0%, and 11% of the variance in criminal charges was attributable to the youth, therapist, and provider organization, respectively. Despite the lack of outcome variance at level 2 of the model, the therapist random effect was maintained in subsequent analyses because (a) the effects of organizational climate and structure are modeled at both therapist- and provider organization-level and (b) the continuous RRM provides only an approximation of the actual criminal charge variance components given the non-normal distribution of the outcome.

Unconditional and variable exposure models

Based on the fully unconditional three-level Poisson RRM, the average number of post-treatment criminal charges per youth during the follow-up period (i.e., the event rate ratio, ERR) was 5.91, 95% CI_ERR = 4.88 – 7.16. Adjusted for the variable exposure (i.e., length of time at risk for receiving charges), the average number of post-treatment criminal charges per youth per year was 1.43, 95% CI_ERR = 1.18 – 1.74. Given an average length of follow-up of 4.15 years, the yearly rate of 1.43 provides an estimated total number of charges, 5.93, roughly equivalent to the total rate of 5.91 from the fully unconditional model. This model included a random effect for therapist and for provider, allowing the average yearly rate of post-treatment charges to vary between therapists and between providers, respectively. All subsequent linear and non-linear models maintain these random effects and the variable exposure term.

Covariate model and random effects specification

Next, covariates commonly associated with charges (i.e., age, gender, ethnicity, and number of lifetime pre-treatment charges) were entered in the variable exposure RRM. In addition to the random intercepts described above and based on the results of the likelihood ratio tests, random effects were modeled for each covariate, with the exception of age, within providers but not within therapists. Thus, for example, the association of gender with the rate of post-treatment charges was constrained to be identical across the therapists within a given provider but it was allowed to vary from provider to provider. All subsequent non-linear models maintain these random effects.

The results¹ indicated that older youth and females had significantly lower rates of post-treatment charges, ERR = 0.91, 95% CI_ERR = 0.84 – 0.98, and ERR = 0.57, 95% CI_ERR = 0.46 – 0.71, respectively; and African-American youths and youths with more lifetime pre-treatment criminal charges had significantly higher rates of post-treatment charges, ERR = 1.29, 95% CI_ERR = 1.07 – 1.55, and ERR = 1.05, 95% CI_ERR = 1.03 – 1.06, respectively.

The following sections describe the findings from models evaluating therapist and organization effects of organizational constructs on post-treatment charges, therapist and organization effects of organizational constructs on therapist adherence, the direct effect of therapist adherence on post-treatment charges, and therapist and organization effects of organizational constructs and therapist adherence on post-treatment charges. The results are presented in Table 1.

Table 1.

Therapist, Organization, and Therapist Adherence Effects on Post-treatment Charges

	Charge Counta				Charge Counta				TAM-R
Predictor	γ	SE	DF	p	γ	SE	DF	p	γ	SE	DF	p
Job Satisfaction
Intercept	0.313	0.114	37	.010	0.376	0.137	37	.010	0.629	0.013	37	<.001
Provider Mean	−0.014	0.031	37	.652	−0.030	0.033	37	.363	0.002	0.005	37	.636
Therapist Deviation	−0.016	0.008	281	.049	−0.011	0.009	281	.210	0.004	0.002	281	.032
TAM-R					−0.676	0.238	38	.008
Decision Making
Intercept	0.346	0.116	37	.005	0.401	0.140	37	.007	0.628	0.012	37	<.001
Provider Mean	−0.106	0.051	37	.045	−0.098	0.057	37	.092	0.004	0.008	37	.577
Therapist Deviation	−0.019	0.021	281	.377	−0.025	0.023	281	.286	−0.008	0.005	281	.094
TAM-R					−0.677	0.241	38	.008
Hierarchy
Intercept	0.313	0.113	37	.009	0.368	0.137	37	.011	0.628	0.012	37	<.001
Provider Mean	−0.007	0.041	37	.866	0.022	0.045	37	.624	0.003	0.007	37	.691
Therapist Deviation	0.013	0.011	281	.238	0.010	0.012	281	.412	−0.002	0.002	281	.445
TAM-R					−0.660	0.239	38	.009
Emotional Exhaustion
Intercept	0.328	0.114	37	.007	0.393	0.138	37	.008	0.631	0.013	37	<.001
Provider Mean	−0.064	0.062	37	.314	−0.069	0.067	37	.306	0.008	0.010	37	.395
Therapist Deviation	−0.001	0.012	281	.911	−0.001	0.013	281	.945	−0.007	0.003	281	.007
TAM-R					−0.691	0.239	38	.007
Role Conflict
Intercept	0.318	0.114	37	.009	0.381	0.138	37	.009	0.629	0.013	37	<.001
Provider Mean	0.017	0.047	37	.724	0.020	0.050	37	.697	0.002	0.008	37	.818
Therapist Deviation	0.009	0.010	281	.398	−0.004	0.011	281	.714	−0.003	0.002	281	.158
TAM-R					−0.680	0.240	38	.008
Procedural
Intercept	0.308	0.111	37	.009	0.364	0.133	37	.010	0.628	0.012	37	<.001
Provider Mean	−0.031	0.099	37	.757	−0.073	0.105	37	.492	−0.018	0.015	37	.254
Therapist Deviation	0.022	0.026	281	.398	0.010	0.027	281	.721	−0.002	0.005	281	.749
TAM-R					−0.711	0.239	38	.006
Growth & Advancement
Intercept	0.315	0.114	37	.009	0.373	0.138	37	.011	0.631	0.012	37	<.001
Provider Mean	−0.006	0.034	37	.859	.004	0.036	37	.913	0.012	0.005	37	.027
Therapist Deviation	−0.040	0.015	281	.010	−0.041	0.017	281	.013	0.006	0.003	281	.051
TAM-R					0.238	0.238	38	.009

Note: T-ratio test statistic (omitted) computed as (γ / SE). Event Rate (omitted) computed as exp(γ_ijk). TAM-R = Therapist Adherence Measure – Revised.

^aModel adjusted for variable length of exposure, age, gender, ethnicity, and number of lifetime pre-treatment charges.

Therapist and Organization Effects of Organizational Climate and Structure on Post-Treatment Charges and Therapist Adherence

Job Satisfaction, Decision Making, and Growth and Advancement were significantly associated with the annual rate of post-treatment charges. Specifically, therapists who provided higher ratings of Job Satisfaction and Growth and Advancement relative to others in their provider organization (i.e., the average level) had a significantly lower average rate of youth post-treatment charges, ERR = 0.98, 95% CI_ERR = 0.97 – 1.00 and ERR = 0.96, 95% CI_ERR = 0.93 – 0.99, respectively. Provider organizations with a higher average Participation in Decision-Making score had a lower average rate of youth post-treatment charges, ERR = 0.90, 95% CI_ERR = 0.81 – 1.00.

Job Satisfaction, Emotional Exhaustion, and Growth and Advancement were significantly associated with therapist adherence. Specifically, therapists who perceived higher levels of Job Satisfaction relative to the average for their provider organization had a significantly higher average level of caregiver-reported therapist adherence, γ₀₁₀ = 0.004, 95% CI = 0.0001 – 0.008. Also, therapists who perceived higher levels of Emotional Exhaustion relative to their provider organization had a significantly lower average level of adherence, γ₀₁₀ = −0.007, 95% CI = −0.013 – −0.001. Finally, holding constant the effects of individual therapist reports, provider organizations with higher average Growth and Advancement scores had significantly higher adherence, γ₀₀₁ = 0.012, 95% CI = 0.002 – 0.022. In addition, therapists who reported higher Growth and Advancement relative to the average for their organization had higher caregiver-reported therapist adherence, nearly reaching statistical significance, γ₀₁₀ = 0.006, 95% CI = 0.0001 – 0.02, p = .051.

Direct Effect of Therapist Adherence on Post-Treatment Charges

Holding constant the effects of the other covariates, the level of therapist adherence experienced by a family during the treatment episode was significantly associated with a lower rate of post-treatment charges, γ₅₀₀ = −0.629, SE = 0.239, T = −2.63, DF = 38, p = < .013, ERR = 0.53, 95% CI_ERR = 0.33 – 0.87. Thus, the annual rate of post-treatment charges associated with the highest level of adherence was 47% lower than that for the lowest level of adherence, and the annual rate of post-treatment charges for an adherence score 1 SD above the mean was 29% lower than that for a score 1 SD below the mean.

Therapist and Organization Effects of Organizational Climate and Structure and Therapist Adherence on Post-Treatment Charges

Finally, the effects were modeled of therapist adherence and each organizational variable on the annual rate of youth post-treatment charges. For each model, the effect of therapist adherence was negative and significant, with event rate ratios ranging from 0.46 to 0.49, such that higher levels of therapist adherence were associated with a lower rate of post-treatment charges. That is, the rate of post-treatment charges associated with the highest level of adherence was approximately 50% of that associated with the lowest level of adherence. Only one of the organizational variables was significantly associated with the rate of post-treatment charges when therapist adherence was included in the model. Youth treated by therapists who perceived higher levels of Growth and Advancement relative to the average level for their provider organization had a significantly lower annual rate of youth post-treatment charges. Specifically, a Growth and Advancement score one unit higher than the organization average was associated with a 4% lower annual rate of post-treatment charges (i.e., ER = 0.96, 95% CI_ER = 0.93 – 0.99). The effects of Job Satisfaction and Decision Making observed on post-treatment charges in the absence of therapist adherence were not observed in the presence of adherence.

Discussion

The current investigation is the first to our knowledge to examine prospectively, in a large sample of youth and families, therapists, and provider organizations, relations between therapist adherence to an evidence-based treatment for youth and its long- term sequelae. The investigation found caregiver ratings of therapist adherence during a treatment episode predicted lower rates of long-term criminal activity among delinquent youth treated with MST, independently and in the presence of the effects of organizational climate and structure modeled at therapist and provider levels. Two climate indicators – Job Satisfaction and Growth and Advancement; and one structure indicator – Participation in Decision Making – predicted such criminal activity when therapist adherence effects were held constant. When both organizational and therapist adherence effects were modeled simultaneously, however, only therapist perceptions of Growth and Advancement relative to the provider organization average related to youth criminal activity. In addition, the rate differential in post-treatment criminal charges associated with Growth and Advancement was considerably lower (4%) than that associated with adherence (47%).

The circumscribed nature of climate and structure effects on youth outcomes found in this investigation prompts several interpretations. One is that linkages between climate, services, and youth outcomes found in studies of public statewide child welfare organizations may not characterize mental health service organizations implementing evidence-based treatments. For example, the majority of provider organizations in the current study (93.7%) were private entities contracted by public juvenile justice, child welfare, and mental health agencies to treat youth with serious antisocial behavior. As such, the participating organizations resemble child-serving mental health organizations nationally, the majority of which are also privately held, and report starting new clinical service programs regularly (Schoenwald, Chapman et al., 2008). In addition, although the nature of service provided to youth was not described in the child welfare and juvenile justice case management studies (Glisson & Green, 2006; Glisson & Hemmelgarn, 1998), it is likely these services were characterized by the indeterminacy of process and outcomes common in the human services and in treatments used by mental health professionals in practice contexts (Weisz et al., 1995a, 1995b). In contrast, the treatment model implemented in the current study, while more individualized and flexible relative to some empirically supported treatments for youth, is reasonably well specified. For example, there are manuals and training protocols for practitioners, clinical supervisors, expert consultants, and organizations hosting MST programs. Eligibility criteria for the target population and expected outcomes are clearly defined and objectively measured. Perhaps the impact of climate and structure on services and outcomes is more circumscribed when (a) specific treatment protocols for specific target populations are implemented; and (b) implementation support is specified at multiple levels of the organizational context (Fixsen et al., 2005; Klein & Knight, 2005).

An alternate explanation is that the climate and structure measures did not operate similarly with the current sample (i.e., therapists who voluntarily chose to implement an evidence-based treatment employed by organizations that chose to start MST programs). Psychometric examination of the original OCQ and POS scales using conventions established in previous studies of organizational climate, structure, and culture suggest they performed similarly in the current sample. For example, interrater agreement for all but one scale was medium to high. A third possibility is that the statistical modeling approach used in this study partially accounts for the lack of robust climate or structure effects. Both therapist and organization level effects were modeled simultaneously, and this approach differs from that taken in some studies examining relations among climate, structure, and client outcomes (see, e.g., Glisson & Hemmelgarn, 1998; Morris & Bloom, 2002; Morris et al., 2007). The approach taken here reflected twin interests in how MST therapists perceived organizations employing them and the organizations. Those interests were borne in part of the contrasting circumstances characterizing randomized effectiveness trials of MST and its transport to usual care settings. In each of the trials, 3–4 therapists at a time were employed either by the university or a community agency to implement MST. Transporting MST involved many therapists employed by many organizations; and, as such provided a first opportunity to examine empirically how MST therapists perceive organizations and the organizations themselves. Modeling both therapist and organization level effects illuminated the importance of individual therapist perceptions relative to organizational average levels of climate indicators (e.g., job satisfaction, growth and advancement, emotional exhaustion) in predicting caregiver-reported therapist adherence. Such findings are consistent with theory and research suggesting that although perceptions of the climate of a workplace are often shared (as evidenced via statistical methods) climate is essentially an indicator of individual perceptions, whereas other constructs including structure are a property of organizations (Glisson, 2002).

Limitations

Several limitations of this investigation warrant attention. First, the evaluation of youth outcomes at the extended follow-up (on average 4 years post-treatment) was possible only for archival data, and not for the youth behavior and functioning outcomes examined post-treatment and at 6- and 12-month follow-up (Schoenwald et al., 2003; Schoenwald, Carter et al., 2008). Insofar as the significant long-term reductions in juvenile crime and cost savings associated with such reductions prompted service systems to request the transport of MST, this focus on long-term criminal outcomes in treatment transport is warranted. This focus, however, potentially limits the pertinence of findings regarding organizational and adherence effects on long-term youth outcomes to the transport of MST and other empirically supported treatments for juvenile offenders.

Second, the statistical modeling approach in this investigation examined both therapist and organization effects on outcomes. As noted in the Discussion, this approach contrasts with models that capture either the individual level response, or aggregate individual responses to reflect an organizational score on the basis of statistical indicators of the appropriateness of such aggregation (see, e.g., Aarons & Savitsky, 2006; Glisson & Hemmelgarn, 1998; Morris & Bloom, 2002). As noted in the Measures section, the current sample was characterized by both considerable interrater agreement and noteworthy therapist variance on organizational scales. In addition, the substantive questions of the investigation were best served by modeling both therapist and organizational level data effects, although not by modeling the contextual effect (the difference between the organizational and individual level effects, Raudenbush & Bryk, 2002).

Third, rather than aggregating across subscales within climate, or structure, separate models examined relations among each of the distinct climate and structure subscales, therapist adherence, and youth outcomes. The decision to model the effect of each scale on therapist adherence and youth criminal outcomes reflected our substantive research questions and the first time use of the organizational climate and structure measures with a sample of therapists and mental health organizations implementing an evidence-based treatment. The substantive question driving the investigation was: Which, if any, specific aspects of climate and structure would impact therapist adherence to MST and youth outcomes? As noted in the Introduction, select subscales of climate and structure had previously been found to predict short-term outcomes (youth discharge circumstances, behavior, and functioning); and, therapist adherence moderated some of those effects (Schoenwald et al., 2003). Thus, the decision was made to retain the focus on distinctive aspects of climate and structure in the current investigation of long-term outcomes, as evidence suggested each might have different effects on adherence, and/or outcomes. We recognize, however, the possible inflation of Type 1 error is a limitation of this modeling strategy, and are cautious in our interpretations of significant organizational findings. The lack of data on organizational covariates reinforces the need to be cautious about interpreting such findings.

Implications for Practice, Research, and Policy

The current findings suggest therapist adherence is a robust predictor of criminal activity among delinquent youth treated with MST in usual care settings up to on average 4.1 years post-treatment. Further, the effects of select aspects of organizational climate and structure on therapist adherence and youth outcomes appear to operate at the individual rather than provider organization average level. Moreover, with the exception of Growth and Advancement, these effects appear to wash out in the presence of therapist adherence. These findings suggest more clinical and research attention to therapist adherence is needed in treatment effectiveness trials (Perepletchikova, Treat, & Kazdin, 2007) and in the transport of effective treatments to usual care settings. The findings suggest linkages between adherence and outcomes established in randomized trials can be replicated in usual care settings when using a treatment-model with specific training and ongoing clinical support. In addition, therapist perceptions of select aspects of organizational climate and structure can affect adherence, suggesting attention to therapist experiences of the organizational context could be an asset in clinical implementation support strategies.

Additional research is needed to illuminate which aspects of mental health service provider organizations in general, and of organizations implementing evidence-based treatments specifically, have the potential to affect the implementation and outcomes of specific evidence-based treatments. For example, the current study did not assess organizational culture, leadership, and resources, constructs identified in organizational theory and research as potentially important to innovation implementation (Glisson, 2002; Simpson, 2002). The expansion within the past five years of the number of sites attempting the implementation of evidence-based treatments for youth has facilitated the conduct of at least two studies currently underway that examine these issues (Chamberlain et al., 2007; Glisson & Schoenwald, 2005). Both studies are multi-site randomized controlled trials testing the effects of organizational and community development interventions designed to support the implementation of evidence-based treatments for youth on the climate, structure, and culture of organizations, and on the implementation and outcomes of the treatments.

The current findings also have implications for policy efforts designed to support the widespread application of evidence-based interventions in usual care, such as state legislation, federal regulations, and service funding requirements linked to the use of evidence-based treatments. Such efforts are likely to fall short of their promise absent strategies to ensure the effective implementation of the interventions in usual care settings. The current findings suggest, for example, that therapist adherence is relatively more important than organizational climate and structure to long-term criminal outcomes of youth receiving MST in usual care settings. However, therapist perceptions of select organizational climate and structure variables can affect such adherence. In the case of therapist perceptions of relatively greater opportunities for growth and advancement relative to the organizational average, a significant association with youth criminal charge outcomes remained in the presence of adherence. Thus, both model-specific clinical support and organizational intervention may be needed to ensure the desired implementation and outcomes of a specific treatment in usual care settings. Accordingly, the scope of policies to stimulate the use of specific evidence-based treatments should address empirically identified barriers to their effective implementation at multiple levels of the practice context.