Randomized Trial of MST and ARC in a Two-Level EBT Implementation Strategy

Abstract

A two-level strategy for implementing evidence-based mental health treatment was assessed in a 2 × 2 randomized trial with 615 delinquent youth in 14 rural Appalachian counties. The implementation strategy included (1) the introduction of a Multisystemic Therapy (MST) program for delinquent youth in each county, and (2) the ARC (for Availability, Responsiveness and Continuity) organizational intervention for implementing effective community-based mental health services in randomly assigned counties. Within each county, youth were randomly assigned to the MST program or to usual services programs, yielding 4 treatment conditions (MST plus ARC, MST only, ARC only, control). Results of multilevel mixed-effects regression analyses found both main and interaction effects of MST and ARC on youth outcomes. Total problem behaviors as measured by the Child Behavior Checklist (CBCL) decreased significantly to non-clinical levels in the MST plus ARC condition by the six-month post-test, but were at clinical levels for youth in the MST only, ARC only, and control conditions. In addition, youth in the MST plus ARC condition entered out-of-home placements at less than half the rate (16%) of youth in the control condition (34%).

Several government reports and special issues of research journals have called for the development of effective mental health services implementation strategies (IOM, 2006; NIMH, 2006; 2008). They suggest that, first, implementation strategies that simply train clinicians in a new clinical treatment model (evidence-based or not) are insufficient to implement effective services (e.g., Sholomskas et al., 2005). Second, implementing effective mental health services requires strategies that remove service barriers in the organizational and community contexts in which the services are provided (e.g., Chamberlain et al., 2008; Schoenwald & Henggeler, 2003). Third, the pace of scientific advancement in strategies for addressing implementation barriers in organizational and community contexts lags far behind the development of new treatment models.

All three points are addressed here with an experimental study of a two-level strategy that includes (1) the implementation of an evidence-based treatment (EBT) program for delinquent youth, Multisystemic Therapy (MST; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009), and (2) an organizational intervention labeled ARC (for Availability, Responsiveness and Continuity) that addresses service barriers in the organizational and community social contexts (Glisson, 2007, 2008; Glisson, Dukes, & Green, 2006). The study’s rationale is based on evidence that service barriers are created by organizational and community social contexts through the norms and values that govern expectations about the way things are done, shared beliefs about the cause, prevention and treatment of mental health problems, and existing organizational and community service structures. As a result, social contexts affect (1) which EBTs are adopted, (2) the effective implementation of EBTs, and (3) the sustainability of EBTs. The current study focuses on implementation and assesses the outcomes of a two-level implementation strategy in 14 rural, Appalachian counties.

Implementing Mental Health Services in Rural Communities

Rural communities vary widely on many factors, including population density, income, ethnicity, and social structure. The more impoverished and isolated rural areas present significant mental health risks to children and effective mental health services are needed for the children who are most at risk (Barth, Wildfire & Green, 2006; Costello, Farmer, Angold, Burns, & Erkanli, 1997; Robbins, Dollard, Armstrong et al, 2008; Spoth, Trudeau, Guyll Shin, & Redmond, 2009; Starr, Campbell & Herrick, 2002). Challenges to implementing effective mental health services in many rural communities include physical isolation, poor communication infrastructures, a lack of transportation, limited resources, social mores that emphasize self-sufficiency, closed social systems, and stigmatized views about mental health issues (Boydell, Stasiulis, Barwick, Greenberg & Pong,, 2008; Fox, Berman, Blank, & Rovnyak, 1999; DHHS, 2004; Rost, Fortney, Fischer & Smith, 2002).

Research in rural and community mental health suggests these challenges can be met with contextual interventions that (1) build grass roots support for mental health services, (2) develop social networks among service providers, stakeholders and opinion leaders, (3) link mental health service programs to the socio-political institutions of the community, and (4) provide organizational structures and processes for identifying and addressing service barriers (Beeson et al., 1998; Boydell et al, 2008; Bjorklund & Pipard, 1999; Fox, Blank, Rovnyak, & Barnett, 2001; Molgaard, 1997; St. Lawrence & Ndiaye, 1997).

The ARC Organizational Intervention Model

The ARC intervention model addresses the challenges to implementing effective mental health services with three strategies. First, it provides organizational tools (e.g., teamwork, goal setting, feedback systems) required for identifying and addressing service barriers. Second, ARC introduces service organizations to principles of effective service systems (e.g., mission-driven, results-oriented, participation-based). Finally, ARC addresses service provider behaviors and attitudes (e.g., rigidity, resistance, detachment) that discourage service improvement efforts. The ARC model incorporates manual-guided activities within 12 intervention components. These components are implemented in three stages to develop the collaboration, participation, and innovation within organizational and community social contexts necessary for effective implementation (Glisson, 2007, 2008; Glisson et al., 2006). The activities are introduced and guided by an ARC specialist who works with (1) communities to create support for services to the target population (e.g., delinquent youth), (2) service organizations to facilitate improvements in service delivery, and (3) key individuals to develop social networks among service providers, stakeholders and opinion leaders (e.g., judges, principals).

The ARC specialist forms action groups of service providers and stakeholders using planned processes to identify and remove service barriers by changing procedures and practices within the organization and community. Service barriers addressed in the ARC process cover a wide range of issues, such as poor cooperation between institutions (e.g., mental health service providers and school systems), youths’ lack of access to support services (e.g., transportation, recreational services), and unnecessary bureaucratic red tape within service systems (e.g., inefficient referral processes, excessive documentation requirements).

Implementing MST in Rural Communities

MST is an intensive family and community based treatment model for delinquent youth that has been identified by independent reviewers, federal agencies, and consumer groups as among the most effective treatments for serious antisocial behavior in adolescents. The published results of randomized controlled trials testing the effects of MST with chronic, violent, and substance abusing delinquents and of the implementation and outcomes of MST in community settings have been summarized in peer-reviewed journals (e.g., Eyberg, Nelson & Boggs, 2008; Kazdin & Weisz, 1998; Schoenwald, 2008) government reports (e.g., National Institute on Drug Abuse, 1999; National Institutes of Health, 2006; U.S. Public Health Service, 2001) and books (e.g., Henggeler et al., 2009; Hoge, Guerra, & Boxer, 2008; Weisz, 2004).

An aspect of MST that may be particularly amenable to implementation in rural areas is its home-based model of service delivery, which addresses the lack of transportation, the time required for travel to a therapist’s office, the caregiver’s need for childcare, and the potential stigma associated with going to a mental health clinic. Similarly, the low caseloads (4–6 families) and flexible hours required for MST (e.g., clinicians are available 24 hours/day, 7 days/week) allow therapists to expend intensive and sustained effort based on the individual capacities of each family. The strength-focused nature of MST and its emphasis on indigenous support systems may also be well suited to rural families (Fox et al., 2001).

Rural communities also present significant challenges to the implementation of MST in isolated areas: (1) Treatment progress can be harder to maintain if therapist travel time reduces session frequency or duration; (2) obtaining support from the youth’s ecological system can be more difficult when therapists must travel significant distances; (3) limited organized recreational activities make sustainable sources of prosocial peer interaction for the youth more difficult to develop; and (4) field supervision, collegial coverage of cases, and exchange of information among colleagues are diminished.

Implementing MST with ARC

ARC is expected to improve the implementation of MST and youth outcomes in these impoverished rural communities in several ways. The intra-organizational components of ARC are expected to align therapist job requirements with the demand characteristics of MST, improve treatment team functioning and reduce job stress. Greater alignment of bureaucratic rules and regulations with the demand characteristics of MST could facilitate adherence to MST protocols in the face of extended geographical distances, relative isolation, and limited community resources.

The inter-organizational components of ARC are expected to improve cooperation among key institutions (e.g., service providers, schools) and build collaboration among service providers and stakeholders to improve access to vocational, recreational and other resources that provide opportunities for community support and pro social peer interaction. Such cooperation can affect system related outcomes that depend on services and support from multiple community institutions, and is expected to affect the efficiency of MST implementation and the distribution of therapist efforts across systems in the youth’s social ecology. For example, ARC may decrease the time therapists spend in contact with community institutions on a case-by-case basis, thereby enhancing efficiency; and/or may result in shifting the distribution of time the therapist intervenes in one or more systems (e.g., family, school, peer, and neighborhood).

The present study was designed to test the main and interaction effects of ARC and MST within rural settings to assess the extent to which implementing ARC in conjunction with MST influenced its implementation and impact. Our specific predictions were:

• Hypothesis 1

  MST treatment model fidelity, efficiency of therapeutic effort within community and service systems, and MST treatment outcomes are superior in counties that receive the ARC organizational intervention compared to counties not receiving ARC.

• Hypothesis 2

  Psychosocial (reduced behavioral problems) and systems outcomes (reduced out of home placements) are superior for youth in counties receiving the ARC organizational intervention compared to counties not receiving ARC.

• Hypothesis 3

  Psychosocial (reduced behavioral problems) and systems outcomes (reduced out of home placements) are superior for youth who receive MST compared to youth who receive the usual services.

Method

The rural Appalachian region of eastern Tennessee is the poorest in the state and one of the poorest in the nation (Appalachian Regional Commission, 2000). Counties from the region were selected that (1) are not included within a Metropolitan Statistical Area, (2) have a majority of their populations in communities of fewer than 2,500 residents, (3) have per capita incomes below that of Tennessee ($28,641), (4) have proportions of children in poverty above that of Tennessee (17.6%), and (5) have no cities with populations that exceed 15,000. Fourteen (14) of the poorest, least populated counties in eastern Tennessee meeting these criteria were selected for the study and all agreed to participate.

The study incorporated a 2 × 2 factorial design. First, the ARC intervention was randomized at the county level. Six of the 14 sampled counties were randomly selected to participate in the ARC intervention. There were no significant differences between ARC and non-ARC county means, respectively, on total population (34,536 vs. 35,896; .95 CIμ₁–μ₂ = −16,792, 19,511; p = .873), per capita income ($15,351 vs. $14,650; .95 CIμ₁–μ₂ = −2,749, 1,347; p = .470), and proportion of children in poverty (20.8% vs 21.8%; .95 CIμ₁–μ₂ = −4.7, 6.8; p = .780). Second, delinquent youth were randomly assigned within each county to either the MST program that was introduced in each county or to the usual services programs.

Selection and Randomization of Youth

A power analysis for clustered data estimated that a sample of 560 youth within 14 counties was necessary for an adequate test of the primary hypotheses specifying the effects of MST (randomization of youth within each county) and ARC (randomization of counties). Power was estimated for modest effects (.35 standard deviation units), using RMASS software for 3-level clustered data with repeated measures at the first level (Roy, Bhaumik, Aryal & Gibbons, 2007), as .90 with .01 Type I error rate for MST effects (randomization at second level), and as .80 with .05 Type I error rate for the analysis of ARC effects (randomization at third level).

Between October, 2003, and September, 2007, youth referred to juvenile court in each of the 14 counties who met the following criteria were recruited and randomly assigned to the MST program or to the usual services programs. Selection criteria required that the youth (1) not have a sibling in the study (2) not be psychotic, (3) live with at least one parent or long-term caregiver, (4) be 9 to 17 years of age, (5) be referred to juvenile court for a status offense or delinquent behavior, (6) be at risk of out-of-home placement (e.g., repeated court referrals), (7) have a DSM IV diagnosis other than adjustment disorder, (8) be TennCare (Tennessee’s Medicaid waiver program) eligible, (9) have a Global Assessment of Functioning (GAF) score of less than 51 (Greenberg & Rosenheck, 2005; Hilsenroth et al, 2000), (10) have behavioral or psychiatric symptoms that require intervention, and (11) have an IQ greater than 54. Criteria 1–5 were specified by the research team and 6–11 by TennCare as eligibility criteria for the necessary category of funded services. The youth were assessed by University of Tennessee Children’s Mental Health Services Research Center (UT CMHSRC) licensed mental health providers assigned to each participating juvenile court.

A four-step participant recruitment process included each juvenile court’s youth services officer (YSO) and three UT CMHSRC staff – a research specialist, licensed mental health clinician, and data manager: (1) During routine intake, the YSOs identified youth who met selection criteria 1–6 and notified the UT research specialists. (2) The research specialists met with the youth and caregivers to describe the study, confirm that initial criteria were met, and obtain informed consent using protocols approved by NIMH and the UT and Medical University of South Carolina (MUSC) IRBs. (3) If the caregiver and youth provided their consent, the clinician conducted the mental health screening. (4) Following confirmation that the youth met the clinical criteria, the mental health clinician informed the research specialist who then contacted the data manager for the random assignment to treatment condition and informed the families. Assignment to MST or usual services was determined by a predetermined, concealed randomization of sequence numbers based on the order of recruitment.

As shown in Figure 1, a total of 674 youth and their caregivers agreed initially to participate, met the clinical criteria, and were randomly assigned to the MST treatment program or to the usual services programs. Both MST and usual services were funded by TennCare, Tennessee’s Medicaid waiver program, and billing and support were the same across all counties and conditions.

Figure 1.

Recruitment and random assignment of youth within counties

Participants

Youth participants

Of the randomly assigned youth, 615 participated in the study using a non-blind protocol, including 284 in ARC counties and 331 in non-ARC counties, with 316 youth receiving MST and 299 receiving usual services. As shown in Table 1, the youth averaged 14.90 years of age (SD = 1.59), 69 percent were male and reflecting the racial characteristics of the rural Appalachian region of eastern Tennessee, 91 percent were white.

Table 1.

Characteristics of youth in sample

	Min.	Max.	Mean	SD
Age of participants	9	17	14.9	1.6
Family income (monthly)	0	6000	1161.2	801.8
GAF raw score (baseline)	20	50	45.8	4.0
CBCL Total Problem T score (baseline)	23	90	65.4	9.6
Participant	Frequency	Percentage
Gender
Male	425	69.1
Female	190	30.9
Ethnicity
Caucasian	562	91.4
African American	28	4.6
Biracial	15	2.4
Hispanic	6	1.0
Native American	2	.3
Other	2	.3
Most Frequent DSM Diagnoses
ADHD	174	28.4
Oppositional Defiant Disorder	146	23.8
Disruptive Behavior Disorder	112	18.3
Substance Related Disorders	86	14.0
Depressive Disorder	75	12.2
Bipolar Disorder	63	10.3
Conduct Disorder	56	9.1
Dysthymic Disorder	54	8.8
Major Depressive Disorder	50	8.2
Generalized Anxiety Disorder	35	5.7
PTSD	23	3.8
Adjustment Disorder	23	3.8

Over half (53%) of the youth had two or more mental health diagnoses. As shown in the Table 1, the most common diagnoses for the youth included ADHD (28%), oppositional defiant (24%), disruptive behavior (18%), substance abuse (14%), depressive disorder (12%), bipolar (10%), conduct disorder (9%), dysthymic (9%), major depression (8%), and anxiety (6%).

Youth assigned to the MST condition received MST for an average of 105.22 days (SD = 44.30). There was no difference in length of MST treatment between ARC (104.89 days) and non-ARC counties (105.49 days). The mean duration of usual services was 186.6 days (SD = 138.3). There was no significant difference in the duration of usual services between non-ARC counties (184.6 days) and ARC counties (188.5 days). Youth assigned to usual services received inpatient (24%) and outpatient (90%) mental health services, and family and parent focused mental health treatment (50%) from a wide variety of providers, including individual practitioners (43%), mental health centers (41%), in-home therapists (39%) and physicians (5%).

Caregivers

The youths’ primary caregivers served as respondents for assessing the youths’ behavior problems, out-of-home placements, and treatment fidelity. The primary caregivers were most often female (80 %), were the youth’s biological mother (67 %), biological father (12%), or grandmother (11%), usually unmarried (86 %), and had family incomes averaging $1,162 per month (SD = $801).

MST therapists

MST therapists were employed by Tennessee’s largest, private, mental health service organization which has multiple offices throughout the state. Seven treatment teams provided MST to the 14 counties during the study. Each team served either two ARC or two non-ARC counties with an average size of five therapists per team. Therapists were mostly female (82%) with an average age of 29 years, and four years of experience providing mental health services, with almost half holding masters degrees (45%). Thus, the sample resembled therapists implementing MST nationwide (Schoenwald, 2008). Informed consent was obtained from all therapists at the beginning of the study and from new therapists who replaced those who left. There was 30% turnover in the current study, which compares favorably to the 39% rate (annual rate of 21%, range from 0 – 50%) observed during the two and a half year clinical implementation phase of the multi-site MST transportability study (Sheidow, Schoenwald, Wagner, Allred & Burns, 2006). MST therapists were not blind to study conditions for two reasons. First, randomization to the ARC intervention occurred at the county level and, second, several components of the ARC intervention involve therapist participation.

ARC and MST Intervention Conditions

ARC intervention

The ARC intervention was guided by a trained ARC specialist (a Ph.D. industrial organizational psychologist) who directed the manual guided activities for each of 12 components sequenced in three stages (Glisson, 2009; Glisson et al., 2006). For example, in the first stage (collaboration) the components begin with the ARC specialist working with community stakeholders and upper management in targeted mental health agencies to plan implementation activities, endorse the implementation process, communicate a clear vision for the implementation effort, set high performance standards and actively support change. As another example, the first component in the next stage (participation) begins with the ARC specialist directing activities that build teamwork within newly formed action groups of service providers and stakeholders. These activities facilitate information sharing and support participative decision-making within those teams to identify service barriers. The first component in the third stage (innovation) begins with the ARC specialist helping action groups form short term and long term goals focused on removing the service barriers identified in the previous stage.

The ARC specialist worked with various service providers, service managers, YSOs, juvenile court judges, school administrators, county and city officials, and other opinion leaders to identify and address barriers to services for delinquent youth. Between September of 2003 and December of 2006, the ARC specialist devoted approximately 24 hours per week in the field (four hours per county per week) and 12 hours per week in the office (two hours per county per week) in record keeping, contacting service providers and stakeholders by phone and email, and planning and arranging meetings in each county assigned to ARC. Approximately 70% of the time on ARC activities was spent with service organizations and 30% was spent with other community stakeholders.

Examples of the types of barriers identified in the ARC intervention included: (1) schools allowing service providers insufficient access to teachers and limited collaboration between service providers and schools, (2) MST therapists needing additional collegial support to resolve problems encountered in their most difficult cases, (3) poor communication between courts and service providers, and (4) the reluctance of individual therapists to identify service problems in their own service system. These and other implementation barriers were addressed in the ARC intervention process under the guidance of the ARC specialist. For example, (1) procedures were developed in collaboration with school officials and service providers to provide efficient access to teachers; (2) MST treatment teams developed a system that prioritized their discussion of cases so that treatment team meeting time was devoted to the most difficult cases; (3) weekly meetings were established between court officials (e.g., YSOs) and community service providers to share information about services to delinquent youth; and (4) a team process was created to identify and address service barriers.

Multisystemic Therapy (MST) clinical protocol

MST is an intensive, family-based treatment developed for delinquent youth at imminent risk of incarceration or other out-of-home placements. MST specifically targets factors in each youth’s social ecology (family, peer, school, neighborhood, and community systems) (Henggeler et al., 2009). MST assessment and intervention activities are guided by nine principles and a specified analytic process. Interventions typically include improving caregiver discipline and monitoring practices, reducing family conflict, improving affective relations, decreasing youth association with deviant peers, increasing association with prosocial peers, improving school or vocational performance, and developing an indigenous support network of family, neighbors, and friends. Specific treatment techniques are integrated from those therapies that have the most empirical support, including cognitive behavioral, behavioral, and pragmatic family therapies..

MST implementation protocol

In addition to the manuals for therapists, the MST quality assurance system consists of several elements (Henggeler et al., 2009; Schoenwald, 2008): (a) Site assessment; (b) 5-day orientation for therapists and clinical supervisors;(c) on-site clinical supervision guided by a manual-based supervision protocol (Henggeler & Schoenwald, 1998); (d) weekly consultation with an MST expert trained in a consultation protocol (Schoenwald, 1998); (e) quarterly booster training for therapists and supervisors; and (f) feedback on therapist and supervisor adherence to MST protocols. The MST quality assurance system is deployed through MST Services (MSTS), an organization licensed by the Medical University of South Carolina and its 21 Network Partners (Schoenwald, 2008; in press). The MST teams in the study were licensed throughout the duration of their participation.

Measures of MST Implementation and Fidelity

MST implementation was assessed using several measures pertaining to the primary therapist --the therapist treating the family for the entire treatment episode or, for families treated by more than therapist, the one providing treatment for the majority of the family’s treatment episode. In the current and prior studies of MST implementation, no differences between primary and other therapists emerged for therapist (e.g., demographics, professional experience), family (e.g., demographics, problem severity), or treatment (length, completion) variables (Schoenwald, Carter, Chapman, & Sheidow, 2008; Schoenwald, Chapman, Sheidow, & Carter, 2009).

MST Therapist Logs

The Therapist Log is completed weekly for each family to identify the subsystems of the youth’s social ecology that are addressed. The logs yield three indicators: (a) whether the therapist addressed a particular subsystem during the specified week; (b) how many minutes the therapist spent working with that subsystem; and (c) therapist ratings of progress or deterioration in that subsystem.

MUSC research assistants trained MST therapists to complete the logs, contacted each therapist by telephone or e-mail weekly for the most recent log, and reviewed all logs upon receipt. Logs were obtained for 295 of 316 (93%) families participating in MST and logs provided by primary therapists for 287 of the families (91%) were used in the analyses. Therapists’ consistency of log ratings for the treated families, as indexed by the level 3 reliability coefficients produced by HLM modeling of repeated ratings (level-1) within families (level-2) within therapists (level-3), were as follows. For indicators (a) and (b), reliabilities in the majority of the domains ranged from .67 to .84. For indicator (c), reliabilities in the majority of domains ranged from .60 to .69.

MST fidelity indicators

Two indicators of therapist fidelity to MST were used, one molar and one molecular (Perpletchikova, Treat, & Kazdin, 2007; Waltz, Addis, Koerner, & Jackson, 1993) The 28 item MST Therapist Adherence Measure- Revised (TAM-R; Henggeler, Borduin, Schoenwald, Huey, & Chapman, 2006) is a valid, reliable measure of therapist conformance to the nine MST treatment principles (Schoenwald et al, 2008; Schoenwald et al, 2009; Schoenwald, Sheidow, Letourneau, & Liao, 2003). The measure is administered monthly to caregivers who respond on the basis of the prior week’s treatment. TAM-R data were collected monthly by phone by the University of Tennessee Social Science Research Institute (UT SSRI) and data from 243 (76%) of families in the MST condition were analyzed. Among families with excluded data, 7.6% rated a therapist other than the primary therapist, 7.2% provided ratings outside of parameters for valid data collection (i.e., > 14 days after ratings were due or > 30 days after treatment ended), and 10.2% provided no ratings. Data were neither differentially missing by ARC versus non-ARC condition nor as a function of therapist or family, as evidenced by null findings of HLM unconditional means models predicting the odds of TAM-R data availability.

A molecular indicator of adherence, the MST audio coding adherence system, focuses on the interactions within a treatment session. Audio-taped sessions were coded in an independent laboratory at the University of Southern California under the direction of Dr. Stanley Huey using a validated system from other trials of MST (Huey, 2001). The lab randomly selected and coded three tapes per family from early, mid, and late treatment. A single rater rated each tape, multiple raters rated tapes for the same therapist in different sessions, and multiple raters rated tapes for the same therapist-family pair in different sessions. Two methods were used to evaluate possible rater influence: (1) A Many-Facet Rasch Model (MFRM; Linacre, 1989) evaluating the leniency/severity of raters showed little variation in the levels of rater severity (i.e., rater severities covered a range of 0.8 logits); however, the statistical test of the equality of rater severity across raters was rejected (Myford & Wofle, 2003, 2004). (2) Two four-level measurement models were specified and estimated in MlwiN (Rasbash, Charlton, Browne, Healy, & Cameron, 2009) to evaluate item ratings nested within measurement occasions that were nested within families (or therapists) who were nested within raters. In models with families at level 3, items accounted for 75% of variance, with families, occasions, and raters accounting for 9%, 13%, and 3%, respectively. In models with therapists at level 3, items accounted for 74% of variance, with therapists, occasions, and raters accounting for 11%, 12%, and 2%, respectively.

Coded tapes were available for 46% (144 of 316) of the families. Missing tapes were attributable to several factors including: an IRB prohibition against taping sessions involving individuals outside of the family; technical problems including broken tape recorders and poor audio quality (25%); available tapes for a family limited to therapists other than the primary therapist (8%); family opposition to taping sessions and some therapists forgetting to bring recorders to the home. Results of HLM (families nested within therapists) unconditional models of the log odds of a family having a coded tape showed a random effect at the therapist level (p < .001), but no differences between ARC and non-ARC conditions.

MST supervisor adherence was measured using the 43-item Supervisor Adherence Measure (SAM; Schoenwald, Henggeler, & Edwards, 1998). Scale development, reliability, and validity have been detailed elsewhere (Schoenwald, 2008; Schoenwald, Sheidow, & Chapman, 2009). Four valid SAM subscales predict therapist adherence and youth behavioral problems through one-year post-treatment. MUSC research assistants contacted each MST therapist every other month to complete the SAM. The level of supervision adherence is the mean of therapist reports on a supervisor during a family’s treatment.

Measures of Youth Outcomes

Youth behavior problems

Youth behavior problems were measured with the Child Behavior Checklist (CBCL). The CBCL describes 113 problem behaviors applicable to children aged 5 to 18 years (Achenbach, 1991). Caregivers were asked to rate the extent to which the behavior was true of their child during the previous 6 months on a scale that ranged from 0 (not true) to 2 (very often or often true). Given the broad range of diagnoses for the participating youth shown in Table 1, standardized Total Problem Behavior T-scores based on norms for age and gender that combine information from both Externalizing Behavior and Internalizing Behavior scales, were calculated for each youth. As illustrated in Figure 1, UT CMHSRC research specialists administered the CBCL in personal interviews with each caregiver at baseline within 24 hours of the youth appearing in court and at 6 months, 12 months and 18 months following baseline. Alpha reliabilies ranged between .94 and .95 across waves.

Youth out-of-home placements in state custody

Youth placed in state custody in out-of-home placements were determined by interviews with the youth’s caregiver via monthly phone calls placed by the UT SSRI and in-depth personal interviews conducted by UT CMHSRC research specialists at 6 months, 12 months and 18 months following baseline. Out-of-home placements in state custody were identified for 23 percent of the youth in the sample.

Results

The research questions are divided into two groups. The first group focuses on MST implementation and fidelity using data collected from MST therapists and the caregivers of the youth who received MST (i.e., half of the sample). The second group of questions focuses on outcomes among youth in the full sample, comparing MST and usual services in ARC counties and non-ARC counties. Different multilevel random effects models were used to answer each group of questions. For questions regarding MST implementation and fidelity, youth who received MST were nested within MST therapists. For questions regarding differences in youth outcomes across all conditions in the full sample, youth were nested within counties.

The rationale for using these multilevel models begins with two considerations. First, some but not all MST therapists treated youth in two counties (in either two ARC or two control counties but not in both). Therefore, youth who received MST could be nested within MST therapists or nested within counties but not hierarchically nested within both. We elected to nest youth within MST therapists, rather than nesting youth within counties, as a more appropriate strategy for assessing MST implementation and fidelity. Second, youth from the full sample could be nested within counties but not within MST therapists because only half of the youth received MST. Therefore, the analyses of MST and ARC effects on youth outcomes in the full sample nested youth within counties. The data were analyzed with HLM 6 Linear and Nonlinear Modeling software (Raudenbush, Bryk & Congdon, 2005). The unstandardized regression weights from these analyses are interpreted below in relation to the scaling of each instrument described in the preceding measurement sections.

MST Implementation Indicators

ARC effects on MST implementation were examined for youth assigned to the MST condition nested within MST therapists, as explained above. This approach is not without consequences, as described by Meyers and Beretvas (2006). Computation of Intraclass Correlations (ICCs) for the MST implementation and fidelity indicators showed non-trivial clustering of data at the county level (ICC = .10). Thus, following procedures specified by Hedges (2007) to correct significance tests for un-modeled clustering, we calculated adjusted outcome p values for an ICC level of .10. That is, the p values were penalized for the un-modeled dependency in the implementation data attributable to the nesting of therapists in counties.

As noted in the introduction, ARC was expected to enhance MST fidelity and the efficiency of MST therapist intervention efforts overall, specifically with community and service systems. Redistribution of intervention activities across the subsystems in the youths’ ecology was also considered likely in the ARC condition, although specific predictions were not made about shifts in intervention time for each subsystem.

Therapist time spent addressing systems and subsystems

A three-level model was used to estimate the effect of ARC on the time MST therapists spent addressing each subsystem. Weekly Therapist Logs were nested within youth who were nested within therapists. A three-level model assessed the minutes spent for each system/subsystem where Y = minutes, X₁ = ARC, e = repeated measure random effect, r₀ = youth random effect, u₀₀ = therapist random effect:

• Level 1: Repeated Weekly Therapist Log

  $$\mathrm{Y}={\mathrm{\pi }}_0+\mathrm{e}$$
• Level 2: Youth Level

  $${\mathrm{\pi }}_0={\mathrm{\beta }}_{00}+{\mathrm{r}}_0$$
• Level 3: Therapist Level

  $${\mathrm{\beta }}_{00}={\mathrm{\gamma }}_{000}+{\mathrm{\gamma }}_{001}{\mathrm{X}}_1+{\mathrm{u}}_{00}$$

MST therapists in the ARC condition spent on average fewer minutes weekly working with the youth’s family system (β −42.3, SE = 11.8, p= .001); and with the youth’s caregiver (β −25.4, SE = 8.4, p = .004). Within the family system, MST therapists in the ARC condition spent less time weekly working conjointly with the primary caregiver and family members (β −28.3, SE = 10.16, p = .007); with the other caregiver and family members (β − 14.8, SE = 6.95, p = .036); and, with the primary caregiver and youth together (β −33.0, SE = 9.24, p = .001). For the remaining 13 (of 16) subsystems, there were no differences between ARC and non-ARC counties.

Probability of a therapist addressing specific systems and subsystems

A three-level Bernoulli model with a logit link function was used to analyze the therapist’s probability of addressing a system/subsystem where φ = probability of addressing a system/subsystem. The three-level model is the same as the previous model except for Level 1 where:

$$\text{Log}[\mathrm{\phi }/(1-\mathrm{\phi })]={\mathrm{\pi }}_0.$$

Counter to expectations, the results showed equivalent probability across ARC and non-ARC conditions that a particular system or subsystem was addressed in a given week.

Therapists’ progress ratings

Because therapist ratings of progress in a given week are judged in relation to prior weeks in treatment, cumulative ratings of progress were examined. The effect of ARC on the cumulative amount of progress in treatment was estimated using a three-level model identical to the first model described above except Y = the cumulative progress rating. Differences were found for most (9 of 16) subsystems, all favoring the ARC counties. For two of the nine subsystems in which the differences in progress were found -- Family with Other, and Natural Community with Caregiver -- MST therapists in the ARC condition provided significantly lower initial ratings of progress, and subsequently judged families in the ARC condition as experiencing more cumulative progress in treatment. These findings suggest ARC may have sensitized therapists to the potential insularity of rural families and importance of cultivating family-community linkages in rural communities. Table 2 presents the nine subsystems in which therapists in the ARC condition reported significantly greater cumulative progress throughout treatment. Several of these subsystems reflect the interface of the caregiver or youth with the community or formal service systems, further suggesting ARC increased MST therapist awareness of the importance of these interfaces.

Table 2.

Significant Differences by Condition in Cumulative Progress Ratings

Domain/Subdomain	Intercept			Slope
	Coefficienta	SE	95% CI	Coefficientb	SE	95% CI
Family w. Primary CG	−0.19	0.52	−1.21, 0.83	0.10***	0.02	0.06, 0.14
Family w. Other	−1.03*	0.37	−1.76, −0.30	0.18***	0.02	0.14, 0.22
Individual Child	−1.06	0.52	−2.08, −0.04	0.28***	0.02	0.24, 0.32
Natural Community w. CG	−1.41*	0.54	−2.47, −0.35	0.28***	0.03	0.22, 0.34
Peer w. Youth	−0.36	0.41	−1.16, 0.44	0.09**	0.02	0.05, 0.13
Service System w. CG	−0.56	0.49	−1.52, 0.40	0.20***	0.02	0.16, 0.24
Service System wo. CG	0.19	0.50	−0.79, 1.17	0.06*	0.02	0.02, 0.10
School w. CG	−0.03	0.54	−1.09, 1.03	0.13***	0.03	0.07, 1.03
School w. Youth	0.35	0.54	−0.71, 1.41	0.10**	0.03	0.06, 0.14

Note. CG = Caregiver, Intercept df = 71 for all outcomes.

^aCoefficient for ARC condition on intercept (First weekly Log).

^bCoefficient for ARC condition on slope (Cumulative change in progress across weekly Logs).

^*p<.05,

^**p<.01,

^***p<.001.

MST Fidelity

Therapist adherence as reported by caregiver

Analyses of ARC effects on the MST Therapist Adherence Measure (TAM-R) were conducted using two-level models where Y = family average TAM-R, X₁ = ARC, r = youth level random effect, u₀ = therapist level random effect.

• Level 1: Youth Level

  $$\mathrm{Y}={\mathrm{\beta }}_0+\mathrm{r}$$
• Level 2: Therapist Level

  $${\mathrm{\beta }}_0={\mathrm{\gamma }}_{00}+{\mathrm{\gamma }}_{01}{\mathrm{X}}_1+{\mathrm{u}}_0$$

There were no differences in caregiver reported MST therapist adherence between ARC and non-ARC conditions.

Therapist ratings of supervisor adherence

Using the same model, with Y = family average SAM, completed SAM measures were included in the analyses when primary therapists for families rated their primary supervisors, resulting in 257 SAMs provided by 91 percent of the therapists. There were no differences between ARC and non-ARC conditions.

Audio-coded therapist adherence

Ratings of randomly selected therapy sessions for audio-coding were analyzed using a three-level model with number of coded sessions nested within families, nested within therapists. The three-level model was similar to the previous model for assessing minutes spent on each system/subsystem, except that in Level 1, Y = therapist adherence. There were no differences in audio-coded therapist adherence ratings between ARC and non-ARC conditions.

Youth Outcomes in the Full Sample

The analyses of ARC and MST main and interaction effects examined two outcome criteria for all youth in the sample. The two criteria were (1) out-of-home placements and (2) CBCL Total Problem T-Scores. Two HLM models, an “unconditional” model with no condition effects and a “conditional” model, were examined in the analyses of both outcome criteria (Hedeker, & Gibbons, 2006; Raudenbush & Bryk, 2002).

Out-of-home placements

The effects of ARC and MST on youth out-of-home placement were examined with a two-level model to estimate the probability of a binary outcome (each youth either did or did not enter an out-of-home placement in the 18 month follow-up period). The analysis used a binomial sampling model based on the Bernoulli distribution and a logit link function with the log of the odds of custody as the outcome variable (Raudenbush & Bryk, 2002, pp. 294–309). The conditional model estimated the log of the odds of out-of-home placement for the main effect of MST at level one (youth) and for the main effect of ARC at level two (county).

The following two-level model estimated out of home placement where φ = probability of out of home placement, φ/(1−φ) = odds of out of home placement, X₁ = MST, X₂ = ARC, r = youth level random effect, and u_0, u₁ = county level random effects:

• Level 1: Youth level

  $$\text{Log}[\mathrm{\phi }/(1-\mathrm{\phi })]={\mathrm{\beta }}_{\mathrm{o}}+{\mathrm{\beta }}_1{\mathrm{X}}_1$$
• Level 2: County level

  $$\begin{array}{l}{\mathrm{\beta }}_{\mathrm{o}}={\mathrm{\gamma }}_{00}+{\mathrm{\gamma }}_{01}{\mathrm{X}}_2+{\mathrm{u}}_0\\ {\mathrm{\beta }}_1={\mathrm{\gamma }}_{10}+{\mathrm{\gamma }}_{11}{\mathrm{X}}_2+{\mathrm{u}}_1\end{array}$$

As shown in Table 3, out-of-home placements were significantly lower for the youth who received MST within each county and for the youth in counties that received the ARC intervention. The exponent of the adjusted log-odds intercept indicates a probability of out-of-home placement for youth who received usual services in a county without ARC of .34 (.518 = .34/.66).

Table 3.

HLM analysis of youth entering out-of-home residential placements using Bernoulli distribution with logit link function

Model	Variable	b	SE	exp b	t	df	p
Unconditional
	Constant	−1.189***	.115	.305	−10.367	13	.000
	County			Chi-square
	Variance	.043		15.842		13	.257
Conditional
	Constant	−.657***	.173	.518	−3.795	12	.003
	ARC	−.587*	..296	.556	−2.181	12	.050
	MST	−.751**	..277	.472	−2.710	12	.019
	ARC x MST	.365	.431	1.44	.848	12	.413

The adjusted relative odds of a youth entering an out-of-home placement in a county that received the ARC intervention are 56 percent [exp (−.587)] of the odds of out-of-home placement in a county that did not participate in the ARC intervention. This represents a 44 percent reduction in the odds of out-of-home placement for the estimated main effect of ARC.

The adjusted relative odds of a youth entering an out-of-home placement who received MST treatment are 47 percent [exp (.75)] of the odds of placement for youth who did not receive MST. This represents a 53 percent reduction in the odds of placement for the HLM estimated main effect of MST treatment. The significant main effects of MST and ARC together reduced the rate at which youth entered out-of-home placements by more than half, from 34 percent in the control condition (no MST and no ARC) to 16 percent of the youth in the ARC plus MST condition.

CBCL Total Problem Behavior T Scores

Results of two HLM analyses are reported for youth CBCL Total Problem Behavior T scores. The first examines MST and ARC effects on youth outcomes at six months after intake when both MST treatment and the average length of usual services were complete. The second examines MST and ARC effects on the trends in youths’ rates of change over the 18 month follow-up period.

The first two-level model estimated total problem outcome scores at six months, controlling for baseline scores, where Y = six month CBCL Total T score, X₁ = baseline CBCL Total T score (mean centered), X₂ = MST, X₃ = ARC, r = youth level random effect, and u_0, u₁ = county level random effects:

• Level 1: Youth level

  $$\mathrm{Y}={\mathrm{\beta }}_{\mathrm{o}}+{\mathrm{\beta }}_1{\mathrm{X}}_1+{\mathrm{\beta }}_2{\mathrm{X}}_2+\mathrm{r}$$
• Level 2: County level

  $$\begin{array}{l}{\mathrm{\beta }}_{\mathrm{o}}={\mathrm{\gamma }}_{00}+{\mathrm{\gamma }}_{01}{\mathrm{X}}_3+{\mathrm{u}}_0\\ {\mathrm{\beta }}_2={\mathrm{\gamma }}_{10}+{\mathrm{\gamma }}_{11}{\mathrm{X}}_3+{\mathrm{u}}_1\end{array}$$

As shown in Table 4 and Figure 2, a significant MST x ARC interaction effect indicated that youth in the ARC counties benefited more from MST treatment at the six month outcome than the youth in the non-ARC counties. The adjusted mean CBCL Total Problem T score for youth receiving MST in ARC counties (57.31) is in the non-clinical range (i.e., < 60), and significantly lower than the mean scores for youth in the three other conditions (60.75 – 60.84), all of which are in the clinical range (i.e., > 60) at six months. The level of total problem behaviors of youth assigned to MST in the ARC counties as estimated by the CBCL therefore indicate that the youth in that condition experienced significant clinical improvement after six months, while the problem behavior levels of youth in the other three conditions did not.

Table 4.

HLM analysis of CBCL Total Problem T score outcomes at six months

Model	Variable	Coefficient	SE	t-ratio	df	p-value
Unconditional
	Constant	59.998***	1.381	43.436	13	.000
	County variance	23.747
	Individual variance	103.741
	Chi-square	148.734***			13	.000
	ICC	.186
Conditional
	Constant	60.840***	1.305	46.609	12	.000
	CBCL baseline	.715***	.041	17.594	510	.000
	ARC	−.094	2.004	.047	12	.964
	MST	−.042	1.000	−.042	12	.968
	MST x ARC	−3.390*	1.472	−2.302	12	.040

Figure 2.

MST x ARC interaction effect on CBCL Total Problem T score outcomes at 6 months

Second, a three-level piecewise linear growth model examined the rate of change of each youth’s total problem scores over 18 months in two periods (Raudenbush & Bryk, 2002). Period 1 captures the youths’ rate of change between baseline and 6 months. Period 2 captures the rate of change between 6 and 18 months. The three-level model for the piecewise trend analysis over four waves (0, 6, 12, 18 months) where Y = CBCL Total T score, X₁ = Period 1 (0 to 6 months), X₂ = Period 2 (6 months to 18 months), X₃ = MST, X₄ = ARC, e = repeated measures random effect, r_0, r_1, r₂ = youth level random effects, u_00, u₁₀, u₁₁, u_20, u₂₁ = county level random effects is:

• Level 1: Repeated measures level

  $$\mathrm{Y}={\mathrm{\pi }}_{\mathrm{o}}+{\mathrm{\pi }}_1{\mathrm{X}}_1+{\mathrm{\pi }}_2{\mathrm{X}}_2+\mathrm{e}$$
• Level 2: Youth level

  $$\begin{array}{l}{\mathrm{\pi }}_0={\mathrm{\beta }}_{00}+{\mathrm{r}}_0\\ {\mathrm{\pi }}_1={\mathrm{\beta }}_{10}+{\mathrm{\beta }}_{11}{\mathrm{X}}_3+{\mathrm{r}}_1\\ {\mathrm{\pi }}_2={\mathrm{\beta }}_{20}+{\mathrm{\beta }}_{21}{\mathrm{X}}_3+{\mathrm{r}}_2\end{array}$$
• Level 3: County level

  $$\begin{array}{l}{\mathrm{\beta }}_{00}={\mathrm{\gamma }}_{000}+{\mathrm{u}}_{00}\\ {\mathrm{\beta }}_{10}={\mathrm{\gamma }}_{100}+{\mathrm{\gamma }}_{101}{\mathrm{X}}_4+{\mathrm{u}}_{10}\\ {\mathrm{\beta }}_{11}={\mathrm{\gamma }}_{110}+{\mathrm{\gamma }}_{111}{\mathrm{X}}_4+{\mathrm{u}}_{11}\\ {\mathrm{\beta }}_{20}={\mathrm{\gamma }}_{200}+{\mathrm{\gamma }}_{201}{\mathrm{X}}_4+{\mathrm{u}}_{20}\\ {\mathrm{\beta }}_{21}={\mathrm{\gamma }}_{210}+{\mathrm{\gamma }}_{211}{\mathrm{X}}_4+{\mathrm{u}}_{21}\end{array}$$

Caregivers of 567 (92%) of the 615 youth in the sample completed CBCLs for two or more waves and were included in the piecewise trend analysis. Figure 1 shows that missing data increased over time with the most missing data occurring at the final 18 month wave (43%). A pattern-mixture model for the analysis of missing data found no differences in the main and interaction effects of ARC and MST between youth with and without data at each wave (Hedeker & Gibbons, 2006). We also conducted pattern mixture tests specifically accounting for missing data from youth placed out-of-home and found no differences in outcome effects.

As reported in Table 5, the unconditional piecewise growth model shows significant downward linear trends in problem behaviors in both periods. However, the rate of change in the second period, 6 to 18 months (−1.575), is less than the rate of change in the first period, baseline to 6 months (−5.498). The ICCs for the two periods indicate a much higher proportion of variance at the county level in Period 1 than Period 2, and the random variance terms for the county-level trend in Period 2 is not significant. Following strategies outlined by Raudenbush & Bryk (2002) for analyses of growth trends, the non-significant county-level random error term in Period 2 was fixed at zero in the conditional model. The youth-level random error term for Period 1 was not significant but was allowed to vary in the conditional model to allow youth level variance in Period 1 trends.

Table 5.

HLM piecewise growth analysis of CBCL Total Problem T scores during 18 month follow-up

Model	Variable	Coefficient	SE	t-ratio	df	p-value
Unconditional
	Baseline	65.377***	.901	72.569	13	.000
	Period 1 growth rate (0 to 6 months)	−5.498***	.840	−6.541	13	.000
	Period 2 growth rate (6 to 18 months)	−1.575***	.323	−4.877	13	.000
				Chi-square
	County baseline variance	8.545***		89.141	13	.000
	County Period 1 variance	7.424***		73.760	13	.000
	County Period 2 variance	.491		18.082	13	.154
	Youth baseline variance	42.744***		893.013	449	.000
	Youth Period 1 variance	4.900		458.780	449	.364
	Youth Period 2 variance	8.257***		613.026	449	.000
	ICC (period 1)	.602
	ICC (period 2)	.056
Conditional
	Baseline	65.388***	.917	71.294	13	.000
	Period 1 growth rate
	Intercept	−4.873**	1.287	−3.788	12	.003
	ARC	.664	1.924	.345	12	.736
	MST	−.551	.928	−.593	565	.553
	ARC x MST	−2.901*	1.369	−2.119	565	.034
	Period 2 growth rate
	Intercept	−1.345**	.509	−2.642	565	.009
	ARC	−1.593*	.727	−2.191	565	.029
	MST	−.261	.702	−.372	565	.709
	ARC x MST	2.259*	1.025	2.205	565	.028

The HLM piecewise growth analysis in the conditional model shown in Table 5 shows a significant MST x ARC interaction in Period 1, indicating that total problem scores for youth in the MST plus ARC condition declined immediately after baseline at a significantly higher rate (−7.66 points) than total problem scores in the other conditions. Although the total problem behaviors continued to decrease for youth in all four conditions during the second follow-up period, a significant MST x ARC interaction for Period 2 indicates that the youth in the MST plus ARC conditions experienced less decline in problem scores (−.94) in the second period. Thus, the steep initial downward trend in total problem behaviors among youth in the MST plus ARC condition during the first six months began to flatten in the 6 to 18 month period. The differences in problem levels were eliminated by the end of the second period between the MST plus ARC (55.85), ARC (55.30), MST (56.75), and control (57.30) conditions as a result of the differences in rates of change between the four conditions during the second follow-up period.

Discussion

The two-level strategy for implementing an evidence-based treatment program for delinquent youth was effective in reducing out of home placements and reducing the youths’ total problem behaviors reported by the youth’s caregivers. Both levels of the implementation strategy, (1) the MST treatment program for randomly assigned delinquent youth within each county and (2) the ARC organizational intervention in randomly assigned counties, contributed to effective outcomes. There were MST and ARC main effects on out of home placements and an MST x ARC interaction effect on youth problem behaviors.

The significant main effects indicated that youth who received MST and youth in ARC counties were each significantly less likely to enter out-of-home placements. As a result, the combined effects of both factors on the rate of out-of-home placements resulted in youth who received MST in ARC counties being placed out-of-home at less than half the rate (16%) of youth who received the usual services in the non-ARC counties (34%).

Youth who received MST in the ARC counties also improved at a faster rate in the first six months after intake, resulting in statistically and clinically significant lower levels of psychosocial problem behaviors in the MST plus ARC condition than in the other conditions. CBCL Total Problem Behavior T scores for youth in the MST plus ARC condition were in the non-clinical range after six-months while CBCL Total Problem Behavior T scores were at clinical levels for youth in the other three conditions (MST only, ARC only, control).

The main and interaction effects of ARC and MST on youth outcomes, combined with equal levels of MST fidelity in ARC and non-ARC counties, suggest the mechanisms of ARC action pertain to the organizational and community social contexts of the implementation effort rather than to effects on fidelity to the MST treatment model. The findings indicate that ARC affected therapist perceptions of the rural community context of the families they served in potentially important ways. Specifically, therapists in ARC counties reported greater clinical progress throughout treatment in 9 of 16 intervention subsystems, primarily those reflecting the interface of the caregiver or youth with the community and service system. In addition, although MST therapists in ARC and non-ARC conditions were equally likely to address all systems and subsystems in the youth’s ecology, and to dedicate the same amount of time to treatment overall, MST therapists in the ARC condition spent less time with three subsystems, including family sessions with the primary caregiver, family sessions with the other caregiver, and with the youth’s caregiver. These findings provide partial support for the hypothesis that the inter-organizational components of ARC facilitated greater therapist sensitivity to, and improvement in relations between, the family, community, and service system in these rural and impoverished counties. In addition, ARC led to some redistribution of therapeutic effort suggestive of greater therapist efficiency (less time spent) in the family domain.

Study Limitations

The experimental design of this study provides a rigorous test of the effects of a two-level strategy for implementing a new evidence-based treatment program, particularly with delinquent youth in rural and impoverished communities. However, four study limitations suggest caution in the interpretation of some of the findings. (1) Youth out-of-home placement was reported by the youths’ primary caregivers rather than archival sources (i.e. official records). (2) The piecewise trend analyses of CBCL Total Problem scores were based on data from 92% of the sample with two or more reports, but missing data increased over time with the most occurring at the final 18-month wave. However, no differences were found in the effects of ARC and MST between youth with and without data at each wave, so concerns about bias are somewhat mitigated. Results of pattern mixture tests of missing data accounting for youth placed out-of-home also showed no differences in outcome effects of ARC and MST. (3) The analyses of MST fidelity and implementation had less power because of the smaller sample size (i.e., only youth who received MST). Also, as noted in the Results section, data on one MST fidelity indicator – audio coded treatment sessions – were available for just under half of the families treated with MST. Although there were no differences between ARC and non-ARC counties in the pattern of missing fidelity data, the significant random effect for therapist in analyses predicting tape availability suggest the null findings for audio data could be attributable to therapist related biases. (4) Finally, given the rural Appalachian setting of the study, there are limits on the generalization of our findings to other types of settings.

Implications for Future Research

This study provides support for the use of contextual interventions that address organizational and community barriers to the implementation of mental health services and evidence-based treatment programs (Glisson, 2002; Glisson et al., 2008). The effectiveness of the two-level strategy described here underscores the need for additional research on implementation strategies that build support for mental health treatment programs within the organizational and community contexts in which they are introduced (NIMH, 2006, 2008; Simpson & Flynn, 2007). This study indicates the ARC model for developing collaboration and participation among service providers and community stakeholders in identifying and removing service barriers improved the outcomes of an EBT program for delinquent youth. Understanding the mechanisms of action through which ARC exerts these effects on youth outcomes, the implementation of an evidence-based treatment, and linkages between implementation and outcomes, is a priority of future research. In addition, there is much more to learn about the attributes of organizational implementation strategies that contribute to the adoption and sustainability of effective treatments in diverse organizational and community settings.

Implications for Clinical Practice

The ARC intervention is designed to develop collaboration, participation and innovation among members of service organizations and community stakeholders in implementing services for a specific population. ARC strategies are designed to identify and address contextual barriers to service quality and outcomes using a series of structured intervention components that include direct service providers, service system administrators, and representatives from courts, schools, and other stakeholders. Purveyors of several evidence-based treatments, including but not limited to MST, have designed strategies to accomplish similar aims, although few are as comprehensive as the model used here and few are being tested experimentally (e.g., Chamberlain et al, 2008; Henggeler, Sheidow, Cunningham, Donohue, & Ford, 2008). The evidence from this experimental study of the implementation of a new evidence-based treatment program in each of 14 rural Appalachian counties suggests that interventions aimed at developing organizational and community collaboration, participation and innovation in identifying and removing contextual barriers to mental health services are important to success in clinical practice.