Using a Cross-Study Design to Assess the Efficacy of Motivational Enhancement Therapy–Cognitive Behavioral Therapy 5 (MET/CBT5) in Treating Adolescents with Cannabis-Related Disorders

Abstract

Objective:

To evaluate how adolescents with marijuana problems who received a research-based treatment (five sessions of motivational enhancement therapy plus cognitive behavioral therapy [MET/CBT5]) in an experimental setting would have fared had they received exemplary community-based outpatient treatment.

Method:

Twelve-month outcomes representing six domains (substance use problems, substance use frequency, emotional problems, illegal activities, recovery, and institutionalization) were assessed for youth who received MET/CBT5 in the Cannabis Youth Treatment study and youth who received outpatient treatment from one of three community-based programs selected for evidence of efficacy. Groups were matched on pretreatment characteristics using a propensity score weighting strategy.

Results:

Youth who received MET/CBT5 exhibited greater reductions in substance use frequency, substance use problems, and illegal behaviors 12 months after treatment entry than had they entered the community-based outpatient programs. Results showed no evidence that youth who received MET/CBT5 would have fared better with respect to emotional problems, the likelihood of being institutionalized, or achieving a "recovery" status at 12 months had they received community-based treatments.

Conclusions:

The community-based treatments used in this study have not been assessed relative to "treatment as usual" but were selected as "exemplary" models of adolescent treatment. There is no evidence in this study that these exemplary programs yielded superior 12-month outcomes for the treatment of adolescents with marijuana problems; youth receiving MET/CBT5 experienced greater reductions in substance use and illegal activities. Thus, MET/CBT5 may be a promising treatment for community-based providers to adopt to treat these clients.

One strategy for ensuring that adolescents with substance use problems receive high-quality treatment is to encourage community-based treatment programs to adopt treatment approaches that have demonstrated some degree of efficacy in rigorously designed studies (Institute of Medicine, 2006). Beginning in 2003, the Substance Abuse and Mental Health Services Administration's (SAMHSA's) Center for Substance Abuse Treatment funded 38 treatment programs to deliver one such treatment: five sessions of motivational enhancement therapy plus cognitive behavioral therapy (MET/CBT5) (Sampl et al., 2001).

MET/CBT5, offered in five sessions over the course of 6–7 weeks, was the shortest of five promising brief interventions compared in the Cannabis Youth Treatment (CYT) study (Diamond et al., 2002). It consists of two components. Adolescents first received two individual motivational enhancement therapy sessions to help resolve their ambivalence about whether they have a drug problem and increase their motivation to stop using marijuana. They then received three group cognitive behavioral therapy sessions designed to teach basic skills for refusing drug offers, establishing a supportive social network of recovery, developing a plan for non-drug-using activities, and coping with unanticipated high-risk situations (Diamond et al., 2002; Sampl et al., 2001).

In the CYT study, MET/CBT5 was compared with four other brief interventions where, using an intent-to-treat approach, it was shown to be as efficacious as the other treatments and one of the more cost-effective treatments (Dennis et al., 2004). The CYT study did not explore how the treatment performed relative to the usual care youth typically receive in community-based drug and alcohol treatment programs. As such, there is little empirical evidence that MET/ CBT5 has the potential to produce better outcomes than are typically observed. Nonetheless, MET/CBT5 is now widely considered to be an inexpensive first-line intervention for treating substance-using behaviors among adolescents with cannabis use problems (Riley et al., 2008). Policy makers' confidence in the approach is reflected in SAMHSA's investment of $7 million to facilitate the adoption of MET/CBT5 by 38 treatment programs under the Effective Adolescent Treatment funding program (SAMHSA, 2003).

Although the evidence suggests that MET/CBT5 generates comparable results to other evidence-informed treat-ments, it has yet to be evaluated against treatments delivered outside experimental settings. A notable exception is a recent randomized, controlled trial in one facility that compared a seven-session version of the treatment (MET/CBT7) with that facility's standard, evidence-informed outpatient program. In that study, clients who received MET/CBT7 had somewhat lower increases in the percentage of days abstinent over the 12-month follow-up, although the effect sizes were small and economic analyses indicated that MET/CBT7 was the more cost-effective intervention (Godley et al., 2010).

In the present study, we evaluated how MET/CBT5 fared relative to nonexperimental treatments using data from a standardized assessment of adolescents similar to those in the CYT study but who enrolled in one of three well-established outpatient treatment programs that were selected to participate in SAMHSA's Adolescent Treatment Model (ATM) program. The programs were the following: (a) Chestnut Health Systems' (Bloomington, IL) outpatient adolescent services, which blends Rogerian, behavioral, cognitive, and reality therapies in a combination of individual and group counseling sessions for 1–8 hours of services per week (or 9–12 hours per week for intensive cases) (Godley et al., 2003); (b) Epoch Counseling Center (Baltimore, MD), which provides motivational interviewing and psychoedu-cational services in group and family counseling sessions (Battjes et al., 2003); and (c) Teen Substance Abuse Treatment (Maricopa County, AZ), which uses cognitive-behavioral and family systems theories and provides 9 hours of services weekly that include in-home individual and family sessions, teen group therapy, and multifamily group therapies (Stevens et al., 2003).

These three programs were selected because they were able to demonstrate 2 years of continuous operation, willingness to evaluate the effectiveness of their program, willingness to develop and make publicly available a treatment manual, some evidence or testimony suggesting the merits of their services, and willingness to collaborate with the other selected program sites (Dennis et al., 2003). By meeting these criteria, SAMHSA identified these programs as "exemplary" (Stevens and Morral, 2003). However, at the time of funding, they were not required to demonstrate effectiveness relative to other treatment programs. Since that time, each has been evaluated and provided evidence that the clients engaged in their programs exhibited some reductions in adverse behaviors (Battjes et al., 2004; Godley et al., 2004; Stevens et al., 2007). In the absence of a representative sample of contemporary outpatient programs, the three ATM outpatient programs offer an attractive comparison sample because clients were assessed with the same instruments used in the CYT study and at the same 3-month intervals and because ATM sites had high rates of recruitment and follow-up (described below). To the extent that they outperform standard care in community-based outpatient treatment, they represent a conservative comparison group.

Combining data from ATM and CYT, we use principles developed in the Rubin Causal Model (Holland, 1986; Rubin, 1974) to measure the efficacy of MET/CBT5 compared with community-based treatment. Similar to a randomized, controlled trial, the method we use enables us to identify a population of adolescents that received community-based outpatient treatment whose pretreat-ment characteristics are closely matched with those who received MET/CBT5. We used these samples to examine how youth receiving MET/CBT5 would have fared 12 months after intake had they received treatment at one of these three community-based treatment programs. We compared the effectiveness of the two treatment approaches on six outcomes presented in the main findings from the CYT experiment (Dennis et al., 2004): (a) the percentage of adolescents in recovery, defined as no use or abuse/dependence problems while living in the community, at 12 months after enrollment; (b) the percentage institutionalized within the 90 days before the 12-month follow-up; (c) substance problems; (d) substance use frequency; (e) engagement in illegal activities; and (f) emotional problems at the 12-month follow-up. Our study provides one of the first evaluations of the efficacy of this widely disseminated adolescent treatment intervention.

Method

CYT study sample and the MET/CBT5 intervention

Details of the CYT study design and results have been previously published, including descriptions of the study rationale, design, and analysis plan (Dennis et al., 2002); the treatments that were tested (Diamond et al., 2002); the clients (Tims et al., 2002); and the main findings (Dennis et al., 2004). In the CYT study, conducted in 1997, 600 adolescents were randomized to one of five different outpatient treatment interventions (including MET/CBT5) offered at one of four treatment sites (University of Connecticut Health Center, Farmington, CT; Operation PAR, St. Petersburg, FL; Chestnut Health Systems, Bloomington, IL; and Children's Hospital of Philadelphia, Philadelphia, PA). The 600 adolescents were selected to represent youth "appropriate for and [that] typically present to publicly funded outpatient treatment" (Dennis et al., 2004, p. 199): 12- to 18-year-old recent cannabis users who reported at least one symptom of cannabis abuse or dependence. Participants were compensated for completing each follow-up assessment. Among youth randomized to MET/CBT5 in the CYT study, 74% fully completed the prescribed treatment and only 5% were randomized but did not receive any treatment (Dennis et al., 2004). The current analysis is based on only the 174 youth who were randomized to receive MET/CBT5 and who had 12-month follow-up data. The 12-month follow-up rate was 95%.

ATM sample

Youth in the community-based treatment condition were drawn from three outpatient treatment programs funded in 1998–1999 as part of the ATM program supported by the Center for Substance Abuse Treatment. In two funding periods, a total of 10 adolescent treatment programs were funded to collect detailed survey information on clients at intake and follow-up (Dennis et al., 2003). A description of facility eligibility criteria for participation in the ATM study and overviews of the three treatment programs are presented in the introduction section of this article; more detail can be found in Stevens and Morral (2003). In analyses that controlled for differences in client characteristics, none of the three programs appeared to perform better relative to each other (Morral et al., 2006) and were thus combined for the current analysis.

Our analysis used intake and 12-month follow-up data from clients treated at the three ATM outpatient sites. A total of 463 adolescents were assessed at these programs at intake, and 431 (93%) were recontacted at 12 months after intake.

Exclusion and inclusion criteria

The target population for the CYT study was adolescents with cannabis-related disorders who would typically present themselves to publicly funded outpatient treatment programs across the United States. Participants were eligible if they were ages 12 to 18, self-reported one or more criteria for can-nabis abuse or dependence as specified in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR; American Psychiatric Association, 2000), had used cannabis in the past 90 days or 90 days before being sent to a controlled environment, and were appropriate for outpatient or intensive outpatient treatment (Mee-Lee and American Society of Addiction Medicine, 2001). The exclusion criteria were the following: (a) use of alcohol for 45 or more days during the 90 days before admission; (b) use of other drugs for 13 or more days during the 90 days before admission; (c) one or more acute medical or psychological problems; (d) insufficient mental capacity to understand and provide informed consent or participate in treatment; (e) living outside of the program's catchment area; (f) having a history of repeated, violent behavior or severe conduct disorder; or (g) not being proficient in the English language to consent to treatment or research interviews (Dennis et al., 2004). Applying these same criteria to the ATM sample resulted in 108 youth (25% of the ATM outpatient sample with 12-month follow-up) being excluded from the sample.

Assessment instrument

Data documenting pretreatment client characteristics and 12-month treatment outcomes were obtained from the Global Appraisal of Individual Needs (GAIN; Dennis, 1999). The GAIN is one of the most widely used assessment tools in studies of adolescent treatment and was used to assess clients in the ATM and CYT at intake and 3, 6, 9, and 12 months after intake. The GAIN has eight sections assessing background and demographic characteristics (at baseline only); substance use; physical health; risk behaviors; mental health; and environment, legal, and educational/vocational problem areas. The GAIN includes more than 100 symptom count, change score, and service utilization indices, most with Cronbach's α greater than .85 (Dennis et al., 1999). Test-retest analyses of key GAIN substance use and substance use problem scales in a sample of 210 adolescents indicated good reliability over a 90-day interval (r > .72 for each index) (Dennis et al., 2000).

Outcomes

We examined treatment outcomes in six areas: recovery, institutionalization, substance use problems, substance use frequency/intensity, emotional problems, and criminal activities. Recovery is a dichotomous (0/1) variable that combines the traditional outcome of abstinence from drugs with that of being free in the community and, therefore, at risk for using drugs. Recovery equals 1 if there is no use of any illicit drugs in the 3 months before the follow-up interview and if the person is not in a controlled environment during that interval. Institutionalization in the past 90 days is also dichotomous and indicates whether a youth spent 1 or more of the past 90 days in any of several types of controlled environments in which drug use and liberty were substantially constrained for the whole day (e.g., jail, inpatient treatment, group homes, or probation camps).

The Past Month Substance Problem Scale (Dennis et al., 2010) is a 16-item count of recent symptoms of DSM-IV drug and alcohol abuse. Inter-item reliability is excellent (α = .92 in the ATM data set). The Substance Frequency Scale (Dennis et al., 2010) is a seven-item scale that sums days of use during the past 90 days for alcohol, marijuana, and other illicit drugs. Inter-item reliability is good (α = .80 in the ATM data set). The Emotional Problem Scale (Dennis et al., 2010) is a seven-item scale assessing psychological distress recency, frequency, and severity during the 90 days preceding assessment (α = .80 in the ATM data set). The Illegal Activities Scale (Dennis et al., 2010) is a three-item scale assessing criminal behavior recency, frequency, and severity during the past 90 days (α = .74 in the ATM data set). For the latter four continuous measures, we modeled change between the baseline and 12-month assessment (i.e., Y₁₂-Y₀).

Analytic approach

Our analytic approach was designed to answer the following question: What is the relative efficacy of MET/CBT5 compared with community-based treatment programs for youth like those who entered the CYT experiment? To answer this question, we used propensity score weighting to compare the outcomes of youth receiving MET/CBT5 in CYT with those of similar clients receiving outpatient care in ATM. Specifically, we computed the average treatment effect on the treated population (ATT; Hirano et al., 2003; Rosen-baum and Rubin, 1983, 1984, 1985; Wooldridge, 2002). In our application, the estimated ATT is the expected treatment effect of receiving MET/CBT5 versus community-based treatment for the adolescent clients like those who actually received MET/CBT5 in the CYT study. The results did not allow us to interpret how those receiving care in the ATM study would have fared had they received MET/CBT5 in the CYT study.

Computing the ATT is a quasi-experimental strategy drawing on Rubin's Causal Model (Holland, 1986; Rubin, 1974), requiring that we estimate what the outcomes of youth in MET/CBT5 would have been had they instead entered the community treatments. We did this by creating propensity score weights that emphasized members of the ATM sample that looked similar to the MET/CBT5 group with respect to 108 pretreatment variables corresponding to patient placement criteria established by the American Society of Addiction Medicine (Mee-Lee and American Society of Addiction Medicine, 2001) and that have been used in previous investigations examining the effectiveness of adolescent drug treatment (McCaffrey et al., 2004; Mor-ral et al., 2006). Many of these pretreatment characteristics have been shown in prior work to influence drug and alcohol treatment outcomes, including pretreatment levels of substance use (Alford et al., 1991; Jenson et al., 1993; Kennedy and Minami, 1993; Shoemaker and Sherry, 1991), symptoms associated with emotional well-being and criminality (Alford et al., 1991; Brown et al., 1996, 2000; Myers et al., 1995), academic/scholastic attendance/performance (Rush, 1979; Shoemaker and Sherry, 1991), employment (Rush, 1979), sociodemographics (Alford et al., 1991; Cady et al., 1996; Friedman and Glickman, 1986; Friedman et al., 1986; Shoemaker and Sherry, 1991), and social/familial substance use (Richter et al., 1991; Shoemaker and Sherry, 1991). Missing values in the 108 pretreatment variables were rare (M = 1.22%, with 103 of the pretreatment variables having less than 5% missing) and were controlled for by balancing the two groups on missing value indicators for each variable as well as the 108 pretreatment variables described above.

The propensity score weights used in our analysis were computed using generalized boosted models (GBM), a flexible, nonparametric estimation technique that can regress the treatment indicator (MET/CBT5 vs. community treatment) onto a large number of pretreatment (baseline) covariates. GBM adaptively captures the functional form of the relationship between the covariates and treatment with less bias than traditional regression approaches (Lee et al., 2010;McCaffrey et al., 2004). Propensity score weight modeling was implemented using the twang (Toolkit for Weighting and Analysis of Nonequivalent Groups) package in R, and balance between the treatment and control groups was assessed across a number of twang diagnostic criteria (Ridgeway et al., 2010).

To show how comparable the MET/CBT5 and community-based treatment groups were after applying the propensity score weights, we calculated the pre- and post-weighting standardized mean difference (SMD) for each of the 108 pretreatment characteristics in the propensity score model. The SMD is estimated by (Ridgeway et al., 2010):

Here, μ̂_j denotes the estimated mean value for the treatment and control conditions (j = t and c, respectively); σ̂ denotes the estimated standard deviation for the variable in question and is estimated using the observed variance in the (unweighted) treatment condition. Values of 0 for an SMD thus represent no difference in means, whereas values of ±1 represent one standard deviation difference between the two groups. Standardized differences greater than 0.25 (and correspondingly less than −0.25) are considered to be "moderate effect size differences" (Cochran, 1968). Variables with SMD of greater than +0.25 or less than −0.25 were included as covariates in the outcome regression analyses described below to control for any lingering confounding of the variable after weighting (Neugebauer and van der Laan, 2005).

We estimated the ATT for each of our outcomes separately. For binary outcomes (recovery and institutionaliza-tion), we fit weighted logistic regression models to obtain the predicted means and standard errors for each group where the outcome was regressed on an indicator of treatment condition (1 = MET/CBT5; 0 = community-based treatment) and covariates that remained imbalanced after propensity score weighting (see Table 1 for the list of variables). For the four continuous outcomes (changes in Past Month Substance Problem Scale, Substance Frequency Scale, Emotional Problems Scale, and Illegal Activities Scale), we estimated the ATT on change score outcomes using weighted linear regression models in the same manner.

Table 1.

Patient characteristics in the Cannabis Youth Treatment (CYT) study, the original Adolescent Treatment Model (ATM), and reduced ATM before and after weighting (means)

Variable	MET/CBT5 n = 174) M	ATM (all) (n = 431) M	ATM (only elig.) (n = 323) M	ATM (weighted) (ESS = 115) M
Demographics
Female	0.213	0.213	0.183	0.201
White	0.672	0.671	0.700	0.729
Black/African American	0.264	0.132	0.127*	0.177
Hispanic	0.029	0.100	0.080*	0.029
Other	0.034	0.095	0.093*	0.065
Age, years	15.695	15.752	15.749	15.634
In school in the past 90 days	0.879	0.824	0.818	0.894
Pattern of substance use
Past Month Substance Problem Scale	3.759	3.146	2.898	3.603
Substance Frequency Scale	0.174	0.147	0.127*	0.157
Weekly or daily marijuana use	0.695	0.530	0.553*	0.672
Weekly or daily alcohol use	0.178	0.191	0.146	0.154
Other risk behaviors in 90 days before intake
Multiple sexual partners	0.386	0.336	0.303	0.304
Had sex without barrier protection	0.235	0.294	0.263	0.233
Any needle use	0.000	0.009	0.006	0.004
Used at work, at school, or while babysitting	0.356	0.369	0.356	0.382
Used in hazardous situation	0.483	0.464	0.443	0.452
Lifetime history of behavioral interventions
Institutionalization, days in past 90	5.408	13.179	13.111*	5.419
Drug and alcohol treatment	0.277	0.390	0.365	0.257
Mental health treatment	0.260	0.444	0.460*	0.400*
Juvenile justice system involvement	0.856	0.907	0.898	0.812

Notes: MET/CBT5 = five sessions of motivational enhancement therapy plus cognitive behavioral therapy; elig. = eligible; ESS = effective sample size.

^*Standardized mean difference compared with CYT study sample > 0.25. Means were compared only after exclusion criteria were applied to ATM sample.

Results

Participant characteristics

Tables 1 and 2 show how youth from the CYT study (n = 174) compare with youth from the following: (a) the total ATM outpatient sample (n = 431); (b) the reduced ATM sample, after applying exclusion criteria (n = 323); and (c) the propensity-score weighted ATM outpatient sample after applying exclusion criteria (effective sample size = 115). The reduced ATM sample looked similar to the total ATM sample, except that those in the reduced sample had higher levels of weekly or daily marijuana and alcohol use, and a greater proportion had a marijuana use disorder at baseline. Means were not compared between the ATM sample before applying exclusion criteria and the CYT study sample.

Table 2.

Patient clinical characteristics in the Cannabis Youth Treatment (CYT) study, the original Adolescent Treatment Model (ATM), and reduced ATM before and after weighting (means)

Variable	MET/CBT5 (n = 174) M	ATM (all) (n = 431) M	ATM (only elig.) (n = 323) M	ATM (weighted) (ESS = 115) M
Substance use disorder
Any marijuana disorder	0.828	0.557	0.616*	0.808
Marijuana dependence	0.445	0.413	0.422	0.484
Marijuana abuse	0.379	0.253	0.291	0.355
Any alcohol disorder	0.339	0.325	0.334	0.342
Other substance use disorders	0.132	0.114	0.080	0.126
Past-year psychological problem
Emotional Problem Scale	0.218	0.278	0.272*	0.263*
Any internal disorder	0.322	0.471	0.440*	0.400
Any external disorder	0.626	0.696	0.669	0.613
Physical, sexual, or emotional victimization
Lifetime history of victimization	0.580	0.712	0.684	0.607
Past year	0.362	0.479	0.457	0.419
Past 90 days	0.190	0.265	0.252	0.204
Extensive victimization	0.367	0.518	0.478	0.403
Violence and illegal activity
Illegal Activities Scale	0.224	0.226	0.213	0.203
Any illegal activity	0.902	0.968	0.966	0.939
Property crimes	0.586	0.710	0.703	0.623
Interpersonal crimes	0.443	0.689	0.669*	0.535
Drug-related crimes	0.626	0.824	0.814*	0.726
No. of substance, psychological, behavioral, or legal problems
0	0.011	0.009	0.012	0.021
1	0.029	0.021	0.019	0.033
2	0.138	0.072	0.084	0.063
3–12	0.822	0.88	0.885	0.883

Notes: MET/CBT5 = five sessions of motivational enhancement therapy plus cognitive behavioral therapy; elig. = eligible; ESS = effective sample size.

^*Standardized mean difference compared with CYT study sample > 0.25. Means were compared only after exclusion criteria were applied to ATM sample.

Before weighting the ATM reduced sample to look like the CYT sample, the two groups differed (i.e., had SMDs ≥ 0.25 / ≤ −0.25) on 31 of 108 client characteristics (note that not all 108 characteristics are included in Tables 1 and 2 and that actual SMD values are not presented in the tables; all 108 characteristics are, however, presented in supplementary materials available on request). The greatest differences included the proportion of time spent in controlled environments in the 90 days before enrollment (SMD = −0.85), with ATM youth spending more time on average in controlled environments, and current regular receipt of drug and alcohol treatment, with 9% of youth in the ATM reporting currently receiving treatment at intake versus 2% of the CYT sample (SMD = −0.58). The ATM sample also had higher rates of past attendance at Alcoholics Anonymous or other self-help groups (SMD = −0.75) and criminal violence (SMD = −0.52). As expected, the CYT had higher rates of youth reporting that marijuana was the drug for which they most needed treatment (SMD = 0.48).

After weighting, most of these differences were attenuated. There were only five pretreatment variables with absolute SMD greater than 0.25 after weighting: reporting health problems (SMD = −0.45), three different indices of mental health problems (SMDs = −0.32, −0.30, and −0.26), and recency of gambling (SMD = 0.25). Thus, these variables were included as covariates in the treatment effect regression models presented below.

Efficacy of Cannabis Youth Treatment study versus Adolescent Treatment Model outpatient programs

Our estimates of the average treatment effect on the treated (ATT) population are presented graphically in Figure 1 (for the dichotomous outcomes) and in Table 3 (for the continuous outcomes). As shown, there was no evidence that youth who received MET/CBT5 would have fared better with respect to the outcomes examined here had they received treatment at an ATM outpatient program. On the contrary, we find that youth who received MET/CBT5 had significantly greater reductions than youth in the propensity-weighted ATM outpatient sample in past-month substance problems (MET/CBT5 change [Δ] = −1.79, ATM Δ = −0.64, difference = −1.15, 95% CI [−2.21, −0.09]), substance fre-quency (MET/CBT5 Δ = −0.07, ATM Δ = −0.02, difference = −0.05, 95% CI [−0.01, −0.09]), and illegal activities (MET/ CBT5 Δ = −0.09, ATM Δ = 0.01, difference = −0.10, 95% CI [−0.15, −0.05]). For recovery and institutionalization as well as changes in emotional problems, there were no significant differences found between the youth receiving MET/CBT5 and the propensity-weighted ATM outpatient sample.

Figure 1.

Five sessions of motivational enhancement therapy plus cognitive behavioral therapy (MET/CBT5) and "exemplary" outpatient programs: 12-month dichotomous outcomes. Y axis represents percentage of persons meeting outcome criteria at 12 months after propensity score weighting.

Table 3.

Five sessions of motivational enhancement therapy plus cognitive behavioral therapy (MET/CBT5) and "exemplary" outpatient programs (Adolescent Treatment Model [ATM]): 12-month change scores

Variable	MET/CBT5	ATM (weighted)	Difference [95% CI]
Past Month Substance	−1.79	−0.64	−1.15
Problem Scale			[−2.21,−0.09]
Substance Frequency	−0.07	−0.02	−0.05
Scale			[−0.09,−0.01]
Emotional Problems	−0.07	−0.03	−0.03
Scale			[−0.08, 0.02]
Illegal Activities	−0.09	0.01	−0.10
Scale			[−0.15,−0.05]

Note: CI = confidence interval.

Discussion

Community-based treatment providers positioned to adopt MET/CBT5 have limited evidence to evaluate whether adopting the treatment will yield outcomes similar to or better than the outcomes their programs currently produce. In the absence of evidence from a randomized, controlled trial comparing MET/CBT5 with community-based treatment, we used quasi-experimental methods to estimate MET/CBT5's efficacy relative to three "exemplary" community-based programs. The evidence suggests that youth who received MET/ CBT5 had significantly better outcomes with respect to substance use and criminal activity than those who received care in the three nonexperimental settings. On the other hand, we found no evidence to suggest that youth who received MET/CBT5 in the CYT study would have fared differently with regard to emotional problems or institutionalization rates. We note that although we did not see a significant difference between the two conditions on the likelihood of recovery, the observed 10% difference favoring MET/CBT5 on recovery rates may be nonsignificant because we only have power (β = .80; α = .05) to detect a 16% difference between conditions as significant for binary measures.

Is public investment in disseminating MET/CBT5 justified? Our results suggest that for youth like those entering the CYT study, which was designed to include youth who would typically be admitted to publicly funded outpatient programs with cannabis use problems (Tims et al., 2002), MET/CBT5 offers care that yields better outcomes than what is offered at exemplary outpatient treatment programs in terms of substance problems and frequency and illegal activities. This is similar to findings by Godley et al. (2010) that a variant of MET/CBT5 performs as well as another noteworthy community-based treatment. However, although the treatment performed well relative to other treatments we examined in this study, these results do not apply to potentially more severely affected persons referred to inpatient or residential treatment. Perhaps more important, these outcomes are not optimal: At 12 months, only a third of the youth in the CYT and ATM samples met recovery criteria and a quarter had been institutionalized in the past 90 days, suggesting significant room to improve the treatments delivered to adolescents with substance use problems.

The relative effectiveness of MET/CBT5 is particularly noteworthy because it is a brief and comparatively inexpensive intervention. As described above, the treatment consists of five sessions designed to be administered over a 6-week period; an episode of treatment was estimated to cost between $1,100 and $1,600 (Dennis et al., 2004; French et al., 2002). This is generally shorter than the community-based programs, which lasted between 2 and 5 months (Battjes et al., 2004; Godley et al., 2004; Stevens et al., 2007). Post hoc analyses confirm that mean adolescent self-reports of time spent in outpatient treatment in the past 90 days were significantly lower among those who received MET/CBT5 in the CYT study than the weighted ATM sample. Although these differences were significant at 3-, 6-, and 12-month assessments, the greatest differences existed at the 3-month assessment and likely reflect the scheduled treatment differences.

Although our analytic strategy succeeded in identifying a comparison group of youth in community-based treatment with more than 100 diverse pretreatment characteristics closely matched to those from the CYT study, the possibility remains that some important but unobserved group differences exist, and that these differences, not treatment differences, explained the differences in group outcomes. Moreover, we cannot rule out the possibility that observed differences in outcomes resulted not from the specific differences in the treatments provided but rather from the differences in the contexts in which the treatments were provided. That is, in contrast to community treatment settings, the CYT study was conducted as an experiment, with resources that might be uncommon in community settings. There are differences between experimental and real-world conditions in which adolescents are treated for behavioral problems. Our adjustment strategy attempts to control for differences related to observable client characteristics, but there likely are differences in characteristics of therapists (e.g., training, caseload) and in treatment settings (e.g., intensive supervision of counselors by researchers) that could affect client outcomes (Weisz et al., 1995).

This study helps to fill the gap in data on the efficacy of widely used approaches to treating adolescents with alcohol and drug problems. Rigorous assessment of the relative efficacy of alternative treatment approaches can be expensive and time consuming to obtain; in addition, ethical concerns may bar researchers from using an untreated control group (Dennis et al., 2004). Thus, our strategy for leveraging existing outcomes data to generate evidence regarding the relative efficacy of a widely used therapy answers recent calls for increased public investment in the generation of comparative effectiveness research to guide program development and financing decisions (Tunis et al., 2003).

Recent advances in causal modeling approaches for evaluating treatment efficacy from observational studies can provide policymakers with valuable insights into problems that otherwise lack rigorous empirical support. Nevertheless, these methods, like the propensity score weighting approach used in this study, can be data intensive. This is because the approach requires the uniform administration of clinical assessments in multiple treatment settings over extended periods of time. The present study can therefore be viewed as a return on SAMHSA's substantial long-term investment in the development and implementation of standardized assessment tools and follow-up data collection. As of December 2009, more than 200 Center for Substance Abuse Treatment grantees had administered the GAIN and approximately 75 more were scheduled to be collected in 2010, representing a substantial investment in federal dollars.

Although the advantages of our approach are considerable, there are also important limitations to consider. Perhaps most important, the outpatient programs that comprise the nonexperimental control condition were not selected to be representative of community-based treatment providers in the United States. It is likely that some programs in the United States perform better or worse than those included in this study. Had their outcomes been used, the efficacy of MET/ CBT5 could have been found to be substantially worse or better than the ATM programs. In addition, data from the ATM study started to be collected in 1998–1999, and the offerings at community-based programs may have changed since that time.

Our approach for creating propensity score weights also has statistical limitations. Although a nonparametric approach offers the ability to control for a number of pretreat-ment covariates and permutations of these variables, it, too, may be imperfect and pretreatment differences between the groups on unmeasured variables may still exist. This includes differences in constructs not included in our model and un-correlated with those that are included as well as those we purport to adjust for but which may be subject to measurement error.

In summary, we combined clinical trial data with observational data on standard treatment to assess the efficacy of MET/CBT5 by using propensity score methods to simulate a balanced, usual-care control group. We found that MET/ CBT5 performed better with regard to substance problems, substance use frequency, and criminal behavior. This analysis provides evidence that for these three outcomes, MET/ CBT5 performs better than what is considered exemplary community-based care. For the three other outcomes (recovery, institutionalization, and emotional problems) there was no evidence that MET/CBT5 and the exemplary programs differed. Although the observed 12-month outcomes remain far from ideal, its relatively low cost makes MET/CBT5 a promising option for treating adolescents with cannabis-related disorders compared with more intensive, longer-term approaches.