The Comparative Effectiveness of Outpatient Treatment for Adolescent Substance Abuse: A Meta-Analysis

Abstract

Meta-analysis was used to synthesize research on the effects of outpatient treatment on substance use outcomes for adolescents with substance use disorders. An extensive literature search located 45 eligible experimental or quasi-experimental studies reporting 73 treatment-comparison group pairs, with many of the comparison groups also receiving some treatment. The first analysis examined 250 effect sizes for the substance use outcomes of adolescents receiving different types of treatment relative to the respective comparison groups. As a category, family therapy programs were found to be more effective than their comparison conditions, whereas no treatment programs were less effective. However, not all treatment types were compared with each other in the available research, making it difficult to assess the comparative effectiveness of the different treatments. To provide a more differentiated picture of the relative improvement in substance use outcomes for different treatments, a second analysis examined 311 pre-post effect sizes measuring changes in substance use for adolescents in the separate treatment and comparison arms of the studies. The adolescents in almost all types of treatment showed reductions in substance use. The greatest improvements were found for family therapy and mixed and group counseling. Longer treatment duration was associated with smaller improvements, but other treatment characteristics and participant characteristics had little relationship to the pre-post changes in substance use. Based on these findings family therapy is the treatment with the strongest evidence of comparative effectiveness, although most types of treatment appear to be beneficial in helping adolescents reduce their substance use.

Recent estimates from the Treatment Episode Data Set indicate that approximately 11% of the 1.8 million substance abuse treatment admissions in 2007 were for adolescents under age 20, the majority of whom presented with marijuana/hashish as the primary substance abused (SAMHSA, 2009). With such large numbers of adolescents in substance abuse treatment, it is important to know whether such treatments are effective and, if they are not all equally effective, which are most effective. The study reported here uses meta-analysis to investigate the findings of experimental and quasi-experimental studies that compare different outpatient substance abuse treatments for adolescents with control conditions or with each other. In particular, it examinestheir collective findings about (a) the comparative effectiveness of different types of treatment, (b) the magnitude of change in adolescents’ substance use after they enter treatment, and (c) the relationship of various method, participant, and treatment characteristics to those changes in substance use.

Research on Treatment Effectiveness for Adolescents

Adolescents with substance use disorders differ from their adult counterparts in several ways and thus may have different treatment needs (Winters, 1999). For instance, adolescents may be more susceptible to peer influences, be more vulnerable to adverse effects from substances because of smaller body size and lower tolerance levels, and, because of their developmental stage, experience greater long-term cognitive and emotional damage from substance abuse (Brown, Tapert, Granholm, & Delis, 2000; Tapert, Caldwell, & Burke, 2001; Winters, 1999). That the causes and consequences of substance use disorders may differ for adolescents and adultsimplies that evidence regarding the effectiveness of treatment for adolescents should be based on research conducted with adolescents, not inferred from research with adults.

The number of studies focusing specifically on the effectiveness of adolescent substance abuse treatment has burgeoned over the last decade (Dennis et al., 2004; Waldron & Turner, 2008; Williams & Chang, 2000). A few previous attempts have been made to systematically summarize the results of this research in order to assess the extent to which treatment for adolescents is effective and which types of treatment are most effective. Some of these reviews have included studies of both adults and adolescents, leaving ambiguity about whether the results can be applied specifically to adolescents. One of the earliest meta-analyses of alcohol treatment studies, for instance, found brief interventions and motivational enhancement programs to be most effective, but did not differentiate effects for adults and adolescents (Miller et al., 1995).

Other meta-analyses that combined results for adults and adolescents examined only one or two specific treatment types and therefore had limited scope, though they reported beneficial effects for the treatments reviewed. One such review, for instance, found family therapy to be more efficacious than non-family therapies (Stanton & Shadish, 1997). Another found that methadone maintenance treatment reduced opiate use and criminal activities (Marsch, 1998). And still another found that motivational interviewing reduced alcohol use and drug addiction (Burke, Arkowitz, & Menchola, 2003). While these reviews are informative, whether the same conclusions would apply specifically to adolescents with substance use disorders is unclear.

Among the reviews focused exclusively on treatment for adolescents, some were traditional narrative reviews that did not characterize treatment effects in a way that allowed them to be compared. Deas and Thomas (2001), Waldron (1997), Weinberg et al., (1998), and Williams & Chang (2000), for example, reviewed the efficacy of adolescent substance abuse treatment but did not quantify the outcomes associated with different treatment modalities or consider differences associated with methodological, participant, or treatment characteristics. Other narrative reviews were restricted to randomized trials and did not consider whether strong quasi-experimental studies provided useful evidence (e.g., Ozechowski & Liddle, 2000; Waldron, 1997) or focused on a specific treatment such as pharmacotherapy, family therapy, or cognitive behavioral therapy (e.g., Waldron, 1997; Waldron & Kaminer, 2004; Waxmonsky & Wilens, 2005).

The only relatively comprehensive systematic review specifically of outpatient adolescent substance abuse treatment effectiveness to date is the meta-analysis conducted by Waldron and Turner (2008) of findings from 17 randomized clinical trials and the 46 treatment conditions embedded within them. Their results showed generally beneficial treatment effects that were especially positive for Multidimensional Family Therapy, Functional Family Therapy, and cognitive behavioral therapy. Analysis of treatment differences, however, was restricted to these three treatment modalities. Further, by focusing only on randomized trials, this meta-analysis excluded the larger body of studies using quasi-experimental research designs. Though the latter must be interpreted with care because of their methodological vulnerabilities, they may nonetheless provide useful information about the nature and range of treatment effects that adds to that available from the limited number of randomized studies.

The Current Study

In an attempt to more fully synthesize the adolescent substance abuse treatment effectiveness literature, the current study addresses three broad research questions. First, what is the comparative effectiveness of different types of outpatient treatment for adolescents with substance use disorders? Second, what is the magnitude of change in substance use for such adolescents after entry into outpatient treatment programs? Third, what differences between the characteristics of the participant samples, the treatment programs, and the study methods are related to those changes in substance use? We address these research questions through a meta-analysis of the findings of experimental and quasi-experimental studies comparing substance use outcomes after receipt of treatment for adolescents with clinical levels of substance abuse.

This meta-analysis expands and extends prior research in several important ways. First, an attempt was made to be comprehensive across all available studies judged capable of providing useful information about comparative treatment effects and across all treatment modalities. In addition, an extensive coding scheme was applied that extracted a great deal of information from the study reports about the research methods, outcome measures, participant sample characteristics, and treatment characteristics. Furthermore, newly developed meta-analysis techniques were applied to address the complexity and diversity of the substance use outcomes reported in these studies. Multiple outcomes are often reported that represent different substances (e.g., alcohol, marijuana, or other drugs) measured in various ways (e.g., abstinence, 30 day use, frequency of use, and problems associated with use). Within any participant sample, these multiple outcomes are not statistically independent and, as such, create problems if analyzed together. Rather than eliminate informative outcome data, however, we have retained all the substance use outcomes from each study in the analyses and applied new techniques for handling their statistical dependencies (Hedges, Tipton, & Johnson, 2010).

A further complication is that relatively few studies compare a specific treatment with a relatively neutral no-treatment or placebo control condition. For understandable reasons, neither researchers nor service providers find it acceptable to allocate adolescents presenting with substance use disorders to presumptively ineffective control conditions. Instead, most studies compare a focal treatment—the one of primary interest to the researcher—with an alternate treatment, though that may only be relatively unspecified “practice as usual.” In such comparisons, any differences in outcomes are not only a function of the effectiveness of the focal treatment but also of that of the comparison treatment. Because the latter varies from study to study, the comparative effectiveness of the different treatments examined in different studies is difficult to assess.

We addressed this situation in two different ways. First, we analyzed effect sizes representing differences in substance use outcomes for adolescents in different treatment-comparison combinations while attempting to statistically control for characteristics of the methods, participant samples, and general features of the treatments in the respective conditions.Meta-regression models were used to create effect sizes that were adjusted for pretest-posttest time interval, attrition rate, substance use outcome type, baseline substance use severity, gender mix of the participant sample, race/ethnicity mix of the participant sample, and mean age of the participant sample. The goal of this first approach was therefore to compare different treatment-comparison combinations net of the potential confounding characteristics of the samples and studies. In the second approach, we analyzed the pre-post changes in substance use for adolescents in each separate arm of the comparisons reported in the studies while again controlling for factors other than type of treatment, such as participant characteristics, that might influence those changes. We then compared the resulting estimates of the magnitude of substance use reduction across the different types of treatment.

Methods

Eligibility Criteria, Search Strategy, and Studies Included

The data used in this meta-analysis were generated from published and unpublished research reports meeting predefined inclusion criteria. Studies were required to involve an explicit identifiable substance abuse treatment with the aim of reducing, remediating, or eliminating substance use or substance related problems. The treatment had to be delivered on an outpatient basis to adolescent participants 12-20 years old who met DSM criteria for substance abuse or dependence or the equivalent. We chose to focus solely on outpatient programs given the variability in participant populations, treatment modalities, and treatment intensity between outpatient and residential programs. Results for at least 10 participants per condition on at least one posttest measure of substance use must have been reported with statistics that allowed estimation of an effect size. Eligible research designs included those using random assignment to treatment conditions and nonrandomized comparison studies that employed matching or statistical controls on baseline substance use or risk variables. The treatment conditions compared could include no treatment, placebo treatment, general practice as usual, or two distinct treatments. Finally, to be eligible, studies had to be reported in English in 1980 or after.

Studies were identified from a variety of sources including electronic databases such as Dissertation Abstracts International, ERIC, NCJRS, ProQuest, PsycINFO, PubMed, Social Services Abstracts, and Sociological Abstracts in 2008. Search terms included keywords such as adolescent substance treatment, program evaluation, assessment, the names of specific treatment modalities (e.g., motivational enhancement therapy), and the names of specific types of substances (e.g., alcohol, marijuana, cocaine). Hand-searches were also conducted on the conference proceedings of the College of Problems on Drug Dependence and the Joint Meeting on Adolescent Treatment Effectiveness. Reference lists from retrieved studies were reviewed for potentially eligible studies as well as the references in prior literature reviews and meta-analyses. Further attempts to identify unpublished or overlooked studies were made by contacting researchers in the field of adolescent substance abuse treatment, attendees at the Joint Meeting on Adolescent Treatment Effectiveness, and current and prior CSAT, NIAAA, and NIDA grantees. Eligible study reports were coded on over 500 items related to general study context (e.g., date and country), study methods, treatment characteristics, sample characteristics, outcome measures, and effect sizes. The coding team conducting the search, coding, and construction of the final database was experienced with meta-analysis and followed the general procedures described in reports of other similar meta-analyses (e.g., Wilson et al., 2003).

This extensive literature search yielded a total of 45 eligible published and unpublished studies reported from 1981 through 2008, nearly all of which used a random assignment design. All of the studies compared outcomes for adolescents in a given treatment program (e.g., family therapy) with those for adolescents in a comparison condition, most often an alternative treatment of some sort. In some cases the comparison condition was presumed to be inferior to the focal treatment that was the main interest of the study; in other cases the study compared two treatment programs with neither hypothesized to be inherently inferior to the other.

Because some studies had three or more treatment conditions or groups being compared, group comparison effect sizes were available for 73 different treatment-comparison group pairs. These 73 treatment-comparison pairs were all unique combinations of experimental groups, but they were not independent because some pairs included the same comparison group arm. For instance, one study may have contributed three unique pairs based on three treatment conditions: family therapy versus control condition, cognitive behavioral therapy versus control condition, and family therapy versus cognitive behavioral therapy. The substance use outcomes reported for the 73 treatment-comparison combinations were coded into 250 standardized mean difference effect sizes representing post-treatment differences in substance use between the conditions compared (102 for alcohol, 40 for marijuana/cannabis, 81 for mixed substance use, and 27 for specific substances other than alcohol or marijuana, e.g., cocaine or heroin).

In addition to the posttest differences between treatment and comparison conditions, many of the eligible studies also provided information about change in substance use over time for the adolescents in each condition. Among the 45 studies contributing group difference effect sizes, 44 also reported sufficient information at pretest to allow calculation of pre-post change in substance use separately for each group arm (pretest-posttest change in the treatment group, pretest-posttest change in the control group, etc.). Those 44 studies provided pretest-posttest change information for 79 unique treatment or comparison conditions that generated 311 pre-post effect sizes representing pre-treatment to post-treatment changes in substance use within one of those conditions (139 for alcohol, 40 for marijuana/cannabis, 105 for mixed substance use, and 27 for other specific substances).

Statistical Procedures

Standardized mean difference effect sizes (d) for the substance use outcomes were calculated as the difference between the posttest means for the treatment and comparison conditions divided by the pooled standard deviation (Lipsey & Wilson, 2001):

$$d=\frac{{\overline{X}}_T-{\overline{X}}_C}{S_{pooled}}$$

Pre-post mean change effect sizes for substance use outcomes were similarly calculated as the difference between the posttest and pretest means divided by the pooled standard deviations. When means and standard deviations were not reported, effect sizes were calculated when possible from other statistics that were reported as outlined in Lipsey & Wilson (2001). Cox transformations were used to estimate the standardized mean difference effect sizes for dichotomous outcomes, as described by Sanchez-Meca and colleagues (2003). All effect sizes were given algebraic signs such that positive values indicated better results (i.e., lower substance use) than the comparison group for whichever treatment group was designated as the focal one in a given analysis, or better results at post-treatment than pre-treatment.

To account for any bias associated with small samples, all effect sizes were adjusted with the small-sample correction factor to produce unbiased estimates (Hedges g; Hedges, 1981). The small sample corrected effect sizes and their standard errors were calculated as follows where n_T and n_C are the respective sample sizes of the focal treatment and comparison groups:

$$g=\left[1-\left(\frac{3}{4(n_T+n_C)-9}\right)\right]\ast d$$

$$S{E}_g=\sqrt{\frac{n_T+n_C}{n_T{n}_C}+\frac{g^2}{2(n_T+n_C)}}$$

All eligible effect sizes were included in each analysis, which, in most cases, meant multiple effect sizes from the same participant sample and, in some cases, effect sizes that shared a comparison group (e.g., when three conditions were compared pairwise with each other). The associated statistical dependencies were handled in each analysis by first grouping the effect sizes into clusters with dependent effect sizes within clusters and independence between clusters. Group difference effects sizes were clustered within pairs of treatment and comparison conditions. Pre-post effect sizes were clustered within the study arms contributing those effect sizes. To account for the within cluster statistical dependencies that resulted from this procedure, all analyses used the random effects robust standard error estimation technique developed by Hedges, Tipton, and Johnson (2010). The robust standard error technique requires that an estimate of the mean correlation (ρ) between all the pairs of effect sizes within a cluster be estimated for calculation of the between-study sampling variance estimate, τ². In all analyses, we estimated τ² with ρ = .80; sensitivity analyses showed that the findings were robust across different reasonable estimates of ρ.

All analyses were weighted using inverse variance weights so that the contribution of each effect size was proportionate to its statistical precision, which largely reflects the size of the sample on which it is based (Hedges & Olkin, 1985; Lipsey & Wilson, 2001). As suggested by Hedges and colleagues (2010), we used a conservative approach in calculating the weights by assuming an overall weight for each cluster of dependent effect sizes based on the sample size of the study, then dividing that weight across however many effect sizes were in that cluster. Thus the weight for each effect size i within each cluster j was calculated as:

$$w_{ij}=\frac{1}{k(\overline{S}{\overline{E}}_g^2+{\tau }^2)}$$

wherek is the number of effect sizes per cluster, $\overline{S}{\overline{E}}_g^2$ is the mean sampling variance for the effect sizes in a cluster, and τ² is the estimate of the sampling variance across all the effect sizes from all of the studies. This weighting function thus reflects the size of the sample on which each effect size was based, but required that each effect size from that sample be weighted only according to its proportional share among the multiple other effect sizes in that same cluster. This procedure is approximately equivalent to averaging all the effect sizes from the same cluster and applying the inverse variance weight calculated from the number of cases in the cluster to that mean effect size, except it keeps the effect sizes separate and allows the individual characteristics of each to be represented in the analysis.¹

An examination of the box plots for the effect size distributions identified a small number of effect size and study sample size outliers with the potential to distort the analysis. These were recoded to the corresponding lower or upper fence values (Tukey, 1977) to ensure that they did not exercise a disproportionate influence on the analysis results. Finally, a small number of missing values on method, participant, or treatment variables used in the final analyses were imputed using the expectation-maximization (EM) algorithm in SPSS (Graham, Cumsille, & Elek-Fisk, 2003).

Results

Study Characteristics

Table 1 summarizes the method, participant, and treatment characteristics of the studies contributing group difference effect sizes for the first meta-analysis. The number of different treatment-comparison group pairs represented in this data set (k) was 73; the number of substance use effect sizes (n) was 250. Most of the comparison group pairs (84%) were reported in journal articles, with an average publication date in 2001, although 37% were published prior to 2000. Assignment to conditions in virtually all of these treatment-comparison pairs (99%) was random. The mean pretest to posttest attrition rate was .17, although approximately 13% of the experimental groups showed attrition rates of .40 or more.²

Table 1. Descriptive Statistics for Characteristics of Studies Contributing Group Difference Effect Sizes (k = 73; n = 250).

	Mean	SD a	Range
Method Characteristics
Publication year	2001	5.8	1981 - 2008
Conducted in the U.S. (1=yes)	.90	.30	0 - 1
Journal publication (1=yes)	.84	.37	0 - 1
Randomized design (1=yes)	.99	.12	0 - 1
Average attrition rate	.17	.17	0 - .70
Established survey instrument (1=yes)b	.61	.49	0 - 1
Time span of outcome measure (days)b	64.9	45.3	28 - 267
Pretest-posttest interval (days) b	157.3	147.3	28 - 1456
Effect size based on means/SDs (1=yes)b	.64	.48	0 - 1
Pretest group difference effect sizebc	.04	.48	−1.18 - 2.57
Group equivalence effect sizec	.27	.29	−.40 - 1.23
Participant Sample Characteristics
Percent male	67.8	17.6	0 - 90
Percent white	60.8	26.9	0 - 100
Age (years)	16.5	1.56	14 - 20
Clinical comorbidity (1=yes)	.47	.50	0 - 1
Substance severity factor score	1.70	.61	1 - 3
Delinquency level/police contact	2.55	1.32	1 - 5
Referred from prior treatment (1=yes)	.32	.47	0 - 1
Treatment Characteristics
Delivered in group format (1=yes)	.32	.47	0 - 1
Level of family involvement	1.57	.64	1 - 3
Implementation quality	.24	.59	−1.22 - 1.06
Duration (days)	76.4	67.5	1 - 380
Frequency of treatment contact	2.80	1.29	1 - 6

Note:

^aSD = standard deviation

^bEstimates calculated at the effect size level (n = 250); all others calculated for the k=73 group comparisons.

^cMean estimates weighted using inverse variance weights.

Sixty-one percent of the effect sizes were based on outcome measures from established survey instruments (e.g., Global Appraisal of Individual Needs), and the mean time span covered by the outcome measure was substance use in the past 65 days. The mean pretest-posttest interval was about 157 days. The mean effect size comparing treatment and comparison conditions on pretest substance use was .04. The mean composite group equivalence effect size, an average measure of baseline group equivalence on risk factors other than the pretest measure (e.g., gender, age, race, delinquency, other substance use measures), was .27. The mean values on both the pretest and group equivalence effect sizes would be expected to be close to zero given that virtually all studies used randomization to try to produce initial equivalence between the groups compared. Their positive values, especially on the composite group equivalence variable, indicate that, on average, the baseline measures slightly favored the focal treatment groups over their respective comparison groups.

In terms of the characteristics of the participants, most of the samples in the treatment-comparison pairs were predominantly male (68%), white (61%), and had an average age of 16. Nearly half (47%) of these samples included adolescents with clinical levels of psychiatric comorbidity (e.g., oppositional defiant disorder, major depressive disorder). The mean level of baseline substance use severity was a moderate 1.70 measured as an average scale score on three items ranging from 1 (low) to 3 (high) for baseline alcohol, marijuana, and mixed substance use severity respectively (Cronbach’s α =.81). The delinquency levels of the adolescent samples ranged from 1 (most adolescents had no police contact) to 5 (correctional institutionalization), with the average of 2.6 indicating samples where most of the adolescents had at least some arrest or police contact history. Approximately 32% of the samples were referred to treatment from prior treatment.

Regarding the characteristics of the treatments represented in these treatment-comparison pairs, 32% were provided in group settings (versus individual settings) and many included at least some level of family involvement (this measure ranged from 1=family never present to 3=family always present). Implementation quality, with a mean of .24, was measured as an standardized scale score ranging from −1.2 to 1.1 that was based on four items: explicitly manualized treatment, treatment with a standard script or protocol, researcher (versus practitioner) providing treatment, and treatment set up for research purposes (versus routine practice) (Cronbach’s α =.70). The average treatment duration was 76 days. The frequency of treatment contact was rated on an ordinal scale ranging from 1 (little to no contact) to 6 (almost continuous contact), and the average program had contact with participants a couple of times per week.

The First Analysis: Comparing Treatment Effects

Though most of the treatment conditions included multiple treatment elements, each was coded according to the primary type of treatment provided to participants (see Appendix A for a description of the treatment type categories). Within those categories, some treatments were additionally coded to identify specific named treatments, e.g., ACRA, MDFT, MET/CBT-7. Overall, the most prevalent treatment types were family therapy, motivational enhancement therapy/motivational interviewing (MET), psychoeducational therapy (PET), and cognitive behavioral therapy (CBT). The various types of treatment were compared with each other and with control conditions in many different combinations. For instance, 14 studies compared cognitive behavioral therapy (CBT) to some other condition. However, only two of those involved a no treatment control group; the others compared CBT to another type of treatment, e.g., family therapy, MET/CBT, and so forth. (Table 2, discussed in more detail later, shows the number of studies and effect sizes available for each comparison).³

Table 2. Mean Covariate Adjusted Posttest Effect Sizes and 95% Confidence Intervals for Each Treatment Category versus Available Comparison Conditions.

Treatment Combination	k	n	Mean	95% CI
Behavioral Therapy
vs. CBT	1	1	−.67	(−1.35, .02)
vs. Family	1	2	−.01	(−5.56, 5.53)
vs. MET/CBT	1	2	.36	(−1.45, 2.16)
vs. Practice as usual	1	6	−.01	(−.45, .43)
vs. All of the above	4	11	.00	(−.52, .53)
Family Therapy
vs. Behavioral	1	2	.01	(−5.53, 5.56)
vs. CBT	3	12	.53	(−.30, 1.36)
vs. Group and mixed counseling	7	24	.32*	(.18, .47)
vs. MET/CBT	5	10	.11	(−.32, .55)
vs. PET	5	14	.45*	(.02, .88)
vs. Practice as usual	4	26	.09	(−.27, .46)
vs. All of the above	25	88	.26*	(.13, .38)
Group and Mixed Counseling
vs. CBT	1	4	−.62	(−2.21, .96)
vs. Family	7	24	−.32*	(−.47, −.18)
vs. MET	1	8	−.45	(−1.26, .36)
vs. MET/CBT	5	5	.20	(−.14, .55)
vs. PET	3	4	−.16	(−.36, .04)
vs. No treatment	1	25	−.40	(−1.38, .58)
vs. All of the above	18	70	−.10	(−.28, .07)
vs. All treatments (excluding no treatment)	17	45	−.10	(−.28, .08)
Cognitive Behavioral Therapy (CBT)
vs. Behavioral	1	1	.67	(−.02, 1.35)
vs. Family	3	12	−.53	(−1.36, .30)
vs. Group and mixed counseling	1	4	.62	(−.96, 2.21)
vs. MET/CBT	2	6	−.59	(−3.07, 1.88)
vs. PET	3	7	.16	(−.74, 1.05)
vs. Practice as usual	2	10	.51	(−2.25, 3.27)
vs. No treatment	2	3	.49	(−3.62, 4.59)
vs. All of the above	14	43	−.02	(−.35, .31)
vs. All treatments (excluding no treatment)	12	40	−.07	(−.43, .28)
Motivational Enhancement Therapy (MET)
vs. Group and mixed counseling	1	8	.45	(−.36, 1.26)
vs. PET	5	20	.01	(−.33, .36)
vs. Skills	1	2	.22	(−2.45, 2.89)
vs. Practice as usual	1	5	.18	(−.69, 1.05)
vs. No treatment	9	35	.23*	(.13, .32)
vs. All of the above	17	70	.17*	(.07, .27)
vs. All treatments (excluding no treatment)	8	35	.11	(−.11, .32)
MET/CBT
vs. Behavioral	1	2	−.36	(−2.16, 1.45)
vs. CBT	2	6	.59	(−1.88, 3.07)
vs. Family	5	10	−.11	(−.55, .32)
vs. Group and mixed counseling	5	5	−.20	(−.55, .14)
vs. PET	1	2	.33	(−2.96, 3.62)
vs. All of the above	14	25	−.07	(−.29, .14)
Psychoeducational Therapy (PET)
vs. CBT	3	7	−.16	(−1.05, .74)
vs. Family	5	14	−.45*	(−.88, −.02)
vs. Group and mixed counseling	3	4	.16	(−.04, .36)
vs. MET	5	20	−.01	(−.36, .33)
vs. MET/CBT	1	2	−.33	(−3.62, 2.96)
vs. No treatment	2	8	.00	(−1.25, 1.24)
vs. All of the above	19	55	−.15	(−.32, .01)
vs. All treatments (excluding no treatment)	17	47	−.16	(−.34, .01)
Pharmacological Treatment
vs. Placebo	7	13	.30	(−.19, .79)
Skills Training
vs. MET	1	2	−.22	(−2.89, 2.45)
Practice as Usual
vs. Behavioral	1	6	.01	(−.43, .45)
vs. CBT	2	10	−.51	(−3.27, 2.25)
vs. Family	4	26	−.09	(−.46, .27)
vs. MET	1	5	−.18	(−1.05, .69)
vs. All of the abovea	7	43	−.13	(−.35, .09)
No Treatment
vs. CBT	2	3	−.49	(−4.59, 3.62)
vs. Group and mixed counseling	1	25	.40	(−.58, 1.38)
vs. MET	9	35	−.23*	(−.32, −.13)
vs. PET	2	8	.00	(−1.24, 1.25)
vs. All of the above	14	71	−.20*	(−.31, −.10)

Notes: k = number of treatment-comparison group pairs; n = number of effect sizes. All estimates adjusted for baseline group equivalence and pretest differences, pretest-posttest time interval, attrition rate, substance use outcome type (alcohol, marijuana, other drugs), baseline substance use severity, gender mix, race/ethnicity mix, and mean age of the participant sample.

^aTotals do not sum to 43 and 7 because one treatment comparison group pair contributed 4 effect sizes that compared a combined CBT and family therapy program with practice as usual. That k = 1, n = 4 case is only represented once in the “all of the above” category.

^*p< .05

However, every treatment type was not compared with every other treatment type and control condition. Numerous treatment-comparison combinations did not appear in this research literature (e.g., behavioral therapy vs. MET) and thus direct comparisons between many treatment types could not be made. A further complication is that many studies compared different versions of similar treatments with each other (e.g., a motivational interviewing intervention compared to another motivational interviewing intervention that included mailed feedback). Many of these effect sizes were, not surprisingly, very small and often close to zero. In other cases, they were not so small but the treatment variants they compared were not of general interest. Comparisons of such very similar treatments, therefore, were not included in the analysis and are not represented in any of the tables showing results of those analyses.

One rapidly developing meta-analysis technique for extracting the greatest amount of comparative effectiveness information possible from a limited set of pairwise comparisons such as those in Table 2 is to include indirect estimates along with the direct ones (Hoaglin et al., 2011; Jansen et al., 2011). For instance, if Treatment A is compared with Treatment B in one set of studies and with Treatment C in another set, then the difference between the AB and AC effect sizes provides an indirect estimate of the unobserved BC effect sizes. For this technique to be used, it is necessary to assume that the indirect estimates of the BC effect sizes are consistent with the direct estimates. This consistency assumption can be checked by comparing the indirect estimates with the corresponding direct ones when the latter are available. For the studies in this meta-analysis, we found very poor consistency in preliminary analysis when we conducted that check, even when using the covariate-adjusted effect sizes described below. We have, therefore, included no indirect effect size estimates in our analyses.

To estimate the comparative effectiveness of different adolescent substance abuse treatment types from the direct comparisons available, a meta-regression model using robust standard errors was first fit to the effect sizes. The results of this analysis were then used to adjust the posttest group difference effect sizes for the potentially confounding effects of differences across studies on key methodological and sample characteristics. These covariate adjustments held all effect sizes at zero for the baseline group equivalence and pretest differences, and held them at the mean values across all studies for (a) pretest-posttest time interval, (b) attrition rate, (c) substance use outcome type (alcohol, marijuana, other drugs), (d) baseline substance use severity, (e) gender mix of the participant sample, (f) race/ethnicity mix, and (g) mean age of the participant sample.

Because a given treatment could be the focal treatment condition for some comparisons and the comparison condition for others, the meta-regression model used to create the covariate-adjusted effect sizes predicted the absolute value of the group difference posttest effect size. This procedure eliminated the original directionality of the effect sizes, so the meta-regression model also included a dummy variable indicating whether the original posttest effect size was positive. The group equivalence, pretest difference, and attrition rate covariates were then given directional signs so that positive values represented enhancing influences on the effect size and negative values represented diminishing influences. For instance, if the pretest difference on a particular substance use outcome favored the designated focal treatment, it was expected to enhance the absolute value of the corresponding posttest effect size in that instance and the pretest difference covariate was coded as a positive value for the meta-regression. If the pretest difference and original posttest effect size had opposite signs, the pretest difference would act to decrease the absolute value of the corresponding posttest effect size and the pretest covariate was coded as a negative value in that instance. The covariate adjusted effect sizes were then created by adding the residuals from this meta-regression model to a constant value calculated as the predicted value for each treatment type comparison holding the baseline group equivalence and pretest difference effect sizes at zero and all other covariates in the model at their mean values.

Table 2 shows the random effects covariate-adjusted mean posttest effect sizes for each treatment type in turn versus the other treatment or control conditions with which it was paired in the available studies. Positive mean effect sizes indicate that the designated treatment type exhibited, on average, better outcomes than the comparison treatment type; negative mean effect sizes indicate it had worse outcomes. Note that the effect sizes for the identified treatment types shown in the table are not mutually exclusive. Each treatment type was also represented as a comparison condition in a different mean effect size estimate. So, for instance, the one effect size comparing behavioral therapy to CBT is also represented (with opposite sign) as the effect size comparing CBT to behavioral therapy.

Table 2 also shows the 95% confidence intervals for each mean effect size. Those confidence intervals are generally quite wide because of the small number of unique treatment-comparison combinations in most instances. Any consideration of the mean effect sizes in Table 2, therefore, should recognize that even widely divergent mean values for different treatment-comparison pairs may have confidence intervals that overlap zero and each other. For instance, CBT, on average, had worse outcomes than family therapy (mean effect size of −.53), but better outcomes than practice as usual (mean effect size of .51). However, neither of those mean effect sizes was significantly different from zero.

Thus although Table 2 shows that some treatment types tended, on average, to show somewhat larger, smaller, or about the same effects as the aggregate of all the treatment conditions with which they were compared, most of those mean effect sizes were not statistically significant. Among the few exceptions was family therapy, which showed a positive (and often statistically significant) mean effect size across all the comparisons in which it was involved. Also, the mean effect size for MET vs. the other conditions with which it was compared was always positive and often statistically significant. More than half of the MET studies, however, compared outcomes with no-treatment control conditions rather than with other treatments. Only the no-treatment control conditions exhibited a mean effect size for contrasts with all other conditions with which they were compared that was significantly negative, indicating worse outcomes on average.

The small number of studies available for each comparison and the associated low statistical power for reliably detecting substance use outcome differences between different treatment conditions allows little differentiation of more and less effective types of treatment. Examining the direction and magnitude of the mean effect sizes for the different comparisons, however, does reveal a general pattern that is more easily seen graphically than in Table 2. Figure 1 displays the treatment types with arrows connecting those compared in the available research. Each arrow points to the treatment type with the better outcomes in that comparison. The treatment types are arrayed so that those favored in fewer comparisons are farther to the left while those favored in more comparisons are further to the right. The thickness of the arrows is proportionate to a composite indicator of the magnitude of the respective effect sizes and the number of studies on which they are based. The thickest arrows represent the relationships with the largest effect sizes and the most studies while the thinnest ones represent the relationships with the smallest effect sizes and the fewest studies.

Figure 1. Comparisons Between Different Treatment Types.

Notes: The arrows point to the treatment type with the more positive outcomes in each comparison. The thickness of each arrow indicates the magnitude of the mean effect size and the number of studies on which it is based, each equally weighted. The treatment types are arrayed from left to right with those to the right generally showing larger positive effects than those to the left with which they are compared.

As Figure 1 illustrates, the treatment types can be roughly divided into four groups:

1. No-treatment and placebo control conditions. These control conditions are presumptively less effective than any treatment condition and show effect sizes consistent with that assumption in all comparisons but one (no-treatment vs. group/mixed counseling). Furthermore, the mean effect size for all the treatment types compared with a no-treatment condition is statistically significant, giving support to the view that most of the treatment types produce better outcomes than no treatment.

2. PET, group/mixed counseling, and practice as usual. The outcomes of these treatments compare unfavorably with almost every treatment with which they are compared. They may be more effective than no-treatment control conditions, but the evidence for that is rather limited.

3. CBT, MET/CBT, MET, Behavioral therapy, and Pharmacological treatment. CBT shows better outcomes than any of the many treatment types in Groups 1 and 2 above with which it has been compared. The pattern of mean effect sizes for comparisons between CBT, MET/CBT, and behavioral therapy are inconsistent in a way that does not allow them to be easily differentiated. MET has not been directly compared with any of those three treatments, but shows favorable outcomes relative to the treatments in Group 2 and no-treatment controls that are roughly similar to those of CBT. Moreover, the mean effect size for MET vs. all the conditions with which it has been compared is statistically significant, though much of that advantage comes from comparisons with no-treatment controls. Pharmacological treatment has only been compared with placebo control conditions, so little can be said about its relative effectiveness in comparison to the other treatments in this group. The magnitude of its mean effect size for those placebo comparisons, however, suggests that its effects may be similar to those other treatments.

4. Family therapy. Family therapy compares favorably with every treatment with which it has been compared, including the alternate treatment types in Group 3 above. Furthermore, the overall mean effect size of .26 for those comparisons is statistically significant. An effect size of that magnitude can be better understood in terms of the reduction in substance use it represents. For instance, a common measure of substance use outcomes in these studies is the number of days an adolescent used marijuana in the past month, based on the Timeline Follow back (TLFB). An effect size of .26 on that measure represents a reduction from an average of 10 days in the past month to 6 days in the past month. Although this may be a modest substantive impact, it still equates to an almost 40% reduction in days used marijuana.

In general, however, these comparisons of outcomes between different types of treatments do not provide much insight into the extent to which substance use is reduced in either treatment arm. For that, we must examine pre-post change in substance use. In addition, comparing the substance use reductions across the treatment arms in all the available studies provides another perspective on the comparative effectiveness of the different treatment types. These considerations motivated the second analysis described below.

The Second Analysis: Differential Change in Substance Use

In the second analysis, the treatment and comparison arms of the 73 treatment-comparison group pairs used in the first analysis were separated and pretest-posttest effect sizes representing change between the beginning and end of the treatment period were examined. Some of the 146 individual arms of those 73 pairs were duplicates originating from studies in which more than one treatment was contrasted with the same comparison condition. In other instances, the pretest baseline means were not reported for substance use outcomes that contributed to the prior analysis. Pre-post effect sizes, therefore, could be computed for only 98 treatment and comparison group arms from only 44 of the 45 studies contributing to the first analysis described above. Those adolescent samples provided 311 pre-post effect sizes for analysis. The inability to represent all the study arms and all the substance use outcomes that had contributed to the previous group comparison analysis in the pre-post analysis reported below means the results of the two are not fully comparable. Differences can come from the different analysis approaches used, but also from the fact that the same studies and outcomes are not represented in both analyses.

Across all the 311 pre-post substance use effect sizes, the random effects mean was .52 (p< .001; 95% CI [.44, .60]), indicating that adolescents exhibited significant decreases in their substance use after entry into treatment. The mean reductions were greatest for marijuana use (ḡ = .58, p< .001, 95% CI [.38 .77]) and mixed substance use (ḡ = .65, p< .001, 95% CI [.52 .77]), and smallest for alcohol (ḡ = .31, p< .001, 95% CI [.22 .39]) and other specific (e.g., cocaine) substance use (ḡ = .13, p< .05, 95% CI [.01 .25]). Effect sizes of these magnitudes can again be better understood in terms of the substance use reductions they represent. Using the number of days used substances in the past month (e.g., from the TLFB), these effect sizes represent magnitudes equivalent to a pre-post reduction from 2 to 0.6 days of alcohol use, 13 to 6 days of marijuana use, 10 to 5 days of mixed substance use, and from 3.5 to 2.7 days of other substance use. There was also evidence of substantial heterogeneity in the pre-post effect sizes (χ²= 398.06, p< .001, τ² = .16; I² = 75.8%), indicating that there are differences across the arms that influence the magnitude of adolescents’ reductions in substance use after entry into treatment.

There are various method, participant, and treatment characteristics of the different study arms that may account for at least some of the variation in the observed pre-post effect sizes. To examine the influence of such characteristics, we selected variables representing three distinct categories of study characteristics: those related to the study methods, the nature of the adolescent participants, and features of the treatment. We then fit a series of nested meta-regression models that examined the contribution of each of these sets of variables. Model I included methodological variables and assessed the potential for method differences to be confounded with the substantive variables of interest.⁴ Model II then added demographic characteristics of the participants to examine whether gender, race/ethnicity, or age distinguished adolescents who typically responded better or worse to treatment irrespective of the nature of the treatment. For similar reasons, Model II also included a set of participant characteristics of more direct clinical relevance—whether the sample included comorbid cases, their mean delinquency level, and the baseline severity of their substance use. Model III then added three general characteristics of the treatment provided to those participants—duration of treatment, frequency of contact, and a general indicator of quality of implementation. This model allowed assessment of the general contribution of the amount of treatment irrespective of the specific treatment modality. All these models additionally controlled for substance use outcome type and whether the treatment arms were indicated in the original studies as those of focal interest or as comparison conditions.

Table 3 presents the unstandardized regression coefficients (b) from these models along with their robust standard errors and standardized regression coefficients (β). As shown there for Model I, none of the method variables had a significant relationship with pre-post effect sizes. Rather, the notable feature of Model I is the statistically significant negative coefficients for alcohol and other substance outcomes. These results indicate that the pre-post change for these substances was, on average, smaller than for mixed substance use—the omitted reference value in this set of dummy codes. There was no significant difference for pre-post change on marijuana use versus mixed substance use, however. Overall, therefore, the treatments represented in these studies had significantly larger effects on marijuana and mixed substance use than on alcohol and other substance use (e.g., cocaine, heroin). Moreover, these differences remained statistically significant when the other variables for the more complete models in the series were added.

Table 3. Coefficients and Robust Standard Errors from Nested Meta-Regression Models Predicting Pretest-Posttest Effect Sizes (k = 79; n = 311).

	Model I			Model II			Model III			Model IV
	b	se	β	b	se	β	b	se	β	b	se	β
Alcohol outcomea	−.42*	.09	−.40	−.36*	.16	−.34	−.32†	.17	−.31	−.47*	.15	−.45
Marijuana outcomea	−.11	.15	−.07	−.19	.16	−.12	−.17	.16	−.10	−.27*	.13	−.17
Other substance outcomea	−.53*	.20	−.18	−.70*	.20	−.24	−.49*	.21	−.17	−.54*	.20	−.19
Arm in non-focal position	−.15	.09	−.16	−.16†	.09	−.17	−.08	.10	−.08	−.04	.11	−.04
Outcome timinga	−.01	.01	−.21	−.01	.01	−.22	−.01	.01	−.13	−.01	.01	−.05
Pre-post interval	.01†	.01	.36	.01†	.01	.38	.01†	.01	.35	.00	.01	.26
Study attrition	.21	.26	.07	.18	.26	.06	.20	.28	.06	−.12	.26	−.04
Randomized control group study	.13	.14	.13	.12	.13	.11	.24	.14	.23	.09	.13	.09
Percent male in sample				.01	.01	.13	.01	.01	.03	.01	.01	.02
Percent white in sample				−.01	.01	−.10	−.01	.01	−.09	−.01	.01	−.11
Average age				−.03	.03	−.10	−.01	−.03	−.01	−.01	.03	−.04
Clinically comorbid participants				.01	.11	.01	−.03	.11	−.03	.09	.11	.10
Delinquency level of sample				−.09	.06	−.23	−.01	.07	−.03	−.16*	.05	−.43
Baseline substance use severity				−.08	.09	−.10	−.09	.10	−.10	.01	.08	.01
Frequency of treatment contact							.09	.07	.20
Treatment duration							−.01*	.01	−.31
Implementation scale score							−.01	.08	−.01
Behavioral therapy										.37	.38	.17
CBT										.38†	.20	.20
Family therapy										.59*	.19	.44
Group/mixed counseling										.51	.19	.37
MET										.22	.14	.18
MET/CBT										.32	.20	.20
PET										.14	.14	.14
Pharmacological therapy										−.33	.28	−.10
Practice as usual										−.06	.01	−.13

Note:

^aCoefficients shown for the between-study effects of these covariates that varied within and between studies.

^†p< .10.

^*p< .05.

The Model II results show that none of the characteristics of the participant samples that were examined—gender, race/ethnicity, age, clinical comorbidity, delinquency, and baseline substance use severity—was significantly associated with reduction in substance use net of the method characteristics. Finally, Model III indicated that, net of the method and participant characteristics, neither frequency of treatment contact nor general implementation levels had significant independent relationships with pre-post improvement. Treatment duration (in days), however, did show a significant negative association, indicating that, on average, adolescents in longer treatment programs showed less improvement. For instance, holding all the other variables at their means, the predicted pretest-posttest effect size for 1-day treatment programs (i.e., brief interventions) was .68, versus .61 for 30-day programs, .45 for 90-day programs, .37 for 120-day programs, and .14 for 210-day programs. Note that comorbidity, delinquency, and baseline substance use severity were controlled in these comparisons, so the differences cannot be readily explained by longer treatment for adolescents with more severe problems.⁵ Despite this significant negative relationship, it is important to note that, on average, participants in all types of programs reduced their substance use between pretest and posttest. Participants in longer-duration programs simply reported less improvement over time.

The primary contribution of the first three regression models summarized in Table 3, however, is not so much the identification of specific variables with independent relationships to pre-post improvement. More important for present purposes is the ability of the full set of variables in Model III to control for more general differences between the treatment arms that might be confounded with the effects of the specific treatment types of interest from a comparative effectiveness perspective. Though most of the method, participant, and general treatment characteristics were not significantly related to pre-post effect sizes, some of that is due to low statistical power, as suggested by the standardized coefficients that show relationships of moderate magnitude that are, nonetheless, not statistically significant.

To estimate the substance use reduction associated with each treatment type while adjusting for any potential confounding with the variables shown in Table 3, covariate-adjusted pre-post effect sizes were estimated from the meta-regression shown in Model II. Note that this model does not include the treatment dummy indicators or the three variables shown in Model III -- frequency of treatment contact, treatment duration, and the implementation scale score. We did not want to hold these variables constant in this analysis because they refer to inherent characteristics of each treatment modality as delivered. Controlling out these differences between treatments, therefore, could control out some of the distinctive characteristics of the treatment modalities on which this analysis focuses.

To obtain covariate-adjusted pre-post effect sizes for each treatment modality, the residual from Model II was added to the constant value for the predicted effect size when the other predictors were held at their weighted overall mean values. Figure 2 shows the random effects means and 95% confidence intervals for these covariate-adjusted pre-post effect sizes for the treatment types represented in at least four independent samples, listed in ascending order of mean effect size. The vertical line at zero represents no improvement from pretest to posttest and mean effect sizes to the right of that line indicate that, on average, there were improvements, e.g., increases in abstinence, decreases in frequency of use. Confidence intervals that do not include zero indicate that the mean pre-post effect size was statistically significant. As Figure 2 shows, all the treatment types exhibited both positive and statistically significant improvements in substance use over time with the exception of practice as usual. The group/mixed counseling treatments and family therapy showed the largest adjusted mean pre-post effect sizes, although there was substantial overlap among the confidence intervals for most of the treatment types. Results for the three treatment types represented with fewer than four independent samples (not shown in Figure 2) all yielded covariate adjusted mean pre-post effect sizes that were not significantly different from zero: Pharmacological (ḡ = −.11, 95% CI [−1.85, 1.63], n = 2, k = 2); Placebo (ḡ = .14, 95% CI [−3.59, 3.88], n = 3, k = 2); Behavioral (ḡ = .54, 95% CI [−.94, 2.02], n = 8, k = 3).

Figure 2. Adjusted Mean Pretest-Posttest Effect Sizes by Treatment Type.

Note: Means to the right of zero indicate reduced substance use over time (i.e., lower frequency, more abstinence). Estimates adjusted for substance outcome type, method, and participant characteristics. Confidence intervals based on robust standard errors that account for clustering within studies. Treatment types with k < 4 (behavioral, pharmacological, placebo) are omitted from the figure; see text for results.

Especially notable in Figure 2 is the overlap in confidence intervals for many of the treatment types and the ‘no treatment’ control arms from those studies that used such controls. The regression coefficients on the treatment type dummy codes for Model IV in Table 3 directly test the difference between the mean covariate-adjusted effect size for each treatment type shown and the mean for the ‘no treatment’ control groups (omitted as the reference category in that regression), net of the other treatment types. As can be seen there, only the family therapy mean pre-post effect size was significantly larger than that for the no treatment controls.

Overall, the comparisons of the effects of the various treatment types resulting from this pre-post change analysis were substantially similar to those resulting from the group comparison analysis reported earlier with a few notable exceptions. Relative to the other treatments, group/mixed counseling and PET showed stronger effects in this analysis and MET/CBT and pharmacological treatment showed weaker effects. As in the earlier analysis, however, few of these differences were statistically significant. Moreover, because of the limited reporting of pretest values on the substance use outcome measures, not all the treatment arms represented in the group comparisons could be included in the pre-post analysis, thus the two sets of results are not strictly comparable.

Discussion and Conclusions

Controlled studies of the effects of treatment for adolescents with substance use disorders are a relatively recent development. Of the 45 studies that met the eligibility criteria for this meta-analysis, nearly 63% were reported after 2000. There are many gaps in this growing research literature and more studies will be needed before relatively definitive conclusions can be reached about the treatment approaches that are most effective for adolescents with different substance abuse issues and histories. One admitted limitation of the current study is the time lag between our literature search (2008), data collection, and data analysis; and given the rapid development of this field there are several recently published studies that were not included in this meta-analysis. Thus, an important area for future research will be to update these meta-analytic findings to incorporate results from ongoing and new research studies. Nonetheless, the currently available research does provide an encouraging pattern of findings that address several key questions about the treatment of adolescent substance use disorders.

Treatment efficacy

The most fundamental question that might be asked of this research is simply whether there is evidence of treatment efficacy; that is, whether any of the treatments that have been studied are better than no treatment at all. Only four of the distinct treatment types we identified in this meta-analysis were studied in controlled comparisons with no-treatment control conditions (group/mixed counseling, CBT, MET, and PET). The mean effect size across these comparisons was statistically significant and favored treatment.

More comparisons, though less well controlled, were possible with the pre-post change effect sizes computed for each arm of the available research studies and covariate-adjusted for differences in participant characteristics, type of substance use outcome, measurement characteristics, and attrition. The means for these pre-post effect sizes showed greater substance use reduction for all but one of the distinct treatment types than were found for the no-treatment control arms of the studies that used them. That difference, however, was statistically significant only for family therapy.

Taken together, these findings provide some evidence for the general efficacy of treatment relative to no treatment, though the pattern of evidence is clearly in that direction and there is no indication that treatment produces worse outcomes. On the other hand, comparison with no-treatment conditions is not the only test of the efficacy of the different treatment types. Treatments have most often been compared with other treatments in the available research, which sets a higher bar for demonstrating efficacy than comparison with a no-treatment control. Nevertheless, results from the pre-post analysis indicated an almost universal reduction in substance use between treatment entry and termination regardless of treatment type. This could, of course, result largely or entirely from spontaneous remission on the part of the adolescent participants or even regression to the mean given that entry into treatment is likely to come at a point where substance use problems are especially severe. Nonetheless, it is also consistent with the expected effects of effective treatment. Given the indications that at least some treatments are effective in reducing substance use, it is encouraging to see widespread reductions among the adolescents in the research studies.

Comparative effectiveness

From a practical perspective, the most important question to ask of the research findings on treatment of adolescent substance use disorders is which, if any, treatment works best, and for which adolescents. Ideally, for purposes of assessing comparative effectiveness, multiple independent studies would be available comparing each treatment type with every other treatment type. The research conducted to date falls well short of that ideal with many comparisons for which there are no studies at all and others with too few to yield stable results (as is evident in Table 2).

The best we were able to do to assess comparative effectiveness under these circumstances was to examine the effect sizes for the outcomes of each treatment type compared to whatever diverse treatment or control conditions happened to be used in the available studies. Any differences in those effect sizes for different treatment types, however, is a function of what each was compared with as well as how effective it is in its own right. To make those effect sizes somewhat more comparable across studies as indicators of differential treatment effects, we used covariate-adjusted versions of them that were statistically adjusted for differences in the characteristics of the participant samples and study methods.

Further indications that at least some distinct treatment types, on average, are efficacious were shown by the statistically significant mean effect sizes indicating better outcomes for family therapy and MET than for the various treatment and control conditions with which they have been compared. CBT, MET/CBT, and behavioral therapy were also favored in the comparisons in which they were involved, but those mean effects sizes fell short of statistical significance.

These patterns were largely replicated when the covariate-adjusted pre-post effect sizes for the individual treatment arms were compared. Family therapy, behavioral therapy, CBT and MET were among the treatment types showing the largest substance use reductions while placebo and no treatment controls were among those showing the smallest reductions. The most convincing and consistent comparative effectiveness finding was for family therapy, which showed relatively large positive effects relative to other treatments in both analyses.

Any conclusions about the general advantages of family therapy must, nonetheless, be tentative, given that there were several treatment types with which it was never compared and others for which there are too few studies to yield confident results. It is notable that, as described in the Appendix, most of the studies with family therapy involved well-known name brand programs (i.e., Functional Family Therapy, Multidimensional Family Therapy, Family Support Network, and Multisystemic Therapy). These programs have a more extensive basis in research than many of the other treatments in this meta-analysis and especially well-developed treatment protocols.

The most general finding across the different analyses, however, was a lack of statistically significant differentiation between the substance abuse outcomes of the various distinct treatment types represented in the available studies. That could mean that they are all equally ineffective, but the overall pattern of evidence is more consistent with the conclusion that most are at least somewhat more effective than no treatment but, with the exception of family therapy, not clearly more effective than each other.

Another area in which little significant differentiation was found relates to the characteristics of the adolescent samples used in these studies. We attempted to code all the baseline information reported in the studies about those characteristics and include them in the analysis to identify subgroups more or less responsive to treatment. The analysis of pre-post reductions in substance use provided the most direct evidence and showed no differences related to gender, race/ethnicity, age, baseline substance use severity, comorbidity, or delinquency level. Further analyses not reported here examined the interactions of these variables with the different distinct treatment types and found only scattered chance levels of statistical significance.

Taken at face value, this rather surprising lack of relationships between participant characteristics and treatment effects is perhaps an encouraging finding. It indicates that treatments are relatively robust in their effects, that is, produce similar outcomes for adolescents with different demographic characteristics and substance use issues and histories. Such a conclusion is likely premature, however. Few of the research studies in the meta-analysis broke out treatment effects for different participant subgroups, so most of what could be examined was differences in the aggregate samples.

The one variable related to participant characteristics that was clearly associated with substance use outcomes was the substance at issue. Both the group comparison and pre-post regression analyses showed that reductions in substance use were smaller for alcohol and other substances (e.g., heroin and cocaine) than for marijuana. This pattern appeared when treatment and participant characteristics were statistically controlled and within studies with multiple outcomes for the same participants receiving the same treatment. Marijuana use thus appears to be more responsive to treatment than alcohol or hard drug abuse.

In additional exploratory analysis of pooled data from Chestnut Health System’s Global Appraisal of Individual Needs (GAIN) database (Dennis et al., 2008), we conducted analysis analogous to that of the second meta-analysis reported here to examine whether there were similar relationships between client and program characteristics and pre-post changes in substance use among adolescents in more routine practice programs. Analyses were based on data from 102 outpatient treatment programs serving over 9,000 adolescents across the United States. Those results (available upon request from the authors) provided similar findings in terms of (a) the almost universal reduction in substance use between treatment entry and termination regardless of treatment type, (b) smaller reductions in alcohol use among adolescents than marijuana use, and (c) largest reductions in substance use for group and mixed counseling programs. However, results from the GAIN data did indicate that routine practice providers of adolescent substance abuse treatment may see less improvement if they serve client populations that are primarily male, have high levels of psychiatric comorbidity, or have higher levels of alcohol-related problems. More research of this type is needed to compare findings from research studies to outcomes observed in routine practice settings, thereby providing useful practice based evidence about treatment effectiveness.

Any practical implications of the findings from the current study should be considered within the context of the limitations of the study as well as feasibility and cost for treatment providers. For instance, although some types of treatment tended to out-perform or under-perform relative to other treatment types, we would conclude that in general, there is a broad range of treatments for adolescents with substance use disorders that seem to be effective for reducing adolescents’ overall levels of substance use. Practitioners tasked with choosing a specific treatment program to implement should therefore also consider the costs of implementation associated with different treatment types—an issue that is not addressed here. For instance, cost effectiveness research from the Cannabis Youth Treatment Study suggests that the cost per day of abstinence is significantly higher for branded family therapy programs relative to MET/CBT and ACRA programs (Dennis et al., 2004). Future research comparing the outcomes of different treatments would, therefore, provide better practical guidance if they would report comparative cost as well as comparative effectiveness.

Appendix A

Treatment Type Categories, Acronyms, and Treatment Descriptions for the Treatments in the Group Differences Meta-analysis.

General Treatment Type (Acronym)	Description
Behavioral therapy	Behavioral or contingency management therapy based on the principles of rewards, punishment, and reinforcement
	Adolescent Community Reinforcement Approach (ACRA)  (k = 3; n = 10)
	Generic behavioral (k = 1; n = 1)
Cognitive behavioral therapy (CBT)	Therapy that helps clients recognize situations in which they are most likely to use, and how to avoid and appropriately cope with those situations
	Generic CBT (k = 14; n = 43)
Family therapy	Therapy involving one or more family members that addresses family relationships and processes and seeks to understand individual behavior within the context of the family
	Family Support Network (FSN) (k = 2; n = 4)
	Functional Family Therapy (FFT) (k = 5; n = 15)
	Multidimensional Family Therapy (MDFT) (k = 7; n = 20)
	Multisystemic Therapy (MST) (k = 5; n = 26)
	Generic family therapy (k = 5; n = 21)
Group/mixed counseling	Generic counseling or ‘talk therapy’ delivered in multiple formats (group, individual, family) focusing on day to day life issues that does not fall in other clearly defined treatment categories
	Chestnut Health Systems Outpatient (CHS) (k = 4; n = 4)
	Seven Challenges (7-C) (k = 1; n = 2)
	Generic multi-service Package (k = 1; n = 6)
	Generic group/mixed (k = 13; n = 64 )
Motivational enhancement therapy/motivational interviewing (MET)	Therapy using the motivational enhancement/interviewing strategies that use reflective listening, open ended strategies, and comparisons of behavior to normative standards
	Generic MET (k = 17; n = 70)
Motivational enhancement + cognitive behavioral therapy (MET/CBT)	Therapy that combines MET and CBT therapeutic strategies
	MET/CBT-5 (k = 3; n = 6)
	MET/CBT-7 (k = 5; n = 5)
	MET/CBT-12 (k = 1; n = 2)
	Generic MET/CBT (k = 5; n =12)
Psychoeducational therapy (PET)	Educational therapy that teaches clients about substance abuse and substance-related issues
	Generic PET (k = 19; n = 55)
Pharmacological therapy	Therapy employing psychoactive drugs to affect thinking, feeling, or behaviors
	Pemoline (k = 1; n = 2)
	Acamprosate (k = 1; n = 2)
	Fluoxetine (k = 1; n = 1)
	Disulfiram (k = 1; n = 2)
	Cyanamide (k = 1; n = 2)
	Tianeptine (k = 1; n = 2)
	Naltrexone (k = 1; n = 2)
Skills training	General skills training treatment program that does not fall in another clearly defined treatment category
	Relaxation skills training (k = 1; n = 2)
Practice as Usual	Comparison groups receiving practice as usual – generally includes community treatment or case management received as standard practice as opposed to the treatment program of interest.
	Generic practice as usual (k = 7; n = 43)
No treatment condition (No Tx)	Group received no clearly specified treatment or intervention
	No Treatment (k = 14; n = 71)