Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Jan 1.
Published in final edited form as: Psychiatr Serv. 2014 Nov 17;66(1):41–48. doi: 10.1176/appi.ps.201300517

Provision of Mental Health Services as a Quality Indicator for Adolescent Substance Use Treatment Facilities

Rajeev Ramchand 1, Beth Ann Griffin 2, Sarah B Hunter 3, Marika Suttorp Booth 4, Daniel F McCaffrey 5
PMCID: PMC4388191  NIHMSID: NIHMS632116  PMID: 25219932

Abstract

Objective

We sought to test whether adolescents receiving substance use treatment at facilities offering full (can treat all psychiatric conditions) or partial (do not treat severe/persistent mental illness) mental health services have better 12-month substance use and mental health outcomes.

Methods

Data was collected from 3,235 adolescents served at one of 50 adolescent treatment facilities who were assessed at baseline and at 12-months. Propensity scores were applied to compare client outcomes from three types of facilities (full, partial, or no mental health services); weighted linear models were estimated to examine outcomes.

Results

Youths attending facilities offering full or partial mental health services had better substance use outcomes than youths attending facilities offering no such services. There was no evidence of a difference in substance use outcomes between facilities offering full versus partial services, nor evidence of differences on mental health outcomes.

Conclusions

These preliminary findings suggest that the availability of mental health services may be a useful quality indicator for adolescent substance use treatment facilities. More research is needed to examine specific types of mental health services offered at different facilities.

Keywords: substance use treatment, adolescent, propensity scores, mental health treatment, quality indicator

Introduction

Substance use disorders are among the key contributors to adolescent morbidity and mortality in the United States (1), necessitating effective treatment for youths with such disorders. Unfortunately, consumers have little information to ensure adolescents are provided with effective treatments because few practices or processes have been shown to reliably predict adolescent outcomes (2–6). When specific treatment practices are known to correspond with superior client outcomes and these practices are not universally implemented, they may serve as quality indicators that distinguish better performing facilities from those with worse performance (7, 8). The current study tests whether the provision of mental health services is an appropriate structural quality indicator for adolescent substance use treatment facilities.

Notable efforts have been made to identify quality metrics for substance use treatment services, most of which derive from expert consensus. Experts have recommended “performance measures” (e.g., client engagement) (9–11), “key adolescent substance use treatment elements” (e.g., staff trained in adolescent development and co-occurring mental disorders) (12), and “effective addiction treatment principles” (e.g., “[M]any drug-addicted individuals also have other mental disorders”) (13). Many of these ‘measures’, ‘elements’, and ‘principles’ overlap; studies have examined how clinical practice conforms or adheres to some of these guidelines (12, 14, 15) but only one process measure - client engagement - has yet been associated with positive client outcomes (4, 16, 17).

One salient structural characteristic that may serve as an ideal quality indicator is whether facilities offer mental health services. Over half of adolescents entering treatment report having mental health problems (18–24), suggesting that co-occurring disorders among adolescents are the “norm rather than the exception” (25, p. 200). Facilities are encouraged to offer “integrated care” (12, 13, 25). However, only a minority receive mental health services (21, 26), possibly because only about half of these facilities offer such services (5). To date, no study has examined whether the provision of mental health services at the facility-level might be associated with better client outcomes in youth treatment settings. The current study tests whether the availability of mental health services might serve as an appropriate structural quality indicator for adolescent substance use treatment facilities. We do so using data from 50 facilities that offered different levels of mental health services and served over 6,000 adolescents. Using casemix adjustment, we examined whether youths who received treatment at facilities offering mental health services fared better at 12-months post-intake than youths attending facilities not offering such services.

Methods

Sample

Data were from facilities that were supported by one of 9 discretionary grant programs sponsored and initiated by the Substance Abuse and Mental Health Services Administration - Center for Substance Abuse Treatment (SAMHSA CSAT) between 1998 and 2008 (see acknowledgement section for list of discretionary grant programs). Though the focus and initiation year of each discretionary program differed, all facilities were required to collect intake (i.e., baseline) and 12-month post-intake (i.e., outcome) information from youths receiving funded services. All youths were 12–18 years of age, and we restricted our analysis to outpatient programs since most adolescents in the United States receive outpatient care (27). During that period, CSAT supported 108 facilities. We excluded 29 sites that were missing facility-level data on services offered and 29 sites missing outcome data (22 were missing all 12-month data; 7 did not collect data on two of the measures used for this analysis at baseline or follow-up). The final analytic sample consisted of 50 different facilities serving 6,623 clients. This sample was further restricted to 3,235 clients with complete 12-month outcomes.

Measures

Both facility- and client-level data were used for the current study.

Facility Characteristics

To obtain facility-level data on provision of mental health services, in 2010 we used a tailored survey design (28) to survey administrative or clinical staff via mail or email (identified staff indicated a preference during initial contact) at the funded facilities. Provision of mental health services was assessed with the following question: “Does this location treat both the substance abuse problems and psychiatric problems of dually diagnosed adolescents?” Full mental health services was defined as responding “Yes, can treat all psychiatric conditions (e.g. have psychiatrist and/or licensed social worker or psychologist on staff)”;’ partial mental health services was defined as responding “Yes, except for severe/persistent mental illness (includes actively psychotic, danger to self & others).” Those who responded “No” were categorized as providing no mental health services. Staff that completed the survey indicated whether responses referred to the time that the client-level data was collected or at the time of the 2010 survey.

We examined 20 other facility-level characteristics that could potentially co-occur with the different mental health services and also be related to the outcomes. These characteristics were drawn from both our own survey as well as characteristics collected in the National Survey of Substance Abuse Treatment Services (N-SSATS) that we were able to match to the selected facilities in the year in which the grant was initially funded. Specific variables were whether the facility offered particular assessments (comprehensive substance abuse assessment or diagnosis); therapy and counseling (family counseling, aftercare counseling), testing (biological drug testing, HIV testing, other STD testing, TB screening), transitional and other ancillary services (assistance with obtaining social services, discharge planning, employment counseling or training, case management, HIV/AIDS education counseling, or support, other types of HIV-related interventions), and models of care (cognitive behavioral, 12-step, therapeutic community, motivational enhancements, motivational incentives, and other) as well as an indicator of whether the facility had any licensure, certification, or accreditation. For most items, facility representative were asked, “Which of the following services are provided by this facility at this location (mark all that apply).”

Client Characteristics

All client characteristics were assessed via self-report using the Global Appraisal of Individual Needs (GAIN) (29, 30), which all facilities were required to use at treatment intake and at 12-months. Casemix adjustment controlled for 22 baseline variables across five domains listed in Table 1: demographics, substance use, mental health, criminal justice involvement, and sexual risk.

Table 1.

Nonresponse weighted baseline characteristics of study sample across treatment facilities offering full, partial, or no mental health services, before and after casemix adjustment.

Full Partial None Total
Unweighted Weighted Unweighted Weighted Unweighted Weighted Unweighted
Value ASMD *100 Value ASMD *100 Value ASMD *100 Value ASMD *100 Value ASMD *100 Value ASMD *100 Value SD
Female (%) 29 10 28 8 23 3 24 2 23 4 23 4 24 43
Race (%)
White 67 43 55 19 54 17 47 4 23 45 44 3 45 50
Black 13 3 14 3 13 2 13 2 11 3 13 2 12 33
Hispanic 7 45 17 23 18 22 25 6 51 51 29 3 28 45
Other 13 6 15 0 16 3 15 0 15 1 15 1 15 36
Current Living Situation (%)
House 91 15 90 13 81 13 87 3 86 3 87 5 85 35
Friend’s House/Unsupervised Dorm 6 6 7 3 6 5 7 3 10 9 9 6 7 26
Other 3 15 3 15 13 24 7 1 4 13 4 13 7 26
Substance Use
Frequency-Past 90 days (Mean) .10 2 .12 9 .10 8 .11 1 .12 9 .11 4 .11 .13
Problems-Past Month (Mean) 2.03 14 2.44 2 2.62 04 2.51 0 2.71 6 2.52 0 2.51 3.29
Problems-Past Year (Mean) 5.47 23 6.07 10 6.99 11 6.35 3 6.67 4 6.38 2 6.49 4.41
Does not recognize their substance use as a problem (%) 11 1 10 4 15 12 12 3 8 11 10 3 11 32
Past receipt of treatment (Mean) .27 15 .29 13 .44 03 .42 01 .48 07 .43 01 .41 .99
Marijuana as primary substance under treatment (%) 44 04 47 00 46 01 46 01 49 04 46 00 47 50
Tobacco Dependence (Mean) 40.83 18 39.91 15 36.68 07 34.81 02 26.70 19 32.81 03 33.96 38.74
Mental Health
Emotional Problems .18 18 .19 13 .23 13 .21 01 .21 01 .21 01 .21 .19
Problem Orientation .51 12 .62 05 .88 13 .71 01 .62 05 .65 02 .69 1.48
Internal Mental Distress 6.37 11 6.97 03 7.66 05 7.00 03 7.40 02 6.88 04 7.24 8.07
Behavior Complexity 8.86 12 9.38 6 10.84 12 9.62 03 9.55 4 9.31 7 9.85 8.01
Criminal Justice Involvement
Criminal Justice System Involvement .31 25 .33 21 .50 15 .42 0 .43 2 .40 6 .42 .46
Illegal Activities .09 9 .10 4 .11 2 .10 1 .11 3 .10 1 .10 .12
Total Arrests .23 8 .23 8 .27 2 .27 .33 7 .30 2 .29 .70
Crime Violence 5.58 21 5.96 14 7.02 4 6.56 4 7.37 10 6.70 2 6.79 5.73
Drug Crimes .40 22 .46 17 .67 4 .58 4 .73 11 .61 2 .63 1.00
Experiences in controlled environments .06 26 .07 24 .23 30 .14 0 .11 12 .11 12 .14 .30
Institutionalization 5.69 24 6.28 22 19.09 28 11.50 11 8.96 11 8.72 12 11.88 25.74
Sexual Risk 13.05 6 13.33 1 13.47 04 13.29 00 13.30 0 13.19 2 13.30 4.51
Open in a new tab

ASMD=Absolute Standardized Mean Difference; SD=Standard Deviation

Outcomes

We examined four outcomes. Substance use frequency was measured with a scale that assessed the average proportion of the past 90 days where alcohol and other drugs were used. Substance problems in the past month were assessed using endorsement of up to 16 symptoms associated with criteria for substance dependence, abuse, health and psychological problems, and lower severity symptoms. Internal reliability was good (α = .80) for the substance use frequency scale (31) and excellent (α = .92) for the substance problems scale (30); a test-retest analysis of adolescents found that both scales also showed good reliability over a 90-day interval (r > .72 for both) (32). Internalizing problems was measured with a 43-item scale that comprises a count of past-year symptoms related to internalizing disorders with excellent reliability (α = .94) (30) and reflects severe levels of mental distress, such as traumatic stress and homicidal and suicidal thoughts (33). Externalizing problems were measured with the Behavior Complexity Scale, which is a count of 33 symptoms relating to externalizing disorders, also with excellent reliability (α = .94) (30) and that reflects inattention and less so symptoms of conduct disorder (33).

Analysis

Non-Response Weights

Because we had limited outcome data from 50 treatment facilities, we first estimated nonresponse weights using Generalized Boosted Models (GBM) (34). These weights made the clients with 12-month outcomes match the original baseline sample of 7,964 clients from 72 facilities (50 with 12-month outcomes, and 22 that systematically did not collect any 12-month outcomes) with respect to 22 pre-treatment variables described above.

Casemix Adjustment

Next, we estimated propensity score weights to adjust for baseline differences among youths who attended substance use treatment facilities with full, partial, or no mental health services. We created weights for youths in each group to match the characteristics of the overall analytic sample at baseline with respect to the distributions of the 22 baseline variables described above. Propensity score weights were also created using GBM (35). Estimation of both the non-response and propensity score weights used the twang package in R (36, 37).

After propensity score weighting, we calculated the pre- and post-weighting absolute standardized mean difference (ASMD) from the overall analytic sample for each group on each of the pretreatment variables. Value of 0 represents no difference in means while values of 1 represent one standard deviation difference between a particular group and the overall analytic sample. For each variable, we took the maximum ASMD across the three treatment groups to summarize how well balanced the groups are relative to the overall analytic sample. We denote group differences to be notable when the maximum ASMD is greater than .20 (38).

Facility-level adjustment

The level of mental health services offered may correlate with other services or practices. While such clustering would not necessarily invalidate the provision of mental health services as a quality indicator, we nonetheless set out to assess how sensitive our findings were to adjustment for additional facility-level confounds. Of the aforementioned 19 characteristics, facilities offering different levels of mental health services differed meaningfully on 13 characteristics. We did not have enough facilities within each group to estimate either propensity score weights to balance on these characteristics or regression models that controlled for all 13 characteristics. We therefore estimated a series of outcome models adjusting for seven of the most imbalanced covariates (biological drug testing, case management, accreditation/licensing, and reporting 12-step treatment orientation, therapeutic community, motivational enhancement, or motivational incentives as primary care model) one at a time, noting when results from the models were qualitatively different than those from the model that did not include it. Two characteristics yielded such differences: reporting a 12-step treatment orientation or motivational incentives as a primary care model. Given adjustment for these measures affects the estimated impact of partial mental health services facilities, our final models control for these two variables.

Outcome Analysis

We used weighted linear regression models to estimate the association between mental health service provision and each outcome. The models used dummy indicators for type of facility, along with those covariates which, after weighting, had ASMD>.20 and two facility-level characteristics. All models controlled for clustering of clients within facilities and predictive margins/recycled mean predictions were used to compute the estimated mean 12-month outcomes for each facility group (39). Recycled mean predictions/marginal means compute the predicted mean for each group assuming that all youth in the analytic sample had attended facilities within that group. From there, predicted mean group differences along with appropriate standard errors can be estimated using methods detailed elsewhere (34). Models were fit using the survey package in R.

Results

Description of the Sample

Prior to non-response weighting, those with 12-month data were, at baseline, more likely to recognize that they had a substance use problem than the entire baseline sample. After weighting this difference was attenuated.

Thirteen sites (n = 849 clients) reported offering full mental health services, 18 facilities (n = 1,227) offered partial mental health services only, and 19 facilities (n = 1,159) did not offer any mental health services. Of the 31 sites offering any mental health services, 18 reported that these services were offered during the funding period and 12 reported currently offering them, one site did not answer the question about whether responses were current or during funding period. The overall sample to which we weighted each subgroup is presented in the last column of Table 1. In aggregate, approximately one-quarter of the sample was female, and half was White. Half were in treatment for marijuana as the primary substance. According to thresholds established by the developers of the GAIN instrument (40), the youths in the sample were marked by moderate levels of frequent substance use in the past month and moderate to high levels of substance use problems in the past year.

Casemix Adjustment

Weighting by both propensity score and non-response weights balanced the data from the three groups of facilities. Before weighting, the groups were notably different on almost half of the 22 variables. After applying propensity score weights, four differences remained: youths in facilities with full mental health services still had lower levels of experiences in a controlled environment, with the criminal justice system, and of being institutionalized relative to the overall sample. There also remained some imbalance in that the full mental health service facilities treated a lower proportion of Hispanic clients. These four variables were included as covariates in our analyses.

Substance Use Outcomes

Predicted mean outcomes across the three groups for both substance problems and substance frequency are presented in Figures 1 and 2, respectively; differences between groups are presented in Table 2. Youths attending facilities providing mental health services used fewer substances and reported fewer substance use problems at 12 months than similar youths who attended facilities without mental health services. For example, the mean value on the substance problem scale was 1.13 among clients attending facilities with full mental health services and 1.22 among clients attending facilities with partial mental health services relative to 1.86 for those with no mental health services, differences that were statistically significant (full versus none difference= .73, 95% Confidence Interval (CI)=.17 to 1.30; full versus partial difference=.64, 95% CI =.25 to 1.03). However, youths from all three facilities were above the substance problem threshold of 1 that may signify abuse, and all were under the substance frequency threshold of 0.14 that may identify individuals with considerable difficulty stopping without significant assistance and/or a controlled environment (40). There was no statistically significant evidence of a difference between facilities offering full as compared to partial mental health services on these outcomes.

Figure 1.

Figure 1

Open in a new tab

Predicted mean substance use frequency at 12-months across treatment facilities offering full, partial, or no mental health services after case-mix adjustment. The substance use frequency scale ranges from 0–1.

Figure 2.

Figure 2

Open in a new tab

Predicted mean past month substance use problems reported at 12 months across treatment facilities offering full, partial, or no mental health services after case-mix adjustment. The substance problem scale ranges from 0 to 16.

Table 2.

Adjusted mean differences in 12-month outcomes across treatment facilities offering full, partial, or no mental health services.

Substance Use Outcomes Mental Health Outcomes
Frequency Problems Internal Distress Behavior Problems
Mean Difference 95% CI Mean Difference 95% CI Mean Difference 95% CI Mean Difference 95% CI
Partial-Full .01 −.03 to .04 .09 −.53 to .72 .02 −1.46 to 1.51 .15 −1.72 to 2.02
None-Full .03 .00 to .06 .73 .17 to 1.30 −.98 −2.34 to.39 −.77 −2.63 to 1.09
None-Partial .02 .00 to .04 .64 .25 to 1.03 −1.00 −2.33 to .33 −.92 −3.07 to
Open in a new tab

Mental Health Outcomes

In Figure 4, we present predicted mental health outcomes across the three facility types; differences between outcomes are presented in Table 2. None of the differences were statistically significant.

Discussion

Quality indicators are practices not universally implemented but known to correspond with better client outcomes (7, 8); structural quality indicators are features of organizations related to the capacity to provide high quality care (41). Almost half of substance use treatment facilities do not offer mental health services, thereby meeting the criteria of not being universally implemented. The current analysis found preliminary support that this feature corresponds with better client outcomes: facilities offering either full or partial mental health services were associated with better substance use outcomes as compared to facilities reporting no mental health services. In a field with a notable absence of quality indicators, the availability of mental health services is a potentially meaningful quality indicator, though future research is needed to confirm these results.

The lack of evidence relating to mental health outcomes is noteworthy, suggesting that simply offering mental health services does not necessarily translate to improved outcomes for adolescents receiving substance use treatment. There exists a “paucity” of evidence-based treatment interventions for dually diagnosed youths (25). Though some such treatments exist with proven benefits under experimental settings, the effectiveness of these treatments, and adoption of these practices, in community-based settings is unknown. More evidence-based treatments are needed for dually diagnosed youths. Research is also needed to better understand the specific types of mental health services currently used to treat youths in substance use treatment settings, and how these might be improved. In the current study, the measures of “full” and “partial” mental health services were provided by administrative or clinical staff responding to a single survey item – more refined measures should be used to better specify such services.

An important limitation in this study is that we cannot fully isolate the effect of the mental health service provision in our analyses because the groups differ on other characteristics measured at the facility-level and possibly on unmeasured characteristics too. The existence of these differences do not necessarily invalidate mental health service provision as a quality indicator to the degree that it is commonly offered with these other characteristics that are the causal factors associated with positive outcomes. However, if the relationships are unique to our sample or could easily be disrupted if mental health service provision were used as a quality indicator, then such a use would be invalid. While beyond our ability to examine this issue more fully, we acknowledge this to be an important topic for future research.

Though this contribution provides important information, certain other limitations should be kept in mind. First, data from the current study do not derive from a sample of youths in treatment settings designed to be representative of community-based treatment providers in the U.S, and prior work indicates that CSAT-funded facilities may over-represent programs that, on average, are more likely to offer ancillary mental health and HIV and other STD services (5). Also, data from the current study represents an 11-year time frame (from 1998 to 2008). Although these two features limit the generalizability of the current findings, the analytic dataset does represent one of the largest longitudinal datasets of adolescent substance use treatment in the United States currently available. Furthermore, recent analyses of adolescent treatment facilities across a similar time period showed little change in service provision (5). Another limitation is that the propensity score weights we estimated may not adjust for group differences on unmeasured variables, as well as those variables subject to measurement error. However, we were able to control for a large set of baseline variables which reduces the risk of an omitted variable and measurement error bias (43). While our nonresponse weights make youths in our sample with 12-month outcomes look similar to the baseline sample of youths on 22 baseline characteristics, these weights do not protect effect estimates from attrition bias which may occur if, for example, clients with the worst outcomes were most likely to be lost to follow-up. The challenge of retaining clients in substance use treatment studies has been noted (44), and there is reason to suspect that clients with both superior and inferior outcomes are more likely to be lost to follow-up, limiting the types of sensitivity analyses that can be applied (45).

Figure 3.

Figure 3

Open in a new tab

Predicted mean mental health outcomes across treatment facilities offering full, partial, or no mental health services after case-mix adjustment. The Internal Mental Distress scale ranges from 0 to 43; the Behavior Complexity Scale ranges from 0 to 33.

Acknowledgments

Funded by National Institute on Drug Abuse (NIDA) Grant Number 5R01DA017507 and supported by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) contract #270-07-0191 using data provided by the following grantees: Targeted Capacity Expansion/HIV (Study: TCE; CSAT/SAMHSA contracts #270-2003-00006, #270-2007-00004C, #277-00-6500 and grantees: TI-16386, TI-16400), Young Offenders Reentry Program (Study: YORP; CSAT/SAMHSA contract #270-2003-00006 and #270-2007-00004C; and grantees: TI-16992, TI-17070, TI-17071), Drug Court (Study: DC; CSAT/SAMHSA contract #270-2003-00006 and #270-2007-00004C; and grantees: TI-17433, TI-17476, TI-17490), Assertive Adolescent Family Treatment (Study: AAFT; CSAT/SAMHSA contract #270-2003-00006 and #270-2007-00004C; and grantees: TI-17604, TI-17728, TI-17755, TI-17761, TI-17765, TI-17769, TI-17779, TI-17788, TI-17812, TI-17830), Cannabis Youth Treatment (Study: CYT; CSAT/SAMHSA contracts #270-97-7011, #270-00-6500, #270-2003-00006; and grantees TI-11317, TI-11321), Adolescent Treatment Model (Study: ATM; CSAT/SAMHSA contracts #270-98-7047, #270-97-7011, #277-00-6500, #270-2003-00006; and grantees: TI-11423, TI-11422, TI-11871, TI-11894, TI-11892, TI-11874), Strengthening Communities–Youth (Study: SCY; CSAT/SAMHSA contracts #277-00-6500, #270-2003-00006; and grantees: TI-13313, TI-13356, TI-13308, TI-13323), Effective Adolescent Treatment (Study: EAT; CSAT/SAMHSA contract #270-2003-00006; and grantees:TI-15678, TI-15670, TI-15486, TI-15413, TI-15562, TI-15514, TI-15672, TI-15478, TI-15447, TI-15545, TI-15671, TI-15527, TI-15489, TI-15485, TI-15584, TI-15421, TI-15586, TI-15469, TI-15677), Targeted Capacity Expansion (Study: TCE; CSAT/SAMHSA contract #270-2003-00006, #270-2007-00004C, #277-00-6500 and grantees: TI-15458). The authors thank these grantees and their participants for agreeing to share their data to support this secondary analysis. The opinions about this data are those of the authors and do not reflect official positions of the government or individual grantees.

Footnotes

Disclosures: None for any author

Contributor Information

Rajeev Ramchand, RAND Corporation.

Beth Ann Griffin, RAND Corporation.

Sarah B. Hunter, RAND Corporation

Marika Suttorp Booth, RAND Corporation.

Daniel F. McCaffrey, RAND Corporation

References

  • 1.Park MJ, Paul Mulye T, Adams SH, et al. The health status of young adults in the United States. Journal of Adolescent Health. 2006;39:305–317. doi: 10.1016/j.jadohealth.2006.04.017. [DOI] [PubMed] [Google Scholar]
  • 2.Williams RJ, Chang SY. A Comprehensive and Comparative Review of Adolescent Substance Abuse Treatment Outcome. Clinical Psychology: Science and Practice. 2000;7:138–166. [Google Scholar]
  • 3.Garner BR, Hunter BD, Modisette KC, et al. Treatment staff turnover in organizations implementing evidence-based practices: turnover rates and their association with client outcomes. Journal of Substance Abuse Treatment. 2012;42:134–142. doi: 10.1016/j.jsat.2011.10.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Garnick DW, Lee MT, O’Brien PL, et al. The Washington circle engagement performance measures’ association with adolescent treatment outcomes. Drug and Alcohol Dependence. 2012;124:250–258. doi: 10.1016/j.drugalcdep.2012.01.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Hunter SB, Griffin BA, Booth MS, et al. Assessing the generalizability of the CSAT-sponsored GAIN dataset: Are the CSAT sites representative of adolescent treatment programs in the U.S.? Journal of Substance Abuse Treatment. 2013;46:238–243. doi: 10.1016/j.jsat.2013.07.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Tanner-Smith EE, Wilson SJ, Lipsey MW. The comparative effectiveness of outpatient treatment for adolescent substance abuse: a meta-analysis. Journal of Substance Abuse Treatment. 2013;44:145–158. doi: 10.1016/j.jsat.2012.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Hermann RC, Palmer RH. Common ground: a framework for selecting core quality measures for mental health and substance abuse care. Psychiatric Services. 2002;53:281–287. doi: 10.1176/appi.ps.53.3.281. [DOI] [PubMed] [Google Scholar]
  • 8.McGlynn EA. Choosing and evaluating clinical performance measures. The Joint Commission Journal on Quality Improvement. 1998;24:470–479. doi: 10.1016/s1070-3241(16)30396-0. [DOI] [PubMed] [Google Scholar]
  • 9.Garnick DW, Lee MT, Chalk M, et al. Establishing the feasibility of performance measures for alcohol and other drugs. Journal of Substance Abuse Treatment. 2002;23:375–385. doi: 10.1016/s0740-5472(02)00303-3. [DOI] [PubMed] [Google Scholar]
  • 10.Garnick DW, Lee MT, Horgan CM, et al. Adapting Washington Circle performance measures for public sector substance abuse treatment systems. Journal of Substance Abuse Treatment. 2009;36:265–277. doi: 10.1016/j.jsat.2008.06.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.McCorry F, Garnick DW, Bartlett J, et al. Developing performance measures for alcohol and other drug services in managed care plans. The Joint Commission Journal on Quality Improvement. 2000;26:633–643. doi: 10.1016/s1070-3241(00)26054-9. [DOI] [PubMed] [Google Scholar]
  • 12.Brannigan R, Schackman BR, Falco M, et al. The quality of highly regarded adolescent substance abuse treatment programs: results of an in-depth national survey. Archives of Pediatric and Adolescent Medicine. 2004;158:904–909. doi: 10.1001/archpedi.158.9.904. [DOI] [PubMed] [Google Scholar]
  • 13.National Institute on Drug Abuse. Principles of drug addiction treatment : a research-based guide. 2. Washington, DC: National Institute on Drug Abuse; 2009. [Google Scholar]
  • 14.Garnick DW, Horgan CM, Chalk M. Performance measures for alcohol and other drug services. Alcohol Research & Health. 2006;29:19–26. [PMC free article] [PubMed] [Google Scholar]
  • 15.Mark TL, Song X, Vandivort R, et al. Characterizing substance abuse programs that treat adolescents. Journal of Substance Abuse Treatment. 2006;31:59–65. doi: 10.1016/j.jsat.2006.03.017. [DOI] [PubMed] [Google Scholar]
  • 16.Garner BR, Godley MD, Funk RR, et al. The Washington Circle continuity of care performance measure: predictive validity with adolescents discharged from residential treatment. Journal of Substance Abuse Treatment. 2010;38:3–11. doi: 10.1016/j.jsat.2009.05.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Garnick DW, Horgan CM, Lee MT, et al. Are Washington Circle performance measures associated with decreased criminal activity following treatment? Journal of Substance Abuse Treatment. 2007;33:341–352. doi: 10.1016/j.jsat.2007.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Chan YF, Dennis ML, Funk RR. Prevalence and comorbidity of major internalizing and externalizing problems among adolescents and adults presenting substance abuse treatment. Journal of Substance Abuse Treatment. 2008;34:14–24. doi: 10.1016/j.jsat.2006.12.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Diamond G, Panichelli-Mindel SM, Shera D, et al. Psychiatric syndromes in adolescents seeksing outpatient treatment for marijauan with abuse and dependency in oupatient treatment. Journal of Child & Adolescent Substance Abuse. 2006;15:37–54. [Google Scholar]
  • 20.Grella CE, Hser YI, Joshi V, et al. Drug treatment outcomes for adolescents with comorbid mental and substance use disorders. Journal of Nervous and Mental Disease. 2001;189:384–392. doi: 10.1097/00005053-200106000-00006. [DOI] [PubMed] [Google Scholar]
  • 21.Jaycox LH, Morral AR, Juvonen J. Mental health and medical problems and service use among adolescent substance users. Journal of the American Academy of Child and Adolescent Psychiatry. 2003;42:701–709. doi: 10.1097/01.CHI.0000046846.56865.F9. [DOI] [PubMed] [Google Scholar]
  • 22.Robbins MS, Kumar S, Walker-Barnes C, et al. Ethnic differences in comorbidity among substance-abusing adolescents referred to outpatient therapy. Journal of the American Academy of Child & Adolescent Psychiatry. 2002;41:394–401. doi: 10.1097/00004583-200204000-00011. [DOI] [PubMed] [Google Scholar]
  • 23.Sterling S, Weisner C. Chemical dependency and psychiatric services for adolescents in private managed care: implications for outcomes. Alcoholism: Clinical and Experimental Research. 2005;29:801–9. doi: 10.1097/01.alc.0000164373.89061.2c. [DOI] [PubMed] [Google Scholar]
  • 24.Garnick DW, Horgan CM, Acevedo A, et al. Criminal justice outcomes after engagement in outpatient substance abuse treatment. Journal of Substance Abuse Treatment. 2014;46:295–305. doi: 10.1016/j.jsat.2013.10.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Hawkins EH. A tale of two systems: co-occurring mental health and substance abuse disorders treatment for adolescents. Annual Review of Psychology. 2009;60:197–227. doi: 10.1146/annurev.psych.60.110707.163456. [DOI] [PubMed] [Google Scholar]
  • 26.Chan YF, Godley MD, Godley SH, et al. Utilization of mental health services among adolescents in community-based substance abuse outpatient clinics. The Journal of Behavioral Health Services & Research. 2009;36:35–51. doi: 10.1007/s11414-007-9100-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Substance Abuse and Mental Health Services Administration. DASIS Series S-61, HHS Publication No. (SMA) 12-4701. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2012. Center for Behavioral Health Statistics and Quality: Treatment Episode Data Set (TEDS): 2000–2010 National Admissions to Substance Abuse Treatment Services. [Google Scholar]
  • 28.Dillman D, Smyth J, Christian L. Internet, mail and mixed-mode surveys: The tailored design method. 3. Hoboken, NJ: John Wiley; 2009. [Google Scholar]
  • 29.Dennis ML. Global Appraisal of Individual Needs (GAIN) Administration guide for the GAIN and related measures (Version 1299) Bloomington, IL: Chestnut Health Systems; 1999. [Google Scholar]
  • 30.Dennis ML, Chan YF, Funk RR. Development and validation of the GAIN Short Screener (GSS) for internalizing, externalizing and substance use disorders and crime/violence problems among adolescents and adults. American Journal of Addiction. 2006;25:80–91. doi: 10.1080/10550490601006055. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Ramchand R, Griffin BA, Suttorp M, et al. Using a cross-study design to assess the efficacy of motivational enhancement therapy-cognitive behavioral therapy 5 (MET/CBT5) in treating adolescents with cannabis-related disorders. Journal of Studies on Alcohol and Drugs. 2011;72:380–389. doi: 10.15288/jsad.2011.72.380. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Dennis M, Babor T, Diamond G, et al. The Cannabis Youth Treatment (CYT) experiment: Preliminary Findings. Rockville, MD: Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration, Department of Health and Human Services; 2000. [Google Scholar]
  • 33.Stucky BD, Edelen MO, Ramchand R. A psychometric assessment of the GAIN Individual Severity Scale (GAIN-GISS) and Short Screeners (GAIN-SS) among adolescents in outpatient treatment programs. Journal of Substance Abuse Treatment. 2014;46:165–173. doi: 10.1016/j.jsat.2013.07.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Graubard BI, Korn EL. Predictive margins with survey data. Biometrics. 1999;55:652–659. doi: 10.1111/j.0006-341x.1999.00652.x. [DOI] [PubMed] [Google Scholar]
  • 35.McCaffrey DF, Griffin BA, Almirall D, et al. A tutorial on propensity score estimation for multiple treatments using generalized boosted models. Statistics in Medicine. 2013;32:3388–3414. doi: 10.1002/sim.5753. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Ridgeway G, McCaffrey DF, Morral AR. Toolkit for the Weighted Analysis of Nonequivalent Groups: A Tutorial for the twang Package. 2006. [Google Scholar]
  • 37.Ridgeway G, McCaffrey DF, Morral AR, et al. Toolkit for Weighting and Analysis of Nonequivalent Groups: A Tutorial for the Twang Package. 2012. [Google Scholar]
  • 38.Cohen J. A Power Primer. Psychological Bulletin. 1992;112:155–159. doi: 10.1037//0033-2909.112.1.155. [DOI] [PubMed] [Google Scholar]
  • 39.Graubard BI, Korn EL. Predictive margins with survey data. Biometrics. 1999;55:652–659. doi: 10.1111/j.0006-341x.1999.00652.x. [DOI] [PubMed] [Google Scholar]
  • 40.Dennis ML, Ives M, Funk R. GAIN-I encyclopedia of supplemental documentation on scales and other calculated variables. Bloomington, IL: Chestnut Health Systems; 2008. Available from: http://www.chestnut.org/LI/gain/Scales_Crosswalks/GI_Scales_and_Variables.xls. [Google Scholar]
  • 41.Agency for Healthcare Research and Quality (AHRQ) National Quality Measures Clearninghouse: Selecting Structure Measures for Clinical Quality Measurement. Undated [cited 2014 April 28]; Available from: http://www.qualitymeasures.ahrq.gov/tutorial/StructureMeasure.aspx.
  • 42.Harrison PA, Asche SE. Outcomes monitoring in Minnesota: treatment implications, practical limitations. Journal of Substance Abuse Treatment. 2001;21:173–83. doi: 10.1016/s0740-5472(01)00199-4. [DOI] [PubMed] [Google Scholar]
  • 43.Steiner P, editor. Implications of covariate measurement error for propensity scores; 2013 Fall Meeting of the Society for Research on Educational Effectiveness; 2013 March 8; Washington, DC. 2013. [Google Scholar]
  • 44.Scott CK. A replicable model for achieving over 90% follow-up rates in longitudinal studies of substance abusers. Drug and Alcohol Dependence. 2004;74:21–36. doi: 10.1016/j.drugalcdep.2003.11.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Griffin BA, McCaffrey D, Ramchand R, et al. Assessing the Sensitivity of Treatment Effect Estimates to Differential Follow-Up Rates: Implications for Translational Research. Health Services and Outcomes Research Methodology. 2012;12:84–103. doi: 10.1007/s10742-012-0089-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES