Substance Use, Education, Employment, and Criminal Activity Outcomes of Adolescents in Outpatient Chemical Dependency Programs

Abstract

Although the primary outcome of interest in clinical evaluations of addiction treatment programs is usually abstinence, participation in these programs can have a wide range of consequences. This study evaluated the effects of treatment initiation on substance use, school attendance, employment, and involvement in criminal activity at 12 months post-admission for 419 adolescents (aged 12 to 18) enrolled in chemical dependency recovery programs in a large managed care health plan. Instrumental variables estimation methods were used to account for unobserved selection into treatment by jointly modeling the likelihood of participation in treatment and the odds of attaining a certain outcome or level of an outcome. Treatment initiation significantly increased the likelihood of attending school, promoted abstinence, and decreased the probability of adolescent employment, but it did not significantly affect participation in criminal activity at the 12-month follow-up. These findings highlight the need to address selection in a non-experimental study and demonstrate the importance of considering multiple outcomes when assessing the effectiveness of adolescent treatment.

Introduction and Background

According to the National Survey on Drug Use and Health,¹ 8.3% of youths between the ages of 12 and 17 were diagnosed with substance abuse or dependence in 2005. Research has demonstrated that substance abuse or dependence among adolescents is associated with a variety of problems related to mental health status, academic performance, family functioning, and risk of infectious diseases.^2,3 These adolescents also pose a high risk of continuing abuse into adulthood. As chemical dependency (CD) treatment for adolescents has become more specialized, the focus of interventions has been widened in an attempt to address some of these other problem areas.^2,4,5 The American Academy of Child and Adolescent Psychiatry has recommended that interventions target problems in family, vocational, and legal domains in addition to providing treatment for substance use problems.⁶ Treatment success depends not only on the attainment of abstinence by the adolescent but also on spillover effects that benefit the adolescent, his/her family, and society as a whole.

Clinical evaluations generally view the benefits of CD treatment in terms of abstinence and reduced substance-related problems. Because abstinence may not be a realistic outcome for all adolescents, advocates of a harm reduction approach consider treatments that address the consequences of substance use to be effective interventions even if abstinence is not attained.^7,8 In addition, economic evaluations of CD treatment for adults have identified benefits to society that include reductions in criminal activity, decreased use of healthcare, and higher employment earnings.^9,10 While some of the expected social benefits of adolescent treatment programs overlap with those of adult programs (i.e., reductions in criminal activity and healthcare utilization), improvement in other domains may be more relevant for adolescents than adults and vice versa.⁵ For instance, it is desirable that treatment programs for adolescents reinforce school engagement and educational attainment. Investment in education during adolescence carries potential benefits such as improved lifetime earnings, cognitive functioning, and health.^11–13 Increased employment, on the other hand, may be a less desirable outcome in treatment programs for adolescents. Several studies have found positive associations between adolescent employment or work intensity and substance abuse.^14–16 The association appears to be influenced by differential selection, whereby adolescents with educational disengagement, problem behaviors, and substance abuse problems choose to be employed.^17–19 There is also some evidence that increased exposure of working adolescents to delinquent peers may lead to substance use.²⁰

This paper evaluated the effectiveness of CD treatment for adolescents based on two clinical outcomes (abstinence and decrease in substance use problems) and three behavioral outcomes (school enrollment, employment, and involvement in criminal activity).^*,²¹ The sample consisted of 419 adolescents recruited at four Kaiser Permanente (KP) Northern California Chemical Dependency Recovery Programs (CDRP). Treatment was expected to promote abstinence, decrease the number of symptoms related to substance use, increase the probability of school attendance, and decrease the probability of involvement in criminal activity. The effect of treatment on employment is an open empirical question. On the one hand, employment may have an adverse influence on adolescents because it increases contact with peer groups that do not facilitate sobriety, permits access to funds that can be used to obtain substances, and shifts the adolescent’s focus from school involvement.²² On the other hand, certain types of employment—in particular, jobs that are not time-intensive—may teach responsible behavior and be desirable for adolescents doing well at school.

The gold standard to test these hypotheses would be a randomized controlled trial that balanced observed and unobserved characteristics between those who received treatment and those who did not.²³ When treatment is randomly assigned, different outcomes between the two groups could be attributed to the treatment received. In the absence of randomization, however, the decision to initiate treatment and the corresponding outcomes are jointly determined, and changes in outcomes may reflect unmeasured differences between individuals in the treatment and control groups. In naturalistic settings, simple multivariate regression techniques, such as probit, logit, or ordinary least-squares regressions, can lead to biased estimates of treatment effects.

Because the study at hand used a non-randomized observational design, the present analysis was conducted using instrumental variables (IV) estimation, an econometric technique designed to address selection bias. The technique simulates randomization on the basis of one or more instrumental variables that are correlated with treatment but uncorrelated with the outcome being investigated. IV estimation has been applied to evaluate traditional medical interventions such as treatment of acute myocardial infarction²⁴ as well as various mental health and behavioral interventions.^25–28 To the authors’ knowledge, this is the first study to use an IV technique to evaluate outcomes of a CD program for adolescents. Our results demonstrate the importance of incorporating estimation techniques that address selection into program evaluations of non-randomized trials.

Data

Sample

Kaiser Permanente Northern California (NCKP) is a not-for-profit managed care integrated health care delivery system with approximately 3.4 million members. Eighty-eight percent of the members are commercially insured, whereas 10% receive coverage through Medicare and 2% through Medicaid. The majority of the membership has some college education and earns middle-class household incomes.²⁹ Outpatient CD and mental health services are provided internally within the health plan.

The sample consisted of 419 adolescents at treatment intake at four NCKP CD programs that provide intensive structured outpatient CD treatment (Table 1). The programs are representative of intensive outpatient adolescent treatment programs throughout the USA. These programs focus on abstinence from alcohol and other drug use as well as larger aspects of recovery.³⁰ Group and individual sessions address substance-related issues such as developing non-substance using social supports, improving family relationships, improving school attendance and performance, decreasing criminal behavior, and making positive decisions regarding employment and on-going education.^31,32

Table 1.

Individual, program, and community characteristics, by site

	Site 1	Site 2	Site 3	Site 4
Sample (N=419)	51	141	100	127
Treatment exposure
Initiated treatment (%)**	66.67	63.83	78.00	81.10
Average length of stay (days) for adolescents who initiated treatment	113.38 (110.46)	127.62 (108.44)	84.90 (62.68)	80.85 (57.88)
Individual characteristics
White (%)**	29.41	60.28	54.00	41.27
African American (%)**	49.02	85.11	5.00	19.05
Hispanic (%)*	11.77	15.60	28.00	20.63
Other race (%)	9.80	15.60	13.00	19.05
Female (%)	35.29	31.91	40.00	31.50
Age	16.20 (1.20)	16.13 (1.32)	15.95 (1.22)	16.30 (1.22)
Lived in residential program (%)	11.77	10.64	5.00	4.72
Measure of substance abuse severity at baseline*	3.30 (2.44)	4.70 (3.50)	4.56 (3.17)	4.945 (3.381)
Parent graduated college (%)	45.65	24.09	13.00	20.00
Parent or family members had alcohol/drug problems (%)**	41.18	46.10	49.00	38.58
Referral from legal system (%)*	21.57	27.66	31.00	43.31
Program characteristics
Treatment module	Intensive, structured, outpatient treatment
Pretreatment assessment offereda	Yes	No	Yes	Yes
Physical proximity of psychiatric and CD departments (jointly located/separated)a	Separated	Separated	Jointly located	Jointly located
Community characteristics
Population aged 25 and over with less than high school diploma (%)**	18.50	11.95	12.55	18.12
Unemployment rate in civilian labor force (%)**	7.22	5.01	4.77	6.01
Median household income (1999)	$59,282 ($36,946)	$54,385 ($17,744)	$57,880 ($14,916)	$55,812 ($15,442)

Standard deviations in parentheses for continuous variables. Kruskal–Wallis test compared distributions across sites.

^*Statistically significant at p<0.05

^**Statistically significant at p<0.01

^aFrom Sterling and Weisner (2005)³⁴

The first phase of treatment begins with intake/assessment and is followed by group treatment sessions held three times per week for 8 weeks. During the next 8 weeks, clients attend two group sessions per week to focus on continuing recovery and relapse prevention. The last phase, aftercare, lasts up to 1 year and involves a weekly group session. Parents are usually required to attend twice a week in the first phase and weekly thereafter.³³

Between May of 2000 and June of 2002, all patients entering the adolescent program were recruited to participate in the study. Parents provided written consent and adolescents verbally consented. At baseline, adolescents completed a computerized self-administered questionnaire as well as a paper-and-pencil questionnaire, and a parent or guardian completed a paper-and-pencil questionnaire. Six and twelve months later, patients and parents were interviewed by telephone. A more detailed description of the program and study protocols can be found in papers by Campbell et al.³³ and Sterling and Weisner.³⁴ Human subjects approval was obtained annually from institutional review boards at the University of California, San Francisco, and the Kaiser Foundation Research Institute.

Measures

Outcomes

Two clinical outcomes and three behavioral outcomes were analyzed in this paper. All outcomes were constructed on the basis of the adolescents’ responses to the 12-month follow-up questionnaires. The first clinical outcome was abstinence, which took the value of 1 if the adolescent reported not having consumed any substances (excluding tobacco) in the past 30 days and 0 otherwise. While abstinence is an important outcome of CD programs, it is a difficult target for some adolescents. A more widely attainable program goal is to decrease the severity of substance use problems. To assess this goal, a measure of substance use problem severity was constructed from responses to 14 questions from the Comprehensive Adolescent Severity Inventory, a semi-structured self-report questionnaire based on the Addiction Severity Index that measures health and functioning in various life areas.^24,35,36 The problem severity measure was based on the total count of “yes” responses to 14 substance dependence and abuse symptom questions that referred to the past 30 days at 12 months follow-up.³³ A lower number for this measure corresponds to less severe substance use problems.

In addition to providing treatment for substance use problems, adolescent CD interventions directly target family, vocational, and legal problems.⁶ To reflect these dimensions, three behavioral outcomes were constructed based on responses to the following questions at 12 months follow-up:

1. Are you currently going to school or any other educational program?

2. In the past 6 months (since our last telephone interview with you), have you been picked up or charged with any legal offense?

3. Do you have a job or paid employment?

Each of the outcomes was set equal to 1 if the adolescent responded “yes” and zero otherwise. As noted earlier, treatment was expected to promote and improve school attendance, to decrease criminal activity, and to have an ambiguous effect on adolescent employment.^14–16,20 Unfortunately, we did not have ideal measures of employment status and were therefore unable to determine whether an effect on employment worked through intensity or participation per se. Focus groups held as part of the study revealed that the jobs being held by the adolescents attending the CD program typically did not promote recovery or enhance personal development. Clinical staff felt that many of the jobs exposed adolescents to peer groups that did not facilitate sobriety, provided them access to funds that could be used to obtain substances, and often shifted the adolescent’s focus from other important activities such as school. The comments received from these focus groups and the fact that most of the sample was still of school age at the time of the 12 months follow-up led us to expect some decrease in employment at follow-up.

Treatment initiation

Treatment initiation, the explanatory variable of interest, was set equal to 1 (0 otherwise) if the patient had at least two visits within 60 days after intake (the first visit after intake was an orientation session). The literature does not clearly define when treatment initiation begins, but this definition was developed to be consistent with the programs’ aims and was used by the program clinicians and NCKP’s regional Adolescent Chemical Dependency Coordinating Committee, which oversees the adolescent CD programs in the area.^33,34 The number of visits was extracted from CD utilization data from the KP administrative database.

Treatment initiation was selected as the primary explanatory variable for two reasons. First, there was no defined point in time for treatment completion. Although the program can last up to 1 year including aftercare, time in treatment varied depending upon individual characteristics. Second, the instrumental variables available to address selection were more predictive of treatment initiation than length of stay in treatment. The robustness checks presented later in the paper also include a discussion of several variations of the analysis using other measures of treatment intensity such as length of stay in days and a dichotomous variable indicating a length of stay above the median.

Control variables

Outcomes at baseline

The objective of the study was to identify whether treatment initiation contributed to decreased substance use problems and, at the same time, helped increase, maintain, or decrease school attendance, employment, and involvement in criminal activity. To measure change, each of the specifications included a dichotomous indicator for the outcome variable at baseline. The analyses of school attendance, employment, and substance use problem severity at follow-up included a similar baseline dichotomous indicator. When criminal activity at follow-up was the outcome, the baseline control was a dummy variable indicating whether the adolescent had ever been on probation or parole, picked up or charged with any legal offense, or in jail or detention. A specific control for abstinence at baseline was not included in the analysis of abstinence because there was no variation in this outcome at baseline; all of the adolescents were using substances at intake.

Individual characteristics

The models included a continuous measure for age at baseline as well as indicators for gender and race/ethnicity (white, African-American, Hispanic, and other race/ethnicity). A binary measure was included in the model for individuals who attended a residential program (general hospital, residential psychiatric treatment, residential drug or alcohol treatment, group home, halfway house, monitored independent living, jail, detention center) during the 6 months before baseline. All of the models controlled for substance use problem severity 6 months before baseline using the measure adopted from the analysis of the same group of adolescents by Campbell et al.³³

Family characteristics

Family environment has been shown to impact substance use by adolescents both positively and negatively.³⁷ Our models contained an indicator for whether the parent had graduated from college. The analysis also contained a binary measure for familial drug or alcohol problems.

Referral sources

Motivation to adhere to treatment may vary depending on the primary reason for attending treatment. Many adolescents who seek treatment are ordered to attend by the court system or by their parents.² The core analysis included an indicator for referral from the legal system. Because the data also contained information about other referral sources (e.g., school or mental health provider), these were used to test the robustness of the core results. If adolescents selected one or more referral sources, missing responses to other questions about referral were coded as zero.

Community level and site characteristics

Although the four treatment sites were similar in terms of structure and service delivery, the locations and surrounding communities differed. Therefore, in addition to controlling for individual characteristics, site dummies^* and other related variables were added to account for program and community level heterogeneity. The adolescents’ home addresses at baseline were linked to geocoded census 2000 data. Three variables based on these data were entered into our core models: the percentage of the population aged 25 and over with less than a high school diploma, median household income in 1999, and the percentage unemployed in the civilian labor force (aged 16 and over).

Instrumental variables

For identification reasons, the application of IV techniques requires the use of at least one variable or “instrument” that is correlated with the potentially endogenous explanatory variable³⁸ (in this case, treatment initiation) but not significantly related to the outcome of interest. Distance to treatment and other travel barriers were likely to be associated with treatment initiation but unlikely to be significantly correlated with substance use, schooling, criminal activity, or employment. Other studies have successfully used geographic distance or geographic location as instrumental variables.^26,39,40

Mapquest’s driving directions function estimated the distance between the treatment center and the zip code of the adolescent’s residence in miles and the travel time in minutes. The average of the distance to the treatment facility from the adolescent’s home zip code and to the home zip code from the treatment facility was calculated.^* Another instrument in the analysis was based on parents’ responses at baseline to the question, “How difficult is it for your child to travel to or get transportation to this program?” A variable was constructed with a value of 1 if the parent responded “very difficult” and zero otherwise.

Descriptive statistics

Although the programs had similar staffing levels and treatment approaches, four sites were recruited for the study to ensure that the sample was ethnically diverse (i.e., more Hispanics) and to increase the study’s generalizability. Table 1 shows differences in client profiles, program characteristics, and community characteristics by site. Sites 1 and 2 had the lowest proportion of adolescents initiating treatment, but had a relatively high retention of clients among those who initiated. Sites 3 and 4 had higher rates of treatment initiation but shorter average lengths of stay. In addition to length of stay and treatment initiation, there were significant differences across sites in terms of percentage of white clients, African American clients, and Hispanic clients; severity of substance use problem; referral source; and community-level characteristics such as the unemployment rate and the percentage of the population without a high school diploma.

Table 2 displays the mean values of each variable by treatment initiation status. Research participants were between 12 and 18 years of age, with a mean age of 16 years. Forty-nine percent were white, 16% African American, 20% Hispanic, and 15% other race (Asian or Native American). There were very few statistically significant (p<.05) differences in individual characteristics between those who initiated treatment and those who did not. Adolescents initiating treatment had less difficulties traveling to treatment. Adolescents in Site 2 were less likely to initiate treatment than participants in other sites, whereas Site 4 was the most successful in recruiting clients. For a more comprehensive analysis of enabling and disabling factors for treatment initiation using this same dataset, see Sterling et al.⁴¹

Table 2.

Mean values for all variables, by treatment initiation status

Variable	Full sample (N=419; 100%)	Initiated treatment (N=305; 72.79%)	Did not initiate treatment (N=114; 27.21%)	N
Outcomes (12-month follow-up)
Attended school (%)	82.55	81.79	84.62	384
Involved in criminal activity (%)	22.66	23.21	21.15	384
Employed (%)	40.63	40.36	41.35	384
Abstinent (%)**	0.47	0.52	0.36	384
Substance use problem severity (%)	0.96 (1.77)	0.85 (1.57)	1.27 (2.22)	383
Instrumental variables for treatment initiation
Distance to treatment (miles)	10.47 (9.33)	10.02 (9.05)	11.65 (10.00)	417
Very difficult traveling to treatment (%)**	7.3	4.5	14.81	397
Difficult traveling to treatment (%)**	37.78	33.91	48.15	397
Controls
Attended school at baseline (%)	0.83	0.82	0.87	415
Ever involved in criminal activity pre baseline (%)	0.51	0.54	0.45	417
Employed at baseline (%)	0.56	0.55	0.61	418
Substance use problem severity at baseline (%)	4.57 (3.3)	4.68 (3.21)	4.28 (3.52)	417
Site 1 (%)	12.17	11.15	14.91	419
Site 2 (%)**	33.65	29.51	44.74	419
Site 3 (%)	23.87	25.57	19.3	419
Site 4 (%)*	30.31	33.77	21.05	419
Percentage of population aged 25 and over with less than high school diploma (%)	14.81	15.43	13.18	404
Percentage of civilian labor force unemployed (%)	5.54	5.63	5.31	404
Median household income (1999)	$56,257.98 ($19,948.09)	$55,939.45 ($18,676.32)	$57,098.78 ($23,042.47)	404
Female (%)	34.13	32.79	37.72	419
Age	16.15 (1.25)	16.13 (1.29)	16.19 (1.17)	419
White (%)	49.28	50.66	45.61	418
African American (%)	15.79	17.11	12.28	418
Hispanic (%)	19.62	18.75	21.93	418
Other race (%)	15.31	13.49	20.18	418
Lived in residential program (%)	7.64	7.87	7.02	419
Parent graduated college (%)	22.58	20.82	27.27	403
Parent or family members had alcohol/drug problems (%)	43.91	45.25	40.35	419
Referral from legal system (%)	32.54	34.21	28.07	418

Standard deviations in parentheses for continuous variables. Kruskal–Wallis test compared distributions for those who initiated treatment with those who did not.

^*Statistically significant at p<0.05

^**Statistically significant at p<0.01

Missing data

Twelve-month follow-up information was unavailable for 35 adolescents. Excluding adolescents without any 12-month follow-up information left 384 adolescents in the final analysis sample. Fifteen adolescents with 12-month follow-up information did not have 15-digit geocodes. One observation was missing the type of referral, another did not report ethnicity, and a third did not have information on substance use severity at baseline. Two observations were missing information on distance to treatment. In addition, 11 parents did not complete the parent questionnaire, two did not respond to the question about parental education, and five did not answer whether it was difficult to travel to treatment. Missing data were imputed using mean imputation for the core analysis. Alternate methods and results for imputation are discussed in the sensitivity analysis.

Methods

The goal of this analysis was to estimate the effect of CD treatment initiation on five outcomes at the 12-month follow-up: abstinence, substance use problem severity, school attendance, involvement in criminal activity, and employment. Except for substance use problem severity, which was a continuous measure, all other outcomes were dichotomous. Single-equation models were first used to estimate the relationship between treatment initiation and the selected outcomes. The general estimating equation was

$$Y_{if}^{\ast }={\alpha }_0+{\alpha }_1{T}_i+{\alpha }_2{Y}_{ib}+X_{ib}^{\prime }{\alpha }_3+u_i$$ (1A)

where i stands for observation i, subindex b indicates that the variable was measured at baseline, and subindex f represents measurement at follow-up. $Y_{if}^{\ast }$ is a latent measure of outcome Y for patient i at the 12-month follow-up, T is an indicator for treatment initiation, and X is a vector of the adolescent’s demographic and other covariates. Y_ib is a control for the outcome at baseline. Finally, α₀, α₁, α₂, and α₃ are a set of coefficients to estimate, and u_i is a random error term.

When Y* is continuous (as in the case of substance use problem severity), Y* = Y and Eq. 1A takes the form of a linear regression. For dichotomous outcomes (abstinence, school attendance, employment, and criminal activity), we assume that Y=1 if Y*>0 and that Y=0 otherwise. In these cases, Eq. 1A can be rewritten as

$$Pr(Y_{if}=1\mid T_i,Y_{ib},X_{ib})=f({\alpha }_0+{\alpha }_1{T}_i+{\alpha }_2{Y}_{ib}+X_{ib}^{\prime }{\alpha }_3)$$ (1B)

where f(.) can be either the logistic function (logit regression) or the normal density function (probit regression). We chose to work with probit regression because the methods we used to address selection are also based on the normal density. Equation 1B, however, could have also been estimated using a logit model.

The coefficient of interest is α₁, the effect of treatment initiation on each of the outcomes. Ideally, if all the relevant variables that affect the outcome could be taken into account, α₁ would provide an unbiased estimate of the effect of treatment initiation on the dependent variable. Yet, there may be unobserved characteristics that differentiate those who initiated treatment from those who did not. A limitation of estimating Eqs. 1A or 1B with single-equation regression (either linear regression or probit/logit regression) is that single-equation techniques fail to address selection into treatment because of differences in unmeasured dimensions of severity, motivation, or other unobserved characteristics of the adolescent. The coefficient estimates for the treatment initiation variable will be biased if variables that are correlated with treatment and determine Y are omitted from Eqs. 1A and 1B. The expected direction of this bias cannot be determined a priori.²⁶ Adolescents with high motivation, for instance, may be more likely to initiate and comply with treatment. These adolescents, however, would also be more likely to decrease substance use, attend school, respect the law, or achieve other successful outcomes in various life areas. The bias of single-equation estimation in this case would be in the positive direction. A good outcome may not be the exclusive result of the treatment intervention but would capture the higher motivation of those entering treatment. Conversely, adolescents with greater unobserved severity may be more likely to attend treatment. This unmeasured severity would cause a downward bias in the single-equation estimate. Treatment would appear to have a smaller impact on school attendance, criminal activity, or employment than the true effect.^26,42

IV estimation attempts to remove the unobserved variation potentially correlated with the error term in Eqs. 1A and 1B from the main explanatory variable.⁴³ To use IV estimation, valid instrumental variables are needed that significantly predict the likelihood of initiating treatment without predicting the outcomes of interest.^*,⁴⁴ Adolescents would essentially be sorted into two treatment groups based on variation in these instrumental variables.²⁵ As mentioned in the “Data” section, this analysis used distance between the adolescent’s residence and the location of the CDRP and/or parental assessment of the difficulty of traveling to treatment as instrumental variables. The rationale for the choice of these instruments was that clients who lived far away from the CDRP or who had other travel difficulties would face higher barriers to treatment and be less likely to initiate. At the same time, these factors should not significantly influence the outcomes under study. Table 1 shows that a higher proportion (p<0.01) of parents of adolescents who did not initiate treatment reported that it was very difficult to travel to treatment.

IV estimation can be implemented using different techniques that depend on the distribution of the variables under analysis. Two-stage least squares, the most commonly used IV technique, is appropriate when the outcome of interest is continuous. The IV estimation for substance use problem severity included the following equations:

$$T_i={\gamma }_0+Z_{ib}^{\prime }{\gamma }_1+X_{ib}^{\prime }{\gamma }_2+{\upsilon }_i,$$ (2)

$$Y_{if}={\beta }_0+{\beta }_1{T}_i^{\ast }+{\beta }_2{Y}_{ib}+X_{ib}^{\prime }{\beta }_3+{\epsilon }_i,$$ (3)

where Z is a set of excluded instrumental variable(s) that identified the treatment initiation equations, and $T_i^{\ast }$ is the predicted value of treatment initiation from the first stage reduced-form regressions. Equation 3 estimates the likelihood of having a certain level of substance use problem severity given the first stage prediction of treatment initiation.

When both the outcome and treatment variables are dichotomous, a recursive bivariate probit model that assumes joint determination of treatment initiation and outcome leads to the consistent estimation of the treatment effect.⁴³ A recursive bivariate probit model for four of the five outcomes was specified as follows:

$$\begin{array}{l}{Y}_{if}^{\ast }={\beta }_0+{\beta }_1{T}_i+{\beta }_2{Y}_{ib}+X_{ib}^{\prime }{\beta }_3+{\epsilon }_i\\ Y_i=1if{Y}_i^{\ast }>0,0\text{otherwise}\end{array}$$ (4)

$$\begin{array}{c}{T}_i^{\ast }={\gamma }_0+Z_{ib}^{\prime }{\gamma }_1+X_{ib}^{\prime }{\gamma }_2+{\upsilon }_i\\ T_i=1if{T}_i^{\ast }>0,0\text{otherwise}\\ E[{\epsilon }_i]=E[{\upsilon }_i]=0\\ \mathit{Var}[{\epsilon }_i]=\mathit{Var}[{\upsilon }_i]=1\\ \mathit{Cov}[{\epsilon }_i,{\upsilon }_i]=\rho \end{array}$$ (5)

Equation 5 estimated the likelihood of initiating treatment, whereas Eq. 4 estimated the likelihood of abstaining from drugs or alcohol, attending school, participating in criminal activity, or being employed at the 12-month follow-up. Z′ represented the instrumental variables for treatment initiation, and ε and υ were error terms for the outcome and treatment initiation equations. Under the assumption that ε and υ are jointly normally distributed with means equal to 0, variances equal to 1, and correlation equal to ρ, this system of equations can be estimated as a recursive bivariate probit model using maximum likelihood methods.

Greene⁴³ and others have shown that if the error terms are not correlated, then the model above reduces to independent probit equations that can be estimated separately. If the null hypothesis of ρ=0 is rejected (i.e., the error terms of Eqs. 4 and 5 are correlated), then the recursive bivariate probit will generate unbiased and consistent estimates even when the same unobservable factors simultaneously affect the outcomes of interest and treatment initiation. Although, technically, these models can be estimated without any exclusion restrictions, the existence of valid instruments allows identification of the model without relying on untestable assumptions about the distribution of the error terms.

For each outcome, two tests were conducted to assess whether the instrumental variables were reliable and valid to identify the treatment initiation equation independently of the outcome equations of interest (see results for these tests in Table 3). First, instrumental variables reliability was assessed by evaluating the statistical significance of the instruments Z in Eqs. 2 and 5 using Wald and F tests. The instrumental variables were individually or jointly significant (p<0.05) in explaining the variation in treatment initiation and predicted treatment initiation in the expected direction (i.e., distance to treatment and difficulty traveling were negatively related to treatment initiation). Second, when appropriate, tests of exclusion restrictions⁴⁵ were conducted to ensure that the instrumental variables were excludable from the outcome equations (Eqs. 3 and 4).^* Various combinations of instrumental variables were used in each of the specifications depending on which combination passed the validity tests.

Table 3.

Selected first stage results, tests of reliability and validity of instrumental variables

	School attendance at 12 months follow-up	Criminal activity at 12 months follow-up	Employment at 12 months follow-up	Substance use problem severity at 12 months follow-up	Abstinence at 12 months follow-up
Method	Bivariate probit	Bivariate probit	Bivariate probit	Two-stage least-squares	Bivariate probit
Instrumental variables	Distance to treatment facility	Distance to treatment facility; very difficult traveling to treatment	Distance to treatment facility	Distance to treatment facility; very difficult traveling to treatment	Distance to treatment facility; very difficult traveling to treatment
First stage dependent variablea	Treatment initiation	Treatment initiation	Treatment initiation	Treatment initiation	Treatment initiation
Distance to treatment Facility	−0.013* (0.006)	−0.011 (0.008)	−0.013* (0.006)	−0.003 (0.003)	−0.013 (0.009)
Very difficult traveling to treatment	–	−0.792** (0.273)	–	−0.277** (0.102)	−0.734* (0.293)
Tests of reliability and validity of instrumental variables
Joint significance of instrumental variablesb	4.02* (0.045)	11.31** (0.004)	5.79* (0.016)	4.91** (0.008)	10.98** (0.004)
Test of exclusion restrictionsb	n/a	0.45 (0.504)	n/a	0.6781	0.25 (0.616)

^*Statistically significant at p<0.05

^**Statistically significant at p<0.01

^aCoefficients and robust standard errors (in parentheses) are reported. Regressions control for site, percentage in community with less than high school degree, percentage in community unemployed, median household income in community, gender, age, race, participation in residential program in 6 months before baseline, measure of substance abuse severity, parent’s education level, familial alcohol/drug problem, and referral from the legal system.

^bWald test employed for bivariate probit models. Chi-square values and p values (in parentheses) are reported. F statistics and p values (in parentheses) are reported for two-stage least-squares models. When only one instrument is used, the joint significance test is the same as the individual test for the single variable.

^cChi-square values and p values (in parentheses) are reported for bivariate probit models, and Hansen J statistic is reported for two-stage least-squares models.

As a final point, models that estimate simultaneous equations (either two-stage least-squares or bivariate probit models) are less efficient statistically than single equation models. When the sample size is relatively small, as in this analysis, the statistical power of the IV models raises some concern. Failure to uncover statistically significant effects may not necessarily imply that effects are not present. An additional problem associated with IV estimation is that identification may not be achieved when instruments are weak (i.e., when the instruments do not estimate treatment initiation with enough precision). This concern, however, is more likely to be problematic in two-stage least-square models than in bivariate probit models where identification is also achieved through nonlinearities.

Results

Estimation results for treatment outcomes

Tables 4 and 5 report selected estimation results for the single-equation estimation (before addressing selection) and for the IV estimation, which accounts for selection bias. The tables display coefficient estimates, standard errors (in parentheses), and marginal effects^* (in squared brackets) which were computed at the sample means. In the single equation estimation, treatment initiation did not significantly predict school attendance, employment, or criminal activity (Table 4). Results showed that those who initiated treatment were significantly more likely to be abstinent at follow-up and had lower substance use problem severity (Table 5). The effect of treatment initiation on substance use problem severity approached significance at the 5% level.

Table 4.

Selected estimation results for education, criminal activity, and employment outcomes^a

Estimation method	School attendance at 12 months follow-up		Criminal activity at 12 months follow-up		Employment at 12 months follow-up
	Probit	Bivariate probit	Probit	Bivariate probit	Probit	Bivariate probit
Treatment initiation	−0.075	1.539**	−0.039	−0.362	0.018	−1.563**
	(0.195)	(0.217)	(0.174)	(0.755)	(0.158)	(0.245)
	[−0.014]	[0.482]	[−0.011]	[−0.109]	[0.007]	[−0.558]
Dependent variable at baseline	0.184 (0.224)	0.309 (0.203)	0.277 (0.174)	0.254 (0.177)	0.580** (0.158)	0.326 (0.179)
Site 1b	0.302 (0.337)	0.210 (0.288)	−0.004 (0.279)	−0.100 (0.299)	0.068 (0.249)	0.057 (0.229)
Site 3b	−0.075 (0.230)	−0.265 (0.202)	−0.047 (0.198)	−0.011 (0.219)	0.412* (0.196)	0.480** (0.171)
Site 4b	0.258 (0.241)	0.014 (0.210)	0.109 (0.201)	0.162 (0.225)	0.195 (0.190)	0.345* (0.172)
Percentage with less than high school degree	−0.304 (1.059)	−0.737 (0.823)	−1.607 (1.007)	−1.375 (1.117)	−1.351 (0.923)	−0.360 (0.836)
Percentage unemployed	−3.141 (3.245)	−2.101 (2.291)	−1.138 (2.582)	−2.579 (2.722)	−1.421 (2.468)	−2.335 (2.242)
Median household income (1999)	0.044 (0.061)	0.045 (0.053)	−0.027 (0.055)	−0.020 (0.054)	−0.092 (0.049)	−0.074 (0.049)
Female	0.224 (0.190)	0.266 (0.160)	−0.519** (0.172)	−0.489** (0.175)	0.224 (0.151)	0.037 (0.158)
Age	−0.562** (0.111)	−0.446** (0.099)	−0.043 (0.059)	−0.016 (0.062)	0.265** (0.067)	0.197** (0.066)
African Americanc	0.053 (0.292)	−0.023 (0.254)	0.274 (0.240)	0.302 (0.257)	−0.262 (0.223)	−0.059 (0.217)
Hispanicc	−0.303 (0.231)	−0.090 (0.192)	−0.182 (0.210)	−0.215 (0.217)	−0.216 (0.193)	−0.268 (0.171)
Other ethnicityc	−0.051 (0.236)	0.188 (0.220)	0.032 (0.214)	−0.090 (0.246)	−0.416 (0.224)	−0.488** (0.177)
Lived in residential program	−0.383 (0.294)	−0.308 (0.294)	0.329 (0.294)	0.373 (0.303)	0.059 (0.274)	0.007 (0.250)
Substance use problem severity at baseline	0.057* (0.029)	0.025 (0.025)	0.032 (0.025)	0.028 (0.026)	−0.016 (0.021)	0.001 (0.019)
Parent graduated college	−0.054 (0.231)	−0.029 (0.192)	−0.166 (0.193)	−0.163 (0.192)	0.048 (0.182)	−0.017 (0.167)
Familial alcohol/drug problems	−0.300 (0.180)	−0.302 (0.157)	0.138 (0.151)	0.143 (0.152)	−0.123 (0.148)	−0.042 (0.135)
Referral from legal system	−0.674** (0.191)	−0.583** (0.178)	0.205 (0.179)	0.227 (0.182)	−0.228 (0.166)	−0.052 (0.156)
Missing geocoded data	−0.325 (0.339)	−0.326 (0.335)	−0.051 (0.377)	0.038 (0.396)	1.277** (0.453)	0.895** (0.344)
Missing parent’s education	0.386 (0.473)	0.270 (0.308)	−0.005 (0.406)	−5.534** (0.295)	0.926* (0.386)	0.692 (0.384)
Test of exogeneity of treatment initiation (p value)d		(0.001)**		(0.651)		(0.309)
N	384	382	384	364	384	382

^*Statistically significant at p<0.05

^**Statistically significant at p<0.01

^aCoefficients, robust standard errors (in parentheses), and marginal effects (in brackets) are reported. Instrumental variables include distance from residence to treatment facility and parental assessment of difficulty traveling to treatment (see Table 3).

^bComparison is Site 2 (the largest site).

^cComparison is whites.

^dWald test employed to test H₀: ρ=0 for bivariate probit models. Chi-square values and p values (in parentheses) are reported. The null for this test is that treatment initiation is uncorrelated with the residuals in the outcome equation.

Table 5.

Selected estimation results for substance use outcomes^a

Estimation method	Substance use problem severity at 12 months follow-up		Abstinence at 12 months follow-up
	OLS	Two-stage least-squares	Probit	Bivariate probit
Treatment initiation	−0.435	−2.399	0.486**	0.987
	(0.247)	(1.688)	(0.155)	(0.788)
			[0.189]	[0.363]
Site 1b	−0.245 (0.304)	−0.237 (0.341)	−0.252 (0.253)	−0.212 (0.257)
Site 3b	−0.180 (0.264)	−0.007 (0.330)	−0.006 (0.181)	−0.004 (0.214)
Site 4b	−0.160 (0.270)	0.080 (0.366)	−0.074 (0.180)	−0.070 (0.212)
Percentage with less than high school degree	−0.275 (1.077)	0.340 (1.161)	0.189 (0.866)	−0.059 (0.939)
Percentage unemployed	4.137 (4.462)	2.589 (4.368)	−1.279 (2.341)	−1.162 (2.505)
Median household income (1999)	−0.042 (0.053)	−0.041 (0.066)	0.029 (0.047)	0.024 (0.047)
Female	−0.137 (0.216)	−0.294 (0.231)	0.190 (0.146)	0.224 (0.149)
Age	−0.072 (0.075)	−0.070 (0.090)	−0.129* (0.054)	−0.147* (0.060)
African Americanc	−0.268 (0.221)	−0.001 (0.352)	0.176 (0.216)	0.159 (0.248)
Hispanicc	0.094 (0.260)	0.066 (0.298)	−0.019 (0.179)	−0.038 (0.195)
Other ethnicityc	0.112 (0.304)	−0.131 (0.343)	0.090 (0.196)	0.191 (0.222)
Lived in residential program	−0.221 (0.403)	−0.374 (0.391)	−0.032 (0.282)	0.031 (0.282)
Substance use problem severity at baseline	0.027 (0.034)	0.047 (0.037)	−0.035 (0.021)	−0.037 (0.022)
Parent graduated college	0.035 (0.260)	−0.015 (0.285)	0.205 (0.164)	0.177 (0.160)
Familial alcohol/drug problems	0.084 (0.186)	0.153 (0.218)	0.028 (0.138)	−0.008 (0.140)
Referral from legal system	0.127 (0.222)	0.206 (0.254)	−0.278 (0.153)	−0.298 (0.160)
Missing geocoded data	1.077 (0.737)	0.903 (0.956)	−0.318 (0.361)	−0.187 (0.398)
Missing parent’s education	0.192 (0.416)	−0.799 (0.654)	–	–
Test of exogeneity of treatment initiation (p value)d		(0.230)		(0.549)
N	383	363	384	364

^*Statistically significant at p<0.05

^**Statistically significant at p<0.01

^aCoefficients, robust standard errors (in parentheses), and marginal effects (in brackets) are reported. Instrumental variables include distance from residence to treatment facility and parental assessment of difficulty traveling to treatment (see Table 3).

^bComparison is Site 2 (the largest site).

^cComparison is whites.

^dWald test employed to test H₀: ρ=0 for bivariate probit models. p value of C statistic is reported for two-stage least-squares models. The null for this test is that treatment initiation is uncorrelated with the residuals in the outcome equation.

After addressing selection through the IV models, treatment initiation had a positive and statistically significant effect on the probability of attending school and a negative and statistically significant effect on the likelihood of being employed at the 12-month follow-up (Table 4). Adolescents initiating treatment had a 48-percentage-point higher probability of attending school during the follow-up period than those who did not initiate treatment. The likelihood of employment at the 12-month follow-up was 56-percentage-point lower for adolescents who had initiated treatment. The estimated marginal effect of treatment initiation on criminal activity was in the expected direction (negative) but was not significant.

Tests of independence between treatment initiation and outcomes

Tests of exogeneity were conducted to determine whether there was independence between unobserved client characteristics associated with treatment initiation and determinants of outcomes. Any finding of correlation between the error terms in the treatment and outcome equations would suggest that unobserved characteristics are affecting both the decision to initiate treatment and the outcome, thus, biasing the single-equation regression results. Wald tests of the independence of Eqs. 4 and 5 were used to assess exogeneity when the estimating technique was bivariate probit. The C-statistic test of exogeneity of treatment initiation was utilized when using two-stage least squares. The null hypothesis in both cases was that there was no significant correlation between treatment initiation and the unexplained portion of the outcomes regression. Results of these specification tests did not reveal selection bias in the case of criminal activity (p value of 0.651), abstinence (p value of 0.549), substance use problem severity (p value of 0.230), or employment (p value of 0.309). The null hypothesis of exogeneity was rejected, however, in the case of schooling (p value of 0.001).

While statistical precision of the point estimates may be questioned because of the small sample size and the inefficiency of simultaneous equation estimation relative to single-equation regression, these results indicate that selection bias was a concern in the schooling specification. The presence of an omitted variable capturing unobserved dimensions of severity of drug use could explain why the effect of treatment initiation on the likelihood of attending school was not significant in the single-equation estimations but became large and significant once IV estimation was used. Because traditional regression techniques cannot fully control for these unobserved/unmeasured differences, IV techniques have the potential to eliminate bias and produce more reliable estimates of treatment effects.

Sensitivity Analysis

Length of stay

The initial goal of this study was to investigate whether schooling, criminal activity, and employment at 12 months follow-up were affected by treatment duration.^* Two different measures of treatment duration were constructed, one measuring days in treatment over 12 months and another indicating whether the number of days in treatment for each patient exceeded the median for the sample. Given the concerns about selection into treatment, the analysis depended on the availability of valid instrumental variables that could predict these measures of duration with precision. Unfortunately, distance to treatment (in miles and minutes travel time) was not significant in predicting a continuous measure of length of stay, conditional on initiating treatment. These results highlight some of the challenges associated with identifying intuitively appealing and statistically sound IVs. The failure to identify adequate IVs in the treatment duration analysis was the main reason this paper shifted focus from treatment duration to treatment initiation as the key explanatory variable. It is important to note that the inability to use IV methods in this setting does not necessarily imply that treatment duration is unrelated to treatment outcomes or that single-equation models are biased.

Distance to treatment in minutes was statistically significant in explaining length of stay above the sample median, conditional on initiating treatment. In the analysis of median length of stay, the exogeneity tests indicated that single-equation estimation was preferred for school attendance, criminal activity, abstinence, and substance use problem severity. In the case of employment, bivariate probit estimation was unstable (i.e., led to unrealistic estimates of ρ). Results from probit estimation showed that receiving treatment for at least the median length of stay increased the likelihood of attending school at 12 months (p<0.05) among those who initiated treatment. No significant effects were found for any of the other outcomes. The analysis was repeated using within-site median length of stay rather than the overall sample median. Results were similar (statistically and in magnitude) as those described above, with two exceptions. The employment model converged and the test of ρ indicated that selection was a concern. Adolescents who remained in treatment for longer than the median length of stay at their treatment site were more likely to attend school (marginally significant at p<0.07) and less likely to be employed (p<0.01) at the 12-month follow-up.

Sensitivity of core results to different sets of control and instrumental variables

A thorough sensitivity analysis was conducted to examine the robustness of the core results to the inclusion of different controls and IVs. First, the core models were reestimated with only three site dummies and three community-level variables. The following groups of control variables were then added in stages: the individual controls (gender, age, African-American race, Hispanic race, other race, lived in a residential program, and substance use problem severity at baseline), family controls (parent’s education and familial drug/alcohol problem), and different combinations of referral variables (legal, school, and mental health professional referral). Next, the control for the outcome at baseline was excluded from our core bivariate probit models. Finally, we repeated the analysis with the addition of controls for mental health problems. Two scales for externalizing and internalizing behavioral problems were obtained by combining several subscales from the Youth Self-Report, a structured self-report instrument that measures child and adolescent problems in several domains. The magnitude, sign, and significance of the marginal effects for treatment initiation and the tests of exogeneity generally remained the same throughout our sensitivity analyses. In the case of employment, some of the sensitivity specifications had difficulty converging, and results became unstable once numerous controls were added. Small sample size, limited degrees of freedom, and weak IVs could be related to this instability. When the employment equations did converge, exogeneity was rejected in the most parsimonious model but not in the other specifications.

A second sensitivity test involved the use of different combinations of IVs to conduct simultaneous equation estimation. When parental report of difficulty traveling to treatment was used to evaluate the effects of treatment initiation on employment, results were similar in magnitude to those in the core analysis, but treatment initiation was no longer significant. Using average distance as the only IV to evaluate criminal activity, abstinence, and substance use problem severity increased the sample size, but its predictive power was generally weak. Results from the specification that used the measure of difficulty traveling in addition to distance as an instrument to investigate schooling were consistent with those in the core analysis.

Imputation methods

As mentioned earlier, mean imputation was used in the core analysis to address missing data. While mean imputation may decrease overall sample variation relative to other imputation methods, this method was used because it made interpretation easier, particularly given multiple tests of the validity and reliability of IVs. To check the robustness of our results, we first repeated the analysis without imputing any of the missing observations and then used multiple imputation. Results from these exercises are available upon request.

When missing observations were not imputed, the results were similar to those under mean imputation. Adolescents who initiated treatment were significantly more likely to be attending school and abstinent at 12 months. Bivariate probit results showed a negative effect of treatment initiation on employment, but exogeneity could not be rejected for this outcome, and the probit estimates were not significant. No significant effects were detected for the other outcomes, although the effect of treatment initiation on substance use problem severity approached significance at a 7% level.

Turning to multiple imputation, five imputed datasets were generated using the ice command in STATA. Only missing variables at baseline and IVs were imputed. For each of the imputed datasets, we conducted tests of validity and reliability of the instrumental variables and estimated the effect of treatment initiation on each outcome using both single-equation and simultaneous-equation regressions. Coefficients and standard errors derived from these estimations were then combined manually across the five imputed datasets using Rubin’s rules.⁴⁶ The IVs were more precise in estimating treatment initiation with mean imputation than multiple imputation. Simultaneous-equation estimation again indicated that treatment initiation had a significant effect on schooling. In four out of five of the imputed datasets, the test of exogeneity could not be rejected, probably as a result of reduced precision in the estimation. Simultaneous-equation estimation of employment was unstable in two of the datasets. Results for the other outcomes were generally similar to those in the core analysis.

Implications for Behavioral Health

The estimation of treatment effects when dealing with non-randomized trials is challenged by the selection of patients based on unobserved characteristics. IV techniques were used to estimate the effects of initiation of CD treatment on several adolescent clinical and behavioral outcomes (abstinence, substance abuse problems, school attendance, involvement in criminal activity, and employment). The results illustrate the unique outcomes associated with adolescent interventions and highlight the importance of considering multiple outcomes and addressing selection when evaluating treatment effectiveness. The analysis also revealed some of the limitations of small sample size and the challenges of obtaining reliable and valid instruments when conducting IV analysis.

This study adds to the growing literature suggesting that CD treatment for adolescents can improve a range of clinical and behavioral outcomes.^47–50 Evaluating effectiveness in terms of multiple outcomes provides a richer context in which to examine the impact of a program. With so much individual variation, some adolescents who improve in one area after treatment might not improve in another. Sindelar et al. (2004)⁵¹ concluded that evaluations of adult addiction programs should consider multiple outcomes because statements regarding a program’s effectiveness and implications for policy may not be correct when only one outcome is analyzed. Our findings suggest that this principle also applies to adolescent programs. Although the primary outcome of interest in clinical evaluations of CD treatment is abstinence or reduced substance use, participation in these programs can have more far-reaching consequences. As seen in evaluations of adult interventions, capturing multiple outcomes and social externalities is necessary and important, as clinical and behavioral outcomes do not always show a strong correlation⁵² and the value of the benefits in multiple life domains often outweigh the costs of an intervention.^9,53

Just as treatment approaches for adolescents have been differentiated from those for adults, evaluations of adolescent interventions should also take into account the unique needs of this population. The evaluation of employment illustrated this point. If this were an evaluation of adults in treatment, a positive association between employment and treatment initiation would be expected. Among adolescents, however, the expected effect of treatment on employment is more ambiguous. Employment may be competing with school objectives or may be associated with increased drug and alcohol use when adolescents are still in school. Our analysis of employment as an outcome was limited by sample size constraints as well as the inability to distinguish jobs in terms of type and intensity. Future research should continue to explore this dimension of adolescent behavior.

Most published outcome evaluations of adolescent addiction programs are primarily based on clients in public programs. A notable strength of this study was the ability to focus on a group of adolescents in a private managed care organization. Extending the available research in this area is especially pertinent, as managed care organizations are increasing their role in the provision of CD services in both private and public treatment systems.^54,55 Conversely, these results may have less generalizability, and findings from this population of adolescents are not necessarily representative of others in public treatment programs or with other types of insurance.

An important contribution of this paper has been to demonstrate the potential problem of selection bias in evaluations of adolescent CD treatment programs. Specifically, differences between single equation and simultaneous equation estimates for schooling highlighted why selection bias may be an important concern when evaluating observational studies.⁴⁰ IV approaches have been used in various areas of health services research but have yet to be fully integrated into the evaluation of addiction programs. IV techniques may be especially relevant given that randomized trials often pose ethical dilemmas and selection bias is a prominent concern when a non-experimental design is used. Collecting data on measures that could be used as instruments when designing surveys to evaluate interventions could also help improve these types of analyses.^*,⁴²

This study has several limitations. First, the measure of abstinence is based on self-reports, which is subject to over-reporting. Second, the ideal IV would have been a precise measure of the distance between the treatment program and the adolescent’s school or between the program and the parent’s work, as these may be more relevant to participation in treatment. Unfortunately, this information was unavailable. Third, the use of IV techniques was a challenge in this analysis given the relatively small number of observations. Applying these methods to larger samples could result in more robust and consistent estimates. Although larger than those used in other evaluations of adolescent treatment programs, the sample size was nevertheless small. As a result of the loss of efficiency when IV estimation is used, observational studies often require larger samples than randomized trials.⁴² This difference is magnified further if the instruments are weak.⁴² The power in the statistical analysis is therefore lower than ideal.

Despite these limitations, this analysis suggests that participation in intensive outpatient adolescent treatment improved school attendance and promoted abstinence 1 year after intake. The results also clarified the need to account for potential unobserved variables that could bias results of effectiveness studies. Applying the techniques described in this paper to larger samples with more refined outcome measures may result in more consistent estimates that enable researchers to determine program effectiveness without implementing an expensive experimental design.