The Washington Circle Engagement Performance Measures' Association with Adolescent Treatment Outcomes

Abstract

Background

For adolescents, substance use disorder (SUD) treatment outcomes (e.g., abstinence, problematic behaviors) often cannot be measured soon enough to influence treatment trajectory. Although process measures (e.g., treatment engagement) can play an important role, it is essential to demonstrate their association with outcomes. This study explored the extent to which engagement in outpatient treatment was associated with outcomes and whether demographic/clinical characteristics moderated these relationships.

Methods

This is a prospective study of adolescents (N=1,491) who received outpatient treatment for SUDs at one of 28 treatment sites taking part in a national evidence-based practice implementation initiative. Information from the Global Appraisal of Individual Needs interviews at intake and six-month follow-up, as well as encounter data, were used. Adjusted hierarchical logistic models were used to estimate effects of engagement on six-month outcomes.

Results

Sixty-one percent of adolescents engaged in outpatient treatment. Adolescents engaging in treatment had significantly lower likelihoods of reporting any substance use (OR 0.60, 95% CI 0.41, 0.87), alcohol use (OR 0.63, 95% CI 0.45, 0.87), heavy alcohol use (OR 0.53, 95% CI 0.33, 0.86), and marijuana use (OR 0.64, 95% CI 0.45, 0.93). This association of engagement with abstinence outcomes was not limited to any particular group. Treatment engagement, however, was not associated with adolescents' self-report of illegal activity or trouble controlling behavior at follow-up.

Conclusion

At the individual level, the Washington Circle engagement measure was a predictor of some positive outcomes for adolescents in outpatient treatment. Efforts to better engage adolescents in treatment could improve quality of care.

1. Introduction

The treatment goals for adolescents with substance use disorders (SUDs) include stopping the use of alcohol or drugs and avoiding problematic behaviors. Performance measures of treatment processes, such as treatment engagement, can play an important role in assessing early treatment success. Process measures, which quantify whether a health care service supported by scientific evidence is provided to or on behalf of a client, can indicate whether services are provided in a timely and consistent manner that is likely to lead to a favorable clinical outcome. Furthermore, process measure data, compared with outcome measure data, which quantify the actual resulting “health state” of the client, are typically less expensive to collect (National Quality Measures Clearinghouse, 2011). If they are found to predict treatment outcome, process measures such as engagement can be obtained when there is still time to change course in how treatment is provided. If failure to engage results in significantly worse outcomes than for those who engage, treatment programs could implement process improvement actions to improve engagement and hence clinical outcomes.

The purpose of this study was to examine how adolescents' engagement in outpatient treatment relates to a range of outcomes such as any substance use, alcohol use, heavy alcohol use, marijuana use, illegal activities, and control of problem behaviors at six months follow-up. Specifically, using a process measure of treatment engagement, we explored: the extent to which treatment engagement predicts outcomes for adolescents such as substance use or problem behaviors, such as illegal activity or trouble controlling behavior; and the extent to which the predictive relationships between engagement and these outcomes are moderated by adolescents' demographic and clinical characteristics.

1.1 Background

It is important to demonstrate that meeting the criteria of process measures is significantly related to increased probability of improved outcomes. Indeed, acceptance of process measures relies on establishing an association between compliance with process measures and outcomes such as abstinence, improved school or work performance, or reduced criminal justice involvement. Research in this area is key, both for process measures focused on substance use disorders (Garnick et al., 2007; Harris et al., 2009a; Horgan and Garnick, 2005) and medical conditions (Zingmond et al., 2011).

Less attention has been focused on this topic among adolescents, although one study examined a process measure for continuity of care following residential substance use treatment (measured as at least one service within 14 days of discharge). Adolescents with continuity of care after discharge had increased likelihood of abstinence at their three-month follow-up interview (Garner et al., 2010). Thus far, adults have been the focus of research on the relationship between outcomes and the measure of treatment engagement initially developed by the Washington Circle and used in this study. Engagement in outpatient treatment was associated with lower criminal justice involvement in Oklahoma (Garnick et al., 2007) and Washington state (Campbell, 2009) and with improved individual level legal outcomes and reduced substance use among outpatients in the Veterans Administration (Harris et al., 2010b). Similar measures (2-6 visits in the first month of treatment) were associated with both individual and facility-level positive outcomes at outpatient and intensive outpatient Veterans Administration clinics for patients with alcohol use disorders (Harris et al., 2009b). This study expands this area of research on treatment engagement and outcomes to an adolescent population. Given that individuals' responses to treatment are not static over their lifespan, including in their response to substance use treatment (Haegerich and Tolan, 2008; Tolan et al., 2007), it is important to consider adolescents as a separate group and to examine the effect of engagement by them on treatment outcomes, as it may differ from the response among adults.

This engagement measure has been used widely and demonstrated to be feasible for use in the private (Garnick et al., 2002) and public sectors (Garnick et al., 2011; Garnick et al., 2009), adopted by the National Committee on Quality Assurance (2007), and endorsed by the National Quality Forum (2010). It also has been implemented by the VA and included in the Stage 1 set of Clinical Quality Measures for meaningful use initiatives in the HITECH Act (Blumenthal and Tavenner, 2010).

Outpatient treatment engagement occurs when an individual who already has initiated outpatient treatment receives two additional services within 30 days after initiation. (Outpatient initiation, intended to capture the very initial stage of treatment, occurs when an individual with a new treatment episode receives a second treatment service within 14 days; Garnick et al., 2009.). It is broadly accepted that treatment retention is key to clients' success, with most addicted individuals needing at least three months in treatment to significantly reduce or stop their substance use (National Institute on Drug Abuse, 2009) and longer treatment often is necessary (Arria, 2003; Luchansky et al., 2006b), especially for those with more severe problems (Simpson, 2004). Initiation and engagement in treatment first must occur, however, in order to reach the goal of longer retention in treatment. Therefore, we hypothesize that adolescent clients who meet the criteria for engagement will report improved outcomes. Additionally, because the measure defines engagement as comprised of two visits after initiation, and because we had the opportunity to test its effect if defined differently, we ran ancillary models with engagement defined alternatively as three or four post-initiation visits, hypothesizing similar results.

2. Methods

2.1. Study design and sample

This is a prospective study of 1,491 adolescents (84% follow-up rate with follow-up sample somewhat younger) who received outpatient treatment for SUDs between June 2007 and May 2010 at one of 28 treatment sites which took part in the Assertive Adolescent and Family Treatment (AAFT) project. Specific referral and recruitment procedures varied between sites, however, all 28 sites were required to recruit adolescents (12-18 years of age) identified as having a problem with substance use. Each site received financial resources of approximately $300,000 per year for up to three years to implement AAFT through a SAMHSA Center for Substance Abuse Treatment (CSAT) initiative to increase adherence to use of evidence-based practices (EBPs) in SUD treatment (Godley et al., 2011a). The sites are spread across 12 states (regions of the country represented: Northeast [5], Southeast/South Atlantic [8], Midwest [2], Southwest [4], and West [9]) and serve a diverse range of communities, including urban and rural areas, colonias or Mexican-American communities located on the Texas-Mexican border, and Native American communities. We excluded four groups of clients, specifically those: (1) entering during the first quarter of the study (n = 6) to avoid issues of program start-up, (2) entering during the last two quarters (n = 124) to avoid issues of program close out, (3) who were in a controlled environment for a major portion of the measurement time period (n = 212), and (4) with missing data (n = 72 to 78 depending on outcome).

2.2. Description of the Assertive Adolescent and Family Treatment (AAFT) Study

AAFT is focused primarily on two evidence-based practices, Adolescent Community Reinforcement Approach (A-CRA) and Assertive Continuing Care (ACC; Godley et al., 2007; Godley et al., 2006; Godley et al., 2001). These are, respectively, outpatient and continuing care adaptations of adult-focused CRA (Hunt and Azrin, 1973), and comprise a behavior therapy that involves 12 to 14 sessions with the adolescent, parent, and adolescent and parent together over a 90 day period, followed by 90 days of home-based ACC (Godley et al., 2011b). These evidence-based practices focus on the interaction between youth and their environments, and are individual and family-centered and community-based. Families/primary caregivers are an integral part of treatment and their inclusion in the process increases the likelihood of improving adolescents' recovery environment and reducing substance use and related problems. Both A-CRA and ACC, which seek to rearrange environmental contingences to make non-using behaviors more reinforcing than using behaviors, are effective treatments for adolescents with SUDs (e.g., Dennis et al., 2004; Garner et al., 2010; Godley et al., 2007; Godley et al., 2011a; Ruiz et al., 2011).

2.3. Data sources

Data included information from interviews at intake and six-month follow-up, as well as encounter data with dates of treatment services. To evaluate the SUD treatment of their adolescent clients, all programs utilized the widely accepted Global Appraisal of Individual Needs (GAIN; Dennis et al., 2008). The GAIN main scales have been shown to demonstrate good internal consistency and test–retest reliability and to be highly correlated with other measures of use, including timeline follow-back methods, urine tests, collateral reports, treatment records, and blind psychiatric diagnosis (e.g., Dennis et al., 2006; Lennox et al., 2006).

2.4. Measures

2.4.1. Outcomes

We investigated two categories of treatment outcomes, substance use and problem behavior. Substance use outcomes include: any substance use, alcohol use, heavy alcohol use and marijuana use. Problem behavior outcomes include: involvement in illegal activities and trouble controlling behavior. Each outcome is dichotomous and based on self-report at the six-month follow-up assessment, which asked the client to report use of specific substances or specific problem behaviors during the most recent three month period.

2.4.2. Main variable of interest

Engagement, as defined for outpatient treatment by the Washington Circle specification for the public sector (Garnick et al., 2009), was our main variable of interest. Engagement is specified as two additional treatment services within 30 days from the date of treatment initiation, which itself is specified as one treatment service within 14 days of an index service beginning a new episode. The initiation service is not included in the count of treatment services that comprise engagement. Treatment services, which count towards initiation or engagement, must be performed in the presence of the adolescent, but could involve treatment with the adolescent alone or in conjunction with their parent(s). We also tested the sensitivity of our models to the choice of engagement as our main variable of interest and the specification of engagement. In particular, we used initiation as the major predictor of outcomes instead of engagement. We also examined alternative specifications for engagement, requiring either three or four additional services within the 30 day time period after initiation. Outcomes of these sensitivity analyses are provided in the Results section.

2.4.3. Other independent variables

Choice of other independent variables for our models relied on theory and began with a list of factors in the GAIN, which previous research suggested would relate to our outcomes. However, because the GAIN provides a rich source of information with several variables measuring the same or similar constructs, a further selection among this initial list was necessary to guard against multi-collinearity or over-specification (Lee et al., 2011). The five general domains categorizing variables available in the GAIN included client demographics, clinical factors, behavioral problems, hospitalizations, and medical history. Correlations, chi-square tests, and other analyses between candidate independent variables and each outcome were used to determine which variables, if any, in each domain, would help explain variation in our outcomes and thus make useful adjustment to our analyses. These preliminary analyses indicated that variables for hospitalizations and medical history would not be helpful to our analyses and thus these two domains were dropped. The remaining three domains each contained a number of independent variables which demonstrated usefulness for at least some of our outcomes. Among demographic characteristics, these variables included age, race/ethnicity, sex, and homelessness. Among clinical variables, the preliminary analyses suggested General Victimization Scale (GVS) and separate indicators for Major Depressive Disorder (Depression), Generalized Anxiety Disorder (Anxiety), Attention-Deficit Hyperactivity Disorder (ADHD), and Conduct Disorder. Among the behavioral variables, we retained the variables for Substance Use Problems (SPSM), truancy, worry about peer pressure to use substances, and criminal justice system involvement. A baseline variable for each outcome also was included in the model specific to that outcome.

2.5. Statistical analysis

Preliminary analyses were conducted to provide overall, and by engagement, descriptive statistics for all independent variables and outcomes at both intake and six month follow-up. In addition, McNemar tests (McNemar, 1947) were performed to assess the significance of changes in these outcomes over time. Main analyses consisted of hierarchical logistic regressions modeling outcomes at six month follow-up as a function of engagement, after adjustment for baseline characteristics, and certain interactions with engagement. Hierarchical models are widely used in health services research when patients are clustered, such as by site, by staff member within site, or by both.

In determining the specification of our models, our first issue was whether to use a change score or analysis of covariate approach. Preliminary analyses indicated some significant baseline differences in outcome measures between engaged and non-engaged groups and also moderate correlations between baseline and follow-up outcomes within subjects. These results directed us towards the analysis of covariance approach (Vickers and Altman, 2001). A second specification issue concerned which potential interactions with engagement should be included as independent variables in our models. To inform this decision, we employed the common approach in exploratory research of performing a set of simpler models (Hosmer and Lemeshow, 2000), which include only the single covariate itself, engagement, and their interaction. Interactions of covariates with engagement were included in our final models if they had p-values of 0.25 or less in these simpler models. A p-value of 0.25 was selected as a screening criterion because it has been shown that relying on smaller p-values (e.g., 0.05) for screening can result in the omission of important variables (Hosmer and Lemeshow, 2000). We also tested the interaction of engagement with the baseline status of the outcomes and found that none were significant at a p-value 0.25. Thus, separate analysis of those presenting with and without the baseline characteristics was not necessary (Harris et al., 2010a). To address another specification of concern, we tested the need to include random effects in our models for sites and/or clinicians within sites. We discovered that the proportions of variance due to random effects of sites and clinicians were similar and of sufficient size to include both in our models.

3. Results

3.1. Study subjects

Frequencies for study subjects' demographic, clinical and behavioral characteristics and outcome variables at baseline, and for initiating and engaging in treatment, are displayed in Table 1. Of the sample of 1,491 adolescents, the majority were aged 15 to 16 (51%), predominately male (74%), approximately a third each were White and Latino, over half reported having been victimized, and there were substantial levels of Depression, ADHD, and Conduct Disorder. Over two-thirds had high truancy rates in the 90 days preceding intake, 63% reported criminal justice involvement at intake, and nearly two-thirds reported some degree of recent substance use related problems. The predominant drug used at intake was marijuana (77%), followed by alcohol (64%). Seventy-seven percent of the sample initiated and 61% engaged in treatment.

Table 1.

Demographic, Clinical, Behavioral and Treatment Characteristics of Adolescents Receiving Outpatient Treatment Under Assertive Adolescent and Family Treatment (AAFT).^a

	N (%)
Demographics
Age
12-14	266 (18%)
15-16	759 (51%)
17-18	466 (31%)
Race/Ethnicity
White	494 (33%)
Black	212 (14%)
Latino	487 (33%)
Mixed/Otherb	298 (20%)
Female	391 (26%)
Homeless	102 (7%)
Clinical
General Victimization Scalec
Low	583 (39%)
Moderate	293 (20%)
High	615 (41%)
Depressiond	505 (34%)
Anxietye	173 (12%)
ADHDf	674 (45%)
Conduct Disorderg	713 (48%)
Behavioral
Truancyh
< 31 Days in School	992 (67%)
31+ Days in School	499 (33%)
Peer Pressurei	499 (33%)
Criminal Justice Involvementj	939 (63%)
Substance Use Problems Scalek
Low	555 (37%)
Medium	464 (31%)
High	472 (32%)
Baseline Outcomes
Alcohol Usel	949 (64%)
Heavy Alcohol Usem	676 (45%)
Marijuana Usen	1,149 (77%)
Illegal Activitieso	681 (46%)
Trouble Controlling Behaviorsp	1020 (69%)
Treatment Characteristics
Initiationq	1,142 (77%)
Engagementr	911 (61%)

^aN = 1,491 other than Illegal Activities (n = 1,485) and Trouble Controlling Behaviors (n = 1,488).

^bMixed/Other: Adolescents who self-identified as of mixed race/ethnicity or other race/ethnicities including Native American (3.4%), Asian (1.4%), Pacific Islander (0.2%), mixed (14.3%) and other (0.9%).

^cGeneral Victimization Scale (GVS): Count of types of lifetime victimization (including physical, emotional and sexual), and the number of traumagenic factors involved (including origination, duration, type and relation of perpetrator, etc.). Low is none, Moderate is 1-3, and High is 4+.

^dMajor Depressive Disorder in the 12 months preceding intake.

^eGeneralized Anxiety Disorder in the 12 months preceding intake.

^fADHD required clients to endorse six or more symptoms in the 12 months preceding intake related to inattention, hyperactivity/impulsivity, or both inattentive and hyperactive type.

^gPast-year Conduct Disorder required three or more endorsed symptoms in the 12 months preceding intake.

^hIn school less than 31 days in the 90 days prior to intake.

ⁱFears old friends will try to get him/her to drink or use drugs again.

^jInvolved in criminal justice system at the time of intake.

^kSubstance Use Problems Scale (SPSM): Count of the number of 16 types of problems related to substance use that client endorses in the month prior to intake. Low is none, Medium is 1-3, and High is 4+.

^lSelf-reported alcohol use at intake and for the prior 90 days.

^mheavy alcohol use at intake and for the prior 90 days. Heavy alcohol use is reported as either being drunk or consumed 5 or more drinks in a day.

ⁿSelf-reported marijuana use at intake and for the prior 90 days.

^oSelf-reported illegal activities (things that “might get them in trouble or be against the law besides using alcohol or other drugs”) at intake and for the prior 90 days.

^pSelf-reported trouble paying attention, controlling behavior, or following rules, at intake and for the prior 90 days.

^qThe Washington Circle measure of outpatient treatment initiation is at least one additional outpatient treatment service within 14 days of a new episode of outpatient treatment (defined as a 60-day prior period without treatment services) (Garnick et al., 2002; Garnick et al., 2009).

^rThe Washington Circle Engagement is at least two additional outpatient treatment services within 30 days after the initiation date (Garnick et al., 2009).

In addition, as shown in Table 2, the proportion of clients reporting each negative outcome decreased significantly between intake and 6-month follow-up (p<.0001 in all cases).

Table 2.

Substance Use and Problem Behavior Outcomes for Adolescents Receiving Outpatient Treatment Under Assertive Adolescent and Family Treatment (AAFT).^a

Outcome	Definition	Intake Proportion (95% CI)	6-month Follow-up Proportion (95% CI)b
Any Substance Use	Dichotomous self-report of any substance use in the prior three months	89% (87, 91)	60% (58, 63)
Alcohol Use	Dichotomous self-report of any alcohol use in the prior three months	64% (61, 66)	45% (43, 48)
Heavy Alcohol Use	Dichotomous self-report of being drunk or consuming 5+ drinks in the prior three months	45% (43, 48)	32% (30, 34)
Marijuana Use	Dichotomous self-report of any marijuana use in the prior three months	77% (75, 79)	46% (43, 48)
Illegal Activities	Dichotomous self-reported illegal activities (things that “might get them in trouble or be against the law besides using alcohol or other drugs”) for the prior 90 days.	46% (43, 48)	29% (27, 31)
Trouble Controlling Behavior	Dichotomous self-reported trouble paying attention, controlling behavior, or following rules for the prior 90 days.	69% (66, 71)	50% (47, 52)

^aN = 1,491 other than Illegal Activities (n = 1,485) and Trouble Controlling Behaviors (n = 1,488).

^bAll differences between proportion at intake and at six-months follow-up are significant at the p<.0001 level.

3.2. Multivariate regression results

3.2.1. Substance use outcomes

Adolescents who engaged in treatment had significantly lower likelihoods of reporting all four substance use measures: any substance use (OR 0.60, 95% CI 0.41, 0.87), any alcohol use (OR 0.63, 95% CI 0.45, 0.87), heavy alcohol use (OR 0.53, 95% CI 0.33, 0.86), and any marijuana use (OR 0.64, 95% CI 0.45, 0.93) (Table 3).

Table 3.

Association of Engagement^a With Substance Use^b at Six Months Follow-up.^c

	Any Substance Use		Any Alcohol Use		Heavy Alcohol Used		Any Marijuana Use
	OR	95% CI	OR	95% CI	OR	95% CI	OR	95% CI
Individual—Demographics
Age (reference: 15-16)
12-14	0.70*	.52, .96	.48**	.27, .84	.42**	.23, .79	.92	.68, 1.25
17-18	1.22	.94, 1.59	1.28	.85, 1.91	1.20	.79, 1.80	1.04	.81, 1.35
Race/Ethnicity (reference: White)
Black	.87	.60, 1.26	.52**	.35, .77	.32***	.20, .51	1.19	.83,1.73
Latino	1.10	.80, 1.52	1.07	.77, 1.49	.84	.60, 1.19	.97	.70, 1.34
Mixed/Othere	1.20	.85, 1.69	1.02	.72, 1.44	.88	.61, 1.26	1.28	.92, 1.80
Female	.49***	.38, .65	.52***	.39, .69	.41***	.30, .56	.53***	.40, .70
Homeless	.95	.60, 1.51	1.03	.65, 1.63	1.19	.74, 1.90	1.15	.73, 1.79
Individual—Clinical
General Victimization Scale (GVS)f (reference: Low)
Moderate	1.15	.84, 1.56	1.22	.89, 1.69	1.01	.60, 1.71	.99	.61, 1.61
High	1.44**	1.09, 1.90	1.51**	1.14, 2.00	.97	.63, 1.50	1.05	.70, 1.58
Depressiong	1.30	.96, 1.76	1.29	.95, 1.75	1.36	.99, 1.87	1.29	.96, 1.73
Anxietyh	1.10	.72, 1.66	.92	.62, 1.37	.77	.51, 1.17	1.01	.68, 1.49
ADHDi	1.04	.80, 1.36	.96	.73, 1.26	.96	.72, 1.28	1.00	.77, 1.29
Conduct Disorderj	1.21	.93, 1.58	1.27	.97, 1.66	1.33*	1.00, 1.76	1.23	.95, 1.60
Individual--Behavioral
Truancy (reference: 31+ Days in School)k
< 31 Days in School	1.40**	1.10, 1.80	1.45**	1.14, 1.85	1.35*	1.05, 1.74	1.32*	1.04, 1.67
Peer Pressurek	.88	.68, 1.14	.90	.70, 1.16	1.25	.96, 1.64	.96	.75, 1.23
Criminal Justice Involvementm	.87	.68, 1.11	.79	.62, 1.02	.94	.72, 1.23	.83	.65, 1.06
Substance Use Problems (SPSM)n (reference: Low)
Medium	1.07	.69, 1.66	1.33*	1.00, 1.77	1.34	.98, 1.82	1.29	.98, 1.71
High	2.02**	1.24, 3.30	1.57**	1.15, 2.12	1.56**	1.13, 2.15	1.94***	1.42, 2.63
Baseline Outcomeso
Any Substance Use	2.56***	1.71, 3.81	--	--	--	--	--	--
Alcohol Use	--	--	3.23***	2.51, 4.17	3.19***	2.39, 4.27	--	--
Marijuana Use	--	--	--	--	--	--	2.58***	1.89, 3.51
Treatment Characteristics&Interactions
Engaged (reference: Not Engaged)a	.60**	.41, .87	.63**	.45, .87	.53**	.33, .86	.64*	.45, .93
Age 12-14 × Engage	--	--	1.48	.74, 2.95	1.63	.75, 3.54	--	--
Age 17-18 × Engage	--	--	1.42	.84, 2.39	1.38	.81, 2.37	--	--
GVS Moderate × Engage	--	--	--	--	.86	.43, 1.72	1.12	.60, 2.07
GVS High × Engage	--	--	--	--	1.39	.80, 2.41	1.34	.81, 2.22
SPSM Medium × Engage	1.64	.95, 2.83	--	--	--	--	--	--
SPSM High × Engage	1.17	.65, 2.09	--	--	--	--	--	--
Variance Partitions
Site	.0171		.0196		.0199		.0193
Staff	.0170		.0222		.0133		.0234

Note:

^*p≤.05,

^**p≤.01.

^***p≤.001. Dashes indicate that a specific variable is not included in the model.

^aThe Washington Circle Engagement is at least two additional outpatient treatment services within 30 days after the initiation date (Garnick et al, 2009).

^bSelf-reported use of the indicated substance for the 90 days prior to six-months post-intake assessment.

^cn = 1,491

^dHeavy alcohol use: reported either being drunk or consumed 5 or more drinks in a day.

^eMixed/Other: Native American (3.4%), Asian (1.4%), Pacific Islander (0.2%), mixed (14.3%) and other (0.9%).

^fGeneral Victimization Scale (GVS): Count of types of lifetime victimization including physical, emotional and sexual), and the number of traumagenic factors involved (including origination, duration, type and relation of perpetrator, etc.). Low is none, Moderate is 1-3, and High is 4+.

^gMajor Depressive Disorder in the 12 months preceding intake.

^hGeneralized Anxiety Disorder in the 12 months preceding intake.

ⁱADHD required clients to endorse six or more symptoms in the 12 months preceding intake related to (a) inattention, (b) hyperactivity/impulsivity, or (c) both inattentive and hyperactive type.

^jPast-year Conduct Disorder required three or more endorsed symptoms in the 12 months preceding the intake assessment.

^kIn school less than 31 days in the 90 days prior to intake.

^lFears old friends will try to get him/her to drink or use drugs again.

^mInvolved in criminal justice system at the time of intake.

ⁿSubstance Use Problems Scale (SPSM): Count of the number of 16 types of problems related to substance use that client endorses in the month prior to intake. Low is none, Medium is 1-3, and High is 4+.

^oSelf-reported indicated substance use at intake and for the prior 90 days.

Several demographic characteristics were significantly associated with lower likelihoods of substance use outcomes. Specifically, younger adolescents (aged 12-14) had lower likelihood of all substance use except marijuana, female adolescents had lower likelihood on all four substance abuse outcomes, and adolescents who were Black had lower likelihood for both alcohol use and heavy alcohol use.

Behavioral and clinical characteristics significantly predictive of higher likelihood of substance use outcomes included frequent truancy from school (all outcomes), high levels of victimization (any substance use and any alcohol use), high levels of substance use problems at intake (all outcomes) and having symptoms of Conduct Disorder (heavy alcohol use only). None of the interactions examined between engagement and other main effects were significant.

3.2.2. Problem behavior outcomes

Treatment engagement was not a significant predictor for either problem behavior outcome (Table 4). However, being female or an older adolescent did significantly decrease the likelihood of reporting either trouble controlling behavior or illegal activity. Additionally, being involved in the criminal justice system at intake significantly decreased the likelihood of trouble controlling behavior. On the other hand, there was greater likelihood of involvement in illegal activities at follow-up if the adolescent scored in the moderate or high range on the GVS, scored medium or high on the SPSM, or came into treatment having been already engaged in illegal activities in the 90 days prior to intake. There was also greater likelihood of trouble controlling behavior at six months follow-up if the adolescent had symptoms of ADHD at intake or had trouble controlling behavior in the 90 days before intake. Only the interaction between engagement and younger age (12-14) in the trouble controlling behavior model was significant, so that younger adolescents who engaged in SUD treatment were most likely to continue having difficulty controlling behavior.

Table 4.

Association of Engagement^a With Problem Behaviors at Six Months Follow-up.

	Illegal Activitiesbc		Trouble Controlling Behaviorde
	OR	95% CI	OR	95% CI
Individual--Demographics
Age (reference: 15-16)
12-14	1.15	.83, 1.59	.68	.40, 1.14
17-18	.69*	.52, .92	.51***	.34, .76
Race/Ethnicity (reference: White)
Black	.71	.38, 1.34	.91	.62, 1.32
Latino	.92	.55, 1.56	.93	.67, 1.29
Mixed/Otherf	.81	.46, 1.41	.91	.65, 1.29
Female	.60**	.44, .81	.65**	.50, .86
Homeless	.96	.59, 1.55	1.13	.72, 1.78
Individual--Clinical
General Victimization Scale (GVS)g (reference: Low)
Moderate	1.57**	1.12, 2.20	.86	.53, 1.41
High	1.40*	1.03, 1.90	1.20	.80, 1.81
Depressionh	1.28	.94, 1.76	1.04	.78, 1.40
Anxietyi	.91	.61, 1.37	1.47	.98, 2.19
ADHDj	1.31	.98, 1.73	1.73***	1.33, 2.24
Conduct Disorderk	1.17	.88, 1.57	--	--
Individual--Behavioral
Truancy (past 90 days)l (reference: 31+ Days in School)
< 31 Days in School	1.10	.85, 1.43	1.14	.90, 1.45
Peer Pressurem	1.05	.81, 1.37	1.04	.81, 1.33
Criminal Justice Involvementn	1.23	.94, 1.61	.72**	.57, .92
Substance Use Problems (SPSM)o (reference: Low)
Medium	1.38*	1.01, 1.87	1.11	.85, 1.46
High	1.43*	1.03, 1.98	1.23	.92, 1.65
Baseline Outcomesp
Illegal Activities (Baseline and 90 days prior)	1.96***	1.51, 2.55	--	--
Trouble Controlling Behavior (Baseline and 90 days prior)		--	2.07***	1.57, 2.73
Treatment Characteristics & Interactions
Engageda (reference: Not Engaged)	.86	.55, 1.34	.70	.46, 1.07
Age 12-14 × Engage	--	--	2.03*	1.07, 3.85
Age 17-18 × Engage	--	--	1.31	.78, 2.20
Black × Engage	1.34	.60, 3.00	--	--
Latino × Engage	.86	.46, 1.59	--	--
Other × Engage	1.56	.78, 3.10	--	--
GVS Mod × Engage	--	--	1.31	.71, 2.45
GVS High × Engage	--	--	1.44	.87, 2.39
Variance Partitions
Site	.0283		.0252
Staff	.0206		.0193

Note:

^*p≤Note:. 05,

^**p≤.01,

^***p≤. 001. Dashes indicate that a specific variable is not included in the model.

^aThe Washington Circle Engagement is at least two additional outpatient treatment services within 30 days after the initiation date (Garnick et al., 2009).

^bSelf-reported illegal activities (things that “might get them in trouble or be against the law besides using alcohol or other drugs”) for the 90 days prior to six-months post-intake assessment.

^cn = 1,485.

^dSelf-reported trouble paying attention, controlling behavior, or following rules for the 90 days prior to six-months post-intake assessment.

^en = 1,488

^fMixed/Other: Native American (3.4%), Asian (1.4%), Pacific Islander (0.2%), mixed (14.3%) and other (0.9%).

^gGeneral Victimization Scale (GVS): Count of types of lifetime victimization including physical, emotional and sexual), and the number of traumagenic factors involved (including origination, duration, type and relation of perpetrator, etc.). Low is none, Moderate is 1-3, and High is 4+.

^hMajor Depressive Disorder in the 12 months preceding intake.

ⁱGeneralized Anxiety Disorder in the 12 months preceding intake.

^jADHD required clients to endorse six or more symptoms in the 12 months preceding intake related to (a) inattention, (b) hyperactivity/impulsivity, or (c) both inattentive and hyperactive type.

^kPast-year Conduct Disorder required three or more endorsed symptoms in the 12 months preceding the intake assessment.

^lIn school less than 31 days in the 90 days prior to intake.

^mFears old friends will try to get him/her to drink or use drugs again.

ⁿInvolved in criminal justice system at the time of intake.

^oSubstance Use Problems Scale (SPSM): Count of the number of 16 types of problems related to substance use that client endorses in the month prior to intake. Low is none, Medium is 1-3, and High is 4+.

^pSelf-reported indicated problem behaviors at intake and for the prior 90 days.

3.3. Sensitivity Analyses

To check the robustness of our results under alternative model specification, three types of sensitivity analyses were conducted. Specifically we modeled: 1) including a facility level engagement variable; 2) redefining engagement as three or four visits after initiation, rather than the two that is the current standard; and 3) including initiation as the main variable of interest rather than engagement.

First, including a facility level variable for percent of clients engaged did not have a significant effect on outcomes in any of our models, nor did it significantly change the magnitude of other effects. Second, compared with the average engagement rate of 61% when defined as two or more treatment visits after initiation, when we tested more stringent definitions, the average engagement rate was 42% for three or more visits and 22% for four or more visits. Redefining engagement to represent 3 or 4 visits within the 30 day time period after initiation instead of two resulted in only minor differences in the regression coefficients. For marijuana, any substance use, and problem behavior outcomes, the results were similar across all definitions. For alcohol and heavy alcohol, however, engagement was significant only using the original definition. Third, when we included initiation as a predictor variable rather than engagement, initiation was predictive of abstinence from alcohol and from any substance use but was not predictive of heavy alcohol use, marijuana use, involvement in illegal activities, or trouble controlling behavior. Most of the significant results related to other explanatory variables were the same.

3.4. Variance Partitioning

Using the Linear Threshold Model (Goldstein et al., 2002) to partition the variation in outcome due to staff and site components, we found that the proportion of variance due to staff ranged from 1.3% for heavy alcohol use to 2.3% for marijuana, and the proportion of variance due to site ranged from 1.7% for any substance use to 2.8% for illegal activity.

4. Discussion

4.1 Association of Engagement and Outcomes

Treatment engagement was significantly associated with a lower likelihood of reporting use of alcohol or other drugs at six months follow-up among adolescents receiving outpatient A-CRA and ACC. These results are consistent with earlier evidence for adolescents using a slightly different measure of engagement (Balsa et al., 2009) and adults in the Veterans Administration (Harris et al., 2010b; Harris et al., 2009b). In contrast to our findings pertaining to substance use, treatment engagement was not associated with adolescents' self-report of illegal activity or trouble controlling behavior at six month follow-up. While no prior studies have examined the link between adolescent outpatient treatment engagement using the Washington Circle measure specification and problem behavior outcomes, most research addressing the effect of treatment retention on criminal justice involvement shows a link between retention and reduced criminal justice involvement or conduct disordered behavior (Hser et al., 2001; Hser et al., 2004; Whitmore et al., 2000), although one study did not (Balsa et al., 2009). There also have been studies linking either adult treatment engagement using the Washington Circle measure specification and reduced criminal justice involvement (Campbell, 2009; Garnick et al., 2007), or a modified measure used by the Veterans Administration and positive results on the ASI measure that includes legal outcomes (Harris et al., 2010b). None of these studies, however, provide us with information that can explain why the engagement measures did not predict decreased problematic behaviors in this study.

The association of engagement with better abstinence outcomes was not limited to any particular group. Adolescents' demographic, clinical or behavioral characteristics did not moderate the relationships between engagement and outcomes, with one exception. This general lack of significance of the interactions of engagement with almost all client characteristics implies that efforts to improve engagement are appropriate across all adolescents in outpatient treatment. With regard to the single instance in which a characteristic (youth age 12-14 years) did moderate the relationship between engagement and an outcome (i.e., trouble controlling behavior), we speculate that engagement alone is not powerful enough to overcome the difficulty the youngest adolescents may have controlling behavior. This may be particularly true if those who present with severe substance use issues are first offered treatment in the least restrictive environment, outpatient treatment, but may eventually be stepped-up to residential treatment. It also is possible that the youngest adolescents may be physically escorted to treatment by parents, ensuring technical participation and engagement but not the sort of participation that is required to effect change.

As to be expected, prior behavior was a highly significant predictor of outcome. Adolescents who reported difficulty with a behavior -- whether substance use, illegal activity, or trouble controlling behavior -- at the beginning of treatment were most likely to continue to report such difficulty six months later. These results were robust to the different modeling approaches undertaken as part of the sensitivity analyses described above.

4.2 Influence of Adolescents' Characteristics

Our main focus was treatment engagement, but consistent with the literature on outcomes for adolescents, we also found strong associations of certain demographic, clinical and behavioral characteristics with some outcomes. These are also important to consider in crafting client-centered approaches to treatment, noting, of course, that engagement in outpatient treatment is not a final outcome but merely on the causal path to positive outcomes.

With the exception of marijuana use, being younger at intake (12-14 years) was associated with reduced likelihood of substance use at follow-up, as also demonstrated elsewhere (Balsa et al., 2009; Chung et al., 2004). This is also consistent with National Survey on Drug Use and Health population data which show that heavy substance use, abuse, and dependence symptoms increase incrementally through the teen years peaking in the early 20's before descending (Dennis and Scott, 2007). In contrast, being older (17-18) was associated with reduced self-report of engaging in illegal activities and trouble controlling behaviors. Older adolescents may have underreported or curtailed illegal activities under threat of adult sanctions or they may be more likely to already be more closely monitored by the juvenile justice system.

Consistent with prior research (Chung et al., 2004; Godley et al., 2011b), adolescent males were less likely to be abstinent than are females at follow-up after treatment. Adolescent females had reduced likelihood of reporting illegal activities or trouble controlling behavior, consistent with other studies focused on both adolescents (Balsa et al., 2009) and adults (Garnick et al., 2007; Hser et al., 2003b).

We also found that adolescents with the most serious clinical issues or behavioral problems at intake (e.g., high levels of victimization, Conduct Disorder, or truancy) were more likely to still report using some substances at six month follow-up and some problem behaviors, as also reported by other studies that do not consider treatment engagement (Arria, 2003; Godley et al., 2004b; Luchansky et al., 2006a; Williams et al., 2008). By and large, however, we did not confirm other research showing that co-morbid mental health disorders negatively influence abstinence (Godley et al., 2004a; Hser et al., 2003a; Tomlinson et al., 2004). Like previous research, we did not find that criminal justice involvement reported at intake among adolescents is associated with later substance abuse outcomes (Godley et al., 2004a) nor with illegal activities, although it is associated with reduced trouble controlling behavior at 6-month follow-up. This might be explained by the fact that both criminal justice involvement and treatment engagement entail supervision and, in the case of criminal justice involvement, sanctions. These could serve to curtail problematic behaviors including illegal activities or, at least, self-reports of illegal activities.

Race/ethnicity was significant only for Black adolescents and alcohol outcomes. Compared with White adolescents, Black adolescents were less likely to report alcohol use and heavy alcohol use at six-month follow-up. This is congruent with studies showing that a smaller proportion of Black adolescents and adults use alcohol than their White counterparts (Dauber et al., 2009; Jones-Webb, 1998). The current findings, however, contrast with findings from an earlier examination of this dataset, which did not find any differences in substance use outcomes by race/ethnicity (Godley et al., 2011b). These differences in findings related to race/ethnicity are not too surprising, however, given the differences in the two studies' sample size and analytic approach. For example, in the Godley study, sites dropped from the study were significantly more likely to have a larger proportion of African-American participants compared to those retained. Additionally, in the Godley study, engagement was not included as an explanatory variable and the outcomes were measured differently.

4.3 Limitations

Despite the advantages offered by using the rich set of adolescents' characteristics, treatment measures, and outcomes provided in GAIN data, there are limitations to our study. First, each of the treatment programs in this study received federal grant funds, training, and on-going coaching to support implementation of an evidence-based practice (Godley et al., 2011a). Thus, while the current findings suggest outpatient engagement is related to important substance use outcomes when an evidence-based practice is implemented well, it is not known to what extent the current findings can be generalized to settings where evidence-based practices are not being implemented or where implementation is poor. Nor is it known how well these findings generalize to residential treatment for adolescents and additional research is needed to confirm these findings in other settings. Second, like other non-randomized, observational studies, there may be unobserved or uncollected variables, which if included in our models would influence both engagement and outcomes (e.g., adolescents with stronger motivation at entrance to treatment may have both higher engagement and better outcomes). Consistent with the Behavioral Model of Health Services Use (Andersen and Davidson, 2007), which posits that an array of individual and contextual characteristics influence health services use, we include a rich collection of adolescents' clinical and behavioral characteristics in our models, so the potential effect of unobserved variables on our outcomes is probably lessened. Third, while estimating client-level effects is our main focus, it still is important to take into account and suitably adjust for facility-level effects (Finney et al., 2011; Harris et al., 2010b). If our sample of adolescents had come from more facilities, it might have been useful to include facility-level covariates in our hierarchical model. Indeed, we tested facility-level engagement rates, which were not significant. We do, however, note that these facility random effects contribute less than 3% of the total variance in our models, and thus facility level characteristics do not represent a major factor for this study. Additionally, we did run sensitivity analyses controlling for facility engagement rates, with no statistically significant alteration of results. Finally, both outcomes and most covariates in our models come from adolescents' self-report. However, the GAIN data was collected by interviewers who passed a rigorous training and quality assurance certification process teaching them to ask questions in a standardized manner and provide appropriate assurances of confidentiality. Moreover, the GAIN has shown good concurrent validity with the Form 90, a timeline follow-back method as well as biometric assessments of substance use (Dennis et al., 2004), lending confidence in this information which is widely used in adolescent treatment research.

4.4 Implications

This is the first study to examine the association between outpatient treatment engagement and outcomes among adolescents. After adjusting for demographic, clinical and behavioral characteristics, and the baseline measure of each respective outcome, treatment engagement was a strong predictor of substance use outcomes. These results contribute to the growing evidence that, at the individual level for outpatient treatment, the Washington Circle treatment engagement measure can be an important predictor of positive substance use outcomes. Given that most treatment programs do not yet routinely collect post-treatment outcome data, a performance measure that demonstrates predictive validity may be very useful. Moreover, it offers an almost immediate indication of clients for whom extra efforts will be needed to enable treatment success, and can be calculated solely from encounter data.

Performance measures are taking a more key role in driving quality improvement, accountability and efficiency in behavioral health in the context of health reform, the proliferation of electronic health records, and interest in addiction care that spans different stages of substance use recovery (McLellan et al., 2007). Thus, findings like those reported here that contribute to the evidence associating performance measures focused on processes of treatment with treatment outcomes take on vital importance. The Patient Protection and Affordable Care Act (2010) has the potential to expand treatment for substance use disorders to more Americans. With that expansion, accountability for quality and measures for incentive-based approaches will become even more important (Blumenthal and Tavenner, 2010; Bremer et al., 2008). As mental health and SUD treatment parity move forward (Barry et al., 2010), performance measures focused on substance use disorders also are urgently needed. Additionally, as electronic health records become common (Blumenthal and Tavenner, 2010), more providers of SUD treatment will be equipped to calculate and use both the treatment engagement measure that is the focus of this paper and also more nuanced approaches to measuring processes and outcomes in real time throughout the course of treatment.

There is an emerging consensus that optimal treatment includes following individuals longitudinally, monitoring their treatment and outcomes periodically, and making individualized patient-centered adjustments (McKay, 2009). Moreover, new treatment models will undoubtedly emerge with the transformation of the treatment system anticipated in the next decade, an increased focus on integration of specialty treatment for substance use disorders and medical care, and the growth of electronic health records. While these changes continue to unfold, treatment engagement remains an important tool for monitoring performance and justification for its use is strengthened with new evidence of an association with substance use outcomes in adolescent outpatient treatment.