Are Washington Circle Performance Measures Associated With Decreased Criminal Activity Following Treatment?

Abstract

This study examines the association between adherence to during-treatment process measures of quality (defined as initiation and engagement in treatment as developed by the Washington Circle) and outcome measures (defined as arrests and incarcerations) in the following year. The data come from the Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS) administrative data system linked to data from state agencies involved in criminal justice. Clients who initiated a new episode of outpatient treatment and who engaged in treatment were significantly less likely to be arrested or incarcerated during the following year. Initiation of substance abuse treatment alone, without engagement in treatment, was not significantly associated with arrests or incarcerations. These findings validate the clinical importance of the Washington Circle performance measures of initiation and engagement. These “process of care” measures can make a difference when used as a target for quality improvement at treatment facilities.

1. Introduction

Quality measures based on processes of care that are known to lead to positive outcomes are vitally important to the substance abuse treatment system. Substance abuse treatment providers are often able to influence control processes, such as the types and timing of services that they provide to clients, but providers are less able to influence outcomes which are influenced by multiple factors outside of treatment services. The fact that providers can influence processes led the Washington Circle to develop substance abuse performance measures designed to capture a floor of minimally acceptable services in the early stages of treatment of substance abuse (McCorry, Garnick, Bartlett, Cotter, & Chalk, 2000).

These substance abuse performance measures have been adopted by a national accrediting organization and are used by the Veterans Administration. Whether these process-based performance measures are associated with improved outcomes has not been studied, however. We designed this study to test whether such an association exists with criminal activity. Focusing on new episodes of outpatient treatment, we examine the association between meeting the measures of initiation and engagement in substance abuse treatment and arrests or incarceration during the following year.

The impetus for Washington Circle’s developing the performance measures was the body of evidence found in many studies that have shown the effectiveness of substance abuse treatment on post treatment outcomes. These studies often showed that clients who stay longer in treatment have better outcomes including substance use, employment, and criminal justice involvement (Etheridge, Craddock, Hubbard, & Rounds-Bryant, 1999; Gossop, Marsden, Stewart, & Rolfe, 1999; Hubbard, Craddock, & Anderson, 2003; Simpson, Joe, & Rowan-Szal, 1997; Zarkin, Dunlap, Bray, & Wechsberg, 2002).

1.1 Substance Abuse Performance Measures

Using the classic categories of quality measures (Donabedian, 1980), the Washington Circle measures can be categorized as “process” measures that are focused on adherence to recommendations for clinical practice based on evidence or consensus. By 2000, the Washington Circle had developed a model of the process of care in four domains representing the continuum of substance abuse services -- prevention/education, recognition, treatment, and maintenance (McCorry et al., 2000). Three measures, identification, initiation, and engagement, were further developed and pilot tested (Garnick, Lee et al., 2002), resulting in their adoption by the National Committee for Quality Assurance (NCQA, 2004), the Veterans Administration (Harris, 2006), and the state of Oklahoma (Oklahoma Department of Mental Health and Substance Abuse Services, 2006) An additional set of states is considering refinements to these measures using data they routinely collect on publicly-funded clients.

For new episodes of outpatient services, the focus of this study, these measures are defined as:

Initiation: the percent of adults with an index outpatient service for substance abuse or dependence and any additional substance abuse services within 14 days following the index service that denotes the beginning of a new episode of treatment.

Engagement: the percent of adults with substance use disorders that receive two additional substance abuse services within 30 days after the initiation of care.

By focusing on the early stage of treatment, these measures capture necessary first steps before the longer-term continuation in treatment that may be clinically recommended.

These performance measures have multiple applications including accountability, quality improvement and, more recently “pay for performance” approaches in which providers or provider groups receive financial incentives based on their performance (Horgan & Garnick, 2005). To be most useful, performance measures need to be easily measured and credible to all stakeholders, including payers and providers. Exploring the relationship between process measures and outcomes is an important element in establishing that credibility.

1.2 Criminal Justice Outcomes After Substance Abuse Treatment

Criminal justice involvement is a key outcome measure to consider because of the high prevalence of criminal activity among individuals with substance abuse problems and the large associated social costs (Rajkumar & French, 1997). Therefore, a decrease in criminal justice involvement is an important outcome measure of the effectiveness of substance abuse treatment. Analyses of data from the Drug Abuse Treatment Outcome Studies (DATOS) show that longer time in outpatient treatment, defined as six months to a year or more than a year are not associated with decreased predatory illegal behavior although longer stays in residential treatment reduced criminal justice involvement (Hubbard et al., 2003). In a study focused on length of stay and treatment completion, Zarkin and colleagues (Zarkin et al., 2002) report that among adults that stayed at least a week in outpatient treatment, those with longer lengths of stay were less likely to commit a crime during the one-year followup period but that treatment completion did not have a statistically significant effect.

Additional evidence comes from analyses of data from a national treatment outcomes study conducted in England that have shown promising findings that treatment leads to reduction in criminal activities (Gossop et al., 1999; Gossop, Marsden, Stewart, & Rolfe, 2000; Groppenbacher, Bemis Batzer, & White, 2003; Healey, Knapp, Marsden, Gossop, & Stewart, 2003). Groppenbacher and colleagues found a reduction in arrests after residential treatment; and those who entered outpatient treatment following residential treatment had even better outcomes than clients who only had residential treatment (Groppenbacher et al., 2003). Gossop and colleagues found a significant reduction in crimes committed and in the number of clients who committed crimes at one-year follow-up (Gossop et al., 1999).

1.3 Use of State Administrative Data for Research

When sufficient resources and effort are provided to support accurate data collection, state administrative data systems hold a wealth of readily available information which can be used to answer research questions and monitor performance (Alterman, Langenbucher, & Morrison, 2001; McCarty, McGuire, Harwood, & Field, 1998). Integration of such data from multiple state databases can be even more important for research and evaluation (Garnick, Hodgkin, & Horgan, 2002; Saunders & Heflinger, 2004). A critical aspect of this study, focusing on Oklahoma substance abuse treatment, is the ability to link data collected by the state substance abuse agency to other state databases such as for employment and criminal justice. Such data linkage enables us to explore the relationship between substance abuse treatment and improved outcomes.

Many of the previous studies of criminal justice outcomes using linked state treatment and criminal justice information have been focused on data from Washington State. Analysis of a sample of Supplemental Security Income (SSI) recipients with prior criminal involvement in Washington State who entered substance abuse treatment had a reduction in risk for arrests, felony convictions, and convictions for lesser crimes in the year after treatment. Furthermore, those with treatment episodes of at least 90 days were associated with lower risk for criminal activity.(Luchansky et al., 2006) Luchansky and colleagues (Luchansky, Krupski, & Stark, 2007) also compared outcomes for methamphetamine patients with those who used other substances. Criminal justice involvement for methamphetamine patients was found to be similar to those who used other hard drugs, but was higher than for those who used alcohol and marijuana.

New Mexico (Ellis, McCan, Price, & Sewell, 1992), Oregon, Oklahoma, and Connecticut also have used existing state databases to conduct research. In a review of outcome studies that use state administrative databases, the Oregon State Study showed that clients who completed treatment had fewer incarcerations and 75 percent higher wages than those who did not complete treatment (Alterman et al., 2001). The state of Connecticut linked criminal justice data with its substance abuse treatment database and found that rates of arrests in the year after treatment decreased by 12 percent for those individuals treated in 2000, 2001, or 2003. Men and non-whites had the greatest reduction in arrest rates post- treatment.(Connecticut Department of Mental Health and Addiction Services, 2005) Finally, in Vermont, the proportion of substance abuse service recipients who were charged with a crime in fiscal year 2004 decreased by 28% from the three months prior to treatment compared with the three months after treatment (J Pandiani & Hunter, December 30, 2005). Using older data from 1999–2001, there also was a two-thirds decline in arrest rates from the year before to the year after treatment, although changes in incarceration were much lower (J. Pandiani & Simon, August 20, 2004).

1.4 Analytic Approach of This Study

Our study of linked Oklahoma data differs from previous work in three ways. First, we focus on the effect of meeting criteria for treatment initiation and engagement, performance measures defined by the Washington Circle (www.washingtoncircle.org), while previous work was focused on treatment completion and length of stay. We are able to calculate initiation and engagement using encounter data that includes information on the date, location and type of services for each client, while most previous studies have relied on data on admissions and discharges. Second, to measure criminal justice involvement, we use information from Oklahoma State criminal justice agencies which had been linked with treatment data, while most previous work used self-report of crime.

Third, as an analytic approach, we use a survival analysis model with a censored, continuous-valued outcome measuring time-to-adverse criminal justice event, while previous substance abuse treatment studies have been evaluated with a dichotomous outcome model where treatment outcomes measure a client’s reduction in substance use and criminal activity based on their status six to twelve months post-treatment. Such a model might be considered appropriate for an acute disease, where the effect of treatment might be measured by how frequently it results in a definitive positive outcome. For chronic diseases, in which treatment often is periodic and ongoing across time, measurement of the effect of an individual treatment episode is more suitably measured with a continuous-valued outcome such as time to next adverse event (e.g., time to next hospitalization, new treatment episode, or other negative consequence). Survival analysis is appropriate given the chronic nature of substance abuse (McLellan, McKay, Forman, Cacciola, & Kemp, 2005). Thus, we examine the impact of treatment on the continuous-valued outcome of time to criminal justice involvement in the form of arrests and incarceration. Specifically, our study uses Cox proportional hazard models to address the following research question:

Is achieving substance abuse treatment initiation or engagement as defined by the Washington Circle associated with a reduction in a client’s likelihood of post-treatment arrests and incarcerations?

2. Methods

2.1 Data Source

This study covers adult clients treated in publicly-funded Oklahoma substance abuse outpatient programs as recorded in the Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS) administrative data system. This data system includes information from their Integrated Client Information System (ICIS), which contains a Client Data Core with client self-reported information on admissions as well as information on dates and types of services. The pre-admission information is collected by the clinician through a structured interview using the Addiction Severity Index assessment instrument. All information is reported to a statewide, central repository in a standardized format, following the federal Drug and Alcohol Services Information System (DASIS) standards. Data are validated through a series of system edits to ensure responses are logical and fall within the allowed value ranges. If errors are found, the treatment record is returned to the provider agency for correction. Preadmission information must be deemed complete and accurate before services can be reported and reimbursed.

Also linked are data from state agencies involved in criminal justice, including Department of Corrections (incarcerations), Department of Public Safety (DUI convictions), and Oklahoma State Bureau of Investigation (arrests), and Employment Security Commission (which provides quarterly employment records). Arrest information is collected by county and city law enforcement agencies and reported in the Uniform Crime Reports standardized format to the Oklahoma State Bureau of Investigation. The database is comprised of arrests occurring within the state, excluding federal crimes. All felonies and serious misdemeanors are included, with less serious misdemeanors included if they are committed with one of the other more serious categories.

Information from these databases was linked using probabilistic matching on name, gender, birth date and social security number based on well established methods (Jaro, 1995). The academic members of our research team were able to access this data by ensuring client confidentiality through masking of some information on exact dates of service (instead using count of days from an index service happening some time in a year period) according to the regulations of the federal Health Insurance Portability and Accountability Act (HIPAA) of 1996. To be included in the study, an adult client age 18 years or older had to have received any substance abuse treatment in 2001. Linked data for one year pre and one year post treatment were used to construct the variables in our analysis.

Construction of the sample took place in several steps. An initial draw from the ICIS identified 14,855 adults receiving any outpatient treatment in 2001. To avoid incomplete or out-of-date information on clients, our analysis was restricted to adults with a new treatment episode during the year 2001 for whom current socio-demographic data was recorded (assessment had to be within a month before the first service of the new treatment episode, the so called “index” service). A new episode was defined as a substance abuse service that occurred after a 60-day period with no other substance abuse treatment. We excluded three groups of clients from the outpatient sample: 7,258 clients who did not have a new episode of outpatient treatment in 2001; 915 youth under 18 years of age at the beginning of a new treatment episode, and 1,354 clients for whom current client socio-demographic information was not recorded. This resulted in an analysis sample of 5,328 outpatient clients.

The outcome variable in our Cox proportional hazard models was time from index service to first arrest or incarceration after treatment engagement. This variable was censored by the earlier of 365 days after discharge or time to residential treatment admission or death - the last two representing competing events that so effectively change the likelihood of arrest or incarceration, that the only solution is to censor any such client. Such an outcome variable presents a minor analytic dilemma because 158 clients died, were incarcerated or were arrested within 45 days of index service, the last day by which a subject’s engagement status could be determined. Either inclusion or exclusion of these subjects in a survival analysis generates a small bias; however the results were no different whether these subjects were included or not. Therefore, we excluded these subjects from the analysis because the majority did not engage and the resulting bias of exclusion would be in favor of the null hypothesis of no treatment effect. After exclusion of these additional 158 subjects, the final analysis file consisted of 5,169 outpatient clients.

2.2 Variables

Based on the literature regarding key influences on arrests and incarceration for clients undergoing treatment for substance use disorders, we included the client-level regressors listed below in our survival analysis models. Variables providing information on client socio-demographics, self reported substance use, and referral source come from information in the ICIS database. Prior year arrest and DUI comes from the linked criminal justice data and prior year employment comes from the linked Employment Security Commission data. Variables on substance abuse initiation and engagement were calculated from the services information that includes date and location of services. In addition, indicator variables (yes/no) were included for each treatment facility with 50 or more outpatient clients in the sample.

Demographics

age (18–20/21–30/31–44/45+)

gender (female/male)

race (white/black/Native American/other)

education (less than high school/high school graduate or more)

marital status (single/married/divorced, separated or widowed)

homeless (yes/no)

Substance Use

alcohol abuse (none/<3 times in past week/3+ times in past week)

marijuana abuse (none/<3 times in past week/3+ times in past week)

amphetamine abuse (yes/no use in past month)

methamphetamine use (none/<3 times in past week/3+ times in past week)

cocaine abuse (yes/no use in past month)

heroin abuse (yes/no use in past month)

other drug abuse (yes/no use in past month)

Prior Year Characteristics and Referral Source

arrest or incarceration by quarter in year prior to index (yes/no for each quarter)

any DUI in year prior to index (yes/no)

any employment in year prior to index (yes/no)

referral source (self or significant other/employer/social services/health services/criminal justice)

The primary variables of interest in the model were treatment initiation and engagement as defined by the Washington Circle, which were calculated from the services information in the ICIS. An outpatient client was considered to have initiated if he or she had another substance abuse service (excluding detoxification) within 14 days after the index service. The client was considered to have engaged in treatment if he or she met the initiation criteria and had two additional substance abuse services within 30 days of initiation. We also explored the sensitivity of our results to changes in these definitions of allowing initiation to include services within 21 days of the index services and engagement to include services within 45 or 60 days of initiation. Other regressors were included in the survival models to estimate and adjust for the effect of other client level factors on outcomes. To test whether the effects of initiation and engagement might differ among subgroups of clients, auxiliary models were run which included interactions between initiation and engagement and other client characteristics.

2.3 Analytic Technique

Our analyses consist of multivariate Cox proportional hazards regressions to examine the effect of treatment initiation and engagement status on hazard of arrest/incarceration, after adjusting for potential differences in program and other confounding covariates at the client-level. Cox proportional hazards regressions are widely used in the analysis of time-to-event data with right censoring (Box-Steffensmeier & Jones, 2004; Fisher & Lin, 1999), meaning that for some observations the period of monitoring ends before the event occurs. Rather than resorting to the inherently biased step of omitting or truncating such observations, Cox proportional hazard regressions make maximum unbiased use of information available in the data in carrying out the analysis. In this study, time to arrests/incarcerations cannot be determined and are, therefore, marked as ‘right censored’, if the client died, entered a residential program, or the arrest/incarceration has not yet occurred within 365 days of the index service.

3. Results

3.1 Descriptive Statistics

Table 1 shows summary demographic characteristics of the full sample and of three subgroups: those who had an index event but did not initiate treatment, those who initiated but did not engage in treatment, and those who initiated and engaged in treatment. Comparison of means show that summary demographic characteristics across the three sub-samples were fairly similar. The full sample of outpatient clients were mostly between the ages of 21 to 30 (36%) and 31 to 44 (40%), male (60%), white (68%), and high school graduates (65%).

Table 1.

Demographic Characteristics of the Outpatient Analytic Sample

		Means (Standard Deviations)
Variables	Full sample (N = 5,169)	a. Clients With Index Service Only (N = 1,644)	b. Clients Who Initiated Only (N = 874)	c. Clients Who Initiated and Engaged (N = 2,651)
Demographics
Age
18–20	.11 (.32)	.12 (.32)	.13 (.34)	.10 (.31)
21–30 a vs. c**	.36 (.48)	.38 (.48)	.38 (.48)	.34 (.47)
31–44	.40 (.49)	.38 (.49)	.37 (.48)	.41 (.49)
45+ a vs. c*	.13 (.34)	.11 (.32)	.12 (.33)	.14 (.35)
Gender
Male	.60 (.49)	.61 (.49)	.57 (.49)	.61 (.49)
Female	.40 (.49)	.39 (.49)	.43 (.49)	.39 (.49)
Race
White	.68 (.47)	.68 (.46)	.68 (.46)	.68 (.47)
Native American	.16 (.36)	.14 (.35)	.18 (.38)	.16 (.37)
Black	.13 (.33)	.14 (.35)	.11 (.31)	.12 (.33)
Other	.04 (.19)	.03 (.18)	.03 (.17)	.04 (.20)
Education
High School	.65 (.48)	.64 (.48)	.63 (.48)	.66 (.47)
Not High School	.35 (.48)	.36 (.48)	.37 (.48)	.34 (.47)
Marital Status
Married a vs. c*	.28 (.45)	.26 (.44)	.29 (.46)	.30 (.46)
Never married a vs. c*	.36 (.48)	.38 (.49)	.37 (.48)	.35 (.48)
Divorced, separated, widowed	.35 (.48)	.36 (.48)	.34 (.47)	.35 (.48)
Homeless Living Arrangement a vs. c**	.02 (.15)	.03 (.18)	.03 (.17)	.02 (.13)

^*Pairwise comparisons noted are significant at * p<.05,

^**p<.01 using a Bonferroni adjustment.

In terms of substance use, referral source and prior year characteristics, there are a few significant pair wise comparisons, but the differences are small (Table 2). A slightly lower proportion of clients who were self-referred or referred by a significant other engaged in treatment (28%) compared to the proportion who had an index service only (33%) and those who initiated only (36%). Most clients were either self-referred or referred by a significant other (31%) or by the criminal justice system (46%). A lower proportion of clients who initiated only (41%) were referred from the criminal justice system compared with clients with an index service only (46%) or initiation and engagement (48%). The majority was employed in the year prior to treatment (69%).

Table 2.

Susbstance Use, Referral Source, and Prior Year Characteristics of the Outpatient Analytic Sample

		Means (Standard Deviations)
Variables	Full sample (N = 5,169)	a. Clients With Index Service Only (N = 1,644)	b. Clients Who Initiated Only (N = 874)	c. Clients Who Initiated and Engaged (N = 2,651)
Substance Use
Alcohol
None in past month	.63 (.48)	.63 (.48)	.63 (.48)	.63 (.48)
< 3 times in past week	.22 (.41)	.23 (.42)	.23 (.42)	.20 (.40)
≥3 time in past week	.15 (.36)	.15 (.35)	.15 (.35)	.16 (.37)
Marijuana
None in past month	.74 (.44)	.74 (.44)	.75 (.43)	.74 (.44)
< 3 times per week	.12 (.32)	.12 (.33)	.11 (.31)	.12 (.32)
≥3 time per week	.14 (.34)	.13 (.34)	.14 (.35)	.14 (.34)
Amphetamine
None in past month a vs. c**	.97 (.18)	.98 (.14)	.97 (.16)	.96 (.20)
Any use in past month a vs. c**	.03 (.18)	.02 (.14)	.03 (.16)	.04 (.20)
Methamphetamine
None in past month	.89 (.31)	.90 (.30)	.89 (.32)	.89 (.31)
< 3 times in past week	.04 (.20)	.04 (.20)	.05 (.22)	.04 (.19)
≥3 time in past week	.06 (.25)	.06 (.23)	.06 (.24)	.07 (.26)
Cocaine
None in past month	.93 (.25)	.93 (.26)	.93 (.25)	.93 (.25)
Any use in past month	.07 (.25)	.07 (.26)	.06 (.25)	.07 (.25)
Heroin
None in past month a vs. c*	.96 (.19)	.97 (.17)	.96 (.18)	.95 (.21)
Any use in past month a vs. c*	.04 (.19)	.03 (.17)	.03 (.18)	.04 (.21)
Referral from
Self or significant other a vs. c**, b vs. c***	.31 (.46)	.33 (.47)	.36 (.48)	.28 (.45)
Employer or school	.02 (.13)	.02 (.12)	.01 (.12)	.02 (.13)
Criminal justice a vs. b*, b vs. c***	.46 (.50)	.46 (.50)	.41 (.49)	.48 (.50)
Social service a vs. c**	.13 (.34)	.11 (.32)	.14 (.35)	.14 (.35)
Health service	.07 (.26)	.07 (.26)	.08 (.27)	.07 (.26)
Prior Year Characteristics
Employed in Prior year	.69 (.46)	.72 (.45)	.67 (.47)	.69 (.46)
Driving Under the Influence (DUI) in prior year	.14 (.35)	.13 (.34)	.14 (.34)	.15 (.36)
Arrest or Incarceration in prior year
1st quarter of prior year	.06 (.25)	.06 (.25)	.06 (.24)	.07 (.25)
2nd quarter of prior year	.08 (.27)	.08 (.27)	.07 (.26)	.07 (.26)
3rd quarter of prior year	.09 (.29)	.10 (.30)	.08 (.27)	.10 (.30)
4th quarter of prior year	.09 (.29)	.09 (.28)	.08 (.27)	.10 (.30)

^*Pairwise comparisons noted are significant at * p<.05,

^**p<.01,

^***p<.001 using a Bonferroni adjustment.

The overall initiation rate among outpatient adult clients was 68% and the engagement rate was 51% (Table 3). Initiation and engagement rates by selected client characteristics are also shown in Table 3. Clients age 45 or older initiated treatment at a significantly higher rate (72%) compared to clients age 21 to 30 (66%). Men and women initiated and engaged in substance abuse treatment at about the same rate. Compared to black clients who initiated at a rate of 64%, Native Americans initiated treatment at a significantly higher rate of 72%. However, there was no significant difference in engagement rates among the three racial groups. The engagement rates were higher for older clients as indicated by an engagement rate of 53% for those age 31 to 44 and an engagement rate of 56% for those age 45 or older compared to 47% for those age 18 to 20. Clients 45 or older also engaged in treatment at a higher rate compared to clients age 21 to 30.

Table 3.

Outpatient Initiation and Engagement Rates

	Initiation Rate (Standard Deviation)	Engagement Rate (Standard Deviation)
Total Outpatient (N=5169)	68% (0.46)	51% (0.50)
Selected Client Characteristics
Age
18–20 (a)	66% (0.47)	47% (0.50) a vs. d**
21–30 (b)	66% (0.47)) b vs.d*	48% (0.50) b vs. d**
31–44 (c)	69% (0.46)	53% (0.50) b vs.c**
45+ (d)	72% (0.45)	56% (0.50)
Gender
Male (e)	68% (0.47)	52% (0.50)
Female (f)	69% (0.46)	50% (0.50)
Race
White (g)	68% (0.47)	51% (0.50)
Black (h)	64% (0.48) h vs. i	49% (0.50)
Native American (i)	72% (0.45)	53% (0.50)

^*Pairwise comparisons noted are significant at * p<.05,

^**p<.01 using a Bonferroni adjustment.

3.2 Cox Proportional Hazards Regression

Table 4 provides results of our Cox proportional hazards regression for clients first treated with outpatient services. The table shows that among the outpatient clients, those who initiated and engaged in treatment were significantly less likely have arrests or incarcerations (hazard ratio = 0.73, 95% CI = 0.62 – 0.87). Initiation of substance abuse treatment alone without engaging in treatment was not significantly associated with the likelihood of arrests/incarcerations.

Table 4.

Time-to-Event Analysis of Arrest or Incarceration Following Index Outpatient Substance Abuse Treatment (N=5,169)

Independent Variables	Hazard Ratio (95% CI)
Treatment Process Variables (ref = index only)
Initiation Only	0.94 (0.76 –1.16)
Initiation and Engagement	0.73 (0.62 – 0.87) ***
Demographics
Age (ref = 18–20)
Age 21–30	0.91 (0.72 –1.17)
Age 31–44	0.95 (0.73 – 1.23)
Age 45+	0.68 (0.48 – 0.96)*
Gender (ref = male)
Female	0.76 (0.64 –0.90) ***
Race (ref = White)
Native American	1.11 (0.90 – 1.38)
Black	1.26 (1.01 –1.57)*
Other race	0.79 (0.51 – 1.24)
Education (ref= less than high school)
High School or greater	0.86 (0.74 – 1.00)*
Marital Status (ref = never married)
Married	0.86 (0.71 – 1.05)
Divorced, separated, or widowed	0.91 (0.75 – 1.12)
Living Arrangement (ref = not homeless)
Homeless	1.40 (0.94 – 2.10)
Substance Use (ref = none for each category)
Alcohol, < 3 times in past week	0.90 (0.74 – 1.08)
Alcohol, ≥ 3 times in past week	0.90 (0.72 – 1.12)
Marijuana, < 3 times in past week	1.19 (0.95 – 1.49)
Marijuana, ≥ 3 times in past week	1.30 (1.06 – 1.61)*
Amphetamine, any use in past month	1.27 (0.86 – 1.88)
Methamphetamine, < 3 times in past week	1.07 (0.74 – 1.54)
Methamphetamine, ≥ 3 times in past week	1.38 (1.03 – 1.84)*
Cocaine, any use in past month	1.18 (0.91 – 1.53)
Heroin, any use in past month	0.72 (0.38 – 1.38)
Other Drugs, any use in past month	1.00 (0.58 – 1.71)
Prior Year Characteristics (ref = none for each category)
Employed in Prior year	1.10 (0.93 – 1.30)
Driving Under the Influence (DUI) in prior year	1.54 (1.27 – 1.88)****
Arrest or Incarceration in Prior Year
Arrest or Incarceration 1st quarter of prior year	1.81 (1.45 – 2.26)****
Arrest or Incarceration 2nd quarter of prior year	1.80 (1.46 – 2.22)****
Arrest or Incarceration 3rd quarter of prior year	1.84 (1.51 – 2.23)****
Arrest or Incarceration 4th quarter of prior year	2.04 (1.69 – 2.47)****
Referral Source (ref = self or significant other)
Employer or school	0.45 (0.18 – 1.10)
Criminal justice	1.04 (0.86 – 1.26)
Social service	1.03 (0.76 – 1.38)
Health service	0.86 (0.60 – 1.23)

Note: A hazard ratio above 1.00 indicates that arrest or incarceration is more likely; a hazard ratio below 1.00 indicates that arrest or incarceration is less likely.

^*p<.05

^**p<.01

^***p<.001

^****p<.0001

Other client characteristics associated with lower hazard of arrests and incarcerations were age, sex, and education. Outpatient clients who were 45 or older were less likely be arrested or incarcerated (hazard ratio = 0.68, 95% CI = 0.48 – 0.96) compared to those age 18 to 20 and the interaction of age over 45 and treatment engagement was significantly associated with reduced arrests or incarcerations (data not shown). In addition, women were less likely to be arrested or incarcerated (hazard ratio = 0.76, 95% CI = 0.64 –0.90) compared to men, although the interaction of female and treatment engagement was not significant (data not shown). Those who were high school graduates were less likely to be arrested or incarcerated (hazard ratio = 0.86, 95% CI = 0.74 – 1.00) compared to those who did not graduate high school. A variable with a marginally significant hazard ratio was the indicator for being referred to treatment by an employer or school (hazard ratio = 0.45, 90% CI = 0.18 – 1.10, p<.08).

Variables associated with greater likelihood of arrests/incarcerations were being black (hazard ratio = 1.27, 95% CI = 1.01 – 1.57), although interactions of race and treatment engagement were not significant (data not shown). In addition, those reporting at admission use of marijuana more than three times in the past week (hazard ratio = 1.30, 95% CI = 1.06 – 1.61), use of methamphetamines more than three times in the past week (hazard ratio = 1.38, 95% CI = 1.04 –1.84), and those having a history of being arrested/incarcerated or having a DUI in the year before the index service were all associated with an increase likelihood of being arrested or incarcerated. The strongest predictor of criminal activities after treatment was criminal behavior before treatment. The hazard ratios for arrest/incarceration in the each quarter of the year prior to the index treatment ranged from 1.80 to 2.04 while the hazard ratio for driving under the influence (DUI) in the prior year was 1.53 (95% CI = 1.26 – 1.88).

3.3 Sensitivity Testing

We tested the sensitivity of our results in three areas: alternate definitions of initiation and engagement, approaches to accounting for clustering of clients within programs, and definition of criminal justice involvement. First, we tested whether the regression results for initiation and engagement were stable across alternate definitions of these process measures to examine if the results are found only using the exact time periods specified by the Washington Circle. Using a longer time period for initiation of 21 rather than the original 14 days, the initiation rate increased from 68% to 74%. Similarly, expanding the time period for engagement showed that the engagement rates rose by no more than 5%, ranging from 51% for a 30 day window, to 54% for 45 days, and 56% for 60 days. For all combinations of the definitions we tested, the regression results remained similar to the results found with the original Washington Circle specifications.

Second, we tested the sensitivity of our results to using differing analytic techniques. Our analyses deal with the issue of systematic outcome differences among programs by including 0–1 indicators for the 33 programs with at least 50 clients represented in the analysis sample. The remaining 470 adult outpatient clients, treated at 29 smaller programs, are combined to form a reference group, who receive zeros on all program indicators. In the semi-parametric Cox proportional hazards model, this ‘indicator’ method of dealing with the nesting of outcomes within program implies that the baseline hazard functions of similar clients in different programs (say i and j), vary by the constant ratio, exp(β_i − β_j), where β_i represent the program effect estimated in the regression. We test the sensitivity of results on this strong ‘constant ratio’ assumption by comparing them to results of two alternative modeling approaches, ‘shared frailty’ (Gutierrez, 2002) and stratified (Kleinbaum & Klein, 2005). In a shared frailty model, the ratio of baseline hazard functions of similar clients in different programs, i and j, are again expressed as exp(ω_i − ω_j), but the ω_i’s are considered random effects satisfying certain distributional requirements rather than coefficients estimated in the regression. Using a stratified approach, the baseline hazard functions of clients in different programs are allowed to be totally unrelated to each other. Analyses using these two alternative approaches show virtually no differences in either the size or significance of any effect of interest, providing evidence that our results are robust with respect to the particular method of addressing program differences. Third, we tested the sensitivity changing the definition of criminal justice involvement to limit it to arrests only. We found no difference in our key finding of an association of substance abuse treatment engagement with decreased arrests, a stability that is similar to other reports of a high degree of correspondence across four criminal justice outcomes including any arrest, felony arrest, any conviction, and felony conviction (Luchansky et al., 2007).

4. Discussion

Our analyses indicate that among Oklahoma outpatient clients, engagement in substance abuse treatment is associated with a decrease in the likelihood of arrests and incarcerations. Initiating substance abuse treatment alone does not appear to be associated with significant outcome differences. Previous research has shown the positive benefits of longer lengths of treatment and of treatment completion, but the present findings show that engagement in the early stages of substance abuse treatment, for clients with even two additional visits within 30 days after the initiation of outpatient care is associated with a positive difference in outcomes. Moreover, this finding is robust to various specifications of the initiation and engagement measures in which we altered the time periods for calculating engagement. This finding demonstrates an important association between treatment engagement, a performance measure based on the concept of a “process measure” of quality of care, and reduced criminal justice involvement, an “outcome measure.”

Researchers and policymakers have debated about the superiority of outcome measures (patients’ health or disability) or process measures (health care services provided to patients) (Mant, 2001; McAuliffe, 1979; Rubin, Pronovost, & Diette, 2001). Outcome measures have the advantage of being focused on goals that are important in their own right and make intuitive sense (such as abstinence, improved functioning, reduced criminal justice involvement, or employment). However, there may be a time lag before such outcomes are apparent and outcomes often are expensive to collect. Improved outcomes often depend on patient compliance or multiple processes of care, and while outcome measures may suggest specific areas of care that may require quality improvement, further investigation is typically necessary to determine how to influence outcomes. Process measures, on the other hand, which are used to assess adherence to clinical practices based on evidence or consensus, offer the advantages of being more immediately actionable and often useful for identifying specific areas of care that may require improvement. However, pitfalls include the danger of focusing on processes that do not relate to outcomes and concerns that process measures do not always provide compelling feedback.

Despite some ongoing debate, there is general consensus that both types of measures are important and that process measures are useful.(Krumholz, Normand, Spertus, Shahian, & Bradley, 2007) For process measures that focus on the provision of treatment services, it is important to demonstrate the probability of improved outcomes for individuals who meet the measure’s specifications. The gold standard for establishing causality regarding the association between process measures and outcomes is a randomized clinical trial, an approach that often is used in substance abuse to establish the impact of specific treatment approaches on outcomes such as abstinence, treatment utilization, or costs. This approach is not practical when it is not possible to randomize clients into different kinds of treatment. Thus, we adopted a widely used alternate approach to examine the association between process measures and outcomes and constructed a model controlling for other factors that also influence the key outcome of interest. Moreover, the results reported here focus on “treatment as usual” by providers across a state whereas clinical trials generally are focused on formal treatment strategies.

4.1 Monitoring and Quality Improvement

Our analysis offers initial support for using substance abuse performance measures based on concepts of process of care, such as the Washington Circle measure for engagement, to target initiatives to improve the quality care in outpatient substance abuse treatment. The research we report here suggests that services delivered during this early phase are associated with better outcomes. Because engagement can be monitored on an immediate basis rather than waiting for an adverse outcome to occur, this measure can be used as an indicator that providers need to take steps to target improvement. Having identified that need, they can turn to a range of resources for guidance on next steps. For example, one current initiative aimed at working with providers to determine how to improve early retention in treatment is the Network for the Improvement of Addiction Treatment (NIATx, 2006). In addition, there are current efforts to define (Miller, Zweben, & Johnson, 2005) and identify evidence-based practices for substance use disorders (National Quality Forum, 2005).

While the main focus of our research is on the relationship between process and outcome measures, our approach also is related to a newly proposed approach to collecting outcome data. Many studies tend to measure outcomes after treatment at specified time such as six-months, or one-year post-treatment. McLellan and colleagues suggest, however, that for many clients, both treatment for substance abuse and monitoring of the outcomes that they achieve need to be ongoing in nature, as it is in many other chronic conditions. Therefore, they argue, outcome measures should be collected at multiple points of time during the course of treatment and used to direct the course of treatment, rather just at a specified time post-treatment (McLellan et al., 2005). If substance abuse treatment, employment, and criminal justice data -- such as we used for the research reported here -- could be merged on a real time basis, this information could be used to supplement treatment and outcome information collected directly from patients for continuous treatment monitoring.

4.2 Additional Influences on Criminal Justice Outcomes

Some important differences in criminal justice outcomes were found among groups based on prior criminal justice involvement, race and extent of self-reported use of some drugs, along with some other sociodemographic characteristics of clients in the sample. These findings suggest that it is key to focus on specific groups in efforts to improve treatment outcomes. Treatment engagement, while associated with reduced criminal justice involvement, is only one component of potential influences on outcomes that are largely influenced by clients’ characteristics and past experience.

The strongest predictor of whether a client will have arrests or incarcerations after treatment is their criminal justice history in the year prior to treatment, although referral to treatment from the criminal justice system was not significant in our regression results. Similar to others’ reports, we also show that those who had arrests for driving under the influence or other arrests/incarcerations were more likely to have arrests/incarcerations after treatment. Past research found clients arrested in the year prior to treatment were more likely to be readmitted (Luchansky, He, Krupski, & Stark, 2000) and more likely to have an additional arrest after treatment (Luchansky et al., 2007; Zarkin et al., 2002). We also found that black clients who start treatment in outpatient care are more likely to have arrests and incarcerations after treatment compared to white clients, even after controlling for employment status, education, prior arrests and other factors. We did not find, however, a significant interaction between race and treatment engagement, indicating that the lower criminal justice involvement activity for clients who become engaged is independent of race.

It is crucial to consider the implications of these results regarding prior year arrests and race in concert, however. Certainly additional efforts related to substance abuse treatment need to be directed at black clients with previous criminal justice involvement to reduce the likelihood of further arrests and incarcerations after treatment. Such efforts may include culturally competent treatment programs (Howard, 2003), programs that support clients as they reconnect with their communities (Butzin, Martin, & Inciardi, 2005; Cooke, 2005), wraparound services such as employment, legal, or family services (Schmidt, Greenfield, & Mulia, 2006), and further exploration of the role of criminal justice referral to treatment as an influence on achieving treatment engagement.. Also key are earlier interventions that identify and intervene with individuals before they establish patterns of repeated arrests and substance use.

Some of the self-reported types and frequency of drug use were significantly associated with later criminal justice involvement. We found that clients who reported heavier marijuana or heavier methamphetamine use at intake were more likely to have arrests or incarcerations after treatment. This finding is consistent with one report that those treated on an outpatient basis for methamphetamine use were less likely to complete treatment if they were daily methamphetamine users and if they also reported alcohol or heroin use (Brecht, Greenwell, & Anglin, 2005). However, a recent study comparing methamphetamine and other drug users’ outcomes showed no significant differences in post-discharge criminal justice involvement compared with users of other hard drugs but worse outcomes compared with users of alcohol (Luchansky et al., 2007). These results suggest that while engagement is a necessary first step, efforts to retain clients in treatment for longer periods of time need to be strengthened, especially for populations with severe problems.

Positive factors related to improved criminal justice outcomes include findings showing that women were less likely to be arrested or incarcerated after treatment, although there was no association between gender and engagement. This gender difference has also been reported in previous work (Banks, Pandiani, & Bramley, 2001). In addition, Banks and colleagues found that men ages 35 to 49 were less likely to engage in criminal activities after treatment; the findings of the present study show all clients 45 years old or older less likely to be arrested or incarcerated after treatment. We found an interaction between age and engagement suggesting that older clients who are engaged have less criminal justice involvement. Moreover, older clients may be more likely to have the motivation or the resources to engage in treatment.

4.3 Limitations

Because we only used data from one state, the results reported here should be interpreted only as initial evidence of the relationship between process and outcomes for substance abuse treatment. Oklahoma offered an excellent opportunity for research because of its ongoing merged data, but differs from other states. More than half of Oklahoma’s population lives in the two largest metropolitan areas, while the rest of the state is considered rural or frontier. While three-quarters of the state’s population is white, Oklahoma has the largest percent of Native Americans of any state and each of the 39 recognized tribes has its own unique culture and traditions. Other distinctive characteristics of Oklahoma’s population include the distribution of racial groups, with counties in the eastern half of Oklahoma tending to have greater proportions of Native Americans, higher proportions of people who are older and a higher proportion of people with incomes less than 100% of the federal poverty level. In fiscal year 06, alcohol was the leading drug of choice for persons presenting for substance abuse treatment at 36%; however, methamphetamine had risen to tie with marijuana as the second highest drug of choice, both at 21%; followed by cocaine at 10%.

In addition, issues of data completeness need to be considered because our data may not include all treatment (independent variables) or all criminal activity (dependent variable). Incomplete data on treatment might occur, for example, if individuals receiving services through the Oklahoma Department of Mental Health and Substance Abuse Services also received other services funded by additional sources such as Medicaid. To the extent that the database does not link substance abuse treatment services provided from all funding sources, the results of this study would be biased because we may have erroneously identified new episodes or because we may have underestimated initiation or engagement rates. An examination of earlier data from 1996–1998 shows that among adult clients in Oklahoma with only substance abuse services, 97% were treated only under ODMHSAS, with 3% under Medicaid and no shared clients. For adults with both substance abuse and mental health services, 73% of clients were treated only under ODMHSAS, 21% under both ODMHSAS and Medicaid and 6% only through Medicaid (Coffey et al., 2001).

Incomplete data on criminal justice activity occurs because the outcome is based on state justice data and only includes criminal activities that result in arrest or incarceration. An alternative measure of criminal activity could be based on client self report or a merging of self-report and state justice data. Each method is subject to some bias, however; state data because all crimes do not result in arrest or incarceration and self report data because respondents may misreport their criminal activity and because of follow-up contact bias whereby those at higher risk may be less likely to be included in follow-up data collection (Rohrer, Vaughan, Cadoret, & Zwick, 1999). Another consequence of data incompleteness is omitted variable confounding. Omitted variable confounding occurs, when a variable not included in the model significantly effects outcome and also correlates with variables included in the model. In such circumstances, the correlated variable in the model receives credit that more appropriately should be attributed to the omitted variable. In our model, use of Alcoholics Anonymous (AA), Narcotics Anonymous (NA) or other community support organizations is possibly such an omitted variable. It is likely that use of community support will positively associate with prior criminal justice involvement and with criminal justice referral because of participation as a condition of probation or parole. Despite this recognition, however, an indicator variable for use of AA, NA or other community support cannot be included in our model, primarily because the AA, NA or community support organization generally keeps no records of activity or even attendance, although the criminal justice system might keep such information. Nonetheless, it is difficult to include participant’s activity in these organizations as part of an analytic model.

5. Conclusion

Adherence to performance measures can influence an outcome, such as criminal justice involvement, that is critical both to clients and society. This study offers initial evidence that for outpatient clients in the public sector, adherence to substance abuse performance measures is associated with a lower likelihood of the serious negative outcomes of arrests or incarcerations in the year after beginning a new treatment episode. Long acknowledged as a key outcome of treatment of substance use disorders, decreased criminal justice involvement is one of the ten domains in the National Outcome Measures (NOMs) developed by the Substance Abuse and Mental Health Services Administration (SAMHSA, 2006).

Our study has implications for quality improvement initiatives, payment policies, and further research. The implication of our findings for improving the quality of substance abuse treatment is that providers in the public sector should be encouraged to use performance measures that focus on the early stages of treatment because of the association we show with better outcomes, which are the ultimate goal of treatment. Performance measures based on the concept of “process of care” can make a difference because they are actionable and can be used to target quality improvement initiatives at substance abuse treatment facilities. In terms of payment, our results offer initial justification for incentive payments to providers that include measures of clients’ engagement in early stages of treatment, an approach already adopted in Delaware. In terms of research, new initiatives by additional states to calculate the same process measures explored here offer the potential for further study of the relationship between adherence to process measures and outcomes of substance abuse treatment. Expansion of the research reported here might include further analyses of types of criminal justice involvement or analyses focused on other outcomes such as housing or employment.