Abstract
This study, a collaboration between an academic research center and Washington State’s health, employment and correction departments, investigates the extent to which treatment engagement, a widely adopted performance measure, is associated with employment, an important outcome for individuals receiving treatment for substance use disorders. Two-stage Heckman probit regressions were conducted using 2008 administrative data for 7,570 adults receiving publicly-funded treatment. The first stage predicted employment in the year following the first treatment visit and three separate second stages models predicted number of quarters employed, wages, and hours worked. Engagement as a main effect was not significant for any of the employment outcomes. However, for clients with prior criminal justice involvement, engagement was associated with both employment and higher wages following treatment. Clients with criminal justice involvement face greater challenge regarding employment, so the identification of any actionable step which increases the likelihood of employment or wages is an important result.
Keywords: engagement, employment, substance use disorders, criminal justice, performance measures
Introduction
Employment has been shown to improve stability, health and social wellbeing.1 The purpose of this study is to determine whether meeting a performance measure for substance use disorders -- treatment engagement -- at the beginning of a new episode of outpatient treatment for alcohol and drug disorders is associated with employment outcomes. Such a performance measure that captures the early aspects of treatment is useful because it can track services provided and point to the need to intervene while there is still time to influence the trajectory of treatment. The economic recession that began in late 2007, and the resulting high rate of unemployment, greatly impedes finding employment for individuals being treated for substance use disorders. Moreover, people who also have recently been involved with the criminal justice system are particularly negatively impacted by economic conditions and may face even more significant challenges obtaining employment.
Many studies have focused on employment as a treatment outcome, but this is the first study to explore the association between meeting a process-based performance measure, treatment engagement, and client’s employment. This is important because a performance measure allows a treatment system or provider organization to target approaches to improve outcomes. A treatment engagement measure has been demonstrated to be feasible for use in the private sector,2 adopted by the National Committee on Quality Assurance,3 and endorsed by the National Quality Forum.4 It also has been adapted by the Veterans Health Administration and included in the Stage 1 set of Clinical Quality Measures for meaningful use initiatives in the HITECH Act.5 For use in public sector settings engagement for clients beginning a new episode of outpatient treatment is defined as receipt of another treatment service within 14 days of beginning treatment and at least two additional treatment sessions within the next 30 days.6
Employment after substance abuse treatment is an important outcome but often difficult to accomplish7 because finding employment is influenced by multiple factors outside of the control of the treatment system or the client. These factors include local or national economic stability, geographical location and job availability. In examining the employment trajectories of individuals who are treated for substance use disorders, other studies have focused on a range of client attributes that may be associated with post-treatment employment. As reviewed below, these include: prior employment experience, prior criminal justice involvement, skills or vocational training, treatment characteristics such as length of stay in treatment or completion of treatment, and demographic characteristics such as gender and race/ethnicity.
In many studies, employment prior to or at the beginning of treatment was the strongest indicator of whether clients would be employed after treatment.8–12 Employment experience prior to treatment was found to differ by type of drug abused. Clients who reported methamphetamine was their primary drug of choice had worse job histories prior to treatment, although after treatment users of methamphetamine had similar employment outcomes as clients using other hard drugs or alcohol.9
For clients who are being treated for substance use disorders and who also have criminal justice histories, employment provides an opportunity to be rewarded for productivity, develop supportive networks and engage in positive behaviors that compete with criminal activity, thus facilitating a reduction in substance use relapse and criminal justice recidivism. In addition to the monetary and psycho-social benefits associated with employment, criminally involved clients are often required to seek and maintain employment in order to meet requirements of parole and/or probation.7 However, clients involved in the criminal justice system may face significant barriers when seeking work because of the stigma and restricted available resources for job seeking associated with previous incarceration. Often, individuals exit prison with limited vocational training and education, and insufficient strategies and support networks to secure employment upon release. Offenders with stronger pre-incarceration employment histories, as well as those holding an in-prison job have higher probabilities of securing employment upon release.13
Previous research focused on arrest as the outcome14 and prior criminal justice involvement proved to be a strong independent covariate. This led us to hypothesize that prior criminal justice involvement might moderate the impact of engagement for both arrests and other outcomes such as employment. The current paper provides the opportunity to test this hypothesis. Previous research also has shown that aspects of treatment such as length of stay or treatment completion are associated with higher subsequent employment.10,15 In addition, there is evidence that substance abuse treatment paired with vocational support dramatically increased clients’ monthly earnings.12
Employment both before and after substance abuse treatment varies by clients’ race/ethnicity and gender. African Americans often enter treatment with more problematic work histories than their White or Latino counterparts, and their employment problems sometimes worsen from treatment intake to follow-up.16 Men report higher employment rates at treatment entry, while women have shown a greater increase in employment after treatment than men, though their earnings from work do not increase as much as for men.17,18
To conduct this analysis, the research team went beyond the more typical approach of analyzing administrative and client self-reported data from solely the substance abuse treatment system. Rather, analysts from the Division of Behavioral Health and Recovery in Washington State and academic partners collaborated to combine substance abuse treatment data with employment data that is reported on a quarterly basis and with data from state criminal justice agencies. Washington State offers a rich source of data that has been regularly used to study employment outcomes.9,12,19 This study extends this work by focusing specifically on outpatient treatment engagement through the following two questions:
To what extent is engagement associated with clients being formally employed (i.e., as reported to the state’s Employment Security Department) at any time during the year after beginning a new episode of outpatient treatment?
For those with any employment, to what extent is engagement associated with how much clients are employed as measured by the numbers of quarters employed, hours worked and wages earned in the formal sector?
Methods
Data
This study focuses on adult clients who received publicly-funded substance abuse treatment services in Washington in 2008, as recorded by the Division of Behavioral Health and Recovery (DBHR) of Washington State’s Department of Social and Health Services. The Washington administrative data system, TARGET, collects two categories of information: treatment admission data on client demographics (e.g., age, gender, race/ethnicity), living situation, treatment referral source, and substance use, and client encounter data, including dates and types of services received. Client substance abuse treatment data was linked with employment data obtained from Washington State’s Employment Security Department (ESD), criminal justice data obtained from the Washington Department of Corrections (information on incarcerations), and the Washington State Patrol (information on arrests). Finally, this client based dataset was merged with information on each treatment facility’s primary focus, ownership, and provision of employment ancillary services, as obtained from the 2008 National Survey of Substance Abuse Treatment Services (NSSATS), an annual census survey of substance abuse treatment facilities conducted by the Substance Abuse and Mental Health Services Administration (SAMHSA). NSSATS data from the 2007 and 2009 surveys were used for data fields that were found to be missing in the 2008 file.
Washington State’s DBHR used probabilistic linking20 to combine employment and criminal justice data with each client’s treatment data and then removed all client-identifying information from the dataset used by the study team for analyses.
Sample Selection
The study sample was made up of 8,536 adults ages 21 and over who began an outpatient substance abuse treatment episode in calendar year 2008. A new outpatient treatment episode was defined as an outpatient treatment admission (called the index) proceeded by a 60-day period without any treatment services, although detoxification could have occurred during this period. If a client had more than one outpatient treatment episode during the year, only the first episode was used in the analysis.
In forming the study sample, clients ages 18 to 20 were excluded from the study because factors with the potential to influence their post treatment employment in the formal sector might be much different than the factors affecting older adults. Also, clients were excluded who had been in a residential treatment setting (N=415) or incarcerated (N=14) for 30 days or more after the index visit, because clients who were in such controlled environments would not have been employed. Clients who died (N=7) in the year after index were also excluded. Also, 524 clients were excluded from the sample because they had missing data in at least one of the independent variables used in the analyses. Finally, six agencies with only one person remaining after the above exclusions were excluded, as it would be impossible to determine separate facility aggregate and individual client level values for engagement and employment at baseline, key variables in the analyses. After all exclusions, the final analytic sample consisted of 7,570 clients (89% of the original sample).
Employment Outcome variables
This study focused on four employment outcomes, a general dichotomous “any employment” outcome, and three additional measures, which quantified the level of employment as noted below. All four outcomes were based on Washington State’s ESD records of the client working in the state’s formal sector during the four quarters in the year after the calendar quarter in which the client’s treatment index took place. Importantly, the three levels of employment variables were only created for clients, who positively reported “any employment”. The three variables calculated quantities for: a) total number of quarters employed (range: 1 – 4); b) total number of hours worked; and c) total wages earned. For hours worked and total wages earned, values above the 99th percentile were considered outliers and “topcoded”. That is to say, the clients identified were assigned the value for the outcome equal to the 99th percentile.
Treatment Engagement
For creating the analytic variable indicating whether a client engages in outpatient treatment or not, the study employed the Washington Circle (WC) public sector specifications for initiation and engagement.6 A client who receives at least one service within 14 days after the index is considered to have initiated treatment. Treatment engagement is then defined as receiving at least two additional services within 30 days after the initiation visit. Under the WC public sector specifications, the index visit is defined as the date of the first treatment service, not the date of admission. In Washington this distinction may matter less than in other states, since a therapeutic activity usually takes place on the same date as the outpatient treatment admission. In addition, the models included a facility-level variable for the proportion of outpatient adult clients (before exclusions) admitted in 2008, who engaged in treatment. Facility engagement rates were calculated by dividing the number of clients who met the engagement criteria in 2008 by the total number of clients who had a new treatment episode in each facility. Facility engagement rates were calculated prior to the client exclusions. In order to reduce the possible effect of multi-collinearity, the individual client’s engagement variable was centered at the facility-level percent engagement. The estimated coefficient for this centered variable represents the impact of a client’s own engagement beyond the impact from the facility’s level of engagement.
Covariates
The Behavioral Model of Health Services Use21 suggests that health services utilization and outcomes are influenced by both individual and contextual factors. Covariates used in the study’s regression models consist of client and facility-level variables which previous research has found to be associated with treatment outcomes. The four categories of client-level variables included: a) client demographics, b) employment and criminal justice involvement, c) time of index visit, and d) substance use and treatment. The two categories of facility-level variables included: a) client characteristics aggregated at the facility level, and b) facility characteristics derived from the NSSATS. Unless noted, the data used to determine client-level covariate values were collected by staff at the treatment facilities at treatment admission and are based on client self-report.
Client demographics
Demographics included age, gender, race/ethnicity, education, marital status, and homelessness. In addition, an indicator for whether the client resided in an urban area was created by linking client’s zip code of residence to corresponding Rural Urban Commuting Area Codes.22
Employment and criminal justice involvement
The covariate for prior employment is based on the client’s self-reported employment status at admission. Although employment data from ESD may have been preferable, such data are only available by calendar quarter, and thus it would have been impossible to determine whether employment indicated in the quarter of the index visit occurred before or after the index visit date. Similar to client engagement, employment status was centered at the facility-level proportion of clients employed. The model also included a covariate for client’s criminal justice involvement prior to treatment based on data from Washington DOC or State Patrol. Clients were determined to have criminal justice involvement, if they had an arrest or were incarcerated at any point in the year prior to their index visits. Because we theorized that employment might be strongly affected by both engagement and prior criminal justice involvement, but the effect from the presence of both factors might differ significantly from the sum of each effect alone, the models also included the interaction between these two variables.
Time of Index visit
The models included dummy indicators for the 2008 calendar quarter in which the index visit occurred in order to account for the rise in unemployment in Washington during this year.
Substance use and treatment
Two covariates concern the client’s use of their primary, secondary, and/or tertiary substances of abuse: first, whether they used the substance in the month prior to treatment admission, and second, the earliest age of use of these substances. The one treatment-related covariate was the referral source with categories for self/family, community agency or group (e.g., child protective services), criminal justice system (e.g., department of corrections), and other (e.g., employer).
Client characteristics aggregated at the facility level
In addition to facility engagement rate, models also included the proportion of clients at the facility who reported being employed at admission. Similar to facility engagement rates, this variable was calculated prior to the client exclusions.
Facility Characteristics
The models also included facility-level covariates for primary focus (substance abuse only vs. mental health only or in combination with substance abuse), ownership (public, private non-profit, or private for-profit), and whether facility provides employment counseling or training. Because only 3 out of 173 Washington outpatient treatment facilities had a mental health focus only, these facilities were combined with those with a mixed mental health/substance abuse focus in forming the variable for facility’s primary focus.
Analysis
After inspecting the composition of the sample and the percentages of engaged clients by client characteristic, the analyses utilized multivariate analyses to determine how client and facility characteristics combined together to impact each of the four employment outcomes (any employment, and quarters employed, total hours worked, and total wages earned). For each multivariate analysis a form of Heckman selection modeling was used, which accounted for clustering within facility. The Heckman procedure23 is comprised of two steps: (1) a probit equation predicting “any employment” (i.e., employment in one of the four quarters after the beginning of a new episode of outpatient treatment), and (2) a linear regression equation associating treatment engagement with the employment outcome after adjusting for other explanatory variables. To adjust for potential selection effects from using only employed subjects in the second stage of the Heckman procedure, the specification must include an additional variable derived from the first stage probit model (the so-called inverse Mill’s ratio variable). Choice of other variables for the two stages of the Heckman procedure represents a challenge, as there is no universally accepted rule for what constitutes appropriate overlap in variables between the two stages and what is considered too much. As noted by Briggs24 “In some illustrations of the Heckman Model, it has been suggested that the covariates in the selection function should contain one or more variables related to the probability of treatment selection, but excluded from outcome prediction.25,26 In other illustrations, only covariates excluded from outcome prediction have been included in the selection function.23 In either case, it is typically assumed that the additional variables included in the selection function are strong predictors of treatment assignment, yet uncorrelated with the outcome of interest.” In any case, to guard against over-specifying the second stage of the model, it was decided to use only half the variables from the first stage and to choose variables, which were the most important and the most actionable. That is to say, variables were more likely to be included in the second stage of the model, if identification of their significance could lead to an improvement effort by the program or facility staff. Thus, the first stage probit regression included the two engagement variables together with all individual client and facility characteristics of interest, while the second stage linear regressions included all these same variables except the substance use variables, the rural/suburban place of residence variable, the index quarter indicator, and all the facility characteristics variables (though it does retain the variables of client characteristics aggregated at the facility level). Also, as required by the Heckman procedure, the inverse Mills ratio variable is included in the second stage model in order to take the selection effect of employment into account. This particular implementation of the Heckman Model also accounted for clustering within facility by using Generalized Estimating Equations in the linear regression stage of the analysis.
Results
In addition to computing distributions on all variables of interest in the sample (see Tables 1 and 2), chi-square tests were used to examine further the relationships between treatment engagement and client characteristics in the models. The results provide a profile of who becomes engaged in treatment, and confirm the appropriateness of using multivariate regression to relate treatment engagement with an outcome, when that outcome is also influenced by other client or facility characteristics.
Table 1.
Client characteristics at admission and engagement rates
| Client Characteristic | Sample Characteristic N (%) | Engagement Rate of subgroup (%) |
|---|---|---|
| Overall | 7,570 | 70.5 |
| Female | 2,841 (37.5) | 69.8 |
| Male | 4,729 (62.5) | 71.0 |
| Age | ||
| 21–25 | 1,483 (19.6) | 71.8a |
| 26–30 | 1,360 (18.0) | 73.1b |
| 31–44 | 2,867 (37.9) | 71.6c |
| 45+ | 1,860 (24.6) | 66.1a,b,c |
| Race/ethnicity | ||
| White | 4,475 (59.1) | 73.6a,b,c |
| Black | 666 (8.8) | 62.2a,d |
| Latino | 792 (10.5) | 79.8b,d,e,f |
| American Indian | 1,060 (14.0) | 57.0c,e,g |
| Other | 577 (7.6) | 69.0f,g |
| Education | ||
| No high school degree | 2,203 (29.1) | 70.2 |
| High school degree | 4,378 (57.8) | 70.4 |
| More than high school | 989 (13.1) | 71.7 |
| Homeless | 692 (9.1) | 57.7a |
| Not homeless | 6,878 (90.9) | 71.8a |
| Married | 1,857 (24.5) | 71.7 |
| Unmarried | 5,713 (75.5) | 70.2 |
| Rural/Suburban | 2,061 (27.2) | 70.6 |
| Urban | 5,509 (72.8) | 70.4 |
| Employed at baseline (self-report) | 2,129 (28.1) | 74.5a |
| Not employed at baseline | 5,441 (71.9) | 69.0a |
| Criminal justice involvement in prior year | 3,845 (50.8) | 72.2a |
| No Criminal justice involvement in prior year | 3,725 (49.2) | 68.9a |
| Quarter in 2008 when index occurred | ||
| First | 1,960 (25.9) | 72.5a |
| Second | 1,983 (26.2) | 71.0b |
| Third | 1,937 (25.6) | 71.3c |
| Fourth | 1,690 (22.3) | 66.9a,b,c |
| Substance(s) used in past month1 | ||
| Alcohol use | 2,202 (29.1) | 66.7a |
| No alcohol use | 5,368 (70.9) | 72.1a |
| Marijuana | 1,116 (14.7) | 66.2a |
| No marijuana | 6,454 (85.3) | 71.3a |
| Cocaine | 493 (6.5) | 61.5a |
| No cocaine | 7,077 (93.5) | 71.2a |
| Opiates | 441 (5.8) | 60.8a |
| No opiates | 7,129 (94.2) | 71.2a |
| Methamphetamines | 418 (5.5) | 71.8 |
| No Methampheatamines | 7,152 (94.5) | 70.5 |
| Other drug | 150 (2.0) | 66.0 |
| No other drug | 7,420 (98.0) | 70.6 |
| Age of first use2 | ||
| < 10 | 981 (13.0) | 70.6 |
| 11–14 | 2,675 (35.4) | 70.5 |
| 15–17 | 2,331 (30.8) | 70.1 |
| 18–20 | 910 (12.0) | 71.5 |
| 21+ | 673 (8.9) | 70.9 |
| Treatment referral source | ||
| Self/family | 696 (9.2) | 66.8a,b,c |
| Community agency/self-help group | 1,610 (21.3) | 72.7a,d |
| Criminal justice | 4,135 (54.6) | 73.2b,e |
| Other | 1,129 (14.9) | 59.9c,d,e |
Substance was listed as a primary, secondary, or tertiary drug and frequency of use was one or more times per month.
Earliest age of first use of any of the substances reported as primary, secondary, or tertiary substance of abuse.
Superscripts connote pairs that are significantly different at the overall p < .05 level, using a Bonferroni correction for multiple comparisons.
Table 2.
Client outcomes in year following the index quarter
| Variable | Total | Clients engaged | Clients not engaged |
|---|---|---|---|
| All clients | |||
| N | 7,570 | 5,340 | 2,230 |
| Employed in year following index quarter (%) | 43.0 | 44.7 | 38.8**,b |
| Clients employed in year following index quarter | |||
| N | 3,252 | 2,386 | 866 |
| Number of quarters employed – mean (s.d.) | 2.8 (1.2) | 2.8 (1.2) | 2.7 (1.2)*,a |
| Hours worked – mean (s.d.) | 842 (731) | 866 (738) | 776 (708)**,b |
| Wages earned ($) – mean (s.d.) | 12,217 (13,906) | 12,537 (13,979) | 11,338 (13,672)*,a |
Differences by engagement status of proportions or means significant at *p < .05;
p < .01.
Effect size levels based on Cohen’s d value:
implies effect size is < .10,
implies effect is < .20.
Client characteristics and engagement
As shown in Table 1, the clients were predominantly male (62.5%) and over age 30 (62.5%). White clients were the largest group (59.1%), although Black, Latino, and American Indian clients each represented about a tenth of the sample. Most clients had at least a high school degree (70.9%), most were not homeless (90.9%), about three quarters were not married, and nearly three quarters lived in urban areas. Most clients did not report being employed at baseline (71.9%) and about half had criminal justice involvement in the year before treatment. Clients reported a wide range of types of drugs at admission and most reported first using any substance as before age 18 (79.2%).
The rates of engagement varied significantly by some client characteristics (Table 1). Older clients aged 45 and over had a lower engagement rate (66.1%) than younger clients. Clients who were Latino had a significantly higher engagement rate (79.8%) than all other racial ethnic groups, and Blacks and American Indians had a significantly lower engagement rate (62.2% and 57.0%, respectively) than Whites (73.6%). Homeless clients were less often engaged (57.7%) than clients who were not homeless (71.8%). Individuals who reported employment at treatment admission were engaged at significantly higher rate than those who were unemployed, while clients with prior criminal justice involvement defined as an arrest or incarceration in the year before their new outpatient treatment episode were more engaged than those who were not. Clients who began their outpatient treatment episode during the fourth quarter in 2008 had slightly lower rates of engagement than those who started in an earlier quarter of the year. Distributions by engagement status of the types of substances used at admission were mixed. Clients who reported using alcohol, marijuana, cocaine or opiates in the last 30 days had lower engagement rates than clients reporting not using those specific substances, while no differences were found between clients reporting methamphetamine or “other” drugs and those who did not report using those substances. Clients referred by the criminal justice system had a significantly higher rate of engagement (73.2%) than those who entered treatment through self or family-referrals (66.8%). No significant differences by engagement status were found for clients’ gender, education, marital status, urban/rural place of residence, or age of first substance use.
Employment outcomes in the year following treatment index
As shown on Table 2, there was a significant difference of having any employment during the year following the treatment index visit between clients who were engaged and those who were not (44.7% vs. 38.8%, p<.01). On average engaged clients were employed more quarters (mean = 2.8, s.d. = 1.2) worked more hours (mean = 866, s.d. =738) and had significantly higher earnings (mean = $12,537, s.d. =$13,979) than those not engaged (mean = 2.7, s.d. = 1.2 and mean = 776, s.d. = 708, mean = $11,338, s.d. = $13,672 respectively).
Facility Characteristics
Among the 173 facilities offering outpatient treatment for substance use disorders (Table 3), sixty percent reported substance abuse as their primary focus of treatment and fifty-seven percent reported that they were nonprofit organizations. Less than half (42.2%) of the facilities offered employment counseling. There was wide variation in facility engagement rates (not shown in table). The mean facility engagement rate was 0.71 (s.d. = 0.21), with the lowest quartile ranging from 0.00 to 0.56, the second quartile ranging from 0.56 to 0.72, the third quartile ranging from 0.72 to 0.88, and the highest quartile from 0.88 to 1.00. Similarly, the proportion of clients at the facility reporting being employed at treatment admission also varied widely. The mean was 0.29 (s.d. = 0.21) and the lower two quartiles ranged from 0.00 to 0.14 and from 0.14 to 0.28, while the higher two quartiles ranged from 0.28 to 0.43 and from 0.43 to 0.90.
Table 3.
Facility characteristics (N = 173)
| Characteristic | Mean (s.d.)/% |
|---|---|
| Primary Focus | |
| Substance abuse (%) | 59.5 |
| Mental health or mix substance abuse/mental health (%) | 40.5 |
| Ownership | |
| Public (%) | 23.1 |
| For profit (%) | 20.2 |
| Nonprofit (%) | 56.7 |
| Facility provides employment counseling/training (%) | 42.2 |
Multivariate Results for engagement and employment outcomes
In the regression models, the client level variable indicating engagement was not significantly associated with having any employment in the year after treatment (Table 4) or the number of quarters, wages or hours of employment (Table 5). However, the interaction between a client’s prior criminal justice involvement and engagement was found to significantly increase the likelihood of any employment in the year after the index visit by 6% (95% CI = (0.01, 0.11), p<.05) (Table 4) and had a positive effect on wages earned post treatment (Table 5). The facility level variable indicating proportion engaged was also not associated with having any employment in the year after treatment, nor wages or hours of employment. However, proportion engaged was negatively associated with the number of quarters employed (95% CI = (−0.52, −0.02), p<.05).
Table 4.
Factors associated with any employment in year following the index quarter, N = 7,570).
| Variable | Marginal Effect | 95% Confidence Interval |
|---|---|---|
| Engagement and its interaction | ||
| Engagementa | 0.00 | (−0.04, 0.04) |
| Engagement X prior year criminal justice involvement | 0.06* | (0.01, 0.11) |
| Client characteristics aggregated at the facility level | ||
| Facility engagement rate | 0.05 | (−0.04, 0.14) |
| Proportion of clients reporting employment at baseline | 0.32** | (0.15, 0.49) |
| Facility Characteristics | ||
| Primary focus is s.a. (reference: m.h. or mix s.a. & m.h.)b | 0.08** | (0.05, 0.12) |
| Ownership (reference: public) | ||
| For profit | −0.02 | (−0.09, 0.05) |
| Nonprofit | −0.01 | (−0.08, 0.05) |
| Facility provides employment counseling/training | 0.00 | (−0.04, 0.04) |
| Client demographics | ||
| Female | −0.02 | (−0.05, 0.00) |
| Age (reference: 45+) | ||
| 21–25 | 0.23** | (0.19, 0.27) |
| 26–30 | 0.20** | (0.16, 0.24) |
| 31–44 | 0.15** | (0.11, 0.18) |
| Race/ethnicity (reference: White) | ||
| Black | 0.02 | (−0.03, 0.07) |
| Latino | −0.07 | (−0.13, 0.00) |
| American Indian | 0.05 | (−0.00, 0.10) |
| Other | −0.02 | (−0.07, 0.04) |
| Education (reference: no H.S. degree) | ||
| High school degree | 0.10** | (0.07, 0.13) |
| More than H.S. | 0.14** | (0.09, 0.18) |
| Homeless | 0.01 | (−0.04, 0.05) |
| Married | 0.01 | (−0.02, 0.04) |
| Rural/Suburban (reference: urban) | 0.02 | (−0.03, 0.06) |
| Employment and criminal justice involvement | ||
| Employed at baseline (self-report)a | 0.31** | (0.27, 0.35) |
| Prior year criminal justice involvement | −0.02 | (−0.06, 0.03) |
| Quarter in 2008 when index occurred (reference: first) | ||
| Second | 0.00 | (−0.03, 0.03) |
| Third | −0.04 | (−0.08, 0.00) |
| Fourth | −0.08** | (−0.11, −0.04) |
| Substance Use and Referral Source | ||
| Substance(s) used in past month | ||
| Alcohol | 0.01 | (−0.02, 0.03) |
| Marijuana | −0.01 | (−0.05, 0.03) |
| Cocaine | −0.01 | (−0.07, 0.05) |
| Opiates | −0.08** | (−0.13, −0.03) |
| Methamphetamines | −0.05 | (−0.10, 0.00) |
| Other Drug | −0.01 | (−0.10, 0.08) |
| Age of first use (reference: 21+) | ||
| < 10 | 0.01 | (−0.04, 0.06) |
| 11–14 | 0.06* | (0.01, 0.11) |
| 15–17 | 0.06** | (0.02, 0.11) |
| 18–20 | 0.04* | (0.00, 0.09) |
| Treatment referral source (reference: Self/family) | ||
| Community organization/agency | 0.02 | (−0.02, 0.07) |
| Criminal Justice | 0.02 | (−0.02, 0.06) |
| Other | −0.01 | (−0.07, 0.05) |
Notes:
Engagement and baseline employment status are centered around the facility mean.
s.a. = substance abuse; m.h. = mental health.
p < .05,
p < .01.
Table 5.
Factors associated with number of quarters worked, wages earned, and hours worked in year after post-index quarter, N = 3,252)
| Second Stage Dependent Variables
|
||||||
|---|---|---|---|---|---|---|
| Quarters Employed | Total Hours | Total Wages | ||||
|
| ||||||
| Coeff. | (95% CI) | Coeff. | (95% CI) | Coeff. | (95% CI) | |
| Engagement and its interaction | ||||||
| Engagementa | 0.04 | (−0.11, 0.18) | 26 | (−62, 114) | 183 | (−1481, 1846) |
| Engagement X prior year criminal justice involvement | 0.11 | (−0.07, 0.29) | 92 | (−14, 198) | 2,440* | (480, 4401) |
| Client characteristics aggregated at the facility level | ||||||
| Facility engagement rate | −0.27* | (−0.52, −0.02) | −52 | (−212, 108) | −904 | (−4399, 2592) |
| Proportion of clients reporting employment at baseline | 0.98** | (0.52, 1.44) | 765** | (451, 1078) | 15,184** | (9854, 20514) |
| Client demographics | ||||||
| Female | −0.02 | (−0.10, 0.05) | −72** | (−120, −24) | −2,565** | (−3520, −1610) |
| Age (reference: 45+) | ||||||
| 21–25 | −0.20 | (−0.44, 0.04) | −174** | (−303, −44) | −2,745** | (−4766, −724) |
| 26–30 | −0.21* | (−0.42, 0.00) | −113 | (−235, 9) | −1,271 | (−3223, 682) |
| 31–44 | −0.12 | (−0.29, 0.05) | −75 | (−172, 22) | −453 | (−2062, 1157) |
| Race/ethnicity (reference: White) | ||||||
| Black | −0.11 | (−0.27, 0.05) | −74 | (−165, 17) | −1,695* | (−3358, −32) |
| Latino | 0.14* | (0.01, 0.27) | 124** | (33, 212) | −1,069 | (−2625, 486) |
| American Indian | −0.06 | (−0.18, 0.05) | 69 | (−8, 146) | 2,338** | (741, 3934) |
| Other | −0.20** | (−0.33, −0.06) | −54 | (−142, 33) | −908 | (−2562, 746) |
| Education (reference: less than high school grad) | ||||||
| H.S. grad | 0.05 | (−0.07, 0.17) | 20 | (−44, 83) | 761 | (−402, 1924) |
| More than H.S. | 0.10 | (−0.06, 0.25) | 53 | (−48, 155) | 3,109** | (1006, 5211) |
| Homeless | −0.24** | (−0.41, −0.08) | −176** | (−260, −92) | −2,971** | (−4658, −1284) |
| Married | −0.01 | (−0.10, 0.09) | 1 | (−58, 60) | 664 | (−433, 1761) |
| Employment and criminal justice involvement | ||||||
| Employed at baselinea | 0.46** | (0.26, 0.66) | 330** | (196, 464) | 7,077** | (4737, 9417) |
| Prior year criminal justice involvement | −0.33** | (−0.47, −0.19) | −199** | (−273, −124) | −3,847** | (−5325, −2369) |
| Treatment referral source | ||||||
| Treatment referral source (reference: Self/family) | ||||||
| Community Org./Agency | −0.10 | (−0.26, 0.07) | −58 | (−163, 47) | −1,359 | (−3276, 558) |
| Criminal Justice | −0.04 | (−0.21, 0.13) | −48 | (−149, 53) | −1,365 | (−3263, 532) |
| Other | −0.13 | (−0.32, 0.06) | −117* | (−224, −9) | −1,768 | (−3972, 436) |
Notes:
Engagement and baseline employment status are centered around the facility mean.
p < .05,
p < .01
Multivariate results for other client and facility characteristics and employment outcomes
The first step of the Heckman procedure, which examined the relationship between treatment engagement and any employment after adjusting for other client and facility characteristics, showed a number of significant results (Table 4). Among facility characteristics, having substance abuse treatment as the primary focus and having higher proportions of clients reporting employment at baseline both predicted having a higher probability of clients employed in the year post index. Among client characteristics, being younger than 45 years of age, having a high school education or more, and being employed at baseline predicted a higher probability of being employed in the year after treatment index. Also, clients who began substance use in the age ranges of (11–14), (15–17) and (18–20) were more likely to be employed than those reporting first use at 21 and older. On the other hand, clients who reported opiate use in the last 30 days and clients with a treatment index occurring during the fourth quarter in 2008 were significantly less likely to be employed in the year following their index.
Table 5 shows that the facility-level variable indicating the proportion of clients reporting prior employment at baseline had a significant positive effect on the number of quarters employed, hours worked and wages earned. In addition, some client demographics had an impact on the extent of employment. Being female had a negative significant effect on both total hours worked (95% CI = (−120, −24), p<.01) and earned wages (95% CI = (−$3520, −$1610), p<.01), and clients within the age range of (21–25) had significant negative effects on hours worked and wages earned in comparison to those 45 and older, while clients within the age range of (26–30) had a significant negative effect on quarters worked in comparison to those 45 or older. Employment outcomes varied by race/ethnicity; being Black had a negative significant effect on total wages earned in comparison to Whites, but reporting Latino race/ethnicity had a positive significant effect on the number of quarters employed (95% CI = (0.01, 0.27), p<.05) and total hours worked (95% CI = (33, 212), p<.01). Being American Indian was associated with a positive effect on total wages earned, while clients categorized as “Other” were associated with a negative significant effect on number of quarters employed. Having more than a high school education was associated with higher earnings, while homelessness had a significant negative effect on the number of quarters employed, hours worked and wages earned. Prior employment had a significant positive effect on the number of quarters employed, hours worked and amount earned. In contrast, previous criminal justice involvement significantly negatively affected all three employment outcomes.
Discussion
Overall, neither the facility nor the client-level treatment engagement variable proved to affect employment. Clients with prior year criminal justice involvement were not significantly less likely to have any employment. However, among clients with prior criminal justice involvement, those who became engaged in a new episode of outpatient treatment were more likely to have any employment in the year after the quarter during which they began treatment and to have higher wages during that year. Among those employed, however, their number of quarters employed, total hours worked and total wages were all lower than clients without criminal justice involvement. These findings are particularly important because in the year prior to the start of their outpatient treatment, about half of Washington clients had some type of criminal justice involvement, including being arrested or incarcerated. Court involved clients might be expected to have higher rates of engagement in substance abuse treatment than other subgroups, in order to meet the conditions for release from incarceration.
A range of outcomes can be considered when exploring the association of process measures of treatment for substance use disorders and outcomes, including use of alcohol or drugs, housing, criminal justice involvement and employment.27 Among these outcomes, employment may be a particularly difficult one to achieve for clients treated in the public sector. As shown in the sample reported here, these clients generally do not have strong educational backgrounds with most having no more than a high school degree and most reporting not being employed at the start of treatment. Among these clients, those with criminal records face additional barriers in finding employment. Those who were incarcerated may have lost work skills, former prisoners are barred from certain occupations, and employers are often less willing to hire individuals with a criminal history.13
This study also confirmed previous findings that employment after treatment for substance use disorders is related to prior employment, gender, age, race/ethnicity, education and type of drug use. Consistent with other studies, self-reported prior employment was a predictor of post-treatment employment and wages.8–10,12,15,19 While gender was not a significant predictor of gaining any employment after beginning treatment, being female was negatively associated with hours worked, and in both this study and previous analyses using Washington data,19 negatively associated with wages earned among those who were employed after beginning treatment. Being younger than 45 years of age predicted greater probability of being employed, but being younger (ages 21 to 25) was associated with fewer hours worked and fewer wages earned among those who were employed. Although race/ethnicity was not associated with the probability of being employed, being Black was associated with lower total wages earned compared to being White, while being American Indian was associated with more wages earned compared to White. Also, being Latino was associated with more quarters and hours worked post index compared to White. Consistent with previous research,19 clients with higher education were more likely to be employed after treatment and having some high school was associated with higher total wages. In addition, clients who reported opiate use in the 30 days prior to intake were significantly less likely to engage in treatment and they were less likely to be employed in the year following treatment. Use of process measures to monitor treatment to ensure that this population achieves a minimal floor of services may be useful in providing additional resources to opiate users to help them engage in treatment and increase their chances of achieving employment. One unexpected result of the analyses was that the clients whose age of first use was between 11–20 had positive estimates, which seems to imply that these clients were more likely to be employed compared to clients whose first use was at age 21 or older. This finding, however, is undoubtedly influenced by the presence of the highly significant employed prior to treatment variable in the model. A check of correlations shows that employed prior to treatment was strongly correlated with age of first use of 21 or older, and this correlation undoubtedly impacted the apparent age of first use results.
Employment outcomes are influenced by the general economic environment. The US economy entered into a recession in December of 2007, and witnessed the deterioration of the labor market throughout 200828, the year in which all index visits occurred. In the state of Washington, the seasonally adjusted unemployment rate increased from 4.1% at the beginning of 2008 to 9.4% at the end of 2009.29 Consistent with this, the results showed that clients entering treatment in the fourth quarter of 2008 were less likely to gain employment than clients entering in earlier quarters. This confirms the conjecture that the general economic environment has a strong influence on employment among recipients of treatment for substance use disorders.
Facility type was also found to be associated with employment status. Clients treated at facilities with substance abuse treatment as the primary focus, compared to those treated at facilities with a mental health or MH/SA mixed focus, were more likely to be employed. Also, the higher percentage of clients in a facility who report being employed at baseline, the more likely the client is to be employed after treatment. It is possible that being in a facility with a greater proportion of clients employed at baseline helps other clients find jobs through networking, but it is also possible that this factor merely reflects greater employment opportunities in these facilities’ surrounding vicinity. Less understandable, however, is that being treated at a facility with a higher percentage of engaged clients was associated with fewer quarters worked. One possible explanation of this anomaly is that more clients from such facilities, particularly more clients with criminal justice backgrounds, were becoming employed in the year after treatment, but they were not as likely to become employed quickly or obtain steady employment, thus they had fewer quarters in which they worked. Another possible explanation is that facilities with a higher percentage of engage clients might have them participating in in treatment for longer periods of time, thus delaying employment.
Limitations
Despite the rich opportunity for research offered by linking information across three departments of a state as they worked to deal with substance abuse treatment, employment, and criminal justice, there are limitations to this study. First, the study could only include information on employment in the formal sector of the state’s economy, the sector which offers regular wages and hours, which carries employment rights, and on which income tax is paid. These results do not reflect the extent to which clients may find employment in the informal sector, often referred to colloquially as working or being paid “under the table” or the “underground economy.” In fact, individuals with a history of substance use disorders or criminal justice involvement may turn to the informal sector because their options for finding work in the formal sector are worse than what they can find in the informal sector.13,30 This study did not include information on arrests/incarcerations occurring outside of the state of Washington, thus limiting the ability to fully assess prior criminal justice involvement on employment. Second, like other non-randomized, observational studies, there may be unobserved or uncollected variables, which if included in the models would influence both engagement and outcomes (e.g., clients with stronger motivation at entrance to treatment may have both higher engagement and better outcomes). Consistent with the Behavioral Model of Health Services Use21 which posits that a rich array of individual and contextual characteristics may influence health services use, the study included a substantial collection of clients’ clinical and behavioral characteristics in the regression models, so the potential effect of unobserved variables on outcomes is probably lessened. Third, the study focused on only one state, so the results may not be broadly generalizable. Finally, it is important to note that most of the significant effects were small (with all Cohen’s d less than .20), so caution should be taken in considering the implications of this research. However, this study does contribute to the growing literature on the association between the process measure, engagement in outpatient treatment and outcomes. However, this study does contribute to the growing literature on the association between the process measure, engagement in outpatient treatment and outcomes.
Implications for Behavioral Health
The implications of the findings of this research fall into three areas: the importance of both employment support services and engaging clients with criminal justice involvement in outpatient substance abuse treatment, utilization of performance measures for monitoring quality of service provision, and collaboration across organizations to conduct research.
For substance abuse treatment providers, assisting clients with prior criminal justice involvement obtain employment may not be their primary role. However, there is a current focus on patient-centered care and treatment of the whole person that would behoove providers to focus their attention on their clients’ broad range of needs. A large segment of the population receiving substance abuse treatment in the public sector enters with involvement with the criminal justice system. Individuals reentering the community after incarceration often do so with limited supportive community resources and do not receive coordinated service delivery. For individuals with prior criminal justice involvement finding and maintaining employment is an important factor in reducing recidivism, however lack of job skills and assistance is a hindrance to employment for this population. Getting employment for this population requires a multi-faceted approach, including job training, assistance with job search and support with other basic needs. To the extent that these opportunities for support are offered by or coordinated through the substance abuse treatment system has far reaching implications for client recovery. In addition to employment-focused services, the findings reported in this paper indicate that helping this population engage in treatment also may influence their becoming employed after treatment because outpatient substance abuse treatment engagement was shown to be related to employment outcomes for clients with prior criminal justice involvement.
Moreover, it is possible, through the use of the outpatient treatment engagement measure, for treatment providers to monitor whether clients are on track to becoming engaged in outpatient treatment and to intervene early to take steps to help them to engage in treatment. This study adds to the evidence supporting the association between a process-focused measure of performance for the treatment of substance use disorders and outcomes.14,31,32
Finally, through collaboration across sectors and the merging of data from multiple agencies such as across substance abuse treatment, employment, and criminal justice agencies, this research has shown how performance measures in one sector can be associated with outcomes in another. One critical next step is for policy makers and treatment providers both in Washington State and other states to consider how to improve on engaging clients in outpatient care through providing feedback to providers, internal quality improvement efforts (e.g., implementing financial incentives), and continuing efforts to collaborate across agencies.33–35
Acknowledgments
This research was supported by the National Institute on Alcohol Abuse and Alcoholism grant # R01AA017177-01A2.
Footnotes
Conflict of Interest Statement
None of the authors has any conflict of interest related to this work. The views expressed herein are not necessarily those of the Washington State Department of Social and Health Services.
References
- 1.Kessler RC, House JS, Turner JB. Unemployment and health in a community sample. J Health Soc Behav. 1987 Mar;28(1):51–59. [PubMed] [Google Scholar]
- 2.Garnick DW, Lee MT, Chalk M, et al. Establishing the feasibility of performance measures for alcohol and other drugs. J Subst Abuse Treat. 2002 Dec;23(4):375–385. doi: 10.1016/s0740-5472(02)00303-3. [DOI] [PubMed] [Google Scholar]
- 3.National Committee for Quality Assurance. The state of health care quality 2007. Washington, D.C: National Committee for Quality Assurance; 2007. [Google Scholar]
- 4.National Quality Forum. [Accessed August 17, 2010];NQF-Endorsed Standards: NQF # 0004, Initiation and Engagement of Alcohol and Other Drug Dependence Treatment: a. Initiation, b. Engagement. 2010 http://www.qualityforum.org/Measures_List.aspx?keyword=substance+use&from=header#k=initiation&e=1&st=&sd=&s=&p=1.
- 5.Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. NEJ M. 2010;363(6):501–504. doi: 10.1056/NEJMp1006114. [DOI] [PubMed] [Google Scholar]
- 6.Garnick DW, Lee MT, Horgan CM, Acevedo A Washington Circle Public Sector W. Adapting Washington Circle performance measures for public sector substance abuse treatment systems. Journal of Substance Abuse Treatment. 2009 Apr;36(3):265–277. doi: 10.1016/j.jsat.2008.06.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Magura S, Staines GL, Blankertz L, Madison EM. The effectiveness of vocational services for substance users in treatment. Substance Use & Misuse. 2004;39(13–14):2165–2213. doi: 10.1081/ja-200034589. [DOI] [PubMed] [Google Scholar]
- 8.Zanis DA, Coviello D, Alterman AI, Appling SE. A community-based trial of vocational problem-solving to increase employment among methadone patients. Journal of Substance Abuse Treatment. 2001 Jul;21(1):19–26. doi: 10.1016/s0740-5472(01)00177-5. [DOI] [PubMed] [Google Scholar]
- 9.Luchansky B, Krupski A, Stark K. Treatment response by primary drug of abuse: Does methamphetamine make a difference? Journal of Substance Abuse Treatment. 2007 Jan;32(1):89–96. doi: 10.1016/j.jsat.2006.06.007. [DOI] [PubMed] [Google Scholar]
- 10.Arria AM TOPPS-II Interstate Cooperative Study Group. Drug treatment completion and post-discharge employment in the TOPPS-II Interstate Cooperative Study. Journal of Substance Abuse Treatment. 2003 Jul;25(1):9–18. doi: 10.1016/s0740-5472(03)00050-3. [DOI] [PubMed] [Google Scholar]
- 11.Veach LJ, Remley TP, Jr, Kippers SM, Sorg JD. Retention predictors related to intensive outpatient programs for substance use disorders. Am J Drug Alcohol Ab. 2000 Aug;26(3):417–428. doi: 10.1081/ada-100100253. [DOI] [PubMed] [Google Scholar]
- 12.Luchansky B, Brown M, Longhi D, Stark K, Krupski A. Chemical dependency treatment and employment outcomes: results from the ‘ADATSA’ program in Washington State. Drug and Alcohol Dependence. 2000 Aug 1;60(2):151–159. doi: 10.1016/s0376-8716(99)00153-2. [DOI] [PubMed] [Google Scholar]
- 13.Visher CA, Debus S, Yahner J. Employment after Prison: A Longitudinal Study of Releasees in Three States. Washington, DC: Urban Institute Justice Policy Center; 2008. http://www.urban.org/UploadedPDF/411778_employment_after_prison.pdf. [Google Scholar]
- 14.Garnick DW, Horgan CM, Lee MT, et al. Are Washington Circle performance measures associated with decreased criminal activity following treatment? Journal of Substance Abuse Treatment. 2007 Dec;33(4):341–352. doi: 10.1016/j.jsat.2007.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Zarkin GA, Dunlap LJ, Bray JW, Wechsberg WM. The effect of treatment completion and length of stay on employment and crime in outpatient drug-free treatment. Journal of Substance Abuse Treatment. 2002 Dec;23(4):261–271. doi: 10.1016/s0740-5472(02)00273-8. [DOI] [PubMed] [Google Scholar]
- 16.Fosados R, Evans E, Hser YI. Ethnic differences in utilization of drug treatment services and outcomes among proposition 36 offenders in California. Journal of Substance Abuse Treatment. 2007 Dec;33(4):391–399. doi: 10.1016/j.jsat.2007.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Grella CE, Scott CK, Foss MA. Gender differences in long-term drug treatment outcomes in Chicago PETS. Journal of Substance Abuse Treatment. 2005;28:S3–S12. doi: 10.1016/j.jsat.2004.08.008. [DOI] [PubMed] [Google Scholar]
- 18.Oggins J, Guydish J, Delucchi K. Gender differences in income after substance abuse treatment. Journal of Substance Abuse Treatment. 2001 Apr;20(3):215–224. doi: 10.1016/s0740-5472(01)00163-5. [DOI] [PubMed] [Google Scholar]
- 19.Wickizer TM, Campbell K, Krupski A, Stark K. Employment outcomes among AFDC recipients treated for substance abuse in Washington state. Milbank Q. 2000;78(4):585–608. doi: 10.1111/1468-0009.00186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Campbell KM. Impact of record-linkage methodology on performance indicators and multivariate relationships. Journal of Substance Abuse Treatment. 2009 Jan;36(1):110–117. doi: 10.1016/j.jsat.2008.05.004. [DOI] [PubMed] [Google Scholar]
- 21.Andersen RM, Davidson PL. Improving Access to Care in America: Individual and Contextual Indicators. In: Andersen RM, Rice TH, Kominski GF, editors. Changing the US Health Care System: Key Issues in Health Services Policy and Management. 3. San Francisco: Jossey-Bass Inc; 2007. [Google Scholar]
- 22.WWAMI Rural Health Research Center. [Accessed December 1, 2011];RUCA data. 2007 http://depts.washington.edu/uwruca/ruca-download.php.
- 23.StataCorp. Heckman two-stage procedure. Stata Statistical Software: Release 11. College Station, TX: StataCorp LP; 2009. [Google Scholar]
- 24.Briggs DC. Casual Inference and the Heckman Model. Journal of Education and Behavioral Statistics. 2004;29(4):397–420. [Google Scholar]
- 25.Lalonde RJ. Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review. 1986;76(3):604–620. [Google Scholar]
- 26.Greene WH. Econometric Analysis. New York: Macmillan Publishing Company; 1993. [Google Scholar]
- 27.McLellan AT, Cacciola JC, Alterman AI, Rikoon SH, Carise D. The Addiction Severity Index at 25: origins, contributions and transitions. Am J Addiction. 2006 Mar-Apr;15(2):113–124. doi: 10.1080/10550490500528316. [DOI] [PubMed] [Google Scholar]
- 28.National Bureau of Economic Research. Determination of the December 2007 Peak in Economic Activity. 2008 http://www.nber.org/dec2008.pdf.
- 29.Washington State Employment Security Department. Resident Civilian Labor Force and Employment in Washington State Seasonally Adjusted. 2012 https://fortress.wa.gov/esd/employmentdata/home.
- 30.McLain RJ, Alexander SJ, Jones ET. Gen Tech Rep PNW-GTR-755. Portland, OR: U.S. Department of Agriculture, Forest Service, Pacific Northwest Research Station; 2008. Incorporating Understanding of Informal Economic Activity in Natural Resource and Economic Development Policy. [Google Scholar]
- 31.Garnick DW, Lee MT, O’Brien PL, et al. Exploring the Association of the Washington Circle Engagement Performance Measure and Adolescents’ Outcomes. Drug and Alcohol Dependence. doi: 10.1016/j.drugalcdep.2012.01.011. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Harris AH, Humphreys K, Bowe T, Tiet Q, Finney JW. Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes? J Behav Health Serv Res. 2010 Jan;37(1):25–39. doi: 10.1007/s11414-008-9142-2. [DOI] [PubMed] [Google Scholar]
- 33.Horgan C, Garnick DW. The quality of care for adults with mental and addictive disorders: Issues in performance measurement. [Accessed January 3, 2012];Background paper for the Institute of Medicine. 2005 http://sihp.brandeis.edu/ibh/pdfs/IOMJan05submitted.pdf.
- 34.Evans E, Grella CE, Murphy DA, Hser YI. Using administrative data for longitudinal substance abuse research. Journal of Behavioral Health Services and Research. 2010 Apr;37(2):252–271. doi: 10.1007/s11414-008-9125-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Hser YI, Evans E. Cross-system data linkage for treatment outcome evaluation: lessons learned from the California Treatment Outcome Project. Eval Program Plann. 2008 May;31(2):125–135. doi: 10.1016/j.evalprogplan.2008.02.003. [DOI] [PMC free article] [PubMed] [Google Scholar]