Evidence from regression‐discontinuity analyses for beneficial effects of a criterion‐based increase in alcohol treatment

Abstract

Brief interventions effectively reduce alcohol problems; however, it is controversial whether longer interventions result in greater improvement. This study aims to determine whether an increase in treatment for people with more severe problems resulted in better outcome. We employed regression‐discontinuity analyses to determine if drinking driver clients (n = 22,277) in Ontario benefited when they were assigned to a longer treatment program (8‐hour versus 16‐hour) based on assessed addiction severity criteria. Assignment to the longer16‐hour program was based on two addiction severity measures derived from the Research Institute on Addictions Self‐inventory (RIASI) (meeting criteria for assignment based on either the total RIASI score or the score on the recidivism subscale). The main outcome measure was self‐reported number of days of alcohol use during the 90 days preceding the six month follow‐up interview. We found significant reductions of one or two self‐reported drinking days at the point of assignment, depending on the severity criterion used. These data suggest that more intensive treatment for alcohol problems may improve results for individuals with more severe problems. Copyright © 2012 John Wiley & Sons, Ltd.

Introduction

Current evidence demonstrates the value of alcohol treatment in reducing consumption and improving health and social outcomes (Anton et al., 2006; Babor et al., 2010; Drummond et al., 2009; Mann et al., 1994; Project Match Research Group, 1997; Smart and Mann, 2000). However, the value of longer versus shorter interventions has not been established, and studies demonstrating that briefer interventions are as effective as longer interventions continue to be reported. For example, in a recent meta‐analysis of the effects of brief interventions for alcohol problems in primary care settings, Kaner et al. (2009) found that although these interventions reduced alcohol consumption over follow‐up intervals, longer interventions had little additional effect over brief interventions.

A related issue, in dealing with alcohol use disorders as well as other problems, is the challenge of how to respond to individuals with different levels of problem severity and needs (e.g. Drummond et al., 2009; Fonagy, 2010; Watzke et al., 2010). For example, it is widely believed that individuals with more serious problems are most suited to interventions that are longer or more intensive, and many guidelines for dealing with individuals with alcohol problems are based on this assumption (e.g. Health Canada, 2004; Mee‐Lee et al., 2001). Nevertheless, there is little direct research support for severity‐based assessment‐assignment schemes that involve assignment of individuals with more severe problems to longer or more intensive treatment programs. Evidence cited in favor of severity‐based assignment schemes has often come from between‐study rather than within‐study comparisons (e.g. Mann et al., 1988) or naturalistic studies (e.g. Simpson et al., 1999).

A central problem in obtaining the evidence needed to validate or refute severity‐based assignment schemes is the study design needed to obtain evidence. Randomized trials have become the gold standard in treatment outcome evaluations, and evaluation of the effects of a severity‐based assignment scheme might involve random assignment of individuals with a range of problem severity to treatment interventions of varying intensity or duration. This type of study is a variant of the treatment matching design, with severity as the matching variable. Research efforts to evaluate specific matching hypotheses have not yet evaluated severity‐based assignment schemes, but in general it has proven difficult to find strong evidence supporting treatment matching based on randomized trials.

Nevertheless, severity‐based assignment schemes are common and intuitively appealing. One area where treatment of alcohol problems has, in recent years, been introduced on a systematic basis is in the provision of remedial programs for convicted drinking drivers. In many or most jurisdictions in the developed world individuals who are caught or convicted for a drinking‐driving offence are encouraged or required to attend remedial programs where their levels of alcohol problems are assessed and treatment services are provided (Mann et al., 1988; Dill and Wells‐Parker, 2006; Nickel, 1990). Substantial evidence demonstrates that these programs have beneficial effects on measures of substance use, health and traffic safety (e.g. Dill and Wells‐Parker, 2006; Macdonald et al., 2004; Mann et al., 1988, 1994; Wells‐Parker et al., 1995, 2009). Severity‐based assignment schemes are commonly seen as an appropriate basis for these programs (e.g. Health Canada, 2004; Voas et al., 2011). However, as Wells‐Parker et al. (1995) noted, widespread acceptance of the underlying assumptions of severity‐based assignment has prevented evaluation of the validity of these assumptions.

The regression‐discontinuity (RD) design, a quasi‐experimental research design first introduced by Thistlewaite and Campbell (1960), seems particularly well‐suited for the evaluation of severity‐based assignment schemes. In the RD design the assignment rule to treatment conditions or control groups is previously known and followed consistently. A pre‐treatment measure, such as alcohol problem severity, is used to assign individuals to two or more groups, e.g. control and treatment groups, or shorter and longer intervention groups. This pre‐treatment measure has to be chosen so that it is linked to outcome via linear or other continuous relationship as measured by regression. If there is an effect for the different interventions, then the regression line is interrupted or discontinued at the threshold used to allocate individuals into different treatment conditions, and this effect can be statistically ascertained against chance.

Of particular interest is that the RD design, when properly implemented, is considered to provide an unbiased estimate of the intervention effect and the ability to draw causal inference around the point of discontinuity is similar to that of a randomized trial (Campbell and Stanley, 1967; Imbens and Lemieux, 2008; Murray et al., 2010; Shadish and Cook, 2009; Trochim, 1990). Linden et al. (2006) identified five conditions when the RD design could profitably be used: (1) strict observance to the cutoff point for assignment, (2) correct identification of the functional relationship between the assignment variable and the outcome variable, (3) large sample size, (4) all individuals in the study must be drawn from the same population, and (5) all participants receiving “treatment” receive the same level and amount of intervention. Although the RD design has been recommended for alcohol and drug research (Hester and Miller, 1988), it has only rarely been employed. In an early study, Daniels et al. (1992) employed an RD design to assess the impact on follow‐up risk drinking of severity‐based assignment of computerized advice to reduce consumption and provision of a self‐help manual in a group of 547 general hospital patients. Although significant reduction in risk drinking was observed, no additional benefit of the computerized advice and self‐help manual was found. RD designs have been used more recently in studies employing aggregate level data. Carpenter and Dobkin (2009) applied RD analysis to examine the impact of the minimum legal drinking age of 21 in the United States on alcohol consumption, arrests and mortality rates among young people. They found that reaching the legal drinking age results in significant increases in the proportion reporting alcohol consumption in the previous month, a 6% increase in arrests, and a 9% increase in mortality rate. Hasin and Beseler (2009) employed RD methods to assess evidence for dimensional versus categorical nature of alcohol problems in data from the 2001–2002 National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) sample. Their results supported a dimensionality perspective, with no evidence of discontinuities that would reflect a categorical perspective.

Beginning in 2000, all convicted drinking drivers in Ontario seeking relicensing were required to complete all components of the Back on Track (BOT) remedial measures program. The BOT program is based on brief interventions for alcohol problems (Annis et al., 1996; Sobell and Sobell, 1993) and includes completion of an assessment, an 8‐hour or a 16‐hour program, and a six‐month follow‐up interview. Assignment to the shorter or longer program is based on assessment of alcohol problem severity. The BOT program provides an ideal context for the evaluation of a severity‐based assignment scheme with the RD design. In this paper, we describe RD analyses of the main outcome measure, alcohol use, which was the major focus of program content, and for control purposes, tobacco use, which was not addressed in program content. Thus two hypotheses are tested: (1) assignment to the longer 16‐hour intervention will be associated with measurable improvement at the point of discontinuity in the self‐reported number of days of alcohol use at six‐month follow‐up; (2) assignment to the longer 16‐hour intervention will not be associated with measurable improvement at the point of discontinuity in the self‐reported number of days of tobacco use at six‐month follow‐up.

Method

Design and setting

Since 2000, convicted drinking drivers must complete the BOT mandatory provincial remedial program in Ontario, Canada, if they seek relicensing following a period of license suspension. All BOT participants are required to complete an assessment, a treatment program and a follow‐up interview. The Centre for Addiction and Mental Health (CAMH) manages and operates the BOT program on behalf of the government of Ontario. Service delivery is subcontracted to local providers in about 30 locations across the province, while CAMH delivers the program in Toronto. The BOT program is designed to meet stringent requirements in terms of geographic accessibility, consistency in assessment and assignment to interventions, and consistency in service delivery at different locations (Shuggi et al., 2002).

The assessment includes the Research Institute on Addictions Self‐inventory (RIASI; Mann et al., 2009; Nochajski et al., 1995), an addictions screening instrument that was designed specifically for use with convicted drinking drivers. The RIASI is a 52‐item instrument that measures distal (hostility/aggression, sensation seeking, depression, anxiety, interpersonal competence, childhood risk factors, social problems such as criminal history, health issues) and proximal factors (current drinking habits, pre‐occupation with alcohol, alcohol beliefs, use of alcohol to alleviate problems, and family history) associated with alcohol or drug problems. An overall score (RIASI‐T) is based on the full instrument, and the recidivism score (RIASI‐R) is based on a subset of 15 items found by Nochajski et al. (1997) to be the strongest predictors of drinking driving recidivism. In developing the RIASI, Nochajski and colleagues reported Chronbach's alpha coefficients ranging between 0.81 and 0.89 across different groups of drinking driving offenders (Nochajski et al., 1995). The recidivism subscale was found to correctly identify 80% of recidivists over a two‐year period (Nochajski et al., 1997; Nochajski, 2002). Good agreement of RIASI scores with other measures of alcohol and drug problems, including the Alcohol Dependence Scale (Skinner and Horn, 1984), the Drug Abuse Screening Test (Skinner, 1982) and the Adverse Consequences of Substance Use Scale (Mann et al., 2006) have been reported in both New York and Ontario samples (Nochajski et al., 1995; Nochajski, 2002; Shuggi et al., 2006; Mann et al., 2006).

In the assessment, clients provide information on number of days of alcohol and other drug use in the 90 days preceding the assessment. Additionally, information is collected on client demographics. Based on the assessment scores individuals are assigned to complete a treatment program (8‐hour or 16‐hour).

Six months following completion of their assigned treatment program, clients are required to complete a follow‐up interview where they provide information on alcohol and other drug use and problems in the 90 days preceding the interview. The proportion of clients who complete the six‐month follow‐up once they have begun the assessment is 97% (Rootman et al., 2005).

An RD design was implemented, with allocation to either the 8‐hour or the 16‐hour intervention based on cutoff scores shown to be indicators of alcohol problems and predictive of drinking driving recidivism (Nochajski et al., 1995, 1997). Those individuals with RIASI‐T scores of nine or more or RIASI‐R scores of six or more received the 16‐hour treatment intervention, and all others the 8‐hour intervention. Assessments were conducted by trained clinicians using encrypted web‐based technology. Program assignments were made by computer, based on the results of the assessment as described earlier.

Participants

The data were drawn from administrative records of BOT. The analyses included data from 22,277 individuals who completed their assessments beginning November 1, 1999 and had completed the follow‐up interview by April 30, 2005. The majority of clients (96%) completed the follow‐up within seven months of participating in their assigned program.

The sample was predominantly male (88%); average age was 44 years and approximately 75% were younger than 50 years of age. They had achieved an average of 12 years of schooling, and reported a mean income between $20,000 and $49,999. Most reported being married (44%) or single (36%), and 72.1% reported being currently employed. The mean scores on the RIASI‐T and RIASI‐R were 6.8 and 3.3 respectively, and 25.8% reported that they had had an additional drinking driving offence, in addition to the present one, at some time in the past. Approval for the study protocol was obtained through the CAMH Research Ethics Board. Confidentiality of data was maintained at all times.

Intervention

Assignment to the 8‐hour or 16‐hour program is based on a threshold score being reached on either the RIASI‐T (> 8) or the RIASI‐R (> 5), both scores which were considered indicative of higher levels of alcohol problems and increased risk of drinking driving recidivism. The 8‐hour program is provided in groups of 10 to 25 clients, and the 16‐hour program is provided in groups of 6 to 15 clients. The programs aim to help participants: learn about the effect of alcohol and other drugs on driving performance; learn about the health and behavioral effects of alcohol intake; understand the consequences of impaired driving; assess their own attitudes, beliefs and behaviors in relation to impaired driving; and enhance coping skills to enable them to avoid situations that involve alcohol, other drugs and driving, develop plans to avoid another impaired driving offence, and to deal with potential triggers to drinking and drinking‐driving. The 16‐hour program has an increased emphasis on coping skills in relation to educational activities.

Main outcome measures

Six months following completion of their assigned treatment program, clients are required to complete a follow‐up interview where they provide information on alcohol, tobacco and other drug use and problems in the 90 days preceding the interview. The self reported number of days using alcohol in the 90 days preceding the six‐month follow‐up served as the primary outcome measure. Assignment to the longer 16‐hour intervention is hypothesized to be associated with measurable improvement at the point of discontinuity. Self‐reported days of alcohol use, which was the major focus of the BOT program content. Self‐reported tobacco use in the 90 days preceding the six‐month follow‐up is the control outcome measure. Use of tobacco is not expected to be associated with assignment to the longer 16‐hour program since it is not addressed in BOT program content.

Self‐report measures of substance use are the most common outcome measures in addiction treatment research, and research indicates that these reports have good reliability and validity in these contexts (Del Boca and Darkes, 2003; Del Boca and Noll, 2000; Mann et al., 1983). Comparisons of self‐report measures with biological or observational measures of substance use show good concordance between them, with the largest differences typically occurring in the heaviest users (e.g. Colón et al., 2010; Northcote and Livingston, 2011). While under‐reporting has been observed (Johnson et al., 2009; Lapham et al., 2002), Frone (2006) notes that there are likely no better methods to measure substance use.

Analyses

Data were analyzed using the SAS statistical software package. The approach to RD analysis taken here follows the recommendations of Trochim (1984, 1990). For our analyses, we consider clients assigned to the 16‐hour program as constituting the “treatment” condition, and clients assigned to the 8‐hour program as constituting the “control” condition. Self‐reported alcohol and tobacco use of clients in both programs declined substantially between assessment and follow‐up (see Table 1). The RD analyses, then, will serve to determine if there is any additional value conferred by assignment to the longer, more intensive (reflecting smaller group size) treatment program. Separate analyses were conducted for assignment to treatment condition based on the RIASI‐T score and assignment based on the RIASI‐R score.

Table 1.

Self‐reported number of days of alcohol and tobacco use at assessment and follow‐up (past 90 days) for assignment based on the RIASI‐T and RIASI‐R

Substance	Program groups	Assessment Mean (SD)	Follow‐up Mean (SD)	P Value
Assignment based on RIASI‐T
Alcohol	8‐hour	12.12 (16.10)	9.61 (13.57)	<0.001
	16‐hour	16.07 (19.08)	10.26 (14.38)	<0.001
Tobacco	8‐hour	46.95 (43.83)	43.27 (44.00)	<0.001
	16‐hour	62.30 (40.24)	57.62 (42.09)	<0.001
Assignment based on RIASI‐R
Alcohol	8‐hour	12.09 (16.11)	9.60 (13.57)	<0.001
	16‐hour	17.16 (19.22)	10.44 (14.29)	<0.001
Tobacco	8‐hour	46.67 (43.85)	42.99 (44.00)	<0.001
	16‐hour	68.70 (36.77)	63.11 (40.03)	<0.001

In conducting the analysis, Trochim (1990) recommends starting with an over‐specified model with higher order polynomial terms than the true model. Quadratic or higher polynomial terms were included in preliminary models to test for a curvilinear relationship between the outcome measure and pre‐treatment measure (RIASI‐T or RIASI‐R), however, these terms were not significant and therefore were not included in the final models.

In preliminary analyses we also included the variables age, gender, education, and self‐reported number of days of use of the substance being considered (alcohol and tobacco) at the assessment interview. Age, gender, and education were not significant contributors to the model and thus were not retained in the final models.

RD models were built having the self‐reported number of days of alcohol use as the main outcome measure (Table 2). Separate models were built for each of the instruments, RIASI‐T and RIASI‐R. The overall score (RIASI‐T) is calculated based on the full instrument (52 items), and the recidivism score (RIASI‐R) is calculated based on a subset of 15 items for each of the 22,277 individuals. On the model built for the RIASI‐T instrument, the 16‐hour program group (or treatment group) consisted of 5631 clients who were assigned to the 16‐hour program based on the RIASI‐T instrument only, that is, individuals with RIASI‐T scores of nine or more.

Table 2.

Effects of program assignment, RIASI‐T and RIASI‐R scores and self‐reported number of drinking days at assessment on self‐reported number of drinking days at follow‐up

Variables	Regression coefficient	Standard error	P Value
Assignment based on RIASI‐T (Equation (1) )
Constant	10.50	0.21	<0.001
Assignment effect	–1.10	0.30	0.001
RIASI‐T score	0.09	0.04	0.04
Assignment effect × RIASI‐T score	–0.18	0.05	0.003
Days drinking at assessment	0.50	0.01	<0.001
Assignment effect × Days drinking at assessment	–0.09	0.01	<0.001
Assignment based on RIASI‐R (Equation (2) )
Constant	10.87	0.26	<0.001
Assignment effect	–1.96	0.37	<0.001
RIASI‐R score	0.21	0.07	0.003
Days drinking at assessment	0.50	0.01	<0.001
Assignment effect × Days drinking at assessment	–0.13	0.02	<0.001
Assignment effect × RIASI‐R score × Days drinking at assessment	–0.35	0.18	<0.001

The final RD model for the RIASI‐T is as follows:

$$\mathit{Z}={\mathrm{c}}_0+{\mathrm{c}}_1\mathit{A}+{\mathrm{c}}_2\mathit{X}+{\mathrm{c}}_3\mathit{A}*\mathit{X}+{\mathrm{c}}_4\mathit{Y}+{\mathrm{c}}_5\mathit{A}*\mathit{Y}+{\mathrm{e}}_{\mathrm{i}}\text{,}$$ (1)

where A denotes the assignment (treatment) variable (1 = 16‐hour program, 0 = 8‐hour program) based on the RIASI‐T, X indicates the instrument or pre‐treatment measure (scores on the RIASI‐T), Z denotes the self‐reported number of days of alcohol at follow‐up), and Y indicates the self‐reported number of days of alcohol at assessment.

On the models built for the RIASI‐R the 16‐hour program group (treatment group) consisted of 3116 clients who were assigned to the 16‐hour program based on the RIASI‐R instrument only, that is, individuals with RIASI‐R scores of six or more.

The final model for the RIASI‐R is as follows:

$$\mathit{Z}={\mathrm{c}}_0+{\mathrm{c}}_1\mathit{A}+{\mathrm{c}}_2\mathit{X}+{\mathrm{c}}_3\mathit{Y}+{\mathrm{c}}_4\mathit{A}*\mathit{Y}+{\mathrm{c}}_5\mathit{A}*\mathit{X}*\mathit{Y}+{\mathrm{e}}_{\mathrm{i}}\text{,}$$ (2)

where A denotes the assignment (treatment) variable (1 = 16‐hour program, 0 = 8‐hour program) based on the RIASI‐R, X indicates the instrument or pre‐treatment measure (scores on the RIASI‐R), Z denotes the outcome variable (self‐reported number of days of alcohol at the follow‐up), and Y indicates the self‐reported number of days of alcohol use at the assessment.

Similarly, Table 3 show results for the RD models having the control outcome measure as the self‐reported days of tobacco use.

Table 3.

Effects of program assignment, RIASI‐T and RIASI‐R scores and self‐reported number of days using tobacco at assessment on self‐reported number of days using tobacco at follow‐up

Variables	Regression coefficient	Standard error	P Value
Assignment based on RIASI‐T (Equation (1A))
Constant	48.62	0.54	<0.001
RIASI‐T score	0.54	0.11	<0.001
Assignment effect × RIASI‐T score	–0.48	0.14	<0.001
Days using tobacco at assessment	0.77	0.01	<0.001
Assignment effect × Days using tobacco at assessment	–0.02	0.01	0.05
Assignment based on RIASI‐R (Equation (2A) )
Constant	50.94	0.69	<0.001
RIASI‐R score	1.31	0.18	<0.001
Assignment effect × RIASI‐R score	–1.35	0.44	0.004
Days using tobacco at assessment	0.75	0.01	0.001

The final model for the RIASI‐T is as follows:

$$\mathit{Z}={\mathrm{c}}_0+{\mathrm{c}}_1\mathit{X}+{\mathrm{c}}_2\mathit{A}*\mathit{X}+{\mathrm{c}}_3\mathit{Y}+{\mathrm{c}}_4\mathit{A}*\mathit{Y}+{\mathrm{e}}_{\mathrm{i}}\text{,}$$ (1A)

where A denotes the assignment (treatment) variable (1 = 16‐hour program, 0 = 8‐hour program) based on the RIASI‐T, X indicates the instrument or pre‐treatment measure (scores on the RIASI‐T), Z denotes the outcome variable (self‐reported number of days of tobacco at the follow‐up), and Y indicates the self‐reported number of days of tobacco use at the assessment.

The final model for the RIASI‐R is as follows:

$$\mathit{Z}={\mathrm{c}}_0+{\mathrm{c}}_1\mathit{X}+{\mathrm{c}}_2\mathit{A}*\mathit{X}+{\mathrm{c}}_3\mathit{Y}+{\mathrm{e}}_{\mathrm{i}}\text{,}$$ (2A)

where A denotes the assignment (treatment) variable (1 = 16‐hour program, 0 = 8‐hour program) based on the RIASI‐R, X indicates the instrument or pre‐treatment measure (scores on the RIASI‐R), Z denotes the outcome variable (self‐reported number of days of tobacco at the follow‐up), and Y indicates the self‐reported number of days of tobacco use at the assessment.

For all equations, the intercept term (A) shows the main effect and the interaction term (A*X) correspond to the slope in the regression line at the cutoff point. Another measure, the self‐reported number of days of substance use (Y) in the assessment period was included to improve the goodness of fit. As well, the interaction terms between other covariates, assignment and pre‐treatment measures were included in the models. The cutoff point was subtracted from the pre‐treatment measure to avoid the problem of collinearity.

If a discontinuity is observed at the cutoff point, it indicates that a treatment effect occurred. The discontinuity represents a statistically significant change in the slope or the y‐intercept, or both. If the interaction term is significant, we observe a change in the slope between the regression lines of the control group (8‐hour program) and the treatment group (16‐hour program). If the main effect term for the assignment (dummy coded) variable shows statistical significance, we observe a change in the intercept at the cutoff point between the regression lines of the control and treatment group (Trochim, 2006).

Results

Alcohol and tobacco using days before and after treatment

Table 1 presents the mean self‐reported number of days using alcohol and tobacco in the 90 days preceding assessment and the 90 days preceding the six‐month follow‐up for all participants. As can be seen, there were significant reductions in self‐reported number of days using both substances at follow‐up. These results provide evidence that program participation was associated with a beneficial effect for participants. However, as this is a simple pre‐post analysis, alternative interpretations of the cause of the decline (e.g. regression to the mean, simple passage of time) cannot be ruled out.

RD models of alcohol consumption

Table 2 shows the final RD models for our main outcome measure, self‐reported number of days using alcohol at follow‐up. The results show a significant and negative effect of the assignment variable in each of the analyses for the RIASI‐T and RIASI‐R. This means that clients assigned to the longer 16‐hour program based on the RIASI‐T or RIASI‐R show significant reductions in the self‐reported days of use of alcohol at the point of discontinuity (where the assignment took place). For clients assigned to the longer program based on the RIASI‐T score, the reduction corresponded to approximately one less drinking day (95% confidence interval [CI]: –1.72, –0.55) in the 90 days preceding follow‐up, and for clients assigned to the longer program based on the RIASI‐R score, the reduction corresponded to approximately two fewer drinking days (95% CI: –2.69, –1.23).

The results also show a very clear relationship between days of drinking and problem levels as measured by the RIASI‐T or RIASI‐R score at assessment and self‐reported days of drinking at follow‐up. Self‐reported drinking days at assessment strongly predicted self‐reported drinking days at follow‐up. As well, higher levels of alcohol‐related problems at assessment as measured by the RIASI‐T or RIASI‐R score significantly predicted more self‐reported drinking days at follow‐up.

Finally, for the RAISI‐T models the interaction between the assignment variable and the RIASI‐T score was significant as well as the interaction between the assignment variable and self‐reported number of drinking days at assessment. For the RIASI‐R models, the three‐way interaction of assignment variable with RIASI‐R score and self‐reported number of drinking days at assessment was significant.

Figure 1 presents the effects of pre‐treatment RIASI‐T scores, and assignment to the longer program at a RIASI‐T score of nine or more, on self‐reported number of drinking days at follow‐up. The mean of the outcome measures at each RIASI‐T data point varying from 1 to 20 is shown. Data for RIASI‐T scores larger than 20 are not presented because they represented fewer than 100 clients. The results demonstrate the reduction of about one drinking day at the point of assignment to the 16‐hour program. The results also illustrate the interaction of the cutoff point with the RIASI‐T scores at assessment. Prior to the score where assignment to the 16‐hour program occurs, the relationship between the RIASI‐T score and follow‐up drinking days is linear and positive. However, after the score where assignment to the 16‐hour treatment program occurs, the relationship appears flat, that is, there appears to be little relationship between RIASI‐T score and follow‐up drinking days.

Figure 1.

Self‐reported mean days of alcohol use at follow‐up by RIASI‐T score.

Figure 2 presents the effects of pre‐treatment RIASI‐R scores, and assignment to the 16‐hour program at a RIASI‐R score of six or more, on self‐reported number of drinking days at follow‐up. Again, the mean of the follow‐up measures at each RIASI‐R data point varying from 1 to 11 is presented in order to better visualize the findings, and data for values larger than 11 are not included in the figure because they represent fewer than 100 clients. The results here are similar to those seen with the RIASI‐T score. At the point of discontinuity where assignment to the longer program occurred, the incremental benefit of a reduction of about two drinking days can be seen. As well, prior to the cutoff for assignment to the longer 16‐hour program a clear positive relationship between RIASI‐R score and drinking days at follow‐up was seen, but after assignment this relationship is not apparent. The change of slope reflects the significant interaction between assignment and the RIASI‐R scores.

Figure 2.

Self‐reported mean days of alcohol use at follow‐up by RIASI‐R score.

RD models of tobacco

Table 3 presents the final RD models for our control outcome measure, self‐reported number of days using tobacco at follow‐up. For tobacco, the RD models do not include the assignment variable as its effect is not significant in either analysis. This suggests that clients sent to the 16‐hour program based on the RIASI‐T and RIASI‐R do not show specific significant reductions in the days of use of tobacco at the point of discontinuity (where the assignment took place).

The results show significant relationships between self‐reported number of days of tobacco use and addiction problem levels as measured by the RIASI‐T or RIASI‐R score at assessment and tobacco use at follow‐up. Higher numbers of self‐reported tobacco‐using days at assessment strongly predicted higher numbers of self‐reported tobacco‐using days at follow‐up. As well, higher levels of alcohol‐related problems at assessment as measured by the RIASI‐T or RIASI‐R score significantly predicted higher numbers of self‐reported tobacco‐using days at follow‐up. Interestingly, the interaction between the assignment variable and the instrument was significant for the RIASI‐T and RAISI‐R models. The interaction between the assignment variable and number of tobacco using days at assessment was significant for the RIASI‐T model only.

Discussion

The RD design, when properly implemented, is considered equivalent to randomized designs, around the point of discontinuity, in ability to draw causal inferences (Imbens and Lemieux, 2008; Linden et al., 2006; Shadish and Cook, 2009; Trochim, 1984, 1990). In spite of the strength of this design, it has only rarely been used in alcohol research (Carpenter and Dobkin, 2009; 2011; Daniels et al., 1992). The BOT program provides an important opportunity to employ this design because of the large sample size available, the consistency with which individuals were assigned to one of two treatment interventions, and the consistency with which services were provided at different locations. Thus, these results are of great interest in illustrating powerful methods to evaluate the impact of alcohol and drug treatment programs, as well as in providing important information on the impact of a criterion‐based alcohol treatment assignment regimen.

Two hypotheses guided this research. The first hypothesis was that assignment to the longer treatment program at the point of discontinuity (the cutoff point for assignment on the RIASI‐T and RIASI‐R) would be associated with a significant reduction in days using alcohol. We observed an overall reduction in days using alcohol between assessment and six‐month follow‐up, and the hypothesis assumes that at least a portion of this beneficial effect was due to the specific activities and contents of the interventions. In other words, if the clients were benefiting from the educational and therapeutic activities that were occurring in both the 8‐hour and 16‐hour programs, then assigning them to receive more of these activities at the cutoff points on the assessment instruments would be reflected in a specific and measurable improvement in outcome at the point of assignment. The results confirmed this hypothesis. Clients sent to the longer treatment program based on the RIASI‐T and RIASI‐R showed significant reductions in the days of use of alcohol at the point of discontinuity (where the assignment took place), in addition to the overall benefits suggested by the pre‐post analysis. For clients assigned to the 16‐hour treatment program because they exceeded the cutoff point on the RIASI‐T score, the reduction corresponded to approximately one less drinking day, and for clients assigned to 16‐hour treatment program because they exceeded the cutoff point on the RIASI‐R score, the reduction corresponded to approximately two fewer drinking days, in the 90 days preceding follow‐up. To put these effects in context, in terms of the average self‐reported number of drinking days seen at assessment (13.12) a one day reduction is a decline in drinking days of about 7.6%, while a two day reduction is a decline of about 15.2%.

These results provide support for providing longer or more intensive treatment for alcohol problems on the basis of severity‐based criteria. They also support the proposition that specific activities in the two programs contribute importantly to the overall beneficial results seen with alcohol use measures in the pre‐post analysis. That is, the ability of the RD analysis to demonstrate that an increase in the “amount” of program that clients received was associated with a specific and significant beneficial influence on outcomes supports the proposition that the overall benefits seen in the pre‐post analysis were at least partly due to program effects. It is also the case that other differences between the education and treatment programs reflecting program intensity, such as differences in group size, could have contributed to the discontinuity effect.

The second hypothesis was that no specific benefits of assignment to the longer treatment program would be observed on the measure of tobacco use. The analyses supported this hypothesis as well: assignment to the longer program based on the RIASI‐T or RIASI‐R was not associated with a significant reduction in days using tobacco at the point of discontinuity. Tobacco use is not addressed in the BOT program. As noted earlier, there was a significant reduction in days using tobacco between assessment and follow‐up. However, there was no reason to link this reduction in any specific way to differential program content or program activities. Thus, for tobacco, the reduction in use would seem to be most likely due to such factors as regression to the mean, simple passage of time, similar effects on tobacco use of each program regardless of assessment problem level, or a general increase in health consciousness and health behaviors rather than specific effects of components of the BOT program. The failure to find discontinuity effects supports this suggestion.

Two other results are of interest. First, we observed that the discontinuity effect was larger for assignments based on the RIASI‐R score than for those based on the RIASI‐T score (two days versus one day reduction in drinking). While these differences were not compared directly, this result may suggest that assignments based on the RIASI‐R could have a larger impact than those based on the RIASI‐T. It may be possible that the subset of items in the RIASI‐R are somehow more relevant to the nature of the problems experienced by this population. Participants are convicted drinking drivers and the RIASI‐R scale was developed to focus on a key problem measure for this population (recidivism), and thus the larger impact of assignment based on the RIASI‐R scale may reflect the salience of this measure to the specific problems that this population experiences. Second, the slope of the regression line appears to change with assignment to the treatment program. One possible contributor to this is that there is an increase in variability of the data points as RIASI scores increase, and since the regression lines after assignment to treatment are based on a smaller dataset with a higher standard deviation this could act to skew the means and affect the apparent slope of the line. As well, since the change occurs at about the point where assignment occurs, the difference in slopes could also be related to program effects. One possibility is that the approximately linear relationship between problem levels and outcomes seen at lower levels of problems when participants were streamed to the shorter, less intense program reflects a relatively low impact of that program such that the linear relationship between problem level and outcome was minimally influenced by program participation. For those assigned to the longer, more intensive treatment program, participation in this program may have affected the simple linear influence of pre‐program problem levels on outcomes. While interesting, these interpretations are speculative and require further investigation.

Several limitations must be considered in examining these results. One limitation is that results are based on self‐report measures which may be subject to a variety of factors that may act to affect their validity. For example, individuals may be motivated to exaggerate or minimize their use of alcohol and other drugs. Additionally, while participants are not required to attend the program by courts or police, attendance is necessary if they wish to obtain a driver's license following the period of mandatory license suspension. Thus, there may be demand characteristics in the program that may act to influence participants' self‐reports. While the available evidence indicates that self‐report measures of alcohol and drug use are generally reliable and valid (e.g. Harrison et al., 1993; O'Malley et al., 1983), under‐reporting of drug use and previous legal problems in this population has been noted in the literature (Lapham et al., 2002). Thus, while of substantial interest, these results need to be replicated and extended in further research.

Nevertheless, these results have important implications for understanding the effects of severity‐based assignment systems for alcohol treatment, and in health contexts more generally (e.g. Mee‐Lee et al., 2001; Murray et al., 2010), in demonstrating a beneficial influence of such a system and also in illustrating a powerful method for evaluating those systems. The RD design also provides important opportunities for assessing client factors that may moderate program effectiveness (Hester and Miller, 1988). A severity‐based assignment scheme in which there are two levels of programs is a variant of a more general treatment matching strategy (e.g. Project Match Research Group, 1997), in which individuals with specific characteristics are streamed to specific forms of programs thought to be more effective. Thus, RD analysis also permits an assessment of the potential of individual characteristics to modify the impact of program assignment, and hence to identify additional factors that might be usefully employed as assignment criteria in order to maximize beneficial effects of health interventions (e.g. Doss and Atkins, 2006).

Declaration of interest statement

The authors have no competing interests.