Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2008 Dec 1.
Published in final edited form as: Addict Behav. 2007 May 3;32(12):2819–2836. doi: 10.1016/j.addbeh.2007.04.020

Illusory Predictors: Generalizability of Findings in Cocaine Treatment Retention Research

Angela L Stotts 1, Marc E Mooney 1, Shelly L Sayre 1, Meredith Novy 1, Joy M Schmitz 1, John Grabowski 1
PMCID: PMC2249556  NIHMSID: NIHMS32700  PMID: 17543472

Abstract

Treatment retention is of paramount importance in cocaine treatment research as treatment completion rates are often 50% or less. Failure to retain cocaine patients in treatment has both significant research and clinical implications. In this paper we qualitatively and quantitatively demonstrate the inconsistency found across analyses of retention predictors in order to highlight the problem. First, a qualitative review of the published literature was undertaken to identify the frequency of predictors studied and their relations to treatment retention. Second, an empirical demonstration of predictor stability was conducted by testing a common set of variables across three similar 12-week cocaine clinical trials conducted by the same investigators in the same research clinic within a 5 year period. Results of the literature review indicated inconsistently selected variables of convenience, widely varying statistical procedures, and discrepant findings of significance. Further, quantitative analyses resulted in discrepancies in variables identified as significant predictors of retention among the three studies. Potential sources of heterogeneity affecting the consistency of findings across studies and recommendations to improve the validity and generalizability of predictor findings in future studies are proposed.

Keywords: Cocaine treatment, Retention, Methodology, Retention predictors

1. Introduction

Treatment retention is of paramount importance in addiction studies, and has been stressed as a primary endpoint, particularly for cocaine treatment (Satel and Kosten, 1991, Montoya et al., 1995, Lavori et al., 1999). While retention is poor across the majority of substance abuse treatment studies, treatment completion rates in cocaine clinical trials are often 50% or less (e.g., Johnson et al, 2006; Levin et al., 2006; Schmitz et al, 2001). Failure to retain patients in treatment has both significant research and clinical implications. From a research methodology perspective, retention is necessary to maintain internal validity, as patient dropout may bias the composition of groups as well as reduce statistical power (Friedman et al., 1998). If dropouts differ from non-dropouts on characteristics other than completion status, external validity is also threatened. Treatment effects may not generalize to all treatment-seeking, cocaine dependent individuals, but rather to the rarefied subset who completed a clinical trial. Reporting outcomes on those with less therapeutic exposure, or on only those who completed treatment, can result in biased estimates of intervention efficacy.

From a clinical perspective, some evidence suggests that participants who leave treatment programs prematurely fare worse than those who remain for the duration (Bleiberg et al., 1994, De Leon, 1991). The seminal paper by Baekeland and Lundwall (1975) on treatment retention in a variety of settings suggested that “dropping out” is a negative treatment indicator. Within the drug abuse literature, however, the effect of retention may vary by substance. For example, opiate-dependent patients who remain in treatment are more likely to be drug free and have lower relapse rates (Stark, 1992). Among cocaine-dependent samples, however, the relation between time in treatment and outcome has been less consistent (Carroll et al., 1994). More recent cocaine treatment studies, however, including an investigation of 5-year outcomes, suggest better outcomes for patients who have more exposure to treatment, particularly those with a high severity of drug and psychosocial problems (Simpson et al., 1999, 2002).

Because of both methodological and clinical issues, predictors of attrition and retention have become a major focus of research. Numerous studies in the past 15 years have attempted to identify key predictors of retention (e.g., Washton, 1990, Kleinman et al., 1992, Agosti et al., 1996, Alterman et al., 1997, Siqueland et al., 1998, Kampman et al., 2001, Milligan et al., 2004, Sayre et al., 2002, Means et al., 1989). The authors of this paper have been among the contributors to this literature (e.g., Sayre et al., 2002). A cursory review of the retention literature, however, indicates large inconsistencies in methods and findings. For example, some studies have found demographic characteristics such as age, race, gender, and marital status to be predictive of retention (e.g., Kleinman et al., 1992, Sayre et al., 2002, Carroll et al., 1994), while others have not (e.g., Hoffman et al., 1994, Carroll et al., 1991). Drug use severity variables have also been related to retention in some studies (e.g., Means et al., 1989, Gainey et al., 1993), but not in others (Kleinman et al., 1992). With notable exceptions (e.g., Joe et al., 1999), the result of such studies has been a large body of literature seemingly of limited use with findings of questionable generalizability.

In this paper we qualitatively and quantitatively demonstrate the inconsistency found across cocaine treatment predictor analyses in order to sufficiently highlight the problem. The focus will be on cocaine treatment retention due to the especially poor cocaine treatment retention rates and also to avoid systematic sources of inconsistency (i.e., variations in retention related to different substances of abuse). Potential sources of heterogeneity affecting the consistency of findings across studies and recommendations to improve the validity and generalizability of predictor findings in future studies are proposed. First, a qualitative review of the published literature was undertaken to identify the frequency of predictors studied and their relations to treatment retention. Second, we conducted an empirical demonstration of predictor stability by testing a common set of variables across three similar 12-week cocaine clinical trials conducted by the same investigators in the same research clinic (Grabowski et al., 2000, Schmitz et al., 2001). Together, this work identifies a number of problems in predicting cocaine treatment retention and solutions for improving methodology and potential for generalizability in this area of research.

2. Methods: Qualitative study

2.1. Study selection

Studies published between January 1980 and August 2004 were considered in the current paper. Initial identification of studies for the review portion of this paper began with a search of three widely used computer databases, PsychINFO (American Psychological Association), MEDLINE (National Library of Medicine), and Web of Science (Thompson Corporation). Queries were conducted employing combinations of the word cocaine with the key words attrition, retention, dropout, prediction, predictors, and correlates. Reference lists provided in the collected literature were further examined for studies appropriate for the present review (i.e., “snowballing”). In addition, papers that met inclusion criteria were submitted for cited reference searches through the Web of Science database to identify publications that subsequently cited the index publications.

2.2. Study inclusion criteria

Studies included in the present review were those that: (a) involved outpatient treatment (pharmacological, psychological, or both); (b) involved participants with a primary diagnosis of cocaine abuse, dependence, or problem cocaine use; (c) evaluated multiple predictors or correlates of treatment completion or attrition; (d) were written in English or had been translated into English; and (e) were published in peer-reviewed journals.

2.3. Study exclusion criteria

Conference abstracts were not included in the present review, as they report data of a preliminary nature and generally offer limited description of research methods. Unpublished studies and dissertations were also not included, as they have not undergone the rigorous peer-review process.

2.4. Review variables

In studies meeting inclusion criteria, information concerning study design, analysis techniques, and predictors of treatment retention were abstracted. Subsequently, evaluative judgements using standardized coding forms forced predictors into one of seven superordinate (e.g., demographics) and 27 subordinate (e.g., gender) categories. The superordinant and subordinate categories are listed in Table 1. Inter-rater agreement was excellent for both superordinant categories (Cohen’s κ = 0.91, 95% C.I., 0.86, 0.95), and subordinate categories (Cohen’s κ = 0.80, 95% C.I., 0.73, 0.85), where κ values > .74 are considered in the excellent range (Cicchetti, 1994).

Table 1.

Frequency of Predictor Use and Statistically Significant Relationship to Retention in 23 Treatment Studies for Cocaine Abuse or Dependence

Predictor Reported (N) Significance
 
Demographic   + − NS UK
  Age 10 3 2 5 0
  Education 12 3 1 7 1
  Employment 10 4 0 6 0
  Income 2 1 0 1 0
  Marital Status 6 1 0 5 0
  Other 1 0 1 0 0
  Race 8 5 0 3 0
  Residence 6 1 0 5 0
  Sex 7 2 0 5 0
  Social 3 0 1 2 0
Family          
  Family Problems 5 2 0 3 0
  Other 1 1 0 0 0
Legal          
  Arrests 4 0 1 3 0
  Legal 6 1 0 5 0
Psychological          
  Axis I/II 7 2 3 2 0
  Depression 4 1 0 3 0
  Impulsivity 2 0 1 0 1
  Other 25 6 1 16 2
Psychotherapy          
  Therapy/Therapist 5 0 0 5 0
Substance Abuse          
  Axis I/II 2 0 0 2 0
  Current Treatment 4 1 0 3 0
  Previous Treatment 8 1 1 6 0
  Recent Use 17 1 2 14 0
  Cocaine Route 8 0 1 7 0
  Severity 24 5 2 15 2
  Toxicology 6 1 2 3 0
  Other 8 1 2 5 0
Other 13 3 1 9 0
  Total 214 46 22 140 6
Open in a new tab

Note. + = statistically significant and positively related; − = statistically significant and negatively related; NK = not significant; UK = significance unknown/unreported.

3. Results: Qualitative Study

A total of 23 studies were identified that met inclusion criteria (see Appendix A). Results of our literature review indicated marked variability in sample size (Median = 128, Range = 50 – 428), number of predictors (Median = 7, Range = 1 – 45), and observation period (Median = 74 days, Range = 30 – 360). The majority of reports employed data from prospective pharmacotherapy or psychotherapy clinical trials (n = 16, 70%), while a minority employed observational datasets from treatment centers (n = 7, 30%). The types of analyses in order of most frequent use were: logistic regression (n = 8, 30.4%), ANOVA/ANCOVA (n = 6, 26.1%), survival analysis (i.e., Cox or Kaplan-Meier regression; n = 5, 21.7%), linear regression (n = 2, 8.7%), and bivariate analysis (i.e., t-tests, zero-order correlations; n = 2, 8.7%).

A total of 201 predictor variables were evaluated across the 23 studies. Due to the heterogenous nature of the analyses and predictor variables, we chose to summarize our findings in vote-tally format (Hunter and Schmidt, 1990). However, due to the fallibility of inferences about significant versus non-significant variables, we do not formally assess predictors for plurality (Hunter and Schmidt, 1990). The most frequent superordinate categories were found in the following order: substance abuse (n = 85, 38.3%), demographic (n = 65, 29.3%), psychological (n = 38, 17.1%), other (n = 13, 5.9%) legal (n = 10, 5.0%), family (n = 6, 2.7%), and psychotherapy (n = 5, 2.3%). Table 1 reflects the lack of unanimity across studies on the statistical significance of various predictor variables.

4. Summary: Qualitative Study

A qualitative literature review was performed to determine the consistency of variable selection and findings in studies designed to identify predictors of retention in treatment for cocaine abusing or dependent patients. The literature specific to cocaine treatment retention indicated that: (1) numerous variables have been inconsistently selected and tested; (2) statistical techniques vary widely; and (3) discrepant findings of significance exist across studies. The majority of selected variables were arbitrarily chosen post-hoc (e.g., demographic variables), devoid of theory, and analyzed with test statistics intended for testing pre-specified hypotheses (Moye, 2000).

5. Method: Quantitative Study

5.1. Participants & Procedures

The selected studies were conducted at the Treatment Research Clinic (TRC), a component of the Substance Abuse Research Center in the Department of Psychiatry and Behavioral Science at the University of Texas-Houston Medical School. The clinic itself is described elsewhere (Grabowski et al., 1997). The data presented in this paper were collected over a five-year period (1994–1999) as part of three randomized clinical trials for the treatment of cocaine dependence. All participants were recruited through advertisements in local media sources. Participants took part in an initial telephone screen to determine initial eligibility for an evaluative appointment. To be included in the trials, callers had to be: (a) English-speakers; (b) between the ages of 18 and 55; and (c) current users of cocaine. Common exclusion variables included: (a) pregnancy; (b) current dependence on substances other than cannabis or nicotine; (c) current Axis I psychotic, affective, or anxiety disorders (DSM-IV, (American Psychiatric Association, 1994); and (d) serious medical conditions.

Eligible callers were invited to participate in a 3–10 day pre-treatment evaluation, which included a physical exam, lab work, psychiatric assessment, and a series of questionnaires assessing demographics, substance use, and psychosocial functioning. Upon successful completion of the pre-treatment evaluation, participants began the 12-week, outpatient treatment programs which incorporated both therapy and medication components. Each of the three clinical trials was approved by the University of Texas-Houston Committee for the Protection of Human Subjects and is described in further detail below.

Activities common to clinic visits across the three studies included: (a) receiving medication, (b) provision of urine samples at each visit; (c) collection of vital signs; and (d) completion of self-report questionnaires. Participants were each compensated $10–20 per week during treatment. Sample sizes vary slightly from the original reports of the following studies due to missing data on various measures.

Study 1

This study (October 1994–October 1997) was a 12-week trial examining the effectiveness of three doses of risperidone (2, 4 or 8 mg) compared with placebo in an abuse/dependent population of 80 cocaine users who entered the treatment phase (Grabowski et al., 2000). Following stabilization on the medication, participants attended the clinic twice a week for treatment. All participants attended cognitive behavioral therapy once per week.

Study 2

This study (May 1995–November 1999) was a 12-week relapse prevention trial examining the effectiveness of naltrexone (50 mg) compared with placebo in a population of 80 cocaine dependent individuals who achieved initial abstinence (5 consecutive days with cocaine-free urine samples Schmitz et al., 2001). Participants of the program attended the clinic twice per week during the treatment phase. Cognitive behavioral therapy was provided twice per week for the first 8 weeks of treatment and once per week during the final 4 weeks of treatment.

Study 3

This study (April 1995–August 1998) was a 12-week trial examining the effectiveness of two doses of dextro-amphetamine sulfate (final dose 30 or 60 mg sustained release) compared with placebo in an abuse/dependent population of 106 cocaine users entering the treatment phase (Grabowski et al., 2001). Following stabilization on the medication, participants attended the clinic twice a week for treatment. During week 5, a dose increase from 15 mg to 30 mg or from 30 mg to 60 mg occurred. Cognitive behavioral therapy was provided weekly.

5.2. Measures

Across the three studies, common measures were selected based on the frequency analysis of variables employed in the literature review and availability. Measures included a demographic questionnaire, the Addiction Severity Index (ASI, McLellan et al., 1992), the Cocaine Craving Scale (CCS, Halikas et al., 1991)(measured on a 10 cm visual analogue scale), the University of Rhoade Island Change Assessment (URICA, McConnaughy et al., 1983), the Beck Depression Inventory (BDI, Beck et al., 1961), and laboratory measures of benzoylecgonine (BE: cocaine positive status was defined as a BE value ( 300 ng/mL, Preston et al., 1997).

5.3. Outcomes

Treatment completion was evaluated in two ways to promote comparison with the published literature. First, treatment completion was defined dichotomously (0 = dropped out before end of treatment period; 1 = completed treatment). Second, treatment completion was evaluated as a continuous variable based on the number of days in treatment (1 to 84 days).

5.4 Analysis

All analyses were conducted using the Statistical Analysis System, Version 9.1 (SAS, 2004). Type-I error rate was controlled on a studywise basis, using Bonferroni adjustments. Three analytic strategies were implemented based on those most commonly reported in the literature review: (a) bivariate correlation; (b) logistic regression; and (c) Cox regression (i.e., survival analysis). In addition, as an extension of current methods, bootstrap resampling methods were implemented to evaluate the stability of the identified predictors for logistic regression (Steyerberg et al., 1999, Sauerbrei and Schumacher, 1992, Chen and George, 1985, Austin and Tu, 2004). The bootstrap is used to assess variability of test statistics, allowing one to estimate an empirical distribution function by repeated sampling from the observed data. Bootstrap methods are particularly useful for approximating the distribution of test statistics in small samples (Austin and Tu, 2004). Regression models were constructed with consideration of multicollinearity (i.e., redundant measures of drug and alcohol use were excluded) with the following variables included: (a) medication (covariate; 0 = placebo; 1 = active); (b) age; (c) education; (d) gender; (e) race (0 = White; 1 = Non-white); (f) marital status (0 = married; 1 = non-married); (g) employment status (0 = unemployed; 1 = employed), (h) number of days of cocaine use in past 30 days, (i) alcohol use in past 30 days, (j) number of previous drug treatments, (k) intake urine status (0 = cocaine negative; 1 = cocaine positive [BE value ≥ 300 ng/mL]), (l) craving score, (m) motivation (0 = unmotivated; 1 = motivated) (dichotomized scoring of the URICA, see Stotts et al., 2001), (n) ASI Family composite, (o) ASI Psychiatric composite, (p) ASI Legal Composite, and (q) BDI score. Automatic variable selection criteria were as follows: forward (entry, p ≤ .10; backward, (retention, p ≤ .10), and stepwise (entry, p ≤ .25, retention, p ≤ .10)(Austin and Tu, 2004).

6. Results

6.1. Sample Description

The current study samples are presented in Table 2. Across the three studies, most participants were young, not married, non-white males with at least a high school education. The majority of participants used cocaine by smoking it, and had used cocaine for nearly a decade. Retention curves for the three studies are shown in Figure 1.

Table 2.

Sample Characteristics of Three Studies

  All Study 1 Study 2 Study 3
  N = 266 N= 80 N = 80 N= 106
 
  M/% (SD) M/% (SD) M/% (SD) M/% (SD)
Demographic        
  Age 36.4 (6.4) 35.5 (6.7) 35.3 (7.2) 34.4 (6.7)
  Gender (% female) 23 25 29 25
  Race (% non-white) 66 63 59 63
  Education 12.9 (2) 12.7 (2) 12.4 (1.9) 12.7 (2)
  Married 23 24 21 29
  Employed 41 44 41 53
Drug Use        
  Drug Composite 0.3 (0.1) 0.3 (0.1) 0.3 (0.1) 0.2 (0.1)
  Alcohol Composite 0.2 (0.2) 0.2 (0.2) 0.2 (0.2) 0.1 (0.2)
  Alcohol (Past 30 Days) 10.3 (10.3) 12.1 (10.5) 6.4 (8.9) 12.0 (5.5)
  Years Alcohol 11.6 (8.6) 11.9 (9.1) 12.2 (8.8) 12 (10.2)
  Cocaine (Past 30 Days) 12.7 (9.3) 12.3 (8.4) 12.5 (9.6) 13.1 (9.6)
  Years Cocaine 8 (5.3) 8 (5.3) 8.2 (5.1) 8 (5.4)
  Cocaine Route 78 78 75 81
  Drug Treatments 1 (1.4) 1.3 (2.2) 1.1 (1.8) 1.9 (3.2)
  % Cocaine + at intake 62 74 38 71
  Craving Score 73.2 (22.8) 70.2 (23) 71.3 (25.1) 65.3 (20.4)
  % High Motivation 23 19 35 17
Psychosocial        
  Family Composite 0.2 (0.2) 0.2 (0.2) 0.2 (0.2) 0.2 (0.3)
  Psychiatric Composite 0.1 (0.1) 0.1 (0.2) 0.1 (0.2) 0.2 (0.2)
  Employment Composite 0.5 (0.3) 0.5 (0.3) 0.5 (0.3) 0.4 (0.3)
  BDI Score 11.4 (5.6) 11.6 (5.8) 10.1 (5.4) 12.1 (5.5)
Legal        
  Legal Composite 0.2 (0.4) 0.2 (0.4) 0.2 (0.4) 0.4 (0.5)
  #of Convictions 25.5 (38.9) 15.1 (31.4) 14.7 (31.1) 1.7 (2.7)
  % Probation 11 15 10 25
Open in a new tab

Figure 1.

Figure 1

Open in a new tab

Retention curves indicating percentage of the sample remaining by days in treatment for three cocaine treatment studies.

6.2. Correlates of Retention

Bivariate associations between selected variables are shown in Table 3. Overall, correlation coefficients were quite low. Following Bonferroni adjustments, no predictors achieved statistical significance, whether retention was modeled as a continuous outcome (i.e., days of treatment completed) or dichotomous outcome (i.e., treatment completed or not)

Table 3.

Correlations between Attrition and Selected Predictors

  Study 1 Study 2 Study 3
  N = 80 N = 80 N = 106
  Days Completion Days Completion Days Completion
 
Demographic            
  Age 0.19 0.19 −0.09 −0.06 0.02 0.05
  Gender 0.02 0.12 0.00 −0.04 −0.15 0.00
  Race (% non-white) 0.11 0.14 −0.02 −0.02 −0.15 −0.15
  Education −0.01 −0.04 0.26 0.21 0.14 0.10
  Married −0.02 −0.04 −0.12 −0.18 0.05 0.05
  Employed 0.04 0.07 −0.02 −0.07 0.05 0.06
Drug Use            
  Drug Composite 0.04 −0.02 −0.04 −0.01 0.02 0.05
  Alcohol Composite 0.18 0.13 −0.07 −0.04 0.05 0.04
  Alcohol (Past 30 Days) 0.15 0.10 0.05 0.05 −0.02 0.01
  Years Alcohol −0.02 0.06 −0.01 −0.11 −0.03 −0.04
  Cocaine (Past 30 Days) −0.11 −0.15 0.09 0.05 0.02 0.09
  Years Cocaine 0.00 0.11 0.03 0.04 0.05 0.06
  Cocaine Route 0.04 0.08 0.12 0.05 −0.19 −0.16
  Drug Treatments −0.03 0.00 −0.03 −0.03 0.10 0.01
  % Cocaine Positive 0.01 0.09 −0.05 −0.09 −0.04 −0.13
  Cocaine Level (ng/mL) 0.06 −0.08 0.17 0.19 0.17 0.10
  Craving Score 0.09 0.02 −0.19 −0.18 0.12 0.09
  % High Motivation 0.11 0.04 0.05 0.01 0.05 0.06
Psychosocial            
  Family Composite 0.14 0.13 0.00 −0.06 0.00 −0.07
  Psychiatric Composite 0.10 0.00 0.14 0.16 −0.10 −0.15
  Employment Composite −0.02 0.03 −0.04 −0.09 −0.20 −0.11
  BDI Score 0.05 −0.01 −0.13 −0.13 −0.14 −0.18
Legal            
  Legal Composite 0.01 −0.04 0.04 0.10 0.14 0.01
  # of Convictions 0.11 0.02 0.16 0.17 −0.03 0.02
  % Probation −0.05 −0.02 −0.06 −0.04 0.08 −0.03
Open in a new tab

Note. Correlations coefficients are Pearson, phi, and point biserial coefficients depending on data types. No effects reached statistical significance following Bonferonni adjustment.

6.3. Regression Analyses

Full-rank regression models were evaluated for each study for both retention outcomes (study completion, days completed). Although p <.05 was considered the cutoff for significance, variables achieving p <.10 are also considered.

6.3.1. Logistic regression

In both studies 1 and 2, less cocaine use in the 30 days preceding entry to the trial predicted retention, χ²(1) = 5.64, p = 0.0175, and χ²(1) = 3.15, p = 0.0761, respectively. In study 3, greater years of education, χ²(1) = 3.91, p = 0.0481, and fewer days of alcohol use in the 30 days preceding the trial, χ²(1) = 2.72, p = 0.0994, predicted retention.

6.3.2. Cox regression

No predictor variables were shared in any of three models. In study 1, participants with greater days of cocaine use in the 30 days before treatment, χ²(1) = 7.20, p = 0.0073, and greater craving for cocaine at the start of treatment, χ²(1) = 3.80, p = 0.0513, dropped out earlier. In study 2, being older, χ²(1) = 7.62, p = 0.0058, and drinking alcohol more frequently in the last 30 days, χ²(1) = 3.92, p = 0.0478, was associated with earlier drop out. In study 3, education level positively predicted longer time in treatment, χ²(1) = 7.61, p = 0.0058, along with fewer days using alcohol in the month preceding treatment, χ²(1) = 3.91, p = 0.0478, fewer previous drug and alcohol treatments, χ²(1) = 3.32, p = 0.0686, and lower ASI legal composite scores, χ²(1) = 3.33, p = 0.0679.

6.4. Automatic Variable Selection

In order to illustrate the instability of models constructed through AVS, logistic regression models were evaluated in two ways. First, we examined the models selected through forward, backward, and stepwise AVS techniques. Second, we employed bootstrap resampling to evaluate the stability of model selection using 1,000 samples.

6.4.1. Individual studies

Results of AVS logistic regressions for each study are shown in Table 4.

Table 4.

Logistic Regression Analyses for Predictors of Retention Using, Backward, Forward, and Stepwise Retention Procedures

  Study 1
 
Variable Forward Backward Stepwise
Age χ²(1) = 5.27, p = 0.022 χ²(1) = 5.77, p = 0.016 χ²(1) = 2.86, p = 0.091
Cocaine Use χ²(1) = 3.11, p = 0.078 χ²(1) = 3.87, p = 0.049 –
Sex – χ²(1) = 2.71, p = 0.1 –
ASI Family χ²(1) = 2.76, p = 0.097 – –
       
  Study 2
 
Variable Forward Backward Stepwise
Education χ²(1) = 4.29, p = 0.038 χ²(1) = 4.77, p = 0.029 χ²(1) = 2.86, p = 0.091
       
  Study 3
 
Variable Forward Backward Stepwise
Cocaine Use – χ²(1) = 5.27, p = 0.022 –
Intake Urine – χ²(1) = 3.17, p = 0.075 –
ASI –   –
Psychiatric   χ²(1) = 5.27, p = 0.022  
Open in a new tab

In study 1, only greater age and fewer days of cocaine use were consistently predictive of study completion across the AVS techniques, whereas the ASI family composite (forward AVS) and sex (backward AVS) were retained with only 1 technique. In study 2, greater years of education was a stable predictor of study completion. In study 3, only the backward AVS selected a model, showing greater cocaine use in the 30 days preceding treatment, intake urine sample positive for cocaine, and fewer psychiatric complaints were predictive of study completion.

6.4.2. Bootstrap resampling

Results of bootstrap resampling analyses are presented in Figure 2.

Figure 2.

Figure 2

Open in a new tab

For each study, the number of times each variable was selected as a significant predictor of retention in 1000 derived models using bootstrap resampling analyses with three automated variable selection procedures (Forward, Backward, and Stepwise).

In study 1, the typical model contained about 4 variables (forward, M = 4.3, SD = 1.8; backward, M = 4.8, SD = 2.6; and stepwise, M = 3.0, SD = 1.4). Inspection of Figure 2 illustrates that only age and cocaine use in the 30 days preceding treatment were included in more than half of 1000 derived models.

In study 2, the typical model contained about 4 variables (forward, M = 4.3, SD = 1.8; backward, M = 4.3, SD = 2.0; and stepwise, M = 3.3, SD = 1.6). Few variables were frequently included with any consistency, and in some cases, inclusion strongly varied by AVS approach (e.g., age, cocaine use, and education).

In study 3, the typical model contained about 4 variables (forward, M = 3.5, SD = 1.4; backward, M = 4.1, SD = 1.9; and stepwise, M = 3.3, SD = 1.6). Only cocaine use in the 30 days preceding treatment and intake urine sample positive for cocaine showed some consistency as predictors of treatment dropout.

7. Summary: Quantitative Study

Results of quantitative analyses of three cocaine pharmacotherapy treatment studies provide further evidence suggesting the instability of post-hoc analyses conducted to identify retention predictors. Despite the fact that the cocaine trials were conducted by the same investigators within the same research clinic within a 5 year period, almost no overlap existed across studies in predictor variables found to be associated with cocaine treatment retention. Further, logistic regression models on the same variables using three automated variable selection methods (i.e., forward selection, backward selection, and stepwise selection) within each study produced varying results. Finally, bivariate correlations between the predictor variables and two separate measures of retention—one continuous and one dichotomous—indicated a notable absence of association for all variables, similar to what has been found in past studies (e.g., Kleinman et al, 1992; Roberts & Nishimoto, 1996). Somewhat surprising is the fact that medication condition did not predict retention in any of the three studies given the visually-different retention curves associated with each. Methodological explanations discussed subsequently may account for the lack of contribution found for medication and perhaps other seemingly relevant variables.

8. Conclusions

Cumulative results of the qualitative and quantitative studies highlight significant inconsistencies that cast doubt on the meaningfulness or generalizability of the current literature on cocaine treatment retention. Generalizability can be defined as the ability of a “system” or model to provide accurate predictions in a different sample of patients. Justice et al. (1999) state that, “…a system that can only predict outcomes in the sample in which it was developed is useless.” Thus, in the cocaine treatment field, failure to reliably replicate findings regarding predictors of retention indicates that little progress has been made in identifying key characteristics associated with treatment drop out, precluding the development of more efficient and effective strategies to enhance retention.

Several plausible explanations can be invoked for the lack of reliable findings in this area of research. First, the majority of studies examining retention have tested a large number of variables without benefit of theoretical models defined a priori, nor even a rationale for selecting various predictors. Researchers are rejecting this type of search for simple, random predictors of drop-out (e.g., Harris, 1998). Further, in order to develop a systematic and coherent literature, investigators need to move beyond research on correlates of attrition and begin to propose and test theoretical models with hypothesized causal mechanisms and clearer implications for preventing attrition, rather than rely on completely data-driven systems which maximize the likelihood of chance findings (Streiner, 1994, Henderson and Denison, 1989). At minimum, researchers should select dynamic variables (e.g., motivation, mood) that may provide implications for treatment development and which may be influenced by intervention, rather than static, “tombstone” variables (e.g., race). For example, Moeller and colleagues (2001) specifically investigated the impact of impulsivity on both cocaine use and retention and found it to be a significant predictor, suggesting that cocaine treatment may benefit from including components targeting impulsivity. McKay et al. (2000) similarly investigated the prognostic significance of antisocial personality disorder (ASPD) in cocaine patients, which led to implications for treatment based on outcomes related to this diagnosis. Once potentially important constructs such as impulsivity and ASPD are identified, more complex models estimating the relative contribution for each and potential interactions with other variables can be developed and tested. Joe and colleagues (1999) provide an excellent example of testing complex models in their study examining retention, as defined by treatment process components (session attributes and therapeutic involvement), within the context of patient factors (e.g., motivation). Without theoretical models, the extremely large number of possible variable interactions cannot be narrowed, as it is currently not technologically possible to identify and test all complex interactions of potentially important variables. Thus, we are left with simplistic models which are not intuitively sufficient to explain a complex human behavior such as treatment retention. Although opposing arguments can be made, proposing theoretical models that can be subjected to falsification has been the foundation of deductively rigorous science.

The failure to reproduce cocaine retention predictor findings across studies and samples may also be attributed to the lack of specification of important contextual factors, including therapist, treatment program, study protocol, and sample characteristics (e.g., inclusion, exclusion criteria). Even seemingly insignificant factors such as staff turnover, program location, and clinic rules/atmosphere (e.g., relaxed vs. strict observance of clinic rules) may account for more variance in retention than one might expect (Harris, 1998). Improvements in client retention may be possible with fairly minor modifications in such factors, e.g., use of a structured induction to clarify expectations and reduce anxiety, cultural sensitivity training, and monetary incentives for continuing clients (Zweben and Li, 1981, Walitzer et al., 1999, Petry, 2000). Thus, contextual factors that may significantly influence retention in treatment need to be clearly identified in all studies of treatment retention.

Statistical methods used to identify significant predictor variables in cocaine treatment retention outcomes are also variable across studies. These methods all have associated advantages and disadvantages and inherently rely on the nature of the dependent variable selected to measure retention or attrition, also inconsistently chosen across studies. For example, logistic regression analyses are often used when retention is defined dichotomously (e.g., treatment completer vs. non-completer). Results from these analyses are easy to interpret, however, information is lost by collapsing retention data to one binary category. Linear or Cox regression procedures are typically employed with continuous measures of retention (e.g., number of sessions attended). These methods use more of the available data, however interpretability is often sacrificed. Regardless, differing measures assessed cross-sectionally using differing statistical analyses of retention are likely to result in the inconsistent identification of important variables. The consistent use of more sophisticated statistical methods, such as the longitudinal modeling of complex relationships among patient characteristics, process variables, and change over time (e.g., Joe et al., 1999) may assist in the development of reliably reproducible models for understanding retention in treatment.

Equally if not more problematic are the commonly used automated variable selection (AVS) methods. Several potential problems with the use of these selection methods have been identified: (1) R² values are often biased high; (2) estimated standard errors are often biased low; and (3) results are dependent upon the correlation between the predictor variables. As a result, AVS methods increase the likelihood of finding chance associations in a sample when the true association in the population is actually weaker. In addition, the usual test statistics on which these methods are based were intended for testing hypotheses established a priori and not post-hoc (Streiner, 1994, Steyerberg et al., 1999, Austin and Tu, 2004, Steyerberg et al., 2001).

Small sample sizes also pose a challenge for cocaine treatment retention studies and may lead to discrepant prediction models. Harrell et al (1984, in Altman and Royston, 2000) suggest that for regression modeling the EPV (events per variable) should be at least ten times the number of potential prognostic variables included in the model. Others argue that more complex rules-of-thumb are needed to estimate minimum sample size as a function of the number of predictors as well as effect size (Green, 1991). Regardless, with limited outcome events or observations and a large number of baseline characteristics included, the resulting regression coefficients for individual variables may not be trustworthy, and may represent spurious associations (Concato et al, 1993). Results of models having fewer than 10 outcome events per independent variable are questionable in terms of accuracy, and the usual tests of statistical significance may be invalid. Therefore, recommended limits on subject to variable ratios need to be adhered to more stringently. Compounding the problem of small sample size is the dilemma of missing data. If losses to follow-up are large and possibly related to outcome events, (i.e., not missing at random), then prognostic estimates may be seriously biased (Kelley and Maxwell, 2003, Harrell et al., 1985).

Possibly most problematic, however, is the lack of an accepted convention or requirement for validation of findings prior to publication. Within the substance abuse treatment field, initial predictor models of retention have rarely been cross-validated on independent samples, and according to the results of the current study are unlikely to confirm initial findings due to factors described previously (e.g., contextual differences, reliance on methods which capitalize on chance variables). Methods to maximize internal validation or accuracy of findings drawn from the original sample may be a necessary first step to enhance generalizability. At minimum, retention predictor studies should make use of internal validation procedures such as data exclusion (random split sample) when sample size permits or resampling techniques (e.g., bootstrapping). While in the current paper we used bootstrap resampling to illustrate the instability of prognostic models identified through AVS, they can be used to validate empirically derived models, and to achieve less biased parameter estimates and accompanying standard errors (Schumacher et al., 1997). Ideally, internal validation procedures would be followed by external validation procedures, i.e. testing the accuracy of the system in another sample at the same site or at different or multiple geographic sites (by independent investigators).

Concerns about prognostic modeling discussed here in the context of substance abuse treatment retention are in general not new, however, due to time constraints and pressures to publish they are largely ignored. The result is a literature on which no firm conclusions can be drawn, and more importantly which provides no coherent direction for intervention. Based on identified weaknesses of the literature, several recommendations can be made for future studies evaluating factors associated with treatment retention (See Table 5). Similar recommendations can be applied to prediction studies of other substance abuse treatment outcomes and are not limited specifically to retention. The recommendations are fairly ambitious and will take a coordinated effort among substance abuse researchers. Failing to address these issues, however, will perpetuate a literature of inconsistent, contradictory, and non-generalizable findings from which no meaningful conclusions can be drawn.

Table 5.

Recommendations for Improving the Quality of Findings in Prognostic Treatment Retention Studies.

Recommendations
▪ Prospectively develop theoretical models of retention/attrition using dynamic variables
   
▪ Develop conventions for specifying categories of key contextual factors that may impact attrition (e.g., precise characterization of subjects, procedures, treatment program)
   
▪ Specify conventions for statistical methods and for measurement of retention/attrition to promote consistency across studies
   
▪ Specify minimum sample size and missing data requirements
   
▪ Validate findings prior to publication using split-sample or re-sampling methods
Open in a new tab

Acknowledgements

This research was supported by the National Institute on Drug Abuse research grants DA09262 & DA06143. Part of the data was presented at the annual meeting of the College on Problems of Drug Dependence in Orlando, Florida, June 2005.

Appendix A

23 Published Trials Evaluating Predictors of Treatment Retention/Completion in Cocaine-Abusing Patients

Author Sitea N
Agosti et al., 1991 Study 60
Means et al., 1989 Clinic 81
Kleinman et al., 1992 Clinic 86
Gainey et al., 1993 Study 110
Agosti et al., 1996 Study 198
Roberts and Nishimoto, 1996 Clinic 369
Alterman et al., 1996 Study 95
Alterman et al., 1997 Study 256b
Gottheil et al., 1997 Study 447
Siqueland et al., 1998 Study 286
McMahon et al., 1999 Clinic 54
Mulvaney et al., 1999 Study 87
Pena et al., 1999 Study 294
Alterman et al., 2000 Study 160
McKay et al., 2000 Study 127
Kampman et al., 2001 Study 128
Moeller et al., 2001 Study 50
Sterling et al., 2001 Clinic 116
Sayre et al., 2002 Study 165
Siqueland et al., 2002 Study 487
Levin et al., 2004 Clinic 135
Milligan et al., 2004 Study 111
Patkar et al., 2004 Clinic 141
Open in a new tab
a

Note: Site denotes population: Clinic = non-select sample, Study = research sample.

b

Sample size consists of 4 samples combined.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Agosti V, Nunes E, Ocepeck-Welikson K. Am J Drug Alcohol Abuse. 1996;22:29–39. doi: 10.3109/00952999609001643. [DOI] [PubMed] [Google Scholar]
  2. Agosti V, Nunes E, Stewart JW, Quitkin FM. The International Journal of the Addictions. 1991;26:327–334. doi: 10.3109/10826089109058888. [DOI] [PubMed] [Google Scholar]
  3. Alterman AI, Kampman K, Boardman CR, Cacciola JS, Rutherford MJ, McKay JR, Maany I. Drug and Alcohol Dependence. 1997;46:79–85. doi: 10.1016/s0376-8716(97)00049-5. [DOI] [PubMed] [Google Scholar]
  4. Alterman AI, McKay JR, Mulvaney FD, Cnaan A, Cacciola JS, Tourian KA, Rutherford MJ, Merikle EP. Drug Alcohol Depend. 2000;59:215–221. doi: 10.1016/s0376-8716(99)00124-6. [DOI] [PubMed] [Google Scholar]
  5. Alterman AI, McKay JR, Mulvaney FD, McLellan AT. Drug and Alcohol Dependence. 1996;40:227–233. doi: 10.1016/0376-8716(95)01212-5. [DOI] [PubMed] [Google Scholar]
  6. Altman DG, Royston P. Statistics in Medicine. 2000;19:453–473. doi: 10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5. [DOI] [PubMed] [Google Scholar]
  7. American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV. Washington, DC: American Psychiatric Association; 1994. [Google Scholar]
  8. Austin PC, Tu JV. Journal of Clinical Epidemiology. 2004;57:1138–1146. doi: 10.1016/j.jclinepi.2004.04.003. [DOI] [PubMed] [Google Scholar]
  9. Baekeland F, Lundwall L. Psychological Bulletin. 1975;82:738–783. doi: 10.1037/h0077132. [DOI] [PubMed] [Google Scholar]
  10. Beck A, Ward C, Mendelson M, Mack J, Erbaugh J. Archives of General Psychiatry. 1961;49:599–608. doi: 10.1001/archpsyc.1961.01710120031004. [DOI] [PubMed] [Google Scholar]
  11. Bleiberg JL, Devlin P, Croan J, Briscoe R. Int J Addict. 1994;29:729–740. doi: 10.3109/10826089409047906. [DOI] [PubMed] [Google Scholar]
  12. Carroll KM, Rounsaville BJ, Gawin FH. Am J Drug Alcohol Abuse. 1991;17:229–247. doi: 10.3109/00952999109027549. [DOI] [PubMed] [Google Scholar]
  13. Carroll KM, Rounsaville BJ, Gordon LT, Nich C, Jatlow P, Bisighini RM, Gawin FH. Arch Gen Psychiatry. 1994;51:177–187. doi: 10.1001/archpsyc.1994.03950030013002. [DOI] [PubMed] [Google Scholar]
  14. Chen CH, George SL. Stat Med. 1985;4:39–46. doi: 10.1002/sim.4780040107. [DOI] [PubMed] [Google Scholar]
  15. Cicchetti DV. Psychological Assessment. 1994;6:284–290. [Google Scholar]
  16. De Leon G. In: Improving drug abuse treatment, NIDA Research Monograph 106. Pickens RW, Leukefeld CG, Schuster CR, editors. Rockville, MD: National Institute on Drug Abuse; 1991. pp. 218–244. [PubMed] [Google Scholar]
  17. Friedman LM, Furburg CD, DeMets DL. Fundamentals of clinical trials. New York: Springer; 1998. [Google Scholar]
  18. Gainey RR, Wells EA, Hawkins JD, Catalano RF. International Journal of the Addictions. 1993;28:487–505. doi: 10.3109/10826089309039643. [DOI] [PubMed] [Google Scholar]
  19. Gottheil E, Sterling RC, Weinstein SP. Journal of Addictive Diseases. 1997;16:1–14. doi: 10.1300/J069v16n02_01. [DOI] [PubMed] [Google Scholar]
  20. Grabowski J, Arnoni G, Elk R, Rhoades H, Schmitz J. NIDA Res Monogr. 1997;175:158–181. [PubMed] [Google Scholar]
  21. Grabowski J, Rhoades H, Schmitz J, Stotts A, Daruzska LA, Creson D, Moeller FG. J Clin Psychopharmacol. 2001;21:522–526. doi: 10.1097/00004714-200110000-00010. [DOI] [PubMed] [Google Scholar]
  22. Grabowski J, Rhoades H, Silverman P, Schmitz JM, Stotts A, Creson D, Bailey R. Journal of Clinical Psychopharmacology. 2000;20:305–310. doi: 10.1097/00004714-200006000-00003. [DOI] [PubMed] [Google Scholar]
  23. Green SB. Multivariate Behavioral Research. 1991;26:499–510. doi: 10.1207/s15327906mbr2603_7. [DOI] [PubMed] [Google Scholar]
  24. Halikas JA, Kuhn KL, Crosby R, Carlson G, Crea F. Comprehensive Psychiatry. 1991;32:22–27. doi: 10.1016/0010-440x(91)90066-l. [DOI] [PubMed] [Google Scholar]
  25. Harrell FE, Jr, Lee KL, Califf RM, Pryor DB, Rosati RA. Stat Med. 1984;3:143–152. doi: 10.1002/sim.4780030207. [DOI] [PubMed] [Google Scholar]
  26. Harrell FE, Lee KL, Matchar DB, Reichert TA. Cancer Treatment Reports. 1985;69:1071–1077. [PubMed] [Google Scholar]
  27. Harris PM. American Journal of Evaluation. 1998;19:293–305. [Google Scholar]
  28. Henderson DA, Denison DR. Psychological Reports. 1989;64:251–257. [Google Scholar]
  29. Hoffman JA, Caudill BD, Koman JJ, Luckey JW, Flynn PM, Hubbard RL. Journal of Addictive Diseases. 1994;13:115–128. doi: 10.1300/j069v13n04_01. [DOI] [PubMed] [Google Scholar]
  30. Hunter JE, Schmidt FL. Methods of meta-analysis: Correcting error and bias in research findings. Newbury park, CA: Sage; 1990. [Google Scholar]
  31. Joe GW, Simpson DD, Broome KM. Drug and Alcohol Dependence. 1999;57:113–125. doi: 10.1016/s0376-8716(99)00088-5. [DOI] [PubMed] [Google Scholar]
  32. Justice AC, Covinsky KE, Berlin JA. Annals of Internal Medicine. 1999;130:515–524. doi: 10.7326/0003-4819-130-6-199903160-00016. [DOI] [PubMed] [Google Scholar]
  33. Kampman KM, Alterman AI, Volpicelli JR, Maany I, Muller ES, Luce DD, Mulholland EM, Jawad AF, Parikh GA, Mulvaney FD, Weinrieb RM, O'Brien CP. Psychology of Addictive Behaviors. 2001;15:52–59. doi: 10.1037/0893-164x.15.1.52. [DOI] [PubMed] [Google Scholar]
  34. Kelley K, Maxwell SE. Psychological Methods. 2003;8:305–321. doi: 10.1037/1082-989X.8.3.305. [DOI] [PubMed] [Google Scholar]
  35. Kleinman PH, Kang SY, Lipton DS, Woody GE, Kemp J, Millman RB. American Journal of Drug and Alcohol Abuse. 1992;18:29–43. doi: 10.3109/00952999209001609. [DOI] [PubMed] [Google Scholar]
  36. Lavori PW, Bloch DA, Bridge PT, Leiderman DB, LoCastro JS, Somoza E. Journal of Clinical Psychopharmacology. 1999;19:246–256. doi: 10.1097/00004714-199906000-00008. [DOI] [PubMed] [Google Scholar]
  37. Levin FR, Evans SM, Vosburg SK, Horton T, Brooks D, Ng J. Addict Behav. 2004;29:1875–1882. doi: 10.1016/j.addbeh.2004.03.041. [DOI] [PubMed] [Google Scholar]
  38. McConnaughy EI, Prochaska JO, Velicer WF. Psychotherapy: Theory, Research, and Practice. 1983;20:368–375. [Google Scholar]
  39. McKay JR, Alterman AI, Cacciola JS, Mulvaney FD, O'Brien CP. J Nerv Ment Dis. 2000;188:287–296. doi: 10.1097/00005053-200005000-00006. [DOI] [PubMed] [Google Scholar]
  40. McLellan AT, Kushner H, Metzger D, Peters R, Smith I, Grissom G, Pettinati H, Argeriou M. J Subst Abuse Treat. 1992;9:199–213. doi: 10.1016/0740-5472(92)90062-s. [DOI] [PubMed] [Google Scholar]
  41. McMahon RC, Kouzekanani K, Malow RM. Journal of Substance Abuse Treatment. 1999;16:17–22. doi: 10.1016/s0740-5472(97)00317-6. [DOI] [PubMed] [Google Scholar]
  42. Means LB, Small M, Capone DM, Capone TJ, Condren R, Peterson M, Hayward B. Int J Addict. 1989;24:765–783. doi: 10.3109/10826088909047312. [DOI] [PubMed] [Google Scholar]
  43. Milligan CO, Nich C, Carroll KM. Psychiatric Services. 2004;55:167–173. doi: 10.1176/appi.ps.55.2.167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Moeller FG, Dougherty DM, Barratt ES, Schmitz JM, Swann AC, Grabowski J. J Subst Abuse Treat. 2001;21:193–198. doi: 10.1016/s0740-5472(01)00202-1. [DOI] [PubMed] [Google Scholar]
  45. Montoya ID, Levin FR, Fudala PJ, Gorelick DA. Drug Alcohol Depend. 1995;38:213–219. doi: 10.1016/0376-8716(95)01101-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Moye LA. Statistical Reasoning in Medicine: The Intuitive P-Value Primer. New York: Springer-Verlag; 2000. [Google Scholar]
  47. Mulvaney FD, Alterman AI, Boardman CR, Kampman K. Journal of Substance Abuse Treatment. 1999;16:129–135. doi: 10.1016/s0740-5472(98)00017-8. [DOI] [PubMed] [Google Scholar]
  48. Patkar AA, Murray HW, Mannelli P, Gottheil E, Weinstein SP, Vergare MJ. Journal of Addictive Diseases. 2004;23:109–122. doi: 10.1300/J069v23n02_08. [DOI] [PubMed] [Google Scholar]
  49. Pena JM, Franklin RR, Rice JC, Foulks EF, Bland IJ, Shervington D, James A. American Journal on Addictions. 1999;8:319–331. doi: 10.1080/105504999305721. [DOI] [PubMed] [Google Scholar]
  50. Petry NM. Drug Alcohol Depend. 2000;58:9–25. doi: 10.1016/s0376-8716(99)00071-x. [DOI] [PubMed] [Google Scholar]
  51. Preston KL, Silverman K, Schuster CR, Cone EJ. Addiction. 1997;92:717–727. [PubMed] [Google Scholar]
  52. Roberts AC, Nishimoto RH. American Journal of Drug and Alcohol Abuse. 1996;22:313–333. doi: 10.3109/00952999609001662. [DOI] [PubMed] [Google Scholar]
  53. SAS. Cary, NC: SAS Institute Inc.; 2004. [Google Scholar]
  54. Satel SL, Kosten TR. Journal of Nervous and Mental Disease. 1991;179:89–96. doi: 10.1097/00005053-199102000-00005. [DOI] [PubMed] [Google Scholar]
  55. Sauerbrei W, Schumacher M. Stat Med. 1992;11:2093–2109. doi: 10.1002/sim.4780111607. [DOI] [PubMed] [Google Scholar]
  56. Sayre SL, Schmitz JM, Stotts AL, Averill PM, Rhoades HM, Grabowski JJ. Am J Drug Alcohol Abuse. 2002;28:55–72. doi: 10.1081/ada-120001281. [DOI] [PubMed] [Google Scholar]
  57. Schmitz JM, Stotts AL, Rhoades HM, Grabowski J. Addict Behav. 2001;26:167–180. doi: 10.1016/s0306-4603(00)00098-8. [DOI] [PubMed] [Google Scholar]
  58. Schumacher M, Hollander N, Sauerbrei W. Stat Med. 1997;16:2813–2827. doi: 10.1002/(sici)1097-0258(19971230)16:24<2813::aid-sim701>3.0.co;2-z. [DOI] [PubMed] [Google Scholar]
  59. Simpson DD, Joe GW, Broome KM. Archives of General Psychiatry. 2002;59:538–544. doi: 10.1001/archpsyc.59.6.538. [DOI] [PubMed] [Google Scholar]
  60. Siqueland L, Crits-Christoph P, Frank A, Daley D, Weiss R, Chittams J, Blaine J, Luborsky L. Drug and Alcohol Dependence. 1998;52:1–13. doi: 10.1016/s0376-8716(98)00039-8. [DOI] [PubMed] [Google Scholar]
  61. Siqueland L, Crits-Christoph P, Gallop R, Barber JP, Griffin ML, Thase ME, Daley D, Frank A, Gastfriend DR, Blaine J, Connolly MB, Gladis M. American Journal on Addictions. 2002;11:24–40. doi: 10.1080/10550490252801611. [DOI] [PubMed] [Google Scholar]
  62. Stark MJ. Clinical Psychology Review. 1992;12:93–116. [Google Scholar]
  63. Sterling RC, Gottheil E, Weinstein SP, Serota R. Addiction. 2001;96:1015–1022. doi: 10.1046/j.1360-0443.2001.967101511.x. [DOI] [PubMed] [Google Scholar]
  64. Steyerberg EW, Eijkemans MJ, Habbema JD. J Clin Epidemiol. 1999;52:935–942. doi: 10.1016/s0895-4356(99)00103-1. [DOI] [PubMed] [Google Scholar]
  65. Steyerberg EW, Eijkemans MJC, Harrell FE, Habbema JDF. Medical Decision Making. 2001;21:45–56. doi: 10.1177/0272989X0102100106. [DOI] [PubMed] [Google Scholar]
  66. Stotts AL, Schmitz JM, Rhoades HM, Grabowski J. Journal of Consulting and Clinical Psychology. 2001;69:858–862. doi: 10.1037//0022-006x.69.5.858. [DOI] [PubMed] [Google Scholar]
  67. Streiner DL. Can J Psychiatry. 1994;39:191–196. doi: 10.1177/070674379403900401. [DOI] [PubMed] [Google Scholar]
  68. Walitzer KS, Dermen KH, Connors GJ. Behav Modif. 1999;23:129–151. doi: 10.1177/0145445599231006. [DOI] [PubMed] [Google Scholar]
  69. Washton AM. Hazelden Educational Materials. Center City, MN: 1990. [Google Scholar]
  70. Zweben A, Li S. Am J Drug Alcohol Abuse. 1981;8:171–183. doi: 10.3109/00952998108999123. [DOI] [PubMed] [Google Scholar]

RESOURCES