A Preliminary Study of the Effects of Individual Patient-Level Feedback in Outpatient Substance Abuse Treatment Programs

Abstract

The purpose of the current study is to examine the effects of feedback provided to counselors on the outcomes of patients treated at community-based substance abuse treatment programs. A version of the Outcome Questionnaire (OQ-45), adapted to include drug and alcohol use, was administered to patients (N=304) in three substance abuse treatment clinics. Phase I of the study consisted only of administration of the assessment instruments. Phase II consisted of providing feedback reports to counselors based on the adapted OQ-45 at every treatment session up to session 12. Patients who were found to not be progressing at an expectable rate (i.e., “off-track”) were administered a questionnaire that was used as a second feedback report for counselors. For off-track patients, feedback compared to no feedback led to significant linear reductions in alcohol use throughout treatment and also in OQ-45 total scores and drug use from the point of the second feedback instrument to session 12. The effect for improving mental health functioning was evident at only one of the three clinics. These results suggest that a feedback system adapted to the treatment of substance use problems is a promising approach that should be tested in a larger randomized trial.

1. Introduction

There have been several efforts at developing systems to assess individual patient outcomes and provide feedback to providers in mental health settings based on these outcomes. Lambert et al. (2001, 2001a, 2001b; 2003) have employed a feedback system in which clinicians are provided with data based on patient self-assessments using the Outcome Questionnaire-45 (OQ-45) (Lambert, Lunnen, Umphress, Hansen, & Burlingame, 1994). Four large-scale studies aimed at evaluating the effects of providing research-based feedback on patient progress have been conducted (Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Lambert, Whipple, et al., 2001; Lambert et al., 2002; Whipple et al., 2003). Each of the studies evaluated the effects of providing therapists (and sometimes patients) with feedback on a patient’s improvement through the use of time-series graphs and warnings for patients who were not demonstrating expected treatment responses (signal-alarm cases). Expected treatment response is calculated from regression analyses, derived from large samples, in which expected rate of change over time is calculated for patients at various levels of initial distress/functioning. In these studies, patients not progressing as expected are termed “off-track.” Patients who are progressing as expected are referred to as “on-track.” Across the four studies, the effect sizes (standardized mean differences) for the difference between off-track patients for whom therapists received feedback and treatment as usual controls (no feedback) ranged from 0.34 to 0.92 (Lambert, Harmon, Slade, Whipple, & Hawkins, 2005). The percent of patients who showed reliable or clinically significant change in these studies was also markedly different between feedback (ranging from 35% to 56%) and no feedback (21%) treatment conditions.

Additional studies have examined whether the addition of clinical support tools into the OQ-45 feedback system enhances the effects of providing feedback for off-track patients (Whipple et al., 2003; Harmon et al., 2007; Slade, Lambert, Harmon, Smart, & Bailey, 2008). In these studies, off-track patients completed assessments of the severity of problems in multiple domains, including therapeutic alliance, social support, life events, and motivation. Following these assessments, counselors were able to view patients’ assessment scores and were provided with specific suggestions for improving treatment in order to address specific problem areas. In two investigations, effect sizes for the difference between off-track patients whose therapists received either feedback alone or feedback plus clinical support tools were significant and ranged from .31 (Harmon et al, 2007) to .70 (Whipple et al., 2003). These same investigations found that the effect sizes for the difference between off-track patients whose therapists received feedback plus clinical support tools versus feedback without clinical support tools from .15 (Whipple et al., 2003) to .31 (Harmon et al., 2007), suggesting that the addition of clinical support tools enhanced the effect of feedback alone. In addition, Shimokawa, Lambert, and Smart (2010), using meta/mega-analytic techniques, combined data from the six clinical trials and found that negative change rates were four times less likely when progress feedback and clinical support tools were employed. This finding held up even though treatment-as-usual and feedback conditions were delivered by the same clinicians.

The studies conducted by Lambert and colleagues were all implemented in mental health clinics. In the addiction field, performance improvement feedback methods have been implemented on a clinical basis. These include a Methadone Treatment Quality Assurance System that provided performance improvement reports on a quarterly basis to supervisors in 70 Veterans Affairs (VA) clinics (Ducharme & Luckey, 2000; Phillips et al., 1995) and the Quality Enhancement Research Initiative, which provided performance monitoring, feedback, and dissemination of best practice guidelines to administrators and clinicians in VA settings (Finney, Willenbring, & Moos, 2000). Another outcomes assessment system for use in VA substance abuse clinics has the potential to provide feedback at the program level by collecting baseline and six-month follow-up data (Tiet, Byrnes, Barnett, & Finney, 2006).

There has been little research evaluating feedback systems in a substance abuse treatment context. One research study that examined the impact of providing feedback reports on patient attendance data to clinicians in a substance abuse treatment clinic found improvements in attendance (McCaul & Svikis, 1991). We previously reported on the results of a randomized trial involving 20 substance abuse treatment programs that examined the efficacy of providing caseload (average patient outcomes) feedback to substance abuse counselors (Crits-Christoph et al., 2010). In this study, clinicians compared their average outcomes for all of their patients during a given week (irrespective of how long patients had been in treatment that week) to outcomes for the clinic as a whole that week. Outcome was assessed in terms the therapeutic alliance, treatment satisfaction, and drug/alcohol use. No effects of feedback were found. Clinicians were apparently relatively unmotivated by feedback reports that were oriented towards helping them with patients as a group rather than feedback reports for specific patients. These results led us to reformulate our approach so that patients were tracked weekly beginning at their first counseling session and individual patient feedback was provided to clinicians. In addition, the new approach included tracking patient functioning and symptoms, rather than only the alliance and drug/alcohol use, so that a broader assessment of patient concerns and progress was available to clinicians. We also wanted to direct clinician attention to patients who were not progressing in treatment because such patients likely required consideration of changes in the focus of treatment. To meet these goals, a modified version of the feedback system used in the Lambert and colleagues’ studies was created. No previous studies have evaluated the effects of providing such individual patient outcome feedback in the context of substance use treatment.

The current study, therefore, was designed to evaluate the effects of providing individual patient feedback to substance abuse counselors. Based on the above successful studies by Lambert and colleagues with mental health populations, and the failure of our previous counselor-level feedback system, we conducted a preliminary study of an adapted OQ-45 feedback system for substance abuse treatment. Feedback reports based on the adapted OQ-45 and clinical support tools for off-track patients were evaluated. Our hypotheses were that, compared to no feedback, (1) receiving regular treatment progress feedback would result in improvement in OQ-45 scores and drug/alcohol use for off-track patients, and (2) provision of clinical support tools for off-track patients would lead to relatively greater patient improvements over time.

2. Materials and Methods

2.1 Study Design

The study consisted of two phases. In the first phase, patients seeking outpatient treatment for alcohol or drug use problems received assessments only using the modified OQ-45 instrument which was administered at the beginning of treatment sessions. During the second phase, patients in the same clinics were administered the modified OQ-45 at treatment sessions, and the counselors for these patients received immediate feedback reports based on the patient OQ-45 assessment. In addition to feedback, counselors were provided clinical support tools in Phase II if a patient became off-track. The study protocol was approved by Institutional Review Boards affiliated with the participating clinics and Universities.

2.2 Participants

2.2.1 Clinics

Three community-based outpatient substance abuse treatment clinics (non-methadone maintenance), one each in the New York City, Philadelphia, and Salt Lake City, participated.

2.2.2 Clinicians

To be eligible to participate in the study, clinicians at the participating clinics needed to be treating adult (age 18 or above) clients with individual counseling, provide written informed consent, and work a minimum of 20% of time at the clinic.

2.2.3 Patients

Eligible patients (aged 18 or above) were those seeking a new course of individual outpatient treatment (within two weeks of their intake session). In addition, only patients whose individual counseling appointments were scheduled with a minimum frequency of twice monthly were eligible. All participating patients provided written informed consent.

2.3 Assessments

2.3.1 Modified Outcome Questionnaire-45 (OQ-45)

Patient progress in this study was tracked using the Outcome Questionnaire (OQ-45), a 45-item self-report scale developed specifically for the purpose of tracking and assessing patient outcomes in a therapeutic setting. The OQ-45 is scored using a 5-point scale (0=never, 1=rarely, 2=sometimes, 3=frequently, 4=almost always), which yields a possible range of scores from 0 to 180. High scores on the OQ-45 indicate greater levels of symptoms and/or poorer functioning. The primary modification of the OQ-45 for the current study was to add two items measuring the number of days in the past week that the patients used (1) alcohol and (2) drugs. The total score, which provides a global assessment of functioning, and the individual alcohol and drug use items were used in this study. The OQ-45 also has three subscales that measure quality of interpersonal relations, social role functioning, and symptom distress; however, for the purpose of the current study only the total score was analyzed.

The OQ-45 is a well-established instrument that has been validated across a broad range of normal and client populations. Lambert et al. (2004) reported an internal consistency reliability (Cronbach’s alpha) for the OQ-45 of .93 and a 3-week test-retest reliability value of. 84 for the OQ-45 total score. Concurrent validity of the OQ-45 total score has been examined by correlating it with the Symptom Checklist-90 (SCL-90; Derogatis, 1977), Beck Depression Inventory (BDI; Beck, Steer, & Garbin, 1988), Zung Depression Scale (Zung, 1965), and the State-Trait Anxiety Inventory (STAI; Spielberger, 1983). All of the concurrent validity figures with the OQ-45 and these instruments were significant at the .01 level with a range of r’s from. 50 to .85 (Lambert et al., 2004). Most important, the OQ-45 has been shown to be sensitive to the effects of interventions on patient functioning (Vermeersch, Lambert, & Burlingame, 2000; Vermeersch, Whipple, Lambert, Hawkins, Burchfield, & Okiishi, 2004).

The OQ-45 was administered immediately prior to every treatment session in both Phase I and Phase II up to a maximum of 12 post-baseline treatment sessions or six months (whichever came first). All OQ-45 data was collected electronically using PDAs or computer kiosks. Patients were handed the PDAs (or instructed to go to computer kiosk) by clinic receptionists, research assistants, or counselors to complete the OQ-45. Data was captured electronically and sent (via an encrypted wireless connection) to a computer and stored in a database. In Phase II, reports were viewed electronically, via a secure, password-protected computer system, and were available immediately after the patient completed measures via PDA or computer kiosk. Feedback reports based on OQ scores were the same reports as those used in previous studies (Slade et al., 2008), with the exception that patient scores on the alcohol and drug use questions were added to the reports and tracked over time so that increasing, or decreasing, drug/alcohol use would be evident to the counselor.

2.3.2 Assessment for Signal Clients (ASC)

During Phase II, when patients were identified as being off-track based upon their scores on the OQ-45, therapists were given the opportunity to use clinical support tools which provided suggestions about ways to improve treatment. Upon being identified as off-track, patients were asked to complete the Assessment for Signal Clients (ASC; Lambert, et al., 2007), a 40-item self-report measure designed to assess the severity of problems that may be impeding treatment progress - specifically, problems with therapeutic alliance, motivation, social support, and stressful life events. The ASC uses a 5-point scale (0=strongly disagree, 1= slightly disagree, 2=neutral, 3=slightly agree, and 4=strongly agree).

After the patient completes the ASC, a feedback report is made available to therapist. Therapists are trained to examine the report, noting the domains and the specific items of concern. The web-based feedback report contains a link to a list of interventions that target each problem domain identified by the ASC. In addition, therapists are instructed to refer to the ASC manual that provides a decision tree to facilitate clinical decisions. The decision tree enables the therapist to walk through the domain(s) of concern, and it directs the therapist to specific therapeutic intervention suggestions to be considered and utilized. Each of the four domains and its relevant clinical attributes are described in detail with concern to how those attributes may have an impact on patient symptomatology, clinical presentation, treatment prognosis, and therapeutic relationship. In addition to providing specific suggestions for clinical consideration, the manual also refers therapists to additional literature for further edification.

Some specific examples of treatment interventions for the Therapeutic Alliance domain are: Work with resistance by retreating when necessary and being supportive; Allow the client to assert their negative feelings about the relationship; Give and ask for feedback on the therapeutic relationship. Examples of interventions for the Social Support domain are: Assess client’s social network; Utilize progressive desensitization to aid client in overcoming social anxieties; Encourage pursuit of hobbies, participation in clubs, service projects. Examples of interventions for the Motivation domain are: Assess how important it is for the client to make target changes, having them rate the desire to make specific changes on a scale of 1-10; Discuss the client’s important values, goals, and aspirations; Ask open-ended questions to see if the client is ready to make a commitment to change. For the fourth domain, Life Events, the manual instructs the therapist to explore the stressful event and to facilitate the development of coping strategies for the patient to help the patient establish stability.

If it appears that difficulty within the four domains does not adequately address the patient’s treatment failure (i.e., going “off-track”) therapists are trained to reassess the patient’s diagnostic formulation, to consider whether an alternate kind of treatment could be more effective, and/or whether a referral for a medication consultation is an appropriate avenue to pursue.

2.4 Procedures

Counselors were given an orientation to the study and trained in the interpretation of feedback reports. In addition to providing a graph of OQ-45 total scores and a table of drug and alcohol use scores for each completed OQ, the feedback reports contained a color-coded feedback statement. A feedback report with alert status designated as “white” declared: “The client is functioning in the normal range. Consider termination.” Green feedback stated: “The rate of change the client is making is in the adequate range. No change in the treatment plan is recommended.” Yellow feedback included the statement: “The rate of change the client is making is less than adequate. Recommendations: Consider altering the treatment plan by intensifying treatment, shifting intervention strategies, and monitoring progress especially carefully. This client may end up with no significant benefit from therapy.” A red feedback report alerted the clinician: “The client is not making the expected level of progress. Chances are that he or she may drop out of treatment prematurely or have a negative treatment outcome. Steps should be taken to carefully review this case and decide on a new course of action.” For data analysis purposes, all patients receiving a red or yellow message were considered off-track.

If a patient was off-track based on his rate of improvement in OQ-45 scores, the ASC was administered using PDAs or computer kiosks. Patients were handed the PDAs (or instructed to go to the computer kiosk) by clinic receptionists, research assistants, or their counselors. Once the data was captured electronically, it was sent via an encrypted wireless connection to a computer and stored in a database. The ASC was administered only once during Phase II of the study, at the time of the first occurrence of the patient being off-track.

2.5 Statistical Analysis

The initial preliminary analysis compared the Phase I and Phase II cohorts on baseline variables and patient characteristics. Primary outcome analyses included all patients who had at least a baseline and one treatment session’s OQ-45 assessment. The primary analysis tested whether feedback (Phase II) results in a significantly greater rate of change in OQ-45 total scores than no feedback (Phase I) for off-track patients. This involved a longitudinal analysis of all OQ-45 data collected from baseline through session 12. The longitudinal analysis used a random effects multilevel regression model (Goldstein, 1995) with a random slope and random intercept for each subject. Due to non-linear change of individual patient profiles over time, a logarithmic transformation of time was performed. To test the main hypotheses, the weighted averages of subject-specific slope estimates were compared between the feedback and no-feedback groups (Phase I and II) for off-track patients to identify the group with the most rapid improvement. Data from all subjects were used, with a specific contrast examined for the subgroup of patients who were off-track. An additional analysis specifically examined the impact of the clinical support tools (ASC). For these analyses, only data from the time the ASC was administered (when the patient became off-track) and subsequent treatment sessions were used. We again used a random effects multilevel regression model and compared the rates of change for the feedback and no feedback off-track patients using the data subsequent to the administration of the ASC. Further analyses examined change in OQ-45 total scores from baseline to endpoint (last available session up to session 12) using analysis of covariance (baseline value as covariate). Drug and alcohol use and the subscales of the OQ-45 were analyzed in the same way as OQ-45 total scores. We used PROC MIXED in SAS (Littell et al., 2006) to conduct the random effects hierarchical regression analyses.

Effect sizes (converted to Cohen’s d), derived from the F-test for mixed effects model, were calculated as $d=2\sqrt{\frac{F}{df}}$, where F is the F-test statistic for the effect of interest in the multilevel design (Rosenthal & Rosnow, 1991; Verbeke & Molenberg, 2000).

3. Results

3.1 Baseline Characteristics of Sample

3.1.1 Patients

Across sites, a total of 304 patients provided intake data and at least one post-baseline assessment in either Phase I or Phase II of the study (142 in Pennsylvania; 107 in Utah; 53 in one NY clinic, 2 patients in a second NY clinic). Of the 304 patients who had at least one post-baseline assessment, there were 165 in Phase I and 139 in Phase II. Of those in Phase I, 62 (37.6%) were identified as off-track at some point in their treatment. Of those in Phase II, 54 (38.8%) were off-track.

Overall, the sample of patients (combining Phase I and Phase II) that provided a baseline assessment and at least one post-baseline assessment was 56% male and 37% Caucasian. The typical patient reported that alcohol had been a problem for about 11 years, while drug use had been a problem for about 13 years. Over 80% of patients reported no drug use at baseline and over 75% reported no alcohol use at baseline. There were no significant differences between the Phase I and Phase II samples, or significant differences between the on-track and off-track patients within each phase, on any demographic or descriptive variables (Table 1).

Table 1. Characteristics of Patients.

Characteristic	No Feedback (Phase I) N=165		Feedback (Phase II) N=139
	On-track N=103	Off-track N=62	On-track N=85	Off-track N=54
Gender (% men)	58.8% (60/102)	64.5% (40/62)	52.7% (39/74)	62.0% (31/50)
Age, M (SD)	38.8 (11.4)	40.2 (10.3)	38.9 (10.2)	40.3 (9.4)
Race
White	43.0% (43/100)	28.8% (17/59)	41.5% (34/82)	31.5% (17/54)
Black	37.0% (37/100)	45.8% (27/59)	40.2% (33/82)	51.9% (28/54)
Latino	14.0% (14/100)	18.6% (11/59)	14.6% (12/82)	7.4% (4/54)
Other	6.0% (6/100)	6.8% (4/59)	3.7% (3/82)	9.3% (5/54)
Marital Status (Single)	66.3% (67/101)	70.5% (43/61)	62.2% (51/82)	64.2% (34/53)
Years of School (Less than H.S.)	33.0% (33/100)	33.9% (20/59)	23.2% (19/82)	30.8% (16/52)
Employment Status (Unemployed)	76.5% (75/98)	72.4% (42/58)	73.2% (60/82)	85.2% (46/54)
Duration of Alcohol as problem, M (SD)	9.8 (11.3)	11.7 (12.6)	10.2 (11.3)	11.9 (11.9)
Duration of drugs as problem, M (SD)	13.0 (10.8)	12.0 (10.3)	11.8 (10.9)	13.8 (12.0)

In terms of baseline values on the outcome measures (see baseline, Table 2), there was a significant difference in OQ-45 total scores between the on-track and off-track patients during Phase I, with off-track patients reporting greater symptoms on the OQ-45 than the on-track patients (t [163] = −2.30, p = .02). On-track and off-track patients did not differ significantly on alcohol or drug use at baseline. There were also no significant differences between the off-track patients in Phase I and the off-track patients in Phase II on any of the outcome measures at baseline.

Table 2. Descriptive Data for Alcohol Use, Drug Use, and OQ-45 Total Score at Each Treatment Session.

	Alcohol Use				Drug Use				OQ-45
	No Feedback		Feedback		No Feedback		Feedback		No Feedback		Feedback
Session	On-track	Off-track	On-track	Off-track	On-track	Off-track	On-track	Off-track	On-track	Off-track	On-track	Off-track
0	0.33 (0.88) 103	0.52 (1.57) 62	0.51 (1.16) 85	0.85 (1.75) 54	0.58 (1.45) 103	0.26 (1.04) 62	0.38 (1.31) 85	0.61 (1.34) 54	66.6 (31.2) 103	77.1 (22.7) 62	68.8 (29.8) 85	68.3 (26.1) 54
1	0.32 (0.97) 103	0.41 (1.19) 61	0.45 (1.04) 85	0.50 (1.06) 54	0.39 (1.25) 103	0.21 (0.66) 61	0.28 (1.12) 85	0.41 (1.00) 54	61.5 (28.0) 103	83.6 (23.9) 61	63.9 (26.3) 85	75.9 (25.7) 54
2	0.29 (1.15) 90	0.53 (1.40) 62	0.35 (1.15) 72	0.63 (1.50) 85	0.10 (0.50) 90	0.24 (1.02) 62	0.32 (1.36) 72	0.38 (1.09) 52	59.8 (27.2) 90	83.4 (22.8) 62	60.4 (25.4) 72	76.6 (26.7) 52
3	0.12 (0.56) 81	0.49 (1.34) 59	0.10 (0.55) 68	0.46 (1.18) 50	0.10 (0.44) 81	0.17 (0.50) 59	0.18 (0.90) 68	0.34 (0.96) 50	54.6 (26.4) 81	85.8 (20.5) 59	57.9 (27.0) 68	74.4 (27.2) 50
4	0.13 (0.84) 72	0.69 (1.66) 51	0.07 (0.32) 57	0.30 (0.73) 46	0.17 (0.95) 72	0.37 (1.17) 51	0.05 (0.40) 57	0.15 (0.47) 46	54.1 (25.1) 72	83.3 (23.2) 51	53.6 (23.7) 57	74.9 (25.3) 46
5	0.03 (0.17) 66	0.91 (1.96) 46	0.21 (1.02) 52	0.20 (0.51) 44	0.05 (0.21) 66	0.33 (1.16) 46	0.29 (1.36) 52	0.27 (1.11) 44	53.9 (26.8) 66	83.9 (23.5) 46	49.8 (23.1) 52	74.5 (25.7) 44
6	0.03 (0.18) 62	0.48 (1.35) 42	0.07 (0.26) 43	0.23 (0.66) 40	0.02 (0.13) 62	0.05 (0.31) 42	0.16 (0.69) 43	0.40 (1.28) 40	54.7 (24.3) 62	84.7 (19.3) 42	52.1 (23.1) 43	77.6 (24.5) 40
7	0.12 (0.49) 60	0.39 (1.20) 41	0.38 (1.27) 40	0.35 (1.23) 37	0.08 (0.46) 60	0.24 (0.66) 41	0.30 (1.22) 40	0.14 (.48) 37	55.0 (22.8) 60	86.7 (21.9) 41	52.9 (23.7) 40	71.8 (22.6) 37
8	0.04 (0.20) 51	0.35 (0.77) 34	0.13 (0.42) 32	0.21 (0.64) 34	0.14 (0.98) 51	0.32 (1.09) 34	0.19 (1.06) 32	0.32 (1.30) 34	55.1 (25.0) 51	88.0 (21.2) 34	52.3 (27.7) 32	75.4 (25.9) 34
9	0.05 (0.21) 44	0.18 (0.64) 33	0.13 (0.43) 31	0.10 (0.41) 29	0.00 (0.00) 44	0.18 (0.77) 33	0.29 (1.30) 31	0.31 (1.34) 29	54.3 (23.5) 44	87.6 (23.5) 33	52.2 (26.4) 31	74.4 (25.4) 29
10	0.00 (0.00) 38	0.28 (0.75) 29	0.08 (0.40) 25	0.08 (0.27) 26	0.00 (0.00) 38	0.00 (0.00) 29	0.28 (1.40) 25	0.00 (0.00) 26	56.0 (22.6) 38	87.8 (21.8) 29	49.2 (21.9) 25	70.7 (28.2) 26
11	0.03 (0.17) 33	0.59 (1.62) 27	0.08 (0.41) 24	0.04 (0.20) 25	0.00 (0.00) 33	0.37 (1.24) 27	0.29 (1.43) 24	0.00 (0.00) 25	54.6 (22.3) 33	86.9 (29.2) 27	46.4 (23.8) 24	71.2 (28.3) 25
12	0.00 (0.00) 24	0.37 (1.01) 19	0.05 (0.21) 22	0.14 (0.65) 21	0.00 (0.00) 24	0.00 (0.00) 19	0.23 (1.07) 22	0.00 (0.00) 21	51.7 (24.5) 24	85.5 (25.8) 19	47.1 (23.3) 22	73.3 (25.0) 21

Note. Mean (SD), N given. Session 0 = baseline visit.

3.1.2 Clinicians

Across the three sites, there were a total of 38 clinicians who had at least one patient that completed one or more post baseline assessments. Of these clinicians, 60% were female, 29% were African American, and 8% identified as Hispanic or Latino. The mean age of participating clinicians was 44 years old. In terms of education, 61% of the clinicians had a Masters Degree, 18% had a Bachelor’s Degree, and 16% had a Doctoral Degree. Fifty-five percent reported having five or more years of experience as a counselor.

3.2 Effects of Feedback

3.2.1 Analysis of Baseline to Session 12

Mean (SD) values for alcohol use, drug use, and OQ-45 Total Scores over the 12 treatment sessions are given in Table 2; estimated trend lines from the longitudinal multilevel regression analyses are provided in Figure 1. For alcohol use, there was a significant (t [301] = 2.3, p = .023; d = .26) difference in slopes between the off-track patients in Phase I compared to the off-track patients in Phase II. Although the off-track patients in Phase II (feedback) began treatment at somewhat higher levels of alcohol use, they improved substantially over time and by the end of 12 sessions were using alcohol at a rate similar to the on-track patients in Phase I and Phase II. In contrast, the off-track patients in Phase I (no feedback) showed very little improvement in alcohol use over 12 treatment sessions. The longitudinal analysis of drug use yielded similar results, though the comparison of off-track patients in Phase I and Phase II did not reach significance (t [301] = 1.8, p = .076; d =.20). Analysis comparing the slopes of change on OQ-45 total scores for off-track patients in Phase I versus off-track patients in Phase II showed no significant difference (t [301] = 1.1, p = .27; d =. 13).

Figure 1.

Estimated trend lines for OQ-45 Total Scores, Alcohol Use, and Drug Use over Treatment Sessions.

3.2.2. Analysis from the Point of Going Off-Track to Session 12

At the session at which patients went off-track, there were no significant differences between the feedback and no-feedback groups (OQ-45: t [114] = 1.1, p = .27; alcohol use: t [111] = .62, p = .54; drug use: t [114] = −.36, p = .72). However, from the off-track point onwards to session 12, there were significant differences in rates of improvement between the feedback and no feedback groups for OQ-45 total scores (t [112]= 2.54, p = .013, d = .48) and drug use (t [112] = 1.99, p = .049, d =. 38), but not alcohol use (t [112] = −.08, p = .94, d = .02). Estimated trend lines for the alcohol use, drug use, and OQ-45 total scores, beginning with the session patients were identified as off-track, are provided in Figure 2.

Figure 2.

Estimated trend lines for OQ-45 Total Scores, Alcohol Use, and Drug Use From the Off-Track Session to Session 12.

3.2.3 Site Interactions

Across all assessments from baseline to session 12, there were no significant site by feedback (Phase I vs. Phase II) interactions for the off-track patients on the longitudinal (slope) analyses (alcohol: F [2, 301] = .21, p = .81); drug: F [2, 301] = .34, p = .71; F [2, 301] = 1.84, p = .16). Using only the data from the off-track point to session 12, there was a significant site by treatment interaction for the OQ-45 total score (F [2, 109] = 4.98, p = .0073, eta-squared=0.05). This interaction was a function of a clear effect for clinical support tools feedback on OQ-45 total scores at the Salt Lake City site (t [109] = 4.02, p < .0001; d = .78), but no evidence of an effect at the Philadelphia (t [109] = .28, p = .78; d = .05) and New York City (t [109] = .21, p = .83; d = .04) sites. There were no significant site interactions in regard to rates of improvement from the point the patient went “off-track” until session 12 for alcohol use (F [2, 109] = 2.1, p = .12) or drug use (F [2, 109] = .52, p = .60).

3.2.4 Retention in Treatment

By session 12, about 35% of patients were still in treatment. There were no significant differences in retention rate between the feedback and no feedback “off-track” groups. The median (M, SD) length of treatment for each group was: on-track, no feedback: 7.0 (4.10); on-track, feedback: 6.0 (4.11); off-track, no feedback: 9.0 (3.64); off-track, feedback: 9.0 (3.68).

4. Discussion

The results of this study revealed that, for patients who were not progressing as expected (“off-track”), feedback to substance abuse counselors led to superior treatment outcomes compared to no feedback. The effects of feedback were evident on general psychiatric symptoms/functioning (OQ-45 scores) and alcohol and drug use, though findings on these measures emerge at different points in the feedback process. The every-session feedback reports that contained information on OQ-45 total scores and drug/alcohol use had a positive impact (compared to no feedback) over time (slope) on alcohol use and at endpoint on the OQ-45. Without feedback to their counselors, such off-track patients show little average improvement in alcohol use over time, while patients of counselors who received feedback improved to a level of alcohol use similar to patients who were on-track. Thus, feedback appeared to successfully bring the off-track patients back on-track.

In contrast to alcohol use, linear improvement (slopes) in OQ-45 scores and drug use were not evident for the off-track feedback patients until the point where they were identified as off-track. Once a patient was identified as off-track, the ASC was administered and clinical support tools based on personalized information from the ASC were provided to counselors. It seems reasonable that a consistent linear effect for feedback was seen only after a counselor was made aware of the fact that a given patient was off-track in regard to OQ-45 scores. It may likely take the large red ALERT that characterizes an off-track feedback report to get the attention of a substance abuse counselor in regard to psychiatric symptoms/functioning, as opposed to substance use. Moreover, the clinical support tools contained in the report generated from the ASC may provide novel suggestions for substance abuse counselors, allowing these counselors to turn their focus to issues such as the quality of the therapeutic alliance, social supports, consideration of psychiatric medications, and attention to life stresses, rather than only focusing on substance use. This feedback effect for clinical support tools, however, was only apparent for OQ-45 total score outcomes at the Salt Lake City. Because the OQ-45 with clinical support tools was in use at the Utah clinic before the implementation of the current study, the counselors at that clinic, compared to the Philadelphia and New York City clinics, were more familiar with the clinical support tools feedback report. This greater familiarity with- and confidence in-the clinical support tools feedback report may have facilitated the implementation of successful strategies to improve OQ-45 outcomes at the Salt Lake City site. This site effect for the OQ-45 suggests that more extensive experience with the use of the clinical support tools reports may improve the impact of these feedback reports. However, no treatment by site interaction was evident on the drug use outcome measure; counselors at all sites were able to use the clinical support tools feedback to improve drug use outcomes.

One important finding of this study was that a large percent (38.2%) of patients in the participating substance abuse treatment facilities were identified as being off-track at some time during their course of treatment. In mental health samples, the percent of off-track patients has typically ranged from 15 to 40% with the higher rates associated with clinics treating more seriously disturbed patients (Hawkins, et al., 2004). This larger number of off-track patients in substance abuse clinics may indicate that psychiatric symptoms often are neither identified nor addressed as a focus of treatment within substance abuse treatment clinics. Alternatively, it may be that psychiatric symptoms and overall functioning fluctuate within a substance abusing population in relation to relapses and binges with drug/alcohol alternating with periods of abstinence. Such large fluctuations in symptoms/functioning are likely to trigger an off-track alert in the OQ-45 system, as normative (on-track) progress is defined as a linear improvement in symptoms/functioning over time. Regardless of the cause of these common off-track cases, it appears useful that substance abuse counselors have a mechanism for regularly tracking psychiatric symptoms and current functioning, and the OQ-45 feedback system provides such a mechanism.

Although improvement in alcohol use and drug use were evident, the sample was characterized by low levels of self-reported use, with 80% of patients reporting no drug use at baseline and over 75% reported no alcohol use at baseline. This low level of use is consistent with data reported on 20 substance abuse treatment programs (Crits-Christoph et al., 2010). Some patients are mandated to drug treatment after a legal problem or a citation for driving under the influence charge and are already alcohol/drug-free before starting treatment. In addition, many patients can readily stop using alcohol/drugs for brief periods. Treatment is then oriented towards sustaining recovery over time rather than reducing heavy usage. Thus, the pattern of alcohol/drug use over time rather than usage at a given point in time (e.g., termination of treatment) is the most relevant outcome of substance abuse treatment. The episodic, relapsing nature of drug/alcohol addiction likely explains the impact of feedback on endpoint OQ-45 total scores: feedback, compared to no feedback, was associated with less deterioration from baseline.

This project was a preliminary study and consequently there are a number of limitations to it. Foremost is that the study did not randomly assign patients to feedback or no feedback conditions and instead relied on a Phase I (no feedback) and Phase II (feedback design) at the participating clinics. The patient population may have changed over time (though no differences on measured variables were evident), confounding interpretations of comparisons between Phase I and Phase II off-track patients. Another limitation of this study is the lack of diagnostic information on patients. This makes it difficult to know the types of patients that might benefit from use of an outcomes assessment/feedback system. A further limitation is that no information was collected on the types of interventions that were implemented by clinicians in response to feedback reports. Thus, the study provides no information on how feedback achieved relatively better outcomes. Future research will need to address the mechanism of feedback effects.

Despite these limitations, the current study suggests that use of outcome assessment/feedback systems may be of value for enhancing treatment outcomes within substance abuse treatment clinics. Further research with larger sample sizes and employing random assignment is needed to confirm the preliminary findings reported here.