Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2008 Dec 1.
Published in final edited form as: J Subst Abuse Treat. 2007 May 14;33(4):363–371. doi: 10.1016/j.jsat.2007.01.007

A Feasibility Study of a Web Based Performance Improvement System for Substance Abuse Treatment Providers

Robert Forman a,b, Paul Crits-Christoph b,*, Övgü Kaynak b, Matt Worley b, Donald A Hantula c, Agatha Kulaga d, John Rotrosen d, Melissa Chu d, Robert Gallop f, Jennifer Potter g, Patrice Muchowski h, Kirk Brower i, Stephen Strobbe i, Kathy Magruder j, A’Delle H Chellis k, Tad Clodfelter l, Margaret Cawley m
PMCID: PMC2111171  NIHMSID: NIHMS34641  PMID: 17499954

Abstract

We report here on the feasibility of implementing a semi-automated performance improvement system - Patient Feedback (PF) - that enables real-time monitoring of patient ratings of therapeutic alliance, treatment satisfaction, and drug/alcohol use in outpatient substance abuse treatment clinics. The study was conducted in 6 clinics within the National Institute on Drug Abuse Clinical Trials Network. It involved a total of thirty-nine clinicians and 6 clinic supervisors. Throughout the course of the study (4 week training period, 4 week baseline, 12 week intervention, 4 week post-intervention assessment, 1 year sustainability phase) there was an overall collection rate of 75.5% of the clinic patient census. In general, the clinicians in these clinics had very positive treatment satisfaction and alliance ratings throughout the study. However, one clinic had worse drug use scores at baseline than other participating clinics, and showed a decrease in self-reported drug use at post-intervention. Although the implementation of the PF system proved to be feasible in actual clinical settings, further modifications of the PF system are needed to enhance any potential clinical usefulness.

Keywords: Performance Improvement, Substance Abuse, Feedback, Alliance, Treatment Satisfaction

1. Introduction

Performance improvement methods, originally developed by industrial psychologists and applied in manufacturing settings, have been increasingly applied to the treatment of mental health and substance use disorders. Performance improvement methods vary and go by different names (e.g. “continuous quality improvement”), but in general employ common methods that include: 1) an organization identifies desired outcomes; 2) objective, measurable indicators reflecting achievement of those outcomes are specified; 3) methods for efficiently measuring indicators are developed; 4) indicator measurement and feedback processes are implemented; 5) team processes are employed to review feedback; 6) action plans, based on available knowledge and evidence are implemented to improve performance; 7) steps 1–6 are repeated with new indicators identified by the organization’s stakeholders as deemed necessary.

In the mental health field, several studies have used patient outcome monitoring with feedback to psychotherapists as a way to attempt improvement of clinical performance (Howard, Brill, Lueger, O’Mahoney, & Grissom, 1995; Howard, Kopta, Krause, & Orlinsky, 1986; Howard, Lueger, Maling, & Martinovich, 1993; Howard, Moras, Brill, Martinovich, & Lutz, 1996; Lueger, Lutz, & Howard, 2000; Lueger et al., 2001; Lambert, Hansen, & Fitch, 2001; Lambert, et al., 2003; Lambert, Whipple, et al., 2001; Cook & Dixon, 2005; Wells et al., 2000).

In regard to addiction treatment, the U.S. Veteran’s Administration (VA) has implemented a Methadone Treatment Quality Assurance System that provided performance improvement reports on a quarterly basis to supervisors in 70 VA clinics (Ducharme & Luckey, 2000; Phillips et al., 1995). More recently, the VA has implemented the Quality Enhancement Research Initiative (QUERI), which provides performance monitoring, feedback, and dissemination of best practice guidelines to administrators and clinicians (Finney, Willenbring, & Moos, 2000; Humphreys, Hamilton, & Moos, 1996; Moos et al.,1998). In another effort, Hermann et al. (2000) described the implementation of a quality improvement system within the Cambridge Behavioral Health System, and reported data from non-randomized evaluations of the system. McCaul and Svikis (1991) demonstrated that feeding back patient attendance data to clinicians in a substance abuse treatment clinic yielded improvements in attendance in a pre/post design; and Andrzejewski, Kirby, Morral, and Iguchi (2001) found that provision of graphed feedback to drug clinicians about their adherence to a research protocol yielded a 71% increase in protocol adherence.

The purpose of the current article is to report on an evaluation of the feasibility of implementing a semi-automated performance improvement system, called Patient Feedback (PF) that enables real-time monitoring of therapeutic alliance and treatment satisfaction by clinicians and supervisors working in outpatient substance abuse treatment clinics. As a potential means to reduce drug and alcohol use, improvements in therapeutic alliance and treatment satisfaction were targeted. Therapeutic alliance was chosen because it consistently predicts the outcome of psychotherapy and counseling (Martin, Garske, & Davis, 2000), and has also been found to predict outcome in substance abuse settings (Gillaspy, Wright, Campbell, Stokes, & Adinoff, 2002). Treatment satisfaction was included because it is commonly an element of quality monitoring systems in addiction treatment programs (National Treatment Center Study, 2005). By monitoring therapeutic alliance and treatment satisfaction, the PF system allows clinicians to assess the interim effectiveness of their efforts and to make modifications if needed (Anderson, Crowell, Hantula, & Siroky, 1988; Deming, 1986; Juran & Godfrey, 1999). Additionally, the regular assessment of these dimensions, in itself, may influence clinician behavior by signaling that alliance and satisfaction are priorities of the organization (Alvero et al., 2001; Berwick et al., 1990; Nicol & Hantula, 2001).

2. Materials and Methods

2.1 Study Design

The study was conducted in 6 community-based outpatient drug-free substance abuse treatment clinics associated with the National Institute on Drug Abuse (NIDA) Clinical Trials Network (CTN). These clinics were connected to CTN research centers at New York University (2 clinics), McLean Hospital/Harvard University, University of Michigan, Medical University of South Carolina, and Duke University. To be included in the study, the clinic had to have at least 4 clinicians who were currently conducting group counseling sessions (at least once a week) and were able to attend a monthly supervision meeting. In addition, clinics needed to have Internet access for their clinicians and supervisors. Data collection for the feasibility phase began in April 2004 and ended in December 2004.

Because this project was a preliminary investigation of the feasibility and impact of using the PF system, all clinics and clinicians received the feedback intervention (i.e., there was no control group). The study consisted of 5 phases: 1) pre-intervention procedures (4 weeks), 2) baseline (4 weeks), 3) intervention (12 weeks), 4) post-intervention assessment (4 weeks), and 5) sustainability (1 year).

2.2 Participants

All patients who were actively participating in group counseling sessions for substance abuse problems at the 6 clinics were eligible to be involved in the study. Thus, the study involved the full range of patients seen in outpatient settings, including those with substance abuse and substance dependence diagnoses (although diagnosis was not assessed in this study). Patients were oriented to the study during the intake process of the clinic. Participation was voluntary. Because the PF survey was anonymous, the study did not track the total number of patients who participated. At each assessment point during the study, the sample of patients within each clinic could be somewhat different from the previous assessment (as former patients ended treatment and new patients were enrolled). The study baseline period included all participating patients currently in treatment at that point in time. Thus, patients were not necessarily assessed at the point they began their course of treatment.

All clinicians and supervisors working in the 6 clinics were eligible for study participation, with the exception of any clinician who did not conduct group counseling sessions on a weekly basis. Because the clinicians were considered the human subjects in this study, they gave written consent to participate (patients were given an orientation to the study and its procedures, but because the patient assessment was anonymous and free of risk, patients were not consented). The study and the clinician consent form were approved by the local Institutional Review Board’s for each participating clinic.

2.3 Patient Feedback (PF) System

The PF survey is a 1 page, 12 survey item questionnaire with optical mark recognition and instructions to the respondent. The PF survey was developed in accordance with quality improvement indicator development guidelines (JCAHO, 1998; Meyer, 1994) and with input from 22 clinic supervisors and 18 researchers. Beginning with a pool of potential items, PF survey development team members ultimately selected 12 items for inclusion in the PF survey. Selected items met two quality improvement criteria: 1) the item monitored something staff could improve and, 2) improvement in the item’s domain was thought to be associated with increases in outcomes that are important to the organization (Meyer, 1994). Three survey item domains were ultimately agreed upon: 1) therapeutic alliance; 2) group treatment satisfaction; and 3) self-reported substance use. The survey also included a domain for demographic information.

The therapeutic alliance scale used in Patient Feedback was a 4-item scale (Crits-Christoph et al., unpublished) derived from items from the widely used 24-item California Psychotherapy Alliance Scale (CALPAS),(Gaston, 1991). This 4-item scale correlates 0.80 with the CALPAS total score (deleting the 4 items from the total) and has an internal consistency reliability (Cronbach’s alpha) of 0.78 (Crits-Christoph et al., unpublished). Each item assesses 1 of the 4 primary dimensions of the alliance identified by Gaston (1991). Because of its brevity and adequate psychometric properties, this 4-item alliance scale was ideal for inclusion in the PF survey. Each item is rated on a 5-point scale that begins with “not at all” = 1, “a little bit” = 2, “moderately” = 3, “quite a bit” = 4 and “very much so” = 5.

The treatment satisfaction items were adapted from the (ECHO), an 88-item standardized measure used to monitor patient satisfaction in behavioral healthcare settings (Eisen et al., 1999; Shaul et al., 2001). The 3 items chosen were considered to be the most relevant items for capturing patient satisfaction in substance abuse treatment, while at the same time avoiding duplication of the therapeutic alliance domain. The 3 ECHO items selected each had a high correlation (ranging from 0.50 to 0.67) with an overall rating of satisfaction with treatment (Shaul et al., 2001). The original wording of ECHO items was modified to accommodate the needs of this study. For example, the ECHO item: “In the last 12 months, how often did the people you went to for counseling or treatment explain things in a way you could understand?” was shortened and made consistent with the timeframe of this survey. Each item is rated on the same 5-point scale that therapeutic alliance items were rated. A scale score consisting of the average of the 3 items was created (Cronbach’s alpha = 0.76).

Demographic and length of stay items were included so that feedback reports could monitor performance separately by patient gender, ethnicity, and length of stay in the treatment program, with categories for less than 1 week, 1 to 4 weeks, 1 to 3 months, and greater than 3 months. Clinicians and supervisors could then target improvement initiatives to address discrepancies within specific patient characteristics.

One item asking about alcohol consumption and one asking about drug use were included. These 2 substance items were adapted from the Drug and Alcohol sections of the Addiction Severity Index (McLellan et al., 1980). The “number of days” metric was used instead of “dollars spent” because the cost of substances varies considerably across the country, and patients do not always pay for drugs (e.g., trading sex for drugs, or using surplus from drug sales). Unlike the other PF items, responses to these items were not reported to supervisors or clinicians; instead they were used in our evaluation of the impact of using the PF system.

2.4 Implementation of PF System and Study Procedures

2.4.1 Study Phases

In the pre-intervention phase, the clinic staff (clinicians and supervisors) gave written informed consent for participation and were instructed on the use of the PF system and interpretation of feedback reports. Also during this phase, patients were instructed on how to complete the PF survey. The baseline phase consisted of 2 assessments over a 4 week period that was designed to give clinic staff practice at collecting and faxing forms to the data center. The intervention phase consisted of the collection of PF surveys every other week with feedback reports generated for each survey collection period. Additionally, clinic supervisors held monthly staff meetings to discuss treatment strategies, and a monthly newsletter was provided to the clinics. The post-intervention assessment period consisted of another 4 weeks of bi-weekly collection of PF surveys with feedback reports, but without requirement (for the study) of monthly staff meetings. The final phase of the study, the sustainability phase, was designed to give clinics open access to the PF system to use on a voluntary basis. This phase was included to evaluate whether or not clinics considered the PF system useful enough, and feasible enough, to continue on their own as a quality improvement process. The sustainability phase lasted 1 year.

2.4.2 Administration of PF Survey to Patients

Potential patient participants were first given a written orientation to the study. A member of the clinic staff (not their clinician) reviewed the orientation form with patients and answered any questions they had. The orientation form described the purpose of the study and made it clear that participation was voluntary and fully confidential. Instructions for completing the survey were given provisions were specified if the patient had difficult reading. Patients were informed that the surveys were to be placed in a locked box and only the designed clinic project assistant and research staff would access to the box (not the clinicians).

For a scheduled assessment week during the study, the PF survey was given to all eligible patients in group counseling sessions during that week who agreed to participate. The survey was administered by the project assistant (not the clinician) at the end of group sessions after the clinician had left the room, and then dropped in a locked box to ensure anonymity at the patient level and confidentiality at the clinician level. A designated clerical staff member at each clinic (the project assistant) was responsible for collecting all PF surveys, completing Attendance Forms, and faxing them to a central Data Management Unit for the study. The faxed surveys were read by optical scanning software on a fax server, with the data automatically flowing into a database. Forms that could not be scanned (because of stray marks on the page or other problems) were processed manually to insure the data in the database matched what was on the forms. Custom software created by the investigators then automatically calculated scale scores for the therapeutic alliance and treatment satisfaction scales (averaging item scores for each scale) and created time-series graphs of the scores (see PF reports below).

2.4.3 PF Reports for Clinicians and Supervisors

Using the PF survey, the PF data system automatically produced two types of feedback reports: 1) Clinic Reports, aggregating PF survey data from all participating clinicians at a given clinic into a single report; and 2) Caseload Reports, in which PF survey data from individual clinician caseloads were aggregated into a single report. These Feedback Reports were provided to study clinicians and supervisors as tabbed Microsoft Excel workbooks displaying graphs and tables. Each workbook was posted within 1 week onto password protected websites accessible to the clinicians and supervisors. Supervisors were not given access to the individual clinician Caseload Reports because the feedback was intended to serve a developmental, not evaluative function and also because research has shown that supervisors who are not facile in feedback can de-motivate performance through improper use of performance data and feedback systems (Anderson, Crowell, Doman, & Howard, 1988; Daniels & Daniels, 2004). As additional surveys were collected over time, data were added to the graphs and tables to develop a time-series presentation. Graphs included for each participating clinic and clinician were: 1) Therapeutic Alliance by Length of Stay; 2) Therapeutic Alliance by Ethnicity; 3) Therapeutic Alliance by Gender; 4) Treatment Satisfaction by Length of Stay; 5) Treatment Satisfaction by Ethnicity; and 6) Treatment Satisfaction by Gender. Beneath each graph, a data table for each graph was appended. Finally, one data table presented the combined data for all of the graphs in the current Report. When fewer than 5 patient surveys were available for a graph/table, that graph/table was not produced so as to prevent a clinician from connecting the data to any particular patients. Feedback reports included an additional graph that displayed average patient attendance data for each clinic and clinician. Although attendance data for each individual clinician was originally included in the PF system, these data were confounded by clinician absences, co-leading of groups, and clinicians providing coverage for other clinicians (i.e, leading their groups), so these data were not examined. The PF surveys were not collected if a clinician was absent or another clinician led their group, so the basic survey data was not confounded.

The time-series graphs derived from the PF survey presented the percent of respondents who rated the quality indicator (therapeutic alliance or treatment satisfaction) as “very much so.” A numerical value of 3.5 on the scale scores was used for “very much so,” since the scales were averages of item ratings. Clinicians and supervisors were trained during the Team Training to read the graphs, and a Feedback Manual provided additional guidance in Feedback Report interpretation. In addition to reviewing the Clinic Report as a team, individual clinicians were encouraged to examine their own Caseload Reports, and were free to discuss their Caseload Reports with other clinicians and/or their supervisor on a voluntary basis. Supervisors could use Clinic Report data to guide decisions about training, supervision, and resource allocation. Supervisors could also incorporate Clinic Reports into their reports to regulatory agencies, Board of Directors, funding agencies and other stakeholders.

2.4.4 Team Meetings

On a monthly basis, clinic supervisors led a team meeting in which clinic staff viewed and discussed the most recent Clinic Report. During this meeting the staff identified quality indicators that they would improve (e.g., ratings of therapeutic alliance by women), action steps they planned to initiate during the next month, and who would be responsible for implementing the action steps. The participants in the team meeting, the indicators selected, and the action plan were documented by the supervisor using a PF Team Meeting Form. Team Meeting Forms were faxed on a monthly basis to the Data Management Unit, along with the feedback surveys. These meetings were structured according to JCAHO quality improvement guidelines (JCAHO, 1998); the PF Manual and team training provided guidance to the staff on how to conduct these meetings.

2.4.5 PF Newsletter

On a monthly basis, an electronic newsletter was published by the research team, posted on the website, and distributed to staff at each of the 6 clinics, as well as other interested parties. This monthly newsletter, the PF Newsletter, 1) recognized the accomplishments of participating clinic teams; 2) presented actual data from participating clinics; 3) promoted the use of evidence-based practices: and 4) provided information about quality improvement and the PF System. The provision of social recognition for the clinic teams is derived from Stajkovic and Luthans (1997), and Alvero et al. (2001) and relies on the results of feedback studies across a variety of settings (e.g., Crowell, Anderson, Abel, & Sergio, 1988; Huberman & O’Brien, 1999; Nicol & Hantula, 2001). In each issue of PF Newsletter, team innovations and achievements of individual clinics were recognized with depictions of actual Clinic Reports (with permission and only focusing on clinic-wide scores, not individual clinician reports), team photographs, quotes, and identification of the staff by name (all with signed permission). Clinics were encouraged to use the PF Newsletter in their outreach activities; several clinics enlarged and posted in their lobby copies of past issues in which their program was highlighted.

2.4.6 Training

All training conducted in the study was provided during 4 Internet-based conferences. Internet-based conferences combined the telephone and Internet to create a virtual meeting, eliminating the cost and inconvenience associated with travel. Trainees viewed Microsoft PowerPoint® slides controlled by the presenter over the Internet, while interacting with the presenter by telephone. The first 1-hour Internet-based conference training provided clinic staff with an orientation to the PF System including: 1) quality improvement principles; 2) PF survey; 3) PF Reports; 4) PF Website 5) PF Newsletter, 6) confidentiality of feedback; and 7) overview of study procedures. The second 1-hour Internet-based conference was attended by the clinic support staff and clinic supervisors; it reviewed the study procedures in detail, including how to distribute, collect, transmit faxes, and store the PF surveys. The third 1-hour Internet-based conference focused on training in the use of computers, the Internet, and the PF website. This training was attended by the supervisor and clinic project assistant. The final Internet-based training occurred shortly after the second PF survey collection during the baseline period. During this team training, research staff reviewed with the clinic team their first actual Clinic Report. In addition, this session included information on: 1) quality improvement and the PF surveys; 2) data integrity; 3) the team meeting process; 4) the PF Manual; 5) the PF Website; and 6) the PF Newsletter.

3. Results

3.1 Characteristics of Sample

Across the 6 clinics, 39 clinicians (19 males, 20 females) and 6 clinic supervisors (3 males, 3 females) agreed to participate in the study; 3 eligible clinicians declined to participate. On average, these clinicians had worked 10.6 years as a clinician. Educationally, 29 (74.4%) held masters degrees, 7 (17.9%) bachelor degrees, 2 (5.1%) doctorates, and 1 (2.5%) was a physician. Twenty-seven (69.2%) identified themselves as Caucasian, 11 (28.2%) as African-American, and 1 (2.6%) as Asian.

At the first assessment beginning the intervention phase, the average alliance score for clinicians was 4.36 (SD = 0.35), reflecting a very good alliance for the average patient in their caseload (on the 1 to 5 scale). Average treatment satisfaction at this assessment was also very good (M = 4.41; SD = 0.34), on the 1 to 5 scale.

As mentioned, the anonymous nature of the PF survey and changing patient samples over time preclude a precise description of all of the patients who participated. However, using data obtained from the first assessment only (N=327 unique patients), the general nature of the patient population could be summarized on the available descriptors. The patient sample was 68% men and 32% women. About 50% identified themselves as Caucasian, 39% African-American, 8% Latino, and 3% other. Five-and-a-half percent (5.5%) of patients had been in treatment less than 1 week; 27.3% from 1 to 4 weeks; 22.4% from 1 to 3 months; 44.8% more than 3 months.

The average frequency of drug use for patients at the first baseline assessment was 0.91 days (SD = 1.85; N=327) in the past week, and the average frequency of alcohol use was 0.74 (SD = 1.63; N=325) days in the past week.

3.2 Feasibility Assessment

Feasibility was examined in terms of participants’ compliance with study procedures and researchers’ ability to implement the components of the PF system. Throughout the course of the study, a total of 2,814 PF surveys were collected and faxed by the 6 participating clinics. Based on estimates from clinic directors of the total census of patients within the clinic at the time the PF system was implemented, an overall collection rate of 75.5% of the clinic census was obtained. The PF Website operated continuously throughout the feasibility study, with a total of nearly 5,000 web views (“website hits”) by the 50 participating staff members (clinicians, supervisors, project assistants). The PF Newsletter was published on a monthly basis with a circulation of nearly 300 recipients. One hundred percent of study participants, including project assistants, supervisors and clinicians, completed the required trainings. All of the supervisors conducted monthly Team Meetings, and faxed the Team Meeting form. Eighty-six percent of the eligible clinicians participated in the monthly team meetings. In total, 564 Feedback Reports were posted to the PF website. All PF Reports were posted within 1 week of faxing, and 83% were posted within 48 hours. A total of 89 Caseload Reports were viewed by clinicians during the baseline and intervention phases. The mean number of reports viewed per clinician during these phases was 2.3 out of 8 total reports, with 12 clinicians never accessing their PF reports, and 15 clinicians accessing at least 3 reports, over the course of the baseline and intervention phase.

An important aspect of feasibility as a clinical tool is the sustainability of the PF system after the intervention study. During the 1-year sustainability phase - in which clinics were allowed, but not obligated, to use the PF system as often as they wished or not at all - 3 clinics continued to use it at the same rate (every other week) as during the intervention phase; 2 clinics faxed PF surveys on about a monthly basis; 1 clinic chose to use the system on a semi-annual basis.

3.3 Impact on PF on Alliance, Treatment Satisfaction, and Drug/Alcohol Use

Because the study was a pilot/feasibility study and not powered (N= 6 clinics) to detect changes in these organizations over time, formal statistical analyses are not presented. Overall, there was little evidence for change in drug and alcohol use over time (see Figure 1). However, as mentioned, these clinics and clinicians were on average performing very well, with average drug and alcohol use very low across all assessments. Similarly, alliance and treatment satisfaction ratings were very high across all assessments. Across all of the assessments during the intervention phase, the average alliance and treatment satisfaction ratings only varied between 4.3 and 4.4 (minor variation on the 1 to 5 scale).

Figure 1.

Figure 1

Open in a new tab

Average Counselor (N=39) Performance on Patient Drug and Alcohol Use (past week)

Although significance testing of differences in performance of the 6 clinics over time could not be performed, the graphs of average clinician performance on the drug and alcohol outcomes were visually inspected for each clinic. Five clinics began the intervention period at relatively low average levels of patient drug and alcohol use. In contrast, 1 clinic (Clinic “A” in Figure 2) began the intervention phase at relatively higher levels of drug use. At this clinic there appeared to be some evidence for improvement in drug use outcomes over time. Scores decreased from an average of 1.5 days per week of use to 1.1 days per week of use at the end of the feedback period, before increasing slightly again (to 1.24) at the post-intervention assessment.

Figure 2.

Figure 2

Open in a new tab

Average Drug Use (past week) Scores at Each of Six Participating Clinics

4. Discussion

This project demonstrated that the implementation of a semi-automatic performance improvement system directed at clinicians in addiction treatment facilities was generally feasible from both a research and clinical perspective. Research procedures were performed with the involvement and cooperation of clinical and administrative staff in the community-based clinics. Moreover, it is notable that clinics continued to use the PF system after the intervention phase without any additional support from the research infrastructure, suggesting that the implementation of the intervention is sustainable. It is unclear, however, whether sustainability of use of the PF system can only be achieved after completing a research study during which the research infrastructure was substantially involved. For routine clinical use of PF, it would need to be feasible and sustainable without any involvement of research infrastructure (other than initial training). Before such clinical implementation is recommendation, however, it would be important to evaluate the sustainability not only of the use of the system but of any positive intervention effect. The relative burden of outgoing assessments, attending to feedback reports, and team meetings with discussion of feedback needs to be weighed against the any positive impact on counselor performance (average patient outcomes) that are achieved through using the system.

The feasibility assessment yielded only one piece of information that is less encouraging: the typical clinician accessed their feedback reports only 2.3 times (out of a maximum of 8) over the course of the study. Because the literature on quality improvement suggests that there should not be negative consequences to participating in such improvement interventions (Daniels & Daniels, 2004; Deming, 1986), supervisors did not know if clinicians did or did not access their reports. However, regardless of whether or not an individual clinician accessed their feedback report, all clinicians participated in team meetings and therefore were potential beneficiaries of the suggestions for performance improvement and action steps that arose out of these meetings. To increase clinicians’ access to feedback reports in future studies, positive reinforcers for accessing reports such as monetary rewards could be considered. Alternatively, feedback reports might also be mailed to clinicians, in case internet access or computer skills are a barrier to accessing web-based reports. It may be important, however, to first evaluate the reasons why some clinicians did not access their feedback reports. One possibility might be that certain clinicians were aware (after a single feedback report access or patients’ direct reports to them) that their performance was very good in either absolute terms or relative to their co-workers. Such clinicians might see little need to engage in performance improvement and lack the motivation to spend time viewing reports on the web.

In general, average alliance, treatment satisfaction, and drug/alcohol use outcomes were very favorable across all assessments. An important factor restricting the amount of improvement that might be seen with the PF intervention in this study was the fact that 44.8% of the overall study patient sample had been in treatment for more than 3 months. Although many patients drop out of substance abuse counseling early, the patients who stay in treatment accumulate over time, so that at any random point in time, most patients attending a clinic have been in treatment for a long duration (longer than the average treatment duration). The disadvantage of taking a “snapshot” of how clinicians are performing at a given point in time, regardless of how long current patients have been in treatment, is that outcome assessment at that point in time minimizes the actual amount of change that has occurred for many patients since the initiation of treatment. Previous research on treatment of substance dependence (e.g., Crits-Christoph et al., 1999) indicates that most improvement occurs over the first month of counseling. Thus, by excluding this phase of treatment for many patients, the amount of potential improvement due to the PF intervention was limited: the average days per week of alcohol use (0.74) and drug use (0.91) were low at baseline. Feedback reports displaying these low values conveyed to most clinicians that their performances were excellent right from the beginning and this may have reduced clinicians’ motivation for continuing to access their feedback reports over time. Other studies of feedback interventions in mental health treatment (Lambert, Hansen, et al., 2001; Lambert et al., 2002) conduct assessments as patients initiate and continue treatment, thereby incorporating a baseline assessment that captures patients at their most dysfunctional (when they initially begin treatment). In contrast, PF was designed to be a clinician performance improvement system that could be implemented relatively easily with the full clinic census. If patients were all assessed from their respective treatment initiation sessions, data collection would need to be continuous over a long period of time to obtain an adequate sample of patients for each clinician. Such an extended assessment process could not yield timely, cost-effective data for quality improvement reviews.

In fact, performance improvement studies in other areas of medicine (e.g., Snyder & Anderson, 2005) often find that performance indicators in general show relatively positive performance for many providers or health care workers. Only a minority of providers/health care workers are typically found to be performing at a sub-optimal level and in need of substantial improvement. In the current study, there was evidence, by visual inspection, that one clinic had higher average drug and alcohol use at the baseline assessment compared to the other clinics. This same clinic appeared to show improvements over time during the intervention phase of the project.

While we had too few clinics to quantitatively examine factors that might explain differences between clinics, it is noteworthy that the clinic that displayed improvement over time was the only clinic that was not already using team meetings as part of its organization structure. The team meetings that were part of the PF intervention may have been a more meaningful addition to this clinic than to the other clinics that already had team meetings. This clinic in particular used the Team Meetings to develop an “orientation to treatment” group format for new patients. This group format involved placing new patients into an orientation group for 30 days, and then transitioning these newly oriented patients into regular treatment groups. Other clinics also developed improvement plans during the Team Meetings, such as: giving special attention to ethnicities rating low on therapeutic alliance; reviewing lists of scheduled patients to increase accuracy of attendance records; calling no-shows immediately; and engaging long-term patients with new patients to help them feel more connected.

The finding that one clinic in particular could benefit more from this type of intervention suggests a strategy for clinical use of the PF system: clinics might implement the system as a onetime assessment tool as part of their quality improvement efforts and then only proceed to use the feedback process on a continuing basis if the initial assessment yielded evidence of the need for performance improvement for the clinic as a whole, or for a subset of clinicians.

Although some clinics might show the need for performance improvement using the PF system as implemented in the current study, two major modifications might be needed and tested before such a system could have broader clinical usefulness. The first is to focus feedback on the subset of patients that have recently (e.g., within 1 month) begun treatment. Because treatment failures are more likely to be represented in these early-in-treatment patients, this modification of the PF system would likely yield a wider range of alliance, treatment satisfaction, and drug/alcohol use scores, thereby permitting room for improvement on the measures. A second possible modification is the development of alternative instruments that lend themselves to feedback better by having adequate variation. Moreover, tracking drug and alcohol use with single item scales may not be adequate.

Because this study did not have a control group, we cannot rule out that other factors besides the feedback intervention may have been responsible for the improvement in drug use outcome at the one clinic. The process of measuring outcome, the general success of treatment, or simply the passage of time, may in part be responsible for improvements seen over the course of the intervention phase. To sort out these different factors, a randomized controlled trial that compares feedback/team meetings with no feedback/team meetings would be necessary.

Other limitations of the current study include the fact that only 6 clinics, each motivated and already connected to a research network, participated. The extent to which the PF system would be successful in clinics that are less motivated or less comfortable with research is not known. Another limitation is that only 75% of patients currently in treatment completed the PF survey at the participating clinics. It is possible that the less motivated, and less satisfied patients, did not complete the PF survey, thereby leading to a biased sample. The generalizability of PF across different types of drugs of abuse, and different treatment settings, is also not known. In addition, its use with individual counseling (all treatments were group counseling in the current study) needs to be examined. Further research is needed to enhance the effect of the Patient Feedback system and to understand any potential mechanism of change.

Acknowledgments

The preparation of this manuscript was funded in part by NIDA grants U01-DA130431, and R01-DA018935. We wish to thank all of the clinicians and cljnic staff who participated in this study.

Footnotes

Correspondence concerning this article should be addressed to Paul Crits-Christoph, Room 650, 3535 Market St. Philadelphia, PA 19104, Email: crits@mail.med.upenn.edu.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Andrzejewski ME, Kirby KC, Morral AR, Iguchi MY. Technology transfer through performance management: The effects of graphical feedback and positive reinforcement on drug treatment clinicians’ behavior. Drug and Alcohol Dependence. 2001;63:179–186. doi: 10.1016/s0376-8716(00)00207-6. [DOI] [PubMed] [Google Scholar]
  2. Alvero AM, Bucklin BR, Austin J. An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings. Journal of Organizational Behavior Management. 2001;21:3–29. [Google Scholar]
  3. Anderson DC, Crowell CR, Doman M, Howard GS. Performance Posting, Goal Setting, and Activity-Contingent Praise as Applied to a University Hockey Team. Journal of Applied Psychology. 1988;73:87–95. [Google Scholar]
  4. Anderson DC, Crowell CR, Hantula DA, Siroky LM. Task clarification and individual performance posting for improving cleaning in a student-managed university bar. Journal of Organizational Behavior Management. 1988;9:73–90. [Google Scholar]
  5. Berwick DM, Godfrey AB, Roessner J. Curing health care: New strategies for quality improvement. San Francisco, CA: Jossey–Bass; 1990. [Google Scholar]
  6. Cook T, Dixon MR. Performance feedback and probabilistic bonus Contingencies among employees in a human service organization. Journal of Organizational Behavior Management. 2005;25:45–63. [Google Scholar]
  7. Crits-Christoph P, Gibbons MB, Forman R, Hu B, Hearon B, Worley M. A brief alliance scale for community-based research. University of Pennsylvania; 2004. Unpublished manuscript. [Google Scholar]
  8. Crits-Christoph P, Siqueland L, Blaine J, Frank A, Luborsky L, Onken LS, et al. Psychosocial treatments for cocaine dependence: National Institute on Drug Abuse Collaborative Cocaine Treatment Study. Archives of General Psychiatry. 1999;56:493–502. doi: 10.1001/archpsyc.56.6.493. [DOI] [PubMed] [Google Scholar]
  9. Crowell CR, Anderson DC, Abel DM, Sergio JP. Task clarification, performance feedback and social praise: Procedures for improving the customer service of bank tellers. Journal of Applied Behavior Analysis. 1988;21:65–71. doi: 10.1901/jaba.1988.21-65. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Daniels AC, Daniels JE. Performance Management: Changing Behavior that Drives Organizational Effectiveness. Atlanta, GA: Performance Management Publishers; 2004. [Google Scholar]
  11. Deming WE. Out of the Crisis. Cambridge, MA: MIT Center for Advanced Educational Services; 1986. [Google Scholar]
  12. Ducharme LJ, Luckey JW. Implementation of the methadone treatment quality assurance system: Findings from the feasibility study. Evaluation & the Health Professions. 2000;23:72–90. doi: 10.1177/01632780022034499. [DOI] [PubMed] [Google Scholar]
  13. Eisen SV, Shaul JA, Clarridge B, Nelson D, Spink J, Cleary PD. Development of a consumer survey for behavioral health services. Psychiatric Services. 1999;50:793–798. doi: 10.1176/ps.50.6.793. [DOI] [PubMed] [Google Scholar]
  14. Finney JW, Willenbring ML, Moos RH. Medical Care. Vol. 38. 2000. Improving the performance of VA care for patients with substance use disorders: The QUERI Substance Abuse Module; pp. I-105–I-112. [DOI] [PubMed] [Google Scholar]
  15. Gaston L. The reliability and criterion-related validity of the California Psychotherapy Alliance Scale. Psychological Assessment. 1991;3:68–74. [Google Scholar]
  16. Gillaspy JA, Wright AR, Campbell C, Stokes S, Adinoff B. Group alliance and cohesion as predictors of drug and alcohol abuse treatment outcomes. Psychotherapy Research. 2002;12:213–229. [Google Scholar]
  17. Hermann RC, Regner JL, Erickson P, Yang D. Developing a quality management system for behavioral health care: The Cambridge Health Alliance Experience. Harvard Review of Psychiatry. 2000;8:251–60. [PubMed] [Google Scholar]
  18. Howard KI, Brill P, Lueger RJ, O’Mahoney MT, Grissom G. Integrating outpatient tracking assessment. Philadelphia, PA: Compass Information Services, Inc; 1995. [Google Scholar]
  19. Howard KI, Kopta SM, Krause MS, Orlinsky DE. The dose-effect relationship in psychotherapy. American Psychologist. 1986;41:159–164. [PubMed] [Google Scholar]
  20. Howard KI, Lueger RJ, Maling M, Martinovich Z. A phase model of psychotherapy outcome: Causal mediation of change. Journal of Consulting and Clinical Psychology. 1993;61:678–685. doi: 10.1037//0022-006x.61.4.678. [DOI] [PubMed] [Google Scholar]
  21. Howard KI, Moras K, Brill PL, Martinovich Z, Lutz W. Evaluation of psychotherapy: Efficacy, effectiveness, and patient progess. American Psychologist. 1996;51:1059–1064. doi: 10.1037//0003-066x.51.10.1059. [DOI] [PubMed] [Google Scholar]
  22. Humphreys K, Hamilton EG, Moos RH. Substance abuse treatment in the Department of Veterans Affairs: System structure, patients, and treatment activities. Palo Alto, CA: Program Evaluation and Resource Center; 1996. [Google Scholar]
  23. Huberman WL, O’Brien RM. Improving therapist and patient performance in chronic psychiatric group homes through goal-setting, feedback, and positive reinforcement. Journal of Organizational Behavior Management. 1999;19:13–36. [Google Scholar]
  24. Joint Commission on Accreditation of Healthcare Organizations (JCAHO) Using Performance Measurement to Improve Outcomes in Behavioral Health Care. Oakbrook Terrace, IL: JCAHO; 1998. [Google Scholar]
  25. Juran JM, Godfrey AB, editors. Juran’s Quality Handbook. 5. New York: McGraw-Hill; 1999. [Google Scholar]
  26. Lambert M, Hansen N, Finch A. Patient-focused research: Using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology. 2001;69:159–172. [PubMed] [Google Scholar]
  27. Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology. 2003;10:288–301. [Google Scholar]
  28. Lambert MJ, Whipple JL, Smart DW, Vermeersch DA, Nielsen SL, Hawkins EJ. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research. 2001;11:49–68. doi: 10.1080/713663852. [DOI] [PubMed] [Google Scholar]
  29. Lambert MJ, Whipple JL, Vermeersch DA, Smart DW, Hawkins EJ, Nielsen SL, et al. Enhancing psychotherapy outcomes via providing feedback on patient progress: A replication. Clinical Psychology and Psychotherapy. 2002;9:91–103. [Google Scholar]
  30. Lueger RJ, Lutz W, Howard K. The predicted and observed course of psychotherapy for anxiety and mood disorders. The Journal of Nervous and Mental Disease. 2000;188:127–134. doi: 10.1097/00005053-200003000-00001. [DOI] [PubMed] [Google Scholar]
  31. Lueger RJ, Howard KL, Martinovich Z, Lutz W, Anderson E, Grissom G. Assessing treatment progress of individual patients using expected treatment response models. Journal of Consulting and Clinical Psychology. 2001;69:150–158. [PubMed] [Google Scholar]
  32. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical Psychology. 2000;68:438–450. [PubMed] [Google Scholar]
  33. McCaul ME, Svikis D. Improving patient compliance in outpatient treatment: clinician-targeted interventions. In: Pickens CG, Leukefeld G, Schuster CR, editors. Improving Drug Abuse Treatment. Rockville, MD: National Institute on Drug Abuse; 1991. (NIDA Research Monograph No.106, pp. 204–217). DHHS Publication No. (ADM) 91–1754. [Google Scholar]
  34. McLellan AT, Luborsky L, O’Brien CP, Woody GE. An improved diagnostic instrument for substance abuse patients, The Addiction Severity Index. Journal of Nervous and Mental Diseases. 1980;168:26–33. doi: 10.1097/00005053-198001000-00006. [DOI] [PubMed] [Google Scholar]
  35. Meyer C. How the right measures help teams excel. Harvard Business Review. 1994;72:95–102. [Google Scholar]
  36. Moos RH, Finney JW, Cannon D, Finkelstein A, McNicholas L, McLellan AT, et al. Outcomes monitoring for substance abuse patients: I. Patients’ characteristics and treatment at baseline. Palo Alto, CA: VA Health Care System, Program Evaluation and Resource Center and HSR&D Center for Health Care Evaluation; 1998. [Google Scholar]
  37. National Treatment Center Study. Clinical Trials Network Summary and Comparison Report. Athens, GA: Institute for Behavioral Research, University of Georgia; 2005. NTCS Report No. 10. [Google Scholar]
  38. Nicol N, Hantula DA. Decreasing delivery drivers’ departure times. Journal of Organizational Behavior Management. 2001;21:105–116. [Google Scholar]
  39. Phillips CD, Hubbard RL, Dunteman G, Fountain DL, Czechowicz D, Cooper JR. Measuring program performance in methadone treatment using in-treatment outcomes: An illustration. Journal of Mental Health Administration. 1995;22:214–225. doi: 10.1007/BF02521117. [DOI] [PubMed] [Google Scholar]
  40. Shaul JA, Eisen SV, Clarridge BR, Stringfellow VL, Fowler FJ, Cleary PD. Experience of care and health outcomes (ECHO™) survey field test report: Survey evaluation 2001 [Google Scholar]
  41. Snyder C, Anderson G. Do quality improvement organizations improve the quality of hospital care for Medicare beneficiaries? Journal of the American Medical Association. 2005;293:2900–2907. doi: 10.1001/jama.293.23.2900. [DOI] [PubMed] [Google Scholar]
  42. Stajkovic AD, Luthans F. A meta-analysis of the effects of organizational behavior modification on task performance. Academy of Management Journal. 1997;40:1122–1149. [Google Scholar]
  43. Wells KB, Sherbourne C, Schoenbaum M, Duan N, Meredith L, Unützer J, et al. Impact of disseminating performance improvement programs for depression in managed primary care: A randomized controlled trial. The Journal of the American Medical. 2000 doi: 10.1001/jama.283.2.212. [DOI] [PubMed] [Google Scholar]

RESOURCES