The Effectiveness of Clinician Feedback in the Treatment of Depression in the Community Mental Health System

Abstract

Objective

We describe the development and evaluation of a clinician feedback intervention for use in community mental health settings. The Community Clinician Feedback System (CCFS) was developed in collaboration with a community partner to meet the needs of providers working in such community settings.

Method

The CCFS consists of weekly performance feedback to clinicians as well as a clinical feedback report that assists clinicians with patients who are not progressing as expected. Patients in the randomized sample (N=100) were pre-dominantly female African-Americans, with a mean age of 39.

Results

Satisfaction ratings of the CCFS indicate that the system was widely accepted by clinicians and patients. An HLM analysis comparing rates of change across conditions controlling for baseline gender, age, and racial group indicated a moderate effect in favor of the feedback condition for symptom improvement (t(94) = 2.41, p = .017, d = .50). Thirty-six percent of feedback patients compared to only 13% of patients in the no feedback condition demonstrated clinically significant change across treatment (χ²(1) = 6.13, p = .013).

Conclusions

These results indicate that our CCFS is acceptable to providers and patients of mental health services, and has the potential to improve the effectiveness of services for clinically meaningful depression in the community mental health setting.

While many have suggested that mental health treatment outcomes in community-based settings could be improved through the dissemination of empirically-supported psychotherapies (Stirman, Crits-Christoph, & DeRubeis, 2004; Barlow, Levitt, & Bufka, 1999; Chorpita et al., 2002; Henggeler, Schoenwald, & Pickrel, 1995), such efforts have a variety of hurdles, including the cost of training clinicians in new methods and the resistance of clinicians to adopting new approaches that are discrepant from their own preferred style of therapy. Provider feedback systems that help clinicians identify patients who are not improving as expected may be an important alternative approach to improving psychotherapeutic treatment outcomes in community settings.

A meta-analysis of studies of mental health status feedback in psychotherapy (Knaup, Koesters, Schoefer, Becker, & Puschner, 2009) indicates an overall small effect (d = .10) for clinician feedback systems compared to control conditions. However, Lambert and Shimokawa (2011) review two feedback systems that have advantages over other simple mental health status feedback methods. Both the Partners for Change Outcome Management System (PCOMS; Miller, Duncan, Sorrell, & Brown, 2005) and the Outcome Questionnaire-45 (OQ-45; Lambert, Gregersen, & Burlingame, 2004) include feedback on patient progress in treatment and include patient ratings of important clinical variables to maximize treatment outcomes. Lambert and Shimokawa (2011) found a moderate combined effect size across three well-designed studies (Reese, Norsworthy, & Rowlands, 2009; Anker, Duncan, & Sparks, 2009) of the PCOMS (d = .47) indicating that patients whose clinicians received feedback were better off than 68% of treatment-as-usual patients.

Six controlled studies have been published that examine the effects of providing OQ-45 progress feedback to clinicians (Lambert, Whipple, Smart, Vermeersch, & Nielsen, 2001; Lambert et al., 2002; Whipple et al., 2003; Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Harmon et al., 2007; Slade, Lambert, Harmon, Smart, & Bailey, 2008). Pooling data across studies, Lambert and Shimokawa (2011) report that for off track patients (i.e., those who are not progressing as expected across the early sessions of treatment) there were moderate effects for those whose clinicians received weekly OQ-45 feedback (d = .58) and for those whose clinician received weekly OQ-45 feedback plus feedback based on the OQ-Clinical Support Tools (d = .81) compared to treatment-as-usual patients.

Since the publication of this review, seven additional controlled studies of OQ-45 feedback to mental health patients have been conducted (de Jong, van Sluis, Nugter, Heiser, & Spinhoven, 2012; Crits-Christoph et al., 2012; Hansson, Rundberg, Osterling, Ojehagen, & Berglund, 2012; Probst et al., 2013; Simon et al., 2013; Simon, Lambert, Harris, Busath, & Vazquez, 2012; Amble, Gude, Stubdal, Andersen, & Wampold, 2014). Three of the six studies demonstrated a significant positive effect of two-level feedback on patient outcome (Probst et al., 2013; Simon et al., 2013; Simon et al., 2012; Amble et al., 2014) although the effect sizes for two of these studies were much smaller than in previous investigations ranging from .12 to .36. Crits-Christoph et al. (2012) also demonstrated a positive effect of feedback for substance abuse patients, but this investigation did not include a randomized control group. In contrast, two recent investigations did not provide support for the effectiveness of clinician feedback. De Jong et al. (2012) reported a minimal positive effect of feedback post hoc only for the subset of off track patients whose clinicians self-reported using feedback. Lastly, Hansson et al. (2012) did not find significant positive effects of feedback on patient outcome, as measured by the OQ-45.

We developed a community clinician feedback system targeted to the treatment of depression in the community mental health setting working closely with patients, clinicians, and administrators. Our system was designed to be broadly applicable to patients seeking outpatient treatment for clinically meaningful depression in order to facilitate implementation in the community. We specifically focused on designing a system that met the challenges of delivering mental health services in the community setting. Although the performance feedback studies based on the OQ-45 and PCOMS conducted to date have been impressive, these feedback interventions have not been developed to be sensitive to the broader range of functioning found in community mental health centers and the efficacy of these feedback interventions has never been tested in the community mental health system, which treats the population sector most in need of cost-efficient interventions. Of the three studies of the PCOMS reviewed by Lambert and Shimokawa (2011), one was conducted in a university counseling center, one in a graduate program training clinic, and one with a sample of clients in couples counseling. Studies demonstrating moderate effects for the OQ-45 have focused largely on Caucasian samples and mostly university counseling center patients (mean age 22 in three studies conducted).

Our goal was to develop and test a clinician feedback intervention that not only built on the strengths of existing feedback systems by providing weekly feedback on progress in treatment and by including additional clinical help for patients who were not progressing as expected across treatment, but also that addressed some weaknesses for implementing feedback specifically in the community mental health setting. A collaborative effort between the investigators and our community partner indicated that in order to meet the needs of the community setting, a feedback system would need to: a) be easy to disseminate with no training required by clinicians to use or understand the reports; b) include measures that were reliable and validated in community mental health settings; and c) go beyond the existing systems in providing clinical feedback to clinicians for patients who were not progressing in treatment as expected.

In community mental health centers where clinicians have large difficult caseloads and little free time, additional clinical tools for improving clinical outcomes with patients who are not progressing as expected are vital. Some of the studies of the OQ system did incorporate clinical support tools for off track patients, which included additional patient measures of the therapeutic relationship, motivation, life events, and support networks. We conducted a series of focus groups with community clinicians and supervisors to determine what additional clinical constructs these providers would want to consider when making important decisions regarding alteration of the treatment course to assist patients who were not progressing as expected. Our patient-specific assessment was designed to help shed light on why certain patients are not progressing as expected and can provide important clinical foci that may be crucial to changing the course of treatment for clinicians treating depression in the community.

The goals of the current manuscript are to a) describe the Community Clinician Feedback System (CCFS), b) present the development of community recovery curves that serve as the basis for identifying whether patients are progressing as expected, c) provide descriptive statistics of clinician and patient satisfaction ratings of the CCFS, d) present the results of a randomized clinical trial comparing the effectiveness of the CCFS versus no feedback in the treatment of depression in the community mental health setting, and e) evaluate the effectiveness of the CCFS clinical report for the subset of patients who went off track for clinical improvement.

We hypothesized that patients in the community mental health setting whose clinicians received feedback regarding their progress in treatment would have significantly better outcomes compared to patients whose clinicians did not receive feedback. Although previous investigations of the OQ-45 demonstrated that feedback was effective only for the subset of patients who went off track for clinical improvement, our system was designed to improve outcomes for the full intent-to-treat population in the community mental health setting where clinicians may have heavy caseloads and patients seeking services often have complex diagnostic syndromes. Whereas providers in settings such as university counseling centers may not need help with on track cases, we hypothesized that providers in the community mental health setting may benefit from the weekly progress reports for all patients in treatment. In these community settings, it may be important to keep close track of weekly progress even for on track patients to help prevent deterioration. We further hypothesized that for the subset of feedback patients who go off track for clinical improvement, symptom course will be improved after the clinicians receive the clinical report.

Method

All study procedures were conducted in compliance with the City of Philadelphia and the University of Pennsylvania Institutional Review Boards.

Setting

The present study was conducted through a partnership with a large-scale non-profit, private, community mental health center (CMHC) serving predominantly publicly funded patients in Philadelphia. This agency provides mental health, mental retardation, and substance abuse treatment services to both child and adult populations. The patients served include low-income individuals who receive public assistance or other forms of support for medical and behavioral health services. The CMHC outpatient program serves approximately 2000 patients each year. Most outpatients are offered and receive concurrent pharmacological and psychotherapeutic services.

Participants

Patients

Patient participants in the present study were recruited from the general population of patients seeking services for depression at the CMHC. To minimize burden on the intake staff, we used existing organizational intake procedures with an added research interest assessment. The screening procedure included the administration of the Quick Inventory for Depressive Symptomatology (QIDS; Rush et al., 2003) to identify patients with a clinically meaningful level of depressive symptoms (i.e., score of 11 or above on the QIDS). Research by Rush et al. indicates that scores of 11 or greater on the QIDS are comparable to a score of 14 or above on the 17-item Hamilton Rating Scale for Depression (HAMD; Hamilton, 1960).

In accordance with standard intake procedures at the clinic, every adult patient seeking outpatient services at the CMHC was asked to complete the QIDS, as part of a larger packet of clinic forms, while in the waiting room before an intake appointment. During the intake interview, the CMHC intake clinician examined the QIDS and then asked patients who scored an 11 or higher if they were interested in participating in a research study. For those patients who were interested, the intake clinician completed a brief checklist that identified inclusion and exclusion criteria for the study. The intake clinician referred eligible patients who scored 11 or above on the QIDS to the research staff, were interested in participating in the research program, were able to read English at or above the 4^th grade level, were between 18 and 65 years of age, were not being referred out for immediate substance abuse or medical treatment, and did not meet criteria for any psychotic disorder. Patients were allowed to receive concurrent medication as part of their services at the site. Medication usage was not monitored in this protocol. A member of the research staff then met with the referred patient immediately following their intake appointment, depending on availability, or contacted the interested patient via telephone within 48 hours to schedule a brief baseline assessment. At the baseline assessment, a research assistant met with the referred patient to describe the study protocol and complete informed consent. Patients who provided consent then completed a demographic questionnaire and the 24-Item Behavior and Symptom Identification Scale (BASIS-24; Eisen, Normand, Belanger, Spiro, & Esch, 2004) and were randomized to treatment condition. Patient randomization was performed using a computer generated randomization algorithm to assign patients to a clinician who was either receiving feedback reports or no feedback reports. Assignments were in blocks of four resulting in balance in assignment for every four patients. Clinician assignment to receive feedback reports versus no feedback reports was stratified as a function of therapist experience; therefore guaranteeing balance in patients assignment dependent on clinician experience.

Three hundred and ninety-four patients scored an 11 or above on the QIDS at their intake with the clinic and were considered for referral to the study (see consort chart in Figure 1). Of the 394 patients, 164 (42%) were referred to the research staff for potential participation in the study and 230 (58%) were ruled out by the CMHC intake clinician. One hundred patients (61.0%) completed a baseline assessment and were randomized to treatment with a clinician who received weekly feedback reports or a clinician who did not receive weekly feedback reports. Of the 50 patients randomized to the feedback condition, 39 (78%) attended at least one therapy session and of the 50 patients randomized to the no feedback condition, 46 (92%) attended at least one therapy session.

Figure 1.

Recruitment flow diagram for feedback versus no feedback conditions.

Baseline demographic characteristics are presented in Table 1 for both the randomized sample and the modified intent-to-treat sample included in all analyses. The majority of patients were female, single, African-American, unemployed, with less education than a high school diploma/GED. Primary diagnoses based on the community clinicians’ intake evaluation included major depressive disorder (43%), post-traumatic stress disorder (21%), depressive disorder not otherwise specified (10%), adjustment disorder (9%), mood disorder not otherwise specified (4%), bipolar disorder (2%), and other diagnoses (11%). Average baseline scores on the BASIS-24 were 2.17 (sd = .61) for the randomized sample and 2.02 (sd = .58) for the intent-to-treat sample.

Table 1.

Demographic Characteristics of Patients

Characteristics	Randomized Sample n=100	Intent-to-Treat Sample n=85
Gender (Female)	71.0%	74.1%
Marital Status
Single	51.0%	48.2%
Married/Cohabitating	18.0%	16.5%
Separated/Divorced	29.0%	32.9%
Widowed	2.0%	2.4%
Ethnicity (Hispanic)	4.0%	3.5%
Race
African-American	78.0%	76.4%
Caucasian	6.0%	7.1%
Mixed Race	7.0%	7.1%
Other	9.0%	9.4%
Employment
Full-Time	3.0%	2.3%
Part-Time	6.0%	5.9%
Homemaker	5.0%	5.9%
Unemployed	76.0%	76.5%
Student	10.0%	9.4%
Highest level of education
< High School Diploma	24.0%	27.1%
High School Diploma/GED	22.0%	18.8%
Some College	40.0%	40.0%
College Graduate	8.0%	9.4%
Post-graduate or Professional degree	4.0%	4.7%
Age, years m(sd)	39.76(11.82)	40.65(11.77)

Clinicians

The CMHC where we conducted this trial employs around 40 master’s level clinicians, both salaried and fee-for-service. Three to four full-time and two to three part-time psychiatrists provide medical coverage to the clinic’s patients. We recruited 20 clinicians to participate in the current program of research by placing flyers advertising the study in clinicians’ mailboxes. Clinicians who responded then met with a research assistant who described the study and completed an informed consent with clinicians who were interested in participating. After consenting to the study protocol, the clinicians completed a demographics questionnaire and were randomized to treatment condition using a computer generated randomization algorithm to assign clinicians to either receive feedback reports or not receive feedback reports balancing number of years of experience as a clinician (clinicians were nested within treatment condition). Clinicians were defined as low experience if they had less than or equal to two years of clinical experience and high experience if they had greater than two years’ experience. Random assignments were generated separately for the two levels of experience. Random assignments were generated in blocks of four such that assignment guaranteed a balance of two feedback and two no feedback clinicians per block.

All clinicians held at least a master’s level degree. Clinicians in both conditions received a $75 honorarium for every three patients treated as part of this investigation. Clinicians in the feedback condition were asked to complete a satisfaction survey regarding their experience with the CCFS after completing treatment with their third randomized patient. Clinicians received $20 for completion of the satisfaction survey. Demographic characteristics are presented in Table 2 for the sample of 20 clinicians. The majority of clinicians were female, white, with a mean age of 34.44 years (sd = 12.17). Fifty percent of therapists had more than two years of experience.

Table 2.

Demographic Characteristics of Clinicians

Characteristics	Feedback Condition n=10	No Feedback Condition n=10	Full Sample n=20
Gender (Female)	70%	80%	75%
Ethnicity (Hispanic)	0%	0%	0%
Race
Caucasian	60%	70%	65%
African American	10%	30%	20%
Asian	20%	0%	10%
Other	10%	0%	5%
Highest Degree
Master’s	90%	90%	90%
Doctoral	10%	10%	10%
Years of Experience
2 or Less	50%	30%	40%
Greater than 2	50%	70%	60%
Age, years m(sd)	34.50 (10.46)	43.78 (13.17)	34.44 (12.17)

Outcome and Progress Assessments

QIDS

The QIDS is designed to be a brief self-report measure to assess depressive symptoms using the criteria outlined in the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) in 16 items. The QIDS demonstrated good internal consistency (Cronbach’s alpha = .86) and total scores on the QIDS were highly correlated (r = .81) with scores on the 17-item HAMD in a sample of patients with chronic major depression (Rush et al., 2003).

The BASIS-24

(Eisen, Normand, Belanger, Spiro, & Esch, 2004). All patients completed the BASIS-24 on paper at the baseline assessment and on a handheld computer before each subsequent therapy session. The BASIS-24 consists of 24 items designed to measure mental health status across six domains: depression/functioning, interpersonal relationships, psychotic symptoms, alcohol/drug use, and emotional lability. The measure has demonstrated good construct and discriminate validity, acceptable test-retest reliability, and internal consistency (Eisen et al., 2004). Further research on the measure’s use in racially and culturally diverse populations including whites, African Americans, and Latinos demonstrates reliability, concurrent validity, and sensitivity across groups (Eisen, Gerena, Ranganathan, Esch, & Idiculla, 2006), making the measure appropriate for the present study. In the current sample, the BASIS-24 had adequate internal consistency (Cronbach’s alpha = .72) at baseline.

We implemented a one-page feedback report based on the BASIS-24 to inform clinicians of the progress of their patients (see sample in Appendix). Based on our focus groups, clinicians working in the community thought that the most important information was whether or not their patient was progressing as expected. The feedback report consisted of a single sheet of paper that included a colored graph showing the line of expected recovery along with the patient’s actual BASIS-24 scores for all sessions completed to date. At the top of the report, a green label indicated that a patient was progressing as expected and a red label indicated that a patient was not progressing as expected (i.e., was off track). The report also contained a written description of the patient’s progress at the bottom of the sheet. The wording of the written description was developed through a series of focus groups with community providers. This feedback is brief and simply presents to the community clinicians whether their patient is on track and therefore can continue treatment as planned or whether their patient is off track and they should reconsider the treatment course. This one page report was modelled after the weekly progress report utilized in the OQ-45 system but was simplified to include only a red versus green signal, simplified language to describe the patient’s progress, and a simple graph of general symptom course.

Development of Recovery Curves for the BASIS-24

We developed the CCFS to build on the strengths of other effective feedback systems. Like the OQ-45, our goal was to provide a weekly report that indicated each patient’s progress to date compared to other patients who began treatment at similar levels of distress. We derived average recovery curves for the BASIS-24, which served as the basis of the feedback reports that indicate whether or not a given patient is on or off track for a successful therapeutic outcome. The recovery curves were derived from a national dataset of 3200 outpatient participants who completed the BASIS-24 at treatment intake with a subset of 850 patients who completed the measure again at 4 to 8 weeks following intake (Eisen et al., 2004; Eisen, Ranganathan, Seal, & Spiro, 2007). This dataset provided an excellent opportunity to examine recovery curves across a large, racially diverse sample of outpatients, many of whom were treated in mental health centers in the public sector. Analyses were based on the sample of 546 outpatients who had two assessment points within 16 weeks. For the construction of recovery curves, we assumed an individual’s change would depend on the amount of time elapsed from baseline and the magnitude of their baseline assessment. We constructed the model through an ordinary least squares regression (OLS) equation Y_Δi = b₁Week_i + b₂Week_i × Y_0i + e_i, where Y_Δi is the observed change for the i^th person, Week_i is the amount of time between assessments measured in weeks, using the data on 546 mental health outpatients. As with any regression equation, we represented individual errors as e_i. The model assumed the amount of change at baseline was zero (i.e., regression through the origin) and a linear interactive relationship between time and baseline, as well as model-based assumptions for OLS-regression (i.e., homoscedasticity, independence of the observations, and normality of the residuals. The recovery curve was derived based on 80% prediction estimates from the estimated regression equation. For any given week from baseline and a given baseline score, the recovery curve is a bowed trajectory, with deviations in linearity depending on the magnitude of the baseline score and amount of time. The recovery curves constructed by Lambert et al. (2002) were based on 50 blocks of the baseline score from their sample. Our approach has three advantages: (1) we have a unique recovery curve for each possible score on the BASIS-24 at baseline determined by the interaction term in our model, (2) our model is accurate when the elapsed time is 0 due to the regression through the origin, and (3) each recovery curve is formed as a function of the individual’s baseline assessment, therefore no patients fall on the border between blocks. Individuals with observed values above the 80% prediction interval are classified as off track. We validated the model using 168 outpatients with clinically meaningful depressive symptoms seeking treatment at the CMHC. Study specific recovery curve algorithms, using the same process described above, were derived for the local CMHC sample. Comparing this local CMHC sample to the Eisen et al. (2004) national sample, we observed an agreement of κ=0.871 (95% CI: 0.793-0.945) for designation of off track status corresponding to very good agreement per Byrt (1996).

Community Clinician Feedback Questionnaire (CCFQ)

The CCFQ was designed as a second-level feedback assessment to assist clinicians with patients who went off track for clinical improvement. Participants who went off track during treatment based on the weekly BASIS-24 feedback reports completed the CCFQ at their earliest convenience after being identified as off track. The clinical report derived from the CCFQ was provided to the clinician prior to the next treatment session. To develop the content and format of the CCFQ, we conducted a series of focus groups with both community providers and experts in treating depression. We included the following constructs deemed important by our expert and clinician groups: demographic background, treatment motivation, attitudes and expectations about treatment, therapeutic alliance, suicide risk, substance use, perceived social support, psychosocial stressors, violence potential, personality disorder, interpersonal distress, interpersonal patterns, cognitive distortions, compensatory skills, and trauma history. We began by selecting reliable and valid measures of each construct available in the public domain (see Table 3). We administered the full battery of measures to a sample of 210 patients receiving outpatient services at the CMHC (Scott, Connolly Gibbons, Schauble, Thompson, & Crits-Christoph, 2011) to derive abbreviated scales. We also modified the wording of some items in order make the measure more community-friendly and applicable for self-report use. All items were modified to fit on a 4-point Likert scale ranging from “strongly disagree” to “strongly agree.” We conducted psychometric analyses to eliminate items that were highly correlated and items that did not demonstrate high reliability. The resulting subscales with reduced numbers of items all demonstrated high Cronbach’s alphas (all above .85) and high correlations with the original full scales (all r’s above 0.9; see Table 1) for the sample of 210 patients. All subscales also demonstrated adequate Cronbach’s alphas in the current effectiveness sample (see Table 3). The resulting 235-item assessment is cost efficient (all scales are based on publicly available measures at no cost) and reliably addresses the constructs most important to community providers in evaluating the treatment course for patients who are not improving as expected.

Table 3.

The Reliability of the Scales Included in the CCFQ

Measure	Number of Items on Original Scale	Number of Items on Revised Measure	Cronbach’s Alpha for Derivation Sample N=210	Correlation of CCFQ Subscale with Original Measure	Cronbach’s Alpha for Current Sample N=41
TMQ Ryan, Plant, & O’Malley, 1995	22	12	.853	.986	.705
AAE Moras & Jones, 1992	3	3	N/A	N/R	N/A1
WAI-C Horvath & Greenberg, 1989	12	8	.892	.967	.900
RFL Linehan, Goodstein, Nielsen, & Chiles, 1983	16	12	.892	.978	.827
AUDIT Saunders, Aasland, Babor, & Grant, 1993	15	10	.923	.978	.792
PSS-FF Procidano & Heller, 1983	23	12	.887	0.912	.899
PSS Cohen, Kamarck & Mermelstein, 1983	10	10	.831	N/R	.697
AQ Bryant & Smith, 2001	12	9	.866	N/R	.862
IIP-PD-25 Kim & Pilkonis, 1999	25	25	.935	N/R	.814
IIP-32 Barkham, Hardy, & Startup, 1996	32	32	.958	N/R	.868
SUIP-R Connolly Gibbons et al., 1999	28	25	.939	.996	.897
DAS Weissman & Beck, 1978	40	26	.911	.995	.899
WOR-SR Connolly Gibbons & Crits-Christoph, 2011	65	20	.916	.982	.896
WAI RD/DD Weinberger, 1991	18	18	.767	N/R	.713
THQ Green, 1996	24	24	.951	N/R	.671

Note: N/A indicates that only single item scores used. N/R indicates that the scale was not revised for inclusion in CCFQ. TMQ= Treatment Motivation Questionnaire; AAE = Attitudes and Expectations Form; WAI-C = Working Alliance Inventory – Client Version; RFL = Reasons for Living Questionnaire; AUDIT = Alcohol Use Disorders Identification Test; PS-FF = Perceived Social Support from Friends and Family; PSS = Perceived Stress Scale; AQ = Aggression Questionnaire; IIP-PD-25 = 25-item Inventory of Interpersonal Problems-Personality Disorder Scale; IIP-32 = 32-item Inventory of Interpersonal Problems; SUIP-R = Self-Understanding of Interpersonal Problems –Revised; DAS = Dysfunctional Attitudes Scale; WOR-SR = Ways of Responding Questionnaire – Self Report; WAI RD/DD = Weinberger Adjustment Inventory, sum of Repressive Defensiveness and Denial of Distress scales; THQ = Trauma History Questionnaire.

We developed algorithms to generate clinical reports from the CCFQ. Our programming scores each continuous construct included in the CCFQ and determines whether the patient scored in the lowest third, middle third, or highest third of the distribution for that factor in the community sample. We developed text to descriptively characterize the scores in each third of the distribution relative to other patients receiving services at the clinic. For categorical measures, we developed text to represent each categorical answer emphasizing how each categorical answer compares to the judgments of others that receive services in the community setting.

Clinician Satisfaction Survey

We developed a brief clinician satisfaction survey to evaluate the feasibility of implementing the CCFS in the community setting. Clinicians were asked to rate the BASIS-24 reports and the clinical reports based on the CCFQ separately on four 7-point scales ranging from “not at all” to “extremely,” including usefulness of the reports, ease with which one understands the language of the reports, clarity of the reports, and accuracy of the reports. Clinicians were also asked how often they reviewed the BASIS-24 and CCFQ reports and were asked to complete a checklist of possible ways they used the reports. Finally, clinicians were asked to rate their overall satisfaction with the CCFS on a 7-point Likert scale ranging from “not at all” to “extremely” and were asked whether they would recommend the system to other clinicians. Clinicians completed the satisfaction survey after treating 3 cases within this protocol.

Patient Satisfaction Survey

We also implemented a brief patient satisfaction survey. Patients were asked to rate both the BASIS-24 and the CCFQ across five 7-point scales (ranging from “not at all” to “very much”), including the degree to which they liked completing the assessment, how useful they found completing the assessment, whether they felt the reports helped improve treatment, the frequency with which the reports were shared with them during treatment, and the helpfulness of reviewing reports during the treatment sessions. Patients were asked to complete the satisfaction survey at the conclusion of their participation in the study.

Procedure

Patients were randomly assigned to 8 weeks of individual therapy with a clinician who would receive weekly feedback reports or a clinician who would not receive weekly feedback reports. All assessments and therapy sessions occurred at the community clinic and were scheduled at the patient’s convenience. Patients were reimbursed $5 for completing the baseline assessment, $5 for completing the BASIS-24 before each weekly therapy session, and $25 for completing the CCFQ if the patient went off track. Patients were reimbursed with gift cards to a local convenience store.

All patients were asked to complete the BASIS-24 on a handheld computer administered by the research assistant when they arrived at the clinic for weekly therapy appointments. Patients completed the BASIS-24 on a user-friendly screen that presents one question at a time and requires that each question be answered before the next question appears. All answers were instantly captured electronically and transmitted to a secure server computer in the study office at the clinic. An event-driven program scored the BASIS-24 and compared scores across each visit for a given patient with the expected trajectory of other patients starting treatment at similar levels of distress. Then, the program automatically generated a one-page BASIS-24 report indicating the patient’s progress to date, which was automatically printed on a printer located in the clinic’s reception area. The BASIS-24 reports were collected by the research assistant and then handed to clinicians randomized to the feedback condition before each weekly therapy appointment. The feedback reports of patients randomized to clinicians who did not receive weekly reports were filed in a secure location by the research assistant and not given to the clinician.

Any patient whose report indicated that they were not progressing as expected across the early sessions of treatment (i.e., off track) was asked by the research assistant to schedule an appointment to complete the CCFQ before the patient’s next scheduled weekly therapy appointment. After completing the CCFQ, the research assistant scored the measure and generated the written report based on the algorithms developed for the measure. For patients who were randomized to a clinician who received feedback, the second-level reports were placed in the clinicians’ mailboxes or handed to them before the next scheduled therapy assessment with the client. For patients who were randomized to a clinician who did not receive feedback, the second-level reports were filed by the research assistant and not distributed to the clinician.

Statistical Analysis

The primary analyses were based on a modified intent-to-treat sample that includes 85 patients who were randomized to treatment condition, completed the BASIS-24 at baseline, and attended at least one treatment session. The effectiveness of the CCFS was examined using hierarchical linear models (HLM) to compare the rate of change across treatment weeks on the total score of the BASIS-24 for all patients whose clinicians received feedback to patients whose clinicians did not receive feedback, controlling for baseline gender, age, and racial group. One advantage of the HLM is the flexibility in handling missing data, while still accommodating the clustering within an individual attributable to the repeated measures. The HLM analyses used all available data for each of the 85 patients. Age was included as a continuous covariate. Racial group represented Caucasian versus minority patients. To examine the effectiveness of the CCFQ clinical report for patients who went off track, we conducted a hierarchical linear model (HLM) comparing the rate of change on the BASIS-24 prior to going off track with slope of change following receipt of the CCFQ clinical report for the subset of patients who went off track in the feedback condition, controlling for gender, age, and race. For this analysis, we conducted a piecewise model looking at change from baseline until each patient went off track with a second leg of time looking at change after the point where patients went off track representing a connected piecewise model (Cudeck & Klebe, 2002; Cudeck & Harring, 2007; Gallop, Dimidjian, Atkins, & Muggeo, 2011). In addition, we conducted a hierarchical linear model comparing the rate of change from the point at which the patient went off track, and the clinician received the clinical report through week 8 for feedback patients compared to no feedback patients.

In order to evaluate clinically meaningful change on the BASIS-24, we computed the percent of patients in each treatment group who achieved reliable change, the percent who had clinically significant change, and the percent who had both reliable and clinically significant change (Jacobson & Truax, 1991). For the computation of reliable change, the reliability of the BASIS-24 was estimated from the endpoint ratings since intake ratings were specifically restricted by our inclusion criteria. Clinically significant change was computed using criterion C described by Jacobson and Truax (1991). Computations for the BASIS-24 were based on the normative mean and standard deviation provided by the Mental Health Services Evaluation Department of McLean Hospital (S. Eisen, personal communication, January 16, 2012). We also report the percentage of patients in each treatment group demonstrating clinical deterioration, defined as patients whose scores on each measure increased by the reliable change index.

Results

Clinician Satisfaction

Clinicians rated high overall satisfaction with our feedback system (m = 5.0, sd = .6 on 1 to 7 scale where 7 was extremely satisfied). Eighty-nine percent of the nine clinicians surveyed reported that they would recommend this feedback system to other clinicians working in the community. For the BASIS-24 reports, clinicians rated the reports as useful (m = 4.6, sd = 1.0) and the language as understandable (m = 6.1, sd = .8). They also endorsed good report clarity (m = 5.0; sd = 1.8) and good accuracy of the reports (m = 4.8, sd = .8). Eighty-nine percent of clinicians said they always reviewed the BASIS-24 reports. When asked how they used the reports, 66.6 % indicated that they reviewed the reports directly with their clients, 22.2% indicated that they did not use the reports, and 11.2 % indicated that they reviewed the reports individually and used it to make clinical notes. For the CCFQ clinical reports, 100% of clinicians said they always reviewed the reports and they rated high overall usefulness of the reports (m = 5.3, sd = .5). Clinicians rated the clinical reports as highly understandable (m = 6.2, sd = .8), of high clarity (m = 5.8, sd = .4), and accurate in representing their patients’ problems (m = 5.2, sd = .8). When asked how they use the reports, 83.3% indicated they reviewed the report directly with their clients, 66.7% indicated that they took the reports into account when creating treatment plans, 50.0% indicated that they felt they had a better understanding of their clients after reading the report and that changed how they approached the client, 33.3% indicated that they focused treatment on a new problem based on the report, 16.7% indicated that they tried new therapeutic techniques suggested in the reports, 16.7% indicated that they reviewed books or clinical materials based on suggestions in the reports, 16.7% indicated that they reviewed the report with their supervisor, and 16.7% indicated that they did not use the reports.

Patient Satisfaction

Our patient satisfaction survey also indicates that patients seeking services for treatment of depression in the community mental health setting found our feedback system to be a valuable part of their treatment experience. On a 1 to 7 scale ranging from “not at all” to “very much,” the 34 patients who completed the survey rated that they liked completing the BASIS-24 weekly (m = 6.1, sd = 1.4), found the measure useful (m = 5.8, sd = 1.4), and thought that completing the BASIS-24 improved treatment (m = 4.8, sd = 2.0). Regarding completion of the CCFQ, patients who went off track indicated that they liked completing the measure (m = 5.3, sd = 1.7), found it useful (m = 5.1, sd = 1.9), and felt that it improved their treatment (m= 4.4, sd =2.4). Patients found it somewhat useful on average to review the reports with their clinician (m = 4.2, sd = 2.4).

Effectiveness of Feedback

Because our system was designed to help with the progress of all patients receiving services for depression, we first conducted hierarchical linear models to compare the slope of change on the BASIS-24 for all patients whose clinicians received feedback compared to patients whose clinicians did not receive feedback, controlling for baseline gender, age, and racial group (Figure 2). Patients attended a median of 6.0 (sd = 3.1) sessions. The average score on the BASIS-24 across the sample at baseline was 2.17 (sd = .61). An HLM analysis comparing rates of change across conditions indicated a moderate effect in favor of the feedback condition for symptom improvement (t(94) = 2.41, p = .017, d = .50).

Figure 2.

BASIS-24 scores across treatment sessions for patients whose clinicians were randomized to receive feedback compared to no feedback condition.

These results indicate effectiveness of the comprehensive two-level feedback system but do not delineate the contributions of each level of feedback provided to clinicians. The BASIS-24 feedback alone may have been responsible for the demonstrated effectiveness of the feedback system. However, our system was designed to give additional valuable clinical feedback to clinicians just for the subset of patients who were not progressing adequately across treatment. Figure 3 displays the actual symptom course for the 18 patients who went off track in the feedback condition compared to their expected trajectory if they had continued on the slope of change established while their clinicians were receiving the BASIS-24 reports alone. Figure 3 is derived from the model-based estimates produced from the fitted within group piecewise model. The break point represents the average BASIS-24 score at the point where patients on average went off track. The points prior to going off track represent the average slope estimate across patients prior to going off track and the subsequent points after going off track are based on the on-average slope estimates after going off track. Slope estimates represent the on-average change per time, where time is per session. The estimates after going off track if there were no feedback provided are based on the assumption that the change post off track would remain equal to the rate of change prior to going off track producing a linear trajectory over the course of the entire longitudinal period.

Figure 3.

BASIS-24 scores before and after off track session for patients whose clinicians were randomized to receive feedback reports.

Although feedback from the BASIS-24 appears effective for the sample as a whole, those patients who eventually went off track were increasing in symptoms across the early sessions of treatment and began improving from the point where they went off track and their clinicians received the second-level clinical report. There was a moderate within group effect (t(123) = -2.83, p = .006, d = .57) comparing slope of change prior to going off track with slope of change following receipt of the second-level clinical report, indicating that this second-level report is an important component of the feedback system for patients who go off track. Without this second-level report, our estimated trajectory indicates that they would have continued increasing in symptoms with only the weekly BASIS-24 reports. In fact, our analysis of the subsample of patients who went off track indicates a small size effect (although not statistically significant) for symptom improvement from the point at which they went off track for patients whose clinicians received the feedback reports compared to patients whose clinicians did not receive feedback (t(41) = 1.20, p = .233, d = .38).

Based on the criterion outlined by Jacobson and Truax (1991), an alpha coefficient of .80 with an intake standard deviation of .58 for the BASIS-24 results in a reliable change index of 0.72 points and a clinically significant change criterion of 1.32. Fifteen percent of patients in the feedback condition and 9% of those in the no feedback condition had reliable change (χ²(1) = .91, p = .340). Thirty-six percent of feedback patients and 13% of patients in the no feedback condition demonstrated clinically significant change across treatment (χ²(1) = 6.13, p = .013), while 10% of feedback patients and 4% of patients in the no feedback condition demonstrated both reliable change and clinically significant change (χ²(1) = 1.12, p = .289). No patients in either treatment condition demonstrated clinical deterioration as measured by the BASIS-24.

Discussion

These results indicate that our CCFS is feasible to administer, acceptable to providers and patients of mental health services, and has the potential to improve the effectiveness of services for clinically meaningful depression in the community mental health setting. Community mental health settings are in need of cost-efficient and easily disseminated interventions that can improve treatment outcomes. However, dissemination of any intervention requires the participation of both providers and patients to ensure the success of the intervention. It is noteworthy in this busy environment where time is at a premium, almost 90% of clinicians always reviewed the BASIS-24 when provided and 100% reviewed the CCFQ reports. The clinicians who used the system found it useful, very understandable, clear, and accurate. The majority of clinicians said that they would recommend this system to other clinicians to support clinical services.

Our results indicate that the CCFS is effective in improving outcomes for patients in outpatient treatment for depression in the community mental health setting. The effect sizes found here were moderate in size, comparable to those found in other feedback studies such as investigations of the OQ feedback system (Lambert & Shimokawa, 2011). However, these effects are not directly comparable because the current study used only patients with depressive symptoms. In addition, standard care in CMHC settings is associated with relatively low response rates, likely due to the severity of disorders and degree of comorbidity evident in CMHC settings. Contrary to studies of the OQ-45, our results indicate that feedback to clinicians is successful in improving outcomes for all patients receiving services for depression, not simply for the subset of patients who go off track for clinical improvement. Lambert and colleagues have interpreted the OQ-45 results as indicating that weekly feedback on progress may not be useful for clinicians in treating cases that are progressing as expected because these cases are doing well and the clinician can continue with the treatment as planned. However, in the community setting, most patients have complex diagnostic syndromes, severe psychosocial stressors, and frequent crises. In this patient population, it may be extremely important for clinicians to closely monitor symptom course for all patients to help prevent and address clinical deterioration before the point that a patient goes off track for clinical improvement. Our results indicate that the CCFS is useful in this community setting in assisting clinicians to keep patients on track to begin with, in addition to helping correct the course of treatment for patients who go off track.

Because our system consists of two levels of feedback, weekly BASIS-24 reports for all patients as well as the more comprehensive CCFQ clinical report for the subset of patients who go off track, it is difficult to identify which elements of the feedback system are responsible for its effectiveness. One might suppose that it is merely the weekly BASIS-24 reports that are responsible for the effectiveness of the system and perhaps the CCFQ reports are not necessary. To evaluate the specific effectiveness of the CCFQ reports for the subset of feedback patients who do go off track for clinical improvement, we compared the slope of change prior to going off track to the slope of change after the patient goes off track and the clinician receives the CCFQ report. Our results show a significant improvement in symptom course, with patients actually increasing in symptoms prior to going off track and then revealing symptom decreases on average after the clinician receives the CCFQ clinical report. However, comparing the symptom course after the point where a patient goes off track did not reveal a statistically significant difference between the feedback group and the no feedback group. These results indicate that the CCFQ may be a critical component to this clinician feedback system but suggest further evaluation of the utility of the clinical report is needed. Although the weekly BASIS-24 progress reports assist clinicians in keeping on track patients progressing adequately, the inclusion of the CCFQ at the point where a patient goes off track for clinical improvement may help the clinician to correct the course of treatment.

Our analysis of clinically significant change also indicates an important benefit for this feedback system compared to no feedback. In the no feedback condition, only 13% of patients met criteria for clinically significant improvement across the 8 week period. In contrast, 36% of patients in the feedback condition showed clinically significant change. Given the ease of disseminating this system and the low cost, the CCFS significantly improves patients’ chances of experiencing clinically significant improvement. These results indicate that it might make sense to combine the use of this feedback system with an evidence-based psychotherapy to improve outcomes further in the community mental health setting. In a pilot examination of Supportive-Expressive Psychotherapy for depression in the same setting (Connolly Gibbons et al., 2012), 50% of patients demonstrated clinically significant change. It may be that the combination of an evidence-based psychotherapy and feedback is needed to insure that the majority of patients in the community mental health setting experience clinically meaningful change.

Our clinician survey asked clinicians to rate how they used our feedback system in clinical practice. The majority of clinicians said that they used both the BASIS-24 and the CCFQ report in the session with the patient. It is likely that the reports, in addition to simply raising the clinician’s awareness, are effective in helping the clinician and patient to focus the treatment session on a discussion of the therapeutic progress together. This focus on test data to help the dyad redirect the therapeutic process towards improvement is congruent with the principles of collaborative or therapeutic assessment articulated by Finn, Fischer, and Handler (2012). Indeed, the current findings suggest that the use of a simple and accessible instrument like the CCFS can extend the benefits of therapeutic assessment approaches to the CMHC setting.

The community mental health patients in our study were also very satisfied with the CCFS. Patients who participated in the research protocol on average provided very high ratings when asked whether they liked completing the assessments (m = 6.1 for BASIS-24 and 5.3 for CCFQ). This is an extremely important finding for disseminating this system in clinical practices that serve racially diverse populations. It may be that other feedback systems also result in high satisfaction ratings. Such ratings should be considered in dissemination efforts. If patients like completing the assessments, rather than seeing the assessments as a burden, the system has the potential to be successful in a community mental health setting where resources are stretched.

There are multiple limitations to the current trial. First, we used the BASIS-24 both as an integral part of our intervention and as our outcome assessment. To control for the effect of completing the assessment, we had patients in the no feedback condition complete the weekly BASIS-24 although their clinicians did not receive the reports. Future investigations should use independent observer-rated assessments of outcome and should consider implementing a broader outcome battery to examine the breadth of effects possible with clinician feedback systems. Second, our results generalize to the populations of patients seeking services for clinically meaningful depression in the community mental health system. These results may not generalize to all practice settings and patient populations. The system was designed to meet the specific needs of the community mental health setting; however, the high satisfaction ratings from both clinicians and patients indicate that this feedback system may be useful in a variety of practice settings with diverse patient populations. Another limitation is that we were not able to evaluate whether match between therapist and patient race, or other variables, influenced the effectiveness of the feedback intervention. Future research would be needed to delineate the effectiveness of the system for specific racial and diagnostic groups. We did not gather information on medication utilization so it is possible that symptom course was influenced by medication. Finally, we evaluated the utility of the second level clinical report by comparing the slope of change prior to going off track with the slope of change after the clinician receives the clinical report for patients in the feedback condition who went off track using a piecewise regression model. This approach does have limitations for evaluating causality. We have not ruled out the possibility that unmeasured variables are responsible for the change in symptom trajectory for this subgroup rather than the clinical report. Future research would be needed to evaluate the causal relation between the second level clinical report and improvement in symptom course.

Public Health Significance.

This study suggests that therapist performance feedback is an effective intervention for the treatment of depression in the community mental health setting.