The effects of therapist feedback on the therapeutic alliance and alcohol use outcomes in the outpatient treatment of alcohol use disorder

Abstract

Background:

It is widely accepted that the therapeutic alliance (TA) is a mediator of psychotherapy effects, but evidence is sparse that the TA is an actual mechanism of behavior change. The purpose of this study was to provide the first systematic evidence regarding the TA as a mechanism of change in the treatment of alcohol use disorder (AUD).

Methods:

Participants were 155 adult men and women presenting for individual outpatient treatment of AUD. Each was randomly assigned to one of 6 experienced therapists, who did or did not receive over 3 study phases post-session participant feedback on his/her ratings of the TA. All participants received a 12-session version of cognitive behavioral therapy for AUD. Participants rated the TA by use of the California Psychotherapy Alliance Scale (CALPAS) and reported their daily alcohol consumption between-sessions and for one year post-treatment by use of the Timeline Followback Interview. Multilevel statistical models that partitioned within- and between-participant effects and between-therapist effects were run to test the effects of feedback condition on the alliance and alcohol use, and the effects of the alliance on alcohol use.

Results:

The study’s main hypotheses that feedback causes an enhanced therapeutic alliance, and that the alliance is associated with better alcohol use outcomes, were not supported.

Conclusions:

Several methodological and substantive reasons for the pattern of findings are suggested, as well as directions for future research that would advance study of the TA as a mechanism of change in psychotherapy and in studying therapist effects on outcomes in general.

Introduction

Decades of research on psychotherapy process and outcomes have consistently shown an overall positive relation of small to medium magnitude between the therapeutic alliance and treatment outcomes (Fluckiger et al., 2018). This essential result has been obtained across different approaches to psychotherapy and client populations. Accordingly, the therapeutic alliance, or a collaborative relationship between therapist and client that consists of an emotional bond and shared beliefs in the tasks and goals of treatment (Bordin, 1979), has been identified as a mechanism of change that is common to psychotherapeutic approaches and that is independent of any specific effects on outcomes that varying approaches may have. Studies of adult clients being treated for a substance use disorder have revealed findings similar to those found in the broader psychotherapy literature, albeit with some tendency for more modest effect sizes (Fluckiger et al., 2018; Richardson et al., 2012). A review of research on the alliance in substance use disorders treatment (Meier et al., 2005) showed that the alliance, especially when rated by clients early in treatment, is a consistent predictor of engagement, early improvements in treatment, and more positive posttreatment outcomes. Studies by Barber et al. (2001) and Crits-Christoph et al. (2009) showed that these conclusions hold when controlling for symptom change (alcohol or other drug use in this context) occurring prior to the measurement of the alliance.

Research on the alliance and substance use disorders treatment predominantly has featured one or two measurements of the alliance during (typically earlier) treatment and testing associations between the alliance so measured and outcomes post-treatment and during follow-up. However, such a design does not yield data needed to provide empirical evidence on whether the alliance mediates behavior change initiated in psychotherapy, which as noted is the current consensus in the field. Rather, evaluation of the alliance as a “common mechanism” of change requires multiple assessments of both the alliance during treatment and symptoms during and following treatment (Kazdin and Nock, 2003). Falkenström and colleagues (2013) measured both the alliance and symptoms session-by-session among clients in primary care psychotherapy. Participants were most commonly treated for anxiety, relationship problems, and depression. Falkenström et al. evaluated the relationship of the alliance with symptom levels reported at the next treatment session, controlling for symptom change prior to the assessment of the alliance. The results from the session-to-session analyses showed that alliance ratings predicted subsequent symptomatic change, and that symptomatic change predicted alliance. The particularly noteworthy finding from this study was the session-to-session prediction of symptom change by the therapeutic alliance, even when controlling for symptom change preceding the session. Zilcha-Mano et al. (2016) reported similar findings in adults presenting for outpatient treatment of a range of psychiatric disorders (about 50% for mood disorders, no participants reported for treatment of a substance use disorder) in an urban medical setting. As part of a clinical trial, these participants received either cognitive behavior therapy (CBT) or alliance-focused therapy (AFT) as their assigned treatment. The results showed that prior session alliance ratings were positively related to subsequent session symptom reduction independent of prior symptom level. As in the Falkenstr m et al. study (2013), the session-by-session alliance findings were independent of treatment condition.

Connors et al. (2016) reported the first study of within-treatment client changes in both the alliance and alcohol consumption in individuals presenting for outpatient treatment of alcohol use disorder (AUD). All participants received 12-session individual CBT treatment for AUD. Participant reports of the alliance were recorded after every treatment session, and participants were asked to report on their daily alcohol use since their last treatment session. The results showed that alliance ratings following a given session were inversely related to alcohol involvement (indexed by percentage of days drinking and percentage of days heavy drinking) in the days up to the next treatment session, controlling for alcohol involvement reported at the previous session. Using the same data base as Connors et al. (2016), Prince et al. (2016) used latent profile analysis to show that within-treatment participant reports of the alliance could be defined by three profiles, which independently predicted alcohol involvement outcomes at 4 months following treatment. As predicted, the pattern of alliance enhancement over the course of treatment predicted less alcohol involvement following treatment.

In summary, research involving both general psychiatric and AUD treatment adult outpatients has shown consistent findings of participant alliance ratings predicting subsequent session symptom reports, controlling for prior session symptom level. Given the generally positive relation that has been found between providing therapists with feedback on client change in symptoms during treatment and eventual treatment outcomes (Prescott et al., 2017), it would seem that providing therapists feedback from their clients on the course of the alliance could increase overall levels of the therapeutic alliance. The literature on the alliance suggests that such an increase in turn would predict better patient outcomes both during and following treatment.

One study of the effects of providing therapists feedback on the alliance has been reported. Zilcha-Mano and Errázuriz (2015) randomly assigned adults presenting for general outpatient psychiatric treatment to one of five therapist feedback conditions: no feedback control, weekly raw score feedback on patients’ general psychiatric functioning as measured by their responses to the 30-item Outcome Questionnaire (OQ) items, weekly raw score feedback on participants’ ratings of the therapeutic alliance, as measured by the Working Alliance Inventory (WAI), weekly feedback on both the WAI and the OQ raw scores, or weekly feedback of Lambert’s OQ progress feedback report, respectively. This study was more an effectiveness rather than an efficacy trial and involved a relatively large number of participants, N = 547, with the most frequent diagnosis of depressive disorder (68.7%). No participant was reported as being treated for a substance use disorder. Therapists (n = 28) were free to use their preferred therapeutic approach, and there was no limit on the number of sessions completed in treating a participant. The analyses of this many-faceted study that are of central interest here are tests of the moderating effects of feedback condition on the association between the alliance and treatment outcome (OQ total score, range 0–120, higher score = greater distress). Only two of the five feedback conditions were included in this analysis, control and feedback only on the alliance. The results showed that feedback moderated the alliance-outcome relation, such that the positive relation between within-patient component of the alliance and outcome was stronger for patients whose therapists received alliance feedback.

The purpose of this experimental study was to extend investigation of the effects of providing therapists patient feedback on the course of the alliance and alcohol use outcomes in adult outpatients receiving individual CBT treatment for AUD. Participants were randomly assigned to therapists who did or did not receive weekly feedback on participants’ reports of the therapeutic alliance. The study was divided into three Phases (1, 2, and 3). In Phase 1, no therapist received feedback, and in Phases 2 and 3 whether therapists received feedback was based on their randomly assigned experimental condition. The 12-item CALPAS was used to obtain participants’ weekly reports of the alliance, and at each weekly (of 12) CBT session participants were asked to report their daily alcohol consumption since their last session. Participants’ alcohol consumption also was measured over the course of one year following treatment completion. It was hypothesized that providing therapists with participants’ reports of the alliance enhances alliance levels over the course of treatment, and that alliance level is inversely associated with alcohol involvement outcomes during and following treatment.

Materials and Methods

Participants

Individuals seeking outpatient AUD treatment were recruited from the community through local newspaper and radio advertisements. Inclusion criteria were (a) being between 18 and 65 years of age, (b) meet DSM-IV-TR (American Psychiatric Association, 2000) criteria for a diagnosis of current alcohol dependence, (c) living within commuting distance of the program site, (d) demonstrating basic reading skills sufficient to complete assessment materials; and (e) willing to sign a consent form stating a willingness to participate in and complete all facets of the investigation. Exclusion criteria were (a) meet criteria for a current psychotic disorder, (b) presenting with gross neurocognitive impairment, as evidenced by poor performance on a structured mental status examination (MMSE; Vertesi et al., 2001), or (c) in alcohol or other drug treatment during the previous 12 months (except for mutual-help groups). This latter exclusion criterion was applied to reduce the likelihood that the alliance that emerged in the present treatment was confounded with that formed in a recent previous treatment. Individuals with other psychiatric disorders were not excluded, but those with concurrent drug use disorders were accepted for the study only if they had a primary alcohol dependence disorder. The participant recruitment flow chart is displayed in Figure 1. The present analyses are based on the 155 participants (n = 51 female) who attended at least two treatment sessions. Table 1 is a summary of participant demographic and descriptive information. Participants did not differ statistically by treatment condition on any baseline demographic or alcohol consumption variables. It is important to note that patients were enrolled in the study September 2012 – August 2014. DSM-5 was published in 2013. Because the project had enrolled about one half of its total participants by that date, we continued using DSM-IV-TR diagnostic terms and criteria.

Figure 1.

Participant Recruitment Flow Chart

Table 1.

Descriptive Statistics for Overall Sample and by Treatment Condition

	Overall (n = 155)	Control (n=104)	Feedback (n=51)	F Statistic or χ2	p-value	Partial η2 or ϕ
Age (mean)	48.65 (8.75)	48.53 (8.73)	48.90 (8.88)	.062	.804	.000
Gender (% female)	32.9%	34.6%	29.4%	420	.587	.052
Race (% minority)	5.8%	6.7%	3.9%	.494a	.719	−.056
Marital Status (% married)	52.3%	50.0%	56.9%	.646	.495	.065
Employment (% employed)	67.1%	68.3%	64.7%	.197	.717	−.036
Income (household)	6.53 (2.31)	6.57 (2.39)	6.45 (2.14)	.087	.769	.001
Any Past Treatment	18.4%	19.8%	15.7%	.382	.659	−.050
PDA (baseline)	.261 (.266)	.266 (.257)	.253 (.286)	.072	.788	.000
PHD (baseline)	.578 (.326)	.582 (.309)	.571 (.362)	.038b	.845	.000

Note: PDA = Percent Days Abstinent; PHD = Percent Heavy Drinking Days (defined as 5 or more standard drinks for men and 4 or more standard drinks for women; NIAAA, 2004; Wechsler and Nelson, 2001).

Gender 0 = female, 1= male; Race 0 = Caucasian, 1 = minority/other (Caucasian 94.2%; 2.6% were African American, and 3.2% were more than one race or unknown). Marital Status 0 = single, separated, divorced, or widowed, 1 = married/cohabiting (Married 46.5%, Cohabitating 5.8%, Single Never Married 18.1%, Separated 8.4%, Divorced 20.6%, Widowed 0.6%). Employment 0 = unemployed, 1 = employed (67.1% reported part-time or full-time employment; 20% unemployed; 7.7% disabled; 5.2% retired). Income examined as continuous variable using the 9-point scale (1= 0-$4999 to 9 = $100,000 or greater). Any past treatment 0= no, 1= outpatient, inpatient, or AA.

^aExpected count below 5 in the one cell

^bLevene test was significant, however, equal variances not assumed was still not significant with a p-value = .644

Procedure

Individuals that called the project phone number were screened for initial inclusion and provided a description of the treatment program. If the initial eligibility criteria were met, participants were scheduled for a baseline/intake interview (~ 90 minutes). During the baseline appointment, informed consent was obtained and measures administered. The research was approved by and conducted in compliance with the University at Buffalo Social and Behavioral Sciences Institutional Review Board.

All participants received 12 weekly sessions of standard Cognitive Behavioral Therapy (CBT; Kadden et al., 1992) for alcohol dependence (the first session was 90 minutes and the successive sessions were 60 minutes each) through the research site’s outpatient clinical program. Therapists were blind to the study hypotheses, trained on the study protocol, and supervised weekly, including reviews of audiotaped sessions to ensure compliance to the treatment manual.

At the end of each treatment session, participants in a private area completed two computer-administered measures of the therapeutic alliance (based on their perceptions of that day’s session): the California Psychotherapy Alliance Scale (CALPAS; Gaston, 1991) and the Working Alliance Inventory (WAI; Horvath and Greenberg, 1986; Tracey and Kokotovic, 1989). The CALPAS ratings were used to generate feedback to the therapists and was the measure of the therapeutic alliance used to test the study’s hypotheses. The WAI will be used in subsequent analyses not included in this paper, though it is notable that total scores on the two scales correlated r = .77, n = 1523 observations, lending confidence that both measure the same therapeutic alliance construct. The CALPAS was computer-scored, and a feedback form was generated for the therapist that incorporated information on the patient’s perceptions of the alliance and guidance on techniques and strategies that might be used to enhance the alliance or maintain an alliance that is perceived by the patient as strong. In the Feedback condition, the feedback form was available to the therapist (and the supervising psychologist) immediately after the session electronically as well as via hard-copy. Therapists were asked to review the feedback forms following the treatment session (i.e., that day) and also prior to the participant’s next session. Participants were aware that therapists might be given feedback on the alliance ratings they provided. However, to decrease the salience of the alliance ratings being collected after each session, participants also completed a brief session report that gathered such information as how difficult it had been to stay abstinent, how much stress they had been experiencing, their confidence in remaining abstinent, and alcohol consumed, if any, since the last session.

Each of the 6 therapists in Study Phase 1 saw all participants in the No Feedback (NF) condition. After a “wash-out” period for purposes of completing treatment with the final enrollees in Phase 1, Phase 2 recruitment began, with Therapists 1 and 2 seeing participants in the Feedback condition and Therapists 3, 4, 5, and 6 seeing participants each in the No Feedback condition. After another “wash-out” period, Phase 3 recruitment began, with Therapists 1, 2, 3, and 4 seeing participants each in the Feedback condition and Therapist 5 and 6 seeing participants in the No Feedback condition. Within each phase, participants were assigned randomly to therapists, and therefore to feedback condition. Each therapist was assigned approximately 9–10 participants per phase.

Feedback form.

Participants’ California Psychotherapy Alliance ratings were computer scored to yield a composite CALPAS score and four subscale scores (Patient Commitment, Patient Working Capacity, Therapist Understanding and Involvement, and Working Strategy Consensus). These scores were presented graphically on the therapist’s computer screen and as a printed report, along with the associated descriptive text and guidance. In addition, a graph was shown, depicting the course of the alliance ratings for the current and prior weeks’ sessions. Feedback was divided for the overall alliance score and for each of the four subscales into three categories (i.e., high, moderate, low) in order to provide feedback to therapists depending on the current status of the alliance as rated by the participant. This is not meant to imply that there are discrete cutoffs that distinguish a high alliance from a moderate alliance or a moderate alliance from a low alliance. Because alliance scores tend to be negatively skewed (with high scores being more common), “high” ratings were limited to a perfect score of 7; scores below 7 generated constructive feedback. Anchor wording from CALPAS items (not at all, a little bit, somewhat, moderately, quite a lot, very much so) were used to demarcate the moderate and low feedback categories. The midpoint between “moderately” and “quite a lot” was chosen as the cut off between low and moderate feedback because the word “moderately” was deemed to reflect a substantially vulnerable alliance.

The feedback text was informed by Bordin’s (1979) theory of alliance dimensions as well as empirical studies of factors that benefit the alliance, such as encouraging discussions with the patient concerning the status of the relationship (e.g., Foreman and Marmar, 1985), maintaining a flexible stance as concerns therapeutic technique (e.g., Castonguay et al., 1996), and encouraging patients to talk about instances when they felt misunderstood (e.g., Rhodes et al., 1994). Wording within the feedback text also was included regarding therapist behaviors that are thought to be associated with strong alliances, such as expressing empathy and providing a safe working environment (e.g., Lambert and Barley, 2001; Norcross, 2010; Watson and Greenberg, 1994). In accord with the idea that the alliance ratings lie on a continuum, the feedback text was general (albeit guided by empirical alliance research) and somewhat overlapping between categories. The final version of the feedback text was informed by pilot testing. The purpose of the therapist feedback was to provide suggestions, based on the current week’s ratings, for improving the alliance (or maintaining a strong alliance). Therapists were encouraged to use suggestions on the feedback report. Furthermore, therapists were asked not to discuss alliance feedback scores with patients (unless brought up by the participant).

After receiving the feedback for a given session, the therapist completed a brief series of questions to assess the extent to which (a) the participant’s ratings of the alliance are consistent with the therapist’s perceptions, (b) the feedback provided was helpful, and (c) they had applied previously-provided feedback.

The feedback and guidance on the therapeutic alliance was supplemented with weekly in-person supervision that included focus on the therapeutic alliance for therapists in the Feedback condition. Supervision related to each participant began with discussion of the weekly feedback report and the participant’s clinical status and presenting concerns. Therapists were encouraged to discuss whether the report was in agreement with their impressions of the alliance. The feedback was used to guide discussion about planning for future sessions, particularly with regard to improving or maintaining the alliance and the participant’s clinical status. Strategies suggested in the feedback report were used as a starting place to guide specific approaches that might most benefit that participant’s needs. For therapists in the NF condition, supervision focused on participants’ clinical concerns and progress in a CBT framework. Therapeutic alliance was not the focus of discussion unless a therapist expressed specific concerns about the alliance. Therapists in the NF condition were unaware that FB therapists were receiving feedback reports and were blind to the study’s focus on therapeutic alliance. Representative therapist feedback text is provided in Online Supplement 1.

Therapist Selection and Training.

Six therapists delivered the individual treatment. Each had at least 5 years’ experience in AUD treatment and was credentialed (Certified Addictions Counselor). The therapists participated in intensive training, which included background/rationale for treatment, review of the treatment manual, review of previously taped examples of CBT sessions, and extensive supervised practice exercises. Information on the alliance was not included in the training. The therapists were then assigned training cases that were audio-recorded for review of (a) adherence, (b) skillfulness, and (c) appropriate structure and focus. The therapists were supervised weekly in pairs (Therapists 1 and 2, 3 and 4, and 5 and 6, respectively) for the duration of the study.

Research assessments.

At the baseline, end of treatment, and 3, 6, 9, and 12-month follow-ups, participants completed a questionnaire packet (measures described later) and a Timeline Followback interview on any alcohol consumption. Participants were paid $50 for completing each assessment. These in-person assessments included breath tests to ensure that data were gathered when participants were alcohol-free. When an in-person arrangement could not be made, the data were gathered by phone and mail. To encourage accurate self-reports, participants were assured that the data they provide at follow-up would not be made available to the therapists. In addition, participants received $15 for completing the CALPAS, WAI, and Patient Session Report after each treatment session.

Measures.

Mini International Neuropsychiatric Interview (MINI; Sheehan, 1998; Sheehan et al., 1997). The MINI was used to confirm meeting diagnostic criteria for current alcohol dependence and to assess psychotic disorders and mood disorder with psychotic features.

Mini-Mental State Examination (MMSE; Vertesi et al., 2001). The standardized MMSE is a brief and reliable assessment that measures potential cognitive deficits. Scores range from 0 to 30, with scores between 26 and 30 considered “normal” in the average population.

Demographic characteristics, and current status information (e.g., marital status, employment) were obtained using a comprehensive background questionnaire administered during the baseline assessment.

Timeline Follow-back (TLFB; Sobell and Sobell, 1992). The TLFB is a calendar-based retrospective recall interview of daily alcohol use. The TLFB was used to assess number of drinking days and number of heavy drinking days over the 6-month period prior to the baseline assessment, throughout the treatment period, and throughout the 12-month follow-up period. The reliability and accuracy of the TLFB measure have been consistently demonstrated in this population for both alcohol and other substance use (e.g., Ehrman and Robbins, 1994; Sobell and Sobell, 1992, 1996).

Treatment measures – client.

California Psychotherapy Alliance Scale (Gaston, 1991). The CALPAS was used to assess client perceptions of the therapeutic alliance. The 24-item instrument was designed to measure Patient Commitment, Patient Working capacity, Therapist Understanding and Involvement, and Working Strategy Consensus. Ratings are made on a 7-point Likert scale ranging from 1 (“not at all”) to 7 (“very much so”). The measure has been shown to possess acceptable internal consistency and reliability (Cecero et al., 2001).

Client Session Report - After each session, clients completed a Client Session Report (Project MATCH Research Group, 1997). The measure assessed (a) how difficult it was to stay abstinent, (b) how much stress clients experienced (c) clients’ confidence in remaining abstinent, (d) how motivated the client was not to drink, and (e) the amount of alcohol consumed, if any, on each day since the last treatment session.

Treatment measures – therapist.

Therapist Report on Feedback Form – At the completion of treatment of each client, therapists provided 4 ratings on a Likert-like scales on the extent to which: (1) therapists conserved the TA feedback that the client provided after each session, (2) the client’s alliance ratings were consistent with the therapist’s perceptions, (3) the client’s feedback was helpful, and (4) therapists had applied the feedback.

Data Analytic Strategy

Several sets of analyses were conducted, using a similar strategy described by Zilcha-Mano and Errázuriz (2015). Specifically, multilevel modeling (MLM; HLM 7.03, Raudenbush et al., 2017) was used to examine changes in both drinking and therapeutic alliance during treatment and post-treatment, as well as the relationship between therapeutic alliance and drinking. The data were structured on three levels: measurements were nested within participants, and participants were nested within therapists. To account for the correlation between measurements within-participant and measurements of participants with the same therapist, random intercept of participants nested within therapist and the random intercept of therapist were modeled, and all analyses were conducted as a three-level hierarchical nested model to protect against potentially biased estimates regardless of significance (see Crits-Christoph and Mintz, 1991; Wampold and Serlin, 2000). Further, data obtained at the end of each session were used for during treatment analyses, with drinking indicators of either percentage of days abstinent or percentage of days heavy drinking between each session (i.e., treatment session one represents drinking behaviors between session 1 and session 2). Both between-therapist scores and scores over time were grand mean centered. In addition, both within-person effects and between-person TA scores are deviation scores so that no further centering is needed. All models were estimated using maximum likelihood estimation. The general approach to handling missing data in multilevel modeling was used. Therefore, in this study, cases were deleted on level 1 and not on levels 2 and 3, thereby retaining the full sample. Parameters were then estimated on all available data. Further detail on statistical modeling procedures are presented in Online Supplement 2.

Results

Descriptive Statistics on Session Attendance and Therapist Ratings of Feedback

Due to our interest in the effects of therapeutic alliance feedback on overall therapeutic alliance scores and drinking outcomes, only data from participants who attended at least 2 sessions (n= 155; No-Feedback condition n=104, Feedback condition n=51) were utilized (i.e., at least one session in which feedback was potentially given to the therapist). In the experimental condition, 6 participants were dropped from the analyses, as 3 had no sessions and 3 had 1 session. In the control condition, 14 participants were not included in the analyses, as 6 had no sessions and 8 had 1 session. The overall mean number of sessions attended was 9.84 (SD=3.34), and the mean number of days between sessions was 8.54 (5.32). The counterpart mean (SD) number of sessions findings for the control sessions was 9.60 (3.35) and 10.33 (3.31) for the feedback sessions, which did not differ statistically. The mean (SD) number of days between the control sessions was 8.53 (5.31) and 8.54 (5.36) for the feedback sessions. These means also did not differ statistically.

The end of treatment therapist report on feedback data were collapsed across all therapists and all clients in cases when feedback was provided and data available (Ns for items = 51). The data suggest predominant support and valuing of the feedback. Therapists reported that they reviewed and considered the feedback provided to a great extent in all 51 cases, therapists viewed participants’ TA ratings as considerably or very consistent with those of the therapists’ perceptions in over 93% of the cases, therapists viewed the TA feedback as considerably or very helpful in their treatment of over 88% of the cases, and therapists reported that they applied the feedback to a considerable or great extent in treatment of over 96% of the cases.

Feedback and Change in CALPAS scores During Treatment

Results from model comparisons indicated that the random quadratic model was the best fitting model (deviance test of quadratic versus quadratic with random slope; χ2= 42.06, p < .001). A summary of results is presented at the top of Table 2. There was a significant increase in CALPAS score during treatment (i.e., significant linear fixed effect), with such increases slowing towards the end of treatment (i.e., significant quadratic fixed effect).

Table 2.

Final Growth Curve Models During Treatment

	CALPAS			PDA			PHD
	b	SE	p-value	b	SE	p-value	b	SE	p-value
Intercept	6.163	.063	<.001	.716	.026	<.001	.134	.017	<.001
Time (linear fixed effect)	.050	.009	<.001	.041	.006	<.001	−.018	.004	<.001
Time2 (quadratic fixed effect)	−.002	.0007	.003	−.002	.0004	<.001	.0009	.0003	.002
Random Effects Variances	Var	χ2	p-value	Var	χ2	p-value	Var	χ2	p-value
Intercept (level 3)	.012	17.43	.004	.00001	3.497	>.500	<.00001	5.029	.413
Intercept (level 2)	.190	878.18	<.001	.094	2019.74	<.001	.040	1171.23	<.001
Time (linear random effect)	.008	287.88	<.001	.004	601.87	<.001	.0013	454.886	<.001
Time2 (quadratic random effect)	.00003	209.68	<.001	.00002	401.06	<.001	<.00001	347.97	<.001
Level-1, e	.063	---	---	.0116	---	---	.008	---	---
Moderation by Feedback Condition
	b	SE	p-value	b	SE	p-value	b	SE	p-value
Intercept	6.167	.065	<.001	.716	.031	<.001	.131	.021	.001
X Feedback	−.011	.088	.904	−.004	.054	.946	.010	.036	.788
Time (linear fixed effect)	.049	.010	<.001	.049	.007	<.001	−.022	.005	<.001
X Feedback	−.024	.026	.342	−.025	.013	.051	.013	.008	.124
Time2 (Quadratic fixed effect)	−.002	.0008	.007	−.003	.0005	<.001	.0013	.0004	<.001
X Feedback	.002	.002	.376	.002	.0009	.024	−.001	.0006	.091
Random Effects Variances	Var	χ2	p-value	Var	χ2	p-value	Var	χ2	p-value
Intercept (level 3)	.012	17.43	.004	.00001	3.98	>.500	<.00001	4.813	>.500
Intercept (level 2)	.190	878.18	<.001	.094	2016.63	<.001	.040	1172.31	<.001
Time (linear random effect)	.008	287.88	<.001	.004	596.64	<.001	.0013	450.258	<.001
Time2 (quadratic random effect)	.00003	209.68	<.001	.00001	398.32	<.001	<.00001	345.62	<.001
Level-1, e	.063	---	---	.012	---	---	.008	---	---

Note: CALPAS = California Psychotherapy Alliance Scale; PDA = Percent Days Abstinent; PHD= Percent Heavy Drinking; Feedback = 0 (no feedback), 1 (feedback); b = unstandardized estimate; SE = standard error; Var = Variance estimate.

The effect of Feedback condition on changes in CALPAS scores during treatment was examined next. Results indicated no significant differences between those in the No-Feedback versus Feedback condition at the start of treatment (i.e., intercept), nor in the growth during treatment (i.e., non-significant on both the linear or quadratic effect; see bottom of Table 2).

Change in Drinking Behaviors During Treatment

Similar to the CALPAS, results from the model comparison analyses indicated that the random quadratic model best fit the data for both percentage of days abstinent (quadratic versus quadratic with random slope; χ²₍₃₎ = 125.278, p <.001) and percentage of days heavy drinking (χ²₍₃₎ = 57.917, p <.001). Specifically, there was a significant increase in percentage of days abstinent and decrease in percentage of days heavy drinking during treatment (i.e., significant linear fixed effect), with such increases for percentage of days abstinent and decreases for percentage of days heavy drinking slowing towards the end of treatment (i.e., significant quadratic fixed effect; see top of Table 2 for summary and Figure 2).

Figure 2.

Growth Curve for PDA (top panel) and PHD (bottom panel) across Treatment (Tx) Sessions by Feedback Condition

In addition, results indicated that Feedback condition approached statistical significance on the linear effect (p = .051) and was significant on the quadratic effect (see Table 2 and Figure 2). As expected, no difference at the intercept or starting values of percentage of days abstinent were noted between feedback conditions. After plotting the growth curves by feedback condition, several follow-up analyses were conducted. Results showed no difference between Feedback and No-Feedback at the end of treatment (i.e., time centered at session 12; b = −.033, SE = .042, p =.434). Next, follow-up analyses explored if there was a difference mid-way through treatment, i.e., session 6 and session 7. At session 6, there was a marginal difference, b = −.078, SE = .041, p = .060; and at session 7 the difference was significant, b = −.081, SE = .021, p < .001, such that those in the No Feedback condition had a higher percentage of days abstinent when compared to those in the Feedback condition. Taken together, receiving feedback resulted in a slower but constant improvement throughout treatment, which eventually caught up to the No-Feedback condition, whereas the no Feedback condition showed more rapid improvement that slowed over time.

With regard to percentage of days heavy drinking, no significant differences were found on the linear or quadratic effect (see Table 2, and Figure 2). Further, as expected, there was no difference at the intercept or starting values of percentage of days heavy drinking between Feedback and No Feedback conditions, nor was there a difference at the end of treatment (i.e., session 12; b = .027, SE = .029, p =.352). Finally, additional follow-up analyses revealed no significant differences at either session 6 (b =.048, SE = .027, p = .077) nor session 7 (b = .050, SE = .027, p = .068). Thus, taken together, feedback condition showed little effect on percentage of days heavy drinking.

Therapeutic Alliance Effects on Drinking Behavior During Treatment

To examine the relation between the therapeutic alliance and drinking behaviors during treatment, we conducted two 3-level MLMs in which the within-person, between-person, and between-therapist effects of alliance were entered as predictors on percentage of days abstinent or percentage of days heavy drinking controlling for time. Initial analyses revealed that Feedback condition did not moderate the within-person effect of alliance on percentage of days abstinent (b = .018, SE = .032, p = .568); thus, Feedback condition as a moderator of this effect was subsequently dropped from the analysis (see Table 3 for summary of final models). Results indicated that the within-person effect was significant (p = .015), such that as higher CALPAS scores (relative to a person’s own mean) at the end of the previous session predicted higher percentage of days abstinent during the interval until the next session. Further, results indicated a significant feedback condition × between-person effect interaction (p = .035), such that those with higher overall CALPAS scores and in the Feedback condition had higher percentage of days abstinent (estimate = .386, SE = .0070, p < .001). Further, although the relationship was stronger among those in the feedback condition (as indicated by the significant interaction), overall mean CALPAS scores for those in the No Feedback condition were also significant (estimate = .208, SE = .047, p < .001), such that higher overall mean therapeutic alliance score was associated with higher percentage of days abstinent. Finally, the between- therapist CALPAS effect was non-significant (p = .346).

Table 3.

Effect of CALPAS and Feedback Condition on Session to Session Drinking Behaviors (Final Model)

	PDA			PHD
	b	SE	p-value	b	SE	p-value
Intercept	.869	.020	<.001	.061	.013	.011
X Between-Therapist CALPAS	−.193	.181	.346	.222	.124	.146
X Feedback	−.063	.033	.060	.041	.023	.079
X Between-Person CALPAS	.208	.047	<.001	−.158	.027	<.001
X Feedback × Between-Person CALPAS	.179	.084	.035	---	---	---
Time	.018	.002	<.001	−.008	.001	<.001
Within-Person CALPAS*	.038	.015	.015	−.036	.016	.033
Random Effects Variances	Var	χ2	p-value	Var	χ2	p-value
Intercept (level 3)	<.00001	5.285	.258	<.00001	3.190	>.500
Intercept (level 2)	.037	3147.86	<.001	.017	2469.63	<.001
Time (linear random effect)	.0006	756.53	<.001	.0002	506.76	<.001
Within-person WAI (random effect)	.0031	146.94	.463	.015	261.42	<.001
Level-1, e	.014	---	---	.008	---	---

Note: CALPAS = California Psychotherapy Alliance Scale; PDA = Percent Days Abstinent; PHD= Percent Heavy Drinking; Feedback = 0 (no feedback), 1 (feedback); b = unstandardized estimate; SE = standard error; Var = Variance estimate

^*Feedback Condition was also examined as a cross-level moderator on the Within-Person CALPAS effect, but was found to be non-significant in both models and thus trimmed to examine the main effect of the Within-Person CALPAS effect (see results section).

With regard to heavy drinking days, feedback condition did not moderate the within- person effect of alliance on percentage of days heavy drinking (b = .005, SE = .035, p = .885) nor the between-person CALPAS effect (estimate = −.158, SE = .105, p = .135) and thus the cross-level interaction (level 1) and feedback × between-person CALPAS effect interaction on level 2 were dropped from the final model (see Table 3). Results from the final model indicated that the within-person CALPAS effect was significantly associated with percentage of days heavy drinking (p =.011), such that the higher one’s therapeutic alliance score at the end of a treatment session predicted lower percentage ofdays heavy drinking in the interval before the next session. Results further indicated a significant association between the between-person CALPAS effect and percentage of days heavy drinking (estimate = −.158, SE = .027, p < .001), such that those with higher overall CALPAS scores had lower percentage of days heavy drinking regardless of treatment condition. Finally, the association between the between-therapist CALPAS effect and percentage of days heavy drinking was non-significant (p = .146).¹

Post-Treatment Drinking: Effect of Feedback Condition and Between-Person Alliance Scores

To further examine the effect of Feedback Condition and between-person alliance effects on drinking following treatment, analyses were conducted using summaries of percentage of days abstinent and percentage of days heavy drinking during the post-treatment assessment intervals (i.e., total of 5 assessments). Results from model fit comparison analyses indicated that the random linear model best fit the data for percentage of days abstinent (linear versus random linear model; χ2 = 107.58, p < .001), and that the addition of a quadratic effect was non-significant (χ2= 0.0001, p > .500). Similar results were found for percentage of days heavy drinking model fit (linear versus random linear model; χ2 = 82.49, p < .001), with the addition of a quadratic effect being non-significant (χ2 = 0.152, p > .500). Overall, results from both outcomes indicated a small but significant decline in percentage of days abstinent and small but significant increase in percentage of days heavy drinking over the course of the follow-up intervals (see top of Table 4 for summary).

Table 4.

Final Growth Curve Models During Post Treatment

	PDA			PHD
	b	SE	p-value	b	SE	p-value
Intercept	.811	.022	<.001	.095	.016	.002
Time (linear fixed effect)	−.015	.006	.014	.011	.005	.029
Random Effects Variances	Var	χ2	p-value	Var	χ2	p-value
Intercept (level 3)	<.00001	2.517	>.500	<.00001	3.534	>.500
Intercept (level 2)	.0639	1243.044	<.001	.032	812.07	<.001
Time (linear random effect)	.0036	492.770	<.001	.002	396.72	<.001
Level-1, e	.0144	---	---	.012	---	---
Moderation by Feedback Condition and Alliance Effects
	b	SE	p-value	b	SE	p-value
Intercept	.830	.234	<.001	.087	.018	.008
X Between-Therapist CALPAS	−.196	.218	.420	.355	.164	.097
X Feedback Condition	−.060	.041	.149	.023	.031	.460
X Between-Person CALPAS	.231	.058	.<.001	−.149	.044	<.001
X Feedback Condition × Between-Person CALPAS	.186	.106	.082	−.124	.080	.124
Time (linear fixed effect)	−.015	.007	.014	.009	.006	.124
X Feedback Condition	.010	.012	.438	.004	.010	.674
X Between-Person CALPAS	.0001	.018	.994	−.0002	.015	.990
X Feedback Condition × Between-Person CALPAS	−.016	.031	.608	−.007	.026	.779
Random Effects Variances	Var	χ2	p-value	Var	χ2	p-value
Intercept (level 3)	<.00001	4.107	.392	<.00001	3.22	>.500
Intercept (level 2)	.049	955.80	<.001	.026	668.26	<.001
Time (linear random effect)	.004	487.83	<.001	.002	395.70	<.001
Level-1, e	.014	---	---	.012	---	---

Note: CALPAS = California Psychotherapy Alliance Scale; PDA = Percent Days Abstinent; PHD= Percent Heavy Drinking; Feedback = 0 (no feedback), 1 (feedback); b = unstandardized estimate; SE = standard error; Var = Variance estimate. The model was also explored removing the interaction term to examin main effects, but results continue to remain non-significant or marginal, thus full models tested are presented (see results section for more detail

Next, we examined the effects of feedback condition, between-person alliance effect and between-therapist alliance effect on post-treatment drinking. Results indicated that the feedback condition × between-person effect did not moderate the trajectory of percentage of days abstinent during the post-treatment intervals (p = .608), nor was it associated with starting values (i.e., on the intercept, p = .082). To further explore the independent effect of both feedback condition and between-person alliance on percentage of days abstinent, the condition × between-person alliance interaction was removed from the model. Results indicated that neither feedback condition (b = .010, SE = .012, p = .441) nor between-person alliance effect (b = −.005, SE = .015, p = .737) moderated the trajectory of percentage of days abstinent during post-treatment. In contrast, the between-person alliance effect was significantly and positively related to percentage of days abstinent at the end of treatment (b = .286, SE = .049, p < .001), such that the higher a participant’s overall mean alliance score the higher the percentage of days abstinent at the end of treatment continuing through the 12-month follow-up. There was no effect of feedback condition (b = −.055, SE = .045, p = .220) or between-therapist alliance rating on the intercept (b = −.145, SE = .218, p = .542).

Finally, similar to percentage of days abstinent, the feedback condition × between-person alliance interaction was non-significant on the trajectory of percentage of days heavy drinking over the course of post-treatment intervals (p = .779), and was not significant on starting values, p = .124. The between-therapist alliance effect also was non-significant (p = .097). Further inspection of the main effects (i.e., removal of the feedback × between-person alliance effect interactions), also revealed that neither feedback (b = −.004, SE = .010, p = .669) nor the between-person alliance effect moderated the trajectory of percentage of days heavy drinking over the course of follow-up (b = −.003, SE = .012, p = .832). However, similar to percentage of days abstinent analyses, the between-person alliance effect was significantly related to percentage of days heavy drinking at the end of treatment (b = −.187, SE = .037, p < .001), such that the higher a participant’s overall mean alliance score the lower the percentage of days heavy drinking at the end of treatment continuing through the 12-month follow-up. There was no effect of feedback condition (b = −.023, SE = .031, p = .469) or between-therapist alliance rating on the intercept (b = .317, SE = .164, p = .126).

Discussion

The results of this study did not support its main hypotheses, as feedback condition affected neither alliance scores nor drinking within or following treatment. Feedback did interact with the between-patient component of variance in the alliance data, but that result was not predicted. Therefore, the data did not support the hypothesized relations among the feedback manipulation, the alliance, and drinking outcomes. There are several factors that could plausibly help to explain this pattern of findings. The first is that this study did not include procedures that could be used to assess what therapists actually did in response to the alliance feedback that they received following each session. In this regard, although the therapist survey data suggest that therapists valued the alliance feedback and tended to incorporate it in their work with clients, there may have been considerable variability among the therapists in the degree and manner in which that happened. Notably, the now considerable literature on therapist feedback informed treatment (Schuckard et al., 2017) has shown clearly that there may be little relation between therapists’ reports of the degree of usefulness of session-by-session feedback about client functioning and how much they actually use it, and, as discussed further later, that the latter variable seems to be the critical determinant of the strength of relation between feedback and client or patient outcomes.

A second factor that may help to explain the lack of a feedback effect is the restricted range that was observed in CALPAS scores. In this study, overall patients’ CALPAS scores started high and tended to increase slowly for about the first two-thirds of the treatment sessions and then plateaued over the rest of treatment. At that point, the average CALPAS item score was close to the maximum of seven. Such a restricted range for change may have been insensitive to the effects of therapist feedback. In this regard, given the reasonable chance for ceiling effects to occur in the CALPAS (as well as in the other major alliance measure, the WAI (Mallinckrodt & Tekie, 2016)), future research likely would benefit by systematically measuring changes in alliance levels during the course of treatment, especially declines, which have been shown to predict treatment outcomes (Larrson et al., 2018). Although within-treatment sessions changes were not monitored systematically in this study, it is noteworthy that the project’s clinical supervisor (author RLD) observed few such declines in the alliance across therapists and patients.

Besides limitations in the CALPAS, ceiling effects in any alliance measure could result from a failure to establish a climate in which clients feel safe in being honest about their perceptions of how much (or little) therapy is helping them or about the therapeutic relationship. Outside of such a context, clients’ reports could tend to be far more positive in some cases than an accurate report of the alliance or outcomes would be. Of course, client feedback about the alliance or about therapy outcomes in general can be only as effective as the accuracy of the information that the client provides (Maeschalck and Barfknecht, 2017). In this study, there was no explicit effort made to create a context that normalized the solicitation and application of client feedback, which could have resulted in an attenuation of the power of the feedback manipulation to some unknown degree. This point is central not only for research on alliance feedback but also for its implementation into clinical practice.

This study replicated the Connors et al. (2016) finding of an inverse relation between level of the alliance at the end of a given treatment session and the degree of alcohol involvement as measured by both percentage of days abstinent and percentage of days heavy drinking during the time to the following session. However, this study did not replicate Prince et al.’s (2016) findings of an inverse relation between the alliance and alcohol involvement following treatment. Besides the feedback manipulation tested in this study, the major difference between this study and Connors et al. and Prince et al. was in how the alliance data were analyzed. In this regard, the two latter studies did not partition variance in the alliance into between- and within-patient effects and between-therapist effects as was done in this study. That difference may have resulted in an overall reduction in the strength of relation between patients’ alliance reports and their drinking following treatment.

As mentioned earlier, the one effect involving feedback condition found in this study was the feedback by between-patients measure of the alliance interaction in predicting drinking during treatment. Average alliance scores during treatment were related to percentage of days abstinent but only for patients in the Feedback condition. Zilcha-Mano and Errázuriz (2015) found the same interaction in their data but with the within-patient alliance effect, though they offered no explanation of this finding. In the current study, the feedback manipulation may have reduced within-patient variability, but there still was sufficient between-therapist variance to produce the moderation effect. More substantively, it can be speculated that, relevant to a point made earlier, therapists may have applied feedback more diligently on an alliance from participants that tended to be more positive, with positive effects on the following week’s frequency of alcohol consumption.

Not investigated systematically in this study but seems critical is to understand what discriminates among therapists who continue to enhance their therapeutic skills as a result of using client feedback from those who do not. In this regard, two factors that seem important are study and repeated review of the client feedback data that they receive and engagement in deliberate practice toward the end of improved clinical skills (Chow et al., 2015). In this study, therapists reviewed alliance feedback in weekly supervision. Using suggestions that were offered on the feedback reports, therapists brainstormed with the supervisor and the other therapist they were paired with in supervision to tailor the feedback to their individual clients. Ultimately, how therapists used the feedback was at their discretion. Future research might consider development of specific guidelines derived from a data base of clinical cases that consists of patterns of feedback over the course of therapy and associated therapeutic action that followed from them (Maeschalck and Barfknecht, 2017).

There are several strengths of this study that warrant mention. An experimental design was used that allowed inferences about the causal connection between therapist feedback and alliance changes in treatment. Furthermore, the design of using repeated measurements of both the alliance and (alcohol use) outcomes enhanced the ability to make potential causal inferences about the effects of alliance feedback to therapists. Another strength is that this study involved the participation of experienced therapists who used an evidence-based treatment that is implemented widely in the outpatient treatment of AUD.

There also were limitations of this study that should be considered in interpreting its findings. Self-report was used to measure both alcohol consumption and the alliance. However, the Timeline Followback method of obtaining retrospective self-reports of daily alcohol and other drug use has been studied and used extensively in substance use disorder treatment research and has been shown to have excellent reliability and validity in the context of treatment research (Robinson et al., 2014). In addition, the primary interest in this study was patients’ perceptions of the alliance, and for that patients’ self-reports are the best source. Furthermore, its measurement limitations notwithstanding, the CALPAS is a widely used measures of the alliance that has demonstrated good indices of reliability and validity (Hatcher, 2010). Also notable is that this study’s convenience sample was representative of individuals presenting for treatment at the clinical research site, but its findings cannot be seen as representative of all patients presenting for AUD treatment. Finally, the sample size for the feedback condition (n=51) was modest.

In conclusion, the results of this study did not support its hypothesis of a causal connection between therapist feedback about the alliance, change (improvement) in the strength of the alliance, and subsequent alcohol involvement outcomes. Nevertheless, the pattern of findings yielded by this methodologically strong and innovative study suggests several topics for future alliance research. Given the data on the effects of session-by-session feedback to therapists about their patients’ functioning, foremost among these may be understanding therapists’ decisions to apply feedback about the alliance to their practice (or not), how they go about doing so, and what modifications in their practice may be most effective. Such research has the potential to improve the effectiveness of therapist training and ultimately patient outcomes on a wide scale. In addition, moderator analyses, based on between-therapist, between-patient effects may be another productive line of future research, as suggested by the Zilcha-Mano and Errázuriz (2016), given the proportion of variance in the alliance due to the random patient effect in alliance growth during treatment that was found in the current study. Overall, it would appear that continued study of ways to enhance therapists’ relationship building and technical skills in conducting psychotherapy ultimately would begin to address the relative inattention paid to therapist effects in psychotherapy and thereby enhance patient outcomes.