Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Apr 1.
Published in final edited form as: J Consult Clin Psychol. 2018 Apr;86(4):398–402. doi: 10.1037/ccp0000293

When therapist estimations of the process of treatment can predict patients rating on outcome: the case of the working alliance

Sigal Zilcha-Mano 1, J Christopher Muran 2, Catherine F Eubanks 3, Jeremy D Safran 4, Arnold Winston 5
PMCID: PMC5901886  NIHMSID: NIHMS940097  PMID: 29648859

Abstract

Objective

It has been demonstrated that patient perspective on alliance can predict subsequent treatment outcome as reported by the therapist, but not the other way around. This study aimed to investigate the circumstances in which therapists can provide estimations of alliance capable of predicting patient perceptions of subsequent session outcome. The study focused on two potential indicators: time in treatment and treatment efficacy.

Method

Data of 107 treatment completers assigned to either cognitive behavioral therapy or alliance focused therapy were analyzed. Data included session-to-session assessments of alliance and the session outcome measure across the 30 weeks of treatment, as well as pre-treatment to post-treatment assessments of treatment efficacy using the Symptom Checklist-90-Revised. An actor-partner interdependence model of longitudinal hierarchically nested data, disentangled for within- and between-patients effects, was used. The interactive effects of time and treatment efficacy and their combined effect were examined.

Results

At the sample level, as expected, the therapist perspective of alliance did not significantly predict patient perception of subsequent session outcome, but significant interaction effects were detected. Therapists’ perspective on the alliance was a stronger predictor of patients’ perception of subsequent session outcome when therapy was more rather than less effective. This effect was evident mainly early in treatment, where greater variability across dyads was found.

Conclusions

Findings suggest that although therapists’ ratings of the alliance are not consistently predictive of patients’ rating of subsequent session outcome, they are better predictors in more than in less effective treatments.

Keywords: alliance, APIM, treatment efficacy, alliance-focused treatment

The working alliance is one of the most reliable predictors of treatment outcome. Nevertheless, the ability of the therapist’s perspective of the alliance to predict outcome has shown much variability (Horvath et al., 2011), and its ability to predict treatment outcome as evaluated by patients or external observers has been challenged in recent years (Zilcha-Mano et al., 2015). Whereas patient perspectives of alliance have been found to significantly predict both their own and their therapists’ perception of outcome, therapist perspectives were found to be a consistent predictor only of their own perception of outcome, not of their patients’ (Kivlighan et al., 2016; Kivlighan et al., 2014; Zilcha-Mano et al., 2016).

The failure of the therapists’ perspective of alliance to predict patient perceptions of outcome raises grave concerns about the ability of therapists to identify harmful processes that may increase the risk that patients would not see improvement in treatment or might drop out of treatment. This risk is consistent with findings suggesting that therapists fail to identify indicators that predict patients’ evaluations of the efficacy of treatment for them. For example, therapists’ evaluation of the process of treatment was not found to be an indicator of what patients report as unsuccessful treatment that resulted in deterioration (Hannan et al., 2005). These findings are particularly troubling when coupled with research evidence showing that therapists are reluctant to recognize deterioration processes and tend to overestimate improvement rates (Norcross, 2003; Walfish, McAlister, O’Donnell, & Lambert, 2012). This troublesome finding attests to the importance of improving therapists’ ability to identify harmful processes, which may result in patients evaluating the treatment as unhelpful. One promising path to achieve this is to identify factors that determine the circumstances under which therapists can estimate therapeutic elements, such as aspects of the working alliance, that affect patient evaluations of treatment outcome.

Only one study to date examined the circumstances under which the therapist’s perspective of alliance can predict patient perceptions of outcome over the course of treatment (Kivlighan et al., 2016), with the therapist’s perspective of alliance being a stronger predictor of patient perception of session quality in more effective treatments than in less effective ones. Time in treatment was also identified as a potential moderator, but not directly for the alliance, but rather for the extent to which the therapeutic relationship was genuine (Gelso et al., 2014). It is an open question whether time in treatment and treatment efficacy may indicate when therapist perspective on alliance can predict patient perception of subsequent session outcome. Although of great importance, no study to date has examined this question while disentangling the trait-like (between-patients differences) and state-like (within-patient changes) components of alliance (Zilcha-Mano, 2016, 2017). When exploring the process of change during treatment, it is crucial to focus on the state-like component of alliance, disentangling it from general trait-like individual differences, which may act as confounders (Falkenström et al., 2016; Zilcha-Mano, 2017).

This study aimed at investigating the circumstances in which therapists can provide estimations of alliance capable of predicting patient perceptions of subsequent session outcome, focusing on the state-like component of alliance. We examined this question in a dataset that has previously been used to demonstrate the ability of patient-rated alliance to predict therapist-rated outcome, and the inability of therapist-rated alliance to predict patient-rated outcome (Zilcha-Mano et al., 2016). We expect that although at the sample level, the therapist’s perspective of alliance does not predict patient perception of subsequent session outcome, when therapy is more rather than less effective, therapists’ perspective on the alliance will predict patients’ perception of subsequent session outcome. We also expected time in treatment to serve as a significant moderator, although not enough findings are available to suggest the direction of the effect.

Method

Participants

Data of 107 patients, who were assigned to either cognitive-behavioral therapy (CBT) or alliance focused therapy (AFT) and who completed the last session of treatment, were included. Given the focus on treatment efficacy as a potential factor, only completers’ data were used, where the post-treatment outcome assessment was collected. The study was approved by the IRB of the relevant institution. Demographic and diagnostic information for this subsample appears in Table 1.

Table 1.

Demographic and clinical patient characteristics

Variable
Demographics
 Age in years, mean (SD) 43.69 (SD = 13.48)
 Female 68 (63.6%)
 Caucasian 88 (87.1%)
 Single 59 (55.1%)
 College graduate 42 (39.3%)
 Employed 85 (79.4%)
Primary Axis I diagnosis
 Mood disorder 55 (51.4%)
 Anxiety disorders 21 (19.6%)
 Adjustment disorder 5 (4.7%)
 Multiple Axis I diagnoses 56 (52.3%)
Primary personality disorder diagnosis
 PD NOS 29 (25.5%)
 Dependent 1 (0.9%)
 Avoidant 16 (15%)
 Obsessive-compulsive 10 (9.3%)
 Paranoid 2 (1.9%)
 Narcissistic 1 (0.9%)
 Negativistic 2 (1.9%)
 Depressive 6 (5.7%)
 Multiple personality disorder diagnoses 16 (15%)
Open in a new tab

Notes: Values shown as % (n) unless otherwise noted. PD = personality disorder.

Treatment and therapists

Two treatment models were used: CBT, which focused on symptom reduction and schema change, using strategies such as self-monitoring, cognitive restructuring, behavioral exercises, and experimentation (N = 52, Turner & Muran, 1992), and AFT, also known as Brief Relational Therapy (Muran, 2002; Muran et al., 2005), which is based on relational concepts and involves negotiating alliance ruptures between the patient and therapist (N = 55, Safran & Muran, 2000). Both were manualized and designed to treat patients in a fixed, 30-session, one-session-per-week format. External observer coding of adherence revealed high adherence in both models. Ninety-six therapists participated in the study.

Measures

Working alliance

The quality of the working alliance was assessed with the 12-item patient and therapist versions of the Working Alliance Inventory (WAI; Tracey & Kokotovic, 1989). Internal reliability range across time points was .88–.94 for patients and .83–.93 for therapists.

Treatment efficacy

Treatment efficacy was assessed with the Global Severity Index (GSI) of the Symptom Checklist-90-Revised (SCL-90-R: Derogatis, 1983). Internal reliability of the GSI was .86 pre-treatment and .84 post-treatment.

Session outcome

Session outcome was assessed with the one-item session outcome measure (Muran et al., 1992), after each session, for 30 weeks. A single item was used to accommodate the time constraints of patients and therapists and to minimize self-report burnout. The one item was answered by both patients (“To what extent are your presenting problems resolved?”) and therapists (“To what extent are your patient’s presenting problems resolved?”) on a Likert scale, ranging from 1 (not at all) to 9 (completely).1

Procedure

After describing the study to patients, written informed consent was obtained. Patients completed the SCL before starting treatment and at termination. Patient and therapist session outcome and working alliance ratings were collected at each session. Further details on the design and procedures are described elsewhere (Muran, 2002; Muran et al., 2005).

Statistical analyses

To examine when therapist-rated alliance at one session can predict patient-rated outcome at the next session, we used an Actor-Partner Interdependence Model (APIM, Kenny et al., 2006) of longitudinal hierarchically nested models of sessions within individuals, individuals within therapeutic dyads, and dyads within therapists. Each alliance effect was introduced in the model with two terms, one for therapist-rated, the other for patient-rated alliance. Analyses were conducted using the SAS PROC MIXED procedure (SAS Institute, 2003), assigning random effects to dyads and therapists. A model of fixed effect of log of time and a random intercept was found to demonstrate the best model fit in predicting session outcome based on the log likelihood test and the AIC criterion. The APIM enabled us to adequately test the therapist’s partner effect, that is, the effect of therapist-reported alliance on patient-reported outcome, which was the focus of the present study. We examined two potential factors to determine when therapist-rated alliance at one session can predict patient-rated outcome at the next session: treatment efficacy (as defined by changes in SCL from pre- to post-treatment) and time in treatment (defined as the present session number). Given our interest in within-person effects, we followed Wang and Maxwell’s (2015) method to disentangle within- and between-patients effects, by centering patient-reported and therapist-reported alliance within the individual patient’s mean, and using the individual patient’s mean for patient-reported and therapist-reported alliance as the between-patients effect. We used the between-patients effects as covariates, together with patient pre-treatment symptom severity, as evaluated by the SCL.

Results

The estimated variance of the therapist’s random effect was null, whereas the variance of the dyad-level random effect was significant (S2 = 0.99, p < 0.0001, ICC = .34).2 The partner effect of therapist-reported alliance on patient-reported subsequent session outcome was not significant (B = 0.03, SE = 0.04, t(6594) = 0.72, p = .47), but significant interaction effects were found. A significant interaction was found between treatment efficacy and changes in within-patient alliance, as rated by the therapist, in predicting subsequent session outcome, as rated by the patient (B = −.23, SE = 0.11, t(4029) = −2.06, p = .03): in more effective treatments, an increase in therapist-reported alliance predicted a greater increase in patient-reported session outcome (partner effect) than in less effective treatments. There was no significant interaction for time (log of time) in treatment (B = .06, SE = 0.07, t(4033) = 0.84, p = .39), but a significant 3-way interaction was found between the partner effect, time, and treatment efficacy (F(1,4069) = 12.34, p = .0004; Figure 1): at early sessions of more effective treatment, higher levels of therapist-rated alliance predicted better patient-rated outcome than in less effective treatment (B = 1.29, SE = 0.37, t(4068) = 3.42, p = .0006 for more effective, and B = −0.46, SE = 0.13, t(4052) = −3.52, p = .0004 for less effective). This effect was restricted to early sessions and was not significant in later ones (B = −0.02, SE = 0.11, t(4041) = −0.19, p = .84 for more effective, and B = 0.15, SE = 0.10, t(4044) = 1.35, p = .17 for less effective). In more effective treatments, the variance in the outcome between patients was higher in early sessions (S2 = 1.60, S.E. = 0.53, Z = 0.85, p = .001) than in later ones (S2 = 0.10, S.E. = 0.10, Z = 1.05, p = .14), producing greater variance to be explained by potential interaction effects. All findings remained similar when controlling for treatment condition.3

Figure 1.

Figure 1

Open in a new tab

Three-way interaction of the role of time and treatment efficacy in the association between therapist-reported alliance and patient-reported subsequent session outcome. Axis Y represents the coefficients of the within-patient therapist-reported alliance predicting subsequent patient-rated outcome. Treatment efficacy was based on treatment outcome (measured as the change in SCL from pre- to post-treatment).

Discussion

Whereas the findings do not support the claim that therapist perspective on alliance is generally a good predictor of subsequent patient-reported session outcome, they do suggest that therapist perspective on alliance can be more predictive of outcome in some circumstances than in others. Specifically, in more effective treatments, therapist perspective on the alliance had a stronger relation to patient perception of outcome, than in less effective ones. These findings are consistent with the two previous studies that focus on treatment efficacy as a moderator (Kivlighan et al., 2016; Kivlighan, Marmarosh, & Hilsenroth, 2014). Findings also revealed that the interaction effect of treatment efficacy was evident early but not later in treatment, perhaps because of greater variability of the association between alliance and session outcome earlier in treatment. This finding is consistent with the only other study that examined the moderating effect of both time and treatment efficacy, and found that in more effective treatment patient-rated alliance was a significant predictor of patient-rated session outcome in early sessions, but not in later ones (Kivlighan et al., 2016).4

The findings suggest that when a patient is deemed to be off-track for success, therapists should be cautious in assuming that their ratings of alliance are meaningful indicators of improvement for the next session outcome. One possible post hoc explanation is that therapists may be resistant to reporting problematic alliance in their poor outcome cases, or oblivious to it (Safran et al., 2001). The finding that patient-rated alliance is a predictor of subsequent patient- and therapist-reported treatment outcome may support such a proposition. Another possibility is that more competent therapists are more skillful in detecting an aspect of treatment (as manifested here in the alliance) that results in what patients view as better subsequent outcome, and also conduct more effective treatments.

The findings suggest that when treatment is effective, therapists tend to consistently produce alliance estimates that predict outcome. This may introduce a bias in the therapists’ estimations regarding their ability to monitor the alliance when treatment is not proceeding as expected (e.g., Sanitioso et al., 1990), and may erroneously cause therapists to assume that treatment failures are not the result of poor alliance. Thus, therapists should exercise caution when inferring that poor alliance is not responsible for poor outcome, because in the case of poor outcome their judgment should be treated with skepticism. Our findings are consistent with evidence that providing therapists with patient progress feedback when treatment is off-track may be a particularly effective way of reducing treatment failure (Lambert, 2015; Shimokawa, Lambert, & Smart, 2010). The findings suggest that when treatment is predicted to be less effective, therapists should be made aware of this fact, and receive feedback on patient alliance and other process variables. Without the benefit of systematic feedback, therapists should continuously monitor the process and outcome with patients, and actively solicit feedback from them. Our post hoc analyses regarding the moderating effect of treatment conditions point to the potential benefits of training therapists on this topic.

This is the first study to examine the circumstances under which therapist perspective on alliance can predict subsequent patient-rated outcome, disentangling the within-patient effects from more trait-like characteristics of the patient and the dyad (e.g., general tendencies to rate the alliance as stronger). Other strengths of the study include the use of session-to-session assessments, separate measures of treatment efficacy and session outcome, and statistical analyses that account for the interdependence between patients and therapists and made possible the establishment of a correct temporal relationship between alliance and outcome, which is especially important when focusing on therapists’ perspective of alliance (Zilcha-Mano, Solomonov et al., 2015). The study has several limitations, including (a) a small sample size, which precluded us from examining potential 4-way interactions focusing on differences between treatment conditions in these moderation effects. Future studies should test this 4-way interaction to investigate whether treatment that incorporates training procedures designed specifically to improve therapists’ ability to work constructively with weakened alliances and negative therapeutic processes produces different effects from treatment that does not include such a focus. Other limitations include: (b) the inability to infer causality regarding the role that treatment efficacy plays in the ability of therapist-rated alliance to predict subsequent patient-rated outcome because no temporal relationship could be established; (c) inclusion only of patients who completed the post-treatment SCL assessment at the end of the 30 sessions (thus, excluding those who dropped out); (d) use of a single item to evaluate session outcome; and (e) reliance on patient report on outcome, which may capture only part of the session outcome.

Public Health Significance.

Only for patients who ultimately have favorable outcomes does therapist-rated alliance predict next-session patient-rated improvement. For patients who ultimately have unfavorable outcomes, the therapists’ judgment of alliance quality is a poor prognosticator of patient experience. Although poor alliance could still contribute to poor outcome, the results suggest that therapists cannot trust themselves to make this connection on the spot. Training time and effort should be allocated to enhance the therapist’s ability to identify negative relational processes, including by exploring patient experience, especially when the therapy is off-track.

Footnotes

1

Treatment efficacy and session outcome are related but distinct constructs. Whereas one focuses on the extent to which the treatment as a whole was effective in reducing the patient’s symptoms, the other focuses on the extent to which the patients’ problems were resolved at any given session. The constructs were assessed using separate measures. Calculating the correlation between the outcome measure as used in the main model (each weekly level of the session outcome) and treatment efficacy as used in the main model (changes in SCL from pre-treatment to post-treatment) revealed an average correlation of −0.09, with the highest across all the 30 correlations being −.265 (which became insignificant after adjusting for the multiple correlations). Thus, although the slope of change in session outcome over the course of treatment and the slope of change in overall treatment outcome from pre-treatment to post-treatment were significantly associated, the patients’ scores on the session outcome throughout treatment were not significantly correlated with the change in SCL from pre- to post-treatment.

2

There were no significant differences in mean SCL reduction from pre- to post-treatment between the AFT (M = 0.20, SD = 0.44) and CBT (M = 0.29, SD = 0.47) conditions (t(105) = −.99, p = .32). Nor was there significant interaction between treatment condition and time (in log of session number) in predicting session outcome (F(1,8214) = 0.36, p = .55).

3

We conducted two post hoc analyses to test the potential moderating effect of type of treatment on the moderating effect of time and treatment success. The analyses revealed: (a) a 3-way interaction between time, type of treatment, and the alliance partner effect, which resulted in a significant effect (F(1,6625) = 6.1, p = .01), suggesting that therapist-rated alliance affects patient-rated outcome mostly in AFT, mostly later in treatment; this significant interaction confirms the previously reported interaction with treatment condition that was found in this data (Zilcha-Mano, Muran et al., 2016); and (b) a 3-way interaction between treatment efficacy, type of treatment, and the alliance partner effect, which resulted in a non-significant effect (F(1,4023) = 0.21, p = .65). Findings should be interpreted with caution because of the small sample size.

4

We found no significant interaction effect of time in treatment. This is consistent with the only other study that examined time as moderator of therapist-alliance effect on patient-outcome (Kivlighan et al., 2016). Even when focusing on the real relationship aspect of the therapeutic relationship, the findings regarding the potential moderating effect of time were mixed (Kivlighan et al., 2016 vs. Markin, Kivlighan, Gelso, Hummel, & Spiegel, 2014). It is also important to stress that treatments were relatively long in the present study (30 sessions).

References

  1. Derogatis LR. SCL-90-R revised manual. Baltimore, MD: John Hopkins School of Medicine; 1983. [Google Scholar]
  2. Falkenström F, Ekeblad A, Holmqvist R. Improvement of the working alliance in one treatment session predicts improvement of depressive symptoms by the next session. Journal of Consulting and Clinical Psychology. 2016;84(8):738–751. doi: 10.1037/ccp0000119. http://dx.doi.org/10.1037/ccp0000119. [DOI] [PubMed] [Google Scholar]
  3. Gelso C. A tripartite model of the therapeutic relationship: Theory, research, and practice. Psychotherapy Research. 2014;24:117–131. doi: 10.1080/10503307.2013.845920. [DOI] [PubMed] [Google Scholar]
  4. Hannan C, Lambert MJ, Harmon C, Nielsen SL, Smart DM, Shimokawa K, Sutton SW. A lab test and algorithms for identifying patients at risk for treatment failure. Journal of Clinical Psychology: In Session. 2005;61:155–163. doi: 10.1002/jclp.20108. [DOI] [PubMed] [Google Scholar]
  5. Horvath AO, Del Re A, Flückiger C, Symonds D. Alliance in individual psychotherapy. Psychotherapy. 2011;48:9–16. doi: 10.1037/a0022186. [DOI] [PubMed] [Google Scholar]
  6. Kivlighan DM, Jr, Hill CE, Gelso CJ, Baumann E. Working alliance, real relationship, session quality, and client improvement in psychodynamic psychotherapy: A longitudinal actor-partner interdependence model. Journal of counseling psychology. 2016;63(2):149–161. doi: 10.1037/cou0000134. http://dx.doi.org/10.1037/cou0000134. [DOI] [PubMed] [Google Scholar]
  7. Kivlighan DM, Jr, Marmarosh CL, Hilsenroth MJ. Client and therapist therapeutic alliance, session evaluation, and client reliable change: A moderated actor-partner interdependence model. Journal of Counseling Psychology. 2014;61(1):15–23. doi: 10.1037/a0034939. [DOI] [PubMed] [Google Scholar]
  8. Lambert MJ. Progress feedback and the OQ-system: The past and the future. Psychotherapy. 2015;52(4):381–390. doi: 10.1037/pst0000027. [DOI] [PubMed] [Google Scholar]
  9. Markin RD, Kivlighan DM, Jr, Gelso CJ, Hummel AM, Spiegel EB. Clients’ and therapists’ real relationship and session quality in brief therapy: An actor partner interdependence analysis. Psychotherapy. 2014;51:413–423. doi: 10.1037/a0036069. [DOI] [PubMed] [Google Scholar]
  10. Muran JC. A relational approach to understanding change: Plurality and contextualism in a psychotherapy research program. Psychotherapy Research. 2002;12:113–138. [Google Scholar]
  11. Muran JC, Safran JD, Samstag LW, Winston A. Evaluating an alliance-focused treatment for personality disorders. Psychotherapy: Theory, Research, Practice, Training. 2005;42:532–545. [Google Scholar]
  12. Muran J, Safran J, Samstag L, Winston A. Patient and therapist postsession questionnaires, Version 1992. New York, NY: Beth Israel Medical Center; 1992. [Google Scholar]
  13. Safran JD, Muran JC. Negotiating the therapeutic alliance: A relational treatment guide. New York: Guilford; 2000. [Google Scholar]
  14. Safran JD, Muran JC, Samstag LW, Stevens C. Repairing alliance ruptures. Psychotherapy. 2001;38(4):406. doi: 10.1037/a0022140. [DOI] [PubMed] [Google Scholar]
  15. Shimokawa K, Lambert MJ, Smart DW. Enhancing treatment outcome of patients at risk of treatment failure: meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting & Clinical Psychology. 2010;78(3):298. doi: 10.1037/a0019247. [DOI] [PubMed] [Google Scholar]
  16. Tracey TJ, Kokotovic AM. Factor structure of the working alliance inventory. Journal of Consulting and Clinical Psychology. 1989;1:207–210. [Google Scholar]
  17. Turner AE, Muran JC. Cognitive-behavioral therapy for personality disorders: A treatment manual. San Diego, CA: Social & Behavioral Documents; 1992. [Google Scholar]
  18. Walfish S, McAlister B, O’Donnell P, Lambert MJ. An investigation of self-assessment bias in mental health providers 1. Psychological Reports. 2012;110:639–644. doi: 10.2466/02.07.17.PR0.110.2.639-644. [DOI] [PubMed] [Google Scholar]
  19. Zilcha-Mano S. Is alliance really therapeutic? A systematic answer based on recent methodological developments. American Psychologist. 2017 doi: 10.1037/a0040435. [DOI] [PubMed] [Google Scholar]
  20. Zilcha-Mano S, Solomonov N, Chui H, McCarthy KS, Barrett MS, Barber JP. Therapist-reported alliance: Is it really a predictor of outcome? Journal of Counseling Psychology. 2015;62(4):568–578. doi: 10.1037/cou0000106. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES