Therapist effects and the outcome—alliance correlation in cognitive behavioral therapy for panic disorder with agoraphobia

Abstract

Although the alliance–outcome correlation is well established, no published studies to date have separated between therapists’ and patients’ contributions while controlling for early symptom change. In this study, we examined therapist effects in two trials of CBT for panic disorder with agoraphobia (PDA) and the impact of therapists’ and patients’ contribution to the alliance on outcome and attrition in one trial. Alliance ratings were obtained from patients and therapists early and late in treatment (n = 133). Data were analyzed using multi-level modeling controlling for early symptom change. No therapist effects were found. The patients’ contribution to the alliance predicted outcome (in both panic severity and anxiety sensitivity) and attrition. The therapists’ contribution to the alliance predicted attrition but not outcome. Results suggest that the patient's contribution to the alliance plays an important role in CBT for PDA and that including common factors into research on CBT may help elucidate treatment processes.

Although the last few decades have seen tremendous advances in psychotherapy research (cf. American Psychological Association Presidential Task Force on Evidence-Based Practice, 2006; Nathan & Gorman, 2007; Roth & Fonagy, 2005), several issues still remain contentious. One controversial issue is that of therapist effects on outcome of psychotherapy, or the importance of differences in outcome among therapists (e.g. Baldwin & Imel, 2013; Garfield, 1997; Hill, 2006; Siev, Huppert, & Chambless, 2009; Wampold, Imel, & Miller, 2009). Another controversial issue is the role of the therapeutic alliance (defined as the combination of agreement on goals and how to achieve the goals, and the personal bond between the patient and therapist; Bordin, 1979) and its relationship with the outcome of therapy (Crits-Christoph, Connolly Gibbons, Hamilton, Ring-Kurtz, & Gallop, 2011; Flückiger, Del Re, Wampold, Symonds, & Horvath, 2012; Horvath, Del Re, Flückiger, & Symonds, 2011; Webb et al., 2011).

These issues have been less explored in CBT for anxiety disorders, and specifically in treatment of panic disorder with agoraphobia (PDA). Studies of CBT for anxiety disorders often utilize structured treatment manuals, which potentially diminish therapist and alliance effects (Baldwin & Imel, 2013; Crits-Christoph & Gallop, 2006). There is only one published study that examined therapist effects (or differences) in outcomes CBT for PDA (n = 183; Huppert et al., 2001). Significant effects were found, though they varied greatly depending on the outcome measure (1e 18%), with therapists explaining 8% of the variance in the overall severity of panic disorder and 18% of the variance in anxiety sensitivity. Although the alliance was not measured in the trial, Huppert et al. suggested that more therapists’ skill in developing a therapeutic alliance could account for some of these therapist effects, a notion raised by other researchers as well (Baldwin & Imel, 2013). One major limitation of the Huppert et al. study is that the data were analyzed using analyses of variance (ANOVAs) and correlations and not multi-level modeling, (e.g. Baldwin & Imel, 2013; Elkin, Falconnier, Martinovich, & Mahoney, 2006). Therefore, the first goal of the current study was to reanalyze the Huppert et al. (2001) data using multi-level modeling. This approach is more suitable for the dataset than previous analyses, because it takes into account both therapist and patient levels of variance, thus avoiding potential type I and II errors. In addition, we were interested in replicating the findings regarding therapist effects in another large multi-site CBT for panic disorder trial (The Longitudinal Treatment Study dataset: Aaronson et al., 2008; Allen et al., 2010; White et al., 2013), while also examining whether the therapeutic alliance, on the therapist and the patient levels, can partly explain differences in outcome.

The therapeutic alliance has been suggested as a potential predictor of therapist effects (Baldwin & Imel, 2013; Huppert et al., 2001), and indeed a moderate and consistent correlation exists between the therapeutic alliance and psychotherapy outcome regardless of treatment orientation (including CBT) or type of disorder (including anxiety disorders; see Flückiger et al., 2012; Horvath et al., 2011; Martin, Garske, & Davis, 2000 for recent meta-analyses). However, most studies to date have examined the alliance-outcome correlation at the patient level only (i.e., the patient's alliance correlated with the patient's outcome). To the best of our knowledge, only three studies so far have examined the outcome–alliance correlation at the therapist level (specific “therapist contribution” via the average alliance score per therapist) in addition to the patient level (“patient contribution”), none of which were conducted on samples of CBT treatments. All of these studies found that treatment outcome was predicted by therapist level alliance, but only one of them found that the patient level alliance predicted outcome as well (Baldwin, Wampold, & Imel, 2007; Crits-Christoph et al., 2009; Zuroff, Kelly, Leybman, Blatt, & Wampold, 2010). Therefore, the preliminary and contrasting findings emphasize the importance of conducting more studies that directly examine these issues.

Most studies to date (including the ones that examined therapist and patient level contributions of the alliance on outcome) did not control for symptom change prior to measurement of the alliance when examining the alliance-outcome correlation, and those who did produced mixed results (e.g. Barber, 2009; Webb et al., 2011). Therefore, it is still not clear whether the therapeutic alliance is a consequence or cause of symptom change in therapy (DeRubeis, Brotman, & Gibbons, 2005).

Theoretically, in CBT for the anxiety disorders, a good therapeutic alliance may improve therapy outcome by promoting exposure to feared situations (through greater trust in the therapist) and by preventing patient dropout. However, earlier studies exploring the relationship of outcome with the therapeutic alliance and related therapist behaviors in CBT for PDA produced mixed findings (e.g. Keijsers, Hoogduin, & Schaap, 1994; Williams & Chambless, 1990), while recent studies reported no significant relationship between alliance and panic symptom change in moderate size samples (Casey, Oei, & Newcombe, 2005; Ramnerö & Öst, 2007). Thus, more data are needed on the alliance in CBT using modern statistics and methodology.

Even less research has been done on the topic of therapist effects on attrition rates and on the alliance-attrition correlation. The findings regarding therapist effects on attrition are mixed (Elkin et al., 2006; Huppert et al., 2001; Najavits & Weiss, 1994; Wilson, Wilfley, Agras, & Bryson, 2011), while stronger patient level alliances predicted lower rates of attrition in a recent meta-analysis (N = 1301; Sharf, Primavera, & Diener, 2010).

The Longitudinal Treatment Study dataset (LTS; Aaronson et al., 2008; Allen et al., 2010; White et al., 2013) provides excellent opportunity to examine the issues of the therapeutic alliance and therapist effects on outcome and attrition in a large, multi-site trial of CBT for PDA. In the initial study phase of the LTS, all patients were treated with CBT (Aaronson et al., 2008; White et al., 2010). Patients were then triaged into two clinical trials according to response status. Responders were randomized to booster sessions or no booster sessions (White et al., 2013) while non-responders were randomized to either continued CBT or to paroxetine (Payne et al., 2013). In this study, data were analyzed from the initial treatment phase only where all patients (n = 379) received CBT from 23 therapists.

In sum, the current study sought to first reanalyze using multilevel modeling Huppert et al.'s (2001) data from the multicenter trial for panic disorder (MCSTPD, Barlow, Gorman, Shear, & Woods, 2000) and then as a next step to examine alliance and therapist effects on outcome and attrition in the LTS. The therapeutic alliance was obtained from two different perspectives (patient and therapist) and at two different time points (early and late in therapy). Analyses were conducted using multilevel modeling with three levels (therapist, patient and time) while controlling for early symptom change. Therefore, we were able to separate the patient and therapist contributions to the alliance (by using multi-level modeling), while also examining differences in patients’ and therapists’ perspectives on these contributions (by collecting both patients’ and therapists’ ratings of the alliance). On the basis of Huppert et al. (2001), we expected to find moderate therapist effects on outcome in both samples. In addition, we expected a higher level of therapeutic alliance at the patient and therapist levels to be correlated with greater symptom reduction and lower attrition, even after controlling for symptom change prior to the measurement of the therapeutic alliance, and that alliance would account for a significant portion of the therapist effects. Given the inconsistent findings in the literature, we did not have an a priori hypothesis as to whether the therapist or patient perspective of the alliance would be more predictive of outcome and attrition nor about therapist effects on attrition.

Method

The multicenter collaborative study for the treatment of panic disorder (MCSTPD)

For full details of the trial and the previous therapist effects analysis see Barlow et al. (2000) and Huppert et al. (2001). In the current analyses, all CBT groups (e.g., CBT only and CBT combined with imipramine/placebo pill) were combined and analyzed together.

Participants

Of the 205 patients diagnosed with primary panic disorder with or without agoraphobia that were randomized to CBT, 183 patients (70 men, 113 women) had sufficient data to be included in our analyses.

Therapists

Fourteen doctoral-level therapists were identified for the study. These therapists had data from 4 or more patients. Seven therapists were men and 7 women, 13 of whom were psychologists and 1 of whom was a psychiatrist. The average age of the therapists was 35.7 years, with varied experience conducting general psychotherapy from 2 to 20 years (M = 8.9, SD 5.6) and in conducting CBT from 1 to 18 years (M = 5.9, SD=5.1 years). Nine their orientation as primarily CBT, whereas 5 described themselves as other (i.e., eclectic or psychodynamic). All therapists were trained to competency and certified in conducting panic control treatment prior to participating in the active phase of treatment. Supervision continued during biweekly conference calls throughout the study. Trained raters rated high levels of adherence and competency during treatment throughout the study (adherence mean = 5.72, SD = 0.70; competency mean = 5.59, SD = 1.06; scales range from 1 to 7).

Measures

A number of panic-related measures were collected at pretreatment and post-acute-treatment phase (i.e., after 12 weeks of treatment), all of which are described in Huppert et al. (2001). Among other measures, independent evaluators evaluated patients using the Panic Disorder Severity Scale e Independent Evaluator Version (PDSS-IE; Shear et al., 1997). Self-report measures included the Anxiety Sensitivity Index (ASI; Reiss, Peterson, Gursky, & McNally, 1986).

The longitudinal Treatment Strategies (LTS) Study

Participants

Of the 379 participants that were enrolled in the study and entered treatment, 350 participants whose therapists treated more than 3 patients were identified for the study. These patients were treated by 17 therapists at 4 sites. Of this sample, 253 patients completed the acute phase of the study and received a post-treatment diagnostic interview and assessment; 97 patients either dropped out or were removed from the study. Inclusion criteria included the following: (a) a principal Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV; APA, 1994) diagnosis of panic disorder with or without agoraphobia; (b) age 18 years or older; (c) no substance abuse or dependence within the last 6 months; (d) absence of active suicide potential within the last 6 months; (e) absence of any history of psychosis, bipolar I disorder, bipolar II disorder, or cyclothymia; (f) no current application pending or existing for a medical disability claim; (g) no significant cognitive impairment; (h) free from current uncontrolled general medical illness requiring intervention; (i) absence of concurrent psychotherapeutic treatment directed at anxiety or panic disorders; and (j) no concurrent psychopharmacological treatment that may have antipanic effects (participants were required to discontinue medication/s by session 9). Data on patient characteristics in the current sample (N 350) are presented in Table 1 (see Aaronson et al., 2008 for full description of study procedures and White et al., 2010 for a description and analysis of attrition). Fig. 1 presents the patient flow and sample sizes in the acute phase of the study. Self-report ratings of the therapeutic alliance were obtained from participants and therapists in session 3 and/or 9 for a portion of the entire sample (the administration of the WAI was only initiated in the second half of the acute phase). Session 3 and 9 were chosen to examine change in alliance during therapy and whether the alliance-outcome correlation is affected by the timing of the alliance rating (early vs. late in therapy). We removed from analyses data from patients whose therapists treated less than 4 patients (who had WAI data). Therefore, the session 3 subsample (henceforth S3 subsample) consisted of 133 patients (ninety females) treated by 11 therapists (N patients per therapist = 5–28, M = 12.09), out of which 36 patients either dropped out from treatment or were removed from the study. Ninety five patients in the S3 subsample completed the PDSS-IE assessment at post-treatment. The session 9 subsample (henceforth S9 subsample) consisted of 133 patients (eighty six females) treated by 11 therapists (N patients per therapist = 5–33, M = 12.09), out of which 8 patients either dropped out or were removed from the study. One hundred and twenty four patients in the S9 subsample completed the PDSS-IE assessment at post-treatment.

Table 1.

Patient characteristics in the Longitudinal Treatment Study (n = 350).

Gender (females, n, %)	226 (64.6%)
Age (years, M (SD))	37.7 (11.9)
Age (years, range)	18-70
Ethnicity (n, %)
Caucasian	305 (87.1%)
African-American	18 (5.1%)
Asian or Pacific Islander	19 (5.4%)
American Indian or Alaskan Native	4 (1.1%)
Other	4 (1.1%)
Marital status (n, %)
Married	174 (49.8%)
Never married	132 (37.7%)
Separated, divorced, widowed	43 (12.3%)

Note. Data on age and marital status were not available for a small percentage (<1%) of the patients.

Fig. 1.

Patient flow and sample sizes in the acute phase in the Longitudinal Treatment Study. WAI = working alliance inventory. N_T = number of therapists. N_P = number of patients. These numbers are for patients who provided data from self-report outcome measures (PDSS-SR and ASI). The numbers differ for Panic Disorder Severity Scale Independent Evaluator Version because some patients who completed treatment did not receive a post-treatment assessment (N_P = 7 for general therapist effects analyses; N_P = 2 for WAI analyses).

Therapists

Seventeen therapists that treated more than 3 patients were identified for the study. Data on characteristics of these therapists are presented in Table 2. As the table shows, the therapists were relatively experienced, both in general and in CBT in specific. Therapists were trained in the protocol at the Boston site and group supervision continued throughout the study period. Additionally, adherence monitoring was done on randomly selected tapes. Results were provided to the therapist and site supervisor and discussed as needed.

Table 2.

Therapist characteristics in the Longitudinal Treatment Study (n = 17).

Gender (females, n, %)	9 (52.9%)
Age (years, M (SD))	40.1 (9.2)
Number of patients treated in study (M (SD))	20.6 (15.2)
Number of patients treated in study (range)	7-60
Experience conducting psychotherapy (Median (range))
Number of years	9 (2-27)
Number of hours	5000 (227-38,600)
Number of patients	400 (14-1960)
Number of PDA patients	30 (7-1000)
Experience conducting CBT (Median (range))
Number of years	6(1-20)
Number of hours	2250 (140-17,280)
Number of patients	150(7-1250)
Number of PDA patients	24 (1-900)
Orientation (n, %)
CBT	14 (82.4%)
Psychodynamic	2 (11.8%)
Family Therapy	1 (6.8%)
Degree (n, %)
PhD or PsyD	10 (58.8%)
MA	7 (41.2%)

Note. CBT = Cognitive Behavioral Therapy. PDA = Panic Disorder with Agoraphobia. All therapist characteristics describe therapist status upon entry to the study.

Measures

Panic Disorder Severity Scale – Independent Evaluator Version (PDSS-IE)

The PDSS-IE (Shear et al., 1997, 2001) is a 7-item measure for assessing the frequency, distress, and avoidance caused by panic attacks. Each item is rated on a 0 (none/mild) to 4 (extreme/severe) scale. Interrater reliability (N 434, p_I 0.993) and internal consistency (α = 0.81–0.86) were both current study (Aaronson et al., 2008).

Panic Disorder Severity Scale – Self-Report Version (PDSS-SR)

The PDSS-SR is a self-report version of the respective independent evaluator measure (Houck, Spiegel, Shear, & Rucci, 2002). Internal consistency coefficients in the different sessions in the current study were excellent (α = 0.88–0.91).

Anxiety Sensitivity Index (ASI)

The ASI (Reiss et al., 1986) is a 16-item self-report scale used to evaluate anxiety focused on panic related bodily sensations. Each item is rated on a 0 (very little) to 4 (very much) scale. It has been used extensively in research on PDA, and internal consistency coefficients in the different sessions in the current study were excellent (α ½ 0.87–0.93).

Working Alliance Index – Short Form (WAI-SF)

The WAI-SF (Tracey & Kokotovic, 1989) is a 12-item self-report scale used to assess the therapeutic relationship between therapist and patient, both from the therapist's perspective (WAI-T) and the patient's perspective (WAI-P). Each item is rated on a 1 (never) to 7 (always) scale. The scale yields both a total score and three 4-item subscales, each assessing different aspects of the therapeutic relationship, as originally defined by Bordin (1979): goals (patient and therapist agreement on goals), tasks (patient and therapist agreement on how to achieve the goals) and bond (development of a personal bond between patient and therapist). Internal consistency coefficients in the different sessions and subsamples in the current study were generally excellent (α = 0.80–0.97), except for the patient WAI-goal subscale which showed moderate internal consistency (α = 0.64–0.68).

Procedure

The Institutional Review Boards at all 4 sites approved this study. The Acute phase of the study consisted of 11 individual cognitive-behavioral therapy (CBT) sessions for panic disorder using a modified version of Panic Control Treatment (Barlow & Craske, 2007). CBT was administered in individual format (less than 90 min in length), and the course of treatment was delivered in less than 19 weeks. All treatment sessions were audiotaped, and 15% of sessions were randomly selected for rating for adherence and competence ratings across the 11 sessions of acute treatment. The Adherence Rater was a doctoral-level clinician (i.e., PhD clinical psychologist or MD psychiatrist) who was trained and certified in the treatment protocol. In the larger clinical trial, following the Acute Phase, patients were randomized to either responder or nonresponder arms of the study. Responders were randomized to receive either (a) Assessment only (no additional treatment) or (b) Maintenance CBT (1 session a month for 9 months). Nonresponders were randomized to receive either (a) Additional CBT for panic disorder (6 sessions over 3 months) or (b) a medication trial (3–12 months). Because of the addition of different treatments following the acute CBT sessions, the current article focuses exclusively on the acute phase of the trial in which all patients received CBT. Eligible and interested participants were evaluated at intake using the PDSS-IE and ASI, and then entered 11 sessions of CBT. Self-report measures (PDSSSR and ASI) were collected from patients in each session. After CBT termination, patients were evaluated using the PDSS-IE and ASI.

Data analytic approach

Data were analyzed using linear and generalized multi-level models (Raudenbush & Bryk, 2002). Analyses were performed using SPSS Version 20.0 Mixed Models and were adjusted for repeated measures with nested random effects at the patient and therapist levels with full maximum likelihood estimation method. Effect sizes of treatment and therapist effects were examined in the patient sample that included therapists who treated at least four patients in either dataset. The magnitude of therapist effects were examined using the percent of variance attributed to the therapist level from the total variance in the sample, and the significance level of that variance.

In the LTS, prediction of treatment outcome and attrition using alliance scores was carried out in each subsample separately. Separate models were fitted for each outcome measure (PDSS-IE, PDSS-SR, ASI and number of sessions until dropout). Due to interest in the contribution of each subscale of the WAI individually (cf. Webb et al., 2011), separate models were also fitted for each WAI scale (total, task, bond, and goal). Patient and therapist WAI were entered simultaneously in the same models as they theoretically capture the same construct but were not highly correlated.

When applicable, fitting of level one growth curves and covariance structures was performed using model fit estimates (–2 log likelihood). Using this approach, a linear growth curve was chosen for all analyses, with first-order homogeneous autoregressive covariance structure. Then, each participant's growth curve was predicted from therapist and patient specific parameters (symptom level at intake, therapist and patient therapeutic alliance scores). Therapist level alliance scores were calculated as the average of the respective patient level scores for each therapist. Therapist level alliance scores were included in the models even if there were insignificant variances at the therapist level, because recent writing on mediation suggests that a significant indirect effect (therapists-alliance-outcome/attrition) can exist without the existence of a significant direct effect (therapists-outcome/attrition; MacKinnon & Fairchild, 2009).

To identify relevant predictors, we started with a base model that included symptom level at intake as a covariate. We then added the respective WAI scale/subscale scores (both patient and therapist) in both levels (patient and therapist level). We then applied a backward step-wise procedure, retaining predictors with p value <0.15 until a final model was reached. Following Fournier et al. (2009), we chose a conservative approach in which predictors were required to be associated with both slopes and intercepts to be considered predictive of outcome. Intercepts were centered to represent estimated endpoint scores at session 11 (thereby accounting for the rationale for a two-level model suggested by Kim, Wampold, & Bolt, 2006). Missing data in any of the potential predictors were imputed using data from the nearest time point (n = 2 in S3 subsample only).

For all models, continuous predictors at the patient level were centered at the group mean and at the therapist level – at the grand mean. To control for early symptom improvement while predicting outcome, symptom change (as measured by PDSS-SR/ASI) from session 1 to session 3 was calculated by regressing symptom level at session 3 on symptom level at session 1 and taking the unstandardized residual. WAI scores at session 3 were then modified by regressing scores on early symptom change as measured by PDSSSR or ASI, depending on type of analyses (PDSS-SR early symptom change was used in analyses of PDSS-SR, PDSS-IE and number of sessions until dropout; ASI for ASI analyses only), and using the resultant residual. A similar procedure was not employed for WAI at session 9 because by that session most of the change in PDSS scores has already occurred (a change of 4.5 points out of a total change of 6 points). Therefore, controlling for earlier symptom change would be expected to leave a very small and clinically insignificant change to be explained by session 9 WAI. Effect Sizes (ES) for predictors in the linear multi-level analyses were calculated from likelihood ratio statistics (Lipsitz, Leong, Ibrahim, & Lipshultz, 2001) and are reported in a percent reduction in variance metric (R²).

Results

Effect size of treatment in the LTS

The effect size of treatment on PDSS-IE was calculated as pre-to-post Cohen's d and was found to be large, t(255) 21.83, p < 0.001, d = 1.36, 95% CI [1.19, 1.53].

Effect sizes of treatment on PDSS-SR and ASI were calculated as the percent of variance which was explained by the addition of time as a predictor to a model without time as a predictor at level 1. Using this approach, we found that the effect sizes of treatment were large for both PDSS-SR, t(309)= -18.95, p < 0.001, ES = 0.40 and ASI t(305) = 24.02, p < 0.001, ES = 0.68.

Therapist differences in treatment outcome

To examine therapist effects on outcome, basic models were fitted that show the distribution of variance in outcome in the different levels (therapist, patient and in the case of PDSS-SR and ASI I the LTS also time).

In the MCSTPD using multi-level models, we found no evidence for therapist effects on the PDSS-IE (S² = 0.00, p = 1.00, ES = 0.00). For the ASI, variance at the therapist level accounted for 14.4% of the total variance, although it was not significantly different than zero (S² = 16.35, p = 0.17), meaning that differences between therapists in outcome were not significant.

In the LTS, analyses on the entire sample showed that only 2.2% of the variance in PDSS-IE was attributable to the therapist level, and this estimate was not statistically significant (S² = 0.43, p = 0.46). Similarly, we found that none of the variance in PDSS-SR (S² = 0.00, p = 1.00) belonged to the therapist level, while 59% of the variance (S² = 18.67, p < 0.001) belonged to the patient level. Similar results were obtained for ASI, where 1.8% of the variance (S² = 3.4, p = 0.21) belonged to the therapist level, while 62.4% of the variance (S² = 119.02, p < 0.001) belonged to the patient level.

Therapist differences in attrition

Variance in completion status at the therapist level was small and insignificant (S² = 0.16, p 0.07). In addition, only 4.3% of the variance (S² = 0.39, p = 0.19) in number of completed session was attributable to the therapist level.

Prediction of treatment outcome using WAI^2,3

Baseline Models without WAI

The results of the 3 null models in S3 subsample (PDSS-IE, PDSS-SR and ASI) are displayed in Tables 1–3, respectively. PDSS-IE multi-level analyses showed that the estimated mean post-treatment PDSS-IE score in S3 subsample was 6.1. On the PDSS-SR in S3 subsample there was an estimated mean decrease of 0.6 points in each session, and the estimated mean session 11 score was 4.9. On the ASI in S3 subsample, there was an estimated mean decrease of 1.8 points in each session, and the estimated mean session 11 score was 14.5. All analyses controlled for symptom severity at intake.

Table 3.

Results of Final Multi-Level Models Predicting Post-Treatment Panic Disorder Severity Scale (Independent Evaluator version) Scores using session 3 Working Alliance Inventory Scores (n = 95) after using Backwards Stepwise Elimination. Blank spaces mean that the measure was not included in the final model. Predictors that are not presented here never met criteria for inclusion in the final model (p < 0.15).

Model		Regression coefficients					Model ES
		Intercept	Symptoms at intake	Patient level WAI-T	Patient level WAI-P	Therapist level WAI-P
Null	b	6.07**	0.28**
	df	13	90
	t	13.02	2.83
WAI	b	6.10**	0.36**		−0.20**		14.32%
Total	df	12	91		95
	t	15.46	3.80		−4.02
WAI	b	6.07**	0.34**	-0.30*	−0.37*	−0.85	16.54%
Task	df	95	95	95	95	95
	t	15.78	3.67	-2.16	−2.50	−1.77
WAI	b	6.07**	0.34**		−0.45**		12.21%
Bond	df	13	90		95
	t	14.14	3.61		−3.64
WAI	b	6.15**	0.34**		−0.43*		9.42%
Goal	df	95	92		5
	t	15.31	3.50		−2.87

Note. WAI-T = Working Alliance Index – Therapist version; WAI-P = Working Alliance Index – Patient version; ES = Effect Size; df = Degrees of Freedom.

^*p < 0.05.

^**p < 0.01.

Prediction of PDSS-IE outcome using WAI at session 3

Final models of the stepwise analysis are presented in Table 3, separately for each WAI subscale. Patient level WAI-P and all subscales were predictive of post-treatment PDSS-IE (ES 14.32% for total scores, 16.54% for the task subscale, 12.21% for the bond subscale, and 9.42% for the goal subscale), such that higher WAI scores were associated with lower PDSS-IE scores at post-treatment. In addition, higher patient level WAI-T scores were predictive of lower post-treatment PDSS-IE scores, but only for the task subscale. Therapist level WAI-P and WAI-T scores were not predictive of outcome as measured by the PDSS-IE.

Prediction of PDSS-SR outcome using WAI at session 3

Final models of the stepwise analysis are presented in Table 4, separately for each WAI subscale. As the table shows, patient level WAI-P and subscales were predictive of PDSS-SR outcome (ES 6.41% for total scores, 3.66% for the task subscale, 5.96% for the bond subscale, and 5.30% for the goal subscale). The b estimates can be interpreted as representing the change in estimated Session 11 PDSS-SR scores or slopes per unit of the predictor above the sample mean, controlling for all of the other predictors in the model. For example, in the final model of WAI-total, the b estimate is 0.12 for WAI-P is –0.12 and 0.01 for Time*WAI-P, indicating that for each unit of WAI-total above the sample mean, Session 11 PDSS-SR score is expected to be lower by 0.12 points and the average decrease in PDSSSR per session is expected to be greater by 0.01 points (controlling for all other predictors in the model). Patient level WAI-T scores and therapist level WAI-T and WAI-P scores were not predictive of outcome as measured by the PDSS-SR.

Table 4.

Results of Final Multi-Level Models Predicting Post-Treatment Panic Disorder Severity Scale (Self Report version) Scores from session 3 Working Alliance Inventory Scores (n = 133) after using Backwards Stepwise Elimination. Blank spaces mean that the measure was not included in the final model. Predictors that are not presented here never met criteria for inclusion in the final model (p < 0.15).

Model		Regression coefficients						Model ES
		Intercept	Symptoms at intake	Patient level WAI-P	Time	Time* symptoms at intake	Time* patient level WAI-P
Null	b	4.88**	0.22**		−0.60**	−0.06**
	df	221	239		8	177
	t	13.17	2.60		−15.03	−6.40
WAI Total	b	5.01**	0.25**	−0.12**	−0.58**	−0.05**	−0.01**	6.41%
	df	225	241	295	197	208	243
	t	13.50	3.00	−2.61	−15.46	−6.01	−2.92
WAI Task	b	4.97**	0.24**	−0.26*	−0.59**	−0.05**	−0.03*	3.66%
	df	224	240	297	192	204	245
	t	13.38	2.89	−2.10	−15.31	−6.07	−2.01
WAI Bond	b	4.99**	0.25**	−0.28*	−0.58**	−0.05**	−0.03**	5.96%
	df	225	242	290	200	207	239
	t	13.49	2.94	−2.53	−15.32	−6.07	−2.75
WAI Goal	b	4.96**	0.24**	−0.23*	−0.59**	−0.05**	−0.03**	5.30%
	df	257	276	286	277	284	289
	t	13.35	2.86	−1.98	−15.43	−6.04	−2.68

Note. WAI-P = Working Alliance Index – Patient version; ES = Effect Size; df = Degrees of Freedom.

^*p < 0.05.

^**p < 0.01.

Prediction of ASI outcome using WAI at session 3

Final models of the stepwise analysis are presented in Table 5, separately for each WAI subscale. Results show that patient level WAI-P and all of its subscales predicted ASI outcome significantly (ES 13.70% for total scores,14.50% for the task subscale,12.56% for the bond subscale, and 9.99% for the goal subscale). Higher WAI-P scores predicted lower Session 11 ASI scores and greater decreases in ASI with each session. In addition, higher therapist level WAI-T scores were predictive of lower session 11 ASI scores, and greater decreases in ASI with each session, but only for the task subscale. Patient level WAI-T and Therapist level WAI-P were not predictive of outcome as measured by the ASI.

Table 5.

Results of Final Multi-Level Models Predicting Post-Treatment Anxiety Sensitivity Index Scores from session 3 Working Alliance Inventory Scores (n = 133) after using Backwards Stepwise Elimination. Blank spaces mean that the measure was not included in the final model. Predictors that are not presented here never met criteria for inclusion in the final model (p < 0.15).

Model		Regression coefficients								Model ES
		Intercept	Symptoms at intake	Patient level WAI-P	Therapist level WAI-T	Time	Time* symptoms at intake	Time* patient level WAI-P	Time* therapist level WAI-T
Null	b	14.45**	0.37**			−1.79**	−0.04**
	df	4	203			5	54
	t	17.32	5.51			−18.02	−5.01
WAI Total	b	14.81**	0.40**	−0.41**		−1.74**	−0.04**	−0.04**		13.70%
	df	9	237	294		12	96	135
	t	16.62	6.12	−4.48		−17.51	−4.87	−3.58
WAI Task	b	15.29**	0.39**	−0.93**	−1.42*	−1.69**	−0.04**	−0.08**	−0.19**	14.50%
	df	272	264	300	288	121	117	146	134
	t	19.45	5.92	−3.72	−2.40	−18.38	−5.00	−2.92	−2.78
WAI Bond	b	14.70**	0.37**	−0.97**		−1.75**	−0.04**	−0.09**		12.56%
	df	11	243	287		13	96	127
	t	15.72	5.68	−4.30		−16.64	−5.27	−3.28
WAI Goal	b	14.79**	0.43**	−0.93**		−1.76**	−0.04**	−0.09**		9.99%
	df	266	268	282		114	115	126
	t	19.15	6.26	−3.68		−19.25	−4.37	−3.18

Note. WAI-T = Working Alliance Index – Therapist version; WAI-P = Working Alliance Index – Patient version; ES = Effect Size; df = Degrees of Freedom.

^*p < 0.05.

^**p < 0.01.

Prediction of outcome using WAI at session 9⁴

Results in the S9 subsample followed a similar pattern to those of the S3 subsample. Outcome in all three measures (PDSS-IE, PDSS-SR and ASI) was predicted by patient level WAI-P scores and all of its subscales, with higher WAI scores predicting better outcome. All other WAI scores (therapist level WAI-P, patient and therapist level WAI-T) did not predict outcome, except for therapist level WAI-P (total, task and goal, but not bond) predicting lower post-treatment PDSS-IE scores. Full results of these analyses are available from the first and second authors.

Prediction of attrition using WAI

Analyses in this section focused on prediction of completion status and number of sessions until dropout using WAI at session 3. We did not predict attrition using WAI at session 9, because in this late stage of treatment attrition rates were very small (n 8 out of 133). A model with intake PDSS-IE level as predictor served as null model, to control for initial symptom severity while interpreting the results.

The results of the null and final models are displayed in Table 6. As the null model shows, symptom level at intake was not predictive of number of sessions until dropout. Therapist level WAI-P scores and all subscales were predictive of number of sessions until dropout, such that higher WAI scores were predictive of lower relative risk for dropout. In addition, higher patient level WAI-P scores (total and task subscale only) were also predictive of lower risk for dropout. In contrast, therapist level and patient level WAI-T scores were not predictive of number of sessions until dropout.

Table 6.

Results of Multi-Level Models Predicting Number of Sessions until Dropout using session 3 WAI Scores (n = 133) after using Backwards Stepwise Elimination. Blank spaces mean that the measure was not included in the final model. Predictors that are not presented here never met criteria for inclusion in the final model (p < 0.15).

Model		Regression coefficients
		Intercept	Symptoms at intake	Patient level WAI-P	Therapist level WAI-P
Null	b	−3.57**	0.00
	Exp (b)	0.03	1.00
WAI Total	b	−3.63**	0.01	−0.02*	−0.10**
	Exp (b)	0.03	1.01	0.98	0.90
WAI Task	b	−3.65**	0.01	−0.07*	−0.27**
	Exp (b)	0.03	1.01	0.93	0.76
WAI Bond	b	−3.59**	0.01	−0.05	−0.25**
	Exp (b)	0.03	1.01	0.95	0.78
WAI Goal	b	−3.63**	0.00		−0.34**
	Exp (b)	0.03	1.00		0.71

Note. WAI-T = Working Alliance Index – Therapist version; WAI-P = Working Alliance Index – Patient version; b = Cox regression coefficient for prediction of number of sessions until dropout; Exp(b) = Relative risk for dropout.

^*p < 0.05.

^**p < 0.01.

Discussion

The present study examined therapist effects on outcome and attrition in CBT for PDA and the therapeutic alliance at the patient and therapist levels as a predictor of change in therapy. In a large-scale clinical trial (LTS), we found that the patient's contribution to the therapeutic alliance, as perceived by patients themselves, usually predicted outcome and attrition, both early (session 3) and late (session 9) in the course of treatment, even after controlling for early symptom improvement. The large effect sizes and the consistent results across outcome measures lend further support to the validity of our findings. The therapist's contribution to the therapeutic alliance and the therapeutic alliance as perceived by therapists usually did not predict outcome. Also, there were no significant therapist effects in either the LTS and the MCSTPD, although therapist effects on the ASI in the MCSTPD may have been insignificant due to underpowered analyses.

To the best of our knowledge, this is the first study to demonstrate alliance-outcome and alliance-attrition relations in CBT for PDA using validated measures of the alliance (cf. Casey et al.., 2005; Ramnerö & Öst, 2007). In addition, this is the first study that separated between therapists’ and patients’ contributions to the alliance-outcome correlation while also controlling for early symptom change (cf. Baldwin et al., 2007; Crits-Christoph et al., 2009; Zuroff et al., 2010). Unlike our study, all previous studies found that the therapists’ contribution predicted outcome, while only one of these studies found that the patients’ contribution also predicted outcome of therapy (Zuroff et al., 2010). Our findings suggest that the patient's perspective of the therapeutic alliance may play an essential role in facilitating better outcome. Follow-up studies are needed to examine the possible mechanisms of this relationship (e.g. does better alliance lead to greater compliance to exposures and more dropping of safety behaviors?).

In the present study all aspects of the patient level therapeutic alliance (agreement on goals, agreement on how to achieve the goals, and development of a personal bond between the participants) predicted outcome. This finding is inconsistent with a recent analysis of observer rated alliance done by Webb et al. (2011) in which only a factor assessing the agreement on tasks and goals predicted outcome of cognitive therapy for depression, while a factor assessing the affective bond between patient and therapist did not. This raises the possibility that the therapeutic alliance plays different roles in the treatment of different disorders, or that method variance (i.e., patient perspective vs. observer perspective) may lead to different results.

Our findings suggest that scientists and practitioners should collect two or even three different perspectives of the alliance (patients, therapists and observers) rather than relying on one perspective only. Along similar lines, patients also rated the alliance as stronger than therapists on average, which may reflect either different anchoring that each group uses (i.e., therapists likely compare to their history of other patients, while patients focus on the current relationship on the basis of various unknown criteria), or a level of discomfort that patients have in negatively judging their therapists. However, the latter would suggest a less accurate depiction of the alliance which should then be a worse predictor of outcomes, and we found the contrary.

There was one significant result at the therapist level: alliance from the patient perspective was more consistent as a predictor of attrition, more so than alliance at the patient level. It could be that even though the therapist's ability to form a therapeutic alliance may keep a patient in therapy (possibly by instilling motivation in the patient), other patient factors (e.g. rigidity, difficulties with emotion regulation or tolerance) may still prevent the patient from responding to the treatment.

The lack of therapist effects on outcome in our data stands in contrast to a recent meta-analysis although some of the effect sizes in our samples were comparable in size to those reported (3% in randomized controlled trials (RCTs) and 7% in naturalistic studies; Baldwin & Imel, 2013). The lack of therapist effects is also in contrast to the effects that were previously found in the MCSTPD using ANOVAs (Huppert et al., 2001). It is possible that treatment effects in anxiety disorders (i.e., exposure and cognitive change procedures) overwhelms any potential variability that exists among therapists or that in RCTs use of carefully selected and trained therapists and rigorously controlled treatments may decrease variability in outcome between therapists (Crits-Christoph & Gallop, 2006). Along these lines, Baldwin and Imel (2013) found therapist effects to be larger in field studies than in RCTs and Crits-Christoph et al. (1991) found that use of a manual was related to decreased therapist effects, suggesting that standardization may help ensure uniform results. Overall, large naturalistic field studies (Elkin et al., 2006; Okiishi, Lambert, Nielsen, & Ogles, 2003, Okiishi et al., 2006; Wampold & Brown, 2005) and studies that compare manualized to nonmanualized therapy (e.g., Schulte, Künzel, Pepping, & Schulte-Bahrenberg, 1992) may provide better opportunities for examining therapist effects.

A number of reasons may account for the different results obtained using ANOVAs and multi-level modeling in the MCSTPD analyses such as overreliance on outliers in ANOVA, different estimation methods (ordinary least squares vs. maximum likelihood) or the differences in standard errors for therapists that are estimated using these methods. It seems important to use advanced statistical methods that are suitable to the data design at hand, as therapist effects seem to be highly sensitive to the data analytic approach used (Elkin et al., 2006; Kim et al., 2006). This also means designing studies in a manner that allows enough power to detect therapist effects. According to a recent analysis (Crits-Christoph et al., 2011), as many as 50 therapists treating 60 patients each are needed to provide an adequate therapist-level alliance score e a criteria not met by the present study. It should be noted, in addition, that psychotherapy researchers have long recognized the importance of reporting the clinical relevance of a finding to complement the statistical inference (e.g. Jacobson & Truax, 1991). Nevertheless, clinical significance measures have not been developed yet for advanced statistical methods such as mutli-level modeling. Therefore, as the use of these advanced methods increases, psychotherapy researchers should also focus on developing advanced clinical significance measures.

Several limitations of the study should be noted. Despite accounting statistically for alternative explanations (i.e. controlling for early symptom change), the conclusions of this study are still limited by the correlational design and reliance on self-reported therapeutic alliance measures (rather than observers’ ratings).

The small number of therapeutic alliance measurements may also be a limitation, as a recent analysis suggests that aggregation of at least four measurements is required to achieve a stable estimate of the therapeutic alliance (Crits-Christoph et al., 2011). In addition, multiple measurements over time would allow for time-lagged analyses to better demonstrate the temporal relationship between alliance and symptom change and other predictors of change (e.g. Weiss, Kivity, & Huppert, in press). Measuring alliance only in session 3 and 9 also meant excluding from analyses patients who dropped out earlier in treatment. This limitation also introduced difficulties untangling early alliances from early symptom change.

In sum, this is the first study to show that the patient contribution to the alliance (as perceived by patients) is related to outcome of CBT for PDA using current methodological standards. Moreover, this is the first study to show that patient level alliance is more predictive of outcome than therapist level alliance when controlling for early symptom change. The findings suggest that including common factors into research on CBT may help elucidate treatment processes. In addition, these findings point to the possible importance of monitoring therapeutic alliance from the patient perspective in CBT for PDA. Indeed, collecting patient ratings on the therapeutic alliance throughout treatment and using them as feedback for therapists (Lambert & Shimokawa, 2011) or repairing alliance ruptures as they arise (Safran, Muran, & Eubanks-Carter, 2011) has been shown to improve outcomes in other populations and merits study in CBT for the anxiety disorders as well. Finally, given the high ratings of early alliance, more experimental research should examine whether methods to improve the alliance lead to better outcomes (e.g. Crits-Christoph et al., 2006).