Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Jul 1.
Published in final edited form as: Behav Res Ther. 2010 Mar 17;48(7):599–606. doi: 10.1016/j.brat.2010.03.011

The Process of Change in Cognitive Therapy for Depression: Predictors of Early Inter-Session Symptom Gains

Daniel R Strunk a,*, Melissa A Brotman b, Robert J DeRubeis c
PMCID: PMC2878902  NIHMSID: NIHMS190832  PMID: 20362978

Abstract

Although Cognitive Therapy for depression is an efficacious treatment, questions about the aspects of the therapy that are most critical to successful implementation remain. In a sample of 60 Cognitive Therapy patients with moderate to severe depression, we examined three aspects of therapists’ adherence to Cognitive Therapy techniques, the patients’ facilitation or inhibition of these techniques, and the therapeutic alliance as predictors of session-to-session symptom improvement across the first five therapy sessions. Two elements of therapist adherence (viz., cognitive methods and negotiating content / structuring sessions) emerged as the strongest predictors of symptom improvement. Patient facilitation or inhibition of therapist adherence also predicted subsequent symptom change. Neither adherence to behavioral methods / homework nor the therapeutic alliance was a significant predictor in parallel analyses. Although alliance scores did not predict subsequent symptom change, they were significantly predicted by prior symptom change. These findings support the model of change that motivates Cognitive Therapy for depression, and they highlight the potential role of patient facilitation of therapists’ adherence in treatment response.

Keywords: Cognitive therapy, depression, adherence, therapeutic alliance, patient

Cognitive therapy (CT) is a well-standardized, effective treatment for major depression (Strunk & DeRubeis, 2001). Recent investigations have demonstrated that CT is at least as effective (DeRubeis, Gelfand, Tang, & Simons, 1999; DeRubeis et al., 2005) and may result in more enduring effects relative to pharmacotherapy (Hollon et al., 2005; Hollon, Stewart, & Strunk, 2006). However, the means by which CT achieves these effects is the subject of ongoing debate (Garrat, Ingram, Rand, & Sawalani, 2007).

In this paper, we examine two therapy processes: adherence to CT techniques and the therapeutic alliance. We extend previous CT process research by acquiring multiple assessments of process variables, employing longitudinal models that incorporate repeated measures, and including a larger sample of cases compared to most prior CT process research. Our method of scoring our adherence measure is an improvement over previous research as it is based on a recent factor analysis by our group (Strunk, Cooper, & Ryan, 2010). In addition, we explore a new construct, the extent to which patients facilitate or inhibit specific therapists’ efforts to adhere to CT. We also retain important advantages that have been present in some previous studies. Our predictor variables are: (1) assessed early in treatment, during the time when symptom change tends to be occurring most rapidly and when process measures are less likely to reflect therapists’ responses to the course of treatment (Stiles & Shapiro, 1994); and (2) assessed prior to the starting point of the symptom change they predict.

Role of Adherence and the Therapeutic Alliance in CT

Psychotherapy process researchers use the term adherence to refer to manual-specified therapist actions. Based on a factor analysis of a broader measure of adherence to CT for depression, DeRubeis and Feeley (1990) have described two types of adherence: concrete and abstract. Concrete adherence involves methods designed to help patients identify and correct inaccurate, negative thoughts, behavioral techniques, and the use of homework assignments. Abstract techniques involve broader discussions of therapy-relevant issues, including efforts to further understand the meaning of patients’ beliefs and conversations about the progress of CT sessions.

The therapeutic alliance is defined as the collaborative and affective bond between therapist and patient (Gaston, Marmar, Gallagher, & Thompson, 1991; Krupnick et al., 1996; Martin, Garske, & Davis, 2000; Raue, Goldfried, & Barkham, 1997). An expanded definition by Bordin (1979) separates the alliance into three components: goal, task, and bond. Goal refers to a mutual agreement on the specified outcomes that are the target of the intervention; task involves an agreement on the current approach to achieve the goals; and bond, reflects the development of a positive therapist-patient relationship characterized by mutual liking and trust.

Both the alliance (Horvath & Symonds, 1991; Klein et al., 2003; Martin et al., 2000; Wampold, 2001) and therapist adherence (Barber, Crits-Christoph, & Luborsky, 1996; DeRubeis & Feeley, 1990; Feeley, DeRubeis, & Gelfand, 1999) have been shown to correlate with outcome in psychotherapy, thus satisfying one of the three tests that should be applied to observational investigations of the relation between a process variable and an outcome variable. Two other important tests, nonspuriousness and temporal precedence of the process variable (Feeley et al.; Judd & Kenny, 1981), are much less often conducted in process-outcome studies. Spuriousness cannot be ruled out in observational research, but one plausible source of it can be reduced by assessing process measures early in treatment, thereby limiting the opportunity for third variables that influence outcome to also influence measures of therapeutic processes.

Early measures of process variables may be particularly revealing because the rate of symptom change tends to be greatest early in the course of treatment (Ilardi & Craighead, 1994). In most studies of the relation between the therapeutic alliance and symptom change, temporal precedence has not been established (see Feeley et al., 1999). In turn, any alliance-outcome relation could simply reflect the effect of symptom change on the therapeutic alliance. Two widely-cited meta-analyses (Horvath & Symonds, 1991; Martin et al., 2000) have documented the statistical relation between the alliance and outcome across a variety of psychotherapies. However, these meta-analyses have included studies in which measures of the therapeutic alliance predicted symptom change over the full course of treatment. In such studies, symptom improvement prior to the measurement of the alliance may have contributed to the developing therapeutic relationship, and thus may be at least partly responsible for the observed alliance-outcome relationships. Therefore, in order to ensure the temporal precedence of a process measure obtained at a given session, the symptom change experienced prior to that session must not be included in the measure of symptom change.

In the few studies of the relation between therapeutic alliance and outcome that have focused on prediction, rather than simply association, the strength of the relation has been inconsistent. Barber and colleagues (2000) reported a significant predictive relationship between the therapeutic alliance and outcome in a diagnostically heterogeneous group of patients being treated with supportive-expressive dynamic psychotherapy. Similarly, Klein et al. (2003) found that early alliance predicted outcome in a group of chronically depressed patients treated with Cognitive-Behavioral Analysis System of Psychotherapy (CBASP) alone or with medication. However, in two studies of CT (DeRubeis & Feeley, 1990; Feeley et al., 1999), the alliance-outcome relation was nonsignificant; the correlation was small in one case and negative in the other. Given the small number of studies and the mixed findings produced to date, further appropriately designed studies of the alliance-outcome relationship are needed.

Strategy for Use of Repeated Measures of Process and Outcome Variables

Researchers have utilized a variety of approaches to investigate the process of change in psychotherapy. In addition to selecting widely varying unit sizes for ratings (from single utterances to several sessions), researchers have also used a variety of strategies for the determining the interval over which change in outcome measures is to be examined (from change over very brief time intervals to change over an entire course of therapy). Despite this variability across studies, surprisingly little is known about the precise time period over which the effects of psychotherapy interventions become evident as changes on symptom measures. This is likely because researchers have not generally focused on identifying the specific time lag between therapeutic interventions and the resulting symptom improvements. Studies of adherence and alliance in CT for depression have often utilized statistical models in which ratings of these process measures for a single therapy session are examined as predictors of change over a substantial portion of the full course of treatment (for examples, see Ablon & Jones, 1999; DeRubeis & Feeley, 1990; Feeley et al., 1999; Hill, O’Grady, & Elkin, 1992).

Although these studies can reveal process-outcome relationships in CT, focusing on repeated assessments of process measures (across multiple sessions) as potential predictors of symptom change from one session to the next has distinct advantages. First, there is reason to believe that the lag between therapeutic activities and consequent symptom improvements may be as brief as the time between two consecutive therapy sessions. In studies focused on identifying predictors of very large session-to-session symptom changes (i.e., sudden gains), researchers have successfully identified predictors in the immediately preceding session (Tang & DeRubeis, 1999; Tang, DeRubeis, Beberman, & Pham, 2005; Tang, DeRubeis, Hollon, Amsterdam, & Shelton, 2007). Thus, though data on this issue are insufficient, we suspect that the time lag over which many therapeutic activities produce symptom improvements is likely to be brief. Second, psychotherapy process research can utilize repeated measures of process and outcome variables to obtain more precise estimates of the relation of early process variables and subsequent symptom improvement. Third, more sophisticated longitudinal analyses can allow for the use of all available data, including participants with missing values, assuming that the values are missing at random (Brown & Prescott, 1999). Despite the advantages of using repeated assessments of process measures (across multiple sessions) as potential predictors of symptom change from one session to the next, researchers have rarely employed this approach. To our knowledge, no examination of the process of change in CT for depression has included the use of these methods.

Purpose of this Study

We sought to investigate several therapy process variables as predictors of symptom improvement in CT for depression. We build upon prior CT process-outcome research by employing repeated measurements of process variables early in treatment, when change is occurring rapidly and when the process measures are least likely to be contaminated by third variables. We examine three factors of adherence, informed by a recent factor-analytic investigation (Strunk et al., 2010). Although a factor analysis of adherence items has been previously reported (DeRubeis & Feeley, 1990), that analysis employed a small sample and was conducted before guidelines for conducting exploratory factor analysis were well-developed (see Fabrigar, Wegener, MacCallum, & Strahan, 1999). Process measures over the first several sessions were examined as predicting subsequent improvement assessed at the beginning of the subsequent session.

In addition to examining therapists’ adherence to therapeutic techniques and the therapeutic alliance, we developed a measure to assess patients’ facilitation or inhibition of therapist-offered CT techniques. CT is thought to achieve its effects by therapists and patients collaboratively engaging in specific techniques aimed at modifying depressive cognitions. Insofar as this collaborative element is helpful, patients’ contributions to CT-specific strategies may be an important determinant of treatment response. This construct of patient facilitation / inhibition of therapist’s adherence is likely related, but distinct from constructs such as patient motivation. Whereas patient motivation is a plausible determinant of patient’s behavioral responses to therapists’ CT-specific efforts, this motivation is not synonymous with facilitative or inhibitory behaviors. In addition, unlike previous studies which have focused on patient behaviors more generally (e.g., Ablon & Jones, 1999), we focus specifically on patient actions that reflect their contributions to the use of CT-specific strategies.

We hypothesized that adherence in early sessions (particularly adherence to methods designed to promote cognitive change) would predict subsequent symptom reduction from session-to-session over the first five sessions, whereas the therapeutic alliance would not. We also proposed that patients who facilitated therapists’ concrete adherence to CT would show more improvement in symptoms than patients who were less engaged in CT.

Method

Participants

Patients

Patients were 60 adults with a primary Axis I diagnosis of Major Depressive Disorder who were randomly assigned to the CT condition of a two-site (University of Pennsylvania and Vanderbilt University), clinical trial of CT, pharmacotherapy, and placebo for moderate to severe depression (see DeRubeis et al., 2005). Patients met criteria for a current episode of depression according to the Structured Clinical Interview for DSM-IV Diagnosis (First, Spitzer, Gibbon, & Williams, 2001) and scored 20 or higher on the modified 17-item version of the Hamilton Rating Scale for Depression (Hamilton, 1960). Those with psychotic features, a history of bipolar disorder, current substance abuse, borderline personality disorder, antisocial personality disorder, schizotypal personality disorder, or a primary diagnosis of an Axis I disorder other than Major Depressive Disorder were excluded from participation. Participants gave written informed consent prior to entering the acute phase of the study.

In our sample of CT patients, 58% were women and ages ranged from 19 to 68 years (M = 40, SD = 12). Most patients were Caucasian (78%); 12% were African American, and 10% were of other ethnicities. One third of patients were married or co-habitating with their partners.

Therapists

Four male and two female clinicians served as cognitive therapists. Five of the therapists were licensed Ph.D. psychologists, and one was a psychiatric nurse practitioner (MSN). Four of the therapists had extensive CT experience (7 to 21 years) prior to the study initiation; two of the therapists began the trial with two years of CT experience and received additional training from the Beck Institute for Cognitive Therapy during the trial. All therapists followed the procedures outlined in standard texts of CT for depression (Beck et al., 1979; Beck, 1995). Sessions were twice weekly for the first 4 to 12 weeks (at the therapist’s discretion) and once weekly thereafter.

Measures

Depressive Symptoms

The Beck Depression Inventory-II (BDI-II; Beck, Steer, & Brown, 1996), a self-report measure, was used as a session-to-session indicator of depression severity.

Observer-rated process measures

Therapist Adherence

The Collaborative Study Psychotherapy Rating Scale (CSPRS; Hollon et al., 1988), containing 90 Likert-type items rated on a 7-point scale, was originally created to distinguish therapist behaviors in CT, interpersonal psychotherapy, and clinical management of pharmacotherapy. Higher scores indicate that a therapist engaged in a behavior more extensively. In our investigation, a subset of the CSPRS items was used to measure adherence to three subscales of therapy-specific techniques, as identified by Strunk and colleagues’ (2010) factor analysis. The 9-item Cognitive Methods subscale evaluates therapists’ efforts to help the patient re-evaluate his or her thoughts. For example, an item is, “Did the therapist help the client to use currently available evidence or information (including the client’s prior experiences) to test the validity of the client’s beliefs?” The 8-item Negotiating / Structuring subscale evaluates therapists’ efforts to negotiate therapy content and structure the therapy session. An example item is, “Did the therapist negotiate with the client assignments, changes in direction, or major emphases of the session in a way that gave the client the opportunity to have input?” Finally, the 5-item Behavioral Methods / Homework subscale evaluates use of behavioral strategies and the use of homework. An example item is, “Did the therapist encourage the client to record feelings, activities or events between sessions or review the client’s record of feelings, activities, or events?” Together, these scales reflect the extent of therapists’ adherence to the techniques of CT.

Therapeutic Alliance

The short form of the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989; Tracey & Kokotovic, 1989) is a set of 12 Likert-type items used to assess the therapeutic alliance. The total score was used as the primary measure of the alliance, following previous work with the WAI (see Horvath & Symonds, 1991; Klein et al., 2003; Martin et al., 2000). The WAI also breaks the alliance construct into three components: goal, task, and bond (Bordin, 1979; Horvath & Luborsky, 1993). The Goal subscale concerns the mutual agreement and importance placed upon specified targets of therapeutic intervention. The Task subscale refers to in-session behaviors that indicate a mutual agreement on how to achieve the goals. The Bond subscale refers to the positive personal attachments between the patient and therapist, including mutual trust, acceptance, and confidence.

Patient Facilitation / Inhibition

The CSPRS was modified and expanded (CSPRS-R; Brotman et al., 1999) to include assessments of patients’ facilitating or inhibiting therapist adherence. To assess patient facilitation vs. inhibition of the therapist-offered CT techniques,19 items were used. Each of these items assessed the patient’s facilitation or inhibition of therapists’ efforts as assessed by 19 of the 22 therapist adherence items described above. We judged 3 therapist adherence items inappropriate for use in this capacity. For all but 2 of the 19 items, the item read “Did the client impede or facilitate the therapist’s efforts in this domain?” The domain was therefore largely defined on the basis of the original CSPRS item assessing therapist adherence. For two other items, modifications were made to item wording for the sake of clarity. Raters also consulted a detailed manual (available upon request) with additional instructions for rating specific Patient Facilitation / Inhibition items. All Patient Facilitation / Inhibition items were rated on a 7-point Likert-type scale, assessing the patients’ responses to each of the therapist techniques assessed in the therapist adherence scales. At the extremes of the scales, a rating of “1” indicates that the patient strongly inhibited the progress of therapy for the identified therapist behavior; a rating of “7” indicates that the patient strongly facilitated the process of therapy for the identified therapist behavior. A rating of “4” indicates that the patient was neither facilitative nor inhibitory during the session. Thus, patients who are engaged in making an effort to use therapist-offered CT techniques obtain higher facilitation ratings. Patients who decline to try CT-offered strategies or those who raise significant objections obtained lower ratings.

Procedure

All process measures were rated by two raters trained in CT at the University of Pennsylvania (DRS and MAB). Each tape was rated independently by the raters, who were blind to outcome. The same judges rated all process measures for the first four sessions of CT. Sessions were observed and rated sequentially to assess the continuity of CT over the course of therapy, with the aim of yielding more accurate ratings of adherence and alliance. Therapy sessions were both video- and audiotaped. Raters watched and listened to videotape whenever possible; audiotapes were used only if the videotape was missing or presented some technical difficulty, such as poor sound quality.

Psychometric Properties of the Scales

Random effects intraclass correlation coefficients (ICCs) were calculated to assess the interrater reliabilities of the scales as rated independently by the two raters. ICC estimates were corrected for the number of raters. For the three therapist adherence scales, coefficients were: .62 for Cognitive Methods, .76 for Negotiating / Structuring, and .72 for Behavioral Methods / Homework. As shown in Table 1, although Cognitive Methods and Negotiating / Structuring were highly correlated (r = .75), Behavioral Methods / Homework was not highly correlated with either Cognitive Methods or Negotiating / Structuring (rs were .03 and .31, respectively). In contrast, the correlations among Patient Facilitation / Inhibition scales (formed on the basis of the therapist adherence scales) were all high (rs ranged from .59 to .80). Because previous research has not examined the factor structure of patient facilitation / inhibition, and factors based on therapist adherence were all highly correlated, a single overall Patient Facilitation / Inhibition scale was used for the primary analyses. The ICC for the Patient Facilitation / Inhibition index was .69. The ICC for the total WAI score was .79; coefficients for the subscales were .75 for Task, .76 for Goal, and .75 for Bond. These ICCs are similar to those reported by DeRubeis and Feeley (1990) and Feeley et al. (1999).

Table 1.

Average Correlations among Process Measures and Concurrent BDI-II scores Across Sessions 1 through 4

1 2 3 4 5 6
1. Cognitive Methods --
2. Negotiating / Structuring .58(4/4) --
3. Behavioral Methods / Homework .03(0/4) .36(3/4) --
4. Patient Facilitation / Inhibition .29(2/4) .53(4/4) .24(2/4) --
5. Alliance .36(3/4) .48(4/4) .29(2/4) .53(4/4) --
6. BDI-II −.02(0/4) .03(0/4) .17(1/4) −.04(0/4) .06(0/4) --
Open in a new tab

Note. Correlations reported are the average of the correlations at each of the first four sessions. Significance tests were performed at each session. The fractions in superscript indicate at how many of the first four sessions the relationship was significant (p < .05).

Results

Prior to examining predictors of symptom change, we examined the relationships among process measures as well as between each of these measures and the concurrent BDI-II scores. Correlations among these variables were computed separately at each of the first four sessions, and Fisher’s r to z transformations were used to convert the correlations into z-scores. These correlations were then averaged over the first four sessions. The averaged z-scores were then transformed back to r values to produce a mean correlation across sessions. The resulting mean correlations are reported in Table 1. With only one exception, process measures were all significantly correlated with one another at two or more sessions. The sole exception was the correlation between Cognitive Methods and Behavioral Methods / Homework, which was small and non-significant at each session. Alliance was associated with each of the three adherence measures as well as with Patient Facilitation / Inhibition of adherence. BDI-II scores were generally unrelated to process measures. The only process measure that was significantly associated with BDI-II scores was Behavioral Methods / Homework, and this relationship was only significant at one of the four sessions examined.

Analytic Approach

To examine session-to-session symptom change, the relation of a process measure in a given session to symptom change in the following session was examined in all 60 CT patients. Because a site by treatment interaction was identified in the primary analyses of the efficacy of the treatments, and since this interaction was driven in part by a small but non-significant site differences in the effect of CT (DeRubeis et al., 2005), site was entered as a covariate in these analyses. There was no evidence that the process measures were related to outcome differentially between the sites in any of the analyses we report. That is, there was no evidence of a site by process measure interaction. Finally, for ease of interpretation, signs have been adjusted so that a positive relationship always indicates that a process measure is associated with positive outcomes. Thus, a positive relationship indicates that a process measure is associated with more positive outcomes (i.e., lower symptom scores) in the next session.

Analyses of Session-to-Session Symptom Change

Repeated measures regression was performed using SAS Proc Mixed (Littell, Milliken, Stroup, & Wolfinger, 1996). In these analyses, the relation between the process measure of interest and symptom change in the following session was examined using data from several sessions in a single omnibus test.

A vector of BDI-II scores from sessions 2 through 5 served as the dependent variable. A vector of BDI-II scores from the prior session (1 through 4) was entered as a covariate, with BDI-II scores at a given session covaried out of the BDI-II scores of the subsequent session. Separate repeated measure regression models were performed for each of the independent variables of interest. If the test of the effect of the independent variable was significant, it indicated that the process measure predicted BDI-II scores at the following session after covarying the BDI-II scores from the current session. Four covariance structures (autoregressive, unstructured, compound symmetry, toeplitz) were examined using Akaike’s Information Criterion (AIC), Schwarz’s Bayesian Criterion, and −2 Res Log Likelihood. Unstructured was identified as the model with the best fit. Prior to addressing specific hypotheses, possible therapist effects on session-to-session outcome were examined. Therapist was not a significant predictor of session-to-session symptom change (p = .3).

As shown in Table 2, the Cognitive Methods therapist adherence scale was the strongest of three significant predictors of subsequent symptom improvement. High levels of Cognitive Methods were followed by greater symptom improvement across the early sessions. The Negotiating / Structuring scale was similarly predictive of session-to-session improvement. Patient Facilitation / Inhibition of therapist adherence was also significantly related to inter-session improvement, although the magnitude of this relationship was somewhat smaller. The alliance failed to predict short-term symptom improvement. We also examined the alliance subscales as predictors of inter-session symptom change. Neither the Task nor the Bond subscales were significant predictors in these models (r = .17, t = 1.27, p = .2 and r = .06, t = .42, p = .7, respectively). However, there was a non-significant trend for the Goal subscale to predict session-to-session BDI-II scores (r = .24, t = 1.84, p = .07).

Table 2.

Relationship between Process Measures and Symptom Improvement in the Following Session

Predictors of Subsequent Session-to-Session Symptom Change on the Beck Depression
Inventory-II
r t
Adherence
 Cognitive Methods .44 3.67***
 Negotiating / Structuring .38 3.11**
 Behavioral Methods / Homework −.04 −.30
Patient Facilitation / Inhibition .26 2.04*
Alliance Total Score .15 1.18
Open in a new tab

Note. r values represent the relationship between the amount of a process variable at session n and the degree of depression in session n + 1. Signs have been adjusted so that positive signs indicate that the process measure leads to lower depression scores in the following session. Site was included as a covariate in each model.

*

p < .05.

**

p < .01.

***

p < .001.

In an effort to examine whether the predictive relation between the therapist adherence scales and inter-session symptom change may be partly accounted for by the (non-significant) relationship of alliance and subsequent symptom change, we added the alliance as a predictor in each of the adherence analyses. The possibility of an interaction between alliance and each of the three adherence scales was also explored. None of these potential interactions between an adherence scale and alliance was significant (all ps > .1). The interaction terms were therefore dropped from the model. Therapist Cognitive Methods remained a significant predictor after controlling for alliance (r = .42, t = 3.49, p = .0009). Similarly, Negotiating / Structuring remained a significant predictor after controlling for alliance (r = .36, t = 2.94, p = .005). Alliance remained non-significant after controlling for either of these adherence scales (ps > .4). We also entered the alliance in a model with all of the significant predictors identified in previous models (viz., Cognitive Methods, Negotiating / Structuring, and Patient Facilitation / Inhibition). Only Cognitive Methods emerged as a unique predictor in this model (r = .31, t = 2.42, p = .02). Thus, when either of the adherence scales that emerged as a significant predictor was entered into the model with alliance, the adherence scale remained a significant predictor of subsequent symptom change, whereas the alliance did not. In similar models that included adherence and each of the alliance subscales, adherence remained significant in each model, and none of the alliance subscales was significant after adherence was covaried. Therefore, therapist adherence (specifically Cognitive Methods and Negotiating / Structuring), but not alliance, uniquely predicted improvement in inter-session intervals.

Magnitude of Symptom Change Predicted

The magnitude of change predicted by the process variables in session-to-session analyses provides important information about the potential clinical significance of any significant predictors. Our session-to-session analyses examine change in BDI-II scores from session 1 to session 5. Among the 52 patients with data available at sessions 1 and 5, the average decrease in BDI-II scores between these sessions was 5.1 (SD = 6.1). The Cohen’s d for this difference was .53, a medium effect. This decrease in BDI-II scores represents 32% of the average change among patients who completed the full 16-week treatment. To examine the reliability of this change in BDI-II scores, we calculated reliable change index scores on the BDI-II for each patient (Jacobson & Truax, 1991). From session 1 to 5, 67% of patients exceeded a cutoff of 1.96 SDs that has been proposed to indicate reliable change.

Prior Change and Process Measures

Because the same two raters made all ratings sequentially, it is possible that their ratings of process measures were biased by indirect knowledge of prior symptom improvement. Such bias could then contribute to biased estimates of the relation of a process measure to subsequent outcome. With this possibility in mind, we examined the association between process measures at sessions 2, 3 and 4 with symptom change from the previous session to the current session.

To do so, we used a repeated measures regression very similar to the approach we used for our primary session-to-session analyses. In these models, BDI scores at the current session (sessions 2 through 4) served as the vector of dependent variables. The prior session BDI and site were entered as covariates. As with the primary session-to-session models, unstructured was identified as the best-fitting covariance structure. Each process measure was examined in a separate model. Of all the process measures, only the alliance was significantly associated with prior symptom change, with higher alliance scores following greater symptom improvement (see Table 3). In addition, a non-significant trend emerged for the relation of Patient Facilitation / Inhibition with prior symptom improvement.1 Interestingly, the two therapist adherence measures which emerged as the strongest predictors of subsequent symptom change were unrelated to prior symptom change. Thus, it does not appear that the relation between these adherence measures and subsequent change can be attributed to any biasing effect of raters’ inferences about the magnitude of prior change.

Table 3.

Relation of Process Measures to Prior Session-to-Session Symptom Change on the Beck Depression Inventory-II

r t
Adherence
 Cognitive Methods −.13 −1.02
 Negotiating / Structuring .06 .44
 Behavioral Methods / Homework −.12 −.90
Patient Facilitation / Inhibition .25 1.94†
Alliance Total Score .34 2.76**
Open in a new tab

Note. r values represent the relationship between the amount of a process variable at session n and the degree of depression in session n (controlling for depression at session n - 1). Positive signs indicate that the process measure is associated with greater prior symptom change. Site was included as a covariate in each model.

†

p < .1.

*

p < .05.

**

p < .01.

Discussion

Our findings provide support for the view that important elements of therapist adherence to the methods of CT predict subsequent symptom change during treatment for depression. As a predictor of session-to-session symptom change, adherence to cognitive methods was the strongest of all the predictors examined. Negotiating / Structuring and Patient Facilitation / Inhibition of adherence also predicted session-to-session symptom change. In contrast, although a non-significant trend was obtained with the Goals subscale of the alliance measure, neither the alliance scores overall, or any of the subscales, yielded significant prediction of session-to-session improvement.

Therapist Adherence

Nearly one-third of the total magnitude of symptom improvement occurred in the first two weeks, during which patients participated in four therapy sessions. Whereas factors reflecting therapists’ use of cognitive therapy techniques, and their efforts to structure the session each predicted session-to-session change, variation in adherence to behavioral methods did not. These findings are therefore consistent with Beck et al.’s (1979) proposal that the primary active ingredient in CT for depression involves therapists’ efforts to encourage patients to examine and correct maladaptive thought patterns, and do not support a view that gives primacy to the role of behavioral approaches in the treatment of more severe depression (Dimidjian et al., 2006). Moreover, none of the three adherence scales were related to prior symptom change, suggesting that the predictive relations we observed were not the result of any biasing effect that could be attributed to raters’ inferences about symptom change already experienced by the patients whose sessions they rated.

It is important to distinguish our ratings of therapist adherence from ratings of therapist competence (see Sharpless & Barber, 2009). Raters evaluating competence consider not only the frequency and extent of therapist behaviors but also the context and appropriateness of the therapeutic strategies employed by the therapist. Thus, one might expect that competence ratings would be stronger predictors of outcome than adherence ratings. However, current approaches to rating competence may be more open to the influence of the idiosyncratic views of raters, which undermine reliability and predictive power. In fact, Jacobson and Gortner (2000) reported very poor reliability among expert raters of therapist competence in CT. (In a separate paper, we examine the predictive relation of competence ratings and subsequent symptom change (Strunk, Brotman, DeRubeis, & Hollon, 2009). Despite obtaining respectable estimates of inter-rater reliability, the magnitude of the competence-outcome association we observed was not as large as the effects for Cognitive Methods and Negotiating / Structuring in the current paper.)

Alliance

With regard to our efforts to find a predictive relationship between symptom change and the alliance, or any of its components and outcome, only one non-significant trend emerged, that between subsequent symptom change and the alliance component that reflects agreement on goals. When we examined adherence and alliance variables simultaneously, adherence measures were identified as significant predictors of symptom reduction. Our analyses of prior symptom change showed the alliance was predicted by prior symptom change. These results extend research showing that the alliance is likely to improve after particularly large symptom improvements, called sudden gains (Tang & DeRubeis, 1999). In our sample, across a number of early sessions, the alliance appeared to be a function of prior symptom change.

These findings are consistent with the small number of psychotherapy process studies which have examined predictors of subsequent symptom change in CT specifically (DeRubeis & Feeley, 1990; Feeley et al., 1999). With regard to the magnitude of the relation between alliance and outcome, our findings are also remarkably consistent with the evidence from dozens if not hundreds of studies of the association of the alliance and outcome. In meta-analyses of this larger group of studies (studies which have primarily not ruled out temporal confounds) the effect size estimates for the alliance-outcome relationship have been small (r = .22, Martin et al., 2000; r = .26, Horvath & Symonds, 1991). Thus, our results failed to support the view that variations in the formation of an alliance with a patient accounts for variation in symptom relief in CT. However, our data were collected in the context of a clinical trial in which all therapists made efforts to form good working relationships with their patients. Thus, if therapists were to adopt a strategy in which they adhered to some principles of treatment, but showed a disregard for the alliance, poorer outcomes would likely follow. Our data do not allow us to address this issue.

When Klein et al. (2003) reported that alliance assessed early in CBASP significantly predicted the subsequent rate of symptom change, they suggested that differences in statistical power or in the treatments under investigation (CT vs. CBASP) may have led to these discrepant findings. In the present study, the effect size estimate for the alliance-outcome relationship was small (i.e., an r of .15). This is quite similar to the effect size estimates that we calculated for the three statistical models reported by Klein et al. (rs of .10, .14, and .15; Rosnow & Rosenthal, 1996). Differences in power therefore may explain the differences between our results and those reported by Klein and colleagues, as their sample was nearly seven times as large as the present sample. However, the developers of CBASP also placed more importance on the therapeutic role of the alliance than did the developers of CT (McCullough, 1999; Beck et al., 1979). Regardless of whether the alliance plays a similar role in CT and CBASP, evidence published to date using the appropriate temporal sequencing has failed to find an alliance-outcome relationship for CT.

Patient Facilitation / Inhibition

The present findings further suggest that patients’ facilitation or inhibition of therapist adherence is an important early determinant of change in the following session inasmuch as this facilitation predicts symptom change early in treatment. Patients who were, for example, able and willing to provide specific examples of events or cognitions tended to show symptom gains in the following session. Patient Facilitation / Inhibition not only predicted subsequent symptom change, but there was also a trend for Patient Facilitation / Inhibition to be predicted by prior symptom change. Thus, patient’s facilitative efforts not only may contribute to symptom change, but that symptom change in turn may lead patients to increase their efforts at facilitation (i.e., a reciprocal relationship). Our measure of Patient Facilitation / Inhibition of therapist adherence shares some conceptual overlap with the alliance measures—particularly the alliance subscale assessing agreement on the means of achieving therapy goals (i.e., the Task subscale). However, the scales differ from each other in important ways. Whereas the alliance items assess agreement on the way to achieve any therapy goals, the Patient Facilitation / Inhibition items assess more directly patients’ compliance with specific therapist behaviors prescribed by the CT treatment manual. Patient Facilitation / Inhibition involves more than simply agreeing with the approach to achieving therapy goals. It reflects patients’ efforts to work actively in pursuit of the specific goals of CT.

Finally, it is important to note that the depression level during the prior session was controlled for in all of the session-to-session models. Thus, our results do not simply reflect the fact that patients with milder depressive symptoms were more engaged in treatment and therefore showed additional therapeutic gains. However, our analyses of process measures and prior symptom change suggest that while Patient Facilitation / Inhibition predicted subsequent symptom change, it was also predicted by prior symptom change. Thus, a cautionary note is warranted. It is possible that Patient Facilitation / Inhibition scores may have been biased by raters’ indirect knowledge of prior symptom change. While we are not aware of any reason this would be the case specifically for this measure, the pattern of association leaves this possibility open. Nonetheless, our session-to-session models indicate that when patients were actively involved in CT techniques, they tended to show improvements in depressive symptoms in the following session.

Limitations

There are several limitations that must be considered in this investigation. First, the independent variables in this study, namely adherence and alliance, were not manipulated. Thus, any causal inferences must be made with caution. Unfortunately, manipulation of these process variables is quite difficult. Second, there was no control or comparison group in our study. Therefore, as we discussed in the context of the CBASP, we do not know if the findings we obtained are specific to CT. In addition to the Klein et al. (2003) findings regarding CBASP, an early measure of the alliance has also been shown to predict subsequent symptom change in supportive-expressive therapy (r = .30, Barber et al., 2000). Thus, the alliance may be an important factor leading to improvement in therapies which use the therapist-patient relationship as a primary therapeutic tool.

Third, although our focus on therapist adherence to specific interventions revealed important predictive relations between therapist behaviors and outcome, it is important to note that interventions may not exert their effects through proposed mediators. For example, it is possible that the effect of therapists’ use of Cognitive Methods might be due to variables other than cognitive change among patients. Future research to address this issue is needed. Fourth, sessions were rated sequentially, and the same raters rated all process measures. There are several benefits to performing sequential ratings, including that raters were able to track and assess the continuity of CT assignments over the course of therapy. If a patient was given a homework assignment at the end of a session, raters were able to determine if the therapist reviewed that homework in the following session. Similarly, knowledge of a prior session could help a rater judge whether the therapist and patient agree on the current approach to achieve their goals. However, there are disadvantages to sequential ratings where raters rate all constructs as well. It is possible that through exposure to multiple sessions, raters could determine the extent to which patients had experienced prior symptom change. Knowledge of this prior change may have biased ratings. Although the lack on an association between prior change and therapist adherence suggests these ratings were not substantially biased, the relation between prior symptom change and Patient Facilitation / Inhibition allows for the possibility of bias. While we have no reason to expect bias would uniquely impact this measure, future research on patient facilitation / inhibition (and other process variables) might use separate raters for each session to guard against the possibility of such bias.

Future Research

This investigation was undertaken in the context of a larger study in which adherence and alliance were observed. Future investigations should include attempts to vary the levels of alliance and adherence systematically. When such variables are manipulated, the cause-effect relationship under investigation can be determined with greater certainty.

Although this investigation focused on the early part of CT, when symptom change occurs most rapidly, research might also investigate other portions of treatment; for example, work on identifying and modifying core beliefs typically occurs late in treatment in CT (Beck, 1995). An investigation of this component of CT would likely require a focus on later sessions.

Finally, researchers are encouraged to examine the role of therapist adherence, patient facilitation / inhibition and the therapeutic alliance in other therapeutic modalities. In such studies, investigations of process measures assessed using longitudinal analyses and appropriate temporal sequencing may greatly facilitate the elucidation of key change processes. A better understanding of which factors are therapeutic in CT, as well as in other interventions, has the promise of informing theories of therapeutic change. With such knowledge, researchers and clinicians may refine therapeutic procedures, leading to more effective interventions for those who seek psychotherapy.

Implications and Conclusion

These findings speak to the validity of the cognitive model of depression. The positive adherence results provide indirect support for Beck’s original ideas (1967; 1976) that depression derives from maladaptive, negative thinking patterns. Our results suggest therapist adherence, such as therapist efforts to encourage patients to evaluate their thoughts and beliefs, appear to be driving symptom change. Variations in the therapeutic alliance were not related to subsequent symptom change in CT. Identifying process variables most predictive of change should inform efforts to train therapists in CT as this treatment is disseminated.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1

Rather than process measures being a function of symptom change since the last session, it is also possible that process measures may be biased by raters’ exposure to symptom change from session 1 to each session rated. To examine this possibility, we ran the repeated measures regression analyses just described except rather than covarying BDI scores at the prior session, BDI scores at session 1 were entered as covariates for each session. Thus, this model examines the relationship of a process measure with symptom change occurring between session 1 and the rated session. Results closely paralleled those reported in Table 3, with only one previously non-significant result being significant. The significant effect for the alliance remained significant. The one result which was now significant was that patient facilitation / inhibition was significantly related to greater prior symptom change (r = .30, t = 2.39, p = .02).

References

  1. Ablon JS, Jones EE. Psychotherapy process in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology. 1999;67:64–75. doi: 10.1037//0022-006x.67.1.64. [DOI] [PubMed] [Google Scholar]
  2. Barber JP, Connolly MB, Crits-Christoph P, Gladis L, Siqueland L. Alliance predicts patients’ outcome beyond in-treatment change in symptoms. Journal of Consulting and Clinical Psychology. 2000;68:1027–1032. doi: 10.1037//0022-006x.68.6.1027. [DOI] [PubMed] [Google Scholar]
  3. Barber JP, Crits-Christoph P, Luborsky L. Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology. 1996;64:619–622. doi: 10.1037//0022-006x.64.3.619. [DOI] [PubMed] [Google Scholar]
  4. Beck AT. Depression: Clinical, experimental, and theoretical aspects. New York: Harper & Row; 1967. [Google Scholar]
  5. Beck AT. Cognitive therapy and the emotional disorders. New York: Meridian; 1976. [Google Scholar]
  6. Beck AT, Rush AJ, Shaw BF, Emery G. Cognitive therapy of depression. New York: Guilford; 1979. [Google Scholar]
  7. Beck AT, Steer RA, Brown GK. Manual for the Beck Depression Inventory. 2nd ed. San Antonio, TX: The Psychological Corporation; 1996. [Google Scholar]
  8. Beck JS. Cognitive therapy: Basics, beyond. New York: Guilford; 1995. [Google Scholar]
  9. Bordin ES. The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research and Practice. 1979;16:252–260. [Google Scholar]
  10. Brotman MA, Strunk DR, Stirman SW, DeRubeis RJ. The Collaborative Study Psychotherapy Rating Scale-Revised (CSPRS-R): Revised and expanded version. University of Pennsylvania: Philadelphia; 1999. Unpublished manuscript. [Google Scholar]
  11. Brown H, Prescott R. Applied Mixed Models in Medicine. Chichester, England: Wiley; 1999. [Google Scholar]
  12. DeRubeis RJ, Feeley M. Determinants of change in cognitive therapy for depression. Cognitive Therapy and Research. 1990;14:469–482. [Google Scholar]
  13. DeRubeis RJ, Gelfand LA, Tang TZ, Simons AD. Medications versus cognitive behavior therapy for severely depressed outpatients: Mega-analysis of four randomized comparisons. The American Journal of Psychiatry. 1999;156:1007–1013. doi: 10.1176/ajp.156.7.1007. [DOI] [PubMed] [Google Scholar]
  14. DeRubeis RJ, Hollon SD, Amsterdam JD, Shelton RC, Young PR, Salomon RM, et al. Cognitive Therapy vs Medications in the Treatment of Moderate to Severe Depression. Archives of General Psychiatry. 2005;62:409–416. doi: 10.1001/archpsyc.62.4.409. [DOI] [PubMed] [Google Scholar]
  15. Dimidjian S, Hollon SD, Dobson KS, Schmaling KB, Kohlenberg RJ, Addis ME, et al. Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology. 2006;74:658–670. doi: 10.1037/0022-006X.74.4.658. [DOI] [PubMed] [Google Scholar]
  16. Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ. Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods. 1999;4:272–299. [Google Scholar]
  17. Feeley M, DeRubeis RJ, Gelfand LA. The temporal relation of adherence and alliance to symptom change in cognitive therapy for depression. Journal of Consulting and Clinical Psychology. 1999;67:578–582. doi: 10.1037//0022-006x.67.4.578. [DOI] [PubMed] [Google Scholar]
  18. First MB, Spitzer RL, Gibbon M, Williams JBW. Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition With Psychotic Screen (SCID-I/P W/ PSY SCREEN) New York, NY: Biometrics Research, New York State Psychiatric Institute; 2001. [Google Scholar]
  19. Garrat G, Ingram RE, Rand KL, Sawalani G. Cognitive processes in cognitive therapy: Evaluation of the mechanisms of change in the treatment of depression. Clinical Psychology: Science & Practice. 2007;14:224–239. [Google Scholar]
  20. Gaston L, Marmar C, Gallagher D, Thompson L. Alliance prediction of outcome beyond in-treatment symptomatic change as psychotherapy processes. Psychotherapy Research. 1991;1:104–112. [Google Scholar]
  21. Hamilton M. A rating scale for depression. Journal of Neurology, Neurosurgery & Psychiatry. 1960;23:56–62. doi: 10.1136/jnnp.23.1.56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hill CE, O'Grady KE, Elkin I. Applying the Collaborative Study Psychotherapy Rating Scale to rate therapist adherence in cognitive-behavior therapy, interpersonal therapy, and clinical management. Journal of Consulting and Clinical Psychology. 1992;60:73–79. doi: 10.1037//0022-006x.60.1.73. [DOI] [PubMed] [Google Scholar]
  23. Hollon SD, DeRubeis RJ, Shelton RC, Amsterdam JD, Salomon RM, O'Reardon JP, et al. Prevention of Relapse Following Cognitive Therapy vs Medications in Moderate to Severe Depression. Archives of General Psychiatry. 2005;62:417–422. doi: 10.1001/archpsyc.62.4.417. [DOI] [PubMed] [Google Scholar]
  24. Hollon SD, Evans MD, Auerbach A, DeRubeis RJ, Elkin I, Lowery A, et al. Development of a system for rating therapies for depression: Differentiating cognitive therapy, interpersonal psychotherapy and clinical management pharmacotherapy. Vanderbilt University: Nashville; 1988. Unpublished manuscript. [Google Scholar]
  25. Hollon SD, Stewart MO, Strunk DR. Enduring effects for cognitive behavior therapy in the treatment of depression and anxiety. Annual Review of Psychology. 2006;57:285–315. doi: 10.1146/annurev.psych.57.102904.190044. [DOI] [PubMed] [Google Scholar]
  26. Horvath AO, Greenberg LS. Development and validation of the working alliance inventory. Journal of Counseling Psychology. 1989;2:223–233. [Google Scholar]
  27. Horvath AO, Luborsky L. The role of the therapeutic alliance in psychotherapy. Journal of Consulting and Clinical Psychology. 1993;61:561–573. doi: 10.1037//0022-006x.61.4.561. [DOI] [PubMed] [Google Scholar]
  28. Horvath AO, Symonds BD. Relation between working alliance and outcome in psychotherapy: A meta-analysis. Journal of Counseling Psychology. 1991;38:139–149. [Google Scholar]
  29. Ilardi SS, Craighead WE. The role of nonspecific factors in cognitive-behavior therapy for depression. Clinical Psychology: Science and Practice. 1994;1:138–156. [Google Scholar]
  30. Jacobson NS, Gortner ET. Can depression be de-medicalized in the 21st century: Scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy. 2000;38:103–117. doi: 10.1016/s0005-7967(99)00029-7. [DOI] [PubMed] [Google Scholar]
  31. Jacobson NS, Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting & Clinical Psychology. 1991;59:12–19. doi: 10.1037//0022-006x.59.1.12. [DOI] [PubMed] [Google Scholar]
  32. Judd CM, Kenny DA. Process analysis: Estimating mediation in treatment evaluations. Evaluation Review. 1981;5:602–619. [Google Scholar]
  33. Klein DN, Schwartz JE, Santiago NJ, Vivian D, Vocisano C, Castonguay LG, et al. Therapeutic alliance in depression treatment: Controlling for prior change and patient characteristics. Journal of Consulting and Clinical Psychology. 2003;71:997–1006. doi: 10.1037/0022-006X.71.6.997. [DOI] [PubMed] [Google Scholar]
  34. Krupnick JL, Sotsky SM, Simmens S, Moyer J, Elkin, Watkins J, Pilkonis PA. The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental Health treatment of depression collaborative research program. Journal of Consulting and Clinical Psychology. 1996;64:532–539. doi: 10.1037//0022-006x.64.3.532. [DOI] [PubMed] [Google Scholar]
  35. Littell RC, Milliken GA, Stroup WW, Wolfinger RD. SAS System for Mixed Models. Cary, NC: SAS Institute, Inc; 1996. [Google Scholar]
  36. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical Psychology. 2000;68:438–450. [PubMed] [Google Scholar]
  37. McCullough JP. Treatment for Chronic Depression : Cognitive Behavioral Analysis System of Psychotherapy (CBASP) New York: Guilford; 1999. [DOI] [PubMed] [Google Scholar]
  38. Raue PJ, Goldfried MR, Barkham M. The therapeutic alliance in psychodynamic-interpersonal and cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology. 1997;65:582–587. doi: 10.1037//0022-006x.65.4.582. [DOI] [PubMed] [Google Scholar]
  39. Rosnow RL, Rosenthal R. Computing contrasts, effect sizes, and counternulls on other people’s published data: General procedures for research consumers. Psychological Methods. 1996;1:331–340. [Google Scholar]
  40. Sharpless BA, Barber JP. A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology. Clinical Psychology Review. 2009;29:47–56. doi: 10.1016/j.cpr.2008.09.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Stiles WB, Shapiro DA. Disabuse of the drug metaphor: Psychotherapy process-outcome correlations. Journal of Consulting and Clinical Psychology. 1994;62:942–948. doi: 10.1037/0022-006x.62.5.942. [DOI] [PubMed] [Google Scholar]
  42. Strunk DR, DeRubeis RJ. Cognitive therapy for depression: A review of its efficacy. Journal of Cognitive Psychotherapy: An International Quarterly. 2001;15:289–297. [Google Scholar]
  43. Strunk DR, Brotman MA, DeRubeis RJ, Hollon SD. Therapist competence in Cognitive Therapy for Depression: Predicting subsequent symptom change. 2009 doi: 10.1037/a0019631. Manuscript under review. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Strunk DR, Cooper AA, Ryan ET. Adherence and alliance in Cognitive Therapy for Depression when combined with medication: Predictors of retention and subsequent symptom change. 2010 Manuscript in preparation. [Google Scholar]
  45. Tang TZ, DeRubeis RJ. Sudden gains and critical sessions in cognitive-behavioral therapy for depression. Journal of Consulting and Clinical Psychology. 1999;67:894–904. doi: 10.1037//0022-006x.67.6.894. [DOI] [PubMed] [Google Scholar]
  46. Tang TZ, DeRubeis RJ, Beberman R, Pham T. Cognitive Changes, Critical Sessions, and Sudden Gains in Cognitive-Behavioral Therapy for Depression. Journal of Consulting and Clinical Psychology. 2005;73:168–172. doi: 10.1037/0022-006X.73.1.168. [DOI] [PubMed] [Google Scholar]
  47. Tang TZ, DeRubeis RJ, Hollon SD, Amsterdam J, Shelton R. Sudden gains in cognitive therapy of depression and depression relapse/recurrence. Journal of Consulting and Clinical Psychology. 2007;75:404–408. doi: 10.1037/0022-006X.75.3.404. [DOI] [PubMed] [Google Scholar]
  48. Tracey TJ, Kokotovic AM. Factor structure of the Working Alliance Inventory. Psychological Assessment. 1989;1:207–210. [Google Scholar]
  49. Wampold BE. The Great Psychotherapy Debate. Mahwah, NJ: Lawrence Erlbaum; 2001. [Google Scholar]

RESOURCES