Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Jun 1.
Published in final edited form as: Behav Res Ther. 2020 Mar 24;129:103608. doi: 10.1016/j.brat.2020.103608

Outcomes, Skill Acquisition, and the Alliance: Similarities and Differences between Clinical Trial and Student Therapists

Lizabeth A Goldstein a,b,c,*, Abby D Adler Mandel a,d, Robert J DeRubeis e, Daniel R Strunk a
PMCID: PMC7329568  NIHMSID: NIHMS1582894  PMID: 32276239

Abstract

Considerable evidence from clinical trials supports the efficacy of cognitive therapy (CT) of depression. Less is known about outcomes when provided in other contexts, such as when provided by student therapists. We conducted a non-randomized comparison of student therapists vs. clinical trial therapists on change in depressive symptoms, dropout, change in CT skills, and therapeutic alliance among 100 clients with moderate to severe depression. Treatment manual and duration were the same. Clients treated by student therapists had largely comparable outcomes on depressive symptom change, therapeutic alliance, and CT skills. Results supported non-inferiority of student therapists on change in depressive symptoms, but non-inferiority was not supported when using an interviewer evaluated measure of depression. Evidence of non- inferiority was also obtained for client CT skills and therapeutic alliance. In fact, conventional superiority analyses indicated student therapists outperformed clinical trial therapists on alliance and CT skills. The rate of dropout among student therapists (30%) was numerically higher than among clinical trial therapists (17%) and our results did not support non-inferiority on dropout. CT provided by student therapists can achieve outcomes similar to those in a clinical trial, but more research about dropout is needed.

Keywords: cognitive therapy, depression, therapeutic alliance, therapeutic outcomes, therapist trainees, treatment dropouts

Cognitive therapy (CT) has been established as an efficacious treatment for depression across a number of clinical trials (Gloaguen, Cottraux, Cucherat, & Blackburn, 1998; Hollon, Thase, & Markowitz, 2002). In these trials, CT is often provided by highly qualified therapists. To what extent the effects of this treatment are achieved by less expert therapists remains largely unknown. In a recent meta-analysis, Johnsen and Friborg (2015) found that trained psychologists achieved significantly better outcomes using CT for depression than did psychology students. Concerns about the ability of trainees (i.e., psychology students) to provide effective treatments led these authors to recommend that students should provide CT for clients “with primarily mild, or at the maximum, a moderate degree of depression” (p. 15). With students spending several years under supervision prior to licensure and several thousand students completing their doctoral training each year (Association of Psychology Postdoctoral and Internship Centers, 2020), a substantial portion of the psychological services provided are provided by students. Thus, understanding whether student-provided psychological treatments produce benefits comparable to those achieved by trained therapists is critically important.

However, to determine whether student therapists can deliver CT in a manner that yields comparable outcomes to those achieved in clinical trials, research methods must be as similar as possible across clients seen by each type of therapist. This includes using the same treatment manual, providing CT for the same duration and, to the extent possible, treating clients with the same features. One step toward achieving comparability across samples of clients is to use the same inclusion and exclusion criteria. In addition, a comparison focused on the treatment of moderate to severe depression would be most informative, as it is only in the context of such cases that established treatments have demonstrated specific benefits (i.e., yielding greater symptom reductions than non-specific control conditions; Fournier et al., 2010; Driessen, Cuijpers, Hollon, & Dekker, 2010). Moreover, there is evidence that therapist effects are larger among clients with more severe symptoms (Saxon & Barkham, 2012). In this paper, we compare symptom reduction achieved by student therapists and clinical trial therapists in their work with moderate to severely depressed clients. Beyond considering symptom outcomes, we also examine three other important factors: client dropout, clients’ ratings of the therapeutic alliance, and client acquisition of CT skills. To place our analyses in context, we first review available evidence with regard to the generalizability of CT’s effects on symptoms and then provide our rationale for focusing on the three other variables of interest.

How robust are the effects of CT?

Several studies have compared the outcomes of depressed clients treated with CT in clinical trials with the outcomes of clients treated with CT in standard clinical settings, as described below. Across these seven studies, settings have included: private practice (Persons, Bostrom, & Bertagnolli, 1999), the UK’s National Health Service (Westbrook & Kirk, 2005), an outpatient CT clinic (Gibbons et al., 2010), a university outpatient clinic (Lutz, Schiefele, Wucherpfennig, Rubel, & Stulz, 2016), a community mental health center (Merrill, Tolbert, & Wade, 2003), an outpatient medical center clinic (Forand, Evans, Haglin, & Fishman, 2011), and an outpatient mood disorders clinic (Peeters et al., 2013). Many of these studies reported including trainees or students as therapists (Forand et al., 2011; Gibbons et al., 2010; Lutz et al., 2016; Westbrook & Kirk, 2005), and several others included therapists who were likely to be less experienced or expert than therapists providing CT in clinical trials. Although results of these studies generally indicated that outcomes did not differ across clinical trials and clinical practice, in two studies there were differences favoring clinical trial therapists among clients who had higher initial symptom severity (Forand et al., 2011; Gibbons et al., 2010). Nonetheless, the interpretation of the comparisons presented in these studies has been clouded by one or more key methodological differences across clinical trial and clinical practice samples. Next, we consider these methodological differences.

One important difference has been duration of treatment. In each the seven studies noted above, clinical practice involved open duration treatment whereas the clinical trial comparators did not. Therefore, if CT in clinical practice had been less effective, it may have caught up to results of clinical trials by providing treatment for a longer period. Gibbons and colleagues (2010) addressed this concern by cutting off data for inclusion at 15 weeks (or 20 sessions in another analysis) across the clinical practice and clinical trial samples. Using this approach, the clinical practice sample exhibited comparable symptom improvement to the clinical trial sample overall; however, the clinical practice sample underperformed the clinical trial sample among more severely depressed clients. In addition, this analysis does not address the possibility that knowledge of the limited duration of treatment may have influenced the therapists’ delivery of CT or the clients’ efforts to make use of the treatment. Thus, comparisons of CT in clinical practice and clinical trials would ideally examine treatment of the same planned duration.

Second, studies have varied with regard to whether therapists were asked to follow a treatment manual. For example, Westbrook and Kirk’s (2005) examination of data from the UK National Health Service involved therapists using “no standard treatment manual” (p. 1246). Similarly, Forand and colleagues (2011) examined “nonstandardized nonmanualized CBT” (p. 612). In contrast, clinical trials of CT have nearly universally reported using manuals, most often Beck et al. (1979).

A third key methodological issue involves the likelihood of differences in pre-treatment client characteristics across studies. Clients have differed across a number of demographic variables as well as whether clients met criteria for major depressive disorder (MDD). Even when MDD was assessed, the assessment has often not been on the basis of semi-structured interviews in the clinical practice samples, raising concerns about diagnostic validity. This concern also extends to questions about the presence and frequency of comorbid conditions. The use of clear, well-assessed inclusion and exclusion criteria utilized in clinical trials have tended to be lacking in the clinical practice comparison samples examined in the literature. In an exception to this, Lutz et al. (2016) compared CT in a German outpatient clinic and a U.S. based clinical trial. They restricted the German sample to those that met trial inclusion and exclusion criteria and then adjusted for pre-treatment differences across samples using propensity score matching. Following the restriction and adjustment, their results suggested that CT appeared to perform comparably to that observed in the trial; however, this comparability may have been achieved by providing more than twice as many sessions in the German clinic.

In summary, the existing literature offers some evidence that the outcomes achieved by CT in clinical settings are promising, but a number of differences across clinical trial and clinical practice samples limit inferences from these comparisons. Even when authors have been careful to ensure comparability on some dimensions, other differences have often complicated the comparisons. In addition, the results have been less promising when considering more severely depressed clients. Therefore, whether student therapists can achieve comparable outcomes remains a vital question (Johnsen & Friborg, 2015).

Beyond Symptomatic Outcomes

Beyond examining therapeutic outcome, there is a need to examine other variables characterizing CT provided by clinical trial vs. student therapists. We focus on the following three: (1) clients’ mastery of the skills of CT, (2) client dropout, and (3) clients’ rating of the therapeutic alliance.

First, we examined clients’ acquisition of the skills of CT. Acquisition of these skills is a major target of treatment, with most therapeutic activities aimed at achieving this goal. CT skill acquisition has been related to both acute outcome as well as risk of relapse following treatment (Barber & DeRubeis, 2001; Adler, Strunk, & Fazio, 2015; Strunk, DeRubeis, Chiu, & Alvarez, 2007). Second, we examined dropout, as it is an important aspect of therapeutic outcome. According to one meta-analysis (Swift & Greenberg, 2012), diverse treatments for a variety of diagnoses have been associated with a higher risk of dropout when provided by trainees as compared to more experienced therapists. However, this relation is less clear among cognitive behavioral therapies specifically (Fernandez, Salem, Swift, & Ramtahal, 2015). Finally, we examined the therapeutic alliance. This variable was of special interest because of the common concern that student therapists may not garner the respect or confidence of their clients. In addition, the alliance is the most studied psychotherapy process variable, which has been found to exhibit a modest, but reliable association with outcome across diverse psychotherapies (Martin, Garske, & Davis, 2000). In studies examining the alliance as a predictor of subsequent symptom change in CT for depression specifically, the evidence has been less consistent, with just two studies identifying evidence of an alliance-outcome relation (Strunk, Cooper, Ryan, DeRubeis, & Hollon, 2012; Webb et al., 2011; for an in depth discussion, see Pfeifer & Strunk, 2015).

Purpose of this Study

The purpose of this study was to compare outcomes achieved by clients treated by student therapists vs. clinical trial therapists. To do so, we compare outcomes among clients treated at our university-based training clinic with clients treated in the CT condition of the DeRubeis et al. (2005) trial. Student therapists and clinical trial therapists all provided CT for the same duration, followed the same treatment manual, and worked with clients who met common inclusion and exclusion criteria. Beyond examining outcome, we also examine potential differences in client dropout, clients’ acquisition of CT skills, and clients’ rating of the alliance.

Methods

Participants

Clinical trial data were drawn from the CT condition of a clinical trial comparing CT, pharmacotherapy (paroxetine augmented by lithium carbonate or desipramine hydrochloride if necessary), and pill placebo in the treatment of moderate to severe depression (DeRubeis et al., 2005). This study occurred across two sites: the University of Pennsylvania (with highly experienced cognitive therapists) and Vanderbilt University (with one highly experienced and two less-experienced cognitive therapists). All 60 clients assigned to the CT condition of the clinical trial were included in our analyses. Student therapist data were drawn from a research study of CT conducted at a training clinic at The Ohio State University (Adler et al., 2015). Of the 67 participants in that study, we included the 41 who had baseline Hamilton Rating for Depression (HRSD) scores of 20 or higher. As a result, clients across both samples had been given a primary diagnosis of MDD per the Structured Clinical Interview for DSM-IV and a HRSD score of 20 or higher. We also excluded one participant in the training clinic sample who met criteria for borderline personality disorder (an exclusion in the clinical trial). This results in a final training clinic sample of 40 participants. Across both samples, potential participants were excluded if they met criteria for substance dependence in the past 6 months or had a history of Bipolar I disorder or psychosis. For full details, see Online Supplement A.

Whereas participants from the CT condition of the clinical trial were not on psychotropic medication, participants from the training clinic were allowed to be on medication if their dose was stable for 1 month prior to intake and they made a commitment to maintaining a stable dose over the course of the study. Of the 40 training clinic participants, 18 were on medications. Neither study had any exclusion based upon previous participation in any psychotherapy. All participants provided informed consent prior to participation in the study, and the study was approved by an Institutional Review Board.

Therapists

Five clinical trial therapists were licensed PhD-level psychologists, and one was a psychiatric nurse practitioner. Psychotherapy experience ranged from 5 to 21 years at the outset of the clinical trial. The highly experienced CT therapists had 7–21 years of specific CT experience, whereas the two less experienced therapists had 2 years of CT experience and received additional training through the Beck Institute for Cognitive Therapy during the trial. In addition, at each clinical trial site, study therapists met weekly to discuss cases.

At the training clinic site, four advanced graduate students in a clinical psychology PhD program with 1–2 years of clinical experience provided treatment. Prior to the study, they received approximately 100 hours of training in CT, including extensive experiential role- playing. The last author provided weekly individual and group supervision.

Across both studies, all sessions were video recorded. Independent ratings of recorded sessions by student and clinical trial therapists were conducted using the Cognitive Therapy Scale (Academy of Cognitive Therapy, n.d.). Though the raters and the number of observations differed across these evaluations, evidence suggested that therapists tended to deliver sessions with a score of 40 or higher (a score indicating acceptable competence). For additional details about competence ratings, see Strunk, Brotman, DeRubeis, and Hollon (2010) for the clinical trial therapists and Adler et al. (2015) for the student therapists.

Measures

Structured Clinical Interview for DSM-IV (SCID-I: First, Spitzer, Miriam, & Williams, 2002; SCID-II: First, Gibbon, Spitzer, Williams & Benjamin, 1997).2 Trained interviewers administered these measures at intake to determine diagnostic status for MDD as well as other Axis I and Axis II disorders. For a limited number of cases, evaluators reviewed recordings of these interviews and provided assessments of MDD diagnosis. At the training clinic, 12 cases were assessed and 100% reliability was achieved (k = 1.00). In the clinical trial, 12 cases were assessed and good reliability was achieved (k = .80; DeRubeis et al., 2005).

Hamilton Rating Scale for Depression (HRSD; Hamilton, 1960; Williams, 1988). Trained interviewers administered the 17-item version of the HRSD, modified to assess atypical symptoms. At the training clinic, evaluators administered the interview at intake, week 4, and post-treatment. In the clinical trial, evaluators administered the interview at intake, weekly for the first four weeks and then bi-weekly through the post-treatment assessment. To assess reliability in each study, evaluators reviewed recordings of a small number of interviews. At the training clinic, 15 interviews were assessed and inter-rater reliability was high (ICC = .99; Adler et al., 2015). In the clinical trial, 24 interviews were assessed and inter-rater reliability was also high (ICC = .96; DeRubeis et al., 2005).

Beck Depression Inventory – 2nd Edition (BDI-II; Beck, Steer, & Brown, 1996). This 21-item self-report questionnaire measures depressive symptom severity, with responses ranging on a scale from 0–3 and total scores ranging from 0 (no symptoms of depression) to 63 (severe depression). Across all clients, the instructions were modified slightly to ask participants about their symptoms over the past week. Previous research has shown the BDI-II has high internal consistency (a =. 92 among outpatients) and test-retest reliability (r = .93 among outpatients across one week; Beck, Steer, & Brown).

Ways of Responding Scale (WOR; Barber & DeRubeis, 1992). Participants are asked to imagine themselves in six stressful scenarios (e.g., repeated job application rejections). Following each scenario, they are asked to imagine vividly a set of negative thoughts (e.g., “Will I ever get a job? There just doesn’t seem to be any point in applying.”), after which they are asked to report what their further thoughts and actions would be. Raters from a three-person team, blind to assessment point, study, and client made ratings of the overall quality of the client’s response for each scenario (WOR Quality) on a 1 to 7 scale, with 1 reflecting little if any use of CT skills (e.g., blaming self and drinking to reduce negative emotions) and 7 reflecting optimal CT skills (e.g., reviewing evidence that job market is competitive, asking friend to review resume and planning to continue applying). Scores on these indices reflect the average rating for each client across six scenarios. Prior to making study ratings, raters completed approximately 30 hours of training, including practice on non-study questionnaires. Agreement was high, with the ICC for WOR quality corrected for 3 raters estimated at .97.

Working Alliance Inventory (WAI; Busseri & Tyler, 2003; Horvath & Greenberg, 1989; Tracey & Kokotovic, 1989). We assessed the therapeutic alliance with the 12-item client- rated version of the WAI. Across all clients, we examined the WAI as assessed during the second week of CT. The WAI assesses the quality of both the affective aspects of the therapeutic relationship and the agreement between therapist and client on overall treatment goals and on the steps to achieve therapeutic goals. A total score reflects the alliance, with higher scores indicating a stronger therapeutic alliance. Previous research has demonstrated the WAI has high internal consistency reliability (a =. 91 at session four; Busseri & Tyler).

Beck Anxiety Inventory (BAI; Beck, Epstein, Brown, & Steer, 1988). This 21-item self- report questionnaire measures the severity of anxiety symptoms. Responses for each item range from 0 to 3, allowing for a total scores ranging from 0 to 63, with higher scores indicating more severe symptoms of anxiety. This measure was utilized for possible inclusion in creating propensity scores (described below). Previous research has shown the BAI has high internal consistency reliability (a =. 92) and test-retest reliability (r = .75 over 1 week; Beck, Epstein, Brown, & Steer).

Procedures

Following the intake assessment, clients were offered 16 weeks of CT, with twice-weekly sessions offered for the first four weeks and then tapering to once-weekly thereafter. Therapists administered CT per Beck, Rush, Shaw, and Emery (1979). Treatment was offered free of charge, and clients were offered an honorarium for study assessments and tasks before, during, and after treatment. The amount offered for assessments over the course of acute treatment (including the intake and post-treatment evaluation) was $100 in the clinical trial and $110–$140 in the training clinic.

Clients were considered treatment dropouts if they stopped attending therapy sessions and either directly declined to continue treatment or did not respond to multiple attempts to contact them. Therapists at both the clinical trial and student sites made significant efforts to re-engage clients in treatment.

Analytic Strategy

Although the DeRubeis et al. (2005) trial was a two site study, because there was no site difference in outcome in the CT condition, data from the sites were combined to form the clinical trial sample. Thus, comparisons focus on the clinical trial (n = 60) vs. the student therapist (n = 40) sample. Initial HRSD scores were included as a covariate in models of CT skills, alliance, and dropout. In addition, as the comparison between clients in the clinical trial and those treated by student therapists was not randomized, we calculated propensity scores for each dependent variable of interest selecting from the following baseline variables: age, gender, marital status, race, education level, income level, single episode vs. recurrent MDD, BAI score, comorbid anxiety disorder diagnosis, comorbid post-traumatic stress disorder diagnosis, and comorbid personality disorder diagnosis. Some clients were missing baseline data for income (n = 7) and BAI score (n = 3); values were imputed using single (stochastic) imputation. Variables that predicted each specific outcome at the p < .10 level were included in a model used to generate a propensity score for use in models of that outcome. The predictors retained were as follows: (1) for BDI-II: age, marital status, race, income, and BAI score; (2) for HRSD: marital status, race, and income; (3), for dropout: income and BAI score; (4) for CT skills: PTSD status, and (5) for the alliance: age. These propensity scores were entered as covariates into their respective subsequent analyses.

To assess whether differences between clients treated by student vs. clinical trial therapists were smaller than a pre-specified margin (i.e., delta), we conducted one-sided non- inferiority analyses. Using this approach, evidence of non-inferiority would indicate that clients treated by the student therapists scored either better or no worse than a pre-specified margin (delta) compared to clinical trial therapists. One-sided 95% confidence intervals were calculated for the mean difference between clients of student therapists and clients of clinical trial therapists, which must remain within the margin to be considered non-inferior. For symptom change, CT skill change, and the alliance, we used a delta reflecting a Cohen’s d of .3, the approximate magnitude of the drug-placebo difference observed in the depression literature (Kirsch et al., 2008).3 This resulted in deltas set to 3.0 for the BDI-II, 2.0 for HRSD, .28 for CT skill change, and 3.0 on the alliance. For dropout, we conducted a non-inferiority test of proportions, with success defined as treatment completion. We set the delta to 10%, meaning that the analysis tests whether clients seen by student therapists achieved completion rates of no worse than 10% lower than that of clinical trial therapists.

For comprehensiveness, we also conducted conventional superiority analyses to evaluate potential differences between clients seen by student versus clinical trial therapists. We examined changes in depressive symptoms in separate hierarchical linear models (HLM) for the HRSD and BDI-II. Although the samples differed in the amount of data available for the HRSD, all available data were included in the models. In the clinical trial, the HRSD was administered at intake, weeks 1, 2, 3, 4, 6, 8, 10, 12, and 14 of treatment, and post-treatment. In the training clinic, the HRSD was administered only at intake, week 4, and post-treatment. The BDI-II was collected at the same assessment points as the HRSD in addition to at each therapy session. To reflect typical and shared BDI-II administration time-points, the BDI-II data used in the models was limited to 22 data points per case, including assessments twice weekly for the first 4 weeks of treatment and then weekly up to the 16th week of treatment. Each model included two levels: a within subject level (level 1) and a between subject level (level 2). Subject specific randomly varying intercepts and slopes were modeled at level 2. Because the rate of change was non-linear for both the HRSD and BDI-II, we considered the fit using log and square root transformations along with a standard linear representation of time. For both the HRSD and BDI-II, the square root transformation was chosen as it was the best fitting model on the basis of AIC and BIC values. In HLMs, the effects of greatest interest were those involving an interaction of therapist type (i.e., student vs. clinical trial therapists) by time, which if significant would indicate a difference in the slopes of depressive symptom change. Effect size estimates were based on subject-specific slopes.

For analyses of WOR scores, we evaluated potential differences in change in CT skills using a regressor approach, in which therapist type was examined as a predictor of each post- treatment skill variable while controlling for pre-treatment levels of that same skill. Baseline HRSD and propensity score were also included as additional covariates. For analyses of dropout, we evaluated therapist type as a predictor of risk of dropout using logistic regression. Therapist type, baseline HRSD score, and propensity score were entered into the model. Finally, we evaluated therapist type as a predictor of differences in the alliance at week 2, with baseline HRSD and propensity score included as covariates.

Results

As summarized in Table 1, the baseline demographic and diagnostic information was similar across samples. Just over half of clients were female, and about a third were married or cohabiting. A majority of clients had recurrent depression.

Table 1.

Pre-treatment Demographic and Diagnostic Characteristics by Therapist Type

Characteristics Student Therapists (N = 40) Clinical Trial Therapists (N = 60)
Intake HRSD, M ± SD 23.85 ± 2.68 24.00 ± 3.37
Intake BDI-II, M ± SD 31.69 ± 7.31 30.93 ± 8.64
Intake BAI, M ± SD 18.70 ± 9.30 15.42 ± 9.93
Age, M ± SD, years 37.38 ± 12.60 40.30 ± 11.51
Gender, n (% female) 24 (60) 35 (58)
Married or cohabiting, n (%) 15 (38) 18 (30)
Caucasian, n (%); 33 (83) 47 (79)
Education, n (%)
 HS or less 5 (13) 19 (32)
 At least some college 26 (65) 31 (52)
 At least some grad/prof 9 (23) 10 (17)
Income, n (%)
 Less than $50,000 27 (68) 47 (78)
 $50,000–99,999 11 (28) 9 (15)
 $100,000 or greater 2 (5) 4 (7)
Recurrent Depression, n (%) 26 (65) 45 (75)
PTSD Diagnosis, n (%) 5 (13) 13 (22)
Any Axis II Diagnosis, n (%) 23 (58) 27 (45)
Open in a new tab

Symptom Change

HRSD and BDI-II means by time of assessment are illustrated in Figures 1 and 2, respectively. As the figures show, depressive symptom severity decreased over time among clients seeing both types of therapists. For the non-inferiority analysis, we used linear regression to calculate adjusted post-treatment symptom severity scores, accounting for pre-treatment symptom severity and propensity scores. Symptom outcomes of student therapists were non- inferior to the outcomes of clinical trial therapists on the BDI-II, but not the HRSD. On the BDI- II, with delta set to 3.0 (as described in the Analytic Strategy section above), results supported non-inferiority, t(73) = 2.25, p = .01, 95% of observed delta [−∞, 1.67]. The observed difference in post-treatment BDI-II scores as estimated in this model was 2.10 points, with the mean of post-treatment BDI-II being numerically lower among clients of student therapists.4 On the HRSD, with delta set to 2.0, results did not support non-inferiority, t(76) = 1.08, p = .14, 95% of observed delta [−∞, 2.92]. The observed difference was .31 HRSD points, with model-estimated post-treatment severity being numerically higher for clients of student therapists. See Figure 3 (panels A and B) for a graphical depiction of these tests.

Figure 1.

Figure 1.

Open in a new tab

HRSD means by therapist type, adjusted for propensity score at each time point. For the student therapist sample, HRSD interviews were only conducted at intake, week 4, and week 16. HRSD propensity scores are based on marital status, race, and income.

HRSD = Hamilton Rating Scale for Depression.

Figure 2.

Figure 2.

Open in a new tab

BDI-II means by therapist type, adjusted for propensity score at each time point. BDI-II propensity scores are based on age, marital status, race, income, and BAI score.

BDI-II = Beck Depression Inventory.

Figure 3.

Figure 3.

Open in a new tab

Non-Inferiority tests for Depressive Symptoms, CT Skills, Treatment Completion and the Alliance.

Note. For panels A, C, and E, the 95% confidence intervals not including delta shows evidence of non-inferiority. For panels B and D, results did not support non-inferiority.

Next, we examined potential differences between clients provided CT by student vs. clinical trial therapists with regard to the rate of symptom improvement on both the HRSD and BDI-II. Using separate models for each measure of depressive symptoms, we included propensity score and propensity score by time interaction as covariates. On the HRSD, the therapist type by time interaction reflecting potential differences in the slope of symptom change between student and clinical trial therapists was non-significant (d = .002, t(460) = .09, p = .93). Similarly, in the model of BDI-II scores, the therapist type by time interaction was also not significant (d = .15, t(1235) = .05, p = .96). Thus, for both the HRSD and the BDI-II, we failed to find significant differences in the slope of symptom change between student and clinical trial therapists.

CT Skills

We first evaluated non-inferiority of WOR Quality scores of clients seen by student therapists as compared with clients seen by clinical trial therapists, adjusting for baseline HRSD, baseline WOR quality score, and propensity score. On WOR Quality, with the delta of .28, clients of the student therapists (adjusted M = 4.72, SD = .98) were non-inferior to clients of expert therapists (adjusted M= 4.21, SD = .91, t(67) = 3.32, p = .0007, 95% of observed delta [0.09, ∞]. This is shown graphically in Figure 3, panel C.

We then used conventional superiority analyses to test potential differences in the change in CT skills between clients seen by student vs. clinical trial therapists. Using regression, we entered WOR Quality score at post-treatment as a dependent variable and evaluated differences by therapist type with the following covariates included: intake WOR Quality, baseline HRSD, and propensity score. Unexpectedly, we found greater improvements in CT skills among clients seen by the student therapists as compared with those seen by clinical trial therapists, d = .54, F(1,64)= 4.71, p = .03.

Dropout

The rates of dropout were: 10 of 60 clients (17%) treated by clinical trial therapists, and 12 of 40 clients (30%) treated by student therapists. Non-inferiority of treatment completion among clients of student therapists as compared with clients of clinical trial therapists was not supported with delta set to 10%, Z = −.39, p = .65, 95% of observed delta [−.30, ∞].5 Using logistic regression, we also examined therapist type as a predictor of risk of dropout with baseline HRSD score and propensity score included as covariates. In this model, therapist type was not a significant predictor, with clients of student therapists not at differential risk of dropout as compared with clients of clinical trial therapists (OR = .43, CI = .16–1.18, χ2 = 2.70, p =.10).

Alliance

We compared client-rated therapeutic alliance scores during the second week of treatment from clients seen by student therapists and clinical trial therapists. Baseline HRSD scores and propensity score were included as covariates. For our non-inferiority test, we set delta to 3.0 on the WAI. The test indicated that clients of the student therapists (adjusted M = 73.53, SD = 8.05) were non-inferior to clients of the clinical trial therapists (adjusted M = 67.55, SD = 9.24), t(86) = 4.50, p < .0001, 95% of observed delta [2.67, ∞]. This is depicted graphically in Figure 3, panel D.

We then used conventional superiority analyses to compare WAI scores among clients of student therapists versus clinical trial therapists, covarying for baseline depression and propensity score. Unexpectedly, clients of the student therapists were found to report more positive therapeutic alliances than clients of the expert therapists, d = .69, F(1,84) = 8.96, p = .004.

To place our primary findings in context, we also examined the relationships among change in depressive symptoms, acquisition of CT skills, and the early assessment of the therapeutic alliance across clients of clinical trial and student therapists. As shown in Table 2, both alliance and skill acquisition were moderately correlated with therapeutic gains on the BDI. Reduction of symptoms on the HRSD was moderately correlated with skill acquisition, but was not significantly related to the alliance. Change in depressive symptoms as indexed by the BDI-II and HRSD was strongly related.

Table 2.

Correlations of Alliance, Skill Acquisition, and Regressed Change in Depressive Symptoms

1 2 3 4
1. CT Skills Post-treatment --
(adjusted for pre-treatment scores)
2. Alliance .33** --
64
3. BDI-II Post-treatment -.56*** -.26* --
(adjusted for pre-treatment scores) 65 70
4. HRSD Post-treatment -.48*** -.19 .80*** --
(adjusted for pre-treatment scores) 67 73 75
Open in a new tab

Note.

*

p < .05

**

p < .01

***

p < .001.

CT skills post-treatment and post-treatment depressive symptoms were estimated using residuals of a model in which post-treatment scores were predicted from pre-treatment scores. Alliance reflects scores at week 2.

BDI-II = Beck Depression Inventory-II; HRSD: Hamilton Rating Scale for Depression. Sample size for each analysis is listed below the correlation coefficient.

Discussion

This study provides initial evidence that CT conducted by student therapists can achieve outcomes largely non-inferior to those achieved by therapists in a clinical trial. We examined self-reported and interviewer-evaluated depressive symptoms, acquisition of CT skills, dropout, and the therapeutic alliance. Regarding symptom change, the non-inferiority analysis of the BDI-II supported a lack of difference between clients treated by student therapists and those treated by clinical trial therapists. However, this was not the case for the HRSD. Though the observed difference on the HRSD was small, the upper limit of the 95% confidence interval (2.92) exceeded the pre-determined margin of 2 points. Superiority analyses did not reveal a significant difference by therapist type.

Results from several previous studies have suggested that clients treated with CT in naturalistic settings could achieve comparable symptom outcomes as those observed among clients treated with CT in clinical trials (e.g., Merrill, Tolbert, & Wade, 2003; Lutz et al., 2016; Peeters et al., 2013; Persons, Bostrom, & Bertagnolli, 1999; Westbrook & Kirk, 2005). However, in the two studies that examined how outcomes may have differed by initial depressive symptom severity, the authors found that clinical trial outcomes were superior to those of naturalistic therapy among the more severely depressed clients (Forand, Evans, Haglin, & Fishman, 2011; Gibbons, Fournier, Wiltsey Stirman, DeRubeis, Crits-Christoph, & Beck, 2010). Had we relied solely on superiority analyses as the authors of these studies did, we would have concluded that, even among our samples of moderately to severely depressed clients, we failed to find evidence of a difference between student and clinical trial therapists. We regard our results as particularly informative as they were obtained in the context of the most methodologically similar comparison of student and expert clinical trial therapists providing CT of depression in the literature. Nonetheless, given our primary emphasis on tests of non-inferiority, we can only conclude that we obtained evidence of non-inferiority in analyses of the BDI-II. While there are several aspects of these measures that might account for the difference in findings across the BDI-II and HRSD, we think differences in the symptoms assessed, the frequency of assessments, and the use of distinct clinical evaluator groups across samples are particularly important to note. A recent meta-analysis highlighted differences in self-report vs. clinician-rated measures of depressive symptoms, though their results suggested that clinician-rated measures tend to be more sensitive to change (Cuijpers, Li, Hofmann, & Andersson, 2010). Unlike the BDI, the HRSD includes a number of symptoms not included in MDD diagnostic criteria (e.g., agitation, psychic and somatic anxiety, and hypochondriasis) and more items assessing sleep disruption. While the BDI-II and HRSD have the advantage of being widely used, use of measures that do not have these substantive differences would aid researchers interpreting differences in findings across clinician and self-report measures in future research.

Although we cannot be sure what accounts for the degree of comparability in symptom change we found across student and clinical trial therapists, the following three factors deserve careful consideration: (1) our efforts to provide careful training and supervision for student therapists; (2) the use of similar inclusion and exclusion criteria; and (3) the use of fixed duration treatment among student therapists to match that provided by clinical trial therapists. Our training and supervision efforts exceeded those utilized in the previous studies we have cited. While we suspect these efforts may have been important to achieving positive outcomes, the labor required was, on our view, modest enough to be practical in other contexts. The current study also emphasized maintaining methodological commonalities across student therapists and clinical trial therapists. The inclusion and exclusion criteria at the training clinic closely paralleled those in the comparison clinical trial, so the main differentiating variable was likely the expertise of the therapists. Further, we calculated propensity scores to minimize the impact of lack of randomization to therapist type. At the least, relative to previous studies, we can be more confident that the samples being compared had been obtained with similar exclusion and inclusion criteria, provided with the same duration of treatment, and use of the same treatment manual.

Beyond symptom change, we also evaluated three important variables: clients’ mastery of CT skills, client dropout, and the therapeutic alliance. One of our more robust findings was that clients treated by student therapists exhibited an even higher level of CT skill acquisition than clients treated by expert therapists. This provides initial evidence that student therapists can engage a primary target of this therapy and facilitate clients’ development of CT skills. Our work also raises the prospect of evaluating therapist training efforts through monitoring skill acquisition among clients that therapists treat. We hope the estimates obtained in our sample will serve as a useful benchmark for subsequent studies. Clients of student therapists also endorsed higher levels of therapeutic alliance than clients of clinical trial therapists. We encourage future work evaluating these variables in other samples. Interpreting our findings regarding dropout is more challenging. The non-inferiority and superiority analyses were both not statistically significant. Thus, we cannot draw any strong conclusions about the rate of dropout among clients of student therapists vs. clinical trial therapists. Despite being non- significant, we regard the magnitude of the difference in dropout rates (17 % vs. 30%) as potentially important and in need of additional research. The direction of the difference is also consistent with a broader meta-analytic finding suggesting student therapists may have higher rates of dropout (Swift & Greenberg, 2012). These dropout rates occurred despite substantial efforts made by clinical trial and student therapists to re-engage clients in treatment.

Although we have emphasized the difference in whether clients were treated by clinical trial or student therapists, there were some other differences between the clinical trial and student therapist contexts that are important to consider. To be eligible to participate in the clinical trial, participants had to consent to be randomized any of three treatment conditions (pharmacotherapy, placebo, or CT). There was no randomization in the student therapist sample. In addition, the training clinic site allowed clients on stable doses of psychotropic medication to participate, whereas the clinical trial participants had to be medication-free. We cannot rule out the possibility that such differences may have played some role in the outcomes observed. In addition, it is important to note that there could still be important client differences not considered in our analysis. There may be some clients for whom clinical trial therapists, who may be seen as having more gravitas, would have an advantage in keeping clients engaged in treatment. This is an important question for future research.

Fortunately, many procedures were similar between the clinical trial and training sites, suggesting the comparison of clinical trial vs. student therapists test is likely to be informative. All participants were seen at university-based clinics at major research universities. All participants consented to participate in research, including consent to additional assessments and allowing recordings of therapy sessions for research purposes. Across the clinical trial and training clinic, therapists met to discuss cases regularly. Treatment manual, session frequency, and treatment duration were all based on the same protocol. All participants participated in a lengthy intake process that included structured clinical interviews and a number of self-report questionnaires. Participants in both the clinical trial and student therapist samples were provided similar small honorariums for their participation in research tasks.

We note a few additional limitations of this study. First, and most critically, because clients were not randomized to clinical trial vs. student therapists, we cannot rule out the possibility of unmeasured differences in the clients at the sites that may have contributed to our results, such as possible regional differences. A randomized clinical trial designed to test the question of therapist expertise would provide a more decisive test of this difference. In the meantime, the use of similar inclusion and exclusion criteria combined with use of propensity scores for key diagnostic and demographic variables address several potential sources of differences. There are additional limitations worth consideration. Our sample sizes were not as large as would be desirable and the unequal number of participants in the clinical trial and student therapist samples meant that power was further compromised to some degree. In addition, therapists in the training clinic were all students in the same doctoral program in clinical psychology. Whether our results would generalize to a broader sample of student therapists from different programs requires further study. Finally, we have focused on the difference between student therapists and clinical trial therapists, but defining therapist “expertise” has been a contentious issue. Eells, Lombart, Kendjelec, Turner, and Lucas (2005) identified several constructs that are similar or related to expertise, including “competence, experience, proficiency, effectiveness, or mastery” (p. 580). It is possible that different aspects of expertise are differentially important to therapeutic outcomes. Nonetheless, we conducted additional analyses comparing outcomes of the student therapists vs. only the four clinical trial therapists considered to be experts specifically in CT of depression. The results were largely similar to the full sample and are included in Online Supplement B.

In sum, we found that symptom outcomes of clients treated by student therapists did not appreciably differ from outcomes of clients treated by clinical trial therapists when measured via self-report, though this comparability was not as clear when considering clinician-evaluated symptom severity. Additionally, clients treated by student therapists were not only non-inferior with regard to CT skill acquisition among clients of clinical trial therapists, but in conventional analyses clients of student therapists surprisingly achieved greater gains in CT skills. The same pattern was seen for the therapeutic alliance. While the rate of client dropout was numerically higher among student therapists than clinical trial therapists, we failed to find evidence of non- inferiority or a conventional group difference in dropout. For this reason, we encourage future work examining dropout among student therapists. Taken together, results of this study hold promise for the potential that CT of depression may achieve good outcomes as it is more widely disseminated and increasingly provided by new therapists.

Supplementary Material

1

  • We compared clients with major depression treated with cognitive therapy delivered by clinical trial vs. student therapists

  • Clients treated by student therapists had largely comparable outcomes on symptom change, alliance, and cognitive therapy skills.

  • There was some evidence that student therapists had higher client dropout than clinical trial therapists.

Acknowledgments

Funding Sources

This work was supported by the National Center for Research Resources [grant number TL1RR025753]; the National Institute of Mental Health [grant numbers MH50129, MH55875, and MH01697]; Advanced Fellowship Program in Mental Illness Research and Treatment, the Office of Academic Affiliations, Department of Veterans Affairs; and a Career Development Award, Veterans Affairs [grant number RX002952]. The funding sources had no involvement in study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. Declarations of interest: none

1Abbreviations:

BAI

Beck Anxiety Inventory

BDI-II

Beck Depression Inventory-II

CT

cognitive therapy

HLM

hierarchical linear modeling

HRSD

Hamilton Rating Scale for Depression

MDD

major depressive disorder

SCID

Structured Clinical Interview for DSM-IV

WAI

Working Alliance Inventory

WOR

Ways of Responding

Footnotes

2

The cited versions of the SCID were administered at the student therapist site. DeRubeis et al. (2005) administered slightly different versions, as the Text Revision of the DSM-IV was released between the two studies. However, diagnostic criteria remained the same between the two versions of the interviews, including criteria for major depressive disorder (First & Pincus, 2002).

3

For ease of interpretation, a d of .3 was calculated for each relevant measure and rounded to the nearest whole number in the units of each scale (HRSD, BDI-II, WAI). For the WOR score, this value was rounded to the nearest hundredth decimal place.

4

Note that the observed mean difference between clients of student therapists and clinical trial therapists was −2.10. The upper bound of the one-sided 95% confidence interval of this value was 1.67. This value is less than the delta of 3.0; thus, the results of this analysis support non-inferiority of clients of student therapists as compared with clients of clinical trial therapists.

5

For ease of interpretation, results of the non-inferiority analysis without covariates are presented for dropout. To include baseline HRSD and propensity score as covariates, we obtained residuals from a logistic regression model with these variables as predictors of dropout. Analysis of these residuals also failed to support non-inferiority of dropout rates for clients of student therapists.

CRediT Author Statement

Lizabeth Goldstein: conceptualization; data curation; formal analysis; investigation; methodology; project administration; software; visualization; writing – original draft; writing – review and editing

Abby Adler Mandel: data curation; formal analysis; funding acquisition; investigation; project administration; resources; writing – original draft; writing – review and editing

Robert DeRubeis: data curation; funding acquisition; investigation; project administration; resources; supervision; writing – review and editing

Daniel Strunk: conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; visualization; writing – original draft; writing – review and editing

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Academy of Cognitive Therapy (n.d.) Certification process. Retrieved from http://www.academyofct.org/certification-process/
  2. Adler AD, Strunk DR, & Fazio RH (2015). What changes in cognitive therapy for depression? An examination of cognitive therapy skills and maladaptive beliefs. Behavior Therapy, 46, 96–105. doi: 10.1016/j.beth.2014.09.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Association of Psychology Postdoctoral and Internship Centers (2020, January 23). APPIC Match Statistics. Retrieved from https://www.appic.org/Internships/Match/Match-Statistics
  4. Barber JP, & DeRubeis RJ (1992). The ways of responding: A scale to assess compensatory skills taught in cognitive therapy. Behavioral Assessment, 14, 93–115. [Google Scholar]
  5. Barber JP, & DeRubeis RJ (2001). Change in compensatory skills in cognitive therapy for depression. Journal of Psychotherapy Practice & Research, 10, 8–13. [PMC free article] [PubMed] [Google Scholar]
  6. Beck AT, Epstein N, Brown G, & Steer RA (1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893–897. doi: 10.1037/0022-006X.56.6.893 [DOI] [PubMed] [Google Scholar]
  7. Beck AT, Rush AJ, Shaw BF, & Emery G (1979). Cognitive therapy of depression. New York: Guilford Press. [Google Scholar]
  8. Beck AT, Steer RA, & Brown BK (1996). Beck Depression Inventory manual (2nd ed.). San Antonio, TX: Psychological Corporation. [Google Scholar]
  9. Busseri MA, & Tyler JD (2003). Interchangeability of the working alliance inventory and working alliance inventory, short form. Psychological Assessment, 15, 193. doi: 10.1037/1040-3590.15.2.193 [DOI] [PubMed] [Google Scholar]
  10. Cuijpers P, Li J, Hofmann SG, & Andersson G (2010). Self-reported versus clinician-rated symptoms of depression as outcome measures in psychotherapy research on depression: A meta-analysis. Clinical Psychology Review, 30, 768–778. [DOI] [PubMed] [Google Scholar]
  11. DeRubeis RJ, Hollon SD, Amsterdam JD, Shelton RC, Young PR, Salomon RM, . . . Gallop R (2005). Cognitive therapy vs medications in the treatment of moderate to severe depression. Archives of General Psychiatry, 62, 409–416. doi: 10.1001/archpsyc.62.4.409 [DOI] [PubMed] [Google Scholar]
  12. Driessen E, Cuijpers P, Hollon SD, & Dekker JJ (2010). Does pretreatment severity moderate the efficacy of psychological treatment of adult outpatient depression? A meta- analysis. Journal of Consulting and Clinical Psychology, 78, 668–680. doi: 10.1037/a0020570 [DOI] [PubMed] [Google Scholar]
  13. Eells TD, Lombart KG, Kendjelic EM, Turner LC, & Lucas CP (2005) The quality of psychotherapy case formulations: A comparison of expert, experienced, and novice cognitive-behavioral and psychodynamic therapists. Journal of Consulting and Clinical Psychology, 73, 579–589. doi: 10.1037/0022-006X.73.4.579 [DOI] [PubMed] [Google Scholar]
  14. Fernandez E, Salem D, Swift JK, & Ramtahal N (2015). Meta-analysis of dropout from cognitive behavioral therapy: Magnitude, timing, and moderators. Journal of Consulting and Clinical Psychology. Advance online publication. doi: 10.1037/ccp0000044 [DOI] [PubMed] [Google Scholar]
  15. First MB, Gibbon M, Spitzer RL, Williams JBW, & Benjamin LS (1997). Structured Clinical Interview for DSM-IV Axis II Personality Disorders. (SCID-II). Washington, D.C.: American Psychiatric Press, Inc. [Google Scholar]
  16. First MB, & Pincus HA (2002). The DSM-IV Text Revision: Rationale and potential impact on clinical practice. Psychiatric Services, 53, 288–292. doi: 10.1176/appi.ps.53.3.288 [DOI] [PubMed] [Google Scholar]
  17. First MB, Spitzer RL, Miriam G, & Williams JBW (2002). Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition. (SCID- I/P). New York: Biometrics Research, New York State Psychiatric Institute. [Google Scholar]
  18. Forand NR, Evans S, Haglin D, & Fishman B (2011). Cognitive behavioral therapy in practice: Treatment delivered by trainees at an outpatient clinic is clinically effective. Behavior Therapy, 42, 612–623. doi: 10.1016/j.beth.2011.02.001 [DOI] [PubMed] [Google Scholar]
  19. Fournier JC, DeRubeis RJ, Hollon SD, Dimidjian S, Amsterdam JD, Shelton RC, & Fawcett J (2010). Antidepressant drug effects and depression severity: a patient-level meta-analysis. JAMA, 303, 47–53. doi: 10.1001/jama.2009.1943 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gibbons CJ, Fournier JC, Wiltsey Stirman S, DeRubeis RJ, Crits-Christoph P, & Beck AT (2010). The clinical effectiveness of cognitive therapy for depression in an outpatient clinic. Journal of Affective Disorders, 125, 169–176. doi: 10.1016/j.jad.2009.12.030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Gloaguen V, Cottraux J, Cucherat M, & Blackburn I (1998). A meta-analysis of the effects of cognitive therapy in depressed patients. Journal of Affective Disorders, 49, 59–72. doi: 10.1016/S0165-0327(97)00199-7 [DOI] [PubMed] [Google Scholar]
  22. Hamilton M (1960). A rating scale for depression. Journal of Neurology, Neurosurgery and Psychiatry, 23, 56–62. doi: 10.1136/jnnp.23.1.56 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hollon SD, Thase ME, & Markowitz JC (2002). Treatment and prevention of depression. Psychological Science in the Public Interest, 3, 39–77. [DOI] [PubMed] [Google Scholar]
  24. Horvath AO, & Greenberg S (1989). Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36, 223–233. [Google Scholar]
  25. Johnsen TJ, & Friborg O (2015). The effects of cognitive behavioral therapy as an anti- depressive treatment is falling: A meta-analysis. Psychological Bulletin, 141, 747–768. doi: 10.1037/bul0000015 [DOI] [PubMed] [Google Scholar]
  26. Kirsch I, Deacon BJ, Huedo-Medina TB, Scoboria A, Moore TJ, & Johnson BT (2008). Initial severity and antidepressant benefits: A meta-analysis of data submitted to the Food and Drug Administration. PLoS Medicine, 5, e45. doi: 10.1371/journal.pmed.0050045. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lutz W, Schiefele AK, Wucherpfennig F, Rubel J, & Stulz N (2016). Clinical effectiveness of cognitive behavioral therapy for depression in routine care: A propensity score based comparison between randomized controlled trials and clinical practice. Journal of Affective Disorders, 189, 150–158. doi: 10.1016/j.jad.2015.08.072 [DOI] [PubMed] [Google Scholar]
  28. Martin DJ, Garske JP, & Davis MK (2000). Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical Psychology, 68, 438–450. doi: 10.1037/0022-006X.68.3.438 [DOI] [PubMed] [Google Scholar]
  29. Merrill KA, Tolbert VE, & Wade WA (2003). Effectiveness of cognitive therapy for depression in a community mental health center: A benchmarking study. Journal of Consulting and Clinical Psychology, 71, 404–409. doi: 10.1037/0022-006X.71.2.404 [DOI] [PubMed] [Google Scholar]
  30. Peeters F, Huibers M, Roelofs J, van Breukelen G, Hollon SD, Markowitz, . . . Arntz A (2013). The clinical effectiveness of evidence-based interventions for depression: A pragmatic trial in routine practice. Journal of Affective Disorders, 145, 349–355. doi: 10.1016/j.jad.2012.08.022 [DOI] [PubMed] [Google Scholar]
  31. Persons JB, Bostrom A, & Bertagnolli A (1999). Results of randomized controlled trials of cognitive therapy for depression generalize to private practice. Cognitive Therapy and Research, 23, 535–548. doi: 10.1023/A:1018724505659 [DOI] [Google Scholar]
  32. Pfeifer BJ, & Strunk DR (2015). A primer on psychotherapy process research: Considering cognitive therapy of depression. Australian Psychologist, 50, 411–415. doi: 10.1111/ap.12147 [DOI] [Google Scholar]
  33. Saxon D, & Barkham M (2012). Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk. Journal of Consulting and Clinical Psychology, 80, 535–546. doi: 10.1037/a0028898 [DOI] [PubMed] [Google Scholar]
  34. Strunk DR, Brotman MA, DeRubeis RJ, & Hollon SD (2010). Therapist competence in cognitive therapy for depression: Predicting subsequent symptom change. Journal of Consulting and Clinical Psychology, 78, 429–437. doi: 10.1037/a0019631 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Strunk DR, Cooper AA, Ryan ET, DeRubeis RJ, & Hollon SD (2012). The process of change in cognitive therapy for depression when combined with antidepressant medication: Predictors of early inter-session symptom gains. Journal of Consulting and Clinical Psychology, 80, 730–738. doi: 10.1037/a0029281 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Strunk DR, DeRubeis RJ, Chiu AW, & Alvarez J (2007). Patients’ competence in and performance of cognitive therapy skills: Relation to the reduction of relapse risk following treatment for depression. Journal of Consulting and Clinical Psychology, 75, 523–530. doi: 10.1037/0022-006X.75.4.523 [DOI] [PubMed] [Google Scholar]
  37. Strunk DR, Brotman MA, DeRubeis RJ, & Hollon SD (2010). Therapist competence in cognitive therapy for depression: Predicting subsequent symptom change. Journal of Consulting and Clinical Psychology, 78, 429–437. doi: 10.1037/a0019631 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Swift JK, & Greenberg RP (2012). Premature discontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 80, 547–559. doi: 10.1037/a0028226 [DOI] [PubMed] [Google Scholar]
  39. Tracey TJ, & Kokotovic AM (1989). Factor structure of the Working Alliance inventory. Psychological Assessment, 1, 207–210. [Google Scholar]
  40. Webb CA, DeRubeis RJ, Amsterdam JD, Shelton RC, Hollon SD, & Dimidjian S (2011). Two aspects of the therapeutic alliance: Differential relations with depressive symptom change. Journal of Consulting and Clinical Psychology, 79, 279–283. doi: 10.1037/a0023252 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Westbrook D, & Kirk J (2005). The clinical effectiveness of cognitive behavior therapy: outcome for a large sample of adults treated in routine practice. Behaviour Research and Therapy, 43, 1243–1261. doi: 10.1016/j.brat.2004.09.006 [DOI] [PubMed] [Google Scholar]
  42. Williams JB (1988). A structured interview guide for the Hamilton Depression Rating Scale. Archives of General Psychiatry, 45, 742–747. doi: 10.1001/archpsyc.1988.0180032005800 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1582894-supplement-1.docx (23.1KB, docx)

RESOURCES