Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 May 1.
Published in final edited form as: Am J Psychiatry. 2016 Feb 12;173(5):481–490. doi: 10.1176/appi.ajp.2015.15040492

Divergent Outcomes in Cognitive Behavioral Therapy and Pharmacotherapy for Adult Depression

Jeffrey R Vittengl 1,*, Robin B Jarrett 2, Erica Weitz 3, Steven D Hollon 4, Jos Twisk 5, Ioana Cristea 6, Daniel David 7, Robert J DeRubeis 8, Sona Dimidjian 9, Boadie W Dunlop 10, Mahbobeh Faramarzi 11, Ulrich Hegerl 12, Sidney H Kennedy 13, Farzan Kheirkhah 14, Roland Mergl 15, Jeanne Miranda 16, David C Mohr 17, A John Rush 18, Zindel V Segal 19, Juned Siddique 20, Anne D Simons 21, Pim Cuijpers 22
PMCID: PMC4934129  NIHMSID: NIHMS797067  PMID: 26869246

Abstract

Objective

Although the average depressed patient benefits moderately from cognitive-behavioral therapy (CBT) or pharmacotherapy, some experience divergent outcomes. We tested frequencies, predictors, and moderators of negative (deterioration, extreme non-response) and unusually-positive (superior improvement, superior response) symptomatic outcomes.

Method

Sixteen randomized clinical trials comparing CBT versus pharmacotherapy for unipolar depression provided individual patients’ (N=1700) Hamilton Rating Scale of Depression (HAM-D) and/or Beck Depression Inventory (BDI) scores pre- and post-treatment. We tested demographic and clinical predictors and treatment moderators of any deterioration (increases ≥1 HAM-D or BDI points), reliable deterioration (increases ≥8 HAM-D or ≥9 BDI points), extreme non-response (post-treatment HAM-D ≥21 or BDI ≥31), superior improvement (HAM-D or BDI decreases ≥95%), and superior response (post-treatment HAM-D or BDI =0) using multilevel models.

Results

About 5–7% of patients showed any deterioration, 1% reliable deterioration, 4–5% extreme non-response, 6–10% superior improvement, and 4–5% superior response on the HAM-D or BDI. Superior improvement on the HAM-D (OR=1.67) only and attrition (OR=1.67) were more frequent in pharmacotherapy versus CBT. Patients with deterioration and superior response had lower, whereas patients with extreme non-response and superior improvement had higher, symptoms pre-treatment.

Conclusions

Deterioration and extreme non-response (a) occur infrequently in randomized clinical trials comparing CBT versus pharmacotherapy for depression, and (b) mirror superior improvement and superior response in distributions of symptom changes and end-states, respectively. Pre-treatment symptom levels help forecast negative and unusually-positive outcomes but do not guide selection of CBT versus pharmacotherapy. Pharmacotherapy may produce clinician-rated superior improvement and attrition more frequently than does CBT.

Keywords: depression, cognitive therapy, cognitive behavioral therapy, pharmacotherapy, deterioration, extreme non-response, superior improvement, superior response

Introduction

Cognitive behavioral therapy (CBT) and pharmacotherapy are, on average, equally and moderately efficacious acute-phase treatments for unipolar depression [1]. For example, about 50–60% of patients receiving CBT or pharmacotherapy for depression respond versus 40% receiving pill placebo [2–3], and the average CBT or pharmacotherapy patient has post-treatment symptom levels roughly 0.3 SD below patients receiving pill placebo [4–5]. However, these favorable averages may obscure some patients’ negative outcomes [6]. Moreover, beyond conventionally-defined response (≥50% symptom reduction) and remission (post-treatment Hamilton Rating Scale for Depression (HAM-D) [7] scores ≤7), only patients with unusually-positive outcomes have no residual symptoms with attendant dysfunction [8].

Here we focused on patient-level treatment outcomes diverging from the average. We analyzed two specific negative outcomes: Deterioration (symptom severity increases from pre- to post-treatment) [9] and extreme non-response (severe depressive symptoms post-treatment) [10], although many others are possible [11]. In complement, we defined two unusually-positive outcomes: Superior improvement (≥95% symptom reduction from pre- to post-treatment) and superior response (no depressive symptoms post-treatment). We estimated the frequency of these outcomes and tested antecedents in a large patient-level database (N=1700) pooled from 16 randomized clinical trials comparing CBT to pharmacotherapy. Such analyses may clarify which patients are more likely to have negative or unusually-positive outcomes, help match treatments to patients, and via matching increase treatments’ overall benefits [12].

The frequency and causes of deterioration, extreme non-response, superior improvement, and superior response in CBT and pharmacotherapy for depression are unclear. Perhaps 5–10% of psychotherapy patients show reliable deterioration (increases exceeding the symptom measure’s reliable change threshold), with evidence of higher risk among children than adults, in routine practice compared to randomized clinical trials, and in “psychotherapy” considered more broadly than CBT specifically [9,13]. Reliable deterioration may occur with similar frequency in pharmacotherapy for depression [14]. The concept of extreme non-response was introduced in CBT research [10] but observed rates have varied widely from 6–22% of CBT patients in two samples [10,15]. Finally, based on distributions of symptom scores, superior response and superior improvement possibly occur in <10% of treated depressed patients [8,16]. The current large-sample analyses were designed to clarify these outcome frequencies in pharmacotherapy versus CBT for depression.

Patient characteristics may predict negative and unusually-positive outcomes in CBT or pharmacotherapy for depression. Pre-treatment markers of greater pathology (e.g., comorbidity, suicidality, unemployment, prior hospitalizations) have predicted less improvement in CBT [17–18] and pharmacotherapy [19–20]. In addition, higher pre-treatment depressive symptoms, plus poorer pre-treatment functioning and therapeutic alliance, have predicted extreme non-response in CBT [10,15]. Broadly, pre-treatment symptom levels may relate directly to post-treatment symptoms levels but inversely to the direction of symptom change [21]. Thus, greater pre-treatment severity may predict higher probabilities of extreme non-response and superior improvement but lower probabilities of superior response and deterioration. In contrast, demographic variables have been inconsistent predictors of CBT or pharmacotherapy outcomes [18,22], perhaps because detecting modest effects without large individual-patient samples is difficult.

Whether treatments or illness trajectories cause particular outcomes is difficult to determine [6]. Unipolar depression is often episodic, and mood reactivity within episodes is common [23]. Thus, deterioration and extreme non-response could be adverse effects of treatment, unrelated to treatment, or even benefits of treatment (i.e., some patients would have been worse off untreated). Similarly, superior improvement and superior response may sometimes reflect “spontaneous” improvements due to biochemical, learning, or life events unrelated to treatment.

Pharmacotherapy and CBT likely share some active ingredients, including patients’ and clinicians’ expectancies of benefit and clinicians’ regular supportive interactions with patients. In addition, CBT may reduce negative emotion via top-down processes including patients’ skill acquisition and use [24], whereas pharmacotherapy may act directly on neurochemical pathways to reduce negative emotion in bottom-up processes [25]. Different hypothesized mechanisms highlight the possibility that CBT and pharmacotherapy may produce divergent outcomes for different patient subpopulations.

Data from randomized clinical trials support identification of potential causes of patient outcomes [6]. Different frequencies of deterioration, extreme non-response, superior improvement, and superior response between patients randomized to CBT or pharmacotherapy, for example, would suggest that these events may result from components of particular treatment protocols. Moreover, randomized clinical trials allow testing moderators of outcomes. Moderators are interactions between treatments (e.g., CBT versus pharmacotherapy) and patient characteristics that provide information about individuals’ differential outcomes in one treatment versus another [18].

The goals of this report were to (1) estimate the proportions of patients with negative (deterioration, extreme non-response) and unusually-positive (superior improvement, superior response) outcomes in CBT or pharmacotherapy for depression; (2) test predictors of these outcomes, including the type of treatment (CBT versus pharmacotherapy) and patient demographic and clinical characteristics; (3) test whether patient characteristics moderate treatment effects (CBT versus pharmacotherapy) on these outcomes; and (4) contrast pre-treatment characteristics of patients with opposing outcomes (deterioration versus superior improvement; extreme non-response versus superior response).

Method

Identification and Selection of Studies

We analyzed individual patient data pooled from randomized clinical trials comparing CBT to pharmacotherapy for adults (≥18 years) with unipolar depression, defined by major depressive disorder, dysthymic disorder, or by elevated depressive symptoms on standard measures. Psychological interventions with cognitive restructuring as a core component defined CBT for study selection [1].

We accessed an existing database of trials to identify potential studies for this meta-analysis (www.evidencebasedpsychotherapies.org). The database was updated through January 2014 for the current search, and 24 studies met inclusion criteria. We contacted study authors by email to invite their participation. We requested patient-level data including pre- and post-treatment depressive symptom scores, and demographic and clinical characteristics. Authors provided patient-level data for 16 trials described in Online Supplement 1.

Participants

The analyzed sample included 1700 adults from 16 randomized trials comparing CBT (n=794) with pharmacotherapy (n=906) for depression. As shown in Table 1, most patients were women, of majority ethnicity, educated beyond high school, and of middle age. Most patients (96.8%) had major depressive disorder, 1.4% had dysthymic but not major depressive disorder, and 1.8% had no depression diagnoses but elevated depressive symptoms pre-treatment (HAM-D M=16.19, SD=4.78; BDI M=21.38, SD=9.36).

Table 1.

Descriptive Statistics for Individual Patient Meta-analysis Variables

Variable K studies N patients Mean/% SD
Demographics
 Female 16 1700 69.4%
 Age (years) 14 1626 37.38  11.64
 Minority ethnicity 11 1390 14.0%
 Education (>12 years) 12 1439 65.5%
 Employed full time 11 1289 52.0%
Clinical Characteristics
 Major depressive disorder 16 1700 96.8%
 Dysthymic disorder   7   913 18.3%
 Double depression   7   913 15.7%
 Comorbid diagnosis 11 1417 32.1%
Depressive symptom severity
 HAM-D pre-treatment (with missing values) 14 1466 19.18   4.58
 HAM-D pre-treatment (with imputed values) 14 1471 19.18   4.58
 HAM-D post-treatment (with missing values) 14 1037   8.60   6.49
 HAM-D post-treatment (with imputed values) 14 1471   9.17   6.32
 BDI pre-treatment (with missing values) 14 1400 30.44   9.79
 BDI pre-treatment (with imputed values) 14 1491 30.34   9.81
 BDI post-treatment (with missing values) 14 1074 11.16   9.84
 BDI post-treatment (with imputed values) 14 1491 12.04   9.55
Open in a new tab

Note. K and N refer to the number of studies and patients, respectively, with data available on the variable, whereas % refers to the proportion of patients with the characteristic. Double depression = diagnoses of both major depressive and dysthymic disorders. Comorbid diagnoses are those other than depressive disorders. HAM-D = 17-item Hamilton Rating Scale for Depression. BDI = Beck Depression Inventory-II.

Measures

Depressive Symptom Severity

Fourteen studies used the clinician-report HAM-D, 14 studies used the patient-report Beck Depression Inventory (BDI), and 12 used both measures. The 17-item HAM-D yields scores from 0–52. The 21-item BDI yields scores from 0–63. We converted first-edition BDI [26] scores to the slightly higher second-edition metric [27] before analysis. Higher HAM-D and BDI scores mark greater depressive symptoms.

Operational Definitions

Deterioration

“Any deterioration” was an increase of ≥1 points on the HAM-D or BDI from pre- to post-treatment. “Reliable deterioration” was an increase meeting the p < .05, two-tailed, change thresholds of ≥8 HAM-D [28] or ≥9 BDI [29] points. The any-deterioration group contained all patients with deterioration, whether reliable or not.

Extreme Non-response

Following previous standards [10,15], extreme non-response was defined as post-treatment BDI scores ≥31. We converted this BDI threshold [30] to define extreme non-response as HAM-D scores ≥21.

Superior Improvement

Superior improvement was pre- to post-treatment decreases in HAM-D or BDI scores ≥95%, marking improvement beyond traditional response thresholds (≥50%) and contrasting with deterioration.

Superior Response

Superior response was post-treatment HAM-D or BDI =0, marking the absence of measured depressive symptoms and contrasting with extreme non-response.

Statistical Analyses

We multiply imputed missing BDI and HAM-D among studies using these measures to allow intent-to-treat analyses. We used the Markov chain Monte Carlo method in PROC MI to generate 10 complete data sets, computed outcomes for each patient in the complete datasets, conducted multilevel analyses in each dataset with PROC GLIMMIX, and pooled the results via PROC MIANALYZE in SAS 9.3 software (SAS Institute, Inc., Cary, NC). Because patient data were nested within studies, multilevel analyses controlled for the random effect of study. Logistic regression models predicted binary outcomes (e.g., presence versus absence of deterioration) from the fixed effects of treatment (CBT versus pharmacotherapy), a predictor (e.g., age, gender), and the interaction of treatment and the predictor (i.e., the moderator effect). We contrasted dichotomous and continuous pre-treatment characteristics of patients with opposing outcomes in logistic and normal linear multilevel models, respectively.

Previous Analyses of the Individual Patient Database

Whereas the current analyses focused on outcomes diverging from the mean, past studies using the individual patient database focused on averages and conventional outcomes [31–32]. Average treatment effects did not differ significantly between the 8 excluded versus 16 included studies, suggesting that the included studies are representative [32]. Among included studies, pharmacotherapy versus CBT produced statistically significantly lower average post-treatment HAM-D (but not BDI) scores, although the difference (0.88 HAM-D points) was small [32]. Conventionally-defined response (HAM-D decrease of ≥50%) and remission (post-treatment HAM-D ≤7), respectively, did not differ significantly in pharmacotherapy (63.4%; 51.0%) versus CBT (57.5%; 47.0%) [32]. Finally, gender [31] and symptom severity [32] did not moderate CBT versus pharmacotherapy’s effects on post-treatment symptom averages.

Results

Frequency and Treatment Differences in Deterioration and Non-response

Table 1 displays descriptive statistics for the variables analyzed. Controlling the random effect of study, attrition (defined as post-treatment HAM-D and BDI both missing) was more frequent in pharmacotherapy (18.5%) than in CBT (12.0%), OR=1.67, p<.01, but did not correlate significantly with baseline HAM-D or BDI (p>.25). Patients began treatment with moderate to severe depressive symptoms and ended with mild symptoms, on average, on the HAM-D and BDI.

Figure 1 shows scatterplots of HAM-D and BDI scores pre- and post-treatment with regions marking analyzed outcomes (see Online Supplement 2 for plots separating CBT and pharmacotherapy). Visual analysis of scatterplots suggested that patients with deterioration versus superior improvement, and extreme non-response versus superior response, marked opposite tails of symptom-change and post-treatment symptom-level continua, respectively.

Figure 1.

Figure 1

Open in a new tab

Depressive symptom severity before and after treatment (cognitive behavioral therapy or pharmacotherapy) for depression. Larger dots represent more patients. Plotted data include imputation of missing values. HAM-D = Hamilton Rating Scale for Depression. BDI = Beck Depression Inventory-II.

The correlation between continuous HAM-D and BDI scores was moderately high (.85), pooling pre- and post-treatment observations. Similarly, percentage agreement between the HAM-D and BDI was moderately high in identifying patients with versus without deterioration (92.3%), reliable deterioration (98.2%), extreme non-response (94.3%), superior improvement (88.8%), and superior response (92.3%). Nonetheless, clinician (HAM-D) and patient (BDI) reports sometimes differed regarding the presence of analyzed outcomes (see Figure 2).

Figure 2.

Figure 2

Open in a new tab

Patients with analyzed outcomes by clinician- (HAM-D = Hamilton Rating Scale of Depression) and patient- (BDI = Beck Depression Inventory-II) report among 12 studies using both measures. Any deterioration and reliable deterioration, respectively, are increases of ≥1 and ≥8 points on the HAM-D, and ≥1 and ≥9 points on the BDI, from pre- to post-treatment. Extreme non-response is post-treatment HAM-D ≥21 or BDI ≥31. Superior improvement is a reduction of HAM-D or BDI scores by ≥95%. Superior response is a post-treatment HAM-D or BDI =0.

Our first goal was to estimate the frequency of outcomes (see Table 2). Overall, about 5–7% of patients showed any deterioration (pre- to post-treatment increases ≥1 points on the HAM-D or BDI), 1% showed reliable deterioration (increases of ≥8 HAM-D or ≥9 BDI points), 4–5% showed extreme non-response (post-treatment HAM-D ≥21 or BDI ≥31), 6–10% showed superior improvement (HAM-D or BDI decreases of ≥95%), and 4–5% showed superior response (post-treatment HAM-D or BDI =0). About 13% of patients had any negative outcome (deterioration or extreme non-response) and 15% had an unusually-positive outcome (superior improvement or superior response) on the HAM-D or BDI.

Table 2.

Estimated Proportions of Patients with Negative Outcomes and Unusually-Positive Outcomes

Outcome Overall Cognitive-behavioral therapy Pharmacotherapy Treatment difference
Est. % 95% CI Est. % 95% CI Est. % 95% CI p
Negative Outcomes
Any Deterioration
 HAM-D 7.1 4.7, 10.7 7.7 4.8, 12.0 6.6 4.1, 10.5 .48
 BDI 5.2 3.4, 8.0 5.4 3.3, 8.8 5.0 3.0, 8.2 .78
Reliable Deterioration
 HAM-D 0.9 0.4, 1.8 1.2 0.5, 2.6 0.6 0.2, 1.9 .32
 BDI 1.1 0.5, 2.5 0.8 0.3, 2.5 1.4 0.6, 3.4 .37
Extreme Non-response
 HAM-D 5.4 3.7, 7.7 5.3 3.4, 8.1 5.5 3.6, 8.2 .88
 BDI 4.3 2.9, 6.4 4.6 2.9, 7.3 4.1 2.5, 6.5 .62
Any Negative Outcome 13.3 9.9, 17.6 14.1 10.1, 19.2 12.7 9.1, 17.6 .54
Unusually-Positive Outcomes
Superior Improvement
 HAM-D 6.4 4.4, 9.2 4.8 2.9, 7.7 7.7 5.2, 11.3 .04
 BDI 9.8 6.4, 14.7 8.9 5.6, 13.9 10.6 6.8, 16.1 .32
Superior Response
 HAM-D 3.9 2.6, 5.9 3.2 1.8, 5.5 4.5 2.9, 7.0 .23
 BDI 5.4 3.0, 9.4 4.5 2.4, 8.3 6.2 3.4, 10.9 .17
Any Unusually-Positive Outcome 15.1 10.6, 21.0 12.8 8.5, 18.7 16.9 11.7, 23.7 .06
Open in a new tab

Note. Est. = estimated proportion of patients derived from multilevel logistic regression models that adjusted for the random effect of study. Analyses included multiple imputation of missing HAM-D (Hamilton Rating Scale for Depression) and BDI (Beck Depression Inventory-II) scores. Any deterioration and reliable deterioration, respectively, are increases of ≥1 and ≥8 points on the HAM-D, and ≥1 and ≥9 points on the BDI, from pre- to post-treatment. Extreme non-response is post-treatment HAM-D ≥21 or BDI ≥31. Superior improvement is a reduction of HAM-D or BDI scores by ≥95%. Superior response is a post-treatment HAM-D or BDI =0. Any negative outcome is deterioration and/or extreme non-response, and any unusually-positive outcome is superior improvement and/or superior response on either measure. Fourteen studies each used the HAM-D and BDI, whereas 12 studies used both measures.

Frequencies of deterioration, extreme non-response, and superior response did not differ significantly between CBT and pharmacotherapy. Treatment with pharmacotherapy versus CBT increased patients’ chance of superior improvement significantly on the HAM-D (OR=1.67) but not on the BDI.1

Prediction and Moderation of Outcomes

Our second and third goals were to test predictors and moderators of outcomes. We tested predictors (main effects) and moderators (interaction effects) in a series of multilevel logistic regression analyses (see Online Supplement 4 for regression coefficients). Patient characteristics listed in Table 1 did not significantly moderate differential treatment effects (CBT versus pharmacotherapy), but several statistically significant predictors were evident.

First, lower pre-treatment depressive symptoms predicted deterioration, but only within measures (HAM-D or BDI). Patients with lower pre-treatment HAM-D scores had a higher chance of any (OR=1.92) and reliable (OR=2.46) deterioration on the HAM-D.2 Similarly, patients with lower pre-treatment BDI scores had a higher chance of any (OR=2.48) and reliable (OR=2.19) deterioration on the BDI.

Second, higher pre-treatment symptom severity predicted extreme non-response both within and between measures. Patients with higher pre-treatment HAM-D scores had a higher chance of extreme non-response on the HAM-D (OR=1.89) and BDI (OR=1.48). Similarly, patients with higher pre-treatment BDI scores had higher chance of extreme non-response on the BDI (OR=2.25) and HAM-D (OR=1.65).

Third, younger age (OR=1.36)2 and higher pre-treatment HAM-D scores (OR=1.33) predicted superior improvement on the HAM-D. Finally, full-time employment (OR=1.98) and lower pre-treatment BDI scores (OR=1.46) predicted superior response on the BDI.

We followed-up pre-treatment symptom scales’ significant prediction of outcomes within the HAM-D by testing specific depressive symptoms captured by individual scale item scores (8 studies provided item-level HAM-D data). Lower pre-treatment depressed mood (OR=1.60), less guilt (OR=1.37), and fewer general somatic symptoms (OR=1.56) predicted any deterioration.3 Individual HAM-D items did not predict reliable deterioration significantly. Initial insomnia (OR=1.58), middle insomnia (OR=1.57), and weight loss (OR=1.70) predicted extreme non-response. Finally, gastro-somatic symptoms predicted superior improvement (OR=1.51).

Contrasts of Outcome Groups

Our fourth goal was to contrast the pre-treatment characteristics of patients reaching opposing outcomes, deterioration versus superior improvement and extreme non-response versus superior response. As shown in Table 3, patients with any deterioration versus superior improvement on the HAM-D had lower pre-treatment HAM-D scores. Similarly, patients with any deterioration versus superior improvement on the BDI had lower pre-treatment BDI scores. Finally, patients with extreme non-response versus superior response on the HAM-D or BDI had higher pre-treatment scores on both symptom measures.

Table 3.

Pre-treatment Characteristics of Patients ending Treatment in Opposing Outcome Groups

Pre-Treatment Variable Any Deterioration
versus
Superior Improvement on the HAM-D		 Any Deterioration
versus
Superior Improvement on the BDI		 Extreme Non-response
versus
Superior Response on the HAM-D		 Extreme Non-response
versus
Superior Response on the BDI
DET SI DET SI ENR SR ENR SR
Age (years) 37.4 (1.8) 35.4 (1.9) 38.2 (2.0) 37.0 (1.7) 37.3 (1.9) 36.1 (2.1) 38.3 (2.1) 37.7 (1.9)
Female 68.4% (8.2) 76.0% (7.1) 70.5% (7.2) 76.4% (5.0) 72.5% (8.1) 72.2% (8.4) 62.9% (7.7) 74.7% (5.9)
Minority ethnicity 15.0% (6.9) 9.7% (4.8) 6.2% (3.9) 11.0% (4.6) 12.5% (6.4) 8.6% (5.7) 5.8% (3.7) 13.1% (5.7)
Education (>12 years) 78.2% (8.4) 70.6% (9.2) 71.4% (10.1) 79.2% (7.3) 68.2% (10.1) 70.2% (10.1) 68.7% (10.1) 80.5% (7.7)
Employed 50.5% (9.3) 48.3% (9.1) 39.4% (11.0) 50.2% (10.2) 48.6% (9.5) 45.1% (10.5) 53.3% (11.4) 58.6% (10.8)
Double depression 15.0% (8.3) 5.0% (4.9) 12.8% (7.9) 11.8% (6.8) 12.3% (†) 0.7% (†) 5.6% (6.8) 6.4% (6.3)
Comorbid diagnosis 35.6% (14.3) 29.5% (13.5) 31.2% (14.6) 28.1% (13.1) 37.9% (14.8) 29.3% (14.1) 31.7% (14.8) 27.8% (13.5)
HAM-D 17.3 (0.7) 20.1 (0.7)* 19.2 (0.8) 19.4 (0.7) 22.0 (0.8) 18.3 (0.8)* 20.7 (0.8) 19.0 (0.8)*
BDI 28.5 (1.6) 29.7 (1.6) 23.7 (1.6) 29.6 (1.4)* 33.0 (1.9) 28.2 (1.8)* 35.7 (1.7) 26.7 (1.6)*
Open in a new tab

Note. Means/percentages (standard errors) estimated in multilevel models controlling for the random effect of study. HAM-D = Hamilton Rating Scale for Depression. BDI = Beck Depression Inventory-II. DET = any deterioration (increases) in scores. SI = superior improvement (decreases ≥95%). ENR = Extreme non-response (post-treatment HAM-D ≥21 or BDI ≥31). SR = superior response (post-treatment HAM-D or BDI =0).

†

The statistical model failed to converge because events were too rare in some cells; inferential contrast not computed; estimates from raw data.

*

Pairwise contrast p < .05, two-tailed.

Discussion

Among 1700 depressed patients from 16 randomized clinical trials comparing CBT to pharmacotherapy, we found that 13% experienced negative and 15% experienced unusually-positive symptomatic outcomes. Analyzed negative outcomes were any deterioration (5–7% had increases ≥1 HAM-D or BDI points), reliable deterioration (1% had increases ≥8 HAM-D or ≥9 BDI points), and extreme non-response (4–5% had post-treatment HAM-D ≥21 or BDI ≥31). Negative outcomes may be more common in practice settings outside of research [12,14] because frequent assessment of patients, feedback to clinicians about patients’ progress, and monitoring of treatment fidelity (e.g., clinicians’ competence and protocol adherence) in many randomized clinical trials parallel interventions shown to reduce deterioration [13]. Patients’ unusually-positive outcomes included superior improvement (6–10% had HAM-D or BDI decreases ≥95%) and superior response (4–5% had post-treatment HAM-D or BDI =0). Because few patients receiving CBT or pharmacotherapy reached absent (superior response) or minimal (superior improvement) residual depressive symptoms, our results highlight the potential value of continuing, sequencing, or augmenting treatments [19,33].

Treatment with pharmacotherapy versus CBT increased patients’ odds of superior improvement from the clinician (HAM-D; 7.7% versus 4.8%) but not patient (BDI) perspective. Pharmacotherapy also predicted greater attrition relative to CBT (18.5% versus 12.0%), and among the subset of studies with placebo arms, fewer negative outcomes overall relative to pill placebo (16.2% versus 24.8%). If replicated, more frequent superior improvement during pharmacotherapy than in CBT may signal operation of an unknown moderator (e.g., to be revealed by genotyping) among patients who tolerate treatment (e.g., don’t drop). In addition, younger age (among adults) predicted superior improvement on the HAM-D, whereas employment predicted superior response on the BDI. These latter findings fit broader patterns of better outcomes for patients with less (versus more) severe pathology [17–20]. Treatment modality did not change patients’ odds of deterioration, extreme non-response, or superior response on the HAM-D or BDI.

Pre-treatment symptom levels predicted outcomes and varied significantly between patients with negative versus unusually-positive outcomes. In general, pre-treatment symptom levels related directly to post-treatment symptom levels and inversely to the direction of symptom change. More specifically, lower pre-treatment symptom severity increased risk for deterioration, and patients who deteriorated had lower pre-treatment symptoms than did patients with superior improvement, within measures (HAM-D or BDI). In addition, higher pre-treatment symptoms increased odds of superior improvement within the HAM-D only. Specific depressive symptoms (HAM-D items) most relevant to these predictions included less depressed mood, less guilt, and fewer general somatic symptoms (deterioration), and more gastro-somatic symptoms (superior improvement).

Psychotherapy may produce negative, or fail to evoke unusually-positive, outcomes when psychotherapists are confrontational and over-interpret common experiences and patient characteristics as pathological [34]. However, lack of generalization across measures in our analyses of deterioration and superior improvement also suggests methodological artifacts, such as regression to the mean within measures. The patient (BDI) and clinician (HAM-D) measures may also capture somewhat different information [35] and have finite reliability and validity [36], as do all measures. Finally, patients with lower symptom scores pre-treatment may be entering treatment as a depressive episode is waxing, and their deterioration and lack of superior improvement during treatment might reflect the natural course of illness.

Higher pre-treatment depressive symptoms also predicted extreme non-response, and patients with extreme non-response had higher pretreatment symptoms than did patients with superior response, both within and between the HAM-D and BDI. In addition, lower-pre-treatment symptoms increased patients’ odds of superior response on the BDI. Specific depressive symptoms (HAM-D items) that predicted extreme non-response on the HAM-D included initial insomnia, middle insomnia, and weight loss. Although CBT and pharmacotherapy benefit many severely-depressed patients [37], our analyses suggest that severe depression at the beginning of treatment adds risk for severe depression, and reduces likelihood of being symptom-free, at the end of pharmacotherapy or CBT.

Pre-treatment symptom levels and patient characteristics analyzed here (age, gender, education, ethnicity, pre-treatment symptom severity, double depression, comorbidity) did not moderate treatment effects on deterioration, extreme non-response, superior improvement, or superior response. Consequently, the current results do not support differential treatment selection (CBT versus pharmacotherapy) from symptom levels and patient characteristics to prevent negative or potentiate unusually-positive outcomes. Other patient characteristics or events outside of treatment may account for these outcomes and are potential targets for future research. Even so, empirical guidance for selecting CBT or pharmacotherapy for a given patient based on average post-treatment symptom scores is available [16,38].

Based on the current findings, we recommend assessing symptom levels frequently and longitudinally (especially among patients with high pre-treatment severity) and pursuing rapid corrective action (e.g., increased session frequency, switching or augmenting treatment) if patients do not progress adequately [12]. However, other negative outcomes and adverse events, including discrete behaviors (e.g., suicide), psychosocial outcomes (e.g., divorce; domestic violence), dropping out of treatment, and medication side effects (e.g., sexual dysfunction) arguably are distinct from depressive symptom levels [11] and may require different preventive efforts. Similarly, positive outcomes distinct from depressive symptoms are also important therapeutic targets (e.g., social-interpersonal functioning) [39].

The current study has additional limitations that temper our conclusions. For example, although the sample was large, low base-rates of divergent outcomes and attrition (presumably often negative) may have limited detection of predictors and moderators. In addition, many important patient characteristics (e.g., personality profiles, depressive cognitive content, therapeutic alliance, social support) were not analyzed and could be targets for future research on predictors and moderators of divergent outcomes [10,40]. Moreover, our results from randomized clinical trials may not generalize fully to routine clinical practice [12,14]. Similarly, generalization of our findings to other negative (e.g., adverse events, side effects) or positive (e.g., improvement in work or social functioning) outcomes not studied here is unknown. Finally, our analyses focused on trials of CBT versus pharmacotherapy and do not address combinations, sequences, dissemination, or the quality of implementation of treatments.

Preventing negative and potentiating unusually-positive outcomes may improve the overall efficacy of CBT and pharmacotherapy for depression. In this effort, future research might profitably test replication and mechanisms of the current findings. For example, greater odds of clinician-rated superior improvement among younger (versus older) adults and those treated with pharmacotherapy (versus CBT), and greater odds of patient-rated superior response for patients with full-time (versus less) employment, possibly reflect measurable intra-patient, environmental, or interacting mechanisms. Similarly, specific depressive symptoms linked with clinician-rated extreme non-response (insomnia, weight loss), deterioration (less depressed mood, less guilt, fewer general somatic symptoms), and superior improvement (gastro-somatic symptoms) may reflect sampling or measurement error but may also reveal patient subpopulations with different treatment-response trajectories.

The current results clarify expectations for outcomes in acute-phase CBT or pharmacotherapy for adult depression. First, whereas the majority of patients responded by conventional standards [32], we found that a few patients (13%) had negative outcomes and some (15%) had very positive outcomes. Second, pre-treatment symptom levels help forecast negative and unusually-positive outcomes: Patients with severe pre-treatment symptoms are more likely to have large drops in symptoms (superior improvement) and unlikely to get worse (deteriorate), but they are also more likely to end treatment with severe symptoms (extreme non-response) and unlikely to end treatment symptom-free (superior response). Conversely, patients with milder pre-treatment symptoms are more likely to end treatment symptom-free (superior response) and unlikely to end treatment with severe symptoms (extreme non-response), but they are also more likely to get worse (deteriorate) and less likely to have large drops in symptoms (superior improvement). Finally, choosing pharmacotherapy versus CBT may increase patients’ odds of both discontinuing treatment and clinician-rated superior response.

Supplementary Material

Supplement

Acknowledgments

Dr. Vittengl is a reviewer for UpToDate. NIMH grant MH-45043 (Dr. Jarrett) provided data. Dr. Jarrett is a paid consultant to UpToDate and her medical center receives fees for cognitive therapy she provides to patients. Dr. Hollon is supported by NIMH grant MH60713 and MH01697. Dr. DeRubeis is supported by NIMH grant MH60998. Dr. Dunlop reports grants from Bristol Myers Squibb, grants from GlaxoSmithKline, grants and personal fees from Pfizer, grants from Takeda, grants from Assurex, grants from Otsuka, personal fees from MedAvante, personal fees from Hoffmann LaRoche, outside the submitted work. Within the last three years, Prof. Hegerl was an advisory board member for Lilly, Lundbeck, Takeda Pharmaceuticals, Servier and Otsuka Pharma; a consultant for Nycomed; and a speaker for Bristol-Myers Squibb, Medice Arzneimittel, Novartis and Roche Pharma. Dr. Mergl reports a Consultancy Agreement with Nycomed, a Takeda company. Dr. Mohr is supported by the National Institute of Heath grants R01 MH100482, R01 MH095753, P20 MH090318, R34 MH095907 and reports a consulting relationship with Otsuka Pharmaceuticals. In the last 3 years, Dr. Rush has received consulting fees from Brain Resource Ltd., Duke-NUS, H. Eli Lilly, Lundbeck A/S, Medavante, Inc; National Institute of Drug Abuse, Santium Inc.,Takeda USA, University of Colorado, University of Texas Southwestern Medical Center at Dallas; speaking fees from the University of California at San Diego, Hershey Penn State Medical Center, and the American Society for Clinical Psychopharmacology; royalties from Guilford Publications and the University of Texas Southwestern Medical Center; a travel grant from CINP and research support from Duke-National University of Singapore. Dr. Cuijpers has received royalties from Servier, Atheneum publishers, and HB publishers, and speaking fees from the University of Trier, Vanderbilt University, the VGCt, the NVGRT, and has received grant support from ZonMw, the European Commission, and the NutsOhra Foundation.

Footnotes

Disclosures

Ms. Weitz and Drs. Twisk, Cristea, David, Dimidjian, Faramarzi, Kennedy, Kheirkhah, Miranda, Segal, Siddique, and Simmons report no financial interests or conflicts of interest.

1

Among the five trials including pill placebo arms, deterioration, extreme-non response, superior improvement, and superior response did not differ significantly among CBT, pharmacotherapy, and placebo, perhaps due to lower statistical power (see Online Supplement 3). However, the frequency of any negative outcome was lower in pharmacotherapy versus placebo, OR=0.59.

2

Odds ratios for continuous predictors reflect changes of 1 SD on the predictor. Table 1 shows SDs.

3

Odds ratios for HAM-D items reflect changes of 1 point on the item.

Contributor Information

Jeffrey R. Vittengl, Department of Psychology, Truman State University, Kirksville, USA.

Robin B. Jarrett, Department of Psychiatry, The University of Texas Southwestern Medical Center, Dallas, USA

Erica Weitz, Department of Clinical Psychology and EMGO Institute for Health and Care Research, Amsterdam, VU University Amsterdam, The Netherlands

Steven D. Hollon, Department of Psychology, Vanderbilt University, Nashville, USA

Jos Twisk, EMGO Institute for Health and Care Research, Amsterdam, VU University Amsterdam, The Netherlands

Ioana Cristea, Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, Cluj, Romania

Daniel David, Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, Cluj, Romania

Robert J. DeRubeis, Department of Psychology, University of Pennsylvania, Philadelphia, USA

Sona Dimidjian, Department of Psychology and Neuroscience, University of Colorado, Boulder, USA

Boadie W. Dunlop, Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, USA

Mahbobeh Faramarzi, Fatemeh Zahra Infertility and Reproductive Health Research Center, Babol University of Medical Sciences, Babol, Iran

Ulrich Hegerl, Department of Psychiatry and Psychotherapy, University of Leipzig, Leipzig, Germany

Sidney H. Kennedy, Department of Psychiatry, University of Toronto, Toronto, Canada

Farzan Kheirkhah, Department of Psychiatry, Faculty of Medicine, Babol University of Medical Sciences, Babol, Iran

Roland Mergl, Department of Psychiatry and Psychotherapy, University of Leipzig, Leipzig, Germany

Jeanne Miranda, Health Services Research Center, Neuropsychiatric Institute, University of California, Los Angeles, USA

David C. Mohr, Center for Behavioral Intervention Technologies, Feinberg School of Medicine, Northwestern University, Chicago, USA

A. John Rush, Duke–National University of Singapore Graduate Medical School, Singapore

Zindel V. Segal, Department of Psychology, University of Toronto – Scarborough, Toronto, Canada

Juned Siddique, Department of Preventative Medicine, Feinberg School of Medicine, Northwestern University, Chicago, USA

Anne D. Simons, Department of Psychology, University of Notre Dame, Notre Dame, USA

Pim Cuijpers, Department of Clinical Psychology and EMGO Institute for Health and Care Research, Amsterdam, VU University Amsterdam, The Netherlands

References

  • 1.Cuijpers P, Berking M, Andersson G, Quigley L, Kleiboer A, Dobson KS. A meta-analysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Can J Psychiatry. 2013;58:376–385. doi: 10.1177/070674371305800702. [DOI] [PubMed] [Google Scholar]
  • 2.Iovieno N, Tedeschini E, Ameral VE, Rigatelli M, Papakostas GI. Antidepressants for major depressive disorder in patients with a co-morbid axis-III disorder: a meta-analysis of patient characteristics and placebo response rates in randomized controlled trials. Int Clin Psychopharmacol. 2011;26:69–74. doi: 10.1097/YIC.0b013e328340775e. [DOI] [PubMed] [Google Scholar]
  • 3.Cuijpers P, Karyotaki E, Weitz E, Andersson G, Hollon SD, van Straten A. The effects of psychotherapies for major depression in adults on remission, recovery and improvement: A meta-analysis. J Affect Disord. 2014;159:118–126. doi: 10.1016/j.jad.2014.02.026. [DOI] [PubMed] [Google Scholar]
  • 4.Cuijpers P, Turner EH, Mohr DC, Hofmann SG, Andersson G, Berking M, Coyne J. Comparison of psychotherapies for adult depression to pill placebo control groups: A meta-analysis. Psychol Med. 2014;44:685–695. doi: 10.1017/S0033291713000457. [DOI] [PubMed] [Google Scholar]
  • 5.Dunlop BW, Thase ME, Wun CC, Fayyad R, Guico-Pabia CJ, Musgnung J, Ninan PT. A meta-analysis of factors impacting detection of antidepressant efficacy in clinical trials: The importance of academic sites. Neuropsychopharmacology. 2012;37:2830–2836. doi: 10.1038/npp.2012.153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Dimidjian S, Hollon SD. How would we know if psychotherapy were harmful? Am Psychol. 2010;65:21–33. doi: 10.1037/a0017299. [DOI] [PubMed] [Google Scholar]
  • 7.Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry. 1960;23:56–62. doi: 10.1136/jnnp.23.1.56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Zimmerman M, Posternak MA, Chelminski I. Is the cutoff to define remission on the Hamilton Rating Scale for Depression too high? J Nerv Ment Dis. 2005;193:170–175. doi: 10.1097/01.nmd.0000154840.63529.5d. [DOI] [PubMed] [Google Scholar]
  • 9.Ogles B, Lambert M, Sawyer J. Clinical significance of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. J Consult Clin Psychol. 1995;63:321–326. doi: 10.1037//0022-006x.63.2.321. [DOI] [PubMed] [Google Scholar]
  • 10.Coffman SJ, Martell CR, Dimidjian S, Gallop R, Hollon SD. Extreme nonresponse in cognitive therapy: Can behavioral activation succeed where cognitive therapy fails? J Consult Clin Psychol. 2007;75:531–541. doi: 10.1037/0022-006X.75.4.531. [DOI] [PubMed] [Google Scholar]
  • 11.Linden M, Schermuly-Haupt M. Definition, assessment and rate of psychotherapy side effects. World Psychiatry. 2014;13:306–309. doi: 10.1002/wps.20153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Lambert MJ. Prevention of treatment failure: The use of measuring, monitoring, and feedback in clinical practice. Washington, DC: American Psychological Association; 2010. Predicting negative treatment outcome: Methods and estimates of accuracy; pp. 83–105. [Google Scholar]
  • 13.Lambert MJ. The efficacy and effectiveness of psychotherapy. In: Lambert MJ, editor. Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change. 6th. New York, NY: Wiley; 2013. pp. 169–218. [Google Scholar]
  • 14.Blais MA, Malone JC, Stein MB, Slavin-Mulford J, O’Keefe SM, Renna M, Sinclair SJ. Treatment as usual (TAU) for depression: A comparison of psychotherapy, pharmacotherapy, and combined treatment at a large academic medical center. Psychotherapy. 2013;50:110–118. doi: 10.1037/a0031385. [DOI] [PubMed] [Google Scholar]
  • 15.Koenig AM, Jarrett RB, Gallop R, Barrett MS, Thase ME. Extreme nonresponse to acute phase cognitive therapy for depression: An attempt to replicate and extend. Behav Ther. 2014;45:300–313. doi: 10.1016/j.beth.2013.12.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.DeRubeis RJ, Cohen ZD, Forand NR, Fournier JC, Gelfand LA, Lorenzo-Luaces L. The Personalized Advantage Index: translating research on prediction into individualized treatment recommendations. A demonstration. PLoS One. 2014;9:e83875. doi: 10.1371/journal.pone.0083875. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Jarrett RB, Minhajuddin A, Kangas JL, Friedman ES, Callan JA, Thase ME. Acute phase cognitive therapy for recurrent major depressive disorder: who drops out and how much do patient skills influence response? Behav Res Ther. 2013;51:221–230. doi: 10.1016/j.brat.2013.01.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Driessen E, Hollon SD. Cognitive behavioral therapy for mood disorders: efficacy, moderators and mediators. Psychiatr Clin North Am. 2010;33:537–555. doi: 10.1016/j.psc.2010.04.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Rush AJ, Wisniewski SR, Warden D, Luther JF, Davis LL, Fava M, Trivedi MH. Selecting among second-step antidepressant medication monotherapies: Predictive value of clinical, demographic, or first-step treatment features. Arch Gen Psychiatry. 2008;65:870–880. doi: 10.1001/archpsyc.65.8.870. [DOI] [PubMed] [Google Scholar]
  • 20.Souery D, Oswald P, Massat I, Bailer U, Bollen J, Demyttenaere K, Mendlewicz J. Clinical factors associated with treatment resistance in major depressive disorder: Results from a European multicenter study. J Clin Psychiatry. 2007;68:1062–1067. doi: 10.4088/jcp.v68n0713. [DOI] [PubMed] [Google Scholar]
  • 21.Tedlow J, Fava M, Uebelacker L, Nierenberg AA, Alpert JE, Rosenbaum J. Outcome definitions and predictors in depression. Psychother Psychosom. 1998;67:266–270. doi: 10.1159/000012290. [DOI] [PubMed] [Google Scholar]
  • 22.Van HL, Schoevers RA, Dekker J. Predicting the outcome of antidepressants and psychotherapy for depression: a qualitative, systematic review. Harv Rev Psychiatry. 2008;16:225–234. doi: 10.1080/10673220802277938. [DOI] [PubMed] [Google Scholar]
  • 23.American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th. Washington, DC: American Psychiatric Association; 2013. [Google Scholar]
  • 24.Beck AT, Haigh EA. Advances in cognitive theory and therapy: the generic cognitive model. Annu Rev Clin Psychol. 2014;10:1–24. doi: 10.1146/annurev-clinpsy-032813-153734. [DOI] [PubMed] [Google Scholar]
  • 25.Ma Y. Neuropsychological mechanism underlying antidepressant effect: a systematic meta-analysis. Mol Psychiatry. 2015;20:311–319. doi: 10.1038/mp.2014.24. [DOI] [PubMed] [Google Scholar]
  • 26.Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry. 1961;4:561–571. doi: 10.1001/archpsyc.1961.01710120031004. [DOI] [PubMed] [Google Scholar]
  • 27.Beck AT, Steer RA, Brown GK. Beck Depression Inventory–Second Edition Manual. San Antonio, TX: Psychological Corporation; 1996. [Google Scholar]
  • 28.Grundy CT, Lambert MJ, Grundy EM. Assessing clinical significance: Application to the Hamilton Rating Scale for Depression. J Ment Health. 1996;5:25–33. [Google Scholar]
  • 29.Seggar LB, Lambert MJ, Hansen NB. Assessing clinical significance: Application to the Beck Depression Inventory. Behavior Therapy. 2002;33:253–269. [Google Scholar]
  • 30.Vittengl JR, Clark LA, Thase ME, Jarrett RB. Nomothetic and idiographic symptom change trajectories in acute-phase cognitive therapy for recurrent depression. J Consult Clin Psychol. 2013;81:615–626. doi: 10.1037/a0032879. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Cuijpers P, Weitz E, Twisk J, Kuehner C, David D, DeRubeis R, Hollon SD. Gender as predictor and moderator of outcome in cognitive behavior therapy and pharmacotherapy for adult depression: An “individual patient data” meta-analysis. Depress Anxiety 2014. 2014;31:941–951. doi: 10.1002/da.22328. [DOI] [PubMed] [Google Scholar]
  • 32.Weitz E, Hollon SD, Twisk J, van Straten A, David D, Cuijpers P. Does baseline depression severity moderate outcomes between CBT and pharmacotherapy? An individual participant data meta-analysis. JAMA Psychiatry In press. [Google Scholar]
  • 33.Jarrett RB, Minhajuddin A, Gershenfeld H, Friedman ES, Thase ME. Preventing depressive relapse and recurrence in higher-risk cognitive therapy responders: a randomized trial of continuation phase cognitive therapy, fluoxetine, or matched pill placebo. JAMA Psychiatry. 2013;70:1152–1160. doi: 10.1001/jamapsychiatry.2013.1969. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Boisvert C, Faust D. Iatrogenic symptoms in psychotherapy. A theoretical exploration of the potential impact of labels, language, and belief systems. Am J Psychother. 2002;56:244–259. doi: 10.1176/appi.psychotherapy.2002.56.2.244. [DOI] [PubMed] [Google Scholar]
  • 35.Dunlop BW, McCabe B, Eudicone JM, Sheehan JJ, Baker RA. How well do clinicians and patients agree on depression treatment outcomes? Implications for personalized medicine. Hum Psychopharmacol. 2014;29:528–536. doi: 10.1002/hup.2428. [DOI] [PubMed] [Google Scholar]
  • 36.Cusin C, Yang H, Yeung A, Fava M. Rating scales for depression. In: Baer L, Blais MA, editors. Handbook of clinical rating scales and assessment in psychiatry and mental health. Totowa, NJ: Humana; 2010. pp. 7–35. [Google Scholar]
  • 37.Hollon SD. Cognitive and behavior therapy in the treatment and prevention of depression. Depress Anxiety. 2011;28:263–266. doi: 10.1002/da.20797. [DOI] [PubMed] [Google Scholar]
  • 38.Simon GE, Perlis RH. Personalized medicine for depression: Can we match patients with treatments? Am J Psychiatry. 2010;167:1445–1455. doi: 10.1176/appi.ajp.2010.09111680. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Renner F, Cuijpers P, Huibers MJ. The effect of psychotherapy for depression on improvements in social functioning: a meta-analysis. Psychol Med. 2014;44:2913–2926. doi: 10.1017/S0033291713003152. [DOI] [PubMed] [Google Scholar]
  • 40.Mohr DC. Negative outcome in psychotherapy: A critical review. Clinical Psychology: Science and Practice. 1995;2:1–27. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement
NIHMS797067-supplement-Supplement.docx (698.4KB, docx)

RESOURCES