The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: an umbrella review and meta‐analytic evaluation of recent meta‐analyses

Abstract

Mental disorders represent a worldwide public health concern. Psychotherapies and pharmacotherapies are recommended as first line treatments. However, evidence has emerged that their efficacy may be overestimated, due to a variety of shortcomings in clinical trials (e.g., publication bias, weak control conditions such as waiting list). We performed an umbrella review of recent meta‐analyses of randomized controlled trials (RCTs) of psychotherapies and pharmacotherapies for the main mental disorders in adults. We selected meta‐analyses that formally assessed risk of bias or quality of studies, excluded weak comparators, and used effect sizes for target symptoms as primary outcome. We searched PubMed and PsycINFO and individual records of the Cochrane Library for meta‐analyses published between January 2014 and March 2021 comparing psychotherapies or pharmacotherapies with placebo or treatment‐as‐usual (TAU), or psychotherapies vs. pharmacotherapies head‐to‐head, or the combination of psychotherapy with pharmacotherapy to either monotherapy. One hundred and two meta‐analyses, encompassing 3,782 RCTs and 650,514 patients, were included, covering depressive disorders, anxiety disorders, post‐traumatic stress disorder, obsessive‐compulsive disorder, somatoform disorders, eating disorders, attention‐deficit/hyperactivity disorder, substance use disorders, insomnia, schizophrenia spectrum disorders, and bipolar disorder. Across disorders and treatments, the majority of effect sizes for target symptoms were small. A random effect meta‐analytic evaluation of the effect sizes reported by the largest meta‐analyses per disorder yielded a standardized mean difference (SMD) of 0.34 (95% CI: 0.26‐0.42) for psychotherapies and 0.36 (95% CI: 0.32‐0.41) for pharmacotherapies compared with placebo or TAU. The SMD for head‐to‐head comparisons of psychotherapies vs. pharmacotherapies was 0.11 (95% CI: –0.05 to 0.26). The SMD for the combined treatment compared with either monotherapy was 0.31 (95% CI: 0.19‐0.44). Risk of bias was often high. After more than half a century of research, thousands of RCTs and millions of invested funds, the effect sizes of psychotherapies and pharmacotherapies for mental disorders are limited, suggesting a ceiling effect for treatment research as presently conducted. A paradigm shift in research seems to be required to achieve further progress.

Mental disorders represent a worldwide public health concern1, 2. Psychotherapies and pharmacotherapies are recommended as first line treatments3, 4. However, evidence has recently emerged suggesting that the efficacy of both types of treatment may have been overestimated, due to several shortcomings of clinical trials, such as publication bias, researcher allegiance, or use of weak comparison groups (in particular, waiting list)5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16. A realistic estimate of the efficacy of psychotherapies and pharmacotherapies is important to obtain.

Meta‐analyses and systematic reviews of randomized controlled trials (RCTs) are thought to provide the highest level of evidence ¹⁷ . However, not only individual RCTs but also meta‐analyses may be affected by the above‐mentioned biases6, 18, 19. To avoid overestimating treatment efficacy, meta‐analyses need to take risk of bias systematically into account6, 18, 19, 20. Furthermore, estimates of efficacy depend upon the comparator against which treatments are tested. Waiting list conditions, for example, represent a relatively weak comparator, leading to larger effect sizes6, 8, 21, 22. Comparisons with placebo or treatment‐as‐usual (TAU) provide better estimates of the true efficacy of a treatment6, 22.

The most recent comprehensive review of meta‐analyses of both psychotherapies and pharmacotherapies in mental disorders, including 61 meta‐analyses, was published in 2014, reporting a medium effect size (standardized mean difference, SMD = 0.50) ⁸ . Some of the included meta‐analyses, however, used waiting list comparisons in the assessment of overall efficacy. In addition, the authors seem to have just averaged the extracted effect sizes, without performing a meta‐analytic evaluation including weighting effect sizes. Furthermore, a large number of studies and meta‐analyses have been published since 2014.

For all these reasons, we carried out an up‐to‐date umbrella review of recent meta‐analyses of psychotherapies and pharmacotherapies for the main mental disorders in adults which used placebo or TAU as comparison groups and formally assessed risk of bias or quality of studies. As the primary outcome, we used the effect size for target symptoms of the relevant disorder.

METHODS

The study protocol of this umbrella review was registered in advance at PROSPERO (International Prospective Register of Systematic Reviews), registration number: CRD42020155452.

Inclusion criteria

Meta‐analyses of RCTs comparing psychotherapies or pharmacotherapies to placebo or TAU in adults with mental disorders published since 2014 were eligible. We also considered meta‐analyses comparing psychotherapies vs. pharmacotherapies head‐to‐head, or their combination to either monotherapy. Only meta‐analyses which formally assessed risk of bias or quality of studies were included. If multiple meta‐analyses fulfilling the inclusion criteria were available for one condition, all of them were included. Reporting of SMD or other measures of between‐group effect size was required.

All types of pharmacotherapy or psychotherapy were eligible for inclusion. Meta‐analyses examining specific subgroups (e.g., treatment resistant depression, primary care patients, the elderly), psychiatric or somatic comorbidities (e.g., depression in cancer patients), specific settings (e.g., group therapy only, or inpatient therapy) or augmentation strategies (e.g., psychostimulants added to antipsychotic drugs in schizophrenia), or focusing on secondary outcomes (e.g., quality of life in depression) were not included. These inclusion criteria are consistent with the above‐mentioned 2014 review ⁸ , except for excluding waiting list comparisons and requiring meta‐analyses to have assessed risk of bias or quality of studies. Both standard and network meta‐analyses were eligible.

Combining data of patients receiving TAU or placebo with those of patients on waiting list has been shown to inflate effect sizes8, 22, 23. On the other hand, mixing data of patients on TAU with those receiving specific therapies (e.g., cognitive‐behaviour therapy, CBT) can be expected to underestimate the effect size of the treatment in question. Therefore, meta‐analyses mixing data of TAU or placebo with waiting list or active treatments were excluded.

Search for studies

We searched PubMed and PsycINFO and individual records of the Cochrane Library for meta‐analyses of RCTs of psychotherapies and/or pharmacotherapies for mental disorders in adults published between January 2014 and March 2021.

Four reviewers independently searched for studies. Decision on inclusion was made by consensus including another rater. Search terms were meta‐analy* or metaanaly* combined with the thesaurus of the individual databases concerning each disorder. To provide comparable results, we used the syntax applied in the previous most comprehensive review ⁸ .

Data extraction

We focused on effect sizes and 95% confidence intervals (CIs) for the target symptoms of the relevant disorder (primary outcome). We extracted between‐group SMDs and related measures (Cohen's d, Hedges' g) as reported in the meta‐analyses. Odds ratios (ORs) and hazard ratios (HRs) were converted to SMDs24, 25. Data on relative risk (RR) were extracted as reported. We used Cohen's convention of d=0.2, d=0.5 and d=0.8 as indicating small, medium and large effect sizes ²⁶ , corresponding to success rate differences of 11%, 28% and 43%; numbers needed to treat of 9, 4 and 2; ORs of 1.43, 2.48 and 4.25; RRs of 1.22, 1.86 and 3.00; and HRs of 1.3, 1.9 and 2.824, 25, 27, 28, 29. Intention‐to‐treat data were preferred whenever available.

If meta‐analyses took risk of bias into account by, for example, additionally reporting data separately for low risk of bias studies or correcting for publication bias, we listed all reported effect sizes but preferably focused on the corrected or high quality data for interpreting results.

Rates of remission and response were included as secondary outcomes when available. Dichotomous variables have some limitations ³⁰ , but complementarily to SMDs they can provide useful information about efficacy.

One author extracted data (type of treatment and disorder, number of studies, number of participants, type of comparator, risk of bias, adverse events/side effects, and effect sizes). Data were cross‐checked independently by two raters each.

Quality assessment

The quality of the included meta‐analyses was independently assessed by two raters. For the purpose of this review, we used the items 1 to 9 of the Checklist for Systematic Reviews and Research Syntheses31, 32, complemented by item 12 of AMSTAR 2²⁰ (“Was the impact of risk of bias in individual studies on results of the meta‐analysis taken into account?”) and an additional item addressing whether the meta‐analysis was registered. In case of disagreement between raters, consensus ratings were used.

Data synthesis

The results of the largest meta‐analyses for each condition, i.e. those including most RCTs, are presented and evaluated separately. Additionally, these independent meta‐analyses were included in second‐order meta‐analyses combining their summary effect sizes across all the different mental disorders ³³ . This allowed to obtain a weighted effect of psychotherapy or pharmacotherapy across all mental disorders, and weighted effects for the benefits of combined therapy, and for the comparative efficacy of psychotherapy vs. pharmacotherapy. The analysis was performed by Comprehensive Meta‐Analysis (CMA, Version 3) using a random effects model based on SMDs and their CIs via the CMA analysis option “generic estimates”.

Heterogeneity was assessed using the I² statistic. If meta‐analyses did not report an overall effect size, but effect sizes for specific treatments and comparisons, the effect sizes of the relevant comparisons were aggregated by CMA and the resulting overall SMDs were entered into the second‐order meta‐analyses across disorders. Only effect estimates based on at least two primary RCTs were used.

RESULTS

Included meta‐analyses

The search retrieved 23,601 items, reduced to 19,500 after removing duplicates, which were screened by titles and abstracts. Full‐text evaluation was carried out for 319 papers. One hundred and two meta‐analyses fulfilled the inclusion criteria (see Figure 1 and supplementary information). These encompassed 69 meta‐analytic comparisons of pharmacotherapies with placebo or TAU, 26 comparisons of psychotherapies with placebo or TAU, 11 comparisons of psychotherapies vs. pharmacotherapies head‐to‐head, and 13 comparisons of combined psychotherapy and pharmacotherapy to either monotherapy6, 12, 13, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134. The 102 meta‐analyses encompassed 3,782 RCTs (range: 2 to 522) and 650,514 patients (range: 65 to 116,477) (see supplementary information).

Figure 1.

PRISMA flow chart. RoB – risk of bias, TAU – treatment as usual

Across all meta‐analyses, the mean number of positively rated items in the quality assessment was 8.71±1.43 (range: 4 to 11). The items 10 (item 12 of AMSTAR 2, addressing whether the meta‐analyses took the impact of bias on results into account) and 11 (study registration) were the least frequently fulfilled (48% and 47%, respectively). The quality of meta‐analyses was not significantly different between psychotherapies and pharmacotherapies (mean of positively rated items: 8.95±1.12 for psychotherapies, 8.68±1.54 for pharmacotherapies, t=0.74, p=0.46).

Psychotherapies and pharmacotherapies vs. TAU or placebo

In the largest meta‐analyses, the effect sizes of both psychotherapies and pharmacotherapies in comparison to TAU or placebo were small (SMD<0.50) for most disorders and treatments (see Figure 2 and supplementary information). Medium effect sizes were found only for pharmacotherapies of obsessive‐compulsive disorder (OCD) (SMD=0.56) ⁷² , bulimia nervosa (SMD=0.61) ⁸⁰ , and somatoform disorders (SMD=0.50) ⁹¹ , and for psychotherapies of post‐traumatic stress disorder (PTSD) (SMD=0.54) ⁵⁴ and borderline personality disorder (SMD=0.57) ⁹³ . Large effect sizes were only reported for psychotherapy of OCD (SMD=1.03) ⁷⁴ , with, however, a substantial proportion of patients taking concomitant pharmacotherapy72, 74. Overall, risk of bias was often high (see Figure 2 and supplementary information).

Figure 2.

Effect sizes in the largest meta‐analyses of pharmacotherapies (squares) and psychotherapies (circles) in comparison to placebo or treatment‐as‐usual (TAU). PHA – pharmacotherapy; PSY – psychotherapy, SMD – standardized mean difference, * – adjusted for risk of bias, ° – adjusted for small‐study effects, MDD – major depressive disorder, GAD – generalized anxiety disorder, SAD – social anxiety disorder, OCD – obsessive‐compulsive disorder, PTSD – post‐traumatic stress disorder, PD – personality disorder, ADHD – attention‐deficit/hyperactivity disorder, H – high, M – medium, L – low, U – uncertain, NR – not reported. Where SMD is not provided, this means that no valid meta‐analysis reporting this value was available.

For psychotherapies and pharmacotherapies, second‐order random effects meta‐analyses in comparison to placebo or TAU yielded statistically significant but small SMDs of 0.34 (95% CI: 0.26‐0.42, I²=66.33%) and 0.36 (95% CI: 0.32‐0.41, I²=70.61%), respectively, across disorders (see Figure 2). For the aggregated data of psychotherapies and pharmacotherapies, the SMD was 0.35 (95 CI: 0.31‐0.39, I²=68.23%).

Depressive disorders

For psychotherapies of depressive disorders, the largest meta‐analysis reported a small SMD of 0.31, adjusted for biases, in comparison to TAU⁵¹ (see Figure 2). Taking all included meta‐analyses into account, psychotherapy achieved effect sizes (SMDs) between 0.11 and 0.61 in comparison to placebo or TAU6, 12, 37, 50, 51, 52, except for one outlying meta‐analysis reporting a large SMD post‐therapy (1.11), reduced to 0.27 at 3 to 12 month follow‐up and associated with a high risk of bias ⁵² . The majority of effect sizes were small (<0.50).

Only between 1% and 17% of studies of psychotherapy for depression were found to show a low risk of bias. When meta‐analyses took risk of bias into account, they consistently found a decrease in effect sizes (see supplementary information).

Across all forms of psychotherapy, remission from major depressive disorder (Hamilton Depression Rating Scale, HAM‐D <7) was achieved in 43% of patients, with no significant differences between the various psychotherapies ⁵ . Response (50% reduction of HAM‐D score) was achieved in 54% of patients ⁵ . TAU was superior to no treatment with regard to remission (33% vs. 23%), but inferior to psychotherapy (33% vs. 43%) ¹³⁵ .

The largest meta‐analysis of pharmacotherapy for depressive disorders reported a SMD of 0.30³⁶ (see Figure 2). All effect sizes (SMD) achieved by pharmacotherapy in comparison to placebo were below 0.50, ranging from 0.19 to 0.41. The exception was ketamine, which achieved large short‐term effects (0.83, 0.88) 24 hours and 3‐4 days after treatment, dropping to 0.31 after 7 days13, 34, 35, 36, 37, 38, 39, 41, 42, 43, 44, 45, 46, 47, 48, 49. Most effect sizes in terms of RRs were small as well (≤1.22).

The mean response rate for selective serotonin reuptake inhibitors (SSRIs) was 51% vs. 39% for placebo ³⁵ , corresponding to a small effect size ²⁷ .

Many trials of pharmacotherapy in depression showed a high risk of bias13, 35, 36, 42 (see supplementary information).

Anxiety disorders

In the largest meta‐analyses of anxiety disorders, psychotherapies achieved SMDs between 0.28 and 0.44^6,55,66 (see Figure 2). Overall, psychotherapies of anxiety disorders achieved SMDs compared to TAU or placebo between 0.01 and 0.726, 54, 55, 59, 65, 66, 71, except for two outlying effect sizes in generalized anxiety disorder (1.44, 1.32), each based on three studies only6, 55. Two effect sizes of psychotherapy (CBT) in social anxiety disorder were medium (0.72, 0.56) ⁶⁶ , but most effect sizes were small (see supplementary information).

Overall, only 17% of psychotherapy studies in anxiety disorders were found to show a low risk of bias ⁶ .

In the largest meta‐analyses for anxiety disorders, pharmacotherapies achieved SMDs in comparison to placebo between 0.33 and 0.45^53,56,64 (see Figure 2). Overall, effect sizes for pharmacotherapy were between 0.01 and 0.56, with the majority of effect sizes being small53, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 66, 67, 69, 70 (see supplementary information). RR ranged between 1.20 and 4.03, with most values being small, one medium (monoamine oxidase inhibitors), and one large (benzodiazepines, RR=4.03) ⁶⁹ .

For social anxiety disorder and generalized anxiety disorder, pharmacotherapy yielded response rates of 52% and 56%, respectively, versus 32% and 41% with placebo59, 69.

Obsessive‐compulsive disorder

For psychotherapy (CBT) of OCD, the largest meta‐analysis reported a large SMD (1.03) ⁷⁴ (see Figure 2). Considering all meta‐analyses, large SMDs in comparison to placebo were reported (0.91‐1.46)72, 74. At follow‐up of on average of 15.1 months after the end of treatment, SMDs decreased from 0.57 to 0.06 for all comparators ⁷⁴ . Follow‐up results were not available for a comparison against placebo. Most psychotherapy trials included patients taking stable doses of antidepressants72, 74, possibly overestimating effect sizes in favour of psychotherapy ⁷² .

For pharmacotherapy of OCD, the largest meta‐analysis reported a medium effect size (SMD=0.56) ⁷² (see Figure 2). Considering all meta‐analyses, small to medium SMDs were reported (0.45‐0.66).

For most studies of psychotherapy and pharmacotherapy, the risk of bias was high (see Figure 2 and supplementary information).

Post‐traumatic stress disorder

For psychotherapy (CBT) of PTSD, the largest meta‐analysis reported a medium effect size compared to TAU (SMD=0.54) ⁵⁴ (see Figure 2), which was stable at follow‐ups of up to 12 months after end of therapy ⁵⁴ . For PTSD related to childhood maltreatment, a SMD of 0.50 in comparison to TAU/placebo was found, which was reduced to 0.21 after adjusting for small sample size ⁷⁹ .

For pharmacotherapy of PTSD, the largest meta‐analysis reported a small SMD in comparison to placebo (0.21) ⁷⁶ (see Figure 2). Considering all meta‐analyses, effect sizes achieved by pharmacotherapy in comparison to placebo were heterogeneous (SMDs: –0.10 to 0.97)75, 76, 77, 78. Risk of bias was often high77, 78. A large SMD was obtained with phenelzine (0.97), a medium one with mirtazapine (0.79), desipramine (0.52) and olanzapine (0.51), all based on only one RCT except for olanzapine ⁷⁵ . For SSRIs and serotonin and norepinephrine reuptake inhibitors (SNRIs), a medium SMD was reported (0.50) ⁷⁷ . For all other drugs, SMDs were <0.50 (from –0.10 to 0.47).

Personality disorders

For psychotherapy of personality disorders, only a meta‐analysis of borderline personality disorder was available, which reported a medium SMD in comparison to TAU (0.57), with a high risk of bias in most studies (see Figure 2) ⁹³ .

An update for developing a Cochrane report of pharmacotherapy in borderline personality disorder did not provide meta‐analytic results since the authors did not find robust evidence ¹³⁶ .

Somatoform disorders

For psychotherapy of somatoform disorders, the largest meta‐analysis reported a small SMD (0.19, see Figure 2) in comparison to enhanced care, with high risk of bias due to lack of blinding ⁹² . For pharmacotherapy of somatoform disorders, the largest meta‐analysis reported a medium SMD (0.50, see Figure 2) in comparison with placebo ⁹¹ .

Considering all meta‐analyses, heterogeneous SMDs (from 0.13 to 0.91) were reported for pharmacotherapy, based on two or three RCTs, with a high risk of bias for most RCTs ⁹¹ .

Eating disorders

For psychotherapy of bulimia nervosa, no recent meta‐analysis fulfilled the inclusion criteria. For pharmacotherapy, the largest meta‐analysis reported a medium SMD in comparison with placebo (0.61, see Figure 2) ⁸⁰ . Considering all meta‐analyses, considerable heterogeneity among classes of drugs were found (SMDs: 0.10‐1.00) ⁸⁰ .

For psychotherapy of binge eating disorder, no meta‐analysis fulfilled the inclusion criteria. For pharmacotherapy, the largest meta‐analysis reported a small to medium SMD in comparison with placebo (0.45, see Figure 2) ⁸² . Considering all meta‐analyses, a small to medium effect size compared to placebo was found for pharmacotherapy (SMD=0.45, RR: 1.67, 2.61)81, 82. One of these meta‐analyses reported a high ⁸² , the other a medium to low risk of bias ⁸¹ .

For psychotherapy of anorexia nervosa, the largest meta‐analysis reported a small SMD in comparison with TAU (0.14, see Figure 2) ⁸⁵ . Overall, the effect sizes in comparison to TAU or placebo were small (SMD=0.10‐0.31, RR: 0.97, 1.28)84, 85, 86. For pharmacotherapy, the largest meta‐analysis reported a small effect size (SMD=0.25) ⁸³ .

Substance use disorders

For both psychotherapy and pharmacotherapy of substance use disorders, the largest meta‐analysis reported small SMDs in comparison with TAU or placebo (0.23 and 0.26, respectively, see Figure 2)95, 96. For psychotherapy, the effect size decreased at follow‐ups ≥8 months after end of treatment (SMD=0.05) ⁹⁶ . Considering all meta‐analyses, small effect sizes were found for pharmacotherapy (SMDs: 0.07 to 0.35, RR: 0.32‐1.39)94, 95.

Insomnia

For psychotherapy, no recent meta‐analysis fulfilled the inclusion criteria. The quality of studies was found to be low ¹³⁷ . For pharmacotherapy, the largest meta‐analysis reported a small SMD in comparison with placebo (0.27, see Figure 2) ⁹⁰ . Overall, for pharmacotherapy of insomnia, small to medium SMDs were reported (0.07 to 0.58)88, 89, 90. One meta‐analysis provided effect sizes only for one of eight outcome measures, with large SMDs (0.88‐1.38), suggesting selective reporting ⁸⁷ .

Attention‐deficit/hyperactivity disorder (ADHD)

For psychotherapy of ADHD in adults, no meta‐analysis could be included ¹³⁴ . For pharmacotherapy, the largest meta‐analysis reported a small to medium SMD in comparison with placebo (0.45, see Figure 2) ⁹⁹ . Considering all meta‐analyses, the effects of pharmacotherapy were heterogeneous (from 0.16 to 0.97)98, 99, 100, 101, 102. Large SMDs were found for amphetamines98, 100, 102, small to medium SMDs for methylphenidate100, 101. Risk of bias was often high or unclear, and level of evidence was low to very low98, 100.

Schizophrenia spectrum disorders

Results of psychotherapy in schizophrenia spectrum disorders were evaluated in the context of pharmacotherapy (i.e., patients usually received concomitant medication). The largest meta‐analysis reported a small SMD in comparison with TAU (0.33, see Figure 2) ¹¹¹ . Considering all meta‐analyses, small effect sizes compared to nonspecific controls were found for overall symptoms, positive and negative symptoms (SMDs: 0.32, 0.24 and 0.08, respectively) ¹³⁸ . In comparison to TAU, psychotherapy achieved small to medium SMDs for negative symptoms (0.15‐0.58)110, 112.

For psychotherapy, a response rate of 13% for overall symptoms and 25% for positive symptoms was found¹³⁹ (a reduction of symptoms of at least 50% was required). The response rate decreased considerably if researcher allegiance (authors evaluated the therapy that they developed) was taken into account (from 13% to 4.9%) ¹³⁹ .

For acute pharmacological treatment of schizophrenia, the largest meta‐analysis reported an overall SMD of 0.45 for target symptoms, reduced to 0.38 after adjusting for publication bias (Figure 2) ¹⁰³ . These results are consistent with meta‐analyses on specific drugs, such as quetiapine (SMD=0.33), cariprazine (SMD: 0.32‐0.37), lurasidone (SMD: 0.34‐0.47), and aripiprazole and brexpiprazole (RR=1.1)104, 107, 108, 109. A large effect size was reported for clozapine (SMD=0.89) ¹⁰³ . Large and medium SMDs were achieved by long‐acting injectable antipsychotics in the maintenance treatment of non‐affective psychoses (RR: 1.75‐3.70) ¹⁰⁷ .

For the acute treatment of schizophrenia with pharmacotherapy, differences in response rates in comparison with placebo were small (23% vs. 14%) ¹⁴ .

Bipolar disorder

For psychotherapy of bipolar disorder, the largest meta‐analysis reported a small effect size in comparison to TAU (SMD=0.18, see Figure 2) ¹²¹ , with small effect sizes for both depression and mania symptoms (SMDs: 0.23 and 0.05, respectively) and for relapse prevention post‐therapy (RR: 1.52) ¹²¹ . At follow‐up 26 to 78 weeks post‐therapy, SMDs were 0.21 and 0.38; RR for relapse was 1.35 ¹²¹ . Psychotherapy was given in the context of concomitant pharmacotherapy.

For the acute treatment of mania, the results for pharmacotherapy are heterogeneous. One meta‐analysis reported medium SMDs for cariprazine (0.51‐0.52) ¹¹⁹ , and another reported a small effect size for aripiprazole (SMD=0.16) ¹¹⁵ . These two meta‐analyses included only three RCTs. A third meta‐analysis reported a very large SMD (1.51) for tamoxifen ¹²⁰ , based on two RCTs with small samples (16 and 66 cases, respectively), making the results questionable. This study represents a clear outlier.

For the acute treatment of bipolar depression, the largest meta‐analysis of pharmacotherapy reported heterogeneous results, with effect sizes (SMDs) between 1.41 and –1.84 ¹¹³ . Large effect sizes were achieved by fluoxetine (1.41), divalproex (1.25), lurasidone (1.15), moclobemide (1.09), cariprazine (0.85) and imipramine (0.86) ¹¹³ , all of them based, however, on only 0‐3 direct comparisons. Some drugs achieved medium effect sizes (olanzapine, phenelzine, tranylcypromine). The effect sizes of all other drugs were small ¹¹³ . Quetiapine achieved an almost medium effect size (SMD=0.48) based on 11 direct comparisons ¹¹³ .

For the prevention of manic/hypomanic/mixed episodes, effect sizes of lithium were almost medium (RR=1.85) ¹¹⁴ . Medium effect sizes were reported for olanzapine (RR=2.88) and risperidone (RR=2.88); large effect sizes for aripiprazole (once monthly) and asenapine (RR: 3.31, 4.81) ¹¹⁴ . The results for asenapine are based on only one RCT ¹¹⁶ . For all other drugs, effect sizes were small ¹¹⁴ .

For the prevention of any mood episode relapse, a large effect size was found for asenapine (RR=3.82), with the caveat mentioned above ¹¹⁴ . Medium effect sizes were reported for quetiapine and olanzapine114, 116, 118. Small effect sizes for the prevention of any mood episode were achieved by lithium (RR: 1.60, 1.61) and several other drugs114, 118. Earlier meta‐analyses reported heterogeneous results for the prevention of any relapse by lithium (SMD: 1.12, 0.47) ¹⁴⁰ .

For the prevention of depressive episodes, quetiapine and asenapine achieved medium effect sizes (RR: 2.08, 2.60). For all other drugs, including lithium (RR=1.26), effect sizes were small ¹¹⁴ . Small effect sizes were found for antidepressants (RR=1.56) ¹¹⁷ .

Psychotherapies vs. pharmacotherapies

Head‐to‐head comparisons of psychotherapies vs. pharmacotherapies yielded small effect sizes for all disorders examined, i.e., depressive disorders, anxiety disorders, PTSD and OCD (SMDs: 0.00‐0.24, see Figure 3)37, 66, 74, 126. A second‐order random effects meta‐analysis across the effect sizes of the largest meta‐analyses (Figure 3) yielded a non‐significant SMD of 0.11 (95% CI: –0.05 to 0.26, I²=61.99).

Figure 3.

Effect sizes in the largest meta‐analyses for head‐to‐head comparisons of psychotherapies (PSY) vs. pharmacotherapies (PHA). SMD – standardized mean difference, ° – adjusted for small‐study effects, MDD – major depressive disorder, SAD – social anxiety disorder, OCD – obsessive‐compulsive disorder, PTSD – post‐traumatic stress disorder, H – high, M – medium, L – low, U – uncertain, NR – not reported

Considering all included meta‐analyses, no substantial differences in short‐term efficacy between psychotherapies and pharmacotherapies in depressive disorders, anxiety disorders and PTSD were found12, 37, 66, 122, 123, 124, 125, 126, with only a few exceptions. In OCD, psychotherapy achieved medium to large SMDs (0.61‐0.95) in comparison to SSRIs ⁷² , but most psychotherapy trials included patients taking stable doses of antidepressants, affecting results in favour of psychotherapy. Most studies of psychotherapy and pharmacotherapy in OCD had a high risk of bias ⁷² . With regard to long‐term efficacy, psychotherapy achieved a large SMD compared to pharmacotherapy in PTSD (0.83) ¹²⁶ . For other disorders, no head‐to‐head comparisons fulfilled the inclusion criteria.

Combining psychotherapy and pharmacotherapy

In the largest meta‐analyses (Figure 4), effect sizes (SMDs) in favour of the combined treatment were small for most disorders, that is depressive disorders (0.37, 0.15) ³⁷ , social anxiety disorder (combined vs. pharmacotherapy: 0.40) ⁶⁶ , OCD (combined vs. psychotherapy: 0.25) and PTSD (0.09, 0.12) ¹²⁶ . Effect sizes (SMDs) were medium in favour of the combined treatment vs. psychotherapy in social anxiety disorder (0.52) ⁶⁶ and for the combined treatment vs. pharmacotherapy (SSRIs) in OCD (0.73), based on a network meta‐analysis including only one direct comparison with very small samples ⁷² . A large effect size was found only for the combined treatment vs. pharmacotherapy in ADHD (0.80) ¹³⁴ , based on only two RCTs showing a high risk of bias in at least one domain.

Figure 4.

Effect sizes in the largest meta‐analyses for combined therapy vs. pharmacological (squares) or psychological (circles) monotherapy. SMD – standardized mean difference, ° – adjusted for small‐study effects, COM – combined therapy, PHA – pharmacotherapy, PSY – psychotherapy, MONO – monotherapy, MDD – major depressive disorder, SAD – social anxiety disorder, OCD – obsessive‐compulsive disorder, PTSD – post‐traumatic stress disorder, ADHD – attention‐deficit/hyperactivity disorder, H – high, M – medium, L – low, U – uncertain, NR – not reported

A second‐order random effects meta‐analysis across the effect sizes of the largest meta‐analyses yielded a statistically significant but small SMD of 0.31 (95% CI: 0.19‐0.44, I²=53.02) in favour of the combined treatment (Figure 4).

Considering all included meta‐analyses, most effect sizes (SMDs) achieved by the combined treatment compared to either monotherapy in depressive disorders, anxiety disorders, PTSD, OCD and ADHD were small (0.09‐0.48) when risk of bias was taken into account12, 37, 66, 72, 74, 126, 128, 129, 131, 132, 134. Exceptions were the superiority of the combined treatment in long‐term outcome of PTSD over pharmacotherapy (SMD=0.96, based on only two direct comparisons) ¹²⁶ , and the superiority of the combined treatment over psychodynamic therapy in social anxiety disorder (SMD=0.68), based on a network meta‐analysis including zero direct comparisons for the condition ⁶⁶ , making a study of inconsistencies impossible ¹⁴¹ .

In several of these meta‐analyses, risk of bias was high in several domains, or results were based on only a few or small studies66, 72, 126, 134.

DISCUSSION

In this field‐wide assessment of psychotherapies and pharmacotherapies for mental disorders in adults, we included evidence from 102 meta‐analyses with 3,782 RCTs and 650,514 patients. We found small benefits overall for both types of interventions, with an average SMD of 0.35 and moderate heterogeneity across conditions ¹⁴² . This finding challenges the result of the previous most comprehensive review, which reported an overall medium effect size (SMD=0.50) across psychotherapies and pharmacotherapies, based on 61 meta‐analyses with 852 RCTs and 137,126 patients ⁸ . This latter estimate seems to be due to including waiting list comparators and averaging effect sizes without performing a random effects meta‐analytic evaluation.

According to the results of this umbrella review and second‐order meta‐analyses, there is an additional gain of psychotherapy and pharmacotherapy in the treatment of mental disorders in adults, but this is small in terms of effect sizes ²⁶ . Conditions for which very extensive evidence was available (e.g., depression) almost always had such modest effect sizes when only studies with low risk of bias were considered, or efforts were made to correct for bias. Medium or large effect sizes were found only for few conditions, and most of the effects sizes ≥0.50 were associated with a high risk of bias and/or limited evidence. Nevertheless, the argument still holds that, although there are some medications for general medical conditions with clearly higher effect sizes, psychotropic agents or psychotherapies are not generally less efficacious than those medications ¹⁴⁰ .

Some limitations and features of this umbrella review should be discussed as they affect the interpretation of overall evidence. First, several meta‐analytic comparisons included only a few studies, affecting statistical power and external validity of results.

Second, the results of network meta‐analyses need some extra caution141, 143. It has been argued that these meta‐analyses can only provide observational evidence, since the comparisons between treatments are both direct and indirect, and the latter are not randomized ¹⁴⁴ . As a related issue, transitivity (similar distribution of effect modifiers) can be controlled statistically only for known modifiers, in contrast to controlling all modifiers by randomization. Some of the network meta‐analyses included in this review encompassed only a few or even no direct comparisons of specific treatments66, 72, 75. Statistical power may be low if only a few studies with small samples and large heterogeneity are included141, 145. Thus, some inconsistencies between direct and indirect comparisons may not have been detected ¹⁴⁵ , possibly affecting effect size estimates.

Third, we followed Cohen's convention of small, medium and large effect sizes ²⁶ . However, the clinical relevance of these estimates is not clear. The clinical benefit of an intervention needs to be determined by comparison with a benchmark such as the minimal clinically important difference (MCID) ¹⁴⁶ . For the HAM‐D, for example, a minimal clinically relevant improvement has been claimed by some authors to correspond to a 7‐point difference ¹⁴⁷ or to an SMD of 0.88 ¹⁴⁸ . If this is correct, in psychotherapies or pharmacotherapies for depression, effect sizes of 0.30, 0.40 or even 0.50 correspond to a difference on the HAM‐D of 2 or 4 points (i.e. <7) which cannot be detected by clinicians and can therefore hardly be regarded as clinically significant. In schizophrenia, a SMD of 0.73 is required for a minimal clinical improvement of 15 points on the Positive and Negative Syndrome Scale (PANSS) ¹⁴⁹ , implying that SMDs below 0.73 are not detectable by clinicians and may not be clinically significant.

For a better judgment, the CIs of the effect size estimates may be compared with the proposed MCID values ¹⁵⁰ . It has been argued that, if the upper limit of the 95% CIs is smaller than the MCID, effect sizes can be regarded as “definitely clinically not important” ¹⁵⁰ . For the vast majority of meta‐analyses on depression and schizophrenia, this would be the case if SMDs of 0.88 and 0.73 are used as MCID. However, even if the summary effect sizes are substantially smaller than the MCID, there is heterogeneity in treatment responses across patients. Therefore, a minority of patients may still achieve large benefits from treatment.

Fourth, identical effect sizes may have different clinical importance in different patient populations (e.g., according to disorders, gender or age) and for different outcomes (e.g., mortality) ¹⁵¹ . For outcomes including vital events (e.g., rates of suicide) small differences in success rates may be clinically important, whereas for continuous measures of (often transient) depression, anxiety or other symptoms, small differences of a few scale points may not ¹⁵² . Of the meta‐analyses on the treatment of depression, for example, only a few examined hard outcomes such as suicidal behaviour41, 42, 44. In the meta‐analyses on schizophrenia and bipolar disorder, data on suicidal behaviour were not reported, except for one meta‐analysis ¹¹⁸ . Future studies and meta‐analyses should include such important hard outcomes.

Fifth, TAU as a comparator was found to be superior to no treatment in depression (with regard to remission, 33% vs. 23%) ¹³⁵ . However, TAU is a heterogeneous condition, and effect sizes achieved depend on the type of treatment actually delivered. Larger effect sizes may be achieved in comparison to weaker forms of TAU23, 153. This applies to psychological placebo as well ¹⁵⁴ .

Sixth, the results of RCTs may not necessarily represent real‐world effectiveness ¹⁵⁵ . In clinical practice, patients often suffer from multiple mental disorders, and treatments are usually tailored to the individual patients' needs. This applies, for example, to treatment duration. Most of the treatments included in this review were short‐term ⁶ . Data on longer‐term treatments are widely missing from RCT research.

In summary, a systematic re‐assessment of recent evidence across multiple meta‐analyses on key mental disorders provided an overarching picture of limited additional gain for both psychotherapies and pharmacotherapies over placebo or TAU. A ceiling seems to have been reached with response rates ≤50% and most SMDs not exceeding 0.30‐0.40. Thus, after more than half a century of research, thousands of RCTs and millions of invested funds, the “trillion‐dollar brain drain” ² associated with mental disorders is presently not sufficiently addressed by the available treatments. This should not be seen as a nihilistic or dismissive conclusion, since undoubtedly some patients do benefit from the available treatments. However, realistically facing the situation is a prerequisite for improvement. Pretending that everything is fine will not move the field forward ¹⁵⁶ , nor will conforming and producing more similar findings ¹⁵⁷ .

A paradigm shift in research seems to be required to achieve further progress. Suggestions for such a shift have recently been made ¹¹ , for example, for improving methodological quality and replicability (e.g., open science158, 159), improving available treatments – e.g., by personalized management160, 161, 162, defining specific targets and outcomes ¹⁶³ , considering response to previous treatments (staging)164, 165, switching or augmentation strategies ¹⁶⁶ – or developing new treatments (e.g., exploration of out‐of‐the box ideas and accidental discoveries ¹⁶⁷ ). A focus on prevention (e.g., in educational or occupational settings)168, 169 may improve the situation as well.

Improving treatment strategies for mental disorders can be regarded as a central health challenge of the 21st century.

ACKNOWLEDGEMENTS

The authors are grateful to M. Huhn for providing effect sizes for the pharmacological treatment of schizophrenia. They thank J. Matzat, an official patient representative of the Federal Joint Committee in Germany, for approving the paper. They are also indebted to M. Wolf for his support in producing the figures, to S. Uysal for his help in the evaluation of the quality of the included meta‐analyses and the review of the extracted data, and to B. Leowald and L. Feix for support in searching for meta‐analyses. The paper is dedicated to the late H. Kächele, a pioneer of psychotherapy research. Supplementary information on the study is available at https://osf.io/yvg2c/.