A systematic review of network meta-analyses for pharmacological treatment of common mental disorders

Abstract

Question

Network meta-analyses (NMAs) of treatment efficacy across different pharmacological treatments help inform clinical decision-making, but their methodological quality may vary a lot depending also on the quality of the included primary studies. We therefore conducted a systematic review of NMAs of pharmacological treatment for common mental disorders in order to assess the methodological quality of these NMAs, and to relate study characteristics to the rankings of efficacy and tolerability.

Study selection and analysis

We searched three databases for NMAs of pharmacological treatment used in major depression, generalised anxiety disorder (GAD), social anxiety disorder (SAD), post-traumatic stress disorder (PTSD), obsessive-compulsive disorder (OCD) and specific phobia.

Studies were appraised using the International Society for Pharmacoeconomics and Outcomes Research checklist of good research practices for indirect-treatment-comparison and network-meta-analysis studies.

Findings

Twenty NMAs were eligible for inclusion. The number of randomised controlled trials per NMA ranged from 11 to 234, and included between 801 to more than 26 000 participants. Overall, antidepressants were found to be efficacious and tolerable agents for several disorders based on rankings (45%) or statistical significance (55%). The majority of NMAs in this review adhered to guidelines by including a network diagram (70%), assessing consistency (75%), making use of a random effects model (75%), providing information on the model used to fit the data (75%) and adjusting for covariates (75%).

Conclusions

The 20 NMAs of depression and anxiety disorders, PTSD and/or OCD included in this review demonstrate some methodological strengths in comparison with the larger body of published NMAs for medical disorders, support current treatment guidelines and help inform clinical decision-making.

Background

Worldwide, psychiatric disorders have become a priority and are the leading cause of disability.¹ Globally, depression and anxiety disorders account for the fifth highest burden of disease based on disability-adjusted life years.¹ According to one recent meta-analysis of the epidemiology of common mental disorders (major depression, generalised anxiety disorder (GAD), panic disorder (PD), obsessive-compulsive disorder (OCD), post-traumatic stress disorder (PTSD), social anxiety disorder (SAD) and specific phobia) these disorders occur globally in 29.2% of people across their lifetime.² For many disorders, a range of different treatment options are available.³ Given their efficacy and tolerability,^4–6 antidepressants are recommended for these conditions by National Institute for Health and Care Excellence guidelines^{7 8} and WHO guidelines.⁹

Standard pairwise meta-analyses have provided clinicians with an overview of medication efficacy and tolerability through the quantitative synthesis of data on treatment effect size estimates from randomised controlled trials (RCTs).⁹ Nonetheless, standard pairwise meta-analyses can only be used to draw strong conclusions about interventions that have been directly compared with one another, and are not designed to support comparisons between all potential interventions at clinicians’  disposal.^{10 11} Network meta-analysis (NMA) methods combine direct head-to-head comparisons with indirect comparisons between all interventions in a network of clinical trials,^{10 12 13} and may be particularly relevant when competing interventions are available.^{11 14}

Recognition of the potential utility of NMAs has been signalled by a rapid recent increase in the number of publications using this.^15–20 Caution is advised in interpreting NMAs, however, as their validity depends on methodological assumptions such transitivity (ie, the distribution between the effect modifiers is similar across treatment comparisons) and consistency (ie, indirect and direct evidence is in agreement).¹¹ Previous systematic reviews of NMAs across medical conditions have documented that violations of these assumptions are common and may arise for a number of reasons.^{11 21 22} For instance, although transitivity was mentioned in 33% of the 353 NMAs included in Petropoulou 2017 overall, this proportion increased over time, such that 77% of the networks published in 2015 discussed transitivity.²²

Failure to consider these factors would, to a large extent, undermine the utility of NMAs in informing comparative efficacy choices.¹⁵ Indeed, NMAs have been criticised for generating false-positive results,^{15 23} and leading to conclusions about the superiority of particular medications that are unwarranted, as current evidence may not support the choice of one second-generation antidepressant over another in terms of differences in efficacy and effectiveness.^{23 24}

Objective

In light of these considerations, we decided to (A) conduct a systematic review of published NMAs that have assessed the efficacy of pharmacological treatment for common mental disorders, (B) review the quality of the methods of the published NMAs using a quality rating approach that has been designed specifically for NMAs and (C) discuss differences in rankings of the efficacy and tolerability of pharmacological treatment in terms of methodological and clinical characteristics of the NMAs by disorder.

Study selection and analysis

Eligibility criteria

NMAs were considered eligible for inclusion in this review if they included RCTs of pharmacotherapy in treating adult participants (18–65 years) with common mental disorders (depression, GAD, PD, OCD, PTSD, SAD and specific phobia) diagnosed according to the Diagnostic Statistical Manual for Mental Disorders (DSM-III and later) or the International Classification of Diseases ((ICD-10). NMAs containing RCTs that included participants with comorbid secondary mental disorders were also included. Participants with substance use disorders were excluded. Studies were not restricted by language, publication date or setting.

Search strategy

NMAs of medication for the treatment of common mental disorders in adults published up until 22 March 2017 were identified by searching Scopus, PubMed Central and the Cochrane Library. Parallel search strategies were employed, including a general strategy using broad search terms incorporating pharmacological classes, as well as a more specific query incorporating generic medication names. Both search queries included the following terms: adults, common mental disorders, pharmacotherapy, network meta-analysis and the abbreviation NMA.

Study selection and data extraction

Two authors (TW and JI) assessed the relevance of each study first by title and abstract, followed by the retrieval of full-text articles that passed the initial screen for further inspection. General descriptive information was extracted (see online supplementary table 1) with specific emphasis placed on the methodological quality of each published NMA^{22 25 26}.

We also extracted data (where provided) from rankograms or cumulative ranking probability plots (the surface under the cumulative ranking curve),¹¹ indicating the best treatment, in terms of both efficacy and proportion of patients who left the study early (as a proxy measure of treatment tolerability and acceptability).

Appraisal of reporting standards and methodological quality of included NMAs

The extent to which the included studies complied with recommended standards for reporting NMA methodology was assessed by applying the checklist of good research practices from the International Society for Pharmacoeconomics and Outcomes Research guidance document, an instrument that was specifically designed for the purpose of evaluating the quality of NMAs.¹⁵ In addition, we modelled our approach on that of Chambers et al ²⁵ and the updated review by Zarin et al ²⁶ and Petropoulou et al (2017) who assessed the following criteria: study method, study transparency and reproducibility, and the presentation of study findings^{22 25 26}.

Findings

Description of search

Four hundred and thirty-five studies were found across the three databases searched. Of these studies, 135 were duplicates. The abstracts for the remaining 300 studies were scanned for eligibility, of which 259 studies were excluded for a variety of reasons (see the selection flow chart, figure 1). Full-text articles were subsequently retrieved for further assessment of the 41 studies that passed the initial screening phase. After independent reviewing, 21 failed to meet inclusion criteria, leaving 20 eligible for inclusion. Each NMA study included published and/or unpublished RCTs (with the inclusion of cross-over studies, quasi experimental designs and/or open label studies, where study designs were reported) (see figure 1).

Figure 1.

Flow chart of NMAs included in the systematic review. NMA, network meta-analysis; RCT, randomised controlled trial.

Description of the included NMAs

NMAs investigating treatment of depressive disorders (ie, major depressive disorder) represent 70% (number of studies (N)=14) of the included studies.16 18 24 27–37 Two of the 14 NMAs reported an additional primary diagnosis of Parkinson’s disease³¹ and sexual dysfunction,³³ and two NMAs included patients with drug-resistant depression.^{35 3} Of the remaining six NMAs, two investigated the treatment of GAD,^{19 38} two SAD,^{17 39} one PTSD²⁰ and one OCD.⁴⁰ Diagnostic criteria (DSM defined, n=12; ICD 10, n=4) was based on standardised measures and/or research diagnostic criteria. Recruitment was conducted in inpatient and/or outpatient settings across health technology and WHO regions (n=10, for example, the UK, Australia and Canada) with regards to the clinical effectiveness, safety and cost-effectiveness of interventions employed. The social, ethical and legal aspects of these technologies were also assessed.⁴¹

Across all 20 included NMAs, the number of RCTs per NMA ranged from 11³¹ to 234,²⁴ and sample size from 801³¹ to more than 26 000^{16 33} participants. Year of publication for the 20 NMAs ranged from 2008³⁹ to 2017³⁶ (also see figure 2). Only 1 of the 20 included NMAs reported funding by industry.¹⁹ The mean journal impact factor (based on 2015/2016 ResearchGate ratings) across the 19 peer reviewed and published NMAs was 3.11 (SD: 1.32). The remaining NMA was published as a report.²⁰

Figure 2.

Descriptive characteristics of year of publication by disorder. GAD, generalised anxiety disorder; MDD, major depressive disorder; NMAs, network meta-analysis; OCD, obsessive compulsive disorder; PTSD, post-traumatic stress disorder; SAD, social anxiety disorder.

Studies assessed a range of medications administered according to either fixed or flexible doses. The most common agents were antidepressants (eg, selective serotonin reuptake inhibitors, serotonin and norepinephrine reuptake inhibitors and tricyclic antidepressants, including immediate release and/or extended release). The individual RCTs reported by these NMAs were conducted over a period ranging from 2 weeks,^{17 35} to more than 12 months.¹⁶ Fourteen of the included NMAs included a placebo group as a comparator. The remaining 30% of the included NMAs did not include a placebo comparator.16 24 27 32 36 40

Figure 3.

Descriptive characteristics of the number of NMAs by disorder and medication class. 5-HT2, receptor antagonists; CNS, central nervous system; GAD, generalised anxiety disorder; MAOIs, monoamine oxidase inhibitors; MDD,  major depressive disorder; NARI, norepinephrine reuptake inhibitor; NASSAs, noradrenergic and specific serotonergic antidepressants; NDRI, norepinephrine and dopamine reuptake inhibitor; NMAs, network meta-analysis; OCD, obsessive compulsive disorder; PTSD, post-traumatic stress disorder; RIMAs, reversible inhibitors of monoamine oxidase A; SAD, social anxiety disorder; SARIs, serotonin and reuptake inhibitors; SNRIs, serotonin and norepinephrine reuptake inhibitors; SSRE, selective serotonin reputake enhancer; SSRI, selective serotonin reputake inhibitor; TCA, tricyclic antidepressants; TecA; tetracyclic antidepressant.

The outcomes assessed across the 20 NMAs varied from the assessment of treatment response in 14 NMAs (ie, treatment efficacy),17–19 24 27 29–34 36 38 39 dropouts due to any cause (n=5) and/or dropouts due to side effects (n=12).18–20 27 31 33–35 37 38 40 Standardised and self-report measures were used to assess these outcomes and were specific to the different disorders. Additional outcomes were symptom severity, remission and relapse. Probability rankings were provided for the outcome of cost-effectiveness for three NMAs and reference 38.^{16 32}

Assessment of standard of reporting of the included NMAs

Study method

Fifteen of the 20 NMAs employed a Bayesian framework (using Markov Chain Monte Carlo estimation methods),17 19 20 24 27 28 30–36 38 40 with a Frequentist approach being employed in addition in two of the NMAs.^{17 36} Both direct and indirect estimates were calculated using a random effects model for each of the 15 NMAs, with a fixed effects model also employed by 1 NMA.³⁶ Risk of bias (RoB) and quality assessment was assessed for 15 NMAs, based on a variety of instruments, including the Cochrane RoB tool and the Grading of Recommendations Assessment, Development and Evaluation approach.17 19 20 24 27–31 33 35–38 40 Three of the 15 NMAs did not report the findings for RoB assessment, however.^{31 35 38} Overall the RCTs included were rated unclear or high for RoB. Publication bias was reported by four NMAs.29 35 37 39

Fifteen NMAs reported adjusting for covariates,17 18 20 24 27–35 38 40 and provided information about the model used to fit the data.17 18 20 24 27–35 38 40 Additional sensitivity analyses were reported by 13 NMAs.16 18 20 24 27–30 32 33 35 38 40 Fifteen NMAs reported that the assessment of consistency would be calculated across comparisons, however, the findings were only reported by 13 of these studies.17 19 27 28 31–33 35–40 In addition, 2 of the 13 NMAs by Gartlehner et al (2011) and Linde et al ¹⁸ reported the use and method of node splitting across direct and indirect evidence as a method of assessing inconsistency with a Z-test.^{10 18 24} An assessment of heterogeneity in the effect sizes reported across studies was obtained in 13 NMAs by calculating the I squared statistic (I²), the χ² statistic (χ²), the Q statistic and/or the Gelman-Rubin statistic.17 19 20 24 27 28 31–33 35 37 39 40 Four NMAs reported the τ statistic to explain heterogeneity in the effect estimates.17 18 38 40

Thirteen NMAs reported testing of transitivity assumptions via the comparison of effect modifiers across studies, including dosage,27 31 32 34 treatment duration,^{28 32 34} sample size,^{28 31} drug-placebo comparisons,^{18 28} recruitment settings^{18 32 36} and baseline symptom severity and similarity.^{29 34 39}

Transparency and reproducibility

The majority of the NMAs documented which databases were used (95%),16 18–20 24 27–40 as well as the specific search terms that were used to identify trials (80%),17–20 24 27–33 35 38–40 and the date of the last search (100%). Twelve NMAs provided additional search strategy queries as supplementary information.17 19 20 24 27 30 33 35–38 40 All of the NMAs extracted data from contributing clinical studies and 95% of the NMAs provided a study characteristic table. Only 3 of the 15 NMAs that used a Bayesian framework to calculate rankings provided the model code that they used to conduct their analysis.^{17 38 40}

Presentation of study findings

Half of the NMAs included in this review used standard network graphs and plots to visually represent their data (50%).17 18 20 28–31 33 35–37 The most common figure reported was a flow diagram that provided information regarding the eligibility of the RCTs and number of RCTs included in the network. In addition, 14 NMAs displayed their results with a forest plot indicating effect estimates across comparisons and outcomes.17 18 20 24 28–37 39 Ten NMAs provided a full matrix of head-to-head comparisons (for direct and indirect comparisons).17–19 27–29 31 32 35 37 Nine NMAs employing a Bayesian (n=7) or Frequentist framework (n=2) reported the probability that particular agents were the most effective treatment and ranked treatments accordingly as best, second best, third best and so on.16 19 27 31 32 35 36 38 40 Based on the results for rankings (n=9) and statistical significance (n=20) for the included NMAs, antidepressants were often (55%, 11/20 NMAs) rated as the most efficacious and/or tolerable treatment across disorders.

Conclusions

Twenty NMAs investigating pharmacological treatment for depression and anxiety disorders, PTSD and/or OCD were included in the review. Antidepressants were rated as the most effective and/or tolerable agents in 11 (55%) of the NMAs, as assessed either by rankings or by overall statistical significance. The remaining studies reported rankings or overall statistical significance for 5HT1A partial agonists,^{28 33} anticonvulsants,^{20 38} antipsychotics,^{28 35 36} dopamine antagonists,³¹ acetyl-l-carnitine,²⁸ the antimanic agent lithium,³⁵ the noradrenergic and specific serotonergic antidepressant mirtazapine²⁷ and the thyroid hormone.³⁵

The potential clinical utility of NMAs for evaluating the relative efficacy and tolerability of different classes of antidepressants is somewhat undermined by evidence for methodological quality across the 20 NMAs in the review. More than a quarter of NMAs did not provide a network diagram, report RoB assessment or conduct sensitivity analyses. Fifteen NMAs reported the results of tests for consistency in effect estimates across trials, and 13 NMAs evaluated transitivity. Only four NMAs assessed and reported publication bias, a bias that may impact the overall estimate of effects and ranking of medications.⁴² Moreover, the sample size and number of RCTs for the NMAs of depression were larger than the remaining NMAs. On a more positive note, more than 70% of the NMAs included in the review accounted for the influence of variability of covariates on effect estimates (eg, meta-regression or logistic regression), made an assessment of model fit and extracted data from clinical studies where they provided a table of study characteristics.

The fact that only three NMAs provided their model code may reflect convergence on standard software routines for this purpose, or that the model code has become easily accessible and freely available, as previously noted for Bayesian frameworks.²⁵ The majority of the NMAs did not take full advantage of reporting tools designed to convey design-specific considerations (eg, inconsistency or ranking plots). Moreover, only 10 NMAs provided a full matrix of effect estimates for head-to-head comparisons (for direct and indirect comparisons), and even fewer (n=9) reported treatment rankings. Lack of presentation and utilisation of visual graphics provided by these NMAs lends weight to the critique that NMAs are complex and mainly used by researchers with strong statistical skills.¹³ Failure to report rankings may partly reflect the concern that positions in these ranks are sensitive to small and clinically non-significant differences in reported treatment effects.^{43 44}

Nevertheless, the NMAs included in this review performed favourably with respect to seven aspects of quality, compared with other recent, and more inclusive, systematic surveys of published NMAs for medical disorders.^{22 25 26} Compared with Chambers et al (2015), Zarin et al ²⁶ and Petropoulou et al (2017) the NMAs included in this review more frequently provided a network diagram of both direct and indirect comparisons (70% compared with 61%, 48%, 26%, respectively); assessed consistency (75% compared with 69%, 53%, 30%, respectively) and made use of a random effects model (75% compared with 70%, 49%, 74%, respectively).^{22 25 26} A higher proportion of NMAs in this review also modelled the data (75% vs 40% and 48%) and adjusted effect estimates for covariates (75% vs 29% and 18%), compared with Chambers et al ²⁵ and Petropoulou et al (2017), respectively.^{22 25} The heterogeneity assumption was explored in 65% of the NMAs reported on in this paper, compared with 56% for Zarin et al ²⁶ and Petropoulou et al (2017).^{22 26} Finally, more than three quarters (77%) of the NMAs included in Petropoulou et al (2017) did not discuss transitivity, compared with 65% in our review.²²

The relatively good standing of the NMAs in our review may partly reflect their relatively recent publication, consistent with Petropoulou et al’s (2017) finding that more recently published NMAs adhere to more rigorous methodological standards.²² For instance, Petropoulou et al (2017) reported an increase in the number of NMAs discussing transitivity and inconsistency from 0% to 86% when comparing NMAs published in 2005 with those published in 2015.²² In addition, although the NMAs in our review performed favourably compared with those assessed in studies of medical disorders with respect to multiple methodological features, optimal methods were not always employed, with the evaluation of consistency being a particular case in point. Nonetheless, the NMAs included in this review show strong methodological quality and relatively favourable adherence to reporting for the treatment of common mental disorders, factors that support their replication across the clinical spectrum.

Clinical implications

The 20 NMAs of depression and anxiety disorders, PTSD and/or OCD included in this review reflect the growing evidence base of trials on the pharmacological treatment for these disorders, support current treatment guidelines and help inform clinical decision-making.⁴³ The included NMAs in this review demonstrated superiority with respect to a number of aspects of methodological quality than recent surveys of NMAs published across medical disorders; we have relatively high level of confidence in the findings of the NMAs included in our review. Nevertheless, studies employing NMA methods going forward may gain from addressing some of the shortcomings identified in this review.