Overoptimistic Literature and Methodological Biases Favoring Cognitive Behavioral Therapy for the Prevention of Psychosis

Indicated prevention in young people at Clinical High Risk for Psychosis (CHR-P) originated in Australia more than 20 years ago¹ and subsequently impacted national and international clinical guidelines² and diagnostic manuals.³ While the most recent umbrella reviews (reviews of meta-analyses) demonstrated substantial achievements in detection and prognostic assessment of young CHR-P individuals,⁴ the most updated network meta-analysis found no robust evidence to favor cognitive behavioral therapy (CBT) compared with the control condition (ie, needs-based interventions).⁵ An independent pairwise meta-analysis by the Cochrane group corroborated these findings, concluding “there was no convincing unbiased, high-quality evidence” that any type of intervention is more effective than needs-based interventions⁶ (two other pairwise meta-analyses were published^7,8 but either used older data⁷ than the network and Cochrane meta-analyses or were discussed elsewhere, with significant concerns in regard to study inclusion^8,9). Overall, these studies cautioned that uncertainty of evidence is high for CBT in preventing psychosis among CHR-P individuals.^5,6,10

In contrast to these cautionary evidence-based findings, a recent pairwise meta-analysis included 10 CBT trials and identified robust evidence that “cognitive behavioural therapy is effective in reducing both psychosis transition rates and attenuated psychotic symptoms” over more than 2-year follow-up.¹¹ However, this meta-analysis appears to have major methodological problems as well as explicit errors.

First, the authors of this meta-analysis stated that an update was necessary because of the “low-quality of evidence” ¹¹ of published trials included in the latest network/Cochrane meta-analyses,^5,6 and state robust trials would require at least 300 CHR-P individuals.¹¹ No high-quality, large-scale, randomized controlled trials of CBT have been published since the network/Cochrane meta-analyses^5,6—rather, only three new CBT trials have emerged, one from Italy and two from China.¹¹ The latter two did not acknowledge registering the trial protocol and admitted utilizing “low-intensity CBT” ¹¹ (fewer than the minimum 16 sessions recommended by the NICE QS80¹²) as well as specific contents of their CBT protocol “were not clear.” ¹¹ Accordingly, these two trials were rated as at high risk of bias within the meta-analysis.¹¹ Furthermore, although these studies enrolled first-degree relatives of schizophrenia patients, they did not clarify whether all trial participants presented with functional impairment, which is necessary to meet the Genetic Risk and Deterioration subgroup of the CHR-P criteria beyond familial risk.^1,4 The other (small) Italian CBT trial (n = 58) by Pozza et al¹³ has several weaknesses relating to the measurement of outcomes, incorrect interpretation of Kaplan–Meier outputs, selective reporting, and failure to adhere to CONSORT guidance (eg, failure to register the trial).¹⁴ As a result, we suggested claims made by this study that CBT can prevent psychosis, should be tempered.¹⁴ The authors of the meta-analysis acknowledged that all 10 trials included were at high risk of bias or at unclear risk.¹¹ It is difficult to understand how strong conclusions relating to the effectiveness of CBT for prevention of psychosis could be reached with no trials free of this degree of bias.

Second, but of principal concern, is the presence of numerous data extraction errors, which may have increased the likelihood of the meta-analytic results being significant in favor of CBT. For example, the short-term (within 6-month) meta-analytic forest plot and analysis¹¹ omit the trial by Morrison et al (two transitions in the CBT and three in the control group at 6-month follow-up),¹⁵ which was included in previous meta-analyses,¹⁶ as well as the 6-month data from Pozza et al.¹³ Other examples include the medium-term (6–12 months) meta-analytic forest plot, which reports one transition in the CBT group for Pozza et al,¹¹ while the Kaplan–Meier curve in Pozza et al shows that there are at least three transitions in the CBT group at 12 months.¹³ Furthermore, the number of transitions in the CBT arm of McGorry included at 12 months (four) diverge from the number (seven) indicated by the authors of this trial in previous meta-analyses.¹⁷ The long-term (12–24 months) meta-analytic forest plot omits the follow-up data provided by Pozza et al at 61 weeks/14 months; these data are rather used in the ultra-long-term (more than 24 months) forest plot.¹¹ The transitions extracted from Bechdolf et al in the 6–12 and 12–24 months analyses do not reflect the primary outcome of the meta-analysis (ie, a transition from a CHR-P state to frank psychosis¹¹) but rather the progression from an “early initial prodromal phase” of risk (in which symptoms, disability, and biological deficits are less severe than CHR-P) to any later stage, including subthreshold psychosis (ie, the CHR-P state itself); the data used therefore diverge from those employed by previous meta-analyses.¹⁷

Third, while the authors stated that their literature search was planned to include unpublished literature,¹¹ they failed to include the large CBT trial, PREVENT (n = 216, as did the other pairwise meta-analysis^8,9), which published baseline data a decade ago.¹⁸ While the final manuscript is yet to be published, preliminary findings showing no statistical significance for CBT in preventing psychosis were presented at a major international conference in 2016¹⁹ and included in previous meta-analyses.⁵ Publication biases were not formally investigated with funnel plots because of the small number of studies, and no other sensitivity analyses for publication biases (eg, trim and fill, fail-safe N) were attempted.¹¹ Consequently, this meta-analysis could not exclude that its findings may be affected by publication biases.¹¹

Fourth, a previous umbrella review demonstrated no evidence that CBT impacts other clinical outcomes such as acceptability of treatments, severity of attenuated positive/negative psychotic symptoms, depression, symptom-related distress, social functioning, general functioning, and quality of life.¹⁰ The authors of the meta-analysis confirmed all these findings but report a “robust” effect that CBT improves attenuated psychotic symptoms.¹¹ This statement conflicts with the small effect size approaching the non-significance level (standardized mean difference = −0.24, 95% CI: −0.43, −0.06), which is unlikely to make any appreciable difference to patients in practice.¹¹

Given these major methodological biases and explicit errors listed above, the analyses, results, and conclusions of this meta-analysis¹¹ should be corrected, and the record amended. While we caution against presenting meta-analyses without full and transparent reporting, to demonstrate the potential impact of these errors we have repeated the medium-term (6–12 months) meta-analysis after removing the biased studies (one Chinese study and Pozza et al¹³), adding PREVENT and amending the transitions: the updated risk ratio for CBT vs control interventions to prevent transition to psychosis at 12 months is 0.635 (95% CI: 0.391–1.029, P = .065, random effects model), which shows no significant meta-analytic evidence that CBT can robustly prevent transition to psychosis. Furthermore, to avoid multiple testing every time a new CBT trial is being published, meta-analyses that regularly update the evidence given an a priori sample size would be required in this field.¹⁰

Overall, we conclude that the lack of robust meta-analytic evidence to favor CBT to prevent psychosis, as appraised by the most recent network meta-analysis⁵ and the Cochrane meta-analysis,⁶ still stands. Transparent appraisal of limitations of knowledge is a prerequisite for any reliable scientific advancements. The lack of robust meta-analytic evidence to favor CBT aligns with its “black box” mechanism of action²⁰ and uncharted side effects,²¹ particularly in vulnerable minorities.²² Collegial and global initiatives combining individual participant data meta-analyses, experimental therapeutics, strategies to control risk enrichment, innovative youth mental health services, adaptive trial designs, stratification, and precision medicine approaches will hopefully deliver effective interventions to prevent psychosis in these individuals.¹⁰