Concerns about ‘A comparison of electronically-delivered and face to face cognitive behavioural therapies in depressive disorders: A systematic review and meta-analysis’

Recently, a meta-analysis found that electronically delivered therapist-supported cognitive behavioural therapy (eCBT) was more effective than face-to-face cognitive behavioural therapy (ftfCBT) in reducing depression symptom severity, with a very large standardized mean difference (SMD) of 1.73.¹ Considering that SMDs above 0.80 are usually considered large,² that a meta-analytic comparison of CBT with passive controls resulted in SMDs around 0.50 to 0.70,³ and that other studies reported differing results,^4,5 we were surprised to see such an extreme effect. Our concerns were strengthened further by an addendum stating that the direction of the effect was misinterpreted in the original publication.⁶ Thus, we decided to reproduce the primary analysis of the study using data published in a corrigendum.⁷

Two of us have independently examined the included trials to extract relevant data. We were unable to reproduce some of the extracted means and standard deviations (the data required to calculate SMDs), with notable disagreement regarding around ten percent of the data points (Supplement Table S1).

Meta-analysis uses individual study estimates of the effect of interest (effect size) together with the standard error of the estimates to obtain a pooled effect. An essential feature of effect sizes is that they “provide a description of the size of observed effects that is independent of the possibly misleading influences of sample size”.⁸ We noticed that the authors' calculation of the effect size estimates for individual studies is different from the conventional calculation of SMDs⁹ in a critical way (Supplement Table S2). As they report in the corrigendum,⁷ instead of using standard deviations, they used standard errors in the denominator of the SMD formula. By doing so, they did not only make their effect size heavily dependent on sample size, but they also introduced considerable bias in comparison to conventional SMD, amounting to an overestimation by a factor of about ten in studies with two hundred participants (Supplement Table S3). Consequently, their effect size cannot be considered SMD. Furthermore, its statistical properties are completely unknown, including its sampling distribution and a way to calculate its standard error. Consequently, the results obtained with these calculations are uninterpretable. We do not have any reason to believe that the analysis of the secondary outcomes was substantially different (for example, for global functionality, a SMD point estimate of 36.28 was reported¹). Because individual study estimates are the basis for virtually all meta-analytic calculations and subsequent analyses, this carries over to the performed heterogeneity tests, subgroup analyses, tests for possible publication bias, sensitivity analyses, and so on.

Further confusion comes from the fact that the authors offer several, methodologically distinct and partly conflicting, interpretations of their original findings, including that eCBT was “more effective than” (i.e., superior to) ftfCBT, that eCBT was “at least as effective as” (i.e., non-inferior to) ftfCBT, and that eCBT was “as effective as” (i.e., equivalent to) ftfCBT.¹ In their addendum, they report that actually ftfCBT was found superior to eCBT.⁶ Most surprisingly, even when they learned that their statistically significant primary result points exactly in the opposite direction of what they thought previously, they stated that “[t]he overall conclusions are essentially the same” nonetheless.⁶ Thus, the reported conclusions seem arbitrary. It is worth noting that our re-analysis using re-extracted data and conventional SMD as effect size but in every other aspect the same meta-analytic approach as applied by the authors does not clearly align with any of the authors’ interpretations. Our analysis showed no statistically significant superiority of one treatment over the other, which, however, should not be interpreted as evidence of equivalence¹⁰ (Fig. 1, Supplement Table S4).

Fig. 1.

Re-calculated meta-analysis with re-extracted data and conventional standardized mean difference as effect size. Note. eCBT, electronically delivered therapist-supported cognitive behavioural therapy; ftfCBT, face-to-face cognitive behavioural therapy; SD, standard deviation; IV, inverse variance; CI, confidence interval; Total, number of participants. Descriptive group statistics refer to change from baseline in depressive symptom severity, with positive values indicating decrease in symptom severity. The standardized mean difference is pooled so that positive values indicate superiority of electronically delivered therapist-supported cognitive behavioural therapy and negative values indicate superiority of face-to-face cognitive behavioural therapy. This analysis identified considerable statistical heterogeneity among the included trials and showed no statistically significant superiority of one treatment over the other. Importantly, these findings should not be interpreted as evidence of equivalence.¹⁰

In summary, reproducibility of the data extraction was limited, the formulas used in the analyses were problematic and led to statistically biased and uninterpretable results, and the authors’ conclusions seemed largely disconnected from the data.

Contributors

LK conceptualized and designed the work, contributed substantially to the acquisition of the data, performed the analyses, interpreted the findings, and wrote the first draft. SL contributed substantially to the interpretation of the findings and revised the work critically for important intellectual content. MK contributed substantially to the conception and design of the work, acquired the data, contributed substantially to the analyses and the interpretation of the findings, and revised the work critically for important intellectual content. LK and MK have directly accessed and verified the underlying data reported in the manuscript. All authors had full access to all the data in the study, approved the final version of the work, and agree to be accountable for all aspects of the work.

Data sharing statement

All data that underlie the results reported in this work are included in the Supplement.

Declaration of interests

None of the authors reports a conflict of interest related to the content of the work.