Response to “Comparing the Effectiveness of EMDR and TF-CBT for Children and Adolescents: a Meta-Analysis

With interest we read the article by Lewey and colleagues describing the results of a meta-analysis, entitled “Comparing the Effectiveness of EMDR and TF-CBT for Children and Adolescents: A Meta-Analysis” (Lewey et al. 2018). In this article, the authors presented the available evidence on the effectiveness of Trauma Focused-Cognitive Behavioral Therapy (TF-CBT) and Eye Movement Desensitization and Reprocessing (EMDR) therapy for children and adolescents with symptoms of Posttraumatic Stress Disorder (PTSD). The authors included studies from 1989 until 2015 that applied TF-CBT and/or EMDR therapy in children and adolescents with full or subclinical PTSD, and compared these treatments to waiting list, head to head or other active treatment control conditions. Based on the results of this meta-analysis, the authors concluded that both TF-CBT and EMDR therapy were effective in treating PTSD, and that TF-CBT was marginally more effective than EMDR. We are highly concerned that the conclusions of the meta-analysis by Lewey et al. (2018) are based upon major methodological limitations and incorrect information. Considering the fact that results of meta-analyses have a great impact on the development of international treatment guidelines and mental health policies, we consider it our obligation to share our viewpoint with the readers of this journal.

The most important methodological flaw of the meta-analysis was the consistent reversal of the signs of the reported effect sizes of the EMDR conditions, thereby favouring the control condition (waiting list or another active condition) over EMDR therapy. In paragraph “overall effect size and heterogeneity analysis”, the authors defined the minus sign (−) of an effect size as indicating a decrease in measured symptoms following treatment. Accordingly, in the result section (moderator analyses) they concluded that samples with TF-CBT displayed the largest mean effect size of −.813, with a 95% confidence interval ranging from −1.077 to −.549, and that EMDR had a mean effect size of .959 with a confidence interval of .398 to 1.521, thus reflecting an increase of PTSD symptoms rather than a decrease. Also, in Fig. 1 (Lewey et al. 2018) it is shown that for most of the EMDR studies, the control condition outperformed the EMDR treatment. However, all original EMDR studies that included a comparison group reported a decrease of PTSD symptoms post treatment. For example, in the study of Kemp et al. (2010) 27 children (aged 6 to 12 years) suffering from PTSD symptoms after a motor vehicle accident were randomly assigned to EMDR or a waiting list control condition. The effect size displayed in Fig. 1 (Lewey et al. 2018) indicated that the waiting list was more effective compared to the EMDR group, while in fact the study results showed that EMDR was superior to the control condition. The same error appeared to have occurred in case of the studies conducted by Chemtob et al. (2002), Jaberghaderi et al. (2004) and De Roos et al. (2011). As a matter of fact, none of the included studies EMDR resulted in poorer outcomes compared to the control condition. Thus, it seems that the authors made the mistake of reversing plus and minus signs. If this error would be corrected it is likely that, possibly with a marginal difference, EMDR therapy studies yielded the largest mean effect size (−.959 for EMDR versus −.813 for TF-CBT), rather than the smallest effect-size. This is in sharp contrast to the current conclusion drawn by the authors.

Another vital concern pertains to a number of methodological and statistical issues with regard to the conducted meta-analysis. Firstly, the authors have performed three different analyses of publication bias checks to account for possible unpublished data, namely the Duval and Tweedie’s trim and fill procedure, the Classic fail-safe N, and the Orwin’s fail-safe N. Although these procedures in itself were correct, this did not solve the main issue that rigorous inclusion criteria were needed to make sure that included studies were of high quality and trials were comparable on potential effect modifiers to be able to obtain unbiased pool estimates. Since the authors did not rate the quality of the included studies, which is a common procedure in meta-analyses (Moher et al. 2009), this has likely resulted in the inclusion of studies that would otherwise not have met the inclusion criteria or would be given less weight in the statistical analyses. As a consequence, studies were included that did not report on basic characteristics like age and/or gender (4/30 studies) or did not report on both pre- and post-intervention means (10/30 studies), yet no statistical correction was applied to account for this.

Outcome measures were not specified per study. Instead, in Table 6 (Lewey et al. 2018) a wide variety of outcome measures is displayed of which some were not valid instruments for measuring post-traumatic stress symptoms (e.g., the Child Behaviour Checklist, CBCL). Also, it was not clear whether the data were pooled across reporter (i.e., self, parent). This is important because evaluations by parents and self-report of children for these types of symptoms have been found to show only modest agreement (Kassam-Adams et al. 2006; Stover et al. 2010). Since outcome measures and reporter were not specified per study, it cannot be judged whether an imbalanced distribution of these effect modifiers existed between TF-CBT and EMDR studies which may have further biased comparisons.

Another important point we want to make pertains to the classical Random Effects Model (REM) based upon calculated effect sizes per study. The comparison group was highly heterogeneous, whereas a substantial number of the included studies (11 out of 30) did not use a randomized design to allocate treatment. Given these conditions a network meta-analysis (NMA) or mixed treatment meta-analysis (MTM) is preferable (Caldwell 2014; Rouse et al. 2017; Tonin et al. 2017). The key feature of NMA is that it allows the synthesis of direct and indirect estimates for the relative effects of many competing treatments for the same health condition and that results from both randomized and non-randomized studies can be combined (Tonin et al. 2017). An important advantage of this approach is that networks can be created that account for the amount of direct evidence available in the literature and/or the volume of studies referring to each intervention. The latter is also crucial given that only N = 138/1192 (11.6%) included participants received EMDR as treatment condition in contrast to N = 1054/1192 (88.4%) receiving TF-CBT (for this calculation we followed the numbers of the treatment groups as displayed in Table 2 (Lewey et al. 2018).

Our final point of concern are a series of inconsistencies in the display of the treatment and comparison conditions. Firstly, the authors did not identify two direct comparison studies between trauma-focused CBT versus EMDR in their own selection for the meta-analysis, namely Jaberghaderi et al. (2004) and de Roos et al. (2011). Secondly, in the EMDR studies of Puffer et al. (1998) and Chemtob et al. (2002) the control condition is listed as OTHER, while in fact it concerned a delayed EMDR treatment group. Thirdly, the study of Oras et al. (2004) is erroneously displayed as a study in which EMDR is compared with another treatment (Table 2 and Fig. 1 (Lewey et al. 2018). This study did not include a comparison group, but EMDR therapy was incorporated within traditional psychodynamic therapy.

In summary, the conclusion of the authors that TF-CBT was marginally more effective than EMDR therapy cannot be justified based upon the studies that were reviewed given the major methodological and statistical flaws. Besides a range of inconsistencies, the most important reasons limiting the informative value of this study were the errors in calculating the effect sizes of the EMDR studies, the lack of risk of bias assessments for the included studies, the unspecified outcome parameters per study, and the suboptimal choice for the REM. What the meta-analysis does suggest is that, regarding the treatment of PTSD in children and adolescents, the effects of both CBT and EMDR are replicable and robust. However, more methodologically sound studies with large samples are needed to determine whether there are differences in the effectiveness of trauma therapies used for children and adolescents, such as TF-CBT and EMDR therapy.

Compliance with Ethical Standards

Conflict of Interest

Carlijn de Roos and Ad de Jongh receive income from a published book(s) about EMDR and for training postdoctoral professionals in EMDR. Nanda Rommelse, Rogier Donders, Rik Knipschild and Iva Bicanic have no conflict of interest.