Response to: “Updated and rectified meta-analysis shows no effect of propranolol versus placebo on traumatic memory reconsolidation disruption”

In their letter to the editor, Steenen and colleagues1 argue that the conclusions of our meta-analysis2 on the clinical efficacy of reconsolidation impairment using propranolol are “incorrect in the context of (...) psychotrauma-related symptomatology.” They claim that we did not assess risk of bias and critique our omission of 3 unpublished studies (Aikins,3 Saladin,4 and Orr5). We are not convinced by their attempt to rectify our analysis.

Although this is a matter of debate,6 we were asked during the JPN peer review of our manuscript to omit unpublished data from our meta-analysis because the methodological quality of such studies is difficult to verify. Indeed, we could not verify important methodological information with respect to the unpublished results (n = 6) of Aikins,3 such as the adequacy of blinding methods. Moreover, we verified methodological information in the study by Orr5 with the author himself and concluded that the data from this pre-emptively terminated (rather than unpublished) study with n = 5 randomized participants were not suitable for meta-analytic purposes. The inclusion of studies with very small samples in a meta-analysis (n ≤ 3 per group) is bound to compromise the precision of the overall effect estimate.6

We also could not verify important information such as the outcomes used to compute the effect size for Saladin’s unpublished study of reconsolidation interference in participants with posttraumatic stress disorder (PTSD) and comorbid alcohol dependence.4 Considering that Steenen and colleagues1 argue against including studies of addiction (but see Gisquet-Verrier and colleagues7), it is unclear why they included Saladin’s study in their analysis. But even more problematic, we found discrepancies across public forums in how Saladin reported the results of this unpublished study,4 casting reasonable doubts on the validity of the data used by Steenen and colleagues.1 We also could not replicate the large effect size estimate of d = −3.59 that Steenen and colleagues1 reported for the Saladin study,4 as it seems to have been erroneously computed using standard errors rather than standard deviations on a measure of subjective distress to trauma and alcohol cues (based on data available at https://clinicaltrials.gov/ct2/show/results/NCT01055171?term=propranolol+and+alcohol&draw=2&rank=2&view=results). Scrutiny indicates that we were justified to not include such unpublished studies.

Steenen and colleagues1 re-analyzed our data by excluding a number of studies that they deemed had high risk of bias (retaining only 4), adding to these the 3 unpublished studies mentioned above. They also included 2 studies published after our paper was completed and accepted for publication (see Steenen and colleagues8 and Elsey and Kindt9). Exactly how they concluded that this set of studies had low risk of bias is not specified. Moreover, their computation of the effect size for Elsey and Kindt’s phobia study suggests a possible outcome selection bias.9 It appears that Steenen and colleagues1 computed an effect size of d = 0.45 using the Behavioural Avoidance Test (reporting n = 0 per group when it appears to have been n = 13 per group). However, Elsey and Kindt also measured phobia severity with a post-treatment self-report questionnaire.9 All available outcome data should be included in a meta-analysis to minimize bias;10 this does not seem to have been considered by Steenen and colleagues.1

We agree that it would have been appropriate to include these 2 studies in our meta-analysis had they been available. However, a re-analysis of our data with these studies included indicates that our finding remains (Figure 1). Although not mentioned by Steenen and colleagues,1 we noted in our original meta-analysis that study quality was assessed and did not significantly moderate the overall outcome effect.2

Figure 1.

Forest plot results: reconsolidation interference by disorder. CI = confidence interval; LL = lower limit; Pbo = placebo; Ppnl = propranolol; SE = standard error; UL = upper limit.

Considering these important issues, Steenen and colleagues’1 results cannot and should not constitute evidence against our rigorous meta-analysis. Rather, we argue that our meta-analysis provides empirical support in favour of a reconsolidation-based treatment method involving narrative and pharmacological interventions that are more efficacious than simply giving “some tablets of propranolol.”