Rudolph et al. Respond to “Power in Mediation Analysis”

We thank Dr. VanderWeele for highlighting the need to improve power and sample-size calculations for mediation analyses (1)—currently a weak point in the literature—and for a thoughtful discussion of our paper (2). We agree that power and sample-size calculations for mediation have been neglected and that much more work on this topic is merited, given the increasing interest in mediation questions in health and the corresponding use of methods that estimate path-specific mediation effects (Elizabeth A. Stuart, Johns Hopkins Bloomberg School of Public Health, unpublished manuscript, 2020).

We also wholeheartedly agree that more work is needed to disseminate methods for power and sample-size calculations for mediation analyses and to lower the barriers to implementation for applied researchers, especially those who are less computationally inclined and for whom our simulation tutorial may not be appropriate. VanderWeele writes that such methods for power and sample size must be very easy to use in order to engender appreciable uptake (1) and provides examples of successful use of the Valeri and VanderWeele (3) and Imai et al. (4) approaches for estimation of direct and indirect effects. One way to improve accessibility might be for future grant-funded projects to have the explicit objective of creating Stata programs (StataCorp LLC, College Station, Texas), SAS macros (SAS Institute, Inc., Cary, North Carolina), R packages (R Foundation for Statistical Computing, Vienna, Austria) and/or simple interactive Web-based tools that implement simulation-based approaches behind the scenes and which have clear interfaces for parameter settings and display of results. Sadly, such (crucial) efforts tend not to be supported through existing grant mechanisms, which often highlight methodological advances but not always broad dissemination. To conduct such activities well requires expertise in dissemination and dedicated resources.

Simulation-based power analysis where the simulation mimics key characteristics of the data is, we believe, the most accurate way to calculate power, because, as VanderWeele points out (1), estimator performance (of which power is one aspect) can vary across finite sample scenarios. We considered 13 scenarios, each under 3 different sample sizes and representing relatively large and small effect sizes, and demonstrated large variations in terms of estimator-specific power across these finite-sample scenarios (2). In one scenario, power spanned nearly the entire 0%–100% range, despite the estimators’ having appropriate confidence interval coverage. Considering this starting point, we agree that there would be value in even more comprehensive simulation studies that give general practical guidance about which estimator may provide the most power while continuing to provide appropriate coverage. We considered estimands on the risk difference scale and with a single binary treatment, mediator, and outcome. However, when considering a binary outcome, alternative scales, like the risk ratio and odds ratio, may be a better match for the research question. In addition, power and estimator performance can differ markedly depending on the types and characteristics of variable distributions (e.g., continuous, binary, rare) and the existence of multiple and perhaps interrelated mediating variables. In the paper, we compared 4 estimation approaches plus an analytical equation-based method, but as VanderWeele rightly points out (1), there exist many others, with more being added nearly every month.

In conclusion, we appreciate the opportunity to raise awareness of the limitations of current practice for calculating power in mediation analyses and to highlight the need for additional work to further lower barriers to implement the simulation approach we illustrate here. As mediation approaches become more widespread, accurate power estimation becomes even more important to prevent investigators in underpowered studies from concluding that an indirect effect does not exist, when in fact they lacked the power to detect such an effect. We hope that barriers to implementing accurate power and sample-size estimation will continue to be lowered to the point where this will be one of the first steps researchers undertake when considering any mediation research question.