Response to “Comment on ‘A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods’”

We appreciate the careful consideration by Keil et al. of our paper¹ comparing single-index mixture exposure models in simulations, including a novel permutation test for weighted quantile sum regression (WQSr) and quantile g-computation (QGC). We agree that mean absolute error (MAE) and mean absolute percent error (MAPE) assess accuracy and not bias. Therefore, we should have stated that the component-specific coefficients estimated by QGC were far less accurate that those of the WQSr models in our simulations. We appreciate the opportunity to make this correction.

However, we differ with Keil et al.’s characterization of QGC being more accurate than WQSr in their simulation scenario with 1 signal and 13 noise components. We reproduced this simulation using the permutation test version of WQSr instead of the training/test split version, which we recommended against using in our paper due to low power. Signal component MAPE was 11.6% for both WQSr and QGC, and MAE for all components was 0.018 for WQSr and 0.026 for QGC, indicating that WQSr is more accurate overall.

Keil et al. noted that WQSr is well-suited to the condition of unidirectional mixture effects used in our simulations.¹ It remains unclear how it will perform when run in both directions for the case of bidirectional mixture effects. Similarly, performance of QGC has not been characterized with complex bidirectional mixture effects having more than two signal components.

Keil et al. argued our treatment of failed WQSr iterations (i.e., those with no detectable signal in the positive direction) underestimated error. Imputing zero for failed WQSr iterations accurately represents the result but may underestimate error for versions of WQSr with high failure rates, such as repeated holdout (RH). We adopted this approach partly to ensure a fair evaluation of the RH versions of WQSr, although we recommended avoiding these models because of their high failure rates. We recommended the permutation test WQSr model, for which $\le 1\%$ of simulations failed, so our conclusions would not be affected by alternative approaches.

In our alcohol/caffeine mixture analogy, Keil et al. noted that the exposures counteract each other and that QGC can provide that information in addition to direction-specific effects. However, it is not straightforward to characterize beneficial and adverse effects using QGC in complex situations with more than two components, such as a mixture of endocrine-disrupting chemicals with pro- and antiandrogenic mechanisms. It would be difficult to ascertain the beneficial and adverse effects of such a mixture with QGC.

WQSr and QGC are both appealing choices depending on the aims of a given analysis. QGC is well-suited to estimating an overall (i.e., positive plus negative) mixture effect. It also provides estimates of individual component coefficients, although our simulations suggest these may be inaccurate. WQSr is well suited to providing separate estimates of the beneficial and adverse (i.e., positive and negative) effects of the mixture. We believe that in many contexts it is critical to separately detect and characterize beneficial and adverse mixture effects, which is why we recommended WQSr with a permutation test to maximize power.

Refers to https://doi.org.10.1289/EHP12404