Comment on “A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods”

Recently, Day et al.¹ compared weighted quantile sum regression (WQSr) with quantile-based g-computation (QGC) using simulations and a worked example.² They wrote that “mixture component-specific coefficients estimated by [QGC] were far more biased than those of any of the WQSr models.”¹ Their results do not support this claim because bias was not assessed in isolation.

Instead, Day et al. assessed mean absolute percent error (MAPE), a measure of accuracy that combines bias and variability. To assess bias, we repeated one of their simulations (correlated mixture, ${\mathrm{\beta }}_1=0.2$) and examined mean percent bias (MPB: average bias/truth) of the component-specific coefficients. MPB of their weighted quantile sum regression bootstrap sample permutation test (WQSBSPT) approach was 2.5–8 times higher than that for QGC (Table 1). MPB for the “mixture effect” was 80 times higher for WQSBSPT than QGC. Thus, the results support a countermanding claim: QGC was far less biased than WQSBSPT. We consequently disagree that their results suggest “caution when interpreting [QGC] coefficients.”¹

Table 1.

Comparison of mean percent bias for QGC and WQSBSPT.

	Mean percent bias (%) (200 samples of $n=500$)
Estimand	QGC	WQSBSPT
Component coefficient, “high” effect size	2	$-5$
Component coefficient, “low” effect size	6	48
Overall effect size	0.1	8

Note: Mean percent bias for two contrasting approaches (QGC without bootstrapping and weighted quantile sum regression using WQSBSPT) to estimate the effects of a mixture using the data simulation methods of Day et al.¹ with a “correlated mixture” developed with the empirical covariance matrix reported in the appendix of their paper. QGC, quantile-based g-computation without bootstrapping; WQSBSPT, weighted quantile sum regression bootstrap sample permutation test.

Day et al.’s simulations assume unidirectional causal exposures, an ideal setting for maximizing WQSr accuracy. A reanalysis using our previously published simulation² with 1 causal exposure and 13 noise exposures yielded a 3-fold better component coefficient MAPE for QGC than WQSBS-Split (12% and 36%, respectively). Similar to Day et al.’s simulations, this simulation assumes no counteracting exposures. Thus, accuracy results do not generalize across different plausible scenarios.

We also note a fundamental flaw in some simulations. When WQSr methods failed to return a result, Day et al. imputed ${\widehat{\mathrm{\beta }}}_1=0$. When analyzing data simulated with ${\mathrm{\beta }}_1=0$, the authors therefore imputed an estimate with perfect accuracy (no bias, no variability), which exaggerated WQSr performance. Because 86% of Weighted quantile sum regression, random subset, repeated holdout (WQSRS-RH) fits failed to return a result when ${\mathrm{\beta }}_1=0$, their WQSRS-RH results are not credible. Other WQSr results were also impacted.

Finally, we respond to the authors’ coffee/alcohol analogy. The analogy says that the effects of coffee and alcohol would cancel out one another in a QGC regression such that QGC would erroneously report no joint effect of substance use. If the joint effect of coffee and alcohol on a health outcome of interest is truly null, QGC will tell us so² and also estimate the independent effects of alcohol and coffee. That is no error; it is crucial to understand when therapeutic agents counteract hazardous agents in a mixture. Alternatively, WQSr will yield biased independent effects of alcohol and coffee and no joint effect.³

Refers to https://doi.org.10.1289/EHP10570