Table 4.
Simulation results for DGP 2 and 3, over 1000 replications: normal endpoint, strong confounder–endpoint association, good and poor overlap.
| Relative bias (%) | Variance | RMSE | 95% CI coverage (%) | |
|---|---|---|---|---|
| DGP 2: Normally distributed endpoint, strong confounder–endpoint association, good overlap | ||||
| (d1) Q and g misspecified parametric | ||||
| OLS regression | −45.9 | 0.052 | 0.292 | 86 |
| IPTW | −59.1 | 0.067 | 0.350 | 98 |
| PS matching | −34.0 | 0.099 | 0.342 | 96 |
| WLS | −50.2 | 0.059 | 0.315 | 87 |
| TMLE | −45.7 | 0.041 | 0.272 | 86 |
| BCM | −31.4 | 0.074 | 0.299 | 90 |
| (d2) Q and g machine learning | ||||
| Regression (Q super learner) | −8.6 | 0.025 | 0.162 | 96 |
| IPTW (g boosted CART) | 41.0 | 0.036 | 0.251 | 99 |
| WLS (Q OLS, g boosted CART) | 2.6 | 0.022 | 0.149 | 100 |
| TMLE (Q SL, g boosted CART) | 3.1 | 0.011 | 0.106 | 95 |
| BCM (Q SL, g boosted CART) | 9.8 | 0.029 | 0.174 | 98 |
| DGP 3: Normally distributed endpoint, strong confounder–endpoint association, poor overlap | ||||
| (d1) Q and g misspecified parametric | ||||
| OLS regression | −119.2 | 0.050 | 0.527 | 40 |
| IPTW | −160.6 | 0.082 | 0.703 | 71 |
| PS matching | −81.1 | 0.100 | 0.453 | 84 |
| WLS | −137.9 | 0.063 | 0.606 | 39 |
| TMLE | −129.7 | 0.046 | 0.561 | 35 |
| BCM | −73.8 | 0.072 | 0.399 | 74 |
| (d2) Q and g machine learning | ||||
| Regression (Q super learner) | −22.0 | 0.046 | 0.233 | 94 |
| IPTW (g boosted CART) | 100.6 | 0.034 | 0.442 | 82 |
| WLS (Q OLS, g boosted CART) | −12.8 | 0.025 | 0.165 | 99 |
| TMLE (Q SL, g boosted CART) | 5.6 | 0.019 | 0.139 | 87 |
| BCM (Q SL, g boosted CART) | 12.3 | 0.034 | 0.191 | 98 |
Note: In DGPs 2 and 3, the true ATE was 0.4 and the biases, using a naive estimator based on the mean difference, were 80 and 190%, respectively.