TABLE 4.

Performance of targeted learning and estimating equation estimators by study within the HAL-based simulations

Method	Study ID	TrueATE	Variance	Bias	MSE	rMSE	Coverage	Coverage2	CIwidth
A-IPTW	1	−0.0109	0.0056	−0.0373	0.0070		0.704	0.912	0.1658
	2	0.0507	0.0005	−0.0012	0.0005		0.934	0.958	0.0849
	3	0.0007	0.0003	0.0007	0.0003		0.954	0.948	0.0737
	4	−0.0468	0.0019	0.0109	0.0020		0.928	0.950	0.1612
	5	−0.0136	0.0020	−0.0046	0.0020		0.926	0.952	0.1640
	6	0.2523	0.0010	−0.0266	0.0017		0.672	0.868	0.0829
	7	−0.0310	0.0012	0.0093	0.0013		0.862	0.938	0.1098
	8	−0.0442	0.0037	0.0037	0.0037		0.914	0.952	0.2112
	9	0.0089	0.0001	−0.0005	0.0001		0.940	0.948	0.0362
	10	0.0203	0.0006	−0.0001	0.0006		0.954	0.954	0.0961
C-TMLE	1	−0.0109	0.0219	−0.0947	0.0309		0.836	0.890	0.4956
	2	0.0507	0.0006	0.0016	0.0006		0.956	0.954	0.0993
	3	0.0007	0.0004	0.0018	0.0004		0.948	0.948	0.0782
	4	−0.0468	0.0026	0.0046	0.0026		0.948	0.950	0.2005
	5	−0.0136	0.0027	−0.0087	0.0027		0.928	0.950	0.1882
	6	0.2523	0.0011	−0.0124	0.0012		0.942	0.944	0.1295
	7	−0.0310	0.0025	0.0012	0.0025		0.936	0.948	0.1875
	8	−0.0442	0.0049	−0.0014	0.0049		0.922	0.960	0.2524
	9	0.0089	0.0001	−0.0008	0.0001		0.942	0.950	0.0409
	10	0.0203	0.0006	0.0007	0.0006		0.952	0.940	0.1037
CV-A-IPTW	1	−0.0109	0.0565	−0.0262	0.0572		0.926	0.954	0.7789
	2	0.0507	0.0006	0.0031	0.0006		0.960	0.954	0.0985
	3	0.0007	0.0004	0.0009	0.0004		0.966	0.950	0.0793
	4	−0.0468	0.0025	0.0063	0.0025		0.956	0.948	0.2008
	5	−0.0136	0.0024	−0.0062	0.0024		0.940	0.946	0.1881
	6	0.2523	0.0012	−0.0045	0.0012		0.938	0.944	0.1301
	7	−0.0310	0.0020	0.0030	0.0020		0.936	0.940	0.1737
	8	−0.0442	0.0045	0.0000	0.0045		0.950	0.952	0.2553
	9	0.0089	0.0001	−0.0002	0.0001		0.942	0.948	0.0394
	10	0.0203	0.0006	0.0011	0.0006		0.962	0.944	0.1026
CV-IPTW	1	−0.0109	0.1632	−0.1129	0.1759		0.984	0.944	2.3263
	2	0.0507	0.0008	−0.0012	0.0008		1.000	0.948	0.2686
	3	0.0007	0.0005	0.0020	0.0005		1.000	0.954	0.1831
	4	−0.0468	0.0032	0.0270	0.0040		1.000	0.936	0.5065
	5	−0.0136	0.0028	−0.0202	0.0032		0.982	0.936	0.2817
	6	0.2523	0.0014	−0.0057	0.0014		1.000	0.954	0.3305
	7	−0.0310	0.0033	0.0017	0.0033		1.000	0.948	0.5111
	8	−0.0442	0.0062	0.0006	0.0062		1.000	0.948	0.6842
	9	0.0089	0.0001	0.0143	0.0003		0.998	0.756	0.1016
	10	0.0203	0.0007	0.0006	0.0007		1.000	0.956	0.2451
CV-TMLE	1	−0.0109	0.1868	−0.0291	0.1876		0.590	0.938	0.7006
	2	0.0507	0.0006	0.0031	0.0006		0.958	0.966	0.0985
	3	0.0007	0.0004	0.0009	0.0004		0.966	0.958	0.0793
	4	−0.0468	0.0025	0.0064	0.0025		0.956	0.954	0.2008
	5	−0.0136	0.0024	−0.0063	0.0024		0.940	0.958	0.1881
	6	0.2523	0.0013	0.0046	0.0013		0.936	0.958	0.1301
	7	−0.0310	0.0020	0.0030	0.0020		0.938	0.946	0.1737
	8	−0.0442	0.0044	0.0000	0.0044		0.950	0.964	0.2551
	9	0.0089	0.0001	−0.0002	0.0001		0.942	0.952	0.0394
	10	0.0203	0.0006	0.0011	0.0006		0.962	0.956	0.1026
IPTW	1	−0.0109	0.0280	−0.0736	0.0334		0.890	0.932	0.5945
	2	0.0507	0.0008	−0.0028	0.0008		1.000	0.948	0.2544
	3	0.0007	0.0005	0.0022	0.0005		1.000	0.954	0.1789
	4	−0.0468	0.0033	0.0276	0.0040		1.000	0.926	0.4889
	5	−0.0136	0.0028	−0.0201	0.0032		0.978	0.940	0.2712
	6	0.2523	0.0014	−0.0091	0.0015		0.998	0.942	0.2536
	7	−0.0310	0.0033	0.0023	0.0033		1.000	0.946	0.4838
	8	−0.0442	0.0062	0.0003	0.0062		1.000	0.950	0.6504
	9	0.0089	0.0001	0.0172	0.0004		0.992	0.684	0.0990
	10	0.0203	0.0007	0.0007	0.0007		1.000	0.958	0.2410
TMLE	1	−0.0109	0.3860	−0.3235	0.4906		0.100	0.920	0.1681
	2	0.0507	0.0006	0.0005	0.0006		0.932	0.960	0.0849
	3	0.0007	0.0004	0.0007	0.0004		0.946	0.948	0.0737
	4	−0.0468	0.0020	0.0099	0.0021		0.916	0.950	0.1611
	5	−0.0136	0.0022	−0.0052	0.0022		0.922	0.948	0.1640
	6	0.2523	0.0010	−0.0015	0.0010		0.806	0.942	0.0828
	7	−0.0310	0.0013	0.0079	0.0014		0.852	0.932	0.1098
	8	−0.0442	0.0040	0.0020	0.0040		0.900	0.952	0.2111
	9	0.0089	0.0001	−0.0003	0.0001		0.936	0.948	0.0362
	10	0.0203	0.0006	0.0001	0.0006		0.952	0.954	0.0961

Abbreviations: Coverage, coverage using 95% Wald-type confidence intervals (CI) based upon standard error estimates, where “Coverage2” uses the true sample variance; CI width, average width of the “Coverage” CI’s; Variance, true sample variance; MSE, mean-squared error; rMSE, relative (to the IPTW estimator in denominator) mean-squared error.