Table 1.
Distributions of Comparison Metrics Produced by Propensity Score Matching, Coarsened Exact Matching, and Ordinary Least Squares Regression When All Confounders Are Measured
| Data Structure and Analytical Model | Mean Point Estimate (SD, Empirical SE)a | Mean Software SE | Type I Error, % | Mean Analytical Sample Size (n) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Data Structureb | Analysis Model as Applied to Data Structurec | PSM | CEM | OLS | PSM | CEM | OLS | PSM | CEM | OLS | PSMd | CEM | OLS |
| 1. Confounder-outcome linear, good support | Correctly specified | 0.00 (0.11) | 0.00 (0.01) | 0.00 (0.01) | 0.14 | 0.01 | 0.01 | 0.8 | 5.2 | 5.2 | 20,000 | 19,952 | 20,000 |
| 2. Confounder-outcome linear, poor support | Correctly specified | 0.02 (0.66) | 0.00 (0.02) | 0.00 (0.02) | 0.85 | 0.01 | 0.02 | 1.6 | 18.2 | 4.8 | 19,938 | 19,652 | 20,000 |
| 3. Confounder-outcome quadratic, good support | Misspecified | −0.56 (26.91) | −0.30 (21.15) | −0.25 (18.28) | 32.57 | 17.28 | 18.52 | 3.2 | 11.0 | 4.9 | 19,970 | 19,567 | 20,000 |
| 4. Confounder-outcome quadratic, poor support | Misspecified | 0.88 (64.89) | 0.08 (36.18) | 0.18 (21.90) | 92.26 | 17.94 | 21.69 | 7.9 | 32.9 | 5.4 | 19,864 | 18,176 | 20,000 |
| 5. Confounder-exposure discontinuity | Misspecified | 0.04 (1.27) | 0.00 (0.03) | 0.00 (0.02) | 1.39 | 0.01 | 0.02 | 3.6 | 33.0 | 5.2 | 25,534 | 19,764 | 20,000 |
| 6. Confounder-outcome discontinuitye | Misspecified | 0.27 (1.29) | −0.13 (0.21) | 0.01 (0.45) | 1.86 | 0.29 | 0.52 | 3.5 | 2.1 | 2.5 | 19,895 | 19,493 | 20,000 |
| 7. Confounder-exposure and -outcome discontinuity | Misspecified | 0.05 (1.28) | 0.00 (0.37) | 11.59 (0.43) | 1.39 | 0.42 | 0.46 | 3.5 | 2.3 | 100.0 | 25,534 | 19,763 | 20,000 |
| 8. Exposure ratio 8:92, confounder-outcome linear | Correctly specified | 0.56 (439.65) | 0.00 (0.13) | 0.00 (0.03) | 547.87 | 0.13 | 0.03 | 1.7 | 6.3 | 4.7 | 3,410 | 248 | 20,000 |
| 9. Exposure ratio 8:92, confounder-outcome quadratic | Misspecified | 36.92 (482.75) | −127.02 (53.76) | 130,394.90 (2,264.48) | 750.41 | 490.30 | 1,415.84 | 2.0 | 0.0 | 100.0 | 3,411 | 249 | 20,000 |
Abbreviations: CEM, coarsened exact matching; OLS, ordinary least squares; PSM, propensity score matching; SD, standard deviation; SE, standard error.
a The empirical SE equals the SD of the point estimates across 10,000 replications. This empirical SE is the gold standard (as opposed to the SE produced by the software). By comparing the mean empirical SE (i.e., the SD of the point estimate) with the mean software SE, we can determine whether the software SE is estimated accurately.
b In all data structures, the true relationship between the exposure and the outcome was null.
c The analysis was implemented in the same way across all 9 data structures, assuming a linear relationship between x1 and the exposure and x1 and the outcome. When applied to the various data structures, the same analysis model could either be correctly specified or misspecified. Additional details on the nature of the misspecification are provided in the “Analytical approach” subsection of the Methods section.
d Our implementation of PSM weighs the number of unexposed observations to equal the number of exposed observations, so the analytical sample size is double the number of exposed observations for which a match exists within the caliper we set of 0.01. Therefore, the size of the PSM analytical sample is always double the number of included exposed observations, and it sometimes exceeds the total number of observations in the original sample.
e See Web Table 4 for more details on the results from the “confounder-outcome discontinuity” data structure.