Table 5. Associations between PSM methodology reporting and statistical significance of primary outcome.
| Methodologic Characteristic | Proportion of Studies (n, %)‡ | Overall P-value† | Univariate Logistic Regression | Multivariate Logistic Regression* | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Overall | Reported Significant Results | Did not Report Significant Results | OR | 95% CI | P-Value | OR | 95% CI | P-Value | ||
| (N = 92) | (N = 66) | (N = 26) | ||||||||
| Algorithm Reported | 52 (56.2) | 34 (51.5) | 18 (69.2) | 0.123 | 0.47 | 0.18–1.24 | 0.127 | |||
| Ratio Reported | 87 (94.6) | 63 (95.5) | 24 (92.3) | 0.549 | 1.75 | 0.28–11.1 | 0.550 | |||
| Replacement Reported | 22 (23.9) | 12 (18.2) | 10 (38.5) | 0.040 | 0.36 | 0.13–0.97 | 0.044 | |||
| Paired Stats Method | 77 (83.7) | 56 (84.9) | 21 (80.8) | 0.633 | 1.33 | 0.41–4.36 | 0.634 | |||
| Reproducibility Score | 0.006 | |||||||||
| 0 or 1 | 12 (13.0) | 7 (10.6) | 5 (19.2) | Ref. | - | - | ||||
| 2 or 3 | 64 (69.6) | 52 (78.8) | 12 (46.2) | 3.10 | 0.84–11.45 | 0.090 | ||||
| 4 | 16 (17.4) | 7 (10.6) | 9 (34.6) | 0.56 | 0.12–2.52 | 0.447 | ||||
| Fully Reproducible | 16 (17.4) | 7 (10.6) | 9 (34.6) | 0.006 | 0.22 | 0.07–0.69 | 0.009 | 0.26 | 0.07–1.02 | 0.053 |
| Covariates Included | 89 (96.7) | 63 (95.5) | 26 (100) | 0.269 | ||||||
| Covariates Justified | 33 (35.9) | 20 (30.3) | 13 (50.0) | 0.076 | 0.43 | 0.17–1.10 | 0.079 | 0.87 | 0.28–2.68 | 0.810 |
| Illustrated Co-Variate Balance | 62 (67.4) | 43 (65.2) | 19 (73.1) | 0.465 | 0.69 | 0.25–1.88 | 0.467 | |||
| Use of SMD | 18 (19.6) | 11 (16.7) | 7 (26.9) | 0.264 | 0.54 | 0.18–1.60 | 0.268 | |||
| Regression Model Reported | 57 (62.0) | 36 (54.6) | 21 (80.8) | 0.020 | 0.28 | 0.10–0.85 | 0.024 | 0.32 | 0.10–1.00 | 0.051 |
| Goodness of Fit Test | 3 (3.3) | 2 (3.0) | 1 (3.9) | 0.840 | 0.78 | 0.07–9.00 | 0.843 | |||
| Sensitivity Analysis | 4 (4.4) | 3 (4.6) | 1 (3.9) | 0.882 | 1.19 | 0.12–12.0 | 0.882 | |||
| Baseline Characteristics | 33 (35.9) | 20 (30.3) | 13 (50.0) | 0.076 | 0.44 | 0.17–1.10 | 0.080 | 0.43 | 0.14–1.33 | 0.142 |
OR = Odds Ratio; CI = Confidence Interval
†Overall p values were calculated by chi-squared analysis, Fisher’s Exact Test, or the Wilcoxon Rank Sum Test where appropriate
‡ Reported as # studies reporting methodological characteristic/total # of studies in subgroup (%) for categorical variables and median (first quartile, third quartile) for continuous non-parametric variables.
*For the multivariate analysis, a mixed effect logistic model with random effect for journal was used