TABLE 3.
Performance of different methods for synthesizing standardized mean differences based on 10,000 replicates of simulated meta-analyses with N = 20 studies under various settings.
| Method | τ = 0.1 | τ = 0.5 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ni = 6–14 | ni = 30–70 | ni = 6–14 | ni = 30–70 | |||||||||
| Bias | RMSE | CP, % | Bias | RMSE | CP, % | Bias | RMSE | CP, % | Bias | RMSE | CP, % | |
| θ = 0: | ||||||||||||
| Cohen’s d, Eq. (4) | 0.001 | 0.144 | 97.7 (95.6) | 0.000 | 0.068 | 95.2 (95.0) | 0.000 | 0.177 | 95.4 (95.9) | 0.002 | 0.129 | 93.4 (95.3) |
| Cohen’s d, Eq. (5) | 0.001 | 0.144 | 96.3 (96.1) | 0.000 | 0.068 | 95.2 (95.0) | 0.000 | 0.178 | 94.6 (96.4) | 0.002 | 0.129 | 93.4 (95.3) |
| Cohen’s d, Eq. (6) | 0.001 | 0.148 | 95.9 (96.1) | 0.000 | 0.068 | 95.1 (95.0) | −0.001 | 0.184 | 94.4 (96.3) | 0.002 | 0.129 | 93.4 (95.2) |
| Hedges’ g, Eq. (10) | 0.001 | 0.129 | 97.7 (95.8) | 0.000 | 0.067 | 95.2 (95.0) | 0.000 | 0.157 | 95.6 (96.1) | 0.002 | 0.127 | 93.4 (95.3) |
| Hedges’ g, Eq. (11) | 0.001 | 0.129 | 96.4 (96.2) | 0.000 | 0.067 | 95.2 (95.0) | 0.000 | 0.159 | 94.7 (96.5) | 0.002 | 0.127 | 93.4 (95.2) |
| Hedges’ g, Eq. (12) | 0.001 | 0.133 | 95.9 (96.1) | 0.000 | 0.067 | 95.2 (95.0) | −0.001 | 0.164 | 94.4 (96.3) | 0.002 | 0.127 | 93.4 (95.2) |
| Hedges’ g, Eq. (13) | 0.001 | 0.135 | 97.0 (95.8) | 0.000 | 0.067 | 95.3 (95.0) | 0.000 | 0.167 | 95.1 (95.9) | 0.002 | 0.127 | 93.4 (95.2) |
| Hedges’ g, Eq. (14) | 0.001 | 0.128 | 97.8 (95.8) | 0.000 | 0.067 | 95.2 (95.0) | 0.000 | 0.157 | 95.7 (96.1) | 0.002 | 0.127 | 93.4 (95.3) |
| Hedges’ g, Eq. (20) | 0.001 | 0.154 | 95.8 (95.1) [96.5] | 0.000 | 0.069 | 95.1 (94.7) [95.2] | −0.001 | 0.192 | 94.1 (95.1) [95.2] | 0.002 | 0.130 | 93.4 (95.0) [93.5] |
| Hedges’ g, Eq. (21) | 0.001 | 0.153 | 96.3 (94.9) [96.2] | 0.000 | 0.069 | 95.1 (94.7) [95.1] | −0.001 | 0.192 | 94.3 (94.8) [94.4] | 0.002 | 0.130 | 93.4 (95.0) [93.5] |
| Hedges’ g, Eq. (22) | 0.001 | 0.156 | 94.9 (95.3) [97.2] | 0.000 | 0.069 | 94.9 (94.8) [95.2] | −0.001 | 0.193 | 93.9 (95.3) [96.0] | 0.002 | 0.130 | 93.4 (95.0) [93.6] |
| θ = 0.3: | ||||||||||||
| Cohen’s d, Eq. (4) | −0.012 | 0.145 | 97.6 (95.8) | −0.001 | 0.068 | 95.2 (94.9) | −0.023 | 0.179 | 95.5 (95.8) | −0.003 | 0.130 | 93.3 (95.1) |
| Cohen’s d, Eq. (5) | −0.010 | 0.145 | 96.3 (96.2) | −0.001 | 0.068 | 94.9 (94.9) | −0.016 | 0.179 | 94.7 (96.3) | −0.003 | 0.130 | 93.3 (95.1) |
| Cohen’s d, Eq. (6) | −0.002 | 0.148 | 96.0 (96.1) | 0.000 | 0.069 | 95.0 (94.9) | −0.006 | 0.184 | 94.7 (96.2) | −0.003 | 0.131 | 93.3 (95.1) |
| Hedges’ g, Eq. (10) | −0.042 | 0.136 | 96.9 (94.9) | −0.006 | 0.068 | 95.1 (94.9) | −0.051 | 0.166 | 94.6 (95.4) | −0.008 | 0.128 | 93.3 (95.0) |
| Hedges’ g, Eq. (11) | −0.041 | 0.136 | 95.2 (95.2) | −0.006 | 0.068 | 94.9 (94.9) | −0.046 | 0.166 | 93.9 (95.8) | −0.008 | 0.128 | 93.3 (95.0) |
| Hedges’ g, Eq. (12) | −0.034 | 0.137 | 95.0 (95.5) | −0.005 | 0.068 | 94.9 (94.9) | −0.038 | 0.169 | 94.0 (96.0) | −0.008 | 0.129 | 93.3 (95.0) |
| Hedges’ g, Eq. (13) | −0.029 | 0.138 | 96.5 (95.4) | −0.005 | 0.068 | 95.0 (94.9) | −0.035 | 0.170 | 94.7 (95.7) | −0.008 | 0.129 | 93.3 (95.0) |
| Hedges’ g, Eq. (14) | −0.043 | 0.136 | 96.9 (94.8) | −0.006 | 0.068 | 95.1 (94.9) | −0.053 | 0.166 | 94.6 (95.3) | −0.008 | 0.128 | 93.3 (95.0) |
| Hedges’ g, Eq. (20) | 0.000 | 0.155 | 95.7 (95.2) [96.5] | 0.000 | 0.069 | 95.0 (94.7) [95.1] | −0.001 | 0.193 | 94.6 (95.4) [95.6] | −0.001 | 0.131 | 93.3 (94.9) [93.5] |
| Hedges’ g, Eq. (21) | 0.000 | 0.154 | 96.3 (95.2) [96.2] | 0.000 | 0.069 | 95.0 (94.7) [95.0] | −0.002 | 0.192 | 94.8 (95.2) [94.9] | −0.001 | 0.131 | 93.3 (94.9) [93.4] |
| Hedges’ g, Eq. (22) | 0.000 | 0.157 | 95.1 (95.3) [97.1] | 0.000 | 0.069 | 94.8 (94.7) [95.2] | −0.002 | 0.194 | 94.4 (95.7) [96.3] | −0.001 | 0.131 | 93.3 (94.9) [93.4] |
| θ = 0.8: | ||||||||||||
| Cohen’s d, Eq. (4) | −0.030 | 0.152 | 97.2 (95.3) | −0.002 | 0.070 | 95.4 (95.0) | −0.054 | 0.191 | 94.5 (95.1) | −0.006 | 0.131 | 93.6 (95.3) |
| Cohen’s d, Eq. (5) | −0.023 | 0.151 | 95.9 (96.0) | −0.002 | 0.070 | 95.3 (95.1) | −0.037 | 0.188 | 94.0 (95.8) | −0.005 | 0.131 | 93.6 (95.3) |
| Cohen’s d, Eq. (6) | −0.004 | 0.153 | 96.0 (96.1) | −0.001 | 0.070 | 95.2 (95.1) | −0.013 | 0.190 | 94.3 (96.3) | −0.004 | 0.131 | 93.6 (95.4) |
| Hedges’ g, Eq. (10) | −0.108 | 0.171 | 92.3 (89.2) | −0.015 | 0.071 | 94.7 (94.4) | −0.131 | 0.209 | 87.8 (89.5) | −0.019 | 0.130 | 93.4 (95.2) |
| Hedges’ g, Eq. (11) | −0.105 | 0.170 | 89.7 (90.1) | −0.015 | 0.071 | 94.5 (94.4) | −0.118 | 0.203 | 87.6 (91.0) | −0.019 | 0.130 | 93.3 (95.2) |
| Hedges’ g, Eq. (12) | −0.088 | 0.163 | 91.1 (91.8) | −0.014 | 0.071 | 94.6 (94.5) | −0.097 | 0.195 | 89.8 (92.6) | −0.018 | 0.130 | 93.4 (95.2) |
| Hedges’ g, Eq. (13) | −0.076 | 0.158 | 94.0 (92.6) | −0.014 | 0.071 | 94.6 (94.5) | −0.089 | 0.193 | 91.4 (93.0) | −0.018 | 0.130 | 93.4 (95.2) |
| Hedges’ g, Eq. (14) | −0.111 | 0.173 | 92.1 (88.6) | −0.015 | 0.071 | 94.7 (94.4) | −0.135 | 0.211 | 87.3 (89.0) | −0.019 | 0.130 | 93.3 (95.2) |
| Hedges’ g, Eq. (20) | 0.001 | 0.162 | 95.7 (95.1) [96.8] | 0.001 | 0.071 | 95.2 (94.9) [95.4] | 0.001 | 0.200 | 94.4 (95.3) [95.6] | −0.001 | 0.131 | 93.5 (95.1) [93.6] |
| Hedges’ g, Eq. (21) | 0.000 | 0.160 | 96.4 (94.7) [96.3] | 0.001 | 0.071 | 95.3 (94.9) [95.3] | 0.000 | 0.199 | 94.5 (95.0) [94.5] | −0.001 | 0.131 | 93.4 (95.1) [93.5] |
| Hedges’ g, Eq. (22) | 0.001 | 0.164 | 95.0 (95.2) [97.1] | 0.001 | 0.071 | 95.2 (95.0) [95.5] | 0.000 | 0.201 | 94.2 (95.4) [96.3] | −0.001 | 0.131 | 93.5 (95.1) [93.6] |
Abbreviation: CP, coverage probability of 95% confidence interval; RMSE, root mean squared error.
Notation: N, no. of studies; τ, between-study standard deviation; ni, study-specific sample size; θ, true overall standardized mean difference.
Note: coverage probabilities include those based on conventional meta-analysis method in Equation (23) (outside parentheses and square brackets), based on the Hartung–Knapp–Sidik–Jonkman method in Equation (24) (inside parentheses), and based on the method in Equation (26) (inside square brackets).