TABLE 2.
Performance of different methods for synthesizing standardized mean differences based on 10,000 replicates of simulated meta-analyses with N = 5 studies under various settings.
| Method | τ = 0.1 | τ = 0.5 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ni = 6–14 | ni = 30–70 | ni = 6–14 | ni = 30–70 | |||||||||
| Bias | RMSE | CP, % | Bias | RMSE | CP, % | Bias | RMSE | CP, % | Bias | RMSE | CP, % | |
| θ = 0: | ||||||||||||
| Cohen’s d, Eq. (4) | 0.000 | 0.299 | 97.6 (95.4) | −0.001 | 0.137 | 95.2 (94.6) | 0.001 | 0.367 | 95.4 (95.8) | 0.002 | 0.260 | 89.6 (95.4) |
| Cohen’s d, Eq. (5) | 0.000 | 0.303 | 96.3 (96.0) | −0.001 | 0.137 | 95.2 (94.6) | 0.001 | 0.371 | 94.0 (96.1) | 0.002 | 0.260 | 89.6 (95.4) |
| Cohen’s d, Eq. (6) | 0.000 | 0.312 | 96.0 (95.9) | −0.001 | 0.138 | 95.2 (94.6) | 0.000 | 0.383 | 93.6 (96.0) | 0.002 | 0.260 | 89.6 (95.4) |
| Hedges’ g, Eq. (10) | 0.000 | 0.267 | 97.6 (95.6) | −0.001 | 0.135 | 95.3 (94.6) | 0.001 | 0.327 | 95.5 (96.0) | 0.002 | 0.255 | 89.6 (95.4) |
| Hedges’ g, Eq. (11) | 0.000 | 0.271 | 96.3 (95.9) | −0.001 | 0.135 | 95.2 (94.7) | 0.001 | 0.330 | 94.1 (96.2) | 0.002 | 0.255 | 89.6 (95.4) |
| Hedges’ g, Eq. (12) | 0.000 | 0.279 | 96.0 (95.8) | −0.001 | 0.135 | 95.2 (94.7) | 0.000 | 0.341 | 93.7 (96.0) | 0.002 | 0.256 | 89.6 (95.4) |
| Hedges’ g, Eq. (13) | 0.000 | 0.281 | 97.1 (95.5) | −0.001 | 0.135 | 95.3 (94.6) | 0.000 | 0.345 | 94.7 (95.8) | 0.002 | 0.256 | 89.6 (95.4) |
| Hedges’ g, Eq. (14) | 0.000 | 0.266 | 97.6 (95.6) | −0.001 | 0.135 | 95.3 (94.6) | 0.001 | 0.326 | 95.5 (96.0) | 0.002 | 0.255 | 89.7 (95.4) |
| Hedges’ g, Eq. (20) | −0.001 | 0.313 | 96.2 (95.2) [96.9] | −0.001 | 0.137 | 95.2 (94.5) [95.3] | −0.001 | 0.382 | 93.7 (95.4) [94.8] | 0.002 | 0.260 | 89.3 (95.2) [89.5] |
| Hedges’ g, Eq. (21) | 0.000 | 0.309 | 96.9 (94.9) [96.6] | −0.001 | 0.137 | 95.2 (94.5) [95.2] | 0.000 | 0.381 | 94.4 (95.1) [94.0] | 0.002 | 0.260 | 89.4 (95.2) [89.4] |
| Hedges’ g, Eq. (22) | −0.001 | 0.315 | 95.1 (95.5) [97.4] | −0.001 | 0.137 | 95.0 (94.6) [95.3] | −0.001 | 0.382 | 92.6 (95.6) [95.5] | 0.002 | 0.260 | 89.3 (95.2) [89.6] |
| θ = 0.3: | ||||||||||||
| Cohen’s d, Eq. (4) | −0.001 | 0.300 | 97.7 (95.6) | 0.000 | 0.138 | 95.6 (95.0) | −0.007 | 0.372 | 95.2 (95.3) | 0.000 | 0.263 | 88.5 (94.8) |
| Cohen’s d, Eq. (5) | 0.002 | 0.303 | 96.4 (95.9) | 0.000 | 0.138 | 95.4 (95.0) | −0.001 | 0.378 | 93.8 (95.8) | 0.000 | 0.263 | 88.4 (94.8) |
| Cohen’s d, Eq. (6) | 0.010 | 0.312 | 96.1 (95.8) | 0.000 | 0.138 | 95.3 (95.0) | 0.007 | 0.391 | 93.5 (95.8) | 0.000 | 0.264 | 88.4 (94.8) |
| Hedges’ g, Eq. (10) | −0.032 | 0.270 | 97.5 (95.7) | −0.005 | 0.135 | 95.6 (95.0) | −0.037 | 0.334 | 95.1 (95.5) | −0.006 | 0.259 | 88.5 (94.8) |
| Hedges’ g, Eq. (11) | −0.030 | 0.272 | 96.4 (96.0) | −0.005 | 0.136 | 95.4 (95.0) | −0.033 | 0.338 | 93.9 (95.7) | −0.005 | 0.259 | 88.5 (94.8) |
| Hedges’ g, Eq. (12) | −0.023 | 0.280 | 96.0 (95.9) | −0.005 | 0.136 | 95.4 (95.0) | −0.026 | 0.349 | 93.5 (95.7) | −0.005 | 0.259 | 88.5 (94.8) |
| Hedges’ g, Eq. (13) | −0.020 | 0.281 | 97.0 (95.6) | −0.005 | 0.136 | 95.6 (95.0) | −0.023 | 0.352 | 94.6 (95.4) | −0.005 | 0.259 | 88.6 (94.8) |
| Hedges’ g, Eq. (14) | −0.033 | 0.269 | 97.6 (95.6) | −0.005 | 0.135 | 95.6 (95.0) | −0.039 | 0.333 | 95.2 (95.5) | −0.006 | 0.259 | 88.5 (94.8) |
| Hedges’ g, Eq. (20) | 0.003 | 0.311 | 96.3 (95.4) [97.1] | −0.001 | 0.137 | 95.3 (94.9) [95.4] | 0.002 | 0.393 | 93.6 (95.0) [94.6] | 0.000 | 0.264 | 88.3 (94.6) [88.4] |
| Hedges’ g, Eq. (21) | 0.002 | 0.307 | 97.0 (95.2) [96.6] | −0.001 | 0.137 | 95.4 (94.9) [95.4] | 0.001 | 0.391 | 94.2 (94.8) [93.9] | 0.000 | 0.264 | 88.3 (94.6) [88.3] |
| Hedges’ g, Eq. (22) | 0.002 | 0.313 | 95.3 (95.6) [97.5] | −0.001 | 0.138 | 95.1 (94.9) [95.5] | 0.001 | 0.393 | 92.7 (95.3) [95.4] | 0.000 | 0.264 | 88.2 (94.6) [88.5] |
| θ = 0.8: | ||||||||||||
| Cohen’s d, Eq. (4) | −0.006 | 0.308 | 98.0 (95.8) | 0.000 | 0.141 | 95.8 (95.0) | −0.023 | 0.386 | 94.6 (95.0) | 0.000 | 0.261 | 89.2 (95.2) |
| Cohen’s d, Eq. (5) | 0.004 | 0.313 | 96.4 (95.9) | 0.000 | 0.141 | 95.6 (95.0) | −0.009 | 0.390 | 93.5 (95.4) | 0.001 | 0.261 | 89.2 (95.2) |
| Cohen’s d, Eq. (6) | 0.023 | 0.324 | 96.1 (95.9) | 0.000 | 0.141 | 95.6 (95.0) | 0.013 | 0.402 | 93.4 (95.4) | 0.002 | 0.261 | 89.2 (95.2) |
| Hedges’ g, Eq. (10) | −0.087 | 0.289 | 96.7 (94.9) | −0.013 | 0.139 | 95.6 (94.9) | −0.103 | 0.359 | 93.5 (94.3) | −0.013 | 0.257 | 89.2 (95.1) |
| Hedges’ g, Eq. (11) | −0.082 | 0.292 | 95.1 (95.2) | −0.013 | 0.139 | 95.3 (94.9) | −0.093 | 0.360 | 92.0 (94.7) | −0.012 | 0.257 | 89.1 (95.1) |
| Hedges’ g, Eq. (12) | −0.065 | 0.296 | 95.0 (95.2) | −0.013 | 0.139 | 95.3 (94.9) | −0.075 | 0.365 | 92.2 (94.8) | −0.012 | 0.257 | 89.1 (95.1) |
| Hedges’ g, Eq. (13) | −0.056 | 0.294 | 96.5 (95.1) | −0.012 | 0.139 | 95.5 (94.9) | −0.068 | 0.366 | 93.5 (94.6) | −0.012 | 0.257 | 89.1 (95.0) |
| Hedges’ g, Eq. (14) | −0.090 | 0.289 | 96.7 (94.8) | −0.013 | 0.139 | 95.6 (94.9) | −0.107 | 0.359 | 93.4 (94.2) | −0.013 | 0.257 | 89.2 (95.1) |
| Hedges’ g, Eq. (20) | 0.005 | 0.334 | 96.3 (95.0) [97.0] | −0.002 | 0.141 | 95.5 (94.9) [95.7] | −0.001 | 0.405 | 93.3 (94.4) [94.3] | 0.002 | 0.261 | 89.0 (94.9) [89.2] |
| Hedges’ g, Eq. (21) | 0.002 | 0.327 | 97.1 (94.8) [96.8] | −0.002 | 0.141 | 95.7 (94.9) [95.6] | −0.004 | 0.403 | 93.9 (94.1) [93.7] | 0.002 | 0.261 | 89.1 (94.9) [89.1] |
| Hedges’ g, Eq. (22) | 0.003 | 0.337 | 95.0 (95.3) [97.4] | −0.002 | 0.141 | 95.3 (94.9) [95.7] | −0.004 | 0.405 | 92.1 (94.7) [95.0] | 0.002 | 0.261 | 89.1 (95.0) [89.3] |
Abbreviation: CP, coverage probability of 95% confidence interval; RMSE, root mean squared error.
Notation: N, no. of studies; τ, between-study standard deviation; ni, study-specific sample size; θ, true overall standardized mean difference.
Note: coverage probabilities include those based on conventional meta-analysis method in Equation (23) (outside parentheses and square brackets), based on the Hartung–Knapp–Sidik–Jonkman method in Equation (24) (inside parentheses), and based on the method in Equation (26) (inside square brackets).