Table 2.
Results from simulation analyses for datasets labeled by clinicians, research methodologists, and full-text selection
| # | N | SelectCl | SelectEx | SelectFT | WSS95Cl | WSS95Ex | WSS95FT | RRF10Cl | RRF10Ex | RRF10FT |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 195 | 11 | 6 | 5 | 32.31 (6.37) | 57.70 (4.80) | 61.41 (2.14) | 29.09 (8.31) | 30.00 (16.73) | 20.00 (11.18) |
| 2 | 277 | 8 | 4 | 4 | 43.33 (5.47) | 59.40 (6.82) | 62.31 (9.15) | 28.57 (13.23) | 25.00 (16.67) | 33.33 (27.22) |
| 3 | 640 | 20 | 14 | 12 | 55.76 (2.54) | 73.55 (1.54) | 77.40 (2.45) | 43.16 (5.56) | 52.75 (7.90) | 62.88 (10.59) |
| 4 | 311 | 34 | 20 | 11 | 73.15 (1.43) | 72.98 (2.48) | 78.12 (4.10) | 66.22 (4.10) | 71.32 (11.39) | 73.64 (9.24) |
| 5 | 56 | 18 | 9 | 8 | 48.89 (0.00) | 70.12 (3.35) | 70.28 (3.13) | 28.10 (2.52) | 45.83 (12.50) | 41.07 (5.05) |
| 6 | 188 | 18 | 12 | 7 | 40.33 (2.47) | 45.34 (8.99) | 86.76 (1.78) | 47.06 (6.69) | 40.15 (11.92) | 78.57 (15.85) |
| 7 | 435 | 109 | 22 | 6 | 32.70 (1.25) | 66.10 (1.08) | 64.19 (11.75) | 25.63 (3.31) | 62.55 (5.93) | 46.67 (20.66) |
| 8 | 428 | 45 | 45 | 45 | 66.42 (1.26) | 66.34 (1.08) | 66.92 (1.23) | 60.45 (5.34) | 61.26 (5.52) | 60.86 (5.31) |
| 9 | 342 | 3 | 1a | 1a | 97.99 (0.70) | NA | NA | 100.00 (0) | NA | NA |
| 10 | 397 | 6 | 4 | 4 | 74.78 (2.53) | 93.78 (0.87) | 93.13 (0.15) | 63.33 (8.16) | 100.00 (0) | 100.00 (0) |
| 11 | 218 | 6 | 5 | 4 | 79.55 (1.82) | 82.80 (0.43) | 79.95 (2.51) | 46.67 (20.66) | 40.00 (13.69) | 33.33 (0) |
| 12 | 335 | 5 | 5 | 4 | 61.30 (14.84) | 61.05 (14.20) | 96.68 (0.92) | 65.00 (22.36) | 60.00 (33.54) | 100.00 (0) |
| 13 | 480 | 35 | 16 | 9 | 65.09 (4.03) | 73.12 (4.00) | 95.76 (0.34) | 78.74 (10.14) | 89.17 (16.67) | 100.00 (0) |
| 14 | 772 | 21 | 10 | 8 | 34.84 (16.34) | 95.72 (0.32) | 96.27 (0.48) | 85.95 (19.72) | 100.00 (0) | 100.00 (0) |
| Total | 5074 | 339 | 173 | 128 | 50.15 (17.14) | 69.24 (11.51) | 75.76 (12.16) | 48.31 (23.32) | 62.78 (21.20) | 65.58 (23.25) |
aA simulation study with only 1 relevant paper cannot be executed because it needs at least 1 relevant record as training data and 1 relevant record as target paper to detect
SelectCl, Ex, FT = number of records included by the clinician, research methodologist, and full-text selection
WSS95Cl, Ex, FT = Work Saved over Sampling measured at a recall level of 95% for dataset labeled by the clinician, research methodologist, and full-text selection
RRF10Cl, Ex, FT = Relevant References Found after screening 10% of all publications (RRF10) for dataset labeled by the clinician, research methodologist, and full-text selection