Table 3.
Interreviewer agreement on the titles-first screening step versus the title and abstract screening step, by reviewer pair
| Reviewer pair | 1 Neurologist A and pulmonologist |
2 Pulmonologist and neurologist B |
3 Neurologist B and gynecologist/obstetrician |
4 Gynecologist/obstetrician and general surgeon |
5 General surgeon and neurologist A |
Overall |
|---|---|---|---|---|---|---|
| Citations reviewed (n) | 586 | 584 | 614 | 588 | 593 | 2965 |
| % agreement on titles-alone classification (95% CI) | 90.6 (88.2–93.0) | 87.8 (85.2–90.5) | 93.8 (91.9–95.7) | 93.5 (91.5–95.5) | 93.7 (91.8–95.7) | 91.9 (91.0–92.9) |
| Kappa statistic on titles-alone classification (95% CI) | 0.51 (0.40–0.62) | 0.40 (0.30–0.51) | 0.63 (0.52–0.74) | 0.67 (0.57–0.77) | 0.53 (0.39–0.66) | 0.54 (0.49–0.59) |
| % agreement on titles and abstracts classification (95% CI) | 96.2 (94.7–97.8) | 95.3 (93.7–97.1) | 95.2 (94.4–97.5) | 97.4 (96.2–98.7) | 96.6 (95.2–98.1) | 96.3 (95.6–97.0) |
| Kappa statistic on titles and abstracts classification (95% CI) | 0.56 (0.39–0.72) | 0.43 (0.25–0.60) | 0.56 (0.40–0.71) | 0.72 (0.59–0.86) | 0.49 (0.29–0.68) | 0.56 (0.49–0.63) |
Note: Percent agreement = (number of citations with a consensus of a “yes” “yes” or “no” “no” response in a reviewer pair)/total number citations screened.
Abbreviation: CI, confidence interval.