Table 6.
Semi-automation test with RobotAnalyst using a training set of dually reviewed citations from a review-of-review and randomly selected citations with labels from full-text screening
|
Traditional search Total citations: 3181 Title-abstract screening: 201 includes/2980 excludes Full-text screening: 59 includes/3122 excludes |
|||
|---|---|---|---|
|
Training set: ROR citations + 30% random citations Labeled citations: 1063 (33%) Training set labels: TP (40), FP (0), TN (1023) Unlabeled citations assigned inclusion prediction by ML algorithm: 2118 |
|||
| Inclusion prediction: 0.3 | Inclusion prediction: 0.4 | Inclusion prediction: 0.5 | |
| Predicted includes | 2094 | 1765 | 676 |
| Predicted excludes | 24 | 353 | 1442 |
| Sensitivity | 98% | 84% | 80% |
| Specificity | 34% | 44% | 79% |
| Missed citations | 1 | 3 | 12 |
| Burden | 99% | 89% | 55% |
| Time savings (min) | 12 | 177 | 721 |
FP false positive, ML machine learning, TP true positive, TN true positive