Table 5.
Semi-automation test with RobotAnalyst using a training set of dually reviewed citations from a review-of-review with labels from full-text screening
|
Traditional search Total citations: 3181 Title-abstract screening: 201 includes/2980 excludes Full-text screening: 59 includes/3122 excludes |
||||
|---|---|---|---|---|
|
Review of reviews Total SR citations: 30 Total primary study citations: 95 Title-abstract screening: 76 includes/49 excludes Full-text screening 33 includes/43 excludes Number of citations not screened: 3030 |
Training set: ROR citations Labeled citations: 125 (4%) Training set labels: TP (33), FP (0), TN (92) Unlabeled citations assigned inclusion prediction by ML algorithm: 3056 |
|||
| Inclusion prediction: 0.3 | Inclusion prediction: 0.4 | Inclusion Prediction: 0.5 | ||
| Predicted includes | NA | 3040 | 2819 | 1166 |
| Predicted excludes | NA | 16 | 237 | 1890 |
| Sensitivity | 54% | 100% | 97% | 69% |
| Specificity | 100% | 3% | 10% | 63% |
| Missed citations | 26 | 0 | 2 | 18 |
| Burden | 4% | 99% | 93% | 41% |
| Time savings (min) | 1561 | 8 | 118.5 | 945 |
FP false positive, ML machine learning, NA not applicable, TP true positive, TN true positive