Table 4.
Semi-automation test with RobotAnalyst using a training set of dually reviewed randomly selected citations with labels from title and abstract screening
|
Traditional database search Total citations: 3128 Title-abstract screening: 148 includes/2980 excludes Full-text screening: 46 includes/3082 excludes |
|||
|---|---|---|---|
|
Training set Labeled citations: 938 (30%) Training set labels: TP (15), FP (32), TN (891) Unlabeled citations assigned inclusion prediction by ML algorithm: 2190 | |||
| Inclusion prediction: 0.3 | Inclusion prediction: 0.4 | Inclusion prediction: 0.5 | |
| Predicted includes | 2168 | 1970 | 1363 |
| Predicted excludes | 22 | 220 | 827 |
| Sensitivity | 100% | 93% | 74% |
| Specificity | 30% | 36% | 55% |
| Missed citations | 0 | 3 | 12 |
| Burden | 99% | 93% | 74% |
| Time savings (min) | 11 | 110 | 413.5 |
FP false positive, ML machine learning, TP true positive, TN true positive