Table 1.
Terminology | Description |
---|---|
Estimated recall | The estimated percent of how many studies at title/abstract level have been identified among those that will be passed through to full-text screening. As this is calculated based on a set of records that have not been completely screened, the estimated recall may differ from the true recall. |
Final include | A primary study included in the completed systematic review. |
Iteration | A set of records that is used to assign a score around the likeliness of inclusion and prioritize the remaining unscreened records in order from highest relevance to lowest relevance. |
Modified screening approach | An approach to modify how screening is being performed. For example, changing from: (i) dual-independent screening to liberal accelerated screening; (ii) dual-independent screening to single-reviewer screening; or (iii) assigning the remaining records to the AI reviewer to exclude, with a human reviewer(s) also screening these records as a second reviewer. |
Prioritized screening | Through active machine learning, the presentation of records to reviewers is continually adjusted based on the AI’s estimated likelihood of relevance. The frequency of adjustment may differ by software application. |
Screening burden | The total number of records at title/abstract to be screened. |
Stop screening approach | An approach to screening whereby the remaining records are not screened once a certain threshold has been achieved (e.g., estimated recall @ 95%). These records are assumed to be excluded. |
Record not yet identified [i.e., title/abstract false negative (FN)] | When an estimated recall (at any %) or true recall of less than 100% is used, these are the records that would have been included based on the title/abstract to be further reviewed at full-text screening, but were not yet identified. Had these records been screened at title/abstract and further screened based on the full text, they may have been excluded or included in the final review (i.e., a final include). |
Title/abstract include [i.e., title/abstract true positive (TP)] | Records included based on the title/abstract to be further reviewed based on the full text. These records may then be excluded at full-text review or included in the final review. |
Training set | One or more iterations which inform the machine learning to score and prioritize the remaining unscreened records. |
Title/abstract exclude [i.e., true negative (TN)] | Records considered excluded based on title/abstract screening. |
True recall |
This is only known once all references have been screened and includes the percentage of the actual number of records that were title/abstract includes. True recall % calculated as: [title/abstract TP / (title/abstract TP + title/abstract FN)] |