Erratum
Following publication of our article [1], it has come to our attention that two of the formulae in Table 1 were incorrect. The formulae for the measures of precision and burden have been corrected (Table 1). We are publishing this erratum to update these formulae to the following:
Table 1.
Definitions of performance measures reported in the studies
| Measure | # | Definition | Formula |
|---|---|---|---|
| Recall (sensitivity) | 22 | Proportion of correctly identified positives amongst all real positives | |
| Precision | 18 | Proportion of correctly identified positives amongst all positives. | |
| F measure | 10 | Combines precision and recall. Values of β < 1.0 indicate precision is more important than recall, whilst values of β > 1.0 indicate recall is more important than precision | Where β is a value that specifies the relative importance of recall and precision. |
| ROC (AUC) | 10 | Area under the curve traced out by graphing the true positive rate against the false positive rate. 1.0 is a perfect score and 0.50 is equivalent to a random ordering | |
| Accuracy | 8 | Proportion of agreements to total number of documents. | |
| Work saved over sampling | 8 | The percentage of papers that the reviewers do not have to read because they have been screened out by the classifier | |
| Time | 7 | Time taken to screen (usually in minutes) | |
| Burden | 4 | The fraction of the total number of items that a human must screen (active learning) | |
| Yield | 3 | The fraction of items that are identified by a given screening approach (active learning) | |
| Utility | 5 | Relative measure of burden and yield that takes into account reviewer preferences for weighting these two concepts (active learning) | Where β is the user-defined weight |
| Baseline inclusion rate | 2 | The proportion of includes in a random sample of items before prioritisation or classification takes place. The number to be screened is determined using a power calculation | Where n i = number of items included in the random sample; n t = total number of items in the random sample |
| Performance (efficiency) a | 2 | Number of relevant items selected divided by the time spent screening, where relevant items were those marked as included by two or more people | |
| Specificity | 2 | The proportion of correctly identified negatives (excludes) out of the total number of negatives | |
| True positives | 2 | The number of correctly identified positives (includes) | TP |
| False negatives | 1 | The number of incorrectly identified negatives (excludes) | FN |
| Coverage | 1 | The ratio of positives in the data pool that are annotated during active learning | Where L refers to labelled items and U refers to unlabelled items |
| Unit cost | 1 | Expected time to label an item multiplied by the unit cost of the labeler (salary per unit of time), as calculated from their (known or estimated) salary | timeexpected × costunit |
| Classification error | 1 | Proportion of disagreements to total number of documents | 100 % − accuracy % |
| Error | 1 | Total number of falsely classified items divided by the total number of items | |
| Absolute screening reduction | 1 | Number of items excluded by the classifier that do not need to be manually screened | TN + FN |
| Prioritised inclusion rate | 1 | The proportion of includes out of the total number screened, after prioritisation or classification takes place | Where nip = number of items included in prioritised sample; ntp = total number of items in the prioritised sample |
Precision =
Burden =
Footnotes
The online version of the original article can be found under doi:10.1186/2046-4053-4-5.
Contributor Information
Alison O’Mara-Eves, Email: a.omara-eves@ioe.ac.uk.
James Thomas, Email: j.thomas@ioe.ac.uk.
John McNaught, Email: John.McNaught@manchester.ac.uk.
Makoto Miwa, Email: makoto-miwa@toyota-ti.ac.jp.
Sophia Ananiadou, Email: Sophia.Ananiadou@manchester.ac.uk.
References
- 1.O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Rev. 2015;4:5. doi: 10.1186/2046-4053-4-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
