Table 2. Adaptive iterative reviewing: IED nominations are iteratively presented to the reviewer one-by-one using 10 events per iteration.
Subject | #IEDs | IEDs confirmed up and until iteration | ||||||||||||||
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | ||
S1 | 13 | 8 | 11 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 |
S2 | 11 | 4 | 6 | 6 | 8 | 8 | 8 | 8 | 8 | 9 | 9 | 9 | 9 | 9 | 9 | 9 |
S3 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 |
S4 | 36 | 4 | 8 | 14 | 17 | 20 | 24 | 27 | 29 | 30 | 31 | 32 | 32 | 33 | 33 | 35 |
S5 | 19 | 8 | 12 | 15 | 18 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 |
S6 | 6 | 4 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 6 | 6 | 6 |
S7 | 79 | 9 | 17 | 26 | 36 | 45 | 52 | 58 | 61 | 62 | 68 | 68 | 70 | 72 | 74 | 74 |
S8 | 5 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 5 | 5 | 5 | 5 |
S9 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
S10 | 12 | 3 | 10 | 11 | 11 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 |
S11 | 7 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
S12 | 7 | 3 | 4 | 5 | 6 | 6 | 6 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 |
S13 | 19 | 10 | 17 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 | 19 |
S14 | 14 | 9 | 12 | 12 | 12 | 13 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 |
S15 | 6 | 5 | 5 | 5 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 |
TOTAL | 241 | 78 | 120 | 144 | 164 | 179 | 191 | 201 | 206 | 209 | 216 | 217 | 220 | 224 | 226 | 228 |
MEAN | 32% | 50% | 60% | 68% | 74% | 79% | 83% | 85% | 87% | 90% | 90% | 91% | 93% | 94% | 95% |
Nomination certainties are updated after every iteration, thereby allowing nominations with higher likelihoods to be shown first. Template reliabilities were also updated at the end of each review. This was done by updating the true and false positives of contributing templates and recalculating their reliability scores accordingly, as shown in Appendix S1.