Table 2.
Overview information on the considered combinations of reference sets and datasets
| Reference set | Positive controls | Negative controls | Dataseta | Positive controls N ≥ 1b n (%) |
Positive controls N ≥ 3b n (%) |
Negative controls N ≥ 1b n (%) |
Negative controls N ≥ 3b n (%) |
|---|---|---|---|---|---|---|---|
| Harpaz | 62 | 75 | VigiBase | 41 (66) | 29 (47) | 36 (48) | 24 (32) |
| Social 0.4 | 13 (21) | 5 (8) | 17 (23) | 8 (11) | |||
| Social 0.5 | 8 (13) | 5 (8) | 8 (11) | 2 (3) | |||
| Social 0.6 | 3 (5) | 2 (3) | 2 (3) | 2 (3) | |||
| Social 0.7 | 3 (5) | 1 (2) | 2 (3) | 2 (3) | |||
| Social 0.8 | 3 (5) | 1 (2) | 2 (3) | 2 (3) | |||
| Social 0.9 | 3 (5) | 1 (2) | 2 (3) | 2 (3) | |||
| Social 0.99 | 3 (5) | 1 (2) | 2 (3) | 2 (3) | |||
| WEB-RADR | 200 | 5332 | VigiBase | 197 (98) | 180 (90) | 5072 (95) | 3853 (72) |
| Social 0.4 | 98 (49) | 75 (38) | 2527 (47) | 1879 (35) | |||
| Social 0.5 | 85 (42) | 56 (28) | 2294 (43) | 1653 (31) | |||
| Social 0.6 | 46 (23) | 26 (13) | 1461 (27) | 879 (16) | |||
| Social 0.7 | 42 (21) | 20 (10) | 1345 (25) | 772 (14) | |||
| Social 0.8 | 37 (18) | 19 (10) | 1267 (24) | 679 (13) | |||
| Social 0.9 | 35 (18) | 17 (8) | 1216 (23) | 624 (12) | |||
| Social 0.99 | 34 (17) | 14 (7) | 1176 (22) | 585 (11) | |||
| Forum posts | 61 (30) | 28 (14) | 1657 (31) | 886 (17) |
a‘Social 0.X’ means social media data from Twitter and Facebook, with a post-level threshold on the indicator score of 0.X. For forum posts, an indicator score threshold of 0.7 was used
bThese figures refer to the specific time points at which data were extracted for positive and negative controls for the purposes of receiver operating characteristic analysis