Table 3.
Recommendations relating to the use of social media for signal detection
| Recommendation | Observations |
|---|---|
| When evaluating signal detection algorithms for social media data, complement overall performance analyses (e.g. receiver operating characteristics) with manual post-level assessment as a sanity check: if a large proportion of the automatically identified posts do not contain the medicinal product or adverse event term indicated, overall results must be interpreted with considerable caution | During manual inspection of post text corresponding to a social media signal, only 39.6% of the posts contained the drug and medical event of interest as an actual adverse experience. In the subset of posts with indicator score of 0.7 or above, the corresponding result was 67.3% (72 of 107 posts) |
| In evaluation of signal detection methods, proprietary reference sets should be avoided if possible | Practically, working with our WEB-RADR SD reference set has been very cumbersome since all data extraction had to be performed locally at several different sites by those authorised to access the de-anonymised controls. Further, such a reference set cannot be critically inspected or re-used outside the specific study where it was used. Finally, certain types of analyses become impossible to perform, such as aggregation based on characteristics of the medicines or adverse event terms |
| If setting up a safety surveillance system based on social media today, it is more important to first improve and calibrate adverse event recognition than the algorithms for statistical signal detection | We have generally seen small differences between different algorithms and in our exploratory study, a more advance method like SbD provided no added benefit [15, 23]. No signal detection algorithm can extract information unless the data it depends on are of adequate quality and are well calibrated. In both the Caster and Dietrich studies, medical event recognition was a significant hurdle [15] [Dietrich submitted 2019] |
SbD ‘Signal Before Detect’ algorithm, SD signal detection