Table 4.

Recommendations relating to adverse event recognition in social media data

Recommendation	Observations
For methods developed for AE recognition in social media, evaluate its performance on a standard reference data set such as that produced by WEB-RADR, to facilitate comparison of methods	We compared the classification performance of the NLP workflow for medicinal product–AE pair recognition, when evaluated on an independent sample from the same dataset that was used for training of the predictive models in the workflow with the performance on the AE recognition reference set. Recall dropped from 52% to 20%, and precision from 53% to 38%. Evaluation of a previously published method for detection of AE posts also displayed a drop in performance: from 50% precision and 92% recall (0.65 F1 score) in the publication to 37% precision and 63% recall (0.46 F1 score) on the WEB-RADR reference dataset. This illustrates the risk of overestimating the classification performance of a method if an independent dataset from another context is not used in the evaluation
Consider the use of machine learning technology to support the recognition of social media data relevant for pharmacovigilance	Less than 2% of tweets assessed in the development of the AE recognition reference set contain AE terms [Dietrich 2019, submitted]. A large proportion of irrelevant data will exist in any social media dataset. As such, employing automated processes may enable AE recognition whilst reducing the effort required for manual review
Human curation should be used in conjunction with automated processes aimed at identifying potential AEs from social media with methods available today	The NLP workflows for medicinal product–AE recognition and coding were evaluated to have a precision equal to 38%. This means that the majority of automatically recognised medicinal product–AE pairs are incorrectly classified. Human curation has the potential to detect and discard such pairs and thereby increase the precision. The content of social media posts underlying signals of disproportionate reporting (SDRs) in the signal detection study was inspected and found to be severely lacking in content and interpretability. In fact, of the posts inspected, only 40% of posts contained the correct drug and the correct event as an adverse experience, pointing to a significant issue with ADR recognition [15]
If available, use existing mappings between verbatim text and MedDRA® terms from spontaneous reporting systems to improve sensitivity in medical event recognition for social media	Our study showed that the inclusion of historical mappings from VigiBase verbatims to a dictionary of MedDRA® LLTs almost tripled the number of captured AEs. Generalisability beyond VigiBase as a source of mappings and Twitter as the domain of application is unknown
Consider incorporating information on medicinal product indications in automated AE recognition, thereby reducing the likelihood of falsely categorising an indication as an AE	In the AE recognition reference set, 18.5% of patients mention indications for use of a medication in conjunction with product names and AEs [Dietrich 2019 submitted]. In addition, patients may describe symptoms of their underlying conditions and AEs in the same post, making it difficult for automated processes to determine which medical conditions or symptoms are AEs versus related to a product’s indication. Absolute removal of indication-related posts may not be beneficial or result in more accurate automated coding; for example, in a post where a patient mentions that a medical product aggravated the condition that the medicine is meant to be treating

ADRs adverse drug reactions, AEs adverse events, LLTs Lowest Level Terms, NLP natural language processing