Table 4.

Evaluation and performance metrics

Author^<	NLP/Text Mining	Primary Evaluation Metric	Comparative Evaluation
Brennan & Aronson, 2003¹⁶	NLP	Number of matched terms to vocabularies; reported matched frequencies higher for nursing complemented vocabularies	Compared several models on nursing, MeSH, and SNOMED terms
Portier et al, 2013³³	Text mining	Descriptive results including sentiment analysis	None reported
Freifeld et al, 2014²⁷	NLP	Automated, dictionary-based symptom classification had 72% recall and 86% precision	Results of annotation were compared to FDA Adverse Event Reporting System data
Gupta et al, 2014¹⁹	NLP	Extracts symptoms and conditions with a F-measure of 66–76%	Compared performance of two other programs, the OBA and the MetaMap annotator, with baseline and default parameters
Park & Ryu, 2014²⁵	Text mining	Descriptive results including symptoms and clinical distinctions	None reported
Janies et al, 2015²⁸	NLP	No reported algorithm evaluation metrics	None reported
Jimeno-Yepes et al, 2015²⁰	NLP	Highest performing model (Micromed+Meta) had precision, recall, and F-measure as 72%, 60%, and 66%, respectively	Compared across exact and partials matches for five models
Karmen et al, 2015²¹	NLP	Average precision of 84% and an average F-measure of 79%	Compared algorithm results to independent expert ratings
Liu & Chen, 2015²³	NLP	Average F-measure of 90% for drug entity extraction and average F- measure of 80% for medical event extraction	Compared several methods across patient-authored forums
Nikfarjam et al, 2015²⁴	NLP	86%, 78%, and 82% for precision, recall, and F-measure, respectively	Comparison between several methods including SVM, ADRMine, MetaMap, and a lexicon-based method
Tighe et al, 2015³²	Text mining	Descriptive results including the average degree centrality of the reduced pain tweet corpus graph was 60.7	Compared sentiment for relevant terms to objective terms
Eshleman & Singh, 2016¹⁸	NLP & text mining	Precision exceeding 85% and F- measure over 81%	Compared sentiment analysis to graph topology with co-occurring symptoms
Lee & Donovan, 2016³⁵	Text mining	Descriptive results for symptom findings	None reported
Marshall et al, 2016²⁹	Text mining	Descriptive results including cooccurrence and clustering for symptom findings	None reported
Topaz et al, 2016³⁰	NLP	Descriptive results including symptom extraction	None reported
Sunkureddi et al, 2016³⁴	NLP	Descriptive results including frequency ranking of reactions and patients’ concerns	None reported
Cocos et al, 2017¹⁷	NLP	Approximate match F-measure for RNN of 75% for ADR identification	Compared the BLSTM-RNN ADR classification, a baseline lexicon system, and a condition random-field model
Cronin et al, 2017³⁶	NLP	Logistic regression for medical communications with AUC of 0.899	Compared naive bayes, logistic regression, and random forest across different types of patient portal messages
Lamy et al, 2017²²	NLP	No reported algorithm evaluation metrics	None reported
Lu et al, 2017³¹	Text mining	Descriptive results including sentiment scores, clustering of groups, and Jaccard similarities	None reported
Patel et al, 2017²⁶	NLP	No reported algorithm evaluation metrics	Compared method between two datasets

Note.

Studies have been arranged in chronological order to assess trends over time; ADR=adverse drug reaction; ADRMine (a machine learning-based concept extraction system that uses Conditional Random Fields); AUC=area under the curve; BLST=Binarized Long Short-Term Memory Network; FDA=Federal Drug Administration; F-measure=also known as F1 score or F-score in the published literature; MeSH=Medical Subject Headings; NLP=Natural Language Processing; OBA=open biomedical annotator; MetaMap (tool for recognizing Unified Medical Language System [UMLS] concepts in text); RNN=recurrent neural network; SVM=support vector machine