Automatic Extraction of Drug Indications from FDA Drug Labels

Abstract

Extracting computable indications, i.e. drug-disease treatment relationships, from narrative drug resources is the key for building a gold standard drug indication repository. The two steps to the extraction problem are disease named-entity recognition (NER) to identify disease mentions from a free-text description and disease classification to distinguish indications from other disease mentions in the description. While there exist many tools for disease NER, disease classification is mostly achieved through human annotations. For example, we recently resorted to human annotations to prepare a corpus, LabeledIn, capturing structured indications from the drug labels submitted to FDA by pharmaceutical companies. In this study, we present an automatic end-to-end framework to extract structured and normalized indications from FDA drug labels. In addition to automatic disease NER, a key component of our framework is a machine learning method that is trained on the LabeledIn corpus to classify the NER-computed disease mentions as “indication vs. non-indication.” Through experiments with 500 drug labels, our end-to-end system delivered 86.3% F1-measure in drug indication extraction, with 17% improvement over baseline. Further analysis shows that the indication classifier delivers a performance comparable to human experts and that the remaining errors are mostly due to disease NER (more than 50%). Given its performance, we conclude that our end-to-end approach has the potential to significantly reduce human annotation costs.

Introduction

Drug-disease treatment relationships, i.e. drugs and their indications, are among the top searched topics in PubMed^1,2. The primary application of this information is to inform healthcare professionals and patients for questions³ like “what are the indications of fluoxetine.” Such information is also valuable for developing computational methods for predicting and validating results of novel drug indications^4–6 and drug side effects⁷, controlling data-entry and medication errors in the electronic medical records^8–10, feeding the Google Knowledge Graph, and assisting PubMed Health^® (http://www.ncbi.nlm.nih.gov/pubmedhealth/) editors to cross-link drug and disease monographs¹¹. Given the variety of applications, it is important to build a structured and normalized drug indication repository, or a computable “gold standard” of drug-disease relationships, from credible drug resources. Most existing high-quality drug resources, such as DailyMed¹², DrugBank¹³, MedlinePlus¹⁴, and MedicineNet¹⁵ are described in free text. Figure 1 presents excerpts from DailyMed descriptions, aka FDA drug labels, for two drugs.

Figure 1.

Illustration of Disease Named Entity Recognition (NER) and Classification Problems (Source: DailyMed Drug Labels)

Figure 1 also illustrates the two key steps in creating a drug indication repository from existing free-text resources: (i) disease named entity recognition (NER) to identify all the disease mentions (underlined) from drug narratives, and (ii) disease classification to distinguish indications from other disease mentions, demonstrated using checks for yes and ‘X’ for no. Disease NER, in general, is not a new problem; there exist high-performing and state-of-the-art tools such as MetaMap¹⁶, DNorm^{17, 18}, and KMCI¹⁹ that automatically identify UMLS disease concepts from biomedical or clinical narratives. Also, many text-mining methods^20–25 have been proposed to extract disease mentions from drug resources such as DailyMed, MedlinePlus, and even Wikipedia.

The disease classification problem has not received much attention, however. Through detailed analysis of 100 FDA drug labels, we learned that even in the designated “INDICATIONS AND USAGE” section, about half of the disease mentions are not indications and include false positives such as characteristics of indications, contraindications, side effects, usages of another drug, unrelated diseases, etc²⁶. There have been a few earlier attempts to address the disease classification problem in context of building a drug indication repository, e.g. SIDER-2 contains structured indications from FDA drug labels by filtering the disease mentions that overlap with its side effects repository²³, and Wei et al.²¹ classify a given disease mention as an indication based on its frequency of occurrence across multiple structured and unstructured resources. However, the performances of these hand-crafted rule-based methods are limited either in terms of precision²³ or recall²¹. More recently, we resorted to manual expert annotations to identify indications from pre-recognized diseases in FDA drug labels, and curated a source-linked resource called LabeledIn^{24, 26}. Upon systematic comparison with an existing automatically-curated resource²⁷, we concluded that human judgment is a reliable solution to the classification problem. However, it is time-consuming, and hence, expensive to be scalable with respect to building a repository with wide coverage of drugs.

In this study, we present an automatic end-to-end indication extraction framework that given an FDA drug label, (1) extracts the relevant section, (2) recognizes and normalizes all the disease mentions, (3) more importantly, classifies the disease mentions as “indications” vs. “non-indications,” and (4) outputs the indication mentions and corresponding UMLS disease concepts that could be directly used to populate the target drug indication repository. The main contribution of this work is the use of supervised machine learning to address the under-explored disease classification problem in (3) and achieve a performance that is comparable to human experts.

Methods

The overall framework to extract drug indications from FDA drug labels is shown in Figure 2. As our data source, we use the FDA drug labels from the DailyMed website¹², which contains the most up-to-date drug labels submitted to FDA by drug manufacturers. The output of the framework is a set of structured and precisely normalized drug-disease relationships that can be used to automatically populate a computable drug indication repository. In the following subsections, we describe our disease NER and classification methods in further detail.

Figure 2.

Overall Framework to Extract Drug Indications from FDA Drug Labels

Disease Named Entity Recognition

The goal of the method is to identify all the disease mentions, or indication candidates, from the textual descriptions of a given drug label. For this purpose, we prepared a disease lexicon using two seed ontologies, MeSH and SNOMED-CT, respectively useful for annotating scientific articles^{17, 28, 29} and clinical documents^30–32. The lexicon consists of 77,464 concepts taken from: (i) the disease branch in MeSH, and (ii) the 11 disorder semantic types (UMLS disorder semantic types excluding ‘Finding’) in the SNOMED-CT as recommended in a recent shared task ³⁰. As for the automatic tool, we applied MetaMap¹⁶, a highly configurable program used for mapping biomedical texts to the UMLS identifying the mentions, offsets, and associated CUIs. We experimented with multiple settings of MetaMap, and the optimal setting for this study is illustrated in Figure 3.

Figure 3.

Disease Named Entity Recognition (NER) Method

The drug labels may contain overlapping disease mentions, e.g. the phrase “skin and soft tissue infections” denotes two specific diseases, “skin infections” and “soft tissue infections.” While the final results by MetaMap do not return such overlapping mentions, these are captured in the intermediate results of MetaMap, known as the Metathesaurus candidates. Hence, we utilized these candidate concepts in our method. MetaMap provides two types of candidate concepts: contiguous and dis-contiguous, e.g. in the phrase “skin and soft tissue infections”, “soft tissue infections” is a contiguous candidate, and “skin+infections” is a dis-contiguous candidate. We found that MetaMap returns different sets of dis-contiguous candidates with and without the term processing feature. Hence, we conducted two runs of MetaMap for comprehensive results. Also, the word sense disambiguation feature was turned on to disambiguate mentions that may map to multiple CUIs, e.g. “depression.” In order to restrict the returned candidates to specific semantic types from two vocabularies as mentioned above, we used a lookup against our custom disease lexicon as opposed to running multiple rounds of MetaMap for the two vocabularies. Finally, candidates with overlapping spans (e.g. “moderate to severe pain”) were resolved in the following manner: (i) when both candidates were contiguous, the longer candidate was selected, (ii) when one candidate was dis-contiguous - (a) if the merged span contained conjunctions (e.g. “or,” “and”) or prepositions (e.g. “to”), then the merged span was pre-annotated and both CUIs were retained, e.g. the elliptical coordination in “skin and soft tissue infections,” (b) if the two mentions were related by a parent-child UMLS relationship (e.g., the phrase “acute bacterial otitis media” maps to hierarchically related concepts “acute + otitis media” and “otitis”), then the longer mention was retained, else, the shorter mention was retained (e.g. the phrase “drug hypersensitivity reactions” maps to non-hierarchically related concepts “drug + reactions” and “hypersensitivity reactions”).

Disease Classification

Next, we model indication extraction as a binary classification problem that judges whether a disease mention identified through NER method is an indicated usage of the drug. To build a high-quality gold standard, it is very important to judge and remove irrelevant disease mentions, e.g. “Difficulty falling asleep” from drug label A, and “cancer” from drug label B in Figure 1. This problem could also be perceived as another flavor of classifying diseases in clinical narratives into linguistic or temporal bins^{33, 34}.

To address the classification problem, we used the support vector machines (SVM), a discriminative binary classifier known to be highly accurate and widely used in text classification. Given a set of data points (or examples) with known labels, the SVM assigns scores to each data point and finds a hyperplane that partitions the examples into two distinct bins, positive and negative based on the assigned score. For this problem, a data point is a disease mention at a given location, e.g. in Figure 1, the drug label A has five data points, and drug label B has three data points. We hand-picked the classification features based on our previous study with human experts, experimented with several features, and selected a set of linguistic, contextual, semantic, and dictionary-based features for this problem as described and exemplified in Table 1. Other features considered but not selected for the final classifier include the relative location of sentence in the drug label and the presence of drug name in the sentence containing the mention.

Table 1.

SVM Features for Disease Classification Problem

Feature (Description)	Example1		Example2
	Data Point	Feature Value(s)	Data Point	Feature Value
Mention (Represented Surface-value)	Difficulty falling asleep (Drug Label A in Fig 1)	Difficulty falling asleep	Vomiting (Drug Label B in Fig 1)	Vomiting
Neighboring Tokens (left/right 5 tokens within sentence)		may, frequently, be, associated, with		the, prevention, of, -DISEASE-, and, associated, with, initial, and, repeat
Location (of sentence)		2		1
NDF-RT Match (the relationship of this disease concept with a concept catalogued in NDFRT35 for this drug)		No Match		Exact Match
Semantic Category (of the UMLS concept corres.to mention)		Mental or Behavioral Dysfunction		Sign or Symptom

To train the SVM, we relied on a recently curated high-quality annotated text corpus, LabeledIn²⁴, that contains labeled or marketed indications for 250 frequently searched drug ingredients on PubMed Health. LabeledIn is unique among existing structured drug indication resources^{21, 22, 27, 35, 36} in that it is human-validated, source-linked, and normalized to the most precise concepts in widely used UMLS vocabularies (SNOMEDCT, MeSH, and RxNorm). LabeledIn was created through double-annotation (88% kappa agreement) of 500 drug labels with assistance from a high-recall disease NER tool. In particular, for each drug label, the expert was presented with all disease mentions identified using an NER tool and was asked to assign a yes/no judgment. At the backend, the offsets of the disease mentions and the associated expert judgments were recorded. This helped in creating a fine-grained annotated text corpus that contained all disease mentions along with the associated human judgments. This becomes the training dataset for this problem as described in first row of Table 2. Since our goal is to automatically populate a computable gold standard of drug indications, we evaluate the results of the classifier at concept-level. For this purpose, we created an evaluation dataset that contains all the disease concepts (CUIs) present in a given drug label and the respective expert-determined yes/no assignments; this dataset is described in the second row of Table 2. There is a clear distinction between distribution of examples in both datasets because a mention identified as positive is likely to be repeated several times in a given drug label, e.g. the CUI represented by “RLS” has four occurrences in drug label A in Figure 1.

Table 2.

Classification Data Points for 500 FDA Drug Labels

Data Point Definition	#Data Points (+,−)
Disease Mentions with offset	5,336 (70%, 30%)
Disease Concepts (UMLS CUIs)	3,013 (55%, 45%)

Results

We first measured the baseline performance on 500 drug labels from LabeledIn by assuming all diseases to be indications. The result is shown in the first row of Table 3. The result is a high recall and low precision. The recall is less than perfect because of the cases where the NER method identified a less-specific concept than annotated by the human experts, e.g., from the phrase “Acute Bacterial Otitis Media,” the NER identified “Otitis Media” whereas annotators expanded the indication to the entire phrase.

Table 3.

Indication Extraction Performance at Concept-level (micro-averaged)

Indication Extraction Method	Precision(%)	Recall(%)	F1-measure(%)
Disease NER (baseline)	54.99	93.93	69.37
Disease NER + NB	80.16	79.65	79.91
Disease NER + ME	85.02	86.16	85.59
Disease NER + SVM	86.99	85.58	86.28

We then conducted the experiment including a classification method in the framework. We experimented with three different classifiers, SVM, Maximum Entropy (ME), and Naïve Bayes (NB). We used an internal C++ implementation of the classifiers to conduct the experiments with 5,336 training examples and used 10-fold cross validation for evaluation against the evaluation dataset. To align the classifier results with our concept-level evaluation benchmark, we post-processed the results where multiple occurrences of the same concept were classified differently, e.g. SVM may classify the two occurrences of “RLS” in drug label A (Figure 1) as positive and negative, respectively. We resolved this conflict by preferring the positive over negative classification. Finally, we selected SVM as it delivered the highest overall classification accuracy (optimized at score threshold 0.4) and hence leads to the highest indication extraction F1-measure on our data as shown in Table 3. The importance of each feature used for SVM is shown through the feature ablation experiment in Table 4 listing the features in decreasing order of their importance.

Table 4.

Results of Feature Ablation Experiment (SVM)

Ablated (Removed) Feature	F1-measure(%)
Neighbors	79.75
Mention	83.09
Location	84.97
NDF-RT Match	85.11
Semantic Category	86.01

Using the results from the SVM-based extraction method, we randomly selected the 100 erroneous cases (50/220 false positives and 50/143 false negatives) for further analysis. Figure 4 shows the different categories of false positive errors that we observed. The two largest categories (shown separately) correspond to errors due to the NER component of the framework: either only partly identifies the correct concept (Boundary Error) or confuses with some other entities (Not a Disease Mention). About 36% of false positives represent boundary errors, e.g. from the phrase “carcinoma of the prostate,” the disease tagger identified “carcinoma” and the classifier identified this mention as positive. This example is considered as a false positive as it does not match with our gold standard that contains the complete mention “carcinomas of the prostate.” About 20% of false positives include non-disease mentions, such as part of an organization name (e.g. the “cancer” society), disease homonym (“seizure” used in a non-biological context), etc.

Figure 4.

Analysis of 50 Randomly Selected False Positives

The remaining categories of false positive errors are due to indication misclassification as described next. About 18% include the diseases mentioned in some other context and unrelated to actual indications. Another 18% category represents the advanced cases, such as pre-existing conditions and conditions that cause indications. In our earlier study to create LabeledIn using expert annotations, we found some situations where experts needed to refer to additional references and drug properties to make their judgment. Consider the two drug labels in Figure 5: the “HIV infection” and “epilepsy” have similar feature values, e.g. neighbor tokens and semantic categories; however, “HIV infection” should be identified as a negative example and “epilepsy” as positive. Such cases are difficult to learn by a domain-independent classifier. Finally, the remaining few include characteristics of indications, contraindications, or side effects. In terms of false negatives, less than 5% were because of NER tool limitations, and the remaining represent misclassification errors because of poor structure of sentences (such as lists) and noisy neighbors (such as parenthesis), advanced cases, etc.

Figure 5.

Example of Training Dataset showing an advanced case of classification (“HIV infection” and “epilepsy”)

Since our final goal is to minimize human effort, we also studied how many concepts can be correctly classified by the SVM, and thereby eliminated from human judgment in a traditional machine-assisted annotation pipeline²⁶, without incurring any errors. We investigated whether the scores (or ranks) assigned by SVM could be used to identify some perfectly classified examples. Figure 6 shows the variation of error rate with respect to the proportion of highest/lowest ranked concepts bypassed from further human validation; we found that we could safely eliminate 2.2% top-scored disease concepts and 5.1% bottom-scored disease concepts from further human validation.

Figure 6.

Effect of Eliminating Highest/Lowest Ranked Concepts on Error Rate

Conclusion

In this study, we have addressed the disease recognition and classification problems to prepare a comprehensive drug indication repository serving many critical applications. In the context of curation of a drug indication repository, there have been many important efforts primarily based on automatic recognition of disease mentions from drug narratives^20–24. However, an important next step of classifying the automatically identified diseases as “indication”/“not indication” is largely missing from existing work, and hence the results from existing methods suffer from lower precision requiring further validation and cleaning by human experts. Taking a step forward, this study presents a framework that not only recognizes disease mentions from drug narratives, but further distinguishes indications from other disease mentions using an SVM-based classification method, thus minimizing the need to have humans in the pipeline. We experimented with 500 FDA drug labels corresponding to frequently-sought human drugs, and configured a disease NER tool to deliver an optimal recall on drug labels. We find that the combination (recognition + classification) framework achieves 32% improvement in precision and 17% improvement in F1-measure over a baseline method that is based only on recognition.

The proposed framework delivered an 86% F1-measure on 500 drug labels whereas two human experts in a previous study using similar NER mechanism²⁶ delivered 88% joint F1-measure on 100 drug labels after their first round of annotation. We consider these performances to be equivalent to each other, given the upper limit (94% recall) on classification performance due to natural language limitations of the NER tool. We also observed that over half of the errors are due to the NER limitations. This clearly indicates that the proposed classifier can act as an independent or complementary annotator, saving annotation costs and time²⁶. Furthermore, we find that about 7% of the training examples fed to the classifier could be altogether bypassed from further human judgments without incurring any errors. In terms of improvement, we found that several erroneous cases required advanced domain knowledge and are difficult to learn. Also, many false negative errors could be resolved by refining the neighbor calculation algorithm and adding formatting features to the classifier. Our future work also includes improving the technical aspects of the classifier such as automatic feature generation and handling imbalanced data. Lastly, we observed that certain annotated drug indications are specific to certain procedures/conditions (e.g. “nausea” and “vomiting” are associated with “cancer chemotherapy” in Figure 1 drug label B). Such information is not yet captured through NER or classification modules of our current pipeline, and requires further processing of drug labels.