Semantic Processing to Identify Adverse Drug Event Information from Black Box Warnings

Abstract

Adverse drug events account for two million combined injuries, hospitalizations, or deaths each year. Furthermore, there are few comprehensive, up-to-date, and free sources of drug information. Clinical decision support systems may significantly mitigate the number of adverse drug events. However, these systems depend on up-to-date, comprehensive, and codified data to serve as input. The DailyMed website, a resource managed by the FDA and NLM, contains all currently approved drugs. We used a semantic natural language processing approach that successfully extracted information for adverse drug events, at-risk conditions, and susceptible populations from black box warning labels on this site. The precision, recall, and F-score were, 94%, 52%, 0.67 for adverse drug events; 80%, 53%, and 0.64 for conditions; and 95%, 44%, 0.61 for populations. Overall performance was 90% precision, 51% recall, and 0.65 F-Score. Information extracted can be stored in a structured format and may support clinical decision support systems.

Introduction

The prescription drug market in the United States is the largest in the world at $266 billion dollars per year in annual sales in 2010 ¹. These lifesaving drugs help save patients’ lives and improve the quality of life for millions, but there are trade-offs. Each year there are as many as two million injuries, hospitalizations, and deaths from adverse drug events (ADEs) ². In fact, one study has listed ADEs of hospitalized patients as high as the 4th leading cause of deaths in the United States ³. Data suggest that patients who experience ADEs spend an average of 8.25% longer in the hospital with greater morbidity ⁴. In addition to the higher human suffering, these events lead to a substantial financial cost to an already financially stretched healthcare system. The total cost for ADEs in the United States is estimated at $75 billion annually ². The cost savings from these unnecessary tragedies could be invested in other potential cost saving technologies, such as health information exchange.

An adverse drug event is defined by the World Health Organization (WHO) as “any untoward medical occurrence that may present during treatment with a pharmaceutical product but which does not necessarily have a causal relationship with this treatment ⁵.” An ADE could best be described as an unintended negative outcome that occurs as a result of taking a drug. These could range from rash and hives for a mild reaction to severe reactions, such as respiratory complications or death. Information on the nature of the ADEs caused by a drug is essential in preventing negative outcomes. Furthermore, information that surrounds the ADE such as populations and conditions at greater risk, and what to do in case of an event are also of great importance in preventing the events themselves.

Given the magnitude of adverse drug events on our health care system, one proposed approach to improving efficacy and safety is to make better use of drug safety information. Currently, accurate, relevant, and easy to read drug information is difficult to acquire. Much of the essential information is found in the US Food and Drug Administration (FDA) structured product labels ⁶. The labels contain details about pharmacology, warnings, precautions, contraindications, and especially, the black box warnings ⁷. However, this information is locked in free text, making its use by automated systems impossible. It would be useful to have this data in a structured format, which would allow access, for example, by clinical decision support systems. Several studies have attempted to extract information from structured product labels using both statistical data mining techniques and natural language processing. However, the scope of these methods is limited in that focus is only on extraction of the actual adverse drug events.

In this paper, we test the feasibility of using SemRep (a semantic based natural language processing system) ⁸ as a general method to extract information from FDA black box warnings labels, to include adverse drug events, conditions at-risk, and susceptible populations. Our preliminary work on this topic can be found here ⁹.

Background

Black box warnings

Currently, the US Food and Drug Administration defines and detects adverse events for drugs that are currently on the market. The FDA maintains the Adverse Event Reporting System (AERS), which contains drug-use profiles, population databases, and active surveillance systems ¹⁰. In addition to this knowledge source, the FDA and the National Library of Medicine manage the DailyMed Web site, which currently contains 47,385 drug labels stored in an electronic format known as the structured product label (SPL) ⁷. This format is the approved standard of the Health Level 7 (HL7) consortium and has been adopted by the FDA for exchanging product and facility information ¹¹. The labels include a wide range of information on approved drugs: Description of the drug, Clinical pharmacology, Indications and Usage, Contraindications, Warnings, Precautions, Adverse Reactions, Overdosage, Dosage & Administration, Patient Counseling Information, Supplemental Patient Material, Patient Package Insert, Highlights, Full Table of Contents, Medication Guide, and Black Box Warnings (see Figure 1 for a black box warning example). The information contained in the labels is independently vetted by the FDA when the drug is approved and is considered to be the gold standard for drug information. The FDA can force changes to the labels after the drug is approved, if new safety concerns or information warrant such change. Not all drugs contain black box warnings and this section is one of the most stringent sections of the label and is used for drugs that have the potential for serious adverse drugs events.

Figure 1.

Sample black box warning label for Levaquin® (generic name: levofloxacin.)

Black box warnings include a variety of pertinent information for the prescribing physician. Therefore mining them may have significant clinical impact. Examples of pertinent information include severe adverse drug events, potential susceptible populations such as pregnant women, and conditions that would put a patient at particular risk of adverse events. Given the richness and importance of this information, we chose to analyze black box warning labels.

Automatic methods for identifying adverse drug events

Data mining and natural language processing have been used to extract information about adverse drug events from existing sources of biomedical information. Bates et al. reviewed the research on automatic extraction of adverse drug events. They claim that information technology, especially natural language processing, can be effectively used to identify adverse events in electronic medical records. They suggest that natural language processing will likely become prevalent as new sources of health information continue to become available ¹².

Harpaz et al. have explored data mining to discover new adverse drug events². They utilized data-mining analysis and evaluated potential new data sources for ADE discovery such as electronic medical records, administrative claims data, and the biomedical literature. Others have tried knowledge discovery from social media sites. There has been a growing appreciation that social media serves as an innovative method to harness data directly from the consumer. Social media offers great promise as a ‘non-traditional source’ of data and may serve as an effective tool for the identification of new adverse drug events. One such study by Nikfarin et al. sought to mine the social media Web site DailyStrength for patients self-reporting adverse drug events. They utilized NLP rules-based methods to identify adverse events that may be under reported via traditional methods ¹³.

Other researchers have focused their attention on SPLs. Rubrichi et al. utilized conditional random fields and compared it to support vector machines for extracting information from SPLs ¹⁴. They sought to automatically extract active ingredient and interaction effects. The work demonstrates the ability of automated methods to successfully pull information from drug labels. Though the paper had promising results, the approach focused only on the extraction of active ingredient and interaction effects. Fung et al. focused on extracting drug indications from DailyMed ¹⁵. They utilized natural language processing and compared their final results to the national drug file-reference (NDF-RT) and the Semantic MEDLINE Database ¹⁶. The study demonstrates the ability of NLP tools to extract information from SPLs.

Although finding previously unidentified adverse drug events from emerging sources of health data is important, there is still the problem of accessing currently available ADE data from curated sources such as DailyMed. Several research groups have evaluated methodologies to utilize this resource, with a fair amount of success. Bisgin et al. used a topic modeling approach to analyze black box warnings, adverse events, and the warnings and precautions sections of prescription drug labels ¹⁷. They used a statistical hierarchical method. The study produced insight into the connection between various drugs and common adverse drug events such as liver failure and hepatic injury. Another approach was developed by Freidlin and Duke. They created a natural language processing application (SPLICER) to extract adverse events from SPLs. From the output of their system, a standardized ADE knowledge base was created ¹⁸. The system was effective in extracting a total of 534,125 adverse drug events from a total of 5602 product label, with precision of 95% and recall of 93%. SPLICER uses a rule-based and algorithmic approach after adjustments with training data and works in a three step process. It first parses the text, after which the ADE extractor finds patterns likely to contain adverse events. Finally, terms are mapped to the Medical Dictionary for Regulatory Activities (MEDRA). The extracted data is then translated to SNOMED CT codes that can be stored as a value set for clinical decision support. This could lead to better clinical decisions by identifying problematic adverse drug events before the patient receives that drug. The focus of this research is only on adverse drug events; though useful for the clinical decision support, it would be ideal if a knowledge resource contained more general information about the ADE such as at-risk conditions and susceptible populations. Semantic approaches to natural language processing may provide the ability to create a more general approach to extract a broad array of information from structured product labels.

SemRep

SemRep is a semantic natural language system developed at the National Library of Medicine ¹⁷. The system first tokenizes input text and then performs a lexical lookup to the SPECIALIST Lexicon ¹⁹. A parser identifies noun phrases for each sentence, and these are mapped to Unified Medical Language System (UMLS) concepts by MetaMap ²⁰. SemRep extracts semantic predications which consist of three parts: a subject, an object, and a predicate, which indicates the relationship between the subject and object. For example, from (1) SemRep produces the predication in (2).

1. Lactic acidosis is a metabolic complication due to metformin.

2. Metformin CAUSES Acidosis, Lactic.

In addition to CAUSES, SemRep predicates used in this study were PREDISPOSES (identifies risk factors), ADMINISTERED_TO (e.g. “Drug ADMINISTERED_TO PATIENT”), and PROCESS_OF (e.g. “Disease PROCESS_OF Human”). We tested the ability of SemRep to extract information from FDA structured product labels. We focused on black box warnings due to the fact that they are the most serious warning that the FDA issues to drug manufacturers. It is believed that this approach (utilizing semantic predications) can unlock a richer, more generalizable, and deeper understanding of the corpus of drug data than by the many of the previously mentioned methods.

Methods

The DailyMed site housed at the National Library of Medicine provides a rich repository of structured product labels. For this study we only processed the black box warnings of the SPL. Figure 2 gives an overview of the study.

Figure 2.

The overall process used to analyze black box drug warning labels with SemRep.

2,000 drug structured product labels were randomly selected from the DailyMed website. Out of those, 900 had black box warnings, which were identified by using the LOINC code 34066-1 (BOXED WARNING SECTION) in the XML text.

We decided to extract three items of information from the free text of the black box warning: the adverse event itself, conditions which put patients at risk for events, and populations at risk for events. These are defined as:

• Adverse event: An adverse event to the drug. (e.g. “Lactic acidosis is a metabolic complication due to metformin.”)

• Condition: Disorders which put the patient at greater risk for adverse events. (e.g. “Patients with cardiovascular disease may be at greater risk…”

• Population: Groups at greater risk for an adverse event. (e.g. “Pregnant women should not receive methotrexate.”)

Rules applied to SemRep output

Three rules were developed to systematically extract the three items of information from SemRep predications:

1. Adverse event: Object of CAUSES OR Object of PREDISPOSES

2. Condition: Subject of PROCESS_OF

3. Population: (Object of ADMINISTERED_TO OR Object of PROCESS_OF) IF the object Concept is not general, such as “Patient” “Individual,” and “Personnel”

1. To identify adverse events, objects of “CAUSES,” or objects of “PREDISPOSES” were exploited. An example for the event category is “Lactic acidosis is a metabolic complication due to metformin.” The predication extracted from the sentence output was “Metformin CAUSES Acidosis, Lactic.” Lactic acidosis, as the object, was retained as an adverse event.

2. To identify conditions, subjects of PROCESS_OF were retained. For example, from the text “Patients with cardiovascular disease may be at greater risk…,” SemRep extracted the “predication Cardiovascular Disease PROCESS_OF Patients.” Since “Cardiovascular Disease” is the subject, it was retained as a condition for which the drug would put a patient at a greater risk of an adverse event.

3. To identify the population, the object of PROCESS_OF was retained, if it was not “Patient,” “Individual,” or “Personnel.” This was done because these concepts are not specific enough to identify a (useful) population that would be at greater risk for taking the drug. For example, the predication “Acidosis, Lactic PROCESS_OF Pregnant Women” was extracted from the text “Fatal lactic acidosis has been reported in pregnant women”, “Pregnant Women,” as the object of PROCESS_OF, was retained as an at-risk population.

Evaluation

To build a gold standard, we used 100 randomly selected black box warnings from the 900 extracted from the DailyMed website. The second author (MF) identified at-risk conditions, adverse events, and susceptible populations manually in each of these 100 warnings. We ran SemRep on the same 100 warnings and the rules described above were applied to the output to automatically identify at-risk conditions, susceptible populations, and adverse events. Output from the system (SemRep + rules) was then compared to the gold standard. Performance metrics were calculated for conditions, adverse events, and populations using precision, recall and F-Score.

Results

The results demonstrated that SemRep performed well as a novel tool to identify information from black box warnings (Table 1). Precision was 95% for susceptible populations, with recall 44%, and an F-Score of 0.61. For at-risk conditions, precision was 80%, recall was 53%, and the F-Score was 0.64. For adverse events, precision was 94% with recall of 52%, and F-Score of 0.67.

Table 1.

Performance metrics on the ability of semantic processing to extract information from FDA black box warning labels. N = Total number of instances.

	Condition	Population	Adverse event	Overall
Precision	80%	95%	94%	90%
Stu Recall	53%	44% Ten	52%	51%
F-Score	0.64	0.61 tem	0.67	0.65
N	84	83	317	484

Discussion

Results were promising in this study to determine the feasibility of using SemRep and post-processing rules to extract adverse event information (including susceptible populations and at-risk conditions) from black box warnings. Since recall was lower than precision, we performed an error analysis concentrating on the false negatives. Twenty-five random false negatives were analyzed. The most common error type was due to anaphora. Information needed for its resolution often does not appear in the sentence being processed. The following text provides an example.

“Only physicians experienced in immunosuppressive therapy for organ transplant patients should prescribe Cyclosporine…The drug increased susceptibility to infection and the possible development of lymphoma.”

In order to resolve the sortal anaphoric element the drug, SemRep would require access to its antecedent cyclosporine in the preceding sentence, which it currently does not have. A future version of SemRep will have the ability to resolve this type of anaphora.

A second common cause of error was that some semantic interpretation depends on inferencing, as in the following sentence:

“There are serious and life-threatening events associated with tamoxifen. Uterine malignancies consist of both endometrial adenocarcinoma and uterine sarcoma.”

In order to determine the adverse events associated with tamoxifen, it is necessary to infer that uterine malignancies and endometrial adenocarcinomas are life threatening. This is a challenging problem for natural language processing, which SemRep does not address.

Since precision was lower for at-risk-conditions, we also looked at false positives in this category. The method uses a simple rule to extract conditions-at-risk with the assumption that in the black box warnings the subjects of PROCESS_OF would be mostly these conditions. In spite of 80% precision, there were eleven false positives where the condition mentioned was not the condition-at-risk, but an indication of the drug. The following sentence provides an example.

“Fluvoxamine Maleate Tablets are not approved for use in pediatric patients except for patients with obsessive compulsive disorder”

In this case, the antidepressant fluvoxamine can be used in children with obsessive compulsive disorder. The rule erroneously extracted obsessive compulsive disorder as condition-at-risk.

The ability of a natural language processing system to extract adverse events, but also other types of information, from black box warnings, such as susceptible populations and at-risk conditions is essential for clinical applications such as clinical decision support systems. The information extracted from black box warnings and stored in structured format can be matched against the electronic health record and prevent serious adverse events. For example, if a clinician is considering prescribing an angiotensin-converting-enzyme inhibitor such as enalapril for a patient who is pregnant, information extracted from the black box warning (i.e. Drug: enalapril, Population: pregnant women, Adverse event: fetal death), could provide an alert concerning the serious potential consequences.

Given the number of patients affected by adverse drug events, there is a need to produce general structured data that can be used to provide clinicians, patients, and informatics applications with readily accessible information. It would be particularly valuable to have accurate, up-to-date information for clinical decision support systems available to the clinician at point of care. This could be accomplished by giving the clinical decisions support tool access to the data from drug warning labels that would, for example, help physicians identify patients most at risk for adverse events, especially those with relevant preexisting conditions.

In the future, we would like to expand the information that we extract from black box warnings beyond populations, conditions, and adverse events. Information such as what to do in the event of an adverse event, contra-indications, warnings, precautions, and drug dosage would all be useful expansions to the current research. It is likely that writing rules that apply to SemRep output, and that capture this additional information, will be straightforward. Additionally, we would like to process the entire corpus of prescription drug labels on the DailyMed site. Lastly, we need to modify SemRep to address the linguistic errors identified in the error analysis.

Limitations

This study has limitations. First, the sample size for analysis was rather small with 100 labels evaluated. It remains to be seen how the performance of the system will scale in the larger set of 900 labels that we currently have identified. Second, the gold standard was created by one physician who has experience with SemRep. It would be ideal if physicians not involved in natural language processing system were involved in gold standard creation. Finally, we did not compare our methods to other approaches (i.e. statistical and machine learning), since in this exploratory study our major objective was to determine the feasibility of a rule-based approach.

Conclusions

Having reliable automatic access to drug and prescribing information from FDA package insert labels provides notable benefit to health care. Such access would likely translate into better health care, fewer deaths, and fewer adverse events to prescription drugs. While previous approaches such as topic modeling showed promise to evaluate package insert labels, the additional information available in semantic search technologies will likely underpin significant improvements to mining drug labels. Overall, SemRep was useful as a novel tool to extract information from FDA black box warning labels. The lower recall of the system was largely due to deficiencies in anaphora resolution and the challenging problem of inferencing.