Automated Determination of Publications Related to Adverse Drug Reactions in PubMed

Abstract

Timely dissemination of up-to-date information concerning adverse drug reactions (ADRs) at the point of care can significantly improve medication safety and prevent ADRs. Automated methods for finding relevant articles in MEDLINE which discuss ADRs for specific medications can facilitate decision making at the point of care. Previous work has focused on other types of clinical queries and on retrieval for specific ADRs or drug-ADR pairs, but little work has been published on finding ADR articles for a specific medication. We have developed a method to generate a PubMED query based on MESH, supplementary concepts, and textual terms for a particular medication. Evaluation was performed on a limited sample, resulting in a sensitivity of 90% and precision of 93%. Results demonstrated that this method is highly effective. Future work will integrate this method within an interface aimed at facilitating access to ADR information for specified drugs at the point of care.

Introduction and Background

Adverse drug reactions (ADRs) were shown to be highly prevalent and to be responsible for increased hospitalization rates, morbidity and mortality, particularly in older adults1. Timely dissemination of up-to-date information about ADRs at the point of care can significantly improve medication safety and prevent ADRs2, 3. MEDLINE is the largest freely available catalogue of peer-reviewed evidence-based medical resources which provides public access to indexed abstracts4. It includes references to the most recent updates on ADRs associated with both established and recently introduced medications. Easy and efficient access to this information for clinical decision making by providers as well as by automated decision support systems can improve patient safety and quality of care5.

PubMed is an online service that gives users the ability to retrieve the information collected in MEDLINE by entering simple keywords or using advanced search options and query filters4. However, recent studies demonstrated significant barriers in the ability of physicians to find information they need using PubMed6–8. The use of PubMed to answer clinical care questions on a daily basis was limited because it was difficult to retrieve a highly relevant set of articles within the time constraints of a routine medical visit or clinical encounter8. The average physician-generated search retrieved between 31–46% of relevant articles7, 8. On average, physician-generated queries contained less than 3 search terms8. Several studies analyzing PubMed use by physicians confirmed serious barriers physicians faced in using PubMed in routine clinical practice including lack of time to develop efficient search strategies, retrieval of large numbers of irrelevant articles, and limited familiarity with efficient PubMed search techniques9–11. In order to facilitate dissemination of evidence-based research results into routine clinical practice, effective methods for addressing barriers in using PubMed by physicians are Previous studies demonstrated that use of specialized PubMed search filters improved the efficiency of physician searches9, 10. Such filters, represented by verified PubMed queries that address highly specific clinical questions showed good results9, 10 but did not focus on ADRs. Other studies used MESH and textual terms, and others used machine learning to detect articles concerning specific adverse events, pairs of drug and adverse events, or specific drugs12–16. However, search filters aimed at effective retrieval of articles which include information on all ADRs for a particular medication received limited attention. Such a filter can potentially address the above mentioned physician challenges in using PubMed at the point of care and promoting medication safety by disseminating research results in routine care settings. Thus, the goal of this project is development and initial evaluation of a specialized type of PubMed query aimed at identifying all MEDLINE references to articles containing information about ADRs related to a particular drug.

Methods

Developing the Search Query

A total of 1,644 unique PubMed identifiers (PMID’s) were obtained from an online corpus of adverse drug event abstracts (https://sites.google.com/site/adecorpus/home) and the corresponding abstracts were collected15. A statistical package called WordStat (http://provalisresearch.com/products/content-analysis-software/) was used to analyze properties of these abstracts, including high frequency text words and phrases, MeSH terms tagged as major and minor article topics, MeSH subheadings and supplementary concepts, which also contain names of drug and drug classes. High frequency text words and phrases were analyzed. The textual terms were found to be either too general to be useful for retrieving only adverse event articles, or were very specific for particular adverse events so that they were not generalizable. Minor MeSH terms were ignored since the topics they describe are not the main foci of the articles. Of the properties analyzed, high frequency major MeSH terms with subheadings were the most useful, since they describe the main focus of the article and cover a broad range of adverse events. Statistical analysis showed that a majority of the highest frequency major MeSH terms were Pharmacological Action (PA) terms with the “adverse event” subheading. This led to an initial base search method (“drugname” [TIAB] AND (“pharmacological action/adverse event” [MAJR]) which finds articles that have both the specified drugname in either the title or abstract and one of the drug’s corresponding PAs with the subheading “adverse event” as a major MeSH term. In order to obtain the appropriate PA(s) of a drug, the MeSH resource was used (http://www.nlm.nih.gov/bsd/disted/meshtutorial/pharmacologicalactionterms/) which assigns most substances to one or more PA terms. The additional subheadings “toxicity” and “poisoning” were added, after a review of MeSH subheadings, as they each denote an adverse event. An example of the initial search with two PA variables and all three subheadings is shown in Figure 1.

Figure 1.

Base ADR Search Strategy

The initial base search was tested on numerous drugs during development. The results were reviewed manually by the developer (HA) to identify inappropriate abstracts, the majority of which included animal studies, review articles evaluating drugs other than the desired drug, and articles where the adverse drug reaction was caused by a drug other than the desired drug. The search was iteratively refined accordingly by adding additional filters aimed at increasing the accuracy of the search. Three additional filters were designed to eliminate these unwanted results without removing the desired results, as described below:

1. Animal studies were removed using the filter AND (“humans”[MESH]), which limits the search to abstracts tagged with the MeSH term “humans”. Animal studies which were tagged with the MeSH term “humans” were still being retrieved, and therefore the additional filter NOT ((“Rodentia” [MESH]) OR (“Artiodactyla” [MESH] OR (“Carnivora” [MESH]) OR (“Lagomorpha” [MESH])) was added, which eliminated all articles tagged with the MeSH term for animals within these subgroups.

2. Review articles where the primary drug being reviewed was not the specified drug frequently caused errors in retrieval because the adverse effects of the drug being reviewed are typically compared with other drugs that do not have the adverse effects. In addition, the review articles contained reference lists which overlapped with original research articles identified by the very query we have been testing. These articles were filtered out using the search string NOT (review[pt] NOT (“drugname”[TI])) to remove review articles from the results except when the specified drug is being reviewed. For example, this filter eliminated retrieval of the PubMED article http://www.ncbi.nlm.nih.gov/pubmed/15172038 because the title was “Safety overview of new disease-modifying antirheumatic drugs”, and leflunomide, the drug which was specified, was mentioned in the article along with a list of other drugs. This strategy allowed removal of review articles devoted to a general overview of broad groups of medications with diverse mechanisms of action but are not specifically focused on the drug of interest and provide little information about its ADRs.

3. Many false positives were also removed with the filter NOT ((“induced” [TI] NOT (“drugname” [TI])) because it eliminates abstracts with the word “induced” in the title when the specified drug name is not in the title, as the foci of these articles concern adverse effects of different drugs from the specified ones. For example, this filter eliminated retrieval of the PubMED article http://www.ncbi.nlm.nih.gov/pubmed/23462722, because it has the title “Sarcopenia and body mass index predict sunitinib-induced early dose-limiting toxicities in renal cancer patients.”

In this example, the drug Sorafenib is mentioned in the abstract in reference to the medication regimen of study subjects and its ADRs are not discussed in the article.

An example of a full search, with two pharmacological actions and the above filters, is shown in Figure 2.

Figure 2.

Final ADR Search Strategy

Evaluation

In order to evaluate the method, precision and recall were measured. To ensure broad representation of all medication groups for the precision testing, one representative generic drug was chosen for each of the nine most frequently prescribed therapeutic categories17. The particular representative drugs were chosen from the list of medications routinely prescribed for older adults in whom ADRs are most frequently described (Table 1). In addition, the drugs chosen for the precision testing were not in the development set. The search query shown in Figure 2 was modified to make the results review manageable. One filter required that an abstract existed in English because the abstracts had to be reviewed manually to establish accuracy. To reduce the effort further, abstracts were also limited to ones published in the last ten years by major clinical journals (e.g. the filter JSUBSETAIM[TW] was added). These additional filters did not change the logic of the query and were used only to limit the number of articles retrieved. A physician informaticist (JF) and biomedical informaticist familiar with medical language and terminology (CF) performed the manual review but did not develop the method. A query was generated for each drug name by automatically mapping the drug name to the corresponding pharmacological action(s) so that the query described above could be generated. The queries that were automatically generated were then executed, resulting in a list of abstracts for the specified drugs. Each expert independently read each of the retrieved abstracts to determine whether or not they discussed an adverse drug event for the specified drug. An abstract was considered correct if the study reported one or more ADRs and if the outcome was found to be positive or negative. Inter-rater agreement was computed. Precision was calculated to be the ratio of correct abstracts over all retrieved abstracts.

Table 1.

List of medications used to evaluate precision

Category	Drug Name	Generic
Lipid-lowering agent	Lipitor	Atorvastatin
Anti-depressant	Lexapro	Escitalopram
Beta Blocker	Lopressor	Metoprolol
ACE Inhibitor	Accupril	Quinapril
Antidiabetic Agent	DiaBeta	Glyburide
Respiratory Agent	Atrovent	Ipratropium
Anti-Ulcerants	Nexium	Esomeprasole
Diuretic	Lasix	Furosemide
Anti-Epileptics	Neurontin	Gabapentin

In order to evaluate recall of the method, we established a reference set that was manageable. A physician (JF) read all articles that were published in the Annals of Pharmacotherapy in the most recent 3 months which were fully indexed at the time of analysis (June-August 2013). This journal was chosen because it is a high quality peer-reviewed journal that covers a broad range of drugs, adverse events, and types of publications, such as case reports, research studies, and review articles. The physician recorded each article that discussed an adverse drug event during that time period along with the corresponding drug(s), forming the reference standard. The query shown in Figure 2 was modified so that the queries were restricted to that single journal, to June-August of 2013, and to the drugs in the abstracts forming the reference standard. Other than limiting the publications to be reviewed to a manageable size, the basic logic of the queries remained the same. Recall was calculated as the ratio of correct articles retrieved by the automated query for the drugs in the recall reference standard.

Results

The search query that was used to evaluate precision resulted in retrieval of a total of 58 articles for the nine drugs (Table 1). Of those 54 mentioned one or more ADRs, resulting in a precision of 93%. Inter-rater agreement was determined to be 100%. The recall reference standard consisted of 72 articles, out of which ten articles mentioned ADRs for specific drugs. Of the 10 ADR articles, 9 were automatically retrieved by our method for the specific drugs identified in the manual review, resulting in a sensitivity of 90%.

Discussion

In this study an effective search strategy was introduced to identify articles containing information related to ADRs associated with a specific medication. The precision of the resulting PubMed queries was 93% as assessed based on nine widely used medications from corresponding nine major medication classes. Recall was tested on a complete 3-month list of all publications in a representative journal and found to be 90%. Our results conform to previous reports on successful development of specialized search filters that used MeSH terms as part of their search strategy9, 10. As MeSH terms are assigned by trained abstractors according to established guidelines18, use of this information as part of the search strategy resulted in development of a robust ADR search filter for PubMed. The use of subheadings, particularly the one establishing an adverse event relation and the assignment of major topics, were particularly useful in establishing precision.

Prior work concerning development of automated methods that retrieve abstracts related to ADR literature has been focused on search strategies aimed at retrieval of articles discussing drugs associated with a specific ADR or ADR-drug pairs12–14, 16, 19, 20. Some methods rely on PubMed and MeSH terms with or without keywords, whereas others use machine learning methods with MeSH terms, textual terms, or a combination of both as features. Evaluation studies to measure performance of these methods were generally based on use of reference standards consisting of a set of specific ADR-drug pairs.

There has been little work regarding automated retrieval for the task we describe in this paper, which involves obtaining MEDLINE abstracts for specific drugs or groups of drugs where any ADRs are discussed. A paper that reported on a task similar to ours studied five search approaches for use in the MEDLINE or EMBASE where four of the approaches involved retrieval of ADR articles for specified drugs only12. The search queries were developed manually by using a combination of text and indexing terms and were tested on seven new anti-epileptic drugs. The most sensitive search had a sensitivity of 98.6% but a precision of 2.8%, and the highest precision reported for any of the searches was 2.9%. Our method differs from related work in that it is based on use of the most relevant MeSH headers and subheaders which were derived from statistical analysis of the MeSH terms and supplementary concepts, and the precision is much higher.

Previous studies clearly demonstrated that clinicians are willing and able to use MEDLINE information in the process of routine care delivery as long as existing barriers to PubMed use for MEDLINE searches are addressed 8, 11. A recent study demonstrated thet physicians’ acceptance of the MEDLINE system for practicing evidence-based medicine was predicated by perceived usefulness and perceived ease of use11. Our query is a first step in development of an application that can be potentially useful within a physician’s workflow. An application can be embedded in an electronic medical record system, or made available via mobile tools for physicians to further facilitate perceived ease of use and usefulness, or the filter can be added to PubMed as a clinical search query. The approach we used may also be extended to other specific types of medication safety questions, such as drug-food or drug-herb interactions as well as to other indexed information repositories such as EMBASE. Further work may include specialized medication safety tools which would dynamically generate drug specific queries delivering adverse reaction summaries along with the most relevant and recent information about the drug(s) of interest to assist physician with evidence-based decision making in the process of care delivery.

Our work has some limitations. Since our approach relies on MeSH terms, the most recent articles may be missed if they have not been assigned MeSH terms at the time of the query. The quality of indexing by abstractors may also be a limiting factor. However, the National Library of Medicine employs experienced and well trained abstractors who follow strict abstraction guidelines4. This is confirmed by the fact that during the development phase we observed high consistency in MEDLINE indexing with regards to ADR information. Another limitation is the small number of articles used to measure precision and sensitivity. Generating a reference standard for precision requires the use of an expert to read the articles that were retrieved by the queries to determine the ground truth. However, generating a reference standard for sensitivity is a more time-consuming task since all articles in a specific time period or journal(s) have to be read to establish a relevant ADR relation. In the future, we plan on extending our reference set to include more drugs and a larger time period in order to obtain a larger test set. The method we used to establish a test set for sensitivity, although time-consuming, aims to minimize bias since it does not focus on any set of adverse events or set of drugs. Broadening the sensitivity test set will enable us to experiment further with some of the filters. For example, filter 2 described above seemed to remove some true positive and generate false positive results and should be studied further.

Conclusion

The method we described in this paper generates queries to obtain relevant articles focused on ADRs for specific drugs, and it demonstrated high sensitivity and precision in this initial study. It appears to be very promising and should be studied further. We believe it will enable development of applications that focus on efficient access to adverse event information for clinical decision making by providers, researchers, as well as by automated decision support systems, and that it can lead to improved medication safety for patients.