An automated tool for detecting medication overuse based on the electronic health records

Abstract

Purpose

Medication overuse is a serious concern in healthcare as it leads to increased expenditures, side effects and morbidities. Identifying overuse is only possible through excluding appropriate indications that are primarily mentioned in unstructured notes. We developed a framework for automatic identification of medication overuse and applied it to proton pump inhibitors (PPIs).

Methods

We first created an indications knowledgebase using data from drug labels, clinical guidelines, expert opinion and other sources. We also obtained the list of current problems for 200 randomly selected inpatients who received PPIs using a natural language processing system and the discharge summaries of those patients. These problems were checked against the indications knowledge base to identify overuse candidates. Two gastroenterologists manually reviewed the notes and identified cases of overuse. Results from the automated framework were compared to the manual review.

Results

Reviewers had high inter-rater reliability in finding indications (agreement = 92.1%, Cohen’s κ = 0.773). In 137 notes included in final analysis, our system identified indications with a sensitivity of 74% (95%CI = 59% – 86%) and specificity of95% (95%CI = 87% – 98%). In cases of appropriate use where the automated system also found one or more indications, it always included the correct indication.

Conclusions

We created an automated system that can identify established indications of medication use in electronic health records with high accuracy. It can provide clinical decision support for identifying potential overuse of PPIs, and could be useful for reducing overuse and also to encourage better documentation of indications.

Introduction

Overuse of medication is defined as the administration of medications in the absence of an appropriate indication, and it is a serious concern in healthcare: it leads to increased prescriptions and healthcare expenditures¹, and more importantly, it increases medication related side effects and complications without providing any benefit to the patients. Overuse is defined as delivery of clinically inappropriate care^2,3, and medication overuse is the administration of medication in the absence of an established indication. Previous studies have shown that proton pump inhibitors (PPIs), antidepressants, antipsychotics, statins, non-steroidal anti-inflammatory drugs (NSAIDs) and cyclooxygenase-2 (COX-2) inhibitors are subject to widespread overuse^4–6. For the purpose of this study, we focused on the overuse of proton pump inhibitors (PPIs).

PPIs are a group of gastric acid-suppressing drugs with high potency. Established indications for use include eradication of Helicobacter pylori in patients with peptic ulcers, prevention of gastric ulcers induced by NSAIDs, treatment of gastric ulcers, Zollinger-Ellison syndrome, acid-induced esophagitis, Barrett’s esophagitis and severe gastroesophageal reflux disease (GERD)⁷. Earlier studies showed that PPIs are more potent and effective than other acid-suppressing drugs ^8,9; consequently, PPIs have gradually replaced histamine receptor antagonists (H₂RAs) over the last two decades ^10,11. In 2009, PPIs had the third rank in the sales of medicines in the United States (US); in 2010 esomeprazole (Nexium®) was second in total sales among prescription drugs^12,13. In the US, some PPIs are now available without prescription, and their use is not reflected in the aforementioned statistics.

PPIs are frequently overused and compliance to guidelines has been reported to be as low as 31–33% ^4,14,15. Aside from the direct costs associated with overuse ^16,17, serious adverse effects have been reported from long-term acid suppression with PPIs. Several studies have demonstrated associations between PPIs and Clostridium difficile colitis ^18–20. Long-term use of PPIs may also lead to Vitamin B12 malabsorption ²¹, bone fractures^22–24, iron deficiency ²⁵, interstitial nephritis^26,27, and gastric carcinoids²⁸, although these associations are not supported by strong evidence ²⁹. PPIs may increase the risk of pneumonia ³⁰, and may decrease the effectiveness of clopidogrel – a frequently used anti-platelet agent – to cause fatal cardiovascular events ^31–33. Most of these side effects of PPIs arise after long-term use of these drugs; however, PPIs are very frequently used for long periods ^4,34. Recent guidelines on PPI use include warnings regarding these long-term effects⁷, and in February 2012 the Food and Drug Administration issued a warning specifically regarding the association between PPIs and C. difficile colitis³⁵.

Overuse of PPIs has been studied by manual review of patient records or via patient interviews^{4,11,14,15,36,37}. In all these studies, the primary method has been to rule out any established indication for PPI use, thereby identifying patients receiving the drug without an appropriate indication. This requires collecting a list of all existing conditions and diseases the patient has, and comparing it to the set of established indications. Although conditions and diseases affecting a patient can be recorded in the electronic health records (EHRs) in a structured form by means of electronic problem lists, studies have shown that these sources of information are neither comprehensive nor reliable ^38–41. Thus the information needed to make the decision about the appropriateness of use of PPIs must be extracted from narrative, unstructured notes in a process that is typically difficult to automate. This issue is not unique to studies of overuse, and has been a well-known limitation for using electronic health records in clinical decision support (CDS)^42,43.

Natural language processing (NLP) can be used to convert narrative information into a computable format and has been proven to be accurate and efficient in clinical studies ^42,44. Previous studies have shown that using an NLP engine can assist with automated generation of problem lists for patients with high accuracy ^45,46. This study develops a framework for automated identification of patients who are potentially subject to medication overuse and applies the framework to the overuse of PPIs. We also aim to identify the potential challenges in generalizing this framework to overuse associated with other groups of medications or other types of treatments.

Methods

Our framework consists of two arms (Figure 1): a knowledgebase of established indications for using a medication, and a list of current problems and conditions that a patient has. Here, we describe each of these components in more detail. Although we describe the process for the specific case of PPIs, the framework can similarly be used for other groups of medications.

Figure 1.

An overview of the framework used for identifying overuse of Drug X. See text for description. Abbreviations: AERS = Adverse Effect Reporting System, NDF-RT = National Drug File Reference Terminology, NLP = Natural Language Processing.

To develop the knowledgebase of indications (Figure 1, top-left), we began with collecting a list of medical conditions in which PPIs are used by reviewing drug labels and clinical guidelines. We also analyzed the reasons for PPIs use reported by healthcare professionals in the Adverse Effects Reporting System (AERS)⁴⁷. While the primary purpose for AERS is to report adverse effects, one mandatory field in submissions to AERS is the intended use of medication reported; we used this information to generate a list of common uses for PPIs. We also retrieved the indications listed for PPIs in the National Drug File Reference Terminology (NDF-RT)⁴⁸. Subsequently, a committee of three attending faculty gastroenterology specialists reviewed the list of possible indications with respect to the scientific evidence behind them, and selected a smaller list of all appropriate on-label and off-label indications for PPI use in adults, which we refer to as “established indications”. Each of the established indications was mapped to a single corresponding concept in the Unified Medical Language System (UMLS). The UMLS is a compendium of numerous controlled vocabularies in the biomedical sciences where each concept (e.g. a disease, a drug, a procedure, or a gene) is identified using a unique identifier⁴⁹. Mapping the indications to concepts in the UMLS ensures that these indications are coded in a consistent form that can also be used to represent patient problems in the next step. Table 1 provides the listing of established indications of PPI use after being mapped to UMLS concepts.

Table 1.

A listing of established indications for proton pump inhibitors developed by gastroenterology specialists and subsequently mapped to the UMLS concepts.

CUI	Condition
C0013295	Duodenal ulcer
C0013298	Duodenitis
C0013299	Duodenogastric reflux
C0030920	Peptic ulcer
C0030922	Peptic ulcer hemorrhage
C0151966	Duodenal ulcer hemorrhage
C0341245	Erosive duodenitis
C0854225	Duodenal ulcer, obstructive
C2741638	Stress ulcer
C0004763	Barrett's esophagus
C0014868	Esophagitis
C0014869	Reflux esophagitis
C0017168	Gastroesophageal reflux disease
C0151970	Oesophageal ulcer
C0155789	Oesophageal varices hemorrhage
C0267055	Erosive esophagitis
C0267075	Esophagitis ulcerative
C0013395	Dyspepsia
C0017181	Gastrointestinal hemorrhage
C0038358	Gastric ulcer
C0079487	Helicobacter infection
C0235325	Gastric hemorrhage
C0237938	Gastrointestinal ulcer
C0267158	Reflux gastritis
C0267166	Gastroduodenitis
C0341163	Gastric ulcer perforation
C0341164	Gastric ulcer hemorrhage
C0343378	Helicobacter gastritis
C2010560	Gastritis hemorrhagic
C2243088	Gastritis erosive
C0041909	Upper gastrointestinal hemorrhage

Abbreviations: UMLS = Unified Medical Language System, CUI = Concept Unique Identifier.

The other component of our framework is to collect current problems for patients who are receiving the medication in question (in this case, PPIs) using the EHR (Figure 1, top-right). We identified such patients by searching records of medication orders in the EHR, and retrieved the medical records collected for those patients during their hospitalization period. Depending on the availability and quality of the EHR data, patient problems can be obtained from structured problem lists or from narrative reports. In our data set, comprehensive and reliable problem lists were not available; therefore, we used narrative reports as the only source for obtaining patient problems. For the purpose of this study, we only used discharge summaries, and we used the MedLEE (Medical Language Extraction and Encoding) NLP system to extract the concepts in the notes. MedLEE is a rule-based NLP program that is especially designed to parse clinical texts, i.e. it takes natural language clinical text as an input, and represents the data in a standardized computable form as the output to facilitate accurate retrieval of information⁵⁰. It is capable of identifying negation and temporality identifiers, and of mapping the medical concepts found in the notes to the UMLS concepts. We generated a list of patient’s current conditions coded into UMLS concepts, taking care to exclude negated conditions, family history, and conditions strictly happening in the past (i.e. a past medical history of an acute disease was not considered as a current problem, but if a chronic problem such as diabetes mellitus was mentioned in the past medical history, we included it in the patient’s current problems).

In a pilot study, the accuracy of our system in identifying indications in a training set of notes was measured by comparing it to manual review by the corresponding author. We estimated our method has an agreement of 75% in predicting overuse (95% CI = 63 – 86%), and used this information in sample size calculation for the current study. Assuming a null accuracy of 50% (pure chance) and using sample size calculation for proportions, we determined that a total sample size of 115 was needed to be able to identify accuracy of 0.75 using a type-I error threshold of 0.05 and a statistical power threshold of 0.80. Since some of the exclusion criteria (see below) could only be assessed after the sampling was done, to allow for possible exclusions and subgroup analyses without critical decline in statistical power, we used a sample size of 200. The sample used in this study did not contain any of the records or patients included in the pilot phase.

We used a computer program to randomly select 200 patients who were admitted to the New York Presbyterian Hospital in 2010 and had one or more orders to receive PPIs during admission. For patients with multiple hospitalizations in 2010, we used a computer program to randomly choose one of the hospitalizations in which PPIs were prescribed, and discarded the others. We used this data to evaluate our framework. Patients whose current conditions did not include an established indication for PPI use were classified as “overuse candidates”. We call them candidates because establishing overuse is not completely possible by reviewing discharge summaries alone (even by manual review) since a valid indication might not be documented properly in the discharge summary. We used a single inclusion criterion: to have an order in the EHR to receive any PPI during their hospitalization. Our exclusion criteria included not having a discharge summary recorded in the EHR (that is because discharge summaries are usually not recorded for hospitalizations shorter than two days at our center), history of intubation during the hospitalization, and age less than 18 years.

A gastroenterologist manually reviewed the notes and identified whether an established indication was present. A gastroenterology fellow independently reviewed 40 of these notes so that we could assess inter-rater reliability. We facilitated the review process by providing the experts with an interface that helped them browse the notes in a secured and encrypted environment and record the indications they found in the notes using a free-text input box. Our reviewers were not involved in the development of the framework. We measured inter-rater reliability using concordance and Cohen’s Kappa⁵¹. We finally compared the output of the framework with the results of manual review by the experts to calculate the sensitivity and specificity of our method. The Institutional Review Board of Columbia University approved the protocol of this study.

Results

From the original 200 notes selected for manual review by experts, 23 were excluded because they were associated with children. We also excluded 40 notes because they were associated with patients with a history of intubation. There is evidence suggesting that PPIs may be beneficial for preventing stress ulcers in patients receiving prolonged intubation ⁵², but guidelines are vague and the decision is generally made on a case-to-case basis. All remaining 137 notes were manually reviewed by experts and categorized by them as either “appropriate use” or “overuse candidate”. The inter-rater reliability of two reviewers was high (agreement = 92.1%, Cohen’s κ = 0.773). Table 2 summarizes the baseline characteristics of patients included in this study. It should be noted that distribution is impacted by the choice of PPI recommended by the hospital formulary.

Table 2.

Baseline characteristics of patients included in the study. Continuous values are summarized as mean (standard deviation), and counts are documented as frequency (percent).

Demographics
Age (years)	65.5 (19.8)
Gender
Female	79 (58%)
Male	58 (42%)
Ethnicity
African-American	17 (12%)
Caucasian	39 (28%)
Hispanic	32 (23%)
Other*	49 (36%)
Medication use
Esomeprazole	116 (85%)
Pantoprazole	6 (4%)
Omeprazole	2 (2%)
Lansoprazole	1 (1%)
Multiple PPIs	12 (9%)
Total	137 (100%)

^*Patients with other ethnicities (e.g. Asian, or Indian) and those for whom ethnicity information is unknown are marked as “Other” in our EHR.

Abbreviations: PPI = Proton Pump Inhibitor.

Out of all 137 notes reviewed, only 43 contained an indication based on manual review. We measured the accuracy of the system using the well-known performance metrics sensitivity and specificity. Sensitivity is defined as TP/(TP+FN) where TP is the number of times the system correctly found that there is an appropriate indication, and FN is the number of times the system did not find an appropriate indication but manual review did. Likewise, specificity is defined as TN/(TN+FP) where TN is the number of times the system correctly identified the absence of appropriate indications, and FP is the number of times the system found an appropriate indication but manual review did not. Our framework identified indications in 37of the notes and in comparison to the manual review it had a moderate sensitivity (74%, 95%CI = 59% – 86%) and a high specificity of (95%, 95%CI = 87% – 98%). In cases that were not classified as overuse candidates, the number of indications returned by our framework ranged from 1 to 4 (mean = 1.43, standard deviation = 0.74, median = 1, mode = 1). In 28 cases, only one indication was returned. In all true positive cases, the indication documented by manual review was correctly returned by the framework. There was no difference in the number of indications returned for true positives versus false positives (Mann-Whitney P-value = 0.3491). The main reasons for receiving PPIs included GERD (17 cases), gastrointestinal hemorrhage (13 cases) and peptic ulcer (4 cases).

The mean ± standard deviation of the time spent by manual reviewers reviewing each discharge summary was 83 ± 52 seconds. Processing the same discharge summaries using the NLP system took 1.88 ± 1.38 seconds, and the matching query took 27 milliseconds on average (all measurements were done on a commodity server with 43 GB of memory and one 4-core 2.93 GHz processor).

Discussion

We were able to create a framework using NLP that can be used for automated identification of established indications of medication use in narrative reports with high accuracy, and an automated framework to identify overuse candidates. The success of our approach is a product of the comprehensiveness of the list of indications, accuracy of extracting patients’ problems, completeness of documentation, and ability to translate the established indications determined by clinicians into unique concepts in the knowledgebase.

Identifying the reasons for the prescription of a medication using NLP was previously studied by multiple groups of researchers as part of the Institute for Integrating Biology and the Bedside (I2B2) “medication extraction challenge”. Studies conducted in response to that challenge all showed a low performance in extracting the reasons for prescription of medications, with their F-measure ranging from 0.03 to 0.525^53,54. In contrast, our system was able to identify the indications for PPI administration in adults with high sensitivity and specificity (F-measure = 0.80). While our results are not directly comparable with those studies because they are obtained using different data sets, we believe the higher accuracy of our approach has in part resulted from intentionally restricting our search for certain concepts (i.e. the list of established indications). Another study in which the search for medication-problem associations was limited to those previously described in a knowledgebase also yielded comparable results (sensitivity of 67.5% and specificity of 86%) ⁴⁸.

Given the subjective nature of the task as illustrated by the inter-rater agreement of 92% on the reference standard, any system would be unlikely to be able to perform with 100% accuracy. Therefore, we further analyzed the output of our framework to investigate the false positives. We found that at least in 2 of the 5false positive cases, a second reviewer could interpret the indication found by the system as appropriate. Other reasons for false positives included differential diagnoses that were suggested in one section of the note (e.g. “patient complained of epigastric pain during admission, likely secondary to GERD”) and ruled out later in the note, indicating that more complex reasoning concerning the section of note where the information occurred would be required to determine appropriateness of medication use. Another cause of error was the NLP errors in interpreting temporality (e.g. history of gastritis from several years ago was incorrectly identified as an ongoing chronic event).

Similarly, false negatives were predominantly cases in which a manual reviewer documented an indication that was not in the original list of appropriate indications (e.g. hiatus hernia, coagulopathy, post-surgical ileus, and gastric outlet obstruction), an inevitable consequence of lack of expert agreement on an inclusive list of indications. Most of the remaining false negatives were the result of NLP limitations; for instance, the term “GI ulcer” was not recognized as “peptic ulcer disease” because it was not obvious to the NLP system that the location of bleed was in the upper gastrointestinal tract (as determined by the expert reviewers). Additionally, in some cases the decision about the appropriateness of PPI use was too complex to be implemented in our framework. For instance, in one patient who had a low risk for gastric ulcer, the expert reviewer determined that prescribing a PPI prophylactically would be appropriate because that patient was a Jehovah’s Witness who would be likely to refuse transfusions if significant gastric bleeding occurred.

Our work is not without limitations. First, we processed only discharge summaries for possible indications. Further indications might be revealed in other documents such as admission notes or progress notes. Some researchers have also expanded the information used to identify indications by complementing the data from medical records with data collected directly from the patients, for example through interviews⁴. While this may serve to eliminate under-documentation of patient’s conditions, it introduces recall bias and precludes automation.

Our method is unable to distinguish overuse from under-documentation. While the majority of indications for PPI use are severe and significant events (such as gastrointestinal bleeds, or peptic ulcers), one can still expect that some less serious indications (e.g. GERD) may not be documented in the medical records if they are not the primary reason for admission, or if the patient started receiving PPIs long before they were admitted to the hospital. However, this issue also affects manual review of the notes by experts, and should not affect the results of our evaluation. Last, our method currently ignores indirect indications; for instance, if the PPIs were prescribed solely to prevent or diminish the gastro-irritant effects of a co-prescribed medication (e.g. NSAIDs, steroids, aspirin, or alendronate) our framework would mark this as overuse.

The performance of the system could be improved by promoting better documentation, extending the list of indications to include less frequent indications, and introducing more complex logic into the process of identifying patient problems. We also anticipate that the performance would be higher if structured problem lists were in use and were kept up to date. In future work, we will try to address these limitations and demonstrate the generalizability of this framework by applying it to other cases of overuse.

In summary, our automated framework compares favorably with expert manual review. In the future, this framework could provide clinical decision support for identifying overuse of medications both to reduce overuse and to encourage better documentation of indications.

Key Points.

• Medication overuse can be automatically identified with high accuracy by automated processing of electronic health records

• Automatically identifying overuse is hindered by the same factors affecting manual identification: vagueness of indications, and inaccurate documentation of medical findings