Abstract
Background
Idiosyncratic drug induced liver injury (DILI) is an uncommon but important cause of liver disease that is challenging to diagnose and identify in the electronic medical record (EMR).
Aim
To develop an accurate, reliable, and efficient method of identifying patients with bonafide DILI in an EMR system.
Methods
527,000 outpatient and ER encounters in an EPIC-based EMR were searched for potential DILI cases attributed to 8 drugs. A searching algorithm that extracted 200 characters of text around 14 liver injury terms in the EMR were extracted and collated. Physician investigators reviewed the data outputs and used standardized causality assessment methods to adjudicate the potential DILI cases.
Results
A total of 101 DILI cases were identified from the 2,564 potential DILI cases that included 62 probable DILI cases, 25 possible DILI cases, 9 historical DILI cases, and 5 allergy only cases. Elimination of the term “liver disease” from the search strategy improved the search recall from 4% to 19% while inclusion of the 4 highest yield liver injury terms further improved the positive predictive value to 64% but reduced the overall case detection rate by 47%. RUCAM scores of the 57 probable DILI cases were generally high and concordant with expert opinion causality assessment scores.
Conclusions
A novel text searching tool was developed that identified a large number of DILI cases from a widely used EMR system. A computerized extraction of dictated text followed by the manual review of text snippets can rapidly identify bonafide cases of idiosyncratic DILI.
Keywords: Hepatotoxicity, drug-induced liver injury, data mining, Electronic Medical Record
Introduction
Idiosyncratic drug-induced liver injury (DILI) is a serious adverse drug reaction (ADR) that can lead to a wide range of clinical manifestations varying from asymptomatic laboratory test abnormalities to acute liver failure (ALF) resulting in transplant or death1,2,3. Idiosyncratic DILI is largely independent of the dose and duration of suspect drug exposure which makes it exceedingly difficult to predict its development and outcome4. In addition, there are no reliable clinical, genetic, or environmental risk factors implicated in DILI pathogenesis. Idiosyncratic DILI is infrequent in the general population with an estimated annual incidence of only 15 to 20 cases per 100,000 patient years attributed to a multitude of agents5. However, establishing a diagnosis of DILI is challenging due to the frequent use of multiple medications and the need to exclude other more common causes of acute and chronic liver injury such as pancreaticobiliary disease, alcohol, and viral hepatitis. Furthermore, there are no objective laboratory tests to confirm a diagnosis of DILI. Therefore, causality assessment is largely based upon the temporal association between drug exposure and liver injury onset, exclusion of competing causes, and clinical improvement after drug discontinuation6. In this setting, it is not surprising that making a diagnosis of DILI is frequently delayed or even missed altogether7.
Prior pharmacoepidemiology studies have attempted to use various searching strategies to identify DILI patients in large electronic medical record (EMR) databases8,9. The International Classification of Diseases codes (ICD-9 codes) are widely used to code the common principal diagnoses of hospitalized patients and billing of physician services. Although there are over 40,000 codes included in the ICD-9 index, there are no specific codes for idiosyncratic DILI. We previously demonstrated that using common ICD-9 codes for general liver injury such as “jaundice” (782.4) and “toxic hepatitis” (573.3) that were cross-linked with a text search of the EMR for specific drugs has a poor sensitivity and specificity for identifying patients with idiosyncratic DILI10. Other studies using more sophisticated search algorithms of dictated text in the EMR for liver injury terms in combination with specific drug names, diagnosis codes, and laboratory values improved the accuracy of identifying bonafide DILI cases11. However, these approaches required extensive computer software development that may not be generalizable.
Studies using text searching methods have also been shown to be an efficient and practical means to identify other ADR's 12. For example, Honigman et al found that free text searching was superior to diagnosis codes, allergy codes, and a computerized event monitoring system for detecting various ADR's with an overall sensitivity of 91% and a positive predictive value (PPV) of 7.2%13. Similarly, Field et al found that a free text search was useful in identifying ADR's but with a lower sensitivity of 39%14. However, to date there has been no study that uses free text searching to identify DILI cases. The aim of the current study is to develop a sensitive and specific free text searching algorithm of the EMR to identify DILI cases attributed to eight commonly used drugs with characteristic liver injury patterns and phenotypes.
Methods
The DILIN Retrospective study
The National Institutes of Digestive Diseases, Diabetes, and Kidney Diseases (NIDDK) established the Drug Induced Liver Injury Network (DILIN) in 2003 to improve our understanding of the etiologies, risk factors, and outcomes of patients with DILI in the United States15,16. The network currently consists of 6 clinical sites and a data coordinating center. The Idiosyncratic Liver Injury associated with Drugs (ILIAD) retrospective study was initiated in 2004 to enroll patients with liver injury attributed to a limited number of drugs with well-recognized phenotypes16. The ILIAD study initially targeted patients with clinically apparent liver injury due to amoxicillin-clavulanate, phenytoin, valproate, and isoniazid that developed after 1994. The list of targeted drugs was expanded in 2006 to include quinolone antibiotics, trimethoprim- sulfamethoxasole, minocycline/ tetracycline, and nitrofurantoin. All subjects who participate in the ILIAD study are required to sign a written informed consent document approved by the local Institutional Review Board. A waiver to search the EMR at the University of Michigan for subjects that may be eligible for the ILIAD protocol was obtained for the current study.
Electronic Medical Record Database
The University of Michigan Health System is a tertiary care referral center, including a longstanding liver transplant program, with over 900 licensed inpatient beds and over 1 million outpatient visits per year. As of August 2012, all outpatient and emergency room (ER) encounters were captured in the EpicCare Ambulatory and ASAP ED products (EPIC, Madison, WI). The text content of all Epic notes are stored in the Epic chronicles database and are accessible for analysis via Epic Clarity (EPIC, Madison, WI) which is populated nightly from the source document system.
Searching the Epic EMR database for the current study involved a multistep process. A subset of the Clarity tables pertinent to provider notes and associated encounters was extracted into an Oracle (Oracle Corporation, Redwood City, CA) database. Custom Java code then took the notes in the database tables and pushed them to an Apache Solr (Apache Software Foundation, V4.7.2) instance via Solrj, a Java API for creating and reading Solr indexes. Text searches were then prototyped using the Solr admin user interface. Finally, Pentaho Data Integrator (PDI, Pentaho, Orlando, FL) was used to perform searches against the Solr instance and record the output of the search results into additional tables so that Excel (Microsoft, Redmond, WA) spreadsheets could be created for further review. The Solr schema used for the index included a number of fields in addition to the note text to facilitate searching. Each document that was indexed included related fields such as the type of encounter, the type of provider, encounter date, patient identification number, and the type of note. Together this additional metadata was used to exclude notes written by non-credentialed providers, patient instructions, and phone notes.
Each individual file in the spreadsheet contained up to four 200 character “snippets” (SNIPS) of text taken directly from the medical chart that contained the search terms (listed below). A 200 character limit (or ~ 30 words) was chosen to provide the physician reviewers enough clinical context from the note to make an assessment of the possibility that this was a true DILI case. In addition, the first 32,000 characters of the entire document were also exported to the excel sheet.
Medical Record Search Techniques
Initial EMR Search
The text of all outpatient provider and ER Epic encounters created between August 1, 2012 and December 31, 2013 were searched for one of the 14 pre-specified “liver injury” terms combined with a single “drug term”. The 14 “liver injury” terms were selected as medical terms that would likely indicate a possible diagnosis of DILI: drug-induced liver injury, drug-induced liver toxicity, drug-induced liver damage, drug-induced liver disease, drug-induced hepatotoxicity, drug-induced hepatitis, liver damage, liver disease, drug hepatotoxicity, hepatotoxicity, liver injury, liver toxicity, adverse liver reaction and DILI. The 8 drugs that were targeted were the same as those used for enrollment into the ILIAD study that included antibiotics (augmentin, isoniazid, nitrofurantoin, trimethoprim/sulfamethoxazole, minocycline, fluoroquinolones) and antiepileptic drugs (valproic acid, phenytoin). A list of the generic and brand name variations of each drug used in the search are provided in Supplemental Materials and Methods.
The types of documents searched in Epic included progress notes, consult notes, letters, history and physical notes, and emergency department provider notes. Documents were only included if they were authored by an attending physician, resident physician, physician assistant, or nurse practitioner.
Refinement of EMR search algorithm
In an attempt to improve the efficiency of the search strategy, a stepwise removal of the initial 14 liver injury terms was undertaken. Initially, we eliminated the liver injury terms that yielded no DILI cases; we also repeated the search after removing the liver injury terms that yielded a small number of potential DILI cases. The sensitivity, specificity, and positive and negative predictive value of each of the searches were then determined using the initial search as the “gold standard” comparator. The term search sensitivity indicates search recall while search precision reflects positive predictive value.
Medical Record Review
The output data contained in the excel spreadsheets were initially reviewed by a physician investigator (LH). If there was no evidence of liver injury, or if an alternative cause of liver injury was apparent such as viral hepatitis, hepatic ischemia, or alcoholic liver injury, the case was considered a “non-DILI case”. If a diagnosis of DILI could not be excluded based upon the initial review of the excel data, medical records and notes in the EMR were further reviewed to determine whether DILI was present. The following data was then manually extracted from all of the potential DILI cases: patient age, gender, significant medical history, timing of drug initiation, liver biochemistries over time, available viral hepatitis serologies, autoimmune markers, liver biopsy results (if available), clinical setting (outpatient, ER, inpatient), gastroenterology consultation, and clinical outcomes including the need for liver transplant and mortality. This information was then reviewed by an experienced hepatologist (RJF) and the data extractor (LH) to determine the likelihood of DILI. DILI cases caused by acetaminophen overdose, chemotherapy, and experimental drug therapy were excluded.
Causality Assessment
All potential DILI cases identified from the EMR search were initially classified as being either a non-DILI case or a DILI case. Expert opinion was used to categorize the DILI cases as follows: 1= “Probable DILI” if DILI was more likely than not to have caused the liver injury (i.e. > 50% probability); 2= “Possible DILI” if alternative etiologies of liver injury were more likely but DILI was still possible (i.e. ≤ 50% probability of DILI); 3= “Historical DILI” if a document referred to a remote episode of DILI with incomplete medical information; and 4=“Allergy only” if a hepatotoxic reaction was listed as a drug allergy but the reaction could not be verified in the EMR. RUCAM scores were also calculated for all probable and possible DILI cases in which the timing between drug exposure and liver function test trend was known14.
Estimates of Time to review cases
The time needed to review the snippets of text and narrative of each case and determine if it represented a non-DILI or true DILI case was estimated by the physician reviewer (LH).
Statistical Analysis
Descriptive statistics were reported for baseline variables. Normally distributed data are reported as mean ± standard deviation while non-normally distributed data are reported as median (range). The sensitivity, specificity, positive and negative predictive value of each searching algorithm was determined. P values were calculated using Fischer exact test for categorical data.
Results
Initial EMR search
Between August 1, 2012 and December 31, 2013, there were 87,000 emergency room visits and 440,000 outpatient clinic visits recorded in the EPIC EMR. The initial search for documents that included at least one of the 14 liver injury terms and a suspect drug name yielded 2,564 potential DILI cases (representing 4,577 documents) (Figure 1). After manual review of the 2,564 records, there were 101 (4%) DILI cases identified which included 62 probable DILI cases, 25 possible DILI cases, 9 historic DILI cases, and 5 allergy only cases (Figure 2). Most of the non-DILI cases did not have any evidence of liver injury and were identified in our search because they contained two unrelated search terms. Figure 1 illustrates an example of such a case. In addition, some non-DILI cases had pre-existing liver disease with no evidence of acute DILI. The liver injury terms that contributed to the 101 DILI cases are displayed in Table 1. The term “liver disease” was not very useful in identifying DILI cases despite the large number of cases that contained those text words. Furthermore, the terms “drug-induced hepatotoxicity “, “drug-induced liver damage”, “drug hepatotoxicity”, and “adverse liver reaction” were also not identified in any of the DILI cases.
Figure 1. Example of Data Outputs.
Snippets of text (SNIPS) containing 200 characters around each search term were imported into an Excel file. A physician reviewed the SNIPS to determine if the case was a DILI or non-DILI case. If the available SNIPS were non-revealing, the entire document text consisting of up to 32,000 characters was reviewed. Case 1 illustrates a “probable DILI” case in a 69 year old man who developed cholestatic hepatitis following augmentin exposure. In contrast, Case 2 illustrates a non-DILI case wherein the term “liver disease” was incidentally mentioned as a pertinent negative in the medical history.
Figure 2. Data output.
Search 1 combined 14 liver injury terms with 8 drug terms. Of the 2,564 cases, 101 (4%) were DILI cases. The non-DILI cases either had no evidence of liver injury or a clear alternative explanation for the liver injury.
Table 1.
Utility of the 14 individual liver injury terms to identify potential DILI cases and true DILI cases.
| Liver Injury term | Potential DILI cases (n) | True DILI cases (n) | Predictive Value (% true DILI) | True DILI cases found ONLY with this term (n) |
|---|---|---|---|---|
| Drug-induced liver toxicity | 1 | 1 | 100 | 0 |
| Drug Induced liver injury | 41 | 36 | 88 | 0 |
| DILI | 16 | 13 | 81 | 4 |
| Drug-induced hepatitis | 38 | 14 | 37 | 5 |
| Liver injury | 127 | 47 | 37 | 3 |
| Drug-induced liver disease | 4 | 1 | 25 | 0 |
| Hepatotoxicity | 151 | 15 | 10 | 10 |
| Liver damage | 45 | 4 | 7 | 2 |
| Liver toxicity | 89 | 4 | 4 | 0 |
| Liver disease | 2268 | 57 | 3 | 19 |
| Drug-induced hepatotoxicity | 1 | 0 | NA | 0 |
| Drug-induced liver damage | 0 | 0 | NA | 0 |
| Drug hepatotoxicity | 0 | 0 | NA | 0 |
| Adverse liver reaction | 0 | 0 | NA | 0 |
| Total | 2781 | 192 | 43 |
Refinements of EMR search
To improve the efficiency of the search strategy, the term “liver disease” was eliminated and the search of the Clarity database was repeated. With this approach, the total number of cases to be reviewed was reduced from 2564 to 431 potential DILI cases. A careful review of these 431 cases demonstrated that 82 (19%) were DILI cases and that 19 previously identified DILI cases from the initial search were no longer included (Table 2). In addition, the elimination of the four terms that did not yield any DILI cases along with the four other liver injury terms with a detection rate of ≤10% (i.e. “hepatotoxicity”, “liver damage”, “liver toxicity”, “ liver disease”) further reduced the number of cases to be reviewed to 166 potential DILI cases. Of these, 66 were DILI cases leading to a significantly higher positive predictive value (40%) compared to the initial search strategy.
Table 2.
Utility of using 4 to 14 combined liver injury terms
| Search Strategy | Total potential DILI cases | Total true DILI cases | # DILI cases missed | Search sensitivity | Search specificity | PPV | NPV | Estimated time to review search output |
|---|---|---|---|---|---|---|---|---|
| #1: All 14 liver injury terms | 2564 | 101 | 0 | 100% | 99.5% | 4% | 100% | 29 hours |
| #2: All 14 liver injury terms except “liver disease” | 431 | 82 | 19 | 81% | 99.9% | 19% | 100% | 10 hrs |
| #3: Six high yield liver injury terms* | 166 | 66 | 35 | 65% | 99.9% | 40% | 100% | 6 hrs |
| #4: Four very high yield liver injury terms† | 84 | 54 | 47 | 53% | 99.9% | 64% | 100% | 5 hrs |
PPV=Positive predictive value. NPV= Negative predictive value
Six high-yield liver injury terms Drug induced liver toxicity, drug-induced liver injury, DILI, Drug induced hepatitis, Liver injury, drug-induced liver disease
Four very high yield liver injury terms: Drug induced liver toxicity, drug-induced liver injury, DILI, Drug induced hepatitis
Finally if only the four highest yield liver injury terms from Table 1 were used (which includes drug-induced liver toxicity, drug-induced liver injury, DILI, and drug-induced hepatitis) the total number of potential DILI cases was only 84. Since 54 of these 84 cases were DILI cases, the positive predictive value increased from 4% to 64%. However, this approach also reduced the overall detection rate of DILI cases from 100% to 53%. (Figure 3).
Figure 3. Variations of the initial EMR search.
Search #1 included all liver injury terms (n=14). Search #2 included all of the liver injury terms except for the term “liver disease. Search #3 included only 6 high-yield liver injury terms while Search #4 included the 4 liver injury terms with the highest yield.
Time to Complete record review
The investigators estimated that it took approximately 5 minutes to review the 4 snippets and narrative in the excel spreadsheet for cases which represented DILI. Conversely, the investigators estimated that it only took approximately 30 seconds to identify a non-DILI case from the available snippets. With these estimates, the total amount of time to complete the initial search was 29 hours to identify 101DILI cases (17 minutes/DILI case). In contrast, it only took about 5 hours to review the 84 charts using the 4 high yield liver injury terms that yielded 54 DILI cases (5 minutes/DILI case). (Table 2).
Clinical characteristics of the DILI cases
The suspect drugs, presenting features, and outcomes of the 62 probable DILI cases were reviewed and compared to the 39 other possible DILI, historic DILI, and allergy only cases (Table 3). The most commonly identified drugs in the probable DILI cases tracked very closely to the drugs targeted in this search. Specifically, augmentin, nitrofurantoin, trimethoprim-sulfamethoxasole, and isoniazid were most frequently identified as the suspect drug. However, other drugs beyond those targeted in the search were identified in 23% of the probable DILI cases and 33% of the other DILI cases.
Table 3.
Clinical and laboratory data of the 101 true DILI cases.
| Probable DILI cases | Possible, historic, and allergy only DILI cases | |
|---|---|---|
| N | 62 | 39 |
| Drug Name | ||
| Augmentin | 11 (18%) | 4 (10%) |
| Nitrofurantoin | 8 (13%) | 2 (5%) |
| Trimethoprim/sulfamethoxazole | 6 (10%) | 7 (18%) |
| Isoniazid | 5 (8%) | 2 (5%) |
| Ciprofloxacin | 5 (8%) | 1 (3%) |
| Minocycline | 4 (6%) | 2 (5%) |
| Levofloxacin | 2 (3%) | 1 (3%) |
| Valproate | 2 (3%) | 1 (3%) |
| Phenytoin | 1 (2%) | 0 |
| “Other” drug | 14 (23%) | 13 (33%) |
| Multiple drugs/Unknown drug | 4 (7%) | 6 (15%) |
| Mean age at DILI onset ± std dev (yrs) | 51 ± 18 | 48 ± 20 |
| Number of patients <18yrs (%) | 2 (3.2%) | 5 (15.1%) |
| Year of DILI onset | ||
| Prior to 2007 | 6 (10%) | 2 (5%) |
| 2007-August 2012 | 16 (26%) | 11 (28%) |
| August 2012-2013 (study period) | 40 (65%) | 20 (51%) |
| Unknown | 0 | 6 (15%) |
| Female gender | 39 (63%) | 24 (62%) |
| Ethnicity | ||
| Caucasian | 56 (84%) | 30 (77%) |
| African American | 6 (10%) | 3 (8%) |
| Asian | 1 (2%) | 3 (8%) |
| Other | 3 (5%) | 3 (8%) |
| Underlying Liver disease | 7 (11%) | 10 (26%) |
| Setting at DILI onset | ||
| Outpatient | 23 (37%) | 13 (50%) |
| Emergency Room only | 4 (6%) | 1 (4%) |
| Hospitalized | 35 (56%) | 12 (46%) |
| Unknown | 0 | 13 |
| Jaundice at DILI onset | 29 (47%) | 5 (42%) |
| Lab profile at DILI onset | ||
| Mean initial R value* | 8.7 ± 8.9 | 11.9 ± 17.9 |
| R≤ 2 (cholestatic) | 13 (21%) | 6 (30%) |
| R 2-5 (mixed) | 12 (19%) | 5 (25%) |
| Cases R≥ 5 hepatocellular | 37 (60%) | 9 (45%) |
| Unknown | 0 | 19 |
| Liver Biopsy performed | 28 (45%) | 11 (42%) |
| Liver transplant required due to DILI | 1 (2%) | 0 |
| Seen by gastroenterologist for evaluation | 54 (87%) | 21 (84%) |
R = ((Initial ALT)/(ULN))/( (Initial alk phos/ (ULN))
Percentage based on denominator using only cases with known values DILI=Drug Induced Liver Injury. LFT=Liver Function Test. RUCAM= Roussel-Uclaf Causality Assessment Method ULN = Upper limit of normal
The mean age of the 62 probable DILI cases was 51 years and 63% were female. Seven patients (11%) had underlying liver disease and the median duration of suspect drug use was 10 days (range: 1 day to 3 years). Interestingly, only 43% of the probable DILI cases were diagnosed and managed as an outpatient or in the ER while 57% were hospitalized for their DILI episode. Forty-seven percent of the probable DILI cases were jaundiced at presentation, 45% underwent liver biopsy as part of their diagnostic evaluation, and 87% were seen by a gastroenterologist for evaluation. During follow-up, 1 patient required liver transplantation but there were no deaths attributed to DILI.
The probable DILI cases were less likely to have underlying liver disease (11% vs. 26%) compared to the other DILI cases (possible, historic, allergy only) and also less likely to be managed as outpatients (37% vs 50%). However, the remainder of the clinical features in the two groups of patients were otherwise similar (Table 3).
Causality Assessment
Data to calculate RUCAM scores was available in 57 of the cases that were adjudicated by expert opinion as being probable DILI. The median RUCAM score was 7 points (range:−3 to 11), with the majority of patients (61%) ranking as “Probable” or “Highly probable” on the RUCAM scale and 33% scored as “”Possible” (Table 4).
Table 4.
RUCAM scores for cases adjudicated as “Probable” or “Possible” DILI by expert opinion.
| RUCAM Score | Probable DILI Cases* N=57 | Possible DILI Cases* N=17 |
|---|---|---|
| Not DILI (≤0) | 2 | 2 |
| Unlikely DILI (1-2) | 3 | 4 |
| Possible DILI (3-5) | 17 | 10 |
| Probable DILI (6-8) | 28 | 1 |
| Highly Probable DILI (>8) | 7 | 0 |
| Median RUCAM Score (range) | 7 (−3 to 11) | 3 (−3 to 7) |
RUCAM scores could not be calculated in 5 of the 62 probable DILI cases and 8 of the 25 Possible DILI cases due to missing data.
P < 0.005 by Fischer exact test comparing Possible and Probable scores
RUCAM = Roussel-Uclaf Causality Assessment Method
Data to calculate RUCAM scores was also available in 17 of the cases that were adjudicated by expert opinion as possible DILI. In support of a lower overall likelihood of DILI, the median RUCAM score was only 3 points (range −3 to 7), with the majority of patients (59%) scoring as only “Possible” DILI on the RUCAM scale (Table 4).
Discussion
The Drug Induced Liver Injury Network was established in 2003 to improve our understanding of the causes, mechanisms, and outcomes of patients with idiosyncratic DILI attributed to drugs and HDS products in the United States15,16. However, identifying patients with bonafide DILI for enrollment into mechanistic, epidemiological and clinical studies is difficult, given that DILI is a relatively infrequent event and a clinical diagnosis of exclusion. In addition, multiple clinical and pathological patterns of liver injury can be seen with a specific drug and there are no reliable, objective laboratory tests to confirm a diagnosis of suspected DILI18.
Investigators have used a variety of methods to search computerized EMR and administrative databases for potential DILI cases with varying success 8-11. In the current study, a novel text searching method was developed to peruse a large number of outpatient and ER encounters contained in a widely used commercial EMR product (EPIC) at a large tertiary care center. Rather than using surrogate terms and codes for liver disease employed by medical billers and coders, specific medical terms that providers use when describing a patient with potential DILI in their dictated note were tested in combination with a list of the generic and trade names of 8 well characterized hepatotoxic drugs. Over the 17 month study period, 2,564 potential DILI cases were identified from a database of over 500,000 outpatient and ER visits. A manual review of the 200 characters surrounding up to 4 text snippets extracted from the EMR of each identified liver injury term was undertaken along with a review of the entire extracted dictation when necessary. With this methodology, non-DILI cases could be quickly eliminated from further consideration (mean case review time ~ 30 seconds). DILI cases were also rapidly identified with a mean case review time of 5 minutes. Using this searching strategy, a total of 101 DILI cases were identified over the 17 month study period using 14 liver injury terms. Interestingly, only 60 of the 101 DILI cases actually transpired during the study time period while the remainder had occurred previously with over 15% having taken place prior to 2007. These data indicate that direct text searching of the EMR can be used for contemporaneous identification of ADR's as well as prior DILI events.
Because a large number of potential DILI cases were flagged in our initial search, we wanted to identify which liver injury terms were the most useful for a DILI case. Per Table 2, the elimination of “liver disease” from the liver injury terms markedly reduced the number of potential DILI cases (i.e. false positives) while maintaining a high sensitivity for DILI cases. Further reduction of the number of liver injury terms improved efficiency with an improved positive predictive value. Although use of the 4 highest yield liver injury terms dramatically reduced the total number of charts that needed to be reviewed from 2,564 to 84, the total number of DILI cases detected with this more restrictive approach was also reduced by 45%. Therefore, additional studies in other EMR databases are needed to confirm the optimal search strategy. The results of our study are similar to those of Overby et al who used a more complex computerized algorithm involving hierarchical searching methods11. In their study, a series of liver injury terms that were initially developed in a retrospective dataset of EMR's were shown to perform with a high degree of sensitivity and specificity for DILI cases in a prospective manner.
The direct text searching strategy employed in the current study yielded nearly five-fold the number of idiosyncratic DILI cases compared to a prior study we did involving ICD-9 codes over a 10 year time period 10. Furthermore, the current study included only outpatient and ER records whereas the prior study looked at outpatient, ER, and inpatient records. However, the EMR systems searched in the two studies were different (EPIC vs Careweb) and the current study focused on 8 drugs whereas the prior study looked only at 4 drugs. Nonetheless, a five-fold increase in the detection of DILI cases represents a substantially improved yield. We believe that the improved rate of DILI case detection is primarily due to improved specificity of the search terms used and the ability to search the actual physician text, rather than relying on surrogate billing codes. Furthermore, the total number of potential DILI cases to review was substantially reduced as well as the amount of time required to confidently exclude a non-DILI case via the use of extracted text snippets. Therefore, we feel that the new approach is a simple and efficient means to identify DILI cases.
A large proportion of the 101 DILI cases were determined to be probable (i.e. > 50% probability) or possible DILI (< 50% probability) via expert opinion causality assessment. The remaining DILI cases that were only mentioned in the EMR with no source documents to verify their accuracy were classified as “historic” DILI or “allergy only” cases. Prior studies have demonstrated that the inter-individual reliability and kappa scores are higher when utilizing a standardized expert opinion causality assessment method compared to the RUCAM instrument 19-21. In order to further assess the quality of the probable and possible DILI cases, RUCAM scores were also calculated. Of note, RUCAM scores could not be calculated for all cases particularly when the timing between suspect drug initiation and subsequent LFT's could not be confidently determined; this is a clear limitation of the retrospective nature of our study. However, the RUCAM scores that could be calculated were generally concordant with expert opinion assessments with the median score of the 57 probable DILI cases being substantially higher than the 17 possible DILI cases (7 vs 3, p < 0.005). Therefore, we feel that the cases identified through our search methodology are bonafide DILI cases and in fact, many of the eligible patients have met the enrollment criteria for the ongoing ILIAD study (R Fontana personal communication).
Interestingly, our text search method that was only targeting 8 pre-specified drugs also found some DILI cases attributed to other drugs in 27% of the DILI cases (Table 3). A careful review of these cases demonstrated that the targeted drugs were either being concomitantly prescribed or considered for treatment but not administered during the DILI episode. For example, a patient with azithromycin-induced liver injury was also noted to have an allergy to ciprofloxacin; this chart was found in our initial search because it contained the words “liver injury” as well as “ciprofloxacin”, even though the episode of DILI was actually referring to azithromycin-induced liver injury. Going forward, further refinement of our searching algorithm by using additional natural language processing tools, which relies upon the context of the note to help generate meaning from the text, may be needed to improve the ability to identify DILI due to a specific drug. For example, phrases that imply alternative causes of liver injury, i.e. “azithromycin-induced liver injury” could be eliminated, and specific phrases that imply DILI due to one of the target drugs (i.e. drug-induced liver injury due to minocycline) could be preferentially included.
The clinical presentation and features of the probable DILI cases identified in the current study are similar to what has been described in the ongoing DILIN prospective registry study 2,3. Although this study involved searching only the outpatient EMR, nearly 56% of the probable DILI cases had been hospitalized for the DILI episode and 47% were jaundiced at presentation. As noted in prior studies, a large proportion of the probable DILI patients underwent a liver biopsy for diagnostic evaluation presumably due to the difficulty in confidently establishing a diagnosis of DILI22,23.
Limitations of this study include that it was conducted retrospectively. However, natural language processing algorithms to identify DILI cases using liver injury terms by other investigators have demonstrated good performance when used both retrospectively and prospectively11. Second, this study was limited to a single large referral center that also performs liver transplantation. Therefore, our data may reflect a higher frequency and severity of DILI than seen in other centers. However, the objective and simple text searching method used can be replicated and potentially validated in other centers. Furthermore, the EMR database searched in the current study (i.e. EPIC) is currently used by over 40% of other medical systems in the US. Lastly, it is possible that there are additional liver injury terms beyond the ones selected in Table 1 that could provide a higher yield and greater specificity for potential DILI cases.
In summary, a text search information retrieval tool that combined a series of 14 liver injury terms with 8 drug names was able to identify 101 DILI cases including 62 probable cases from over 500,000 patient encounters over a 17 month period in a large computerized EMR database. Stepwise reduction of the liver injury search terms further improved the searching efficiency at a moderate cost of overall case detection. The identified DILI cases generally had high causality scores by both expert opinion and RUCAM scoring. Furthermore, the clinical characteristics, presenting features, and outcomes of the probable DILI cases were consistent with what has been reported in other prospective studies of DILI in the US and other western countries 3,5,22. Refinement and enhancement of these methods using additional natural language processing algorithms and incorporation of computerized laboratory and pathology data hold great promise in improving our ability to reliably and efficiently identify patients with idiosyncratic DILI.
Supplementary Material
Acknowledgments
Financial Support: The DILIN network is structured as a UO1 cooperative agreement with funds provided by the National Institutes of Diabetes and Digestive and Kidney Diseases (NIDDK) under grant 2U01-DK-065184-06 (Michigan).
Abbreviations
- ADR
Adverse drug reaction
- ALF
Acute liver failure
- ALT
Alanine Aminotransferase
- DILI
Drug Induced Liver Injury
- DILIN
Drug Induced Liver Injury Network
- EMR
Electronic Medical Record
- ER
Emergency room
- ICD-9
International Classification of Diseases
- NPV
Negative Predictive Value
- PPV
Positive Predictive Value
- RUCAM
Roussel-Uclaf Causality Assessment Method
- ULN
Upper limit of normal
Footnotes
Specific author contributions: All of the authors were involved in the study design, acquisition of data, analysis and interpretation of data as well as critical review of the final manuscript draft.
Potential conflicts of interest:
Dr. Fontana has received research support from Vertex Pharmaceuticals, BMS, Janssen, and Gilead.
Dr. Heidemann and Mr. Law have no conflicts.
References
- 1.Chalasani NP, Hayashi PH, Bonkovsky HL, et al. ACG Clinical Guideline: The diagnosis and Management of Idiosyncratic Drug-induced liver injury. Am J Gastroenterol. 2014;109:950–966. doi: 10.1038/ajg.2014.131. [DOI] [PubMed] [Google Scholar]
- 2.Fontana RJ, Hayashi PH, Gu J, Reddy KR, Barnhart H, the DILIN network Idiosyncratic Drug-induced liver injury is associated with Substantial Morbidity and Mortality within 6 months from onset. Gastroenterology. 2014;147:96–108. doi: 10.1053/j.gastro.2014.03.045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Chalasani N, Fontana RJ, Bonkovsky HL, Watkins PB, et al. Causes, clinical features, and outcomes from a Prospective study of Drug-induced Liver injury in the United States. Gastroenterology. 2008;135:1924–1934. doi: 10.1053/j.gastro.2008.09.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Fontana RJ. Pathogenesis of Idiosyncratic Drug-induced Liver Injury and Clinical Perspectives. Gastroenterology. 2014;146:914–928. doi: 10.1053/j.gastro.2013.12.032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Bjornsson ES, Bergmann OM, Bjornsson HG, Kvaran RB, Olafsson S. Incidence, Presentation, and Outcomes in Patients with Drug-Induced Liver Injury in the General Population of Iceland. Gastroenterology. 2013;144:1419–1425. doi: 10.1053/j.gastro.2013.02.006. [DOI] [PubMed] [Google Scholar]
- 6.Hayashi PH, Fontana RJ. Clinical features, diagnosis, and natural history of Drug Induced Liver injury. Sem Liv Dis. 2014;34:134–144. doi: 10.1055/s-0034-1375955. [DOI] [PubMed] [Google Scholar]
- 7.Garcia-Cortes M, Stephens C, Lucena MI, et al. Causality assessment methods in Drug induced liver injury: Strengths and weaknesses. J Hepatol. 2011;55:683–691. doi: 10.1016/j.jhep.2011.02.007. [DOI] [PubMed] [Google Scholar]
- 8.Meier Y, Cavallaro M, Roos M, Pauli-Magnus C, Folkers G, Meier PJ. Incidence of drug induced liver injury in medical inpatients. Eur J Clini Pharmacol. 2005;61:135–143. doi: 10.1007/s00228-004-0888-z. [DOI] [PubMed] [Google Scholar]
- 9.Duh MS, Walker AM, Kronlund KH. Descriptive epidemiology of acute liver enzyme abnormalities in the general population of central Massachusetts. Pharmacoepid Drug Saf. 1999;8:275–283. doi: 10.1002/(SICI)1099-1557(199907)8:4<275::AID-PDS427>3.0.CO;2-D. [DOI] [PubMed] [Google Scholar]
- 10.Jinjuvadia K, Kwan W, Fontana RJ. Searching for a Needle in a Haystack: Use of ICD-9-CM Codes in Drug-Induced Liver Injury. The American Journal of Gastroenterology. 2007;102:2437–2443. doi: 10.1111/j.1572-0241.2007.01456.x. [DOI] [PubMed] [Google Scholar]
- 11.Overby CL, Pathak J, Gottesman O, Haerian K, et al. A collaborative approach to developing an electronic health record phenotyping algorithm for drug-induced liver injury. Journal of American Medical Informatics Association. 2013;20:e243–3252. doi: 10.1136/amiajnl-2013-001930. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Warrer P, Hansen EH, Juhl-Jensen L, et al. Using text-mining techniques in electronic patient records to identify ADRs from medicine use. Br J Clin Pharmacol. 2011;73:674–684. doi: 10.1111/j.1365-2125.2011.04153.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc. 2001;8:254–66. doi: 10.1136/jamia.2001.0080254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Field TS, Gurwitz JH, Harrold LR, et al. Strategies for detecting adverse drug events among older persons in the ambulatory setting. J Am Med Inform Assoc. 2004;11:492–8. doi: 10.1197/jamia.M1586. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hoofnagle JH. Drug Induced Liver Injury Network. Hepatology. 2004;40:773. doi: 10.1002/hep.20445. [DOI] [PubMed] [Google Scholar]
- 16.Fontana RJ, Watkins PB, Bonkovsky HL, et al. Drug-Induced Liver Injury Network (DILIN) Prospective Study. Rational, Design, and Conduct. Drug Safety. 2009;32(1):55–68. doi: 10.2165/00002018-200932010-00005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Danan G, Benichou C. Causality Assessment of Adverse Reactions to Drugs—I. A Novel Method Based on the Conclusions of International Consensus Meetings: Application to Drug-Induced Liver Injuries. Journal of Clinical Epidemiology. 1993;46:1323–1330. doi: 10.1016/0895-4356(93)90101-6. [DOI] [PubMed] [Google Scholar]
- 18.Fontana RJ, Seeff LB, Andrade RJ, et al. Standardization of nomenclature and causality assessment in drug-induced liver injury: summary of a clinical research workshop. Hepatology. 2010;52:730–742. doi: 10.1002/hep.23696. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Rochon J, Protiva P, Seeff LB, et al. Reliability of the Roussel Uclaf Causality Assessment Method for assessing causality in drug-induced liver injury. Hepatology. 2008 Oct;48(4):1175–1183. doi: 10.1002/hep.22442. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Rockey DC, Seeff LB, Rochon J, et al. Causality assessment in drug-induced liver injury using a structured expert opinion process: comparison to the Roussel-Uclaf causality assessment method. Hepatology. 2010 Jun;51(6):2117–2126. doi: 10.1002/hep.23577. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Hayashi PH, Barnhart HX, Fontana RJ, et al. Reliability of causality assessment for drug, herbal, and dietary supplement hepatotoxicity in the Drug Induced Liver Injury Network. Liv International. 2014 doi: 10.1111/liv.12540. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Andrade RJ, Lucena MI, Fernandez MC, et al. Drug-induced liver injury: an analysis of 461 incidences submitted to the Spanish registry over a 10-year period. Gastroenterology. 2005;129(2):512–521. doi: 10.1016/j.gastro.2005.05.006. [DOI] [PubMed] [Google Scholar]
- 23.Kleiner D, Chalasani N, Lee WML, et al. Hepatic histological findings in suspected drug-induced liver injury: systematic review and clinical associations. Hepatology. 2014;59:661–670. doi: 10.1002/hep.26709. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.