Use of Text Searching for Trigger Words in Medical Records to Identify Adverse Drug Reactions within an Intensive Care Unit Discharge Summary

Summary

Purpose

To evaluate the performance of using trigger words (e.g. clues to an adverse drug reaction) in unstructured, narrative text to detect adverse drug reactions (ADRs) and compare the use of these trigger words to a targeted chart review for ADR detection within the intensive care unit (ICU) discharge summary note.

Materials

A retrospective medical record review was conducted. Evaluation of ADRs occurred in two phases – targeted chart review of the ICU discharge summary notes in Phase 1 and targeted chart review using specific words and phrases as triggers for ADRs in Phase 2.

Results

Four hundred ADRs were documented in 223 patients for Phase 1. For Phase 2, there were 219 ADRs identified in 120 patients. 138 real or accurate ADRs were identified from Phase 1 and 47 duplicate events. 34 ADRs from Phase 2 were not identified in Phase 1. Fifteen of the ADRs were inaccurately presumed in Phase 2. Fifty-eight of 127 text triggers identified at least one ADR. Low and moderate frequency trigger words were more likely to have PPVs > 5%.

Conclusions

Targeted chart review using specific words and phrases as triggers for ADRs is a reasonable approach to identify ADRs and may save time compared to other methods after further refinement leads to a more accurately performing trigger word list.

Introduction

Despite continued diagnostic and therapeutic advancements made in health care, more than two million adverse drugs reactions (ADRs) are estimated to occur in the United States annually, with approximately 33% being deemed preventable [1]. The opportunity for prevention underlies the importance of identifying these ADRs to reduce both the frequency and severity of future events. Numerous strategies for ADR detection have been studied to date. Voluntary reporting is the most common method utilized by hospitals and health systems to identify ADRs, with as many as 95% reporting in this manner [2, 3]. Despite its widespread use, voluntary reporting is insensitive and nonspecific for identifying ADRs [4–6]. One study found that the use of voluntary reporting identified less than 1% of ADRs identified by the investigators using triggers [4]. Another method of ADR detection, comprehensive manual chart review or non-targeted chart review, is an approach that involves the review of the entire chart and has been shown to typically detect more ADRs than other strategies [7]. However comprehensive manual review of medical records is a laborious process that cannot realistically be performed on a daily basis due to resource constraints [7–10].

There is a clear need for a more efficient approach to ADR detection that can accurately account for events that occur. Strategies to engage a more targeted approach to chart review would involve:

1. identifying treatment such as the administration of antidote medications as a trigger to an ADR occurrence;

2. text searching for specific words and phrases that may trigger the detection of an ADR;

3. review of a select section of the medical record [7].

These methods can alleviate some of the disadvantages of non-targeted review [4, 7]. One particular targeted computer-based ADR detection method that has been developed is an electronic screening method designed to search for text that may be indicative of the occurrence of an ADR within a hospital discharge summary note [11]. Murff and colleagues compared this electronic screening method with comprehensive manual chart review and found that the benefits of the automated system lied in its ability to optimize resources and process a greater number of patient charts. Forster and colleagues also examined the electronic screening method using specific words and phrases in narrative text and concluded that its high specificity for ADRs could allow for replacement of comprehensive chart review [13].

There is value in detecting ADRs specific to the critically ill patients. ADRs are of particular concern in the intensive care unit (ICU) as this vulnerable patient population experience ADRs with increased frequency and severity [14–16]. A study using targeted manual chart review demonstrated value in utilizing patient discharge summaries notes from the ICU as compared to hospital discharge notes for the identification of ADRs that occurred during ICU care [8]. An ADR was documented for 49% of patients in the ICU discharge summary note as compared to 25% of patients in the hospital discharge summary notes. The authors concluded that ICU discharge summary notes could effectively be used as a tool for the detection of ADRs within an ICU.

The purpose of this study was to evaluate the performance of using specific words and phrases in narrative text as triggers to detect ADRs and to compare use of these trigger words for ADR detection with a targeted chart review of the ICU discharge summary notes written by practitioners. Automation of this detection process through the use an electronic screening tool would allow greater applicability to general practice leading to an increased detection of ADRs and opportunities to create preventative systems based on detected information.

Methods

A retrospective electronic medical record review of the Medical Archival Retrieval System database was conducted for medical ICU patients at a large tertiary, academic medical center containing more than 150 ICU beds with 32 beds in the medical ICU. The electronic medical record review was from January 1, 2012, to April 30, 2012. Patients were included if they were at least 18 years old and had a minimum ICU length of stay (LOS) of 24 hours.

The review of electronic health records for the ICU discharge summary notes was conducted in a stepwise approach (Phase 1 and Phase 2). ICU discharge summary notes were considered the progress note on the final day of the patient’s ICU stay. Two pharmacists (AM, MK) conducted an independent electronic medical record review of de-identified ICU discharge summary notes for all patients included in the analysis. ICU discharge summary notes were reviewed by pharmacists for the documentation of ADRs by practitioners. Pharmacists utilized two scales to assess the ADRs objectively with accuracy and consistency; these included the Harvard Medical Practice Scale (MPS) and a modified National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) Index for classifying medication events [17–19].

The Harvard MPS was used to rank the strength or confidence of the wording in the documentation provided by the practitioner [12, 17, 18]. The Harvard MPS is a 6-point scale with

1. = little to no evidence for management causation;

2. = slight to modest evidence for management causation;

3. = management causation not likely; less than 50–50 but close call;

4. = management causation more likely than not, more than 50–50 but close call;

5. = moderate/strong evidence for management causation and

6. = virtually certain evidence for management causation.

ADRs were included if they received a 2 or greater score on the Harvard MPS, which required consistency by two reviewers. These criteria for a possible ADR were determined a priori based on a previous evaluation [12]. The modified NCC MERP Index was used to help score and standardize event severity [19]. It was modified in language from its original wording of medication errors and applied to ADRs, limiting the categories of use from letters D through I with

• D being an ADR that resulted in no harm or intervention to preclude harm;

• E an ADR that contributed to or resulted in temporary harm and required intervention;

• F an ADR that contributed to or resulted in temporary harm and required initial or prolonged hospitalization;

• G an ADR that may have contributed to or resulted in permanent harm;

• H an ADR occurred that required intervention to sustain life and

• I an ADR that may have contributed to or resulted in the patient’s death.

The findings of the Phase 1 review were evaluated by an additional pharmacist (either AH, AW), who was not included in the initial assessment. These pharmacists were trained using the same standardized methods as the primary reviewers. The purpose of the second review was to reconcile any discrepancies related to the presence of an ADR found by the initial reviewers. After a discrepancy was identified, the pharmacist performed an independent assessment of the chart and graded the event with the aforementioned scales. Agreement between two of the three reviewers using the inclusion criteria for ADRs provided earlier was considered a positive event.

Each ICU discharge summary note was then cross-referenced with a list of trigger words by use of text searching (Phase 2). Trigger words were derived from our previous study and from the literature [11, 12]. In our previous study, ADRs were identified in the ICU discharge note and the reviewer documented possible trigger words creating a comprehensive list [11]. In addition, a study evaluating trigger words in the hospital discharge summary was reviewed for words to include [12]. A total of 127 trigger words were used. Two additional pharmacists (AA, CK) were responsible for reviewing the ICU discharge summary note that contained trigger words. These pharmacists were provided 3 lines of text from the report to review. The line before the trigger word, the line with the trigger word and the line after the trigger word were reviewed. These pharmacists reviewed the summaries individually and categorized them with regards to whether an ADR possibly occurred through use of the same scales in Phase 1. After the individual assessment was completed for all the triggered ICU discharge summary notes, an additional pharmacist acted as an evaluator of discrepancies in a similar manner to the Phase 1 review. ► Figure 1 depicts a summary of the methods.

Fig. 1.

Summary of Methods

Analysis

A description of the ADRs including reaction and severity are provided for those ADRs identified in Phase 1. More than one drug could be associated with an ADR in the drug description for Phase 1; however the overall analysis was done by ADR event and not drug. The comprehensive review in Phase 1 was used as the comparator to Phase 2 to confirm the accuracy of events identified using the trigger and 3-line method ADRs. Events occurring in Phase 2 using trigger words were considered “real or accurate” when Phase 1 ADRs agreed or a Phase 2 event that was not identified in Phase 1 was confirmed by a pharmacist reviewer. An inaccurate identification of an event using trigger words in Phase 2 occurred when an ADR was found in Phase 2 that was not identified in Phase 1 and the absence of an ADR was verified by a pharmacist reviewer. Missed events occurred when Phase 1 identified an event that was not found in Phase 2, termed missed events-Phase 2. Missed events were also discovered for Phase 1, despite the initial impression that this should be negligible and this was termed missed events -Phase 1.

For Phase 2, an ADR identified in Phase 1 could be associated with more than one text word; however this was an evaluation of trigger words so the positive predictive value (PPV) was calculated for each trigger word. A true negative event occurred when a trigger word from the list of 13,493 was found in the note and no corresponding ADR was identified during the Phase 2 review, then when compared to Phase 1 there was no ADR identified for that patient. To calculate the PPV for each trigger word, we divided the number of times that a trigger word was used and an ADR was confirmed (i.e., the number of true-positives), by the number of times the trigger word was used with or without an ADR being confirmed (i.e., the sum of true-positives and false-positives).

Results

A cohort of 380 unique patient visits and 1502 notes on the day of discharge from the ICU were evaluated. The average age of the cohort was 58.5±17.4 years old with 56.8% male. The majority of patients were Caucasian (72.4%) and 11.3% were African American. The average ICU length of stay was 9.7±15.1 days and a median of 4.0 days.

Phase 1 – Targeted Chart Review of ICU Discharge Summary Notes

Four hundred ADRs were documented in 223 patients. The use of the Harvard MPS determined that 17% of ADRs had a confidence level of 2, 7% of ADRs had a confidence level of 3, 24% of ADRs had a confidence level of 4, 35% of ADRs had a confidence level of 5, and 17% of ADRs had a confidence level of 6. The 3 most common locations in the note that an ADR was described were plan, impression/plan and history of present illness. The most common ADRs were bleeding/supratherapeutic INR, mental status changes, acute kidney injury, thrombocytopenia, hypotension and Clostridium difficile. The severity of the events was D (24.3%), E (56.8%), F (17.5%) and G (1.5%). The average time spent reviewing events by electronic medical record review was 4 minutes and 36 seconds with a minimum time of 20 seconds and a maximum time of 34 minutes and a median time of 3 minutes and 12 seconds.

Phase 2 – Trigger Word Review

There were 219 ADRs identified in 120 patients. One hundred eighty-five of these ADRs were identified in Phase 1; however this translates into 34.5% (138/400) real/accurate events because there were duplicates (n = 47) of the same event using triggers for identification. The remaining 34 ADRs from Phase 2 were not identified in Phase 1. There were 6.8% (15/219) of the ADRs that were inaccurate indicating they were not actually ADRs and were overestimated by reading the 3 text lines. Nineteen (8.8%) of the ADRs were actual and missed events – Phase 1 with total electronic medical record review despite having 2 reviewers. Also, there were 262 missed events-Phase 2. ► Figure 2 illustrates the overlap in identification of events between Phase 1 and Phase 2. The severity of the 138 ADRs was D (17.4%), E (62.3%), F (18.1%) and G (2.2%) per determination in Phase 1 where the detail of the event existed. The amount of time was less than 1 minute for reviewing 202 of the events and 1–5 minutes for 17 of the events using the trigger method. The average time spent reviewing the same events by electronic medical record review in Phase 2 was 4 minutes and 40 seconds with a minimum time of 32 seconds and a maximum time of 34 minutes and a median time of 3 minutes and 12 seconds.

Fig. 2.

Description of the overlap for events identified in Phase 1 and Phase 2

Trigger words were cited a total of 13,493 times in the 1502 notes or on average 9 times per note. There were 45.7% (58/127) of triggers that identified at least one ADR as shown in ► Table 1. The number of true negative trigger words was 13,289. The 10 most commonly used triggers were blood, 2/2, due to, low, positive, elevated, d/c, hold, control and DVT. None of the triggers with high (>100) frequency of use in the chart had a PPV of >5% but a few had PPVs of >4% including concern, and related. Triggers with moderate (20–100) frequency of use in the chart had several with PPVs of >5% included drop, FFP, held, hemoptysis, HIT, hypoglycemia, opiates, pancytopenia, rash, rising, stop and supratherapeutic. Frequent use of trigger words with a low yield for ADR detection was because the trigger word was used in context other than an ADR. For example, rash was identified 54 times but only a true ADR in 3 cases because the patient had a rash but was not due to an ADR or the patient had a history of a rash described in the chart. Triggers with low (<20) frequency of use in the chart had several with PPVs >5% including allergy, bradycardic, contrast nephropathy, drug fever, drug-induced, effect of, high INR and prolonged QTc.

Table 1.

Trigger Words in Phase 2 Associated with at Least 1 Real Adverse Drug Reaction Confirmed by Pharmacists

Trigger Word*	Number of Trigger Words in Notes(n= 13,493)	ADR Identified Using Trigger Word in Phase 2(n=204)	Positive Predictive Value (%)
2/2	859	25	2.9
Accident	6	-	-
ACLS	1	-	-
Acute renal failure	9	-	-
Agitation	62	2	3.2
AKI	192	3	1.6
Allergic	11	-	-
Allergy	6	1	16.7
AMS	109	-	-
ARF	11	-	-
Arrest	86	-	-
Bleed	153	2	1.3
Blood	3525	4	0.11
Bradycardia	40	1	2.5
Bradicardic	9	1	11.1
C diff	100	1	1.0
Changes	148	3	2.0
Code	73	-	-
Coma	3	-	-
Complication	5	-	-
Concern	312	15	4.8
Condition C	59	-	-
Contrast Nephropathy	2	2	100
Control	313	1	0.32
d/c	393	8	2.0
D50	4	-	-
Dc’d	13	-	-
Decrease	102	3	2.9
Deficiency	18	-	-
Delirium	93	2	2.2
Desaturation	20	-	-
Diarrhea	101	1	1.0
Discontinue	41	-	-
DKA	33	-	-
Drop	83	5	6.0
Drowsy	22	-	-
Drug fever	1	1	100
Drug-induced	2	1	50
Due to	711	21	3.0
DVT	240	1	0.42
Effect of	3	1	33
Elevated	380	8	2.1
Epistaxis	15	-	-
Error	1	-	-
ETOH abuse	15	-	-
FFP	44	3	6.8
Fluid resuscitation	6	-	-
Glucagon	4	-	-
Hallucinations	11	-	-
Held	71	5	7.0
Hematoma	18	-	-
Hemoptysis	35	2	5.7
High INR	2	1	50
HIT	61	6	9.8
Hives	1	-	-
Hold	346	4	1.2
Hyperglycemia	15	-	-
Hypernatremia	16	-	-
Hypoglycemia	29	3	10.3
Hypotension	195	3	1.5
Hypotensive	150	5	3.3
Hypoxemia	47	-	-
Hypoxia	43	-	-
Hypoxic	29	1	3.4
Iatrogenic	5	-	-
Incorrect	3	-	-
Ingestion	25	-	-
Injury	74	-	-
Interaction	2	-	-
Intubate	7	-	-
Intubation	137	1	0.73
Lethargic	25	1	0.04
Low	570	7	1.2
Marijuana	8	-	-
Melena	41	-	-
Mental status	201	5	2.5
Metabolic acidosis	1	-	-
NAC	12	-	-
Narcan	23	-	-
Non-compliance	1	-	-
Noncompliance	1	-	-
OD	19	-	-
Opiates	25	3	12
Overdose	63	-	-
Overload	66	1	1.5
Oversedation	4	-	-
Pancytopenia	22	2	9.0
Polysubstance	19	-	-
Positive	545	1	0.18
PRBC	78	-	-
Prolonged QTc	2	1	50
Pulmonary Fibrosis	35	-	-
Rash	54	3	5.6
Rebound	31	-	-
Related	167	8	4.8
Renal Disease	6	-	-
Renal Failure	87	1	1.1
Respiratory Distress	76	1	1.3
Respiratory Failure	205	-	-
Rhadomyolysis	1	-	-
Rise	26	-	-
Rising	28	2	7.1
SAH	15	-	-
Secondary to	275	6	2.2
Sedated	97	2	2.1
Sedating Medications	1	-	-
Seizure	164	-	-
Serotonin Syndrome	1	-	-
Shortness of breath	45	-	-
SOB	162	1	0.62
Somnolent	114	1	0.88
Stop	26	3	11.5
Subtherapeutic	7	-	-
Suicide attempt	19	-	-
Supratherapeutic	20	3	15
SVT	8	-	-
Switch	48	2	4.2
Tamponade	17	-	-
Temporal	25	1	4
Thrombocytopenia	31	1	3.2
Toxic	25	-	-
Transfusion	49	1	2.0
UGIB	9	-	-
Unresponsive	79	-	-
Vit K	4	-	-
Vitamin K	16	-	-
Withdrawal	35	-	-

^* Results do not indicate change in upper and lower case. For example, we identified both SOB and sob and both Hold and hold

Discussion

This evaluation demonstrates that one-third of ADRs are detected using a targeted medical record review of trigger words. This approach appears to be more time efficient than a comprehensive medical record review as most reviews took less than 1 minute; however the true cumulative time advantage needs to be confirmed with a more refined list of trigger words. This potential resource benefit will be particularly important when pharmacovigilance entails a tremendous number of medical records. We also report which trigger words are useful. Positive predictive value is influenced by the number of events identified such that the PPV is lower for frequently (>100 times) occurring trigger words compared to the low (<20) frequency trigger words. It is not surprising that in the chart, one can find the word blood written >3500 times, similarly, 2/2, hold and low are used frequently to describe other non-ADR events. For those trigger words occurring with high frequency, a pharmacovigliance program should focus on words that performed well such as 2/2, concern and related. ADRs occurring in the ICU are the primary focus of this study, unlike previous studies [12, 13], since ADR surveillance systems will need prevention strategies specific to the environment due to the variation in drugs used, ADR causes and ADR severity [13–15].

Surveillance systems typically depend on voluntary reporting and no matter how streamlined this process becomes with the use of tools and automation underreporting still remains its limitation [5, 20]. The ICU discharge summary note, as a targeted medical record review, is a source to identify ADRs with 57% of patients having an ADR documented in the medical record in our study. This rate is comparable to the previous study that reported 49% of patients with ADR documentation in the ICU discharge summary note [12]. Again, resource efficiency with this approach remains the concern for application and the identification of well performing trigger words and an automated approach will simplify the process. It is known that different methods of ADR detection identify different events, making a comprehensive surveillance system that uses different methods ideal [4, 7, 10, 21]. One would not expect a targeted trigger approach using text would yield the same results as a targeted trigger review using laboratory values or antidote medications, but this has not been tested. Also, the trigger words used in our study were used to find recognized or documented events and antidote medications as triggers could be useful to find undocumented ADRs. The ICU discharge note is a practical addition to an ADR surveillance system and the 3-line text approach may be a practical means of incorporation.

The identification of events using a retrospective approach, meaning identifying events after the patient has left the hospital affords institutions the opportunity to make systematic changes to prevent future events [7]. The 3-line text approach identifies the events and culminating the information provides areas for suggested improvement. There still may be a need for a more comprehensive review of the medical record once an ADR is identified with the text word search to determine the cause of the ADR and direct systematic changes. This decision of a detailed review will depend on the goals of the institution.

The next step in trigger word surveillance would be to test more words. Other steps could include educating practitioners on documenting and discussing ADRs in the medical record using common language. Emphasizing sections of the notes (i.e. plan) that commonly yields an ADR description leading to detection may further simplify the process and allow for fewer triggers words not related to an ADR. Further, development of a deep phenotype that extends beyond a word to phrases may permit an even more accurate method of detection [22]. Another future direction is linking trigger words to specific ADRs as the current list is a broad evaluation for general ADRs.

Limitations

The evaluation of text was specific to the ICU discharge note so the most useful trigger words may not apply to notes of the entire ICU stay or non-ICU notes, thus limiting generalizability. Also, we assumed the notes on the last day of the ICU stay were the discharge notes as our electronic health record does not have a specific note entitled “ICU discharge summary”. The inclusion of all notes on the day of discharge included 1502 notes in 380 patients due to multiple services writing notes on that day. This selection could have been more streamlined. We used a rigorous approach to the comprehensive ICU medical record review by including 2 pharmacists, yet, trigger words in phase 2 still identified 34 events that were missed in phase 1. This does identify human error and an opportunity for automation to assist. The severity analysis in phase 1 was applied to phase 2 because as we proceeded with the 3-line medical record review, we realized there was insufficient information to perform a severity analysis.

We identified a list of 127 words based on the literature and previous work, still other words could be tested [11, 12]. We used some abbreviations and different word variations; however misspellings were not accounted for in this method. We identified one-third of ADRs, however two-thirds were not captured so the investigation of other words may enhance these results. Also, the trigger words we evaluated were for recognized events documented in the medical record by the practitioner, a different set of triggers may be needed to find unrecognized or undocumented events [5].

Conclusion

Discharge summary notes are a source of ADR information. Targeted chart review of the ICU discharge summary note is an alternative method to identify ADRs. Targeted chart review using trigger words is a useful approach to identify ADRs and may save time compared to other methods. Use of trigger words runs a risk of low yield in ADR detection for some words so ultimate resource efficiency cannot be determined until the trigger word list is refined. Trigger words other than those evaluated in this study should be investigated to maximize ADR detection. Developing a surveillance system that extends beyond the known limitations of a system dependent on voluntary reporting while maintain the efficient use of resources can consider the use of a targeted chart review with electronic screening using trigger words. A comprehensive pharmacovigilance program will incorporate multiple methods of ADR detection.

Funding Statement

Funding Not obtained for this project.