Evaluation of accuracy of IHI Trigger Tool in identifying adverse drug events: a prospective observational study

Abstract

Aims

Adverse drug events (ADEs) can seriously compromise the safety and quality of care provided to hospitalized patients, requiring the adoption of accurate methods to monitor them. We sought to prospectively evaluate the accuracy of the triggers proposed by the Institute for Healthcare Improvement (IHI) for identifying ADEs.

Methods

A prospective study was conducted in a public university hospital in 2015 with patients over the age of 18. Triggers proposed by IHI and clinical alterations suspected to be ADEs were searched daily. The number of days in which the patient was hospitalized was considered as unit of measure to evaluate the accuracy of each trigger.

Results

A total of 300 patients were included in this study. Mean age was 56.3 years (standard deviation (SD) 16.0), and 154 (51.3%) were female. The frequency of patients with ADEs was 24.7% and with at least one trigger was 53.3%. From those patients who had at least one trigger, the most frequent triggers were antiemetics (57.5%) and ‘abrupt medication stop’ (31.8%). The sensitivity of triggers ranged from 0.3 to 11.8% and the positive predictive value ranged from 1.2 to 27.3%. Specificity and negative predictive value were greater than 86%. Most patients identified by the presence of triggers did not have ADEs (64.4%). No triggers were identified in 40 (38.5%) ADEs.

Conclusions

IHI Trigger Tool did not show good accuracy in detecting ADEs in this prospective study. The adoption of combined strategies could enhance effectiveness in identifying patient safety flaws. Further discussion might contribute to improve trigger usefulness in clinical practice.

What is Already Known about this Subject

• Occurrence of adverse drug events (ADEs) in the process of providing health care compromises quality of care and patient safety.

• Institute of Health Improvement (IHI) Trigger Tool has been used worldwide on retrospective reviews of hospital patient records to search for ADEs.

• To our knowledge, the accuracy of the IHI Trigger Tool in recognizing ADEs has not been tested in prospective studies.

What this Study Adds

• IHI triggers did not show good accuracy in detecting ADEs in this prospective study, showing a global positive predictive value of 0.084.

• Overall, 160 (53.3%) patients had at least one trigger and, among these, 57 (35.6%) had at least one ADE, but only 36 (22.5%) showed a clinically plausible association.

• The use of triggers would not impact real‐time changes in clinical practice, but its assessment would help hospitals planning strategies and alternative approaches to improve trigger usefulness for patient safety.

Introduction

The occurrence of adverse events with consequent compromising of the quality of care and safety provided to the patient is of great concern in the process of providing health care. Adverse events are also considered the third most common cause of death in the United States 1. Of these, adverse drug events (ADEs) are among the most common episodes 2, 3, 4, 5, 6, 7 which justified the launch of the World Health Organization's third Global Challenge for Patient Safety, Medication without Harm 8.

In the search for strategies to improve patient safety, ADEs require investigation and analysis, as they pinpoint flaws in the process of the use of medications and provide important information for the improvement of health care provided to patients 9, 10. The monitoring of ADEs using ‘triggers’ is a trend in healthcare services in developed countries aimed at improving and enhancing the process of detecting adverse events 10, 11, 12, 13, 14, 15, 16. Several studies conducted in hospitals in many countries used this method proposed by the Institute for Healthcare Improvement (IHI) or its adapted versions to search for ADEs 5, 11, 12, 17, 18, 19, 20. The method, entitled the IHI Global Trigger Tool, is based on the retrospective review of a random sample of hospital patient records using triggers to identify the events that caused harm to the patient 21.

Although the use of triggers has been extensively referred to as a relevant tool in the field of patient safety, previous studies have found poor performance of the retrospective use of triggers in predicting ADEs from patient chart reviews 10, 22, 23, 24, 25. The ability of prospective use of triggers in detecting ADEs is still to be elucidated and this evaluation could potentially contribute to the quality and safety of patient care. Thus, we designed this study with a robust methodology to screen for ADEs in patients with positive and negative triggers. We aimed to prospectively evaluate the accuracy of these triggers to identify ADEs in patients hospitalized at a public university hospital in Brazil.

Methods

Study setting and subjects

This was an observational, prospective study conducted from 26 January to 6 March 2015. The study was carried out in a large‐scale, general, public university hospital that is a regional reference centre in specialized care. The research protocol was approved by the Research Ethics Committee from the Universidade Federal de Minas Gerais (code number 28222514.3.0000.5149). All of the participants or respective legal guardians signed an informed consent form.

This study included patients over the age of 18 who were hospitalized during the collection period within the medical and surgical clinical units, independent of whether any of the IHI triggers we selected was present or not during hospitalization. Patients in respiratory isolation, with communication difficulties and without available responsible family member were excluded.

Definitions

The following World Health Organization (WHO) 26 definitions were adopted for this study. (i) ADE: any incident resulting from the process of the use of medication that results in harm or injury to the patient, including adverse drug reactions and medication errors. (ii) Preventable ADE: any ADE that would not have occurred if the patients had received the proper health care according to the normal standards indicated for the circumstance in which the event occurred. Adverse drug reactions resulting from medication errors were considered preventable.

Variables and data collection

Data collection was performed by two independent teams trained to focus on triggers or on clinical alterations screened to be scrutinized afterwards for the presence of ADEs by the group of specialists. The twelve triggers from the second edition of the IHI Global Trigger Tool from the Medication Module Triggers group were used as references 21: (i) five medications: diphenhydramine, vitamin K, naloxone, flumazenil and antiemetics (metoclopramide, bromopride and ondansetron) were considered present when they were prescribed and their administration was registered by the nursing staff; (ii) five laboratory parameters: glucose <50 mg dl⁻¹, positive exam for Clostridium difficile, partial thromboplastin time (PTT) >100 s, international normalized ratio (INR) >6.00, serum creatinine increased twice the level of basal creatinine (first creatinine after inclusion of the patient was considered as the baseline); (iii) two information about patient's clinical evolution and care: excessive sedation/hypotension (considered present when there was an entry in the medical records using the expression ‘excessive sedation’ or ‘very drowsy’, accompanied by a blood pressure lower than 90/60 mmHg on the same day) 12 and ‘abrupt medication stop’ (considered present when there was an unexpected suspension of medication due to a sudden change in patient condition).

Considering the ADEs related to the triggers we selected for this study, and those ADEs found in other studies 4, 5, 12, 17, 20, we used 18 clinical alterations to screen for ADEs. As mentioned before, these clinical alterations were collected separately from the triggers and were represented by: change in respiratory pattern, lethargy/drowsiness, hypoglycaemia, hyperglycaemia, haemorrhage, bradycardia, tachycardia, renal lesion, hepatic lesion, constipation, diarrhoea, dysentery, skin rash, hypotension, hypertension, nausea and vomiting, falls, and tremors. The parameters pre‐determined in the study protocol were used to identify these clinical alterations. Data were collected by Pharmacy students supervised by clinical pharmacists from medical records, laboratory exams results, medical prescriptions and nurses' records.

The assessment of presence or absence of ADEs and their classification were performed by clinical pharmacists, nurses and physicians with large clinical experience in ADE assessment. These professionals will be referred to in this study as specialists. To minimize the disagreement among teams in judging ADEs that have been reported by other studies 24, 27, we adopted standard procedures to assess ADEs according to five steps, as follows:

• Step 1Patient recruitment.
• Step 2Data of interest were collected daily. Data regarding prescriptions and laboratory exam results were extracted from the computer system used in the hospital.
• Step 3A group of collaborators searched daily exclusively for the triggers proposed by the IHI to detect ADEs. Another group of collaborators performed a separate search exclusively for the clinical alterations of interest. Therefore, collaborators involved in the verification of the triggers did not search for clinical alterations and vice versa, in order to minimize collection and interpretation bias.
• Step 4The identified clinical alterations were evaluated by a pharmacist and a nurse separately and were classified according to the following options: event resulting from the natural course of the disease/comorbidity or ADE. Divergent cases and those assessed as with an ADE by the pharmacist and the nurse were submitted to the first physician reviewer. Later, the cases in which the first physician reviewer did not agree with the pharmacist and/or the nurse were forwarded to a second physician reviewer. When there was a disagreement among the physicians, a meeting was set up to reach a consensus on the issue (Figure 1).
• Step 5All of the cases defined as ADEs were evaluated by two pharmacists to identify the main suspected medication and verify the plausible association of the event with the triggers identified. ADEs were also classified by the severity of the harm caused to the patient (mild, moderate, severe and death) and by their preventability (preventable, not preventable), according to WHO standards 26.

Figure 1.

Flowchart of the process of identification and absolute frequency of the adverse drug events, identified in 300 patients studied at a university hospital

Statistical analysis

The determination of accuracy between the trigger and the ADE was performed only when a plausible association was identified. Accuracy was assessed using the number of days in which patients were hospitalized as a unit of measure, given that the analysis per patient would hinder the possibility of investigating false‐positive and true‐negative cases. Analyses were later performed using the number of patients as a unit of measure. The measures of accuracy used herein were: sensitivity (Se), defined as the proportion of days in which there was a trigger with an ADE within the total number of days in which the ADE existed; specificity (Sp), defined as the proportion of days in which there was no trigger within the total number of days in which the ADE did not exist; positive predictive value (PPV), defined as the proportion of days in which the ADE appeared within the days in which the trigger was present; the negative predictive value (NPV), defined as the proportion of days in which there was no ADE during the days in which there was no trigger.

As there was dependence among the patients' data during the different days of hospitalization, this study adopted the non‐parametric bootstrap resampling method to calculate the confidence interval percentages. One thousand bootstrap resamples were considered for each interval. A level of significance of 0.05 was adopted in all of the analyses. Analyses were later performed using the number of patients as a unit of measure. The statistical analyses for the description of the variables were conducted using the SPSS software (version 21.0, SPSS Inc., Armonk, NY), while the measures of accuracy were performed using the R software (version 3.2.2, R Core Team, 2015).

Results

A total of 300 patients with an average age of 56.3 years (SD, 16.0) and predominantly female (51.3%) were included. The total number of hospitalization days was 2776 for the total of patients in the study, with a median of 7.0 (IQR, 3.0–13.0) hospitalization days.

In all, 739 clinical alterations were identified, of which 104 (14.1%) were classified as ADEs after assessment by specialists, as shown in Figure 1. The prevalence of ADEs was 37.5 events per 1000 patients per day or 24.7 events per 100 patients. Regarding the degree of severity of harm to the patient, 49% of the ADEs were considered mild, 47.1% moderate and 3.9% severe. With regard to preventability, 39 (37.5%) were considered preventable. Hypotension was the most common ADE (21.2%), and 31.8% of these were considered preventable. The second most frequent ADE was constipation (18.3%), with 63.2% classified as preventable as they occurred without laxative interventions (diet and/or medication) when these were necessary. Hyperglycaemia occurred in 14 (13.5%) and haemorrhage in 13 (12.5%) of the cases. These four types of more frequent ADEs corresponded to 65.4% of the total number of ADEs (Table 1).

Table 1.

Characterization of adverse drug events identified in 300 patients studied in a university hospital

	ADEsa (n = 104)
	n (%)	With trigger	Preventable
Hypotension	22 (21.2)	13	7
Constipation	19 (18.3)	13	12
Hyperglycaemia	14 (13.5)	7	4
Haemorrhage	13 (12.5)	5	2
Renal lesion	10 (9.6)	8	8
Drowsiness	7 (6.7)	5	3
Nausea/vomiting	5 (4.8)	4	0
Diarrhoea	4 (3.8)	4	0
Hypoglycaemia	3 (2.9)	2	2
Change in respiratory pattern	2 (1.9)	1	0
Skin rash	2 (1.9)	2	0
Tachycardia	2 (1.9)	0	0
Hypertension	1 (1.0)	0	1

^aADEs, Adverse drug events.

The classes of medications most involved in the occurrence of the ADEs included drugs acting on the cardiovascular system (31.7%) and the nervous system (25.9%). Forty medications were suspected to be involved in ADEs identified in this study, of which the most frequent were morphine (13.5%), carvedilol (7.7%), tramadol (6.7%) and warfarin (6.7%).

Triggers were found in 64 (61.5%) of the 104 ADEs identified (Table 1). The most frequent were antiemetics (57.5%) and ‘abrupt medication stop’ (31.8%) (Table 2). Five of the triggers investigated (flumazenil, naloxone, glucose <50 mg dl⁻¹, positive result for Clostridium difficile and INR >6.00) were not identified in any of the days of study.

Table 2.

Frequency of triggers by category, frequency of triggers with plausible association with adverse drug events, sensitivity, specificity, positive predictive value, negative predictive value in 300 patients studied in a university hospital

Triggers	Absolute frequency (n) a	Plausible association (n) b	Se (95% CI)	Sp (95% CI)	PPV (95% CI)	NPV (95% CI)
Antiemetics	336	4	0.014 (0.000–0.036)	0.867 (0.828–0.902)	0.012 (0.000–0.027)	0.889 (0.855–0.919)
Abrupt medication discontinuation	185	35	0.118 (0.083–0.162)	0.939 (0.928–0.950)	0.189 (0.134–0.257)	0.899 (0.869–0.928)
Serum creatinine (2x basal level)	22	6	0.019 (0.000–0.071)	0.994 (0.986–0.998)	0.273 (0.000–0.720)	0.888 (0.859–0.919)
Diphenhydramine	15	1	0.003 (0.000–0.010)	0.994 (0.988–0.999)	0.067 (0.000–0.286)	0.887 (0.856–0.914)
Vitamin K	12	2	0.006 (0.000–0.016)	0.996 (0.991–1.000)	0.167 (0.000–0.667)	0.886 (0.856–0.914)
Excessive sedation, hypotension	11	1	0.003 (0.000–0.010)	0.996 (0.991–1.000)	0.091 (0.000–0.500)	0.886 (0.858–0.915)
PTT > 100 s	3	0	0.000 (0.000–0.000)	0.999 (0.997–1.000)	0.000 (0.000–0.000)	0.886 (0.854–0.914)

PTT, partial thromboplastin time; Se, sensitivity; Sp, specificity; PPV, positive predictive value; NPV, negative predictive value; CI, confidence interval.

^aNumber of days in which the trigger was present.

^bNumber of days in which the trigger was present and there was an ADE.

Triggers were found in 545 (19.6%) and ADEs were observed in 316 (11.4%) of the 2776 hospitalization days, with days in which more than one trigger or ADE was identified. However, a clinically plausible association of ADEs with a trigger was only found in 49 (1.8%) days. The individual PPV of the triggers ranged from 0.012 to 0.273, and the global PPV was 0.084. Frequencies of the presence of triggers, their plausible association with ADEs and measures of accuracy are shown in Table 2.

It was observed that 160 (53.3%) patients had at least one of the triggers and, among these, 57 (35.6%) had at least one ADE, but only 36 (22.5%) had a clinically plausible association. By contrast, among the 140 patients with no trigger, an ADE was observed in 17 (12.1%) (Figure 2).

Figure 2.

Flowchart characterizing the 300 studied patients regarding the presence of trigger and adverse drug event at a university hospital

Discussion

This study provides a relevant contribution to the assessment of accuracy of the triggers proposed by the IHI to identify ADEs. Our results showed that these triggers were not helpful as a predictor of ADEs. We assessed the occurrence of ADEs not only in patients with positive triggers but also in patients with negative triggers, considering that the IHI Trigger Tool would help predict the occurrence of ADEs. Thus, we expected that positive triggers would be more frequently associated with the occurrence of ADEs than negative triggers. However, most patients presenting at least one trigger did not have ADEs, and 38.5% of ADEs had no trigger during follow‐up. Moreover, among the 12 investigated triggers, only six had a plausible association with some ADE.

We could state that the goal of IHI Trigger Tool lies in facilitating the retrospective selection of records more likely to contain ADEs 2, 28. Its utility would focus on future planning of actions for hospital risk management based on this retrospective screening. However, previous studies have highlighted the poor performance of this procedure to search for ADEs 10, 22, 23, 24, 25. Our results showed that prospective use of triggers also did not show good ability to detect patients experiencing ADEs. In both procedures, we should take into account that most triggers are measured after the harm has already taken place. Thus, retrospective or prospective use of triggers would not impact real‐time changes in clinical practice to prevent inpatients from suffering the occurrence of ADEs, but they would help prescribers and other healthcare professionals recognize the main types of ADEs and high‐risk patients. As recommended by other authors 29, the adoption of combined strategies to detect ADEs would be more effective to identify patient safety flaws. For instance, rapid response teams have been referred to as a feasible approach to identify adverse events, with a detection of twice as many events as the hospital's safety reporting system 30. While using chart review methods, target samples should be enriched in ADEs of interest rather than selecting charts at random.

The individual evaluation of the triggers showed that naloxone, antiemetics, ‘abrupt medication stop’ and vitamin K triggers deserve recognition regarding their performance in identifying ADEs. As in other studies 17, 20, trigger naloxone, an opioid antagonist, was not identified at any time. Nevertheless, morphine was the medication that was most frequently associated with ADEs, given that the majority of cases were related to constipation. It is possible that the institution in which the study was performed, as it provides medical care to a significant number of haematologic‐oncologic patients, can benefit from the use of the trigger ‘administration of enema’, which has been shown by O'Leary et al. 5 to be effective in detecting constipation.

The trigger antiemetics, though more frequent, had low plausible association with ADEs and the lowest level of PPV due to the large number of false‐positive cases. This case is in accordance with the results from other studies 11, 19 and brings concerns as to the use of this trigger in monitoring the quality of medical care provided, given that they are widely used in hospital clinical practices to treat nausea/vomiting, which represent ADEs that are generally nonpreventable. The ‘abrupt medication stop’ was one of the most frequent and had the second best PPV, which runs in line with results identified in prior studies 12, 17, 24. Nonetheless, Franklin et al. 11, and Lau and Kirkwood 19 found contradictory results, which can be related to the different research methods used. The cost–benefit of the employment of this trigger in monitoring the occurrence of ADEs should be evaluated, as the verification of its presence is quite burdensome, complex and subjective, and demands search for complementary information, which is not always available. By contrast, despite a low PPV, vitamin K stood out, as it was able to identify a severe ADE related to the error in the process of warfarin use, given that its applicability has already been pointed out in other studies 5, 12. It is also important to emphasize that, among studies that use IHI triggers, varied results were found regarding the frequency, type and severity of the adverse events identified when these triggers were present 5, 11, 12, 17, 18, 19, 24, 28. Although the frequency of ADEs (24.7%) in this study is within the broad spectrum of estimates (7–27%) from similar studies 12, 17, 18, 19, 28, it is still of great concern to patient safety. The identification of 37.5% of all ADEs observed as preventable is highly relevant as an opportunity to improve the quality and safety of the use of medications.

Some factors may well explain the differences in these results, such as: the profile of patients involved in the study; subjectivity in the assessment of ADEs, which can be confused with the natural course of baseline disease and/or comorbidities; the occurrence of ADEs with no proposed IHI triggers; the different adaptations and/or adjustments in the IHI triggers by the researchers; and the intrinsic problems of these triggers. Nevertheless, regardless of differences, the findings of this study suggest weaknesses in the characteristics of these triggers, which results in low sensitivity and a significant number of false‐positive cases.

The strengths of this study include its prospective character, which minimized the loss of information due to the absence or low quality of the health professional notes on patients' medical records; the use of standardized clinical alterations as a reference in the search for ADEs with and without triggers; and the definition of the number of days that the patient was hospitalized as unit of measure, allowing the sensitivity, specificity and NPV to be calculated for each trigger and not only the proportion of ADE identified by the trigger or the PPV, as shown by other authors 11, 12, 17, 20. Another strength of this study was the minimization of bias, given that divergences in the specialists' reports were resolved by consensus through the clinical assessment.

There are some limitations that should be addressed. This is an observational study and drawing conclusions is difficult due to confounders and bias. In addition, this was a single‐centre study and it was not possible to ensure the detection of all of the ADEs, despite standardized procedures and all the efforts to identify as many ADEs as possible.

Conclusion

The IHI Trigger Tool did not show good accuracy in detecting ADEs when applied prospectively in this study. The adoption of combined strategies and alternative approaches to improve trigger usefulness could enhance effectiveness for patient safety. Further discussion on the IHI Global Trigger Tool might contribute to improve its usefulness in clinical practice.

Competing Interests

There are no competing interests to declare.

We acknowledge Programa de Pós‐graduação em Ciências da Saúde: Infectologia e Medicina Tropical of Universidade Federal de Minas Gerais, Pró‐Reitoria de Pesquisa da Universidade Federal de Minas Gerais, Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), Instituto de Avaliação de Tecnologias em Saúde (IATS) and Fundação de Amparo à Pesquisa de Minas Gerais (FAPEMIG) for their support. We would like to thank all patients who participated in the study and the valuable collaboration of the volunteer research team.

Contributors

All authors had access to the data. M.D.G.S., M.A.P.M., L.G.V., J.A.Q.O. and A.L.P.R. participated in the conception and design of the study; M.D.G.S., L.G.P., J.A.Q.O., R.R.M., J.L.P.S. and A.L.P.R. participated in the data analysis and interpretation of results; M.D.G.S., M.A.P.M., L.G.V. and A.L.P.R. participated in drafting of the article. All authors reviewed and approved the final version of the manuscript.

Silva, M. D. G. , Martins, M. A. P. , Viana, L. G. , Passaglia, L. G. , de Menezes, R. R. , Oliveira, J. A. Q. , da Silva, J. L. P. , and Ribeiro, A. L. P. (2018) Evaluation of accuracy of IHI Trigger Tool in identifying adverse drug events: a prospective observational study. Br J Clin Pharmacol, 84: 2252–2259. 10.1111/bcp.13665.