Performance of trigger tools in identifying adverse drug events in emergency department patients: a validation study

Abstract

Aims

Trigger tools are retrospective surveillance methods that can be used to identify adverse drug events (ADEs), unintended and harmful effects of medications, in medical records. Trigger tools are used in quality improvement, public health surveillance and research activities. The objective of the study was to evaluate the performance of trigger tools in identifying ADEs.

Methods

This study was a sub‐study of a prospective cohort study which enrolled adults presenting to one tertiary care emergency department. Clinical pharmacists evaluated patients for ADEs at the point‐of‐care. Twelve months after the prospective study's completion, the patients' medical records were reviewed using eight different trigger tools. ADEs identified using each trigger tool were compared with events identified at the point‐of‐care. The primary outcome was the sensitivity of each trigger tool for ADEs.

Results

Among 1151 patients, 152 (13.2%, 95% confidence intervals (CI) 11.4, 15.3%) were diagnosed with one or more ADEs at the point‐of‐care. The sensitivity of the trigger tools for detecting ADEs ranged from 2.6% (95% CI 0.7, 6.6%) to 15.8% (95% CI 10.6, 22.8%). Their specificity varied from 99.3% (95% CI 98.6, 99.7) to 100% (95% CI 99.6, 100%).

Conclusion

The trigger tools examined had poor sensitivity for identifying ADEs in emergency department patients, when applied manually and in retrospect. Reliance on these methods to detect ADEs for quality improvement, surveillance, and research activities is likely to underestimate their occurrence, and may lead to biased estimates.

What is Already Known about this Subject

• Trigger methods are widely used to identify adverse drug events and adverse drug reactions.

• Trigger methods have not been compared with prospective identification of adverse drug events to understand their completeness and accuracy

What this Study Adds

• The current available trigger tools have poor sensitivity for identifying adverse drug events in emergency department patients.

• More robust methods for the detection and monitoring of adverse drug events are needed.

Introduction

Adverse drug events (ADEs), unintended and harmful events related to medication use, are a common source of patient injury within healthcare 1, 2. They are a leading cause of ambulatory and emergency department visits, unplanned hospital admissions and deaths 3, 4, 5, 6. Patients who present to emergency departments with ADEs spend more days in hospital and incur nearly double the healthcare costs over 6 months compared with patients presenting without medication‐related morbidity 7.

The majority of ADEs are, in retrospect, deemed preventable 5, 8. However, identifying and implementing practice changes that result in sustained and measurable reductions of ADEs has proven challenging: Efforts directed at reducing medication errors—failures in the process of medication delivery, have not optimized the safety of healthcare, in part because many errors are intercepted before reaching the patient 1, 9, 10. In contrast, medications administered without error may result in harm due to toxic reactions or dosing problems 11. It is possible that greater improvements in patient safety may be achieved by changing the focus of preventative efforts from error to harm reduction 1.

Until recently, a major barrier to developing evidence‐based harm reduction strategies was the inability of healthcare institutions to identify and monitor ADEs over time in an efficient and consistent manner, so that data could inform the development and evaluation of quality improvement strategies 1. Conventional approaches to identifying ADEs included voluntary reporting, chart reviews and mining administrative data using patient safety indicators and discharge diagnoses. These methods are resource‐intensive, provide inconsistent results over time, or significantly under report patient safety events, prompting continued investigation into more practical and efficient methods to identify adverse outcomes 12, 13, 14, 15, 16.

An alternate method for identifying ADEs was described in 1974 and has since been adapted to various healthcare settings 17. Trigger tools use a two step chart review methodology. In a first step, trained reviewers screen a random selection of patient records and flag those containing a trigger or clue, that may consist of an individual word, phrase, order or laboratory value 1, 17, 18. In a second step, a trained nurse or physician reviews the flagged records to determine whether an adverse event occurred. Today, trigger methods are in widespread use in quality improvement and increasingly in pharmacosurveillance and research activities 17, 19. For example, the National Electronic Injury Surveillance System‐Cooperative Adverse Drug Event Surveillance Project (NEISS‐CADES) uses trigger methods to describe and monitor outpatient ADEs treated in USA emergency departments 19, 20.

Despite their widespread application, trigger methods have not been compared to prospective identification of ADEs to understand their completeness and accuracy. Therefore, our aim was to evaluate the sensitivity and specificity of published trigger tools in identifying adverse drugs events in comparison with events identified in a prospective sample of adult emergency department patients.

Methods

Study setting and design

This study was an a priori planned sub‐study of a prospective observational study to derive clinical decision rules to identify emergency department patients at high‐risk of ADEs 21. This study was conducted in the one Canadian tertiary care hospital, Vancouver General Hospital, with an annual census of 84 000 patients. The University of British Columbia Clinical Research Ethics Board (H10‐01632) reviewed and approved the study protocol and waived the need for informed consent.

Enrolment and data collection

The patient enrolment and data collection procedures for the parent study have been described in detail elsewhere 21. Briefly, patients presenting to the emergency department between July 1 2008 and January 24 2009 were eligible for enrolment. Within each data collection shift, clinical pharmacists applied a systematic patient selection algorithm to ensure a representative sample of patients (see Supplemental Figure, Appendix S1) 5. All patients 19 years of age or older who reported using at least one prescription or over the counter medication in the 2 weeks prior to presentation and spoke English or had a translator available were deemed eligible. Patients were excluded if they were transferred directly to an admitting service, presented for a scheduled revisit (e.g. intravenous antibiotics), left against medical advice or before the pharmacist assessment was complete, exhibited violent behaviour or had previously been enrolled. After prospective data collection was complete, we excluded patients if their medical record could not be located or if data on inclusion and exclusion criteria were missing.

Three clinical pharmacists collected demographic and clinical information from patients at the point‐of‐care, and reviewed the medical record in the emergency department. They verified medication histories using PharmaNet, a province wide prescription filling database. All admitted patients were followed until hospital discharge and consenting patients were contacted after discharge by telephone when necessary to determine whether the patient met the primary or secondary outcome. Pharmacists evaluated whether the patient's visit was due to an ADE arising prior to the patient's presentation to the Emergency Department, using three standardized causality algorithms 22, 23, 24. Inter‐rater reliability of this assessment algorithm was evaluated to ensure reliability and was previously reported (kappa 0.75, 95% CI 0.52, 0.98) 21. The pharmacists then interviewed treating physicians using a standardized questionnaire to determine whether the physician believed the patient had suffered an ADE. An independent committee adjudicated all discordant or uncertain cases to establish the final diagnosis 21.

Assessment of trigger methods

Research assistants (AK and CM), who had not been involved with the primary study, manually reviewed the medical records of all enrolled patients 6–12 months after the date of their emergency department visit and after all admitted patients were discharged. This included a review of both the electronic records as well as the paper chart of included patients. We applied the two step chart review methodology described by the Institute for Healthcare Improvement 17, 19. First, we piloted a data collection form containing the individual triggers of eight published trigger tools (Appendix S2) 1, 17, 18, 25, 26, 27, 28, omitting triggers for non‐medication‐related adverse events, triggers not applicable to our patient population and the triggers for medications that had not been prescribed in our cohort of patients. The final form contained 63 individual triggers and the NEISS‐CADES algorithm which contains the triggers allergic reaction, adverse effect, side effect, secondary to, ingestion, toxicity, medication error, angioedema, anaphylaxis, rash, bleeding and hypoglycaemia 19. We trained two research assistants in the use of the data collection form (AK and CM) and randomly selected 21 charts for review to measure the inter‐rater agreement between their assessments. The research assistants reviewed the medical records of all patients to identify individual triggers and determine whether the NEISS‐CADES criteria were met. Records for which one or more triggers were flagged as having been met proceeded to the second stage review. In the second review stage, a trained nurse specializing in patient safety and quality (CP) who applies trigger tools in her work reviewed the flagged charts for the presence of an ADE. The first and second stage reviewers were blinded to the ADE determination from the primary study. A trigger tool or the NEISS‐CADES algorithm were considered to have flagged an ADE,,if any of its component triggers were present during the first review step and the second review step identified an ADE (see Appendix S4 for a study flow chart).

Outcome measures

The ADE outcomes identified in the primary study were considered the criterion standard for the sub‐study. ADEs were defined as ‘utoward and unintended symptoms, signs or abnormal laboratory values arising from the appropriate or inappropriate use of prescription or over the counter medications’ 13, 21. Pharmacists classified ADEs into seven categories: (i) Adverse drug reactions, defined as ‘noxious and/or unintended responses to medication which occurred despite appropriate drug dosage for prophylaxis, diagnosis or therapy of the indicating medical condition’ 29, events caused by (ii) non‐adherence, (iii) subtherapeutic dosing, (iv) supratherapeutic dosing, (v) receiving the wrong drug, (vi) suffering from an untreated indication or (vii) using a drug without a treatment indication 11, 30. The pharmacists individually assessed causality at the point‐of‐care using previously adapted standardized algorithms, ruling out ADEs for any events in which physicians identified alternate diagnoses 22, 23, 24, and assessed preventability using criteria developed by Hallas et al. 31. We rated the severity of the events as severe if the event caused death or required hospital admission, moderate if the event required a change in medical management and mild if the event required no change in management 5. ADEs arising from the administration of medications in the Emergency Department or in hospital were not included in the analysis.

Statistical analysis

We used descriptive statistics to summarize the baseline characteristics of the patient population.

We reported the inter‐rater reliability of collecting data on individual triggers by using the prevalence adjusted and biased adjusted kappa (PABAK) score 32. We estimated the sensitivity of the trigger tools by dividing the number of records with one or more ADEs identified using the trigger tool by the total number of records with ADEs identified during the prospective study, multiplied by 100 33. We examined the specificity of the trigger tools by dividing the number of records that screened negative using the trigger tool without ADEs by the total number of charts without ADEs, multiplied by 100. We decided a priori to conduct a sensitivity analysis and calculate the sensitivities and specificities of the tools for moderate and severe adverse drug reactions. We reported 95% CIs of the sensitivity and specificity.

Results

Adverse drug events identified at the point‐of‐care

Among 1566 patients who were approached for enrolment, 1160 were included in the prospective study (Figure 1) 21. The charts of eight patients could not be located at the time of data collection for the trigger tool evaluation and two additional charts were unavailable at the time of the NEISS‐CADES evaluation. One additional record had insufficient data to determine the inclusion and exclusion criteria and was excluded.

Figure 1.

Flow diagram of included patients

The baseline characteristics of the study population are listed in Table 1. Among 1151 patients, 152 (13.2%, 95% CI 11.4, 15.3%) were diagnosed with 164 ADEs at the point‐of‐care (Table 2). Clinical pharmacists classified 34.8% (57/164) of these events as adverse drug reactions, 66.5% (109/164) of them as being directly related to the patient's chief complaint and 69.5% (114/164) as preventable. Most events (81.1%, 133/164) were rated as moderate in severity and warfarin, paracetamol (acetaminophen) with codeine, aspirin, phenytoin, olanzapine and hydrochlorothiazide were the most commonly implicated medications.

Table 1.

Characteristics of included patients

Characteristics	All patients (n = 1151)	Patients without adverse drug events (n = 999)	Patients with ≥1 adverse drug event (n = 152)
Age (years) mean (SD)	51.6 (20.9)	50.9 (20.6)	56.1 (21.7)
Gender, n (%)
Female	610 (53.0%)	538 (53.9%)	72 (47.4%)
Male	541 (47.0%)	461 (46.1%)	80 (52.6%)
Arrived from, n (%)
Home	1048 (91.1%)	917 (91.9%)	131 (86.2%)
Homeless or shelter	29 (2.5%)	19 (1.9%)	10 (6.6%)
Nursing home	49 (4.3%)	40 (4.0%)	9 (5.9%)
Other	24 (2.1%)	22 (2.2%)	2 (1.3%)
English language, n (%)
Speak English	1112 (96.6%)	968 (96.9%)	144 (94.7%)
Translator available	39 (3.4%)	31 (3.1%)	8 (5.3%)
Emergency department treatment location, n (%)
Acute care	509 (44.3%)	406 (40.8%)	103 (67.8%)
Minor care	639 (55.7%)	590 (59.2%)	49 (32.2%)
Canadian triage acuity score, n (%)
1	5 (0.43%)	3 (0.30%)	2 (1.3%)
2	174 (15.1%)	150 (15.0%)	24 (15.8%)
3	507 (44.0%)	423 (42.3%)	84 (55.3%)
4	425 (36.9%)	387 (38.7%)	38 (25.0%)
5	40 (3.47%)	36 (3.6%)	4 (2.6%)
Most common chief complaints, n (%)
Abdominal pain	103 (8.6%)	96 (9.6%)	7 (4.6%)
Chest pain	77 (6.7%)	72 (7.2%)	5 (3.3%)
Shortness of breath	68 (5.9%)	55 (5.5%)	13 (8.6%)
Lower extremity pain	57 (5.0%)	54 (5.4%)	3 (2.0%)
Back pain	51 (4.4%)	47 (4.7%)	4 (2.6%)
Number of prescription medications, median (IQR)	2 (1,5)	2 (1,5)	4 (2,7)
CAM use, n (%)	137 (12.0%)	116 (11.7%)	21 (13.8%)
Over the counter medication use, n (%)	853 (74.6%)	758 (76.5%)	95 (62.5%)
Number of prescribing physicians, median (IQR)	1 (1, 2)	1 (1, 2)	2 (1, 2)
Number of comorbid conditions, median(IQR)	2 (1, 3)	1 (0, 3)	2 (1, 4)
Followed by a general practitioner, n (%)	993 (87.0%)	859 (86.9%)	134 (88.2%)
Disposition from emergency department, n (%)
Home	936 (81.4%)	829 (83.1%)	107 (70.4%)
Admitted	210 (18.3%)	167 (16.7%)	43 (28.3%)
Transferred	3 (0.26%)	1 (0.1%)	2 (1.3%)
Died in ED, n (%)	1 (0.09%)	1 (0.1%)	0 (0%)

CAM, complimentary or alternative medications; ED emergency department

Table 2.

Characteristics of 164 adverse drug events identified at the point‐of‐care in 152 patients

Characteristics	Adverse drug events (n = 164)
Severity, n (%)
Severe	21 (12.8%)
Moderate	133 (81.1%)
Mild	10 (6.1%)
Preventability, n (%)
Preventable	114 (69.5%)
Non‐preventable	50 (30.5%)
Relationship to chief complaint, n (%)
Chief complaint‐related	109 (66.5%)
Incidentally found	55 (33.5%)
Classification, n (%)
Adverse drug reactions	57 (34.8%)
Non‐compliance or drug withdrawal	36 (22.0%)
Untreated indication	26 (15.9%)
Subtherapeutic dose	21 (12.8%)
Wrong drug	10 (6.1%)
Supratherapeutic dose	9 (5.5%)
Drug without indication	4 (2.4%)
Drug interactions	1 (0.6%)
Most common culprit medications, n (%)
Warfarin	14 (8.5%)
Paracetamol with codeine	11 (6.7%)
Aspirin	9 (5.5%)
Phenytoin	7 (4.3%)
Olanzapine	6 (3.7%)
Hydrochlorothiazide	6 (3.7%)
Cephalexin	5 (3.0%)
Hydromorphone	5 (3.0%)
Glyburide	4 (2.4%)
Morphine	4 (2.4%)

Evaluation of trigger tools

The kappa scores for the inter‐rater agreement between the first stage reviewers with regard to the presence or absence of individual triggers ranged from 0.81 (95% CI 0.55, 1) to 1 (95% CI 1, 1). Appendix S3 (see supplemental table, Appendix S3) describes the number of records that were flagged as positive based on the individual triggers. The most commonly flagged triggers were >6 h in the emergency department, unplanned hospitalization or transfer to a higher level of care, rising serum creatinine, rising blood urea nitrogen level or creatinine greater than twice baseline and the use of an antiemetic.

Table 3 presents the diagnostic test characteristics of the trigger tools we evaluated. The sensitivities of the trigger tools for records containing one or more ADEs ranged from 2.6% (95% CI 0.7, 6.6%) to 12.5% (95% CI 7.9, 19.1%). For records with at least one moderate or severe adverse drug reaction their sensitivities ranged from 4.1% (95% CI 0.5, 14.0%) to 22.5% (95% CI 12.2, 37.0%). Using the flags of the individual trigger tools as a basis for conducting the second stage review, between 140 (87.2%) and 157 (95.7%) of the 164 ADEs were missed. These methods also missed between 101 (88.6%) and 113 (99.1%) of the 114 ADEs classified as preventable. The specificities of the tools ranged from 99.3% (95% CI 98.6, 99.7%) to 100% (95% CI 99.6, 100%). Their positive predictive values were consistent between tools and ranged between 57.1% (95% CI 18.4, 90.1%) and 100% (95% CI 63.1, 100%). Their negative predictive values varied between 87.1% (95% CI 84.9, 88.9%) and 88.2% (95% CI 86.1, 90.0%).

Table 3.

Diagnostic performance characteristics of the trigger methods for adverse drug events

Trigger tool	Target population	Patients with >1 adverse drug event
		Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)
GlobalGriffin & Resar 17	Hospitalized	12.5% (7.9, 19.1)	99.3% (98.6, 99.7)	73.1% (52.1, 88.4)	88.2% (86.1, 90.0)
GlobalResar et al. 1	Hospitalized	10.5% (6.3, 16.8)	99.5% (98.8, 99.8)	76.2% (52.8, 91.8)	88.0% (85.9, 89.8)
Adverse drug event Singh et al. 26	Ambulatory	11.2% (6.8, 17.6)	99.7% (99.1, 99.9)	85.0% (62.1, 96.8)	88.1% (86.0, 90.0)
Adverse drug event Rozich 18	Hospitalized	12.5% (7.9, 19.1)	99.6% (99.0, 99.9)	82.6% (61.2, 95.1)	88.2% (86.1, 90.0)
Adverse drug event Hug 28	Hospitalized	12.5% (7.9‐19.1)	99.6% (99.0, 99.9)	82.6% (61.2, 95.1)	88.2% (86.1, 90.0)
Adverse drug reaction‐Cantor et al. 25	Ambulatory	5.3% (2.3‐10.1)	100% (99.6, 100)	100% (63.1, 100)	87.4% (85.3, 89.2)
Electronic‐Wolff 27	ED	2.6% (0.7, 6.6)	99.7% (99.1‐99.9)	57.1% (18.4, 90.1)	87.1% (84.9, 88.9)
NEISS‐CADES 3	ED	15.8 (10.6‐22.8)	99.5% (98.8, 99.8)	82.8% (64.2, 94.2)	88.6% (86.6, 90.4)

ADE adverse drug event; ADR adverse drug reaction; ED Emergency Department; NEISS‐CADES National Electronic Injury Surveillance System‐Cooperative Adverse Drug Events Surveillance System; PPV, positive predictive value

Evaluation of the NEISS‐CADES algorithm

The kappa scores for the inter‐rater agreement between the first stage reviewers with regard to whether a NEISS‐CADES trigger was present or absent was 1 (95% CI 1, 1). The NEISS‐CADES algorithm had a sensitivity of 15.8% (95% CI 10.6, 22.8%) and a specificity of 99.5% (95% CI 98.8, 99.8%) for detecting records with at least one ADE and a sensitivity of 38.8% (95% CI 25.5, 53.8%) and specificity of 99.1% (95% CI 98.3, 99.6%) for moderate or severe adverse drug reactions (Table 4). The NEISS‐CADES algorithm missed 137 (83.5%) of the 164 prospectively identified ADEs and 100 (87.7%) of preventable events. The algorithms' positive predictive value was 82.8% (95% CI 64.2, 94.2%) and its negative predictive value 88.6% (95% CI 86.6, 90.4%)

Table 4.

Number and proportion of missed adverse drug events, by event characteristics and trigger method used

Trigger method used for screening	Adverse drug events (n = 164)	Moderate adverse drug events (n = 133)	Severe adverse drug events (n = 21)	Preventable adverse drug events (n = 114)	Chief‐complaint related adverse drug events (n = 109)
Griffin & Resar 17 , n (%)	140 (87.4%)	115 (85.8%)	15 (71.4%)	101 (88.6%)	93 (85.3%)
Singh et al. 26, n (%)	143 (87.2%)	118 (88.7%)	15 (71.4%)	103 (90.4%)	94 (86.2%)
Rozich et al. 18 , n (%)	140 (87.4%)	115 (85.8%)	15 (71.4%)	101 (88.6%)	93 (85.3%)
Hug et al. 28 , n (%)	140 (87.4%)	115 (85.8%)	15 (71.4%)	101 (88.6%)	93 (85.3%)
Cantor et al. 25 , n (%)	154 (93.9%)	123 (92.5%)	21 (100%)	108 (94.7%)	101 (92.7%)
Resar et al. 1 , n (%)	144 (87.8%)	119 (89.5%)	15 (71.4%)	104 (91.2%)	95 (87.2%)
Wolff & Bourke 27 , n (%)	157 (95.7%)	128 (96.2%)	19 (90.5%)	113 (99.1%)	105 (96.3%)
NEISS‐CADES 19 , n (%)	137 (83.5%)	113 (85.0%)	15 (71.4%)	100 (87.7%)	90 (82.6%)

NEISS‐CADES National Electronic Injury Surveillance System‐ Cooperative Adverse Drug Event Surveillance

Discussion

Our objective was to evaluate the classification performance of eight published trigger tools for ADE identification in a sample of adult emergency department patients. To our knowledge, our study is the first to validate trigger tools by comparing adverse events identified by the tools to events identified at the point‐of‐care by clinical care providers. Our findings indicate that the sensitivities of the trigger tools and the NEISS‐CADES algorithm were uniformly low, ranging from 2.6% to 15.8% compared with the prospective standard. All trigger tools missed the majority of ADEs deemed preventable. In contrast, their specificities for ADEs were high, indicating few false positive cases.

Over the past 15 years, trigger tools have been widely adopted by healthcare institutions internationally to identify and monitor adverse events related to medical care 17. Their implementation followed recognition of their feasibility and ease of use and was driven by a pressing need to find and apply new methods to capture adverse event data for quality improvement 17, 34. As ‘big data’ from comprehensive electronic health records became available for data mining, trigger tools were also applied as a means of identifying adverse outcomes within large databases for surveillance and research 35. However, the rapid and widespread uptake of trigger methods preceded their rigorous validation, which is intended to ensure that the intended outcomes are identified consistently, completely and accurately 36. The practice of generating and analyzing trigger‐derived data in advance of robust validation is problematic as it limits our ability to understand their inherent limitations.

The assessment of any trigger tool's sensitivity, its ‘miss’ rate, requires a comparison of outcomes identified using the tool with events identified by an independent robust criterion standard for all patients within a study sample 36. Previous attempts at validating trigger tools have used retrospective methods, voluntary reporting and administrative data to identify adverse event outcomes 16, 33, 35, 37, 38, 39, 40, 41. This is problematic, as adverse event outcomes are poorly identifiable within medical records due to incomplete documentation, underreported within existing adverse event reporting platforms and poorly identifiable within administrative data 12, 13, 14, 15, 16, 42. Therefore, none of these methods is a robust standard for comparison and, as a result, led to overestimations of sensitivity 36. In contrast to previous validation attempts, our study is the first to compare the performance of trigger tools to independent, prospectively established outcomes.

Five previous studies reported the sensitivity of the same trigger tools we assessed and reported sensitivities ranging from 33–94.9 % 16, 33, 39, 41, 43. Two focused on adverse event identification in adults 16, 33 and paediatrics 41, and two on ADE identification in adults 39, 43. Previous studies are likely to have overestimated the sensitivity of the trigger tools due to incomplete outcome determinations and, in some cases, outcome determinations not being independent of the trigger tool application. This occurred when the same trigger methods were applied to pre‐screen charts or find the outcomes of interest 16, 33, 39. All previous validation studies using retrospective methods have missed all undocumented outcomes 16, 33, 39, 41, 43. In one study, the same research pharmacist completed the identification of ADE outcomes prior to applying the trigger tool and, therefore, would have been biased towards finding a higher number of events 43. Another used the outcomes identified by the trigger method being evaluated, combined with outcomes identified by voluntary reporting and application of indicators in administrative data 16. For these reasons, none of the previous studies reporting sensitivity meet the methodological standards required to provide unbiased estimates of sensitivity.

Most previous attempts at validating trigger tools have focused on their positive predictive value, the proportion of flagged or trigger positive records containing the outcome of interest 33, 35, 37, 39, 41, 43, 44, 45, 46. This metric is useful in providing an estimate of the yield within the flagged records and enables an estimate of the excess workload generated by false positive flags 35. Since trigger tools are used as a screening method and are meant to be applied to large data sets, trigger tools with high positive predictive values are key in minimizing the false positive rate, as long as the sensitivity of these tools is maintained to a reasonable degree. In our study, the positive predictive value of the tools varied between 57.1–100%, consistent with ranges reported in previous studies. A few studies have reported lower positive predictive values when evaluating a subset of laboratory based triggers, signalling adverse events in only 15% of cases 45, or adaptations of existing tools, signalling events in 4% and 17% of cases 43, 46, 47. The latter studies call into question the signal‐to‐noise ratio and any efficiency gained by employing trigger tool methodology to prescreen records.

In our study, we found high inter‐rater reliability of individual triggers compared with the level of agreement reported by others 33, 40, 41, 44, 46, 48. This may be due to multiple factors. We trained research assistants extensively, attempted to discuss and minimize the degree of subjective interpretation required to apply the triggers in advance of piloting and asked research assistants to review the emergency department record of only one hospital, minimizing variation and shortening the length of time required to complete the review. As a result, our reviewers were able to evaluate the complete emergency department record within the set time limit suggested by the Institute for Healthcare Improvement 48. Finally, the sample from which we report inter‐rater agreement is small, and thus our estimates are more uncertain compared with other studies.

In our team's work in deriving clinical decision rules to predict patients at high risk for ADEs, drug and disease specific variables were not useful, as was seen in this study 21. This was likely because ADEs tend to be heterogeneous in nature. The trigger tools developed so far contain a large number of disease and drug‐specific manifestations, laboratory values or antidotes, which in our analysis were not discriminatory. In our work on predicting ADEs, our team found that drug and disease specific variables predicted too small a subset of events to be useful, e.g. a trigger using an INR cutoff was only useful for events related to warfarin, which constituted an important but small minority of events. However, more generic variables such as ‘medication change within 28 days’ could predict multiple types of ADEs, because harm generally occurred shortly after a drug had been initiated. We believe that more sensitive trigger tools might be developed by using clinical decision rule methodology in which clinical judgment is initially used to define lists of candidate predictor variables. Subsequently, a prospective observational study is conducted in which data on candidate predictor variables and the study outcome are collected. Then, statistical associations between the candidate predictor variables and outcome are carefully explored in a systematic way to parse out potentially useful variables and combine them to a decision algorithm 36.

To our knowledge, our study represents the first external validation of the NEISS‐CADES algorithm, which is presently in use to monitor and describe the public health burden of outpatient ADEs treated in USA emergency departments 49. We estimated its sensitivity at 15.8%, indicating that 84.2% of ADEs may be missed in this important public health surveillance system. When a narrower case definition was applied, focusing on moderate and severe adverse drug reactions only, its sensitivity was increased to 38.8%. However, NEISS‐CADES missed the majority of preventable ADEs, indicating that opportunities for prevention and system‐wide change may be missed if we rely solely on this case finding method 27.

Our findings have important implications for quality improvement initiatives, public health surveillance and research efforts that use trigger‐derived data as a means of developing and evaluating strategies to improve clinical care. In 2010, Landrigan et al. published the results of a longitudinal study evaluating whether investments in quality improvement strategies over 5 years led to a measurable improvement in adverse event rates in a sample of hospitals in North Carolina 50. The study found little evidence of improvement over time and concluded that harm remained common. The results of our study may offer additional insight into these findings. In our study, the majority of preventable ADEs were not identifiable using the trigger methods we evaluated. Thus, it is possible that the poor sensitivity of the global trigger tool for preventable events may have limited the authors' ability to detect improvement. Alternatively, if the healthcare institutions used trigger‐derived data to prioritize and develop quality improvement strategies, it is plausible that few preventable adverse outcomes were in fact targeted, and that the programs did in fact have less impact on safety than anticipated.

Limitations

Our study is not without limitations. Our study was conducted in one academic centre and,therefore, our results may not be generalizable to other types of institutions. We evaluated all tools against ADE cases that presented to emergency departments, even though some tools were specifically developed for other healthcare settings. It is possible that the sensitivity of the tools may be greater for other types of hospitals and care settings. However, two of the tools we evaluated were specifically designed for emergency departments and one had the highest while the other the lowest sensitivity for ADEs. Among the tools we evaluated, three were designed for adverse events while five had been developed to identify adverse events to medications. This was likely reflected in the lower positive predictive value of those tools not specifically designed for ADEs.

Differences between the definitions of ADE used in developing the trigger tools and that used in our study may have contributed to the low sensitivity we found. For this reason we completed a sensitivity analysis, in which we used the narrowest definition for ADEs possible, adverse drug reactions, and excluded all mild events. Even in this analysis the sensitivity of the trigger tools remained low.

In conclusion our results suggest that eight commonly used trigger tools, including the NEISS‐CADES algorithm currently used for public health surveillance of outpatient ADEs in the US, suffer from poor sensitivity. The majority of missed ADEs were preventable, suggesting the need for continued development of more robust methods to detecting and monitor ADEs. Our results highlight the importance of validating retrospective ADE case finding methods against a robust prospective standard, so that refinement may be attempted prior to their widespread implementation.

Competing Interests

All authors completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf and declare no support from any organization for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.

Supporting information

Appendix S1 Patient enrolment algorithm. We used a systematic selection algorithm to ensure enrolment of a representative sample during prospective data collection [21].

Appendix S2 Trigger tools are composed of individual triggers used to screen medical records. We evaluated all individual triggers that could plausibly be used to identify adverse drug events in adults and omitted those triggers intended to identify events unrelated to medication use (e.g. intensive care unit acquired pneumonia). Triggers consisting of serum drug concentrations or antidotes to medications that are not used in the outpatient setting were excluded (e.g. protamine). Orders to check the levels of the following medications are not common in the emergency department and were omitted from this study: ciclosporin [26], phenobarbital [26], procainamide [26], quinidine [26], theophylline [18, 26], lidocaine [18], gentamicin [18], tobramycin [18] and amikacin [18]. Letters indicating complaints from family members could not be obtained for our chart review and were therefore excluded from the study [1].

Appendix S3 Presence of 64 triggers in the records of patients, by adverse drug event status.

Appendix S4 Study flow diagram.

Supporting info item

Supporting info item

Supporting info item

Supporting info item

Karpov, A. , Parcero, C. , Mok, C. P. Y. , Panditha, C. , Yu, E. , Dempster, L. , and Hohl, C. M. (2016) Performance of trigger tools in identifying adverse drug events in emergency department patients: a validation study. Br J Clin Pharmacol, 82: 1048–1057. doi: 10.1111/bcp.13032.