Prospective Validation of Clinical Criteria to Identify Emergency Department Patients at High Risk for Adverse Drug Events

Abstract

Objectives

Adverse drug events (ADEs) cause or contribute to one in nine emergency department (ED) presentations in North America and are often misdiagnosed. EDs have insufficient clinical pharmacists to complete medication reviews in all incoming patients, even though pharmacist‐led medications reviews have been associated with improved health outcomes. Our objective was to validate clinical decision rules to identify patients presenting with ADEs so they could be prioritized for pharmacist‐led medication review.

Methods

This multicenter, prospective study was conducted in two tertiary and one community hospital in Canada. We enrolled 1,529 adults presenting to EDs over 12 months. We applied two clinical decision rules and collected baseline variables prior to assessments by clinical pharmacists and physicians. We compared the physician and pharmacist diagnoses with the decision rule results. The primary outcome was a moderate or severe ADE, defined as an unintended and harmful event related to medication use or misuse, which required a change in medical therapy, diagnostic testing, consultation, or admission. An independent committee adjudicated uncertain and discordant cases. We calculated the diagnostic accuracy of both rules.

Results

Among 1,529 patients, 184 (12.0%) were diagnosed with an ADE. Rule 1 contained the variables 1) having a preexisting medical condition or having taken antibiotics within 1 week and 2) age > 80 years or having a medication change within 28 days. They had a sensitivity of 91.3% (95% confidence interval [CI] = 86.3%–95.0%) and a specificity of 37.9% (95% CI = 35.3%–40.6%) for ADEs.

Conclusions

Our study validated clinical decision rules that can be applied by clinical pharmacists to limit the number of patients requiring medication review, while identifying the majority of patients presenting with clinically significant ADEs.

CME Information: Validation of the Pediatric NEXUS II Head Computed Tomography Decision Instrument for Selective Imaging of Pediatric Patients with Blunt Head Trauma

CME Editor: Corey Heitz, MD

Authors: M. Hohl, MD, FRCP(C), Katherin Badke, Amy Zhao, Maeve E. Wickham, MSc, Stephanie A. Woo, Marco L.A. Sivilotti, MD, FRCP(C), and Jeffrey J. Perry, MD, CCPF(EM)

If you wish to receive credit for this activity, please refer to the website: http://www.wileyhealthlearning.com/aem

Educational Objectives

After reading the article, participants should be able to discuss the criteria identifying ED patients at high risk for adverse drug reactions.

Activity Disclosures

No commercial support has been accepted related to the development or publication of this activity.

No conflicts of interest or financial relationships relevant to this article were reported.

This activity underwent peer review in line with standards of editorial integrity and publication ethics. Conflicts of interest have been identified and resolved in accordance with John Wiley and Sons, Inc.'s Policy on Activity Disclosure and Conflict of Interest.

Accreditation

John Wiley and Sons, Inc. is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

John Wiley and Sons, Inc. designates this journal‐based CME activity for a maximum of 1.0 AMA PRA Category 1 Credit^™. Physicians should only claim credit commensurate with the extent of their participation in the activity.

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within 1 hour. To successfully earn credit, participants must complete the activity during the valid credit period, which is up to two years from initial publication. Additionally, up to 3 attempts and a score of 70% or better is needed to pass the post test.

Preventable adverse events related to medical care are a common cause of emergency department (ED) visits and hospitalizations and a leading cause of dealth.1, 2 Of deaths attributable to medical care, those related to medications are the most common.3, 4 Prospective studies indicate that adverse drug events (ADEs) cause or contribute to one in nine ED visits, indicating a large burden of disease.5, 6 With the population aging and medication use expanding, these numbers are expected to continue to rise.

Physicians working in EDs and on inpatient units do not recognize a medication‐related cause in 20% to 50% of ADEs.7, 8, 9, 10 Lack of timely recognition, correction, and communication of ADEs may prolong harmful medication use and contributes to the excess morbidity, health services use, and costs associated with these events.9, 11, 12, 13, 14, 15 In contrast to physicians, clinical pharmacists whose training and professional practice focus on medication management are more likely to recognize medication‐related presentations.16 Pharmacist‐led medication review in high‐risk patients in the ED has been associated with reduced hospital length of stay among those requiring admission.17, 18

Clinical pharmacists remain a scarce and expensive resource, making routine medication review in all incoming patients untenable.19, 20, 21 As a result, the majority of patients presenting to EDs with clinically significant ADEs—recognized or not—are discharged without medication review.5 Evidence‐based criteria can enhance the identification and treatment of patients presenting with ADEs and are needed to ensure that high‐risk patients are evaluated by clinical pharmacists to optimize their outcomes and reduce subsequent health services utilization.

Our group previously prospectively derived clinical decision rules that allow care providers in EDs to identify incoming patients presenting with ADEs.6 In the derivation study, the objective was to derive clinical decision rules that were sensitive for the detection of adverse events. This would allow their identification early in a patient's hospital course so that patients could be referred to a clinical pharmacist for medication review. In the present study, our goal was to validate these rules (Figures 1 and 2) by assessing their diagnostic accuracy in a new cohort of patients. A secondary aim was to evaluate the accuracy of variables collected by nurses, to evaluate whether nurses could apply the rules. Based on derivation, our hypothesis was that at least one rule would maintain a sensitivity of >90% in identifying patients with moderate or severe ADEs.

Figure 1.

Rule 1 used to screen for moderate or severe ADEs, and its classification performance. *During clinical decision rule derivation having taken medication in the previous 2 weeks was an inclusion criterion; however, during piloting the triage nurses applying the rule asked that this criteria be built into the first step of each rule to enhance its ease of use and functionality. †Medication changes included medication stops and starts, and changes to dose, frequency or route of administration. ADE = adverse drug event.

Figure 2.

Rule 2 used to screen for moderate or severe ADEs, and its classification performance. *During derivation having taken medication in the previous 2 weeks was an inclusion criterion; however, during piloting the nurses applying the rule asked that this criteria be built into the first step of each rule to enhance its ease of use and functionality. †Medication changes included medication stops and starts, and changes to dose, frequency or route of administration. ADE = adverse drug event; CTAS = Canadian Triage Acuity Score.

Methods

Study Design and Setting

This was a prospective cohort study that was conducted in two Canadian teaching hospitals (Vancouver General Hospital, Vancouver, British Columbia; and the Ottawa Civic Hospital, Ottawa, Ontario) and one urban community center (Lions Gate Hospital, North Vancouver, British Columbia) with a combined annual ED census of 215,000 visits. One of these sites had participated in the previous derivation study.6

The research ethics boards of all participating sites approved the study protocol and waived the need for informed consent for study enrollment, prospective data collection, and subsequent chart review (for follow‐up after the ED visit). Written informed consent was mandated for any follow‐up telephone calls. We set out to validate two clinical decision rules in this study, as evaluating more than one rule at a time increases the chances of successfully validating at least one rule and allowed us to target different performance metrics for different EDs.

Selection of Participants

All patients who presented to a participating ED between August 2014 and September 2015 during a scheduled data collection shift were eligible for enrollment. We scheduled day (0800–1559 hours), evening (1600–2359 hours), and weekend shifts proportional to the volume of incoming patients during the same time interval in the prior fiscal year. We did not schedule data collection shifts between 0000 and 0759 hours as the number of eligible patients presenting at nighttime had been small during the prior derivation study, rendering nighttime enrollment inefficient and costly.6

Enrollment of consecutive eligible patients during data collection shifts would have overwhelmed our ability to complete data collection forms and outcome assessments without disrupting the flow of patients through the EDs. To minimize the study's impact on patient flow and enroll a representative sample of patients, research pharmacists used a previously developed algorithm to systematically select patients for the study from among all incoming patients.5, 6 Pharmacists tallied the number of patients presenting in the hour prior to their start time, from which they randomly selected one patient using an online random number generator. As sicker patients tend to linger in the ED and are systematically different from patients discharged rapidly, this strategy allowed us to avoid the selection bias that would have occurred had pharmacists randomly selected from among all patients in the ED. After completing enrollment and data collection on the first patient, research pharmacists used fixed time intervals (e.g., 45 minutes) from the first patient's presentation to approach subsequent eligible patients. This was quicker than retallying patients who had presented within the past hour and ensured that lingering patients could not be sampled more than once. When applied in a previous study,5 this algorithm yielded a sample representative of the age, sex, and triage acuity of all presenting patients (unpublished data).

We enrolled patients who were 19 years of age or older, reported using at least one prescription or over‐the‐counter medication in the 2 weeks prior to presentation, and who either spoke English or had a translator available when they presented to the ED. We excluded patients if they exhibited violent behavior; presented with intentional self‐poisoning, needlestick injury, or sexual assault; were previously enrolled; presented for a scheduled revisit; were transferred directly to an admitting service; were triaged to a fast‐track zone (in which the time to patient disposition was too rapid for enrollment); or left against medical advice or prior to seeing the physician and pharmacist.

Intervention

After enrollment, research assistants placed data collection forms containing standardized clinical variables and the clinical decision rules to be validated in patient charts (Figures 1 and 2). The standardized clinical variables were ones that were associated with ADEs in the derivation study, but were not used in our final models.6 ED nurses who were unfamiliar with the study were briefly oriented to the study procedures and data collection forms at the beginning of data collection shifts by research assistants. Nurses completed data collection forms during their initial patient assessments and prior to the research pharmacist's assessments. Nursing data collection forms were removed from patient charts prior to research pharmacist assessments. We incentivized nurses to use the forms with monthly prize draws; each completed form was counted as one entry toward the prize.

Research pharmacists, both residency‐trained clinical pharmacists (KB and AZ) who normally worked in the ED or on a hospital ward, collected demographic and clinical information in the ED, including diagnostic test and imaging findings from emergency physicians and by chart review. They obtained best‐possible medication histories using provincewide electronic medication dispensing data from PharmaNet and the Ontario's Drug Profile Viewer for all patients in whom data were available and from patient, family and care provider interviews: The pharmacists reviewed the electronic information for medication accuracy, completeness, and adherence by interviewing patients and care providers. They determined any over‐the‐counter or alternative therapies by patient and care provider interviews. When required, they called the patients’ family physicians and community pharmacists for clarification. Given the inclusion of patients who might not be able to provide an accurate history (e.g., patients with delirium) and the difficulty of asking nurses to collect detailed, high‐quality medication data in busy EDs, we prespecified that when nursing variables were missing or inaccurate, we would replace them with variables derived from hospital registration, laboratory, and medication‐dispensing data (i.e., PharmaNet for patients enrolled in British Columbia and Ontario Drug Benefits Program for low‐income and >65 year‐old patients enrolled in Ontario). Variables derived from electronic medication‐dispensing data were verified by research pharmacists when collecting a best possible medication history and were documented prior to outcome assessments.

Outcome Ascertainment

Research pharmacists evaluated whether or not the patient presented with an ADE using a validated causality algorithm, previously adapted to our ADE definition.22 After completing and documenting their outcomes assessments, pharmacists interviewed the treating physician using a standardized questionnaire to determine whether or not the physician believed the patient had suffered an ADE (yes/no/uncertain) and to identify any alternative diagnoses for the patient's presentation. When the physician and pharmacist determinations of a patient's ADE status were concordant (ratings yes/yes or no/no), this was considered the criterion standard. If there was any disagreement (ratings yes/no, yes/uncertain, or no/uncertain), or if both ratings were uncertain, an independent committee, composed of a pharmacist and physician otherwise uninvolved in the study, adjudicated the case using a previously developed algorithm (Figure 3).5

Figure 3.

Adjudication procedure for uncertain or discordant events. In the ED each patient was evaluated by a clinical pharmacist and the treating emergency physician independently and blinded to each other's evaluations. The ratings were combined while the patient was still in the ED, but after each rater had documented their assessment. If there was any disagreement about the rating (i.e., yes/no, yes/uncertain, no/uncertain, etc.), or if one or both of the evaluations were uncertain, the case proceeded to independent adjudication by a committee consisting of a pharmacist and physician who were not in any other way involved in the study. ADE = adverse drug event

If enrolled patients were admitted to hospital, we followed their course of care until discharge by chart review and telephoned consenting patients after hospital discharge, if follow‐up was necessary to determine whether or not an ADE was present upon presentation (e.g., for the results of Clostridium difficile toxin assays pending at discharge).

Outcome Definition

Varying case definitions of ADEs exist.23, 24, 25, 26 We defined the primary outcome, an ADE, as an “untoward and unintended event arising from the appropriate or inappropriate use of a prescription or over‐the‐counter medication.”23, 25, 26 ADEs included adverse drug reactions, a response to a prescription or over‐the‐counter drug that is noxious and unintended and occurs at doses normally used in man for the prophylaxis, diagnosis, or therapy of disease,26 and events due to nonadherence or drug withdrawal, errors in prescribing, dispensing or medication administration, drug interactions, supra‐ or subtherapeutic dosing, untreated indications, and inappropriate drug use. The severity of all ADEs was rated as: 1) severe, if the event caused death or required admission; 2) moderate, if it resulted in a change in medical management (medical therapy, a diagnostic procedure or consultation); and 3) mild, if the event required no change in therapy.5 To meet the outcome definition we prespecified that events had to be categorized as at least moderate in severity, based on the actions of the treating physician. All events identified in this study are described in more detail in Data Supplement S1 (available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1111/acem.13407/full). Events were categorized as related to the chief complaint, if the patient's chief complaint was a direct result of the ADE (e.g., “vomiting blood” in a patient with an nonsteroidal anti‐inflammatory induced gastric ulcer), and categorized as incidentally found if it the chief complaint was unrelated to the ADE (e.g., “fall” in a patient found to have an international normalized ratio of 6).

Data Analysis

We calculated the inter‐rater reliability of the outcome measure during the pilot period using Cohen's kappa and reported it with 95% confidence intervals (CIs).27 The sample size calculation was based on the clinical decision rule maintaining a sensitivity of ≥90% with a desired precision of ±5%.6, 28 Given a conservative prevalence of 10% for moderate and severe ADEs in prior studies, we estimated requiring a sample size of 1,500 to capture 150 outcomes.5, 6 This would yield an accuracy of 95%CIs between 85% to 94% for a rule with an estimated 90% sensitivity.

We performed a planned interim analysis at midpoint of data collection. First, we assessed the quality of data collected by nurses by evaluating the proportion of patients misclassified for four key variables for which objective comparisons were available in administrative data: age category (i.e., ≥80 years), renal failure (i.e., creatinine ≥150 μmol/L), number medications (i.e., use of ≥4 medications), and recent medication changes. To be acceptable for uptake into clinical practice, our goal was to validate rules that were parsimonious and accurate. A priori, we aimed for fair specificity while maintaining sensitivity above 90%.19 We collected additional potential predictor variables and measured their univariate associations with the study outcome using two‐sided Student's t‐tests and the Pearson chi‐square test.

We calculated the sensitivity, specificity, and post‐test predictive values for each of two rules for identifying patients presenting with one or more ADEs with 95% CIs. For the purpose of estimating the proportion of patients who would be identified as high risk by the rules if implemented in clinical practice, we emulated a prior implementation study, in which we categorized all excluded patients as low risk, as they were either clinically deemed to be a very low risk of an ADE (e.g., on no medications within 2 weeks, presenting with needlestick injuries, sexual assault, imaging results), or medication review in the ED would not be feasible without incurring substantial additional resources or changes in patient flow (e.g., not English speaking and no translator available or presented for a direct admission).17, 18 Thus, we determined the proportion of patients who would be identified as high risk by the rules in clinical practice, by determining the proportion who screen positive, over all approached patients (N = 2,513). We used Stata/SE, version 13, for all analyses.

Results

Characteristics of Study Subjects

We enrolled 1,529 patients between September 2014 and August 2015. The patient flow diagram is depicted in Figure 4. The mean (±SD) age of enrolled patients was 59.3 (±20.9) years, 55.7% were female, and the median number of prescribed medications was 5 (interquartile range [IQR] = 3–9; Table 1).

Figure 4.

Patient flow. *No moderate or severe ADEs identified. †At least one moderate or severe ADE identified. ADE = adverse drug event.

Table 1.

Characteristics of Enrolled Patients

Characteristics	Number of Patients (N = 1,529)
Age (years)	59.3 (±20.9)
Age > 80 years	331 (21.7)
Female	851 (55.7)
Arrived by ambulance (n = 14,92)	496 (33.24)
CTAS
1	18 (1.2)
2	418 (27.4)
3	794 (52.0)
4	273 (17.9)
5	23 (1.5)
Most common chief complaints (n = 1,432)
Abdominal pain	127 (9.72)
Chest pain	127 (9.72)
Shortness of breath	62 (4.72)
Most common comorbid conditions
Hypertension	497 (32.5)
Mental health diagnosis	385 (25.2)
Atrial fibrillation	153 (10.0)
Comorbid conditions	0.96 (±1.05)
Most common prescription medications
Levothyroxine	228 (14.9)
Pantoprazole	218 (14.3)
Atorvastatin	208 (13.6)
Ramipril	165 (10.8)
Metformin	159 (10.4)
Prescription medications	5 (3–9)
Complementary and alternative medication use	708 (47.64)
Illicit drug use	123 (8.22)
Followed by a general practitioner	1331 (88.26)
Hospital admission	237 (15.5)
Died in hospital	0 (0.0)

Data are reported as mean (±SD), n (%), or median (IQR).

CTAS = Canadian Triage Acuity Score; IQR = interquartile range.

Assessment of Outcomes

The inter‐rater reliability of the pharmacist assessments of ADEs during the pilot period was 0.86 (95% CI = 0.6–1.0). In 1,369 (89.5%) of patients, the clinical pharmacist's and treating physician's outcome assessment was concordant; all others were adjudicated by an independent committee (Table 2). In total, 170 patients with missing physician assessments (n = 46) or discordant or uncertain ratings (n = 124) were adjudicated, of whom 85 met the final outcome definition. Three patients with a missing physician assessment were ultimately diagnosed with an ADE. Physicians did not recognize the ADE in 63/181 (34.8%) patients and were uncertain about 28/181 (15.5%) patients who met the final outcome definition.

Table 2.

Outcomes Assessment by Provider Group

Pharmacist Rating	Physician Ratinga
	No ADE	ADE	Uncertain	Total
No ADE	1,200	13b	20b	1,233
ADE	79b	99	15b	193
Uncertain	32b	7b	18b	57
Total	1,311	119	53	1,483

All cases in whom a rating was missing, or in whom ratings were discordant or uncertain, were adjudicated by an independent committee otherwise uninvolved in the study. Cases in which the pharmacist's and physician's ratings were concordant (ADE/ADE or no ADE/no ADE) were considered final.

ADE = adverse drug event.

^aPhysician assessments were missed in 46 patients; all of which went to adjudication.

^bAll discordant and uncertain cases (n = 124) were adjudicated.

Table 3.

Univariate Association Between Standardized Variables and ADEs

Characteristic	Outcome Measure		Difference (95% CI)
	No ADE (n = 1,345)	ADE (n = 184)
Age (years)	58.4 (±20.9)	66.1 (±19.8)	7.7 (4.5 to 10.9)a
Age cutoff > 80 years	274 (20.4)	57 (31.0)	10.6 (4.0 to 17.9)a
Female	742 (55.2)	109 (59.2)	4.1 (–3.6 to 11.4)
CTAS
1	16 (1.2)	2 (1.1)	0.1 (0.0 to 1.2)
2	357 (26.6)	61 (33.3)	6.6 (0.0 to 14.1)
3	692 (51.5)	102 (55.4)	3.9 (–3.7 to 11.5)
4	257 (19.1)	16 (8.7)	10.4 (5.1 to 14.4)a
5	21 (1.6)	2 (1.1)	0.5 (0.0 to 1.6)
Creatinine > 150 μmol/Lb	25 (5.4)	18 (19.2)	13.8 (6.9 to 22.7)a
Illicit drug use	106 (7.9)	17 (8.9)	1.0 (–2.7 to 6.1)
Present medication use
Antihypertensives	436 (32.4)	90 (48.9)	16.5 (8.9 to 24.1)a
Aspirin	254 (18.9)	43 (23.3)	4.5 (–1.5 to 11.4)
Opioid	157 (11.7)	43 (23.3)	11.7 (5.9 to 18.5)a
Antibiotic	149 (11.1)	35 (19.0)	7.9 (2.6 to 14.4)a
Insulin/hypoglycemic	110 (8.2)	31 (16.8)	8.7 (3.7 to 14.9)a
Benzodiazepines	136 (10.2)	20 (10.9)	0.8 (–3.4 to 6.3)
Antiepileptic	44 (3.3)	9 (4.7)	1.6 (–1.0 to 5.8)
Medical history
Renal failure	64 (4.8)	17 (9.2)	4.5 (0.9 to 9.6)a
Diabetes	136 (10.1)	31 (16.9)	6.7 (1.7 to 13.0)a
Heart failure	97 (7.2)	24 (13.0)	5.8 (1.5 to 11.6)a
Atrial fibrillation	107 (8.0)	22 (12.0)	4.0 (0.0 to 9.6)
Mental health diagnosis	212 (15.8)	33 (17.9)	2.1 (–3.1 to 8.6)
On > 3 prescription medicationsb	684 (52.4)	141 (82.0)	25.8 (18.6 to 31.9)a
On > 4 prescription medicationsb	560 (42.8)	126 (73.3)	26.8 (19.3 to 33.6)a
Antibiotic use within 7 days	146 (10.9)	38 (19.9)	9.0 (3.6 to 15.4)a
Ambulance arrivalb	424 (32.2)	72 (40.7)	7.6 (0.4 to 15.2)a
Hospitalized in the past 28 daysb	217 (18.8)	40 (25.2)	5.6 (0.0 to 12.4)
Medication changes within 28 days	863 (64.2)	154 (83.7)	19.5 (12.7 to 24.9)a
Assistance taking medicationsb	229 (17.5)	45 (25.7)	7.4 (1.4 to 14.4)a

Data are reported as mean (±SD) or n (%).

^aStatistically significant at p < 0.05.

^bDenominator less than 1,345 for patients without and 184 for patients with ADEs.

ADE = adverse drug event; CTAS = Canadian Triage Acuity Score.

Among enrolled patients, 184 (12.0%, 95% CI = 10.4%–13.8%) were diagnosed with 202 moderate or severe ADEs meeting the primary outcome definition and thus required a change in medical management according to the treating emergency physicians (Figure 4, Data Supplement S1). These included 96 (6.3%, 95% CI = 5.1%–7.6%) adverse drug reactions. Of 184 patients, 76.6% (95% CI = 69.8%–82.5%) experience ADEs that were chief complaint‐related, and 23.4% (95% CI = 17.5%–30.2%) had events that were incidentally found by pharmacists. None of these were fatal. In 34.5% (95% CI = 27.5%– 42.1%) of cases ultimately attributed to ADEs, emergency physicians did not attribute the presentation to a drug (Table 2). They were uncertain about whether an adverse event had occurred in an additional 16.1% (95% CI = 11.0%–22.4%).

Main Results

Standardized variables strongly associated with ADEs included age; use of opioids, antihypertensives, or antibiotics; recent medication changes; and the number of prescription medications (Table 2). Nurses misclassified 2.7% (8/311) of patients over 80 years old as being younger, 60.5% (26/43) of patients with creatinine values of over 150 μmol/L as not having renal failure, 45.4% (465/1023) of patients taking four or more regular medications as taking less, and 40.7% (319/783) of patients with recent medication changes as having not had changes to their medications. As a result, we assessed the rules’ accuracy using variables derived from hospital registration, laboratory, and PharmaNet data for medication‐related variables.

Rule 1 identified 168/184 patients experiencing an ADE for a sensitivity of 91.3% (95% CI = 86.3%–95.0%) and a specificity of 37.9% (95% CI = 35.3%–40.6%; Figure 1). These criteria identified 39.9% of all incoming patients as high risk, after excluding patients who did not meet inclusion criteria and would have therefore been deemed “low risk” by the rule (N = 2,513 patients).

Rule 2 identified 176/184 ADEs, yielding a sensitivity of 95.7% (95% CI = 91.6%–98.1%) and a specificity of 22.8% (95% CI = 20.6%–25.2%; Figure 2). These criteria identified 48.3% of all incoming patients as high risk after excluding patients who did not meet inclusion criteria and would have therefore been deemed low risk by the rule (N = 2,513 patients).

Discussion

We validated criteria to identify ED patients presenting with clinically significant ADEs in a new cohort of patients. Both sets of criteria we evaluated were sensitive for the detection of this outcome. Implementation of either tool would allow clinical pharmacists to rapidly screen patients entering EDs for their risk of ADEs prior to proceeding with full medication review. We previously demonstrated the value of this strategy in a prospective implementation study in one tertiary and two community hospitals, in which pharmacist‐led medication review in the ED was associated with a reduction in subsequent hospital bed utilization in high‐risk patients.17, 18 Given the multitude of other tasks pharmacists are expected to fulfill in busy EDs, these criteria can help identify high‐risk patients likely to benefit from medication review, as well as low‐risk patients in whom the intervention can safely be omitted. In addition to validating the clinical decision rules, our study confirms the high proportion of patients who present to EDs with clinically significant ADEs,5, 6, 29 as well as the significant proportion of events that physicians are unlikely to attribute to a medication‐related cause without pharmacist assessment,7, 8, 10 highlighting the need to improve care for this patient group.

Even though the specificity of our rules was limited, ADEs were so common that pharmacists identified clinically significant events in every fifth high‐risk patient. Without the rules in place, all patients would have needed to be seen by a clinical pharmacist to redress the high rate of ADEs that would not have been attributed to a medication‐related cause by physicians. We believe that reducing this number by 60% represents a substantial improvement.

Medication‐ and laboratory‐based variables collected by nurses at the point of care were inaccurate. Therefore, we validated the criteria using variables derived from hospital registration, laboratory, and outpatient medication dispensing data with clinical pharmacists verifying the medication history with the patient or his or her family. As a result, the rules can be implemented by clinical pharmacists as a rapid screening tool in centers that do not yet have access to electronic medication dispensing or other electronic data, allowing those hospitals to reduce hospital bed utilization in high‐risk patients.17, 18

With the near universal adoption of electronic health records in the United States over the past decade, their accelerating uptake throughout Canada, and plans to develop electronic drug information hubs in the Canadian provinces, these rules may be automated and integrated into electronic medical records to provide patient‐level decision support.30, 31 Even in jurisdictions without access to electronic medication dispensing data, algorithms can be developed that allow the electronic medical record to autopopulate the rules with known data elements (e.g., age), so that the pharmacists can efficiently screen patients by entering only data not already contained in the electronic medical record. Once integrated into electronic medical records, automated screening is likely to facilitate their widespread uptake.

Strengths of this study include adherence to methodological standards for the validation of clinical decision rules, and patient enrollment from tertiary care and community hospitals in two provinces, enhancing their generalizability.32 Study subjects were selected using a standardized enrollment algorithm to ensure a representative sample. The outcome measure was clearly defined, prospectively ascertained, and independently assessed by two raters, a pharmacist and a physician. If any disagreement or uncertainty occurred, an independent committee adjudicated the case. Clinical variables used as predictors were standardized and collected prior to determination of the patient's outcome. The most commonly implicated culprit medications and ADEs were consistent with those identified in population‐based estimates from the United States, and therefore, the rules are likely generalizable to other hospitals and jurisdictions.33, 34 We tested the accuracy and effectiveness to two different sets of criteria: less sensitive but simpler criteria for centers with less clinical pharmacists and more sensitive criteria for centers with greater pharmacist availability.

Limitations

There is generally a tradeoff between the sensitivity and specificity of diagnostics tests. As a result, we deliberately did not set the desired sensitivity of our criteria to 100%, allowing us to maintain a higher specificity that would facilitate the uptake of this tool into clinical practice. While some clinicians may view this as a disadvantage, recognition of 90% of events represents a substantial improvement of the current practice in which only 51% to 62% of events are identified by emergency physicians.7, 8, 10

We assessed the accuracy of nursing variables given the challenges experienced with prospective data collection in busy EDs in other studies and found their accuracy inconsistent. In retrospect, we might have been able to mitigate this by training nurses more carefully on the application of the rules. However, based on our observations we concluded that rules based on data collected by nurses at the point of care may not achieve high accuracy and we used variables derived from hospital registration, laboratory, and medication dispensing data that had been verified by clinical pharmacists. This means that the criteria can be applied by clinical pharmacists who work in EDs as rapid screening tools to identify high‐risk patients. Alternatively, the criteria can be integrated as electronic algorithms into electronic medical records with access to electronic data to provide automated patient‐level decision support tools; however, this will be limited by the ability of medication dispensing data to reflect the real‐world medication regimen of the patient. Due to a restrained study budget, we were unable to enroll patients at nighttime, limiting the generalizability of our findings, as nighttime data collection was inefficient and costly in derivation.6 While we hope future validation studies will address this limitation, current pharmacist staffing models in most North American EDs generally exclude nighttime coverage, making our study generalizable to those patients in whom the rules would applied.35 Finally, as we regarded ADEs as diagnoses of exclusion, some presentations that are commonly multifactorial (e.g., falls, delirium) may have been underrepresented in our outcomes, as we would have categorized these as non‐ADEs unless alternative diagnoses had been ruled out.

Conclusions

We validated clinical criteria that enable ED pharmacists to rapidly and efficiently screening patients for adverse drug events, allowing them to capture the majority of these events at the earliest time point within a hospital encounter and limit harm. The enthusiasm for more widespread application of medication management interventions and current Canadian hospital accreditation standards reflects the relevance of this strategy to current health care practices. In future studies, these rules can serve as a starting point to develop risk stratification methods for patients in other practice settings in which adverse drug events are common.

The authors acknowledge the physicians and nurses working at Vancouver General, Lions Gate, and the Ottawa Civic Hospitals. This study would not have been possible without their dedication and support. The authors give particular thanks to Drs. Gina Gill, Frank Scheuermeyer, and Christina Bui for their service on the adjudication committee. The authors thank Dr. Vi Ho, Kelsey Seal, Puneet Vashisht, Carl Chan, and Robert Yao for their assistance in data collection. Finally, the entire research team thanks all patient participants for their willingness to be involved in the research process and patience with data collection.

Supporting information

Data Supplement S1. Adverse drug events meeting the primary outcome definition (N = 202) identified in 184 of 1529 patients.

Academic Emergency Medicine 2018;25:1015–1027.29517818