Automated Surveillance for Adverse Drug Events at a Community Hospital and an Academic Medical Center

Abstract

Objectives

To compare the rates and nature of ADEs at an academic medical center and a community hospital using a single computerized ADE surveillance system.

Design

Prospective cohort study of patients admitted to two tertiary care hospitals.

Outcome Measure

Adverse drug events identified by automated surveillance and voluntary reporting.

Methods

We implemented an automated surveillance system across an academic medical center and a community hospital. Potential events identified by the computer were reviewed in detail by medication safety pharmacists and scored for causality and severity. Findings were compared between the two hospitals, and with voluntary reports from nurses and pharmacists.

Results

Over the 8 month study period, 25,177 patients were admitted to the university hospital and 8,029 to the community hospital. There were 1,116 ADEs in 900 patients at the university hospital for an overall rate of 4.4 ADEs per 100 admissions. At the community hospital, 399 patients experienced 501 ADEs for a rate of 6.2 events per 100 admissions. Rates of antibiotic-associated colitis, drug-induced hypoglycemia, and anticoagulation-related ADEs were significantly higher at the community hospital compared with the university hospital. Computerized surveillance detected ADEs at a rate 3.6 times that of voluntary reporting at the university hospital and 12.3 times that at the community hospital.

Conclusions

Operation of a common automated ADE surveillance system across hospitals permits meaningful comparison of ADE rates in different inpatient settings. Automated surveillance detects ADEs at rates far higher than voluntary reporting, and the difference may be greater in the community hospital setting. Community hospitals may experience higher rates of certain types of ADEs compared with academic medical centers.

Introduction and Background

Adverse drug events (ADEs) have long been recognized to be one of the principle sources of harm to hospitalized patients, ^1–3 and improving the safety of medication management has evolved into a prime focus of quality improvement in healthcare settings. Despite unanimous agreement regarding the magnitude of the medication safety challenge, estimates of the frequency of adverse drug events vary widely, with different measurement strategies yielding different figures. Chart review has been considered the “gold standard” for ADE detection, although different chart review strategies have yielded divergent findings; ^4,5 from a practical point of view, chart review is heavily resource-intensive, precluding its use in screening large numbers of patients in an ongoing fashion.

Surveillance approaches to ADE detection—screening for specific data suggestive of the occurrence of an ADE—permit the ongoing monitoring of large numbers of patients for ADEs with fewer resources than chart review. Several groups have demonstrated the efficacy of automated surveillance for detecting ADEs. A study by Classen et al. ⁶ using computerized surveillance yielded 731 verified ADEs over an 18 month period, compared with only 9 ADEs detected by traditional voluntary reporting. This study used a series of simple rules examining chemistry, hematology, drug level, and drug order data. A subsequent study by Jha et al. ⁷ refined this methodology with the use of more complex rules. The authors showed a high rate of ADE detection, identifying 275 ADEs by computer compared with 23 by voluntary reporting during the 8 month study period. The Agency for Healthcare Research and Quality's Critical Analysis of Patient Safety Practices ranked the use of computerized ADE detection under the category “High Strength of Evidence” for its impact and effectiveness. ⁸

In spite of its proven utility, automated ADE surveillance has not been widely implemented, and most reports to date have focused on systems in tertiary care academic medical centers. Little is known about ADE rates in community hospitals, and we are aware of no published experience with automated ADE surveillance in community hospitals. In the belief that a systematic method for ADE detection was critical to ensuring the effectiveness of medication safety interventions, we implemented a single, custom-built computerized ADE surveillance system at a community hospital and an academic medical center. This report describes findings from eight months of operation of this system at both hospitals.

Methods

All patients admitted to the two hospitals are monitored by the ADE surveillance system. The data reported here represent the period from March 1 to October 31, 2005. Durham Regional Hospital and Duke University Hospital are members of Duke University Health System. Durham Regional Hospital is a full-service community hospital of 391 licensed beds serving an eight county area and admitting approximately 16,000 patients annually. It is staffed primarily by community physicians, most of whom admit solely to Durham Regional Hospital. The core information systems consist of Siemens Medical Systems components. The hospital uses the Siemens barcode-based automated medication administration system.

Duke University Hospital operates 1,019 licensed beds and admits approximately 36,000 patients annually. Orders are written primarily by house staff, and all medical staff are members of the medical school faculty. The hospital information systems consist of a combination of custom-built and commercial products, including the McKesson pharmacy system and a locally developed core hospital information system. The hospital was implementing McKesson computerized physician order entry (CPOE): at the beginning of the study several floors were operational; at the conclusion, approximately 80% of the hospital was live on CPOE. A single centralized laboratory facility serves both community and university hospitals. Information systems components common to both hospitals include a single Cerner laboratory system and a common clinical data repository that receives real-time data from the principle patient care information systems, including laboratory, radiology, and other diagnostic reports. The two hospitals are connected by a high-bandwidth wide area network.

A detailed description of the design, technology, rules development, and operations of the ADE surveillance system is reported elsewhere. ⁹ Briefly, the surveillance system operates on a DB2 database that resides on a mainframe computer that operates Duke University Hospital's core hospital information system. The rule engine is written in PL/I, and the Web-based evaluation application is written in C++. The system queries patient data from each hospital daily to identify potential ADEs in hospitalized patients. The order set operates 69 rules. Examples of rule types include: orders for antidotes (e.g., Naloxone); toxic drug levels; and combinations of laboratory values or trends and medication orders (for example, a rapidly falling platelet count in a patient on heparin therapy). The list of rules is reported elsewhere. ⁹ Patient medication data are extracted from the different pharmacy systems at the two hospitals each day for querying purposes. Laboratory and demographic data from both hospitals are queried in the common clinical data repository.

The daily batch queries produce lists of potential ADEs that are subsequently evaluated by pharmacists specially trained in medication safety. Specifically, these pharmacists receive training in the use of the Naranjo algorithm for determining causality of events, ¹⁰ and the Duke seven-point severity scoring system (see definitions, ▶). ⁹ The evaluators review the patient's medical record (approximately 80% of cases can be resolved based on review of the electronic portions of the record; the other 20% require review of the paper chart) to determine whether the alert represents an ADE. Events are scored for causality using the Naranjo algorithm, ¹⁰ and for severity. ⁹ Events with a causality score of 5 or greater (probable or definite) and a severity score of 3 (transient harm to the patient) or greater are considered ADEs (▶). Multiple alerts triggered by a single drug administration (e.g., multiple administrations of D50 following a single dose of insulin) are scored as a single ADE. The evaluator records the names of the drug or drugs involved, and writes a brief narrative of the event. The pharmacist evaluators compare their findings among themselves and then with a physician reviewer. Kappa statistics for inter-rater reliability between pharmacists, and between pharmacists and physician reviewer were greater than 0.88 for causality and severity scoring. ⁹ The rules were reviewed for positive predictive value and low-yield rules removed from operation, and rules were added or modified based upon early experience prior to the beginning of the study period.

Table 1.

Table 1. Overall Adverse Drug Events and Rates per 100 Admissions by Severity

Severity Index	University Hospital		Community Hospital
	ADEs (%)	ADEs/100 Admissions	ADEs (%)	ADEs/100 Admissions
3: Transient adverse patient effects occurred which required some corrective therapy, increased length of stay by 1–2 days or resulted in lab values, vital signs or medication effects outside the desirable parameters.	973 (87)	3.9	419 (84)	5.2
4: Significant adverse patient effects occurred which required aggressive intervention such as code, intubation, transfer to ICU, interventional drug therapy or increased length of stay > 2 days.	141 (13)	0.56	78 (16)	0.97
5: Permanent adverse patient effects occurred such as paralysis, brain damage, disability or loss of limb, organ or bodily function.	1 (0.1)	0.00	1 (0.2)	0.01
6: Patient death	1 (0.1)	0.00	3 (0.6)	0.04
Total ∗	1116 (100)	4.4	501 (100)	6.2

^∗ Totals may differ due to rounding.

ADEs = Adverse Drug Events; ICU: Intensive Care Unit.

Duke University Health System operates an electronic reporting system for safety incidents or concerns. The system offers single portal reporting of all manner of incidents including medication errors and adverse drug events. Staff members can report anonymously if desired. The system has been in use at both the university and the community hospital since 2004, and is heavily utilized, generating approximately 580 reports per month. Approximately 40% of these reports are medication-related. Hospital pharmacists reviewing these reports assign each a severity score using the same 7-point system referenced above. We compared volumes of voluntary reports scored by hospital pharmacists as severity score 3 or greater from hospital areas covered by the surveillance system (e.g., not outpatient areas), with the volumes of reports detected by the ADE surveillance system during the first 4 months of the study period; after finding consistent report ratios for 4 months in a row, we discontinued this time-consuming comparison.

Statistical analysis was performed using the chi-square test to compare categorical variables and a two-tailed t-test to compare continuous variables. We used SAS statistical package, version 8.2 for all analysis (SAS Institute, Inc., Cary, NC). For study purposes, data were de-identified and made anonymous to comply with HIPAA and Federal privacy standards; and the work was granted exemption from requirement for Institutional Review Board approval.

Results

Between March 1 and October 31, 2005, the university hospital cared for 25,177 inpatients for 148,416 total patient days; 8,029 patients were admitted to the community hospital and spent a total of 54,589 patient days in hospital. Overall ADE incidence rates at the two hospitals are shown in ▶. During the eight month study period, 900 patients admitted to the university hospital experienced 1,116 ADEs, or 4.4 ADEs per 100 admissions. At the community hospital 399 patients suffered a total of 501 ADEs, for a rate of 6.2 ADEs per 100 admissions. The ratio of ADEs to total alerts (including duplicate alerts) was almost identical at the two hospitals (0.0663 at the community hospital and 0.0665 at the university hospital, p = 0.965).

Events representing the full range of severity were detected at both hospitals (▶). At the university hospital there were 973 events of severity 3, 141 of severity 4, 1 of severity 5, and one death; the single death was associated with an anticoagulant. At the community hospital there were 419 events of severity index 3 (transient adverse patient effects); 78 of severity 4 (significant adverse patient effects requiring aggressive intervention), 1 of severity 5 (permanent adverse patient effects), and three ADE-associated deaths. Two of the deaths were associated with anticoagulants and one with opiates.

ADEs detected by automated surveillance fell into six broad categories as shown in ▶. Among these categories there was a significantly higher incidence at the community hospital of drug-induced hypoglycemia (p = 0.009), anticoagulation-related events (p = 0.015), and antibiotic-associated C. difficile colitis (p < 0.0001). Rates of narcotics-related events, drug-induced hyperkalemia, and nephrotoxin-induced renal compromise were comparable between the hospitals.

Table 2.

Table 2. ADEs by Category at University and Community Hospitals

Category	University Hospital		Community Hospital		p-value
	ADEs (%)	ADEs/100 Admissions	ADEs (%)	ADEs/100 Admissions
Hypoglycemia	257 (23)	1.0	110 (22)	1.4	∗ 0.009
Anticoagulants	224 (20)	0.9	96 (19)	1.2	∗ 0.015
Hyperkalemia	215 (19)	0.9	58 (11)	0.7	0.26
Narcotics/Benzodiazepines	150 (13)	0.6	62 (12)	0.8	0.084
C. difficile colitis	127 (11)	0.5	117 (23)	1.5	∗ <0.0001
Nephrotoxins and Increased Cr	81 (7.3)	0.3	26 (5.1)	0.3	0.98
Miscellaneous	62 (5.6)	0.2	32 (6.9)	0.4	∗ 0.025
Total	1116 (100)	4.4	501 (100)	6.2

^∗ Statistically significant differences observed.

The marked difference in the rates of antibiotic-associated C. difficile colitis became apparent early in the study period, and we brought the disparity to the notice of the infection control officer for the community hospital. During the summer months the infection control group instituted an aggressive campaign to reduce the incidence of C. difficile infection. Bleach cleansing was instituted of every room occupied by patients with C. difficile following discharge. Every medical staff member was sent a letter describing the problem and reminding them to employ soap and water for hand washing, rather than alcohol foam which is ineffective in eradicating C. difficile spores. Posters were placed throughout the care units containing the same information. Following the implementation of these measures, the incidence of antibiotic-associated C. difficile colitis as detected by ADE surveillance at the community hospital decreased markedly (▶).

Figure 1.

Antibiotic-associated C. difficile colitis at the community hospital and the university hospital. See text for description of intervention.

At both hospitals, approximately 17% of all patients experiencing ADEs suffered more than one event (▶). The combinations of ADEs in these patients varied widely; common combinations included repeat episodes of hypoglycemia requiring reversal; repeat episodes of hyperkalemia requiring reversal, or repeat episodes of transient renal impairment due to nephrotoxins. Many multi-ADE patients experienced apparently unrelated ADEs. At the university hospital, those experiencing large numbers of events (5 or more) were principally transplant patients or oncology patients with multiple medications and admissions and complex medical histories. These patients experienced events such as drug-induced renal failure, drug-induced hyperkalemia, antibiotic-associated C. difficile colitis, narcotics-related oversedation, and drug-induced peripheral neuropathy. One patient with diabetes suffered multiple episodes of hypoglycemia requiring reversal with 50% dextrose due, in part, to delays in adjusting their insulin regimen following events. At the community hospital, one patient who experienced 6 ADEs had multiple problems including peripheral vascular disease, and suffered narcotics-related oversedation and antibiotic-associated C. difficile colitis; another patient with an artificial mitral valve suffered 7 ADEs including drug-induced hyperkalemia and benzodiazepine-associated oversedation; each of these patients also suffered two separate warfarin-related bleeding episodes.

Table 3.

Table 3. Patients with Multiple Adverse Drug Events

ADEs	Patients (%)
	University Hospital	Community Hospital
1	744 (83)	333 (83)
2	117 (13)	43 (11)
3	27 (3)	15 (4)
4	7 (0.7)	6 (1)
5	2 (0)	0 (0)
6	2 (0)	1 (0)
7	1 (0)	1 (0)
Total	900 (100)	399 (100)

To examine more closely those events that had more than a minor transient impact on patients, we analyzed severe events (Severity Index ≥ 4, requiring aggressive intervention) separately. Among more severe events anticoagulation-related ADEs stood out as the most important category at both hospitals; in addition, such events were significantly more frequent at the community hospital (0.12 per 100 admissions at the university hospital versus 0.31 per 100 admissions at the community hospital, p < 0.001). Warfarin-related events comprised the principle source of anticoagulation events detected at both hospitals; among severe warfarin-related events, the majority originated in the outpatient setting (e.g., the patient had an elevated International Normalized Ratio (INR) value at the time of admission to the hospital): 73% of those at the university hospital, and 92% of warfarin events at the community hospital. As a rough measure of tightness of anticoagulation control, we examined INR values at the time of ADE detection. Comparison of mean INR scores at detection for all warfarin-related events (severity ≥ 3) revealed a significant difference in level of anticoagulation control between the two patient groups, with a mean INR of 6.2 at the university hospital, versus 10.1 at the community hospital (p < 0.0001).

We compared ADE detection by automated surveillance with detection by voluntary reporting using the health system's electronic reporting system during the first 4 months of the study period; we stopped performing exhaustive comparisons after this as 4 months' data was compelling and side-by-side comparisons are resource-intensive. During these 4 months, 144 medication-related reports associated with inpatients were assigned a severity level of 3 or greater at the university hospital, while 520 ADEs were detected by ADE surveillance. At the community hospital, 23 medication-related events were reported by voluntary reporting, while 283 ADEs were detected by ADE surveillance. Thus ADE surveillance detected ADEs at a rate 3.6 times greater than voluntary reporting at the academic medical center, and 12.3 times greater at the community hospital.

Discussion

We have described the application of a system for computerized surveillance for ADEs at an academic medical center and a community hospital, and findings from eight months' operation of the system at these hospitals. We observed 4.4 and 6.2 ADEs per 100 admissions at the university hospital and the community hospital, respectively. The overall incidence of ADEs detected at the two hospitals was very comparable to the findings of other investigators using similar methods. In his 1991 report, Classen ⁶ reported an incidence of 1.67 per 100 admissions; Jha et al. ⁷ subsequently detected 4.1 events per 100 admissions using an extended version of Classen's rules. Like these and other investigators, ¹¹ we observed that automated ADE surveillance detects events at rates far higher than those of voluntary reporting, in spite of an aggressive reporting culture at our academic medical center.

The community hospital had a significantly higher incidence of C. difficile colitis during the first six months of the study period. The reasons for the greater initial incidence of C. difficile colitis at the community hospital are not certain, but we know of several likely contributing factors. This hospital receives a large number of patients from a chronic care facility whose patient population is at high risk for antibiotic-associated colitis, and which has had a high incidence of C. difficile infection. In addition, in some of the care units at the community hospital, sinks are located at the back of the unit, a geography that is not conducive to frequent soap and water hand washing. The hospital's infection control group launched an aggressive campaign aimed at reducing the spread of C. difficile, which appears to have been effective, as ADE surveillance detected a dramatic reduction in C. difficile-related events at the community facility in the last several months of the study period; rates at the university hospital remained unchanged. It is too early to determine whether this improvement will be sustained.

Among events of greater severity, a significantly higher incidence of warfarin-related over-anticoagulation events was noted at the community hospital, and INR at presentation in these events was significantly higher compared with the university hospital. We hypothesize that this discrepancy is related to the different physician groups involved and ease of access to expert resources for outpatient anticoagulation management. Community hospital patients are cared for by community physicians with limited access to specialized anticoagulation management resources, whereas university hospital physicians have access to a hospital-based specialty clinic for coagulation management. In response to these findings, we are developing a health system-wide outpatient anticoagulation management program that will facilitate comprehensive management of patients on warfarin.

A high proportion of patients at both hospitals suffered multiple ADEs. Indeed, experiencing one ADE appears to put patients at risk for a second. While a patient admitted to the university hospital had on average a 3.6% chance of experiencing one or more ADEs, a patient who has suffered one ADE has on average a 17% chance of experiencing another. There appear to be several reasons for this observation. First, many of these patients were receiving medications that would not routinely be discontinued following an ADE. Therefore it is not surprising to find that a number of patients experienced “repeat” ADEs of the same type, due either to failure to adjust medication regimens following a first occurrence, or to difficulty achieving an optimal regimen in the face of complex disease physiology. Second, complex patients suffered combinations of ADEs reflective of the hazards of hospitalization (e.g., C. difficile colitis) and treatment with multiple medications. Finally, it is possible that our methodology overestimates the relative incidence of multiple ADEs because it under-detects idiosyncratic ADEs (also called adverse drug reactions). The surveillance system rules principally target known and expected effects of medications with narrow therapeutic indices (e.g. warfarin, opiates, insulin), and do not target or detect many cases of idiosyncratic reactions (such as allergic reactions), which are less likely to occur multiple times over a short time period.

It is not possible to know with certainty the relative contribution of the different medication safety-related information technology systems in use to the observed differences in ADE rates between the two hospitals. At the time of the study, the partially implemented CPOE system at the university hospital provided only minimal basic safeguards for dosing (e.g., overdose flags for multiple times the maximum listed dose for any drug), no drug-laboratory value warnings, and no protocols guiding the specific dosing of anticoagulants, insulin, or opiates. The use of a bar code medication administration system at the community hospital may have made the administration process safer at that hospital than at the university hospital, but there is no way to know this without an extensive comparative analysis. More importantly, it is likely that other, more significant differences between the university and community hospital—patient mix; clinical pharmacist ratios and practice patterns; nursing staffing patterns or experience levels; physician mix—dwarf the effects of differences in information technology use between the two settings.

The strengths of automated surveillance include efficiency and scalability. The traditional gold standard for adverse event detection is chart review, ^2–4 which generally involves manual review of a sample of patient charts for evidence of ADEs. The efficiency and rapidity of automated surveillance permits monitoring of every patient admitted to the hospital. Jha et al. ⁷ demonstrated that an ADE surveillance system similar to that which we employ required one fifth the person-hours of manual chart review. Automated surveillance is not affected by the myriad factors that influence voluntary reporting rates; our implementation has demonstrated consistent performance and high levels of agreement between medication safety pharmacist operators, and detects ADEs at a far higher rate than voluntary reporting.

Our study of automated ADE surveillance has several important limitations. We have not compared findings from automated surveillance with a gold standard (such as chart review) to validate the system's performance. Indeed, we are aware that our implementation of automated surveillance is less comprehensive than chart review in detecting ADEs, because we are constrained by the range of clinical data types available to the surveillance system. At present the surveillance system has electronic access to laboratory, pharmacy and demographic data only. We are unable to detect signs and symptoms of ADEs that might be recorded, for example, in nursing notes, or reflected in changes in vital signs. We expect in the future to incorporate rules based upon data from electronic clinical documentation systems; previous reports have demonstrated that expansion of data capture to include documentation data can significantly expand the number of ADEs detected. ^7,12 Finally, in spite of being more efficient than a chart review-based process, automated surveillance nonetheless requires specialized resources; two medication safety pharmacists spend approximately 80% of their time evaluating the alerts generated by the system from the two hospitals.

There is a potential source of bias in the study regarding our observation of differences in ADE rates between the two hospitals. Our pharmacist evaluators could not be blinded to the location of the automated alerts, and it is possible that some bias attached to their evaluations based on location. However, the fact that the ratio of alerts generated to ADEs detected was the same at the two hospitals suggests that the differences were due to different underlying event rates. In addition, the use of explicit causality and severity criteria serve to further reduce the impact of any potential bias upon our findings. Nonetheless, a contribution due to bias cannot be ruled out.

Conclusion

We have successfully implemented automated surveillance for ADEs at a community hospital and an academic medical center using the same computer system, rules base and evaluation methodology, permitting comparison of event patterns between hospitals. ADE rates in several categories were significantly higher at the community hospital. The system detects ADEs in a consistent fashion, bringing to our attention large numbers of events of which we were previously unaware, and detecting a change in incidence of a specific outcome (C. difficile colitis incidence) consistent with a change in practice targeting that outcome. Automated surveillance detects ADEs at rates far higher than voluntary reporting, and the difference may be greater in the community hospital setting. We hope in the future to expand the system's range of event detection as additional electronic care systems are implemented at our hospitals, and to extend automated ADE surveillance to the ambulatory arena.