Abstract
Abstract Background: Adverse drug events (ADEs) are both common and costly. Most hospitals identify ADEs using spontaneous reporting, but this approach lacks sensitivity; chart review identifies more events but is expensive. Computer-based approaches to ADE identification appear promising, but they have not been directly compared with chart review and they are not widely used.
Objectives: To develop a computer-based ADE monitor, and to compare the rate and type of ADEs found with the monitor with those discovered by chart review and by stimulated voluntary report.
Design: Prospective cohort study in one tertiary-care hospital.
Participants: All patients admitted to nine medical and surgical units in a tertiary-care hospital over an eight-month period.
Main Outcome Measure: Adverse drug events identified by the computer-based monitor, by chart review, and by stimulated voluntary report.
Methods: A computer-based monitoring program identified alerts, which were situations suggesting that an ADE might be present (e.g., an order for an antidote such as naloxone). A trained reviewer then examined patients' hospital records to determine whether an ADE had occurred. The results of the computer-based monitoring strategy were compared with two other ADE detection strategies: intensive chart review and stimulated voluntary report by nurses and pharmacists. The monitor and the chart review strategies were independent, and the reviewers were blinded.
Results: The computer monitoring strategy identified 2,620 alerts, of which 275 were determined to be ADEs. The chart review found 398 ADEs, whereas voluntary report detected 23. Of the 617 ADEs detected by at least one method, 76 ADEs were detected by both computer monitor and chart review. The computer monitor identified 45 percent; chart review, 65 percent; and voluntary report, 4 percent. The ADEs identified by computer monitor were more likely to be classified as “severe” than were those identified by chart review (51 versus 42 percent, p =.04). The positive predictive value of computer-generated alerts was 16 percent during the first eight weeks of the study; rule modifications increased this to 23 percent in the final eight weeks. The computer strategy required 11 person-hours per week to execute, whereas chart review required 55 person-hours per week and voluntary report strategy required 5.
Conclusions: The computer-based monitor identified fewer ADEs than did chart review but many more ADEs than did stimulated voluntary report. The overlap among the ADEs identified using different methods was small, suggesting that the incidence of ADEs may be higher than previously reported and that different detection methods capture different events. The computer-based monitoring system represents an efficient approach for measuring ADE frequency and gauging the effectiveness of ADE prevention programs.
In today's competitive health care market, hospitals are searching for ways to provide better-quality care at lower cost. Adverse events represent an important aspect of quality of care, and these events are both costly and frequent.1,2,3 In the Harvard Medical Practice Study,4 nearly 4 percent of hospitalized patients in New York state suffered an injury due to medical treatment. Of these, nearly 20 percent were due to drugs.5 Other studies of hospitalized patients have also shown that adverse drug events (ADEs) are common,2,6,7,8,9,10,11,12,13,14 although the reported rates have varied depending on the criteria used for defining events and the methods by and intensity with which they were sought.15
An ADE is different from an adverse drug reaction (ADR). The World Health Organization defines an ADR as “an effect which is noxious and unintended, and which occurs at doses used in man for prophylaxis, diagnosis, or therapy.”16 This definition is restrictive because it considers only incidents in which the use of a drug is appropriate, whereas many adverse events are due to error.17 Therefore, we have chosen as our primary outcome the adverse drug event (ADE), defined as “an injury resulting from medical intervention related to a drug.”2 These events include non-preventable occurrences such as unpredictable drug rashes, expected ones such as complications of chemotherapy, and incidents caused by errors in prescribing, dispensing, or administering drugs.
In a previous study,17 we compared the yield in identification of ADEs from intensive chart review by nurses with that from stimulated confidential voluntary report, in which the nurses and pharmacists on study units were asked daily whether any adverse events had occurred. We found that chart review identified many more ADEs than did stimulated voluntary report. Although chart review is an effective method for detecting ADEs, its expense makes it impractical for ongoing quality monitoring in hospitals.2
Use of a computer-based monitor is much less expensive than chart review, and previous studies18,19 have shown that it identifies many more ADEs than does spontaneous reporting. However, such computer monitors are still not widely used,19 because of their scarcity and doubts about their effectiveness. Classen et al.18 developed an ADE monitor at LDS Hospital in Salt Lake City, Utah. They found mainly moderate to severe ADEs and discovered that such events occurred in 2 percent of all admissions. Since they did not perform concurrent, independent chart reviews, it is unclear how the rates and types of ADEs found by their computer monitor would compare with those found by chart review.
We performed a prospective study with the following goals: 1) to compare the rate and type of ADEs detected using a computer-based monitor strategy with those found using a strategy of either chart review or stimulated voluntary report, 2) to evaluate the capture rate and positive predictive value of the detection rules used in the computer-based monitor, and 3) to compare the resources used by the different ADE detection methods.
Methods
Setting
Brigham and Women's Hospital is a 726-bed tertiarycare teaching hospital. Its internally developed information system, the Brigham Integrated Computer System, manages administrative, financial, and clinical information and provides clinical-results reporting and computer-based physician order entry applications.20 The system also includes a computer-based event detection application that uses rules to detect a wide variety of clinical events.21 The ADE screening rules developed for this project ran under this event detection application.
Patients
Patients studied included all adults admitted to nine units over an eight-month period from October 1994 through May 1995. These units included two medical intensive care units, one surgical intensive care unit, four medical general care units, and two surgical general care units, representing approximately one third of the hospital's beds and about a third of the patient-days on non-obstetric units during the study period.
Outcomes
Our main outcome measures were ADEs detected by the different strategies. Patients admitted to the hospital because of an ADE were excluded from the chart review and the stimulated voluntary report studies, because the main focus of these strategies was to measure in-hospital ADEs in order to evaluate the effect of an intervention to prevent such events. However, because ADEs that lead to hospitalizations are important from a health systems perspective, we collected and report these data gathered by the computer monitor. We also looked for potential ADEs, defined as “incidents with potential for injury related to a drug.” We measured the capture rate and positive predictive value of individual rules in the computer monitor rule base. Finally, we analyzed the person-hours involved in each ADE detection scheme.
Rule Base
We developed ADE detection rules that consist of Boolean combinations of simple medical conditions such as new medication orders, laboratory results above or below certain numeric thresholds, and medication orders associated with changes in laboratory values over time (see Appendix). For example, one combination would be “patient on medication X AND patient's serum lab > Y.”21 New combinations and conditions have been added over time to capture new kinds of medical events. The rules were created using a computer-based rule editor21 that permitted the ADE screening rules used in this study to be created and changed by non-programmers.
We used published rules from the LDS Hospital study18 as the starting point for developing our ADE monitor rule base. The published rules identified three main types of conditions: 1) orders for known antidotes (e.g., naloxone); 2) laboratory abnormalities (e.g., elevated serum potassium); and 3) laboratory abnormalities occurring in the presence of certain drugs (e.g., an increase in serum creatinine in a patient with an order for gentamicin).
Based on the LDS Hospital experience, we anticipated a high number of false-positive alerts. During the study, we therefore changed rules with low positive predictive value to increase their efficiency. For example, we modified a rule intended to detect allergic reactions to medications. The original rule identified all patients who received an order for diphenhydramine. In order to capture only new events, we added a second condition that excluded patients who had received a previous dose of diphenhydramine within the past seven days. We also added a third condition that excluded patients receiving a concurrent order for transfusion or paclitaxel, because diphenhydramine is routinely given in our hospital as a premedication when these are ordered.
Finally, we added new rules to the rule base during the study. For example, we created a rule that generated an alert when a patient had a falling or low platelet count in the setting of an order for a histamine-2 blocking agent. At the end of the study, there were 52 unique rules in the rule database.
Case Identification
Daily, for patients on the study floors, the computer generated a list of alerts, defined as instances that met the conditions of an ADE rule. The computer report included the patient's name, medical record number, bed location, date of event, and the specific condition met. A trained reviewer performed a targeted review of each patient's medical chart to assess whether an alert was associated with an ADE or potential ADE. Probable ADEs noted in the chart, although not directly related to the identifying alert, were also documented and included for further review. Multiple computer-generated alerts could be associated with a single ADE. The reliability of assessment of whether alerts represented ADEs or potential ADEs was checked in a sample of 100 alerts; the kappa value was.53, and the percentage agreement was 89 percent.
All incidents thought by the reviewer to represent an ADE or potential ADE were subsequently classified by a physician experienced in evaluating ADEs according to whether an ADE or potential ADE was present, using a previously described approach.2 The ADEs thus identified were also evaluated for severity and preventability. Severity was classified as significant, serious, life-threatening, or fatal,22 and events were judged preventable if they were felt to be due to an error or could have been averted by any means currently available.23 Reliability for the judgments made using this approach has previously been reported2; for judgments about whether an incident was an ADE, the kappa values were.81-.98; for preventability, kappa was.92; and for severity, kappas were.32-.37. Concurrent with this ADE monitor study with independent reviewers, an ADE detection study was conducted on the same nursing units. These reviews used a previously described methodology2 including daily manual chart review as well as stimulated voluntary report (i.e., confidential, voluntary reports solicited from the nursing and pharmacy staff about possible events). Physicians were not asked to provide reports because they are generally not geographically based in this hospital. The independent reviewers were blinded to the data generated by the computer monitor. Each suspected case identified by the reviewers was evaluated and assessed for severity and preventability in the same manner as for the ADE monitor strategy.
Analysis
We compared ADEs identified by each strategy to determine the yield and degree of overlap. The ADEs detected by each method were also compared for severity and preventability. For each screen, the positive predictive value was calculated: the numerator was the number of screens associated with an ADE, and the denominator the total number of screens of that type (sometimes an ADE was associated with more than one screen). Changes to the ADE monitor rule base were evaluated for their impact on the positive predictive value and capture rate. Comparisons between categoric variables were made using the chi-squared test with one degree of freedom. All analyses were performed using the statistical software SAS.24
Results
During the study there were 21,964 patient-days on the nine study units. Overall, 617 in-hospital ADEs and 86 potential ADEs were identified by the three detection methods, so the crude ADE rate was 28.1 ADEs per 1,000 patient-days (▶). After adjusting for our sampling scheme, which included a disproportionately high fraction of intensive care unit patient-days, the hospital-wide rate was 21.0 ADEs per 1,000 patient-days.
Table 1.
Number and Ratio of Adverse Drug Events (ADEs) Detected by Each Method
|
Computer Monitor
|
Chart Review
|
Voluntary Report
|
Totals* |
|
|---|---|---|---|---|
| No. (Ratio) | No. (Ratio) | No. (Ratio) | No. (Ratio) | |
| ADEs | 275 (9.6) | 398 (13.3) | 23 (0.7) | 617 (21.0) |
| Preventable ADEs | 70 (2.3) | 109 (3.6) | 9 (0.4) | 166 (5.6) |
|
Potential ADEs
|
2 (0.1)
|
23 (0.9)
|
61 (2.9)
|
86 (4.0)
|
| Note: The ratios are the number of events per 1,000 patient days. They were adjusted for the sampling scheme to reflect the rate for all beds in the hospital. | ||||
The totals do not equal the sum of ADEs detected by individual methods, because many events were identified by more than one method.
The computer monitor strategy detected 275 ADEs and only 2 potential ADEs during the study period. The adjusted rate for the monitor strategy was 9.6 per 1,000 patient-days. Over the same period, the chart review method captured 398 ADEs and 23 potential ADEs for an adjusted ADE rate of 13.3 per 1,000 patient-days. Voluntary reporting identified 23 ADEs and 61 potential ADEs with an associated adjusted rate of 0.7 ADEs per 1,000 patient-days.
Only 76 (12 percent) of the 617 ADEs were detected by both chart review and computer monitor, whereas 3 (0.5 percent) were identified by both computer monitor and voluntary report (▶). There were 281 severe ADEs, of which 139 (49 percent) were identified by the computer monitor, 169 (60 percent) were identified by chart review, and 11 (4 percent) were identified by voluntary report. Although chart review detected more severe events than other methods, a larger fraction of the ADEs detected by the computer monitor were classified as severe compared with those found by chart review (51 versus 42 percent, p =.04). Of the ADEs detected by voluntary report, 48 percent were severe. Of the 166 preventable ADEs discovered during the study, 70 (42 percent) were identified by the computer monitor, 109 (66 percent) were identified by chart review, and nine (5 percent) were captured by voluntary report. There were nonsignificant trends toward finding more preventable ADEs using both chart review and voluntary report (27 percent for chart review versus 23 percent for computer monitor [p =.16], and 39 percent for voluntary report versus 22 percent for computer monitor [p =.07]).
Table 2.
Preventability of Adverse Drug Events (ADEs) Detected by Each Method or by Multiple Methods
| No. by Any Method |
Computer Monitor
|
Chart Review
|
Voluntary Report
|
Computer Monitor and Chart Review
|
Computer Monitor and Voluntary Report
|
|
|---|---|---|---|---|---|---|
| No. (%) | No. (%) | No. (%) | No. (%) | No. (%) | ||
| All ADEs | 617 | 275 (45) | 398 (65) | 23 (4) | 76 (12) | 3 (1) |
| Non-preventable ADEs: | 451 | 205 (45) | 289 (64) | 14 (3) | 54 (12) | 3 (1) |
| Not severe | 288 | 121 (48) | 196 (68) | 9 (3) | 36 (13) | 2 (1) |
| Severe | 163 | 84 (52) | 93 (57) | 5 (3) | 18 (11) | 1 (1) |
| Preventable ADEs: | 166 | 70 (42) | 109 (66) | 9 (5) | 22 (13) | 0 (0) |
| Not severe | 48 | 15 (31) | 33 (69) | 3 (6) | 3 (6) | 0 (0) |
| Severe | 118 | 55 (47) | 76 (65) | 6 (5) | 19 (16) | 0 (0) |
The types of events detected by the chart review were substantially different from the types captured by the monitor (▶). Chart review was more effective than the computer monitor at detecting events manifested primarily by symptoms. For example, chart review more frequently identified change in mental status: Of the 125 cases of change in mental status, 93 (74 percent) were identified by chart review, whereas only 44 (35 percent) were captured by the computer monitor. Similarly, chart review more frequently captured cases of nausea and vomiting (44 for chart review versus 7 for computer monitor), rigors (21 for chart review versus 5 for computer monitor) and hypotension (35 for chart review versus 21 for computer monitor).
Table 3.
Types of Adverse Drug Events (ADEs) Detected by Each Method
|
Computer Monitor
|
Chart Review
|
Voluntary Report
|
Total No. | |
|---|---|---|---|---|
| No. (%) | No. (%) | No. (%) | ||
| Change in mental status | 44 (35) | 93 (74) | 4 (3) | 125 |
| Allergic reaction | 50 (50) | 74 (73) | 4 (4) | 101 |
| Hypotension | 21 (41) | 35 (69) | 0 (0) | 51 |
| Nausea and/or vomiting | 7 (14) | 44 (90) | 0 (0) | 49 |
| Bleeding | 19 (56) | 15 (44) | 1 (3) | 34 |
| Arrhythmias | 26 (76) | 13 (38) | 2 (6) | 34 |
| Diarrhea | 25 (78) | 11 (34) | 0 (0) | 32 |
| Renal failure | 30 (94) | 4 (13) | 0 (0) | 32 |
| Rigors | 5 (22) | 21 (91) | 0 (0) | 23 |
| Respiratory failure | 10 (59) | 7 (41) | 2 (12) | 17 |
| Hypoglycemia | 14 (88) | 7 (44) | 0 (0) | 16 |
| Flushing | 3 (38) | 5 (63) | 1 (13) | 8 |
| Ileus | 0 (0) | 6 (100) | 0 (0) | 6 |
| Other† |
21 (24)
|
63 (71)
|
9 (10)
|
89
|
|
Total‡ |
275 (45)
|
398 (65)
|
23 (4)
|
617
|
| * The totals in this column do not equal the sum of events detected by individual methods, because many ADEs were identified by more than one method. | ||||
All other ADEs, including paresthesias, weakness, and fever.
The percentages do not add up to 100 because many ADEs were identified by more than one method.
The computer more reliably identified events associated with changes in laboratory values, such as renal failure (30 for computer monitor versus 4 for chart review) and hypoglycemia (14 for computer monitor versus 7 for chart review). Somewhat surprisingly, the monitor was also more effective at capturing cases of diarrhea (25 for computer monitor versus 11 for chart review), mainly because these events usually had an order for an anti-diarrhea medication, which the monitor was more reliable at identifying.
Admissions Caused by ADEs
In addition to the events reported above, during the study period the computer monitor detected 87 admissions to the hospital due to an ADE. Fifteen (17 percent) of these events were allergic reactions, 15 (17 percent) were from altered mental status, and 14 (16 percent) were due to bleeding. Twenty-one of these 87 ADEs (24 percent) were preventable and 69 (79 percent) were classified as severe.
Evaluation of Individual Rules
The computer ADE monitor's 52 rules generated 2,620 alerts. Among these, 450 alerts were found to be associated with 363 ADEs. The overall positive predictive value of the ADE monitor rules was 17 percent.
Of the 2,620 alerts, “nephrotoxin and rise in serum creatinine” generated the largest number of alerts. There were 375 such alerts, representing 14 percent of all the alerts. Among these, 65 were associated with 43 ADEs (positive predictive value, 17 percent). Of our original rules, the most efficient was for an order for 50-percent dextrose, which generated 100 alerts (4 percent of all alerts), of which 27 were associated with 24 ADEs (positive predictive value, 27 percent). Most of these ADEs were instances of hypoglycemia due to insulin or other hypoglycemic agents, but some were cases of hyperkalemia treated with insulin and 50-percent dextrose.
The positive predictive value of the individual computer rules varied from 9 to 28 percent (▶). Because the positive predictive value of some rules was initially low, we made a number of changes to the rules during the study. As noted previously, the diphenhydramine rule was changed twice. First, the exclusion of diphenhydramine orders associated with paclitaxel increased the positive predictive value of the rule from 7 to 9 percent. Next, the exclusion of the diphenhydramine orders associated with a transfusion further increased the positive predictive value to 23 percent. Despite these changes, the ADE capture rate for the diphenhydramine rule remained stable; it was originally 1.7 per 1,000 patient-days and later, after the paclitaxel and transfusion exclusions, was 1.8 per 1,000 patient days.
Table 4.
Frequency and Positive Predictive Value of Adverse Drug Event (ADE) Monitor Rules
| No. Alerts (N) | No. Alerts Associated with ADEs (A) | Positive Predictive Value (A/N) | |
|---|---|---|---|
| Nephrotoxin and rise in creatinine* | 375 | 65 | 17 |
| Prednisone | 313 | 27 | 9 |
| Diphenhydramine | 300 | 42 | 14 |
| Allergy entered | 113 | 21 | 19 |
| Hydrocortisone | 154 | 16 | 10 |
| Prednisone and diphenhydramine | 111 | 14 | 13 |
| Dextrose 50% | 100 | 27 | 27 |
| Sodium polystyrene sulfate | 92 | 11 | 12 |
| Oral metronidazole or vancomycin | 87 | 14 | 16 |
| Serum potassium > 6.5 mmol/L | 66 | 8 | 12 |
| Serum digoxin > 1.7 ng/mL | 61 | 14 | 23 |
| Vitamin K and previous warfarin | 60 | 17 | 28 |
| Serum vancomycin >50 mg/L | 54 | 10 | 19 |
| Triamcinolone and beta-blocker | 51 | 5 | 10 |
| Others |
683
|
161
|
23
|
| Total | 2620 | 450 | 17 |
See footnote to appendix (p. 313) for list of drugs included as nephrotoxins.
We also modified the vitamin K rule during the study to improve its positive predictive value for detecting possible warfarin overdose. Initially, the rule was triggered simply by an order for vitamin K. In the first month of the study, this rule generated 32 alerts, 4 (12.5 percent) of which were determined to be ADEs. We changed the rule to be true only if there was also a previous or current order for warfarin. During the remaining seven months of the study, there were 28 alerts, 13 (46 percent) of which were associated with an ADE. Although there was a substantial improvement in the positive predictive value of the rule, the capture rate of the rule decreased (from 2.0 ADEs per 1,000 patient days to 0.7 ADEs per 1,000 patient days), because the rule no longer identified cases of vitamin K depletion due to diarrhea from medications, which had represented two of the four initial ADEs. However, we were able to capture some of these cases by other rules that identified orders for anti-diarrhea medications.
Overall, during the study, we changed 9 of the 49 original rules in the ADE monitor rule base, resulting in substantial improvements in the rules' positive predictive values, and we added three new rules, allowing us to detect additional events. During the first eight weeks of the study, the ADE monitor had a positive predcictive value of 16 percent and a crude event capture rate of 16.6 ADEs per 1,000 patient days. During the final eight weeks of the study, the positive predictive value increased to 23 percent and the capture rate increased slightly to 17.8 ADEs per 1,000 patient days.
Comparison of Work Required
The ADE monitor required a total of 11 person-hours per week to detect ADEs on the nine study floors. Generating the list of computer alerts required one person-hour per week, and ten person-hours per week were needed to review the associated charts. In contrast, the chart review and voluntary report methodologies together required 60 person-hours per week to track all the patients on the study floors and review their charts. Fifty-five hours were spent on manual chart review and five hours on receiving voluntary report and documenting the cases. Overall, the monitor detected 69 percent as many cases as chart review but required only 20 percent as much time. Entering the rules for the ADE monitor required two programmer-weeks, for an approximate cost of $2,000. The entire event monitor project required two person-years to build, but the ADE monitor represents only a minor proportion of what the event monitor does. Ongoing maintenance requires 1 to 2 programmer-hours per month.
Discussion
Chart review and the computer monitor were both effective at capturing large numbers of ADEs, while stimulated voluntary report detected only a small number of these events. However, voluntary report was much more effective at detecting potential ADEs than either chart review or the computer monitor. There was surprisingly little overlap between events found by these detection methods, especially between the computer monitor and chart review. The monitor was particularly good at identifying events associated with quantitative changes, such as renal failure, whereas chart review was better at identifying events associated only with symptoms such as change in mental status. Adverse drug events identified by the monitor tended to be more severe, whereas those detected by chart review were more often preventable. The computer monitor strategy was the most efficient, because it required substantially fewer hours than chart review and detected many more ADEs than voluntary report.
Classen et al.18 found a rate of 2.0 ADEs per 100 hospital admissions using their computer monitor supplemented by voluntary report. Our ADE rate detected by the monitor strategy was 4.1 ADEs per 100 admissions. The ADEs detected by Classen et al. included few mild ADEs, whereas our ADE monitor found substantial numbers of such events, which probably accounts for some of the difference in rates.
Another issue is the degree of overlap between methods, which was smaller than expected. There is some precedent for finding relatively little overlap in adverse events detected by different methodologies. O'Neil et al.25 compared chart review with stimulated physician voluntary report for adverse events and found that both strategies detected a similar number of events but that the overlap was only 20 percent. These differences could be related to problems with interrater reliability as to whether events are present for specific incidents. However, using the same methodology as in this study, we previously found that once incidents were identified, interrater reliability regarding whether an ADE was present was high, with kappa values of 0.81 and 0.71.2,17 Thus, it appears that disagreement about the presence of an ADE is not the issue. Instead, the cause of the low overlap was that many events were missed by each of the detection methodologies. To better understand this, we further examined events that were found by one method but not by the others.
In these analyses, we found that chart review was more effective at identifying events in which the patient had significant symptoms, but it was much less so at identifying events in which the main effect was laboratory abnormalities. Chart review missed many cases associated with laboratory abnormalities because of lack of attribution or poor documentation. For example, patients receiving a nephrotoxin in addition to many other medications had rises in serum creatinine without another cause of renal failure, but the events were not explicitly noted as changes in renal function in the chart. Other patients received naloxone for narcotic overdose or 50-percent dextrose for hypoglycemia but had no progress notes related to these events. Although chart review was more effective at identifying changes in mental status due to benzodiazepines or narcotics when no antidote was given, it often failed to capture changes in mental status due to toxic levels of anti-epileptic medications. However, because of the laboratory abnormalities associated with these events, the computer monitor's efficacy in sifting through large amounts of clinical data enabled it to detect these types of events.
Many ADEs detected by chart review were missed by the computer monitor. The most common reason was lack of need for treatment or testing beyond cessation of the medication. For example, patients given narcotics who become obtunded often recover without requiring naloxone. Such ADEs, which are associated with symptoms alone, currently could be detected by the computer monitor only if physicians enter such events into the patient's computer database as an allergy, intolerance, or adverse reaction. To our surprise, only 14 percent of new drug allergic reactions (rash, hives, anaphylaxis) that were detected by the chart review study were entered into the computer by physicians. It is essential to improve this rate, because future allergic reactions cannot be prevented if allergy data are not entered consistently into the computer. This illustrates the point that computer-based checks are only as effective as the data they contain.
Our ability to implement certain potential rules was limited by the accessibility of the clinical data in our patient database. For example, the rule editor was not able to easily access microbiology data, so results of bacterial cultures or assays could not be obtained. Therefore, we could not create a rule that looked for a positive Clostridium difficile toxin assay. Fortunately, most of the patients thought to have pseudomembranous colitis are treated with oral metronidazole or oral vancomycin, so we were able to capture many of these events by a rule that detected orders for the oral administration of either of these medications. Although the rule lacked the specificity of a positive Clostridium difficile toxin assay, it nevertheless allowed us to detect six ADEs during the study.
Because we are continually increasing the amount of clinical information available in coded form, we were able to develop new rules during the study period. For instance, we developed the ability to detect new patient allergies entered into the computer system. Since patients normally have previous medication allergies entered at the time of admission, we excluded all allergies entered into the computer on the day of the patient's hospitalization. Although this approach missed allergic reactions that occur on the day of admission or that cause an admission, it allowed us to avoid reviewing the large number of allergy orders that represented past allergic events. During the 19 weeks that this rule was in effect, we were able to detect 18 ADEs that otherwise would have been missed.
Developing the ADE monitor required substantial effort to refine previously published rules. We found that continuous monitoring and improvement of the rules was vital to maintaining efficient and sensitive rules. With the addition of more coded medical information, further improvement in the positive predictive value and sensitivity of the rules should increase the efficiency of computer-based strategies. Specifically, the ability to evaluate electronic provider notes should substantially improve the effectiveness of the computer monitor; according to D. Classen (personal communication; August 1997), the current version of the LDS computer monitor identifies many events using these data. As clinical practices change, with the introduction of new drugs and changes in the use of existing ones, the computer monitor will have to be updated continuously to remain effective. It will be essential to have computer rule editors that allow the rules to be maintained with minimal programming effort.
The ADE monitor-based strategy required substantially fewer person-hours than did chart review. Thus, we feel it is the most practical method for ongoing quality assessment. It can be supplemented by encouraged voluntary report (possibly facilitated by online data entry), which identifies many potential ADEs. An effective strategy for ADE detection might entail a pharmacist reviewing all records of patients with computer-generated alerts and voluntarily reported events, as is done at LDS Hospital.18 To cover our entire 726-bed hospital, such a strategy would require 33 person-hours per week, a substantial but not unreasonable commitment. Few hospitals currently devote this great an effort to the detection of ADEs, but given the magnitude of the problem, we feel such effort is justified. The Joint Commission on Accreditation of Healthcare Organizations asks hospitals to monitor their rates of adverse events. Computer-based monitors are probably the most practical tools for tracking ADEs with only modest resource consumption.
This study has several limitations. First, since there is no independent “gold standard,” we can only estimate the number of ADEs missed by the monitor and how representative the detected events are of all the ADEs that occurred on the study floors. Second, the results reported here represent an undercount because certain features, such as the allergy-entry program, became available only during the study period. Finally, because practice patterns differ among institutions, the capture rate and positive predictive value of specific rules may be, to some extent, idiosyncratic to our hospital. As other institutions utilize ADE monitors, issues about generalizing these data will have to be addressed.
We have presented the development and evaluation of a computer-based ADE monitor, which was based on similar monitors created by others. The monitor was effective at identifying ADEs and compared favorably with chart review and voluntary report methods, particularly in terms of effort per event identified. The types of events identified by the computer-based monitor differed from those identified by chart review; many events could not be found by the monitor given the current level of coding of information. These data suggest that ADEs may be more common than indicated by previous studies, including our own,2 that used only one method for detecting events. As new clinical information (drug administration records, vital signs) becomes available in coded form, many of the events currently missed by the computer monitor should become accessible. The development process required considerable attention to optimize the positive predictive value, sensitivity, and specificity of the rules. Changes to the monitor's rules will continue as more clinical information becomes available. Since they represent a highly efficient strategy for identifying ADEs, computer-based ADE monitors seem likely to become the primary strategy for tracking these serious and costly events.
Acknowledgments
The authors thank William W. Churchill, MS, RPh, and Priscilla Dasse and the Risk Management Foundation for their support and encouragement.
Appendix
Rule Base of the Adverse Drug Event (ADE) Monitor
Receiving diphenhydramine AND no diphenhydramine within last 7 days AND patient not on paclitaxel AND no blood transfusion in last 1 day.?
Receiving “nephrotoxin” AND blood creatinine has risen > 0.5 mg/dL in last 1 day
Receiving atropine sulfate
Receiving betamethasone dipropionate 0.05%
Receiving oral metronidazole or oral vancomycin
Receiving charcoal (activated)
Receiving dextrose 50% in water
Blood alkaline phosphatase > 350 U/L
Receiving racemic epinephrine hcl
Receiving phytonadione (vitamin K) AND order for warfarin within last 14 days
Receiving protamine sulfate
Receiving ranitidine AND platelet count has fallen to less than 50% of previous value
Serum carbamazepine > 12.0 μg/mL
Serum digoxin > 1.7 ng/mL
Serum amikacin results > 25 mg/L
Receiving benzodiazepine AND receiving anti-epileptic
Serum bilirubin > 10 mg/dL
Receiving calamine lotion
Serum cyclosporine > 500 μg/L
Receiving Digibind
Receiving diphenoxylate with atropine
Receiving flumazenil
Receiving glucagon
Receiving naloxone
Receiving fluocinolone acetonide
Serum potassium > 6.5 mmol/L
Serum lidocaine > 5.0 μg/mL
Blood eosinophils > 6%
Receiving kaopectate
Receiving loperamide
Receiving opium and belladonna
Serum n-acetyl procainamide > 20 μg/mL
Receiving opium tincture deodorized
Serum phenytoin results within last 1 day are > 20 μg/mL
Serum phenobarbital results within last 1 day are < 45 μg/mL
Receiving prednisone AND diphenhydramine
Receiving prednisone AND no prednisone AND no solumedrol within last 7 days
Serum procainamide > 10 μg/mL
Serum aspartate amino transferase > 150 U/L AND no prior result > 150 U/L in last 7 days
Serum theophylline > 20 μg/ML
Serum tobramycin > 10 mg/L
Serum valproate > 120 μg/mL
Serum quinidine > 5 μg/mL
Receiving Diprolene 0.05%
Serum gentamicin > 10 mg/L
Receiving hydrocortisone AND no hydrocortisone within last 7 days
Receiving triamcinalone and a beta-blocker
Receiving prednisone AND receiving epinephrine
Serum alanine aminotransferase > 150 U/L AND no result > 150 U/L in last 7 days
Receiving sodium polystyrene sulfonate
Serum vancomycin > 50 mg/L
This work was supported by research grant R01-HS07107-01 from the Agency for Health Care Policy and Research, by a grant from the Harvard Risk Management Foundation, and by the Carl W. Walter Fund of Harvard Medical School.
Footnotes
Nephrotoxins were aminoglycosides, angiotensin converting enzyme (ACE) inhibitors, acyclovir, amphotericin B, carboplatin, cisplatin, cyclosporine, foscarnet, ifosfamide, nonsteroidal anti-inflammatory agents, pentamidine, and vancomycin.
References
- 1.Leape LL. Error in medicine. JAMA. 1995;272: 1851-7. [PubMed] [Google Scholar]
- 2.Bates DW, Cullen D, Laird N, et al. Incidence of adverse drug events and potential adverse drug events: implications for prevention. JAMA. 1995;274: 29-34. [PubMed] [Google Scholar]
- 3.Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. JAMA. 1997;227: 307-11. [PubMed] [Google Scholar]
- 4.Brennan TA, Leape LL, Laird N, et al. Incidence of adverse events and negligence in hospitalized patients: results from the Harvard Medical Practice Study I. N Engl J Med. 1991;324: 370-6. [DOI] [PubMed] [Google Scholar]
- 5.Leape LL, Brennan TA, Laird NM, et al. The nature of adverse events in hospitalized patients: results from the Harvard Medical Practice Study II. N Engl J Med. 1991;324: 377-84. [DOI] [PubMed] [Google Scholar]
- 6.Faich GA. National adverse drug reaction reporting, 1984-89. Arch Intern Med. 1991;151: 1645-7. [PubMed] [Google Scholar]
- 7.Faich GA. Adverse-drug-reaction monitoring. N Engl J Med. 1986;314: 1589-92. [DOI] [PubMed] [Google Scholar]
- 8.Melmon KL. Preventable drug reactions: causes and cures. Semin Med Beth Israel Hosp. 1971;284: 1361-7. [DOI] [PubMed] [Google Scholar]
- 9.Keith MR, Bellanger-McCleery RA, Fuchs JE Jr. Multidisciplinary program for detecting and evaluating adverse drug reactions. Am J Hosp Pharm. 1989;46: 1809-12. [PubMed] [Google Scholar]
- 10.Seidl LG, Thornton G, Smith JW, et al. Studies on the epidemiology of adverse drug reactions, III: reactions in patients on a general medical service. Bull Johns Hopkins Hosp. 1966;119: 299-315. [Google Scholar]
- 11.Jick H. Adverse drug reactions: the magnitude of the problem. J Allergy Clin Immunol. 1984;74: 555-7. [DOI] [PubMed] [Google Scholar]
- 12.Ogilvie RI, Ruedy J. Adverse drug reactions during hospitalization. Can Med Assoc J. 1967;97: 1450-7. [PMC free article] [PubMed] [Google Scholar]
- 13.Hurwitz N, Wade OL. Intensive hospital monitoring of adverse reactions to drugs. Br Med J. 1969;1: 531-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Berry LL, Segal R, Sherrin TP, Fudge KA. Sensitivity and specificity of three methods of detecting adverse drug reactions. Am J Hosp Pharm. 1988;45: 1534-9. [PubMed] [Google Scholar]
- 15.Karch FE, Lasagna L. Adverse drug reactions: a critical review. JAMA. 1975;234: 1236-41. [PubMed] [Google Scholar]
- 16.International drug monitoring: the role of the hospital. World Health Organ Tech Rep Ser. 1969;425: 5-24. [PubMed] [Google Scholar]
- 17.Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med. 1993;8: 289-94. [DOI] [PubMed] [Google Scholar]
- 18.Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients. JAMA. 1991;266: 2847-51. [PubMed] [Google Scholar]
- 19.Grasela TH, Walawander CA, Kennedy DL, Jolson HM. Capability of hospital computer systems in performing druguse evaluations and adverse drug event monitoring. Am J Hosp Pharm. 1993;50: 1889-95. [PubMed] [Google Scholar]
- 20.Teich JM, Hurley JF, Beckley RF, Aranow M. Design of an easy-to-use physician order entry system with support for nursing and ancillary departments. Proc 16th Annu Symp Comput Appl Med Care. 1992: 99-103. [PMC free article] [PubMed]
- 21.Kuperman GJ, Teich JM, Bates DW, McLatchey J, Hoff TG. Representing hospital events as complex conditionals. Proc 19th Annu Symp Comput Appl Med Care. 1995: 137-41X. [PMC free article] [PubMed]
- 22.Folli HL, Poole RL, Benitz WE, Russo JC. Medication error prevention by clinical pharmacists in two children's hospitals. Pediatrics. 1987;79: 718-22. [PubMed] [Google Scholar]
- 23.Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. JAMA. 1995;274: 35-43. [PubMed] [Google Scholar]
- 24.SAS Institute Inc., Release 6.03. NC: Cary, 1988:
- 25.O'Neil AC, Petersen LA, Cook EF, Bates DW, Lee TH, Brennan TA. Physician reporting compared with medical-record review to identify adverse medical events. Ann Intern Med. 1993;119: 370-6. [DOI] [PubMed] [Google Scholar]