A Systematic Review of the Performance Characteristics of Clinical Event Monitor Signals Used to Detect Adverse Drug Events in the Hospital Setting

Abstract

Objective

We conducted a systematic review of pharmacy and laboratory signals used by clinical event monitor systems to detect adverse drug events (ADEs) in adult hospitals.

Design and Measurements

We searched the MEDLINE, CINHAL, and EMBASE databases for the years 1985–2006, and found 12 studies describing 36 unique ADE signals (10 medication levels, 19 laboratory values, and 7 antidotes). We were able to calculate positive predictive values (PPVs) and 95% confidence intervals (CIs) for 15 signals.

Results

We found that PPVs ranged from 0.03 (95% CI, 0.03–0.03) for hypokalemia, to 0.50 (95% CI, 0.39–0.61) for supratherapeutic quinidine level. In general, antidotes (range = 0.09–0.11) had the lowest PPVs, followed by laboratory values (range = 0.03–0.27) and medication levels (range = 0.03–0.50).

Conclusion

Data from this study should help clinical information system and computerized decision support producers develop or improve existing clinical event monitor systems to detect ADEs in their own hospitals by prioritizing those signals with the highest PPVs.

Introduction and Background

Clinical decision support (CDS) systems have been shown to improve patient care and treatment outcomes by providing physicians and other health care providers with patient-specific information that is intelligently filtered and presented at appropriate times. ¹ Clinical event monitors, one of the most common types of CDS systems, provide feedback through alerts and reminders to health care providers when triggered by certain information available in electronic format (i.e., by signals). ² Clinical event monitors can be used to detect medication-related problems by processing pharmacy order signals ^{3, 4} and laboratory test result signals, ⁵ generated by systems with varying levels of automation and sophistication. ⁶

The most clinically significant medication-related problems are adverse drug events (ADEs). Various definitions have been proposed and used throughout the literature to describe ADEs. For this paper, we use the Institute of Medicine definition which defines ADEs as “injuries resulting from a medical intervention related to a drug.” ^7–9 ADEs are common and occur in 2.4–5.2 per 100 hospitalized adult patients. ^10–13 A meta-analysis of fatal ADEs suggest that these events are between the fourth and sixth leading causes of death in the United States. ¹⁴ Each ADE is estimated to increase the length of hospital stay by 2.2 days and to increase the hospital cost by $3,244. ¹⁵

Compared with manual methods of ADE detection (e.g., chart review or voluntary reporting), clinical event monitors are less expensive and faster, and they often identify ADEs not normally detected by clinicians during the course of routine hospital care. ^16–19 Through the early detection and prevention of ADEs, clinical event monitors can improve the quality of care, while reducing health care costs by as much as $760,000 per year in a teaching hospital. ^20–23 Despite the potential benefits of clinical event monitors and the fact that several prominent national organizations have recommended their use to detect ADEs, ^24,25 few health care systems have implemented them. ²⁶ Moreover, when they have implemented them, they have done so in non-standardized ways that make it difficult to compare and synthesize the results. ^9,27 The lack of generalizability of results in turn contributes to the problems and suboptimal performance of hospitals in the U.S. health care system. ¹

To begin to address these concerns and to help clinical information system and CDS producers develop, select, or improve systems to detect ADEs, we conducted a systematic review of individual pharmacy and laboratory signals that are currently used by clinical event monitors to detect ADEs in the adult hospital setting. When possible, we calculated the positive predictive values (PPVs) of individual signals.

Methods

Study Identification and Eligibility

Before we implemented our literature search, we established criteria for inclusion and exclusion of studies. We included studies that met the following four criteria: their results were published between January 1, 1985, and July 1, 2006; they described a clinical event monitoring system to detect ADEs in an adult hospital setting; they described laboratory or pharmacy ADE signals; and they provided PPVs or information to allow the calculation of PPVs for individual ADE signals. We excluded studies if they focused on ADE prevention rather than detection (e.g., if they focused on computerized physician order entry systems) as this has recently been reviewed elsewhere. ²⁸ We also excluded studies if they described non-laboratory or non-pharmacy ADE signals, including signals to monitor physiologic data (e.g., blood pressure or heart rate) or administrative data (e.g., diagnostic or procedural codes [ICD-9 or CPT]), or if they described free-text search strategies to detect potential ADEs. Because of concerns that non–peer-reviewed data might introduce bias into our systematic review, ^29,30 we also excluded studies in which data was presented as an abstract, poster presentation, or editorial.

Information Sources and Search Strategy

We searched OVID MEDLINE, OVID CINHAL, and EMBASE for articles published in all languages between January 1, 1985, and July 1, 2006. In OVID, we searched for the following medical subject headings (MeSH) keywords, and text words: adverse drug event, adverse drug reaction, adverse drug reaction reporting systems, clinical event monitor, clinical decisions support systems, clinical laboratory information systems, clinical pharmacy information system, computer generated signals, decision support system, drug monitoring, medication errors, and physiologic monitoring. In EMBASE, we searched for the above terms plus the following EMTREE keywords: computer assisted drug therapy and drug surveillance program. We supplemented the computerized search by reviewing the reference lists of all articles selected for inclusion.

Study Selection, Data Extraction, and Review Criteria

Two reviewers (SMH and RLA) independently assessed each article for eligibility criteria, with adjudication by a third reviewer (JTH) in cases of disagreement. While reviewing each study that met the eligibility criteria, the same two authors (SMH and RLA) used standardized forms to independently extract and record: hospital characteristics (e.g., teaching or community hospital, number of beds); patient characteristics (e.g., number of patients included); the signals monitored by the hospitals; and, data necessary to record or calculate positive predictive values. To collect the necessary data to calculate a PPV, we reviewed the data from each signal in the individual included studies. For every signal in an included study, we recorded the number of times that a specific signal fired and the number of times that a health professional determined that the signal represented an ADE. Study authors were contacted by e-mail for data clarification when necessary.

Signals from each of the studies that met eligibility criteria were included and combined if they measured the same parameter (e.g., digoxin level, serum potassium level, or use of vitamin K) independent of the reference interval or dosage used in the particular study. Signals were then grouped into one of three categories: antidote signals (triggered by administration of medications given to counteract the effects of a poison, toxin, or other agent with toxic effects), medication level signals (triggered by elevated or supratherapeutic drug levels), and laboratory result signals (triggered by abnormal values in blood tests).

Quantitative Data Synthesis and Statistical Analysis

To calculate a study-specific PPV for each signal, we divided the number of times that a signal fired and an ADE was confirmed (i.e., the number of true-positives), by the number of times the signal fired with or without an ADE being confirmed (i.e., the sum of true-positives and false-positives). PPVs were chosen as the performance characteristic of interest since the majority of studies conducted a targeted verification of signal firings and did not include a corollary gold-standard measure, such as an independently conducted chart review looking for the presence of ADEs. As a result, the sensitivity and specificity of individual signals used to detect ADEs could not be calculated.

To determine the appropriateness of computing a pooled PPV, we compared the individual study-specific PPVs using the chi-square test for homogeneity of proportions. ³¹ For those signals for which there was no evidence of heterogeneity (p > 0.05) we calculated an overall estimate of pooled PPVs and corresponding 95% confidence intervals (CIs). We used a generalized estimating equations (GEE) model by combining the PPVs for signals reported in at least two studies. This model included an exchangeable correlation structure to account for within-study correlation, using the total number of signal firings in each study as the weighing factor. ^32–34 We also examined the sensitivity of the overall PPV estimates using a fixed effects model recommended in the meta-analytic literature. ³⁵

To determine whether certain studies were heavily influencing the overall PPV estimate for each signal, we performed an influence analysis in which we excluded studies, one at a time, and reestimated the overall PPVs. We also examined the cumulative effect on the overall PPV estimate by adding studies, one at a time, ordered by year of publication, and hospital bed size. If there were any publication bias, it would most likely be caused by the greater probability of publication of studies with a larger number of firings or of studies with a smaller number of firings but a greater PPV. We examined this possibility by visually inspecting a scatter plot of the PPV and the square root of the number of signals (which is proportional to the reciprocal of the standard error) and testing for a significant linear trend between them. If we found a lack of data points near the origin or a statistically significant negative linear trend, we would consider it to be evidence of publication bias. ³⁶ We conducted all statistical analyses with either SAS version 8.2 for Windows (SAS Institute, Inc., Cary, NC) or Stata version 9.0 for Windows (StataCorp, LP, College Station, TX).

Results

Of the 6,649 titles that were initially identified, 4,243 were from MEDLINE, 859 were from CINHAL, and 1,547 were from EMBASE. After removing duplicates and going through a thorough screening process (▶), we identified 12 observational studies that met our eligibility criteria. 18, 37–48 ▶ lists the 12 studies and the characteristics of the study sites. All but two of the studies were conducted in teaching hospitals.

Figure 1.

Flow diagram of included and excluded studies.

Table 1.

Table 1 Characteristics of Studies Included in the Systematic Review

Author/Year/Reference	Study Site
Evans et al., 199137	500-bed tertiary teaching hospital
Azaz-Livshits et al., 199838	34 bed medical ward in teaching hospital
Jha et al., 199839	726 bed tertiary teaching hospital
Raschke et al., 199846	650 bed community teaching hospital
Levy et al., 199918	34 bed medical ward in teaching hospital
Dormann et al., 200041	9 bed medical ward in a teaching hospital
Brown et al., 200048	238 bed Veterans Administration Medical Center
Jha et al., 200142	726 bed tertiary care teaching hospital
Thuermann et al., 200243	86 bed neurology department in teaching hospital
Dormann et al., 200444	29 bed gastroenterology ward in teaching hospital
Silverman et al., 200447	726 bed tertiary care teaching hospital
Hartis et al., 200545	1,952 beds in six community hospitals

Of the total 36 signals that we identified in two or more publications and included in our analysis, 7 were administrations of antidotes, 10 were supratherapeutic medication levels, and 19 were abnormal laboratory test results. Fifteen signals (three antidotes, eight laboratory tests, and four medication levels) contained no evidence of heterogeneity (p > 0.05) and were pooled to calculate overall PPVs and 95% CIs. Naloxone was not included in the analysis because of the 12 studies that met eligibility criteria, only one study provided sufficient information about naloxone to calculate PPVs. ³⁷ Because we could not calculate a pooled PPV (our primary unit of analysis) with the PPV from only one study, naloxone was not included in our systematic review.

Of the antidote signals (▶), sodium polystyrene administration, had the lowest pooled PPV 0.09 (95% CI, 0.06–0.13), and metronidazole or vancomycin administration had the highest 0.11 (95% CI, 0.06–0.20). Of the laboratory test result signals (▶), hypokalemia had the lowest pooled PPV 0.03 (95% CI, 0.03–0.03), and hypoglycemia had the highest 0.28 (95% CI, 0.24–0.32). Of the medication level signals (▶), cyclosporine had the lowest pooled PPV 0.03 (95% CI, 0.02–0.06) and quinidine had the highest 0.50 (95% CI, 0.39–0.61). Among the pooled signals considered, the antidote category had the lowest PPVs (range = 0.09–0.11), followed by the laboratory test result category (range = 0.03–0.27), and the medication level category (range = 0.03–0.50).

Table 2.

Table 2 Signals Associated with Antidotes ^∗

Signal	Number of Studies	PPV Range	p-value Test for Heterogeneity	Overall Estimate of PPV† (95% CI)	Overall Estimate of PPV‡ (95% CI)
Vitamin K given	3	0.02–0.30	<0.01	—	—
Activated charcoal given	2	0.08–0.45	0.03	—	—
Antihistamine (e.g., diphenhydramine or hydroxyzine) given	3	0.03–0.14	<0.01	—	—
Oral metronidazole or vancomycin given	2	0.07–0.16	0.06	0.11 (0.06–0.20)	0.10 (0.06–0.14)
Antidiarrheal (e.g., loperamide, diphenoxylate, bismuth) given	3	0–0.11	0.06	0.09 (0.07–0.13)	0.07 (0.00–0.15)
Sodium polystyrene (Kayexalate®) given	3	0.06–0.12	0.44	0.09 (0.06–0.13)	0.08 (0.05–0.12)
Oral or topical steroids (e.g., prednisone, prednisolone) given	2	0.04–0.09	<0.01	—	—

^∗ Naloxone not included as data were available from only a single study.

^† PPV calculated using GEE pooled estimate and CI.

^‡ PPV calculated using fixed effects pooled estimate and CI.

PPV= positive predictive value.

Table 3.

Table 3 Signals Associated with Laboratory Test Results

Signal	Number of Studies	PPV Range	p-value Test for Heterogeneity	Overall Estimate of PPV ∗ (95% CI)	Overall Estimate of PPV† (95% CI)
Serum creatinine elevated or increasing	5	0.08–0.39	<0.01	—	—
Hypoglycemia (as indicated by low or decreasing glucose)	2	0–0.33	0.49	0.27 (0.27–0.27)	0.10 (0.00–0.27)
Hyperbilirubinemia (as indicated by high or increasing bilirubin)	4	0.05–0.39	<0.01	—	—
Hyponatremia (as indicated by low or decreasing sodium)	2	0.24–0.33	0.72	0.25 (0.23–0.28)	0.25 (0.09–0.41)
Blood urea nitrogen (BUN) elevated or increasing	3	0–0.30	0.41	0.22 (0.14–0.32)	0.17 (0.08–0.26)
Eosinophilia (as indicated by high or increasing eosinophils)	5	0–0.62	<0.01	—	—
Hyperkalemia (as indicated by high or increasing potassium)	5	0–0.67	<0.01	—	—
Alanine aminotransferase (ALT) elevated or increasing	3	0.12–0.38	<0.01	—	—
Anemia (as indicated by a low or decreasing hemoglobin/hematocrit)	5	0.12–0.30	0.14	0.19 (0.12–0.29)	0.16 (0.11–0.22)
Partial thromboplastin time (PTT) elevated or increasing	3	0.04–0.92	<0.01	—	—
Gamma-Glutamyl Transferase (GGTP) elevated or increasing	4	0.03–0.19	0.03	—	—
Alkaline phosphatase (ALP) level elevated or increasing	5	0–0.31	<0.01	—	—
Aspartate aminotransferase (AST) elevated or increasing	4	0.01–0.23	<0.01	—	—
Agranulocytosis or leukopenia (as indicated by low or decreasing white blood cells)	4	0.09–0.5	0.15	0.11 (0.07–0.17)	0.10 (0.04–0.15)
International normalized ratio (INR) elevated or increasing	4	0.05–1.0	<0.01	—	—
Lactate dehydrogenase (LDH) elevated or increasing	3	0.02–0.17	0.06	0.06 (0.02–0.14)	0.03 (0.00–0.06)
Thrombocytopenia (as indicated by low or decreasing platelets)	4	0.03–0.12	0.01	—	—
Hypocalcemia (as indicated by low or decreasing calcium)	2	0–0.11	0.25	0.06 (0.02–0.18)	0.02 (0.00–0.08)
Hypokalemia (as indicated by low or decreasing potassium)	2	0–0.03	0.86	0.03 (0.03–0.03)	0.03 (0.01–0.04)

^∗ PPV calculated using GEE pooled estimate and CI.

^† PPV calculated using fixed effects pooled estimate and CI.

PPV= positive predictive value.

Table 4.

Table 4 Signals Associated with Supratherapeutic Medication Levels

Signal	Number of Studies	PPV Range	P-value Test for Heterogeneity	Overall Estimate of PPV ∗ (95% CI)	Overall Estimate of PPV† (95% CI)
Quinidine	2	0.43–0.60	0.56	0.50 (0.39–0.61)	0.50 (0.22–0.78)
Phenobarbital	3	0–1.0	<0.01	—	—
Theophylline trough	5	0.25–1.0	0.01	—	—
Vancomycin peak or trough levels	3	0.18–0.33	0.31	0.26 (0.22–0.32)	0.26 (0.20–0.32)
Procainamide	3	0–0.42	<0.01	—	—
Lidocaine	3	0.17–0.50	0.51	0.19 (0.17–0.21)	0.18 (0.09–0.28)
Aminoglycoside antibiotic	3	0.04–1.0	<0.01	—	—
Digoxin	8	0.08–1.0	<0.01	—	—
Phenytoin	7	0.07–1.0	<0.01	—	—
Cyclosporine	2	0–0.04	0.29	0.03 (0.02–0.06)	0.03 (0.00–0.06)

^∗ PPV calculated using GEE pooled estimate and CI.

^† PPV calculated using fixed effects pooled estimate and CI.

PPV= positive predictive value.

There were no meaningful differences in overall PPV estimates calculated with GEE models or fixed effects models. The influence analysis suggested that the removal of certain studies affected the PPVs for particular signals. For example, when the Evans et al. study was removed from the analysis of the signal for agranulocytosis or leukopenia, the pooled PPV increased from 0.11 to 0.23. ³⁷ Similarly, when the Theurmann et al. study was removed from the analysis of the anemia signal, the PPV increased from 0.19 to 0.26. ⁴³ No effects were noted on the overall PPV estimates when stratified by study year or bed size.

Some evidence of publication bias was found for the signal agranulocytosis or leukopenia. Specifically, a significant negative association between the number of firings and the PPV (p < 0.05), suggested the possibility that smaller studies with lower PPVs may not have been published and may therefore have eluded our systematic review. For the remaining signals, we found no evidence of publication bias.

Discussion

This systematic review analyzed the performance characteristics of individual pharmacy and laboratory signals that are currently used by clinical event monitors to detect ADEs in the adult hospital setting. Our review of the PPVs of 36 signals from 12 studies published between 1985 and 2006 revealed two important findings.

First, there was evidence of significant between-study heterogeneity for the majority of signals, limiting our ability to pool the PPVs of signals across studies. Of the 36 signals identified in two or more publications, 21 contained evidence of heterogeneity and could therefore not be pooled to calculate overall PPVs. There are at least two plausible explanations for this heterogeneity. First, it may be due to the use of different reference intervals for therapeutic medication levels and laboratory values in different studies. Second, it may be attributable to the different hospital and/or patient characteristics which affect the underlying prevalence of ADEs. This is particularly important because PPVs are by definition affected by the underlying prevalence of the condition of interest.

The second important finding was that there was significant variability in the PPVs for different individual signals, both across studies and within signal categories (e.g., antidotes, medication levels, and laboratory test results). The overall PPV estimates for the 15 pooled signals in the analysis ranged from 0.03 for hypokalemia to 0.50 for a supratherapeutic quinidine level. Moreover, antidotes had the lowest PPVs, followed by laboratory test results, and medication levels. It is not surprising that PPVs were highest for medication levels. For this category of signal, the prior odds of an ADE are increased, since the underlying assumption is that patients in each case are already receiving the medication of interest and their prescribing clinicians are aware of the possibility of an ADE. ⁴⁹ In contrast, the other two categories of signals would not necessarily be expected to be associated with an ADE. Laboratory values are often abnormal because of the onset or worsening of medical conditions unrelated to the use of medications. Likewise, the majority of antidotes analyzed in our study can be used to treat multiple medical conditions, only a fraction of which are related to the presence of an ADE.

Limitations and Strengths

Our systematic review has several limitations that deserve mention. First, systematic reviews of effect sizes often limit their selection of studies to those involving randomized controlled trials (RCTs). ⁵⁰ However, analyzing RCTs is not always feasible or preferable for evaluating the performance characteristics of individual signals used to detect ADEs. ^51,52 For purposes of our analysis, we did not limit our systematic review to RCTs, so we were not able to apply instruments commonly used to assess the quality of RCTs. ^53,54 Second, although we found 12 studies that could be included in the overall analysis, we found few studies that covered each ADE signal. This may have limited our ability to identify the dependence of overall PPVs on factors such as facility bed size and to detect publication bias, a problem to which all systematic reviews are susceptible. ^55,56 Third, our analysis focused on data that is widely available in electronic format (such as laboratory and pharmacy information) and was thus biased against data that cannot be readily computed. It also excluded some sources of electronic data available to enhance ADE detection, such as administrative data (e.g., ICD-9 and CPT codes), allergy rules, and free-text searching of clinician progress and discharge notes. ⁵⁷

Despite these limitations, we believe that our results are important and represent the most comprehensive information available on the performance characteristics of ADE signals in the adult hospital setting. Our analysis employed the “best practice” methods recommended for conducting systematic reviews of the literature. ⁵⁰ Moreover, in keeping with suggestions of the Roadmap for National Action on Clinical Decision Support, the study was designed to capture, organize, and assess studies available internationally. ¹

Implications

While the benefits of health information technology are clear at least in theory, adapting information systems to health care has proven difficult, partly because there are so many non-standardized and independent approaches to creating and representing clinical knowledge and CDS systems. ^58,59 In this regard, our systematic review may provide a foundation for and influence the future design and implementation of computerized decision support systems used to detect ADEs in the hospital setting. Having comprehensive information on the performance characteristics of individual signals may help hospitals prioritize the signals to be included in their systems to maximize the detection of ADEs and to minimize the number of false-positive alerts (i.e., alert burden), which is a growing problem. ^60,61 To further reduce false-positive alerts, investigators have also begun to integrate data from multiple sources, including pharmacy, laboratory, and demographic data. ^62,63 Taking the false-positive rate into account is especially important when large-scale information systems are being developed, since as many as 30% of information system projects fail and a significantly larger number have cost overruns. ⁶⁴

The fact that many of the signals to detect ADEs have relatively low PPVs should not impede the adoption of clinical event monitors. ⁶⁵ In many respects, the monitors can be treated as a type of screening test that allows for early ADE identification and intervention, and thereby reduces morbidity and mortality rates. ⁶⁶ Indeed, the monitors have been shown to detect ADEs not normally detected by clinicians during the course of routine care, and to decrease the length of time until diagnosis and treatment. ^18,19,67 Screening tests such as fecal occult blood testing to detect colorectal cancer are recommended despite having PPVs that range from 0.02 to 0.18 in adults over 50 years old, and are thus similar to the ranges of some signals described in our study. ⁶⁸

Recommendations for Future Work

Additional studies are needed to improve the performance characteristics of individual ADE signals and CDS systems, apply these systems to other clinical environments, develop interoperable systems, and perform economic analyses of these systems. Studies have suggested that ADE detection rates can be improved by combing multiple data sources and having a better understanding of the context of the data as they relate to patients’ underlying medical conditions. ^69–72 Investigators have begun to use clinical decision support systems to detect ADEs in other clinical care settings, such as ambulatory care clinics and nursing homes. ^57,73–75 These systems may be particularly useful in the nursing home setting where patients are frail, have multiple comorbid medical conditions, and take more medications per patient than in any other clinical setting. ^74,76,77 Since most systems lack standardized methods to export or share ADE algorithms, additional studies are required to develop interoperable systems. ^78,79 Additional cost-benefit and cost-effectiveness studies are needed not only to determine the rational selection, optimal use, and potential success of systems used to detect ADEs, but also to determine the costs of developing and maintaining the systems and of responding to true-positive and false-positive alerts.

Conclusions

Our systematic review provides the PPVs of pharmacy and laboratory signals used to detect ADEs in the adult hospital setting, and suggests that the PPVs of individual signals vary widely. Our findings should help clinical information system and clinical decision support producers create and modify clinical decision support systems to detect ADEs in their own institutions. Future studies are needed to improve the performance characteristics of individual ADE signals and CDS systems, apply these systems to other clinical environments, develop interoperable systems, and perform economic analyses of the systems.