Abstract
Objective: To examine various strategies for the identification of adverse drug events (ADEs) among older persons in the ambulatory clinical setting.
Design: A cohort study of Medicare enrollees (n = 31,757 per month) receiving medical care from a large multispecialty group practice during a 12-month observation period (July 1, 1999 through June 30, 2000).
Measurements: Possible drug-related incidents occurring in the ambulatory clinical setting were detected using signals from multiple sources.
Results: During the tracking period, there were 1,523 identified ADEs, of which 421 (28%) were considered preventable. Across all sources, 23,917 signals were found; 12,791 (53%) were potential incidents that led to review of a patient's medical record and 2,266 (9%) were presented to physician reviewers. Although the positive predictive value (PPV) for reports from providers was high compared with other sources (54%), only 11% of the ADEs and 6% of the preventable ADEs were identified through this source. PPVs for other sources ranged from a low of 4% for administrative incident reports to a high of 12% for free-text review of electronic notes. Computer-generated signals were the source for 31% of the ADEs and 37% of the preventable ADEs. Electronic notes were the source for 39% of the ADEs and 29% of the preventable ADEs. There was little overlap in the ADEs identified across all sources.
Conclusion: Our findings emphasize the limitations of voluntary reporting by health care providers as the principal means for detection of ADEs and suggest that multiple strategies are required to detect ADEs in geriatric ambulatory patients.
Development of approaches to prevent adverse drug events (ADEs) requires the efficient identification and analysis of the range of preventable drug-related adverse events occurring in a clinical setting. However, unbiased identification has proven to be a difficult step. Reliance on spontaneous reporting has been found to systematically underestimate the rate of ADEs.1,2,3 Manual chart review is highly labor intensive and costly.3 Computerized detection has been examined in several studies set in hospitals3,4,5,6,7 and one study in the ambulatory setting.8,9 As computerized administrative data and electronic medical records become more available in this setting, they may provide more efficient opportunities for identifying adverse events.10,11 This represents an important frontier in medical informatics and is an important practical implication of text searching and natural language processing, as emphasized in a recent Institute of Medicine Report.12
We conducted a study of a large population of Medicare enrollees receiving medical care in the ambulatory setting to evaluate the incidence and preventability of ADEs among ambulatory geriatric patients and classify preventable events by the stage of the pharmaceutical care process at which the error occurred. We identified 1,523 ADEs for a rate of 48.0 per 1,000 person-years of observation. The rate of preventable ADEs was 13.3 per 1,000 person-years.13
Nested within this previously reported study, we conducted an examination of the relative productivity of each of the strategies employed for identifying ADEs, including computerized signals, electronically recorded clinic notes, manual reviews of hospital discharge records and reports from emergency department visits, and spontaneous reports from health care providers. This aspect of the study focused on comparisons of the positive predictive values (PPVs) of signals from each source, the types of ADEs that each identified, and the amount of overlap across sources. There are large differences in the availability and ease of access to these sources across ambulatory settings, but little evidence to support decisions about their use in patient safety research and quality improvement efforts. This study was directed at filling this evidence gap by clarifying the relative contribution that each source makes to detecting ADEs in this setting.
Methods
Study Setting and Population
This study was conducted in the setting of a large multispecialty group practice closely aligned with a New England–based health maintenance organization (HMO). The multispecialty group practice employs 217 physicians and includes 30 ambulatory clinic sites. The group practice provides medical care to more than 30,000 persons aged 65 or older, more than 85% of whom are enrolled in a Medicare + Choice Plan (Medicare risk contract with the health plan), with the remainder being traditional fee-for-service Medicare enrollees. Subjects for this study included all persons aged 65 or older receiving health care services delivered by the group practice in the ambulatory setting from July 1, 1999 through June 30, 2000. Residents of long-term care facilities were excluded from the study. The previously published study includes a detailed description of the population.13
The project was approved by the Institutional Review Board (IRB) of the University of Massachusetts Medical School, and the IRB of the group practice and the HMO; it was carried out under the auspices of the health plan and medical group quality management committees as part of peer-review and quality improvement activities. There was no direct contact by study personnel with either patients or health care providers during the study.
Signals of Drug-related Incidents
We limited the study to drug-related incidents occurring in the ambulatory clinical setting. We detected drug-related incidents using multiple sources: (1) reports from health care providers including physicians, advanced practitioners, nurses, and pharmacists (via an Intranet system for electronic submission of event reports, an ADE telephone hotline, e-mailed copies of progress notes to the drug utilization group, or interoffice mailing of cards reporting ADEs); (2) manual review of hospital discharge summaries; (3) manual review of notes from emergency department visits; (4) computer-generated signals; (5) automated free-text review of electronic clinic notes; and (6) manual review of administrative incident reports from the group's affiliated pharmacies concerning medication errors. Ambulatory medical records were selected for review based on information derived from these sources. Initial medical record reviews and abstractions were performed by clinical pharmacist investigators trained in the study protocol. When this review indicated a possible drug-related incident, the abstracted information was presented to physician reviewers for classification.
All available discharge summaries relating to hospitalizations for the population over the study period were obtained. The information contained in each discharge summary was reviewed for evidence of a drug-related incident that led to the admission to the hospital. Similarly, the notes from all emergency department visits were reviewed for evidence of a drug-related incident leading to the emergency department visit. Drug-related incidents that occurred during hospitalizations and emergency department visits were excluded. Clinical pharmacist investigators performed the reviews of the discharge summaries and emergency department notes manually.
Computer-generated signals of possible drug-related incidents included elevated drug levels, abnormal laboratory results, the use of medications considered to be antidotes, and diagnoses (ICD-9 codes) that could reflect an ADE. All computer-generated signals were derived monthly and downloaded onto a personal computer for review by the pharmacist investigators.
Most outpatient notes (more than 80%) were available in electronic form as part of an electronic medical record. Notes were analyzed using computer-based free-text searching to identify potential drug-related incidents, similar to the process described by Honigman et al.8,9 This effort included the Micromedex M2D2 medical data dictionary (Denver, CO) with extensive adaptation to include locally used synonyms and abbreviations for the phrases of interest. Our methodology linked drugs from drug classes to known and reported adverse effects. Specific drugs within each drug class of interest were identified and the national drug codes (NDCs) of relevant drugs on the group practice's formulary were obtained. On a monthly basis, we ran a computer program to identify eligible patients who were dispensed these drugs. This file was linked to the electronic notes via medical record numbers. Any electronic notes containing key phrases that suggested the presence of adverse effects associated with that drug class were identified and the drug and patient information as well as the signaling phrase and the lines of text before and after it were extracted for review by the pharmacist investigators. (A complete list of drugs and text phrases is available from the authors.)
A signal from any of these sources triggered a review of the patient's medical record by a pharmacist investigator. If there was no evidence in the record of a drug-related incident related to the signal, the investigator completed a report with an explanation. After an extensive training period, reliability between pharmacist investigators on the identification of drug-related incidents for presentation to physician reviewers was assessed. During the course of the study, we drew 80 signals of possible drug-related incidents randomly across all signal sources for review by pairs of pharmacist investigators. There was agreement between clinical pharmacist investigators 84% of the time (κ = 0.67).
Classification of Events
The primary outcome of the study was an ADE, defined as an injury resulting from the use of a drug. This definition of an ADE is consistent with definitions used in previous studies.14,15,16,17 Drug-related incidents were presented by the pharmacist investigators to pairs of physician reviewers selected from among four of the authors (JHG, DWB, LRH, JMR). The reviewers independently classified incidents using structured implicit review to determine whether an ADE was present and, if so, whether it was preventable. To determine whether an ADE had occurred, the physician reviewers took into consideration the temporal relation between the drug exposure and the event and any coexisting conditions of the patient as well as whether the event reflected a known effect of the drug. ADEs were considered to be preventable if they were due to an error and were preventable by any means available.14 The structured implicit review process has been used in numerous previous studies relating to ADEs across various clinical settings.14,16,17,18,19,20 When the physician reviewers disagreed on the classification of an incident regarding the presence of an ADE or its preventability, they discussed the issues; consensus was reached in all instances in which there was initial disagreement. We compared the initial assessments of the physician reviewers across all reviews. For judgments about the presence of an ADE, the κ value for agreement was 0.81 and 0.67 for preventability.
Analysis
We assessed the ADEs identified from each source to determine the productivity, as indicated by the PPV, and overlap. We first determined the proportion of signals from each source presented to physician reviewers and characterized the reasons why they were not presented. The PPV for each source was calculated: The numerator was the number of signals produced by that source that were classified as ADEs, and the denominator was the total number of signals from that source. These calculations were repeated for preventable ADEs. We performed parallel calculations for individual types of computer-generated signals and for signals from electronic notes. PPVs were selected as the focus of our analyses to provide research studies and quality improvement efforts with information that can be used to estimate the potential costs and productivity of various methods of detecting ADEs in their own settings. At the end of the data collection period, all ADEs that occurred in the same individuals during the course of the year of follow-up were intensively evaluated to identify duplicates. Duplicate events were retained in analyses of the productivity of signal sources and were separately analyzed to determine the amount of overlap across sources.
Results
During the year of observation, 1,523 ADEs were identified, of which 421 (28%) were considered preventable.13 The number of signals provided by each source and the decisions of pharmacist investigators are summarized in ▶. This preliminary review stage eliminated 21,651 of the 23,917 signals. Among the hospital discharge summaries and notes from emergency department visits, the majority (87% and 93%, respectively) contained no information suggesting that an ADE was a component of the reason for the admission. Among reports from providers, nearly one-fourth concerned patients who were not eligible for the study or incidents that occurred during an inpatient stay. This was encountered much less often for signals from the other sources. Among signals that led to review of the patients' medical records, nearly three-fourths of the computer-generated and electronic note signals were eliminated by the investigators after the medical record reviews. This was most often the result of a clear alternative explanation for the symptoms captured by the signal, such as a nondrug-related symptom or known medical condition. In addition, computer-generated signals and those from electronic notes generated a number of individual signals from the same source that referred to the same incident. For example, abnormal blood urea nitrogen and creatinine test results from the same date frequently reflect a single underlying event. These repeats existed for 9% of the computer-generated signals. Repeated signals from electronic notes were often generated by several different phrases linked to the same drug and resulting from the same patient encounter; 11% of the signals from this source were repeats. Pharmacist investigators accommodated this by clustering their reviews of computer-generated and electronic note signals by patient rather than by signal type, enabling them to identify and eliminate these repeated signals efficiently. In these cases, the specific signal linked to the classification as an ADE was chosen randomly.
Table 1.
Initial Review of Signals
| Source of Signal | No. of Signals | No Indication of an ADE (%)∗ | Patient Ineligible (%)† | No Evidence of ADE in Medical Record (%)‡ | Second Indication of the ADE in the Same Source (%) | Presented to Physician Reviewers (%) |
|---|---|---|---|---|---|---|
| Provider reports | 322 | 0 | 24 | 16 | 1 | 59 |
| Hospitalizations | 3,203 | 87 | 1 | 4 | < 1 | 7 |
| Emergency department visits | 8,303 | 93 | 1 | 3 | < 1 | 3 |
| Computer-generated signals | 6,929 | 0 | 5 | 76 | 9 | 11 |
| Electronic notes | 5,048 | 0 | 2 | 73 | 11 | 15 |
| Incident reports | 112 | 0 | 6 | 5 | < 1 | 88 |
| Total | 23,917 | 44 | 2 | 39 | 5 | 9 |
Case investigators reviewed the reports from emergency department visits and hospital discharge summaries and did not further pursue those for which there were no indications that an ADE was in any way responsible for the presentation or admission.
A patient was ineligible if he or she was an inpatient, resided in a nursing home, or was not a patient of the medical group at the time of the signal.
Case investigators considered a possible ADE to be unsupported by the medical record if there was inadequate relevant information in the record, if the symptoms were explained by other aspects of the patient's condition, if the drug was not taken by the patient at the relevant time, or if there was an error in the signal.
Of the 2,266 drug-related incidents presented to the physician reviewers, 1,523 (67%) were classified as ADEs. The productivity of each signal source is presented in ▶. Among the sources of signals, the highest PPV was for provider reports (54%). Review of hospital discharge summaries and notes from emergency department visits produced very low PPVs (5% and 2%, respectively). Computer-generated signals and indicators in electronically recorded clinic notes had intermediate PPVs (7% and 12%, respectively). For preventable ADEs, PPVs were generally low, with a maximal rate of 8% for provider reports.
Table 2.
Signals from Each Source, Their Classification as ADEs, and Their PPVs
| Source of Signal | No. of Signals | No. of ADEs | PPV | No. of Preventable ADEs | PPV for Preventable ADEs |
|---|---|---|---|---|---|
| Provider reports | 322 | 173 | 54 | 27 | 8 |
| Hospitalizations | 3,203 | 169 | 5 | 58 | 2 |
| Emergency department visits | 8,303 | 193 | 2 | 70 | 1 |
| Computer-generated signals | 6,929 | 466 | 7 | 157 | 2 |
| Electronic notes | 5,048 | 591 | 12 | 121 | 2 |
| Incident reports | 112 | 5 | 4 | 3 | 3 |
▶ summarizes the percentage of the ADEs and preventable ADEs that were identified through each source. Provider reports captured only 11% of the ADEs identified during the year and only 6% of the preventable ADEs. Hospital discharge summaries and notes from emergency department visits captured 11% and 13%, respectively, of the ADEs, and 17% and 14%, respectively, of the preventable ADEs. Computer-generated signals and electronic notes accounted for the largest percentages of ADEs (31% and 39%, respectively) and preventable ADEs (37% and 29%, respectively). Few of the ADEs were identified through administrative incident reports from the pharmacies.
Table 3.
Percentage of ADEs Identified through Each Source
| Source of Signal | % of ADEs in This Source∗ | % of Preventable ADEs in this Source∗ |
|---|---|---|
| Provider reports | 11 | 6 |
| Hospitalizations | 11 | 14 |
| Emergency department visits | 13 | 17 |
| Computer-generated signals | 31 | 37 |
| Electronic notes | 39 | 29 |
| Incident reports | 4 | 2 |
Percentages total more than 100% because some ADEs were identified through more than one source.
Details on the productivity of individual computer-generated signals are provided in ▶. Not included in this table are codes that produced no signals during the course of the year (ICD-9 codes for poisoning by antibiotics; antiinfectives; central nervous sytem depressants; drugs affecting the autonomic nervous system, skin, and mucous membranes; poisoning by other drugs and medical substances; dispensing of naloxone, digoxin immune FAB, protamine sulfate, or flumazenil; and laboratory tests indicating serum quinidine levels over 5 μg/mL, serum procainamide levels greater than 12 μg/mL, or white blood cell counts less than 3,000/μL among those taking clozapine or ganciclovir). Overall, laboratory values indicating out-of-range drug levels had the highest PPV for all ADEs (26%) and for ADEs that were preventable (15%). Signals based on other out-of-range laboratory values, dispensing of drugs commonly used as antidotes, and ICD-9 codes indicating drug-related problems had mean PPVs for any ADE of 5.5% for each group. Laboratory values and ICD-9 codes had mean PPVs of 2% for preventable ADEs, whereas dispensing of antidotes had a mean PPV of less than 1% for preventable ADEs.
Table 4.
Signals from Computer-generated Signals, Their Classification as ADEs, and Their PPVs
| Computer-generated Signals | No. of Signals | No. of ADEs | PPV for ADE | No. of Preventable ADEs | PPV for Preventable ADEs |
|---|---|---|---|---|---|
| Drug levels | |||||
| Serum valproate >120 μg/mL | 4 | 2 | 50 | 2 | 50 |
| Serum theophylline >20 μg/mL | 13 | 2 | 15 | 1 | 8 |
| Serum phenobarbital >45 μg/mL | 2 | 0 | 0 | 0 | 0 |
| Serum phenytoin >20 μg/mL | 66 | 17 | 26 | 11 | 17 |
| Serum cyclosporine >400 ng/L | 3 | 1 | 33 | 0 | 0 |
| Serum digoxin >2.0 ng/mL | 154 | 39 | 25 | 21 | 14 |
| Serum carbamazepine >13.0 μg/mL | 7 | 3 | 43 | 2 | 29 |
| Laboratory results | |||||
| Serum alkaline phosphatase >350 U/L | 146 | 1 | 1 | 0 | 0 |
| Serum bilirubin >4.0 mg/dL | 62 | 0 | 0 | 0 | 0 |
| Serum potassium < 2.9 mmol/L | 77 | 37 | 48 | 21 | 27 |
| Serum potassium >6.0 mmol/L | 139 | 34 | 24 | 14 | 10 |
| Blood eosinophils >9% | 412 | 9 | 2 | 1 | < 1 |
| Serum aspartate aminotransferase >84 U/L | 234 | 11 | 5 | 1 | < 1 |
| Serum alanine aminotransferase >80 U/L | 288 | 11 | 4 | 2 | 1 |
| Serum urea nitrogen >60 mg/dL | 749 | 55 | 7 | 21 | 3 |
| International normalized ratio >5 | 604 | 27 | 4 | 16 | 3 |
| Platelet count < 50,000 μL | 147 | 81 | 5 | 0 | 0 |
| Serum creatinine >2.5 mg/dL | 945 | 26 | 3 | 6 | 1 |
| l-Thyroxine and thyroid-stimulating hormone < 0.3 μU/mL | 449 | 7 | 2 | 4 | 1 |
| Clostridium difficile testing | 347 | 17 | 5 | 3 | 1 |
| Glucocorticoid and hemoglobin A1c >6% | 302 | 33 | 11 | 6 | 2 |
| Antidotes/treatments | |||||
| Prednisone and diphenhydramine | 12 | 2 | 17 | 0 | 0 |
| Phytonadione (vitamin K) | 35 | 2 | 6 | 1 | 3 |
| Sodium polystyrene sulfonate | 69 | 4 | 6 | 4 | 6 |
| Glucagon | 1 | 0 | 0 | 0 | 0 |
| Hydroxyzine and prednisone | 277 | 5 | 2 | 1 | < 1 |
| Oral vancomycin | 13 | 1 | 8 | 0 | 0 |
| Nystatin | 812 | 54 | 7 | 1 | < 1 |
| Diagnoses (ICD-9 codes) | |||||
| Poisoning by | |||||
| Psychotropic agents | 8 | 1 | 13 | 1 | 13 |
| Analgesics and antipyretics | 15 | 2 | 13 | 2 | 13 |
| Agents that affect blood | 10 | 0 | 0 | 0 | 0 |
| Hormones and synthetics | 4 | 1 | 25 | 0 | 0 |
| Anticonvulants/antiparkinsonian drugs | 9 | 0 | 0 | 0 | 0 |
| Sedatives and hypnotics | 9 | 0 | 0 | 0 | 0 |
| Central nervous system stimulants | 2 | 0 | 0 | 0 | 0 |
| Cardiovascular drugs | 54 | 4 | 7 | 4 | 7 |
| Gastrointestinal drugs | 2 | 0 | 0 | 0 | 0 |
| Water, minerals, etc. | 3 | 0 | 0 | 0 | 0 |
| Muscle drugs | 1 | 0 | 0 | 0 | 0 |
| Other drugs | 22 | 1 | 5 | 1 | 5 |
| Late effect of drugs | 1 | 0 | 0 | 0 | 0 |
| Dermatitis due to substances taken internally | 64 | 4 | 6 | 0 | 6 |
| Allergic contact dermatitis | 6 | 0 | 0 | 0 | 0 |
| Drug neuropathy | 1 | 0 | 0 | 0 | 0 |
| Urticaria due to drug | 57 | 2 | 4 | 0 | 0 |
| Aspirin-induced gastritis | 252 | 14 | 6 | 4 | 2 |
| Urticaria, contact | 5 | 0 | 0 | 0 | 0 |
▶ presents parallel information for the indicators that were searched in electronically recorded clinical notes. There was wide variability in the number of signals produced by these indicators. The largest number was produced by symptoms of peripheral edema among patients taking calcium channel blockers with 1,186 signals. Among the signals from electronic notes, PPVs for ADEs ranged from 5% to 31%. Although five signals produced no preventable ADEs, several of the remaining signals had PPVs for preventable ADEs of 10% and greater.
Table 5.
Signals from Electronic Clinic Notes, Their Classification as ADEs, and Their PPVs
| Signals from Electronic Notes | No. of Signals | No. of ADEs | PPV for ADEs | No. of Preventable ADEs | PPV for Preventable ADEs |
|---|---|---|---|---|---|
| Angiotensin-converting enzyme inhibitors and cough | 814 | 72 | 9 | 13 | 2 |
| Selected antidepressants and anorexia | 22 | 1 | 5 | 0 | 0 |
| Selected antidepressants and constipation | 77 | 6 | 8 | 1 | 1 |
| Selected antidepressants and hypotension | 27 | 1 | 4 | 0 | 0 |
| Selected antidepressants and insomnia | 63 | 4 | 6 | 1 | 2 |
| Selected antidepressants and nervousness | 139 | 5 | 4 | 1 | 1 |
| Beta-blockers and bradycardia | 257 | 39 | 15 | 4 | 2 |
| Hypoglycemics hypoglycemia | 293 | 51 | 17 | 19 | 6 |
| Hypoglycemics and tremor | 77 | 4 | 5 | 0 | 0 |
| NSAIDs and bleeding | 324 | 17 | 5 | 6 | 2 |
| NSAIDs and gastrointestinal complaints | 110 | 7 | 6 | 5 | 5 |
| NSAIDs and nausea | 209 | 40 | 19 | 7 | 3 |
| NSAIDs and renal failure/insufficiency | 18 | 1 | 6 | 0 | 0 |
| Selected antidepressants and dry mouth | 22 | 10 | 45 | 3 | 14 |
| Diuretics and hyponatremia | 39 | 8 | 21 | 5 | 13 |
| Diuretics and hypotension | 112 | 16 | 14 | 11 | 10 |
| Warfarin and bleeding | 339 | 46 | 14 | 7 | 2 |
| Digoxin and nausea | 154 | 12 | 8 | 3 | 2 |
| Opioids and constipation | 186 | 58 | 31 | 25 | 13 |
| Calcium channel blockers and peripheral edema | 1,186 | 79 | 7 | 7 | 1 |
| Proton pump inhibitors and diarrhea | 130 | 9 | 7 | 0 | 0 |
| Antibiotics and diarrhea | 420 | 77 | 18 | 4 | 1 |
NSAIDs = nonsteroidal anti-inflammatory drugs.
Overall, only 5% of the ADEs were identified through more than one source. The source with the highest percentage of ADEs captured in other sources was review of notes relating to emergency department visits (12%), with the majority of these identified through computer-generated signals and/or electronic notes. Among preventable ADEs, only 4% were identified in a second source.
Discussion
In this study, the highest numbers of ADEs were found through electronic text searching and computer-generated signals with provider reports, hospital discharges, and emergency department notes identifying much lower numbers. A low percentage of ADEs were reported by providers; only 11% of all ADEs and 6% of the preventable ADEs found during the year were identified through provider reports. This finding parallels previous studies of ADEs among inpatients for whom the proportion of ADEs reported by providers has averaged approximately 5%.5 During the study period, repeated attempts were made to enhance provider reports through in-service training sessions and the development of multiple reporting options including a Web-based mechanism based on the medical group's Intranet. Previous research has demonstrated a correlation between the intensity of prompting and the rate of provider reports of adverse events,21 suggesting that these activities may have been responsible for the slightly higher rate of ADEs reported by providers in this study. Provider reports did have the highest PPV for preventable ADEs compared with other methodologies. However, overall, these data suggest that electronic strategies identify many more ADEs than other sources.
Comparing the productivity of signals to that of undirected chart review in an ambulatory population is complex. The one year of follow-up in this study included 334,045 person-months for Medicare + Choice enrollees and 47,039 outpatient visits by older adults who were not enrolled in that program. Undirected chart reviews would have required assessment of all the visit notes, laboratory test results, medication prescriptions, and reports from hospitalizations and emergency department visits for approximately 381,084 person-months. Based on this estimate and the 1,523 total ADEs identified in this study, the PPV for a person-month of chart review would be 0.4%.
One previous study of ADEs in the ambulatory setting employed incident detection methods similar to the computer-based approaches in our study.8 Although the earlier study included the full age-range of patients seen by a group practice and the current study was limited to elders, there are a number of similarities in the findings. Notably, the rate of ADEs detected was similar (55 per 1,000 patients with visits during one year in the earlier study vs. 48 per 1,000 patient-years in the current study) as was the overall PPV across the computer-based signal sources (7.5% in the earlier study vs. 8.8% in the current study). The earlier study generated many fewer computer signals from laboratory test results and drug prescribing, probably due to the use of higher thresholds and requirements for combinations of information as well as a difference in the age of the patient population; the earlier study found a lower PPV for these signals (3% in the earlier study vs. 6.8% in the current study). In contrast, free-text reviews of electronic notes in the earlier study were much more expansive, identifying 22,792 incidents for review compared with only 5,048 in the current study. Here again, the earlier study found a lower PPV (7.5% in the earlier study vs. 12% in the current study). Although the current study expanded the sources of signals to include manual reviews of hospital discharge summaries and emergency department reports as well as reports from providers, the majority of ADEs were identified through computer-based sources.
We also found very little overlap among the ADEs identified in the various sources that we employed; only 5% of ADEs were found in more than one source. Previous studies have also found low rates of overlap,1,3,22 although not as low as in this setting. This low overlap suggests that every one of the sources that we investigated has low sensitivity in the ambulatory setting. In the current study, there were notable differences in the types of events captured by different sources. Not surprisingly, ADEs for which the major effects are symptomatic were found most often in the electronic notes whereas those that result in out-of-range laboratory tests appeared in the computer-generated signals. Only one type of event, ADEs that consisted of drug-related falls, were found almost exclusively through review of hospitalizations and emergency department visits. Aside from these exceptions, event types were identified across all sources despite the lack of overlap.
In summary, our results suggest that all these sources contribute important, independent information about the occurrence of ADEs in a population treated in the ambulatory setting. Investigators designing studies in similar settings and those attempting quality improvement projects aimed at reducing ADEs should include multiple sources if the aim is to approximate the true underlying rate. Recent work from the Institute for Healthcare Improvement based in hospital settings has suggested that health care systems without easy access to computerized information may be able to replicate some of these signals through the use of “triggers” in paper-based medical records,23 and this may be an option in ambulatory settings as well. Where specific sources are omitted, the details presented here will assist developers in understanding the events that they are likely to miss. Overall, the low PPVs suggest that extensive investigator time will be required to use any of the sources investigated.
Our findings suggest several ways in which the search for ADEs could be made more efficient and less labor-intensive, primarily through enhancements in the use of automated clinical data. In particular, the systems accessed in this study could not easily track changes in laboratory values or drug dispensing over time and did not include entries of new allergies. Our use of electronically recorded clinical notes was based on simple searches for specific keywords and phrases among patients using various drug classes. Natural language processing could improve the PPVs of these indicators through pattern matching and rule-based techniques,24,25,26 and algorithms for detection of ADEs could be developed through machine learning. The initial review of hospital discharge summaries and emergency department visits was a major component of pharmacist investigators' time in our study. For settings in which these reports are captured in electronic form, a similar process could be used to automatically search for indications of drug-related incidents.27 As electronic medical record systems are increasingly used in ambulatory care settings, more sophisticated approaches may be possible. For example, output from computer-generated signals could include clinical notes from the relevant time period, eliminating the need to obtain and search paper medical records.
Our study has several limitations. ADEs were identified only through the methods described so that negative predictive values could not be measured. We did not perform time-motion studies of the pharmacist investigators so that the relative efficiency of the various strategies employed could not be assessed. The study was conducted in the context of a single multispecialty group practice providing care to elderly persons residing in a single geographic area, and the vast proportion of the study population was composed of Medicare + Choice enrollees. This particular setting is ideal for such research because automated data on medications, laboratory results, and electronic clinic notes are readily available. However, the patterns of ADEs and their identification in various sources are likely to differ at other sites. The extent to which possible drug-related incidents could be determined during the medical record reviews is dependent on the quality and extent of record keeping. This medical group used integrated medical records with extensive documentation in a well-organized record. This clarity may not be available at other ambulatory sites, which would reduce the ability to identify ADEs.
Conclusion
Enhanced surveillance and reporting systems for ADEs are required. Our findings emphasize the limitations of voluntary reporting by health care providers and suggest that multiple strategies must be employed to detect ADEs among older persons in the ambulatory setting.
Supported by research grant AG 15979 from the National Institute on Aging, Bethesda, MD. The contents are solely the responsibility of the authors and do not necessarily reflect the official views of the National Institute on Aging.
The authors thank Jill Auger, RPh, and Leslie Garber, RPh, for assistance with data collection relevant to this study. Mary Ellen Stansky and Jackie Cernieux, MPH, are acknowledged for their assistance with technical aspects of this study and Bessie Petropoulos for assistance with manuscript preparation.
References
- 1.O'Neil AC, Petersen LA, Cook EF, Bates DW, Lee TH, Brennan TA. Physician reporting compared with medical-record review to identify adverse medical events. Ann Intern Med. 1993;119:370–6. [DOI] [PubMed] [Google Scholar]
- 2.Cullen DJ, Bates DW, Small SD, Cooper JB, Nemeskal AR, Leape LL. The incident reporting system does not detect adverse drug events: a problem for quality improvement. Jt Comm J Qual Improv. 1995;21:541–8. [DOI] [PubMed] [Google Scholar]
- 3.Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc. 1998;5:305–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients. JAMA. 1991;266:2847–51. [PubMed] [Google Scholar]
- 5.Koch KE. Use of standardized screening procedures to identify adverse drug reactions. Am J Hosp Pharm. 1990;47:1314–20. [PubMed] [Google Scholar]
- 6.Jha AK, Kuperman GJ, Rittenberg E, Teich JM, Bates DW. Identifying hospital admissions due to adverse drug events using a computer-based monitor. Pharmacoepidemiol Drug Saf. 2001;10:113–9. [DOI] [PubMed] [Google Scholar]
- 7.Senst BL, Achusim LE, Genest RP, et al. Practical approach to determining costs and frequency of adverse drug events in a health care network. Am J Health Syst Pharm. 2001;58:1126–32. [DOI] [PubMed] [Google Scholar]
- 8.Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc. 2001;8:254–66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Honigman B, Light P, Pulling RM, Bates DW. A computerized method for identifying incidents associated with adverse drug events in outpatients. Int J Med Inf. 2001;61:21–32. [DOI] [PubMed] [Google Scholar]
- 10.Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology. J Am Med Inform Assoc. 2003;10:115–28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform. 2003;36:131–43. [DOI] [PubMed] [Google Scholar]
- 12.Aspden P, Corrigan JM, Wolcott J, Erickson SM, editors. Patient Safety Achieving a New Standard for Care. Washington, DC: The National Academies Press, 2003. [PubMed]
- 13.Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289:1107–16. [DOI] [PubMed] [Google Scholar]
- 14.Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events: implications for prevention: ADE Prevention Study Group. JAMA. 1995;274:29–34. [PubMed] [Google Scholar]
- 15.Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. JAMA. 1995;274:35–43. [PubMed] [Google Scholar]
- 16.Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adverse drug events in nursing homes. Am J Med. 2000;109:87–94. [DOI] [PubMed] [Google Scholar]
- 17.Leape LL, Cullen DJ, Clapp MD, et al. Pharmacist participation on physician rounds and adverse drug events in the intensive care unit. JAMA. 1999;282:267–70. [DOI] [PubMed] [Google Scholar]
- 18.Kaushal R, Bates DW, Landirgan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA. 2001;285:2114–20. [DOI] [PubMed] [Google Scholar]
- 19.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention of prevention on serious medication errors. JAMA. 1998;280:1311–6. [DOI] [PubMed] [Google Scholar]
- 20.Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. JAMA. 1997;277:307–11. [PubMed] [Google Scholar]
- 21.Rozich JD, Haraden CR, Resar RK. Adverse drug event trigger tool: a practical methodology for measuring medication related harm. Qual Saf Health Care. 2003;12:194–200. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Welsh CH, Pedot R, Anderson RJ. Use of morning report to enhance adverse event detection. J Gen Intern Med. 1996;11:454–60. [DOI] [PubMed] [Google Scholar]
- 23.Dormann H, Muth-Selbach U, Krebs S, et al. Incidence and costs of adverse drug reactions during hospitalization: computerized monitoring versus stimulated spontaneous reporting. Drug Saf. 2000;22:161–8. [DOI] [PubMed] [Google Scholar]
- 24.Spyns P. Natural language processing in medicine: an overview. Methods Inf Med. 1996;35:285–301. [PubMed] [Google Scholar]
- 25.Friedman C, Hripcsak G. Natural language processing and its future in medicine. Acad Med. 1999;74:890–5. [DOI] [PubMed] [Google Scholar]
- 26.Hripcsak G, Friedman C, Alderson PO, DuMouchel W, Johnson SB, Clayton PD. Unlocking clinical data from narrative reports: a study of natural language processing. Ann Intern Med. 1995;122:681–8. [DOI] [PubMed] [Google Scholar]
- 27.Murff HJ, Forster AJ, Peterson JF, Fiskio JM, Heiman HL, Bates DW. Electronically screening discharge summaries for adverse medical events. J Am Med Inform Assoc. 2003;10:339–50. [DOI] [PMC free article] [PubMed] [Google Scholar]