Using Computerized Data to Identify Adverse Drug Events in Outpatients

Abstract

Objective: To evaluate the use of a computer program to identify adverse drug events (ADEs) in the ambulatory setting and to evaluate the relative contribution of four computer search methods for identifying ADEs, including diagnosis codes, allergy rules, computer event monitoring rules, and text searching.

Design: Retrospective analysis of one year of data from an electronic medical record, including records for 23,064 patients with a primary care physician, of whom 15,665 actually came for care.

Measurement: Presence of an ADE; sensitivity and specificity of computer searches for ADE.

Results: The computer program identified 25,056 incidents, which were associated with an estimated 864 (95 percent confidence interval [CI], 750–978) ADEs. Thus, the ADE rate was 5.5 (CI, 5.2–5.9) per 100 patients coming for care. Furthermore, in 79 (CI, 68–89) ADEs, the patient required hospitalization, resulting in an estimated rate of 3.4 (CI, 2.7–4.3) admissions per 1,000 patients. The sensitivity of the search methods for identifying ADEs was estimated to be 58 (CI, 18–98) percent, and the estimated specificity was 88 (CI, 87–88) percent. The positive predictive value was 7.5 (CI, 6.5–8.5) percent, and the negative predictive value was 99.2 (CI, 95.5–99.98) percent. Compared with age and gender-matched controls with no positive screen, patients with ADEs had twice as many outpatient visits and were taking nearly three times as many drugs. Antihypertensives, ACE-inhibitors, antibiotics, and diuretics were associated with 56 (CI, 47–65) percent of ADEs. Among ADEs, 23 (CI, 16–32) percent were lifethreatening or serious, and 38 (CI, 29–47) percent were judged preventable.

Conclusion: Computerized search programs can detect ADEs, and free-text searches were especially useful. Adverse drug events were frequent, and admissions were not rare, although most hospitals today do not identify them. Thus, such detection programs demonstrate “value-added” for the electronic record and may be useful for directing and assessing the impact of quality improvement efforts.

Adverse drug events (ADE) are common and costly and are responsible for significant morbidity and mortality among hospitalized patients. According to one estimate, the costs of problems with medications may be higher than the total cost of cardiovascular or diabetes care.¹ Both the Food and Drug Administration and the Joint Commission on Accreditation of Healthcare Organizations emphasize the need for reporting ADEs as important markers of the quality of medical care.^2,3 In addition, the American Society for Health-Systems Pharmacists recommends that all health care systems develop ongoing ADE reporting programs.⁴ It has been estimated that ADEs account for up to 106,000 deaths annually in the United States.^5,6 Most studies of ADEs have been performed with hospitalized patients,^7–11 and 2 to 5 percent of all hospital admissions each year are due to ADEs.^12,13 In one study of inpatients, nearly 28 percent of ADEs that occurred were preventable.¹⁴

Although ADEs are common in inpatients, routine identification of these events has generally been ineffective. Spontaneous reporting is the most widely used technique, but it identifies only 5 percent of events.¹⁵ In inpatients, manual chart review is more effective but too costly to be routinely practical.¹⁶ Another alternative is computerized detection, which has been effective in identifying ADEsat several inpatient sites.^6,16–19

Although inpatient ADEs have received considerable recent attention, relatively less is known about the incidence, severity, preventability of, and risk factors for ADEs in the ambulatory population.²⁰ The incidence of outpatient ADEs has been reported to range from 2.6 to 50.6 percent, depending on the data collection methods and the definition of ADEs used.^21–33 Estimation of rates is made even more difficult because of under-reporting by both the provider and patient as well as subjectivity in assessing causality between the event and the specific drug therapy. In addition, medication administration is vastly less centralized and controlled than in inpatient settings, so that compliance becomes a key issue.

In 1992, there were 908 million ambulatory visits to physician offices, outpatient clinics, and emergency departments in the United States.³⁴ As a result of these visits, an estimated 1.5 billion prescriptions were dispensed annually from ambulatory care pharmacies.²¹ The large number of outpatient visits and the significant number of potential drug problems associated with these clinical encounters make it vital to understand and describe ambulatory clinical practice patterns and the impact ADEs have in this setting. This information could help prevent unnecessary hospital admissions and decrease health care costs.

To better characterize ADEs in the outpatient setting, we undertook a study to 1) determine the frequency and types of ADEs; 2) compare the ability of different computer search methods to identify these ADEs; and 3) determine the severity of ADEs in these patients.

Methods

Setting

Brigham and Women's Hospital is a 667-bed institution delivering primary, secondary, and tertiary care in Boston, Massachusetts. It has approximately 170 clinicians who serve as primary care physicians. Primary practitioners work at a diverse array of clinical sites, including hospital-based practices, community-based practices, and neighborhood health centers; almost all use an electronic ambulatory record. Since 1993, all information pertaining to patients and their visits has been collected and stored in the Brigham Integrated Computer System (BICS). This information is stored at the patient level and includes demographics (name, age, gender, race, insurance type etc.), patient problem lists, current medication lists, allergies, a dictated note for clinic visits or telephone communication with patients, diagnoses based on that visit using coded ICD-9 classification, pertinent laboratory data, and the primary care provider (house staff and attending physicians). In addition, all hospitalization and emergency department visits for this health system are coded and entered.

Patients

From July 1995 through June 1996, we collected information on all patient visits to primary care practices using the electronic record. This database included 23,064 patients, of whom 15,665 actually came for care, accounting for 88,514 visits (Table 1▶). For patients with a primary care physician in the system, in the largest managed care plan in the group, less than 10 percent of utilization occurs with other providers or facilities. The study was approved by the Brigham and Women's Hospital human research committee.

Table 1.

Patient Characteristics

	No.	%
Patients	15,665	–
Visits	88,514	–
Visits with notes	78,421	–
Female patients (% by patient)	11,618	74
Race (% by patient):
Caucasian	7,177	46
Black	2,077	13
Hispanic	781	5
Asian	155	1
Other	5,475	35
Payor (% by patient)*:
HMO	8,833	56
Commercial	4,364	28
Government (Medicare)	3,967	25
Medicare managed care	205	1
Medicaid	821	5
Medicaid managed care	368	2
Self pay	1,963	13
Physician visits (% by provider type):
Fellow	1,089	2
Resident
1st Year	29	<1
2nd Year	882	1
3rd Year	1,993	2
Faculty	74,329	84
Specialist	9,352	11
Unknown	840	1
Laboratory tests per patient per year	23.75	–
Drugs prescribed per patient per year	1.29	–
Drugs prescribed per visit	3.35	–

Note: The mean age of patients, as of Jun 30, 1996, was 47.9 years.

* Some patients had more than one payor.

Outcome Measures

The primary outcome was the ADE, defined as “an injury resulting from an intervention related to a drug.”¹⁴ ADEs were considered preventable if an error in the medication process could be identified. The Naranjo algorithm was utilized to assess the likelihood that an ADE was caused by a specific drug.³⁵

The gold standard for the presence of an ADE was chart review, and the method used to assess whether an ADE was present in a specific computer-identified incident was as follows: Each chart was reviewed by one of four physicians trained in ADE evaluation (B.H., J.L., J.R., D.W.B.). They were given the date that the incident occurred and access to the electronic medical record. They then reviewed data, including the medication list, laboratory values, and the notes for that day and several months before and after the incident. For example, if the signal was ACE (angiotensin-converting enzyme) inhibitor and cough, they would assess whether the patient had symptoms suggestive of an upper respiratory tract infection, whether the ACE inhibitor was discontinued, and whether the symptoms resolved with cessation of the ACE inhibitor. Reliability of chart evaluation for the presence of an ADE was assessed by having two independent reviewers evaluate a stratified random sample of 60 incidents over-sampled for events. The kappa was 0.49, suggesting moderate agreement.

ADEs were also reviewed to determine severity, which was categorized as fatal, life-threatening, serious, or significant. Severity was classified as causing death, permanent disability, hospitalization, multiple ambulatory or emergency department visits, a laboratory abnormality requiring only therapy change, or a clinic visit requiring prescription change. Preventability of ADEs was classified as definitely preventable, probably preventable, probably not preventable, and definitely not preventable.¹⁴ In the analyses, results were collapsed into preventable (definite plus probable) and not preventable (definitely not plus probably not). Reliability for judgments regarding severity and preventability using this approach has previously been reported as a kappa of 0.32–0.37 for severity and of 0.92 for preventability.¹⁴

Incident Identification

To identify computer incidents, data were downloaded from BICS into an Access 95 relational database and were accessed through Open DataBase Connectivity (ODBC) to ensure scalability. Visual Basic 5.0 was used to reformat the information so that the investigators could review the records, which were “cleaned” for spelling, syntax, and abbreviations. A computer program was developed consisting of four search methods—ICD-9 codes, allergy rules, computer event monitoring rules, and an automated chart review using text searching of the electronic medical record. This was then used to query and narrow the data to identify possible ADEs, referred to as “incidents.” These were subsequently reviewed manually by the investigators to determine whether an actual ADE had occurred present. Figure 1▶ shows how patient visits, incidents, and ADEs were selected.

Figure 1.

Flow diagram of patients, patient visits, computer-identified incidents, and adverse drug events (ADEs).

Search Methodologies for the Detection of Computer-identified Incidents

IDC-9 Classification Rules

The providers entered ICD-9 codes for each patient visit after the patient encounter, from an abbreviated electronically coded list made available at the point of service. The database was then searched for ICD-9 codes that have been previously found to be associated with the presence of adverse drug events³⁶ (list available from authors).

Allergy Rules

A database containing the patient's medications was accessed; the database included medications prescribed, dose interval, and quantity provided. Discontinued medications were presented. Over-the counter-medications were rarely reported. Using M²D₂ (Micromedex, Denver, Colorado), the patient-drug allergy list was extended to include product names, generics, and ingredients where necessary. Incidents were considered present if any prescription for a medication to which a patient had a known allergy was ordered or a new allergy was recorded. Both medications and ingredients in a medication were used to search through the medication list data.

Computer Event Monitoring Rules

The computer event monitoring rules used Boolean combinations of simple medical conditions such as new medication orders, laboratory results above or below certain numeric thresholds, and changes in laboratory values over time (Table 2▶); they were based on previously used and published inpatient rules that were then modified for the outpatient setting.^16,18

Table 2.

Types of Computer Incidents That Identify Adverse Drug Events (ADEs)

Rule Class		No. of CIIs	Weighted ADE	%CII with ADE*	PPV/(95% CI)†
Allergy		214	104	5.8	48.6 (41.9–55.3)
ICD-9		248		5	0.32.0 (0.6–4.6)
Text searches		22,792	1,637‡	90.6	7.2 (6.8–7.5)
Drug–laboratory rules		1,802	60	3.3	3.3 (2.5–4.2)
Subtypes:
1	On new order (no orders within last 2 weeks) of diphenhydramine	20	1	0.1	5.0 (0.1–24.9)
2	On oral vancomycin	14	1	0.1	7.1 (0.2–33.9)
3	On drugs that increase LFTs (AST/ALT/bilirubin) and blood alkaline phosphate > 350 U/L	48	1	0.1	2.1 (0.1–11.1)
4a	On phytonadione AND on warfarin	0	0	0	–
4b	On warfarin AND international normalized ratio (INR) > 5	179	12‡	0.6	6.7 (3.5–11.4)
5a	On ranitidine and platelet count > 100,000 and < 250,000/mm3	0	0	0	–
5b	On ranitidine and platelet count > 100,000 and < 250,000/mm3	16	0	0	0
6a	On carbamazepine and serum carbamazepine > 12.0 mcg/mL	57	0	0	0
6b	On carbamazepine and white blood count < 3,500/mm3	40	1	0.1	2.5 (0.1–13.2)
7a	On digoxin and serum digoxin > 1.7 ng/mL	53	5	0.3	9.4 (3.1–20.7)
7b	On digoxin and serum potassium < 3.5 mmol/L	43	4	0.2	9.3 (2.6–22.1)
8	On cyclosporine and serum bilirubin > 10 mg/dL	0	0	0	–
9	On cyclosporine and serum cyclosporine > 500 mcg/L	36	2	0.1	5.6 (0.7–18.7)
10	On drugs that increase potassium and serum potassium > 6.5 mmol/L	22	3	0.2	13.6 (2.9–34.9)
11	Blood eosinophils > 6%	923	18‡	1.0	2.0 (1.1–2.9)
12	On Kaopectate	0	0	0	–
13	On loperamide	8	0	0	0
14	On procainamide and serum n-acetyl procainamide > 20 mcg/mL	0	0	0	–
15	On phenytoin and serum phenytoin > 20 mcg/mL	48	1	0.1	2.1 (0.1–11.1)
16	On phenobarbital or primidone, and serum phenobarbital > 45 mcg/mL	13	0	0	0
17	Prednisone and diphenhydramine ordered on the same visit	3	0	0	0
18	On procainamide and serum procainamide > 10 mcg/mL	0	0	0	–
19	On HMG CoA reductive inhibitors and serum aspartate amino transferase > 150 U/L	13	2	0.1	15.4 (1.9–45.5)
20	On theophylline and serum theophylline > 20 mcg/mL	3	2	0.1	66.7 (9.4–99.2)
21	On valproate and serum > 120 mcg/mL	3	0	0	0
22	On quinidine and serum quinidine > 5 mcg/mL	0	0	0	–
23	On HMG CoA reductive inhibitors and serum alanine aminotransferase > 150 U/L	21	2	0.1	9.5 (1.2–30.4)
24	On sodium polystyrene sulfonate	0	0	0	–
25	On topical steroids and no history of psoriasis	153	1	0.1	0.7 (0.02–3.7)
26	On new order (no orders within last 2 weeks) losartan	10	2	0.1	20.0 (2.5–55.6)
28	On clozapine and white blood count < 3,500/mm3	0	0	0	–
28a	On diuretic class A and serum potassium < 3.0 mmol/L	45	1	0.1	2.2 (0.1–11.8)
28b	On diuretic class B and serum potassium > 5.5 mmol/L	0	0	0	–
29	On NSAIDs and serum potassium > 5.5 mmol/L	31	1	0.1	3.2 (0.1–16.7)
overall		25,056	1,806	100	7.2 (6.9–7.5)

Abbreviations: CII indicates computer-identified incidents; PPV, positive predictive value; LFTs, liver function tests; AST, aspartate aminotransferase; ALT, alanine aminotransferase; NSAIDs, nonsteroidal anti-inflammatory drugs.

*Percentage of all ADEs found.

†Positive predictive values and 95% confidence intervals presented as percentages.

‡Actual number divided by the sampling fraction: allergy, 50.0%; term searches, 9.7%; Brigham rule 4b, 52.0%; Brigham rule 11, 10.8%.

Text Searching

Visit notes were examined electronically using the Micromedex M²D₂ medical data dictionary. This data-mining tool is a clinical lexicon server consisting of a controlled vocabulary of medical concepts and drug terminology that allows multiple relationships between multiple medical terms and events.

A full description of this tool has been given elsewhere.³⁷ In brief, the lexical methods applied against ADE terminology identified language at two levels—“morphologic” (singular/plural forms, variant spellings) and “semantic” (synonym links based on the intent of words, or codes applied to words, e.g. ICD-9 codes). A methodology was developed that semantically linked drugs and drug classes to known and reported adverse effects and their synonyms.

The first step was to catalog each word found in the visit notes and combinations of words up to five in a row. This was achieved by breaking tables into records, then into sentences, and then into words and word combinations. Since this is not a full-blown natural language processor, sentences that contain negative terms (like NOT CANCER or RULE OUT CANCER) would be excluded instead of an attempt being made to understand each sentence context. In addition, words were stored as all upper case, which limited comparisons in which capitalization was important and caused certain acronyms like AN to not be found when the next step was completed. The full word and word combinations were then filtered through the M²D₂ structure containing terms that typically indicate an ADE. The resulting list was then joined into core master concepts, tying all the synonyms for a particular ADE for a patient on the same visit together. Mentions of vertigo and dizziness would be considered one incident in this case.

Only adverse effects occurring with a frequency of greater than 1 percent were considered, to decrease the number of false-positive results; the 1 percent threshold was arbitrary. Also, a severity category of “high” was used to eliminate minor adverse occurrences. Incidents were identified using all possible drug names (as in the allergy drug list) and adverse effect combinations. In many cases, more than one drug that the patient was taking could cause a particular effect. Each drug/adverse effect combination was listed as a separate incident. Table 3▶ provides a random sample of patients and, for each, the drug used, the associated medical record term, and the adverse effect term.

Table 3.

Random Sampling of Medical Record Terms Found to Be Associated with Adverse Effects of Drugs Used in Our Patient Population

Pt ID	Drug Name	Medical Record Term	Adverse Effect Term
113	Relafen (nabumetone)	Dry mouth	Xerostomia
113	Prilosec (omeprazole)	Dry mouth	Xerostomia
115	Naprosyn (naproxen)	Eruption	Erythema
115	Pepcid (famotidine)	Swelling	Edema
115	Prilosec (omeprazole)	Diarrhea	Diarrhea
115	Pepcid (famotidine)	Diarrhea	Diarrhea
118	Relafen (nabumetone)	Epistaxis	Epistaxis
121	Motrin (ibuprofen)	Swelling	Edema
127	Tenormin (atenolol)	Depression	Depression
129	allopurinol	Renal insufficiency	Renal failure
129	Indocin (indomethacin)	Low blood sugar	Hypo- glycemia
130	Motrin (ibuprofen)	Wheezing	Broncho- spasm
130	lisinopril	Cough	Cough
152	Lasix (furosemide)	Hypokalemia	Hypokalemia
156	captopril	Stress	Anxiety
156	Indocin (indomethacin)	Tiredness	Fatigue
156	cimetidine	Depression	Depression
156	cimetidine	Stress	Anxiety
168	atenolol	Sexual problems	Impotence
170	Nicorette (nicotine gum)	Palpitations	Palpitations
171	Lasix (furosemide)	Anemia	Anemia
197	Lasix (furosemide)	Eruption	Rash

Sampling Strategy

A list of incidents was generated using the computerized search methods. Because of the large number of incidents identified, random samples of incidents from each search method were selected for further manual review by the investigators. Sampling fractions for manual review were as follows: allergy search, 50 percent; ICD-9, 100 percent; text searches, 9.7 percent; and computer monitor, 100 percent of all rules except Rule 4b (52 percent) and Rule 11 (10.8 percent) (Table 2▶). For the search methods that were not totally reviewed manually, final adverse event rates were extrapolated on the basis of the sampling fraction.

Data were collected on each patient with a confirmed ADE, identified by chart review, and entered into a computerized data sheet. This included information about gender, race, age, payer group, provider type (house officer, fellow, or faculty), number and type of laboratory tests ordered, number and name of drugs prescribed, and all drugs and class of drugs the patient was taking at the time an ADE was identified. In addition, a problem list was generated for all patients with confirmed ADEs, and any comorbidities were assessed using the Charlson scoring method.³⁸

Using the above approach to assess the frequency of ADEs in the population not identified as having an ADE, we selected a control group. This control group was matched for gender and age within a two-year range, and subjects were randomly selected using a random number generator. The visit selected as the visit for review was that with the date closest to the date of the case visit. This visit note and the related data for the control group were then reviewed by two investigators (J.R. and D.W.B.) to determine whether an ADE was present.

The patients with ADEs were also compared with the subjects in the control group to identify differences in the number of ambulatory visits, the number of drugs the patient was taking, and the new drugs prescribed, as well as comorbidity.

ADE Case Examples

Case 1.

A patient was receiving multiple medications, including warfarin, for mitral valve replacement. The INR (international normalized ratio) became increasingly elevated over a six-week period, increasing from 3.0 to 5.4 IU. The patient complained of increasing back pain, which was eventually diagnosed as a retroperitoneal hematoma requiring hospitalization. The computer search was able to identify this as an incident by two methods—1) rule 4b: “on warfarin and an INR >5”; and 2) text searching: match of warfarin with dictated note containing terms “hemorrhage” and “hematoma.”

Case 2.

A patient with hypertension and congestive heart failure was receiving multiple medications, including captopril. The patient developed throat fullness and urticaria secondary to the captopril, which was discontinued with resolution of symptoms. The computer search identified this as an incident by three methods: 1) allergy search: rule for new drug and allergy identified; 2) rule 25: “on topical steroids and no history of psoriasis”: 3) text searching: match of “throat fullness” and “urticaria” with captopril.

Analysis

To determine the total number of ADEs and account for the instances in which sampling was done, we multiplied by one over the sampling fraction to estimate the number of ADEs for that pool. All ADE-related data were weighted accordingly. For each incident type, positive predictive values (PPVs) were then calculated. The numerator was the number of incidents associated with an ADE, and the denominator was the total number of computer-identified incidents of that type. Individual ADEs were often identified by more than one computer screening method. To project the total number of ADEs, we therefore divided the total number of ADEs identified by any screen by the mean number of incidents associated with an ADE.

To assess the sensitivity and specificity of the computer program (the likelihood that any one or more of the four strategies would detect an ADE), we used the following approach. For sensitivity, we divided the number of visits in which an ADE was identified by one or more of the approaches, and divided this figure by the total estimated number of visits in which an ADE was present (the false negative rate was estimated by identifying the number of ADEs in the control group). The exact binomial 95 percent confidence interval for the estimates of true positive and false negative ADEs were calculated. Specificity was assessed by dividing the frequency of visits with no positive screen by the number of visits in which no ADE was present. Because the control group included only patients who had no positive screen for any of the computer search methods, we could only estimate sensitivity and specificity of the combined approach, not of the individual strategies.

To calculate P values for differences between ADE and control groups, we used the chi-square test for proportions and the t-test for means. All analyses were performed using the statistical software SAS SAS Institute Inc., Cary, North Carolina).³⁹

Results

Between July 1, 1995, and June 30, 1996, 23,064 patients were followed, among whom 15,665 visited the ambulatory practices of the Brigham and Women's Hospital. This latter group accounted for 88,514 visits (5.65 visits/patients/year) and was considered the study population. In this group, 74 percent of patients were female and the mean age was 47.9 years (see Table 1▶). Of the total population of the clinic (23,064), 16,802 (73 percent) were female and the mean age was 48.6 years. Each patient with ambulatory visits had, on average, 23.75 individual laboratory tests done annually, 1.29 new drugs prescribed annually, and was taking 3.35 unique drugs per year. In this group, 84 percent were seen by faculty and 87 percent had some type of insurance coverage.

Frequency of Incidents and ADEs

Altogether, the computer program identified 25,056 incidents in the study population of 15,655; after adjusting for the sampling strategy, an estimated 864 (CI, 750–978) ADEs were identified (Figure 2▶ ; see also Table 2▶). Thus, the ADE rate was 5.5 (CI, 5.2–5.9) per 100 patients per year among the group coming for care, and 3.7 (CI, 3.5–4.0) per 100 patients per year among all patients.

Figure 2.

Performance characteristics of the combination of the four search methods for detecting adverse drug events.

Of the 864 ADEs, hospitalization was required in 79 (CI, 68–89), so that 9.1 (CI, 7.3–11.3) percent of ADEs resulted in hospitalization. Viewing this from another perspective, of the ambulatory population coming for care (n=15,665), 1,706 patients were hospitalized to Brigham and Women's Hospital for any cause during the one-year study period, accounting for 2,316 admissions. Thus, we estimate the proportion of admissions caused by an ADE to be 3.4 (CI, 2.7–4.3) percent (79 of 2,316 admissions) .

Assessment of Performance Characteristics of the Computer Screening Strategy

The projected total of 864 (CI, 750–978) ADEs was based on 121 ADEs identified by manual chart review, which occurred in 97 patients. Each visit that included an ADE was matched with a randomly selected control visit, in which the control patient had an age within two years of the age of the case patient, and the same gender. The control group included 121 visits and 118 patients. On manual chart review of all the control group visits, only one ADE was detected. On the basis of this information (Figure 2▶), the sensitivity of the search methods for identifying ADEs was estimated to be 58 (CI, 18–98) percent and the estimated specificity 88 (CI, 87–88) percent. The PPV was 7.5 (CI, 6.5–8.5) percent, and the negative predictive value (NPV) was 99.2 (CI, 95.5–99.98) percent. Among the 121 visits with ADEs, 100 were identified by one search method, and 21 were identified by more than one search method. In addition to the manual review by the investigators, each ADE was confirmed by the Naranjo method for determining likelihood that an event was due to a specific drug: 15 (12 percent) were classified as possible; 64 (53 percent) as probable, and 42 (34.7 percent) as definite.

The relationship between computer-identified incidents and ADEs was often “many to one” (approximately 2.1:1), since for some ADEs many incidents pointed toward the same event (Figure 1▶). Of the four methodologies used to identify incidents, the text-searching method had by far the highest number of incidents (22,792) and also pointed to the largest number of ADEs, although the PPV was relatively low at 0.07 (CI, 0.07–0.08). Allergy rules, on the other hand, yielded the lowest number of incidents (214), but 104 of those were ADEs, for a PPV of 0.49 (CI, 0.42–0.55). ICD-9 searches had the lowest PPV (0.02; CI, 0.0–0.05). The entire computer rules search had an overall PPV of 0.03 (CI, 0.03–0.04). However, if one of these rules with an extremely low yield was eliminated (“blood eosinophils > 6 percent”), then the PPV would nearly double to 0.05 (CI, 0.03–0.06). In addition, three computer rules had a moderately high PPV (Table 2▶); “on theophylline with a serum theophylline > 20 mcg/mL” (PPV, 0.67; CI, 0.09–0.99); “on new order…losartan” (PPV, 0.20; CI, 0.03–0.56); and “on drugs that increase potassium and serum potassium > 6.5 mmol/L” (PPV, 0.14; CI, 0.03–0.35).

Patient Characteristics

The patient characteristics of the cohort of ADEs and controls that were manually reviewed show that patients with ADEs had double the number of outpatient visits (Table 4▶). In addition, they had more than twice as many new drugs prescribed (5.34 vs. 1.68) and were taking nearly three times the number of drugs (4.37 vs. 1.93). The drug classes that caused the most ADEs were antihypertensives, ACE inhibitors, antibiotics, and diuretics; combined, these four groups were associated with 56 (CI, 47–65) percent of all ADEs (Table 5▶). Dermatologic, central nervous system, and gastrointestinal events were the most common types of events (Table 6▶).

Table 4.

Comparison Between Patients with ADEs and Those in Control Group

	ADE Group	Control Group
	No. (%)	No. (%)	P Value
No. of patients*	97	118	–
Total outpatient visits	1,437	778	–
Average visits/patient	14.81	6.59	–
No. of female patients	73 (75.3)	87 (73.7)	0.80
Race:
Caucasian	58 (59.8)	75 (63.6)
Black	17 (17.5)	20 (16.95)
Hispanic	6 (6.2)	5 (4.2)	0.97
Asian	1 (1.0)	1 (0.85)
Other	15 (15.5)	17 (14.4)
Average age (years):	59.8	61.5	0.44
<31	2 (2.1)	2 (1.7)
31–40	12 (12.4)	14 (11.9)
41–50	9 (9.3)	9 (7.6)	0.98
51–60	24 (24.7)	27 (22.9)
61–70	23 (23.7)	27 (22.9)
>70	27 (27.8)	39 (33.0)
Insurance†:
HMO	39 (40.2)	42 (35.6)	0.49
Commercial	24 (24.7)	33 (28.0)	0.59
Government (Medicare/Medicaid)	49 (50.5)	63 (53.4)	0.68
Self-pay/indigent	13 (13.4)	7 (5.9)	0.06
Average new drug prescriptions/patient/year	5.34	1.68	<0.01
Average number of drugs patient was receiving	4.37	1.93	<0.01

*One hundred twenty-one visits with ADEs occurred in 97 patients while 121 visits without ADEs (control) occurred in 118 patients.

† Some patients had more than one insurance status.

Table 5.

Distribution of Adverse Drug Events (ADEs) by Medication Class and Specific Medication

Drug Category	No.	%
Antihypertensive:	33	27.3
Atenolol	7	5.8
Nifedipine	7	5.8
Amlodipine	5	4.1
Diltiazem	4	3.3
Terazosin	3	2.5
Propranolol	3	2.5
Other	7	5.8
ACE inhibitor:	25	20.7
Lisinopril	18	14.9
Enalapril	4	3.3
Other	4	3.3
Antibiotic:	10	8.3
Trimethoprim	3	2.5
Other	7	5.8
Diuretic:	9	7.4
Furosemide	4	3.3
Triamterene	4	3.3
Hydrochlorothiazide	1	0.8
Anticoagulant:	8	6.6
Warfarin	8	6.6
Antidepressant:	8	6.6
Paroxeti	3	2.5
Other	6	5.0
Cardiovascular:	8	6.6
Digoxin	8	6.6
Cholesterol-lowering:	6	5.0
Lovastatin	3	2.5
Other	3	2.5
Pepticulcer:	6	5.0
Omeprazole	3	2.5
Ranitidine	3	2.5
NSAID:	6	5.0
Ibuprofen	2	1.7
Naproxen	2	1.7
Other	3	2.5
Antiseizure:	4	3.3
Phenytoin	4	3.3
Carbamazepine	1	0.8
Other	16	13.2

Note: There were 121 ADEs; many ADEs were associated with more than one drug category. In addition, within categories, ADEs were sometimes associated with more than one medication, so that the numbers for subcategories may sum to more than the total for a category. In all, 68 (56.2 percent) of the ADEs were associated with one or more drugs from the following classes: antihypertensives, ACE inhibitors, antibiotics and diuretics.

Table 6.

Organ Systems and Signs and Symptoms for Adverse Drug Events

	ADE Group, n = 121	Control Group n = 121
Skin:	31 (26)	0
Rash	16	0
Pedal edema	5	0
Angioedema	3	0
Other	7	0
Central nervous system:	24 (24)	1
Dizziness	9	0
Fatigue	7	0
Other	8	1
Gastrointestinal:	22 (18)	0
Nausea	6	0
Diarrhea	6	0
Other	10	0
Respiratory:	19 (16)	0
Cough	18	0
Shortness of breath	1	0
Hematologic:	9 (7)	0
Overanticoagulation without Bleeding	4	0
Overanticoagulation with Bleeding	3	0
Other	2	0
Cardiovascular:	5 (4)	0
Hypotension	3	0
Tachycardia	2	0
Other	5 (4)	0

Note: Percentages appear in parentheses.

Regarding severity, no ADE was fatal. Seven (prevalence, 5.8 percent; CI, 2.4–11.6 percent) were life threatening; 21 (prevalence, 17.4 percent; CI, 11.1–25.3 percent) were serious, and 93 (prevalence, 76.9 percent; CI, 68.3–84.0 percent) were significant. Of the ADEs reviewed, 9.1 (CI, 4.6–15.7) percent resulted in hospitalization, 15.7 (CI, 9.7–23.4) percent required multiple ambulatory or emergency department visits, 12.4 (CI, 7.1–19.6) percent had a laboratory abnormality requiring a change in therapy, and 62.8 (CI, 53.6–71.4) percent required at least one additional clinic visit for prescription changes. Forty-six ADEs (prevalence, 38 percent; CI, 29–47 percent) were felt to be preventable. If these percentages are extrapolated over the entire clinic population, then 0.34 (CI, 0.27–0.43) percent of this outpatient practice had to be hospitalized secondary to an ADE; 0.59 (CI, 0.49–0.70) percent required multiple ambulatory or emergency department visits, and 2.35 (CI, 2.16–2.56) percent required at least one additional clinic visit for prescription changes.

Discussion

We found that ADEs were frequent and that a computerized detection approach could identify these events with moderate sensitivity in a primary care setting that uses an electronic record. In addition to the symptoms involved, these ADEs resulted in many additional ambulatory visits and hospitalization in an important proportion of cases. Many of the ADEs were judged to be preventable. Currently, hospitals do not routinely detect these events.

Most previous studies of ADEs in outpatients used chart review and self-report in cohorts of patients and extrapolated their findings on the basis of total annual visits, to arrive at an estimated occurrence rate.^21–33 Our approach uses the information contained in the electronic record to identify incidents that may be associated with ADEs. Compared with manual chart review, this approach to estimating an occurrence rate is faster and much less expensive. The ADE rate we identified undoubtedly represents a lower bound, since our assessment suggests that we missed 42 percent of the ADEs recorded in the medical record; in addition, other work suggests that many ADEs in the ambulatory setting are not even recorded in the medical record.⁴⁰

Studies of ADEs in ambulatory settings using patient surveys report higher rates of ADEs (30–50 percent).^{28,29,31,33,40–42} Studies based on chart reviews in small clinic populations in which ADEs are defined as events requiring consultation or therapy generally report a lower incidence (1–3 percent).^26,27,30 Kellaway and McCrae³¹ surveyed patients who were prescribed new medications and had recently been discharged from the hospital, and 41 percent said they “certainly or probably” had a reaction to a drug. Most of these patients, however, also had a significant medical or surgical problem, which initiated the hospitalization, thus making it difficult to definitively assess the relationship between the symptoms and the drugs. Similar problems of causality were encountered in Martys' study of a general practice in England,³³ in which 41 percent of patients described adverse events secondary to newly prescribed drugs. Klein et al.²⁹ and Darnell et al.²⁸ used telephone surveys to detect ADEs among patients in a medical clinic and a high-rise building of elderly persons (study populations, 200 to 250) and reported that 30 percent had ADEs. These rates are much higher than our findings, but these samples included high-risk populations. Campbell et al.²⁷ used ICD claims of a 5 percent sampling of a Kaiser population in Oregon and reported an incidence of 3.1 percent for ADEs. Our findings of 2 percent ADEs using ICD-9 claims are similar. In addition, he reported that 30 percent of patients with ADEs required more than two additional clinic visits, a rate twice what we found, but only 0.23 percent of those with ADEs in that study required hospitalization, compared with 9.1 percent in this population.

Several studies identify cardiovascular drugs (including ACE inhibitors), CNS (central nervous system) agents, antibiotics, and diuretics, as the leading causes of ADEs in the outpatient setting.^24,26,27 More data (including the associated medications) are available regarding the epidemiology of ADEs in inpatients than in outpatients. ADEs occur in 4 to 7 percent of hospitalized patients.^5,7,8,14,18 The proportions of events that are life threatening was reported as being higher (1 to 2 percent) than in this outpatient population, but the incidence of serious and significant events was similar. Nevertheless, even a significant event in an outpatient can result in multiple visits to a clinic or emergency department. For hospitalized patients, ADEs resulted in prolonged length of stay and higher health care costs.^7,8 Although we did not do a cost analysis, it is likely that repeated visits to providers will result in higher health care costs as well.

Persons with ADEs in this study were different from those without ADEs. Although they were similar with respect to age, gender, and race, the patients with ADEs had many more visits to their providers, were receiving more drugs, and had more new drugs prescribed. The profiles of those patients with ADEs were also different from the profile reported in studies of inpatients, in the symptoms they manifested and the drugs that caused the ADE.^11,13,14,16 This outpatient population had fewer CNS agents producing ADEs, and more ADEs were associated with antihypertensive and antidepressant medications. Drugs used for long-term therapies seemed to have more ADEs associated with their use, perhaps in part reflecting lack of documentation of ADEs associated with short-term therapies. In addition, the frequency of skin manifestations tended to be higher in outpatients whereas the frequencies of CNS and gastrointestinal complaints were similar.

Our finding that 38 percent of outpatient ADEs were preventable was generally similar to the reported proportion in inpatients, which has ranged from 28 to 50 percent.¹⁴ Such preventable ADEs represent important opportunities for improvement in quality of care. Most of the ADEs determined to be preventable were found by the allergy rule. Allergy prevention is an area that can readily be addressed by alerts delivered using computerized prescribing. However, even with this system in the inpatient setting, drugs to which patients were allergic were still not only ordered but administered.⁴² This occurred because providers frequently failed to add new allergy information into the medical record. Other types of decision support that may be useful are alerts about drug–drug interactions, interpretation of critical laboratory results, and laboratory results in relation to drugs prescribed, dose limits, and guided dosing algorithms.⁴³ In addition to computer decision support, pharmacists represent an important safety net, especially in the outpatient setting, in which education is pivotal.

The tool we used could be improved further; several aspects of the search process deserve mention. ICD-9 terms identifying ADEs—such as E codes (mechanism of illness or injury), which are specific to drug events—were not routinely used by the physicians at the study institution. The list of ICD-9 codes included some for nonspecific allergic reactions, such as allergic rhinitis and conjunctivitis and adverse reactions to food substances [693.1–693.9] rather than being specific for drug events. In addition, physicians often used these codes inappropriately. If coding were more accurate and if the list of ICD-9 codes were narrowed to be more drug oriented, the sensitivity of this search method would be higher.

Not surprisingly, recording a new allergy to a medication or prescribing a medication to which the patient was allergic can be a very sensitive marker for adverse events. However, in this electronic medical record, true allergies were not separated from sensitivities such as nausea due to codeine or erythromycin. Although not true allergies, drug sensitivities are often listed by the provider and entered into the electronic medical record as allergies, reducing the specificity of this method.

We modified the computer rules for inpatient ADE detection described by Jha et al.,¹⁶ in order to use them in an outpatient practice. After being piloted, some of the rules required modification, while others were eliminated because of different practice patterns or less frequent use of some laboratory tests. Many of these rules would require additional modification to be more sensitive in ambulatory settings.

Textual searching of the electronic medical record is potentially important because it requires only that dictated text be available electronically. This detection method required major modifications, since the first searches yielded a database with too many “hits.” Some of these adjustments included eliminating sentences with negative terms (e.g., “no,” “not”) or ambiguous terms (e.g., “rule out”), limiting the terms used for adverse events (“e.g., ovarian disease,” “cancer”) and adjusting for terms that are used for adverse events and primary conditions (e.g., “swelling,” “rash,” “irritation”). The PPV remained low even after these modifications.

Since ADEs are both costly and frequent^43–46 and represent an important aspect of quality of care,¹⁵ many attempts at providing better monitoring of these events at a lower cost have been attempted. Our program for ADE detection expanded on some of those developed earlier by Classen et al.¹⁸ and Bates et al.⁴³ by including a greater breadth of detection approaches.

This study has a number of limitations. It was performed at only one institution, so the results may not be generalizable to other outpatient settings. However, previous studies provide evidence that these events are common. In addition, this institution had an advanced electronic medical record, which provided access to multiple patient events. Collating and reformatting electronic information may not be possible in all institutions, since most such records are currently not coded; however, most electronic information can be reformatted to be searched by this approach.

The searches undoubtedly missed many events that were documented; thus, the sensitivity could be improved. Furthermore, many other events were probably not noted in the record⁴⁰—in particular, telephone contacts, which are variably documented. In addition, only utilization that occurred within this network could be identified.

Another limitation is that only those medications prescribed at these outpatient settings were analyzed. Over-the-counter medications, which may have also caused adverse events, were not investigated. The lexicon used for text searching was in the early stages of development. By including only the most common important adverse effects, the sensitivity would have been improved. Also, the point estimates for sensitivity and specificity, especially for sensitivity, are uncertain. Finally, the PPV of the rules was only 7 percent, which is good enough for signal detection but not sufficient for use as a quality monitor. If this proportion could be improved to 80 percent , for example, it might be possible to use this as an independent quality tool.

Conclusions

In this study, we developed a program that combines four computer search methods, including text searching of the electronic medical record, to detect ADEs in outpatient settings. Although further refinements to this methodology should improve the overall accuracy of detection, these data demonstrate that the methodology of combining several searching tools can be successful in retrospectively detecting with moderate sensitivity ADEs in the electronic medical record.