Using Electronic Medical Records to Enhance Detection and Reporting of Vaccine Adverse Events

Abstract

We implemented an automated vaccine adverse event surveillance and reporting system based in an ambulatory electronic medical record to improve underreporting and incomplete reporting that prevails in spontaneous systems. This automated system flags potential vaccine adverse events for the clinician when a diagnosis is entered, prompts clinicians to consider the vaccine as a cause of the condition, and facilitates reporting of suspected adverse events to the Vaccine Adverse Event Reporting System (VAERS).

During five months, a total of 33,420 vaccinations were administered during 14,466 encounters. There were 5,914 follow-up contacts by vaccinees within 14 days of the vaccination visits; 686 (11.6%) generated an alert. Clinicians submitted VAERS reports for 23 of these (0.69 per 1,000 vaccine doses), which is almost 6 times the dose-based reporting rate to VAERS. ¹ Clinician surveys indicated that it took a minimal amount of time to respond to the alerts. Of those who felt that an alert corresponded to an actual vaccine adverse event, the majority used the reporting feature to file a VAERS report.

We believe that elicited surveillance via real time prompts to clinicians holds substantial promise. By coupling simplified reporting with the initial prompt, clinicians can consider and report a vaccine adverse event electronically in a few moments during the office visit.

Introduction

Clinician reporting of vaccine adverse events to public health authorities serves as a key method to detect adverse events during the post-licensure period. The CDC and FDA rely heavily on the Vaccine Adverse Event Reporting System (VAERS), ² the passive (from the federal perspective) surveillance system in the US for identifying adverse effects of approved vaccines. Since VAERS is a spontaneous reporting system, it relies on clinicians, as well as parents and others, to initiate submission of reports. However, spontaneous reporting systems suffer from incomplete recognition of potential adverse events, administrative barriers to reporting, and incomplete case documentation. ³ True adverse events are therefore underreported; additionally, events that are not causally related to vaccination may be overreported. ⁴ One study found that the completeness of reporting varies widely, from 68% reporting efficiency for vaccine-associated polio to less than 1% for rash following MMR vaccine. ⁵ In addition to the problems associated with incomplete reporting, the current spontaneous reporting system cannot assess incidence rates of events because it contains no information about the number of individuals who have been immunized. ⁴ However, VAERS data are vital for generating hypotheses about potential vaccine adverse events that can then be examined using a rigorous epidemiological study design.

Information technology techniques have been developed to supplement spontaneous systems and address these limitations. Computerized adverse event monitors have been used to check for adverse drug events, ^6,7 nosocomial infections ⁶ and errors in antimicrobial prescribing. ⁸ One study using a computer simulation model detected 32 times more medication errors than voluntary reporting. ⁹ Natural language processing has been shown to detect adverse events in text documents, ¹⁰ and has recently been used to detect vaccine adverse events as well. ¹¹ Clinical decision support functions have been integrated into electronic medical record (EMR) systems for disease surveillance and patient safety applications. Haller et al. integrated an incident reporting system into their EMR to report on accidents and preventable complications during surgical procedures. ¹² Others implemented interventions within the EMR to improve laboratory monitoring after medication dispensing. ^13,14 Tamblyn et al. and Smith et al. implemented computer alerts to clinicians after prescribing potentially inappropriate medications to elderly patients. ^15,16 However, real time clinical decision support for potential vaccine adverse events has not been previously described.

To assist clinicians with recognizing adverse vaccine reactions and to simplify reporting, we developed and implemented an automated enhancement to an existing EMR system (EpicCare) used in the ambulatory setting. The enhancement identifies potential vaccine adverse events, prompts clinicians during an encounter to consider vaccine adverse events, and facilitates reporting to VAERS. The system is automated and requires minimal resources to maintain. Our approach overcomes some of the limitations of spontaneous reporting systems by recognizing possible adverse events in real time in the ambulatory setting and producing more complete and timely reports to VAERS. It is the first system we are aware of that stimulates clinicians’ recognition of potential adverse reactions to vaccines, facilitates their assessment and documentation of these events, and supports reporting to public health authorities.

Methods

Setting

We developed an enhancement to the EpicCare EMR, as used at Harvard Vanguard Medical Associates (HVMA), a multi-specialty provider group with 14 health centers that care for approximately 330,000 patients in the greater Boston area. HVMA has used EpicCare Classic, an electronic medical record system developed by Epic Systems Inc. (Madison, WI), exclusively since 2000 for all clinical encounters. HVMA and the Institutional Review Board considered this intervention a new function of the EMR, and not human subjects research. The provider survey was considered an evaluation of this new function. Therefore, it was granted an exemption from human subjects protection regulations.

Alerting and Response Processes

When the HVMA clinician entered a diagnosis into the EMR during an office visit or telephone encounter within 14 days following vaccination, a dialog box opened (▶). The clinician was notified that a vaccination had been administered within the preceding 14 days, was asked whether the diagnosis might represent a vaccine adverse reaction, and if so, to answer a few brief questions about the current diagnosis (▶). If the clinician thought it was a possible adverse event, s/he could choose to report it to VAERS. The VAERS form was automatically populated with data from fields in the EMR and the dialog box, so that no additional data entry was required. The VAERS form was sent via facsimile to VAERS; electronic transmission to VAERS is also possible, but was not the focus of this intervention.

Figure 1.

Best practice alert dialog box.

Figure 2.

Vaccine adverse event SmartSet.

Diagnosis Codes

In order to capture unanticipated types of adverse reactions, alerts were presented for all diagnoses unless they appeared on an “exclusion list” of ICD-9 diagnosis codes that we categorized as being relatively common and/or very unlikely to be vaccine adverse events. We included relatively common diagnoses on this list to minimize the number of alerts sent to clinicians. Although some of these conditions could be true vaccine adverse events, we felt it was more important to minimize the number of false positive alerts than to include relatively minor conditions previously identified as vaccine adverse events.

The exclusion code list was developed by two clinicians who reviewed all ICD-9 codes and added to the list those that were very unlikely to be vaccine adverse events (such as well child checkups, injuries and congenital conditions) and very common conditions (such as upper respiratory infections, conjunctivitis, asthma and otitis media). We did not formally validate this list. We reviewed frequencies of diagnosis codes during various post-vaccination time windows to estimate the numbers of alerts that would be sent to clinicians. Four clinicians beta-tested the initial list, and we revised the list based on recommendations from the beta testers. The complete exclusion list is shown in the appendix.

When a clinician entered a diagnosis in the EMR during a telephone or office visit encounter, the system determined whether any vaccines had been administered during the previous 14 days. If so, then each diagnosis was compared to the exclusion code list. If the diagnosis code was not on the exclusion code list, it generated an alert. We excluded diagnoses entered on the same day as the vaccination, to avoid alerts based on diagnoses entered during the immunization visit that were unrelated to the vaccination.

Dialog Box

The dialog box was the interface with the user. It opened via the EpicCare Best Practice Alert function to alert the clinician to a possible vaccine adverse event if the alert criteria were met. The initial dialog box notified the clinician that the diagnosis entered was possibly related to a recent vaccination, and referred the clinician to an EpicCare SmartSet, a decision support question set (▶). The first question asked if the clinician thought the current visit was related to a vaccine reaction. If the clinician entered “no,” there were no further questions. Otherwise, the clinician was asked to provide brief details about the potential adverse event, and could choose whether to report it automatically to VAERS. The clinician could answer them immediately, or return to answer them later.

VAERS Form

We developed an HTML version of the VAERS form that can populate with data from the EMR. The final question in the dialog box asked whether the clinician wanted to report the potential adverse event to VAERS. If the clinician answered yes, the reporting feature populated the VAERS form with demographic, provider and vaccination data from the EMR and data the clinician entered into the alert dialog box. This electronic file was automatically sent to a designated HVMA clinician who printed and submitted it via facsimile to VAERS. This electronic-to-paper method was used to avoid the resource investment required to establish a PHIN-MS compliant messaging system that would allow fully electronic reporting from the EMR to VAERS.

Clinician Questionnaires

Two months after we implemented the system we administered an anonymous clinician survey. The purpose of the survey was to obtain clinician feedback on the system, including usefulness and the amount of time, if any, that it added to encounters. The survey was mailed to all HVMA pediatric clinicians who were eligible to receive alerts. The first wave was mailed 9 weeks after the alerting system was implemented. The survey was one page with mostly closed-ended questions, asking clinicians about their experience with the alerting system during the first 2 months post-implementation in terms of frequency of alerts, time added to the encounter, and whether the alerts tended to identify potential adverse events or not. We mailed a second wave of surveys to all clinicians after five weeks as a reminder for those who did not respond to the first mailing.

Analysis

We analyzed data on children 0–17 years of age who had a telephone or office visit encounter at HVMA between January 20–June 20, 2005 to determine the frequency of alerts. We excluded from the analysis 602 immunizations administered during separate visits within 14 days of each other to avoid confusion as to which immunization visit was associated with the potential adverse event. We did not send alerts to urgent care clinicians.

Comparison Data

We obtained information about the number of vaccine adverse events reported nationally for individuals under 18 years of age during the period of this intervention from the VAERS system (Jane Woo, personal communication). We computed the reporting rate (reports per immunization) by estimating the number of vaccine doses administered nationally to this age group during the same period.

Observations

Alerts and Reports

A total of 33,420 vaccinations were administered by 157 providers during 14,466 encounters. There were 5,914 follow-up telephone and office visit encounters within 14 days of a vaccination visit. Of these follow-up encounters, 686 (11.6%) had at least one diagnosis that was not on the exclusion code list, thus generating an alert. Because of an error that left some common codes off the exclusion list, the actual number of alerts sent was larger than this (36.2%). We report here the number of valid alerts sent. Overall, clinicians submitted 23 reports indicating potential vaccine adverse events (0.69 reports per 1,000 vaccine doses). If urgent care clinicians had received alerts, an additional 470 follow-up encounters would have triggered an additional 73 alerts. The most common adverse reactions reported were fever, rash and edema.

Comparison to National Reporting

During the period of this intervention, 4,439 reports were spontaneously submitted to VAERS for children and adolescents nationwide (personal communication, Jane Woo). During this period, we estimate that 38,194,286 vaccine doses were administered to this age group, yielding an approximate reporting rate of 0.12 per 1,000 vaccine doses.

The reports generated by our system were submitted on the date the child presented with the potential vaccine adverse event, compared to a median reporting delay to VAERS of 16 days (personal communication, Jane Woo). In addition, reports generated by our system provided complete data on vaccination date and onset date, compared to spontaneously-submitted VAERS reports that included 6% with incomplete vaccination date data and 10% with incomplete onset date data (personal communication, Jane Woo).

Clinician Survey

We mailed surveys to 176 pediatric providers, of whom 124 returned completed surveys, for a response rate of 74%. Of 101 clinicians who reported that they had received an alert, 35 (35%) reported that they answered the questions all or most of the time, but only 12 (12%) reported that they believed any of the alerts corresponded to an actual vaccine side effect. The most common reasons cited for not completing the questions were that the clinicians thought it was unnecessary and that they were too busy. The majority of respondents reported that it took 1–2 minutes to answer the questions if they felt the encounter was related to the recent vaccination and less than 1 minute if it was not. Thirty percent of clinicians reported that they received more than one alert per week and an additional 35% reported that they received a few alerts each month. Of those who had received an alert, 50% felt the alerts occurred too often. Of note, these responses were provided during the period that more alerts were being sent than intended.

Post implementation, of the 12 clinicians who felt that an alert corresponded to an actual vaccine adverse event, 8 (two-thirds) of them used the reporting feature to submit a VAERS report. Of those who said that they submitted a report, 50% said that they had never reported a vaccine adverse event to a public health authority before implementation of this elicited surveillance system.

Discussion

We implemented and evaluated a vaccine adverse event elicited surveillance and reporting system within a large multi-specialty group practice that used an EMR for routine ambulatory care. Optimal surveillance systems should be able to identify rare adverse events and assess causality. ¹⁷ Our system can do the first, and can facilitate the causality assessment by obtaining the clinician’s input while the event is fresh. We believe that this system increased the completeness of reporting because the rate of reporting (0.69 per 1,000 vaccine doses) was several-fold higher than our estimate of the national reporting rate in this age group during the period of our intervention (0.12 per 1,000 vaccine doses), or the overall rate of 0.11 per 1,000 net doses reported in the literature. ¹ It is possible, however, that the higher reporting rate we observed was unrelated to the elicited surveillance system. Among other potential explanations is the possibility that this practice’s patients brought more adverse reactions to the attention of their providers than is typical, or that other characteristics of the electronic medical record were responsible for a higher reporting rate. We have anecdotal evidence against the latter explanation, since some clinicians who submitted reports via this system told us that they had never reported a vaccine adverse event before this.

These reports were submitted on the date of the follow-up encounter and contained complete data on patient and provider demographics, adverse outcome(s) and vaccines (vaccination date, vaccine type[s], lot numbers and manufacturers). Our intervention provided reports that were more complete and timely than reports submitted spontaneously to VAERS. This automated system prompted clinicians to consider vaccine adverse events and facilitated reporting to VAERS by auto-populating the VAERS form with data within the EMR and the alert dialog box.

Another EMR-based system recently adapted to assess vaccine adverse events is the MediClass system. ¹¹ This system uses natural language processing and knowledge-based methods to identify potential vaccine adverse events in textual chart notes. This system is able to detect possible vaccine adverse events and may prove useful in epidemiological studies; however, this natural language processing system does not currently operate in real time and so it does not ask the clinician to note explicitly whether a vaccine adverse event is likely.

Computerized provider order entry with clinical decision support has been integrated into EMRs to improve medication safety. 13,14,15,16,18,19 Haller et al. implemented an incident reporting system into their EMR. ¹² Our system extends previous work on detection of adverse events and the implementation of clinical decision support systems to the vaccine safety arena, and includes a reporting component so that data are submitted to VAERS. Further validation and monitoring of the alerts would be beneficial to ensure that clinicians continue to use this enhancement, and to monitor for “alert fatigue.” Further work to move this system to a fully electronic messaging system from the EMR to VAERS would be valuable.

There are a number of limitations to this system. First, it is necessary that the EMR includes information about vaccines that have been administered. Second, we excluded diagnoses that occurred on the same day as the vaccination. We did this to avoid generating alerts based on diagnoses entered during the immunization visit that were unrelated to the vaccination. We were mindful that we might miss some immediate hypersensitivity-type reactions by excluding same day follow-up encounters. In addition, since alerts triggered only for follow-up encounters within 14 days following vaccination, we would miss adverse reactions that occurred beyond 14 days.

Finally, the alert is based on an exclusion code list. Therefore, any diagnoses on this list that are true vaccine adverse events would not be detected via this system. For instance, if a vaccine caused otitis media, this system would not prompt the clinician to consider it as a vaccine adverse event. We attempted to minimize this limitation through careful diagnosis code selection and revisions to the code list based on feedback we received from the provider survey. At the same time, we did not want to generate too many false positive alerts. We know that clinicians might stop answering the questionnaire if they receive too many alerts, so we plan to continue monitoring the numbers of alerts and we remain receptive to clinician feedback on codes that trigger alerts frequently. The number of alerts was acceptable after we corrected the error that generated more alerts than intended, as evidenced by the group practice’s decision to continue using this alerting system as part of routine practice after the trial evaluation ended.

There are a number of implications for adoption of this system into routine use. It is based in a proprietary EMR, which limits its portability. We were not able to implement complex logic because of the potential impact it would have on system response time. Thus, we were limited in the degree to which we could fine-tune the code to minimize false positive alerts. Upgrades to the EMR required upgrading our system each time as well. Moving this system outside the EMR would overcome some of these challenges, but would lose the real-time feature of this decision support system.

We believe that elicited surveillance via real time prompts to clinicians holds substantial promise, particularly when coupled with simplified reporting. We therefore believe it is worthwhile to add these capabilities to electronic medical records in the ambulatory setting.