Advanced Clinical Decision Support for Vaccine Adverse Event Detection and Reporting

Abstract

Importance:

Reporting of adverse events (AEs) following vaccination can help identify rare or unexpected complications of immunizations and aid in characterizing potential vaccine safety signals.

Objective:

To create an electronic health record (EHR) module to assist clinicians with AE detection and reporting.

Design:

We developed an open-source, generalizable clinical decision system called Electronic Support for Public Health–Vaccine Adverse Event Reporting System (ESP-VAERS) to facilitate automated AE detection and reporting using EHRs. ESP-VAERS prospectively monitors patients’ electronic records for new diagnoses, changes in laboratory values and new allergies for up to 6 weeks following vaccinations. When suggestive events are found, ESP-VAERS sends a secure electronic message to the patient’s clinician. The clinician is invited to affirm or refute the event, add comments, and if they wish, submit an automated, pre-populated electronic case report to the national VAERS. High probability AEs following vaccination are reported automatically even if the clinician does not respond.

Setting:

We implemented ESP-VAERS in December 2012 at the MetroHealth System, an inpatient and outpatient integrated healthcare system in Ohio with nearly 1 million encounters per year. We queried the VAERS database to determine MetroHealth’s baseline reporting rates from 1/2009–3/2012 and then assessed changes in reporting rates with ESP-VAERS.

Participants:

All patients receiving vaccinations between 12/04/2012 and 08/03/2013 and their clinicians.

Exposure:

ESP-VAERS

Main outcome and measure:

The odds ratio of a VAERS report submission during the intervention period compared to the comparable pre-intervention period.

Results:

In the 8 months following implementation, 91,622 vaccinations were given. ESP-VAERS sent 1,385 messages to responsible clinicians describing potential AEs (15 per 1000 vaccinations, mean 0.4 alerts per clinician per month (range 0–8)). Clinicians reviewed 1,304 (94%) messages, responded to 209 (15%), and confirmed 16 for transmission to VAERS. An additional 16 high probability AEs were sent automatically. Reported events included seizure, pleural effusion, and lymphocytopenia. The odds of a VAERS report submission during the pilot period were 30.2 (95% CI, 9.52–95.5) times greater than the odds during the comparable pre-pilot period.

Conclusion and relevance:

An open-source EHR-based clinical decision support system can increase AE detection and reporting rates in VAERS.

Introduction

Routine vaccination is a cornerstone of preventive healthcare. Vaccines have dramatically decreased the incidence of many serious infectious diseases.¹ While pre-licensure clinical trials are helpful to characterize the basic safety profile of vaccines, rare adverse events (AEs) may only become apparent after widespread use in the community. To this end, rigorous post-marketing vaccine safety surveillance is critical to building public and professional trust in vaccines.

The Centers for Disease Control and Prevention (CDC) and the US Food and Drug Administration (FDA) jointly operate the Vaccine Adverse Event Reporting System (VAERS). VAERS is a passive reporting system that accepts spontaneous AE reports from clinicians, pharmaceutical companies, and the public. VAERS reports vary in quality and completeness and under-reporting, especially of mild and self-limiting AEs, appears common.^2,3 Reasons for clinician under-reporting might include failure to associate an acute health event to recent vaccines, lack of awareness of VAERS, the misperception that only serious events should be reported, or lack of time to report. VAERS reports often lack critical data such as vaccine lot numbers and the precise date of vaccination.⁴ Currently, no widespread, automated mechanisms exist to facilitate detection and electronic reporting of AEs to VAERS by clinicians. Consequently, the utility of VAERS data is diminished by substantial under-reporting and sparse documentation of patients’ clinical status and potential explanations of their conditions.^2,5

One potential adjunct is to take advantage of the increasing penetration and functionality of electronic health record (EHR) systems. Adding surveillance and AE reporting capacity to EHRs offers a practical and efficient means to monitor large numbers of patients, integrate AE reporting into physicians’ workflow, elicit clinician comments in a timely manner, and efficiently submit reports. We describe the development and implementation of an EHR-based AE detection and reporting system called Electronic Support for Public Health–Vaccine Adverse Event Reporting System (ESP-VAERS) that builds upon an automated vaccine AE surveillance system based in an ambulatory electronic medical record.⁴ ESP-VAERS identifies possible AEs following vaccination, prompts clinicians for input when appropriate, and has the capability to submit secure electronic reports to VAERS. The ESP-VAERS design is open-source and compatible with any modern EHR system.

Methods

Study setting

ESP-VAERS was implemented in the MetroHealth System, a tertiary care academic health system in northeast Ohio that provides a full range of inpatient and outpatient primary and specialty care services and has a network of community health centers that primarily focus on primary care and preventive health services. The MetroHealth system includes >400 primary care and specialty care physicians and 373,000 established patients with nearly 1 million medical encounters per year. Both the academic medical center and the community provider network are served by an integrated EpicCare EHR system.⁶ All adult and pediatric patients served by MetroHealth were included in the study.

MetroHealth utilizes the Electronic Support for Public Health network (ESPnet) public health surveillance platform.⁷ ESPnet is open-source software that facilitates automated detection and reporting from EHRs to health departments.^8,9 ESPnet is populated nightly with structured data from all patients seen throughout the healthcare system within the preceding 24 hours. These data include demographics, diagnoses, laboratory reports, prescriptions, and vaccines. ESPnet organizes these data into tables, applies algorithms to detect events of public health interest, and when appropriate, sends electronic case reports to the state health department. MetroHealth has been using ESPnet for automated notifiable disease surveillance and electronic reporting to the Ohio Department of Health since 2009.¹⁰

Development of ESP-VAERS

We developed a new ESPnet module, ESP-VAERS, to monitor patients’ EHRs for 42 days following each vaccination for possible new onset AEs. ESP-VAERS identifies every vaccine administered and prospectively records the patient’s new diagnostic codes, laboratory tests, allergy lists, and medication prescriptions during the 6-week surveillance period. We developed and applied algorithms designed to detect both expected and unexpected AEs. When a possible AE is identified based on the algorithms, it is recorded in a registry database table and the clinician is securely messaged electronically through the EHR to consider if the event should be reported to VAERS (Figure 1).

Figure 1.

Flowchart of possible vaccine related adverse events

Algorithm development

We divided ICD-9 diagnosis codes into 3 categories: 1) codes for severe and/or potential high probability diagnoses that have previously been associated with vaccines,^11,12 2) codes for diagnoses of undetermined significance that could conceivably be associated vaccines, and 3) codes for diagnoses that are not associated with receipt of vaccines (e.g., well child visits, fractures, etc.) (Appendix). Category 1 included ICD-9 codes for severe and potential high probability diagnoses included in the VAERS Table of Reportable Events Following Vaccination¹¹ and the Vaccine Injury Table¹² and ICD-9 codes defined as immunization reactions. The onset intervals and exclusion criteria for each diagnosis were based on the VAERS Table of Reportable Events Following Vaccination.¹¹ Category 2 included all ICD-9 codes other than the potential high probability and severe event codes in Category 1, the extremely low probability codes in Category 3, and any codes present in the individual patient’s record within the preceding 36 months. Onset intervals and additional exclusion criteria for these codes were based on clinical expertise, literature review, and review of alerts using retrospective and pilot project data. Codes for routine care and diagnoses very unlikely to represent AEs following vaccination were placed into Category 3 and did not trigger an alert.

We assessed for significant changes in key laboratory tests following vaccination including hemoglobin, white blood cell count, platelet count, creatinine, alanine aminotransferase, aspartate transaminase, alkaline phosphatase, bilirubin, sodium, potassium, calcium, creatinine kinase and partial thromboplastin time. We set different thresholds for significant changes for each laboratory test (Appendix). In general, a significant change was defined as substantial worsening compared to the patient’s most recent test result or an abnormal result without a prior record of any results since ESP’s inception on January 1, 2009. ESP also followed the patient’s coded allergy list for 30 days after vaccination. The appearance of a new allergy to the index vaccine was considered suggestive of an adverse event.

Possible AE messages were sent to clinicians’ Epic InBasket, a secured system for clinicians to communicate with one another regarding patient care within the EHR.⁶ Each message contained a brief summary of the patient, the vaccination and the suspected AE, and a hyperlink to a web form with more information. Selecting the hyperlink opened a web form with details about the vaccination, adverse event, and options to approve and send the notification to VAERS with optional comments or cancel the notification and add optional comments. The hyperlink also included 3 questions about the utility and acceptability of the message:

1. Was this message helpful? (Yes/No)

2. Did it interrupt your workflow? (Yes/No)

3. Has the number of messages recently been Appropriate or Too Frequent?

Reports of severe and potential high probability AEs following vaccination, such as anaphylaxis and encephalitis, were submitted automatically even if the clinician did not respond. Reports sent to VAERS included patient’s demographics, vaccine, lot number, date of vaccination, possible adverse event, date of adverse event, and any free-text comments provided by clinicians. If multiple vaccines were given simultaneously, information on all vaccines were included in the report as per the standard VAERS protocol.¹³ Cases approved for transmission were sent to VAERS as HL7 version 2.3.1 messages using CDC’s PHIN-MS secure messaging protocol.¹⁴

Following implementation and testing by co-investigators and clinician collaborators, MetroHealth clinicians were offered a brief training session to ensure that they were familiar with the project and comfortable managing the notifications. Support for clinicians using the ESP-VAERS system was provided through handouts, teaching sessions, and an EHR message to all clinicians. The ESP-VAERS study period ran from December, 04, 2012, the date that the system was implemented in all MetroHealth practices, to August 03, 2013.

Historical reports

We queried the VAERS database for all reports sent from the state of Ohio from January 01, 2009 to March 31, 2012, limiting the search to reports sent from December 4^th -August 3^rd in each year to match the study period. Based on the city, the name of the clinic or hospital, the address of the clinic or hospital and/or the provider name, we identified reports sent from the MetroHealth System and verified these reports in the EHR. The number of vaccinations in the historical period was determined by running ESP-VAERS on all MetroHealth data from January 01, 2009 to August 03, 2009, December 04, 2009 to August 03, 2010, December 04, 2010 to August 03, 2011, and December 04, 2011 to March 31, 2012.

Statistical analysis

Using logistic regression, we compared the odds of a VAERS report after vaccination in the pilot period to the odds of a report in the historical period. We controlled for the time of year by restricting the historical reports to the period of calendar time relevant to the pilot.

The study was approved by the Institutional Review Boards of the MetroHealth System and Harvard Pilgrim Health Care Institute. We employed best-practice methods to ensure web application security, strong message encryption, and other techniques to ensure that these data were protected at or beyond standards set by the American National Standards Institute Healthcare Information Technology Standards Panel.^15,16

Results

During the 8-month study period, 91,622 vaccinations were administered. The most common vaccines given were the combined tetanus, diphtheria and pertussis vaccine (Tdap) (15,279 doses) and the inactivated influenza vaccine (13,629 doses) (Table 1). ESP-VAERS sent 1,385 messages to responsible clinicians describing possible AEs following vaccination, corresponding to a rate of 15 messages per 1,000 vaccinations (Figure 2). The average number of alerts per clinician was 0.4 messages per month, and the range was between 0 and 8 messages per month.

Table 1.

Vaccines administered at MetroHealth between December 04, 2012 and August 03, 2013

Vaccines Administered* N = 91,622
Vaccine	Count (%)
Tdap	15,279 (17%)
IIV	13,629 (15%)
PCV13	7,561 (8%)
Hib	7,051 (8%)
Hepatitis A Pediatric	6,508 (7%)
HPV	5,424 (6%)
DTaP-Hepatitis B-IPV	5,099 (6%)
PPSV23	4,230 (5%)
Pentavalent rotavirus	3,087 (3%)
Other	23,754 (26%)

Abbreviations: Tdap-combined tetanus, diphtheria and pertussis; IIV- inactivated influenza vaccine; PCV- pneumococcal conjugate vaccine; Hib-haemophilus influenzae type B; HPV-human papillomavirus; DTaP- diphtheria, tetanus and pertussis; IPV-inactivated polio vaccine; PPSV- Pneumococcal polysaccharide vaccine.

^*Some adverse events were counted against multiple vaccines if the patient received more than one vaccine at the index encounter.

Figure 2.

Flowchart of the vaccine related data from MetroHealth between December 04, 2012 and August 03, 2013

*93% of the messages opened

Clinicians opened 1,304 messages (94%), responded to 209 messages (15%), and confirmed 16 for transmission of reports to VAERS (Table 2). The diagnoses for transmitted reports included seizure, eosinophilia, Bell’s palsy, pleural effusion, lymphocytopenia, leukopenia, cellulitis, febrile seizure, rash, viral exanthem, rubella symptoms, fever and mild neutropenia, fever and hypothyroidism. Of the 16 confirmed messages, 15 included custom comments from the healthcare provider. The remaining 193 alerts were designated by clinicians as not associated with the vaccination. The most common diagnoses, designated as not associated with the vaccination, included bronchospasm, nonspecific skin eruption, cellulitis, fever and seizure. These five diagnostic categories accounted for 22% of all of the alerts that were considered not associated with the vaccination.

Table 2.

Clinician confirmed VAERS reports at MetroHealth between December 04, 2012 and August 03, 2013

VAERS Reports: Clinician Confirmed (n=16)
Adverse Event	Vaccine(s) Administered
Seizure	IIV
Eosinophilia	IIV
Eosinophilia	PCV13
Bell’s palsy	Tdap
Pleural effusion	IIV, Tdap
Lymphocytopenia	Tdap
Leukopenia	Tdap
Cellulitis	DTaP-IPV, MMRV
Febrile seizures, rash	MMR, Varicella
Viral exanthem, rubella symptoms	DTaP, Hepatitis B, IPV, Hib, IIV, MMR, Varicella, PCV13
Fever and mild neutropenia	PPSV23
Rash	DTaP-IPV, MMRV
Rash	Hepatitis A, Hib, MMR, PCV13, Varicella
Rash	Hepatitis A, MMRV
Fever	DTaP-HepB-IPV, Hib, PCV13, Rotavirus
Hypothyroidism	Shingles, Tdap

Abbreviations: IIV- inactivated influenza vaccine; PCV- pneumococcal conjugate vaccine; Tdap- combined tetanus, diphtheria and pertussis; DTaP- diphtheria, tetanus and pertussis; IPV-inactivated polio vaccine; MMRV-measles, mumps, rubella and varicella; MMR-measles, mumps and rubella; PPSV- Pneumococcal polysaccharide vaccine; Hib- haemophilus influenzae type B.

An additional 16 high probability AEs following vaccination were sent automatically (Table 3). All of the high probability reports were coded by clinicians as immunization reactions. These included: possible allergic reaction, fever alone, fever and a local reaction, fever and rash, cellulitis and fussiness. Thus, 32 VAERS reports were sent for 91,622 vaccines over an 8-month period for a net reporting rate of 34.9 VAERS reports per 100,000 vaccinations (Table 4).

Table 3.

Automatically sent VAERS reports from MetroHealth between December 04, 2012 and August 03, 2013

VAERS Reports: Automatically Sent (n=16)
ICD-9 Codes for	Adverse Event	Vaccine(s)
Immunization Reaction	Reaction to vaccine - possible allergic reaction	IIV
	Fever, local immunization reaction	DTaP-IPV, MMRV
	Rash, fever	Hepatitis A, MMR, Varicella
	Rash, fever	Hepatitis A, MMR, Varicella
	Fever	DTaP-Hepatitis B-IPV, Hib, PCV13, Rotavirus
	Cellulitis	IIV, Meningococcal
	Fussiness	Hepatitis A, MMRV
	Local immunization reaction	DTaP-IPV, MMRV
	Local immunization reaction	DTaP-IPV, HepA, MMRV
	Local immunization reaction	Hepatitis A, MMR, Varicella
	Local immunization reaction	HPV, Meningococcal, Tdap
	Local immunization reaction	IIV
	Local immunization reaction	IIV, PPSV23, Tdap
	Local immunization reaction	PPSV23
	Local immunization reaction	Shingles
	Local immunization reaction	Tdap

Abbreviations: IIV- inactivated influenza vaccine; DTaP- diphtheria, tetanus and pertussis; IPV-inactivated polio vaccine; MMRV-measles, mumps, rubella and varicella; MMR-measles, mumps and rubella; Hib- haemophilus influenzae type B; PCV- pneumococcal conjugate vaccine; HPV-human papillomavirus; Tdap-combined tetanus, diphtheria and pertussis; PPSV- Pneumococcal polysaccharide vaccine.

Table 4.

VAERS reports sent from MetroHealth during the study period as compared to the pre-study period.

Results
MetroHealth Data	VAERS Reports	Time period	Reports/ Month	Vaccinations	Reports/ Vaccination
Pre-pilot Data	3	27 months	0.11 report/ month	274,080 vaccines	1.09 reports/ 100,000 vaccines
Pilot Data	32	8 months	4 reports/ month	91,622 vaccines	34.9 reports/ 100,000 vaccines

Of the 1,160 alerts without a response and not automatically sent, the most common diagnoses included nonspecific skin eruptions, eosinophilia, seizure, fever, leukopenia and lymphocytopenia, accounting for 18% of the 1,160 alerts with no response.

We identified three reports in VAERS that were sent by MetroHealth clinicians from January 01, 2009 to August 03, 2009, December 04, 2009 to August 03, 2010, December 04, 2010 to August 03, 2011, December 04, 2011 to March 31, 2012 prior to the inception of ESP-VAERS. During this period, 274,080 vaccines were administered at MetroHealth corresponding to a reporting rate prior to initiation of the ESP-VAERS pilot of 1.09 reports per 100,000 vaccinations. The odds of a VAERS report submission during the study period were 30.2 (95% CI, 9.52–95.5) times greater than the odds during the comparable pre-intervention period.

Many clinicians felt that the messages were helpful (115/209; 55%) and did not interrupt workflow (116/209; 56%). The majority of the clinicians (166/209; 79%) stated that the number of messages was appropriate.

Discussion

Our implementation of ESP-VAERS demonstrates that EHRs can facilitate identification of possible AEs following vaccination, engage clinicians within their existing workflows to comment on events, and generate and submit secure reports to VAERS. Implementation of ESP-VAERS was associated with a 30.2 fold increase in the odds of MetroHealth submission of AE reports to VAERS. Many reported AEs were medically significant, including seizure, lymphocytopenia and Bell’s palsy, and other AEs such as pleural effusions have only rarely been reported following vaccination.¹⁷ Such reporting might provide hypothesis generating information to CDC and FDA that results in further assessment of a possible association. We determined that an open-source EHR-based clinical decision support system improves the detection and reporting of AEs following vaccination.

Through this pilot, we developed new logic for automated identification of potential high probability and possible AEs following vaccination, we developed capacity for querying clinicians via the EHR from an external source, and we implemented automated secure HL7 messaging to VAERS. Such a system is dependent upon clinician acceptance and participation. Many of the clinicians found that the messages were helpful and did not interrupt their workflow. Most found that the number of messages sent was appropriate.

EHRs offer an increasingly available opportunity for low cost monitoring and reporting of AEs. EHRs offer three potential advantages over existing claims based systems: 1) EHRs have access to much richer data streams, including vital signs and laboratory test results, 2) EHR data streams are updated in near real-time, permitting more timely AE detection compared to claims databases, and 3) EHR systems can query patients’ clinical providers for comments on possible AEs in near-real time, adding richness and specificity to AE detection.

ESP-VAERS provides potential advantages over traditional passive reporting systems: ESP-VAERS prompts clinicians to recognize possible AEs and automates components of the VAERS reporting process. Thus, the increased rate of VAERS report submission during the intervention period might be due to a combination of alerting clinicians to possible AEs that they may have otherwise missed or disregarded, notifying clinicians to consider reporting the AE to VAERS, and facilitating the report submission process. In addition, because they contain rich data on large numbers of patients, automated EHR surveillance systems can facilitate estimation of possible AE incidence densities in defined populations. In contrast, data from passive surveillance yields adverse event counts only for those reports submitted. Finally, ESP-VAERS reports can be pre-populated with information from the EHR, allowing for additional comments from a clinician, thus limiting the likelihood of reports with missing key data fields. This is in contrast to traditional passive VAERS reports which have been found to be missing essential variables including age, date of birth, vaccination date and date of adverse event onset.¹⁸ As ESP-VAERS is not embedded within an EHR system, but rather sits outside of the EHR, it does not burden routine EHR operations. Use of EHR systems is increasingly common, so a generalizable and portable automated AE surveillance approach based on existing EHR systems potentially offers a feasible solution to quickly ramp up AE surveillance to provide clinically rich reports at relatively low marginal cost.

Limitations to ESP-VAERS include the large number of alerts and the inter-related risk of clinician alert fatigue. Although clinicians opened 94% of the AE reports we sent, they only elected to respond to 15%. The current algorithm is restricted to single codes for new diagnoses and historical events. Increasing the sophistication of the algorithm and program to use combinations of ICD-9 codes and laboratories for diagnoses and exclusion criteria may reduce the number of false positive alerts that clinicians are receiving. Furthermore, the upgrade to ICD-10 may decrease the false positive alerts given the greater richness of ICD-10 codes for specific diagnoses compared to ICD-9 codes.¹⁹ Further investigation into the large number of alerts that lack a response is also warranted, including a better understanding of why some alerts were opened and some were not.

An open-source, EHR-diagnostic clinical decision support system with advanced predictive algorithms using ICD-9 codes, laboratory values, allergies and the potential for medication prescriptions can significantly improve the detection and reporting of AEs following vaccination to VAERS. This type of open-source, advanced clinical decision support system represents new opportunities for clinical decision support to improve public health.