Monitoring vaccine safety using the vaccine safety Datalink: Assessing capacity to integrate data from Immunization Information systems

Abstract

Background

The Vaccine Safety Datalink (VSD) uses vaccination data from electronic health records (EHR) at eight integrated health systems to monitor vaccine safety. Accurate capture of data from vaccines administered outside of the health system is critical for vaccine safety research, especially for COVID-19 vaccines, where many are administered in non-traditional settings. However, timely access and inclusion of data from Immunization Information Systems (IIS) into VSD safety assessments is not well understood.

Methods

We surveyed the eight data-contributing VSD sites to assess: 1) status of sending data to IIS; 2) status of receiving data from IIS; and 3) integration of IIS data into the site EHR. Sites reported separately for COVID-19 vaccination to capture any differences in capacity to receive and integrate data on COVID-19 vaccines versus other vaccines.

Results

All VSD sites send data to and receive data from their state IIS. All eight sites (100%) routinely integrate IIS data for COVID-19 vaccines into VSD research studies. Six sites (75%) also routinely integrate all other vaccination data; two sites integrate data from IIS following a reconciliation process, which can result in delays to integration into VSD datasets.

Conclusions

COVID-19 vaccines are being administered in a variety of non-traditional settings, where IIS are commonly used as centralized reporting systems. All eight VSD sites receive and integrate COVID-19 vaccine data from IIS, which positions the VSD well for conducting quality assessments of vaccine safety. Efforts to improve the timely receipt of all vaccination data will improve capacity to conduct vaccine safety assessments within the VSD.

1. Introduction

COVID-19 vaccine distribution is underway in the United States (US), and states are now offering vaccines broadly to all age-eligible individuals.[1], [2] In addition to the three COVID-19 vaccine products currently authorized by the US Food and Drug Administration (FDA) under the Emergency Use Authorization (EUA) mechanism (Pfizer-BioNTech, Moderna, and Janssen [3], [4], [5]), other vaccines in various stages of clinical development may be offered to individuals in the US in coming months.[6] Some of these products have been developed with platform technologies not previously used in vaccines, and none have had the benefit of long-term pre-licensure evaluation prior to being recommended to the general public. This has created a critical need for timely and accurate post-authorization assessment of COVID-19 vaccine safety.

The VSD is a collaborative safety surveillance system led by the Centers for Disease Control and Prevention (CDC), set up to conduct timely analyses of vaccine safety once vaccines are authorized or approved and administered in the general population. Past evaluation efforts have established that the VSD, which includes a population of about 12.4 million (3.8% of the US population),[7], [8] is a critical source for vaccine safety data and is among the first large-scale efforts to examine vaccine safety for the novel COVID-19 vaccines in the US.[9].

The VSD includes electronic health record (EHR) data from members of eight integrated healthcare systems, which allows for capture of vaccination records along with comprehensive data on demographics, comorbidities, and healthcare utilization.[10] Vaccination data systematically collected by VSD sites includes information on vaccines administered within the healthcare systems, as well as those administered externally, received through medical or pharmacy claims or data exchange with jurisdictional Immunization Information Systems (IIS), along with other validated external sources. IIS are population-based databases that record all immunization doses administered by participating providers to persons within a given geographic area.[11] Accurate capture of data on vaccines administered outside of the health system is critical for vaccine safety research, especially in the case of the COVID-19 vaccine, where many are receiving the vaccine in non-traditional health system settings.

An initial evaluation of VSD’s incorporation of state and local IIS data was conducted from 2009 to 2010, beginning prior to the novel influenza H1N1 pandemic, as part of a pre-pandemic planning exercise. Results of this evaluation showed that four of the [then] seven data-contributing VSD sites did not receive any vaccination data from their state or local IIS, in part due to lack of a consolidated state IIS in some regions, as well as issues related to adult immunization data capture and state privacy laws.[12] This finding led to efforts to improve the capacity for VSD sites to receive IIS data, recognizing that the VSD’s access to these externally-administered doses was critical for conducting real-time assessments of vaccine safety.

As observed with the pH1N1 vaccine, COVID-19 vaccines are being administered in a variety of settings outside of the medical home, including pharmacies and mass vaccination sites.[13] In preparation for this, the CDC established strict reporting guidelines for the COVID-19 vaccine.[14] Providers are required to document all COVID-19 vaccination within 24–72 h of administration; many states are facilitating this process through their existing IIS.[15]

With these concerted efforts to establish linkages between healthcare providers and IIS, and with the current emphasis on conducting timely safety assessments of COVID-19 vaccines in the US, we revisited the VSD’s capacity to integrate IIS data into its safety assessments in general and for the COVID-19 vaccines specifically. Our aim was to describe the status of IIS data exchange at each of the eight data-contributing VSD sites, focusing on the potential value of IIS data integration for COVID-19 vaccine safety evaluations conducted by the VSD.

2. Methods

VSD sites are independent research entities embedded within integrated health systems, with access to the electronic health and claims data from their respective health systems, as approved by Institutional Review Boards (IRBs) at each site. As of 2021, there are eight VSD data-contributing sites, representing eight different health systems. Each health system individually determines if vaccination data are shared with and/or received from their jurisdictional IIS, as well as whether, when and how those data are received, stored and incorporated into a patient’s EHR. The availability of these data to the health system then determines the VSD site’s ability to incorporate IIS data into VSD datasets and analyses. Available data is then translated into standardized codes (e.g. HL7 for vaccination data) and organized in a relational database, using standardized rules and definitions across VSD sites. This standardization is essential to the VSD’s ability to combine and compare data across sites.

Four VSD sites operate in states where there is one statewide IIS representing all the members served by the health system. (Table 1 ) The other four VSD sites require coordination with more than one IIS. Both HealthPartners (HP) and Kaiser Permanente Northwest (KPNW) provide medical care spanning two states (HP: MN and WI; KPNW: OR and WA) and thus have two statewide IIS representing their members. Two VSD sites are located in California, Kaiser Permanente Northern and Southern California (KPNC and KPSC, respectively), where the state population was represented in nine regional registries until a transition to an updated centralized IIS (California Immunization Registry 2, CAIR2) in 2017. CAIR2 consolidated seven regional registries into one centralized IIS, although two regional registries (the San Joaquin Regional Registry and the San Diego Regional Registry) continue to operate independently. KPNC and KPSC both include members who may be represented in CAIR2 or by one additional regional registry. The transitional state of the IIS in California over the past decade has presented ongoing challenges in establishing bidirectional data exchange between the VSD sites and the various IIS. Given that CAIR2 represents most of the state population, we decided to focus on this IIS for the present evaluation at the two California VSD sites.

Table 1.

Status of jurisdictional IIS data exchange at eight VSD sites – August 2021.

VSD Site	Health System (enrollment1)	IIS	Send data (to IIS)	Receive data (from IIS)2		Integrate data into VSD data3
				COVID-19 Vaccines	Other Vaccines3	COVID-19 Vaccines	Other Vaccines4
DH	Denver Health (179 K)	CIIS	All data	Automated	Automated	All data	All data
HP	HealthPartners (473 K)	MIIC	All data	Automated	Automated	All data	All data*
		WIR	All data	Trigger-based*	Trigger-based*	Reconciled data*	Reconciled data
KPCO	Kaiser Permanente Colorado (471 K)	CIIS	All data	Trigger-based*	Trigger-based*	All data	All data
KPNW	Kaiser Permanente Northwest (554 K)	ALERT IIS	All data	Trigger-based	Trigger-based	All data	All data
		WA IIS	All data	Trigger-based	Trigger-based	All data	All data
KPWA	Kaiser Permanente Washington (792 K)	WA IIS	All data	Automated	Automated	All data	All data
MFC	Marshfield Clinic (172 K)	WIR	All data	Automated	Automated	All data	All data
KPNC	Kaiser Permanente Northern California (4.2 M)	CAIR2	All data	Automated	Trigger-based	All data	Reconciled data
KPSC	Kaiser Permanente Southern California (4.5 M)	CAIR2	All data	Automated	Trigger-based	All data	Reconciled data

1 Enrollment represents total number of enrolled health plan members in 2021

2 Sites indicated how the data from IIS were received by the health system (Automated vs. Trigger-based, defined below).

Automated: IIS data on all health system members are shared on a routinized and automated batched basis (typically daily, weekly, or monthly)

Trigger-based: Specific healthcare events (e.g., ambulatory visits, scheduling visits, birthdays, etc.) will act as a trigger where a request for immunization data is pushed out to the IIS, from the health system, and all relevant immunization data is returned and made available in the EHR.

Trigger-based*: The asterisk denotes health system efforts are in progress to transition to an automated batched transfer of data.

3 Sites indicated whether IIS data are integrated into the VSD data (All data vs. Reconciled data, defined below.)

4 Other Vaccines refers to all vaccination data, other than COVID-19 vaccines.

All data: All IIS data received by the health system are available for use in VSD research.

All data*: The asterisk denotes vaccine data are available and in the process of being added.

Reconciled data: Only reconciled data (data that has been reviewed and accepted by a provider, manually or as part of a batched reconciliation) is made available for use in VSD research.

Reconciled data*: The asterisk denotes health system efforts are in progress to eliminate the need for individual reconciliation of the vaccine record before being available for use in VSD research.

CIIS: Colorado Immunization Information System

MIIC: Minnesota Immunization Information Connection

ALERT IIS: ALERT Immunization Information System (Oregon)

WAIIS: Washington Immunization Information System

WIR: Wisconsin Immunization Registry

CAIR: California Immunization Registry

2.1. VSD site survey

In January 2021, we surveyed each of the eight data-contributing VSD sites to evaluate the status of IIS data exchange and integration into VSD databases for each associated health system. Data managers at each VSD site were asked to provide the status on three IIS domains: (1) the health system’s ability to send data to their jurisdictional IIS; (2) the health system’s ability to receive data from their jurisdictional IIS; and (3) the VSD site’s ability to integrate jurisdictional IIS data into VSD research databases. For domains (2) and (3), sites were asked to distinguish between ‘all vaccines’ and ‘COVID-19 vaccines’, as it was anticipated that sites with barriers to complete vaccine data receipt and/or integration may have set up special capacity for collecting and sharing data on COVID-19 vaccines, specifically, given the urgency of accessing COVID-19 vaccination status for members.

Sites were also asked to describe whether IIS data receipt occurred through an “automated” or “trigger-based” process. In “automated” approaches there is a standardized, automated transfer of IIS data containing immunization records for all health system members to the health system on a regular basis (typically daily, weekly, or monthly). In the “trigger-based” approach, specific healthcare events (e.g., ambulatory visits, scheduling visits, birthdays, etc.) initiate a request for immunization data from the IIS to be transferred to the health system.

The survey question on IIS data integration asked sites to specify whether available IIS data were routinely integrated into the vaccine data that is made available to the VSD site, or whether the availability of IIS data was dependent on a reconciliation process, where providers must accept vaccine records before they are integrated into the vaccine data.

All VSD activities are approved by local institutional review boards.

3. Results

3.1. VSD site survey

All sites (n = 8, 100%) participated and site responses are represented in Table 1. Sending data to IIS and receiving data from IIS

All eight sites reported routinely sending data to their jurisdictional IIS. All sites also reported receiving IIS data. Of note, two of the sites (KPNW and HP) engage in data exchange with two separate state IIS and reported on the status of exchange separately for each IIS.

COVID-19 Vaccines: Six of the eight VSD health systems receive COVID-19 vaccination data through an automated batched process; two sites (KPCO and KPNW) receive the data through a trigger-based mechanism but KPCO noted that, as of August 2021,their health system is actively working on transitioning to an automated batched process specifically for COVID-19 vaccines. At KPNW, although the health system requires reconciliation of the IIS data before integrating it into the EHR, the VSD site data manager is provided with separate data tables from both the Oregon and Washington IIS which are included in VSD studies without waiting for provider reconciliation. Three VSD sites (HP, KPNC, KPSC) only recently transitioned to automated receipt of COVID-19 vaccine data due to the realization that health systems may not obtain complete capture of member COVID-19 vaccines on a timely basis if the IIS data is only being received following a specific health system trigger. At KPSC and KPNC, these efforts include a hybrid approach where COVID-19 vaccination IIS data are received through an automated query process daily for members with encounters and supplemented with a batch process to query the IIS for members without a COVID-19 vaccination record. At HP, the health system was recently granted permission to receive automated data from the Minnesota IIS, which represents the majority of externally reported vaccination data, however a small subset of data received from the Wisconsin IIS is only received following a trigger-based event.

Other Vaccines: Half of the sites (n = 4) receive data for all other vaccines through an automated batched process and half receive all (n = 2) or some (n = 2) other vaccination data through a trigger-based mechanism. For all non-COVID-19 vaccinations, KPNC, KPNW, and KPSC rely on a trigger-based mechanism where the IIS is queried for members with health system encounters. As above, the VSD data manager at KPNW is able to use the IIS data tables for inclusion in VSD studies, which bypasses the need to wait for reconciliation in the EHR.

3.2. Data integration

COVID-19 Vaccine data: All eight VSD sites are able to integrate IIS data for COVID-19 vaccines into VSD research databases, without requiring any reconciliation process following receipt at the health system. The only exception to this is the small subset of vaccination data that HP receives from the Wisconsin IIS, which is subjected to reconciliation prior to being integrated into the HP EHR.

Other vaccine data: Six sites are able to integrate all IIS vaccination data into VSD research databases. For the Denver Health, Marshfield Clinic, and Kaiser Permanente Washington sites, the IIS data are directly integrated into the EHR, following health system data quality and deduplication efforts. As noted above, at KPNW the IIS data from both the Oregon and Washington IIS are received and stored in a separate data table which is then made available to the VSD data manager. The fifth site (HP) was recently granted permission to initiate a weekly query of the Minnesota IIS for VSD-specific projects and is now able to integrate all vaccination data into research databases. For KPCO, once the trigger event occurs, IIS data is queried and automatically integrated into the member EHR and made available for use in VSD research.

Two sites (KPNC, KPSC) continue to rely on triggers for non-COVID-19 vaccination data. For KPSC and KPNC, as well as the small proportion of HP data coming in from the WI IIS, the state IIS is queried when a member has a triggering event and any non-COVID-19 vaccination data received must be reconciled before being integrated into the EHR and then available for use in VSD research databases.

4. Discussion

All eight VSD health systems now routinely send vaccine data to their state or local IIS, and all are receiving IIS data, which is then integrated into the EHR and available for use by the VSD sites. This represents an improvement in establishing linkages between VSD site EHRs and IIS in comparison to the evaluation from 2009 to 10, where only three of the seven data-contributing VSD sites reported such capacities[12]. The successful integration of IIS data into the EHR across all VSD sites primarily benefits clinical providers who are now using more complete capture of vaccination records to determine whether a patient presenting for care is due for any vaccines. But this improvement is also a hopeful step toward increasing completeness, timeliness, and accuracy of vaccination data used for studies of vaccine safety and is reassuring for interpretation of VSD research on COVID-19 vaccination.

Prior to the SARS-CoV-2 pandemic, only four of the eight VSD sites benefited from automated integration of IIS data, for all vaccines, into the health system’s EHRs. Three additional VSD sites have since seen positive shifts in the timelines of vaccination data capture and the integration of COVID-19 vaccination data (at a minimum) is now happening on an automated basis. With this, seven of the eight VSD sites now have ready access to timely COVID-19 vaccination data for use in VSD research studies. One VSD site, however, does still rely on a trigger-based approach for receiving any vaccination data, which means potential delays in the availability of the data to the EHR and the VSD site. While this site is ultimately able to integrate the vaccination data once it is received, there are opportunities to improve real-time access to IIS data. However, the ability to effect this change resides with the health system, and not the research staff working on the VSD. Further, individual state legislation and IIS logistical hurdles may prohibit the automated sharing of IIS immunization information in some cases, providing an additional challenge to gaining access to immunization records for research purposes.[16].

COVID-19 vaccination data is not currently required to undergo provider reconciliation, prior to being available for use in VSD research, However, several VSD sites do have a reconciliation process for other vaccines. It is not unexpected that healthcare systems have a process for reviewing and validating externally reported data before incorporating it into a medical record. Providers have sometimes expressed reluctance toward automatically incorporating IIS data due to concerns about data quality and limited representation of adults in IIS,[17] but these concerns may now be outdated. IIS data quality has improved markedly in the past 10 years,[18] and the reliability of these data has been demonstrated through research generating timely estimates of annual influenza coverage for the CDC and state and local public health programs.[19], [20] The CDC, together with the American Immunization Registry Association (AIRA) and participating IIS, has developed IIS functional standards which include specific criteria for evaluating the data quality of IIS in the US. AIRA collaborates with the IIS community to develop and implement measures and tests to assess IIS alignment with the standards, demonstrating improvement toward and achievement of standards and quality thresholds in the areas of transport, submission, query, clinical decision support, and data quality.[21], [22] A recent report on adult data capture in IIS shows that 100% of IIS include adult populations, which likely means that adults are increasingly represented in IIS contributions to VSD EHRs. These broad investments in IIS, paired with the federal mandate on reporting of COVID-19 vaccination and CDC’s encouragement of the use of IIS for managing COVID-19 vaccine allocation and administration, have positioned IIS well for being a consolidated repository of COVID-19 vaccination data for patients across the lifespan. Recognizing the importance of timely capture of externally administered COVID-19 doses within the EHR, the VSD site that currently relies on triggers to receive IIS data is actively advocating for a transition away from trigger-based to automated transfer of COVID-19 vaccination data. If successful, the benefits of IIS data inclusion can be fully evaluated and serve as a catalyst for moving entirely away from trigger-based systems for all vaccines, thereby improving timeliness of updated immunization data that is available for clinical decision making, Healthcare Effectiveness Data and Information Set (HEDIS) reporting, and for vaccine studies conducted by the VSD.

The value of integrating IIS data into a health system’s EHR for assessments of vaccine safety have not been evaluated. Research has demonstrated that including data from IIS can increase appropriate administration of vaccines, [23], [24], [25] by providing historical records of vaccination, thereby reducing misclassification of vaccination status. [13], [14], [15] These studies suggest that strengthening links between EHR and IIS are critical to improve the ability to conduct research related to vaccination status[26] and reinforce the need for integration of IIS data into EHR systems for timely and unbiased evaluations of vaccine safety. As described by McCarthy et al, the addition of IIS data can improve the accuracy of analyses conducted by the VSD.[12] Not only does it provide the benefit of a larger vaccinated population, thus greater power for detecting rare adverse events, but it reduces biases that come from misclassification of vaccination status. For example, studies that compare vaccinated to unvaccinated individuals are less likely to incorrectly classify someone as unvaccinated if they are including IIS data, and studies using self-controlled case series may avoid the inadvertent exclusion of vaccinated individuals. A previous assessment of pH1N1 vaccine data capture for two VSD sites determined that up to 30% of pH1N1 vaccination data for members aged 4–17 years came exclusively from the IIS; systematic exclusion of a subgroup this large from a study of vaccine safety would introduce substantial bias. With the current emphasis on use of IIS for centralized documentation of COVID-19 vaccination, it is possible that the data contributions from IIS would be even greater for the current pandemic vaccination program.

In contrast to other vaccines which are more commonly administered within these integrated healthcare systems, COVID-19 vaccines are offered in more diverse settings, free of charge, and outside of large medical centers, making the capture of data from IIS even more critical. Further, the CDC’s requirement that doses of COVID-19 vaccine be reported to a jurisdictional system (e.g. IIS) within 72 h of administration highlights the timeliness of vaccination data made available through linkages with IIS.[27] A preliminary review of data from KPNW, in February 2021, revealed that 63% of COVID-19 vaccinations (representing 61,757 doses of vaccine from December 14, 2020 through February 20, 2021) were made available by the IIS and were not included in the member’s EHR. It is likely that use of centralized mass vaccination clinics is driving substantial contributions of data from IIS at this site. With movement to more localized vaccine clinics, this contribution will likely decrease, however this finding reinforces the critical value of receiving IIS data particularly in these early stages of vaccination.

Limitations of this evaluation include the dynamic nature of COVID-19 data receipt and integration at each VSD health system, which made it difficult to capture the most current status of data availability to the VSD site. Some VSD sites had substantial challenges in accessing the appropriate IT staff to understand the structure of the data systems, making it hard to know how the receipt and integration of IIS data worked at their particular site.

For the remainder of the current COVID-19 vaccination campaign, and thereafter, the VSD will continue to be an important collaborative surveillance system for evaluating the safety of the COVID-19 vaccines. The health systems represented by the eight data-contributing VSD sites have made enormous progress in establishing automated data exchanges with jurisdictional IIS, increasing the likelihood of complete capture of routine and COVID-19 vaccination data and strengthening the rigor of studies evaluating vaccine safety. The current momentum around the need for real-time assessments of vaccine safety, like the recent publication by Klein et al[28], may serve as a tipping point to transition health systems toward the automated receipt of all vaccine data from IIS on an ongoing basis.

Funding disclosure:

Funding for this study was provided by the Centers for Disease Control and Prevention, Contract 200–2012-53584. Findings and conclusions of this study are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: [Michael Jackson reports a relationship with Sanofi Pasteur Inc that includes: funding grants. Allison Naleway reports a relationship with Pfizer that includes: funding grants. James Donahue reports a relationship with Janssen Inc that includes: funding grants. Lina Sy reports a relationship with Moderna that includes: funding grants. Lina Sy reports a relationship with GlaxoSmithKline that includes: funding grants. Lina Sy reports a relationship with Dynavax that includes: funding grants. Lina Sy reports a relationship with Seqirus Inc. that includes].