Assessing the Immunization Information System and electronic health record interface accuracy for COVID-19 vaccinations

Abstract

Objective

Our objective is to assess the accuracy of the COVID-19 vaccination status within the electronic health record (EHR) for a panel of patients in a primary care practice when manual queries of the state immunization databases are required to access outside immunization records.

Materials and Methods

This study evaluated COVID-19 vaccination status of adult primary care patients within a university-based health system EHR by manually querying the Kansas and Missouri Immunization Information Systems.

Results

A manual query of the local Immunization Information Systems for 4114 adult patients with “unknown” vaccination status showed 44% of the patients were previously vaccinated. Attempts to assess the comprehensiveness of the Immunization Information Systems were hampered by incomplete documentation in the chart and poor response to patient outreach.

Conclusions

When the interface between the patient chart and the local Immunization Information System depends on a manual query for the transfer of data, the COVID-19 vaccination status for a panel of patients is often inaccurate.

BACKGROUND

This project started with a seemingly simple idea in 2 primary care clinics in December 2021, in response to the Omicron wave of the COVID-19 pandemic. We decided to identify adult patients within our practices who were not vaccinated against COVID-19, then provide targeted outreach to those patients. However, there were unanticipated difficulties identifying unvaccinated patients. This led to an assessment of the accuracy of the COVID-19 vaccination status in our EHR, with a focus on the interface between the patient chart and the local Immunization Information Systems (IIS). The EHR and IIS interface has implications beyond COVID-19 vaccines. Accurate information at the point of care for all vaccines impacts immunization recommendations, targeted outreach, and population risk assessment.^1,2

The utility of a robust exchange of information between immunization databases and other health information systems is illustrated by researchers in Wisconsin who quantified the COVID-19 vaccination rates for people experiencing homelessness by linking the Homelessness Management Information System with the Wisconsin Immunization Registry. They discovered significantly lower rates of immunization among people experiencing homelessness than in the general population and provided targeted outreach with mobile vaccination clinics.³ Comprehensive immunization records within the EHR also affect the administration of the new 20-valent and 15-valent conjugate pneumococcal vaccines. The decision to vaccinate adults with one of the new vaccines depends on whether the patient has received the 13-valent conjugate vaccine or the polysaccharide pneumonia vaccine previously.⁴ Comprehensive vaccine information within the EHR assists with clinical decision making.

Since vaccinations are given in a variety of settings outside the primary care clinic, comprehensive vaccination records within a patient chart depend on knowledge of and access to local repositories for immunization data. These repositories are part of the IIS network. In a 2015 survey of Family Physicians and General Internal Medicine Physicians, just 61% of Family Physicians and 22% of General Internists thought the IIS could help determine patient’s immunization needs, indicating a critical gap in knowledge of the IIS. The most frequently reported barrier was the IIS not updating the EHR.⁵ More recently, efforts have been made to increase use of the IIS across healthcare entities.^6,7 This is especially important as more vaccines are given in pharmacies and other vaccination sites.^8,9 The United States IIS network has seen an estimated 10-fold increase in submissions and queries since the start of the COVID-19 pandemic, highlighting its important role in immunization informatics.¹

The IIS within the United States has undergone considerable adaptation and growth since its inception, which was in part spurred by the measles insurgence of 1989 to 1991. During that outbreak, a delay in immunization of a child due to unclear vaccination status led to the child’s death from measles, which instigated renewed interest in the development of a system to track vaccinations.¹⁰ The current immunization tracking system is a network of 62 independent databases housing immunization records for a specific geopolitical area, interconnected to the site of vaccination, healthcare providers, patients, and public health entities. Each IIS is independently operated, therefore impacted by local policy and resource limitations.^11,12 Data are exchanged between systems on a cloud-based routing service called Immunization Gateway. Population level data exchange on the Immunization Gateway is a rapidly evolving aspect of immunization informatics and is identified in Dr Lenert’s article on public health and health system collaboration as a key aspect of the next level of IIS utility.¹³

The transfer of immunization information from the IIS to the EHR occurs via a manual query or automatic query. The manual query is done by a user of the EHR by selecting to initiate a query of the IIS while within a patient’s chart. An automatic query is triggered by parameters set by the users of the electronic health system and does not require any action by individual providers. Once the immunization data is brought into the EHR system, it needs to be reconciled into the patient’s individual chart. The reconciliation process depends on data coherence between the IIS, EHR, and the individual’s chart. The reconciliation process can be done automatically if certain criteria are met, or manually.

In December 2021, at the start of this project, the adult vaccine information stored in the state IISs was only brought into our EHR by a user initiated manual query of the state database. The object of this study is to assess the accuracy of the COVID-19 vaccination status for a panel of patients in a primary care practice when the transfer of information relies on user initiated manual queries of the IIS.

MATERIALS AND METHODS

On December 15, 2021, we ascertained the COVID-19 vaccination status, as defined within our EHR, for 9181 patients established in our University of Kansas Family Medicine and Internal Medicine clinics. We included all patients ages 18 or older with a Wyandotte County, Kansas address. The EHR system used is Epic. The COVID-19 vaccination status is defined within the EHR in 4 different ways: fully vaccinated, partially vaccinated, postponed, or unknown. The status is “unknown” if there is no record of a COVID-19 vaccine. If a vaccine is given within our health system, the placement of the order and documentation of the procedure automatically populates the chart with the immunization record. For vaccines given outside the health system, either self-reporting by the patient or a query of the IIS are the mechanisms for bringing the information into the medical record. When the IIS is queried, if there is no record of a COVID-19 vaccine administered, the vaccine status remains “unknown” within the EHR. There is not an option to change the status to unvaccinated.

We defined fully vaccinated as one shot of the Janssen (also known as the Johnson and Johnson) vaccine or 2 shots of mRNA-1273 (Moderna) or the mRNA-BNT162b2 (Pfizer) vaccines. Partially vaccinated patients might be patients who had one shot and chose to not complete a second dose, or patients who were partially through the series during the time of the query. The “postponed” vaccination status is sometimes used by providers if a patient has declined the vaccine, but the postponed status is not used consistently. Booster shots were not factored into the vaccination status.

At the time of the initial assessment on December 15, 2021, the immunization data for vaccines administered within our health system was easily visible and searchable within the EHR. However, if the patient was vaccinated outside the health system, the data was only brought into our EHR by self-reporting or a manual query of the IIS on an individual basis, usually in anticipation of an office visit. Although many clinicians were doing this manual query of the IIS at the time of the office visit, it was not done systematically or consistently.

The primary intervention of this study was performing a manual query of the IIS for all patients with “unknown” vaccination status. Since our clinic is in Kansas, very near the state line with Missouri, our EHR can simultaneously query the Kansas and Missouri Immunization Information Systems. The manual query process starts with accessing the patient chart, followed by activating the query for outside immunizations and reconciling the data pulled into the EHR (see Figure 1).

Figure 1.

Process for a manual query of the Immunization Information System from the electronic health record.

RESULTS

The initial population of 9186 adult patients living in Wyandotte County, Kansas has similar demographics to the subgroup of patients with “unknown” vaccination status, as shown in Table 1. Race data is self-reported and taken from the EHR.

Table 1.

Demographics of adult patients established within the primary care clinics, living in Wyandotte County Kansas, at the time of the initial data query on December 15, 2021

	All adult clinic patients n (%)	Adult clinic patients with unknown vaccination status n (%)
Age (average)	50.1	45.5
Sex
Male	3624 (39.5)	1683 (40.9)
Female	5562 (60.5)	2431 (59.1)
Race
Black or African American	3371 (36.7)	1514 (36.8)
White or Caucasian	3237 (35.2)	1265 (30.7)
Asian	577 (6.7)	310 (7.5)
Two or more	104 (1.1)	54 (1.3)
Other/declined	1897 (20.7)	971 (23.6)
Total	9186	4114

Of the 9186 patients, 4114 (44.8%) had “unknown” vaccination status and 5065 (55.1%) were either partially or fully vaccinated. In comparison, the CDC reported that as of December 31, 2021, 85.6% of adults ages 18 years and older nationwide and 82.7% of adults in Kansas received at least one dose of a COVID-19 vaccine.¹⁴ This indicated our clinic vaccination rates were potentially significantly lower than national and state vaccination rates. However, as shown in Table 2, a manual query of the state IIS for the 4114 patients with “unknown” vaccination status indicated 1807 (43.9%) were already vaccinated, bringing our vaccination rate to 74.8%, closer to the state and national adult vaccination rates.

Table 2.

Vaccine status breakdown before and after manual query of IIS

	Initial vaccination status n (%)	Results of manual query of IIS for patients with “unknown” vaccination n (%)	Post query vaccination status n (%)
Vaccinated total	5065 (55.1)	1807 (43.9)	6872 (74.8)
Fully vaccinated	4613 (50.2)	1491 (36.2)	6104 (66.4)
Partially vaccinated	452 (4.9)	316 (7.7)	768 (8.4)
“Unknown” status	4114 (44.8)	2292 (55.7)	2292 (25.0)
“Postponed”	7 (0.1)	4 (0.1)	11 (0.1)
Patient deceased	0 (0.0)	11 (0.3)	11 (0.1)
Total	9186	4114	9186

The next step was an attempt to determine if the data coming from the IIS was comprehensive. Was the IIS capturing most of the vaccines given to patients outside the health system? To accomplish this, we tried to determine if the 2292 patients who still had “unknown” vaccination status after our manual query were unvaccinated, or if they were vaccinated and the immunization was not captured by the IIS.

Two methods of clarifying the vaccination status were tried: chart review or direct patient contact (Figures 2 and 3). Both methods were labor intensive and of limited value due to incomplete documentation or lack of response to outreach. There were slightly better outcomes with phone calls and patient portal messages, but these efforts were considerably more time consuming. Only a small proportion (3–9%) of the patients with no record of a vaccine in the IISs indicated they had received the vaccine, which suggests the IIS is capturing most of the vaccines administered. Neither method achieved statistical significance.

Figure 2.

Results of a chart review of 100 charts of patients with “unknown” vaccination status after the manual query of the IIS.

Figure 3.

Results of outreach to 170 patients with “unknown” vaccination status after the manual query of the IIS. Outreach consisted of phone messages or patient portal messages or both.

DISCUSSION

This project started with what seemed like a simple goal: identify patients not vaccinated against COVID-19, then provide targeted outreach to those patients. However, it was much harder than anticipated to obtain an accurate list of unvaccinated patients. At the time of the initial assessment in December 2021, the interface between the clinic EHR and the IIS was only activated for adult patients by a manual query of the IIS on an individual, patient-by-patient basis. When a manual query for all 4114 patients with unknown vaccination status was done, we found 44% of our patients with “unknown” vaccination status were previously vaccinated. The inaccuracy of our lists of potentially unvaccinated patients was a barrier to targeted patient outreach. Once the information from the IIS is pulled into the EHR, it appears reasonably comprehensive. Efforts to quantify exactly how comprehensive were hampered by difficulty obtaining vaccination information from chart review or patient outreach.

Since embarking on this study, the University of Kansas Health System has improved the interface between the IIS and the EHR. As of January 20, 2022, both the Kansas and Missouri IIS are queried automatically when a patient arrives within our health system for any appointment type. This includes arrival for office visits, labs, x-rays, and hospital admissions. An automatic query dependent on a patient’s interaction with the health system may still miss patients that are not coming to the health system often but will capture significantly more patients than a connection that relies on manual queries. However, by failing to update records of the patients who do not visit the health system frequently, data sets still have potential for inaccuracy when compared to a bulk automatic update process independent of encounters within the health system. In addition to the automatic query, the COVID-19 immunization data is now automatically reconciled within the patient’s chart if the demographic information is coherent, and the immunization information is properly formatted. This ensures the vaccine information is easily visible within the patient dashboard.

The COVID-19 pandemic revealed a number of weaknesses in public health infrastructure, including shortcomings at the intersection of public health and primary care. This study highlights both the importance of and difficulties in data sharing between primary care providers and public health entities such as IIS. Instead of viewing public health in a hierarchical structure, the importance of a web of communication is apparent, where primary care practices access population health data to provide proactive care to a panel of patients.¹³ Reimagining immunization informatics, such that all vaccine providers are connected to a centralized vaccine registry and information flows freely within the system while maintaining patient privacy is critical to the strengthening of this important intersection.

CONCLUSION

When the interface between the patient chart and the local IIS depends on a manual query for the transfer of data, the COVID-19 vaccination status for a panel of patients is incomplete.

Significant numbers of vaccinated patient are not captured due to the cumbersome process for accessing vaccine data. Chart review and patient outreach to verify the accuracy of the data in the IIS was limited by poor response to outreach efforts but suggested the data coming from the IIS is reasonably comprehensive.

FUNDING

This research received no specific grant from any funding agency in the public, commercial or non-for-profit sectors.

AUTHOR CONTRIBUTIONS

SM: conceptualization, data gathering, writing original draft, editing. MM: conceptualization, data gathering, writing original draft, editing. LM: data gathering and interpretation, writing original draft, editing. CG: methodology, data interpretation, writing original draft, editing. BC: conceptualization, writing original draft, editing. MB: data gathering, writing original draft, editing. KK-V: data gathering, editing. AC: data gathering. TZ: data gathering. KHR: data gathering. JW: conceptualization, methodology, data gathering and interpretation, writing original draft. All authors reviewed the final draft and approved submission.

ETHICS APPROVAL

This study was reviewed by the University of Kansas Medical Center Institutional Review Board and was designated as a quality improvement project.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The data underlying this article are available in Dryad Digital Repository at https://doi.org/doi:10.5061/dryad.z612jm6gr.