Abstract
Background
Accurate determination of COVID-19 vaccination status is necessary to produce reliable COVID-19 vaccine effectiveness (VE) estimates. Data comparing differences in COVID-19 VE by vaccination sources (i.e., immunization information systems [IIS], electronic medical records [EMR], and self-report) are limited. We compared the number of mRNA COVID-19 vaccine doses identified by each of these sources to assess agreement as well as differences in VE estimates using vaccination data from each individual source and vaccination data adjudicated from all sources combined.
Methods
Adults aged ≥18 years who were hospitalized with COVID-like illness at 21 hospitals in 18 U.S. states participating in the IVY Network during February 1–August 31, 2022, were enrolled. Numbers of COVID-19 vaccine doses identified by IIS, EMR, and self-report were compared in kappa agreement analyses. Effectiveness of mRNA COVID-19 vaccines against COVID-19-associated hospitalization was estimated using multivariable logistic regression models to compare the odds of COVID-19 vaccination between SARS-CoV-2-positive case-patients and SARS-CoV-2-negative control-patients. VE was estimated using each source of vaccination data separately and all sources combined.
Results
A total of 4499 patients were included. Patients with ≥1 mRNA COVID-19 vaccine dose were identified most frequently by self-report (n = 3570, 79 %), followed by IIS (n = 3272, 73 %) and EMR (n = 3057, 68 %). Agreement was highest between IIS and self-report for 4 doses with a kappa of 0.77 (95 % CI = 0.73–0.81). VE point estimates of 3 doses against COVID-19 hospitalization were substantially lower when using vaccination data from EMR only (VE = 31 %, 95 % CI = 16 %–43 %) than when using all sources combined (VE = 53 %, 95 % CI = 41 %–62%).
Conclusion
Vaccination data from EMR only may substantially underestimate COVID-19 VE.
Keywords: COVID-19, SARS-CoV-2, Vaccine effectiveness, mRNA vaccines, Test-negative study design
1. Introduction
Post-licensure COVID-19 vaccine effectiveness (VE) studies have been essential for guiding vaccine policy decisions. These studies have commonly been conducted using the test-negative study design, in which patients with medically attended acute respiratory illness are enrolled and tested for SARS-CoV-2 [1], [2]. COVID-19 VE is estimated using multivariable logistic regression, comparing the odds of COVID-19 vaccination among patients testing positive versus negative for SARS-CoV-2 [2]. Bias can be introduced from several sources, including misclassification of COVID-19 vaccination status, which may result in substantial bias of VE estimates because it is the exposure in these analyses [2], [3], [4].
In the United States, there are four main sources of COVID-19 vaccination data: (1) immunization information systems (IIS), also known as state or jurisdictional vaccine registries, which are “confidential, population-based, computerized databases that record all immunization doses administered by participating providers to persons residing within a geopolitical area” [5]; (2) electronic medical records (EMR), which typically contain vaccine administration information from a single health system or multiple health systems with linked records; (3) self-report, which is information provided directly from the patient or patient surrogate; and (4) follow-up with vaccine providers, although this has been done less frequently during the COVID-19 pandemic given multiple and diverse vaccine providers (e.g., pharmacies, mass vaccination sites, etc.) and because it is labor intensive and inefficient. Each of these data sources has limitations. The United States does not have a national IIS in which all doses of vaccines administered to anyone residing in the United States are recorded. Instead, there are 64 unique jurisdictional (states, cities, or territories) IISs that submit vaccination data to CDC for surveillance of vaccination coverage and to inform program operations [5], [6]. However, documentation of vaccinations varies considerably in each IIS as they are independently governed and have historically been less complete for adult than for childhood vaccinations [7], [8]. By 2020, before COVID-19 vaccine rollout, only 68% of adults aged ≥19 years had ≥1 vaccination documented in an IIS compared with 94% of children aged < 6 years who had ≥ 2 vaccinations documented [7]. Similarly, the level of completeness in an EMR varies as patients might receive vaccine doses outside the health care system (e.g., at retail pharmacies) or across different health care systems that do not have interoperable EMR systems. Self-reported vaccination status can also vary, as shown by previous evaluations of influenza and pneumococcal VE [9], [10]. Limitations of self-reported vaccination history include the potential for social desirability bias and recall bias.
Many COVID-19 VE assessments in the United States use only one or two of these three vaccination data sources. The Investigating Respiratory Viruses in the AcutelY Ill (IVY) Network, a multicenter, prospective surveillance network that enrolls hospitalized adults with acute respiratory illness, uses all three sources—IIS, EMR, and self-report—to estimate COVID-19 VE [11]. Understanding differences in COVID-19 vaccination information available in each source and the effect of these differences on COVID-19 VE estimates is needed to interpret VE estimates that rely on limited sources of vaccination data. We assessed agreement of COVID-19 vaccine doses among IIS, EMR, and self-report, and compared VE estimates from each source with VE estimates generated by integration of all three sources of vaccination data.
2. Methods
2.1. Study design and population
During February 1–August 31, 2022, adults aged ≥ 18 years admitted for COVID-like illness (CLI) to 21 hospitals in 18 states within the IVY Network were eligible for inclusion in this test-negative, case-control analysis. CLI was defined as presence of fever, cough, shortness of breath, loss of taste, loss of smell, use of respiratory support (high flow nasal cannula, non-invasive ventilation, or invasive mechanical ventilation) or new pulmonary findings on chest imaging consistent with pneumonia. Case-patients with CLI were included if they tested positive for SARS-CoV-2 by reverse transcription polymerase chain reaction (RT-PCR) or antigen test within 14 days of illness onset. Control-patients with CLI who tested negative for SARS-CoV-2 by RT-PCR during the same period were included.
2.2. Data sources and definitions
Demographic and clinical data were obtained from EMR review and patient (or proxy) interview. COVID-19 mRNA vaccination status was ascertained by trained surveillance personnel upon enrollment from IIS, EMR, patient (or proxy) interview and, rarely, from CDC COVID-19 vaccination card. Given the small number of patients with available COVID-19 vaccination card data (132/7322, 1.8%), this source of vaccination data was omitted from source-based analyses. For each source of vaccination data, the type of vaccine (i.e., product), number of vaccine doses received, and known (or, in the case of self-report, estimated) date of vaccine receipt was recorded.
Table 1S summarizes the definitions used to interpret and classify available vaccine information by source. There are three categories of responses for self-reported vaccination status, but only two categories for IIS and EMR. Patients who self-report receipt of COVID-19 vaccination are classified as having, “evidence of COVID-19 vaccination.” Of these, only patients with known or estimated vaccination dates were included in VE analyses. Patients who report no receipt of COVID-19 vaccination are classified as unvaccinated. Additionally, patients missing self-reported COVID-19 vaccination history, either because they were too ill to participate in an interview (and no proxy was available) or because they refused to answer vaccination questions, are classified separately as, “missing.” For IIS and EMR sources, distinguishing between unvaccinated and missing COVID-19 vaccination is challenging because the absence of documented information can have multiple interpretations. Thus, patients who are identified in IIS (indicating prior receipt of adult vaccinations) but are missing documentation of COVID-19 vaccination and patients who are not identified in IIS are classified as having, “no evidence of COVID-19 vaccination.” For classification of vaccination status using EMR data, patients without documentation of any COVID-19 vaccine doses were classified as “no evidence of COVID-19 vaccination.”
2.3. Inclusion and exclusion criteria
Patients from sites that did not report COVID-19 vaccination by source, with receipt of non-mRNA doses, with missing self-reported COVID-19 vaccination, or who withdrew participation from this activity were excluded from descriptions of patient characteristics and COVID-19 vaccination source agreement analyses. Additional exclusions were applied for COVID-19 VE analyses that were stratified by number of mRNA COVID-19 vaccine doses received. Patients were excluded from VE analyses if they had received a non-mRNA vaccine, had only received one mRNA dose, had an immunocompromising condition2 (because VE analyses were stratified by number of doses, which were different for the immunocompromised population), had received any mRNA vaccine prior to CDC recommendations [12], had illness onset > 10 days before test date or > 14 days before hospitalization, or had missing data on variables included in the multivariable models.
For COVID-19 VE estimates by source, four vaccination groups were defined: 1) patients without evidence of COVID-19 vaccination or unvaccinated by self-report before illness onset; 2) patients who received 2 doses of COVID-19 mRNA vaccine ≥ 14 days before illness onset; and 3) patients who received 3 doses of COVID-19 mRNA vaccine ≥ 7 days before illness onset; 4) patients who received 4 doses of COVID-19 mRNA vaccine ≥ 7 days before illness onset.
2.4. Statistical analyses
Characteristics of hospitalized adults by IIS, EMR, and self-report and by vaccination status were summarized with frequencies, proportions, median and interquartile ranges. In the absence of a single criterion (gold) standard source of vaccination data, we used each vaccination source as a reference group in three independent agreement analyses as follows: 1) IIS vs. EMR, 2) EMR vs. self-report, and 3) IIS vs. self-report. Proportion of COVID-19 vaccine doses detected by each source was calculated as the number of doses detected by one source divided by the total number of doses detected by both sources. Unweighted Cohen’s kappa with bootstrapped confidence intervals were used to assess agreement between vaccine data sources, where higher values indicate better agreement between data sources. Kappa values are often interpreted as 0–0.20 as no agreement, 0.21–0.39 as minimal agreement, 0.40–0.59 as weak agreement, as 0.60–0.79 moderate agreement, as 0.80–0.90 strong agreement, and > 0.90 almost perfect agreement [13]. Patients who were missing self-reported vaccination history due to illness, absent proxy, or refusal to answer questions about vaccination were excluded from source agreement analyses. Their demographic and clinical characteristics were described using frequencies and proportions.
Several IVY Network sites include bi-directional IIS and EMR systems, which we defined as EMR systems in which data from IIS can be simultaneously viewed and/or edited during a patient encounter. Because such systems may improve the quality of data in each repository, we stratified agreement analyses by sites with and without bi-directional IIS systems to assess whether interoperability affected proportion of doses detected in either source. We also used unadjusted logistic regression to assess associations with discordance between IIS and EMR, defined as any difference between COVID-19 vaccine doses recorded in IIS and EMR.
To illustrate the effect of using different sources of vaccination status data on VE estimates, we calculated VE against COVID-19-associated hospitalization using vaccination status ascertained by self-report alone, IIS alone, EMR alone, and all sources combined (Fig. 1S). We used established statistical methods for calculating VE [14]. In VE analyses disseminated from the IVY Network, we use a combination of self-report, IIS and EMR for vaccination status classification [15], [16]. The goal of the current analysis is to quantify the bias that may result from using only one of these vaccination sources. Vaccine effectiveness of 2, 3, or 4 doses (among immunocompetent adults aged ≥ 50 years) against COVID-19-associated hospitalization was assessed by comparing the odds of COVID-19 mRNA vaccination by dose with no evidence of COVID-19 vaccination (or unvaccinated, for VE estimates based on self-reported vaccination status only) between case-patients and control-patients. Using multivariable logistic regression models, VE was calculated as (1 – adjusted odds ratio [aOR]) × 100. Models were adjusted for U.S. Department of Health and Human Services region, admission date in biweekly intervals, age group (18–49, 50–64, and ≥ 65 years), sex, and self-reported race and ethnicity. Source specific VE estimates were generated, and differences between estimates with nonoverlapping 95% CIs were considered to be statistically significant. Analyses were conducted using Stata 17.1 (StataCorp LP, College Station, TX). All tests were two-tailed with the threshold for statistical significance set at α = 0.05.
This activity was reviewed by CDC and was conducted consistent with applicable federal law and CDC policy (45C.F.R. part 46.102(l)(2), 21C.F.R. part 56; 42 U.S.C. §241(d); 5 U.S.C. §552a; 44 U.S.C. §3501 et seq.).
3. Results
During February 1–August 31, 2022, a total of 7322 patients enrolled in the IVY Network. After excluding patients from two sites without COVID-19 vaccination recorded by source (n = 469), recipients of non-mRNA COVID-19 vaccines (n = 360), patients missing self-reported COVID-19 vaccine doses (n = 1950), and patients who withdrew (n = 44), a total of 4499 patients were included in vaccination source agreement analyses (Fig. 2S). Patients missing self-reported COVID-19 vaccination history, compared with those who self-reported COVID-19 vaccination history, had a higher proportion of dementia (44 % vs. 17 %), ICU admission (39 % vs. 20 %) or invasive mechanical ventilation (17 % vs. 6 %) (Table 2S ).
Across all sources of vaccination data, the median age of participants who had received ≥ 1 COVID-19 vaccine dose was 66 years and 58–61 years for those without evidence of COVID-19 vaccination (Table 1 ). Of 4499 patients included, ≥1 COVID-19 vaccine dose was observed most frequently by self-report (n = 3570, 79 %), then IIS (n = 3272, 73 %), and least frequently by EMR (n = 3057, 68 %). The proportion of COVID-19 vaccinated patients was similar across sources by demographics, employment status, medical insurance status, healthcare utilization in the previous year, and clinical characteristics (Table 1).
Table 1.
Characteristics of hospitalized adults by source of mRNA COVID-19 vaccination and vaccination status — IVY network, February 1–August 31, 2022, N = 4499.
| Characteristic |
Source of mRNA COVID-19 vaccination |
|||||
|---|---|---|---|---|---|---|
|
EMR n (%) or Median (IQR) |
IIS n (%) or Median (IQR) |
Self-report n (%) or Median (IQR) |
||||
| Evidence of mRNA COVID-19 vaccination with ≥ 1 dose | No evidence of COVID-19 vaccination* | Evidence of mRNA COVID-19 vaccination with ≥ 1 dose | No evidence of COVID-19 vaccination† | Evidence of mRNA COVID-19 vaccination with ≥ 1 dose | Unvaccinated | |
| Total | n = 3057 | n = 1442 | n = 3272 | n = 1227 | n = 3570 | n = 929 |
| Number of mRNA COVID-19 vaccine doses | ||||||
| Unvaccinated/no evidence of vaccination | 0 (0) | 1442 (1 0 0) | 0 (0) | 1227 (1 0 0) | 0 (0) | 929 (1 0 0) |
| 1 dose | 1080 (35) | 0 (0) | 989 (30) | 0 (0) | 1120 (31) | 0 (0) |
| 2 doses | 1395 (46) | 0 (0) | 1601 (49) | 0 (0) | 1793 (50) | 0 (0) |
| 3 doses | 372 (12) | 0 (0) | 461 (14) | 0 (0) | 501 (14) | 0 (0) |
| ≥4 doses | 210 (7) | 0 (0) | 221 (7) | 0 (0) | 156 (5) | 0 (0) |
| Age, years | 66 (55–76) | 61 (47–72) | 66 (55–76) | 60 (45–70) | 66 (55–76) | 58 (43–69) |
| Age group, years | ||||||
| 18–49 | 538 (18) | 391 (27) | 564 (17) | 365 (30) | 638 (18) | 291 (31) |
| 50–64 | 862 (28) | 480 (33) | 941 (29) | 401 (33) | 1016 (28) | 326 (35) |
| ≥65 | 1655 (54) | 571 (40) | 1765 (54) | 461 (38) | 1914 (54) | 312 (34) |
| Sex, Female | 1506 (49) | 697 (48) | 1641 (50) | 562 (46) | 1755 (49) | 448 (48) |
| Race/Ethnicity | ||||||
| White, non-Hispanic | 1855 (61) | 873 (61) | 1980 (61) | 748 (61) | 2175 (61) | 553 (60) |
| Black, non-Hispanic | 531 (17) | 267 (19) | 572 (17) | 226 (18) | 608 (17) | 190 (20) |
| Hispanic | 454 (15) | 214 (15) | 489 (15) | 179 (15) | 530 (15) | 138 (15) |
| Other, non-Hispanic | 177 (6) | 72 (5) | 194 (6) | 55 (4) | 212 (6) | 37 (4) |
| Unknown | 40 (1) | 16 (1) | 37 (1) | 19 (2) | 45 (1) | 11 (1) |
| US Census Region§ | ||||||
| Northeast | 440 (14) | 220 (15) | 517 (16) | 143 (12) | 545 (15) | 115 (12) |
| South | 1300 (43) | 710 (49) | 1369 (42) | 641 (52) | 1520 (43) | 490 (53) |
| Midwest | 523 (17) | 149 (10) | 540 (17) | 132 (11) | 567 (16) | 105 (11) |
| West | 794 (26) | 363 (25) | 846 (26) | 311 (25) | 938 (26) | 219 (24) |
| Bi-directional EMR/IIS¶ | 2090 (68) | 996 (69) | 2185 (67) | 901 (73) | 2410 (68) | 676 (73) |
| COVID-19 RT-PCR Test Results | ||||||
| Negative | 1619 (53) | 683 (47) | 1759 (54) | 543 (44) | 1910 (54) | 392 (42) |
| Positive | 1437 (47) | 759 (53) | 1512 (46) | 684 (56) | 1659 (46) | 537 (58) |
| ≥3 chronic medical condition categories** | 1504 (49) | 485 (34) | 1563 (48) | 426 (35) | 1667 (47) | 322 (35) |
| Immunocompromised†† | 979 (32) | 271 (19) | 994 (30) | 256 (21) | 1073 (30) | 177 (19) |
| Cardiovascular disease | 2242 (73) | 903 (63) | 2384 (73) | 761 (62) | 2571 (72) | 574 (62) |
| Pulmonary disease | 1100 (36) | 451 (31) | 1159 (35) | 392 (32) | 1251 (35) | 300 (32) |
| Obesity (BMI ≥30 kg/m2) | 1174 (39) | 549 (40) | 1272 (40) | 451 (38) | 1362 (39) | 361 (40) |
| Diabetes mellitus | 1103 (36) | 418 (29) | 1164 (36) | 357 (29) | 1260 (35) | 261 (28) |
| Long-term care facility resident | 127 (4) | 77 (5) | 159 (5) | 45 (4) | 177 (5) | 27 (3) |
| Healthcare visits in the last year | 3 (2–4) | 2 (1–3) | 3 (2–4) | 2 (1–4) | 3 (2–4) | 2 (1–4) |
| Currently employed | 600 (20) | 308 (23) | 640 (20) | 268 (23) | 705 (21) | 203 (23) |
| Any medical insurance§§ | 3004 (98) | 1319 (91) | 3208 (98) | 1115 (91) | 3484 (98) | 839 (90) |
Abbreviations: EMR = electronic medical record; IIS = immunization information system; IQR = interquartile range; BMI = body mass index.
For EMR, no evidence of COVID-19 vaccination status refers to patients without documented COVID-19 vaccination receipt in EMR.
For IIS, no evidence of COVID-19 vaccination includes scenarios in which the patient’s record is available in the IIS and there is no documentation of COVID-19 vaccine receipt as well as scenarios in which there is no record of patient in the IIS.
Hospitals by U.S. Census region included Northeast: Baystate Medical Center (Springfield, Massachusetts), Beth Israel Deaconess Medical Center (Boston, Massachusetts), Montefiore Medical Center (Bronx, New York); South: Vanderbilt University Medical Center (Nashville, Tennessee), University of Miami Medical Center (Miami, Florida), Emory University Medical Center (Atlanta, Georgia), Johns Hopkins Hospital (Baltimore, Maryland), Wake Forest University Baptist Medical Center (Winston-Salem, North Carolina), Baylor Scott & White Health (Temple, Texas); Midwest: University of Iowa Hospitals and Clinics (Iowa City, Iowa), University of Michigan Hospital (Ann Arbor, Michigan), Hennepin County Medical Center (Minneapolis, Minnesota), Barnes-Jewish Hospital (St. Louis, Missouri), Cleveland Clinic (Cleveland, Ohio), Ohio State University Wexner Medical Center (Columbus, Ohio); West: Stanford University Medical Center (Stanford, California), UCLA Medical Center (Los Angeles, California), UCHealth University of Colorado Hospital (Aurora, Colorado), Oregon Health & Science University Hospital (Portland, Oregon), Intermountain Medical Center (Murray, Utah), University of Washington (Seattle, Washington).
Hospitals without bi-directional EMR/IIS include: Baystate Medical Center (Springfield, Massachusetts), Beth Israel Deaconess Medical Center (Boston, Massachusetts), Wake Forest University Baptist Medical Center (Winston-Salem, North Carolina), University of Michigan Hospital (Ann Arbor, Michigan), UCLA Medical Center (Los Angeles, California).
Chronic medical condition categories included at least one condition in the following categories: cardiovascular, neurologic, pulmonary, gastrointestinal, endocrine, kidney, hematologic, or autoimmune.
Immunocompromising conditions included: active solid tumor malignancy (defined as treatment for the malignancy or newly diagnosed malignancy in the past 6 months), active hematologic malignancy (such as leukemia, lymphoma, or myeloma), HIV infection with or without AIDS, congenital immunodeficiency syndrome, previous splenectomy, previous solid organ transplant, immunosuppressive medication, systemic lupus erythematosus, rheumatoid arthritis, psoriasis, scleroderma, or inflammatory bowel disease, including Crohn’s disease or ulcerative colitis.
Medical insurance categories include Medicare, Medicaid, Active military (Tricare), Veterans Affairs, Other government insurance, Private insurance, Other.
Agreement analyses were conducted between IIS vs. EMR (Table 2a ), EMR vs. self-report (Table 2b ), and IIS vs. self-report (Table 2c ). In each of these analyses, the proportion of patients with a specific number of COVID-19 vaccine doses detected by one source (e.g., IIS) among doses identified by two sources (e.g., IIS and EMR) is shown along with the Cohen’s kappa agreement for each dose. For patients who had received ≥ 2 COVID-19 vaccine doses, a higher proportion of patients self-reported extra doses than were documented in either IIS or EMR (Table 2b, Table 2c, Table 3, Table 4 ). Comparing IIS and EMR, a higher proportion of patients had 3 COVID-19 doses documented in IIS (0.90; 95 % CI = 0.88–0.91) compared with EMR (0.78; 95 % CI = 0.76–0.80) as well as ≥ 4 doses documented in IIS (0.91; 95 % CI = 0.88–0.93) vs. EMR (0.73; 95 % CI = 0.70–0.77). Additionally, although kappa agreement was moderate overall for each of the three source comparisons, agreement was highest between IIS and self-report for 4 doses with a kappa of 0.77 (95 % CI = 0.73–0.81).
Table 2a.
Number of mRNA COVID-19 vaccine doses among hospitalized adults recorded in immunization information systems (IIS) and electronic medical records (EMR) — IVY Network, February 1–August 31, 2022, N = 4499.
| Number of mRNA COVID-19 vaccine doses | Frequency |
Proportion by IIS* (95 % CI) |
Proportion by EMR† (95 % CI) |
Kappa (95 % CI) |
||
|---|---|---|---|---|---|---|
| Both (a) |
IIS only (b) |
EMR only (c) |
||||
| No evidence of vaccination§ | 1080 | 147 | 362 | 0.77 (0.75–0.79) | 0.91 (0.89–0.92) | 0.73 (0.72–0.74) |
| 1 dose | 129 | 92 | 81 | 0.73 (0.68–0.78) | 0.70 (0.64–0.75) | 0.58 (0.52–0.63) |
| 2 doses | 775 | 214 | 305 | 0.76 (0.74–0.79) | 0.83 (0.81–0.85) | 0.67 (0.66–0.69) |
| 3 doses | 1215 | 386 | 180 | 0.90 (0.88–0.91) | 0.78 (0.76–0.80) | 0.72 (0.70–0.73) |
| ≥4 doses | 326 | 135 | 46 | 0.91 (0.88–0.93) | 0.73 (0.70–0.77) | 0.76 (0.69–0.83) |
Abbreviations: IIS = immunization information system; EMR = electronic medical record; CI = confidence interval.
Proportion detected IIS = a + b/a + b + c.
Proportion detected EMR = a + c/a + b + c.
For IIS, no evidence of vaccination includes scenarios in which the patient’s record is available in the IIS and there is no documentation of COVID-19 vaccine receipt as well as scenarios in which there is no record of patient in the IIS. For EMR, no evidence of vaccination refers only to patients without documented COVID-19 vaccine doses since all hospitalized patients have a medical record.
Table 2b.
Number of mRNA COVID-19 vaccine doses among hospitalized adults by electronic medical records (EMR) and self-report — IVY Network, February 1–August 31, 2022, N = 4499.
| Number of mRNA COVID-19 vaccine doses | Frequency |
Proportion by EMR* (95 % CI) |
Proportion by self-report† (95 % CI) |
Kappa (95 % CI) |
||
|---|---|---|---|---|---|---|
| Both (a) |
EMR only (b) |
Self-report only (c) |
||||
| Unvaccinated/no evidence of mRNA COVID-19 vaccination§ | 890 | 552 | 39 | 0.97 (0.97–0.98) | 0.63 (0.60–0.65) | 0.66 (0.65–0.69) |
| 1 dose | 87 | 123 | 69 | 0.75 (0.70–0.80) | 0.56 (0.50–0.62) | 0.45 (0.36–0.55) |
| 2 doses | 801 | 279 | 319 | 0.77 (0.75–0.79) | 0.80 (0.78–0.82) | 0.64 (0.61–0.67) |
| 3 doses | 1222 | 173 | 571 | 0.71 (0.69–0.73) | 0.91 (0.90–0.92) | 0.64 (0.62–0.67) |
| ≥4 doses | 315 | 57 | 186 | 0.67 (0.63–0.71) | 0.90 (0.87–0.92) | 0.69 (0.67–0.71) |
Abbreviations: EMR = electronic medical record; CI = confidence interval.
Proportion of doses detected by EMR = a + b/a + b + c.
Proportion of doses detected by self-report = a + c/a + b + c.
For EMR, no evidence of COVID-19 vaccination indicates patients without documented COVID-19 vaccination receipt in EMR. For self-report, unvaccinated indicates persons who self-reported no prior receipt of COVID-19 vaccination (n = 929).
Table 2c.
Number of mRNA COVID-19 vaccine doses among hospitalized adults by immunization information systems (IIS) and self-report — IVY Network, February 1–August 31, 2022, N = 4499.
| Number of mRNA COVID-19 vaccine doses | Frequency |
Proportion by IIS* (95 % CI) |
Proportion by self-report† (95 % CI) |
Kappa (95 % CI) |
||
|---|---|---|---|---|---|---|
| Both (a) |
IIS only (b) |
Self-report only (c) |
||||
| Unvaccinated/no evidence of mRNA COVID-19 vaccination§ | 873 | 354 | 56 | 0.96 (0.95–0.97) | 0.72 (0.70–0.75) | 0.78 (0.72–0.78) |
| 1 dose | 94 | 127 | 62 | 0.78 (0.93–0.83) | 0.55 (0.49–0.61) | 0.48 (0.41–0.55) |
| 2 doses | 822 | 167 | 298 | 0.77 (0.75–0.79) | 0.87 (0.85–0.89) | 0.71 (0.69–0.74) |
| 3 doses | 1399 | 202 | 394 | 0.80 (0.79–0.82) | 0.90 (0.89–0.91) | 0.72 (0.70–0.74) |
| ≥4 doses | 384 | 77 | 117 | 0.80 (0.76–0.83) | 0.87 (0.84–0.89) | 0.77 (0.73–0.81) |
Abbreviations: IIS = immunization information system; CI = confidence interval.
Proportion of doses detected by IIS = a + b/a + b + c.
Proportion of doses detected by self-report = a + c/a + b + c.
For IIS, no evidence of COVID-19 vaccination status includes patients identified in IIS who lack documentation of COVID-19 vaccine receipt as well as patients who are not identified in IIS. For self-report, unvaccinated includes only persons who self-reported no prior receipt of COVID-19 vaccination (n = 929).
Table 3.
Bivariate associations* of discordance in number of mRNA COVID-19 vaccine doses between immunization information systems (IIS) and electronic medical records (EMR) — IVY Network, February 1–August 31, 2022, N = 4499.
| Characteristic |
Unadjusted OR (95 % CI) |
P- value |
|---|---|---|
| Age group, years | ||
| 18–49 | Reference | |
| 50–64 | 1.32 (1.07, 1.64) | 0.009 |
| ≥65 | 1.43 (1.18, 1.75) | <0.001 |
| Race and Ethnicity | ||
| White, non-Hispanic | Reference | |
| Black, non-Hispanic | 1.00 (0.82, 1.21) | 0.98 |
| Hispanic | 0.71 (0.57, 0.89) | 0.002 |
| Other, non-Hispanic | 1.07 (0.79, 1.45) | 0.66 |
| Unknown | 0.66 (0.32, 1.35) | 0.26 |
| SARS-CoV-2 RT-PCR-positive test results | 0.68 (0.59, 0.78) | <0.001 |
| Bi-directional EMR/IIS§ | 0.55 (0.48, 0.64) | <0.001 |
| Currently employed | 0.80 (0.67, 0.97) | 0.021 |
| Long-term care facility resident | 2.34 (1.75, 3.14) | <0.001 |
Abbreviations: OR = odds ratio; CI = confidence interval.
Additional factors that were assessed and determined not to be significantly associated with discordant data between IIS and EMR include the following: sex, college educations, healthcare worker, hospital admissions in the previous year, and healthcare visits in the previous year.
Hospitals with bi-directional IIS and EMR systems include: Montefiore Medical Center (Bronx, New York); Vanderbilt University Medical Center (Nashville, Tennessee); University of Miami Medical Center (Miami, Florida); Emory University Medical Center (Atlanta, Georgia); Johns Hopkins Hospital (Baltimore, Maryland); Baylor Scott & White Health (Temple, Texas); University of Iowa Hospitals and Clinics (Iowa City, Iowa); Hennepin County Medical Center (Minneapolis, Minnesota); Barnes-Jewish Hospital (St. Louis, Missouri); Cleveland Clinic (Cleveland, Ohio); Ohio State University Wexner Medical Center (Columbus, Ohio); Stanford University Medical Center (Stanford, California); UCHealth University of Colorado Hospital (Aurora, Colorado); Oregon Health & Science University Hospital (Portland, Oregon); Intermountain Medical Center (Murray, Utah); University of Washington (Seattle, Washington). Hospitals without bi-directional IIS and EMR include: Baystate Medical Center (Springfield, Massachusetts); Beth Israel Deaconess Medical Center (Boston, Massachusetts); Wake Forest University Baptist Medical Center (Winston-Salem, North Carolina); University of Michigan Hospital (Ann Arbor, Michigan); UCLA Medical Center (Los Angeles, California).
Table 4.
Vaccine effectiveness (VE) for prevention of COVID-19-associated hospitalization among immunocompetent* adults by number of mRNA COVID-19 vaccine doses received and by source of vaccination — IVY Network, February 1–August 31, 2022, N = 2952.
| Number of mRNA COVID-19 vaccine doses by source of vaccination | No. of vaccinated COVID-19 case-patients/total no. of case-patients (%) | No. of vaccinated control-patients/total no. of control-patients (%) | Adjusted VE† % (95 % CI) |
|---|---|---|---|
| 2 doses | |||
| EMR | 359/922 (39) | 385/909 (42) | 20 (3–34) |
| IIS | 334/830 (40) | 372/780 (48) | 32 (16–45) |
| Self-report | 389/785 (50) | 407/716 (57) | 35 (19–47) |
| Combined EMR, IIS, self-report | 437/819 (53) | 547/845 (65) | 36 (19–49) |
| 3 doses | |||
| EMR | 381/944 (40) | 441/965 (46) | 31 (16–43) |
| IIS | 448/944 (47) | 535/943 (57) | 42 (29–52) |
| Self-report | 493/889 (55) | 599/908 (66) | 49 (37–59) |
| Combined EMR, IIS, self-report | 506/888 (57) | 655/953 (69) | 53 (41–62) |
| 4 doses§ | |||
| EMR | 58/621 (9) | 89/613 (15) | 56 (34–70) |
| IIS | 72/568 (13) | 121/529 (23) | 65 (48–76) |
| Self-report | 90/479 (19) | 139/546 (25) | 46 (24–62) |
| Combined EMR, IIS, self-report | 69/451 (15) | 108/406 (27) | 70 (54–81) |
Abbreviations: VE = vaccine effectiveness; CI = confidence interval; EMR = electronic medical record; IIS = immunization information system.
Excludes patients with immunocompromising conditions defined as: active solid tumor malignancy (defined as treatment for the malignancy or newly diagnosed malignancy in the past 6 months), active hematologic malignancy (such as leukemia, lymphoma, or myeloma), HIV infection with or without AIDS, congenital immunodeficiency syndrome, previous splenectomy, previous solid organ transplant, immunosuppressive medication, systemic lupus erythematosus, rheumatoid arthritis, psoriasis, scleroderma, or inflammatory bowel disease, including Crohn’s disease or ulcerative colitis.
Multivariable logistic regression models were used to determine vaccine effectiveness, with vaccine status as the primary independent variable, case status as the dependent variable, and the following covariates: admission date (biweekly intervals), age (18–49, 50–64, and ≥ 65 years), sex, self-reported race and ethnicity, and U.S. Health and Human Services region of the admitting hospital.
Among immunocompetent adults aged ≥ 50 years.
Older age and residence in a long-term care facility were associated with higher odds of discordance in vaccination status by IIS and EMR, while Hispanic ethnicity, SARS-CoV-2 RT-PCR-positive test results, current employment, and bi-directional IIS/EMR systems at the enrollment site were associated with lower discordance in vaccination status by IIS and EMR (Table 3). In agreement analyses that were stratified by sites with and without bi-directional IIS/EMR systems, kappa coefficient was significantly higher for all doses, except 4 doses, among sites with bi-directional IIS/EMR systems than sites in which IIS was independent of EMR (Table 3S).
For estimation of mRNA VE against COVID-19-associated hospitalization by source of vaccination data, additional exclusions were applied and a total of 2952 participants were included in these analyses (Fig. 2S). VE point estimates were overall lower for VE analyses restricted to a single source of vaccination data compared with VE obtained from all three sources of vaccination data (Table 4). Specifically, VE point estimates by COVID-19 vaccine dose were consistently, but not statistically, lower across all analyses using EMR as the only source of vaccination data compared with VE estimated by other independent sources or by all sources combined. VE of 3 doses by EMR-only to prevent COVID-19-associated hospitalization was 31 % (95 % CI = 16 %–43 %), which was lower than VE of 3 doses estimated by all sources combined at 53 % (95 % CI = 41 %–62 %), although confidence intervals overlapped. There was no difference in COVID-19 VE estimates that were stratified by sites with and without bi-directional IIS/EMR systems (Table 4S).
4. Discussion
After more than a year of COVID-19 vaccination in the United States, findings from this multistate network show that reporting of COVID-19 vaccine doses was highest among patients with available self-reported COVID-19 vaccination history, followed by IIS, and then EMR sources. Furthermore, hospitals with bi-directional IIS and EMR systems had better agreement of documented COVID-19 doses than those with separate systems. Combined use of all three sources of vaccination data consistently resulted in higher COVID-19 VE point estimates against COVID-19 hospitalization compared with VE estimates based on independent sources only, especially EMR only, although these differences were not statistically significant. To minimize misclassification bias, integration of vaccination sources among IIS, EMR, and self-report remains the optimal approach for conducting VE analyses in the United States.
The findings from this analysis confirm and build on a previous analysis from this network, which was conducted early after the rollout of COVID-19 vaccines in 2021 [17]. In the previous analysis, self-reported COVID-19 vaccination status had high agreement with documented sources of vaccination data (IIS and EMR). Since then, despite increasing complexity of the COVID-19 vaccine series [12], the current analysis confirms previous findings that, when available, self-reported COVID-19 vaccination history remains an important adjunct source of vaccination data. However, self- (or proxy-) reported vaccination history in an acutely ill, hospitalized cohort was only available among 73% of enrolled participants in this analysis. Missing self-reported vaccination history among a substantial proportion of the hospitalized population, specifically among older, more severely ill patients in the ICU or those with dementia, limits its use as the only source of vaccination information in VE studies among a hospitalized cohort.
Separating documented sources of COVID-19 vaccination in this analysis into doses detected by IIS and EMR also yields important insights. A higher proportion of patients with receipt of ≥3 COVID-19 vaccine doses were identified in IIS compared with EMR, suggesting a possible shift of booster doses administered in higher proportions at retail pharmacies than in the site’s healthcare system. Additionally, COVID-19 VE point estimates against COVID-19 hospitalization based on IIS-only data were 10–12% points higher than COVID-19 VE point estimates based on EMR-only data. Strengthening IISs had been a core goal of COVID-19 vaccine implementation plans [18]. Limited initial COVID-19 vaccine supply combined with tiered rollout of vaccines to priority groups and the need for timely administration of a second dose led to substantial investments in U.S. immunization infrastructure to monitor COVID-19 vaccine distribution and coverage in near real-time [19], [20]. Because the federal government procured all COVID-19 vaccines used in the U.S., the federal government was also able to mandate that vaccination providers report administration of COVID-19 vaccine to IISs within 24–72 hours as part of the COVID-19 vaccine provider enrollment agreement, which led to substantial improvements in IIS data quality for COVID-19 vaccination [21]. The extent to which such reporting practices will continue once the COVID-19 public health emergency ends and the federal government is no longer the only payor for COVID-19 vaccines used in the U.S., or if reporting mandates are removed, remains unclear. If reporting practices return to pre-pandemic experience, evidence from influenza studies suggests that reporting of adult vaccinations to IISs was low and varied by jurisdiction, with some IISs demonstrating improved documentation over time (e.g., Michigan), but not others [22], [23].
An additional component of strengthening IISs through the COVID-19 pandemic had been to make them bi-directional with EMRs to improve documentation in both repositories and to allow for comprehensive COVID-19 vaccination history to be available at point-of-care for clinician decision-making and for COVID-19 VE assessments [5], [6]. Indeed, this analysis showed that having bi-directional IIS and EMR systems, in which data from both systems could be viewed simultaneously and/or edited, led to improved agreement between these sources among 14 hospitals with bi-directional IIS and EMR systems, but agreement was not perfect likely because of differences in editing practices by site for either source. However, the potential effect of this harmonization on COVID-19 VE estimates was not discernible, likely due to the small sample size available to stratify VE estimates by sites with and without bi-directional systems.
This analysis is subject to at least five limitations. First, because self-reported vaccination history is the only source in which the difference between vaccinated, unvaccinated, and missing information is clear, the importance of self-reported COVID-19 vaccination history might be overestimated by excluding patients unable to provide self-reported data. Similarly, a related limitation is that absence of COVID-19 vaccination data in either an IIS or EMR can only be interpreted as “no evidence of COVID-19 vaccination,” which does not distinguish between unvaccinated persons and those with missing data. Second, because patients missing self-reported vaccination data were more likely to be severely ill, VE estimates based on self-report only might be skewed towards hospitalized patients who are less sick and, therefore, more likely to be vaccinated. This could contribute towards the higher VE point estimates observed by self-reported vaccination status. Third, inaccurate self-reported vaccination status could have occurred by false overreporting of COVID-19 vaccination due to potential social desirability bias, recall bias, or inaccurate responses from proxies. While only plausible COVID-19 vaccine doses with known or estimated vaccination dates and product names were used in VE analyses to minimize this bias, we did not routinely contact participants who self-reported a higher number of COVID-19 vaccine doses for additional verification. Fourth, this analysis was limited to data collected after February 1, 2022, when the IVY Network began documenting vaccine doses identified by IIS and EMR separately. Fifth, in the absence of a gold standard repository of COVID-19 vaccination data, agreement analyses were performed using each source as a gold standard, which prevented calculation of sensitivity and specificity of dose detection by source.
Despite these limitations, this analysis demonstrates the emerging importance of IIS as a source of adult COVID-19 vaccination data relative to EMRs, and supports continued use of integrated sources of COVID-19 vaccination data to minimize misclassification bias in COVID-19 VE analyses conducted in the U.S. As the public health emergency for COVID-19 concludes in the U.S. on May 11, 2023, ongoing monitoring of the completeness of these sources is needed as payment and procurement of COVID-19 vaccines become de-centralized and reporting practices for adult vaccinations evolve.
5. Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention (CDC).
6. Data availability
No additional data are available.
7. Notes
A full list of investigators and collaborators in the Investigating Respiratory Viruses in the Acutely Ill (IVY) Network are listed in Appendix A.
Funding
This work was funded by the United States Centers for Disease Control and Prevention (contract 75D30122C12914 to Vanderbilt University Medical Center).
Declaration of Competing Interest
All authors have completed and submitted the International Committee of Medical Journal Editors (ICJME) disclosure form. Funding for this work was provided to all participating sites by the United States Centers for Disease Control and Prevention. The authors declare the following financial interests/personal relationships, which may be considered as potential competing interests: Samuel Brown reports institutional funds from Janssen for influenza research. Jonathan Casey reports a travel grant from Fischer and Paykel, outside the submitted work. Steven Chang reports consulting fees from PureTech Health and Kiniksa Pharmaceuticals and participating as a DSMB member for a study at UCLA, outside the submitted work. Manjusha Gaglani reports grants from CDC, CDC-Abt Associates, CDC-Westat, and Janssen, and served as co-chair of the Infectious Diseases and Immunization Committee for the Texas Pediatric Society, outside the submitted work. Kevin Gibbs reports grants from NIH and DoD, as well as support for MHSRS 2022 travel from the DoD, outside the submitted work. Adit Ginde reports receiving grants from NIH, DoD, AbbVie, and Faron Pharmaceuticals, outside the submitted work. Michelle N. Gong reports grants from NHLBI, CDC, AHRQ, speaking at medicine grand rounds at New York Medical College, travel support for the ATS executive meeting and serving as ATS Chair Critical Care Assembly, DSMB membership fees from Regeneron, and participating on the scientific advisory panel for Endpoint, outside the submitted work. Carlos Grijalva reports grants from NIH, CDC, AHRQ, FDA, Campbell Alliance/Syneos Health; receiving consulting fees from and participating on a DSMB for Merck, outside the submitted work. David Hager reports receiving grants from NIH, outside the submitted work. Akram Khan reports receiving grants from United Therapeutics, Johnson & Johnson, 4D Medical, Eli Lily, Dompe Pharmaceuticals, and GlaxoSmithKline; and serves on the guidelines committee for Chest, outside the submitted work. Adam Lauring reports receiving grants from CDC, FluLab, NIAID, and Burroughs Wellcome Fund, and consulting fees from Sanofi and Roche for consulting on oseltamivir and baloxavir respectively, outside the submitted work. Emily Martin reports grants from Merck and NIH, outside the submitted work. Tresa McNeal reports receiving a grant from CDC, receiving a one-time payment for participating as a virtual webinar panelist for Clinical Updates in Heart Failure, and being a Practice Management Committee member for Society of Hospital Medicine, outside the submitted work. Ithan D. Peltan reports grants from NIH, Janssen Pharmaceuticals and institutional support from Asahi Kasei Pharma and Regeneron, outside the submitted work. Jennifer Wilson reports grants from NHLBI, outside the submitted work. No other potential conflicts of interest were disclosed.
Footnotes
Immunocompromising conditions: active solid tumor malignancy (defined as treatment for the malignancy or newly diagnosed malignancy in the past 6 months), active hematologic malignancy (such as leukemia, lymphoma, or myeloma), HIV infection with or without AIDS, congenital immunodeficiency syndrome, previous splenectomy, previous solid organ transplant, immunosuppressive medication, systemic lupus erythematosus, rheumatoid arthritis, psoriasis, scleroderma, or inflammatory bowel disease, including Crohn’s disease or ulcerative colitis.
Supplementary data to this article can be found online at https://doi.org/10.1016/j.vaccine.2023.05.028.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
References
- 1.International Vaccine Access Center. Results of COVID-19 Vaccine Effectiveness Studies: An Ongoing Systematic Review. https://view-hub.org/covid-19/effectiveness-studies (accessed November 28, 2022).
- 2.World Health Organization . World Health Organization; Geneva: 2021. Evaluation of COVID-19 vaccine effectiveness: interim guidance. [Google Scholar]
- 3.Sullivan S.G., Tchetgen E.J., Cowling B.J. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. Am J Epidemiol. 2016;184(5):345–353. doi: 10.1093/aje/kww064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Foppa I.M., Ferdinands J.M., Chung J., et al. Vaccination history as a confounder of studies of influenza vaccine effectiveness. Vaccine. 2019 doi: 10.1016/j.jvacx.2019.100008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Centers for Disease Control and Prevention. Immunization Information Systems. Available at: https://www.cdc.gov/vaccines/programs/iis/about.html (accessed July 31, 2022).
- 6.Shen A.K., Sobczyk E.A., Coyle R., Tirmal A., Hannan C. How ready was the US vaccination infrastructure and network of immunization information systems for COVID-19 vaccination campaigns: Recommendations to strengthen the routine vaccination program and prepare for the next pandemic. Hum Vaccin Immunother. 2022;18(5) doi: 10.1080/21645515.2022.2088010. https://www.tandfonline.com/doi/full/10.1080/21645515.2022.2088010 2088010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Centers for Disease Control and Prevention. IISAR Data Participation Rates and Maps. Available at: https://www.cdc.gov/vaccines/programs/iis/annual-report-iisar/rates-maps-table.html (accessed July 31, 2022).
- 8.CDC Progress in immunization information systems — United States, 2012. MMWR. 2012;62(49):1005–1008. [PMC free article] [PubMed] [Google Scholar]
- 9.Irving S.A., Donahue J.G., Shay D.K., et al. Evaluation of self-reported and registry-based influenza vaccination status in a Wisconsin cohort. Vaccine. 2009;27(47):6546–6549. doi: 10.1016/j.vaccine.2009.08.050. [DOI] [PubMed] [Google Scholar]
- 10.Mangtani P., Shah A., Roberts J.A. Validation of influenza and pneumococcal vaccine status in adults based on self-report. Epidemiol Infect. 2007;135(1):139–143. doi: 10.1017/S0950268806006479. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Tenforde M.W., Self W.H., Zhu Y., Naioti E.A., et al. Influenza and Other Viruses in the Acutely Ill (IVY) Network. Protection of mRNA vaccines against hospitalized COVID-19 in adults over the first year following authorization in the United States. Clin Infect Dis. 2022 May;17:ciac381. doi: 10.1093/cid/ciac381. [DOI] [Google Scholar]
- 12.Centers for Disease Control and Prevention. Archived COVID-19 vaccination Schedules. https://www.cdc.gov/vaccines/covid-19/clinical-considerations/archived-covid-19-vacc-schedule.html (accessed November 25, 2022).
- 13.McHugh M.L. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22(3):276–282. [PMC free article] [PubMed] [Google Scholar]
- 14.Chua H., Feng S., Lewnard J., et al. The Use of Test-negative Controls to Monitor Vaccine Effectiveness: A Systematic Review of Methodology. Epidemiology. 2020;31(1):43–64. doi: 10.1097/EDE.0000000000001116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Adams K., Rhoads J.P., Surie D., Gaglani M., Ginde A.A., McNeal T., et al. Vaccine effectiveness of primary series and booster doses against covid-19-associated hospitalization admissions in the United States: living test negative design study. BMJ. 2022;379 doi: 10.1136/bmj-2022-072065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Surie D., Bonnell L., Adams K., Gaglani M., Ginde A.A., Douin D.J., et al. IVY Network. Effectiveness of monovalent mRNA vaccines against COVID-19-associated hospitalization among immunocompetent adults during BA.1/BA.2 and BA.4/BA.5 predominant periods of SARS-CoV-2 omicron variant in the United States - IVY Network, 18 States, December 26, 2021-August 31, 2022. MMWR Morb Mortal Wkly Rep. 2022;71(42):1327–1334. doi: 10.15585/mmwr.mm7142a3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Stephenson M., Olson S.M., Self W.H., et al. Ascertainment of vaccination status by self-report versus source documentation: Impact on measuring COVID-19 vaccine effectiveness. Influenza Other Respir Viruses. 2022 Nov;16(6):1101–1111. doi: 10.1111/irv.13023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Department of Health and Human Services and the Department of Defense. From the Factory to the Front Lines: The Operation Warp Speed Strategy for Distributing a COVID-19 Vaccine. Report to Congress. Available at: https://www.hhs.gov/sites/default/files/strategy-for-distributing-covid-19-vaccine.pdf (accessed July 13, 2022).
- 19.Centers for Disease Control and Prevention. “Administration announces $200 million from CDC to jurisdictions for COVID-19 vaccine preparedness.” Available at: https://www.cdc.gov/media/releases/2020/p0924-200-million-jurisdictions-covid-19-preparedness.html#::text=CDC%20is%20awarding%20%24200%20million,for%20the%20COVID%2D19%20vaccine (accessed July 31, 2022).
- 20.Trotter A.B., Abbott E.K., Coyle R., Shen A.K. Preparing for COVID-19 vaccination: A call to action for clinicians on immunization information systems. Ann Intern Med. 2021;174(5):695–697. doi: 10.7326/M20-7725. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Centers for Disease Control and Prevention. CDC COVID-19 Vaccination Program Provider Requirements and Support. Available at: https://www.cdc.gov/vaccines/covid-19/vaccination-provider-support.html (accessed on July 31, 2022).
- 22.Petrie J.G., Fligiel H., Lamerato L., Martin E.T., Monto A.S. Agreement between state registry, health record, and self-report of influenza vaccination. Vaccine. 2021;39:5341–5345. doi: 10.1016/j.vaccine.2021.07.090. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Nowalk M.P., D’Agostino H.E.A., Zimmerman R.K., et al. Agreement among sources of adult influenza vaccination in the age of immunization information systems. Vaccine. 2021;39:6829–6836. doi: 10.1016/j.vaccine.2021.10.041. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
No additional data are available.