Abstract
Introduction:
Many vaccination studies rely on self-reported vaccination status, with its inherent biases. Accuracy of influenza vaccination self-report has been evaluated periodically, typically using the medical record as the gold standard. The burgeoning of electronic medical records (EMRs) and immunization information systems (IISs) and the rise of adult vaccine administration in community pharmacies suggest the need for a reevaluation of self-reported vaccination status.
Methods:
Vaccination data from self-report, the state IIS, the health system EMR and other sources were compared for participants in outpatient and inpatient influenza vaccine effectiveness studies for four seasons (2016–2017 to 2019–2020). Agreement among the sources was calculated along with sensitivity and specificity. Tests for trend assessed changes in completeness of the Pennsylvania - Statewide IIS (PA-SIIS) data over time.
Results:
With self-report as the gold standard, agreement with the local EMR, PA-SIIS, and all sources was 62%, 77% and 85%, respectively. Sensitivity of the EMR was 42% (95% CI=41, 43) and specificity was 91% (90, 92). With PA-SIIS-as the gold standard, agreement with the local EMR and all sources was 77% and 78%, respectively. Sensitivity of all sources combined was 96% (95, 97) and specificity was (63% (62, 64). Capture of influenza vaccinations in the IIS has not consistently improved over time, with a significant increase among children (P=0.001), no change among working-age adults and a decrease among older adults (P=0.004). However, PA-SIIS provided the largest percentage of verified vaccines (69.3%) compared with EMR (43.3%) and other sources (12.4%).
Conclusion:
Both self-report and PA-SIIS are good estimates of actual vaccine uptake. When high accuracy data are required, such as for vaccine effectiveness studies, triangulation using multiple sources should be conducted.
Keywords: Adult influenza vaccination, Immunization Information Systems, vaccination status, vaccine registries, electronic medical records
Introduction
Self-report is frequently used to estimate vaccination coverage in national and statewide surveys such as the National Immunization Survey (NIS) [1], National Health Interview Survey (NHIS) [2], and the Behavioral Risk Factor Surveillance Survey (BRFSS) [3], as well as numerous intervention and quality improvement studies. In the past, the medical record has been used as the “gold standard” against which self-reported vaccination has been measured. Studies by MacDonald et al. and Zimmerman et al. reported sensitivity of 98%−100% and specificity of 38%−71% for self-report of adult influenza vaccines [4, 5]. Newer studies using 2007–2008 data, compared self-report and electronic medical records (EMRs), finding similar agreement between them [6, 7]. These studies were conducted before EMRs were mature and widely used for example, in 2008, use of EMRs by healthcare providers was estimated at 10% [8].
As part of the American Recovery and Reinvestment Act of 2009, expansion of health technology including EMRs was encouraged and financed through the HITECH Act [9] that provided incentives to health care providers to adopt EMRs, in part to improve health care quality [8]. Since that time, considerable effort has been made to convert paper charts to electronic systems, enable under resourced practices to purchase EMR software, train vaccine providers how to consistently record vaccines in the EMR, and use EMR software to prompt providers to vaccinate. Currently, 96% of hospitals and 86% of office-based physicians use EMRs [10]. EMRs have distinct advantages over paper charts, such as legibility, consistent recording in a designated location, and electronic search capability that is more rapid and likely more accurate than manual chart reviews.
Concomitantly, Immunization Information systems (IISs) or vaccine registries have matured. IISs are defined as “confidential, population-based databases of all immunization doses administered by participating providers to persons residing within a given geopolitical area” [11]. These regional (usually statewide) repositories of vaccine administrations have enormous value for ordering vaccines, increasing vaccinations through patient level reminder and recall, provider assessment and feedback [12], and policy making. As of 2016, 55 U.S. jurisdictions, including 49 states and 6 cities had implemented IISs; childhood participation in IISs was 94% in 2016 [13] and adult participation was 24.5% in 2012 [14]. Completeness of vaccine registries varies across jurisdictions because they are governed by their individual jurisdictional policies and regulations [11]. Functional standards for IISs, which were first developed in 2001 and revised several times since then, include having the capability for bidirectional information exchange with EMRs to enhance accuracy of both repositories [15]. However, the Pennsylvania Statewide IIS (PA-SIIS) and 40 others do not mandate adult influenza immunization reporting [16].
For over a decade, the Centers for Disease Control and Prevention (CDC) has funded influenza vaccine effectiveness (VE) studies, namely, the outpatient U.S. Influenza Vaccine Effectiveness Network (US Flu VE) and more recently, the inpatient Hospitalized Influenza Vaccine Effectiveness Network (HAIVEN). These studies use a test-negative case-control design to determine if the rate of influenza vaccination differs among influenza cases compared with non-influenza acute respiratory cases. Thus, validity of the VE estimates is dependent upon accurate assessment of vaccination status. Investigators in both studies use a variety of sources for obtaining vaccination status including self-report, electronic medical records (EMR), and state immunization registries. VE studies are a newer use of state immunization registries; their validity for this purpose has rarely been tested [12, 17].
The major changes in the immunization infrastructure environment, as well as emerging uses for vaccination data warrant a reexamination of the accuracy of the sources of immunization information. The purpose of this study is to determine agreement among different sources of influenza vaccination data from participants in the combined CDC outpatient and inpatient VE studies from a single local health system over four influenza vaccination seasons, 2016–2017 through 2019–2020.
Methods
The University of Pittsburgh Institutional Review Board approved this study. Vaccination data were derived from an outpatient (US Flu VE Network) and an inpatient (HAIVEN) test-negative, case-control influenza VE studies [18, 19]. Consented participants or parents of minor children completed an enrollment interview with questions about receiving influenza vaccine during the season of enrollment that included anytime since July 1. They were asked to provide information about where and when they received the vaccine to assist with verification. A vaccine information sheet and addressed, stamped envelope was given to those who stated that they had information at home, but could not remember it during enrollment, to complete and return to the study team. The vaccinations recorded at enrollment or received in the mail were considered to be self-reported. At the end of enrollment for each season, requests for influenza vaccination data for all participants were made to the health system EMR and the Pennsylvania Statewide IIS (PA-SIIS). There are established systems for requesting data from each of these sources, such as the University of Pittsburgh’s Clinical Translational Science Institute’s Health Record Research Request (R3) system that provides clinical data from the health system’s EMR for research studies.
Most clinics (doctor’s offices, hospitals, health centers, mass vaccination clinics, etc.) that report immunization data to the PA-SIIS use an HL7 interface (PA-SIIS personal communication 2021). This health system’s EMR and PA-SIIS use a bidirectional interface such that immunization data entered into the EMR is “live-feed” uploaded to PA-SIIS; whereas, the health system’s EMR routinely queries PA-SIIS in batches to receive immunization data reports that are then uploaded to the EMR. These reports are typically automatically integrated into the immunization record
Finally, research assistants telephoned pharmacies, work sites, community clinics, primary care providers, etc. to confirm self-reported vaccinations that could not be confirmed in the EMR or PA-SIIS. Copies of the signed consent forms were provided to these sources to facilitate release of the information. Verbal vaccine verifications were recorded by research assistants or documentation was faxed to the research office for recording. Vaccinations were considered verified if they were located in PA-SIIS, the EMR or reported from one of the other sources. Vaccinations that were recorded in the EMR or PA-SIIS but had not been reported by the participant, were called “discovered” vaccinations. They are reported herein, but were not included in the agreement analyses.
Statistical Analysis
Baseline measures by influenza season are reported using counts and percentages. Differences in vaccination records across age and racial groups and the differences in reporting between EMR and PA-SIIS (bi-directional data transfer) were tested by Chi-square statistic. The latter comparisons were made as a test of the quality of the bidirectional data transfer. We compared the percent of self-reported influenza vaccines confirmed in PA-SIIS over the four seasons to determine if reporting changed over time using the Cochran-Armitage trend test [20] with two-sided p-values.
We used two vaccination sources as the gold standard, self-report and PA-SIIS in separate sets of analyses. The percent agreement between the gold standard and the other measures was calculated as the number of vaccinated and unvaccinated participnts that agreed in both sources divided by the total number of participants. To assess the test’s validity to discriminate between two verification sources, we evaluated percent agreement, Cohen’s kappa, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the ROC curve. These values were calculated for all four seasons combined and by age group and race category. The kappa statistic was interpreted as 0–0.20 no agreement, 0.21–0.39 minimal agreement; 0.40-.059 weak agreement; 0.60–0.79 moderate agreement; 0.80–0.90 strong agreement and; >0.90 almost perfect agreement [21].
We also compared the bidirectional transfer of immunization data between PA-SIIS and the EMR by dividing the vaccinations recorded in both sources by: a) the vaccines in PA-SIIS only; and b) the vaccines in the EMR. All statistical analyses were carried out using SAS version 9.4 (SAS Inc., Cary, NC).
Results
A total of 9,930 participants were enrolled in the combined VE studies over the four seasons. In general, approximately one fifth were children, over half (56%) were adults 18–64 years and about one fourth were older adults ≥65 years old. Over half (57.5%) of the participants were female, 27.6% were non-white, 2.4% identified as Hispanic and 60% self-reported influenza vaccination. Table 1 summarizes the demographic characteristics of participants included in these analyses overall and for each season.
Table 1.
Baseline characteristics of participants by influenza season 2016–2017 through 2019–2020
| Season | |||||
|---|---|---|---|---|---|
| Characteristics | 2016–2017 (N=2294) | 2017–2018 (N=2324) | 2018–2019 (N=2839) | 2019–2020 (N=2473) | Overall (N=9930) |
| n (%) | n (%) | n (%) | n (%) | n (%) | |
|
| |||||
| Age Group | |||||
| 6 months - 17 years | 417 (18.2) | 399 (17.2) | 686 (24.2) | 534 (21.7) | 2036 (20.5) |
| 18 – 64 years | 1250 (54.5) | 1360 (58.6) | 1552 (54.7) | 1387 (56.3) | 5549 (56.0) |
| ≥65 years | 627 (27.3) | 563 (24.2) | 601 (21.2) | 541 (22.0) | 2332 (23.5) |
| Female sex, ref. = male | 1300 (56.7) | 1316 (56.7) | 1644 (57.9) | 1439 (58.5) | 5699 (57.5) |
| Non-white race, ref. = white | 519 (22.7) | 635 (27.6) | 912 (32.4) | 652 (26.4) | 2718 (27.6) |
| Hispanic ethnicity, ref. = non-Hispanic | 45 (2.0) | 48 (2.1) | 81 (2.9) | 62 (2.5) | 236 (2.4) |
| Self-reported vaccination | 1427 (62.2) | 1362 (58.6) | 1679 (59.1) | 1476 (59.7) | 5944 (59.9) |
Figure 1 shows the vaccination status and source of vaccine information for the study population and Table 2 shows the self-reported vaccinations, discovered vaccinations, as well as the number and percent of vaccinations verified through the various sources. Ten percent of vaccinations were “discovered,” that is, vaccinations that were recorded among those who had reported not being vaccinated. PA-SIIS represented the largest source of verified vaccinations (76.3%; range = 72.9%–92.5% across the various seasons and age groups), followed by the EMR (47.7%; range = 40.6% – 65.9%) and other sources (13.7%; range = 10.1% – 16.4%). Combining all sources, 94.8% of self-reported vaccines could be verified. The highest verification in both PA-SIIS and the EMR was for children and the lowest verification in both sources was for working age adults (P≤0.001 across all age groups). Non-white participants compared with white participants had significantly higher verification in PA-SIIS and the EMR (P<0.001).
Figure 1.
Sources of vaccination data in study population
Table 2.
Recorded influenza vaccinations from various sources for all participants for influenza seasons 2016–2017 through 2019–2020
| Vaccinations verified through | ||||||
|---|---|---|---|---|---|---|
| Season/group | Self-reported vaccination N | PA-SIISa N (%) | EMRb N (%) | Other sourcesc N (%) | All sourcesd N (%) | Discovered vaccinatione N (%) |
|
| ||||||
| Overall | 5944 | 4534 (76.3) | 2833 (47.7) | 814 (13.7) | 5634 (94.8) | 596 (10.6) |
| 2016–2017 | 1427 | 1075 (75.3) | 679 (47.6)f | 195 (13.7)f | 1309 (91.7) | 132 (10.1) |
| 2017–2018 | 1362 | 1073 (78.8) | 588 (43.2)f | 208 (15.3)f | 1313 (96.4) | 116 (8.8) |
| 2018–2019 | 1679 | 1278 (76.1) | 869 (51.8)f | 169 (10.1)f | 1641 (97.7) | 208 (12.7) |
| 2019–2020 | 1476 | 1108 (75.1) | 697 (47.2)f | 242 (16.4)f | 1371 (92.9) | 140 (10.2) |
| 6 mon-17 years | 1039 | 961 (92.5)f | 685 (65.9)f | 110 (10.6)g | 1100 (105.9) | 192 (17.5) |
| 18–64 years | 3054 | 2228 (73.0)f | 1239 (40.6)f | 459 (15.0)g | 2868 (93.9) | 286 (10.0) |
| ≥65 years | 1846 | 1345 (72.9)f | 908 (49.2)f | 243 (13.2)g | 1663 (90.1) | 117 (7.0) |
| White race | 4491 | 3379 (75.2)f | 2002 (44.6)f | 630 (14.0) | 4179 (93.1) | 322 (7.7) |
| Non-white race | 1424 | 1133 (79.6)f | 816 (57.3)f | 178 (12.5) | 1425 (100.1) | 266 (18.7) |
PA-SIIS = Pennsylvania Statewide Immunization Information System
EMR = Health system electronic medical record
Other sources = Pharmacies, employer records, insurer records, physician offices, etc.
All sources = PA-SIIS +EMR + Other sources; vaccines recorded in more than one source are only counted once for All sources.
Discovered vaccination = vaccines found in various sources among those who self-reported no vaccination. .%=discovered/all sources
P≤0.001 for differences across groups
P≤0.01 for differences across groups
Agreement analyses were conducted using: 1) self-report as the gold standard; and 2) PA-SIIS as the gold standard, shown in Tables 3 and 4, respectively. When self-report was compared with EMR records, agreement was low (62%), as indicated by a kappa of 0.29 (minimal agreement). When self-report was compared with PA-SIIS only, agreement increased to 77%, and kappa was 0.54 (weak agreement). Using all possible sources of vaccination data agreement with self-reported vaccinations rose to 85%, with a sensitivity and specificity of 85% and kappa of 0.69 (moderate agreement). Supplemental Tables 1–3 show the agreement values with self-report for age and race groups.
Table 3.
Agreement between self-report (gold standard) and alternate sources for influenza vaccination status, N=9930
| EMR | |||||||
|---|---|---|---|---|---|---|---|
| Vaccinations reported in each source | Agreement | Kappa (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | Sensitivity, % (95% CI) | Specificity, % (95% CI) | Area under ROC curve |
 Unvaccinated = 3641 |
62% | 0.29 (.28, .31) | 88 (87, 89) | 51 (50, 52) | 42 (41, 43) | 91 (90, 92) | 67% |
|
|
|||||||
| PA-SIIS | |||||||
 Unvaccinated = 3525 |
77% | 0.54 (.52, .55) | 90 (89, 91) | 65 (64, 67) | 69 (67, 70) | 88 (87, 89) | 78% |
| All other sources | |||||||
 Unvaccinated = 3390 |
85% | 0.69 (.67, .70) | 89 (88, 90) | 79 (78, 80)) | 85 (84, 86) | 85 (84, 86) | 85% |
Table 4.
Agreement between PA-SIIS (gold standard) and alternate sources for influenza vaccination status, N=9930
| EMR | |||||||
|---|---|---|---|---|---|---|---|
| Vaccines reported in each source | Agreement | Kappa (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | Sensitivity, % (95% CI) | Specificity, % (95% CI) | Area under ROC curve |
 Unvaccinated = 5110 |
77% | 0.52 (.51, .54) | 90 (89, 91) | 72 (71, 73) | 56 (55, 58) | 95 (94, 95.3) | 75% |
|
| |||||||
| Self-report, EMR, and other sources | |||||||
 Unvaccinated = 3390 |
78% | 0.57 (.56, .58) | 68 (67, 70) | 95 (94, 96) | 96 (95, 97 | 63 (62, 64) | 79% |
When PA-SIIS was used as the gold standard and compared with the EMR, agreement was 77%, and kappa was 0.52. Adding self-report and other sources to the EMR data resulted in agreement with PA-SIIS of 78%, sensitivity of 96%, specificity of 63% and kappa of 0.57 (weak agreement). Supplemental Tables 4 and 5 show the agreement values with PA-SIIS for age and race groups.
Figures 2a and 2b show the percent of self-reported influenza vaccines confirmed in PA-SIIS over the four seasons. Capture of influenza vaccines in the state registry increased significantly over time among children (test for trend P=0.001), decreased significantly for older adults (P=0.004), but did not change significantly among younger adults, nor in either racial group. The transfer of data between the EMR and PA-SIIS was further explored and is shown in Table 5. The percent of PA-SIIS vaccines found in the EMR (transfer from EMR to PA-SIIS) overall was high at nearly 90% and was 90% or higher for most seasons and age and race groups. Notable exceptions were the 2018–2019 season (77.7%), working-age adults (85.4%) and non-white participants (86%). Over the four seasons, transfer from the EMR to PA-SIIS increased significantly (P=0.036). Conversely, the transfer of data from PA-SIIS to the EMR was considerably lower at 56.2% overall and the changes over time were not significant (P=0.447).
Figure 2a and 2b.
Percent of self-reported influenza vaccinations captured in PA-SIIS each season by age group (2a) and race (2b)
Table 5.
Completeness of bidirectional interface between PA-SIIS and EMR
| Vaccination recorded in: | Vaccinations sent from: | ||||
|---|---|---|---|---|---|
| Both PA-SIIS and EMR | PA-SIIS only | EMR only | PA-SIIS to EMR | EMR to PA-SIIS | |
| (a) | (b) | (c) | (a/b*100) | (a/c*100) | |
|
| |||||
| Overall | 2547 | 4534 | 2833 | 56.2% | 89.9% |
| Season | |||||
| 2016–2017 | 640 | 1075 | 679 | 59.5% | 94.3% |
| 2017–2018 | 556 | 1073 | 588 | 51.8% | 94.6% |
| 2018–2019 | 675 | 1278 | 869 | 52.8% | 77.7% |
| 2019–2020 | 676 | 1108 | 697 | 61.0% | 97.0% |
| Age group | |||||
| 6m-17 years | 656 | 961 | 685 | 68.3% | 95.8% |
| 18–64 years | 1058 | 2228 | 1239 | 47.5% | 85.4% |
| >= 65 years | 833 | 1345 | 908 | 61.9% | 91.7% |
| Race | |||||
| White | 1832 | 3379 | 2002 | 54.2% | 91.5% |
| Non-White | 702 | 1133 | 816 | 62.0% | 86.0% |
Discussion
This study compared several sources of influenza vaccination status that have been used in influenza research studies. These sources include EMR, PA-SIIS, other records and self-report. National and regional surveys of vaccine receipt to determine vaccination coverage (NHIS, NIS, BRFSS) rely strictly on self-report, with the general acknowledgement of the recall bias and response bias inherent to that source of vaccination status. In influenza VE studies, self-reported vaccination status has been verified using EMR data, IIS records, employee health records and manual tracking by research assistants to confirm vaccination performed in private physician’s offices, pharmacies and health plan records, among others. This effort to verify vaccination using all reasonable sources is made because the accuracy of VE estimates is directly related to the accuracy of vaccination status. Thus, understanding the relationships among these sources of influenza vaccination information is essential.
Previous studies of influenza vaccine self-report among older adults compared with medical charts and electronic databases as the “gold standards” have reported sensitivity of 98%−100% indicating that self-report of receiving influenza vaccine was considered to be highly reliable; whereas, self-report of not receiving influenza vaccine was not as reliable, with specificity of 38%−71% [4, 5]. This level of agreement was confirmed in more recent studies comparing the EMR and self-report [6, 7]For example, when the EMR was the gold standard, sensitivity of self-report was 93% and specificity was 65% overall and among those age 65 years and older, sensitivity was 96% and specificity was 50% [6, 7]. When self-report was the gold standard, sensitivity of the EMR ranged from 51% to 89% [6, 7];.In the current study, 10% of the total influenza vaccinations had been reported by participants as not received, yet were discovered in other sources including the EMR. In contrast, a study in the UK, where nearly all vaccines are administered through the primary care office, 5% of vaccinations reported in the EMR were self-reported as not received [22].
In this study, agreement between self-report and EMR alone was 62%, compared with 68%, 89% and 92% agreement reported previously [4, 5]. Agreement between self-report and PA-SIIS was 77%, and agreement between self-report and combined EMR, PA-SIIS and other sources was 85%.
These agreement values may reflect an increase in vaccination sites available to this study’s participants compared with previous studies. For example, this study included a large proportion of working age adults who might receive worksite influenza vaccines that we were unable to confirm or may reflect the addition of pharmacy vaccination sites that has occurred since the licensure of pharmacists as vaccinators. These sites may not routinely report vaccinations to the IIS, thus reducing the number of vaccines that can be verified.
If self-report is not a suitable gold standard for influenza vaccination status, is the IIS a viable option? Both EMRs and IISs have matured and become more widely used [13] over the past two decades. IISs are considered to be less prone to bias than other sources because they are population based and are not limited to those who receive vaccines in medical settings [12]. However, their level of completeness may determine their usefulness for research. Previous studies have examined the completeness of local IISs compared with EMRs and their suitability for research. Five studies were identified, all of which included only children [23–27]. Completeness ranged from 82% for influenza vaccination [26], to 97% [25] and 100% [24] for up-to-date childhood vaccines in clinics that used direct electronic transfer of data from the EMR to the IIS. Jackson et al. reported at least 97% match between the IIS and EMR data for primary data elements including vaccine and date of administration [27]. The authors concluded that analyses using IISs should be limited to only those variables that are IIS-required because they have high internal validity, completeness and accuracy. A study conducted in Spain in 2011–2012 reported a 31% overestimation of influenza vaccination coverage among adults by self-report compared with a computerized vaccination registry that was linked to an EMR [28]. Additional barriers to accurate record-keeping are identical names for different people or multiple records for the same person with slightly or completely different names.
Over four years, we did not find a consistent increase in the influenza vaccination coverage reported in PA-SIIS. Influenza vaccination coverage increased significantly in children, decreased significantly in older adults, and did not change among working age adults, while coverage decreased in white participants and increased in non-white participants. This lack of consistent improvement in coverage may again reflect increased use of non-medical office vaccination sites that are not required to report to PA-SIIS. Other sources of error may be imperfections in bi-directionality of the transfer of vaccination data. We found that the transfer from the EMR to PA-SIIS was high, but not so from PA-SIIS to the EMR. This disparity may derive from the intricacies of creating interfaces that a) respect patient privacy in the EMR; b) allow easy incorporation of IIS data into the EMR; and c) direct the data to the appropriate place in the EMR where electronic data searches of immunizations are made. Another, potential, factor that may impede data transfer from the IIS to health system EMRs is the need to communicate with an assortment of EMR software packages within its jurisdictional boundaries. Furthermore, as health systems expand, they may have facilities in more than one IIS jurisdiction and therefore have different reporting rules applied to them.
It is also possible that we did not observe increased adult vaccination reporting to PA-SIIS over time because it has peaked and will not increase further without changes to state laws that mandate adult vaccination reporting by all vaccine administrators, and in other jurisdictions, remove opt in/opt out and consent barriers to inclusion in the IIS.
Can the IIS be used as a reliable source of influenza immunization data for research? A 2013 review of IIS use in public health research found only three VE studies, none of which was for influenza [12]. A 2012 study of influenza VE was conducted among children using the New York City Immunization Registry. The authors concluded that registry data can be used to estimate VE in a pandemic when rapid turnaround in results is needed and the IIS is well established [17]. When PA-SIIS was used as the gold standard against the EMR and all other sources used for influenza VE studies, agreement between the sources was modest indicating that IISs may be acceptable sources of vaccination data for some studies, but when high accuracy data are required, self-reported data, verified using multiple sources, may be a better choice.
As of 2015, 96% (51) of the 53 IISs in the U.S. were authorized to collect immunization data in all age groups, including adults, while only 12 had a mandate to collect adult immunization data [16]. Some IIS programs collecting adult immunization data allow adults to opt out, or require explicit consent for vaccination data to be transferred to the IIS. These stipulations potentially affect the quality and quantity of adult immunization data in IISs across the country and may limit their usefulness as the gold standard for vaccination information. Other efforts to improve IIS reporting might include IIS reminder information included with every vaccine shipment, educational presentations to businesses and agencies that offer vaccination services or host vaccination clinics, and incentives to those groups for reporting to the IIS. Organizations such as the Association of Immunization Registries of America (AIRA), the National Adult and Influenza Immunization Summit (NAIIS) provide resources that promote documentation of immunizations in IISs.
Since these data were collected, we have experienced a coronavirus pandemic that resulted in the development of new COVID-19 vaccines. Unlike other adult vaccines, reporting of all COVID-19 vaccines is required. This requirement should result in significantly better capture of these vaccinations by the IISs. Furthermore, as of this writing, countless businesses, agencies and events require proof of vaccination before admission, thus enhancing self-report accuracy. Future research comparing PA-SIIS as the gold standard for COVID-19 vaccines may demonstrate the value of mandatory vaccine reporting to state immunization information systems.
Strengths and Limitations
Data from two influenza VE studies that used nearly identical means to assess self-reported vaccination status over four influenza seasons were combined to increase sample size. There may have been other sources of data that were not used to confirm self-report because the return on the time investment was too low. Research indicates that over 40% of adults receive the influenza vaccine in non-medical settings [29]. If reporting of adult and/or influenza vaccines is not mandatory for these settings, many self-reported vaccinations may not be verified in the IIS. The findings in this study may not be generalizable to all locales. For example, some states may have stricter reporting policies which may result in higher agreement between self-report and the IIS, or may not include adults at all in the IIS. By comparing PA-SIIS influenza vaccine coverage over four years, we found that in Pennsylvania, childhood reporting continues to improve, but not adult reporting. The interfaces between EMRs and IISs are continually improving and past results may not correspond with current functionality.
Conclusions
Both self-report and IIS provide reasonable estimates of actual vaccine uptake. While IISs may provide valid and highly accurate reports of childhood influenza vaccinations, their use for adult influenza research may be less reliable without reporting mandates. When high accuracy data are required, such as for vaccine effectiveness studies, triangulation using multiple sources should be conducted.
Supplementary Material
Funding:
This work was supported by the Centers for Disease Control and Prevention (CDC) [5U01IP001035] and by National Institutes of Health (NIH) [UL1TR001857]. This work represents the views of the authors and not the CDC or NIH. It is subject to CDC’s public access policy.
Competing interests:
RKZ and JMR have research funding from Sanofi Pasteur. MPN, GKB, TMS and HAD have research funding from Merck & Co., Inc. on an unrelated project. MPN has participated in the Merck pneumococcal expert forum. SGS and MS have no conflicts to report. GKB serves as an advisor or consultant at New World Medical Inc., Ocular Therapeutix, Inc., and Sight Sciences, Inc.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Literature Cited
- 1.Increased Interseasonal Respiratory Syncytial Virus (RSV) Activity in Parts of the Southern United States [https://emergency.cdc.gov/han/2021/han00443.asp]
- 2.About the National Health Interview Survey [https://www.cdc.gov/nchs/nhis/about_nhis.htm]
- 3.Behavioral Risk Factor Surveillance System (BRFSS) [https://www.cdc.gov/brfss/]
- 4.Mac Donald R, Baken L, Nelson A, Nichol KL: Validation of self-report of influenza and pneumococcal vaccination status in elderly outpatients. Am J Prev Med 1999, 16(3):173–177. [DOI] [PubMed] [Google Scholar]
- 5.Zimmerman RK, Raymund M, Janosky JE, Nowalk MP, Fine MJ: Sensitivity and specificity of patient self-report of influenza and pneumococcal polysaccharide vaccinations among elderly outpatients in diverse patient care strata. Vaccine 2003, 21(13–14):1486–1491. [DOI] [PubMed] [Google Scholar]
- 6.Greene SK, Shi P, Dutta-Linn MM, Shoup JA, Hinrichsen VL, Ray P, Nordin JD, Kuckler L, Weintraub ES, Yih WK: Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med 2009, 37(6):552–555. [DOI] [PubMed] [Google Scholar]
- 7.Rolnick SJ, Parker ED, Nordin JD, Hedblom BD, Wei F, Kerby T, Jackson JM, Crain AL, Euler G: Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine 2013, 31(37):3928–3935. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Congressional Intent for the HITECH Act [https://www.ajmc.com/view/ajmc_10dechit_stark_sp24tp28?p=1] [PubMed]
- 9.Health Information Technology for Economic and Clinical Health (HITECH) Act. In.; 2009.
- 10.Health IT Dashboard [https://dashboard.healthit.gov/quickstats/quickstats.php]
- 11.Immunization Information Systems (IIS) [https://www.cdc.gov/vaccines/programs/iis/about.html]
- 12.Curran EA, Bednarczyk RA, Omer SB: Evaluation of the frequency of immunization information system use for public health research. Hum Vaccin Immunother 2013, 9(6):1346–1350. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Murthy N, Rodgers L, Pabst L, Fiebelkorn AP, Ng T: Progress in Childhood Vaccination Data in Immunization Information Systems - United States, 2013–2016. MMWR Morb Mortal Wkly Rep 2017, 66(43):1178–1181. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Centers for Disease Control and Prevention (CDC): Progress in immunization information systems - United States, 2012. MMWR Morb Mortal Wkly Rep 2013, 62(49):1005–1008. [PMC free article] [PubMed] [Google Scholar]
- 15.Immunization Information Systems (IIS) Functional Standards [https://www.cdc.gov/vaccines/programs/iis/functional-standards/func-stds-v4-0.html]
- 16.Martin DW, Lowery NE, Brand B, Gold R, Horlick G: Immunization information systems: a decade of progress in law and policy. J Public Health Manag Pract 2015, 21(3):296–303. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hadler JL, Baker TN, Papadouka V, France AM, Zimmerman C, Livingston KA, Zucker JR: Effectiveness of 1 dose of 2009 influenza A (H1N1) vaccine at preventing hospitalization with pandemic H1N1 influenza in children aged 7 months-9 years. J Infect Dis 2012, 206(1):49–55. [DOI] [PubMed] [Google Scholar]
- 18.Ohmit SE, Thompson MG, Petrie JG, Thaker SN, Jackson ML, Belongia EA, Zimmerman RK, Gaglani M, Lamerato L, Spencer SM et al. : Influenza vaccine effectiveness in the 2011–2012 season: protection against each circulating virus and the effect of prior vaccination on estimates. Clin Infect Dis 2014, 58(3):319–327. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Ferdinands JM, Gaglani M, Martin ET, Middleton D, Monto AS, Murthy K, Silveira FP, Talbot HK, Zimmerman R, Alyanak E et al. : Prevention of Influenza Hospitalization Among Adults in the United States, 2015–2016: Results From the US Hospitalized Adult Influenza Vaccine Effectiveness Network (HAIVEN). J Infect Dis 2019, 220(8):1265–1275. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Agresti A: A Survey of Exact Inference for Contingency Tables. Statistical Science 1992, 7(1):131–153. [Google Scholar]
- 21.McHugh ML: Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012, 22(3):276–282. [PMC free article] [PubMed] [Google Scholar]
- 22.Mangtani P, Shah A, Roberts JA: Validation of influenza and pneumococcal vaccine status in adults based on self-report. Epidemiol Infect 2007, 135(1):139–143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Boyd TD, Linkins RW, Mason K, Bulim I, Lemke B: Assessing immunization registry data completeness in Bexar County, Texas. Am J Prev Med 2002, 22(3):184–187. [DOI] [PubMed] [Google Scholar]
- 24.Kolasa MS, Chilkatowsky AP, Clarke KR, Lutz JP: How complete are immunization registries? The Philadelphia story. Ambul Pediatr 2006, 6(1):21–24. [DOI] [PubMed] [Google Scholar]
- 25.Koepke R, Petit AB, Ayele RA, Eickhoff JC, Schauer SL, Verdon MJ, Hopfensperger DJ, Conway JH, Davis JP: Completeness and accuracy of the wisconsin immunization registry: an evaluation coinciding with the beginning of meaningful use. J Public Health Manag Pract 2015, 21(3):273–281. [DOI] [PubMed] [Google Scholar]
- 26.Hendrickson BK, Panchanathan SS, Petitti D: Evaluation of immunization data completeness within a large community health care system exchanging data with a state immunization information system. J Public Health Manag Pract 2015, 21(3):288–295. [DOI] [PubMed] [Google Scholar]
- 27.Jackson ML, Henrikson NB, Grossman DC: Evaluating Washington State’s immunization information system as a research tool. Acad Pediatr 2014, 14(1):71–76. [DOI] [PubMed] [Google Scholar]
- 28.Jimenez-Garcia R, Hernandez-Barrera V, Rodriguez-Rieiro C, Carrasco Garrido P, Lopez de Andres A, Jimenez-Trujillo I, Esteban-Vasallo MD, Dominguez-Berjon MF, de Miguel-Diez J, Astray-Mochales J: Comparison of self-report influenza vaccination coverage with data from a population based computerized vaccination registry and factors associated with discordance. Vaccine 2014, 32(35):4386–4392. [DOI] [PubMed] [Google Scholar]
- 29.National and State-Level Place of Flu Vaccination among Vaccinated Adults in the United States, 2014–15 Flu Season [https://www.cdc.gov/flu/fluvaxview/place-vaccination-2014-15.htm]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.