Abstract
Objectives:
Having accurate influenza vaccination coverage estimates can guide public health activities. The objectives of this study were to (1) validate the accuracy of electronic health record (EHR)–based influenza vaccination data among pregnant women compared with survey self-report and (2) assess whether survey respondents differed from survey nonrespondents by demographic characteristics and EHR-based vaccination status.
Methods:
This study was conducted in the Vaccine Safety Datalink, a network of 8 large medical care organizations in the United States. Using EHR data, we identified all women pregnant during the 2018-2019 or 2019-2020 influenza seasons. Surveys were conducted among samples of women who did and did not appear vaccinated for influenza according to EHR data. Separate surveys were conducted after each influenza season, and respondents reported their influenza vaccination status. Analyses accounted for the stratified design, sampling probability, and response probability.
Results:
The survey response rate was 50.5% (630 of 1247) for 2018-2019 and 41.2% (721 of 1748) for 2019-2020. In multivariable analyses combining both survey years, non-Hispanic Black pregnant women had 3.80 (95% CI, 2.13-6.74) times the adjusted odds of survey nonresponse; odds of nonresponse were also higher for Hispanic pregnant women and women who had not received (per EHR data) influenza vaccine during current or prior influenza seasons. The sensitivity, specificity, and positive predictive value of EHR documentation of influenza vaccination compared with self-report were ≥92% for both survey years combined. The negative predictive value of EHR-based influenza vaccine status was 80.5% (95% CI, 76.7%-84.0%).
Conclusions:
EHR-based influenza vaccination data among pregnant women were generally concordant with self-report. New data sources and novel approaches to mitigating nonresponse bias may be needed to enhance influenza vaccination surveillance efforts.
Keywords: influenza vaccination, pregnancy, misclassification, electronic health records, nonresponse bias
Influenza viruses cause substantial morbidity and mortality every year in the United States and globally,1-4 including among pregnant women who encounter higher risk of severe disease compared with nonpregnant women of similar age.5-7 Vaccination is the primary means of influenza prevention,8,9 and improving influenza vaccination coverage is a national public health priority. 10 Influenza vaccination coverage among pregnant women in the United States is primarily monitored through in-person,11,12 telephone, 13 and internet-based surveys,5,14,15 with respondents asked to report their vaccination status. While survey-based methods for assessing influenza vaccination coverage have strengths, important limitations are operant.13,16 Self-report of vaccination may be inaccurate by mistake or because of social desirability bias, survey response rates are generally falling, and survey-based methods are subject to nonresponse bias. 17 In addition, if survey response is more likely from vaccinated people than from unvaccinated people, 18 influenza vaccination coverage based on national surveys could overestimate true coverage.
Electronic health record (EHR) 19 and claims data 20 have also been used to estimate influenza vaccination coverage during pregnancy, an approach that may mitigate some limitations of survey-based methods. EHR data can be used to reliably identify pregnant women, 21 assess other risk factors for influenza disease,8,9 and obtain demographic information such as race and ethnicity. In addition, if EHR data indicate that a pregnant woman was vaccinated during the current influenza season, the positive predictive value (PPV) is high compared with self-report. 18 However, if EHR data do not indicate influenza vaccination, it is possible that a pregnant woman was vaccinated outside the medical home, such as at a workplace or pharmacy.18,22 In this circumstance, EHR data would underestimate true coverage.
Having accurate and timely influenza vaccination coverage estimates, particularly in groups at increased risk of serious influenza disease,5-7 is necessary to guide public health activities8,9 toward achieving national public health goals. 10 The objectives of the current study were to (1) assess by survey the validity of EHR-based influenza vaccination data among pregnant women and (2) assess whether survey respondents differed from survey nonrespondents by demographic characteristics and EHR-based vaccination status.
Methods
Study Setting
This study was conducted in the Vaccine Safety Datalink (VSD), a collaboration between the Centers for Disease Control and Prevention and 8 large medical care organizations (referred to as “sites”).23-25 Participating VSD sites included Marshfield Clinic, HealthPartners, Denver Health, Kaiser Permanente (KP) Washington, KP Northwest, KP Northern California, KP Southern California, and KP Colorado. Denver Health cares for publicly insured, privately insured, and uninsured patients26,27; the remaining VSD sites primarily serve insured patients.23-25 The demographic and socioeconomic characteristics of the VSD population are similar to those of the US population, 25 while influenza vaccination coverage among pregnant women in the VSD is generally higher than coverage indicated by national estimates. 19
Vaccination data at VSD sites, which are used for clinical and research purposes, are derived from several data sources. Vaccines ordered and administered within VSD sites represent a high proportion of available vaccine records. In addition, vaccine data from claims and state immunization information systems (at 6 VSD sites) are added to a patient’s EHR.19,28,29 Influenza vaccines administered in pharmacies would be integrated into EHR-based vaccination data if a claim were submitted to insurance or if the pharmacy submitted vaccine data to a state immunization information system. In the current study, the term “EHR-based vaccination data” refers to all available electronic vaccination data integrated into an individual’s record.
Study Overview
We identified all women at VSD sites who were pregnant during the 2018-2019 or 2019-2020 influenza season. We then conducted surveys among a sample of pregnant women who did and did not appear vaccinated during the same season according to EHR data. Pregnant women were surveyed during only 1 influenza season. Respondents were asked to report their influenza vaccination status for the current season, which was compared with EHR-based vaccination data. We considered vaccination self-report as the criterion standard.
Study Population: Identification of Pregnant Women
The VSD has developed and validated an algorithm to identify pregnancies with a high degree of accuracy. 21 The algorithm can identify current pregnancies in near-real time, using weekly updates of VSD EHR-based data. 30 The following data sources are used: pregnancy-related International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes, 31 prenatal procedure codes, and estimated delivery and last menstrual period dates from obstetric flowsheets.21,30
Using this algorithm, we identified all women aged 18-49 years who were pregnant at any time from August 1 through January 31 of the respective influenza season (eg, for the 2018-2019 survey, pregnant from August 1, 2018, through January 31, 2019). We required women to have continuous health insurance enrollment through the influenza season (for Denver Health, 1 outpatient visit was used as a proxy for health insurance enrollment). We excluded women with an ICD-10-CM code indicating an adverse pregnancy outcome such as spontaneous abortion, stillbirth, or anencephaly. We excluded from analysis women with multiple pregnancy episodes during a single influenza season, women with a diagnosis code for a vaccine allergy, and women with presumed influenza vaccine data errors (eg, receipt of a nonindicated vaccine, such as high-dose influenza). For the 2018-2019 survey, 439 of 107 126 pregnant women were excluded for vaccine allergy or invalid vaccine data; for the 2019-2020 survey, 409 of 97 661 pregnant women were excluded.
Survey Design and Administration
Sampling procedures
After identifying all eligible pregnant women in the VSD as our sampling frame, we randomly sampled women for survey administration. We stratified sampling for each survey year by VSD site and EHR-based influenza vaccination status, with women who appeared unvaccinated in EHR data oversampled. The total sample of pregnant women was 1247 for the 2018-2019 survey and 1748 for the 2019-2020 survey. For the 2018-2019 survey, we sampled 999 unvaccinated (per EHR data) and 248 vaccinated (per EHR data) women; for the 2019-2020 survey, the numbers sampled were 1460 unvaccinated and 288 vaccinated. In the 2018-2019 survey, non-Hispanic Black women were less likely to respond than women from other racial and ethnic groups; consequently, non-Hispanic Black women were oversampled for the 2019-2020 survey. With a sample of 999 EHR-unvaccinated women (ie, the 2018-2019 survey), a 50% response rate, and 60% of EHR-unvaccinated women reporting they were unvaccinated, the study was powered to achieve a CI width of 8.8% around the estimated proportion confirming their vaccination status.
Survey instrument
We developed and pilot-tested a survey instrument with 8 individuals, and then revised it accordingly. We based survey questions on published survey instruments; whenever possible, we used exact wording from prior instruments.32-38 Survey content included whether the respondent received an influenza vaccine and, if so, at what location; whether influenza vaccine had been recommended and offered by a health professional; if the respondent was not vaccinated, the primary reason why not; and attitudes about influenza vaccination. We also asked women to confirm that they were currently or had recently been pregnant; we excluded from analyses women who did not confirm pregnancy on the survey.
Survey administration
For the 2018-2019 influenza season, survey administration began March 15, 2019; for the 2019-2020 influenza season, survey administration began February 18, 2020. Surveys were in the field for 15 weeks. Women received up to 3 mailed surveys, up to 5 emails with a unique hyperlink to an internet-based survey, and up to 2 automated telephone reminders. Outreach stopped once the survey was completed or a person requested no further contact. In nearly all aspects, survey administration was consistent across survey years and VSD sites. One VSD site required that participants receive a pre-survey letter with an opportunity to opt out from the survey and did not permit email contact; consequently, at this site, participants received an additional mailed survey. For the 2018-2019 influenza season, a Spanish-language version of the survey was sent to women with an EHR designation of preferred language Spanish. Because of resource constraints, the 2019-2020 survey was available in English only. The internet form of the survey used Research Electronic Data Capture. 39 Respondents received a $20 gift card for completing the survey.
Analytic Methods
Pregnant women were considered survey respondents if they answered the primary survey question of “Since July 1 of [the current influenza season] have you had a flu vaccination?” We used the Pearson χ2 test to compare respondents with nonrespondents, with P < .05 considered significant. Self-reported vaccination status was treated as the criterion standard for all analyses. We accounted for the stratified sampling design, included a finite population correction, and incorporated inverse probability weighting for sampling and survey response probabilities. For the 2018-2019 season analyses, sampling weights accounted for VSD site and EHR vaccination status; for the 2019-2020 season analyses, weighting also accounted for oversampling non-Hispanic Black women. Weighted percentages for survey responses and EHR vaccination validity measures (sensitivity, specificity, PPV, and negative predictive value [NPV]) were reported with Clopper–Pearson 95% CIs. 40 In the context of this study, sensitivity is interpreted as follows: among all individuals who self-report influenza vaccination, what percentage have EHR documentation of influenza vaccination? When presenting descriptive frequencies, we combined survey results across influenza seasons. We estimated κ coefficients and associated CIs by using bootstrap replication variance estimation.
Two multivariable logistic regression models were developed: a model assessing factors associated with survey nonresponse (among the entire sample) and a model assessing factors associated with self-reporting influenza vaccination (among women unvaccinated according to EHR data). We explored correlation within VSD site and tested for the interaction between race and ethnicity and vaccination status. Covariates, identified a priori based on prior publications,11,14,15,19 included the following: age group (18-24, 25-34, 35-49 y), race and ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, non-Hispanic Asian, non-Hispanic Other [multiracial, Hawaiian, American Indian/Alaska Native, and unspecified other race and ethnicity]), chronic health condition (yes/no), gravidity (first pregnancy, pregnant before), vaccinated for influenza in prior season (yes/no), health care provider vaccination recommendation (yes/no), health insurance type (private, Medicaid, none), education (≤high school, some college/college graduate, advanced degree), annual household income (<$75 000, ≥$75 000), and VSD site. Covariates were retained in the final models regardless of significance. Multivariate analyses accounted for the study design and sampling strategy, 41 study year, VSD site, EHR vaccination status, and race and ethnicity. VSD site was treated as a fixed effect. In addition, we conducted a subanalysis among pregnant women unvaccinated according to EHR data, examining factors associated with self-reporting influenza vaccination (eg, EHR false negatives). We conducted all analyses using SAS version 9.4 (SAS Institute, Inc).
The KP Colorado Institutional Review Board approved this study, and other study sites ceded oversight to KP Colorado. Written consent was not required for survey administration, and survey participants could opt out in writing or by telephone.
Results
Survey Sample
For the 2018-2019 influenza season, a total of 106 687 survey-eligible pregnant women were identified, of whom 1247 (1.2%) were sampled for survey administration (Table 1). For the 2019-2020 influenza season, a total of 97 244 survey-eligible pregnant women were identified, of whom 1748 (1.8%) were sampled.
Table 1.
Characteristics of respondents and nonrespondents to surveys administered to women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink study sites a
| Characteristic b | 2018-2019 Respondents, no. (%) | 2018-2019 Nonrespondents, no. (%) | P value c | 2019-2020 Respondents, no. (%) | 2019-2020 Nonrespondents, no. (%) | P value c |
|---|---|---|---|---|---|---|
| Total | 630 (100.0) | 617 (100.0) | 721 (100.0) | 1027 (100.0) | ||
| Age, y | <.001 | <.001 | ||||
| 18-24 | 86 (13.7) | 176 (28.5) | 116 (16.1) | 248 (24.1) | ||
| 25-34 | 405 (64.3) | 342 (55.4) | 432 (59.9) | 582 (56.7) | ||
| 35-49 | 139 (22.1) | 99 (16.0) | 173 (24.0) | 197 (9.2) | ||
| Race and ethnicity d | <.001 | <.001 | ||||
| Non-Hispanic White | 398 (63.2) | 244 (39.5) | 299 (41.5) | 211 (20.5) | ||
| Non-Hispanic Black | 28 (4.4) | 77 (12.5) | 234 (32.5) | 550 (53.6) | ||
| Hispanic | 107 (17.0) | 164 (26.6) | 99 (13.7) | 165 (16.1) | ||
| Non-Hispanic Asian | 38 (6.0) | 46 (7.5) | 37 (5.1) | 31 (3.0) | ||
| Non-Hispanic Other | 18 (2.9) | 32 (5.2) | 15 (2.1) | 15 (1.5) | ||
| Missing | 41 (6.5) | 54 (8.8) | 37 (5.1) | 55 (5.4) | ||
| Chronic health condition e | .29 | .12 | ||||
| Yes | 126 (20.0) | 109 (17.7) | 174 (24.1) | 282 (27.5) | ||
| No | 504 (80.0) | 508 (82.3) | 547 (75.9) | 745 (72.5) | ||
| Vaccinated for influenza in current season (per EHR) | <.001 | .004 | ||||
| Yes | 169 (26.8) | 79 (12.8) | 141 (19.6) | 147 (14.3) | ||
| No | 461 (73.2) | 538 (87.2) | 580 (80.4) | 880 (85.7) | ||
| Vaccinated for influenza in prior season (per EHR) | .002 | <.001 | ||||
| Yes | 160 (25.4) | 111 (18.0) | 206 (28.6) | 207 (20.2) | ||
| No | 470 (74.6) | 506 (82.0) | 515 (71.4) | 820 (79.8) |
Abbreviation: EHR, electronic health record.
Vaccine Safety Datalink23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.
Individuals unvaccinated for influenza in the current season (per EHR) were oversampled in both survey years; non-Hispanic Black women were oversampled in the 2019-2020 survey.
Using the Pearson χ2 test, with P < .05 considered significant.
For respondents and nonrespondents, data on race and ethnicity were obtained from the EHR. Other race and ethnicity included 37 multiracial, 15 Hawaiian, and 9 American Indian/Alaska Native women, as well as 19 women with an unspecified other race and ethnicity.
Survey Response
For the 2018-2019 influenza season, 630 of 1247 (50.5%) women responded; for the 2019-2020 influenza season, 721 of 1748 (41.2%) women responded (Table 1). Nearly all respondents (94.9% in 2018-2019, 97.1% in 2019-2020) confirmed they were or had been pregnant. Response rates were highest among non-Hispanic White pregnant women (62.0% in 2018-2019, 58.6% in 2019-2020) and lowest among non-Hispanic Black pregnant women (26.7% in 2018-2019, 29.9% in 2019-2020). Respondents differed significantly from nonrespondents by age group, race and ethnicity, and EHR-documented influenza vaccination in current and prior influenza seasons (all P < .001).
In multivariable analyses, non-Hispanic Black pregnant women had 3.80 (95% CI, 2.14-6.74) times the adjusted odds of not responding to the survey compared with non-Hispanic White pregnant women (both survey years combined) (Table 2). The adjusted odds of nonresponse were higher for Hispanic pregnant women than for non-Hispanic White pregnant women and for those who had not received (per EHR data) influenza vaccine during current or prior seasons compared with those who had received the influenza vaccine.
Table 2.
Multivariable analyses of characteristics associated with nonresponse to influenza vaccination surveys administered to women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink study sites a
| Characteristic | 2018-2019 Influenza season, aOR (95% CI) b | 2019-2020 Influenza season, aOR (95% CI) b | Both seasons combined, aOR (95% CI) b |
|---|---|---|---|
| Age, y | |||
| 18-24 | 1.85 (0.75-4.54) | 1.03 (0.26-4.05) | 1.28 (0.61-2.70) |
| 25-34 | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| 35-49 | 0.75 (0.36-1.56) | 0.50 (0.17-1.46) | 0.59 (0.31-1.13) |
| Race and ethnicity c | |||
| Non-Hispanic White | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Non-Hispanic Black | 4.67 (2.10-10.39) | 2.79 (1.20-6.48) | 3.80 (2.14-6.74) |
| Hispanic | 4.62 (2.07-10.33) | 2.68 (0.77-9.41) | 3.49 (1.54-7.88) |
| Non-Hispanic Asian | 1.57 (0.58-4.26) | 0.45 (0.14-1.44) | 0.84 (0.35-2.03) |
| Non-Hispanic Other d | 4.12 (1.26-13.44) | 2.28 (0.60-8.72) | 2.42 (0.78-7.51) |
| Missing | 1.69 (0.40-7.08) | 1.88 (0.74-4.79) | 1.27 (0.45-3.63) |
| Chronic health condition e | |||
| Yes | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| No | 2.10 (0.89-4.96) | 0.87 (0.30-2.52) | 1.30 (0.69-2.44) |
| Vaccinated for influenza in current season (per EHR) | |||
| Yes | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| No | 1.87 (1.10-4.96) | 1.20 (0.52-2.76) | 1.46 (0.92-2.30) |
| Vaccinated for influenza in prior season (per EHR) | |||
| Yes | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| No | 1.47 (0.68-3.18) | 2.52 (0.82-7.69) | 1.93 (1.02-3.64) |
Abbreviations: aOR, adjusted odds ratio; EHR, electronic health record.
Vaccine Safety Datalink23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.
Each column represents a separate multivariable regression model; models were adjusted for all variables listed and for Vaccine Safety Datalink site.
For respondents and nonrespondents, data on race and ethnicity were obtained from the EHR.
Other race and ethnicity included multiracial, Hawaiian, and American Indian/Alaska Native women and women with an unspecified other race and ethnicity.
Net Implied Bias in Coverage Estimates
In the survey sample for both years combined, the sampling probability weighted prevalence of having received influenza vaccine (per EHR data) was 70.6% (95% CI, 66.2%-74.6%). The sampling probability weighted prevalence of vaccination (per EHR data) among survey respondents was 75.9% (95% CI, 71.0%-80.3%). The net implied bias in coverage would therefore be 5.3% overall. Stratified by race and ethnicity, the net implied bias in coverage would be 6.5% among non-Hispanic Black, 5.2% among non-Hispanic White, and 1.1% among Hispanic pregnant women.
EHR Vaccination Data Compared With Self-Report
Combining results across both survey years, among pregnant women who appeared vaccinated according to EHR data, 8 of 310 self-reported they were not vaccinated (weighted percent false positives: 0.6%; 95% CI, 0.1%-2.3%). Combining both survey years, among pregnant women who appeared unvaccinated according to EHR data, 304 of 1041 self-reported they were vaccinated (weighted percent false negatives: 19.5%; 95% CI, 16.0%-23.3%).
The sensitivity, specificity, and PPV of EHR documentation of influenza vaccination were high for 2018-2019, 2019-2020, and both years combined (Table 3). For example, the sensitivity for both years combined was 92.4% (95% CI, 89.9%-94.5%). The NPV was the lowest of the validity measures in 2018-2019, 2019-2020, and both years combined. The κ coefficient showed strong agreement (κ, both years combined, 0.84; 95% CI, 0.81-0.87) between EHR documentation of influenza vaccination and self-report.
Table 3.
Sensitivity, specificity, positive and negative predictive values, and κ coefficient for EHR documentation of influenza vaccination compared with self-reported vaccination status among women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalinka,b
| Characteristic | No. | Sensitivity | Specificity | Positive predictive value | Negative predictive value | κ coefficient |
|---|---|---|---|---|---|---|
| Year 1 survey (2018-2019) | 630 | 92.7 (89.2-95.4) | 98.8 (96.1-99.8) | 99.6 (97.0-100.0) | 82.0 (76.8-86.4) | 0.86 (0.82-0.89) |
| Year 2 survey (2019-2020) | 721 | 92.1 (87.4-95.5) | 97.4 (92.5-99.5) | 99.1 (95.9-100.0) | 78.8 (72.7-84.1) | 0.83 (0.78-0.88) |
| Years 1 and 2 combined | 1351 | 92.4 (89.9-94.5) | 98.1 (95.7-99.4) | 99.4 (97.7-99.9) | 80.5 (76.7-84.0) | 0.84 (0.81-0.87) |
| Years 1 and 2 combined, by specific strata | ||||||
| Age group, y | ||||||
| 18-24 | 202 | 91.7 (80.5-97.6) | 97.8 (91.1-99.8) | 98.6 (86.7-100.0) | 86.9 (78.4-93.0) | 0.87 (0.78-0.97) |
| 25-34 | 837 | 92.2 (88.8-94.8) | 97.7 (94.0-99.4) | 99.3 (96.8-100.0) | 79.0 (73.8-83.6) | 0.83 (0.79-0.87) |
| 35-49 | 312 | 93.5 (88.5-96.8) | 99.7 (96.9-100.0) | 99.9 (95.6-100.0) | 80.7 (72.4-87.4) | 0.86 (0.80-0.92) |
| Race and ethnicity c | ||||||
| Non-Hispanic White | 697 | 91.7 (87.5-94.9) | 97.8 (92.5-99.7) | 99.2 (96.2-100.0) | 79.0 (73.1-84.1) | 0.83 (0.78-0.88) |
| Non-Hispanic Black | 262 | 83.6 (75.4-89.9) | 97.7 (92.3-99.7) | 96.7 (89.0-99.5) | 88.3 (82.3-92.8) | 0.82 (0.75-0.90) |
| Hispanic | 206 | 95.4 (88.1-98.8) | 97.8 (92.6-99.7) | 99.3 (90.6-100.0) | 87.4 (79.8-92.8) | 0.90 (0.84-0.95) |
| Non-Hispanic Asian | 75 | 90.4 (75.9-97.7) | 100.0 (NA) | 100.0 (NA) | 67.7 (49.5-82.7) | 0.76 (0.61-0.90) |
| Non-Hispanic Other d | 33 | 98.2 (76.4-100.0) | 100.0 (NA) | 100.0 (NA) | 87.1 (67.2-97.2) | 0.92 (0.83-1.00) |
| Missing | 78 | 87.0 (72.7-95.5) | 100.0 (NA) | 100.0 (NA) | 68.7 (51.0-83.1) | 0.75 (0.59-0.91) |
| Chronic health condition e | ||||||
| Yes | 300 | 94.2 (88.9-97.4) | 97.0 (91.3-99.4) | 99.3 (93.9-100.0) | 79.1 (69.7-86.6) | 0.84 (0.77-0.90) |
| No | 1051 | 91.8 (88.4-94.4) | 98.4 (95.4-99.7) | 99.4 (97.3-100.0) | 80.9 (76.5-84.7) | 0.84 (0.81-0.88) |
| Vaccinated for influenza in prior season (per EHR) | ||||||
| Yes | 366 | 95.4 (92.3-97.5) | 93.9 (80.8-99.1) | 99.6 (97.1-100.0) | 53.9 (41.7-65.7) | 0.66 (0.56-0.77) |
| No | 985 | 87.4 (81.7-91.8) | 98.7 (96.2-99.7) | 98.8 (95.4-99.9) | 85.7 (81.8-89.0) | 0.85 (0.80-0.89) |
Abbreviations: EHR, electronic health record; NA, not applicable.
Vaccine Safety Datalink23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.
All values are percentage (95% CI) unless otherwise indicated. Point estimates and 95% CIs accounted for sample design strata, sample and response weights, and finite population correction; κ coefficient and associated 95% CIs were estimated with bootstrap replication variance estimation; 95% CIs for all other measures were estimated with Clopper–Pearson.
Data on race and ethnicity were obtained from the EHR.
Other race and ethnicity included multiracial, Hawaiian, and American Indian/Alaska Native women and women with an unspecified other race and ethnicity.
The validity of EHR documentation of influenza vaccination was similar across age groups and chronic health condition status (Table 3). The NPV was lower among women who (per EHR data) had received influenza vaccine the prior season (53.9%; 95% CI, 41.7%-65.7%) than among women who were unvaccinated (per EHR data) in the prior season (85.7%; 95% CI, 81.8%-89.0%).
We conducted a subanalysis among pregnant women unvaccinated according to EHR data to examine factors associated with self-reporting influenza vaccination (eg, EHR false negatives). In multivariate analyses, for both years combined, the pregnancy being the first pregnancy, having influenza vaccination recommended by a health care provider, and having received (per EHR data) influenza vaccine in the prior season were positively associated with self-reporting influenza vaccination (Table 4).
Table 4.
Multivariable analyses of characteristics associated with failure to capture data on influenza vaccination receipt within EHR data (EHR false negatives) among women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink a
| Characteristic | 2018-2019 Influenza season, aOR (95% CI) b | 2019-2020 Influenza season, aOR (95% CI) b | Both seasons combined, aOR (95% CI) b |
|---|---|---|---|
| Age, y | |||
| 18-24 | 0.73 (0.26-2.06) | 0.99 (0.34-2.95) | 0.70 (0.30-1.64) |
| 25-34 | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| 35-49 | 0.56 (0.18-1.78) | 0.68 (0.26-1.81) | 0.54 (0.27-1.08) |
| Race and ethnicity c | |||
| Non-Hispanic White | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Non-Hispanic Black | 0.29 (0.08-1.10) | 1.01 (0.40-2.50) | 0.72 (0.34-1.49) |
| Hispanic | 0.80 (0.18-3.52) | 1.07 (0.37-3.08) | 0.98 (0.42-2.27) |
| Non-Hispanic Asian | 2.30 (0.60-8.77) | 2.71 (0.62-11.83) | 2.27 (0.88-5.86) |
| Non-Hispanic Other d | 0.80 (0.05-14.01) | 0.47 (0.10-2.15) | 0.38 (0.09-1.54) |
| Missing | 0.79 (0.21-3.06) | 3.31 (1.09-10.09) | 1.86 (0.81-4.29) |
| Chronic health condition e | |||
| Yes | 1.32 (0.47-3.72) | 0.96 (0.35-2.67) | 1.21 (0.65-2.27) |
| No | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Gravidity | |||
| First pregnancy | 3.01 (1.26-7.20) | 2.18 (0.91-5.24) | 2.70 (1.48-4.92) |
| Pregnant before | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Rather not say | 0.01 (0.01-0.01) | 0.46 (0.03-7.31) | 0.87 (0.14-5.46) |
| Vaccinated for influenza in prior season (per EHR) | |||
| Yes | 4.67 (1.97-11.03) | 7.97 (3.40-18.65) | 7.30 (4.00-13.31) |
| No | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Health care provider recommended influenza vaccination | |||
| Yes | 2.36 (0.71-1.09) | 8.08 (2.51-26.00) | 4.46 (1.63-12.17) |
| No | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Rather not say | 0.24 (0.03-2.12) | 0.46 (0.03-7.31) | 0.34 (0.05-2.39) |
| Type of health insurance | |||
| Private | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Medicaid | 0.43 (0.17-1.09) | 0.65 (0.25-1.66) | 0.56 (0.28-1.10) |
| No coverage | 0.08 (0-8.86) | 0.36 (0.03-3.84) | 0.12 (0-4.16) |
| Rather not say/not sure | 0.45 (0.11-1.95) | 1.46 (0.23-9.20) | 1.13 (0.33-3.93) |
| Education | |||
| ≤High school | 0.79 (0.13-4.91) | 1.01 (0.31-3.28) | 1.13 (0.40-3.22) |
| Some college/college graduate | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Advanced degree | 4.77 (1.59-14.27) | 2.13 (0.79-5.72) | 2.97 (1.40-6.30) |
| Rather not say | 0.14 (0.01-2.33) | 0.84 (0.04-19.23) | 0.94 (0.11-8.05) |
| Annual household income, $ | |||
| <75 000 | 0.47 (0.18-1.24) | 0.67 (0.24-1.86) | 0.56 (0.29-1.07) |
| ≥75 000 | 1 [Reference] | 1 [Reference] | 1 [Reference] |
| Rather not say | 0.71 (0.14-3.70) | 1.19 (0.34-4.20) | 0.94 (0.36-2.44) |
Abbreviations: aOR, adjusted odds ratio; EHR, electronic health record.
Vaccine Safety Datalink23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.
Each column represents a separate multivariate regression model; models were adjusted for all variables listed and Vaccine Safety Datalink site.
Data on race and ethnicity were obtained from the EHR.
Other race and ethnicity included multiracial, Hawaiian, and American Indian/Alaska Native women and women with an unspecified other race and ethnicity.
Additional Vaccine-Related Information
Among pregnant women who reported they were vaccinated, 92.6% (95% CI, 86.5%-96.5%) reported receiving a recommendation from a health professional for influenza vaccination; among unvaccinated pregnant women, 83.9% (95% CI, 79.4%-87.8%) reported receiving a recommendation. Among pregnant women who reported they were vaccinated, 85.2% (95% CI, 75.7%-92.0%) reported being offered an influenza vaccine at a health care visit; among unvaccinated pregnant women, 84.4% (95% CI, 80.0%-88.2%) reported being offered a vaccine.
All respondents who self-reported receiving influenza vaccine were asked the location of their vaccination. For women unvaccinated per EHR data who reported vaccination (EHR false negatives), 35.6% (95% CI, 25.9%-46.4%) were vaccinated in a workplace, 24.2% (95% CI, 15.3%-34.9%) in a hospital, and 36.1% (95% CI, 27.4%-45.6%) in a physician’s office, clinic, or health center. For women vaccinated per EHR data who reported vaccination (EHR true positives), 1.1% (95% CI, 0.3%-3.1%) were vaccinated in a workplace, 18.7% (95% CI, 9.5%-31.2%) in a hospital, and 79.4% (95% CI, 67.1%-88.7%) in a physician’s office, clinic, or health center.
Discussion
To achieve high influenza vaccination coverage overall 10 and among groups at high risk from influenza,5-7 accurate and timely data on vaccination coverage are needed to guide public health activities.8,9 However, no single data source can provide a complete picture of coverage. 16 In this study, using self-report as the criterion standard, we found relatively close agreement between influenza vaccine receipt in EHR-based data and self-report, with high sensitivity, specificity, PPV, and κ. The NPV was lower at 80.5%, indicating that 19.5% of women who appeared unvaccinated in EHR data self-reported receiving influenza vaccine during the current season. In addition, using EHR data to characterize survey nonrespondents, we detected substantial risk of nonresponse bias across race and ethnicity and vaccination status, a finding with important implications for survey-based assessments of influenza vaccination coverage.
Aside from an investigation after the 2009 H1N1 influenza outbreak in the United States 43 and a report from Australia, 44 we are unaware of other studies assessing the accuracy of EHR-based influenza vaccination data among pregnant women. Accuracy has been assessed in other US populations, with some studies using self-report as the criterion standard18,22 and others treating EHR 45 or immunization information system46,47 data as the standard. Sy et al 18 compared EHR-based influenza vaccination data with self-report among 50- to 79-year-old members of a large health care organization during the 2007-2008 influenza season: the NPV of EHR data was 79.5%. This value is surprisingly close to the estimate of 80.5% from our study. We anticipated finding a higher NPV among pregnant women in the current study, given the expansion of immunization information systems and the improved capacity of these systems to send immunization data to EHR systems. 48 Workplace vaccination programs may contribute to the lower-than-anticipated NPV; integrating workplace-based influenza vaccination into state immunization information systems may be an important strategy to improve accuracy.
Our findings also highlight the risk of nonresponse bias in surveys of influenza vaccination. Non-Hispanic Black and Hispanic pregnant women and the unvaccinated (per EHR data) were significantly less likely to respond to the survey than non-Hispanic White pregnant women. Non-Hispanic Black people, including those who are pregnant, have consistently lower influenza vaccination rates11,12,15 and express less confidence in influenza vaccine safety and efficacy 49 than non-Hispanic White people; similar findings have also been observed among Hispanic people.12,49 Because our survey was introduced as concerning “the flu (which is also called influenza) and flu vaccination,” unvaccinated individuals and those with low vaccine confidence might have been less willing than vaccinated individuals and those with high vaccine confidence to complete the survey. National surveys, particularly if immunization-focused, may encounter similar behaviors. This type of nonresponse bias could lead to an overestimation of influenza vaccination coverage, and weighting to a referent population may not necessarily correct for this bias.
Limitations
Our study findings had several limitations. First, survey nonresponse may have influenced our results; differential nonresponse to the survey could have negatively or positively affected the validity measures we assessed. Second, while we treated self-report as the criterion standard, self-reported vaccination can be inaccurate, and we could not review outside records (such as from a workplace) to verify respondents’ self-report. Third, misclassification of pregnancy status could have occurred, although nearly all survey respondents confirmed their pregnancy status. Fourth, patients at VSD sites are largely insured (Denver Health also cares for uninsured individuals), and the VSD population may not be generalizable to the overall population of pregnant women in the United States.
Conclusions
The current study can be viewed as part of a longer-term goal of enhancing the nation’s ability to monitor influenza vaccination coverage. The study highlights the strengths of using EHR-derived data for this purpose, including the ability to identify large populations of pregnant women, as well as the limitations, including missing some influenza vaccination occurring outside the medical home. Current EHR-based systems are also not representative of all states and may underrepresent important sociodemographic groups such as the uninsured. Integrating data from multiple sources, including from surveys,5,11-15 EHR-based networks, 19 and immunization information systems, 48 continues to be needed to monitor influenza vaccination coverage.
Acknowledgments
The authors acknowledge Bradley Crane, MS, and Stephanie Irving, MHS, at the Center for Health Research, Kaiser Permanente Northwest, Portland, Oregon, for their contributions to this work.
Footnotes
Authors’ Note: Matthew F. Daley, Liza M. Reifler, and Jo Ann Shoup had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Disclaimer: The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Liza M. Reifler has an immediate family member who owns stock in Merck, Abbott Laboratories, and AstraZeneca. Allison L. Naleway has received research support from Pfizer unrelated to the current study. Michael L. Jackson has received research support from Sanofi Pasteur unrelated to the current study. Huong McLean has received research support from Seqirus unrelated to the current study. Nicola Klein reports research support from GlaxoSmithKline, Sanofi Pasteur, Protein Science (now Sanofi Pasteur), Merck, and Pfizer unrelated to the current study.
Funding: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: this research was funded by the Centers for Disease Control and Prevention, through a Task Order (contract no. 200-2012-53582-21454), issued as part of the Vaccine Safety Datalink project (contract no. 200-2012-53582). Coauthors from the Centers for Disease Control and Prevention were involved in design and conduct of the study, interpretation of data, and review and approval of the article.
ORCID iDs: Matthew F. Daley, MD 
https://orcid.org/0000-0003-1309-4096
Hilda Razzaghi, PhD, MSPH https://orcid.org/0000-0002-8053-9748
References
- 1.Centers for Disease Control and Prevention. Estimates of deaths associated with seasonal influenza—United States, 1976-2007. MMWR Morb Mortal Wkly Rep. 2010;59(33):1057-1062. [PubMed] [Google Scholar]
- 2.Reed C, Chaves SS, Daily Kirley P, et al. Estimating influenza disease burden from population-based surveillance data in the United States. PLoS One. 2015;10(3):e0118369. doi: 10.1371/journal.pone.0118369 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Centers for Disease Control and Prevention. Disease burden of flu. 2021. Accessed March 16, 2021. https://www.cdc.gov/flu/about/burden/index.html
- 4.Rolfes MA, Foppa IM, Garg S, et al. Annual estimates of the burden of seasonal influenza in the United States: a tool for strengthening influenza surveillance and preparedness. Influenza Other Respir Viruses. 2018;12(1):132-137. doi: 10.1111/irv.12486 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Lindley MC, Kahn KE, Bardenheier BH, et al. Vital signs: burden and prevention of influenza and pertussis among pregnant women and infants—United States. MMWR Morb Mortal Wkly Rep. 2019;68(40):885-892. doi: 10.15585/mmwr.mm6840e1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Mertz D, Geraci J, Winkup J, Gessner BD, Ortiz JR, Loeb M.Pregnancy as a risk factor for severe outcomes from influenza virus infection: a systematic review and meta-analysis of observational studies. Vaccine. 2017;35(4):521-528. doi: 10.1016/j.vaccine.2016.12.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Prasad N, Huang QS, Wood T, et al. Influenza-associated outcomes among pregnant, postpartum, and nonpregnant women of reproductive age. J Infect Dis. 2019;219(12):1893-1903. doi: 10.1093/infdis/jiz035 [DOI] [PubMed] [Google Scholar]
- 8.Grohskopf LA, Alyanak E, Broder KR, et al. Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2020-21 influenza season. MMWR Recomm Rep. 2020;69(8):1-24. doi: 10.15585/mmwr.rr6908a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Grohskopf LA, Alyanak E, Broder KR, Walter EB, Fry AM, Jernigan DB.Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2019-20 influenza season. MMWR Recomm Rep. 2019;68(3):1-21. doi: 10.15585/mmwr.rr6803a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Office of Disease Prevention and Health Promotion. Healthy People 2030: increase the proportion of people who get the flu vaccine every year—IID-09. 2021. Accessed March 19, 2021. https://health.gov/healthypeople/objectives-and-data/browse-objectives/vaccination/increase-proportion-people-who-get-flu-vaccine-every-year-iid-09
- 11.Ding H, Kahn KE, Black CL, O’Halloran A, Lu PJ, Williams WW.Influenza vaccination coverage among pregnant women in the U.S., 2012-2015. Am J Prev Med. 2019;56(4):477-486. doi: 10.1016/j.amepre.2018.11.020 [DOI] [PubMed] [Google Scholar]
- 12.Lu PJ, Hung MC, O’Halloran AC, et al. Seasonal influenza vaccination coverage trends among adult populations, U.S., 2010-2016. Am J Prev Med. 2019;57(4):458-469. doi: 10.1016/j.amepre.2019.04.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kennedy ED, Ahluwalia IB, Ding H, Lu PJ, Singleton JA, Bridges CB.Monitoring seasonal influenza vaccination coverage among pregnant women in the United States. Am J Obstet Gynecol. 2012;207(3 suppl):S9-S16. doi: 10.1016/j.ajog.2012.06.069 [DOI] [PubMed] [Google Scholar]
- 14.Kahn KE, Black CL, Ding H, et al. Influenza and Tdap vaccination coverage among pregnant women—United States, April 2018. MMWR Morb Mortal Wkly Rep. 2018;67(38):1055-1059. doi: 10.15585/mmwr.mm6738a3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Razzaghi H, Kahn KE, Black CL, et al. Influenza and Tdap vaccination coverage among pregnant women—United States, April 2020. MMWR Morb Mortal Wkly Rep. 2020;69(39):1391-1397. doi: 10.15585/mmwr.mm6939a2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Lu PJ, Santibanez TA, Williams WW, et al. Surveillance of influenza vaccination coverage—United States, 2007-08 through 2011-12 influenza seasons. MMWR Surveill Summ. 2013;62(4):1-28. [PubMed] [Google Scholar]
- 17.Tourangeau R, Plewes TJ, eds. Nonresponse in Social Science Surveys: A Research Agenda. National Academies Press; 2013. [Google Scholar]
- 18.Sy LS, Liu IL, Solano Z, et al. Accuracy of influenza vaccination status in a computer-based immunization tracking system of a managed care organization. Vaccine. 2010;28(32):5254-5259. doi: 10.1016/j.vaccine.2010.05.061 [DOI] [PubMed] [Google Scholar]
- 19.Groom HC, Henninger ML, Smith N, et al. Influenza vaccination during pregnancy: influenza seasons 2002-2012, Vaccine Safety Datalink. Am J Prev Med. 2016;50(4):480-488. doi: 10.1016/j.amepre.2015.08.017 [DOI] [PubMed] [Google Scholar]
- 20.Hong K, Lindley MC, Zhou F.Coverage and timing of influenza vaccination among privately insured pregnant women in the United States, 2010-2018. Public Health Rep. Published June 23, 2021. doi: 10.1177/00333549211026779 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Naleway AL, Gold R, Kurosky S, et al. Identifying pregnancy episodes, outcomes, and mother–infant pairs in the Vaccine Safety Datalink. Vaccine. 2013;31(27):2898-2903. doi: 10.1016/j.vaccine.2013.03.069 [DOI] [PubMed] [Google Scholar]
- 22.Greene SK, Shi P, Dutta-Linn MM, et al. Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med. 2009;37(6):552-555. doi: 10.1016/j.amepre.2009.08.022 [DOI] [PubMed] [Google Scholar]
- 23.Baggs J, Gee J, Lewis E, et al. The Vaccine Safety Datalink: a model for monitoring immunization safety. Pediatrics. 2011;127(suppl 1):S45-S53. doi: 10.1542/peds.2010-1722H [DOI] [PubMed] [Google Scholar]
- 24.McNeil MM, Gee J, Weintraub ES, et al. The Vaccine Safety Datalink: successes and challenges monitoring vaccine safety. Vaccine. 2014;32(42):5390-5398. doi: 10.1016/j.vaccine.2014.07.073 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Sukumaran L, McCarthy NL, Li R, et al. Demographic characteristics of members of the Vaccine Safety Datalink (VSD): a comparison with the United States population. Vaccine. 2015;33(36):4446-4450. doi: 10.1016/j.vaccine.2015.07.037 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Gabow P, Eisert S, Wright R.Denver Health: a model for the integration of a public hospital and community health centers. Ann Intern Med. 2003;138(2):143-149. doi: 10.7326/0003-4819-138-2-200301210-00016 [DOI] [PubMed] [Google Scholar]
- 27.Eisert SL, Durfee MJ, Welsh A, Moore SL, Mehler PS, Gabow PA.Changes in insurance status and access to care in an integrated safety net healthcare system. J Community Health. 2009;34(2):122-128. doi: 10.1007/s10900-008-9136-2 [DOI] [PubMed] [Google Scholar]
- 28.Mullooly J, Drew L, DeStefano F, et al. Quality of HMO vaccination databases used to monitor childhood vaccine safety. Vaccine Safety Datalink Team. Am J Epidemiol. 1999;149(2):186-194. doi: 10.1093/oxfordjournals.aje.a009785 [DOI] [PubMed] [Google Scholar]
- 29.McCarthy NL, Gee J, Weintraub E, et al. Monitoring vaccine safety using the Vaccine Safety Datalink: utilizing immunization registries for pandemic influenza. Vaccine. 2011;29(31):4891-4896. doi: 10.1016/j.vaccine.2011.05.003 [DOI] [PubMed] [Google Scholar]
- 30.Naleway AL, Crane B, Irving SA, et al. Vaccine Safety Datalink infrastructure enhancements for evaluating the safety of maternal vaccination. Ther Adv Drug Saf. 2021;12:20420986211021233. doi: 10.1177/20420986211021233 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.American Medical Association. International Classification of Diseases, 10th Revision, Clinical Modification. American Medical Association Press; 2018. [Google Scholar]
- 32.Robbins C, Boulet SL, Morgan I, et al. Disparities in preconception health indicators—Behavioral Risk Factor Surveillance System, 2013-2015, and Pregnancy Risk Assessment Monitoring System, 2013-2014. MMWR Surveill Summ. 2018;67(1):1-16. doi: 10.15585/mmwr.ss6701a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Hill HA, Elam-Evans LD, Yankey D, Singleton JA, Dietz V.Vaccination coverage among children aged 19-35 months—United States, 2015. MMWR Morb Mortal Wkly Rep. 2016;65(39):1065-1071. doi: 10.15585/mmwr.mm6539a4 [DOI] [PubMed] [Google Scholar]
- 34.Hill HA, Elam-Evans LD, Yankey D, Singleton JA, Kang Y.Vaccination coverage among children aged 19-35 months—United States, 2016. MMWR Morb Mortal Wkly Rep. 2017;66(43):1171-1177. doi: 10.15585/mmwr.mm6643a3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Parsons VL, Moriarity C, Jonas K, Moore TF, Davis KE, Tompkins L.Design and estimation for the National Health Interview Survey, 2006-2015. Vital Health Stat 2. 2014;(165):1-53. [PubMed] [Google Scholar]
- 36.Williams WW, Lu PJ, O’Halloran A, et al. Surveillance of vaccination coverage among adult populations—United States, 2015. MMWR Surveill Summ. 2017;66(11):1-28. doi: 10.15585/mmwr.ss6611a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Centers for Disease Control and Prevention. National early-season flu vaccination coverage, United States, November 2017. 2017. Accessed May 17, 2018. https://www.cdc.gov/flu/fluvaxview/nifs-estimates-nov2017.htm
- 38.Centers for Disease Control and Prevention. National early-season flu vaccination coverage, United States, November 2016. 2016. Accessed May 17, 2018. https://www.cdc.gov/flu/fluvaxview/nifs-estimates-nov2016.htm
- 39.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. doi: 10.1016/j.jbi.2019.103208 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Korn EL, Graubard BI.Confidence intervals for proportions with small expected number of positive counts estimated from survey data. Surv Methodol. 1998;24:193-201. [Google Scholar]
- 41.Yu M, Li Y, Qiu M.Statistical inference of the relative concentration index for complex surveys. Stat Med. 2019;38(21):4083-4095. doi: 10.1002/sim.8284 [DOI] [PubMed] [Google Scholar]
- 42.Grohskopf LA, Sokolow LZ, Broder KR, Walter EB, Fry AM, Jernigan DB.Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2018-19 influenza season. MMWR Recomm Rep. 2018;67(3):1-20. doi: 10.15585/mmwr.rr6703a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Specker B, Wey B, Fuller JE, et al. 2009 H1N1 and seasonal influenza immunization among pregnant women: a comparison of different sources of immunization information. Matern Child Health J. 2014;18(3):681-687. doi: 10.1007/s10995-013-1293-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Regan AK, Mak DB, Moore HC, et al. Surveillance of antenatal influenza vaccination: validity of current systems and recommendations for improvement. BMC Public Health. 2015;15:1155. doi: 10.1186/s12889-015-2234-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Rolnick SJ, Parker ED, Nordin JD, et al. Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine. 2013;31(37):3928-3935. doi: 10.1016/j.vaccine.2013.06.041 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Irving SA, Donahue JG, Shay DK, Ellis-Coyle TL, Belongia EA.Evaluation of self-reported and registry-based influenza vaccination status in a Wisconsin cohort. Vaccine. 2009;27(47):6546-6549. doi: 10.1016/j.vaccine.2009.08.050 [DOI] [PubMed] [Google Scholar]
- 47.King JP, McLean HQ, Belongia EA.Validation of self-reported influenza vaccination in the current and prior season. Influenza Other Respir Viruses. 2018;12(6):808-813. doi: 10.1111/irv.12593 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.American Immunization Registry Association. Landscape analysis: IIS adult vaccination data capture and data utilization. May2021. Accessed July 19, 2021. https://repository.immregistries.org/files/resources/60958446cdad0/adultlandscape_final.pdf
- 49.Dudley MZ, Limaye RJ, Salmon DA, et al. Racial/ethnic disparities in maternal vaccine knowledge, attitudes, and intentions. Public Health Rep. 2021;136(6):699-709. doi: 10.1177/0033354920974660 [DOI] [PMC free article] [PubMed] [Google Scholar]