A Risk Prediction Model to Identify Newborns at Risk for Missing Early Childhood Vaccinations

Abstract

Background

Approximately 30% of US children aged 24 months have not received all recommended vaccines. This study aimed to develop a prediction model to identify newborns at high risk for missing early childhood vaccines.

Methods

A retrospective cohort included 9080 infants born weighing ≥2000 g at an academic medical center between 2008 and 2013. Electronic medical record data were linked to vaccine data from the Washington State Immunization Information System. Risk models were constructed using derivation and validation samples. K-fold cross-validation identified risk factors for model inclusion based on alpha = 0.01. For each patient in the derivation set, the total number of weighted adverse risk factors was calculated and used to establish groups at low, medium, or high risk for undervaccination. Logistic regression evaluated the likelihood of not completing the 7-vaccine series by age 19 months. The final model was tested using the validation sample.

Results

Overall, 53.6% failed to complete the 7-vaccine series by 19 months. Six risk factors were identified: race/ethnicity, maternal language, insurance status, birth hospitalization length of stay, medical service, and hepatitis B vaccine receipt. Likelihood of non-completion was greater in the high (77.1%; adjusted odds ratio [AOR] 5.6; 99% confidence interval [CI]: 4.2, 7.4) and medium (52.7%; AOR 1.9; 99% CI: 1.6, 2.2) vs low (38.7%) risk groups in the derivation sample. Similar results were observed in the validation sample.

Conclusions

Our prediction model using information readily available in birth hospitalization records consistently identified newborns at high risk for undervaccination. Early identification of high-risk families could be useful for initiating timely, tailored vaccine interventions.

We developed a prediction model using demographic and birth hospitalization information from electronic medical records that consistently identified newborns at risk for missing future vaccines. Prediction models may aid providers as they initiate early, tailored vaccine interventions.

INTRODUCTION

Childhood vaccination is the most effective way to prevent many infectious diseases at individual and population levels [1]. The rise of vaccine hesitancy, coupled with vaccine-preventable disease outbreaks [2], and the precipitous decline in childhood vaccine administrations during the COVID-19 pandemic [3] highlight the continued importance of promoting and increasing childhood vaccination coverage.

Given the wide variation in vaccine uptake by sociodemographic and clinical characteristics [4–7], it can be difficult for providers to easily and accurately identify individual infants at risk for missing future childhood vaccinations whose parents may benefit from tailored interventions [8–10]. Risk prediction models using patient characteristics to identify those at the highest risk for adverse outcomes and behaviors are increasingly used to complement clinical decision-making [11, 12]. Two previous risk prediction models for childhood vaccinations have been reported: one to predict vaccination coverage among preschool-age emergency department patients [13] and another to predict receipt of select childhood vaccines among patients at urban, safety net community health centers [14]. The target populations, vaccinations, and utility of these risk prediction models were limited.

A population-based risk prediction model using readily available information to identify patients at risk for missing early childhood vaccines is needed. Ultimately, such a model could be further developed into a clinical support tool to prompt early initiation of vaccination interventions. During the COVID-19 pandemic and in its aftermath, it is critical to capitalize on every opportunity to address vaccine hesitancy and on-time vaccination.

Our objective was to develop a clinically relevant statistical model using sociodemographic, clinical, and birth hospitalization characteristics to predict the risk of an infant failing to complete the recommended 7-vaccine series [15] by age 19 months.

METHODS

Study Sample and Setting

The study sample and setting have been described previously [16, 17]. In brief, all Washington State infants with a birth weight ≥2000 g who received birth hospitalization care between January 1, 2008, and December 31, 2013, at the University of Washington Medical Center (UWMC), a large academic medical center in Seattle, Washington, were included. Infants born weighing <2000 g were excluded because recommended hepatitis B (HepB) vaccination schedules differ for these infants [15]. Infants without complete admission and discharge data and those who transferred to UWMC after birth were also excluded (Supplementary Figure). The study was approved by Seattle Children’s Hospital and Washington State Institutional Review Boards.

Data Sources

Sociodemographic, birth hospitalization, and clinical data, including vaccine doses given during the birth hospitalization, were retrospectively collected from electronic medical records (EMR). These EMR data were linked using select identifiers and a standardized matching algorithm to vaccine records obtained from the Washington State Immunization Information System (WAIIS) in order to capture vaccinations through age 18 months in other clinical settings. Infants with <2 documented doses in WAIIS or an inactive WAIIS record (ie, due to death, moved out of state) were excluded, consistent with national and state reporting standards [18]. Providers contribute data to WAIIS by manual entry, batch files, or direct Health Level Seven linkage [19]. Although provider reporting is not mandated, WAIIS is highly complete and has both internal and external validity [20]. The Centers for Disease Control and Prevention estimates that ≥95% of Washington State children less than 6 years of age participated in WAIIS during the study period (2008-2015) [21].

Measures

The primary outcome for the prediction model was failure to receive the recommended 7-vaccine series, comprised of 4 diphtheria-tetanus-(whole-cell or acellular) pertussis (DTP/DTaP), 3 poliovirus, 1 measles-mumps-rubella, 3 Haemophilus influenzae type b, 3 HepB, 1 varicella, and 4 pneumococcal conjugate vaccine doses, by age 19 months [15]. The 19-month cutoff was selected a priori to reflect timely 7-vaccine series completion in accordance with ACIP recommendations [15]. This age cutoff has been used previously to define timely receipt of recommended vaccines, beyond which an infant is considered delayed for vaccination [22].

Independent Variables

Maternal and Infant Sociodemographic Characteristics

Sociodemographic data included infant sex (male, female), maternal language (English, Spanish, other), insurance status (public, private), and race/ethnicity. Race/ethnicity was recorded at the point of care by hospital staff and categorized using US Census Bureau classifications [23] and collapsed into Hispanic, non-Hispanic white, non-Hispanic black, Asian, or multi-racial/other. Urban vs rural residency was measured using Rural Urban Commuting Area (RUCA) codes to categorize geographic areas as primarily rural or urban based on census tract and commuting data [24]. Area-level income was measured for each patient based on the median household income in his/her ZIP code using 2010 Census Bureau data [25, 26]. ZIP-level median household income was stratified into equal income quartiles.

Clinical and Birth Hospitalization Visit Characteristics

HepB vaccination during the birth hospitalization, as recommended by the Advisory Committee on Immunization Practices (ACIP) during the study period, was assessed. Gestational age was categorized as <37 or ≥37 weeks. Birth hospitalization length of stay was calculated as the number of hours between admission and discharge and categorized as <24, ≥24 and <48, ≥48 and <96, or ≥96 hours. Medical service during the birth hospitalization included newborn nursery, intermediate care nursery, or neonatal intensive care unit (NICU).

Statistical Analysis

An adverse risk model was derived based on a random sample of two-thirds of the study population. Within the derivation sample, we used a k-fold cross-validation approach with k = 5 folds, repeated 5 times, to iteratively create test and validation samples. Among each set of k-1 folds, backward regression identified candidate variables for the adverse risk model. All available variables (described above) were included. No assumptions were made as to the magnitude or direction of the associations with non-completion of the 7-vaccine series; as such, a two-sided alpha = 0.01 level was used given the large sample size. The predictive capacity of the candidate variables from each regression was tested in the kth fold using area under the receiver operating characteristic (ROC) curve (AUC), a common metric to evaluate model performance in this setting [27, 28]. This process was repeated 5 times, creating 25 models and their associated error values. A best set of predictor variables was defined as those comprising the top 5 models with the highest AUC. For each variable from this set, the adverse category (ie, the category associated with a higher likelihood of 7-vaccine series non-completion) was identified. Regression coefficients for each retained predictor variable in these top 5 models were compared. To enhance the predictive ability of the model, weighting was assigned based on the average regression coefficients for each retained predictor variable. For each patient in the entire derivation set, the total number of weighted adverse risk factors was calculated. The observed distribution of the number of adverse risk factors was used to establish groups at low, medium, or high risk for non-completion of the 7-vaccine series. Given that HepB vaccine served as both an independent variable (HepB birth dose) and dependent variable (as part of the 7-vaccine series), we performed a sensitivity analysis to reevaluate the risk model using a 6-vaccine series (ie, excluding HepB vaccine) [29].

All risk models were evaluated using logistic regression. Low risk was used as the reference category for non-completion of the 7-vaccine series in the derivation set and then tested in the remaining one-third of patients (ie, independent validation sample). Models were first unadjusted and then adjusted by unretained predictors, as the retained variables were embedded in the risk score. Sensitivity and specificity were calculated to compare predicted vs true failure to complete the 7-vaccine series by age 19 months. P-values for all models were based on 2-tailed tests and considered significant at P < .01 unless otherwise noted. Stata 14.0 (Stata Corp. 2015, College Station, TX) was used for all analyses.

RESULTS

Cohort Characteristics

We included a total of 9080 infants weighing ≥2000 g who received birth hospitalization care at the study hospital between 2008 and 2013. The majority were non-Hispanic white, publicly insured, had an English-speaking mother, resided in urban areas, and were born at term gestation (Table 1). Two-thirds of the study population (n = 6053) was randomly assigned to the derivation sample and the remaining one-third (n = 3027) to the validation sample. A lower proportion of infants in the validation vs derivation sample were born at <37 vs ≥37 weeks gestation (P = .01). The other characteristics were similar between the 2 samples. Non-completion of the 7-vaccine series by age 19 months was 46.8% in the derivation sample and 45.7% in the validation sample (P = .85). Compared with infants in the final study sample, those excluded due to <2 WAIIS vaccine doses (n = 458) were more likely to be male, non-Hispanic white, privately insured, have an English-speaking mother, live in a high-income ZIP code, and have a shorter birth hospitalization length of stay (P < .01 for all comparisons).

Table 1.

Demographic, Clinical, and Birth Hospitalization Characteristics of Patients in the Derivation and Validation Cohorts, 2008-2013

Characteristica	Derivation Cohort N = 6053 (%)	Validation Cohort N = 3027 (%)	P-value
Infant sex			.85
Male	3095 (51.1)	1554 (51.3)
Female	2958 (48.9)	1473 (48.7)
Race/ethnicity			.95
Hispanic	767 (14.4)	403 (15.0)
Non-Hispanic white	2604 (48.8)	1297 (48.4)
Non-Hispanic black	1165 (21.8)	594 (22.2)
Asian	723 (13.6)	358 (13.4)
Multi-racial/other	75 (1.4)	28 (1.0)
Maternal language			.42
English	4307 (75.9)	2179 (76.3)
Spanish	557 (9.8)	296 (10.4)
Other	809 (14.3)	381 (13.3)
Insurance status			.37
Private	2477 (43.8)	1270 (44.8)
Public	3175 (56.2)	1562 (55.2)
Rural-urban residence			.89
Rural	157 (2.6)	80 (2.6)
Urban	5894 (97.4)	2946 (97.4)
Estimated household incomeb			.16
$20 135-42 799	262 (4.4)	103 (3.4)
$42 800-50 844	734 (12.2)	375 (12.5)
$50 845-62 239	2392 (39.7)	1182 (39.3)
$62 240-174 729	2630 (43.7)	1350 (44.9)
Gestational age, wks			.01
<37	692 (11.4)	292 (9.7)
≥37	5359 (88.6)	2734 (90.4)
Birth hospitalization length of stay, h			.81
<24	546 (9.0)	274 (9.0)
≥24 to <48	2635 (43.5)	1321 (43.6)
≥48 to <96	2192 (36.2)	1104 (36.5)
≥96	680 (11.2)	328 (10.8)
Medical service			.07
Newborn nursery	4802 (79.3)	2464 (81.4)
Intermediate care	919 (15.2)	411 (13.6)
Neonatal ICU	332 (5.5)	152 (5.0)
Received HepB vaccinec	4584 (75.7)	2274 (75.1)	.53
Risk categoryd			.82
Low	3543 (58.5)	1792 (59.2)
Medium	1948 (32.2)	955 (31.6)
High	562 (9.3)	280 (9.3)

Abbreviations: HepB, hepatitis B; ICU, intensive care unit.

^aAll proportions shown in the table are based on known data in total study sample (n = 9080). Number and percentage of total missing data are as follows: race/ethnicity = 1066 (11.7%), maternal language = 551 (6.1%), insurance status = 596 (6.6%), rural-urban residence = 3 (0.03%), income estimate = 52 (0.6%), and gestational age = 3 (0.03%).

^bBased on 2010 US Census Bureau ZIP-code level median household income and stratified into equal income quartiles.

^cReceived the HepB vaccine before birth hospitalization discharge.

^dLow risk category (0-2 risk score), medium (3-4 risk score), and high (5-7 risk score).

Predictors of 7-Vaccine Series Non-Completion

Based on selection techniques, 4 candidate predictors (infant sex, income estimate, urban-rural residence, and gestational age) were removed from the final trimmed models, and 6 variables were included in the final model: race/ethnicity, maternal language, insurance status, birth hospitalization length of stay, birth hospitalization medical service, and HepB vaccine receipt during the birth hospitalization. The composite AUC for the top 5 models was 0.67.

Within the derivation sample, the strongest predictor for 7-vaccine series non-completion by age 19 months was not receiving the HepB vaccine during the birth hospitalization (adjusted odds ratio [AOR] 2.9; 99% confidence interval [CI]: 2.4, 3.5) (data not shown). Other risk factors associated with series non-completion included non-Hispanic white race/ethnicity, maternal language of English, public insurance, birth hospitalization stay <24 hours, and NICU medical service during the birth hospitalization.

Using the 6 retained variables in the model, each infant’s predictors were summed and a risk score was assigned to each participant. HepB birth dose receipt contributed approximately twice the predictive value of 7-vaccine series completion compared with other retained variables and, thus, was assigned twice the weight of other variables in the model. The observed distribution of adverse risk factors was used to identify 3 risk groups: low (0-2 risk score), medium (3-4 risk score), and high (5-7 risk score) risk for 7-vaccine series non-completion. Most infants were in the low (58.5%) or medium (32.2%) risk groups, with the remainder (9.3%) in the high-risk group (Table 1). Higher scores indicated an increased risk for 7-vaccine series non-completion by age 19 months. For example, a privately insured, Hispanic infant with a Spanish-speaking mother who received a timely HepB birth dose and was discharged from the newborn nursery service ≥24 hours would be considered low risk (see scoring scheme in Table 2). In contrast, a non-Hispanic white infant with an English-speaking mother, a missed HepB birth dose, and a birth hospitalization stay <24 hours would be considered at high risk for not completing the 7-vaccine series by age 19 months.

Table 2.

Risk Calculator for Failure to Complete the 7-Vaccine Series^a by Age 19 Months

Infant Characteristic		Points
HepB vaccinationb	No	2
	Yes	0
Race/ethnicity	Non-Hispanic (NH) white	1
	Hispanic, Asian, NH black, or multi-racial/other	0
Maternal language	English	1
	Spanish or other language	0
Insurance type	Public	1
	Private	0
Hospital serviceb	NICU	1
	Newborn or intermediate care	0
Length of stayb	<24 h	1
	≥24 h	0

Abbreviations: HepB, hepatitis B; NICU, neonatal intensive care unit.

^aFour doses of diphtheria-tetanus-(whole-cell or acellular) pertussis (DTP/DTaP), 3 poliovirus, 1 measles-mumps-rubella, 3 Haemophilus influenzae type b, 3 HepB, 1 varicella, and 4 pneumococcal conjugate vaccines.

^bDuring birth hospitalization.

Table 3 presents the relationship between the risk group and 7-vaccine series non-completion. Within the derivation cohort, 38.7% of infants in the low-risk category failed to complete the series by 19 months compared with 52.7% and 77.1% of those in the medium- and high-risk categories, respectively. Similar results were seen in the validation sample.

Table 3.

Proportion of Derivation and Validation Cohorts Who Did Not Complete7-Vaccine Series by Age 19 Months by Risk Category

Risk Categorya	Derivation Cohort (N = 6053)		Validation Cohort (N = 3027)
	n	%	n	%
Low	1371/3543	38.7	664/1792	37.1
Medium	1027/1948	52.7	504/955	52.8
High	433/562	77.1	214/280	76.4

^aLow (0-2 risk score), medium (3-4 risk score), and high (5-7 risk score).

Likelihood of 7-Vaccine Series Non-Completion

Within the derivation cohort, the adjusted odds of 7-vaccine series non-completion were much higher among infants in the high (AOR 5.6; 99% CI: 4.2, 7.4) and medium (AOR 1.9; 99% CI: 1.6, 2.2) vs low-risk category (Table 4). Similar results were seen in the validation sample. The analysis assessing 6-vaccine series completion (ie, excluding HepB vaccine) showed similar results when comparing infants categorized as high (AOR 4.2; 99% CI: 3.6, 5.0) and medium (AOR 1.6; 99% CI: 1.5, 1.8) vs low-risk (data not shown). Model sensitivity and specificity were 67.8% and 51.7%, respectively.

Table 4.

Odds Ratios (OR) and 99% Confidence Interval (CI) of 7-Vaccine Series Non-Completion by Age 19 Months in the Derivation and Validation Cohorts

	Derivation Cohort (N = 6053) OR (99% CI)		Validation Cohort (N = 3027) OR (99% CI)
Risk levela	Unadjusted	Adjusted	Unadjusted	Adjusted
Low	Ref	Ref	Ref	Ref
Medium	1.8 (1.5, 2.0)	1.9 (1.6, 2.2)	1.9 (1.5, 2.3)	1.9 (1.5, 2.4)
High	5.3 (4.0, 6.9)	5.6 (4.2, 7.4)	5.5 (3.8, 8.1)	5.5 (3.7, 8.2)

^aLow (0-2 risk score), medium (3-4 risk score), and high (5-7 risk score).

Discussion

In a large, diverse cohort from Washington State, we established that a multi-dimensional risk prediction model can delineate groups of infants with widely varying risk of failure to complete the recommended 7-vaccine series by age 19 months. The risk model was restricted to a small number of explicitly defined variables that can be reliably measured and easily obtained from the EMR during an infant’s birth hospitalization. Moreover, factors above and beyond receipt of the initial HepB birth dose contributed meaningful independent information about risk of future undervaccination. Considering that the 7-vaccine series is universally recommended, yet approximately 30% of US children aged 24 months have not received the full series [7], a risk model that identifies a subset of infants with 5-fold greater odds of non-completion may be highly beneficial to providers and practices as they initiate timely, tailored vaccine interventions.

Our risk prediction model builds upon prior models [13, 14] to accurately and efficiently identify newborns early in the vaccination process who may benefit most from multimodal interventions. Failure to receive the HepB birth dose during hospitalization was the strongest individual predictor of 7-vaccine series non-completion, consistent with previous studies [17, 30]. This could reflect parental perception of low infection risk, limited understanding of disease severity, vaccine safety concerns during the newborn period, or general vaccine hesitancy [31]. Importantly, our model extends beyond this single risk factor to include other key sociodemographic, clinical, and birth hospitalization factors that collectively offer a broad perspective regarding a child’s risk for undervaccination. Public insurance has been associated with undervaccination, and disparities in access to high-quality health care likely contribute [6]. Infants cared for in the NICU setting often exhibit higher medical acuity, complex healthcare needs, and competing visit priorities, which could hinder vaccine uptake [32]. In addition, English-speaking, non-Hispanic white families have been shown to exhibit higher vaccine hesitancy [5, 6].

Using our risk prediction model, we envision that an overall risk score could be calculated for each infant during the birth hospitalization and communicated to the outpatient care team (ie, via the EMR or discharge paperwork). Outpatient practices and providers could then use this information to tailor early visits with and support of families identified as high risk for undervaccination in conjunction with other interventions known to increase childhood vaccination. They could consider a range of evidence-based interventions to meet the individual needs of each family. For example, strategies such as transportation assistance, reminder-recall initiatives, or case management could be employed to ensure equitable access to preventive care services, including necessary vaccinations [10, 33]. The risk prediction tool could also alert practice staff to schedule additional or longer visits with high-risk families. This may prompt providers to address barriers to vaccination and initiate vaccine conversations before they might otherwise do so. Increased time for repeated conversations may be particularly useful since over half of physicians spend 10–19 minutes and 8% spend ≥20 minutes discussing vaccines with parents who have significant concerns [34].

Formal screening could also be coupled with tools (ie, via clinical decision support in the EMR) [10, 35] to promote the use of evidence-based provider communication techniques. Studies demonstrate the effectiveness of providers offering a strong vaccine recommendation, using a presumptive approach to initiate the recommendation, employing motivational interviewing, and tailoring communication based upon a family’s unique concerns and readiness to vaccinate [9, 10, 36–40]. These techniques are universally applicable but may be particularly useful for vaccine-hesitant families, including those likely captured using the present model. In one study, infants of vaccine-hesitant parents had fewer days of undervaccination when a presumptive recommendation was used in multiple visits over time [38]. In another study, vaccine-hesitant parents recalled that provider motivational interviewing specifically convinced them to vaccinate their child [36]. Providers report that these approaches are easy to use, save time, and increase self-efficacy in discussing vaccines with vaccine-hesitant parents [39].

Limitations

This study has several limitations. First, use of prospective data and external validation are considered the most stringent prediction model designs [41]. Our model used retrospective data and was internally validated. These limitations may be partially compensated for by our use of a large patient sample, an iterative cross-validation technique, and inclusion of a limited number of predictors, all of which may enhance the quality of prediction models [27, 41]. Second, the age of our dataset (2008-2013 birth cohort) may be a limitation, although early childhood vaccination coverage has been relatively stable in recent years (aside from notable declines during the COVID-19 pandemic) [7]. Third, race/ethnicity data abstracted from the EMR were both self-reported and ascribed by hospital staff, potentially diminishing their accuracy. Similarly, our area-level estimates of median household income lacked patient-level specificity, which may have limited our assessment of socioeconomic and geographic factors as determinants of vaccination. However, we used a standardized approach to measuring household income based upon Census Bureau data [26], lending support to its inclusion in our analyses. Fourth, the moderate predictive power of the current model suggests that major determinants of missed vaccinations remain to be studied. Ideally, other important predictors of infant vaccination, such as parental attitudes about childhood vaccination, would also be included [4]. However, it may be impractical to routinely measure or obtain these data for all patients. It is worth noting that Washington State has one of the highest nonmedical exemption rates in the United States [42]; thus, our results may not be generalizable to settings with a less vaccine-hesitant population. Moreover, although underreporting of vaccine administrations to WAIIS could explain why some infants had less than 2 documented vaccine doses in WAIIS, their sociodemographic characteristics (ie, high-income, non-Hispanic white) were consistent with those of parents who delay or decline early childhood vaccines [5, 6, 16, 17]. Our model may be strengthened by including these patients given their similarity to the highest risk group. We expect this effect to be minimal, however, given the small proportion of infants excluded for this reason. Finally, the study population did not include infants who were born out of hospital, but future inclusion of these families could potentially strengthen our prediction model given that they are more likely to refuse standard newborn preventive care including HepB vaccine [31].

Conclusions

This study demonstrates that a prediction model using readily available infant sociodemographic, clinical, and birth hospitalization data may help identify newborns at risk for undervaccination by age 19 months. Future prospective studies are needed to confirm whether this model can be developed into an enhanced childhood vaccination prediction tool that is clinically useful in a variety of practice settings. Since parental vaccine decision-making often begins prenatally, a similar tool could be adapted to pregnancy for earlier identification. Novel methods for identifying and counseling high-risk families may be particularly timely given growing vaccine hesitancy, resulting vaccine-preventable disease outbreaks, and declining childhood vaccinations during the COVID-19 pandemic.

Notes

Acknowledgments. The authors gratefully acknowledge Steve Senter and Nicholas Dobbins at the University of Washington Institute of Translational Health Sciences and colleagues at the Washington State Immunization Information System for their assistance on this study.

Financial support. This work was supported in part by the National Center for Advancing Translational Sciences (Award UL1 TR002319) and the Mentored Clinical Scientist Research Career Development Award from the Agency for Healthcare Research and Quality (AH K08HS025470).

Potential conflicts of interest. J. A. E. receives research support from Chimerix, AstraZeneca, Novavax, and GlaxoSmithKline. J. A. E. was a consultant for Meissa Vaccines and Sanofi Pasteur.

The other authors have no other potential conflicts of interest or financial relationships relevant to this article to disclose. All authors have submitted the ICMJE Form for Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.