Development and Validation of a Clinical Prediction Tool for Seasonal Influenza Vaccination in England

Matthew M Loiacono; Nicholas Mitsakakis; Jeffrey C Kwong; Gabriela B Gomez; Ayman Chit; Paul Grootendorst

doi:10.1001/jamanetworkopen.2020.7743

. 2020 Jun 29;3(6):e207743. doi: 10.1001/jamanetworkopen.2020.7743

Development and Validation of a Clinical Prediction Tool for Seasonal Influenza Vaccination in England

Matthew M Loiacono ^1,^2,^✉, Nicholas Mitsakakis ³, Jeffrey C Kwong ^3,^4,^5,^6,⁷, Gabriela B Gomez ^8,⁹, Ayman Chit ^1,², Paul Grootendorst ²

¹Vaccine Epidemiology and Modeling, Sanofi Pasteur, Swiftwater, Pennsylvania

²Leslie Dan Faculty of Pharmacy, University of Toronto, Ontario, Canada

³Dalla Lana School of Public Health, University of Toronto, Ontario, Canada

⁴University Health Network, Toronto, Ontario, Canada

⁵ICES, Toronto, Ontario, Canada

⁶Public Health Ontario, Toronto, Ontario, Canada

⁷Department of Family and Community Medicine, University of Toronto, Ontario, Canada

⁸Vaccine Epidemiology and Modeling, Sanofi Pasteur, Lyon, France

⁹London School of Hygiene and Tropical Medicine, Department of Global Health and Development, London, United Kingdom

Accepted for Publication: April 7, 2020.

Published: June 29, 2020. doi:10.1001/jamanetworkopen.2020.7743

^✉

Corresponding Author: Matthew M. Loiacono, MSc, Vaccine Epidemiology and Modeling, Sanofi Pasteur, 1 Discovery Dr, ATTN: Matthew M. Loiacono, B60 - 360, Swiftwater, PA 18370 (matthew.loiacono@sanofi.com).

Author Contributions: Mr Loiacono had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Loiacono, Mitsakakis, Chit, Grootendorst.

Acquisition, analysis, or interpretation of data: Loiacono, Mitsakakis, Kwong, Gomez, Chit.

Drafting of the manuscript: Loiacono, Gomez, Grootendorst.

Critical revision of the manuscript for important intellectual content: Loiacono, Mitsakakis, Kwong, Gomez, Chit, Grootendorst.

Statistical analysis: Loiacono.

Obtained funding: Chit.

Administrative, technical, or material support: Gomez, Chit.

Supervision: Mitsakakis, Chit, Grootendorst.

Conflict of Interest Disclosures: Mr Loiacono and Drs Gomez and Chit reported being full-time employees of Sanofi Pasteur. Mr Loiacono reported receiving nonfinancial support from Sanofi Pasteur during the conduct of the study. Dr Grootendorst reported receiving grants from Sanofi Pasteur Canada in the form of financial support for an academic conference he organized and grants from Sanofi US to provide stipends for a PhD student (Mr Loiacono) under his supervision outside the submitted work. No other disclosures were reported.

Funding/Support: This study was funded by Sanofi Pasteur.

Role of the Funder/Sponsor: The funder reviewed and approved the design of the study; enabled collection of the data via a multistudy CPRD data license; and reviewed and approved the manuscript. The funder had no role in conduct of the study; management, analysis, and interpretation of the data; preparation of the manuscript, or decision to submit the manuscript for publication.

Additional Contributions: Daniel Gibbons, MPhil, from UK Behavioral Insights team, provided advice regarding the clinical utility of the prediction model in a practice setting as well as the methodology to construct the model. He received no compensation for his contribution.

^✉

Corresponding author.

PMCID: PMC7324952 PMID: 32597991

Key Points

Question

Can a prediction tool using patient characteristics measured via routinely collected primary care accurately estimate a patient’s probability of receiving the seasonal influenza vaccine during an upcoming influenza season?

Findings

In this prognostic study that used retrospectively collected primary care data of 324 284 patients from 2011 to 2016 to train and validate logistic regression models, patient-level seasonal influenza vaccine uptake among at-risk adults in England was predicted with misclassification rates ranging from 0.12-0.16, sensitivity ranging from 0.71-0.94, and specificity ranging from 0.68-0.90.

Meaning

The results of this study suggested that it may be possible to predict patient-level seasonal influenza vaccine uptake solely based upon patient characteristics identified via primary care records.

This prognostic study assesses the development and validation of a clinical prediction tool to estimate the uptake of seasonal influenza vaccine among at-risk adults in England from primary care data.

Abstract

Importance

Timely identification of patients likely to miss seasonal influenza vaccination (SIV) could help health care practitioners tailor services and gain efficiency.

Objective

To develop and validate a predictive model of SIV uptake among at-risk adults.

Design, Setting, and Participants

This prognostic study constructed a prediction model for vaccine uptake by adults at increased risk of influenza-associated complications. Drawing from the Clinical Practice Research Datalink database’s records of primary care data of 324 284 adults routinely collected at general practices across England from January 2011 to December 2016, logistic regression models were trained on data from patients registered from January 2012 to December 2013 and validated with out-of-sample data from patients registered from January 2015 to December 2016. Data were extracted from the database December 2018 and analyzed between September 2019 and December 2019.

Exposures

Covariates included sex, age, race/ethnicity, smoking status, socioeconomic status, previous pneumococcal vaccination, prior season SIV uptake, and clinical risk conditions.

Main Outcomes and Measures

The main outcome was patient-level SIV uptake. Model performance was measured via misclassification rate, Brier score, sensitivity, specificity, and area under the curve.

Results

The training data sets consisted of 324 284 (aged 18 to 64 years) and 186 426 (aged 65 years or older) patients. The mean (SD) age in the training data among patients aged 18 to 64 years was 45 (13) years; 161 487 (49.8%) were women, and 102 133 (31.5%) were categorized as white. Among patients aged 65 years or older, the mean (SD) age was 77 (8) years; 96 169 (51.6%) were women, and 64 996 (34.9%) were categorized as white. The validation data sets consisted of 35 210 patients aged 18 to 64 years and 25 497 aged 65 years or older. The mean (SD) age in the validation data set among patients aged 18 to 64 years was 42 (14) years; 17 296 (49.1%) were women, and 13 346 (37.9%) were categorized as white. Among patients aged 65 years or older, the mean (SD) age was 73 (8) years; 13 135 (51.5%) were women, and 9641 (37.8) were categorized as white. Among patients aged 18 to 64 years, SIV uptake was 35.9% (95% CI, 35.7%-36.0%) and 32.6% (95% CI, 32.1%-33.1%) for the training and validation data sets, respectively. Among patients aged 65 years or older, SIV uptake was 83.1% (95% CI, 82.9%-83.2%) and 76.1% (95% CI, 75.5%-76.6%) for the training and validation data sets, respectively. Prior season SIV uptake and pneumococcal vaccination status were the best predictors of SIV uptake. Predicted SIV uptake probabilities for patients aged 18 to 64 years were reliable, but biased toward underpredicting, whereas, among patients aged 65 years or older, they were variable and biased toward overpredicting. Briefly, in out-of-sample validation among patients aged 18 to 64 years, misclassification rates were 0.163 to 0.164, Brier scores were 0.124 to 0.125, area under the receiver operating characteristic curve values ranged from 0.876 to 0.877, sensitivity ranged from 0.705 to 0.720, and specificity ranged from 0.896 to 0.902. In patients aged 65 years or older, misclassification rates were 0.120 to 0.125, Brier scores were 0.0953 to 0.0959, area under the receiver operating characteristic curve was 0.877, sensitivity ranged from 0.919 to 0.936, and specificity ranged from 0.680 to 0.753.

Conclusions and Relevance

This study suggests that data obtained from primary care records could accurately predict SIV uptake among at-risk adults. Further research is needed to assess the feasibility and efficacy of implementing this model in clinical settings.

Introduction

Each year, millions of individuals develop severe influenza disease globally, and as many as 500 000 individuals die from influenza-associated complications.¹ While most are susceptible to influenza, the risk of influenza-related morbidity and mortality is greatest among specific subgroups, including young children, adults older than 65 years, pregnant women, those with certain chronic health conditions, and those who are immunosuppressed.^2,3

The seasonal influenza vaccine (SIV) remains the most effective means of reducing influenza-associated morbidity and mortality, especially among clinical risk groups.¹ Despite the known efficacy and safety of SIVs, as well as the provision of free or highly subsidized vaccines to eligible patients in many jurisdictions, SIV uptake among at-risk adults is suboptimal and has, at best, stagnated for more than a decade across many regions, including North America and Europe.^4,5,6

Countless efforts have been made to improve SIV uptake through patient- and health care professional (HCP)–level interventions, yet the gap between realized and optimal coverage remains.^7,8 Notwithstanding, numerous studies have highlighted the pivotal role of HCPs in influencing and advising a patient’s health-related behaviors, such as smoking cessation and cancer screening.^9,10 A similar role is taken with vaccinations, where HCP communications and recommendations have been shown to increase uptake of various adolescent and adult vaccines.^11,12,13 Leveraging this unique role, HCP-level interventions represent a promising avenue through which the SIV coverage gap may be addressed.

Two commonly used HCP-level interventions include patient recall-reminder systems and software-based HCP-directed prompts, which have exhibited varying degrees of effectiveness.^{14,15,16,17,18,19} Among recall-reminder systems, automated communication systems are most easily implemented, whereas personalized phone calls or even home visits may be more effective, but are substantially more resource intensive to implement.^17,20 Software-based prompts via electric health record (EHR) systems that remind HCPs to vaccinate a patient at the time of consultation are similarly easy to implement and have been shown to be effective across various health systems.^21,22 However, these prompts may be rendered ineffective if used too frequently (known as prompt fatigue) and, further, do not provide any unique insights into the patient’s likelihood of being vaccinated.^23,24

As for characterizing a patient’s likelihood of being vaccinated, prior SIV uptake determinant studies have shown that patient characteristics—including sociodemographic characteristics, health-related behaviors, and comorbidities—are associated with uptake.^25,26,27,28 Therefore, it stands that a patient’s SIV uptake could reasonably be predicted based on such characteristics, allowing HCPs to effectively identify patients who are less likely to be vaccinated during an upcoming season.

Given stringent time constraints faced by HCPs, this insight may help them to optimally allocate their time and resources, through selective use of resource-intensive interventions as well as custom tailoring of software prompts to reflect the patient’s likelihood of SIV uptake.²⁹ Rapid improvements in the quality of primary care data can presumably be leveraged to generate real-time insights into a patient’s likelihood of receiving vaccination.³⁰ However, research in this area is limited; to our knowledge, only 1 study has attempted to construct a predictive model of SIV uptake using routinely collected primary care data.³¹

In this study, we investigated the feasibility of developing and validating an SIV uptake prediction model based only on patient characteristics attainable from primary care data to estimate the probability of patient-level SIV uptake among a population of at-risk adults in England. Using data from the UK’s Clinical Practice Research Datalink (CPRD) database for model training and validation, we assessed the predictive performance of 3 forms of logistic regression models.

Methods

Data Source

Data used for model training and validation were derived from the CPRD database,³² as described in Loiacono et al.³³ Briefly, a reference cohort (3 391 975 participants) was originally constructed to identify adults aged 18 years or older in the CPRD database who were registered to English practices for a minimum of 365 consecutive days between January 2011 and December 2016. The inclusion and exclusion criteria for this reference cohort were specified to identify patients with minimal gaps in registration and records of high enough quality for research purposes, as determined by the CPRD’s acceptability metric.³² A detailed diagram of the reference cohort construction is available in eFigure 1 in the Supplement.

From this reference cohort, we identified nonoverlapping cohorts of at-risk patients and constructed model training and out-of-sample validation data sets (Figure 1). Specific clinical risk conditions for inclusion, as defined by the National Health Service, included pregnancy, chronic renal disease, chronic heart disease, chronic respiratory disease, chronic liver disease, diabetes, immunosuppression, chronic neurological disease, and morbid obesity.³⁴ Training data consisted of patients registered to their practice from January 2012 to December 2013. Out-of-sample validation data consisted of patients registered to their practice from January 2015 to December 2016 and who were not present in the training data set. Two years of enrollment were required to assess the patient’s SIV uptake during the prior season. Data were extracted from the CPRD database in December 2018 and analyzed from September 2019 to December 2019.

Figure 1. — ^aAs constructed in Loiacono MM et al.³³

^bPatients aged 65 years or more without indices of multiple deprivation data were excluded (47 170 patients [20.2%] for training data; 9547 patients [27.2%] for validation data).

Among the training and validation data sets, patients without at least 1 clinical risk condition during both the observed and prior influenza season were excluded. While the National Health Service also identifies patient age as a risk condition (≥65 years), in this study only older adults with at least 1 additional clinical risk condition were included to focus model prediction on older patients at greatest risk of influenza-associated morbidity and mortality. Training and validation data sets were stratified by patient age (18-64 years and ≥65 years). This study received approval by the Independent Scientific Advisory Committee of CPRD. Informed consent from study participants was not required, given that individual-level consent was provided prior to data collection, and all data were deidentified prior to CPRD’s collection. The development and validation of the prediction model in this study was performed in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.³⁵

Variables

The dependent variable measured was patient-level SIV uptake. Uptake was assessed annually between September 1 and December 31, a timeframe encompassing most SIVs administered.³⁶ Patients vaccinated outside of this window were excluded as outliers (10 203 patients [0.5%] excluded from training data; 1204 patients [0.1%] excluded from validation data). Covariates included sex, age, race/ethnicity, smoking status, socioeconomic status, history of pneumococcal vaccination, prior season SIV uptake, and clinical risk conditions.³⁴ Age was treated as a continuous variable. Ethnicity was defined as specified by the Office of National Statistics.³⁷ Patient socioeconomic status was approximated using the Indices of Multiple Deprivation (IMD), a socioeconomic measure of the patient’s area of residence.³⁸ Identification of variables for inclusion in models was based on prior SIV uptake determinants research.³³

Pneumococcal vaccination history was determined by whether a patient had at least 1 record of a pneumococcal vaccination prior to December 31 of the given year. Clinical risk conditions were identified using Read codes as arranged by Primary Care Information Services, including: pregnancy, chronic renal disease, chronic heart disease, chronic respiratory disease, chronic liver disease, diabetes, immunosuppression, chronic neurological disease, and morbid obesity.³⁹ Time-varying patient characteristics were assessed prior to September 1 of the given year, based on the most recent record. Missing values for ethnicity, smoking status, and morbid obesity were coded as unknown. All codes used for data extraction from the CPRD database are described in detail in Loiacono et al.³³

Model Training and Validation

We evaluated 3 types of logistic regression models: stepwise, least absolute shrinkage and selection operator (LASSO), and ridge. We opted to use logistic regression models due to their inherently transparent development process and ease of implementation, all while maintaining strong predictive abilities.⁴⁰ All variables were considered for modeling across both age strata except for pregnancy and patient IMD. Pregnancy was not used in models for patients aged 65 years or more. Patient IMD was only used in models for patients aged 65 years or more, given prior evidence of the association with SIV uptake specifically in this age stratum.³³

Stepwise models were trained using a backward stepwise algorithm that systematically reduced the model via minimization of the Akaike information criterion.⁴¹ For the LASSO and ridge models, an optimal value of λ, or the penalty coefficient for the loss functions, was determined via 10-fold cross-validations using the 2013 training data, in which the optimal λ minimized model deviance (eFigure 2, eFigure 3, eFigure 4, and eFigure 5 in the Supplement).⁴² Both the backward stepwise algorithm and LASSO autonomously performed feature or variable selection, which resulted in a reduced model, whereas ridge maintained the full model.

Models were trained on the 2013 training data set and validated on the 2016 out-of-sample validation data set. Given the limited prior research to guide cutoff selection, an uninformative cutoff of 0.5 was specified, a priori, to classify a patient’s SIV uptake status based on their predicted probability (ie, a predicted probability ≤0.5 did not receive SIV; a predicted probability >0.5 received SIV). For models trained on the 2013 training data set, estimated coefficients were reported.

Statistical Analysis

To assess the out-of-sample predictive performance of the models, the following performance metrics were calculated: misclassification rate, Brier score, area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Misclassification rate (ie, proportion of patients with an incorrectly predicted SIV uptake status, in which 0 indicates no misclassification) was calculated as the sum of false positives and false negatives divided by the total number of predictions. Brier score (accuracy of a probabilistic prediction ranging from 0 to 1, in which 0 indicates perfect accuracy) was calculated as Σ(y_i – p_i)2 / n, where y_i and p_i were the observed SIV uptake status and probabilistic prediction for patient i, respectively, among n total patients.⁴³ The AUC is a measure of the model’s discrimination power, ranging from 0 to 1, in which 1 indicates perfect prediction. Sensitivity and specificity are true-positive and true-negative rates, ranging from 0 to 1, in which 1 indicates perfect true-positive or true-negative prediction.

Uncertainty measures for all performance metrics (95% CIs) were calculated using normal approximation methods (eAppendix 1 in the Supplement). Calibration plots were constructed based on the methods previously described in Gerds et al.⁴⁴ Receiver operating characteristic curves and kernel density plots were constructed and are available (eFigures 6-12 in the Supplement). Additionally, results from a sensitivity analysis, in which models were trained and validated through a within-sample 10-fold cross-validation using the 2013 training data set, are available in eTable in the Supplement. All analyses were performed in R version 3.4.3 (R Project for Statistical Computing). Further details on specific R packages are available in eAppendix 2 in the Supplement.

Results

Summary of Patient Characteristics

Training data consisted of 324 284 patients aged 18 to 64 years (mean [SD] age, 45 [13] years; 161 487 women [49.8%]; 102 133 categorized as white [31.5%]) and 186 426 patients aged 65 years or older (mean [SD] age, 77 [8] years; 96 169 women [51.6%]; 64 996 categorized as white [34.9%]) (Table 1). Validation data consisted of 35 210 patients aged 18 to 64 years (mean [SD] age, 42 [14] years; 17 296 women [49.1%]; 13 346 categorized as white [37.9%]) and 25 497 patients aged 65 years or older (mean [SD] age, 73 [8] years; 13 135 women [51.5%]; 9641 categorized as white [37.8%]). Across both age strata, SIV uptake (prior and current season) and lower pneumococcal were noted in the validation data relative to the training data. Among patients aged 18 to 64 years, SIV uptake was 35.9% (116 316 patients; 95% CI, 35.7%-36.0%) and 32.6% (11 493 patients; 95% CI, 32.1%-33.1%) for the training and validation data sets, respectively. Among patients aged 65 years or older, SIV uptake was 83.1% (154 872 patients; 95% CI, 82.9%-83.2%) and 76.1% (11 493 patients; 95% CI, 75.5%-76.6%) for the training and validation data sets, respectively. Pneumococcal vaccine uptake among patients aged 18 to 64 years was 23.4% (75 984 patients) and 16.7% (5882 patients) for training and validation sets, respectively; for patients aged 65 years or older, uptake was 85.1% (158 661 patients) for the training set and 74.9% (19 099 patients) for the validation set.

Table 1. Summary of Characteristics of Patients in the Training and Validation Data Sets, Stratified by Patient Age.

Characteristic	Patients by age group, No. (%)
	Training data (season 2013)		Validation data (season 2016)
	18-64 y (n = 324 284)	≥65 y (n = 186 426)	18-64 y (n = 35 210)	≥65 y (n = 25 497)
Men	162 797 (50.2)	90 257 (48.4)	17 914 (50.9)	12 362 (48.5)
Women	161 487 (49.8)	96 169 (51.6)	17 296 (49.1)	13 135 (51.5)
Age, mean (SD), y	45 (13)	77 (8)	42 (14)	73 (8)
Race/ethnicity
Asian	10 330 (3.2)	3051 (1.6)	1932 (5.5)	660 (2.6)
Black	6570 (2.0)	1540 (0.8)	1306 (3.7)	276 (1.1)
Mixed	67 229 (20.7)	37 464 (20.1)	8312 (23.6)	5975 (23.4)
Other	2943 (0.9)	1056 (0.6)	608 (1.7)	257 (1.0)
Unknown	135 079 (41.7)	78 319 (42.0)	9706 (27.6)	8688 (34.1)
White	102 133 (31.5)	64 996 (34.9)	13 346 (37.9)	9641 (37.8)
Patient IMD
1, Least deprived	NA	44 729 (24.0)	NA	7329 (28.7)
2	NA	43 936 (23.6)	NA	5288 (20.7)
3	NA	40 268 (21.6)	NA	5196 (20.4)
4	NA	32 341 (17.3)	NA	4267 (16.7)
5, Most deprived	NA	25 152 (13.5)	NA	3417 (13.4)
Smoking status
Never	173 300 (53.4)	92 422 (49.6)	18 831 (53.5)	12 411 (48.7)
Current	74 142 (22.9)	17 897 (9.6)	8396 (23.8)	3723 (14.6)
Former	73 721 (22.7)	75 480 (40.5)	7491 (21.3)	9166 (35.9)
Unknown	3121 (1.0)	627 (0.3)	492 (1.4)	197 (0.8)
Pneumococcal vaccine uptake	75 984 (23.4)	158 661 (85.1)	5882 (16.7)	19 099 (74.9)
Prior season SIV uptake	112 148 (34.6)	153 159 (82.2)	10 970 (31.2)	19 278 (75.6)
Current season SIV uptake, No. (% [95% CI])	116 316 (35.9 [35.7-36.0])	154 872 (83.1 [82.9-83.2])	11 493 (32.6 [32.1-33.1])	19 393 (76.1 [75.5-76.6])
Clinical risk conditions^a
1	272 932 (84.2)	113 597 (60.9)	30 683 (87.1)	17 854 (70.0)
≥2	51 352 (15.8)	72 829 (39.1)	4527 (12.9)	7643 (30.0)

Open in a new tab

Abbreviations: IMD, Indices of Multiple Deprivation; NA, not applicable; SIV, seasonal influenza vaccine.

^{^a}

Among patients aged 65 or more, age was not counted as a clinical risk condition.

Model Training

Estimated coefficients for trained models are presented in Table 2. The most influential predictors across both age strata were prior season SIV uptake and pneumococcal vaccination status. Otherwise, among patients aged 18 to 64 years, pregnancy (estimated coefficient for stepwise model, 1.80; LASSO, 1.76; ridge, 1.24), diabetes (estimated coefficient for stepwise model, 0.88; LASSO, 0.85; ridge, 0.69), and chronic neurological disease (estimated coefficient for stepwise model, 0.68; LASSO, 0.64; ridge, 0.48) had the greatest contributions. Among patients aged 65 years or older, estimated coefficients were overall smaller and less variable. For stepwise and LASSO models, only a small number of predictors were autonomously excluded from the models among patients aged 18 to 64 years (eg, race [mixed, other, and white] and chronic liver disease) as well as among patients aged 65 years or older (eg, race [mixed, other, and white], chronic renal disease, chronic liver disease, and chronic neurological disease), indicating that most predictors contributed nontrivially to the model fit.

Table 2. Estimated Coefficients for All Models Trained on the Entire 2013 Training Data Set, Stratified by Age.

Variable	Estimated coefficients by age group
	18-64 y			≥65 y
	Stepwise	LASSO	Ridge	Stepwise	LASSO	Ridge
Intercept	–3.91	–3.91	–3.19	–0.60	–0.73	–0.81
Women	0.22	0.21	0.17	–0.09	–0.09	–0.08
Age	0.032	0.032	0.025	–0.015	–0.014	–0.008
Ethnicity
Asian	1 [Reference]	1 [Reference]	1 [Reference]	1 [Reference]	1 [Reference]	1 [Reference]
Black	–0.21	–0.10	–0.09	–0.22	–0.12	–0.12
Mixed	–0.076	NA^a	0.022	–0.073	NA^a	0.025
Other	–0.060	NA^a	0.024	–0.072	NA^a	–0.005
Unknown	–0.22	–0.14	–0.10	–0.18	–0.10	–0.08
White	–0.071	NA^a	0.034	–0.037	0.026	0.055
Patient IMD
1, Least deprived	NA^b	NA^b	NA^b	1 [Reference]	1 [Reference]	1 [Reference]
2	NA^b	NA^b	NA^b	–0.087	–0.047	–0.028
3	NA^b	NA^b	NA^b	–0.092	–0.053	–0.040
4	NA^b	NA^b	NA^b	–0.140	–0.095	–0.071
5, Most deprived	NA^b	NA^b	NA^b	–0.110	–0.067	–0.059
Smoking status
Never	1 [Reference]	1 [Reference]	1 [Reference]	1 [Reference]	1 [Reference]	1 [Reference]
Current	–0.12	–0.11	–0.10	–0.24	–0.23	–0.21
Former	0.061	0.054	0.086	0.073	0.068	0.077
Unknown	–0.74	–0.61	–0.50	–0.26	–0.22	–0.29
Pneumococcal vaccination	1.07	1.07	0.96	1.48	1.48	1.30
Prior season SIV uptake	3.06	3.05	2.43	3.60	3.60	2.99
Pregnant	1.80	1.76	1.24	NA^c	NA^c	NA^c
Chronic renal disease	0.32	0.26	0.25	NA^a	–0.01	–0.02
Chronic heart disease	0.42	0.39	0.30	0.11	0.09	0.08
Chronic respiratory disease	0.20	0.16	0.02	0.06	0.05	0.06
Chronic liver disease	NA^a	NA^a	0.008	NA^a	0.020	0.024
Diabetes	0.88	0.85	0.69	0.15	0.13	0.12
Immunosuppression	0.52	0.46	0.35	0.13	0.10	0.08
Chronic neurological disease	0.68	0.64	0.48	NA^a	NA^a	–0.01
Obesity
Morbid (BMI ≥40)	–0.27	–0.26	–0.27	–0.11	–0.10	–0.10
Unknown	–0.41	–0.40	–0.39	–0.14	–0.13	–0.20

Open in a new tab

Abbreviations: BMI, body mass index (calculated as weight in kilograms divided by height in meters squared); IMD, Indices of Multiple Deprivation; LASSO, least absolute shrinkage and selection operator; NA, not applicable; SIV, seasonal influenza vaccine.

^{^a}

Variable removed by autonomous feature selection (stepwise and LASSO only).

^{^b}

Patient IMD manually excluded from models among patients aged 18 to 64 years because of consistent insignificance.

^{^c}

Variable excluded from models for patients aged 65 years or more because of lack of pregnancy records.

Model Validation

Within each age strata, the 3 model types performed similarly overall (Table 3). Misclassification rates were lowest for the stepwise and LASSO models among both patients aged 18 to 64 years (0.163, 95% CI, 0.159-0.166) and 65 years or older (0.120, 95% CI, 0.116-0.124). Brier scores were highest for the stepwise and LASSO models among patients aged 18 to 64 years (0.125, 95% CI, 0.122-0.127), whereas they were highest for the ridge model among those aged 65 years or older (0.0959, 95% CI, 0.0932-0.0985). All models across both age strata had comparable AUCs (0.877, 95% CI, 0.873-0.881). Sensitivity was highest for the LASSO model among those aged 18 to 64 years (0.721, 95% CI, 0.717-0.725) and the ridge model among those aged 65 years or older (0.936, 95% CI, 0.933-0.939). Specificity was highest for the ridge model among those aged 18 to 64 years (0.902, 95% CI, 0.899-0.905) and the LASSO model among those aged 65 years or older (0.755, 95% CI, 0.750-0.760).

Table 3. Performance Measures for Out-of-Sample Validation of Stepwise, LASSO, and Ridge Models, Among Patients Aged 18 to 64 Years and 65 Years or Older.

Model	Performance measure^a
Model	Misclassification rate	Brier score	AUC	Sensitivity	Specificity
Patients aged 18-64 y
Stepwise	0.163 (0.159-0.166)	0.125 (0.122-0.127)	0.877 (0.873-0.881)	0.720 (0.716-0.724)	0.897 (0.894-0.899)
LASSO	0.163 (0.159-0.166)	0.125 (0.122-0.127)	0.877 (0.873-0.881)	0.721 (0.717-0.725)	0.896 (0.893-0.899)
Ridge	0.164 (0.160-0.167)	0.124 (0.122-0.126)	0.876 (0.873-0.880)	0.705 (0.701-0.710)	0.902 (0.899-0.905)
Patients aged ≥65 y
Stepwise	0.120 (0.116-0.124)	0.0954 (0.093-0.098)	0.877 (0.873-0.881)	0.920 (0.916-0.923)	0.753 (0.747-0.758)
LASSO	0.120 (0.116-0.124)	0.0953 (0.093-0.098)	0.877 (0.873-0.881)	0.919 (0.916-0.922)	0.755 (0.750-0.760)
Ridge	0.125 (0.121-0.129)	0.0959 (0.093-0.099)	0.877 (0.873-0.881)	0.936 (0.933-0.939)	0.680 (0.675-0.686)

Open in a new tab

Abbreviations: AUC, area under the receiver operating character curve; LASSO, least absolute shrinkage and selection operator; SIV, seasonal influenza vaccine.

^{^a}

Misclassification rate measures the proportion of patients with an incorrectly predicted SIV uptake status (based upon a cutoff of 0.5). Brier score measures the accuracy of a probabilistic prediction, ranging from 0 to 1, where 0 indicates perfect accuracy. AUC measures the model’s discrimination power, ranging from 0 to 1, where 0.5 indicates an inability to appropriately classify a patient’s SIV uptake and 1 indicates perfect prediction. Sensitivity measures the true positive rate; specificity measures the true negative rate.

Predicted probabilities for patients aged 18 to 64 years were overall reliable and comparable among the 3 model types (Figure 2). For the intermediate range of predicted probabilities (eg, 0.25-0.50), prediction was biased toward underpredicting. The ridge model exhibited the highest degree of underprediction bias, particularly for probabilities of 0.50 or more. For patients aged 65 years or older, predicted probabilities were substantially more variable and overall biased toward overpredicting (Figure 2). Prediction bias of the 3 model types was similar for probabilities greater than 0.75. However, for probabilities between approximately 0.20 and 0.60, the ridge model’s bias toward underpredicting was notably greater than those of the stepwise or LASSO models.

Discussion

This study found that a clinical prediction tool using patient-level characteristics identified via primary care records was able to estimate the probability of patient-level SIV uptake among at-risk adults with an overall high degree of accuracy. As for the specific methods tested, performance of the 3 types of models were comparable, exhibiting only minor discrepancies across the various performance metrics. Relative to the stepwise model, neither the LASSO nor ridge models performed substantially better. These findings suggest that the simplest approach, the stepwise regression, is well suited for this type of prediction model.

Model performance did however differ notably between the 2 age strata. Among younger adults, estimated probabilities were closer to the observed SIV uptake probabilities, whereas estimated probabilities were substantially more variable among older adults. This may be explained in part by the imbalanced outcome variable (eg, approximately 70%-80% vaccine uptake among older adults) but may also be indicative of a lower overall degree of explanatory power for predictors in the models among patients aged 65 years or more. Considering this, either model may reliably be used to identify patients who are least likely to be vaccinated (eg, probability ≤25%), but the model among younger adults may be better suited for evaluating the patient’s specific probability of being vaccinated.

Our work establishes a new possible foundation from which future clinical tools and interventions may be developed, in which the insights gained from these predictions may guide HCP resource allocation. By identifying patients with a low probability of being vaccinated, HCPs can deploy more resource-intensive outreach efforts, such as personalized phone calls. Additionally, the geographical distribution of patients with low probabilities to be vaccinated could be further investigated for instances of clustering, which would highlight areas with potential limitations to health care access.

As for HCP-level interventions, this model could be integrated into EHR systems, to deliver HCP-directed reminders via software prompts at the time of patient consultation. With the model performing real-time calculations behind the scenes, EHR-based prompts could be customized to not only remind the HCP to vaccinate the patient, but also serve the HCP with a unique insight into the patient’s inherent likelihood of being vaccinated and helping them tailor the conversation accordingly. Given the time constraints placed upon HCPs during a consultation, this may help them more efficiently use their time.⁴⁵

For example, a patient can be characterized as unlikely, moderately likely, or highly likely to be vaccinated based on their predicted probability, and EHR prompts could be constructed accordingly. For patients flagged as highly likely, the prompt may advise the HCP to make presumptive and time-saving recommendation (eg, “today we will be giving you your flu shot”).⁴⁶ For patients flagged as unlikely, the prompt may include a list of the patient’s clinical risk conditions, suggesting that the HCP initiate a personalized dialogue with regard to the patient’s specific condition(s) and the importance of vaccination, which would be followed by a presumptive recommendation.⁴⁷ As for those patients flagged as moderately likely, the prompt may advise the HCP to make a presumptive recommendation, but also to be prepared to discuss the patient’s specific risk conditions if they exhibit hesitancy. This scenario demonstrates 1 way these models potentially can be used to autonomously deliver time-optimizing, patient-tailored guidance to HCPs.

Limitations

Despite the strong predictive performance of these models, this study has some limitations that must be acknowledged. First, we explicitly opted to treat missing values as unknown, rather than the more common approach of imputation or exclusion of observations with missing data.⁴⁸ However, in a clinical setting, it is likely that values for some predictors may be missing in the patient’s records, such as ethnicity, smoking status, or body mass index. By explicitly modeling these missing values, we preserve their explanatory power among patients that have a known value while retaining the model’s ability to predict among patients with missing values, thereby maximizing the model’s clinical utility. As for excluding patients aged 65 years or older without IMD data, this was decided to reflect real-world circumstances, in which a patient’s IMD measure can simply be obtained via cross-referencing their area of residence with an IMD lookup table.

Second, while the use of CPRD’s database allowed us to capture a wide breadth of patients and perform true out-of-sample validation, a fundamental strength of this study, it also introduced its own respective disadvantages. Given that our model was trained on England-specific data, it may not be applicable to other regions. Nevertheless, the framework that we have implemented here can be replicated elsewhere using similar sources of data to train and validate country-specific prediction models. Doing so would ensure that the model’s predictive capabilities are best suited to the respective health system and its patients.

Additionally, we noted a decrease in both pneumococcal vaccine uptake in the validation data sets, relative to the training data sets. As explained in Loiacono et al,³³ this observed drop in vaccine coverage over time may be explained by the increasing number of CPRD-enrolled practices dropping out of data collection over time, or perhaps even the increase in pharmacy-administered vaccines and the subsequent lack of appropriate data transfers to general practitioners. Nevertheless, within the framework of predictive modeling, these differences are of less concern, as similarity of baseline characteristics between training and validation data are not a prerequisite.

Third, although the model was capable of accurate prediction, it does not explicitly explain why the patient was likely or unlikely to be vaccinated. This is an inherent limitation of using large primary care databases, given that we must model a patient’s vaccine uptake as a function of only the characteristics that we can confidently measure. Thus, this study does not account for other known determinants of SIV uptake, such as personal beliefs, opinions, and other social factors that could not be accurately measured in CPRD’s database.⁴⁹ Similarly, it is possible that the patient-level attitudes and behavior underlying vaccine uptake may vary over time, making comparison between different years difficult. Nevertheless, the inclusion of a lagged independent variable (ie, prior season SIV uptake) in the model allowed for flexibility in this regard, given that it effectively encompassed and adjusted for changes in patient attitudes based on their historical actions.

Conclusions

The results of this study suggest that primary care records can be leveraged to provide future insights into patient preventive health behaviors such as SIV uptake. Logistic regression models can predict SIV uptake with high accuracy, and the modeling approach implemented here can likely be adapted to other countries and databases. Future research is needed to assess the feasibility of implementing this model in a clinical setting as well as to evaluate its potential effectiveness with regard to improving SIV uptake. Similarly, future studies may wish to investigate the performance of additional methods, such as decision tree learning, to identify a smaller subset of critical predictors that may be used as a decision tool, thus simplifying the model’s implementation.

Supplement.

eFigure 1. Overview of the selection of the reference cohort

eFigure 2. Binomial deviance versus log(Lambda) for 10-fold within-sample cross-validation of LASSO model among patients aged 18-64 years

eFigure 3. Binomial deviance versus log(Lambda) for 10-fold within-sample cross-validation of ridge model among patients aged 18-64 years

eFigure 4. Binomial deviance versus log(Lambda) for 10-fold within-sample cross-validation of LASSO model among patients aged 65+ years

eFigure 5. Binomial deviance versus log(Lambda) for 10-fold within-sample cross-validation of ridge model among patients aged 65+ years

eFigure 6. ROC curve for out-of-sample validation of stepwise model among patients aged 18-64 years

eFigure 7. ROC curve for out-of-sample validation of LASSO model among patients aged 18-64 years

eFigure 8. ROC curve for out-of-sample validation of ridge model among patients aged 18-64 years

eFigure 9. ROC curve for out-of-sample validation of stepwise model among patients aged 65+ years

eFigure 10. ROC curve for out-of-sample validation of LASSO model among patients aged 65+ years

eFigure 11. ROC curve for out-of-sample validation of ridge model among patients aged 65+ years

eFigure 12. Density plots of predicted SIV uptake probabilities for out-of-sample validation for stepwise, LASSO, and ridge models among patients aged (a) 18-64 (b) 65+ years

eTable 1. Performance measures from sensitivity anaylsis (10-fold within-sample cross-validation) for stepwise, LASSO, and ridge models among patients aged (a) 18-64 and (b) 65+ years

eMethods 1. Measures of uncertainty

eMethods 2. Statistical software

Click here for additional data file.^{(906.1KB, pdf)}

References

1.World Health Organization . Influenza (seasonal). 2009. Accessed September 1, 2019. https://www.who.int/news-room/fact-sheets/detail/influenza-(seasonal)/
2.US Centers for Disease Control and Prevention . People at high risk for flu complications. Published 2018. Accessed August 5, 2019. https://www.cdc.gov/flu/highrisk/index.htm
3.Mertz D, Kim TH, Johnstone J, et al. Populations at risk for severe or complicated influenza illness: systematic review and meta-analysis. BMJ. 2013;347:f5061. doi: 10.1136/bmj.f5061 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.US Centers for Disease Control and Prevention . Estimates of influenza vaccination coverage among adults—United States, 2017-18 flu season. Published 2018. Accessed August 7, 2019. https://www.cdc.gov/flu/fluvaxview/coverage-1718estimates.htm
5.World Health Organization . Prevention and control of influenza pandemics and annual epidemics. World Health Assembly; WHA56.19. 2003. Accessed June 2, 2020. https://www.who.int/immunization/sage/1_WHA56_19_Prevention_and_control_of_influenza_pandemics.pdf
6.Jorgensen P, Mereckiene J, Cotter S, Johansen K, Tsolova S, Brown C. How close are countries of the WHO European Region to achieving the goal of vaccinating 75% of key risk groups against influenza? results from national surveys on seasonal influenza vaccination programmes, 2008-2009 to 2014-2015. Vaccine. 2018;36(4):442-452. doi: 10.1016/j.vaccine.2017.12.019 [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Ndiaye SM, Hopkins DP, Shefer AM, et al. Task Force on Community Preventive Services . Interventions to improve influenza, pneumococcal polysaccharide, and hepatitis B vaccination coverage among high-risk adults: a systematic review. Am J Prev Med. 2005;28(5)(suppl):248-279. doi: 10.1016/j.amepre.2005.02.016 [DOI] [PubMed] [Google Scholar]
8.Shefer A, Briss P, Rodewald L, et al. Improving immunization coverage rates: an evidence-based review of the literature. Epidemiol Rev. 1999;21(1):96-142. doi: 10.1093/oxfordjournals.epirev.a017992 [DOI] [PubMed] [Google Scholar]
9.Park ER, Gareen IF, Japuntich S, et al. Primary care provider-delivered smoking cessation interventions and smoking cessation among participants in the National Lung Screening Trial. JAMA Intern Med. 2015;175(9):1509-1516. doi: 10.1001/jamainternmed.2015.2391 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Peterson EB, Ostroff JS, DuHamel KN, et al. Impact of provider-patient communication on cancer screening adherence: a systematic review. Prev Med. 2016;93:96-105. doi: 10.1016/j.ypmed.2016.09.034 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Johnson DR, Nichol KL, Lipczynski K. Barriers to adult immunization. Am J Med. 2008;121(7)(suppl 2):S28-S35. doi: 10.1016/j.amjmed.2008.05.005 [DOI] [PubMed] [Google Scholar]
12.Ylitalo KR, Lee H, Mehta NK. Health care provider recommendation, human papillomavirus vaccination, and race/ethnicity in the US National Immunization Survey. Am J Public Health. 2013;103(1):164-169. doi: 10.2105/AJPH.2011.300600 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Nichol KL, Mac Donald R, Hauge M. Factors associated with influenza and pneumococcal vaccination behavior among high-risk adults. J Gen Intern Med. 1996;11(11):673-677. doi: 10.1007/BF02600158 [DOI] [PubMed] [Google Scholar]
14.Kontopantelis E, Doran T, Gravelle H, Goudie R, Siciliani L, Sutton M. Family doctor responses to changes in incentives for influenza immunization under the U.K. Quality and Outcomes Framework pay-for-performance scheme. Health Serv Res. 2012;47(3 Pt 1):1117-1136. doi: 10.1111/j.1475-6773.2011.01362.x [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Ministry of Health and Long-Term Care, Negotiations Branch . Information and procedures for claiming the cumulative preventive care bonus. Revised March 2018. Accessed May 4, 2020. http://www.health.gov.on.ca/en/pro/programs/ohip/bulletins/11000/bul11190_cpcb.pdf
16.Blue Cross Blue Shield . 2017 Performance recognition program. Published 2017. Accessed May 4, 2020. https://thephysicianalliance.org/images/FilesDocuments/2017_BCN_BCBSM_PRPBooklet_Final122016.pdf
17.Vann JCJ, Jacobson RM, Coyne-Beasley T, Asafu-Adjei JK, Szilagyi PG. Patient reminder and recall systems to improve immunization rates. Cochrane Database Syst Rev. 2005;(3):CD0033941. doi: 10.1002/14651858.CD003941.pub2 [DOI] [Google Scholar]
18.National Institute for Health and Care Excellence . Flu vaccination: increasing uptake. Published August 22, 2018. Accessed November 23, 2019. https://www.nice.org.uk/guidance/ng103/evidence/3-increasing-flu-vaccination-uptake-in-clinical-risk-groups-pdf-6532083616
19.Patel MS, Volpp KG, Small DS, et al. Using active choice within the electronic health record to increase physician ordering and patient completion of high-value cancer screening tests. Healthc (Amst). 2016;4(4):340-345. doi: 10.1016/j.hjdsi.2016.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Thomas RE, Lorenzetti DL. Interventions to increase influenza vaccination rates of those 60 years and older in the community. Cochrane Database Syst Rev. 2014;(7):CD005188. doi: 10.1002/14651858.CD005188.pub3 [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Newby KV, Parsons J, Brooks J, Leslie R, Inglis N. Identifying strategies to increase influenza vaccination in GP practices: a positive deviance approach. Fam Pract. 2016;33(3):318-323. doi: 10.1093/fampra/cmw016 [DOI] [PubMed] [Google Scholar]
22.Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Arch Intern Med. 2000;160(3):301-308. doi: 10.1001/archinte.160.3.301 [DOI] [PubMed] [Google Scholar]
23.Szilagyi PG, Serwint JR, Humiston SG, et al. Effect of provider prompts on adolescent immunization rates: a randomized trial. Acad Pediatr. 2015;15(2):149-157. doi: 10.1016/j.acap.2014.10.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Zazove P, McKee M, Schleicher L, et al. To act or not to act: responses to electronic health record prompts by family medicine clinicians. J Am Med Inform Assoc. 2017;24(2):275-280. doi: 10.1093/jamia/ocw178 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Burns VE, Ring C, Carroll D. Factors influencing influenza vaccination uptake in an elderly, community-based sample. Vaccine. 2005;23(27):3604-3608. doi: 10.1016/j.vaccine.2004.12.031 [DOI] [PubMed] [Google Scholar]
26.Endrich MM, Blank PR, Szucs TD. Influenza vaccination uptake and socioeconomic determinants in 11 European countries. Vaccine. 2009;27(30):4018-4024. doi: 10.1016/j.vaccine.2009.04.029 [DOI] [PubMed] [Google Scholar]
27.Sammon CJ, McGrogan A, Snowball J, de Vries CS. Factors associated with uptake of seasonal and pandemic influenza vaccine among clinical risk groups in the UK: an analysis using the General Practice Research Database. Vaccine. 2012;30(14):2483-2489. doi: 10.1016/j.vaccine.2011.11.077 [DOI] [PubMed] [Google Scholar]
28.Mangtani P, Breeze E, Stirling S, Hanciles S, Kovats S, Fletcher A. Cross-sectional survey of older peoples’ views related to influenza vaccine uptake. BMC Public Health. 2006;6(1):249. doi: 10.1186/1471-2458-6-249 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Konrad TR, Link CL, Shackelton RJ, et al. It’s about time: physicians’ perceptions of time constraints in primary care medical practice in three national healthcare systems. Med Care. 2010;48(2):95-100. doi: 10.1097/MLR.0b013e3181c12e6a [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Bollaerts K, de Smedt T, McGee C, et al. ADVANCE: towards near real-time monitoring of vaccination coverage, benefits and risks using European electronic health record databases. Vaccine. 2019;S0264-410X(19)31051-5. doi: 10.1016/j.vaccine.2019.08.012 [DOI] [PubMed] [Google Scholar]
31.Shaham A, Chodick G, Shalev V, Yamin D. Personal and social patterns predict influenza vaccination decision. BMC Public Health. 2020;20(1):222. doi: 10.1186/s12889-020-8327-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Herrett E, Gallagher AM, Bhaskaran K, et al. Data resource profile: Clinical Practice Research Datalink (CPRD). Int J Epidemiol. 2015;44(3):827-836. doi: 10.1093/ije/dyv098 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Loiacono MM, Mahmud SM, Chit A, et al. Patient and practice level factors associated with seasonal influenza vaccine uptake among at-risk adults in England, 2011 to 2016: an age-stratified retrospective cohort study. Vaccine X. 2020;4:100054. doi: 10.1016/j.jvacx.2020.100054 [DOI] [PMC free article] [PubMed] [Google Scholar]
34.National Health Service . Who should have the flu vaccine? Revised July 2019. Accessed September 1, 2019. https://www.nhs.uk/conditions/vaccinations/who-should-have-flu-vaccine/
35.Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. 2015;162(1):55-63. doi: 10.7326/M14-0697 [DOI] [PubMed] [Google Scholar]
36.National Health Service . The national flu immunisation programme 2016/17 letter. Published May 26, 2016. Accessed October 1, 2019. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/529954/Annual_flu_letter_2016_2017.pdf
37.Office for National Statistics . Harmonised concepts and questions for social data sources, secondary standards: social capital. Published May 2015. Accessed May 5, 2019. https://gss.civilservice.gov.uk/wp-content/uploads/2016/03/P3-Ethnic-Group-June-16-1.pdf
38.Leeser R. English Indices of Deprivation 2015: Greater London Authority Intelligence briefing. Published 2015. Accessed November 4, 2019. https://data.london.gov.uk/dataset/indices-of-deprivation
39.Primary Care Information Services . Seasonal influenza vaccine uptake reporting specification: collection 2017/2018. University of Nottingham PRIMIS team; 2017. Accessed September 1, 2019. http://www.inps.co.uk/sites/default/files/seasonal_flu_lqd_specification_2017_v9.3.pdf
40.Shipe ME, Deppen SA, Farjah F, Grogan EL. Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis. 2019;11(suppl 4):S574-S584. doi: 10.21037/jtd.2019.01.25 [DOI] [PMC free article] [PubMed] [Google Scholar]
41.Venables WN, Ripley BD. Modern Applied Statistics With S-PLUS. Springer Science & Business Media; 2013. [Google Scholar]
42.Kassambara A. Machine Learning Essentials: Practical Guide in R. STHDA; 2018. [Google Scholar]
43.Ferro CA. Comparing probabilistic forecasting systems with the Brier score. Weather Forecast. 2007;22(5):1076-1088. doi: 10.1175/WAF1034.1 [DOI] [Google Scholar]
44.Gerds TA, Andersen PK, Kattan MW. Calibration plots for risk prediction models in the presence of competing risks. Stat Med. 2014;33(18):3191-3203. doi: 10.1002/sim.6152 [DOI] [PubMed] [Google Scholar]
45.Paterson P, Meurice F, Stanberry LR, Glismann S, Rosenthal SL, Larson HJ. Vaccine hesitancy and healthcare providers. Vaccine. 2016;34(52):6700-6706. doi: 10.1016/j.vaccine.2016.10.042 [DOI] [PubMed] [Google Scholar]
46.Hofstetter AM, Robinson JD, Lepere K, Cunningham M, Etsekson N, Opel DJ. Clinician-parent discussions about influenza vaccination of children and their association with vaccine acceptance. Vaccine. 2017;35(20):2709-2715. doi: 10.1016/j.vaccine.2017.03.077 [DOI] [PMC free article] [PubMed] [Google Scholar]
47.Connors JT, Slotwinski KL, Hodges EA. Provider-parent communication when discussing vaccines: a systematic review. J Pediatr Nurs. 2017;33:10-15. doi: 10.1016/j.pedn.2016.11.002 [DOI] [PubMed] [Google Scholar]
48.Hunsberger S, Murray D, Davis CE, Fabsitz RR. Imputation strategies for missing data in a school-based multi-centre study: the Pathways study. Stat Med. 2001;20(2):305-316. doi: [DOI] [PubMed] [Google Scholar]
49.Jain A, van Hoek AJ, Walker JL, Mathur R, Smeeth L, Thomas SL. Identifying social factors amongst older individuals in linked electronic health records: an assessment in a population based study. PLoS One. 2017;12(11):e0189038. doi: 10.1371/journal.pone.0189038 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials