Skip to main content
Human Vaccines & Immunotherapeutics logoLink to Human Vaccines & Immunotherapeutics
. 2019 Apr 22;15(7-8):1803–1814. doi: 10.1080/21645515.2019.1575711

Using the precaution adoption process model to clarify human papillomavirus vaccine hesitancy in canadian parents of girls and parents of boys

Ovidiu Tatar a,, Gilla K Shapiro a,b, Samara Perez a,b, Kristina Wade a, Zeev Rosberger a,b
PMCID: PMC6746468  PMID: 30735442

ABSTRACT

Background: Achieving optimal human papillomavirus (HPV) vaccine uptake can be delayed by parents’ HPV vaccine hesitancy, which is as a multi-stage intention process rather than a dichotomous (vaccinated/not vaccinated) outcome. Our objective was to longitudinally explore HPV related attitudes, beliefs and knowledge and to estimate the effect of psychosocial factors on HPV vaccine acceptability in HPV vaccine hesitant parents of boys and girls.

Methods: We used an online survey to collect data from a nationally representative sample of Canadian parents of 9–16 years old boys and girls in September 2016 and July 2017. Informed by the Precaution Adoption Process Model, we categorized HPV vaccine hesitant parents into unengaged/undecided and decided not. Measures included sociodemographics, health behaviors and validated scales for HPV and HPV vaccine related attitudes, beliefs and knowledge. Predictors of HPV vaccine acceptability were assessed with binomial logistic regression.

Results: Parents of boys and girls categorized as “flexible” hesitant (i.e., unengaged/undecided) changed over time their HPV related attitudes, behaviors, knowledge and intentions to vaccinate compared to “rigid” hesitant (i.e., decided not) who remained largely unchanged. In “flexible” hesitant, greater social influence to vaccinate (e.g., from family), increased HPV knowledge, higher family income, white ethnicity and lower perception of harms (e.g., vaccine safety), were associated with higher HPV vaccine acceptability.

Conclusions: HPV vaccine hesitant parents are not a homogenous group. We have identified significant predictors of HPV vaccine acceptability in “flexible” hesitant parents. Further research is needed to estimate associations between psychosocial factors and vaccine acceptability in “rigid” hesitant parents.

KEYWORDS: Human papillomavirus, vaccine, hesitancy, parents, knowledge, attitudes, intentions

Introduction

Human papillomavirus (HPV) vaccines are considered a “game changer” in cancer prevention in both women and men.1 HPV vaccines are highly effective in reducing high-risk HPV (e.g., 16 and 18) infection prevalence and offer protection against cervical, vaginal, vulvar, penile, anal, and oropharyngeal cancers.2 Worldwide, over 270 million doses of HPV vaccines have been distributed and the Global Advisory Committee on Vaccine Safety of the WHO considers the HPV vaccine extremely safe.2 Globally, 95 countries have implemented or will implement national HPV vaccination programs (mostly for females) by the end of 2018.3,4 HPV immunization is recommended before the onset of sexual activity–to develop immunity before the first exposure to the virus–and comprises 2 or 3 doses for those aged 9–14 and >15 (or immunocompromised) respectively.2

Worldwide, completion of the recommended HPV vaccine uptake in females aged 10–20 years is very low (6.1%) and great disparities in uptake exist between countries of high income (32.1%) and low and medium income (0.1–7.2%).3 Geographically, best full-course HPV vaccination coverage in females aged 10–20 years has been achieved in Oceania (notably in Australia), Northern America, and Europe (35.9%, 36.6% and 31.1% respectively). Latin America and the Caribbean (19.0%) and especially Africa (1.2%) and Asia (1.1%) significantly lag behind.3 In Canada, all 10 provinces and 3 territories have implemented gender-neutral, publicly funded, school-based HPV vaccination programs.5 Full-course HPV vaccine uptake in girls and boys in Canada continues to be sub-optimal, as only 3 provinces report uptake rates >80% (Newfoundland & Labrador–89.2% in girls–, Prince Edward Island–82.7% in girls and 81.4% in boys–and Nova Scotia–84.9% in boys–).6-8

Despite existing national school-based programs, HPV vaccine uptake can be jeopardized by parents’ hesitancy towards HPV vaccination for their children. As an example, in the Republic of Ireland, a group called Reactions and Effects of Gardasil Resulting in Extreme Trauma (REGRET), greatly influenced parents’ hesitancy by questioning the safety profile of the HPV vaccine.9 Consequently, vaccine uptake in school-based vaccination programs for girls dropped from 72.3% in the academic year 2015/2016 to 51% one year later.9,10

Rather than a dichotomous behavior (vaccinated/not vaccinated), the WHO SAGE Working Group describes vaccine hesitancy as a continuum refusal process and defines it as a “delay in acceptance or refusal of vaccination despite availability of vaccination services”.11 Using a stage of change-based framework allows for a nuanced perspective on HPV vaccination intentions by examining distinct stages of vaccine intentions. The Precaution Adoption Process Model (PAPM) posits that preventive health behavior can include six nominal intention stages: 1) unaware of the health behavior; 2) unengaged in the decision; 3) undecided; 4) decided not to act; 5) decided to act (intending); and 6) acting (vaccinated).12

Informed by PAPM to measure HPV vaccination intentions in parents of girls and boys, Perez et al. (2017) and Shapiro et al. (2018) found strong associations in cross-sectional studies between parents’ who decided not to vaccinate and perceived benefits of HPV vaccination, cues to action (e.g., recommendation of a healthcare provider (HCP)), harms (e.g., adverse effects related to the vaccine) and perceived susceptibility to HPV infection.13,14 In unengaged or undecided parents, these associations were more inconsistent, suggesting that decided not vaccine hesitant parents have more fixed attitudes and beliefs related to HPV vaccination compared to parents who were unengaged or undecided.13,14 Importantly, Shapiro et al. (2018) found that HPV vaccination policy (i.e., availability of publicly funded school-based vaccination program) has little influence on the decided not (~9% of parents of boys and in parents of girls reported being in this stage) compared to unengaged or undecided hesitant parents (~38% in parents of boys and ~20% in parents of girls for which all jurisdictions had implemented school-based vaccination programs).14 These results suggest that HPV vaccine hesitant parents do not represent a homogenous group and that latent differences between unengaged/undecided and decided not parents exist.

To our knowledge, this is the first study to explore longitudinally knowledge, attitudes, beliefs and intentions related to the HPV vaccine among hesitant parents (objective 1) and to estimate the associations between psychosocial factors (i.e., socio-demographics, attitudes and beliefs) and HPV vaccine intentions in HPV vaccine hesitant parents of boys and girls (objective 2). The goal of this study is to understand better the intricacies of the vaccine hesitancy concept that might lead to more effective targeted interventions.

Results

After data cleaning, our final samples at Time 1 and Time 2 were 3,604 and 1,758 respectively. In parents of girls, 175 parents were unengaged/undecided and 85 were decided not and provided valid answers at both time-points. We included in the analyses 322 and 84 unengaged/undecided and decided not parents of boys respectively.

Psychosocial correlates of unengaged/undecided and decided not stages

At baseline (Time 1), in both parents of girls and boys, those who were unengaged/undecided (compared with decided not) perceived higher susceptibility and severity of HPV infections, more benefits of the HPV vaccine, higher social influence and considered HPV vaccine affordability and accessibility an issue (Table 1). Conversely, parents who were decided not had higher HPV and HPV vaccine knowledge, reported more harms associated with the HPV vaccine, higher conspiracy and hesitancy beliefs and perceived themselves more competent to make health-related decisions for their child (Table 1). In parents of girls, more decided not parents received an HPV vaccine recommendation from a HCP and the mean age of their daughter was one year older than in unengaged/undecided parents. In parents of boys, unengaged/undecided had higher income than decided not parents and a higher proportion reported having two or more children. (See Appendix A).

Table 1.

Comparison of knowledge, attitudes and beliefs of hesitant (unengaged/undecided versus decided not) parents of girls and parents of boys at Time 1.

  Unengaged and undecided n = 175
Decided not n = 85
 
  M (SD) M (SD) T-test (95% CI) *
Parents of Girls
General HPV Knowledge 12.9 (5.3) 15.1 (4.4) −3.38; −0.93
HPV Vaccine Knowledge 5.7 (2.6) 7.3 (1.9) −2.20; −1.08
Susceptibility 4.6 (1.2) 2.7 (1.4) 1.54; 2.24
Severity 5.9 (1.1) 5.3 (1.3) 0.23; 0.89
Benefits 4.6 (0.9) 2.9 (1.0) 1.46; 1.97
Affordability 3.4 (1.4) 2.3 (1.2) 0.72; 1.39
Accessibility 3.0 (1.0) 2.5 (1.2) 0.25; 0.83
Harms 4.3 (1.1) 5.8 (1.0) −1.81; −1.26
Influence 4.2 (0.8) 3.7 (0.9) 0.29; 0.75
Self-Efficacy 5.8 (1.1) 6.5 (0.8) −0.98; −0.52
Conspiracy (VCBS) 3.7 (1.5) 5.0 (1.3) −1.79; −1.09
VHS: Confidence 2.1 (0.7) 2.9 (0.9) −1.04; −0.60
VHS: Risk
3.4 (0.9)
4.1 (0.7)
−0.90; −0.51
 
Unengaged and undecided n = 322
Decided not n = 84
 
Parents of Boys
General HPV Knowledge 13.2 (5.7) 15.6 (5.1) −3.67; −1.16
HPV Vaccine Knowledge 5.9 (2.9) 7.1 (2.5) −1.76; −0.49
Susceptibility 4.6 (1.1) 2.9 (1.3) 1.32; 1.95
Severity 5.8 (1.0) 5.0 (1.5) 0.54; 1.24
Benefits 4.8 (0.9) 3.1 (1.1) 1.46; 1.99
Affordability 4.7 (1.3) 3.8 (1.6) 0.55; 1.32
Accessibility 3.3 (1.1) 3.0 (1.1) 0.09; 0.62
Harms 3.8 (1.2) 5.4 (1.3) −1.91; −1.27
Influence 3.9 (0.9) 3.2 (1.0) 0.52; 1.00
Self-efficacy 5.7 (1.0) 6.3 (0.9) −0.83; −0.39
Conspiracy (VCBS) 3.2 (1.4) 4.5 (1.5) −1.63; −0.91
VHS: Confidence 1.9 (0.6) 2.8 (0.9) −1.05; −0.61
VHS: Risk 3.2 (0.9) 3.8 (0.9) −0.83; −0.40

Note: * Welch two sample t-test; CI = confidence interval; M = mean; SD = standard deviation; VCBS = Vaccine Conspiracy and Beliefs Scale; VHS = Vaccine Hesitancy Scale. HPV = human papillomavirus. In bold significant higher means.

Change of HPV vaccine intention stage over time

From Time 1 to Time 2, 17.7% and 24% of unengaged/undecided parents of girls changed to decided to and vaccinated respectively, while in parents decided not at Time 1, 7.1% and 2.3% changed to decided to and vaccinated respectively (Table 2). More decided not parents of girls at Time 1 did not change their intention stage and remained decided not at Time 2 (70.6%), compared to 44% of unengaged/undecided who remained unengaged/undecided over time (CI: 0.14; 0.39) (Table 2). In parents of boys, over time, 19.6% and 9.9% of those unengaged/undecided changed to decided to and vaccinated respectively while in parents decided not at Time 1, 0% and 1.2% changed to decided to and vaccinated respectively (Table 2). As in parents of girls, more decided not parents of boys at Time 1 did not change their intention stage and remained decided not at Time 2 (70.2%), compared to 59.6% of unengaged/undecided who remained unengaged/undecided over time (CI: −0.005; 0.22) (Table 2).

Table 2.

Changes in PAPM stages of hesitant (unengaged/undecided and decided not) parents of girls and parents of boys from Time 1 to Time 2.

    Unengaged/Undecided Time 1 (n = 175)
  Decided not Time 1 (n = 85)
 
    % (n)   (%) n Test of proportions§ 95% CI
Parents of Girls
Time 2 (n = 175) Unaware* 2.3 (4) Time 2 (n = 85) 0 (0)
  Unengaged 5.7 (10)   1.2 (1) −0.01; 0.09
  Undecided 38.3 (67)   18.8 (16) 0.08; 0.31
  Decided not 12.0 (21)   70.6 (60) −0.70; −0.47
  Decided yes 17.7 (31)   7.1 (6) 0.02; 0.07
 
Vaccinated
24.0 (42)
 
2.3 (2)
0.14; 0.30
 
 
Unengaged/Undecided Time 1 (n = 322)
 
Decided not Time 1 (n = 84)
Test of proportions§ 95% CI
Parents of Boys
Time 2 (n = 322) Unaware* 5.3 (17) Time 2 (n = 84) 2.4 (2) −0.02; 0.08
Unengaged 17.1 (55)   8.3 (7) 0.01; 0.17
Undecided 42.5 (137)   17.9 (15) 0.14; 0.35
Decided not 5.6 (18)   70.2 (59) −0.75; −0.54
Decided yes 19.6 (63)   0 (0)
Vaccinated 9.9 (32)   1.2 (1) 0.04; 0.14

Note: * represents parents’ inaccurate reporting of PAPM stage at Time 2. § represents 2-sample test for equality of proportions with Yates continuity correction. Bold indicates significantly higher proportions. In this table, we present the change in intention stage in parents who responded both at Time 1 and Time 2. Attrition from Time 1 to Time 2 in parents of girls was 187 and 94 for unengaged/undecided and decided not respectively. Attrition from Time 1 to Time 2 in parents of boys was 323 and 68 for unengaged/undecided and decided not respectively.

Change of attitudes and behaviors over time

In parents of boys and parents of girls who changed over time from unengaged/undecided to decided to or vaccinated, we found (at Time 2) significantly increased HPV and HPV vaccine knowledge, increased risks related to the HPV infection, increased benefits of HPV vaccination, increased social influence and self- efficacy and decreased attitudes related to HPV vaccine associated harms, affordability and accessibility as well as decreased vaccine conspiracy and hesitancy beliefs (Table 3). In contrast, in parents of boys and parents of girls who did not change their intention stage over time (i.e., unengaged/undecided or decided not at Time 1 and Time 2), knowledge and attitudes and beliefs remained largely unchanged (Appendix B).

Table 3.

Change of knowledge, attitudes and beliefs in Unengaged/Undecided parents at Time 1 who changed to Decided To or Vaccinated at Time 2.

  Parents of girls
Parents of boys
  Unengaged/Undecided at T1 (n = 175) versus Decided To/Vaccinated at T2 (n = 73)
Unengaged/Undecided at T1 (n = 322) versus Decided To/Vaccinated at T2 (n = 95)
  Mean T1 Mean T2 95% CI Mean T1 Mean T2 95% CI
General HPV knowledge 12.89 16.21 −4.49; −2.13 13.21 15.77 −3.67; −1.45
HPV Vaccine knowledge 5.70 7.77 −2.62; −1.51 5.94 7.31 −1.96; −0.78
Risk 4.64 5.46 −1.12; −0.52 4.62 5.57 −1.19; −0.71
Severity 5.85 6.07 −0.48; 0.04 5.85 6.10 −0.47; −0.03
Benefits 4.63 5.23 −0.85; −0.34 4.81 5.58 −0.97; −0.57
Affordability 3.39 2.37 0.65; 1.40 4.72 3.64 0.70; 1.45
Accessibility 3.03 2.35 0.41; 0.95 3.32 2.73 0.31; 0.88
Harms 4.30 3.19 0.80; 1.43 3.82 3.03 0.50; 3.03
Cues to action (influence) 4.23 5.24 −1.27; −0.75 3.95 5.11 −1.37; −0.95
Self-efficacy 5.77 6.25 −0.71; −0.24 5.73 6.18 −0.65; −0.26
Conspiracy (VCBS) 3.52 2.99 0.15; 0.90 3.24 2.63 0.30; 0.92
VHS: Confidence 2.08 1.91 −0.01; 0.34 1.96 1.66 0.17; 0.43
VHS: Risk 3.43 2.89 0.31; 0.76 3.20 2.91 0.07; 0.50

Note: In bold significant confidence intervals (CI) for 2 sample Welch t-test. At Time 1 we report the number of parents who also provided responses at Time 2. HPV = human papillomavirus

Psychosocial predictors of HPV vaccine intentions

In bivariate analyses of both parents of girls and parents of boys, perception of increased risk of HPV infection, increased benefits of HPV vaccination, higher social influence, self-efficacy related to HPV vaccination and receiving recommendation to vaccinate their child from a HCP were associated with increased odds of vaccine acceptability (i.e., decided to and vaccinated) (Table 4). In contrast, worries related to affordability and accessibility, increased beliefs related to HPV vaccine harms and increased vaccine conspiracy and vaccine hesitancy beliefs were associated with decreased odds of HPV vaccine acceptability (Table 4). Parents of girls reporting an annual family income ≥100,000 Canadian dollars had lower vaccine acceptability compared to those earning <100,000 Canadian dollars (Table 4).

Table 4.

Bivariate logistic regression analysis for boys and girls at Time 2 for decided to/vaccinated versus unengaged/undecided/decided not (reference category).

  Boys (n = 305)
Girls (n = 171)
  OR (CI) OR (CI)
Knowledge, attitudes and beliefs
General HPV Knowledge 1.03 (0.98; 1.09) 1.14 (1.06; 1.23)
HPV Vaccine knowledge 1.11 (0.99; 1.24) 1.19 (1.03; 1.39)
Susceptibility 2.66 (2.00; 3.54) 2.55 (1.81; 3.58)
Severity 1.36 (1.03; 1.80) 1.22 (0.87; 1.70)
Benefits 2.98 (2.15; 4.13) 2.76 (1.83; 4.15)
Affordability 0.81 (0.69; 0.95) 0.66 (0.53; 0.84)
Accessibility 0.64 (0.51; 0.81) 0.62 (0.46; 0.84)
Harms 0.52 (0.41; 0.64) 0.44 (0.32; 0.59)
Influence 4.41 (3.00; 6.47) 3.79 (2.43; 5.90)
Self-Efficacy 1.77 (1.33; 2.34) 2.30 (1.56; 3.39)
VCBS 0.70 (0.58; 0.84) 0.70 (0.56; 0.87)
VHS: Confidence 0.33 (0.21; 0.52) 0.49 (0.30; 0.81)
VHS: Risk 0.69 (0.52; 0.90) 0.42 (0.28; 0.63)
Vaccine policy change (Ontario, Quebec, Manitoba)
Vaccine policy no change (reference) (reference)
change 0.82 (0.49; 1.39) 0.77 (0.41; 1.44)
Sociodemographics
Parents’ age 0.99 (0.96; 1.03) 0.95 (0.91; 1.01)
Age of the child 0.92 (0.83; 1.03) 0.91 (0.79; 1.04)
Education Elementary/high school (reference) (reference)
University 1.36 (0.63; 2.92) 1.34 (0.62; 2.87)
Number of children One child (reference) (reference)
Two children 1.14 (0.62; 2.10) 1.04 (0.50; 2.20)
Three or more children 1.09 (0.56; 2.12) 1.21 (0.53; 2.80)
Parents’ gender male (reference) (reference)
female 1.19 (0.72; 1.96) 0.91 (0.48; 1.70)
Income<100K (reference) (reference)
≥100K 1.58 (0.94; 2.65) 0.47 (0.23; 0.93)
Prefer not to answer 1.93 (0.83; 4.49) 0.81 (0.28; 2.34)
Ethnicity Other (reference) (reference)
White 0.55 (0.29; 1.05) 0.81 (0.38; 1.76)
HCP recommendation No (reference) (reference)
Yes 4.05 (2.07; 7.96) 2.19 (1.07; 4.50)

Note: OR = adjusted odds ratio for decided to/vaccinated versus unengaged/undecided/decided not (reference category). For knowledge, attitudes and beliefs OR is reported for one-unit increase. In bold, significant OR. HPV = human papillomavirus

In multivariate analysis of unengaged/undecided hesitant parents (Time 1) who changed to vaccine acceptors (decided to/vaccinated) or remained hesitant (unengaged/undecided or decided not) at Time 2, higher social influence (e.g., family, friends, HCPs) was associated with increased odds of vaccine acceptability in parents of girls (OR = 2.91, CI: 1.50; 5.65) and parents of boys (OR = 2.89, CI: 1.77; 4.74). Increased beliefs related to HPV vaccine harms (e.g., vaccine is unsafe, insufficient research done) was associated with decreased odds of vaccine acceptability in parents of girls (OR = 0.47, CI: 0.25; 0.88) and parents of boys (OR = 0.63, CI: 0.40; 1.01). In parents of girls, for one unit increase in HPV knowledge we found 21% increased odds of vaccine acceptability (OR = 1.21, CI: 1.05; 1.39). In parents of boys, being white (compared to other ethnicities) was associated with lower odds of HPV vaccine acceptability (OR = 0.39, CI: 0.17; 0.90) (Table 5). Parents of girls earning ≥100,000 Canadian dollars had lower odds of accepting the HPV vaccine (OR = 0.36, CI: 0.13; 0.99) (Table 5).

Table 5.

Multivariate logistic regression analysis for boys and girls at Time 2 for decided to/vaccinated versus unengaged/undecided/decided not (reference category).

  Boys (n = 305)
Girls (n = 171)
  AOR (CI) AOR (CI)
General HPV Knowledge 0.99 (0.91; 1.10) 1.21 (1.05; 1.39)
HPV Vaccine knowledge 1.00 (0.83; 1.19) 1.13 (0.84; 1.53)
Susceptibility 1.46 (0.93; 2.30) 1.39 (0.76; 2.55)
Severity 0.67 (0.43; 1.05) 1.22 (0.68; 2.19)
Benefits 1.14 (0.63; 2.06) 1.11 (0.44; 2.80)
Affordability 0.89 (0.70; 1.13) 0.70 (0.47; 1.03)
Accessibility 1.09 (0.76; 1.56) 1.62 (0.94; 2.78)
Harms 0.63 (0.40; 1.01) 0.47 (0.25; 0.88)
Influence 2.89 (1.77; 4.74) 2.91 (1.50; 5.65)
Self-efficacy 1.29 (0.86; 1.93) 1.48 (0.81; 2.72)
Conspiracy (VCBS) 1.09 (0.73; 1.61) 1.51 (0.86; 2.65)
Hesitancy confidence 0.80 (0.36; 1.77) 2.11 (0.75; 5.91)
Hesitancy risk 1.32 (0.80; 2.24) 0.58 (0.26; 1.29)
Vaccine policy no change (reference) (reference)
change 0.96 (0.48; 1.92) 0.83 (0.31; 2.23)
HCP recommendation No (reference) (reference)
Yes 2.23 (0.91; 5.46) 1.47 (0.53; 4.08)
Income <100K (reference) (reference)
≥100K 1.47 (0.75; 2.90) 0.36 (0.13; 0.99)
Prefer not to answer 1.68 (0.56; 5.05) 0.76 (0.13; 4.35)
Ethnicity Other (reference) (reference)
White 0.39 (0.17; 0.90) 0.50 (0.15; 1.68)

Note: HPV = human papillomavirus. AOR: adjusted odds ratio for decided to/vaccinated versus unengaged/undecided/decided not (reference category). For knowledge, attitudes and beliefs AOR is reported for one-unit increase. In bold, significant AOR. Final model fit diagnostics for boys: BIC = 363, goodness of fit (Cessie van Houwelingen) p = 0.95, VIF<3.06, C = 0.87. Model fit diagnostics with all variables from bivariate for boys: BIC = 395, goodness of fit (Cessie van Houwelingen) p = 0.71, VIF<3.30, C = 0.87. Final model fit diagnostics for girls: BIC = 229, goodness of fit (Cessie van Houwelingen) p = 0.64, VIF<3.85, C = 0.90. Model fit diagnostics with all variables from bivariate for girls: BIC = 252, goodness of fit (Cessie van Houwelingen) p = 0.34, VIF<4.47, C = 0.91

Discussion

Informed by PAPM, HPV vaccine hesitant parents (i.e., those who have not vaccinated their child against HPV) were categorized into unengaged (i.e., have not thought about vaccinating their child), undecided (about vaccinating their child against HPV), and parents who decided not to vaccinate (Figure 1). This categorization is consistent with the WHO SAGE definition of vaccine hesitancy as a multifaceted rather than a dichotomous (i.e., vaccinated/not vaccinated) decision. PAPM facilitates a better understanding of vaccine hesitancy by excluding parents who are not even aware that the HPV vaccine can be given to their child. Previous research has shown that in the absence (or incipient stages) of an HPV vaccination program for boys, the unaware group accounts for 28–58% of a randomly selected sample of parents13,14 and have significantly lower HPV knowledge compared to unengaged, undecided and decided not hesitant parents.14

Figure 1.

Figure 1.

Precaution Adoption Process Model (PAPM) vaccination intention stages.

To better understand the longitudinal decision of vaccine hesitant parents we dichotomized vaccine hesitant parents–in parents of boys and girls–as two distinct entities: “flexible” hesitant–corresponding to vaccine intention stages unengaged/undecided–and “rigid” hesitant, corresponding to the intention stage decided not. We suggest these two entities based on existing differences between “flexible” and “rigid” hesitant parents related to their intentions to give the HPV vaccine to their children and differences in parents’ knowledge, attitudes and behaviors related to the HPV vaccine. This study found that significantly more “flexible” hesitant parents changed to HPV vaccine acceptors over time, compared to “rigid” hesitant who remained unchanged. Moreover, we found an important change from Time 1 to Time 2 in HPV and HPV vaccine knowledge and attitudes and beliefs (e.g., increased benefits of HPV vaccination, increased social influence, decreased harms) in “flexible” hesitant compared to “rigid” hesitant parents. In addition to the objectives of this study, we combined data from this study with data collected by Perez et al. (February 2014)15 and found–on a total sample of 6,721 Canadian parents of boys and girls–that “flexible” hesitant are about 3.6 times more numerous than “rigid” hesitant (29% versus 8% respectively). Specific messaging targeting ‘flexible’ hesitant parents and designed to address their concerns (based on our results) should be integrated into educational materials that are distributed to parents at the time of implementation in either school, clinic or physician-based programs.

In “flexible” hesitant parents, we found that increased influence from family, friends and HCP related to HPV vaccination (parents of girls and parents of boys), increased HPV vaccine knowledge (parents of girls) and decreased perceptions of harms related to the HPV vaccine (parents of girls) are positively related to HPV vaccine acceptability (i.e., decided to or vaccinated). Moreover, we conducted a sensitivity analysis on a combined sample of 418 parents of girls and parents of boys at Time 1 who received a HCP recommendation related to the HPV vaccine (See Appendix C) and found that the strength of a HCP recommendation was positively associated with HPV vaccine acceptability (OR = 1.69, 95% CI; 1.21–2.36). In their systematic review and meta-analysis, Newman et al. (2018) found additional parent factors associated with HPV vaccine uptake to include perceived HPV vaccine benefits, affordability (i.e. HPV vaccine covered by insurance), and child’s age.16 These differences could be explained by including in our analyses only parents who changed from “flexible” hesitant at Time 1 and by defining the outcome (i.e., decided to or vaccinated versus vaccine hesitant) informed by a multi-stage intention model (i.e., PAPM), as opposed to Newman et al. (2018) who defined their outcome binomially (vaccinated/not vaccinated).16

Importantly, the natural experiment represented by the introduction of a free, school-based vaccination program for boys in Ontario, Quebec and Manitoba from Time 1 to Time 2 was not significantly associated with vaccine acceptability in “flexible” hesitant parents. In our opinion, most parents became aware of the new program (as parental consent is required for minors); however, it is possible that key aspects (e.g., vaccine safety) were not covered in the information provided to parents, or that the information provided–consistent with results published by Tulsieram et al. (2018) who analyzed the readability and coherence of seven Canadian provincial ministry of health’s HPV information websites–was not adequately tailored to the level of understanding of lay population17 . Our results show that tailored information for “flexible” hesitant parents should address the excellent safety profile of the vaccine, the need to consult a HCP and discuss with other members form the community who have vaccinated their children.

Our study is not without limitations. First, HPV vaccination status is based solely on parents’ self-reports of vaccination status as HPV vaccination registries are in various stages of implementation across Canada. We consider that by including in the definition of vaccine acceptability both actual uptake and parents’ decision to vaccinate (i.e., decided to stage), we have partially circumvented the recall bias. Second, we were limited by the small number of “rigid” hesitant parents who changed over time to decided to or vaccinated (n = 8 and n = 1 in parents of girls and parents of boys respectively) and could not conduct multivariate analyses in this group of hesitant parents. Third, the applicability of our results could have been affected by an important attrition rate. Nevertheless, the negligible effect size of the differences in sociodemographics, knowledge, attitudes and beliefs between parents who responded at Time 2 and those who were lost to follow-up indicate that the two groups were almost identical in respect to the variables analyzed (See appendices D and E). Lastly, our study was conducted in a high-income country with a well-developed healthcare system and efficient vaccination programs, making our results likely less generalizable to low or medium income countries and different healthcare systems.

Methods

Data collection

We used a web survey and a cross-sectional design to collect data from a national representative sample of Canadian parents of 9–16 years old boys and girls. Details of the study methodology are presented in detail elsewhere.18 Briefly, parents were recruited by Canada’s largest market research and polling firm, Leger–The Research Intelligence Group–from its national panel of over 400 000 members which is built to be nationally and regionally representative with respect to gender, age, education level, household composition and income. Collecting data from a representative sample of Canadian parents was ensured by using Leger’s proprietary software–designed in concordance to Canada’s census data–to generate the initial sample pool and continuously adjust the composition of the target groups during the recruitment process. Data collection was performed from August-September 2016 (Time 1) and parents who participated at Time 1 were invited to participate in June-July 2017 (Time 2). At Time 1, all Canadian jurisdictions and 3 provinces (Alberta, Nova Scotia and Prince Edward Island) had in place publicly, school-based vaccination programs for girls and boys respectively. Between Time 1 and Time 2, three provinces (Ontario, Manitoba and Quebec) initiated free, school-based HPV vaccination programs for boys. The study received approval by the Integrated Health and Social Services University Network for West-Central Montreal (CODIM-FLP-16–219).

Measures

We used the PAPM to situate parents’ stage of intention to vaccinate their [child] with the HPV vaccine.12 Informed by PAPM, parents selected one of 6 response options: “I was unaware that the HPV vaccine could be given to [child]” (stage 1, unaware), “I have never thought about vaccinating [child] against HPV” (stage 2, unengaged), “I am undecided about vaccinating [child] against HPV” (stage 3, undecided), “I have decided I do not want to vaccinate [child] against HPV” (stage 4, decided not), “I have decided I do want to vaccinate [child] against HPV” (stage 5, decided to) and “[child] has already received the HPV vaccine” (stage 6, vaccinated). Parents who were unengaged, undecided or decided not to vaccinate, were considered HPV vaccine hesitant. As opposed to measuring intentions dichotomously (vaccinated/not vaccinated), the PAPM allows for HPV vaccine hesitancy to be defined more precisely, by excluding parents who were unaware of the HPV vaccine (as these parents did not have the opportunity to make an informed decision) and parents who had decided to vaccinate (and were obviously no longer hesitant) (Figure 1).

Correlates of parents’ HPV vaccine hesitancy included HPV and HPV vaccine knowledge, attitudes and beliefs1 related to the HPV vaccine, sociodemographics, HPV vaccination policy and behaviors. HPV knowledge and HPV vaccine knowledge were measured with validated scales; the HPV knowledge scale had an internal consistency of α = 0.90 (23 items, e.g., “HPV can be passed on during sexual intercourse”) and HPV Vaccine knowledge scale of α = 0.78 (11 items, e.g., “The HPV vaccines offer protection against all sexually transmitted infections”).19 Total knowledge scores were calculated by assigning 1 point to correct answers and zero points for incorrect or ‘Don’t know’ answers. We used the validated HPV Attitudes and Beliefs Scale (HABS)20 to measure the following constructs: susceptibility (risk) (3 items, α = 0.92, e.g., “Without the HPV vaccine, my [child] would be at risk of getting HPV later in life), severity (threat) of HPV infection and associated diseases (3 items, α = 0.84, e.g., “It would be serious if my [child] contracted HPV later in life”), benefits of HPV vaccination (10 items, α = ,0.94 e.g., “The HPV vaccine is effective in preventing HPV”), affordability (3 items, α = ,0.87 e.g., “The HPV vaccine costs more than I can afford”), accessibility (4 items, α = 0.79, e.g., “Dealing with getting the HPV vaccine for my [child] would be simple”), harms related to the HPV vaccine (6 items, α = 0.93, e.g., “The HPV vaccine is unsafe”) and social influence (8 items, α = 0.91, e.g., “Other parents in my community are getting their [child] the HPV vaccine”). Self-efficacy was measured with 4 items (α = 0.89 e.g., “I am competent to make decisions about the vaccines [child] receives”. We measured vaccine conspiracy with the Vaccine Conspiracy Beliefs Scale (VCBS) (7 items, α = 0.95, e.g., “Vaccine safety data is often fabricated.”)21 and vaccine hesitancy by using the Vaccine Hesitancy scale (VHS) which consists of two subscales: confidence (7 items, α = 0.92, e.g., “Childhood vaccines are important for my child’s health”) and risks (7 items, α = 0.64, e.g., “New vaccines carry more risks than older vaccines”).22 Vaccine attitudes were measured on a seven-point Likert-type rating scales ranging from ‘1-strongly disagree’ to ‘7-strongly agree’ and average scores for each sub-scale were calculated.

Sociodemographics included continuous variables (i.e., parent’s and child’s age) and categorical variables (i.e., education, number of children, parent’s gender, income and ethnicity). HPV vaccine policy change encompass provinces who implemented free HPV vaccination programs for boys from Time 1 to Time 2 (i.e., Ontario, Quebec and Manitoba) versus provinces with no change in policy. Health behaviors comprise a binomial variable i.e., receiving versus not receiving a HPV vaccine recommendation from a healthcare provider (HCP).

Data analysis

We used statistical data cleaning methods (i.e., psychometric synonyms and bogus items) to flag inattentive or unmotivated responders.18 Responders flagged at either Time 1 or Time 2 were excluded from the analyses.

Informed by PAPM, we dichotomized hesitant parents into: unengaged/undecided and decided not as we consider that both unengaged and undecided parents have not formed a definitive intention (i.e., they are in more initial stages of decision-making) as opposed to parents who have clearly decided not to give the HPV vaccine to their child. We conducted analyses separately for parents of boys and parents of girls and for each group of hesitant parents separately and included only parents who provided responses at both Time 1 and Time 2.

Consistent with objective 1 we explored 1) differences in psychosocial correlates between unengaged/undecided and decided not at Time 1, 2) the change of HPV vaccine intention stage from Time 1 to Time 2 for the two groups of hesitant parents and 3) the change of attitudes and behaviors over time for each group of hesitant parents. For categorical variables, we used two-sample test of proportions and reported 95% confidence intervals (CI) to highlight significant differences. For continuous variables (e.g., attitudes, knowledge) we used the Welch two sample t-test and reported the 95% CI for differences in means.

To estimate the effect of psychosocial predictors (i.e., socio-demographics, attitudes and beliefs) on HPV vaccine intentions in HPV vaccine hesitant parents of boys and girls (objective 2), we included only parents who were unengaged or undecided at Time 1 and used binomial logistic regression to analyze the Time 2 data. The outcome variable included two categories of parents: acceptors of the HPV vaccine (i.e., decided to or vaccinated) and HPV vaccine hesitant (unengaged/undecided or decided not). For nominal predictors, we report the odds ratio (OR) and 95% CI of accepting the HPV vaccine (versus HPV vaccine hesitant) for each category versus the reference category (e.g., female versus male). For continuous predictors (e.g., attitudes), we report the change (OR) and 95% CI represented by a one-unit score increase. First, we conducted bivariate analyses between each predictor and the outcome. Then, we ran the multivariate model (final model) with predictors significantly associated with the outcome in bivariate analyses and predictor variables of interest i.e., policy change, ethnicity.14 We used following logistic regression model diagnostic criteria1: Rank Discrimination Index C where C = 0.5 indicates random guessing and C = 1 perfect discrimination 2) Variation Inflation Factor (VIF) with a cut-off value of <10 to flag multicollinearity23 and 3) Cessie–van Houwelingen goodness-of-fit test whereby p > 0.05 suggests no evidence to reject a good fit.24 We calculated the Bayesian Information Criterion (BIC)25 for the regression model including all variables and the final model and retained the model with the lowest BIC value. Analyses were performed with R 3.4.3 for Windows.

Conclusion

In a Canadian sample of parents of 9–16-year-old boys and girls, we have shown that HPV vaccine hesitancy is not a homogenous entity and consists of those that are “flexible” hesitant who are more likely to later accept HPV vaccination, and those that are “rigid” hesitant, who tend to remain unchanged over time. Different interventions are needed for these two groups. For HPV vaccine “flexible” hesitant parents, interventions should target existing barriers of HPV acceptability such as increased beliefs of harms related to HPV vaccine, poor communication with friends, family and healthcare providers about HPV vaccination and low HPV knowledge. Future research is needed to evaluate the effect of psychosocial factors on HPV vaccine acceptability in “rigid” hesitant parents, and what interventions are most appropriate for this group.

Appendix A.

Sociodemographics of hesitant (unengaged/undecided and decided not) parents of girls and parents of boys at Time 1

Parents of Girls
 
Unengaged and undecided
n=175
Decided not
n=85
 
  n (%) n (%) Test of proportions (95% CI) **
Education      
Elementary/high school 35 (20.0) 13 (15.3) -0.05; 0.14
University 140 (80.0) 72 (84.7) -0.14; 0.05
Number of children      
One child 44 (25.2) 17 (20.0) -0.06; 0.16
Two children 83 (47.4) 44 (51.8) -0.17; 0.09
Three or more children 48 (27.4) 24 (28.2) -0.12; 0.11
Parent’s gender      
Male 64 (36.6) 23 (27.1) -0.02; 0.21
Female 111 (63.4) 62 (72.9) -0.21; 0.02
Income      
<100.000 104 (59.4) 42 (49.4) -0.03; 0.23
≥100.000 55 (31.4) 27 (31.8) -0.12; 0.12
Prefer not to answer 16 (9.2) 16 (18.8) -0.19; -0.003
Ethnicity      
White 140 (80.0) 73 (85.9) -0.15; 0.04
Other 35 (20.0) 12 (14.1) -0.04; 0.15
HCP recommendation-yes 13 (7.4) 18 (21.2) -0.23; -0.04
HCP recommendation-no 162 (92.6) 67 (78.8) 0.04; 0.23
  M (SD) M (SD) T-test (95%CI) *
Parent’s age 42.5 (6.9) 43.3 (6.5) -1.96; 1.96
Age of the child 11.4 (2.3) 12.4 (2.3) -1.58; -0.38
Parents of boys
 
Unengaged and undecided
n=322
Decided not
n=84
 
  n (%) n (%) Test of proportions (95% CI) **
Education      
Elementary/high school 42 (13.1) 19 (22.6) -0.19; 0.001
University 280 (86.9) 65 (77.4) -0.001; 0.19
Number of children      
One child 76 (23.6) 21 (25.0) -0.12; 0.09
Two children 150 (46.6) 29 (34.5) 0.01; 0.24
Three or more children 96 (29.8) 34 (40.5) -0.22; 0.01
Parent’s gender      
Male 128 (39.8) 29 (34.5) -0.06; 0.17
Female 194 (60.2) 55 (65.5) -0.17; 0.06
Income      
<100.000 168 (52.2) 51 (60.7) -0.20; 0.03
≥100.000 121 (37.6) 21 (25.0) 0.02; 0.23
Prefer not to answer 33 (10.2) 12 (14.3) -0.12; 0.04
Ethnicity      
White 273 (84.8) 71 (84.5) -0.08; 0.09
Other 49 (15.2) 13 (15.5) -0.09; 0.08
HCP recommendation-yes 16 (5.0) 3 (3.6) -0.03; 0.06
HCP recommendation-no 306 (95.0 81 (96.4) -0.06; 0.03
  M (SD) M (SD) T-test (95%CI)
Parent’s age 44.8 (7.2) 43.3 (7.5) -0.29; 3.33
Age of the child 13 (2.3) 12.6 (2.4) -0.21; 0.92

Note: * Welch two sample t-test; ** 2-sample test for equality of proportions; CI = confidence interval; M = mean; SD = standard deviation; VCBS = Vaccine Conspiracy and Beliefs Scale; In bold significant higher means or proportions. HCP= healthcare provider. Included are parents who responded at Time 1 and Time 2

Appendix B.

Knowledge, attitudes and behaviors in Decided not and Unengaged/Undecided parents who did not change their vaccination intention stage from Time 1 to Time 2

  Parents of Girls
Parents of Boys
Decided NOT at Time 1
(n = 85)
versus
Decided NOT at Time 2
(n = 60)
Unengaged/Undecided at Time 1
(n = 175)
versus
Unengaged /Undecided at Time 2
(n = 77)
Decided NOT at Time 1
(n = 84)
versus
Decided NOT at Time 2
(n = 59)
Unengaged/Undecided at Time 1
(n = 322)
versus
Unengaged OR Undecided at Time 2
(n = 192)
Mean T1 Mean T2 95% CI Mean T1 Mean T2 95% CI Mean T1 Mean T2 95% CI Mean T1 Mean T2 95% CI
General HPV knowledge 15.05 16.55 -2.91; -0.09 12.89 13.36 -1.80; 0.86 15.62 16.80 -2.65; 0.29 13.21 14.79 -2.51; -0.65
HPV Vaccine knowledge 7.34 7.53 -0.87; 0.49 5.70 6.62 -1.63; -0.21 7.06 7.48 -1.17; 0.34 5.94 6.73 -1.25; -0.34
Risk 2.75 3.12 -0.85; 011 4.64 4.48 -0.15; 0.47 2.98 3.06 -0.53; 0.38 4.62 4.65 -0.21; 0.16
Severity 5.29 5.28 -0.44; 0.43 5.85 5.93 -0.36; 0.21 4.96 4.95 -0.48; 0.50 5.85 5.86 -0.18; 0.16
Benefits 2.92 2.85 -0.28; 0.42 4.63 4.67 -0.25; 0.18 3.09 2.89 -0.19; 0.60 4.81 4.82 -0.16; 0.15
Affordability 2.34 2.53 -0.62; 0.24 3.39 3.33 -0.32; 0.43 3.78 3.05 0.21; 1.25 4.72 4.25 0.20; 0.72
Accessibility 2.49 2.51 -0.42; 0.39 3.03 2.98 -0.26; 0.36 2.96 2.95 -0.36; 0.37 3.32 3.29 -0.15; 0.22
Harms 5.84 6.03 -0.51; 0.14 4.30 4.23 -0.26; 0.40 5.41 5.62 -0.63; 0.20 3.82 3.91 -0.29; 0.12
Cues to action (influence) 3.71 3.73 -0.32; 0.28 4.23 4.17 -0.18; 0.30 3.19 3.28 -0.41; 0.23 3.95 4.01 -0.21; 0.10
Self-efficacy 6.52 6.30 -0.11; 0.55 5.77 5.41 0.06; 0.66 6.34 6.28 -0.29; 0.39 5.73 5.62 -0.08; 0.30
Conspiracy 4.95 5.06 -0.52; 0.30 3.52 3.54 -0.40; 0.35 4.50 4.59 -0.60; 0.43 3.24 3.26 -0.27; 0.23
Hesitancy (confidence) 2.90 2.99 -0.40; 0.20 2.08 2.13 -0.22; 0.12 2.79 2.78 -0.33; 0.34 1.96 2.03 -0.19; 0.04
Hesitancy (risks) 4.13 3.98 -0.11; 0.41 3.42 3.43 -0.24; 0.23 3.81 3.92 -0.39; 0.16 3.20 3.16 -0.13; 0.19

Note: In bold significant confidence intervals (CI) for 2 sample Welch t-test. At Time 1 we report the number of parents who also provided responses at Time 2. HPV = human papillomavirus

Appendix C.

Multivariate logistic regression analysis at Time 1 for all parents who received a HCP recommendation related to the HPV vaccine (n=418). Outcome is decided to/vaccinated versus unengaged/undecided/decided NOT (reference category).

  All parents at Time 1
AOR (95% CI)
General Knowledge 0.93 (0.86; 1.02)
Benefits 1.97 (1.13; 3.42)
Harms 0.45 (0.32; 0.64)
Influence 2.22 (1.35; 3.66)
Strength of HCP recommendation 1.69 (1.21; 2.36)
Childs’ gender
male
(reference)
female 5.91 (2.43; 14.40)
Income
<100K
(reference)
≥100K 1.86 (0.79; 4.37)
Prefer not to answer 1.39 (0.44; 4.43)
Ethnicity
Other
(reference)
White 0.39 (0.14; 1.07)

Note:

HCP = healthcare provider AOR = adjusted odds ratio for decided to/vaccinated versus unengaged/undecided/decided not (reference category). For knowledge, attitudes/beliefs and strength of HCP recommendation AOR is reported for one-unit increase. In bold, significant AOR.

Benefits were measured with a scale of 10 items, the harms scale included 6 items and influence 8 items. General HPV knowledge was measured with 23 items. Benefits (e.g., “The HPV vaccine has many benefits”), harms (e.g., “the HPV vaccine is unsafe”) and influence (e.g., “My family thinks it is a good idea to vaccinate my child against HPV”) were measured on a 7-point Likert scale where 1= strongly disagree and 7= strongly agree. HCP’s strength of recommendation was measured on a 5-point Likert scale where 1= the recommendation does not influence at all the decision to vaccinate their son/daughter and 5 = the recommendation influenced a lot. Model fit diagnostics: BIC=252, goodness of fit (Cessie van Houwelingen) p=0.27, VIF<1.82, C=0.94.

Appendix D.

Differences in sociodemographics between parents who participated at Time 2 and those who did not participate at Time 2

 
Participated at Time 2 (n=1758)
Did NOT participate at Time 2 (n=1846)
 
 
  n (%) n (%) Test of proportions (95% CI) Effect size (Cohen’s h)
Education        
Elementary/high school 305 (17.3) 329 (17.8) -0.03; 0.02 -0.01
University 1453 (82.7) 1517 (82.2) -0.02; 0.03 0.01
Number of children        
One child 411 (23.4) 409 (22.2) -0.02; 0.04 0.03
Two children 820 (46.6) 849 (46.0) -0.03; 0.04 0.01
Three or more children 527 (30.0) 588 (31.8) -0.05; 0.01 -0.04
Parent’s gender        
Male 597 (34.0) 627 (34.0) -0.03; 0.03 -0.0001
Female 1161 (66.0) 1219 (66.0) -0.03; 0.03 0.0001
Income        
<100.000 907 (51.6) 962 (52.1) -0.04; 0.03 -0.01
≥100.000 674 (38.3) 680 (36.8) -0.02; 0.05 0.03
Prefer not to answer 177 (10.1) 204 (11.1) -0.03; 0.01 -0.03
Ethnicity        
White 1467 (83.4) 1547 (83.8) -0.03; 0.02 -0.01
Other 291 (16.6) 299 (16.2) -0.02; 0.03 0.01
HCP recommendation-yes 216 (12.3) 204 (11.1) -0.01; -0.03 0.04
HCP recommendation-no 1542 (87.7) 1642 (88.9) -0.03; 0.01 -0.04
  Mean Mean T-test (95%CI)  
Parent’s age 44.08 42.98 0.64; 1.54 0.16
Age of the child 12.62 12.54 -0.07; 0.23 0.03

Note: In bold significant higher proportions or means. Effect size values <0.2 are considered small and the difference between proportions or means trivial even if statistically significant (two-sample test of proportions or Welch two sample t-test).

Appendix E.

Differences in knowledge, attitudes and beliefs between parents who participated at Time 2 and those who did not participate at Time 2

 
Participated at Time 2 (n=1758)
Did NOT participate at Time 2 (n=1846)
 
 
  Mean Mean T-test (95% CI) Effect size (Cohen’s d)
General HPV Knowledge 13.34 12.51 0.44; 1.22 0.14
HPV Vaccine Knowledge 6.30 5.84 0.27; 0.65 0.16
Susceptibility 5.01 4.90 0.01; 0.20 0.08
Severity 5.95 5.93 -0.055; 0.09 0.02
Benefits 4.97 4.88 0.01; 0.16 0.08
Affordability 3.57 3.60 -0.14; 0.08 -0.02
Accessibility 2.77 2.86 -0.16; -0.01 -0.08
Harms 3.49 3.53 -0.14; 0.05 -0.03
Influence 4.69 4.59 0.02; 0.17 0.08
Self-Efficacy 6.07 6.00 0.01; 0.14 0.08
VCBS 3.14 3.25 -0.20; -0.01 -0.07
VHS: Confidence 1.93 2.01 -0.12; -0.03 -0.10
VHS: Risk 3.05 3.06 -0.07; 0.05 -0.01

Note: In bold significant higher means. Effect size values <0.2 are considered small and the difference between means trivial even if statistically significant (Welch two sample T-test).

Funding Statement

This work was supported by the Canadian Cancer Society Research Institute (CCSRI) [704036].

Note

1.

Internal consistency (Cronbach’s α) for all included scales was calculated at Time 1, n = 3604.

Disclosure of potential conflicts of interest

No potential conflict of interest was reported by the authors.

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Citations

  1. World Health Organization Human papillomavirus vaccines: WHO position paper, May 2017; 2017. May 12 [accessed 2019 Jan 7]. http://apps.who.int/iris/bitstream/handle/10665/255353/WER9219.pdf?sequence=1.

Articles from Human Vaccines & Immunotherapeutics are provided here courtesy of Taylor & Francis

RESOURCES