Applying the Hornik & Woolf Approach to Identify Messaging Themes and Improve COVID-19 Vaccine Confidence among Federally Qualified Health Centers’ Workforce in Wisconsin

Abstract

The Federally Qualified Health Centers (FQHCs) in Wisconsin serve the most under-resourced communities in the state and are trusted sources in local communities. Although healthcare workers can be leveraged to champion COVID-19 vaccines, existing vaccine hesitancy among the FQHC workforce itself calls for research to identify promising messaging themes that can boost their vaccine confidence. In spring 2021, through a partnership with the Wisconsin Primary Health Association, we took a community-engaged approach to develop and field a survey including 46 beliefs (Ms = 1.36–4.25, SDs = 0.81–1.46, all on 5-point Likert scales) among employees of 10 out of the 17 FQHCs in Wisconsin. In total, 347 clinical team members and 349 non-clinical staff members (a) indicated their levels of (dis)agreement with all 46 belief items and (b) reported vaccine acceptance (dichotomized, acceptance = 77.6%) as well as recommendation intentions (dichotomized, yes = 73.9%). We carried out the Hornik & Woolf analyses in a multilevel logistic regression framework with bootstrapping to rank order all beliefs, segmented by subgroup and behavioral outcome. Our results suggest that communication-based interventions should promote beliefs such as those related to perceived safety and efficacy, rather than normative pressure from peers, while aiming to reduce doubts about information withholding and manipulation, the safety of the mRNA technology, the approval process, and “unnatural” ingredients in the vaccines. Subgroup-specific belief rankings are also provided. This study demonstrates the usefulness of incorporating the H&W approach into community-engaged research with local healthcare systems to improve health messaging for vaccine promotion.

Introduction

Vaccine hesitancy remains a major barrier to public health efforts in containing and eventually ending the COVID-19 pandemic in the United States. Unlike previous forms of resistance against highly efficacious and safe vaccines (e.g., the MMR and the HPV vaccines), COVID-19 vaccine hesitancy is marked by deepening distrust in the very institutions responsible for vaccine messaging and rollout including governments, public health officials, and media (Hudson & Montelpare, 2021; Tram et al., 2021). Pockets of hesitancy are more likely to be found among racial and ethnic minority groups, lower-income households, and in rural areas (Kelly et al., 2021; Robinson et al., 2021; Salomon et al., 2021). Once surfaced and exacerbated by the pandemic, this “trust gap”, particularly among under-resourced populations, is likely to persist and undermine vaccine messaging even after COVID-19 is eventually brought under control. We argue that a localized vaccine messaging approach that recruits clinical staff serving the local community as vaccine ambassadors is promising in filling this trust gap and reducing vaccination disparities. That said, an urgent issue that needs to be tackled first is to address existing hesitancy among clinical team members and non-clinical staff themselves (Browne et al., 2021; Li et al., 2021; Meyer et al., 2021). This report demonstrates the usefulness of employing the Hornik and Woolf (Hornik & Woolf; 1999; Hornik et al., 2019) approach (hereafter, H&W) to identify messaging themes that are promising in boosting vaccine confidence among the workforce of the Federally Qualified Health Centers (FQHCs) serving under-resourced communities in the state of Wisconsin.

Since the pandemic began, FQHCs have been tasked with COVID-19 testing and vaccine distribution for the state’s most under-resourced populations, including racial and ethnic minorities and low socioeconomic groups residing in Wisconsin’s rural areas. They are the gatekeepers for community members’ access to health information related to COVID-19 vaccines. Compared with other messengers such as government representatives, scientists unfamiliar to the local community, or even doctors serving other communities, local community members’ own doctors and nurses are among the most trusted sources of health information.^11,12 Survey data showed that community members’ willingness to take COVID-19 vaccines is increased with a provider recommendation (Head et al., 2020; Reiter et al., 2020), underscoring the potential persuasive power of providers in patients’ vaccine decision-making (Reiter et al., 2020; Rahman et al., 2015; Shay et al., 2018; Solís Arce et al., 2021). In rural areas where trust in institutional authorities has declined more rapidly than in their urban counterparts (Cramer, 2016), leveraging trustable healthcare clinical team members to champion vaccines represents a promising way to help close the gap in rural-urban vaccination rates. Despite this potential, the proportion of healthcare workers hesitating to take the vaccines for themselves was comparable to the general public (Browne et al., 2021; Li et al., 2021; Meyer et al., 2021). Hesitancy is higher among healthcare workers who are supporting staff, female, younger, belonging to racial and ethnic minorities, and living in rural areas (Browne et al., 2021; Li et al., 2021; Meyer et al., 2021; Shekhar et al., 2021). Therefore, it is necessary to develop targeted communication interventions to improve healthcare workers’ vaccine acceptance and willingness to recommend to undecided community members.

Although many studies have examined messaging themes and strategies to nudge the public to take COVID-19 vaccines (Bokemper et al., 2021; Borah, 2022; Diament et al., 2022), systematic research that aims to identify promising messaging themes for healthcare workers remains scarce. We took a community-engaged approach to partner with the Wisconsin Primary Health Association (WPHCA), the member association for the 17 FQHCs in Wisconsin. The WPHCA leadership recognized the need to address hesitancy among their workforce during the early phase of vaccine rollout. We collaborated with them to first develop and field a survey to elicit an initial list of belief candidates and then employed the H&W method to identify promising messaging themes. The H&W method is grounded in behavioral change theories (e.g., the theory of reasoned actions (Fishbein & Ajzen, 2009), the integrated model for behavior change (Fishbein & Capella; 2006), and the health belief model (Janz & Becker; 1984) that highlight the roles of health beliefs in predicting and affecting targeted behaviors. In the case of COVID-19 vaccines, a communication campaign targeting healthcare workers can be designed to address a wide variety of beliefs, ranging from legitimate concerns about the likelihood of severe side effects and efficacy against new variants to ungrounded misconceptions (e.g., COVID-19 vaccines will cause infertility; Browne et al., 2021; Li et al., 2021; Meyer et al., 2021). Given resource constraints and the urgency to roll out a campaign quickly, campaign designers cannot target all of them and have to choose high-priority beliefs from a long list of candidates. The H&W method is designed to address this critical need during the formative research. It aims to rank all candidate beliefs by integrating two critical pieces of information: a) each belief’s strength of association with behavioral intention, and b) the proportion of the population who have not yet held the belief and therefore are possible to be influenced by the message intervention. In the past, this method has been applied to facilitate the selection of beliefs for the Food and Drug Administration’s national The Real Cost campaign preventing youth from tobacco use (Brennan et al., 2017). Although initially proposed as a technique to analyze cross-sectional survey data, the causal validity of the H&W approach has been supported in a two-wave panel study conducted with a large nationally representative sample of U.S. youth and young adults (Hornik et al., 2019). In this study, the H&W ranking of 16 smoking-related beliefs produced using the cross-sectional Wave 1 data substantially mirrors (r = .53) the ranking produced using adjusted odds ratios that characterize how Wave 1 beliefs predicted Wave 2 smoking behaviors in a lagged model controlling for Wave 1 behaviors. In another experimental study, Lee and colleagues (2016) found that messages targeting the more promising beliefs identified through the H&W approach indeed reduced smoking intentions through changing the targeted and highly ranked beliefs. Although messages targeting the less promising beliefs also reduced smoking intentions, these effects were mediated through changing untargeted beliefs that were again highly ranked by the H&W approach. These studies provide evidence to support the causal validity of the H&W approach in the context of tobacco control.

We report the first H&W analysis of promising and unpromising beliefs among FQHCs’ workforce. We addressed the nesting structure of our dataset, provided targeted rankings for subgroups, and quantified estimation uncertainties. Our H&W belief rankings can provide immediate utility for communication-based interventions addressing COVID-19 vaccination hesitancy among healthcare workers. This approach can also facilitate the selection of messaging themes in communication-based interventions collaborating with local communities and local healthcare systems. Since we applied the H&W approach to analyze cross-sectional data, causality cannot be established. Whether targeting highly ranked beliefs can produce intended behavioral changes (i.e., vaccine uptake) still needs to be confirmed in follow-up research using longitudinal or experimental designs.

Method

Approach and Sample

From winter 2020 to spring 2021, we conducted key informant interviews (N = 12) with FQHC clinical teams and non-clinical staff through the partnership with the WPHCA. This qualitative formative research helped generate an initial list of belief items related to FQHC employees’ concerns about COVID-19 vaccines, which we expanded with a review of instruments used in national surveys, such as the Kaiser Family Foundation Surveys and several waves of rolling cross-sectional surveys conducted by the COVID States Project (Lewis, 2020; Lazer et al., 2021). In spring 2021, upon WPHCA’s review and approval, they helped email our survey link to all clinical and non-clinical staff members (N = 1,472) from 10 FQHCs that had agreed to participate in this study. These 10 FQHCs collectively served approximately 176,000 Wisconsin residents in 2020, and most of their patients come from rural and historically disenfranchised communities. With a response rate of about 47.3%, in total, 696 participants completed the survey including: (a) 347 clinical team members (role = 1) with direct patient contact (e.g., physicians, nurses, medical assistants, dentists, dental hygienists) and (b) 349 non-clinical staff members (role = 0) including service/outreach and facility support staff. In our analytical sample, 67.7% of participants (n = 471) resided in rural areas (rural = 1), classified based on the county where each FQHC is located and the Wisconsin Office of Rural Health’s Rural Wisconsin Health report. See Table 1 for details on sample characteristics. The protocol was approved by the corresponding author’s Institutional Research Board.

Table 1.

Sample characteristics and summary of vaccine acceptance and recommendation intention by demographic

		Percentage	Vaccine acceptance	Recommendation
Age	18–44	60.7%	73.1% (n = 361)	70.0% (n = 342)
	45–64	37.3%	85.3% (n = 224)	81.8% (n = 214)
	≥ 65	2.0%	91.7% (n = 12)	91.7% (n = 12)
Gender	Female	88.4%	75.3% (n = 590)	71.7% (n = 562)
	Male	11.0%	94.6% (n = 74)	91.2% (n = 68)
	Non-binary	0.4%	100% (n = 3)	100% (n = 3)
Race	Black or African American	3.9%	80.0% (n = 25)	60% (n = 25)
	American Indian or Alaska Native	1.6%	72.7% (n = 11)	70% (n = 10)
	Asian	2.4%	81.3% (n = 16)	92.9% (n = 14)
	Native Hawaiian or Pacific Islander	0.4%	100% (n = 3)	66.7% (n = 3)
	White	88.4%	77.3% (n = 591)	74.1% (n = 560)
	Other	3.3%	85.7% (n = 21)	81.0% (n = 21)
Ethnicity	Hispanic/Latino	5.8%	83.3% (n = 36)	81.1% (n = 37)
	Non-Hispanic/Latino	94.2%	77.5% (n = 627)	73.6% (n = 592)
Education	Post-graduate degree(e.g. M.A., Ph.D., M.D., J.D., etc)	25.1%	93.5% (n = 170)	87.0% (n = 162)
	Bachelor’s degree	23.6%	82.9% (n = 158)	78.1% (n = 151)
	Associate degree	19.3%	67.7% (n = 130)	67.5% (n = 120)
	Trade/technical/vocational training	13.7%	69.2% (n = 91)	70.0% (n = 89)
	Some college credit, no degree	11.4%	70.7% (n = 75)	56.3% (n = 71)
	High school graduate, diploma or the equivalent (for example: GED)	6.8%	60% (n = 45)	67.4% (n = 43)
	Some high school, no diploma	0.1%	0% (n = 1)	0 (n = 1)
Rural/Urban	Rural	67.7%	75.1% (n = 471)	70.5% (n = 457)
	Urban	32.3%	82.9% (n = 225)	80.5% (n = 216)
Occupation type	Clinical team members	49.9%	79.9% (n = 335)	77.0% (n = 310)
	Non-clinical staff	50.1%	75.4% (n = 338)	70.8% (n = 329)

Measures

Vaccine acceptance.

Participants were asked to report whether they had the first or second dose of a COVID-19 vaccine. For those who did not, they further indicated how likely they were to take the one-dose or two-dose vaccine once it became available (5-point Likert scale, 1 = very unlikely and 5 = very likely). Participants who had already taken the vaccine were combined with those who reported very likely or likely to get a vaccine—we recoded this group as 1 and others as 0. We asked participants to report intentions because not all FQHC employees could access COVID-19 vaccines when the survey was fielded.

Recommendation intention.

Participants were asked to report their intentions to recommend the COVID-19 vaccine to other people. Clinical team members were asked how likely they were to recommend either a one-dose or two-dose vaccine to their patients, once available, on a scale from 1 (very unlikely) to 5 (very likely). Similarly, non-clinical staff were asked about their intentions to recommend vaccines to others in general. This variable was also dichotomized, coding participants who reported either very likely or likely as 1 and others as 0.

Belief items.

We developed 46 belief items (see Table 2 for details) through a series of focus groups, interviews, and literature review of national surveys, all measured using a 5-point Likert scale (1 = definitely disagree and 5 = definitely agree). Participants were asked to report their level of agreement or disagreement with each of these belief items, with the order of display randomized. Belief items were further divided into hesitancy-fanning beliefs and confidence-boosting beliefs. Hesitancy-fanning beliefs refer to those associated with higher levels of vaccine hesitancy, such as concerns about safety and efficacy and doubts about information transparency. Confidence-boosting beliefs refer to beliefs associated with an increased intention for vaccination acceptance or recommendation, such as participants’ efficacy beliefs in explaining the science and approval process and expected protection for vulnerable others. All the hesitancy-fanning beliefs were reverse-coded. Following the H&W literature (Hornik et al., 2019; Brennan et al., 2017), all belief items were dichotomized, collapsing somewhat agree and definitely agree into belief = 1 (strong believers) and all other responses into belief = 0 (potential believers).

Table 2.

Descriptive statistics and PPG scores by behavioral outcome for each of the 46 belief items

Beliefs	Mean	SD	PPG for vaccine acceptance	PPG for recommendation intention
Overall, the vaccine would be safe for me.	4.03	1.16	19.04	16.55
The vaccine would effectively prevent me from catching the coronavirus.	3.33	1.24	17.87	17.01
I am likely to experience severe side effects.	2.59	1.15	6.33	5.68
I feel confident in my ability to explain the science behind the vaccines to patients.	3.18	1.25	12.96	16.56
I feel confident in my ability to explain the FDA emergency authorization process to patients.	3.08	1.27	11.82	14.94
I feel confident in my ability to explain the vaccines’ pros and cons to patients.	3.46	1.23	13.43	15.97
If I get the early versions of the vaccine, it would be less effective than later versions.	2.30	1.00	3.11	4.1
Vaccination could lead to birth defects.	2.31	1.07	6.81	7.23
Vaccination could permanently alter my DNA.	1.65	1.01	4.17	4.31
COVID-19 is caused by 5G and is a form of radiation poisoning transmitted through radio waves.	1.36	0.81	1.97	3.02
Vaccination will make it easier for the government to track and monitor my movements.	1.54	0.99	4.12	4.21
Even if I take the vaccine, the immunity that it provides would not last long.	2.58	1.14	7.98	7.7
If I take the vaccine, the live virus it contains might infect people around me.	1.49	0.87	2.56	3.26
The vaccine has been forced upon me.	1.64	1.08	5.41	5.62
I’m afraid that I will get sterilized after getting vaccinated.	1.58	0.99	3.66	3.79
The COVID-19 vaccines were developed much faster than usual because of political pressure.	3.03	1.34	12.33	12.46
Politics have a larger influence than science on vaccine approval and distribution.	3.07	1.31	8.94	8.91
Big pharmaceutical companies want to profit from the COVID-19 vaccines.	3.37	1.22	6.23	6.42
The approval process was too hasty.	2.51	1.30	11.62	11.12
Negative information about the vaccine is being withheld from the public.	2.36	1.26	10.55	9.94
People are deceived about the vaccines---they have more risks than often portrayed.	2.24	1.22	8.15	8.05
There is limited reliable information about the vaccines’ effectiveness and safety.	2.62	1.36	11.47	11.93
We do not know yet whether the new technology (mRNA) used to produce some of the approved COVID-19 vaccines is safe.	2.63	1.31	12.65	12.44
There is conflicting or contradictory information about the vaccines’ effectiveness and safety.	2.80	1.35	9.29	10.57
Arguments about the pros and the cons of the vaccines are equally strong and convincing.	2.35	1.18	3.88	5.13
People who have already been tested positive for COVID-19 can skip the vaccine.	1.74	1.02	5.46	4.42
If I get vaccinated earlier than others, I will put myself at higher risks.	1.75	0.98	1.73	3.05
If I get the vaccine, I will join others’ efforts in bringing our life back to normal soon.	4.21	1.09	15.21	13.82
If I get the vaccine, I will help achieve herd immunity.	4.07	1.12	16.27	15.7
If I get the vaccine, I will help protect other vulnerable people in my community.	4.25	1.07	14.4	13.53
If I get the vaccine, I will be putting something unnatural into my body.	2.58	1.30	9.76	8.71
If I get the vaccine, I will be taking in harmful biomedical materials.	2.00	1.17	7.8	6.91
Most of my colleagues are willing to get the vaccines when they arrive.	3.97	1.02	5.31	7.36
Most health professionals in the US would choose to get the vaccine when vaccines become available.	3.96	0.96	8.24	9.27
Most people who are important to me think I should get the vaccine.	3.82	1.29	17.62	17.19
I will be judged by others in my profession if I refuse to get the vaccine.	3.15	1.27	−1.28	0.14
If I get the current approved vaccines, they will be outdated when new variants of the virus get widely spread.	2.58	1.10	5.05	4.57
Even if I get the vaccine, it will not be strong enough to protect me from getting infected with the new variants.	2.73	1.12	6.54	6.56
If most people are vaccinated, I do NOT need to get the vaccine.	1.67	0.99	4.87	3.94
If everyone gets the vaccine, I will be protected anyway even without getting vaccinated myself.	1.80	1.05	3.77	3.18
The virus is a hoax.	1.44	0.87	2.83	3.23
The virus is manmade.	2.27	1.34	7.94	6.56
The spread of the virus is a deliberate attempt by governments to gain political control.	1.89	1.24	7.86	7.36
The spread of the virus is a deliberate attempt by a group of powerful people to take control.	1.90	1.24	7.35	6.5
The spread of the virus is a deliberate attempt by a group of powerful people to make money.	1.84	1.19	6.94	6.43
The government is trying to cover up the link between vaccines and autism.	1.66	1.01	4.22	4.14

Note: The response options were strongly disagree, disagree, neither disagree nor agree, agree, strongly agree.

Statistical Analysis

The H&W method ranks belief items by calculating each belief’s percentage to gain (PPG) score. To simplify the presentation of this method, we define vaccine endorsers as those either with high acceptance themselves (vaccine acceptance = 1) or high intention for recommendation (recommendation intention = 1). Others were termed potential endorsers. In our context, for a given belief item, its PPG score takes the difference between (a) the percentage of vaccine endorsers among the strong believers for that belief item and (b) the percentage of vaccine endorsers overall. If we assume that each belief is perfectly influential in changing the targeted behavior, conceptually, the PPG score estimates the additional proportion (compared with the current overall percentage in the targeted population) of vaccine endorsers the campaign can expect to convert from the potential endorsers category, should the campaign be successful in persuading everyone to strongly endorse the targeted belief—i.e., moving everyone to the strong believers category. Both experimental and longitudinal evidence supports the validity of relative belief rankings produced by the H&W method in prior work (Hornik et al., 2019; Brennan et al., 2017).

To account for the nested structure of our dataset (e.g., clustering due to participants employed by the same FQHC), we estimated PPG scores within a multilevel logistic regression framework to adjust for dependence among observations. Moreover, we incorporated moderators to provide separate PPG estimates for subgroups (i.e., rural versus urban, clinical teams versus non-clinical staff), paving the way to producing targeted belief rankings. Notably, we added interaction terms to estimate subgroup-specific PPG scores—in this way, the multilevel regression models (with interaction terms) were estimated on the full sample instead of smaller subgroup segments to preserve statistical power. Furthermore, we separated beliefs that were associated with increased vaccination confidence and were hence worth promoting in the campaign (i.e., confidence-boosting beliefs), from those associated with vaccination hesitancy and hence in need of mitigation (i.e., hesitancy-fanning beliefs). Lastly, we used bootstrapping to quantify estimation uncertainties for each PPG.

For a given belief item, we first estimated two probabilities from the multilevel logistic regression models for each of the four subgroups: (a) the subgroup-specific conditional probability of being vaccine endorsers among the strong believers and (b) the subgroup-specific marginal probability of being vaccine endorsers in that subgroup. Then, we subtracted the conditional probability from the marginal probability to calculate that item’s PPG score for a given subgroup. To quantify estimation uncertainties, we used bootstrapping (samples = 1,000) to generate the 84% confidence interval (CI) for each subgroup-specific PPG score in Figures 1–4. It is well known that even two overlapped 95% CIs may still indicate a significant difference in the two means from normal distributions (Maghsoodloo & Huang, 2010). In contrast, 84% of CIs have the preferable property such that the lack of overlapping (i.e., two disjoint CIs) can be taken to directly indicate the significant difference (Maghsoodloo & Huang, 2010) —i.e., two different PPG scores in our case. An example of estimating the PPG scores for the confidence-boosting belief “If I get the vaccine, I will help protect other vulnerable people in my community” and for the hesitancy-fanning belief “Vaccination could permanently alter my DNA”, respectively for each of the four subgroups, can be found in Table 3a and 3b. For the whole sample, PPG scores (projecting a belief item’s impacts on vaccine acceptance and recommendation, respectively) for all 46 belief items are summarized in Table 2.

Figure 1a.

Estimated Percentage to Gain for Rural – Confidence-boosting beliefs

Figure 4b.

Estimated Percentage to Gain for Non-clinical Staff – Hesitancy-fanning beliefs

Table 3a.

Example of calculating the PPG score for a vaccine confidence boosting belief

Belief: “If I get the vaccine, I will help protect other vulnerable people in my community.”
		Strong believers (=Strong agree/somehow agree)	Potential believers (=all other responses)	Overall	Potential Percentage to Gain
Whole sample	% Vaccine acceptance = 1	93.5%	28.8%	79.1%	PPG = 93.5–79.1 = 14.4%
	% Believers in targeted population	77.7%	22.3%
Urban	% Vaccine acceptance = 1	94.2%	39.4%	84.5%	PPG = 94.2–84.5 = 9.7%
	% Believers in targeted population	82.4%	17.6%
Rural	% Vaccine acceptance = 1	93.2%	25.3%	75.6%	PPG = 93.2–75.6 = 17.6%
	% Believers in targeted population	76.6%	23.4%
Clinical staff	% Vaccine acceptance = 1	94.3%	28.6%	81.3%	PPG = 94.3–81.3 = 13%
	% Believers in targeted population	80.2%	19.8%
Non-clinical staff	% Vaccine acceptance = 1	92.7%	28.9%	77.1%	PPG = 92.7–77.1 = 15.6%
	% Believers in targeted population	75.5%	24.5%

Notes. For the whole sample, the conditional probability of being vaccine endorsers (i.e., vaccine acceptance = 1) among the strong believers (i.e., strongly agree or somewhat agree with the belief item) was estimated to be 93.5%. The conditional probability of being vaccine endorsers among the potential believers (i.e., reporting lower levels of agreement with the belief item) was estimated to be 79.1%. The marginal probability of being vaccine endorsers in the targeted population—which, in this case referred to the entire sample—was estimated to be 79.1%. Then, for the whole sample, the PPG score was calculated as 93.5%−79.1% = 14.4%.

In this example, the total proportion of strong believers in the whole sample was estimated to be 77.7% and the corresponding estimated proportion for potential believes was 22.3%. These proportions were informative but not necessary for calculating the PPG score. For other sub-groups (urban, rural, clinical staff, non-clinical staff), targeted PPG scores were provided following the same formula.

Table 3b.

Example of calculating the PPG score for a vaccine hesitancy fanning belief

	Belief: “Vaccination could permanently alter my DNA.
		Strong disbelievers (=Strongly disagree/somewhat disagree)	Potential disbelievers (=all other responses)	Overall
Whole sample	% Vaccine acceptance = 1	83.3%	17.0%	79.1%	PPG = 83.3–79.1 = 4.2%
	% Disbelievers in targeted population	94.8%	5.2%
Urban	% Vaccine acceptance = 1	87.0%	12.7%	83.6%	PPG = 87.0–83.6 = 3.4%
	% Disbelievers in targeted population	95.8%	4.2%
Rural	% Vaccine acceptance = 1	80.4%	17.3%	77.0%	PPG = 80.4 – 77.0 = 3.4%
	% Disbelievers in targeted population	94.3%	5.7%
Clinical staff	% Vaccine acceptance = 1	85.6%	8.5%	82.0%	PPG = 85.6–82.0 = 3.6%
	% Disbelievers in targeted population	95.8%	4.2%
Non-clinical staff	% Vaccine acceptance = 1	81.1%	22.6%	76.5%	PPG = 81.1–76.5 = 4.6%
	% Disbelievers in targeted population	93.9%	6.1%

Notes. For the whole sample, the conditional probability of being vaccine endorsers (i.e., vaccine acceptance = 1) among the strong dis-believers (i.e., strongly disagree or somewhat disagree with the belief item) was estimated to be 83.3%. The conditional probability of being vaccine endorsers among the potential disbelievers (i.e., reporting lower levels of disagreement with the belief item) was estimated to be 17.0%. The marginal probability of being vaccine endorsers in the targeted population—which, in this case referred to the entire sample—was estimated to be 79.1%. Then, for the whole sample, the PPG score was calculated as 83.3%−79.1% = 4.2%.

In this example, the total proportion of strong disbelievers in the whole sample was estimated to be 94.8% and the corresponding estimated proportion for potential disbelieves was 5.2%. These proportions were informative but not necessary for calculating the PPG score. For other sub-groups (urban, rural, clinical staff, non-clinical staff), targeted PPG scores were provided following the same formula.

Notably, the H&W approach aims to rank a large pool of beliefs without necessarily clustering them into belief factors. This analytical strategy is justifiable for two reasons. First, from a practical standpoint, addressing highly ranked individual beliefs rather than belief “clusters” can help alleviate the burden of campaign message development. When resources are scarce, campaign designers are more likely to succeed in creating effective messages to address the most promising single belief rather than a group of many beliefs. Second, from an analytical perspective, the H&W approach aims to calculate the PPG score for each individual belief. Even when a set of beliefs are highly correlated, this would not induce multicollinearity because no more than one belief would be simultaneously entered into the regression model. Therefore, we followed the existing literature (Hornik & Woolf; 1999; Hornik et al., 2019) and analyzed each belief individually. This allows campaign designers to choose which belief or beliefs to target, and beliefs that tend to cluster together would likely appear next to each other on the final ranking.

Results

Overall intentions for vaccine acceptance and recommendation

Table 1 presents the characteristics of the analytical sample (range 696–639) with the largest size, as the specific sample sizes vary from one belief to another, given missing values and list-wise deletion. The sample was predominantly white (88.4%) and female (88.4%), and the majority had a Bachelor’s (23.6%) or post-graduate degree (25.1%). 79.9% of clinical team members and 75.4% of non-clinical staff reported vaccine acceptance. Fewer clinical team members (77.0%) and non-clinical staff members (70.8%) indicated a willingness to recommend vaccines. Fewer participants from rural clinics reported vaccine acceptance = 1 (rural: 75.1% vs. urban: 82.9%) or recommendation intention = 1(rural: 70.5% vs. urban: 80.5%).

The H&W Analysis of Promising Message Themes

In the section below, a series of rankings of beliefs based on the revised H&W method are presented separately by subgroup (rural versus urban residency, clinical teams versus non-clinical staff), behavioral outcome (vaccine acceptance, recommendation intention), and belief type (confidence-boosting versus hesitancy-fanning beliefs). Given the large set of ranked beliefs, we provided a summary of top-ranked beliefs for each of these subgroups (see Table 4 and Table 5 for highlights). These beliefs were identified using a global cut-off point (PPG = 11.62% for vaccine acceptance, PPG = 11.93% for recommendation intention) corresponding to the top 75th percentile in the distribution of PPG scores across all the beliefs in the overall sample, respectively for each behavioral outcome. For readers interested in the complete H&W rankings to contrast top- and bottom-ranked beliefs in each subgroup, Figures 1–4 present all estimated PPG scores along with confidence intervals to facilitate visual comparisons.

Table 4.

Summary of top promising beliefs in predicting vaccine acceptance

	Confidence-boosting beliefs (PPG%)	Hesitancy-fanning beliefs (PPG%)
Overall	Safety (19.04%) Effectiveness (17.87%) Important others think I should get (17.62%) Help reach herd immunity (16.27%) Help go back to normal life (15.21%) Help protect vulnerable population (14.40%) Confidence in explaining pro and con (13.43%) Confidence in explaining science (12.96%) Confidence in explaining FDA process (11.82%)	The vaccine was developed faster due to political pressure (12.33%) Don’t know about mRNA safety yet (12.56%) Approval process was too hasty (11.62%)
Rural	Safety (22.48%) Important others think I should get (20.94%) Effectiveness (20.45%) Help reach herd immunity (20.18%) Help go back to normal life (18.48%) Help protect vulnerable population (17.34%) Confidence in explaining pro and con (13.94%) Confidence in explaining science (12.41%)	The vaccine was developed faster due to political pressure (12.70%) Don’t know about mRNA safety yet (11.85%)
Urban	Safety (15.65%) Effectiveness (15.56%) Confidence in explaining science (13.52%) Confidence in explaining pro and con (12.99%) Confidence in explaining FDA process (12.86%) Important others think I should get (12.69%)	The vaccine was developed faster due to political pressure (14.18%) Don’t know about mRNA safety yet (11.85%) Conflict/contradictory information (13.01%) Negative information withheld (11.68%)
Clinical	Safety (17.44%) Effectiveness (15.90%) Important others think I should get (14.80%) Help reach herd immunity (14.59%) Help go back to normal life (13.45%) Help protect vulnerable population (12.63%)	The vaccine was developed faster due to political pressure (13.00%)
Non-clinical	Safety (20.08%) Effectiveness (19.25%) Important others think I should get (18.66%) Help reach herd immunity (16.68%) Help go back to normal life (15.37%) Confidence in explaining pro and con (14.91%) Help protect vulnerable population (14.57%) Confidence in explaining science (14.56%) Confidence in explaining FDA process (12.75%)	Don’t know about mRNA safety yet (13.62%) Approval process was too hasty (12.36%) The vaccine was developed faster due to political pressure (11.85%) Limited reliable information (11.83%)

Notes: top-ranked beliefs identified using a global cut-off point (PPG = 11.62%) corresponding to the 75th percentile in the distribution of PPG scores across all the beliefs in the overall sample. Outcome is vaccine acceptance.

Table 5.

Summary of top promising beliefs in predicting recommendation intention

	Confidence-boosting beliefs (PPG%)	Hesitancy-fanning beliefs (PPG%)
Overall	Important others think I should get (17.19%) Effectiveness (17.01%) Confidence in explaining science (16.56%) Safety (16.55%) Confidence in explaining pro and con (15.97%) Help reach herd immunity (15.7%) Confidence in explaining FDA process (14.94%) Help go back to normal life (13.82%) Help protect vulnerable population (13.53%)	Don’t know about mRNA safety yet (12.44%) The vaccine was developed faster due to political pressure (12.46%) Limited reliable information (11.93%)
Rural	Important others think I should get (20.48%) Safety (20.16%) Help reach herd immunity (19.46%) Effectiveness (19.41%) Confidence in explaining science (19.05%) Confidence in explaining pro and con (18.73%) Confidence in explaining FDA process (17.94%) Help go back to normal life (17.07%) Help protect vulnerable population (15.55%)	The vaccine was developed faster due to political pressure (13.19%) Limited reliable information (12.54%) Don’t know about mRNA safety yet (12.53%) Most health professionals choose to get vaccinated (11.95%)
Urban	Effectiveness (14.30%) Confidence in explaining science (13.04%) Confidence in explaining pro and con (12.80%) Safety (12.36%)	The vaccine was developed faster due to political pressure (12.31%) Conflict/contradictory information (12.07%)
Clinical	Confidence in explaining FDA process (13.08%) Effectiveness (12.94%) Confidence in explaining pro and con (12.59%) Safety (12.48%) Important others think I should get (12.37%) Confidence in explaining science (12.09%)	The vaccine was developed faster due to political pressure (13.07%)
Non-clinical	Effectiveness (19.99%) Safety (19.71%) Confidence in explaining science (19.50%) Important others think I should get (18.78%) Help reach herd immunity (18.09%) Confidence in explaining pro and con (17.90%) Help go back to normal life (15.20%) Confidence in explaining FDA process (15.08%) Help protect vulnerable population (14.05%)	Don’t know about mRNA safety yet (13.96%) Limited reliable information (13.74%) Approval process was too hasty (12.22%)

Notes: top-ranked beliefs identified using a global cut-off point (PPG = 11.93%) corresponding to the 75th percentile in the distribution of PPG scores across all the beliefs in the overall sample. Outcome is recommendation intention.

Rural FQHCs—top confidence-boosting beliefs to promote

For employees in rural FQHCs, highly ranked confidence-boosting beliefs (Figure 1a) for vaccine acceptance were those about participants’ perceived safety and effectiveness and those associated with community-oriented values. In contrast, beliefs emphasizing normative pressure from peers in the profession were less promising. Regarding vaccine recommendation, promising themes included those mentioned above and additional efficacy items about confidence in explaining the science, the FDA emergency authorization process, and vaccines’ pros and cons to patients.

Rural FQHCs—top hesitancy fanning beliefs to mitigate

For participants from rural FQHCs, for vaccine acceptance, top hesitancy-fanning beliefs (Figure 1b) that need to be mitigated included concerns about the development, approval, and distribution of vaccines as manipulated by political pressure and influences. Concerns about the safety of mRNA technology were also strongly associated with vaccine hesitancy. Additional beliefs ranked high were concerns about information transparency and reliability (e.g., negative information about the vaccine is being withheld from the public). Top-ranked hesitancy-fanning beliefs predicting recommendation intention were similar to those mentioned above. Additional high-ranking beliefs concern the conflicting or contradictory information about the vaccines’ effectiveness and safety.

Figure 1b.

Estimated Percentage to Gain for Rural – Hesitancy-fanning beliefs

Urban FQHCs—top confidence-boosting beliefs to promote

For urban FQHCs, the most promising beliefs associated with both vaccine acceptance and recommendation intention (Figure 2a) were perceived safety and effectiveness, and the participants’ own efficacy in explaining the science, pros and cons of the vaccines. The normative perception from important others was also ranked high. In contrast, normative pressure from peers in the profession and colleagues was ranked the lowest.

Figure 2a.

Estimated Percentage to Gain for Urban – Confidence-boosting beliefs

Urban FQHCs—top hesitancy fanning beliefs to mitigate

The rankings of hesitancy-fanning beliefs were consistent for both vaccine acceptance and recommendation intention (Figure 2b). Participants’ concerns about political pressure were again ranked high, along with their concerns about the safety of mRNA technology, unreliable information, and confusion from conflicting information.

Figure 2b.

Estimated Percentage to Gain for Urban – Hesitancy-fanning beliefs

Clinical staff—top confidence-boosting beliefs to promote

For clinical team members, top-ranked beliefs for vaccine acceptance (Figure 3a) included safety and effectiveness judgments and those associated with community-oriented values. Regarding recommendation intention, besides safety and effectiveness evaluation and normative influences from important others, providers’ perceived efficacy in explaining science, the FDA emergency authorization process, and vaccines’ pros and cons to patients were also highly ranked.

Figure 3a.

Estimated Percentage to Gain for Clinical Staff – Confidence-boosting beliefs

Clinical staff—top hesitancy fanning beliefs to mitigate

The rankings of hesitancy-fanning beliefs were consistent for both behavioral outcomes (Figure 3b). Top beliefs were concerns about political influence, mRNA technology, and information quality.

Figure 3b.

Estimated Percentage to Gain for Clinical Staff – Hesitancy-fanning beliefs

Non-clinical staff—top confidence-boosting beliefs to promote

For both behavioral outcomes, the most promising themes for staff members were similar to clinical staff (Figure 4a). Beliefs related to the safety and effectiveness of vaccines, normative perceptions of important others, and community-oriented values emerged as the most promising themes. Again, efficacy in explaining the science behind the vaccines and the pros and cons of the vaccines were highly ranked beliefs.

Figure 4a.

Estimated Percentage to Gain for Non-clinical Staff – Confidence-boosting beliefs

Non-clinical staff—top hesitancy fanning beliefs to mitigate

For both behavioral outcomes, non-clinical staff members’ concerns about mRNA technology safety, political influence, and information quality were most highly ranked (Figure 4b). Their worries about the likelihood of severe side effects were among the top beliefs as well.

Discussion

This report presented results from the first H&W analysis that identified promising messaging themes to improve COVID-19 vaccine confidence among healthcare clinical team members and non-clinical staff from FQHCs serving under-resourced communities in Wisconsin. To provide better estimates for subgroup-specific belief rankings and to address data clustering where participants were nested under clinics, we adopted a multilevel logistic regression modeling framework with bootstrapping to carry out the H&W analyses. Compared with previous research applying the H&W method, the multiple modeling framework we adopted along with bootstrapping helps produce uncertainty estimates even for clustered data, which we encourage future research to report when feasible. We identified consistently highly ranked beliefs that can be adopted in blanket pro-vaccine messaging across FQHCs. When resources are available for targeted communication, we also produced subgroup-specific belief rankings to answer the call for targeted messaging in health communication campaigns (Brennan et al., 2017; Kreuter & Wray; 2003). It is worth noting that when generating subgroup-specific H&W rankings, we did not carry out the H&W analysis on each of the stratified subsets of the sample. Instead, we included interaction terms between each belief item and the dummies indicating subgroup membership in the overall multilevel logistic regression model and estimated the conditional probability and the marginal probability (two quantities necessary to calculate the PPG score for a particular belief) from that model applied to the combined sample. We believe this is a better analytical strategy, because it preserves statistical power while permitting calculating subgroup-specific PPG scores.

Consistency across subgroups to support blanket messaging

Although the majority of surveyed healthcare workers have already received available COVID-19 vaccines, approximately 20%–30% of participants remained hesitant, comparable to hesitancy levels found in the general population (Browne et al., 2021; Meyer et al., 2021; Shekhar et al., 2021). The success of leveraging healthcare workers as vaccine ambassadors is hinged upon improving vaccine confidence among themselves. Towards that end, our results showed that confidence-boosting beliefs such as those focusing on perceived safety and efficacy, were consistently on the top-five lists across subgroups, and beliefs about being judged by others were consistently ranked low and should be avoided in campaigns.

As for hesitancy-fanning beliefs, campaigns should focus on reducing doubts about information withholding and manipulation by those with political or economic motives, the safety of the mRNA technology, the approval process, and “unnatural” ingredients in the vaccines. These messaging themes are consistent with previous survey research (Li et al., 2021; Meyer et al., 2021). Contrary to our expectations, misconceptions such as the virus is a hoax and COVID-19 is caused by 5G were consistently ranked low and did not warrant intensive messaging for correction, despite heavy media coverage of these topics around the period for data collection. Although such misconceptions might be more widely shared among the public, our results suggest one should refrain from generalizing such expectations to healthcare workers without testing for their association with behavioral intentions. We have shared these recommended messaging themes with the WPHCA and hope these results can inform communication interventions elsewhere targeting healthcare workers in similar settings.

Subgroup-specific rankings to support targeted messaging

The multilevel logistic regression framework allowed us to test potential heterogeneity in belief rankings between subgroups. Indeed, for healthcare workers in urban settings, messaging should focus on improving knowledge on the science and the FDA approval process and then promoting beliefs about one’s ability to explain these crucial aspects of vaccine development; in contrast, for healthcare workers in rural areas, campaigns should target beliefs about normative influences of important others and achieving herd immunity. Hesitancy-fanning beliefs did not seem to differ substantially between rural and urban healthcare workers. When comparing clinical teams with non-clinical staff, for example, concerns about severe side effects were much more promising among non-clinical staff. In practical settings, these results can help campaign designers plan for targeted messaging when it is feasible to segment the audiences. Targeting is an established strategy to improve message persuasiveness³⁰ and hence is worth considering whenever feasible.

Implications for community-engaged health communication research

During health crises such as the COVID-19 pandemic, community partners such as local healthcare systems often need rapid-response research and digestible findings to help them make immediate decisions. The H&W method is well-suited for this purpose because it does not require more time-consuming longitudinal or experimental designs. Furthermore, another attraction for the H&W method is its ability to gather insights directly from the targeted audience, which can help mitigate biases in theme selection if carried out solely by researchers, funders, or campaign designers. That said, researchers are also recommended to be transparent with community partners that the H&W method is designed to identify high-priority message themes, not necessarily the fully developed message products.

Limitations

Firstly, the results from H&W approach are calculated using cross-sectional data and we cannot establish the causal relationships between beliefs and vaccine uptake. Future research is encouraged to empirically assess the causal validity of belief rankings—that is, whether recommended messaging themes would indeed better improve vaccine acceptance and recommendation than lower-ranked messages. Previous research applying the H&W to guide tobacco control campaigns has provided encouraging results (Hornik et al., 2019; Lee et al., 2016), which increased our confidence in reported recommendations. Furthermore, as in typical H&W applications, we relied upon self-reported behavioral intentions as the key criteria variables to benchmark PPG scores. Although behavioral intentions do not always translate into actual behavioral changes, recent research has demonstrated that for the specific purpose of message ranking and evaluation, behavioral intentions can provide valid diagnostic signals (O’keefe, 2021). Lastly, given that the pandemic is rapidly changing, replication studies are encouraged to better track healthcare workers’ vaccine-related beliefs and concerns as they evolve over time.

Conclusion

Healthcare workers have the potential to serve as local champions for COVID-19 vaccines, but targeted messaging is needed to address their existing vaccine hesitancy. Through community-engaged research in partnership with the WPHCA, we showed the feasibility of applying the H&W approach to identify high-priority beliefs and facilitate the development of messages targeting the FQHC workforce, who are critical to the success of improving COVID-19 vaccination in under-resourced communities including rural residents.

Availability of data and material (data transparency)

Data and study materials are available upon request to the corresponding author.