Scaling up the discovery of hesitancy profiles by identifying the framing of beliefs towards vaccine confidence in Twitter discourse

Abstract

Our study focused on the discovery of how vaccine hesitancy is framed in Twitter discourse, allowing us to recognize at-scale all tweets that evoke any of the hesitancy framings as well as the stance of the tweet authors towards the frame. By categorizing the hesitancy framings that propagate misinformation, address issues of trust in vaccines, or highlight moral issues or civil rights, we were able to empirically recognize their ontological commitments. Ontological commitments of vaccine hesitancy framings couples with the stance of tweet authors allowed us to identify hesitancy profiles for two most controversial yet effective and underutilized vaccines for which there remains substantial reluctance among the public: the Human Papillomavirus and the COVID-19 vaccines. The discovered hesitancy profiles inform public health messaging approaches to effectively reach Twitter users with promise to shift or bolster vaccine attitudes.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10865-022-00328-z.

Introduction

Social media microblogging platforms, specifically Twitter, have become highly influential and relevant to shaping attitudes towards vaccination. With 206 million daily active users as of 2021, Twitter has substantial reach and daily exposure being the most popular social network for news consumption (Auxier & Anderson, 2021; Statista, 2021). Moreover, Twitter allows people to express their beliefs about vaccine confidence or hesitancy, their trust or mistrust in vaccines as well as their stance on civil rights and vaccination mandates. Vaccine hesitancy was defined by the 2011 interdisciplinary World Health Organization’s Strategic Advisory Group of Experts (SAGE) Working Group on Vaccine Hesitancy as the “delay in acceptance or refusal of vaccines despite its availability” (MacDonald & group, 2015). Moreover, the SAGE working group recognized at least three universal factors (3C Model) contributing to vaccine hesitancy, subsequently developing the 3C vaccine hesitancy model consisting of (1) vaccine confidence, (2) vaccine complacency, and (3) vaccine constraints (practical vaccine barriers). The 3C and subsequent vaccine hesitancy models (Betsch et al., 2018) have shown vaccine confidence to play a significant role and explain the most substantial proportion of variance underpinning vaccine doubt that in turn contributes to individuals not vaccinating. Therefore, in this study we explored the degree to which vaccine confidence was framed on social media and how it informs profiles of vaccine hesitancy for both HPV and COVID-19—the two most controversial yet effective and underutilized vaccines for which there remains substantial reluctance among the public. Hence, our first research question asks:

• RQ1 How is confidence in the HPV and the COVID-19 vaccines framed in the Twitter discourse?

Answering this question was possible by casting the search for framings of vaccine confidence as a Natural Language Processing (NLP) problem operating on Twitter. To our knowledge, there are no current NLP techniques capable of identifying how vaccine confidence is framed in social media discourse. Social science stipulates that discourse almost inescapably involves framing (Boydstun et al., 2014; Chong & Druckman, 2007; Entman, 2007)—a strategy of highlighting certain issues to promote a certain interpretation or attitude. For example, when misinformation is used in framing vaccine confidence, it typically results in vaccine hesitancy. Similarly, when civil rights are highlighted in a particular framing, it determines vaccine refusal, while when trust in vaccines is increased, it leads to vaccine acceptance, and eventual uptake. Recent work in NLP concerning automatic recognition of framings targeted the study of political bias and polarization in social and news media (Baumer et al., 2015; Boystun et al., 2014; Card et al., 2015; Field et al., 2018; Roy & Goldwasser, 2020; Tsur et al., 2016), mainly addressing the recognition of 15 cross-cutting dimensions of political framing (Card et al., 2015), e.g., economic dimensions, fairness and equality or policy prescription and evaluation. Although recent Twitter content analysis (Rao et al., 2021) revealed that there is significance correlation between polarized attitudes towards vaccines and political dimensions, to our knowledge, no NLP methods have yet been developed to identify vaccine hesitancy framings, although vaccine hesitancy is often discussed in social/news media.

In this paper we present a Question/Answering (Q/A) solution for the identification of hesitancy framings, enabled by the questions introduced in (Rossen et al., 2019), as the Vaccine Confidence Repository (VCR). Q/A is an established NLP framework that consists of the automatic processing of the question language which enables the identification of its answer in large collections of documents (Strzalkowski & Harabagiu, 2006). When the questions are complex, (Harabagiu & Hickl, 2006; Scialom et al., 2019) the answer is automatically processed by summarizing multiple passages deemed relevant to the question. Given that large collection of tweets discussing either the COVID-19 or the HPV can be retrieved from the Twitter API, questions targeting the confidence in vaccines (available from the VCR) can be used in a Q/A system operating on the index of tweets to capture the framing of vaccine hesitancy.

The questions from the VCR were informed by the antivaccine content analysis of Kata (Kata, 2010, 2012). Kata’s analysis of the content of anti-vaccination websites was among the first to reveal six classes of content attributes related to vaccine hesitancy or resistance: (1) concerns about the safety and effectiveness of vaccines; (2) the consideration of alternative medicine; (3) the interaction of civil liberties with vaccination programs; (4) reference to conspiracy theories; (5) the influence of morality, religion and ideology on choice of vaccination and (6) the usage of misinformation and falsehoods. This pioneering work on categorizing vaccine hesitancy or resistance informed more than 500 studies. In the study reported in Rossen (2019), 3–4 questions were generated by researchers targeting each of the classes of content attributes reported by Kata (2010, 2012), producing the VCR. The VCR questions were used in that study as survey links available from Facebook pages and parenting forums. Nearly 300 Australian visitors answered the questions, enabling the discovery of hesitancy profiles. We were inspired by this work, which aimed to discover vaccine hesitancy profiles, and instead of soliciting answers from Twitter users, we decided to (a) retrieve the tweets that answer the same questions; and (b) infer from them the vaccine hesitancy framings. Moreover, we were interested to examine if this methodology of discovering hesitancy framings can inform the discovery of hesitancy profiles. Furthermore, we applied the methodology not only to the HPV vaccine, but also to another vaccine, namely the COVID-19 vaccine. For this purpose, we extended the VCR with questions addressing confidence in the newer COVID-19 vaccine, modifying the original VCR questions when necessary. Not only were we able to discern hesitancy framings from Twitter for both vaccines, but when analyzing these framings, we noticed that they often relied on misinformation. In this work, we considered misinformation as any misconception, references to conspiracy theories, or any flawed reasoning. This allowed us to address the second research question:

• RQ2 What specific misinformation about the HPV and COVID-19 vaccines is propagated on Twitter?

Answering RQ2 entails discovering the specific misinformation that was unveiled by answering the questions from VCR, but also and importantly, the derivation of a taxonomy of misinformation that is used to frame confidence in the HPV or the COVID-19 vaccines. Misinformation has exploded on social media platforms such as Twitter (Cacciatore, 2021; Hou et al., 2021; Wawrzuta et al., 2021), but it is less known which misinformation themes are propagated and what concerns they address. Building a taxonomy of misinformation to uncover HPV and COVID-19 vaccine related misinformation is needed, against which inoculation interventions can be prepared. A growing literature suggests that vaccine acceptance depends to a large extent on public trust and related confidence in the safety and efficacy of vaccines (Larson et al., 2018; Latkin et al., 2021; Siegrist, 2021). To further understand the way in which trust in vaccines impacts vaccine confidence, we considered a third question in our study:

• RQ3 What trust issues are associated with the HPV and COVID-19 vaccines in Twitter conversations?

The multidimensional concept of trust involves not only trust in the vaccine, but also trust in the healthcare practitioners who administer vaccines, the healthcare systems, public health authorities and governments who advocate for vaccination. Trust is increasingly important especially in the context of high uncertainty for vaccine decision-making such as with the recent coronavirus pandemic, rapidly changing emerging science on what is known about Coronavirus as example, changing vaccine recommendations, growing science illiteracy, and the growing number of vaccines being recommended. Under these conditions of uncertainty, the public depends increasingly on the expertise, judgements, competency, and transparency in sharing what is known about vaccines. The case of trust and vaccination carries with it a history of vaccine development and missteps, but social movements and reactions render trust in vaccines highly variable and locally specific (Larson et al., 2016). In the context of our study, we explored trust erosion, or trust increase in vaccination. The answer to RQ3 led to the development of two trust taxonomies for each vaccine, namely a taxonomy for trust building and a taxonomy for eroding trust. These taxonomies revealed a constellation of concerns that addressed several trust themes impacting confidence in vaccines. Interestingly, we found that many of the trust concerns that we discovered aligned with a multitude of definitions of trust, ranging from the individual level, e.g. trust involving the overall reluctance to obtain vaccination due to fear of side effects (Latkin et al., 2021), to societal and system levels of trust in science and public health authorities (Siegrist, 2021; Sutton et al., 2020). Explaining the differences in vaccine trust or in the tension between civil rights and vaccine mandates is made possible by considering the moral aspects of the vaccine hesitancy framings. Consequently, we also addressed the research question:

• RQ4 What moral dimensions characterize the confidence in the HPV and COVID-19 vaccines on Twitter?

Because previous work in social psychology considered the Moral Foundations Theory (MFT) (Haidt & Graham, 2007; Haidt & Joseph, 2004) as a theoretical framework advocating that there is a small number of moral values emerging from evolutionary, social, and cultural origins, that human support. The moral values are referred as Moral Foundations (MFs) and include Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion and Purity/Degradation. The contrasting Care/Harm MFs are concerned with care for others, generosity, compassion, ability to feel pain of others, sensitivity to suffering of others, prohibiting actions that harm others, or the opposite respectively. The contrasting Fairness/Cheating MFs are concerned with fairness, justice, reciprocity, reciprocal altruism, rights, autonomy, equality, proportionality, prohibiting cheating, or the opposite respectively. The contrasting Loyalty/Betrayal MFs are concerns with group affiliation and solidarity, virtues of patriotism, self-sacrifice for the group, prohibiting betrayal of the group, or the opposite respectively. The contrasting Authority/Subversion MFs are concerned with fulfilling social roles, submitting to authority, respect for social hierarchies or tradition, leadership, prohibiting rebellion against authority, or the opposite respectively. Finally, the contrasting Purity/Degradation MFs are concerned with associations with the sacred or holy, religious notions which guide how to live, prohibiting violation of the sacred or disgust and contamination, respectively. We used these MFs when analyzing hesitancy framings and encoded each framing with the MFs they evoked. These moral encodings proved to be very informative in the discovery of the hesitancy profiles based on vaccine confidence, as revealed by Twitter discourse. Ultimately, in this study we were most interested to answer the research question:

• RQ5 What hesitancy profiles can be discerned from Twitter for the HPV and COVID-19 vaccines?

Answering this question entails discovering how hundreds of thousands of Twitter users frame their confidence in vaccines and what stance they have towards these framings. This was possible because we had access to tweets discussing the HPV vaccine authored by 192,487 users and tweets discussing the COVID-19 vaccine authored by 2,268,358 users. However, we found that only 138,779 Twitter users framed their confidence in the HPV vaccine and only 665,798 users framed their confidence in the COVID-19 vaccines. Nevertheless, we hypothesized that Twitter authors evoking vaccine hesitancy framings in similar ways, with respect to their adoption or rejection of misinformation, their erosion or building trust in the vaccines, the vaccine literacy they have or lack, their stance on the respect of civil rights as well as their focus on certain moral foundations, must belong to the same hesitancy profile.

To our knowledge, this is the first study that aims at the automatic discovery of hesitancy profiles at large scale, especially by using vaccine hesitancy framings discovered automatically as answers to a set of questions about confidence in vaccines. We believe that this method uncovered hesitancy profiles that provide a more nuanced interpretation than those reported in Rossen (2019), which were based on answers provided by human participants that indicated their agreement with each VCR question. This is because hesitancy framings were further characterized by linking them to the misinformation or trust taxonomies while also considering the moral foundations and the vaccine literacy. This ontological characterization of the hesitancy framings allowed us to interpret the hesitancy profiles against the stance tweet authors had when evoking their vaccine confidence, instead of only relying on the quantified attitudes as answers to the VCR questions. We were pleasantly surprised by the hesitancy profiles revealed by the method presented in this paper, and the insightful interpretations that could be derived. We believe that these profiles identify where interventions can be delivered on the Twitter platforms, and most importantly, what vaccine hesitancy issues the interventions need to consider. Finally, as the method was successfully applied for two different vaccines, it highlights its portability for considering confidence across vaccines and deriving vaccine-specific hesitancy profiles.

Methods

Overview of methodology

The methodology that we employed for uncovering vaccine hesitancy profiles from the Twitter discourse addressing vaccine confidence is illustrated in Fig. 1, which shows the four main processing steps. In Step 1, we performed the identification of the vaccine hesitancy framings for either the HPV or COVID-19 vaccines as answers to questions asking about confidence in vaccines. Step 2 scaled-up the discovery of vaccine hesitancy by automatically discovering (a) all the tweets that evoked any of the hesitancy framings identified in Step 1 and (b) the stance of the tweet author towards the framing. In the Step 3 ontological commitments of the hesitancy framings identified in Step 1 are derived. First, the framings are categorized and then taxonomies of misinformation or trust are derived, while also recognizing in the hesitancy framings the implied Moral Foundations (MF) provided by the Moral Foundations Theory (MFT) (Haidt & Graham, 2007; Haidt & Joseph, 2004), health literacy and impact of civil rights on vaccine hesitancy. These ontological commitments along with all the stance of tweet authors towards the framings, identified in Step 2, inform the representation of each Twitter user framing their vaccine hesitancy. The user representations enabled the discovery of the hesitancy profiles in Step 4 of the method.

Fig. 1.

Overview of Methodology: Step 1: Identifying hesitancy framings using Question/Answering; Step 2: Scaling-up the discovery of tweets evoking any hesitancy framing; Step 3: Derivation of ontological commitments of the framings; Step 4: Discovery of vaccine hesitancy profiles

Step 1: Identification of vaccine hesitancy framings

The Q/A framework that was used for identifying how vaccine hesitancy is framed on Twitter introduces some novelties in the typical Q/A architectures that consist of a Question Processing Module (QMP) which allows any question to be transformed in queries used by a Tweet Processing Module (TMP) to identify text passages relevant to a question, informed by an index where documents have been processed; and an Answer Processing Module (APM) which extracts the answer. First, in the QPM, instead of processing directly the questions, we automatically generated five attitude-invoking questions from each question, as shown in Fig. 1 by relying on regular expressions. The rationale for generating attitude-evoking question stems from the belief, supported by prior research in opinion-based Q/A (Yu & Hatzivassiloglou, 2003) that tweets expressing attitudes are more likely identified by attitude-evoking question than by complex questions. The questions processed in the QPM consist of (a) a general question asking about confidence in the HPV or COVID-19 vaccine, i.e. Q1: “How confident are you in the safety of the HPV/COVID-19 vaccine? “ and (b) a set of 18 questions from the Vaccine Confidence Repository (VCR), introduced by Rossen et al. (2019). The entire list of VCR questions that were used is available in the supplemental material.

The VCR questions concern five major belief themes resulting from the analysis of the content expressed in anti-vaccination sites (Kata, 2010, 2012). The question belief themes are: (T1) vaccines are unsafe and unnatural; (T2) vaccines are ineffective; (T3) there is redundant vaccination; (T4) parents should be free to choose whether or not to vaccinate their children and (T5) vaccination is a conspiracy. For each belief theme, three or four questions were formulated. Table 1 lists one question from theme T2 (vaccines are ineffective) pertaining to the quest for vaccine confidence in the COVID-19 vaccine as well as a question from T2 used for inquiring about confidence in the HPV vaccine along with the attitude-invoking questions generated from them. There were 95 (19 × 5) attitude-evoking questions generated for each vaccine.

Table 1.

Examples of automatically generated attitude-evoking questions

Question about COVID-19 vaccine: Given all the COVID-19 variants, the protective effects of the vaccine do not last

Automatically generated questions evoking different attitudes towards vaccine confidence:

Q.C1: Are you completely sure that given all the COVID-19 variants, the protective effects of the vaccine do not last?

Q.C2: Are you somehow sure that given all the COVID-19 variants, the protective effects of the vaccine do not last?

Q.C3: Are you on the fence about the protective effects of the vaccine given the COVID-19 variants?

Q.C4: Why do you think that given the COVID-19 variants; the protective effects of the vaccine probably will last?

Q.C5: Why are you convinced that given all the COVID-19 variants, the protective effects of the vaccine will last?

Question about HPV vaccine: The more people who get vaccinated the greater the protection against disease

Q.H1: Do you strongly believe that because more people get the HPV vaccine, the greater the protection against the diseases caused by HPV infection?

Q.H2: Are you somehow certain that the more people get the HPV vaccine, the greater the protection against the diseases cause by HPV infection?

Q.H3: Why are you on the fence regarding the fact that if more people get the HPV vaccine, then the greater the protection against the diseases caused by the HPV infection?

Q.H4: What makes you think that even if the more people who get the HPV vaccine, it will probably not change the protection against the diseases caused by the HPV infections?

Q.H5: Why are you convinced that even if more people get the HPV vaccine, it will not change the protection against the diseases caused by the HPV infection?

The TPM uses the attitude-evoking questions resulting from the QPM to find relevant tweets for them, based on a relevance model, implementing the BM25 vector ranking model (Beaulieu et al., 1997). The collection that was searched for relevant tweets was obtained by using the Twitter streaming API for each vaccine. For the HPV vaccine, we used the Twitter historical API with the following query “(human papillomavirus vaccination) OR (human papillomavirus vaccine) OR gardasil OR cervarix OR (hpv vaccine) OR (hpv vaccination) OR (cervical vaccine) OR (cervical vaccination) lang:en”, 1,833,380 total tweets, with 969,372 retweets and 864,008 original tweets from 625,354 total authors. These tweets were authored in the time frame initiating on January 1st, 2008, and end ending on May 1st, 2021 (~ 13 years). A large fraction of these tweets, which were duplicates likely due to spam bots, required filtering. Locality Sensitive Hashing (LSH) (Das et al., 2007) is a well-known method used to remove near-duplicate documents in large collections. We performed LSH, with term trigrams, 100 permutations, and a Jaccard threshold of 50%, on our original tweets collection to produce the collection $C_T^{\mathit{HPV}}$ = 422,078 unique original tweets. The tweets from $C_T^{\mathit{HPV}}$ were authored by $A_T^{\mathit{HPV}}$ = 192,487 users. Using the same methodology, we used the query “(covid OR coronavirus) vaccine lang:en” for retrieving for the COVID-19 vaccines a collection of 19,021,575 total tweets, with 9,888,104 retweets and 9,133,471 original tweets from 4,382,289 total users obtained from the Twitter streaming API, which resulted after near-duplication removal into the collection $C_T^{COVID-19}$ = 5,865,046 unique original tweets, authored by $A_T^{COVID-19}$ = 2,268,358 users in the time span January 17th, 2021–July 21st, 2021 (~ 6 months). We used Lucene (lucene.apache.org) to index in $I^{\mathit{HPV}}$ the tweets from $C_T^{\mathit{HPV}}$ and in $I^{COVID-19}$ the tweets from $C_T^{COVID-19}$. For each of the attitude-evoking questions pertaining to the confidence in the HPV vaccine, a set of ranked tweets from $C_T^{\mathit{HPV}}$ are retrieved. The ranking is produced by the scoring function from the BM25 relevance model, operating on the indexes $I^{\mathit{HPV}}$. Similarly, for each of the attitude-evoking questions pertaining to the confidence in the COVID-19 vaccine, a set of ranked tweets from $C_T^{COVID-19}$ was retrieved, using or $I^{COVID-19}$.

In the APM, from the ranked list of tweets retrieved by the relevance model, we considered (1) only the top 300 ranked tweets, and (2) we merged all the top ranked tweets retrieved for all attitude-evoking questions, aiming to judge their relevance. The judgements were performed by two experts in question answering. A total of 1523 tweets for the HPV vaccine, and 2388 tweets for the COVID-19 vaccine were judged as being relevant by two researchers from the Human Language Technology Research Institute at University of Texas at Dallas. Cohen’s Kappa score was 0.81, which indicates strong agreement between annotators (0.8–0.9) (Zapf et al., 2016). As shown in Fig. 1, tweets that were judged to be truly relevant were categorized with respect to the attitude of the tweet authors towards the predication of the question. The same human judges that evaluated the relevance have also judged whether the tweet is (a) against; (b) doubts or (c) accepts the predication of the question. This was essential for the process of inferring the hesitancy framings, which is the goal of the APM. Hesitancy framings were inferred from tweets that shared the same attitude towards a question. We were inspired by work in query-based summarization (Baumel et al., 2016; Yulianti et al., 2018), in which an abstractive summary is created to highlight the most informative aspects of multiple documents that answer a query. In our case, the tweets had the role of documents, and considering that the tweets were already retrieved based on the processing of attitude-evoking questions, two computational linguistics experts selected the discourse units that are shared by a set of tweets, from which the framing was generated, informed by the pyramid method. The pyramid method (Nenkova & Passonneau, 2004) is an empirically grounded method for content selection that quantifies the centrality of viewpoints.

Table 2 illustrates examples of the attitudes assigned to tweets relevant to a question inquiring about the safety of COVID-19 vaccines as well as hesitancy framings inferred using the pyramid method. In generating the framing A, first, a discourse unit from Tweet 1 was selected, while a second discourse unit was selected from Tweet 3, to infer the framing, similarly to an abstractive summary. The linguists also have inspected all other discourse units of the tweets, deciding whether they are (a) central to the issues discussed across all tweets sharing the same attitude and (b) offering the same response to the inquiry question before selecting them to be used in the framing. When a discourse unit expressed new content, it was selected for the inference of the framing. In this way, Step 1 produced $framing{s}^{COVID-19}$ and $framing{s}^{\mathit{HPV}}$, thus informing the answer to RQ1.

Table 2.

Example of question used to inquire about the confidence in COVID-19 vaccines, answered by relevant tweets

Q(COVID-19): Vaccines have not been adequately tested for safety

Tweets AGAINST the question predication:

Tweet 1: Scientists have been working on Coronavirus vaccines for decades; once COVID-19 was isolated they could focus all their attention on that. Vaccine has been tested, tracked, and is safe. Make sure everyone understands this

Tweet 2: But it is misleading to suggest that adverse results in previous mRNA vaccine trials make the Covid 19 vaccines unsafe, particularly when widespread testing of such vaccines has been carried out on humans

Tweet 3: The #COVIDVaccines have been tested on a wide range of people to make sure they’re safe and effective for everyone

Framing A. Scientists have been working on Coronavirus vaccines for decades. The COVID-19 vaccine has been tested, tracked and it is safe

Tweets DOUBTING the question predication:

Tweet 4: I am seeing folks hesitating to get the COVID-19 vaccine because “it hasn’t been adequately tested”, but then I see articles about the lingering effects of having COVID-19, and think, “Huh, I'll take my chances with the vaccine.”

Framing B. Even if vaccine was not tested for a long time, it is not worth having the lingering effects of COVID-19

Tweets ACCEPTING the question predication

Tweet 5: The COVID-19 vaccine is untested. It takes at least 5 years to test a vaccine they say. The problem robert kennedy Jr has with them is they dont do what they are suppose to do. It has only been 2 years since COVID 19

Tweet 6: The Covid Vaccines have NOT been adequately tested. Therefore, this vaccine satisfies The Nuremberg Code conditions and definitions for “medical research and experimentation”. Coerced Vaccinations would violate The Nuremberg Code of The Geneva Convention

Framing C. The COVID-19 vaccines have not been tested for at least 5 years, as they should

The question belongs to the theme of unsafe and unnatural vaccinations. The relevant tweets are categorized according to the attitudes expressed towards the question predication. Examples of inferred framings for each attitude category are shown

Step 2: Scaling-up the discovery of hesitancy framings

While hesitancy framings were inferred from tweets that were relevant to the questions from the VCR, we expect that many other tweets from the collection $C_T^{\mathit{HPV}}$ = 422,078 unique tweets may evoke any of the $framing{s}^{\mathit{HPV}}$, and similarly, many tweets from the collection $C_T^{COVID-19}$ = 5,865,046 unique tweets may evoke any of the $framing{s}^{COVID-19}$. Therefore, in Step 2 we aimed to discover all tweets that evoke any of the framings identified in Step 1. For this purpose, each framing was used as a query, to retrieve tweets that are deemed relevant to the framing. The retrieved tweets were judged by three language experts as being relevant or irrelevant, with inter-judge agreement computed using the Kappa score, yielding a score of 0.82 for relevant tweets for the HPV vaccine framings and 0.84 for relevant tweets for the COVID-19 vaccine framings. Then we used a Natural Language Processing (NLP) method taking advantage of Deep Learning to recognize all tweets evoking any vaccine hesitancy framing. The NLP method, detailed in (Weinzierl & Harabagiu, 2021), uses a supervised learning framework. This entails that we have divided the judged tweets into a training set, a validation set, and a testing set. For the HPV vaccine, the training set contains 4128 tweets, out of which 3703 tweets evoked a framing. For the same vaccine, the validation set had 459 tweets, out of which 424 tweets evoked a framing, while the testing set had 1147 tweets, out of which 1024 evoked a framing. For the COVID-19 vaccines, the training set contains 7604 tweets, out of which 6684 tweets evoked a framing, the validation set had 845 tweets, out of which 748 tweets evoked a framing and the testing set had 2113 tweets, out of which 1838 evoked a framing.

The training set of tweets informed the construction of a of a fully connected graph for each framing $F_i$ (FCG-$F_i$) which was bootstrapped through link-prediction to discover additional tweets evoking $F_i$. The bootstrapping of FCG-$F_i$ took advantage of the fact that a deep learning representation of the graph of FCG-$F_i$ can be learned with knowledge embedding models, e.g. TransE (Bordes et al., 2013), TransD (Ji et al., 2015), TransMS (Yang et al., 2019) or TuckER (Balazevic et al., 2019). When knowledge embedding models learn to represent each node and each relation from a graph, they rely on a link scoring function, which we have used to predict if a tweet evokes a framing $F_i$, and thus should be included in the graph FCG-$F_i$. In addition, as we detail in (Weinzierl & Harabagiu, 2021), we designed a neural architecture that also considered the language from each tweet, not only its neural representation to make the prediction of a tweet that should be linked in each FCG-$F_i$. This method allowed us to discover that there were 282,651 tweets in $C_T^{\mathit{HPV}}$ that evoked some $framing{s}^{\mathit{HPV}}$ and 1,256,369 tweets in $C_T^{COVID-19}$ that evoked some $framing{s}^{COVID-19}$.

In Step 2, as illustrated in Fig. 1, the scaling-up of the discovery of hesitancy framings also involves the automatic identification of the stance. To discover the stance of each tweet addressing any of the framings, we relied on a second NLP method using deep learning, detailed in (Weinzierl et al., 2021), by stacking several layers of lexico-syntactic, semantic, and emotion Graph Attention Networks (GATs) (Velickovic et al., 2018) to learn and refine all the possible interactions between these different linguistic phenomena, before classifying a tweet as (a) agreeing; (b) disagreeing or (c) having no stance towards the framing of interest. Stance discovery was made possible by the stance judgements produced by the three language experts which judged whether tweets from the $C_T^{\mathit{HPV}}$ or $C_J^{COVID-19}$ collection were relevant to $framing{s}^{\mathit{HPV}}$ or $framing{s}^{COVID-19}$. Whenever a tweet evoked a framing, it received a probabilistic distribution [$p^{\mathit{Accept}}$, $p^{\mathit{Reject}}$, $p^{\mathit{NoStance}}$] with respect to the framing. Using the method detailed in (Weinzierl et al., 2021), we discovered that there were 137,261 tweets that accepted some framing from $framing{s}^{\mathit{HPV}}$ and 54,946 tweets that rejected some framing from $framing{s}^{\mathit{HPV}}$. Similarly, we have found that there are 877,481 tweets that accepted some framing from $framing{s}^{COVID-19}$ and 447,716 tweets that rejected some framing from $framing{s}^{COVID-19}$.

Step 3: Deriving the ontological commitments of hesitancy framings

The hesitancy framings identified in Step 1 were first categorized by language experts as expressing (1) misinformation; (2) evoking issues of trust in vaccines; (3) pertaining to civil rights or (4) expressing morality issues. The decision of whether a framing contained misinformation was based on finding evidence on the Web, as retrieved by search engines, that the framing expressed known misconceptions, or conspiracy theories. In addition, whenever flawed reasoning was observed, the framing was categorized as misinformation. One researcher with expertise in Web search and an expert on Public Health independently judged the framings that contain misinformation. The two researchers adjudicated their differences and decided that out of the 64 framings inferred for the confidence in the HPV vaccines, 21 of them (33%) expressed misinformation. Similarly, out of the 113 framings inferred for the safety of COVID-19 vaccines, misinformation was present in 38 of them (34%). Table 3 illustrates some examples of misinformation.

Table 3.

Examples of misinformation expressed in vaccine hesitancy framings

Examples of hesitancy framings for the HPV vaccine expressing misinformation

Framing H-M1. The HPV vaccine may cause the immune system to attack the body

Framing H-M2. The HPV Vaccine specifically was designed to destroy a young girl's immune system

Framing H-M3. The HPV vaccine is linked to nervous and immune system disorders in some young women and girls

Examples of hesitancy framings for COVID-19 vaccine expressing misinformation

Framing C-M1. The COVID-19 Vaccine is a satanic plan to microchip people

Framing C-M2. Strong immune system is all you need to prevent COVID-19 infection

Framing C-M3. Because the chances of survival if infected with COVID-19 are 99.99%, people should have the right to decide whether to vaccinate

Examples H-M1 to H-M3, pertaining to the HPV vaccine, articulate misinformation about the effects of the vaccine on the immune system. Each framing articulates a different nuance of these effects. However, all three framings share the theme of the vaccine’s effects on the immune system, an observation that motivated us to derive a taxonomy of misinformation based on the themes and the concerns raised by the misinformation articulated throughout framings. Similarly, the framings C-M1 to C-M3 cover the theme that the COVID-19 vaccines are unnecessary. Given these observations, all framings expressing misinformation were organized in a misinformation taxonomy, by inspecting common themes and concerns and concerns they addressed. The Misinformation Taxonomy has three layers of abstraction: themes → concerns → framings → tweets. Given that in Step 2 we automatically recognized all tweets that evoke framings that express misinformation, the next layer of abstraction concerns grouping all framings that addressed the same concern, and finally all concerns that share the same theme—and the highest level of abstraction. This allowed us to answer RQ2.

While expressing misinformation is also seeding mistrust in the vaccines, to our surprise, many other framings addressed the issue of trust in the safety of vaccines, although not expressing any misinformation. As with judging misinformation, two researchers (one public health expert and a sociolinguist expert) made independent judgements about whether a framing is eroding or increasing trust in vaccine safety or does not convey any trust issue. The inter-judge agreement was computed using the Kappa score, yielding a score of 0.8 for trust expressed about the HPV vaccine and 0.82 for trust expressed about the COVID-19 vaccines. After adjudicating the judgements, we found that there were 21 hesitancy framings that increase trust and 20 hesitancy framings that erode this trust in the HPV vaccine. Similarly, 27 framings were found to increase the trust while 25 framings eroded trust in COVID-19 vaccines.

Table 4 illustrates examples of both forms of trust in the HPV or the COVID-19 vaccines. We derived two separate taxonomies for trust in the vaccines: a taxonomy of eroding trust and a taxonomy of building trust in the vaccines. Like the derivation of the Misinformation Taxonomy, common themes were first identified, which were further categorized by the concerns they raised, generating the three levels of abstraction: themes → concerns → framings → tweets.

Table 4.

Examples of framings that erode or increase the trust in vaccine confidence

Examples of framings which are eroding the trust in the HPV vaccine

Framing HT-1. The HPV vaccine only prevents cancer caused by HPV, you can get cancer many ways including genetics

Framing HT-2. The Gardasil vaccine promotes promiscuity

Framing HT-3. Children are given too many vaccines

Examples of framings which are increasing trust in the HPV vaccine

Framing HT + 1. Vaccinate against HPV is best at younger age

Framing HT + 2. Cancer and other diseases caused by HPV (human papillomavirus) can be prevented with HPV vaccine

Framing HT + 3. Vaccine immunity is more protective than natural immunity

Examples of framings which are eroding the trust in the COVID-19 vaccine

Framing CT-1. It is not known if the COVID-19 vaccines will provide protection against future variants

Framing CT-2. Some breakthroughs of fully vaccinated people have happened

Framing CT-3. Needs proof that the vaccine will not kill in 2 years those taking it

Examples of framings which are increasing trust in the COVID-19 vaccine

Framing CT + 1. Scientists have been working on Coronavirus vaccines for decades. The COVID-19 vaccine is a success

Framing CT + 2. HPV vaccine is safe and efficient

Framing CT + 3. Incentives increase the likelihood of taking the COVID-19 vaccine

In Step 3 of our methodology, additional ontological commitments were produced for hesitancy framings that showcase vaccine literacy of its absence. For example, framings HT + 1 and HT + 2 illustrated in Table 4 were coded as showcasing vaccine literacy, whereas framings HT-1 and HT-2 display lack of vaccine literacy. In this study, we have relied on the definition of vaccine literacy reported in (Biasio et al., 2021), which considers vaccine literacy as the competence to find, understand and use health and vaccination information. We found that 17 of the hesitancy framings addressing the HPV vaccine showcased vaccine literacy, while 15 displayed a lack of literacy. Similarly, 27 of the framings used for the COVID-19 vaccine showcased vaccine literacy, while 21 displayed a lack of literacy.

A small number of framings identified in Step 1 addressed civil rights issues. From all the framings that were inferred for the HPV vaccine, 12 framings address civil right issues, while from all the framings inferred for the COVID-19 vaccine 28 framings addressed civil rights issues. Examples of framings that were categorized as expressing civil rights are listed in Table 5. The ontological commitments that were considered for the framings addressing civil rights encoded two possible situations: (1) framings implying that vaccination should be prioritized over civil rights (e.g., framing C.CR.3 from Table 5); and (2) framings implying that civil rights should always be prioritized (e.g., framing H.CR.3 from Table 5).

Table 5.

Examples of framings that address civil rights

Examples of framings for the HPV vaccine which address civil rights

Framing H.CR.1. Religious exemptions for the Gardasil vaccine guarantee Freedom of Religion

Framing H.CR.2. Teenagers should be able to make their own decision about HPV vaccination

Framing H.CR.3. It is a parent right to decide if children get vaccinated

Examples of framings for the COVID-19 vaccine which address civil rights

Framing C.CR.1. COVID-19 vaccine passports are acceptable only for travel but not for other activities

Framing C.CR.2. Refusing the COVID-19 vaccine when working in healthcare is unacceptable

Framing C.CR.3. Vaccination against COVID-19 should be mandatory

The final categorization concerned moral issues highlighted by the framings. Examples of such framings are provided in Table 5. Nevertheless, we considered to reveal all the Moral Foundation (MFs) implied in each framing. Previous work (Johnson & Goldwasser, 2018, 2019) has shown that there are correlations between stances towards framings and moral convictions that justify the stances. To further explore the correlation between MFs and hesitancy framings, a computational linguist and an expert in public health have independently assigned MFs to all the framings for the HPV and the COVID-19 vaccines. The inter-judge agreement was computed using the Kappa score, yielding a score of 0.89 for the HPV vaccine and 0.85 for the COVID-19 vaccines. These annotations enabled us to answer question RQ4.

Step 4: Discovering vaccine hesitancy profiles

The previous three steps of our methodology provided information that allowed us to generate a representation of the users involved in the discourse about COVID-19 vaccines and the users participating in the discourse about the HPV vaccine. A vectorial representation was produced for each Twitter user that evoked $framing{s}^{\mathit{HPV}}$ or $framing{s}^{COVID-19}$ in any of their tweets. This The Vector User Representation (VUR) has entries for (a) the themes from the misinformation taxonomy; (b) the themes from the taxonomy for building trust or from the taxonomy for eroding trust; (c) a quantification of the vaccine literacy or lack of; (d) a quantification of the impact of civil rights on vaccination; and (e) a quantification of each of the MFs, as illustrated in Fig. 2. These values of the VUR are computed as: (1) values $v_{\mathit{Theme}}$ quantifying the conceptualization of misinformation or trust taxonomy themes in each user’s tweets; (2) values $v_{\mathit{Literacy}}^{+/-}$ quantifying the vaccine literacy (or lack of) of the framings referred by users in their tweets; (3) values $v_{\mathit{CR}}^{+/-}$ quantifying a user’s preference of vaccination mandates over civil rights ( +) or the respect of civil rights, regardless of public health circumstances (−); and (4) values $v_{\mathit{MR}}^i$ quantifying the support of each of the MFs. Central to the computation of these four types of values is the quantification of each $Framin{g}_X$ evoked by a user in its tweets, in a value $v_{\mathit{Framing}}^X$. To compute $v_{\mathit{Framing}}^X$ we note that any tweet $t$ authored by the same user has: (1) a $Framin{g}_X$, that it evokes; (2) a stance that reflects if the tweet $t$ is (a) accepting the $Framin{g}_X$, quantified by the probability $p_X^{\mathit{Accept}}\left(t\right)$; or it rejects the $Framin{g}_X$, quantified by the probability $p_X^{\mathit{Reject}}\left(t\right)$; or the tweet has no stance towards $Framin{g}_X$, quantified by the probability $p_X^{\mathit{NoStance}}\left(t\right)$, where the distribution ($p_X^{\mathit{Accept}}\left(t\right)$, $p_X^{\mathit{Reject}}\left(t\right)$, $p_X^{\mathit{NoStance}}\left(t\right)$) is produced by the automatic stance detection from Step 2 of the methodology. We made the assignment of $v_{\mathit{Framing}}^X\left(t\right)=\text{max}\left(p_X^{\mathit{Accept}}\left(t\right),p_X^{\mathit{Reject}}\left(t\right)\right)$, preferring a quantification provided by the dominant stance. Moreover, when the dominant stance was the rejection of the $Framin{g}_X$, we changed the polarity, i.e. $v_{\mathit{Framing}}^X\left(t\right)=-v_{\mathit{Framing}}^X\left(t\right)$, such that positive values assigned to $Framin{g}_X$ are interpreted as acceptance of the framing, whereas negative values represent rejection of the framing. Furthermore, the quantification of each $v_{\mathit{Theme}}$ is based on the observation that framings that are mapped into the misinformation or trust taxonomies can be ontologically characterized by some $Them{e}_Y$ from one of these taxonomies illustrated in Tables 6, 7, 8 and 9. If a user generates only one tweet $t$ that evokes $Framin{g}_X$, then $v_{\mathit{Theme}}^Y=v_{\mathit{Framing}}^X\left(t\right)$. However, when a user generates multiple tweets $t_i$ that evoke same $Framin{g}_X$, then $v_{\mathit{Theme}}^Y$ is computed as the average of $v_{\mathit{Framing}}^X\left(t_i\right)$. Moreover, if the same user generates tweets which refer to multiple framings $Framin{g}_j$ categorized under the $Them{e}_Y$ of one of the taxonomies, the value $v_{\mathit{Theme}}^Y$ becomes the sum of framing values, i.e. $v_{\mathit{Framing}}^j\left(t_i\right)$. Because of this, for some themes the values in the user representation may be outside the interval [− 1, + 1]. The values $v_{\mathit{Literacy}}^{+}$ result from taking the average value of all $v_{\mathit{Framing}}^X\left(t\right)$ that were annotated as exhibiting vaccine literacy, while $v_{\mathit{Literacy}}^{-}$ results from taking the average value of all $v_{\mathit{Framing}}^X\left(t\right)$ that showcase lack of vaccine literacy, given every tweet $t$ of a user. Finally, the evaluation of the values $v_{\mathit{MR}}^i$, for each of the $i=1,\dots ,9$ Moral Foundations (MF), is generated by taking the average value of all $v_{\mathit{Framing}}^X\left(t\right)$, for each framing that was annotated with the $M{F}_i$. When the VURs for all users were generated, hesitancy profiles were discerned by using the k-Means clustering algorithm (Lloyd, 1982), experimenting with $K=2,\dots ,10$ possible clusters. The final number of clusters was determined by the Elbow method [Thorndike 1953]. We found $K=5$ to be optimal for both HPV and COVID-19 vaccine hesitancy profiles.

Fig. 2.

Vector User Representation for Discovery of Hesitancy Profiles

Table 6.

A Taxonomy of Misinformation about the HPV vaccine

THEME 1: Unsafe HPV vaccine	THEME 2: HPV Vaccine Ingredients
CONCERN1: Gardasil creator says the vaccine is as dangerous as HPV *CONCERN2: Some scientists explain why the HPV vaccine is unsafe CONCERN3: Multiple countries ban the HPV vaccine CONCERN4: HPV vaccine proven unsafe in Europe CONCERN5: Lack of research about HPV vaccine safety	*CONCERN1: The HPV vaccine contains toxins CONCERN2: The HPV vaccine contains Borax CONCERN3: The HPV vaccine contains lab-engineered DNA
THEME 3: Testing of the HPV Vaccine	THEME 4: Alternatives to HPV Vaccine
CONCERN1: HPV vaccine was not tested on boys CONCERN2: Vaccine tested against placebo with high amounts of aluminum CONCERN3: Not tested for carcinogenicity or impairment of fertility CONCERN4: Tested on minorities and underserved because it contains sterility formula	*CONCERN1: Homeopathic medicines are alternatives to HPV vaccine *CONCERN2: Vitamins are alternatives to HPV vaccine CONCERN3: Mushroom extract is an alternative to HPV vaccine
THEME 5: Unnecessary HPV vaccine	THEME 6: Effect on Immune System
CONCERN1: The HPV vaccine prevents the same cancer preventable by a Pap smear *CONCERN2: A strong immune system fights off most strains of HPV CONCERN3: Improved living standards, not vaccinations, reduced infections	CONCERN1: HPV vaccine alters children immune system, even of an unborn child *CONCERN2: HPV vaccine may cause immune system to attack body CONCERN3: HPV vaccine was designed to destroy young girls’ immune system
THEME 7: Not effective HPV vaccine	THEME 8: Adverse Events of HPV vaccine
CONCERN1: The HPV vaccine causes cancer CONCERN2: The HPV vaccine does not cover all cancer-causing HPV strains CONCERN3: The HPV vaccine causes more cancer cases than it prevents	CONCERN1: Deaths/ Injuries caused by HPV vaccine CONCERN2: The HPV Vaccine causes Infertility CONCERN3: HPV vaccine leads to mental retardation CONCERN4: HPV vaccine causes paralysis
THEME 9: HPV vaccine and promiscuity	THEME 10: HPV vaccine information is concealed
CONCERN1: Girls receiving the HPV vaccine will become promiscuous CONCERN2: Not getting the HPV vaccine is not putting others at risk if you are not promiscuous	CONCERN1: Pharmaceutical companies conceal information about HPV vaccine safety CONCERN2: Pharmaceutical companies hide cancer cure and release HPV vaccines *CONCERN3: The Government conceals information about the safety of HPV vaccines

Table 7.

A Taxonomy of Misinformation about the COVID-19 Vaccines

THEME 1: Unsafe COVID-19 vaccine	THEME 2: COVID-19 Vaccine Ingredients
CONCERN1: Vaccine unsafe because the virus is a bioweapon. *CONCERN2: Some scientists explain why the COVID-19 vaccine is unsafe CONCERN3: Bill Gates admits the COVID-19 vaccine is unsafe CONCERN4: The COVID-19 vaccine makes you gay CONCERN5: The COVID-19 vaccine makes you 5G compatible CONCERN 6: The COVID-19 vaccine renders pregnancies risky	*CONCERN1: The COVID-19 vaccine injects a toxin in your bloodstream CONCERN2: The COVID-19 vaccine uses nanotechnology CONCERN3: The vaccine is gene therapy that activates a toxin in your body CONCERN4: The mRNA vaccine contains the virus
THEME 3: Testing of the COVID-19 Vaccine	THEME 4: Alternatives to COVID-19 Vaccine
CONCERN1: No-long term studies of side-effects of COVID-19 vaccine CONCERN2: No COVID-19 vaccine efficacy or safety data CONCERN3: Vaccine has not been tested for at least 5 years	*CONCERN1: Homeopathic medicines are alternatives to HPV vaccine *CONCERN2: Vitamins are alternatives to COVID-19 vaccine CONCERN3: Hydroxychloroquine as alternative to vaccine CONCERN4: Garlic as alternative to vaccine CONCERN5: Mushroom extract is an alternative to HPV vaccine
THEME 5: Unnecessary COVID-19 vaccine	THEME 6: Effect on Immune System
CONCERN1: The vaccine is a satanic plan to microchip population *CONCERN2: A strong immune system is all you need CONCERN3: Chances of surviving infection are 99.99% CONCERN4: People with severe allergies should not be vaccinated	CONCERN1: COVID-19 vaccine overwhelms the immune system *CONCERN2: COVID-19 vaccine may cause immune system to attack body CONCERN3: COVID-19 vaccine overrides the immune system
THEME 7: Not effective COVID-19 vaccine	THEME 8: Adverse Events of COVID-19 vaccine
CONCERN1: The vaccine does not protect against COVID-19 infection CONCERN2: Natural Immunity lasts longer than vaccine-induced immunity CONCERN3: People better protected by immunity gained through infection then immunity gained through vaccination	CONCERN1: COVID-19 vaccine interacts with people’s DNA CONCERN2: COVID-19 vaccine replaces the genetic code with a synthetic one
	THEME 10: COVID-19 vaccine information is concealed
	CONCERN1: Pharmaceutical companies conceal information about breakthroughs and reinfections CONCERN2: The Federal Government lied about vaccines to reduce the information about COVID-19 treatments *CONCERN3: The Government conceals information about the safety of COVID-19 vaccines

Table 8.

The Taxonomy for building Trust in HPV vaccines and the Taxonomy for eroding Trust in the HPV vaccines

Taxonomy of building TRUST in HPV vaccines

Taxonomy Of eroding TRUST In HPV vaccines

THEME 1: Trust in Safety of HPV vaccines *CONCERN 1: HPV vaccine is safe; I gave it to my kids! *CONCERN 2: Information about the safety of vaccines is available on VAERS COCNERN 3: There are no serious allergy reactions to HPV reported THEME 2: Motivation for taking HPV vaccines CONCERN 1: Cancer and other diseases caused by HPV (human papillomavirus) can be prevented with HPV vaccine CONCERN 2: Vaccinate against HPV is best at younger age CONCERN 3: Vaccines can save lives and eliminate diseases CONCERN 4: Vaccination is the most effective way of preventing infectious diseases *CONCERN 5: Vaccine immunity is more protective than natural immunity THEME 3: Trust in Effects of HPV vaccines CONCERN 1: HPV vaccine improves children’s immune systems CONCERN 2: The HPV vaccine does not suppress the immune system like the natural HPV infection CONCERN 3: Not having side effects after the HPV vaccine depends on the immune system CONCERN 4: HPV vaccine cannot cause cancer or any other disease THEME 4: Trust in role of HPV vaccines for Public Health CONCERN 1: HPV vaccine can completely eradicate cervical cancer CONCERN 2: Herd Protection through vaccination CONCERN 3: Vaccinations are the key to controlling infectious disease and the secondary conditions associated with infection CONCERN 4: The benefits of the HPV vaccine extend to people who aren't vaccinated — meaning the more people who are vaccinated, the better CONCERN 5: To eradicate several cancers, HPV vaccination should be compulsory and free THEME 5: Trust in the role of pharmaceutical companies for preventing cancer with HPV vaccines *CONCERN 1: Vaccines are not profitable for pharmaceutical companies CONCERN 2: HPV Vaccines prevent cancer, regardless of the profit of pharmaceutical companies THEME 6: Trust in Doctors/Science CONCERN 1: Safety and efficacity of HPV vaccines supported/explained by doctors CONCERN 2: Safety and efficacity of HPV vaccines supported/explained by science

THEME 1: Lack of Trust in Safety of HPV vaccines CONCERN 1: HPV vaccine may provoke autoimmune diseases CONCERN 2: Children/Teens with allergy should not be vaccinated CONCERN 3: Vaccines alter the immune system of a child THEME 2: De-Motivation in taking the HPV vaccines CONCERN 1: The HPV vaccine only prevents cancer caused by HPV; you can get cancer many ways including genetics CONCERN 2: Children are given too many vaccines CONCERN 3: The Gardasil vaccine promotes promiscuity THEME 3: Lack of Trust in Effects of HPV vaccines CONCERN 1: There are some side effects specific to age, but they aren't really about the child's immune system THEME 4: Lack of Trust in role of HPV vaccines for Public Health CONCERN 1: Why mandate a children vaccine that protects against cancer that appears later in life CONCERN 2: If vaccines work, the vaccinated have nothing to fear from the unvaccinated THEME 5: Lack of Trust in the role of pharmaceutical companies for preventing cancer with HPV vaccines *CONCERN 1: Pharmaceutical companies produce ineffective HPV vaccine for profit CONCERN 2: Mandating the vaccines is benefitting the pharmaceutical companies THEME 6: Lack of Confidence in Doctors/ Science CONCERN 1: Doctors do not tell the truth about HPV vaccines CONCERN 2: Researchers/scientists do not tell the truth about HPV vaccines CONCERN 3: Casting doubt about published research of HPV vaccine THEME 7: Lack of Trust in Ingredients of HPV vaccines CONCERN 1: HPV Vaccine contains aluminum adjuvant THEME 8: Lack of Trust in the testing of the HPV vaccine CONCERN 1: HPV vaccine rushed into use without adequate testing CONCERN 2: Distrust of FDA testing procedures CONCERN 3: HPV vaccine was not tested if it causes cancer

Table 9.

The Taxonomy for building Trust in COVID-19 vaccines and the Taxonomy for eroding Trust in the COVID-19 vaccines

Taxonomy of increasing TRUST in COVID-19 vaccines

Taxonomy of eroding TRUST in COVID-19 vaccines

THEME 1: Trust in Safety of COVID-19 vaccines *CONCERN 1. HPV vaccine is safe and efficient. Scientists have been working on Coronavirus vaccines for decades. The COVID-19 vaccine has been tested, tracked and it is safe *CONCERN 2. The Government has provided plenty of safety information about the COVID-19 vaccines THEME 2: Motivation for taking the COVID-19 vaccines CONCERN 1: Given the risks of COVID-19, it is unlikely that building natural immunity is a good idea CONCERN 2. Even if vaccine not tested for a long time, it is not worth having the lingering effects of COVID-19 CONCERN 3. The COVID-19 vaccine is not making you immune to the infection, but it is mitigating the effect of the infection CONCERN 4: The vaccines trigger your body to naturally create immunity more reliably than getting COVID *CONCERN 5. Natural antibodies last a few months and only partially protect you, while the vaccine instructs your immune system to produce antibodies that last longer and protect you better CONCERN 6: COVID-19 vaccines protect against the emerging variants CONCERN 7. Incentives for taking the COVID-19 vaccine CONCERN 8: Children over 12 should be vaccinated to avoid distant learning THEME 3: Trust in Effects of COVID-19 vaccines CONCERN 1: Vaccination against COVID-19 Strengthens the immune system CONCERN 2: More likely to get thrombosis from flying economy than from Astra Zeneca CONCERN 3: Johnson and Johnson COVID-19 vaccine allegedly preferred over Pfizer or Moderna for those with allergies THEME 4: Trust in role of COVID-19 vaccines for Public Health CONCERN 1: Getting the COVID-19 vaccine will protect my patients, family and friends who cannot get the jab. Vaccination is key in protecting yourself and others against COVID-19 CONCERN 2: Because of the risk of new variants that may escape the COVID-19 vaccines, the decision to not vaccinate puts everyone at risk CONCERN 3: People can choose not to vaccinate if they pay a higher medical bill when needing hospitalization because of COVID-19 THEME 5: Trust in role of pharmaceutical companies in fighting COVID-19 infections *CONCERN 1: COVID-19 vaccines are not profitable for pharmaceutical companies unless they are safe and effective THEME 6: Trust in Ingredients of COVID-19 vaccines CONCERN 1: The mRNA COVID-19 Vaccine uses the RNA of COVID-19 which leaves your body soon after you get vaccinated THEME 7: Trust in Testing of COVID-19 vaccines CONCERN 1: COVID-19 vaccination trials for children proven vital

THEME 1: Lack of Trust in Safety of COVID-19 vaccines CONCERN 1: if the COVID-19 vaccines are completely safe, why no accidental death policy CONCERN 2: Vaccine exemptions should be available because the COVID-19 vaccines are experimental CONCERN 3: Children should not be vaccinated against COVID-19 because there is no legal accountability for adverse events CONCERN 4: Governments of the Western world advocate COVID-19 vaccines without long-term studies of safety or efficacy, while Asian governments control COVID-19 infections with traditional safety measures THEME 2: De-Motivation in taking the COVID-19 vaccines CONCERN 1: The COVID-19 vaccine does not provide immunity against infection CONCERN 2: People having severe allergy should be monitored for 30 min after receiving the Pfizer COVID-19 vaccine CONCERN 3: It is not known if the COVID-19 vaccines will provide protection against future variants THEME 3: Lack of Trust in Effects of COVID-19 vaccines CONCERN 1: Lack of confidence in mRNA vaccines and their long-term effects CONCERN 2: Needs proof that the vaccine will not kill in 2 years those taking it CONCERN 3: Wait one year to see if there are no long-lasting side effects CONCERN 4: Astra Zeneca vaccine determines blood clots CONCERN 5: Fear vaccine may worsen existing conditions THEME 4: Lack of Trust in role of COVID-19 vaccines for Public Health CONCERN 1: People that do not believe that COVID-19 is real or do not believe that masks work should not be receiving vaccines be exempt from the vaccination CONCERN 2: Because the authorities advocate so hard for COVID-19 vaccination should be the main reason for refusing the vaccine CONCERN 3: Preference for getting COVID-19 and fighting it off than vaccinating THEME 5: Lack of Trust in role of pharmaceutical companies in fighting COVID-19 infections *CONCERN 1: Pharmaceutical companies will profit because COVID-19 waves will never end, thus requiring annual boosters THEME 6: Lack of Trust in Ingredients of COVID-19 vaccines CONCERN 1: The COVID-19 Vaccine injects the dead SARS-COV2 virus in your body THEME 7: Lack of Trust in Testing of COVID-19 vaccines CONCERN 1: COVID-19 vaccines were ‘rushed,’ so they could still be unsafe CONCERN 2: AstraZeneca used outdated information in its COVID-19 vaccine trials

Results

Step 1 of our method produced 113 $framing{s}^{COVID-19}$ and 64 $framing{s}^{\mathit{HPV}}$. They answer RQ1:

• RQ1 How is confidence in the HPV and the COVID-19 vaccines framed in the Twitter discourse?

A more nuanced answer to RQ1 is provided by the categorization $framing{s}^{COVID-19}$ and $framing{s}^{\mathit{HPV}}$ performed in Step 3 the methodology, yielding a more in-depth understanding of how confidence in the two vaccines is framed. Figure 3, which illustrates the distribution of the framing categories across all the themes covered by the VCR questions (Rossen et al. (2019). Surprisingly, for the HPV vaccine, misinformation framings were inferred across all question themes, except for question Theme 4, where civil rights dominated. We expected to find a lot of misinformation in the framings answering the questions from theme 5, but we were startled to find plenty of misinformation answering questions from themes 1, 2 and 3. Moreover, the framings answering the general theme also contained misinformation, indicating that misinformation is pervasive in the framing of confidence in the HPV vaccines. As shown in Fig. 3, misinformation (indicated in red shading) is also present in framings answering questions about confidence in the COVID-19 vaccines. We also noticed that there is a higher percentage of framings that erode trust (yellow shading) in the COVID-19 vaccines than in the HPV vaccine. Surprisingly, there is a substantial percentage of framings that build trust across both vaccines. Framings involving civil right issues were inferred as answers mostly to questions from Theme 4 about parents’ right to decide whether to vaccinate their children for both vaccines. Vaccine literacy (blue shading) seems to be present in most framings except Theme 5 (vaccine conspiracy theories) in the case of COVID-19.

Fig. 3.

Distribution of Hesitancy Framings Categories

The results of Step 2 of the method, aiming to scale up the discovery of framings on the entire Tweet collections discussing the vaccines, were evaluated using the following metrics: Precision ($P$), Recall ($R$) and $F1$-measure. $P$ computes the number of tweets correctly identified to evoke any framing, out of all tweets that the system reported in in (Weinzierl & Harabagiu, 2021) automatically pinpointed. $R$ measures how many tweets were identified by the system reported in in (Weinzierl & Harabagiu, 2021) to evoke a framing out all of the tweets that were judged to do so, and $F1=2PR/\left(P+R\right)$. We measured $P=$ 80.1%; $R=$ 83.2%, and $F1=$ 81.6% when evaluating on the test collection for the HPV vaccine while $P=$ 71.5%, $R=$ 75.8%, and $F1=$ 73.6% when evaluated in the test collection for COVID-19 vaccines. In Step 2 we also recognized the stance of the tweet authors against the framing they evoke. The stance detection system we used, detailed in (Weinzierl et al., 2021), recognized the “Accept” stance with $F1=$ 86.5%; the “Reject” stance with $F1=$ 70.5% for the $framing{s}^{\mathit{HPV}}$, while for the $framing{s}^{COVID-19}$ it recognized the “Accept” stance with $F1=$ 87.6% and the “Reject” stance with $F1=$ 71.5%. These results indicate that these two automatic systems performed quite well.

The results of Step 3 involve the creation of several taxonomies, including the Misinformation Taxonomy, illustrated in Tables 6 and 7; The Trust Taxonomies illustrated in Tables 8 and 9. Ten misinformation themes were discovered in the Misinformation Taxonomy for the HPV vaccine, illustrated in Table 6 while nine misinformation themes were discovered for COVID-19 vaccination, illustrated in Table 7. Asterix among concerns denotes that these concerns were common across vaccines. Although higher order themes were similar across vaccines (with HPV vaccination having one additional promiscuity theme), concerns across vaccines differed in number and content sharing only 21% of concerns. Of the 33 concerns, only 7 were shared across vaccines. This suggests that misinformation is tailored to the worries that are vaccine specific. The Misinformation Taxonomy derived for the HPV vaccine as well as the Misinformation Taxonomy derived for the COVID-19 vaccine answer the question:

• RQ2 What specific misinformation about the HPV and COVID-19 vaccines is propagated on Twitter?

It is important to also compare the number of framings expressing misinformation in both taxonomies, as it represents the lowest level of abstraction, now shown in Tables 6 and 7. 21 framings for the HPV vaccine and 38 framings for the COVID-19 vaccine expressed misinformation.

Two different Trust taxonomies discovered for each vaccine provided the answers to:

• RQ3 What trust issues are associated with the HPV and COVID-19 vaccines in Twitter conversations?

Table 8 illustrates these taxonomies for the HPV vaccine, whereas Table 9 illustrates both taxonomies for the COVID-19 vaccine. Trust in the HPV vaccine is characterized by 6 themes expressing 21 different concerns, while the taxonomy that encodes erosion of trust in the HPV vaccine used 18 concerns distributed across 8 different themes. When comparing the taxonomy encoding knowledge that builds trust in the HPV vaccine with the same taxonomy for the COVID-19 vaccine, we observed only 4 common concerns, which are marked with * in Tables 8 and 9. However, the same comparison on the trust eroding taxonomies leads to the observation that here is only one shared concern, marked with * in Tables 9 and 10.

Table 10.

Twitter hesitancy profiles discovered for the HPV vaccine

Profile 1: PROMOTERS (14,403 users; 21%) Themes for Eroding Trust in Vaccine: − 0.05: Lack of Trust in Safety of HPV Vaccines Themes for Building Trust in Vaccine:  + 1.01: Motivation for Taking HPV Vaccines  + 0.99: Trust in Role of HPV Vaccines for Public Health  + 0.77: Trust in the Role of Pharmaceutical Companies for preventing Cancer with HPV Vaccines  + 0.07: Trust in Doctors/Science Vaccine Literacy:  + 2.08: Having literacy| −0.08: Lacking literacy Civil Rights:  + 0.77: Vaccines more important than civil rights Moral Foundations:  + 2.81: Care|+ 2.77: Authority|+ 0.89: Loyalty |+ 0.89: Fairness| − 0.11: Subversion| − 0.10: Harm |− 0.09: Degradation| − 0.05: Purity

Profile 2: TRUSTERS (19,338 users; 28%) Themes for Building Trust in Vaccines:  + 0.45: Motivation for Taking HPV Vaccines  + 0.27: Trust in Role of HPV Vaccines for Public Health  + 0.17: Trust in Doctors/Science  + 0.10: Trust in Effects of HPV Vaccines  + 0.09: Trust in the Safety of HPV Vaccines Vaccine Literacy:  + 0.93: Having literacy| − 0.05: Lacking literacy Civil Rights:  + 0.06: Civil rights above all Moral Foundations:  + 0.94: Care|+ 0.84: Authority|+ 0.23: Fairness|  + 0.19: Loyalty|+ 0.11: Betrayal |− 0.11: Subversion − 0.08: Harm| − 0.06: Degradation

Profile 3: DEBUNKERS (21,904 users; 32%) Misinformation Themes: − 0.10: Not effective HPV Vaccine − 0.06: Adverse Events of HPV Vaccine Themes for Eroding Trust in Vaccine: − 0.12: Lack of Trust in Safety of HPV Vaccines − 0.11: Lack of Trust in the Testing of the HPV Vaccines − 0.09: Lack of Trust in Role of HPV Vaccines for Public Health − 0.08: De− Motivation in Taking the HPV Vaccines Themes for Building Trust in Vaccine: − 0.05: Trust in the Safety of HPV Vaccines Vaccine Literacy: − 0.14: Having literacy| − 0.25: Lacking literacy Civil Rights: − 0.20: Civil rights above all Moral Foundations: − 0.35: Degradation|− 0.35: Subversion | − 0.32: Authority| − 0.25: Purity| − 0.25: Harm| − 0.18: Care| − 0.13: Cheating| − 0.07: Betrayal Profile 5: SKEPTICS (8,787 users; 13%) Misinformation Themes:  + 0.32: Unsafe HPV Vaccine |+ 0.21: Not effective HPV Vaccine  + 0.11: Testing of HPV Vaccine|+ 0.07: HPV Vaccine Ingredients  + 0.11: Information about HPV Vaccines is Concealed  + 0.09: Adverse Events of HPV Vaccine| Themes for Eroding Trust in Vaccine:  + 0.27: Lack of Trust in Doctors/Science  + 0.19: Lack of Trust in Safety of HPV Vaccines  + 0.12: De− Motivation in Taking the HPV Vaccines  + 0.05: Lack of Trust in the Testing of the HPV Vaccines Themes for Building Trust in Vaccine: − 0.11: Trust in Doctors/Science Vaccine Literacy:  + 0.23: Lacking literacy| − 0.09: Having literacy Civil Rights:  + 0.18: Civil rights above all Moral Foundations  + 0.61: Subversion|+ 0.40: Harm|+ 0.35: Betrayal|  + 0.24: Cheating|+ 0.22: Degradation|+ 0.17: Purity|  + 0.06: Fairness

Profile 4: MISINFORMERS (4673 users; 7%) Misinformation Themes:  + 0.91: Information about HPV Vaccines is Concealed  + 0.69: Not effective HPV Vaccine  + 0.18: Unsafe HPV Vaccine  + 0.16: HPV Vaccine Ingredients  + 0.14: Testing of HPV Vaccine  + 0.07: Adverse Events of HPV Vaccine Themes for Eroding Trust in Vaccine:  + 0.38: Lack of Trust in the Role of Pharmaceutical Companies for Preventing Cancer with HPV Vaccines  + 0.16: Lack of Trust in Doctors/Science  + 0.13: Lack of Trust in Safety of HPV Vaccines  + 0.07: De− Motivation in Taking the HPV Vaccines  + 0.06: Lack of Trust in the Testing of the HPV Vaccines Themes for Building Trust in Vaccine:  + 0.70: Trust in the Role of Pharmaceutical Companies for Preventing Cancer with HPV Vaccines − 0.09: Trust in the Safety of HPV Vaccines − 0.08: Trust in Doctors/Science Vaccine Literacy: − 0.12: Having literacy|+ 0.12: Lacking literacy Civil Rights:  + 0.09: Civil rights above all Moral Foundations:  + 2.80: Betrayal |+ 1.59: Harm|+ 0.61: Authority| − 0.06: Care  + 0.58: Cheating|+ 0.41: Subversion|+ 0.27: Degradation

In Step 3 we also produced annotations of the implied Moral Foundations (MFs), answering the research question:

• RQ4 What moral dimensions characterize the confidence in the HPV and COVID-19 vaccines on Twitter?

Overall, there were 111 moral foundations (MFs) annotated in the 64 confidence framings for the HPV vaccine and 230 MFs annotated in the 113 confidence framings used for the COVID-19 vaccines. Interestingly, 1 framing was coded with 4 MFs, 32 with 3 MFs, 95 with 2 MFs and 48 framings with one MF. The predominant MF in the confidence framings for the HPV vaccine was a tie between Authority and Subversion, while the predominant MF in the framings for the COVID-19 vaccine was Harm. This indicates that the most common approach by which each vaccine is morally framed in public discourse shifts depending on the vaccine. The framings that expressed misinformation mostly implied the MF Subversion. The framings that conveyed trust erosion in vaccines predominantly conveyed the MFs Subversion and Harm, while the framings that built trust in the vaccine implied the MFs Care and Authority. Framings that involved civil rights issues were predominantly implying the MF Fairness, while framings that involved literacy issues were predominately implying the MF of Care.

The results of Step 4 are the vaccine hesitancy profiles discovered for each of the two vaccines. Table 10 lists the hesitancy profiles and their characteristics for the HPV vaccine, while Table 11 lists those for the COVID-19 vaccines. The hesitancy profiles answer the research question:

• RQ5 What hesitancy profiles can be discerned from Twitter for the HPV and COVID-19 vaccines?

Table 11.

Twitter Hesitancy Profiles for the COVID-19 Vaccine

Profile 1: PROMOTERS (48,447 users; 9%) Misinformation Themes: − 0.07: Unnecessary COVID-19 Vaccine − 0.07: Testing of COVID-19 Vaccine − 0.07: Adverse Events of COVID-19 Vaccine  + 0.06: COVID-19 Vaccine Ingredients − 0.04: Unsafe COVID-19 Vaccine |− 0.04: Effect on Immune System − 0.03: Not effective COVID-19 Vaccine Themes for Eroding Trust in Vaccine:  + 0.35: De-Motivation in Taking the COVID-19 Vaccines − 0.10: Lack of Trust in Safety of COVID-19 Vaccines Themes for Building Trust in Vaccine:  + 1.03: Trust in Role of COVID-19 Vaccines for Public Health  + 0.83: Motivation for Taking COVID-19 Vaccines  + 0.22: Trust in the Safety of COVID-19 Vaccines  + 0.05: Trust in Effects of COVID-19 Vaccines Vaccine Literacy: + 2.32: Having literacy| 0.04: Lacking literacy Civil Rights: + 0.56: Vaccines more important than civil rights Moral Foundations:  + 1.63: Care|+ 1.82: Authority |+ 0.71: Loyalty |+ 0.98: Fairness| − 0.37: Subversion| 0.64: Harm |0.03: Degradation| 0.05: Purity

Profile 2: AMBIVALENT (267,087 users; 48%) Misinformation Themes  + 0.08: Information about COVID-19 Vaccines is Concealed  + 0.07: COVID-19 Vaccine Ingredients  + 0.03: Unsafe COVID-19 Vaccine Themes for Eroding Trust in Vaccine:  + 0.08: De-Motivation in Taking the COVID-19 Vaccines  + 0.04: Lack of Trust in Safety of COVID-19 Vaccines  + 0.03: Lack of Trust in Effects of COVID-19 Vaccines Themes for Building Trust in Vaccines:  + 0.08: Trust in Role of COVID-19 Vaccines for Public Health  + 0.06: Trust in the Safety of COVID-19 Vaccines  + 0.05: Motivation for Taking COVID-19 Vaccines Vaccine Literacy:  + 0.26: Having literacy Civil Rights:  + 0.27: Civil rights above all|  + 0.17: Vaccines more important than civil rights Moral Foundations:  + 0.15: Care|+ 0.45: Authority|+ 0.34: Fairness|  + 0.17: Loyalty|+ 0.10: Betrayal |+ 0.26: Subversion|  + 0.36: Harm|+ 0.08: Degradation

Profile 3: DEBUNKERS (199,014 users; 35%) Misinformation Themes: − 0.11: Unnecessary COVID-19 Vaccine − 0.06: Unsafe COVID-19 Vaccine|− 0.05: Testing of COVID-19 Vaccine − 0.05: Information about COVID-19 Vaccines is Concealed − 0.03: Effect on Immune System − 0.03: Adverse Events of COVID-19 Vaccine Themes for Eroding Trust in Vaccine: − 0.27: Lack of Trust in Role of COVID-19 Vaccines for Public Health − 0.12: Lack of Trust in Safety of COVID-19 Vaccines − 0.07: Lack of Trust in the Role of Pharmaceutical Companies in Fighting COVID-19 Infections − 0.03: Lack of Trust in Effects of COVID-19 Vaccines Vaccine Literacy: − 0.04: Having literacy Civil Rights: − 0.26: Civil rights above all Moral Foundations: − 0.35: Degradation |− 0.80: Subversion | − 0.09: Authority|− 0.11: Purity| − 0.52: Harm| − 0.09: Care| − 0.07: Cheating|− 0.09: Betrayal Profile 5: SKEPTICS (41,044 users; 7%) Misinformation Themes:  + 0.20: Information about COVID-19 Vaccines is Concealed  + 0.06: COVID-19 Vaccine Ingredients|+ 0.04: Adverse Events  + 0.05: Testing of COVID-19 Vaccine|+ 0.04: Effect on Immune System  + 0.04: Not effective vaccine|+ 0.04: Unnecessary COVID-19 Vaccine Themes for Eroding Trust in Vaccine:  + 1.31: De-Motivation in Taking the COVID-19 Vaccines  + 0.07: Lack of Trust in Safety of COVID-19 Vaccines  + 0.04: Lack of Trust in the Role of Pharmaceutical Companies in Fighting COVID-19 Infections  + 0.03: Lack of Trust in Testing/Effects of COVID-19 Vaccines Vaccine Literacy: + 0.23: Lacking literacy| − 0.09: Having literacy Civil Rights: + 0.67: Civil rights above all Moral Foundations  + 1.68: Subversion|+ 1.00: Harm|+ 1.37: Betrayal|+ 0.11: Cheating|  + 0.14: Degradation|+ 0.06: Purity|+ 0.20: Fairness|+ 0.31 Authority

Profile 4: MISINFORMERS (5617 users; 1%) Misinformation Themes:  + 0.83: Information about COVID-19 Vaccines is Concealed  + 0.57: Testing of COVID-19 Vaccine  + 0.41: COVID-19 Vaccine Ingredients  + 0.40: Effect on Immune System  + 0.28: Adverse Events of COVID-19 Vaccine  + 0.27: Unsafe COVID-19 Vaccine  + 0.25: Not effective COVID-19 Vaccine  + 0.23: Unnecessary COVID-19 Vaccine  + 0.19: Alternatives to COVID-19 Vaccine Themes for Eroding Trust in Vaccine:  + 1.54: De-Motivation in Taking the COVID-19 Vaccines  + 0.85: Lack of Trust in Safety of COVID-19 Vaccines  + 0.41: Lack of Trust in the Role of Pharmaceutical Companies in Fighting COVID-19 Infections  + 0.36: Lack of Trust in Effects of COVID-19 Vaccines  + 0.31: Lack of Trust in Role of COVID-19 Vaccines for Public Health  + 0.20: Lack of Trust in Testing of COVID-19 Vaccines Themes for Building Trust in Vaccine: − 0.13: Trust in the Safety of COVID-19 Vaccines − 0.10: Trust in Role of COVID-19 Vaccines for Public Health − 0.07: Trust in Effects of COVID-19 Vaccines − 0.07: Motivation for Taking COVID-19 Vaccines  + 0.03: Trust in the Role of Pharmaceutical Companies in Fighting COVID-19 Infections Vaccine Literacy:  + 0.57: Having literacy |+ 0.55: Lacking literacy Civil Rights:  + 2.46: Civil rights above all  + 0.53: Vaccines more important than civil rights Moral Foundations:  + 1.93: Betrayal |+ 4.82: Harm|+ 1.68: Authority| − 0.06: Care  + 0.67: Cheating|+ 4.90: Subversion|+ 0.98: Degradation |  + 2.02 Fairness|+ 0.87 Loyalty |+ 0.52 Purity

Five hesitancy profiles were derived for the HPV vaccine, interpreted by the prototypical vector user representations, available in Table 10, generated as the centroid of each cluster corresponding to a profile. Similarly, five hesitancy profiles were recognized for the COVID-19 vaccine, with interpretations made possible by their corresponding prototypical vector user representations, available in Table 11.

Discussion

Qualitatively, vaccine confidence for both HPV and COVID19 was expressed in framings covering a range of themes from general vaccine safety issues to individual level concerns (unsafe, adverse effects, ingredients, overwhelming the immune system), to vaccine development, testing and transparency concerns, to questioning vaccine efficacy, whether vaccinating is necessary, and alternatives. We therefore uncover not only vaccine confidence themes on social media, which may have been recognized in prior literature (Dunn et al., 2017; Islam et al., 2020; Shapiro et al., 2017; Wawrzuta et al., 2021), but we also uncover users’ stance toward those vaccine confidence themes across millions of users at scale.

Quantitatively, we inferred a larger number of hesitancy framings for the COVID-19 vaccine (113 framings) than for the HPV vaccine (64 framings), perhaps because we operated on a larger number of tweets in the $C_T^{COVID-19}$ collection (5,865,046 unique tweets), which is an order of magnitude larger than the number of tweets in the collection $C_T^{\mathit{HPV}}$ collection (422,078 unique tweets). But we also believe that the quantitative differences may be explained by the question/answering framework we designed to find the vaccine confidence framings, illustrated in Fig. 1. We noticed that the experts judged a larger number of tweets relevant to the COVID-19 vaccine questions than they did for the HPV vaccine questions. This provides a second, and perhaps better explanation for why we obtained a different number of framings between the two vaccines, highlighting the finding that in Twitter discourse, people have a greater number of vaccine confidence issues for the COVID-19 vaccine than for the HPV vaccine.

The vaccine hesitancy framings that we collected in $framing{s}^{COVID-19}$ and $framing{s}^{\mathit{HPV}}$ allowed us to discover not only the fact that the distribution of framing categories varies between the vaccines, but also the fact in Twitter discourse about vaccine confidence is impacted not only by misinformation, but also by the erosion of trust in vaccines. Some of the questions from VCR invite confidence framings that rely on misinformation. For example, Q11: “Homeopathic medicines are an effective alternative to conventional vaccines” produces as answers only misinformation framings, as shown in Fig. 3. However, other questions, such as Q9: “The more people who get vaccinated the greater the protection against disease” are answered by framings that either build or erode trust in vaccines clearly showcasing vaccine literacy problems. Not surprisingly, framings answering the question Q13: “It is important that people are able to make their own decisions about vaccination” are dominated by civil rights issues for both vaccines. Another interesting observation derived from the analysis of Fig. 3 is that framings relying on misinformation were inferred as answers to 11 questions when considering the HPV vaccine, while for the COVID-19 vaccine, misinformation was present in the framings answering 14 questions. Hence misinformation plays an important role in answering more VCR questions for the COVID-19 vaccine.

While much interest has been shown in identifying misinformation on social media platforms, relatively few studies have considered addressing the problem of identifying the specific misinformation that is propagated on Twitter or other social media platforms (Luo et al., 2019; Margolis et al., 2019; Massey et al., 2020; Reiter et al., 2018). Typically, known misinformation can be identified, through methods such as (Weinzierl & Harabagiu, 2021) by relying on Wikipedia web pages or similar sources that collect debunked specific misinformation. However, our method of identifying vaccine hesitancy framings produced an interesting byproduct, namely the discovery of framings that contained specific misinformation, which we further organized in misinformation taxonomies specific to the HPV or the COVID-19 vaccines. It is also important to note that the two misinformation taxonomies comprise typologies that discovered a greater number of misinformation themes than the typology of misinformation reported in Jamison et al (2020), which was adapted from Kata’s ontology. Interestingly, although most of misinformation concerns from the misinformation taxonomies that we derived are vaccine-specific, and so are the framings. This accounts for the generative power of Misinformers and the tailoring of the misinformation to vaccines. But vaccine confidence is not impacted only by misinformation, as we have seen in Fig. 3. From the $framing{s}^{\mathit{HPV}}$, 21.8% increased trust in the safety of vaccines (n = 21/96) while 20.8% of frames eroded trust in the safety of the HPV vaccine (N = 20/96). From the $framing{s}^{COVID-19}$, 24.1% increased trust in the safety of the vaccines (n = 27/112) while 22.3% eroded trust in the safety of vaccines (n = 25/112). This motivated our decision to derive two different trust taxonomies for each vaccine. To our knowledge, this is the first time when trust in vaccines has considered either the erosion of trust or the increase in trust. Moreover, the empirically derived taxonomies of trust in the COVID-19 and HPV vaccines reveal a number of findings. First, Twitter discourse on vaccination expressed vaccine attitudes in relation to trust to a substantial degree and in approximate equal proportions in relation to building or eroding trust across vaccines. Second, trust in vaccines was expressed across both HPV and COVID19 vaccines at the individual level (e.g., confidence in vaccine over natural immunity), the family level (e.g., vaccination protects families), to the system level (e.g., vaccine prevents cancer regardless of pharma profit, government provides and makes transparent vaccine information). Thus, a social-ecological framework contextualized trust at multiple levels on twitter (Latkin et al., 2021).

Qualitatively, some of the trust themes were also recognized in recent research on COVID-19 vaccination and trust by Latkin et al (2021). Trust in the vaccine literature has commonly been measured with single items (Larson et al., 2018), has often been operationalized by measuring credibility source (Larson et al., 2018; Sutton et al., 2020) but has rarely been operationalized or measured as perceived motivations or erosion of trust or as moral values that align. Only recently has one trust scale been developed that measures parental confidence in source credibility and trust in various sources but also measures the norms that vaccination is important for children, a protective measure that all teenagers should get vaccinated (Frew et al., 2019). It is important to remember that the hesitancy framings involving trust are unsolicited as opposed to survey results. Twitter framings that erode trust focus on instilling fear or casting doubt, de-motivating vaccination at the individual level, but also raise attention on the failure and incompetence of institutional systems and experts. This has been highlighted in Recreancy and social capital theory, which explains loss of trust and credibility in contentious public health disasters by emphasizing institutional failures of responsibility (Freudenburg, 1993).

It is important to note our usage of the implied Moral Foundations (MFs) is also new in its application to understanding vaccine confidence and hesitancy framings on social media. More importantly, we have associated each implied MFs with a stance. The stance of each author of a tweet towards the framing (s)he evokes is transferred to the MFs implied by the same framing. Based on this observation, clear moral attitudes emerge within the hesitancy profiles across both the HPV and COVID-19 vaccines. Promoters of both the HPV and COVID-19 vaccines tend to strongly accept framings, which espouse Care, Authority, Loyalty, and Fairness, with rejection of Subversion. Alternatively, Misinformers of both the HPV and COVID-19 vaccines tend to adopt framings in stark moral contrast than Promoters, whose moral foundations of Betrayal, Harm, and Subversion oppose those moral stances of Loyalty, Care, and Authority respectively. Misinformers tend to have much stronger moral stances than Promoters, which indicates morality plays a key role in the motivation of those spreading misinformation at scale. A similar pattern is found when comparing the Skeptic profiles, where moral foundations of Subversion, Betrayal, and Harm are adopted towards both the HPV and COVID-19 vaccines. In contrast, we find slightly differing moral profiles across vaccines when comparing the Trusters of the HPV Vaccine to the Ambivalent of the COVID-19 Vaccine. The two groups share in their adoption of Authority and Fairness moral foundations, but the HPV Vaccine Trusters adopt Care, while the COVID-19 Vaccine Ambivalent adopt Harm. The Debunkers share in rejection of Subversion but differ across other moral foundations. HPV Vaccine Debunkers equally tend to reject Degradation, Subversion, and Authority, while COVID-19 Vaccine Debunkers focus much more on rejecting Subversion, with a secondary focus on Harm and Fairness.

Vaccine hesitancy profiles as person centered audience segmentation for targeted campaigns

HPV and COVID19 vaccine hesitancy profiles highlight a constellation of accept and reject stances across various vaccine hesitancy framings, which will inform future messaging campaigns. The potential range of messaging targets spans inoculating against specific misinformation to tapping into moral frameworks to importantly, ways to bolster trust or debunk messaging that erodes trust. Vaccine stance identified from social media importantly, should be distinguished in its value for reflecting unsolicited attitudes toward vaccines in contrast to survey research (Hornik et al., 2020). Although five profiles were discerned for each vaccine, there are substantial differences both quantitatively in the relative size of profiles, and qualitatively, how profiles distinguish users.

Interest to public health interventionists involves strategically targeting profile members whose stance suggest their vaccine attitudes are amenable to change, or alternatively, whose vaccine attitudes may already be positive but need strengthening (Roskos-Ewoldsen et al., 2002). With this goal in mind, Promoters (21%) and Debunkers (32%) who make up more than half of the HPV vaccine users, express to a large degree support for vaccination in their high motivation to vaccinate, their trust, being vaccine literate, supportive of mandating vaccination, and appeal to moral frames of care, authority and loyalty in the case of Promoters. These users may respond to authoritative appeals, mandating vaccination, and trust appeals that emphasize the importance of public health. For COVID-19 vaccine profiles on the other hand, whose stance may also already be positive but in need of strengthening, Promoters make up a much smaller subgroup (9%) while Debunkers make up 35%. Bolstering positive vaccine attitudes may be achieved with trust messaging that emphasizes the importance of vaccinating for public health i.e., the collective, and motivating vaccination by emphasizing moral values of care (preventing harm), authority, loyalty, and fairness for Promoters. By contrast, for Debunkers—who make up a substantial subgroup (almost 200,000 users), morality messaging should be avoided with this subgroup who reject moral framings. An emphasis on moral messages with this subgroup may boomerang (Fishbein et al., 2002).

Of greater interest are profiles whose framing-stance scores suggest these profile users are ambivalent, on-the-fence, or skeptical whose members are more likely to be unvaccinated and hold vaccine attitudes amenable to change. Among HPV vaccine profiles, Trusters comprised the second largest subgroup (28%) after Debunkers and make up a substantial group in size relatively speaking (nearly 20,000). The pool of HPV vaccine users overall was smaller than that of COVID19 vaccine users – representing a vastly larger population of Twitter users. Trusters are accepting of a range of vaccine themes that build trust (e.g., motivated to vaccinate, trust in the role of public health, doctors, science, and the effects of the vaccine). These findings on frame-stance score suggest that Trusters are likely to respond favorably to messages that build trust. These users are literate and their motivation to vaccinate can be tapped possibly through moral value appeals of care, authority and fairness. Avoiding messaging that emphasizes vaccine mandates is warranted for this subgroup given Trusters’ weak yet existing stance on civil rights above all irrespective of public health circumstances. Findings from our study therefore inform not only messaging that users may respond to but also messaging that should be avoided in order to prevent potential iatrogenic message effects (Fishbein et al., 2002; Moos, 2005).

In comparison to Trusters among HPV vaccine profiles, the Ambivalent, who make up the largest subgroup (48%) of COVID-19 vaccine hesitancy profiles with 267,087 users, also reveal a weak yet existing frame-stance score across trust themes in COVID-19 vaccination. These users truly are on the fence, both accepting and rejecting trust framings and whose motivation to vaccinate needs to be strengthened. This subgroup may benefit from significantly bolstering trust and motivation coupled with inoculating against misinformation and utilizing moral appeals of authority and preventing harm. Both Trusters (HPV) and the Ambivalent (COVID19) are ripe for receiving inoculation messages against misinformation across vaccine safety, effectiveness, the testing process, transparency, ingredients, and adverse reactions. Similar misinformation domains have been recognized in the literature across HPV and COVID-19 yet never in this social media vaccine frame-stance context at this scale (Calo et al., 2021; Head et al., 2018; Jamison et al., 2020; Kim et al., 2020; Loomba et al., 2021; Massey et al., 2020; Sundstrom et al., 2021; Van der Linden et al., 2016; Zimet et al., 2013).

A smaller subgroup of vaccine profiles, the Skeptics (13% for HPV; 7% for COVID19), exhibit frame-stance scores that are accepting of most misinformation and erosion of trust framings. These users are illiterate, strongly de-motivated to vaccinate, and with whom moral values of subversion, harm and betrayal resonate as well as civil right above all irrespective of public health circumstances. Reaching these users presents more challenges. These subgroups’ vaccine stances are not as extreme as Misinformers who widely distrust vaccination and actively propagate misinformation. The Skeptics exhibit weak frame-stance scores on many fronts suggesting these could be targeted to shift vaccine attitudes.

Stengths and limitations

Strengths of the methodology for discovering vaccine hesitancy framings presented in this paper include (1) the Q/A framework that was used as a starting point to identify framings; (2) the discovery at scale of tweets that evoke the hesitancy framings and (3) the identification of the stance of the tweet authors towards vaccine confidence framings. These sophisticated natural language processing methods have the advantage of operating at the pragmatic level of language processing, in contrast with the topic processing methods, which operate at the lexical level. The framings are insightful because they enabled us to identify misinformation, trust in vaccines, civil rights and morality issues that were discussed. The framings also revealed the vaccine literacy of Twitter users. Moreover, by discovering framings at scale, this method considers the viewpoint of 138,779 Twitter users regarding their confidence in the HPV vaccine (i.e., users who specifically expressed their stance toward HPV and COVID-19 vaccine confidence) and of 665,798 Twitter users regarding their confidence in the COVID-19 vaccines. By identifying the stance of tweets towards each framing we have used a more advanced form of affect processing of the language in Tweets than the one afforded by sentiment analysis. This is because sentiment analysis operates at the lexical level, considering the positive, negative, or neutral orientation of words to infer the sentiment of a tweet. In contrast, stance identification considers the interaction of lexical, syntactic and semantic features of a tweet’s language with emotions to derive the attitude of a tweet towards a specific framing and results in the identification of subgroups of users, which is informative to a greater degree for public health campaign design. For example, a tweet may have positive sentiment, but its stance may be rejecting a given framing.

The discovery of the hesitancy profiles, which is unique to the method presented in this paper, is another notable strength, made possible by (a) the recognition of $framing{s}^{\mathit{HPV}}$ and $framing{s}^{COVID-19}$ at scale; (b) the identification of the stance of tweet authors evoking any of these framings and (c) the representation of the users authoring these tweets which takes into account the ontological commitments of the hesitancy framings. The method described in the paper has also some important limitations. First, we do not know if $framing{s}^{\mathit{HPV}}$ and $framing{s}^{COVID-19}$ represent all the framings of vaccine confidence in the two types of vaccines. To address this issue, we would need to consider additional questions that address vaccine confidence and find if new framings were inferred for the new questions. Second, the Misinformation and Trust Taxonomies have the same limitation of completeness, which may impact the completeness of the hesitancy profiles. A third limitation of the method for discovering hesitancy profiles derives from our exclusive focus on vaccine confidence, while hesitancy should also account for vaccine convenience and complacency.

While the interpretation of the hesitancy profiles is insightful, future work will need to test and validate these profiles for both user vaccination status and for profile member responsiveness to strategic messaging. Six vaccine relevant value frameworks characterize these vaccine profiles for HPV and COVID-19 vaccination—two voluntary and underutilized vaccines shown to be safe and effective. Derived from millions of tweets and unsolicited vaccine attitudes expressed on Twitter—these factors contribute as first steps to identify and characterize the complexity in vaccine hesitancy profiles at such scale. The implication with these profiles indicates promise to reach vastly larger number of unvaccinated with more precise and strategically targeted messaging. The misinformation and trust taxonomies that informed the vaccine profiles shed light on nuanced differences and similarities among subgroups and between vaccines in regard to stance on moral foundations, trust, and misinformation dimensions that contribute to vaccine attitudes and importantly informs which vaccine attitudes may be accessible and amenable to change for each subgroup. Prior research has demonstrated the importance of only some of the hesitancy framings categories in characterizing hesitancy profiles. Loomba et al (2021) as one example, were the first to quantify the impact of misinformation exposure on vaccine hesitancy.

Conclusion

In this paper we have presented a method capable of identifying how confidence in the HPV vaccine is framed in a collection of 422,078 unique tweets and how confidence in the COVID-19 vaccines is framed in a collection of 5,865,046 unique tweets. The categorization of these hesitancy framings enabled the derivation of misinformation and trust taxonomies as well as the analysis of vaccine literacy, the implied moral foundations and the tension between vaccine mandates and civil rights, which allowed us to discover several profiles of hesitancy for each vaccine. The discovery of these profiles was made possible by (a) the automatic recognition of all tweets from that evoke any of the hesitancy framings; and (b) the automatic identification of the stance the tweet authors have towards the evoked framings.

This novel methodology sheds light on what has been known but rarely modelled in this detailed, in-depth manner, namely the heterogeneity that makes up vaccine attitude profiles. Furthermore, this novel modeling approach captures user stance toward vaccine framings that uncovers the attitude orientation and informs messaging that can tap into which vaccine attitudes may be accessible (Roskos-Ewoldsen et al., 2002). These results begin to disentangle the complex attitudes shaping vaccine attitudes. Furthermore, such a person or user-centered approach to characterizing vaccine hesitancy profiles recognizes the importance of uncovering subgroups with similar vaccine hesitancy stance across multiple ontological dimensions. The patterns of vaccine hesitancy framings across multiple value frameworks inform public health messaging approaches to effectively reach profiles with promise to shift or bolster vaccine attitudes.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

All authors contributed to the study conception and design. Material preparation and data collection was performed by MW, and data analysis was performed by all authors. Multiple drafts of the manuscript were written by MW, SH, and SH, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

The authors did not receive support from any organization for the submitted work.

Data availability

We will make the taxonomies constructed for this work public on GitHub upon acceptance.

Code availability

We will make the code for this work available on GitHub upon acceptance.

Declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethics approval

All procedures performed in this study were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study was approved by The University of Texas at Dallas Institutional Review Board (IRB-21-515 stipulated that our research met the criteria for exemption #8(iii) of the Chapter 45 of Federal Regulations Part 46.101.(b)).

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent for publication

The authors affirm that research participants provided informed consent for publication of aggregated results.

Human and animal rights and Informed Consent

All procedures followed were in accordance with ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2000. Informed consent was obtained from all patients for being included in the study.