“Do not inject our babies”: a social listening analysis of public opinion about authorizing pediatric COVID-19 vaccines

Abstract

Designing effective childhood vaccination counseling guidelines, public health campaigns, and school-entry mandates requires a nuanced understanding of the information ecology in which parents make vaccination decisions. However, evidence is lacking on how best to “catch the signal” about the public's attitudes, beliefs, and misperceptions. In this study, we characterize public sentiment and discourse about vaccinating children against SARS-CoV-2 with mRNA vaccines to identify prevalent concerns about the vaccine and to understand anti-vaccine rhetorical strategies. We applied computational topic modeling to 149 897 comments submitted to regulations.gov in October 2021 and February 2022 regarding the Food and Drug Administration's Vaccines and Related Biological Products Advisory Committee's emergency use authorization of the COVID-19 vaccines for children. We used a latent Dirichlet allocation topic modeling algorithm to generate topics and then used iterative thematic and discursive analysis to identify relevant domains, themes, and rhetorical strategies. Three domains emerged: (1) specific concerns about the COVID-19 vaccines; (2) foundational beliefs shaping vaccine attitudes; and (3) rhetorical strategies deployed in anti-vaccine arguments. Computational social listening approaches can contribute to misinformation surveillance and evidence-based guidelines for vaccine counseling and public health promotion campaigns.

Introduction

Parental hesitancy about the childhood vaccination schedule has threatened vaccine-preventable disease prevention in the United States in recent decades.¹ For diverse reasons ranging from mistrust in government and the pharmaceutical industry to a preference for a “natural” lifestyle, parents are increasingly declining or delaying vaccines for their children. The COVID-19 pandemic has further disrupted childhood immunization in multiple ways: In addition to many missed well-child visits at which routine immunization happens and dips in kindergarten vaccination coverage,² the COVID-19 vaccine has prompted new parental concerns that threaten to spill over into attitudes and decisions about the existing childhood immunization schedule.³ The pandemic also increased political polarization around vaccination, amplifying policy tension around parental freedoms vs the public good.⁴

In this context, designing effective vaccination counseling guidelines, public health campaigns, and school-entry mandates requires a nuanced understanding of the “information ecology” in which parents are making vaccination decisions. Social media and other online channels, which are major drivers of vaccine hesitancy,⁵ produced a flood of both accurate and inaccurate information during the COVID-19 pandemic.⁶ This “infodemic” cast widespread doubt on the safety and effectiveness of COVID-19 vaccines, particularly for children.⁷ Anti-vaccine activist groups also undermined trust in healthcare professionals and vaccine promotion.⁸ Building on the World Health Organization's infodemic management framework,⁹ medical communication researchers have stressed the importance of conducting “social listening” to track misinformation and translate insights into actionable best practices.¹⁰

A critical outstanding question in the field remains how best to “catch the signal” about circulating misinformation in order to effectively counter it in clinical practice and public health promotion. Machine learning and natural language processing (NLP) methods such as topic modeling have previously been used to analyze online COVID-19 discourse.^11-13 This method allows for the detection of latent topics, or “signals,” within large, “noisy” social media data streams, such as tweets. Pairing topic modeling with thematic and discursive analysis allows for further sensemaking. This approach has enabled researchers to discover trends in public attitudes toward vaccination, diagnose the harms of misinformation, and generate hypotheses about the mechanisms by which misinformation operates.^14,15

In this study, we applied topic modeling to a novel dataset: public comments submitted through the US government's regulations.gov portal in response to two planned meetings of the United States Food and Drug Administration’s (FDA’s) Vaccines and Related Biological Products Advisory Committee (VRBPAC) to extend emergency use authorization (EUA) of the Pfizer-BioNTech COVID-19 mRNA vaccine to children aged 5-11 years (October 2021) and 6 months to 4 years (February 2022). In advance of public hearings or committee meetings pertaining to federal regulatory issues, an online docket is opened on regulations.gov to accept public comments about the rulings. While a typical docket receives a median of 242 public comments,¹⁶ over 170 000 public comments were published on the two dockets related to the two scheduled VRBPAC meetings. Although prior studies have manually analyzed smaller numbers of public comments submitted to regulations.gov,^17,18 only one other study that we found conducted a large-scale analysis using machine learning methods.¹⁹

Our aim was to use this large, organically generated, and untapped dataset of public comments to detect signals about parental concerns and misinformation regarding the pediatric COVID-19 vaccines. To this end, we united the ability of NLP to distill large text datasets into latent topics with the ability of thematic and discursive analysis to translate these topics into meaningful insights. These insights can then be deployed to improve the responsiveness and precision of public health messaging, immunization services, and pediatric vaccine counseling.

Data and methods

Data

We used the regulations.gov application programming interface (API) to download all public comments published on two dockets (links to the API and dockets are available at the study's Open Science Framework [OSF] site).²⁰ Docket FDA-2021-N-1088 was created on October 13, 2021, and remained open for public comment until October 25, 2021. The purpose of the associated meeting was to amend the EUA for the Pfizer-BioNTech COVID-19 mRNA vaccines for administration to children aged 5-11 years. Docket FDA-2022-N-0082 was created on February 4, 2022, and remained open until February 14, 2022, following the announcement of the associated meeting's postponement. The planned purpose of this meeting was to further amend the EUA for vaccine administration to children aged 6 months to 4 years.

Analytic approach

Topic modeling

To prepare the regulations.gov dataset for topic modeling, we first removed numbers, special characters, URLs, and stop words (ie, common words with little meaning, such as “and,” “for,” and “what”). We then tokenized the comments into unigrams (ie, split comments into single-word units) and lemmatized them (ie, converted words into their root form) using the Python library spaCy.²¹ We subsequently deduplicated comments to prevent generating topics that reflected repetition due to duplication rather than repetition due to semantic relatedness.²² Some duplicates seemingly originated from templates distributed by anti-vaccine entities to encourage commenting; templates were either duplicated exactly or edited to result in what we termed “fuzzy duplicates.” We removed exact duplicates and retained fuzzy duplicates to strike a balance between ensuring a robust topic model and capturing variation among template-inspired comments. In an exploratory analysis (see Supplementary material, Table S1, Figures S1 and S2), we estimated the prevalence of fuzzy duplicates related to the most frequently duplicated template comment by computing Ratcliff/Obershelp text sequence similarity between this template and all other comments in the dataset.²³

We generated topics using a latent Dirichlet allocation (LDA) topic modeling algorithm provided by the differential language analysis toolkit Mallet interface.²¹ Latent Dirichlet allocation groups words with high probabilities of co-occurrence into a pre-specified number of interpretable topics. Each text input to the model (in this case, each comment) is described by a probability distribution of topics, and each topic is described by a distribution of highly associated words.²⁴ While other NLP approaches are well suited to generating and validating highly predictive models, LDA outperforms other topic modeling methods when the goal is to generate insights, producing qualitatively superior topics.²⁵ Following established LDA model selection practices,²⁶ we estimated eight candidate models with 25, 50, 75, 100, 125, 150, 175, and 200 topics. We also designated the words “child,” “covid,” and “vaccine,” as stop words to exclude, as they appeared across comments too frequently to help in differentiating topics. For each topic, we used log-likelihood word frequencies to identify the 20 most highly associated words, and we used comment-level probability distributions to identify the 10 most representative comments. All co-authors reviewed the candidate model outputs for semantic validity (ie, whether topics could be readily interpreted based on their most highly associated words) and granularity (ie, whether topics were broad or specific), reaching the consensus that the 100-topic model was optimal.

Thematic and discursive analysis

To facilitate thematic analysis, all authors independently assigned each of the 100 topics a semantic label based on its 20 most highly associated words and 10 most representative comments. The 10 most representative comments for each topic were also coded as negative, neutral, or positive toward the COVID-19 vaccine and/or the proposed EUA. All authors then consolidated the topic labels, discussed their interpretation, and confirmed low-quality topics to exclude due to lack of conceptual clarity. Next, all authors independently conducted a thematic analysis, grouping topics into themes and higher-level domains based on common features and meanings.²⁷ Individual topics could be grouped into multiple themes. We identified themes related to both the substantive content of comments and discursive or rhetorical characteristics.²⁸ Two authors (A.M.B. and A.M.G.) worked iteratively to finalize the mapping of topics to themes and to further consolidate, synthesize, define, describe, and interpret the themes and domains. All authors reviewed the final thematic analysis.

Results

We downloaded 130 471 comments from the October 2021 docket and 42 638 from the February 2022 docket. After removing 23 212 exact duplicate comments (4890 unique comments), 149 897 comments were included in the topic model. Out of the 100 generated topics, we dropped five low-quality topics from the thematic analysis based on the 20 most highly associated words being non-English (one topic), filler words (two topics), or words referring to regulations.gov or the VRBPAC meeting (two topics). We identified no topics that reflected positive or neutral sentiment toward the COVID-19 vaccine. Of the 10 most representative comments for each of the 100 topics (1000 total comments), only five (0.5%) were coded as neutral or positive. The deduplicated dataset, topic modeling code, and model results (including the 20 most highly associated words and the 10 most representative comments for each topic) are available at the study's OSF site.²⁰

We identified 13 themes across three high-level domains (see Figure 1). The “What” domain (five themes) captured commenters’ arguments against authorizing the COVID-19 vaccine for children. The “Why” domain (four themes) captured worldviews that may have shaped commenters’ vaccine attitudes. The “How” domain (four themes) captured discursive and rhetorical strategies underlying commenters’ approach to persuasion. Domains and themes are described below and summarized in Tables 1-3, which provide examples of topics and representative comments associated with each theme.

Figure 1.

Schematic of domains and themes emerging from analysis of public comments on the EUA of COVID-19 vaccines for pediatric use. Source: Authors’ own analysis of 149 897 public comments submitted to Docket FDA-2021-N-1088 from October 13 to 25, 2021, and Docket FDA-2022-N-0082 from February 4 to 14, 2022. The dockets pertained to amendment of the EUA for the Pfizer-BioNTech COVID-19 mRNA vaccines for administration to children aged 5-11 years and 6 months to 4 years, respectively. Notes: N/A.

Table 1.

Themes, example topics, and example comments for the “What” domain (arguments against the COVID-19 vaccine for children) emerging from thematic and discursive analysis of public comments on the EUA of COVID-19 vaccines for pediatric use.

Theme (number of topics mapped to this theme)	Example topics mapped to theme	Example comments from mapped topics
Unsafe (n = 19)	(T4) long, term, body, develop, still, new, affect, future, grow, could, impact, technology, generation, development, idea, consequence, brain, stage, growth, negative (T56) heart, blood, clot, myocarditis, cause, issue, problem, attack, stroke, damage, neurological, increase, inflammation, disorder, etc, pericarditis, cancer, bell, palsy, infertility (T81) effect, term, long, side, unknown, short, possible, possibly, worth, immediate, detrimental, known, excellent, longterm, gamble, underutilized, devoid, disqualify, certainty, undone	(T4) Do not approve the Covid vaccine for 5-12 year olds. Their bodies and brains are still developing and who knows what this vaccywill do to them. (T56) I do not support the Vaccine for children of any age. These vaccines cause blood clots, strokes, and heart attacks. (T81) Absolutely NO! The long term effects are UNKNOWN. There are so many side effects already for adults. NO ONE knows the long term effects down the road.
Rushed development process (n = 17)	(T31) experimental, drug, jab, inject, subject, inoculation, untested, unproven, stage, circumstance, unapproved, inoculate, jabbed, inhumane, innoculations, innoculation, serum, concoction, horrendous, cocktail (T58) long, term, safety, study, data, efficacy, lack, short, complete, effectiveness, determine, review, ensure, adequate, conduct, sufficient, collect, reckless, peer, longitudinal (T86) time, enough, research, rush, wait, feel, information, testing, understand, determine, period, sure, amount, comfortable, anti, sufficient, soon, conduct, diligence, collect	(T31) There is no way you should even consider giving an experimental drug to children. Look at what happened with chantex and that was an approved drug. What is wrong with you people? (T58) Are there publised peer-reviewed clinical data on safety and efficacy of vaccine in children age 11 and below? If there are, can we have access to clinical data. (T86) This vaccine has not been out long enough- nor has it been researched enough yet for children. Until it has been out longer I will take a wait and see attitude!
Ineffective (n = 7)	(T18) prevent, transmission, spread, infection, disease, symptom, contract, reduce, therefore, others, mild, neither, catch, hospitalization, transmit, spreader, severity, lessen, contraction, argument (T74) variant, vaccinated, spread, infection, prevent, unvaccinated, delta, current, likely, viral, become, infect, strain, booster, load, israel, protection, omicron, transmit, breakthrough (T79) get, vaccinate, still, spread, sick, vaccinated, fully, transmit, others, unvaccinated, catch, pass, someone, hospitalize, besides, easily, vaxxed, pas, argument, powell	(T18) It's absolutely asinine to mandate an experimental drug for anyone. The FDA has the burden of proof that the vaccine would prevent infection. The vaccine does not prevent infection nor does it prevent the spread of Covid. (T74) With vaccinated infecting unvaccinated and the children are 99.9% unaffected by Covid-19 or the Delta Variant what motive other than making money is there for vaccinating children? (T79) I strongly disagree, that kids supposed to be vaccinated. First of all they have thair own immunity system, they dont need a vaccine. You dont have to be that smart to understand, that vaccine dose not help. It is official information—that any vaccinated person can ged covid. So, i dont see any reason to vaccinate the kids.
Unnecessary (n = 17)	(T12) risk, low, high, serious, illness, extremely, complication, severe, minimal, category, reward, worth, associate, incredibly, carry, ratio, factor, burden, incidence, sever (T27) flu, like, cold, polio, common, deadly, measles, childhood, different, seasonal, pox, booster, chicken, eradicate, season, corona, influenza, mmr, similar, cure (T85) die, chance, likely, ill, less, seriously, zero, almost, become, sick, least, car, survive, accident, percent, hospitalize, severely, statistically, catch, drown	(T12) I am not in support of children being vaccinated as children are not at as severe a risk as pethaps the elderly.are or those who.underlyng conditions. (T27) It is 99.7% Recovery! Not Polio. No TB. Not Whooping Cough. No mandates for CHILDREN. Use your common sense. Not about money or politics. (T85) The 5-11 age group has no risk of mortality from COVID! They are more likely to die from a car crash or being struck by lightning. Do not move forward with unnecessary recommendations at the hands of for profit pharmaceutical companies.
There are better alternatives (n = 4)	(T37) immunity, natural, antibody, recover, herd, strong, build, protection, naturally, superior, provide, fight, robust, acquire, expose, lasting, gain, recognize, achieve, quickly (T38) treatment, effective, ivermectin, early, treat, therapeutic, doctor, protocol, proven, medication, medicine, patient, alternative, cure, vitamin, hospital, option, hydroxychloroquine, instead, zinc (T92) healthy, food, exercise, instead, care, vitamin, focus, eat, help, promote, chemical, lifestyle, etc, diet, nutrition, medicine, proper, teach, obesity, educate	(T37) What happened to bodily autonomy? What happened to natural immunity? Where is the evidence that vaccinated immunity is superior? What are the standards for herd immunity? (T38) Stop the vaccine mandates and open up medications such as Ivermectin, Hydrochloroquin and Buedesonide so the country can move on to normalcy. (T92) In your health guidance, including influenza and Covid19, please put much more focus on healthy lifestyle approaches to health—like healthy diet, proper nutrition, clean air and water, and restful sleep. Thank you.

Source: Authors’ own analysis of 149 897 public comments submitted to Docket FDA-2021-N-1088 from October 13 to 25, 2021, and Docket FDA-2022-N-0082 from February 4 to 14, 2022. The dockets pertained to amendment of the EUA for the Pfizer-BioNTech COVID-19 mRNA vaccines for administration to children aged 5-11 years and 6 months to 4 years, respectively.

Notes: Example topics and comments are keyed to the full annotated 100-topic database posted on Open Science Framework (20). “T” designates the topic number.

EUA, emergency use authorization; FDA, Food and Drug Administration.

Table 3.

Themes, example topics, and example comments for the “How” domain (discourse and rhetoric in arguments against the COVID-19 vaccine for children) emerging from thematic and discursive analysis of public comments on the EUA of COVID-19 vaccines for pediatric use.

Theme (number of topics mapped to this theme)	Example topics mapped to theme	Example comments from mapped topics
Appeals to ethos (n = 13)	(T24) science, follow, look, support, real, political, sense, scientific, actual, back, politics, money, true, agenda, behind, fear, common, ignore, statistic, truly (T63) symptom, day, sick, mild, son, two, recover, cold, daughter, positive, last, husband, fever, fine, three, mother, antibody, week, grandchild, ill (T64) dr, doctor, scientist, expert, malone, listen, robert, read, article, peter, review, physician, mccullough, md, video, technology, document, fauci, paper, university	(T24) You need to follow the real science not the science the FDA, pharma, and fauci want us to follow it's all political (T63) All 4 of my children had Covid. Harmless. No symptoms and 2 with mild runny noses. I see no reason for a vaccination. I will not choose this for my children. There is no data. (T64) Geert Vanden Bossche Robert Malone Peter McCullough Ryan Cole Byram Bridle Roger Hodkinson Pierre Kory Thousands of medical pros around the world…enough said Wake up people, lend an ear to these people, and do your jobs!
Appeals to pathos (n = 12)	(T29) stop, madness, insanity, nonsense, end, poison, immediately, tyranny, abuse, hurt, insane, mandating, pure, absolute, forcing, experimenting, craziness, killing, vaccinating, playing (T46) god, may, evil, pray, soul, love, day, jesus, mercy, name, lord, bless, hand, truth, answer, earth, judge, man, shall, conscience (T77) absolutely, wrong, push, evil, criminal, beyond, morally, idea, abuse, completely, innocent, disgust, unethical, shame, nothing, ethically, pure, immoral, level, onto	(T29) Stop this heinous madness, STOP this maniacal Idiocy…STOP making making Covid 19 Vaccines mandatory for kids and for adults. No…Stop…Cease… (T46) Jesus Christ was VERY clear about millstones. Your final judgment will be with God. Trying to save as many souls as possible from eternal damnation. (T77) It is shameful and utterly unbelievable that our supposedly free government would stoop to the totalitarian suggestion of mandatory vacintion of children 5 to 11. My prayer is that you will not carry out such an evil plan!
Appeals to logos (n = 21)	(T6) risk, benefit, outweigh, far, potential, weigh, ratio, analysis, possible, reward, associate, cost, worth, minimal, greatly, vs, known, exceed, negligible, perceived (T42) death, report, vaers, adverse, event, injury, reaction, number, cdc, thousand, accord, reporting, database, website, pull, combine, market, halt, record, actual (T73) death, cdc, report, case, state, accord, per, number, million, hospitalization, among, condition, mortality, total, website, united, compare, occur, hospitalize, comorbidities	(T6) There is no need to vaccinat 5-11. Risk don't outweigh the benefits. The financial cost don't outweigh the benefits. What are the benefits? What are all the cost $$$? What are all the risk? (T42) I am against any child receiving any COVID vaccination at this time. As stated in the VAERS website as of October 152 021, there’re has been reported: A—83 412 hospitalizations; B—24 806 permanent disabilities; C—2631 miscarriages; D—17 128 deaths. These statistics do not even include the unreported cases. (T73) Do not mandate covid vaccination for anyone, especially not for children! According to a July 19, 2021 WSJ op-ed piece by Dr. Marty Makary (Professor at Johns Hopkins), “there have been only 335 people under 18 that have died from COVID, the age group which has a population of 73.1 million. That equates to 335/73.1M = 0.00046% chance or 1 death in 218 208 children (a child is nearly twice as likely to die from a lightning strike; 1 in 138 849). Statistically children under 18 are at zero risk of death from COVID.”
Templated language (n = 15)	(T2) approval, adverse, still, vulnerable, potentially, neurological, event, teen, development, available, extremely, base, acknowledge, occur, immunological, firmly, concerned, company, nih, impact (T19) low, approval, severe, extremely, base, outweigh, significant, available, teen, pose, unknown, much, include, firmly, number, addition, large, confirm, month, uk (T36) support, evidence, unknown, yet, believe, clearly, administer, rush, call, available, current, caution, virtually, unnecessary, versus, face, exercise, reject, fear, history	(T2) I am opposed to the approval of the COVID-19 vaccines for children. Their immunological and neurological systems are still in development, making them more vulnerable to adverse effects than adults. Especially, given the seriousness of the adverse events that have occurred in teens and not acknowledged by the CDC, FDA, NIH, and the vaccine manufacturing companies. (T19) I am firmly against the approval of the COVID-19 vaccines for children. Based on available data, including the UK study on kids & COVID, the benefits do not outweigh the risks. It has been confirmed that children are at extremely low risk of getting severe COVID or long COVID, or of dying from COVID. By contrast, the data is showing the COVID vaccines are posing a significant risk to teens and young adults. This is in addition to all the unknown risks, which will only be discovered over time. (T36) I call upon our government and the regulators not to repeat mistakes from history and to reject the calls to vaccinate children against COVID-19. Extreme caution has been exercised over many aspects of the pandemic, but surely now is the most important time to exercise true caution—we must not be the generation that, through unnecessary haste and fear, risks the health of children.

Source: Authors’ own analysis of 149 897 public comments submitted to Docket FDA-2021-N-1088 from October 13 to 25, 2021, and Docket FDA-2022-N-0082 from February 4 to 14, 2022. The dockets pertained to amendment of the EUA for the Pfizer-BioNTech COVID-19 mRNA vaccines for administration to children aged 5-11 years and 6 months to 4 years, respectively.

Notes: Example topics and comments are keyed to the full annotated 100-topic database posted on Open Science Framework (20). “T” designates the topic number.

EUA, emergency use authorization; FDA, Food and Drug Administration

The “What” domain: arguments against the COVID-19 vaccine for children

The themes in the “What” domain captured five arguments against giving the COVID-19 vaccine to children: unsafe, rushed, ineffective, unnecessary, and better alternatives (Table 1). The first theme highlighted arguments that the vaccine was unsafe, referencing adverse effects purportedly caused by the vaccine in both the short term (eg, myocarditis) and the long term (eg, future infertility) or listing harmful vaccine ingredients. Many claims about side effects or harms were described as “proven,” citing “expert” websites, “scientific” papers, or the Vaccine Adverse Event Reporting System database. The second theme highlighted arguments that the vaccine development process had been rushed. The accelerated vaccine development timeline opened the door for claims that the EUA for the “experimental” or “unproven” vaccine based on “novel” mRNA technology was being sought “too soon.” Calls to action included “long-term” placebo-controlled double-blind safety trials and a “wait and see” pause on vaccinating children until study results were available.

The third theme highlighted arguments about the vaccine's ineffectiveness, particularly against novel viral variants. These comments claimed that a vaccine with less-than-complete effectiveness against infection, transmission, or mortality risk was “useless.” Evidence offered about the vaccine's ineffectiveness included references to CDC data, pharmaceutical company data, and well-known individuals who had died of COVID-19 despite being vaccinated (eg, Colin Powell). The fourth theme highlighted arguments that the vaccine was unnecessary for children due to the perceived low risk posed to them by COVID-19, the vaccine's particularly poor risk/benefit ratio for children, and the robustness of children's immune systems. Some comments contrasted the lack of necessity for COVID-19 vaccines to the necessity of vaccines against diseases perceived as higher risk to children (eg, measles and polio).

The final theme highlighted arguments for better alternatives to the COVID-19 vaccine. Comments frequently endorsed pharmaceuticals such as hydroxychloroquine and ivermectin (which had been widely debunked as effective treatments for COVID-19 by October 2021), citing their perceived safety and effectiveness compared to the vaccine. Some comments also endorsed “natural” vitamins and supplements or a general “healthy” lifestyle—including sleep, nutrition, and sunshine—as a better prevention strategy than vaccination.

The “Why” domain: worldviews shaping attitudes toward the COVID-19 vaccine for children

The themes in the “Why” domain captured four distinct beliefs underlying attitudes toward the COVID-19 vaccine: personal autonomy, innocence of childhood, mistrust in institutions, and human experimentation (Table 2). The personal autonomy theme was related to bodily autonomy, personal freedom, and government overreach, particularly in reference to COVID-19 vaccine mandates. While mandates were not set to be discussed at either of the VRBPAC meetings, many comments portrayed the EUA as a “slippery slope” or conflated it with a mandate. This theme also captured beliefs that vaccination should be a parental choice that accounts for individual preferences and medical circumstances, rather than a “one-size-fits-all” strategy. The innocence of childhood theme referenced beliefs about the “purity” of children's bodies and their right to a “normal” childhood. Comments portrayed vaccines as “toxic” substances that could “violate” children's bodies, interfere with development or puberty, and disrupt “normal” life. Some comments also referenced other COVID-19 mitigation strategies such as mask use as disruptive to “normalcy.”

Table 2.

Themes, example topics, and example comments for the “Why” domain (worldviews shaping attitudes toward the COVID-19 vaccine for children) emerging from thematic and discursive analysis of public comments on the EUA of COVID-19 vaccines for pediatric use.

Theme (number of topics mapped to this theme)	Example topics mapped to theme	Example comments from mapped topics
Personal autonomy (n = 13)	(T14) government, parent, decision, choice, care, decide, overreach, childrens, responsibility, authority, federal, parental, personal, guardian, business, responsible, healthcare, agency, reach, whether (T16) medical, choice, one, individual, fit, size, personal, family, person, every, doctor, care, medicine, different, always, religious, procedure, exemption, belief, everyone (T95) school, public, pull, attend, education, require, student, teacher, order, california, home, homeschool, remove, district, governor, requirement, private, ca, comply, homeschooling	(T14) Do not make vaccines mandatory, these decisions should be left up to the individual and the minors guardians!!!!!! This is an impingement on our rights as Americans!!!! (T16) Please do not make vaccines mandatory for children. Each individual has health considerations and religious considerations. We are as unique as our fingerprints and there is no “one size fits all” when it comes to the health of ourselves and our children. Our world is being torn apart by mandates. (T95) Vote no on Covid mandates for school children! I will pull my 3 children out of CA public schools if this is a requirement!
Innocence of childhood (n = 8)	(T10) shot, please, baby, infant, toddler, little, authorize, reconsider, innocent, small, ready, precious, grandchild, forward, extend, okay, tiny, shots, newborn, mo (T75) mask, school, family, wear, pandemic, social, protection, mental, face, society, normal, community, learn, home, member, teacher, activity, access, option, physical (T91) health, issue, life, cause, condition, affect, problem, serious, could, complication, underlying, unless, exist, threaten, major, rest, underlie, pre, lifelong, elderly	(T10) Do not inject our babies with these so called vaccines! The AHA has said that they are causing heart problems an miocardidis and our babies do not need to be subjected to this! These babies are innocent, do not endanger their lives!! (T75) I am totally and completely against kids getting vaccinated and having to wear masks. Both are dangerous to their health and their emotional and mental health. (T91) Children are at no risk of Covid unless they have underline conditions. There's more healthy kids that do not need anything.
Mistrust in institutions (n = 22)	(T11) fauci, world, control, lie, biden, gate, administration, bill, agenda, dr, gain, order, nazi, house, nih, white, plan, president, china, pay (T21) science, follow, look, support, real, political, sense, scientific, actual, back, politics, money, true, agenda, behind, fear, common, ignore, statistic, truly (T43) fda, trust, public, american, lose, agency, credibility, faith, continue, organization, job, interest, political, pharmaceutical, suppose, longer, institution, confidence, ignore, corrupt	(T11) Welcome to the perpetual war on everything brought to you by Foundation Capitalism, and the Military Industrial Biologics Vortex :- ( This is more than government over reach. It is like killing an ant with a sledge hammer. Anthony Fauci is the Josef Mengele of our time, and Bill Gates is the Joseph Goebbels. (T21) If we truly follow the science our young children do not need this vaccine and will do more harm. I urge you to follow the science and not politics. Thank you. (T43) Americans won't stand for this. You are losing credibility by the second. If you suggest a covid vaccine for children for a virus that they aren't affected by (SARS Cov-2) then you will be putting nails in the coffin of what's left in terms of credibility of the fda.
Human experimentation (n = 9)	(T26) code, nuremberg, crime, humanity, law, violation, human, violate, consent, hold, illegal, constitution, criminal, nuremburg, act, accountable, cease, experimentation, unconstitutional, break (T34) therapy, gene, experimental, mrna, alter, genetic, call, technology, dna, untested, animal, definition, inoculation, unconscionable, modification, modify, traditional, novel, massive, aka (T55) use, pig, guinea, lab, rat, experiment, human, treat, precious, animal, shield, deserve, experimentation, pawn, approved, acceptable, race, unapproved, mouse, asset	(T26) No human can be made subject to a forced medical experiment. I simply refer you to the Nuremberg Code. The penalty for violation of this code is death. That is what all complicit actors and aiders and abettors will face with the absolute certainty of tomorrow's sunrise. (T34) Nobody should be getting this INOCULATION, Especially children!!! Children should not be subject to Experimental Gene Therapy at all!!! Say NO to the Covid-19 gene therapy inoculation for EVERYONE!!! (T55) Children and pregnant women are not lab rats nor test subjects. Have you not learned anything from the Thalydomide issues?????

Source: Authors’ own analysis of 149 897 public comments submitted to Docket FDA-2021-N-1088 from October 13 to 25, 2021, and Docket FDA-2022-N-0082 from February 4 to 14, 2022. The dockets pertained to amendment of the EUA for the Pfizer-BioNTech COVID-19 mRNA vaccines for administration to children aged 5-11 years and 6 months to 4 years, respectively.

Notes: Example topics and comments are keyed to the full annotated 100-topic database posted on Open Science Framework (20). “T” designates the topic number.

EUA, emergency use authorization; FDA, Food and Drug Administration.

The mistrust in institutions theme revealed a perceived lack of credibility of government institutions (especially regulatory agencies) and the pharmaceutical industry, leading to opposition toward the FDA's vaccine authorization decisions. Comments accused these entities of “lying,” of being “corrupt,” and of having ulterior profit motives to cover up adverse event data. In particular, commenters expressed outrage and disbelief that the pharmaceutical industry was indemnified against vaccine harms, and claimed that the indemnification would not be necessary if the FDA knew the vaccines were safe, making the fact of legal liability itself a source of distrust. The final theme reflected extreme moral opposition to human experimentation, referring to the “experimental” vaccines as “poison”, “bioweapons,” or “gene therapy.” The vaccine and vaccine trials were deemed “murderous,” “genocidal,” and a “violation of the Nuremberg code.” Those carrying out vaccine trials were referred to as “Nazi” doctors, and government officials (including Dr. Anthony Fauci and President Joe Biden) were referred to as “dictators,” “criminals,” or “communists” who would be held accountable for their “crimes against humanity.”

The “How” domain: discourse and rhetoric in arguments against the COVID-19 vaccine for children

The “How” domain (Table 3) contained three themes based on the Aristotelian rhetorical literature, categorizing persuasive techniques deployed in the comments as appeals to ethos, pathos, and logos.^29,30 Appeals to ethos tap into the credibility and character of the speaker, established by referencing personal experience or moral authority, or by discrediting the credibility of opponents. For example, comments employed personal anecdotes (eg, grandchildren who suffered no ill effects due to COVID-19), references to moral values, or claims about “higher” authorities (eg, Bible verses referenced to imply that Jesus would disapprove of vaccinating children).

Appeals to pathos use emotion and affect to persuade. Several topics were characterized by expressions of “outrage” at “insane” regulators who were considering the authorization of the vaccine. Vivid imagery (eg, referring to vaccination as “playing Russian roulette”), references to religious figures, and language intended to provoke strong negative emotions (eg, guilt, shame, and fear) were common. Appeals to logos stress reasoning through facts, statistics, and proofs. Comments employing these appeals referenced risk/benefit calculations, provided statistics or cited “experts,” emphasized the “logical” or “self-evident” conclusions that could be drawn about the vaccines, or highlighted the need to follow established research practices (eg, longer-term data collection).

A fourth theme in the “How” domain highlighted the influence that anti-vaccine entities exerted through templated language. Although exact duplicate comments were excluded from the model, the retained fuzzy duplicates reflected commenters’ adoption and adaptation of this language. For example, the preprocessed text with the most exact duplicates, at 647 instances, corresponded to a template referencing a “UK study” (Table 3, Topic 19). Our exploratory analysis of fuzzy duplicates related to this template demonstrated that 0.80% of all comments had Ratcliff/Obershelp similarity equal to or greater than 0.5 (range = 0-1; see Supplement).

Discussion

We leveraged an underutilized source of public opinion—comments submitted to the US government's regulations.gov portal—to characterize opposition to the EUA of the COVID-19 vaccine for children. Our analysis spotlights the emerging influence of online information on rulemaking and the legislative process.^16,31,32 We identified three unique domains that enabled a nuanced understanding of commenters’ arguments (the “What”), their underlying worldviews (the “Why”), and the characteristics of anti-vaccine discourse (the “How”). While the specific arguments against vaccinating children aligned with the existing parental vaccine hesitancy literature,¹ all three domains have important implications for clinicians who counsel vaccine-hesitant parents and for public health entities designing public health promotion campaigns. We discuss four critical implications in the following.

First, our analysis emphasizes the need to develop and deploy infoveillance and social listening systems that can rapidly detect signals about public opinion, misinformation, and activism and translate those signals into clinical guidance. Clinician awareness of the beliefs and attitudes that are going to walk in the door can lag behind reality given the speed of the news cycle and social media. Opinion surveys and knowledge–attitudes–beliefs research that inform clinician practice guidelines take a long time to field and analyze and are mostly likely to capture only the respondent's specific self-reported beliefs (our “What” domain), not the underlying worldviews informing them or the rhetorical strategies that are most influential. Supplementing existing opinion surveys with rapid machine learning-driven scans of large data streams produced organically can supercharge clinical guidance and allow policymakers and public health officials to address emergent concerns.

Second, clinicians must bring an awareness of the multiple dimensions of vaccine hesitancy to their counseling of vaccine-hesitant parents. While most clinicians (and public health campaigns) are prepared to address specific arguments against vaccination (eg, toxic ingredients, a rushed development process), our research reveals the importance of foundational beliefs as well—which are harder both to detect in a brief clinical encounter and to address or counter using conventional health education and promotion techniques.³³ In addition to the themes of personal autonomy and the innocence of childhood identified in our “Why” domain, which have been associated with parental vaccine hesitancy in work on the moral foundations framework,^34,35 we add to this prior work by identifying foundational beliefs related to mistrust in institutions and outrage against perceived human experimentation. Erosion of trust in the government and the pharmaceutical industry was a common theme in policy debates surrounding the COVID-19 vaccine;³⁶ understanding both its source and its implications for vaccination decisions is a critical next step for improving the effectiveness of vaccine counseling. Misinformation about vaccine-based human experimentation has also long been characterized in the literature,³⁷ but the reach and power of this trope took many by surprise during the pandemic.³⁸

A third important implication of our findings is the need for a diverse set of evidence-based vaccine counseling strategies. While some parents have concerns about specific vaccine ingredients (eg, aluminum), others fear adverse effects or do not perceive the need for vaccination based on risk, benefit and harm perceptions, or preferences for “natural prevention” or “healthy living”. These concerns all require different counseling messages, something that clinicians are already well aware of. However, the foundational beliefs related to trust, autonomy, or purity that we uncovered in our second “Why” domain, as well as the rhetorical strategies we characterized in the “How” domain, must also inform approaches to and content of vaccine counseling. Both pro- and anti-vaccine messages that employ rhetorical strategies have been shown to be more effective than those that focus on facts (or misinformation) alone.^39,40 Promising vaccine counseling and promotion techniques have been developed that deploy default framing,⁴¹ social norms,⁴² motivational interviewing,⁴³ and psychological inoculation theory.⁴⁴ Protocolizing these strategies for implementation in clinical settings and public health campaigns should be a research and practice priority.

A final implication of our findings is the importance of potential spillover of specific COVID-19 vaccine concerns to the broader childhood vaccination schedule.³ In other words, will parents who were not previously vaccine hesitant now transfer worries about toxic ingredients, a rushed development process, or overwhelming a child's immune system to routine childhood immunization. If this spillover occurs, this may suggest that clinicians should separate COVID-19 vaccination promotion from discussions of the child's other vaccines. On the other hand, research on the human papillomavirus vaccine suggests that bundling multiple vaccines into a single counseling or vaccination session can be an effective way to reduce concerns about a single vaccine perceived to be problematic.⁴⁵

We are not the first team to analyze public comments submitted to health-related rulings on regulations.gov; previous studies have done so on rulings related to breast implants,¹⁸ dietary guidelines,¹⁹ and machine learning-enabled medical devices.¹⁷ While most prior studies manually analyzed a few hundred to a few thousand comments, we leveraged the efficiency of LDA to analyze nearly 150 000 comments. Our results align with one evaluation of public comments demonstrating that comments were characterized by high emotion, frequent logical flaws or inconsistencies in arguments, and low credibility.¹⁷ Another study similarly noted increases in quantity and declines in substance and utility of public comments that accompanied the ability to comment via web-based platforms.¹⁶ In contrast, an analysis of public comments on the 2020 Dietary Guidelines found that comments showed a diversity of opinions and often provided scientific (non-anecdotal) evidence for claims.¹⁹ Our work is also situated in a rich body of work examining discursive elements in online discussions of vaccines and anti-vaccine sentiment.⁵ A prior study of anti-COVID-19 vaccine online discourse identified ethos-related strategies (discrediting the establishment narrative; describing a unified front of “ordinary people” rising up against the global elite) and logos-related strategies (encouraging skeptics to “do their own research” and disproportionately representing risks and benefits of vaccination).⁴⁶ These two bodies of research support our claim that organically generated public comments and a discursive analytic approach are both conducive to generating novel insights about public opinion that can inform clinical practice and policy.

Our study has some important limitations to note. Our dataset comprises over 170 000 comments submitted by the public to two dockets on regulations.gov. Members of the public who are motivated and able to submit comments via this website and in response to this specific docket and issue are not representative of the general public, and their views on childhood vaccination and COVID-19 vaccination are similarly highly selected. However, this selection serves the goals of the study well, as we aimed to characterize discourse around opposition to the vaccine in order to inform public health promotion and clinical counseling guidance, rather than draw inferences about the overall views of the public from a representative sample of individuals.

A second limitation is using data from two very specific timeframes during the COVID-19 vaccine roll-out to distill insights about the information ecology in which parents were making and continue to make the COVID-19 vaccination decision for their children. We argue that input from the public during these high-salience moments is particularly well suited to our research goal of characterizing COVID-19 vaccine discourse. A final limitation is the subjectivity inherent to thematic and discursive analysis, including the interpretation of topics modeled via LDA. We addressed this potential limitation with an established, collaborative, and iterative process to maximize rigor and reproducibility. Although beyond the scope of our analysis, future work that incorporated sentiment analysis might also add nuance and rigor to our understanding of both thematic structure and discursive content.

Conclusion

Our results offer an instructive case study of the intense public awareness and scrutiny of the development, testing, and regulatory approval processes for pediatric COVID-19 vaccines. These processes are now playing out in plain sight and in real time, in the context of increased political and cultural polarization. Clinicians and public health agencies struggled to respond to what was, for many, an unanticipated level of resistance to the COVID-19 vaccine generally and particularly for children. Previous “playbooks” for vaccine promotion and distribution proved nearly useless. Our results point to the promise of social listening methods that can rapidly characterize the thematic (the “What” and “Why”) and discursive (the “How”) content of public opinion about the vaccine. Outputs of these analyses can inform the iterative development and testing of new playbooks detailing evidence-based clinical counseling guidelines and public health campaign messaging. As new vaccines join the public health arsenal and new pandemics disrupt society, a continued focus on the pathways through which public opinion and online discourse drive beliefs and behaviors is needed.

Supplementary material

Supplementary material is available at Health Affairs Scholar online.

Conflicts of interest

Please see ICMJE form(s) for author conflicts of interest. These have been provided as supplementary materials.