Social Media Vaccine Websites: A Comparative Analysis of Public and Moderated Websites

Abstract

The internet is an important source of vaccine information for parents. We evaluated and compared the interactive content on an expert moderated vaccine social media (VSM) website developed for parents of children 24 months of age or younger and enrolled in a health care system to a random sample of interactions extracted from publicly available parenting and vaccine-focused blogs and discussion forums. The study observation period was September 2013 through July 2016. Three hundred sixty-seven eligible websites were located using search terms related to vaccines. Seventy-nine samples of interactions about vaccines on public blogs and discussion boards and 61 interactions from the expert moderated VSM website were coded for tone, vaccine stance, and accuracy of information. If information was inaccurate, it was coded as corrected, partially corrected or uncorrected. Using chi-square or Fisher’s exact tests, we compared coded interactions from the VSM website with coded interactions from the sample of publicly available websites. We then identified representative quotes to illustrate the quantitative results. Tone, vaccine stance, and accuracy of information were significantly different (all p < .05). Publicly available vaccine websites tended to be more contentious and have a negative stance toward vaccines. These websites also had inaccurate and uncorrected information. In contrast, the expert moderated website had a more civil tone, minimal posting of inaccurate information, with very little participant-to-participant interaction. An expert moderated, interactive vaccine website appears to provide a platform for parents to gather accurate vaccine information, express their vaccine concerns and ask questions of vaccine experts.

Many parents express concerns about the safety and effectiveness of vaccines (Freed, Clark, Butchart, Singer, & Davis, 2010). Numerous studies have shown the internet is an important source of vaccine information for concerned parents (Glanz et al., 2013; Jones et al., 2012; Larson et al., 2013). However, there is a vast amount of vaccine information on the internet; both the quality of information and the methods by which that information is presented vary significantly (Betsch et al., 2012; Ward, Peretti-Watel, Larson, Raude, & Verger, 2015). On one end of the spectrum, there are health system– and government-sponsored websites that feature accurate and carefully curated information in support of vaccination (Wilson, Atkinson, & Deeks, 2014), but these online resources generally lack interactive technologies (Grant et al., 2015). On the other end, there are staunchly anti-vaccine websites that use various interactive social media technologies to disseminate their messages (Moran, Lucas, Everhart, Morgan, & Prickett, 2016). Along this continuum, there are also numerous parenting blogs and discussion boards where parents can discuss vaccine-related topics with their peers.

It has been suggested that pro-vaccine websites should adopt interactive social media technologies to better engage parents and reinforce their messages (Grant et al., 2015; Witteman & Zikmund-Fisher, 2012). Adopting such technologies could help empower parents, build trust in the medical community, and ultimately lead to more informed vaccine decision making; however, they pose potential risks (Kaplan & Haenlein, 2010; Kata, 2010; Moorhead et al., 2013). Allowing the public to openly voice their concerns about vaccines could lead to misinformation, volatile discussions (Anderson, Brossard, Scheufele, Xenos, & Ladwig, 2014; Coe, Kenski, & Rains, 2014), and create a need to employ expert staff to carefully moderate discussions, respond to complicated questions, and correct inaccuracies. These potential risks may deter health care systems, government agencies, and pro-vaccine advocacy groups from creating and maintaining an interactive online vaccine resource for parents.

In this article, we report the results of a substudy within the Colorado Vaccine Social Media (VSM) study, a randomized controlled trial to evaluate the effectiveness of a web-based VSM intervention to increase childhood vaccine acceptance. The trial evaluated three outcomes: vaccine uptake in infants, parental vaccine attitudes and beliefs, and the tone and content of the social interaction on the website. The VSM intervention was shown to be effective in increasing vaccine rates and reducing parental vaccine concerns (Daley, Narwaney, Shoup, Wagner, & Glanz, 2018; Glanz et al., 2017). The objective of this current study was to use an observational analysis to evaluate and compare the content of the interaction on the VSM study website to a random sample of interaction extracted from publicly available parenting and vaccine-focused blogs and discussion forums.

Method

Vaccine Social Media Website

The VSM website was developed for pregnant women enrolled in Kaiser Permanente Colorado (KPCO), an integrated health care delivery system with more than 670,000 members in the Denver area (Glanz et al., 2017). Details of website development have been described previously (Shoup et al., 2015). In summary, participants were randomized to one of three groups: website with vaccine information and interactive social media components (VSM); website with vaccine information only or usual care (see Daley et al., 2018; Glanz et al., 2017, for details). This substudy focuses solely on the vaccine social media arm participants. The VSM website contained detailed vaccine information supported by science, as well as interactive components that included a blog, discussion board, chat room, and an “ask an expert” question portal (Glanz et al., 2017). For the interactive components, the research team created bimonthly blog posts covering emerging vaccine-related topics, hosted monthly chat rooms with vaccine experts, and allowed participants to privately ask our team of experts vaccine-related questions. To access the website, participants were required to have a login and password, and all of the interaction was carefully monitored by the research team to ensure privacy, filter abusive language, and prevent bullying (Glanz et al., 2017).

The observation period from this study was between September 2013 and July 2016. During this period, 542 women were consented into the study and had unlimited access to the website until their child was 2 years of age. Their partners could access the website as well. Participants had a mean age of 31.6 years, mostly White, and college educated (Table 1). One hundred eighty-nine (34.9%) visited the VSM website at least once, with a mean of 1.9 visits (SD = 1.8) and a range of 1 to 15 visits. More vaccine-hesitant parents (41.2%) visited the website than nonhesitant parents (34.0%), although the difference was not statistically significant (p = .24). Vaccine hesitancy was assessed using Parent Attitudes and Childhood Vaccines survey, which has 15 items that scale from 0 to 100. Scores of 50 or above are considered vaccine hesitant (Opel et al., 2013). All participant comments and questions were collected and archived for the analysis. From this single-study website, there were 61 threads of interactions from the VSM website with the following breakdown: ask questions of vaccine experts (n = 36; 59%), chat sessions (n = 15; 25%), blogs (n = 7; 11%), and discussion boards (n = 3; 5%). Within each thread of interaction, there were several comments or questions. During our study period, social media participation rates ranged from 1.2% to 9.1% per trimester.

Table 1.

Demographics of the Vaccine Social Media Cohort, N = 542.

Characteristics	Vaccine social media arm (n = 542)
Mother’s age at enrollment
Mean (SD), years	31.6 (4.4)
Mother’s race, n (%)a
White	476 (87.8)
Black	12 (2.2)
Asian/Pacific Islander	21 (3.9)
Multiracial/Other	31 (5.7)
Missing	2 (0.4)
Mother’s ethnicity, n (%)a
Hispanic	47 (8.7)
Non-Hispanic	489 (90.2)
Missing	6 (1.1)
Number of children, n (%)a
Pregnant with first child	246 (45.4)
Have other child/children	296 (54.6)
Missing	0 (0)
Annual household income, n (%)a
Less than $40,000	45 (8.3)
$40,000 to $80,000	163 (30.1)
$81,000 to $120,000	188 (34.7)
More than $120,000	118 (21.8)
Missing	28 (5.2)
Mother’s education, n (%)a
High school or less	21 (3.9)
Some college	72 (13.3)
College	222 (41.0)
Graduate school	227 (41.9)
Missing	0 (0)
Vaccine hesitancy by PACVb
Median (IQR)	13.0 (26.0)

Note. SD = standard deviation; IQR = interquartile range; PACV = Parent Attitudes and Childhood Vaccines.

^aPercentages represent column percentages; may exceed 100% due to rounding.

^bPACV, a validated screening questionnaire assessing vaccine hesitancy on a scale from 0 to 100; higher scores indicate a higher degree of hesitancy.

The KPCO Institutional Review Board approved the study.

Publicly Available Interactive Vaccine Websites

To contextualize interactions on the VSM website and conduct a comparative descriptive analysis, we sought to identify a representative sample of online interactions centered on vaccine-related topics. During the observation period, we selected a sample of interactions extracted from publicly available parenting and vaccine-focused blogs and discussion forums. First, we conducted separate website searches on Google, Yahoo, Bing, and Ask search engines with each of the following six phrases: “parenting blogs and parenting discussion boards,” “mommy blogs and mommy discussion boards,” “baby shots blogs and baby shots discussion boards,” “immunization blogs and immunization discussion boards,” “vaccine blogs and vaccine discussion boards,” and “vaccination blogs and vaccination discussion boards.”

Next, from each search engine and search phrase, the top 150 results (or the total number of available results if less than 150) were extracted and duplicates removed. This resulted in 1,249 websites (Figure 1). Eligible websites were U.S. based, publicly available, had search functionality, and had a blog, discussion board, and/or “ask an expert” portal on the website, resulting in 367 eligible websites. Eligible websites were then categorized into two groups based on their intended audience: Parenting/Mommy websites (n = 146) and websites with vaccine themes (n = 221).

Figure 1. Flow diagram of website eligibility.

^aEach search query was limited to the top 150 returned as these were the most relevant. ^bKeywords included blogs and discussion boards preceded by “vaccine,” “vaccination,” “immunization,” “parenting,” “mommy,” and “baby shots.” ^cIneligible criteria included website was outside the United States; not publicly available; no search functionality; no blog, discussion board, and/or ask an expert portal; PDF, book or article without an interactive option; top 10 website listing; advertisement for a conference or other promotional item. ^dExclusions from the websites included websites without vaccine content or websites without vaccine comments within the study time period of September 2013 to May 2016. ^eExclusions from the websites included websites without vaccine comments within the study time period of September 2013 to May 2016.

On each website, the next step was to identify vaccine content using the search function for the following keywords: “vaccines,” “vaccination,” “immunizations,” “immunize,” and “baby shots.” We removed websites without vaccine content and websites without vaccine-related discussions or comments within the eligible study period. This resulted in 39 Parenting/Mommy websites with vaccine content and 122 websites with vaccine themes.

Next, interactions were selected from all 39 of the Parenting/Mommy websites and a random sample of 40 (33%) of the 122 websites with vaccine themes without replacement using the RANUNI function in SAS software version 9.4 (SAS Institute, Cary, NC). On websites with blogs, a blog entry and up to 10 of the comments were randomly selected (n = 56 blogs; 71%). On websites with discussion forums, one vaccine discussion thread was randomly selected (n = 23 threads; 29%).

Analytic Procedure

Interaction was coded using a priori codes. The coding took place between May 2016 and July 2016, over 10 sessions lasting between 2 and 6 hours each. Six members of the study team (JAS, KJN, NMW, CRK, KSG, JMG) participated in the coding sessions and were blinded to the source of interaction.

We created codes for thematic categories that have been identified in previous assessments of online vaccine resources (Bean, 2011; Guidry, Carlyle, Messner, & Jin, 2015; Kata, 2010; Moran et al., 2016). These included the tone of interaction, vaccine stance, and accuracy of information (Table 2). Throughout the coding process, we also allowed for new codes to surface from the data.

Table 2.

Codes, Definitions, and Response Options of the Website Interaction.

Code	Definition	Response options
Tone	Description of the way people interact with one another within the text sample	Antagonistica Argumentativea Civil conversation Civil debate Informative Patronizing, provocative, or sarcastica Supportive
Vaccine stance	The overall position and attitude about vaccines within the text sample	Positive toward vaccines Unsure about vaccines Negative toward vaccines
Accuracy of information	Correctness of the information discussed within the text sample	Accurate information Mix of accurate information and inaccurate information Inaccurate information Cannot determine
Inaccurate information corrected	Disposition of inaccurate information posted	Inaccurate information corrected Inaccurate information partially corrected Inaccurate information not corrected

^aCoded as a negative tone.

Codes were summarized and compared between the VSM and publicly available websites using quantitative and qualitative approaches. First, the relative frequencies of the codes were compared between the two sets of websites with chi-square or Fisher’s exact statistical tests. Second, the study team identified representative quotes to contextualize the quantitative results.

Although the coding process was blinded, we wanted to mitigate any potential biases created by the possibility that the VSM and publicly available websites could be identified by the coding team. Thus, a 15% random sample of the interaction (n = 21 threads) from the VSM and publicly available websites was re-coded by an independent qualitative researcher who was blinded to the study objectives as well as the source of the interaction. The independent researcher coded for tone and vaccine stance. Accuracy of information was not assessed by the independent researcher due to the expertise required in vaccine content. Intercoder reliability was assessed with a Cohen’s kappa coefficient statistic to measure coding consistency. The independently re-coded random sample of interaction demonstrated good reliability, with values all above .70 (κ_tone = .879; κ_{vaccine stance} = .773; Landis & Koch, 1977).

Results

There were significant differences in the coded interactions for tone, vaccine stance, accuracy of information, and corrected inaccurate information between the public websites and VSM website (Table 3). The tone on public websites was more antagonistic, while the tone on the VSM website was more informative. On the VSM website, vaccine stance was significantly more positive, information was more accurate, and any inaccurate information was corrected. Of the 79 public websites, 3 were moderated, 64 were not moderated, and moderation status could not be determined on 12 websites. Of the 40 websites that offered a vaccine theme, 11 were classified as anti-vaccine, 19 were pro-vaccine, and 10 were neutral. Parenting websites covered a broad spectrum of information and were not classified as being specifically pro- or anti-vaccine.

Table 3.

Codes for Website Interaction, Frequency and Percentage, by Website Type.

Codes	Public websites, n (%)a	Vaccine social media website, n (%)a	p valueb
Tone			<.001
Antagonistic	17 (21.5)	0 (0.0)
Argumentative	9 (11.4)	0 (0.0)
Civil conversation	16 (20.3)	12 (19.7)
Civil debate	10 (12.7)	1 (1.6)
Informative	4 (5.1)	47 (77.0)
Patronizing, provocative, sarcastic	10 (12.7)	1 (1.6)
Supportive	13 (16.5)	0 (0.0)
Totalsc	79 (100)	61 (100)
Vaccine stance			<.001
Positive toward vaccines	18 (22.8)	42 (68.9)
Unsure about vaccines	24 (30.4)	13 (21.3)
Negative toward vaccines	37 (46.8)	6 (9.8)
Totalsc	79 (100)	61 (100)
Accuracy of informationd			<.001
Accurate information	17 (21.5)	53 (86.9)
Mix of accurate and inaccurate information	33 (41.8)	8 (13.1)
Inaccurate information	29 (36.7)	0 (0.0)
Totalsc	79 (100)	61 (100)
Inaccurate information corrected within website interactionsd			<.001
Inaccurate information corrected	0 (0.0)	8 (100)
Inaccurate information partially corrected	29 (46.8)	0 (0.0)
Inaccurate information not corrected	33 (53.2)	0 (0.0)
Totalsc	62 (100)	8 (100)

^aThe VSM website had 61 interactions coded and the public websites had 79 samples of interaction coded.

^bChi-square test of independence or Fisher’s exact test when expected cell count <5.

^cColumn totals; may exceed 100% due to rounding.

^dInformation was fact-checked against publicly available scientific information and gold standards.

Website Content Analyses

The quotations that appear below may have been shortened for clarity (where indicated) but are otherwise presented with fidelity to their original form, including emphases, misspellings, and grammatical errors.

Tone.

On the VSM website, most of the interaction was coded as civil or informative (98.3%). Below is an example of a civil, informative exchange on the VSM website between a vaccine expert and a parent about the required doses of influenza vaccine for her child:

(Post 1, Study Participant): My baby is 9 mo. old and I have a few questions about the flu vaccine. First, why does he need two? Can he just get one?

(Reply to Post 1, Study Team Member): To answer your first question, children aged 6 months to 8 years old need two shots if it’s the first time they are getting the flu vaccine. The first shot “primes” the baby’s immune system (i.e., gets the immune system ready) because it probably has never seen influenza before. The second shot …

In contrast to the VSM website, public websites were more likely to have a negative tone (1.6% vs. 45.6%). The following is an example of an antagonistic exchange from a publicly available blog:

(Post 2): You are an idiot. People should have their kids get vaccinated.

(Reply to Post 2): … YOUR the (uneducated) idiot. Sorry, not sorry. I’m guessing you don’t have children because you would have done your research just like most parents …

(Reply to reply of Post 2): … I do have a kid and you are still an idiot. You clearly are not doing research correctly.

Vaccine Stance.

Compared with the VSM website, public websites tended to display a more negative or unsure stance about vaccines. In some instances, a negative vaccine stance on public websites was conveyed through civil conversation. Below is a cordial discussion on a public website between individuals with concerns about vaccine safety.

(Post 3): … I’m delaying all vaccines for as long as possible. They are not safe. All of them. I could never inject such poisons into my child’s pure body.

(Reply to Post 3): All of them, we have a history of reactions in my family as well as a family history of allergies to some of the ingredients and my husband had a seizure after his last one when he was six. With a history (of vaccine reactions) on both sides I’m not taking any chances …

(Reply to Post 3): … We separate all of the vaccines at 2 months just to make sure non give her a reaction and if so we would know which one did. But we vaccinate! … Herd immunity only gets you so far people.

In certain instances, discussions on public websites had a positive vaccine stance but were sarcastic and antagonistic toward vaccine hesitant parents:

(Post 4): Nobody is forcing these ignorant paranoid antiscience knucle-draggers to do anything whatsoever. … If they want to put their kid in the public spaces occupied by other children they need to do their part to suppress whooping cough, measles, influenza, etc. They are shirkers …

(Reply to Post 4): … one thing that is obvious—rabid anti-vaxxers have tons of free time to write ridiculously long and meandering blog posts ALL DAY LONG on multiple sites …

The interaction from the VSM website in the following example displays a particularly negative stance on vaccines, followed by a response from the study team that used engagement in civil conversation that conveyed empathy, expertise, and honesty (Covello, 2003).

(Post 5, Study Participant): Within seconds of reading this I was put off, the fact that it was discussed on how to publish results regardless of outcome makes me wonder are you really telling us the truth? It says the rotavirus causes diarrhea, and you want me to inject my kid with all the additives, preservatives and who knows what else to “protect” them from something the CDC says most kids under three contract at least once. Then a side effect could be diarrhea, or worse and the vax is not even 100% effective. Um no thanks.

(Reply to Post 5, Study Team Member): This is a topic many parents feel passionately about, and we appreciate that you took the time to bring up these points. To your first point . . . our intent was to explain the results from two different studies showing that the rotavirus vaccines have some risks. You also brought up some really good points about weighing the risks and benefits of vaccines. We agree that it sounds odd that the vaccine could both prevent and cause diarrhea at the same time. The difference is in the severity of the diarrhea …

Accuracy of Information.

More than half of the sampled interactions on public websites contained inaccurate and uncorrected information (53.2%), some of which was conveyed through civil conversation. The following is from a public website:

(Post 6): I am curious what you all do to support LO [Little One] before and after vaccines. My 7 week old has first vaccines in two weeks. We do two vaccines at a time with monthly shot appts btwn doctor visits to spread them out.

(Reply to Post 6): Look into a charcoal poultice. I haven’t done it. … Charcoal draws out toxins, if you put a poultice over the kidneys it can draw out bad stuff. If the baby were older I’d give it orally …

(Reply to Post 6): Someone mentioned that sesame oil draws out heavy metals. I haven’t looked further into it yet though.

In contrast to the public websites, inaccurate information on the VSM website (below) was corrected by the study team (100%):

(Post 7, Study Participant): Babies are exposed to so many different things but they do not normally receive them directly into their muscles and blood stream.

(Reply to Post 7, Study Team Member): Vaccines are injected into muscle (or beneath the skin) but not into the blood stream. Doing this fortunately does “work” in that it creates the immune response that protects against those bad diseases.

Discussion

In this descriptive comparative content analysis, we observed significant differences in tone, vaccine stance, and accuracy of information between publicly available websites and the VSM website. As expected, publicly available websites tended to be contentious and have a negative stance toward vaccines with significant amounts of inaccurate and uncorrected information. Creating an expert moderated social media resource, in contrast, elicited a more civil tone and participants were less likely to post inaccurate information.

At consent into the study, it was explicitly clear to participants that the VSM website would be moderated by a research team affiliated with their health care provider. Therefore, the overall positive tone on the VSM website could be explained by the Hawthorne effect, in which people behave differently when they know they are being observed (Landsberger, 1958; McCarney et al., 2007). Alternatively, it is also possible that parents who were prone to instigate arguments and post inaccurate information chose not to participate in the study or interact on the website. While these explanations appear to threaten the validity of our results, they also suggest that an expert moderated resource promotes civility and deters antagonistic participants.

Almost all the interactions on the VSM website were between participants and the study team. There was very little participant-to-participant interaction. We have two possible explanations for this. One, creating another forum for parents to interact with other parents is not novel. As shown in our analysis, the internet is rife with other discussion forums and blogs in which parents can discuss parenting topics with their peers. Two, previous research has found that approximately 1% of people who visit social media websites participate by posting or initiating interactions (van Mierlo, 2014). In our study, the rate of social media participation ranged from 1.2% to 9.1% per trimester, suggesting a higher than expected participation rate. It is possible that more participant-to-participant interaction would be observed if the VSM website was scaled to a much larger population, such as across the entire Kaiser Permanente Health Plan or by making the website publicly available.

Our website was developed as a safe, heavily moderated environment for parents to obtain vaccine information. Therefore, it may seem skewed to compare the accuracy and tone of interactions on public websites to those of a health plan-centered website moderated by experts. Given that vaccines are a potentially sensitive and controversial topic, providers may still be reluctant to set up and implement an interactive website on vaccines out of concern that it would attract contentious discussions and become an unwieldly source of misinformation. This comparison provides evidence that a carefully moderated, vaccine-focused social media website can foster civil conversations and perhaps even deter individuals who wish to initiate arguments and spread misinformation.

Exposure to anti-vaccine messaging through social media can heighten parents’ concerns and reduce their intentions to vaccinate (Betsch, Renkewitz, Betsch, & Ulshofer, 2010; Venkatraman, Garg, & Kumar, 2015). Our data demonstrated that these harmful messages can be generated through interactions that are argumentative and contentious, as well as through civil, supportive conversations between individuals with similar views (Tan, Niculae, Danescu-Niculescu-Mizil, & Lee, 2016). At present, it is not known if either of these modes of message delivery are more influential on vaccination attitudes, intentions, or behaviors. Although research in health communications has focused on the dangers of vitriolic dialogue (Anderson et al., 2014; Kata, 2010), it may be the seemingly benign, civil interaction though which inaccurate information is disseminated that proves to be more harmful to online information seekers. Future health communication research could explore the cognitive processes by which these different modes of online communication influence parents’ vaccine attitudes and behaviors. Studies have shown that sources of vaccine information sponsored by the National Institutes of Health and Centers for Disease Control and Prevention are influential (Getman et al., 2017); these types of resources tend to lack interactive components and are rarely used by vaccine hesitant parents (Grant et al., 2015). In contrast, the VSM website was used by both hesitant and nonhesitant participants, and the tone of the interaction on the website was civil. This suggests that an expert-moderated social media website can effectively engage parents across a spectrum of vaccine attitudes and beliefs.

Our study had limitations. Although the study team was blinded to the source of interaction during the coding process, the differences in interactions between the VSM and publicly available websites were apparent. This potential for bias was assessed by independently re-coding a 15% random sample of the interactions, which demonstrated high reliability. Our study may have also lacked generalizability, since the social media website was administered in a single, integrated health care system in Colorado. Additionally, we compared patient interaction from one study website to a much larger random sample of publicly available websites. Analyzing the interaction from a single study website may have limited the study’s generalizability. Last, our analysis was limited to one thread from each of the public websites, perhaps limiting the representativeness of the interactions. Each thread, however, was selected using a random process, suggesting our comparisons were internally valid.

Implications for Practice

On the expert moderated VSM website, the interactions with participants were civil, and we did not observe negative vitriolic interaction between participants. Our prior analyses (Daley et al., 2018; Glanz et al., 2017) established the effectiveness of the VSM website on increasing vaccine rates and decreasing vaccine concerns among parents. The additional results from the analysis of the interaction on the VSM website suggests that large health systems, government agencies, and professional groups can adopt expert moderated social media platforms to engage parents in vaccine discussions. An expert moderated, interactive vaccine website appears to provide a safe platform from which parents can gather accurate vaccine information, express their concerns, and ask questions.