Abstract
This study evaluated the topics, accuracy, and credibility of X (formerly Twitter) Community Notes addressing COVID-19 vaccination.
Social media can magnify health misinformation, especially about vaccination.1 Platform countermeasures have included censoring, shadowbanning (limiting distribution without disclosure), and adding warning labels to problematic content. Yet, evaluating these countermeasures is challenging due to restrictive public disclosures about their inner workings.2
In late 2022, X (formerly Twitter) introduced Community Notes, a crowdsourced misinformation countermeasure. Anonymous volunteer contributors independently identify posts containing misinformation and propose corrections called “notes.” Notes labeled as helpful by contributors who disagreed on past notes (to rely on a diversity of perspectives) are shown alongside the original post.3 Because Community Notes is open source, we were able to evaluate the topics, accuracy, and credibility of notes addressing COVID-19 vaccination.
Methods
Notes from the first year of Community Notes (December 12, 2022, to December 12, 2023) were obtained from X’s public data page. We filtered for notes that were visible on X that mentioned “vaccin*” and “covid*” or “coronavirus.”
A random sample of notes was double-annotated by M.R.A. and N.D. to determine topic, accuracy, and credibility. Open coding, which entails deriving labels from review of raw data, was used to determine the primary topic of each note. Axial coding was used to resolve open codes into 4 overarching topics (adverse events, conspiracy, vaccine recommendations, and vaccine effectiveness); all labeled notes were resolved to a primary subject label, or the 4 topics described all annotated notes. Notes were categorized as entirely (scientifically supported), partially (scientifically debated), or not (scientifically unsupported) accurate. Annotators were instructed to use their training, experience, and primary sources to evaluate accuracy. Since notes require citations, top-level domains in citations were rated as having high (primary sources, such as peer-reviewed journals or government websites), moderate (reputable secondary sources, such as major news outlets or fact checkers), or low (less reputable secondary sources, such as blogs or tabloids) credibility. When notes cited multiple sources, the highest credibility domain was used. Annotations were reviewed and disagreements adjudicated by a third clinician-author (D.M.S.).
Weekly rates of notes, the prevalence of note labels with bootstrapped 95% CIs, and total view counts for noted posts were computed with Python, version 3. The study, using public data (45 CFR §46), was exempted from ethical review.
Results
Of the 45 783 notes made visible on X, 657 mentioned COVID-19 vaccination. Monthly rates increased from 22 to 186 notes during the study (Figure). Of the 205 randomly sampled notes, there was strong agreement on note topics (90% agreement, Cohen κ = 0.83), source credibility (87% agreement, Cohen κ = 0.77), and accuracy (96% agreement, Cohen κ = 0.90) before resolving disagreements.
Figure. Community Notes Addressing COVID-19 Vaccine Misinformation.
A weekly time series of the volume of notes made publicly visible between December 12, 2022, and December 12, 2024, that discussed COVID-19 vaccination is shown.
The predominant note topic was adverse events (51%; 95% CI, 44%-58%), followed by conspiracy theories (37%; 95% CI, 31%-44%), vaccine recommendations (7%; 95% CI, 4%-11%), and vaccine effectiveness (5%; 95% CI, 2%-8%). Ninety-seven percent (95% CI, 96%-99%) of notes were entirely accurate, 2% (95% CI, 0%-4%) partially accurate, and 0.5% (95% CI, 0%-1%) inaccurate. Forty-nine percent (95% CI, 42%-56%) of notes cited high, 44% (95% CI, 37%-51%) moderate, and 7% (95% CI, 4%-11%) low credibility sources.
Post view data were available for 189 of 205 posts, totaling 201 281 364 views (mean number of views, 1 064 981; 95% CI, 689 821-1 548 471). Example notes are provided in the Table.
Table. Example Community Notes With Annotations.
Original posta | Community Noteb | Note topicc | Note credibilityd | Note accuracye | Post view countf | Post creation dateg |
---|---|---|---|---|---|---|
|
|
Conspiracy | Moderate credibility | Entirely accurate | 1.6 Million | September 4, 2023 |
|
|
Adverse events | High credibility | Entirely accurate | 18.4 Million | December 17, 2022 |
The monovalent Moderna and Pfizer-BioNTech COVID-19 vaccines are no longer authorized for use in the United States. |
|
Recommendations | High credibility | Entirely accurate | 6 Million | April 19, 2023 |
|
|
Effectiveness | Moderate credibility | Entirely accurate | 2.2 Million | August 25, 2023 |
Praying for [Celebrity]
|
|
Adverse events | Low credibility | Entirely accurate | 3.0 Million | June 6, 2023 |
Reported with text slightly modified to shield online identities.
Reported verbatim.
Refers to the subject matter of the note.
Refers to the credibility of the most credible domain cited in support of the note.
Refers to the accuracy of the note.
The number of views associated with the post attached to the note and reported on X.
Refers to the date the post was published on X.
Discussion
A sample of Community Notes added to posts on X containing COVID-19 vaccination misinformation primarily addressed adverse events and conspiracy theories, were accurate, cited moderate and high credibility sources, and were attached to posts viewed hundreds of millions of times.
The US Food and Drug Administration commissioner recently urged health professionals to redouble their vaccine education efforts.4 The small number of notes addressing posts with COVID-19 vaccine misinformation suggests opportunities for health professionals to contribute to this mission via participating in Community Notes.
The primary limitation of this study is that only note quality was studied, but these attributes are predictive of effectiveness (eg, higher credibility yields greater persuasiveness5). Additional limitations include a narrow focus on COVID-19 vaccination, a small sample, human judgments were used to assess accuracy, user engagement with notes was not studied, and effects on perceptions or behaviors were not studied.
Investigations of other health topics and note influence (including unintended effects6) are needed. More social media firms should open-source their misinformation countermeasures for evaluation by independent scientists to illuminate, foster public trust in, and scale the most effective strategies.
Section Editors: Kristin Walter, MD, and Jody W. Zylke, MD, Deputy Editors; Karen Lasser, MD, Senior Editor.
Data Sharing Statement
References
- 1.Khullar D. Social media and medical misinformation: confronting new variants of an old problem. JAMA. 2022;328(14):1393-1394. doi: 10.1001/jama.2022.17191 [DOI] [PubMed] [Google Scholar]
- 2.Broniatowski DA, Dredze M, Ayers JW. “First do no harm”: effective communication about COVID-19 vaccines. Am J Public Health. 2021;111(6):1055-1057. doi: 10.2105/AJPH.2021.306288 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.About Community Notes on X. X Help Center. Accessed December 10, 2023. https://help.twitter.com/en/using-x/community-notes#:~:text=Community%20Notes%20aim%20to%20create,publicly%20shown%20on%20a%20post
- 4.Marks P, Califf R. Is vaccination approaching a dangerous tipping point? JAMA. 2024;331(4):283-284. doi: 10.1001/jama.2023.27685 [DOI] [PubMed] [Google Scholar]
- 5.Pornpitakpan C. The persuasiveness of source credibility: a critical review of five decades’ evidence. J Appl Soc Psychol. 2004;34(2):243-281. doi: 10.1111/j.1559-1816.2004.tb02547.x [DOI] [Google Scholar]
- 6.Nyhan B. Why the backfire effect does not explain the durability of political misperceptions. Proc Natl Acad Sci U S A. 2021;118(15):e1912440117. doi: 10.1073/pnas.1912440117 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Sharing Statement