Skip to main content
JAMA Network logoLink to JAMA Network
. 2024 Apr 24;331(19):1670–1672. doi: 10.1001/jama.2024.4800

Characteristics of X (Formerly Twitter) Community Notes Addressing COVID-19 Vaccine Misinformation

Matthew R Allen 1, Nimit Desai 1, Aiden Namazi 2, Eric Leas 3, Mark Dredze 4, Davey M Smith 5, John W Ayers 5,
PMCID: PMC11044002  PMID: 38656757

Abstract

This study evaluated the topics, accuracy, and credibility of X (formerly Twitter) Community Notes addressing COVID-19 vaccination.


Social media can magnify health misinformation, especially about vaccination.1 Platform countermeasures have included censoring, shadowbanning (limiting distribution without disclosure), and adding warning labels to problematic content. Yet, evaluating these countermeasures is challenging due to restrictive public disclosures about their inner workings.2

In late 2022, X (formerly Twitter) introduced Community Notes, a crowdsourced misinformation countermeasure. Anonymous volunteer contributors independently identify posts containing misinformation and propose corrections called “notes.” Notes labeled as helpful by contributors who disagreed on past notes (to rely on a diversity of perspectives) are shown alongside the original post.3 Because Community Notes is open source, we were able to evaluate the topics, accuracy, and credibility of notes addressing COVID-19 vaccination.

Methods

Notes from the first year of Community Notes (December 12, 2022, to December 12, 2023) were obtained from X’s public data page. We filtered for notes that were visible on X that mentioned “vaccin*” and “covid*” or “coronavirus.”

A random sample of notes was double-annotated by M.R.A. and N.D. to determine topic, accuracy, and credibility. Open coding, which entails deriving labels from review of raw data, was used to determine the primary topic of each note. Axial coding was used to resolve open codes into 4 overarching topics (adverse events, conspiracy, vaccine recommendations, and vaccine effectiveness); all labeled notes were resolved to a primary subject label, or the 4 topics described all annotated notes. Notes were categorized as entirely (scientifically supported), partially (scientifically debated), or not (scientifically unsupported) accurate. Annotators were instructed to use their training, experience, and primary sources to evaluate accuracy. Since notes require citations, top-level domains in citations were rated as having high (primary sources, such as peer-reviewed journals or government websites), moderate (reputable secondary sources, such as major news outlets or fact checkers), or low (less reputable secondary sources, such as blogs or tabloids) credibility. When notes cited multiple sources, the highest credibility domain was used. Annotations were reviewed and disagreements adjudicated by a third clinician-author (D.M.S.).

Weekly rates of notes, the prevalence of note labels with bootstrapped 95% CIs, and total view counts for noted posts were computed with Python, version 3. The study, using public data (45 CFR §46), was exempted from ethical review.

Results

Of the 45 783 notes made visible on X, 657 mentioned COVID-19 vaccination. Monthly rates increased from 22 to 186 notes during the study (Figure). Of the 205 randomly sampled notes, there was strong agreement on note topics (90% agreement, Cohen κ = 0.83), source credibility (87% agreement, Cohen κ = 0.77), and accuracy (96% agreement, Cohen κ = 0.90) before resolving disagreements.

Figure. Community Notes Addressing COVID-19 Vaccine Misinformation.

Figure.

A weekly time series of the volume of notes made publicly visible between December 12, 2022, and December 12, 2024, that discussed COVID-19 vaccination is shown.

The predominant note topic was adverse events (51%; 95% CI, 44%-58%), followed by conspiracy theories (37%; 95% CI, 31%-44%), vaccine recommendations (7%; 95% CI, 4%-11%), and vaccine effectiveness (5%; 95% CI, 2%-8%). Ninety-seven percent (95% CI, 96%-99%) of notes were entirely accurate, 2% (95% CI, 0%-4%) partially accurate, and 0.5% (95% CI, 0%-1%) inaccurate. Forty-nine percent (95% CI, 42%-56%) of notes cited high, 44% (95% CI, 37%-51%) moderate, and 7% (95% CI, 4%-11%) low credibility sources.

Post view data were available for 189 of 205 posts, totaling 201 281 364 views (mean number of views, 1 064 981; 95% CI, 689 821-1 548 471). Example notes are provided in the Table.

Table. Example Community Notes With Annotations.

Original posta Community Noteb Note topicc Note credibilityd Note accuracye Post view countf Post creation dateg
  • Marburg Virus is baked into the covid shots and will be activated by four 1 min pulse waves at 15GHz from 5G towers throughout the country.

  • The government has been planning this for years now and it will be “released” in October 2023.

  • #plandemic #CovidIsNotOver #marburg #5g #zombieapocalypse

Conspiracy Moderate credibility Entirely accurate 1.6 Million September 4, 2023
  • So the FDA finally came out and admitted that Pfizer’s shot for Covid causes blood clots?

  • Only 2 y late!

  • A recent published literature review on vaccines and blood clots did not find strong evidence to support this claim. However, it did find that “occurrence of blood clots in COVID-19 is up to 10 times more common than from the vaccines’ injection.”

  • https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9055170/

Adverse events High credibility Entirely accurate 18.4 Million December 17, 2022
The monovalent Moderna and Pfizer-BioNTech COVID-19 vaccines are no longer authorized for use in the United States. Recommendations High credibility Entirely accurate 6 Million April 19, 2023
  • In a society with uncorrupted legacy media every front page would announce this in letters two inches high.

  • From the CDC:

  • NEW COVID VARIANT

  • MORE LIKELY

  • TO INFECT

  • THE VACCINATED

Effectiveness Moderate credibility Entirely accurate 2.2 Million August 25, 2023
Praying for [Celebrity]
  • I can’t imagine how many individuals are dealing with similar side effects and the media doesn’t care.

  • [Celebrity] Ends Up ‘Paralyzed And Blind’ From Covid Vaccination, Sources Claim

Adverse events Low credibility Entirely accurate 3.0 Million June 6, 2023
a

Reported with text slightly modified to shield online identities.

b

Reported verbatim.

c

Refers to the subject matter of the note.

d

Refers to the credibility of the most credible domain cited in support of the note.

e

Refers to the accuracy of the note.

f

The number of views associated with the post attached to the note and reported on X.

g

Refers to the date the post was published on X.

Discussion

A sample of Community Notes added to posts on X containing COVID-19 vaccination misinformation primarily addressed adverse events and conspiracy theories, were accurate, cited moderate and high credibility sources, and were attached to posts viewed hundreds of millions of times.

The US Food and Drug Administration commissioner recently urged health professionals to redouble their vaccine education efforts.4 The small number of notes addressing posts with COVID-19 vaccine misinformation suggests opportunities for health professionals to contribute to this mission via participating in Community Notes.

The primary limitation of this study is that only note quality was studied, but these attributes are predictive of effectiveness (eg, higher credibility yields greater persuasiveness5). Additional limitations include a narrow focus on COVID-19 vaccination, a small sample, human judgments were used to assess accuracy, user engagement with notes was not studied, and effects on perceptions or behaviors were not studied.

Investigations of other health topics and note influence (including unintended effects6) are needed. More social media firms should open-source their misinformation countermeasures for evaluation by independent scientists to illuminate, foster public trust in, and scale the most effective strategies.

Section Editors: Kristin Walter, MD, and Jody W. Zylke, MD, Deputy Editors; Karen Lasser, MD, Senior Editor.

Supplement.

Data Sharing Statement

jama-e244800-s001.pdf (15.4KB, pdf)

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

Data Sharing Statement

jama-e244800-s001.pdf (15.4KB, pdf)

Articles from JAMA are provided here courtesy of American Medical Association

RESOURCES