References to unbiased sources increase the helpfulness of community fact-checks

Abstract

Community-based fact-checking is a promising approach to address misinformation on social media at scale. However, an understanding of what makes community-created fact-checks helpful to users is still in its infancy. In this paper, we analyze the determinants of the helpfulness of community-created fact-checks. For this purpose, we draw upon a unique dataset of real-world community-created fact-checks and helpfulness ratings from X’s (formerly Twitter) Community Notes platform. Our empirical analysis implies that the key determinant of helpfulness in community-based fact-checking is whether users provide links to external sources to underpin their assertions. On average, the odds for community-created fact-checks to be perceived as helpful are 2.33 times higher if they provide links to external sources. Furthermore, we demonstrate that the helpfulness of community-created fact-checks varies depending on their level of political bias. Here, we find that community-created fact-checks linking to high-bias sources (of either political side) are perceived as significantly less helpful. This suggests that the rating mechanism on the Community Notes platform successfully penalizes one-sidedness and politically motivated reasoning. These findings have important implications for social media platforms, which can utilize our results to optimize their community-based fact-checking systems.

Introduction

Misinformation on social media can have real-world consequences. Among other instances, negative effects of misinformation have been repeatedly observed in the contexts of public safety^1–4, public health^5–7, and elections^8–10. Recognizing this, professional fact-checkers and fact-checking organizations (e.g., https://snopes.com, https://politifact.com) routinely fact-check social media rumors in order to identify potentially misleading information¹¹. However, due to limited resources, these organizations struggle to keep up with the volume of content generation^12–14. This problem has prompted growing interest in delegating fact-checking to non-professional contributors in the crowd^12,15–20.

A community-based approach to fact-checking is promising as it offers the capacity to conduct large numbers of fact-checks at high frequency and low costs^16,19. Additionally, it alleviates some of the trust concerns associated with conventional forms of fact-checking^21,22. The underlying rationale of community-based fact-checking draws on the concept of “wisdom of the crowds”—the idea that the biases and errors of individual users can be offset by aggregating input from a diverse group^23–25. Recent experiments show that while the judgement of individual fact-checkers can be inconsistent and unreliable²⁴, even fairly small groups of non-experts can achieve an accuracy comparable to those of experts^{15–17,19,20}.

However, while the crowd may be capable of accurately detecting misinformation, it does not automatically entail that all users will decide to do so¹⁷. Crucial challenges encompass lack of engagement in critical thinking²⁶, politically motivated reasoning^27,28, and manipulation attempts²⁹. Each of these behaviors can reduce the effectiveness of community-based fact-checking systems. For instance, there could be purposeful efforts by users to manipulate the fact-checking process by reporting social media contents as misleading based purely on non-conformance with their preconceptions or to achieve partisan ends²⁹. Furthermore, the high level of (political) polarization among social media users^30,31 can lead to significantly different interpretations of facts or even entirely different sets of accepted facts³². Hence, crucial requirements in community-based fact-checking systems are sophisticated rating systems and fact-checking guidelines that promote helpful fact-checks. However, little is known regarding the question of what makes a community fact-check helpful.

Previous research has analyzed determinants of helpfulness in the context of customer reviews on online platforms such as https://Amazon.com and https://Yelp.com, yet not for community-created fact-checks of social media posts. For example, earlier works have found that meta-characteristics such as the age of the review, the rating, and the length are important determinants of the helpfulness of customer reviews^33–38. Yet, despite apparent similarities, community-based fact-checking on social media substantially differs from customer reviews. While customer reviews commonly share (subjective) personal experiences with a product, the goal in fact-checking on social media is to carry out an accurate (and objective) assessment of a social media post. Furthermore, community-based fact-checking on social media must deal with politically biased views and a highly polarized user base^28,39. Ensuring high levels of trust with the fact-checkers’ assessments is thus comparatively more important—and more difficult to attain. To this end, modern community-based fact-checking systems typically step away from exclusively labeling potentially misleading social media content. Instead, they encourage users to write short textual fact-checking assessments and link to external sources (e.g., media outlets, scientific papers) to underpin their assertions. However, an understanding of how (and which) external sources affect the helpfulness of community-created fact-checks is largely missing. Shedding light on this question represents the goal of this research.

In the present work, we conduct an empirical analysis of the relationship between external sources in community-created fact-checks on social media and their perceived helpfulness. For this purpose, we utilize a unique dataset encompassing community-created fact-checks for social media posts obtained from X’s “Community Notes” platform (formerly “Birdwatch”)⁴⁰. In contrast to earlier (small-scale) crowd-based fact-checking initiatives, Community Notes allows users to identify misleading posts directly on X. Specifically, the Community Notes feature allows users to tag posts they consider misleading and supplement them with written notes that provide context to the post (e.g., by referring to external sources). An integral feature of Community Notes is that it implements a rating system, which provides users with the capability to rate the helpfulness of notes contributed by other users. These ratings are intended to facilitate the identification of the context which people find most helpful. For our analysis, we gather all community-created fact-checks from the Community Notes during an observation period of more than 8 months. Subsequently, we implement explanatory regression models to holistically analyze how the presence of external sources is linked to the helpfulness of the corresponding fact-check. Furthermore, we study how the helpfulness varies with regards to the level of political bias of the external sources provided in the community-created fact-checks.

Our empirical analysis implies that linking to external sources is the key determinant of helpfulness in community-based fact-checking. On average, the odds for community-created fact-checks to be perceived as helpful are 2.33 times higher if they link to external sources. Furthermore, we demonstrate that the helpfulness of community-created fact-checks varies depending on their level of political bias. Here, we find that community-created fact-checks linking to high bias sources (of either political side) are perceived as significantly less helpful. This suggests that the rating mechanism on the Community Notes platform successfully penalizes one-sidedness and politically motivated reasoning. Our findings have important implications for social media platforms that can utilize our results to optimize their fact-checking guidelines and promote helpful fact-checks.

Background

Social media has emerged as a dominant platform for sharing information online, with a global user base exceeding 4.59 billion in 2022, expected to approach six billion by 2027^9,41. The shift from traditional media to social media has essentially transferred the responsibility of content quality control from trained journalists to everyday users⁴² and created a fertile environment for the proliferation of misinformation⁴³. Numerous studies have examined the spread of misinformation on social media, suggesting that it is more infectious, that is, receives more reposts than the truth^7,11,44–48. Existing research highlights several factors contributing to this dynamic, such as emotional appeal^{46,47,49–51}, novelty of the content¹¹, automated bot activity^4,52, and partisan motives^53–55. The challenge is further exacerbated by advances in AI enabling its creation at unprecedented speed and scale^56,57.

Misinformation on social media can have severe real-world consequences, posing risks not only to individuals but also to society as a whole^{2,9,56,58–60}. Exposure to misinformation on social media has been associated with various adverse outcomes, including misperceptions during elections^8–10, threats to public safety^1–4, and risky behaviors during public health crises^5–7. For instance, during the COVID-19 pandemic, belief in vaccine-related misinformation led to greater hesitancy and refusal to vaccinate, thereby increasing individuals’ risk of infection^61,62. Hence, to protect users from being affected by misinformation and prevent harm, it is both warranted and required to implement effective countermeasures to fight against its spread⁶³.

Containing the spread of misinformation on social media necessitates accurate identification approaches¹⁶. Current measures of identifying misinformation fall under two primary categories. The first entails human-based approaches that rely on professionals or fact-checking organizations like Politifact and Snopes to verify the veracity of posts^43,64. The second category entails machine learning-based systems, which attempt to automatically classify misinformation by leveraging content-based elements (e.g., images, text, video), context-based elements (e.g., time and location), or propagation patterns^65,66. However, both approaches exhibit inherent drawbacks. Verification performed by experts typically delivers reliable results but grapples with scalability owing to the scarcity of professional fact-checkers. Conversely, while detection powered by machine learning provides scalability, it frequently underperforms in terms of prediction accuracy⁶⁷. Consequently, this indicates the necessity for approaches to fact-checking that combine accuracy with scalability.

As an alternative, recent research has suggested delegating the task of fact-checking misinformation on social media to non-experts in the crowd^{12,15–20,68}. The intuition is to harness the wisdom of crowds to identify misleading posts²⁴. Different from expert-based approaches, which are hindered by the limited pool of professional fact-checkers, community-based approaches make it possible to identify misinformation at a high volume¹⁶. Additionally, community-based fact-checking tackles the problem of user skepticism towards professional fact-checks²¹. Existing works imply that although assessments from single users might be inconsistent and unreliable, they tend to be highly accurate when collated²⁴. Experimental research has shown that crowds can be remarkably accurate in recognizing misleading content on social media platforms, indicating that even fairly small ensembles of non-experts can achieve results comparable to those of experts^15–17.

Although the crowd might be capable of correctly identifying misinformation, it does not automatically entail that all users will decide to do so¹⁷. Critical challenges encompass lack of engagement in critical thinking²⁶, politically motivated reasoning^27,28, and manipulation attempts²⁹. Each of these behaviors can reduce the effectiveness of community-based fact-checking systems. For example, users might deliberately sabotage the fact-checking mechanism by reporting social media content that refutes their personal belief, irrespective of its actual truthfulness²⁹. Furthermore, the stark polarization among social media users^30,31 might results in different interpretations of facts or even completely different sets of acknowledged facts³². Indeed, prior (small-scale) attempts towards community-based fact-checking, like TruthSquad, Factcheck.EU, and WikiTribune^69,70 were confronted with quality issues regarding user-created fact-checks^15,71. This highlights the difficulty of implementing real-world community-based fact-checking systems that preserve both high level of quality and scalability. Core requirements to counter the aforementioned challenges encompass advanced rating systems and fact-checking guidelines that foster helpful context^17,20.

In an attempt to address these challenges, the social media platform X (formerly Twitter) launched its community-based fact-checking system Community Notes (formerly known as “Birdwatch”)^40,72. Different from earlier crowd-based fact-checking initiatives, Community Notes allows users to identify misinformation directly on the platform. Community Notes also implements a rating mechanism that allows users to rate the helpfulness of other users’ fact-checks. However, given the novelty of the platform, research studying how users interact with Community Notes is still relatively scant. Early works have primarily analyzed the targets of community fact-checkers^39,72–75 and the spread of community fact-checked posts on X^57,76–80. While politically motivated reasoning might pose challenges^39,72, research suggests that community notes can successfully reduce users’ belief in false content and their intentions to share misleading posts^22,79. Our study adds by studying the link between external sources in community-created fact-checks and their helpfulness.

Research questions

Helpfulness of external sources (RQ1)

Community-based fact-checking systems can provide fact-checkers with the option to link to external sources (i. e., websites) to support their assessments of social media posts^40,72. Multiple considerations lead us to expect that fact-checks that make use of this option and do link to external sources are perceived as more helpful by other users. First, the presence of links to external sources is likely to make the fact-check more credible. Arguments tend to be more credible if they provide more information in support of the advocated position⁸¹. It is well documented that the advisees’ perception of the credibility of the advisor is an important determinant of helpfulness^82,83. Second, users may be unmotivated to invest the necessary effort of validating the assertions made in the fact-checks. In this scenario, more justifications for a position may make users more confident in their assessment⁸⁴. Third, the presence of links to external sources may reflect the fact-checkers’s involvement and knowledge. The more effort and expertise the fact-checker puts into writing the fact-check, the more likely it is that it will provide high-quality information that presents helpful context to other users. Taking these arguments together, community fact-checks that link to external sources may contain more credible arguments presented by better-informed fact-checkers that are more helpful to other users. RQ1 states:

RQ1

Are community fact-checks linking to external sources perceived as more helpful?

Political bias in external sources (RQ2)

The internet has given rise to an unprecedented prominence and popularity of politically biased sources of information⁸⁵. This raises the question of whether the effect of external sources in community-created fact-checks on helpfulness varies depending on their level of political bias. Fact-checkers can link to websites with high (e.g., partisan websites such as https://breitbart.com) or low political bias (e.g., mainstream media outlets). In general, politically biased sources tend to be perceived as less credible than non-biased sources⁸⁶. We expect that users perceive politically biased sources as less helpful because individuals have a well-developed association between credible sources and truthful information^87,88. In other words, it may be easy for individuals to rely on the simple-decision rule “experts are usually correct” when judging the likely authenticity of a fact-check. Furthermore, people are generally more persuaded by high-credibility sources^87–89, which can even make them more likely to agree with counter-attitudinal viewpoints⁸⁹. It is thus plausible that fact-checks leveraging source credibility are more likely to be perceived as helpful. Based on this reasoning, we hypothesize that fact-checks linking to external sources with high political bias are perceived as less helpful than those linking to external sources with low political bias. RQ2 states:

RQ2

Is linking to low bias sources in community fact-checks perceived as more helpful than linking to high bias sources?

Partisan asymmetry (RQ3)

Politically biased sources typically have a distinct partisan leaning, favoring either conservative (i. e., right-leaning) or liberal (i. e., left-leaning) opinions⁹⁰. At the same time, social media is characterized by “us versus them” mentality (i. e., a partisan-laden perception), which can result in the dismissal of viewpoints and facts from the political out-group⁹¹. Contextualized to community-based fact-checking, linking to politically biased sources may implicitly reveal information about the political orientation of the fact-checker—which may be polarizing to users with opposing (political) views. Assuming a high level of political diversity among the users participating in community-based fact-checking (i. e., both fact-checkers and raters), this would imply that biased sources of either political side are less likely to be perceived as helpful by a large share of users. However, the assumption of high political diversity may not hold true in real-world community-based fact-checking systems such as X’s Community Notes. The reason is that users engaging in community-based fact-checking are self-selected and, thus, are not necessarily representative of the overall user base on social media or society as a whole. There is ample evidence that the political left and the political right use social media in different ways, a phenomenon known as ideological asymmetry⁹². For example, adherents of the left have been found to be less tolerable to the spread of misinformation and have greater trust in fact-checking^53,92. It is thus conceivable that the self-selected fact-checking community is more likely to identify with one side of the political spectrum—and that it may read its own political leanings into fact-checks. However, an understanding of whether politically biased sources are more helpful if they are left-leaning or right-leaning is missing. Hence, RQ3 states:

RQ3

Are politically biased sources perceived as more helpful if they are left-leaning or right-leaning?

Data and empirical model

Data

This work examines the helpfulness of community-created fact-checks from X’s Community Notes⁴⁰. Launched to the public in October, 2021, Community Notes is a novel platform to counter misinformation circulating on X through the power of collective intelligence. The Community Notes platform allows X users to identify posts they perceive as misleading and supplement them with textual written notes, as illustrated in Fig. 1. Community Notes are limited to 280 characters where each URL (i. e., website) accounts for a single character. Community Notes can be attached to any post on X.

Fig. 1.

Example of a Community-Created Fact-Check (“Community Note”) on X.

Community Notes comes with a rating system allowing users to assess the helpfulness of notes submitted by others. Similar to other popular websites like https://Amazon.com, these user-generated ratings aim to identify and elevate the visibility of the most helpful and relevant context. To surface helpful community notes that appeal broadly across heterogeneous user groups, the feature uses a bridging-based rating system⁹³. This system calculates the helpfulness score for each Community Note based on the ratings made by others. Only notes that are rated as helpful by multiple contributors with heterogeneous rating histories are publicy displayed to all users on X⁹⁴.

Data collection

We retrieved all Community Notes and corresponding original posts from the official roll-out of the Community Notes in October 2022 until June 2023 from the Community Notes site (https://birdwatch.twitter.com). Following earlier work on helpfulness^34,35,95, we only consider fact-checks for which the helpfulness has been assessed at least once by users (i. e., fact-checks that received at least one helpful or unhelpful vote). The resulting dataset contains a total number of 41,128 Community Notes (i. e., community-created fact-checks), and 2,848,825 ratings (i. e., helpfulness votes). We utilized the historical API provided by X to correlate the postID referenced in every Community Note with the original post (i. e., the post that was subject to fact-checking) and collected the following information about each original post and the account of its author: (i) the number of followers, (ii) the number of followees, (iii) the account age, (iv) whether the user has been verified by X, (v) the post age.

Links to external sources

Subsequently, we extracted each link to external websites from the text explanations in the Community Notes (see Fig. 1). To this end, we implemented string extraction of links in the Python programming language with the help of the built-in re package. Each of the extracted links was then reduced to its domain name (e.g., https://cnn.com). Among all Community Notes, 88.66% contain at least one link to an external source. The majority of Community Notes with external sources contained only a single link, whereas 35.58% of the Notes contained multiple external links. The most common sources in Community Notes are links to Media Outlets (e.g., CNN) and Public Authorities (e.g., government websites), which represent 50.09% and 18.25% of all links in our dataset, respectively. This is followed by Social Media Posts (13.97%; e.g., other posts on X), Scientific Literature (7.04%; e.g., articles in scientific journals), Encyclopedias (5.82%; e.g., Wikipedia), and Third-Party Fact Checkers (3.34%; e.g., snopes.com). A small fraction (1.50%) of links pointed to sources not captured by these categories and were grouped as Other.

Political bias

To determine the political slant of the external sources, we utilized the website Media Bias/Fact Check (https://mediabiasfactcheck.com), which provides assessments of political bias (left and right) for a great deal of websites. The bias ratings from Media Bias/Fact Check are a common choice in previous literature⁹⁶ and are based on criteria such as the factuality of reporting, one-sidedness, and strength of political affiliations. We used Media Bias/Fact Check to collect information about (i) the bias magnitude (low, medium, high), and (ii) the bias direction (left, undirected, right) of the external sources in Community Notes.

By matching the bias rating from Media Bias/Fact Check to the extracted domain names, we were able to obtain bias scores for 50.35% of all links in Community Notes. Figure 2 illustrates the distribution of the detected bias magnitudes and directions. External sources with medium bias are most common in our sample (51.51%), followed by low bias (39.82%). Notes containing highly biased sources are relatively rare (8.67%). Regarding the bias direction, left-leaning external sources are more prevalent in Community Notes with approximately 52% (11,089) of Notes having a clear left-leaning bias, while only 13.61% of Notes have a clear right-leaning bias. Approximately 34.35% of external sources are politically neutral (i.e., undirected bias). Examples of the most common domains referenced in Community Notes are reported in Table 1.

Fig. 2.

Distribution of political biases in Community Notes. (A) Bias magnitude ordered from Low to High. (B) Bias direction, separated into Left, Undirected, and Right.

Table 1.

Most frequent domains referenced in the text explanations of Community Notes.

Domain	Frequency
Overall
twitter.com	5747
wikipedia.org	2298
apnews.com	1220
snopes.com	1127
youtube.com	1086
Low bias
wikipedia.org	2298
reuters.com	979
cdc.gov	715
nature.com	524
thehill.com	340
Medium bias
apnews.com	1220
snopes.com	1127
nytimes.com	835
npr.org	819
washingtonpost.com	765
High bias
dailymail.co.uk	295
state.com	179
giszodo.com	97
vox.com	97
washingtonexaminer.com	91

Variable definitions

We are interested in analyzing factors that determine the helpfulness of community-created fact-checks. To this end, the dependent variable is the number of (helpful votes), which denotes the number of users who voted “Yes” in response to the question “Is this note helpful?” The total number of users who responded to this question is denoted by .

The explanatory variables in our study can be divided into two groups: (1) variables that are given by the community-created fact-check (i. e., Community Note); and (2) variables that provide information about the original post (summary statistics and cross correlations are provided in the Supplementary Table S1 and Fig. S1).

Fact-checking variables

Our key explanatory variable is External Source, which is a binary label denoting whether a link to an external website has been provided as part of a Community Note ( if true, otherwise 0). Additionally, Media Bias/Fact Check provides ratings on the political bias of the external sources. We gather information about the magnitude and direction of the political biases for each Community Note. The resulting variable ranges from 0 to 2. Here a value of 0 refers to sources with low bias, a value of 1 refers to sources with medium bias, and a value of 2 refers to high biased sources (in either political direction). If a Community Note contains multiple external links, we take the mean of the bias scores of the individual links. We follow the same approach to calculate individual scores for the ( for left-leaning, 0 for undirected, and for right-leaning.) For instance, if there are two links in a Community note pointing in different directions, the mean would be zero.

We use additional control variables to account for common content characteristics of the Community Notes that may affect their helpfulness: (i) we control for the length () of the Community Notes (excluding links), (ii) we calculate the using the Gunning-Fog readability index, and (iii) we use the sentimentr package⁹⁷ in combination with the built-in NRC lexicon⁹⁸ to measure the positive/negative ⁹⁹ of the Community Note.

Original post variables

In our empirical analysis, we also control for characteristics of the original (i. e., the fact-checked) post. First, we control for the sentiment of the post (), analogous to the sentiment of the Community Note. Second, we control for the social influence of the author of the original post. The variables include the number of followers (), the number of followees (), the account age (), and how many days have passed since the post was first published (). We log-transform each of these variables to account for fat-tailed distributions in our later analysis. Additionally, we use a binary variable to control for whether the account has been verified by X ( if true, otherwise 0). Third, we use a binary variable denoting whether the original post covers a political topic ( if true, otherwise 0). To this end, we fine-tuned (and manually validated) a pre-trained TwHIN-BERT language model¹⁰⁰ for our task (see SI, Supplement D for implementation details). The classifier achieved a high macro-averaged score of 0.755 in predicting topic labels.

Model specification

Following previous research modeling helpfulness^33,35, we model the number of helpful votes, HVotes, as a binomial variable with probability parameter and Votes trials. Our key explanatory variable that allows us to analyze RQ1 is , a binary variable denoting whether a Community Note includes links to external sources ( when true, 0 otherwise). We control for multiple content characteristics of Community Notes, namely, the length (), text complexity (), and . Furthermore, we control for various characteristics of the fact-checked post. The control variables include the number of and , the account age (), whether the account is , the post age (), the post sentiment (), and a binary dummy indicating whether the fact-checked post covers a political topic ( when true, 0 otherwise). This yields the following regression model:

1

2

with intercept , error term , and representing fact-checker-specific random intercepts that capture unobserved heterogeneity across fact-checkers (e. g., differences in expertise). We estimate Eqs. 1 and 2 using maximum likelihood estimation and mixed-effects generalized linear models. To facilitate the interpretability of our findings, we z-standardize all continuous variables, allowing us to compare the effects of regression coefficients on the dependent variable measured in terms of standard deviations.

In order to analyze RQ2 and RQ3, we focus on the subset of community-created fact-checks that contain at least one link to external sources rated by Media Bias/Fact Check. The key explanatory variable that allows us to analyze RQ2 is , i. e., the severity of political bias of the links provided in Community Notes. To study RQ3, we include additional interaction term between and , which allows us to analyze whether politically biased sources are more/less helpful depending on whether they are left-leaning or right-leaning. All controls are analogous to the previous model.

Note that we analyze a wide range of additional model variants as part of an extensive set of robustness checks. In all of these analyses, we observe consistent results.

Empirical results

Helpfulness of external sources (RQ1)

We now analyze factors that determine the helpfulness of community-created fact-checks (RQ1). For this purpose, we draw upon a mixed-effects binomial regression model with the share of helpful votes as the dependent variable. The coefficient estimates for our primary explanatory variables are visualized in Fig. 3 (see Supplementary Table S3 for full estimation results).

Fig. 3.

Mixed-effects binomial regression analyzing the helpfulness of external sources in explaining the share of helpful votes. Shown are coefficient estimates with 95% CIs. Unit of analysis is the fact-check level (). Fact-checker-specific random effects are included.

Our findings suggest that the content characteristics of Community Notes play an important role in determining their helpfulness: the coefficient for Word Count (coef. = 0.012, OR = 1.012, ) is positive and statistically significant, Text Complexity (coef. = , OR = 0.961, ) is negative and statistically significant, whereas Sentiment lacks statistical significance at common significance levels. For a one standard deviation increase in the explanatory variable, the estimated odds of a helpful vote increase by 1.21% for Word Count, and decrease by 3.92% for Text Complexity. Consequently, the perceived helpfulness of Community Notes is higher if they incorporate longer explanations with less complex language.

We further note that the social influence attributed to the account that disseminates the original post has an effect on the perceived helpfulness of Community Notes. Here, the largest effect sizes are estimated for Verified, Followers, and Account Age. The odds of receiving a helpful vote for Community Notes reporting posts from verified accounts are 16.77% higher (coef. = 0.155, OR = 1.168, ) than for unverified accounts. A one standard deviation increase in the number of followers decreases the odds of a helpful vote by 19.99% (coef. = , OR = 0.800, ). A one standard deviation increase in the time since the account was published is associated with a 1.69% decrease (coef. = , OR = 0.983, ) in the estimated odds of a helpful vote. In sum, there is a lower level of helpfulness for posts from high-follower and older accounts, and a higher level of helpfulness for Community Notes fact-checking posts from verified accounts. Furthermore, we find that fact-checks for posts covering a political topic are significantly less helpful (coef. = , OR = 0.970, ).

To analyze RQ1, we assess how the presence of links to external sources in Community Notes is linked to their helpfulness. The coefficient estimate for External Source is 0.844 (OR = 2.326, ), which implies that the odds of Community Notes linking to external sources to be perceived as helpful are 2.33 times higher than for those not containing links to external sources. Notably, this is, by far, the largest effect size across all variables in our model.

Political bias in external sources (RQ2)

Next, we analyze the role of political bias regarding the helpfulness of external sources in community-created fact-checks (RQ2). For this purpose, we additionally include the variable into the regression model and restrict our analysis to Community Notes containing at least one link to an external source rated by Media Bias/Fact Check, resulting in 21,307 observations. The control variables are analogous to the previous model.

The coefficient estimates (see left panel in Fig. 4 for marginal effects, and Supplementary Table S4 for full estimation results) imply that linking to politically biased sources in community-created fact-checks is perceived as significantly less helpful. Specifically, a one standard deviation increase in is associated with a 3.05% decrease (coef. = , OR = 0.969, ) in the odds of a Community Note being perceived as helpful. To put this number into perspective, this implies that a community-created fact-check providing a link to a highly biased website (e.g., Breitbart) is approximately 10.16% less likely to be perceived as helpful than a fact-check linking to a low biased website (e.g., Reuters).

Fig. 4.

Marginal effects (with 95% CIs) of bias magnitude (left panel) and bias direction (right panel) on the share of helpful votes. Unit of analysis is the fact-check level (). Fact-checker-specific random effects are included.

Political vs. non-political posts

We now assess the role of political bias for political vs. non-political posts. For this, we include an interaction term between and into our regression model. After including the interaction, the estimated coefficient for the direct effect of is still negative and statistically significant (coef. = , OR = 0.944, ). This implies that for non-political posts, a one standard deviation increase in is associated with an 5.64% decrease in the odds of a Community Notes to be perceived as helpful. Furthermore, we observe that the coefficient of the interaction between and is positive and statistically significant (coef. = 0.062, OR = 1.064, ), which implies that the effect of bias in external sources is more positive for political posts. For political posts, we can assess the effect size by calculating the exponent of the sum of the coefficients¹⁰¹ of and . The resulting OR is 1.004, which is not statistically significant at common significance levels. This implies that bias in external sources is not statistically significantly linked to lower helpfulness for political posts.

Partisan asymmetry (RQ3)

Next we analyze whether there is a partisan asymmetry, i. e. whether politically biased sources are more helpful if they are left-leaning or if they are right-leaning (RQ3). The marginal effects are visualized in the right panel of Fig. 4 (see Supplementary Table S5 for full estimation results).

We find that the presence of both right-leaning and left-leaning biased sources in community notes significantly reduces their perceived helpfulness. However, the magnitude of the effect sizes significantly differ: the inclusion of left-leaning biased sources reduces the perceived helpfulness by, on average, 1.29% (ME , OR = 0.987, ), whereas the inclusion of right-leaning biased sources reduces the perceived helpfulness by, on average, 0.60% (ME , OR = 0.994, ). In contrast, the inclusion of sources with undirected political bias is not associated with a statistically significant change in perceived helpfulness (ME , OR = 1.002, ). Overall, this implies that external sources with the same level of political bias are rated as the least helpful if they are left-leaning.

Political vs. non-political posts

We assess the role of bias direction for political vs. non-political posts. To this end, we extend our regression model with an additional interaction term between and . When a right-leaning biased source is included in a community note concerning a political post, there is no statistically significant change () in the perceived helpfulness. However, the inclusion of a similar bias in a community note on a non-political post results in a 0.90% decrease in perceived helpfulness (ME , OR = 0.991, ). This variation between the two effects is statistically significant (). For left-leaning biased sources and sources with undirected political bias, we observe no statistically significant differences between political vs. non-political posts (each ).

Exploratory analyses & robustness checks

Multiple exploratory analyses and checks validated our results and confirmed their robustness. Specifically, we (1) controlled for fact-checks notes that contain multiple external sources, (2) analyzed helpfulness across different types of media categories (e.g., media outlets, scientific literature) and (3) additional topics (e.g., science, health, economy), and (4) conducted a variety of additional robustness checks. In all of these checks, we find consistent results and our hypotheses continue to be supported. In the following, we provide a summary of the main findings (see Supplementary Materials for details).

Multiple external sources

Our main analysis focuses on the presence of at least one external source in community-created fact-checks (i. e., a binary variable). However, 34.64% of Community Notes contain multiple external links. We explicitly control for the number of links authors provide as part of their fact-check (see Supplementary Table S6). The coefficient for is slightly positive and statistically significant (coef. = 0.040, OR = 1.041, ), implying that including multiple links to external sources in community-created fact-checks increases helpfulness.

Analysis across media types

We further explore how the helpfulness of external sources varies across different media types. For this purpose, two trained research assistants manually assigned media categories (e.g., media outlets, scientific literature) to each external source in our dataset (multiple selection possible). The most common sources in Community Notes are links to Media Outlets and Public Authorities, which represent 50.09% and 18.25% of all links in our dataset, respectively. This is followed by Social Media Posts (13.97%), Scientific Literature (7.04%), Encyclopedias (5.82%), and Third-Party Fact Checkers (3.34%). 1.50% of all links could not be assigned to these categories and were labeled as Other.

Subsequently, we repeat our regression analysis with binary variables denoting the presence of the corresponding source categories as part of the Community Notes (see Supplementary Table S6). The coefficient estimates for media types and their frequencies are visualized in Fig. 5. We find that community-created fact-checks linking to fact-checks from third-party fact-checkers (e.g., https://snopes.com) are perceived as particularly helpful (coef. = 0.265, OR = 1.303, ), followed by links to (coef. = 0.174, OR = 1.190, ), posts (coef. = 0.173, OR = 1.189, ), and (coef. = 0.085, OR = 1.089, ). Community-created fact-checks linking to are associated with a small and marginally significant increase in perceived helpfulness (coef. = 0.027, OR = 1.027, ), whereas links to (e.g., Wikipedia) show no significant effect.

Fig. 5.

Analysis across media types. (A) Results of a mixed-effects binomial regression analyzing the helpfulness of external sources across different media types. Shown are coefficient estimates with 95% CIs. Unit of analysis is the fact-check level (). Fact-checker-specific random effects are included. (B) Frequencies of media types across community notes.

These results highlight notable heterogeneity in how the type of external reference influences perceived helpfulness and suggest that source credibility, perceived authority, and contextual familiarity may play an important role in shaping community evaluations.

Analysis across fine-grained topics

While our main analysis distinguishes between political and non-political posts, we now examine how more fine-grained topics relate to the perceived helpfulness of Community Notes (see Fig. 6). To do so, we again apply a pre-trained TwHIN-BERT language model to classify posts into five topics: Politics, Health, Economy, Science, and Other (see Supplement D for implementation details). In terms of frequency (multiple topics per post possible), Politics is the most common topic, accounting for 42.10% of all Community Notes, followed by Health (16.90%), Economy (16.20%), and Science (9.82%). A total share of 33.40% of all posts fall into the Other category.

Fig. 6.

Analysis across fine-grained topics. (A) Results of a mixed-effects binomial regression analyzing the helpfulness of external sources across different topics. Shown are coefficient estimates with 95% CIs. Unit of analysis is the fact-check level (). Fact-checker-specific random effects are included. (B) Frequencies of topics across community notes.

We then re-estimate our regression model and include the fine-grained topic labels as dummy variables. The results show that Community Notes on Economy-related posts are rated most helpful (coef. = 0.108, OR = 1.114, ), followed by Science-related (0.082, OR = 1.085, ) and Health-related (0.011, OR = 1.011, ) posts. In contrast, Community Notes addressing posts on Politics are rated significantly less helpful (coef. = , OR = 0.974, ). Altogether, this reinforces our earlier results: Community Notes for non-political content tend to be perceived as more helpful than those for political content.

Additional checks

We performed an extensive series of supplementary analyses (see Supplementary Materials): (1) we controlled for outliers in the dependent variables; (2) we computed the variance inflation factors for each independent variable and validated that they are below the critical threshold of four; (3) we included quadratic effects; (4) we reestimated our models without including fact-checker-specific random effects; (5) we re-estimated our models using bootstrapped standard errors to assess the robustness of our findings to mild overdispersion in our data; (6) we repeated our regression analysis by modeling the total count of votes (helpful and unhelpful) as the dependent variable; (7) we included alternative categorizations of factual reporting and domain quality from Media Bias/Fact Check and previous research¹³. (8) we re-coded bias magnitude and bias direction into a single variable; (9) we re-coded bias magnitude into a factor variable. All the aforementioned analyses supported our findings.

Discussion

Our empirical findings contribute to research on misinformation on social media platforms and community-driven fact-checking. Whereas previous experimental studies have been primarily centered around the question of whether a crowd can accurately evaluate content on social media¹⁶, an understanding of “what makes community-created fact-checks helpful” has remained largely absent. In this study, we hypothesized that linking to external sources is the key determinant of helpfulness in community-based fact-checking (RQ1). Our primary rationale was that fact-checks are more credible and persuasive if they provide more information in support of their assertions. Furthermore, the presence of links to external sources may reflect the fact-checkers’s expertise. We found strong support for this hypothesis: on average, the odds for Community Notes to be perceived as helpful were 2.33 times higher if they link to external sources. Notably, this effect size was larger than for any other considered predictor of helpfulness (i. e., content characteristics, author characteristics).

We further analyzed whether the link between external sources in community-created fact-checks and helpfulness varies depending on their level of political bias (RQ2). Our rationale was that community-created fact-checks leveraging source credibility may be more likely to be effective. Consistent with this notion, we found that linking to high bias sources (e.g., “alternative” news outlets) in community-created fact-checks is perceived as less helpful. We also compared the helpfulness across various sub-categories of external sources. Here we found that community-created fact-checks linking to fact-checks from third-party fact-checking organizations (e.g., https://snopes.com) are perceived as particularly helpful. In contrast, community-created fact-checks linking to encyclopedias (e.g., Wikipedia) and scientific literature are perceived as less helpful.

Furthermore, our study provides new insights into the debate on whether political one-sidedness among the user base might hamper community-based fact-checking. The reason for these concerns is that users participating in community-based fact-checking may not be free of partisan motifs and political bias, but rather read their own political leanings into fact-checks. For instance, studies suggest that users may systematically reject content from those with whom they disagree politically^28,39. In this regard, our empirical findings are encouraging: although authors of community fact-checks are more likely to link to left-leaning sources, biased sources of either political side are rated as less helpful by other users (RQ3). This suggests that the rating mechanism on the community notes platform indeed penalizes one-sidedness and politically motivated reasoning. Such crowd-based correction aligns with prior observations of the “wisdom of the crowd” on other online platforms, such as Wikipedia and Stack Overflow, where collective input tends to support relatively high-quality information¹⁰².

From a practical perspective, social media platforms should closely monitor the potential of community-created fact-checking systems. While not a silver bullet and subject to limitations—such as difficulty achieving consensus on polarized topics^28,39 and limited coverage, particularly in low-visibility parts of the platform where many false claims go unchecked^14,103—these systems may still serve as a valuable complement to existing content moderation strategies. Their promise lies in three key advantages: (i) they enable the identification of misinformation at a larger scale, (ii) they address the trust problem associated with conventional fact-checking, and (iii) they identify misinformation that is of direct interest to actual social media users—and which may go unnoticed by third-party fact-checking organizations.

As such, our findings offer practical insights for the design of more effective community-based fact-checking tools. Specifically, our results suggest that ranking systems should put strong emphasis on links to unbiased external sources provided in fact-checks. In this light, the recent decision by X¹⁰⁴ to require external sources in Community Notes can be seen as a step in the right direction. However, community-based fact-checking systens could go even further. While helpful fact-checks can be identified through voting systems, accumulating high numbers of votes requires time. To accelerate this process, social media platforms could build on our our findings to develop systems that facilitate an early detection of potentially helpful fact-checks (e. g., by accounting for political bias in external sources), thereby helping to prevent unhindered dissemination of misleading social media posts.

As with any other research, our study has a number of limitations. Although we performed an extensive series of robustness checks, there may be additional unobserved factors affecting users’ perceived helpfulness of a specific fact-check that we cannot control for in our study. For instance, our approach struggles to account for subjective characteristics in the perception of raters (e. g., users’ knowledge). As such, it is important to mention that observational data cannot reveal how users cognitively process the information in fact-checks. It would thus be an interesting extension to analyze the reception of community-created fact-checks in an experiment with neurophysiological measurements¹⁰⁵. Our study is also limited by the accuracy and availability of the bias ratings for websites, specifically those from Media Bias/Fact Check. However the bias ratings from Media Bias/Fact Check are a common choice in previous literature⁹⁶ and rely on distinctive source characteristics such as the factuality of reporting. Ultimately, our conclusions are confined to the sphere of community-based fact-checking on X’s Community Notes platform. Further research is necessary to understand if the observed patterns are generalizable to other crowd-sourced fact-checking platforms.

Conclusion

Community-based fact-checking systems require sophisticated rating systems and fact-checking guidelines that promote helpful context. In this work, we empirically investigate the helpfulness of the context provided in community-created fact-checks on X’s community-based fact-checking system Community Notes. Our analysis suggests that linking to external sources is the key determinant of helpfulness in community-based fact-checking. Furthermore, we find that the rating mechanism on the Community Notes platform successfully penalizes political one-sidedness in fact-checking. Our study has important implications for social media platforms that can utilize our results to optimize their fact-checking guidelines and promote helpful fact-checks.

Author contributions

K.S. and N.P. conceived the study. K.S. analyzed the data. K.S. and N.P. wrote and reviewed the paper.

Funding

Open Access funding enabled and organized by Projekt DEAL.

Data availability

We adhere to X’s data usage policies. Raw data on community notes is available via https://communitynotes.x.com/guide/en/under-the-hood/download-data. Aggregate data analyzed during the current study is available upon request. Data requests can be directed to Kirill Solovev (kirill.solovev@wi.jlug.de).

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-09372-6.