Facts do not speak for themselves: Community norms, dialog, and evidentiary practices in discussions of COVID-19 on Reddit

Abstract

The present study sought to explore the distinct discourse norms and evidentiary practices in discussions of COVID-19 in four subcommunities on Reddit. Qualitative analysis found that communities differed in the degree to which they reinforce and augment Reddit’s platform-wide norms for dialog and evidence use. One of the three communities (r/AskTrumpSupporters) differed from the rest by establishing discourse norms for turn-taking between politically opposed users and structuring dialog around authentic questions aimed at understanding alternative points of view. Quantitative analyses revealed that this community significantly differed from the other communities in the proportion of dialogic exchanges and in the use of evidentiary practices (sourcing, source evaluation, and interpretation of evidence). Excerpts of dialog from this community are used to illustrate findings. We conclude with implications for educators interested in preparing youth to critically engage with scientific information they encounter in public discourse.

Introduction

In the current political climate, terms like “alternative facts” and “fake news” have entered public discourse as justifications for avoiding thoughtful dialog about information and its implications for personal and political decision-making. Scientific information is no exception. As the COVID-19 pandemic has shown, key sources of science news and health recommendations have come under attack, particularly on social media (Nguyen and Catalan, 2020), reinforcing concerns about polarization (Taylor et al., 2018), the growth of antiscience skepticism (Edis, 2020) and the proliferation of misinformation about the pandemic (Allington et al., 2020). Studies suggest that social media may amplify these trends due to a variety of factors, including the broad reach of influencers, the relative ease of circulation, persistent exposure to misinformation and the absence of editorial review (for systematic reviews, see Baram-Tsabari and Schejter, 2019; Suarez-Lledo et al., 2021 and Wang et al., 2019).

However, overgeneralizing the quality of information on social media runs the risk of neglecting variation that exists within and between platforms. Context matters and social media platforms vary considerably in the degree to which the spread of misinformation is controlled by moderators and users (Cinelli et al., 2020). Another way to frame the problem is not by considering the proliferation of scientific misinformation, but by the degree to which information is vetted and discussed within a particular online community. A recent experimental study (Pennycook et al., 2020) showed that misinformation about COVID-19 was shared online because users failed to think about the accuracy of information, and that prompts to consider information accuracy mitigated the effect of sharing misinformation. Dialog about information accuracy can trigger epistemic vigilance (Sperber et al., 2010), or the critical evaluation of evidence and arguments used to ground knowledge claims, by holding individuals accountable for the quality of their information and reasoning.

Online communities vary in the discourse norms governing the use of information as evidence (Fiesler et al., 2018), which can have a direct impact on the quality of critical dialog about evidence in a particular community or forum (Zhang et al., 2021). The present study aims to explore the distinct discourse norms and evidentiary practices of four online subcommunities on the social media platform Reddit. We believe that developing a better understanding of the variations between these online communities may provide important insights into communication educators interested in fostering critical dialog about evidence among youth on social media.

Literature review

Social media as a source of information: Emerging practices and platform differences

At this point, the potential for social media to facilitate the spread of misinformation is well established (Suarez-Lledo et al., 2021; Wang et al., 2019), and we have seen the emergence of a range of efforts to combat this spread. Organizations like the World Health Organization routinely intersperse their posts about health information with fact-checking reminders. Platforms like Twitter and Instagram have programmed reminders to visit the CDC website for accurate Covid information, which pop up when terms like #covid are entered or when a non-institutional source includes information about COVID. However, this still requires consumers to monitor their exposure to misinformation.

Research focused on user practices to adapt to the challenges of misinformation on social media have found that teens and young adults are aware of the need to critically evaluate information. Surveys find that participants see the need to attend to source credibility (Common Sense Media, 2017) and follow up by going directly to news sites or searching for additional information (American Press Institute (API), 2015). Furthermore, social correction of misinformation has become a common practice, with a recent large-scale survey finding that such corrections happen at a rate similar to that of sharing misinformation (Rossini et al., 2021).

Researchers also have turned attention to the importance of understanding the affordances of different platforms for sharing, discussing and evaluating information. For example, studies find social corrections are more common on WhatsApp compared to Facebook, due to features of the former that enable intimate relationships and responsibility to others (Rossini et al., 2021; Waruwu et al., 2020). Looking at more public platforms, a classification analysis of news stories on Reddit and Twitter suggests different and complementary affordances of each platform. Breaking news tended to arrive on Twitter first and last longer, making it useful for tracking stories, while Reddit tended to more quickly bring together additional information and critical dialog across diverse perspectives (Chen and Sanchez, 2019; Priya et al., 2019). A unique feature of social media is the ability to “lurk” or view discourse without directly participating in it. This can be beneficial in the case of those who report witnessing social corrections of misinformation on Facebook and WhatsApp at higher rates than those who directly engage in this practice (Rossini et al., 2021).

Taken together these studies highlight the importance of attending to the quality of discourse in online communities as a way of understanding the mechanisms through which social media can foster a critical stance toward scientific information exchanged in online settings. Whether individuals directly engage in critical dialog or merely observe, the dialogic process opens up opportunities to engage in epistemic thinking about evidence and its use in the service of scientific arguments in online settings.

Critical dialog, discourse norms, and evidentiary practices: Citing, sourcing and interpreting online information

Several cognitive factors contribute directly to the proliferation and persistence of misinformation online, including selective exposure (Winter et al., 2016), confirmation bias, (Modgil et al., 2021) and myside bias (Diana et al., 2019). These biases are exacerbated online when individuals have limited access to alternative points of view in a mutually reinforcing cycle often referred to as an “echo chamber” (Quattrociocchi, 2017). However, studies also show that political discourse on social media can actually increase exposure to cross-cutting perspectives (Kim, 2011), particularly in open forums where group-membership is not restricted (van Raemdonck, 2020).

Under the right conditions, exposure to alternative perspectives can open the door to critical dialog where individuals press one another to substantiate claims with evidence (Kuhn and Zillmer, 2015), prompting a set of evidentiary practices that support epistemic vigilance (Barzilai and Chinn, 2018; Sperber et al., 2010). One such evidentiary practice is sourcing, or considering the author, publisher or publication date when citing evidence (McGrew, 2020). Sourcing evidence in argumentative contexts, in turn, triggers a second evidentiary practice: the evaluation of source credibility for bias (Braasch and Bråten, 2017). Finally, a third evidentiary practice fostered in the critical dialog is the interpretation of evidence to draw valid conclusions. Individuals are more likely to invest additional cognitive resources into interpreting evidence and questioning whether it is being used correctly to support a claim when they identify a source and detect potential bias in the presentation of information (Gierth and Bromme, 2020). Together, these three evidentiary practices are key elements in what Duncan et al. (2018) refer to as a layperson’s grasp of evidence. In many cases, nonexperts are not equipped with the knowledge or skills to adequately engage in “analyzing, evaluating, interpreting, and integrating evidence” (Duncan et al., 2018: 914) as scientists do. Instead, nonexperts must develop a set of evidentiary practices that help them select which sources to consult, critically evaluate the trustworthiness of these sources and validly interpret what these sources have concluded and why.

While critical dialog can have a mitigating effect on cognitive biases, it can also exacerbate them. Cognitive biases are influenced by individuals’ motivations while engaging with information in online contexts (Winter et al., 2016). In a study contrasting the effects of motivated reasoning, Winter et al. (2016) found that accuracy motivation led to a more balanced selection of sources, whereas defense motivation) led to the selection of belief-consistent sources. Our goals when engaging in dialog about evidence can also impact cognitive biases. Villarroel and colleagues (2016) found that arguing to persuade was more likely to trigger confirmation bias than arguing to deliberate. To facilitate critical discourse and heighten epistemic vigilance, an online community must facilitate interactivity between users so that they engage with each other’s content and reasoning by probing, elaborating or critiquing their ideas (Ariel and Avidar, 2015). One way to achieve this is by setting community norms around constructive, critical dialog. Ultimately, the way that individuals enter into critical dialog matters, as the framing of the conversation and its conduct influence our motivations when interacting with others around the use and interpretation of evidence.

Norms for political discourse, particularly in the context of deliberative democracy, have a rich theoretical tradition with clear implications for the empirical study of dialog (Habermas, 2006). However, much of that work is concerned with the question of whether deliberative norms lead to rational, consensus-based decision-making. Our aims here are more modest and more focused. We are less concerned with whether discourse norms for deliberation lead to rational decision-making, and more concerned with how discourse norms in a community make room for evidence-based dialog, an essential component of rational decision-making. We ask whether the norms used to define why and how a community comes together to exchange information align with whether and how individuals utilize evidentiary practices in dialog with one another.

Reddit as a context for understanding evidentiary practices on social media

There are several qualities that make Reddit a particularly useful context for studying norms and practices for evidence-based dialog on social media. First, it is structured as a user-moderated online discussion forum, facilitating the exchange of information in long-form, interactive posts. Within a given subreddit, the posts that are displayed first are those that have been up-voted by community members and within each post, users can view the history of threaded discussion comprising nested replies from users. Second, the platform is divided into over 130,000 active communities (known as subreddits) each with its own set of norms and practices used to augment the policies established by the company (Amaya et al., 2021). As such, the platform affords the opportunity to examine how distinct communities establish and enforce discourse norms that govern the exchange and critique of information. Third, Reddit is a popular site, ranking seventh overall for website visits in 2021 in the United States and 15th worldwide, with over 430 million active monthly users (Statista, 2021).

Reddit’s structure as a platform provides a useful context to compare evidentiary practices in a set of communities with distinct norms for critical dialog. The early months of the COVID-19 pandemic presented a unique opportunity to examine these phenomena, as it raised the stakes for sourcing, evaluating and interpreting scientific information, while also polarizing opinions about how governments should act on information being disseminated by health organizations.

The present study

The present study examines norms and practices associated with evidence-based, critical dialog about COVID-19 in four Reddit communities. Our aim is to characterize how dialog about scientific information unfolds in each setting in light of the affordances and constraints set by each community’s discourse norms. We present samples of dialog to illustrate key trends in the data and conclude with a discussion of the implications of these findings for educators. Our study was guided by the following research questions:

1. What discourse norms are established for dialog and evidence use in each community and how do these norms reiterate, extend or augment the general discourse norms set by the Reddit platform?

2. Are differences between the four communities observed in dialogic interaction and evidentiary practices (citing sources, evaluating sources and interpreting evidence)?

Methods

Sampling

Four subreddit communities were selected for study based on differences in their community size, focus and the likelihood of hosting discussions related to the COVID-19 pandemic. Two focused on COVID-19 exclusively, and two focused on political discussions with threads frequently addressing COVID-19. r/Covid, is a community of approximately 10.3 thousand users ¹ dedicated to “sharing information about Covid-19.” r/Coronavirus, with 10.3 million users, is dedicated to “monitor[ing] the spread of the disease Covid-19.” r/Politics, a community of 7.5 million users, is a forum for “news and discussions on U.S. politics” as it is reported in major news outlets. r/AskTrumpSupporters is a community of approximately 91.7 thousand users that describes itself as a “Q&A subreddit to understand Trump supporters, their views, and the reasons behind those views.”

Threads about COVID-19 were selected from each community in two 3-week periods coinciding with the first two peaks of the pandemic in the U.S. (April and July 2020). The first 10 threads, sorted using Reddit’s best match criterion, were selected for coding, resulting in 20 threads from r/Covid, r/Coronavirus and r/Politics and 19 from r/AskTrumpSupporters. Within each of the 20 threads, up to 50 first-level replies to the original post were then selected for coding, resulting in the analysis of 111 first-level replies in r/Covid, 992 in r/Coronavirus, 988 in r/Politics and 142 in r/AskTrumpSupporters, or a total of 2233 first-level replies to original posts in all. The downthread dialog, or subthreads, following each of these 2233 first-level replies were also analyzed.

Data analysis

Community norms

The norms of each community were analyzed by identifying differences in posted rules and norms as they relate to evidence use and productive, evidence-based discourse. We organized norms into three categories—macro, meso and micro norms—adapted from Chandrasekharan et al. (2018). In our analysis, macro norms are defined as platform-wide agreements, policies and wikis posted by Reddit, which apply to discourse in all of its subcommunities. Meso norms represent platform macro norms that are explicitly restated or elaborated on by a particular subreddit community. Meso norms for each subreddit community were identified using three sources of information: (1) subcommunity rules posted on their homepage; (2) FAQs and wiki pages supplementing the posted rules; and (3) policies for assigning flair (or community-specific tags that users use to identify themselves or the nature of the content in their post). Micro norms are defined as any community-specific rules and norms posted by the subcommunity used to augment Reddit’s macro norms and they were identified using the same three sources used for identifying meso norms. These three categories differ from those defined by Chandrasekharan et al. (2018) in that their categories emerged from an analysis of moderator enforcement of rules, whereas ours emerged from an analysis of the posted rules themselves.

To compare communities, norms related to the use of dialog and evidence were identified by means of thematic content analysis for each of the four subcommunities. A comprehensive list of norms was generated and then used to identify and tabulate the similarities and differences between groups.

General dialogicity

Dialogicity, a measure used to operationalize social interactivity (Ariel and Avidar, 2015), refers to the number of instances in which users’ posts generate interaction with or between other users. This was calculated as the total number of comments under a post divided by the total number of top-level replies (i.e. replies directly to the original post).

Evidentiary practices: Sourcing, evaluation of sources, and interpretation of evidence

Finally, the presence of evidentiary practices within a thread was coded using a scheme adapted from Shi (2020) to examine the use of evidence in argumentation. For our purposes, evidence refers to numeric data used to establish or challenge claims about symptoms, transmission, infection rate, mortality rate, testing, treatment or cure for COVID-19. We identified three types of evidentiary practice: (1) the sourcing of evidence; (2) the evaluation of sources; and (3) the interpretation of evidence. Each subthread following a post (up to 50) was coded as containing evidentiary practices if any of these three practices were present in a subthread. Each subthread received a rating of 0 or 1 for the presence of evidentiary practices and a total was calculated by adding across subthreads. Since there were different numbers of subthreads for each post, the values were standardized by dividing the total instances of evidentiary practices by the number of subthreads to capture the proportion of subthreads with evidentiary practices. Interrater reliability, calculated using 15% of the data to compare a lead rater’s coding against each of two other raters, reached .97 and 1.0 (Cohen’s κ), respectively.

Findings

Qualitative analysis of community norms

In the sections that follow, the norms related to dialog and evidence are presented for each community and compared for similarities and differences in the use of macro, meso and micro norms. A summary of these comparisons can be found in Table 1.

Table 1.

Community norms addressing evidence and dialog.

Community	Macro	Meso	Micro
r/Covid	Yes	None	None
r/Coronavirus	Yes	• No incivility • No redundancy • No off-topic posts	• No edited titles • Use high-quality sources • No politics • Flair: Vaccination status (optional)
r/Politics	Yes	• No incivility • No redundancy • No off-topic posts	• No edited titles • Use high-quality sources • Post links to sources • Use timely information • Flair: Politician (required)
r/AskTrump Supporters	Yes	• No incivility • No redundancy	• Cite sources and context • Initial post must be an authentic question from a non-supporter to Trump supporter • All posts from NS must include a clarifying question • No piling multiple questions in a single post • No links to other threads, must respond to others directly • Flair: Trump supporter, Nonsupporter undecided (required)

NS: trump nonsupporter.

Community norms for r/Covid

There were no community-specific meso or micro norms, rules or FAQs posted on this subreddit. Instead, the community relies solely on Reddit’s macro norms to establish expectations. Like all subreddits, the r/Covid sidebar contains links to Reddit’s content policy, which includes broad rules that can serve to shape the quality of evidence and evidence-based discourse in the Reddit community. Specifically, Rule 1 (“Remember to be human”) prohibits bullying or harassment; Rule 2 entreats users to post authentic content; and Rule 5 prohibits impersonating others, which would presumably include impersonating experts in a field. While r/Covid does not specifically invoke Reddit’s community-wide macro norms, it is reasonable to assume that these macro norms would also apply in this subreddit. The reddiquette norms most relevant to the quality of evidence and evidence-based discourse include reminders to (1) keep submission titles factual and free of opinion; (2) use original source content, providing direct links where possible; (3) read content and up- or downvote based on how appropriate it is, not on whether you agree or disagree; and (4) state reasons for downvoting or reporting a post, providing constructive feedback, when downvoting or reporting a post. Reddiquette also asks users to refrain from posting hoaxes.

Community norms for r/Coronavirus

In addition to providing links to Reddit’s terms of service and policies (macro norms) on their homepage, r/Coronavirus offers 9 community rules (Table 1). The majority of these rules are meso norms, reinforcing platform-wide injunctions against incivility, redundancy, off-topic dialog, advertising, and spam. Of the remaining micro norms, three are germane to evidence use. One rule prohibits altering titles from the original sources to “ensure that the article’s content is represented accurately.” A second steers discourse away from editorializing about politicians and establish that “posts about what has happened are preferred to posts about what should happen,” and exceptions are made for discussions of “executive and legislative leadership and provincial or state authorities with large active outbreaks.” And a third rule establishes a commitment to the quality of information and sources in the subcommunity. The subreddit’s wiki page provides links to what they describe as the “most reliable information on [the] novel coronavirus outbreak,” which include the U.S. Centers for Disease Control, U.K. Public Health Department, and the World Health Organization’s (WHO) COVID-19 page. Finally, users can choose flair to indicate whether they are partially vaccinated, fully vaccinated, or waiting to be vaccinated against COVID-19.

Community norms for r/Politics

In addition to posting links to Reddit’s terms of service and policies, r/Politics provides a direct link to the platform’s reddiquette page, suggesting a greater emphasis on compliance with these macro norms in their subcommunity. The community’s 12 rules (Table 1) reinforce meso norms for discourse like ensuring that posts are on-topic, avoiding redundancy, and maintaining civil discourse. Like other political forums on Reddit, r/Politics includes micro norms on the importance of germane, evidence-based and productive discourse in the subcommunity. To enforce these standards, they have included micro norms for keeping posts timely and refraining from altering titles. On their wiki page, r/Politics also establishes micro norms that govern content in the subcommunity. Specifically, all top-level posts must open a topic for discussion with a link to a timely news item (within 2 weeks) about U.S. politics, using the original title from the source. The sources, themselves, must be on the community’s list of approved domains, which include a combination of legacy news sites; reputable online sources frequently cited by legacy news sites; research organizations, policy think tanks, or political advocacy groups frequently cited by legacy news sites; and official governmental agencies and outlets. These extensions of Reddit’s content policies underscore the subcommunity’s emphasis on providing information and evidence that can be sourced and fact-checked. Finally, all politicians and representatives from news organizations must self-identify using flair.

Community norms for r/AskTrumpSupporters

Of the four subreddits, r/AskTrumpSupporters maintains the most complex set of discourse norms (Table 1), although they have fewer rules than r/Politics overall. Among their 8 rules, there is one meso norm that calls for civility in discourse and another aimed at reducing redundancy. The remaining six rules establish micro norms for structuring dialog, requiring that all top-level posts be open-ended, sourced questions from nonsupporters, responded to by Trump supporters with only clarifying questions from undecided or nonsupporters thereafter. To protect the authenticity and integrity of these dialogs, another micro norm requires that users identify themselves truthfully as Supporters, Nonsupporters, or Undecided using flair. Two more micro norms set standards for reporting rule violations while refraining from direct confrontation with other users. Together these latter two rules can be seen as attempts to “thread the needle” between encouraging community-wide monitoring of discourse, while also trying to avoid flared tempers between users.

The wiki goes on to define the characteristics of good questions, including recommendations to avoid closed-ended questions or long opinion pieces. Instead, the moderators recommend that questions follow the formula “what is your view and why” and include sources for any of the premises of questions in order (1) to ensure that the quality of responses is not “dependent solely on the wording” of the question, (2) to provide context for the question and (3) to avoid “unnecessary discussion attacking the premise of the question and encourag[e] discussion on the question itself.” The moderators emphasize the importance of “focusing on the view, not the person.” The wiki page addresses the quality of responses and follow-up questions, enjoining users to be specific, aware of their own assumptions and patient. The wiki also recommends that users provide detailed responses in the interest of making their views understood or to better understand the views of others. The moderators ask users not to stack multiple questions in one post, explaining “no one likes being overwhelmed when trying to explain their position.”

In short, while the micro norms on this site do address citing sources, they also focus on encouraging authentic dialog in which speakers from opposite sides of a political spectrum seek to understand each other’s perspectives. The emphasis is on engaging in two-way dialog, rather than expounding on one’s own perspective. And although there is clearly an expectation that users will engage in argumentation, the emphasis is on exploring how others respond to evidence that challenges their perspective, rather than critiquing them or their politics.

Quantitative analyses

Given the observed differences in community norms, we turn to a between-group analysis (ANOVA) of dialogicity, a measure of dialogic interaction between users, and evidentiary practices, or the sourcing, critique of sources and interpretation of information used as evidence.

Dialogicity

A one-way ANOVA was used to test Dialogicity (i.e. the average length of threads calculated as total posts/top-level posts) by Community (subreddit) (see Figure 1). Overall, significant differences were found (F(3, 75) = 36.70, p < .001, η² = .595). Post hoc tests suggested a significant difference (with Bonferroni correction) between r/AskTrumpSupporters, where the average value was M = 27.64 (SD = 16.35) compared to r/Politics (M = 6.20 (SD = 2.50)), r/Covid (M = 3.40 (SD = 1.45)), r/Coronavirus (M = 5.97 (SD = 1.66)) (p < .001 for all comparisons), suggesting that the highest levels of dialogic engagement overall occurred in r/AskTrumpSupporters. No other post hoc comparisons were significant.

Figure 1.

Comparison of dialogicity by community.

Evidentiary practices: Sourcing, source evaluation, evidence interpretation

Next, threads were analyzed for the presence of evidentiary practices (i.e. any instances of sourcing, source evaluation, and evidence interpretation) either in top-level replies or downthread discussion (Figure 2). The total frequency was counted for each subthread (up to 50) in a thread and divided by the number of coded subthreads threads to yield the proportional appearance of evidentiary practices across threads in each community. Overall, significant differences by Community were found for Evidentiary Practices (F(3, 75) = 40.98, p < .001, η² = .621). Post hoc tests suggested a significant difference (with Bonferroni correction) between r/AskTrumpSupporters, where the value was M = .68 (SD = .31) compared to r/Politics (M = .15 (SD = .12)), r/Covid (M = .20 (SD = .24)), r/Coronavirus (M = .01 (SD = .004)) (p < .001 for all comparisons), suggesting that the highest levels of Evidentiary Practices overall occurred in r/AskTrumpSupporters. In addition, there was a significant difference between the r/Covid and r/Coronavirus communities’ Evidentiary Practices (p < .05), but not after Bonferroni correction.

Figure 2.

Comparison of evidentiary practices by community.

Illustrative dialogs from r/AskTrumpSupporters

In this section, we present sample data from r/AskTrumpSupporters to illustrate what high dialogicity and high use of evidentiary practices looked like in our data sample. Our aim is to illustrate what we found at the intersection of these two findings to explore both the potential advantages and limitations of online discourse about evidence in this subcommunity.

In the exchange excerpted in Table 2, LD (a Trump nonsupporter) asks ZC (a Trump supporter) for the source of the claim that 30% of American counties had no new cases of COVID-19. This can be seen as a “clarifying question” per the community’s discourse rules and illustrates one type of evidentiary practice (a request for the source of data being cited). ZC responds in turn 2 with a link to the news conference where Trump made the claim, followed by a link to a Washington Post article that reported the same data (an evidentiary practice of citing sources). However, in turn 3 OR (a Trump nonsupporter) then replies by pointing out that the quote from the Washington Post was taken out of context, quoting the article further to show that the author of the article was criticizing Trump’s use of the data as being misleading. This reply illustrates examples of both (1) “evaluating a source” by critiquing Trump’s and perhaps also ZC’s biased presentation of the data; and (2) “interpreting evidence” by explaining that a presentation of the percentage of counties reporting new cases obfuscates the actual proportion of new cases by population. In responding in this way, OR focuses on challenging the framing of the data and its interpretation, rather than focusing on Trump’s or ZC’s motives for doing so.

Table 2.

Sample epistemic dialog (ATS003).

Turn	User ID (flair) a	Text
1	LD (NS)	Can you link the source of this 30% of the nation’s counties have had no new cases data?
2	ZC (TS)	It’s what Trump said in a briefing yesterday [link to source]: The quote was “Nationwide, more than 850 counties, or nearly 30% of the country, have reported no new cases in the past 7 days.” Edit: Since you all seem super fixated on this one point, here is confirmation by the Washington Post [link to source]. That first figure, the more than 850 counties one, is accurate. Data from Johns Hopkins University as of April 15 indicate that 905 counties had added no new confirmed infections since April 8. But more than half of those counties did not have any cases in the first place. Of counties where the virus had been identified by April 8, more than 2500 saw new cases over the next week.
3	OR (NS)	It is a bit misleading without context, though. From that source [Washington Post]: Those figures aside, Trump’s presentation of the 850 counties as “30 percent of the country” is far from the mark. . .The counties with no cases reported from April 8–15—including those which have reported no cases at all—make up less than 6 percent of the population. Counties in which more than 94 percent of the population live saw new cases crop up over that 7 day period.” So, if there are ~3200 counties in the US, and ~900 added no new cases this past week, the implication is that a third of the country/population saw no new cases.

NS: trump nonsupporter; TS: trump supporter.

^aUser self-identification as a TS or NS.

The exchange in Table 2 illustrates several potential advantages to the community norms in r/AskTrumpSupporters. The community rules around self-identification and replies establish a structured dialog across differing perspectives. This structure promotes the critical exchange of information and allows for a close interrogation of the source and interpretation of evidence supporting a view. Norms governing turn-taking heighten the potential for dialogicity by making room for responses to each question or critique from non-supporters. This dialogicity, and the rules around questioning, increase the potential for interactivity by holding participants accountable to addressing one another’s ideas. Such a structure can simply provoke heated exchanges that produce very little insight into the topic of discussion, but two additional discourse norms may mitigate this issue. First, an emphasis on asking clarifying questions can serve to frame the exchange as an information-seeking dialog, rather than a dispute, as we can see in turns 1 and 2; second, the emphasis on productive discussion, including sincere questions and an emphasis on sourced evidence rather than opinions, can serve to focus discourse on the grounds of participant’s beliefs, rather than on their character, which is the case in all three turns.

The exchange in Table 2 also illustrates some potential limitations of the discourse norms in this community. First, this is not simply an information-seeking conversation and the nonsupporters are clearly stretching the boundaries of what it means to ask clarifying questions. In turn 3, we see that OR is critiquing the source and the interpretation of the data without asking a question. Arguably, this turn enhances the potential of the discussion to produce effective critical dialog, despite the community norms. Second, the rule requiring nonsupporters to ask only clarifying questions might lead to stilted exchanges, or as in this case, the rule may be broken to allow for authentic critical dialog. In any case, the exchange in Table 2 results in a noteworthy process of fact-checking and careful sourcing of information, as well as an opportunity for third-party readers to see the importance of context when interpreting information.

In the exchange excerpted in Table 3, we see a different outcome of evidentiary dialog. Just prior to the excerpt, DD (nonsupporter) has asked FS (Trump supporter) to share their views on Hydroxychloroquine, a treatment for COVID-19 that President Trump had recently endorsed. DD has asked FS to respond to a blog post from a site that aggregates and summarizes medical information about COVID-19. In turn 1, FS objects to the request, claiming that the blog post is a biased opinion piece (i.e. evaluating the source). In turn 2, DD responds by apologizing and suggesting that FS read the original articles sourced in the blog post (i.e. sourcing), which come from reputable medical journals. DD provides further context for their original question by explaining that the articles offer evidence against the validity of studies supporting the efficacy of Hydroxychloroquine as a treatment (i.e. interpreting data). FS responds with a critique of the journal articles in turn 3, citing the absence of data under conditions others have claimed were critical for the success of the treatment. To close, DD asks FS again to read the blog post, as it addresses that additional data, offering to read an article in return as a show of good faith.

Table 3.

Sample epistemic dialog (ATS018).

Turn	User ID (Flair) a	Text
1	FS (TS)	[Quoting User DD] Would you be willing to read this breakdown of case studies on Hydroxychloroquine? [link to source]– Not coming from a source that is clearly a biased opinion piece. The very first paragraph makes it clear the author has an agenda to push. I wouldn’t expect it to accurately represent all sides. I’ve seen and heard from enough healthcare professionals to know there are two sides to this issue.
2	DD (NS)	Ok, sorry I used that source. I was just trying to save myself the effort of posting the studies he was referring to. I will link to a couple of those now: From the New England Journal of Medicine [link to source] Published in Clinical Infectious Diseases [link to source] Both of these studies found no benefit of Hydroxychloroquine. I recommend reading the article, despite it being extremely negative on Hydroxychloroquine. The author is a doctor and the website “Science Based Medicine” is a good source for medical information. The reason I ask you to read the article is because he also goes into depth talking about the studies being used to tout Hydroxychloroquine as beneficial. He goes into detail pointing out methodological flaws in those studies. It’s a good article. Will you at least read the “Method” and “Results” sections of the two studies I linked above? It’s maybe a 1 minute read each.
3	FS (TS)	[Quoting User DD] Both of these studies found no benefit of Hydroxychloroquine. Neither of these studies cite that they use it in combination with zinc. Every healthcare professional I’ve heard that talks about the benefits of Hydroxychloroquine also state they use it in combination with zinc. Zinc seems to be in some regard a catalyst of Hydroxychloroquine to fight the infection, and without it seems to be far less effective. It doesn’t surprise me that studies without the use of zinc have poor results.
4	DD (NS)	Again, I highly recommend you read the article I posted. He discusses that aspect of it as well. Would you be willing to make a deal? I’ll read one of your articles if you read this one?

FS: fullstep; DD: Delphic12; NS: trump nonsupporter; TS: trump supporter.

^aUser self-identification as a TS or NS.

Like the previous exchange, the excerpt in Table 3 illustrates the potential for discourse norms to structure productive dialog about sourced information. However, this exchange also illustrates sincere attempts at authentic engagement that come with the requirement of ending posts with clarification questions. Although turn 4 is less of a request for clarification and more of an offer for reciprocity, it at least adheres to the community rule for nonsupporters to ask questions. What makes the exchange authentic is the good faith effort by each poster to respond to the other, even when the response is a rejection. While it is unlikely that either this exchange or the exchange in Table will lead to a change in views on either side of the political spectrum, they represent a clear commitment to the critical analysis of information through sourcing, evaluation, and interpretation fueled by community norms that prioritize evidentiary practices and dialog. They also demonstrate how the combination of evidentiary norms and practices combine with dialogic norms and practices in this particular community. Although the speakers themselves may not walk away with refined views on either the evidence or the conclusions they have chosen to draw from the evidence they have selected, they have engaged in an exchange that is visible to nonparticipants to consider. This final point has educational implications that we will take up in the discussion of our findings.

Discussion

This study sought to examine the dialogic and evidentiary norms and practices of four Reddit communities and to compare communities in the use of dialog and evidentiary practices. Our analysis of community norms identified several noteworthy similarities and differences in discourse norms. All four communities provided links to Reddit’s macro norms, but only the smallest of the four communities, r/Covid, leaned entirely on these norms without posting their own rules, wikis or FAQs. This finding squares with other research on Reddit, which suggests that the best predictor of the presence of subreddit rules is the size and popularity of the subreddit (Fiesler et al., 2018). The remaining three communities (r/Coronavirus, r/Politics and r/AskTrumpSupporters) posted meso norms that emphasized staying on-topic, avoiding redundancy, and maintaining civility. Two of the communities (r/Politics and r/AskTrumpSupporters) shared micro norms enjoining users to cite original source content with direct links where possible, and the third (r/Coronavirus) asked users to draw from high-quality sources. However, Micro norms also divided the three communities. Whereas r/Politics and r/Coronavirus focused more on defining acceptable sources and using accurate titles, r/AskTrumpSupporters focused more on setting norms for the conduct of dialog, including rules on authentic questioning, focusing on other’s views rather than one’s own and turn-taking between speakers with opposing political perspectives.

These differences between micro norms aligned with between-group differences in the presence of dialog and evidentiary practices in the four communities. Dialogicity, or the average length of threaded discussion in response to posts was significantly higher on r/AskTrumpSupporters than the other three communities. This finding may come as no surprise, given the focus on dialog in this community. Evidentiary practices, on the other hand, which were emphasized equally in all but one community (r/Covid), were also significantly higher in r/AskTrumpSupporters. That is, users in this community were more likely to engage in sourcing, source evaluation, and/or source interpretation than the other three communities. One potential explanation for this finding is that because this community emphasized highly structured dialogic exchange across differing opinions, it heightened epistemic vigilance while also guiding discourse toward authentic dialog about the issues and away from diatribes and screeds. In this sense, our findings also align with extant research that may be of particular interest to educators. Van Raemdonck (2020) argues that open subreddits tend to draw diverse perspectives that can avoid an echo chamber effect, but can nonetheless heighten polarization through a hardening of positions in light of critical dialog. However, Chen and Sanchez (2019) have found that such polarization can be mitigated when discourse involves greater reciprocity (sustained dialog between a small group of people). Our data align with these studies, suggesting that subreddits that attract users with diverse perspectives and draw them into structured, reciprocal dialog may increase the degree to which users engage in perspective-taking, an effective strategy for mitigating polarization (Saveski et al., 2021).

Of course, community norms and structured dialog cannot possibly hold up against the active desire to shut oneself off from alternative points of view or challenging data and here an important distinction should be made. Nguyen (2020) distinguishes between echo chambers (social epistemic contexts where individuals distrust and undermine sources that challenge their views) and epistemic bubbles (social epistemic contexts where individuals rarely encounter sources that challenge their views). While norms and epistemic practices may not affect individuals in echo chambers who actively choose to seal themselves off, we might hope to mitigate the effect of epistemic bubbles in open forums where community norms encourage interactive, productive, evidence-based dialog across diverse perspectives.

Limitations and suggestions for further study

Our findings come with some caveats. First, because participants in this study self-selected into their communities and because there were no controlled comparisons, we cannot make inferences about the causal relationships between discourse norms, dialog, and evidentiary practices. Indeed, both the discourse norms and dialog about evidence may reflect rather than direct the users who have chosen to engage with these communities. A randomized controlled experiment would allow us to test relationships, causality, and directionality more carefully.

Second, our findings are comparative and it is important to acknowledge the ways in which dialog fell short of the kind of sustained discourse that one would associate with highly productive deliberative communities. Likewise, much of the discourse came from lay readers who were informed enough to read scientific reports, but most likely lacked the kind of expertise or breadth of knowledge to mount a legitimate critique of the science they commented on. In other words, while there are noteworthy examples of engagement in our data, they do not necessarily present an ideal model for deliberative discourse. Future research might explore gaps between expert and novice evidentiary discourse to better inform the design of interventions to promote critical dialog on social media.

Despite these shortcomings, there is much to learn from the discussions found in these subreddits. Our qualitative data illustrate how social media can be a fruitful context for exploring evidentiary thinking about sources, source bias, and the interpretation of data. Under more controlled conditions, we might better understand how to structure dialog to even greater effect to help students become acquainted with the epistemic aims, ideals, and reliable processes (Chinn et al., 2020) that inform critical dialog about health recommendations and public policy.

Implications for teaching and learning

For the present, our findings demonstrate that meaningful differences in the amount and quality of critical dialog about scientific evidence can be found on social media and we see several implications for educators interested in preparing youth to critically engage with the scientific information that they encounter in public discourse. First, our data highlight three evidentiary practices worth considering in framing discussions of scientific information. Identifying, critiquing, and interpreting sources of scientific information are critical to fostering a layperson’s grasp of evidence (Duncan et al., 2018). Whether students choose to actively engage in dialog or simply “lurk” as third-party observers, they can learn how critical dialog can be used to interrupt the proliferation of misinformation and serve as a foundation for personal decision-making. Second, public platforms like Reddit offer a useful point of departure for defining the features of productive critical discourse about evidence, particularly as it emerges in natural dialog among nonexperts. Educators interested in teaching about social media, dialog, and the public understanding of science should help students critically examine when and how public dialog about scientific information breaks down and when and how it becomes productive, particularly in light of key features like community norms, discourse aims and dialogic engagement. Finally, educators can use social media as a context for helping students make informed choices about which online communities to turn to for productive dialog about scientific information. Learning how to choose discourse communities on social media, based on the quality of critical dialog and evidentiary practices, may ultimately be the most useful adaptive strategy to combat the proliferation of fake news and alternative facts both within and beyond the context of social media.

Author biographies

Mark Felton is a Professor of Teacher Education in the Lurie College of Education at San Jose State University. His basic and applied research examines the use of deliberative dialog as a context for promoting evidence-based reasoning about social, scientific, and civic issues.

Ellen Middaugh is an Assistant Professor of Child and Adolescent Development in the Lurie College of Education at San Jose State University. Her research focuses on the influence of varied social contexts on youth civic identity development and on the implications of digital media for positive youth development.

Henry Fan is an undergraduate research assistant and senior in computer science at San Jose State University. He studies adult learning in STEM education, with a focus on student empowerment and agency through choice.