Estimating the effect size of moral contagion in online networks: A pre-registered replication and meta-analysis

Abstract

Over 5 billion people now use social media platforms. As our social lives become increasingly entangled with online social networks, it is important to understand the dynamics of online information diffusion. This is particularly true for the political domain, as political elites, disinformation profiteers, and social activists all use social media to gain influence by spreading information. Recent work found that emotional expressions related to morality (moral-emotion expression) are associated with increased diffusion of political messages—a phenomenon we called “moral contagion.” Here, we perform a large, pre-registered direct replication (N = 849,266) of Brady et al. using the dictionary methods from the original paper, as well as new large-language models. We also conduct a meta-analysis of all available data testing moral contagion (5 labs, 27 studies, N = 4,821,006). The estimate of moral contagion in the available population is positive and significant (IRR = 1.13, 95% CI = [1.06, 1.20]), such that for each additional moral–emotional word in a post, the expected number of shares was 13% greater. The mean effect size of the pre-registered replication (IRR = 1.17) better estimated the population effect than the original study (IRR = 1.20). Contrary to prior work, we find that the moral contagion model substantially outperforms nonsense models of diffusion (“XYZ contagion model”). Moral contagion was also conceptually replicated when moral–emotional content was measured using state-of-the-art natural language processing methods. These findings reveal that the moral contagion effect is highly robust across datasets and methods.

Significance Statement.

Social media platforms have become a central avenue for sharing political information. Understanding how political messages spread is crucial, as political leaders, activists, and even disinformation agents use online networks to sway public opinion. Our study provides robust evidence, across multiple datasets, topics, time periods, and methods, that messages containing moral–emotional language are more likely to be shared on social media—showcasing the broad generalizability of “moral contagion.” This research highlights the robust association between moral–emotional language and online information diffusion, offering insights for those studying political communication, social media, and information spread.

Estimating the effect size of moral contagion in online networks

In the digital age, more people than ever before communicate about politics, morality and news via online social networks (2). Over 5 billion people are now online and the average user is online for several hours per day (3). Social media once promised to strengthen democracy and egalitarianism, yet these platforms also contribute to the rise of political disinformation (4), affective polarization (5), hate speech and harassment (6), and organization of political violence (7). Therefore, examining the psychological processes that underlie these phenomena is key for understanding when and why online political conversations take these forms (8, 9).

Recent work found that emotional expressions related to the domain of morality (moral–emotional expression) are associated with increased diffusion of political messages in online social networks—a phenomenon we called “moral contagion.” In over 500,000 Twitter messages, the use of moral–emotional language was associated with increased sharing across political topics of gun control, same-sex marriage, and climate change, and moral–emotional tweets were more likely to spread through ingroup networks (1).

On one hand, moral–emotional expression online may also help raise awareness of moral issues and spark collective action (10–12). On the other, the spread of moral emotions—particularly negative moral emotions like moral outrage—has been linked to increased group identification and worsened intergroup relations (1, 13) as well as social pressure to express more negative emotions (8, 14, 15). Furthermore, moralized rhetoric online is linked to offline violence (16). Given the importance of moral contagion in contentious political conversations online, there is a need to understand the extent to which the general phenomenon replicates in samples outside of the original sample (2017).

In general, it is important to replicate empirical findings since several large replication projects have found that many original studies in psychology either did not replicate or overestimated the effect size of the phenomenon of interest (17–19). Relatedly, other work has questioned the generalizability of psychological findings (20, 21). This is likely to be the case in domains where the topics under investigation are sensitive to the social context (22), such as political and moral psychology (23). This is especially important because prior work claimed that moral contagion does not replicate well out of the original samples used in Brady et al. (1), and also claimed that the moral contagion model performed “no better” than a nonsense model that counted the number of “X,” “Y,” and “Z”s in social media messages to predict their diffusion (‘XYZ contagion” model; Burton et al. (24)). Here, we sought to address limitations of previous work by testing how well moral contagion replicated outside of the original samples used in Brady et al. (1). We tested moral contagion across all available published samples as well as a large, pre-registered replication. We also tested Burton et al. (24)'s key claim that moral contagion models perform no better than the nonsense “XYZ contagion” model.

Furthermore, natural language processing has improved rapidly since the original paper documenting moral contagion (1) was published. In particular, the advent of large language models that can outperform dictionary methods as well as several traditional machine learning methods across multiple languages (25) presents an opportunity to determine whether moral contagion is robust to various measurement strategies. Therefore, we used both dictionary methods as well as state-of-the-art methods based on large language models to conceptually replicate moral contagion in online networks.

Current research

In the current article, we estimated the effect size of moral contagion and assessed its generalizability using two approaches. First, we conducted a large, pre-registered direct replication of prior work (N = 849,266) spanning real social media conversations about 5 contentious political topics over a number of years in US politics. Second, we searched the PsychINFO citation database and the Open Science Framework database (using the terms “moral contagion” and “moral–emotional language”) to perform a meta-analysis of all published and unpublished replications of the original moral contagion studies (1) that we were able to locate as of 2023 April. Our meta-analysis includes 4,821,006 observations spanning 27 studies from 5 different research labs.

The results of our pre-registered direct replication as well as the meta-analysis of existing datasets find that the estimate of moral contagion in the available population of studies is positive and significant, revealing that it generally replicates in a wide variety of topics and platforms. We also find that the large, pre-registered replication better estimated the effect size of the available population of studies than the original study, and that heterogeneity exists in the effect size across studies/datasets. Our pre-registered replication also found that the moral contagion model significantly outperformed a nonsense “XYZ contagion” model in every dataset, in contrast to claims made by Burton et al. (24). Finally, moral contagion replicates when moral–emotional language is measured using the large language model GPT-3.5, demonstrating that moral contagion replicates with advanced measures in natural language processing. These findings highlight the importance of using replication and a pre-registered analysis plan when aiming to generate accurate estimates of effect size, and testing how robust empirical estimates are to different computational methods in a rapidly changing methodological environment.

A pre-registered replication of moral contagion

We first conducted a pre-registered, large-scale direct replication of the moral contagion effect (N = 849,266; see https://osf.io/gd734) (1). We closely followed the guidelines and standards widely recommended by replicability advocates to increase the precision of the effect size estimate and provide a rigorous test of generalizability (26, 27). These standards include performing a direct replication, using a larger sample size than the original study, and utilizing pre-registration to guard against questionable research practices. All research was conducted in accordance with the New York University (NYU) University Committee on Activities Involving Human Subjects (IRB no. 12-9058). Data collection was ruled “exempt” due to our use of public tweets only.

To perform the replication, we focused on five contentious political topics from recent years in US politics that we expected to show evidence of moral contagion (Justice Kavanaugh Supreme Court hearings, the government shutdown over the border wall, the Trump administration's ban of transgendered individuals in the military, the killing of George Floyd, and mask wearing during the COVID-19 pandemic). All of these topics were distinct from the original moral contagion research (which focused on same-sex marriage, climate change, and gun control; Brady et al. (1)) allowing us to assess the generalizability of moral contagion across time and topics.

We then connected to Twitter's premium API with the goal of collecting 200,000 tweets from each of the five topics at specific dates (when the topic of interest was at its peak controversy in US politics). The topics, their start dates, the data collection method, and statistical models were all pre-registered at the OSF link above before collection began. (See Table 1 for all topics and dates of the collection periods.) We note that for two of our datasets, 200,000 tweets could not be collected because there were not enough tweets available from Twitter's API at the exact date we chose before we collected the data.

Table 1.

Data description for the pre-registered replication.

Political Topic	Date	N	Description
Brett Kavanaugh Supreme Court Hearings	2018 September 16	195,567	Hearings are held to address sexual assault allegations against Supreme Court nominee Brett Kavanaugh
President Trump Opts to Keep the Government Shutdown	2019 January 16	200,000	President Trump keeps the government shut down as leverage to build a wall at the US−Mexico border
Transgender Ban in the Military Revived by Supreme Court	2019 January 23	53,649	Supreme court revives a ban on transgender individuals in the military
George Floyd Death	2020 March 27	200,000	Protests erupt as a video is released showing George Floyd killed by a police officer
COVID-19 mask wearing	2020 May 27	200,000	Conversations about wearing face masks 2 months after COVID-19 hit the United States

Topics, start date for data collection, sample sizes (N), and description of topics used in the pre-registered replication of Brady et al. (1).

To analyze each dataset, we exactly replicated the analysis pipeline of Brady et al. (1) directly based on code from its OSF page: https://osf.io/59uyz/. This included preprocessing steps to prepare tweets for sentiment analysis and wrangling key covariates from metadata. Our outcome variable was the retweet count for each tweet (measure of contagion), which was provided as metadata from Twitter's API. We also note that because we collected historical data from Twitter's Premium API, our retweet count variable is more accurate than previous studies, since collecting tweets in real-time does not account for retweets that occur outside the window of data collection.

To form our three word-category predictor variables, we computed the frequency of distinctly moral, distinctly emotional, and moral–emotional words (e.g. “hate,” “disgust,” “greed”) present in each tweet to determine how these factors predicted retweet count. Importantly, the moral–emotional category was formed by taking the words that were present in both the moral and emotional lexicons, so that remaining words formed the distinctly moral and distinctly emotional categories. For each dataset, our pre-registered statistical models included our three word-category predictor variables, as well as covariates known to affect retweet counts independent of message content (1, 28–31), including the number of Twitter followers the original message author had, whether media or a URL was attached to the message, and whether the message author was a “verified” Twitter user. Including these pre-registered covariates known to impact retweet counts in our model reduces omitted variable bias (32); see also SI Appendix, Section S1). We fit a negative-binomial model with maximum likelihood estimation to account for overdispersion in retweet counts (33). Descriptive statistics for all variables can be found in SI Appendix, Section S2, Tables S5–S7, Figs. S1–S5.

Analyzing each dataset revealed a positive and significant association between moral–emotional language and message diffusion for 4 out of 5 datasets: Kavanaugh: IRR (incident rate ratio) = 1.05; Trump wall: IRR = 1.07; Transgender Ban: IRR = 1.66; COVID-19: IRR = 1.12, all ps < 0.001 (see Fig. 1). The George Floyd dataset was the exception with a negative association: IRR = 0.97, P < 0.001. The mean effect size across all datasets was IRR = 1.17. In other words, for each additional moral–emotional word in a social media post, the expected number of shares was 17% greater. This effect size was very similar, albeit slightly smaller, than the moral contagion effect estimated by Brady et al. (1), where the mean effect size was IRR = 1.20. As such, these large pre-registered experiments replicated the basic moral contagion effect with a similar magnitude to the original effect and found that it generalized to at least four new topics.

Fig. 1.

Meta-analysis of studies examining moral contagion in online networks. A meta-analysis of the moral contagion phenomenon using a generic inverse-variance weighting method with a random effects model. The effect size for moral contagion (the association between moral–emotional language and message diffusion) showed a significant, positive effect across 27 studies, N = 4,821,006. Effect size represents an incident rate ratio produced from negative binomial regression models. The dotted line represents an IRR of 1, which indicates a zero effect. Values >1 represent a positive association between moral–emotional language and retweet counts, values <1 indicate a negative association.

For distinctly emotional language, a significant negative association with message diffusion was observed for three out of five datasets: Trump Wall: IRR = 0.95; Floyd: IRR = 0.98; COVID-19: IRR = 1.04, all ps < 0.001. We observed significant positive associations for the Kavanaugh dataset: IRR = 1.07, P < 0.001 and a nonsignificant association for the Transgender Ban dataset: IRR = 0.98, P = 0.55. The mean effect size across all datasets was 1.00.

For distinctly moral language, a significant, positive association with message diffusion was also observed for two out of five datasets: Kavanaugh: IRR = 1.18; and the Trump Wall dataset: IRR = 1.05, all ps < 0.001. We observed significant negative associations for both the Floyd dataset: IRR = 0.97, and the COVID-19 dataset: IRR = 0.95, all ps < 0.001; and a nonsignificant association for the Transgender Ban dataset: IRR = 1.01, P = 0.832. The mean effect size across all datasets was 1.03.

In our pre-registered replication, we also found that the moral contagion model outperformed Burton et al. (24)'s nonsense “XYZ contagion” model as determined by comparing AIC/BIC model fit statistics, see SI Appendix, Section S1. Furthermore, models with our pre-registered covariates consistently outperformed models without them, see SI Appendix, Section S1. Together, these findings directly challenge two key claims made by Burton et al. (24): moral contagion does not replicate well out of the original Brady et al. (1) sample, and the moral contagion model performs “no better” than a nonsense “XYZ contagion” model. Our meta-analysis finds that moral contagion is robust across a wide number of topics.

Replicating the moral contagion effect with large language model GPT

To test the methodological robustness of these results we next conducted a replication analysis using the large-language model GPT-3.5. While Brady et al. (1) used dictionary methods, natural language processing has advanced significantly since that original paper was published, and recent methods appear to be far more effective than traditional dictionary methods at detecting manually annotated psychological constructs (25). Recent work has incorporated machine learning classifiers to measure moral emotions (14, 15) and large-language models such as GPT represent the state-of-the-art in natural language processing. GPT has been found to be an effective tool to detect psychological constructs (such as sentiment, discrete emotions, and various moral constructs) across multiple languages (25). Importantly, GPT was found to be far more correlated with the judgments of human annotators (r = 0.59 to 0.77) than dictionary methods (r = 0.20 to 0.30). Because of these methodological advancements in the field, it is important to test the robustness of the moral contagion with more recent methods, especially because recent work claimed that dictionary methods are poor measures of moral emotions in social media text (24, 34).

GPT analysis

We analyzed all five datasets from the pre-registered replication with the GPT-3.5 API. Because GPT-3.5 has similar performance to the GPT-4 API for the English language (and was substantially less expensive), we chose to use GPT-3.5 over GPT-4 (see Rathje et al. (25)). While we did not perform a separate study to validate GPT-3.5, GPT's ability to capture the main constructs we measured (morality, sentiment, and discrete emotions) as rated by human annotators was previously validated in several studies including work by the authors (Abdurahman et al. (35); Rathje et al. (25)). See Rathje et al. (25) for detailed information on annotation procedures, details on the human annotators, and limitations regarding using GPT.

All of the GPT analysis used prompts that were similar to previously validated prompts (25). Specifically, our prompts asked GPT-3.5: to what extent does the following social media post contain (i) content related to morality or (ii) moral outrage (iii) emotional content and (iv) moral AND emotional content. For each prompt, we included at the end “Answer only with a number, with 0 being ‘not at all’ and 10 being ‘very much.’ Here is the social media post:” and then we included the social media posts. These prompts were meant to get both at the core construct of interest (moral and emotional language), a similar construct (moral outrage), in addition to measuring moral and emotional language separately. Further, to look at whether certain discrete emotions were associated with moral contagion (or whether moral–emotional language overlapped with various discrete emotions), we also prompted GPT about sentiment and various discrete emotions (including sadness, joy, anger, and fear). All of the prompts are shown in SI Appendix, Tables S8 and S9. We used the GPT-3.5 API and code adapted from Rathje et al. (25) to run these analyses, and used GPT-3.5's more recent “batch API” (https://platform.openai.com/docs/guides/batch) for the sentiment and discrete emotions analysis to enable faster analysis.

We found that the GPT output was correlated with dictionary methods, though the correlations were small. Specifically, the number of moral words in a post was correlated with GPT's rating of whether a post was related to morality, r = 0.16, 95% CI = (0.16, 0.16), P < 0.001. The number of emotional words in a post was correlated with GPT's rating of a post's emotionality, r = 0.24, 95% CI = (0.23, 0.24), P < 0.001. The number of moral–emotional words in a social media post was correlated with GPT's ratings of a post's moral and emotional language, r = 0.27, 95% CI = (0.27, 0.27), P < 0.001. The number of moral–emotional words in a social media post was also correlated (albeit less strongly) with GPT's ratings of moral outrage, r = 0.17, 95% CI (0.17, 0.18), P < 0.001. While these correlations are not extremely high, they are very similar to the range of correlations between GPT output and dictionary output for other psychological constructs (see Rathje et al. (25)).

We also explored whether the moral–emotional dictionary correlated with other constructs, such as sentiment and discrete emotions. The number of moral–emotional words in a post positively correlated with GPT's ratings of a post's negativity (r = 0.09, 95% CI [0.09, 0.09], P < 0.001), anger (r = 0.14, 95% CI = [0.14, 0.14], P < 0.001), sadness (r = 0.20, 95% CI = [0.20, 0.20], P < 0.001), and fear (r = 0.15, 95% CI = [0.15, 0.15], P < 0.001). It negatively correlated with GPT's rating of joy (r = −0.05, 95% CI [−0.05, −0.05], P < 0.001). Importantly, none of these correlations were as strong as the moral–emotional dictionary's correlation with GPT's ratings of a post as moral–emotional (r = 0.27), suggesting that the moral–emotional dictionary from Brady et al. (1) seems to better capture the construct of moral emotion as opposed to simply capturing sentiment or discrete emotions. We report further correlation matrices between each GPT construct, the GPT “batch API” and non-batch API analysis, and the correlations between all GPT constructs and dictionaries in SI Appendix, Figs. S3–S5.

Models

Our outcome variable was the retweet count for each tweet (measure of contagion). Instead of word frequency, we used GPT ratings from our four different prompts assessing content (i) emotionality, (ii) morality, (iii) moral–emotionality, and (iv) moral outrage for each tweet to determine how these factors predicted retweet count. For each dataset, our statistical models included our GPT rating predictor variables, as well as covariates outlined above (number of Twitter followers the original message author had, whether media or a URL was attached to the message, and whether the message author was a “verified” Twitter user). We fit a negative-binomial model with maximum likelihood estimation to account for overdispersion in retweet counts (33). Since our prompts created a different scale than the counts used in the dictionary method, we standardized the GPT variables for better effect size comparison (see SI Appendix, Tables S10–S14 for output of all models using standardized predictor variables for both dictionary and GPT measurements).

When measuring moral–emotional language using GPT, we found a positive and significant association between moral–emotional language and message diffusion for three out of five datasets: Kavanaugh: IRR = 1.10; Transgender Ban: IRR = 1.39; COVID-19: IRR = 1.39, all ps < 0.001. We observed a significant negative association for the George Floyd dataset: IRR = 0.91, P < 0.001, and a nonsignificant positive association for the Trump wall dataset: IRR = 1.01, P = 0.55. Results were thus consistent with those obtained using the dictionary method—except for the Trump Wall dataset. Overall, the mean effect size across all datasets was IRR = 1.16, meaning that each standard deviation increase in GPT's moral–emotional score was associated with an 16% increase in sharing. This closely replicated the positive association between moral–emotional language and sharing counts found using dictionary-based methods. In fact, this effect size is larger than estimated using dictionary-based methods, where the average effect size of standardized model outputs predicted an 8% increase in sharing for every standard deviation increase in moral–emotion words present in the message. Results for all predictors in GPT models are included in SI Appendix, Tables S10–S14.

For emotional language, a positive and significant association with message diffusion was observed for three out of five datasets: Kavanaugh: IRR = 1.07; Floyd: IRR = 1.14; COVID-19: IRR = 0.8, all ps < 0.001. We observed nonsignificant positive associations for the Transgender Ban dataset: IRR = 1.04, P = 0.38; and the Trump wall dataset: IRR = 1.01, P = 0.58.

For moral language, a positive and significant association with message diffusion was also observed for three out of five datasets: Kavanaugh: IRR = 1.04; Floyd: IRR = 1.31; COVID-19: IRR = 1.17, all ps < 0.001. We observed significant negative associations for both the Transgender Ban dataset: IRR = 0.35, P = 0.001; and the Trump Wall dataset: IRR = 1.19, P = 0.001.

Finally, we observed a significant and positive association between moral outrage and message diffusion for three out of five datasets: Trump Wall: IRR = 1.20; Kavanaugh: IRR = 1.15; COVID-19: IRR = 1.20, all ps < 0.001. We observed a significant negative association for Floyd: IRR = 0.93, P < 0.001 and a nonsignificant positive association for the Transgender Ban dataset: IRR = 0.35, P = 0.001

We also explored whether adding other constructs, such as “sentiment” and discrete emotions (sadness, anger, fear, joy, and negativity) would change the effect of moral–emotional language. The addition of each of these constructs into the moral contagion model did not change the effect of moral–emotional language, suggesting that moral–emotional language is a unique predictor of message diffusion over and above other non-moral discrete emotion constructs. Results for all predictors with discrete emotions in GPT models are included in SI Appendix, Tables S15–S39.

A meta-analysis of all available replications of moral contagion

To provide a robust estimate of the effect size of moral contagion, we pulled all available published and unpublished data by searching PsychINFO's database for articles using the keywords “moral contagion” and “moral–emotional language” and then selected papers that performed a replication of the main findings of the paper by reading the abstract (1). Next, we searched OSF's database of studies using the same terms and selected replications of Brady et al. (1) for access to data, where required. Studies that were included either had their data available on OSF to reproduce or listed adequate information in their paper for us to perform the meta-analysis (i.e. the effect size and dispersion for each study was reported in Results). Overall, we were able to collect data from five independent labs that have examined the association between moral–emotional words and retweet counts using the same method as Brady et al. (1). The final meta-analysis included 27 studies consisting of total N = 4,821,006, see Table 2.

Table 2.

Studies included in meta-analysis of moral contagion.

Paper	No. of studies	Total N (analyzed)	publication
Ariztia (36)	1	25,560	Unpublished
Brady et al. (1)	3	313,003	Proceedings of the National Academy of Sciences
Brady et al. (10)	2	286,255	Journal of Experimental Psychology: General
Pre-registered Replication	5	849,266	Current manuscript
Burton et al. (24, 34)a	5	207,622	Cognitive Science Society Proceedings/Nature Human Behaviour
Jordan (37)	3	188,129	Unpublished
Rathje et al. (38)	8	2,951,322	Proceedings of the National Academy of Sciences

Table lists authors, date of publication or posting of paper if unpublished, number of studies, the total number of observations across studies, and where the paper was published (or whether it was unpublished).

^aBurton et al. (24, 34) did not include covariates known to impact retweet count, and thus effect sizes were estimated using models that only contained the three word-category predictors. Both publications use the same dataset.

The statistical model used for all studies in the meta-analysis was the same as Brady et al. (1) and our replication study above, with the exception of one lab that did not collect covariates for their analysis (see Table 2). For those studies, we used the reported effect size and standard errors from the model that did not include covariates.

We performed the meta-analysis using a generic inverse-variance weighting method as implemented in the R package meta. As a more conservative estimate of the meta-analytic effect as well as to improve estimates of generalizability, we estimated a random effects model (20, 21) using the Hartung–Knapp–Sidik–Jonkman method to estimate between study variance (τ²; IntHout et al. (39)). To estimate the confidence interval on τ², we used the Q-profile method (40). We present the meta-analytic estimate of moral contagion in Fig. 1. All parameters used in the model estimation can be viewed in the R scripts available at: https://osf.io/96fwc/.

The meta-analytic estimate of moral contagion is positive and significant when all available data to date are taken into account. Results of the random effects model estimated an effect size of IRR = 1.13, 95% CI = (1.06, 1.20), see Fig. 1. It is interesting to note that this meta-analytic estimate of effect size was similar, albeit slightly smaller, than Brady et al. (1)'s original estimate (IRR = 1.20), and very close in size to our pre-registered replication study above (IRR = 1.15). Overall, however, the meta-analysis found that the results of Brady et al. (1) generalize out of its original sample to numerous political topics, lab groups, and social media platforms. As such, the moral contagion effect seems highly robust across contexts.

In order to examine the datasets for evidence of publication bias, we conducted Egger's intercept test (41), which is a statistical test for asymmetry in the relationship between study precision and study effect size. Egger's intercept test found that the pattern of results in our meta-analysis were not indicative of publication bias, intercept = 3.049, P = 0.171. Similarly, Fig. 2 demonstrates that there is no correlation between precision of a study and the estimated effect size (potential publication bias would be indicated by lower standard errors being associated with greater effect sizes; Egger et al. (41)). Furthermore, a P-curve analysis of all studies in the meta-analysis revealed that evidential value for moral contagion was present, see SI Appendix, Fig. S5.

Fig. 2.

Plotting the relationship between study precision and effect size. Each observation represents a study, where the Y axis represents the standard error associated with the moral contagion estimate, and X axis represents the effect size in terms of the IRR. The red line highlights an IRR of 1.0, which indicates a zero effect. In the meta-analysis, lower study precision was not systematically related to effect size, which suggests that publication bias is not a contributor to the meta-analytic estimate. Note that studies with the largest samples (at the top of the graph) were most likely to replicate the moral contagion effect.

Discussion

Our large, pre-registered replication and meta-analysis of 27 studies that included over 4,821,006 messages provides robust evidence of moral contagion in online social networks. In our replication, we found clear evidence that moral–emotional language is associated with increased sharing across multiple political topics and time periods. We also replicated moral contagion using GPT3.5 to measure moral–emotional language in social media text, demonstrating the moral contagion is robust to measurement type. Together, these results provide strong evidence of the association between moral–emotional language and sharing and demonstrate the generalizability of moral contagion: a significant effect was found for various political topics, across multiple social media platforms (Twitter and Facebook), across time periods and among ordinary users as well as political elites.

Our findings also directly challenge previous claims made by Burton et al. (24) that moral contagion may not replicate well out of the original Brady et al. (1) sample, and that the moral contagion model performs “no better” than a nonsense “XYZ contagion” model. Since Burton et al. (24) used smaller samples than the Brady et al. (1) and did not pre-register their data collection or analysis plans, our findings suggest that following established standards for replication as we did here (26, 27) allow for more precise estimates of effect sizes. While our findings challenge claims made by Burton et al. (24), our replication and meta-analysis did provide evidence of heterogeneity in moral contagion as some studies produced negative coefficients or null findings. For example, we did find one case where moral contagion was not replicated: when people were discussing the killing of George Floyd (an event that precipitated massive international protests in support of Black Lives Matter). There are several reasons that can explain this heterogeneity, such as sampling error, real contextual features of the topic or social network that serve as moderators for moral contagion, or issues with the replication strategy and analytic decisions of the researchers. For example, Burton et al. (34) did not collect covariates included in the models of Brady et al. (1) which limits the ability to accurately estimate the effect size in these datasets due to the extremely skewed nature of social media data. The heterogeneity we discovered highlights the importance of using pre-registration (including a detailed analysis plan that specifies critical covariates) and identifying concrete boundary conditions for a phenomenon to help guide future replications and extensions of the research (for an example applied to moral contagion, see Brady et al. (8)).

Our results demonstrate how using big data to test psychological hypotheses affords researchers the opportunity to conduct large-scale replications that help estimate effect size with better precision. Although our results underscore the general stability of the findings of moral contagion, it is still important to acknowledge several limitations with the methodological approach taken by Brady et al. (1); most importantly the fact that word-count methods can produce noisy results and the data were purely correlational. These limitations point to the fact that meta-analyses following one method are not enough; there is a need to examine psychological phenomena with various methodological approaches alongside directly replicating the broader finding. In this article, we also used advances in natural language processing that measure moral emotion expressions with much greater precision to replicate our results, and previous studies have done the same (14). Furthermore, our present analysis cannot establish a causal link between moral–emotional language and sharing because messages with moral–emotional language may differ from messages without moral–emotional language on other aspects not controlled for in our analysis. However, other studies using carefully controlled behavioral experiments have found that moral–emotional language does in fact cause an increase in political message sharing (13). Large-scale direct replications combined with multimethod approaches can help assess the robustness of psychological findings in the era of big data.

Conclusion

In sum, the current research reveals that the findings of Brady et al. (1) replicate in multiple out-of-sample datasets using both dictionary-based and large-language model methods for measuring moral and emotional content, which highlights the general robustness of the moral contagion phenomenon. Our estimate of the effect size from a large, pre-registered direct replication estimated the population effect size more precisely than the original study, which reinforces the importance of replicating work following pre-registration standards to gain the most accurate expectations of effect sizes in future work. Our analysis suggests that moral emotion expressions are associated with the diffusion of political messages across numerous topics, time periods, authors, and social media platforms.

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

The authors declare no funding.

Author Contributions

William J. Brady (Conceptualization, Software, Formal analysis, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review & editing), Steve Rathje (Software, Formal analysis, Validation, Methodology, Writing—review & editing), Laura K. Globig (Software, Formal analysis, Validation, Methodology, Writing—review & editing), and Jay J. Van Bavel (Conceptualization, Supervision, Funding acquisition, Writing—review & editing)

Preprints

This manuscript was posted on a preprint: https://osf.io/preprints/osf/s4w2x_v2.

Data Availability

All deidentified data and analysis scripts are available at the Open Science Framework (OSF) at: https://osf.io/96fwc/.