Quantifying the vulnerabilities of the online public square to adversarial manipulation tactics

Abstract

Social media, seen by some as the modern public square, is vulnerable to manipulation. By controlling inauthentic accounts impersonating humans, malicious actors can amplify disinformation within target communities. The consequences of such operations are difficult to evaluate due to the challenges posed by collecting data and carrying out ethical experiments that would influence online communities. Here we use a social media model that simulates information diffusion in an empirical network to quantify the impacts of adversarial manipulation tactics on the quality of content. We find that the presence of hub accounts, a hallmark of social media, exacerbates the vulnerabilities of online communities to manipulation. Among the explored tactics that bad actors can employ, infiltrating a community is the most likely to make low-quality content go viral. Such harm can be further compounded by inauthentic agents flooding the network with low-quality, yet appealing content, but is mitigated when bad actors focus on specific targets, such as influential or vulnerable individuals. These insights suggest countermeasures that platforms could employ to increase the resilience of social media users to manipulation.

Significance Statement.

We show that social media users are vulnerable to adversarial manipulation tactics, through which bad actors can amplify exposure to content that threatens, for example, democratic elections and public health. While tactics such as flooding the network with low-quality yet appealing content are damaging, getting users to follow inauthentic accounts has the most detrimental impact. Bad actors can increase harm by maximizing coverage rather than targeting particular individuals, such as influential ones. The varying degrees of harm associated with these tactics highlight tradeoffs and specific areas on which platforms could focus as they develop deterrence strategies against manipulation.

Introduction

The vision of social media as the modern public square has been challenged as users have become victims of manipulation by astroturf (1, 2), trolling (3), impersonation (4), and misinformation (5–7). False news have been reported to spread virally—similarly to reliable information (8) or even more (9) depending on operational definitions. These kinds of manipulation exploit a complex interplay of socio-cognitive (10, 11), ideological (7), and algorithmic (12, 13) biases. The exploitation is enabled or greatly facilitated by inauthentic accounts that impersonate people with malicious intent. Many such accounts can be coordinated by a single entity (14). If this is done through software, such accounts are commonly referred to as social bots (15, 16). Inauthentic and/or coordinated accounts have been observed to amplify disinformation (8), influence public opinion (14, 17–19), commit financial fraud (14, 20), infiltrate vulnerable communities (3, 21, 22), and disrupt communication (23, 24).

Can social media be manipulated to the point that they no longer function as a public square? Under what conditions? It is difficult to carry out empirical experiments and analyzes in the real world to explore these questions (25, 26). One challenge is the limited size of experiments in the wild, stemming from both costs and ethical concerns about the potentially harmful nature of content from bad actors. A second difficulty is the limited data from social media platforms available to researchers (27), exacerbated by recent events such as changes at Twitter/X. These difficulties have led, for example, to conflicting accounts about whether disinformation campaigns on social media can sway elections (28–33). Evidence suggests that these operations mainly impact specific vulnerable communities (34, 35). However, we lack a comprehensive quantitative understanding of how coordinated inauthentic tactics can disrupt online communities. This prevents the informed design of moderation or regulatory policies to protect the online public square from manipulation.

Here we introduce SimSoM, a minimalistic model of a generic social media platform. The model allows us to explore scenarios in which an information-sharing network is manipulated by malicious actors controlling inauthentic accounts, and to measure the consequences of such information operations. We assume that bad actors aim to spread low-quality information. While there are different kinds of low-quality content in reality—disinformation, conspiracy theories, malware, or other harmful messages—our model uses an abstract definition of low-quality content that encompasses these different types. The impact of the manipulation is measured in terms of the quality of information to which users are exposed in the network.

We find that the presence of manipulation is sufficient to suppress quality information, driving low-quality content to spread virally in the network. We also examine network vulnerabilities that may amplify the effects of manipulation, and evaluate the overall information quality as a result of different malicious tactics. We focus on four well-documented tactics commonly employed in influence operations across various platforms (36): (i) infiltrating a community, for example through social bots (6, 37), follow trains (38), or by impersonating news outlets (36, 37); (ii) generating deceptively appealing content, such as novel narratives (9) or emotional messages (39); (iii) flooding the network with high volumes of content by posting at high frequency to artificially inflate popularity/engagement indicators (8, 37, 40), and possibly deleting content to avoid detection (41); and (iv) targeting specific users, such as influential (8) or vulnerable individuals (6). Insights from analyzes of these tactics are instrumental in developing countermeasures to increase the resilience of social media and their users against manipulation. We discuss mitigation steps that platforms could take and the issues that arise from regulations aimed at protecting human speech from suppression.

Results

We model information diffusion in a social media platform such as Twitter/X, Instagram, or Mastodon. The information system is a directed network with nodes representing accounts and links representing follower relations. Similar to real-world platforms, content circulates through messages that appear in news feeds. Agents can post new messages or reshare messages from their feeds, generated by their friends, i.e. the accounts they follow. The information diffusion process is illustrated in Fig. 1. Messages represent information that could take the form of text, links, hashtags, images, or other media. An agent can introduce a new message into the system or, alternatively, select a message from their news feed to reshare. Messages created and reshared by an agent then appear on the news feeds of their followers.

Fig. 1.

Illustration of the SimSoM model. Each agent has a limited-size news feed, containing messages posted or reposted by friends. Dashed arrows represent follower links; messages propagate from agents to their followers along solid links. At each time step, an active agent (colored node) either posts a new message (here, m20) or reposts one of the existing messages in their feed, selected with probability proportional to their appeal a, social engagement e, and recency r (here, m2 is selected). The message spreads to the node’s followers and shows up on their feeds.

Even though people prefer quality content (42), their sharing behavior is mediated by other factors such as laziness (43) and message appeal. To account for this, each message m in the model has two intrinsic and independent attributes, appeal  $a_m$ and quality  $q_m$. Everything else being equal, messages with higher appeal are more likely to be reshared (Fig. 1). Quality, on the other hand, represents objective, desirable properties of content such as the originality of an idea or the accuracy of a claim. Here we naively represent quality and appeal as scalar values. Deceptive posts may have low quality yet high appeal. For example, false news and junk science articles have low quality—most people would not share them knowingly. Yet such low-quality content may be even more likely to spread virally than high-quality information (9). Low-quality content may be novel, clickbait, ripped from headlines, and/or may appeal to people’s political, emotional, or conspiratorial bias. Worse yet, bad actors can employ generative AI to produce such content at scale (37).

The model captures bias towards appeal as well as two other ingredients that are prioritized by the ranking algorithms of social media platforms, namely social engagement and recency (44). An agent selects a message to reshare from the news feed, which is an inventory of distinct messages recently shared by the agent’s friends. The message is selected with probability proportional to: (i) its appeal; (ii) its social engagement, defined as the number of times it has been shared by the agent’s friends; and (iii) its recency, which decreases with time in the feed (see details in Methods).

Unlike authentic agents, whose intention is to consume and share high-quality information, we define inauthentic agents as accounts that are controlled by bad (adversarial) actors to spread low-quality content among authentic agents. We refer to these accounts as “bad actors” or “inauthentic agents” throughout this paper. Such accounts may be controlled by humans (trolls), software (social bots), or a mixture (cyborgs). The model has three parameters to model manipulation tactics by bad actors: infiltration, deception, and flooding, explained next.

Infiltration describes how bad actors amplify exposure to their messages by getting authentic accounts to follow them (Fig. 2a). Bad-actor infiltration of the social network is modeled by a parameter γ, the probability that each bad actor is followed by each authentic agent. Unless otherwise stated, we assume that authentic agents follow bad actors uniformly at random. Figure 2b illustrates the effective suppression of information quality when γ is high.

Fig. 2.

Subnetworks modeling authentic accounts (purple nodes) and bad actors (yellow nodes). a) Illustration of the follower link structure. Solid links indicate follower relations within each subnetwork. Both subnetworks have hub and clustering structure that mimics or derives from online social networks. Dashed links represent authentic accounts following bad actors, according to the infiltration parameter γ, which represents the probability that an authentic node follows any given bad actor. When $\gamma =0$ there is no infiltration and bad actors are isolated, therefore harmless; the opposite extreme $\gamma =1$ indicates complete infiltration, such that bad actors are followed by all authentic accounts. b) Effects of bad-actor infiltration γ on the quality of messages in synthetic networks with ${10}^3$ authentic agents and 100 inauthentic agents. For illustration purposes, both the authentic and inauthentic subnetworks in this panel are generated with the same method used for the inauthentic subnetworks in our experiments (see Methods). Node size represents the number of followers. The darker an authentic agent node, the lower the quality of messages in their feed.

The quality q and appeal a of messages originating from authentic accounts are drawn independently from two distinct distributions, reflecting empirical evidence that these messages tend to have high quality and low appeal (see Supplementary material). In contrast, we assume that bad actors can manipulate information in the network by creating messages with low quality ($q=0$) and deceptively high appeal. The appeal differential of content from bad actors is modeled by the deception parameter ϕ. In the absence of deception ($\varphi =0$), the appeal of bad-actor messages is drawn from the same distribution as those from authentic accounts. If $\varphi >0$, it represents the probability that bad-actor content has $a=1$, making it irresistible (see details in Methods).

Flooding is another tactic inauthentic accounts can use to amplify their influence, by crowding out high-quality information. To model this, the parameter θ is defined as a boost of engagement for bad-actor content (see Methods).

SimSoM lets us explore information diffusion on social media, including the properties of authentic accounts that might render them vulnerable to adversarial attacks and the effects of different manipulation tactics. In the next sections, we present the results from simulations of the model on online communities derived from an empirical follower network ($N\approx {10}^4$ Twitter accounts). The network has both scale-free structure (hubs) and high clustering (triangles), structural characteristics that are ubiquitous in socio-technical networks. It also has a realistic community structure, with two well-separated clusters of accounts capturing political polarization (see Methods).

Once the system reaches a steady state, in which the message quality across the network has stabilized (see Methods), we record the mean quality of the messages in the feeds of authentic agents. These measurements are further averaged across simulation runs with the same parameters but different random seeds. We simulate the information diffusion process in networks with different structures to explore whether social network features render communities vulnerable to bad-actor tactics. Similarly, we evaluate the impact of these tactics by evaluating the model with varying levels of bad-actor infiltration (γ), deception (ϕ), flooding (θ), and targeting specific types of accounts. We report on the relative quality, defined as the ratio of the average quality of authentic agents to that of a baseline without bad actors (see Methods).

Network vulnerabilities

Key structural features of the social network may play a role in amplifying our vulnerability to manipulation by bad actors. The empirical network has two features that are ubiquitous in social media: the presence of hubs and highly clustered communities. We can explore how overall quality is affected by these features through three alternative networks constructed by shuffling links while preserving hubs, cluster structure, or neither (see Methods). Figure 3 shows that the presence of clusters does not significantly affect the overall quality (purple vs. blue) because we assume that agents in each cluster are equally likely to follow bad actors. On the other hand, having hubs makes a network more vulnerable to manipulation: the relative system quality is significantly lower in the presence of hubs, both when there are clusters (purple vs. green, $p<{10}^{-3}$) and not (blue vs. orange, $p<{10}^{-3}$). This is because node in-degrees and out-degrees are highly correlated (Spearman correlation 0.9, $p<{10}^{-4}$) in the empirical network. Therefore, when authentic followers are concentrated among hubs, high-quality content is also concentrated among those hubs. This implies that high-quality content has more competition and becomes obsolete more quickly compared to the case in which this content is uniformly distributed among authentic nodes. The same does not apply to content from bad actors because this content spreads uniformly among authentic users. (The scenario in which content from bad actors mostly concentrates among hubs is explored later.)

Fig. 3.

Impacts of different network structural features on the average information quality, relative to the scenario without bad actors. The original network (“hubs $+$ clusters”) is visualized along with shuffled networks in which links from the original network are rewired while preserving clusters, hubs, or neither (“random”). Node size and color represent, respectively, the number of followers of an account and their political leaning ranging from liberal to conservative (red to blue, see Methods). Yellow nodes are bad actors. Pairwise statistical significance is calculated using the Mann–Whitney U test (*** for $p<{10}^{-3}$); only significant differences are reported.

Infiltration, deception, and flooding tactics

Bad actors may maximize their message spread by combining various manipulation tactics. We systematically quantify the effects of these tactics through simulations varying the parameters for infiltration (${10}^{-4}\le \gamma \le {10}^{-1}$), deception ($0\le \varphi \le 1$), and flooding ($1\le \theta \le 64$). See Methods for further details. Figure 4a,b,c illustrates the effects of individual malicious tactics on the overall quality of information spreading through the network, compared to a baseline scenario without bad actors. We observe that infiltration is the most harmful manipulation tactic: when authentic agents have a $\gamma =10\mathrm{\%}$ probability of following each bad actor, the average quality in the system is reduced to less than half. Flooding and deception have smaller effects. When low-quality content has $\theta =64$ times more exposure than authentic content, quality is reduced to less than 70%. When bad actors generate content with maximum appeal exclusively ($\varphi =1$), quality is reduced to about 70%.

Fig. 4.

Effects of individual and combined tactics by bad actors on the system’s message quality, relative to the scenario without bad actors. a) Varying infiltration γ, without flooding ($\theta =1$) or deception ($\varphi =0$). Shading represents 95% confidence intervals across runs in panels a–c. b) Varying flooding θ with infiltration $\gamma =0.01$ and no deception ($\varphi =0$). c) Varying deception ϕ with infiltration $\gamma =0.01$ and no flooding ($\theta =1$). d) Joint infiltration and flooding with no deception. e) Joint infiltration and deception with no flooding. f) Joint deception and flooding with infiltration $\gamma =0.01$.

Similarly, Figure 4d,e,f shows the effects of pairs of tactics combined. Infiltration is dominant, but more harm can be done in combination with flooding or deception: the average quality is reduced to 40% when $\gamma =0.1$ and $\theta =64$ (Fig. 4d) or $\varphi =1$ (Fig. 4e). Combining flooding and deception ($\theta =64,\varphi =1$, Fig. 4f) only results in marginal loss of quality (below 70%). With all three tactics combined ($\gamma =0.1,\theta =64,\varphi =1$, not shown), bad actors can further reduce the quality to 30%.

Reshare and exposure cascades

Empirical evidence has shown that among fact-checked claims, low-quality content (debunked claims) tends to have larger retweet cascades than high-quality content (confirmed claims) (9, 45). SimSoM allows us to examine how different factors may contribute to such a virality pattern. Figure 5 illustrates the effects of bad-actor tactics on the size of reshare cascades for content generated by both authentic agents (“high-quality”) and inauthentic agents (“low-quality”). We observe that low-quality cascade size is boosted by appeal. This is in line with the hypothesis that factors like novelty make false news more viral (9). However, low-quality cascades can be boosted most effectively through high bad-actor infiltration. Importantly, empirical analyses that do not reconstruct actual cascades from the data may underestimate intermediary amplification by inauthentic accounts, even when those accounts are removed as originators (cascade tree roots). On the other hand, our model lets us reconstruct the likely reshare cascades that include intermediary amplification by inauthentic accounts (see Methods).

Fig. 5.

Complementary cumulative distributions of reshare cascade sizes for low- and high-quality content, generated by inauthentic and authentic agents, respectively. The plots are based on 10 simulations. a) Effect of bad-actor infiltration γ, with no flooding ($\theta =1$) or deception ($\varphi =0$). b) Effect of flooding θ, with low infiltration ($\gamma ={10}^{-3}$) and no deception ($\varphi =0$). c) Effect of deception ϕ, with low infiltration ($\gamma ={10}^{-3}$) and no flooding ($\theta =1$).

To date, empirical data from social media platforms has allowed researchers to measure reshare cascades, but not exposure. As a result, few studies focus on exposure (46). Using the SimSoM model, it is possible to reconstruct not only likely reshare networks but also exposure networks, thus estimating how many accounts are exposed to (i.e. view) a message even if they do not reshare it (see Methods). Figure 6 compares the sizes of reshare and exposure cascades. In general, reshares underestimate exposures by roughly one order of magnitude. Excluding the smallest cascades and the largest ones (to avoid finite-size effects), we observe that exposure networks grow sub-linearly with reshare networks: $s_v\sim s_r^{\nu }$ where $s_v$ and $s_r$ are the exposure and reshare cascade sizes, respectively, and $\nu <1$ is the scaling exponent. This means that as messages go viral, exposures grow more slowly than reshares. The exponent is higher for low-quality ($\nu \approx 0.8$) than for high-quality content ($\nu \approx 0.6$), suggesting that for each extra reshare, messages posted by inauthentic agents gain more views.

Fig. 6.

Scaling between reshare and exposure cascade sizes. a) Scaling for low-quality messages (posted by inauthentic agents). b) Scaling for high-quality messages (posted by authentic agents). The exposure cascade size is averaged across messages with the same reshare cascade size, based on 10 simulations. The dashed lines provide a linear scaling reference, while the solid lines show the slopes (exponents) ν of power-law fits for reshare cascades of size between 10 and 1,000, yielding $\nu =0.80\pm 0.01$ (low-quality messages) and $\nu =0.56\pm 0.01$ (high-quality messages). The largest reshare and exposure cascades (corresponding to the circles in panels a and b) are also visualized for c) low-quality and d) high-quality messages, based on one simulation. Node colors are the same as in Fig. 3; node size represents out-degree, or influence. Here we use $\theta =1,\varphi =0,\gamma ={10}^{-2}$; the results are similar for other γ values.

Targeting tactics

The above results show that bad actors can use inauthentic accounts to infiltrate and disrupt an online public square. We have thus far assumed that all authentic agents have the same probability of following inauthentic ones, reflecting a scenario in which bad actors do not focus their efforts on specific potential followers. However, those interested in manipulating the network may want to maximize the spread of low-quality content through the community by targeting certain groups of accounts.

As an example, an adversary might target influentials based on the assumption that having such followers can multiply their impact—a message reshared by an influential account has a higher chance of going viral. Targeting influentials is well within the capability of bad actors and even automated accounts; the number of followers, often used as a proxy for influence (47), is public information on all social media platforms. A bad actor can easily interact with accounts having many followers by mentioning and/or following them (38, 48); other ploys include retweeting, quoting, and/or liking their tweets. There is empirical evidence of preferential targeting by bad actors that spread misinformation (1, 8). Targeting politically active accounts or habitual misinformation spreaders are also conceivable tactics.

An important question, then, is whether tactics targeting specific authentic accounts do in fact increase the manipulative power of bad actors. To explore this question, the model can be extended to account for various features of social media users. Here we consider five features that are available in the empirical data: number of followers (hubs tactic), propensity to share misinformation (misinformation tactic), political partisanship (partisanship tactic), or specific political leaning (liberal and conservative tactic). We then introduce a preferential targeting tactic for adversarial actors, according to which authentic agents have different probabilities of following inauthentic ones, proportional to one of these features. See Methods for details.

Figure 7 shows the impact of these targeting tactics on information quality. Counterintuitively, preferential targeting is less harmful than random targeting: the distribution of quality is uneven so that the targeted population is worse off, but other parts of the community are spared. Targeting tactics therefore tend to backfire if we assume that bad actors intend to maximize the spread of their content across the full community.

Fig. 7.

Effects of targeting tactics. a) Average information quality resulting from each tactic, as well as the default random targeting, relative to the scenario without bad actors. We highlight significant differences calculated using The Mann–Whitney U test (^**** for $p<{10}^{-4}$). b) Suppression of quality in the empirical network when bad actors specifically target influential accounts (hubs), and when they target politically left- (liberal) and right-leaning (conservative) accounts. The network has ${10}^3$ authentic agents (purple nodes) and 50 inauthentic agents. Node size represents the number of followers. The darker an authentic agent node, the lower the quality of messages in their feed. Significant changes due to targeting tactics are only observed when bad-actor infiltration is sufficiently high, therefore we use $\gamma ={10}^{-1}$ in experiments for both panels.

Targeting hubs, for example, results in a network with significantly higher average quality ($p<{10}^{-4}$). The amplification power of an influential is counterbalanced by the concentration of low-quality content, which has fewer chances to be reshared; the majority of other agents are left relatively free from manipulation. Note that this is analog to the reason why the presence of hubs leads to high-quality content being forgotten more quickly when hubs are not targeted by bad actors, as seen earlier.

The harm of manipulation through the network also diminishes when bad actors target accounts sharing a lot of misinformation or with specific political leaning ($p<{10}^{-4}$). The echo-chamber structure of the empirical network (Fig. 7b) helps interpret the latter finding: low-quality messages get shared and become obsolete rapidly within one densely connected partisan cluster, sparing the rest of the network.

Discussion

Social media platforms have enhanced the so-called attention economy, in which abundant content must compete for our scarce attention (49). But how to ensure that accurate, relevant, timely information wins this competition? To date, the policies that govern social media have been mainly guided by the concept of a free marketplace of ideas, rooted in John Milton’s centuries-old reasoning that truth prevails in a free and open encounter of opinions (50). In an ideal world, the wisdom of the crowd (51) would realize this vision by combining the opinions of many users (52). Unfortunately, several aspects of modern social media challenge the illusion of a public square or marketplace in which wise crowds access and select quality information (53, 54). First, information production is affected by information consumption (55), creating incentives to produce appealing but not necessarily high-quality content. Second, social influence undermines the wisdom of crowds because the information and opinions to which we are exposed online may be inauthentic, correlated through coordination (56, 57), or dominated by few influential individuals (58). Third, confirmation bias (59) can increase vulnerability to misinformation in social media (11, 60). Finally, the structure of information flow networks can distort perceptions and increase vulnerability to malicious actors (61, 62).

Exploration using SimSoM quantifies how manipulation by bad actors can prevent social media from functioning as a public town square. Simulations of the model reveal that making low-quality content highly appealing plays a lesser role compared to other harmful tactics by inauthentic accounts. This suggests that novelty, for example, may not provide the primary explanation for the virality of fake news, as previously hypothesized (9). Tactics that inflate engagement indicators, such as flooding, can erode the system’s quality just as well. More importantly, we find that infiltrating the network is a dominant harmful tactic available to bad actors—these accounts can drastically suppress quality by inducing only a small fraction of the community into following them.

A wealth of previous models and experiments have focused on information diffusion and popularity. Several studies have investigated the role played by network mechanisms affecting the popularity of individual posts, including exogenous and endogenous bursts of attention (63, 64), memory (65), novelty (66, 67), and position bias (68, 69). This literature considers the popularity of pieces of information in isolation. Market-like environments in which many messages compete for limited attention have received less consideration. Exceptions have considered the cost of learning about quality (70), distortions of quality assessments that result from aggregate knowledge of peer choices (56), and confirmation bias in the spread of misinformation (71). The SimSoM model proposed here extends the model of Weng et al. (72), who demonstrated that some posts inevitably achieve viral popularity irrespective of quality in the presence of competition among networked agents with limited attention. The model was formalized as a critical branching process and studied analytically, predicting that the popularity of posts follows a power-law distribution with heavy tails (73–75).

The current SimSoM model has several limitations. First, it neglects many mechanisms of actual online social media that may contribute to message exposure, such as search, personalization, and other details of secret platform algorithms. For example, increasingly popular feed ranking algorithms are based less on what is shared by social connections and more on out-of-network recommendations. Bad-actor tactics could be affected differently by such algorithmic changes (see Supplementary material). While the present model captures several universal criteria of feed algorithms (appeal, social engagement, recency), future work should investigate the vulnerability of different algorithmic affordances. Second, real user behaviors are driven by complex cognitive processes. Beyond finite attention, such cognitive aspects of information sharing are not explored here. Instead, the model assumes that all authentic agents follow a simple probabilistic rule in selecting the content to be shared. Large language models have recently been proposed as a way to model agents with more realistic behaviors (76). In addition, SimSoM models information diffusion through simple contagion. While this approach is in line with prior modeling (77), theoretical (45), and empirical (78) studies, complex contagion (25) can also play a role in the spread of certain types of harmful information (26). Despite these limitations, the current model’s predictions are consistent with empirical findings about the difference in virality between low- and high-quality information on Twitter (8, 9). This suggests a reasonable balance between model realism and generality.

We can think of inauthentic accounts as zealots, a minority of agents committed to a particular view. Opinion dynamics models have shown that a critical minority of active zealots can quickly drive a system to consensus toward their opinion (79–82). In our social media model, we only explore the capacity to suppress information quality rather than to drive consensus to a particular opinion.

While the structure of online social networks evolves over time, the follower network used in our experiments provides a more realistic setting than synthetic networks to run our simulations. In particular, it captures features of real social networks—hubs, clustering, and political homophily structure—that have been observed consistently over the years and across platforms (83–85) and that are known to play key roles in information diffusion (86–88). Furthermore, our sampling procedure captures a community of accounts that are both active and vulnerable to misinformation, as needed to study manipulation tactics that target such communities. Finally, the assumption that the follower network is static during an information-spreading simulation is consistent with observations that the dynamics of unfollowing are slow (89).

Our results suggest that, surprisingly, inauthentic accounts do not need to target hubs (influentials); they can do more damage by connecting to random accounts. Future research should focus on whether the strategic placement of bad actors within/across polarized online communities can sway users toward a particular outcome, for example by distorting popularity perceptions (61, 62). Further work is also needed to characterize the effects of strategic targeting when the attacker has limited resources.

The insights gained from the present findings suggest several countermeasures to increase the resilience of social media to manipulation. The first lesson is that we must make it more difficult for bad actors to infiltrate the network. Platform efforts to detect and take down deceptive accounts must be strengthened, especially as tactics to hack follower networks get more sophisticated (38).

Recent models show that caps on the depth and/or breadth of diffusion networks can decrease the ratio of distorted messages received by social media users (90). While these models assume that distortions occur randomly, SimSoM demonstrates how adversarial actors can exploit vulnerabilities by flooding our news feeds, thereby crowding out quality information. A countermeasure would be to challenge accounts that post at very high rates to prove that they are human. Users could also be warned when they follow accounts that post low-quality content and/or when friend accounts are suspended or take suspicious actions, such as changing names/handles (14) or posting and deleting large volumes of content (41).

Our findings suggest that a winning tactic for malicious accounts is to target all social media users, and not only the most influential, if their goal is to spread harm widely across the network. Literacy programs may provide some protection against disinformation. This is also suggested by a binary agreement model that assumes each agent may be committed to truth or disinformation (91). Social media platforms could lead by educating users about their vulnerability to manipulation and deception. The simplest version of this is through accuracy reminders, which can improve content quality during both posting (92) and resharing (93). Flooding reduces the effectiveness of such accuracy interventions (94). Our model could be extended to explore ways to combine friction and accuracy nudges by, e.g. lowering the probability that low-quality messages are reshared by agents after they are exposed to warnings.

Given that inauthentic accounts can be used to suppress human speech, granting them (or any entity that controls them) unlimited free-speech rights would seem to lead to a logical contradiction (95). Yet, ironically, efforts by social media platforms to moderate abusive accounts and even research on social bot detection have been assailed by some with charges of censorship (96). These questions may have significant repercussions on regulations designed to protect the online public square (53).

Methods

Social media diffusion model

SimSoM ^a is a parsimonious agent-based model inspired by the long tradition of representing the spread of ideas as an epidemic process where messages are passed along the edges of a network (97). The model simulates a directed follower network, as in Twitter/X, Mastodon, or Threads. Nodes represent agents (users) and links represent follower relations, which may or may not be reciprocal. The direction of a link goes from the follower to the followed (friend) account, capturing the flow of attention; when a friend’s post is reshared by a follower, information spreads in the opposite direction. In line with previous work (45, 77, 78), the diffusion process is modeled as simple contagion, where each exposure to a message results in the same resharing probability. We assume that the structure of the network is static (no unfollowing/blocking of accounts) during the information-spreading process. In contrast to classical epidemiological models, new messages are continuously introduced into the system in an exogenous fashion.

At each time step, an agent i produces a new message with probability μ or chooses one of the messages in their news feed to be reshared with probability $1-\mu$ (Fig. 1). The new or reshared message is then added to the news feeds of i’s followers. We set $\mu =0.5$, reflecting the empirical average ratio for English-language tweets (98). Based on empirical data, agents are assumed to have limited-size inventories: only the most recent $\sigma =15$ messages are retained in each news feed (see Supplementary material). The results are robust for different values of μ and σ (see Fig. S1 in Supplementary Material).

We assume authentic agents prefer to reshare messages posted by their friends that are appealing, recent, and popular. This is based on empirical evidence that users are more likely to share popular content according to engagement signals (99). The probability that an agent shares a message from their news feed, allowing it to spread, is proportional to the message’s appeal, social engagement, and recency. More explicitly, let $M_i$ be the feed of i ($|M_i|=\sigma$). The probability of message $m\in M_i$ being selected is $P(m)=a_m{e}_m{r}_m/\sum _{j\in M_i}{a}_j{e}_j{r}_j$ where $a_m$ is the appeal of message m, $e_m$ is the social engagement, i.e. the number of times it was (re)shared by i’s friends, and $r_m$ is the recency. Message recency decays with time as a stretched exponential function $r_m(t)=e^{-0.4t^{0.4}}$, where t is the “age,” or the time passed since m was first introduced to the agent’s news feed. This decay function is based on empirical online news engagement data (67). To model flooding, bad actors can be more active than authentic agents, sharing a message θ times such that it appears to have higher engagement than authentic content.

Quality and appeal

Both the quality q and the appeal a of new messages are defined in the unit interval. Informed by empirical data, q and a for authentic accounts are assumed to be independent (see Fig. S2 in Supplementary Material). For authentic accounts, quality q is drawn from an exponential distribution $P(q)=C{e}^{\tau q}$ where high-quality information is more common than low-quality information. The term $C=\frac{\tau }{e^{\tau }-1}$ is a normalizing constant such that ${\int }_0^{1}P(q)dq=1$; the exponent $\tau =10$ is estimated empirically (see Fig. S3 in Supplementary Material). We independently draw appeal from the distribution $P(a)=(1+\alpha )(1-a{)}^{\alpha }$, with $\alpha >1$ (Fig. S4a in Supplementary Material). We set $\alpha =4$. This choice for the appeal probability density function reproduces a broad distribution of reshares comparable to that observed in real-world information diffusion networks: few messages go viral while the majority do not (Fig. S4b in Supplementary Material).

On the other hand, we assume that bad actors strictly generate low-quality messages ($q=0$). The potentially deceptive nature of this content is modeled by the deception parameter ϕ ($0\le \varphi \le 1$), the probability that a bad-actor message is irresistibly appealing. With probability ϕ, we set $a=1$, and with probability $1-\varphi$ we draw appeal from the same distribution as for authentic accounts, $P(a)=(1+\alpha )(1-a{)}^{\alpha }$. If $\varphi =0$, bad actors and authentic accounts generate messages with appeal drawn from the same distribution. If $\varphi >0$, bad-actor messages are more likely to have high appeal; the larger ϕ, the greater the potential virality of low-quality content by bad actors.

Follower network

We run simulations on a follower network derived from empirical Twitter data. This network was constructed from a 10% random sample of public tweets between 2017 June 1 and 30 (100). The data include users (excluding likely automated accounts) that shared at least 10 links to news sources, at least one of which was to a source labeled as low-quality. Only news sources with known political valence were considered. Based on the shared links, most accounts in the dataset have an associated partisanship score (from $-1$ for left-leaning to $+1$ for right-leaning) defined as the average political bias of the news sources they share; and a misinformation score defined as the fraction of posts linking to low-credibility sources. Both partisanship and misinformation scores, included in the original network dataset, were based on news source labels from third-party fact-checkers (84).

From the original dataset, we select nodes with both partisanship and misinformation attributes. We further reduce the size of the network to speed up our simulations. We apply k-core decomposition to select $N=\mathrm{10,006}$ nodes forming the $k=94$ core. Finally, we remove a random sample of edges to decrease the density of this core while preserving the average in/out-degree (number of friends/followers) of the original network ($k=0$ core). This results in $E=\mathrm{1,809,798}$ edges; each node has on average approximately 180 friends/followers (Fig. 3, “hubs+clusters” network).

Bad-actor subnetwork

Since the empirical network described above does not include likely inauthentic accounts, we use it to model the subnetwork of authentic accounts. We then add a subnetwork representing inauthentic accounts that infiltrate the system. The ratio between the sizes of the two subnetworks is described by β, i.e. for an authentic agent subnetwork of N nodes, the inauthentic subnetwork is composed of $\beta N$ nodes (Fig. 2). Since there are many types of inauthentic accounts (trolls, social bots, cyborgs), estimating the percentage of these on social media is a very difficult task. We thus set $\beta =0.05$ following a rough estimation of the prevalence of bots on Twitter.^b Note that from the perspective of information quality in the model, increasing the prevalence of bad actors is equivalent to increasing their infiltration. Therefore, we focus on the effects of varying γ rather than β.

The inauthentic account subnetwork is created using a directed variant of the random-walk growth model (101). This captures the presence of hubs and clustering (directed triads). Specifically, the network is initialized with four fully connected nodes. We then add one new node at a time, assuming that each has fixed out-degree $k_{out}=3$. Once a new node i comes into the network, it links to (follows) a randomly selected target node (friend) j. Each of the remaining $k_{out}-1=2$ friends are selected as follows: with probability expressed by a parameter p, i follows a random friend of j’s; with probability $1-p$, i follows another randomly selected node. Following friends of a friend has the effect of generating closed, directed triads and approximates a preferential attachment process, giving rise to hub nodes with high in-degree. The parameter p thus models both hubs and clustering. We use $p=0.5$.

In addition, the bad-actor subnetwork is designed to manipulate information flow and collective attention by spreading certain messages. Therefore, we assume that bad actors get random authentic accounts to follow them: we add a directed link from each authentic node to each bad-actor node with probability γ. (In the next subsection, we present alternatives to this random targeting tactic.) The parameter γ models the degree of infiltration of the network by bad actors (Fig. 2). When $\gamma =0$, there is no infiltration and bad actors are isolated, therefore harmless; the opposite extreme $\gamma =1$ indicates complete infiltration such that bad actors dominate the network. Because we are not concerned with the quality of messages on bad-actor news feeds, they do not follow or reshare content from authentic agents for the analyzes in this paper.

Bad-actor targeting tactics

In the scheme described above, the authentic agents that follow bad actors do so randomly. We also study scenarios in which bad actors target certain accounts as potential followers (Fig. 7). In all of these scenarios, each bad actor is still followed by $\gamma N$ authentic accounts on average, but these targets are selected according to some criterion, such as whether they are politically active or have many followers.

Each targeting tactic is modeled by making the probability that an authentic agent i follows bad actors proportional to some feature $f(i)$. In the hubs tactic, we set $f(i)=k_{in}(i)$, the number of followers of i. Misinformation and partisanship attributes of authentic accounts in the empirical network allow us to model other targeting scenarios. In the misinformation tactic, $f(i)$ is set to i’s misinformation score, i.e. their propensity to share misinformation. The partisanship score is used in the liberal and conservative tactics, while its absolute value is used in the partisanship tactic.

Overall quality

The effects of manipulation on authentic agents are quantified by the quality of all content in circulation. The overall quality of the information system at time t is measured by the average quality across all the messages visible through the feeds of the authentic agents:

$$Q_t=\frac{1}{\sigma N}\sum _{i=1}^N\sum _{m\in M_i}{q}_{i,m,t},$$

where $q_{i,m,t}$ is the quality of the message in the mth position in authentic user i’s feed at time t.

The system’s quality at each time step t is calculated with an exponential moving average ${\overline{Q}}_t=\rho {\overline{Q}}_{t-1}+(1-\rho )Q_t$. As the simulation takes place, some of the messages become obsolete quickly, while others live longer and infect a larger fraction of the network. The simulation ends once the system reaches a steady state in which the difference between two consecutive values of the quality moving average is smaller than a threshold, i.e. $|{\overline{Q}}_t-{\overline{Q}}_{t-1}|/{\overline{Q}}_{t-1}<ϵ$. See Supplementary material for further details about model convergence to a steady state. The average quality reported for all analyzes is calculated at the end of the simulation.

Cascade construction

A reshare cascade is a tree that begins when an agent i (root) posts a new message m. After that, a new link is created when a follower j of i reshares m. We say that i is the parent of j in the tree. Similarly, an agent’s exposure to a message is defined as having that message on their feed while being activated—we assume that an agent who is about to share or reshare a post has seen the content on their feed. A link in the exposure cascade for message m is created between an activated agent with m in their feed and their friend who had shared m. When constructing both reshare and exposure cascades, m may have been reshared by more than one of an agent’s friends. When this occurs, one of them is selected at random to be the parent in the cascade tree. This choice does not affect a cascade tree’s size, but might affect its structure. Other alternatives, such as selecting as parent the friend who most recently shared m, can be explored if one desires to analyze the structure of the reshare or exposure cascades. By definition, reshare cascades can be as small as one and exposure cascades can be as small as zero.

To capture the complete diffusion cascades, the size distributions plotted in Fig. 5 include only cascades of “extinct” messages, those that have become obsolete and are no longer present in any news feed by the end of simulations.

Simulation framework and parameters

For each set of parameters, we run 10 simulations starting from random conditions; the reported average quality is the average across these runs.

We explore values of the bad-actor parameters to cover broad ranges of manipulation tactics. The infiltration parameter γ is a probability, and we consider values in the unit interval on a logarithmic scale to focus on more realistic low-probability values. The deception parameter ϕ also represents a probability and we examine the full unit interval. Finally, the flooding parameter θ is a multiplicative factor on the volume of posts, and we explore the range 1–64 on a logarithmic scale to capture the high volumes observed in empirical analyzes (41). In scenarios with bad actors in the system, the default parameters are $\gamma =0.01$, $\varphi =0$, and $\theta =1$. Except for the simulations exploring inauthentic targeting tactics, authentic followers of the bad actors are selected at random.

The parameters $\rho =0.8$ and $ϵ=0.0001$ were tested to ensure that the system’s quality stabilizes at the steady state (see Supplementary material).

We shuffle the follower network in various ways to derive the scenarios reported in Fig. 3. The “hubs+cluters” network is the original one. In the “hubs” network, we shuffle the original network while preserving the degree distribution and destroying any community clustering. In the “clusters” network, the outgoing endpoint of each edge in the original network is rewired to another node within the same cluster; this preserves the cluster structure and destroys the hub structure. In the “random” shuffle, all edges are rewired at random with uniform probability, destroying all hub and clustering structure.

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

This work was carried out in part while X.L. visited the Observatory on Social Media at Indiana University, with support by the China Scholarship Council. B.T.T., A.F., and F.M. were supported in part by the Knight Foundation. B.T.T. was funded in part by the Ostrom Workshop. A.F. and F.M. were funded in part by DARPA (grants W911NF-17-C-0094 and HR001121C0169). F.M. and B.T.T. were supported in part by the Swiss National Science Foundation (Sinergia grant CRSII5_209250). F.M. was supported in part by the Democracy Fund, and Craig Newmark Philanthropies. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.

Author Contributions

F.M. and A.F. formulated the research and developed the methodology. F.M. and B.T.T collected the data. B.T.T and X.L. wrote the code and performed analyses. All authors prepared the manuscript.

Preprints

This manuscript was posted on a preprint at doi.org/10.48550/arXiv.1907.06130.

Data Availability

Code and data to implement the model and reproduce results are available at github.com/osome-iu/SimSoM.