Malicious Actors on Twitter: A Guide for Public Health Researchers

Abstract

Social bots and other malicious actors have a significant presence on Twitter. It is increasingly clear that some of their activities can have a negative impact on public health.

This guide provides an overview of the types of malicious actors currently active on Twitter by highlighting the characteristic behaviors and strategies employed. It covers both automated accounts (including traditional spambots, social spambots, content polluters, and fake followers) and human users (primarily trolls). It also addresses the unique threat of state-sponsored trolls. We utilize examples from our own research on vaccination to illustrate.

The diversity of malicious actors and their multifarious goals adds complexity to research efforts that use Twitter. Bots are now part of the social media landscape, and although it may not be possible to stop their influence, it is vital that public health researchers and practitioners recognize the potential harms and develop strategies to address bot- and troll-driven messages.

In their editorial, Allem and Ferrara pose the question, “Could social bots pose a threat to public health?”¹ Their answer is a resounding “Yes.” Social bots and other malicious actors are a disruptive force on online social networks. Although bots and trolls have been widely covered in the media for their role in politics, only in the past year has their activity garnered significant attention in public health.^1–5 The threat of social bots to public health is multifaceted. Bots can directly influence users by spreading content that works against public health goals, such as antivaccine propaganda or spam for products such as e-cigarettes.^6,7 The volume of bot-produced posts can also distort efforts to use social media data to gauge public sentiment, potentially limiting the usefulness of novel surveillance efforts. More fundamentally, the practice of public health depends on clear communication between practitioners and the communities they serve, and the interference of malicious actors could erode public confidence in online communication.

In this commentary, we highlight the diversity of malicious actors on Twitter, describe some characteristic behaviors, and introduce ways to recognize social bots and other malicious actors in context. This commentary evolved from our mixed-methods research studying vaccination discourse on Twitter and a tutorial presented at the Social Computing, Behavioral–Cultural Modeling, and Prediction and Behavioral Representation in Modeling and Simulation conference in July 2018.⁸

BACKGROUND

The term “bot” can connote different things depending on context. Pew defines bots broadly as “accounts that can post content or interact with other users in an automated way and without direct human input.”⁹ Bots, by definition, are not human. Some bots try to imitate humans by mimicking their online behaviors. A social bot “automatically produces content and interacts with humans on social media, trying to emulate and possibly alter their behavior.”^10(p96) Further blurring the line between human and machine, there are “cyborg” accounts that alternate between human control and automation.¹¹ While it is nearly impossible to distinguish automated accounts with certainty, there are bot-like behaviors that indicate an account could be automated.

Bots exist on all social media platforms; however, our research and most existing scholarship is focused on Twitter bots. Some bot-like behaviors may be consistent across platforms, but more research is necessary to identify characteristic behaviors by platform. The true number of automated accounts on Twitter is unknown, but a recent Pew report suggests that up to two thirds of all activity on Twitter is from automated accounts.⁹ Even a single automated account can have an outsized impact, as the 500 most active bots shared 22% of the total links on Twitter.⁹ When focusing specifically on social bots, a recent estimate classified between 9% and 15% of Twitter accounts as social bots.¹² With 336 million active monthly Twitter users worldwide, even conservative estimates would translate into millions of social bots.

Not all automated accounts engage in malicious behavior—in fact, many automated accounts operate within the bounds of Twitter’s terms of service. Generally, these accounts are clearly identified as bots. For example, @netflixbot automatically tweets as new media is added to Netflix. Organizations and other users may also utilize automated tools to post content. Malicious applications of automation violate Twitter’s terms of service by using accounts to falsely represent a human, post spam or malware, or engage in phishing (using accounts to induce individuals to reveal personal information). Current research focuses on distinguishing between bots and humans, but future research may focus on the likelihood that bots will engage in harmful activities.

BOT DETECTION

Bot detection is not foolproof and continues to get more difficult as bots evolve in sophistication.¹⁰ Currently, the best bot-detection techniques rely on a combination of methods including complex algorithms and machine learning processes.¹³ For the average user, Botometer (https://botometer.iuni.iu.edu) is a publicly available tool for Twitter bot detection. Botometer relies on rule-based detection, which assesses thousands of account features to derive a score estimating likelihood of automation.¹⁴ However, the creators of Botometer emphasize, “While bot scores are useful for visualization and behavior analysis, they don’t provide enough information by themselves to make a judgement about an account.”¹⁵

It is also possible to assess accounts manually. According to Nimmo, hallmarks of an automated user are the “three A’s”: activity, anonymity, and amplification, all of which can be observed by scrutinizing an account history and profile.¹⁶ Automated accounts tend to have irregularly high levels of daily activity.¹⁶ The Oxford Internet Institute estimates that any account posting more than 50 tweets per day is suspicious and is likely to be automated.¹⁶ Second, automated account profiles tend to contain very little personal information or rely on generic information. The least sophisticated bots may simply have handles that are alphanumeric scrambles and will leave profile details blank.¹⁶ Others may use stock photos or steal profile pictures off the Web but often fail to match profile information to photos (e.g., typically male names with female profile pictures).¹⁶ Finally, many automated accounts are created to spread information and amplify specific content. Frequently retweeting similar content could be a sign of automation. It is also suspicious if the number of retweets and likes on posts regularly exceeds the number of account followers.¹⁶ In general, it is easier to recognize irregular activity on Twitter if you are familiar with the typical discourse, popular accounts, and hashtags associated with a given topic.

MALICIOUS ACTORS

The world of bots is diverse, with specialized bots to accomplish specific goals. There are also human users who seek to disrupt online discourse. Our goal is not to provide a comprehensive overview of bot-detection strategies but, rather, to highlight some hallmarks of bot-like behavior as we have observed it. Our published work has employed a variety of bot-detection strategies, some requiring advanced computational skills and others based on straightforward manual analysis.^7,13 Much of this work has been done by using published data sets of confirmed bots or trolls. Each strategy has its strengths and weaknesses; manual annotation can reveal a great deal of information about a single account but is time- and labor-intensive, and automated methods can quickly assess many accounts but tend to provide only a big-picture perspective of accounts in aggregate. We believe that even without advanced tools, all social media users should at least be aware of different actors and the types of strategies that they employ. Understanding the diversity of malicious actors is important to fully understand the ways public opinion can be manipulated on Twitter. We present a partial list of the types of malicious actors, their identifiable characteristics, and examples from our work with vaccines (Table 1).

TABLE 1—

Taxonomy of Malicious Actors on Twitter: Characteristics, Examples, and Potential Effects

	Identifiable Characteristics	Examples From Research on Vaccines	Potential Effects on Surveillance of Attitudes and Intentions
Automated accounts
Traditional spambots	High activity level, featuring commercial content14	Of the vaccine-related content posted by known traditional spambots,17 the majority were job openings in vaccine-related fields (MA)	Minimal effects; content is usually not relevant to indicators of vaccine uptake or attitudes
	May not try to hide promotional nature of account
	Anonymous profiles may have empty profile photos and minimal account information16
Social spambots	Difficult to detect manually17	Suspected botnet identified when 61 known social spambots17 simultaneously posted identical messages about vaccines and Rocky Mountain Fever (a rare disease with no known vaccine; MA)	May promote antivaccine content to gain followers and ad revenue, with negative implications for attitudes; minimal effect on surveillance of intentions
	Moderate activity level	Known social spambots17 share similar types of vaccine content as general users, high proportions of retweeted content, news headlines, and links to outside content including alternative products (MA)
	Post a mix of regular and promoted content; large friend networks, sometimes including many other social bots	When we compared total vaccine-related activity from known social spambots against general vaccine-related activity, we found that social spambots post more polarizing vaccine content (CA)7
	Account profiles may use stolen or generic images and names16	In that same study, social spambots featured more antivaccine content than provaccine content (CA)7
	A single account may tweet in multiple languages16
	May synchronize with other social bots to tweet similar content as part of a botnet
Content polluters	Difficult to detect manually18	Known content polluters18 share similar types of vaccine content as general users, including retweeted news headlines and links to outside sites (MA)	May promote antivaccine content to gain followers and ad revenue, with negative implications for attitudes; minimal effect on surveillance of intentions
	Highest levels of activity, tweeting more frequently than all other bot types13	When we compared total vaccine-related activity from known content polluters against general vaccine-related activity, we found that content polluters post more polarizing vaccine content (CA)7
	Large friend networks13	In that same study, content polluters featured more antivaccine content than provaccine content (CA)7
	High levels of amplification, lots of retweeted content	Antivaccine content from known content polluters includes posts promoting vaccine alternatives, antivaccine conspiracy theories, and stories of extreme vaccine side effects (MA)
Fake followers	Low activity level, infrequent posting13	Very little vaccine-related content from known fake followers; only examples were links to pornography using random strings of keywords as text (e.g., “hot single mom adult vaccines”; MA)	Minimal effects
	Large friend networks13
	Account information stolen from real accounts or comprising generic information19
Semiautomated accounts
Cyborgs	Very difficult to identify	Accounts with intermediate botometer scores (not clearly a bot or a human) posted more polarizing vaccine content than general users7; these accounts could include some cyborg accounts (CA)	May have negative implications for attitudes and intentions if account operator masquerades as human user
	Could be a bot-assisted human11	Suspected cyborgs include active antivaccine accounts that can be linked to real people, but display high levels of activity and amplification like fully automated accounts (MA)
	Could be a human-assisted bot11
	Post more frequently than either average humans or average bots11
Malicious humans
Trolls	Intentionally disruptive behavior	We have no way of confirming trolling behavior, but suspect that some violent threats and extreme conspiracy theories could be coming from trolls (MA)	May have negative implications for attitudes and intentions as account operator masquerades as human user, but unless automation is used, will likely not materially affect aggregate results
	Motivated to harm
	Sometimes characterized by aggressive, violent, or sexually explicit content
State-sponsored trolls	Varies by state; Russian trolls are known to sow discord by promoting debate over politically charged issues; Chinese trolls are paid to divert conversations away from politically sensitive topics20	#VaccinateUS hashtag used by Russian trolls to target both pro- and antivaccine concerns (MA)7	May have negative implications for attitudes and intentions as account operator masquerades as human user, but sponsored coordination means that material effect on aggregate results is plausible
		Known Russian trolls posted more polarized vaccine content than general accounts, with roughly equal proportions of pro- and antivaccine messages (CA)7

Note. CA = identified in computational analysis; MA = identified in manual analysis.

Spambots are a class of bots that exist to post and repost unsolicited advertising content.¹⁷ Spambots violate Twitter’s terms of service. However, spambots serve a significant role in the online economy and, therefore, continue to thrive. Traditional spambots are typically easy to detect, as they post commercial content with high frequency. A subset of spambots have adapted to evade detection methods by mimicking human users. These social spambots post a combination of promoted and general content to better imitate a real account. Furthermore, the name “spambot” may be misleading: social spambots may promote a narrative, rather than a product. These methods are effective; typical human users on Twitter cannot reliably distinguish between social spambots and genuine accounts.¹⁷

Content polluters are a type of social bot that utilizes diverse strategies to aggressively distribute unsolicited content. Content polluters were first identified by Lee et al. by setting a “honeypot” trap, utilizing a generic online profile to attract and monitor bot-like accounts.¹⁸ They identified a class of bots engaged in disruptive techniques: blasting friend networks with spam, inserting @mention messages to legitimate users, and relying on reciprocity in follow requests to build networks.¹⁸ Like with other social bots, distinguishing content polluters from genuine users is a difficult task.

Social bots can be organized to work in unison to maximize impact. A social botnet links together social bots under a single botmaster’s control to engage in coordinated malicious activities while reducing risk of detection.²¹ Social botnets can manipulate public opinion by creating the impression of consensus on a topic by utilizing linked bot accounts to tweet and retweet messages in an echo chamber.¹⁷ Botnets can also manipulate “digital influence” on Twitter by skewing metrics of popularity and reach.¹⁶ Organized bot-driven narratives have been described as “astroturf” campaigns, as they mimic and subvert true “grassroots” types of activism.²⁰

Not all bots distribute content. An online marketplace has developed to allow individuals to purchase fake accounts to boost their number of followers. The New York Times drew attention to the Devumi company, which for the low price of $49 guaranteed to produce 5000 Twitter followers.¹⁹ The Times exposed celebrities, athletes, and politicians who had purchased fake followers attempting to bolster their online presence. These fake followers are a type of amplification bot that can be used to promote the reach of a specific account or narrative. The typical fake follower account may mirror the account of an actual human, sometimes using a stolen photo and profile information, but will follow thousands of accounts, with few of those accounts following back.¹⁹

Humans do not need automation to engage in malicious activities online. One common tactic is trolling. A troll is a user who “constructs the identity of sincerely wishing to be part of the group in question, including professing, or conveying pseudo-sincere intentions, but whose real intention(s) is/are to cause disruption and/or to trigger or exacerbate conflict for the purposes of their own amusement.”^22(p237) While there has been little empirical research on trolling, a survey of heavy Internet users found that 5.6% self-reported enjoying “trolling” others.²³ To complicate matters, automated accounts can also be programmed to engage in trolling behaviors.

There is substantial evidence to suggest that several nations are engaging in “targeted online hate and harassment campaigns” that are designed to “intimidate and silence individuals critical of the state” or to “persecute perceived opponents at scale.”^24(p1) These forms of state-sponsored trolling vary by context. Typical strategies include making accusations of collusion or treason, using violent hate speech or sexual harassment, and the distribution of memes to spread disinformation.²⁴ In the United States, Russian state-sponsored trolling has received the most attention. The hallmark of Russian trolls is to exacerbate both sides of a controversial topic, with the goal of sowing discord. However, in China, state-sponsored trolls are paid to promote propaganda and distract from threatening political debate.²⁴

WHY THIS MATTERS IN PUBLIC HEALTH

We use vaccination as an example of how malicious actors wield significant influence in online discourse on a topic of great public health significance. Most Americans believe vaccines are safe and effective, yet antivaccine content has an outsized impact online. As researchers, we have found it difficult to know if this activity is driven by a small group of dedicated antivaccine “activists” or if this content is promoted by automated accounts. The 2015 Defense Advanced Research Projects Agency bot challenge identified networks of illicit bots spreading antivaccine content, including the conspiracy theory hashtag #CDCWhistleblower.²⁵ Similarly, our results suggest that certain types of bots are utilized to post antivaccine messaging.⁷ To us, this suggests that bots are at least partially inflating antivaccine activity on Twitter and employing a diverse set of strategies to do so. To continue to assess online discourse on vaccination without consideration of these actors would only serve to legitimize bot activity and undermine the validity of the research.

Bots are part of the social media landscape and will only increase in sophistication and reach. This presents multiple challenges for both researchers and public health practitioners. For public health researchers, particularly those working with social media data, this could mean several things. First, the distorting effect bots have on online discourse will continue to grow. Large numbers of social bots posing as legitimate users and bot-nets amplifying a specific message could distort any surveillance efforts, if not taken into consideration in advance. Any surveillance approach that relies on social media data needs to proceed cautiously, craft careful research questions, develop search strategies with an understanding of how different types of bots are likely to have an impact on that research question, and should be cross-validated against trusted metrics.

Second, the diversity of actors and their multifarious goals add layers of complexity to online discourse. Some bots may be churning out content that negatively affects public health as a primary goal. For others, this content may simply be means to a different end, such as spreading malware, selling products, creating discord, prompting others to question authority, undermining trust, or simply to get attention. It is not always possible to know why a bot was created or what motivates a malicious actor.

Third, combatting bot messaging is more critical than combatting the medium. Bots will continue to evolve to evade detection, so relying on bot detection to shut down bot accounts is not enough.

In large part, the success of a bot-driven narrative depends on how willing the public may be to engage with promoted content. Recent news articles have demonstrated that bots and trolls are very successful in leveraging social controversy into clicks. Understanding how different malicious actors are spreading messages and understanding which ones are successful may give public health workers an idea of which ideas appeal to the public (such as #CDCWhistleblower) and which ideas do not (such as #vaccinateUS). Thinking like a bot may be the first step in combatting them, both online and off.

Online, increasing the presence of “official” health narratives may not be enough.² It may even be counterproductive to engage with bot-driven narratives directly, as it may simply “feed the trolls” and encourage further bot engagement. Offline, it is important for practitioners to use trusted relationships to dispel misinformation directly. Increasing social media literacy may help alert the public to malicious users and increase recognition of bot-driven narratives. These challenges also demand that both researchers and practitioners will need to forge new partnerships with computer science professionals to stay abreast of the rapidly changing technology. These challenges come at a time of reduced resources for both practice and research to address public health threats. Yet, a collaborative effort is essential to effectively respond to this growing threat.

The issue of bots and other malicious actors points to a much larger issue, that of how to manage the spread of misinformation online. While misinformation is not a new phenomenon, new technologies allow for the amplification of misinformation at a scale that is unprecedented. There are known and trusted strategies for addressing misinformation in the field of health communication, but more research is needed to fully understand how misinformation spreads online. A commentary from Chou et al. highlighted the need to assess the prevalence of misinformation online and the need to understand how it is spread³; we echo this sentiment and emphasize the need for empirical studies to assess the role of bots and other malicious actors in the spread of misinformation across multiple platforms and to develop data-driven solutions to this growing threat. This research will be necessary to allow researchers, practitioners, and policymakers to limit the impact of malicious actors on public health.