Are you Scared Yet?

Abstract

In March 2012, the CDC launched “Tips from Former Smokers,” a $54 million national campaign featuring individuals experiencing long-term health consequences of smoking. The campaign approach was based on strong evidence that anti-tobacco ads portraying fear, graphic images, and personal testimonials are associated with attitudinal and behavior change. Yet it was also controversial; critics cited the danger that viewers might reject such intensely graphic messages. Tasked with informing this debate, our study analyzes the corpus of Tips campaign-related tweets obtained via the Twitter Firehose. We provide a novel and rigorous method for media campaign evaluation within the framework of the Extended Parallel Process Model. Among the relevant Tweets, 87% showed evidence of message acceptance, while 7% exhibited message rejection.

Nearly five decades after the first US Surgeon General’s Report on Smoking and Health (US Department of Health and Human Services, 2014; US Public Health Service Office of the Surgeon General, 1964), an estimated 443,000 Americans still die each year from smoking-related disease. Thus tobacco use remains the leading preventable cause of death and disease in the United States (Centers for Disease Control and Prevention, 2011) and in the world (World Health Organization, 2011). For this reason, on March 15, 2012, the Centers for Disease Control and Prevention (CDC) launched “Tips from Former Smokers” (Tips), a $54 million national media campaign that used real-life stories from smokers suffering from the long-term health consequences of smoking, including amputation, tracheotomy, paralysis, and heart surgery (Flock, 2012). The combination of graphic images, fear appeals, and personal testimony in the ads was expected to elicit a negative reaction to smoking or to the thought of smoking. Each ad also prompted smokers to call the national quit line (1-800-Quit-Now) or visit the national quit website (www.smokefree.gov) for free cessation resources (Rigotti & Wakefield, 2012).

The campaign’s graphic and emotional approach was based on evidence that anti-tobacco ads with negative emotional appeals are associated with higher recall, and are more effective at promoting cessation than humorous or non-emotive ads (Biener, McCallum-Keeler, & Nyman, 2000; Biener, Ji, Gilpin, & Albers, 2004; Durkin, Biener, & Wakefield, 2009; Farrelly et al., 2012; Flock, 2012). Yet the fear strategy employed by Tips is not without controversy. First, some scholars have raised ethical concerns about the use of fear in such campaigns. For instance, Gass and Seiter (2011) consider unethical the use of threats, because to be effective such messages must evoke a state of “psychological distress” in recipients (Gass & Seiter, 2011, p. 348). Second, researchers consider that there are alternatives to fear that remain unexplored, such as guilt (Coulter & Pinto, 1995) or positive emotions (Lewis, Watson, White, & Tay, 2007), which warrant comparative research with fear. Finally, fear appeals can fail to evoke a response, or worse yet, may boomerang (Witte, 1992). Precisely the last argument may serve as fodder for the two previous ones if evidence showed that fear appeals fare poorly in terms of responses among the population. One such population response may appear in messages on the rapidly-diffusing social networking platform Twitter. Did the Tips campaign scare Twitter users? And if so, did the fear appeals go too far?

This study presents an analysis of Twitter messages about the CDC “Tips from Former Smokers” campaign. The main objective is to determine whether the Twitter conversation contained messages that displayed high perceived threat—i.e. in terms of high severity and susceptibility—versus low perceived threat or disregard. These terms are taken from the Extended Parallel Process Model (Witte, 1992), which we use as an analytic framework. We hope to make a novel contribution to the literature by (a) informing the debate on whether fear appeals represent persuasive public health messaging strategies; and (b) using large datasets—big data—gathered from Twitter to represent immediate responses to a specific campaign, spontaneously generated from within individuals’ natural environments. Indeed, population responses gathered in naturalistic or realistic settings, such as Twitter, are “desperately needed” in the literature (Witte & Allen, 2000, p. 605).

The Tips Campaign

Since the Master Settlement Agreement of 1998, state-sponsored tobacco control media campaigns have been a central component of comprehensive tobacco control efforts in the US (Emery et al., 2012; Szczypka et al., 2005), and there is a growing body of evidence suggesting that such campaigns are associated with less smoking among youth and adults, more anti-smoking attitudes among youth, and higher rates of intentions to quit and increased numbers of quit attempts among adults (Emery et al., 2005, 2012; Hopkins et al., 2001; US Department of Health and Human Services, 2000; Wakefield et al., 2008). Moreover, there is strong evidence that US states with the longest-running media campaigns have significantly lower smoking prevalence than those with more recent campaigns (Hopkins et al., 2001; US Department of Health and Human Services, 2000). With few exceptions, however, levels of exposure to state-sponsored anti-tobacco campaigns have been falling steadily since 2003, as states struggled with economic downturns that resulted in broad cuts to tobacco control programs (Szczypka et al., 2005). At the same time, the rate of decline in smoking prevalence across the US has slowed significantly. While smoking prevalence decreased from 42.4% to 20.9% between 1965 and 2005, between 2005 and 2010 prevalence fell to just 19.3% (CDC, 2011; Syamlal & Mazarek, 2011).

The Tips campaign marked the first national media campaign to address tobacco use, and also the first time the federal government had used paid advertising to prevent smoking and encourage quitting (Harris, 2012). The program was made possible by funding from the Patient Protection and Affordable Care Act 2010 (McAfee, Davis, Alexander, Pechacek, & Bunnell, 2013). The 12-week campaign reached every media market in the United States, with advertising buys on billboards, television, radio, print, and Internet websites targeting adults 18-54 years old (Flock, 2012).

Evidence suggests that Tips achieved its goal of reducing smoking in the US: in early results, the CDC reported a 132% increase in call volume to its national quit line, compared to the same 12-week period in 2011; and unique visits to the national quit website increased by 428% compared to 2011. Overall, 718,090 additional calls and unique website visits were recorded during the Tips campaign (Augustson et al., 2012). While the campaign was estimated to result in significant increases in cessation help-seeking, quit attempts, and communications between nonsmokers and smokers about the dangers of smoking (Augustson et al., 2012; McAfee et al., 2013), it is still unclear whether viewers were more likely to reject than to embrace the hard-hitting messages. Data from population surveillance surveys will provide further evidence about longer-term outcomes such as level of successful quitting. However, social media data—so-called big data—may offer deeper insights into how viewers reacted to the ads: that is, whether audiences engaged with the ads in ways that might support behavior change, or rejected or disregarded the messages before they became incorporated into such a thought process.

Twitter Response to the Tips Campaign

Twitter, the microblogging social media platform with more than 500 million users, has been called the world’s largest focus group (St. Amand, 2013), providing a platform for unfiltered expression (Papacharissi & de Fatima Oliveira, 2012). Additionally, because social media messages reflect unprompted musings, they may offer more accurate insights into users’ thought processes than would be available in a traditional focus group setting (e.g., Wilkinson, 1998). Some criticism of focus group research evaluating the effectiveness of television advertising and/or programming has centered on the fact that focus groups are conducted in unnatural laboratory settings wherein participants are given little or no choice regarding the segments they view (Goldman & Glantz, 1998). Since Twitter communications (called “tweets”) are limited to 140 characters, each tweet generally reflects a single idea or thought. The impromptu reactions of Twitter users may represent qualitative feedback akin to focus group responses, but generated in a natural setting uncompromised by the artificial environment and deliberate, targeted exposure inherent in focus group evaluations.

Rapidly changing trends in media consumption make Twitter a particularly valuable resource for evaluating viewer responses to television programming and advertising campaigns. Multiple screening across media platforms is increasingly common: according to a 2012 report from Ericsson ConsumerLab, 62% of people worldwide reported using social networking sites and forums while watching television (Ericsson ConsumerLab, 2012). In June 2012 one in three Twitter users reported that they had posted tweets about the content of television programs while viewing, an increase of 27% from only five months prior (Bauder, 2012). To put this behavior in the context of population and frequency, in a three-month period in 2013, 19 million unique authors composed 263 million tweets about live television programming (Nielsen Media Research, 2013). Thus an analysis of tweets regarding specific televised messages is likely to provide a practical and meaningful tool for evaluating reactions to messaging that have traditionally been measured via focus groups.

The Persistent Debate about Fear’s Role in Persuasion

Persuasive messages containing fear appeals are intended to influence audiences toward desirable outcomes. But these messages can also yield two undesirable outcomes: no effects at all (Byrne & Hart, 2009; Witte, 1992)—studies that are difficult to report given the results-oriented bias in scholarly publications—or an increase in the undesirable behavior, such as higher rates of smoking intention (Wolburg, 2006). However, the puzzling findings on the persuasive effects of fear fit into a coherent theory under Witte’s Extended Parallel Process Model (EPPM) (Witte, 1992). The EPPM explains that strategic messages must surpass a lower threshold of fear appeals in order for individuals to notice them; otherwise they are ignored. But the fear-persuasion relationship is not linear, since eliciting too much fear may have a boomerang effect (Byrne & Hart, 2009). This upper threshold is more complex and involves the combination of efficacy in the message (response efficacy) and efficacy inherent in the individual (self-efficacy). Accordingly, there is a critical point at which high levels of efficacy turn that soaring fear response into danger control: a rational, protective motivation response toward successfully accepting the message. However, if efficacy is low, individuals react to elevated levels of fear with fear-control responses—rejecting the message (Witte, 1992). Analyses based on tests of the EPPM conclude that acceptance or rejection of fear-inducing messages depend upon the interplay between the level of fear elicited, levels of efficacy (either response- or self-efficacy), and contextual factors (Witte & Allen, 2000; Witte, 1992, 1998).

According to the EPPM, the fear dosage is critical because its success depends on the specific combination of efficacy levels in the message and within individuals, which is unknown a priori. Consequently, messages frequently go awry. A strong body of evidence suggests that when individuals are confronted with a message that evokes fear, they will disregard or reject the message (Byrne & Hart, 2009; Henriksen, Dauphinee, Wang, & Fortmann, 2006; Witte, 1992, 1994; Wolburg, 2006). Evidence from anti-drug campaigns also suggests that public service announcements that aim to deter unhealthy behaviors have been associated with a boomerang effect (for a review, see Byrne and Hart, 2009; Hornik, Jacobsohn, Orwin, Piesse, & Kalton, 2008), or normalization of that behavior (Terry-McElrath, Emery, Szczypka, & Johnston, 2011; Werb et al., 2011).

Nevertheless, findings from a fear appeals meta-analysis suggest that any message with a high level of fear appeals—regardless of the level of efficacy (response- or self-efficacy)—produces larger desirable effects than any message with a low level of fear appeals (Witte & Allen, 2000). The Tips campaign contained high levels of fear appeals, represented by graphic descriptions of health effects such as cancer, facial damage, stoma, amputation, and hair loss. Therefore, we hypothesize that there will be more Twitter messages indicating high perceived threat compared to messages describing low perceived threat among tweets related to the Tips campaign (H1).

Similarly, Witte and Allen (2000) contend that well-constructed fear appeal messages accompanied with (at least) some response efficacy produce (at worst) weak positive effects on attitudes, intentions, and behaviors. Hence message acceptance appears to be more likely than message rejection or message disregard when efficacy is present. In the case of Tips, the messages in the ads provided viewers with information about resources to help them quit—the 1-800-Quit-Now help line and/or the smokefree.gov website—which is what Witte and colleagues would describe as message efficacy. In this way the CDC aimed to empower viewers in addition to evoking a fear response with the Tips campaign’s graphic images. As a result, we also hypothesize that among tweets indicating high perceived threat, there will be more message acceptance than message rejection (H2).

Methods and Measures

Our approach to collecting and assessing Tips-related tweets from the corpus of Twitter messages during the period while Tips advertisements were being broadcast in the US represents an innovative strategy for applying methodological rigor to the analysis of big data.

Data Collection

Data were obtained from a vendor (Gnip, Inc.; http://www.gnip.com) licensed to provide access to the entire corpus of Twitter data, using a data streaming process referred to as the “Firehose.” Unlike accessing the publicly available data via the Twitter streaming API (Twitter’s Application Program Interface), which samples approximately 1% of Twitter content (https://dev.twitter.com/docs/streaming-apis), the Firehose provides real-time access to 100% of all tweets and metadata. Potentially relevant tweets were filtered from the Firehose using a broad set of content-specific keywords, following methods proposed by Stryker and colleagues (Stryker, Wray, Hornik, & Yanovitzky, 2006).

Keyword Selection

To select relevant keywords for this study, a team of six researchers previewed the Tips advertisements prior to campaign launch. A comprehensive keyword list was generated based on ad content, likely tobacco-related behavior and policy topics, and expert knowledge, in consultation with the CDC (the Tips campaign sponsor). The keyword list was designed to be as comprehensive as possible in order to capture the entirety of Tips-related tweets, encompassing non-standard English usages, slang terms, and misspellings. Keywords were screened across three conversational areas related to tobacco use, policy and marketing. The first and broadest search contained keywords associated with tobacco-related behavior such as smoking, tobacco, and cigarette. A second search focused on conversations around tobacco control policy using keywords such as tobaccofree, smokefree, and quitnow (Tobacco Policy). Finally, the third search focused on keywords describing specific content features appearing in each of the televised CDC Tips ads such as CDC Tips, hole in throat, and amputee (Ad Specific). In all cases, words were searched for their logical variants and misspellings. Please see list of keywords in the Appendix. During the first two weeks of the Tips campaign, the volume and content of incoming tweets yielded by the broad keyword filters were actively monitored to identify potential related keywords that could detect additional campaign-relevant tweets. No additional terms were identified during this review. Tweets were collected into the three archives during the course of the campaign beginning March 15 through June 9, 2012.

Metadata

Metadata represent ancillary information—data about the data—that is embedded in each tweet, and included in the corpus of data collected via the Twitter Firehose. These metadata include a tweet ID (a unique numerical identifier assigned to each tweet), the username and biographical profile of the account used to post the tweet, geolocation (if enabled by the user), number of followers of the posting account, the number of accounts the posting account follows, the posting account’s Klout score (a measure of social media influence), as well as any hashtags, URL links, and media content attached to the tweet. These data were used to describe the concentration and potential diffusion of Twitter messages related to the Tips campaign.

Filtering for engagement with Tips

Researchers postulated that, after watching a Tips ad, a Twitter user would engage the CDC Tips campaign by tweeting using words to describe a televised commercial. To collect engagement tweets, each tweet in the three archives (Tobacco Behavior, Tobacco Control, Ad Specific) was filtered for keywords (and variants) that described televised commercials (ad, commercial, campaign). Because multiple keywords might occur in a single tweet, in some instances the data collected via the Firehose placed multiple copies of the same tweet into the archive. De-duplication, using the unique identifier from the metadata, assured that the archive contained only one occurrence of each tweet.

Human Coding

At several stages in the data collection, human coders were used to assess relevance and code message content. These coders received training that consisted of viewing all ads that were part of the Tips campaign, then meeting as a group to review, discuss, and refine coding criteria. Questions about and discrepancies with the coding criteria were resolved by consensus.

Precision/relevance

The Tips engagement tweets were assessed for relevancy, using a combination of human coding and machine classification to eliminate false positives from the collected data (Mitchell, 1997). A group of six trained coders were paired into three teams. Each team classified one-third of a random sample (1,350 tweets; 450 tweets per coding team) of the engagement tweets for relevance to the Tips campaign (relevant vs. non-relevant). Inter-coder reliability was calculated using a Kappa score averaged across the three teams, and was found to be acceptably high (K = 0.93) (Landis & Koch, 1977). The human-coded tweets were then used to train a naïve Bayes classifier to automatically classify the larger dataset of Tips engagement tweets for relevance. Precision was calculated as the percent of Tips-relevant tweets yielded by the keyword filters.

Recall

To assess whether the retrieved tweets were representative of, and generalizable to, all of the Twitter content that might be relevant to the Tips campaign, a random sample of approximately 13,000 Tweets that were not retrieved by the engagement filters was pulled from the archives of smoking behavior, policy, and ad-specific tweets collected during the period of March 15-June 9, 2012. Seven teams of trained human coders each reviewed a set of over 3,600 unretrieved tweets to identify Tips-relevant content. Inter-coder reliability was acceptable (K=0.65¹). Recall was calculated as the fraction of Tips-relevant tweets that were correctly retrieved by the keyword filters during the approximately three-month collection period. This fraction was weighted appropriately to account for the disproportionate sampling of retrieved and unretrieved tweets to identify Tips-relevant content.

Content Coding

The body of Tips-relevant tweets was further coded for fear appeals (message acceptance, message rejection, or disregard). Both message acceptance and message rejection involve an appraisal of high perceived threat, while disregard represents an appraisal of low perceived threat. Fear appeals codes were based on the EPPM model (Witte & Allen, 2000; Witte, 1992, 1994, 1998). Message acceptance was defined as tweets reflecting evidence that the viewer experienced fear and/or perceived a threat to their own or to a smoker’s health after viewing a Tips ad. For example, a tweet stating, “That smoking commercial with the lady with the hole in her neck scares the living F*@# out of me,” would be coded as fear message acceptance. Fear message rejection was defined as content reflecting doubt about the threat message, the viewer’s inability to deal with the fear, or the viewer being overwhelmed by fear. In such cases, the individual could use denial (e.g. “That smokefree commercial is bullsh*%! My grandma has been smoking since she was ten and she doesn’t have a hole in her da@# neck”); defensive avoidance (e.g. “Ughhhhhh that smoking commercial is on! Turn immediately”); or reactance (e.g. “I cant help but light a cigarette afta i see tha smokers commercial”) to cope with fear, and such tweets would be coded as message rejection. Finally, low-threat or disregard messages were defined as those without emotional content or any personal commentary related to fear. For example, tweets such as “CDC: Tips from Former Smokers - Terrie’s Ad: http://t.co/zOa0yD2T graphic antismoking ad” reflect minimal or no threat appraisal.

A standardized code set was constructed, and eight human coders were paired into four coding teams to classify a random sample (1,400) of the Tips-relevant tweets for message acceptance, rejection, or disregard. Inter-coder reliability for content coding was acceptable (K=0.75). This code set was then used to train the naïve Bayes classifier to machine classify the corpus of Tips-relevant tweets.

Results

During the period in which the CDC’s first Tips From Former Smokers campaign was broadcast across the 210 media markets in the US (March 15 through June 9, 2012), 37 keywords and their variants (e.g. plurals, misspellings) representing three broad tobacco constructs and engagement filters yielded nearly 17 million tweets from the Twitter Firehose. The Tobacco Behavior archive yielded the most tweets (approximately 16.9 million) while the Tobacco Control and Ad Specific archives contained considerably fewer at 77,000 and 54,000 tweets respectively. After de-duplication, there were 245,319 unique tweets that were potentially relevant to the CDC’s Tips From Former Smokers televised advertising campaign.

Precision

A trained naïve Bayes classifier determined that 193,491 (79%) of tweets pulled from the Firehose using our keyword filters were identified as Tips-relevant. Thus the corpus of relevant tweets came to be 193,491 and the precision of the keyword filters was 79%.

Recall

Among the random sample of approximately 13,000 unretrieved tweets, human coders found 16 tweets to be relevant to Tips. Adjusting for the sampling fraction (0.14%) of non-retrieved tweets, recall was calculated at 94%.

Message acceptance, rejection, and disregard

Approximately 87% (167,867) of the Tips-relevant tweets were classified as message acceptance, 7% (14,281) as message rejection, and 6% (11,521) as message disregard. Thus a majority of Twitter messages related to Tips displayed high perceived threat (94%) as opposed to low perceived threat or disregard (H1). Moreover, the results also indicate that there was more message acceptance than message rejection among high-perceived threat tweets (H2). A list of examples of tweets representing each of the three fear response codes is included in Table 1.

Table 1. Examples of Tweets Representing Each Fear Message Code.

Fear Type	Tweet Text
Message Acceptance	Most effective antismoking ad ever is the stroked-out mom who has to get sponge baths from her adult son.works on a multigenera
	Just saw the scariest tobacco commercial…so thankful I stopped that shit finally #feelinghealthieralready
	Whoever decided there needed to be an antismoking commercial involving stomas deserves to be shot.
	That antismoking commercial actually makes me wanna quit
	That antismoking commercial with people having holes in their neck is fucking nasty
	I swear these cigarette ad commercials are scary as hell! #toxic
	That commercial about smoking with the amputees is enough for me to never wanna see a cigarette again
	new anti-smoking commercial showing people that became amputees from cigarette smoking is crazy. i like my legs. use them both
	RT @xAdoreTIFFANY: Tht Cigarette Commercial Where The Lady Gotta Hole In Her Throat Is Scary AF
	That new cigarette commercial fucks wit me
Message Rejection	As a life long nonsmoker I don’t feel I should have to be subjected to the Crypt Keeper-esk gal in the antismoking commercial.
	The fucked up thing is…whenever i see that lady from the anti smoking ad with a hole in her throat…it just makes me want a cigarette
	Do not waste my tax dollars on #AntiSmoking ads. How about an ad on irresponsible government policies and damage it does to our freedom?
	@WadddupAshley =O y? i want us to be on a “you don’t always die from tobacco” commercial one day
	Watching the commercial with the ppl that have a hole in they throat from smoking cigarettes, only makes me want to go smoke a cigarette.SMH
	Yo, this commercial with the cigarette smokers with holes in their throat can bleaux meh.
	Crycrycry RT @SCAFFBEEZY: This new non smoker commercial with the bald lady and hole in her throat urkss me!! She’s dumb
	I thought it was robots talking but then it was a smoking commercial
	RT @kushbarbiee: tht smoking commercial < iCant
	If I was that lady in the cig commercial. I would hook a bong to my neck hole!!
Message Disregard	The CDC’s New Anti-Smoking Ad Campaign: What Do You Think? – Stop Smoking Center – Everyday Health http://5.co.JgBhYP5D
	C.D.C. Finances Nationwide Antismoking Ad Campaign. (Wish the article would have given credit to the ad agency.) http://t.co/eT
	US Backs Antismoking Ad Campaign http://t.co/RPwCKylh
Message Disregard	C.D.C. Finances Nationwide Antismoking Ad Campaign, a First: http://t.co/nLeE4yy8
	Yahoo CDC Launches New Graphic Antismoking Ad Campaign: Washington, D.C. -- This week CDC launched…http://t.co/CANGJIHc
	CDC: Tips from Former Smokers – Terrie’s Ad: http://t.co/zOa0yD2T graphic antismoking ad #cancersticks
	US for the first time backs antismoking ad campaign: http://t.co/rCUTFmOK
	Anti-Smokin Campaign gd Public Policy or Heavv Handd Propaganda? http://t.co/Ptdg3JTO #argument #cigarette #propaganda #thinkin
	CDC – Tips from Former Smokers – Buerger’s Disease Ad http://t.co/qlsu0WoJ via @LYBIO
	CDC ad campaign reveals harsh reality of smoking-related diseases http://t.co/EdLxfSFY

Concentration and potential diffusion of messages

Analysis of the metadata revealed that 166,857 unique users generated the 193,491 Tips-related tweets. Our results, however, are based on population—not sample—data (the corpus of 193,491 tweets). Therefore the assumption of independence among units (which would be violated here) does not pertain, especially given that our unit of analysis is the tweet rather than the individual—and individuals may have different reactions to tweets depending on specific ad, context, or mood. In total these tweets represented at least 39 million potential impressions (defined as tweets delivered to users’ timelines). This definition of impressions is a standard measure used by Twitter (http://help.tweetreach.com/entries/276589-What-do-you-mean-by-reach-exposure-and-impressions-).

Discussion

This study provides strong evidence that the controversial “Tips from Former Smokers” campaign was neither rejected nor dismissed by viewers despite its use of graphic and emotionally evocative imagery and themes. A corpus of 193,491Tips-relevant tweets was collected during the course of the 2012 campaign, and the vast majority of these tweets reflected message acceptance. The Tips campaign has been validated via traditional survey-based research methodology to be effective at increasing population-level smoking cessation, with estimates of a relative 12% increase in quit attempts and nearly half a million quality-adjusted life-years potentially added to the US population (McAfee et al., 2013). Our qualitative analysis adds to that evidence by offering insight into the intermediate thought processes by which the audience interpreted and reacted to the ads in a natural setting. As postulated by health behavior theory, an affective response (such as fear) to an ad influences the viewer’s attitudes and beliefs, which in turn may influence the associated behavior (Batra & Ray, 1986; Fishbein & Ajzen, 1975). Our results offer support for the campaign’s hard-hitting graphic messaging approach. This qualitative evaluation of how people naturalistically processed and reacted to Tips messages provides ammunition for the argument that the fear-based campaign may in fact have been better received than would other types of messaging strategies. Given the controversy surrounding such graphic imagery (Gass & Seiter, 2011), this contribution is substantial.

Although this study is not a direct test of the EPPM, Witte’s model (1992) offers a theoretical framework guiding our analyses. Rather than offering evidence from a controlled—but sometimes sterile—environment, our study used a real-life, natural setting to gather actual responses to a mass media campaign. Thus our findings shed light on campaign reactions for the specific subpopulation of Twitter users. Study results offer support for our hypotheses that more Twitter messages relevant to the Tips campaign demonstrate high perceived threat compared to messages displaying low perceived threat (H1); and that Tips-relevant Twitter messages indicate higher levels of message acceptance than of message rejection (H2).

Our study employed a novel research strategy designed to take advantage of the rapid diffusion of new media platforms and the spontaneous, unfiltered audience reactions those platforms allow and encourage. As the “world’s largest focus group” (St. Amand, 2013), Twitter provides an effective source from which to gather quantities of information about a very focused topic. Our methods for selecting and testing the precision and recall of keywords, filtering for relevance, and generating a clean dataset to code for content represent important steps toward helping build standards for research rigor when analyzing sets of “big data” such as those gathered from the Twitter Firehose. With this keyword selection and evaluation process, we were able to achieve a practical balance of precision and recall. The calculation of precision is an important methodological step for maximizing the external validity of the search terms (Stryker, Wray, Hornik, & Yonovitzky, 2006), as well as the internal validity of the analyses. The calculation and reporting of recall also provides strong evidence of the external validity of this research.

Our research has some limitations. First, because Twitter is not widely diffused among the US adult population, analyzing Twitter messages provides limited generalizability to the US population as a whole. The proportion of Internet users who reported using the Twitter platform more than doubled between November 2010 and May 2013, at which time the percentage stood at 18% (Brenner & Smith, 2013). Further, although the Firehose enables collection of metadata attached to each tweet, those metadata do not allow full demographic characterization of the individuals generating the tweets. Thus it is impossible to precisely report the demographic characteristics associated with users who tweeted about the Tips campaign. Consequently, we cannot describe the similarities or differences between those who tweeted about the Tips campaign and others who also saw the ads but did not tweet about them. Further, there is no method to determine whether the messages were generated by smokers, former smokers, or individuals who have never contemplated smoking.

It should also be noted that, although the estimated number of potential impressions made by the Tips conversation on Twitter was high (39 million), it is highly unlikely that every follower viewed every tweet. This number must be considered within the context of common Twitter behavior. Further, since Twitter metadata do not provide follower IDs, there is no method for determining the number of unique followers who received an impression (reach).

As a result of the absence of demographic information or smoking status among individual Twitter users, the external validity of inferences drawn from Twitter data depends in part upon available information about the characteristics of Twitter users at the population level. The Pew Internet & American Life Project describes Internet users who use Twitter as predominantly young (30% aged 18-29), African American (27%) or Latino (28%); they also tend to be urban dwellers and middle- to higher-income. Twitter use is comparable across all levels of education (Brenner & Smith, 2013).

While Twitter users are not necessarily representative of the US population, Twitter is used disproportionately by members of communities typically underrepresented in traditional research settings (i.e. young adults and African Americans). Furthermore, young adults have higher smoking prevalence than the general population, and African Americans suffer disproportionately from tobacco-related diseases (Dietz, Sly, Lee, Arheart, & McClure, 2013; Peters et al., 2012; Stingone, Funkhouser, Weissler, Bell, & Olshan, 2013). Hence, Twitter may give us an ear into conversations that might not be captured using traditional research methodology.

The research reported here took place without any direct participant recruitment, contact, or behavioral observations. As a result, analysis of Twitter messages is not ideal for examining outcomes, as it does not offer a measure of actual behavior (i.e., quitting smoking). However, according to the Health Belief Model (Becker, 1974) and the Theory of Planned Behavior (Ajzen, 1985), behavior is preceded by cognition and attitudes. Thus messages that are dismissed prior to being cognitively processed will have little or no impact on behavior (Ajzen, 1985; Becker, 1974; Fishbein & Ajzen, 1975). Studying audience reactions to a media intervention using big data such as Twitter messages, then, offers a process evaluation demonstrating potential attitudes and intentions, which are good predictors of future behaviors.

A further consideration is that, while our archives of data gathered from the Twitter Firehose likely represent the entirety of Tips-relevant tweets, they do not represent the entire Twittersphere. Our calculations of precision and recall justify the conclusion that the analyses were performed on the census of tweets related to the Tips campaign, and those analyses were approached from within a theoretical framework. Thus, the study’s limitations are outweighed—at least in part—by the methodological rigor with which it was conducted.

In sum, our research provides strong evidence that Twitter reactions to Tips gave the campaign life beyond the ads; each of the nearly 200,000 tweets about the campaign created a ripple effect that extended the ads’ reach. Beyond that, our study showed that the graphic emotional approach employed by Tips had the desired result of jolting the audience into a thought process that might have some impact on future behavior. Nearly everyone is aware that smoking is unhealthy (Slovic, 2001), which poses a challenge for tobacco control media campaigns to present old information in ways that capture attention from the audience. Our analysis of Twitter data leads us to believe that Tips succeeded in doing so.

To date, research on big data gathered from Twitter is so new that the emerging literature presents no standardized methodology, making direct comparison between studies difficult. This study describes a rigorous approach to collecting and cleaning Twitter data, reporting internal and external validity of the data collection process, and analyzing massive amounts of qualitative data. As such, this research represents an important step toward establishing practical and rigorous methodological standards for using “big data” in social science research.

Appendix: List of Keywords for CDC Tips Campaign

Tobacco Behavior	Cig(s)
	Cigarette(s)
	Nicotine
	Smoke(s)
	Smoker
	Smoking
	Tobacco
Tobacco Policy	@cdcgov
	@cdctobaccofree
	@drfriedencdc
	@fdatobacco
	@smokefreegov
	antitobacco
	antismoking
	CDC
	quitline
	quitnow
	secondhand + smoke
	smokefree
	smokefree.gov
	tobaccofree
Ad Specific	#cdctips
	amputation
	amputee
	Buerger’s+Disease
	heart+attack
	hole+neck
	hole+throat
	lung+cancer
	stoma
	stroke
	throat+cancer
	Tips + Former + Smokers
Engagement	ad
	commercial
	campaign
	PSA