Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Jul 1.
Published in final edited form as: J Comput Mediat Commun. 2014 Jan 10;19(4):975–990. doi: 10.1111/jcc4.12057

Effects of Online Comments on Smokers’ Perception of Anti-Smoking Public Service Announcements

Rui Shi 1, Paul Messaris 2, Joseph N Cappella 3
PMCID: PMC4281938  NIHMSID: NIHMS643712  PMID: 25561825

Abstract

On YouTube anti-smoking PSAs are widely viewed and uploaded; they also receive extensive commentary by viewers. This study examined whether such evaluative comments with or without uncivil expressions influence evaluations by subsequent viewers. Results showed PSAs with positive (i.e. anti-smoking) comments were perceived by smokers as more effective than PSAs with negative (pro smoking) comments. Smokers in the no comment condition gave the highest perceived effectiveness score to PSAs. Smokers’ readiness to quit smoking moderated the effect of comments on PSA evaluation. Smokers reading negative uncivil comments reported more negative attitude toward quitting and a lower level of perceived risk of smoking than those reading negative civil comments but positive civil and positive uncivil comments didn't elicit different responses.

Keywords: Anti-Smoking PSA, Recommendation Systems, Social Influence

Information in the age of the Internet no longer flows one way from media to audience. Audiences create media as well as comment on it. Social media such as Facebook have existed for only seven years and various news websites allow comments in response to virtually every news story. Amazon not only allows potential buyers to read previous buyers’ reviews of the product, but also recommends books that they might find interesting based on what “people like you” have bought. These are all examples of online recommendation systems, which include various interactive features allowing individuals to express and share their opinions. It is important to understand the online recommendation systems because whatever media product it accompanies - a piece of news, a music video, a commercial product, or an anti-smoking public service announcement (PSA) - the information carried by the recommendation system synchronizes with the media product and thus becomes a part of the message sent to the audience.

Among various forms of online recommendation system, commentary is of particular interest because of the amount of information it carries. While rating scores only describe the valence of the social commentary, comments can simultaneously tell viewers how good or bad other people feel about something and why they feel this way. Incivility, or sometimes referred to as “flaming”, has been a problem common to online comments and reviews. The use of uncivil language has been found to be more prevalent and obvious in computer mediated communication than in face to face situations (Orenga, Zornoza, Prieto, & Peiro, 2000), and it has been argued that people are less sensitive to other people’s feelings when they are virtual and anonymous (Moor, Heuvelman & Verleur, 2010).

This study explores one form of online recommendation system, i.e. commentary, and its potential impact on audiences’ responses to the core message that is the object of the commentary. The objects selected here are health messages, specifically anti-smoking ads.

Impact of Online Recommendation System

Recommendation systems can affect people by creating the sense that a social norm - or at least a social consensus -- exists and in the process pressing for conformity. The perceived social norm can lead people to match the opinion or behavior of others because people have the internal desire to be accurate, to fit in, and to maintain a positive self-concept (Cialdini & Goldstein, 2004; Petty & Cacioppo, 1986).

In a study investigating why experts usually failed to predict the market performance of cultural products such as songs, books and movies, researchers found it was the social influence of recommendation system that made success of the products unpredictable (Salganik, Dodds & Watts, 2006). In this online experiment, subjects were given the chance to download unknown songs from unknown bands for free. Those in the control condition were only given a list of the song names while those in social influence condition were given the name of each song as well as how many times each song had been downloaded by previous listeners. It was found that the best songs usually ranked high and the worst songs usually ranked low on the number of downloads but the rest of the songs fluctuated from the top to the bottom. Unless a song is of superb quality, its success depends on the recommendations it received at the time it enters the market. A second experiment kept the same design but tried to strengthen people’s perception of the social norm by arranging songs in rank order in addition to presenting the number of downloads. The effect of social information was found to be larger than in the first experiment thus making prediction of a song’s success even less likely.

Not only are people influenced by the online recommendation systems when they don't have clear preferences but, even when they have explicitly expressed their preferences, they sometimes reverse their choices after knowing what other people prefer (Zhu, Huberman, & Luon, 2011). If people’s evaluation and preference can be swayed by rating scores or ranking information left by previous users, it would not be surprising that comments left by previous audience could also create a norm influencing subsequent audience preferences. Walther, DeAndrea, Kim, and Anthony (2010) showed that anti-marijuana PSAs on YouTube were evaluated by college students as more effective when accompanied by positive (that is anti- drug and pro-ad) comments compared with the same PSAs accompanied by negative (that is pro drug and anti-ad) comments. The effect of comments on PSAs’ perceived effectiveness was particularly strong for those who identified with the prior commenters.

The amount of research on the influence of online comments is limited since they are a relatively new phenomenon. Comments are different from rating scores because they describe both valence of the norm and the arguments and rationales supporting the norm too. They can carry valenced information in a way that is substantive or not with uncivil comments far from unusual. Walther and colleagues (2010) directly addressed the effects of online comments, but only in a relative sense: they assessed the PSA in a positive-comment condition relative to that in the negative-comment condition. The results indicate that positive comments reinforced the PSA better than negative comments, but without a control group it is impossible to know whether comments of either valence actually improve or harm the effectiveness of the PSA than when it stands alone. The current study aims to address this question by including a control group. In addition to comment valence, Walther et al. (2010) manipulated message intensity and profanity in the original design but this factor yielded no effects. Since their population was of college age, the absence of effects from incivility may not generalize to a a more representative population. The Walther et al. study is a significant step in understanding the impact of recommendation systems. We will add to their work by examining a different content domain (tobacco control), employing a representative sample of smokers, including a no comment control to assess the impact of commentary at all, and distinguishing factorially positive and negative substantive comments from civil and uncivil ones.

Uncivil language is common on the internet. Cautions have been raised on the uncivil atmosphere surrounding political discussion on social network sites (Kushin and Kitchener, 2009) as well as on the prevalence of profanity among online health service sites and forums from which women seek help (Finn & Banach, 2000). A survey of YouTube users finds flaming to be common among commenters who used offensive language because they perceive flaming as normative (Moor, Heuvelman & Verleur, 2010).The impact of uncivil postings on the internet has been explored but findings have been inconclusive. People report they sometimes refrain from uploading videos to avoid flaming comments (Moor, Heuvelman & Verleur, 2010). In an experimental test in the political arena, participants’ intention to participate in discussion was not influenced by the impolite language involved. Discussants in the uncivil condition were perceived as more dominant and less credible, but incivility did not decrease the persuasiveness of the content (Ng & Detenber, 2005).

Social Influence and Conformity

Although the research on recommendation systems is relatively new and limited in its breadth and depth, other research that predates recommendation systems but focuses on social influence under certain and uncertain conditions is relevant to the possible effects of recommendations on outcomes such as attitudes, beliefs, intentions and behaviors. These studies have potential implications for online recommendation systems with civil and uncivil commentary.

If online comments are an indicator of social norms, then research on conformity effects as one form of social influence may help us to understand the potential power of online commentary. The autokinetic experiments conducted by Muzafer Sherif (1935, 1936) were among the earliest investigations into people’s compliance with social norms and associated perceptions. When tested individually subjects reported a large range of lengths the dot of light moved, but when they were then put into a group of two or three and asked to make the estimation out loud, their estimates converged without any discussion. This study shows people’s perceptions of the physical world can be shaped by others’ expressed opinions. The Asch paradigm examined conformity effects under a more extreme circumstance, i.e. a minority one against a unanimous majority (e.g. Asch, 1956) demonstrating people’s tendency to conform to the majority even when stimuli were clear and majority opinion erroneous.

Social influence can also shape people’s attitudes and beliefs. Personal attitudes can be based on perceived public opinion (Sonck & Loosveld, 2010). In both laboratory and field experiments, attitudes were found to be swayed by reports of public opinion polls, especially when initial attitudes are neutral or when the issues of little personal relevance (Giner-Sorolila & Chaiken, 1997; Kang, 1998). Social norms can function as heuristic cues (as would be suggested by the ELM by Petty, Cacioppo, 1986) to direct belief change when people are not highly motivated to process the information presented.

Although the general research on profanity and incivility in traditional settings is extensive, general conclusions are difficult to infer as contexts play a significant role in the direction of effects. On the one hand, incivility has been shown to decrease trust (Mutz and Reeves, 2005), credibility (Bostrom, Baseheart & Rossiter, 1973), liking (Mutz, 2007), perceived socio-intellectual status and aesthetic quality (Mulac, 1976) of the speaker. These factors contribute to a communicator’s persuasiveness (e.g. Eagly & Chaiken, 1975). On the other hand, incivility increases people’s perception of the speakers’ intensity (i.e. dynamism -- how passionate, strong, and enthusiastic the speaker is) and thus increase their persuasiveness (Scherer & Sagarin, 2006).

The relationship between incivility and persuasion becomes more complicated when taking into account people’s pre-existing attitudes. In some cases people tend to forgive uncivil comments if they agree with the content, but they punish the speaker for incivility if the speaker is on the opposing side (Mutz, 2007). In other cases people reward uncivil speakers in pro-attitudinal condition but do not differentiate civil and uncivil speakers in counter-attitudinal conditions (Scherer, 2007).

These previous studies on social influence and incivility in traditional settings help explain the possible underlying mechanisms of possible effects and thus postulate potential impact of online recommendation systems.

Effectiveness of Anti-Smoking PSAs

Many health messages are posted on YouTube including anti-smoking PSAs. There are various motives for people or organizations to upload the PSAs online, sometimes to attempt to reach out, sometimes to poke fun or ridicule. Regardless of the motive, those PSAs are there and their presence is almost a cottage industry.

A great deal of research has been directed at the elements that help to create effective PSAs especially in the tobacco control arena. Researchers have evaluated the role of argument strength (Strasser et al., 2009), content themes (Sutfin, Szykman, & Moore, 2008), and various format features like smoking cues (Kang, Cappella, Strasser, & Lerman, 2009), and sensation value (Strasser et al., 2009).

However, less emphasis has been given to role of the social context within which PSAs operate and the impact of contextual factors on overall ad effectiveness. Durkin and Wakefield (2006) for example found that daily smokers who discussed the anti-smoking PSAs after watching them were more motivated to quit. More relevant to the current study is an experiment that compared the effects of an anti-smoking PSA on college students when it was accompanied by discussion of different sources. Findings showed that the PSAs followed by friends’ discussion about the harm of smoking had a larger impact on students’ attitude toward smoking compared with the PSAs followed by strangers’ discussion (Samu & Bhatnagar, 2008). The result further illustrates that social influence has the potential to alter - positively or negatively -- the effectiveness of anti-smoking PSAs.

Hypotheses and Research Question

The current study was built in a YouTube-like visual context with anti-smoking PSAs as the media product under investigation. On YouTube, anti-smoking PSAs are widely viewed and uploaded; they also receive extensive commentary by viewers. Viewers of these health messages are also exposed to the comments accompanying them. This study examined whether such evaluative comments influence judgments about the PSAs when viewed subsequently by others.

Based on previous studies on online recommendation system and social influence it was hypothesized that:

Ht. Comment valence influences smoker’s perceived effectiveness (PE) of the PSA such that positive (i.e. pro-ad or anti-smoking) comments make the PSA more persuasive and negative ((i.e. anti-ad or pro-smoking) comments make the PSA less persuasive.

People’s tendency to conform to a perceived norm has been found to be particularly strong under conditions of uncertainty or self-doubt (Tesser, Campbell, & Mickler, 1983; Wooten & Reed, 1998). In this study two uncertain scenarios were considered: a) when people were unsure of the PSA, i.e. when the ad was weak or ambiguous; and b) when people were unsure of themselves or their identity as smokers, i.e. when they were more or less ready to quit smoking. Thus it was further hypothesized:

H2. PSA quality moderates the influence of comments on people’s PE of the PSA such that the effect of comments will be particularly strong when people are unsure of the PSA.

H3. Smokers’ readiness to quit smoking moderates the influence of comments on their PE of the PSA such that the effect of comments will be particularly strong when smokers are more (versus less) ready to quit smoking.

Since findings from the previous studies on incivility are mixed and lead to conflicting predictions we only raise a research question:

RQ. How will uncivil comments influence people’s PE of the PSA, attitudes, perceived risk, and intention to quit smoking?

Method

Participants and Study Design

A total of 592 adult smokers were recruited through Knowledge Networks, a survey research company which has developed a representative panel of adults in the United States. To be eligible for the study, a subject must be a regular smoker who (a) smokes cigarettes currently, (b) smokes an average of five or more cigarettes a day in the past week, and (c) smoked more than 100 cigarettes in their lifetime. Panelists who failed to meet all three criteria were thanked and excluded from the study sample.

The final working sample has a mean age of 49.47 years old. Most of the participants had finished high school (93.8%), approximately half (50.8%) were males, 82.1% were White, 7.1% were African-American, 5.6% were Hispanic, 2.7% mixed, and 2.5% marked “"other". Subjects have smoked 33.83 years on average (SD = 12.91), and they have previously quit smoking 4.62 times (SD = 8.45) on purpose for more than one complete day. Fagerstrom Test for Nicotine Dependence (FTND) was used to measure participants’ addiction level (Heatherton, Kozlowski, Frecker, & Fagerstrom, 1991). Participants for the current study scored 4.17 on average (SD = 2.21) which indicates moderate nicotine dependence.

This study adopted a 2 comment valence (positive vs. negative) × 2 comment tone (civil vs. uncivil) × 2 PSA quality (strong vs. weak) mixed experiment design. Comment valence and comment tone were manipulated between subjects and PSA quality within subjects. There were two control conditions as well, one with no comment and the other with mixed comments.

Anti-Smoking PSAs and Comments

Four strong and four weak PSAs were selected from a pool of 99 previously rated anti-smoking PSAs. Strong PSAs were those rated as top ten in perceived effectiveness by a separate sample of adult smokers; weak PSAs were those rated in the bottom ten.

Every PSA had ten comments attached to it. Comments were defined as positive if they supported the PSA or were against smoking; comments were defined as negative if they criticized the PSA or supported smoking. Comments are categorized as uncivil if they a) include swear words (e.g. fuck; shit; damn, etc.); or b) involve personal attack (e.g. smokers should all die); or c) are insulting (e.g. stupid, retarded); or d) use extremist references (e.g. Nazi, Hitler).

To avoid case-category confound (Jackson, 1992), a pool of 292 comments was created. These were evenly distributed across four comment treatment conditions, which means a quarter of the comments in the pool were positive civil (PC), a quarter positive uncivil (PU) a quarter negative civil (NC), and the last quarter negative uncivil (NU). All the comments were selected from YouTube comment page and some were modified slightly to fit the treatment conditions.

Procedure

When entering the online survey participants were told the purpose of the study was to “test a new website where people can share health related video clips, public service announcements and advertisements.” They answered questions about their smoking history and current level of readiness to quit smoking before the PSA viewing. Subjects were randomly assigned to one of the six comment conditions (PC, PU, NC, NU, Control - mixed comments, and Control - no comment). One strong PSA and one weak PSA were randomly selected out of the eight PSAs for each participant and the selected two PSAs were shown in random order.

For treatment conditions, ten comments were randomly generated from the comment pool for each PSA according to the condition. It is not common to see 10 comments in a row on YouTube with the same valence and tone. To increase the ecological validity of the study, comments were selected in the ratio of 8/2 for valence and 8/2 for tone for the treatment groups. When combined, every Valence × Tone condition keeps a ratio of 7:1:1:1. So for example a PSA in the PU condition would be accompanied by 7 PU comments, 1 PC, 1 NC and 1NU yielding 8 positive and 2 negative comments; 8 uncivil and 2 civil comments. For the mixed comments control group, comments were selected in a ratio of 5/5 for valence and 5/5 for tone. The no comment control group received only the PSA. The order of the ten selected comments was randomized before they were presented to the participants. PE of the PSA was surveyed right after each exposure to a PSA and its accompanying comments.

The design strategy we use gives every subject a unique set of ten comments for each PSA. In other words, the chance of two participants watching the same PSA accompanied by the same ten comments in the same order was virtually zero. Therefore, any observed effects cannot be attributed to a specific set of comments attached to each condition. The usual case-category confound was effectively eliminated.

Measures

Dependent Variables

Perceived Effectiveness of the PSA was assessed with three components - convincingness, confidence, and smoking-related thought (Zhao, Strasser, Cappella, Lerman & Fishbein, 2011). Convincingness was measured with an item read (a) “This ad was convincing"; confidence was measured with an item said (b) “Watching this ad helped me feel confident about how to best deal with smoking"; smoking-related thought was measured using the difference of two items: (c) “The ad put thoughts in my mind about quitting smoking”, and (d) “The ad put thoughts in my mind about wanting to continue smoking". All items were measured on a 5-point scale ranging from “1- strongly disagree” to “5- strongly agree". PE score of the PSA was calculated as PE = [a + b + (c - d)/2 + 3] / 3 (Cronbach’s α = .74 for the three components). Table 2 shows the mean and standard deviation of PE across experiment conditions.

Table 2.

Sample Size and Mean (standard deviation) of Perceived Effectiveness across Conditions

N Strong PSA Weak PSA Two PSAs Averaged
Positive Civil (PC) 108 3.25 (.74) 2.56 (.78) 2.90 (.63)
Positive Uncivil (PU) 85 3.27 (.75) 2.52 (.68) 2.89 (.59)
Negative Civil (NC) 98 3.12 (.75) 2.39 (.78) 2.73 (.63)
Negative Uncivil (NU) 113 2.98 (.73) 2.35 (.76) 2.64 (.64)
Control - Mixed 98 3.13 (.83) 2.52 (.87) 2.82 (.73)
Control – None 90 3.47 (.60) 2.61 (.78) 3.04 (.58)
Open in a new tab

Note. N = 567 for strong PSA; N = 574 for weak PSA; N = 550 for PSA averaged.

Attitude, perceived risk, and intention to quit were asked once after exposure to the PSAs and comments. Attitude was measured with a five point scale composed of eight items describing the possible benefits of quitting and harms of smoking (Cronbach’s α = .91). Examples are “I would add years to my life if I quit smoking completely and permanently in the next 3 months”, and “If I continue to smoke cigarettes at my current pace, it is likely that I will get heart disease."

Subjects indicated their perceived risk level by locating how much they “worry about your health risks because of your smoking”, “think about the serious health effects of smoking”, and “feel afraid or anxious about your smoking” on a four point scale ranging from “not at all” to “very much” (Cronbach’s α = .88).

Finally participants’ intention to quit smoking in the next three months was assessed on a four point scale (definitely will not - definitely will) that included three items: “quit smoking completely and permanently”, “reduce the number of cigarettes you smoke in a day”, and “talk to someone (friend, family member, spouse) about quitting smoking” (Cronbach’s a = .84).

Moderators and Covariates

A modified version of the Ladder of Contemplation (Biener & Abrams, 1991) was used to measure participants’ level of readiness to quit smoking. Participants were asked to choose a number between 0 to 10 to indicate where they were now at in thinking about quitting smoking. Statement at 0 read “I have no thoughts about quitting smoking”, and 10 meant “I am taking action to quit smoking". Study sample averaged 5.52 on the 11 point ladder ( SD = 2.87). Subject’s readiness to quit was recoded into three categories for the analysis: a score of 0 to 2 was considered of low readiness to quit (20.17%), 3 to 7 were coded as moderate readiness (49.28%), and subjects scoring 8 to 10 were of high readiness (30.55%).

People’s familiarity with online comments in general was measured with the question “How often do you read comments left by previous viewers?” using a 4-point scale ranging from “1- never” to “4- always” (M = 2.38, SD = .73).

Subjects in all but the no-comment conditions were asked the number of comments read. 55.3% of the respondents reported reading all the comments, 28.3% read some, 10% read a few, and 6.4% none of the comments. Subjects across conditions did not differ on the number of comments read. This variable served as a covariate in the model and no one was dropped from the study as a result of their response to this question.

Mediators

Transportation describes how much people felt “transported” into the ad and became involved with it (Green & Brock, 2000). It was measured on a five-point scale ranging from “strongly disagree” to “strongly agree” with five items (Cronbach’s α = .86 and .92 for the first and second exposure respectively). Sample items are: “I could picture myself in the scene of the events shown in the ad”, and “The events in the ad are relevant to my everyday life.”

Subjects’ emotional reaction to the PSA was assessed with five items asking how much they agree or disagree that they felt afraid, guilty, angry, hopeful, and proud. The first three emotions were averaged to indicate Negative Emotion, and the last two Positive Emotion.

Two scales were created to measure subjects’ reactance to the PSA. The Exaggeration scale asked to what extent did people think the information presented about smoking was exaggerated / dishonest / fake / insulting / stupid (Cronbach’s a = .92 and .93 for the first and second exposure). The Manipulation scale has four items measuring how much people felt they were manipulated by the ad (Cronbach’s a = .90 and .92 for the first and second exposure). Items included “this ad tried to make a decision for me”, and “this ad tried to pressure me".

Result

Manipulation Check

The comment manipulation was successful. Among the comment conditions subjects in the positive condition considered the comments to be most favorable toward the PSA, and subjects in the negative condition perceived the comments to be most critical towards the PSA (Mpositive = 3.27, SD = .78; Mmixed = 2.71, SD = .81; Mnegative = 2.25, SD = .76), F (2, 457) = 78.59, p < .001. Those in the civil condition thought the comments were respectful, and those in the uncivil condition believed the comments were impolite (Mcivil = 2.65, SD = .85; Mmixed = 2.37, SD = .83; Muncivil = 2.06, SD = .87), F (2, 459) = 22.19, p < .001. All six single degree of freedom contrasts (favorable - critical: positive vs. negative, positive vs. mix, negative vs. mix; respectful - impolite: civil vs. uncivil, civil vs. mix, uncivil vs. mix) were significant after applying Holm Bonferroni to adjust for family-wise error with the weakest contrast having a mean difference of .28 on a five point scale, t (276) = 2.60, p = .01.

Hypothesis Testing

A four-way mixed design ANCOVA was first conducted, with two levels of comment valence (positive vs. negative), two levels of comment tone (civil vs. uncivil), the third factor (within subject) for PSA quality (strong vs. weak), and the fourth factor included three levels on smokers’ readiness to quit (low vs. moderate vs. high). Number of comments read served as the covariate in the model.

Table 1 presents the results of the hypothesis tests and table 2 the means by condition for PE. A significant main effect was found for comment valence, F(1, 357) = 6.20, p = .01, indicating positive comments generated a higher PE than negative comments.

Table 1.

Analysis of Covariance of Perceived Effectiveness across PSA Quality by Comment Valence, Comment Tone, and Smokers’ Readiness to Quit

Source df F η 2 p
Between subjects
Number of Comments Read (N) 1 16.14* .04 .00
Comment Valence (V) 1 6.20* .02 .01
Comment Tone (T) 1 0.52 .00 .47
Readiness to Quit (R) 2 15.31* .08 .00
V × T 1 0.85 .00 .36
V × R 2 3.16* .02 .04
V × T × R 2 1.74 .01 .18
Subjects within-group 357 (.69)
Within subjects
PSA Quality (Q) 1 15.27* .04 .00
Q × V 1 0.75 .00 .39
Q × R 2 6.31* .03 .00
Q × Subjects within group 357 (.34)
Open in a new tab

Note. Values enclosed in parentheses represent mean square errors. Interaction terms T×R, Q×N, Q×T, Q×V×T, Q×V×R, Q×T×R, Q×V×T×R were included in the model but were not reported in the table since they are of little theoretical value and their effects were not significant.

*

p < .05

Strong PSAs remained relatively strong and weak PSAs remained relatively weak no matter what comments accompanied them, F(1, 357) = 15.27, p < .001, but PSA quality did not interact with comment valence (Q × V in Table 1), F(1, 357) = .75, p = .39. Thus, the hypothesis that the effect of comments would be particularly strong with weak PSA was not supported.

The interaction term between comment valence and smokers’ readiness to quit was significant (V × R in Table 1), F(2,357) = 3.16, p = .04, indicating the effect of comments’ valence on PE depends on smokers’ readiness to quit. Planned contrasts showed that positive and negative comment conditions do not differ on PE for smokers who are not ready to quit, t (357) = −.94, p =.35, but for those who are moderately or highly ready to quit smoking positive comments make them evaluate the PSA as more effective than the negative comments, t moderate readiness (357) = 3.07, p = .002; t high readiness (357) = 2.51, p = .013.

Deleterious Effect of Comments

The two control conditions (the mixed comments condition and the no comment condition) were not included in the analysis above because we sought to replicate Walther et al.’s findings with a different sample and context. The effects of any directional comments (pro or con) on the ads require comparison to two types of controls - no comment and balanced comments. So further analyses were conducted to compare the average PE across four valence conditions (positive, negative, mixed, none).

One-way ANOVA shows significant differences among the four comment valence conditions, F(3, 546) = 7.31, p<.001. Planned contrasts indicate that positive comments fail to improve PE compared with the no comment condition. On the contrary, positive comments had a lower PE than the no comment condition though the difference was only marginally significant, t(549) = 1.72, p = .08. As can be seen from Table 2, PSAs with no comments received the highest PE of all conditions. Planned contrasts comparing the no comment group with all the other groups combined showed the no comment condition had a significantly higher PE, t(549) = 3.21, p = .002, indicating overall the existence of comments decreases the PE of the anti-smoking PSAs.

Since the no comment condition achieves the highest PE, further analysis was conducted to examine how PSA quality moderates the influence the three types of comments’ have on PE compared with the no comment condition. As shown in Figure 1, compared with the no comment condition strong PSAs suffered significantly in all three comment conditions while weak PSAs were affected only by negative comments.

Figure 2.

Figure 2

Open in a new tab

Smokers’ Attitude Towards Quitting, Perceived Risk, and Intention to Quit Across Comment Treatment Conditions. Error bar showed the 95% CI.

The number of comments people read was related to judgments of perceived effectiveness of the ads (See table 1). For both positive and negative conditions those who read a few or some comments (Mpositive = 3.02, SD = .60; Mnegative = 2.84, SD = .59) perceived the PSA to be significantly more effective than those who read all the comments (Mpositive = 2.79, SD = .62; Mnegative = 2.57, SD = .66), tpositive(173)= 2.42, p = .017; tnegative(171) = 2.70, p = .003. One interpretation of this result is that the more comments read directly affected PE of the ads in a deleterious way regardless of the valence of the comments. However, it is also possible that viewers’ judgments of the ad’s effectiveness decreased the likelihood that they would read the comments that followed.

Comments as Distractors

To unravel the counter-intuitive results about comments and explore the causal direction a mediation analysis was conducted.

Mediation analysis (see Figure 3 for path model and coefficients) using joint-significance tests showed negative comments decreased PE because they lowered subjects’ transportation level, made them believe the PSA was exaggerated, and elicited their negative emotions like fear, guilt, and anger. Positive comments decreased PE only because they lowered subjects’ transportation level, indicating they distracted people from the PSA and lowered their level of engagement.

Figure 3.

Figure 3

Open in a new tab

Mediation analysis for strong PSAs. Standardized path coefficients with S.E. in parentheses. Model fit: X2 = 10.00 (p =.02); RMSEA = .06, p = .25; CFI = .99; TLI = .93; SRMR = .009. Exogenous variables in the model are dummy variables with the no comment condition as the base category. The same model was nm for weak PSAs and the result pattern was the same except the positive to transportation path was only marginally significant.

To confirm the distraction effect, we coded participants’ responses to the free recall task, which asked subjects to write down everything they could remember from the two PSAs they just watched. Recall of the first and the second PSA were coded separately on whether it was blank (yes/no), inaccurate (yes/no) or irrelevant (yes/no) by two independent coders who were blind to the hypotheses and conditions (Cohen’s Kappa >.78 for all six categories). To compute accuracy score, blanks were treated as missing data, an accurate recall was given a value of +1, an inaccurate recall a −1, and an irrelevant comment a 0. An overall accuracy score was then calculated for each individual by adding their scores for both PSAs. The overall accuracy score ranges from -2 (recall both PSAs incorrectly) to +2 (recall both correctly).

ANOVA showed positive, negative, mixed, and no comment conditions differ significantly on their mean accuracy score, F(3, 366) = 2.61, p = .05. Planned contrast showed that people in the no comment condition indeed remember the PSAs better than those in the comment conditions, t(366) = -2.30, p =.02. The no comment (M = .67, SD = .87) condition scored higher on recall accuracy than the positive condition (M = .29, SD = .96), t(366) = −2.46, p = .01, and the negative condition (M = .30, SD = .96), t(366) = −2.46, p = .01), but it did not surpass the mixed comment condition (M = .46, SD = .90), t(366) = −1.19, p = .23.

These results suggest that both positive and negative comments can have deleterious effects on smokers’ evaluation of the anti-smoking PSA. While the negative comments created an anti-ad or pro-smoking norm that lowered smokers’ PE of the ad, the positive comments harmed the PSA mainly because they were a distraction.

Incivility and Attitude, Perceived Risk, and Intention to Quit

Neither the main effect of comment tone nor its interaction with comment valence had a significant effect on PE in the four-way mixed design ANCOVA. To explore the possible effects incivility has on the persuasiveness of the comments, smokers’ attitude, perceived risk and intention to quit were compared across the four comment treatment conditions (PC, PU, NC, NU). A one-way MANOVA was conducted with comment treatment condition as fixed factor and attitude, perceived risk, and intention as dependent variables. Result revealed a significant multivariate main effect for comment condition, Wilks' λ = .93, F(9, 915.236) = 3.04, p = . 001. Given the significance of the overall test, the univariate main effects were examined. Significant differences among treatment conditions were obtained for attitude, F(3, 378)= 6.04, p = .001, and perceived risk F(3, 378) = 4.62, p = .003, but not for intention, F(3, 378) = 1.14, p = .33.

As shown in Figure 2, orthogonal contrasts reveal that the positive civil and the positive uncivil groups do not differ on attitudes, t(382) = 1.03, p = .30) or perceived risk, t(393) = −.24, p = .81, but those reading the negative uncivil comments score significantly lower than those reading the negative civil comments on attitude toward quitting, t(382) = 3.31, p = .001, and perceived risk of smoking, t(393) = 2.97, p = .003. Since all subjects in the study were smokers, positive comments could be considered as counter-attitudinal and negative comments pro-attitudinal to the subjects. Therefore this set of findings supports Scherer’s (2007) argument that incivility improves the persuasiveness of pro-attitudinal messages. In other words, smokers reward incivility on their own side but ignore incivility on the opposing side.

Further analysis showed that the strengthening effect of incivility on negative comments was not moderated by people’s familiarity with online comments, but it was moderated by age (See Table 3). Unlike the majority of the sample, senior smokers (age 60 and above, 19.4% of the sample) do not appreciate the anti-PSA / pro-smoking comments that are uncivil.

Table 3.

Summary of Simple Regression Analyses for Variables Predicting Attitude and Perceived Risk

Attitude Perceived Risk
Model B SE B β B SE B β
Condition (NC vs. NU) −1.34 .77 −.82 −.89 .77 −.53*
Age −.04 .02 −.74* −.03 .02 −.40
Familiarity .23 .36 .20 .47 .37 .40
Condition × Age .03 .01 1.04* .02 .01 .82*
Condition × Familiarity −.15 .21 −.28 −.20 .21 −.40
Open in a new tab

Note.

*

p<.05.

Discussion

This study found smokers’ perceived effectiveness of the anti-smoking PSAs was influenced by comments following the PSAs. PSAs with positive comments were perceived by smokers as more effective than PSAs with negative comments, which replicated the general findings of Walther et al. (2010). However, smokers in the no comment condition gave the highest PE score to PSAs. Also, consistent with the literature on uncertainty and social influence, smokers’ prior readiness to quit smoking moderated the effect of comments on their perception of the PSAs. The most hardcore smokers considered the PSAs ineffective no matter what comments they saw. For smokers in the middle or at the top of the contemplation ladder, however, positive comments made PSA more effective compared with negative comments.

The most surprising finding from the study is that positive comments failed to improve PSA evaluation over the no-comment exposure to ads. Quite opposite to the hypothesis, even positive comments significantly decreased smokers’ perceived effectiveness if the ad was strong. The key question then is: why did the positive condition fail to generate a higher PE than the no comment condition? Three possible explanations are offered:

  • 1) Comments distracted audience from the PSA, decreased their level of engagement and thus decreased PE. As demonstrated in the result section, mediation analysis and the test of recall indicated the presence of comments was a distraction.

  • 2) The stimuli were not pure. As mentioned in the method section, comments in the positive condition consisted of eight positive comments and two negative comments for each PSA. It is thus possible that PE was dragged down by the two negative comments mixed in the positive condition.

    The current study adopted an 8-2 ratio for comments for ecological validity. However, the 8 - 2 approach can be considered as a two-sided message while a pure positive condition would be a one-sided message. Meta-analysis of two-sided messages indicates that they are consistently more persuasive than one-sided messages only when they are refutational, i.e. when they acknowledge and refute the opposing arguments. When they are not refutational, then it is possible that two-sided messages (without refutation) would be inferior to one-sided messages in their persuasiveness (O’Keefe, 1999). Comments in the present study were not refutational but were sequential and sometimes unrelated - as is often the case online. They could not be considered to be a coherent persuasive message. So a pure positive condition could still elicit higher PE than the current stimuli did. What remains unclear, however, is whether it will be able to elicit significantly higher PE than the no comment control condition. Of course, the reality of online recommendations is that they are not pure conditions but are very likely to represent a range of points of view, sometimes refutational in a substantive sense but with each side refuting the other. In this sense, our findings are realistic and representative.

  • 3) Positive comments were resources for smokers to counter-argue. Positive comments were of the same valence as anti-smoking PSA, but the creator of the comments has less credibility and authority than creators of the PSAs. When subjects, who were all smokers, tried to counter argue, those in the positive condition may ignore the PSA and focus on the comments instead because they are an easier target to attack.

Studies using rating scores or ranking information to create social influence succeeded in promoting positive group’s evaluation from no information control (Cohen & Golden, 1972; Salganik, Dodds & Watts, 2006), which indicate comments may function differently from rating or ranking. As mentioned before, commentary is of particular interest among forms of online recommendation system because it carries more information than rating and ranking. While rating scores only describe the valence of the social norm, comments can describe both valence of the norm and the arguments and rationales supporting the norm. On the one hand, the arguments and rationales embedded in the comments may strengthen the influence of online recommendation system by making social norm more salient; on the other hand, they may weaken the influence by providing viewers with resources to counter-argue or by overwhelming viewers with too much information. To test the third explanation, future studies can compare the effects of different kinds of online recommendation system, positive rating scores versus positive comments for example.

Implications

The detrimental effect of comments on PE, attitude, and perceived risk found by the current study seems to suggest anti-smoking PSAs would be better off without comments, especially if the PSAs are strong or if the target audience is somewhat ready to quit smoking.

Social media have received a great deal of attention from public health officials in part because of the personalness of these media and in part because of the power they give to audiences to contribute. This power is double-edged as the present study shows. Contributors can offer extensive comments and these in turn can undermine the impact of health messages when the comments are not moderated in some way. Our data suggest that under some conditions the intended message is lost or reduced in effectiveness during the diffusion process.

Understanding the influence of online commentary for a variety of topics, under different conditions of trustworthiness and bias of the commentators, and with various degrees of control by the web sites involved will require concerted effort by researchers as we seek to understand this version of social influence as manifested in the emerging social media environment. One thing is very clear: it is no longer possible to consider the influence of news or other messages in the public information environment apart from the comments which follow them.

Figure 1.

Figure 1

Open in a new tab

Perceived effectiveness for strong and weak PSAs across four comment conditions. Error bars showed the 95% CI. For strong PSAs the none condition differed significantly from each of the three other conditions. For weak PSAs only the negative comment condition differed significantly from the none condition.

Contributor Information

Rui Shi, Wake Forest University, Annenberg School for Communication, University of Pennsylvania, 336-749-8713, 3620 Walnut St., Philadelphia, PA, 19104.

Paul Messaris, University of Pennsylvania, Lev Kuleshov Professor of Communication, Annenberg School for Communication, University of Pennsylvania, 215-898-4208, 3620 Walnut St., Philadelphia, PA, 19104

Joseph N. Cappella, Michigan State University, Gerald R. Miller Professor of Communication, Annenberg School for Communication, University of Pennsylvania, 215-898-7059, 3620 Walnut St., Philadelphia, PA, 19104

Reference

  1. Asch S. Studies of independence and conformity: A minority of one against a unanimous majority. Psychological Monographs: General and Applied. 1956;70(9):1–70. [Google Scholar]
  2. Bostrom RN, Baseheart JR, Rossiter C. r. The Effects of Three Types of Profane Language in Persuasive Messages. Journal of Communication. 1973:23461–475. [Google Scholar]
  3. Biener L, Abrams DB. The contemplation ladder: Validation of a measure of readiness to consider smoking cessation. Health Psychology. 1991;10:360–365. doi: 10.1037//0278-6133.10.5.360. [DOI] [PubMed] [Google Scholar]
  4. Cialdini R, Golstein N. Social influence: Compliance and conformity. Annual Review of Psychology. 2004;55:591–621. doi: 10.1146/annurev.psych.55.090902.142015. [DOI] [PubMed] [Google Scholar]
  5. Cohen J, Golden E. Informational social influence and product evaluation. Journal of Applied Psychology. 1972;56(1):54–59. [Google Scholar]
  6. Durkin S, Wakefield M. Maximizing the impact of emotive antitobacco advertising: Effects of interpersonal discussion and program placement. Social Marketing Quarterly. 12. 2006:3–14. [Google Scholar]
  7. Eagly AH, Chaiken S. An attribution analysis of the effect of communicator characteristics on opinion change: The case of communicator attractiveness. Journal of Personality and Social Psychology. 1975;32(1):136–144. [Google Scholar]
  8. Finn J, Banach M. Victimization online: The downside of seeking human services for women on the internet. CyberPsychology & Behavior. 2000;3(5):785–796. [Google Scholar]
  9. Giner-Sorolila R, Chaiken S. Selective use of heunrstic and systematic processing under defense motivation. Personality and Social Psychology Bulletin. 1997;23(1):84–97. [Google Scholar]
  10. Green MC, Brock TC. The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology. 2000;79(5):701–721. doi: 10.1037//0022-3514.79.5.701. [DOI] [PubMed] [Google Scholar]
  11. Heatherton TF, Kozlowski LT, Frecker RC, Fagerstrom KO. The Fagerstrom test for nicotine dependence: A revision of the Fagerstrom tolerance questionnaire. British Journal of Addiction. 1991;86:1119–1127. doi: 10.1111/j.1360-0443.1991.tb01879.x. [DOI] [PubMed] [Google Scholar]
  12. Jackson S. Message Effects Research. Guilford; New York: 1992. [Google Scholar]
  13. Kang Y, Cappella JN, Strasser AA, Lerman C. The effect of smoking cues in antismoking advertisements on smoking urge and psychophysiological reactions. Nicotine & Tobacco Research. 2009;11(3):254–261. doi: 10.1093/ntr/ntn033. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Kang M. The influence of public opinion polls on public opinion: The role of motivation and ability in the elaboration likelihood model. Dissertation Abstracts International Section A: Humanities and Social Sciences. 1997;59 (1998-95015-092; AAM9825262) [Google Scholar]
  15. Kushin MJ, Kitchener K. Getting political on social network sites: Exploring online political discourse on facebook. First Monday. 2009;14(11) [Google Scholar]
  16. Moor PJ, Heuvelman A, Verleur R. Flaming on YouTube. Computers in Human Behavior. 2010;26(6):1536–1546. [Google Scholar]
  17. Mulac A. Effects of obscene language upon three dimensions of listener attitude. Communication Monographs. 1976;43(4):300–307. [Google Scholar]
  18. Mutz DC. Effects of ‘in-your-face’ television discourse on perceptions of a legitimate opposition. American Political Science Review. 2007;101(4):621–635. [Google Scholar]
  19. Mutz DC, Byron R. The new videomalaise: Effects of televised incivility on political trust. American Political Science Review. 2005;99(1):1–16. [Google Scholar]
  20. Ng EWJ, Detenber BH. The impact of synchronicity and civility in online political discussions on perceptions and intentions to participate. Journal of Computer-Mediated Communication. 2005;10(3) [Google Scholar]
  21. O’Keefe DJ. How to handle opposing arguments in persuasive messages: A meta-analytic review of the effects of one-sided and two-sided messages. Communication Yearbook. 1999;22:209–249. [Google Scholar]
  22. Orenga V, Zornoza AM, Prieto F, Peiró JM. The influence of familiarity among group members, group atmosphere and assertiveness on uninhibited behavior through three different communication media. Computers in Human Behavior. 2000;16:141–159. [Google Scholar]
  23. Petty RE, Cacioppo JT. Communication and persuasion: Central and peripheral routes to attitude change. Springer-Verlag; New York: 1986. [Google Scholar]
  24. Price V, Stroud N. Public attitudes toward polls: evidence from the 2000 U.S. presidential election. International Journal of Public Opinion Research. 2006;18(4):393–421. [Google Scholar]
  25. Salganik MJ, Dodds PS, Watts DJ. Experimental study of inequality and unpredictability in an artificial cultural market. Science. 2006;311(5762):854–856. doi: 10.1126/science.1121066. [DOI] [PubMed] [Google Scholar]
  26. Samu S, Bhatnagar N. The efficacy of anti-smoking advertisements: The role of source, message, and individual characteristics. International Journal of Nonprofit and Voluntary Sector Marketing Special Issue: Social Marketing. 2008;13(3):237–250. [Google Scholar]
  27. Scherer CR. Indecent influence: What mediates the relationship between swearing and persuasion. Dissertation Abstracts International: Section B: The Sciences and Engineering. 2007;67(9-B) Retrieved from www.csa.com. (2007-99006-361; AAI3235678) [Google Scholar]
  28. Scherer CR, Sagarin BJ. Indecent influence: The positive effects of obscenity on persuasion. Social Influence. 2006;1(2):138–146. [Google Scholar]
  29. Sherif M. A study of some social factors in perception. Archives of Psychology. 1935;27(187) [Google Scholar]
  30. Sherif M. The psychology of social norms. Harper & Brothers; 1936. [Google Scholar]
  31. Sonck N, Loosveld G. Impact of Poll Results on Personal Opinions and Perceptions of Collective Opinion. International Journal of Public Opinion Research, 2010;22:230–255. [Google Scholar]
  32. Strasser AA, Cappella JN, Jepson C, Fishbein M, Tang KZ, Han E, et al. Experimental evaluation of antitobacco PSAs: Effects of message content and format on physiological and behavioral outcomes. Nicotine & Tobacco Research. 2009;11(3):293–302. doi: 10.1093/ntr/ntn026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Sutfin EL, Szykman LR, Moore MC. Adolescents’ responses to anti-tobacco advertising: Exploring the role of adolescent smoking status and advertisement theme. Journal of Health Communication. 2008;13(5):480–500. doi: 10.1080/10810730802198961. [DOI] [PubMed] [Google Scholar]
  34. Tesser A, Campbell J, Mickler S. The role of social pressure, attention to the stimulus, and self-doubt in conformity. 1983;13(3):217–233. [Google Scholar]
  35. Walther JB, DeAndrea D, Kim J, Anthony JC. The influence of online comments on perceptions of anti-marijuana public service announcements on YouTube. Human Communication Research. 2010;36(4):469–492. [Google Scholar]
  36. Wooten D, Reed A., II Informational influence and the ambiguity of product experience: Order effects on the weighting of evidence. Journal of Consumer Psychology. 1998;7(1):77–99. [Google Scholar]
  37. Zhao X, Strasser A, Cappella JN, Lerman C, Fishbein M. A measure of perceived argument strength: Reliability and validity. Communication Methods & Measures. 2011;5:48–75. doi: 10.1080/19312458.2010.547822. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Zhu H, Huberman BA, Luon Y. To switch or not to switch: Understanding social influence in recommender systems. Computing. 2011 Retrieved from http://arxiv.org/abs/1108.5147.

RESOURCES