Efficiently and Effectively Evaluating Public Service Announcements: Additional Evidence for the Utility of Perceived Effectiveness

Abstract

Recent research has made significant progress identifying measures of the perceived effectiveness (PE) of persuasive messages and providing evidence of a causal link from PE to actual effectiveness (AE). This article provides additional evidence of the utility of PE through unique analysis and consideration of another dimension of PE important to understanding the PE-AE association. Current smokers (N =1,139) watched four randomly selected anti-smoking Public Service Announcements (PSAs). PE scores aggregated by message were used instead of individual PE scores to create a summed total, minimizing the likelihood that PE perceptions are consequences of an individual’s intention to quit, supporting instead the PE→AE order. Linear regression analyses provide evidence of PE’s positive and significant influence on smoking cessation-related behavioral intentions.

Efficient procedures for evaluating the effectiveness of messages for the purposes of persuasion would be a useful tool for those designing public health campaigns, political and social issue campaigns, and certainly for researchers seeking to evaluate various types of message effects as they undertake theory testing. A good deal of attention and very significant progress has been made on this issue the past decade (e.g., Dillard, Shen, & Vail, 2007; Dillard, Weber, & Vail, 2007; Dillard & Ye, 2008; Fishbein, Hall-Jamieson, Zimmer, von Haeften, & Nabi, 2002). The purpose of the present paper is to advance the research base by providing data that will enhance researchers’ confidence in perceived effectiveness (PE) as a valid, if imperfect, measure of actual effectiveness (AE).

The Need for an Efficient Measure of PE

The most direct way to evaluate the effectiveness of a message is to conduct a test in the field with the appropriate target population. For example, Public Service Announcements (PSAs) to be used in a public campaign can be compared on the outcome variables of interest against some control. For messages aimed at behavior change (assuming the proper design and controls), this is perhaps the most accurate way to measure message effectiveness. Unfortunately, this strategy requires running at least a small-scale campaign in the field and then evaluating its success in changing behavior, which can take months or longer, rendering it impractical and expensive.

The obvious problem with measuring AE is its inefficiency. Researchers and campaign designers need to know about the effectiveness of messages in advance of testing the final campaign and before deploying campaign resources. One strategy employed by advertisers, for example, has been to test preliminary versions of an advertisement in early stages of development before implementing the more expensive full-fledged version. But even these preliminary versions need to be evaluated in some way for their potential effectiveness in order to choose the best message for production and implementation.

Research on PE

Prior research has significantly advanced the development of efficient measures of PE and the validation of these measures relative to important outcomes such as attitude and behavioral intention. Although quite encouraging, the results of these studies need to be bolstered by other data. Specifically, three issues need to be addressed: target outcomes, causal direction, and representativeness of the samples employed.

Target Outcomes

In experimental work, PE has been positively correlated with judged realism, amount learned from the ad, negative emotion in response to the ad, attitude toward the advocacy of the ad, cognitive responses (thought-listing difference scores), and intention to engage in advocated behavior, and negatively correlated with positive emotional responses (Barrett, Cappella, Fishbein, Yzer, & Ahern, 2011; Dillard, Shen, et al., 2007; Fishbein et al., 2002). In addition, a meta-analysis of 40 studies (approximately 3,000 respondents) found a significant effect size, r = .41 with 95% of the effects positive, for the PE-AE (in this case attitude) relationship (Dillard, Weber, et al., 2007).

Having attitude, cognitive response, intention, and ultimately behavioral change data are necessary steps in validating the utility of any PE measure. Prior research has largely focused on attitude toward the message advocacy as an indication of AE. However, in assessing the PE-AE relationship, the gold standard for outcomes would be behavior change rather than attitude, emotion, or some other indicator of behavior. Short of behavior change itself, which is difficult to test and certainly not efficient, a change in specific intentions regarding behaviors pertinent to the target audience is a reasonable surrogate (for meta-analyses of the intention-behavior relationship see Hale, Householder, & Greene, 2002; Webb & Sheeran, 2006). In the five studies conducted by Dillard, Shen, et al. (2007), only one used intention to act as an outcome measure. Additionally, the measures employed were general rather than the very specific measures advocated by Fishbein and Ajzen (2010), in part because they are more likely to correlate with behavior. To build on the previous findings, the validation presented here focuses on intention to act in specific ways where the intention is highly relevant to the individual and the time frame and behavior are quite specific. If data can be marshaled to show that messages high in PE are associated with specific intentions consistent with the messages and pertinent to the targets of the messages, while those low in PE are not (or are less so), then confidence in PE as a test of a message will be enhanced.

Causal Direction of the PE-AE Relationship

In a study that did utilize behavioral intention as a measure of actual message effectiveness, Zhao, Strasser, Cappella, Lerman, and Fishbein (2011) evaluated responses to anti-smoking PSAs and found that perceived argument strength was positively associated with intentions to quit smoking, r = .44, p < .001, which can be interpreted as a relationship between PE and AE. Another study presented in the same article showed a similar pattern for adolescents exposed to anti-drug messages and intention to use marijuana regularly, r = −.35, p < .001 (Zhao et al., 2011). While the correlations reported in these studies are substantial and consistent, they do not allow inferences about causal direction between PE and AE because it is just as likely that those intending to quit smoking or those adolescents with low drug use intentions evaluate the related ads they see in more positive ways. Causality is very much in question even though pertinent intentions were carefully assessed.

Dillard, Shen, et al. (2007) also attempted to establish that PE is antecedent to AE, and not the reverse, by running structural equation models with their data in both directions. In both cases, one study used attitude as a representation of AE and the other behavioral intention, the PE → AE order fit the data better than did the reverse order model. These findings are certainly consistent with PE being causally prior to AE, but they cannot be definitive in the same way as an experiment with random assignment. Of the five studies described in the article, only the fifth manipulated PE by telling students that a given message was or was not effective and examined the consequences of this manipulation on behavioral intention. The results showed that messages thought to be effective were more likely to be associated with increased behavioral intention consistent with the message’s advocacy. This elegant experimental approach allowed the researchers to control message content completely and only manipulate the perception of effectiveness. In the end, this highly suggestive finding needs to be generalized to a larger set of messages using PE measures to select messages in advance, randomly assigning individuals to messages received, and testing an outcome in the form of intention or behavior that is pertinent to the individual.

Representativeness of Samples

Participants

Not surprisingly, a majority of the participants in PE-AE related research have been college students. All of the samples employed by Dillard, Shen, et al. (2007) and the majority of the samples in Dillard, Weber, et al.’s (2007) meta-analysis were college students. Student samples surely do not invalidate the findings, but if PE measures are to be employed in the context of public campaigns where the audience is much broader than a college population, it is necessary to show that the PE–AE relationship holds in these samples. While a few studies have focused on adolescent samples consisting of middle school and high school students (e.g., Barrett et al., 2011; Fishbein et al., 2002), even fewer studies have recruited nationally representative adult samples.

Messages

A variety of message topics have been studied in this context. Dillard, Weber, et al.’s (2007) meta-analysis provides an overview of the variety of message topics, ranging from school exams to sexually transmitted diseases to flossing to city taxes. However, the number and variety of message themes within each topic has been much more limited, likely because of technology and available resources. Our research contributes to message sample representativeness by including a large number of ads (N = 100) within the same topic (anti-smoking) but with a variety of message themes (e.g., death, secondhand smoke, strategy for quitting). The theme of the message can affect PE because the theme is linked to the argument of the message. Therefore, it is important to include a variety of message themes, not just message topics, when examining the utility of PE.

Measurement of PE

Measures of PE have varied greatly with no agreed upon set of items used for message assessment (Yzer, Vohs, Luciana, Cuthbert, & MacDonald, 2011). In fact, different operational measures of PE have been significantly correlated with different outcome measures (e.g. Barrett et al., 2011; Dillard, Shen, et al., 2007). To better understand message effectiveness, Yzer et al. (2011) examined the conceptualization and operationalization of PE and the affective antecedents to PE in the context of anti-drug PSAs. Using seven global PE statements from previous research, the authors conducted a factor analysis which revealed two PE factors: convincingness and pleasantness. However, Yzer and colleagues point out that the items most widely used are part of the convincing factor, which only captures part of how PE has been conceptualized.

Examining the differences between the two dimensions of PE more closely, Yzer et al. (2011) found that the different factors of PE correlated with different retrospective affective responses. The convincing factor was highly correlated with arousal (e.g., excited vs. bored), r = .74, and the pleasantness factor was highly correlated with valence (e.g., happy vs. unhappy), r = .66. In addition, the authors found that the anti-drug PSAs the adolescents rated highest in PE were also the messages with a combination of high arousal and negative valence ratings, which is consistent with a great deal of fear appeal research. Even if affect is an antecedent of PE, Yzer et al.’s results indicate that PE is strongly correlated with affective responses. Thus, both cognitive and affective responses should be taken into consideration when assessing the PE of a message. Our study examines the impact of both emotional responses to PSAs and more traditional PE items on behavioral intentions.

Hypothesis

The case for the validity of PE of persuasive messages is solid but incomplete. The set of messages that have been tested with measures of PE needs to grow; the population tested needs to be more representative; the outcomes need to target behaviors (or intentions likely to tap those behaviors) relevant to the target audience; and a range of PE judgments needs to be tested in order to show that high doses of PE have greater influence than moderate doses and moderate doses more influence than low.

In the present study, a general population of adult smokers was exposed to four anti-smoking PSAs randomly selected from a large set (N = 100). After viewing the PSAs, viewers indicated their intention to quit smoking permanently and completely as well as other intentions concerning actions that can assist with the cessation process. We were primarily interested in intention to quit because the specific language and time frame provides a conservative test of PE and because it is the intention closest to the behavioral gold standard by which many anti-smoking campaigns are judged successful or not. Included in the responses after each PSA are evaluative assessments of each message’s persuasive qualities.

There are three advantages to using a large sample of smoking cessation messages. First, the messages covered a range of themes and stylistic features, allowing messages to range in terms of perceived effectiveness. Second, both the message and message order were randomly assigned to participants, meaning each participant should have had a unique viewing experience. Third, a large sample of messages allowed us to use an aggregate PE score instead of individual PE scores to predict behavioral intentions, minimizing the likelihood that the causal direction is anything other than from PE to individual behavior intentions (AE). The aggregate PE score was created by taking the average score for each message (from the set of respondents who viewed the message) and assigning it in place of each individual score for that message. Using the average score for each message helps control for other factors that may influence an individual’s evaluation of it (e.g., readiness to quit and message order) and provides what we believe to be a less biased and more accurate reflection of the message’s potential influence on behavior. The average score for each of the four messages an individual saw were then summed to create an overall aggregate PE score (see Figure 1). The large sample of smoking cessation messages allowed us to aggregate the scores this way because each person watched a different set of PSAs, so each person still ended up with a unique overall PE score. Thus, the aggregate scores were used to address causality because the behavioral intentions of one individual could not cause the message evaluations of the other individuals who watched the same message. This procedure is similar to conducting a study in two parts with separate samples. For example, suppose participants of the first sample rated the PE of all the messages and the participants of the second sample were randomly assigned to view four messages and subsequently rated their intentions to engage in smoking cessation-related behaviors. If we used the PE scores from the first sample to calculate a sum PE score for each participant in the second sample and used it to predict the behavioral intentions of the second sample participants, there is no way the intention scores could be causally prior to the PE scores because they came from a different sample of people. This hypothetical study using two separate samples is virtually identical to the methodology used in the current study with one sample.

Figure 1.

The creation of participant sum perceived effectiveness scores aggregated by video. Individual PE ratings of each video were first aggregated by video, then aggregate video PE scores were substituted for each individual’s PE score for the videos they watched. Finally, the aggregate video scores were summed to create a unique summed aggregate PE score for each participant. In the figure, P = participant, PE = perceived effectiveness score, V = video, x = the total number for that variable.

We hypothesized that exposure to anti-smoking messages rated as persuasive in aggregate would be associated with higher reported intentions to quit smoking, reduce smoking, and talk about quitting with others at the individual level. Because messages were assigned to persons at random, and because aggregate scores across PSAs are used to predict message strength, PSAs and their evaluations are a random variable. To insure that the results from the random variable (perceived message effectiveness) are not confounded by other factors, controls like readiness to quit (stage of change), need for cognition, and other individual differences were included in all analyses.

Method

Data were collected from two different samples with only minor differences in design; these are referred to as Study 1 and Study 2. The method for both studies is presented in this section; all differences between the studies are presented in text.

Participants

Participants (Study 1 N = 566, Study 2 N = 630) were drawn from a nationally representative sample and were current smokers who reported smoking at least five cigarettes a day and more than 100 cigarettes in their lifetime. The mean age for Study 1 was 49.60, SD = 11.01, range: 19–66. For Study 2, the mean age was 46.20, SD = 11.85, range: 18–65. All participants were part of the KnowledgePanel, established and maintained by Knowledge Networks (KN). Potential KnowledgePanel participants are first selected through random digit dialing and address-based sampling and are then contacted by trained interviewers and recruiters at KN. Participants without a computer or internet access are provided with the necessary equipment at no charge if they would like to participate in the KnowledgePanel. Incentive points can be earned for completing surveys and redeemed for cash. KnowledgePanel participants are notified about surveys for which they qualify and each panelist determines which surveys s/he would like to complete. Recruitment was done by KN through their KnowledgePanel and continued until the desired number of completes was obtained.

Dropped participants in Study 2

Twenty-six individuals were dropped from analyses because their video viewing times were either under 25 seconds (all PSAs were at least 25 seconds in length) or over 600 seconds (the longer time allowed for differences between broadband, dial-up, and other internet connection users). In addition, 49 responses from 39 individuals were deleted due to the video viewing time of one or two PSAs. The other responses of these individuals were included in aggregate calculations and the individual behavioral responses used in analyses.

PSA Videos

One hundred anti-smoking television PSAs (60 in Study 1, 40 in Study 2) were selected from a collection compiled from various sources including the American Legacy Foundation and various state health agencies. All PSAs were professionally produced and many, but not all, had aired as part of either state or national campaigns between 1998 and 2007. The major themes of the PSAs were (a) disease and death, (b) selling disease and death, (c) endangers others-secondhand smoke, (d) endangers others–burden (negative consequences your family may suffer if you become ill and/or die), (e) smokers’ negative life circumstance, (f) marketing tactics (how tobacco companies market their product), and (g) strategy for quitting. However, these themes are general categorizations of the primary themes; many of the PSAs had two or even three themes.

In both studies, each participant watched four PSAs that were embedded within the online questionnaire. Within the survey, both PSA selection and order were completely random. Thus, each person was likely to receive a different set of smoking cessation PSAs. When there was PSA overlap, individuals were likely to have received them in a different order and in a different context. In the context of each study and the sets of potential PSAs, total PSA exposure is truly a random variable.

Procedures

Participants were randomly selected from active KnowledgePanel members; individuals who participated in the first study were excluded from the selection process and were therefore not allowed to participate in the second study. Selected participants received an email notifying them of a new survey with a link to the questionnaire. Reminder emails were sent to all initial participants after three days of non-response before additional participants were randomly selected and emailed. Surveys could be taken by KnowledgePanel members on any computer with an internet connection. First, a screen providing a brief description of the study was displayed; continuing to the next screen provided implied consent. After viewing the initial screens, individuals answered several questions about smoking, their health, and individual differences. Participants then watched one PSA and answered questions about that specific message; this procedure was repeated for each of the next three PSAs. Finally, individuals answered questions about their behavioral intentions, self-efficacy, and beliefs and attitudes related to smoking and health. At the end of the questionnaire, participants were thanked for their time and were given an opportunity to provide additional thoughts or feedback. On average, it took between 23 and 28 minutes for participants to complete either study.

Independent Measures

Perceived message effectiveness

PE was assessed by four items on a 5-point Likert-type scale (1 = SD, 5 = SA). The items included “This ad was convincing” and “Watching this ad helped me feel confident about how to best deal with smoking.” Similar to Zhao et al. (2011), favorable and unfavorable thoughts about the message were also measured on this scale: “The ad put thoughts in my mind about quitting smoking” and “the ad put thoughts in my mind about continuing to smoke.”

To create the PE score, unfavorable thoughts (b) were first subtracted from favorable thoughts (a). The resulting score was divided by two and added to three ([(a-b)/2]+3) so that it was on the same 5-point scale as the other PE items. The newly created thoughts score was then added to the other two items and the sum of those items was divided by three (Cronbach’s α = .75, m = 2.98, SD = .82). The PE scores were then aggregated across respondents by PSA (by study to preserve any differences in PE that may exist). Finally, the mean PE scores for the four PSAs each individual saw were added together to create a sum aggregate PE score (see Figure 1), range: 4–20. The mean and standard deviation can be found in Table 1.

Table 1.

Means, Standard Deviations, and Correlations of Major Variables

	Mean (SD)	1	2	3	4	5	6	7	8	9	10	11	12
1. Aggregate PE	11.91 (.57)	1
2. Readiness	5.50 (2.89)	.08*	1
3. NFC	2.50 (.77)	−.06	−.06*	1
4. Dependence	4.22 (2.35)	.05	−.07*	.14**	1
5. Aggregate Fear	9.60 (.84)	.82**	.04	−.04	.07*	1
6. Aggregate Pride	8.08 (.33)	.45**	.08**	−.03	.05	.37**	1
7. Aggregate Hope	9.31 (.57)	.64**	.09**	−.05	.04	.43**	.83**	1
8. Aggregate Guilt	10.12 (.83)	.85**	.05	−.06*	.06	.92**	.42**	.50**	1
9. Aggregate Anger	9.82 (.54)	.44**	.04	−.04	.04	.63**	.03	.05	.55**	1
10. Quit	2.20 (.75)	.09**	.55**	−.03	−.05	.06*	.10**	.10**	.07*	.04	1
11. Reduce	2.81 (.83)	.13**	.56**	−.08**	−.09**	.08*	.11**	.13**	.09**	.02	.66**	1
12. Talk	2.52 (.95)	.15**	.46**	−.05	.00	.12**	.09**	.11**	.12**	.07*	.54**	.60**	1

^*p ≤ .05,

^**p ≤ .01 (2-tailed, N = 1,139, listwise deletion)

Emotional responses to PSAs

Yzer et al. (2011) found that the pleasant dimension of PE, captured by semantic differential items that assessed the ad as good-bad and negative-positive, was highly correlated with the valance (happy vs. unhappy) dimension of affect. In other studies, emotional connotations of messages have been significantly correlated with PE (e.g., Biener, McCallum-Keeler, & Nyman, 2000; Dillard & Peck, 2000) and highly emotional narratives have been positively associated with smoking quit attempts (Durkin, Biener, & Wakefield, 2009), providing more evidence for the need to account for emotional responses to messages as well as traditional cognitive components of PE.

The five potential emotional responses chosen were fear, guilt, anger, hope, and pride. The items asked the respondents to indicate their agreement with the statement “I felt…” on a 5-point Likert-type scale (1 =SD, 5 = SA). Emotional response scores were also aggregated by PSA (by study) and individual sum scores were created based on the four messages each person saw, score range: 4–20. Means and standard deviations for each emotional response are presented in Table 1.

Dependent Measures

Behavioral intentions

Properly measured behavioral intentions have often been used in communication and psychological research as a good predictor of behavior (Hale et al., 2002; Webb & Sheeran, 2006). Individuals responded to three different behavioral intention items¹ on a 4-point scale (1 = definitely will not, 4 = definitely will). Items asked participants how likely it was in the next three months they would “quit smoking completely and permanently,” “reduce the number of cigarettes you smoke in a day,” and “talk to someone (friend, family member, spouse) about quitting.” These three items were examined both as an overall measure of smoking cessation intention and individual behavioral intentions. Means and standard deviations are provided in Table 1 for the individual intentions.

Control Measures (Pre-PSA Exposure)

Variables that may affect participant responses to the PSAs were also measured and included in analyses for control purposes. The means and standard deviations for all control measures are also included in Table 1.

Readiness to quit

An individual’s intention to act or not act must start with a decision about the behavior; this idea is outlined in detail in the transtheoretical model’s construct stages of change (Prochaska & DiClemente, 1982). Therefore, it was expected that an individual’s readiness to quit smoking would have an influence on their future quitting-related behavior, regardless of the PE of the message.

The participant’s level of mental and/or physical preparation to attempt to quit smoking was represented by a modified ladder of contemplation (Biener & Abrams, 1991). Response options ranged from 0 (I have no thoughts about quitting smoking) to 10 (I am taking action to quit smoking). Biener and Abrams (1991) have validated this scale as a measure of motivation to stop smoking.

Nicotine dependence

Participants also responded to the Fagerström Test for Nicotine Dependence (FTND), a revision of the Fagerström Tolerance Questionnaire (FTQ), that is meant “to provide a short, convenient self-report measure of dependency on nicotine” (Heatherton, Kozlowski, Frecker, & Fagerström, 1991, p. 1119). Items were a mix of open-ended and fixed response. Fixed response options were either on a 4-point scale (range: 0–3) or a 2-point scale (range: 0–1) (Heatherton et al., 1991). Cronbach’s alpha was .59, which is similar to previous research that reports Cronbach’s alpha ranges of .61 to .64 (Heatherton et al., 1991; Pomerleau, Carton, Lutzke, Flessland, & Pomerleau, 1994).

Both the FTQ and FTND exhibit only moderate levels of internal consistency (see Heatherton et al., 1991). However, prior research has assessed the test-retest reliability of the scales and the relationship of the scales to biochemical indicators of smoking heaviness and other measures of external validity (e.g., years smoking, other addiction scales) (Heatherton et al., 1991; Pomerleau et al., 1994). Pomerleau et al. (1994) found the FTND had a higher Cronbach’s alpha and better test-retest reliability than the FTQ, although the scales were not significantly different on their measures of external validity. Given that the FTND is meant to measure a biochemical experience and we are using it as a control variable, we do not believe the low reliability negatively affects our data analyses.

Need for cognition

Need for cognition is an individual difference variable that influences the persuasion processes (for a review see Cacioppo, Petty, Feinstein, & Jarvis, 1996) and was therefore included to control for any message elaboration differences that occurred between individuals. Four items were selected from Cacioppo and Petty’s (1982) scale and two were reverse coded. Participants responded to items on a 5-point scale (1 = a lot like me, 5 = not at all like me). Cronbach’s alpha for this scale was low (.56), which was likely due to the small number of items and fact that two were reverse coded. Negatively worded items and reverse coding have been found to reduce scale reliability and validity (e.g., Barnette, 2000; Schriesheim, Eisenbach, & Hill, 1991). Regardless of the low reliability, need for cognition was kept because factor analysis suggested the items loaded best on one factor and because it was a control variable.

Preliminary Analyses

A correlation matrix of the major variables of interest is presented in Table 1. The data sets were combined for analyses with a dummy variable for study employed to control for study effects. It is important to note that readiness to quit was moderately correlated (p ≤ .01) with all of the behavioral intentions, which is not surprising considering the original test of the readiness to quit measure. Biener and Abrams (1991) provided the significant correlation between readiness to quit and intention to quit as evidence of concurrent validity for their measure (.64 correlation, p < .001). In addition, two of the labels provided on their ladder (10: I am taking action to quit smoking, and 8: I am starting to think about how to reduce the number of cigarettes I smoke a day) are very similar to two of our intentions (“How likely is it that in the next 3 months you will quit smoking completely and permanently”, “How likely is it in the next 3 months you will reduce the number of cigarettes you smoke in a day”). Thus, the similarity between these measures is expected to produce positive and significant relationships. It is for this reason that controlling for readiness to quit smoking was considered crucial to testing PE’s relationship with behavioral intention. If PE is significantly associated with the behavioral intentions when controlling for readiness to quit, it provides evidence that PE explains variance in behavioral intention over and above that due to an individual’s readiness or stage of change.

The correlation matrix also shows moderate to high levels of correlation between the aggregate emotion variables. Four of the variables yielded highly correlated pairs, fear and guilt (r = .92) and hope and pride (r = .83). Because of the high correlations and to avoid collinearity issues, these pairs were summed to create index scores of negative and positive emotional responses, fear-guilt (Cronbach’s α = .84, m = 19.73, SD = 1.64, range: 8–40) and hope-pride (Cronbach’s α = .78, m = 17.39, SD = .87, range: 8–40). Anger was moderately correlated with guilt (r = .55, p ≤ .001) and fear (r = .63, p ≤ .001) and one of the behavioral intentions (r = .07, p = .02). However, because of concerns over the vague wording of the anger item, it was not used in the analyses.²

We conducted an exploratory factor analysis in SPSS 18 using the maximum likelihood procedure to determine the appropriateness of creating an overall behavioral intention scale. The results showed that the three intentions loaded best on one dimension; factor loadings ranged from .83 to .86. Therefore, the intentions were summed to create an overall intention score with a Cronbach’s alpha of .82, demonstrating adequate internal reliability. The mean score was 7.52 with a standard deviation of 2.18 (range: 3–12).

Analysis Plan

In order to fully examine the data, two analysis strategies were used. First, we ran linear regression models with each of the three independent variables (aggregate PE, aggregate guilt-fear, aggregate hope-pride) and the overall behavioral intention score. Second, separate linear regression models were estimated for each behavioral intention, again with each of the three independent variables. The three independent variables could not be included in the same model estimate because of collinearity; aggregate PE was significantly correlated with all of the aggregate emotion evaluations (Pearson’s r ranged from .45 to .86) and all of the aggregate emotion scores were significantly correlated with each other (see Table 1)³. In addition to the control variables listed above, study (Study 1 = 0, Study 2 = 1) and demographic information (age, gender, racial/ethnic identity, education, income) were also included as controls⁴. Race/ethnicity and education were dummy coded for the regressions (European American/White = 0; less than high school education = 0; all other categories were compared to these). The results of both strategies are detailed and presented below. All regression analyses were conducted in STATA 11.

Results

Table 2 contains the results for the three independent variables on each intention including the overall intention score; the table also includes the R² for each model (including all controls). ^5–6 The results of the control variables are only discussed in text. Stata employs listwise deletion by default and resulted in a final N = 1,139. We kept this strategy because the number of missing observations was not great, less than 5% of the possible observations (1,196), and would likely not change results.

Table 2.

Summary of Separate Linear Regressions of the IVs on the Six Behavioral Intentions (with Study as a Control Variable)

IV	Intention (DV)
Aggregate Perceived Effectiveness	Overall Intention	Quit Smoking	Reduce Cigarettes	Talk to Someone
B	.36	.05	.13	.17
SE	.10	.04	.04	.05
β	.10*	.04	.09***	.11***
R2	.41	.33	.36	.26
Aggregate Fear-Guilt
B	.10	.01	.02	.06
SE	.04	.01	.02	.02
β	.07*	.03	.05	.10**
R2	.40	.33	.35	.25
Aggregate Hope-Pride
B	.15	.04	.06	.04
SE	.06	.02	.03	.03
β	.06*a	.04#	.07**	.04 a
R2	.40	.33	.35	.25

Note. Each independent variable was run in a separate regression equation with the same control variables: study, readiness to quit, nicotine dependence, need for cognition, race/ethnicity, age, gender, income, and education.

^aDummy code for study (Study 1 = 0, Study 2 = 1) was significant.

^*p ≤ .05,

^**p ≤ .01,

^***p ≤ .001,

^#p ≤ .12

Demographic Differences

Demographic variables (age, gender, racial/ethnic identity, education, income) were included in all regression models as controls because of the large and diverse samples, and because many public health practitioners are interested in such differences. However, because they are not a primary interest in this study, only the significant differences are reported here. Consistent differences in behavioral intentions were found based on race/ethnicity, gender, and age. For all behavioral intentions, African Americans/Blacks reported significantly higher levels of intention than European Americans/Whites (p-values ≤ .001). Women were significantly more likely than men to report intentions to reduce the number of cigarettes smoked a day (p-value < .001), and age was positively and significantly associated with intention to talk to someone about quitting (p-value < .001).

Overall Intention

The aggregate PE score, aggregate fear-guilt index, and aggregate hope-pride index were each significantly associated with overall intention to engage in smoking cessation behaviors. Readiness to quit was the only control variable significant in each regression model, p < .001.

Separate Behavioral Intentions

Quit smoking completely and permanently

None of the three independent variables were significantly associated with this intention. In each model, study neared significance (p-values ranged from .11 to .15) and readiness to quit was again a significant predictor (p < .001).

Reduce the number of cigarettes smoked a day

Both aggregate PE and aggregate hope-pride were significant. Readiness to quit was positive and significant (p < .001), while need for cognition was negative and significant (p-values ranged from .01 to .04) in all three regressions.

Talk to someone (friend, family member, spouse) about quitting smoking

Aggregate PE and aggregate fear-guilt were significantly associated with intention to talk to someone about quitting. Again, readiness to quit was a significant predictor (p < .001).

Additional Analyses

In several of the regression models, the control variable for study was significant or near significance (p ≤ .15). As a result, the findings may be affected by differences in characteristics of the two studies. Thus, the two data sets were compared on each of the major variables. The aggregate PE scores were significantly different (t = 18.18, p < .05, equal variances not assumed); PE scores were significantly higher in Study 1 (m = 12.20, SE = .02) than Study 2 (m = 11.66, SE = .02). There was also a significant difference in responses to the fear-guilt index (t = 27.20, p < .05, equal variances not assumed); Study 1 again had higher scores (m = 20.81, SE = .06) than Study 2 (m = 18.76, SE = .05). Study 1 also had significantly higher hope-pride scores (m = 17.77, SE = .03) than Study 2 (m = 17.04, SE = .03), t = 16.01, p < .05. However, the two studies were similar in terms of the participant samples including nicotine dependence, readiness to quit, and gender (p-values ranged from .14 to .48).

To better understand the differences between the studies on the independent variables, we re-ran the regression models for each study separately. Because our primary interest is in the most difficult intention, to quit completely and permanently in the next three months, only it and the overall intention measure were re-estimated. The results of the additional analyses are presented in Table 3, which again includes the R² for each model with control variables.

Table 3.

Summary of Linear Regressions by Study

	Intention (DV)
IV	Study 1		Study 2
Aggregate Perceived Effectiveness	Overall Intention	Quit Smoking	Overall Intention	Quit Smoking
B	.38	.05	.37	.06
SE	.13	.05	.15	.05
β	.10**	.04	.08*	.03
R2	.37	.31	.44	.36
Aggregate Fear-Guilt
B	.10	.01	.10	.02
SE	.05	.02	.06	.02
β	.07*	.02	.05	.03
R2	.36	.31	.44	.36
Aggregate Hope-Pride
B	.25	.08	.07	.00
SE	.09	.03	.09	.03
β	.09**	.08*	.02	.00
R2	.36	.31	.44	.36

Note: Each independent variable was run in a separate regression equation (by study) with the same control variables: readiness to quit, nicotine dependence, NFC, race/ethnicity, age, gender, income, and education.

^*p ≤ .05,

^**p ≤ .01

Discussion

We conducted a conservative test of the PE→AE relationship and found additional evidence that PE precedes AE. Using aggregated message scores to predict individual behavioral intentions reduces the possibility that individuals motivated to engage in behavioral changes prior to message exposure also rated those messages as more effective and then reported higher intentions to engage in related behaviors, diminishing the influence of the individual’s motivations prior to message exposure on their perception of the messages. In addition to using aggregate PE scores, we also controlled for individual factors that are known to influence behavioral intentions, such as readiness to quit smoking (which was the most significant predictor variable). Obtaining significant results on specific intentions thus supports the claim that PE precedes AE. Our study also asked specific intention questions framed in a specific time period, included a large sample of smoking cessation PSAs (randomly assigned so no individual saw the same combination of messages), and included a nationally representative sample of adult smokers. Therefore, three issues in previous PE research were directly addressed: causal direction, measures of intention, and representativeness of the samples. It should also be noted that the primary behavioral intention evaluated here is difficult to change with a few anti-smoking ads. Any success in changing smokers’ intentions to quit suggests that PE is a good indicator of message effectiveness.

PE, Emotional Responses, and Smoking Cessation Intentions

PSAs rated as more effective, as judged by both traditional cognitive responses and emotional responses, were associated with intentions to engage in smoking cessation activities, although not uniformly or under all conditions tested. The two most important findings are: (a) PE was positively and significantly associated with the overall intention measure and (b) the hope-pride index was positively associated with intention to quit smoking completely and permanently in the next three months (near significance in the combined data, significant in Study 1). Although significant results for overall intention and intention to quit smoking were only obtained in one of the data sets, it is particularly encouraging as all analyses included several control variables that were known or anticipated to have significant relationships with smoking cessation intentions. In addition to the control variables that made for a conservative test, the wording of the intention measures was particularly strong. Emphasizing that the intention measures refer to permanent and complete behavior change within a short time frame makes them an acceptable and more realistic substitute for measures of actual behavior, which adds to our confidence in the predictive validity of PE.

While intention to quit smoking is the most desirable intention from a public health perspective, it is also the most difficult to obtain and least likely to occur after exposure to a PSA (or several). That PE and emotional responses significantly predicted some of the other smoking cessation-related behavioral intentions is also important from a practical standpoint. PE significantly predicted the other smoking cessation-related behavioral intentions: reduce the number of cigarettes smoked a day in the next three months, and talk to someone (friend, family member, spouse) about quitting smoking in the next three months. These results suggest that PSAs can at least contribute to behavioral intentions, which may be the most we can expect from an anti-smoking campaign (as opposed to expecting outright behavior change).

Other Influences on Smoking Cessation Intentions

In addition to the relationship between PE and smoking cessation intentions, two other patterns emerged. First, readiness to quit had a clear and significant association with all smoking cessation intentions. As previously discussed, this result is not surprising, especially considering how Biener and Abrams (1991) created their smoking cessation intention scale, but it is worth pointing out as this result is consistent with the smoking cessation literature. Second, African American/Black individuals were consistently more likely to intend to engage in these cessation-related behaviors than European American/White individuals. No other racial/ethnic identity was significantly different from European American/White individuals on any of the cessation intentions.

Differences Between the Studies

The results from the two studies yielded inconsistent results. The studies were secondary analyses from existing data sets with the two sets of PSAs selected, originally, for different reasons. The ads used in Study 1 had greater variance in the PE and emotional response scores than those is Study 2. In Study 1, the ads were chosen from a large sample (about 1,000) of anti-smoking PSAs to maximize variance in argument strength and message sensation value. The highs and lows on these factors drove the selection of ads. In Study 2, ads were selected based on message theme (e.g., disease and death, positive frame of quitting, etc.) to fill in the gaps left by Study 1 so that when combined, the variety of message themes was an approximation of the set as a whole. As a result, the three message themes judged to be the least effective in both studies–marketing tactics messages, cosmetics messages, and smokers’ negative life circumstances messages⁷–made up a greater majority of the ads in Study 2. These three themes accounted for approximately 30% of the ads used in Study 2, while comprising 11% of the ads in Study 1. The higher probability of seeing one or more of the less effective themes dropped the PE scores in Study 2 and resulted in less variance among the participants’ responses. In the end, the two studies together yield a set of themes that map, in their frequency, the full set of themes in the 1,000 ads. Therefore, in our view, the results from the full set of combined PSAs are more representative of anti-smoking ads in general.

A strong argument can be made that PE is a good measure of AE. In the combined data, PE predicts overall intention and for each of the three components of intention, the direction is positive for all and near (t > 1.4) or past significance for two of the three individual intention measures. Two of the three intention measures are about quitting or reducing smoking, while the other is about quitting with help, which is known to increase the success rate for quitting by a factor of two (Sutherland, 2002). The two sets of aggregate emotion scores – one positive and one negative – add no variance explained of any consequence to any of the intention measures, aggregate or individual. These findings suggest that emotional characteristics are inconsequential additions to any measures of PE employed here and tap primarily into believability and convincingness.

We would offer one caveat to the above claims. The most important intentions are those linked to reducing smoking directly, with intentions to seek quit help secondary. In the combined data, hope-pride was at or near significance in predicting intentions to quit and reduce smoking, mostly due to the effects of Study 1. These results may seem too inconsistent to be trustworthy, except that some survey studies of intentions to quit smoking have reported substantial increases in variance explained when hope and pride – emotions clearly related to elevated levels of positive self-affirmation – were included as predictors (Cappella, 2007). We would suggest serious consideration of asking questions about pride and hope, which are appeals to positive self-affirmation, in evaluating ads for their effectiveness.

Implications

These results support the use of PE as an indicator of potential behavior change (AE) and demonstrate the utility of a concise closed-ended measure of PE. Although changes in behavior are ultimately the gold standard for successful messages (see Snyder, 2001), messages associated with healthy behavioral intentions are important to identify for two reasons. First, the link between behavioral intention and actual behavior is positive and significant, suggesting that by predicting behavioral intention we can predict actual behavior. As demonstrated in this study, messages that are perceived as more effective are associated with intentions to engage in behaviors consistent with the message advocacy. Even when the focus of the message is to advocate a difficult behavior, like smoking cessation, effective messages affected important intentions like reducing the number of cigarettes smoked a day, while controlling for several key variables. The results of this study provide additional evidence of the important role persuasive mediated messages play in behavior change in combination with well understood factors like readiness (stages of change) and efficacy in health related behaviors.

As has been pointed out by other scholars, tracking and measuring actual behavior is difficult, time consuming, and costly (e.g., Dillard, Weber, et al., 2007; Snyder, 2001). One of the goals of this study was to provide evidence of the utility of a short measure of PE. Based on a measure of argument strength developed and tested by Zhao et al. (2011) and message evaluation by Fishbein et al. (2002), this measure of PE uses only four items and had an adequate internal reliability. Also made clear in this measure is the referent, which has not been consistently specified in previously used measures (Dillard, Weber, et al., 2007) and has been found in some studies to measure multiple referents (Dillard & Ye, 2008). Specifically, the thought items used the language, “the ad put thoughts in my mind about…”, making third-person effects less likely. Using a short and clear measure of PE is advantageous for researchers and practitioners alike, providing accurate evaluations of messages that can be used with a variety of populations (e.g., young, less educated) without taking up valuable time or resources.

Practically, the results of this study provide guidance for researchers, marketers and others who want to evaluate the effectiveness of PSAs. Determining the ads with the highest PE scores will allow evaluators to select the ads with the greatest potential to influence behavioral intentions. In the context of these studies, that means selecting the ads that were associated with people intending to quit smoking completely and permanently, reduce the number of cigarettes they smoke daily, and intending to engage in more than one smoking cessation related activity – all within the next three months. Addressing the pleasant-unpleasant dimension of PE by assessing emotional responses to the ads also aids in selecting the PSAs with the most potential to positively affect intentions. While emotional responses did not result in more variance explained than PE, different emotional responses (positive or negative) were associated with different behavioral intentions. Most important, in this context, messages that elicit feelings of hope and pride may contribute to intentions to reduce smoking behaviors directly (e.g., quitting and reducing the number of cigarettes).

Study Limitations and Strengths

As with any study, there are limitations. Self-reported behavioral intention was used as the outcome measure instead of actual behavior, which would have been more desirable. Due to the time and cost associated with collecting behavioral measures, intention was selected as an adequate, if imperfect, alternative. Self-selection bias is also a limitation of any online questionnaire (e.g., Wright, 2005). While participants were randomly selected from the KnowledgePanel, they ultimately chose whether or not to participate and there may be important differences between individuals who chose to participant and those who did not. The low variance among the PE scores in both studies may have also limited analyses, but of course, the messages selected were not designed to be ineffective. Just the opposite is the case. So the limited variance may reflect our decision to use a range of professionally designed and produced PSAs.

The design of the studies also presents a limitation; they were not designed as true experiments. That is, there was no condition and no pre-determination of strong or weak messages. However, our analysis represents this treatment in a more natural, if somewhat messy, way. Participants received a random combination of four PSAs; they might see four weak or four strong or a combination of weak and strong. Our results suggest that those who receive a combination of the ads rated as most effective have the greatest intention to engage in smoking cessation-related behaviors in the near future. Another approach would be to design and run an experiment that gives participants either all ads rated high in PE or all ads rated low in PE. Finally, the results may be interpreted to be small to moderate effect sizes. In the combined data, standardized regression coefficients ranged from .09 to .11 for aggregate PE on behavioral intentions, and .06 to .10 for aggregate emotional responses on behavioral intentions. However, smoking related campaigns have one of the lowest effect levels on behavior change, mean r = .05, likely due to the addictive nature of cigarettes (Snyder et al., 2004). The fact that we obtained significant results in the context of anti-smoking ads is encouraging for their impact in natural settings. As Snyder et al. (2004) argue, small experimental effects can have substantive effects in a population. In their example, 5% of a city population of 100,000 still equates to 5,000 people changing their behavior. Snyder and colleagues also argue we should not expect health campaigns to result in large numbers of people changing their behavior, but should have more modest expectations.

A primary strength of this study is that it is a conservative test. We controlled for many factors, including readiness to quit, which is clearly the strongest predictor of future smoking-related intentions. We also asked participants to focus on specific intentions and a specific time period, just as Fishbein and Ajzen (2010) recommended. In addition, instead of using each individual’s perception of message effectiveness to predict their future smoking cessation-related intentions, an aggregate score was calculated from the ratings of each individual who saw that message, providing a measure free of individual bias and one that more accurately reflects the persuasive potential of each message. Also, the sample was drawn nationally from adults aged 18 and older and included only individuals who reported being current and regular cigarette smokers. Finally, messages were selected randomly from a large pool instead of being categorized into less and more effective messages and no two individuals saw the same PSAs in the same order.

Conclusion

These analyses were conducted to provide additional evidence of the utility of PE as a predictor of actual effectiveness. Results demonstrated that even when controlling for key behavioral intention predictors, PE was positively associated with smoking cessation-related intentions. This study adds to the literature supporting PE as a measure of a successful message and one that can be used by researchers and practitioners alike to help identify and construct successful health messages.