Applying Quantitative Approaches to the Formative Evaluation of Antismoking Campaign Messages

Abstract

This article shares an in-depth summary of a formative evaluation that used quantitative data to inform the development and selection of promotional ads for the antismoking communication component of a social marketing campaign. A foundational survey provided cross-sectional data to identify beliefs about quitting smoking that campaign messages should target, as well as beliefs to avoid. Pretesting draft ads against quantitative indicators of message effectiveness further facilitated the selection and rejection of final campaign ads. Finally, we consider lessons learned from the process of balancing quantitative methods and judgment to make formative decisions about more and less promising persuasive messages for campaigns.

Introduction

Given the association between smoking, chronic disease, and mortality (Centers for Disease Control and Prevention [CDC], 2012a, 2012b; World Health Organization, 2011), the U.S. Surgeon General’s National Prevention Strategy includes “tobacco free living” as a top priority (National Prevention Council, 2011). Social marketing programs that include media campaigns are one vehicle for delivering antismoking interventions to large numbers of adult smokers (National Cancer Institute, 2008). Included in many of these campaigns are mediated or “promotional” materials used to disseminate messages to target audiences (Luca & Suggs, 2010). In addition to product, price, and place, promotion is a key element of the social marketing mix. Ensuring that promotional materials contain messages that are persuasive before a campaign begins can contribute to the efficacy of a social marketing program overall. Formative evaluation provides an opportunity to assess the potential influence of messages on smokers’ cessation beliefs and behaviors.

While the literature provides examples of both quantitative and qualitative methods for formative selection and pretesting of messages, qualitative methods such as focus groups and in-depth interviews seem to dominate antismoking evaluations (e.g., Bradley, Thorson, Bothner, & Allen, 2000; McCausland et al., 2009). Focus groups have utility because they allow for open discussion of ideas, increasing the possibility of discovering new information related to the outcome of interest that investigators might not have otherwise considered (Hull, Gasiorowicz, Hollander, & Short, 2013). Still, quantitative approaches, such as closed-ended surveys, have other strengths that focus groups do not offer. For example, survey data from representative samples of the target population provide an understanding of how campaign-relevant indicators are distributed in the audience (Fishbein & Ajzen, 2010). Questionnaire data also allow investigators to test which of those indicators are associated with a campaign’s outcome behavior of interest (Niederdeppe, Porticella, & Shapiro, 2012). Finally, closed-ended surveys are an efficient method for collecting data from large, heterogeneous groups to measure perceptions of campaign messages, compare reactions to messages, and reduce bias in estimating message responsiveness (Wakefield et al., 2013).

Although theory-driven approaches germane to quantitative pretesting of message strategies are described in the literature (Hornik & Woolf, 1999; Zhao, Strasser, Cappella, Lerman, & Fishbein, 2011), applications of these approaches for the development of antismoking social marketing campaigns have not been widely published. In response, this paper describes a two-part formative evaluation for the communication component of an antismoking social marketing campaign. The evaluation relied on quantitative data to develop messages promoting the campaign product of quitting smoking with help. The field-tested approaches that we report may serve as useful models for formative evaluation planning in future campaigns.

Campaign Initiative

In 2009, the American Recovery and Reinvestment Act allocated US$650 million dollars for Communities Putting Prevention to Work, an initiative to reduce chronic disease stemming from tobacco use and obesity. As part of that initiative, the Centers for Disease Control and Prevention (CDC) awarded funds to the Philadelphia Department of Public Health (PDPH) for the Get Healthy Philly program. Adult smoking prevalence in Philadelphia ranks highest out of the ten largest U.S. cities (PDPH, 2012). To curb this tobacco use, Get Healthy Philly included a 16-month media campaign, as well as a comprehensive set of other programmatic activities that have been reported previously (PDPH, 2012). Together, the program’s components addressed important social marketing benchmarks, including audience targeting, reducing the costs of quitting with aids, formative investigation to understand audience beliefs and behaviors, and pretesting of intervention messages (Luca & Suggs, 2010). This paper details the formative analysis and message pretesting phases of the media campaign. Program planners from PDPH designed this campaign in partnership with a Philadelphia-based advertising agency and an evaluation team from the Annenberg School for Communication at the University of Pennsylvania.

Low-income and African American smokers living in Philadelphia were selected as the campaign’s target audiences for several reasons. The rate of smoking among low-income Philadelphians (37%; Public Health Management Corporation [PHMC], 2008) is higher than the general Philadelphia population smoking rate (25%; PDPH, 2012; PHMC, 2010); and although the African American smoking rate is similar to that of the general Philadelphia population, African Americans are at a relatively greater risk for mortality and morbidity due to smoking (U.S. Department of Health and Human Services, 1998). Planners further narrowed the target audience to smokers who were ready to quit, speculating that smokers who have no desire to stop smoking might require more extensive interventions than a short-term media campaign can provide.

To develop specific messages for the media campaign, planners considered institutional, social, and individual routes to affect quitting behavior. For example, institutional routes might utilize media advocacy to target secondhand smoke policies, and social routes could use media to change societal expectations about quitting. Individual routes employ direct appeals to influence the audience, such as messages targeting smokers’ attitudes and beliefs about quitting. Compared to individual routes, institutional and social routes can take longer to have effects (Hornik, 2002). The media campaign ultimately targeted individual smokers because it was reasonable to expect that media would reach individuals during the 16-month campaign period.

The media campaign aimed to increase quitting with the help of quit aids, such as nicotine replacement therapy (NRT) or through seeking the advice of doctors. Most smoking cessation attempts happen without help (CDC, 2011) presumably because quitting “cold turkey” can be performed spontaneously. Quitting with help requires more planning, such as a plan for buying NRT or making a doctor’s appointment. Despite the convenience of cold turkey quitting, smokers who try to quit with help are twice as likely to quit successfully relative to smokers who try to quit without any aids (Fiore et al., 2008). Program activities outside of the media campaign attended to the costs associated with quit aid use. Namely, the health department offered two giveaways of free NRT via a telephone quitline, and focused on expanding Medicaid and private insurance coverage of quit-smoking medications (PDPH, 2012). These efforts, and the utility of quit aids, made quitting with help a reasonable, accessible and valuable product to promote to the campaign’s target audience.

Part 1: Foundational Survey

The first phase of formative evaluation provided an opportunity to understand Philadelphia smokers’ cessation and quit aid behaviors and beliefs. This phase also required a methodological approach for selecting potential campaign message strategies. Scholars have argued that beliefs related to the targeted behavior should be the foci of message strategies (Fishbein & Cappella, 2006). Such beliefs are the proximal determinants of behavioral intention according to the Integrative Model of behavioral prediction (Fishbein & Ajzen, 2010), so changing those beliefs through message promotion should in turn change intention and subsequent behavior. This theory of behavioral prediction informed an approach to formative evaluation that takes advantage of cross-sectional quantitative data to select beliefs for promotional messages (Hornik & Woolf, 1999). Niederdeppe, Porticella, and Shapiro (2012) recently applied this approach to identify beliefs that could become message strategies in campaigns to bolster nutrition policy. We used the approach to conduct a telephone-based survey to assess cessation beliefs in a representative sample of the target audience. The data indicated which beliefs are linked to intention to quit with help and should therefore drive campaign message content. Institutional Review Boards (IRBs) at the University of Pennsylvania and PDPH approved all study procedures.

Method

Sample

Adult smokers from Philadelphia County, Pennsylvania were recruited and surveyed over a 3-week period in the summer of 2010. Data collectors from Social Science Research Solutions employed a dual-frame design, randomly generating both landline and cell phone numbers with Philadelphia telephone exchanges. Cell phone respondents completed the interview directly. For landlines, one adult smoker was randomly selected from the household. Eligible participants were 18 years old or older, Philadelphia residents, had smoked at least 100 cigarettes in their lifetimes, and were currently smoking. Of the 4,314 phone numbers successfully contacted, 14.3% (n = 620) were eligible, and 81% (n = 501; 371 landline, 130 cell phone) of eligible participants completed the survey. The final response rate was 37% using the American Association for Public Opinion Research (2006) response rate 3 for calculation.

Measures

Intention to quit with help

The main outcome of interest, intention to quit with help, was a binary measure calculated from a set of intentions to engage in five quit behaviors: using NRT, using prescription medications, going to quit programs, calling quitlines, and seeking quitting advice from doctors. In addition to reporting whether they had heard of and used these quit aids in the past, participants reported their intention to use the aids in the next 3 months. Those who reported that they probably or definitely would use at least one of the five aids were coded as intending to quit with help. Participants who had not heard of a particular quit aid did not receive the respective quit aid intention question and are missing on that intention measure. Participants who reported no intention to quit with help were coded as intending to quit without help if, on a separate question about overall intention to quit, they reported that they definitely will, probably will, or probably will not quit smoking in the next 3 months. The remaining participants who had no intention to quit with help and reported that they definitely will not quit smoking in the next 3 months were coded as having no intention to quit with or without help. While increasing quitting with any form of help was the campaign’s main objective, we also evaluated intention to use each of the five quit aids as separate binary outcomes (definitely not or probably will not use the aid vs. probably will or definitely will use the aid).

Cessation beliefs

Cessation belief items came from multiple sources, including published literature on the predictors of smoking cessation (e.g., McKee, O’Malley, Salovey, Krishnan-Sarin, & Mazure, 2005), results from a telephone survey with young adult smokers (Cappella, 2007), antismoking public service announcements that smokers had rated for effectiveness (Strasser et al., 2009), and members of the evaluation team who specialized in cessation research. The survey included 64 belief items: 25 about quitting, 4 about quitting with “outside help” in general, and 35 specific to the five quit aids. Each participant answered only 38 of these belief items (16 about quitting, 4 about quitting with outside help, and 18 about specific aids), which were randomly assigned to reduce the length of the survey interview while still collecting representative data on a large number of beliefs.

Response options for belief items about using outside help and beliefs specific to each of the five quit aids were dichotomous (e.g., agree vs. disagree. Other belief items used four-point response ranges (e.g., very unlikely to very likely), including positive outcome beliefs about quitting and perceived pressure from “other people” to quit and not smoke in public. These beliefs were highly skewed with at least half of respondents selecting the most pro-quitting category, so dichotomization between the pro-quitting category and the other three categories (e.g., very likely = 1; not very likely = 0) best maximized the size of comparison groups for analysis and simplified the approach to examining associations between beliefs and intentions. The remaining beliefs with 4-point response ranges were about negative outcomes of quitting, self-efficacy to manage quitting experiences, and perceived pressure specifically from friends, spouse/romantic partner, family members, and people in Philadelphia. These were less skewed and were dichotomized at the midpoint (e.g., very likely/likely = 1; unlikely/very unlikely = 0) to ensure that these beliefs could be compared against a common standard during analysis.

Smoking characteristics and demographics

Several questions asked participants about their smoking habits and preferences. These included self-reported addiction (yes, maybe, no), nicotine dependence based on scores from the Fagerstrom Test for Nicotine Dependence (Heatherton, Kozlowski, Frecker, & Fagerstrom, 1991), and number of quit attempts in the past 12 months. Participants also reported their demographic characteristics: gender, race/ethnicity, age, education, and income.

Analysis

The primary analysis followed a methodological approach from Hornik and Woolf (1999), which relies on three criteria to identify the most promising beliefs for campaign messages. Leveraging on the expected association between beliefs, intentions and actual behavior (Fishbein & Ajzen, 2010), the first criterion specifies that beliefs must be highly related to behavioral intention. To assess the strength of each belief–intention relationship, we examined odds ratios (ORs) from separate logistic regression models predicting intentions. Intention to quit with help was regressed on each of the beliefs about quitting and quitting with outside help. We also considered whether belief–intention relationships were moderated by demographics (e.g., race), smoking characteristics (e.g., nicotine dependence), or degree of intention to quit regardless of intention to use help. The moderation analysis tested if the interaction of each of these characteristics with each of the beliefs was significantly associated with intentions. We did not specify moderation hypotheses because the chief purpose of this analysis was to offer insight about which beliefs could be promising across the audience and would not ignore or negatively affect any subgroups, particularly members of the campaign target audiences (African American and low-income smokers).

The second criterion prioritizes beliefs that are not already widely accepted by the population, as indicated by belief distributions, to ensure maximum impact. Additionally, using dichotomous cross-tabulations of the belief and intention measures, we calculated an estimate called percentage to gain, which takes into consideration both the first criterion, strength of belief–intention relationships, and the second criterion, belief distributions. The percentage to gain is the maximum additional percentage of the population that would have the desired intention if the whole population (100%) held the desired belief. Table 1 shows how to estimate percentage to gain. In this example, 65% of the population intends to quit with help overall. However, among people who specifically endorse the belief “I would set a good example for others such as children if I quit smoking,” 77% intend to quit with help. If a campaign message could increase endorsement of this belief to 100% of the population (i.e., persuade the 44% who do not already endorse the belief to adopt it), then we estimate that overall intention rates would shift to match that of current believers (i.e., 77%). The potential percentage to gain on intention is thus 12% (77% – 65% = 12%), which represents the maximum possible change in intention if the relationship between the belief and intention is causal, and the campaign is completely successful in convincing everyone of the relevant belief. While 100% belief endorsement as a result of a campaign is not a reasonable expectation, the percentage to gain metric provides a method of ranking beliefs relative to one another in order to choose beliefs that have the highest potential for increasing intention.

Table 1.

Example of Analysis for Calculating “Percentage to Gain” to Identify Promising Beliefs for Campaign Messages

	Belief: I would set a good example for others such as children if I quit smoking		Overall
	Not very likely	Very likely
Intend to quit without help (column %)	n=52, 50%	n=30, 23%	n=82, 35%
Intend to quit with help (column %)	n=52, 50%	n=101, 77%	n=153, 65%
Total (row %)	n=104, 44%	n=131, 56%	n=235, 100%

The third criterion relies on subjective judgment to decide if beliefs that have room to change in the population, are highly related to intention, and, as a result, have a high percentage to gain are also beliefs that might be influenced by communication campaign messages. The evaluation team first eliminated beliefs that did not satisfy the first set of criteria, and then discussed which of the remaining beliefs could become the basis for plausible campaign messages. This process did not involve a formal rating system but instead considered the opinions of team stakeholders. The advertisers working on the campaign also contributed to this subjective decision process, given their expertise in turning beliefs into convincing story lines.

We evaluated the same criteria for promising beliefs with a second set of dependent variables that measured separate intentions to use each of the five quit aids. This analysis examined associations between beliefs specific to each quit aid (e.g., NRT, doctor advice) and intention to use the respective aid. Only participants who reported that they had heard of the quit aid were available for these analyses. Due to space considerations and the fact that the campaign aimed to increase quitting with any form of help rather than with specified quit aids, findings from this second set of analyses are summarized only in the text.

Weighting methodology

We applied post-stratification probability weights in the calculation of cross tabulations and of standard errors for OR estimates. The weights came from the Philadelphia adult smoker population data from the PHMC’s (2008) Southeastern Pennsylvania Household Health Survey.

Results

As listed in Table 2, 53% of participants had an income of less than US$40,000 per year. Non-Hispanic African American smokers comprised 46% of the sample. Overall, the demographic makeup of the final foundational survey sample looked similar to Philadelphia’s smoking population (PHMC, 2010).

Table 2.

Comparison of Foundational Survey Participant Characteristics and Philadelphia Smoker Characteristics

	Foundational Survey		Philadelphia Smokers
	N		N
Gender (female)	501	56%	1,102	53%
Race	497		1,083
African American		46%		44%
White		42%		43%
Hispanic		8		9%
Other		4%		4%
Age—mean years (SD)	494	48.5 (14.4)	1,102	45.6 (15.1)
Education	501		1,097
Less than high school		24%		19%
High school diploma		34%		44%
Some college		27%		21%
College degree or more		15%		16%
Annual household income ≤ US$40,000	422	53%	881	58%
Nicotine dependence	500		--
Very low		37%		--
Low		30%		--
Medium		14%		--
High		15%		--
Very high		5%		--
Self-reported addiction	500		--
Yes		76%		--
No		16%		--
Maybe		9%		--
Quit attempts in last 12 months	497		1,097
None		58%		44%
1 to 2 times		26%		--
3 or more times		16%		--

Note. Foundational Survey data are unweighted. Philadelphia smoker characteristics came from the Philadelphia Public Health Management Corporation (PHMC, 2010) database.

Intention to quit with help varied across participants (Table 3). Over half of participants (54%) intended to quit with help in the next 3 months, but 33% intended to quit without help. Participants reported more intention to seek advice from their doctors relative to using other forms of help (NRT, prescription medications, quit programs, quitlines). Furthermore, smokers who reported using help in the past most often reported seeking advice from their doctors (24% sought doctor advice in the previous year) despite the rising accessibility of over-the-counter quit aids and the ease of access to free counseling via telephone quitlines. Among those who had heard of and ever used one of the other four quit aids, more than half also sought advice from their doctors (data not shown).

Table 3.

Distribution of Intention to Use Help and to Quit Smoking in the Next 3 Months

	N		Definitely will not	Probably will not	Probably will	Definitely will
Intention to use quit aids (help)
Nicotine replacement therapy	348		37%	29%	20%	14%
Prescription medications	330		52%	27%	14%	7%
Go to a quit program	297		37%	35%	22%	6%
Call a quitline	235		40%	32%	22%	6%
Seek advice from doctors	498		26%	33%	27%	15%
Intention to quit smoking (regardless of use of help)	494		17%	34%	30%	19%
Intention to quit with help	497
Intend to quit with help		54%
Intend to quit without help		33%
Do not intend to quit with or without help		13%a

Note. Data are unweighted. Participants were asked about their intention to use each of the quit aids if they reported that they had heard of the aid.

^aSmokers who reported no intention to quit with or without help were excluded from subsequent analyses.

Individuals who reported that they would not use a quit aid and definitely would not quit smoking in the next 3 months (n = 65, 13%) were excluded from remaining analyses so that observed belief–intention relationships would represent a target audience that is at least somewhat ready to quit with or without help. Non-Hispanic Whites were more likely to meet the exclusion criteria than were people who identified as another race or ethnicity, and excluded individuals reported fewer quit attempts in the past year compared to participants retained for analysis.

Table 4 presents the results for all 29 beliefs about quitting and quitting with outside help. The following cessation beliefs—dubbed the “Four Es”—emerged as the most promising candidates for campaign message themes: (1) Easier—If I tried to quit smoking using outside help, it would make quitting easier for me; (2) Example—I would set a good example for others, such as children if I quit smoking; (3) Energy—I would have more energy to do the activities that I enjoy if I quit smoking; (4) Expense—I would have a lot more money to spend on other things instead of cigarettes if I quit smoking. These beliefs were highly related to intention to quit with help (ORs = 2.52–4.65, ps <.05), discriminating well between those who intend to quit with help and those who did not intend to quit with help. Over one third to one half of participants did not already hold these beliefs, and if campaign messages persuaded those individuals to adopt the belief and it was causally related to intention, we could expect 9% to 14% of Philadelphia smokers to also increase their intention to seek help for quitting (the estimates of potential percentage to gain). In contrast, perceived pressure (i.e., what other Philadelphians think of smoking in public) and self-efficacy did not emerge as promising beliefs.

Table 4.

Beliefs Ranked by Their Potential as Promising Message Targets

		Strength of Association With Intention to Quit With Help		Percentage of Smokers Who Do Not Endorse the Belief	Percentage to Gaina
	N	Odds Ratio	95% CI
1. If I tried to quit smoking using outside help, it would make quitting easier for me	418	4.65*	[2.83, 7.63]	43%	14%
2. I would set a good example for others such as children if I quit smoking	235	3.35*	[1.71, 6.57]	43%	12%
3. If I tried to quit smoking using outside help, I would be more successful than if I quit without outside help	419	3.85*	[2.35, 6.33]	40%	11%
4. I would have more energy to do the activities that I enjoy if I quit smoking	226	2.52*	[1.30, 4.87]	51%	11%
5. I would have a lot more money to spend on other things instead of cigarettes if I quit smoking	228	3.05*	[1.58, 5.87]	36%	9%
6. I would reduce my risk of getting health problems such as heart disease, emphysema, and stroke if I quit smoking	246	2.46*	[1.32, 4.56]	37%	8%
7. I would have more hope for living a healthy life if I quit smoking	214	2.34*	[1.24, 4.40]	41%	8%
8. My close friends have often encouraged me to quit smoking	425	2.88*	[1.76, 4.74]	32%	7%
9. I would do less harm to family and friends because they won’t have to breathe my smoke if I quit smoking	223	1.68	[0.89, 3.18]	56%	6%
10. I would experience intense cravings for a cigarette if I quit smoking	225	2.48*	[1.24, 4.99]	23%	5%
11. I would respect myself more if I quit smoking	221	1.57	[0.83, 2.99]	59%	6%
12. My spouse or romantic partner has often encouraged me to quit smoking	377	1.86*	[1.13, 3.06]	38%	5%
13. Most people important to me would approve if I quit smoking	421	2.09*	[1.25, 3.49]	28%	5%
14. I would gain weight if I quit smoking	231	1.56	[0.84, 2.90]	41%	5%
15. I would reduce my risk of dying at an early age due to lung cancer if I quit smoking	221	1.50	[0.79, 2.84]	49%	4%
16. My other family members have often encouraged me to quit smoking	422	2.02*	[1.19, 3.42]	24%	4%
17. People in Philadelphia don’t want other people to smoke around them	253	1.81	[0.94, 3.49]	27%	4%
18. I would be more irritable or tense if I quit smoking	182	1.57	[0.78, 3.17]	32%	3%
19. I would be less able to concentrate if I quit smoking	219	1.20	[0.62, 2.33]	62%	3%
20. If I tried to quit smoking using outside help, it would be embarrassing for meb	419	2.50*	[1.06, 5.92]	8%	2%
21. People in Philadelphia think that there should be fewer and fewer places where smoking is allowed	293	1.22	[0.69, 2.15]	37%	2%
22. I feel confident about controlling weight gain if I quit smoking	255	1.12	[0.63, 2.01]	56%	1%
23. If I tried to quit smoking using outside help, it would be a sign of weaknessb	421	1.09	[0.54, 2.19]	13%	0.3%
24. I feel confident about maintaining concentration if I quit smoking	238	0.98	[0.55, 1.76]	42%	−0.1%
25. I would breathe more easily within weeks if I quit smoking	235	0.96	[0.53, 1.75]	48%	−0.4%
26. I feel confident about managing intense cravings if I quit smoking	266	0.92	[0.50, 1.68]	62%	−1%
27. I feel confident about avoiding situations where it is difficult to refrain from smoking if I quit smoking	243	0.88	[0.48, 1.60]	63%	−2%
28. People in Philadelphia think that it is unacceptable if people smoke in public open spaces	278	0.61	[0.33, 1.14]	24%	−3%
29. I feel confident about managing irritability if I quit smoking	255	0.56	[0.30, 1.03]	59%	−7%

Note: Data are weighted. Underlined beliefs met the criteria for promising campaign messages. Odds ratios reflect the statistical association between having the desirable belief and having a positive intention to quit with help (vs. having an intention to quit without help).

^aPercentage to gain equals the additional percentage of the population that is estimated to have the desired intention if 100% of the population held the desired belief.

^bReverse coded so that an affirmative response (the undesired response) is the reference category.

^*p <.05

Some beliefs about the benefits of NRTs, such as managing cravings and reducing irritability, were possible candidates for increasing intention to use NRT specifically. However, when examining intentions to use each of the other quit aids (prescription medications, quit programs, quitline, or doctor advice), few beliefs about those aids were promising (data not shown).

Moderation analyses involved 576 separate tests of interactions between each of the 64 beliefs and the 9 moderator variables (demographic and smoking characteristics, and overall intention to quit) in their joint associations with intention to quit with help or to use each quit aid. Twenty-eight (5%) of the interaction tests were significant, but there was no consistency in terms of which characteristics moderated each of these belief–intention associations (data not shown). For example, race (African Americans vs. Whites) moderated only four belief–intention associations, and income (<US$40,000 vs. ≥US$40,000) moderated two associations. Associations between the Four Es beliefs and intention to quit with help did not differ by race or income, a notable set of findings given the campaign’s African American and low-income target audiences. Considering the inconsistency in the moderation results across a large number of tests, we could not make a strong case for using separate belief strategies for different subaudiences in this campaign.

Foundational Survey Discussion

The foundational survey provided campaign planners with useful quantitative evidence from a representative sample of Philadelphia smokers. Evaluators used the data to narrow down which beliefs were most promising as part of a campaign message strategy and which beliefs were least promising. Based on the findings, the evaluation team recommended that messages in Philadelphia’s smoking cessation campaign should focus on beliefs about having more energy, saving money, setting a good example by quitting, and about quitting being easier with help. Moderation analysis indicated that the strong associations between these beliefs and intention to quit with help were no different between the campaign’s target audiences and other groups of smokers that the campaign would not target directly. In other words, a message strategy based on these beliefs could be effective across Philadelphia smokers. While some beliefs about using NRT were promising, the evaluation team ultimately decided that these beliefs were not as promising as the Four Es because the campaign aimed to increase quitting with any type of help rather than quitting with a specific aid. Distributions of actual quit aid use indicated that a few specific quit aids were most familiar to Philadelphia smokers, particularly seeking advice from a doctor. This result suggested that ads should model and encourage the use of these aids while quitting.

We recognize, however, that this quantitative approach to belief selection has limitations. For example, using cross-sectional data to choose preferred focus beliefs assumes but cannot assure a causal relationship between beliefs and intentions. Alternatively, intentions might lead to beliefs, or unmeasured confounders might explain the belief–intention associations. If the associations were not causal, success in changing these beliefs would not be effective in changing intentions (or behaviors). Still, this formative evaluation method offers a robust strategy for reducing uncertainty about which beliefs to consider and which ones to avoid when designing messages for campaigns. The next phase of formative evaluation involved pretesting messages intended to promote the promising beliefs that emerged from the foundational survey.

Part 2: Message Pretesting

Pretesting antismoking messages has traditionally involved showing ads (or mock-ups of ads) to focus groups and using open-ended feedback to make judgments about which ad features are successful or unsuccessful (e.g., Bradley et al., 2000; McCausland et al., 2009). Recently, a 10-country study of responses to antismoking messages included both closed-ended survey items that quantitatively rated individual ads and qualitative ratings from focus groups (Wakefield et al., 2013). The formative evaluation for the Philadelphia campaign similarly pretested candidate messages employing a wide variety of measures of message attributes including engagement, emotional response, and reactance. This data collection approach relied on a web survey experiment to collect smokers’ assessments. This study had IRB approval from the University of Pennsylvania and PDPH.

Based on the findings from the foundational survey, ad developers strove to design a series of radio, television, and print ads to convince smokers that it is easier to quit with help and proposed story lines to tap this campaign theme. Each story line included at least one radio, television, and print ad consistent with the campaign’s goal of rolling out its message across three different media. Storyboards with voice-overs were substituted for fully produced television ads. Print ads were designed for display in buses, subway cars, and convenience stores.

The health department (PDPH) selected four story lines to pretest with adult smokers: (1) Last Pack, (2) Smoking Alone, (3) Success Stories, and (4) Try and Quit. Table 5 includes details on the radio, television, and print ads for each story line. “Last Pack” depicts smokers who want to quit but have a hard time making their next cigarette pack their last. “Smoking Alone” shows the downside of smoke breaks: Missing out on other enjoyable activities. “Success Stories” focuses on smokers who quit despite stressful life circumstances. Lastly, “Try and Quit” illustrates how smokers who quit with help are more successful than those who quit without help. Each ad included the health department’s PDPH logo and information about the Philadelphia Quitline (1-800-QUIT-NOW) and website (SmokeFreePhilly.org).

Table 5.

Candidate Ads Used in Message Pretesting

Story Line	Radio	Television	Print	Voice-Over
Last Pack	A son questions his mother about her smoking while they ride in their car. The mother’s response— that it is her last pack— repeats like a broken record.	Ed requests a pack of cigarettes from a convenience store cashier, claiming that this pack will be his last one. The scene is repeated several times, suggesting that Ed continues to smoke despite his desire to quit. In a final scene, Ed asks for a newspaper instead of a pack of cigarettes and confidently states that he “talked to a support group and got something to ease the cravings” and finally quit smoking.	A woman is shown smoking in three repetitive images, each saying “This is my last pack.” A fourth image shows the woman without a cigarette claiming “That was my last pack.” Tagline: “You are twice as likely to quit for good if you quit with help.”	“You are twice as likely to quit for good if you quit with help. For resources, support, and tools, call 1-800-Quit-Now or visit SmokeFreePhilly.org”
Smoking Alone	A woman confesses that she lied to her family to take a smoke break while shopping with them at the mall. She is also conflicted about whether to finish the cigarette or return to enjoying time with her family.	A man smokes outside in the rain while his friends are watching a Philadelphia Eagles football game on television. He argues with himself about his desire to finish the cigarette even though it is taking time away from an enjoyable activity and putting him in an otherwise miserable situation.	A woman smokes behind a movie theater while other families enjoy themselves in front of the theater. Taglines: (a) “Just because you smoke alone doesn’t mean you have to quit alone.” (b) “You’re twice as likely to quit for good if you quit with help.”	“Just because you smoke alone doesn’t mean you have to quit alone. [Talk to a doctor, call a counselor, learn about treatments. (Radio only)] For resources, support, and tools, call 1-800-Quit-Now or visit SmokeFreePhilly.org”
Success Stories	A male cab driver describes frustrations related to his job, but says that he quit with help after visiting the website.	Ad 1: An older male describes his blue collar job washing windows on high rise buildings. Ad 2: A mother describes her stress from taking care of several young children. Both characters say that they no longer smoke to deal with their stressful situations because they finally quit for good after visiting the website where they got information about how to quit smoking with help.	A customer service agent stands in front of a long line of patrons waiting for his attention. The text says: “This used to call for a cigarette. But I quit with SmokeFreePhilly.org.” Taglines: (a) “If he can do it, you can do it.” (b) “You’re twice as likely to quit for good if you quit with help.”	“If he can do it, you can do it. All you need is a little help. Things like medication, counseling services, and nicotine replacements are safe and can even be free. For resources, support, and tools, call 1-800-Quit-Now or visit SmokeFreePhilly.org.” (Radio) “If [he/she] can do it, you can do it. All you need is a little help. Everything you need to succeed is at SmokeFreePhilly.org or 1-800-Quit-Now.” (TV)
Try and Quit	It seems that a woman’s husband has just left her. She says she will not smoke to cope with her heartbreak because she got help to quit smoking at SmokeFreePhilly.org. She then says her husband tried to quit smoking alone but did not succeed, and had left only momentarily to buy another pack of cigarettes.	(Same as Radio)	Ad 1: A downward facing arrow. The ad was intended for subway cars so that each arrow would point to a different patron on the subway. Tagline: “Can Quit. This person logged on to SmokeFreePhilly.org and got the help they needed to quit smoking.” Ad 2: Tagline only: “Quit Try. There’s a difference when it comes to smoking. Get help, and you’ll see it.”	“A lot of people try and quit, and some succeed. Getting help makes you twice as likely to stick with it. Learn about free counseling, medications, and other resources to help you calm your cravings and quit for good at SmokeFreePhilly.org or by calling 1-800-Quit-Now.” (Radio) “A lot of people try to quit, but getting help makes you twice as likely to succeed. Learn about free counseling, medications, and other resources at SmokeFreePhilly.org or 1-800-Quit-Now.” (TV)

Method

Sample

Pretesting of the ads took place over 4 days in October 2010. Adult smokers (N = 501) were recruited from Survey Sampling International’s national opt-in panel to pretest candidate ads through a web-based survey. To be eligible, participants had to be at least 18 years old and smoke three or more cigarettes on a typical day. A randomized block recruitment design ensured that half of the recruited participants were African American, a campaign target audience. In 2010, half of the Philadelphia smokers were African American (PHMC, 2010), so the recruitment design allowed a comparison between African American and Caucasian subsamples. Participants who reported that they had no thoughts about quitting smoking, measured on an 11-point Stage of Change scale (Prochaska & DiClemente, 1983), were excluded from the analysis (14%) because the campaign targeted smokers who were at least somewhat ready to quit. Compared to participants in the analyzed sample, those excluded were more likely to be White, 50 years old or older, and unemployed.

Testing Procedure

Four radio ads, five television ads, and five print ads were pretested. To ensure that rating differences between individual ads could be attributed to the story lines and not to the type of medium or the order in which different media were presented, each participant only rated ads within one medium (e.g., radio). In this way, ratings of campaign messages were not confounded with the type of medium. Participants were randomly assigned to one of the four conditions in which they were exposed to either radio (n = 102), television (n = 120), or print ads (n = 139); or to a control condition (n = 72) with no ad exposure. Within each treatment condition, participants viewed four randomly selected ads representing each of the story lines in random order.

Before ad exposure, participants reported their number of quit attempts in the past year, number of cigarettes smoked per day, intentions to quit smoking in the next 3 months, and demographic characteristics. Survey instructions then asked participants to review draft ads that the city of Philadelphia might use as part of an adult health campaign. The instructions also directed participants in the radio and television conditions to play a test clip and verify that they could hear the test sound on their computers.

Television ads were approximately 30 seconds long, and radio ads were approximately 1 minute long. Print ads were presented on screen for a minimum of 4 seconds but stayed on screen until the participants clicked to the next screen (for the five print ads, median screen time ranged from 11 to 17 seconds). Participants who reported they were not able to hear or view the ad very well or at all were dropped from the analysis. The final number of participant evaluations per ad ranged from 93 to 120.

Survey Measures

Perceived ad effectiveness and engagement

Participants in treatment conditions answered a series of questions measuring perceived ad effectiveness and engagement immediately following exposure to each ad. The perceived effectiveness scale is comprised of five items measuring an ad’s believability, convincingness, importance to the participant, and the extent to which the ad made the participant think about quitting smoking or about continuing to smoke.¹ To measure engagement, participants reported their agreement with four items that reflected how much they were drawn into each ad (e.g., I could picture myself in the scene of the events of the ad). Perceived effectiveness and engagement items used 5-point response scales ranging from strongly disagree to strongly agree. Investigators have demonstrated that perceived effectiveness affects intentions to quit smoking (Zhao et al., 2011) and getting help to quit (Bigsby, Cappella, & Seitz, 2012) among adult smokers. Engaging ads, or ads with compelling narratives, increase attention to the ad’s content and may prevent counterarguing (Slater & Rouner, 2002).

Emotion, reactance, comprehension, and recognition

Perceived effectiveness and engagement were the primary criteria for assessing each ad’s potential to influence quitting with help, but several other message attributes enriched the evaluation. Some evidence indicates that ads eliciting emotional reactions may improve processing or increase motivation (Dillard & Nabi, 2006), and emotionally evocative smoking cessation ads can increase quitting (Durkin, Biener, & Wakefield, 2009). Participants indicated to what extent they felt four emotions while viewing or listening to each ad: anger about being a smoker, hope, worry, and sadness.

Ads with high reactance (Dillard & Shen, 2005) can increase resistance to ad content. Four items evaluated reactance by assessing participants’ trust in the ad (e.g., How much do you think the information presented about smoking is exaggerated) and anger directed at the ad or its sponsors. Emotion items and reactance items used 4-point response scales ranging from not at all to very much. Finally, treatment group participants reported their comprehension and recognition of ads. Comprehension questions assessed participants’ understanding of the primary ad message and recognition questions assessed encoding of specific lines from the ad. Both sets of questions embedded the correct response among several foils.

Beliefs

After evaluating all four ads for the attributes above, treatment group participants also answered questions measuring their endorsement of the consequences of smoking. The items corresponded to the underlying beliefs promoted in each of the story lines: Last Pack—If I get help to quit smoking, I could finally quit for good instead of trying to quit over and over; Smoking Alone—If I quit smoking, I wouldn’t have to miss out on activities with family and friends; Success Stories—If I get help to quit smoking I could get around any barriers to quitting; and Try and Quit—If I get help to quit smoking, I would be more likely to quit smoking than someone who tries on their own. Response options to each of these items were on 5-point scales ranging from strongly disagree to strongly agree. Four other beliefs that were not directly addressed in any of the ads were included as comparisons. Control group participants also answered belief items allowing a comparison between the control (no ad) and the ad conditions.

Analysis

The analytic approach allowed us to make recommendations about story lines based on the relative performance of each story line in pretesting. Story lines were compared using four criteria: assessment of message attributes, effects on beliefs, how assessments differed across media, and how assessments differed across race and gender subgroups. These comparisons led to one of the following conclusions: The story line is promising for all audiences; the story line could be successful for certain segments of the audience; the story line could be successful if certain qualities were adjusted; or the story line may be harmful to the audience.

In the evaluation of the proposed ads, the goal was to choose the best story line across media, not to choose the best ad within each medium. We first evaluated the relative strength of message attributes (perceived effectiveness, engagement, emotion, reactance, comprehension, and recognition) across individual ads within each story line. We expected small mean differences between ad attributes across story lines because ads targeted similar beliefs (e.g., all related to quitting with help) and used similar formats (e.g., most included a narrative). Therefore, testing for significant differences between these mean scores would potentially offer little information about which ads to select or to avoid within each story line.

To decide if one story line tended to perform better or worse than other story lines on these attributes, we imposed a ranking system, whereby attribute scores for individual ads were ordered within a single medium. For example, perceived effectiveness scores for Last Pack, Smoking Alone, Success Stories, and Try and Quit ads within the radio condition were ordered from the best score (1) to the weakest score (4). In cases where there were two executions of an ad within a single medium (e.g., two print versions of Try and Quit ads), only the better scoring execution was included in the ordering system. The rank of perceived effectiveness and engagement scores were given high priority in the selection of final story lines. Emotion, reactance, comprehension, and recognition scores suggested ways to change a particular ad within a story line if perceived effectiveness or engagement scores for that ad were not high.

Evaluating target belief scores served as a secondary check of the potential effects of ads. Independent samples t-tests statistically compared mean belief ratings for those who saw an ad targeting that belief to the mean ratings of the same belief among control group participants (who did not see any ads). If smokers in the treatment conditions endorsed a target belief more than the control group, it would provide evidence that the ad affected belief change favorably. Conversely, lower belief endorsement in the treatment groups compared to control would signal potential “boomerang effects” (Cho & Salmon, 2007), the unanticipated negative effects of ads that could inadvertently reduce intention to quit with help in the population. We expected that statistically significant belief change after a single ad exposure in a message testing setting would be less likely than after multiple exposures, so statistically significant differences were less important than the direction of those differences. Differences in mean belief scores that were negative suggested boomerang effects, while positive mean differences indicated that boomerang effects were unlikely.

Across the three media, we compared the results of the message attributes analysis and the belief analysis. A final subgroup analysis indicated whether results differed within race (Non-Hispanic African American vs. Non-Hispanic White) or gender groups.²

Results

The analysis excluded 68 participants (14%) because they had no intention to quit smoking. Demographic characteristics of the full sample and the analyzed sample were similar (Table 6). Due to space considerations, scores on the attributes prioritized for ad selection (perceived effectiveness and engagement) and beliefs change results are described in the text, but emotion, reactance, comprehension, and recognition scores are reported only in Table 7. There was not enough variance in negative emotions (anger, worry, sadness) between ads, so we considered only the relative ratings of hopefulness and report these findings in the table.

Table 6.

Message Pretesting Participant Characteristics

	Overall	Control	Radio	TV	Print
N	433a	72	102	120	139
Gender (female)	64%	67%	65%	64%	63%
Race
African American	41%	49%	44%	37%	39%
White	51%	43%	53%	53%	54%
Other	8%	8%	4%	11%	7%
Age—mean years (SD)	44.4 (13.8)	43.8 (15.1)	42.5 (14.1)	46.1 (13.7)	44.7 (12.8)
Education (no college)	38%	35%	47%	35%	37%
Employed	46%	58%	47%	40%	43%
Annual household income <US$40,000	68%	69%	73%	66%	66%
Median daily cigarettes	12.0	14.5	12.0	12.0	10.0
One or more quit attempts in last 12 months	63%	64%	68%	63%	58%
Intend to quit smoking in the next 3 months (vs. do not intend)	60%	54%	64%	57%	63%

^aParticipants who reported no thoughts about quitting smoking (0 or 1) on a stage of change scale (ranging from 0 to10) are excluded from this table and from message pretesting analyses (n = 68, 14%).

Table 7.

Target Belief and Message Attribute Scores by Medium Condition and Story Line

		Last Pack			Smoking Alone			Success Storiesa			Try and Quita
		Radio	TV	Print	Radio	TV	Print	Radio	TV 2	Print	Radio	TV	Print 2
	N	99	93	102	100	98	111	100	94	105	98	95	101
Perceived effectivenessb   1–5 scale	Mean (SD)	3.951 (0.71)	3.911 (0.72)	3.701 (0.82)	3.624 (0.82)	3.822 (0.73)	3.692 (0.82)	3.802 (0.69)	3.813 (0.84)	3.533 (0.73)	3.773 (0.72)	3.744 (0.79)	3.474 (0.83)
Engagementb   1–5 scale	Mean (SD)	3.951 (0.92)	3.761 (0.94)	3.471 (1.01)	3.403 (1.09)	3.652 (1.00)	3.462 (1.00)	3.314 (1.00)	3.194 (1.19)	3.273 (1.04)	3.482 (1.05)	3.443 (1.01)	NA
Hopeful   1–4 scale	Mean (SD)	2.623 (1.11)	2.781 (0.95)	2.622 (1.04)	2.394 (1.15)	2.484 (1.05)	2.651 (1.08)	2.751 (1.10)	2.733 (1.10)	2.363 (1.07)	2.692 (1.08)	2.752 (0.98)	2.294 (1.08)
Reactancec   1–4 scale	Mean (SD)	1.341 (0.69)	1.311 (0.51)	1.332 (0.49)	1.494 (0.78)	1.414 (0.78)	1.393 (0.64)	1.443 (0.72)	1.362 (0.57)	1.444 (0.64)	1.352 (0.66)	1.393 (0.61)	1.291 (0.47)
Comprehensiond	% correct	791	763	732	782	782	891	704	851	723	773	764	694
Recognitiond	% correct	801	704	723	792	841	891	704	822	782	792	773	654

Note. NA = Participants did not rate the engagement of “Try and Quit” print ads because these ads did not include any characters with whom to engage. Scores on each attribute are ordered within each medium, with a superscript indicating the order (1 = best score; 4 = weakest score). Scores ranked as 1 or 2 are shaded with the darkest shading indicating scores for the prioritized attributes.

^aTwo versions of television ads for “Success Stories” and print ads for “Try and Quit” were tested; we present only results from the better scoring execution. TV 2 = A mother describes her stress from taking care of several young children; Print 2 = Tagline only (see Table 5 for ad descriptions).

^bPerceived effectiveness and engagement attributes were prioritized when making recommendations about ads and story lines.

^cLower mean scores indicate better ad performance on the reactance scale.

^dComprehension and recognition measures asked participants to select the correct response from a list that included other foils.

Among the prioritized attributes, results favored ads in the Last Pack story line (see Table 7). Perceived effectiveness scores ranked highest for Last Pack radio (M = 3.95, SD = 0.71), television (M = 3.91, SD = 0.72), and print ads (M = 3.70, SD = 0.82) compared to ads in each respective medium in the other story lines. Last Pack engagement scores for radio (M = 3.95, SD = 0.92), television (M = 3.76, SD = 0.94), and print ads (M = 3.47, SD = 1.01) also ranked highest. Mean ratings of the Last Pack targeted belief, If I get help to quit smoking, I could finally quit for good instead of trying to quit over and over, were higher in the three treatment groups than in the control group, though the differences were not significant (mean difference between radio and control = 0.12; mean difference between television and control = 0.12; mean difference between print and control = 0.13). The positive direction of the differences in belief scores indicated that Last Pack ads did not have boomerang effects.

Smoking Alone also performed well relative to the remaining two story lines. Perceived effectiveness and engagement scores for the television and print ads in Smoking Alone ranked second relative to respective scores on the Last Pack television and print ads. However, the Smoking Alone radio ad ranked low on both perceived effectiveness (M = 3.62, SD = 0.82) and engagement (M = 3.40, SD = 1.09) compared to other radio ads. Smoking Alone ads significantly increased ratings of the target belief, If I quit smoking, I wouldn’t have to miss out on activities with family and friends among subjects in the radio and television conditions compared to the control group (mean difference between radio and control = 0.43; mean difference between television and control = 0.48), and belief ratings were also higher (but nonsignificant) in the print condition than in the control group (mean difference between print and control = 0.29). We deemed Smoking Alone an acceptable story line if improvements were made to the radio ad.

Most of the ads in the Success Stories and Try and Quit story lines ranked lowest on the prioritized message attributes with two exceptions. The Success Stories radio ad had the second highest perceived effectiveness score out of all radio ads (M = 3.80, SD = 0.69), and the Try and Quit radio ad had the second highest engagement score relative to other radio ads (M = 3.48, SD = 1.05). Compared to the control group, ratings of the target belief, If I get help to quit smoking, I would be more likely to quit smoking than someone who tries on their own, were significantly higher among treatment group respondents who saw the Try and Quit radio and television ads (mean difference between radio and control = 0.31; mean difference between television and control = 0.38), and ratings were also higher (but nonsignificant) among those who saw the print ad (mean difference between print and control = 0.30). Treatment group respondents who saw any of the Success Stories ads also reported higher endorsement of the belief, If I get help to quit smoking I could get around any barriers to quitting (mean difference between radio and control = 0.13; mean difference between television and control = 0.17; mean difference between Print and control = 0.08). None of the differences in Success Stories belief scores were significant. Due to the low scores on perceived effectiveness and engagement, we did not recommended going forward with Success Stories or Try and Quit even though belief results for these story lines did not indicate boomerang effects.

The overall patterns of results were substantively consistent within male and female subgroups, leading to similar story line recommendations for either group. The findings also indicated similar performance of the radio and television ads among African American and White subgroups. Print ad rankings were nearly consistent for the two races with one exception: African Americans viewing the print ads unexpectedly endorsed the target beliefs less than African Americans in the control group, while White smokers showed the expected effect (results not shown). The Last Pack and Smoking Alone print ads showed the most substantial unintended effects for African Americans. Message attribute ratings of the print ads suggested how to improve these ads to eliminate the potential disparity in ad effectiveness. For example, the message in the Last Pack print ad (Figure 1) might have been unclear based on its relatively lower comprehension and recognition scores among African Americans compared to Whites. We recommended adjusting the ad to model how to quit smoking with help more explicitly.

Figure 1.

Draft of the “Last Pack” print ad.

Message Testing Discussion

The message pretesting phase was critical to understanding which story lines were most likely to be effective in campaign ads. Perceived effectiveness and engagement ratings have been shown to reliably predict actual effectiveness of ads on outcomes (Bigsby et al., 2012; Slater & Rouner, 2002; Zhao et al., 2011). The findings on these and other attributes informed our recommendations about which story lines might perform better than others for both the overall population and within race and gender subgroups. These story lines were Last Pack and Smoking Alone. Results of the belief change analysis increased our confidence that the television and radio ads in these story lines would not cause unintended boomerang effects, but the Smoking Alone radio ad required revision, as did print ads from both story lines for the African American target audience. After considering all of the findings, campaign planners from PDPH chose Last Pack for the campaign, but decided to change the print ad in this story line.

The message pretesting design had some limitations. First, the target beliefs addressed in the ads did not always match the promising beliefs from the foundational survey. Therefore, it was not possible to assess whether these ads had positive, negative, or no effects on one or more of the Four Es beliefs. Furthermore, target beliefs were measured at the end of the survey, after viewing four ads. As a result, four ads had the potential to influence a particular target belief, not just the ad designed to promote that belief. Measuring beliefs directly following only the first ad would have clarified the results of the belief tests, but this method would have been subject to test sensitization effects.

Finally, participants had to evaluate more than one ad on all of the message attributes so there would be enough observations per ad to allow for serious interpretation of the results. In this design, exposure to earlier ads could have influenced evaluations and perceptions of later ads. Ads were presented in random order to reduce the concern of possible order effects. Ideally, each participant would have evaluated a single ad, but recruiting a sample large enough for this kind of design was cost prohibitive.³

Conclusion

This paper demonstrates the use of quantitative methods to guide the formative evaluation of antismoking media campaign messages. The foundational survey identified four promising beliefs to promote in campaign messages. Message pretesting indicated that one of four story lines, Last Pack, yielded higher perceived effectiveness and engagement among adult smokers than did other story lines, and showed little risk for boomerang effects on beliefs about quitting. Evaluators executed these two phases of formative evaluation in a relatively short period of time before the campaign launched.

Several lessons gleaned from this formative evaluation could facilitate future evaluations that use similar methodological approaches. For example, in the foundational survey, some belief items referred to quitting in general rather than quitting with help specifically. It is possible that the observed relationships between beliefs and intention to quit with help would differ if participants reported their beliefs about quitting with help and not about quitting in general. Ensuring that candidate beliefs align directly with the target behavior of interest would strengthen the face validity of formative evaluation findings for future campaigns. Evaluation planners should try to reach consensus about the target behavior for beliefs before developing a foundational survey.

Generating and analyzing results from a foundational survey constitute a first step in a comprehensive formative evaluation. Evaluators must also craft interpretable recommendations based on these findings, which advertisers can then translate into attractive story lines. Regardless of the creative direction of ads, story lines should convey messages promoting beliefs that have the most potential to influence behavioral intention in the population, that is, beliefs identified in the foundational analysis. Out of four story lines pretested for the Philadelphia campaign, only Try and Quit directly dramatized the belief that quitting is easier with help. Other story lines mainly emphasized that quitting is hard and used final voice-overs or taglines to indicate that quitting with help is easier, rather than modeling quitting with help specifically. No story lines focused on the remaining Four Es that emerged from the foundational survey analysis—energy, expense, and example—including the Last Pack story line, which was ultimately selected for the campaign. Pretesting alternative messages promoting these other beliefs and more fully integrating “help” into the narrative may have yielded different recommendations about the best ads for this campaign. The disconnect that occurred between the formative evaluation findings and several of the concepts produced for message pretesting is undesirable from an evaluator’s standpoint, but it does not undermine the quantitative methods used to identify promising message strategies or to test the effectiveness of potential ads. Instead, the disconnect represents a realistic dynamic between program planners and advertisers as they try to balance the empirical evaluation process with the creative process used to make persuasive ads. Ultimately, planners should anticipate the need to articulate and emphasize the message strategies that advertisers should target.

Other lessons learned concern resources for formative evaluation. Message pretesting is truly an iterative process and resources should be allocated with that in mind. As noted above, the Last Pack print ad used in pretesting needed adjustment, so the ad agency created several possible replacements. The final ad that aired was considerably different from the pretested ad. If more time (and money) were available to pretest the advertisers’ new set of proposed ads, planners could have relied on empirical data findings rather than intuition to decide on the final print ad.

To conserve funds, social marketers could use the message pretesting procedure to choose from existing ads rather than producing and selecting new ads (Cotter et al., 2010). The most promising belief items from a foundational survey could be aligned with target beliefs in ads that are already available for use, such as antismoking ads from CDC’s Media Campaign Resource Center (CDC, 2012c). If more than one potential ad promoted the same belief, message pretesting could determine which of those ads were most effective. This approach would be considerably more cost-effective than developing entirely new ads, especially in domains that have previously done a lot of work to create ads.

Finally, the foundational survey and message pretesting results are not intended to dictate the final design of ads, that is, the characters, visuals, and final scripts that make ads attractive. Rather, these approaches provide systematic evidence about which messages strategies and story lines within an ad are most viable and which ones could fail. This evidence, combined with careful judgment, allows planners to make decisions about how to produce successful ads for the communication components of social marketing campaigns.