Perceived Message Effectiveness Measures in Tobacco Education Campaigns: A Systematic Review

Abstract

Target audience ratings of the likely impact of persuasive messages, known as perceived message effectiveness (PME), are commonly used during message development and selection. PME is also used to examine receptivity of messages after they are fully developed or deployed. Despite this, we know little about the conceptual and methodological characteristics of extant PME measures used in the literature. We conducted a systematic review of tobacco education video, print, and audio campaign studies to examine conceptual and methodological characteristics of PME measures. One hundred twenty-six PME measures from 75 studies conducted in 21 countries with more than 61,000 participants were reviewed. Results indicated considerable variability in measures’ focus on general perceptions of a message (i.e., message perceptions) versus perceptions of expected message effects (i.e., effects perceptions). Considerable variability was also found on underlying persuasive constructs, use of referents, and referencing of behavior in PME items and measures. We conclude with several recommendations for future research on PME measurement and validation.

Health communication campaigns are an effective behavior change tool that can impact a broad range of health behaviors (Noar, 2006; Snyder & LaCroix, 2013; Wakefield, Loken, & Hornik, 2010). Prior to launching campaigns, formative work is often conducted to develop and test messages before dissemination. During this process, a critical decision is selecting which messages to use. An approach that has become increasingly common for message selection is the use of target audience ratings of perceived message effectiveness (PME) (Bigsby, Cappella, & Seitz, 2013; Dillard, Weber, & Vail, 2007; Yzer, LoRusso, & Nagler, 2015). This approach is one in which target audience members rate messages on PME and those ratings are used to select messages for use in a campaign. PME is also used as a metric of message receptivity after messages are fully developed or deployed (Duke et al., 2015). However, PME has been conceptualized and measured in many different ways, resulting in a significant problem. How can campaign developers rely on PME ratings for message selection or reception if it is unclear what PME is and what varying PME measures assess? An important step toward bringing clarity to the PME literature is to understand the nature of extant measures, as such an understanding could inform a clearer conceptualization of and recommendations for future PME measurement and validation work. Thus, to advance this growing area of research, we conducted a systematic review of PME measures in tobacco education campaign studies. We chose tobacco education campaigns because they are one of the longest standing campaign literatures and one in which PME has been widely applied.

Actual vs. Perceived Message Effectiveness

A message is effective if it achieves its intended goal. In many health communication campaigns, this goal is to change mediators of behavior - such as attitudes, beliefs, norms, selfefficacy, and intentions - and, ultimately, behavior itself. Message effectiveness is the extent to which message exposure leads to changes in these outcomes. In field trials and experimental work, this is often referred to as actual message effectiveness (Bigsby et al., 2013; Dillard, Weber, et al., 2007). For example, a test of messages designed to increase self-efficacy to quit smoking might examine whether the messages did exactly that - i.e., increased participants’ self-efficacy to quit. Importantly, such an effectiveness test would not be concerned with whether participants perceived the messages as increasing their self-efficacy (Tharp-Taylor, 2012; Wong & Cappella, 2009) but only whether participants’ self-efficacy actually increased.

While message effectiveness refers to actual changes in outcomes, PME is concerned only with perceptions of the likely effectiveness of messages. Broadly speaking, any participant’s judgement of a message that implies effectiveness could be considered relevant to PME. For example, perceiving that a message is believable does not necessarily mean that the message will have an effect, although this could be a necessary (though perhaps not sufficient) condition for effectiveness. Similarly, perceiving that a message is memorable, interesting, or meaningful does not directly equate to effectiveness, but these kinds of perceptions could contribute to message effectiveness. Indeed, such message perceptions are used as the basis for some PME measures (Biener, Ji, Gilpin, & Albers, 2004; Duke et al., 2015; Kim, 2006), and this approach is similar to what Dillard et al. refer to as message attributes in their conceptualization of PME (Dillard, Weber, et al., 2007).

A different class of perceptions, however, are concerned with whether one thinks a message would have effects. For example, the perception that a message would make one think about the health risks of smoking (Mowery, Riedesel, Dreher, Schillo, & Saul, 2016); the perception that a message would increase one’s confidence in quitting smoking (Wong & Cappella, 2009); or the perception that a message would motivate one to not smoke (Donovan, Jalleh, & Carter, 2006). These are also possible elements of PME, given that they are concerned with perceptions of a message’s effectiveness. Such effects perceptions are used as the basis of many PME measures (Bigsby et al., 2013; Brennan, Durkin, Wakefield, & Kashima, 2014; Huang, Friedman, Lin, & Thrasher, 2016), and are similar to what Dillard et al. refer to as message impact in their conceptualization of PME (Dillard, Weber, et al., 2007).

These examples illustrate that many judgements of a message itself or a message’s likely effects could serve as a conceptual basis for PME. While the measures mentioned above have tended to focus on message perceptions or effects perceptions as their theoretical territory, several measures have included items representing both types of perceptions in the same measure (Jasek et al., 2015; Mowery et al., 2016; Wakefield et al., 2005). PME has also been assessed using single item (Biener, 2000; Stewart et al., 2011) as well as multiple-item scales (Jasek et al., 2015; Lee, Cappella, Lerman, & Strasser, 2013), and as a one-factor (Bigsby et al., 2013; Davis, Nonnemaker, Duke, & Farrelly, 2013) or multiple-factor (Brennan et al., 2014; Dillard & Ye, 2008) phenomenon. While some variability in PME measures may exist because researchers are testing messages with different goals, a larger issue seems to be the considerable ambiguity around the conceptualization of PME (Yzer et al., 2015). PME has been conceptualized as perceptions of ad favorability (Brennan et al., 2014), message quality (Choi & Cho, 2016), and persuasiveness (Dillard, Weber, et al., 2007), to name a few. Given these varying conceptualizations, and given that many PME measures rely on intuitive rather than theoretical conceptualizations (Dillard, Shen, & Vail, 2007), it is perhaps unsurprising that PME has been assessed in a myriad of ways (Choi & Cho, 2016; Dillard, Weber, et al., 2007; Yzer et al., 2015).

There is also some debate regarding the extent to which PME is or could be diagnostic of actual effectiveness - i.e., does PME provide data that guides message designers toward more effective messages? O’Keefe (O’Keefe, 2018) recently conducted a meta-analysis in which he examined the consistency between mean ratings of PME and actual effectiveness of messages as tested in separate samples. Comparing 151 message pairs derived from 35 studies, he found that use of PME would only result in choosing a more effective message 58% of the time, which is little better than chance. Dillard and colleagues examined the association between PME and attitudes and found a correlation of r=.41, also suggesting a fair level of disconnect between perceived and actual effectiveness (Dillard, Weber, et al., 2007). There is evidence, however, that PME is associated with changes in behavioral intentions in both cross-sectional (Bigsby et al., 2013) and longitudinal (Brennan et al., 2014; Davis et al., 2013) studies. In addition, at least two studies have found PME to be longitudinally associated with changes in smoking behavior (Brennan et al., 2014; Davis et al., 2016). This mixed evidence may be due - at least in part - to a lack of clear conceptualization and measurement that may cloud the true PME-message effectiveness relationship.

Clearly, the fragmented PME literature is in need of review and synthesis before recommendations for future work can be made. In the current review, we took a broad approach to describing the current landscape of PME measures, using two conceptual tools to guide our work. First, we used Dillard et al.’s (2007) definition of PME, i.e., an estimate of the degree to which a persuasive message will be favorably evaluated - in terms of its persuasive potential - by recipients of that message. Second, we adapted McGuire’s persuasion model (McGuire, 1989) to operationalize “persuasive potential” and allow for categorization of items into persuasive domains. McGuire posited a sequential process by which communication has impact, including a hierarchy of categories ranging from attention, receptivity, comprehension, generating cognitions, acquiring skills, and agreement with the message’s position, to decision to act and post-message actions. While McGuire was focused on the effectiveness of messages, in the current review we are focused on their perceived effectiveness. We thus argue that the persuasive constructs from our adapted version of McGuire’s persuasion model can be identified within existing PME measures, and PME items can be usefully classified for each persuasive construct.

In addition to examining which persuasive constructs PME measures assessed, we also examined two measurement features of interest: 1) the extent to which PME measures made use of referents; and 2) the extent to which PME measures referenced the product or behavior - in this case, cigarettes or smoking. Some commonly used measures in the literature use neither referents nor reference cigarettes or smoking, and such measures tend to be those that focus on message perceptions (e.g., this ad was powerful) (Davis et al., 2016). However, other commonly used measures do use a referent person (e.g., me) and reference cigarettes or smoking, and these measures tend to focus on effects perceptions (e.g., the ad made me feel motivated to try to quit smoking) (Brennan et al., 2014). PME measures that use referents may reference the self or some other individual or group (e.g., best friend, people my age, smokers), while referencing the behavior can be achieved by including content about smoking or tobacco use in PME items.

To characterize PME measures in the tobacco education campaigns literature, we conducted a systematic review of the literature. Our goal was to examine how PME has been measured in studies of tobacco education advertisements in order to characterize the landscape of PME measures and provide recommendations for future conceptualization and measurement of PME.

Method

Search Strategy

We used a comprehensive search strategy to locate studies relevant to this systematic review. The search strategy involved four steps. First, we searched PsycINFO, PubMed, Web of Science, Business Source Premier, Communication and Mass Media Complete, CINAHL, Global Health, and Scopus computerized databases in October of 2016. We paired several PME-related terms such as perceived effectiveness, message effectiveness, and advertising effectiveness with smoking terms, such as cigarette, tobacco, smoking. Second, we searched both Google and Google Scholar using the same sets of search terms, examining the first 100 results from each search. Third, we examined the reference sections of four narrative reviews examining PME or anti-tobacco advertising studies (Choi & Cho, 2016; Dillard, Weber, et al., 2007; National Cancer Institute, 2008; Yzer et al., 2015). Finally, we searched the reference lists of the final set of articles included in our review.

Our review had four inclusion criteria: 1) a study had to use an experimental protocol to test anti-smoking video, print, or audio advertisements; 2) participants had to view advertisements directed at reducing cigarette smoking behaviors; 3) a study had to measure and report quantitative data on PME; and 4) the full study report had to be available in English.

The initial database search yielded 4,377 references, and additional searches yielded 173 references. After removing duplicates, 2,212 references remained. Two trained reviewers independently screened study titles, reducing the number to 528. They then reviewed abstracts, reducing the number to 226. During this process, references were only discarded if both reviewers agreed that a given reference was not relevant to the review. The full text of the remaining 226 articles was then located and screened by the two reviewers, and reasons for study exclusion were tracked. If the two reviewers made a different determination about any of the articles, they consulted a third reviewer to resolve the discrepancy and make a final determination. This process identified 73 articles reporting on 75 studies for the systematic review (Figure 1).

Figure 1.

PRISMA flow diagram showing the study screening process

Article Coding

Two coders independently coded all studies on sample population characteristics (e.g., gender, age group, smoking status), study design characteristics (e.g., recruitment, data collection mode, use of theory), message characteristics (e.g., message themes, products mentioned in messages, message channels), and PME measurement characteristics (e.g., perceptions of messages vs. effects, PME persuasive domains, number of PME measures, number of items in each measure, coefficient alpha, use of referents, referencing of product/behavior, and response scales). All discrepancies that arose during the coding process were resolved through discussion between the two coders and a third reviewer. Krippendorf’s alpha ranged from a low of .79 to a high of 1.0 (percent agreement 89% to 100%). Most categories had perfect agreement.

PME Domains and Constructs

Using both theory and our preliminary review of the empirical literature as a guide, we developed a rubric of persuasive categories and constructs, based on McGuire’s persuasion model (McGuire, 1989). We adapted McGuire’s persuasion model to arrive at nine categories, some of which feature multiple persuasive constructs (see Table 1). This rubric represents a fairly exhaustive examination of potential perceived effectiveness categories and constructs used in the literature, and provided a foundation for our coding of PME measures. For each construct, we indicate whether it tends to be a message perception or an effects perception.

Table 1.

Potential Perceived Message Effectiveness (PME) Categories, Constructs, Definitions, Perception Types, and Example Items

Categories and Constructs	Definition Perception that an ad...	Perception Type	Example Item
Tuning in
Attention	attracts attention	Effects perception	This ad got my attention. (Johnson, 2016)
Receptivity
Credibility	is believable	Message perception	How believable was the message in this ad? (Donovan, Jalleh, & Carter, 2006)
Interest	is interesting	Message perception	The extent to which you found this ad interesting? (Biener, McCallum-Keeler, & Nyman, 2000)
Personal relevance	is personally meaningful	Message perception	The advertisement was relevant to me. (Guillaumier et al., 2015)
Liking	is good or bad	Message perception	Overall, I thought this ad was a very good antismoking ad. (Jonsdottir, Holm, Poltavski, & Vogeltanz-Holm, 2014)
Memory/recall	is memorable	Message perception	This ad was worth remembering. (Duke et al., 2015)
Comprehension
Perceived comprehension	is easy to understand	Message perception	The ad was easy to understand. (Murukutla et al., 2015)
Perceived knowledge	is informative	Message perception	This ad was informative. (Davis et al., 2013)
	teaches one something new	Effects perception	I learned something from the commercial (Bauman et al., 1988)
Generating cognitions
Cognitive elaboration	would make one think about the health risks of smoking or about quitting	Effects perception	How much did the ad make you think about the risk to your health because of smoking? (Wong, Ho, Cappella, Strasser, & Lerman, 2008)
Acquiring skills
Self-efficacy	would boost one’s confidence or provide skills to quit smoking	Effects perception	Watching this ad helped me feel confident about how to best deal with smoking. (Bigsby et al., 2013)
Agreement with the message’s position
Agreement	is something one agrees with	Message perception	I agree with the ad. (Lee et al., 2013)
Argument strength	is convincing or persuasive	Message perception	This ad was convincing. (Cappella & Baek, 2011)
	gives one convincing reasons not to smoke	Effects perception	Did the ad give you good reasons not to smoke? (Murukutla et al., 2015)
Concern/worry	would lead to worry or concern about smoking	Effects perception	This ad made me feel concerned about my smoking. (Brennan, Durkin, Wakefield, & Kashima, 2011)
Decision to act
Motivation to act	would motivate or urge one to take action about their smoking behavior	Effects perception	This ad makes me want to quit smoking. (Niederdeppe et al., 2011)
Post-message actions
Interpersonal communication	would stimulate one to talk about the message	Effects perception	I will talk with my friends about this ad. (Murphy-Hoefer et al., 2008)
Information seeking	would lead one to seek out further information	Effects perception	This ad makes me want to seek out more information about quitting (not assessed in this set of studies)
Overall message evaluation
General effectiveness	is generally “effective”	Effects perception	I think this is an effective antismoking ad. (Pechmann, Zhao, Goldberg, & Reibling, 2003)
Other outcomes
Other	Other message perceptions	n/a	This ad was difficult to watch. (Jasek et al., 2015); This ad talked down to me. (Vardavas, Symvoulakis, Connolly, Patelarou, & Lionis, 2010); This ad was irritating/not irritating. (Jung & Jorge, 2011)

Note. Message perception items tend to ask participants about their perception of the message itself, while effects perception items tend to ask participants about their perception of whether a message is likely to have a particular effect.

The first category (tuning in) concerns message recipients’ perception that a message would attract attention. The second category (receptivity) concerns the ability of a message to resonate with participants at a gut level, including the extent to which a message is perceived as credible, interesting, good/bad (i.e., liking), personally relevant, and memorable. The third category (comprehension) concerns recipients’ perception that a message is understandable and the extent to which it is perceived as teaching something new. The fourth category (generating cognitions) involves recipients’ perception that the message would result in issue-relevant thinking, such as thinking about the health risks of smoking, while the fifth category (acquiring skills) concerns perceptions that a message increases self-efficacy or skills. The sixth category (agree with message’sposition) concerns the extent to which a message recipient agrees with the message, finds it convincing or persuasive, and views the message as one that would cause concern about what was featured in the message (e.g., the dangers of smoking). The seventh category (decision to act) concerns recipients perceptions that a message would motivate them to take action, such as to not smoke or to quit smoking. The eighth category (post-message actions) concerns actions that may take place after viewing the message, such as the perception that a message would stimulate one to talk about it or seek out further information. Finally, the ninth category (overall message evaluation) concerns recipients general perception of the effectiveness of a given message. An “other” category (other outcomes) provided a place for scale items that did not fit into the aforementioned categories.

Results

The 75 studies were published between 1983 and 2016, with a median publication year of 2011 (see Tables 2 & 3 and online supplement). The number of studies using PME increased over time, with just three studies before the year 2000, 30 studies between 2000 and 2009, and 42 studies from 2010–2016 (Figure 2). Studies had a cumulative sample size of N = 61,015 and were conducted in 21 different countries. Most studies were conducted in the United States (79%), followed by Australia (9%) and China (5%). While the majority of studies were conducted in a single country, five studies were conducted across multiple countries (Durkin, Bayly, Cotter, Mullin, & Wakefield, 2013; Murukutla, Bayly, Mullin, Cotter, & Wakefield, 2015; Perl et al., 2015; Wakefield et al., 2011; Wakefield et al., 2003).

Table 2.

Participant and Study Characteristics in the 75 Studies

Variable	k	%
Participant Characteristics
Age groups1
Adults	38	51
Adolescents	26	35
Young Adults	12	16
Children	1	1
NR	1	1
Smoking status1
Smokers	59	79
Non-smokers	32	43
Susceptible/experimenters	16	21
Former smokers	3	4
NR	1	1
Study Characteristics
Country1
United States	59	79
Other countries (Australia, Bangladesh, Britain, China, Egypt, Greece, India, Indonesia, Kenya, Mexico, Nigeria, Philippines, Russia, Senegal, South Korea, Spain, Taiwan, Tunisia, Turkey, and Vietnam)	17	23
Recruitment1
Online	21	28
College/university	15	20
High school	13	17
Market research firm	8	11
Print advertising	7	9
Middle school	6	8
Local organizations	4	5
Other	8	11
NR	1	1
Sampling
Convenience	59	79
Probability	15	20
NR	1	1
Data collection mode
Computer survey	36	48
Paper survey	23	31
In-person interview	3	4
NR	13	17
Experimental design
Single group (within)	40	53
Multiple groups (between)	35	47
Used theory
Yes	55	73
No	20	27
Theories used2
Elaboration Likelihood Model	13	24
Reactance Theory	9	16
Reasoned Action Approach	7	13
Prospect Theory/Regulatory Focus/Framing	7	13
Limited Capacity Model	5	9
Activation Model of Information Exposure	4	7
Protection Motivation Theory	4	7
Transtheoretical Model/Stages of Change	4	7
Extended Parallel Process Model	3	5
Health Belief Model	2	4
Attitude-toward-ad Model	2	4
Social Cognitive Theory	2	4
One of each of the following theories: Accommodation, Appraisal, Attitude Change, Exemplification, Focus, Functional Attitude, Incentive-sensitization, Learning Development, Mass Communication, Moral Development, Self-affirmation, Transportation	12	22

Note.

¹Total sums to more than 75 because some studies had more than one applicable category;

²These percentages calculated only on studies that used a theory (k=55).

Table 3.

Characteristics of Messages Tested in the 75 Studies

Variable	k	%
Message topic
Cigarettes	75	100
Tobacco (general)	25	33
Message theme1
Health effects	59	79
Anti-industry	34	45
Secondhand smoke	30	40
Social influences	26	35
Quitting	26	35
Cosmetic effects	18	24
Addiction	18	24
Death	16	21
Chemicals	9	12
Cost	8	11
Other	3	4
PME used to compare1
Messages	62	83
Message themes	51	68
Message formats/styles	32	43
Socio-demographics	8	11
Message attributes	7	9
Message sources	6	8
Countries	5	7
Referents	2	3
Other	4	5
Message channel1
Video	65	87
Print	11	15
Audio	3	4

Note.

¹Total sums to more than 75 because some studies had more than one applicable category;

²These percentages calculated only on studies that used a theory (k=55).

Figure 2.

Number of PME studies by decade, 1980–2016

Most studies (79%) used convenience sampling. Study participants were recruited online (28%), from colleges or universities (20%), or from high schools (17%), among other recruitment venues. Data collection tended to be conducted via computer (48%) or paper surveys (31%). Studies included participants ranging from age nine to 87 years. Among the 61 studies that reported gender, mean percent was 50.4% female. Among the 47 studies that reported race, mean percentages were 64% white, 20% African American, 15% Hispanic/Latino, 7% Asian, and 8% were mixed race/other. Studies most commonly examined adults (51%), although 35% of studies included adolescents. Seventy-nine percent of studies included smokers, 43% included non-smokers, and 21% included susceptible or experimental smokers.

The studies were almost evenly split between those using a within-subjects design where participants rated all messages (53%) or a between-subjects design where participants were randomized to differing message conditions (47%). For between-subject studies, the number of conditions ranged from two to 12, with a mean of four. Across all studies, participants were shown between one and 28 messages (M= 5.62, SD= 5.09; median = 4). Participants either viewed messages once (73%) or twice (27%). All studies examined messages about cigarettes, while 33% also mentioned tobacco in their messages. A majority of the studies used video messages (87%), while print (15%) and audio (4%) messages were used less frequently. Message themes tested most often were health effects (79%), anti-industry (45%), and secondhand smoke (40%), among other themes. Studies used PME in various ways, including to compare similarly-themed messages to one another (83%), to compare message that used different themes (68%), and to compare messages with different styles or formats (43%). Seventy-three percent of studies were theory-based (73%), with the elaboration likelihood model (24%), reactance theory (16%), reasoned action approach (13%), and message framing theories (13%) being most commonly applied.

PME Measures

We identified 126 measures across the 75 studies (see Table 4). Authors used a range of names to describe their measures (see online supplement), with the most common (52% of measures) being some variation of perceived message effectiveness or persuasiveness (e.g., perceived effectiveness, perceived ad effectiveness, effectiveness, message persuasiveness). The number of PME measures per study ranged from one to five, with a mean of 1.68 (SD= 1.02). Most studies (60%) used a single PME measure. Measures assessed variables from the nine categories which included 16 distinct PME constructs, as well as an ‘other’ category. Across the 126 measures, the number of items in each measure ranged from one to 13 (M= 3.46, SD= 2.51; median= 3). Thirty-eight measures (30%) were single item measures, whereas 88 measures (70%) used multiple items. For the 70 measures that reported reliability, coefficient alpha ranged from .59 to .95 (M= .82, SD= .13). Response formats used in PME measures were agree/disagree (62%), amount/quantity (20%), semantic differential (15%), and degree of certainty (4%), and numeric scales ranged from three to 10 points, with five-point (58%) and seven-point (29%) Likert scales being most commonly used.

Table 4.

Characteristics of Perceived Message Effectiveness (PME) Measures (N=126)

Variable	N	%
Measurement Focus
Message perceptions only	44	35
Effects perceptions only	36	29
Both types of perceptions	46	36
Use of referents
Yes	88	70
No	38	30
Referent used1
Me	71	81
Me + other	11	13
Other	6	7
References product/behavior
Yes	66	52
No	60	48
Response scale used2
Agree/disagree	78	62
Amount/quantity	25	20
Semantic differential	19	15
Degree of certainty	5	4
Response points2
Five-point	73	58
Seven-point	36	29
Four-point	6	5
Ten-point	5	4
Three-point	2	2
Six-point	1	1
Nine-point	1	1
Not applicable	2	2
NR	2	2

Note. N=number of measures.

¹These percentages calculated only on measures that used a referent (k=87).

²Total sums to more than 126 because some measures had more than one applicable category

The 126 measures were made up of message perceptions only (35%), effects perceptions only (29%), or both types of perceptions (36%). The most common constructs (>25%) assessed within PME measures were argument strength (31%), cognitive elaboration (31%), personal relevance (29%), credibility (27%), and motivation to act (26%), while the least commonly assessed (<10%) were concern/worry (9%), interest (8%), general effectiveness (8%), memory/recall (7%), interpersonal communication (7%), agreement (6%) and information seeking (0%; Table 5). Most of these constructs were assessed in measures that included other constructs (Figure 3). Constructs that were most often assessed on their own (i.e., solo) were general effectiveness (50%), liking (40%), and perceived comprehension (29%).

Table 5.

Constructs identified within Perceived Message Effectiveness (PME) Measures (N=126)

	Constructs		Use of Referents		References Behavior		Solo Constructs
Variable	N	%	N	%	N	%	N	%
PME Constructs
Attention	12	10	11	92	0	0	0	0
Credibility	34	27	4	12	2	6	6	18
Interest	10	8	2	20	0	0	0	0
Personal relevance	37	29	25	68	1	3	3	8
Liking	20	16	7	35	4	20	8	40
Memory/recall	9	7	0	0	0	0	0	0
Perceived comprehension	14	11	1	7	0	0	4	29
Perceived knowledge	19	15	12	63	0	0	3	16
Cognitive elaboration	39	31	36	92	20	51	7	18
Self-efficacy	20	16	19	95	20	100	4	20
Agreement	8	6	7	88	0	0	0	0
Argument strength	39	31	6	15	5	13	5	13
Concern/worry	11	9	11	100	10	91	0	0
Motivation to act	33	26	31	94	27	82	7	21
Interpersonal communication	9	7	9	100	1	11	2	22
Information seeking	0	0	-	-	-	-	-	-
General effectiveness	10	8	3	30	4	40	5	50
Other	17	13	6	35	2	12	0	0

Note. N=number of measures. Solo constructs are those that were assessed in their own measure - i.e., not in measures that also included other constructs.

Figure 3.

PME constructs used in measures. Larger circles indicate greater numbers of measures that include the construct. Lines indicate constructs’ co-occurrence with other constructs in the same measure (thicker lines indicate increased frequency of co-occurrence).

Seventy percent of measures used a referent in at least one item while 30% used no referents. Referent use across all PME measures ranged from 0% to 100% of items; the typical measure that used referents had 79% of its items using referents. PME constructs that most often included referents were cognitive elaboration (92% of cognitive elaboration measures used a referent), motivation to act (94%), and personal relevance (68%). Conversely, referents were much less common for the constructs of interest (20%), perceived comprehension (7%), and memory/recall (0%). Among measures that used referents, 81% used ‘me’ (i.e. “this ad makes me want to quit smoking” (Niederdeppe, Farrelly, Nonnemaker, Davis, & Wagner, 2011)), 7% used an ‘other’ (i.e. “in your opinion, was this advertising effective at stopping younger people from smoking?” (Pechmann & Reibling, 2006)), and 13% used a combination of ‘me and other’ (i.e. “this PSA made me want to quit smoking” and “this PSA would make other people want to quit smoking” (Falcone et al., 2011)).

Slightly more than half (52%) of measures referenced the product or behavior in at least one item, such as cigarettes or smoking (i.e. “this ad makes me want to quit smoking’ or “this advertisement influenced me about whether or not to smoke cigarettes’’ (Davis, Nonnemaker, Farrelly, & Niederdeppe, 2011; Murphy-Hoefer, Hyland, & Higbee, 2008)), while 48% never referenced the product or behavior. The use of a product or behavior across all PME measures ranged from 0% to 100% of items; the typical PME measure that referenced a product or behavior had 67% of its items referencing a product or behavior. PME constructs that most commonly referenced behavior were self-efficacy (100%), concern/worry (91%), motivation to act (82%), and cognitive elaboration (51%). Several constructs - including attention, interest, memory/recall, and perceived knowledge - did not ever reference the product or behavior.

Discussion

The purpose of this study was to characterize the conceptual and methodological aspects of PME measures in the anti-tobacco campaigns literature by conducting a systematic review of the literature. We examined 126 measures from 75 studies conducted in 21 countries with more than 61,000 participants. We found large increases in the use of PME over time, suggesting that researchers find PME increasingly useful for evaluating the promise of tobacco education messages for use in campaigns. Results indicated considerable variability in PME measurement, however, on message perceptions versus effects perceptions, underlying persuasive constructs, use of referents, and referencing of behavior in PME measures. While measurement heterogeneity is not unique to the PME literature, the lack of a unifying theory of PME coupled with varying definitions and little evidence of systematic scale development work may have resulted in greater measurement variability than is typical.

In the absence of a clear conceptualization of PME, it is difficult to make recommendations for future measurement. Many observers have noted this lack of a clear conceptualization of PME (Dillard & Ye, 2008; Yzer et al., 2015), and the fact that measures have often been created out of intuitive rather than theoretical bases (Dillard, Shen, et al., 2007). Our review confirms these observations by systematically demonstrating the heterogeneous measurement of PME. We suggest that future PME measures begin with a clear conceptualization of what PME is, followed by item generation and careful scale development.

While both past work (Dillard, Weber, et al., 2007) and the current work finds PME measures to be made up of message perceptions and/or effects perceptions, one direction for future work is a PME conceptualization focused only on expected message effects (i.e., effects perceptions). Some recent work points in this direction, such as Yzer et al. (2015), who have defined PME as “the extent to which a message recipient believes that a health message will affect him or her personally in terms of the particular message objectives” (p. 132). Similarly, Cho and Choi (Cho & Choi, 2010) offered a definition that focuses on “the judgment of the effectiveness of the message in fulfilling its intended outcome in the intended audience” (p. 304) (also see (Choi & Cho, 2016). These PME conceptualizations carve out a theoretically unique space, focusing on the extent to which participants believe a message is likely to have particular effects (i.e., perceived effectiveness). Research following this path could develop, test, and validate PME scales that focus on expected effects only.

Taking this research direction a step further, an effects-based conceptualization of PME would draw on particular persuasive constructs, such as cognitive elaboration, self-efficacy, argument strength, concern/worry, motivation to act, and interpersonal communication. Table 6 illustrates sample message effects PME items from our review and also illuminates actual effectiveness constructs that pair with their perceived effectiveness counterparts. This perceived-actual effectiveness pairing highlights two important points. First, the constructs upon which these PME items are based have both theory (Bandura, 1986; Ferrer, Klein, Persoskie, Avishai-Yitshak, & Sheeran, 2016; Fishbein & Ajzen, 2010; Petty & Cacioppo, 1986; Southwell & Yzer, 2007) and empirical evidence (Bandura, 1997; Brennan, Durkin, Wakefield, & Kashima, 2016; Lee et al., 2013; Petty, Barden, & Wheeler, 2009; Vangeli, Stapleton, Smit, Borland, & West, 2011; Webb & Sheeran, 2006) behind them, which are positive attributes for any PME measure. Secondly, these actual effectiveness constructs can be paired with their associated PME items/measures and applied in studies examining the predictive validity of PME.

Table 6.

Examples of Effects-focused PME Constructs, Items, Actual Effectiveness Constructs, and associated Theories

PME Persuasive Construct	Sample Item	Actual Effectiveness Construct	Theory
Cognitive elaboration	The ad put thoughts in my mind about quitting smoking (Bigsy et al., 2013). Overall, how much did the ad make you think about reasons for not smoking? (Kang, 2007) These ads made me think more about quitting smoking (Mowery et al., 2016)	Cognitive Elaboration/Attitudes	Elaboration likelihood model (Petty & Cacioppo, 1986); Reasoned action approach (Fishbein & Ajzen, 2010)
Self-efficacy	Overall, how much did the ads make you feel that it is easy to quit smoking? (Wong & Cappella, 2009) Watching this ad helped me feel confident about how to best deal with smoking (Bigsby et al., 2013) This PSA gives me the confidence to resist smoking if a friend offers me cigarette (Tharp-Taylor et al., 2012)	Self-efficacy/ Perceived Behavioral Control	Social cognitive theory (Bandura, 1986); Reasoned action approach (Fishbein & Ajzen, 2010)
Argument strength	How convincing is this argument that you should not smoke? (Tharp-Taylor et al., 2012) How strong of an argument is this for not smoking? (Tharp-Taylor et al., 2012) Did the ad give you good reasons not to smoke? (Murukutla et al., 2015)	Attitudes/beliefs	Elaboration likelihood model (Petty & Cacioppo, 1986); Reasoned action approach (Fishbein & Ajzen, 2010)
Concern/worry	This ad made me feel concerned about my smoking (Brennan et al., 2014) Overall, how much did the ads make you worry about your health risks because of your smoking? (Wong & Cappella, 2009) This ad made me feel concerned about smoking around other adults (Murukutla et al., 2015)	Risk perceptions	Risk perception theory (e.g., TRIRISK) (Ferrer et al., 2016)
Motivation to act	The ad made me feel motivated to try to quit smoking (Brennan et al., 2014) To what extent did the ad make you think you should try to stop smoking? (Donovan et al., 2006). The PSA made me want to quit smoking (Falcone et al., 2011)	Intentions	Reasoned action approach (Fishbein & Azjen, 2010)
Interpersonal communication	This ad is one I would talk about (Wakefield et al., 2005) I am likely to talk to someone else about this ad (Murukutla et al., 2015) I will talk with my friends about this ad (Murphy-Hoefer et al., 2008).	Interpersonal communication	Interpersonal campaigns theory (Southwell & Yzer, 2007)

Another important direction for future research is to understand the role of both message and effects perception measures in perceived and actual effectiveness. Past work has examined but not fully answered the question of whether one class of perceptions is more diagnostic than the other or if the two types should be used in combined measures of PME (Dillard, Weber, et al., 2007; Dillard & Ye, 2008). Research to better understand the dimensionality of PME measures is needed, as is work to examine if such measures differentially predict the actual effectiveness of messages. Studies that pit one type of measure against another could empirically examine whether message perceptions, effects perceptions, or combined measures provide the best diagnosticity of messages on actual effectiveness metrics.

In this review, we also examined the role of measurement characteristics such as referents and behavior. Referents were fairly commonly applied in the PME literature (in 70% of measures), and the most common referent was the target of “me” (81% of referent measures used “me”). An effects-focused PME conceptualization virtually requires that measures include a referent in order to clarify the message effect target (e.g., this message motives me to quit smoking), while a message perceptions perspective does not require a referent. While referents were commonly applied in PME measures, they were not used in nearly one-third of measures, nor were they used in all items of many of the measures that applied them. In addition, a small number of measures used an “other” as referent (e.g., this message would motivate people my age to not smoke) or used both “me” and “other” as referents in the same measure.

Additional work is needed on the extent to which the addition of a referent - as well as the type of referent - may affect PME ratings. The limited empirical literature seems to suggest that the use of a referent may improve measurement precision. For example, Dillard and Ye (Dillard & Ye, 2008) examined the influence of referents in PME measures in a rating study examining the nature of PME judgements, using items without referents. The study asked participants to select which referent(s) they were thinking of after they rated each message. Results indicated that: 1) most participants used multiple referents when judging messages, but both the number and type of referents varied; 2) participant use of referents differed by message; and 3) use of more referents was positively associated with PME scores. This study strongly suggests that in the absence of a referent, participants will use different referents when judging messages, potentially biasing PME ratings.

In addition, the third-person effect proposes that people will estimate media messages to have a greater effect on others than on themselves (Sun, Pan, & Shen, 2008). With pro-social messages such as tobacco education advertisements, however, it is possible that a reverse third-person effect may take place (Sun et al., 2008). While tobacco education PME studies show conflicting effects regarding which type of third-person effect will occur (Cheng, 2004; Chock, Fox, Angelini, Lee, & Lang, 2007), such studies all demonstrate that choice of referent affects PME ratings. Additional research on the impact of referents on PME ratings is clearly warranted.

With regard to referencing the product or behavior, we found that only roughly half of measures included some reference to the behavior. This is perhaps not surprising given that many PME items and measures assessed message perceptions rather than effects perceptions. When assessing perceptions of a message (e.g., believable, informative, meaningful, etc.), the behavior is often not referenced. An effects-based perspective on PME, however, requires that the behavior is referenced in most or all items, as we found for effects-focused constructs such as self-efficacy, concern/worry, and motivation to act (Table 6). Interestingly, the argument strength construct was typically assessed with only the term “convincing” (Kang, 2007; Kean, 2004; Rhodes, Roskos-Ewoldsen, Edison, & Bradford, 2008), although it could be applied in items that have participants rate how much a message convinces them or others not to smoke (Cheng, 2004; Tharp-Taylor, 2012). Additional research on PME should examine whether referencing the behavior affects PME ratings.

A discussion of persuasive constructs underlying PME measures raises the question of the extent to which a single PME construct or measure can work across multiple contexts or whether PME measures should differ across contexts. For instance, if increasing self-efficacy is an important goal for a particular message, then self-efficacy may be important to one’s PME measure(s), either alone or in combination with other constructs. If self-efficacy is not a focus, then assessing a self-efficacy PME construct makes little sense. This perspective is in line with recent work suggesting that PME measures should be aligned with the goals of one’s messages (Yzer et al., 2015). This is in contrast with some of the more generic PME measures that have been used in the literature, such as general effectiveness (Anderson & Holody, 2014; Byrne, Guillory, Mathios, Avery, & Hart, 2012), liking (Kuang, 2009; Terry-McElrath et al., 2005), and credibility (Gelb & Pickett, 1983; Kim, 2006). Such more generic measures may not provide the kind of precise tools that message designers seek in gauging the likely effectiveness of their messages in meeting particular goals. Still, future work in this area could compare the performance of a targeted multiple-item PME scale to more generic items assessing liking or generic effectiveness.

A related question is whether PME measures should assess single constructs or combine multiple persuasive constructs together in a single measure. In our review, most measures combined multiple constructs together - i.e., there were few measures that assessed single constructs alone. Thus, for example, when cognitive elaboration was assessed, it was typically (82% of the time) assessed in a measure with other persuasive constructs (Bigsby et al., 2013) (see Figure 3). This decision may rest with whether a researcher is evaluating messages with a very targeted goal or whether messages are intended to impact a broader range of persuasive constructs. In the case of multiple constructs, empirical scale development work will be needed regarding the dimensionality of the PME measure.

Finally, there is the crucial issue of validation - i.e., are PME measures capable of identifying more effective messages? While PME is widely used in studies of tobacco education campaigns and in other areas such as tobacco health warnings (Francis, Hall, Noar, Ribisl, & Brewer, 2017; Noar et al., 2016), an implicit assumption has been that such measures are diagnostic of actual effectiveness. As discussed earlier, evidence of whether messages that score higher on PME are actually more effective varies, and additional validation work is needed. The findings from the current review provide an overview of current measures and suggest several possible directions for item generation and scale development work. Once measures are developed, a variety of issues should be carefully considered when designing validation studies (Table 7). First, people with a more favorable disposition toward a behavior rate messages more generously, resulting in reverse causation problems. Thus, studies that simply show associations between PME and an outcome (e.g., smoking quit intentions) are inadequate because those with greater quit intentions rate messages as more effective to begin with (Davis et al., 2011; Donovan, Leivers, & Hannaby, 1999). Rigorous approaches to overcome this should be brought to bear, including the use of aggregate PME scores (Bigsby et al., 2013), experimental designs with controls (Popova, Neilands, & Ling, 2013), testing perceived and actual effectiveness in separate samples (Gollust, Niederdeppe, & Barry, 2013; Hornikx, 2008), and statistically controlling for baseline smoking characteristics in longitudinal studies (Davis et al., 2016; Davis et al., 2013). Second, the range of messages tested in studies should vary widely, as PME may not be sensitive to small differences in message effectiveness but may be diagnostic when there are larger differences. In fact, O’Keefe (2018) found hints of this in that PME was better at predicting actual effectiveness when larger differences in PME across messages existed (i.e., 67% versus 53% accuracy). Therefore, studies should test messages or message types/themes across a range of levels of perceived and actual effectiveness in order to discover a possible sensitivity threshold for PME. Third, correspondence between PME and actual effectiveness measures should be apparent, and researchers should ask themselves: does the PME measure match the actual effectiveness metric(s) that is being examined? PME measures are most likely to be diagnostic of actual effectiveness when the PME items ask about perceptions of what the messages are trying to achieve, and when the actual effectiveness metrics are in line with the message determinants being targeted. Studies that mismatch PME measures and actual effectiveness outcomes (e.g., “liking” of message predicting motivation to quit smoking or behavior change) are unlikely to be successful, as are PME items that are vague and therefore fail to show correspondence (i.e., general effectiveness). In the end, the ultimate value of PME lies in its promise as a practical tool to help campaign designers choose a set of more effective messages from a larger message pool. To achieve this important goal, greater theoretical development and measurement development work are needed, as are validation studies that clearly illustrate the predictive value of PME.

Table 7.

Key Issues in PME Validation Studies

Problem	Example	Possible Solutions
Reverse Causation	Those who have a more favorable disposition toward a behavior (e.g., those who intend to quit smoking) rate messages more favorably on PME (Davis et al., 2011; Donovan et al., 1999). Validation studies must take this confound into account or risk misinterpreting the association between PME and actual effectiveness	Use aggregate PME scores (Bigsby et al., 2013) Use experimental designs with controls (Popova et al., 2013) Test PME and actual effectiveness in separate samples (Gollust et al., 2013; Hornikx, 2008) Control for baseline differences in disposition towards the behavior (Davis et al., 2013, 2016)
Unknown sensitivity of PME measures	Current PME measures may be unable to offer meaningful guidance on message selection when there are small differences between messages or message types (O’Keefe, 2018)	Studies should test messages with at least moderate expected variability on PME and actual effectiveness, rather than a more narrow range of messages. This will advance an understanding of the sensitivity threshold for PME measures.
Lack of correspondence	Studies should use PME measures that are appropriately matched with their actual effectiveness counterparts. For example, using “liking” as a PME measure (Kuang, 2009; Terry-McElrath et al., 2005) may be inappropriate because the goal of health messages is not to garner liking but to change behavior	Ensure correspondence among 1) determinants that particular messages are expected to change, 2) PME measurement, and 3) actual effectiveness measurement

Note. PME = perceived message effectiveness.

Conclusion

This systematic review examined PME measures in tobacco education campaigns, providing a description of the conceptual and methodological characteristics of these measures. Through an examination of 126 measures from 75 studies, we found considerable variability in whether PME measures were focused on message perceptions, effects perceptions, or both types of perceptions. We also found considerable variability in the use of underlying persuasive constructs, use of referents, and referencing of behavior in PME items. For future work, we offer several recommendations, including new scale development work that carefully considers and tests the value of message versus effects perceptions, a variety of persuasive constructs, use of referents, and referencing of behavior. We also recommend robust validation studies that examine the predictive validity of PME. Advancing the conceptualization, reliability, and validity of PME measures is critical to giving researchers and practitioners more precise tools for gauging the likely effectiveness of messages, thereby increasing the effectiveness of health communication campaigns.