Does Perceived Message Effectiveness Predict the Actual Effectiveness of Tobacco Education Messages? A Systematic Review and Meta-Analysis

Abstract

Target audience ratings of the likely impact of persuasive messages, known as perceived message effectiveness (PME), are commonly used in health communication campaigns. However, applications of PME rely on a critical assumption – i.e., that PME is a valid indicator of the likely effectiveness of messages. To examine the evidence supporting this assumption, we conducted a systematic review and meta-analysis of longitudinal studies in the tobacco education campaigns literature. Six longitudinal studies examining the predictive validity of PME met inclusion criteria. Results indicated that PME ratings were significantly associated with the majority of outcomes studied. In fact, each of the 6 studies found PME to be associated with at least one outcome, and across the 6 studies, PME was associated with message recall, conversations about ads, beliefs about smoking and quitting smoking, quit intentions, and cessation behavior. Meta-analyses demonstrated that PME predicted quit intentions (r=.256, p<.001) and cessation behavior (r=.201, p<.001), revealing effects that were small-to-medium in magnitude. Our results suggest that PME provides some predictive value as to the likely effectiveness of messages, although additional work using different validation designs, with other health behaviors, and among other populations is needed.

Target audience ratings of the likely impact of persuasive messages, known as perceived message effectiveness (PME), are commonly used in the health communication campaigns literature. Reviews have demonstrated the broad use of PME ratings in empirical studies evaluating the likely effectiveness of campaign messages across health topics (Dillard, Weber, & Vail, 2007; Noar, Bell, Kelley, Barker, & Yzer, 2018; Yzer, LoRusso, & Nagler, 2015). In tobacco education campaigns, the use of such ratings has increased exponentially since the year 2000 (Noar, Bell, et al., 2018). Moreover, PME is used in several contemporary national smoking prevention (Duke et al., 2015) and cessation (Davis et al., 2017) campaigns in the development, pre-testing, and preliminary evaluation of campaign messages.

The large growth in use of PME suggests that researchers find it to be a useful tool. In practice, PME is used for both message selection and as an early indicator of campaign receptivity. For message selection, a common approach is pre-testing several messages and selecting only those that score highest on PME to use in a campaign (Mowery, Riedesel, Dreher, Schillo, & Saul, 2016; Sutfin et al., in press). This approach has the potential to increase campaign efficacy by disseminating only the most effective messages. For receptivity, a common approach is assessing a large sample on the final campaign messages either just before or after they have been deployed (Duke et al., 2015; Zhao et al., 2016). This can give campaign planners an early signal as to whether pre-testing led them in the right direction, providing insights regarding possible course corrections (e.g., give greater air time to particular ads). However, these applications of PME rely on a critical assumption – i.e., that PME is a valid indicator of the likely effectiveness of messages. If PME ratings are valid, then their use for these purposes is justified. If PME is an invalid indicator of likely message effectiveness, however, then devoting resources to PME studies makes little sense, and alternative approaches for message selection and receptivity should be used.

Despite scores of studies that have applied PME, surprisingly few studies have attempted to rigorously validate PME. Dillard, Weber, et al. (2007) synthesized the association between PME and attitudes in a meta-analysis of 40 cross-sectional messaging studies across diverse behaviors, finding a correlation between PME and attitudes of r =.41. Dillard, Shen, and Vail (2007) also conducted a series of cross-sectional studies examining the ability of PME to predict attitudes and behavioral intentions, with results that were supportive of PME. However, cross-sectional correlational studies are generally incapable of demonstrating causal evidence for PME because it is just as likely that those with more positive attitudes about the behavior will rate messages as more effective (i.e., reverse causation). Dillard, Shen, et al. (2007) acknowledge this limitation of their work, stating that “we cannot eliminate the possibility that attitudes or intentions cause individuals to judge messages in line with their preexisting orientations…” (p. 482). Indeed, several empirical studies in tobacco education campaigns demonstrate precisely this finding - i.e., those who have a less favorable disposition toward smoking rate tobacco education advertisements as more effective (Biener, McCallum-Keeler, & Nyman, 2000; Davis, Nonnemaker, Farrelly, & Niederdeppe, 2011; Donovan, Leivers, & Hannaby, 1999). This causes reverse causation problems that make interpreting cross-sectional evidence for PME difficult if not impossible.

A more recent attempt to examine the validity of PME ratings across extant topical areas using meta-analysis was provided by O’Keefe (O’Keefe, 2018). He conducted a meta-analysis in which he examined the consistency between mean ratings of PME as tested in one sample and actual effectiveness testing of messages from another sample. Comparing 151 message pairs derived from 35 studies, he found that use of PME would only result in choosing a more effective message 58% of the time, which is little better than chance. While these results paint a fairly dim picture of the validity of PME, they are limited by both the significant heterogeneity in the PME measures used in these studies as well as design decisions that may have artificially reduced the ability of PME to predict actual effectiveness (Noar, Barker, & Yzer, 2018).

Bigsby et al. examined the relationship between PME and quit intentions in two cross-sectional studies of smoking cessation messages (Bigsby, Cappella, & Seitz, 2013). To reduce the chances of reverse causation, they created aggregate PME scores that were not tied to the individual. Linear regressions revealed significant (though modest) associations between aggregate PME ratings and quit intentions, supporting PME as a predictor of actual effectiveness. In another cross-sectional study, Popova et al. utilized an 8-group experimental design with 2 control conditions to examine PME as a predictor of the actual effectiveness of smokeless tobacco prevention messages (Popova, Neilands, & Ling, 2013). While health consequence ads were perceived as most effective, anti-industry ads were actually more effective in decreasing favorable attitudes toward snus, revealing a possible disconnect between perceived and actual effectiveness. Across study conditions, PME was associated with attitude change but only modestly so, r =.06.

While the above studies provide an admittedly mixed record of the ability of PME to predict largely attitudinal and behavioral intention outcomes in cross-sectional and experimental studies, none examined longitudinal outcomes. Because campaigns aim to have impact over time – including affecting behavior change – longitudinal evidence is needed. Longitudinal studies also have advantages regarding temporal order (i.e., ratings of PME assessed first predicting changes in outcomes assessed later in time), strengthening causal claims relative to cross-sectional evidence. To allow campaign researchers to draw stronger conclusions about the usefulness – or lack thereof – of PME, a synthesis of existing longitudinal validation evidence is needed. To achieve such a synthesis, we conducted a systematic review and meta-analysis of PME in tobacco education campaigns. We focused on tobacco education campaigns because they are a long-standing campaigns literature made up of high quality studies, and one in which PME has been extensively applied (Noar, Bell, et al., 2018). The goal of our review was to examine the extent to which PME predicted the actual effectiveness of anti-smoking advertisements in longitudinal studies.

Method

Search Strategy

We used a comprehensive search strategy to locate studies relevant to this systematic review. The search strategy involved four steps. First, we searched PsycINFO, PubMed, Web of Science, Business Source Premier, Communication and Mass Media Complete, CINAHL, Global Health, and Scopus computerized databases in October of 2016 and again in October of 2017. We paired several PME-related terms such as perceived effectiveness, message effectiveness, and advertising effectiveness with smoking terms, such as cigarette, tobacco, and smoking. Second, we searched both Google and Google Scholar using the same sets of search terms, examining the first 100 results from each search. Third, we examined the reference sections of four narrative reviews examining PME or anti-tobacco advertising studies (Choi & Cho, 2016; Dillard, Weber, et al., 2007; National Cancer Institute, 2008; Yzer et al., 2015). Finally, we searched the reference lists of the final set of articles included in our review.

Our review had six inclusion criteria. To be included studies had to: 1) test anti-smoking video, print, or audio advertisements; 2) expose participants to advertisements directed at reducing cigarette smoking behaviors; 3) measure and report quantitative data on PME; 4) assess actual effectiveness of ads (e.g., recall, conversations, attitudes/beliefs, norms, intentions, or behavior) in a longitudinal study; 5) examine the association between PME and actual effectiveness; and 6) the full study report had to be available in English.

The original database search in October 2016 yielded 4,377 references, and additional searches yielded 173 references. After removing duplicates, 2,212 references remained. Two trained reviewers independently screened study titles, reducing the number to 528. They then reviewed abstracts, reducing the number to 226. During this process, references were only discarded if both reviewers agreed that a given reference was not relevant to the review. The full text of the remaining 226 articles was then located and screened by the two reviewers, and reasons for study exclusion were tracked. If the two reviewers made a different determination about any of the articles, they consulted a third reviewer to resolve the discrepancy and make a final determination. This process identified 4 studies. An updated database search in October 2017 yielded an additional 2 studies. Thus, the systematic review had 6 studies (Figure 1).

Figure 1.

PRISMA flow diagram showing the study screening process.

Article Coding

Two coders independently coded all studies on sample population characteristics (e.g., gender, smoking status), study design characteristics (e.g., data collection mode, timing of follow-up), message characteristics (e.g., number of messages), and PME measurement characteristics (e.g., PME scale items, coefficient alpha). All discrepancies that arose during the coding process were resolved through discussion between the two coders and a third reviewer. Mean percent agreement across all coding categories was 95%.

Summarizing study findings.

We summarized all multivariate analyses that examined the ability of PME to predict actual effectiveness outcomes. Given that smokers who are more ready to quit tend to rate ads more favorably (i.e., potential reverse causation), we summarized all variables used as controls in analyses, highlighting smoking-related control variables such as quitting beliefs, quit intentions and smoking behavior. We summarized the multivariate results of analyses, noting which findings were statistically significant (p <.05).

Meta-analysis.

We computed meta-analyses on all study outcomes reported in 2 or more studies. We extracted bivariate data on the association between PME and actual effectiveness outcomes from study reports. When those data were not reported, we contacted the authors to request the data. We converted all statistics to r as a common metric to characterize the association between PME and a given outcome variable. When studies reported more than one measure of PME or a given outcome variable (e.g., two measures of quit intentions), we averaged them together. We then weighted effect sizes by their inverse variance and combined them using random effects meta-analytic procedures (Lipsey & Wilson, 2001). We calculated the Q statistic and I² to examine whether heterogeneity existed among the effect sizes. We did not compute moderator analyses due to the modest numbers of studies in our analyses.

Results

The 6 studies were published between 2013 and 2017 and took place in Australia (50%) and the United States (50%; Table 1). All studies examined adult smokers except for one which examined adolescent non-smokers and smokers (Bigsby, Monahan, & Ewoldsen, 2017). Female respondents represented 55.1% of study samples. Sample sizes ranged from N = 208 – 5,794, with a median sample size of 327. Studies had participants rate between 1 and 8 anti-smoking television advertisements in each study at the baseline session (median = 2). The time between baseline and follow-up ranged from 5 days to 4 months, with a median of 3 weeks.

Table 1.

Characteristics and design of longitudinal studies examining perceived message effectiveness

Study	Sample	Study design	Description and # of Ads	PME measure(s)	Outcomes assessed
Bigsby, Monahan, & Ewoldsen (2016)	N = 244; 60% female; Age 13-17 (M = 14.8); Smokers, Nonsmokers; United States	Participants rated 3 of 6 anti-smoking television advertisements on PME at baseline (computer survey) and took a follow-up phone survey.	6 ads from state-level anti-smoking campaigns. Respondents were shown either: 1) Second-half punch ads focusing on an event or action and obscuring the topic until the last few seconds. 2) Personal narrative ads presenting the story of an individual’s negative experiences or consequences associated with cigarettes.	Message elaboration (5 items, α=.77 - .79): when watching this ad, I did not want to think about smoking; watching this ad made me really think about the bad parts of smoking; this ad had more information about smoking than I personally need; I thought about how this ad related other things I know about smoking; I thought about what I might do based on this ad.	Unaided recall assessed at 3 months
Brennan, et al. (2014)	N = 208, 54.5% female, Age 18+ (M = 31.4); Smokers; Australia	Participants rated 1 of 2 anti-smoking television advertisement on PME at baseline (paper survey) and took a follow-up phone survey.	2 narrative-style ads that had not aired in Victoria, Australia before the experiment: 1) Pam Laffin 2) Rick Stoddard-46 Years Old	Ad-directed perceived effectiveness (3 items, α=.74): made me stop and think; made a strong argument for quitting; taught me something new Personalized perceived effectiveness (3 items, α=.75): was relevant to me; made me feel concerned about my smoking; made me feel motivated to quit smoking	Change in quit intentions (after vs. before ad exposure) Cessation behavior assessed at 5-21 days (mean = 8.5 days). Dichotomized into 2 groups: 1) no change or thought about/decided to quit only or 2) tried to cut down or made a quit attempt and either relapsed or stayed quit
Brennan, et al. (2016)	N = 409; 62.3% female; Age 18+; Smokers; Australia	Participants took a pre-exposure telephone survey, watched a program with anti-smoking advertisements, and then rated PME for 1 of 6 ads in a follow-up telephone survey.	6 ads (Amputation, Mouth Cancer Talks, Carotid, Bronchoscopy, Zita—Tears Apart a Family, and Separation) that had not previously aired in Victoria, Australia were evaluated at the time of their launch. All emphasized serious health effects of smoking through emotional stories and/or graphic imagery	Personalized perceived effectiveness (3 items, α=.75): made me feel motivated to try to quit smoking; made me feel concerned about my smoking; was relevant to me	Change in quit intentions (after vs. before ad exposure)
Brennan, et al. (2017), Study 2	N = 232; 56.9% female; Age 18+; Smokers; Australia	Participants and conversation partners took a pre-exposure survey, viewed an anti-tobacco advertisement, and rated PME for 1 of 2 ads on a post-exposure survey. Participants later completed a follow-up phone survey.	2 narrative-style ads that had not aired in Victoria, Australia before the experiment: 1) Pam Laffin 2) Rick Stoddard-46 years old	Ad-directed perceived effectiveness (3 items, α=.73): made me stop and think; made a strong argument for quitting; taught me something new Personalized perceived effectiveness (3 items, α=.78): was relevant to me; made me feel concerned about my smoking; made me feel motivated to try to quit smoking	All outcomes assessed between 5 and 21 days following exposure (mean = 8.5 days) Post-exposure conversation occurrence Frequency of post-exposure conversations Post-exposure conversation valence (i.e. favorable or unfavorable ad appraisal) Post-exposure conversation content (i.e. quitting or emotion talk)
Davis, et al. (2013)	N = 3,411; 48.0% female; Age 18+; Smokers; United States	Participants rated 8 anti-smoking television advertisements on PME in 1 of 5 conditions at baseline (computer survey). They took a follow-up computer survey in which they were again exposed to the ads.	A pool of >20 ads that mostly came from the CDC Media Campaign Resource Center database. They were allocated into 1 of 5 conditions (some ads duplicated across conditions): 1) why to quit (testimonial) 2) why to quit (graphic images) 3) how to quit only 4) why to quit (testimonial) + how to quit 5) why to quit (graphic images) + how to quit	Perceived ad effectiveness (6 items, α=.94): are worth remembering; grabbed my attention; are powerful; are informative; are meaningful to me; are convincing	Negative feelings about smoking, decisional balance, outcome expectations, desire to quit smoking, quit intentions, quit attempts, self-efficacy assessed at 2 weeks
Davis, et al. (2017)	N = 5,794; 58.4% female; Age 18+; Smokers; United States	Participants rated 8 anti-smoking television advertisements on PME at baseline (computer survey) and then an additional 6 advertisements on PME on a follow-up computer survey.	14 ads from the CDC Tips from Former Smokers Campaign. Most ads depicted former smokers with severe debilitations and illnesses resulting from smoking.	Perceived effectiveness (6 items, α=.95): worth remembering; grabbed my attention; was powerful; was informative; was meaningful; was convincing	Quit attempts assessed at 4 months

Predictive Validity of PME: Recall and Conversations about Ads

Multivariate analyses examining the association between PME and extant outcomes are reported in Table 2. Only a single study examined the association of PME and message recall, finding that PME was significantly associated with unaided recall at follow-up 3 months later, p < .05 (Bigsby et al., 2017). One study also examined associations between PME and conversations about anti-smoking advertisements, reporting mixed results (Brennan, Durkin, Wakefield, & Kashima, 2017). Using two measures of PME (ad-directed perceived effectiveness and personalized perceived effectiveness), the authors found no significant association between either scale and occurrence or frequency of conversations about anti-smoking advertisements about 1 week later. However, the study did find significant associations between both measures of PME and favorable appraisals of anti-smoking advertisements in conversations over the forthcoming week (ad-directed perceived effectiveness and personalized perceived effectiveness, both ps < .05), and a significant association between one measure of PME (ad-directed perceived effectiveness) and unfavorable appraisals of anti-smoking advertisements in conversations (p < .05). It is worth noting that these associations became non-significant in multivariate analyses that controlled for several additional message-oriented variables, such as transportation and emotions felt after watching the ad (see (Brennan et al., 2017)). Results also indicated no significant association between either PME measure and quitting talk content or emotion talk content in conversations about anti-smoking advertisements 1 week later.

Table 2.

Results of longitudinal studies validating perceived message effectiveness by outcome

Study	Analysis	Control Variables	Findings	Sig?
Ad recall
Bigsby et al. (2016)	Multi-level structural equation model	Person level: Valenced attitude accessibility, average reaction time, personal narrative messages, gender, race, perception of message bias, message elaboration, smoking behavior Message level: Perception of message bias, seen message before, seen similar message before	PME (i.e., message elaboration at the message level) significantly associated with unaided recall at 3-month follow-up, b = .20, p<.05.	✓
Conversations about smoking advertisements
Brennan et al. (2017)	Multi-variable logistic or linear regression	Days since tax/plain packaging announcement, clustering at friendship pair level	PME (i.e., ad-directed perceived effectiveness) was not significantly associated with subsequent conversation about the ad seen at ~1-week follow-up, OR = 1.04, ns.	–
			PME (i.e., personalized perceived effectiveness) was not significantly associated with subsequent conversation about the ad seen at ~1-week follow-up, OR = 1.00, ns.	–
			PME (i.e., ad-directed perceived effectiveness) was not significantly associated with conversation frequency about the ad seen at ~1-week follow-up, β =.10, ns.	–
			PME (i.e., personalized perceived effectiveness) was not significantly associated with conversation frequency about the ad seen at ~1-week follow-up, β = .09, ns.	–
			PME (i.e., ad-directed perceived effectiveness) was significantly associated with favorable appraisals of antismoking ads in conversation at ~1-week follow-up, OR = 1.75, p < .05.	✓
			PME (i.e., personalized perceived effectiveness) was significantly associated with favorable appraisals of antismoking ads in conversation at ~1-week follow-up, OR = 1.69, p < .05.	✓
			PME (i.e., ad-directed perceived effectiveness) was significantly associated with unfavorable appraisals of antismoking ads in conversation at ~1-week follow-up, OR = .52, p < .05.	✓
			PME (i.e., personalized perceived effectiveness) was not significantly associated with unfavorable appraisals of antismoking ads in conversation at ~1-week follow-up, OR = 0.83, ns.	–
			PME (i.e., ad-directed perceived effectiveness) was significantly associated with quitting talk in conversation at ~1-week follow-up, OR = 1.84, p< .05.	✓
			PME (i.e., personalized perceived effectiveness) was not significantly associated with quitting talk in conversation at ~1-week follow-up, OR = 2.20, ns.	–
			PME (i.e., ad-directed perceived effectiveness) was not significantly associated with emotion talk in conversation at ~1-week follow-up, OR = 1.73, ns.	–
			PME (i.e., personalized perceived effectiveness) was not significantly associated with emotion talk in conversation at ~1-week follow-up, OR = .92, ns.	–
Attitudes/beliefs about smoking
Davis et al., (2013)	Ordinary least-squares regression	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, negative feelings toward smoking at baseline	PME (i.e., perceived effectiveness) significantly associated with increased negative feelings toward smoking at 2-week follow-up, b = .64, p < .001.	✓
Davis et al., (2013)	Ordinary least-squares regression	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, decisional balance away from smoking at baseline	PME (i.e., perceived effectiveness) significantly associated with increased decisional balance away from smoking at 2-week follow-up, b = .35, p < .01.	✓
Attitudes/beliefs about quitting smoking
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, belief that quitting smoking would improve health at baseline	PME (i.e., perceived effectiveness) significantly associated with increased belief that quitting smoking would improve personal health at 2-week follow-up, OR = 1.97, p < .001.	✓
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, belief in benefit to health of quitting after smoking for 20+ years at baseline	PME (i.e., perceived effectiveness) significantly associated with increased belief in benefit to health of quitting after smoking for 20+ years at 2-week follow-up, OR = 1.80, p < .001.	✓
Self-efficacy to quit smoking
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, self-efficacy at baseline	PME (i.e., perceived effectiveness) not significantly associated with perceptions of success in giving up smoking in next 6 months (self-efficacy) at 2-week follow-up, OR = 1.09, ns.	–
Intentions to quit smoking
Brennan et al., (2014)	Multi-variable logistic regression analysis	Dependency within friendship pairs, number of days since tax increase and plain packaging announcement, ad seen, conversation condition, quit intentions at baseline, daily cigarette consumption	PME (i.e., personalized perceived effectiveness) significantly associated with change in quit intentions at immediate post-exposure, OR = 2.63, p < .01.	✓
			PME (i.e., ad-directed perceived effectiveness) significantly associated with change in quit intentions at immediate post-exposure, OR = 1.70, p <.05.	✓
			In a combined analysis with both PME scales, personalized perceived effectiveness significantly associated with change in quit intentions (OR = 2.57, p < .01) but ad-directed perceived effectiveness not associated (OR = 1.03, ns) at immediate post-exposure.	✓ –
Brennan et al., (2016)	Multi-variable logistic regression analysis	Age, socioeconomic status, sex, educational status, multiple ad exposure, daily cigarette consumption	PME (i.e., personalized perceived effectiveness) significantly associated with change in quit intentions at immediate post-exposure, OR = 2.32, p < .001.	✓
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, desire to quit at baseline	PME (i.e., perceived effectiveness) significantly associated with desire to quit at 2-week follow-up, OR = 1.97, p < .001.	✓
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, serious consideration of quitting in next 6 months at baseline	PME (i.e., perceived effectiveness) significantly associated with serious consideration of quitting in next 6 months (quit intention) at 2-week follow-up, OR = 1.45, p < .05.	✓
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, planning to stop smoking in next 30 days at baseline	PME (i.e., perceived effectiveness) significantly associated with planning to stop smoking in next 30 days (quit intention) at 2-week follow-up, OR = 1.33, p < .05.	✓
Smoking cessation behavior
Brennan et al., (2014)	Multi-variable logistic regression analysis	Dependency within friendship pairs, number of days since tax increase and plain packaging announcement, ad seen, sex, age, number of days to follow-up survey, quit intentions at baseline	PME (i.e., personalized perceived effectiveness) significantly associated with changes in cessation behavior at ~1-week follow-up, OR = 1.47, p< .01.	✓
			PME (i.e., ad-directed perceived effectiveness) not significantly associated with changes in cessation behavior at ~1-week follow-up, OR = 1.06, ns.	–
			In a combined analysis with both PME scales, personalized perceived effectiveness significantly associated with changes in cessation behavior (OR = 1.93, p < .01) but ad-directed perceived effectiveness not associated (OR = .70, ns) at ~1-week follow-up.	✓ –
Davis et al., (2013)	Multi-variable logistic regression analysis	Age, race/ethnicity, income, education, gender, marital status, previous awareness of ads in stimulus, mental anxiety (scaled), recruitment outside panel, quit attempts at baseline	PME (i.e., perceived effectiveness) not significantly associated with quit attempts at 2-week follow-up, OR = 1.16, ns.	–
Davis et al., (2017)	Multi-variable logistic regression analysis	Race/ethnicity, education, income, chronic conditions, mental health conditions, total television gross ratings points, average television consumption, age, sex, desire to quit, previous quit attempts at baseline (past 3 months), daily cigarette consumption	PME (i.e., perceived effectiveness) significantly associated with quit attempts at 4-month follow-up, OR = 1.30, p < .01.	✓

Note. Sig = statistically significant association, p<.05; Underlining indicates smoking-related control variables.

Predictive Validity of PME: Smoking-related Beliefs and Self-Efficacy

One study found a significant association between PME and increased negative feelings about smoking (p <.001) and decisional balance away from smoking (p <.01) at 2-week follow-up (Davis, Nonnemaker, Duke, & Farrelly, 2013). That same study also found significant associations between PME and beliefs regarding the benefit to health of quitting smoking after smoking for over two decades (p < .001) and beliefs that quitting smoking would improve personal health (p < .001) at 2-week follow-up (Davis et al., 2013). Finally, this study found no association between PME and self-efficacy to quit 2 weeks later (Davis et al., 2013).

Predictive Validity of PME: Quit Intentions

Three studies found evidence of an association between PME and quit intentions, but only one of these studies (Davis et al., 2013) assessed quit intentions (and desire to quit) longitudinally. That study found a significant association between PME and a higher desire to quit smoking (p < .001) at 2-week follow-up. They also found a significant association between PME and 2 measures of quit intentions at 2-week follow-up: serious consideration of quitting in the next 6 months (p < .05) and planning to stop smoking in the next 30 days (p < .05) (Davis et al., 2013). Another study found evidence of an association between two PME measures (personalized perceived effectiveness, p < .01 and ad-directed perceived effectiveness, p < .05) and change in quit intentions at immediate post-exposure (Brennan, Durkin, Wakefield, & Kashima, 2014). In addition, when both of these PME measures were included in the same analysis, only the personalized perceived effectiveness measure remained a significant predictor (p < .01). A third study demonstrated an association between PME and change in quit intentions at immediate post-exposure (p < .05) (Brennan, Durkin, Wakefield, & Kashima, 2016).

Meta-analysis.

A meta-analysis of these three studies revealed that PME predicted quit intentions, r=.256 [95% confidence interval, .103 - .398], p<.001 (Figure 2). This effect was statistically heterogenous, Q=20.73, p<.001, I²=90.35.

Figure 2.

Forest plot displaying effect sizes and 95% confidence intervals for quit intentions.

Predictive Validity of PME: Cessation Behavior

Finally, three studies examined the association between PME and smoking cessation behaviors, and two of those studies found evidence of a significant association. One study found one PME measure (personalized perceived effectiveness) to be significantly associated with changes in cessation behavior at about 1-week follow-up (p<.05), but not a second measure (ad-directed perceived effectiveness) (Brennan et al., 2014). When both of these PME measures were included in the same analysis, only personalized perceived effectiveness was a significant predictor (p<.01). Two studies examined the association of PME with quit attempts. One study found no significant association between PME and quit attempts at 2-week follow-up (Davis et al., 2013), while the other found a significant association between PME and quit attempts at 4-month follow-up (p < .01) (Davis et al., 2017).

Meta-analysis.

A meta-analysis of these three studies revealed that PME predicted smoking cessation behavior, r=.201 [95% confidence interval, .167 - .234], p<.001 (Figure 3). This effect was statistically homogenous Q=4.10, p=.129, I²=51.25.

Figure 3.

Forest plot displaying effect sizes and 95% confidence intervals for cessation behavior.

Discussion

This study represents the first synthesis of the longitudinal evidence for the validity of PME ratings in tobacco education campaigns. Our findings indicated that PME ratings were significantly associated with the majority of study outcomes. In fact, each of the 6 studies found PME to be associated with at least one outcome variable, and across the 6 studies, PME was longitudinally associated with message recall, conversations about ads, beliefs about smoking and quitting smoking, quit intentions, and smoking cessation behaviors. Meta-analyses demonstrated that PME predicted quit intentions and cessation behavior, revealing effects that were small-to-medium in magnitude. Our results suggest that PME provides some predictive value as to the likely effectiveness of messages, echoing and bolstering some previous PME findings (Bigsby et al., 2013; Dillard, Shen, et al., 2007; Dillard, Weber, et al., 2007).

Our most important findings concern quit intentions and cessation behavior. Quit intentions are viewed theoretically as most proximal to behavior (Fishbein & Ajzen, 2010), and they have been shown to impact behavior change in both experimental (Webb & Sheeran, 2006) and longitudinal (Vangeli, Stapleton, Smit, Borland, & West, 2011) work. One challenge noted earlier is that quit intentions predict PME – i.e., those who intend to change rate messages more favorably than those who do not intend to change (Biener et al., 2000; Davis et al., 2011; Donovan et al., 1999). To attempt to control for this confound, studies in our review either controlled for baseline quit intentions in analyses (Davis et al., 2013) or predicted change in quit intentions (Brennan et al., 2014; Brennan et al., 2016). Even with these statistical and design controls in place, PME predicted quit intentions in all 3 studies, suggesting that such ratings predict changes in quit intentions beyond differences that exist at baseline.

Cessation behavior is a critically important outcome in our study, as tobacco education campaigns aim to increase quitting behavior. Here again study authors either controlled for baseline quit attempts (Davis et al., 2017; Davis et al., 2013), or examined change in cessation behavior from baseline to follow-up (Brennan et al., 2014). With these controls in place, PME predicted subsequent cessation behavior change in 2 of the 3 studies, again suggesting that higher ratings may be a proxy for messages that are more likely to change behavior.

Our results provide support for the predictive validity of PME ratings, but the heterogeneity across this set of studies should be noted. For instance, while some studies used nationally representative samples, others used smaller convenience samples. The PME and cessation behavior measures used in these studies also varied greatly, as did many aspects of study design, such as the number of messages participants viewed and the length of follow-up. On the one hand, the fact that we observed similar findings across studies suggests that these findings may be fairly robust. On the other hand, we do not yet have enough studies to examine how the heterogeneity across studies affected study results, which reduces our ability to make clear recommendations going forward. As one example, in extant research PME studies vary greatly in how they conceptualize and assess PME (Noar, Bell, et al., 2018), including whether they assess message perceptions or effects perceptions (Baig et al., in press) as well as whether items utilize personal referents (Dillard & Ye, 2008). We do not yet have enough studies to examine which type of PME measure may better predict effectiveness outcomes. As this literature grows, meta-analyses will have the opportunity to examine how variability in study features such as measurement and sampling, as well as study design features such as experimental design and length of follow-up, affect study results. Our meta-analysis of quit intentions found significant heterogeneity among studies, suggesting the presence of moderator variables.

We should also note that PME failed to predict some outcomes (e.g., self-efficacy) and produced mixed results on others (e.g., conversations). This raises the question of what outcomes PME should be expected to predict. If the ultimate goal of smoking cessation messages is to motivate smokers to quit and stimulate them to make quit attempts, then quit intentions and cessation behaviors are key outcomes that should be examined in validation studies. However, if messages attempt to change those outcomes by influencing other processes (e.g., changing beliefs, sparking conversations), then those processes may also be important outcomes to examine. This speaks to the important issue of correspondence (Noar, Bell, et al., 2018). That is, studies should show consistency between what a given PME measure is assessing, what theoretical determinant(s) a message is targeting, and what outcome is being assessed. Therefore, if self-efficacy is being targeted with a particular message, it should be included in the PME measure itself (i.e., perceptions of increased self-efficacy due to the message) and in that case, it is appropriate to examine whether the PME measure predicts increased self-efficacy. If those circumstances are not present, then self-efficacy is unlikely to be an appropriate metric for gauging the success or failure of a given PME measure.

A clear conclusion from this review is that further work on PME validation is greatly needed. Our previous review of the use of PME in tobacco education campaigns yielded 75 studies (Noar, Bell, et al., 2018), indicating that PME is widely used across the tobacco education campaigns literature. That number increases to 92 if we include studies identified in our updated search in October of 2017. And yet only 6 of those 92 studies – or 6.5% – examined longitudinal prediction of PME. Moreover, virtually none of the cross-sectional studies in the set of 92 were designed to validate PME (for an exception, see (Bigsby et al., 2013)), indicating that PME is widely used in the campaigns literature while correspondingly little validation work has taken place. Additional validation work would have implications for a broad range of health communication campaigns, given that PME has utility across health and non-health behaviors (Dillard, Weber, et al., 2007; Noar, Palmgreen, Zimmerman, Lustria, & Lu, 2010; Yzer et al., 2015). In tobacco, validation studies that move beyond studies of adult smokers are needed, such as those focused on youth cigarette and e-cigarette prevention.

It is worth speculating as to why so few validation studies on PME been published in the literature. One reason may be that PME seems intuitive and researchers may assume that it provides data that usefully inform messaging decisions. Another reason may be the difficultly in designing studies that convincingly demonstrate the validity of PME, as well as the resources needed for such studies. Robust validation studies benefit from large samples, potent messages, high message exposure, and longitudinal assessments. Significant resources are needed for such studies, especially for studies examining behavior change, which take time to observe effects. For instance, Davis et al. posited that the reason their 2013 study did not observe significant effects on quit attempts was the modest 2-week timeframe (Davis et al., 2013), which may not have been enough time for a sufficient number of smokers to make quit attempts. Their later 2017 study (with a 4-month follow-up timeframe) did find such effects (Davis et al., 2017). It is likely that robust longitudinal studies will only be possible in cases where adequate resources can be garnered, such for large national campaigns with significant evaluation budgets.

The field needs additional validation studies in tobacco and other health areas, and not all studies will require or be able to garner such resources. Given the difficulties in planning and executing large-scale longitudinal studies, additional ways to design cross-sectional experiments for PME validation should be considered. For instance, to date only a modest number of cross-sectional experimental validation studies exist in the tobacco education campaigns literature (Bigsby et al., 2013; Popova et al., 2013), and more should be undertaken. In addition, to eliminate bias that could be introduced by using the same sample to test both PME and actual effectiveness, future studies could make use of two samples – one sample to rate messages on PME and the other to test the actual effectiveness of such messages. If PME is valid, there should be correspondence between the higher rated messages on PME and the more potent messages according to actual effectiveness metrics (O’Keefe, 2018). Finally, while all 6 of our studies were longitudinal, none included a control group, and in that manner were more observational than experimental. A future longitudinal validation study could examine perceived effectiveness across groups using messages that vary in levels of perceived and actual effectiveness, using control messages to test the presumably stronger messages against.

The most notable limitation of this review is the modest number of longitudinal studies that have examined the predictive validity of PME. Additional studies are needed to increase confidence in the findings reported here. Also, two of the studies by Brennan et al. (Brennan et al., 2014; Brennan et al., 2016) were limited in that the change in intentions outcome was the difference between pre and post-exposure measures of intention at a single timepoint, and was not a longitudinal outcome.

In conclusion, our synthesis of longitudinal studies examining the validity of PME in tobacco education campaigns provides support for the use of PME ratings – i.e., for message selection or as an early indicator of message receptivity. PME ratings predicted the majority of outcomes examined, and all 6 studies found PME to be associated with at least one outcome variable. Across the studies, PME predicted a series of outcomes including increased quit intentions and cessation behavior. This suggests that PME provides some predictive value as to the likely effectiveness of messages, although additional work using different validation designs, with other health behaviors, and among other populations is needed.