On the Conceptual Ambiguity Surrounding Perceived Message Effectiveness

Abstract

Health message quality is best understood in terms of a message’s ability to effectively produce change in the variables that it was designed to change. The importance of determining a message’s effectiveness in producing change prior to implementation is clear: The better a message’s potential effectiveness is understood, the better able interventionists are to distinguish effective from ineffective messages before allocating scarce resources to message implementation. For this purpose, research has relied on perceived message effectiveness measures as a proxy of a message’s potential effectiveness. Remarkably, however, very little conceptual work has been done on perceived message effectiveness, which renders its measures under-informed and inconsistent across studies. To encourage greater conceptual work on this important construct, we review several threats to the validity of existing measures and consider strategies for improving our understanding of perceived message effectiveness.

Measures of perceived effectiveness (PE) are a widely used empirical tool in research on health message effects (e.g., Biener, McCallum-Keeler, & Nyman, 2000; Cho & Boster, 2008; Fishbein, Hall-Jamieson, Zimmer, von Haeften, & Nabi, 2002; Lee, Cappella, Lerman, & Strasser, 2011; Niederdeppe, Farrelly, Nonnemaker, Davis, & Wagner, 2011). From an interventionist’s perspective, the interest in PE is understandable, even obvious. Clearly, if PE measures can predict the likely effects of a health message with sufficient precision, then PE can at the very least help filter out ineffective messages before allocating resources to message implementation. For this reason much of the PE literature addresses the question whether PE measures can in fact predict actual message effectiveness (for an illustrative example and discussion, see Dillard, Weber, & Vail, 2007). That literature is primarily empirical, however, and relatively little systematic work on the conceptual meaning of PE exists.

In most research PE measure selection is based on face validity, if principled at all, perhaps because the meaning of PE seems intuitively obvious. Because of the paucity of a priori theorizing about the conceptual meaning of perceived message effectiveness, little can be said with great confidence about the construct validity of PE. In this paper we address this issue. We first discuss conceptual work on PE. Next, we review variation in operationalization, dimensionality, correspondence with message objectives, and other threats to the validity of currently used PE measures. Finally, we end with ideas for advancing understanding of PE.

Foundations of Perceived Message Effectiveness Research

PE research has in part been informed by advertising research that demonstrated a causal relationship between attitude toward an advertisement and attitude toward the advertised brand (for discussion, see Davis, Nonnemaker, Duke, & Farrelly, 2012; Dillard & Peck, 2000). The mechanism at work is that of consistency; liking a message transfers to feeling favorable toward the brand or behavior that the message promotes. Dillard and Peck (2000) argued that this process may not generalize to health messages. Because health messages make adverse aspects of the particular health issue salient (Lang, Chung, Lee, & Zhao, 2005), liking a health message may not be as relevant as judgments of that message’s persuasiveness (Dillard & Peck, 2000). In a later study, Dillard, Shen and Vail (2007) further specified perceived persuasiveness such that “effectiveness should be a function of the extent to which the appeal demonstrates the severity of a problem and/or the audience’s susceptibility to it” (p. 468). Interestingly, although this is consistent with theory on risk perceptions (e.g., Rippetoe & Rogers, 1987), to the best of our knowledge questions about severity and susceptibility have not been used as empirical measures of PE (but see Pechmann, Zhao, Goldberg, Reibling, & Goldberg, 2003 for the use of PE in the context of severity and susceptibility).

An interpretation of effectiveness perceptions in terms of perceived persuasiveness is common (e.g., Kang & Cappella, 2008; Niederdeppe, Farrelly, Nonnemaker, Davis, & Wagner, 2011; Noar, Palmgreen, Zimmerman, Lustria, & Lu, 2010). After all, persuasion scholarship focuses on how messages might move audiences from one position to another, which at the very least seems relevant to understanding how effective a message is perceived to be. The persuasion literature has deconstructed “effective persuasion” into a wide array of message content and format features, audience characteristics, and contextual factors. It is the complex mix of all these factors that ultimately produces the outcome of a persuasive message (Cappella, 2006). Thus, from a persuasion perspective there are two approaches to predicting message effectiveness. One is to assess all of these factors—for example, by describing message features and then relying on the message effects literature to inform effectiveness estimates. Another is to ask message recipients to judge the persuasiveness of the message. Whether it is possible for message recipients to accurately judge persuasiveness is a question by itself (O’Keefe, 1993), but it is worth noting that a persuasion perspective has implications for conceptualizing PE. If message features affect persuasive outcomes, and the perceived effectiveness of messages that use these features is correlated with the same persuasive outcomes, then those message features should also shape PE. In other words, conceptualizing PE as perceived persuasiveness implies that message features are antecedents or at least sources of PE. For example, Ye (2013) explained PE as “a summative perception of message quality based on various message content and format elements, which can include the soundness, strength, and/or novelty of the presented evidence and arguments as well as edits, cuts, visual images, etc.” (p. 8). Thus PE is “message-oriented, reflecting assessments of message ingredients” (p. 8).

PE has roots in advertising and general persuasion scholarship, but as an area of interest unto itself, PE should be characterized as under-conceptualized. In this regard Dillard and Ye (2008) wrote that most work has used perceived message effectiveness as a “conceptual primitive.” To improve this situation, Dillard and Ye started with a working definition of PE as “an estimate of the degree to which a persuasive message will be favorably evaluated by recipients of that message” (p. 150). They used this definition to address two questions in need of further scrutiny: What are the evaluative dimensions of PE, and which referents do message recipients consider when evaluating a message’s effectiveness? Although their work did not yet inform a theory of PE—which would include its definition, antecedents, and outcomes—it illustrated that measures currently used to assess PE do not reflect theory-based propositions about its meaning. It is important, then, to consider extant PE measures to identify their strengths and weaknesses. We turn to the question of construct validity next.

Existing PE Measures: Threats to Construct Validity

For illustrative purposes we reviewed studies in the health communication domain that used PE measures. To identify studies for our review, we searched for empirical studies that used measures labeled by the studies’ authors in terms of perceived message effectiveness and that were used in the context of health messages. We searched in Google Scholar and in our own personal database, and for each manuscript that we found we next examined its reference list for additional relevant studies. This strategy yielded 23 studies across 22 manuscripts. Table 1 has a description of these studies, including information about PE operationalizations, dimensional structure of PE measures, and antecedents and outcomes of PE.

Table 1.

Operationalizations, Dimensionality, Antecedents, and Outcomes of PE in 23 Health Communication Studies

Study	Perceived message effectiveness operationalization	Dim.	Antecedents	Actual effectiveness operationalization
Anderson, Noar, & Rogers, 2013	Powerful; believable; attention grabbing; memorable; likeable; effective; convincing (to respondents as well as to similar others)	1	Message features (attitudinal, normative, efficacy theme); socio-demographic variables	-
Andsager et al., 2006	Interesting; relevant; aimed at people like me; aimed at people of my race; aimed at people my age; provides useful information; tries to help me	1	Message features (perceived similarity); gender	-
Bigsby et al., 2013	Convincing; helped me feel confident about how to best deal with smoking; put thoughts in my mind about quitting smoking; put thoughts in my mind about continuing to smoke	1	-	Intention
Brennan et al., 2014	Made me stop and think; made a strong argument for quitting; taught me something new; was relevant to me; made me feel concerned about my smoking; made me feel motivated to quit smoking	2	-	Intention
Cho & Boster, 2008	This ad was persuasive; I feel that the ad made its point effectively; I feel that the ad was convincing; both the content and style of the ad were good; the ad was compelling	1	Perceived identification; informativeness; realism	Perceived effect on self; attitude; intention
Davis et al., 2013	These ads are worth remembering; these ads grabbed my attention; these ads are powerful; these ads are informative; these ads are meaningful; these ads are convincing	1	Message features (reasons vs how to quit; graphic vs testimonial)	Emotions; outcome beliefs; confidence; intention; behavior
Dillard & Peck, 2000 Study 1	Persuasive; convincing	1	Recipient features (emotion-focused, not emotion-focused); cognitive responses; affective responses	Attitude toward issue
Dillard & Peck, 2000 Study 2	Persuasive; convincing	1	Intensity of affective response	Attitude toward issue
Dillard & Ye, 2008	Persuasive; effective; convincing; compelling; reasonable; logical; rational; true to life	2	Use of particular referents	-
Fishbein et al., 2002	Was the message convincing?; would it be helpful in keeping your friends from using the drug?; would people your age who have never used drugs be more or less likely to want to try the drug after seeing the ad?; how confident did the ad make you feel about how best to deal with illegal drugs in the real world?	1	Message features (target drug, dramatic portrayals); perceived danger; perceived harmfulness; perceived norms; perceived realism; positive affect; negative affect	-
Hammond et al., 2012	Extent to which message would increase concern about health risks; motivate smokers to quit; prevent youth from smoking; also a measure of overall effectiveness (retained for analysis)	1	Message features (graphics, testimonials); socio-demographic variables	-
Jensen, King, Carcioppolo, & Davis, 2012	Convincing; would people your age who have never been screened be more likely to get screened after reading the pamphlet?; would the pamphlet be helpful in convincing your friends to be screened regularly?	1	Message features (tailored information, visuals)	Intention
Kang & Cappella, 2008	I found this PSA persuasive; I did not find this PSA convincing; I think people different from me would find this PSA persuasive	1	Affective responses; message features (message sensation value)	-
Kang, Cappella, & Fishbein, 2009	This ad was convincing; the ad said something important to me; watching this ad helped me feel confident about how to best deal with using marijuana; if my friends were offered marijuana, this ad would help keep them from using marijuana	1	Message features (marijuana in scenes, argument strength); recipient features (risk for marijuana use)	Perceived effectiveness; liking;, predominant valence
Lee et al., 2011	This ad was convincing; watching this ad helped me feel confident about how to deal with smoking	1	Message features (smoking cues, argument strength)	-
Mitchell, Brown, Morris-Villagran, & Villigran, 2001	The message I saw was compelling; dumb	1	-	Negative thoughts; attitude
Murphy-Hoefer, Hyland, & Higbee, 2008	If I saw this advertisement on TV it would catch my attention; this advertisement influenced me about whether or not to smoke cigarettes; I will talk with my friends about this ad	1	Message features (theme: social norms, health consequences, tobacco industry; design: humor, sarcasm, testimonial, drama)	-
Niederdeppe et al., 2011	How much the ad made me stop and think; how much the ad grabbed my attention; how believable the ad was; how much the ad made me want to quit smoking	1	Message features (reason and how to quit); socio-demographic variables	-
Noar et al., 2010	How effective would a PSA be at persuading someone to use a condom more often; persuading someone to talk to their sexual partner more often about using condoms	1	Message features (perceived threat, attitudes, norms, efficacy, stages of change, sensation value); recipient features (socio-demographic variables); affective responses; cognitive responses; perceived utility	-
Paek, Hove, Kim, Jeong, & Dillard, 2012	Persuasive; effective; convincing; compelling for self/close peer/distant peer/average parent	1	-	Issue importance; intention
Pechmann et al., 2003	Effective for kids my age	1	Message features (theme: disease and death, endangering others, cosmetics, negative lifestyle, refusal skills, marketing tactics, manipulation)	-
Ye, 2013	Effective; convincing; compelling; informative	1	Happiness; fear; anger	Perceived influence; intention
Yzer et al., 2011	Convincing; believable; memorable; good; pleasant; positive; for someone like me	2	Arousal; valence	-

Note. Dim. = number of PE dimensions. Actual effectiveness (AE) variables were included when PE was used to explain the particular AE variable

Variation in operationalization

The range of PE measures described in Table 1 underscores that there is no single agreed upon PE scale. Although some measures are more popular than others, an array of PE measures exists. The most widely used measures ask how “effective,” “convincing,” or “persuasive” a health message is, whereas other measures include such diverse items as “made me stop and think,” “rational,” “dumb,” and “catch attention.”

The problem with these different operationalizations is that even though they are semantically different, each has been assumed to be a valid indicator of PE. A consideration of the predictive validity of PE is illustrative. For example, Pechmann et al. (2003) asked participants to rate anti-smoking messages using the statement, “Overall, I think this ad is effective for kids my age.” The 56 messages Pechmann et al. used were rated as highly similar on the perceived effectiveness measure (M = 3.5; range 3.22–3.65 on a 5-point scale). However, the messages differed in effects on smoking beliefs and intention, which was interpreted as evidence that PE does not predict message effects on outcomes such as attitude and intention. Pechmann therefore concluded that “we do not recommend that advertisements be evaluated on the basis of viewers’ ratings of perceived ad effectiveness” (p. 15). In contrast, other investigators who used different PE measures found that PE is in fact associated with message effects on attitude and even behavior. Dillard, Shen and Vail (2007), for example, asked participants across four studies to rate 15 messages about various topics, including flossing, seat belt use, and alcohol consumption, using items such as how convincing and believable each message was. Each of their four studies showed that PE causally shaped attitude. Brennan, Durkin, Wakefield, and Kashima (2014) used a set of items that they called “personalized PE,” which included ratings of how much two anti-smoking messages were personally relevant, made participants concerned about their smoking, and motivated them to try to quit. These PE items were associated with changes in both quitting intention and actual quit attempts assessed three weeks after PE ratings were obtained. Although the Pechmann, Dillard, and Brennan studies differed on a number of factors, they very clearly differed from one another in the PE measures they used. Because of the different results that these three illustrative studies produced, any conclusion about whether PE should be used for pre-implementation research in intervention design is premature and remains unsubstantiated until formal validity research that compares different PE items is available.

These illustrative findings more generally highlight the need to understand and improve the correspondence between PE conceptualization and operationalization. Seeking consensus on which of the available items are superior PE measures can be an important part of such endeavors. However, we should also address the possibility that the variation in PE operationalizations do not necessarily pose a problem. From this perspective the various measures used to assess PE may reflect different aspects or dimensions of a broader latent PE construct. We turn to that question next.

Dimensionality of PE measures

Previous research has used a variety of measures, but PE has been treated as a unidimensional concept nonetheless. This may not be an issue if these different measures in some way all measure PE. For example, Noar et al. (2010) observed that their measures of cognitive and affective reactions, personal utility, and PE correlated with each other,

raising the possibility that there may be an underlying dimension related to “positive feelings” toward a given PSA. For example, it is possible that perceived effectiveness itself is a broad dimension that could be better assessed through a multiple item scale including items assessing cognitive reaction, emotional reaction, and personal utility, among others. (p. 41)

This is an empirical observation, and, as Noar also points out, a broader conceptual view of PE should go together with empirical work.

We know of three studies in the health communication domain that purposefully explored PE’s dimensionality. In each of these a set of PE measures was submitted to factor analysis. Yzer, Vohs, Luciana, Cuthbert, and MacDonald (2011) found that items such as “believable” reflected a factor that they labeled convincingness, and items such as “pleasant” reflected a valence factor; furthermore, they found that a personal relevance item asking to what extent a PSA was “for someone like me” did not load on either factor. Brennan, Durkin, Wakefield, and Kashima (2014) found that items such as “offers strong argument for quitting” reflected a factor that they labeled ad-directed PE, and items such as “made me concerned about my smoking” reflected a factor that they labeled personalized PE.

This distinction was somewhat similar to one made by Dillard and Ye (2008), who, to the best of our knowledge, were the first to purposively test the dimensionality of PE. Dillard and Ye built on thesis research by Grillova (2002) to empirically differentiate between items such as “persuasive” and “logical,” where the former was thought to reflect message impact and the latter message attributes. Impact and attribute factors correlated strongly with each other (Dillard & Ye, 2008) and had similar associations with attitude (Dillard, Weber, & Vail, 2007). Dillard and colleagues concluded that although impact and attribute PE dimensions can be empirically demonstrated, they may not have very different consequences and therefore researchers may combine impact and attribute items and other items in a single measure (Dillard, Weber, & Vail, 2007; Dillard & Ye, 2008). We do not fully subscribe to this view. Ultimately, conceptual work followed by empirical tests is needed to determine whether PE truly is multidimensional or whether some measures that have been used to assess PE in fact measure something else. There is not yet sufficient evidence to accept that the array of current PE measures can be used interchangeably or that pooling different measures in a unidimensional scale does not lead to important information loss.

Similar measures, variation in conceptual labels

The conceptual ambiguity of PE is further illustrated by the use of measures that some investigators use to assess PE but other investigators use to assess different concepts. For example, Noar et al. (2010) measured personal utility—which they argued is relevant for but distinct from PE—by asking participants “(a) how useful the PSA was in terms of giving them information, ideas, or skills they could use, and (b) how relevant it was to them, meaning that it made them think of their own life” (p. 30). Others used these same or highly similar items to measure PE (e.g., Dillard & Ye, 2008; Fishbein et al., 2002). Or consider Ye (2013), who measured presumed influence by asking respondents about expected effects on themselves and others, which is similar to measures that others have used to assess PE (e.g., Cho & Boster, 2008).

Perhaps even more illustrative is work that does not refer to PE at all but clearly would be deemed relevant by investigators who have used PE. Biener, McCallum-Keeler, and Nyman (2000), for example, were interested in what they called receptivity to anti-smoking efforts, and for this purpose they developed “assessment of ad characteristics” measures. These asked to what extent an ad was sad, frightening, funny, believable, thought provoking, silly, confusing, emotionally moving, entertaining, offensive, phony, reassuring, helpful, and interesting. These items were combined into scales labeled positive emotions, negative emotions, strength of emotion elicited, cognitive quality, thought provoking, and helpfulness. Note that none of these scale labels directly refer to effectiveness, although some scales, such as cognitive quality and helpfulness, and individual items, such as believable and helpful, are similar to those that other investigators have used to measure PE (e.g., Bigsby, Cappella, & Seitz, 2013; Dillard, Shen, & Vail, 2007). This example illustrates that published research exists that could potentially advance understanding of PE, yet might fail to do so because investigators do not always perceive their work as PE research.

Correspondence between measures and objectives

PE measures typically ask about the message’s effectiveness in general—for example, by asking how persuasive a message is. Yet measures that are generally framed do not align with the objectives of theory-based message design. Health messages can seek to inform about basic health facts, raise awareness, offer social support, or improve self-efficacy, to name a few possibilities. A health message is effective if it produces positive change in the variables that it seeks to change. Thus, if the reason for using PE is to obtain a pre-implementation sense of a message’s ability to meet its objectives, then PE should determine the likelihood that a particular message will produce the specific effects that it was designed to have. An inspection of PE measures (as described in Table 1) shows that most do not reflect this at all: While the ultimate goal of PE measures is to determine potential effectiveness in producing change in beliefs about a health behavior, PE measures typically do not directly measure potential belief change. (But notable exceptions exist; e.g., Fishbein et al., 2002; Ye, 2013.)

Correspondence with implied processing style

A number of PE measures appear to assume a cognitively demanding message processing style. These measures’ evaluative anchors (e.g., plausibility, reasonableness, and logic) indicate that they are primarily relevant when message recipients scrutinize the strength of arguments used in a particular message. In contrast, information processing and message effects research make clear that most messages are processed based on heuristic rules that require little if any cognitive effort. The very nature of health messages may trigger heuristic processing. For example, writing about video public service announcements (PSAs), Dillard and Peck (2000) noted that

the message features that define PSAs align well with the conditions that are thought to prompt heuristic processing. PSAs are brief, running between 10 and 60 seconds in length. In addition, they are typically designed to make a single point in a straightforward manner, and they lack much argumentative structure and, therefore, offer little grist for the mill of systematic processing. These observations imply that individuals who are exposed to PSAs may have little choice but to gravitate toward heuristic processing. (p. 463)

If we accept this logic, then the almost singular emphasis in PE measures on cognitively effortful evaluations of a message’s argument strength indicates an important mismatch between measures of the perceived effectiveness of health messages and the style by which these messages are most likely processed. We raise two questions here: First, are extant PE measures (unintentionally) forcing participants to elaborate more than they might in a non-lab situation, thus influencing PE results, and second, are there measures that are able to capture the inherent heuristic response from the health messages in question?

Confounding concerns

O’Keefe (1993) wrote that “respondents who are asked whether they would probably be influenced by a given message will naturally base their answers on their (implicit or explicit) beliefs about what persuades” (p. 231). Similarly, it is possible that variables that may not necessarily have much to do with effectiveness affect PE ratings. For example, the more one believes that risky behaviors have harmful consequences, the more one might view health messages about those behaviors as effective. It is also conceivable that harmful consequences of some drugs, such as meth, are easier to imagine than those of others, such as marijuana.¹ In support of such possibilities Fishbein et al. (2002) found that in their sample of anti-drug PSAs, anti-meth and anti-heroin PSAs were rated as more effective than anti-marijuana PSAs. Findings from our lab replicate these findings. We used a pool of 79 anti-drug PSAs that included 34 PSAs that targeted marijuana, 20 that targeted cocaine or heroin, 16 that targeted meth, and 9 that targeted non-specified drugs. A sample of 190 adolescents rated these PSAs on 7-point perceived convincingness and perceived pleasantness scales (Yzer, Vohs, Luciana, Cuthbert, & MacDonald, 2011). With PSA as the unit of analysis, the results showed that anti-meth PSAs were rated as more convincing (M = 5.51) than anti-marijuana (M = 4.33), anti-non-specified drugs (M = 4.66), and anti-cocaine/heroin PSAs (M = 4.77; F(3, 75) = 14.71, eta² = .37). At the same time, anti-meth PSAs were rated as more unpleasant (M = 2.53) than anti-marijuana (M = 3.99), anti-non-specified drugs (M = 4.04), and anti-cocaine/heroin PSAs (M = 3.62; F(3, 75) = 13.20, eta² = .35). This suggests that when people evaluate a health message, they may think of the health issue at hand, and their evaluation in part reflects the implicit notion that for a health issue so serious the message ought to be effective. This may not, however, mean that people feel that the message is truly effective for themselves.

Specification of referents

The possibility that participants’ effectiveness ratings do not fully reflect perceptions about possible effects on themselves also highlights the importance of understanding who people think of when responding to PE measures. For example, three items in Fishbein et al.’s (2002) PE scale explicitly asked about referents, i.e., friends, peers who have never used drugs, and the participant her/himself. Other researchers have approached this issue more from a third-person effects perspective. Ye (2013), for example, differentiated between PE and presumed influence. Presumed influence asked about expected message effects on themselves, a member of the relevant behavioral group (e.g., a typical binge drinker), and referents whom participants proposed. Similarly, in addition to PE measures, Cho and Boster (2008) asked about perceived effect on self and perceived effect on others.

These examples contrast with most other PE measures, which ask about effectiveness in some way but do not specify for whom the message is effective. For instance, one of the most widely used PE measures asks how convincing a message is but typically does not specify that the rating of interest concerns the message evaluators themselves. Our own data, perhaps unfortunately, can be used to illustrate why personalizing PE measures is a good move (see also Brennan et al., 2014). The questions in our convincingness scale did not specify a referent. We used a separate question to ask how much each message “said something important to me” (Yzer et al., 2011). Across the 79 anti-drug PSAs in our pool, the correlation between the convincingness and personal relevance measures ranged from .01 to .86. Moreover, we found that PSAs were often perceived as more convincing than personally relevant: Across PSAs mean differences between the two 7-point convincingness and personal relevance scales ranged from .01 to 3.09, and for 38 out of 79 PSAs mean differences were greater than one scale point. These findings suggest that convincingness (and perhaps other PE) measures that do not specify referents produce ratings that do not always reflect a consideration of effects on oneself. More generally, unspecified PE measures have referent anchors that are moving targets, which may reduce the precision with which PE ratings can predict message outcomes.

Antecedents of PE

A better understanding of the conceptual meaning of PE can be informed by research on its antecedents. These can include message recipient characteristics (e.g., Bigsby, Cappella, & Seitz, 2013; Niederdeppe et al., 2011), structural message features (e.g., Andsager, 2006; Noar et al., 2010), and mediating processes such as social interaction (e.g., Brennan et al., 2014) and immediate brain responses to message exposure (e.g., Ramsay, Yzer, Luciana, Vohs, & MacDonald, 2011), among others. Determining antecedents would not only clarify conceptual ambiguity but it also would aid the design of health messages. Understanding the factors that influence PE can potentially eliminate some of the guesswork, time, and costs associated with creating an effective health message. For example, the finding that message sensation value positively affects PE (Noar et al., 2010) directs the interventionist to a toolbox of design options for strengthening a message’s sensation value, such as the use of intense images, sound saturation, and narrative content delivery (e.g., Morgan, Palmgreen, Stephenson, Hoyle, & Lorch, 2003).

There is some evidence that message features and message recipient characteristics shape PE through a matching principle, such that messages are perceived as more effective if message format matches message recipients’ personality, value system, or other characteristics (Hullett & Boster, 2001; Noar et al., 2010). Perhaps particularly promising for advancing the conceptual basis for PE is work that focuses on message-induced, affect-based motivational processes. For example, Dillard and colleagues (Dillard & Peck, 2000; Dillard, Shen, & Vail, 2007) found that PE ratings of various PSAs were shaped by discrete emotions that the PSA induced, such as happiness, sadness, fear, and anger. Other work that built on the idea (first advanced in the emotion literature) that arousal and valence indicate approach and avoidance motivational systems found that arousal measured in real time during exposure to PSAs predicted the perceived convincingness of PSAs, and real-time valence predicted perceived pleasantness of PSAs (Yzer et al., 2011). These lines of research suggest that affect-based processes may explain how message features shape PE, which offers ground for advancing PE theory (Bigsby et al., 2013).

Outcomes of PE

Additionally, we can clarify the conceptual meaning of PE by reviewing its associated outcomes. Unfortunately, not much is yet known about the processes and outcomes that PE might affect. As Table 1 illustrates, outcomes of PE, or, as commonly labeled, actual effectiveness (AE), are mostly limited to attitude or intention regarding a particular health behavior (but see Davis et al., 2013). To be sure, there are good theoretical reasons to accept that attitude and intention are potentially important behavioral determinants, but a message should be evaluated on its ability to produce change among those variables that the message was designed to change—and this is not necessarily always attitude or intention, but rather it includes “any of the variety of outcome measures that have traditionally been of interest to applied and theoretical persuasion researchers” (Dillard, Shen, & Vail, 2007, p. 468). For example, the effectiveness of a message that was designed to strengthen self-efficacy regarding the performance of a particular behavior is better determined by correlating PE with self-efficacy than with attitude.

The issue is not simply the number of possible indicators of AE. Persuasive processes are a complex sequence of mediated (and moderated) steps, and thus simple analyses of direct message exposure effects on ultimate outcomes such as attitude, intention, and behavior may lead to under-specification of explanatory processes. If we agree that messages do not directly affect variables such as attitude and intention, then we can use this idea to theorize which mediating variables PE might affect. For example, are messages that score high on PE more likely to induce conversation about the health topic? Do these messages affect specific beliefs about the behavior that were addressed in the message? These and other possibilities demand further scrutiny.

Although considering additional mediating and outcome variables may advance PE theory, we must note that there is evidence from survey, experimental, and meta-analytic research that, across messages and measures, PE to some extent predicts message effects on attitude and intention. For example, in a meta-analysis of 40 effect sizes, Dillard, Weber, and Vail (2007) found an average effect size of r = .41 (95% CI=.38/.43) for the PE-AE relationship, and other work shows that PE influences outcomes such as attitude instead of preexisting attitudes shaping PE (Dillard, Shen & Vail, 2007). Note also that work that has not found support for the PE-AE relationship likely remains unpublished, but it would offer real opportunities for examining when and for which outcome variables PE measures are consequential.

Advancing PE Theory and Measurement

In sum, our review of the literature reveals that there is no consensus on the conceptual definition of PE, and when a definition is offered, unsophisticated conceptualizations are typically proposed. This is problematic for myriad reasons, but specifically it may cause issues for future research. If the corpus of literature on PE is inconsistent with its conceptual definition, then how should future studies define and study it, and how are we to interpret findings?

Before closing with our own summary perspectives on the conceptualization and measurement of PE, we must first note that no conclusive recommendations can be offered with any chance of broad acceptance until theory-driven validation studies become available. A pertinent example is the concept of message exposure. Similar to PE, message exposure has clear significance for understanding health message effects, and as a concept it has intuitive meaning (Slater, 2004). However, until recently remarkably little systematic work on the measurement of exposure had been done (Fishbein & Hornik, 2008), and perhaps as a result, many distinct exposure measures have been in use (Morris, Rooney, Wray, & Kreuter, 2009). Recent validation research in the cancer communication domain has contributed to a better understanding of the conceptualization and measurement of exposure, and we argue that a similar approach should be a next step in research on PE.

Briefly, Hornik and colleagues (Nagler & Hornik, 2012; Romantan, Hornik, Price, Cappella, & Viswanath, 2008; Tan & Hornik, 2014) took a two-step approach. They first categorized various exposure measures based on shared item features (e.g., content specificity, obtrusiveness, and provision of message examples). Next, they evaluated distinct measures’ performance against a set of validity criteria, including convergent validity, predictive validity, and—perhaps particularly important for conceptualization objectives—nomological validity (Cronbach & Meehl, 1955). This work has led to specific recommendations for exposure measures that are likely to be the most valid and relevant when studying topics such as exposure to cancer information, conflicting health information, and cancer-related direct-to-consumer advertising. If such measures are used in future cancer communication research, then they have the potential to greatly improve measurement consistency. A similar validation approach can be useful for testing the performance of available (or new) items as measures of PE. PE measures can be categorized on dimensions such as the ones identified in this review, i.e., convincingness, valence, message impact, and message attributes. A comparison of these categories of measures can at the very least improve measurement consistency.

Of course, empirical validation research needs a conceptual benchmark, and that, as we have argued, is missing from the PE literature. To fuel a discussion on how to best conceptualize PE, we offer our position here. To clarify the ambiguity surrounding PE, we argue for conceptual parsimony: On the basis of our review, we define PE as the extent to which a message recipient believes that a health message will affect him or her personally in terms of the particular message objectives.² This definition has a number of strengths. From a conceptual perspective, the definition delineates the meaning of PE by positioning PE in a process framework of health message effects, in which PE is a mediating state with distinct antecedents and consequences. This clarifies that many measures that are currently used to assess PE in fact might be better seen as antecedents of PE rather than PE itself. For example, perceptions of how much a message will affect someone can be—but not necessarily always have to be—shaped by perceptions of message attributes (e.g., how strong an argument is or how attention-grabbing a message is) or affect induced by a message (e.g., how pleasant or unpleasant a message is). To stay with these examples, attention arguably is a prerequisite for any message to be perceived as effective (or ineffective), but that does not make “attention-grabbing” conceptually the same as PE. Similarly, the definition distinguishes between PE and its consequences, and as such is sensitive to the wide array of variables that different health messages are designed to affect, including, among others, awareness, knowledge, belief change, change in attitude, normative perceptions, self-efficacy, and behavior change.

From a measurement perspective, the definition allows item stem templates (e.g., “How likely is it that this message will ____” or “To what extent is this message effective in ____”) that can be adapted to any message objective (e.g., “____ help[ing] you understand what puts you at risk for diabetes” or “____ make [making] you feel confident that you can use a condom every time you have sex with a new partner”). This produces measures that can be compared across studies yet are tailored to a particular study’s messages. Note also that our definition’s emphasis on expected effects does not make assumptions about message processing styles. Thus PE ratings obtained by measures consistent with our proposed definition are not affected by the depth of processing implied by some currently available measures, such as items asking how plausible a message is. For example, heuristic processing can lead people to get the gist of a message, but they would not be able to comment on the plausibility of the message (as that would have required deeper message scrutiny; Reyna, 2008). In contrast, asking directly about likely effects on themselves would be possible.

In sum, the unidimensional definition of PE that we propose stays as close as possible to what we talk about when we talk about PE: perceptions of how effective a message is in producing intended effects on message recipients. This definition and the measures derived from it solve many problems associated with the array of currently used measures. Our main argument is that many of those measures may in fact not measure PE, but this position needs empirical support. Although at least one study found that a personalized and effect-focused measure outperformed other measures as a predictor of smoking cessation (Brennan et al., 2014), our arguments will be much more persuasive if we systematically test and demonstrate that our measure is distinct from message attribute, message-induced affect, and other measures, and that it mediates effects of these variables on outcomes that are relevant for the messages under study. Such systematic validation research is not currently available. With the objective of advancing understanding and practical utility of PE, we call on other investigators to join us in prioritizing such PE validation research and to engage in further discussion of the conceptualization of PE.