Systematic Review of Measures Used in Pictorial Cigarette Pack Warning Experiments

Abstract

Background

We sought to describe characteristics and psychometric properties of measures used in pictorial cigarette pack warning experiments and provide recommendations for future studies.

Methods

Our systematic review identified 68 pictorial cigarette pack warning experiments conducted between 2000 and 2016 in 22 countries. Two independent coders coded all studies on study features, including sample characteristics, theoretical framework, and constructs assessed. We also coded measurement characteristics, including construct, number of items, source, reliability, and validity.

Results

We identified 278 measures representing 61 constructs. The most commonly assessed construct categories were warning reactions (62% of studies) and perceived effectiveness (60%). The most commonly used outcomes were affective reactions (35%), perceived likelihood of harm (22%), intention to quit smoking (22%), perceptions that warnings motivate people to quit smoking (18%), and credibility (16%). Only 4 studies assessed smoking behavior. More than half (54%) of all measures were single items. For multi-item measures, studies reported reliability data 68% of the time (mean α = 0.88, range α = 0.68–0.98). Studies reported sources of measures only 33% of the time and rarely reported validity data. Of 68 studies, 37 (54%) mentioned a theory as informing the study.

Conclusions

Our review found great variability in constructs and measures used to evaluate the impact of cigarette pack pictorial warnings. Many measures were single items with unknown psychometric properties. Recommendations for future studies include a greater emphasis on theoretical models that inform measurement, use of reliable and validated (preferably multi-item) measures, and better reporting of measure sources.

Implications

Robust and consistent measurement is important for building a strong, cumulative evidence base to support pictorial cigarette pack warning policies. This systematic review of experimental studies of pictorial cigarette warnings demonstrates the need for standardized, theory-based measures.

Implementation of large pictorial warnings on cigarette packaging has emerged as a key tobacco control strategy globally. Cigarette packages provide a combination of high exposure, nearly universal reach to smokers, and very low cost as a medium for conveying health information.¹ Article 11 Guidelines (packaging and labeling of tobacco products) of the World Health Organization Framework Convention on Tobacco Control (FCTC) specify that health warnings should cover at least 50% of the front and back of each package.² The FCTC also recommends cigarette pack warnings include pictorial elements.² To date, more than 100 countries and jurisdictions covering almost 60% of the world’s population have pictorial warnings on cigarette packs.³

As pictorial warnings have expanded globally so has research on their impact. However, what types of constructs and measures are being used to evaluate these warnings, and do they conform to best practices in measurement? A 2008 report from the International Agency for Research in Cancer, which examined multi-item measures used in warning studies, recommended assessing several constructs when evaluating the effectiveness of cigarette pack warnings, including smoking behavior.⁴ A recent review of the effect of pictorial warnings on cigarette packs, which concluded that evidence was insufficient to determine whether the warnings reduce smoking, discovered substantial differences in the studies’ assessment of smoking behavior such that the authors could not systematically compare across studies.⁵ However, other reviews and meta-analyses conclude that pictorial warnings affect many smoking-related outcomes.^6–9 Robust and consistent measurement is important for building a strong, cumulative evidence base on pictorial warnings, which is necessary to advance the science and withstand legal challenges in the United States and globally.

The purpose of our study was to characterize the measures used in experimental studies of pictorial cigarette pack warnings by conducting a systematic review of this literature. We sought to describe the characteristics and psychometric properties of the measures. Based on the findings, we sought to provide recommendations for developing, selecting, and reporting measures in future studies.

Methods

Search Strategy

We located relevant studies using a comprehensive search strategy involving 3 steps, which we originally reported in a meta-analysis on the effects of pictorial cigarette pack warning experiments.⁸ First, we searched PsycINFO, Pubmed, Embase, Web of Science, Communication & Mass Media Complete, Business Source Complete, and CINAHL computerized databases. The search terms were (cigarette* OR tobacco) AND (warning* OR label* OR pictorial OR graphic OR messag* OR text*). Second, we examined the reference sections of 5 narrative reviews of cigarette pack warnings.^1,10–13 Third, we examined the reference lists of the final set of articles included in our review. We included all full-text reports identified in our searches, including peer-reviewed journal articles, books chapters, and gray literature (e.g., dissertations and publicly available reports).

To be included, a study had to use an experimental protocol that evaluated warnings intended for cigarette packs. Studies had to report data on both a pictorial warning condition and a text-only condition. We excluded studies of noncigarette tobacco products such as cigars, smoking cessation public service announcements or multi-component interventions, and warnings embedded in cigarette advertising. We excluded observational studies that asked individuals to recall or report on warnings that they had seen on their own prior to the study. While such studies are pertinent to understanding pictorial cigarette pack warnings, observational methods are quite different as are some of the constructs and measures. (For a review of the observational literature, see Noar and colleagues.^7,9) Finally, articles reporting the studies had to be available in English.

We originally conducted our search in April 2013 for a meta-analysis of pictorial warning experiments, covering the period between 2000 and 2013, which yielded 37 studies from 35 articles/reports.⁸ We updated the search using the same search terms and procedures in April 2016 for this measures review (Figure 1). In the updated search, we identified 4041 references after removing duplicates. Two reviewers independently examined all study titles, reducing the number to 193. They subsequently reviewed abstracts, further reducing the number to 60. Then, the 2 reviewers independently examined the full text of these articles/reports and tracked reasons for study exclusion. If the 2 reviewers made a different determination about a particular article classification, they consulted with a third individual to resolve the discrepancy and make a final determination. The updated search yielded 31 studies from 28 articles/reports. Thus, the entire review includes 68 studies from 63 articles/reports.

Figure 1.

PRISMA flow diagram showing study screening process.

Article Coding

Two authors independently coded the studies (k = 68) using a standardized data abstraction form. They coded all articles on several features, including sample characteristics, application of a theoretical framework, constructs assessed, and characteristics of measures used to assess each construct. Characteristics of measures included the number of items per construct, wording for each item, response scale, construct source, and reliability and validity data. Coders examined the exact item wording or description (rather than what the authors called the measures).

Given the diversity of measures included in this review, we used the Message Impact Framework (MIF) to classify measures, organize reporting of results, and facilitate interpretation of findings (Figure 2).⁸ The framework is based on communication and psychological theories^14–20 as well as tobacco warnings theory and research.^4,21–23 The MIF has 5 theory-based categories. Attention and recall are participants’ attention to warnings and ability to recognize or recall the warnings. Warning reactions are participants’ cognitive, emotional, and physiological reactions to warnings. Knowledge/attitudes/beliefs are participants’ smoking or cigarette pack-related objective knowledge, attitudes, and beliefs. Unlike warning reactions, measures in the knowledge/attitudes/beliefs categories are typically about smoking and not about the warnings themselves. Intentions are participants’ intentions or willingness to act. Intentions also include actions intended to lead to quitting such as information seeking or calling a quitline. Perceived effectiveness refers to participants’ beliefs about how well the warnings work. (See Table 1 for sample measures from each category.) Most categories had perfect agreement, and the mean percentage agreement was 97%. Cohen’s k²⁴ had a mean of 0.95.

Figure 2.

Message Impact Framework with number of studies measuring each construct.

Table 1.

Sample Outcomes Assessed in Experimental Pictorial Warning Studies and Examples of Items/Scales

Construct	Example Item
Attention and recall
Attention attracting	Which health warning is most likely to attract your attention?1
Recall of warning	Try to recall what the warning information on the package stated and type it in the box below.2
Recognition of warning	Which one of the following pictures do you recall seeing on the pack of cigarettes you viewed earlier in the survey?3
Warning reactions
Cognitive elaboration	How much did you think about the harmful effects of smoking?4
Affective reactions	How much did the warning on your cigarette packs make you feel…anxious/not anxious at all, nervous/not nervous at all, fearful/not fearful at all, uneasy/not uneasy at all, upset/not upset at all.5
Credibility	The warning label is believable. The information on the warning label is truthful.6
Message reactance	Participants were asked how irritated, angry, annoyed, and aggravated the warnings made them.7
Knowledge, attitudes, and beliefs
Smoking attitudes	Cigarettes taste good. I enjoy the taste sensations while smoking. When I smoke, the taste is pleasant. I enjoy the flavor of a cigarette. If I’m tense, a cigarette helps me relax. When I’m angry, a cigarette can calm me down. Cigarettes help me deal with anxiety or worry. Smoking calms me down when I feel nervous.8
Perceived likelihood of harm	How likely do you think it is that smoking cigarettes regularly would cause you to develop each of the following [lung cancer, heart disease, mouth or throat cancer] in the next 10 years?9
Perceived severity of harm	I personally consider such a health risk to be a very serious health threat. If I was exposed to such a health risk, this would have considerable consequences on my life.10
Response efficacy	If I quit smoking, my risk to be threatened by such a health risk in the future, will decrease. To quit smoking would be generally an effective method to reduce the risk of such a health hazard.10
Self-efficacy to quit	I believe I have the ability to quit smoking in the next month. I see myself as being capable of quitting smoking in the next month. I feel I have personal control over quitting smoking in the next month. My quitting smoking in the next month would be difficult. I am confident that I will be able to quit smoking in the next month.11
Intentions
Intention to not start smoking	Do you think that you will smoke a cigarette at anytime during the next year?12
Intention to quit smoking	How interested are you in quitting smoking in the next month? How much do you plan to quit smoking in the next month? How likely are you to quit smoking in the next month?13
Behavior
Smoking behavior	During the past 21 days, since you joined the study, have you been refraining from smoking for at least 1 day?14
Perceived effectiveness
Motivate me to quit smoking	If you were a cigarette smoker, would the following labels make you want to quit smoking? If you were a cigarette smoker, which warning label is most likely to cause you to quit? If you were a cigarette smoker, which warning label is least likely to cause you to quit?
Motivate others to quit smoking	I think the design of this package of cigarettes would help discourage current college-aged female smokers to quit. I think the design of this package of cigarettes would help motivate pregnant women who normally smoke to not smoke while they are pregnant. I think the design of this package of cigarettes would help encourage pregnant women smokers to quit.

Results

Study Characteristics

The 68 studies were conducted in 22 different countries, with the most (51%) conducted in the United States, followed by Canada (7%), and Germany (7%). Studies were published as early as 2000; 68% were published between 2011 and 2016. Most studies used samples of smokers only (51%) while fewer included both smokers and nonsmokers (41%). Most studies (71%) included both young adults and adults but no adolescents. Thirteen (19%) studies included adolescents in their sample, while only 4 (6%) studies focused solely on adolescents. Sample sizes ranged from 25 to 15536 (Median = 309); the cumulative sample size across all studies was 75082, and the typical sample was 47% male.

Constructs and Measures

We identified 278 measures across 61 constructs (see Supplementary Appendix 1), with studies assessing between 1 and 11 constructs (M = 4.47, SD = 2.57). Frequently assessed measures included affective reactions (n = 24, 35% of studies), perceived likelihood of harm (n = 15, 22%), intention to quit smoking (n = 15, 22%), perceptions that the warnings motivate people to quit smoking (n = 12, 18%), and credibility (n = 11, 16%) (Table 2). Of 68 studies, 37 (54%) mentioned a theory as informing their study, including prospect theory,^25–27 theory of psychological reactance,^28–31 cognitive dissonance,^32–34 common sense model,^35,36 and terror management theory.³⁷ Several studies also mentioned being informed by theoretical and empirical research on fear appeals.^{10,20,25,35,38–48}

Table 2.

Summary of Measurement Characteristics by Construct^a

	Number of studies (k)	Multi-item	Reliability
Attention and recall	15
Attention attracting	8	1	1 of 1, α=0.86
Attention duration	2	0	n/a
Recall/recognition	5	0	0.95 inter-coder reliability
Response time	3	0	n/a
Warning reactions	42
Affective reactions	24	19	15 of 19, α=0.74–0.98 in 12; r=0.36–0.94 in 3
Avoidance	3	1	1 of 1, α=0.90
Clarity	3	2	1 of 2, r=0.65
Cognitive elaboration	4	0	n/a
Cognitive processing	3	2	2 of 2 (α=0.89–0.91)
Credibility	11	2	2 of 2, α=0.86–0.92
Liking warning	1	0	n/a
Message reactance	4	4	4 of 4, α=0.58–0.93
Smoking cravings	5	3	2 of 3, r=0.33–0.35
Social interactions	2
Social interactions intentions	2	1	0 of 1
Knowledge, attitudes and beliefs	29
Attitudes toward quitting	2	1	1 of 1, α=0.65
Imaginability of harm	1	0	n/a
Knowledge	2	1	n/a
Pack/brand attitudes	8	8	6 of 8, α=0.79–0.95
Smoking attitudes	9	9	5 of 9, α=0.67–0.95
Perceived likelihood of harm	15	12	6 of 12, α=0.78–0.95
Perceived severity of harm	2	1	0 of 1
Response efficacy	1	1	0 of 1
Self-efficacy to quit	4	2	1 of 2, r=0.33
Smoker identity	1	1	1 of 1, α=0.90
Intentions	25
Information seeking	2	0	n/a
Intention to not start smoking	4	2	2 of 2, α=0.86–0.87
Intention to not smoke	1	1	1 of 1, α=0.77
Intention to quit smoking	15	5	5 of 5, α=0.77–0.81 in 4; r=0.75 in 1
Purchase interest	1	0	n/a
Willingness to pay	3	0	n/a
Quitline calls	1	0	n/a
Behavior	4
Smoking behavior	4	0	n/a
Perceived effectiveness	41
Motivate me to cut down on smoking	3	0	n/a
Motivate me to forgo cigarettes	1	0	n/a
Motivate me to not smoke	4	0	n/a
Motivate me to not start smoking	4	0	n/a
Motivate me to smoke more	2	0	n/a
Motivate me to quit smoking	12	7	5 of 7, α=0.83–0.93
Motivate me/others to not smoke (composite)	3	2	2 of 2, α=0.85–0.88
Motivate others to cut down smoking	1	0	n/a
Motivate others to not start smoking	3	0	n/a
Motivate me to encourage others to quit smoking	1	0	n/a
Motivate others to quit smoking	7	2	1 of 2, α=0.89
Be effective for me/others (scale)	14	14	10 of 14, α=0.63–0.94
Be generally effective (no referent)	6	1	1 of 1, α=0.84
Deter buying cigarettes	1	0	n/a
Deter giving cigarettes as gift	2	2	0 of 2
Cause avoidance	1	0	n/a
Communicate harm	1	0	n/a
Increase knowledge	2	0	n/a
Be useful	1	0	n/a

^a Some studies included more than one construct per category.

Authors reported sources of measures only 37% of the time (i.e., either a citation for the measure or the authors indicating they created the items for the study). None of the studies that claimed to have adapted measures described the nature or extent of adaptation. More than half (54%) of the measures were single items. For multi-item measures (n = 116), papers reported reliability data 68% of the time (mean α = 0.88, range = 0.68–0.98). One study on scale development assessed test–retest reliability as well as convergent, discriminant, and predictive validity.²⁹ While few studies reported validity data, some studies indicated that measures were validated in previous research.^30,49 Newer studies were more likely to report reliability data for multi-item measures, sources of measures, and theories.

Attention and Recall

Attention and recall constructs were measured in 15 studies (See Table 2).^{21,25,31,32,39,41,43,44,46,50–55}Attention attracting, the extent to which the warnings attracted or grabbed participants’ attention, was assessed in 8 studies.^{31,39,41,50–52,54,55} Seven studies used single items and 1 used a 2-item scale (α = 0.86).⁵¹ Two studies used objective measures (i.e., not self-report) to assess attention duration (the amount of time participants spent looking at the warnings)^46,56 and 3 used objective measures to assess response time (the time it took participants to complete questions or click forward after viewing the warnings).^25,32,53 Five studies measured recall or recognition of warnings.^{21,31,43,44,55}Unaided recall required participants to remember images or text without assistance,⁴³ whereas recognition involved showing participants text or images and asking which they remembered.^21,44

Warning Reactions

Warning reactions were measured in 42 studies.^{21,26–31,36–39,41–52,54,55,57–68}Affective reactions were assessed in 24 of those studies, mostly with multi-item scales (α range = 0.74–0.98).^{21,27,36,38,39,41–43,45,47,48,54,58–60,64–67} Kees, for example, asked participants if the cigarette warnings made them feel fearful, anxious, or nervous (α = 0.98).⁴³ Another study asked participants how afraid, uncomfortable, or disgusted they felt after seeing the warnings.⁴⁸Avoidance (i.e., trying not to look at or think about the warnings, covering up the warnings, or not buying packs with particular warnings on them) was measured in 3 studies.^31,52,67 Gibson assessed avoidance intentions with a three-item scale (α = 0.81),⁶⁷ while the other studies used a single item to measure avoidance behavior. The 4 studies assessing cognitive elaboration (the extent to which the participant thought about the warning’s content (e.g., the harms of smoking))^31,50,52,63 and the 11 assessing message credibility (perceptions of the believability or truthfulness of the warnings)^{30,41,44–46,51,55,57,58,62,66} used primarily single-item measures. While most studies examined message credibility, 1 study evaluated source credibility.³⁰ Three studies also assessed cognitive processing (a combined measure of noticing, attention, and cognitive elaboration),^21,37,44 2 of which used multi-item scales (α = 0.89–0.91). Four studies assessed message reactance, a negative reaction in response to a perceived threat to one’s freedom,⁶⁹ and all used a multi-item scale adapted from previous research (α = 0.58–0.93).^28–30,67 One of these 4 studies examined source reactance, where participants rated the message source.³⁰ Three of the 5 studies assessing smoking cravings (the extent to which one desires a cigarette) used multiple items (r = 0.33–0.35).^{21,49,61,64,66} Lin, for example, used a 10-item questionnaire to assess smoking cravings.⁴⁹ Three studies also assessed clarity, 2 of which used multiple items (r = 0.65).^35,41,68 Finally, 1 study used a single item to measure liking of the warning.²⁶

Social Interactions

Two studies assessed social interactions intentions or planning to talk to someone about the warnings; one of these studies used multiple items.^41,67

Knowledge, Attitudes, and Beliefs

Knowledge, attitudes, and beliefs were measured in 29 studies.^{21,26,27,32–34,38,42,43,45–48,52,54,55,59,61,66,67,70–74}Knowledge was assessed in 2 studies.^52,55 For example, Fathelrahman asked participants which of the 13 health conditions was related to smoking.⁵² Eight studies assessed pack or brand attitudes, or evaluation of the cigarette pack or brand, six of which used multi-item measures (α = 0.79–0.95).^{34,43,47,59,71,72}Smoking attitudes, or evaluation of favorability or unfavorability of smoking behavior, was assessed in 9 studies, typically with multi-item scales (α = 0.7–0.95).^{21,26,27,34,46,47,55,70,73} Researchers used a wide range of measures to assess smoking attitudes, including whether smoking helps people relax, smoking is glamorous, rebellious, cool, disgusting, or foolish.²¹Perceived likelihood of harm, beliefs about the chances that smoking will cause health-related harms, was assessed in 15 studies usually using multi-item measures.^{21,32–34,38,42,45,48,54,55,66,67,74} Reliability ranged from α = 0.78–0.88.³³Perceived severity of harm (beliefs about the seriousness of the threat) and response efficacy (beliefs that a recommended response will reduce a health threat) were measured in the same study using 4-item scales.⁴⁸ A second study used a single item to assess perceived severity of harm.⁴⁵Self-efficacy to quit, or confidence in one’s ability to quit smoking, was assessed in 4 studies.^21,48,54,61 Two studies used multi-item measures,^48,61 with 1 of those 2 studies reporting a moderate correlation (r = 0.33) between the 2 items.⁶¹ Other beliefs assessed included attitudes toward quitting (2 studies),^45,66 imaginability of harm (1 study),⁷⁴ and smoker identity (1 study).²⁷ Overall, 90% of studies measuring knowledge, attitudes, and beliefs used multi-item measures.

Intentions

Intention constructs were assessed in 25 studies.^{21,26,27,34,37–39,41,45,47,50–52,55,58,61,67,70,71,75–79}Intention to not start smoking, the perceived likelihood of not starting to smoke cigarettes, was assessed in 4 studies, all with samples of nonsmoker adolescents and youth.^21,26,70,71 Two studies reported reliability data (α range = 0.86–0.87).^70,71 One study assessed intention to not smoke among young adult smokers and used a multi-item scale (α = 0.77).²⁷Intention to quit smoking, or perceived likelihood of quitting smoking, was assessed in 15 studies with smoker samples.^{21,34,37,38,41,45,47,51,52,55,61,67,75} Six studies used a single item with a specific time frame, including 4 that assessed intentions to quit in the next 30 days or 6 months.^{21,38,51,61,75} Other studies did not include a time frame and instead asked participants how much they wanted to quit smoking.^21,61 Five of the 15 studies used multi-item measures, with 1 study using a 6-item scale (α = 0.81).³⁷Willingness to pay, how much participants actually paid for cigarettes in an experimental auction, was measured in 4 studies.^76–79 During the auction studies, participants received money for being in the study and could use the money to pay for the “store-price” of a commodity of interest, in this case a pack of cigarettes.⁷⁶ Participants then bid anonymously on the pack of cigarettes. Purchase interest, whether participants would buy a specific pack of cigarettes, was measured in 1 study.⁵⁰ Two studies measured information seeking intentions and behavior,^41,45 and one measured self-reported quitline calls.⁵⁸

Behavior

Four studies assessed smoking behavior.^31,39,55,75 Malouff et al. used self-reports to measure smoking behavior and asked participants whether they had stopped smoking or decreased cigarette use during the past month.⁷⁵ Dijkstra et al. also used self-reports and asked participants, “During the past 21 days, since you joined the study, have you been refraining from smoking for at least 1 day?”³⁹ McQueen et al. assessed smoking behavior objectively by examining the number of packs used since enrollment to estimate behavior.³¹ Evans averaged the number of filters returned each day, an objective measure.⁵⁵

Perceived Effectiveness

Perceived effectiveness—or perceptions of the potential impact of the warnings— was assessed in 41 studies.^{25–27,33–35,37,38,44-46,48,67–71,26,27,31,35,36,38,41,45,54,64,65,67,68,74,80–83} Perceived effectiveness refers to participants’ perceptions of the persuasive potential of a message and was the outcome with the least conceptual clarity: Authors varied greatly in what they named this construct. Terms included “impact,” “effectiveness,” and “perceived intention to quit.” Most studies assessing perceived effectiveness used a single-item measure and included a referent such as self or other.

Perceived effectiveness of the warning to motivate me to cut down on smoking, for example, referred to the perception that a warning would encourage cutting down on smoking. This was assessed in 3 studies using single items such as, “To what extent, if at all, do the warnings cause you to stop from having a cigarette?”^31,52,58 Six studies also assessed a general measure of perceived effectiveness with no referent.^{44,50,54,81,82,84} Sample items include, “Overall, which warning is most effective?”

Some perceived effectiveness measures used multiple items, with many of the more recent studies employing multi-item scales. Perceived effectiveness of the warning to motivate me to quit smoking, for example, refers to the perception of a warning’s perceived motivational value to encourage the participant quitting smoking and was assessed in 12 studies, 9 of which used multiple items (α range = 0.83–0.93).^{35,43,48,52,54,58–60,68,85} One study, for example, asked participants to rate the following 2 statements (α = 0.93): “The information presented on the cigarette package would help me quit smoking,” and “The information presented on the cigarette package motivates me to quit smoking.”⁴³ Other studies using multi-item scales combined items with different referents into 1 scale (α range = 0.63–0.94).^{26,27,38,45,51,58,62,64,67,79,81,86} Cantrell, for example, assessed responses to 3 items: “The pack makes me want to quit smoking,” “The pack will make people more concerned about the health risks of smoking,” and “The pack will prevent young people from starting to smoke.” The 3 items achieved an internal consistency of α = 0.85.⁵¹ One study assessed deter buying cigarettes—whether the warnings were effective in stopping a person from buying cigarettes—using a single item.⁴¹ Finally, deter giving cigarettes as a gift, a perception that a warning would deter wanting to give a cigarettes as a gift, was assessed in 1 study from China.⁸⁵ The study used 3 items to assess this construct, each evaluated separately.

Other

Studies also measured other constructs that did not fit in our MIF: perceived importance,⁵⁷ personal relevance,⁶² perceived ethicality of the warnings,⁴⁷ public support for the warnings,⁸⁷ perceived recall,³¹ intention to switch brands,⁸³ fear control mode,⁴² comparative harm of cigarettes and perception that the warnings left an impression,⁴¹ and value of information.⁷⁸ All constructs except perceived ethicality and fear control mode were measured with single items, and no reliability data were given.

Discussion

Our systematic review examined characteristics and psychometric properties of measures used in experimental studies of pictorial cigarette pack warnings. Results indicated great variability not only in the constructs assessed and measures used but also in how researchers reported information about their measures. Constructs assessed in the studies varied widely from attention-oriented variables,^50,51 to warning reactions,^43,62 to beliefs, attitudes, and intentions,^48,61 to perceived effectiveness,^87,88 with little consistency across studies. Authors varied greatly in what they named their constructs, often using different names to refer to identical constructs. This was especially so for perceived effectiveness measures. This lack of consensus makes comparisons across studies challenging and hinders progress in understanding the effects of pictorial warnings on smoking-related outcomes. Moreover, many measures appeared to be suboptimal. Studies mainly used single-item measures rather than multi-item scales, making it likely that many measures did not fully represent the intended construct. Single item measures typically had no reliability or validity data associated with them, reducing our confidence in the robustness of measurement. Overall, the lack of consensus on constructs and outcome measures leads to fragmentation in this literature and reduces our ability to synthesize common outcomes across studies.

Perceived effectiveness of the warnings was one of the most commonly assessed outcomes (41 of 68 studies). Prior research has suggested that messages rated higher on perceived effectiveness (e.g., messages rated as persuasive) may be more effective in changing attitudes, intentions, and behaviors than those with lower ratings.^89,90 Thus, this outcome may have some value in identifying more effective warnings before they are implemented, although the field lacks consensus with regard to how to best measure this variable. Perceived effectiveness may be particularly useful at earlier stages of research when warnings are being developed; at later stages, measures of actual effectiveness are needed to test the real-world psychological and behavioral effects of warnings.^91,92 However, ambiguity remains in the literature about perceived versus actual effectiveness of warnings.⁹² Measures that assess perceived effectiveness of a warning (e.g., perception of a warning as likely to increase motivation to quit smoking) and those that measure actual effectiveness (e.g., change in quit intentions) are very different study outcomes that researchers should clearly distinguish from one another in study reports. In this review, we found that it was often not readily apparent if a study was, for example, assessing change in actual quit intentions (i.e., actual effectiveness) or perceptions that the warning would change one’s quit intentions (i.e., perceived effectiveness). This lack of transparency could hamper understanding of basic findings and progress in the field.

Most studies (42 of 68) assessed warning reactions, with several of these studies measuring negative affective reactions such as fear.^{21,42,43,47,48,58–60} As with many measures in this review, however, those studies varied greatly in how they conceptualized and measured these constructs. For example, Schneider and colleagues assessed “fear intensity” by asking participants to rate how afraid, worried, uncomfortable, or disgusted they were after seeing pictorial warnings.⁴⁸ This example shows the inconsistency between what authors said they were measuring and what authors actually were measuring (i.e., negative affect vs. fear). Negative affect comprises a range of feelings aversive emotions including fear.⁶⁶ Given the important role that negative affect plays in warning effectiveness,^66,89,93 researchers should engage in a dialogue about the most fruitful ways to measure this construct, as valid and consistent measurement is vital to determining the impact of warnings. Robust measurements of key constructs would increase the ability to generate cumulative knowledge about the impact of warnings. Future studies can further increase theoretical clarity by specifying what they are measuring, specifying what theory or theories the construct is derived from, and naming constructs appropriately.

Many studies reported no reliability data. When reported, most studies used Cronbach’s α, a measure of internal consistency reliability. Authors reported Cronbach’s alpha mainly for perceived effectiveness,^{42–44,48,51,59,62} attitude scales,^{33,43,59,61,70–72} and intentions,^37,70,71 with the reported αs generally in the acceptable range. Only 1 study reported test–retest reliability.²⁹ It is important to show that participants assign the same meaning to survey measures as researchers intend, through qualitative validation (e.g., through consulting an expert panel or cognitive interviewing with lay people) and quantitative validation (e.g., by establishing convergent and discriminant validity). However, very few investigators mentioned that the instruments had been validated in prior studies or their own studies.^49,72

Authors rarely reported sources of measures, and when they did, they most often cited another study as the source of a measure. This may be understandable given that there are not many existing measures specifically developed for pictorial cigarette pack warning studies.^1,4 However, studies provided little or no information about how they developed or adapted items, information that is important according to reporting guidelines.⁹⁴

Several studies did not include sufficient detail on the theories or theoretical perspectives that guided their studies. This is surprising, as many of the constructs used in these studies have links to psychological or communication theories.^15,16,20 A little over half mentioned a theory as informing the study, but even many of those studies listed general perspectives such as “fear appeals.”^20,40 Other theories included stereotype priming model,⁷⁰ terror management,³⁷ cognitive dissonance,³² and psychological reactance.²⁹ No particular theoretical perspective was widely applied across the studies in our review, suggesting a possible lack of consensus as to how researchers believe warnings exert their effects. Other theories could be applied such as theories from advertising that posit that warnings cause people to associate negative things with a product or brand.⁹⁵ Warning researchers are beginning to focus on the mediating mechanisms to explain warning effects, drawing on theories from health communication and advertising,⁴³ which may lead to development of theoretical explanations.

Our review included a large number of recent studies, with the majority of studies published in the last 3 years. Examination of measures across the study period reveal that the measures were fairly similar across the 17 years of studies, with only modest changes. Notably, studies are now beginning to assess constructs such as social interactions and information seeking. Further, the field now has some experimental studies assessing smoking behavior^31,39,55,75 as well as a more recent large trial in the United States which demonstrated that pictorial warnings increased quit attempts and quitting.⁹³ As more experimental studies measure smoking behavior, reviews will be able to further quantify the effect of warnings on behavior. The current review also demonstrated the international nature of warnings research, with experimental studies conducted in Australia, Mexico, Germany, Iran, and 18 other countries. Despite these strengths, greater attention is still needed to improve measurement in pictorial cigarette pack warning studies as the literature matures.

The current review has several limitations. First, the studies reviewed were all experiments, and we did not review the extensive observational literature. While the 2 literatures have much in common, observational studies are often longer term^7,9 and thus use somewhat different measures. Recent systematic reviews of observational warning studies show that while many outcomes are similar to those found here, notable differences include fewer proximal warning outcomes (e.g., emotional reactions) and greater assessment of behavioral outcomes (e.g., quit attempts and smoking prevalence).^7,9 Future research summarizing measures used in the observational literature on pictorial warnings would make a valuable contribution to this area. Second, we were not able to evaluate the psychometric properties of the measures for particular subsamples. We did find that 1 measure—intention to not start smoking⁷¹—was used exclusively with adolescent and youth samples.

Recommendations

As the field advances, it is critical to increase focus on the development, testing, and reporting of validated measures to advance a cumulative body of evidence on warnings (see Table 3). One step toward this goal would be greater use and explication of the theoretical and conceptual underpinnings of studies. Researchers should use theory to guide selection of measures and place greater emphasis on the theoretical and conceptual basis of their studies. One advantage of tying measures to existing theories is that theories can often provide clear conceptual definitions of variables. Researchers, for example, suggest 4 key elements when measuring perceived likelihood of harm: identify the person at risk, focus on specific harms, make the harms contingent on behavior, and designate a time frame.⁹⁶ To illustrate, 1 study asked “If I keep smoking, I feel vulnerable to getting lung cancer at some point in my life.”⁴⁵ The studies measuring perceived likelihood in this review followed some, but not all, of these guidelines. Furthermore, while the perceived message effectiveness literature is not well developed, some guidelines that have been suggested include delineating the behavior and assigning a clear referent individual or group (e.g., this message would motivate me to quit smoking).^90–92 Several studies in this review did not follow these guidelines. As one additional example, studies measuring message reactance tended to assess only the emotional (e.g., anger) component, leaving out the cognitive aspect (e.g., counterarguing) of reactance.^29,69

Table 3.

Recommendations for Cigarette Pack Warning Constructs and Measures

Activity	Recommendations
Development and selection of measures	Use theories and conceptual models to guide choice of outcomes and measures
	Choose outcomes that are proximal and distal to warning exposure, as appropriate
	Use multi-item scales whenever possible
	Where needed, develop and validate new measures
Reporting of measures	Clearly define constructs and report exact wording of items and response scales
	Clearly distinguish perceived effectiveness from actual effectiveness constructs and measures
	Report reliability and validity information; collect new data where it does not yet exist
	Report sources of previously developed measures

The field would also greatly benefit from the use of measurement frameworks to guide studies.⁹⁷ The MIF is one such framework, and it introduces a way to think about how warnings exert their effects, from attracting attention, to changing beliefs, to changing behavior. The framework could assist researchers in identifying what types of constructs and measures are most appropriate, given the stage of warnings research and the particular goals of the study. Early-stage research should emphasize outcomes proximal to warning exposure such as warning reactions and perceived effectiveness because such findings provide valuable evidence for the potential impact of warnings. Later stage research should emphasize more distal outcomes such as quit intentions and quit attempts in order to examine the public health impact of warnings. Both types of research can provide much needed work on theoretical mechanisms of warnings effects.

Regarding the reporting of measures, we recommend authors include exact wording of items (where possible) and response scales, and sources of measures and reliability data, at least in online supplements. Authors should also define their constructs and report validity information where it exists, utilizing multi-item scales when possible and collecting new validation data if it does not yet exist.

Conclusion

The studies identified in this review represent a growing international field from which future studies on the effectiveness of pictorial cigarette pack warnings can build upon. There remains a need for reliable and validated measures as the field continues to grow. Accurate measurement tools are vital to identify factors associated with cigarette pack pictorial warning perceived and actual effectiveness. Data from such studies is critical for building the evidence regarding the role of cigarette pack warnings in impacting smoking-related beliefs and behaviors as well as for informing warnings research for novel and emerging tobacco products.

Supplementary Material

Supplementary data are available at Nicotine & Tobacco Research online.

Funding

Research reported in this publication was supported by The National Cancer Institute and FDA Center for Tobacco Products (CTP) under Award Numbers P30CA016086-38S2 and 1P50CA18090701. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the Food and Drug Administration. The work was also supported by a seed grant from the School of Media and Journalism at UNC Chapel Hill.

Declaration of Interests

None declared.