Abstract
This study examined validity of direct and indirect measures of perceived harm of e-cigarettes and smokeless tobacco (SLT) compared to cigarettes. On direct measures, people compare one product to another, whereas on indirect measures, people rate each product separately and the researcher compares these ratings. Data from youth in Wave 1 of the Population Assessment of Tobacco and Health (PATH) Study (2013–2014) were analyzed (N = 13,651 youth aged 12–17 years). The study included direct measures of perceived harm of e-cigarettes and SLT compared to cigarettes, and indirect measures were created by comparing ratings of the products. Weighted multinomial logistic regressions tested criterion validity by assessing whether direct and indirect measures were associated with criterion variables, including use of e-cigarettes and SLT. Youth were more likely to rate e-cigarettes as less harmful than cigarettes on the indirect measure (67.3%) than the direct measure (50.2%). The same pattern held for ratings of SLT as less harmful than cigarettes (indirect: 29.7%; direct: 11.7%). Direct measures explained unique variance in product use criterion variables even after adjusting for indirect measures, as did indirect measures after adjusting for direct measures. However, the criterion variables were more often associated with the direct measures than the indirect measures. Results offer preliminary support for using both direct and indirect measures when assessing youth’s perceived relative harm of various types of products. However, if researchers cannot include both direct and indirect measures in a study, associations with product use criterion variables support prioritizing direct measures.
Keywords: perceived harm, risk perception, measurement, validity, e-cigarettes, smokeless, tobacco
Introduction
Patterns of tobacco and nicotine-containing product use among U.S. youth have changed over the past five years, with decreases in current cigarette, cigar, and smokeless tobacco (SLT) use and increases in e-cigarette and hookah use (1). People are especially vulnerable to tobacco use initiation during the adolescent development period, as nearly 90% of daily smokers had their first cigarette at or prior to the age of 18 (2).
Harm beliefs play an important role in health behavior (3–6) and are associated with youth tobacco use behavior, including initiation, experimentation, and cessation (7–9). An area of interest in tobacco research is beliefs about relative harm – whether one product is less, equally, or more harmful than another. The belief that e-cigarettes, SLT, and other tobacco and nicotine-containing products are less harmful than cigarettes could motivate youth to use these products (10–15). Tobacco control and regulatory research benefit by taking these harm beliefs into account, as they have implications for initiation and current use among youth.
Reflecting interest among researchers, public health groups, and policymakers, a growing body of research is assessing public perceptions of relative harm for tobacco and nicotine-containing products (14, 16–29). Most of these studies assessed perceptions of relative harm by asking people to directly compare products to cigarettes (14, 16–29) – e.g., “Do you think electronic cigarettes are more harmful than regular cigarettes, less harmful, or are they equally harmful to health?” (28). However, recent studies have introduced a new method in which perceptions of relative harm are measured indirectly – i.e., people are asked to rate the harmfulness of each product on separate questions, and researchers compare the responses to see if one product was rated as more or less harmful than the other (13, 30). These initial investigations have found that direct and indirect comparison methods can yield different results about perceived relative harm. In one study, adult U.S. smokers were more than twice as likely to rate snus (a type of SLT) as less harmful than cigarettes on an indirect measure compared to a direct measure (51.6% vs. 22.1%) (13). The correlation between responses on the direct and indirect measures was surprisingly weak for two measures of the same construct (r = 0.3, R2 = 0.09 or 9% of variance explained) (13). The difference between direct and indirect measures was subsequently replicated for e-cigarettes and SLT (30, 31), suggesting that these two types of measures may either tap into different aspects of perceived relative harm or that one type of measure may be better than the other at assessing perceived relative harm.
Though initial evidence shows that different approaches to measuring relative harm produce different results, several research gaps remain. First, only one prior study has compared the validity of direct and indirect measures of perceived relative harm of tobacco products (31). This prior study examined perceptions of e-cigarettes and SLT and tested whether direct and indirect measures were associated with relevant product use criterion variables (e.g., for e-cigarette perceived relative harm, ever-use of e-cigarettes), finding that the direct measures were more consistently associated with these criterion variables than the indirect measures (31). As the study authors concluded, further validation work is needed comparing direct and indirect measures. Second, prior work in this area is limited because it has only been conducted on adults (13, 30, 31). Youth is a critical period for tobacco product trial (32, 33), and given the establishment of tobacco use and brand preferences at an early age, youth have been the target of tobacco marketing in the past (34). Youth are also the target of tobacco prevention campaigns, some of which focus on communicating product harms (35). Thus, comparative measurement validation studies in youth populations would be useful but are currently lacking.
The present study was designed to address these gaps by comparing the validity of direct and indirect measurement approaches for perceived relative harm in a nationally representative sample of U.S. youth. The study focused on perceptions of two different product types, e-cigarettes and SLT, which have been the focus of many prior studies of perceived relative harm (13–29). We examined the criterion validity of measures, meaning the extent to which the measures significantly correlated with theoretically related criterion variables (36, 37). Specifically, we assessed whether direct and indirect measures of perceived relative harm of each product type (e-cigarettes and SLT) were associated with use of each product type. Use of products is an appropriate criterion variable because public health interest in perceived relative harm is predicated on the notion that harm beliefs have implications for product initiation and use (26, 38, 39). Consistent statistically significant associations between perceived harm measures and product use were taken as evidence in favor of the criterion validity of the measure, as perceived harm is known to be associated both prospectively and cross-sectionally with product use in youth (9, 11, 14, 19, 20, 22).
Method
Data Source and Participants
Data were from Wave 1 of the Population Assessment of Tobacco and Health (PATH) Study (40), a longitudinal study of tobacco use and health. At Wave 1, PATH yields nationally-representative estimates of the U.S. non-institutionalized population aged 12 years and older. Recruitment used address-based, area-probability sampling, with an in-person household screener to select participants. Data were collected in participants’ homes using audio computer-assisted interviews. We analyzed Wave 1 data from youth (age 12–17), collected September 2013 to December 2014 (N = 13,651). The overall youth weighted response rate was 78.4%. Additional details on the study design and data collection procedures are available elsewhere (41).
Survey Instrument
The survey instrument was organized into sections based on product type, with cigarettes first, e-cigarettes second, and SLT subsequent. In the e-cigarette and SLT sections, participants who reported that they had ever seen or heard of the product type (for e-cigarettes, n = 12,178, 89.4%; for SLT, n = 10,123, 75.8%) answered questions about their use and perceived harm of the product. In each section, the direct measure of perceived relative harm immediately preceded the e-cigarette/SLT item used in the indirect measure.
Measures
Direct Measures of Perceived Relative Harm
Direct measures of perceived relative harm for e-cigarettes and SLT were: “Is using e-cigarettes less harmful, about the same, or more harmful than smoking cigarettes?”; “Is using smokeless tobacco less harmful, about the same, or more harmful than smoking cigarettes?” For both items, response options were Less harmful, About the same, More harmful, and Don’t know.
Indirect Measures of Perceived Relative Harm
The indirect measure of e-cigarette perceived relative harm was created based on participants’ responses to two questions: “How much do you think people harm themselves when they use e-cigarettes?”, and “How much do you think people harm themselves when they smoke cigarettes?” Response options for both were No harm, Little harm, Some harm, A lot of harm, and Don’t know. Participants were coded as rating e-cigarettes as Less harmful than cigarettes if their rating for e-cigarettes was lower than their rating for cigarettes, as About the same as cigarettes if their ratings for the products were the same, as More harmful if their rating for e-cigarettes was higher than their rating of cigarettes, and as Don’t know if they rated either product (or both) as Don’t know.
Similarly, the indirect measure of SLT perceived relative harm was created based on the question, “How much do you think people harm themselves when they use smokeless tobacco?” and the analogous question about cigarettes above; the same four categories were created as described above for e-cigarettes.
Product Use Criterion Variables
E-cigarette use was assessed with, “Have you ever used an e-cigarette, such as NJOY, Blu, or Smoking Everywhere, even one or two times?” Participants responding Yes were asked, “When did you last use an e-cigarette, even one or two times?” Options ranged from Earlier today to 5 or more years ago. A three-level variable was created: Participants were coded as “Never users” if they never used an e-cigarette, as “Non-past-30 day users” if they ever used but did not use in the past 30-days, and as “Past 30-day users” if they reported using at least sometime in the past 30 days.
SLT use was assessed with, “Have you ever used any of the following smokeless tobacco products, even one or two times? Choose all that apply” (Snus pouches; Loose snus, moist snuff, dip, spit, or chewing tobacco; I have never used a smokeless tobacco product). Those reporting use of either type of SLT were asked, “When did you last use [SLT type], even one or two times?” Options ranged from Earlier today to 5 or more years ago. Participants were coded as “Never users” if they never used either type of SLT, as “Non-past-30-day users” if they ever used either type of SLT but did not use in the past 30-days, and as “Past 30-day users” if they reported using either type of SLT at least sometime in the past 30 days.
Analysis
We assessed criterion validity using multinomial logistic regressions with two steps to test whether direct and indirect measures of perceived relative harm of each product were associated with use criterion variables. For each product, we assessed this association without controlling for the other perceived harm measure by regressing the criterion variable separately on the direct and indirect measures of perceived relative harm. Next, the criterion variable was regressed on the direct and indirect measures simultaneously to determine whether the direct and indirect measures explained unique variance in the criterion variable.
Analyses were weighted to represent U.S. youth ages 12–17 and to account for the complex sampling design (42). The balanced repeated replication (BRR) method was used as recommended (42). Participants were excluded from any analysis in which they did not have data on one or more variables; no imputation strategy was used to fill missing data. Analyses were conducted in SAS Version 9.3.
Results
Sample Characteristics
Sample demographics and product use behaviors are shown in Table 1. Participants ranged in age from 12 to 17 years old (M = 14.5) with roughly equal numbers between ages 12–13, 14–15, and 16–17. The sample was approximately half female and was representative of the U.S. population in terms of race/ethnicity. Past 30-day product use was 4.6% for cigarettes, 3.1% for e-cigarettes, and 1.6% for SLT, and non-past-30-day use of these products was 8.8%, 7.5%, and 3.2%, respectively.
Table 1.
Sample characteristics and product use: Weighted ns (unweighted %s)
| Variable | n (%) |
|---|---|
| Sex | |
| Female | 6641 (48.7) |
| Male | 6971 (51.3) |
| Age (y) | |
| 12–13 | 4684 (33.7) |
| 14–15 | 4597 (33.3) |
| 16–17 | 4368 (33.0) |
| Race/Ethnicity | |
| Non-Hispanic White | 6478 (54.2) |
| Hispanic | 3880 (22.5) |
| Non-Hispanic Black | 1801 (13.5) |
| Asian | 336 (4.3) |
| Other | 895 (4.8) |
| Cigarettes | |
| Past 30-Day Use | 634 (4.6) |
| Non-Past-30-Day Use | 1204 (8.8) |
| Never Use | 11,813 (86.6) |
| E-cigarettes | |
| Past 30-Day Use | 418 (3.1) |
| Non-Past-30-Day Use | 1034 (7.5) |
| Never Use | 12,199 (89.4) |
| SLT | |
| Past 30-Day Use | 201 (1.6) |
| Non-Past-30-Day Use | 434 (3.2) |
| Never Use | 13,016 (95.2) |
Perceived E-Cigarette Relative Harm Ratings
As shown in Figure 1, participants were more likely to rate e-cigarettes as less harmful than cigarettes on the indirect measure (67.3%, 95% CI: 66.2–68.3%) than the direct measure (50.2%, 95% CI: 49.2–51.3%). They were more likely to rate e-cigarette harm as about the same as cigarette harm on the direct measure (37.8%, 95% CI: 36.8–38.8%) than the indirect measure (29.2%, 95% CI: 28.1–30.2%). The direct and indirect measures were moderately to strongly associated (r = 0.51, p < .0001, excluding the Don’t know category).
Figure 1.
Youth were more likely to rate e-cigarettes as less harmful than cigarettes on the indirect measure vs. the direct measure. Participants with missing data were excluded.
Table 2 shows rates of e-cigarette use by direct and indirect ratings of e-cigarette relative harm. Among participants rating e-cigarettes as less harmful than cigarettes on the direct measure, 5.8% used an e-cigarette in the past 30-days, compared to 0.3% who responded that they “Don’t know” how harmful e-cigarettes are compared to cigarettes.
Table 2.
Rates of youth e-cigarette use by level of perceived relative harm: Unweighted ns and weighted %s.
| Direct Measure of E-Cigarette Perceived Relative Harm | ||||
|---|---|---|---|---|
|
|
||||
| Less Harmful than Cigarettes | Equally Harmful as Cigarettes | More Harmful than Cigarettes | Don’t Know | |
|
|
||||
| Past 30-Day Use | 345 (5.8) | 65 (1.3) | 5 (1.0) | 3 (0.3) |
| Non-Past-30-Day Use | 714 (11.9) | 242 (5.1) | 32 (5.6) | 13 (1.1) |
| Never Use | 4959 (82.3) | 4293 (93.6) | 525 (93.4) | 912 (98.6) |
|
| ||||
| Indirect Measure of E-Cigarette Perceived Relative Harm | ||||
|
|
||||
| Less Harmful than Cigarettes | Equally Harmful as Cigarettes | More Harmful than Cigarettes | Don’t Know | |
|
|
||||
| Past 30-Day Use | 367 (4.6) | 38 (1.0) | 6 (4.3) | 6 (1.7) |
| Non-Past-30-Day Use | 847 (10.4) | 140 (3.8) | 10 (8.5) | 4 (1.0) |
| Never Use | 6943 (85.0) | 3345 (95.2) | 100 (87.3) | 305 (97.4) |
Note: For each cross-tabulation, participants lacking data on either variable were excluded (n = 1543 and n = 1540 for analyses of the direct and indirect measures, respectively). Most of these participants lacked data because they reported never seeing or hearing of e-cigarettes.
Criterion validity of direct measure
When e-cigarette use was regressed on the direct measure of e-cigarette perceived relative harm (Table 3, Model 1), e-cigarette use was consistently related to the direct measure: Participants rating e-cigarettes as less harmful than cigarettes were significantly more likely to be past 30-day users than participants who gave any other response. Similarly, participants rating e-cigarettes as less harmful than cigarettes were also more likely to be non-past-30-day users compared to participants who gave any other response.
Table 3.
Weighted multinomial logistic regression of youth e-cigarette use on direct measures (Model 1), indirect measures (Model 2), and both direct and indirect measures (Model 3) of e-cigarette perceived relative harm: Odds ratios (ORs) and 95% CIs.
| E-Cigarette Use
|
Pseudo R-Squared* | |||
|---|---|---|---|---|
| Past 30-Day Use | Non-Past-30-Day Use | Never Use | ||
| Model 1 | .067 | |||
| Direct Measure: | ||||
| E-cigarettes less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| E-cigarettes about the same | 0.19 (0.14,0.26) | 0.38 (0.32,0.45) | 1.00 (ref) | |
| E-cigarettes more harmful | 0.15 (0.05,0.43) | 0.41 (0.28,0.61) | 1.00 (ref) | |
| Don’t Know | 0.04 (0.01,0.17) | 0.08 (0.04,0.15) | 1.00 (ref) | |
|
| ||||
| Model 2 | .043 | |||
| Indirect Measure: | ||||
| E-cigarettes less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| E-cigarettes about the same | 0.19 (0.13,0.27) | 0.33 (0.27,0.41) | 1.00 (ref) | |
| E-cigarettes more harmful | 0.91 (0.35,2.37) | 0.79 (0.36,1.74) | 1.00 (ref) | |
| Don’t Know | 0.31 (0.12,0.82) | 0.08 (0.03,0.25) | 1.00 (ref) | |
|
| ||||
| Model 3 | .079 | |||
| Direct Measure: | ||||
| E-cigarettes less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| E-cigarettes about the same | 0.25 (0.18,0.35) | 0.48 (0.39,0.60) | 1.00 (ref) | |
| E-cigarettes more harmful | 0.24 (0.08,0.73) | 0.67 (0.42,1.05) | 1.00 (ref) | |
| Don’t Know | 0.05 (0.01,0.22) | 0.13 (0.07,0.24) | 1.00 (ref) | |
| Indirect Measure: | ||||
| E-cigarettes less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| E-cigarettes about the same | 0.41 (0.26,0.63) | 0.48 (0.37,0.62) | 1.00 (ref) | |
| E-cigarettes more harmful | 1.77 (0.66,4.74) | 1.03 (0.45,2.36) | 1.00 (ref) | |
| Don’t Know | 0.90 (0.33,2.46) | 0.19 (0.06,0.59) | 1.00 (ref) | |
Note: Values that are statistically significant at the .05 level are in bold. For each model, participants lacking data on any variable were excluded from analysis.
Pseudo r-squared values are adjusted Nagelkerke pseudo r-square from unweighted models.
Criterion validity of indirect measure
When e-cigarette use was regressed on the indirect measure of e-cigarette perceived relative harm (Table 3, Model 2), participants rating e-cigarettes as less harmful than cigarettes were more likely to be past 30-day users and non-past-30-day users compared to those rating e-cigarettes as about the same as cigarettes and those responding Don’t know, but did not differ from those rating e-cigarettes as more harmful than cigarettes.
Comparison of direct and indirect measures
When e-cigarette use was regressed simultaneously on the direct and indirect measures of e-cigarette perceived relative harm (Table 3, Model 3), participants rating e-cigarettes as less harmful than cigarettes on the direct measure were more likely to be past 30-day users compared to all other groups, and were more likely to be non-past-30-day users compared to those rating e-cigarettes as About the same as cigarettes and those responding Don’t know. For the indirect measure, associations were less consistent: Participants rating e-cigarettes as less harmful were more likely to be past 30-day or non-past-30-day users than were participants rating e-cigarettes About the same as cigarettes, and were more likely to be non-past-30-day users than were participants who responded Don’t know. Adding the indirect measure to a model with the direct measure increased adjusted pseudo r-squared from .067 to .079 (a .012 increase), whereas adding the direct measure to a model with the indirect measure increased adjusted pseudo r-squared from .043 to .079 (a .036 increase). Goodness-of-fit significantly improved in both cases, p<.001.1
Perceived SLT Relative Harm Ratings
As shown in Figure 2, participants were more likely to rate SLT as less harmful than cigarettes on an indirect measure (29.7%, 95% CI: 28.5–30.9%) than on a direct measure (11.7%, 95% CI: 11.0–12.5%). Also, they were more likely to rate SLT harm as about the same as cigarette harm on the indirect measure (63.0%, 95% CI: 61.8–64.2%) than the direct measure (53.8%, 95% CI: 52.6–55.0%). The direct and indirect measures were moderately to strongly associated (r = 0.47, p < .0001, excluding the Don’t know category).
Figure 2.
Youth were more likely to rate SLT as less harmful than cigarettes on the indirect measure compared to the direct measure. They were also more likely to rate SLT as about the same as cigarettes on the indirect measure vs. the direct measure. Participants with missing data were excluded.
Table 4 shows rates of SLT use based on direct and indirect ratings of SLT relative harm. Among participants rating SLT as less harmful than cigarettes on the direct measure, 7.2% had used SLT in the past 30-days, compared to 0% of participants who indicated that they Don’t know how harmful SLT is compared to cigarettes.
Table 4.
Rates of youth SLT use by level of perceived relative harm: Unweighted ns and weighted %s.
| Direct Measure of SLT Perceived Relative Harm | ||||
|---|---|---|---|---|
|
|
||||
| Less Harmful than Cigarettes | Equally Harmful as Cigarettes | More Harmful than Cigarettes | Don’t Know | |
|
|
||||
| Past 30-Day Use | 79 (7.2) | 101 (1.9) | 21 (0.8) | 0 (0) |
| Non-Past-30-Day Use | 63 (5.2) | 248 (4.5) | 100 (3.8) | 9 (0.9) |
| Never Use | 1015 (87.6) | 4985 (93.6) | 2629 (95.4) | 734 (99.1) |
|
| ||||
| Indirect Measure of SLT Perceived Relative Harm | ||||
|
|
||||
| Less Harmful than Cigarettes | Equally Harmful as Cigarettes | More Harmful than Cigarettes | Don’t Know | |
|
|
||||
| Past 30-Day Use | 121 (4.3) | 63 (1.0) | 14 (2.9) | 1 (0.8) |
| Non-Past-30-Day Use | 144 (4.9) | 237 (3.7) | 37 (5.9) | 1 (0.8) |
| Never Use | 2703 (90.8) | 5983 (95.3) | 529 (91.3) | 147 (98.5) |
Note: For each cross-tabulation, participants lacking data on either variable were excluded (n = 3667 and n = 3671 for analyses of the direct and indirect measures, respectively). Most of these participants lacked data because they reported never seeing or hearing of SLT.
Criterion validity of direct measure
When SLT use was regressed on the direct measure of SLT perceived relative harm (Table 5, Model 1), participants rating SLT as less harmful than cigarettes were more likely to be past 30-day SLT users than were participants who gave any other response. Also, they were more likely to be non-past-30-day users of SLT than were participants rating SLT as More harmful and Don’t know.
Table 5.
Weighted multinomial logistic regression of youth SLT use on direct measures (Model 1), indirect measures (Model 2), and both direct and indirect measures (Model 3) of SLT perceived relative harm: Odds ratios (ORs) and 95% CIs.
| SLT Use
|
Pseudo R-Squared* | |||
|---|---|---|---|---|
| Past 30-Day Use | Non-Past-30-Day Use | Never Use | ||
| Model 1 | .042 | |||
| Direct Measure: | ||||
| SLT less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| SLT about the same | 0.25 (0.18,0.34) | 0.82 (0.61,1.09) | 1.00 (ref) | |
| SLT more harmful | 0.10 (0.06,0.17) | 0.68 (0.47,0.97) | 1.00 (ref) | |
| Don’t Know | **0.00 (0.00,0.00) | 0.16 (0.07,0.35) | 1.00 (ref) | |
|
| ||||
| Model 2 | .027 | |||
| Indirect Measure: | ||||
| SLT less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| SLT about the same | 0.22 (0.17,0.30) | 0.72 (0.59,0.89) | 1.00 (ref) | |
| SLT more harmful | 0.66 (0.33,1.35) | 1.20 (0.79,1.82) | 1.00 (ref) | |
| Don’t Know | 0.16 (0.01,2.27) | 0.15 (0.01,1.98) | 1.00 (ref) | |
|
| ||||
| Model 3 | .053 | |||
| Direct Measure: | ||||
| SLT less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| SLT about the same | 0.34 (0.24,0.50) | 0.89 (0.67,1.19) | 1.00 (ref) | |
| SLT more harmful | 0.14 (0.07,0.25) | 0.73 (0.50,1.07) | 1.00 (ref) | |
| Don’t Know | **0.00 (0.00,0.00) | 0.19 (0.09,0.41) | 1.00 (ref) | |
| Indirect Measure: | ||||
| SLT less harmful | 1.00 (ref) | 1.00 (ref) | 1.00 (ref) | |
| SLT about the same | 0.42 (0.29,0.61) | 0.80 (0.65,0.99) | 1.00 (ref) | |
| SLT more harmful | 1.62 (0.74,3.55) | 1.37 (0.87,2.13) | 1.00 (ref) | |
| Don’t Know | 0.94 (0.07,13.43) | 0.38 (0.03,5.08) | 1.00 (ref) | |
Note: Values that are statistically significant at the .05 level are in bold. For each model, participants lacking data on any variable were excluded from analysis.
Pseudo r-squared values are adjusted Nagelkerke pseudo r-square from unweighted models.
No participants responding Don’t know on the direct measure had used e-cigarettes in the past 30-days.
Criterion validity of indirect measure
When SLT use was regressed on the indirect measure of SLT perceived relative harm (Table 5, Model 2), participants rating SLT as less harmful than cigarettes were more likely to be past 30-day users and non-past-30-day users than were participants rating SLT as About the same as cigarettes, but did not significantly differ from participants who gave other responses.
Comparison of indirect and direct measures
When SLT use was regressed simultaneously on the direct and indirect measures of SLT perceived relative harm (Table 5, Model 3), participants rating SLT as Less harmful on the direct measure were more likely to be past 30-day users than were those rating SLT as About the same, More harmful, and Don’t know. Also, they were more likely to be non-past-30-day users than were participants responding Don’t know on the direct measure. Participants responding Less harmful on the indirect measure were more likely to be past 30-day users and non-past-30-day users than were participants rating SLT as About the same, but did not significantly differ from participants who gave any other response. Adding the indirect measure to a model with the direct measure increased adjusted pseudo r-squared from .042 to .053 (a .011 increase), whereas adding the direct measure to a model with the indirect measure increased adjusted pseudo r-squared from .027 to .053 (a .026 increase). Goodness-of-fit significantly improved in both cases, p<.001.2
Supplementary Analyses
Subgroup analyses were conducted among youth who were past 30-day cigarette smokers, non-past-30-day cigarette smokers, and never cigarette smokers. For e-cigarette analyses, results across sub-groups were generally consistent with those in the overall sample (i.e., more consistent associations between e-cigarette use and direct, rather than indirect, measures of perceptions). For SLT analyses, results among non-past-30-day cigarette smokers were similarly consistent. However, for past 30-day cigarette smokers and never smokers, we could not conduct sub-group analyses because the sample sizes of the SLT user groups were too small.
Discussion
Relative harm beliefs are important constructs in tobacco research and regulation, and techniques for measuring these beliefs differ widely. Recent studies have found that the way relative harm is measured can affect estimates of the proportion of respondents that perceive non-cigarette products as less harmful than cigarettes (13, 30, 31). In Popova and Ling’s initial demonstration that direct and indirect measures produce different results, they recommended that tobacco researchers and regulators “consider both direct and indirect measures when perceived risk data are presented as evidence for tobacco regulation” (13, p. e21). Until recently, only a single study had compared the criterion validity of direct and indirect measures among adults (31), finding that direct measures had higher criterion validity (as measured via associations with tobacco use behavior) than indirect measures. The present study extended these findings by examining the validity of direct and indirect measures among youth.
In the present study, differences emerged between youth’s ratings of e-cigarettes and SLT on direct versus indirect comparison measures, mirroring prior findings with adults (13, 30, 31). Associations with product use criterion variables suggest that the direct measures have better criterion validity than the indirect measures: When e-cigarette and SLT use were regressed on the direct measures alone, associations were significant for 11 of 12 regression coefficients, and when e-cigarette and SLT use were regressed on the indirect measures alone, associations were significant for 6 of 12 regression coefficients. Thus, on the direct measure, each level of perceived relative harm appeared to differ meaningfully from the Less harmful level when assessing associations with use, whereas on the indirect measure, differences were less consistent.
We found that direct and indirect measures were uniquely associated with product use criterion variables for both e-cigarette and SLT use, supporting the prior recommendation (13) that tobacco researchers and regulators take into account both types of measures. Additionally, we found that adding the direct measure to models with the indirect measure increased pseudo r-squared more so than did adding the indirect measure to models with the direct measure. Therefore, if researchers cannot include both direct and indirect measures in a study, these findings support prioritizing direct measures.
Additional research would be useful to clarify why direct and indirect measures yield different results. One hypothesis is that the two types of measures may differentially affect cognitive processes, such as information retrieval and processing (43). In direct comparisons, people may place more focus or weight on attributes of the product being evaluated than on attributes of the comparison product (43). Additionally, indirect comparison measures assume that people interpret and use the response scale consistently for different products, which may not be true. Popova and Ling’s original paper on direct versus indirect measures hypothesized that direct measures may bias people toward giving what they see as a “socially appropriate answer” that non-cigarette products are just as harmful as cigarettes (13, p. e22). They hypothesized that indirect measures revealed the “actual differences in perceptions” (13, p. e22). Based on this hypothesis, one might expect indirect measures to be more valid than direct measures, which was not observed in the present study or in a prior study of adults (31). Directly testing this and other hypotheses remains a topic for future research.
Limitations and Strengths
The indirect measures used here were based on items with only five response options (No harm, Little harm, Some harm, A lot of harm, Don’t know), and adding additional options may have improved the validity of the resulting indirect measures. Measures were based on single-items that asked about global harm in general (14, 16–26, 28, 29). The measures also did not specify the frequency or intensity of product use to be assessed. Future studies may consider using multi-item measures to provide a more nuanced assessment of harm perceptions (e.g., having participants evaluate the harms of various frequencies or intensities of product use) and to evaluate perceptions of specific harms (e.g. respiratory harms). Future measurement validation research may also use additional criterion variables, such as tobacco use attitudes, susceptibility, or curiosity, and may evaluate other aspects of the measures’ validity such as whether the measures tap into all important facets of the perceived harm of products (44). A strength of this study is the use of a national sample and its focus on youth, an important demographic in tobacco control research and regulation. Another strength is that criterion validity was assessed for two products and results of these analyses were consistent.
Conclusion
This study evaluated the criterion validity of direct and indirect measures of perceived harm of e-cigarettes and SLT, compared to cigarettes, in a nationally representative sample of U.S. youth. Although indirect measures and direct measures were uniquely associated with product use, direct measures exhibited more consistent associations with product use criterion variables than indirect measures. Given that tobacco use is the primary outcome of interest in public health research on tobacco, these findings provide preliminary support for the use of both types of measures, when possible, and the choice of direct over indirect measures when it is not feasible to include both in research. Continued measurement validation research on harm perceptions would benefit public health research and tobacco regulation.
Footnotes
Pseudo r-squared values and significance of changes in goodness-of-fit were calculated based on unweighted data.
Pseudo r-squared values and significance of changes in goodness-of-fit were calculated based on unweighted data.
Disclaimer: This publication represents the views of the authors and does not represent FDA/CTP position or policy.
Conflict of Interest Statement: The authors have no conflicts of interest to report.
References
- 1.Singh T, Arrazola RA, Corey CG, Husten CG, Neff LJ, Homa DM, et al. Tobacco Use Among Middle and High School Students - United States, 2011–2015. MMWR Morb Mortal Wkly Rep. 2016;65(14):361–7. doi: 10.15585/mmwr.mm6514a1. [DOI] [PubMed] [Google Scholar]
- 2.U.S. Department of Health and Human Services. Preventing Tobacco Use Among Youth and Young Adults -- A Report of the Surgeon General. Rockville, MD: 2012. [Google Scholar]
- 3.Janz NK, Becker MH. The Health Belief Model: A decade later. Health Education Quarterly. 1984;11(1):1–47. doi: 10.1177/109019818401100101. [DOI] [PubMed] [Google Scholar]
- 4.Glanz K, Rimer BK, Viswanath K, editors. Health Behavior and Health Education: Theory, Research, and Practice. 4. San Francisco, CA: Jossey-Bass; 2008. [Google Scholar]
- 5.Weinstein ND. The precaution adoption process. Health Psychol. 1988;7(4):355–86. doi: 10.1037//0278-6133.7.4.355. [DOI] [PubMed] [Google Scholar]
- 6.Brewer NT, Chapman GB, Gibbons FX, Gerrard M, McCaul KD, Weinstein ND. Meta-analysis of the relationship between risk perception and health behavior: the example of vaccination. Health Psychol. 2007;26(2):136–45. doi: 10.1037/0278-6133.26.2.136. [DOI] [PubMed] [Google Scholar]
- 7.Halpern-Felsher BL, Biehl M, Kropp RY, Rubinstein ML. Perceived risks and benefits of smoking: differences among adolescents with different smoking experiences and intentions. Prev Med. 2004;39(3):559–67. doi: 10.1016/j.ypmed.2004.02.017. [DOI] [PubMed] [Google Scholar]
- 8.Krosnick JA, Chang L, Sherman SJ, Chassin L, Presson C. The Effects of Beliefs About the Health Consequences of Cigarette Smoking on Smoking Onset. Journal of Communication. 2006;56:S18–S37. [Google Scholar]
- 9.Song AV, Morrell HE, Cornell JL, Ramos ME, Biehl M, Kropp RY, et al. Perceptions of smoking-related risks and benefits as predictors of adolescent smoking initiation. Am J Public Health. 2009;99(3):487–92. doi: 10.2105/AJPH.2008.137679. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Pepper JK, Brewer NT. Electronic nicotine delivery system (electronic cigarette) awareness, use, reactions and beliefs: a systematic review. Tob Control. 2014;23(5):375–84. doi: 10.1136/tobaccocontrol-2013-051122. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Barrington-Trimis JL, Berhane K, Unger JB, Cruz TB, Huh J, Leventhal AM, et al. Psychosocial Factors Associated With Adolescent Electronic Cigarette and Cigarette Use. Pediatrics. 2015;136(2):308–17. doi: 10.1542/peds.2015-0639. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.O’Connor RJ, McNeill A, Borland R, Hammond D, King B, Boudreau C, et al. Smokers’ beliefs about the relative safety of other tobacco products: findings from the ITC collaboration. Nicotine Tob Res. 2007;9(10):1033–42. doi: 10.1080/14622200701591583. [DOI] [PubMed] [Google Scholar]
- 13.Popova L, Ling PM. Perceptions of relative risk of snus and cigarettes among US smokers. Am J Public Health. 2013;103(11):e21–3. doi: 10.2105/AJPH.2013.301547. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ambrose BK, Rostron BL, Johnson SE, Portnoy DB, Apelberg BJ, Kaufman AR, et al. Perceptions of the relative harm of cigarettes and e-cigarettes among U.S. youth. Am J Prev Med. 2014;47(2 Suppl 1):S53–60. doi: 10.1016/j.amepre.2014.04.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Tomar SL, Hatsukami DK. Perceived risk of harm from cigarettes or smokeless tobacco among U.S. high school seniors. Nicotine Tob Res. 2007;9(11):1191–6. doi: 10.1080/14622200701648417. [DOI] [PubMed] [Google Scholar]
- 16.Sutfin EL, McCoy TP, Morrell HE, Hoeppner BB, Wolfson M. Electronic cigarette use by college students. Drug Alcohol Depend. 2013;131(3):214–21. doi: 10.1016/j.drugalcdep.2013.05.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Choi K, Forster J. Characteristics associated with awareness, perceptions, and use of electronic nicotine delivery systems among young US Midwestern adults. Am J Public Health. 2013;103(3):556–61. doi: 10.2105/AJPH.2012.300947. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Goniewicz ML, Zielinska-Danch W. Electronic cigarette use among teenagers and young adults in Poland. Pediatrics. 2012;130(4):e879–85. doi: 10.1542/peds.2011-3448. [DOI] [PubMed] [Google Scholar]
- 19.Pokhrel P, Fagan P, Kehl L, Herzog TA. Receptivity to e-cigarette marketing, harm perceptions, and e-cigarette use. Am J Health Behav. 2015;39(1):121–31. doi: 10.5993/AJHB.39.1.13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Amrock SM, Zakhar J, Zhou S, Weitzman M. Perception of e-cigarette harm and its correlation with use among U.S. adolescents. Nicotine Tob Res. 2015;17(3):330–6. doi: 10.1093/ntr/ntu156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Choi K, Forster JL. Beliefs and experimentation with electronic cigarettes: a prospective analysis among young adults. Am J Prev Med. 2014;46(2):175–8. doi: 10.1016/j.amepre.2013.10.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Berg CJ, Romero DR, Pulvers K. Perceived harm of tobacco products and individual schemas of a smoker in relation to change in tobacco product use over one year among young adults. Subst Use Misuse. 2015;50(1):90–8. doi: 10.3109/10826084.2014.958858. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Lund I, Scheffels J. Perceptions of relative risk of disease and addiction from cigarettes and snus. Psychol Addict Behav. 2014;28(2):367–75. doi: 10.1037/a0032657. [DOI] [PubMed] [Google Scholar]
- 24.Callery WE, Hammond D, O’Connor RJ, Fong GT. The appeal of smokeless tobacco products among young Canadian smokers: the impact of pictorial health warnings and relative risk messages. Nicotine Tob Res. 2011;13(5):373–83. doi: 10.1093/ntr/ntr013. [DOI] [PubMed] [Google Scholar]
- 25.Chaffee BW, Gansky SA, Halpern-Felsher B, Couch ET, Essex G, Walsh MM. Conditional risk assessment of adolescents’ electronic cigarette perceptions. Am J Health Behav. 2015;39(3):421–32. doi: 10.5993/AJHB.39.3.14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Kiviniemi MT, Kozlowski LT. Deficiencies in public understanding about tobacco harm reduction: results from a United States national survey. Harm Reduct J. 2015;12:21. doi: 10.1186/s12954-015-0055-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Biener L, Nyman AL, Stepanov I, Hatsukami D. Public education about the relative harm of tobacco products: an intervention for tobacco control professionals. Tob Control. 2014;23(5):385–8. doi: 10.1136/tobaccocontrol-2012-050814. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Brose LS, Brown J, Hitchman SC, McNeill A. Perceived relative harm of electronic cigarettes over time and impact on subsequent use. A survey with 1-year and 2-year follow-ups. Drug Alcohol Depend. 2015;157:106–11. doi: 10.1016/j.drugalcdep.2015.10.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Richardson A, Pearson J, Xiao H, Stalgaitis C, Vallone D. Prevalence, harm perceptions, and reasons for using noncombustible tobacco products among current and former smokers. Am J Public Health. 2014;104(8):1437–44. doi: 10.2105/AJPH.2013.301804. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Wackowski OA, Bover Manderski MT, Delnevo CD. Comparison of Direct and Indirect Measures of E-cigarette Risk Perceptions. Tob Regul Sci. 2016;2(1):38–43. doi: 10.18001/TRS.2.1.4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Persoskie A, Nguyen AB, Kaufman AR, Tworek C. Criterion validity of measures of perceived relative harm of e-cigarettes and smokeless tobacco compared to cigarettes. Addictive Behaviors. doi: 10.1016/j.addbeh.2017.01.001. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Soneji S, Sargent J, Tanski S. Multiple tobacco product use among US adolescents and young adults. Tob Control. 2016;25(2):174–80. doi: 10.1136/tobaccocontrol-2014-051638. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Preventing Tobacco Use Among Youth and Young Adults: A Report of the Surgeon General. Atlanta GA: 2012. [PubMed] [Google Scholar]
- 34.Perry CL. The tobacco industry and underage youth smoking: tobacco industry documents from the Minnesota litigation. Arch Pediatr Adolesc Med. 1999;153(9):935–41. doi: 10.1001/archpedi.153.9.935. [DOI] [PubMed] [Google Scholar]
- 35.Duke JC, Alexander TN, Zhao X, Delahanty JC, Allen JA, MacMonegle AJ, et al. Youth’s Awareness of and Reactions to The Real Cost National Tobacco Public Education Campaign. PLoS One. 2015;10(12):e0144827. doi: 10.1371/journal.pone.0144827. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Westen D, Rosenthal R. Quantifying construct validity: two simple measures. J Pers Soc Psychol. 2003;84(3):608–18. doi: 10.1037//0022-3514.84.3.608. [DOI] [PubMed] [Google Scholar]
- 37.Etter JF, Duc TV, Perneger TV. Validity of the Fagerstrom test for nicotine dependence and of the Heaviness of Smoking Index among relatively light smokers. Addiction. 1999;94(2):269–81. doi: 10.1046/j.1360-0443.1999.94226910.x. [DOI] [PubMed] [Google Scholar]
- 38.Shiffman S, Pillitteri JL, Burton SL, Rohay JM, Gitchell JG. Smokers’ beliefs about “Light” and “Ultra Light” cigarettes. Tob Control. 2001;10(Suppl 1):i17–23. doi: 10.1136/tc.10.suppl_1.i17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Slovic P. Rejoinder: the perils of Viscusi’s analyses of smoking risk perceptions. Journal of Behavioral Decision Making. 2000;13(2):273–6. [Google Scholar]
- 40.United States Department of Health and Human Services National Institutes of Health, National Institute on Drug Abuse, United States Department of Health and Human Services, Food and Drug Administration, Center for Tobacco Products. Population Assessment of Tobacco and Health (PATH) Study [United States] Restricted-Use Files. ICPSR36231-v6. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor]; 2016. May 24, Report No. [Google Scholar]
- 41.PATH Study Team. Design and methods of the Population Assessment of Tobacco and Health (PATH) Study. Tobacco Control. 2016 doi: 10.1136/tobaccocontrol-2016-052934. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Westat. PATH Study Restricted Use Files User Guide. Rockville, MD: 2016. [Google Scholar]
- 43.Chambers JR, Windschitl PD. Biases in social comparative judgments: the role of nonmotivated factors in above-average and comparative-optimism effects. Psychol Bull. 2004;130(5):813–38. doi: 10.1037/0033-2909.130.5.813. [DOI] [PubMed] [Google Scholar]
- 44.Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment. 1995;7(3):238–47. [Google Scholar]