Item Development and Performance of Tobacco Product and Regulation Perception Items for the Health Information National Trends Survey

Abstract

Introduction

Emerging tobacco products have become increasingly popular, and the US Food and Drug Administration extended its authority to all products meeting the definition of a tobacco product in 2016. These changes may lead to shifts in public perceptions about tobacco products and regulation, and national surveys are attempting to assess these perceptions at the population level. This article describes the item development and cognitive interviewing of the tobacco product and regulation perception items included in two tobacco-focused cycles of the Health Information National Trends Survey (HINTS-FDA), referred to as HINTS-FDA.

Methods

Cognitive interviewing was used to investigate how respondents comprehended and responded to tobacco product and regulation perception items. Adult participants (n = 20) were selected purposively to oversample current tobacco users and were interviewed in two iterative rounds. Weighted descriptive statistics from the fielded HINTS-FDA surveys (N = 5474) were also calculated.

Results

Items were generally interpreted as intended, and participants meaningfully discriminated between tobacco products when assessing addiction perceptions. Response selection issues involved inconsistent reporting among participants with little knowledge or ambivalent opinions about either government regulation or tobacco products and ingredients, which resolved when a “don’t know” response option was included in the survey. The fielded survey found that a non-negligible proportion of the population do not have clear perceptions of emerging tobacco products or government regulation.

Conclusions

A “don’t know” response option is helpful for items assessing many emerging tobacco products but presents several analytic challenges that should be carefully considered. Multiple items assessing specific tobacco product and regulation perception items are warranted in future surveys.

Implications

The findings from this study can serve as a foundation for future surveys that assess constructs related to emerging tobacco products, harm perceptions across multiple tobacco products, and tobacco-related government regulatory activities. The data provide unique insight into item-specific motivation for selecting a “don’t know” response option for tobacco survey items.

Introduction

The tobacco landscape in the United States has seen many changes in the past decade. Although cigarette use has been steadily declining following decades of public health efforts,¹ emerging tobacco products such as electronic cigarettes (e-cigarettes), hookah, and small cigars have been increasing in popularity.² In addition, the Family Smoking Prevention and Tobacco Control Act was passed in 2009 giving the US Food and Drug Administration (FDA) the authority to regulate tobacco products in the United States. The FDA extended its authority to all other products meeting the statutory definition of a tobacco product, including e-cigarettes, in May 2016.³

These major changes in product development and regulation may lead to shifts in public perceptions about the health effects of tobacco products and their regulation. To adequately capture these potential changes, national surveys such as the Population Assessment of Tobacco and Health (PATH) Study⁴ and the National Youth Tobacco Survey,⁵ as well as other tobacco-focused surveys such as the Project M-PACT (Marketing and Promotions Across Colleges) in Texas,⁶ have included items to assess individuals’ behaviors, knowledge, and attitudes about tobacco products. However, using survey instruments to obtain accurate assessments about emerging tobacco products has proven to be challenging. Key issues identified from previous research include problems identifying the frequency and quantity of product use, discriminating among tobacco products, and clarifying the specific product liquid content added to electronic nicotine delivery systems (eg, nicotine, marijuana, and other drugs).^6,7

As multiple national surveys assess tobacco use and attitudes, item standardization and data harmonization are important to reduce respondent burden, effectively compare across data collection efforts, and increase collaboration opportunities.⁸ Indeed, a 2010 Institute of Medicine report called for efforts to streamline and improve data harmonization within the federal government health surveillance system.⁹ Few existing national surveys related to tobacco contain items that assess knowledge, attitudes, and communication concepts. Thus, the development of new items that explicitly assess perceptions about tobacco product harm and addictiveness, and knowledge and attitudes toward regulation can complement tobacco-related items currently found on existing federal surveys.

To develop these new items, two primarily tobacco-focused cycles of the National Cancer Institute’s Health Information National Trends Survey (HINTS), referred to as HINTS-FDA, were created to provide nationally representative estimates of tobacco-related communication, knowledge, and behavior among adults, and included new and adapted perception items.¹⁰ As part of the formative development of this survey, selected items were cognitively tested with potential survey respondents to modify survey items before instruments were fielded.¹¹ Insights elicited through the cognitive interviewing process are guided by an interviewer who applies intensive interviewing techniques, especially think aloud and verbal probing, to obtain a narrative about how respondents process and answer targeted survey questions.¹¹ Information gathered during the interviews can then be used to modify the final survey instrument to reduce respondent error and item nonresponse.¹¹

The objective of this study was to describe the formative development and performance of selected HINTS-FDA items, which focused on tobacco harm perceptions and beliefs about FDA regulation of tobacco products. We first discuss item origination and development, then describe the cognitive interviewing process used to refine the items, and finally report the final survey items and point estimates from the fielded surveys.

Methods

Initial Item Development

Upon establishing measurement goals for HINTS-FDA, item selection and construction was informed by related items fielded on other large federal questionnaires (eg, PATH). Agencies within the Department of Health and Human Services were consulted to avoid duplication of effort and identify areas of burden reduction between federal surveys. Tobacco product images used on existing federal surveys were included or adapted to accompany the description of several types of tobacco products, including e-cigarettes and smokeless tobacco products. Selected initial items that were developed by the research team or adapted from other surveys underwent additional testing through an iterative cognitive interviewing process.

Cognitive Interviewing Recruitment, Participants, and Process

Adult participants from the Washington, DC, metro area were recruited from the general population in response to advertising in a local newspaper and on the Internet. A total of 20 cognitive interviews with participants were conducted in two iterative rounds of testing, with nine interviews in round one (December 2013) and 11 interviews in round two (January 2014). The sample size selected was consistent with past practice involving cognitive interviewing studies.¹¹ Because of the tobacco-focused nature of the questions, respondents were selected purposively to oversample tobacco users. More than half of participants (n = 11, 55%) were tobacco users, defined as using within the past year: cigarettes, cigars, tobacco pipe, smokeless tobacco, hookah, and/or e-cigarettes. Additional efforts were made to recruit a diverse range of respondents on demographics such as age, sex, and race. Overall, there were more female (n = 12, 60%) than male (n = 8, 40%) participants. Seven participants (35%) were 18–29 years, seven (35%) were 30–54 years, and six (30%) were 55 years or older. Regarding educational attainment, six participants (30%) were high school graduates, six (30%) had completed some college or vocational training, and eight (40%) were college graduates. Participants’ reported race was evenly divided between white and non-white.

Prior to the study commencing, four trained interviewers with expertise as cognitive interviewers observed one interview conducted by the senior methodologist and were trained specifically on the HINTS-FDA protocols, instruments, and research questions. There were two rounds of iterative cognitive testing. The first round included the initial items developed by the study team. The results from the first-round interviews were used to make minor item changes, which were then incorporated into the revised questionnaire used for the second testing round. Each interview was audio recorded and lasted between 1 and 1.5 hours. The interview protocols used a combination of think-aloud and retrospective probe questions to obtain participants’ feedback on the items of interest during testing. Typical follow-up probes used through testing included, “what does it sound like this question was asking to you?”, “how did you come up with your answer to this question?”, and “was it easy or hard to select your answer?” The cognitive interviewing study was approved by the Westat Institutional Review Board and deemed exempt from full review at the National Institutes of Health. Interviewees were compensated $50 for their participation.

Cognitive Interviewing Analysis

Three Westat staff members conducted a qualitative analysis of the cognitive interview data. Written summaries of the interviews, which included selected direct quotations, were first created from reviewing the audio recordings.¹² These summaries were then aggregated across interviews and reviewed for both major themes and specific evidence of problems elicited through probing. The analysis focused on (1) describing respondents’ explanations of the meaning of the answers to each question, (2) identifying and describing any problems that arose during the interview, and (3) offering recommendations for changes to the questions. Analyses were conducted on two different levels. First, interviewers summarized the findings for each question from each interview. These interviewer summaries consisted of descriptions of how participants answered the questions and any problems that occurred during the question–answer process. Next, an interpretive approach for analyzing the cognitive interviews was used. First, themes or patterns of interpretations in the data were identified based on reviewing notes for each question that was tested. We then identified any problematic patterns and determined whether these patterns were more likely to occur in specific characteristics of the subgroups tested (eg, tobacco users). Data were analyzed in accordance with Tourangeau’s four-stage model,^13,14 which identify potential issues that can arise in four stages, including (1) comprehending the intent of the item; (2) retrieving relevant information; (3) judging and estimating the answer; and (4) selecting their answer from the given response options.

HINTS-FDA Data Collection

After cognitive testing was completed and the instrument was finalized, data collection of HINTS-FDA 2015 (N = 3738) occurred from May to September 2015. HINTS-FDA 2017 (N = 1762) was fielded from January to May 2017. Data for both cycles were fielded using a single-mode mailed survey. Within-household sample selection was randomized by selecting the adult in the household having the next birthday. More details about the methodology of HINTS-FDA are available elsewhere.¹⁰ The full instrument and methodology report for both cycles are available on the HINTS Web site at http://hints.cancer.gov.

For this analysis, we calculated weighted univariate descriptive statistics and item performance for selected items included in the HINTS-FDA 2015 and 2017 survey instruments. Data from the two cycles were merged with combined replicate weights using the jackknife replication method. For this analysis, we checked for consistency with the cognitive testing results; that is, we assessed whether the patterns observed in the data would be in any way predictable from the cognitive interview findings.

Results

Overall, the items performed well in both the cognitive interviews and the final fielded surveys. Results in the following sections report first on findings from the cognitive interviewing process and then on the performance of the items in HINTS-FDA.

Cognitive Interviewing Results

Participants were asked the degree to which they thought tobacco products were addictive and harmful, and were queried about their perception of government regulation of tobacco products. In total, four items related to tobacco addiction perceptions and harm perceptions (Table 1) and three items related to government regulation perceptions (Table 2) underwent cognitive testing and are included in the current analysis. Issues identified in the formative development process were related to comprehension and response selection; we did not identify any issues related to retrieval from memory, or judgment and estimation processes. Results are organized in the following sections topically, focusing first on addiction and harm perceptions and then regulatory beliefs.

Table 1.

Selected Harm Perception Cognitive Testing Results and Final Survey Items for Health Information National Trends Survey (HINTS-FDA) 2015 and 2017

Construct	Round 1	Round 2	Primary issue identified	Final version
Item 1: Tobacco product addiction perceptions	Overall, how addictive do you believe each of the following is? (Not at all addictive, Moderately addictive, Very addictive) a. Cigarette smoking b. Cigar smoking c. Smokeless tobacco use d. Electronic cigarettes e. Smoking tobacco in a hookah f. Smoking “roll your own” cigarettes g. Pipe filled with tobacco	Same as Round 1	Mapping issue: - When respondents did not know or did not have an opinion, they either left the question blank or marked “moderately addictive”	Overall, how addictive do you believe each of the following is? (Not at all addictive, Moderately addictive, Very addictive, Don’t Know) a. Cigarette smoking b. Cigar smoking c. Smokeless tobacco use d. Using electronic cigarettes or e-cigarettes (also known as vape pens, e-hookahs, or e-vaporizers) e. Smoking tobacco in a hookah f. Smoking “roll your own” cigarettes g. Smoking a pipe filled with tobacco
Item 2: Nicotine harm and addiction perceptions	Please indicate how much you agree or disagree with the following statements (Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree) a. Nicotine is the main substance in tobacco that makes people want to smoke b. The nicotine in cigarettes is the substance that causes most of the cancer caused by smoking c. Addiction to nicotine is something that I am concerned about	Same as Round 1	Mapping issue: - When respondents did not know or did not have an opinion, they either left the question blank or marked “neither agree nor disagree”	Please indicate how much you agree or disagree with the following statements (Strongly agree, Agree, Disagree, Strongly disagree, Don’t know) a. Nicotine is the main substance in tobacco that makes people want to smoke b. The nicotine in cigarettes is the substance that causes most of the cancer caused by smoking c. Addiction to nicotine is something that I am concerned about
Item 3: Tobacco product use intentions based on addiction claims	If a tobacco product made a claim that it was less addictive than other tobacco products, how likely would you be to use that product? (Very likely, Somewhat likely, Somewhat unlikely, Very unlikely, Don’t know)	Same as Round 1	Mapping issue: - When respondents did not smoke, they were split between selecting “don’t know” and “very unlikely”	If a tobacco product made a claim that it was less addictive than other tobacco products, how likely would you be to use that product? (Very likely, Somewhat likely, Somewhat unlikely, Very unlikely)
Item 4: Tobacco product use intentions based on harm claims	If a tobacco product made a claim that it was less harmful to health than other tobacco products, how likely would you be to use that product? (Very likely, Somewhat likely, Somewhat unlikely, Very unlikely, Don’t know)	Same as Round 1	Mapping issue: - When respondents did not smoke, they were split between selecting “don’t know” and “very unlikely”	If a tobacco product made a claim that it was less harmful to health than other tobacco products, how likely would you be to use that product? (Very likely, Somewhat likely, Somewhat unlikely, Very unlikely)

Table 2.

Selected Regulation Perception Cognitive Testing Results and Final Survey Items for Health Information National Trends Survey (HINTS-FDA) 2015 and 2017

Construct	Round 1	Round 2	Primary issue identified	Final version
Item 1: Knowledge of tobacco product regulation	As far as you know, does anyone review tobacco products before they are marketed to consumers? (Yes, No, Don’t Know)	As far as you know, does anyone review tobacco products before they are allowed to be sold? (If yes) Who reviews tobacco products before they are allowed to be sold? Mark all that apply:  Centers for Disease Control and Prevention (CDC)  Federal Trade Commission (FTC)  Food and Drug Administration (FDA)  Independent groups  National Institutes of Health (NIH)  Surgeon General  Tobacco industry/tobacco companies  US Board of Cancer Prevention (USBCP)  None of the above  Other (specify)	Comprehension issue: - Participants’ understanding of “reviewing” varied widely (eg, reviewing for marketability, safety, or both) - On the basis of survey length and potential for response burden, an administrative decision was made to streamline item to specifically evaluate knowledge of FDA’s activities	Do you believe that the U.S. Food and Drug Administration (FDA) regulates tobacco products in the United States? (Yes, No, Don’t Know)
Item 2: Knowledge of the regulation of product statements	As far as you know, does anyone review statements that tobacco companies make about the contents, chemicals, or health effects of their products before they are marketed to consumers? (Yes, No, Don’t know)	As far as you know, does anyone review statements that tobacco companies make about the contents or health effects of their products? (Yes, No, Don’t know) (If yes) Who reviews statements that tobacco companies make about the contents or health effects of their products? Mark all that apply.  Centers for Disease Control and Prevention (CDC)  Federal Trade Commission (FTC)  Food and Drug Administration (FDA)  Independent groups  National Institutes of Health (NIH)  Surgeon General  Tobacco industry/tobacco companies  US Board of Cancer Prevention (USBCP)  None of the above  Other (specify)	Comprehension issue: - At least two respondents thought about antitobacco advertisements (eg, Surgeon General’s warning placed on tobacco products, antismoking commercials) - On the basis of survey length and potential for response burden, an administrative decision was made to streamline item to specifically evaluate knowledge of FDA’s activities	Do you believe that anyone regulates statements from tobacco companies about the contents or health effects of their tobacco products? (Yes, No, Don’t know) (If yes) Who do you believe regulates statements that tobacco companies make about the contents or health effects of their products? Mark all that apply.  Centers for Disease Control and Prevention (CDC)  Federal Trade Commission (FTC)  Food and Drug Administration (FDA)  National Institutes of Health (NIH)  Surgeon General  Tobacco industry/tobacco companies  None of the above
Item 3: Perceptions of organization qualified to review	How qualified is each of the following to review tobacco products? (Not at all qualified, A little qualified, Somewhat qualified, Very qualified)  Centers for Disease Control and Prevention (CDC)  Federal Trade Commission (FTC)  Food and Drug Administration (FDA)  Independent groups  National Institutes of Health (NIH)  Surgeon General  Tobacco industry/tobacco companies  US Board of Cancer Prevention (USBCP)	How qualified is each of the following to review tobacco products? (Not at all, A little, Somewhat, Very)  Centers for Disease Control and Prevention (CDC)  Federal Trade Commission (FTC)  Food and Drug Administration (FDA)  Independent groups  National Institutes of Health (NIH)  Surgeon General  Tobacco industry/tobacco companies  US Board of Cancer Prevention (USBCP)	Comprehension issue: - Confusion about what constituted “independent group” - On the basis of survey length and potential for response burden, an administrative decision was made to streamline item to specifically evaluate knowledge of FDA’s activities	In your opinion, how qualified is the U.S. Food and Drug Administration (FDA) to regulate tobacco products? (Not at all, A little, Somewhat, Very)

Tobacco Product Addiction and Harm Perceptions

The most common issues that arose from addiction and harm perceptions were response selection issues that occurred when participants felt they did not know much about the addictiveness or harmfulness of the tobacco product or its ingredients. Specifically, when participants were not familiar with emerging tobacco products, had low confidence in their answer, or had ambivalent opinions about the degree of addictiveness and/or harmfulness, they either disregarded the question or indicated that they would select the middle response option (“moderately addictive” or “neither agree nor disagree”). This issue occurred for several sets of tobacco perception items, including emerging tobacco product addiction perceptions (Table 1, Items 1ag) and nicotine harm and addiction perceptions (Items 2ac). For example, for one item querying about nicotine as a carcinogen (Item 2b), seven of the 20 cognitive interviewing participants selected the middle option (“neither agree nor disagree”), and all seven of these respondents reported low confidence in their answer. Either spontaneously or in response to a probe (“what do you know about nicotine?”), these seven respondents indicated that they did not know very much about nicotine, other than that it is addictive. On the basis of these data, final response options for absolute harm and addiction items were modified to include “don’t know” response options, and the middle response category was also removed from Item 2.

In contrast, two related items asked participants how likely they would be to use a product if it made a claim that it was (1) less harmful to health than other tobacco products (Item 4) or (2) less addictive than other tobacco products (Item 3). Unlike harm perception items, these items requested that participants state their intention to perform a behavior instead of reporting their baseline harm perceptions. For these items, a “don’t know” response option was initially included in the cognitive interviews, along with the four-point scale, 1 “very likely” to 4 “very unlikely.” When cognitive interviewing participants did not use tobacco products, they were split between selecting “don’t know” (three participants) or “very unlikely” (six participants) response options. When probed, all nontobacco users who answered “don’t know” and “very unlikely” indicated that they selected their answer because they did not and would not smoke. Given this confusion and overlapping rationale with the “very unlikely” response option, the “don’t know” response option was ultimately removed for the final survey instrument.

Regulatory Beliefs

Reported items focus on knowledge of product regulation (Item 1), knowledge of the regulation of product statements (Item 2), and agency qualification perceptions (Item 3) (Table 2). Although these items were intended to evaluate knowledge and perceptions of government regulatory activities only, comprehension issues arose when we used the more colloquial term “review” instead of “regulate” to denote these activities. In Item 1, which queried on knowledge of tobacco product regulation, most respondents understood “review” to mean reviewing the safety of tobacco products (ie, regulatory activities). However, other participants thought about nonregulatory activities, such as reviewing tobacco products for quality and marketability. The Round 2 item also added a filtered question asking about beliefs of specific agencies’ review. However, the final item was simplified to query specifically about knowledge of the FDA’s regulatory authority over tobacco products.

Similarly, participants reported comprehension issues with the use of “review” in Item 2, which assessed knowledge of the regulation of tobacco company product statements. Although most respondents interpreted “review” as regulating the statements that tobacco companies make, others thought of public health statements from government health agencies (ie, Surgeon General’s warning that are put on tobacco products and antismoking messages on television). The question was revised in the final instrument to use the term “regulate” and to emphasize the source of the statements: “Do you believe that anyone regulates statements from tobacco companies about the contents or health effects of their tobacco products?” The final instrument also included a filtered question about perceptions of regulatory authority concerning product statement review for five government agencies and for the tobacco industry.

The final item (Item 3) assessed perceptions of whether organizations are qualified to review tobacco products. Participants were additionally probed, “what does it mean to be qualified to review tobacco products?” Respondents generally understood the term “qualified” and described it as “having a broad base of information,” “knowledgeable and trustworthy,” and “having scientific background.” In the list of specific agencies and companies, respondents expressed confusion about “independent groups” and the fictitious group “US Board of Cancer Prevention.” The final question was again streamlined to focus specifically on perceptions of FDA regulatory qualifications.

Item Performance on HINTS-FDA 2015 and 2017

Consistent with other nationally representative surveys, final response rates were 33.04% for HINTS-FDA 2015¹⁵ and 34.05% for HINTS-FDA 2017.¹⁶

Don’t Know Response Option

As discussed earlier, response options for several risk perception items in the final survey instrument were modified, based on cognitive testing results, to include a “don’t know” (DK) option. Results from the fielded surveys indicate that this response option was heavily used in selected cases (Figure 1). For example, for the final question asking participants the degree to which they thought smoking tobacco in a hookah was addictive, weighted point estimates suggest approximately 41% of the population would select “don’t know.” A DK response option was also included and often selected in some regulatory perception items. For example, results from the final statement regulation perception item found that about 40% of Americans do not know if anyone regulates tobacco company statements (Figure 2).

Figure 1.

Weighted descriptive statistics and item performance for selected addiction perception items.

Figure 2.

Weighted descriptive statistics and item performance for selected regulation perception items.

Differentiation of Product Perceptions

Results from the fielded studies support the results of cognitive testing, in suggesting that individuals meaningfully distinguished between tobacco products when determining risk perception beliefs (Figure 1). For example, more than three-quarters of participants indicated that cigarette smoking was very addictive, whereas only about a quarter reported that smoking tobacco in a hookah was very addictive.

Discussion

Emerging product developments, changing government regulation, and a rapidly evolving tobacco landscape demand novel ways of conceptualizing and measuring tobacco items in population-based surveillance surveys. The items developed for HINTS-FDA are well positioned to complement item pools found in other federal tobacco-related surveys, and findings from this study extend previous cognitive interviewing and surveillance research work in this area.^4–6 Although previous tobacco research using cognitive interviews has focused largely on optimizing measurement related to frequency and use quantity of tobacco products,^6,7 the items in this study uniquely focus on tobacco product harm perceptions, as well as regulation knowledge and perceptions.

Explicit Don’t Know Response

A major methodological consideration that arose in this study was the appropriate inclusion of an explicit DK response option for harm and regulation perception items. Other federal surveys have accepted volunteered DK responses, but few have explicitly incorporated DK responses into the survey instrument. This may be reflective of the common perception among survey methodologists that DK responses will facilitate participant satisficing, or the use of cognitive shortcuts in formulating a response when motivation is low.¹⁷ However, data from the cognitive interviews suggest that motivation issues do not adequately explain why the DK response option was so heavily used for tobacco harm and regulation perception items in HINTS-FDA. Rather, cognitive interviewing participants primarily identified issues related to response selection; that is, an inability to adequately express their perceived lack of knowledge from the provided list of substantive response options. Consistent with previous research,¹⁸ when a DK response option was initially not explicitly offered, cognitive interviewing participants expressed a lack of knowledge in other ways, most commonly by skipping the item or selecting the middle scale option (eg, “moderately addictive”).

Given the novelty of many tobacco products (eg, e-cigarettes) included in the HINTS-FDA survey instrument, it is unsurprising that a non-negligible proportion of the population has no clear perception of their harmfulness. As individuals typically have preformed attitudes and perceptions only about familiar things,¹⁹ findings from this study suggest that an explicit DK response option may be warranted for select items related to perceptions of emerging tobacco products. Otherwise, harm and regulation perceptions generated on the spot in an attempt to formulate a substantive response may be, at best, uninformative, and at worst, biased and counterproductive when assessed on a national survey.

The high proportion of DK responses for harm and regulation perception items additionally suggests potentially important implications for HINTS-FDA data analysis and interpretation. DK responses have typically been treated as missing data, incorrect responses, or combined with other responses (eg, “refused to answer”) in secondary data analysis. However, related work in risk perception has suggested that individuals who indicate they “don’t know” represent an important subpopulation of public health concern, and filtering out DK responses may bias survey results.^20–23 For example, one study found that individuals who said they did not know their risk of developing cancer engaged in less physical activity, had lower mammography adherence, and had lower fruit and vegetable consumption than those who provided a valid response.²² The risk perception literature also suggests that DK responders have lower socioeconomic status and educational attainment than other responders,^20,21 making them a particularly vulnerable group, as these populations are simultaneously at higher risk for tobacco use.²⁴

In this study, HINTS-FDA weighted point estimates indicate that a high proportion of US individuals do not have harm perceptions of many emerging tobacco products. In some cases, such as smoking tobacco in a hookah, our data suggest more than 40% of the population do not have enough knowledge to report addiction perceptions. Thus, analyzing predictors and outcomes of a DK response can inform tailored educational and behavioral interventions to this group.

However, DK response options may not be warranted in all cases; the type of question or the construct being probed may motivate different reasons for DK responses.²³ Indeed, for HINTS-FDA, we ultimately chose to include an explicit DK option for some items, yet removed the option in other items. Although cognitive interviewing data suggested that addiction perception items were primarily motivated by knowledge deficit, cognitive interview respondents typically did not report lacking the knowledge needed to provide a substantive response to items related to behavioral intention, and therefore, the DK response was removed for behavioral intention items in the final survey instrument. Ultimately, decisions of whether to include a DK response are item specific, and researchers should consider potential theoretical and data trade-offs of different methodological approaches.

Meaningful Distinctions of Addictiveness Perceptions

A second major finding from this study is that respondents who provided valid responses meaningfully distinguished harm between tobacco products. Specifically, a greater proportion of the population (76%) reported that cigarette smoking was “very addictive” than for any other tobacco product listed. In contrast, only about a quarter of the population indicated that they perceived smoking tobacco in a hookah and using e-cigarettes to be “very addictive.” The variation of addictiveness perceptions demonstrates that survey respondents discriminate across tobacco products when responding to harm and addiction perception items.

Nicotine is addictive in any tobacco product, and there may be several explanations for diverse addiction perceptions found in HINTS-FDA. First, many individuals may have stronger perceptions for tobacco products that are more familiar and salient to them, and specifically, products for which there has been decades of public health messaging, such as cigarette smoking. Thus, the high addiction perceptions for cigarette smoking could be reflective of established public service advertisements, educational efforts, and packaging. The FDA recently announced a comprehensive regulatory plan that places nicotine and addiction at the center of its tobacco regulation efforts.^25,26 These findings may help to inform efforts to educate the public about the addictiveness of nicotine in other regulated tobacco products. There are efforts underway to address these issues. For example, the FDA recently expanded its public education campaign to focus on youth use of e-cigarettes.²⁷ Continuing to inquire about addiction and harm perceptions of emerging tobacco products in future HINTS cycles and other surveys can help researchers determine the effectiveness of these new campaigns.

Government Regulation Perception

Previous studies have investigated public perception of the FDA generally, including awareness,²⁸ overall trust,²⁸ and perceptions of credibility and effectiveness of the agency.²⁹ However, this study uniquely queried knowledge of and perceptions of specific regulatory activities that the FDA engages in, such as reviewing statements made by tobacco companies. Data from the full study suggest that such specific questions are warranted. Despite near full awareness of the FDA,²⁸ more than a third of the population does not know if the FDA regulates tobacco products (37.51%) or if anyone regulates statements from tobacco companies about the contents of their tobacco products (39.67%), although both are key regulatory activities of the FDA.

Data from cognitive interviews also suggest that using the more colloquial term “review” was confusing to participants and should be avoided, as participants interpreted review in many unrelated ways, such as for marketability. We ultimately revised the language in the final survey instrument to the term “regulate,” as it is the technically correct definition for the regulatory actions of the agency.

Conclusion

We described the formative development for tobacco and regulatory perception items included in HINTS-FDA. The new items introduced in HINTS-FDA are unique and fill a gap in national survey efforts by providing data on perceptions about tobacco products and their regulation. The results from this cognitive interviewing study suggest that tobacco product harm and government regulation perception items are generally well understood by respondents. Engaging in an extensive cognitive testing process prior to collecting national data was a useful formative development step to improve content validity and response accuracy. Results suggest individuals make meaningful distinctions in their perceptions across tobacco products, and a non-negligible proportion of respondents used “don’t know” for multiple items indicating that they may have not formed perceptions about emerging tobacco products or government regulation.

Strengths of this study include a robust cognitive interviewing protocol, which both oversampled tobacco users and used two rounds of iterative testing. Additionally, efforts were made to standardize items and response options when similar constructs were found in other federal surveys, providing the opportunity for comparability with other federal data sources, data harmonization, and replication. A limitation of this study is that items were cognitively tested 4 years prior to manuscript preparation, and the tobacco marketplace has continued to evolve. Future research may test and adapt these harm and regulation perception items for other and more specific emerging tobacco products (eg, JUUL pods) as they become available and accessible.

Declaration of Interests

None declared.