Using Cognitive Interviewing to Better Assess Young Adult E-cigarette Use

Abstract

Introduction

Characteristics of electronic nicotine delivery systems (ENDS) make assessment of their use a challenge for researchers. Cognitive interviews are a way of gaining insight into participants’ interpretations of survey questions and the methods they use in answering them, to improve survey tools.

Methods

We used cognitive interviews to modify a young adult survey and improve assessment of quantity and frequency of ENDS use, as well as reasons for initiation and use of ENDS products. Twenty-five college students between the ages of 18 and 32 participated in individual cognitive interviews, which assessed question comprehension, answer estimation, retrieval processes, and answer response processes.

Results

Comprehension issues arose discerning between ENDS device types (eg, cigalikes vs. vape pens), and answer estimation issues arose regarding ENDS use as drug delivery systems. These issues appeared to improve when pictures were added specifying the device in question, as well as when specific language naming nicotine as the ENDS product content was added to survey questions. Regarding answer retrieval, this sample of users had problems reporting their frequency of ENDS use, as well as quantifying the amount of ENDS products consumed (eg, volume of e-juice, number of cartridges, nicotine concentration).

Conclusions

Accurate assessment of ENDS products proved challenging, but cognitive interviews provided valuable insight into survey interpretation that was otherwise inaccessible to researchers. Future research that explores how to assess the wide array of ENDS devices, as well as possible population differences among specific device-type users would be valuable to public health researchers and professionals.

Implications

This study extends the current literature by using cognitive interviews to test ENDS assessment questions in a sample of young adults, a population at elevated risk for ENDS use. Problems encountered when answering ENDS use questions underscore the need to develop easily understood ENDS questions that allow for quantification of ENDS use. Future research examining the nature of ENDS product types and different levels of user experience will yield valuable assessment tools for researchers and tobacco control professionals.

Introduction

Electronic nicotine delivery systems (ENDS) are products that deliver vaporized liquid nicotine through a portable, handheld device. They are referred to by many names, among them “e-cigarettes,” “vapes,” and “e-hookah.” The first ENDS were developed in 2003¹ and made available in the United States in 2006.² Multiple characteristics contribute to the growing popularity of ENDS products, among them appealing flavors,³ the ability to use ENDS where cigarettes are prohibited,^1,4 and a belief that ENDS are less harmful than traditional cigarettes.^5,6

Great variety exists across ENDS products, though most contain liquid nicotine, which is suspended in a mixture of propylene glycol and/or vegetable glycerin, along with artificial flavoring. This liquid is vaporized and inhaled after drawing on the device in the same manner as a cigarette.⁶ The smallest ENDS devices are known as “cigalikes,” products that are the same size and shape as traditional cigarettes. These products are often referred to as “first generation” e-cigarettes.^2,7 Some cigalikes are designed to be disposable (ie, discarded at the end of the battery life or when the nicotine cartridge is empty), while others are rechargeable and have replaceable nicotine cartridges. These cartridge-style devices are often referred to as “closed systems.”⁸ In contrast, “vapes,” and “vape pens” usually have refillable reservoirs of “e-liquid,” and come in a wide variety of sizes, tank volumes, and battery voltages, and most with a button that activates the heating element prior to each use.^2,9 Users may add their choice of a wide variety of e-liquids, and often refer to these products as “open systems”⁸ or “second-generation”^2,7 products. At the time of the current study’s implementation, “vape” products offered higher levels of user customization than cigalikes. The terminology of these new and developing products is still evolving, and it is important to note that some products we deemed disposable/rechargeable (“first generation” products) may now share similarities with the “vape style” (refillable) products. As an example, “e-hookah” often has characteristics of both ENDS styles. Product categories for ENDS are not rigidly or always clearly defined, and many products may share characteristics across groups.

The ENDS market has been doubling yearly, and in March 2014 the US retail market was estimated at over $2 billion per year.¹⁰ This rapidly expanding market is evident in the changing rates of awareness and use of ENDS products. Among those over 18, awareness of ENDS products in the United States rose from 40.9% in 2010 to 79.7% in 2013, 8.5% of whom reported ever trying ENDS at least once.¹¹ Among adults surveyed in multiple studies spanning 2009 to 2013, young adults aged 18 to 24 reported the highest ENDS product use.^12–15 Research has demonstrated that younger age contributes to this higher prevalence of ever and recent use, though this past 30-day use has been characterized more as “recent experimentation” than established heavy ENDS product use.¹⁶ This growth in ENDS use underscores the need to develop accurate assessment tools, which are currently limited in effectiveness.^16–20

Unique challenges are encountered when measuring ENDS use versus assessing conventional tobacco products. Among smokers of traditional cigarettes, nicotine intake varies based on individual smoking intensity, including puff and inhalation patterns.^21,22 While ENDS use likely varies in these same patterns, customization options and the absence of federal regulation permit an even greater variation in nicotine delivery across devices. Users may choose various e-liquid tank sizes, heating coils with adjustable settings, mouthpieces, battery sizes, and even mix e-liquids of varying nicotine concentrations, leading to considerable differences from product to product.^6,23,24 These variations make ENDS products difficult to assess compared to traditional cigarettes. Typically a cigarette is finished in one “sitting,” whereas ENDS may be used for one or two puffs and put away for later use.⁹ Research has shown that these characteristics contribute to increased frequency of ENDS use as compared to cigarettes, including use in locations where cigarette smoking is not allowed.^1,4,6 Accurate assessment of the entire landscape of ENDS use is essential to gauge trends and changes in use, and to determine the potential value or risk of these products to public health. Currently there is limited information about how to best assess this new and rapidly growing landscape of ENDS products, particularly among young adults.

The purpose of this study was to improve ENDS assessment through cognitive interviewing in a sample of young adults. Cognitive interviewing is a method of pretesting a survey in a target population to determine how well survey questions meet their intended objectives.²⁵ Cognitive interviewing is described as the evaluation of how respondents comprehend, process, and answer survey questions.²⁶ Participants give explanations of how they interpreted survey questions, elaborations of their answer construction or retrieval, and any difficulties encountered in answering survey items.²⁵ Tourangeau’s four-stage model of the survey response process provided the framework for this study.²⁷ In this model, participants may give inaccurate or unreliable answers because they cannot understand the question (a problem with comprehension), use flawed strategies in creating their answers (judgment and estimation errors), cannot remember or do not possess the relevant information (errors in retrieval), or fail to find an appropriate option among the available answer choices (response errors).^26,27

Methods

Sample and Recruitment

This study was conducted in the context of a larger tobacco-use study, Marketing and Promotions Across Colleges in Texas, or Project M-PACT. This rapid response surveillance study includes 5482 young adults in 24 colleges in the four largest metropolitan areas of Texas (Austin, Dallas/Ft. Worth, Houston, and San Antonio). The Project M-PACT survey assessed trends of 18–29 year old adults’ use of cigarettes, ENDS products, cigar products (including large cigars, cigarillos, and little filtered cigars), hookah, and smokeless tobacco over time, as well as examined their self-reported exposure to tobacco marketing. It also assessed cognitive and affective factors associated with tobacco product use and other risk behaviors such as drug use and alcohol use. Cognitive interviews were used to refine the entirety of the Project M-PACT survey. The present study focuses on improvements made to the ENDS portion of the Project M-PACT survey.

Cognitive interviews were conducted prior to Project M-PACT survey data collection with a convenience sample of 25 college students in Austin, Texas during the summer of 2014. Participants were solicited via advertisements posted at one major university and three local-area community colleges. The aim of cognitive interviews for Project M-PACT was to oversample participants with at least some experience using cigarettes, ENDS products, or alternative products like hookah or smokeless tobacco, in order to evaluate and improve questions pertaining to the experiences of users of these products. Interested participants emailed Project M-PACT about enrolling and were sent an online screener to assess their age, student status and history of tobacco use. Inclusion criteria for cognitive interviews included being a full- or part-time degree- or certificate-seeking student, from 18 to 29 years old. One student was outside this age range (32 years old). Twenty of the 25 participants had at least some experience using tobacco and nicotine products, and 14 had used them during the past 30 days. Cognitive interview participants were ages 18–32 (M = 22.1, SD = 3.2) and approximately half were female (52%). Forty-eight percent identified as non-Hispanic white, 16% as Asian, 12% as Hispanic/Latino, 8% as African American, and 16% as “other” or a combination of two or more race/ethnicities.

Cognitive Interview Procedures and Data Analysis

Participants arrived at the project office to complete a consent form and then the online Project M-PACT survey. Individual cognitive interviews immediately followed, with the participant, one interviewer, and one note-taker present. All interviews were digitally recorded and each lasted approximately 1 hour. Each participant was compensated with $40 cash, and the Institutional Review Board at the University of Texas at Austin approved all study procedures.

Two staff members conducted nearly all of the cognitive interviews and subsequently transcribed all of the interview recordings. For all 3 rounds, problematic questions were identified both by participants who reported difficulty or confusion while answering questions, as well as by the two staff coders who identified discrepancies between the information participants provided and what was intended in the original survey questions. Similar to prior research,²⁸ survey questions were revised by the two coders between rounds of cognitive interviews, known as iterative testing.²⁶ This method provided a means to “test” question revisions to determine whether the identified problems had been resolved in each round. Round 1 consisted of cognitive interviews with 10 participants, and all questions were examined for potential response problems. This round included six ever users and four current (past 30-day) users of ENDS products. Questions identified as problematic during Round 1 were revised based on transcript analysis, and then re-assessed in Round 2 with 10 additional participants (seven ever ENDS users and four current-day users). These interviews primarily focused on problems identified and addressed from Round 1. Problems that persisted with survey questions, as well as any additional problems identified in Round 2, were re-examined and revised for Round 3, with five final participants (five ever ENDS users and three current users). After Round 3, questions and answer choices were either deemed appropriate for inclusion or discarded from the survey.

Measures

Project M-PACT questions were based on existing surveys, including the Population Assessment of Tobacco and Health (PATH),²⁹ the Youth Risk Behavior Surveillance System (YRBSS), the Behavioral Risk Factor Surveillance System (BRFSS), and the National Youth Tobacco Survey (NYTS).^30–32 The Project M-PACT survey consisted of 175 total questions, with skip logic patterns throughout that only asked follow-up questions to participants with relevant experiences. The 30 initial questions concerning ENDS focused on participants’ quantity and frequency of use, including ever and past-30 day ENDS use. Survey items also included reasons for ENDS use, dependence, reported quit attempts, outcome expectations, and future intentions to use ENDS products.

Cognitive Interviews

Cognitive interviews were used to gauge the level of mutual understanding between the researchers’ survey questions and the participants’ interpretation of those questions. Interviewing techniques focused on the main categories for potential errors in Tourangeau’s four-stage model of the survey response process: (1) question comprehension, (2) the answer estimation process, (3) retrieval of the appropriate information, and (4) the final response action.²⁷ Participants were encouraged to use the “think aloud” technique in order to verbally recall the thought process used when answering each survey question,²⁶ providing researchers first-hand insight into how the participant arrived at responses for each question. Each of the cognitive interviews also included direct, tailored questions, or “probes,” about survey questions (Table 1). These prepared probes generated more precise information than think-aloud reporting and allowed comparison of responses to identical probes across interviews. Interviewers also employed spontaneous and emergent probes as new information arose during conversation.

Table 1.

Prepared Probes Used in Cognitive Interviews

1. How did you come up with your answer?

2. How confident are you of your answer?

3. Are there answer choices here that did not make sense?

4. Are there any answer choices missing in this multiple choice question?

5. Could you rephrase this question in your own words?

6. How easy or difficult was this question?

Results

Question problems were mapped onto the four stage model of the survey response process, and revisions to survey questions attempted to improve respondent comprehension, answer retrieval processes, answer judgment and estimation, and offer appropriate answer options. Problems arose with questions that assessed quantity and frequency of ENDS use, including product definitions, periods of recall, and attempts to assess amounts of ENDS products used. These questions were all edited, some remaining in the final survey, while some were eliminated (Tables 2 and 3). Few issues were found with reasons to use, aside from question edits that required expanding available answer options. Questions regarding dependence, quit attempts, outcome expectations, and future intentions to use created no problems for cognitive interview participants, therefore they remained unchanged throughout cognitive interviews and remained in the final survey (Supplementary Table 1). After cognitive interview analysis and question revision, five ENDS questions were removed for the final survey, 25 questions were revised, and 9 questions were added, resulting in 34 ENDS questions.

Table 2.

Question Revisions Based on Comprehension and Answer Estimation Issues

Issue	Original question	Discussion	Revised final question
Comprehension: Participants unable to distinguish between device types	During the last 30 days, have you used a disposable e-cigarette or an e-cigarette with a disposable cartridge? Neither requires the addition of e-liquid/e-juice.	“I was answering [the disposable question] about my use with a vape pen….If we go back, I’ve probably only used that [disposable device] once.”	During the past 30 days, have you used a disposable e-cigarette or an e-cigarette with a disposable nicotine cartridge? Neither requires the addition of e-liquid/e-juice. E-cigarettes of this type are pictured below.
	During the past 30 days, have you smoked a vape pen, personal vaporizer, or any other device that requires the addition of e-liquid/e-juice?	“I didn’t really understand; I thought I was repeating myself.”	During the past 30 days, have you smoked a vape pen, personal vaporizer, or any other device with nicotine e-liquid/e-juice? A device of this type is pictured below.
		“…it was a little disorienting because at first glance I thought [this] it was the same question.”
Comprehension: Confusion regarding product content and ENDS use	How old were you the first time you used an ENDS, (ie, e-cigarette, vape pen, or e-hookah), even one or two puffs?	“You can smoke dry herbs from the e-cigs as well. That kind of confused my answers. I’m being honest—when I smoke e-cigs it’s not tobacco. Mention e-cigs with nicotine or tobacco if that’s what you mean.”	How old were you the first time you used an ENDS product, (ie, e-cigarette, vape pen, or e-hookah), as intended (ie, with nicotine cartridges and/or e-liquid/e-juice) even one or two puffs?
Answer estimation: USE of ENDS with illegal substances potentially over/ underestimating ENDS use

ENDS = electronic nicotine delivery systems.

Table 3.

Question Revisions Based on Retrieval Process Issues

Issue	Original question	Discussion	Revised final question
Retrieval Process: Issues with frequency of ENDS use	During the last 30 days, have you used a disposable e-cigarette or an e-cigarette with a disposable cartridge? Neither requires the addition of e-liquid/e-juice.	“If I add up all the times I’ve done it, it could add up to one.”	No change.
	During the past 30 days, have you smoked a vape pen, personal vaporizer, or any other device that requires the addition of e-liquid/e-juice?	“I do it so casually, that I don’t really recall how many days.”	No change.
	On how many of the last 30 days have you used such a product?	“I just thought that I’ve only done it with friends, so I know it was then. I don’t use it as much [as cigarettes].”	No change.
		“I’ve only used it once and it was [with] my friends.”
Retrieval Process: Issues with quantity of ENDS consumed	What was the size of the bottle in milliliters (ml) from which you obtained your e-liquid/e-juice in the last month?	“I don’t pay attention that much to the size of it.”	Cut.
		“I’m with a friend and they just used the bottle; we smoked as much as we could, like a big group.”
		“I don’t even pay attention to how many milliliters it is; there is small, medium, large, and the extra-large one. It says the milliliters, but depending on the flavor and how much I like it, it will determine how big a bottle I buy.”	Cut.
	On the days you used the product in the last month, how many bottles of e-liquid/e-juice did you usually use per day?
		“I get sick of a flavor quickly, so I’ll buy quite a few. I mix my own flavors. I’ve used it for two months but I don’t know if I’ve used an entire bottle yet.”	Cut.
	On the days you used this device in the past month, approximately how many milliliters (ml) of e-liquid/e-juice did you consume on average in a given day? (≈20 drops = 1ml)
	How many bottles of e-liquid/e-juice did you use in the last 30 days?		Cut.

ENDS = electronic nicotine delivery systems.

Issues With Question Comprehension and Answer Estimation

Comprehension problems and estimation issues necessitated revisions to survey items. Question comprehension requires understanding terminology, as well as establishing a shared, specific meaning between the question researchers are asking and that which participants believe is being asked.²⁶ Answer estimation refers to the consideration a participant undertakes when choosing the level of disclosure to use in answering a question, especially regarding sensitive or private topics.²⁶ Question comprehension issues arose in two primary categories during cognitive interviews: (1) discerning between types of ENDS devices and (2) understanding the content of ENDS products. Device content also created potential issues in the answer estimation process. Table 2 highlights this process with the original question, responses from subjects, and the final question version implemented.

Discerning Between ENDS Types

The original survey asked about two categories of ENDS products: (1) “disposable e-cigarettes or e-cigarettes with a disposable cartridge,” and (2) vape pens, personal vaporizers, or any other devices that “required the addition of e-liquid/e-juice.” Hereafter these products are referred to as “disposable/rechargeable” and “refillable” types, respectively. In Round 1, three of the first six participants who responded regarding past 30-day use noted difficulty discerning between ENDS product types.

For Round 2, the portion of the question that specified device type was underlined (“a disposable e-cigarette or e-cigarette with a disposable nicotine cartridge” and “a vape pen, personal vaporizer, or any other device”) to help draw focus to the specific product description. Some confusion persisted, so photographs representative of these two types of ENDS products were added for Round 3. After inclusion of the photographs, no participants reported issues discerning the type of ENDS product each question specified.

ENDS Product Content

Early in Round 1 of cognitive interviews, one participant indicated her use of ENDS as a vehicle for marijuana intake, rather than nicotine. To avoid possible under or over-estimation of ENDS use based on sensitivity regarding delivery of marijuana or other drugs, all ENDS questions were revised to indicate use of the product “as intended, that is, with nicotine cartridges and/or e-liquid/e-juice” between Rounds 1 and 2. All remaining participants (n = 15) were able to articulate that questions specifically meant using ENDS as a vehicle for nicotine delivery, not marijuana or other substances.

Retrieval Process

The ability to answer survey questions also requires the respondent possesses the relevant information pertaining to referenced experiences. In the original survey, 12 questions attempted to assess frequency of ENDS use, as well as the quantity of disposable e-cigarettes and volume of e-liquid consumed. Participants were probed on their confidence in providing accurate answers and asked to describe the retrieval strategies they used in generating their answers. See Table 3 for original questions, subject responses, and final question versions.

Period of Recall and Frequency of Use

Eighteen ever-users were asked if they had used any ENDS products in the past 30 days. All 18 participants reported no trouble in answering this question, and each felt confident that these answers were accurate. When survey questions asked participants to report the number of days they had used each of the ENDS devices, past 30-day users reported diminished confidence in their answers. One current user noted, “if I add up all the times I’ve done it, it could add up to one [disposable e-cigarette].” In this case, the participant was referencing the entire amount of time he had ever spent smoking a disposable/rechargeable ENDS product. Of the eight past 30-day refillable users, six were confident of their answers. Each of these participants had either used it very infrequently (1 or 2 days) or very frequently (20–30 days) within the past 30 days. The two other participants, who reported 3 and 7 days using the product in the past month, reported less confidence in their answers. Because the majority of participants could at least somewhat confidently recall frequency, these questions were kept in the final survey.

Quantity and Concentration of E-liquid Consumed

In an attempt to assess the quantity of e-liquid consumed, current users of refillable-type ENDS products were asked questions regarding size of e-liquid bottles they purchased, the number of bottles of e-liquid consumed, the concentration of their e-liquid, and their device’s tank volume. Reported confidence in most of these questions varied.

Six participants were probed regarding the size of bottles purchased to fill their devices. Three participants confidently answered they did not know the size of the bottle used, while two participants had little confidence in their answer of 5 and 20 milliliters (mL). Only one participant (who confidently reported using ENDS each of the past 30 days) felt confident that his answer of 30mL was accurate. Thus, this question regarding bottle size was removed from the survey before Round 3 of the cognitive interviews.

In answering the number of bottles of e-liquid consumed daily, all six participants who received this question were confident that the answer was “less than one.” For the total number of bottles used in 30 days, five of these six respondents felt confident that their answers were accurate, though half reported zero total bottles. Questions regarding bottles used per day and per month were discarded before Round 3 of cognitive interviewing.

In another effort to specifically establish an amount of e-liquid consumed, in Rounds 1 and 2, five participants were asked the number of mL of e-liquid used each day. Three respondents chose the smallest option of “less than 1mL per day,” and one respondent chose “1–5mL.” Only one participant answered “6–10mL,” but noted the question was “very difficult” to answer. This question was also discarded before Round 3 of cognitive interviewing.

Questions regarding e-liquid nicotine concentration (in milligrams per milliliter) yielded answers some respondents felt were more accurate. All eight past 30-day refillable users felt confident in their answers, though four attributed this confidence to the fact that they selected “I don’t know” as their answer choice. The remaining four found the question easy to answer, and chose 6mg/mL (two respondents), 18mg/mL, and 24mg/mL respectively. This question was retained for use in the final survey.

Response Process

Adjusting questions based on the response process was the most straightforward procedure in the survey revision process. The original survey included questions that asked participants to indicate their reasons for choosing, trying, and using ENDS products and brands (eg, “this is a popular brand.” or “this brand is affordable.”). Two frequently employed probes were “Are there any answer choices missing in this multiple choice question?” and “Are there any answer choices here that did not make sense?” After Round 1, only one participant suggested an additional answer option. He noted that he chose his brand because “my friends have them.” This answer choice was added for Round 3.

In order to identify any other potential missing “reasons,” these questions were made open-ended during Round 2. Six additional reasons for choosing an ENDS brand were given that did not already exist in our survey draft. These were: (1) “this brand is easily refillable and/or reusable,” (2) “this brand is known for a long battery life,” (3) “this brand has easily adjustable settings,” (4) “this brand is affordable,” (5) “this brand of ENDS products is customizable,” and (6) “this brand is known to be sturdy or durable.” These choices were added to questions regarding reasons to first try a particular brand, as well as reasons to regularly use a chosen brand. After these answer options were added for Round 3, no participants suggested any new answer options.

Discussion

Cognitive interviews provided insight into the interpretations and perceived meanings that participants had of survey questions, as well as the level of disclosure participants might typically use in their responses. Examining interviews using the four-stage model of the survey response process shed light on multiple areas of difficulty assessing products as new and complicated as ENDS devices. Key lessons learned are summarized below, and Supplementary Table 1 provides a listing of the final ENDS questions used in the Project M-PACT survey.

Adding photographs representative of the specific types of ENDS devices improved question comprehension, the ability of participants to discern between disposable/rechargeable ENDS products and refillable types. The PATH survey²⁹ includes photographs of specific device types, and this practice proved valuable in the present study. Participant comprehension was also improved when language regarding the content of ENDS products was included. Much like the practice of “blunting,” where cigars are used as marijuana delivery devices,^33–35 ENDS products may be used to deliver more than just liquid nicotine.^36–38 Adding language that specifically named nicotine cartridges and liquid nicotine as the content of ENDS products allowed participants to comprehend that using ENDS products in this survey meant only as nicotine delivery devices, rather than marijuana or other drug delivery. This language appeared to eliminate any confusion regarding ENDS content while also eliminating potential answer estimation issues regarding sensitive disclosure of ENDS devices as vehicles for marijuana or other drugs.

Some participants described difficulty in reporting the frequency of their ENDS use. While they reported confidence in their ability to report whether or not they had used any ENDS products in the past 30 days, they were less sure about the specific number of days they used either device type. It appears that users are more confident when they answer either very infrequent use or very frequent use. This is not surprising, as research regarding behavior recall supports the idea that rare behaviors are enumerated specifically, while higher-frequency events are subject to more general estimation strategies.³⁹ It is likely that casual ENDS users are less likely to be able to accurately report how often they use the product as compared to rare or everyday users.

Assessment challenges were also common when survey questions attempted to quantify ENDS use. For the two current disposable/rechargeable users, questions pertaining to number of cartridges used each day and each month were kept in the survey, though there was little variability in their responses. Refillable device users frequently reported confidence when they could choose an answer choice of “I don’t know,” yet actual answers regarding bottle size, number of bottles used, and mL of e-juice consumed yielded fewer concrete answers. This may speak to the experience level of users, where more established, committed users of ENDS products may report fewer problems with questions regarding quantity and frequency, while intermittent or experimental users lack details regarding their habits of use. Including “I don’t know” answer options for questions that attempt to quantify the quantity and frequency of ENDS use appears to allow casual ENDS users to answer these questions accurately. To keep the survey at a manageable length, questions that yielded little in the way of usable information regarding quantity of ENDS consumed were cut from the final survey (Table 3).

Few issues were found with reasons to use, aside from question edits which required expanding available answer options. Questions regarding dependence, quit attempts, outcome expectations, and future intentions to use seemed to create no problems for cognitive interview participants. These questions remained unchanged throughout cognitive interviews and remained in the final survey (Supplementary Table 1).

Conclusion

Cognitive interviews yielded insight into the limitations of current questions that assess the ENDS landscape. Through their responses, it became clear that participants had difficulty discerning among ENDS product types, and steps were taken to improve this distinction. Findings also indicated that some participants had problems with items regarding quantity and frequency of use, perhaps because they are not regular users of these products. Nonetheless, it is clear that researchers need to develop survey questions that allow participants of various levels of use to accurately quantify their ENDS consumption. While young adults are the most prevalent experimenters of these products among adults,¹⁶ knowledge of the complicated nature of ENDS devices and specifics about their patterns of use may only be salient for committed, regular users. More research is therefore needed to better understand how to assess the use of ENDS products among populations with varying levels of use.

This study extends existing literature on ENDS assessment, though it has some limitations. Insights obtained in the interviews were open to interpretation and potential bias from researchers. Conducting more cognitive interviews and offering the survey to more participants would allow analysis of inter-rater reliability in identification of problem areas. More participants would also allow a quantitative assessment of changes made after question revisions. Finally, while a great deal was learned about limitations of many current ENDS questions with the current sample, focus groups with committed users of both product types would be valuable in creating survey tools better suited to the wide range of ENDS users.

Despite the aforementioned limitations, the study has a number of strengths. Oversampling tobacco and ENDS users yielded in-depth conversations on a widely variable and emerging product for which there are a limited number of assessment instruments. Testing the survey among a sample of the population that reports the highest rates of ENDS use provided insight into the myriad ways young adults interpret these questions. Lastly, the iterative process was a valuable survey development tool, allowing “testing” of question revisions in an attempt to establish consensus or justify discarding questions before field implementation of the survey.

Future Research

Future research would benefit from focus group studies with ENDS users of various types, especially regarding third generation ENDS products, which are not specifically addressed in our study. This fast-growing subgroup of products are often referred to as “mods” (an abbreviation of the word “modification”), and are refillable and customizable much like second generation devices, though they often require knowledge of electrical concepts like resistance and Ohm’s law, and involve a larger financial investment than most other devices.⁴⁰ To date there is little to no research on the complexities of these devices. Similar to the challenges in assessing first and second generation devices, mods offer a level of customization that will likely make quantification of their use difficult. Using cognitive interviews to test questions developed through future focus groups will be an important step in ENDS surveillance.

Supplementary Material

Supplementary Table 1 can be found online at http://www.ntr.oxfordjournals.org

Funding

This work was supported by the Tobacco Center of Regulatory Science on Youth and Young Adults (P50 CA180906-02) from the National Cancer Institute at the National Institutes of Health and the Food and Drug Administration Center for Tobacco Products. Contributions by OAW were also supported in part by a Career Development Award from the Food and Drug Administration Center for Tobacco Products (K01 CA189301). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the US Food and Drug Administration.

Declaration of Interests

None declared.