The impact of question order on surveillance of nicotine pouch and smokeless tobacco use among adults in the United States

Abstract

Objective

To improve understanding about measurement of smokeless tobacco and nicotine products, we conducted a survey experiment to investigate the effect of question order on calculated prevalence of use.

Methods

We embedded a split sample experiment within the May 2025 Rutgers Omnibus Study; 1891 US adults aged 18–45 were recruited from Amazon MTurk for an online survey about health behaviors, randomized to answer questions about nicotine pouches before smokeless tobacco or vice versa. We compared prevalence by experimental condition using chi-square tests, with alpha = 0.05.

Results

Prevalence of ever smokeless tobacco use was significantly higher among those who received the smokeless tobacco questions first (21.5 % vs 16.4 %, p = .005); however, there was only a small and non-significant difference in prevalence of ever nicotine pouch use (15.5 % vs 17.6 %, p = .23). Those who saw smokeless tobacco questions first were also more likely to name a nicotine pouch brand as their regular brand, suggesting that the smokeless tobacco prevalence in this group was overestimated.

Conclusion

Nicotine pouch use may be misreported as SLT use on surveys that assess smokeless tobacco first. Consequently, efforts to monitor prevalence of smokeless tobacco use as nicotine pouches gain popularity may be undermined by measurement error.

Highlights

• •Evidence of order effects in a survey assessing smokeless tobacco and nicotine pouch use.
• •Smokeless tobacco prevalence differed significantly by question order.
• •Reported brand of product may be useful for validating behavior on surveys.

1. Introduction

Nicotine pouches are small spit-free oral nicotine products that are placed in the mouth between the lip and gum; these products may contain tobacco-derived or synthetic nicotine but do not contain tobacco leaf. Nicotine pouch sales have grown rapidly and consistently in recent years, with monthly sales in the United States (US) tripling from 327 million units in July 2021 to 1046 million in May 2024 (He et al., 2025). Although prevalence in the general population remains relatively low (Dai and Leventhal, 2024; Delnevo et al., 2025), awareness and use is growing, especially among people with a history of using other tobacco and nicotine products (Delnevo et al., 2025; Hrywna et al., 2022; Sparrock et al., 2023). Yet, there remains potential for confusion of nicotine pouches with traditional smokeless tobacco product or medicinal nicotine replacement therapies (Shao et al., 2022; Unger et al., 2022; Morean et al., 2023). Our previous research demonstrated that consumers frequently confuse nicotine pouches with smokeless tobacco and that nicotine pouch question wording and inclusion of a product image can impact estimates of nicotine pouch prevalence (Bover Manderski et al., 2025).

Accurate measurement is critical for effective surveillance of tobacco and nicotine product use, especially as the marketplace continues to rapidly proliferate. Consumer confusion about nicotine pouches can impact responses to not only questions about nicotine pouches but also questions about other oral products such as traditional smokeless tobacco, if people who use nicotine pouches are misreporting their behavior as smokeless tobacco use. Indeed, survey methodology research indicates that question order and location on a survey can impact response behaviors through context effects, whereby the questions in one section of a survey can influence how a participant interprets and responds to subsequent questions (i.e., context effects) (Tourangeau et al., 2000; Johnson et al., 2019; Delnevo, 2010). For example, if a participant answers questions about a specific smokeless tobacco product on a survey and then is asked about smokeless tobacco products more generally, they may interpret the general question to be exclusive of the more specific items that preceded. Additionally, across various surveys our team has conducted (Bover Manderski et al., 2024), anecdotal feedback from participants suggests that those who use nicotine pouches may endorse smokeless tobacco use if they do not anticipate separate questions about nicotine pouches and perceive the product types to be similar. However, no studies have formally evaluated the impact of smokeless and pouch product question order (i.e., asking about nicotine pouches before vs. after smokeless tobacco) on prevalence surveys. To address this gap and improve understanding about measurement of smokeless tobacco and nicotine pouch products, we conducted a survey experiment to investigate the effect of question order on calculated prevalence of product use.

2. Methods

2.1. Study design and population

We embedded a split sample experiment within the May 2025 wave of the Rutgers Omnibus Study. Details about the Rutgers Omnibus Study are published elsewhere (Bover Manderski et al., 2024); briefly, it is a quarterly web-based survey study, initially launched in February 2022, administered to adults ages 18 to 45 and living in the U.S. Omnibus participants are recruited from Amazon Mechanical Turk (MTurk) using the CloudResearch MTurk Toolkit and can participate in multiple waves, enabling both cross-sectional and longitudinal analyses. The Omnibus study protocol has been reviewed and approved by Rutgers Institutional Review Board.

For the current study, we randomized the order of smokeless tobacco and nicotine pouch questions for all May 2025 Omnibus participants (N = 1891), such that approximately half received questions about nicotine pouches before smokeless tobacco, while the other half saw questions in the opposite order.

2.2. Measures

The survey included standard questions about ever (cigarettes: smoked at least 100 in lifetime; other products: ever use, even one time) and current (all products: now use daily or some days) use of cigarettes, e-cigarettes, cigars, smokeless tobacco, and nicotine pouches, as well as sociodemographic characteristics including age (years, measured continuously then categorized as 18–24, 25–34, and 35–45 years), sex (female, male), race/ethnicity (non-Hispanic White, Hispanic, non-Hispanic Black, non-Hispanic Asian, non-Hispanic other race, non-Hispanic multiple races), and educational attainment (some high school or less, high school graduate, associate's degree or some college, bachelor's degree, master's degree or higher).

For both products of interest to the current study (i.e., smokeless tobacco and nicotine pouches), participants were presented with a brief description of the product (the nicotine pouch description also included sample product images) and asked if they had ever used the product even one time (see Supplemental File). Those who endorsed ever using the product were then asked if they currently use the product every day, some days, rarely, or not at all, how many of the past 30 days they had used the product, and what brands they used (multiple choice plus an option to write in their brand, if their brand wasn't listed). For both products, current use was defined as now using the product every day or some days.

As with our previous work, we used responses to questions about brand to gage misclassification of smokeless tobacco and nicotine pouch use. For example, a respondent who endorsed smokeless tobacco use but then provided a nicotine pouch brand as their usual brand would be considered as misclassified with respect to smokeless tobacco use status.

2.3. Statistical analysis

We calculated prevalence of ever and current use of smokeless tobacco and nicotine pouches (separately and combined) by experimental condition and tested for differences using chi square tests, with alpha set at 0.05. We also compared prevalence of potential misclassification, based on reported brand as described above. As a post hoc analysis and consistent with previous studies (Bover Manderski et al., 2025; Kontopantelis et al., 2018), we additionally explored heterogeneity of effect by age, sex, race/ethnicity, and educational attainment using logistic regression models with interaction terms. All analyses were performed using SAS Software version 9.4 (SAS Institute, Cary, NC, USA).

3. Results

Sociodemographic and tobacco use characteristics were similar across study groups (Table 1). Among participants who saw nicotine pouch questions first, a majority were aged 35 to 45 years (53.3 %), female (57.6 %), non-Hispanic White (67.6 %), had at least a bachelor's degree (55.0 %), and had ever used cigarettes (74.1 %), cigars (62.6 %), and/or e-cigarettes (56.7 %). Similarly, a majority of participants who saw smokeless tobacco questions first were aged 35 to 45 years (56.7 %), female (56.2 %), non-Hispanic White (68.3 %), had at least a bachelor's degree (53.9 %), and had ever used cigarettes (73.5 %), cigars (61.4 %), and/or e-cigarettes (56.2 %).

Table 1.

Participant Characteristics by Order of Product Questions, Rutgers Omnibus Study, United States, Wave 13, May 2025 (N = 1891).

	Nicotine Pouch First (N = 955)		Smokeless Tobacco First (N = 936)
	n	%	n	%
Age
18–24	68	7.1	49	5.2
25–34	378	39.6	356	38.0
35–45	509	53.3	531	56.7
Age (Mean, SD)	34.8	6.4	35.3	6.2
Sex
Female	549	57.6	525	56.2
Male	404	42.4	410	43.9
Race/Ethnicity
Non-Hispanic White	646	67.6	639	68.3
Hispanic	103	10.8	107	11.4
Non-Hispanic Black	95	10.0	89	9.5
Non-Hispanic Asian	62	6.5	52	5.6
Non-Hispanic Other	8	0.8	13	1.4
Non-Hispanic Multi-race	41	4.3	36	3.9
Educational Attainment
Some high school or less	13	1.4	18	1.9
High school graduate	121	12.7	128	13.7
Associate's degree or some college	296	31.0	286	30.6
Bachelor's degree	373	39.1	352	37.7
Master's degree or higher	152	15.9	151	16.2
Ever Use
Cigarettes	708	74.1	688	73.5
Cigars	598	62.6	575	61.4
E-cigarettes	541	56.7	526	56.2
Smokeless tobacco	157	16.4	201	21.5
Nicotine pouches	168	17.6	145	15.5
Current Use
Cigarettes	227	23.8	192	20.5
Cigars	57	6.0	53	5.7
E-cigarettes	207	21.7	157	16.8
Smokeless tobacco	19	2.0	21	2.2
Nicotine pouches	43	4.5	35	3.8
First-time participant	306	32.0	289	30.9

Prevalence of ever smokeless tobacco use was significantly higher among those who received the smokeless tobacco questions first relative to those who received the nicotine pouch questions first (21.5 % vs 16.4 %, p = .005, Fig. 1); however, there was only a small and non-significant difference in prevalence of ever nicotine pouch use (15.5 % vs 17.6 %, p = .23) and prevalence of ever smokeless tobacco or nicotine pouch use (25.8 % vs 27.4 %, p = .417). There were no significant differences in current use prevalence for either product, but this may be partially attributed to the branching logic of the survey—respondents had to first endorse ever use of the product before seeing questions about current use. We did not observe evidence of effect modification by age, sex, race/ethnicity, or educational attainment (all interaction term p-values > .2, data not shown).

Fig. 1.

Prevalence of Smokeless Tobacco and Nicotine Pouch Use by Order of Product Questions, Rutgers Omnibus Study, United States, Wave 13, May 2025 (N = 1891).

NOTE: P-values correspond to chi-square tests

Among those who saw smokeless tobacco questions first and reported having a regular smokeless tobacco product brand (n = 21), 14 % named Zyn (i.e., a nicotine pouch brand) as their regular brand, suggesting that the smokeless tobacco prevalence in this group might have been overestimated (data not shown). In contrast, 0 % of those who saw nicotine pouch questions first named a nicotine pouch brand as their usual smokeless tobacco brand. In both experimental groups, there was no evidence of pouch use misclassification, as zero participants gave a smokeless tobacco product as their usual nicotine pouch brand. Moreover, in the group who saw smokeless tobacco questions first, all of the respondents who named a nicotine pouch brand as their regular smokeless tobacco brand subsequently endorsed nicotine pouch use; as such, they may have been misclassified as using both products instead of only nicotine pouches.

4. Discussion

In this split sample survey experiment to assess the impact of question order on calculated prevalence of smokeless tobacco and nicotine pouch use, survey participants who were presented with smokeless tobacco questions first (i.e., before nicotine pouch questions) had higher prevalence of ever smokeless tobacco use, and responses to questions about regular brand suggested that this was driven, at least in part, by measurement error. Specifically, a portion of respondents who used nicotine pouches endorsed smokeless tobacco use when they were asked about smokeless tobacco first and then subsequently endorsed nicotine pouch use when they were presented with the second set of questions. Consequently, this portion of participants would be misclassified as using both products, when they actually use only one.

Although research about the health effects of nicotine pouches is limited, research to date indicates that these products contain lower levels of carcinogens than traditional tobacco (i.e., tobacco-containing products such as cigarettes, cigars, or smokeless tobacco) and are more closely aligned with medicinal nicotine formulations (e.g., nicotine replacement therapy) (Felicione et al., 2025; Travis et al., 2025; Zamarripa et al., 2025). As such, people who switch from traditional tobacco to nicotine pouches may be reducing exposure to harmful constituents; however, such a behavior change is difficult to monitor if people who use pouches are being misclassified as using (traditional) smokeless tobacco. In addition, estimated prevalence of smokeless tobacco product use will be inaccurate if participants in health surveillance studies systematically misreport their use. Notably, ongoing national surveillance systems that assess both smokeless tobacco and nicotine pouch use (e.g., Population Assessment of Tobacco and Health Study (United States Department Of Health And Human Services et al., 2016), National Youth Tobacco Survey (Centers for Disease Control and Prevention, 2023), Tobacco Use Supplement to the Current Population Survey (US Department of Commerce, Census Bureau, 2023)) currently ask smokeless tobacco questions first. These and other prevalence studies may consider adjustments to their question order, such as randomizing order of appearance, to mitigate order effects. Researchers analyzing these data sets should recognize this potential limitation and consider also reporting a combined smokeless tobacco-nicotine pouch measure. In addition, researchers collecting primary data on smokeless tobacco or nicotine pouch use should consider assessing brands used or adding confirmation questions to study instruments (to confirm use of smokeless tobacco and/or nicotine pouches).

This study is subject to limitations. First, despite being a large and diverse sample of US adults, our sample is restricted to MTurk workers ages 18 to 45 years and thus should not be considered nationally representative. In addition, data are self-reported, and some participants may have participated in previous iterations of the Rutgers Omnibus Study. However, the proportion of participants who had participated in a prior wave was similar across study conditions, suggesting that any influence of prior participation would be balanced across groups. Finally, the Omnibus study does not recruit participants according to their tobacco use status; thus, our sample included a relatively small number of respondents who use tobacco, especially smokeless tobacco and nicotine pouches. Additional research among samples of people who use tobacco are warranted to further elucidate misclassification.

This study also has important strengths and implications for tobacco research. The randomized experimental design maximizes the internal validity of our results and demonstrates how assessment of “usual brand” can be used to validate product use when other verification is not feasible, such as on surveys. Although this approach may not be sufficient for verifying use among people who use both smokeless tobacco and nicotine pouches (e.g., a participant might report the brand they use most often, even if they used both product types), it is useful for distinguishing between people who use nicotine pouches from those who use smokeless tobacco. Additional research is needed to improve verification methods among dual users, which could include asking for all brands used or adding explicit confirmation questions.

In summary, people who use nicotine pouches may be misreporting this behavior as smokeless tobacco use on surveys that ask about smokeless tobacco first. Consequently, efforts to monitor changes in prevalence of smokeless tobacco use as nicotine pouches increase in popularity may be undermined by measurement error. These findings especially have implications for ongoing national surveillance systems that assess both product types.

CRediT authorship contribution statement

Michelle T. Bover Manderski: Writing – original draft, Methodology, Investigation, Conceptualization. Nishi J. Gonsalves: Writing – review & editing, Visualization, Validation, Formal analysis. Mary Hrywna: Writing – review & editing, Conceptualization. Cristine D. Delnevo: Writing – review & editing, Resources, Funding acquisition.

Funding

Data collection was funded by the Rutgers Institute for Nicotine & Tobacco Studies. Author effort was supported by the National Cancer Institute (U01CA278695, PI: Cristine Delnevo) and the National Institute on Drug Abuse (R01DA063630, PI: Bover Manderski).

Declaration of competing interest

The authors declare that they have no conflicts of interest.

Appendix A. Supplementary data

Supplemental File

Smokeless Tobacco and Nicotine Pouch Survey Items.

Data availability

Data will be made available on request.