Impact of Question Type and Question Order on Tobacco Prevalence Estimates in US Young Adults: A Randomized Experiment

Abstract

The purpose of this study was to identify the effects of question type, order, and inclusion of product images on tobacco use estimates in a national sample of young adults. Participants aged 18–34 years (N = 4,100) in the Truth Initiative Young Adult Cohort Study (2016) were randomized to one of five question types assessing ever and past 30-day use of tobacco products: (1) “select all that apply” list (checklist, CL); (2) breakout items for each product (B); (3) breakout + images (B + I); (4) CL and B; and (5) CL and B + I. The order of question type was randomly assigned in groups 4 and 5. Bivariate analyses estimated product-specific prevalence by question type/order. Ever cigarette and cigar use prevalence was higher and ever e-cigarette use was lower in B and B + I than in CL. Ever hookah use was higher in B + I than in CL. Past 30-day e-cigarette use was 8.3% higher and past 30-day smokeless use was 13.0% higher in B + I than in CL. In groups 4 and 5, higher prevalence of ever cigarette, cigar, hookah, and past 30-day smokeless use was observed when B was presented first. Question type, order, and inclusion of images affect prevalence estimates of tobacco use.

Implications

This study identifies the effects of question type, order, and inclusion of product images on tobacco use estimates in a national sample of young adults. Ever use and past 30-day use prevalence estimates of specific products were affected by respondents answering breakout items or breakout items with images compared with respondents answering a checklist of items in the survey. Current surveys that include a “select all that apply” list format may be underestimating ever and past 30-day prevalence estimates of tobacco products.

Introduction

From 2011 to 2015, the National Youth Tobacco Survey noted significant changes in the prevalence of some tobacco products among youth¹; this coincided with changes in question type and order. The effect of question type has been analyzed in youth,² but not the effect of question order. The purpose of this study was to identify the effects of question type, order, and inclusion of product images on ever and past 30-day tobacco use estimates in young adults across a range of products (cigarettes, cigars, little cigars/cigarillos, e-cigarettes, smokeless, hookah).

Methods

This study used data from Wave 9 (N = 4,100; Spring 2016) of the Truth Initiative Young Adult Cohort Study, a nationally representative longitudinal cohort; details on the study have been described elsewhere.³ The cohort includes young adults ages 18–34 years at study entry selected from GfK’s Knowledge Panel, an online panel of adults aged 18 and older (http://www.gfk.com/en-us/products-a-z/us/knowledgepanel-united-states/).

The Wave 9 survey included an experiment to compare ever and past 30-day prevalence estimates based on question type and question order. Respondents were randomized to one of five question types: (1) a “select all that apply” list of products (checklist, CL); (2) individual items for each product (breakout, B); (3) breakout items with images for each product (B + I); (4) the checklist and breakout items for each product (CL and B); and (5) the checklist and breakout items with images for each product (CL and B + I). For participants in the latter two conditions, the order of question type (whether respondents received CL or B/B + I items first) was randomly assigned. The order of products described in the checklist and breakout items was constant across conditions. All respondents were asked about ever and past 30-day use of each of the following tobacco products: cigarettes, cigars, little cigars/cigarillos/bidis, e-cigarettes, smokeless tobacco (ie, chewing tobacco, dip, snuff), and hookah/shisha/water pipe (hookah). For each product, never users or those who refused to answer were categorized as never users of [product]. Respondents who reported never using the product or who refused to answer either the ever use question or the past 30-day question were categorized as nonusers of [product].

To avoid within-person inconsistencies of prevalence estimates between question order and type in groups 4 and 5, we categorized respondents into three groups, depending on the first question type they received (CL group, B group, or B + I group). We combined group 1 (CL) with those who received CL items first in group 4 (CL and B) and group 5 (CL and B + I). We also combined group 2 (B) with those who received B items first in group 4 (CL and B), and we combined group 3 (B + I) with those who received B + I items first in group 5 (CL and B + I). Analyses were conducted using Stata/SE 15.0. Welch’s approximation of two-sample t tests with unequal variances was used to assess significant differences in product-specific prevalence by question type/order.

Results

In the full sample (N = 4,100), 39.7% were included in the CL group, 30.7% in the B group, and 29.6% in the B + I group.

Compared with respondents in the CL group (Table 1), those in the B group and B + I group reported significantly higher prevalence estimates of ever cigarette use (CL: 41.0%, B: 49.4%, B + I: 48.5%) and ever cigar use (CL: 19.3%, B: 25.2%, B + I: 23.8%; all p’s < .001).

Table 1.

Impact of Question Type on Ever and Past 30-Day Product-Specific Use in Full Sample (N = 4,100)

	Checklist group (n = 1,627; 39.7%)a	Breakout group (n = 1,260; 30.7%)b	Difference from checklist			Breakout + images group (n = 1,213; 29.6%)c	Difference from checklist
	%	%	%	PR	p f	%	%	PR	p f
Ever use (Y/N—full denominator)d
Cigarettes	41.0	49.4	8.4	1.20	.00	48.5	7.5	1.18	.00
Cigars	19.3	25.2	5.9	1.31	.00	23.8	4.5	1.23	.00
Little cigars/cigarillos	24.3	26.0	1.6	1.07	.32	27.0	2.7	1.11	.10
E-cigarettes	19.6	16.1	−3.4	0.82	.02	16.7	−2.8	0.86	.05
Smokeless	9.0	8.8	−0.2	0.97	.83	9.5	0.4	1.05	.69
Hookah	21.0	21.9	0.9	1.04	.57	24.4	3.4	1.16	.03
Past 30-day use (Y/N—full denominator)e
Cigarettes	11.7	11.9	0.2	1.02	.85	12.3	0.6	1.05	.62
Cigars	1.1	1.5	0.4	1.36	.35	2.1	1.0	1.86	.05
Little cigars/cigarillos	2.8	2.6	−0.2	0.95	.81	3.1	0.3	1.10	.66
E-cigarettes	3.4	2.8	−0.6	0.82	.35	4.3	0.9	1.27	.22
Smokeless	1.4	1.5	0.1	1.07	.83	2.4	1.0	1.70	.06
Hookah	1.7	1.3	−0.5	0.74	.32	2.0	0.3	1.15	.62
Past 30-day use (Y/N—among ever users)
Cigarettes	28.5	24.1	−4.4	0.85	.07	25.3	−3.2	0.89	.21
Cigars	5.1	6.0	0.9	1.17	.63	8.7	3.6	1.70	.09
Little cigars/cigarillos	10.1	10.1	0.0	1.00	1.00	11.3	1.2	1.12	.61
E-cigarettes	17.3	17.2	−0.1	1.00	.99	25.6	8.3	1.48	.03
Smokeless	12.2	17.1	4.9	1.40	.28	25.2	13.0	2.06	.01
Hookah	7.6	5.8	−1.8	0.76	.37	8.1	0.5	1.07	.81

PR, prevalence ratio.

^aThe checklist sample includes those who just received the checklist questions (group 1), those who received the checklist and the breakout items (group 4) with the checklist presented first, and those who received the checklist and the breakout items with pictures (group 5) with the checklist presented first.

^bThe breakout item sample includes those who just received the breakout items (group 2) and those who received the breakout items and checklist (group 4) with the breakout items presented first. Breakout items assessing product-specific ever use: “Have you ever tried [product]?” (yes/no). Breakout items assessing product-specific past 30-day use: “During the past 30 days, on how many days did you use [product]?” (0 days, 1–2 days, 3–5 days, 6–9 days, 10–19 days, 20–24 days, 25–30 days). Respondents who reported using the product at least 1–2 days were categorized as past 30-day users.

^cThe breakout item + picture sample includes those who just received the breakout items with pictures (group 3) and those who received the breakout items with pictures and checklist (group 5) with the breakout items with pictures presented first.

^dRespondents who reported never using the product or refused to answer the question were categorized as “never users.”

^eRespondents who reported never using the product or refused to answer either the ever use question or the past 30-day question were categorized as “nonusers.”

^fWelch’s approximation of two-sample t test with unequal variance. Bolded text denotes statistical significance at p < .05.

The B + I group reported significantly higher prevalence of ever hookah use compared with the CL group (24.4% vs. 21.0%; p = .03). However, those in the B group reported significantly lower prevalence and those in the B + I group reported marginally lower prevalence of ever e-cigarette use compared with those in the CL group (B vs. CL: 16.1% vs. 19.6%; p = .02; B + I vs. CL: 16.7% vs. 19.6%; p = .05). In the full sample, the B + I group reported marginally significant higher estimates of past 30-day prevalence of cigar and smokeless use compared with the CL group (cigar: 2.1% vs. 1.1%; p = .05; smokeless: 2.4% vs. 1.4%; p = .06).

Among respondents who reported ever using the product, breaking out the items and including images resulted in an 8.3% higher estimate of past 30-day e-cigarette prevalence compared with listing e-cigarettes in a checklist (25.6% vs. 17.3%; p = .03) and a 13.0% higher estimate of past 30-day smokeless prevalence compared with listing smokeless tobacco in a checklist (25.2% vs. 12.2%; p = .01).

Among respondents in group 4 (CL and B), those who first received the breakout items reported higher prevalence estimates of ever cigarette and ever cigar use compared to those who first received the checklist of products (cigarettes: 50.0% vs. 40.3%; p = .01; cigars: 25.9% vs. 19.7%; p = .04; Appendix Table 1). There were no significant differences in past 30-day prevalence estimates between those who first received the checklist of products first and those who received the breakout items first.

Among respondents in group 5 (CL and B + I), those who first received B + I reported significantly higher prevalence estimates of ever cigarette use (49.0% vs. 40.3%; p = .01) and ever hookah use (25.3% vs. 19.3%; p = .04). Those who first received B + I also reported significantly higher prevalence of past 30-day smokeless use compared with those who first received CL (2.5% vs. 0.2%; p = .01). Among ever smokeless users in group 5, those who first received B + I reported significantly higher prevalence of past 30-day smokeless use compared to ever users who first received CL (24.4% vs. 3.0%; p = .01).

Conclusions

Question type, order, and inclusion of images affect prevalence estimates of tobacco use. Trend analyses in repeated cross-sectional and longitudinal studies must account for changes in survey items across waves. Recall bias of ever using tobacco products may be reduced by the incorporation of breakout items and images. Current surveys that include a “select all that apply” list format may be underestimating ever and past 30-day prevalence estimates of tobacco products. This underestimation may affect overall tobacco use estimates if analyses combine responses to product-specific prevalence estimates to form overall tobacco use prevalence estimates. Furthermore, subsequent survey questions that are conditional on ever or past 30-day tobacco use should be interpreted with caution when surveys use the “select all that apply” list format to assess tobacco use.

Funding

Data collection for the Truth Initiative Young Adult Cohort Study was funded by Truth Initiative. AJ, AV, AG, and JP were supported by Truth Initiative. AV was also supported by the Tobacco Centers of Regulatory Science (TCORS) award P50DA036114 from the National Institute on Drug Abuse, and the Centers of Biomedical Research Excellence award P20GM103644 from the National Institute of General Medical Sciences. JP was also supported by a National Institutes of Health K01 Career Development Award in Tobacco Control Regulatory Research (K01DA037950). CD was supported by awards R01CA149705 and R01CA190444 from the National Cancer Institute .The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Food and Drug Administration.

Declaration of Interests

None declared.