When national drug surveys “take too long”: An examination of who is at risk for survey fatigue

Abstract

Background:

National surveys are a leading method for estimating prevalence of substance use and other health-related behaviors. However, when a participant perceives a survey as too time-consuming, there is a higher probability of lower quality responses.

Methods:

We examined data from the 2018 to 2019 National Survey on Drug Use and Health, a nationally representative sample of non-institutionalized individuals ages ≥12 in the U.S. (N = 112,184). Participants were asked about 13 drug classes on this hour-long survey, and those reporting use of a drug were asked follow-up questions. We estimated prevalence and correlates of participants stating that the survey took too long to complete.

Results:

An estimated 9.4 % (95 % CI: 8.9–9.8) felt the survey took too long. The more drugs used in the past year, the higher the odds of reporting that the survey took too long. Those reporting use of 8–13 drug classes in particular were at higher odds (aOR = 2.91, 95 % CI: 1.44–5.87). More missing responses was associated with higher odds—particularly when ≥5 drug-related questions were skipped (aOR = 3.26, 95 % CI: 2.26–4.71). Participants who did not speak any English (aOR = 1.74, 95 % CI: 1.31–2.32), have difficulty concentrating (aOR = 1.38, 95 % CI: 1.23–1.54), and/or had trouble understanding the interview (aOR = 3.99, 95 % CI: 3.51–4.53) were at higher odds, as were those who were older and non-white. Higher education and family income was associated with lower odds.

Conclusion:

We identified subgroups of individuals most likely to experience fatigue on a national drug survey. Researchers should recognize that long surveys with extensive follow-up questions may lead to respondent fatigue.

1. Introduction

Collecting self-reported data through surveys has long been among the most common study designs employed in epidemiological drug and health research. Compared to alternative means of data collection, such as biological testing, surveys are advantageous because of their capacity to be deployed to large populations, their ultimate cost-efficiency, and overall practicality of administration (Palamar et al., 2019; Rosay et al., 2007; Safdar et al., 2016). Moreover, as electronically administered surveys have become more popular, the availability of user-friendly sophisticated software to develop and deploy surveys has increased, further adding to the convenience of survey-based study designs (Safdar et al., 2016).

Insofar as epidemiological research on drug use and health-related behaviors continues to rely on survey results, consideration must be given towards factors that can affect the accuracy and quality of survey responses, which are inherently crucial to the integrity of the resulting findings. Germane to this discussion, one of the problems commonly discussed in the survey methods literature is the phenomenon of survey fatigue (Ben-Nun, 2008; Choi and Pak, 2005; Hochheimer et al., 2016; Porter et al., 2004). Also referred to as respondent fatigue, survey fatigue refers to the phenomenon whereby respondents become tired, bored, or disinterested during a survey and provide less thoughtful answers to queries–particularly in later parts of a survey–or prematurely terminate participation as a result (O’Reilly-Shah, 2017). This encompassing definition overlaps with psychological elements of decision fatigue, which refers to the deteriorating quality of decisions after long sessions of decision making (Pignatiello et al., 2018). Respondent fatigue can also lead to “satisficing”, which is the tendency to seek quick answers that are “good enough” to complete the survey or to take shortcuts to complete a survey more quickly rather than invest time and energy towards providing the best responses (Hamby and Taylor, 2016; Krosnick, 1991). In practice, survey fatigue can manifest as skipped questions, text response fields left blank, responses left on the default answer choices, or straight-line answering in which, for example, the first option in a series of multiple-choice questions are selected (Ben-Nun, 2008). Ultimately, these behaviors can introduce undesirable bias into study findings.

While complexity of the survey questions, survey topic, and question type are factors known to influence the extent of survey fatigue, much of the existing research has focused on survey length as being the primary determinant. Although results from studies have been mixed (Rolstad et al., 2011), several studies have found that shorter surveys have produced higher completion rates in addition to higher response rates and higher re-test reliability when compared to longer surveys (Edwards et al., 2002; Galesic and Bosnjak, 2009; Kost and de Rosa, 2018). Guidelines on good survey methodology have long warned that questions towards the end of longer surveys are more likely to obtain a satisficing response (Schuman and Presser, 1996), while a study on survey response quality found that the percentage of skipped questions were somewhat higher in longer surveys (Stolzmann et al., 2019). Studies have begun to delineate differences in survey response based on demographic factors such as participant age and race/ethnicity (Stolzmann et al., 2019), but more research is needed to determine which subpopulations are at highest risk–especially regarding national surveys that estimate drug use.

Survey fatigue is an undesirable problem in epidemiological research that can introduce bias into the data. Most studies investigating the relationship between survey length and survey fatigue have primarily been measured via item response rates, which do not necessarily capture response quality. Moreover, we posit that it is also important to directly measure whether respondents themselves perceive a survey as being too long. Accordingly, to provide greater clarity to the existing research, the present study seeks to identify subgroups of individuals that are more likely to perceive that the National Survey on Drug Use and Health (NSDUH), a major nationally representative survey administered by the Substance Abuse and Mental Health Services Administration, is too long and, by extension, may be at greater risk for survey fatigue. Findings would then allow future researchers to focus their efforts on determining the extent to which survey length affects fatigue and the validity of responses among targeted subpopulations.

2. Methods

2.1. Study population

The NSDUH is an annual nationally representative cross-sectional survey of non-institutionalized individuals in the US (Center for Behavioral Health Statistics and Quality, 2019). The survey is conducted in all 50 states and the District of Columbia, and a multistage sampling design is utilized. Individuals ages 12 and older are eligible and surveys are administered via computer-assisted personal interviewing (CAPI). A trained field interviewer first asks a number of screening questions to obtain information about all household residents. This is followed by an informed consent process prior to conducting the computer-assisted interview (i.e., the full survey). Those deemed eligible and are interested in participating then complete the interview. Sensitive questions are completed using audio computer-assisted self-interviewing (ACASI), where the respondents listen to and read the questions (Substance Abuse and Mental Health Services Administration, 2019). The survey takes 60 min on average to complete as it involves several components that respondents must participate in that can add to the overall duration of the interview. The interview includes modules assessing the use of 13 drug classes. Those reporting use are asked follow-up questions focusing on separate drugs within each class, topics such as when drugs were initiated and recency and frequency of use, and those reporting past-year use also answer questions assessing use disorder as defined by DSM-IV criteria (Center for Behavioral Health Statistics and Quality, 2020a; Substance Abuse Mental Health Services Administration, 2015). The survey also contains additional modules querying demographic characteristics, physical and mental health, and perceived risk and availably of select drugs.

The NSDUH survey is only conducted in English or Spanish, and certified bilingual interviewers are available for respondents who prefer to complete the interview in Spanish. If the respondent does not speak either English or Spanish, the interview is not conducted (Center for Behavioral Health Statistics and Quality, 2020a). Participants receive $30 incentive for completing the survey.

The analysis in the present study focused on data collected in 2018 and 2019–the two most recent cohorts assessed. The weighted interview response rates for 2018 and 2019 were 66.6 % and 64.9 %, respectively. This secondary data analysis was exempt from review at the New York University Langone Medical Center institutional review board.

2.2. Measures

The survey includes separate modules querying the use of 13 drugs or drug classes, including: 1) tobacco, 2) alcohol, 3) marijuana, 4) cocaine, 5) crack, 6) heroin, 7) hallucinogens, 8) inhalants, 9) methamphetamine, 10) prescription opioids, 11) prescription tranquilizers, 12) prescription stimulants, and 13) prescription sedatives. The latter four categories focus mostly on misuse, defined by NSDUH as using the drug in a way not directed by a physician, including use without a prescription, more often, longer, or in greater amounts than directed, or use in any other way as not directed. While participants are first asked about lifetime use, it is reports of past-year use in particular that lead to more follow-up questions; therefore, past-year use was a main focus in this study. We examined indicators of past-year drug use separately, and we also created a sum score variable that indicated whether the participant used 0, 1, 2–3, 4–5, 6–7, or 8–13 drugs or drug classes. We utilized past-year drug use variables that were imputation-revised by NSDUH and thus have no missing data. However, we also created a count variable based on the number of non-imputed variables with missing data for each of the 13 drug classes based on: 1) lifetime use, 2) recency of use, 3) age of first use, and 4) frequency of past-year use. Data were considered missing if a respondent refused to answer, left the answer blank, or responded “don’t know”.

Participants were asked about demographic characteristics, including age, sex, race/ethnicity, education, and annual family income. They were also asked how well they speak English, whether they are deaf or have serious difficulty hearing, are blind or have serious difficulty seeing (even when wearing glasses), and have serious difficulty concentrating, remembering, or making decisions due to a physical, emotional, or mental condition. Finally, at the end of the survey, participants were debriefed. The interviewer recorded whether the participant had trouble understanding questions asked during the interview and whether the participant made any comments about the interview being too long (with responses for both items pre-coded as “yes” and “no”).

2.3. Analysis

We first estimated sample characteristics and whether it was reported that the survey took too long in a univariable manner. We then compared sample characteristics according to whether the participant reported that the survey took too long. These bivariable comparisons were conducted using Rao-Scott chi-square (Heeringa et al., 2010). Next, sample characteristics were fit into a multivariable logistic regression model as independent variables to determine correlates of reporting the survey being too long with all else being equal. This model included the variable indicating the sum of the number of drugs used. The model was then repeated but including indicators for each separate drug or drug class rather than the sum variable. We utilized sample weights to account for non-response, selection probability, the complex survey design, and population distribution. Stata 13 SE (StataCorp, 2013) was used for all analyses and we used Taylor series estimation methods to provide accurate standard errors (Heeringa et al., 2010).

3. Results

We estimate that 9.4 % (95 % confidence interval [CI]: 8.9-9.8) of participants that have taken the NSDUH survey in 2018–2019 felt the survey was too long. Table 1 presents correlates of reporting that the survey took too long according to bivariable tests. We detected significant differences with respect to age, race/ethnicity, education, income, participants’ ability to speak English, difficulty hearing, seeing, and concentrating, and trouble understanding the interview (ps<.001). Subpopulations reporting particularly high prevalence of the survey taking too long were those who had trouble understanding the interview (33.3 %), those who do not speak any English (23.8 %), those reporting difficulty hearing (15.0 %), seeing (14.8 %), and/or concentrating (12.2 %), and those aged ≥65 (16.4 %) or with less than a high school diploma (16.9 %). With regard to reported drug use, those reporting past-year use of alcohol, marijuana, crack, methamphetamine, heroin, and/or misuse of prescription opioids, and tranquilizers, were more likely than those not reporting use to report that the survey took too long in bivariable models (ps<.01). Those reporting use of 8–13 drugs in particular (14.7 %) reported high prevalence of the survey taking too long, with those reporting crack (18.5 %) or heroin use (16.2 %) reporting high prevalence of the survey taking too long. With respect to missing data, a quarter (25.0 %) of those who skipped five or more drug use items reported that the survey took too long.

Table 1.

Sample characteristics and bivariable correlates of participants reporting that the survey took too long to complete (N = 112,184).

	Full Sample		Survey Took Too Long
	N	Weighted % (SE)	No (90.6 %), Weighted % (SE)	Yes (9.4 %), Weighted % (SE)	P
Age, years					<.001
12–17	26,684	9.1 (0.1)	95.2 (0.2)	4.8 (0.2)
18–25	27,863	12.3 (0.1)	95.4 (0.2)	4.6 (0.2)
26–34	17,395	14.6 (0.2)	93.8 (0.3)	6.2 (0.3)
35–49	22,822	22.2 (0.2)	91.9 (0.2)	8.1 (0.2)
50–64	9,818	22.7 (0.2)	88.8 (0.4)	11.2 (0.4)
≥65	7,867	19.0 (0.3)	83.6 (0.7)	16.4 (0.7)
Sex					0.312
Male	53,743	48.5 (0.2)	90.8 (0.3)	9.2 (0.3)
Female	58,706	51.5 (0.2)	90.5 (0.3)	9.5 (0.3)
Race/Ethnicity					<.001
Non-Hispanic White	64,850	62.2 (0.4)	92.3 (0.2)	7.7 (0.2)
Non-Hispanic Black	14,454	12.1 (0.3)	87.5 (0.6)	12.5 (0.6)
Hispanic	21,325	17.1 (0.3)	87.6 (0.4)	12.4 (0.4)
Asian	5,367	5.7 (0.2)	88.2 (0.8)	11.8 (0.8)
Other/Mixed	6,453	2.9 (0.1)	89.7 (0.9)	10.3 (0.9)
Education					<.001
<High School	10,672	11.1 (0.2)	83.1 (0.7)	16.9 (0.7)
High School Diploma	22,701	22.4 (0.2)	89.4 (0.4)	10.6 (0.4)
Some College	28,754	28.1 (0.2)	91.5 (0.3)	8.5 (0.3)
College or more	23,638	29.5 (0.3)	92.2 (0.3)	7.8 (0.3)
(Age <18)	26,684	9.1 (0.1)
Household Income					<.001
<$20,000	19,929	15.2 (0.2)	86.9 (0.6)	13.1 (0.6)
$20,000-$49,999	33,482	28.7 (0.3)	89.2 (0.4)	10.8 (0.4)
$50,000-$74,999	17,527	15.6 (0.2)	91.5 (0.4)	8.5 (0.4)
≥$75,000	41,511	40.5 (0.4)	92.7 (0.3)	7.3 (0.3)
English Speaking Ability					<.001
Very Well	99,460	86.8 (0.2)	91.9 (0.2)	8.1 (0.2)
Well	8,687	8.6 (0.2)	85.0 (0.8)	15.0 (0.8)
Not Well	2,798	3.4 (0.1)	80.2 (1.2)	19.8 (1.2)
Not at All	923	1.2 (0.1)	76.2 (2.4)	23.8 (2.4)
Difficulty Hearing					<.001
No	108,385	94.8 (0.1)	91.0 (0.2)	9.0 (0.2)
Yes	3,663	5.2 (0.1)	85.0 (1.0)	15.0 (1.0)
Difficulty Seeing					<.001
No	107,083	95.5 (0.1)	91.0 (0.2)	9.0 (0.2)
Yes	4,962	4.5 (0.1)	85.2 (0.9)	14.8 (0.9)
Difficulty Concentrating					<.001
No	99,382	91.4 (0.1)	91.0 (0.2)	9.0 (0.2)
Yes	12,568	8.6 (0.1)	87.8 (0.6)	12.2 (0.6)
Trouble Understanding Interview					<.001
No	107,729	95.1 (0.1)	91.9 (0.2)	8.1 (0.2)
Yes	4,455	4.9 (0.1)	66.7 (1.3)	33.3 (1.3)
Past-Year Drug Use
Tobacco	31,040	26.5 (0.2)	90.3 (0.3)	9.7 (0.3)	0.182
Alcohol	67,853	65.2 (0.2)	91.4 (0.2)	8.6 (0.2)	<.001
Marijuana	23,466	16.8 (0.2)	91.5 (0.3)	8.5 (0.3)	0.009
Cocaine	2,731	2.0 (0.1)	90.7 (0.6)	9.3 (0.6)	0.965
Methamphetamine	882	0.7 (0.0)	86.7 (1.7)	13.3 (1.7)	0.006
Heroin	364	0.3 (0.0)	83.8 (2.6)	16.2 (2.6)	0.001
Prescription Opioids	4,501	3.6 (0.1)	87.3 (0.8)	12.7 (0.8)	<.001
Hallucinogens	3,352	2.1 (0.1)	90.8 (0.7)	9.2 (0.7)	0.820
Stimulants	3,005	1.8 (0.0)	92.0 (0.8)	8.0 (0.8)	0.143
Crack	304	0.3 (0.0)	81.5 (2.8)	18.5 (2.8)	<.001
Tranquilizers	2,719	2.0 (0.1)	87.8 (0.9)	12.2 (0.9)	0.001
Sedatives	467	0.4 (0.1)	87.2 (3.0)	12.8 (3.0)	0.192
Inhalants	1,405	0.7 (0.0)	90.3 (1.4)	9.7 (1.4)	0.831
Number of Drugs Used					<.001
0 drugs	36,791	28.0 (0.2)	89.5 (0.4)	10.5 (0.4)
1 drug	59,291	60.2 (0.3)	91.2 (0.3)	8.9 (0.3)
2–3 drugs	13,194	9.6 (0.1)	91.1 (0.5)	8.9 (0.5)
4–5 drugs	2,406	1.6 (0.0)	89.6 (0.9)	10.4 (0.9)
6–7 drugs	623	0.4 (0.0)	87.6 (1.9)	12.4 (1.9)
8–13 drugs	144	0.1 (0.0)	85.6 (4.0)	14.4 (4.0)
Number of Missing Items					<.001
0 missing	105,919	95.1 (0.1)	91.1 (0.2)	9.0 (0.2)
1–2 missing	4,685	3.7 (0.1)	83.8 (1.1)	16.2 (1.1)
3–4 missing	1,120	0.8 (0.3)	81.9 (2.3)	18.1 (2.3)
5 or more missing	725	0.5 (0.0)	75.0 (3.0)	25.0 (3.0)

Note. All data were weighted and adjusted for the complex sample design. Participants age 12–17 were not asked level of education attainment so this age group was omitted from bivariable test. Presciption opioids, tranquilizers, sedatives, and stimulants refer to misuse. SE = standard error.

As shown in Table 2, with all else being equal, age was associated with increased odds of reporting the survey is too long in a dose-response-like manner among those aged ≥26, with those aged ≥65 at 5.13 times the odds (95 % CI: 4.31–6.11) of reporting this compared to those age 12–17. Compared to those identifying as white, all other races/ethnicities were at higher odds of reporting that the survey took too long. Higher education and family income were associated with lower odds of reporting that the survey took too long. Compared to those earning less than a high school diploma, those with higher education were at lower odds for reporting that the survey was too long. Likewise, compared to those earning <$20,000 per year, participants with family income of ≥$50,000 per year were also at lower odds. Those reporting limited English proficiency were at higher odds for reporting that the survey took too long, and compared to those who reportedly speak English very well, the more limited one’s English, the higher the odds that he or she reported the survey taking too long, with those not speaking English well at all at 1.74 times the odds (95 % CI: 1.31–2.32) of reporting the survey being too long. Difficulty concentrating in general (adjusted odds ratio [aOR] = 1.38, 95 % CI: 1.23–1.54) and self-reported trouble understanding the interview (aOR = 3.99, 95 % CI: 3.51–4.53) were also associated with increased odds for reporting that the survey was too long. The more drug-related items skipped by participants, the higher the odds for reporting that the survey took too long, with those skipping five or more items at 3.26 times higher odds (95 % CI: 2.26–4.71) of reporting that the survey took too long, compared to those not skipping any items.

Table 2.

Multivariable model examining correlates of participants reporting that the survey took too long to complete (N = 111,472).

Characteristic	aOR (95 % CI)
Age, years
12–17	1.00
18–25	1.10 (0.97–1.27)
26–34	1.64 (1.43–1.90)
35–49	2.24 (1.93–2.60)
50–64	3.33 (2.85–3.89)
≥65	5.13 (4.31–6.11)
Sex
Male	1.00
Female	1.00 (0.92–1.09)
Race/Ethnicity
Non-Hispanic White	1.00
Non-Hispanic Black	1.79 (1.56–2.06)
Hispanic	1.66 (1.52–1.81)
Asian	1.71 (1.45–2.01)
Other/Mixed	1.34 (1.10–1.64)
Education
<High School	1.00
High School Diploma	0.83 (0.74–0.94)
Some College	0.78 (0.68–0.89)
College or more	0.72 (0.61–0.84)
Household Income
<$20,000	1.00
$20,000-$49,999	0.90 (0.80–1.00)
$50,000-$74,999	0.81 (0.71–0.93)
≥$75,000	0.83 (0.73–0.94)
English Speaking
Very Well	1.00
Well	1.46 (1.26–1.68)
Not Well	1.60 (1.32–1.97)
Not at All	1.74 (1.31–2.32)
Difficulty Hearing
No	1.00
Yes	1.03 (0.87–1.21)
Difficulty Seeing
No	1.00
Yes	1.17 (0.98–1.40)
Difficulty Concentrating
No	1.00
Yes	1.37 (1.23–1.54)
Trouble Understanding Interview
No	1.00
Yes	3.99 (3.51–4.53)
Number of Drugs Used
0 drugs	1.00
1 drug	1.10 (0.99–1.22)
2–3 drugs	1.46 (1.29–1.65)
4–5 drugs	1.81 (1.46–2.24)
6–7 drugs	2.62 (1.80–3.81)
8–13 drugs	2.91 (1.44–5.87)
Number of Missing Items
0 missing	1.00
1–2 missing	1.68 (1.41–2.01)
3–4 missing	1.85 (1.28–2.67)
5 or more missing	3.26 (2.26–4.71)

Note. All data were weighted and adjusted for the complex sample design. Participants age 12–17 were not asked level of education attainment so a missing data indicator was included for this age group for this variable. aOR = adjusted odds ratio; CI = confidence interval.

Concerning past-year drug use (Table 2 continued), compared to those not reporting use of any drugs queried, the higher number of drugs reportedly used in the past year, the higher the odds that the participant reported that the survey took too long. Those reporting that they had used 8–13 drugs queried in the past year were at 2.91 times the odds (95 % CI: 1.44–5.87) of reporting that the survey took too long. Finally, while considering individual drugs or drug classes as independent variables (Table 3), with all else being equal, those reporting past-year use of tobacco (aOR = 1.22, 95 % CI: 1.10–1.35) or marijuana (aOR = 1.15, 95 % CI: 1.02–1.28), or misuse of prescription opioids (aOR = 1.33, 95 % CI: 1.14–1.55) or tranquilizers (aOR = 1.35, 95 % CI: 1.07–1.70) were at higher odds of reporting that the survey took too long compared to those not reporting use.

Table 3.

Multivariable model examining correlates of participants reporting that the survey took too long to complete–focusing on individual drugs as independent variables (N = 111,472).

Past-Year Drug Use	aOR (95 % CI)
Tobacco	1.22 (1.10–1.35)
Alcohol	1.03 (0.93–1.13)
Marijuana	1.15 (1.02–1.28)
Cocaine	0.90 (0.70–1.15)
Methamphetamine	1.17 (0.86–1.60)
Heroin	1.10 (0.64–1.88)
Prescription Opioids	1.33 (1.14–1.55)
Hallucinogens	1.19 (0.98–1.43)
Stimulants	1.08 (0.82–1.41)
Crack	1.12 (0.64–1.98)
Tranquilizers	1.35 (1.07–1.70)
Sedatives	1.15 (0.64–2.09)
Inhalants	1.10 (0.77–1.56)

Note. All data were weighted and adjusted for the complex sample design. This model controlled for all covariates from the other model. Presciption opioids, tranquilizers, sedatives, and stimulants refer to misuse. aOR = adjusted odds ratio; CI = confidence interval.

4. Discussion

Surveys are the leading method of collecting national prevalence data on drug use and other health-related behaviors, but long surveys can cause respondents to experience survey fatigue and bias results. Most of the existing research on the topic has focused on survey length and response rates, and mixed findings in the literature suggest that the impact of survey length on the experience of survey fatigue may vary depending on demographic factors. To this end, the present study seeks to identify subgroups of individuals that are more likely to believe (or report) that the NSDUH survey is too long.

Overall, we found that approximately one in every ten respondents believed that the NSDUH survey took too long to complete. When examining potential correlates of perceiving the survey as taking too long, several associations emerged. For example, age groups above 25 years were at significantly greater odds of reporting that the survey took too long. Specifically, compared to those aged 12–17, those between the ages of 35–49 were at approximately twice the odds, while those aged 50–64 were at nearly triple the odds, and those 65 and older were at over five times the odds. This adds to previous literature suggesting response burden is generally known to be more problematic for older populations (Stone et al., 2007), and may be attributable, at least in part, to a digital divide with regard to age. For example, while 91 % of those aged 65 and over now own a mobile phone, only half (53 %) own a smartphone, in contrast to 96 % smartphone ownership among those aged 18–29 (Pew Research Center, 2019b). Similarly, while nearly three-quarters (73 %) of those aged 65 and over use the internet in some capacity, only 59 % have access to high-speed internet in their homes (Pew Research Center, 2019a). In general, older Americans have consistently adopted technological advances at slower rates (Levine et al., 2016; Pew Research Center, 2017), so it is conceivable that older respondents in the NSDUH may simply not be as familiar with navigating the digital interfaces and, consequently, take longer to complete the CAPIs. Indeed, previous findings show that web-based surveys yield lower response rates among older populations compared to younger populations (Mickael and Morten Bo, 2009), and a systematic review of the use of online surveys among the geriatric population has reported common challenges such as limited access to up-to-date technologies such as high-speed internet, incompatible hardware and software, difficulty with typing, and typeface sizes (Clark, 2002; Remillard et al., 2014).

In addition, it appears that those who identified racially as nonwhite (i.e., Black, Hispanic, Asian, or other) were also at greater odds of reporting that the NSDUH as being too long, as were those reporting lower household income and educational attainment. Perhaps to some extent, these associations may be related to discrepancies in access to modern technologies that vary across socioeconomic levels. For example, the proportion of smartphone ownership varies largely among those with at least a college degree (91 %) versus those with a high school diploma (72 %) or less than a high school education (66 %) (Pew Research Center, 2019b). Similar trends can be observed as it pertains to household income, and the same is true with regard to highspeed internet access at home (Pew Research Center, 2019a). Therefore, these groups of respondents may have required more time on average to complete the CAPIs in a nature similar to older respondents. However, we posit that these findings may also reflect differing levels of language proficiency and question comprehension.

Sociodemographic and economic measures such as household income, educational attainment, and race/ethnicity are known to be associated with lower levels of English proficiency (Ariani and Ghafournia, 2016; Batalova and Fix, 2010; Batalova and Zong, 2016; Kieffer, 2010; Sentell and Braun, 2012), and our findings demonstrate that those with lower levels of English proficiency (i.e., reporting not being able to speak English well or having trouble understanding the survey questions) were at notably increased odds of perceiving and reporting that the survey was too long. This finding is unsurprising given that language and comprehension difficulties with surveys, and the potential resultant need to rely on professional or informal translators, would conceivably increase both the time and cognitive demands required to complete a formal survey. This, in turn, would likely contribute to the feeling that a survey takes too long to complete, and may also increase the risk for survey fatigue or response burden. For the NSDUH, no standardized protocol is noted in the field interviewer manual for assessing English proficiency. If the interviewer is not bilingual, they appear to rely on a member of the household to serve as a translator for screening questions while interview questions cannot be translated for any reason for the survey itself (Center for Behavioral Health Statistics and Quality, 2020b). This is especially problematic if surveys are to accurately assess substance use patterns and associated health conditions among immigrant or ethnic minority populations (Johnson et al., 2006; Wenz et al., 2021).

Upon observing survey-related correlates of perceiving that the NSDUH takes too long, our analysis revealed that respondents who skipped a higher number of drug-related questions were at greater odds for reporting that the survey took too long. It is important to note that the majority of skipped questions were follow-up questions that are meant to probe for other factors (such as frequency, recency of use, or age of first use), which arise after respondents report overall use of a given drug. In particular, those who skipped five or more drug-related survey items were at over three times the odds of perceiving that the survey took too long when compared to those who did not skip any items examined. It is possible that skipped follow-up questions about drug use may be an indicator of perceiving that the survey takes too long, though additional statistical analyses would be needed to confirm this. In any case, the importance of this finding remains as skipped survey questions may introduce bias and reduce the quality of the resulting data, making this is a specific area for future research to further investigate. Our findings contrast those of some previous studies, which found no significant differences in number of missing items between longer and short surveys, though it must be considered that these surveys were not related to drug use (Dirmaier et al., 2007; Subar et al., 2001). In another study on survey response and responses quality, researchers observed that the percentage of missing items was somewhat greater among longer surveys given to patients in a mental health clinic (Stolzmann et al., 2019). These researchers further noted that relatively more skipped items occurred earlier on in longer surveys and speculated that it may have been due to front-end skipping to decrease the anticipated fatigue of longer surveys. Additional research on the impact of survey length and number of missing items as it pertains to drug use appears warranted at this time.

Additionally, our study demonstrates that report of past-year use of marijuana, cocaine, hallucinogens, and/or opioids was independently associated with greater odds of reporting that the NSDUH as being too long. We believe that this is a result of the increased number of follow-up questions that arise when respondents report use of a drug, as described above. Those reporting past-year use of a drug, for instance, are still yet prompted to answer an additional array of questions assessing potential substance use disorders, and this experience would be multiplied for those who report using multiple drugs. It is unsurprising, then, that these respondents are more likely to feel that the survey is too long, and they may also be at greater risk for survey fatigue and satisficing. However, it is further worth noting that respondents may be more inclined to skip responses for this same reason, meaning that the number of skipped questions and previous drug use are intimately intertwined and, consequently, difficult to statistically parse. Future research investigating the relationship between previous drug use and skipped NSDUH questions would be welcome since missing responses introduce unwanted bias and measurement error.

While it is possible that perceiving the survey as being too long may be associated with the actual use of these drugs, the survey sections corresponding to these drugs were objectively longer and potentially more bewildering than other drug sections. For example, those reporting tobacco use are also asked questions about what they smoke (e.g., cigarettes, cigars, pipes), and about any specific brands of products smoked. The hallucinogen section asks about use of many individual drugs including LSD, PCP, ecstasy, ketamine, tryptamines such as DMT, and salvia divinorum. This section also queries lifetime use of peyote, mescaline, and psilocybin. The prescription drug misuse sections, while not measurably longer than other drug sections in terms of number of questions, may be more difficult to complete because participants are specifically asked to think about specific criteria for misuse, not any use (e.g., prescribed use). Therefore, we believe these associations between perception of survey length and reported drug use are more so attributable to the corresponding drug sections on the survey than the use of the drugs themselves.

Finally, our study detected significant associations with respondents who reported “difficulty concentrating” and “trouble understanding the interview” stating the survey took too long to complete. A positive response to these questions could suggest cognitive impairment, especially among older adults, which has implications for survey procedures. However, the NSDUH does not appear to further assess cognitive functioning either through self-report as done in the National Epidemiologic Survey on Alcohol and Related Conditions (Aharonovich et al., 2017) or cognitive testing as the National Health and Nutrition Examination Survey performs for participants aged ≥60 (Centers for Disease Control and Prevention, 2020). A recent study using NSDUH found among adults age ≥50, “difficulty concentrating” was associated with increased odds of prescription opioid and sedative/tranquilizer misuse (Han et al., 2021). This highlights the importance of measuring cognitive functioning for both a more detailed understanding of substance use patterns among people with cognitive impairment and for identifying who may be at risk for survey fatigue. Balancing additional assessments, however, must be weighed against limiting the length of the survey.

4.1. Limitations

Participants were not directly asked whether the survey took too long; rather, it was recorded if they mentioned that it took too long on their own volition. Therefore, it is possible that others also perceived that the survey took too long but did not convey these thoughts. Difficulty seeing, hearing, and concentrating were assessed as general conditions not specific to the interview, so we do not know the extent to which these impairments affected survey administration. The time it took for each individual to complete was not available, and while a variable was available indicating that the respondent had trouble understanding the interview, data are not available regarding how many times a respondent asked for help, an explanation, or a translation. No specific data are available on attention or satisficing during the interview. Missing data was typically due to respondents leaving the answer blank or to breaking off the interview, although in some cases this could have been due to skip-logic in which a previous “don’t know” response led the program to skip a follow-up question, which subsequently leads to a blank response for that item (Center for Behavioral Health Statistics and Quality, 2020a). Indeed, “don’t know” responses can truly reflect a participant honestly replying that he or she does not know the answer, but it can also indicate an unwillingness to think about the question. Many participants who report “don’t know” are in fact willing to answer a more specific follow-up question about the same construct (including prompts) (Caspar et al., 2005), but we determined “don’t know” even as an initial response as missing data in this analysis. We must also keep in mind that differential missingness has been shown to be related to sociodemographic characteristics such as racial minority status (Owens et al., 2001).

4.2. Conclusion

Certain subpopulations appear more likely to be impacted by survey length. In particular, older adults and those with lower levels of English proficiency are at significantly greater odds of perceiving that large surveys such as NSDUH are too long and may be at increased risk of survey fatigue. Future research assessing the extent to which survey length impacts survey fatigue and quality of responses should focus on these subgroups of the population.