Underreporting of drug use on a survey of electronic dance music party attendees

Abstract

Objectives:

Skip-logic is commonly used on electronic surveys in which programs provide follow-up questions to affirmative responses and skip to the next topic in response to non-affirmative responses. While skip-logic helps produce data without contradictory responses, erroneous non-affirmative reports can lead to loss of accurate information. We examined the extent to which type-in drug use responses contradict unreported use in a survey of a high-risk population—electronic dance music (EDM) party attendees.

Design:

We surveyed 1029 EDM party-attending adults (ages 18–40) using time-spacing sampling in 2018. We examined the extent to which reporting of recent drug use via type-in responses occurred after past-year use of the same drugs were unreported earlier on the same survey. Changes in prevalence of use and predictors of providing discordant responses were examined.

Results:

3.6% of participants typed in names of drugs they had used that they did not report using earlier on the survey. Changes in prevalence were not significant when correcting contradictory responses, but prevalence of past-year cocaine use increased from 23.3% to 24.3%. Those with a college degree were at lower odds for providing a discordant response (aOR = 0.13, p = .019). Females (aOR = 2.82, p = .022), those earning ≥$1000 per week (aOR = 11.03, p = .011), and those identifying as gay/lesbian (aOR = 5.20, p = .032) or bisexual or other sexuality (aOR = 15.12, p < .001) were at higher odds of providing a discordant response.

Conclusions:

Electronic surveys that query drug use can benefit from follow-up (e.g. open-ended) questions not dependent on previous responses, as they may elicit affirmative responses underreported earlier in the survey.

Introduction

Self-report via electronic surveys is now the most common means of assessing the extent and nature of drug use in epidemiological research. Compared to other methods such as testing for biologic markers, electronic surveys tend to be more practical, cost-efficient, and allow for the collation of more varied and detailed information (Rosay et al. 2007; Safdar et al. 2016; Palamar, Le, et al. 2019). A crucial aspect of producing accurate estimates of drug use is reliability of the data, particularly as it pertains to consistency of self-reported responses. For example, if a respondent initially does not report lifetime use of a drug (veracity assumed), then that respondent should again respond non-affirmatively when subsequently queried about use of the same drug at the end of the survey or on another survey. When inconsistent responses occur, reliability is diminished because of increased variability, and statistical estimates can be biased (Napper et al. 2010; King et al. 2018). Additional research that allows us to better understand the nature of inconsistent reporting is important in improving the reliability of survey-based designs.

In addition to their practicality, electronic surveys permit the use of skip-logic. In skip-logic methodology, an individual’s response to a question determines whether relevant follow-up questions on that topic will be asked. Aside from reducing participant burden stemming from being asked potentially irrelevant questions, this method is also advantageous in allowing researchers to conveniently query a wide variety of phenomena (Swanson et al. 2014), and making data management and analysis easier since skip-logic prevents inconsistent answers (e.g. reporting no lifetime ecstasy use, but then later reporting past-year ecstasy use).

However, skip-logic has limitations. Several studies examining skip-logic methodology have noted that measurement error can occur. For example, on one survey, nearly half (44.3%) of self-reported aborted suicide attempts would not have been queried, erroneously, owing to respondents answering negatively to gate questions about psychiatric disorders such as depression (Barber et al. 2001). Another study found that prevalence of self-reported past-year cocaine use based on data from the National Household Survey on Drug Abuse (NHSDA) increased when participants were provided multiple chances to report use (Lessler et al. 2000). These types of measurement errors often lead to underreporting whereby use of a drug is (unintentionally) unreported and there is not a subsequent opportunity to report use (Karam et al. 2014).

Other studies have recently investigated conflicting responses within the same surveys, though with a focus on discordant responses between use of a drug class and specific drugs within a drug class. For example, a recent study on the impact of gate questions found that nearly one in ten (9.3%) participants reporting no “bath salt” use via gate question later reported use of a “bath salt” such as mephedrone, methedrone, or methylone, despite these drugs having been listed as examples of “bath salts” in the gate question (Palamar, Acosta, Calderon, et al. 2017). Other studies have investigated contradictory responses on Monitoring the Future (MTF), an annual survey of a nationally representative sample of high school seniors in the US that, until recently, was conducted using pencil and paper. One study found that among those not reporting past-year nonmedical opioid use (with Vicodin and OxyContin listed as examples of opioids), 37.1% and 28.2% later reported past-year nonmedical Vicodin use and/or nonmedical OxyContin use, respectively, on the same survey (Palamar et al. 2016). A similar analysis of contradictory responses in MTF found that among students not reporting past-year nonmedical amphetamine use (with Adderall having been provided as an example of an amphetamine), 28.7% later reported past-year nonmedical use of Adderall on the same survey (Palamar and Le 2017).

In an effort to further examine the extent to which not reporting known drug use may contribute to measurement error on electronic surveys, this paper seeks to investigate the extent to which type-in drug use responses from a high-risk population—electronic dance music (EDM) party attendees (Kurtz et al. 2013; Palamar, Griffin-Tomas, et al. 2015; Hughes et al. 2017; Palamar, Acosta, Ompad, et al. 2017)—contradict unreported use within the same survey.

Methods

Procedure and participants

Participants were surveyed throughout the summer of 2018 using time-space sampling (MacKellar et al. 2007). Each week, parties (primarily at nightclubs) were randomly selected to survey attendees. A list of upcoming EDM parties in NYC (located primarily in Brooklyn and Manhattan) was created each week. The list was based on websites that sell EDM party tickets, EDM party listings on social media (e.g. Facebook), and recommendations from key informants. We considered parties from ticket websites eligible for random selection if ≥15 tickets were purchased for the party by mid-week. Parties were randomly selected each week using R software (R Core Team 2013). Recruitment typically occurred on one to two nights per week, on Thursday through Sunday. Time slots could not be randomly selected as most EDM parties in NYC end at 4am (with a few ending at 5am or 6am). Recruitment was typically conducted between 11:30pm and 1:30am to reach attendees about to enter parties. While most participants (70.4%, n = 724) were surveyed outside of nightclubs, participants were also surveyed outside of two large daytime festivals (29.6%, n = 305), which were not selected via random selection. All parties and festivals were limited to those that solely played EDM music (e.g. festivals that played other music in addition to EDM were excluded).

Passersby were eligible if they were between 18–40 years old, and about to enter the randomly selected party or festival. Recruiters approached passersby (who were alone or in groups), and if eligible, were asked if they would be willing to take a survey about drug use. Since inebriation was a concern (Aldridge and Charles 2008) recruiters tried to ensure that individuals were not visibly intoxicated. Specifically, they ensured that those approached before entering parties did not exhibit slurred speech or display impaired attention or gait. After informed consent was provided, surveys were taken on tablets. Participants were compensated $10 USD for completing the survey. Surveys were administered outside of 22 randomly selected parties plus the two festivals and the overall survey response rate was 74%.

Measures

Participants were asked about their age, sex, race/ethnicity, education, weekly income, and sexual orientation. They were also asked about their frequency of past-year nightclub, festival and/or other EDM party attendance. Participants were then asked about past-year use of various drugs. This investigation focused on use of 11 drugs (i.e. marijuana, ecstasy/MDMA/Molly, powder cocaine, LSD, shrooms, amphetamine [e.g. Adderall], ketamine, synthetic cannabinoids, GHB, methamphetamine, NBOMe) and 6 drug classes. For each drug class, lists of drugs were provided together on the same survey page and affirmative use of any drug in the class was coded as use of the drug class. The classes were prescription opioids (e.g. Vicodin), benzodiazepines (e.g. Xanax), synthetic cathinones (“bath salts”; e.g. methylone), tryptamines (e.g. DMT), 2C series (e.g. 2 C-B), and other new psychedelics (e.g. AL-LAD). With regard to use of amphetamine, opioids, and benzodiazepines, we only queried nonmedical use, which was defined as using without a prescription or in a manner in which it was not prescribed to the user; for example, to get high. Thus, we only queried illegal use of each drug.

The last page asked participants if they had used any drugs in the past day. Those checking ‘yes’ were asked to type in the name(s) of the drug(s) they had used. The survey also asked those reporting past-year use of each drug whether they had experienced an adverse outcome related to use of that drug in the past year (Palamar, Acosta, et al. 2019). Those answering affirmatively were asked to type in names of any other drugs they co-used before the adverse outcome in the past year. Of the drugs that were typed in, we determined those which were not reported elsewhere on the survey. We created indicator variables for each drug and further coded variables adding in type-in responses (for past-day use and for drugs co-used with other drugs before the experience of an adverse effect) for drugs not previously reported. We also coded a variable indicating whether any discordant response occurred—with discord being defined as typing in the name of a recently used drug after answering non-affirmatively when queried about use of that drug earlier in the survey.

Probability weights

Participant selection probabilities were computed based on self-reported frequency of party attendance. Weights included the number of party attendees (tracked via a clicker) who passed a predetermined recruitment line near the party entrance (MacKellar et al. 2007). Weights were inversely proportional to both the frequency of attendance component and to the party-level response rate and number-attending components. Weight components were combined by multiplication and normalized. This up-weighting of those believed to have a lower probability of selection (because they were less likely to go out and be surveyed) and down-weighting of those believed to have a higher probability of selection (because they were more likely to go out and be surveyed) is often used in other studies with venue-based sampling (MacKellar et al. 2007; Jenness et al. 2011).

Analyses

We first examined descriptive statistics of the sample. We then estimated the prevalence of use of each drug and the “adjusted” prevalence of each drug, accounting for additional type-in responses. We also subtracted prevalence from adjusted prevalence to compare estimates. Comparison of estimates was conducted using McNemar tests. Finally, we compared each covariate according to whether a discordant type-in response was provided, and we then fit all covariates into a multivariable logistic regression which produced adjusted odds ratios (aORs) for each covariate. Weights were utilized in all analyses and Taylor series estimation was used to obtain accurate standard errors (Heeringa et al. 2010). Data were analyzed using Stata 13 SE (StataCorp 2013).

Results

Sample characteristics of those who were administered the street-intercept survey are presented in Table 1. The majority of the sample were male (61.3%), heterosexual (81.5%), and had a college degree or higher (58.6%). Overall, 3.6% of participants typed in the name of a drug they had used that they did not report using earlier in the survey. Of those providing a discordant response, 84.4% provided one discordant response and 15.6% provided two discordant responses. As shown in Table 2, prevalence estimates were not drastically altered when adding in discordant type-in responses, although prevalence of powder cocaine use increased by 1% and prevalence of amphetamine use increased by 0.8%. No changes in prevalence were statistically significant.

Table 1.

Sample characteristics (n = 1019).

	Weighted %	n
Age
18–24	49.5	504
25–40	50.5	525
Sex
Male	61.3	604
Female	38.7	425
Race/Ethnicity
White	43.9	494
Black	10.0	84
Hispanic	22.1	207
Asian	15.4	159
Other/Mixed	8.5	85
Education
High school diploma or less	18.2	134
Some college	23.2	245
College degree or higher	58.6	650
Weekly income
<$500	33.8	368
$500–$999	37.5	377
≥$1000	28.7	284
Sexual orientation
Heterosexual	81.5	786
Gay/Lesbian	11.8	108
Bisexual or other sexuality	6.6	135
Number of drugs reportedly used in past year
0–1 drugs	49.0	421
2–4 drugs	39.9	400
≥5 drugs	11.1	208

Number of drugs reportedly used in the past year is the uncorrected number of drugs used—not including updated responses considering type-in responses.

Table 2.

Prevalence of drug use before and after correction given type-in responses.

	Past-year prevalence, Weighted % (n)	Adjusted past-year prevalence, Weighted % (n)	Change in prevalence after adjustment, Weighted % (n)
Any drug	77.9 (840)	78.0 (843)	0.1 (3)
Marijuana	64.8 (735)	65.1 (742)	0.3 (7)
Ecstasy/MDMA/Molly	30.5 (421)	31.1 (423)	0.6 (2)
Powder cocaine	23.3 (345)	24.3 (355)	1.0 (10)
LSD	18.0 (235)	18.3 (240)	0.3 (5)
Shrooms (psilocybin)	14.9 (220)	14.9 (221)	0.0 (1)
Amphetamine	9.8 (147)	10.6 (156)	0.8 (9)
Ketamine	9.4 (166)	9.4 (166)	0.0 (0)
Prescription opioids	9.4 (99)	9.7 (102)	0.3 (3)
“Bath Salts”	6.0 (39)	6.0 (39)	0.0 (0)
Benzodiazepines	5.1 (77)	5.6 (78)	0.5 (1)
Synthetic cannabinoids	3.8 (25)	3.8 (25)	0.0 (0)
GHB	2.3 (47)	2.3 (47)	0.0 (0)
Methamphetamine	2.1 (38)	2.1 (39)	0.0 (1)
Tryptamines	2.1 (33)	2.1 (33)	0.0 (0)
2C Series	1.1 (25)	1.1 (25)	0.0 (0)
NBOMe	1.0 (9)	1.0 (9)	0.0 (0)
Other new psychedelics	0.0 (0)	0.0 (0)	0.0 (0)

Use of amphetamine, benzodiazepines, and opioids refers to nonmedical use. Adjusted prevalence includes type-in responses of drugs in which use was unreported earlier on the survey.

Table 3 presents correlates of discordant type-in responses. Compared to males, females were at more than twice the odds (aOR = 2.82, p = .022) of providing a discordant response. Compared to those with a high school education or less, those with a college degree or more were at lower odds of providing a discordant response (aOR = 0.13, p = .019), and those earning ≥$1000 USD a week were at 11 times higher odds of providing such a response (aOR = 11.03, p = .011). Finally, compared to heterosexual participants, those identifying as a sexual minority were at higher odds of providing a discordant response. Specifically, compared to heterosexuals, gay/lesbian participants were at over five times the odds (aOR = 5.20, p = .032) and those identifying as bisexual or other sexuality were at over fifteen times the odds (aOR = 15.12, p < .001) of providing a discordant response, with nearly a fifth (19.0%) of those identifying as bisexual or other sexuality providing a discordant response.

Table 3.

Correlates of providing a discordant response.

	No discordant response, Weighted %	Any discordant response, Weighted %	aOR	95% CI
Age
18–24	97.8	2.2	1.00
25–40	95.1	4.9	1.74	(0.44, 6.87)
Sex
Male	97.2	2.8	1.00
Female	95.2	4.8	2.82*	(1.16, 6.84)
Race/Ethnicity
White	96.2	3.8	1.00
Black	97.9	2.1	0.59	(0.07, 5.26)
Hispanic	96.6	3.4	0.38	(0.07, 1.97)
Asian	99.4	0.6	0.25	(0.03, 2.25)
Other/Mixed	90.2	9.8	3.73	(0.74, 18.86)
Education
High school diploma or less	95.8	4.2	1.00
Some College	95.4	4.7	0.33	(0.08, 1.44)
College degree or higher	97.1	2.9	0.13*	(0.02, 0.72)
Weekly income
<$500	98.9	1.1	1.00
$500–$999	96.1	3.9	4.97	(0.91, 27.22)
≥$1000	94.0	6.0	11.03*	(1.74, 69.85)
Sexual orientation
Heterosexual	98.4	1.6***	1.00
Gay/Lesbian	91.7	8.3	5.20*	(1.16, 23.41)
Bisexual or other sexuality	81.0	19.0	15.12***	(3.30, 69.41)
Number of drugs reportedly used in past year (Uncorrected)
0–1 drugs	97.3	2.7	1.00
2–4 drugs	95.6	4.4	1.91	(0.54, 6.71)
≥5 drugs	95.8	4.2	1.19	(0.23, 6.15)

The dependent variable indicates whether the participant provided a discordant response by typing in the name of a drug used that was not previously reported as used on the same survey (3.6%).

aOR: adjusted odds ratio; CI: confidence interval.

^*p < .05,

^**p < .01,

^***p < .001.

Discussion

It is known that inconsistent reporting is a common phenomenon in survey-based drug use studies that diminishes the reliability of the data and, by extension, accuracy of estimates. Nevertheless, electronic surveys have numerous feasibility advantages over other methods of assessing drug use and, consequently, will likely remain essential for epidemiologic research in drug use. Moreover, electronic surveys enable the use of skip-logic, which effectively limits the chances for respondents to be routed to questions where they may give answers inconsistent with their responses to previous questions. However, sometimes erroneous non-affirmative reports can lead to loss of information on surveys that utilize skip-logic as follow-up questions are not asked. In this study, we examined the extent to which type-in drug use responses on a survey contradict unreported use on a survey of a high-risk population—electronic dance music (EDM) party attendees.

Our findings suggest that approximately 3.6% of respondents provided discordant responses in which they typed in the names of drugs used (typically within the past 24 h) after having not reported past-year use of the same drug(s) earlier on the survey. This is lower than the incidence of inconsistent responses typically reported in the literature on survey-based studies, which range from 7% to 87%, though it should be noted that the majority of existing studies focus primarily on discord as to pertains to recanting use that was reported months or years earlier (Percy et al. 2005; Harris et al. 2008; Swanson et al. 2014; Palamar, Acosta, Calderon, et al. 2017; Palamar and Le 2017; Taylor et al. 2017; Jensen et al. 2018).

Although we did not find marked differences in prevalence of past-year use of most drugs after adjusting for discordant responses, use of powder cocaine and amphetamine did increase by 1% and 0.8%, respectively. Several studies have also reported relatively pronounced rates of inconsistent self-reporting for cocaine use (Fendrich and Mackesy-Amiti 2000; Percy et al. 2005; Harris et al. 2008; Ledgerwood et al. 2008), including a study examining conflicting responses on the NHSDA, which found that prevalence increased when participants were provided multiple chances to report past-year cocaine use (Lessler et al. 2000). Other studies have found that over one-fifth of amphetamine use is also discordantly reported (Harris et al. 2008). In particular, one study investigating use of nonmedical amphetamine among a nationally representative sample of high school seniors found that over a quarter of students reporting Adderall use also reported no amphetamine use in the same survey (Palamar and Le 2017). We believe that discordant responses involving nonmedical use of prescription drugs may be, in part, due to participant confusion. For example, some participants likely do not read that “nonmedical use” or “misuse” is being queried (as opposed to medical use), while others may be unaware of individual drugs that fall under these drug classes, even when examples of drugs under each drug class are provided for participants (Palamar 2018).

Our study findings further suggest that some groups are more likely to provide discordant responses than others. For example, compared to males, we found that females were at over 180% higher odds of typing in the name of a drug recently used that they did not report using earlier on the same survey. A previous study examining discordant survey responses among high school seniors in the US also found that females were at higher odds than males for reporting nonmedical Vicodin use after reporting no overall nonmedical opioid use (Palamar et al. 2016), and another study found that females were more likely than males to test positive for drug use after reporting they did not use (Fendrich et al. 2004). While females are less likely to report drug use than males in national samples (Johnston et al. 2019), we believe differences may be, in part, due to underreporting. However, we believe more research is needed to investigate potential mechanisms for this phenomenon.

Compared to those with a high school education or less, those with a college degree or higher in this study were at lower odds of typing in the name of a drug used after previously not reporting use on the survey. This corroborates recent findings showing that those with less education were more likely to provide discordant survey responses for marijuana, cocaine, ecstasy, amphetamine, and other ‘hard’ drugs (Jensen et al. 2018). Similarly, Percy et al. (2005) found that the odds of recanting previously reported use declined with increasing educational attainment, though their analysis was limited to use of marijuana and alcohol. Studies have shown that individuals with lower levels of education are more likely than more educated individuals to satisfice on surveys (Hamby and Taylor 2016), which is defined as respondents investing minimal attention towards survey questions and/or taking shortcuts to complete the survey sooner (Tourangeau 2000).

We also determined that those who reported earning more than $1000 USD per week were more likely to provide an inconsistent response. We expected that individuals with higher income would be less likely to provide an inconsistent response, but it may be that some individuals with higher income were less engaged (e.g. as the utility of $10 USD compensation may be less significant for respondents with higher income). In terms of sexual orientation, sexual minorities (i.e. gay/lesbian and bisexual) were significantly more likely to provide discordant responses than heterosexuals. To our knowledge, although previous studies have not yet specifically examined discordant survey responses as it pertains to sexual orientation, studies on self-reports versus biological testing for drug use have found that men who have sex with men (MSM) tend to provide more reliable responses about use of “harder” drugs than their heterosexual counterparts (Fendrich et al. 2008). Additionally, while individuals of sexual minority status—particularly those identifying as bisexual—have higher prevalence of drug use than heterosexual individuals (Medley et al. 2016; Duncan et al. 2019), bisexual individuals and those identifying as other sexuality were at over fifteen times the odds of providing a discordant response in this study, even after controlling for number of drugs used.

Overall, it appears that discordant responding within the same survey may be more of a problem for certain drugs such as cocaine and amphetamine and less of an issue for some other drugs. Assuming respondents did not overreport drug use via type-in responses, results suggest they incorrectly did not report use earlier in the survey when asked about use. It is unknown whether such false negatives are attributable to lack of attention or to satisficing, or perhaps even to mischievousness. In any case, results of this study lead us to question the extent to which brief closed-ended survey questions facilitate the attainment of reliable responses. While relying solely on type-in responses (without providing any closed-ended responses) leads to severe under-reporting of drug use (Kroutil et al. 2010; Palamar, Martins, et al. 2015; Palamar and Le 2019), it does appear that the nature of open-ended type-in responses may inherently prompt respondents to consider more thoroughly the question at hand, perhaps at times leading to more accurate responses. Or, asking more specifically about situational use may aid in recall more so than general questions about any use in the past year. However, asking about situational use may not be feasible (e.g. on short surveys). Ultimately, we believe that including one or more type-in responses in addition to closed-ended questions may help determine instances of underreporting.

When considering the existing literature, discordant responses—in particular, recanting of previously reported responses—appear more likely to occur between a baseline and follow-up survey after a non-negligible amount of time has passed. More research is needed to determine not only who is more likely to provide inaccurate or conflicting data, but also optimal means of handling conflicting responses. Contradictory responses are commonly deleted from national surveys (Brener et al. 2004; Miech et al. 2018) while other researchers delete and impute contradictory data (Center for Behavioral Health Statistics and Quality 2018). Deleting data may bias results as much as inaccurate data, so additional methods for handling conflicting responses would indeed be beneficial. Biological testing can help correct inaccurate survey responses, and perhaps post-hoc interviews can address conflicting responses and determine which response is in fact correct (Johnson and Bowman 2003). In this study, we added type-in responses to previous closed-ended responses as we do not believe type-in responses were instances of over-reporting. Although prevalence of drug use did not substantially change when considering discordant responses, such responses might lead to significant changes in other surveys or in other populations.

We do not believe instances of initial underreporting in this study are due to social desirability concerns as participants later typed in the names of drugs they had recently used. We believe discrepancies are more so due to differences in attention directed toward survey questions. More research is needed to further investigate reasons for and differences resulting from discordant reporting on drug surveys. Likewise, we determined various demographic risk factors for providing discordant responses and we believe future research is needed to determine the mechanisms for these characteristics increasing risk.

Limitations

We believe the main limitation of this study is that type-in responses were limited to drug use in the past day and to drugs associated with adverse effects in the past year. Such type-in responses limited to very recent use and to involvement in adverse effects did not likely allow us to detect less recent past-year use that was underreported earlier. In addition, type-in response options within themselves are commonly associated with underreporting (Kroutil et al. 2010; Palamar, Martins, et al. 2015; Palamar and Le 2019) so it is unlikely that these response boxes picked up all instances of earlier underreported used. Past-day use was not defined for participants so some might have interpreted this as use on the day of the survey and others might have interpreted this as use in the past 24 h. Limited recall of past-day drug use and of drugs used before adverse effects in the past year is also a limitation. It is also possible that some participants used a particular drug for the first time the day of the survey but did not consider that drug when answering about past-year use. Likewise, although the survey only focused on illegal/nonmedical drug use, it is possible some participants typed in names of prescription drugs only used for medical purposes. Although we did not include participants who demonstrated inebriation, it is possible that some participants were in fact inebriated and did not demonstrate visible signs. This investigation only focused on illegal drugs so we did not focus on alcohol use. Results may not be generalizable beyond the New York City EDM scene.

Conclusion

Overall, our findings suggest that prevalence estimates were not significantly affected by inconsistent responses, but we do believe drug use in general was underreported. Electronic surveys that query drug use can benefit from follow-up (e.g. open-ended) questions not dependent on previous responses, as they may elicit affirmative responses intentionally or unintentionally underreported earlier in the survey. However, type-in responses will by no means allow researchers to detect all underreporting. Electronic survey methods that better allow researchers to detect underreporting and/or discordant responses could greatly improve data in future studies.