Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Sep 1.
Published in final edited form as: Drug Alcohol Depend. 2017 Jun 20;178:194–200. doi: 10.1016/j.drugalcdep.2017.05.012

The effects of using answer sheets on reported drug use and data quality in a classroom survey: A cluster-randomized study*

Alvaro Castillo-Carniglia a, Esteban Pizarro b, José D Marín b, Nicolás Rodríguez b, Carolina Casas-Cordero c, Magdalena Cerdá a
PMCID: PMC5548613  NIHMSID: NIHMS886532  PMID: 28654872

Abstract

Background

We compare self-reported prevalence of drug use and indicators of data quality from two different response modes (with and without an independent answer sheet for recording responses) in a survey conducted in 2015 among secondary school students.

Methods

Stratified cluster-randomized study conducted among students in grades 8 to 12 from public, private and subsidized schools in Chile (N = 2,317 students in 122 classes). Measurements included were: percentage reporting substance use (tobacco, alcohol, marijuana, cocaine, ecstasy); number of inconsistent responses; number of item nonresponses; percentage of extreme reports of drug use; percentage reporting using the nonexistent drug, relevón; and completion times.

Results

Compared with those who responded directly in the questionnaire booklet, students who used a separate answer sheet took 17.6 more minutes (95% confidence interval [CI]: 14.4–20.8) to complete the survey and had on average 1.5 more inconsistent responses (95%CI: 0.91–2.14). The prevalence and variance of drug use was higher among those who used an answer sheet for all substances except tobacco; the prevalence ratio (PR) of reported substance use for low-prevalence substances during the past year were: cocaine PR=2.5 (95%CI: 1.6–4.1); ecstasy PR=5.0 (95%CI: 2.4–10.5); relevón PR=4.8 (95%CI: 2.5–9.3).

Conclusions

Using an answer sheet for a self-administered paper-and-pencil survey of drug use among students result in lower quality data and higher reports of drug use. International comparison of adolescent drug use from school-based surveys should be done with caution. The relative ranking of a country could be misleading if different mode of recording answers are used.

Keywords: survey methodology, self-administered, answer sheet, data quality, substance use, adolescents

1. Introduction

Early initiation of substance use is associated with long-term health risks, including an increased likelihood of substance-use disorders in the future and related psychiatric disorders (Cho et al., 2007; McGue and Iacono, 2005). Monitoring the patterns of substance use among school-aged adolescents is important for understanding the magnitude and trend of the problem of early substance use and for identifying targets for evidence-based drug-prevention programs. National surveys of substance use among secondary school students are used to produce estimates that serve these purposes. Producing estimates that are accurate, precise and comparable over time is crucial to monitoring current use and time trends.

According to the 2015 Report of Drug Use in the Americas, Chile has the highest prevalence of use of cocaine, cocaine paste, marijuana and tobacco among school-age children (Observatorio Interamericano de Drogas, 2015). With the exception of marijuana, which has increased rapidly in the past years, Chile has had a consistently higher prevalence of drug use among secondary students (survey with self-administered paper-and-pencil questionnaire), but not necessarily in the general population (survey with face-to-face interview). These results raise concerns about the validity, precision and comparability of the Chilean measurements, in particular regarding the school population.

1.1 School Surveys On Substance Use

In school settings, self-administered surveys targeting students have traditionally been implemented in the classroom, using paper-and-pencil, and some times using machine-readable answer sheets (Centers for Disease Control and Prevention, 2016; United Nations Office on Drugs and Crime, 2003, 2015). Compared to the more typical questionnaire booklets (that can include standardized or non-standardized marks, depending on the survey), the answer-sheet strategy is considered cheaper (e.g., less paper to print) and faster (e.g., machine captured instead of human keying). Additionally, students are familiar with this format that is commonly used for standardize testing.

Surveys using booklets include the national surveys of substance use among secondary school students conducted in Uruguay, Argentina, Spain, Ontario province in Canada, Europe, and the United States (Boak et al., 2013; Hibell et al., 2012; Miech et al., 2015; Observatorio Argentino de Drogas, 2014; Observatorio Uruguayo de Drogas, 2014; Plan Nacional de Drogas, 2012). Surveys using a separate answer sheet to record responses include those conducted in Chile, some state or city versions of the U.S. Youth Risk Behavior Survey, the Global School-based Student Health Survey, and the Global Youth Tobacco Survey (Centers for Disease et al., 2013; Centers for Disease Control and Prevention, 2016; Global Youth Tabacco Survey Collaborative, 2002; Observatorio Chileno de Drogas, 2014; World Health Organization, 2016).

Research on the quality and comparability of substance-use survey data across these two modes remains scarce. Studies of mode effects have compared reported drug use from surveys completed on paper versus computer (Beck et al., 2014; Hallforsa et al., 2000), by mail versus on the web (Callas et al., 2010; McCabe, 2004; McCabe et al., 2002), and by phone versus other modes (Link and Mokdad, 2005; Marcano Belisario et al., 2015). Other outcomes, such as nonresponse (Kongsved et al., 2007; Rolstad et al., 2011) and completion times (Rolstad et al., 2011), have also been compared across modes. To the best of our knowledge no previous studies have explicitly evaluated the potential impact of using a separate answer sheet as a mode of data collection on a survey’s results or data quality.

1.2 Survey Response Process Model

The survey methods literature provides a useful framework to think about mechanisms that could help explain how the use of an answer sheet to record survey responses could affect the quality of self-reports of drug use (and likely other sensitive topics) in the school setting. The survey response process model (Cannell et al., 1981; Strack and Martin, 1987; Tourangeau, 1984; Tourangeau et al., 2000) stipulates that, after hearing a survey question, respondents have to “understand” the question, “retrieve” the relevant information asked for, make a “judgment” as to what answer to provide, and finally “map” the response to the required format. Respondents go through this process with more or less involvement and not necessarily in this order. Survey questions and modes of data collection could also pose stronger demands on different parts of the process.

Figure 1 illustrates a model of the cognitive mechanisms that help explain the effects of using a separate answer sheet to record responses to sensitive behaviors in a school-based setting. We hypothesized that the use of an answer sheet influences two of the four processes in the survey response model – judgment and mapping.

Figure 1.

Figure 1

Open in a new tab

Survey response process model

Regarding the judgment process, the use of an answer sheet to record responses - instead of a questionnaire booklet - should reduce the risks of disclosure by making it more difficult for a third party to see the questions and their corresponding answers. The reduced risk of disclosure may increase the sense of privacy, that should help increase the reports of sensitive behaviors such as drug use, and reduce item nonresponse to these questions.

Regarding the mapping process, using an answer sheet imposes a more difficult task on the respondent – which is first having to locate the place where to mark the responses, and then mark them carefully to comply with the specifications for optical scanning. Just on the operational side having to perform these tasks requires additional time, and thus we expect that respondents using an answer sheet would show longer completion times than respondents using a questionnaire booklet. Taking more time to respond could increase item nonresponse if respondents do not have the time to respond all the survey questions. Item nonresponse increases the variance of the survey estimates (by effectively reducing the sample size) and it could bias the survey reports if the missing data mechanism is not missing at random. As the survey takes longer, respondents probably get tired which could reduce their concentration and commitment to the survey task. A reduction in concentration could produce an increase in inconsistent responses, whereas a decrease in commitment could give rise to random responses (variance) or careless responses (such as reporting a fake drug, or extreme reports of use of drugs).

This study aimed to estimate the effects that using a direct versus answer sheet approach to record responses (currently under use in Chile) had on completion times, data quality (i.e., item nonresponse, inconsistent responses, extreme reports of drug use, reporting use of a fake drug), and the prevalence of reports of substance use. The study also aimed to examine whether differences in completion time mediated the effect of response modality on data quality and whether differences in completion time and data quality mediated the effect of response modality on reported prevalences of substance use. The study used data from a fully randomized experimental design conducted in a school-based setting following the same data collection protocols as those of the Chilean National Drug Surveys among Secondary Students (NDSSS).

2. Methods

2.1 Design And Procedures

This was a stratified, cluster-randomized study conducted in 2015 in which classrooms of students in the same grade comprised the clusters. To avoid potential contamination among subjects, we randomized classrooms instead of students, either to the treatment condition, corresponding to the alternative method under evaluation (i.e., marking nonstandardized responses in the questionnaire booklet), or to the control condition, corresponding to the current method used to record responses in the Nation Drug Survey among School Students (NDSSS) conducted every two years since 2001 (i.e., marking responses on a separate answer sheet). Figure S1 in the supplemental material1 shows an example of both methods used in this study.

All other aspects of data collection were exactly the same for both the treatment and control groups, following the procedures of the 10th version of the NDSSS (Observatorio Chileno de Drogas, 2014). The survey has a self-administered paper-and-pencil questionnaire without skips, which is implemented in the classroom by trained pollsters; teachers and other school authorities are not allowed in the classroom during the survey. When students finish responding, they deposit the answer sheet (control group) or questionnaire (treatment group) in a sealed cardboard box.

All ethical safeguards relevant to human participants were met. We requested written authorization from school principals and, when requested by school authorities, we sent a passive informed consent letter to the parents of children in the selected classes. Pollsters informed students about the study’s objectives and explained that the survey was completely anonymous and voluntary and that the data would be handled under strict confidentiality protocols in accordance with national legislation (Ministerio Secretaría General de la Presidencia, 1999).

2.2 Sample

Fifteen sampling strata were created by combining the three types of schools (public, private and subsidized schools) and the 5 grades comprising the target population (8th through 12th grades), for the 3 largest regions of the country (the Metropolitan, Valparaiso and Biobío regions). Schools with only a single class per grade were excluded prior to sampling.

Two schools per strata were randomly selected using Sampford’s method (Grafstrom, 2009). If the selected school had more than two classes in a given grade, then only two were selected to take part in the study. This resulted in the selection of 180 classes in total (60 per region) within 90 schools (30 per region). The two classes per school were randomly assigned to each of the study arms with a probability of 1/2.

2.3 Variables

To check the balance of the random assignment to experimental conditions, we considered the school type and the composition of the student body by treatment condition, including gender, age, the proportion of students who had ever repeated a grade, marital status of parents, religion and native American background. In Chile, school type is considered a proxy for a student’s socioeconomic status (SES), in which students from private schools have the highest SES; students from public schools have the lowest SES; and students attending subsidized schools are in between.

For the primary analysis of mode differences, we compared student-level estimates of the following variables.

  • Substance use outcomes: We estimated the prevalence of use during the past year and past month of two licit (tobacco, alcohol) and three illicit drugs (marijuana, cocaine [smoked, snorted, or both], ecstasy).

  • Completion times: These were measured in minutes and were calculated based on the difference between the starting time of the survey reported by the pollster (the same for all students in a class) and the finishing time reported by each student in the last question in the survey.

  • Inconsistent responses: The number of inconsistent responses was estimated based on validation programs used to review and clean the raw dataset; for this version of the questionnaire we identified 174 potential logical inconsistencies (e.g., mark drugs use in the past year, but also have never used drugs in lifetime).

  • Item nonresponse: This was calculated as the total number of responses missing (“blanks”) from the questionnaire.

  • Extreme reports of drug use: This was defined as the percentage of persons responding positively to having used all six substances during the past month (crack, ecstasy, stimulants, heroin, methamphetamine and modafinil).

  • Reporting using the fake drug relevón in lifetime and past year.

2.4 Analysis

We used Fisher’s exact test to check the balance between the treatment and control groups in terms of sociodemographic and other potential confounders (Table 1).

Table 1.

Sociodemographic characteristic of the sample, by answer register method. n(%)

Total (n=2,317) On Booklet (n=1,156) On Answer Sheet (n=1,161) P value1
School type
 Public 723 (31.2) 359 (31.1) 364 (31.4) 0.947
 Private with State subsidy 820 (35.4) 413(35.7) 407 (35.1)
 Private 774 (33.4) 384 (33.2) 390 (33.6)
Age
 11 to 14 years 696 (30.8) 336 (29.9) 360 (31.7) 0.625
 15 to 16 years 1019 (45.1) 515 (45.9) 503 (44.3)
 17 to 20 years 546 (24.2) 273 (24.3) 273 (24.0)
Gender
 Men 1,170 (51.2) 599 (49.0) 611 (53.4) 0.036
 Women 1,117 (48.8) 583 (51.1) 534 (46.6)
Ever repeating a grade
 No 1,787 (77.6) 909 (78.8) 878 (76.4) 0.177
 Yes 515 (22.4) 244 (21.2) 271 (23.6)
Marital status of parents
 Married/couple 1,420 (62.3) 714 (62.2) 706 (62.4) 0.890
 Not married/couple 859 (37.7) 434 (37.8) 425 (37.6)
Religion
 Yes 1,519 (66.9) 765 (66.8) 754 (67.0) 0.929
 No/Don’t know 753 (33.1) 381 (33.3) 372 (33.0)
Native American background
 Yes 184 (8.1) 79 (6.9) 105 (9.3) 0.038
 No 2,091 (91.9) 1,069 (93.1) 1,022 (90.7)
Open in a new tab
1

Fisher’s exact test

We used Stata 14.1 (Stata Corporation, College Station, Texas) to estimate the prevalence of drug use, adjusting for all the variables described above that showed a significantly different distribution (P <0.05) among groups due to unintentional imbalance after randomization. To determine the effect of the response modality, we computed adjusted absolute and relative differences for all the outcomes and their respective 95% confidence intervals (95% CI) across response modalities. For this we used generalized linear models, specifying different distributions and link functions depending on the nature of the outcome and the effect measure to be reported. All models were computed with robust variance estimation to account for the clustering of students within classes (Williams, 2000)(27). For example, to estimate the absolute and relative differences of the prevalence of drug use, we ran models specifying a binomial distribution with, respectively, an identity link function or a log link function.

We also estimated whether completion time (T) mediated the relationship between the response modality (RM) and the number of inconsistencies (I) and reports of drug use (DU) (RM-T-I and RM-T-DU) and whether the number of inconsistencies mediated the relationship between the response modality and reports of drug use (RM-I-DU), controlling for the same imbalanced characteristics considered in the previous analysis and accounting for the cluster randomization. For this analysis we defined drug use as cocaine or ecstasy consumption during the past year, because of its sensitivity to the response mode, and because we assumed that the mediation effect (if any) would be similar for other substances. We used the mediation package in Stata (Hicks and Tingley, 2011) based on the work of Imai et al., (2010) on causal mediation analysis. This approach expand the method proposed by Baron and Kenny (1986) to a counterfactual framework, and uses parametric regression to model the mediator conditional on the treatment condition (in our case the response mode), and a separate model for the outcomes conditional on the treatment and the mediator (adjusting covariates imbalance).

3. Results

The sample included 2,317 students (1,161 used an answer sheet and 1,156 responded directly in the questionnaire booklet) in 122 classrooms (61 used each response modality) in 61 schools in 35 municipalities and 3 regions. The school cooperation rate was 67.7% (N schools agreed to take the survey/N schools selected). No student refused to answer the survey. We excluded 11 participants who answered less than 50% of the questionnaire (4 responded in the booklet and 7 used the answer sheet).

Table 1 presents the socio-demographic distribution of the complete sample by response modality. The balance of the groups was suitable for most variables except for gender and ethnic background. Owing to these results, the analyses of mode effects and mediation effects were adjusted for these two variables.

3.1 Effect On Completion Times, Item Nonresponse And Number Of Inconsistencies

Table 2 shows the average completion time and mean number of inconsistencies and item nonresponses for both response modalities, as well as the prevalence of extreme reports of substance use. Students who responded using the answer sheets took an average of 17.6 minutes longer (95% CI = 14.4–20.8) than the students who answered directly in the questionnaire, corresponding to an increase of 49.9% in completion time (95% CI = 39.6–60.9). The percentage extreme reports of drug use and the average number of inconsistencies were also higher among students who used the answer sheet, with an absolute (non-significant) difference of 0.4 percentage points (95% CI = −0.2–1.0) for extreme reporting and 1.5 inconsistencies (95% CI = 0.9–2.1). However, those who used the answer sheet had 1.1 fewer nonresponse items (95% CI = −1.9 to −0.4) than those who responded directly in the questionnaire.

Table 2.

Adjusted absolute and relative effect of the answer sheet on application time, and on over-reporting of drug use, inconsistencies and item nonresponse.

On Booklet On Answer Sheet
Mean1 (95%CI) Mean1 (95%CI) Diff.1 (IC95%) % of change1 (IC95%)
Completion Time (minutes)2 35.36 (33.66 – 37.07) 52.98 (50.25 – 55.70) 17.61 (14.40 – 20.83) 49.9 (39.6 – 60.9)
Extreme reports of drug use (%)3 0.36 (0.02 – 0.70) 0.80 (0.32 – 1.28) 0.44 (−0.15 – 1.03) 125.8 (−26.8 – 596.7)
Inconsistent reports (No.)4 2.00 (1.72 – 2.27) 3.52 (2.93 – 4.11) 1.52 (0.91 – 2.14) 93.0 (57.2 – 136.9)
Item nonresponse (No.)4 3.37 (2.77 – 3.97) 2.25 (1.77 – 2.74) −1.12 (−1.87 – −0.37) −31.3 (−48.0 – −9.3)
Open in a new tab

Diff: Absolute difference between Answer Sheet and Booklet.

1

Adjusted by gender and ethnic background;

2

Estimated with a generalized linear models specifying a normal distribution and Identity/Log link function

3

Estimated with a generalized linear models specifying a binomial distribution and Identity/Log link function.

4

Estimated with a generalized linear models specifying a negative binomial distribution and Identity/Log link function.

3.2 Effect On Reports Of Drug Use

Regarding drug use, absolute differences ranged from 3.2 to 4.2 percentage points depending on the substance, with the exception of tobacco, for which the difference was 0.3 percentage points. Table 3 shows the adjusted absolute (prevalence difference) and relative effects (prevalence ratio) of using answer sheets on reported drug use adjusted by gender and ethnic background. There was no significant difference in the reported use of the most prevalent substances (tobacco, alcohol and marijuana), but for less prevalent drugs, there was an important relative difference across response modalities. Among those who used an answer sheet, reported cocaine use during the past year and past month were, respectively, 2.5 times higher (95% CI = 1.6–4.1) and 3.4 times higher (95% CI = 1.6–7.0) than among those who responded directly in the questionnaire; for ecstasy, among those who used an answer sheet, past-year and past-month reports were, respectively, 5.0 times higher (95% CI = 2.4–10.5) and 3.4 times higher (95% CI = 1.4–8.2). Finally, among those who used an answer sheet, past-year and past-month reports of using the fictional substance relevón were, respectively, 4.8 times higher (95% CI = 2.5–9.3) and 4.9 times higher (95% CI = 2.2–11.0).

Table 3.

Adjusted absolute and relative effect of the answer sheet on drug use reports

On Booklet On Answer Sheet
Adj. Prev. % (95%CI) Coeff. of variation Adj. Prev. % (95%CI) Coeff. of variation Prev. Diff. PP (IC95%)1 PR (IC95%)2
Tobacco
 Past year 35.39 (31.14 – 39.64) 5.24 35.71 (31.12 – 40.3) 5.32 0.32 (−5.95 – 6.59) 1.02 (0.86 – 1.21)
 Past month 22.74 (19.26 – 26.22) 3.09 20.58 (16.71 – 24.44) 2.71 −2.17 (−7.36 – 3.03) 0.93 (0.73 – 1.19)
Alcohol
 Past year 59.07 (53.84 – 64.30) 5.82 63.11 (57.97 – 68.24) 5.47 4.04 (−3.31 – 11.38) 1.07 (0.95 – 1.20)
 Past month 34.21 (28.91 – 39.51) 5.09 36.31 (31.08 – 41.55) 5.33 2.1 (−5.35 – 9.55) 1.06 (0.86 – 1.31)
Marijuana
 Past year 28.77 (24.53 – 33.01) 4.25 32.57 (27.77 – 37.38) 4.87 3.80 (−2.61 – 10.21) 1.13 (0.92 – 1.39)
 Past month 16.54 (13.42 – 19.65) 1.93 20.14 (16.71 – 23.57) 2.63 3.60 (−1.02 – 8.23) 1.21 (0.94 – 1.56)
Cocaine
 Past year 3.03 (2.01 – 4.05) 0.07 6.31 (4.44 – 8.17) 0.40 3.28 (1.22 – 5.34) 2.52 (1.56 – 4.05)
 Past month 1.17 (0.60 – 1.74) 0.01 3.31 (1.99 – 4.63) 0.12 2.14 (0.77 – 3.51) 3.37 (1.63 – 6.99)
Ecstasy
 Past year 1.11 (0.48 – 1.75) 0.01 4.31 (2.65 – 5.98) 0.19 3.20 (1.41 – 4.99) 5.01 (2.40 – 10.45)
 Past month 0.75 (0.24 – 1.25) 0.00 2.05 (1.28 – 2.82) 0.05 1.30 (0.43 – 2.17) 3.36 (1.38 – 8.21)
Relevón
 Lifetime 1.67 (0.82 – 2.51) 0.02 5.88 (4.07 – 7.70) 0.33 4.22 (2.43 – 6.01) 4.78 (2.47 – 9.25)
 Past year 0.96 (0.36 – 1.55) 0.01 3.85 (2.66 – 5.03) 0.14 2.89 (1.63 – 4.15) 4.89 (2.18 – 11.00)
Open in a new tab

Adj. Prev.: Prevalence adjusted by gender and ethnic background; Coeff. of variation: Coefficient of variation ((Standard deviation/Mean) × 100). Prev. Diff: Prevalence difference adjusted by gender and ethnic background; PP: Percentage Points; PR: Prevalence Ratio adjusted by gender and ethnic background.

1

Estimated with a generalized linear models specifying a binomial distribution and Identity link function.

2

Estimated with a generalized linear models specifying a binomial distribution and Log link function.

The variance of prevalence estimates (measured through their coefficient of variation) was higher among those who used the answer sheet for all illicit drugs. For drugs with low prevalence and for relevon the relative difference between the coefficients of variation was larger.

3.3 Results Of The Mediation Analysis

Table 4 presents marginal estimates and 95% confidence intervals of controlled direct effects, average causal mediation effect (ACME), the total effect and the proportion mediated. We did not find evidence of a mediation effect for completion time on inconsistent reports (ACME: −0.06; 95% CI: −0.65 – 0.53) or reported drug use (ACME: 0.0: 95% CI: −0.02 – 0.01). The number of inconsistencies, however, had a significant indirect effect on the relationship between the response modality and reported drug use, with a proportion mediated of 36% (95% CI: 21% – 93%).

Table 4.

Mediation analysis for the effect of the response mode on drug use and inconsistent responses

Marginal effects 95% Confidence Interval
Controlled direct effect
 RM-T-I 1.97 (1.15 – 2.72)
 RM-T-DU 0.06 (0.02 – 0.09)
 RM-I-DU 0.02 (0.00 – 0.05)
Average causal mediated effect
 RM-T-I −0.06 (−0.65 – 0.53)
 RM-T-DU 0.00 (−0.02 – 0.01)
 RM-I-DU 0.01 (0.01 – 0.02)
Total effect
 RM-T-I 1.91 (1.26 – 2.59)
 RM-T-DU 0.05 (0.02 – 0.08)
 RM-I-DU 0.04 (0.01 – 0.06)
Proportion mediated
 RM-T-I −0.03 (−0.04 – −0.02)
 RM-T-DU −0.07 (−0.15 – −0.04)
 RM-I-DU 0.36 (0.21 – 0.93)
Open in a new tab

RM: Response Mode; T: Completion Time; I: Inconsistencies; DU: Drug Use (cocaine or ecstasy)

4. Discussion

In this study we compared two response registration modes for a paper-and-pencil self-administered national drug survey of secondary school students (one with a separate answer sheet and the other in which students marked their preferences directly in the questionnaire). We found that using an answer sheet significantly increased the time taken to respond to the survey, the number of inconsistencies and reports of using the fake drug that was included in the questionnaire. This indicates that the quality of responses to school-based surveys of adolescent substance use could be improved by having students respond directly in the questionnaire rather than on a separate answer sheet. Reports of substance use were also significantly higher and with larger variance among students who used an answer sheet, particularly for substances with a lower prevalence of consumption, such as cocaine and ecstasy. These elevated prevalence estimates were directly attributable to the use of the answer sheet since there was no reason, given the randomization used in the study, why the two groups would have different prevalence levels of substance use. To the best of our knowledge, this is one of the first studies to examine the impact of the response registration mode, specifically the use of an answer sheet, on data quality and prevalence of self-reported substance use in surveys.

In the Americas, Chile has one of the highest prevalence for the use of cocaine and cocaine paste (Observatorio Interamericano de Drogas, 2015). This study suggests that part of the difference in prevalence between Chile and other countries may be due to the use of separate answer sheets to report responses rather than to a real difference in cocaine use. We found that the prevalence of use among those who marked their answers directly in the questionnaire booklet (including answers about their use of relevón) were consistently closer to those observed in countries with similar demographic and epidemiological profiles. For example, the prevalence of past-year snorted cocaine use in Argentinean in 2011 and Uruguayan in 2014 in the school drug surveys were, respectively, 2.7% and 2.1% (Observatorio Interamericano de Drogas, 2015), which are comparable to the 2.4% found in Chile (which considered only snorted cocaine) using the direct questionnaire response modality (4.8% among those who responded using an answer sheet). Also, reports of lifetime use of the dummy drug relevón in all European countries included in the European School Survey Project on Alcohol and Other Drugs (ESPAD) was 1.1%, a percentage relatively close to the one observed in our study among students who responded directly in the questionnaire (Hibell et al., 2012).

The effects of the mode of data collection on indicators of data quality varied in sizes and directions. For example, the use of an answer sheet was associated with a lower number of item nonresponses but also with a high number of inconsistent responses, larger variance, and with a higher proportion of both extreme reports of drug use and using a fake drug. Results of data cleaning processes are not always fully described in national surveys, so the impact of correcting such problems is often unknown. The data cleaning process used by ESPAD suggests that removing surveys with a large proportion of item nonresponses (>50%) or with nonresponses on items about sex and age has a minimal impact on prevalence estimates, but that removing surveys with repetitive response patterns (similar to our extreme report measure) has a small but observable impact (Hibell et al., 2012). Chile and all Latin American countries code survey responses that declare having used a particular drug during the past year or past month as a lifetime prevalence, and they code all cases that declare having used a particular drug during the past month as a past-year prevalence (Observatorio Interamericano de Drogas, 2011). The impact of this data editing procedure will be directly related to the number of inconsistencies in those items and, as we found in our study, larger number of inconsistencies are associated with the use of an answer sheet.

Although in our survey response process model we initially hypothesized that the answer sheet could increase the sense of privacy and therefore improve the validity of self-reports of substance use, we observed that practically all indicators of data quality, including the use of a fake drug, performed worse when using that response mode. This suggests that the higher prevalence observed when using the answer sheet was mainly due to biased responses induced by this response mode, rather than more honest answers.

We also hypothesized that a key driver of the observed differences might be the shorter duration of the direct questionnaire response modality. A survey with a shorter duration may help students have a better attitude toward, and improved concentration during, the course of a survey, resulting in more consistent and honest answers. Herzog and Bachman (1981) found that a longer questionnaire was associated with an increased likelihood of respondents giving identical answers to most items, or a straight-line response pattern, among high school students responding to a drug survey. Galesic and Bosnjak (2009) also showed that the length of the questionnaire was also associated with a faster, shorted, and more uniform response to items later in the questionnaire. However, and contrary to our hypotheses, we did not find evidence in our mediation analysis to support the hypothesis that completion time is a driver for the observed effects of response modality on data quality or reported drug use. In contrast, approximately one-third of the effect of the answer sheet on reported drug use was mediated by the number of inconsistent responses. These suggest that not all the indicators of data quality are equally associated with substance use reports (e.g., item nonresponse vs. inconsistencies responses or extreme reports), and thus they are not equally important when evaluating the methodological validity of a survey.

4.1 Implications

Major studies of substance use among school-aged children use a separate answer sheet, including the Global School-based Student Health Survey, the Global Youth Tobacco Survey and some state or city versions of the US Youth Risk Behavioral Survey (Centers for Disease et al., 2013; Centers for Disease Control and Prevention, 2016; Global Youth Tabacco Survey Collaborative, 2002; World Health Organization, 2016). Our findings suggest that using answer sheets has a significant impact on data quality and reported drug use. This response modality may produce at least two major consequences: (1) biased estimates of the magnitude of the research problem, which may lead to incorrect conclusions and decisions (e.g., the incorrect assignment of resources, biased comparisons between similar surveys within one country or across countries); (2) biased estimates of the association between drug use and some exposures, thus acting as an unmeasured confounder or effect modifier. According to our results, the effect of the answer sheet was not the same for all substances, so its use could also differ across strata and the subgroups of the population being studied.

5. Conclusions

The use of an answer sheet in surveys such as the one conducted in Chile directly affects national statistics of drug use among students, statistics that are used in different international reports to characterize the profile of drug use by country, Chile is usually one of the countries with the highest prevalences of the use of cocaine and cocaine paste, as well as other substances with low prevalences (Observatorio Interamericano de Drogas, 2015). We showed that the relative ranking in terms of adolescent substance use of Latin American countries and possibly other geographic areas (e.g., states within a country) could be partially explained by methodological differences in the substance use surveys. In this case, prevalence differences in cocaine and ecstasy substances were explained mostly by the response mode used in the substance survey of secondary school students. The potential effect of using answer sheets for studies that also consider other risk factors, such as the US Youth Risk Behavior Survey, the Global School-based Student Health Survey or the Global Youth Tobacco Survey, should also be considered. Similarly, any change in survey procedures, such as changing the response modality, should be carefully evaluated.

Supplementary Material

supplement
NIHMS886532-supplement.docx (268.7KB, docx)

Highlights.

  • Response mode in self-administered surveys can significantly affect data quality.

  • Using a separate answer sheets decrease data quality among adolescents.

  • Low data quality can distort substance use estimates and trends.

Acknowledgments

Role of Funding Source

This study was supported by the National Service for Prevention and Rehabilitation of Drug and Alcohol Consumption of Chile (grant number 662237-6-LP15) and a grant from the National Institute on Drug Abuse (R01DA040924-01). ACC was supported by Becas Chile as part of the National Commission for Scientific and Technological Research (CONICYT) and the Robertson Fellowship in Violence Prevention Research.

This study was supported by the National Service for Prevention and Rehabilitation of Drug and Alcohol Consumption of Chile (grant number 662237-6-LP15) and a grant from the National Institute on Drug Abuse (R01DA040924-01). ACC was supported by Becas Chile as part of the National Commission for Scientific and Technological Research (CONICYT) and the Robertson Fellowship in Violence Prevention Research. The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the funding institutions.

Footnotes

*

Supplementary material can be found by accessing the online version of this paper at http://dx.doi.org and by entering doi:...

1

Supplementary material can be found by accessing the online version of this paper at http://dx.doi.org and by entering doi:...

Contributors
  • Alvaro Castillo-Carniglia was involved in the conceptualization of the study, study design, and data acquisition. Conducted the data analyses, and wrote the initial draft of the manuscript. He read and approved the final version of the article.
  • Esteban Pizarro was involved in the conceptualization of the study, study design, and data acquisition. He read and approved the final version of the article.
  • José D. Marín was involved in the conceptualization of the study, study design, and data acquisition. He read and approved the final version of the article.
  • Nicolás Rodriguez was involved in the conceptualization of the study, study design, and data acquisition. He read and approved the final version of the article.
  • Carolina Casas-Cordero was involved in the interpretation of the results and drafting the manuscript. She read and approved the final version of the article.
  • Magdalena Cerdá was involved in the interpretation of the results and drafting the manuscript. She read and approved the final version of the article.

Conflict of Interest

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Baron RM, Kenny DA. The moderator mediator variable distinction in social psychological-research - Conceptual, strategic, and statistical considerations. J Pers Soc Psychol. 1986;51:1173–1182. doi: 10.1037//0022-3514.51.6.1173. [DOI] [PubMed] [Google Scholar]
  2. Beck F, Guignard R, Legleye S. Does computer survey technology improve reports on alcohol and illicit drug use in the general population? A comparison between two surveys with different data collection modes in France. PLoS One. 2014;9:e85810. doi: 10.1371/journal.pone.0085810. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Boak A, Hamilton HA, Adlaf EM, Mann RE. Drug uso among Ontario students, 1977–2013: detailed OSDUHS findings. Centre for Addiction and Mental Health; Toronto, ON: 2013. [Google Scholar]
  4. Callas PW, Solomon LJ, Hughes JR, Livingston AE. The influence of response mode on study results: Offering cigarette smokers a choice of postal or online completion of a survey. J Med Internet Res. 2010;12:e46. doi: 10.2196/jmir.1414. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Cannell CF, Miller PV, Oksenberg L. Research on interviewing techniques. In: Leinhardt S, editor. Sociological Methodology. Jossey-Bass; San Francisco: 1981. pp. 389–437. [Google Scholar]
  6. Centers for Disease, C., Prevention; Brener ND, Kann L, Shanklin S, Kinchen S, Eaton DK, Hawkins J, Flint KH Centers for Disease, C., Prevention. Methodology of the youth risk behavior surveillance system--2013. MMWR Recomm Rep. 2013;62:1–20. [PubMed] [Google Scholar]
  7. Centers for Disease Control and Prevention. [accessed on March 15th 2016];Global School-based Student Health Survey (GSHS) 2016 http://www.cdc.gov/GSHS/
  8. Cho H, Hallfors DD, Iritani BJ. Early initiation of substance use and subsequent risk factors related to suicide among urban high school students. Addict Behav. 2007;32:1628–1639. doi: 10.1016/j.addbeh.2006.11.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Galesic M, Bosnjak M. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin Quart. 2009;73:349–360. [Google Scholar]
  10. Global Youth Tabacco Survey Collaborative, G. Tobacco use among youth: A cross country comparison. Tob Control. 2002;11:252–270. doi: 10.1136/tc.11.3.252. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Grafstrom A. Non-rejective implementations of the Sampford sampling design. J Stat Plan Infer. 2009;139:2111–2114. [Google Scholar]
  12. Hallforsa D, Khatapoushb S, Kadushinb C, Watsonb K, Saxeb L. A comparison of paper vs computer-assisted self interview for school alcohol, tobacco, and other drug surveys. Eval Program Plann. 2000;23:149–155. [Google Scholar]
  13. Herzog AR, Bachman JG. Effects of questionnaire length on response quality. Public Opin Quart. 1981;45:549–559. [Google Scholar]
  14. Hibell B, Guttormsson U, Ahlström S, Balakireva O, Bjarnason T, Kokkevi A, Kraus L. The 2011 ESPAD Report: Substance Use Among Students in 36 European Countries. The Swedish Council for Information on Alcohol and Other Drugs; Stockholm: 2012. [Google Scholar]
  15. Hicks R, Tingley D. mediation: Stata package for causal mediation analysis. 2011 http://EconPapers.repec.org/RePEc:boc:bocode:s457294.
  16. Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychol Methods. 2010;15:309–334. doi: 10.1037/a0020761. [DOI] [PubMed] [Google Scholar]
  17. Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and completeness of questionnaires: A randomized study of Internet versus paper-and-pencil versions. J Med Internet Res. 2007;9:e25. doi: 10.2196/jmir.9.3.e25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Link MW, Mokdad AH. Effects of survey mode on self-reports of adult alcohol consumption: A comparison of mail, web and telephone approaches. J Stud Alcohol. 2005;66:239–245. doi: 10.15288/jsa.2005.66.239. [DOI] [PubMed] [Google Scholar]
  19. Marcano Belisario JS, Jamsek J, Huckvale K, O’Donoghue J, Morrison CP, Car J. Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev. 2015;7:MR000042. doi: 10.1002/14651858.MR000042.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. McCabe SE. Comparison of web and mail surveys in collecting illicit drug use data: A randomized experiment. J Drug Educ. 2004;34:61–72. doi: 10.2190/4HEY-VWXL-DVR3-HAKV. [DOI] [PubMed] [Google Scholar]
  21. McCabe SE, Boyd CJ, Couper MP, Crawford S, D’Arcy H. Mode effects for collecting alcohol and other drug use data: Web and U.S. mail. J Stud Alcohol. 2002;63:755–761. doi: 10.15288/jsa.2002.63.755. [DOI] [PubMed] [Google Scholar]
  22. McGue M, Iacono WG. The association of early adolescent problem behavior with adult psychopathology. Am J Psychiatry. 2005;162:1118–1124. doi: 10.1176/appi.ajp.162.6.1118. [DOI] [PubMed] [Google Scholar]
  23. Miech RA, Johnston LD, O’Malley PM, Bachman JG, Schulenberg JE. Monitoring the Future national survey results on drug use, 1975–2014: Volume I, Secondary school students. Institute for Social Research, The University of Michigan; Ann Arbor: 2015. p. 599. [Google Scholar]
  24. Ministerio Secretaría General de la Presidencia. Sobre potección de la vida privada, Ley No. 19.628. 1999 http://bcn.cl/1lyp5accessed on.
  25. Observatorio Argentino de Drogas. Patrones y magnitud del consumo: diágnostico a nivel país. Secretaría de Programación para la Prevención de la Drogadicción y la Lucha contra el Narcotráfico (SEDRONAR); Bueno Aires: 2014. Sexto estudio nacional sobre consumo de sustancias psicoactivas en estudiantes de enseñanza media. [Google Scholar]
  26. Observatorio Chileno de Drogas. Décimo Estudio Nacional de Drogas en Población Escolar de Chile. Servicio Nacional para la Prevención y rehabilitación del Consumo de Droga y Alcohol; Santiago: 2014. [Google Scholar]
  27. Observatorio Interamericano de Drogas. Protocolo de la encuesta de estudiantes de enseñanza media Sistema interamericano de datos uniformes. Comisión Interamericana para el Control del Abuso de Drogas, Organización de los Estados Americanos; Washington, D.C: 2011. p. 62. [Google Scholar]
  28. Observatorio Interamericano de Drogas. Informe del Uso de Drogas en las Américas. Comisión Interamericana para el Control del Abuso de Drogas, Organización de los Estados Americanos; Washington, D.C: 2015. p. 219. [Google Scholar]
  29. Observatorio Uruguayo de Drogas. Sexta encuesta nacional sobre consumo de drogas en estudiantes de enseñanza media. Junta Nacional de Drogas; Montevideo: 2014. [Google Scholar]
  30. Plan Nacional de Drogas. Encuesta sobre uso de drogas en Enseñanzas Secundarias en España. Ministerio de Sanidad, Servicios Sociales e Igualdad; Madrid: 2012. [Google Scholar]
  31. Rolstad S, Adler J, Ryden A. Response burden and questionnaire length: Is shorter better? A review and meta-analysis. Value Health. 2011;14:1101–1108. doi: 10.1016/j.jval.2011.06.003. [DOI] [PubMed] [Google Scholar]
  32. Strack F, Martin L. Thinking, judging, and communicating: A process account of context effects in attitude surveys. In: Hippler HJ, Schwarz N, Sudman S, editors. Social information processing and survey methodology. Springer Verlag; New York: 1987. pp. 123–148. [Google Scholar]
  33. Tourangeau R. Cognitive science and survey methods: A cognitive perspective. In: Jabine T, Straf M, Tanur J, Tourangeau R, editors. Cognitive aspects of survey methodology: Building a bridge between disciplines. National Academy Press; Washington, DC: 1984. pp. 73–100. [Google Scholar]
  34. Tourangeau R, Rips LJ, Rasinski K. The psychology of survey response. Cambridge University Press; Cambridge: 2000. [Google Scholar]
  35. United Nations Office on Drugs and Crime. Conducting School Surveys on Drug Abuse. United Nations Publications; Austria: 2003. [Google Scholar]
  36. United Nations Office on Drugs and Crime. World Drug Report. United Nations publication; Vienna: 2015. [Google Scholar]
  37. Williams RL. A note on robust variance estimation for cluster-correlated data. Biometrics. 2000;56:645–646. doi: 10.1111/j.0006-341x.2000.00645.x. [DOI] [PubMed] [Google Scholar]
  38. World Health Organization. [accessed on March 15th 2016];Global School-based Student Health Survey. 2016 http://www.who.int/chp/gshs/en/

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplement
NIHMS886532-supplement.docx (268.7KB, docx)

RESOURCES