Testing the Question-Behavior Effect of Self-Administered Surveys Measuring Youth Drug Use

Implications and Contribution

Concern about the possibility of a question-behavior effect has created some opposition to surveys in schools, but there is little research examining evidence for this effect. This study investigated the question-behavior effect on drug use one year after students were first surveyed and found no indication of an increase in drug use.

Introduction

Surveying adolescents about their past experiences with illicit substances is a common practice. National research studies that monitor adolescent drug use through self-report surveys include the University of Michigan’s Monitoring the Future (MTF) study, which started asking young people about their health-risking behaviors annually in 1975 [1, 2], and the Center for Disease and Control and Prevention’s Youth Risk Behavior Survey (YRBS), which has been surveying school-aged children about health-related topics, including drug use, since 1991 [3]. Surveys like the Communities That Care Youth Survey (CTCYS) allow state and local policymakers and community members to measure and monitor adolescent drug use and the risk and protective factors that predict drug use to inform prevention planning activities [4–6]. As policy increasingly relies on the use of data, surveying adolescents about their attitudes and behaviors in the school setting offers a convenient, low-cost method to capture this important information [7].

One concern with youth surveys is that asking young people about their past drug use could lead to changes in future behavior, including the uptake of problem behaviors like early substance use. This process has been studied under many names: the mere measurement effect [8]; the self-prophecy effect [9]; and, as we will call it throughout the rest of this study, the question-behavior effect [10]. The idea that asking about a specific behavior could elicit that behavior is often used as a reason communities should not survey students about drug use. The Centers for Disease Control and Prevention, in referring to their YRBS survey, states, “there is no evidence that simply asking students about health risk behaviors will encourage them to try that behavior” [11]. However, some researchers have argued for a question-behavior effect. In a small study with college-aged students, Williams, Block, and Fitzsimons [10] found that responding to a question about future intent of drug use was related to an increase in illegal drug use behavior (e.g. marijuana, cocaine, heroin use) two months later. Subsequent analyses indicated that survey questions appeared to reinforce current substance use behavior among users but did not increase actual drug use among non-drug users. A 2015 meta-analysis identified 38 health-related studies testing the question-behavior effect in adult samples [12], with seven studies addressing the question-behavior effect on alcohol use. Although there was a small overall effect on health-related behaviors such as use of dental floss and physical activity, there was no evidence in the seven alcohol-related studies of a significant question-behavior effect. However, because the seven studies all involved adult samples, they do not bear on concerns about a question-behavior effect in adolescents. Fitzsimons and Moore [8] warned researchers about the potential harm in measuring behaviors such as drug use and sexual activity with surveillance youth surveys. The researchers acknowledged the importance of youth surveys in identifying potential societal problems but argued for post-administration interventions to counter any possible question-behavior effect.

These prior studies provide limited evidence for a question-behavior effect on risky behavior in older samples and warn of the potential harm to school-age youth, but we know of no study that has examined the question-behavior effect in adolescents. To address this gap, the current study tests the validity of the question-behavior effect on youth drug use in a longitudinal panel of 2,002 elementary school students. The study focuses on two subsamples in the control condition of the longitudinal randomized trial of the Communities That Care prevention system. The initial subsample was asked about drug use in both fifth and sixth grade, while an accretion sample was asked about drug use only in sixth grade. If the prevalence of drug use in sixth graders from the initial sample surveyed in fifth and sixth grade was higher than in the accretion sample surveyed only in sixth grade, after controlling for the influence of other possible predictors of drug use, the difference could indicate a question-behavior effect. This study examined evidence for a long-term or short-term question-behavior effect in a longitudinal panel. If asking about drug use in the fifth grade had a long-term impact on drug use, youth in the initial sample should report higher prevalence of recent or past-30-day substance use in Grade 6 compared to Grade 6 prevalence in the accretion sample. If asking about drug use in the fifth grade had a short-term effect, leading youth to experiment with drugs shortly after being asked about drug use, then youth in the initial sample should report higher prevalence of lifetime substance use in Grade 6, but not higher prevalence of recent use, compared to the prevalence of lifetime substance use reported by the accretion sample in Grade 6.

Methods

Data used in the analyses were collected as part of the Community Youth Development Study (CYDS), a community-randomized trial of the Communities That Care (CTC) prevention framework. CTC is a manualized, community-led prevention planning and monitoring system designed to reduce youth drug use and delinquent behaviors and promote positive youth development. The CTC trial involved 24 communities from seven states that were matched in pairs within state on sociodemographic factors, and then randomly assigned to serve as CTC communities or control communities in the trial. For more information about the trial, longitudinal panel, and study design, see Brown, Graham, Hawkins, Arthur, Baldwin, Oesterle, Briney, Catalano, & Abbott [13].

Sample

In each of the 24 participating communities, parents of eligible fifth graders were approached to gain consent to have their child complete a paper-and-pencil questionnaire about risk, protection, and health-related behavior. The fifth-grade consenting process recruited an initial sample of 3,682 students into the study. To increase the participation rate, a second consenting effort was conducted in the following year, which resulted in the recruitment of an additional accretion sample of 1,146 sixth graders not enrolled in the study in fifth grade. The final sample numbered 4,407 students, or 76% of eligible students.

To avoid possible intervention effects, this study focused only on the 2,002 students who were recruited from control communities. Twelve participants failed an “honesty” screen and were removed from the analysis sample. Two participants who consented and were surveyed in Grade 5 but did not complete the survey in Grade 6 were also removed from the analysis sample. The final sample of 1,988 included 1,349 (67.9%) students surveyed in both Grade 5 and Grade 6 (i.e., the initial sample) and 639 (32.1%) students surveyed in Grade 6 only (i.e., the accretion sample). Students were 11.6 years old (SD = 0.54), on average, when surveyed in Grade 6; 51.0% were male; 61% were white, and 27% were Hispanic. Mean parental education level was 4.03 (SD = 1.31) on a 6-point scale ranging from 1 = grade school or less to 6 = graduate or professional degree. The initial and accretion samples were similar in age, gender, and the level of parental education but differed in that students in the accretion sample were significantly more likely to be non-white (initial sample 36.1% and accretion sample 43.7%, p = .020) and Hispanic (initial sample 24.9% and accretion sample 30.0%, p = .017).

Measures

Data were collected using the Youth Development Survey (YDS), a paper-and-pencil questionnaire about risk, protection, and health-related behavior, including drug use, designed to be completed at school during one 50-minute class period. Starting in the spring of 2004, the YDS was administered to the longitudinal panel of fifth-grade students from experimental and control communities in accordance with procedures approved by the University of Washington Institutional Review Board. The survey was repeated the following year in sixth grade.

This study uses data about student self-reported use of cigarettes, alcohol, inhalants, and marijuana over the past 30 days (e.g., How frequently have you smoked cigarettes during the past 30 days?) and over their lifetime (e.g., Have you ever had more than just a sip or two of beer, wine, or hard liquor (for example, vodka, whiskey, or gin)?). The YDS also contains questions about student race, ethnicity, age at time of survey, and parental education, which were included as covariates in statistical analyses.

Statistical Analysis

Logistic regression analysis was used to examine differences in the prevalence of sixth-grade drug use between initial and accretion samples. Dependent variables included eight Grade 6 drug use outcomes: past 30-day use of cigarettes, alcohol, inhalants, and marijuana; and lifetime use of the same four drugs. Independent variables included the participant’s recruitment group (initial sample = 1, accretion sample = 0); student’s gender (male = 1, female = 0); self-identified race (white = 1, non-white = 0) and self-identified Hispanic ethnicity (Hispanic = 1, non-Hispanic = 0); age of the student at Grade 6; and the educational level of the student’s parents (ranging from 1 = Completed grade school or less to 6 = Graduate or professional school after college). To account for the clustering of students in the 12 control communities, 11 dummy variables indicating community were also included in regression analyses.

Regression coefficients were evaluated for significance using the Wald chi-square test at a .05 Type I two-tailed error rate. A significant positive effect of recruitment group predicting drug use was interpreted as evidence of a question-behavior effect. That is, if asking about drug use in Grade 5 “encouraged” students to use drugs in Grade 6, we would expect sixth-grade prevalence of drug use in the initial sample to be higher than in the accretion sample, after controlling for the influence of covariates. Adjusted odds ratios (AORs) captured both the direction and magnitude of the possible question-behavior effect. All regression analyses and data manipulations were performed using SPSS software (version 19).

Results

Analyses did not find evidence supporting a short- or long-term question-behavior effect on drug use in Grade 6 (Table 1). The likelihood of Grade 6 drug use in students who were asked about their drug use in Grade 5 (the initial sample) was not significantly greater than the likelihood of drug use among those who completed the survey only in Grade 6 (the accretion sample). The rate of past-30-day use of cigarettes, alcohol, and marijuana was nearly identical in both groups, indicating no long-term question-behavior effect. Similarly, the prevalence of self-reported lifetime use in Grade 6 was statistically the same for the initial and accretion samples, with the exception of lifetime alcohol use. However, counter to the question-behavior hypothesis, the likelihood of alcohol use in Grade 6 was significantly greater, not less for those in the accretion sample than for those surveyed in both waves (i.e., the initial sample). The prevalence of lifetime cigarette, inhalant, and marijuana use was also higher (Table 1), although not statistically different. These findings indicate that there was also no evidence for a short-term question-behavior effect.

Table 1.

Grade 6 substance use prevalence in the initial compared to the accretion sample

	Initial sample prevalence	Accretion sample prevalence	Logistic regression coefficienta	Standard error	AORb	p
	Past-30-day substance use prevalence
Cigarettes	1.8%	2.0%	0.14	0.37	1.15	0.70
Alcohol	6.6%	6.5%	0.04	0.20	1.04	0.85
Inhalants	3.9%	5.3%	−0.42	0.24	0.66	0.08
Marijuana	0.9%	0.8%	0.57	0.59	1.76	0.34
	Lifetime substance use prevalence
Cigarettes	10.4%	12.6%	−0.27	0.16	0.76	0.08
Alcohol	20.8%	23.6%	−0.26	0.12	0.77	0.03
Inhalants	10.8%	12.7%	−0.15	0.16	0.86	0.33
Marijuana	2.6%	3.4%	−0.19	0.28	0.83	0.50

^aCoefficients are from logistic regressions comparing the likelihood of substance use in Grade 6 among students in the initial compared to accretion sample. Student gender, age, race/ethnicity, parental education, and dummy variables indicating student’s community were included as covariates in all analyses. Positive coefficients indicate greater substance use prevalence in Grade 6 among students in the initial sample.

^bAOR = adjusted odds ratio. Values greater than 1 indicate higher substance use prevalence in the initial sample.

Discussion

This study does not find any evidence that asking youth about past drug use encourages or causes them to engage in future drug use. Our findings indicated no difference in the prevalence of self-reported drug use among those surveyed in Grade 5 and again in Grade 6 (initial sample) compared to those surveyed in Grade 6 only (accretion sample). If there had been lasting harm from asking youth about past drug use, we would have found higher prevalence of past-30-day substance use in Grade 6 among those in the initial sample who were surveyed in Grade 5. If there had been any short-term experimentation with drugs after taking the survey in Grade 5, we would have found a higher prevalence of lifetime drug use in the group of students surveyed in both Grades 5 and 6. However, the survey data do not show either an increase in long-term use or a short-term rise in use due to experimentation.

These findings should bolster confidence among school administrators, community prevention planners, and community members that surveying students about their drug use in the past will not increase their substance use in the future. Anonymous youth surveys done in a school setting is a cost-effective method of gathering important information about both licit and illicit activities. When aggregated to a school, neighborhood, or community level, these data can inform the selection of evidence-based prevention programs.

This study is not without limitations. First, this study relies on self-reported data from fifth- and sixth-grade students, which could introduce a social desirability bias in outcome measures. However, several studies have validated the use of self-reported drug use data [e.g., 14, 15]), and the use of these data is common in prevention and other social science research [e.g., 16]. Even if self-reports under- or overestimate the prevalence of actual drug use, this effect would likely occur in both samples, therefore not affecting the testing of the question-behavior hypothesis. A second limitation concerns generalizability of the current findings given the fact that the drug use data analyzed came from 12 small to medium-sized communities with populations ranging from 1,921 to 32,885 in seven states in the West, Midwest, and Northeast United States. Although this raises the possibility that the current findings may not generalize to youth from urban and suburban populations or other regions of the country (e.g., the South), it should be noted that previous research on data collected with the CTCYS (from which the YDS was derived) supports the use of these survey instruments across gender and racial/ethnic groups [17]. Finally, generalizability of the findings involving the drug use of fifth and sixth graders to older students should be assessed. The CTCYS is designed to be administered to students in Grades 6, 8, 10, and 12. Additional research could be undertaken to see if findings are replicated for youth in later grades.

Conclusion

We found that asking youth about their past drug use in a survey in Grade 5 did not predict an increase in reported drug use one year later. This suggests that school administrators and community prevention planners should not be concerned about adverse effects of surveying students about past drug use. Anonymous school surveys that collect data on past alcohol and other drug use and the risk and protective factors that predict use are a useful and efficient means to gather epidemiological information about students. These data allow policymakers and community members to monitor emerging trends and to make informed decisions to target programs and policies to prevent unhealthy behaviors among students. Policymakers and community members are encouraged to make decisions based on data; however, opposition to surveying students about health-risking behaviors like drug use can inhibit community support of data-based prevention efforts. This study found that surveying students to obtain information about past drug use did not increase risk for future student drug use.