Comparison of a web‐push survey research protocol with a mailed paper and pencil protocol in the Monitoring the Future panel survey

Abstract

Aims

The experiment tested the effects of a web‐push survey research protocol, compared with the standard mailed paper‐and‐pencil protocol, among young adults aged 19–30 years in the ‘Monitoring the Future’ (MTF) longitudinal study.

Design, Setting and Participants

The US‐based MTF study has measured substance use trends among young adults in panel samples followed biennially, using consistent mailed survey procedures from 1977 to 2017. In 2018, young adult participants in the MTF longitudinal component scheduled to be surveyed at ages 19–30 in 2018 (from high school senior cohorts of 2006–17, n = 14 709) were randomly assigned to receive the standard mail/paper survey procedures or new web‐push procedures.

Measurements

Primary outcomes were responding to the survey and prevalence estimates for past 30‐day use of alcohol, cigarettes, marijuana and illicit drugs.

Findings

The web‐push response rate was 39.07% [95% confidence interval (CI) = 37.889, 40.258]; this was significantly better than the standard MTF response rate of 35.12% (95% CI = 33.964, 36.285). After adjusting for covariates, the web‐push condition was associated with a 19% increase in the odds of responding compared with standard MTF (adjusted odds ratio = 1.188; 95% CI = 1.096, 1.287). Substance use prevalence estimates were very similar and differences became negligible when using attrition weights and controlling for socio‐demographic characteristics.

Conclusions

The web‐push protocol produced a higher response rate than the mailed pencil and paper protocol in the Monitoring the Future panel study, without substantially affecting estimates of substance use once attrition weights and socio‐demographic variables were factored in.

Introduction

Measuring substance use trends is especially difficult when both the behaviors of interest and the preferred modes of assessment are changing with time. A key question is how to keep measuring long‐term trends while still keeping pace with updated data collection strategies. It is essential to both minimize methodological artifacts due to changing strategies and characterize any impact of changing strategies on the substance use estimates. The Monitoring the Future (MTF) study has been measuring substance use trends among young adults in panel samples followed biennially, using consistent mailed paper survey procedures from 1977 to 2017 [1]. As is true for almost all longitudinal surveys [1, 2, 3], MTF panel non‐response attrition has been increasing over the years [4]. In 2018, we conducted an experiment whereby one random half of young adults aged 19–30 years in our primary sample received typical MTF mail/paper survey procedures and the other random half received web‐push procedures as described below. The present paper describes this experiment and our findings.

It is important to note that the present experiment followed a series of smaller‐scale experiments where we tested several iterations of web‐based survey options with participants in auxiliary MTF panel samples followed to ages 19–22. In particular, we honed our web‐push procedures and tested their effects on response rates, socio‐demographic variation in respondents, substance use prevalence estimates and survey costs. Web‐push procedures are defined as contact and data collection procedures that push respondents to complete the web‐based survey, with a mailed paper survey provided only for non‐responders or if requested.

More specifically, this large‐scale implementation of web‐based methods in the longitudinal component of the MTF study is informed by three previous randomized controlled experiments within supplemental samples. In 2014, the first follow‐up at modal ages 19–20 using standard mailed paper questionnaires was compared to three experimental mixed‐mode conditions: standard MTF mail push, web‐push and web‐push + e‐mail. Web‐push + e‐mail was the most promising method, based on response rates and lower costs relative to the other two conditions [5]. A second experiment examined retention rates among the same supplemental sample 2 years later, at ages 21–22, in 2016. Responding via web in 2014 was associated with higher rates of participation in the 2016 (regardless of condition), so the web‐push strategy emerged as a promising route for maintaining respondent engagement while reducing costs [6]. A third study extended the initial experiments by further enhancing the web‐push condition through the introduction of text messaging and quick response (QR) codes in addition to e‐mail, and optimizing the web‐based survey for mobile response. The enhanced web‐push condition substantially increased response rates compared to the standard MTF control group [7]. Based on these promising findings using auxiliary samples at the first two follow‐ups (aged 19–22), we moved forward with this experiment of an enhanced web‐push data collection design with the primary MTF longitudinal sample aged 19–30 in 2018.

Our diligence regarding past experiments was deemed essential, given that MTF provides the field and policymakers with annual updates of substance use trends [4], and is one of the most widely used US national data sets for examining the epidemiology and etiology of substance use among youth and young adults. This move to web‐based procedures alters both our respondent contact and data collection procedures, which has the potential to influence not only participation (i.e. differential selection into the study according to socio‐demographic characteristics), but also prevalence estimates of substance use. Thus, the change in procedures could disrupt the time trends.

Our initial iterative experiments suggested that the move to web‐based procedures was unlikely to disrupt time trends. The current experiment with the MTF primary panel sample of young adults tests the extent of disruption by comparing the effect of a web‐push survey research design, compared to the standard MTF mailed paper‐and‐pencil survey, in terms of response rates and estimates of substance use prevalence.

Research questions

Specific research questions for the current study were: (1) did response rates differ for the web‐push compared to standard MTF data collection conditions overall; (2) did response rates differ across socio‐demographic groups and follow‐up waves; and (3) did prevalence estimates of 30‐day substance use differ for the web‐push compared to standard MTF?

Data and procedures

Monitoring the future (MTF) data

The MTF study includes repeated annual cross‐sectional in‐school surveys of nationally representative samples of US 12th grade students [8, 9]. Each year, approximately 2450 students from the cross‐sectional sample of 12th grade students are randomly selected to participate in the longitudinal portion of the study, as described elsewhere; drug users are oversampled [4]. Each cohort of 2450 students is split in half: one half is randomly assigned to begin 1 year later after high school at modal age 19, and the other half to begin 2 years later at modal age 20. Each respondent is surveyed every 2 years thereafter, with follow‐ups occurring at ages 19–20, 21–22, 23–24, 25–26, 27–28 and 29–30. The current study included young adult participants in the MTF longitudinal component scheduled to be surveyed in 2018 from high school senior cohorts of 2006–17. Those who reported drug use at baseline were oversampled; weights were used to adjust for this sampling procedure. Characteristics of the sample by condition (outlined in more detail below) are described in Supporting information, Table S1.

Consistent with MTF reporting of response rates in the annual reports [4], we have included inactive cases [2] in the denominator for the current set of results, which provides estimates that can be comparable with other MTF studies [3]. The total unweighted number of participants included in the current study are reported in Supporting information, Table S1. A total of 3029 participants (of 14 709) across the six age groups were counted as inactive in 2018 and were subsequently randomized to experimental conditions and included in the denominator of all estimates.

Experimental design

The present study follows a similar protocol used by Patrick et al. [5]. Participants who were eligible for a follow‐up survey in 2018 were randomly assigned (1 : 1) to one of two experimental conditions: standard MTF or web‐push. Respondent contact procedures for the two conditions are summarized in Table 1. Sample size was sufficient for detecting effect sizes produced in previous experiments [5, 6, 7].

Table 1.

Experimental procedures by condition.

		Condition
Order	Action a	Standard MTF	Web‐push condition
1	Selection letter; newsletter	Cp	Cw
2	Questionnaire	P, $	W, $, e‐mail, text (SMS)
3	Reminder postcard	Cp	W, e‐mail
4	Reminder letter	Cp	Pb, W, $, e‐mail
5	Non‐response phone call	Cp	Cpw
6	Final mailing	P	Cpw, P, W

$ = incentive check;

Cp = communication mentioning the paper questionnaire (not including the actual questionnaire);

Cpw = communication mentioning both the paper questionnaire and the web questionnaire;

Cw = communication without mention of mode of survey;

e‐mail = communication duplicated in an e‐mail to those who provided e‐mail addresses;

text (SMS) = communication duplicated in a text message;

W = web survey log‐in information (URL, PIN and QR code).

All occur via postal mail service except for ‘e‐mail’.

^aProcedures are based on Monitoring the Future (MTF) protocol and altered minimally across conditions.

^bThis is the first time web‐push respondents receive a paper questionnaire.

For the standard MTF condition, longitudinal participants were sent one mailing in December, as per usual MTF protocol [4]. The mailing included a letter, which informed participants that they would be receiving a questionnaire and would be paid $25 for their participation, a newsletter containing selected summary results from the study in an informational format and a change of address card for the respondent to update contact information. In April, a paper questionnaire was mailed along with a pencil, pre‐paid return envelope and check for $25 in the participant's name. A reminder postcard was sent 1 week later, and a reminder letter was sent 3 weeks after that (for those participants who had not yet returned their questionnaire). One week later, non‐response phone calls were made to all those who had not yet returned a questionnaire. A final mailing approximately 6 weeks later included a second copy of the paper questionnaire (for those participants who had not yet returned one).

In the web‐push condition, similar to the standard MTF condition, a letter and newsletter were sent to participants notifying them that they would be invited to complete a survey. However, the web‐push condition included a mailed invitation without mentioning a paper survey. Instead of a paper questionnaire, participants in the web‐push condition were sent web survey login information [i.e. survey uniform resource locator (URL) and personal identification number (PIN), as well as a QR code] and a check for $25. Additionally, an e‐mailed version of the invitation to complete an on‐line survey was sent to those participants who provided an e‐mail address. A reminder e‐mail was sent in addition to a reminder postcard that was sent by mail, which was the same as the standard MTF condition except that it requested completion of an on‐line survey. A reminder was sent 1 week later, similar to standard MTF, and telephone calls were made to all those who had not yet returned a questionnaire. However, for the web‐push condition, the reminder was sent via e‐mail as well as a letter, and the letter included the on‐line survey login information and also a paper questionnaire as an additional option. At the same time, those who had provided a mobile phone number and consented to receive texts were sent an SMS (short text message) with the login information. Finally, after the round of non‐response prompting by telephone, the web‐push group was also sent a final paper questionnaire and information about the web survey option. It is important to note that across conditions, we made minimal changes to survey layout, text of communications, and survey content so as not to confound differences in communication with the survey and invitation modes.

In summary, the standard MTF condition was sent two paper questionnaires and no mention of the web. The web‐push condition was sent login credentials to complete the survey on‐line at the first and final reminder, plus up to four e‐mail messages and a text message. Non‐responders in the web‐push condition were also sent up to two paper questionnaires. All procedures are shown in Table 1.

The web version of the survey was programmed using DatStat's Illume Next software. In the baseline survey, respondents were asked to provide an e‐mail address and mobile phone number with permission to receive text messages. In the web‐push condition, 78.9% provided a working e‐mail address and 9.8% provided a mobile phone number and permission for texting [4].

Measures

Our primary focus was on the outcome measure of any response versus no response (yes or no) at 2018 follow‐up. In supplemental analyses, we also examined mode of response among participants in the web‐push condition (web versus paper response). We used measures from baseline (12th grade, in‐school surveys) that were available for both respondents and non‐respondents of the experiment. In addition, in analyses among respondents (e.g. substance use at follow‐up), we also included concurrent characteristics (modal age at 2018 follow‐up) that were provided by respondents at follow‐up (available only for respondents). Weights were used to account for the oversampling of baseline drug users for analyses of response rates and predictors of responding.

Baseline characteristics (12th grade, modal age 18)

Gender was coded as male or female (1 = male, 0 = female). Race/ethnicity was coded as white, black, Hispanic or other. Parent education was coded based on whether at least one parent had at least some college education (compared to high school education or less). Four‐year college plans were coded based on whether the participant indicated that they would ‘definitely’ graduate from a 4‐year college program, compared to other responses (probably will, probably will not and definitely will not). Life‐time substance use measures indicated whether the participant had ever used any alcohol, cigarettes, marijuana or illicit drugs other than marijuana (yes or no).

Concurrent characteristics (at 2018 follow‐up)

Follow‐up number and age at follow‐up were coded as: 1 (age 19–20), 2 (age 21–22), 3 (age 23–24), 4 (age 25–26), 5 (age 27–28) and 6 (age 29–30). College student status was coded as current full‐time enrollment in a 4‐year college (yes or no). Highest education achieved indicated whether the participant had some college experience or more versus a high school degree or less. Employment indicated whether the participant reported a full‐time job, a part‐time job or two or more different jobs (yes), or no outside job or paid employment, laid‐off or waiting to start a job (no) during the first full week in March 2018. Substance use in the past 30 days indicated whether the participant used alcohol, cigarettes, marijuana or illicit drugs other than marijuana (yes or no). MTF attrition weights that account for sample loss since baseline, oversampling of drug users and the complex survey design of the MTF cross‐sectional survey were used only for analyses on past 30‐day drug use at follow‐up [10, 11, 12, 13].

Analytical strategy

The analysis plan was not pre‐registered; results should be considered exploratory. Analyses consisted of cross‐tabulations and logistic regressions using the complex survey design function (to account for the oversampling of drug users and the complex survey design) in Stata version 5. First, cross‐tabulations compare response rates between conditions and among subgroups. Design‐based F‐tests were used to assess statistical significance; pairwise deletion was used for missing data. Two multivariable logistic regressions (with listwise deletion) were used to assess participation: (a) including push condition, follow‐up number and socio‐demographic characteristics (research question 1), and (b) including all previous variables plus interactions between push condition and covariates (research question 2). Interactions were multiplicative terms, dummy variables were compared to the reference group and Benjamini–Hochberg tests with a false discovery rate of 5% adjusted for multiple testing [14]. Post‐estimation global tests (joint F‐tests) were also used to assess statistical significance sets of interactions (i.e. modal age, race/ethnicity) where more than one multiplicative term was needed.

To address research question 3, we assessed substance use across conditions using weights that accounted for attrition, oversampling of drug users and the MTF complex sampling design. We conducted two different logistic regressions for each substance use: (a) a model that included only push condition and (b) a model that included push condition and socio‐demographic characteristics.

Finally, supplemental analyses compared socio‐demographic characteristics across conditions, including logistic regressions with predictors of web response (versus paper) in the web‐push condition and examining differences in 30‐day substance use prevalence across response modes.

Results

R1: overall response rates

Results for overall response rates by condition showed that the response rate for the web‐push condition was significantly higher than the standard MTF condition [web‐push = 39.07%, 95% confidence interval (CI) = 37.889, 40.258] versus standard MTF = 35.12%, 95% CI = 33.964%, 36.285%; design‐based F = 21.771; P < 0.001).

R2: response outcomes by socio‐demographic characteristics

Bivariate results

As shown in Table 2, using drug user oversampling weights, there were significant differences in response rates between conditions for nearly all groups, except for black participants and participants not planning to attend a 4‐year college. There were significant differences only between conditions for follow‐ups 1 and 3 (modal ages 19–20 and 23–24); the difference in response rates across conditions for follow‐ups 2 and 5 (modal ages 21–22 and 27–28) were in the same direction. These differences are visually presented in Fig. 1.

Table 2.

Response rates overall and by baseline characteristics by condition.

	Standard MTF		Web‐push condition		P‐value
	%	SE	%	SE
Total (overall)	35.125	0.006	39.074	0.006	< 0.001
Follow‐up number (age, years)
1 (age 19–20)	31.105	0.014	43.868	0.015	< 0.001
2 (age 21–22)	32.384	0.014	36.023	0.015	NS
3 (age 23–24)	32.489	0.014	39.498	0.015	< 0.001
4 (age 25–26)	36.844	0.015	35.373	0.015	NS
5 (age 27–28)	36.933	0.015	39.083	0.015	NS
6 (age 29–30)	41.149	0.015	40.528	0.015	NS
Sex
Male	30.984	0.008	33.464	0.009	< 0.05
Female	39.134	0.008	44.137	0.009	< 0.001
Race/ethnicity
White	40.444	0.008	42.835	0.008	NS
Black	24.630	0.016	28.624	0.017	NS
Hispanic	27.789	0.014	33.310	0.015	< 0.05
Other	30.776	0.017	40.127	0.017	< 0.001
Parent education
High school or less	28.857	0.011	32.941	0.011	< 0.05
Some college/more	38.434	0.007	42.234	0.007	< 0.001
4‐year college plans
Not definitely	29.415	0.009	30.927	0.009	NS
Definitely	39.775	0.008	45.427	0.008	< 0.001
Any life‐time substance use
Alcohol	32.929	0.007	37.097	0.007	< 0.001
Cigarettes	28.362	0.009	32.242	0.009	< 0.01
Marijuana	28.525	0.008	33.460	0.009	< 0.001
Other illicit drugs	28.626	0.011	35.022	0.011	< 0.001

P‐value is for significance test for respective differences between rates across rows. NS = non‐significant; MTF = Monitoring the Future; SE = standard error.

Figure 1.

Response rate in 2018 by condition for each follow‐up/age group

Multivariable logistic regression results

Main effects

Table 3 shows logistic regression results for predicting any response versus no response in 2018 based on experimental condition, adjusting for baseline characteristics. The web‐push condition was associated with a 19% increase in the odds of responding compared to the standard MTF condition, adjusting for covariates. In terms of follow‐up number/modal ages, each follow‐up had lower odds of responding compared to follow‐up 6 (modal ages 29–30); however, the differences between follow‐ups 1 and 6 and between follow‐ups 5 and 6 were non‐significant. Black, Hispanic and other racial/ethnic participants had lower odds of responding relative to white participants. Parental education and college plans were both positively associated with responding. Participants with life‐time alcohol use at baseline were less likely to respond versus participants with no history of use, and this pattern held for life‐time cigarette use and marijuana use (the relationship for other illicit drug use was non‐significant).

Table 3.

Multiple logistic regression predicting any response (1) versus no response (0) based on experimental condition and baseline characteristics.

	Model 1			Model 2
	aOR	(95% CI)		aOR	(95% CI)
Web‐push (versus standard MTF)	1.188	(1.096, 1.287) ***		0.856	(0.627, 1.166)
Follow‐up number/age, years
1 (age 19–20)	0.905	(0.784, 1.046)		0.682	(0.555, 0.839) ***
2 (age 21–22)	0.747	(0.649, 0.860) ***		0.699	(0.573, 0.853) ***
3 (age 23–24)	0.804	(0.703, 0.918) **		0.688	(0.568, 0.833) ***
4 (age 25–26)	0.820	(0.717, 0.938) **		0.849	(0.703, 1.026)
5 (age 27–28)	0.900	(0.788, 1.028)		0.868	(0.719, 1.047)
6 (age 29–30: reference)
Male	0.684	(0.631, 0.743) ***		0.659	(0.634, 0.800) ***
Race/ethnicity
White (reference)
Black	0.506	(0.436, 0.587) ***		0.496	(0.401, 0.614) ***
Hispanic	0.690	(0.608, 0.783) ***		0.680	(0.568, 0.813) ***
Other	0.755	(0.666, 0.857) ***		0.674	(0.560, 0.812) ***
Parent some college education	1.290	(1.170, 1.422) ***		1.228	(1.069, 1.409) **
4‐year college plans (definite)	1.459	(1.337, 1.593) ***		1.622	(1.432, 1.835) ***
Any life‐time substance use (age 18)
Alcohol use	0.819	(0.740, 0.907) ***		0.865	(0.748, 0.998) *
Cigarette use	0.745	(0.670, 0.828) ***		0.734	(0.632, 0.851) ***
Marijuana use	0.784	(0.705, 0.872) ***		0.771	(0.663, 0.897) **
Other illicit drug use	0.948	(0.853, 1.053)		1.084	(0.935, 1.256)
Web‐push × follow‐up (joint F‐test = 4.80***)
× 1 (age 19–20)				1.748	(1.309, 2.336) ***
× 2 (age 21–22)				1.142	(0.861, 1.514)
× 3 (age 23–24)				1.358	(1.040, 1.775) *
× 4 (age 25–26)				0.926	(0.708, 1.210)
× 5 (age 27–28)				1.072	(0.822, 1.397)
Web‐push × male				1.081	(0.917, 1.273)
Web‐push × race/ethnicity (joint F‐test = 0.90)
× Black				1.028	(0.762, 1.384)
× Hispanic				1.021	(0.793, 1.315)
× Other				1.237	(0.958, 1.595)
Web‐push × parental education				1.100	(0.904, 1.338)
Web‐push × college plans				0.810	(0.680, 0.966) *
Web‐push × alcohol use				0.899	(0.733, 1.101)
Web‐push × cigarette use				1.036	(0.838, 1.282)
Web‐push × marijuana use				1.026	(0.829, 1.268)
Web‐push × illicit drug use				0.758	(0.613, 0.937) *

Weighted n = 9946 (unweighted n = 12 005); model 1 does not include interaction terms.

^* P < 0.05;

^** P < 0.01;

^*** P < 0.001; P‐values for interaction terms account for multiple testing. MTF = Monitoring the Future; CI = confidence interval; aOR = adjusted odds ratio.

Interactions

Interaction terms between socio‐demographic characteristics and condition are shown in model 2 (Table 3). Corroborating bivariate findings, there was a significant interaction between follow‐up and condition. As indicated in Fig. 1, the web‐push condition had a stronger effect for follow‐ups 1 and 3 compared to follow‐up 6 (joint F‐test = 4.80; P < 0.001). We also tested the interaction between follow‐up and condition using a linear measure of follow‐up (results now shown), and similarly a significant interaction was found [adjusted odds ratio (aOR) = 0.90, 95% CI = 0.86, 0.95]. After accounting for multiple testing, there were no significant interactions between condition and demographic characteristics of sex, race/ethnicity, parental education or college plans. There was a significant interaction between life‐time illicit drug use and condition (no other interaction between condition and life‐time substance use was significant). Figure 2 shows this interaction using the predicted probability of responding: the web‐push condition had a stronger effect among participants who reported life‐time illicit drug use relative to participants who reported no life‐time illicit drug use at baseline.

Figure 2.

Predicted probability of responding according to life‐time illicit drug use by treatment condition

R3: substance use at follow‐up

Differences in current substance use by condition using attrition weights are presented in Table 4. Vaping nicotine in the past 30 days was statistically significantly different across conditions (web‐push = 10.88% versus standard MTF = 7.79%; P < 0.05). There were no other significant differences in substance use across conditions. The difference in vaping nicotine in the past 30 days became non‐significant in a logistic regression model that adjusted for socio‐demographic characteristics (Table 5) [5].

Table 4.

Prevalence of substance use in the past 30 days reported at 2018 follow‐up by condition.

	Standard MTF		Web‐push
	%	SE	%	SE
Alcohol	65.655	0.011	65.716	0.011
Cigarettes	12.129	0.008	12.857	0.007
Marijuana	23.529	0.011	24.692	0.010
Illicit drugs	7.348	0.007	6.101	0.005
Vaping (nicotine)	7.790*	0.010	10.875*	0.011

For comparisons,

^*difference was statistically significant (P < 0.05). MTF = Monitoring the Future; SE = standard error.

Table 5.

Logistic regression using attrition weights predicting past 30‐day substance use based on experimental condition and baseline and follow‐up characteristics.

	Alcohol use		Cigarette use		Marijuana use		Other illicit drug use		Vaping
	OR	(95% CI)	OR	(95% CI)	OR	(95% CI)	OR	(95% CI)	OR	(95% CI)
Web‐push (versus standard MTF)
Model 1 (bivariate)	1.002	(0.876–1.148)	1.069	(0.875–1.305)	1.066	(0.914–1.243)	0.819	(0.635–1.058)	1.444	(1.028–2.029)*
Model 2 (adjusted for socio‐demographics)	1.011	(0.875–1.168)	1.130	(0.923–1.385)	1.087	(0.930–1.270)	0.851	(0.659–1.099)	1.406	(0.984–2.008)

Socio‐demographics included race/ethnicity, sex, parents’ education, college aspirations, current student status at follow‐up, highest degree at follow‐up and current employment status at follow‐up.

^* P < 0.05. MTF = Monitoring the Future; SE = standard error; OR = odds ratio; CI = confidence interval.

Supplemental analyses: mode of response among web‐push participants

Socio‐demographic and baseline differences

In terms of mode of response among participants in the web‐push condition, far more responded via web (81.20%) than via paper (18.80%). In bivariate relationships, baseline parental education, college plans and life‐time substance use of cigarettes, marijuana and other illicit drugs were associated with response mode among web‐push participants (Supporting information, Table S2). Specifically, participants with parents who had some college or more and participants with definite plans to attend a 4‐year college were more likely to respond via web relative to participants with parents who had educational levels of high school or less, and participants with no definite plans to attend a 4‐year college, respectively. Participants with no life‐time use of cigarettes, marijuana or other illicit drugs at baseline were more likely to respond via web compared to participants who reported life‐time use of these substances at baseline. However, in a multiple logistic regression model, there were no significant relationships between response mode and baseline or concurrent characteristics (Supporting information, Table S3).

Substance use differences at follow‐up

Substance use differences across response mode are shown in Supporting information, Table S4. Using attrition weights (which account for attrition, drug user oversampling into panel and the complex design of MTF), we found no differences in substance use across response modes except for cigarette use (paper = 18.21% versus web = 11.48%, P < 0.001). This difference in cigarette use across response modes remained significant (P < 0.05) in a logistic regression model that adjusted for socio‐demographic characteristics and life‐time substance use (not shown).

Discussion

The Monitoring the Future panel study is one of the most frequently used data sets concerning the epidemiology and etiology of substance use in young adulthood in the United States. Following promising results from randomized survey research experiments on supplemental samples [5, 6, 7], a decision was made to integrate web‐based data collection procedures in a random half of the panel beginning in 2018. The response rate for the web‐push condition was higher than the standard MTF methods condition, particularly for the first (at modal age 19–20) and third follow‐ups (at modal age 23–24). After adjusting for covariates, the web‐push condition was associated with a 19% increase in the odds of responding compared to the standard MTF condition. We examined the effects of socio‐demographic variables and baseline characteristics. Of particular importance for the study, the web‐push condition had a stronger effect on response among life‐time illicit drug users than non‐users. This relationship was driven by particularly low response rates for standard MTF among participants who report life‐time illicit drug use and a small difference between web‐push and standard MTF among participants with no life‐time illicit drug use. Drug users are particularly likely to be non‐responders, so this is a potential advantage of the web‐based methods. Other advantages of web‐based methods include cost savings [5] and greater flexibility in creating skip patterns and targeted data collection modules, speed of response and improved data quality [15]. Finally, substance use prevalence data were very similar across the two conditions (consistent with prevalence estimates reported in the MTF panel annual report) [4].

Based on these data, collected in 2018, any differences in 30‐day prevalence estimates of substance use become negligible when using attrition weights and controlling for socio‐demographic characteristics (i.e. sex, race/ethnicity and parental education) assessed at baseline. Any changes in the composition of the sample due to increased response are adjusted by attrition weights. We recommend that users of the MTF panel data construct attrition weights (e.g. as in [10–13]) and control for socio‐demographic characteristics. Beyond that, additional modeling of the web‐based experiment with panel participants in 2018 is not necessary. In 2019 the experiment was repeated (data will be available by the end of 2020), and in 2020, web‐push will become the new standard panel data collection protocol for those aged 19–30. Extensions and replications of this work within and outside the United States are needed.

Conclusions

The web‐push design is helpful in reversing the declines in response rates, especially among younger respondents; a particularly promising finding was that the response rate for the web‐push condition at follow‐up 1 (modal age 19–20) was the highest response rate compared to any other age. Our results suggest that, as long as analyses adjust for differential attrition by base year characteristics, additional adjustment for mode of data collection are probably unnecessary when using MTF panel data to examine young adult substance use.

Declaration of interests

None.

Supporting information

Table S1 Baseline (Age 18) Sample Characteristics by Condition

Table S2 Web and Paper Response Rates by Baseline Characteristics among Responders in Web‐Push Condition

Table S3 Multiple Logistics Regression Predicting Web Response (1) versus Paper Response (0) among Participants in Web‐Push Based on Baseline (Age‐18) and Concurrent (Age at Follow‐Up) Characteristics

Table S4 Post 30‐Day Substance Use Prevalence Reported at 2018 Follow‐Up Mode of Response in Web‐Push Condition.

Patrick, M. E. , Couper, M. P. , Parks, M. J. , Laetz, V. , and Schulenberg, J. E. (2021) Comparison of a web‐push survey research protocol with a mailed paper and pencil protocol in the Monitoring the Future panel survey. Addiction, 116: 191–199. 10.1111/add.15158.