Abstract
Aims.
The experiment tested the effects of a Web-Push survey research protocol, compared with the standard mailed paper and pencil protocol, among young adults aged 19 to 30 in the ‘Monitoring the Future’ (MTF) longitudinal study.
Design, Setting & Participants.
The U.S.-based MTF study has measured substance use trends among young adults in panel samples followed biennially, using consistent mailed survey procedures from 1977 to 2017. In 2018, young adult participants in the MTF longitudinal component scheduled to be surveyed at ages 19–30 in 2018 (from high school senior cohorts of 2006–2017, N=14,709) were randomly assigned to receive the standard mail/paper survey procedures or new Web-Push procedures.
Measurements.
Primary outcomes were responding to the survey and prevalence estimates for past 30-day use of alcohol, cigarettes, marijuana, and illicit drugs.
Findings.
The Web-Push response rate was 39.07% (95% CI=37.889, 40.258); this was significantly better than the standard MTF response rate of 35.12% (95% CI=33.964, 36.285). After adjusting for covariates, the Web-Push condition was associated with a 19% increase in the odds of responding compared with standard MTF (AOR=1.188; 95% CI=1.096, 1.287). Substance use prevalence estimates were very similar and differences became negligible when using attrition weights and controlling for sociodemographic characteristics.
Conclusions.
The Web-Push protocol produced a higher response rate than the mailed pencil and paper protocol in the Monitoring the Future (MTF) panel study, without substantially affecting estimates of substance use once attrition weights and sociodemographic variables were factored in.
Introduction
Measuring substance use trends is especially difficult when both the behaviors of interest and the preferred modes of assessment are changing with time. A key question is how to keep measuring long-term trends while still keeping pace with updated data collection strategies. It is essential to both minimize methodological artifact due to changing strategies and characterize any impact of changing strategies on the substance use estimates. Monitoring the Future (MTF) has been measuring substance use trends among young adults in panel samples followed biennially, using consistent mailed paper survey procedures from 1977 to 2017.1 As is true for most all longitudinal surveys (1–3), MTF panel non-response attrition has been increasing over the years (4). In 2018, we conducted an experiment whereby one random half of young adults ages 19–30 in our primary sample received typical MTF mail/paper survey procedures and the other random half received web-push procedures as described below. The present paper describes this experiment and our findings.
It is important to note that the present experiment followed a series of smaller-scale experiments where we tested several iterations of web-based survey options with participants in auxiliary MTF panel samples followed through ages 19–22. In particular, we honed our web-push procedures and tested their effects on response rates, sociodemographic variation in respondents, substance use prevalence estimates, and survey costs. Web-push procedures are defined as contact and data collection procedures that push respondents to complete the web-based survey, with a mailed paper survey provided only for nonresponders or if requested.
More specifically, this large-scale implementation of web-based methods in the longitudinal component of the MTF study is informed by three previous randomized controlled experiments within supplemental samples. In 2014, the first follow-up at modal ages 19/20 using standard mailed paper questionnaires was compared to three experimental mixed mode conditions: Standard MTF Mail Push, Web-Push, and Web-Push+Email. Web-Push+Email was the most promising method based on response rates and lower costs relative to the other two conditions (5). A second experiment examined retention rates among the same supplemental samples two years later, at ages 21/22 in 2016. Responding via web in 2014 was associated with higher rates of participation in the 2016 (regardless of condition), so the Web-Push strategy emerged as a promising route for maintaining respondent engagement while reducing cost (6). A third study extended the initial experiments by further enhancing the Web-Push condition through the introduction of text messaging and quick response (QR) codes in addition to email, and optimizing the web-based survey for mobile response. The enhanced Web-Push condition substantially increased response rates compared to the Standard MTF control group (7). Based on these promising findings using auxiliary samples at the first two follow-ups (ages 19–22), we moved forward with this experiment of an enhanced Web-Push data collection design with the primary MTF longitudinal sample aged 19 to 30 in 2018.
Our diligence regarding past experiments was deemed essential given that MTF provides the field and policymakers annual updates of substance use trends (4), and it is one of the most widely used US national datasets for examining the epidemiology and etiology of substance use among youth and young adults. This move to web-based procedures alters both our respondent contact and data collection procedures, which has potential to influence not only participation (i.e., differential selection into the study according to sociodemographic characteristics) but also prevalence estimates of substance use. Thus, the change in procedures could disrupt the time trends.
Our initial iterative experiments suggested that the move to web-based procedures was unlikely to disrupt time trends. The current experiment with the MTF primary panel sample of young adults tests the extent of disruption by comparing the effect of a Web-Push survey research design, compared to the Standard MTF mailed paper and pencil survey, in terms of response rates and estimates of substance use prevalence.
Research Questions
Specific research questions for the current study were: (1) Did response rates differ for the Web-Push compared to Standard MTF data collection conditions overall? (2) Did response rates differ across sociodemographic groups and follow-up waves? (3) Did prevalence estimates of 30-day substance use differ for the Web-Push compared to Standard MTF?
Data and Procedures
Monitoring the Future (MTF) Data
The MTF study includes repeated annual cross-sectional in-school surveys of nationally representative samples of US 12th grade students (8,9). Each year, approximately 2,450 students from the cross-sectional sample of 12th grade students are randomly selected to participate in the longitudinal portion of the study as described elsewhere; drug users are oversampled (4). Each cohort of 2,450 students is split in half: one half is randomly assigned to begin one year later after high school at modal age 19, and the other half to begin two years later at modal age 20. Each respondent is surveyed every two years thereafter, with follow-ups occurring at ages 19/20, 21/22, 23/24, 25/26, 27/28 and 29/30. The current study included young adult participants in the MTF longitudinal component scheduled to be surveyed in 2018 from high school senior cohorts of 2006–2017. Those who reported drug use at baseline were oversampled; weights were used to adjust for this sampling procedure. Characteristics of the sample, by condition (outlined in more detail below), are described in Supplemental Table 1.
Consistent with MTF reporting of response rates in the annual reports (4), we have included inactive cases2 in the denominator for the current set of results, which provides estimates that can be comparable with other MTF studies.3 The total unweighted number of participants included in the current study are reported in Supplemental Table 1. A total of 3,029 participants (out of 14,709) across the six age groups were counted as inactive in 2018 and were subsequently randomized to experimental conditions and included in the denominator of all estimates.
Experimental Design
The present study follows a similar protocol used in Patrick et al. (5). Participants who were eligible for a follow-up survey in 2018 were randomly assigned (1:1) to one of two experimental conditions: Standard MTF or Web-Push. Respondent contact procedures for the two conditions are summarized in Table 1. Sample size was sufficient for detecting effect sizes produced in previous experiments (5–7).
Table 1.
Experimental Procedures by Condition
| Condition | |||
|---|---|---|---|
| Order | Actiona | Standard MTF | Web-Push Condition |
| 1 | Selection letter; Newsletter | Cp | Cw |
| 2 | Questionnaire | P, $ | W, $, Email, Text(SMS) |
| 3 | Reminder postcard | Cp | W, Email |
| 4 | Reminder letter | Cp | Pb, W, $, Email |
| 5 | Nonresponse phone call | Cp | Cpw |
| 6 | Final mailing | P | Cpw, P, W |
NOTE.— $=incentive check;
Cp=communication mentioning the paper questionnaire (not including the actual questionnaire);
Cpw=communication mentioning both the paper questionnaire and the web questionnaire;
Cw=communication without mention of mode of survey;
Email=communication duplicated in an email to those who provided e-mail addresses;
Text(SMS)=communication duplicated in a text message;
P=paper questionnaire;
W=web survey log-in information (URL, PIN, and QR code).
All occur via postal mail service except for “email.”
This is the first time Web-Push respondents receive a paper questionnaire.
Procedures are based on MTF protocol and altered minimally across conditions.
For the Standard MTF condition, longitudinal participants were sent one mailing in December per usual MTF protocol (4). The mailing included a letter, which informed participants that they would be receiving a questionnaire and would be paid $25 for their participation, a newsletter containing selected summary results from the study in an informational format, and a change of address card for the respondent to update contact information. In April, a paper questionnaire was mailed along with a pencil, prepaid return envelope, and check for $25 in the participant’s name. A reminder postcard was sent one week later, and a reminder letter was sent three weeks after that (for those participants who had not yet returned their questionnaire). One week later, nonresponse phone calls were made to all those who had not yet returned a questionnaire. A final mailing about six weeks later included a second copy of the paper questionnaire (for those participants who had not yet returned one).
In the Web-Push condition, similar to the Standard MTF condition, a letter and newsletter were sent to participants notifying them that they would be invited to complete a survey. However, the Web-Push condition included a mailed invitation without mentioning a paper survey. Instead of a paper questionnaire, participants in the Web-Push condition were sent web survey login information (i.e., survey URL and PIN, as well as a QR code) and a check for $25. Additionally, an emailed version of the invitation to complete an online survey was sent to those participants who provided an email address. A reminder email was sent in addition to a reminder postcard that was sent by mail, which was the same as the Standard MTF condition except it requested completion of an online survey. A reminder was sent one week later, similar to Standard MTF, and phone calls were made to all those who had not yet returned a questionnaire. However, for the Web-Push condition, the reminder was sent via email as well as a letter, and the letter included the online survey login information but also a paper questionnaire as an additional option. At the same time, those who had provided a mobile phone number and consented to receive texts were sent an SMS (text message) with the login information. Finally, after the round of nonresponse prompting by telephone, the Web-Push group was also sent a final paper questionnaire and information about the web survey option. It is important to note that across conditions, we made minimal changes to survey layout, text of communications, and survey content so as not to confound differences in communication with the survey and invitation modes.
In summary, the Standard MTF condition was sent two paper questionnaires and no mention of the web. The Web-Push condition was sent login credentials to complete the survey online at the first and final reminder, plus up to four email messages and a text message. Nonresponders in the Web-Push condition were also sent up to two paper questionnaires. All procedures are shown in Table 1.
The web version of the survey was programmed using DatStat’s Illume Next software. In the baseline survey, respondents were asked to provide an email address and mobile phone number with permission to receive text messages. In the Web-Push condition, 78.9% provided a working email address and 9.8% provided a mobile phone number and permission for texting.4
Measures
Our primary focus was on the outcome measure of any response versus no response (yes or no) at 2018 follow-up. In supplemental analyses, we also examined mode of response among participants in the Web-Push condition (web vs. paper response). We used measures from baseline (12th grade, in-school surveys) that were available for both respondents and nonrespondents of the experiment. In addition, in analyses among respondents (e.g., substance use at follow-up), we also included concurrent characteristics (modal age at 2018 follow-up) that were provided by respondents at follow-up (available only for respondents). Weights were used to account for the oversampling of baseline drug users for analyses of response rates and predictors of responding.
Baseline Characteristics (12th Grade, Modal Age 18).
Gender was coded as male or female (1=male, 0=female). Race/ethnicity was coded as White, Black, Hispanic, or Other. Parent education was coded based on whether at least one parent had at least some college education (compared to high school education or less). Four-year college plans were coded based on whether the participant indicated that they would “definitely” graduate from a four-year college program, compared to other responses (probably will, probably won’t, and definitely won’t). Lifetime substance use measures indicated whether the participant had ever used any alcohol, cigarettes, marijuana, or illicit drugs other than marijuana (yes or no).
Concurrent Characteristics (At 2018 Follow-Up).
Follow-up number and age at follow-up were coded as: 1 (age 19/20), 2 (age 21/22), 3 (age 23/24), 4 (age 25/26), 5 (age 27/28), 6 (age 29/30). College student status was coded as current full-time enrollment in a four-year college (yes or no). Highest education achieved indicated whether the participant had some college experience or more versus a high school degree or less. Employment indicated whether the participant reported a full-time job, a part-time job, or two or more different jobs (yes), or no outside job or paid employment, laid-off, or waiting to start a job (no) during the first full week in March 2018. Substance use in the past 30 days indicated whether the participant used alcohol, cigarettes, marijuana, or illicit drugs other than marijuana (yes or no). MTF attrition weights that account for sample loss since baseline, oversampling of drug users, and the complex survey design of the MTF cross-sectional survey were used only for analyses on past 30-day drug use at follow-up (10–13).
Analytic Strategy
The analysis plan was not pre-registered; results should be considered exploratory. Analyses consisted of cross-tabulations and logistic regressions using the complex survey design function (to account for the oversampling of drug users and the complex survey design) in Stata v.15. First, cross-tabulations compare response rates between conditions and among subgroups. Design-based F-tests were used to assess statistical significance; pair-wise deletion was used for missing data. Two multivariable logistic regressions (with list-wise deletion) were used to assess participation: (a) including push condition, follow-up number, and sociodemographic characteristics (Research Question 1), and (b) including all previous variables plus interactions between push condition and covariates (Research Question 2). Interactions were multiplicative terms, dummy variables were compared to the reference group, and Benjamini-Hochberg tests with a false discovery rate of 5% adjusted for multiple testing (14). Post-estimation global tests (joint F-tests) were also used to assess statistical significance sets of interactions (i.e., modal age, race/ethnicity) where more than one multiplicative term was needed.
To address Research Question 3, we assessed substance use across conditions using weights that accounted for attrition, oversampling of drug users, and the MTF complex sampling design. We conducted two different logistic regressions for each substance use: (a) a model that included only push condition, and (b) a model that included push condition and sociodemographic characteristics.
Finally, supplemental analyses compared sociodemographic characteristics across conditions, including logistic regressions with predictors of web response (versus paper) in the Web-Push condition, and examining differences in 30-day substance use prevalence across response modes.
Results
R1: Overall Response Rates
Results for overall response rates by condition showed that the response rate for the Web-Push condition was significantly higher than the Standard MTF condition (Web-Push=39.07% [95% CI=37.889, 40.258] vs. Standard MTF=35.12% [95% CI=33.964%, 36.285%]; design-based F=21.771; p<.001).
R2: Response Outcomes by Sociodemographic Characteristics
Bivariate Results
Shown in Table 2, using drug user oversampling weights, there were significant differences in response rates between conditions for nearly all groups, except for Black participants, and participants not planning to attend a 4-year college. There were significant differences only between conditions for follow-ups 1 and 3 (modal ages 19/20 and 23/24); the difference in response rates across conditions for follow-up 2 and 5 (modal ages 21/22 and 27–28) were in the same direction. These differences are visually presented in Figure 1.
Table 2.
Response Rates Overall and by Baseline Characteristics, by Condition
| Standard MTF | Web-Push Condition | ||||
|---|---|---|---|---|---|
| % | SE | % | SE | P-value | |
| Total (overall) | 35.125 | 0.006 | 39.074 | 0.006 | <0.001 |
| Follow-up number (age) | |||||
| 1 (age 19/20) | 31.105 | 0.014 | 43.868 | 0.015 | <0.001 |
| 2 (age 21/22) | 32.384 | 0.014 | 36.023 | 0.015 | N.S. |
| 3 (age 23/24) | 32.489 | 0.014 | 39.498 | 0.015 | <0.001 |
| 4 (age 25/26) | 36.844 | 0.015 | 35.373 | 0.015 | N.S. |
| 5 (age 27/28) | 36.933 | 0.015 | 39.083 | 0.015 | N.S. |
| 6 (age 29/30) | 41.149 | 0.015 | 40.528 | 0.015 | N.S. |
| Sex | |||||
| Male | 30.984 | 0.008 | 33.464 | 0.009 | <0.05 |
| Female | 39.134 | 0.008 | 44.137 | 0.009 | <0.001 |
| Race/ethnicity | |||||
| White | 40.444 | 0.008 | 42.835 | 0.008 | N.S. |
| Black | 24.630 | 0.016 | 28.624 | 0.017 | N.S. |
| Hispanic | 27.789 | 0.014 | 33.310 | 0.015 | <0.05 |
| Other | 30.776 | 0.017 | 40.127 | 0.017 | <0.001 |
| Parent education | |||||
| High school or less | 28.857 | 0.011 | 32.941 | 0.011 | <0.05 |
| Some college/more | 38.434 | 0.007 | 42.234 | 0.007 | <0.001 |
| 4-year college plans | |||||
| Not definitely | 29.415 | 0.009 | 30.927 | 0.009 | N.S. |
| Definitely | 39.775 | 0.008 | 45.427 | 0.008 | <0.001 |
| Any lifetime substance use | |||||
| Alcohol | 32.929 | 0.007 | 37.097 | 0.007 | <0.001 |
| Cigarettes | 28.362 | 0.009 | 32.242 | 0.009 | <0.01 |
| Marijuana | 28.525 | 0.008 | 33.460 | 0.009 | <0.001 |
| Other illicit drugs | 28.626 | 0.011 | 35.022 | 0.011 | <0.001 |
Note. P-value is for significance test for respective differences between rates across rows. N.S.=nonsignificant.
Figure 1.
Response Rate in 2018 by Condition for Each Follow-Up/Age Group.
Multivariable Logistic Regression Results
Main Effects.
Table 3 shows logistic regression results for predicting any response versus no response in 2018 based on experimental condition, adjusting for baseline characteristics. The Web-Push condition was associated with a 19% increase in the odds of responding compared to the Standard MTF condition, adjusting for covariates. In terms of follow-up number/modal ages, each follow-up had lower odds of responding compared to follow-up 6 (modal ages 29/30); however, the differences between follow-ups 1 and 6 and between follow-ups 5 and 6 were nonsignificant. Black, Hispanic, and other racial/ethnic participants had lower odds of responding relative to white participants. Parental education and college plans were both positively associated with responding. Participants with lifetime alcohol use at baseline were less likely to respond versus participants with no history of use, and this pattern held for lifetime cigarette use and marijuana use (the relationship for other illicit drug use was nonsignificant).
Table 3.
Multiple Logistic Regression Predicting Any Response (1) versus No Response (0) Based on Experimental Condition and Baseline Characteristics
| Model 1 | Model 2 | |
|---|---|---|
| AOR (95% CI) | AOR (95% CI) | |
| Web-push (versus standard MTF) | 1.188 (1.096, 1.287)*** | 0.856 (0.627, 1.166) |
| Follow-up number/age | ||
| 1 (age 19/20) | 0.905 (0.784, 1.046) | 0.682 (0.555, 0.839)*** |
| 2 (age 21/22) | 0.747 (0.649, 0.860)*** | 0.699 (0.573, 0.853)*** |
| 3 (age 23/24) | 0.804 (0.703, 0.918)** | 0.688 (0.568, 0.833)*** |
| 4 (age 25/26) | 0.820 (0.717, 0.938)** | 0.849 (0.703, 1.026) |
| 5 (age 27/28) | 0.900 (0.788, 1.028) | 0.868 (0.719, 1.047) |
| 6 (age 29/30 - reference) | ||
| Male | 0.684 (0.631, 0.743)*** | 0.659 (0.634, 0.800)*** |
| Race/ethnicity | ||
| White (reference) | ||
| Black | 0.506 (0.436, 0.587)*** | 0.496 (0.401, 0.614)*** |
| Hispanic | 0.690 (0.608, 0.783)*** | 0.680 (0.568, 0.813)*** |
| Other | 0.755 (0.666, 0.857)*** | 0.674 (0.560, 0.812)*** |
| Parent some college education | 1.290 (1.170, 1.422)*** | 1.228 (1.069, 1.409)** |
| 4-year college plans (definite) | 1.459 (1.337, 1.593)*** | 1.622 (1.432, 1.835)*** |
| Any lifetime substance use (age 18) | ||
| Alcohol use | 0.819 (0.740, 0.907)*** | 0.865 (0.748, 0.998)* |
| Cigarette use | 0.745 (0.670, 0.828)*** | 0.734 (0.632, 0.851)*** |
| Marijuana use | 0.784 (0.705, 0.872)*** | 0.771 (0.663, 0.897)** |
| Other illicit drug use | 0.948 (0.853, 1.053) | 1.084 (0.935, 1.256) |
| Web-push × Follow-up (joint F-test=4.80***) | ||
| × 1 (age 19/20) | 1.748 (1.309, 2.336)*** | |
| × 2 (age 21/22) | 1.142 (0.861, 1.514) | |
| × 3 (age 23/24) | 1.358 (1.040, 1.775)* | |
| × 4 (age 25/26) | 0.926 (0.708, 1.210) | |
| × 5 (age 27/28) | 1.072 (0.822, 1.397) | |
| Web-push × Male | 1.081 (0.917, 1.273) | |
| Web-push × Race/Ethnicity (joint F-test=0.90) | ||
| × Black | 1.028 (0.762, 1.384) | |
| × Hispanic | 1.021 (0.793, 1.315) | |
| × Other | 1.237 (0.958, 1.595) | |
| Web-push × Parental education | 1.100 (0.904, 1.338) | |
| Web-push × College plans | 0.810 (0.680, 0.966)* | |
| Web-push × Alcohol use | 0.899 (0.733, 1.101) | |
| Web-push × Cigarette use | 1.036 (0.838, 1.282) | |
| Web-push × Marijuana use | 1.026 (0.829, 1.268) | |
| Web-push × Illicit drug use | 0.758 (0.613, 0.937)* |
Note. Weighted n=9,946 (unweighted n=12,005); Model 1 does not include interaction terms.
p<0.05;
p<0.01;
p<0.001; p-values for interaction terms account for multiple testing
Interactions.
Interaction terms between sociodemographic characteristics and condition are shown in Model 2 (Table 3). Corroborating bivariate findings, there was a significant interaction between follow-up and condition. As indicated in Figure 1, the Web-Push condition had a stronger effect for follow-ups 1 and 3 compared to follow-up 6 (joint F-test=4.80; p<.001). We also tested the interaction between follow-up and condition using a linear measure of follow-up (results now shown), and similarly a significant interaction was found (AOR=0.90, 95% CI=0.86, 0.95). After accounting for multiple testing, there were no significant interactions between condition and demographic characteristics of sex, race/ethnicity, parental education, or college plans. There was a significant interaction between lifetime illicit drug use and condition (no other interaction between condition and lifetime substance use was significant). Figure 2 shows this interaction using the predicted probability of responding: the Web-Push condition had a stronger effect among participants who reported lifetime illicit drug use relative to participants who reported no lifetime illicit drug use at baseline.
Figure 2.
Predicted Probability of Responding According to Lifetime Illicit Drug Use by Treatment Condition
R3: Substance Use at Follow-Up
Differences in current substance use by condition using attrition weights are presented in Table 4. Vaping nicotine in the past 30 days was statistically significantly different across conditions (Web-Push=10.88% vs. Standard MTF=7.79%; p<.05). There were no other significant differences in substance use across conditions. The difference in vaping nicotine in the past 30 days became nonsignificant in a logistic regression model that adjusted for sociodemographic characteristics (Table 5).5
Table 4.
Prevalence of Substance Use in the Past 30 Days Reported at 2018 Follow-Up by Condition
| Standard MTF | Web-Push | |||
|---|---|---|---|---|
| % | SE | % | SE | |
| Alcohol | 65.655 | 0.011 | 65.716 | 0.011 |
| Cigarettes | 12.129 | 0.008 | 12.857 | 0.007 |
| Marijuana | 23.529 | 0.011 | 24.692 | 0.010 |
| Illicit drugs | 7.348 | 0.007 | 6.101 | 0.005 |
| Vaping (nicotine) | 7.790* | 0.010 | 10.875* | 0.011 |
Note. For comparisons,
=difference was statistically significant (p<0.05).
Table 5.
Logistic Regression Using Attrition Weights Predicting Past 30-Day Substance Use Based on Experimental Condition and Baseline and Follow-Up Characteristics
| Alcohol Use | Cigarette Use | Marijuana Use | Other Illicit Drug Use | Vaping | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| OR | (95% CI) | OR | (95% CI) | OR | (95% CI) | OR | (95% CI) | OR | (95% CI) | |
| Web-push (versus standard MTF) | ||||||||||
| Model 1 (bivariate) | 1.002 | (0.876–1.148) | 1.069 | (0.875–1.305) | 1.066 | (0.914–1.243) | 0.819 | (0.635–1.058) | 1.444 | (1.028–2.029)* |
| Model 2 (adjusted for sociodemographics) | 1.011 | (0.875–1.168) | 1.130 | (0.923–1.385) | 1.087 | (0.930–1.270) | 0.851 | (0.659–1.099) | 1.406 | (0.984–2.008) |
Note. Sociodemographics included race/ethnicity, sex, parents’ education, college aspirations, current student status at follow-up, highest degree at follow-up, and current employment status at follow-up.
p<0.05
Supplemental Analyses: Mode of Response among Web-Push Participants
Sociodemographic and Baseline Differences
In terms of mode of response among participants in the Web-Push condition, far more responded via web (81.20%) than via paper (18.80%). In bivariate relationships, baseline parental education, college plans, and lifetime substance use of cigarettes, marijuana, and other illicit drugs were associated with response mode among Web-Push participants (Supplemental Table 2). Specifically, participants with parents who had some college or more and participants with definite plans to attend a 4-year college were more likely to respond via web relative to participants with parents who had educational levels of high school or less, and participants with no definite plans to attend a 4-year college, respectively. Participants with no lifetime use of cigarettes, marijuana, or other illicit drugs at baseline were more likely to respond via web compared to participants who reported lifetime use of these substances at baseline. However, in a multiple logistic regression model, there were no significant relationships between response mode and baseline or concurrent characteristics (Supplemental Table 3).
Substance Use Differences at Follow-up
Substance use differences across response mode are shown in Supplemental Table 4. Using attrition weights (which account for attrition, drug user oversampling into panel, and the complex design of MTF), we found no differences in substance use across response modes except for cigarette use (Paper=18.21% vs. Web=11.48%, p<.001). This difference in cigarette use across response modes remained significant (p<0.05) in a logistic regression model that adjusted for sociodemographic characteristics and lifetime substance use (not shown).
Discussion
The Monitoring the Future Panel Study is one of the most frequently used data sets concerning the epidemiology and etiology of substance use in young adulthood in the United States. Following promising results from randomized survey research experiments on supplemental samples (5–7), a decision was made to integrate web-based data collection procedures in a random half of the panel beginning in 2018. The response rate for the Web-Push condition was higher than the Standard MTF methods condition, particularly for the first follow-up (at modal age 19/20) and the third follow-up (at modal age 23/24). After adjusting for covariates, the Web-Push condition was associated with a 19% increase in the odds of responding compared to the Standard MTF condition. We examined the effects of sociodemographic variables and baseline characteristics. Of particular importance for the study, the Web-Push condition had a stronger effect on response among lifetime illicit drug users than non-users. This relationship was driven by particularly low response rates for Standard MTF among participants who report lifetime illicit drug use and a small difference between Web-Push and Standard MTF among participants with no lifetime illicit drug use. Drug users are particularly likely to be nonresponders, so this is a potential advantage of the web-based methods. Other advantages of web-based methods include cost savings (5), and greater flexibility in creating skip patterns and targeted data collection modules, speed of response, and improved data quality (15). Finally, substance use prevalence data were very similar across the two conditions (consistent with prevalence estimates reported in the MTF panel annual report) (4).
Based on these data collected in 2018, any differences in 30-day prevalence estimates of substance use become negligible when using attrition weights and controlling for sociodemographic characteristics (i.e., sex, race/ethnicity, and parental education) assessed at baseline. Any changes in the composition of the sample due to increased response are adjusted by attrition weights. We recommend that users of the Monitoring the Future panel data construct attrition weights (e.g., as in papers 10–13) and control for sociodemographic characteristics. Beyond that, additional modeling of the web-based experiment with panel participants in 2018 is not necessary. In 2019, the experiment was repeated (data will be available by the end of 2020), and in 2020, web-push will become the new standard panel data collection protocol for those aged 19 to 30. Extensions and replications of this work within and outside of the U.S. are needed.
Conclusions
The Web-Push design is helpful in reversing the declines in response rates, especially among younger respondents; a particularly promising finding was that the response rate for the Web-Push condition at follow-up 1 (modal age 19/20) was the highest response rate compared to any other age. Our results suggest that, as long as analyses adjust for differential attrition by base year characteristics, additional adjustment for mode of data collection are likely unnecessary when using MTF panel data to examine young adult substance use.
Supplementary Material
Acknowledgements
This work was supported by the National Institute on Drug Abuse grants R01DA001411 to R. Miech and R01DA016575 to J. Schulenberg. The content here is solely the responsibility of the authors and does not necessarily represent the official views of the sponsors.
Footnotes
Conflict of Interest.
The authors declare no conflicts of interest.
The baseline survey was entirely school-based paper-and-pencil surveys with a consistent standard protocol from 1975 through 2018 (Miech et al., 2019).
Main substantive results do not change when “inactive” respondents are excluded for analyses.
MTF protocol marks respondents as “inactive” if respondents: (1) have an address that is undeliverable and no new information can be found for that respondent; (2) have not returned a questionnaire for three consecutive attempts and the address is accurate; (3) are confirmed to be deceased; (4) have requested to be removed from the study; or (5) were surveyed in the 12th grade and they were a foreign exchange student from another country and this information was not available to MTF administrators during the respondents’ 12th grade survey. Because inactive participants had not responded for three consecutive survey waves, the number of inactive participants was markedly higher for respondents whose base year was 2011 or earlier (2006 to 2011); differences across base years in the number of inactive participants ranged from 1,015 for base years 2010/2011 to less than 60 for base years 2016/2017.
Starting in 2016, permission to text was requested from the baseline sample; 41.2% of respondents in baseline years 2016 and 2017 provided a cell phone number and granted permission for us to text them. Starting in 2016, permission to text was also requested from participants from previous cohorts through change of address postcards, but respondent completion of change of address cards was very limited.
We ran supplemental analyses that used only drug oversample weights. Past 30-day cigarette use at follow-up was higher for Web-Push participants compared to the Standard MTF condition (respectively, 11.39% vs. 9.57%, p<0.05); moreover, 30-day marijuana use was higher for Web-Push participants relative to the Standard MTF condition (respectively, 22.31% vs. 20.00%, p<0.05). Vaping nicotine in the past 30 days was higher in the Web-Push condition relative to Standard MTF (respectively, 10.60% vs. 7.66%, p<.05). However, these differences were no longer significant in logistic regression models that adjusted for sociodemographics and baseline substance use.
References
- 1.Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. 2007;17(9):643–53. [DOI] [PubMed] [Google Scholar]
- 2.Keeter S, Kennedy C, Dimock M, Best J, Craighill P. Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey. Public Opin Q. 2006;70(5):759–79. [Google Scholar]
- 3.Tolonen H, Helakorpi S, Talala K, Helasoja V, Martelin T, Prättälä R. 25-Year trends and socio-demographic differences in response rates: Finnish adult health behaviour survey. Eur J Epidemiol. 2006;21(6):409–15. [DOI] [PubMed] [Google Scholar]
- 4.Schulenberg JE, Johnston LD, O’Malley PM, Bachman JG, Miech RA, Patrick ME. Monitoring the Future national survey results on drug use, 1975–2018: Volume II, College students and adults ages 19–60 Ann Arbor, MI; 2019. [Google Scholar]
- 5.Patrick ME, Couper MP, Laetz VB, Schulenberg JE, O’Malley PM, Johnston LD, et al. A sequential mixed-mode experiment in the U.S. National Monitoring the Future study. J Surv Stat Methodol. 2018;6(1):72–97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Patrick ME, Couper MP, Jang BJ, Laetz V, Schulenberg JE, Johnston LD, et al. Two-year follow-up of a sequential mixed-mode experiment in the U.S. national Monitoring the Future study. Surv Pract. 2019;12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Patrick ME, Couper MP, Jang BJ, Laetz V, Schulenberg JE, O’Malley PM, et al. Building on a sequential mixed-mode research design in the Monitoring the Future study. In review. [DOI] [PMC free article] [PubMed]
- 8.Miech R, Johnston LD, O’Malley PM, Bachman JG, Schulenberg JE, Patrick ME. Monitoring the Future national survey results on drug use, 1975–2018: Volume I, Secondary school students. Vol. 1 Ann Arbor, MI; 2019. [Google Scholar]
- 9.Bachman JG, Johnston LD, O’Malley PM, Schulenberg JE, Miech RA. The Monitoring the Future project after four decades: Design and procedures (Monitoring the Future Occasional Paper No. 82) Ann Arbor, MI; 2015. [Google Scholar]
- 10.Patrick ME, Terry-McElrath YM, Lanza ST, Jager J, Schulenberg JE, O’Malley PM. Shifting age of peak binge drinking prevalence: Historical changes in normative trajectories among young adults aged 18 to 30. Alcohol Clin Exp Res. 2019;43(2):287–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Patrick ME, Terry-McElrath YM, Lee CM, Schulenberg JE. Simultaneous alcohol and marijuana use among underage young adults in the United States. Addict Behav. 2019;88:77–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Terry-McElrath YM, Patrick ME. Intoxication and binge and high-intensity drinking among US young adults in their mid-20s. Subst Abus. 2016;37(4):597–605. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Terry-McElrath YM, O’Malley PM, Johnston LD, Bray BC, Patrick ME, Schulenberg JE. Longitudinal patterns of marijuana use across ages 18–50 in a US national sample: A descriptive examination of predictors and health correlates of repeated measures latent class membership. Drug Alcohol Depend. 2017;171:70–83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Benjamini Y, Hochberg Y. Controlling the false discovery rate - a practical and powerful approach to multiple testing. J R Stat Soc B. 1995;57:289–300. [Google Scholar]
- 15.Tourangeau R, Conrad FG, Couper MP. The science of web surveys The Science of Web Surveys. New York, NY: Oxford University Press; 2013. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.