Two-Year Follow-up of a Sequential Mixed-Mode Experiment in the U.S. National Monitoring the Future Study

Abstract

This study examines the two-year follow up (data collected in 2016 at modal age 21/22) of an original mixed-mode longitudinal survey experiment (data collected at modal age 19/20 in 2014). The study compares participant retention in the experimental conditions to retention in the standard Monitoring the Future (MTF) control condition (participants who completed an in-school baseline survey in 12^th grade in 2012 or 2013 and were selected to participate in the first follow-up survey by mail in 2014, N=2,451). A supplementary sample who completed the 12^th grade baseline survey in 2012 or 2013 but were not selected to participate in the main MTF follow-up (N=4,950) were recruited and randomly assigned to one of three experimental conditions: 1: Mail Push, 2: Web Push, 3: Web Push + Email in 2014 and again in 2016. Results from the first experiment indicated that Condition 3 (Web Push + Email) was promising based on similar response rates and lower costs (Patrick et al. 2018). The current study examines how experimental condition and type of 2014 response were associated with response in 2016, the extent to which response mode and device type changed from 2014 to 2016, and cumulative cost comparisons across conditions. Results indicated that responding via web in 2014 was associated with greater odds of participation again in 2016 regardless of condition; respondents tended to respond in the same mode although the “push” condition did move respondents toward web over paper; device type varied between waves; and the cumulative cost savings of Web Push + Email grew larger compared to the MTF Control. The web push strategy is therefore promising for maintaining respondent engagement while reducing cost.

Introduction

Longitudinal survey designs provide an important resource for data on development by gathering data on the same people over time (Binder 1998). Retention in these studies has been decreasing in recent years, which raises concerns about the representativeness of the longitudinal samples and possible resulting bias in estimates of long-term outcomes (Lynn 2009, 2018).

In the U.S. national Monitoring the Future study, we explore data collection mode as one avenue to boost sample retention over time. The main question in the experiment reported here is whether the mixed-mode experiment results in differential attrition at the second follow-up survey. That is, are there differences in participant retention at the second follow-up survey based on data collection mode approach (i.e., web push vs. mail push) or by the type of response (web or paper) at the first follow-up survey? Longer-term gains in retention or cost savings from mode changes, regardless of concurrent cross-sectional changes based on study design, would advance the understanding of mode change benefits in longitudinal studies.

This research note examines the longitudinal outcomes of a mixed mode experiment (Patrick et al. 2018) with the national Monitoring the Future (MTF) study. MTF includes a longitudinal component with a nationally-representative sample of 12^th grade students followed through adulthood (Schulenberg et al. 2017, Miech et al. 2017). In 2014, the standard first follow-up (at modal ages 19/20) using mailed paper questionnaires was compared to three experimental mixed mode conditions, described below: 1: Mail Push, 2: Web Push, 3: Web Push + Email. Results from the first experimental follow-up data collection indicated that Condition 3 (Web Push + Email) was promising based on similar response rates and lower costs when compared to the other two conditions (Patrick et al. 2018). At the second follow-up at ages 21/22 in 2016, respondents were assigned to the same condition they received in the first follow-up. The resulting data should allow us to examine how experimental mixed-mode condition prospectively affects participation at later waves, including overall retention, response mode, type of web response (i.e., device), and cumulative cost across two waves of data collection.

Mixed-mode data collection is increasingly being adopted for both cross-sectional and longitudinal surveys (Couper 2017). While there is a growing literature on the effect of mixing modes on response rates and data quality in cross-sectional surveys, there is a paucity of research on longitudinal surveys. Bianchi, Biffignandi, and Lynn (2017) report on the introduction of mixed-mode (web and face-to-face) data collection in Understanding Society, and Al Baghal and Kelley (2017) focus on changes in mode preference in the same study. Similarly, Voorpostel and Ryser (2011) look at the introduction of web to the Swiss Household Panel. However, we know of no studies looking at web and mail mixed-mode surveys across waves of data collection. This note aims to address this gap in the literature.

Research Aims

Research aims for the current study were to examine: (1) whether experimental condition (randomly assigned in 2014 and continued in 2016) and the type of response in 2014 predicted response in 2016; (2) how response mode and device type changed from 2014 to 2016; and (3) cumulative cost comparisons by condition for the 2014 and 2016 data collections.

Method

Monitoring the Future Main Study

MTF is an annual series of surveys of nationally representative samples of 12^th grade students in the 48 contiguous United States (n≈ 15,000 per year, modal age 18 years; Miech et al., 2017). For the MTF follow-up study, approximately 2,450 students from each 12^th grade cohort are randomly selected for mailed follow-up surveys; half of the sample begin the survey one year after high school at modal age 19 and the other half two years after high school at modal age 20 (Schulenberg et al. 2017). Subsequent follow-up surveys are conducted biannually, all via mail. Prior to the first follow-up survey at modal ages 19/20, participants are contacted several times. They are sent: (1) a notification letter (informing them of their selection for follow-up), (2) a newsletter (including a cover letter, a summary of selected MTF study results, (3) a change of address card for updating contact information), (4) an advance letter (alerting them that the survey is coming soon), and (5) then the actual questionnaire mailing. A paper questionnaire is then mailed to each participant along with a pencil, prepaid return envelope, and a check for $25. Reminders are sent one week later (postcard) and three weeks later (letter) to the participants; nonresponse phone calls are made one week after the second reminder, and a final mailing with a second copy of the paper questionnaire is sent about six weeks after the first questionnaire mailing. Subsequent follow-up surveys follow the same protocol, except that the notification letter and newsletter are mailed together and that no advance letter is sent out.

The MTF control group includes participants who completed their in-school baseline survey in 2012 or 2013, and were mailed the first follow-up survey in 2014 and the second follow-up survey in 2016 (N=2,451). MTF oversamples drug-users in 12^th grade to ensure the representativeness of drug-using populations in the follow-up surveys; weights are used to compensate for the differential sampling probabilities (Bachman et al. 2015, Schulenberg et al. 2017).

Experimental Design

A supplementary sample of participants who completed the MTF baseline year survey in 2012 or 2013 but were not randomly selected for the main MTF follow-up was used for the experimental conditions (N=4,950). Participants in the supplementary sample were randomly assigned to one of three experimental conditions in 2014 (as described in Patrick et al. 2018), and this assignment was maintained for the second experimental data collection in 2016 (at modal ages 21 or 22).

Condition 1 (the Mail Push Condition) initially mirrored the mailing procedure of the MTF main study (i.e., selection letter, newsletter, advance letter, paper questionnaire with a check for $25, and reminder postcard at the same time as the MTF main study). The reminder letter in Condition 1, however, provided the option to complete the survey online; web survey login information (i.e., survey URL and a personal identification number [PIN]) was included. Nonresponse phone calls also provided information about both the paper and web response modes. A final mailing included information about the web survey option in addition to a second copy of the paper questionnaire.

In Condition 2 (the Web Push Condition), participants were sent the same selection letter and newsletter as the MTF control group without an indication of a paper survey. An advance letter in Condition 2 stated that they would be sent an invitation to complete an online survey. Instead of a paper questionnaire, participants in Condition 2 were sent web survey login information (i.e., survey URL and PIN) and a check for $25. The reminder postcard sent by mail was the same as the MTF main study except it requested completion of an online survey. The reminder letter included the online survey login information but also a paper questionnaire as an additional option. Nonresponse phone calls provided information about both the paper and web response modes. A final mailing also included a paper questionnaire and online survey information.

Condition 3 (the Web Push+ E-mail Condition) included the identical procedures to Condition 2 with the addition of an e-mailed version of the advance letter, web survey login information, reminder postcard, and reminder letter. Participants selected for this condition who did not provide an email address (23%) or who provided a usable e-mail address (i.e., email was undeliverable, 13%) received the same protocol as Condition 2. Further details on the experimental design are available elsewhere (Patrick et al. 2018).

At the second follow-up in 2016, at modal ages 21/22, participants were assigned to the same experimental condition as at the first follow-up (regardless of their response mode at ages 19/20 in 2014). All eligible respondents from 2014 were invited to participate in 2016, whether or not they responded previously.

Measures

To examine response in 2016 to the second follow-up, we controlled for baseline characteristics (from 12^th grade reports via in-school surveys) that were available for both respondents and nonrespondents. These include class year (1=2013, 0=2012), gender (1=male, 0=female), race/ethnicity (Black, Hispanic, Other, and White [reference category]), parent education (1=at least one parent had at least some college education, 0=no parent had college education), four-year college plans (1=will definitely graduate from a four-year college program, 0=probably will, probably won’t, or definitely won’t), living in urban area (1=urban, 0=rural), and lifetime substance use (i.e., alcohol use, cigarette use, marijuana use, and illicit drug use other than marijuana) in 12^th grade (each separately coded as 1=yes, 0=no).

Mode choice included paper or web. Device type was coded from the paradata available through the software used for web survey implementation (i.e., DatStat Illume). Smartphones included Android, Windows, and iOS phones. Android tablets and iPads were coded as tablets. Personal computers (PCs) included laptop and desktop computers running full-screen browsers.

Estimated cumulative costs included survey mailing materials, postage, staff labor (including fringe benefits), incentives, paper survey scanning, and nonresponse calling for both the first and second follow-up surveys. The calculation did not include 1) costs associated with survey design (for the web survey or the paper survey) and 2) staff time expected to be approximately equal across conditions (i.e., respondent tracking and data cleaning). The costs at both follow-up surveys in 2014 and 2016 were summed for the total cumulative costs.

Results

Table 1 shows response rates by experimental conditions at both waves. At the first follow-up survey, in 2014, the response rate of Condition 2 (38.0%; Web Push) was significantly lower than the MTF control group (43.6%) and Condition 1 (42.5%; Mail Push). At the second follow-up survey, in 2016, no statistically significant differences in response rates were found by conditions; 36.4–38.3% of participants in each condition responded. Among those who did not respond in 2014, 14.8–18.1% responded in 2016; of those who did respond in 2014, 64.3–68.4% responded in 2016. The only significant difference by condition was that 2014 non-responders had a higher 2016 response rate in Condition 2 (18.1%) than in the MTF control group (14.8%).

Table 1.

Response rates by experimental conditions.

	MTF control		Condition 1 (Mail Push)		Condition 2 (Web Push)		Condition 3 (Web Push+Email)
	%	SE%	%	SE%	%	SE%	%	SE%
Response rates each year among all eligible
2014	43.6C2	1.1	42.5C2	1.3	38.0MTF,C1	1.2	40.9	1.2
2016	37.9	1.1	37.2	1.2	36.4	1.2	38.3	1.2
2016 response rates by 2014 response status
Not responded	14.8C2	1.0	16.6	1.3	18.1MTF	1.3	17.0	1.3
Responded	66.1	1.6	64.3	1.9	65.9	2.0	68.4	1.9

Notes: Superscript indicates significant differences in response rates by condition (MTF=Monitoring the Future Control, C1=Condition 1, C2=Condition 2, C3=Condition 3). Only complete cases were counted; partial web cases (3.45% of 2014 web responses [n=34] and 4.15% of 2016 web responses [n=42]) were not included. SE=standard error.

Table 2 reports odds ratios of response at the second follow-up survey in 2016 (at modal ages 21/22). Web response in 2014, compared to paper response in 2014, was associated with significantly greater odds of response at the second follow-up survey in 2016. The experimental conditions were not significantly associated with response in 2016, controlling for 2014 response mode. Regarding other control variables measured at baseline (modal age 18), Blacks and Other racial/ethnic groups had significantly lower odds of response in 2016 compared to Whites. Those who had 4-year college plans at age 18 were significantly more likely to respond in 2016. Lifetime use of illicit drugs other than marijuana at age 18 was significantly associated with lower odds of response in 2016.

Table 2.

Multivariate logistic regression predicting response vs. nonresponse at age 21/22 in 2016 (among 2014 respondents)

	AOR† (95% CI) p
Web response in 2014 (vs. paper response)	1.4 (1.1 – 1.8)*
Condition (C) (vs. MTF)
C1 [Mail Push]	0.9 (0.7 – 1.1)
C2 [Web Push]	0.8 (0.6 – 1.1)
C3 [Web Push+Email]	0.9 (0.6 – 1.2)
Form (vs. Form 1)
Form 2	1.1 (0.9 – 1.4)
Form 6	1.1 (0.9 – 1.5)
Senior class year 2013 (vs. 2012)	0.9 (0.8 – 1.1)
Male	0.9 (0.7 – 1.1)
Race/ethnicity (vs. white)
Black	0.5 (0.4 – 0.7)***
Hispanic	0.8 (0.6 – 1.1)
Other	0.7 (0.5 – 0.9)**
Parent(s) college education (vs. less)	1.1 (0.9 – 1.4)
Definite 4-year college plans (vs. less)	1.5 (1.2 – 1.8)***
Living in urban area (vs. rural)	1.2 (0.9 – 1.5)
Lifetime substance use (age 18)
Alcohol use (vs. no use)	0.9 (0.8 – 1.2)
Cigarette use (vs. no use)	0.9 (0.7 – 1.2)
Marijuana use (vs. no use)	0.8 (0.7 – 1.1)
Other illicit drug use (vs. no use)	0.7 (0.5 – 0.8)***
Used n	2,639
Total N	2,963

^†Adjusted odds ratio,

^*p<.05,

^**p<.01,

^***p<.001. MTF=Monitoring the Future Control, C1=Condition 1, C2=Condition 2, C3=Condition 3

Table 3 presents response mode choices in 2014 and 2016 among those who responded in both waves. In 2016, 56.0% responded by web overall, with significant differences by condition: only 12.0% in Condition 1 responded by web compared to 76.0% in Condition 2 and 82.2% in Condition 3. Most respondents chose the same mode at both waves; 72.4% of 2014 paper responders and 81.4% of 2014 web responders used the same mode in 2016. However, there was a noticeable effect of the “push” strategy, with greater percentages of 2014 paper responders moving to the web in 2016 in Conditions 2 and 3 and more web responders in 2014 moving to paper in 2016 in Condition 1.

Table 3.

Mode choice in 2016 by mode choice in 2014 among responders at both waves

							By Condition
		Total		Condition 1 (Mail Push)		Condition 2 (Web Push)		Condition 3 (Web Push+Email)
2014	2016	%	SE%	%	SE%	%	SE%	%	SE%
Paper responders	Paper responders	72.4	1.9	89.4	1.6	44.8	4.5	39.9	5.3
	Web responders	27.6		10.6		55.2		60.1
Web responders	Paper responders	18.6	1.5	79	5.4	13.6	2.2	12.2	1.8
	Web responders	81.4		21		86.4		87.8
Total	Paper responders	44	1.4	88	1.6	24	2.2	17.8	1.9
	Web responders	56		12		76		82.2

Note: Those who responded in both 2014 and 2016 were included. Total n=1,255 (Condition 1 n=435, Condition 2 n=380, Condition 3 n=440). SE=Standard Error.

Table 4 shows device types among web responders at both follow-up waves. In 2016, 77.5% of web respondents used a PC while 19.7% used a smartphone, and 2.9% used a tablet. Most PC responders in 2014 largely responded by PC in 2016 (80.9%); more than 70% of tablet responders and 60% of smartphone responders in 2014 responded by other devices in 2016.

Table 4.

Device type in 2016 by device type in 2014 among web responders at both waves

2016	Total		2014
			PC		Tablet		Smartphone
	%	SE%	%	SE%	%	SE%	%	SE%
PC	77.5	1.8	80.9	1.9	35.4	11.6	59.9	6.8
Tablet	2.9	0.7	1.9	0.6	25.8	11	4.1	2.8
Smartphone	19.7	1.7	17.2	1.8	38.8	12.2	36	6.7
Total	100.0		100.0		100.0		100.0

Note: Web responders in both 2014 and 2016 were included. Total n=537. PC=personal computer. SE=Standard Error.

The total cost of each experimental design across the two waves was calculated as the cost per targeted participant at both waves, i.e., the cost divided by the number of targeted participants, and as the cost per responder at both waves, i.e., the cost divided by the number of people who actually responded (Table 5). Condition 3 saved 10.7% of the cost per responder (and 11.2 % of the cost per targeted responder), compared to Condition 1 which is similar to the MTF Control procedures. Condition 2 incurred a 3.4% greater cost per responder (although a 5.4% lower cost per targeted responder) than Condition 1.

Table 5.

Estimated cumulative cost differences for 2014 and 2016 follow-ups by condition

	Cost per targeted sample person	Cost per respondent
Mail Push (Condition 1)	--a	--
Web Push (Condition 2)	−5.4%	3.4%
Web Push + Email (Condition 3)	−11.2%	−10.7%

^aCosts are compared to Condition 1, which is most similar to the Monitoring the Future Control design and procedures.

Discussion

We examined the effect of different mail and web mixed-mode designs across two waves of data collection following baseline. The 2014 experiment found that the response rate was lower in the Web Push condition, whereas the Mail Push and Web Push+Email conditions did not differ significantly from the MTF Control. In the 2016 wave, using the same mode assignment as in 2014, no significant differences in response rates by condition were found, but the Web Push+Email did slightly better than the control. Further, the Web Push conditions brought in significantly more responders among those who had not responded in 2014.

We also found that responders tended to continue using the same mode in both waves (77.4% of 2014 paper responders also used paper in 2016; 81.4% of 2014 web responders used web again in 2016), but the Web Push conditions moved more paper responders to the web in 2016, and the Mail Push condition moved more web responders to paper in 2016, suggesting that the choice of mode is influenced by the invitation in the current wave. Among web responders, we found considerable device switching between waves, particularly for mobile device users consistent with Lugtig and Toepoel (2016). Specifically, those who used a PC in 2014 tended to use a PC again in 2016, but more than half of those who used a mobile device (smartphone or tablet) in 2014 used a different device in 2016.

Finally, the Web Push+Email condition resulted in cost savings over the mail control group across two waves of follow-up data collection. The primary reason for this is the increased proportion of responders using the web mode, consistent with the emerging literature on this topic (see Cernat and Lynn 2018, Millar and Dillman 2011). The finding points to the advantages of soliciting email addresses at baseline for use in follow-up surveys.

Strengths of the study include the longitudinal extension of a randomized experimental design based on a nationally-representative sample of high school students in the U.S. Limitations include that the original study does not include high school dropouts and the comparable cost estimates for the MTF Control group were not available. In sum, based on this experiment using nationally representative samples of high school graduates, we find continuing benefits of using a web push strategy for follow-up data collection and of using email addresses (where available) to invite participants.