Building on a Sequential Mixed-Mode Research Design in the Monitoring the Future Study

Abstract

Given the promise of the web push plus e-mail survey design for providing cost-effective and high-quality data (Patrick et al. 2018, 2019) as an alternative to a paper-and-pencil mailed survey design for the longitudinal Monitoring the Future (MTF) study, the current study sought to further enhance the web push condition. The MTF sample is based on US nationally representative samples of 12th grade students surveyed annually. The MTF control group for the current study included participants who completed the in-school baseline survey in the 12th grade and were selected to participate in their first follow-up survey in 2017 via mailed surveys (N = 1,222). A supplementary sample (N = ∼2,450) was assigned to one of the two sequential mixed-mode conditions. Those in condition 1 (N = 1,198), or mail push, were invited to complete mailed surveys and later given a web survey option. Those in condition 2 (N = 1,173), or enhanced web push, were invited to complete a web survey (the same as in the 2014 study, but with the addition of text messages and quick response (QR) codes and the web survey was optimized for mobile devices) and then later given a mailed survey option. Research aims were to examine response rates across conditions, as well as how responses were distributed across mode (paper, web), devices (computer, smartphone, table), and method of accessing the web survey (hand-entered URL, QR code, e-mail link, SMS link). Response rates differed significantly: the MTF control group was 34.2 percent, mail push was 35.4 percent, and enhanced web push was 42.05 percent. The higher response rate in the enhanced web push condition suggests that the additional strategies were effective at bringing in more respondents. Key estimates produced by the enhanced web push condition did not differ from those of the MTF control group.

1. INTRODUCTION

This brief research note describes the results of a follow-up to the study reported in Patrick, Couper, Laetz, Schulenberg, O’Malley, et al. (2018) and Patrick, Couper, Jang, Laetz, Schulenberg, et al. (2019). In that study, conducted in 2014, baseline participants in the annual Monitoring the Future (MTF) study (Bachman, Johnston, O’Malley, Schulenberg, and Miech 2015; Schulenberg, Johnston, O'Malley, Bachman, Miech, et al. 2018; Miech, Johnston, O'Malley, Bachman, Schulenberg, et al. 2019) were randomly assigned to one of three experimental conditions involving a web option: (i) mail push, (ii) web push, and (iii) web push + e-mail. Results were compared to the standard (mail-only) MTF protocol. Response rates for the mail push and web push plus e-mail conditions did not differ significantly from the control condition (described below). However, a substantial proportion (75 percent) of those responding in the web push plus e-mail condition completed the survey online, resulting in cost savings over the control condition.

Given the promise of the web push plus e-mail condition to achieve similar response rates to the control group with similar data quality at lower costs, the current study sought to extend this research, by further enhancing the web push condition relative to the control. Specifically, in addition to supplemental e-mail invitations (as in the early web push plus e-mail condition), we introduced text messaging and quick response (QR) codes to this condition. We also used a mobile optimized design for both experimental conditions, given the relatively high proportion of respondents (16 percent) who used smartphones to complete the survey in 2014, despite the fact that participants were encouraged to use a computer. Our expectation was that these added features would help increase the proportion of web respondents and potentially increase the overall response rate relative to the control condition. This note focuses on the effectiveness of these enhancements. Specifically, we examine the effect of the treatment groups on overall response rates, mode of response and device choice, and the quality of the data.

2. BACKGROUND

The advantages of using e-mail addresses to supplement mailed invitations in a sequential mixed-mode design are already well established (see, e.g., Millar and Dillman 2011; Cernat and Lynn 2018). Less is known about the advantage of using text messages to supplement such invitations.

First, consent rates to receiving text (SMS) messages vary among studies. Crawford, McClain, O’Brien, and Nelson (2013) found that 21 percent of US college student respondents consented, compared with 59 percent of mobile phone users in the Dutch CenterPanel (De Bruijne and Wijnant 2014), 54 percent of American Trends Panel respondents (McGeeney, Yan, Dost, and Gimenez 2016), and 72 percent of US community college students (Deal, Medway, Miller, and Miller 2017). Deal et al. (2017) also reported that text consenters were significantly more likely to complete a follow-up survey than non-consenters (72 versus 61 percent). It is possible that texting is becoming more acceptable in such circumstances, resulting in rates increasing over time.

In terms of effects on response rates, the results are mixed. In an early study among college students in Germany, Bosnjak, Neubarth, Couper, Bandilla, and Kaczmirek (2008) found that SMS prenotifications were more effective than e-mail or no prenotification (51.3 versus 40.1 versus 36.2 percent), but SMS invitations resulted in significantly lower response rates than e-mail invitations (30.1 versus 55.1 percent). Crawford et al. (2013) reported that 46.0 percent of those invited to a follow-up survey by SMS responded, while 67.2 percent of those invited by e-mail did so. They also found much higher smartphone use in the short message service (SMS) invitation group (77.6 percent of respondents) than in the e-mail group (22.6 percent). De Bruijne and Wijnant (2014) found that text message invitations did not result in a significantly higher response rate than e-mail invitations (74 versus 70 percent, p = .33); however, SMS invitations resulted in higher participation on smartphones (71 versus 57 percent). These studies all tested SMS instead of (rather than in addition to) e-mail. McGeeney et al. (2016) achieved similar response rates for text plus e-mail (82 percent) and e-mail-only invitations (81 percent). They also found significantly more smartphone completes in the text plus e-mail condition (56 percent) than e-mail only (33 percent). In a second small experiment, McGeeney et al. (2016) found a non-significant effect of using text plus e-mail reminders (61 percent) relative to e-mail-only reminders (55 percent). Toepoel and Lugtig (2018) obtained only very slightly higher response rates for text plus e-mail (78.5 percent) over e-mail-only invitations and reminders (77.0 percent). These results suggest the need for more research on text messaging to supplement e-mail invitations and communications. Our hypothesis was that adding text messages would increase the proportion of web completes, specifically on a smartphone.

QR codes are two-dimensional barcodes that can be read by image-processing systems (e.g., smartphone camera) to take participants to a website (e.g., to launch a survey). While QR codes are mentioned in the literature (see, e.g., American Association for Public Opinion Research 2014; Dillman, Smyth, and Christian 2014, pp. 303–304), studies reporting their use are rare. Gluck (2012) reported only 2 percent of respondents used a QR code to access and complete the survey. Similarly, Allen, Marlar, and Steele (2016) reported only fifty-six out of 5,000 invitees (1.1 percent) using a QR code to access the web survey. Overall, the response rate for the group with the QR code on the mailed invitation (18.3 percent) was not significantly different from the group with a uniform resource locator (URL) in the mailed invitation (17.0 percent) or both URL and QR code (18.2 percent). Harrison, Henderson, Alderdice, and Quigley (2019) report on two pilot surveys, the second of which included QR codes, along with a number of other enhancements. While the overall response rate increased from 28.7 percent in pilot 1 to 33.7 percent in pilot 2, the proportion of respondents completing the survey online increased from 6.3 to 10.4 percent, suggesting a modest effect of adding the QR code. Given this limited research, we hypothesized a modest positive benefit of adding QR codes.

Finally, mobile optimization is a key element of web survey design, especially those aimed at young adults. A number of studies have demonstrated the value of mobile optimization (see, e.g., Peterson, Griffin, LaFrance, and Li 2017; Arn, Klug, and Kolodziejski 2015; Sarraf, Brooks, Cole, and Wang 2015; Tharp 2015). We did not experimentally vary this feature but hypothesized that mobile optimization would increase the proportion completing the survey on smartphones rather than personal computers (PCs) (desktop or laptop computers) and possibly increase the overall web completion rate.

3. RESEARCH DESIGN

The present study follows the same protocol used in Patrick et al. (2018). In brief, the MTF sample is based on US nationally representative samples of 12th grade students (n = ∼15,000 per year, modal age eighteen years) surveyed annually in high schools in the forty-eight contiguous states. Each year, a random subset is selected to participate in the longitudinal portion of the study, with half of those being surveyed one year later at modal age nineteen years (Schulenberg et al. 2018). The MTF control group for the current study included participants who completed the in-school baseline survey in the 12th grade in 2016 and who were selected to participate in their first follow-up survey in 2017 (N = 1,222). Data collection is via paper questionnaires mailed to sample members. Drug users were oversampled for follow-up. Selection weights were used to adjust for this sampling procedure.

The sample for the experimental groups is a supplementary sample of those who completed the baseline survey in 2016 but were not selected for the main MTF follow-up. A random subset of these cases (N = ∼2,450) was randomly assigned to one of the two sequential mixed-mode conditions (regardless of whether or not they provided an e-mail address and/or phone number for texting). Data collection occurred in parallel with the MTF control in 2017. Condition 1 (N = 1,198) or mail push was the same as in the 2014 study. Condition 2 (N = 1,173) or enhanced web push was the same as in the 2014 study, but with the addition of up to one text message per participant and QR codes on all mailed communications, as described below. In addition, the web survey for both experimental conditions was optimized for mobile devices.

The data collection protocol is described more fully in Patrick et al. (2018) and shown in table 1. Briefly, all groups received a newsletter a few months before data collection notifying them or their selection into the study. The control group and mail push group were sent an invitation letter and the paper questionnaire. The letter for the enhanced web push included a URL and personal identification number (PIN) to complete the web survey and a QR code. About five days later, those in the enhanced web push condition who provided valid e-mail addresses were sent an e-mail containing login information; in addition, those who had provided a mobile phone number and consented to receive texts were sent an SMS with the login information. The letter for each of the groups also included a $25 check. Another week later, nonrespondents from all groups were sent a reminder postcard (with reminder e-mails also going to condition 2). Three weeks after this, the nonrespondents from the control Group were sent a reminder letter without a questionnaire, while nonrespondents from conditions 1 and 2 were sent a reminder letter with a paper questionnaire (with condition 2 also receiving another e-mail). Finally, after a round of nonresponse prompting by telephone calls, nonrespondents from all groups were sent a final paper questionnaire. In summary, the control group was sent two paper questionnaires and no mention of the web option; the mail push group was also sent two paper questionnaires and login credentials to complete the survey online at the first and final reminders; the enhanced web push additionally received up to four e-mail messages and a text message.

Table 1.

2017 Experimental Procedures by Condition

		Condition
Project week	Action	Control: MTF	Condition 1: mail push	Condition 2: web push + e-mail
Week 1	Questionnaire mailing	P, $	P, $	W, $
Week 2	Invitation e-mail			E-mail
Week 2	Reminder postcard	C	C	W, e-mail
Week 4	Reminder mailing	C	C, W	P, W, e-mail
Week 5	Reminder SMS text			Text with URL and PIN
Weeks 6–18	Resend mailings, on request	C, P	C, P, W	P, W
Weeks 6–19	Telephone calls	T	T	T
Week 17	Final mailing	P	P, W	P, W
Week 19	Final reminder			E-mail

Note.— C, control version; P, paper questionnaire; $, incentive check; W, mailed invitation to web survey (URL and PIN); T, telephone calls to non-respondents; e-mail, communication duplicated in an e-mail to those who provided e-mail addresses. PIN supplied in a second e-mail.

In the baseline survey, respondents were asked to provide an e-mail address and mobile phone number with permission to receive text messages. Across the three groups, 91.4 percent provided an e-mail address, while 57.5 percent provided a mobile phone number for texting. However, these were used only in the enhanced web push condition (condition 2), as described above.

4. RESULTS

Response rates were calculated as complete responses divided by invited sample persons, using AAPOR RR1 (American Association for Public Opinion Research 2016), weighted for oversampling of substance users. Overall weighted response rates differed significantly by condition. The weighted response rate for the MTF control group was 34.2 percent (s.e. 1.44), whereas that for mail push was 35.4 percent (s.e. 1.39) and for the enhanced web push was 42.05 percent (s.e. 1.45). The enhanced web push condition achieved a significantly (p < .05) higher response rate than either the control or the mail push. The MTF control condition had a lower response rate in 2017 than was obtained in 2014 (46.2 percent, s.e. 1.51), as did the mail push condition in 2014 (44.6 percent, s.e. 1.76) for the equivalent cohort, which likely reflects declining response rates in surveys in general (see de Leeuw, Hox, and Luiten 2018; Williams and Brick 2018). The higher response rate in the enhanced web push condition suggests that the methods employed in the enhanced web push condition were effective at bringing in more respondents. This holds across a number of demographic subgroups and baseline drug use (see table A.1).

The next analysis is to explore what modes and devices were used to complete the survey. By definition, all of the control condition responded by mail, so our focus turns to the two experimental groups. Table 2 shows the response by mode for each of these conditions.

Table 2.

Mode and Device Type by Condition

	Condition 1 Mail push			Condition 2 Enhanced web push
	N	%	SE	N	%	SE
Paper	335	79.4	1.99	109	22.3	1.89
Web	86	20.6	1.99	377	77.7	1.89
PC	61	71.0	4.91	212	56.0	2.57
Smartphone	25	29.0	4.91	159	42.4	2.56
Tablet	0	–	–	6	1.6	0.66

Note.— Weighted for oversampling of drug users. Only complete cases were included. We first tested whether mode choice (paper versus web) was significantly different by conditions using logistic regression models. Second, we tested whether device choice (computer, smartphone, versus tablet) among web respondents was significantly different by conditions using multinomial regression models. All comparisons (mode choice [paper versus web] and device choice [PC, smartphone, versus tablet]) between conditions were significant at p < .01.

First, there were relatively few breakoffs among web respondents. Only three respondents broke off in condition 1 (two on a PC and one on a smartphone). More broke off in condition 2 (eight on a PC and twenty-nine on a smartphone), but more started the survey online. Breakoff rates were significantly (p < .0001) higher in condition 2 (8.8 percent, s.e. 1.95) than condition 1 (3.4 percent, s.e. 1.39). This is line with other research finding higher breakoff rates among smartphone users (see Mavletova and Couper 2015).

Overall 51.0 percent of respondents in these two conditions completed the survey online, but only 20.7 percent of those in the mail push condition did so, compared with 77.7 percent in the enhanced web push condition. Using e-mail augmentation and pushing sample persons to the web first increased the overall response rate and substantially increased the proportion of online participants.

With respect to device use, 29 percent of web respondents (6.0 percent of all respondents) used a smartphone in the mail push condition, compared with 42.4 percent of web respondents (32.9 percent of all respondents) in the enhanced web push condition. Given that mobile optimization was not mentioned in either condition but was implemented in both conditions, we attribute this to the use of e-mail and text messaging to supplement the mailed invitations. In the 2014 study, the proportion of web respondents in the web push condition was similar (74.7 percent), but only 18.8 percent of web respondents used a smartphone (smartphone use was discouraged in 2014, given that the survey was not optimized for smartphones).

Table 3 shows the method used by respondents in the enhanced web push condition to access the survey. Only fourteen respondents used the link in the text message to access the survey, while only eleven used the QR code in the invitation letter to do so. That is, only 6.6 percent of web respondents used one of these access methods. This suggests that, while the addition of QR codes and text messages may have contributed to the higher response rates for this condition, they did so through means other than facilitating access to the survey.

Table 3.

Access Method among Web Responders in the Enhanced Web Push Condition

	PC, n (%)	Smartphone, n (%)	Tablet, n (%)
E-mail	89 (42.0)	99 (62.3)	1 (16.7)
Hand-entered URLs from postal mailing	123 (58.0)	35 (22.0)	5 (83.3)
QR	–	11 (6.9)	–
SMS	–	14 (8.8)	–
Total	212 (100.0)	159 (100.0)	6 (100.0)

A critical consideration is whether key outcomes differed by the push toward web completion. We found no significant differences between conditions 1 and 2 compared with the MTF control in the prevalence of use of alcohol, cigarettes, or marijuana in the past thirty days, or binge drinking in the past two weeks. We found higher reported use of illicit drugs other than marijuana in the past thirty days in condition 2 (8.7 percent) than in MTF (4.7 percent), but this difference was no longer significant after controlling for baseline characteristics. Comparing across conditions, we find no significant differences in substance use by mode of response (paper versus web) or device type (tables not shown). We thus conclude that the key estimates produced by the enhanced web push condition do not differ from those of the MTF.

5. CONCLUSIONS

The enhanced web push condition, in which e-mail addresses and mobile phone numbers elicited at baseline were used to augment the standard mail protocol, and where respondents were encouraged to complete the survey online, achieved a higher response rate than both the standard MTF mail-only protocol and the mail push condition (in which respondents were given the option of completing the survey online). Whereas the web push plus e-mail condition tested in the previous study (see Patrick et al. 2018) achieved a response rate ratio of 0.94 relative to the control (40.99/43.57 percent), the enhanced web push condition in the present study achieved a ratio of 1.23 relative to the control (42.05/34.20 percent), suggesting that the additional enhancements (text messages, QR codes, and smartphone optimization) together resulted in a substantial improvement to the response rate. Furthermore, a substantial majority (78 percent) of respondents in the enhanced web push condition completed the survey on the web (compared to 75 percent who did so in 2014), resulting in potentially substantial costs savings. This comes at no apparent disadvantage in terms of socio-demographic representation of the sample or in terms of prevalence estimates for key outcomes.

Limitations of the study include the fact that, as in previous studies (Patrick et al. 2018, 2019), the enhanced web push condition replicated paper mailings with digital versions and therefore participants who provided physical mailing addresses, e-mail addresses, and phone numbers for texting received more contacts than participants in the control MTF condition (this is true of previous work on e-mail augmentation; see Millar and Dillman 2011; Cernat and Lynn 2018). In addition, this study is a panel study, conditional on response to a baseline survey where contact information, including e-mail addresses and cell phone numbers, was obtained for those who were willing to give it. Only name and mailing address were required for participation; participants were not excluded if they did not provide e-mail addresses and/or permission to text message. Hence, these results are likely to generate higher rates of technology adoption than the general population. Nonetheless, we expect these results to generalize to other panel or cohort studies where contact information (e-mails and mobile phone numbers) can be solicited at baseline (see Couper and McGonagle 2019).

The collection of e-mail addresses in the first wave of a panel study for use in later waves offers clear advantages. We cannot disentangle the separate effect of adding text messaging, but our findings suggest that, while few respondents used the link in the text message to access the survey, the addition of text messaging may have served to draw attention to the e-mail invitation. The inclusion of QR codes on the invitation letters had little positive effect on response, likely because the web page was listed on the same letter and e-mails and text messages were also used. QR codes still require participant effort (to use the QR reader on their mobile device to go to the web page), while text messages and e-mails require only a click on a link. This finding is in line with previous research on QR codes (e.g., Gluck 2012; Allen et al. 2016). In this population, texts and e-mails provide the greatest ease of use. Our findings also suggest that mobile optimization further increases the proportion of respondents completing the survey online, especially on smartphones. These strategies are effective methods to increase response rates among a cohort of tech-savvy young adults.

Appendix

Table A1.

Response Rates Overall and by Baseline Characteristics, by Condition.

	Control Group (MTF)		Condition 1 (Mail Push)		Condition 2 (Web Push+Email+Text)
	%	SE	%	SE	%	SE
Total (Overall)	34.21C2	1.44	35.42C2	1.39	42.05C1,MTF	1.45
Gender
Male	29.07C2	2.06	33.90	2.05	36.24MTF	2.09
Female	40.88C2	2.13	38.36C2	2.07	49.22C1,MTF	2.13
Race/Ethnicity
White	40.57	2.06	39.62	1.95	44.21	1.97
Black	23.94	3.32	23.94	3.50	31.62	4.02
Hispanic	30.10C2	3.41	35.68	3.54	40.73MTF	3.67
Other	31.11C2	4.13	34.61C2	3.89	47.87C1,MTF	4.12
Parent Education
High School or Less	30.37C2	2.76	32.05	2.72	39.78MTF	2.89
Some College/More	37.05C2	1.79	38.60C2	1.73	44.51C1,MTF	1.75
4-Year College Plans
Not Definitely	30.26	2.28	26.28C2	2.18	35.19C1	2.41
Definitely	38.47C2	1.98	42.69	1.90	47.85MTF	1.93
Residence
Urban	33.33C2	1.62	36.63C2	1.56	43.08C1,MTF	1.63
Rural	37.30	3.10	30.36	3.04	37.91	3.19
Any Lifetime Substance Use
Alcohol	32.49C2	1.81	35.31	1.78	40.35MTF	1.86
Cigarettes	31.49	2.58	29.02	2.58	32.65	2.73
Marijuana	29.30C2	2.01	32.61	2.08	35.93MTF	2.16
Other Illicit Drugs	25.54C2	2.66	33.53	3.08	34.90MTF	3.14

Note—Drug user oversampling weights were used. All comparisons between conditions were non-significant, unless otherwise noted. ^C1 = Percentage was significantly different from that in Condition 1; ^C2 = Percentage was significantly different from that in Condition 2; ^MTF = Percentage was significantly different from that in the MTF Control Group, (p<.05).

 

Megan E. Patrick is Research Professor, University of Michigan, Institute for Social Research, 426 Thompson Street, Ann Arbor, MI 48106-1248, USA.

Mick P. Couper is Research Professor, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

Bohyun Joy Jang is Research Fellow, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

Virginia Laetz is Research Associate Lead, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

John E. Schulenberg is Research Professor, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

Patrick M. O’Malley is Research Professor, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

Jerald Bachman is Research Professor, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

Lloyd Johnston is Research Professor, University of Michigan, Institute for Social Research, 426 Thompson Street Ann Arbor, MI 48106-1248, USA.

 

This work was supported by the National Institute on Drug Abuse grants [R01DA001411 to R. Miech and L. Johnston and R01DA016575 to J. Schulenberg and L. Johnston]. The content here is solely the responsibility of the authors and does not necessarily represent the official views of the sponsors.