Use of Web and Phone Survey Modes to Gather Data From Adults About Their Young Adult Children: An Evaluation Based on a Randomized Design

Abstract

Mode effects on responses to survey items may introduce bias to data collected using multiple modes of administration. The present study examines data from 704 surveys conducted as part of a longitudinal study in which parents and their children had been surveyed at multiple prior time points. Parents of 22-year-old study participants were randomly assigned to one of two mixed-mode conditions: (a) Web mode first followed by the offer of an interviewer-administered telephone mode; or (b) telephone mode first followed by the offer of the Web mode. Comparison of responses by assigned condition on 12 measures showed one statistically significant difference. Analyses that modeled differences by completed mode and the interaction between assigned condition and completed mode found significant differences on six measures related to completed mode. None of the differences indicated that more socially desirable responses were given in interviewer-administered surveys.

Introduction

It has become increasingly common to gather survey data through self-administered Web surveys (Couper, 2000; Dillman, 2000; Shih & Fan, 2008). Administering surveys over the Internet makes it possible to reduce costs compared to interviewer-administered in-person or telephone interview modes (Fricker et al., 2005; McMorris et al., 2009; Roberts, 2007; Schonlau et al., 2009). Also, the Web mode allows respondents flexibility to complete surveys when it is most convenient for them, rather than having to schedule an appointment with an interviewer. Response rates and coverage with Web surveys, however, can be a challenge since some study participants may lack Web access (Fricker et al., 2005; Greenlaw & Brown-Welty, 2009; Weisberg, 2005). A mixed-mode approach to data collection, in which respondents are given the option of completing their surveys in a Web self-administered survey or in a telephone interview, may be a promising strategy that realizes the cost savings of the Web mode while achieving higher response rates associated with a phone survey (de Leeuw, 2005; Dillman & Christian, 2005; Greenlaw & Brown-Welty, 2009; Groves, 2004; Roberts, 2007; Weisberg, 2005).

The danger of mixed-mode approaches to data collection is that the mode in which a respondent completes a survey may affect the respondent’s answers. Dillman and colleagues (1996), reviewing research on differences in responses to telephone and mail surveys, identified seven types of mode differences that can arise due to the presence or absence of the interviewer, the dependence on visual or aural communication, and the interviewer or respondent controlling the pace of the survey and the ordering of the items. Similar mode differences might be expected between telephone and Web surveys which, like mail surveys, are self-administered, usually rely on respondents reading the questions, and commonly allow for respondents to scan ahead in the survey and answer items at their own pace (Greene, Speizer, & Wiitala, 2008).

While there are a number of different types of effects one might expect, the most commonly reported mode effect underlying differences in responses to interviewer- and self-administered surveys is due to social norms being activated by interaction with an interviewer (Dillman, Sangster, Tarnai, & Rockwood, 1996; Greene et al., 2008; Maguire, 2009; Tourangeau & Yan, 2007). This may be a matter of increasing rates of agreement for items that ask respondents to agree or disagree with statement, but are more generally found in social desirability effects. For instance, as reported by Greene and colleagues (2008), respondents may be more likely to report frequent exercise or better eating habits when giving answers to an interviewer over the telephone than if the questions are self-administered. Social desirability effects could be confounded further if certain types of respondents are more likely to choose one mode over another (Fricker et al., 2005; Greene et al., 2008). For instance, if less affluent respondents are more likely to complete the survey by telephone than Web, and telephone surveys result in more socially desirable responses, estimates of the relationship between socioeconomic status and survey responses will be biased. Adjusting for demographic characteristics and common measures of socioeconomic status may not eliminate the bias (Schonlau et al., 2009).

Mode effects of interviewer-administered modes of data collection are well documented in cross-sectional studies (Dillman et al., 1996; Greene et al., 2008; Maguire, 2009; Tourangeau & Yan, 2007); they have received less attention in longitudinal studies where the same respondents have been interviewed multiple times. In the case of longitudinal studies, it may be that participants, having experienced no violation of confidentiality in prior survey waves, more willingly report undesirable behaviors or attitudes, even when giving their answers out loud to an interviewer. On the other hand, participants in longitudinal studies are commonly asked to give extensive locating information so that they can be found for future data collection and have repeated contacts by the study. They may therefore feel less anonymous than participants in cross-sectional studies and, for this reason, be more wary of revealing embarrassing information. Previous research using data from the young adult children in the current study found little evidence of mode effects in a comparison of responses to questions about drug use and sexual behavior by those randomly assigned to a Web-first condition with an in-person follow-up option versus in-person first with Web follow-up (McMorris et al., 2009). The sensitive items, however, were self-administered in the in-person mode, with respondents reading these items off a laptop screen and entering their own answers. A study by Greene and colleagues (2008) compared responses on lifestyle questions for individuals randomly assigned to Web-first and phone-first conditions, all of whom had completed a survey via the Web one year earlier. They found evidence of more socially desirable responses given by those who complied with their phone assignment than those who complied with their Web assignment. However, no study has examined phone versus Web differences in a long-term longitudinal survey where participants have previously completed multiple waves of surveys.

The present study reports on a randomized trial of a mixed-mode approach to data collection to examine potential bias associated with Web and phone administration modes. The sample consisted of parents of 22-year-olds, the primary targets of the longitudinal study. Parents had participated in multiple, annual telephone surveys regarding their children when their children were between the ages of 6 and 18. When their children were age 22, parents were randomly assigned to one of two mixed-mode conditions: (a) a Web-first group asked to complete a survey via the Internet and later offered the opportunity to complete the survey over the phone, or (b) a telephone-first group in which they were first offered the interviewer-administered telephone survey, with the Web follow-up. The guiding hypothesis for this study, based on prior studies of mode bias, is that telephone surveys will result in more socially desirable responses; in other words, parents responding over the phone will report more positive and fewer negative behaviors or characteristics about their children and themselves than those completing the survey via the Web.

Method

Participants

The data used for this study come from the Raising Healthy Children (RHC) project, a longitudinal study of the development of substance use, other problem behavior, and positive behavior. Nested within the project is an experimental test of a social development approach to prevention, with half the sample assigned to the intervention condition (for more details of the intervention see Brown et al., 2005; Catalano et al., 2003; Haggerty et al., 2006). The study population originally consisted of families of first- and second-grade students in 10 suburban public elementary schools in a Pacific Northwest school district. Over the course of 2 years, families of 1,040 students from two grade cohorts were enrolled in the project. These students and their parents were interviewed each subsequent spring until students normally progressing reached the 12th grade. Parent surveys during this period of the project were interviewer administered over the telephone. The content areas of these earlier parent interviews were parenting practices, family cohesion and conflict, and, when children were in elementary and middle school, externalizing behaviors and social competencies of the children. After high school, the youth participants were surveyed twice a year during the 2 years after high school and thereafter annually in the spring. The McMorris and colleagues study (2009) that used data from the RHC project and examined Web versus in-person mode differences in survey responses was based on surveys administered to the older cohort of RHC youth participants when they were in the fall of their 1st year after high school. When the youth participants were approximately age 22 (fall of 2007 for the older grade cohort; fall of 2008 for the younger cohort), their parents were surveyed to provide information on their children’s psychosocial health, progress on various developmental tasks of early adulthood, and the parents’ own income and substance use.

By the time their children were age 22, parents of 113 youth participants had withdrawn from the study during prior waves of data collection. Parents of the remaining 927 youth participants were randomly assigned to either a Web-first or telephone-first condition. Of those assigned, there were 47 sibling pairs and two families with three children in the study, resulting in 878 parents being assigned (438 assigned to the Web-first and 440 assigned to the phone-first mode).

In both conditions, parents were first contacted with an advance letter that told them about the opportunity to do a survey. For those assigned to the Web-first condition, the letter invited them to take the Web survey, instructed them how to access the survey, and provided the URL and a personal access code. They were also given a phone number and email address to contact the project manager if they were unable to complete the survey using the Internet for whatever reason. This group was sent a follow-up letter a week later reminding them to complete their surveys via the Web. Approximately 3 weeks after the advance letter, another reminder letter was mailed that mentioned that a phone interview option would be available and that an interviewer would call. Approximately 30 days after the advance letter, interviewers contacted parents who had not yet completed a survey and asked them to do the survey either on the Web or over the telephone.

For those in the phone-first condition, the advance letter told them an interviewer would be calling soon to schedule a time to complete the survey over the phone. Two days after the advance letter, interviewers began calling and emailing the parents in the phone-first group to schedule phone interviews. Approximately 30 days after the advance letter, another letter was sent to the phone-first group offering the option of completing via the Web and providing the URL and a personal access code. Interviewers continued to call and email the phone-first group, but began offering the Web option in addition to the phone option.

For both the younger and the older cohorts in the RHC project, the field period lasted approximately 3 1/2 months. In both conditions an incentive of $30 was offered for completing a survey, regardless of the completed mode. In both conditions, 352 parents completed a survey (80% of those assigned for both conditions). The average number of days after the start of the field period until a survey was completed was slightly greater in the Web-first condition (Web-first: mean = 40.59 days, sd = 29.73; phone-first: mean = 36.58, sd = 22.14; t = 2.03, p = .043).

The 704 parents who completed the survey constitute the analysis sample for the current study. For those with more than one child in the study, we use data from the survey about their oldest child. Characteristics of the analysis sample and those parents who were assigned to a condition but did not complete the survey (N = 174) are shown by assigned condition in Table 1. Characteristics of completers in the two conditions were similar; thus, there were no significant differences within the analysis sample between the two conditions with respect to any of the background characteristics examined. The percentage of participants whose child was male or who were in the intervention condition were similar for completers versus noncompleters in both the Web-first and phone-first conditions. However, a higher percentage of noncompleters than completers were nonwhite in both the Web-first (X² = 18.95, p < .001) and phone-first condition (X² = 17.49, p < .001). Also, a higher percentage of noncompleters than completers had reported a maximum household income during their child’s high school years of under $40,000 in both conditions (Web-first: X² = 9.95, p < .01; phone-first: X² = 7.52, p < .001). A higher percentage of noncompleters than completers in the phone-first condition were in the older cohort (X² = 7.92, p < .01), while the difference in cohort between noncompleters and completers was nonsignificant in the Web-first condition.

Table 1.

Characteristics of Noncompleters and Completers by Assigned Condition

	Assigned Web-first		Assigned Phone-first
	Noncompleters N = 77 n (%)	Completers N = 345 n (%)	Noncompleters N = 71 n (%)	Completers N = 346 n (%)
Child male	39 (45)	193 (55)	56 (64)	187 (53)
Intervention condition	49 (57)	183 (52)	49 (56)	193 (55)
Older cohort	43 (50)	180 (51)	57 (65)	169 (48)
Nonwhite	31 (36)	54 (15)	29 (33)	49 (14)
Income < $40,000 per year	26 (34)	61 (18)	25 (35)	70 (20)

Of the analysis sample, 81% were the biological mother of the respondent, 15% were biological fathers, and 4% were some other type of guardian (e.g., step or adoptive parent, grandparent, aunt or uncle). The analysis sample was 85% White, 3% African American, 5% Asian, 4% Latino/a, and 3% Native American.

Both phone and Web respondents who chose not to answer a question were offered a ‘don’t know,’ ‘refused,’ or ‘not applicable’ option. This resulted in the rate of skipped items being low for both phone and Web surveys, with no difference by assigned condition or completed mode. Furthermore, respondents received their incentive only on the completion of their survey. Consequently, there were only two partial Web surveys where the parent broke the survey off in the middle and then did not complete the survey. Both were treated as noncompletes for the current study and were excluded from the analysis sample.

The survey took approximately 40 minutes to complete in the phone mode and 32 minutes in the Web mode. The phone interview was conducted with Computer Assisted Telephone Interview (CATI) software. Interviewers asked questions aloud and respondents answered by either giving the number of the response option or giving the actual response (e.g., “yes” or “no”). For the Web survey, most items were asked in a table format, with two to six items with shared response options presented on a single screen. The order in which response options were presented to respondents (i.e., read aloud in phone surveys and presented on the screen in Web surveys) was the same across modes.

Prior studies (e.g., McMorris et al., 2009) have documented the cost savings of conducting surveys by the Web. In this case, the difference in cost is captured by the interviewer time required to conduct telephone interviews, since locating, programming, and equipment costs were similar for the two modes.

Measures

In testing for mode effects, we used measures that reflect the main content areas of the survey. The measures can be considered susceptible to social desirability effects in that they ask parents to report on negative or positive behaviors of their children or of themselves. The Chronbach’s alphas reported below are based on data from the analysis sample of 704. Four scales of psychosocial problems of the 22-year-old children were from the Young Adult Behavior Checklist (Achenbach & Rescorla, 2003). Each of these scales was based on the average of scores on items that offered response options ranging from 0 (never/not true) to 2 (often/very true). The four scales were Depression Problems (13 items, alpha = .84), Anxiety Problems (7 items, alpha = .75), Avoidant Personality Problems (7 items, alpha = .76), and Antisocial Personality Problems (20 items, alpha = .90). The survey included items asking parents how their child was doing in 17 developmental tasks (e.g., finding employment, academic achievement, finding or maintaining healthy relationships with peers, staying in touch with family). Each item offered a response option ranging from 1 (not good at all) to 6 (very good), and the mean was taken across items to form a scale of progress in meeting developmental tasks. While covering distinct developmental tasks, the scale had high internal consistency (alpha = .95). Another question asked about financial support of child: “How much of your child’s total expenses did you and/or your child’s other parent or parent figure pay in the past year?” Response options ranged from 1 (none) to 5 (all).

In addition to questions about the child, parents were asked to report on their own substance use. We examined three dichotomous measures of substance use for whether respondents reported using alcohol, marijuana, or cigarettes in the prior month. Also, parents were asked a series of questions about how many times the child’s biological mother (who was, in most cases, the parent respondent) and biological father (in most cases the spouse of the parent respondent) had engaged in 14 types of antisocial or risky behavior before the age of 15. Response options ranged from 1 (never) to 4 (> 10 times). The mother’s age 15 antisocial behavior scale had an alpha of .85; the father’s age 15 antisocial behavior scale had an alpha of .88. Household income was based on an item which offered an 18-category response option for household income in the prior year ranging from 1 = ‘under $10, 000’ to 18 = ‘$200,001 and over’.

In examining predictors of completed mode and as covariates in analyses of mode differences of responses, we included variables related to the design of the RHC project, demographics, and an indicator of family socioeconomic status. Specifically, these variables included experimental condition, whether the child was in the older or younger grade cohort, gender of the parent respondent, gender of the child, and household income when the child was in high school (based on parent surveys when the child was in Grades 9 – 12, maximum on the same 18-category income measure of income ranges described above).

Missing Data

There were some cases with missing data on some measures. For measures related to the child’s behavior, where missing data was primarily due to parents responding “don’t know” to some items, rates of missing data were low (< 2%). For the measures of mother’s and father’s behavior at age 15, rates were somewhat higher (5% and 13%, respectively) due to parents reporting “don’t know” regarding the behavior of the other parent. Thirteen cases (2% of the analysis sample) lacked data on the parent’s household income during high school. In analysis of co-variance (ANCOVA) models that adjusted for covariates, these 13 cases were dropped.

Results

Predictors of Completed Mode

Mixed-mode data collection can result in differences in the types of respondents who complete in a given mode, for instance, due to differences in Web access or willingness to comply with the mode that is first offered. In order to understand selection effects into completed mode, logistic regression models were used to examine predictors of completed mode, with completing by Web treated as the dependent variable. Separate models were run for those assigned to the Web-first and phone-first conditions.

Of those assigned to the Web-first condition, 250 (71%) completed their survey by Web and 102 (29%) completed their survey over the telephone. Of those assigned to the phone-first condition, 263 (75%) completed by phone and 89 (25%) completed by Web. Table 2 presents the results of logistic regression models predicting completing the survey by Web for the those assigned to Web-first condition and for those assigned to phone-first condition. The results suggest that in both conditions the Web mode selected for different types of respondents, although the pattern of selection effects was somewhat different for the Web-first and phone-first conditions. For those assigned to the Web-first condition, parents of females and higher income parents were more likely to comply with their condition assignment and complete by Web. For those assigned to the phone-first mode, parents who had children in the younger cohort were more likely to not comply with their assigned condition and complete by Web.

Table 2.

Estimates for Logistic Regression Models Predicting Completion of Survey via the Web for Those Assigned to Web-first and Phone-first Conditions

	Web-first (n = 352)		Phone-first (n = 352)
	b (se)	Odds Ratio	b (se)	Odds ratio
Male child	−.53* (.25)	0.59	−.37 (.25)	0.69
Male parent respondent	.66 (.36)	1.93	−.31 (.38)	0.74
Family income	.38** (.09)	1.46	.05 (.08)	1.05
Older cohort	−.24 (.25)	0.78	.55* (.26)	1.73
Intervention condition	.23 (.25)	1.25	.13 (.25)	1.13
Constant	−.75 (.67)	0.47	−1.68* (.70)	0.19
Nagelkerke Pseudo –R2	.11		.04

^*p < .05,

^**p < .01;

b = unstandardized coefficient, se = standard error.

Comparison by Assigned Condition

As a first strategy to assess mode effects on survey responses, comparisons were made by assigned condition, regardless of the mode in which the survey was actually completed. This strategy captures mode effects that operate through assigned condition, increasing the probability of a participants completing their survey in a given mode. Since this analysis was consistent with the randomized design, initially no covariates were included in the analyses, and comparisons were based on t-tests for continuous outcomes and crosstabs for dichotomous outcomes. Secondary analyses were done with ANCOVA and logistic regression models controlling for covariates (experimental condition, grade cohort, gender of child, gender of parent, and family income) with the assumption that inclusion of these covariates adjusted for selection bias that might have been introduced by the differential attrition.

As shown in Table 3, the means or prevalence rates for survey measures were similar across assigned condition for most measures we examined and the standardized effect size for the differences in means (Cohen’s ds) were small. Only 1 of 12 differences was statistically significant at the p < .05 level. Parents assigned to phone-first condition reported that they were responsible for a higher portion of the financial support of their children than parents assigned to the Web-first condition. Secondary analyses that adjusted for covariates showed the exact same pattern of results with respect to differences significant at the p < .05 level. These results suggest that, although approximately three quarters of the participants complied with their condition assignment, mode effects on responses did not produce a consistent pattern of differences by assigned condition.

Table 3.

Means and Prevalence Rates Assigned Condition

	Assigned Web n = 352 m (sd)	Assigned Phone n = 352 m (sd)	t	da
YABCL scales
Depression	2.56 (3.41)	2.51 (3.23)	0.39	0.02
Anxiety	2.60 (2.39)	2.88 (2.23)	−0.92	0.12
Avoidant	1.58 (2.17)	1.42 (1.92)	0.98	0.08
Antisocial	3.24 (5.18)	3.22 (4.59)	−0.03	0.00
Developmental tasks	4.46 (1.06)	4.50 (1.01)	−0.50	0.04
Parent financial support	2.42 (1.18)	2.62 (1.23)	−2.07*	0.17
Antisocial behavior by mom at 15	1.20 (0.38)	1.18 (0.27)	0.14	0.06
Antisocial behavior by dad at 15	1.34 (0.55)	1.32 (0.43)	0.18	0.04
Household income	13.17 (4.40)	13.04 (4.13)	0.57	0.03
Parent substance use past month	n (%)	n (%)		χ2
Cigarettes	97 (28)	92 (26)		.18
Alcohol	220 (62)	223 (64)		.06
Marijuana	13 (4)	12 (3)		.67

^*p < .05

YABCL = Young Adult Behavior Checklist

^aAbsolute value of difference in means between assigned conditions divided by pooled standard deviation.

Comparison by Assigned Condition and Completed Mode

The second strategy for assessing mode effects compared four groups of participants defined by assigned condition and completed mode. Comparisons were done using ANCOVA or logistic regression models, controlling for possible confounds of completed mode that might also be related to outcomes, and estimating both main effects of assigned condition and completed mode as well the interaction effect between assigned condition and completed mode. This strategy can be considered an “as-treated” analysis which, while departing from the randomized design, captures differences in responses by the mode in which a survey is completed that may be due to either to mode effects on responses or to unmeasured selection effects. In assessing mode effects these models make the assumption that selection effects are controlled through the inclusion of covariates.

Table 4 shows means or prevalence rates for four groups defined by assigned condition and completed mode. Of primary interest in the results of ANCOVA analyses are the main effects of completed mode and interaction effects between assigned condition and completed mode. Significant main effects of completed mode in ANCOVA analyses indicate that telephone respondents reported more child anxiety (mean for all completed-by-phone respondents = 3.08 versus mean for all completed-by-Web = 2.41) and antisocial behavior problems (mean for all completed-by-phone = 3.75 versus mean for all completed-by-Web = 2.75) for their children than did Web respondents. Telephone respondents also reported more childhood antisocial behavior by both the child’s biological mother (mean for all completed-by-phone = 1.22 versus mean for all completed-by-Web =1.15) and father (mean for all completed-by-phone = 1.37 versus mean for all completed-by-Web = 1.27) than Web respondents. The effect sizes for these four differences range from .20 to .30. There were also significant assigned-condition-by-completed-mode interaction effects for reports of household income and parent’s self-report of smoking in the prior month. In both cases, these interactions reflect a telephone versus Web difference in the Web-first condition, whereas there was little difference by completed mode among those assigned to the phone-first condition. Among those in the Web-first condition, telephone respondents reported more smoking (42% vs. 22%) and lower income (m = 11.48 vs. m = 13.96) than the Web respondents. As would be expected given random assignment to condition, main effects of condition assignment in the ANCOVA models were nonsignificant in most cases, with the exception that parents in the Web-first condition reported more avoidant problems for their children than parents in the phone-first condition, even after adjusting for completed mode and other model covariates.

Table 4.

Means and Prevalence by Assigned Condition and Completed Mode

	Assigned Web-first		Assigned Phone-first		Assign. F	Compl. F	Compl. da	Assign. by Compl. F
	Completed Web n = 250 m (sd)	Completed Phone n = 102 m (sd)	Completed Web n = 89 m (sd)	Completed Phone n = 263 m (sd)
YABCL scales
Depression	2.47 (3.25)	2.89 (3.84)	2.28 (3.08)	2.65 (3.32)	1.07	1.99	0.12	0.01
Anxiety	2.43 (2.37)	3.17 (2.45)	2.34 (2.34)	3.05 (2.18)	0.59	11.98**	0.30	0.03
Avoidant	1.53 (2.14)	1.77 (2.31)	1.17 (1.85)	1.49 (1.93)	4.09*	3.01	0.15	0.10
Antisocial	2.81 (4.57)	4.34 (6.10)	2.58 (3.80)	3.52 (4.89)	1.83	8.03**	0.25	0.36
Developmental tasks	4.47 (1.08)	4.36 (1.03)	4.44 (1.06)	4.49 (1.01)	0.13	0.011	0.01	0.80
Parent financial support	2.46 (1.19)	2.32 (1.14)	2.48 (1.19)	2.66 (1.26)	2.02	0.72	0.07	1.26
Antisocial behavior by mom at 15	1.15 (0.28)	1.28 (0.49)	1.14 (0.20)	1.20 (0.30)	2.53	11.11**	0.29	1.55
Antisocial behavior by dad at 15	1.26 (0.46)	1.50 (0.66)	1.31 (0.49)	1.32 (0.41)	1.82	5.28*	0.21	3.70
Household income	13.96 (3.55)	11.48 (4.57)	12.81 (4.63)	13.15 (3.91)	0.16	1.44	0.08	9.32**

	Assigned Web-first		Assigned Phone-first		Assign. Wald	Compl. Wald	Assign. by Compl. Wald
	Completed Web n = 250 n (%)	Completed Phone n = 102 n (%)	Completed Web n = 89 n (%)	Completed Phone n = 263 n (%)
Parent subst. use past mo.
Cigarettes	54 (22)	43 (42)	25 (28)	67 (26)	1.00	3.64	6.61*
Alcohol	151 (60)	69 (68)	58 (65)	165 (63)	0.00	0.95	2.26
Marijuana	9 (4)	4 (4)	1 (1)	11 (4)	0.87	0.95	1.01

Note: 2X2 ANCOVA models for continuous outcomes; logistic regression models with mean-centered scoring of assigned and completed modes for dichotomous outcomes. Models adjust for household income, gender of child, gender of parent respondent, cohort, and intervention condition.

^aEffect size (d) is the adjusted effect of completed mode divided by the pooled standard deviation.

YABCL = Young Adult Behavior Checklist

Discussion

Based on prior research comparing results of Web- and telephone-administered surveys, we expected that telephone surveys would make parents less likely to report their own negative behavior or the negative behavior or attributes of their children because they would need to disclose this information to an interviewer. The one result that may be consistent with the social desirability effects is that, in the as-assigned analyses, parents assigned to the phone-first condition reported providing more financial support for their children. It may be that answering this question over the telephone caused some parents to inflate the level of financial support they gave their children. However, the size of this difference was small (d < .20), and generally there was little evidence, from either the comparisons by assigned condition or from the tests of effect of completed mode in the 2×2 ANCOVA models, to support the hypothesis of social desirability effects of phone surveys. The comparisons by assigned condition revealed only one statistically significant difference, which might be expected by chance. The ANCOVAs indicated more statistically significant differences, although in each case parents completing by phone were more likely to report negative behaviors than those who completed by Web. These analyses were done with covariates in an attempt to control for selection effects, but it is likely that selection biases were not fully controlled (Schonlau et al., 2009). In particular, it is likely that parents from more troubled families were more likely to complete their surveys by phone, particularly among those assigned to the Web-first condition. Models predicting completed mode generally found that parents from lower income families were less likely to complete by Web if assigned to the Web-first condition and may point to unmeasured variables related to completed mode and also related to behaviors of the parents and their children. The main implication of these results may be that a mixed-mode data collection strategy can result in different types of participants completing by different modes, either due to Web access or due to willingness to comply with the first mode that is offered. Further, it may not be possible to correct for selection effects with a short list of covariates based on demographic variables or common measures of socioeconomic status.

This study tested for mode effects on responses in a specific community sample, and parents who completed the survey differed in terms of race and income compared to those who did not complete the sample. Generalizability of these results should be considered with respect to the fact they come from the special case of a long-term longitudinal study. Most parents had completed annual parent telephone surveys during the course of the study, answering questions about their children’s behavior and their own behavior. Over the years, many participants had repeated contact with the same interviewers. While this lack of anonymity might contribute to hesitancy about revealing sensitive information about themselves or their families, participants experienced no breach of confidentiality in prior surveys, and may have developed a trust in the project which would reduce any tendency to minimize reports of negative behaviors. Our findings do differ from findings of Greene and colleagues (2008) who found, based on data from a longitudinal study, evidence of social desirability effects in response to questions about diet and exercise. It may be that the content focus of our survey, which was primarily the psychosocial functioning of the parent’s child during early adulthood, was not as sensitive to social norms about more desirable responses.

A strength of the study was the use of an experimental design. Completion rates and demographic and other background characteristics of survey completers were similar across assigned condition, suggesting that comparisons by assigned condition were based on samples representing similar populations. However, the study did have notable limitations in power. While some questions asked about parents’ own substance use, the rate of use of illicit drugs was low and thus the power to detect mode effects on these questions was also low.

Using mixed-mode conditions to assess mode effects further limited power of comparisons by assigned condition, since the design resulted in approximately a quarter of participants not completing in the mode to which they were assigned. The as-treated analyses provided by the 2×2 ANCOVA models partially address this issue, but suffer from the limitation that completed mode is not random and inferences of causal effects of mode rely on the assumption that selection bias is controlled through the use of model covariates. While we were able to include a range of covariates in these models, we lacked (with the exception of household income) prior measures of the outcome variables since the content areas of the parent survey had changed from the surveys administered when participants were in high school. It is thus difficult in these analyses to disentangle mode effects on responses from unmeasured selection effects. A more direct test of mode effects would use a randomized design and restrict participants to their assigned mode, although such a design is challenged by the problem that limited Web access would likely result in differential completion rates. A mixed-mode strategy may be necessary to achieve adequate completion rates and sample coverage while capitalizing on the cost savings provided by Web surveys.