Why Choice of Survey Mode Makes a Difference

Recently, a surveyor who directs a large monthly survey of a general population called to tell me that a simple question he asked on marital status was producing different responses over the Internet than when asked by telephone. In particular, he reported that the percent of respondents who answered “single” had consistently declined by about 10% on the Internet version. The reason for his call was to ask me why the web survey was being answered by more unmarried people than the telephone survey. I responded by asking to see the exact wording of the question for both modes.

Over the telephone the respondent had been read this open-ended question: “What is your marital status?” The interviewer then coded the respondent’s answer into one of the listed categories: single, married, separated, divorced, or widowed. Although identical wording was used for the query on the web, the respondents needed to answer by reading and marking one of the five answer categories. Further examination of the data showed that the percent of married respondents had also decreased slightly, while the percent separated had increased significantly and the divorced and widowed categories had each increased a little. The likely reason for these differences seemed apparent. When asked one’s marital status in an open-ended fashion, single or married are the usual responses one would provide to a casual inquiry from a stranger. The remaining three categories—separated, divorced, or widowed—are more specific responses, which if available, some respondents would choose, but would otherwise not feel a need to volunteer.

There are many reasons in this age of increased survey alternatives that differences are frequently observed in responses across different modes. Yet one of the most frequent is that surveyors inadvertently or sometimes intentionally change how survey questions get asked and answered to better fit the particular survey mode. When I pointed out the difference between offering both the query and response choices on the web, but only the former by telephone, he responded by saying that was the standard way of asking this question over the telephone: “Our interviewers prefer it that way.”

The asking of many kinds of questions is affected by mode choice. In earlier years when face-to-face to interviews were our predominant data collection methodology, surveyors often used long scales with word labels attached to each category. The response task was made easier by providing a “show card” that reminded each respondent of the label for each scale point. The rise of telephone interviewing, with its complete dependence on verbal communication, combined with the desire to limit interview length, led to greater use of shorter polar-point scales (e.g., “On a scale of 1 to 5 where 1 means not at all satisfied and 5 means completely satisfied, and you can use any number from 1 to 5, how satisfied are you?”). To understand the reason for removing word labels for each scale point one needs only to visit a call center and listen to telephone interviewers as they struggle to correctly deliver questions that use fully labeled scales.

The development of Internet surveying has accentuated one particular mode difference, the use of check-all-that-apply questions. The HTML programming used in the construction of most web surveys assigns a radio button (small circle) to question choices when only one answer is to be selected; if more than one answer can be chosen, boxes are provided. The availability of this construction feature appears to be fostering an increase in the use of questions such as, “Please check all of the news sources from which you have gained information about current events in the past week,” followed by choices like daily newspaper, weekly news magazine, radio news program, television news program, and trade publication. If a similar question were asked in a telephone interview, it would be changed to elicit a yes/no answer to each news source. Research has shown that yes/no question formats result in answer choices being selected more often than when a check-all format is used.^1,2

In addition to the tendency for users of different survey modes to construct questions in different ways, sometimes intentionally and sometimes inadvertently, research has shown that certain mode differences are likely to occur even when the question stimulus is identical.³ For example, the simple question, “How would you describe your health, would you say excellent, good, fair or poor?” consistently produces more answers of “excellent” when asked in interviews as opposed to a self-administered format. Respondents have also been reported as more likely to acquiesce, i.e., give agreeable answers, in interviews and more likely to choose from last offered rather than the first offered response alternatives.⁴

In addition to the mode differences between interviewer and self-administered surveys, current research suggests that all self-administrated formats are not the same. Questions consist of more than words. Whether respondents see and read questions, the order in which they process the information they contain, and how they interpret the content depends on the use of numbers, symbols, and graphical composition. For example, a large-scale experiment embedded in the U.S. 2000 Decennial Census showed that the combined use of symbols (arrows) in combination with larger and darker fonts to direct people to the next appropriate question reduced the error of not skipping ahead when directed to do so by about one-third, from 19.7% to 13.5%.⁵ Additional research has revealed that on paper questionnaires, the use of symbols, horizontal vs. vertical layout of scalar categories, the sizes of answer boxes for open-ended questions, and other nonverbal features of question display significantly influence respondent answer choices.⁶

The reality of today’s data collection environment means that we cannot ignore the need to sometimes change survey modes, or mix them by collecting some data by one mode, and the remaining data by another. Declines in response rates for telephone interviews and sampling considerations are pushing more surveyors in the direction of mixed-mode designs. In addition, the move toward relying simultaneously on combinations of telephone, face-to-face, group-administration, mail, the Internet, and/or touchtone data entry by telephone, has in some large survey organizations led to the creation of different specialized survey units, each with its own idea of how questions should be worded and presented to respondents. The trend in such organizations seems to be for maximizing design features for each mode rather than creating an optimal or unimode design that minimizes differences in respondents’ answers across modes.

As a result, we face three interconnected challenges. First, we must find ways to reduce unintentional differences in the question stimulus presented to respondents who answer by different modes. Second, we need to continue research on ways of reducing and/or adjusting for traditional mode differences such as social desirability and acquiescence. Added to this is the third challenge of better understanding how the multiple visual languages used to express questions in self-administered surveys may produce differences in responses within this mode. Attending to each of these issues is essential if mixed-mode surveying is going to be successful in producing accurate estimates from sample surveys.