Tailored Explanation: A Strategy to Minimize Nonresponse in Demographic Items Among Low-income Racial and Ethnic Minorities

Abstract

There is a high rate of nonresponse for demographic items in survey research, particularly for racial and ethnic minority respondents. This present study examined whether providing an explanation to racial and ethnic minority respondents prior to asking a set of demographic questions would increase respondents’ motivation to reduce nonresponse to gender, income, age, and race items. Using a cross-sectional, randomized comparison design, 99 respondents were randomly assigned to two groups. Group 1 did not receive an explanation for asking the demographic questions. Group 2 received an explanation designed to be relevant and meaningful to them concerning the significance and potential use of demographic information for racial and ethnic minority populations. A proportional difference test was used to calculate the differences in the proportion of respondents’ motivation to complete demographic survey items between the groups. A proportional difference effect size (Cohen’s h effect size) was used to determine the magnitude of difference between the two groups. Over 50% of respondents were African Americans. While none of the item non-responses for both groups is statistically significant in terms of proportional differences, there is small (Cohen’s h= 0.184) to moderate (Cohen’s h=0.342) effect in reducing demographic item non-response when an explanation was provided to respondents. Specifically, adding an explanation made the biggest improvement in reporting income. The study findings support the importance of providing participants with an explanation that is relevant and meaningful to increase motivation and thereby minimize item nonresponse.

Incomplete data in surveys is a major concern for researchers. Data incompleteness can be associated with undercoverage, unit nonresponse, and item nonresponse (Groves et al. 2009). This paper focuses on item nonresponse to demographic items including income, race, gender, and age. Item nonresponse refers to when the respondents in the selected sample provide some, but not all, of the requested information (Groves et al. 2009). Item nonresponse can lead to bias and measurement errors. Thus, it is essential to reduce item nonresponse.

Factors That Could Explain Item Nonresponse

Researchers have reported predictors both of nonresponse in general and of income item nonresponse (Ferber 1966; Gronhaug et al. 1988; Bell 1982). For instance, Ferber (1966) and others (Ralph 1984; Riphaln and Serfling 2005) reported that item nonresponse was found to be higher for females compared to males and higher for people with low compared to high educational attainment. In addition, people who are self-employed, homeowners (Sousa-Poza and Henneberger 2000; Pleis and Dahlhamer 2004; Stretcher 2007), in a lower social position, female, older (e.g. age 65 and over; Schrapler, 2001; Ross and Reynolds, 1996; Ralph, 1984), and African American (Ross and Reynolds, 1996) have demonstrated higher item nonresponse rates.

Researchers have suggested that cognitive and motivational factors contribute to item nonresponse including income items (Loosveldt and colleagues 1999a; b; Tourangeau, Rips, and Rasinsk 2000). Beatty and Herman (2002) proposed three contributing factors to item nonresponse in surveys: (1) people’s amount of knowledge about the topic (cognitive), (2) people’s perceptions about how adequate their knowledge is relative to how accurate the question is asking for (cognitive), and (3) people’s willingness to report their responses to the question (motivation). Similarly, Juster and Smith (1997) and Moore, Stinson, and Welniak (1999) also have a model that includes cognitive and motivational factors. Moore et al. (1999) extends their model to include interpretation and comprehension of questions.

On the other hand, Singer, Hippler and Schwarz (1992) suggested that item nonresponse could be due to concerns about confidentiality. Questions regarding income have been reported to be “threatening,” difficult to answer, or thought as an invasion of privacy (Singer et al., 1993; Tourangeau, Rips, & Rasinski, 2001). While other researchers have suggested it could be due to the design of the survey (e.g., response format, poor question wording; Groves et al. 2009).

Strategies to Improve Item Nonresponse

Efforts have been made to decrease item nonresponse by addressing cognitive, motivational, and design issues. For example, researchers have developed open-ended questions that directly ask respondents to report their actual income value or a series of bracketed income questions, known as ‘unfolding bracket’ technique (Heeringa et al. 1993; Juster and Smith 1997). Unfolding bracket is used after the respondent has refused or said “don’t know” to an open-ended income question that asked for their exact income amount. The technique involves the researchers asking the respondent a series of closed-ended questions that place respondents’ income value within a specific range. This technique has been noted to reduce missing financial data by 50% or more (Heeringa et al. 1993; Juster and Smith 1997). Other researchers have reported that the technique has also reduced income nonresponse by 50% (Battaglia et al. 2002; McGrath 2005). Evidence suggests that the unfolding bracket technique is more effective for addressing cognitive causes of income nonresponse than motivational causes.

Moore (2006) created three procedures to elicit responses about income. The first was to use a flexible approach, where respondents can choose how they will in report their income. The second involved an expanded use of closed-ended “range” questions for use as follow up probes in the event of an initial nonresponse. Third was to use dependent interviewing procedures after an event of nonresponse. The dependent interviewing procedure is a follow up technique that interviewers use with respondents by saying, “I have recorded from last time that you received [dollars] from [income source] last February. Does that still sound about right for March, April, and May?” (Moore 2006, pp. 4). Moore (2006) found that there were reductions in item nonresponse with second and third approach.

Other solutions related to the placement of demographic questions in questionnaires. Some researchers contend that the best placement is at the end of the questionnaires (Whitley 1996; Fink et al. 2003; Dillman 2007; Jackson 2009). Stoutenbourgh (2008) argued that including the demographic items at the end would allow the primary survey questions to be answered when participants are most alert and invested in taking the assessment as well as to reserve the “boring demographic” questions until the end. In contrast, other researchers argued for placing demographic items at the beginning of survey interview to increase rapport (Babbie 2008). Despite these suggestions, there are mixed results for both placements of demographic questions. For example, Teclaw, Price, and Osatuke (2012) reported that placement of demographic questions at the beginning of the questionnaire increases item response rate while others do not (Whitley 1996; Fink et al. 2003; Dillman 2007; Jackson 2009).

Many of the efforts used to reduce item nonresponse have been focused on survey design and cognition. There are fewer studies that focus on increasing motivation of respondents to report the demographic items. In addition, current efforts have not considered the context of respondents including literacy and culture. Thus, we tested whether motivation to complete demographic survey items would increase among low income and low educated racial ethnic minorities after they are provided a verbal explanation by the interviewer. The explanation described the importance of providing demographic information and explaining that demographic information could be used to improve care for people from communities like theirs or for populations with similar characteristics. We hypothesize that if people from the same background create the message for people ‘who are like them’ or from similar communities, then it would increase such participants’ motivation to report the desire demographic items, thereby reduce item nonresponse.

METHODS

This study was approved by the University of Wisconsin-Madison Institutional Review Board.

Overview of Study Design

We recently described in detail our evaluation of impact of an established group, the Community Advisors on Research Design and Strategies (CARDS)®, on revising research materials and consent forms to improve recruitment and responsiveness (Jacobson, Bowers, and Krupp, in press; Bowers, Jacobson, and Krupp in press). In brief, the CARDS® is a community group of generally low income, high unemployment, and primarily ethnic and racial minority individuals (Kaiser, Thomas, and Bowers in press). Using materials from a previously completed study of heart failure, participants were asked to assess their likelihood of responding positively to recruiting materials, consenting to participate in the study and responding accurately to demographic items if they were actually enrolled in the study. This paper addresses participant responses to the demographic items.

Design and Sample

A cross-sectional, randomized comparison study was designed to understand whether providing a meaningful explanation to participants would increase participants’ motivation to complete demographic survey items, thereby reduce item nonresponse for demographic items. Prior to collecting data, the researchers met with CARDS® to draft a brief explanation of why responding to demographic items is important. The explanation is different from how researchers more often approach demographic items, which provides either no explanation or a general explanation that the research will improve care for a particular condition, but not making the connection to how people ‘like them’ might benefit from answering those items.

Recruitment was conducted by a person who was from the same community as the study participants and shared the same characteristics. A convenience sample of 99 adult participants was recruited from food banks and other programs (e.g., parenting and childcare programs, women’s support groups, and senior meal programs) at two community centers in low income, racially and ethnically diverse areas in Madison, WI. People were eligible to participate in the study if they were: eligible for food bank support or use of other community center services targeted to low income individuals. They also had an average of high school or no high school education.

Data Collection

Three White researchers from academia collected data with the participants. The researchers met face-to-face with the eligible participants individually on several different dates at the two community centers to conduct the study. To minimize contamination from social conversation about the study, researchers randomized the interview assignments based on day. That is, all participants on the same day, at the same center, were included in the same group. Groups were distinguished by either hearing the CARDS® explanation of why participation in research is important (see Table 1) or hearing no explanation.

Table 1.

Tailored Verbal Script Read by The Researcher

We know that some groups of people are healthier and even live longer than others. For example, women live longer than men. People with higher incomes are healthier and live longer than people with low incomes. So a major goal of health research today is to try to eliminate this difference in health and how long people live, to get rid of these differences between groups of people. One way to do that is to learn more about these differences and to understand the reasons for the differences in health. If we knew more, we could be able to make the changes that would improve the health of people who suffer the most from health problems. So if researchers can find out who is most affected by particular health problems and the reasons they have more health problems, they would be better able to improve care of people who need it the most. This is why researchers always ask questions about the people who are doing the surveys. For instance, researchers usually ask about race, income, education, gender. We know that many people don’t answer these questions. Sometime they leave it blank and sometime they actually give an incorrect answer, thinking that income or race, for example, doesn’t have anything to do with health. This makes it difficult for researchers to learn about differences among groups of people or to develop the most effective treatments.

Participants in each group were shown four demographic questions and then were read the multiple choice responses (see Table 2); ‘if this question appeared on a survey you had agreed to complete, would you ‘Answer the question correctly?, Answer the question incorrectly?, or leave the item blank? Participants were only asked how they would respond to the question but were not actually asked to respond to the items.

Table 2.

Demographic and Motivation Questions

Age	What is your age?
	What would you be most likely to do? ◦Answer the question accurately ◦Leave it blank ◦Give an answer that is not accurate
Race	What is your race?
	What would you be most likely to do? ◦Answer the question accurately ◦Leave it blank ◦Give an answer that is not accurate
Gender	What is your gender?
	What would you be most likely to do? ◦Answer the question accurately ◦Leave it blank ◦Give an answer that is not accurate
Income	What is your annual household income?
	What would you be most likely to do? ◦Answer the question accurately ◦Leave it blank ◦Give an answer that is not accurate

Analysis of Data

The answers to the questions about whether or not the respondents would answer the demographic questions were compared between the two groups. Analysis was done in two steps. First, we examined the differences between the groups on completion and accuracy of the responses for each demographic item: (a) age, (b) gender, (c) race, and (d) income. We used the proportional difference test and then calculated the potential differences in the proportion of participants’ willingness to respond between the groups with 95% confidence.

In the second step, we examined the proportional difference effect size, allowing us to determine the magnitude of nonresponse differences between Group 1 (no explanation) and group 2 (CARDS® explanation) because of the small sample size. We used Cohen’s h (arcsine transformation) effect size (Cohen, 1988). Effect sizes (Cohen’s h) for explanation versus no explanation differences are presented; Cohen’s h statistics are made positive for effects where the explanation “outperforms” the no explanation.

RESULTS

Because the purpose of this study was to examine respondents’ response to the tailored explanation versus no tailored explanation to reporting demographic items, prior to the interview we did not screen for or collect demographic data related to gender, race, gender, or income. However, because we recruited respondents from specific neighborhoods, the respondents were predominantly African American and lower income with an evenly divide of female and men in the study.

In general, there was a higher proportion of intended non-reporting on all demographic items for the group not receiving the explanation (Group 1). For example, as shown in Table 3, 5.6% of participants in group 1 indicated they would not report age compared to 2.2% of participants in group 2 (those receiving explanation), constituting a 3.3% difference. However, this is not statistically significant (CI: −0.135, 0.0667).

Table 3.

Proportional Test And Effect Size of Motivation to Report Demographic Items

Items	Group 1 (%)	Group 2 (%)	Mean difference proportion (%)	CI 95%	p-vale	Cohe n	CI 95%
Age	5.6	2.2	3.4	(−0.14, 0.08)	0.30	0.18	(−1.82, 2.17)
Gender	7.4	2.2	5.5	(−0.16, 0.05)	0.16	0.25	(−1.74, 2.25)
Race	18.5	8.9	9.6	(−0.24, 0.05)	0.09	0.18	(−0.34, 0.71)
Income	35.2	20	15.2	(−0.33, 0.03)	0.05	0.34	(0.09, 0.59)

Note: this calculation was done focusing on leaving blank answers relative to accuracy of responses

While none of the item non-responses for both groups is statistically significant in terms of proportional differences, we calculated an effect size to show the magnitude of the effect. Cohen’s h effect size was calculated. As shown in Table 3, there is small (Cohen’s h= 0.184) to moderate (Cohen’s h=0.342) effect in reducing demographic item non-response when explanation was provided to respondents. Specifically, adding an explanation made a small improvement in reporting gender and age. All of the demographic items (age, gender, race, and income) are going in the same direction, indicating some improvement after the explanation was provided. The largest improvement in item nonresponse was to the income item. For group 1, income question non-response was (35.19%). However, there is a 15.2% in reduction of non-response of income when the explanation was provided to participants, constituting a moderate (Cohen’s h = 0.34) effect size in the desired direction.

DISCUSSION

These findings suggest that providing an explanation to respondents may improve their motivation to respond to demographic items. The increased motivation to report the demographic items may be due to the specific explanation created by CARDS®—an explanation tailored to the context of the respondents. This particular population has been found to mistrust researchers and worry about what the researchers will do with the results of studies (Yancey et al. 2006). This specific explanation focused on how they might benefit from participation. Thus, it may have motivated participants by replacing a fear of how results could be used with an anticipated benefit of using the results. Furthermore, because the verbal explanation emphasized the benefits to ‘people like themselves’ that might result from providing demographic information, participants might have determined that the benefits outweighed any anticipated costs or they see the benefits for their community and not just personal gain. This notion could be explained by Dillman’s (1978) theoretical appeal argument from a social exchange approach for increasing response in mail surveys. The social exchange approach states that respondents are more likely to respond if they believe that in the long run the anticipated benefits of responding outweigh the anticipated costs (Dillman 1978).

It is perhaps not surprising that participants in the group that did not receive a tailored explanation had higher item nonresponse. It could be that without the explanation, participants are less motivated to complete the items for reasons described in the existing literature (Loosveldt et al. 1999a; b). In the absence of the explanation tailored to the context of the respondents, the task of completing the demographic items may seem less attractive (Cialdini 1984).

Although our findings appear to be promising for reducing item nonresponse by using a verbal tailored explanation, we did not compare or test how the verbal explanation differed from having the respondent read a written explanation on their own. Thus, future studies could examine whether verbal or a written explanation is more effective in reducing item nonresponse. It is possible that there could be a social desirability bias occurring after we provided the explanation, but we did not evaluate this. Future research could also examine the effect of a tailored explanation on social desirability. In addition, we did not control for interviewer effect. It is possible that the three researchers who read the questions may have had different effects on response.