Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research

Survey researchers are rightly concerned with measuring the level of potential bias in estimates generated from the surveys.2 Bias in estimates can result from measurement error, processing/editing error, coverage error, and nonresponse error (Federal Committee on Statistical Methodology [FCSM] 2001). It is nonresponse bias that is the focus of this editorial and it is also the subject of the paper by Halbesleben and Whitman (2013) that this editorial accompanies.

Nonresponse bias is a perennial concern for survey researchers as not everyone we attempt to include in our surveys responds. And to the extent that nonrespondents are different from respondents on the key variables the survey was designed to study these differences could bias the very estimates the survey was designed to make. Because we often have very little information about those who do not respond, survey researchers have long focused on the response rate as a key indicator of survey quality. The assumption is that the more nonresponse there is in a survey, the higher the potential for nonresponse bias. Unfortunately, this assumption has served as a problematic diversion for survey research from the real concern of how survey nonresponse potentially biases survey estimates.

What is Known about Response Rates

Response rates lack both validity and reliability as a proxy measure of nonresponse bias. Response rates lack validity in that there is not even a moderate correlation with nonresponse bias (Groves 2006). Groves’ analysis showed that there is a significant amount of variability in nonresponse bias from one estimate to another within the same survey with the same response rate. This study is a particularly strong reminder to the survey research community that nonresponse bias is an estimate level measure and it cannot be easily summarized by a survey level proxy measure such as an overall survey response rate. In addition to Groves’ (2006) paper, many studies have demonstrated that achieving a higher response rate for a survey does not result in significantly different estimates than the same survey using a less aggressive protocol and achieving a lower response rate (Keeter et al. 2000, 2006; Triplett 2002; Blumberg et al. 2005; Holle et al. 2006; Davern et al. 2010).3

Response rates are also unreliable in the sense that two surveys studying the same population and following a similar protocol can derive very different response rates. Differences from one survey to the next (or one survey vendor to next) in handling technical issues related to the final disposition codes that feed into the one of several response rate formulas (American Association for Public Opinion Research [AAPOR] or Council of American Survey Research Organizations) are partially responsible for the nonreliable nature of response rates. However, this is not the only source of variability, as even when examining the same survey with the same set of final disposition codes the calculated response rate can differ depending on assumptions a researcher makes. Skalland (2011) showed that using the final dispositions for the 2008 Behavioral Risk Factor Surveillance System BRFSS he could calculate response rates that vary from 26.9 percent to 65.6 percent depending on the assumptions researchers were willing to make about the proportion on eligible cases among the unobserved cases.

The Fixation on Response Rates

The seemingly easy to compute nonresponse rate has come to serve as the survey research community's proxy for nonresponse bias. Looking at the AAPOR's (the survey industry's main professional organization) annual research conference over the last 3 years demonstrates the industry's fixation on response rates. The terms “nonresponse rate” or “response rate” occurred 767 times in paper titles and abstracts from 2010 to 2012. On the other hand, the terms “response bias” or “nonresponse bias” only occurred 218 times over the same period. This finding suggests that the survey industry is more interested in writing about response rates as a proxy for nonresponse bias than in assessing nonresponse bias and survey quality directly.

In addition to the survey industry, two important institutions explicitly reference response rate targets in their guidance on nonresponse bias. First, The Office of Management and Budget (OMB) reviews most survey protocols funded by the federal government and asks surveys being reviewed to “submit a plan for a nonresponse bias analysis if the expected unit response rate is below 80 percent (see Section 3.2.9).” (http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf). Although OMB's guidance should not be read to imply this, the requirement may lead one to believe that a response rate of 81 percent will not lead to nonresponse bias and gives researchers conducting surveys with high response rates a false sense of security and implies a relationship between response rates and nonresponse bias that has not been justified empirically. Second, influential journal editorial practices can restrict access to publication when surveys have a low response rate. The Journal of the American Medical Association editorial policy states “Survey studies should have sufficient response rates (generally at least 60 percent) and appropriate characterization of nonresponders to ensure that nonresponse bias does not threaten the validity of the findings.” (http://jama.jamanetwork.com/public/instructionsForAuthors.aspx#GeneralInformation). Just as with OMB guidance, the 60 percent response rate statement can take on a gatekeeping function for peer-reviewed publications in top scientific journals. Thankfully, the second portion of the JAMA statement calls on survey researchers to “ensure nonresponse bias does not threaten the validity of findings” and OMB requires a nonresponse bias analysis for the vast majority of surveys (as most surveys do not meet the 80 percent threshold). However, instead of setting a response rate threshold in the same sentence giving guidance regarding nonresponse bias, JAMA and OMB would be better served by requiring surveys to complete some type of nonresponse bias analysis and to have this analysis put in the public domain and referenced in all analytical pieces done using the survey data.

How Has the Fixation on Response Rates Been Harmful?

A substantial sum of a survey's budget can be spent efforts to increase the response rate (including spending money on incentives, advance letters, additional telephone calls, refusal conversion, and follow-up mailings, etc.) (Groves and Peytcheva 2007; Davern et al. 2010). The additional effort and survey funds produce higher response rates, but have not been linked to reductions in nonresponse bias (Groves 2006). Furthermore, surveys that have a higher response rate have not been shown in either actuality or through simulation to produce significantly different estimates from surveys estimates obtained from lower response rate surveys (Keeter et al. 2000, 2006; Triplett 2002; Blumberg et al. 2005; Holle et al. 2006; Davern et al. 2010). The fixation on increasing response rates has been harmful as researchers spend a large portion of the survey's budget increasing response rates that could have otherwise been used to either increase the sample size (using a less aggressive protocol that results in lower response rates at a reduced cost per case) and/or conducting rigorous nonresponse bias analyses (such as those proposed by Johnson and Wislar 2012 and Halbesleben and Whitman 2013). This latter line of work is very important as it will help us better understand the mechanisms that lead to nonresponse bias, which estimates and types of surveys are more likely to be impacted by nonresponse bias, and how bias can be corrected (Groves and Peytcheva 2008).

Survey researchers should consider transferring some of the funds currently spent trying to maximize response rates on conducting nonresponse bias analyses and increasing the effective sample size of a survey. In Davern et al. (2010), the three surveys studied showed that for the same total survey budget approximately twice the effective sample size (e.g., from 10,000 to 20,000 effective sample size completed responses) could be obtained if a less aggressive protocol were followed and the research team were willing to accept an approximately 40 percent lower response rate (e.g., from 45 percent to 26 percent). The paper also showed that the approximately 40 percent reduction in response rate would not have produced one statistically different result across the 33 estimates examined (Davern et al. 2010). All three surveys could have funneled the extra money spent achieving higher response rate into conducting a thorough nonresponse analysis and/or into increasing the effective sample size of the survey.4

Nonresponse Bias Analysis

Nonresponse bias analyses have been outlined by Halbesleben and Whitman (2013) and Johnson and Wislar (2012). In this issue of HSR, Halbesleben and Whitman (2013) propose conducting nonresponse bias studies of the following types:

• Comparison of the sample and population

• Follow-up analysis

• Wave analysis

• Passive/Active nonresponse analysis (focus groups)

• Interest level analysis

• Benchmarking

• Replication

Johnson and Wislar (2012) proposed the following:

• Analysis using data in sampling frame

• Comparisons of survey with other sources

• Analysis using external data sources

• Analysis of paradata

These types of nonresponse analyses should be written up as part of a data quality profile for the survey and the work could be paid for if survey researchers and their sponsors were willing to accept modestly lower response rates. In addition to these nonresponse analyses cited above, survey researchers should examine realization rates (Skalland 2011) for how they might be related to nonresponse bias. The realization rate shows the ratios of the sum of probability of selection (design) weights for a survey relative to the sum of the final poststratified weights. The realization rate allows researchers to see how much adjustment is needed to make the probability of selection weights equal to the population characteristics and they can be calculated for the survey as a whole or for subgroups (e.g., black males, 18–34 years of age).

Good examples of nonresponse bias analysis and data quality profiles are produced by several surveys. For example the National Immunization Survey's detailed total survey error analysis (Molinari et al. 2011) and the Current Population Survey's data quality profile (US Census Bureau 2002). However, as the Federal Committee on Statistical Methodology wrote in 2002 (FCSM 2001) there is a lack of consistency within agencies, and between agencies, with respect to the types of evaluations being published on the survey errors for surveys that make up the federal statistical system (FCSM 2001). The report then goes on to create an excellent taxonomy of the types of analyses and reports that should accompany survey research in the federal statistical system (FCSM 2001). In recent years, OMB has also been providing guidance to researchers in conducting rigorous and consistent survey evaluations.

Conclusion

In addition to the ways in which surveys should be improved to address potential nonresponse bias I believe there are three additional steps the survey industry should consider. First, surveys should produce and make available high quality paradata (data collected about the survey operations) to accompany the data collected on the survey questions themselves, so researchers can access and call histories, contact histories, know what mode the survey was completed in and whether an incentive was provided, and know whether a proxy interview was accepted as they conduct their analyses. These data would provide crucial information needed to conduct the nonresponse bias analyses. Currently paradata are often not considered part of the data file that is released to researchers and they are not included in public use files (one exception is the National Health Interview Survey which has been releasing a paradata file since 2006). Second, organizations conducting surveys should also be judged by their commitment to transparency (such as participation in AAPOR's transparency initiative) (Miller 2010). Making key operational information public would be an important step to allowing a survey's data to be used in the research and allows the research community to be able to put the estimates derived from the survey into appropriate context to understand the potential impact of nonresponse bias as well as other types of survey errors. Finally, every survey sponsored by the federal statistical system, or if the results are being published in a scientific journal, should reference a publicly available quality profile that includes a nonresponse bias analysis. This would go a long way toward helping us understand the potential limitations of inferences made using the survey data as an empirical information base.