Exploring Nonresponse Bias in a Health Survey Using Neighborhood Characteristics

Abstract

Objectives. We examined potential nonresponse bias in a large-scale, population-based, random-digit-dialed telephone survey in California and its association with the response rate.

Methods. We used California Health Interview Survey (CHIS) data and US Census data and linked the two data sets at the census tract level. We compared a broad range of neighborhood characteristics of respondents and nonrespondents to CHIS. We projected individual-level nonresponse bias using the neighborhood characteristics.

Results. We found little to no substantial difference in neighborhood characteristics between respondents and nonrespondents. The response propensity of the CHIS sample was similarly distributed across these characteristics. The projected nonresponse bias appeared very small.

Conclusions. The response rate in CHIS did not result in significant nonresponse bias and did not substantially affect the level of data representativeness, and it is not valid to focus on response rates alone in determining the quality of survey data.

Declining survey response rates over the last decade have raised concerns regarding public health research that uses population-based survey data. Response rates are commonly considered the most important indicator of the representativeness of a survey sample and overall data quality, and low response rates are viewed as evidence that a sample suffers from nonresponse bias.^1,2 Recent survey research literature, however, suggests that response rates are a poor measure of not only nonresponse bias but also data quality.^3–7

The decline in survey response rates over the past several decades has led to a number of rigorous studies and innovative methods to explore the relationship between survey response rates and bias. A meta-analysis that examined response rates and nonresponse bias in 59 surveys found no clear association between nonresponse rates and nonresponse bias.⁸ Some surveys with response rates under 20% had a level of nonresponse bias similar to that of surveys with response rates over 70%. This is because nonresponse bias is either a function of both the response rate and the difference between respondents and nonrespondents in a variable of interest,⁹ or it is a function of covariance between response propensity and a variable of interest.¹⁰ Therefore, response rates alone are not the determinant of nonresponse bias of the survey estimates. Although it may be convenient to use the response rate as a single indicator of a survey's representativeness and data quality, nonresponse bias is a property of a particular variable, not of a survey.

Nonetheless, declining survey response rates increase the potential for nonresponse bias and have raised questions about the representativeness of inferences made from probability sample surveys. Inferences from surveys are based on randomization theory and assume a 100% response from the sample. Although the gap between theory-based assumptions and the reality of survey administration has always been a concern, the increasing deviation from the full response assumption increases this concern.

Nonresponse is multidimensional, not a unitary outcome, and is roughly divided into 3 components: noncontact, refusal, and other nonresponse.⁹ Most examples of nonresponse compose the first 2 components. A study by Curtin et al. found that refusal rates in a telephone survey remained constant between 1979 and 2003, although the contact rates decreased dramatically.¹¹ Another study by Tuckel and O'Neill found the same pattern.¹²

Arguably, different dynamics lead to noncontact and refusal.^13,14 Noncontact (e.g., unanswered phone calls in random-digit-dialed surveys) is related to accessibility. Call screening devices, phone usage, and at-home patterns affect accessibility, and calling strategy (e.g., number of call attempts and timing of calls) directly influences contact rates.^7,12 Refusal occurs only after contact is made. The decision to participate or not is an indicator of the respondent's amenability to the survey and is also influenced by other factors.

Noncontact and refusal may affect different types of potential biases, and these biases may offset one another.^7,15 For example, measures on volunteerism may be biased through noncontact because those who spend much time volunteering may be hard to reach in random-digit-dialed surveys. On the other hand, those who refuse to participate in the same survey may have opinions and behaviors related to volunteerism that differ dramatically from those of persons who are never contacted. Because aggregating noncontact and refusal may obscure our understanding of nonresponse bias, understanding detailed response behaviors along with overall nonresponse bias is important.

The decline in response rates is more rapid for random-digit-dialed telephone surveys than for other survey types. The difficulties inherent in examining nonresponse bias arise from the absence of data on nonrespondents. Unlike face-to-face surveys, in which interviewers make direct observation of the sampled individual and have an opportunity to gather contextual information regardless of response status, such information is scarce in telephone surveys because interviewers do not visit the individual and the interviewer–respondent interaction, if any, remains oral and over the telephone. Follow-up with nonrespondents in a telephone survey can be conducted to study its nonresponse bias, but such efforts are resource intensive. Additionally, unless 100% participation is achieved, there still remains some level of nonresponse.

Alternatively, nonresponse can be studied through the use of the geographic identifiers associated with sampled telephone numbers. Phone numbers from random-digit-dialed sampling frames can be readily associated with a limited number of geographic identifiers, such as zip codes. In addition, most phone numbers can be matched to a postal address and consequently to a census tract and county, which provides a unique opportunity to evaluate patterns of nonresponse as a function of neighborhood characteristics. A few recent nonresponse bias studies have used such contextual data.^16–19

We examined potential nonresponse bias in the 2005 CHIS, a large random-digit-dialed telephone survey, by comparing a wide range of census tract–level neighborhood characteristics by response behavior as well as examining response rates across neighborhood characteristics. Although these characteristics are not specific to individual cases (households), neighborhood characteristics at the census tract level serve as useful proxy indicators of differences in the population. This is because census tracts are relatively permanent small geographic divisions with 1500 to 8000 people that are designed to be homogeneous with respect to sociodemographic characteristics.²⁰ Unlike previous studies that focused on statistical significance, we discuss substantive significance. We explored nonresponse bias in a large, population-based telephone health survey in California. We linked data from the California Health Interview Survey (CHIS) to US Census data at the tract level to compare respondents and nonrespondents across a broad range of neighborhood characteristics.

METHODS

Potential nonresponse bias was evaluated in 3 stages. The data for the first 2 stages was a product of 4 different data sources: (1) 2005 CHIS call history data,²¹ (2) 2000 US Census Summary File 3,²² (3) 2000 Census administrative data,²³ and (4) 2004 general election data.²⁴ CHIS, a biennial random-digit-dialed survey of California's population, is one of the largest random-digit-dialed telephone surveys in the United States. In the 2005 CHIS, interviewers made up to 15 call attempts to each of 225 229 eligible telephone numbers for the screener interview conducted with a household informant to obtain roster information for selecting the adult respondent. When adjusted for the sampling design, the screener interview response rate was 49.8% by the American Association for Public Opinion Research's Response Rate 4.²⁵ The response rate for sampled adults was 54.0%, and the overall response rate for adults was 26.9%.

CHIS call history data provide information about both respondents and nonrespondents, including response behavior at the screener interview and the geographic location of their residence. In this report, we focus on nonresponse at the screener interview stage because a greater number and proportion of the sample failed to be captured at that stage. To eliminate any potential confounding effects, we excluded from the analysis 81 498 individuals from the later stages of data collection who did not necessarily receive the same level of call attempts and attempts to persuade them to respond. There were 10 cases located in 4 census tracts formed after the 2000 Census. Because census neighborhood characteristics were not available, they were also excluded from the further analyses. After these exclusions, the total sample size for the analyses described in this study was 143 721.

All telephone numbers were matched to addresses to the extent possible; the addresses were in turn geocoded to the corresponding census tracts. For those without exact addresses, we used the most likely zip code available from the sample frame and assigned the census tract of the corresponding zip code's centroid. This was done for 29.1% of the total sample. Additionally, each sampled individual was also linked to county.

For each individual in the CHIS call history data set, we had identical contextual information with which to compare (1) respondents, (2) persons who were contacted but refused to respond, and (3) phone numbers at which no contact was made. The contextual information came from the 2000 Census and the November 2004 general election in California. The 2000 Census Summary File 3 provides a series of detailed demographic, socioeconomic, and other characteristics of individuals, families, households, and housing units in the population. It also includes some information on disability. The 2000 Census administrative data include the response rates for the 2000 Census mail version and the hard-to-count score, which is a composite of 12 variables summarizing difficulties in enumeration.²⁶ The last data source, 2004 general election data, was obtained from the University of California, Berkeley Institute of Governmental Studies Statewide Database²⁴ and includes voter registration, registered voters’ party identification, and voting behavior.

Although the CHIS call history data are specific to each individual, the rest are aggregate characteristics at what we refer to as the neighborhood level. All census data were used at the tract level, and the election data at the county level. These 4 data files were merged so that each of 143 721 individuals in the CHIS call history data has variables from 3 other data sources. For example, an unemployment variable indicates the proportion of the population that was unemployed within the census tract in which the sampled individual was located, rather than whether the individual was unemployed.

In the first stage of the study, we divided the sample by response behavior and computed respective means of neighborhood characteristics. Nonresponses were further classified into 3 groups: refusals (e.g., the individual refused, made an appointment for a later call, or requested an advance letter), noncontacts (e.g., telephone rang but no one answered, maximum number of calls, reached answering machine, or questionable ring), and other nonresponse (e.g., hearing and speech problem, language difficulty, or other factor).

The second evaluation used the mean of each neighborhood characteristic for the total sample from the first evaluation and divided the sample into high and low groups (e.g., low proportion of urban population vs high proportion of urban population). We then calculated response rates for the respective groups. Both analyses were expected to show neighborhood characteristics associated with response behaviors.

In the third stage, we added 6 survey variables to the data used in the previous stages, including the current health insurance coverage, self-report of fair or poor health, overweight or obese status, disability status, binge drinking in the last 12 months, and current smoking status. These 6 variables, which were some of the key survey variables, were from the 2005 CHIS data, which included only respondents. We projected individual-level nonresponse bias in light of neighborhood variables demonstrated to be substantively related to the response behavior. We modeled individual-level survey variables with neighborhood characteristics in multilevel logistic regression:

where p_ij is the probability of a survey variable for respondent i in neighborhood j, β coefficients, x_j a vector of an intercept and neighborhood characteristics for neighborhood j, and u_j a random residual error for neighborhood j, allowing the effect of x_j to vary by neighborhood. We selected the following 9 neighborhood characteristics to include as x_j in equation 1: the proportions of non-Hispanic Whites, urban dwellers, never-married persons, linguistically isolated persons, those living in the same house as in 1995, those with less than a high school education, unemployed persons, median household income, and the 2000 Census hard-to-count score. Using the respondent data and census data, we fitted multilevel logistic regression models for survey variables with the neighborhood variables (results are shown in Appendix 2, which is available as a supplement to the online version of this article at http://www.ajph.org). The parameter estimates from the fitted models were applied to predict survey variables for the entire sample. The differences between the observed values for respondents and the predicted values for the total sample were considered to be the projected nonresponse bias.

We expected even small differences to be statistically significant because of the large sample sizes. Consequently, tests of statistical significance may not be meaningful, and we do not report them. We conducted both unweighted and sample-design-weighted analyses using SAS version 9.2 (SAS Institute Inc, Cary, NC). Because unweighted results showed larger differences, we report them to be more conservative. For the nonresponse projection analysis, we used Stata version 9 (StataCorp LP, College Station, TX).

RESULTS

Out of 143 731 cases, 54 969 completed the screener interview and 88 752 did not. More specifically, nonresponses comprised 46 623 refusals, 40 769 noncontacts, and 1362 other nonresponses, giving an unweighted response rate of 38.2%. The sampled individuals were located in 6968 of California's 7049 census tracts and in all of the state's 58 counties. Table 1 shows the distribution of the sample sizes and response and nonresponse rates at the census tract level. In addition, nonresponse was divided into refusal, noncontact, and other nonresponse. On average, 20.6 telephone numbers were sampled from each tract, with a standard deviation of 27.1. The response rate at the census tract level was 43.8% on average. The percentile figures show that there was a dispersion in tract-level response rates, in which the 10th and 90th percentiles were 20.0% and 66.7%, respectively. The average rates were 34.0% for refusal, 21.1% for noncontact, and 1.1% for other nonresponse. From this, we can see that sampled individuals living in different census tracts behave differently with respect to responding to survey requests.

TABLE 1.

Distribution of Census Tract–Level Sample Size and Rates of Response and Nonresponse in a Random-Digit-Dialed Telephone Survey: California Health Interview Survey, 2005

		Response Rate, %	Nonresponse Rate
	Sample Size,a No.		Refusal,%	Noncontact, %	Other Nonresponse, %
Mean	20.6	43.8	34.0	21.1	1.1
SD	27.1	18.9	16.3	16.4	3.7
SE	0.3	0.2	0.2	0.2	0.0
Minimum	1.0	0.0	0.0	0.0	0.0
Maximum	452.0	100.0	100.0	100.0	60.0
1st percentile	2.0	0.0	0.0	0.0	0.0
5th percentile	3.0	13.6	7.7	0.0	0.0
10th percentile	5.0	20.0	15.0	0.0	0.0
25th percentile	8.0	31.8	25.0	10.0	0.0
Median	13.0	43.8	33.3	18.8	0.0
75th percentile	21.0	55.6	42.9	29.4	0.0
90th percentile	43.0	66.7	52.4	42.9	3.7
95th percentile	67.0	75.0	60.0	51.9	7.7
99th percentile	137.0	100.0	80.0	69.6	16.7

Note. The total number of census tracts was 6968.

^aNumber of telephone numbers per census tract.

Table 2 shows the means of 30 different neighborhood characteristics of respondents and nonrespondents. For all the characteristics reported, CHIS respondents and nonrespondents appeared to be very similar—there was almost no difference between these 2 groups in terms of population, size of housing units, gender, age, education, income, employment, and disability. Regarding race, nonrespondents tended to live in areas with more Hispanics and non-Hispanic Asians and fewer non-Hispanic Whites than did respondents. Nonrespondents were more likely to live in urban areas and areas with higher proportions of renters and never-married single persons. The imputation rates for income in the 2000 Census did not differ between respondents’ and nonrespondents’ neighborhoods, nor did the rates for the 2000 Census mail survey response. The census hard-to-count score indicated that respondents tended to live in areas easier to count. (Appendix 1, available as a supplement to the online version of this article at http://www.ajph.org, shows the detailed sample distributions of these characteristics. According to this table, the similarity between respondents and nonrespondents was not because of similarity across census tracts, because tracts appeared to vary one from another. Note that comparisons were done on over 90 variables; characteristics shown in this report tend to present larger differences between respondents and nonrespondents than those not shown.)

TABLE 2.

Distribution of Neighborhood Characteristics in a Random-Digit-Dialed Telephone Survey, by Response Status: California Health Interview Survey, 2005

		Nonresponse
Neighborhood Characteristics	Response	Total	Refusal	Noncontact	Other
Total sample, no.	54 969	88 752	46 621	40 769	1 362
Population size, mean	5 613.3	5 602.9	5 629.0	5 571.5	5 649.0
Male, %	49.4	49.4	49.3	49.5	49.4
Age, y, %
Birth to 17	26.3	25.7	26.3	25.1	25.0
≥ 65	12.0	12.0	11.9	12.0	12.3
Race/ethnicity, %
Hispanic	26.5	27.5	27.1	27.8	28.9
Non-Hispanic White	54.6	51.9	52.9	50.9	45.0
Non-Hispanic African American	5.0	5.3	5.2	5.3	5.4
Non-Hispanic Asian	9.8	11.3	10.7	11.9	16.1
Urban population, %	89.0	92.2	91.5	93.0	96.8
Never married, %	28.0	29.5	28.5	30.5	31.3
1-person household, %	22.1	23.5	22.4	24.8	24.8
Speak English only at home, %	64.9	62.3	63.4	61.4	53.2
Linguistically isolated, %	8.5	9.8	9.1	10.4	14.1
Living in the same house as in 1995, %	51.0	50.2	50.8	49.5	50.3
Less than high school education, %	21.3	22.1	21.7	22.3	25.0
Unemployed, %	4.3	4.2	4.2	4.2	4.6
Not in labor force, %	37.6	37.5	37.5	37.3	38.9
Median household income, $	51 927.1	51 531.0	52 418.4	50 672.1	46 870.0
Below 100% of federal poverty level,a %	12.9	13.4	12.9	13.9	15.4
Has income from social security, %	23.8	22.9	23.4	22.4	22.3
Has at least 1 disability, %	19.0	19.2	19.1	19.4	20.9
No. of housing units	2 105.4	2 124.1	2 108.0	2 142.7	2 121.5
Vacant housing, %	5.8	5.6	5.6	5.6	4.3
Renter occupied housing, %	38.6	42.5	39.7	45.4	49.7
Housing built after 1990, %	14.9	13.4	14.1	12.5	10.6
Housing with no vehicle available, %	8.3	9.7	8.8	10.7	12.6
Median gross rent, $	844.3	858.3	861.2	856.2	821.0
2000 Census hard-to-count score (range = 0–122)	38.3	41.3	39.2	43.5	49.1
2000 Census mail participation, %	76.5	76.3	76.6	75.9	76.0
No income imputed, %	70.7	70.9	70.8	71.0	70.9
Voted in 2004 general election, %	75.7	75.9	75.9	76.0	75.0
Registered Democrat, %	43.1	43.8	43.4	44.2	45.7

Note. Data are mean figures from the Census 2000 Summary File 3, Census 2000 planning data, and 2004 general election data.^22,23,24

^aDetermined by the 1999 US Census.

When we examined nonresponse types in detail, respondents and those who refused to respond appeared to be very similar—practically identical, except for a few variables such as the proportions of non-Hispanic Whites, urban populations, and households speaking only English. Conversely, there were greater differences between respondents and those not contacted than between respondents and nonrespondents as a whole; those not contacted were more likely to live in areas that were urban and that had higher proportions of minority populations, single persons, 1-person households, linguistically isolated persons, renters, and older housing units and had higher census hard-to-count scores. This indicates that not all nonrespondents were the same and that the similarity between respondents and nonrespondents in this study was not caused by the offsetting characteristics of refusals and noncontacts. The characteristics of individuals in the “other nonresponse” category were different from those of both respondents and of those who refused or were not contacted, because they were substantially more likely to be associated with ethnic and linguistic minorities. This is not surprising, because CHIS does not provide all languages spoken by California's population. Given the small proportion of other nonresponse cases among all nonrespondents (about 1.5%), their distinctive characteristics are unlikely to distort the implications about nonresponse bias.

Table 3 provides response rates by neighborhood characteristics. The quartile points of each variable in Appendix 1 were used to divide samples into 4 groups. For instance, areas with a total population of fewer than 4073 persons were categorized into the first quartile, those with 4073 to 5372 persons into the second, those with 5373 to 6757 into the third, and those with 6758 or more into the fourth. Response rates in the respective quartiles were calculated. Overall, the response rates did not vary substantially among the 4 groups for most of the neighborhood characteristics. However, characteristics that differed between respondents and nonrespondents in Table 2 again correlated with differential response rates. The response rates tended to be higher in the areas with higher proportions of non-Hispanic Whites, populations speaking English only, and newer housing, and with lower proportions of non-Hispanic Asians, singles, 1-person households, linguistically isolated persons and renters, and lower census hard-to-count scores. Additional characteristics, such as higher proportions of people receiving social security income or having an available vehicle and lower proportions of registered Democrats, appeared to be associated with positive response rates. Nonetheless, these response rate differences were less than 9 percentage points except for marital status and renter status.

TABLE 3.

Mean Response Rate of a Random-Digit-Dialed Telephone Survey, by Quartile of Neighborhood Characteristics: California Health Interview Survey, 2005

	Response Rate
Neighborhood Characteristics	First Quartile, Mean	Second Quartile, Mean	Third Quartile, Mean	Fourth Quartile, Mean
Total population size, %	38.8	37.0	38.5	38.6
Male, %	37.3	38.1	39.3	38.2
Age, y, %
Birth to 17	34.8	39.9	39.2	39.0
≥ 65	37.0	38.2	39.2	38.6
Race/ethnicity, %
Hispanic	39.1	38.1	39.2	36.7
Non-Hispanic White	34.3	37.7	40.4	40.5
Non-Hispanic African American	41.4	38.2	36.5	37.0
Non-Hispanic Asian	41.9	39.0	37.5	34.6
Never married, %	42.7	40.8	36.5	33.0
1-person household, %	39.3	40.5	39.4	33.9
Speak English only at home, %	34.1	37.1	39.6	42.2
Linguistically isolated, %	41.6	40.0	37.8	33.6
Living in the same house as in 1995, %	35.6	37.7	39.7	39.9
Less than high school education, %	38.2	39.3	39.2	36.4
Unemployed, %	38.5	38.3	37.3	38.9
Not in labor force, %	37.1	38.7	39.1	38.1
Median household income, $	36.9	38.9	38.8	38.4
Below 100% of federal poverty level,a %	38.4	40.1	38.3	36.1
Has income from social security, %	35.1	37.1	39.1	41.7
Has at least 1 disability, %	38.0	39.9	38.0	37.1
No. of housing units	38.7	37.7	39.0	37.6
Vacant housing, %	36.8	38.1	38.0	40.2
Renter occupied housing, %	41.0	41.0	39.8	31.2
Housing built after 1990, %	35.8	35.5	39.0	42.7
Housing with no vehicle available, %	40.9	40.1	38.9	33.1
Median gross rent, $	40.6	38.0	37.3	37.1
2000 Census hard-to-count score (range = 0–122)	40.8	40.2	37.9	34.1
2000 Census mail participation, %	37.3	37.0	38.9	39.7
No income imputed, %	38.4	37.7	38.8	38.1
Voted in 2004 general election, %	40.6	36.5	36.5	40.4
Registered Democrat, %	39.7	42.3	36.1	35.6

Note. Data are mean figures from the Census 2000 Summary File 3, Census 2000 planning data, and 2004 general election data.^22,23,24 The urban population variable was excluded from the analysis because more than 75% of the sample resided in areas with a 100% urban population. For an explanation of quartiles, see the “Results” section.

^aDetermined by the 1999 US Census.

Table 4 presents estimates of survey variables for observed respondents, predicted respondents, predicted nonrespondents, and the total sample, as well as projected nonresponse biases (i.e., differences between the observed respondent estimates and the predicted total sample estimates) for all survey variables. The observed proportion of current health insurance coverage for respondents was 88.7%, and the predicted proportions of current health insurance coverage were 88.0% for nonrespondents and 88.3% for the entire sample; individual-level nonresponse bias for insurance coverage was projected to be 0.4 percentage points higher than estimated. For other survey variables, the absolute nonresponse biases were projected to be as low as 0.1 percentage points and as high as 0.6 percentage points.

TABLE 4.

Estimates of Survey Variables, by Response Status and Projected Nonresponse Bias: California Health Interview Survey, 2005

	Currently Insured, %	Fair or Poor Health, %	Overweight or Obese, %	Have Disability, %	Binge Drinking, %	Current Smoker, %
Respondents observed	88.7	19.3	56.1	34.4	14.2	14.8
Respondents predicted	88.6	19.8	56.5	34.4	14.1	14.7
Nonrespondents predicted	88.0	20.1	55.8	34.0	14.1	14.6
Total sample predicted	88.3	20.0	56.0	34.2	14.1	14.6
Projected nonresponse bias	0.4	−0.6	0.1	0.2	0.1	0.2

DISCUSSION

We evaluated the potential for nonresponse bias in a random-digit-dialed telephone survey using neighborhood characteristics as proxy measures. As noted in the introduction, potential nonresponse bias differed by particular variables. We found that the estimates of this survey may understate proportions of urban area residents, single persons, renters, and racial and linguistic minorities. Potentially, health estimates associated with these characteristics may have been affected; however, the degree of potential underestimation was rather small. Characteristics such as age, gender, income, education, and employment status did not show much association with response behaviors. Although it seemed reasonable to expect that census tract–level response rates would be highly associated with census mail response rates and missing rates of income in the census, the results did not support this expectation. Most importantly, estimates of the proportion of people with disability at the neighborhood level, the most likely correlate of many health characteristics, were almost identical for respondents and nonrespondents. In addition, when using the relationship between neighborhood characteristics and survey variables in the respondent data, we found that the individual-level nonresponse biases in 6 key survey variables were projected to be very small—well under 1 percentage point.

This study examined characteristics at the neighborhood level and as a result has limitations. First, diversity at the individual level may be lost in the aggregate measures. Second, estimates from Census Summary File 3 are subject to sampling errors, and the findings should therefore be interpreted cautiously as proxy measures with sampling variability, not as direct measures of nonresponse bias. Third, although the survey was conducted in 2005, most of the neighborhood data were from 2001. Although it is likely that there have been changes during that period, the changes at the census tract level may not be large.

In spite of these limitations, the findings are compelling. We found that, contrary to the prevailing assumption that nonresponse bias arises from low response rates, the neighborhood characteristics of respondents differed little from those of nonrespondents and most of the observed difference was among households that could not be contacted, as distinguished from those that refused to participate in the screener interview. Our findings are consistent with most previous studies on survey nonresponse. At least for CHIS, relatively high refusal rates do not appear to result in a biased sample. Even differences between noncontact households and respondent households were small, but because this is the fastest-growing segment of nonresponse, it should remain an important focus of efforts to understand response rates and nonresponse bias.

By no means are survey data free from error. Nonresponse is merely 1 of 4 error sources—the others, according to the total survey error paradigm, being noncoverage, sampling, and measurement.⁹ High response rates do not necessarily produce high-quality data. For instance, large financial incentives may be used to increase response rates, which may attract a certain group in the population more than others and lead to systematic measurement error. The overall error may decrease, increase, or stay the same. Response rates are simply one of many ways to summarize the characteristics of a survey and may be a convenient, but not necessarily scientific, tool for summarizing nonresponse bias or data quality. This is well reflected in a statement by the American Association for Public Opinion Research:

[C]onsumers of survey results should treat all response rates with skepticism, since these rates do not necessarily differentiate reliably between accurate and inaccurate data.²⁷

It is evident that a broader spectrum of error sources should be taken into consideration when evaluating survey data quality.

It is noteworthy that recent studies on noncoverage bias in random-digit-dialed telephone surveys found that renters, racial/ethnic minorities, singles, and urban area residents were more likely to be affected by noncoverage.^28,29 The characteristics of these individuals mirror neighborhood characteristics associated with nonresponse shown in this study. It will be important to understand data quality for variables related to these characteristics as a combination of nonresponse and noncoverage biases. Telephone surveys will need to pay attention to these characteristics through, for example, more-rigorous data collection efforts or more-sophisticated adjustment methods.