Telephone health survey estimates: Effects of nonresponse and sample limitations

Abstract

Objective

The objective of this study was to assess nonresponse error in telephone health survey data based on an address‐based sample.

Data Sources

Telephone and in‐person interviews in Greater Boston.

Study Design/Data Collection

Interviewers attempted telephone interviews at addresses that were matched to telephone numbers using questions drawn from federal health surveys. In‐person household interviews were carried out with telephone nonrespondents and at addresses without matching telephone numbers.

Principal Findings

After adjusting for demographic differences, only eight of 15 estimates based on the telephone interviews lay within two standard errors of the estimates when data from all three groups were included.

Conclusions

For health surveys of address‐based samples, many estimates based on telephone respondents differ from the total population in ways that cannot be corrected with simple demographic adjustments.

1. INTRODUCTION

Since the 1980s, telephone surveys have been a critical source of information for those concerned with public health statistics. Prior to that time, the National Health Interview Survey (NHIS), which did and still does rely primarily on in‐person interviews, was the primary source for national health data. However, in the 1970s, telephone service covered well over 90 percent of housing units, Waksberg1 developed a more efficient way to sample telephone numbers connected to housing units via random digit dialing (RDD), and Groves and Kahn2 demonstrated the comparability of RDD‐based surveys to data collected from area probability samples using in‐person interviews. In that context, the Centers for Disease Control (CDC) launched the Behavioral Risk Factor Surveillance System (BRFSS) in 1984.

The distinguishing features of the BRFSS were that the surveys were done by states, so there would be state‐by‐state data, and the surveys were administered by telephone using RDD sampling. Indeed, as noted on the BRFSS website, the fact that telephone survey methodology had evolved to provide a lower cost alternative to in‐person surveys that met high data standards that states could actually carry out was an essential element in the founding of the BRFSS.3

In the last 10 or 15 years, two trends pose challenges for surveys that rely primarily on RDD sampling. First, response rates, particularly to telephone interview efforts, have been falling.4 For example, the average response rates for telephone surveys done by the Pew Research Center had dropped to less than 10 percent.5 The BRFSS has fared somewhat better. In 2014, the median response rates across states were 47 percent. However, the median does not reflect the much lower rates in the bigger states. The minimum response rate in 2014 was 25 percent. Almost all the response rates in larger states in 2015 were below 40 percent.6 In addition, the percentage of homes with landline service, which was the underlying basis for RDD sampling, has been dropping steadily. In 2015, it was estimated that almost 60 percent of US households had only wireless telephone service.7 The new reliance on cell phones makes response rates more challenging. Of particular relevance is that the area codes of cell phone numbers do not necessarily align with where respondents live, for example, when they acquire a cell phone in one state and move to another.

In that context, an alternative approach to general population surveys is to draw an address‐based sample (ABS) and attempt to use multiple strategies, including telephone where possible, to collect data from sampled households.4 While not all addresses within an ABS frame have an associated phone number, many do. Other approaches to data collection could include mail, Interactive Voice Response (IVR), and asking people to respond via the web. However, the purpose of this study is to assess the effects of nonresponse and inability to find a telephone number matched to an address if only the telephone is used to collect data from an ABS sample.

2. METHODS

2.1. The sample

An unclustered probability sample of 1500 residential addresses was drawn by Marketing Systems Group (MSG) from an address‐based sampling frame covering five contiguous Boston area communities. These communities, including three neighborhoods of Boston (Dorchester, Jamaica Plain, and Mattapan) and two of Boston's immediate suburbs (Milton and Quincy), were chosen based on their demographic diversity. The sample was drawn proportionate to the size of each the five neighborhoods and cities.

2.2. Data collection protocols and results

Addresses that were matched to a telephone number (approximately 60 percent of the sample) were sent a letter, accompanied by a two dollar cash incentive, explaining the background and purposes of the survey, assuring confidentiality, and informing them that an interviewer would be calling in the next few days. Soon thereafter, professional interviewers, working from a central phone facility, called each household. If there was more than one person 18 or older living in the household, the interviewer followed a randomized protocol that chose either the oldest or youngest to be the designated respondent. Each telephone number was called a maximum of 12 times; the median and modal number of calls was 6. Calls were placed on various days of the week, primarily during evenings and on weekends. All telephone interviews were conducted in English. In this part of the subsample, 143 respondents (20 percent response rate, AAPOR RR 3) completed the survey.

Addresses that were not matched to a telephone number received a letter, generally similar to the telephone interview letter, informing them that an interviewer would be visiting their homes to complete a personal interview. A 20 dollar postpaid cash incentive was promised upon completion of the interview. Addresses were visited a maximum of 13 times; the median and modal number of visits was 6. All personal interviews were completed in English. The same within household selection protocol was used as in the telephone interviews. In this subsample, 166 respondents (41 percent response rate, RR3) completed the survey.

Approximately half of the nonresponding cases from the telephone subsample (N = 350) were randomly selected for nonresponse follow‐up in‐person interviews. They were sent a second letter informing them that an interviewer would be visiting their home to complete a personal interview and promising a 20 dollar postpaid incentive upon completion of the interview. Each address was visited a maximum of 12 times; the median and modal number of visits was 6. In this nonresponding telephone subsample, 128 respondents (43 percent response rate, RR3) completed the survey.

Data collection began in the fall of 2015 and was completed in April 2016. All the data collection procedures were approved by the university's IRB. Table 1 summarizes the samples and these results.

Table 1.

Data collection results by sample

Sample	Number assigned	Number confirmed ineligible	Estimated number eligible	Number completed	Response rate (AAPOR RR3)
Households with matched telephone numbers	921	82	727	143	20%
Sampled households with telephone numbers that were nonrespondents	350	41	294	128	43%
Households for which there was not a matched telephone number	537	90	409	166	41%

2.3. The survey questions

The survey questions on which this analysis is based were all drawn from three federally sponsored health survey instruments: the Health Information National Trends Survey (HINTS), the BRFSS survey instruments, or the National Health Interview Survey. Questions were chosen to cover a wide range of the issues included in common health surveys. The exact wording and sources of the chosen questions are listed in Appendix S1. However, most of the constructs they measure appear in all three of those surveys.

2.4. The analysis

The focus of the analysis was the extent to which the estimates based on the telephone survey were the same or different from those one would make when we included data from nonrespondents and those excluded from the survey because a matched telephone number could not be found. Based on our sample design and allocation, the probability of selection of households was the same across the sample, but the probability of selection of an adult within a household varied by the number of adults, so all analyses were weighted to adjust for that. Using these final base sampling weights, we calculated and compared the key demographic characteristics of respondents in each of the three groups: telephone respondents, telephone nonrespondents, and those for whom we did not have a telephone number.

To understand the effects of nonresponse and exclusion of addresses for which a telephone match was not possible, we calculated the value of 15 health variables separately for those interviewed by telephone, the telephone nonrespondents who were interviewed in person and the people interviewed in person at addresses for which there was not a matched telephone number. Those estimates also were weighted to adjust for the within‐household selection of an adult. In the same analysis, we also combined the telephone and in‐person data for addresses with a matched telephone, so we could compare estimates from households with and without a telephone match. To combine the telephone and in‐person interviews, we also had to adjust via weighting for the probability of selection of a subsample of nonrespondents who had a chance to be included in the in‐person interviews. Differences in estimates between phone respondents and nonrespondents and between respondents with a matched phone number and those without were tested. Continuous variables, presented as weighted survey means, were compared using a two‐sample t test. Dichotomous variables, presented as weighted proportions, were similarly compared using the linear combination of estimators to determine if the observed difference is statistically significant.

Then, to evaluate the overall likelihood of error in the telephone estimates, we created four estimates for each health variable: the estimate one would make based solely on the telephone interviews; the estimate one would make based solely on interviews from addresses with a telephone match, combining the telephone and in‐person interviews; the estimates one would make based solely on interviews at addresses for which there was not a telephone number match; and, finally, an integrated estimate that used the data from all three datasets.

For all estimates, weighting to adjust for differences in probabilities of selection within the groups was applied, and then, the final sample weights were computed by applying poststratification weighting to the final base weights to match the characteristics of the population in the sample area (based on Census data) with respect to age, gender, marital status, ethnicity/race, and education. The variables chosen for poststratification adjustment are ones commonly used in practice and ones for which accurate Census estimates were available for the area under study.

Final weights for each group were then standardized so that the average weight was 1 and the weighted N's were approximately the same as the actual number of cases. We consider the estimates created from the combination of the initial telephone respondents, interviews with telephone nonrespondents, and interviews with those lacking a telephone match to be the “best combined estimate,” as it uses all the data available to us. Our bottom‐line test was whether or not the estimates based on the telephone survey alone were within two standard errors of the estimates based on the combined data. It should be noted that this approach to identifying an “important” difference is affected by the size of the combined sample. A larger sample would have smaller standard errors, and more estimates from the telephone‐only interviews would lie outside the range defined by two standard errors.

In addition, we computed the difference between the telephone only estimates and the estimates based on the combined data and considered this difference as a measure of the bias related to doing only a telephone survey. Since the estimates based on the combined data include the data from the telephone surveys, these estimates are not independent. Therefore, we estimated the standard errors for our bias measure using a formula that took into account the amount of overlap between the phone survey data and the combined data: Variance of the bias = (1−A)2 [Var(X) + Var(Y)], where Var(X) is the variance from the telephone data, Var(Y) is the variance of the non‐telephone data, and A is the fraction of the combined data that is from the telephone. Using these standard errors, we constructed a statistical test of whether the measure of bias was different from zero.

3. RESULTS

Table 2 presents the demographic characteristics of the respondents in the three groups and compares them with estimates for the sample area based on Census data from the American Community Survey. It can be seen that the groups are very different from one another. The telephone respondents differ most from the Census estimates. They were much more likely to be non‐Hispanic whites (70 vs 44 percent), much less likely to be “never married” (26 vs 46 percent), more likely to be college graduates (51 vs 35 percent), and, most striking of all, much more likely to be over 65 (56 vs 15 percent). The telephone nonrespondents differed from the respondents in being less non‐Hispanic white, less likely to be female and much less likely to be over 65. The biggest difference between the telephone nonrespondents and the Census estimates was in the percent never married (27 vs 46 percent), but they also tended to be more white, better educated, and older than the population as a whole. Meanwhile, one of the most fascinating aspects of Table 2 is that those for whom there was no phone match demographically looked very much like the Census estimates.

Table 2.

Demographic characteristics by sample type for respondents and for population as a whole

Demographic characteristic	Telephone respondents (N = 143)	Telephone nonrespondents (N = 128)	Households with no telephone match (N = 166)	Census estimates for population
Percentage white, non‐Hispanic	70%	56%	38%	44%
Percentage college graduates	51%	50%	40%	35%
Percentage female	61%	48%	52%	53%
Percentage never married	26%	27%	47%	46%
Percentage 65 or older	56%	29%	14%	15%

In Table 3, we present two sets of comparisons: telephone respondents vs telephone nonrespondents who were interviewed in person and data from those addresses with a telephone match vs data from households for which there was not a telephone match. The telephone respondents differed significantly from the interviewed nonrespondents on three variables: they had more chronic conditions, were more likely to have high blood pressure, and were less likely to have gotten a flu shot (<0.05 by t test). They were also more likely to report having heart disease to a nearly significant degree (P = 0.057). It should also be noted that substantively large differences existed in other variables which did not meet the criteria for statistical significance given the sample sizes. Examples are arthritis (32.1 percent for telephone respondents vs 23.8 percent for telephone nonrespondents) and diabetes (22.1 vs 15.2 percent).

Table 3.

Estimates (without demographic adjustments) for 15 health variables by sample and mode

Variable	Sample for which telephone numbers were matched to addresses			Sample without matched telephone numbers
	Phone interviews only (N = 143)	In‐person interviews with phone NRs (N = 128)	Phone interviews plus in‐person interviews with NRs (N = 271)	In‐person interviews with those with no phone match (N = 166)
Mean BMI	29.2	27.5	28.0	27.4
Mean no. conditions	1.4	1.1a	1.2	0.96
Percent diabetes	22.1%	15.2%	17.2%	11.5%
Percent high blood pressure	41.0%	27.5%a	31.4%	28.4%
Percent heart disease	10.3%	4.1%	5.9%	9.7%
Percent lung disease	16.8%	17.8%	17.5%	15.4%
Percent arthritis	32.1%	23.8%	26.2%	11.5%b
Percent depression	16.8%	23.3%	20.7%	20.1%
Percent flu shot in past year	40.1%	53.1%a	49.4%	39.0%
Percent smoke	9.1%	11.5%	10.8%	18.7%b
Mean dr. visits past year	4.2	3.7	3.8	3.3
Mean days missed work past year	2.8	2.6	2.7	2.8
Percent with uninsured in HH past year	6.4%	5.7%	5.9%	7.8%
Percent rate health exc/v good	53.1%	56.9%	55.8%	54.1%
Percent visited dentist in past 6 months	58.8%	54.6%	55.8%	57.4%

Ns may vary slightly due to item nonresponse. Ns are approximately the same for all variables except for Days missed work, for which Ns are 63, 80, 143, and 101, respectively.

^aEstimates from telephone interviews and in‐person interviews with telephone nonrespondents differ by t test (P < 0.05).

^bEstimates from combined sample for addresses for which telephone numbers were matched to addresses and sample for which there was no telephone match differ by t test (P < 0.05).

With respect to the overall comparisons between those addresses with and without a telephone match, those without a telephone match were more likely to smoke and less likely to report having arthritis (P < 0.05). Here, too, there were differences that seem meaningful (49.5 vs 39.0 percent got flu shots and 17.2 vs 11.5 percent with diabetes) but did not reach 0.05 level of significance given the sample sizes.

Table 4 presents estimates that are all adjusted to demographically match the area population. There are seven variables for which the telephone‐based estimates were outside the interval defined by two standard errors from the combined estimate. The telephone respondents reported more diabetes, less depression, were less likely to have gotten a flu shot, were less likely to smoke, they reported more days missed from work due to illness, reported more household members who lacked health insurance, and they reported a higher average BMI. For the five of these seven variables measured as percentages, the differences between the telephone‐based estimates and the combined estimates are all 5 percentage points or greater. When the data from the interviews with telephone nonrespondents are added to the telephone interviews, none of the estimates lies outside two standard errors around the combined estimates. Only one estimate based on interviews with those for whom there was not a telephone match lay outside two standard errors from the combined estimates (percentage with arthritis).

Table 4.

Estimates (with demographic adjustments to match population characteristics) of 15 health variables by sample and mode

Variable	Sample for which telephone numbers were matched to addresses		Sample for which there were not telephone numbers	Total sample of addresses (N = 437)	95% Confidence Interval around total sample estimates	Bias (phone interviews –Total sample)	Standard error of bias
	Phone interviews only (N = 143)	Phone plus in‐person interviews with NRs (N = 271)	In‐person interviews with those with no phone match (N = 166)
Mean BMI	30.4a	28.5	27.2	28.1	27.1‐29.1	2.3b	0.868
Mean no. conditions	1.2	1.1	1.0	1.1	0.93‐1.2	0.1	0.102
Percent diabetes	20.2%a	13%	12%	14%	10%‐19.6%	0.062	0.041
Percent high blood pressure	30%	27%	29%	27%	22%‐33%	0.030	0.043
Percent heart disease	6%	5%	9%	7%	4%‐10%	−0.010	0.021
Percent lung disease	25%	23%	16%	21%	15%‐27%	0.040	0.050
Percent arthritis	18%	16%	11%a	16%	12%‐20%	0.020	0.034
Percent depression	17%a	24%	20%	23%	18%‐29%	−0.060	0.039
Percent flu shot in past year	24%a	48%	40%	42%	36%‐49%	−0.180b	0.043
Percent smoke	10%a	16%	20%	19%	14%‐25%	−0.090b	0.034
Mean dr. visits past year	3.9	3.8	3.4	3.8	3.0‐4.6	0.1	0.550
Mean days missed work past year	3.5a	3.0	2.9	2.8	2.2‐3.4	0.7	0.798
Percent with uninsured in HH past year	13%a	8%	7%	8%	4%‐12%	0.050	0.041
Percent rate health exc/v good	58%	56%	55%	55%	48%‐61%	0.030	0.052
Percent visited dentist in past 6 months	51%	54%	56%	55%	48%‐61%	−0.040	0.052

Ns may vary slightly due to item nonresponse. Ns are approximately the same for all variables except for Days missed work, for which Ns are 63, 143, 101, and 244, respectively.

^aEstimates lie outside the 95% confidence interval around estimates based on combined data from the total sample of addresses.

^bBias Estimate significantly different from 0.0 at 95% confidence level.

Also in Table 4, bias estimates between the telephone data and the combined estimates are displayed along with whether these estimates are statistically significantly different from zero. Three of these seven estimates are statistically significant (percent getting flu shot, percent who smoke, mean BMI), while four are not (percent diabetes, percent depression, percent uninsured, and mean days missed work) even though the differences are likely to be meaningful in a practical context. This is best indicated by differences in 6 percentage points or greater for diabetes and depression.

4. DISCUSSION

Almost all surveys will have some nonresponse. Also, as multimode survey data collection becomes more common, some people in a target population will not have a chance to respond in a particular mode. For any given subject area, it is an empirical question whether leaving out nonrespondents and/or people who cannot be surveyed via a particular mode will adversely affect survey estimates.

Researchers at the Pew Institute have published a series of papers demonstrating that many estimates based on telephone surveys with quite low response rates (as low as 10 percent) match estimates from surveys with higher response rates, particularly after adjustments are made for demographic differences in those who respond.5, 8, 9 Groves10 and Groves and Peytcheva11 showed that low response rates can affect some estimates and not others in the same survey. Hence, we thought it was quite plausible that the responses from those in an address‐based sample who could be interviewed by telephone might produce good estimates of health‐related variables once demographic adjustments were made. That turns out not to be the case. Of the 15 variables studied, only eight were within two standard errors of the combined estimates that included data from the telephone respondents and nonrespondents and those for whom there was not a telephone match. Also, it would have been very difficult to predict which estimates would be good and which distorted: For example, the percentages reporting heart disease and arthritis were similar between the phone and the combined samples, but the percentages reporting diabetes and depression were quite different. The number of visits to doctors was similar, but the number reporting flu shots were quite different. Moreover, the direction of bias differed by topic: the telephone sample was too high for diabetes but too low for depression. Thus, the notion that there is some kind of systematic adjustment that would “fix” the telephone‐based estimates seems fairly implausible.

Which is most to blame for the error, nonresponse, or drawing a sample from those for which a telephone number can be matched to an address? It appears the most likely culprit is nonresponse. The demographic characteristics of those interviewed by telephone are distinctively different from the known characteristics of the population. More informative is the fact that when the data from interviewed nonrespondents are combined with those interviewed by telephone, and the estimates adjusted to try to mirror the demographic characteristics of the area, none of these estimates remain outside of two standard errors from the best combined estimates. In contrast, there was only one adjusted estimate from those lacking a telephone match that lay outside of two standard errors from the combined estimate.

A potential limitation of this study is that the samples were comparatively small, and the weighting also increased the standard errors, which can produce false‐negative conclusions: real differences that do not reach traditional statistical standards. This was best displayed in the tests for “significant” bias. However, both of those facts actually increase the importance of the finding that so many telephone estimates were outside two standard errors around the combined estimates. While one could debate which is the more meaningful measure of when a difference “matters,” we would argue that under the constraint of the small sample sizes in this study, telephone values outside of two standard errors around the combined estimates are the more meaningful indicators of differences that “matter.”

The study was also limited by the comparatively small number of addresses matched to cell phone numbers. The ability to match cell phone numbers and addresses is said to be improving. However, the most important source of error in the telephone estimates was nonresponse.

Surveys that have relied on RDD telephone methods, as is the case for the BRFSS, are having to examine their data collection protocols in the face of declining telephone response rates and the decline of landlines. Especially surveys that focus on populations in a defined geographic area have to be attached to address‐based samples. The question then becomes how best to collect data from those households. All of the options, mail, telephone, IVR, and web have strengths and weaknesses. All of those approaches will have substantial nonresponse rates, particularly in the context of historical standards in the 60‐80 percent range. Some of the attractive options, including telephone and asking people to respond on the Internet, will exclude some potential respondents. The in‐person approach we used here would normally be considered too expensive for most surveys.

It is very tempting to look at studies such as those cited above and hope that the effects of nonresponse and sample frame limitations will not materially affect the estimates. In the absence of evidence to the contrary, these data serve to emphasize the importance of collecting data from as a high proportion of the target population as possible, including nonrespondents and those missed by a primary mode, rather than hoping that some kind of postsurvey adjustments can make up for substantive differences between those who do and do not respond to a primary mode. They also should help researchers using multimode designs with address‐based samples to understand some of the limitations of data collected in a telephone mode.

The problem of how to collect valid health survey data is critical, and the push to try various multimode approaches or use auxiliary variables to support more cost‐effective data collection is germane. These data demonstrate that health‐related estimates based on telephone interviews alone can be quite different from those that include nonrespondents and those addresses that cannot be matched to a telephone number. The results should help reinforce the importance of using survey designs that provide comprehensive sampling of the target populations and achieve high response rates.

Supporting information

 

 

Fowler FJ, Brenner PS, Buskirk TD, Roman A. Telephone health survey estimates: Effects of nonresponse and sample limitations. Health Serv Res. 2019;54:700–706. 10.1111/1475-6773.13110