Reevaluating the Need for Concern Regarding Noncoverage Bias in Landline Surveys

Abstract

Objectives. We used recent data to reexamine whether the exclusion of adults from households with no telephone or only wireless phones may bias estimates derived from health-related telephone surveys.

Methods. We calculated the difference between estimates for the full population of adults and estimates for adults with landline phones; data were from the 2007 National Health Interview Survey.

Results. When data from landline telephone surveys were weighted to match demographic characteristics of the full population, bias was generally less than 2 percentage points (range = 0.1–2.4). However, among young adults and low-income adults, we found greater bias (range = 1.7–5.9) for estimates of health insurance, smoking, binge drinking, influenza vaccination, and having a usual place for care.

Conclusions. From 2004 to 2007, the potential for noncoverage bias increased. Bias can be reduced through weighting adjustments. Therefore, telephone surveys limited to landline households may still be appropriate for health surveys of all adults and for surveys of subpopulations regarding health status. However, for some behavioral risk factors and health care service use indicators, caution is warranted when using landline surveys to draw inferences about young or low-income adults.

In 2006, in this journal, we examined nationally representative survey data from 2004 and early 2005 to determine whether the exclusion of adults without landline telephones biased population-based estimates derived from health-related random-digit-dial telephone surveys.¹ Noncoverage bias is determined both by the magnitude of the difference between persons with and without landline telephones for the variable of interest and by the percentage of persons without landline telephones in the population of interest.² In 2004 and early 2005, only 7.2% of adults did not have landline telephones, and we concluded that “noncoverage is not presently a reason to reject the continued use of general population telephone surveys to help guide public health policy and program decisions.”^1(p931)

In less than 3 years, the percentage of adults without landline telephones more than doubled. In 2007, 13.5% of adults lived in households with only wireless telephones, and an additional 1.7% of adults lived in households without any telephone service.³ Among certain subgroups, the percentage without landlines was even greater, reaching 30.6% for adults younger than 30 years and 21.6% for adults living in low-income households (defined as < 200% of the federal poverty level).

Our previously published conclusion, that noncoverage bias is not a concern,^1,4 needed to be revisited. We therefore used more recent data to reexamine whether the exclusion of adults from households with no telephone or only wireless phones may bias estimates derived from health-related telephone surveys.

METHODS

The National Health Interview Survey (NHIS) remains the leading source for data on the wireless-only population of the United States,^5,6,7 and every 6 months, the National Center for Health Statistics (NCHS) releases a report with the most up-to-date estimates available from the federal government concerning the size and characteristics of this population.³ This NCHS report describes the in-person household survey and the method for determining whether an adult lives in a household with landline telephones (landline household), with only cell phones (wireless-only household), or without any telephones (phoneless household).

We analyzed NHIS data collected in 2007. We selected health measures from data in the NHIS Sample Adult File, which includes health information for 1 civilian adult randomly selected from each interviewed family. In 2007, data on household telephone status and selected health measures were collected from 22 778 randomly selected adults. The overall response rate for the interview with the randomly selected adult was 67.8%.⁷

Variables

In accordance with the established NCHS definitions,³ we classified households as wireless-only if anyone living in the household had a working cellular telephone and if there were no working landline telephones inside the household. Individual ownership or use of the wireless telephone was not ascertained. All adults living in wireless-only households were classified as such. We used a similar approach to classify adults who lived in phoneless households with no working cellular or landline telephones.

Comparisons of adults from landline households with those from wireless-only households and those from phoneless households are regularly updated and disseminated by NCHS for 13 key indicators of health status, health behaviors, health care service use, and health care access.³ These indicators are a subset of the key adult health indicators published quarterly online by the NHIS Early Release Program.⁸ We used the specifications for these indicators that were published online.^3,8

Analyses

We calculated the potential noncoverage bias in landline telephone surveys that exclude wireless telephones and phoneless households by subtracting the population estimate for all adults from the estimated value for adults living in landline households. The population estimate and an unadjusted value for adults living in landline households were estimated from the published NHIS sampling weights without further adjustments.

Previous studies showed that young adults, Hispanic adults, and adults living in poverty are more likely to live in wireless-only households.^1,3 These demographic differences may explain some of the differences in health between adults with and without landlines. Therefore, we estimated an adjusted value for adults from landline households (technically, a predictive margin⁹) with a logistic regression that controlled for differences between adults from landline and nonlandline households on 4 demographic characteristics (age, race/ethnicity, gender, and education). The adjusted bias estimate (i.e., the predictive margin minus the true population value) approximated the actual bias that might be observed from a telephone survey that used sampling weights for adults living in landline households, after those weights were adjusted to match census estimates of the demographic composition of the population with and without landlines for these 4 demographic characteristics. A more complete explanation of this technique is available elsewhere.⁴ These 4 characteristics were selected because they are used for adjusting sampling weights for the Behavioral Risk Factor Surveillance System.¹⁰

Predictive margins, bias estimates, and standard errors were produced with SUDAAN software to account for the complex sample design of NHIS.¹¹ Statistical significance tests (α = 0.05) to determine whether absolute bias was greater than zero were calculated with a standard error of the difference term that accounted for covariance between the 2 groups (adults from landline households and all adults). Then, to determine whether unadjusted and adjusted bias estimates differed from each other, we used a pooled standard error of the difference term to compare the 2 bias estimates.

Because biases for subgroups of the population may not be adequately reflected in observed biases for the overall population, we repeated the analyses for 2 subgroups: young adults aged 18 to 29 years and adults living in low-income households.

RESULTS

Our previous analysis of data from 2004 and 2005 found that the potential noncoverage bias in landline-based telephone surveys did not exceed 2 percentage points for any health-related variable, and we concluded that differences this small were not practically significant.¹ By that admittedly debatable standard, random-digit-dial telephone surveys that excluded cell phone numbers were still viable in 2007 for health status surveys of adults (Tables 1 and 2). The noncoverage bias expected for these measures generally did not exceed 1 percentage point, even when sampling weights were unadjusted and when the sample was limited to populations more likely to live without landline telephones. The unadjusted bias estimate for self-reported health status for low-income adults was an important exception.

TABLE 1.

Unadjusted and Adjusted Estimates Of Noncoverage Bias for Selected Measures of Health-Related Behaviors, Health Status, and Health Care Service Use: United States, 2007

Measure	Population Estimate, % (SE)	Unadjusted Bias, Percentage Points (SE)	Adjusted Bias, Percentage Points (SE)
Health-related behaviors
≥ 5 or more alcoholic drinks in 1 d at least once in past y	20.6 (0.43)	−3.1 (0.29)*	−1.7 (0.24)*
Current smoker	19.8 (0.39)	− 2.0 (0.23)*	−0.9 (0.20)*
Engaged in regular leisure-time physical activity	30.8 (0.49)	−0.6 (0.27)*	−0.5 (0.26)
Health status
Health status described as excellent or very good	60.6 (0.46)	−0.9 (0.24)*	−0.2 (0.24)
Experienced serious psychological distress in past 30 d	2.7 (0.13)	−0.3 (0.07)*	−0.2 (0.07)*
Obese (adults aged ≥ 20 y)	26.7 (0.40)	0.5 (0.18)*	0.4 (0.16)*
Asthma episode in past 12 mo	3.8 (0.16)	−0.1 (0.09)	−0.1 (0.08)
Ever diagnosed with diabetes	7.8 (0.22)	0.7 (0.09)*	0.1 (0.11)
Health care service use
Has usual place to go for medical care	84.1 (0.35)	3.1 (0.22)*	1.3 (0.17)*
Received influenza vaccine during past 12 mo	29.5 (0.43)	2.5 (0.18)*	2.4 (0.17)*
Ever been tested for HIV	36.6 (0.44)	−1.9 (0.22)*	−0.8 (0.21)*
Failed to obtain needed medical care in past 12 mo because of financial barriers	8.0 (0.23)	−1.2 (0.12)*	−0.7 (0.08)*
Currently uninsured	16.7 (0.35)	−2.7 (0.21)*	−0.9 (0.17)*

Note. The population estimate was for all adults with nonmissing data for household telephone status. Unadjusted bias was calculated as the estimated value for adults living in landline telephone households minus the population estimate. Both values were weighted estimates from the 2007 National Health Interview Survey (unweighted sample size = 22 778). To calculate the adjusted bias, the estimated value for adults living in landline telephone households was recalculated (as a predictive margin for logistic regression) to correct for group differences with the full population on measures of age, race/ethnicity, gender, and education.

Source. Centers for Disease Control and Prevention, National Center for Health Statistics, National Health Interview Survey, 2007.⁷

*P ≤ .05 (value significantly different from 0).

TABLE 2.

Unadjusted and Adjusted Estimates of Noncoverage Bias for Selected Measures of Health-Related Behaviors, Health Status, and Health Care Service Use, by Age and Household Poverty Status: United States, 2007

	Young Adultsa (n = 4544)			Low-Income Personsb (n = 6591)
Measure	Population Estimate, % (SE)	Unadjusted Bias, Percentage Points (SE)	Adjusted Bias, Percentage Points (SE)	Population Estimate, % (SE)	Unadjusted Bias, Percentage Points (SE)	Adjusted Bias, Percentage Points (SE)
Health-related behaviors
≥ 5 or more alcoholic drinks in 1 d at least once in past y	34.5 (1.20)	−6.4 (1.00)*	−5.9 (0.99)*	18.8 (0.79)	−4.9 (0.64)*	−3.3 (0.58)*
Current smoker	24.5 (1.00)	−2.5 (0.79)*	−2.5 (0.80)*	26.6 (0.74)	−2.4 (0.49)*	−1.8 (0.45)*
Engaged in regular leisure-time physical activity	36.0 (1.08)	−2.2 (0.86)*	−1.7 (0.86)	22.2 (0.75)	−2.4 (0.54)*	−0.9 (0.47)
Health status
Health status described as excellent or very good	75.5 (0.90)	−0.6 (0.66)	−0.1 (0.67)	45.3 (0.88)	−4.4 (0.62)*	−1.3 (0.57)*
Experienced serious psychological distress in past 30 d	2.3 (0.29)	−0.2 (0.21)	−0.2 (0.22)	6.1 (0.37)	−0.4 (0.24)	−0.6 (0.24)*
Obese (adults aged ≥ 20 y)	20.0 (0.82)	0.6 (0.59)	0.4 (0.60)	30.5 (0.79)	1.4 (0.50)*	0.8 (0.48)
Asthma episode in past 12 mo	4.0 (0.39)	−0.2 (0.30)	−0.2 (0.30)	5.3 (0.36)	0.3 (0.20)	0.3 (0.16)*
Ever diagnosed with diabetes	1.1 (0.29)	0.0 (0.18)	0.0 (0.17)	10.1 (0.50)	1.9 (0.26)*	0.5 (0.26)
Health care service use
Has usual place to go for medical care	69.8 (0.98)	4.0 (0.75)*	4.3 (0.76)*	74.8 (0.79)	3.7 (0.53)*	2.0 (0.47)*
Received influenza vaccine during past 12 mo	13.9 (0.68)	0.0 (0.51)	2.2 (0.46)*	24.8 (0.73)	3.2 (0.45)*	1.9 (0.47)*
Ever been tested for HIV	40.9 (1.00)	−1.9 (0.75)*	−2.2 (0.75)*	41.8 (0.84)	−2.8 (0.54)*	−1.5 (0.50)*
Failed to obtain needed medical care in past 12 mo because of financial barriers	9.5 (0.56)	−1.9 (0.42)*	−2.0 (0.43)*	15.6 (0.59)	−1.5 (0.38)*	−1.6 (0.36)*
Currently uninsured	29.1 (0.95)	−2.2 (0.74)*	−2.9 (0.76)*	32.8 (0.88)	−3.4 (0.57)*	−1.7 (0.54)*

Note. The population estimate was for all adults within the age or income group with nonmissing data for household telephone status. Unadjusted bias was calculated as the estimated value for adults living in landline telephone households minus the population estimate. Both values were weighted estimates from the 2007 National Health Interview Survey. To calculate the adjusted bias, the estimated value for adults living in landline telephone households was recalculated (as a predictive margin for logistic regression) to correct for group differences with the full population on measures of race/ethnicity, gender, and education, as well as age (for low-income adults). For both samples, sample sizes were unweighted.

Source. Centers for Disease Control and Prevention, National Center for Health Statistics, National Health Interview Survey, 2007.⁷

^aAged 18 to 29 years.

^bHousehold income below 200% of the federal poverty level.

*P ≤ .05 (value significantly different from 0).

In addition, when the population of interest included all adults, the exclusion of cell phone numbers from random-digit-dial telephone surveys had little practical effect on adjusted estimates of health-related behaviors and health care service use. After adjustment, we found noncoverage bias greater than 1 percentage point (but less than 2.5 percentage points) only for binge drinking, having a usual place for care, and influenza vaccination. We had substantially similar findings, but with slightly higher levels of potential noncoverage bias, for low-income adults following adjustment.

However, when our population of interest was young adults, noncoverage was a reason to question the validity of landline-only surveys for studies of health-related behaviors and health care service use. In both unadjusted and adjusted estimates, the bias was greatest for binge drinking. Landline-only surveys could underestimate the prevalence of binge drinking among young adults by approximately 6 percentage points. We also found practically significant bias for smoking, HIV testing, financial barriers to medical care, and having a usual place for medical care. Adjustments did not significantly reduce the bias for these estimates.

The viability of landline-only surveys for health insurance estimates is also in doubt. With weighting adjustments, the noninsured rate was underestimated by 0.9 percentage points for all adults, 1.7 percentage points for low-income adults, and 2.9 percentage points for young adults. For all adults and for low-income adults, underestimations of this magnitude would qualify as practically insignificant by our original standard.¹ However, annual changes in insurance rates that are smaller than this magnitude have been considered important.¹² Without the weighting adjustments, the magnitude of the bias was significantly greater for all adults and for low-income adults.

DISCUSSION

For every measure, we found greater unadjusted noncoverage bias in the 2007 data than in the 2004 to 2005 data.¹ The bias more than doubled for estimates of binge drinking, flu vaccination, HIV testing, and financial barriers to needed medical care. The bias nearly doubled for estimates of smoking, having a usual place for care, and health insurance. The increased bias is attributable largely to the rapid increase in the prevalence of adults from wireless-only households. Differences between adults with and without landline telephones did not change substantially.^{1(Table 3),3(Table 4)}

The adjustment used to mimic the effect of demographic weighting attenuated, but did not eliminate, the potential noncoverage bias for some estimates. It is likely that noncoverage bias could be reduced further by the application of other poststratification adjustments. In the past decade, many telephone surveys increased the weights of data from transient telephone households (those that had an interruption in service during the previous year) to compensate for the exclusion of phoneless households.^13–15 Preliminary studies suggest that surveys may be able to compensate for the exclusion of wireless-only households by increasing the weights for wireless-mostly households (those landline households that receive nearly all calls on cell phones)¹⁶ or for transient wireless-only households (those that had only wireless phones at some time during the past year).^17,18 Additional research will be necessary to prove the worth of such adjustments.

Data collection systems that have relied on random-digit-dial surveys for years have been testing methods for including persons from wireless-only households in their samples.¹⁹ For example, the Behavioral Risk Factor Surveillance System (whose findings of declines in smoking and drinking rates have been attributed to noncoverage bias²⁰) recently conducted field tests dialing cell phone numbers²¹ and contacting adults from wireless-only households (and others) by mail.²² Best practices for cell phone surveys are not yet known, but a task force of the American Association for Public Opinion Research recently published an overview of the issues and challenges, including those related to coverage, sampling, nonresponse, measurement, weighting, ethics, and legal restrictions.²³ When it is not feasible to sample persons living in wireless-only households, the task force recommended that researchers discuss how their exclusion may affect the results. Our study should help some researchers with that discussion.

Random-digit-dial surveys with only landline telephone numbers should not be rejected out of hand as a data collection method. This method of sampling and this mode of data collection are still appropriate when exclusion of wireless-only households does not lead to noncoverage bias. This will be true for some survey topics and for some populations, but studies such as ours should be conducted and updated often to be certain when this is the case. Moreover, some topics and populations for which landline-only telephone surveys may not yield valid results now may become acceptable because of advances in sample weighting methods, other methodological innovations, and even changes in the composition of the landline population.

Despite our findings, there may never be a reliable list of topics and populations known to be amenable to a method that excludes wireless-only households, especially because the prevalence of such households continues to grow at a strong and steady pace. The regular inclusion of other samples (of cell phone numbers or of addresses) along with—or in place of—landline telephone numbers may not always be required to generate valid conclusions from the data collected but may be necessary for peace of mind.