A Nonresponse Bias Analysis of the Health Information National Trends Survey (HINTS)

Abstract

We conducted a nonresponse bias analysis of the Health Information National Trends Survey (HINTS) 4 Cycles 1 and 3, collected in 2011 and 2013, respectively, using three analysis methods: comparison of response rates for subgroups, comparison of estimates with weighting adjustments and external benchmarks, and level-of-effort analysis. Areas with higher concentrations of low SES, higher concentrations of young households, and higher concentrations of minority and Hispanic populations had lower response rates. Estimates of health information seeking behavior were higher in HINTS compared to the National Health Interview Survey (NHIS). The HINTS estimate of doctors always explaining things in a way that the patient understands was not significantly different from the same estimate from the Medical Expenditure Panel Survey (MEPS); however, the HINTS estimate of health professionals always spending enough time with the patient was significantly lower than the same estimate from MEPS. A level-of-effort analysis found that those who respond later in the survey field period were less likely to have looked for information about health in the past 12 months, but found only small differences between early and late respondents for the majority of estimates examined. There is some evidence that estimates from HINTS could be biased towards finding higher levels of health information seeking.

Introduction

The Health Information National Trends Survey (HINTS) uses a probability sample to routinely collect nationally representative data about the American public’s use of health-related and cancer-related information. The survey helps to identify new trends and practices in health communication. In addition, the survey provides information about how cancer risks are perceived and offers opportunities for social scientists to test new theories in health communication. The survey has been administered periodically since 2003, with methodology details published elsewhere¹. The most recent version of the survey, HINTS 4, included four data collection cycles over the course of three years, from 2011 to 2014. Rutten et al² published details about the HINTS 4 methodolgy in 2012.

The target population for HINTS is adults age 18 or older in the civilian non-institutionalized population of the United States. HINTS 4 used an address based sample of households in the U.S. based on a two-stage design. In the first stage, a random selection of stratified addresses was selected from a file of residential addresses. In the second stage, the adult with the next birthday was selected from within each sampled household that received a mail questionnaire. As with any survey, HINTS does not achieve complete response from the sampled households. A number of studies have documented a rise in survey nonresponse^3–6. The response rates for all cycles of HINTS 4 are shown in Table 1 and are consistent with other population-based health surveys that employ the same methodology.

Table 1.

HINTS 4 response rates by cycle

Cycle	Data collection dates	Overall response rate (AAPOR RR2)
1	October 25, 2011 – February 21, 2012	36.7%
2	October 9, 2012 – January 11, 2013	40.0%
3	September 6, 2013 – December 30, 2013	35.2%
4	August 20, 2014 – November 17, 2014	34.4%

Nonresponse bias occurs when the response rate is less than 100 percent and those who do not respond to a survey may have answered differently than those who do respond, resulting in biased results that do not accurately reflect the population of interest. The equation in (1) expresses nonresponse bias in a mean (ȳ_R) as a function of the nonresponse rate $\left(\frac{M}{N}\right)$ in the target population, where M is the number of nonrespondents and N the total number in the population, and the difference between the respondents (ȳ_r) and nonrespondents (ȳ_m) on the variables of interest in the target population (Cochran, 1977).

$$\mathit{Bias}({\overline{y}}_R)=\left(\frac{M}{N}\right)({\overline{Y}}_r-{\overline{Y}}_m)$$ (1)

This expression assumes that each sample member has a fixed probability of being a nonrespondent that is expressed as the nonresponse rate. From (1) we can conclude that there will be no nonresponse bias if everyone responds or if there is no difference between the respondents and nonrespondents on the survey variable of interest. Unfortunately, we typically do not know the difference in means between the respondents and nonrespondents for a given sample because the survey values are not available for nonrespondents. Hence, nonresponse bias can only be estimated using a statistical model. It is also possible to follow-up with a sample of nonresponsdents, but this is often expensive and often results in incomplete follow-up with the nonrespondents.

An alternative model of nonresponse, shown in expression 2, assumes that each sampled person has a propensity to respond and defines bias as a ratio of the covariance between the response propensities (σ_yp) and the survey variables of interest, and the average response propensity (p̄). Expression (2) makes it that clear that nonresponse bias occurs when response propensities are correlated with the attributes that are being measured in the survey. For example, HINTS estimates of seeking health information would be biased if individuals who are less likely to seek health information are also less likely to respond to the survey.

$$\text{Bias}({\overline{\mathrm{y}}}_{\mathrm{R}})=\frac{{\mathrm{\sigma }}_{\text{yp}}}{\overline{\mathrm{p}}}$$ (2)

Declining response rates over time have created a lot of interest in studies of nonresponse bias, and response rates are a very commonly used indicator of the quality of a survey. However, there is not always a clear relationship between response rates and nonresponse bias. Groves and Peytcheva (2008) conducted a meta-analysis of 59 methodological studies designed to estimate the magnitude of response bias. Many of the studies they examined enabled them to compare survey responses to known values from the sampling frame or information that was matched to the sample cases from administrative records. Other studies compared early responders with late responders or used responses to screener interviews to describe nonrespondents. They found only a small correlation (~.2) between response rates and absolute relative bias, and most of the variation in nonresponse was across estimates within the same survey rather than between surveys⁷. This suggests that there are important factors within each survey that influence the bias of particular estimates, and that low response rates are not always indicative of nonresponse bias. Other studies have similarly found relatively weak evidence of nonresponse bias related to response rates. Keeter et al. (2000) compared a telephone survey conducted over a five day period with a 36 percent response rate to a more rigorous survey conducted over an eight week period with a 60 percent response rate. They found few significant differences in the estimates across the two surveys. Most of the statistically significant differences in their study were across demographic variables, compared to questions about attention to media and engagement in politics, social trust and connectedness, and other social and political attitudes that were the key outcomes of interest in their study⁸. Curtin, Presser, and Singer (2000) examined the consequences of cutting the number of calls to complete an interview on the Survey of Consumer Attitudes (SCA). They specifically assessed the effect of excluding respondents who required refusal conversion, respondents who required more than five calls to complete the interview, and those who required more than two calls. They found no effect for excluding these groups on cross-sectional estimates of an Index of Consumer Sentiment, which is the primary measure from the SCA; the exclusion of resopndents who required more calls had only a small effect on other annual estimates⁹.

One previous study has examined the potential for nonresponse bias in prior iterations of HINTS. Cantor (2011) studied the HINTS 2 (2005) survey, which was conducted by telephone using a Random Digit Dialing (RDD) methodology. He benchmarked HINTS estimates to “gold standard” in-person surveys such as the Current Population Survey (CPS) and the National Health Interview Survey (NHIS), given that those surveys had higher response rates and coverage rates than HINTS. He found that compared to the CPS estimates, unadjusted HINTS estimates showed a higher percentage of older respondents and a lower percentage of younger respondents. The unadjusted HINTS estimates also showed a lower percentage of male respondents than the CPS, and a slightly lower percentage of non-Hispanic black respondents compared to the CPS. Cantor compared weighted estimates from HINTS (which adjust for these demographic differences) to estimates of health and health service measures on the NHIS. The fully weighted estimate of the percentage reporting “good” or “excellent” health from HINTS was 11 percentage points lower compared to the NHIS. HINTS found higher estimates of reporting negative emotions such as feeling sad or nervous compared to the NHIS. HINTS also found slightly higher percentages of the population who smoke and who have cancer. The HINTS fully weighted estimates of access to health insurance and never visiting a doctor in the past 12 months were within one or two percentage points of the NHIS estimates. Cantor also performed a level of effort analysis using HINTS data which compared the characteristics of easy to contact respondents (i.e., those who took few attempts to contact) to hard to contact respondents (i.e., those who took more attempts to contact). In general, there were small differences between the low and high effort groups. Those requiring more contact attempts were approximately 6–7 percentage points less likely to read the newspaper seven days per week and about five percentage points less likely to go to the Internet first for cancer information than those who required fewer contact attempts. All other differences between the high and low effort groups were less than four percentage points. Overall, the findings suggest that older individuals, less healthy individuals, and those who were more likely to be looking for information about cancer were over-represented in HINTS¹⁰.

The remainder of this paper investigates the nature of response to the HINTS 4 survey and the potential for nonresponse bias. The next section describes the HINTS weighting methodology, which is presented to provide context about estimation procedures that are used to help compensate for nonresponse. Full details of the weighting methodology can be found in Westat (2012) and Westat (2014)^11,12.

HINTS Weighting Methodology

Each cycle of HINTS 4 was delivered with a final sample weight and a set of 50 replicate weights. The final sample weight is used to produce population-level point estimates and the replicate weights are used for estimation of standard errors. The weighting process consisted of four steps: household base weight calculation, household nonresponse adjustment, initial person-level weights, and calibration adjustment.

First, a household level base weight that is equal to the reciprocal of the household’s probability of selection was created for each household in the sample. The base weight varied depending on which sampling stratum the household was from. The three sampling strata for HINTS 4 were areas with high concentrations of minority population, areas with low concentrations of minority population, and counties comprising Central Appalachia. Households in the high minority and Central Appalachia strata were oversampled and are thus given smaller base weights than households from the low minority stratum. An additional adjustment was made to the base weight for households that could receive mail multiple ways.

Next, weighting class cells were formed to adjust for household nonresponse to the survey. The weight for households within each weighting cell were adjusted by a factor equal to the reciprocal of the estimated response rate for the cell. The nonresponse weighting cells were defined based on measured characteristics of the sample households from the sampling frame. A search algorithm was used to identify variables that are highly correlated with household-level response. Characteristics used to form the weighting cells included sampling stratum, experimental treatment group (all adults versus next birthday selection method), Census region, mail route type, metropolitan status, and high Spanish linguistically isolated areas.

Third, each sampled adult in responding households was assigned an initial person-level weight. This weight was calculated by multiplying the nonresponse-adjusted household weight by the reciprocal of the sample person’s within-household probability of selection. In HINTS 4 Cycle 3 and most of the cases in HINTS 4 Cycle 1, this weight was equal to the number of adults in the household since only one person is selected within each household. HINTS 4 Cycle 1 included an experiment that requested all adults in the household to complete the survey. For this subset of respondents, the initial person-level weight was equal to the nonresponse-adjusted household weight if everyone in the household responded. In cases where some household members did not respond and adjustment factor was calculated by dividing the number of eligible adults in the household by the number of responding adults.

Fourth, a calibration adjustment was made to reduce the sampling variance of estimators through the use of reliable auxiliary information. Known population characteristics are used for auxiliary information and are referred to as control totals. In addition, the calibration adjusts for coverage and nonresponse bias assuming that the source of population characteristics is less biased. This was done by benchmarking the HINTS 4 estimates to known control totals. HINTS 4 used a raking process to calibrate to control totals for the following variables available in the 2010 American Community Survey (ACS) for Cycle 1 and the 2012 ACS for Cycle 3: age, gender, educational attainment, marital status, race, ethnicity, and census region. The raking process also included variables from the 2010 National Health Interview Survey (NHIS) for Cycle 1 and the 2012 NHIS for Cycle 3. These variables included the percent with health insurance and the percent that ever had cancer. As a result of this calibration process, estimates calculated from the HINTS 4 data for the control-total variables will agree with the source data for the control totals. For example, if HINTS calibrates to control totals from the ACS for age, the estimated age distribution in HINTS using the calibrated weights will exactly match age distribution for the ACS.

Methods

The nonresponse bias study of HINTS 4, described herein, included three types of analyses: comparison of response rates for subgroups, comparison of estimates with weighting adjustments and external benchmarks, and level-of-effort analysis. These analyses were conducted on HINTS 4 Cycles 1 and 3, collected in 2011 and 2013, respectively.

Comparison of Response Rates for Subgroups

First, differences in base-weighted response rates by subgroups identified on the sampling frame were examined. The goal of this analysis was to demonstrate which variables were correlated with nonresponse. The sampling frame included a number of characteristics of the Census tract of sampled respondents from the American Community Survey (ACS). Survey base weights were used in the calculation of response rates to account for unequal selection probabilities of sampled persons. Chi-square tests of independence were performed to evaluate the relationship between response status and characteristics of sample households, wherein the potential for nonresponse bias was determined by examining whether the sample characteristics were correlated with nonresponse.

Comparison of Estimates With Weighting Adjustments and External Benchmarks

Next, we examined the change in estimates after weighting adjustments and made comparisons to external benchmarks, described below. The extent of the reduction in nonresponse bias due to weighting depends on the correlation of the weighting class variables with survey estimates. The percent distribution of sample characteristics available for both respondents and nonrespondents before and after the nonresponse weighting adjustment was examined. For nonresponse adjustment to be effective, differences between the estimates after the adjustment and the estimates for the population should be reduced.

To compare the fully weighted estimates to external benchmarks based on large, nationally representative, federal surveys, we compared HINTS 4 estimates on health-related characteristics collected for the National Health Interview Survey (NHIS) and the Medical Expenditure Panel Survey (MEPS). NHIS and MEPS are often referred to as “gold standard” surveys in public health, given their in-person household interviewing techniques and their high response rates (~60%).

Level-of-Effort Analysis

We used a level-of-effort analysis to evaluate the potential for nonresponse bias in outcomes that are unique to HINTS (e.g., trust in information about health from different sources, and other health communication-specific items) and therefore not represented on other national surveys. It is assumed that hard-to-reach respondents are similar to nonrespondents and so any differences between the hard-to-reach and easy-to-reach respondents are theoretically indicative of nonresponse bias¹³.

There were three primary mailings in HINTS 4. Sampled households were initially sent a cover letter, questionnaire, return envelope, and $2 bill as a token incentive to complete the survey. A reminder/thank you postcard was sent to sample households approximately two weeks after the initial mailing. A second questionnaire was mailed to sampled households that did not return a completed questionnaire after receipt of the reminder postcard approximately four weeks after the initial mailing. Finally, a third questionnaire was sent to the remaining sample households that did not return a completed questionnaire approximately six weeks after the initial mailing.¹ Table 2 shows the percentage of the final respondents that responded at each stage for Cycle 1 and Cycle 3. For example, 72% of the final number of respondents responded to the first mailing, 22% responded to the second mailing, and only 4% respondend to the third mailing. The level-of-effort analysis shows what the weighted estimate would have been if the data collection period was cut-off after each mailing.

Table 2.

Percentage of respondents at each mailing

Mailing	HINTS 4 Cycle 1	HINTS 4 Cycle 3
1	72%	72%
2	22%	20%
3	4%	8%
Total	100%	100%
N	3696	3185

Results

Comparison of Response Rates for Subgroups

Here, we summarize the response rates for households according to their characteristics that were measured on the sampling frame. Table 3 illustrates the base-weighted response rates by selected characteristics of the household, and illustrates the distribution of the characteristics for the respondents and nonrespondents.

Table 3.

Response rates and distribution of respondents and nonrespondents (%s) by selected characteristics of sample households

Characteristic	HINTS 4 Cycle 1				HINTS 4 Cycle 3
	RR	Resp.	Nonresp.	Sig.	RR	Resp.	Nonresp.	Sig.
	Percent
Region
Northeast	37.0	17.9	18.2	***	36.6	19.0	17.9	***
Midwest	42.1	25.1	20.7		40.1	24.7	20.0
South	34.7	34.6	39.1		31.1	33.2	40.0
West	37.9	22.4	22.0		36.2	23.1	22.1
MSA
Metro	36.8	81.2	83.6	*	34.9	81.7	82.8	*
Non-metro	40.8	18.8	16.4		36.6	18.3	17.2
Route Type
Street address	40.4	78.0	68.9	***	37.5	77.0	69.5	***
Other	29.7	22.0	31.1		29.1	23.0	30.5
Below 100 percent poverty
0–25%	41.2	40.1	34.3	***	38.7	44.8	38.6	***
26–50%	38.4	29.2	28.1		37.0	29.4	27.1
51–75%	34.6	19.3	21.8		30.3	16.1	20.1
76–100%	30.2	11.4	15.8		27.1	9.7	14.2
College degree
0–25%	29.9	12.9	18.0	***	27.6	11.0	15.7	***
26–50%	38.3	23.5	22.7		32.4	19.7	22.4
51–75%	35.4	25.0	27.3		37.2	28.4	26.0
76–100%	42.1	38.7	32.0		38.2	40.8	35.9
Hispanic
0–25%	40.9	32.7	28.3	***	38.7	36.9	31.7	***
26–50%	37.5	32.4	32.3		36.4	35.7	33.8
51–75%	37.8	26.0	25.6		32.4	19.8	22.4
76–100%	28.2	9.0	13.7		25.6	7.6	12.0
Black
0–25%	42.7	38.1	30.6	***	40.2	42.5	34.4	***
26–50%	37.5	31.3	31.3		35.5	32.1	31.7
51–75%	33.8	20.9	24.6		29.5	17.4	22.7
76–100%	30.0	9.7	13.5		27.8	7.9	11.2

We found some difference in response rates by region. Cycle 1 response rates were higher for households from the Midwest (42.1%) and lower for households from the South (34.7%). This was also reflected in the distribution of the respondents and nonrespondents. Households from the Midwest represent 25.1 percent of all responding households compared to 20.7 percent of households that did not respond. Households from the south represent approximately 34.6 percent of all responding households compared to 39.1 percent of households that did not respond. This pattern was similar in Cycle 3. Response rates were higher for households from non-metro areas compared to households from metro areas. Households with a street address had significantly higher response rates than those with some other form of address. Response rates were lower for areas with higher rates of high poverty. Similarly, Areas that have a higher percentage of persons with a college degree have higher response rates than areas with fewer persons with a college degree.

Response rates also differed by the composition of race and ethnicity in sample areas as shown in Table 6. Areas where the Census records higher concentrations of Hispanics have lower response rates. Although the survey was administered in both English and Spanish, response rates were lower for linguistically isolated areas (21.8%) versus other areas (37.7%) in Cycle 1. A similar trend emerged for race. In Cycle 1, areas with black persons representing 0–25 percent of the population had a response rate of 42.7 percent, whereas areas with black persons representing 76–100 percent of the population have a response rate of 30.0 percent.

Table 6.

Changes in estimates of health, health services, medical records, beliefs about cancer, and cancer by mailing for HINTS 4 Cycle 1 and HINTS 4 Cycle 3

Characteristics	HINTS 4 Cycle 1 Mailing			HINTS 4 Cycle 3 Mailing
	1	2	3	1	2	3
Health and health services
Fair or poor health	15.4	15.7	15.1	14.6	14.7	13.9
Very confident or completely confident take care of own health	67.2	68.0	68.0	66.6	67.1	67.9
Feeling nervous, anxious, or on edge	43.1	42.5	42.4	NA	NA	NA
Have a doctor seen most often	65.9	64.7	63.4	63.3	63.7	63.6
Eat 2 or more cups of fruit per day	68.9	68.5	68.9	72.0	73.1	72.7
Eat 2 or more cups of vegetables per day	76.4	76.2	76.4	78.7	78.5	78.4
Access to health insurance	81.3	81.3	81.3	83.0	83.0	83.0
Smoked at least 100 cigarettes in lifetime	38.7	39.0	38.5	43.6	44.6	44.5
Heard of HPV	NA	NA	NA	68.0	67.5	67.7
Quality of care good, fair, or poor	23.7	24.1	25.8	26.1	25.9	26.5
Medical records
Have doctors who maintain medical information in a computerized record system	86.2	85.1*	84.0	88.7	87.8	87.7
Very important doctors share medical information electronically	64.9	64.1	64.4	62.9	62.9	63.0
Very important you should be able to get your own medical information electronically	70.8	70.0	70.3	65.5	65.3	64.9
Beliefs about cancer
Likely or very likely to get cancer in lifetime	36.1	35.7	35.1	31.7	32.3	32.3
Moderately or extremely worried about getting cancer	82.0	81.4	81.8	82.8	82.8	82.5
Agree it seems like everything causes cancer	61.5	62.5	63.1	67.1	66.3	65.8
Agree there’s not much you can do to lower your chances of getting cancer	32.3	32.9	33.7	29.5	29.4	29.2
Agree there are so many different recommendations about preventing cancer, it’s hard to know which ones to follow	71.2	70.8	71.1	74.8	72.7	72.1
Cancer history
Have family members who have had cancer	68.2*	65.3	64.7	66.6	66.7	65.9

We also found that areas with a higher percentage of younger persons (under age 35), regardless of sex, have lower response rates. For example, in Cycle 1 areas with 0–25 percent of the population in the male 18–24 age group have a 40.5 percent response rate, whereas areas with 76–100 percent of the population in this age group have a 32.5 percent response rate.

Comparison of Estimates With Weighting Adjustments and External Benchmarks

Table 4 compares selected HINTS estimates (e.g., Internet use, health measures, and health communication measures) using weights at different stages of the weighting process. The table also includes the estimates from external data sources for comparison. The top panel of the table compares HINTS estimates with comparable estimates from the NHIS. The bottom panel of the table compares HINTS estimates with comparable estimates from the MEPS.

Table 4.

Comparison of HINTS estimates of Internet use, health, and health communications characteristics with different weights and estimates from the NHIS and MEPS

Comparisons with NHIS	HINTS 4 Cycle 1				HINTS 4 Cycle 3
	Base-weighted estimate	Nonresponse adjusted estimate	Final calibrated estimate	NHIS estimate	Base-weighted estimate	Nonresponse adjusted estimate	Final calibrated estimate	NHIS estimate
	Percent
Access to Internet	78.8	78.9	78.1	70.9*	78.9	77.9	78.3	76.4
Excellent, very good, or good health	87.5	87.4	84.9	86.9	87.7	86.8	86.1	87.1
Smoked 100+ cigarettes in life time	40.0	40.0	38.5	40.9	43.2	43.1	44.5	39.8*
Never visited doctor	15.5*	15.5*	21.2	19.0*	14.8*	15.3*	19.4	19.6
Looked for health information on the Internet (Internet users only)	78.7	78.6	78.0	57.9*	80.9	80.7	79.4	61.5*
Used Internet to communicate with doctor (Internet users only)	21.9*	21.9*	19.1	8.0*	33.3*	32.4*	29.7	9.4*
Comparisons with MEPS	Base-weighted estimate	Nonresponse adjusted estimate	Final calibrated estimate	MEPS estimate	Base-weighted estimate	Nonresponse adjusted estimate	Final calibrated estimate	MEPS estimate
Health professionals always explain things in a way you understand	61.3	61.1	61.0	61.4	64.5*	63.9*	61.9	NA
In past 12 months, health professionals always spend enough time with you	46.3	46.2	44.6	52.4*	50.0*	49.5*	46.3	NA

NOTE: The 2011 NHIS estimates are shown in the table for Cycle 1 and the 2013 NHIS estimates are shown for Cycle 3 for all characteristics except “Looked for health information on the Internet” and “Used Internet to communicate with doctor”. These variables are restricted to Internet users only. The NHIS did not include a web use question in 2011 so the table compares the Cycle 1 estimates for these characteristics to the 2012 NHIS estimates. Similarly the measure of access to the Internet from the NHIS for Cycle 1 is estimated from the 2012 NHIS. Significance tests are the result of tests comparing all other estimates with the final calibrated estimates from each cycle of HINTS.

^*p<.05

Table 4 shows that there are small changes between the base-weighted estimates and the nonresponse weight adjusted estimates. The changes in estimates were somewhat larger between the nonresponse weight adjusted estimates and the final calibrated weights. Statistically significant differences between the base-weighted estimates or nonresponse adjusted estimates and the final calibrated estimates are denoted with an asterisk.. The difference in estimates between HINTS and the NHIS vary in size. For example, the HINTS estimate of Internet access in Cycle 1 is significantly higher at 78.1 percent compared to 70.9 percent for the 2012 NHIS. The difference between these estimates is not significant in Cycle 3. The HINTS estimates of the percent in good, very good, or excellent health are only about one or two percentage points lower than the same estimates from the NHIS. Comparisons of estimates of smoking vary between cycles. In Cycle 1, HINTS estimates that 38.5 percent of the population has smoked 100 or more cigarettes in their entire life compared to 40.9 percent from the NHIS, which is not statistically significant. In Cycle 3, the HINTS estimate is almost five percentage points higher than the NHIS estimate, which is statistically significant.

The Cycle 1 estimate for never visiting a doctor in the past 12 months is significantly different from the 2011 NHIS, but the difference is only about two percentage points and the Cycle 3 estimate is not significantly different from the NHIS. Two estimates that showed large differences between HINTS and the NHIS are “looked for health information on the Internet” and “used Internet to communicate with doctor”. Both of these estimates were restricted to Internet users only. The Cycle 1 estimate of looking for health information on the Internet was 78.0 percent compared to 57.9 percent from the 2012 NHIS. The Cycle 1 estimate for using the Internet to communicate with a doctor is 19.1 percent compared to 8.0 percent from the 2012 NHIS. Similar patterns emerged for Cycle 3.

Two questions from HINTS were compared to similar questions from the MEPS. The Cycle 1 estimate of doctors always explaining things in a way that the respondent understands was not significantly different at 61.0 percent compared to 61.4 percent from the MEPS. The Cycle 1 estimate of health professionals spending enough time with the respondent was significantly lower at 44.6 percent compared to 52.4 percent from MEPS.

Level-of-Effort Analysis

Our last analysis examined how the estimates might have changed over the course of the field period for each cycle of HINTS 4. Table 5 shows estimates of demographics, communication, and health communication characteristics. The columns of the table show what the estimate of each characteristic would have been if the field period ended after that specific mailing. An assumption underlying this analysis is that those who are harder to reach are more similar to the nonrespondents than those who are easier to reach¹³. Hence, if the estimates differ between hard to reach households and easy to reach households, this could potentially suggest nonresponse bias.

Table 5.

Changes in estimates of demographics, communication, and health communication by mailing for HINTS 4 Cycle 1 and HINTS 4 Cycle 3

Characteristics	HINTS 4 Cycle 1 mailing			HINTS 4 Cycle 3 mailing
	1	2	3	1	2	3
Demographics
Income $100k or more	13.9	13.5	13.7	14.0	14.0	13.9
Retired	16.9	16.6	16.5	17.3	17.1	17.1
Employed	57.9	57.1	56.1	61.9	62.2	61.7
Own home	58.5	60.5	60.5	61.3	62.2	61.8
Born in USA	87.0	86.7	85.6	85.1*	84.6	83.5
Household with children	32.1	38.3	34.0	38.6	38.1	38.0
Single-person household	19.2	18.6	18.0	19.6	18.8	18.7
Communication
Use Internet	79.9	77.9	78.1	79.2	78.5	78.3
Access Internet through a cellular network	46.2	45.9	47.0	57.2	57.4	56.8
Watch TV more than 5 hours per day	20.4	20.1	19.7	16.6	16.0	16.0
Health communication
Looked for information about health or medical topics from any source in past 12 months	83.3*	81.3*	79.9	81.2*	78.7*	77.5
Very or completely confident could get advice or information on health or medical topics	62.2	61.3	61.2	55.2	55.2	55.5
Trust a doctor regarding health/medical topics a lot	72.8	72.2*	70.9	69.5	69.9	68.9
Trust health information in magazines a lot	3.3	3.0	3.0	NA	NA	NA
Would go to doctor regarding health or medial topics first	52.4	52.0	52.2	50.7	51.0	50.3
Ever sought cancer information for self	53.7	52.4	51.8	56.7	55.7	55.0
Heard or read about genetic tests	37.6	37.2	36.9	37.0	36.3	36.5
Told by doctor that they could choose whether or not to have a test for colon cancer	33.7	34.1	34.5	32.8*	35.0	34.9

Most of the demographic characteristics, in the top panel of Table 5, show only minor changes in the estimates across the field period. Significant differences between the estimates after the initial mailings (mailings 1 and 2) and the estimates after the final mailing (mailing 3) were found for the percent born in the USA in Cycle 3. The table shows that for both Cycle 1 and Cycle 3 there was a trend for the percentage of respondents born in the USA to be lower after the final mailing. In other words, foreign born respondents were more likely to respond later in the field period. We did not observe any significant differences in the estimates across mailings for the communication variables.

In general we observe only small changes in various estimates of health communication, presented in the bottom panel of Table 5. One exception is with the estimates of looking for information about health or medical topics from any source in the past 12 months. The difference between the estimate after the initial mailings and the final mailing was significant in both cycles. The estimates from the initial mailings are 3 to 4 percentage points higher for the initial mailings compared to the estimate after the final mailing. There was also a statistically significant difference in Cycle 1 between the second mailing and the final mailing with the estimate of trusting a doctor regarding health or medical topics a lot; however, there was no difference in Cycle 3. The percent reporting that they were told by a doctor that they could choose whether or not to have a test for colon cancer was significantly higher for the first mailing compared to the final estimate in Cycle 3, but there were no differences in Cycle 1.

Table 6 displays estimates of health care and services, medical record usage, beliefs about cancer, and cancer history. Overall we found few differences in those estimates. Any differences that occured were generally small and were not consistent across cycles.

Discussion

The four cycles of surveys that have been conducted as part of HINTS 4 have experienced response rates ranging from 34 percent to 40 percent. This nonresponse bias analysis used three analysis methods to examine several measures from HINTS 4 Cycle 1 and Cycle 3 to determine the extent to which the data may be impacted by nonresponse bias.

The study first compared response rates for subgroups identified by characteristics of households from the sampling frame. This analysis showed that propensity to respond to HINTS may be affected by several factors. The location of the sampled household was related to response with households in the Midwest having the highest response and the lowest response came households from the South. Areas with higher socio-economic status as measured by poverty level, income, and education have higher response than relatively lower SES areas. Areas with fewer young households also had higher response rates. Finally, areas with higher concentrations of minority and Hispanic populations had lower response rates. The analysis of response rates by subgroups in the HINTS 4 sample are consistent with findings from other surveys. There is unlikely to be any one specific cause of nonresponse to any survey, but there are some explanations to consider. Social capital theory is a frequently posited explanation for nonresponse. The theory suggests that social networks have value and that social contact leads to productive interaction and cooperation that can be manifested in activity such as civic engagement¹⁴. Surveys are one form of civic engagement. Tourangeau and Plewes (2013) highlight that one important source of social capital is education, which is one component of SES⁶. Hence, differences in social capital between groups with different levels of SES could in large part be due to the different levels of social connectedness or social capital between these groups. Topic interest is also known to play a role in survey response¹⁵. It is plausible that younger individuals will have less experience with health care and will be less likely to respond to surveys about health information seeking.

Many of these factors that are associated with response are addressed through weighting adjustments. Although we found relatively small changes between the sample base-weighted estimates and the household nonresponse adjusted estimate, there are often substantial differences with the final calibrated estimates, which are adjusted at the person level. Bias occurs when nonresponse is correlated with the survey variables of interest. The nonresponse weighting adjustments, at the household and person levels, assume that the respondents and nonrespondents are similar within weighting classes. Therefore, bias associated with nonresponse will be reduced to the extent that this assumption holds. In other words, the nonresponse adjustments assume that the data are missing at random given the nonresponse weighting classes.

The comparisons of HINTS estimates to external benchmarks suggests that some of the survey estimates are close to the estimates from other national surveys with higher response rates and a lower likelihood of nonresponse bias. For example, measures of general health, doctor visits, and lifetime smoking are close to estimates from the NHIS. However, estimates related to health information seeking behavior are higher for HINTS compared to the NHIS. It is difficult to attribute all of this difference to nonresponse since the context in which the questions are asked is very different across the two surveys. There were some differences in the question wording between the NHIS and HINTS for these items. For example, the question about using the Internet to communicate with the doctor from HINTS asks about using “e-mail or the Internet,” whereas, the NHIS only mentions e-mail. The NHIS questions ask about using computers, which may exclude the use of mobile devices that can also connect to the Internet. However, one source of differences could be the context in which the items appear. The HINTS 4 questions are in the context of a survey that is about health communication and follow a section in the questionnaire on different types of health communication, whereas the NHIS questions appear in the context of a more general health survey with very few health communication questions. Thus, HINTS respondents may have been ‘primed’ or more likely to answer these questions showing more of the behavior. Mode of administration (mail versus in-person) might also play a role. In comparison to MEPS, the HINTS estimate of “health professionals always explain things in a way that you can understand” was not significantly different from the same MEPS estimate. In contrast, the HINTS estimate of “health professionals always spend enough time with you” was approximately eight percent lower than the MEPS estimate. The slightly lower estimate from HINTS could be a sign of mode differences with respondents being somewhat more willing to admit dissatisfaction with their doctor without an interviewer present.

Finally, since there were relatively few variables available for benchmarking a unique survey like HINTS, we performed a level-of-effort analysis that examined how several estimates changed over the course of the field period. This analysis found relatively few instances of estimates changing significantly over the course of the field period. Most differences that were found were not consistent between Cycle 1 and Cycle 3. However, the level-of-effort analysis did show that those who respond later in the survey are less likely to have looked for information about health or medical topics in the past 12 months. The final estimate is two to three percentage points lower than the estimate at earlier stages of the field period. Assuming that the later respondents are similar to the nonrespondents, this may indicate some bias for this type of characteristic.

Conclusion

Overall, many of the demographic influences on nonresponse to the HINTS such as age and SES can be compensated for with standard weighting procedures. This weighting reduces the bias to the extent that these demographics are correlated with health information seeking behavior. There is some evidence from the level of effort analyses and comparisons with other surveys that estimates in HINTS could be biased towards finding higher levels of health information seeking. However, it is difficult to tell the exact extent of this bias due to the unique nature of HINTS with its focus on health information seeking and mode differences between HINTS and other surveys.