Health Benefits Offer Rates: Is There a Nonresponse Bias?

Abstract

Objective

To determine whether a nonresponse bias exists in the offer rate for health benefits in firms with fewer than 50 workers and to present a simple adjustment to correct for observed bias.

Data Sources

The 2003 Employer Health Benefits Survey (EHBS) conducted by the Kaiser Family Foundation and Health Research and Educational Trust, and a follow-up survey of nonrespondents to the 2003 EHBS.

Study Design

We conducted a follow-up survey to the 2003 EHBS to collect health benefits offering data from firms with fewer than 50 workers. We used McNemar's test to verify that the follow-up survey provided results comparable to the EHBS, and t-tests were used to determine nonresponse bias. We applied a simple weighting adjustment to the EHBS.

Data Collection

The data for both the EHBS and the follow-up survey were collected by the same survey research firm. The EHBS interviews the person most knowledgeable about the firm's health benefits, while the follow-up survey interviews the first person who answers the telephone whether they are the most knowledgeable or not.

Principal Findings

Firms with 3–9 workers were more likely to exhibit a bias than were firms with 10–24 workers and 25–49 workers. Although the calculated bias for each size category was not significant, there is sufficient evidence to warrant caution when reporting offer rates.

Conclusions

Survey nonresponse in the EHBS produces an upward bias on estimates for the offer rates of small firms. Although not significant, this upward bias is because of nonresponse by small firms that do not offer health benefits. Our research is limited in that we only control for differences in the size of the firm.

Over 44 million Americans are without health insurance, leaving the U.S. alone among the industrialized nations of the world in which a substantial share of its population is without health coverage. About 80 percent of the uninsured are families in which the head-of-household works either full- or part-time (Hoffman and Wang 2003). Roughly half of the uninsured work for firms with fewer than 25 workers (Fronstin 2001). Unlike firms with 50 or more workers, many small firms simply do not offer health insurance to their workers (Gabel et al. 2002).

Firms with fewer than 50 workers offering employer-sponsored health benefits have many characteristics distinguishing them from comparably sized firms that do not offer coverage (Fronstin and Helman 2003). These differences may partly explain why some companies find that offering health benefits affects employee recruitment, retention, and performance, and why some companies do not offer health benefits despite the negative impact on their business.

Several surveys collect data on employer-sponsored health benefits from firms with fewer than 50 workers.1 Persuading small firms to participate in these surveys can be daunting. Response rates for the smallest of firms are relatively low, increasing the chance for nonresponse bias.2 The relatively low response rate and the different characteristics of small firms that do and do not offer health benefits increase the probability that survey estimates for the offer rates of small firms will misrepresent the true rate. These problems are further confounded by differences in the incentives that firms may have to participate in these surveys. Interviews with firms participating in the annual Employer Health Benefits Survey (EHBS) sponsored by the Kaiser Family Foundation and Health Research and Educational Trust indicate that the major reason that benefit managers opt to participate in the survey is to obtain information to compare their health plan offerings with that of their peers. Hence, firms that do not offer health benefits lack one of the key incentives for participating in the EHBS. Thus, there are reasons to expect EHBS respondents to be more likely to offer employee health insurance than EHBS nonrespondents.

Biased estimates have significant public policy implications. The media and policy communities sometimes interpret a change in the percentage of small firms offering coverage as a lead indicator of future trends in uninsurance. The small firm offer rate is useful for this purpose because data from employer benefit surveys are usually available a year before information is available from the databases that estimate the number of uninsured.

This article examines the impact of nonresponse bias on estimates of the offer rate for health benefits among small firms. We develop a simple adjustment to the constructed sampling weights to correct for this bias and apply it to the 2003 EHBS. We conclude with suggestions on how to reduce nonresponse bias in future employer benefits surveys.

Background

Survey nonresponse is the failure to obtain complete information from all members of a survey sample (Groves 1989). There are two types of nonresponse: unit nonresponse, which results from sampled units not responding to a survey, and item nonresponse, which results from a survey respondent not answering a particular question from an otherwise complete survey. The bias because of nonresponse is the product of the proportion of nonrespondents (either unit or item) and the difference in the means between respondents and nonrespondents. Therefore, the amount of nonresponse bias is associated with both the response rate and the difference between respondent and nonrespondent estimates (Cochran 1977; Groves and Couper 2002).

There are several ways to analyze nonresponse bias. One approach is to assume that nonrespondents are more like late respondents than early respondents, and then utilize data from the late respondents to determine nonresponse bias (see Filion 1975; Filion 1976; Siemiatycki and Campbell 1984; Fowler 1998; Iannacchionc 1998; Bose 2001 for examples). A second approach is to link secondary data to both respondents and nonrespondents where complete information on the sample is limited (see Stec, Lavrakas, and Stasny 1999 as an example). A third approach is to conduct a follow-up survey (see O'Neil 1979; Black and Safir 2000; Biemer 2001; Bose 2001 for examples).

Survey Data

The EHBS has been conducted annually since 1987. The EHBS asks participating firms as many as 400 questions about its largest health plans offered to its employees. The EHBS draws its sample from a Dun & Bradstreet list of the nation's private and public firms. The 2003 EHBS was collected by Computer-Assisted Telephone Interview from January to early May 2003. The final sample size was 1,856 firms with three or more workers. The response rate was 50 percent.

Every year, the EHBS asks firms question A6 to determine whether or not a firm offers health benefits to its employees. The question reads, “Does your company offer or contribute to a health insurance program for your employees?” Over the years, it has become evident that firms not offering health benefits—especially smaller ones—are less likely to participate in the full EHBS. To gain additional information about health benefit offerings of survey nonrespondents, we ask question A6 of all firms who decline to participate in the full EHBS. An additional 952 firms responded yielding a total sample of 2,808 firms. The item response rate for question A6 was 76 percent.

Steps are taken each year to encourage participation in the final survey by resampling firms that participated in the EHBS during the previous 2 years. The very smallest of firms (three to nine workers) are not included in this panel and are newly drawn each year because of concerns that, among these firms, those that offered health benefits would be more likely to participate again.

Sampling weights are created following a standard process. We first calculate the basic weight and then adjust for unit nonresponse. Next, we trim overly influential weights in order to reduce their overall influence as compared with other respondents. Finally, we apply a poststratification adjustment using a raking method.3 All point estimates and t-tests are conducted with SUDAAN, which computes appropriate variance estimates by controlling for the complex design of the survey.

Methods

To determine whether a nonresponse bias exists, we conducted a follow-up survey of small firms (less than 50 workers) that either did not respond to the 2003 EHBS or only responded to question A6. We limited the follow-up survey to small firms because the vast majority of firms with 50 or more workers offer health benefits to their employees (Gabel et al. 2003). Additionally, firms with 50 or more workers are more likely to participate in the EHBS4 and are less likely to bias the estimate for offer rate. National Research LLC (NR) conducted the full and follow-up survey using the same interviewers.5 The field period for the follow-up survey began 1 week after closing the field period of the original survey and lasted less than 1 week, closing within 2 weeks of the original survey.

The follow-up survey asked the A6 question to those firms not responding to the 2003 EHBS, and reasked the A6 question to those firms that only answered that question in the 2003 EHBS. In the 2003 EHBS, the person most knowledgeable about the firm's health benefits was interviewed. In the 2003 follow-up survey, we asked the first person who answered the telephone (often a receptionist or secretary) whether the firm offered health benefits, regardless of whether that person was the most knowledgeable or not. No attempt was made to remind the firm that they had previously been contacted to participate in the 2003 EHBS.6

Results

National Research LLC obtained follow-up responses from 1,119 firms with fewer than 50 workers out of 1,626 contacted (69 percent response rate). We obtained responses from 407 firms that answered question A6 in both the EHBS and the follow-up survey. Table 1 presents crosstabulations of the responses by firms answering question A6 in both surveys. We conducted a McNemar's test on each crosstabulation to verify that the two approaches for collecting the health benefit offering information provided comparable results.7 The adjusted χ² test statistic for the 10–24 worker group was 1.500 (df=1, p=.307), and 0.600 for the 25–49 worker group (df=1, p=.607), indicating that the differences were not statistically significant. However, for firms with three to nine workers the χ² test statistic was 4.840 (df=1, p=.043), suggesting a significant difference. These findings suggest that comparisons of the results from the two surveys—one asking the question of the “person most knowledgeable about health benefits” and the other asking the person who answered the phone—can be meaningful. However, we must be cautious when analyzing the bias in firms with three to nine workers because of the differences in the way that the follow-up survey was administered as compared with the EHBS.

Table 1.

Comparison of Responses for Firms Responding to Question A6 in Both the EHBS and the Follow-Up Survey

	Number of Firms from Follow-Up Survey Offering Health Benefits
Number of Firms from EHBS Offering Health Benefits	Yes	No	Total
3–9 Workers
Yes	90	18	108
No	7	41	48
Total	97	59	156
10–24 Workers
Yes	125	15	140
No	9	16	25
Total	134	31	165
25–49 Workers
Yes	63	6	69
No	9	8	17
Total	72	14	86

Source: 2003 Kaiser/HRET EHBS and follow-up survey.

EHBS, Employer Health Benefits Survey; HRET, Health Research and Educational Trust.

Our next step was to compare the EHBS respondents (n=979), including those firms responding to the full EHBS and not resurveyed (n=572), with the follow-up respondents who did not participate in the EHBS (n=712). Table 2 presents the proportion of firms offering health benefits by firm size. In all size groupings, the proportion of firms offering health benefits was smaller in the follow-up survey than in the EHBS. In particular, among the very smallest firms (three to nine workers), the offer rate for EHBS respondents was significantly smaller by over 8 percentage points from that computed from the follow-up survey (50.5 versus 58.7 percent).

Table 2.

Unweighted Health Benefit Offer Rates by Firm Size by Survey

	EHBS			Follow-Up Survey
Firm Size	Offer Rate	Sample Size	SE	Offer Rate	Sample Size	SE	p-Value
3–9 Workers	58.7	322	2.73	50.5	368	2.58	.0299
10–24 Workers	80.7	419	1.91	75.6	234	2.79	.1365
25–49 Workers	85.3	238	2.29	81.8	110	3.67	.4220

Source: 2003 Kaiser/HRET EHBS and follow-up survey.

EHBS, Employer Health Benefits Survey; HRET, Health Research and Educational Trust.

These findings suggest that firms with 3 to 49 workers not responding to the EHBS are less likely to offer health benefits, but that a nonresponse bias exists in firms with three to nine workers. We combined the responses to the two surveys together to calculate an unweighted estimate of the offer rate. Table 3 shows the original estimates along with the unweighted offer rate for the combined responses. The table also shows the bias within each size category with its corresponding 95 percent confidence interval. Is this bias large enough to make a difference in a practical sense? Cochran (1977) suggests as a working rule of thumb that the effect of bias on an estimate is negligible if the bias is less than one-tenth of the standard deviation of the estimate. For each size category the bias is less than one-tenth of the standard deviation, suggesting that the bias is negligible. One-sided t-tests were conducted to determine the probability of obtaining a value more extreme than that which would satisfy the above rule of thumb. The results suggest that there is a reasonable chance, particularly for firms with three to nine workers, that the bias is not negligible.

Table 3.

Estimate of Bias in the EHBS and Weighting Adjustment Factors by Firm Size

Firm Size	Original EHBS Offer Rate (SE)	Combined EHBS Offer Rate (SE)	Required Bias*	Offer Rate Bias (SE)	Probability of Bias Greater Than the Required Bias (%)
3–9 Workers	58.7 (2.729)	54.3 (1.863)	4.896	4.348 (3.304)	43.4
10–24 Workers	80.7 (1.908)	78.9 (1.568)	3.905	1.802 (2.470)	19.7
25–49 Workers	85.3 (2.293)	84.2 (1.940)	3.538	1.099 (3.004)	20.8

Source: 2003 Kaiser/HRET EHBS and follow-up survey.

^*The target bias required to determine that the bias is not negligible—equal to one-tenth the standard deviation of the original EHBS offer rate.

EHBS, Employer Health Benefits Survey; HRET, Health Research and Educational Trust.

Working on the assumption that our estimates of the unweighted offer rate are biased upwards for small firms, we calculated and applied an adjustment to the sampling weights. The adjustment formulas are as follows and use information from Table 2:

$$AD{J}_y=\frac{\left({\pi }_1{n}_1+{\pi }_2{n}_2\right)/\left(n_1+n_2\right)}{{\pi }_1}$$$$AD{J}_N=\frac{\left(\left(1-{\pi }_1\right)n_1+\left(1-{\pi }_2\right)n_2\right)/\left(n_1+n_2\right)}{\left(1-{\pi }_1\right)}$$

ADJ_Y calculates the adjustment factor to the weight corresponding to a “yes” response for offering employer-sponsored health benefits in the EHBS while ADJ_N corresponds to a “no” response in the EHBS; π₁ is the proportion responding “yes” in the EHBS and π₂ is the proportion responding “yes” in the follow-up survey; and n₁ is the sample count in the EHBS and n₂ is the sample count in the follow-up survey. Size-specific adjustment factors are shown in the first two columns of Table 4. Adjustments were made prior to poststratification of the weights.

Table 4.

Weighted Offer Rates of the EHBS before and after Weighting Adjustment for Bias by Firm Size

	Weighting Adjustment Factor		Before Weighting Adjustment		After Weighting Adjustment
Firm Size	Offer Health Benefits: Yes	Offer Health Benefits: No	Offer Rate	SE	Offer Rate	SE	Sample Size	p-Value
3–9 Workers	0.9255	1.1059	59.6	3.09	55.4	3.16	322	.1712
10–24 Workers	0.9774	1.0947	78.3	2.55	76.4	2.55	419	.2926
25–49 Workers	0.9870	1.0753	85.0	2.83	83.9	2.93	238	.3789

Source: 2003 Kaiser/HRET EHBS and follow-up survey.

EHBS, Employer Health Benefits Survey; HRET, Health Research and Educational Trust.

Table 4 displays the weighted offer rates for each size category before and after bias adjustment. As expected, the offer rate estimates are smaller following the bias adjustment, but one-sided t-tests show that the adjusted offer rates are not significantly smaller than the unadjusted rates.

Conclusions

Given the nature of employer surveys, and the difficulty in obtaining respondent participation from small firms, researchers need to be aware of the potential bias that can result from nonresponse when estimating employer benefit offer rates for small businesses. Our analysis of the follow-up survey to the EHBS suggests that survey nonresponse produces an upward bias on estimates for the offer rates of small firms. This upward bias, although not significant, is most likely because of higher nonresponse rates among small firms that do not offer health benefits. Results from our follow-up survey show that firms not responding to the original survey were less likely to offer health benefits than those firms who did respond to the original survey, particularly for firms with three to nine workers.

The observed bias within each size category analyzed is negligible. However, there is a two in five chance that we would have observed a nontrivial nonresponse bias in the offer rate for firms with three to nine workers and a one in five chance for firms with 10–24 workers and 25–49 workers. Given this chance for a nontrivial bias, an adjustment to the sampling weights is made. The adjustment results in a reduction in the estimated offer rate for firms with three to nine workers of approximately 4 percentage points, from 59.6 to 55.4 percent. However, the calculated decreases were not significant.

These findings send a “proceed with caution” message to analysts studying offer rates. Because the offer rate bias appears inversely correlated to the response rate, analysts need to address the problem of low response rate. Research organizations can reduce nonresponse bias at the front end of a survey through a number of actions. The most obvious way is to make repeated callback and refusal conversion attempts. There is a difference between nonresponse bias because of noncontact and nonresponse bias because of refusal (Stec, Lavrakas, and Stasny 1999; Black and Safir 2000). Researchers need to focus efforts on both simultaneously or they may still suffer large errors because of nonresponse. While this is time consuming, the result will be a higher participation rate leading to a reduction in the bias because of nonresponse.

Our research is limited in that we only control for differences in the size of the firm when determining nonresponse bias in offer rate estimates. Future analyses should attempt to control for additional firm characteristics such as industry type and the region of the U.S. in which the firm is located. The follow-up survey itself is also a potential limitation. The lower offer rates reported by the follow-up survey respondents may not only be because of systematic differences between respondents and nonrespondents but may also be because of the way the follow-up survey was administered. Researchers should conduct a comparative analysis using methods that only utilize respondents, such as an analysis of late respondents with early respondents. This approach, if successful, would provide a means for analyzing past year surveys for nonresponse bias and to provide an adjustment factor for historical offer rates.