Commentary—Sophisticated Methods but Implausible Results: How Much Does Health Insurance Improve Health?

Jack Hadley and Timothy Waidmann use data from the Health and Retirement Survey (HRS) to analyze the effect of being uninsured on mortality and morbidity of the near-elderly. Consistent with the findings from other work, they conclude that being uninsured is bad for one's health.

However, the Hadley and Waidmann work is far from “more of the same.” They make three methodological contributions. First, rather than simply assuming that insurance status is unaffected by health status, they explicitly recognize that health status may affect insurance status, and use an instrumental variables estimation technique to account for the endogeneity of insurance and health. Second, while most other researchers have limited the outcome under study to mortality, or, in some work, discrete outcomes such as the “likelihood of a major decline in health,” Hadley and Waidmann estimate the effects of insurance on both mortality and a measure of morbidity constructed from self-reported information on health status and on functional limitations. Third, they combine their estimates of the effects of insurance on health status with estimates of the effects of health status on medical expenditures to estimate the effects of insurance on future medical expenditures. Although some have speculated that investments in insurance will lead to improvements in health, and that improved health status will result in lower expenditures in the future, I am not aware of any other attempts to estimate the magnitude of any such effect.

Having started this commentary with the obligatory, and sincere, praise, I must confess to not believing the results. The estimated effect of insurance on mortality is so large as to be implausible, even for those, like myself, who are persuaded that being uninsured is bad for one's health. Hadley and Waidmann analyzed the experience of 3,564 people from the Health and Retirement Survey who were between the ages of 55 and 60 in 1992. They measured baseline health status for these people, determined whether they survived to age 65, and, for survivors, measured their health status at their last HRS interview before they became Medicare eligible.

Two aspects of the Hadley and Waidmann results are most implausible. First, the parameter estimates for the effect of insurance on health status reported in Appendix Table 2 are so large as to make it impossible to believe that they are correct. Health status is measured on a 0–1 scale, where death is given a value of 0, and a respondent who reports excellent health with no limitations in activities is given a value of 1.0. In the instrumental variable model, the estimated effect of insurance on health status at age 65 is similar in size to the estimated effect of baseline health on health status at age 65. That is, a respondent with baseline health status of 1.0 but who was continuously uninsured would be predicted to have the same health status at age 65 as a respondent whose baseline health was very close to 0.0, but who was continuously insured. Similarly, the instrumental variable estimate of the effect of insurance is slightly greater than the sum of the estimated effects of having cancer, diabetes, heart disease, pulmonary disease, hypertension, stroke, and arthritis. That is, being uninsured is estimated to be as bad for one's health as having all of the diseases listed above. While health insurance surely leads to an increase in health care utilization, and almost surely leads to reductions in morbidity and mortality, it is hard to imagine that lack of insurance is as bad for one's health as the combined effect of having cancer, diabetes, heart disease, and a number of other serious illnesses, or that it is as bad for one's future health as the difference between being in very good health at baseline and being almost dead at baseline.

Second (and closely related to the first observation made above), the predicted effect of full insurance coverage on mortality is so large as to be implausible. Of the 3,564 persons in the analytic sample, 6.9 percent died before reaching the age of 65 (Table 1). The mortality of the continuously insured and the partially uninsured was strikingly different: among the 70.6 percent of the sample that was continuously insured, 3.9 percent died before reaching 65; among the 29.4 percent that was uninsured for some or all of the study period, approximately 14 percent died before reaching 65.1 The instrumental variables analysis (Table 5) produces the result that if all of the respondents had been continuously insured throughout the study period, mortality would have declined to 3.9 percent. That is, the IV analysis produce the estimate that if those respondents who were uninsured for some or all of the study had been continuously insured, their mortality rate would have declined from approximately 14 percent to approximately 4 percent.

The magnitude of the estimated effect of insurance on mortality is too large to be credible—Hadley–Waidmann estimate that lack of insurance (for some or all of the observation period) results in an increase in the mortality rate of approximately 350 percent. In contrast, previous analyses of the effects of lack of insurance on mortality for adults age 18–64 suggests that being uninsured at baseline increases the chances of mortality at follow-up by approximately 20–25 percent (Franks, Clancy, and Gold 1993; Sorlie et al. 1994; Kronick 2003). The follow-up period in these analyses varied from 5 to 16 years. It is not clear why the Hadley–Waidmann estimate is an order of magnitude larger than previous estimates; regardless of the reason for the difference, it is hard to believe that more than seven out of every 10 deaths among the partially uninsured near-elderly would have been averted if only they had been covered by health insurance.

If the Hadley–Waidmann estimate is not correct, why might it be wrong? I'm not sure of the answer, but there are two main possibilities. First, the analysis does not consider the effects of income on either insurance status or health status. Low income is the single most significant risk factor for being uninsured, and poverty is clearly very bad for health. I can imagine that income was not included in the analysis because of concerns about endogeneity with employer-sponsored health insurance, but omitting this crucial variable from the analysis certainly biases the coefficients in the observational analysis—the effects of low income on health outcomes will be partially captured by the insurance variable.

Second, the instrumental variables analysis relies on four variables to “identify” the insurance equation—these are factors that are hypothesized to affect insurance status, but to have no direct effect on health status. The variables used are: involuntary job loss in the 5 years prior to the baseline survey; whether the respondent is foreign born; if foreign born, the number of years living in the U.S., and whether the respondent's spouse is a member of a union. I do not think these are good candidates for instruments, because each of these variables is plausibly directly related to health outcomes, independently of their effect on insurance status. Most clearly, involuntary job loss in the past 5 years seems quite likely to be related to changes in health—both as a marker for people in decline (people whose health is deteriorating are more likely to be fired than people who are vigorous), and as a causal factor—people who are fired are likely to become depressed, experience stress, loss of income, and subsequent declines in health. A little less clearly, the foreign born (particularly recent immigrants) may well be sicker (in unmeasured ways) than the native born. In addition to the possibility of unmeasured baseline differences in health between the native and foreign born, the stress of being in a strange culture, and difficulty in figuring out how to navigate the health care and other systems may put the foreign born at greater risk of declines in health, independent of insurance status. Spousal union membership appears to be a somewhat better candidate for an instrument, but as income is excluded from the analysis and respondents whose spouse is a union member may have higher income than other respondents, it may well be that respondents whose spouse is a member of a union would be expected to have better health outcomes than others, independent of insurance status.2

The instrumental variables analysis essentially finds that people who have lost their job (or are foreign born or do not have a spouse who is a union member) are expected to be less likely to be insured, and that these people are much more likely than others to die (or experience other poor health outcomes). The analysis then infers that the reason that people experience poor health outcomes is lack of insurance, but it may well be other factors that are associated with losing a job (or being foreign born or not having a spouse who is a union member) and with poor health. Given the seemingly implausible size of the instrumental variables estimate of the effects of lack of insurance on mortality and the plausible connections between the variables used as instruments and health outcomes, I don't put much credibility in the results of the instrumental variables analysis.

The straightforward observational analysis also overestimates the effects of insurance on health by omitting from the right-hand side variables that are likely to affect health and that are also correlated with lack of insurance. Most notably, as discussed above, the estimated effect of insurance on health is almost certainly biased upwards by a substantial amount because income is omitted from the analysis, and likely biased upwards as well because variables such as involuntary job loss and foreign-born status are omitted.

The observational analysis could produce a somewhat better estimate of the effects of insurance on health by including additional variables that are included in the HRS and are plausibly related to health outcomes (and to insurance status); however, even with improvements, the observational analysis will leave substantial uncertainties about the magnitude of the relationship between insurance and health.

The fundamental problem with the observational analysis is illustrated by considering the stark differences in outcomes between those who lose insurance and those who maintain it during the study period. Approximately 85.5 percent of the sample was insured at baseline in 1992, and 70.6 percent of the sample maintained insurance throughout the study period, while 14.9 percent lost coverage at some point before they reached 65. Mortality among the continuously insured group was 3.9 percent; among those who lost coverage mortality was an astonishingly high 17 percent. The question that needs to be answered is how much of the difference in outcomes between the two groups is because of other factors, and how much, if any, is because of lack of insurance? The observational analysis can control for risk factors that were measured at baseline such as baseline health status, income, race, and education. But fundamentally, something is quite different about the group that lost insurance compared with the group that maintained it. We do not know why some people lost coverage while most maintained it, but losing coverage (especially for the near-elderly) should certainly not be assumed to be a random event. Many people who lose coverage during the study period likely do so because they have lost a job, and that event could well be related to a major decline in health independent of the loss of insurance coverage.

Hadley and Waidmann are well aware of the pitfalls of the observational analysis—they discuss these problems in their review of Baker's work at the beginning of their paper, and plunge into the instrumental variable analysis in an attempt to avoid the inferential conundrums of the observational analysis. However, as discussed above, their instruments are arguably directly related to health outcomes, and thus the IV analysis does not produce credible estimates of the effects of insurance on health.

The connection between better health and lower health expenditures: Hadley and Waidmann creatively meld their analysis of HRS data on the effects of insurance on health among the near-elderly with analysis of Medicare Current Beneficiary Survey data on the relationship between health status and expenditures among new Medicare beneficiaries to estimate the effects of better health on expenditures. As discussed above, the estimated effects of universal coverage on health outcomes are likely to be a very substantial overestimate of the true effects. As a result, the estimates of the effects of universal coverage among the near-elderly on Medicare expenditures are also likely not correct—Hadley and Waidmann likely overestimate the extent to which universal coverage would increase the number of Medicare beneficiaries, and also overestimate the extent to which it would lead to improved health status among those people who reach age 65. These two biases work in counterbalancing directions, and the net effect on the results is not clear.

The Hadley–Waidmann analysis assumes that aggregate health expenditures are determined by the health status of the population—that if the population gets healthier, then utilization and expenditures will decline. If only life were so simple. As decades of work from Jack Wennberg and colleagues have taught us, utilization may be influenced in part by the health care needs of the population, but also by the supply of health care resources, and by the supply-sensitive treatment decisions made by physicians. If the population gets healthier but the supply of physicians and hospital beds does not change, it is likely that, over time, practice patterns will adjust and aggregate levels of utilization may not change. A healthier population may lead to less pressure to increase the supply of resources, and ultimately to a lower rate of growth of expenditures, but these relationships are tenuous. Thus, in the medium run, a healthier stock of 65-year olds may not lead to a decline in the aggregate number of physician visits and hospitalizations.3

More broadly, as an analyst far from the Beltway who is not required to deal with official scorers at the Congressional Budget Office, I am concerned about trying to “sell” universal coverage by focusing on its effects on health expenditures. In state-level debates, we often hear the argument that insuring the uninsured will lead to improved health and reductions in emergency room visits and downstream savings in health expenditures as a result of improved health. This is not an argument that can withstand close scrutiny. The argument for reduced ER visits has been shown to be largely a canard—the insured are as likely to visit the ER as the uninsured (Cunningham and May 2003).

The downstream argument is interesting, and Hadley and Waidmann provide the beginnings of a model to consider it. I discuss above my skepticism about the relationship between the health of the population and the aggregate level of health expenditures. But if we ignore these objections, and model expenditures directly as a function of health (and insurance status), then, if we had good estimates of the effects of insurance on health, we might be able to produce some estimate of the effect of universal coverage on aggregate health expenditures. Hadley and Holahan (2003) estimate that insuring the uninsured will lead to, very approximately, a doubling of health expenditures for the currently uninsured. Given the consensus from most work (the Hadley and Waidmann article notwithstanding) that insurance has a relatively small effect on health, it seems likely that insuring the uninsured would not “pay for itself” through the mechanism of reduced health expenditures, but that is a question at least partially amenable to empirical analysis.

However, I would not recommend that budding health service researchers spend much energy attempting to determine the effects of universal coverage on health expenditures (at least through the intervening mechanisms that universal coverage improves health, and that improved health leads to lower expenditures). Any method of achieving universal coverage would inevitably require major changes in our systems of collecting and spending money in health care, and very likely lead to greater public sector involvement in the health care financing. It is these changes, much more than the changes in the health of the population, that are likely to affect the future path of health care expenditures under universal coverage.

The main arguments for universal coverage are not that it would pay for itself in the form of lower health expenditures, but rather that it would lead to improved health (which is highly valued by most of us), less suffering, fewer financial catastrophes, less pressure on providers who care for disproportionate numbers of the uninsured, a greater sense of equity and community, and, perhaps, greater labor force participation and productivity. These arguments are well worth making and repeating.

During health care financing reform debates, politicians often ask for an estimate of the size of the benefits that should be expected from coverage expansions. Econometric models produce quantitative estimates (albeit often with large confidence intervals) of the expected effects of expanded coverage on health care utilization and expenditures, and on the distribution of financial burden. Yet, we cannot provide a reliable quantitative estimate of the benefits. A much-cited Institute of Medicine (IOM) report estimates that lack of insurance results in 18,000 additional deaths per year, and a loss of $65 to $130 billion annually in health capital (Institute of Medicine 2003). These estimates are valuable markers, and much preferred to not having any estimates at all, but careful examination of the evidence on which they are based leaves us with little confidence in the numbers. The lower bound estimate of $65 billion is derived from the lost value associated with the effect of insurance on mortality, and is based on the estimate that lack of insurance leads to increase of 18,000 deaths per year. The 18,000 estimate for excess deaths results from the estimate that lack of insurance leads to an increase of 25 percent in the mortality; the 25 percent estimate, in turn, is based primarily on the results from two observational studies (Franks, Clancy, and Gold 1993; Sorlie et al. 1994). This is a reasonable estimate, although subject to substantial uncertainty (Kronick 2003). And while the IOM was willing to put a point estimate on the value of lost health capital associated with excess mortality, there was so much uncertainty about the effects of lack of insurance on morbidity that they simply provided a range of somewhere between $0 and $65 billion. This range is uncomfortably wide when talking to politicians or the public.

Better estimates of the magnitude of the effect of lack of insurance on health are much desired. Hadley and Waidmann make a valiant attempt, although one that is ultimately unsatisfying. Even understanding that it is self-serving (and commonplace) for a health services researcher to end with a call for further research, I make that call here. Carefully conducted studies of natural experiments in which large number of people either gain or lose coverage as a result of exogenous factors (i.e., changes in public coverage eligibility causing groups of people to gain or lose coverage; or collective bargaining victories that “bestow” coverage on a large group of previously uninsured workers) would be promising. Alternatively, a randomized trial in which a group of uninsured persons is randomized to receive insurance and followed over a multiyear period would provide a better answer to the question, although it would be expensive to conduct.

I am certainly not suggesting that policy makers should wait for the results of further research before taking action. Being uninsured is bad for health, results in financial catastrophes, creates strains on providers serving the poor (threatening the availability of care for the insured and uninsured alike), and is an assault on our sense of decency and community. And there is no credible countervailing argument about the advantages of living in a society with 45 million uninsured people—although some might argue that the greater level of government involvement in health care financing needed to assure broader coverage will have deleterious effects on the economy more broadly or the health care system in particular, there is little evidence to support this assertion. It would be wonderful to think that an energetic policy entrepreneur will someday soon figure out how to slash through the political thicket that has thwarted previous reform attempts.

But given the very real prospect of political stasis, more firmly grounded quantitative estimates of the benefits of coverage expansions would be beneficial the next time coverage expansion is seriously debated at the state or federal level. The modelers will surely produce estimates of the cost of expanding coverage and the distribution of those costs among business, government, and individuals. It would be highly desirable if they could also produce credible estimates of the benefits.