Abstract
The purpose of this roundtable is to explore the imperfect art of estimating the budget costs of health insurance proposals—called scoring when done by government agencies. The panel addresses the complexities involved in generating these estimates, which usually depend on many untested and untestable assumptions. For example, the Medicare prescription drug “donut hole” was invented so that policymakers could achieve budget targets. These budget scores play a critical role in the design of health policies, as well as in the reform proposals put forth by candidates in an election. The roundtable discusses how policymakers can and do use health policy estimates and budget scores.
Chair: Sherry Glied, Ph.D., is a professor and chair in the Department of Health Policy and Management at Columbia University's Mailman School of Public Health.
Panelists: Linda Bilheimer, Ph.D., is a senior program officer at The Robert Wood Johnson Foundation. Previously, she held the position of deputy assistant director for health at the Congressional Budget Office.
Judith Feder, Ph.D., is a professor and dean of the Georgetown Public Policy Institute at Georgetown University. She was formerly the principal deputy assistant secretary for planning and evaluation at the U.S. Department of Health and Human Services and staff director of the congressional Pepper Commission.
Len Nichols, Ph.D., is vice president of the Center for Studying Health System Change. His experience includes the role of senior advisor for health policy at the Office of Management and Budget.
Kenneth Thorpe, Ph.D., is professor and chair of the Department of Health Policy and Management at the Rollins School of Public Health at Emory University. He was previously deputy assistant secretary at the Department of Health and Human Services.
Timothy Westmoreland, J.D., is professor of both law and public policy at Georgetown University. He has worked for Congressman Waxman's staff in the past and also has experience directing the Medicaid program under a previous administration.
Sherry Glied: Outside of Washington, people often think that estimating the budgetary costs of coverage proposals is something that happens after the details of the legislation have been hammered out. I do not think that is a good characterization of the process. Can you describe a situation when the design of legislation was driven by the numbers, and discuss the implications of that?
Linda Bilheimer: Throughout my time at the Congressional Budget Office (CBO), I cannot think of a single situation when the design of legislation was not driven by the numbers. Every major piece of legislation was extensively negotiated behind the scenes with the cost estimators, who made rounds and rounds of estimates that the public never saw. The key to understanding what drove this is the PAYGO (Pay-As-You-Go) procedures that were established by the Budget Enforcement Act of 1990, which required that any budgetary increases in mandatory programs such as Medicare or Medicaid, or any changes in taxes, be offset by corresponding reductions elsewhere in legislation. Members of Congress clearly wanted to know the implications of PAYGO on their bills well before legislation got to the floor, and cost estimates were done around the clock.
Judith Feder: Even in the absence of PAYGO legislation, legislative proposals are sometimes generated with a predetermined sum in mind. Most recently, $400 billion was the assumed cost limit of the Medicare prescription drug legislation, so the dollar figures drove the entire design. With the Clinton health reform plan, the governing score we were aiming for was zero. We did our best to hit it, and that drove a great many of the decisions we made.
Len Nichols: While we certainly tried to anticipate how CBO would score the Clinton bill, and we had many, many conversations with CBO staff about it, we were stymied because we were not able to give legislative language ahead of time and get CBOs judgment about particular drafts of the bill. Thus, when CBO rendered its verdict about the Clinton plan, we did not have a chance to change it the way we would have if the process had been done informally, as was the norm.
Timothy Westmoreland: Actually, before 1974, none of the process that is now the norm existed. Prior to that, budget decisions were determined from committee to committee, through their negotiations with Cabinet agencies. The Office of Management and Budget (OMB) had a consolidated budget for the executive branch, but Congress didn't have one for the legislative branch.
Then, in 1974, with the Budget and Impoundment Control Act, Congress created a budget process that was an accounting device and was relatively neutral about policy. Between 1974 and 1990, there was no such thing as PAYGO; that was added in 1990. Even without PAYGO, however, there was a lot of estimating going on.
After the Gramm-Rudman Act was passed in the mid-1980s, the budget process went from being simply an accounting device to being a deficit-control and a spending-control device. It is interesting to reflect back and realize that George H.W. Bush, had he been President under the current PAYGO rules, would not have had to break his “no new taxes” pledge. At that time, the budget limits were focused on maximum overall deficits, even if they arose from longstanding law. As a result, former President Bush had to respond to the deficit or face across-the-board cuts. Those rules have since been changed, and now we focus on the budget effects of new legislation using PAYGO.
We have gone from a time when Congress genuinely tried to figure out the parameters first and then determine the cost to a time when we are trying to retro-fit what we can accomplish with a predetermined sum.
Sherry Glied: Is the growing role of estimators a bad thing? Does it have good points? What is the effect of all of this?
Timothy Westmoreland: Estimation has taken on an almost oracular significance in health policy that is unprecedented in democracy. I would say that CBO is more important to health policy than Alan Greenspan is to financial policy. Ironically, it has gone far beyond the point that most estimators would endorse. Estimators will not defend their numbers the way anyone else on the Hill will defend them. In fact, the estimation process often allows politicians to cloak political decisions in the appearance of scientific certainty by saying “that is what CBO has rendered.”
Len Nichols: The good part of the process is that CBO at least serves the function of always keeping the policy conversation centered within a “ballpark” of their estimates. The difficulty is that Congress tends to want a number rather than a range. That is a fundamental tension.
Kenneth Thorpe: Being able to make good estimates is critically important. We have to continue to come up with ways to improve our ability to generate these numbers—whether they are used for domestic policy or tax policy—because they shape economic behavior. They play a dominant role not only in health policy but in all of our budget decisions: They can affect long-term interest rates, people's investment behaviors, and the Federal Reserve.
Linda Bilheimer: One of the most important things that cost estimation does is to require more people to read legislation. The fact is that very few people, including members of Congress, read the legislation that has a profound impact on this country. To do a good cost estimate, you need to learn how to read, understand, and ask informed questions about legislative language.
Back during the time of the Clinton health plan, there was a lot of discussion about what the term “universal coverage” actually meant. No one really thought it meant 100 percent, so there was a lot of playing around with the numbers. For some reason, I believe we wound up with a standard of 95 percent, which was arbitrary.
As I recall, we estimated that one proposal from the Senate had a coverage rate of 94.3 percent. The fact that we actually estimated it to the nearest three-tenths of a percentage point blows my mind. But we did, and then we had to have a meeting with Senate staff to discuss how to get the coverage rate up to 95 percent. Ultimately, someone found a small, specific group of the population that we would estimate could gain coverage, or increase their coverage rate, if some additional language was added to the bill. We projected that making that change would raise the overall coverage estimate to 94.6 percent. We rounded the estimate up to 95 percent and everyone declared victory. That process is an appalling way to design legislation.
Judith Feder: But that is the process we have, and that is what people need to understand. The more budget pressure there is, and the more politically charged the environment is, the greater the chance that you are going to be niggling over decimal points.
There is a tremendous desire in politics for certainty—but these estimates are inevitably uncertain. That is what makes this very difficult and what leads to the glorification of estimates, the false sense of precision, and all the other flaws in this process that have been articulated. Politicians do not want to deal with ambiguity, and I do not think we can make them. It is not our job to fix everything and that is not something that we can fix.
But perhaps we can work to ensure that the amount of influence that estimators have does not become inappropriate. The current process often gives extraordinary power to the people at CBO, who are, after all, only human. They are sometimes outstanding people, but they are just people making their own calls.
The other thing to remember is that estimating is not a pure science; there are a lot of politics in the estimating community. We try to deal with that by developing standards for estimators, promoting communication among those who are doing estimates, being transparent about assumptions, and being clear about how certain estimates are arrived at. Still, we should recognize that certain estimators have implicit, if not explicit, agendas and differing beliefs about what is right or accurate or appropriate and what is not.
Timothy Westmoreland: Estimating is undoubtedly important, but it has taken on a political significance that it cannot withstand. Sensible estimates come with warnings about ranges of uncertainty, assumptions made, variables considered, and standard errors. But those caveats get lost in the legislative process. I do not blame the messenger. It is not the researcher or the estimator who is at fault here; it is the process that we use. Things that good estimators would not defend have become reified into this process, and I find that troubling politically and legally.
Sherry Glied: When we are talking about health reform proposals, we are often dealing with abstractions, at least at the outset. We do not know how—or if—they will work in the future. We did not know in 1992, for example, whether managed competition would work. My question is: With the current process of generating health reform estimates, have we taken some of the policy analysis work out of the policy process?
Kenneth Thorpe: I think the answer to your question depends on which type of estimate you are talking about. If you consider estimates where we have a lot of experience looking at behavioral changes linked to policy tools—such as those done on the Medicare program for policy instruments that we use to increase or decrease spending—we have a pretty good handle on how to do them because we have track records to help us evaluate them. For most things that drive the budget, I think we do a pretty good job of estimating.
On the other hand, it is more difficult to make estimates for big policy proposals that are brand new. With the Medicare drug bill, obviously we have not seen this in the Medicare program before, so there are a lot of assumptions that have to be made. We do not know what the effects of the PPOs (Preferred Provider Organizations) will be—how people will react to them, how many will come in, etc. The same is true with health reform: For the most part, you are providing subsidized coverage to new populations that have not been covered before. We may have some relevant state-level experience to draw from, but, in general, we do not have a lot of information.
For the new programs, it would be helpful if at some point CBO, CMS (the Centers for Medicare and Medicaid Services), or both sat down with staff and laid out a range of estimates in advance; they could then communicate where the uncertainty points are, what is known in terms of the estimates, and how much play there is in them. If federal officials absorbed that into their decision-making process, it would make the process inherently more conservative because I think that they would react more negatively in the areas where the variance is higher.
Len Nichols: I would like to take a step back and remind everyone how we got a CBO. In 1972, we were debating national health reform after Nixon proposed universal health coverage with an employer mandate. At the time, Caspar Weinberger was the Secretary of Housing, Education, and Welfare; the estimates he gave through his testimony before the Ways and Means and Senate Finance committees were considered the gold standard by both committees and all members of Congress. There was no debate about them; it was very pro forma.
It is an astounding contrast to the process we have today. What happened? Congress eventually lost faith in Nixon, and that is how we got the CBO; Congress did not trust the executive branch to play fair. CBO has survived amazingly over all these years, and now, at the end of the day, it has the final word. How much better might it be if we had two public entities charged to decide on these estimates? At least then, members would have to confront the issue when the two entities produced differing estimates. As it stands, if CMS and CBO reach conflicting conclusions, there is no mechanism to guarantee resolution. One number is pigeon-holed through, with CBO having the last word. Perhaps the time has come to redress the balance of power.
Judith Feder: I do not think there is a technical fix to this issue. As Tim was arguing, a legitimate interest in accountability for spending taxpayers' dollars often leads to a skewing of focus to the money. In addition, the question of how much of the money we want to collect from taxpayers versus what we want them to spend is hugely political. I do not know that there is much we can do process-wise to alter that.
In addition, I do not know what the track record is on budget baselines, but I do not think it is fabulous. And those projections are what determine what spending is and will be under current law. So the whole budget process rests on estimates. I do not want to call it fiction, but it rests on estimates, and thereby hides some of the decisions that have to be made about how much we want to tax and how much we want to spend.
Linda Bilheimer: One of the questions that estimators always get asked is: Were you right? Or people will point out that we made baseline projections in 1999 and now it is 2004 and the reality differs greatly from the projections. Baselines are essentially the projections of what you would expect the budget to be in the absence of changes in current law and other unforeseen circumstances. But, of course, everything changes. Therefore, there is no way to go back and validate a baseline. That is a very difficult concept to get people to understand: Almost by definition, you cannot be right.
Timothy Westmoreland: I would like to give an example that underscores that point. In 1992, during the Clinton/Gore transition, I was a principal informant for an OMB estimation process. One day, someone walked into my office and asked me what the price of AIDS drugs would be over the next 10 years. So I asked him to first give me a clue: Do we cure the epidemic or not? Nobody can foresee these kinds of things.
To play out the example a bit further: In 1996, the standard of care for AIDS was AZT, which cost about $1,500 per person per year, whereas by 1997 it was protease inhibitors, at $15,000 per person per year. I am sure many people wanted point estimates in 1996 for good reason, but those numbers would not have been worth much by the following year.
Congress has become so fixated on the question of how much things will cost—which is admittedly quite important—that it sometimes loses sight of what it is buying. If, as Oscar Wilde says, a cynic is defined as someone who knows the cost of everything and the value of nothing, then Congress has become totally cynical.
My technical fix is to incorporate some type of balancing estimate of what we are trying to buy. It would not be hard to incorporate morbidity and mortality data into the process in order to calculate quality- and disability-adjusted life years (QALYs and DALYs) or years of prospective life lost.
Kenneth Thorpe: We have been spending a lot of time focusing on costing out new programs, but we have not discussed the most important thing—which is how the budget baseline is developed. The process that CBO and OMB go through is completely different on the spending versus the revenue side.
The revenue side involves thinking through the key factors that determine the baseline, which are projections of economic growth. There are 7,000 modelers out there in the private sector that are doing projections of GDP growth. CBO and others then look at the projections; they also develop their own and assess how those are positioned against a range of projections developed through very sophisticated models.
On the spending side, however, the amount of time and resources that we devote to evaluating trends in health care spending and the factors that can affect the Medicare program are very different. Obviously we track these things as part of the Trustee's report, but we do not necessarily focus on the thing that matters most—developing the best budget baseline estimates that we can.
We may want to think about how we can do a better job in that regard. Once we get the spending baseline right, then we can worry about how to do deviations from it with these cost estimates. But I am concerned about the asymmetries that are built into the basics of doing a baseline.
Len Nichols: Someone who worked with me on health reform at OMB mentioned that he had attended meetings with representatives from CBO, the Treasury, and OMB, in which they met with revenue forecasters; they homed in on the baseline assumptions—all the way down to the second decimal point—and the group would not adjourn until they reached consensus. It is because of that process that these revenue baselines are reasonable. We have nothing like that on the spending side.
Sherry Glied: There is interest now in expanding the analysis of social programs to include more cost–benefit analyses. Such analyses are regularly conducted when it comes to environmental regulations—like building a dam, for example—but we do not do them a lot for social programs. People tend to view cost-benefit analyses with respect to environmental policy as inherently conservative; they think that these analyses may prevent important regulations from being implemented. Yet, in the social welfare realm, it is just the opposite: People perceive using cost-benefit analyses as inherently liberal, potentially leading to extensions in expensive public programs. Does that make sense to you? What do you think we should be doing on the benefits side for health initiatives?
Judith Feder: The comparison is really interesting. One could characterize the Institute of Medicine's (IOM) recent series of volumes on the uninsured—which measure the benefits of insuring people and the costs to society of uninsurance—as an attempt to make up for the absence of the benefit picture.
Kenneth Thorpe: The problem is we have no scientific basis for doing any of this. If we were going to have a five- or six-fold increase in CMSs or AHRQs research budget, then perhaps within 10 years we could get some information and we could do it. But right now, we are far from having any of the underlying information needed for such an endeavor. If we think that assessing benefits is important, then we should fund that and develop a scientific basis for doing it. Absent that, I think we are fooling ourselves by thinking about it.
Timothy Westmoreland: It is no more science fiction than some of the stuff that is done on the cost side. You build variables. Epidemiologists and statisticians have ways of thinking not just about morbidity and mortality, but of QALYs and DALYs and years of prospective life lost and other important measures.
So I think it is useful to have this discussion. I would agree with Sherry's observation that cost-benefit analyses are more the norm for environmental regulations. In my law school crowd, people are astonished that legislation does not include cost–benefit decisions, because administrative regulations—not just environmental ones—require such analyses at almost every level.
Having said that, I think that people need to recognize that benefits assessments carry some strange artifacts, just as budget estimations do. The first and most obvious one is that it is always cheaper for people to die. For example, I have seen proposals to add pneumococcal vaccination to the Medicare program in which, among other things, the estimators evaluated the costs that would result if people did not die of pneumonia on schedule and instead lived to collect other Medicare benefits. That is not a mistake, and it allows politicians to say—truthfully—that the reason they voted against such proposals is because they are too expensive. What they do not say is: “I am voting to have people die of a preventable disease on an actuarially predicted level.”
Linda Bilheimer: Exactly. The costs of extended life have to be built into the process.
Sherry Glied: On the subcommittee that tried to do benefit estimates for the IOM, there was tremendous uncertainty about the cost of uninsurance and, as a result, the committee came up with an enormous range of estimates. Part of the reason that I think we do better on cost estimates, and why we feel like we know where we are going there, is because we have been forced to do it for 20 or 30 years. We are not there with benefit estimates.
Judith Feder: As the conversation moves forward, it also becomes clear that it is very hard to quantify some very valuable benefits. If it all comes down to dollars—rather than quality of life—then we have gone back to the conservative interpretation of cost–benefit analyses.
Timothy Westmoreland: I would argue that we do not necessarily need to come back to dollars at every point. The administrative process has driven the dollar comparison, but it need not. Just for the sake of blue-sky, let us think about what PAYGO would look like if it were about national mortality. Imagine that you cannot bring legislation to the floor of the House unless you demonstrate that it does not increase mortality in the U.S.
At that point, the three big automakers in this country who want to allow more SUVs would need to find offsets—ways of lowering mortality in other areas—to compensate for the fact that SUVs make a higher contribution than most other vehicles to air pollution and accidents. Perhaps they would institute infant mortality programs because that is where they would get the most bang for the buck.
The point is that there is no place in the Constitution where it says Congress exists only to raise and spend money. It exists to do lots of things that are good for America, and we could use the same models to address other national goals, such as reducing mortality, HIV rates, disability, etc.
Judith Feder: It is interesting to note that, while the scenario that Tim just described sounds fanciful, a similar process actually exists in some areas. For example, as I understand a lot of environmental regulation, economists are not allowed to take costs into account when they have to hit a number in terms of particles per whatever. That is not done on any kind of cost-benefit basis—which frustrates economists in the environmental field.
Linda Bilheimer: Having to do these estimates requires rigor, and we need to think about how these processes would work in the context of the timing of the legislative process. When I worked at CBO, I used to joke about how many Ph.D. theses I had written in a day, because the analyses we would undertake would take 2 years for a student to write about in a dissertation. Due to the demands of the legislative schedule, we had 24 hours or less to produce an estimate.
Sherry Glied: We are now in the midst of an election, and there are going to be cost estimates that come out of each of the Presidential campaigns that will be used to buttress the candidates' own policies and attack those of their opponents. Is this a useful process for voters? If not, is there any way that it can be made more useful?
Kenneth Thorpe: My sense is that those estimates are used to make it easier for candidates to express their priorities in concrete terms. Looking back at the 2000 election, George W. Bush talked about cutting taxes by $1.3 trillion as a way of reinforcing his points about tax policy. Clearly, that is a huge number, but the estimate itself was not the issue; it might just as well have been 1.1 or 1.5 trillion. The point is that he needed a way to signal to his base and to his voters about how he was planning to allocate federal dollars. Using that figure allowed him to communicate that he was intending to devote a big chunk of resources to the tax side.
Estimates serve as a way for candidates to not only express their positions, but to illustrate how their proposed allocation of resources compares to that of their opponent on a given priority. In the 2004 campaign, I think the key difference in the debates will be over the federal budget. It is a question of whether voters want to continue with what President Bush has proposed—tax cuts—and extend them past 2010, which is a $2.5 trillion expenditure; or if they will move toward what Kerry is talking about, which is using $880 billion for health care and education.
Candidates actually get forced into using estimates now because the press is so accustomed to them. Indeed, the media is as budget-driven on the issues as anybody else. They are very sophisticated, and they are not going to allow candidates to say that they are planning to cut taxes or reduce the deficit by half or achieve universal coverage without providing some specifics about how they are going to do it.
Linda Bilheimer: Yet, by our standards, Presidential proposals tend to be fairly broad and generic. They may sound like they have got a lot of detail in them, but they are nothing like legislative language, and so the estimates are often generated by making sweeping assumptions about the way that things work. A two-page summary of a candidate's policy priorities does not provide the level of detail that a cost estimator needs when using the estimation models that we developed.
I think it could be very helpful to have the estimators lay out the underlying assumptions they have put into doing these estimates. We as economists really have an obligation, not just in Presidential campaigns, but in general, to examine the process that goes into creating estimates for proposals that could have a major political impact on the country.
Also there is much we can learn from variation. When analysts come up with different cost estimates, we can discover a lot about the factors that are driving those differences, and gain a deeper understanding of the fundamental policy issues—provided that the assumptions are transparent and that the estimates are presented in a standard way.
Judith Feder: I agree that it is useful to examine the process, except not in a Presidential debate, or at least not in the Presidential primary. There will always be estimates and counter estimates that will be used as ammunition in elections—which are, after all, political battles. I do not think that campaigns are the arena where we want to focus on getting it exactly right. In an election environment, I think that the fewer the estimators, the better.
Timothy Westmoreland: I want to agree with Ken and Linda's characterization of estimates in campaigns as signaling devices. But I would like to give two examples of places where I think they became reified into something more than that.
In 1996, when Clinton and Dole were campaigning against one another, Clinton alleged that Dole's legislation would have cut Medicare. Dole countered that his legislation would actually lead Medicare to be bigger than it ever was before. Interestingly, they were both right: If you compared the Dole legislation to the policy baseline, it lowered the rate of increase, but if you compared it against current actual spending, it expanded the program. But there is no easy way to explain to the public how both candidates were right; it was an issue that could not possibly have been dissected for voters.
Similarly, in 2002, Senator Kennedy proposed to freeze the stair-stepping down of the tax cuts in light of the poor economy. Bush called this a tax increase. Well, it is confusing—the Kennedy proposal was to freeze taxes at the same level as they actually were, but the policy baseline showed them going down. Is that a freeze or an increase? The public genuinely does not understand this. So when all these politicians start in with their “no new taxes” pledges, it is often not clear what they mean; are they referring to baselines or actuals?
That is what troubles me about cost estimates. They become a way for politicians to conceal what they are actually talking about. I agree that they can be signaling devices, but when debates get refined down to the level of baselines and expected versus actual growth rates, they become confusion devices.