The article by Sherry Glied and her colleagues must have been a revelation to everyone in the policy arena who believed that estimates of the cost and effect of health insurance expansions had more to do with science than with art. The level of disagreement among modelers on the most fundamental assumptions involved in such estimates is stunning.
The recommendations made in that article strike me as commonsense methods of improving the consistency—if not necessarily the accuracy—of modeling the cost and effects of proposed policies. To start with, making modelers’ key assumptions transparent should have beneficial effects. By illuminating major differences in assumptions that result in different estimates, transparency could lead to a more open debate and greater consensus among modelers, as well as a better understanding by policymakers of the estimates’ limitations.
For those people drafting legislative proposals to expand health insurance coverage, the Congressional Budget Office (CBO)'s estimate is not just an estimate; it is the estimate, because members of Congress accept the CBO's estimates as the gold standard. Even more important, the CBO's estimates determine what legislation can be passed without triggering a budget point of order in the Senate that would effectively kill a bill unless a supermajority of 60 votes could be assembled to sustain it (House procedures are less restrictive).
On estimates for large, important bills, CBO staff are often willing to sit down with the congressional sponsors and discuss some of their principal assumptions. But the sponsors typically lack the technical expertise to make a compelling argument in favor of an approach different from the one the CBO has chosen, and they are not in a position to assess the CBO's model as a whole (and this is certainly not given to them).
The CBO is understandably reluctant to submit the complete model, since it fears that it will be picked apart by both sides of the issue, selectively challenging individual aspects of the model, many of which require subjective judgment. Moreover, many parts of a CBO model pertain to specific aspects of the legislation in question rather than to more general scientific questions such as the elasticity of demand for health insurance among subgroups of the population. For example, one of the main elements in the debate over Medicare prescription drug coverage was the CBO's estimate of the relative efficiencies of providing coverage through insurance companies competing for the business of individuals, compared with pharmacy benefit managers competing for the government's business.
Policymakers would appreciate knowing more about the CBO's assumptions that Glied and her colleagues identified as central. First, if the CBO revealed its assumptions, outside modelers and economists might be able to debate and test them, with the possibility that better models might ultimately emerge. Second, it would be useful for the CBO to find out how far inside or outside the mainstream its estimates are. The CBO no doubt checks this informally, but more formal opportunities for outside experts to examine the CBO's assumptions would be helpful. Finally, publication of the CBO's assumptions might enable outside modelers to better replicate its estimates and determine their probable impact on different policy options.
This last point is especially important. The CBO has limited resources to create estimates for Congress and a great deal of demand on those resources. The sponsors of expensive legislative proposals often want to examine a number of different options in order to modify those proposals so that they can make better decisions on trade-offs between cost and other policy objectives. But the CBO has a great deal of difficulty estimating the costs of a large number of variants in a timely way. Outside groups or experts can sometimes provide a quicker turnaround, but their estimates are largely worthless for congressional purposes if they are substantially different from the CBO's final estimate.
As Glied and her colleagues make clear, models are the black box of the health insurance policy debate, and the CBO's models are the most powerful black box of all. They exert a tremendous influence on what Congress does or not do. Greater transparency of the kind the article proposes for all models, including the CBO's, could improve the policy process and lead to better predictions of the effect of policy changes.