Using Microsimulation Models to Inform U.S. Health Policy Making

Microsimulation models are an important tool for estimating the potential behavioral and economic effects of public policies on decision making units, including individuals and households, employers, suppliers of health insurance and health care services, and government. Among the broader set of models available to researchers and analysts to inform policy decisions, microsimulation models (MSMs) are distinguished from others by their capacity to analyze the policy impact at the level at which it is intended and to incorporate changing behavioral responses and institutional attributes over the time period being assessed (Chollet 1990; Citro and Hanushek 1991).

There are four basic components of any MSM: (1) the data infrastructure, (2) behavioral assumptions and parameters, (3) statistical methods, and (4) model output. As its foundation, every MSM must have a core data infrastructure. For many health-related MSMs, the core data file is built using at least one survey or administrative database that contains detailed attributes of individual units within the population. A second component of any MSM is the set of parameters or assumptions pertaining to specific behavioral responses of individuals that would be anticipated as a result of a new policy. Typically, modelers utilize a range of estimates about behavior drawn from the scholarly literature. Modelers then use statistical methods to estimate how changes in behavior due to the policy affect designated outcomes of interest to policy makers. Finally, MSMs produce output summarizing aggregate and in many cases, distributional effects of policy scenarios for a defined population.

Recent improvements in data collection, scholarly research evidence, and computing technology over the past two decades have led to the emergence of a new generation of health-related MSMs, including those focusing on health insurance, medical care spending, and population disease burden. Models have been developed and are being used by Federal agencies (e.g., Congressional Budget Office, U.S. Department of Treasury, Centers for Disease Control and Prevention) and private sector entities. Among the latter, MSMs have been developed by research and policy-based organizations as well as consulting firms, universities, and individual academicians.

Throughout the policy development process of the Patient Protection and Affordable Care Act (ACA) of 2010, estimates from several health-related MSMs received considerable attention by policy makers and interested stakeholders seeking objective estimates of the potential effectiveness and economic implications of the legislation. Notably, with the passage of the ACA, many of the major health policy simulation models continue to be used throughout implementation to assess how key provisions may affect specific populations. While potential users or contractors of microsimulation modeling are diverse, there are likely common issues faced by each when choosing a model and modeling strategy.

To provide a foundation for understanding the role of microsimulation in health policy making activities, this article addresses the following three questions:

1. Who uses health-related MSMs and for what purpose?

2. What are the constraints and challenges faced by potential users when making decisions about models and modeling strategy?

3. What are some areas for improvement and investment for the development and use of health-related MSMs in future policy making activities?

Who Uses Health-related MSMs and for What Purpose?

Within the Federal government, health-related MSMs are used extensively by the legislative and executive branches during the policy development phase as alternative designs are identified and assessed in terms of their effectiveness and efficiency for addressing the policy objective at hand. For example, in the early development of the ACA provisions, multiple Federal agencies sought “unofficial” estimates of the number and cost per newly insured person that could be achieved under different combinations of public and private insurance expansion strategies. This was done as a way to “fine-tune” policy recommendations as language was being drafted. Of course, once formal legislation was introduced in Congress in 2009, the nonpartisan Congressional Budget Office (CBO) and the Joint Committee on Taxation (JCT) provided official estimates of cost, revenues, and impact.

While MSMs are extremely useful for evaluating the relative strengths and weaknesses of particular policy designs in the development phase, they are also widely used during implementation. Complex legislation, such as the Affordable Care Act can take years to implement fully. To support this implementation process, MSMs may incorporate new information into modeling assumptions, such as updated expectations about overall economic growth. Another rationale for using MSMs during implementation is that the administrative rule-making process provides additional detail about how specific provisions will be enforced. In turn, this may affect how modelers parameterize particular policies when estimating behavioral responses.

Health-related MSMs also may be used for state-specific policy development such as implementation of federal provisions where states are given discretion. Within the ACA legislation, for example, states have been given the option of planning and developing their own insurance exchanges through which individuals and small employer groups can purchase coverage. Among states that have chosen to do so, many decision makers have considered contracting for MSM services to help them understand the trade-offs associated with different design features (e.g., pooling individual and small risk pools or leaving them separate). Other state governments have contracted for modeling services to estimate whether additional state investments are needed to address second-order effects of federal reform, such as whether a state has adequate provider capacity to address increased demand for medical care resulting from the coverage expansion.

Health-focused advocacy organizations as well as private organizations within the health sector also utilize MSMs. For advocacy organizations, their use is often for producing materials to influence public opinion and/or the legislative process. For health insurers and health care delivery organizations, output generated by MSMs can provide useful information for developing business strategies that will put them in strong positions given changing market conditions that are the result of new policy.

What are the Constraints and Challenges Faced by Potential Users when Making Decisions about Modeling Strategies?

Several constraints and challenges are faced by potential users of MSMs when making decisions about modeling strategies. Within the Federal government, it is not uncommon for there to be a short window of opportunity during which there is support among lawmakers to advance specific agenda items. Thus, legislative committees as well as agencies within the executive branch may be under considerable time pressure to obtain model estimates that can inform decisions about policy positions. Expediency is an important factor.

A second criteria considered by potential users is the reputation of the modeling organization, particularly when considering the use of private-sector MSMs. Because the information created by MSMs can profoundly affect a policy design and its subsequent impact, potential users want to make sure that estimates available for public consumption utilize valid data, current research evidence, and appropriate modeling techniques. Potential users also have expressed the importance of contracting for services from modelers with records of scholarship and/or experience, given the complexities of these models and the difficulty faced by nontechnical experts in understanding the specific approaches used.

Financial constraints represent a third challenge. In contrast with the Federal government, states, and other stakeholders within the private sector may be limited with respect to the resources available for MSM contracting. As potential users weigh contracting decisions, it is vital to consider carefully the scope of work, including the number and types of scenarios being put forth as well as the deliverables requested by the users.

Potential users also must consider the issue of “fit.” Many health-related MSMs are developed from national databases and are designed to generate national estimates. However, many questions about impact may be state specific or even local in nature. In the implementation of the ACA, for example, many states have contracted for MSM analyses to generate state-based estimates. Although these MSMs have been able to make adjustments to the data infrastructure to reflect the population attributes of a particular state, there is considerably more hesitation on the part of modelers to generate estimates that are substate or for small population subgroups within a state.

Finally, potential users of MSMs as well as members of the modeling community face challenges due to the lack of reconciliation across competing models, that is, why different models produce different results for the same outcome of interest. Through discussion with experts, several possible reasons exist. First, there are distinct differences among models both in terms of the data sources and steps used to construct analytic files. Another difference reflects how behavioral assumptions are incorporated into the models. While many health-related MSMs use empirical estimates from the scholarly literature, there is extensive variation in the amount, quality, and certainty regarding the evidence for particular types of behavior. In the context of insurance expansion MSM applications, for example, the scholarly literature has much stronger evidence regarding the price sensitivity of workers in their decision to take up employer-based coverage as compared with estimates of price sensitivity for persons seeking coverage in the individual market. Finally, comparisons of estimates generated by competing models are made more challenging by the use of different reference periods, whereby one model might report the outcome of interest as though it were fully implemented today while another might report its estimate for the first year during which the policy is expected to be fully implemented.

What are Some Areas for Improvement and Investment for the Development and Use of Health-Related MSMs in Future policy making Activities?

In a November 2012 meeting on microsimulation sponsored by the Office of the Assistant Secretary for Policy and Evaluation, U.S. Department of Health and Human Services, several experts put forth ideas to advance the development and use of health-related MSMs in policy making activities. Among the many ideas proposed, two themes emerged, including (1) improvements in the clarity and transparency in the documentation and communication of modeling methods and output and (2) investments in data, measurement, and research evidence to improve certainty about behavioral assumptions used within MSMs.

Improving Transparency and Clarity in the Documentation and Communication of MSM Results

Although MSMs can provide valuable information in the development and implementation of public policies, many potential users and other interested parties have difficulty understanding what is inside the “black box” of health-related MSMs. Overall access to MSM documentation with details of data development, behavioral assumptions, baseline output, and simulated results has become more widespread in recent years. However, as the development of MSMs can include hundreds of decisions related to analytic file development and specification of the models, it is often impractical to document every detail. Some modelers have noted that publicly available documentation reflects what is deemed important to the team creating the model. Others have indicated that private-sector modelers may simply wish to not reveal too much information, lest they lose competitive advantage. The evolving nature of MSMs presents another challenge. There can be considerable gaps in time between the version of documentation available for a given MSM and the actual version of the model in use. Thus, one suggestion for improvement is to document clearly the version of the model being used to generate any estimates.

Beyond understanding any one MSM's “black box,” there is the related issue of how to compare across MSMs' “black boxes” to try to identify reasons why estimates might vary for the same outcome. For potential users and modelers themselves, the issue at hand is whether the variation in results across models is due to differences in data, assumptions, or something else. One proposed strategy is to encourage organized communications among the modeling community to really understand the “nuts and bolts” of each model. Although this may provide valuable “peer-to-peer” learning among modelers, particularly for those within the Federal government, it seems less likely that modelers in the private sector would be as amenable to sharing such detailed information with their competition. Moreover, one expert noted that such activities would yield the largest return if done during the development phase rather than after each model is built and producing output that is being released into the public domain.

When examining existing health-related MSMs, one important way in which these models diverge is in their approach to incorporating or estimating behavioral responses to the policy provisions of interest. Scholarly evidence can vary widely both in terms of published estimates and in the overall number of studies on which to rely. Sensitivity analyses can be useful for assessing how the outcomes change with the modification in a particular model assumption. While the value of sensitivity analyses can provide greater assurance to modelers and analysts, its value to policy makers may be less, as lawmakers often prefer point to interval estimates.

A third strategy to facilitate reconciliation among model estimates, as proposed by Glied, Remler, and Graff Zivin (2002), is to build a reference case, which includes a well-documented set of assumptions and parameters utilized in MSM estimation. Using a reference case, policy makers can more easily compare across models and begin to understand which differences in the results are due to behavioral assumptions and which ones may be due to data. While commended as a good idea in theory, some experts have questioned whether the value gained from this type of exercise would exceed the necessary time, money, and coordination costs.

Investments in Data, Measurement, and Research Evidence

Microsimulation models are only as good as the input data from which they can be developed and estimated. Most health-related MSMs utilize federally sponsored population surveys (e.g., Current Population Survey, Medical Expenditure Panel Survey, American Community Survey), employer surveys, and administrative data to develop the core analytic file. Although progress is being made with respect to the quality and scope of data collection efforts, there are still notable limitations with respect to the types of information being collected and modelers' ability to access existing data sources that are not available in the public domain.

One concern raised by the modeling community is the imbalance in investment between cross-sectional and longitudinal data sources. While cross-sectional data can provide researchers with large sample sizes and greater precision for generating distributional effects, longitudinal data have the advantage of modeling both short- and long-run effects of a policy change for the same set of individuals.

Even among cross-sectional data, no one source contains all the necessary demographic, economic, and health-related information for an individual in the population. Thus, modelers typically rely on multiple data sources along with statistical matching and imputation techniques to generate a comprehensive “picture” of individuals. As some have noted, the process of building the data file does not require just one match or imputation, but these methods may be used at multiple points in analytic file development. Although there have been notable advances in statistical methods and software to execute such techniques, some experts have noted that more work should be done to ensure that in the application of matching and imputation techniques, appropriate covariance structures are maintained between the variable of interest and other key variables in the model.

A third issue recognizes that different types of data sources are used in MSM development and that each has relative strengths and weaknesses. Many existing health-related MSMs rely primarily on survey-based data. As a result, modelers frequently must deal with both underreporting and misreporting of key information. Within demand-side models, the measurement of coverage provides an important example, whereby large disparities existed historically between survey-based estimates of Medicaid enrollment and administrative records. With investments by government and private foundations, researchers have been able to analyze administrative enrollment and survey data together to identify potential reasons for these differences and to propose changes to survey methods to improve the measurement of coverage.

Another context in which there may be important gains from utilizing both administrative and survey data together is in the measurement of income. As noted by several MSM experts, administrative income data (e.g., IRS tax filing information) are likely to be more accurate and complete than survey responses. However, there are clear trade-offs in using administrative data. The first is that tax filing units may not necessarily align with the set of individuals who comprise a household or family. To the extent that household or family income is the relevant measure of income for determining eligibility for public programs, this may be problematic. Second, tax filing data often miss nonfilers who tend to be lower income and are often more likely to qualify for many social- and health-related programs. Surveys are more likely to capture these individuals. Despite these trade-offs, some have noted that having aggregated tabulations from administrative data sources for which the microdata are prohibited from release could still provide value to the modeling community. Notably, such tabulations could provide modelers with a more accurate income distribution, enabling them to gauge the magnitude of the reporting error from surveys and to improve upon existing imputation routines for survey data.

To date, much investment in model development has focused on the demand side of health care markets to predict insurance coverage and spending by individuals. With passage of the ACA, there is now greater need for the development of MSMs that can address policy questions related to the supply side of health care markets. For example, which types of policies are relatively more effective and efficient for increasing capacity in local markets to respond to increased demand for medical care resulting from coverage expansions and demographic shifts? Or, how might hospitals, physicians, and other delivery-focused organizations respond to changes in payment systems (e.g., pay-for-performance, bundled payments, and global capitation)?

Progress in the development of supply-side MSMs will likely require new investment in data collection to ensure comprehensive measurement of insurers and providers. For the former, the introduction of the National Association of Insurance Commissioners' Supplemental Health Care Exhibit (beginning for the 2010 plan year) now permits researchers to more accurately measure the size and degree of competition in insurance markets across states and segments (individual, small group, and large group). However, there is still very limited information about the types and shares of products offered by these firms.

Several data sources exist for measuring provider capacity and specific attributes of providers. Some of the most common include the American Hospital Association Annual Survey, the American Medical Association's Masterfile, and the Community Tracking Study's Health Tracking Physician Survey. Federal surveys sponsored by the National Center for Health Statistics, such as the National Ambulatory Medical Care Survey, can also provide some information for use in supply-side modeling. Currently, the modeling community faces a number of data-related barriers for estimating provider responses to proposed policy changes. Fundamentally, systems must be in place to ensure accurate assessments of provider capacity, including hospitals, physicians as well as nonphysician clinicians. As the organization of care delivery transforms, it is also vital to have consistent collection of information to capture contractual relationships among different provider types (e.g., physician-hospital integration) as well as other economic behavior.

As existing MSMs are refined and new ones are developed, there may be value in identifying and prioritizing areas in which research investments are needed given the lack of evidence. In some instances, the questions that need to be asked are fairly narrow. For example, there is still limited scholarly research on employer decision making with respect to offering health insurance. However, in 2014, with subsidized Exchange-based insurance for low-income Americans without access to affordable coverage, there is an increased need to understand how workers and employers will respond to this new option. For many types of supply-side proposals that encourage innovation in the organization and financing of care, even less scholarly evidence exists to measure the intended and unintended behavioral implications of such policies.

Looking Ahead

The past two decades have seen tremendous advances in the development and use of health-related microsimulation models for policy making activities. Looking ahead, there is still much room for identifying ways to facilitate their use and usefulness in policy making discussions. As the number and types of applications grow, it will be important for the modeling community, government, and other entities that value such tools to make critical investments in data and research to inform model development, particularly in areas that are understudied such as the supply side of health care markets. In addition, efforts to improve the transparency of MSMs, so as to illuminate what is inside the “black box,” will help potential users as they develop their strategy for using microsimulation models in policy development and implementation.