Building Better Models: If We Build Them, Will Policy Makers Use Them? Toward Integrating Modeling into Health Care Decisions

This landmark issue of Medical Decision Making summarizes important deliberations about best practices for simulation modeling.^1–7 This body of work represents the third major set of recommendations for modeling best practices in the past 2 decades.^8,9 The current project, sponsored jointly by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making, builds on and extends prior efforts to set standards for the conduct of modeling.^10–13 These new articles provide modelers with guidance on building more useful models and consumers with benchmarks to judge the quality of the models. These thoughtful guidelines should strengthen the integrity of the model development process and encourage broader use of models in decision making.

Modeling has long been embraced by the medical decision-making community and other researchers concerned with evidence-based practices and outcomes. In 2009, simulation modeling was recommended by the Institute of Medicine as a method to quantify the net impact of medical interventions.¹⁴ More recently, the Patient-Centered Outcomes Research Institute has been charged with using a variety of methods, including modeling, to evaluate the comparative effectiveness of medical interventions (www.pcori.org). In addition, more complex and biologically accurate models are now possible because of computing and information technology advances.

Until recently, however, models affected few coverage and policy decisions in the United States. Models were used in setting American Cancer Society cervical cancer screening guidelines in the 1980s.^15,16 In the late 1980s, the US Congress’ Office of Technology Assessment (OTA) commissioned models to evaluate screening for cervical and breast cancer under the Medicare program. The results were influential in the decisions to extend Medicare benefits to include Pap smears and mammography between 1988 and 1990.^17,18 Unfortunately, the OTA was de-funded in 1995.¹⁹ Cancer modeling gained renewed support in 1999, when the Institute of Medicine released a report on the quality of care and called for greater inclusion of cancer outcomes research at the National Cancer Institute (NCI).²⁰ Meanwhile, outside of the United States, modeling has been and continues to be used to guide policy on a routine basis.^21,22 In the United Kingdom, for example, cost-effectiveness results from models are required for health care coverage decisions.²¹

Modeling of cancer interventions was advanced in the late 1990s with the funding of NCI’s Cancer Intervention and Surveillance Modeling Network (CISNET).¹² Over the past 15 years, CISNET has gained traction in advancing the use of modeling to inform policy and clinical practice. Several aspects of the CISNET modeling approach have contributed to its success, including use of best modeling practices, similar to those outlined in this issue; deployment of more than one model to address a specific research question; development of a template for the description of the models; and commitment to transparency and collaboration. For example, by having multiple modeling teams working together to use a common data set, CISNET has avoided some of the difficulty that can arise when single models come to widely divergent conclusions, as happened in the assessment of spiral computed tomography (CT) scanning.^23–26

Policy makers have approached CISNET to apply their extant models to inform emergent debates in cancer care. The CISNET models have been used successfully to inform Centers for Medicare & Medicaid Services (CMS) reimbursement decisions about fecal immunochemical tests, stool DNA tests, and CT-colonography for colorectal cancer screening.^27–29 They have also been employed by the United States Preventive Services Task Force (USPSTF) to inform their colorectal cancer³⁰ and breast cancer³¹ screening guidelines. Finally, several CISNET models were commissioned by the Healthy People 2010 initiative to determine the feasibility and impact of reaching goals for reductions in smoking prevalence³² and colorectal cancer mortality.³³ There are also examples of models outside of CISNET successfully informing policy and/or practice, including models of selected pharmaceuticals, radiological agents, vaccines, screening for HIV infection, and human papillomavirus testing in women with human immunodeficiency virus.^34–39

Even with these few success stories of models affecting recent policy decisions, there are considerable missed opportunities to use the plethora of existing high-quality models in making important health care decisions. The case of prostate cancer screening is a good example. Leading US organizations, including the USPSTF, the American Society of Clinical Oncology, the American Cancer Society, the American Urology Association, and the National Comprehensive Cancer Network, all offer different and conflicting recommendations about prostate-specific antigen screening.⁴⁰ This is a situation where models have not been used directly but could contribute to the ongoing debates within these professional groups by providing a formal weighting of harm v. benefit.

There have been other obstacles to moving the field of modeling science forward, especially when modeling and politics have collided. For instance, when the USPSTF issued revised breast cancer screening recommendations in November 2009, the final week of debates on the Patient Protection and Affordable Care Act, there was enormous public, political, and scientific push-back about changes in language about the age of screening initiation.^41,42 The CISNET models were even used by some in the radiology community to draw erroneous conclusions about the data.^43,44

Several other factors are likely to contribute to continued resistance to using models in decision making in the United States, including preferences for only using clinical trial evidence, a relative lack of historical modeling standards, and perceptions of models being “black boxes.” There is also variability in transparency based on the model funding source that needs to be considered by end-users of the models. Federal policies mandate data sharing, whereas the privately developed models have strong intellectual property investments that can limit disclosure of their methods.⁴⁵ Another potential barrier to advancing the use of modeling in US health care decisions is the turbulent political climate, including recent discussions about elimination of agencies and initiatives that support modeling.⁴⁶

Notwithstanding these considerable challenges, we should not lose sight of our impressive gains. We have come a long way in the evolution of modeling, and it remains a powerful method to quantify the balance of benefits, harms, and costs of candidate medical policies. High-quality models are especially salient now, when there is an urgent need to address spiraling health care costs related to the demographic pressures of an aging population and new technologies disseminating into use ahead of evidence about their impact. This milieu, coupled with the explosion of knowledge about the biological drivers of health, provides the modeling community with exciting new opportunities, including how to simulate multilevel influences on health, the impact of the “genomic revolution” on health outcomes, and linking “under-the-skin” cellular models to population models.⁴⁷ Past standards have facilitated broader acceptance and use of models. The accompanying standards for best practices in this issue will provide modelers with a roadmap for building even better models and policy makers with formal criteria for selecting models to inform their recommendations. It remains to be seen whether use of these best practices will be practical and, ultimately, whether they will facilitate more widespread use of high-quality models to inform future health care policies.