Developing a Policy Flight Simulator to Facilitate the Adoption of an Evidence-Based Intervention

Abstract

While the use of evidence-based interventions (EBIs) has been advocated by the medical research community for quite some time, uptake of these interventions by healthcare providers has been slow. One possible explanation is that it is challenging for providers to estimate impacts of a specific EBI on their particular organization. To address that concern, we developed and evaluated a type of simulation called a policy flight simulator to determine if it could improve the adoption decision about a specific EBI, the transitional care model (TCM). The TCM uses an advanced practice nurse-led model of care to transition older adults with multiple chronic conditions from a hospitalization to home. An evaluation by a National Advisory Committee, made up of senior representatives from various stakeholders in the U.S. healthcare system, found the policy flight simulator to be a useful tool that has the potential to better inform adoption decisions. This paper describes the simulation development effort and documents lessons learned that may be useful to the healthcare modeling community and those interested in using simulation to support decisions based on EBIs.

Convincing clinicians to adopt evidence-based interventions (EBI) has proved challenging. This paper describes the development and evaluation of a "flight simulator" to help clinicians determine the impact of putting a specific EBI into effect in their practices. We document lessons learned that may be useful to the healthcare modeling community and those interested in using simulation to support decisions based on EBIs.

I. Introduction

While the use of evidence-based interventions (EBIs) has been advocated by the medical research community for quite some time, uptake of these interventions by healthcare providers has been slow. One possible explanation is that it is challenging for providers to estimate impacts of a specific EBI on their particular organization. The randomized controlled trials (RCTs) used to evaluate EBIs measure efficacy independent of the environment of a given organization in the larger healthcare system. Factors such as provider organizational structure, local patient demographics, and payment systems can potentially affect the feasibility and effectiveness of real-world implementation of an EBI. It was hypothesized that employing simulation as a mechanism to produce tailored projections of EBI adoption impacts would alter adoption decisions by increasing decision maker confidence. To evaluate that hypothesis, this study focused on the adoption decision for one specific EBI, the Transitional Care Model (TCM). The study was made up of two components: 1) an effort to develop a simulation that translated TCM RCT and comparative effectiveness results to health care system impacts, and 2) a qualitative analysis of the simulation development’s effect on the adoption decision process. This paper reports the lessons learned from the simulation development effort. The results of the qualitative analysis will be published separately.

To develop the simulation, we applied a ten-step methodology for modeling complex systems and enterprises developed by Rouse [1]. The motivational concept behind this methodology is to develop a policy “flight simulator” that leverages interactive visualizations to allow key stakeholders to “test drive the future.” The assumption was that if stakeholders could see and internalize how the adoption of an EBI would affect their organizations under various scenarios, they would make more informed adoption decisions relative to their business-as-usual decision making approaches. To ground the effort, the Transitional Care Model was employed as a real life test case.

The net result of the effort was a pair of simulations: one modeled an individual provider system’s decision to adopt TCM and the other modeled a subset of the factors that affected widespread adoption of TCM across the US healthcare system. The single provider simulation provides a data-driven projection of the financial impact of adopting TCM for any hospital in the United States. The widespread adoption simulation provides a more stylized and qualitative analysis of how various factors could influence the spread of TCM from a few early adopters to the entire US healthcare system.

It is important to note that the long-term objective of this line of research is to improve decision making in healthcare and not to evaluate the efficacy of a particular policy or intervention. Consequently, the simulations were evaluated for their utility as general decision support tools by a National Advisory Committee (NAC) of senior executives from healthcare systems, insurance companies, and healthcare purchasers. In summary the evaluation was positive and the NAC members felt that the resulting simulations would improve decision making. However, as with any simulation development effort, there were areas for improvement and lessons learned. The intent of this paper is to explain how the simulations were built and to convey the lessons learned to the healthcare modeling community and those interested in facilitating the adoption of EBIs.

The remainder of this paper is organized as follows. Section II provides background on the problem of adoption of EBIs in the US healthcare system, TCM, and modeling healthcare systems. Section III explains the approach to developing and evaluating the simulations. Section IV describes the resulting simulations as well as the results of the NAC evaluation. Section V describes the limitations of the simulations and the lessons learned from this effort both with regard to modeling healthcare systems and developing policy flight simulators in general. Finally, Section VI concludes the paper and describes future work.

II. Background

A. Challenges to Adopting Evidence-Based Interventions

“Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” [2]. Eddy [3] reviews this construct and reports that as little as 15% of medical practice meets this standard. There has been no similar study of care management interventions for populations, but the absence of an evidence base is surely at least as high in that setting. Practice has long been predominantly based on precedent, i.e., making decisions based on learning in medical school and residency, even if that was many years ago.

Why is rigorous evidence of patient benefits often insufficient to ensure the adoption of an EBI when it should be adopted, or inhibit adoption of something unproven by evidence? RCTs may be the gold standard for developing and testing evidence based pharmaceutical and clinical interventions in healthcare, but translating the intervention into practice is a slow process [4]–[8]. RCTs are viewed as idealized implementations of an EBI and thus not reflective of organizational, economic, social, and behavioral constraints of the “real world.” Consequently, translating EBIs into clinical practice has been challenging in healthcare [9], [10]. This means that there have been missed opportunities to improve patient care, even when existing precedence-based care may be ineffective and potentially harmful [3], [11].

Following the landmark studies To Err Is Human [12] and Crossing the Quality Chasm [13], the Institute of Medicine (IOM), now the National Academy of Medicine, formed the Roundtable on Evidence-Based Medicine in 2006. The Roundtable set a goal: “By the year 2020, 90% of clinical decisions will be supported by accurate, timely, and up-to-date clinical information, and will reflect the best available evidence.” [14].

In pursuit of this goal, the IOM teamed with the National Academy of Engineering to bring systems engineering methods, processes, and tools to address the transformation of the delivery system [15]. This reflected the explicit recognition that the delivery system, as a whole, affects the possibilities of achieving this goal. This conclusion was strongly reaffirmed by a 2014 report of the President’s Council of Advisors on Science and Technology [16]. These recommendations motivated the idea of applying simulation techniques developed by the authors to the adoption of EBIs.

B. The Transitional Care Model

The Transitional Care Model (TCM), designed and tested by researchers at the University of Pennsylvania, focuses on improving care and outcomes for chronically ill older adults as they transition from the hospital back to the community. Three RCTs [17]–[19] and one comparative effectiveness trial [20] have consistently demonstrated improved patient reported health, quality of life and experience with care outcomes as well as reductions in rehospitalizations and costs over the past 25 years. Based on this body of research, the then Coalition for Evidence-Based Policy recognized the TCM as a “top-tiered” evidence-based approach that, if scaled, could positively impact the outcomes of hospitalized older adults and reduce total healthcare costs [21]. In partnership with payers and supported by multiple foundations, the clinical and economic findings of the TCM demonstrated in RCTs have been successfully replicated in health systems in the US [22], [23], but uptake of the model has been one system at a time. Traditional adoption strategies (e.g., identifying local champions, multiple meetings with decision makers) consume substantial time and are not as efficient as desired in promoting widespread scaling. One potential strategy to enable decision makers to understand the implications of TCM for their specific organization is through the application of a “policy flight simulator.”

C. Policy Flight Simulators

Policy flight simulators are multi-level computational simulations designed for the purpose of exploring alternative management policies at levels ranging from individual organizations to national strategy [24]. They are explicitly multi-level in the sense that they simultaneously consider the enterprise from multiple perspectives such as organizational structure, business processes, ecosystem behavior, individual practices, etc. The purpose of a policy flight simulator is to enable decision makers to “fly the future before they write the check.” People interact with these simulators almost always in groups rather than individually, often with different stakeholders in conflict about priorities and courses of action.

The flight simulator concept captures the essence of how interactive simulations can enable stakeholders to explore alternative organizational designs computationally rather than physically. Such explorations allow rapid consideration of many alternatives, perhaps as a key step in developing a final vision and plan for transforming an enterprise [25].

Policy flight simulators serve as boundary spanning mechanisms, across domains, disciplines and beyond initial problem formulations. Such boundary spanning results in arguments among stakeholders being externalized. The alternative perspectives are represented by the assumptions underlying and the elements that compose the graphically depicted model projected on the large screen. The debate then focuses on the screen rather than being an argument between two or more people across a table. These characteristics of policy flight simulators made them a natural choice to address the challenges associated with the widespread adoption of TCM.

D. Application to Healthcare Modeling

The application of simulation to the healthcare domain is widespread as prior reviews of the healthcare simulation literature attest [26]–[28]. However, the methods and resulting simulations are often specific to the project involved [26], often to a particular organizational unit or facility with strict boundaries and little potential for reuse [28]. However, the US healthcare system is a complex system, consisting of many interacting organizations, and it has been asserted that complex systems are intrinsically multi-scaled [29], [30]. Consequently, modeling the larger dynamics of the adoption of EBIs requires an explicitly multi-level approach. To that end, Rouse and Serban suggest the use of multi-level policy flight simulators to address the complexity of the healthcare system [31].

While multi-level or multi-scale simulations are fairly widespread in the physical sciences and biology [32]–[34], their applications to complex socio-technical systems such as the US healthcare system have been less common. Most relevant to this study are agent based simulation efforts that perform strategic modeling at the system level, which usually incorporates stakeholders such as governments, hospitals, clinics, nursing homes, physicians, insurance companies, and pharmaceutical companies to assess policies and initiatives [35]–[37].

However, as noted above, simulating the adoption of an EBI required an explicitly multi-level approach. Rouse [1] provides such an approach that evolved from an earlier application of multi-level modeling to evaluate an employee wellness program [38]. It was later formalized and applied to model policies to combat the intrusion of counterfeit electronic parts into a supply chain [39]. Consequently, it provides a promising approach to develop a policy flight simulator for the adoption of the Transitional Care Model.

E. Key Terms

• Provider –An organization that provides health care services to the general public. It may be for-profit, non-profit, or public. For this particular study, we are considering providers that control one or more hospitals.

• Payer –An organization that pays a provider for healthcare services for another person. In the US this may be a private insurance company or a government entity.

• Purchaser –An organization or individual that purchases a healthcare coverage from a payer. In the US, a company would be a purchaser if it purchases private health insurance for its employees.

• Patient –An individual person that consumes health care services from one or more providers.

III. Methods and Procedures

To provide the requisite expertise to execute the study, we employed an interdisciplinary team. Faculty members from the University of Pennsylvania School of Nursing and the Wharton School involved with the development and evaluation of TCM provided expertise on the TCM and associated patient care issues. Faculty members from the Stevens Institute of Technology School of Systems and Enterprises provided expertise on developing multi-level simulations of socio-technical systems. Subject matter expertise on the operation of the US healthcare system was provide by a National Advisory Committee (NAC). This committee consisted of senior healthcare leaders representing two different provider systems, two major health insurance companies – one insurance executive was formerly with the Center of Medicare/Medicaid Services (CMS) –, and one executive from a firm that represented private companies that purchase health insurance for their employees.

For this study, we applied the ten-step methodology for modeling complex enterprise systems developed by Rouse [1] and evaluated by Pennock et al. [39]. While Rouse provides much greater detail on the steps, here we will just provide an outline:

• 1)Decide on the Central Questions of Interest
• 2)Define Key Phenomena Underlying These Questions
• 3)Develop One or More Visualizations of Relationships Among Phenomena
• 4)Determine Key Tradeoffs That Appear to Warrant Deeper Exploration
• 5)Identify Alternative Representations of These Phenomena
• 6)Assess the Ability to Connect Alternative Representations
• 7)Determine a Consistent Set of Assumptions
• 8)Identify Data Sets to Support Parameterization
• 9)Program and Verify Computational Instantiations
• 10)Validate Model Predictions, at Least Against Baseline Data

It is important to note that the methodology is not a fixed procedure and is intended to be tailored. Consequently, it is necessary to describe how each step was executed and tailored through the course of the study. The simulation was developed using a standard spiral development approach, where each spiral involved evaluation by the NAC and sponsor with subsequent refinement of the requirements. Thus, there was a certain amount of iteration over the ten steps. However, for the sake of clarity, we will explain our execution of the ten-steps in a sequential fashion.

A. Step 1: Decide on the Central Questions of Interest

A series of structured interviews were conducted with members of the NAC. The aim of the interviews was to understand the concerns of key stakeholders in the US healthcare system with regard to both the adoption of EBIs in general and the TCM in particular. Questions focused on eliciting critical concerns about EBIs as well as facilitators and barriers that affect EBI adoption decisions. The interviews generated a large number of questions of interest that fell into two categories related to:

• 1)Individual provider adoption decisions; and
• 2)Influencing widespread adoption across the US healthcare system.

As will be seen in the subsequent steps, these two categories suggested differing perspectives that had a substantial impact on the development of the simulation.

B. Step 2: Define Key Phenomena Underlying these Questions

To identify relevant phenomena, the data from the structured interviews were coded using conventional content analyses techniques [40]. A fairly large set of potentially relevant phenomena emerged as important factors that influences both the decision to adopt EBIs, including the TCM, and widespread adoption. Examples include payment models (What will the cost be? Will the cost be covered by existing reimbursement or require an increase in revenue?), patient demographics (How will this work with my population?), organizational culture (Will the organization need to change how it operates to make this work?), and return on investment (How long will it take to break even?).

Despite the fact that the methodology indicates that the modelers should identify “key” phenomena in Step 2, we found that we were really only able to trim the list after we developed the visualizations in Step 3.

C. Step 3: Develop One or More Visualizations of Relationships Among Phenomena

While not explicitly called for in the methodology, we felt that we could not develop useful visualizations without some consideration of the structure of the system that we were attempting to model. In this particular case, we were effectively trying to model a causal chain starting with a decision to adopt an EBI through a series of intermediate consequences to impacts on outcome metrics of interest. This naturally led us to use influence diagrams as a formalism.

We organized the set of phenomena identified through the structured interviews from the NAC into groups defined by each stakeholder perspective (providers, payers, and purchasers). We then built an influence diagram for each perspective. For the first iteration, we were uncritical. We simply attempted to capture implications of what was important to healthcare leaders. This resulted in fairly complex influence diagrams that were difficult to interpret. Furthermore, a computational simulation was not the appropriate mechanism to capture many of these phenomena (e.g., organizational culture). At the same time, we did not want to lose any important information. Consequently, we sorted the elements in the diagrams into three groups: those that we thought could be computationally modeled, those that may not be explicitly modeled but could affect the scenarios that we would evaluate, and those that we thought we would not be able to address in this effort. The last group serves two important functions. First, it is important to inform decision makers of important factors that were not included in the model. Second, these factors may be considered for analysis in future studies.

Following the sorting efforts, the influence diagrams were refactored. As part of the refactoring, the phenomena were grouped and characterized to create cleaner, more interpretable representations. The intent was not to create the perfect influence diagram but rather to provide a basis for further discussion and identification of the key tradeoffs in the next step.

D. Step 4: Determine Key Tradeoffs that Appear to Warrant Deeper Exploration

The influence diagrams developed in Step 3 were discussed with the research sponsor and provided a basis for discussion within the study team itself. Out of this discussion and analysis, we identified a number of key tradeoffs organized by the two categories of questions identified in Step 1. The payers and purchasers were more interested in the systemic issues that emerge from the adoption decisions and interactions of the providers, payers, and patients, but the providers were more interested in individual TCM adoption decisions. Of course, the two categories of questions are interrelated as the rational behavior of the individual providers leads to emergent behaviors and unintended consequences in the US healthcare system as a whole. Thus, we extracted two sets of tradeoffs from the influence diagrams.

Regarding an individual healthcare provider’s decision to adopt, the tradeoff effectively reduces to improving patient outcomes versus maintaining financial viability. For example, under a fee-for-service payment system, reducing patient admissions also means a loss of revenue. So while the provider would like to improve patient outcomes, if the cost is too great, implementation may be infeasible. There are several nuances to this tradeoff:

• •The payment system, whether fee-for-service, capitated, or bundled, could have a dramatic impact on financial viability because it affects who bears the costs and who reaps the benefits
• •The targeted patient population, particularly in terms of DRGs, impacts the benefit/cost trade
• •The bed occupancy rate of the hospital matters: an overcapacity hospital may view reduced 30-day readmissions as a relief, while a below capacity hospital might encounter cost issues due to underutilization
• •It is not certain that the provider will achieve the same outcomes as the RCT. There is a risk that the benefit/cost trade will underperform expectations.

Regarding widespread adoption, the tradeoff is between costs (to both payers and purchasers) and risk avoidance. In essence, the payer and/or purchasers may be willing to expend resources to buy down risk. If we consider the public payer, there is an additional objective of improving the overall health of the US population. Again there are several nuances:

• •The payment system, whether fee-for-service, capitated, or bundled, shifts the incentives and costs among payers and providers
• •Additional evidence beyond RCTs such as peer adoption can reduce providers’ uncertainties regarding outcomes but may be expensive to obtain
• •Transitional care is only one among many number of challenges that a provider must consider. Providers have limited resources, and any initiative that a payer decides to support is competing for attention.
• •Reducing care consumption in one category may increase it in another
• •Long-term demographic trends and shifts in eligibility impact viability, particularly for the public payer
• •There is uncertainty about the impact of an intervention on measures of financial outcomes.

Based on the results of steps one through four, we developed a plan to design and develop two simulations. This plan was briefed to the NAC members for feedback and course corrections.

E. Step 5: Identify Alternative Representations of these Phenomena

As noted by Pennock et al. [39], there is no specific step to develop a simulation architecture in the ten-step methodology. Rather, it seems to be a natural outcome of Steps 5 through 7. Consequently, we executed these steps with this objective in mind. When considering potential representations for the phenomena identified, we note that the two perspectives, single provider adoption and widespread adoption, naturally lead to two different layers of abstraction: the operations of a single hospital versus the interactions of the entire population of hospitals in the US healthcare system. Of course, there are any number of ways one could represent these levels of abstraction. To drive the selection, we started with the constraints. More specifically, there is a relatively fixed set of inputs that a decision maker could affect and output metrics that he or she would be interested in. This effectively determined portions of the simulation ontology. Furthermore, the key tradeoffs naturally lead to certain scenarios of interest that must be properly captured in the simulation. This further constrains the selection.

• Individual provider adoption:

  • •
    Inputs: patient demographics, readmission history, occupancy rates, revenues, TCM characteristics
  • •
    Outputs: estimated annual cash flow resulting from TCM adoption
  • •
    Scenarios: payment system, penalty, workforce, eligibility
• Widespread adoption:

  • •
    Inputs: TCM characteristics, provider beliefs about TCM, competing investment characteristics, care usage rates, cost rates, payment rates
  • •
    Outputs: % of providers adopting TCM
  • •
    Scenarios: payment system, penalty, value of evidence, competition

The individual provider decision is effectively driven by cash flows. An income statement is a natural representation for this level. The widespread adoption decision is effectively driven by the emergent behavior of a population of autonomous decision makers. As a result, an agent-based model is a natural representation for this level.

F. Step 6: Assess the Ability to Connect Alternative Representations

While multi-level or multi-method approaches have become popular in the literature, our past work developing and analyzing such models suggests that they may be counterproductive for some socio-technical problems. Literally implementing a separate representation for each level of abstraction and then connecting them can create internal contradictions and computational issues [39], [41]. Representing the same objects in the real system, in this case patients, providers, and payers, in more than one way can lead to inconsistencies. When implementing a simulation that integrates multiple perspectives, one tends to follow one of two strategies: creating a consistent representation that subsumes all abstractions, or partitioning the state space such that a different abstraction is applied to each partition. In this particular case, we actually had two different sets of decision makers that were interested in two different (but related) causal chains. Our concern was that by attempting to build one simulation that satisfied both, we would end up satisfying neither. Thus, we made a deliberate decision not to computationally connect the levels of abstraction. Instead, we decided to develop two independent simulations that relied on the same basic set of data and assumptions regarding the reference system (i.e., the US healthcare system). From here on, we will refer to these as the single provider adoption model and the widespread adoption model.

G. Step 7: Determine a Consistent Set of Assumptions

Since the intent of the individual provider model was to describe the provider system at a greater level of fidelity than the widespread adoption model, we set the assumptions for that model first. Then we adjusted them as necessary to accommodate the widespread adoption model. In some cases, the assumptions were identical. For example, the penalty formula used by CMS to reduce payments to hospitals with excessive 30-day readmissions was exactly the same in both models. However, in other cases, the assumptions had to be adjusted to accommodate differences in model ontology. For example, the single provider model allowed fine grained adjustments in the specific rates of patient hospital readmissions by different diagnosis groups, whereas the multi-provider model just used an annualized average rate. However, consistency was maintained by deriving the relevant parameters from the same data sets.

H. Step 8: Identify Data Sets to Support Parameterization

Parameters related to the impact of TCM were drawn from published RCTs and comparative effectiveness studies [17]–[20]. Parameters related to individual hospitals and their individual patient populations were drawn from commonly used data sets from CMS including:

• •Provider Utilization and Payment Data Inpatient Public Use File [42]
• •Hospital Cost Report File (CMS-2552-10) [43]
• •Hospital Comparison Data [44]
• •Readmissions Reduction Program (HRRP) [45]

In addition, two partner healthcare systems provided proprietary data sets to use in specific validation scenarios for their systems.

I. Step 9: Program and Verify Computational Instantiations

Because we decided to implement two separate computational instances, we were able to choose a simulation tool appropriate to each perspective as opposed to forcing both into a single tool. Since the single provider model required on-the-fly statistical analysis of large hospital data sets, we implemented it using the statistical software R with Shiny. Shiny provided the web-based, user-friendly interface. Since the widespread adoption model is fundamentally agent-based, we implemented it using a multi-method simulation packaged called AnyLogic. AnyLogic includes built in controls for interface development.

J. Step 10: Validate Model Predictions, at Least Against Baseline Data

Since the intent of both models was to analyze the impacts of adopting TCM where it had not been previously adopted, there was no obvious data set to validate against beyond the data used to parameterize the model. Instead the NAC members reviewed and evaluated the models for validity. In contrast to the previous interviews and discussions with the NAC members, several of the evaluation sessions included additional subject matter experts from the respective NAC member’s organization to provide more thorough scrutiny. We asked the participants to evaluate the simulations using the following criteria:

• •Validity — the extent to which the simulation is technically correct relative to the purposes for which it was developed
• •Acceptability — the extent to which the simulation addresses problems in ways that are compatible with current preferred ways of decision-making and/or potentially useful new ways of multi-stakeholder decision-making
• •Viability — the extent to which use of the simulation for the purposes intended would be worth the time and effort required

We also asked the participants to answer the following questions:

• •Is this a useful way to model the adoption decision?
• •Are there corrections needed in the current model?
• •Are there important aspects of the adoption decision not currently modeled?
• •Are there additional factors for which the analyst should have controls?

The results of this exercise will be discussed in the following section.

IV. Results

As described previously, the output of the ten-step enterprise modeling process was a pair of simulations: the single provider adoption model and the widespread adoption model. We will briefly describe each simulation and then the outcome of the NAC evaluations of the pair.

A. Single Provider Adoption Model

The single provider adoption model offers an integrated, web-based graphical user interface (Fig. 1). The primary output is a probability distribution of a particular hospital’s projected net income after adopting TCM. This captures both the expected value of the outcome and the range of uncertainty about that value. Users may select any particular hospital from more than 3000 hospitals in the US as well as adjust payment system parameters, patient eligibility criteria, bed replacement rates, etc. Uncertainty in the impact of TCM is introduced via a user adjustable subjective probability distribution. The output distribution is generated by aggregating three components: (1) hospital admission loss due to readmission reduction, (2) TCM cost, and (3) hospital readmission payment adjustment, also called the readmission penalty. Numbers of eligible patients and penalty data are actuals drawn from public CMS datasets. Monte Carlo simulation accounts for the potential variation in TCM efficacy, and the result is the distribution of net income.

FIGURE 1.

A portion of the graphical user interface for the single provider adoption model.

The single provider adoption model also provides a hospital comparison view. Many provider systems are made up of multiple hospitals and this view enables them to identify the hospitals where adopting TCM would have the highest potential impact.

B. Widespread Adoption Model

The widespread adoption model is focused on how patients, providers, and payers interact. It is designed around an agent based framework where each entity (a patient, a provider system, or a payer) is an agent in the model. The model describes how each agent type behaves and how it interacts with other agents. Most of the model’s behavior is driven by the provider. Each time step, every provider considers a range of possible EBIs, only one of which is TCM, and determines which, if any, to fund. To do so, each provider projects the financial impacts of each EBI and selects those with highest expected net present value subject to budget constraints. The user may adjust the cost and efficacy of TCM as well as the providers’ beliefs about TCM, payment system characteristics, patient population characteristics, and the arrival rate of competing EBIs. A representative portion of the interface is depicted in Fig. 2.

FIGURE 2.

Notional output screen from the widespread adoption model.

One of the key features of the widespread adoption model is that each provider performs Bayesian updating on its beliefs about the efficacy of an EBI based on the results achieved by its peers. Providers on the NAC indicated that peer results are one of the major drivers of adoption. The most important output is the percentage of providers adopting TCM, but there are also output displays providing adoption rates for other EBIs, provider financials, etc.

Admittedly, the widespread adoption model is more stylized than the single provider adoption model. Since it was not possible to know the decision model for each of the thousands of hospitals in the US, we assumed a risk neutral decision model where providers selected opportunities with the highest expected net present values, though this is adjustable within the simulation code.

C. Single Provider Simulation Evaluation

While the overall goal of the study was to assess the potential for a policy flight simulator to improve EBI adoption decision making in general, interactions with the NAC created the opportunity to evaluate the single provider simulation model for two specific healthcare provider systems. More specifically, the two provider systems associated with NAC members agreed to provide proprietary data relevant to the adoption of TCM for each of their systems. This enabled us to take a more in-depth look at how adopting TCM would impact these two systems.

Each of the provider systems is located in a major US metropolitan area. One system had already attempted to implement an adaptation of the TCM with varied success and the other was considering adopting the TCM.

The first system consisted of seven hospitals, though only four were potential TCM candidates. We used the single provider simulation to project the financial impact of adopting TCM at each of those hospitals. Based on the data available and current penalty rates etc., none of those hospitals would see financial benefits from adoption. The simulation revealed that the readmission penalties imposed by CMS on those hospitals were very low. Consequently, it was not possible to generate enough savings from penalty reductions to offset the cost of TCM. Interestingly, that system had previously discontinued its adaptation of the TCM in all but one specific group, patients with heart failure. However, given that they adopted a variation of TCM rather than the complete model, and heart failure patients were still receiving the adapted intervention, it is difficult to make a direct comparison.

The second system consisted of ten hospitals. We analyzed each of the hospitals using the simulation, and we identified two that were likely to experience positive financial impacts from TCM adoption. The simulation revealed two factors that led to this outcome: 1) The hospitals, at the time, had readmission penalties imposed by CMS which meant it was possible to reduce costs. 2) The hospitals had sufficiently large numbers of patients in the eligible DRGs to spread out the overhead associated with TCM startup. In essence, there was room to capture cost savings, unlike the first system. Representatives from that provider system indicated that they were already in the process of developing transitional care services across their hospitals, and that they would be interested in determining how the TCM could be adopted or adapted to meet their patient populations’ needs.

D. NAC Evaluation of Simulator Utility

Overall, the NAC assessment of the simulations was positive, and the reviewers felt that they would be useful for decision support. They were described as “highly sophisticated tools.” Obviously, these simulations were prototypes, and not surprisingly, the NAC indicated that the simulations would be even more effective if the simulations were tailored to each individual organization. They also indicated that they would want their own staff to go through the assumptions of the simulations in detail prior to their actual use in decision making.

They felt the simulations, assuming they were sufficiently tailored and understood, would be useful for “what if” analyses regarding adoption decisions. One NAC member suggested that the single provider model could be used as a negotiation tool between a payer and a provider to determine what proportion of the TCM cost the payer might share. It was also emphasized that simulations would be useful for examining other EBIs beyond TCM.

Several simulation improvement opportunities were identified through the NAC reviews including the ability to:

• •Compare provider and payer perspectives
• •Select individual DRGs for TCM eligibility
• •Introduce cost sharing between the payers and providers
• •Perform side-by-side comparisons of hospitals

Each of these capabilities were added as a result of this feedback.

Beyond the specific capabilities of the presented simulations, NAC discussions also revealed important limitations of the general model development approach. These will be discussed in detail in the next section.

V. Discussion

While the overall assessment of utility was positive, the TCM simulations were ultimately prototypes intended to test a modeling methodology. Consequently, there were a number of limitations identified via discussions with the NAC. There were also several lessons learned that could inform future attempts to build these types of simulations to address healthcare challenges.

A. Limitations of the Simulations

The following are the major limitations of the two simulations that were identified via the review process. These limitations apply to the simulations as implemented in this study, and suggest improvements that could be made in future evolutions of the tools.

1). Overemphasis of Financial Impacts

The payment system seemed to dominate the behavior of the simulations. That is not surprising since the model simulated financial impacts. Other valid concerns such as technical efficiency in the production of care and patient outcomes were not emphasized in the model. In part this was due to the fact that we had a well understood causal model for the financial impact whereas factors such as patient experience and quality of life were difficult to model due to lack of comparable national data from providers, and even more difficult to convert into a common metric to compare with the financial impacts. The takeaway was that the payment system is the binding constraint in the US healthcare system, and it must be addressed before serious attempts can be made to address the other concerns.

2). Risk of Misleading Senior Decision Makers

One participant in the simulation reviews observed that the simulations would be extremely useful for subject matter experts and policy analysts, but their use by “c-suite” executives would be concerning. In particular, it was noted that the simulations would be useful for analysts to perform “what if” analyses quickly. However, any scenarios presented to senior executives would need to be carefully vetted in advance. As with any simulation, there is a risk that a particular scenario would push it out of the zone of model validity. As a result, direct use of the simulations by someone who does not understand their inner workings risks drawing inaccurate conclusions. This raises a significant point about how policy flight simulators should be used that will be discussed further in the next section.

3). Limited Representation of Provider Decision Making

Factors such as risk attitudes, organizational specific concerns, organizational culture, and informal information sharing relationships among colleagues are critical to understand but nearly impossible to know and represent accurately for the entire collection of healthcare providers in the US. Consequently, these were captured in a stylized way in the widespread adoption model. One can create qualitatively different provider “types” and experiment with potential information sharing networks, but this can only generate qualitative findings as opposed to specific quantitative predictions.

4). Limited Representation of Patient Biological and Psychological Responses

Here there were two issues. First, while we had high quality data on the resource use and cost impact of TCM from RCTs and comparative effectiveness studies, these were effectively point estimates from an ideal research environment. Many of the NAC members were skeptical that real provider organizations would achieve the same results. For example, one common objection by providers is “our patients are different.” This is despite that the fact that RCTs found that the effectiveness of TCM did not vary by demographic characteristics of the patient population such as age and race. They felt this way about all EBIs, not just TCM. While it is infeasible to test for interactions among all possible characteristics of a patient population, one approach would be to ask the respondent to suggest how their patients differ and then explore those differences empirically assuming the characteristic has been measured.

5). Inability to Customize TCM Within the Simulation

Providers were interested in customizing TCM. However, no RCT data exists on these customizations. We simply had no way to predict the effect of a customization of TCM by changing components such as type of clinician or eliminating a component such as home visits when implemented in a particular hospital. This would be true for any EBI. Consequently, we assumed that TCM’s efficacy was the same as what was reported in the RCTs. We then let the users input subjective beliefs about how effective TCM would be in a particular organization. This was not an entirely satisfactory solution as it leads one to question what basis the user had for generating the subjective distribution.

6). Lack of Patient Outcomes

Improving patient outcomes in terms of experience with care, quality of life, etc., are important objectives for an EBI such as TCM. While we initially planned to incorporate these outcomes into the simulations, we found that data on these factors are not consistently collected across the US healthcare system. Patient experience with care and quality of life was collected as a part of TCM studies, but the questions and evaluation scales were not necessarily the same as was what was used at any given hospital. Consequently, we did not have a way to make a fair comparison. Ultimately, we left these factors out of the simulation, but a consistent mechanism for collecting patient outcomes across the US healthcare system might have allowed us to include these important outcomes in the simulations.

B. Lessons Learned About the Approach

Beyond the simulation tools themselves, we also learned several lessons about the modeling approach itself. These lessons suggest future improvements to the ten-step methodology.

1). Despite the Multi-Level Nature of EBI Adoptions, it was not Necessary to Build a Single, Fully Integrated Simulation

The ten-step methodology calls for the development of a multi-level model. In particular, it calls for identifying relevant levels of abstraction and then assessing the ability to connect them. As discussed in Section III, this is difficult to perform in a literal way. This effort reinforced that assertion. Connecting the representations for the single provider level and the widespread adoption level would have implicitly and in some cases explicitly modeled the same patients, providers, and payers twice. This creates a major consistency issue. Originally, our intention was to refactor the representations to create a single integrated model that could address both perspectives simultaneously. However, as noted in the previous section, a lack of accurate representations for the provider decision process pushed us toward more stylized representations to understand widespread adoption. If we had moved forward with our plan to create a single integrated model, we would have limited the utility of the simulation to aid individual adoption decisions because we would have contaminated the single provider results with stylized assumptions.

Consequently, we made the decision to create two separate simulations. We had enough high-quality data to project TCM impacts for an individual hospital with reasonable accuracy. Since the provider is the user for the single provider model, there is no need to create a representation of that decision process in the simulation. For the widespread adoption perspective, stakeholders generally only expect qualitative results as precise predictions are unlikely. Thus, the stylized nature of the widespread adoption model is less of an issue. Ultimately, the NAC reacted positively to two separate simulations. They felt it was the right decision given the differences in concerns of the two groups of stakeholders. While the tendency within the systems modeling community seems to be toward computationally integrating diverse system perspectives, at least in this case, that seemed to be counterproductive. As noted in past work [39], the multi-level view is useful for conceptualizing a system, but when it comes to implementation of a computational model, it should not be taken too literally.

2). Not Everyone Should be able to “Test Drive” all Futures

As noted previously, the purpose of a policy flight simulator is to allow stakeholders to “test drive the future.” One challenge to accomplishing that goal is that there is a tradeoff between the number of “futures” that the simulation can accommodate and the interpretability of the simulation. In an early version of the widespread adoption simulation we included input controls and output displays for virtually everything that the NAC said was important. As a result we had hundreds of controls, and the research sponsor observed that the simulation was confusing and difficult to interpret. To address this, we created two interfaces, a simplified one for use by the stakeholders, and a comprehensive one for our own internal use. Of course, by hiding controls, the stakeholders could no longer explore certain future scenarios of interest. We had to prioritize which scenarios they could explore in the simplified interface.

This issue is closely related to the risk of misleading senior decision makers discussed in Section V.A. It is very difficult, if not impossible, to validate the results of every possible scenario one could generate using a complicated simulation of a socio-technical system. Consequently, it may be inadvisable to provide stakeholders with the ability to explore these scenarios “on the fly,” and instead to focus on a few vetted scenarios. This raises the question of whether one wants a policy flight “simulator” or a policy flight “demonstrator.” The ability to simulate virtually any scenario on the fly may actually be detrimental from a communication standpoint and inadvertently lead to invalid conclusions by key stakeholders. The point was reinforced by feedback from the NAC reviews.

3). Plan for a Family of Simulations to Accommodate the Rapidly Evolving Healthcare System

The ten-step methodology calls for validation of the simulation against at least baseline data. For this effort, we used data on the past performance of hospitals as well as RCT and comparative effectiveness data for the impacts of TCM. The challenge, as one NAC member noted, is that the “baseline is always changing.” For example, the TCM RCTs compared it to standard practice, but standard practice is constantly changing or reacting to policy. Many hospitals may have already started to work to improve their readmission rates, and as a result, adoption of TCM may have a smaller impact than expected for those hospitals. This is a problem when modeling virtually any complex socio-technical system. They frequently change and adapt. Thus, historical data may no longer be representative by the time the simulation is built.

Consequently, the policy flight simulator we built, particularly, the single provider model, is tantamount to extrapolating the trend. Again, this suggests that we may want multiple simulations: one for extrapolating the trend accurately and one or more to identify qualitative shifts in system behavior. This recognition led us to consider a “core-peripheral” approach in some of our past work [39], where the core model captures the trend and the peripheral models are used to explore interesting scenarios. Regardless, the results of this effort suggest that adjustments to the ten-step modeling methodology are required to accommodate the need to both explore a wide range of scenarios but also provide focused communication to the most important stakeholders.

VI. Conclusions and Future Work

The objective of this study was to evaluate whether or not a “policy flight simulator” could improve confidence in decisions of whether or not to adopt an EBI. To that end, we employed a ten-step methodology developed by Rouse for modeling complex systems and enterprises, and applied it to a real world EBI, the Transitional Care Model. The result was two simulations, one representing an individual provider’s decision to adopt TCM and one representing the factors that affect the widespread adoption of TCM across the US healthcare system. An evaluation by senior representatives from various stakeholders in the US healthcare system found the simulations to be useful tools that they felt had the potential to better inform adoption decisions.

As a direct outcome of these discussions, we are considering extending the simulations to support of two future studies. One would involve using the simulations to facilitate negotiations between an actual health insurance company and an actual provider system. The other would involve tailoring the simulations to match the specific needs of an actual provider system and use them to support a TCM adoption decision.

Beyond the application of the simulations to TCM in particular, we also learned a number of lessons with regard to building a multi-level model of the US healthcare system. While a multi-level approach is useful for conceptualizing a complex system, one need not computationally implement a multi-level model. In this study, developing a separate simulation for each level was effective.

Relying on RCT data was extremely limiting, but there seemed to be no alternative. We compensated by allowing users to generate a subjective probability distribution, but this was not very satisfactory. A mechanism that would allow a more credible construction of a subjective distribution through decomposition or otherwise would make for a more useful decision support tool.

There is a tradeoff between the breadth of scenarios that a simulation interface can accommodate and the interpretability of that interface. Furthermore, it is challenging to validate a large number of scenarios. Consequently, the use of an extremely flexible interactive simulation by senior decision makers may actually result in erroneous conclusions. In light of the above lessons learned, we will be updating ten-step modeling methodology to accommodate some of these concerns. This will be documented in future work.

Funding Statement

This work was supported by the Robert Wood Johnson Foundation under Grant 72879.