A Framework for Determining the Return on Investment of Simulation-Based Training in Health Care

Abstract

This article describes a framework that has been developed to monetize the real value of simulation-based training in health care. A significant consideration has been given to the incorporation of the intangible and qualitative benefits, not only the tangible and quantitative benefits of simulation-based training in health care. The framework builds from three works: the value measurement methodology (VMM) used by several departments of the US Government, a methodology documented in several books by Dr Jack Phillips to monetize various training approaches, and a traditional return on investment methodology put forth by Frost and Sullivan, and Immersion Medical. All 3 source materials were adapted to create an integrated methodology that can be readily implemented. This article presents details on each of these methods and how they can be integrated and presents a framework that integrates the previous methods. In addition to that, it describes the concept and the application of the developed framework. As a test of the applicability of the framework, a real case study has been used to demonstrate the application of the framework. This case study provides real data related to the correlation between the pediatric patient cardiopulmonary arrest (CPA) survival rates and a simulation-based mock codes at the University of Michigan tertiary care academic medical center. It is important to point out that the proposed framework offers the capability to consider a wide range of benefits and values, but on the other hand, there are several limitations that has been discussed and need to be taken in consideration.

Introduction

Simulation is used in areas that are difficult to see, expensive to build, dangerous to operate, and so forth. Simulations can also be expensive to develop and maintain, and their ability to meet the requirements set forth in creating them is often difficult to ascertain in advance. Nevertheless, decision makers who must make investment decisions need some means to know that an investment will be fruitful compared with various alternatives that might be available.

Most of the work to find the value of simulation to military training has been organized around cost avoidance. Little has been published in the open literature about a rigorous methodology that takes into consideration the different intangible factors during the life cycle of a simulator and the context of the organization.¹

Current Determination of Return on Investment (ROI) and Value

The literature review showed 3 components that must be considered in determining the value of simulation: quantitative benefits, qualitative benefits, and costs and contributors to value. Putting these 3 factors together in an integrated fashion gives decision makers a view of the value simulation offers.^1,2 The 3 components of qualitative and quantitative benefits and costs are discussed below.

Costs

The costs associated with the deployment of the simulation for health care include the costs of development, acquisition costs, the costs of maintenance, and the costs of operations. Labor to operate, teach, and maintain the simulator is included as appropriate.

Quantitative Benefits

The quantitative benefits generally are easier to recognize and measure. It embraces time savings, reduction in errors, faster time to competence, equipment breakage costs, reduction in alternative training costs, and procedures performed (quantity, time savings, etc.).

These factors were included in an ROI Excel-based tool developed by Frost and Sullivan for Immersion Medical.³ This tool includes major categories of benefits, costs, and other required information. The Frost and Sullivan tool represents the only comprehensive automated mean the authors have seen in the literature for computing ROI for computer-based simulation. Previously developed tools for computing ROI for other purposes have been based on cost avoidance with respect to using actual equipment for training or process improvement.

Qualitative Benefits

Qualitative benefits are the benefits that are hard to measure and transfer into monetary value. Examples of qualitative benefits include the improvement of patient safety, quality of care, employee satisfaction, the reputation of the organization, and others.

The work of Phillips and Phillips⁴ was used as a baseline to consider qualitative aspects related to value and ROI. Phillips recognizes that organizations need an algorithm that facilitates the steps to gather data to make informed investment decisions that can improve organizational training and performance. Phillips introduces an ROI methodology based on Dr Kirkpatrick’s 4 levels of evaluation for measuring performance in Training and Human Performance Technology (HPT) programs: Reaction (level 1), Learning (level 2), Behavior (level 3), and Results (level 4).

Phillips expands on Kirkpatrick’s 4 levels by creating the “Fifth Level” of ROI methodology for training and HPT programs. His modified approach evaluates the business value to the organization based on a particular investment or project to determine a framework for gathering program data to support and improve established training and performance programs. The use of simulation-based training fits well into Phillips’s paradigm.

Phillips’s ROI framework includes techniques used to evaluate the effectiveness of training programs. His approach estimates training impact by obtaining information directly from program participants and then having senior management make adjustments to the estimates. Adjustments are “essential because there are many factors that will influence performance data after training.⁵” Phillips reports that the effectiveness of his techniques lay in the assumption that participants who receive training are capable of estimating or determining how much improvement is related to the actual training program. By carefully crafting a questionnaire, information can be extracted to compute the qualitative aspects of ROI. If the questions are orthogonal, a small number of respondents are needed to achieve statistical power.

Methodology

This article proposes a generic framework that can be used to evaluate the ROI of different types and forms of simulation-based training in health care. The framework will provide a means to include the qualitative part of the benefits in ROI determination. Therefore, a methodology for measuring the nonfinancial benefits is needed and proposed. The framework is an integration of the value measurement methodology (VMM), Frost and Sullivan model, and Dr Jack Phillips approach in measuring the ROI in health care.

The VMM for Nonfinancial Benefits

In July 2001, the Social Security Administration and the General Services Administration along with a team of Booz Allen analysts and thought leaders affiliated with Harvard University’s Kennedy School of Government developed an effective methodology to measure the value of electronic services (i.e., quantitative and qualitative values) that would abide by current federal regulations and under the Office of Management and Budget guidance, which is applicable to the federal government. Later in 2002, the “How-To-Guide and VMM Highlights” document was released. The methodology allows for a decision framework (US Federal CIO Council, 2002) ⁶ to be personalized and adapted to the specific requirements of a project. Through the applications of VMM process, the value of alternatives to a program is articulated and the risk lowered for the considered investment.

As stated above, the VMM help strategists in the Government to consider both tangible and intangible values when making investment decisions and monitoring benefits. Value is derived from the benefits generated directly to users, society, and other stakeholders.

One of the important applications of the VMM methodology in the determination of ROI for simulation-based medical training is considering the value categories of the VMM to identify the tangible and intangible benefits and costs simulation-based medical training. The VMM value categories the authors determined for health care simulation are the following:

• Direct value—benefit of simulation in training users (e.g., PGY1 trainees, etc.)

• Social value—benefit to society (e.g., quality of care, fewer complications, etc.)

• Operational value—decrease in length of stay

• Strategic value—patient safety culture, sustainability

• Financial value—increase revenue, reduced costs

For example, Paige et al⁷ studied the impact of simulation-based interdisciplinary operative team training. All of the participants completed a questionnaire after the training and the majority of them reported that the training would change their practices in the operating room. In addition to that, the training promoted team communication skills, crisis-related teamwork, and improved recognizing operating room errors. All these are forms of operational values that can be captured using the VMM as the operational value is one of the value categories of the methodology. The study was concluded by this qualitative outputs. These outputs require transformation to monetary value and that is when Phillips ROI methodology is applied to quantify the monetary value of this operational improvement. Then it can be considered in the calculation of ROI.

The Framework of the Integrated Methodology

The framework uses the major categories of VMM value structure, which include direct, social, operational, strategic, and financial values, to identify tangible and intangible values and qualitative and quantitative benefits of the medical training simulator. This is accomplished by considering benefits that are preidentified by other studies such as the Frost & Sullivan³ ROI study for simulation based training (SBT) medical training and could be improved by several tools including questionnaires and/or interviews of experts. The next step is to isolate the effects of training using Phillips methodology to monetize the marginal improvement and benefit from using a specific training simulator. The isolation of effects method will vary based on the nature of the selected factors. Finally, the ROI is computed using the outcomes of the previous steps. Figure 1 shows the methodology framework.

Figure 1.

Methodology framework.

Note. ROI = return on investment.

Figure 2 summarizes the cost and value structure of the framework. It also depicts the different parameters that should be considered for ROI evaluation. Section (a) of the figure shows the different types of cost and has 2 categories: project and operational costs. Section (b) shows the VMM categories that help in identifying the key parameters and measures. These measures are classified into qualitative and quantitative. Section (C) shows the transformation of qualitative and quantitative measures into monetary value, tangible values, which is done using Dr Phillips methodology. It also shows that part of the qualitative measures cannot be transformed into monetary value with high credibility, and as a result, will be considered as an intangible value and will not be considered for the calculation part of the ROI but it will help decision makers to make informed decisions.

Figure 2.

Summary of cost and value structures and other factors contributing to ROI.

Note. ROI = return on investment.

Key measures and factors identification

The assessment of the ROI of a medical training simulator starts with the identification of the key impact measurement of the simulator that should be considered in the ROI analysis. For example, in the ROI analysis of a central venous catheter simulation-based training program for the medical intensive care unit at Northwestern Memorial Hospital, Pastrana et al¹ identified medical care costs, length of stay, and number of complications as key impact factors as important measurements for inclusion in analysis. The measures that should be influenced by the simulator depend on the objectives pursued in acquiring the simulator. The identification process is facilitated by considering the major categories of VMM value structure, which include direct, social, operational, strategic, and financial values.

There are several ways and strategies to identify the measures including questionnaires and/or interviews of experts and executives, especially those who are involved in decision making regarding the simulator acquisition. In addition, considering the preidentified measures for the common types of projects is an effective method to begin with. For example, measures identified by other studies like Frost and Sullivan ROI study for simulation medical training and Dr Phillips in his book Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs.

Converting qualitative and quantitative measures to tangible values

The outcome of identification process of key measurements is categorized into qualitative and quantitative measures. Quantitative data are easy to transfer to monetary value and considered as a tangible value. Qualitative data inclusion within the tangible values depends on the level of credibility of the converted data. Values lose credibility if the process used for the conversion is too subjective or inaccurate. The determination of lost value follows the guidelines described by Phillips. Qualitative data have 2 scenarios. The first is when the data can be converted to monetary values with high credibility. In this scenario, the data should be converted and included as a tangible value. The second scenario, if the data cannot be converted to monetary value with high credibility, then, it is considered as an intangible measure. Therefore, tangible measures include qualitative data and quantitative data when converted to monetary values with high credibility.

Steps to convert measures to monetary values

The following steps have to be applied for each measure to convert it to monetary value:

1. Identify the unit of measure: For quantitative measure, it is easier to identify the measuring unit, for example, the number of operations done in the operating room. It is more challenging to identify measuring units for qualitative measures. For example, 1 unit of improvement in patient satisfaction index. In general, for quantitative measures, there are commonly used measures, but for qualitative measures, there have been some measures that are getting more commonly used by the health care community.

2. Determine the value of the unit: Standard values are available for the majority of data types. If the standard value is not available, there are several techniques to identify the value of the unit. The selection of the technique is based on the type of data and the situation in which the data are gathered and intended use. Techniques include analyzing historical data, the use of internal and external experts, the use of external databases, and estimates of participants and managers.

3. Calculate the change in performance: This is the isolation of the impact of the simulator on the specific measure. It is described in the next section.

4. Determine the annual amount of change.

5. Calculate the annual value of the improvement: This can be done by multiplying the annual performance change by the value of the unit.

Isolating and evaluating the impact of the simulator

The next important step is to isolate the effects of the simulator. There are several approaches to isolate the impact of the training simulator. In general, there is no one single best approach to isolate or evaluate the impact of the simulator. Therefore, impact evaluation approach could vary based on the nature of the specific measure being considered. An analytical approach could be applied by using control groups, trend line analysis, or forecasting methods.

The other approach is the estimation. One way to use estimation to identify the human performance improvement developed as a result of the training is Dr Phillips strategy to estimate the improvement. The initial estimate should be done through a questionnaire that trainers and trainees take prior to and after the training to estimate their performance improvement due to the training and confidence levels in the estimations. Decision makers then review and adjust the estimations and identify the important factors and parameters that need to be considered in the calculations of the ROI.⁸

Simulator costs calculation

The other major factor to calculate the ROI is the cost and expenditures of the simulator. The costs can be classified into 2 major categories: project phase costs and operational phase costs. Project costs include initial analysis and assessment costs, the costs of development of the project, acquisition costs, and implementation costs. Operational phase costs include maintenance costs, support, overhead costs, labor, and materials supplies. Both categories should be included in the total costs for credible calculation.

ROI evaluation

The ROI evaluation has to be calculated according to the conditions set by the agency and/or health care business environment. The literature recommends the following formula:

$$\mathrm{ROI}(\%)=\frac{\mathrm{Net\; Benefit\; of\; Simulator}}{\mathrm{Total\; Cost\; of\; Simulator}}\times 100$$

Estimated costs and benefits are among the inputs of the simulator’s ROI calculation process. The presence of uncertainty in investment project always involves the presence of risk on investment, for example, negative ROI in certain scenario.⁹ Therefore, it is essential to consider the different possibilities of costs and benefits to evaluate risks and to work on the mitigation. Prioritizing the risks based on impact is an important step in risk management. In the case of ROI, identifying the key measures that have a major impact on the ROI is essential for risk management.

Sensitivity analysis studies how the uncertainty in a model outputs can be apportioned to diverse sources of uncertainty in inputs of the model.¹⁰ Therefore, performing sensitivity analysis is helpful and could be performed to identify the factors and measures that have the major influence on the ROI. Because the framework involves estimation, which always has a level of uncertainty, we always consider the confidence level in the calculation. The other way to consider uncertainties is using ranges instead of exact values.

Monte Carlo simulation is an effective tool that uses computers and probabilities to generate scenarios when variables are expressed in ranges. For each scenario, a randomly selected value for each unknown variable is used for the calculation. This process is repeated for many times to help decision makes to project the possible scenarios and make informed decisions.⁹

Implementation Case Study

The case study in this section will demonstrate the application of the framework. This case study provides real data from Andreatta et al¹¹ related to the correlation between the pediatric patient cardiopulmonary arrest (CPA) survival rates and a simulation-based mock codes at the University of Michigan tertiary care academic medical center. The objective of this study was to assess the effectiveness of the training on patient outcomes at residents’ confidence in performing resuscitation. This study was conducted over a period of 48 months, in which mock codes were called on an increasing rate and the clinicians responsible for pediatric resuscitation are required to respond just as they would on and the actual CPA event. Events where recorded and performance feedback was given by clinical faculty to the participating clinician’s residents, nurses, allied health, and attending physicians. The CPA survival rate for the hospital before and during the study was examined. The results of this study showed that the survival rate was increased by approximately 50% correlating with the increasing number of mock codes.

The application of the framework will start with developing the cost structure of the training, then the identification of the key parameters to be included in the ROI assessment using the different VMM categories. Next, the conversion of qualitative and quantitative data to tangible values will be executed. After that, the effects of the training will be isolated to evaluate the ROI of the training compared to its cost.

Results

The Cost Structure of the Simulation-Based Training

Table 1 includes the details and totals of the cost of the training. These include start-up costs for developing the scenarios, programming the mannequins, coordinating the delivery, and designing the assessment/evaluation strategies. There are also costs associated with the routine occurrences of the program, which include hourly rates for those who contributed to the start-up as well as the participants who were active during the mock code (average rate for the team is used). The costs for simulator, ancillary equipment, materials, and supplies are included, along with the maintenance agreements for the period of time the program took place. Facility charges are per hour for the code time only.

Table 1.

The Details and Totals of the Cost of the Training.

Item	Per unit cost	Units	Subtotal
Physician start-up	125	40	5000
Physician routine	125	240	30 000
Faculty educator start-up	80	80	6400
Faculty educator routine	80	240	19 200
Coordinator start-up	60	40	2400
Coordinator routine	60	240	14 400
Simulation technician start-up	60	80	4800
Simulation technician routine	60	240	14 400
Simulator purchase	50°000	1	50 000
Simulator maintenance	7500	4	30 000
Materials/supplies	1500	4	6000
Equipment	5000	1	5000
Facilities	250	240	60 000
Participants	900	160	144 000
Total program cost			391 600

Identifying the Key Parameters and Collecting the Data

The framework recommends considering the 5 major categories of value from the VMM: direct, social, operational, strategic, and financial value. Apart from the direct impact on social value, strategic value of patient safety, and financial value of the increased neonatal/pediatric CPA survival rate, this case study will consider the intangible factor of turnover rate of physicians, which has a strategic and financial impact on health care organizations.

Physician turnover is a very costly problem for health care organizations. According to Fibuch and Ahmed,¹² the negative impact of physician turnover should be a big concern for health care organizations as it has an impact on the profitability and the quality of care. In addition to the hiring and training costs, negative impacts such as productivity losses, noteworthy loss of organizational history, knowledge and expertise, disturbance of the morale of the remaining employees, and potential adverse publicity for the organization are expected. Therefore, the study highlighted the importance of incorporating employee retention strategy and considered the opportunities for advancement and learning new skills among the important factors of employee retention strategy. Considering the advancement opportunity provided by simulation-based training justify incorporating the cost saving of employee retention in ROI analysis.

Converting Qualitative and Quantitative Data Into Tangible Values

Computation of turnover costs and understanding its implications in health care are conceptually challenging because of 3 reasons. First, health care is simultaneously driven by market forces and controlled by regulation and as a result, accounting concepts cannot be applied directly to health care without major adjustment. For example, revenue does not equal reimbursement nor does cost equal charges. Second, the mathematical computation of costs is complex and varies with the type of employee and employer as turnover costs for physicians is far more complex than it is for maintenance staff. Third, due to the difficulty of attributing revenues and costs, the net effect of the turnover is almost noncalculable.¹³ Therefore, in this case study, we will consider the out-of-pocket costs that have been mentioned in Waldman et al.¹³

Isolating the Effects of the Training

Waldman et al¹³ used several databases at an academic medical center as a foundation for measuring costs of employee selection, hiring, and training, as well as qualitative and quantitative yardsticks used to measure employee productivity. The study by Waldman et al¹³ drew accounting records and data for specific organizational units within the academic medical center and categorized the costs of turnover by phase of recruiting process: hiring, training, working, and termination. Estimates have been made in few instanced. Even though the study has estimated the average turnover costs for about 6 categories of employee, in this case study, only the turnover cost of physicians and nurses will be considered.

• The average cost of replacing a physician including $36°743 hiring cost, $89°800 training cost, and $43°250 average loss of productivity to bring the total cost to $169°793 (about $170°000).

• The average cost of replacing a nurse including $ 1635 hiring cost, $ 15°825 training cost, and $10°026 average loss of productivity to bring the total cost to $27°486 (about $27°500).

  • Other source has mentioned the average nurse replacement cost as $42°000 and $64°000 in some cases.¹⁴

Note that the costs to train individuals involve mandatory courses, orientation classes, and reimbursed time when not generating charges.

Calculating the ROI

Despite the fact that we could not find actual data on the change of turnover rates for our particular case study, several studies have shown a connection between lower turnover and making investments in the training and development of human resources in health care.^13,14

The following assumptions will be used in ROI calculations: over the 48 months of the simulation-based training, at least 1 physician and 1 nurse every year has preferred not to leave the organization because of the training opportunity.

The annual cost saving of retention will be = $170°000 + $27°500 = $197°500.

The cost saving over the 48 months of implementation = 4 × $197°500 = $790°000.

$$\mathrm{ROI}(\%)=\frac{\mathrm{\$}790,000-391,600}{391,600}\times 100=\mathrm{\%}101.7.$$

Discussion

Clinical personnel, whether they are training or maintaining their abilities, need a safe way to practice decision-making and applied skills as individuals and in teams. Mechanisms for assuring opportunities to practice and rehearse using simulation-based methods have significant benefits for patient safety, not the least is because actual patients are not involved in the processes. Even though real world has stress and tension, simulation-based practice provides an environment that offers the opportunity to focus on building and acquiring skills where learning is facilitated with less stress and tension on the practitioner, especially in high-risk clinical contexts where performance providing care for real patients could negatively affect the learning process. Therefore, simulation-based training offers an essential solution for providing clinical personnel the opportunity to learn, practice, and maintain their abilities without stress or risk to real patients’ lives.¹⁵ Still, administrative decision makers that must determine if the investment in facilitating simulation-based environments is sufficiently beneficial with a convincing ROI compared with various alternatives that might be available.

The integrated framework presented herein enables the determination of ROI with the consideration for both tangible and intangible values and benefits resulting from simulation-based training, including a demonstration of the application of the framework to a specific case study. The application of the framework for this specific case study considered only a single aspect of the value categories of the VMM, with demonstrated %101 ROI for this one aspect alone; a convincing ROI to help the decision-making process. If other aspects were considered in the evaluation, the ROI would be further developed to accommodate both tangible and intangible outcomes and provide a more comprehensive analysis. A limitation of this study is that these data were not available for the analyses conducted for the case study; however, the framework provides a foundation for the types of data that would be beneficial for future studies evaluating the ROI of institutionally supported simulation-facilitated environments.

It is important to point out that the proposed framework offers the capability to consider a wide range of benefits and values that fall under any value category of the VMM including direct, social, operational, strategic, and financial values which enable more comprehensive evaluation for the ROI of the program. However, this would depend on the availability of the information that can help in converting these figures and information into monetary values to incorporate them into the ROI calculations. One example of an obvious added value is the legal obligations and consequences that are avoided as a result of improved clinical outcomes, such as survival rates for the considered case study. Another major value credited to extensive training, which could be achieved using simulation-based methods, is minimizing the medical errors. According to Makary and Daniel,¹⁶ medical error is the third cause of death in the United States, causing about 400°000 preventable deaths during the year 2013 alone. Understanding the real ROI and value of medical training, including highly effective simulation-facilitated methods, provides a foundation for fostering investment in best practices that have a positive impact on patient safety and quality of care. These are major objectives that impact the whole of health care systems globally.