Abstract
Background
Although simulation-based training is becoming widespread in surgical education and research supports its use, one major limitation is cost. Until now, little has been published on the costs of simulation in residency training. At the University of Toronto, a novel competency-based curriculum in orthopaedic surgery has been implemented for training selected residents, which makes extensive use of simulation. Despite the benefits of this intensive approach to simulation, there is a need to consider its financial implications and demands on faculty time.
Questions/purposes
This study presents a cost and faculty work-hours analysis of implementing simulation as a teaching and evaluation tool in the University of Toronto’s novel competency-based curriculum program compared with the historic costs of using simulation in the residency training program.
Methods
All invoices for simulation training were reviewed to determine the financial costs before and after implementation of the competency-based curriculum. Invoice items included costs for cadavers, artificial models, skills laboratory labor, associated materials, and standardized patients. Costs related to the surgical skills laboratory rental fees and orthopaedic implants were waived as a result of special arrangements with the skills laboratory and implant vendors. Although faculty time was not reimbursed, faculty hours dedicated to simulation were also evaluated. The academic year of 2008 to 2009 was chosen to represent an academic year that preceded the introduction of the competency-based curriculum. During this year, 12 residents used simulation for teaching. The academic year of 2010 to 2011 was chosen to represent an academic year when the competency-based curriculum training program was functioning parallel but separate from the regular stream of training. In this year, six residents used simulation for teaching and assessment. The academic year of 2012 to 2013 was chosen to represent an academic year when simulation was used equally among the competency-based curriculum and regular stream residents for teaching (60 residents) and among 14 competency-based curriculum residents and 21 regular stream residents for assessment.
Results
The total costs of using simulation to teach and assess all residents in the competency-based curriculum and regular stream programs (academic year 2012–2013) (CDN 155,750, USD 158,050) were approximately 15 times higher than the cost of using simulation to teach residents before the implementation of the competency-based curriculum (academic year 2008–2009) (CDN 10,090, USD 11,140). The number of hours spent teaching and assessing trainees increased from 96 to 317 hours during this period, representing a threefold increase.
Conclusions
Although the financial costs and time demands on faculty in running the simulation program in the new competency-based curriculum at the University of Toronto have been substantial, augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more efficient learning outcomes.
Clinical Relevance
The higher costs and demands on faculty time associated with implementing simulation for teaching and assessment must be considered when it is used to enhance surgical training.
Introduction
Surgical training programs are beginning to rely more heavily on simulation to supplement traditional teaching methods [11]. Simulation-based training allows trainees to practice skills in a low-pressure environment, without risk to patients, and allows trainees to make and learn from mistakes before they occur in surgery [13]. Previous research has shown that simulation can be highly effective for learning technical and nontechnical surgical skills [3, 4, 9, 14–17] and that the skills learned can transfer to clinical settings [5]. Furthermore, simulation-based training can shorten the learning curve, enhance later intraoperative learning [8, 14], and improve patient outcomes [19]. For example, one study has shown that general surgery residents who have undergone simulation training for laparoscopic inguinal hernia repair have had higher intraoperative performance scores, fewer postoperative and intraoperative complications, and less in-hospital stay compared with residents who used a conventional surgical training curriculum [19].
Despite clear advantages, the costs associated with simulation-based training are a major limitation [19]. Sufficient financial support is required for infrastructure, operating materials, support staff, and faculty time [18]. Although research suggests that the cost is offset by enhanced learning and decreased clinical complications [2], there is a clear need for better cost-effectiveness studies [19]. Zendejas and colleagues [18] conducted a systematic review and found that only 1.6% of studies provided any cost comparison when examining simulation-based training methods compared with other instructional methods in medical education.
There is a paucity of data regarding the costs of using simulation in orthopaedic surgery. For the past 5 years at the University of Toronto, a novel competency-based curriculum has been implemented as a pilot, functioning as a parallel stream of training for residents in the Division of Orthopaedic Surgery [1, 6]. In each of the first 3 years of the pilot program, three of 12 residents who matched to the residency program at the University of Toronto were selected into the competency-based curriculum stream. In its fourth year, five of 12 residents entered the competency-based curriculum. In its fifth year, a decision was made to convert all incoming residents to the competency-based curriculum stream. The competency-based curriculum makes extensive use of high- and low-fidelity simulation as a fundamental element of teaching and assessment. Standardized patients are used for teaching clinical skills, low-fidelity models are used for basic technical skills, and high-fidelity cadaveric models are used for higher order skills. All trainees start residency training by taking part in a 1-month long “Boot Camp,” in which the basic surgical skills and knowledge relating to surgical residency are learned. Trainees then move on to completing specialty-specific modules (clinical rotations) in which simulation is typically used at the beginning of the module for teaching the skills and knowledge required for that module. Simulation is also used at the end of several modules to assess whether trainees have met the necessary competencies. In addition to simulation, the competency-based curriculum training program uses focused teaching curricula, intensive summative and formative feedback sessions, and multiple assessment tools in the clinical environment to assess trainee competence. Augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more effective learning outcomes in the competency-based curriculum residents [14, 15]. In previous work we have shown that enrolling trainees in the month-long, laboratory-based training course (Boot Camp) at the onset of the competency-based curriculum greatly enhances the knowledge and skills of new trainees up to the level of senior residents [12, 15]. These improvements have been shown to be maintained long after completion of the course [14]. Furthermore, this type of simulation-based teaching may privilege nontechnical skill development and later clinical learning [10]. As such, the “Boot Camp” is an essential component of residency training in Toronto. Despite the benefits of the intensive use of simulation, there is a need to consider its financial and human resource implications.
The present study compares the costs of simulation programs in the competency-based curriculum with the historic costs of running simulation programs in the regular stream of resident training both before and after the competency-based curriculum was implemented. In addition to the financial costs, the number of faculty hours dedicated to simulation training is evaluated.
Materials and Methods
To determine the financial costs of simulation in the competency-based curriculum [1, 6], both before and after the implementation of the pilot to the University of Toronto training program, all invoices that were submitted to the division’s administrative office for simulation training were reviewed for each academic year. For reference, the academic year of 2008 to 2009 was chosen to represent an academic year that preceded the introduction of the competency-based curriculum. During this year, two skills sessions using cadavers were given to the residents. The costs involved during these sessions included those related to the purchase and return of cadaver specimens, skills laboratory labor, and the use of associated materials (such as instruments, instrument resterilization, drapes, gowns, etc). The academic year of 2010 to 2011 was chosen to represent an academic year in which the competency-based training program was functioning parallel but separate to the regular stream of training. After 2010 to 2011 and starting with the academic year of 2011 to 2012, all simulation teaching and assessments that were used in the competency-based curriculum program were universally applied to all residents in the regular stream of the residency training program. This measure was implemented because our research demonstrated that, in our context, the augmented simulation experiences our competency-based curriculum residents were receiving were translating to better learning outcomes compared with their regular stream peers [12, 15]. The academic year of 2012 to 2013 was chosen to represent an academic year when simulation was used equally among the competency-based curriculum and regular stream residents.
Simulation training and assessment occurred either in a dedicated surgical skills laboratory (the Surgical Skills Centre, Mount Sinai Hospital, Toronto, Canada) or at various university-affiliated hospital teaching sites. All costs related to simulation were itemized into the categories of cadavers, artificial models, labor (the cost of support staff in the surgical skills laboratory), standardized patients, and other associated materials. No adjustments were made in response to inflation in our analyses.
To further understand the costs of simulation in the training program, a breakdown of the costs of simulation in each competency-based curriculum training module was performed. (The competency-based curriculum residency training program curriculum is comprised of 21 specific modules, of which 17 are clinical modules. To complete this training program, all residents must meet the level of competence at the end of each module.) Simulation is used in 11 of the modules. These include: Introduction to Basic Surgical Skills (also known as the Boot Camp, in which only basic surgical skills and knowledge are taught); Hip and Basic Fractures; Emergency Orthopaedic Surgery; Basic Sports; Arthroplasty; Foot and Ankle; Hand and Upper Extremity; Oncology; Complex Trauma; Complex Arthroplasty; and Advanced Sports. Each module uses simulation for teaching residents the necessary skills and knowledge related to the module at the beginning of the module, for assessing the residents’ level of competence at the end of the module, or both.
To generate the cost of simulation for training per resident for each module, the total cost of the training sessions was divided by the number of residents that attended the sessions (Table 1). The number of residents that attended the sessions in the academic year when the competency-based curriculum training program was functioning in parallel but separate to the regular stream of training was three. The maximum number of residents who attended the training sessions in the academic year when the competency-based curriculum training program was universally applied to all residents in the regular stream of the residency training program was 12. To generate the cost of simulation for evaluation per resident for each module, the total cost of the evaluation sessions was divided by the number of competency-based curriculum residents taking that module that year (Table 1). For the academic year of 2010 to 2011, the number was three, whereas in the academic year of 2012 to 2013, the number was five. The only exception to this was in the Boot Camp module and Sports modules, in which all 12 residents were assessed with simulation, and the Complex Trauma and Oncology modules, in which three residents were assessed with simulation. No regular stream residents used simulation as an evaluation tool during any other modules in that academic year because simulation was only used for competency-based curriculum residents.
Table 1.
Determination of costs of simulation for training and assessment for each module per resident in the University of Toronto orthopaedic surgery residency training program before and after the implementation of the competency-based curriculum (CBC)
| Type of residency training program | Formula for cost of simulation for training per resident | Formula for cost of simulation for assessment per resident |
|---|---|---|
| Training program before CBC implementation |
Cost of simulation for training in entire academic year
12 regular stream residents |
No simulation used for assessment |
| CBC residents separate from regular stream residents |
Cost of simulation for training in each module
3 CBC residents |
Cost of simulation for assessment in each module
3 CBC residents |
| CBC residents plus regular stream residents |
Cost of simulation for training in each module
3 or 5 CBC residents* + 7 regular stream residents |
Cost of simulation for assessment in each module
3 or 5 CBC residents* + 7 regular stream residents for the Boot Camp, Basic and Advanced Sports modules |
* During the 2012–2013 academic year, certain modules had varying numbers of CBC residents on them; this number was three for complex trauma and oncology and five for all other modules.
In addition to competency-based curriculum-specific simulation, all Postgraduate Year 3 residents in the training program (competency-based curriculum and regular stream) have had their essential physician competencies, known as the CanMEDS roles [7] (analogous to the US’ Accreditation Council of Graduate Medical Education’s 6 competency framework), assessed in an annual “CanMEDS Objective Structured Clinical Examination (OSCE)” since 2011 to 2012. Here, residents are assessed in how they perform in standardized clinical scenarios with standardized patients. The costs of using standardized patients was determined by averaging the cost over the last 3 academic years.
The cost of using simulation in the competency-based curriculum does not just relate to direct financial outlay. Faculty are needed to teach and assess trainees during the simulation sessions.
To quantify faculty involvement, the number of hours spent teaching and assessing in each simulation session was totaled for the 3 same academic years used for the cost analysis: 1 year before the implementation of the competency-based curriculum (2008–2009), the year when the competency-based curriculum was functioning parallel but separate from the regular stream of training (2010–2011), and the year when simulation was used equally among the competency-based curriculum and regular stream residents for teaching and assessment (2012–2013). Faculty hours were not converted into a financial value, because we have no known accurate metric of measuring this in our training center.
In addition, other costs relating to the creation and maintenance of the competency-based curriculum were not entered into our analyses. We did not account for the costs of hiring a half-time administrative assistant, sending resident trainees to external review or specialty-specific weekend education courses that were related to the training module they were on, or paying for travel, lodging, and consultation costs for expert consultants in the initial design of the curriculum.
Results
The financial costs of using simulation for teaching and assessment substantially increased when the training program transitioned into having a competency-based curriculum run as a parallel stream of training compared with its traditional stream of training and when then training program transitioned into using simulation program-wide (Table 2). Although the total cost of using simulation to teach residents before the competency-based curriculum was implemented was CDN 10,090 (USD 11,140), the total cost of using simulation to teach and assess all residents in the competency-based curriculum and regular stream programs was approximately 15 times higher (CDN 155,750; USD 158,050).
Table 2.
Costs of simulation training and assessment in the University of Toronto orthopaedic surgery residency training program before and after the implementation of the competency-based curriculum (CBC)*
| Type of residency training program (with total number of hours dedicated to simulation) | Cadaver specimens† | Artificial models‡ | Skills laboratory labor§ | Associated materials|| | Standardized patients | Total cost | Number of residents | Total cost per resident |
|---|---|---|---|---|---|---|---|---|
| Training program before CBC implementation (2008–2009) | ||||||||
| Teaching sessions (16 hours) | 1140 (USD 1259) | 0 | 2240 (USD 2473) | 6710 (USD 7408) | 0 | 10,090 (USD 11,140) | 12 regular stream | 840 (USD 927) |
| CBC residents separate from regular stream residents (2010–2011) | ||||||||
| Teaching sessions (255 hours) | 24,000 (USD 24,228) | 3440 (USD 3473) | 26,900 (USD 27,156) | 16,810 (USD 16,970) | 0 | 71,160 (USD 71,836) | 6 CBC | 11,860 (USD 11,973) |
| Evaluation sessions (102 hours) | 10,500 (USD 10,600) | 7200 (USD 7268) | 2450 (USD 2473) | 9690 (USD 9782) | 0 | 29,840 (USD 30,124) | 6 CBC | 4970 (USD 5017) |
| Total | 34,500 (USD 34,828) | 10,640 (USD 10,741) | 29,360 (USD 29,639) | 26,500 (USD 26,752) | 0 | 101,000 (USD 101,960) | ||
| CBC residents plus regular stream residents (2012–2013) | ||||||||
| Teaching sessions (215 hours) | 43,500 (USD 44,142) | 6110 (USD 6200) | 21,070 (USD 21,381) | 26,180 (USD 26,566) | 0 | 96,860 (USD 98,288) | 60 (14 CBC + 36 regular stream) | 1610 (USD 1634) |
| Evaluation sessions (102 hours) | 24,000 (USD 24,354) | 7140 (USD 7245) | 8270 (USD 8392) | 16,480 (USD 16,723) | 3000 (USD 3044) | 58,890 (USD 59,758) | 35 (14 CBC + 21 regular stream) | 4210 (USD 4272) |
| Total | 67,500 (USD 68,495) | 13,250 (USD 13,445) | 29,330 (USD 29,763) | 42,660 (USD 43,289) | 3000 (USD 3044) | 155,750 (USD 158, 050) | ||
* All costs are in Canadian dollars; †the cost of a cadaver specimen varied on the anatomic region required; for an isolated upper or lower extremity the cost was CDN 1500; for a whole cadaver the cost was CDN 26000; transport fees were CDN 355 per trip; ‡artificial bone models were purchased from Sawbones®, Pacific Research Laboratories (Vashon, WA, USA); soft tissue models were varied and involved pig skin from a pig limb (cost, CDN 30) or artificial models purchased from the Surgical Skills Centre, Mount Sinai Hospital (cost, CDN 24); §skills laboratory staff labor was CDN 45 per hour; part-time student labor was CDN 19.50 per hour; ||the cost of associated materials was variable and depended on what materials were used. For example, the cost of instrument resterilization was CDN 1,844.39, the cost of saw blades was CDN 54 per blade, the cost of renting a C-arm unit for a half-day session was CDN 1100; ¶standardized patients were paid CDN 30 per hour.
Analysis of the cost difference between the academic years when the competency-based curriculum program was functioning parallel but separate to the regular stream of training and when the competency-based curriculum training program was universally applied to all residents in the regular stream of the residency training program indicated that the increased cost was the result of the cost of the Boot Camp. When the Boot Camp was given to the three competency-based curriculum residents only, the total cost was CDN 55,060 (USD 60,790). When the Boot Camp was expanded to all 12 residents in the training program, the cost was CDN 106,720 (USD 108,290) (Table 3). The costs relating to using simulation in teaching stayed the same both before and after simulation were offered to the regular stream residents (because the total cost of cadavers, artificial models, labor, and associated materials remained the same).
Table 3.
The costs of simulation in the competency-based curriculum modules as well as the CanMEDS OSCE*
| Module | Cadavers | Artificial models | Skills laboratory labor | Associated materials | SPs | Total cost | Cost per resident | Number of residents | Total hours | Number of faculty |
|---|---|---|---|---|---|---|---|---|---|---|
| Boot Camp | ||||||||||
| Teaching sessions | 30,000 (USD 30,440) | 3900 (USD 3960) | 16,090 (USD 16,330) | 15,370 (USD 15,600) | 0 | 65,350 (USD 66,310) | 5450 (5530 USD) | 12 | 155 | 1* |
| Evaluation sessions | 18,000 (USD 18,270) | 2250 (USD 2280) | 7420 (USD 7530) | 13,710 (USD 13, 910) | 0 | 41,370 (USD 41,980) | 3450 (3500 USD) | 12 | 32 | 5 |
| All sessions | 48,000 (USD 48,710) | 6140 (USD 6230) | 23,500 (USD 23,850) | 29,080 (USD 29,510) | 0 | 106,720 (USD 108,290) | ||||
| Hip and basic fractures | ||||||||||
| Teaching sessions | 1500 (USD 1520) | 0 | 1210 (USD 1230) | 910 (USD 920) | 0 | 3620 (USD 3670) | 300 (USD 300) | 12 | 8 | 1 |
| Evaluation sessions | 1500 (USD 1520) | 760 (USD 770) | 110 (USD 112) | 1630 (USD 1650) | 0 | 4010 (USD 4590) | 800 (USD 810) | 5 | 12 | 1 |
| All sessions | 3000 (USD 3040) | 760 (USD 770) | 1320 (USD 1340) | 2540 (USD 2580) | 0 | 7630 (USD 7740) | ||||
| Emergency orthopaedics | ||||||||||
| Teaching sessions | 3000 (USD 3040) | 0 | 260 (USD 260) | 1260 (USD 1280) | 0 | 4520 (USD 4590) | 900 (USD 910) | 5 | 8 | 1 |
| Evaluation sessions | 3000 (USD 3040) | 0 | 310 (USD 310) | 220 (USD 220) | 0 | 3520 (USD 3570) | 700 (USD 710) | 5 | 6 | 1 |
| All sessions | 6000 (USD 6090) | 0 | 570 (USD 580) | 1480 (USD 1500) | 0 | 8050 (USD 8170) | ||||
| Basic sports | ||||||||||
| Teaching sessions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| Evaluation sessions | 0 | 960 (USD 970) | 0 | 0 | 0 | 960 (USD 970) | 80 (USD 80) | 12 | 14 | 2 |
| All sessions | 0 | 960 (USD 970) | 0 | 0 | 0 | 960 (USD 970) | ||||
| Arthroplasty | ||||||||||
| Teaching sessions | 1500 (USD 1520) | 0 | 950 (USD 960) | 1170 (USD 1190) | 0 | 3620 (USD 3670) | 300 (USD 300) | 12 | 8 | 2 |
| Evaluation sessions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| All sessions | 1500 (USD 1520) | 0 | 950 (USD 960) | 1170 (USD 1190) | 0 | 3620 (USD 3670) | ||||
| Foot and ankle | ||||||||||
| Teaching sessions | 1500 (USD 1520) | 0 | 840 (USD 850) | 4030 (USD 4090) | 0 | 6380 (USD 6470) | 530 (USD 540) | 12 | 8 | 2 |
| Evaluation sessions | 0 | 0 | 0 | 0 | 0 | 0 | ||||
| All sessions | 1500 (USD 1520) | 0 | 840 (USD 850) | 4030 (USD 4090) | 0 | 6380 (USD 6470) | ||||
| Hand and upper extremity | ||||||||||
| Teaching sessions | 4500 (USD 4570) | 377 (USD 380) | 1970 (USD 2000) | 0 | 6850 (USD 6950) | 570 (USD 580) | 12 | 8 | 1 | |
| Evaluation sessions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| All sessions | 4500 (USD 4570) | 380 (USD 390) | 1970 (USD 2000) | 0 | 6850 (USD 6950) | |||||
| Oncology | ||||||||||
| Teaching sessions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| Evaluation sessions | 1500 (USD 1520) | 0 | 210 (USD 210) | 580 (USD 590) | 0 | 2300 (USD 2330) | 770 (USD 780) | 3 | 6 | 1 |
| All sessions | 1500 (USD 1520) | 210 (USD 210) | 580 (USD 590) | 0 | 2300 (USD 2330) | |||||
| Complex trauma | ||||||||||
| Teaching sessions | 0 | 2210 (USD 2240) | 210 (USD 210) | 340 (USD 350) | 0 | 2770 (USD 2810) | 920 (USD 930) | 3 | 12 | 1 |
| Evaluation sessions | 0 | 2210 (USD 2240) | 210 (USD 210) | 340 (USD 350) | 0 | 2770 (USD 2810) | 920 (USD 930) | 3 | 14 | 1 |
| All sessions | 0 | 4430 (USD 4500) | 430 (USD 440) | 680 (USD 690) | 0 | 5540 (USD 5620) | ||||
| Complex arthroplasty | ||||||||||
| Teaching sessions | 1500 (USD 1520) | 0 | 1120 (USD 1140) | 1130 (USD 1150) | 0 | 3760 (USD 3820) | 310 (USD 310) | 12 | 8 | 2 |
| Evaluation sessions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| All sessions | 1500 (USD 1520) | 0 | 1120 (USD 1140) | 1130 (USD 1150) | 0 | 3760 (USD 3820) | ||||
| Advanced sports | ||||||||||
| Teaching sessions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| Evaluation sessions | 0 | 960 (USD 970) | 0 | 0 | 0 | 960 (USD 970) | 80 (USD 80) | 12 | 14 | 2 |
| All sessions | 0 | 960 (USD 970) | 0 | 0 | 0 | 960 (USD 970) | ||||
| CanMEDS OSCE | ||||||||||
| Evaluation session | 0 | 0 | 0 | 0 | 3,000 (USD 3,040) | 3000 (USD 3040) | 250 (USD 250) | 12 | 4 | 5 |
| Total costs | 67,500 (USD 68,500) | 13,250 (USD 13,450) | 29,330 (USD 29,760) | 42,660 (USD 43,290) | 3,000 (USD 3,040) | 155,750 (USD 158,050) | 16,330 (USD 16,570) | |||
All costs are in Canadian dollars; * in addition to one faculty member, two clinical fellows and two residents were used as full-time teaching assistants in the Boot Camp; cost per resident during teaching sessions represents costs of simulation divided by the number of residents that attended the training sessions. Cost per resident during evaluation sessions represents costs of simulation divided by the number of competency-based curriculum residents that used simulation for their evaluation (regular stream residents did not use simulation for their evaluation); OSCE = Objective Structured Clinical Examination; SPs = standardized patient costs.
The cost breakdown for each training module and the CanMEDS OSCE once all simulation training sessions were shared with the regular stream residents indicated that the vast majority of the costs for simulation in the training program (68.5%) came from the month-long, simulation-based Boot Camp course, whose total cost was CDN 106,720 (USD 108,290) (Table 3). The remainder of the costs came from 10 clinical modules (CDN 47,620; USD 48,320 in total) and the CanMEDS OSCE (CDN 3000; USD 3040).
Once simulation was applied to all trainees in the program, the largest segment of cost was related to the use of cadavers (CDN 67,500/155,750; USD 68,500/158,050; 43%). The next largest segment of cost was related to the use of associated materials (such as gowns and gloves, suture materials, sawblades, instrument processing, etc) in the skills laboratory (CDN 42,660/155,750; USD 43,290/158,050; 27.4%). After this, costs related to the labor of surgical skills laboratory staff were 18.8% (CDN 29,330/155,750; USD 29,760/158,050); costs related to the use of artificial models (such as artificial bone and skin models) were 8.5% (CDN 13,250/155,750; USD 13,450/158,050). It is important to note that the costs of using the surgical skills laboratory, which otherwise could have been substantial, were waived as a result of a cooperative initiative between the laboratory and the Division of Orthopaedic Surgery.
The total number of hours spent teaching and assessing using simulation in the training program before the competency-based curriculum was implemented (2008–2009) was 96 hours (Table 4). The total number of hours spent teaching and assessing using simulation after the introduction of the CBC was 317 hours (Table 4). This represents an overall increase of approximately 3.3 times.
Table 4.
The total number of faculty hours spent teaching and/or assessing resident trainees with simulation in the University of Toronto orthopaedic surgery residency training program before and after the implementation of the competency-based curriculum (CBC)
| Category | Training program before CBC implementation | CBC residents separate from regular stream residents | CBC residents plus regular stream residents |
|---|---|---|---|
| Total number of teaching hours | 96 | 255 | 215 |
| Total number of assessment hours | 0 | 102 | 102 |
| Total number of hours | 96 | 357 | 317 |
Please note that these numbers do not include teaching and assessment performed in the training program without simulation such as that what is routinely done at teaching conferences/rounds, on the ward, clinic, operating room, or emergency department.
Discussion
As surgical education programs begin to rely more heavily on simulation-based training, it is important to consider the financial implications. The present study compared the costs of simulation programs in the new competency-based curriculum at the University of Toronto with the historic costs of running simulation programs in the regular stream of residency training both before and after the competency-based curriculum was implemented. In addition to the financial costs, the number of faculty hours dedicated to sessions where simulation is used for training and assessment was evaluated.
Although the data presented are specific in terms of assessing cost and faculty hours, the main limitation of this study relates to the fact that the data reflect the costs associated with using simulation in one residency training center. However, despite the fact that these costs are specific to Toronto, any institution that seeks to implement similar simulation-based training programs will need to consider the same cost categories and potential financial relationships with the university, industry, and faculty partners in determining a budget. Although there may be variations in specific costs at individual training centers, we believe that the findings from this study are broadly generalizable.
The University of Toronto received government funding for this unique initiative in redesigning residency education. The financial arrangements that the University of Toronto has are likely specific to the Division itself and may not reflect all of the potential costs of using simulation in a competency-based curriculum training program in other centers. For example, one important cost not identified in the data presented is the true cost of renting the surgical skills laboratory for each hour of use. The Mount Sinai Surgical Skills Centre, which is the main skills laboratory for simulation in the training program, has waived the hourly rental rate (CDN 197.50, USD 200) because the Centre was developed as a cooperative initiative between the University’s Division of Orthopaedic Surgery and Department of Surgery. As a result of this relationship, the fixed cost of facility rental was waived. As such, all other costs for simulation were variable, the total value of which depended on the number of residents that were to use simulation at a specific time (including the numbers of cadavers, instrument sets needing preparation and resterilization, number of skills center staff assisting the trainees).
One important cost not identified is that of orthopaedic implants and instrument systems, which were donated through educational grants to the Division. If the costs of rent, plates, screws, nails, arthroplasty components, and other such elements were included, the budget would be substantially more than what was presented. Another indirect cost not identified is the financial cost of faculty time, which was delivered by geographic full-time faculty, fellows, and retired faculty. The number of hours faculty devote to teaching and assessment is significant. Furthermore, the amount of time listed does not include travel time to and from respective offices as well as the cost of transportation, parking, and opportunity cost of the faculty not being clinically active. It is important to note that this teaching and assessment is performed in addition to the teaching and assessment that is done at the hospital-specific level, where trainees are taught by their attending surgeons how to take care of patients in the clinic, operating room, inpatient ward, and emergency department. The last important cost not identified relates to the effect of having the residents away from their clinical duties. In all instances, coresidents, clinical fellows, and staff surgeons had to cover for the residents when they were involved in simulation training and assessment. Although we were unable to determine a dollar value for this, the effect on increasing the patient care workload on those physicians covering for the trainees was substantial.
Despite the clear advantages of using simulation derived from reports emanating from the medical education literature, the associated costs are a major limitation to implementation in mainstream practice [19]. As this study has identified, sufficient financial support is required for infrastructure, operating materials, support staff, and faculty time. Although research suggests that the costs are offset by enhanced learning and decreased clinical complications [2], there is a need for better return-on-investment studies [18, 19]. This is especially the case as it relates to orthopaedic surgery, where to our knowledge no publications exist on the topic. Such studies need to take into account not only the costs involved (including a categorical listing of all resources involved, the monetary value of each resource, and a breakdown of cost per learner for each resource) [17], but also the potential cost savings involved (which would include a cost breakdown of improvements in resource use, operating room efficiency, reduction in patient care errors and events, and improvements in patient safety) [18]. Performing this type of research is complex and will call for close collaboration among surgeons, educators, health economists, and statisticians. Without this evidence, surgical educators might not have the leverage necessary to convince funding bodies to invest in this type of training [18]. Funding bodies may vary from region to region, but may include hospitals, universities, accreditation bodies, medical insurance companies, private donors, and industry.
The current study identifies that there is a substantial cost of implementing simulation for teaching and assessment in a novel, competency-based training curriculum. The extensive use of high- and low-fidelity simulation-based modalities to train and assess residents over a course of 12 modules and one OSCE that assesses the CanMEDS roles has led to financial costs and hours of faculty time that did not exist in the University of Toronto training program before the competency-based curriculum was instituted. The cost of simulation increased by an increment of approximately 15.4 times. The amount of faculty hours dedicated increased by approximately three times. A more categorical analysis of cost has shown that the largest proportion is related to the use of cadavers followed by associated materials, labor, artificial models, and standardized patients. Despite these costs, we believe that they are worth the augmented learner and trainer satisfaction and enhanced learning outcomes [14, 15]. In fact, implementing a month-long, laboratory-based surgical skills training program (“Boot Camp”) at the onset of the competency-based curriculum program has been shown to greatly advance the knowledge and skills of new trainees [12, 15]. These improvements have also been shown to be maintained well into the training program and long after completion of the course [14]. In addition, all trainees who have used simulation in the training program have successfully passed their final licensing examinations.
It is also interesting to note that this type of training using simulation might not only benefit technical skill development and knowledge acquisition, but it may also privilege the development of ancillary abilities such as communication and collaboration skills [10]. By doing so, the use of simulation-based training, especially during the early stages of residency, may promote a more positive learning environment and greatly enhance the efficacy of later clinical learning [10].
Should the intensive use of simulation be used to train and assess residents in all surgical training programs? Although the financial costs and time demands on faculty in running the simulation program in the competency-based curriculum at the University of Toronto have been substantial, augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more efficient learning outcomes [12, 14, 15]. Although our division believes that the training is cost-effective, other training centers will have to decide individually as to whether the cost is worth the benefit. Future studies looking at the return on investment of simulation in teaching and evaluating trainees are needed, especially because simulation will be playing a larger role in medical education. This research, which should look not only at the financial and human resources costs involved, but also at the cost savings involved, will be complex. Without this information, surgical educators might not have the leverage necessary to convince funding bodies (such as hospitals, universities, accreditation bodies, medical insurance companies, private donors, and industry) to invest in this type of training [18].
Acknowledgments
We thank Dr Dan Stojimirovic, Polina Mironova, Lisa Satterthwaite, Shunne Leung, and the entire Surgical Skills Centre team at the Mount Sinai Surgical Skills Laboratory for their assistance in obtaining all financial data for the paper.
Footnotes
The Ontario Ministry of Health and Long-term Care provided funding to the Department of Surgery, University of Toronto, for the Competency-based Curriculum (CBC) initiative (RR).
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
This work was performed at the University of Toronto, Toronto, Ontario, Canada.
References
- 1.Alman BA, Ferguson P, Kraemer W, Nousiainen MT, Reznick RK. Competency-based education: a new model for teaching orthopaedics. Instr Course Lect. 2012;62:565–569. [PubMed] [Google Scholar]
- 2.Cohen ER, Feinglass J, Barsuk JH, Barnard C, OʼDonnell A, McGaghie WC, Wayne DB. Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simul Healthc. 2010;5:98–102. doi: 10.1097/SIH.0b013e3181bc8304. [DOI] [PubMed] [Google Scholar]
- 3.Cook DA. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306:978. doi: 10.1001/jama.2011.1234. [DOI] [PubMed] [Google Scholar]
- 4.Cook DA, Brydges R, Hamstra SJ, Zendejas B, Szostek JH, Wang AT, Erwin PJ, Hatala R. Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc. 2012;7:308–320. doi: 10.1097/SIH.0b013e3182614f95. [DOI] [PubMed] [Google Scholar]
- 5.Dawe SR, Windsor JA, Broeders JAJL, Cregan PC, Hewett PJ, Maddern GJ. A systematic review of surgical skills transfer after simulation-based training: laparoscopic cholecystectomy and endoscopy. Ann Surg. 2014;259:236–248. doi: 10.1097/SLA.0000000000000245. [DOI] [PubMed] [Google Scholar]
- 6.Ferguson PC, Kraemer W, Nousiainen M, Safir O, Sonnadara R, Alman B, Reznick R. Three-year experience with an innovative, modular competency-based curriculum for orthopaedic training. J Bone Joint Surg Am. 2013;95:e166. doi: 10.2106/JBJS.M.00314. [DOI] [PubMed] [Google Scholar]
- 7.Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29:642–647. doi: 10.1080/01421590701746983. [DOI] [PubMed] [Google Scholar]
- 8.Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241:364. doi: 10.1097/01.sla.0000151982.85062.80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Gurusamy KS, Aggarwal R, Palanivelu L, Davidson BR. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2009;21:CD006575. doi: 10.1002/14651858.CD006575.pub2. [DOI] [PubMed] [Google Scholar]
- 10.Hoogenes J, Mironova P, Safir O, McQueen SA, Abdelbary H, Drexler M, Nousiainen M, Ferguson P, Kraemer W, Alman B, Reznick RK, Sonnadara RR. Student-led learning: a new teaching paradigm for surgical skills. Am J Surg. 2015;209:107–114. doi: 10.1016/j.amjsurg.2014.08.037. [DOI] [PubMed] [Google Scholar]
- 11.Reznick RK, MacRae H. Teaching surgical skills—changes in the wind. N Engl J Med. 2006;355:2664–2669. doi: 10.1056/NEJMra054785. [DOI] [PubMed] [Google Scholar]
- 12.Sonnadara R, Garbedian S, Safir O, Mui C, Nousiainen M, Ferguson P, Alman B, Kraemer W, Reznick R. Orthopaedic Boot Camp III: examining the efficacy of student-regulated learning during an intensive laboratory-based surgical skills course. Surgery. 2013;154:29–33. doi: 10.1016/j.surg.2013.05.003. [DOI] [PubMed] [Google Scholar]
- 13.Sonnadara R, McQueen S, Mironova P, Safir O, Nousiainen M, Ferguson P, Alman B, Kraemer W, Reznick R. Reflections on current methods for evaluating skills during joint replacement surgery: a scoping review. Bone Joint J. 2013;95:1445–1449. doi: 10.1302/0301-620X.95B11.30732. [DOI] [PubMed] [Google Scholar]
- 14.Sonnadara RR, Garbedian S, Safir O, Nousiainen MT, Alman B, Ferguson P, Kraemer W, Reznick R. Orthopaedic Boot Camp II: examining the retention rates of an intensive surgical skills course. Surgery. 2012;151:803–807. doi: 10.1016/j.surg.2012.03.017. [DOI] [PubMed] [Google Scholar]
- 15.Sonnadara RR, Van Vliet A, Safir O, Alman B, Ferguson P, Kraemer W, Reznick R. Orthopedic boot camp: examining the effectiveness of an intensive surgical skills course. Surgery. 2011;149:745–749. doi: 10.1016/j.surg.2010.11.011. [DOI] [PubMed] [Google Scholar]
- 16.Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ. A systematic review of skills transfer after surgical simulation training. Ann Surg. 2008;248:166–179. doi: 10.1097/SLA.0b013e318176bf24. [DOI] [PubMed] [Google Scholar]
- 17.Sutherland LM, Middleton PF, Anthony A, Hamdorf J, Cregan P, Scott D, Maddern GJ. Surgical simulation. Ann Surg. 2006;243:291–300. doi: 10.1097/01.sla.0000200839.93965.26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA. Cost: the missing outcome in simulation-based medical education research: a systematic review. Surgery. 2013;153:160–176. doi: 10.1016/j.surg.2012.06.025. [DOI] [PubMed] [Google Scholar]
- 19.Zevin B, Aggarwal R, Grantcharov TP. Surgical Simulation in 2013: why is it still not the standard in surgical training? J Am Coll Surg. 2014;218:294–301. doi: 10.1016/j.jamcollsurg.2013.09.016. [DOI] [PubMed] [Google Scholar]