Surgical Simulation

Abstract

Objective:

To evaluate the effectiveness of surgical simulation compared with other methods of surgical training.

Summary Background Data:

Surgical simulation (with or without computers) is attractive because it avoids the use of patients for skills practice and provides relevant technical training for trainees before they operate on humans.

Methods:

Studies were identified through searches of MEDLINE, EMBASE, the Cochrane Library, and other databases until April 2005. Included studies must have been randomized controlled trials (RCTs) assessing any training technique using at least some elements of surgical simulation, which reported measures of surgical task performance.

Results:

Thirty RCTs with 760 participants were able to be included, although the quality of the RCTs was often poor. Computer simulation generally showed better results than no training at all (and than physical trainer/model training in one RCT), but was not convincingly superior to standard training (such as surgical drills) or video simulation (particularly when assessed by operative performance). Video simulation did not show consistently better results than groups with no training at all, and there were not enough data to determine if video simulation was better than standard training or the use of models. Model simulation may have been better than standard training, and cadaver training may have been better than model training.

Conclusions:

While there may be compelling reasons to reduce reliance on patients, cadavers, and animals for surgical training, none of the methods of simulated training has yet been shown to be better than other forms of surgical training.

In a systematic review of the 30 relevant randomized controlled trials, simulated training (including computer simulation) was not shown to be superior to other forms of surgical training.

Simulation is particularly attractive in the field of surgery because it avoids the use of patients for skills practice and ensures that trainees have had some practice before treating humans.¹ Surgical simulation may or may not involve the use of computers.

One of the barriers to development of virtual reality surgical simulation has been the large amount of computing capacity that has been required to remove delays in signal processing, but this is being addressed by systems that break down the tasks by concentrating on chains of behavior.² Designers of surgical simulators attempt to balance visual fidelity, real-time response and computing power, and cost.³

What Currently Happens in Surgical Training?

Surgical training consists of developing cognitive, clinical, and technical skills, the latter being traditionally acquired through mentoring.⁴ Fewer mentoring opportunities⁵ have led to the use of models, cadavers, and animals to replicate surgical situations and, more recently, to development of surgical skills centers or laboratories. However, the effectiveness of skills laboratories in teaching basic surgical skills (eg, instrument handling, knot tying, and suturing) is not yet proven.^6,7 Reznick⁸ maintains that the teaching and testing of technical skills have been the least systematic or standardized component of surgical education.

Is Simulation an Effective Method of Training?

This review attempts to gauge the instructional effectiveness of surgical simulation⁹ (that repeated use improves performance), as well as construct validity (that the simulator measures the skill it is designed to measure). Validity studies of surgical simulation (mostly computer simulation) have shown mixed results for construct validity while other important aspects of validity (such as predictive validity) and reliability are frequently not tested.^3,9 The ultimate validation is for simulation training to show a positive influence on patient outcomes (adapted from Berg et al⁹).

This review focuses on the use of surgical simulation for training, but simulation is also being used to assess surgeons. For example, the U.K. General Medical Council is using it to assess poorly performing surgeons referred to the Council.¹⁰

Costs of Surgical Training

While the costs of simulation systems can be high, ranging from about U.S. $5000 for most laparoscopic simulators to up to U.S. $200,000 for highly sophisticated anesthesia simulators,¹ traditional Halstedian training is not without cost either. Bridges and Diamond¹¹ calculated that the cost of training a surgical resident in the operating room for 4 years was nearly U.S. $50,000 (measured by the additional time that the resident took to complete procedures). Other providers of surgical training are making substantial investments in surgical skills centers.

MATERIALS AND METHODS

This review is a summary and an update of an assessment carried out for ASERNIP-S (the Australian Safety and Efficacy Register of New Interventional Procedures): www.surgeons.org/asernip-s/.

Inclusion Criteria

Randomized controlled trials (RCTs) of any training technique using at least some elements of surgical simulation compared with any other methods of surgical training, or no surgical training were included for review. Participants in the RCTs could have been surgeons, surgical trainees (residents), medical students, or other groups. Included studies must have contained information on either measures of surgical task performance (whether objective or subjective) or measures of satisfaction with training techniques or both.

Search Strategy

Studies were identified by searching MEDLINE, EMBASE, PreMEDLINE, Current Contents, The Cochrane Library (Issue 2, 2005), scholar.google.com, metaRegister of Controlled Trials, National Research Register (U.K.), NHS Health Technology Assessment (U.K.), and the NHS Centre for Reviews and Dissemination databases (last searched April 2005). PsycINFO, CINAHL, and Science Citation Index were searched on March 25, 2003. Search terms used were surgical simulation or surg* AND (simulat* OR virtual realit*). Additional articles were identified through the reference sections of the studies retrieved.

Articles were retrieved when they were judged to possibly meet the inclusion criteria. Two reviewers then independently applied the inclusion criteria to these retrieved articles. Any differences were resolved by discussion.

Data Extraction and Analysis

The ASERNIP-S reviewer extracted data on to data extraction sheets designed for this review and a second reviewer checked the data extraction.

It was not considered appropriate to pool results across studies, because outcomes were not comparable. Relative risks (RR), for dichotomous outcome measures or weighted mean differences (WMD), for continuous outcome measures with 95% confidence intervals (CI) were calculated for some outcomes in individual RCTs where it was thought that this would aid in the interpretation of results. Calculations were made using RevMan 4.2 (Cochrane Collaboration, 2003). Some of the calculations of continuous measures showed skew (the standard deviation is twice more than the mean), indicating that some of the results were not normally distributed. This usually indicates that there is a strong “floor” or “ceiling” effect for performance of a particular task.¹² Study quality was assessed on a number of parameters such as the quality of methodologic reporting, methods of randomization and allocation concealment, blinding of outcomes assessors, attempts made to minimize bias, sample sizes, and ability to measure “true effect.”

Details of RCTs

A total of 30 relevant RCTs with 760 participants overall were included (Tables 1 and 2).2,3,12–39

TABLE 1. Summary of Included Studies

TABLE 2. Included Studies Grouped According to Training Category

The length of time devoted to training varied greatly between the studies, with the shortest reported time being 10 minutes,³⁷ and the longest being 10 hours.²² In the study by Harold et al,²¹ both training groups were given 1 week to practice using the video box trainer, with the trained group receiving an additional 60-minute course on laparoscopic suturing and in Lehmann et al²⁸ each group carried out a defined training program over 4 days and then switched to the other group on the fifth day. Fifteen studies did not report the length of training but rather the number of sessions or the number of tasks that were practiced.2,3,14,16,17,20,23,25–27,31,34,35 The previous experience of participants varied greatly between the studies, ranging from high school students¹⁴ to third, fourth, and fifth year urology residents³² to expert surgeons.^16,28

Quality of Studies

Only 3 studies^16,18,25 are likely to have demonstrated adequate allocation concealment, with use of sealed or closed envelopes. It was not possible to blind participants for any of the interventions in the included studies. Outcome assessors were blinded in 15 RCTs.13,15,18–22,29–33,36,37,39 One of the 2 outcome assessors was blinded in Wilhelm et al.³⁸ In 14 RCTs, it was not stated whether the outcome assessors were blinded.2,3,12,14,16,17,23–28,30,31 Because the study durations were generally short, most studies had no losses to follow-up.

RESULTS

The 30 included studies have been categorized into 4 types of simulation (computer, video, model, and cadaver) and compared with no training and standard training, as well as with each other.

Computer Simulation Versus No Training (9 Studies) (Table 3)

TABLE 3. Computer Simulation Versus No Training

As may be expected, those trained on computer simulators performed better than those who received no training at all. However, the improvement was not universal.

Computer Simulation Versus Standard Training (5 Studies) (Table 4)

TABLE 4. Computer Simulation Versus Standard Training

When computer simulator trained students were compared with students who received standard training, the superiority of computer simulation was less pronounced than for “no training” comparisons. The computer simulation versus “standard” training comparisons varied, potentially confounded by the different components of “standard” training, as well as by the different intensities of time allowed on the simulator in the computer simulation groups.

Computer Simulation Versus Video Simulation (7 Studies) (Table 5)

TABLE 5. Computer Simulation Versus Video Simulation

Computer simulation showed mixed results, superior in some studies, but not others and was inferior to video simulation in one study. This may have depended on types of tasks, with computer simulation producing better results for tasks such as incisions, but not for knot tying times. However, there were too few studies to determine this.

Computer Simulation Versus Physical Trainer or Model (1 Study)³⁹ (Table 6)

TABLE 6. Computer Simulation Versus Physical Trainer

One study showed computer simulation training to be superior to training on a physical trainer.

Two or More Types of Computer Simulation; MIST-VR (2 Studies) (Table 7)

TABLE 7. Two or More Types of Computer Simulation; MIST-VR

One RCT¹⁴ showed that more demanding training may lead to better performance of surgical tasks on MIST-VR. Another RCT¹² failed to show clear differences between massed and distributed practice on MIST-VR.

Video Simulation Versus No Training (6 Studies) (Table 8)

TABLE 8. Video Simulation Versus No Training

Video simulation groups did not show consistently better results than groups who did not receive training.

Video Simulation Versus Other Forms of Training (5 Studies) (Table 9)

TABLE 9. Video Simulation Versus Other Forms of Training

Generally, no differences were seen between video box training and other forms of training such as bench models or standard training.

Physical or Model Simulation Versus Other Forms of Training, Including No Training (4 Studies) (Table 10)

TABLE 10. Model Simulation Versus Other Forms of Training

While these comparisons showed mixed results, model training may be better than no training and standard training such as instruction from mentors or manuals.

Cadaver Training Versus Standard Training (1 Study)

In Anastakis et al,¹⁵ the cadaver trained group received better scores than the standard training group, which learned independently from the manuals, for the global assessment of operative performance on cadavers (66.5 versus 51.5) and the checklist score (69.5 versus 60.5), although it was not stated whether these differences were statistically significant.

DISCUSSION

Computer simulation generally showed better results than no training at all but was not convincingly superior to standard training (such as surgical drills) or video simulation (particularly when assessed by operative performance). Video simulation did not show consistently better results than groups with no training at all, and there were not enough data to determine if video simulation was better than standard training or the use of models. Model simulation may have been better than standard training, and cadaver training may have been better than model training. Only one RCT³⁹ made a comparison between computer simulation and model training, which showed LapSim to be superior to training on a physical Tower trainer.

With 30 RCTs testing the effect of surgical simulation for training, a clearer result might have been expected. Possible reasons for failing to see a clear benefit for surgical simulation include the following:

1. Small sample sizes. There were only 760 participants in total across all the RCTs, meaning that the ability to detect differences between different forms of training was quite limited.

2. Multiple and confounding comparisons. The power to detect any differences was further diluted by the large number of different comparisons, both within and between studies, and the comparisons may have been confounded by factors such as mentoring, which could have swamped any effect of simulation.

3. Disparate interventions. The component tasks of surgical simulation varied considerably between studies, which may have obscured results for individual tasks such as knot tying. On the other hand, it may be more realistic to test a “package” of surgical skills, but there did not appear to be a consensus as to what might constitute a core set or package of surgical skills.

4. The comparators were not standardized. “Standard” training covered a wide range of activities, some of which were of very minimal intensity (eg, reading a manual, watching a video). Even so, the simulated groups did not always show superiority even over groups with minimal training.

5. The surgical simulations may not have been intensive or long enough to show an effect on training. While some simulator interventions were very brief, a “practice” effect with longer simulator exposure was not generally evident, although such a practice effect may only be apparent for quite high levels of simulator use. However, most simulator interventions (and comparators) showed significant improvements pre and post intervention (or comparator).

6. Ideally, outcome assessors would have been blinded to group allocation because many of the measurements can be quite subjective (eg, global scores). However, such blinding was not routinely used in the studies included in this review.

7. Most studies measured simulator, and not actual, performance. Unless the simulator has previously been validated by assessing operative performance, or the study assesses skills gained on the simulator by assessing operative performance, we cannot be sure that observed improvements on the simulator will translate into improved operative performance. The failure to translate simulator performance into actual operative performance (concurrent validity) was illustrated in a validity study by Paisley et al,¹⁰ which also showed that simulator performance did not discriminate between naive and experienced surgeons (construct validity). The reasons for this apparent lack of concurrent validity cannot be ascertained from this review, but the relatively new nature of simulation (particularly computer simulation and virtual reality) may be significant causes. Perhaps surgical simulation is not yet realistic enough, in regard to representation of anatomy or haptics are simulated inadequately. Perhaps the simulated tasks are isolated or not presented in the right order, or perhaps they are not even the most appropriate tasks.

8. Although it is very difficult to show equivalence (particularly with the small sample sizes and disparate comparisons), a more optimistic interpretation of the findings of this review would be that surgical simulation may be just as good as other forms of surgical training. In other words, can surgical simulation provide an alternative to standard surgical training? Most of the RCTs in this review showed that both simulator and standard training groups improved significantly from baseline, but that participants' final scores usually did not show differences between the simulator and other groups. Apart from the obvious avoidance of experimenting on human patients and not using cadavers or animals, surgical simulation would enable surgical experience earlier in training, make more cases available, give exposure to a wider range of pathologies in a compressed period, allowing acceleration of the learning curve, and give exposure to new procedures and technologies.⁴⁰ Surgical simulation is claimed to be able to save money, although this does not appear to have been substantiated yet. Computer simulation and virtual reality have received much attention, which has eclipsed investigation of other forms of simulation such as models, which may be cheaper and as effective. Unfortunately, many of the studies in surgical simulation have been of poor experimental design, leading Champion and Gallagher⁴¹ to state that “bad science in the field of medical simulation has become all too common.” Even though this review included only RCTs, potentially the most rigorous study design for intervention studies, generally there were significant problems with small sample sizes and the validity and reliability of outcome measurements in particular. The studies by Grantcharov et al,¹⁸ Scott et al,³² and Seymour et al³³ stand out as higher-quality studies since they measured outcomes by actual operative performance, although only some elements were measured in Grantcharov et al.¹⁸

CONCLUSION

While there may be compelling reasons to reduce reliance on patients, cadavers, and animals for surgical training, none of the methods of simulated training (including computer simulation) has yet been shown to be better than other forms of surgical training. In addition, little is known about the real costs (including adverse outcomes in patients) of either simulated or standard surgical training.

Adequately powered, well-designed, and unconfounded RCTs (preferably multicenter with similar protocols) are needed and outcome assessors need to be blinded. Outcomes need to be tested in actual operative circumstances (or on validated systems). In particular, model simulation needs to be further tested against computer simulation. Studies of cost comparisons also need to be done. The RCTs dealt exclusively with technical skills, although other skills such as cognitive skills and communication skills are clearly integral parts of surgical performance.