Abstract
Background
Virtual reality (VR) is an emerging new modality for laparoscopic skills training; however, most simulators lack realistic haptic feedback. Augmented reality (AR) is a new laparoscopic simulation system offering a combination of physical objects and VR simulation. Laparoscopic instruments are used within an hybrid mannequin on tissue or objects while using video tracking. This study was designed to assess the difference in realism, haptic feedback, and didactic value between AR and VR laparoscopic simulation.
Methods
The ProMIS AR and LapSim VR simulators were used in this study. The participants performed a basic skills task and a suturing task on both simulators, after which they filled out a questionnaire about their demographics and their opinion of both simulators scored on a 5-point Likert scale. The participants were allotted to 3 groups depending on their experience: experts, intermediates and novices. Significant differences were calculated with the paired t-test.
Results
There was general consensus in all groups that the ProMIS AR laparoscopic simulator is more realistic than the LapSim VR laparoscopic simulator in both the basic skills task (mean 4.22 resp. 2.18, P < 0.000) as well as the suturing task (mean 4.15 resp. 1.85, P < 0.000). The ProMIS is regarded as having better haptic feedback (mean 3.92 resp. 1.92, P < 0.000) and as being more useful for training surgical residents (mean 4.51 resp. 2.94, P < 0.000).
Conclusions
In comparison with the VR simulator, the AR laparoscopic simulator was regarded by all participants as a better simulator for laparoscopic skills training on all tested features.
There is consensus that education in minimally invasive surgery should be intensified and that more objective assessment of surgeons’ skills should be introduced to ensure high-quality treatment.3 The growing need for training in advanced laparoscopic skills outside the operating room and continued advances in computer technology have led to growing interest in the development of virtual reality (VR) simulators.1,2 Another simulator system is augmented reality (AR), which refers to systems that merges computer graphics and real imagery into a single, coherent perception of an enhanced world around the user. This means that the image one sees on the simulator screen is comprised of a real video image overlaid with a graphics image. This can, for example, be used for directional explanation.
According to various studies2,3,4 force (tactile) feedback is fundamental for good laparoscopic training and results in significantly improved skills transfer to the trainee compared with training without force feedback.2 AR laparoscopic simulators have realistic force feedback because the operator practices on real objects or tissue using surgical laparoscopic instruments. Objective performance assessment is fundamental to provide formative feedback during training, though conventional box trainers lack this ability. AR laparoscopic simulators retain the benefits of a box trainer, such as the realistic haptic feedback, but additionally generate objective measures of performance, similar to VR simulators.1
This study focused on training, simulation, and skills assessment in laparoscopic surgery. Because of the importance of both force feedback and objective assessment, we compared the ProMIS (Haptica) AR laparoscopic simulator with the LapSim (Surgical Sciences) VR laparoscopic simulator. Issues such as realism, didactic value, haptic feedback, and user acceptance were validated by participants. The purpose of this study was to evaluate which training system is of more value in curricula for surgical residents in laparoscopic skills training.
We hypothesized that as a training tool for laparoscopic surgery, the ProMIS AR simulator would be better evaluated by the participants than the LapSim VR simulator.
MATERIALS AND METHODS
Subjects
In total, 90 participants took part in this study. Forty-four participants were tested during the Year Congress of the Dutch Surgical Society 2006, Veldhoven; 34 in the Catharina Hospital, Eindhoven; and and 12 at the International Student Congress of Medical Sciences (ISCOMS) 2006, Groningen, The Netherlands. Participants were allotted to 3 groups based on their clinical laparoscopic experience: experts, who performed more than 100 clinical procedures; intermediates, with less than 100 clinical procedures; and novices, who had no laparoscopic experience.
Equipment
ProMIS Augmented Reality Simulator
The ProMIS AR simulator (Haptica, Dublin, Ireland) (Fig. 1) was used in this study. A torso-shaped mannequin (737-mm long × 508-mm wide × 22-9mm deep) with a black neoprene cover contains an instrument tracking system by means of 3 cameras arranged to identify any instrument inside the simulator from 3 different angles. The left and right cameras are positioned to capture instrument motion, looking in caudal direction of the left and right sides of the mannequin, respectively. The laparoscopic camera is positioned at the pubic symphysis of the mannequin looking cephalad and provides the endoscopic image on the screen.
Figure 1.
ProMIS Augmented Reality laparoscopic simulator (Haptica), with 5-mm endograsps and 5- and 10-mm versaport trocars (Tyco Auto Suture, New Haven, CT, USA).
The camera tracking system captures instrument motion with Cartesian coordinates in the x, y, and z planes at the average rate of 30 (fps). The laparoscopic instruments of choice (Figs. 1 and 2) are marked with 2 rings of yellow electrical tape at the distal end of the shaft, at a fixed distance, to serve as a reference point for the camera tracking system (Fig. 2). For data collection and video imaging display, a Sony Vaio portable notebook computer is used (with a 2.80-GHz Intel Pentium 4 processor running Windows XP Home Edition).
Figure 2.
The needle holders (Karl Storz, Tutlingen, Germany) are marked with the black-yellow tags on the shaft to enable video tracking.
The simulator records time, instrument path length, and smoothness of movement (through changes in instrument velocity and changes in direction) for each instrument (right and left hand) during each separate task within the training module. After completion of the task, ProMIS AR provides measurements and statistics on each performed task and module. In addition, a full video and virtual playback of the trainee’s performance are saved, reviewed, and evaluated. Different trays may be placed in the mannequin for each task, such as the suturing pads for the suture and knot-tying tasks. In this study we used a 1-cm-thick suturing pad, which is used in traditional box trainers.
LapSim Virtual Reality Simulator
The LapSim VR simulator (Surgical Sciences) (Fig. 3) is a PC-based trainer for laparoscopic surgery. This simulator is uses a Dell Precision desktop computer (with a 1.5-GHz dual processor systems, Vidia Quadro2EX 32 MB graphics card, and running Windows 2000) on a 17-in., CRT monitor. The VR laparoscopic instruments (Immersion Medical, San Jose, CA, USA) consist of 2 handles with sensors using a 5-degree-of-freedom tracking system for each instrument. The image on the monitor is a computer-generated virtual reality representation of laparoscopic tasks. The software version used in this study was Basic Skills 3.0.
Figure 3.
LapSim Virtual Reality laparoscopic simulator (Surgical Sciences).
Parameters recorded by the simulator are time, instrument-path length, tissue damage, overall score, and a pass/fail score. These parameters are calculated by the tracking system in each of the instrument handles.
Questionnaire
Participants were asked to fill out a questionnaire that consisted of 3 parts: the 1st part referred to demographics and laparoscopic and/or simulator experience. The second part referred to realism, didactic value, haptic feedback, and usefulness of the ProMIS AR simulator; the feature “didactic value” referred to the additional didactic value of a curriculum with this simulator implemented in it. The last part referred to realism, didactic value, haptic feedback, and usefulness of the LapSim VR simulator. The latter 2 parts had to be answered on a 5-point Likert scale. The questionnaire ended with an open-ended question for general remarks on both simulators.
Protocol
The participants commenced by filling out the first part of the questionnaire. Subsequently, all participants received an introduction about both simulators by means of posters with a short verbal explanation. Information was given about the various training modules available and about the feedback provided by the simulators. Furthermore, the tasks to be performed on both simulators within the scope of this study were clarified. A demonstration video was additionally shown on the ProMIS AR and an instruction text on the LapSim VR before each task. The order in which the participants encountered the simulators was systemically altered (at random) to avoid possible carryover effects. The participants first performed the translocation task, followed by the suturing task(s) (Figs. 4, 5, and 6, 7). After completing the tasks on 1 of the simulators, the participants filled out the corresponding questionnaire. They then performed the equivalent tasks on the other simulator, followed by the last part of the questionnaire.
Figure 4.
LapSim virtual reality simulator: “Lifting and grasping” task.
Figure 5.
LapSim virtual reality simulator: “Suturing” task.
Figure 6.
ProMIS augmented reality simulator: “Translocation” task.
Figure 7.
ProMIS augmented reality simulator: “Suturing” task.
A maximum time limit of 3 minutes for the translocation task on the LapSim and 5 minutes for the remaining tasks was used, as trained surgeons are expected to be able to perform such a task easily within this time. The data (Table 1) recorded by the simulators was extracted to compare the construct validity of both simulators.
Table 1.
Modules and parameters used for this study
| ProMIS augmented reality simulator | LapSim virtual reality simulator | |
|---|---|---|
| Modules | Instrument handling (translocation) | Lifting and grasping (translocation) |
| Suturing and knot tying | Suturing and knot tying | |
| Recorded parameters | Time (seconds) | Time (seconds) |
| Path length (mm) | Pass/fail (P/F) | |
| Smoothness | Score (%) | |
| Path length (mm) | ||
| Tissue damage |
Data Analysis
All data were processed and analyzed using SPSS 13.0. Significant difference in opinion about these 2 simulators was calculated with the paired t-test. The significant difference in the level of skills extracted from the simulators between the 3 groups was calculated with the Kruskal–Wallis test.
RESULTS
Demographics
A total of 90 subjects participated in this study: o30 “experts”, 30 “intermediates” and 30 “novices”. The 3 groups were not homogenous in all demographic aspects. The expert and intermediate groups contained relatively more male participants (76.7% and 93.3%, respectively), whereas amongst the novices, there were more female participants (63.3%). This can be explained by the fact that in the population we tested, gender division was not equal either. Most (96.7%) participants were right handed, which was equally divided in all groups. The age difference was considerably different between groups (Table 2).
Table 2.
Participant demographics
| Demographics | Experts (n = 30) | Intermediates (n = 30) | Novices (n = 30) | Total (n = 90) |
|---|---|---|---|---|
| Mean age (standard deviation) | 45 years (7.49) | 31 years (3.02) | 24 years (4.51) | 33.91 years (10.27) |
| (Min–max) | (28–64) | (28–46) | (21–42) | (21–64) |
| Education | ||||
| Intern | 0 | 0 | 27 | 27 |
| Surgical resident | 1 | 23 | 1 | 25 |
| Other Resident | 0 | 5 | 2 | 7 |
| Surgeon | 29 | 1 | 0 | 30 |
| Other specialist | 0 | 1 | 0 | 1 |
The experts all had laparoscopic suturing experience in the clinical setting (with more than 56% >20 times), more than half of the intermediate group had some suturing experience, and none of the novices had any suturing experience (Table 3). All experts and intermediates had assisted in laparoscopic procedures before, while some novices had not even seen a laparoscopic procedure (Table 3). Most intermediates had prior experience on laparoscopic simulators such as VR systems and box trainers, as these are used in training curricula.
Table 3.
Participant laparoscopic experience
| Experience | Experts (n = 30) | Intermediates (n = 30) | Novices (n = 30) | Total (n = 90) |
|---|---|---|---|---|
| Suturing in clinical setting | ||||
| 0 times | 2 | 14 | 30 | 46 |
| 1–5 times | 3 | 9 | 0 | 12 |
| 5–20 times | 8 | 5 | 0 | 13 |
| >20 times | 17 | 2 | 0 | 19 |
| Procedures: mean (min–max) | ||||
| Seen | 672 (100–3000) | 103 (10–300) | 11 (0–50) | 252 (0–3000) |
| Camera handling | 302 (50–1500) | 56 (0–250) | 2 (0–20) | 115 (0–1500) |
| Assisted | 271 (50–1000) | 52 (0–200) | 2 (0–30) | 103 (0–1000) |
| Simulator | ||||
| No | 10 | 4 | 22 | 36 |
| 1–2 times | 6 | 8 | 7 | 21 |
| 2–5 times | 5 | 8 | 1 | 14 |
| >5 times | 9 | 10 | 0 | 19 |
Realism and Haptics
As presented in Table 4, there was a significant difference in participant opinion with regard to features of the 2 simulator systems. Overall, the ProMIS AR scored higher on all aspects than the LapSim VR simulator, with a P-value of 0,000.
Table 4.
Difference in opinion about ProMIS augmented reality and LapSim virtual reality simulators
| Ratings | ProMIS | LapSim | P-valuea | ||
|---|---|---|---|---|---|
| Mean | Standard deviation | Mean | Standard deviation | ||
| Translocation: haptics | 4.10 | 0.937 | 2.26 | 1.066 | 0.000 |
| Translocation: instrument movement | 4.22 | 0.700 | 2.18 | 1.023 | 0.000 |
| Suturing: realism | 4.43 | 0.657 | 2.49 | 1.072 | 0.000 |
| Suturing: haptics tissue | 3.92 | 0.800 | 1.92 | 0.923 | 0.000 |
| Suturing: resistance needle and thread | 4.15 | 0.708 | 1.84 | 0.926 | 0.000 |
a Significance is calculated with the paired t-test.
The expert group rated “resistance of needle and thread” on the LapSim VR as not good, with a mean of 1.57. Of all the participants, 83.4% judged the “haptic sensations” and 88.9% the “resistance and movements of the instruments” in the ProMIS AR translocations task as good to very good (4 resp. 5 on the Likert scale). These features from the LapSim VR score were rated badly to very badly in 62.2% and 67.8% of respondents, respectively (1 resp. 2 on the Likert scale) (Table 4). On the “suturing” features, the ProMIS AR was rated 2.0 points higher on average than the LapSim VR. The ProMIS AR “suturing task” was considered very realistic by 53.3% of the experts in contrast to 0% for the LapSim VR. The “haptic sensation” and “resistance of needle and thread” of this task were rated 4 and 5 in 70% and 83.3%, respectively, on the ProMIS AR and only 4.4% and 6.6% of respondents, respectively, gave a good rating for the LapSim VR simulator (Table 4). On the ProMIS AR simulator, “haptic sensations of the tissue” was regarded by all participants as less than the other features questioned, but it was still regarded as quite good (mean 3.92). The mean rating remained 2.0 point higher than the rating on the LapSim VR (mean 1.92).
Training Properties
Training properties of both simulators were also part of this study. On the didactic value, the ProMIS AR scored higher than the LapSim VR by all participants (Table 5). The “ProMIS AR simulator as a training tool for surgical residents” scored 4.67 in the expert group and 4.30 in the novice group. The LapSim VR simulator was scored 2.67 and 3.03, respectively, by the same groups for this feature. Of the experts, 66.7% consider the ProMIS AR a very good “tool for training laparoscopic skills to surgical residents,” contrary to 3.3% for the LapSim VR. On the learning properties of the simulators, 83.3% of the experts indicated that the “ProMIS AR can teach trainees the proper skills,” whereas 13.3% stated that the LapSim VR can serve this purpose.
Table 5.
Difference in training properties of the ProMIS augmented reality and LapSim virtual reality simulators
| Ratings | ProMIS | LapSim | P-valuea | ||
|---|---|---|---|---|---|
| Mean | Standard deviation | Mean | Standard deviation | ||
| Training surgical residents | 4.51 | 0.707 | 2.94 | 1.105 | 0.000 |
| Training surgeons | 4.38 | 0.696 | 2.52 | 1.094 | 0.000 |
| Learning proper skills | 4.08 | 0.651 | 2.86 | 1.014 | 0.000 |
| Simulator appeals to me | 4.17 | 0.706 | 2.59 | 1.182 | 0.000 |
a Significance is calculated with the paired t-test.
Opinion on the LapSim VR was divided amongst all participants, which is demonstrated by the wide standard deviations in Table 5. In general, the rating of LapSim VR by the novice group was higher compared with ratings by the intermediate and expert groups. This can be explained because the novice group consists mainly of interns who have not seen, assisted with, or handled a laparoscopic camera before. Therefore, they have no reference point from which to compare the laparoscopic simulators nor to judge on realism. Especially in this group, one may observe a difference between the part of the group that started the study on the LapSim VR first and the other part that started first on the ProMIS AR. But the general opinion on the open-ended question after completion of the training session was that the ProMIS AR simulator was much more realistic and a better training system than the LapSim VR simulator.
Differences in Construct Validity
The current study enabled collection of parameters recorded by both simulators, which are presented in Tables 6 and 7. To assess the parameter significance and determine skill levels, we used the Kruskal–Wallis test. In all tasks investigated in this study, we used a time limit from 300 seconds on each task of the ProMIS AR (“translocation,” “needle handling,” and “knot -tying”). On the LapSim VR, a limit was set at 180 seconds (30 seconds for each handling) for the “translocation” task and 300 seconds for the “suturing and knot-tying” task. On the ProMIS AR simulator, time was shown to be a significant parameter to determine skill level for both “translocation” and “suturing” skills (P = 0.000); on the LapSim VR, there were no significant differences found in this parameter.
Table 6.
Construct validity of the ProMIS augmented reality simulator
| ProMIS | Expert (n = 30) | Novice (n = 30) | P-valuea |
|---|---|---|---|
| Translocation | |||
| Time (seconds) | 122.13 | 195.98 | 0.000 |
| Path length (mm) | |||
| Left | 248.57 | 309.62 | 0.010 |
| Right | 243.28 | 373.99 | 0.000 |
| Smoothness (mm) | |||
| Left | 411.50 | 717.73 | 0.000 |
| Right | 409.63 | 695.40 | 0.001 |
| Suturing | |||
| Time (seconds) | 357.30 | 562.17 | 0.000 |
| Path length (mm) | |||
| Left | 1112.72 | 1220.98 | 0.104 |
| Right | 1068.44 | 1624.92 | 0.000 |
| Smoothness (mm) | |||
| Left | 1135.13 | 1911.70 | 0.000 |
| Right | 1247.07 | 1998.47 | 0.000 |
The data in this table represent the main value. There was a time limit used of 180 sec on the translocation task and 300 sec on the suturing task.
a Significance is calculated with the Kruskal Wallis test.
Table 7.
Construct validity of the LapSim virtual reality simulator
| LapSim | Expert (n = 30) | Novice (n = 30) | P-valuea |
|---|---|---|---|
| Translocation | |||
| Time (seconds) | 119.03 | 128.31 | 0.209 |
| Score (%) | 60.13 | 52.97 | 0.164 |
| Path length (mm) | |||
| Left | 225.21 | 220.27 | 0.525 |
| Right | 193.57 | 200.17 | 0.416 |
| Tissue damage (mm) | 3717.25 | 4501.65 | 0.086 |
| Suturing | |||
| Time (seconds) | 281.15 | 300.04 | 0.128 |
| Score (%) | 90.17 | 89.50 | 0.507 |
| Path length (mm) | |||
| Left | 319.46 | 283.98 | 0.243 |
| Right | 509.22 | 470.51 | 0.264 |
| Tissue damage (mm) | 5707.62 | 10178.00 | 0.009 |
The data in this table represent the main value. There was a time limit used of 180 sec on the translocation task and 300 sec on the suturing task.
a Significance is calculated with the Kruskal Wallis test.
“Smoothness” was calculated on the ProMIS AR by directions and accelerations in instrument movements. From the results depicted in Table 6, one can conclude that all measured parameters on the ProMIS AR were significant to determine skill levels (P < 0.01) except for the “left path length” of the “suturing” task. On the other hand, all parameters measured on the LapSim VR not significant for establishing construct validity, except for the “tissue damage” on the “suturing” task (P = 0.009).
DISCUSSION
Simulator implementation into surgeon training curricula is of paramount importance, and it progressively gains acceptance.5,6 The impact of simulator use is strongly dependent on the correct choice of the system to be used and should be based upon the outcome of validation studies.7 Validation should provide information indicating the usefulness of the chosen system in general and should in particular indicate which system should be used in which training and/or curriculum. Data on (face) validation of the LapSim VR laparoscopic simulator and the ProMIS AR laparoscopic simulator are relatively scarce.2,5,7,8,9,10
Various devices are used to aid training in laparoscopic skills. These devices range from simple box trainers to sophisticated VR trainers. Whereas VR trainers may have some advantages, most participants of such studies feel that inanimate box trainers help more, are more interesting, and are preferred over VR trainers if only 1 trainer is allowed.11 This opinion is supported by the outcome of the current study, as all the participants favored a non-VR environment, with the real instruments used on physical objects, as in traditional box trainers. In the tasks we investigated in this study, we did not use the VR features of the ProMIS AR simulator ,which are present for some modules (e.g., orientation and diathermia), so the video images were similar to those of box trainers.
Various studies have indicated that haptic (force) feedback is very important in laparoscopic simulation and increases the effectiveness of VR training systems.12–14 But this is also one of the most controversial issues in VR laparoscopic simulators design, because it is very difficult and complicated to incorporate realistic haptic feedback into these systems.12 For AR haptics, this is not an issue because of the nature of the system. The outcome of the current study allows the conclusion that ProMIS AR has far better feedback during task performance compared with LapSim VR. Apparently ProMIS AR offers a good alternative to VR laparoscopic simulation, as it retains the benefits of VR and additionally offers more realistic haptic feedback. As shown in Tables 4 and 5, the ProMIS AR simulator rated higher than the LapSim VR simulator on realism, haptic features, and didactic value.
In this study, we encountered the Hawthorn effect,15 which is the influence of the individual performance on the opinion about the simulators. For example, on the LapSim VR simulator, when participants had to perform the “suturing and knot-tying” task, the majority of the participants (in all groups) were not able to pass the needle through the tissue and therefore could not tie knots. This annoyed most of them and was noticeable when they had to fill out the questionnaire. On the ProMIS AR, these suturing skills were tested separately, which caused less frustration.
The outcome of the current study indicates that the construct validity on the “translocation” and “suturing tasks” of the ProMIS AR laparoscopic simulator adequately distinguished between experts and novices. Other studies have shown equal results on the construct validity of this simulator.2,16 This cannot be stated for the LapSim VR laparoscopic simulator, which only shows construct validity on the “tissue damage” of the “suturing task” (Table 6). Previous studies8,10,17–19 on the construct validity of the LapSim VR show conflicting results: only a few parameters were to show construct,17,18 or only when the most (attending surgeons) and least (interns) experienced were compared.8 One study10 even concluded that novices tended to have a better performance compared with the experienced laparoscopists during their first exposure to the LapSim.
To be an effective training tool, the simulator must provide metrics that are meaningful and informative to the trainee. “Time of performance” as a sole parameter might not be the best criterion on which to grade the trainee. The primary issue is that the correct technique is used and a tight knot is made, whereas “time of performance” is a secondary issue. Therefore, it is important to evaluate other metrics recorded by the simulator. In the ProMIS AR simulator, the other parameters, such as “path length” and “smoothness”, give a better impression of the performance and are shown to have a construct validity.
The only measured parameter from the LapSim VR, with construct validity, is “tissue damage”, but this is not sufficient as a sole metric upon which to base assessment of the level of skills of the trainee. This study specifically shows that the ProMIS AR laparoscopic simulator has a better construct validity and is therefore better as a training tool concerning feedback recordings than is the LapSim VR laparoscopic simulator.
CONCLUSION
The outcome of this study allows us to conclude that the ProMIS AR laparoscopic simulator is regarded as a better tool for training surgical residents than the LapSim VR laparoscopic simulator on the limited number of tasks tested. The reason for this outcome is, in our opinion, that AR offers better realism, haptic feedback, didactic value, and construct validity than does VR, and it also gives useful feedback to determine trainee skill levels. It is recommended to implement the ProMIS AR laparoscopic simulator in the training curricula in laparoscopic skills for surgical residents. The place of this simulator in the curricula is in the beginning of the surgical residency, at the cholecystectomy level. It is important that surgical residents not only learn basic laparoscopic skills at this level but also suturing skills for when they, e.g., need to tie the cystic duct when perforations occur in the clinical setting.
Acknowledgments
There are no conflicts of interest in this study, and sponsoring of this study is in name of The Scientific Foundation of the Catharina Hospital Eindhoven, The Netherlands. The study was carried out by objective researchers who have no attachments with the industry.
References
- 1.Hance J, Aggarwal R, Undre S, et al. Evaluation of a laparoscopic video trainer with in-built measures of performance. JSLS 2004;8:S51
- 2.Van Sickle KR. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc 2005;19:1227–1231 [DOI] [PubMed]
- 3.Grantcharov TP, Kristiansen VB, Bendix J, et al. Randomized clinical trail of virtual reality simulation for laparoscopic skills training. Br J Surg 2004;91:146–150 [DOI] [PubMed]
- 4.Seymour N, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236:458–463 [DOI] [PMC free article] [PubMed]
- 5.Roberts KE, Bell RL, Duffy AJ. Evolution of surgical skills training. World J Gastroenterol 2006;28, 12(20):3219–3224 [DOI] [PMC free article] [PubMed]
- 6.Jakimowicz JJ, Cuschieri A. Time for evidence-based minimal access surgery training: Simulate or sink. Surg Endosc 2005;19:1521–1522 [DOI] [PubMed]
- 7.Carter FJ, Schijven MP, Aggerwal R, Grantcharow T, Francis NK, Hanna GB. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc 2005;19:1523–1532 [DOI] [PubMed]
- 8.Duffy AJ, Hogle NJ, McCarthy H, et al. Construct validity for the LAPSIM laparoscopic surgical simulator. Surg Endosc 2005;19:401–405 [DOI] [PubMed]
- 9.Tomulescu V, Popescu I. The use of LapSim virtual reality simulator in the evaluation of laparoscopic surgery skill. Preliminary results. Chirurgia (Bucur) 2004;99(6):523–527 [PubMed]
- 10.Ro CY, Toumpoulis IK, Ashton RC Jr, et al. The LapSim: a learning environment for both experts and novices. Stud Health Technol Inform 2005;111:414–417 [PubMed]
- 11.Madan AK, Frantzides CT, Tebbit C. Quiros RM Participants’ opinions of laparoscopic training devices after a basic laparoscopic training course. Am J Surg 2005;189(6):758–761 [DOI] [PubMed]
- 12.Lamata P, Gómez EJ, Sánchez-Margallo FM, et al. Study of laparoscopic forces perception for defining simulation fidelity. Stud Health Technol Inform 2005;119:288–292 [PubMed]
- 13.Kim HK, Rattner DW, Srinivasan MA. Virtual-reality-based laparoscopic surgical training: the role of simulation fidelity in haptic feedback. Comput Aided Surg 2004;9(5):227–234 [DOI] [PubMed]
- 14.Maass H, Chantier BB, Cakmak HK, et al. Fundamentals of force feedback and application to a surgery simulator. Comput Aided Surg 2003;8(6):283–291 [DOI] [PubMed]
- 15.Schijven M, Jakimowicz J. Face-, expert- and referent validity of the Xitact LS500 Laparoscopy simulator. Surg Endosc 2002;16:1764–1770 [DOI] [PubMed]
- 16.Broe D, Ridgway PF, Johnson S, et al. Construct validation of a novel hybrid surgical simulator. Surg Endosc 2006;20(6):900–904 [DOI] [PubMed]
- 17.Woodrum D, Andreatta P, Yellamanchilli R, Feryus L, Gauger P, Minter RM. Construct validity of the LapSim laparoscopic surgical simulator. Am J Surg 2006;191(1):28–32 [DOI] [PubMed]
- 18.Eriksen JR, Grantcharov T. Objective assessment of laparoscopic skills using a virtual reality stimulator. Surg Endosc 2005;19(9):1216–1219 [DOI] [PubMed]
- 19.Sherman V, Feldman LS, Stanbridge D, Kazmi R, Fried GM. Assessing the learning curve for the acquisition of laparoscopic skills on a virtual reality simulator. Surg Endosc 2005;19(5):678–682 [DOI] [PubMed]