Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 May 1.
Published in final edited form as: Surg Endosc. 2012 Dec 14;27(5):1721–1729. doi: 10.1007/s00464-012-2664-y

Face and Construct Validation of a Virtual Peg Transfer Simulator

Venkata S Arikatla 1, Ganesh Sankaranarayanan 1, Woojin Ahn 1, Amine Chellali 2, Suvranu De 1, GL Caroline 5, John Hwabejire 3, Marc DeMoya 3, Steven Schwaitzberg 2, Daniel B Jones 4
PMCID: PMC3625247  NIHMSID: NIHMS430917  PMID: 23263645

Abstract

Background

The Fundamentals of Laparascopic Surgery (FLS) trainer box is now established as a standard for evaluating minimally invasive surgical skills. A particularly simple task in this trainer box is the peg transfer task which is aimed at testing the surgeon’s bimanual dexterity, hand-eye coordination, speed and precision. The Virtual Basic Laparoscopic Skill Trainer (VBLaST©) is a virtual version of the FLS tasks which allows automatic scoring and real time, subjective quantification of performance without the need of a human proctor. In this paper we report validation studies of the VBLaST© peg transfer (VBLaST-PT©) simulator.

Methods

Thirty-five subjects with medical background were divided into two groups: experts (PGY 4-5, fellows and practicing surgeons) and novices (PGY 1-3). The subjects were asked to perform the peg transfer task on both the FLS trainer box and the VBLaST-PT© simulator and their performance was evaluated based on established metrics of error and time. A new length of trajectory (LOT) metric has also been introduced for offline analysis. A questionnaire was used to rate the realism of the virtual system on a 5-point Likert scale.

Results

Preliminary face validation of the VBLaST-PT© with 34 subjects rated on a 5-point Likert scale questionnaire revealed high scores for all aspects of simulation, with 3.53 being the lowest mean score across all questions. A two-tailed Mann-Whitney performed on the total scores showed significant (p=0.001) difference between the groups. A similar test performed on the task time (p=0.002) and the length of trajectory (p=0.004) separately showed statistically significant differences between the experts and novice groups (p<0.05). The experts appear to be traversing shorter overall trajectories in less time than the novices.

Conclusion

VBLaST-PT© showed both face and construct validity and has promise as a substitute for the FLS to training peg transfer skills.

Keywords: Virtual reality, Surgical training, Face validity, Construct validity, Laparoscopy, Length of trajectory

Introduction

In spite of its many advantages of faster recovery, minimal blood loss and lower cost of treatment, minimally invasive surgery (MIS) requires extensive training and practice over longer periods due to complex hand-eye coordination skills, limited field of view and reduced haptic feedback [1]. The Fundamentals of Laparoscopic Surgery (FLS) has been developed to address the issue of evaluation of laparoscopic surgical skills [2]. The FLS trainer box consists of five tasks: peg transfer, pattern cutting, ligation loop, suturing with intracorporeal and suturing with extracorporeal knot tying. The FLS program and the trainer box is approved and distributed by SAGES. Validation studies of the FLS tasks have been reported in the literature [3] [4]. Passing FLS has become part of board certification in laparoscopic surgery in USA.

Virtual reality (VR) based training simulators are becoming common in the surgical training curricula [5][6]. VR trainers allow objective quantification of surgical skills in real time without the need for proctors, provide unlimited training materials and a safe training environment [7][8]. With funding from the National Institutes of Health, we have started work on developing the virtual version of the FLS tasks. The Virtual Basic laparoscopic Skill Trainer (VBLaST©) program has both hardware and software components, explained in section 2, which allows surgeons to perform the virtual FLS tasks using tool handles which are used in the actual FLS toolbox. FLS even though the standard for board certification in laparoscopic surgery, required trained proctors to administer and score the tests. Moreover the materials need to be constantly replenished. The (VBLaST©system is designed to replicate the FLS tasks with the added advantage of objective automated scoring and the possibility of providing the exams all through the year instead of select few centers and conferences where the FLS exams are currently administered.

In this work we present face and construct validation of the virtual peg transfer simulator (VBLaST-PT©). VBLaST-PT© is capable of high quality multimodal user feedback with an automated scoring system [9] that is modeled after the scoring system for FLS. In this study we aim to establish the quality and accuracy of the simulator through face and construct validation studies. In the case of construct validity we tested the hypothesis that the mean scores obtained from VBLaST-PT© are significantly different for both the groups.

Materials and Methods

The two main components of the VBLaST-PT© simulator are the hardware and software modules which work together to provide high fidelity simulation. In this section we provide a brief description of these two components (for a more detailed description see [10]). We also lay out the design of the validation experiments.

a) Simulation Software

The simulation software involves modeling the peg board, pegs and the tools along with interactions between them. In VBLaST-PT© all six pegs are modeled as rigid objects. While the pegs in the FLS toolbox are not perfectly rigid, they are rather stiff. Our user study does not indicate loss of fidelity based on this assumption. The pegs interact with the rigid peg board (Figure 1) that has twelve posts arranged differently, six on each side. During the simulation, two tool graspers are used to manipulate the position and orientations of the pegs. All the interactions are modeled so as to provide high quality force feedback.

Figure 1.

Figure 1

Open in a new tab

Snapshots of the simulation during various stages of the peg transfer task

In order to detect “picking”, the positions and orientations of the tool jaws were tested with respect to the peg. The picking condition is only satisfied when the tool jaws are on either sides of the peg and the jaws are tightened over the peg. If the peg is picked, we use a virtual coupling mechanism [11] to separate the visual and haptic representations in order to create picking interactions like collision of peg with the peg post and with the other tool. This method renders reasonably accurate forces while displaying appropriate positions for the tool and the peg. The stiffness of the forces is fine tuned based on feedback from the surgeons. Besides the interactions we also automatically record various metrics for offline analysis. We record (a) position of the tools (b) velocity of the tools (c) roll angle of the tools (d) time stamp for state change of each peg (e) overall time to complete the task and (f) state of the peg. Each peg can have three states - it can be picked by one tool, two tools or not picked at all.

The software can automatically detect events including dropping of pegs, transfer of pegs between tools and release of pegs. The software can also decompose the subtasks and provide a detailed analysis of time taken and trajectories for each sub task. This is accomplished through interpreting the states of each peg. A Matlab® script is generated automatically at the end which displays the statistics of the subtasks. Other metrics such as length of trajectory can be automatically obtained once the task is completed.

b) Hardware

We have developed a specialized hardware interface for the VBLaST™ that can be utilized for all the tasks with minor modifications. The main advantage of such a design is its versatility and the ability to use exactly the same tool handles as used in the FLS toolbox. Two Phantom® Omni™ haptic devices from SensableR Inc are used for force feedback. These devices are placed within an aluminum structure shown in Figure 2(a). The aluminum structure also allows for placement of the tools that interface with the haptic devices. Trocars are fitted into a 2-axis gimbal mechanism. This gimbal arrangement allows for rotation in the left-right and forward-backward directions. The rotation and translation of the tool about its own axis is also possible with the trocars assembly.

Figure 2.

Figure 2

Open in a new tab

(a) Tool interface assembly (b) Close-up of the tool handle with potentiometer

Three degree of freedom force feedback is provided with a maximum of 3.3 N. A BEI Duncan 0.5 inch linear potentiometer is connected to each tool (Figure 2 (b)) using a simple attachment that converts the angular motion of the tool handles to the linear motion of the potentiometer. This linear motion, interpreted as an analog voltage signal, is digitized and fed to the simulation software using ADU100 analog-to-digital converter form Ontrak® control systems. We take advantage of the tool-haptic device attachment to directly read the tool articulation from roll angles of the haptic device.

c) Experimental design

Subjects were voluntarily recruited for this Institutional Review Board (IRB) approved study at the Massachusetts General Hospital (MGH) ), Boston and at the Learning Center in the 2012 meeting of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) held in San Diego, CA. We have categorized subjects into two groups - expert and novice, based on their experience level. PGY 4-5, fellow and practicing surgeons were considered ‘experts’ and PGY 1-3 were under the ‘novices’ category.

Before the start of the experiment, subjects were asked to fill out a questionnaire detailing the demographics and previous laparoscopic surgery experience. The subjects were then shown an instructional video describing the peg transfer task. Subjects were given up to two practice trials to become familiar with the procedure and the system. They were then asked to perform one trial of the peg transfer task in the FLS tool box and in the VBLaST-PT© simulator. The order in which the simulators were presented to each subject was randomized. After the completion of the tasks the subjects were asked to answer a feedback questionnaire, consisting of nine questions in total (Table 1) comparing various aspects of the VBLaST-PT© with the FLS peg transfer task. The questions related to visual appearance, haptic feedback, 3D perception, tool movement and overall quality and reliability of the simulation as a training and assessment tool. Each question can be rated on a 5-point Likert scale with 5 representing very good/very satisfactory and 1 representing very poor/not satisfactory.

Table 1.

Feedback questionnaire

Question No.
1 Please rate the degree of realism of the target objects (how realistic they look) in the VBLaST task environment, compared to the corresponding task environment in FLS
2 Please rate the degree of realism of instrument handling (how realistic it feels) in the VBLaST, compared to that in the FLS
3 Please rate the degree of overall realism of the VBLaST simulation (how it looks and feels), compared to the corresponding FLS task.
4 Please rate the quality of the force feedback (sensation of feeling the tools on the target and in the task space) in the VBLaST compared to the FLS.
5 Please rate the degree of usefulness of the force feedback (sensation of feeling the tools on the target and in the task space) in the VBLaST in helping your performance.
6 Please rate the usefulness of VBLaST simulation in learning hand-eye coordination skills, compared to the FLS.
7 Please rate the usefulness of VBLaST simulation in learning ambidexterity skills, compared to the FLS.
8 Please rate the degree of overall usefulness of the VBLaST in learning the fundamental laparoscopic technical skills compared to the FLS.
9 Please rate your assessment of how trustworthy the VBLaST is to quantify accurate measures of performance.
Open in a new tab

For construct validation we have used metrics that are modified from the proprietary ones developed for the FLS [14] with approval from the SAGES FLS committee. The scoring metric is based on completion time and errors. While the formulas are the ones used in the FLS, a different normalization factor has been used in order to arrive at the final score. The normalization factor is chosen to be the best score obtained in the expert group.

In addition to the established metrics of error and time, we present studies involving the length of trajectory (LOT) that quantifies the amount of tool movement in terms of distance travelled by the tool tip in order to complete the task. The total length of hand movement for each hand was computed from the time at which the first peg was picked till the last peg was transferred and the average for the left and right hands was used to compute the LOT for each subject. The LOT was then analyzed for establishing differences between the two groups. IBM SPSS 18.0 was used for statistical analysis.

For the face validity, descriptive statistics for both the combined and the individual feedback scores for expert and novice groups were computed. A two-tailed Mann-Whitney U-test was used to differentiate the responses of the groups. For the completion time and LOT, a two-tailed Mann-Whitney U-test was used to differentiate the scores between the two groups because of non-normality of the data, which was tested using the Shapiro-Wilk test.

Results

The simulator runs on a desktop computer equipped with 2.66GHz Core2 Quad CPU supported by 2.75GB physical RAM and two NVIDIA GeForce 8800 GTX graphics cards. The simulation cycle runs at more than 1000Hz providing high quality force feedback. Figure 3 shows the setup of the simulator.

Figure 3.

Figure 3

Open in a new tab

(a) VBLaST-PT© simulator setup (b) subject performing on the VBLaST-PT© module

The breakdown of the subjects based on the experience level from the demographics data collected during the study is shown in Figure 4. Sixteen experts (PGY 4-5, fellows and practicing surgeons) and eighteen novices (PGY 1-3) participated in the present study. The results from the face validation study are shown in Table 2. The table shows the mean and standard deviation of the scores obtained from the feedback study for each of the nine questions. VBLaST-PT© received high mean scores from the feedback study. The Mann-Whitney U-test, comparing the difference of opinion between expert and novice groups did not show any significance for all the questions. This suggests that there was no difference of opinion between the two groups on all the questions. The lowest mean score received was 3.53, for the quality of instrument handling. The highest mean was 4.08 for the realism of rendering of the target objects. This asserts that the VBLAST-PT© simulator is satisfactory in all aspects of the simulation quality that were examined. There is also a tight agreement amongst the subjects which is evident from the low standard deviation. The maximum standard deviation is 1.05 which is reported for the quality of force feedback. The maximum score received for all the nine questions is 5.0. We computed the scores for the FLS and VBLaST-PT© for comparison. Figure 5 shows the box plot of the scores for novice and expert groups for both FLS and VBLaST-PT©. In the case of VBLaST, the scores are computed using the scoring system mentioned above. A clear difference in the scores is observed in both the plots. However, the scores for VBLaST-PT© are observed to be more spread out for both experts and novice compared to that on the FLS. A two-tailed Mann-Whitney U-test performed on the scores of VBLaST-PT© showed that there is a statistically significant (p=0.001) difference between the groups. FLS also showed a significant difference between the groups with p=0.008.

Figure 4.

Figure 4

Open in a new tab

Plot showing the distribution of subject distribution across experience levels.

Table 2.

Descriptive statistics obtained from the feedback study

Questionnaire Total Expert Novice Mann-Whitney U-test P-value
Mean SD Mean SD Mean SD
Realism in rendering
4.09 0.71 4.25 0.68 3.94 0.72 0.215
Realism in tool handling
3.53 0.71 3.5 0.63 3.55 0.78 0.756
Overall realism compared to FLS
3.65 0.77 3.75 0.58 3.55 0.92 0.617
Quality of force feedback
3.85 1.02 3.94 0.85 3.78 1.16 0.883
Usefulness of force feedback In task completion
4.32 0.88 4.56 0.51 4.11 1.08 0.237
Usefulness: Hand-eye co- ordination
3.91 0.75 4.0 0.73 3.83 0.78 0.505
Usefulness: Ambidexterity skills
3.97 0.83 4.06 0.77 3.89 0.90 0.608
Overall usefulness of VBLaST in training for FLS skill set
3.82 0.95 3.93 0.80 3.72 1.07 0.540
Trustworthiness of VBLaST
3.79 0.74 3.94 0.68 3.65 0.78 0.365
Open in a new tab

Figure 5.

Figure 5

Open in a new tab

Comparison of scores for (a) FLS and (b) VBLaST peg transfer for expert and novice groups. The outliers are shown in circles labeled by the subject number.

Figure 6 shows the results from the analysis of the completion time for both the groups as a box plot. The plots show visibly lower completion time for experts group on FLS and VBLaST-PT©. However the overall time taken is lower on FLS compared to VBLaST-PT© for both the groups. This may be attributed to the differences in handling the system interfaces and to the newness of the system. We expect this difference to decrease with repeated trials. A two-tailed Mann-Whitney U-test performed on the data showed that the completion time between the two groups was statistically significant with p=0.001 for VBLaST-PT© and p=0.02 for FLS. (see Table 3). The outliers shown in figures 5, 6 have not dropped any pegs (expect for outlier subject 33) but rather have recorded unusually high completition time compared to rest of the subjects in their respective skill group.

Figure 6.

Figure 6

Open in a new tab

Box plot comparing the task time (in sec) for experts and novice groups on (a) FLS and (b) VBLaST. The outliers are shown in circles labeled by the subject number.

Table 3.

Data of LOT and completion times for subjects on VBLaST-PT© simulator. LOT-L and LOT-R represents LOT for left and right hand respectively.

Evaluation Criteria Experts Novice Mann-Whitney U-test P-value
Mean SD Mean SD
Task time (sec) 166.68 32.57 227.53 90.52 0.002
LOT-L (cm) 64.59 13.05 83.20 28.50 0.002
LOT-R (cm) 63.67 16.92 85.73 32.59 0.003
LOT- Average (cm) 64.13 14.55 84.46 30.17 0.004
Open in a new tab

An advantage of the VBLaST-PT© is that metrics other than just error and time can be captured in a relatively straightforward manner. One such metric is the length of trajectory (LOT) as discussed above. The results from the analysis of the LOT and the two-tailed Mann-Whitney U test are shown in Table 3. Figure 7 shows the box plot for the LOT categorized with the skill level. A clear distinction between the groups can be observed from the plot. The novice group is shown to have more variability (indicated by the height of the box) in the LOT compared to the experts group. Results from the statistical analysis also show that both left and right LOT and the average LOT were significantly different (P<0.05).

Figure 7.

Figure 7

Open in a new tab

Box plot comparing the average LOT (in cm) for expert and novice groups on VBLaST-PT© simulator.

To see the correlation between the task competition time and the LOT, a scatter plot is provided in Figure 8. The data showed a good linear fit with r2=0.75. Spearman’s correlation analysis of the average LOT and competition time showed that they are highly correlated (ρ = 0.813, p < 0.001). A k-means analysis shows that most of the experts are concentrated at the lower left portion of the graph (blue ellipse) indicating that they take less time with minimal tool movements. On the other hand, the plot shows that the novice group is scattered over a wider portion of the graph indicating a large variation in the skill level among this group. There are novices whose LOT is higher compared to most experts while completing the task in less time. However with experience we expect (yet to be shown empirically) this group of novices to reduce excessive movements while completing the task faster and eventually becoming part of the expert cluster shown. This is expected since some novices may have better motor skills, due to experience with video games, for example [12].

Figure 8.

Figure 8

Open in a new tab

Scatter plot of completion time (in sec) versus average LOT (in cm) for experts and novice groups

Discussion

Development of skills for laparoscopic surgery requires special training and practice. FLS was developed for this purpose. It has been validated and is established as a standardized test for certification in laparoscopic surgery. On the other hand, with the improvement of computing hardware virtual simulators have become alternative training sources for surgeons especially in the minimally invasive surgery. Virtual reality trainers lend many advantages like elimination of animal cadavers or other costly training material during the training phase, training in special pathological cases, long-term cost effectiveness [13]. As a part of initial effort to to incorporate the well established FLS tasks into a virtual reality trainer we present the validation of VBLaST-PT© simulator.

Some examples of simulators that incorporated tasks similar to FLS include LAP Mentor (Simbionix USA, Cleveland, Ohio) [14], Lap-X [15] and LapVR (Immersion Medical, Gaithersburg, MD)[16][17]. However, such simulators must undergo validation before they can be used for performance evaluation. Commercial virtual simulators that involve FLS-like tasks are deficient as some of them, e.g., LapVR, do not have the exact FLS tasks while LAP-X has not been validated. LAP Mentor (Simbionix USA, Cleveland, Ohio) is an essential task module developed to train novice in basic tasks such as camera manipulation, hand-eye coordination, clipping, grasping, cutting and electrocautery. LAP Mentor provides tasks similar to FLS and have been validated recently [18]. However the tasks provided are part of the simulator which has expensive custom hardware and the scoring system adopted is not similar to that used in FLS.

In this paper we reported face and construct validation studies of a virtual peg transfer simulator, being developed to perform the FLS tasks. VBLaST-PT© was found to exhibit good face validity and was capable of differentiating among skill levels. In addition to time and error, we have introduced length of trajectory as an additional metric which may be automatically measured in a virtual simulator and provide an additional means of discriminating between skill levels.

LOT has been shown to indicate a strong correlation with the FLS scores [19]. LOT was also used as a metric to differentiate skill level by some researchers [16,17,18,20]. However, others propose modification of LOT metric. For example in [21] researchers showed that for simple goal oriented tasks LOT as a metric works only during the ‘seeking’ phase. They showed that there is no significant difference between skill groups for the ‘retraction’ phase of the task. Various other metrics like velocity profiles [22], force/torque signatures [23], jerk [24] can also be extracted to differentiate the skill level. One can also use a set of predefined signatures that are independent of the task to differentiate the skill level [25]. We intend to utilize the recorded position and velocity data to perform further analysis in order to find other signatures that differentiate novice from experts. A study characterizing the learning curve on the peg transfer simulator is under way [26]. On completion of the VBLaST, we plan to perform predictive validation to ensure that skills learned in the virtual environment translate to the operating room.

Acknowledgments

Funding: NIBIB/NIH grant # 5R01EB010037

The authors gratefully acknowledge the support of this study by NIH/NIBIB (Grant No. 5R01EB010037). We also thank Alex Derevianko of Massachusetts General Hospital (MGH) for helping us in conducting the experiments and Saurabh Dargar of Rensselaer Polytechnic Institute for helping us during the hardware design phase.

Footnotes

Disclosure

Venkata S Arikatla, Dr. Ganesh Sankaranarayanan, Woojin Ahn, Amine Chellali, John Hwabejire, Marc DeMoya, Steven Schwaitzberg, Daniel B. Jones, Suvranu De and Caroline GL have no conflicts of interest or financial ties to disclose.

References

  • 1.Dawson SL, Kaufman JA. The imperative for medical simulation. Proceedings of the IEEE. 1998;86(3):479–483. [Google Scholar]
  • 2.Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL. Development of a Model for Training and Evaluation of Laparoscopic Skills. The American Journal of Surgery. 1998;175(6):482–487. doi: 10.1016/s0002-9610(98)00080-4. [DOI] [PubMed] [Google Scholar]
  • 3.Dauster B, Steinberg AP, Vassiliou MC, Bergman S, Stanbridge DD, Feldman LS, Fried GM. Validity of the MISTELS simulator for laparoscopy training in urology. Journal of Endourology/Endourological Society. 2005;19(5):541–545. [Google Scholar]
  • 4.McCluney A, Vassiliou M, Kaneva P, Cao J, Stanbridge D, Feldman L, Fried G. FLS simulator performance predicts intraoperative laparoscopic skill. Surgical Endoscopy. 2007;21(11):1991–1995. doi: 10.1007/s00464-007-9451-1. [DOI] [PubMed] [Google Scholar]
  • 5.Liu A, Tendick F, Cleary K, Kaufmann C. A survey of surgical simulation: applications, technology, and education. Presence: Teleoperators and Virtual Environments. 2003;12:599–614. [Google Scholar]
  • 6.Satava RM. Historical Review of Surgical Simulation—A Personal Perspective. World Journal of Surgery. 2007;32(2):141–148. doi: 10.1007/s00268-007-9374-y. [DOI] [PubMed] [Google Scholar]
  • 7.Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual Reality Training Improves Operating Room Performance. Annals of Surgery. 2002;236(4):458–464. doi: 10.1097/00000658-200210000-00008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Jordan JA, Gallagher AG, McGuigan J, McClure N. Virtual reality training leads to faster adaptation to the novel psychomotor restrictions encountered by laparoscopic surgeons. Surgical Endoscopy. 2001;15(10):1080–1084. doi: 10.1007/s004640000374. [DOI] [PubMed] [Google Scholar]
  • 9.Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, Fried GM. Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surgical endoscopy. 2003;17(6):964–967. doi: 10.1007/s00464-002-8828-4. [DOI] [PubMed] [Google Scholar]
  • 10.Arikatla V, Ahn W, Sankaranarayanan G, De S. Towards Virtual FLS: Development of a Peg Transfer Simulator. International Journal of Medical Robotics and Computer Assisted Surgery (IJMRCAS) n.d doi: 10.1002/rcs.1534. (To be submitted) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Zilles CB, Salisbury JK. A constraint-based god-object method for haptic display. IEEE/RSJ International Conference on Intelligent Robots and Systems; 1995. pp. 146–151. [Google Scholar]
  • 12.Rosser JC, Lynch PJ, Cuddihy L, Gentile DA, Klonsky J, Merrell R. The impact of video games on training surgeons in the 21st century. Archives of surgery Chicago Ill 1960. 2007;142(2):181–6. doi: 10.1001/archsurg.142.2.181. [DOI] [PubMed] [Google Scholar]
  • 13.Sutherland LM, Middleton PF, Anthony A, Hamdorf J, Cregan P, Scott D, Maddern GJ. Surgical Simulation. Annals of Surgery. 2006;243(3):291–300. doi: 10.1097/01.sla.0000200839.93965.26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Zhang A, Hünerbein M, Dai Y, Schlag P, Beller S. Construct validity testing of a laparoscopic surgery simulator (Lap Mentor®) Surgical Endoscopy. 2008;22(6):1440–1444. doi: 10.1007/s00464-007-9625-x. [DOI] [PubMed] [Google Scholar]
  • 15.Epona Medical | LAP-X. n.d Retrieved May 13, 2012, from http://www.lapx.eu/en/lapx.html.
  • 16.Iwata N, Fujiwara M, Kodera Y, Tanaka C, Ohashi N, Nakayama G, Koike M, et al. Construct validity of the LapVR virtual-reality surgical simulator. Surgical Endoscopy. 2011;25(2):423–428. doi: 10.1007/s00464-010-1184-x. [DOI] [PubMed] [Google Scholar]
  • 17.Mansour S, Din N, Ratnasingham K, Irukulla S, Vasilikostas G, Reddy M, Wan A. Objective Assessment of the Core Laparoscopic Skills Course. Minimally Invasive Surgery. 2012;2012:1–4. doi: 10.1155/2012/379625. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Pitzul KB, Grantcharov TP, Okrainec A. Validation of three virtual reality Fundamentals of Laparoscopic Surgery (FLS) modules. Studies in health technology and informatics. 2012;173:349–355. [PubMed] [Google Scholar]
  • 19.Ritter E, Kindelan T, Michael C, Pimentel E, Bowyer M. Concurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS) Surgical Endoscopy. 2007;21(8):1441–1445. doi: 10.1007/s00464-007-9261-5. [DOI] [PubMed] [Google Scholar]
  • 20.Larsen CR, Grantcharov T, Aggarwal R, Tully A, Sørensen JL, Dalsgaard T, Ottesen B. Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surgical endoscopy. 2006;20(9):1460–1466. doi: 10.1007/s00464-005-0745-x. [DOI] [PubMed] [Google Scholar]
  • 21.Chmarra MK, Jansen FW, Grimbergen CA, Dankelman J. Retracting and seeking movements during laparoscopic goal-oriented movements. Is the shortest path length optimal? Surgical Endoscopy. 2008;22(4):943–949. doi: 10.1007/s00464-007-9526-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Verner L, Oleynikov D, Holtmann S, Haider H, Zhukov LDPDP. Measurements of the Level of Surgical Expertise Using Flight Path Analysis from da Vinci™. Robotic Surgical System; 2003. [PubMed] [Google Scholar]
  • 23.Rosen JDP, MacFarlane M, Richards C. Surgeon-Tool Force/Torque Signatures - Evaluation of Surgical Skills in Minimally Invasive Surgery. 1999. [PubMed] [Google Scholar]
  • 24.Hwang H, Lim J, Kinnaird C, Nagy A, Panton O, Hodgson A, Qayumi K. Correlating motor performance with surgical error in laparoscopic cholecystectomy. Surgical Endoscopy. 2006 doi: 10.1007/s00464-005-0370-8. [DOI] [PubMed] [Google Scholar]
  • 25.Kowalewski TM, Rosen J, Chang L, Sinanan MN, Hannaford B. Optimization of a vector quantization codebook for objective evaluation of surgical skill. Studies in health technology and informatics. 2004;98:174–179. [PubMed] [Google Scholar]
  • 26.Zhang L, Sankaranarayanan G, Arikatla V, Ahn W, Rideout J, Epstein S, Jones D, et al. Convergent Validation of a Virtual Basic Laparoscopic Skill Trainer (VBLaST) Surgical Endoscopy. n.d (submitted) [Google Scholar]

RESOURCES