Face, content and construct validity of a virtual reality simulator for robotic surgery (SEP Robot)

Abstract

INTRODUCTION

This study aims to establish face, content and construct validation of the SEP Robot (SimSurgery, Oslo, Norway) in order to determine its value as a training tool.

SUBJECTS AND METHODS

The tasks used in the validation of this simulator were arrow manipulation and performing a surgeon's knot. Thirty participants (18 novices, 12 experts) completed the procedures.

RESULTS

The simulator was able to differentiate between experts and novices in several respects. The novice group required more time to complete the tasks than the expert group, especially suturing. During the surgeon's knot exercise, experts significantly outperformed novices in maximum tightening stretch, instruments dropped, maximum winding stretch and tool collisions in addition to total task time. A trend was found towards the use of less force by the more experienced participants.

CONCLUSIONS

The SEP robotic simulator has demonstrated face, content and construct validity as a virtual reality simulator for robotic surgery. With steady increase in adoption of robotic surgery world-wide, this simulator may prove to be a valuable adjunct to clinical mentorship.

Introduction

Surgery is a profession that relies on a surgeon's experience in previous cases and hands-on experience with the appropriate tools. With the number of training hours for surgical trainees decreasing, simulation offers an opportunity to fill some of the gaps.^1,2 Surgical simulators also allow the trainee to develop their psychomotor skills and practice to the required skill level without the risk of danger to patients.^3,4 Simulation is an established component of laparoscopic training and may prove to be a valuable tool in robotic surgery. Several forms of simulation exist such as mechanical trainers, animal models and virtual reality (VR).⁵ VR simulators may prove to be more useful as they provide statistical feedback of performance to the surgeon.

To ensure that VR simulators provide a realistic comparison to real-life surgery, they must undergo scientific validation.^6,7 This study aims to establish face, content and construct validation of the SEP Robot (SimSurgery, Oslo, Norway) in order to determine its value as a training tool.

Subjects and Methods

System

The SEP Robot is a VR robotic simulator (Fig. 1) with a console connected to two instruments with 7° of freedom (DOF) as in clinical robotic surgery. The master console contains a motion tracking device, which detects the position of the controllers in space and recreates these movements onto a computer screen. Like the da Vinci robot (Intuitive Surgical, Sunnyvale, CA, USA) the simulator does not provide haptic feedback. However, unlike the da Vinci system, the images are not three dimensional (3D). The simulator allows the trainee to perform tasks, such as arrow manipulation and suturing. The simulator records time taken, instrument tip trajectory and error scores for each task.

Figure 1.

The SEP Robot, virtual reality simulator.

Subjects and protocol

The tasks used in the validation of this simulator were arrow manipulation and performing a surgeon's knot (Fig. 2). In the arrow manipulation task, the volunteer must grab the balls on either end of the arrow and reposition it as the computer demonstrates without over stretching the arrow. Exercise time and incidences of arrow drop (arrow overstretch leading to the resetting of the original arrow position) were evaluated. This exercise assesses hand-eye co-ordination and the ability to move objects using robotic tools. The second exercise consisted of performing two knots consecutively and alternating the arm handing the needle. Time, equipment drop (thread drop and grab point drop) and a percentage of the maximum stretch of the tightening and winding steps were evaluated for this exercise.

Figure 2.

(A) Task 1: place an arrow; (B) Task 2: surgeon’s knot.

Thirty participants (18 novices and 12 experts) received a demonstration of each task and then attempted it themselves. All participants then filled out a questionnaire to assess face and content validity (Appendix 1, online only). The questionnaire consisted of 19 questions predominantly scored on yes/no answers and a 5-point Likert scale: 5 (excellent/very useful) to 1 (very bad/useless). Construct validity was based on the performance data (metrics) recorded by the simulator.

Statistical analysis

All data were entered into a Microsoft Excel 2003 (Microsoft Corporation, Redmond, WA, USA) spreadsheet. Construct validity was analyzed by comparing the mean performance variables for the two groups (novice and experts), using a two-tailed independent samples t-test. Statistics were calculated using the Statistical Package for Social Sciences, v15.0 (SPSS, Chicago, IL, USA).

Results

Participants

Thirty participants (18 novices, 12 experts) completed the procedures. The novice group consisted of 3rd and 4th year students in the School of Medicine at King's College London. The experienced group consisted of six senior residents and six consultants in urology. The median age of the 12 experienced surgeons was 41 years (range, 32-54 years), whereas the median age of the 18 novices was 22 years (range, 21-23 years). There was no difference between novices and experts regarding sex and handedness. The novices had no experience in robotic surgery, while experienced subjects had performed an average of 148 robotic cases (range, 30-500). No subject had previous experience with the SEP Robot simulator.

Face and content validity

All 30 participants filled out the questionnaire (Appendix 1, online only). Usefulness of tasks, features, and movement realism were scored between a mean value of 3.8 for depth perception and 3.6 for appreciation of training with the instrument. There were no significant differences between the mean values of the scores given by the experts and surgical trainees (mean Likert scores 3.5 and 3.7; Fig. 3).

Figure 3.

Face and content validity.

Participants rated the graphics of the trainer as 3.7 on a Likert-scale. Of participants, 90% rated the trainer realistic and easy to use, 87% considered it generally useful for training and 90% agreed that the simulator was useful for hand-eye co-ordination and suturing.

Construct validity

The simulator was able to differentiate between experts and novices in several respects (Table 1). All the parameters that showed a significant difference between the two groups are shown in Figure 4. The novice group required more time to complete the tasks than the expert group, especially suturing (mean time 78.8 s for experts versus 198.3 s for novices; P = 0.001).

Table 1.

Performance data

Task	Performance evaluation	Novices (n = 18)	Experts (n = 12)	P-value
Place an arrow
Time to accomplish task(s)
	Mean	86.94	57.42	0.03
	SEM	10.69	1.45
	Dropped arrows
	Mean	1.94	0.33	0.005
	SEM	0.416	0.188
Lost arrows
	Mean	0.222	0	0.3
	SEM	0.17	-
Tool collision sum
	Mean	0	0	NS
	SEM	-	-
Close entry sum
	Mean	3.28	0.42	0.02
	SEM	0.946	0.26
Surgeon's knot
Time to accomplish task(s)
	Mean	198.33	78.83	0.000
	SEM	22.17	9.51
Max tightening stretch
	Mean	164.81	72.62	0.002
	SEM	21.064	6.56
Equipment dropped sum
	Mean	2.33	0.33	0.000
	SEM	0.31	0.14
Max winding stretch
	Mean	229.97	85.28	0.027
	SEM	50.44	3.57
Tool collision sum
	Mean	13.56	4.42	0.001
	SEM	1.83	0.9

Figure 4.

Comparison of novices and experts performance in the two tasks.

During surgeon's knot, experts significantly outperformed novices in maximum tightening stretch, instruments dropped, maximum winding stretch and tool collisions in addition to total task time. A trend was found towards the use of less force by the more experienced participants.

Discussion

Technological advances such as robotics and VR siulators will change our surgical practice and training in the future. With ever increasing pressures on surgical performance, the profession is constantly looking for training systems that are novel, reproducible and validated. With surgical trainees operating less¹ than ever before due to shortened training programmes, reduced working hours² and the advancement of medical and minimally invasive therapies, the application of VR technology is evident.

For more than a decade, advancing computer technologies have incorporated VR into surgical training.⁵ This has become especially important in training for minimal invasive procedures, which often are complex and leave little room for error. With the advent of robotic surgery, a valid VR robotic surgery simulator may reduce the learning curve and improve patient safety.^8,9

VR simulators afford a unique opportunity for trainees to practice reality-based surgical skills without any risk to the patients. VR laparoscopic surgical simulators have shown great promise in several areas of surgical training, and repeated practice with some VR simulators has shown improved performance in the operating room in prospective studies.^10-14 The principles of evaluating surgical simulators are well established. Common benchmarks on which simulators arejudged include reliability, face, content, construct, concurrent and predictive validities.⁶ Face validity is an assessment of virtual realism by any user. Content validity is defined as ‘an estimate of the validity of a testing instrument based on detailed examination of the contents of the test items'. The evaluation is carried out by reviewing each item to determine whether it is appropriate for the test and by assessing the overall cohesiveness of the items, such as whether the test contains the steps and skills that are used in a procedure. Establishing content validity is also largely subjective and relies on the judgments of experts about the relevance of the materials used.

Construct validity is regarded as one of the most important aspects of simulator evaluation, because it determines whether the device can differentiate between expert and novice. For a simulator to have construct validity, experts must outperform novices during standardised simulated tasks. Construct validity is defined as ‘a set of procedures for evaluating a testing instrument based on the degree to which the test items identify the quality, ability, or trait it was designed to measure'.

While there are some validated training devices and protocols for standard laparoscopy, the area of robot-assisted surgical training has few well established or validated tools.^15-17 To date, a few papers have shown validity of VR simulators for robotic surgery.^17-23 The first such article appeared in 2008. In a randomised, blinded, pilot study, the validity of a da Vinci robotic VR simulator platform was tested during a paediatric robotic surgery course at the annual American Urological Association meeting in 2007. Participants performed robotic skills tasks on the da Vinci robot and on an offline dV-Trainer. Most felt that VR simulation was useful for teaching robotic skills and the offline trainer was able to discriminate between experts and novices.²³

Our study has shown face, content and construct validity of the SEP Robotic simulator for two specific tasks. Data revealed that the simulator can discriminate novices from experts. For each task, mean procedure time defined skill levels, with experts taking significantly less time than novices. Moreover, the experts showed fewer errors compared with novices in the two tasks and in addition decreased tendency to use unnecessary movements.

There is, however, room for improvement. The tasks assessed here are generic rather than specific to complex procedures such as robotic-assisted radical prostatectomy or robotic hysterectomy. As VR robotic simulators improve, it is expected that they will become more procedure specific. The da Vinci Si system has the ability to be linked with the MIMIC VR robotic simulator²¹ across a computerised interface and is a precursor to more detailed procedural tasks. This new system was unveiled at the World Robotics Symposium in Florida in 2010. Likewise, the RoSS simulator (Roswell Park, NY, USA) provides a new platform for practising robotic surgery²² without causing harm to patients.

Our results in 30 participants need to be validated in larger participant groups. While these early results are encouraging, it remains to be seen as to whether this simulator can achieve predictive validity in a randomised trial.

The findings of this study contribute to the underlying knowledge of how simulators can support surgical training and suggest optimal prospective measures for incorporating the SEP robot into surgical training curricula. It is now an important part of robotic training within the Simulation and Technology Enhanced Learning Initiative (STeLi) of the London Deanery.

Conclusions

The SEP robotic simulator has demonstrated face, content and construct validity as a VR simulator for robotic surgery. With steady increase in adoption of robotic surgery worldwide, this simulator may prove to be a valuable adjunct to clinical mentorship.