Head-to-Head Comparison of Three Virtual-Reality Robotic Surgery Simulators

Abstract

Background and Objectives:

There are several different commercially available virtual-reality robotic simulators, but very little comparative data. We compared the face and content validity of 3 robotic surgery simulators and their pricing and availability.

Methods:

Fifteen participants completed one task on each of the following: dV-Trainer (dVT; Mimic Technologies, Inc., Seattle, Washington, USA), da Vinci Skills Simulator (dVSS; Intuitive Surgical Inc., Sunnyvale, California, USA), and RobotiX Mentor (RM; 3D Systems, Rock Hill, South Carolina, USA). Participants completed previously validated face and content validity questionnaires and a demographics questionnaire. Statistical analysis was then performed on the scores.

Results:

Participants had a mean age of 29.6 (range, 25–41) years. Most were surgical trainees, having performed a mean of 8.6 robotic primary surgeries. For face validity, ANOVA showed a significant difference favoring the dVSS over the dVT (P = .001), and no significant difference between the RM, dVSS, and dVT. Content validity revealed similar results, with a significant difference between the dVSS and dVT (P = .021), a trend toward a difference between the RM and dVT (P = .092), and no difference between the dVSS and RM (P = .99).

Conclusion:

All simulators demonstrated evidence of face and content validity, with significantly higher scores for the dVSS; it is also the least costly ($80,000 for the simulator), although it is frequently unavailable because of intra-operative use. The dVT and RM have similar face and content validity, are slightly more expensive, and are readily available.

INTRODUCTION

As robotic surgery becomes a more common choice for operative procedures because of faster recovery times, lower complication rates, decreased blood loss, and recovery of site-specific functions, surgeons are tasked with improving and retaining skill without affecting patient safety.^1–6 Although medicine is moving away from intraoperative training, the high cost of the da Vinci Surgical System (Intuitive Surgical, Inc., Sunnyvale, CA), the most common commercially available robotic platform, impinges on their capacity for use outside of the operating room (OR). Considering the capacity for harm to the patient and several widely publicized malpractice suits involving the da Vinci, it is crucial that surgeons be adequately trained and maintain their skills.⁷

Virtual-reality (VR) simulators offer opportunities for surgical training outside of the OR as well as standardized skill evaluations.^8–10 As institutions begin instituting their own training and credentialing curriculums for robotic surgery or look to adopting multiplatform curricula such as the Fundamentals of Robotic Surgery program^10–13, we present a recently conducted head-to-head comparison of 3 VR robotic simulators with regard to face and content validity, as well as a differential cost analysis.

There are currently 5 VR simulators that are commercially available for robotic training.^14,15 This project focused on 3 that had already been validated. The dV Trainer (dVT) from Mimic Technologies, Inc. (Seattle, Washington, USA) and da Vinci Skills Simulator (dVSS) from Intuitive Surgical, Inc. (Sunnyvale, CA. USA) have both been the subject of many prior validation studies.^16–25 The RobotiX Mentor (RM) (3D Systems, Littleton, Colorado, USA) is the most recent commercially available VR simulator for robotic skills development and has shown evidence of face, content, and construct validity.²⁶ The other 2 commercially available simulators are the RoSS Robotic Surgery Simulator (Simulated Surgical Systems LLC, San Jose, California, USA) and the SimSurgery Educational Platform (SEP) (SimSurgery, Oslo, Norway). The SEP is predominantly distributed in Europe; in addition, there has been limited validation of both of these platforms.²⁷ The unavailability of the latter 2 in the United States restricts their use.

MATERIALS AND METHODS

Fifteen participants, ranging from senior medical students to experienced robotic systems surgeons, were recruited to participate in this observational, comparative study. Institutional Review Board deemed this project exempt, as was no intervention was involved and no identifiable personal information was gathered. The participants were asked to complete a general demographics questionnaire. They then completed 1 task on each of the 3 simulators. The task was of the participant's choosing and did not have to be consistent among the simulators. We chose this design to obtain a true gauge of the participants' opinions without the influence of mandated procedures. No time limits were set for the tasks and time spent on any particular simulator.

Following each of the tasks, the participants were asked to complete formerly validated questionnaires for evaluation of face and content validity (Figure 1).²⁸ Face validity is considered to be the degree of realism between the system and the demonstrated activity. Alternatively, content validity illustrates the efficacy of a system as an instructional apparatus.

Figure 1.

Face (top) and content (bottom) validation questionnaires.

Statistical analysis was performed by analysis of variance (ANOVA; SPSS version 19 software; IBM Analytics, Armonk, New York) of the scores obtained from the face and content questionnaires, with P < .05 indicating a significant difference. Descriptive analysis and means were performed for the demographics questionnaire when appropriate.

Commercial pricing for the different simulators was obtained by contacting representatives of each company via e-mail.

RESULTS

The 15 participants had an average age of 29.6 (range 25–41) years, and most were surgical residents with an average of 8.6 robotic primary cases (Table 1). All completed tasks on 3 robotic surgical simulators and completed a series questionnaires designed to illustrate face and content validity.

Table 1.

Characteristics of Surgical Patients

Characteristic	Data
Age (y)
Mean	29.6
Range	24–41
Gender (n)
Female	5
Male	10
Years of training (n)	3
Medical student level
Intern	4
PGY2	4
Senior resident	2
Attending	2
Specialty
General surgery	8
Urology	7
Gynecology	0
Average number of robotic cases (as primary surgeon)
Mean	7.53
Range	0–50
Average number of laparoscopic cases (as primary surgeon)
Mean	17.5
Range	0–100
Handedness
Left	2/15
Right	13/15
Musical experience
Yes (n/total)	8/15
Mean years played	12.6
Military deployment (n/total)	0/15
Mean days since last robotic case	168.5
No prior robotic experience (n/total)	9/15
Mean days since last simulation	8.2
No prior robotic simulation experience (n/total)	5/15

The average scores for the individual simulators' respective face and content validity questionnaires are as follows: dVSS, 27.2 /27.73; dVT, 21.4/23.33; and RM, 24.73/26.80. These means are based on total combined scores (range, 5–30), with higher scores equating to greater demonstrated validity (Table 2).

Table 2.

Questionnaire Scores

Robotic System	First Questionnaire Mean	Second Questionnaire Mean	SE First Questionnaire	SE Second Questionnaire
dVSS	27.2	27.733	0.812	0.771
dVT	21.4	23.33	1.341	1.53
RM	24.733	26.80	0.933	0.812

The face validation questionnaire showed the dVSS to be significantly more realistic when compared with dVT (P = .001). There was no difference between the dVSS and RM (P = .316). The RM showed a trend toward being more realistic in comparison to the dVT, although the difference was not significant (P = .092) (Table 3).

Table 3.

Questionnaire Comparisons

Simulator	Comparison	P
Face validation questionnaire
dVSS	dVT	.001
	RM	.316
dVT	dVSS	.001
	RM	.092
RM	dVSS	.316
	dVT	.001
Content validation questionnaire
dVSS	dVT	.021
	RM	1.00
dVT	dVSS	.021
	RM	.092
RM	dVSS	1.00
	dVT	.092

The content validation–based questionnaire produced similar results. The dVSS was seen to have significant training utility compared with the dVT (P = .021). The RM was seen to again have a trend toward significance (P = .092) compared the dVT. There was no difference in utility between the dVSS and RM (P = 1.0) (Figure 2; Table 3).

Figure 2.

Face and content total score comparisons.

The prices of the models ranged from $80,000 (for the da Vinci “backpack” alone) to $137,000 for the RM). Pricing also varies with the purchase of additional software with supplementary exercises for training (Figure 3).

Figure 3.

Cost comparisons.

DISCUSSION

The use of minimally invasive surgical techniques continues to grow; however, the rapid expansion of the minimally invasive approach leads to a dilemma for surgical training on this equipment. The increase in robot-assisted and other forms of minimally invasive surgery has resulted in a greater need for training options for both residents and surgeons who did not receive this training during their residency. There have been ongoing attempts to create a standardized training program; however, training has not been standardized across the different virtual reality simulators. This study's goals were to provide a direct head-to-head comparison of the 3 most common commercially available simulators.

The dVT was one of the first VR simulators commercially available for robotic training and is still one of the most predominant. The console is simulated as a fully adjustable stereoscopic viewer and cable-driven master controller gimbals. The software operates on an external computer used for selecting tasks, managing users and curricula, and observing. All exercises were simulated by Mimic and were graded as an overall percentage composed of individual exercise-specific score modules. In comparison, the dVSS, commonly referred to as the backpack, is an optional add-on for da Vinci Xi and Si systems that is attached to the surgeon's console. Most of the exercises are simulated by Mimic and are also found on the dV- T, although there is a set of additional suturing exercises created by 3D Systems. The RM is a stand-alone simulated console, with nonfixed master controllers and all software developed by 3D Systems.

All 3 simulators offer a variety of VR tasks, ranging from basic orientation for console operation to procedure-specific skills. The dVT and RM have full-length procedures for common robotic cases, such as hysterectomy or prostatectomy. All of these stimulators have been independently validated for face and content validity.^16,21,26 There have been comparisons between the dVT and dVSS, with the dVT showing slightly less overall performance; however, the difference was not directly quantified.²⁹ In this study, we saw that, overall, all 3 simulators showed evidence of face and content validity, but that the dVSS scored significantly higher than its counterparts on both measures. This result is intuitive, given that it incorporates the da Vinci surgeon's console as part of the trainer itself. There was no significant difference between the other 2 trainers, although there was a trend toward a preference for the RM, for both its realism and usefulness for training. Although it may be intuitive that the dVSS would outperform the other 2 simulators, there has been no confirmation in the data up to this point. In addition, it is important to demonstrate that other simulators (such as the RM) are similar in performance to the dVSS.

The other factor evaluated in this study is the role of cost. Besides avoiding injury to the patient or possible liability during the known learning curve for robotic surgery, a motivation is the need to reduce cost during this period. It has been estimated in previous studies that the cost of the average learning curve in terms of operating time exceeds $200,000.³⁰ This estimate does not include the price of the device itself or associated instruments and maintenance. It highlights the need for reliable simulators to help reduce this intraoperative learning curve, improving both patient safety and reducing cost.

The simulators themselves have various costs. The dVSS as a simulator is the least expensive at approximately $80,000; however this does not include the console itself (an additional $500,000 sunk cost), and the loss of OR time for use of the console (estimated at ∼$500/hour), unless an institution is willing to invest in a training-only console (which would decrease the price savings of the sunk cost of the console).²¹ However, as training is often scheduled after normal duty hours, there may not be a limiting factor for availability. The other trainers are similar in price at $110,000 for the dVT and $137,000 for the RM. Both the dVT and RM offer full-length procedural simulations, but the software still differs significantly. The dVT has Maestro AR technology available for partial nephrectomy, hysterectomy, and inguinal hernia repair, and both Xi- and Si-specific prostatectomy procedures. The choice-driven augmented reality interface is generated from virtual instruments overlaid onto intraoperative recordings. The RM modules include hysterectomy, prostatectomy, and lobectomy, each with segmented guidance.

Limitations of this study include the small sample size and the overall lesser experience of the cohort. The sample size limits the conclusions we can draw from the statistical trends, and it may be that, given a larger sample size, the trends established in the comparison of the RM and dVT would have been significant differences. This study, as opposed to previous studies, had a predominantly novice (defined variably in the literature based on previous robotic experience) experience level.^16,21,26 Face and content validity are applicable to nonexperts; however, it limits the generalizability of the findings. This group was excellent for evaluating the simulators for the use of training. Simulators are being developed specifically to decrease the learning curve effect during surgery on actual patients.⁶ In general, trainees will use simulators more frequently than experienced surgeons. The assessment of face validity and usefulness of the simulator can ensure their use.

Future studies will benefit from evaluating skills acquisition and error rates between the simulators, as such data may significantly impact an institution's decision to acquire a certain simulator.

CONCLUSION

The 3 most commercially available VR robotic simulators all showed statistically significant evidence of face and content validity. Although the questionnaire scores for the dVSS outperformed the dVT and RM, the dVSS was limited by availability. The dVT and RM have similar cost and availability, with no difference in preference between them. The training modules available on the different models may be a differentiating feature between these 2 trainers, and both could be considered viable options for institutions seeking robotic training devices.