Preliminary Face and Construct Validation Study of a Virtual Basic Laparoscopic Skill Trainer

Abstract

Background

The Virtual Basic Laparoscopic Skill Trainer (VBLaST^™) is a developing virtual-reality–based surgical skill training system that incorporates several of the tasks of the Fundamentals of Laparoscopic Surgery (FLS) training system. This study aimed to evaluate the face and construct validity of the VBLaST™ system.

Materials and Methods

Thirty-nine subjects were voluntarily recruited at the Beth Israel Deaconess Medical Center (Boston, MA) and classified into two groups: experts (PGY 5, fellow and practicing surgeons) and novice (PGY 1–4). They were then asked to perform three FLS tasks, consisting of peg transfer, pattern cutting, and endoloop, on both the VBLaST and FLS systems. The VBLaST performance scores were automatically computed, while the FLS scores were rated by a trained evaluator. Face validity was assessed using a 5-point Likert scale, varying from not realistic/useful (1) to very realistic/useful (5).

Results

Face-validity scores showed that the VBLaST system was significantly realistic in portraying the three FLS tasks (3.95 ± 0.909), as well as the reality in trocar placement and tool movements (3.67 ± 0.874). Construct-validity results show that VBLaST was able to differentiate between the expert and novice group (P = 0.015). However, of the two tasks used for evaluating VBLaST, only the peg-transfer task showed a significant difference between the expert and novice groups (P = 0.003). Spearman correlation coefficient analysis between the two scores showed significant correlation for the peg-transfer task (Spearman coefficient 0.364; P = 0.023).

Conclusions

VBLaST demonstrated significant face and construct validity. A further set of studies, involving improvement to the current VBLaST system, is needed to thoroughly demonstrate face and construct validity for all the tasks.

Introduction

Minimally invasive surgery (MIS) has many benefits, such as less postoperative pain, shorter hospitalization, and quicker return to normal functioning. MIS, however, requires surgeons and residents to be trained in a more sophisticated manner in performing these procedures. Laparoscopic training is increasingly becoming an important part of the simulation-based curriculum to train surgical residents at many institutions.¹ The Fundamentals of Laparoscopic Surgery (FLS) skill training toolbox, based on the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS), has been adopted by a joint committee of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) and the American College of Surgeons (ACS) as the standard to assess the proficiency of laparoscopic skills.² The FLS trainer box consists of five premanufactured tasks, including peg transfer, pattern cutting, loop ligation of a structure, and suturing. Though the trainer boxes are inexpensive, they are limited in the range of measures for objective evaluation. Even though the scores are rated by a trained examiner, they are open to subjective interpretations. The scores are not immediate, as they are sent off-site for final scoring. Last, tests are offered infrequently at annual meeting and testing centers exclusively.

Virtual-reality–based trainers offer attractive alternatives to inanimate trainers. They are capable of providing multidimensional objective performance measures.^3–8 Virtual-reality trainers have been shown to improve acquisition of skills as well as performance in the operating room.^9–17 The entire evaluation is processed by computer in the simulators, and results can be made available immediately. The Virtual Basic Laparoscopic Skill Trainer (VBLaST^™) is a virtual-reality–based system that is being developed to allow trainees to perform FLS tasks by computer.¹⁸ MISTELS have undergone extensive testing in proving face and construct validity. Therefore, it is essential that VBLaST undergoes similar rigorous validation before it can be widely used by the surgical community. With this goal in mind, the main aims of this study were: 1) to measure the face validity of VBLaST, as judged by both experts and nonexperts, and 2) to measure the construct validity of VBLaST in its ability to differentiate between novice and expert groups.

Materials and Methods

The VBLaST simulator consists of computational software to simulate FLS tasks and a physical interface to connect laparoscopic tools to the haptic device to move the tools for interaction and to provide force feedback to the user. The VBLaST is capable of recreating four FLS tasks, consisting of 1) peg transfer, 2) ligation loop, 3) pattern cutting, and 4) intracorporeal suturing. Details of the computational software aspects of VBLaST can be found in the work of Maciel et al.¹⁸ Three tasks (1–3) were selected for this particular study. Task 3 (pattern cutting) was only included to determine face validity, since the subjects were not able to complete the task in the prescribed time in VBLaST. Figure 1 shows the screenshot of the three tasks, along with the physical FLS equivalent.

FIG. 1.

The three FLS tasks in trainer box and VBLaST^™. (A) and (D) peg transfer; (B) and (E) endoloop; (C) and (F) pattern cutting.

The overall VBLaST system, with the physical interface, is shown in Figure 2. It consists of two replaceable instrumented tools¹⁹ connected to haptic interface devices (PHANToM Omni^®; SensAble Technologies, Inc., Woburn, MA,) placed in an adjustable frame. A transparent polyurethane sheet covered the top of the interface, where the trocars were placed through a hole or simulated port site. A set of platforms were used for subjects with different heights. The instrumented tools consisted of the handles of laparoscopic graspers, scissors, and an endoloop tool. The tools connected to the haptic interface device through a standard audio-jack connector. For loop ligation, a compact string pot attached to the PHANToM Omni (Fig. 3) provided sensor input to tighten the loop around the tube.

FIG. 2.

Overall VBLaST^™ setup with tool interface and display.

FIG. 3.

(A) Ligation loop interface attached to the (B) PHANToM Omni device.

The validation experiment involved subjects performing the three tasks on both VBLaST and FLS in order to compare the level of difference between the novices and experts. For FLS, a proctor was present to time the tasks and then score each subject. For VBLaST, the performance metrics were automatically calculated as soon the subject concluded the experiment. The performance metrics for VBLaST were based on FLS, with a normalized score for each task. Total score was then calculated from the normalized scores, which ranged from a minimum of 0 to a maximum of 100.

Subjects were voluntarily recruited for this Institutional Review Board (IRB)-approved study at the Beth Israel Deaconess Medical Center (Boston, MA). Before the start of the experiment, each subject was asked to fill out a brief questionnaire detailing demographics and previous laparoscopic experience. The subjects were classified into two groups of experts (PGY 5, fellow and practicing surgeons) and novice (PGY 1–4). The subjects were then shown an instructional video detailing three basic FLS tasks. They were asked to perform the three tasks on both VBLaST and the FLS trainer box. The order of the tasks was kept the same for all the subjects, but the sequence in which the two trainers were presented to each subject was random. After finishing the tasks, the subjects completed a questionnaire to rate subjectively the different features in VBLaST relative to the FLS trainer box. The appreciation was expressed in a 5-point Likert scale,²⁰ varying from not realistic/useful (1) to very realistic/useful (5). These scores were used to evaluate the face validity of VBLaST.

Not all of the enrolled subjects were included in the final analysis. There was one expert who refused to complete one of the tasks, stating his frustration with a temporary technical error. Also, there was one subject who did not perform laparoscopy routinely and who deviated very far from the norm. Further, although 50 subjects were enrolled, 11 subjects did not return to complete the testing due to their schedules.

Power analysis

Power analysis was performed using GPOWER software.²¹ Results from the validation of MISTELS²² were used as a guide in choosing the effect size for the difference between skill levels of juniors and the expert group. A total of 17 subjects were determined as sufficient for each group to detect the difference at 80% power for an effect size of 0.5 with α = 0.05 and β = 0.2.

Data analysis

SPSS 17.0 software (SPSS, Inc., Chicago, IL) was used to perform statistical analysis on the data. For face validity, descriptive statistics were obtained for both individual and expert and novice groups combined for all the five questions. A two-tailed Mann-Whitney exact U test was used to differentiate the response between the two groups. For construct validity, descriptive statistics were obtained from the normalized individual and total scores for both VBLaST and FLS. The data were tested for normality by using the Shapiro-Wilk's test available in SPSS 17.0. Both FLS and VBLaST scores were found to be significantly different from normal distribution. So, a two-tailed Mann-Whitney U test, the nonparametric equivalent of the independent samples t-test, were used to compare the difference between the novice and expert groups for both the VBLaST and FLS systems. Spearman correlation coefficient analysis was used to compare FLS and VBLaST scores.

Results

Demographics

There were 22 novices and 17 experts who were included in the analysis. Of the experts, the years of experience after completing residency training ranged from 0 to 22 years, with a total mean experience of 9.2 years. There were 27 (69.3%) men and 12 (30.7%) women who were full participants. Of the novices, 8 (36%) were female. Of the experts, 7 (41%) were women. Five subjects (12.8%) were left-hand dominant, but all were able to utilize the endoloop that was set up on the right-sided trocar.

Face validity

Individual and total face validity scores are shown in Table 1. Overall, the rating for Question 3 was the highest, with a mean score of 3.95 (79%) for the two groups combined and a mean score of 4.10 (82%) for the expert and 3.82 (76%) for the novice group, respectively. Question 4 got the next highest score, with a mean value of 3.67 (74%) for the two groups combined, followed by Question 1, with a mean score of 3.17 (64%) for the two groups combined. Questions 2 and 5 had the lowest ratings, with a mean value of 2.62 (53%) and 2.81 (56%) for the two groups combined. The Mann-Whitney U test, comparing the difference of opinion between expert and novice groups, showed high P-values (P > 0.3), suggesting that there was no difference of opinion between the two groups on all the questions.

Table 1.

Face Validity Scores and Mann-Whitney U Test Results

	Total		Expert		Novice
Questionnaire	Mean	SD	Mean	SD	Mean	SD	Mann-Whitney U test P-value
1. Degree of overall “realism” to movements of actual laparoscopic surgery	3.17	0.794	3.26	0.933	3.08	0.668	0.466
2. Quality of force feedback	2.62	0.882	2.57	0.768	2.65	0.982	0.883
3. Quality of images	3.95	0.909	4.10	0.809	3.82	0.984	0.370
4. “Realism” of equipment (“realism of laparoscopic instruments, trocar placement”)	3.67	0.874	3.57	0.961	3.73	0.810	0.530
5. Your trust in the ability of this device to quantify accurate measures of performance	2.81	0.862	2.78	0.917	2.83	0.834	0.758

SD, standard deviation.

Construct validity

Construct validity scores and the Mann-Whitney U test results are shown in Table 2. For VBLaST, the overall scores show that the experts were able to finish the tasks faster and with fewer errors, compared to the novice group. The peg-transfer task showed the largest difference between the two groups (P = 0.003). This level of difference was not significant for the ligation-loop task (P = 0.365). Overall, for the total combined score, the difference between the two groups was significant (P = 0.015).

Table 2.

Construct Validity Scores and Mann-Whitney U Test Results

	Expert		Novice
Scoring parameter	Mean	SD	Mean	SD	Mann-Whitney U test P-value
VBLaST
Total score	82.58	16.05	68.77	22.20	0.015
Peg transfer	78.07	19.31	57.10	29.00	0.003
Ligation loop	86.98	17.09	80.41	22.74	0.365
FLS
Total score	69.76	13.94	49.13	27.00	0.012
Peg transfer	80.10	14.26	68.85	17.80	0.049
Ligation loop	74.86	17.67	67.43	38.01	0.671
Pattern cutting	54.52	23.88	11.26	47.40	0.001

VBLaST, The Virtual Basic Laparoscopic Skill Trainer; FLS, the Fundamentals of Laparoscopic Surgery training system; SD, standard deviation.

As expected, the total score for FLS tasks showed a significant difference between the two groups (P = 0.012). Among the three tasks, the pattern-cutting task was the most significant (P = 0.001) to differentiate between the expert and novice groups for the sample size of this study. The peg-transfer task showed a difference between the two groups at P = 0.049. The ligation-loop task could not differentiate between the two groups (P = 0.671). Spearman correlation coefficient analysis of VBLaST and FLS scores show that there was a significant correlation between the FLS and the VBLaST peg-transfer tasks (Spearman coefficient 0.364; P = 0.023).

Discussion

This study compared the FLS in a box trainer to an equivalent virtual-reality trainer system with haptic feedback for the three tasks: peg transfer, pattern cutting, and endoloop. VBLaST is considered highly realistic in portraying the FLS tasks, based on the the scores (>70%) that users provided in answer to the questions related to the quality of images and realism of the interface. Moreover, the movements of the virtual instruments were also very realistic. The construct validity results showed that VBLaST was able to differentiate between the expert and novice groups. Of two tasks that were used for scoring, the peg-transfer task showed the most significant difference between the two groups. The significant correlation between the VBLaST and FLS peg-transfer tasks further reinforce that this task was best reproduced in the virtual-reality version. It is also evident that the experts were able to better adapt to the virtual-reality workspace range and motion and were able to finish the tasks much faster. For the peg-transfer task, the lack of bimanual force feedback, when transferring the pegs from one hand to the other, made it difficult for the subjects to determine the completion of the transfer. This resulted in few pegs being dropped during the transfer. For the ligation loop, larger compliance in the interface while moving the endoloop tool complicated the positioning of the loop. This source of error eventually made it impossible to detect any difference between the expert and novice groups for this task and also affected the overall total scores.

Conclusions

In the next phase of the research, force feedback will be added during the bimanual transfer of pegs. A better ligation-loop attachment mechanism will be developed for improved tracking while using the endoloop tool. The pattern-cutting task will also be improved to make it more realistic. The intracorporeal suturing tasks of the FLS will be included in the next phase. The subjective feedback obtained from this experiment will be used to further improve the performance of all the VBLaST tasks for the next study.

Acknowledgements

The authors gratefully acknowledge the support of the NIH/NIBIB through grant #R01 EB005807.

Disclosure Statement

No competing financial interests exist.