Using virtual reality simulation to ensure laparoscopic camera navigation skills of new surgical assistants—a validation study

Abstract

Background

This study aimed to develop and validate a virtual reality (VR) simulation-based test for assessing laparoscopic camera navigation skills, using Messick’s contemporary framework to gather validity evidence and establish an evidence-based pass/fail standard.

Methods

The test was developed through consensus among surgical experts and included eight clinically relevant metrics. A total of 24 participants, comprising 12 novice medical students and 12 experienced surgical residents, were recruited.

Results

The test demonstrated high internal consistency reliability (Cronbach’s α = 0.88) and test–retest reliability (Pearson’s r = 0.84). Significant differences in performance were observed between novices (51.8 ± 13.9) and experienced participants (81.7 ± 6.0, p < 0.001), indicating strong discriminative ability.A pass/fail score of 70% was established using the contrasting groups’ method, with one novice passing and no experienced participants failing. Participants perceived the simulator as realistic and beneficial for skill improvement, though experienced participants were less inclined to use it for further practice.

Conclusions

The study concludes that the VR laparoscopic camera navigation test provides a valid and reliable tool for training and assessing surgical assistants, supporting its integration into mastery learning programs to ensure proficiency before clinical practice.

With the development of surgical technology, laparoscopy is not only applied to simple operations such as gallbladder resection and appendectomy, but also widely used in a variety of complex surgical operations, such as laparoscopic hepatectomy and laparoscopic colorectal surgery. Laparoscopic surgery seems to be associated with significantly faster recovery, shorter length of stay at hospital, reduced analgesic requirements, and even a modest survival advantage [1, 2]. However, the laparoscopic approach is also a very challenging operation as it frequently involves more than one abdominal quadrant, identification and transection of vascular structures, mobilization and resection of the bowel, retrieval of surgical specimens, and performing anastomoses. Therefore, laparoscopic surgery requires good cooperation of the whole surgical team. The laparoscopic assistant must provide the surgeon with a clear and sufficient overview of the operating field in order to ensure the quality of the whole surgical procedure and the safety of the patient. Unfortunately, the operating surgeon is often assisted by inexperienced operating assistants, such as medical students or junior residents, who are tasked with the important role of handling camera navigation. This task is complicated for novices as it requires specific psychomotor and visuospatial skills. Inappropriate handling of the camera results in poor visualization, which can lead to longer operating time [3], surgeon frustration, and compromise patient safety [4–6].

Traditionally, training of laparoscopic assistants is based on apprenticeship training on patients in the clinic, which will lead to a longer learning curve and can have a bad effect on the surgical quality, which is not conducive to patient safety [7]. Simulation-based basic laparoscopic training has been widely used and proven beneficial in training and assessment to prepare future surgeons prior to operating on patients [8]. Likewise, training of novices might result in better assistants who could facilitate superior procedures. How to optimize the training and ensure competency are problems that need to be solved. Given the different learning curve of each individual trainee, a fixed number of training sessions or a certain amount of training time will not ensure that all trainees reach proficiency. According to the mastery learning concept, all trainees must keep practicing until they reach a predetermined pass/fail level of competence. This concept improves the training quality and ensures basic competency, but a well-designed test with solid evidence of validity and a credible pass/fail standard is a pre-requisite for a successful mastery learning training program.

In this study, we aimed to develop the Laparoscopic Navigation Training Rating Scale and gather validity evidence using Messick’s contemporary framework [9, 10]. In addition, we aimed to establish an evidence-based pass/fail level as a standard of proficiency in order to allow mastery learning of laparoscopic camera navigation.

Methods

A single-center validation trial was conducted at the Simulation Center of the First Affiliated Hospital of Guangdong Pharmaceutical University. A simulation-based test of laparoscopic camera navigation skills was developed based on consensus in a group of surgical experts, and validity evidence for the test was gathered using Messick’s contemporary framework.

Development of the simulation-based test

An expert group consisting of one general surgeon, one urologist, one gynecologist, one thoracic surgeon, and a simulation expert tried all camera navigation skills modules (see Tab 1) on the LapSim simulator™ (Surgical Science, Gothenburg, Sweden; Software version: LapSim 2021.6; Hardware version: Haptic model) and reached consensus on which to include in the test. Finally, the camera navigation module under the "Basic Skills Short Course- Intermediate" directory was selected. The difficulty level was medium, the duration was 1 min, and the task objective was to control the camera navigation system to accurately locate 5 stones in the abdominal cavity within the prescribed time. Furthermore, they reviewed all automatic simulator metrics and identified which of these were clinically relevant. The final evaluation scale included eight metrics: missing percentage (%), offset (mm), instrument path length (m), instrument angle path (degrees), angle horizontal error (degrees), camera horizontal alignment (degrees), camera horizontal stability (degrees), and camera horizontal alignment stability (degrees). A realistic range for each metric was set by the above five experts to allow conversion into a 100-point score by the following formula: (worst realistic score—measured score)/(worst realistic score—best realistic score) * 100%. A composite score was then calculated by adding all eight converted scores using the following weighting: 60% for Missed Percentage Score, 10% for Offset Score, and 5% for each of the last six metrics. “The weighting was decided after careful considerations and discussions in the expert group. They agreed that visualizing all the correct areas is the most important task for an assistant in laparoscopy, i.e., low missing percentage was weighted very high (60%). A small offset (total 10%) and efficient camera movements (i.e., short path length and low angle path) (total 10%) are desirable but not crucial for a good surgical outcome, which is why they weighted lower. The same argument was used for perfect angling (total 10%) and stability (total 10%) of the camera. To ensure that a high score equals a good performance (with a low missing percentage, a small offset, a short instrument path length, etc.) we used the formula: Total score = 100—composite score.

Tab 1.

All camera navigation skills modules

Level I Directory	Level II Directory
Camera training	Gastrointestinal
	Gynecology
Camera & instrument navigation	Camera navigation
	Instrument navigation
Camera navigation Gastrointestinal-intermediate	Camera navigation L1 Camera navigation L2 Camera navigation L3 Camera navigation L4
Camera navigation Gastrointestinal-basic	Camera navigation L1 Camera navigation L2 Camera navigation L3
Basic skills long course—difficult	Camera navigation
Basic skills long course- intermediate	Camera navigation
Basic skills long course- basic	Camera navigation
Basic skills short course—difficult	Camera navigation
Basic skills short course- intermediate	Camera navigation
Basic skills short course- basic	Camera navigation

Participants

Medical students in the ninth or tenth semester from the First Clinical Medicine School of Guangdong Pharmaceutical University were recruited for the novice group. Students who had previously participated in laparoscopic training or had experience with laparoscopic surgery were excluded. Residents from the First Affiliated Hospital of Guangdong Pharmaceutical University who had assisted at more than 50 surgical procedures were recruited for the experienced group. All participants signed informed consent for the trial. Furthermore, they answered a questionnaire regarding demographics including age, gender, prior laparoscopic experience, and interest in surgery (1 = not at all interested—5 = very interested).

Gathering validity evidence using Messick’s contemporary framework

All participants performed the camera navigation test two times without receiving any feedback.

The contemporary framework by Messick [9, 10] including five sources of validity evidence (content, response process, internal structure, relationship with other variables, and consequences) was used to explore evidence of validity.

• Evidence for Content: The test was developed based on consensus in a group of four expert surgeons from four different specialties and an expert in medical simulation.

• Evidence for Response Process: All tests of the participants were conducted in a standardized manner without giving feedback. The outcomes were automatically generated simulator metrics which remove any risk of rater bias.

• Evidence for Internal Structure: Internal consistency reliability were calculated across the eight different metrics and test–retest reliability was explored by correlating the participants’ scores in the two tests.

• Evidence for relationship to other variables: The discriminative ability of the metrics and the total composite score were explored by comparing the performances of the two groups.

• Evidence for consequences: The contrasting groups’ method was used to establish a credible pass/fail-score for the total composite score [11].

Study procedure

To ensure a uniform response process, data collection was performed in a standardized manner. On the day of the study, all participants received a comprehensive explanation about the study and its goals and a detailed introduction to the use of the simulator. Familiarity with the simulator navigation application was achieved through a detailed demonstration of the operating procedures. Afterwards, the participants navigated the simulator-navigation test mode while being observed. Each person was tested twice in a row, and no test feedback was given in order to minimize the learning-by-testing effect. After performing the test, members of both groups completed a questionnaire related to the test procedure. The questionnaire was collected on a Likert 5-point scale from "1 = strongly disagree" to "5 = strongly agree".

Statistics

Internal consistency reliability was explored using Cronbach’s alpha across all metrics in each test. The test–retest reliability was explored using Pearson’s correlation to check to what degree the score on the first test was correlated to the score on the second test. Validity evidence for relationship to other variables was established by performing an independent samples t–test comparing the test scores of novices and experienced participants. The contrasting groups’ standard setting method was used to establish a pass/fail standard, and the consequences of this standard were explored by comparing the passing rates of the novices and the experienced using Fisher’s exact test. The Mann–Whitney U test was used to test the differences in the results of the questionnaires between the two groups. All statistical analyses were performed using IBM Statistics for Social Sciences (SPSS) version 26. P-values < 0.05 were considered statistically significant.

Results

Participants

A total of 24 participants, 12 medical students (10 males and 2 females) and 12 experienced surgeons (9 males and 3 females) voluntarily joined the study. There were no dropouts. Residents were from General Surgery, Hepatobiliary or Gastrointestinal (n = 7), Urology (n = 2), and Obstetrics and Gynecology (n = 3). There was no significant difference in gender distribution between the two groups (p = 0.62), but their average age was 23.3 ± 0.5 for novices and 35.9 ± 2.8 for experienced (p < 0.001), showing an expected difference in age between the two groups.

Validity evidence

Analyses regarding internal structure showed that the internal consistency reliability analysis had a Cronbach alpha of 0.88 and a test–retest reliability (Pearson’s r) of 0.84.

Validity evidence for relationship to other variables was established as the average test scores ± SD for the novice and experienced groups were 51.8 ± 13.9 and 81.7 ± 6.0, respectively: p < 0.001 (Fig. 1).

Fig. 1.

The scores of the two groups. The dashed line represents the pass/fail score. Median, minimum, and maximum scores are shown. A significant difference was observed between the novice and the experienced group

Using the contrasting groups’ method, the pass/fail score was calculated to be 70% (Fig. 2). The consequences of this standard were that one novice passed the test (i.e., one false positive) and no experienced failed the test (i.e., no false negatives).

Fig. 2.

Contrasting groups’ method: The intercept of the normal distribution curve of the novice and experienced group (70%) was considered as the pass/fail score

Discussion

Content

The metrics of the laparoscopic simulator used in our study have been explored and used in many international studies regarding surgical competence in several surgical specialties [12–14]. However, validity evidence for the metrics regarding surgical assistants is lacking. Before the experiment, the clinically relevant metrics were jointly selected by four different surgical specialists from Guangzhou (General Surgery, Gynecology, Urology, and Thoracic Surgery) and an expert in medical simulation. Therefore, this scoring table is not only widely applicable to most regions around the world, but also applicable to the surgical specialties that operate in the abdominal cavity. At the same time, in the survey questionnaire, the two groups of participants believed that the test could reflect their true clinical operational level, which counts as additional validity evidence towards content [15]. The validity evidence regarding content is further supported by the fact that the metrics selected for our study align quite well with the outcome parameters chosen by other studies assessing the performance of laparoscopic assistants. The structured assessment of laparoscopic camera navigation skills (SALAS) score also focuses on how well the field is visualized and on correct centering and optimal angling of the scope [16]. The same skills are also covered by a low-cost laparoscopic camera navigational maze that has demonstrated evidence of validity [17].

Response process

Traditional or clinical evaluations require designing a fair approach. For example, in order to reduce bias, it is necessary to hire examiners who have undergone systematic training and conduct blinding. However, the training of raters proposed in most medical education literature is heterogeneous and limited [18–20]. Differences among raters lead to bias in the response process, e.g., if the blinding method fails, the raters may be more inclined towards the expert group [19]. The automatically generated metrics will not be biased towards any user [21] and the virtual reality simulator does not provide any verbal feedback during the procedure, which also minimizes the risk of bias in the entire testing process. To further reduce bias, we provided uniform familiarization for experienced and novices alike with a standardized on-site demonstration that introduced the operating process and precautions before the tests were conducted.

Internal structure

The internal consistency reliability of the evaluation of the camera navigation system in this computer simulator was high, Cronbach α = 0.88. Melchiors et al. argue that Cronbach α is a key factor in evaluating internal consistency and that α at least 0.8 is required in scales, while a Cronbach α exceeding 0.9 indicates item redundancy [22]. The complexity of actual clinical cases may have an impact on the testing results of physicians, and computer simulation evaluation systems can not only simulate various typical cases, but most importantly, the difficulty and process of cases are consistent, which helps to improve the consistency of testing [23].The test–retest reliability of this study is 0.84, which is greater than 0.8, indicating that the camera navigation test is reliable enough to be used for summative assessment.

Relationship to other variables

Our study found that there was a statistically significant difference in scores between the novice group and the experienced group (novice group 51.8 ± 13.9, experienced group 81.7 ± 6.0, p < 0.001) showing that the test could discriminate between novice and experienced participants. At the same time, it was found that there was a big variance in the scores among the novice group, which is consistent with previous literature reports [24]. The reason for this situation may be related to the differences in medical basic knowledge and operating room experience among students, as well as some students being more interested in surgery [23, 24]. The situation of these novice groups can also be classified as being in the cognitive stage of Fitts and Posner’s learning theory, defined by their inconsistent actual performance, so it can be expected that they will score lower and test scores will vary significantly in simulation-based evaluations [25]. The scores of the experienced group are relatively homogenous, which is related to their receiving unified formal training and reducing the differences between them. At the same time, it can be seen from the questionnaire survey results that the experienced participants believe that they do not need to continue using simulators for training, which may be related to their learning curve tending to be stable and the marginal effect of continuing to practice being small [26].

Consequences

Based on the intersection of two normal curves representing beginners and experienced individuals, our newly discovered pass/fail score is 70 points. Due to laparoscopic navigation being a fundamental skill for surgeons in minimally invasive surgery, medical students must acquire this skill as early as possible. One novice passed the test, which may be related to the novice’s great interest in laparoscopy and strong personal practical ability [15]. All experienced individuals passed the test (no false negatives) and this indicates that the test could be used in future mastery learning training of surgical assistants.

Strengths and limitations

The LapSim laparoscopic simulator has accumulated substantial evidence supporting the validity of its use in surgical training and assessment [27]. Spiliotis et al. proposed that surgeons who have undergone simulation training can perform surgeries faster, with higher accuracy, and a lower proportion of intraoperative errors or postoperative complications. Surgical skills obtained through simulation training have strong translatability in the operating room [28]. However, we must acknowledge that there is still a gap between clinical practice and the simulator [29]. Passing our simulator test indicates that the learner is ready to enter the next stage of education where he/she needs continued guidance and training in clinical practice cases to adapt to the transition from simulators to clinical cases. We used a contemporary framework of validity and gathered validity evidence from all five sources but acknowledge that future studies should focus on the transfer of skills to real operating scenarios. Furthermore, our single-center design and relatively small number of participants is a limitation even though our study was big enough to generate significant results and had a size quite typical for studies published in health professions education research [30]. The single-center design also meant that the gender distribution of our participants was typical for China where most medical students and residents are males. This is different in other countries, but we still believe that our results are generalizable as any gender differences in surgical trainees seem to be evened out during training [31].

Future perspectives

Laparoscopic camera simulation navigation is particularly suitable for integration into early simulation courses for young surgical assistants with very limited skills. Simulators allow training outside of the operating room and independent of patients, and are morally and economically meaningful [32]. Learners who consciously practice the program without achieving pass/fail scores in a simulated environment should undergo a review on how to execute the program and what tools to use before learning and practicing psychomotor skills in a safe environment. This will ensure that trainees have appropriate knowledge before participating in the simulation course, avoiding wasting the time and resources of trainees and mentors. This means that the laparoscopic camera navigation test mode can help beginners quickly acquire basic technical skills and reach the plateau stage of their learning curve. To ease implementation, we suggest building in the scoring system of this study as a composite score in the simulator for future training.

Conclusion

In summary, the virtual reality laparoscopic camera navigation system provides a safer, more efficient, and ethical approach for medical students before entering clinical practice. The validity evidence collected in this study provides a robust validity argument for the laparoscopic camera navigation simulation test. This test can be used in future mastery learning training programs to provide feedback to trainees and ensure the end-of-training proficiency.

Declarations

Conflict of interest

Yu Jiongbiao, Yang Yunran, Tang Lidong, Wang Wentao, Ma Jinhuo, Li Xiaowu, Zheng Wang, Lars Konge, Liu Wei have no conflicts of interest or financial ties to disclose.