Effect of head-mounted displays on students’ acquisition of surgical suturing techniques compared to an e-learning and tutor-led course: a randomized controlled trial

Abstract

Background:

Although surgical suturing is one of the most important basic skills, many medical school graduates do not acquire sufficient knowledge of it due to its lack of integration into the curriculum or a shortage of tutors. E-learning approaches attempt to address this issue but still rely on the involvement of tutors. Furthermore, the learning experience and visual-spatial ability appear to play a critical role in surgical skill acquisition. Virtual reality head-mounted displays (HMDs) could address this, but the benefits of immersive and stereoscopic learning of surgical suturing techniques are still unclear.

Material and methods:

In this multi-arm randomized controlled trial, 150 novices participated. Three teaching modalities were compared: an e-learning course (monoscopic), an HMD-based course (stereoscopic, immersive), both self-directed and a tutor-led course with feedback. Suturing performance was recorded by video camera both before and after course participation (>26 h of video material) and assessed in a blinded fashion using the Objective Structured Assessment of Technical Skills (OSATS) Global Rating Score (GRS). Furthermore, the optical flow of the videos was determined using an algorithm. The number of sutures performed was counted, the visual-spatial ability was measured with the Mental Rotation Test (MRT), and courses were assessed with questionnaires.

Results:

Students’ self-assessment in the HMD-based course was comparable to that of the tutor-led course and significantly better than in the e-learning course (P=0.003). Course suitability was rated best for the tutor-led course (x̄=4.8), followed by the HMD-based (x̄=3.6) and e-learning (x̄=2.5) courses. The median ΔGRS between courses was comparable (P=0.15) at 12.4 (95% CI 10.0–12.7) for the e-learning course, 14.1 (95% CI 13.0–15.0) for the HMD-based course, and 12.7 (95% CI 10.3–14.2) for the tutor-led course. However, the ΔGRS was significantly correlated with the number of sutures performed during the training session (P=0.002), but not with visual-spatial ability (P=0.615). Optical flow (R ²=0.15, P<0.001) and the number of sutures performed (R ²=0.73, P<0.001) can be used as additional measures to GRS.

Conclusion:

The use of HMDs with stereoscopic and immersive video provides advantages in the learning experience and should be preferred over a traditional web application for e-learning. Contrary to expectations, feedback is not necessary for novices to achieve a sufficient level in suturing; only the number of surgical sutures performed during training is a good determinant of competence improvement. Nevertheless, feedback still enhances the learning experience. Therefore, automated assessment as an alternative feedback approach could further improve self-directed learning modalities. As a next step, the data from this study could be used to develop such automated AI-based assessments.

Introduction

Learning to perform surgical sutures is one of the most important basic manual skills for prospective surgeons, as well as for physicians and dentists who perform surgical procedures^1,2. Unfortunately, many medical school graduates do not acquire adequate surgical skills during their studies^1,3,4. Furthermore, not all medical schools integrate suturing into their curriculum⁵. Lack of skills can lead to both anxiety and incompetence and thus potential deficiencies in surgical suturing quality and subsequent patient care^1,6. One of the main reasons for unsatisfactory student evaluation seems to be the discrepancy between the number of students and the availability of tutors⁷. To ensure optimal teaching, a ratio of four students to one tutor is recommended¹. However, this may pose major challenges to the capacity of teaching staff⁷, who have been further burdened by possible restrictions due to the coronavirus disease 2019 (COVID-19) pandemic^8,9. Although there exist e-learning/computer-based programs, which could help to teach surgical suturing techniques in a time-independent and self-directed manner. Such programs still rely on an instructor to provide external feedback to students, thereby also encroaching on the capacity of the teaching staff and making the programs not truly self-directed^8,10–12.

Aside from manual dexterity^13,14, the learning experience¹⁵ and visual-spatial ability appear to play crucial roles in the acquisition of surgical skills^16,17. The learning experience results from the totality of influences that affect the trainee during the learning process, such as individualization and the quality of the application or the stimulation of memorization¹⁸. Additionally, visual-spatial ability, which describes the capability of an individual to perceive two-dimensional objects in three dimensions and to classify their position in space, may play a role¹⁷. In this regard, previous studies have shown that individuals with low visual-spatial ability perform worse in acquiring surgical skills than individuals with high visual-spatial ability^17,19.

The use of virtual reality (VR) in medical education has shown great promise²⁰. Different systems are summarized under the term ‘VR’, such as simple (hand-based or screen-based) or enhanced systems (partially or fully immersive). Partially immersive systems include wall projectors, ImmersaDesks, and monocular head-based systems, while fully immersive systems include space-based (vehicle simulation or cave automatic virtual environments) and binocular head-based systems²¹. In recent years, binocular head-mounted display (HMD) VR systems have become popular and widely used in the consumer entertainment market. They could also be used to deliver high-quality learning experiences and to advance the visual-spatial skills of a wide audience by showing immersive, stereoscopic content from a first-person perspective^22,24.

Immersive 360° videos displayed using VR-HMDs without stereoscopy have already shown an advantage in acquiring knot-tying skills²⁵. In laparoscopic surgery, stereoscopic video systems, compared to non-stereoscopic applications, have been found to enable higher-precision performance and an increase in the speed of execution²⁶. Furthermore, stereoscopic learning applications have positive effects in terms of the localization and identification of different structures, such as abdominal anatomical structures, using 3D presentation²⁷. This highlights the potential of intensifying learning through HMDs¹⁵ and the impact of visual-spatial ability in teaching surgical skills and knowledge^17,28,29. In contrast to VR-HMDs, applications enabled by augmented reality (AR) HMDs have already been demonstrated. AR must be distinguished from VR, which extends reality through holography. One randomized controlled trial (RCT) (n=14) was able to show that the additional use of monoscopic AR-HMDs (Google Glass) led to more enjoyment in tutor-led teaching of surgical suture techniques³⁰. Recently, another RCT (n=38) for self-directed training with an AR-HMD (HoloLens 2) showed its comparability to a video course³¹.

However, among the computer-based approaches, to our knowledge, there has been no investigation into the impact of VR-HMDs in teaching surgical suturing and its learning effects using immersive and stereoscopic learning content recorded from a first-person perspective. Therefore, we compared an HMD-based course (no feedback, stereoscopic) with e-learning (no feedback, mesoscopic) and tutor-led (feedback, stereoscopic) courses, respectively, in a multi-arm RCT. The primary endpoint was an improvement in suture quality (learning effect) among novices. The secondary endpoints were student self-assessed improvement, the correlation between improvement in suture quality and visual-spatial ability, and the number of sutures performed during training.

Material and methods

In total, 160 medical and dental students enrolled in this RCT’s courses. Three students did not attend the course despite enrollment. Prior training in a surgical suturing course was denied by all attendees; however, 7 students had to be excluded because they were not novices. The RCT was conducted from March to November 2022, including recruitment and follow-up at a single center (University Hospital RWTH Aachen). Ultimately, the final study cohort included 150 novices with no adverse events. No other groups were added or dropped.

The study was authorized by the local ethics committee of the University Hospital RWTH Aachen (approval code EK 352/21 and EK 22-329) and was registered, including the study protocol, in the German Clinical Trials Register (DRKS00029307). The reporting of the study followed the extension of the CONSORT 2010 statement for reporting multi-arm parallel-group randomized trials³² (Fig. 1).

Figure 1.

CONSORT flow diagram.

Course and evaluation

The cohort was stratified by study program (medicine or dentistry) and randomly divided into three trial arms by block randomization (block sizes of 3, 6, and 9). The random assignment sequence was generated by a statistician. Recruitment was done by P.P., and assignment to the intervention groups was done by B.P. There were no changes in the allocation of the groups.

After course assignment and obtaining consent to participate, the visual-spatial ability of the subjects was measured with a Mental Rotation Test (MRT)³³. MRT was used to analyze the relationship described in the literature for other surgical tasks that involve a relationship between visual-spatial ability and surgical learning. Afterward, the students were given brief written instructions on how to perform the single-button suture and were asked to perform as many single-button sutures as possible within 5 min according to the written instructions. The procedure was recorded and documented with a camera (GoPro Hero 5, GoPro Inc., San Mateo, California, USA). A pre-study questionnaire was administered to collect general and demographic data, such as age, sex, handedness, visual ability, and previous experience with HMDs (Fig. 2).

Figure 2.

Illustration of the study protocol. After completing the Mental Rotation Test (MRT) and the 5-min pretest, the cohort was divided into three groups: e-learning, HMD-based, and tutor-led courses. Knowledge transfer of 40 min was conducted, followed by a 60-min practice period. Surgical skill and satisfaction were then assessed in the form of a posttest and Likert scale questionnaire.

Subsequently, a 40-min theoretical session was conducted. To ensure that the learning content was identical in all three trial arms, a standardized presentation was produced for the tutor-led course with important information on wound healing, suture material, various instruments, the correct execution of the single-button suture, 2D videos on the surgical instruments, suture materials, presentation of the main suturing methods, and how to perform the single-button suture. The 2D instructional video showed a bird’s eye view of the suturing procedure.

An online teaching application identical to the standardized presentation made during the tutor-led course for the e-learning course was derived and presented via a learning platform (https://emedia-medizin.rwth-aachen.de). The web application (a digital lecture) included texts, pictures and videos, and subjects were provided with a computer to visualize the e-learning program to learn independently and in a self-directed manner (Fig. 3A).

Figure 3.

Illustration of the three courses. (A) The self-directed approach in the e-learning course was visualized via a digital learning application and (B) the use of the application was still possible during the self-directed practice period. (C) View of the corresponding immersive and stereoscopic visualization from a first-person perspective and (D) subject in a VR-HMD with the opportunity to continue using it during the practice period. (E) In the tutor-led course, participants were given a lecture with a live demonstration and (F) individual feedback during tutor-led practice period. HMD, head-mounted display; VR, virtual reality.

Students on the HMD-based course received a video version of the standardized lecture presentation, which was displayed on a screen. In addition, after a 2-min adaption period, each participant learned independently through an immersive stereoscopic instructional video using the Oculus Quest 2 VR-HMD (Meta Platforms Inc., United States) and the preloaded video player (Meta Quest TV). For this purpose, a 180° stereoscopic video from a first-person perspective that conveyed the correct way to perform the surgical suture with explanations and instructions from the tutor was recorded (Insta360 EVO, Arashi Vision Inc., China) (Fig. 3C).

After that, the participants underwent 60 min of training. Participants of the e-learning course and the HMD-based course were allowed to continue using the respective platforms for as long as necessary during the learning session (Fig. 3B, D). Only students in the tutor-led course received direct feedback, including instruction, from a tutor (P.P.) (Fig. 3F). The tutor was an oral surgery resident (dental degree), who gained extensive suturing experience while working as a surgical technician for 5 years.

For the suturing procedure, an ethylene vinyl acetate (EVA) patch 4 mm in thickness (Infimedix GmbH, Osteorode, Germany) was provided as a low-fidelity bench model³⁴. Ethilon 5-0 (Johnson & Johnson, New Brunswick, New Jersey, US) was used as the suture material. A Hegar needle holder, surgical forceps, and scissors were provided, and disposable gloves were worn by all participants throughout the course.

Finally, all participants were asked to perform the demonstrated single-button suture technique again for 5 min under video recording. To maintain the subsequent anonymous evaluation, both the pretest and posttest video recordings included only the work environment so that only the gloves were worn and parts of the participants’ forearms were visible. Finally, all students were surveyed about their attended courses using a questionnaire.

The quality of surgical sutures as recorded in the videos was evaluated in a blinded fashion by experienced residents with a dual degree in medicine and dentistry (1× second-last and 2× final year of training) from our department (B.P., M.O., and M.U.) using the Global Rating Score (GRS). The GRS is the summary of eight items (respect for tissue, time and motion, instrument handling, suture technique, procedure of surgery and advance planning, knowledge of specific procedure, quality of final product, and overall performance; see Table 1) of the Objective Structured Assessment of Technical Skills (OSATS) and has been used since the late 1990s^35,36.

Table 1.

Global rating score for the objective assessment of the acquisition of surgical skills according to Denadai et al.⁴⁹ and Martin et al.⁵⁸.

	1	2	3	4	5
Respect for tissue	Often used unnecessary force on tissue or caused damage by inappropriate use of instruments		Carefully handling of tissue but occasionally caused inadvertent damage		Consistently handled tissues appropriately with minimal damage
	1	2	3	4	5
Time and motion	Many unnecessary moves		Efficient time and motion, but some unnecessary moves		Economy of movement and maximum efficiency
	1	2	3	4	5
Instrument handling	Repeatedly makes tentative or awkward moves with instruments		Competent use of instruments, although occasionally appeared stiff or awkward		Fluid moves with instruments and no awkwardness
	1	2	3	4	5
Suture technique	Clumsy and insecure, inadequately tying knots and inability to maintain tension		Careful and slow with most knots placed properly with proper tension		Excellent control of the suture, with proper placement of knots and correct tension
	1	2	3	4	5
Flow of operation and forward planning	Frequently stopped operation or needed to discuss next move		Demonstrated ability for forward planning with steady progression of operative procedure		Obviously planned course of operation with effortless flow from one move to the next
	1	2	3	4	5
Knowledge of specific procedure	Deficient knowledge. Needed specific instruction at most operative steps		Knew all important aspect of the operation		Demonstrated familiarity with all aspects of the operation
	1	2	3	4	5
Quality of final product	Very poor		Competent		Exceptional
	1	2	3	4	5
Overall performance	Very poor		Competent		Exceptional
	1	2	3	4	5
Select the final grade a					( )

^aMinimum 8 and maximum 40 points. 24 points corresponding to a competent level.

The primary endpoint (ΔGRS) was calculated as the difference between pretest and posttest skill and was based on the mean of three scores to account for interrater variability. The video allocation table was only available to P.P. until the end of the evaluation. After completion of the study, all data were anonymized according to the study protocol.

Several unexpected events occurred. Although prior training in a surgical suturing course was denied by all participants, some students (n=7; 2 for e-learning, 3 for HMD-based, and 2 for tutor-led courses) had proficient performance (OSATS GRS ≥24) during the pretest, which was against expectations, and they failed to meet the inclusion criteria of being a novice, according to the study protocol (Fig. 6A–C). Reasons for this may be the result of previous experience in sewing, talent, or not remembering previous suturing training. Due to the anonymous nature of the study, a follow-up survey was not possible. Therefore, we had to exclude these students for the final assessment, who showed a median ∆GRS of 1.

Figure 6.

Algorithm-based evaluation and public dataset. The number of sutures performed in the pretest/posttest and the optical flow, as well as the optical flow per suture, were related to the GRS. (A–C) Each purple square (e-learning course), dot (HMD-based course), and triangle (tutor-led course) represents a measurement in the pretest. In contrast, the green shapes (, , and ) represent posttest measurements. The red shapes (, , and ) represent the cases (n=7) who were not novices. The blue line represents the linear regression model and the gray area the 95% CI. (A) The achieved GRS and the number of sutures performed showed a significant correlation in the comparison of the pretest and posttest scores (R ²=0.73, P<0.001). (B) The pretest measurements showed a lower GRS with a significantly higher optical flow value compared to the posttest results (R ²=0.15, P<0.001). (C) With increasing GRS (x-axis), there was a significant decrease in optical flow per suture (y-axis, logarithmic) (R ²=0.61, P<0.001). (D) Visualization of the optical flow vectors in green. (E) The dataset published as part of this publication includes 314 videos with the corresponding OSATS GRS, which were evaluated by three investigators in a blinded manner. The number of sutures performed is also included. GRS, Global Rating Score.

For a more objective assessment of surgical skills based on the recorded videos, the optical flow (OF) (corresponding to the movement between individual frames) was calculated at a rate of 1 Hz (every 30th frame) for all videos and summed up. The OF was then divided by the number of sutures to obtain a score. The script (Supplementary Code) was programmed in MATLAB (version 2022b) and run on a workstation (AMD Ryzen 3900X, 64 GB memory and RTX 2080 TI OC, 11 GB memory).

Statistical analysis

Statistical analyses were performed using the programming language R (version 4.2.2). P-values <0.05 were considered significant.

A power analysis performed at a difference of 3 with a standard deviation of 4¹¹ and a power of 80% yielded a total sample size of 108 participants. During the peer review of the study, the power was increased to 95%, giving a total sample size of 150 participants, resulting in additional recruitment until the sample size was reached. According to the Shapiro–Wilk test, the primary endpoint (ΔGRS) was not normally distributed, and because of the ordinal nature of the GRS, a Kruskal–Wallis test was performed. A pairwise Mann–Whitney U test was used for subanalysis. Adjustment for multiplicity was applied according to Hochberg’s method³⁷. The correlation between the secondary endpoints and the influence of the number of sutures performed during training and MRT was examined using linear regression. Self-assessment was examined using Fisher’s exact test. Parametric and nonparametric tests, as well as linear regression, were used to examine the additional data in an explorative manner. In addition, linear regression analysis was performed to investigate the relationship between GRS and the number of sutures performed during training. Fisher’s exact test or the χ ² test was used to test group distributions.

Results

A total of 150 novices, including 84 medical students and 66 dental students, participated in the study. This group included 109 females and 41 males. Finally, 51 students participated in the e-learning course, 49 students in the HMD-based course, and 50 students in the tutor-led course.

On average, participants had an age of 22.2 years and were second-year students, with 62.7% of students reporting prior experience with stereoscopic media. Prior use of HMDs was affirmed by 14.6% of respondents. There were no observable systematic differences between the groups (Table 2).

Table 2.

Characteristics of participants (n=150).

	E-learning (n=51)	HMD-based (n=49)	Tutor-led (n=50)	Total (n=150)
Age
Mean (SD)	22.5 (3.8)	21.8 (3.3)	22.2 (4.3)	22.2 (3.8)
Sex
Female	36 (70.6%)	38 (77.6%)	35 (70.0%)	109 (72.7%)
Male	15 (29.4%)	11 (22.4%)	15 (30.0%)	41 (27.3%)
Study program
Medicine	29 (56.9%)	28 (57.1%)	27 (54.0%)	84 (56.0%)
Dentistry	22 (43.1%)	21 (42.9%)	23 (46.0%)	66 (44.0%)
Study year
Mean (SD)	2.2 (1.2)	2.4 (1.1)	2.2 (1.1)	2.3 (1.1)
Semester
Mean (SD)	4.4 (2.2)	4.8 (2.1)	4.2 (2.0)	4.5 (2.1)
Chirality
Right	45 (88.2%)	45 (91.8%)	47 (94.0%)	137 (91.3%)
Left/ambidextrous	6 (11.8%)	4 (8.2%)	3 (6.0%)	13 (8.7%)
Vision
Normal	22 (43.1%)	25 (51.0%)	22 (44.0%)	69 (46.0%)
Corrected	23 (45.1%)	23 (46.9%)	25 (50.0%)	71 (47.3%)
Not corrected	6 (11.8%)	1 (2.0%)	3 (6.0%)	10 (6.7%)
Mental Rotation Test
Mean (SD)	11.9 (5.0)	12.1 (4.7)	12.4 (5.0)	12.1 (4.9)
Computer gaming experience
No experience	15 (29.4%)	15 (30.6%)	18 (36.0%)	48 (32.0%)
Once	9 (17.6%)	11 (22.4%)	8 (16.0%)	28 (18.7%)
Yes, more often	27 (52.9%)	23 (46.9%)	24 (48.0%)	74 (49.3%)
Experience with HMDs
No experience	41 (80.4%)	40 (81.6%)	47 (94.0%)	128 (85.3%)
Once	5 (9.8%)	6 (12.2%)	3 (6.0%)	14 (9.3%)
Yes, more often	5 (9.8%)	3 (6.1%)	0 (0.0%)	8 (5.3%)
Experience with stereoscopic content
No experience	15 (29.4%)	17 (34.7%)	24 (48.0%)	56 (37.3%)
Once	18 (35.3%)	14 (28.6%)	11 (22.0%)	43 (28.7%)
Yes, more often	18 (35.3%)	18 (36.7%)	15 (30.0%)	51 (34.0%)

HMDs, head-mounted displays.

Suturing performance

The correlation in the GRS between the three investigators in the pairwise comparison was strong, with a Pearson’s correlation coefficient between 0.8 and 0.83. This is consistent with the pooled interrater correlation for OSATS GRS of 0.8 found in the literature [95% confidence interval (CI): 0.72–0.86]³⁶.

According to the students’ self-assessments, the HMD-based course was comparable to the tutor-led course and significantly better than the e-learning course (Mann–Whitney U test, P=0.003). In this regard, 37% of students in the HMD-based course, 22% in the tutor-led course, and only 10% in the e-learning course reported that they had improved beyond their expectations (Fig. 4A). In contrast, the median ΔGRS was 12.4 (95% CI: 10.0–12.7) for the e-learning course, 14.1 (95% CI: 13.0–15.0) for the HMD-based course, and 12.7 (95% CI: 10.3–14.2) for the tutor-led course. In the subgroup analysis, the unadjusted ΔGRS of the HMD-based course was significantly higher than that of the e-learning course (Mann–Whitney U test, P=0.039), which was not the case after correction for multiple testing (Mann–Whitney U test, adjusted P=0.114; Fig. 4C). Therefore, the different course modalities were comparable (Kruskal–Wallis test, P=0.15; Fig. 4C).

Figure 4.

Global Rating Score and students’ rating. (A) Self-assessed improvement by participants: worsened (-), unchanged (ø), improved (+), significantly improved (++), and improved beyond my expectations (+++). (B) Course suitability rating: Likert score (1=strongly disagree; 5=strongly agree). (A, B) Course type (y-axis) and percentage of participants with the respective rating (x-axis). (C) The learning effect assessed as ∆GRS (y-axis; primary endpoint) and presented as boxplots for the three course types (x-axis; e-learning=orange; HMD-based=pink; tutor-led=blue). (D) The GRS (y-axis) from the pretest and posttests (x-axis) of the three different courses (panels) presented as boxplots. (C, D) The horizontal line corresponds to the median, the box to the interquartile range (IQR=Q1–Q3) with the minimum (Q1–1.5×IQR) and maximum (Q3+1.5×IQR), and the red dots are outliers. The black dots are individual measurements. GRS, Global Rating Score; HMD, head-mounted display.

However, a multivariable analysis (adjusted for group, age, and sex) showed that ΔGRS was significantly correlated with the number of sutures practiced during the training time [β=0.15±0.49 (estimate±SD), P=0.002] and with the number of sutures in the pretest (β=−1.80±0.29, P<0.001). In a subsequent multivariable analysis (adjusted for age and pretest GRS), there was a significant correlation with the male sex (β=−2.85±1.27, P=0.028), the HMD-based (β=3.12±1.39, P=0.026) and tutor-led courses (β=2.76±1.37, P=0.046), and the number of sutures achieved in the pretest (β=2.03±0.74, P=0.007). Despite this, 31.4% [n=16, risk ratio (RR) 1.43, 95% CI: 0.60–2.43, P=0.37] of students in the e-learning course, 26.5% (n=13, RR 1.21, 95% CI: 0.74–2.76, P=0.64) in the HMD-based course, and 22% (n=11, RR=1.0) in the tutor-led course did not achieve a competent GRS (≥24) in the posttest (Fig. 4D).

Correlations were found between the ΔGRS and the achieved GRS in the pretest (R ²=0.22, P<0.001; Fig. 5A) and the achieved number of sutures in the pretest (R ²=0.15, P<0.001; Fig. 5B). In contrast, no correlation was shown between ΔGRS and MRT (R ²<0.01, P=0.615; Fig. 5C). The number of sutures performed during the training period affected the posttest GRS (R ²=0.18, P<0.001; Fig. 5D) and the number of sutures in the posttest (R ²=0.46, P<0.001; Fig. 5E). Additionally, a correlation between the OF per suture in the posttest could be observed (R ²=0.21, P<0.001; Fig. 5F).

Figure 5.

Visual-spatial ability and training number. Correlations regarding suture quality (GRS), quantity (sutures), the influence of the MRT, training amount, and optical flow (post). (A–F) Each square, dot, and triangle represents individual measurement values. E-learning (), HMD-based (), and tutor-led () courses. The blue line describes a linear regression model. The gray zone is the 95% confidence interval (CI). (A) With increasing pre-GRS (x-axis), the ∆GRS (y-axis) decreased (R ²=0.22, P<0.001). (B) The number of single-button sutures performed in the pretest (x-axis) were correlated with ∆GRS (y-axis) (R ²=0.15, P<0.001). (C) The score of the MRT (x-axis) for the ∆GRS (y-axis) is illustrated. No correlation between the measured visual-spatial ability and the increase in performance could be demonstrated (R ²<0.01, P=0.615). (D) With an increasing number of sutures performed during training, a higher GRS was achieved in the posttest (R ²=0.18, P<0.001). (E) This was also true for the number of sutures achieved in the posttest (R ²=0.46, P<0.001). (F) Similarly, the number of movements required per suture in the posttest decreased with increasing training (R ²=0.21, P<0.001). GRS, Global Rating Score; HMD, head-mounted display.

In an additional computerized analysis, we extracted the OF from the recorded videos, revealing a linear correlation between the GRS and the sutures performed in each test (R ²=0.73, P<0.001), as well as a weak correlation with the OF (R ²=0.16, P<0.001). Interestingly, there was a strong decrease in OF per suture performed (R ²=0.61, P<0.001), starting with an increase in the GRS (Fig. 6).

Qualitative evaluation

The students considered a surgical sutures course to be very important (x̄=4.9, 1 ≙ strongly disagree, 5 ≙ strongly agree) and agreed that it should be firmly integrated into the curriculum (x̄=4.9). The impairment in the acquisition of manual skills as a result of the COVID-19 pandemic was rated high (x̄=4.0). Text-only instructions and explanations of how to perform surgical sutures were generally disliked (x̄=1.3), and students rated the added value of watching an experienced surgeon or trained instructor perform the surgical procedure as high (x̄=4.4). In this context, live demonstrations (x̄=4.7) were rated to be slightly more helpful for the acquisition of manual skills than 2D videos or stereoscopic 180° videos (both x̄=4.4). Furthermore, live demonstrations were considered the most important factor for teaching manual skills and should be permanently integrated into the curriculum (Table 3). In general, a tutor-led course was rated significantly better in its ability to teach surgical suturing techniques (x̄=4.8). However, an HMD-based course (x̄=3.6) was perceived to be clearly better than an e-learning course (x̄=2.5) (Mann–Whitney U test, P<0.001; Table 3; Fig. 4B).

Table 3.

Evaluation of the importance of suturing courses, the influence of the pandemic situation, and the type of teaching materials on manual skills acquisition.

Mean (SD)	E-learning (n=51)	HMD-based (n=49)	Tutor-led (n=50)	Total (n=150)	P a
I think a surgical suturing course is very important in human medical or dental education.	4.9 (0.5)	4.9 (0.3)	4.9 (0.2)	4.9 (0.4)	0.912
The surgical suturing course should be firmly integrated into the curriculum of the medical/dental school.	4.8 (0.5)	4.9 (0.4)	4.9 (0.3)	4.9 (0.4)	0.739
Due to the COVID-19 pandemic and the resulting course restrictions, the acquisition of manual skills in medical/dental teaching is suffering.	3.9 (1.0)	4.1 (1.0)	3.9 (1.1)	4.0 (1.0)	0.981
The use of books or purely text-based teaching materials is sufficient for surgical training in manual skills.	1.3 (0.5)	1.3 (0.6)	1.3 (0.5)	1.3 (0.5)	0.981
Watching an experienced surgeon or trained instructor perform surgical suturing provides clear guidance.	4.2 (0.8)	4.5 (0.7)	4.6 (0.7)	4.4 (0.8)	0.043
The [use of videos | use of stereoscopic 180° videos using an HMD | live demonstrations] can improve medical teaching in terms of manual skill acquisition.	4.6 (0.6)	4.7 (0.6)	4.8 (0.4)	4.7 (0.5)	0.086
I would rate the importance of [videos | stereoscopic 180° videos and HMDs | live demonstrations] in teaching manual skills as follows: (1≙very low, 5≙very high).	4.3 (0.7)	4.3 (0.6)	4.7 (0.5)	4.4 (0.6)	<0.001
The [use of 2D videos | use of stereoscopic 180° videos using an HMD | live demonstration] should be further developed in relation to human/dental teaching and surgical skill acquisition.	4.1 (0.9)	4.3 (0.8)	4.6 (0.8)	4.3 (0.8)	0.003
A [pure e-learning course with videos | pure video course with additional stereoscopic 180° videos using an HMD | live demonstration as part of a tutor-led suture course] can teach surgical suture technique well.	2.5 (1.1)	3.6 (1.2)	4.8 (0.4)	3.6 (1.4)	<0.001

^aExploratory.

HMD, head-mounted display.

The actual course format was rated positively in terms of the amount of information disseminated and the length (x̄=4.6), the usefulness of the instructions (x̄=4.0), and understandability (x̄=4.6) for all three course types, with the presentation or live demonstration scoring the highest. The courses were rated good in terms of the memorability of the information conveyed (x̄=4.8 and 4.5, respectively) (Table 4).

Table 4.

Evaluation of the potential of different intervention methods and camera perspective/position (during live demonstration).

Mean (SD)	E-learning (n=51)	HMD-based (n=49)	Tutor-led (n=50)	Total (n=150)	P a
The format (amount of information and length) of the course conducted was engaging and appropriate.	4.6 (0.6)	4.4 (0.8)	4.8 (0.5)	4.6 (0.7)	0.055
The [e-learning application | recorded lecture | presentation] available in the study adequately provided me with the basic knowledge to perform a single-button surgical suture manually.	3.8 (0.7)	3.9 (0.7)	4.3 (0.7)	4.0 (0.7)	0.001
The [e-learning videos | stereoscopic video via HMD | live demonstration] explained the instructions on how to perform a single-button suture manually in an understandable way.	4.5 (0.8)	4.4 (0.7)	4.9 (0.4)	4.6 (0.7)	<0.001
I was able to remember the information given (on wound care and suture materials).	4.7 (0.6)	4.8 (0.5)	4.9 (0.3)	4.8 (0.5)	0.218
I was able to remember the given instructions (for performing the surgical single-button suture) well and apply them accordingly during the practical exercise.	4.5 (0.5)	4.5 (0.6)	4.6 (0.5)	4.5 (0.6)	0.423
[The use of [videos caused me discomfort and fatigue during use (while watching video) | stereoscopic videos using an HMD caused me dizziness and nausea during use] | The live demonstration caused me fatigue while watching]	1.5 (0.7)	1.6 (0.9)	1.4 (0.7)	1.5 (0.8)	0.659
The chosen perspective of [the videos | the stereoscopic 180° video | my position to the live demonstration] I would evaluate as follows: (1≙poor, 5≙excellent).	4.7 (0.5)	4.0 (0.8)	4.6 (0.7)	4.4 (0.7)	<0.001
A [camera perspective | perspective] from opposite (spectator position) I would evaluate as follows: (1≙poor, 5≙excellent).	2.7 (1.4)	3.0 (1.3)	4.0 (1.2)	3.2 (1.4)	<0.001
The potential of [videos | stereoscopic 180° videos and HMDs | live demonstrations] to provide the viewer with a feeling of immersion (putting oneself in the position of the practitioner), I would evaluate as follows: (1≙very low, 5≙very high).	4.2 (0.7)	4.5 (0.5)	4.2 (0.7)	4.3 (0.7)	0.040

^aExploratory.

HMD, head-mounted display.

In the self-directed courses, significantly more students in the e-learning course (47.1%, n=24) compared to the HMD-based course (18.4%, n=9) felt that feedback would improve learning (χ ²=9.30; P=0.002).

Regarding perspective, students in tutor-led courses most frequently chose the opposite perspective (48%) or the side perspective (42%), followed by the over-the-shoulder perspective (10%). The chosen perspective on the course content in tutor-led (x̄=4.6) and e-learning courses (x̄=4.7) was rated as good; however, the HMD-based course received a lower rating (x̄=4.0). A perspective from the opposite was rated best by the students who participated in the tutor-led course (x̄=4.0), while agreement was lower in the HMD-based (x̄=3.0) and e-learning (x̄=2.7) courses. For all course types, a high potential of putting oneself in the position of the practitioner was reported (x̄=4.2–4.5). Denial of discomfort, fatigue, dizziness or nausea was similar in the corresponding courses (x̄=1.4–1.6) and thus comparable (Table 4).

Regarding the use of HMDs, students stated a preference for the use of HMDs in university teaching in private settings (x̄=4.4 vs. x̄=3.2). Wearing HMDs in the presence of others (x̄=4.2) and being shielded from the environment was rated as a positive experience (x̄=4.3) and therefore perceived as an improvement in concentration (x̄=4.0). However, the use of the HMD for a longer period of time (more than 30 min) was rated as neutral (x̄=3.3).

Discussion

This RCT compared the benefits of using HMDs to teach surgical sutures against an e-learning course and the gold standard of a tutor-led course and included, to our knowledge, the highest number of participants (n=150) among published RCT studies on this topic⁵. Despite their distinctly different characteristics (self-directed/no feedback vs. tutor-led/with feedback; stereoscopic vs. monoscopic), all three types of courses were able to advance students’ surgical suturing skills (Fig. 4). Contrary to our expectations and the desires of the students, feedback from a tutor in our RCT did not show a significant impact on the quality of surgical suturing techniques among novices. Rather, the number of individual sutures performed during training was significantly related to the final competency level. These results are surprising, given that the tutor-led course with feedback is described in the literature as the gold standard¹ and the most common teaching modality⁵. However, the lack of positive effects associated with feedback in the acquisition of basic suturing techniques (Fig. 4C, D) is consistent with a study on the learning of laparoscopic suturing techniques, which shows that only actual training and knowledge of the result are relevant. Further instruction provided offers limited additional benefit³⁸. This is also consistent with an RCT that found that feedback offered no benefit between a video course and a tutor-led course in terms of efficiency and tissue handling during suturing³⁹.

Learning modalities were perceived very differently by the students (Fig. 4A, B). First, the self-assessment of improvement showed that the HMD-based course was best at advancing this factor, followed by the tutor-led course, and lastly, the e-learning course (Fig. 4A). Second, the students rated the tutor-led course best regarding the ability to teach surgical suturing techniques, followed by the HMD-based course. The e-learning course was largely seen as unsuitable (Fig. 4B). Therefore, feedback and the teaching modality seem to have a primary impact on the learning experience. This is consistent with a study showing that additional feedback via videoconferencing in an e-learning suturing course increased user satisfaction⁴⁰. Another study showed that feedback improves students’ confidence⁴¹.

Although the HMD-based course was very similar to the e-learning course and both did not provide feedback, the students’ ratings for each were significantly different (Fig. 4A, B). The HMD-based course, unlike the e-learning course, offers immersion and stereoscopy, and it is taught from a first-person perspective. Immersion describes the feeling of proximity to reality created by digital learning applications and constitutes an important aspect for understanding the effects of digital teaching media⁴². Higher immersion is said to enhance the user’s sense of being much more of a participant than an observer in the constructed space, known as the individual presence. This, in turn, can promote more intensive learning¹⁵. In line with this, students in this study described being shielded from the outside world by the HMD as a pleasant experience and reported being able to concentrate better. Similarly, the benefits of HMDs with non-stereoscopic 360° video (without spatial vision) for the acquisition of knot-tying skills have been demonstrated²⁵. Furthermore, Peden et al.³⁰ showed an increase in satisfaction with a non-stereoscopic AR-HMD (Google Glass).

An additional parameter influencing an enhanced learning experience through a deeper presence could be the spatial understanding of stereoscopic learning content. Stereoscopy is technically generated in HMDs through two different images for the right and left eyes, respectively (Fig. 3C)²⁸. This can lead to an expansion of ‘essential information’, such as improved visual acuity or depth perception⁴³. In this regard, extrinsic stereoscopy must be distinguished from intrinsic visual-spatial ability, the latter describing the ability to mentally manipulate objects in three dimensions^13,17. For this purpose, the visual-spatial ability of subjects can be assessed with the MRT, in which figures must be rotated using the imagination and assigned to a corresponding initial graphic figure³³. Wanzel et al.¹³ have shown that such high-level tests of visual-spatial ability are appropriate for demonstrating novice skills, such as in demonstrating rigid fixations of mandibular bench models. Previous work has shown that there is a linear correlation between knot-tying performance and visual-spatial ability¹⁷. Interestingly, stereoscopic displays were able to provide significant assistance in anatomical skill development to subjects with lower visual-spatial ability²⁸. However, we found neither a correlation between visual-spatial ability and the acquisition of surgical suturing techniques nor an advantage of stereoscopic displays in students with low visual-spatial ability (Fig. 5C). The reason for the lack of correlation could be the influence of our having students perform less complex procedures¹⁶, such as basic suturing techniques.

Aside from the stereoscopic presentation, the perspective on the teaching event also appears to play an essential role in promoting more intensive learning. For the past 100 years, the classic Halsted model of ‘see one, do one, teach one’ has been followed and applied in surgical teaching^19,44. The theoretical basis of this type of learning is mirror neurons, which play a fundamental role in understanding movement⁴⁵. In our tutor-led courses, the most common position chosen by students was the opposite perspective (48%) or the side perspective (42%), which also corresponds to a typical situation in the operating room. Only 10% of the students chose an over-the-shoulder perspective, which never fully corresponds to a first-person perspective. However, a true first-person perspective further enhances learning and can be provided by videos²². Unlike monoscopic 2D videos, stereoscopic videos through HMDs provide a much more uniquely realistic first-person perspective from the proper distance, viewing direction, and response to head movements. In this context, Nagayo et al.³¹, who visualized the step-by-step execution of surgical suturing through holographic representations using AR-HMDs from a first-person perspective, showed that this method was rated very helpful by the users.

It should be noted that the HMD-based course in our study did not offer an interactive VR environment but was based on immersive and stereoscopic 180° first-person videos. VR environments enable a change of viewpoint or even allow users to communicate with the simulations displayed, such as through direct digital feedback, to achieve a maximum form of immersion. The high level of technical preparation required makes these applications very complex. Therefore, it is very difficult to adapt to the needs of each specific medical school. In addition, current controllers for HMDs provide little haptic feedback, so the handling of instruments or tissue cannot be realistically reproduced in a VR environment. Therefore, the use of bench models is, to date, still the preferred approach for the teaching and acquisition of surgical skills. In this regard, studies have shown that the degree of fidelity has less influence on novices’ learning of surgical suturing skills⁴⁶. In this respect, even handmade tools proved to be useful during the COVID-19 pandemic⁴⁷. EVA, as used in our study, has been shown to be as suitable for learning surgical suturing and various related techniques as high-fidelity models, such as pig feet⁴⁸. Supporting this, when learning surgical suturing, the way the procedure is taught seems to have a higher impact on the learner than the material used in the model⁴⁹.

Overall, our results suggest that the HMD-based course offers clear benefits over an e-learning course in terms of the learning experience. Despite the same training duration of 1 h, interestingly, students performed ~3.1 sutures more in the HMD-based course and ~2.8 more in the tutor-led course than in the e-learning course.

However, the following limitations should be considered. Conclusions about the possible long-term retention of skills or the possible cumulative effect of longer training periods cannot be drawn from our study. In this regard, it is important that students train regularly by themselves, as there could be a decrease in learned skills^50,51.

Independent training and automated assessments as an alternative feedback method could enable this. However, most automatic systems for surgical performance assessment are based on kinematic data and require specific hardware⁵², while video-based assessment relies on endoscopic videos⁵². Although we showed that the OF correlates with the GRS and that there is a strong decrease per performed suture with an increase in the GRS (Fig. 6C), this alone would not be enough to replace detailed feedback⁵³.

In this regard, artificial intelligence (AI) algorithms for automated surgical performance assessment have already been used very successfully for endoscopic videos based on public datasets^54,56. However, we are not aware of any public dataset relevant to the evaluation of open surgical sutures such as those presented in our study⁵⁷. Therefore, as part of this study, we have published our dataset (https://doi.org/10.5281/zenodo.7940583) for further development. The dataset includes 314 videos with a total length of 26 h and blinded OSATS annotated by three qualified investigators. This makes it one of the largest video datasets⁵⁷ and to our knowledge the first in its field.

Conclusion

The use of HMDs with stereoscopic and immersive video provides advantages in enhancing the learning experience and should be preferred over a traditional web application for e-learning. Contrary to expectations, feedback is not necessary for novices to achieve a sufficient level of competence in surgical suturing; only the number of surgical sutures performed during training is a good determinant of competence improvement. Nevertheless, feedback still enhances the learning experience. Therefore, automated assessment as an alternative feedback approach could further improve self-directed learning modalities. As a next step, the data from this study could be used to develop such automated AI-based assessments.

Ethical approval

The study approved by the Institutional Review Board (or Ethics Committee) of University Hospital RWTH Aachen (protocol code EK 352/21 and date of approval 09.09.2021 & protocol code EK 22-329 and date of approval 05.10.2022).

Informed consent statement

Informed consent was obtained from all subjects involved in the study.

Sources of funding

This research was supported by the Medical Faculty of the RWTH Aachen University under the grant number SBE-22-2-5. Behrus Puladi was funded by the Clinician Scientist Program of the Medical Faculty of RWTH Aachen University.

Author contribution

B.P., P.P., F.H., and A.B.: conceptualization; P.P., B.P., A.B., and M.Li.: methodology; P.P., B.P., M.Le., and R.R.: software; P.P., B.P., and A.R.: validation; P.P., B.P., F.H., M.Le., F.K., T.W.K., and R.R.: formal analysis; P.P., B.P., M.O., and M.U.: investigation; F.H., B.P., M.Le., M.Li., and R.R.: resources; P.P. and B.P.: data curation; P.P.: writing – original draft preparation; B.P., P.P., R.R., F.H., M.Le., A.B., M.O., M.U., F.K., A.R., M.Li., and T.W.K.: writing – review and editing; P.P. and B.P.: visualization; B.P.: supervision; B.P.: project administration; B.P.: funding. All authors have read and agreed to the published version of the manuscript.

Conflicts of interest disclosure

The authors declare no conflicts of interest.

Research registration unique identifying number (UIN)

1. Name of the registry: German Clinical Trials Register.

2. Unique identifying number or registration ID: DRKS00029307.

3. Hyperlink to your specific registration (must be publicly accessible and will be checked): https://drks.de/search/de/trial/DRKS00029307.

Guarantor

Behrus Puladi.

Data availability statement

The video dataset of this study, including the OSATS assessment by the three investigators, can be downloaded at https://doi.org/10.5281/zenodo.7940583. The remaining data generated and/or analyzed as part of this study are available upon reasonable request from the corresponding author.

Acknowledgements

We would like to thank Jakob van Eys, Daniel Fink, Patrick Buchta, and Uli Heuter from the Audiovisual Media Center of the Medical Faculty RWTH Aachen University for their technical support in the production of the teaching material and Dr. Max Taubert for preparing the random assignment sequence and providing statistical advice.