Randomized trial of a virtual reality tool to teach surgical technique for tibial shaft fracture intramedullary nailing.

Abstract

Introduction:

Active learning methods have accumulated popularity due to improved results in knowledge acquisition as opposed to passive learning methods. For surgical resident physicians with limited training opportunities outside of the operating room due to time constraints, virtual reality (VR) is a relatively inexpensive and time-efficient active training method for procurement of surgical skills. We conducted a simulated intramedullary nailing (IMN) of a tibia to demonstrate VR training programs as a more effective modality of learning orthopaedic surgical techniques compared to passive learning tools such as a standard guide (SG) through trained novice medical students performing a SawBones simulation of intramedullary nail fixation.

Materials and Methods:

First and second-year medical students without prior experience of procedure were recruited and randomized to SG or VR training. Participants were observed performing simulated tibia IMN procedure immediately after training and evaluated by a blinded attending surgeon using procedure-specific checklist and 5-point global assessment scale. Participants returned after 2-weeks for repeat training and evaluation.

Results:

20 participants were recruited and randomized into VR (n=10) and SG (n=10) groups. All 20 participants completed the first phase and 17 completed the second phase of the study.

Aggregate global assessment scores were significantly higher for VR than SG group (17.5 vs. 7.5, p<0.001), including scores in all individual categories. The percentage of steps completed correctly was significantly higher in the VR group compared to the SG group (63% vs. 25%, p<0.002).

Average improvement between the first and second phases of the study were higher in the VR group compared to SG group across all 5-categories of the global assessment scale, and significantly higher for knowledge of instruments (50% vs. 11%, p,0.01).

Discussion:

VR training was more effective than a passive SG in our model of simulated tibia IMN for novice medical students. Virtual reality training may be a useful method to augment orthopaedic education.

INTRODUCTION:

Resident surgeons in training acquire the skills necessary for independent practice by a combination of reading, observation, hands-on exercises and supervised guided operative experience. With advances in orthopaedics, surgeons are expected to master a growing number of procedures. Work-hour restrictions and increased time spent on non clinical administrative tasks reduce the amount of time the trainees can spend apprenticing with expert surgeons in the operating room.^1,2,3 Current training methods include reading surgical technique guides, SawBones simulations and cadaveric dissection. Each of these methods has limitations: technique guides both typically require a baseline familiarity with the specific equipment used and do not allow the trainee to actively demonstrate their knowledge; SawBones simulation does not include the operating room environment or soft tissue aspects of surgery and is not reusable; cadaveric dissections are expensive, time consuming, are not reusable and rarely provide the trainees the opportunity to practice with multiple systems.

As a common orthopaedic surgical procedure, intramedullary nail fixation of tibial shaft fractures (IMN) serves as a good example of this problem. When performed correctly, tibial IMN restores alignment, stabilizes the fracture and allows for immediate weight bearing.⁴ However, failure to perform all steps correctly can lead to increased risk of iatrogenic injury to blood vessels and nerves, and long-term sequelae including non-union, malunion, and infection.⁵ Orthopaedic surgeons learn to master this common surgical procedure during the course of their residency, with initial exposure as early as the first year of training. However, the equipment used for tibial IMN is complex and often brand-specific. A surgeon who is highly proficient using one system may be significantly less so when forced to perform the procedure with a different system, potentially leading to increased operative time and surgical complications.⁶ One method proposed to address these problems is the use of virtual reality (VR) for surgical training. VR simulation allows an active approach for surgeons to become familiar with instruments, equipment, and procedures before performing surgeries on patients.⁷ VR allows the trainee flexibility and user control within the virtual environment. It assists the user to recognize patterns, understand physical processes, and move amongst different frames of reference, a dynamic model that allows users to learn intangible information. Furthermore, training with VR simulators has been shown to improve surgeon performance over time and these skills have been shown to be transferable to improving performance in the operating room. VR “warm ups” have the ability to provide real-time personal feedback which has been shown to significantly improve trainee surgeon performance of laparoscopic colectomies in swine models as well as cholecystectomies.^7,8,9 Successful utilization of VR in surgical training have led to the wide-spread adoption of such systems in general surgery training programs, with at least one program mandating that residents undergo VR training before performing any live surgery.¹⁰ While virtual reality training simulators have been shown to be an effective training modality, especially in arthroscopic surgery,¹¹ VR has not been widely adopted in orthopaedic surgery training programs. As new VR training programs designed for orthopaedic surgery training emerge, they will require careful evaluation to determine if they provide any advantage over the standard training methods or as an adjunctive tool to supplement them.

The aim of this study was to compare the performance of medical students trained using a VR program to those trained with a SG program in a SawBones simulation of a tibial IMN procedure designed to evaluate orthopaedic surgical techniques.

MATERIALS & METHODS

Source of Funding: NIH T32 Regenerative Musculoskeletal Medicine Training Program.

All procedures were performed with UCLA IRB approval. First and second-year medical student volunteers were recruited. Any students with previous reported experience performing or viewing tibial IMN operations (in person or video), previous use of SawBones equipment or VR based gaming systems were excluded. Informed consent was obtained from all participants. All VR system components used were borrowed from Osso VR (http://www.ossovr.com/) and returned to the company at the conclusion of the study.

Study design

A blinded, randomized prospective study was performed on two days, two weeks apart. Two training programs were developed for the study: Senior orthopaedic surgery attending staff at our institution reviewed a virtual reality training module for intramedullary nailing of a tibia (Osso VR), as well as a published surgical technique guide for a tibial IMN system (Synthes). A detailed checklist identified steps required to complete the procedure on a SawBones simulation of tibial IMN using an intact tibia model. The surgical guide and VR training module were edited and modified to contain the same equipment, steps and instructions.

On the first day of the study, twenty medical students (first and second-year) medical students were given hands-on tutorial on drill use, including insertion of different sized bits, chuck mechanism use, and advancing and reversing by third-year orthopaedic surgery residents. All students were observed performing these tasks to ensure uniform competency. Students were then given a sealed envelope, randomly assigning them into one of two groups: standard guide (SG) and virtual reality (VR) training, and separated into two training rooms. Students in both groups were provided with a copy of the checklist, which included only the listed steps of the procedure without specific instructions for each step, and were instructed to use it as a reference during training if needed. A copy of the checklist was placed in the work area of the testing room.

Students in the SG group were given the printed surgical technique guide, which utilized illustrations, photographs and step-by step instructions, and given instructed to review the training material for a maximum of 20-minutes.

Students in the VR group were instructed on using the Oculus Touch Motion controllers hand controls until they were comfortable with the equipment but no longer than 5-minutes. The students were then given the step-by-step VR training module for tibial IMN procedure in tutorial mode, which instructed them on the procedure with written instructions and prompts for each step. After going through the tutorial one time, students were switched over to the testing mode of the module, where clues and instructions are only given when prompted, and allowed to repeat the test module as many times as they wanted for a maximum of 20-minutes total training time. No further instructions were given to either group and no questions about the procedure or training materials were permitted.

Osso VR is a hands-on surgical training platform. For this study the software was run on a standard Osso VR hardware unit utilizing the Oculus Rift virtual reality headset and Oculus Touch motion controllers. When undergoing training, users don the VR headset to enter the virtual operating room and interact with the patient and virtual instrumentation using the Touch controllers. Triggers on the controllers are utilized to grip and operate instruments.

After training, participants were escorted to a separate room to perform a simulated tibia IMN procedure using SawBones. Participants were given 15-minutes to perform the procedure using identically arranged equipment. If they were unable to proceed at any point, they could ask for a prompt, but receive a score of zero for that step. Students were observed by an attending surgeon evaluator who remained blinded to the training assignment of each student until the end of the study. Each subject was filmed and recorded for later detailed analysis of their performance.

Students were then asked to return two-weeks later for re-evaluation. On the second day of the study, students were maintained in their original randomized groups. Students in the SG group again were given 20-minutes to review the printed surgical guide and checklist and students in the VR group were instructed to go through the VR training module once in tutorial mode and then in test mode as many times as they were able for a maximum of 20-minutes. Students were then escorted to the testing room where they performed the SawBones procedure with the same instructions, tools and conditions as the first day.

Evaluation of skills:

The attending surgeon evaluator was provided with the Global Assessment 5-Point Rating Scale and Procedure-Specific Checklist (Adapted from Martin et al., 1997 and Araujo et al., 2014)^8,12 and instructed to observe videos of the participants, which were provided in a randomized manner, without identification of the group assignment. Using the Global Assessment 5-Point Rating Scale, participants were awarded between 1 and 5 points for each category, with low scores awarded for poor performance and high scores for proficiency as described in (Table 1). Participants were also evaluated using the Procedure-Specific Checklist as follows: 1-point for every step performed correctly and 0-points if the step was performed incorrectly or not performed. (Table 2)

Table 1.

Global Assessment 5-Point Rating Scale.

Time and motion	1	2	3	4	5
	Many unnecessary moves.		Efficient time/motion but some unnecessary moves.		Economy of movement and maximum efficiency.
Instrument handling	1	2	3	4	5
	Repeatedly makes tentative or awkward moves with instruments.		Competent use of instruments although occasionally appeared stiff or awkward		Fluid moves with instruments and no awkwardness.
Knowledge of instruments	1	2	3	4	5
	Frequently reached for the wrong instrument or used an inappropriate instrument.		Knew the names of most instruments and used appropriate instrument for the task.		Obviously familiar with the instruments required and their names.
Flow of operation and Forward planning	1	2	3	4	5
	Frequently stopped operating or hesitant in next task or subtask.		Demonstrated ability for forward planning with steady progression of operative procedure.		Obviously planned course of operation with effortless flow from one move to the next
Knowledge of specific procedure	1	2	3	4	5
	Deficient knowledge. Needed specific instruction at most operative steps.		Knew all important aspects of the operation.		Demonstrated familiarity with all aspects of the operation.

Attending surgeon evaluator was provided with this chart prior to the procedure and instructed to rate the participant’s performance immediately after observing them perform the SawBones Tibial IMN simulation.

Table 2.

Procedure-specific Checklist.

	ITEM:	Not Done/Done Incorrectly	Done Correctly
1.	Guidewire Insertion ((wire inserted at appropriate starting point and angle)	0	1
2.	Entry reamer selection (user selects correct reamer, assembles it in his/her hand)	0	1
3.	Nail assembly (user assembles nail, guide, and locking mechanism)	0	1
4.	Nail insertion (user inserts nail as far down sawbone as allowable)	0	1
5.	Proximal Interlock Guides (User selects correct triple guides and inserts through proximal drill guide)	0	1
6.	Proximal Interlock Screw (User places from medial to lateral, through nail in static position)	0	1

Provided to evaluator prior to observing the procedure. Participants were awarded 0 points for steps performed out of sequence.

Statistical analysis was performed using Mann-Whitney-Wilcoxon and Fischer’s exact tests with significance defined as p<0.008 based on Bonferroni correction for multiple comparisons. All statistical analysis was performed using GraphPad Prism Statistics / Data Analysis software, GraphPad Software inc, La Jolla, CA.

RESULTS:

Twenty participants were recruited; 10 were randomized into the SG group and 10 into the VR group. All 20 participants completed the first phase of the study. In phase-I, participants in the VR group completed the procedure an average of 147 seconds faster than SG group, which was statistically significant (VR: 615 sec, SG: 762 sec, p = 0.002). Participants in the VR group completed 38% more steps correctly in the procedure-specific checklist compared to the SG group, which was statistically significant (VR: 63% correct steps, SG: 25% correct steps, p = 0.003).

As measured by the Global Assessment 5-Point Rating Scale, participants in the VR group received significantly higher ratings in all categories compared to the SG group (Table 3), (Figure 1).

Table 3.

Phase-I Global Assessment scores for individual categories and aggregate scores.

Global Assessment Categories:	VR	SG	p-value
Time and Motion	3.6	2.4	0.003 *
Instrument handling	3.4	1.8	0.002 *
Knowledge of instruments	3.3	1.1	0.0002 *
Flow of operation & forward planning	3.5	1.1	0.0002 *
Knowledge of specific procedure	3.7	1.1	0.0002 *
Aggregate Score	17.5	7.5	0.0004 *

Statistical analysis was performed using Mann-Whitney-Wilcoxon test with Bonferroni correction, significance set at p = 0.008 Results were significant in all categories.

Figure 1. Objective Structured Assessment of Technical Skill utilizing the Global Assessment 5-point Rating Scale.

Higher values were awarded for more proficient performance. Statistical analysis was performed using Mann-Whitney-Wilcoxon test, significance was set at p=0.05 (* = p<0.05, ** = p<0.01, *** = p<0.001) and P=0.008 following Bonferroni correction noted with red asterisks as follows:. * = p<0.01, ** = p<0.008 *** = p<0.0008. Results were significant in all categories.

When evaluated by the procedure specific checklist, a higher percentage of participants completed the steps correctly in the VR group compared to the SG group for all categories. The differences for nail insertion and proximal interlock guide trended towards but failed to reach significance following Bonferroni correction. (Figure 2) (Table 4)

Figure 2. Procedure-specific checklist results.

Percent of each group who completed each step correctly. Statistical analysis was performed using Fisher’s exact, * = p<0.05, ** = p<0.01. Following Bonferroni correction, no significance was noted * = p<0.01, ** = p<0.008

Table 4.

Procedure specific checklist % who completed each step during phase-I.

Procedure-specific checklist item	VR	SG	p-value
Guidewire Insertion (wire inserted at appropriate starting point and angle)	70%	50%	0.325
Entry reamer selection (user selects correct reamer, assembles it in his/her hand)	30%	50%	0.325
Nail assembly (user assembles nail, guide, and locking mechanism)	30%	0%	0.1053
Nail insertion (user inserts nail in correct orientation and angle)	90%	30%	0.01
Proximal Interlock Guides (User selects correct triple guides and inserts through proximal drill guide)	70%	10%	0.01
Proximal Interlock Screw (User places from medial to lateral, through nail in static position)	70%	30%	0.09
Average number and % of steps completed correctly by each participant	1.5 (63%)	3.8 (25%)	0.002*

Statistical analysis was performed using Statistical analysis was performed using Fisher’s exact with Bonferroni correction, p=0.008. Although the percent of students who completed each step successfully was not significantly different between the two groups, the average number of steps completed correctly by each student (analyzed using Mann-Whitney with Bonferroni correction p<0.008), was significant.

Seventeen of the students who completed phase-I returned to complete phase-II (9 of the SG group and 8 of the VR group). In phase-II, students in the VR group completed more steps correctly in the procedure-specific checklist compared to the SG group, which was significant (VR: 69% correct steps, SG: 33% correct steps, p = 0.008). When comparing phase-I and phase-II procedure-specific checklist performance within each group, both SG and VR groups showed improvement in overall completion rates, however this was not statistically significant (VR: 6% improvement, SG: 8% improvement, p=0.17)

In phase-II participants in the VR group again scored significantly higher than those in the SG group in all categories on the Global Assessment 5-Point Rating Scale (Table 5). Comparison of scores between phase-I and II found that both groups improved in all categories, though not to a statistically significant level. (Figure 3) When comparing the percentage of participants whose scores improved between phase-I and II, a higher percentage of students in the VR group made improvements compared to the SG group, though this failed to reach significance. (Table 6), (Figure 4)

Table 5.

Phase-II Global Assessment scores for individual categories and aggregate scores.

Global Assessment Categories:	VR	SG	p-value
Time and Motion	3.9	1.4	<0.001 *
Instrument handling	3.9	1.3	<0.001 *
Knowledge of instruments	3.8	1.2	<0.001 *
Flow of operation & forward planning	4.0	1.4	0.002 *
Knowledge of specific procedure	4.4	1.8	<0.001 *
Aggregate Score	19.9	7.2	<0.001 *

Statistical analysis was performed using Mann-Whitney-Wilcoxon test with Bonferroni correction, significance set at p = 0.008 Results were significant in all categories.

Figure 3. Comparison of phase-I and phase-II scores within each group in all global assessment categories.

Statistical analysis was performed using Mann-Whitney-Wilcoxon test, and Bonferroni correction with significance set at p = 0.008. Differences within each group were not statistically significant between Phase-I and Phase II.

Table 6.

Percent improvement between phase-I and phase-II within each group.

Global Assessment Categories:	VR	SG	p-value
Time and Motion	38%	11%	0.18
Instrument Handling	50%	22%	0.59
Knowledge of instruments	50%	11%	0.03
Flow of operation & forward planning	50%	22%	0.10
Knowledge of specific procedure	50%	44%	0.80
Aggregate Score	63%	22%	0.23

Statistical analysis was performed using Mann-Whitney-Wilcoxon test with Bonferroni correction, significance set at p = 0.008. Knowledge of instruments trended towards but failed to reach significance.

Figure 4. Percentage of students whose scores improved between phase-I and phase-II.

Statistical analysis was performed using two-tailed Fisher’s exact test and Bonferroni correction with significance set at p = 0.008. These findings were not statistically significant.

DISCUSSION:

The goal of this study was to evaluate the efficacy of VR training in teaching orthopaedic surgical techniques compared to standard surgical guide training by evaluating the performance of novice medical students performing a SawBones tibial shaft IMN procedure. Our findings show that when participants were evaluated using objective structured assessment of technical skills using the Global Assessment 5-point rating scale, those who received VR training prior to the procedure performed significantly better in all categories than participants who were provided with a printed SG.

When evaluated on a procedure-specific checklist, a higher number of students from the VR group performed each step correctly than those in the SG group. The difference failed to reach statistical significance for each item but the difference between the aggregate numbers of steps performed correctly in each group was significantly higher for the VR group. The differences between the VR group and SG group in both metrics were maintained when students returned for a second test two weeks later. While both groups demonstrated a slight improvement in scores between the first and second phase, a higher percentage of students in the VR cohort improved their performance than those in the SG group, though this failed to reach statistical significance. As neither group was given access to their respective training methods (guide or VR module) or allowed any hands-on practice of the procedure in the two-week period between the two experimental phases, the fact that both groups showed some improvement may be ascribed to having prior experience with the procedure from the first phase, which may have functioned as de facto training for phase two.

In his seminal study published in 2017, Brian George et al, evaluated the readiness of general surgery residents to perform procedures independently across 14 academic programs. His findings demonstrated that only 85% or fewer graduating residents were deemed ready for independent practice through a validated assessment tool. Moreover, analysis of attending surgeon ratings of general surgery residents in their last 6-months of training showed that only 40.3% of residents achieved near-independence in the top five core procedures as defined by the ACGME. These findings were ascribed to multi-factorial issues contributing to an overall diminished resident operative experience, including work-hour restrictions, financial pressure on attendings to increase productivity, changing patient expectations and policies that require extensive attending surgeon hands-on participation; forces, which equally influence residents in all surgical subspecialties^13,14. Exacerbating this issue is the expanding learning curve of many modern procedures and technologies such as minimally invasive techniques and robotics. Proficiency in anterior total hip arthroplasty requires at least 50 repetitions, while orthopaedic robotic systems require 80 or more repetitions for proficiency.¹⁷ Many studies have demonstrated that active learning methods such as problem-based learning as opposed to passive lecture-based learning results in improved knowledge acquisition, heightened transfer of concepts into clinical competence, and increased motivation for self-directed learning due to increased perceived relevance.^15,16 Moreover, meta-analysis conducted on active simulation-based learning, showed a substantial effect on skill, knowledge, and behaviors for patient-related outcomes.¹⁸

VR-based training systems may help address these deficits by providing trainees the opportunity to prepare for surgery independently by taking advantage of self-assessment tools to identify and address specific weaknesses. Furthermore, by training in the safety of the simulated environment, trainees can become better prepared to take advantage of hands-on operative experience more efficiently while reducing the risk to patients and the costs associated with current training methods such as cadaver labs.

The degree of surgical resident autonomy provided by supervising surgeons is a complex issue under increasing scrutiny. While many attending surgeons assume they evaluate levels of proficiency and assign autonomy objectively, there is growing evidence that various biases affect these evaluations. In some cases this may be due to over-estimating the surgical skills of residents who are perceived to be hard-working and dedicated, however there is significant evidence that systematic biases, conscious or otherwise, affect female and underrepresented minority residents at significantly higher rates compared to their Caucasian male counterparts.^{20, 21} These attitudes can manifest by negatively affecting resident confidence, preventing them from gaining independence. In one study, female surgical residents were less confident and rated their abilities lower than their male counterparts despite performing equally well on surgical simulators.²¹ In addition, results from a national survey of 4339 general surgery residents reported that African American residents were the least likely of all races to believe that their operating skills were level appropriate.²²

While these problems in surgical education are symptoms of much larger societal and institutional factors without easy solutions, immersive VR training systems with objective assessment capability may be a valuable tool by allowing attending faculty to review resident performance in a blinded manner and by giving residents an objective assessment of their performance compared to other residents in their programs and nationally.

LIMITATIONS & CONCLUSION

A limitation of our study is the use of medical students rather than surgery residents. An argument can be made that a group of first and second-year medical students may include a significant number of people who will not end up in surgical fields and that the differences in performance based on training method would not be as significant in a group of surgical residents highly selected for their skills and aptitude in this field. However, given the differing level of exposure to the procedure orthopaedic surgery residents gain before starting residency and the small number of residents starting each year, we believed that a novice cohort of students was a more accurate way to measure these differences. In addition, students’ prior differences in drill handling skills were controlled with the initial drill readiness training administered at the beginning of the study.

Another potential limitation is that while our study was sufficiently powered to measure differences between students in the VR and SG groups, we were not sufficiently powered to detect differences within each group between phase I and II, as this effect was smaller and there was dropout of three students. Further work is planned with a larger number of medical students, as well as orthopaedic surgery residents and multiple observers, to obtain sufficient data for statistical significance and assess intra-observer reliability.

Our findings demonstrate that novice medical students trained to perform a simulated tibial IMN procedure performed better when trained through an interactive VR simulation than when using a passive SG and suggests a possible future role for virtual reality training in orthopaedic surgery education, and surgical education at large. Further work will be required to validate these training methods in long-term longitudinal studies of residents in a variety of surgical subspecialties and procedures.

Highlights.

• Aggregate global assessment scores were significantly higher for VR than SG group.

• Percentage of steps completed correctly was significantly higher in the VR group.

• VR group had higher completion of later steps in the procedure.

• VR group scored significantly higher in knowledge of instruments.

• VR can provide subjects with more effective real time feedback early in learning.