Virtual Reality and Augmented Reality—Translating Surgical Training into Surgical Technique

Abstract

Purpose of Review

As immersive learning outside of the operating room is increasingly recognized as a valuable method of surgical training, virtual reality (VR) and augmented reality (AR) are increasingly utilized in orthopedic surgical training. This article reviews the evolving nature of these training tools and provides examples of their use and efficacy. The practical and ethical implications of incorporating this technology and its impact on both orthopedic surgeons and their patients are also discussed.

Recent Findings

Head-mounted displays (HMDs) represent a possible adjunct to surgical accuracy and education. While the hardware is advanced, there is still much work to be done in developing software that allows for seamless, reliable, useful integration into clinical practice and training.

Summary

Surgical training is changing: AR and VR will become mainstays of future training efforts. More evidence is needed to determine which training technology translates to improved clinical performance. Volatility within the HMD industry will likely delay advances in surgical training.

Introduction

Orthopedic surgical training is currently in a state of rapid change. Halsted’s apprenticeship model, where residents and fellows learned their craft under the supervision of a senior trainee or surgeon, drove surgical training into the modern era [1]. In this model, characterized by repetition and graduated responsibility, the patient acted as the primary training modality. However, contemporary surgical trainees face multiple challenges that threaten this paradigm. The increasing complexity of procedures, new technology and techniques, more stringent work hour restrictions, an increasingly litigious legal environment, and increased scrutiny of trainees’ roles in patient care have emphasized the need for innovative education and training techniques. In addition to curriculum standardization and an emphasis on competency-based advancement, the push to provide skills acquisition and assessment in nonclinical environments is an exciting new frontier in surgical training.

Many authors argue that surgical training should mirror pilot training, which leverages the power of simulation before exposing trainees to live situations. Some argue that surgical trainees have “an ethical obligation” to be exposed to all clinical scenarios that can be “reasonably well simulated” before experiencing them with patients. Proponents argue that this technique transfers both instructors’ and learners’ focus to knowledge and skill acquisition and relieves them of the pressure of balancing patients’ safety [2]. This ethical argument is also applied to practicing surgeons who wish to acquire new skills. Over the past 20 years, there has been an explosion of new minimally invasive techniques which often require new skills and tools. These evolving procedures and techniques often impose a learning curve on practicing surgeons that affect early outcomes and complication rates. [3–5] Further, these minimally invasive procedures decrease the viewing field in surgery for learners and educators. The surgeon’s best view is often available to the attending or the resident/fellow, but often not simultaneously [6–9]. This optimal view is critical for goal of graduated autonomy in surgical education. The learner must see as critical portions of the case are performed. The educator must see as it becomes the learner’s time to perform the procedure. With traditional open exposure and arthroscopic procedures on the other end of the spectrum, simultaneous visualization of the surgeon’s best view is possible. It is the middle ground, minimally/less-invasive procedures, where new technology can help bridge these training challenges. It is also possible that technology can help facilitate learning and mastery of new techniques through realistic simulation that can be performed in a nonclinical setting minimizing harm to patients.

Successful surgical skill development though simulation has been well documented. Simulation has been shown to accelerate learning curves for surgical trainees in multiple specialties, including neurosurgery, gynecology, general surgery, and orthopedics [10–13]. Cadaveric models, computer simulation, synthetic models, and animal models can all reliably assess surgical skill [10, 12–15]. While both low- and high-fidelity bench simulations have shown utility and efficacy in orthopedics, they do not allow for the gradual increase in autonomy that surgical trainees need to become proficient and confident. This article provides a narrative review of virtual reality (VR) and augmented reality (AR), two modalities being used to facilitate teaching and assess surgical skills [11].

Mixed Reality

According to Milgram’s theory, mixed reality consists of a continuum existing between our interaction with the real world and a completely virtual world [16, 17]. Virtual reality (VR) is an “immersive, completely artificial computer-simulated image and environment with real-time interaction.” [18]. This mode of simulation can involve any of the senses and is not limited to sight and touch. In contrast, augmented reality (AR) lies between a completely virtual environment and the real world. AR is a broad category that superimposes virtual data on a real-world image.

Mixed reality devices are becoming increasingly ubiquitous in many facets of life. Applications such as Google Translate and Amazon Shop recognize and relay information about objects and words in real time. In healthcare, the mixed reality industry was projected to be worth $641 million by 2018 and $2.4 billion by 2024 [19, 20]. While this is clearly a growing industry, the utility of mixed reality in surgical training has yet to be well characterized.

Appraising Surgical Simulators

The utility and benefits of surgical simulators are best assessed based on face, content, construct, and concurrent validity [21]. Face and content validity are subjective measures of the appropriateness of the simulator’s psychomotor fidelity and of the variables measured, respectively. Construct validity is an objective measure of the simulator’s ability to distinguish novices from experts. Concurrent validity describes the correlation between the variables measured and real-world performance [21].

Virtual Reality

Since the first VR knee arthroscopy simulator was introduced in the 1990s [12], numerous orthopedic simulators and task trainers have been introduced and studied. Modern simulators allow trainees to practice skills such as fracture reduction, sawing, drilling, and arthroscopy. VR-based simulators allow trainees to home diagnostic skills, pre-operative planning techniques, intraoperative decision-making, and surgical techniques outside of the operating room [14].One of the key benefits of VR simulation over real-world simulation is that novices can get immediate constructive feedback and assistance on their performance without the need for face-to-face expert guidance [14].

Ruikar’s review of orthopedic simulators for psychomotor skill and surgical procedure training split orthopedic VR simulators into 3 groups: non-interactive simulators, interactive simulators with visual feedback, and interactive simulators with visio-haptic (tactile) feedback. Non-interactive simulators are defined by their ability to help visualize anatomy and volumetric data. They are primarily used to aid in diagnosis and help to plan and predict surgical outcomes. 3D pre-operative total hip arthroplasty planning is a common application of a non-interactive simulator. While these devices do not improve manual skills, they may help trainees plan a more successful surgery [14].

Interactive simulators are defined by their ability to simulate entire procedures, guiding trainees through key steps. Trainees utilize a mouse, keyboard, or other optical trackers to work through these simulations. Examples of interactive VR simulators range from simple smartphone applications such as ImmersiveTouch Surgery (Yoo, Chicago, Illinois) to intraoperative guidance systems such as Hip Navigator (HipNav, Pittsburgh, Pennsylvania). Applications such as Immersive Touch, which can be downloaded as an app on a smart phone or tablet, can be used by trainees from any location at any time. (This confers an important benefit over visio-haptic simulators, which require a dedicated lab space and can only be used by one trainee at a time.) The few studies assessing the use of interactive simulators with visual feedback in orthopedics demonstrate that there is construct validity when distinguishing novice, intermediate, and expert knowledge in carpal tunnel releases [22] as well as construct, face, and content validity in intramedullary nailing of femur fractures [23]. One downside, however, is that while interactive simulators show benefit in learning about procedures in general, many do not account for the tactile feedback that is inherent in higher-fidelity simulation [14, 24, 25]. Companies such as Fundamental Surgery (FundamentalVR, Boston, Massachusetts) have worked in earnest to partner with academic programs and incorporate VR into surgical training.

Visio-haptic simulators, or those that apply tactile feedback, include shoulder, knee (Fig. 1), and hip arthroscopy simulators, fracture fixation simulators, and orthopedic drilling simulators [21, 25–29]. While haptic simulators demonstrate construct validity and are able to improve performance of specific tasks among both experts and novices [21, 30–34], whether these skills translate to improvements in clinical practice remains an area of active study. Kalun and colleagues evaluated four studies that attempted to answer this question [30, 35–38]. Three of the four studies used a high-fidelity visio-haptic arthroscopic simulator and assessed trainees on both procedural checklists and an arthroscopic global rating scale (GRS). In two of the three studies, training with the haptic simulator led to better knowledge of the surgical steps [36, 37], but only one showed significant improvement on the GRS compared to controls [36]. The authors concluded that the heterogeneity of training protocols, tools, and outcome measures makes it difficult to assess true transfer validity.

Fig. 1.

ArthroSim (TolTech, Aurora, Colorado) visio-haptic knee arthroscopy simulator

Bartlett’s review of VR in orthopedics identified multiple studies showing construct validity of simulators including knee, hip, and shoulder arthroscopy along with hip and intraarticular fracture fixation [21]. However, the authors concluded that while the simulators showed ability to distinguish between novices and experts, many lacked in their ability to differentiate subtler expertise differences between intermediate and expert learners. These findings may limit the utility of these tools in real-world assessment and training for more advanced surgeons [21]. They also noted the heterogeneity of the outcomes measures used in the studies. Overall, they supported the use of knee and shoulder arthroscopy simulators that have been validated but believe that more investigation is needed before their widespread use can be fully supported [21].

While there is evidence supporting the validity of haptic arthroscopic simulators in orthopedic training, haptic VR simulators for drilling and fracture fixation do not offer a clear benefit over lower-fidelity alternatives, which are usually cheaper and more widely available [39–44]. Low-fidelity techniques for teaching basic orthopedic skills such as avoidance of drill plunging [45], cortical screw tightening [46•], and fracture reduction and fixation [31, 43, 44] have shown evidence of transfer and content validity. Several studies in the general surgery realm have shown no additional benefit of high-fidelity simulation compared to low-fidelity simulation, with both methods demonstrating increased skill acquisition compared to textbook review alone [47, 48]. However, head-to-head studies of high- and low-fidelity simulation for these orthopedic surgery tasks have not been performed.

Augmented Reality

AR in orthopedics is a fairly recent but rapidly developing area of study, with over 50 publications on the topic in the past 2 years (Fig. 2). AR allows supplemental data to be incorporated into the surgeon’s real-world sensory inputs and has been integrated into orthopedic procedures as well as surgical training [18, 49–54]. This commonly includes overlaying useful visual data such as relevant imaging into the surgeon’s field-of-view but can also include auditory or sensory feedback, intraoperative navigation, and telementoring or guidance [18, 49–54].

Fig. 2.

Orthoapedic AR publications since 2018 referenced in PubMed

AR in the Operating Room

Blackwell and colleagues first described the use of AR in orthopedics in 1998. [49] The authors postulated that AR could be used for intraoperative guidance (for positioning components and avoiding critical structures), surgical training, and simulation. Since that time, multiple advancements have made AR implementation a reality in many operating rooms. Despite these advancements, there are still many barriers to the widespread use of AR in orthopedics, including integration into the operating room, interface comfort and reliability, and equipment comfort.

AR devices require a display, a position tracking system, and software that transforms and incorporates data [49]. The display can take the form of either a traditional monitor or a head-mounted device (HMDs). HMDs range from simple opaque displays that rest in front of one of the surgeon’s eyes such as the Vuzix M300 (Vuzix, Rochester, New York) (Fig. 3) to semitransparent displays like the Microsoft HoloLens (Redmond, Washington) that overlay information into the extended visual field (Fig. 4). While most HMDs include cameras, videos, and accelerometers that track the wearer’s head position, some also track hand gestures and/or the user’s eye motion [53, 55, 56].

Fig. 3.

Vuzix M300. Simple opaque display

Fig. 4.

The Microsoft Hololens is an example of a head-mounted display that has been utilized in the operating room as an example of augmented reality

The Google Glass (GG, Mountain View, California) HMD was the first commercially available HMD and has been most studied HMD device in the orthopedic literature over the years [12, 18, 57–60]. Many researchers adopted Google Glass because it was lightweight, had a high-resolution video camera, and connected to wireless internet [18]. However, production ceased on Google Glass in 2015 and multiple issues that continue to plague newer HMDs were found including battery life, image quality, line of sight, and network authentication [18, 57].

Multiple other commercially available AR HMDs have been developed since GG was released. The simplest designs, like the Vuzix M300, consist of an opaque viewfinder that sits on glasses frames and uses an Android-based operating system (OS). The Osterhaut Design Group (ODG; San Francisco, California) R7 HMD (Fig. 5) is a popular non-wired HMD with a semitransparent display, an Android-based OS, head gestures, and voice control. There are multiple reports of its promise in intraoperative image guidance, data display, and education [56, 61, 62]. However, ODG recently collapsed following a failed acquisition and the device is no longer available [63].

Fig. 5.

ODG R7 HMD—non-wired HMD with a semitransparent display, Android-based OS, head gestures, and voice control

The Microsoft HoloLens was, until recently, the most advanced commercially available HMD. It has a large wireless semitransparent display that runs on a proprietary OS. Its eye tracking and head and hand gesture-based commands set it apart from its competitors [61, 64]. A 2017 comparison of the HoloLens and the ODG R7 concluded that its contrast perception, text readability, frame rate, and limited system lag made the HoloLens more suited for professional use [61].

As of August 2018, the Magic Leap One HMD was developed and released for use. It is a semitransparent HMD with a wired “lightpack” that handles processing, head and hand gesture controls, and eye tracking [65]. There are no published reports on its use in orthopedics or healthcare. However, reports that the company has partnered with multiple healthcare companies and surgeon training companies suggest that its use may become more widespread in the future [66].

In spine surgery, AR is primarily used to facilitate intraoperative navigation [67]. Elmi-Terander and colleagues used a heads-up display monitor coupled with intraoperative cone-beam CT (CBCT) and an AR surgical navigation system to place thoracic and lumbosacral pedicle screws in 20 patients [68••]. Several bench and cadaveric studies found that HoloLens-assisted pedicle screw placement had up to 97% accuracy when compared to the gold standard technique [69•, 70]. Edstrom and colleagues proposed that AR-assisted surgical navigation of pedicle screw placement decreases radiation exposure when used in conjunction with CBCT [71]. AR has also been used as an adjunct for osteotomy guidance. Kosterhon used a proprietary navigation system to overlay planned osteotomies on a surgical microscope’s field of view, helping surgeons identify the correct osteotomy planes [72].

AR-assisted tumor resection has made headway in orthopedic oncology as well. AR-based navigation systems have been used to resect bone and soft tissue tumors with excellent adherence to surgical margins [73–75], less blood loss, and shorter operative time [76]. While this technology is promising, whether these techniques provide a definitive benefit over the current standard of care remains to be seen [77].

AR has been useful in fracture management, where image overlay or projection has been used to simulate guidewire placement [78•], facilitate intramedullary nail insertion [79–81], and insert sacroiliac screws [82, 83]. Projecting fluoroscopic images into the surgeon’s visual field was shown to improve tip-apex distance, shorter radiation exposure time, and shorter intramedullary nail total insertion time, likely related to time saved by the surgeon not having to move their eyes to look at a fluoroscopy monitor [79].

In adult reconstruction, AR has been used primarily for intraoperative navigation, anatomic referencing, and intraoperative imaging [84–86, 87•, 88••]. Multiple bench studies have demonstrated accuracy with AR-assisted acetabular cup placement [85, 86]. Logishetty and colleagues found no difference accurate placement of the acetabular cup accuracy between groups of novices randomized to surgeon supervision versus HoloLens assistance [87•]. Fallavollita devised a novel c-arm augmented with a camera which allowed for 3 fluoroscopic images to be constructed together to make an intraoperative mechanical axis view with no parallax errors [84]. Finally, Lei and colleagues successfully used the Hololens HMD, 3D printing, and anatomic referencing to perform a total hip arthroplasty on a patient with prior hip arthrodesis [88••]. The HMD and preop scans were overlayed on the patient intraoperatively to assist in placement of the 3D-printed implant [88••].

AR in Education and Training

HMDs have been used in medical [24] and surgical education [62]; however, AR has not demonstrated a clear advantage in terms of skill or knowledge retention when compared to conventional training or VR. However, they did find the VR group was most likely to have adverse effects such as headache, dizziness, and blurred vision [24]. Until the benefit of AR to learners is better understood, caution that should be observed when implementing AR into surgical education, particularly due to its high cost.

AR in TELEMONITORING

Telemonitoring, or remote surgical guidance, is another area where AR and HMDs show much promise. Remote guidance from an experienced surgeon benefits trainees as well as experienced surgeons learning a new skill [89]. Outside of the operating room, telemonitoring can be used to assess post-operative wounds [90, 91] or for remote consultation among colleagues.

The University of Alabama-Birmingham Orthopedic and Neurosurgical departments developed a remote surgical assistance model known as Virtual Interactive Presence and Augmented Reality (VIPAR) [50–52, 91–93]. Their initial proprietary design consisted of screen that combined images from a “local” (intraoperative) camera and a remote camera [52]. This allowed a remote surgeon to provide visual assistance to an operating surgeon [52]. Orthopedic faculty successfully used this system to provide virtual assistance to resident surgeons during shoulder arthroscopy [50]. Both residents and attendings believed it to be a safe and useful adjunct to teaching in the operating room [50]. Later iterations of this technology transitioned to using an iPad as the display and eventually to using the Google Glass HMD [51, 93]. Using the Google Glass VIPAR system, a total shoulder arthroplasty case was performed in Birmingham, AL with remote assistance from a surgeon in Atlanta, GA [93]. While they considered it a successful procedure, users reported secure network connection issues, a limited battery life, and divergent lines of sight between the surgeon and the camera [93].

A similar set-up used the ODG R7 HMD to guide trainees through temporal bone dissections [56••]. The attending physicians uploaded imaging, annotated relevant anatomy in the trainees’ field of view, and provided audible guidance during the procedure [56••]. Users cited poor connectivity, limited line of sight, lack of magnification, and the need for a headlight as factors that limited the practicality of this tool [56••].

Concerns with HMDS

While HMDs have shown promise for surgical navigation and telemonitoring, concerns about battery life, line of sight, secure network access, cost, and HIPAA compliance persist [56, 93]. Furthermore, fragmentation of hardware and software development (i.e., the development of Microsoft HoloLens and Magic Leap One, backed by Microsoft and Google respectively and run off two separate operating systems) may slow the development of useful applications [55, 65].

Another unintended consequence of HMDs is inattentional blindness, or failure to see an object located within one’s visual field because it did not engage the viewer’s attention [94]. There is early evidence that HMDs use may lower a surgeon’s ability to notice foreign bodies intraoperatively [94, 95]. Differences in inattentional blindness between tasks may be affected by cognitive load [94]. Further study is needed to determine whether this phenomenon has sustained or adverse impact among users.

Multiple studies have demonstrated that HMDs can cause side effects such as nausea, headaches, and vertigo [24, 49, 96]. While there is some evidence that these effects are more pronounced in VR simulations compared to AR, a better understanding of potential adverse effects is needed as HMDs become more prevalent in the operating room [24].

Future

Despite the popularity of AR for surgical navigation, the intraoperative use of HMDs and their role in clinical care has not been firmly established. While pilot studies have demonstrated their utility in bench and cadaveric models and in limited clinical studies, surgeons must navigate the practical and ethical challenges associated with implementing this technology in patient care. If positive outcomes over the usual practice are shown in larger clinical studies, HMDs may soon be used to display pertinent information, images, and even fluoroscopy images in the surgeon’s field of view. To increase these devices’ utility in the operating room, more advanced tracking software must be developed. While optical tracking is an accurate way of tracking position intraoperatively, it does suffer from the need to have line of sight, which can limit the surgeon’s freedom in the operating room. One solution is electromagnetic tracking, which overcomes the need for line of sight; however, current technology is limited by interference from metal tools [81, 96–98]. As HMD tracking and spatial registration systems become more advanced, these devices could conceivably replace (instead of augment) bulky surgical navigation systems [96] [49, 54, 78, 96, 99].

Our Experience

The lead author’s institution recently acquired 9 HMDs (Vuzix M300 ×3, ODG R7 ×3, Google Glass ×3) courtesy of funding from an AOTrauma grant to assess the deployability and sustainability of AR in orthopedic training. In our short time with the HMDs, we found multiple issues consistent with those described in the literature, including callibration of the field of view alignment, technical glitches obscuring the surgeon’s field of view, and poor connectivity (Table 1). We also struggled to find an affordable, efficient software to record and stream the surgeon’s best view to both learners and instructors. We found that the Vuzix M300 and ODG R7 HMDs were easiest to use due to being based on an open source Android operating system. The ODG R7 HMDs showed the most promise; however, their parent company went out of business before completion of our study thus highlighting the challenges of volatile product markets when trying to study a new technology.

Table 1.

Experience with three head-mounted displays for use in orthopedic training

	HMD Type
	ODG R7 glasses ×3	Google Glass X ×3	Vuzix M300 glasses ×3
Pros	– Easiest interface to use and set up (touchpad with mouse, finger remote, head gestures)	- Lightweight, unobstrusive	- Lightweight, easy to set up with wifi network because Android based.
			- Multiple software applications to use, and can communicate with other android devices.
	- While FOV is obscured, it is transparent.
Cons	- Camera field of view is slightly off from human field of view. Need to have glasses sit tilted downwards for view to overlap.	- Cannot set up on protected hospital network - Interface is least intuitive to use.	- Opaque viewfinder, so blocks field of vision.
			- Uniocular (some dizziness with testing)
			- Camera field of view difficult to line up with viewer field of view.
	- Heavy with entire field of view affected

Conclusions

Virtual and augmented reality are taking an increasingly important role in surgical education as well as in surgical care. Numerous studies have demonstrated that VR improves skill mastery outside the operating room, though translation of these skills to the clinical environment is difficult to assess. AR is primarily being used for simulated and intraoperative navigation; however, as the devices’ ability to replicate the surgical environment improves, these tools are likely to become more prevalent in both training and patient care. HMDs in particular show promise in surgical training; however, given the volatility within the HMD industry, more study is needed before a clear leader can emerge in terms of both the HMD and the accompanying software.

Funding Information

Dr. McKnight, Dr. Hsu, and Dr. Hwang received funding from the AOTrauma North America Fellows Research Support grant.

Compliance with Ethical Standards

Conflict of Interest

The other authors declare that they have no conflict of interest.

Joseph Hsu is a paid presenter or speaker for Smith & Nephew.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.