Augmented Reality (AR) in Orthopedics: Current Applications and Future Directions

Abstract

Purpose of Review

Imaging technologies (X-ray, CT, MRI, and ultrasound) have revolutionized orthopedic surgery, allowing for the more efficient diagnosis, monitoring, and treatment of musculoskeletal aliments. The current review investigates recent literature surrounding the impact of augmented reality (AR) imaging technologies on orthopedic surgery. In particular, it investigates the impact that AR technologies may have on provider cognitive burden, operative times, occupational radiation exposure, and surgical precision and outcomes.

Recent Findings

Many AR technologies have been shown to lower provider cognitive burden and reduce operative time and radiation exposure while improving surgical precision in pre-clinical cadaveric and sawbones models. So far, only a few platforms focusing on pedicle screw placement have been approved by the FDA. These technologies have been implemented clinically with mixed results when compared to traditional free-hand approaches.

Summary

It remains to be seen if current AR technologies can deliver upon their multitude of promises, and the ability to do so seems contingent upon continued technological progress. Additionally, the impact of these platforms will likely be highly conditional on clinical indication and provider type. It remains unclear if AR will be broadly accepted and utilized or if it will be reserved for niche indications where it adds significant value. One thing is clear, orthopedics’ high utilization of pre- and intra-operative imaging, combined with the relative ease of tracking rigid structures like bone as compared to soft tissues, has made it the clear beachhead market for AR technologies in medicine.

Introduction

The field of orthopedics has been continually transformed by revolutionary new imaging technologies. Starting with the common X-ray in 1985, then moving on to ultrasound (1950s), computerized tomography (CT) (1972), and magnetic resonance imaging (MRI) (1977) with each technological step enhancing the ability of the orthopedic surgeon to diagnose, monitor, and treat musculoskeletal aliments. Advances in these technologies have led to step-wise higher-quality image acquisition at a faster speed, lower cost, and with less radiation exposure. The next revolutionary advancement in imaging may not be in the acquisition, but instead in the display of these images within the realm of extended reality (ER).

Extended reality refers to many different immersive technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) which seek to merge the physical and virtual worlds. Virtual reality (VR) completely immerses the user in a digitally created world using a head-mounted device that creates a 360-degree view of an artificial world and is often combined with 3D audio and haptic feedback devices to create a more immersive experience. VR has seen prominent use in simulation-based medical training with some applications assisting with pre-op planning [1]. However, because the user loses sight of their real environment, there are currently no relevant intra-operative applications.

Augmented reality (AR) overlays or augments the real world with virtual content. The digital content can be displayed through AR glasses, or via screens, tablets, and smartphones. The most recognizable application that brought attention to this technology was Pokémon Go [2]. Because the users maintain their view of the real world, this technology has several interesting clinical use cases. Mixed reality (MR) also known as merged or hybrid reality is a more advanced form of AR in which digital and real-world objects coexist and interact with one another. This allows virtual objects to be anchored in the physical world. A 3D reconstructed pelvis can be anchored over the patient’s native anatomy such that it will remain in the correct anatomic position no matter how the AR device moves in relation to the patient. These applications require significantly more processing power than either VR or AR. For the purpose of this review, the term augmented reality (AR) will be used to encompass both AR and MR.

Augmented Reality

There are three overarching types of augmented reality: projection based, video-see-through (VST), and optical-see-through (OST). In projection-based AR, a 2D or 3D image is projected onto the site of interest (Fig. 1a) [3•, 4–7]. The advantage of this approach is in the direct overlay of the image on the site of interest and in the ability of all nearby observers to simultaneously see and interact with the AR scene. However, its efficacy can be limited by lighting conditions and changes in the surface contours or appearance of the projection site.

Fig. 1.

Three common types of augmented reality (AR). a Projection-based AR demonstrated by a laser projected QWERTY keyboard. b Video-see-through (VST) AR demonstrated by interior decorating mobile application. c Optical-see-through (OST) AR demonstrated by the Microsoft HoloLense displaying surgical guidance lines. Fig. 1a cropped to emphasize the subject, labeled for reuse according to Create Commons Attribution-Share Alike 2.0 Generic License (From Wikimedia Commons) Fig. 1b and c labeled for reuse according to Create Commons Attribution-Share Alike 4.0 International License (From Wikimedia Commons)

In VST, the digital content is superimposed on a live video feed of the real world which can be displayed on a monitor, tablet, or phone screen [8, 9••, 10••, 11–13, 14•, 15–26]. This has been popularized by mobile gaming (e.g., Pokémon Go) and the design industry where it allows for visualization of virtual objects in a physical space, such as a couch in a living room (Fig. 1b).

In OST, the digital content is projected onto an optically clear lens that the user is able to see through. This is typically accomplished via a head-mounted display (HMD) such as Google Glass or the Microsoft Hololense [20, 21, 27, 28, 29•, 30–33]. This can create an incredibly immersive experience; however, only the wearer of the HMD can see and interact with the virtual content which can impede effective communication in the OR (Fig. 1c). Additionally, HMDs are often bulky and present an ergonomic challenge to surgeons who would be required to wear these devices for the duration of a surgery.

A high-fidelity AR experience in the operating room is built upon three key technologies: visual rendering, image registration, and tracking [17, 34–38]. Visual rendering is the process of generating a 2D or 3D image by means of a computer program. The complexity of the rendering and the speed at which it needs to be rendered and refreshed for a given application dictates the required processing power of the AR device, which has important implications for the ergonomics of the system [34, 36]. Image registration is the creation and overlaying of physical and virtual coordinate systems which allows for the accurate placement of virtual content within the physical scene. The accuracy of this process is critical for surgical outcomes as an inaccurate registration of even a few millimeters could lead to drastic surgical complications [34].

Tracking is the process by which objects are followed, and their exact positions within the overlaid virtual and visual coordinate systems are determined. It is critical to track the patient’s anatomy as well as any tools that enter the surgical field. Tracking is done via optical and or electromagnetic signals and can be accomplished with or without the aid of markers with each approach having unique strengths and weaknesses [36].

Marker-based tracking requires physical markers to be placed within the operative field, adding additional time, workflow changes, and potential obstructions. However, markers provide a higher level of accuracy with a lower computational burden than marker-less tracking. The two most common types of markers are spherical markers that either actively emit or passively reflect incident IR light and fiducial markers which have a unique size, shape, or pattern to enable easy registration (Fig. 2) [19, 27, 28, 34, 35, 37, 38, 41–43]. Marker-based tracking for tools most often involves the attachment of IR or fiducial markers in a unique 3D arrangement that allows the machine vision application to accurately determine the 3D position of the tool within the coordinate system (Fig. 2a and c) [19, 27, 28, 37, 41–43].

Fig. 2.

Surgical markers. a Reflective IR tracking spheres used to monitor the 3D position of an acetabular impactor. b Optical fiducial markers to register placement of femoral and tibial cutting guides. c Reflective IR tracking spheres on a posterior-superior-iliac-spine reference frame and awl-tipped tap. Fig. 2a cropped to emphasize the subject, labeled for reuse according to Creative Commons Public Domain Mark 1.0 [39]. Fig. 2b Reprint permission given by Pixee Médical. Fig. 2c labeled for reuse according to Create Commons Non-Commercial-No-Derivatives 4.0 [40]

Marker-less tracking of the patient and or tools can be accomplished using either a single or stereoscopic camera set up in conjunction with computer vision algorithms [6, 8, 14•, 17, 30–32, 44]. The benefits of marker-less tracking are reduced setup time and easier workflow integration. However, this method is far more computationally demanding, and the level of accuracy is not currently acceptable for clinical use. For OST-HMD-based AR, it is also critical to track the location and vantage point of the wearer. This is typically accomplished through some combination of onboard sensors and cameras in the HMD but is sometimes aided by the use of external cameras [34, 36].

Clinical Advantages of AR

Image-based orthopedic procedures require the use of external monitors which demand frequent attention shifts, create hand-eye coordination problems, and impose a large cognitive load, all of which have the potential to reduce efficiency in the OR and impact accuracy creating a higher chance for surgical error [45, 46]. Augmented reality technologies have immense potential to solve these problems and reduce cognitive load, so there is reserve to focus on education, mentorship, and managing complications if they arise [45, 46].

Attention shifts can be eliminated by using OST-HMDs to display fluoroscopic images in the corner of a surgeon’s view of the operative field. When this technology was tested in a sawbones model, orthopedic surgeons no longer needed to avert their eyes and head when placing femoral head guide wires resulting in improved tip apex distance, less radiation exposure, and shorter insertion times than with conventional fluoroscopy (Fig. 3) [33]. Hand-eye coordination problems can be solved by augmenting the operative site with fluoroscopic images in a precise overlay in what has been coined video-augmented-fluoroscopy (VAF) (Fig. 4a). This is accomplished with a camera-augmented mobile C-arm (CamC) that uses a double mirror system to co-register the video and fluoroscopic feeds with an accuracy of <1mm [13, 14•, 15–19, 24, 42]. VAF has been investigated for distal locking of IMNs and for K-wire insertion in a variety of trauma models with initial reports demonstrating faster operative times, lower X-ray requirements, and subjective reports from operators of improved spatial orientation [14•, 17, 19, 24].

Fig. 3.

Using an OST-HMD to eliminate attention shift. a Surgeon shifting attention to fluoroscopy monitor. b Surgeon wearing smart glasses showing fluoroscopy images no longer needs to shift attention away from the operative field. Fig. 3 labeled for reuse Creative Commons Attribution-Noncommercial 4.0 License [33]

Fig. 4.

AR applications in surgery. a Video-augmented-fluoroscopy (VAF) showing fluoroscopic images overlaid on a live video feed of the operative field to aid with start point localization and drilling in a percutaneous bovine model. b Digitally reconstructed radiographs (DRRs) showing digitally rendered coronal, sagittal, and transverse radiographs at the level of the surgeon’s instrument. c Projector overlaying a rending of the patient’s CT scan to assist with start point localization. d Integral videography (IV) system: left side showing system set up and right side showing surgeons view of an in situ 3D visualization of the patient’s knee based on pre-operative CT scans. Fig. 4a reproduced with permission of John Wiley and Sons [14•]. Fig. 4b reproduced with permission of Elsevier [8]. Fig. 4c reproduced with permission of Elsevier [6]. Fig. 4d reproduced with permission of Elsevier [47]

The aforementioned technologies have a limited impact on cognitive load because they still require the surgeon to create 3D mental reconstructions from 2D images, a process that can be counterintuitive and error-prone due to projective simplification [42]. AR technologies can use pre- or intra-operative CT scans to create 3D anatomic overlays and/or provide a live feed of digitally reconstructed radiographs (DRRs) at different viewing angles without the need to move the C-arm or take additional X- rays (Fig. 4b) [42, 48]. The 3D reconstructions are registered to the patient anatomy using markers attached to anatomic landmarks (Fig. 2c) or in more sophisticated systems via optical surface mapping and machine vision approaches [13, 15–17, 48, 49]. Fischer et al. investigated the impact of these successively more complex AR systems, (1) conventional XR, (2) VAF, and (3) 3D visualization with DRRs, on key operative outcomes and showed significant improvements in operative time, radiation exposure, and reduced task load as measured by the Surgical Task Load Index (SURG-TLX) [17].

The other main approach to reducing cognitive load by providing in situ 3D visualizations has been using projection-based methods [3•, 4–7, 50]. Projection-based approaches avoid the ergonomic concerns of HMD’s and the spatial discontinuity of VST while allowing multiple viewers to see and interact with the AR environment. Initial forays into projection-based methods used projectors that directly overlay mapped anatomic information on the patient’s skin (Fig. 4c) [3•, 6, 7]. Gavaghan et al. showed that this visualization approach improved spatial understanding and reduced the need for sight diversion in both a biopsy and bone tumor resection model [7]. In the spine surgery arena, Wu et al. demonstrated that this approach facilitated faster localization of pedicle screw entry points intra-operatively [6]. However, accuracy is limited body habitus and potential changes in posture from pre-operative imaging; thus, this system is unable to guide insertion trajectory [6]. Xu et al improved upon this technology and demonstrated accurate screw guidance with an acceptable degree of error in a pre-clinical model (Fig. 4c) [3•].

Another approach to projection-based AR is autostereoscopic image overlay of integral videography (IV). Autostereoscopy is a means of generating 3D images without the need for special headgear or glasses on the part of the viewer. Integral videography is a system that uses, captures, and reproduces a light field using a 2D array of micro-lenses to create autostereoscopic images. By using a half-silvered mirror device, the operator can then directly perceive the real environment while also viewing a reflected IV video overlay creating the illusion of a 3D image inside the patient’s body (Fig. 4d) [4, 5, 50, 51]. This allows for the precise depth perception that is essential to an AR navigation system and avoids any issues with tissue deformation that would obscure visualization with a conventional projector. Siemionow et al. demonstrated the ability of an optically tracked IV system, to facilitate starting point localization for thoracolumbar instrumentation in a cadaveric model [52•]. However, these systems can be bulky and time-consuming to set up and have the potential to impair the visualization of the operative field.

While the trend in orthopaedics and other surgical disciplines has been a move from open to minimally invasive procedures, has trended in orthopedic and other surgical disciplines, this approach often increases radiation exposure for both the patient and the whole medical team as surgeons need to take repetitive X-ray images to control the procedure and validate the placement of tools and implants [53–62]. AR technologies can allow surgeons to track the movement of their tools and implants in relation to patient anatomy in real time obviating the need for repetitive X-ray exposure. Using the CamC system to co-register fluoroscopic images to the live native scene promises to make fluoroscopic image acquisition more targeted and efficient drastically decreasing the patient and occupational radiation dose. Studies have demonstrated significant reductions in the number of X-rays taken by up to 46% as well as lower occupation radiation exposure as measured by dosimeter [14, 17, 42, 63]. These approaches have primarily been used for establishing starting points for percutaneous access in trauma surgeries.

Another approach to reducing occupational radiation exposure is using 3D virtual reconstructions registered to the patient’s anatomy to provide real-time instrument tracking and DRRs obviating the need for intra-operative fluoroscopy [5, 11•, 64, 65•]. Several groups (Edstrom et al [11•], Elmi-Terander et al [11•, 64], and Burstrom et al [65•]) have utilized intra-operative CBCTs to create and register these 3D models. With CBCT providers can step out of the room or properly shield themselves so that they experience near-zero radiation exposure. Other groups, however, aim to remove ionizing radiation from the operating room altogether. Ma et al. proposed using an intra-operative ultrasound to register a pre-op CT to the patient’s spinal anatomy, while Dibble et al. proposes using a posterior-superior-iliac-crest pin/marker combination to register a pre-op CT to the native anatomy [4, 27]. These solutions have much more market potential as they can be implemented in most operative settings without the requirement for a highly expensive intra-operative CBCT scanner.

A controversial area is the impact of augmented reality technology on operative times, as they introduce new equipment which can require technical setup and complicate surgical workflow. These technologies often increase total procedure time, up to 65 min in some studies [66–70]. Many proposed technologies have claimed to significantly reduce operative time: Fisher et al.’s CBCT-DRR system showed a 60% time-savings for the insertion of K-wires in phantom trauma models, and the Sureshot system showed an 11-min time-savings for the placement of distal locking IMN screws in live patients [17, 71]. However, whether or not these reported modest time improvements translate to real-time benefits is unclear with further research on cost-effectiveness needed.

Furthermore, many technologies that appear to be time-saving in pre-clinical models fail to live up to that promise when utilized in a clinical setting. Elmi-Terander et al demonstrated a decrease in navigation time per-pedicle screw in a cadaveric model using a CBCT AR system. However, when this technology was translated to the clinical setting, the time per pedicle screw was equivalent to that of the freehand technique, and the average operative time was unchanged [9, 15, 64].

As far as accuracy and precision are concerned, a number of groups have studied AR systems in phantom models for a variety of applications: pedicle screw placement [3•, 5, 29•, 31, 64, 65•, 72•, 73], tumor resection [16, 74], distal IMN interlock [4], K-wire or needle placement [21, 33, 42, 75], arthroplasty [13, 43, 44, 76], and vertebroplasty [77] with some groups showing improvement over traditional freehand approaches. However, the jump from pre-clinical sawbones and cadaveric models to live patients is substantial with significantly more degrees of freedom added in.

Spinal surgery and, in particular, pedicle screw placement has received the most attention and as a result [35] has the most technologies at a clinical stage. However, clinical data on the accuracy of augmented reality surgical navigation (ARSN) has been limited and at times contradictory. Su et al compared CT-guided navigation vs. freehand technique and found that ARSN improved screw placement accuracy in the thoracic but not lumbar spine [78]. Chan et al performed a systematic review of small cohorts and found moderate evidence of decreased breach rate using ARSN as compared to freehand [79]. A recent review by Elmi-Terander et al demonstrated improved screw accuracy as well as a higher percentage of screws without a cortical breach in the ARSN as opposed to freehand group [9••].

While data has been mixed on the efficacy of ASRN for pedicle screw placement in the general population, there is compelling evidence that it can improve the accuracy of PS placement in more challenging conditions such as scoliosis in which screws are more likely to be mal-positioned [80••]. Comparing the results of scoliosis surgeries from one surgeon using both an ARSN and FH approach, the ARSN approach allowed the surgeon to increase the PS density and minimize the use of hooks, creating better constructs that will hopefully obviate the need for revision surgery [10••]. In a similar study, using ARSN as opposed to FH led to significantly lower rates of revision surgeries for mal-positioned PS (1.35% vs. 4.38%; p <0.01) [81].

However, there are significant limitations to accuracy that need to be addressed in future systems. Jin et al showed that accuracy decreases significantly as distance from the spinal process reference frame with the IR marker increases. The risk of screw misplacement doubles at 2 levels and quadruples at 3 levels away from the reference frame [80••]. There is also the risk of an operator or assistant inadvertently touching the reference frame during the procedure; if this causes any movement of the tracker, it will need to be repositioned and a repeat CBCT scan taken to re-register the scene [80••].

Conclusion

Orthopedics’ high utilization of pre- and intra-operative imaging combined with the relative ease of tracking rigid structures like bone as compared to soft tissues has made it the clear beachhead market for AR technologies in medicine. AR technologies have the promise of reducing cognitive burden on providers allowing them to operate with more precision and accuracy all while lowering the operative time and radiation exposure. However, it is unclear if current technologies deliver on this promise, and the ability to do so seems contingent upon continued technological progress. The accessibility, efficacy, and costs of these technologies will continue to improve, but their true cost-effectiveness has yet to be determined and remains a major roadblock to continued development. AR technologies appear to be at an inflection point with an increasing number of technologies moving from proof of concept to investigational clinical trials a trend that can be expected to accelerate as technological advancements solve workflow issues, lower costs, and expand access to these technologies.

Author Contribution

Andrew Furman – Idea generation, manuscript research, writing, and preparation

Dr. Wellington Hsu – Idea generation, manuscript editing

Data availability

Not applicable

Declarations

Conflict of Interest

Andrew Furman – No conflicts

Dr. Wellington Hsu – Advisory board member of Stryker, Medtronic, Asahi, Bioventus