Opportunities and challenges of using augmented reality and heads-up display in orthopaedic surgery: A narrative review

Abstract

Background & aim

Utilization of augmented reality (AR) and heads-up displays (HUD) to aid orthopaedic surgery has the potential to benefit surgeons and patients alike through improved accuracy, safety, and educational benefits. With the COVID-19 pandemic, the opportunity for adoption of novel technology is more relevant. The aims are to assess the technology available, to understand the current evidence regarding the benefit and to consider challenges to implementation in clinical practice.

Methods & results

PRISMA guidelines were used to filter the literature. Of 1004 articles returned the following exclusion criteria were applied: 1) reviews/commentaries 2) unrelated to orthopaedic surgery 3) use of other AR wearables beyond visual aids leaving 42 papers for review.

This review illustrates benefits including enhanced accuracy and reduced time of surgery, reduced radiation exposure and educational benefits.

Conclusion

Whilst there are obstacles to overcome, there are already reports of technology being used. As with all novel technologies, a greater understanding of the learning curve is crucial, in addition to shielding our patients from this learning curve. Improvements in usability and implementing surgeons’ specific needs should increase uptake.

1. Introduction

With continuing advances in technology there has been significant interest in harnessing the benefits of augmented reality (AR) and heads-up displays (HUD) in the world of surgery.^1,2 Within orthopaedic surgery these developments have attempted to capture improvements pertinent to patient care whilst delivering benefits to the surgical team.

AR uses technology to deceive the perception of reality with additional information such as being in places other than the real one.³ Using an overlap of virtual and true reality an individual is able to remain aware of their environment and potential dangers.^4,5

Given the current COVID-19 pandemic, optimization of performance is paramount in the face of known and unknown occupational hazards. Surgical training has also changed, including constraints by re-deployment, and working time restrictions. Trainees have reported reduced opportunities to operate as the primary surgeon and have felt that their confidence in their surgical skills have been detrimentally affected.⁶ With this pandemic being the defining healthcare crisis of the modern era, the impact of it on surgical outcomes and training should be taken as an opportunity to implement technological advances to improve the current training environment. Adaptive examples of technological advancements deployed in the pandemic include surgical robots that follow voice commands to assist patient positioning in orthopaedics to reduce the number of staff required in the operating room, thus enforcing the social distancing doctrine.^7,8

Reviews by Lavadière and Jud et al. looked at the historical advances and applicability of AR specifically in orthopaedic surgery.^9,10 Potential for direct visualisation of radiological image overlays on the patient and intraoperative overlay imaging of preoperative planning were denoted however, significant challenges were observed. The impact of COVID-19 has been far-reaching but one upside has been the necessity for innovation and implementation of novel technology, especially where they can improve patient care with reduced direct contact.

AR and HUD use have been documented as early as 2007 in orthopedic surgery, with Ortega et al. assessing the impact and potential benefits of using HUD in spine surgery.¹¹ Subsequent studies have suggested that benefits could include improvement in surgeon's attention to the surgical field, reduction in potential harmful radiation exposure, reducing surgical field obstruction as well as through out-of-hospital services such as rehabilitation.^{5,11, 12, 13, 14}

Another significant area of potential benefit is the reduction of costs associated with surgery. With an average estimated running cost of £1200 per hour in the UK, operating theatres are amongst the most expensive assets in a hospital.¹⁵ Reduced operating times could decrease costs and provide added benefits of greater efficiency in surgery. Furthermore, the known correlation of increased operating time with infection rates is critical given the consequences for patients in the event of such complications.¹⁶ Given the reduction in elective operating enforced by the COVID-19 pandemic, optimising theatre efficiency has never been more relevant.⁶

Large technology firms have attempted to harness this potential in the world of healthcare. One of the earlier movers in this sector was Google Glass (Google, Mountain View, CA) which included options of recording procedures through the camera and also to display medical records, intraoperative imaging as well as for educational purposes.^12,17 Microsoft has also introduced its product Hololens (Microsoft, Redmond, WA) that has used augmented reality to overlay pre-operative models to visualize anatomy.^18,19 Additionally, alternative approaches have included occluding the visual field with reliance on the HUD. An example of this is Oculus Rift (Oculus VR, Menlo Park, CA), which has been successful in the commercial gaming sector. In surgery it has been trialed to create immersive virtual reality operating room educational experiences for trainee surgeons.²⁰

It is conceivable that given such potential benefits, there is a desire to translate the technology into the world of surgery. This review aims to provide an overview of the use of AR and HUD technology in orthopaedic surgery and highlight the benefits and challenges associated with their usage.

2. Methods

We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) framework to undertake and report this review of AR/HUD in Trauma and orthopaedic surgery.

2.1. Eligibility criteria

We sought to identify all original, prospective (randomized and non-randomized) and retrospective studies examining the use of virtual reality in orthopaedic surgery.

Inclusion criteria were any study reporting the use of AR/HUD in any recognised field of trauma and orthopaedic surgery, using any outcome measure. Any primary study design was considered.

2.2. Exclusion criteria

• 1)Reviews and commentaries
• 2)Unrelated to orthopaedic surgery (including other surgical and non-surgical specialties)
• 3)Use of other AR wearables beyond visual aids.

2.3. Search strategy

We performed a review of original articles using the following electronic databases: Medline, Embase, CINAHL and Emcare, Cochrane library and clinical trials registries. We searched the terms “Augmented reality” OR “AR”, “Heads-up display” OR “Head-up display” (with & without hyphen, will use truncation symbol ∗ to include plurals), “Smart glasses” AND “Orthopaedic surgery” OR “orthopaedic procedure” (using truncation for plurals, and wildcards (symbol = ?) for variant spelling. Only articles written English and published between January 1st^, 2000–January 1st^, 2021 were considered.

2.4. Study selection

Three researchers (J.H, P.P and D. G) independently reviewed all studies (title, abstract, and full text) that met inclusion criteria and extracted the relevant data. Any disagreements were resolved between reviewers through discussion.

The search strategy yielded 1004 articles²¹ (Fig. 1). After screening for duplicates using Endnote library (ENDNOTE X3, Thomson Reuters, Carlsbad, CA, USA), and applying exclusion criteria there were 42 papers that remained which met all inclusion criteria and were reviewed as part of the analysis (Fig. 1).

Fig. 1.

This figure shows the preferred reporting items for systematic reviews and meta-analyses flow diagram used for our search strategy.

3. Results

A summary of the papers is found in Table 1. The feasibility and potential benefits of additional use of HUD/AR technology alongside traditional surgical techniques were commented upon and amongst the benefits, the technology provided scope for increased surgical accuracy and precision, decreased radiation exposure for the patient and operating staff, decreased operative time, improved workload perceived by the surgical operator and educational uses and benefits.

Table 1.

Summary of analysed literature.

Category	Authors	Publication year	Key takeaway
Education	Logishetty et al.	2019	Using education of total hip arthroplasty with an AR headset as a feasible and valuable training tool as an adjunct to expert guidance in the OR
Education	Condino et al.	2018	Head mounted device is used to project and find targets on 3D printed mannequins, to improve education of total hip arthroplasty
Education	Pratt et al.	2018	AR can assist accurate identification of anatomical landmarks
Education	Pulijala et al.	2017	VR Surgery is highly useful for surgical trainees as a visualisation aid and for senior surgeons as a practice-based learning tool
Education	Armstrong et al.	2014	Google glass device used intraoperatively to improve communication, safety and efficiency of intraoperative and clinical care
Education	Ponce et al.	2014	Combining real-time AR and HUD (Google glass) in surgery enabled local surgeon to interact with remote surgeon within the local surgical field
General Orthopaedic	Chytas et al.	2019	Preclinical and clinical studies of AR in orthopaedic surgery could lead to improved surgical accuracy and decreased radiation exposure
General Orthopaedic	Fotouhi et al.	2019	Collaborative AR used to jointly co-localize a C-arm x-ray and a HUD has the potential to simplify hand-eye coordination for surgeons
General Orthopaedic	Ewurum et al.	2018	Surgical navigation systems using AR in orthopaedic surgery may enable precise decision-making in the OR by integrating surgical planning, instrument tracking and intraoperative imaging
General Orthopaedic	Gordon et al.	2018	Mixed reality and AR systems superimposing computer-generated images on a visual field could aid orthopaedic practice
General Orthopaedic	Ma et al.	2018	Integral videography based AR navigation in orthopaedics can improve hand-eye coordination for surgeons and enable greater ease of operating
General Orthopaedic	Smith et al.	2016	Improvements in localization, microsurgery and minimally invasive surgery have been made viable through navigational advancements
General Orthopaedic	Chang et al.	2016	HUD can improve perioperative care, intraoperative communication and documentation, surgical outcome as well as surgical training
General Orthopaedic	Pauly et al.	2015	R technology can support surgeons in the challenging task of understanding the spatial relationships between the anatomy, the implants and their tools
General Orthopaedic	Gavaghan et al.	2012	Portable image overlay used to display information on models, patients' skin and cadaver tissue could be an alternative to monitor displays for visualisation of surgical navigation data
Pelvis	Alexander et al.	2020	AR camera projecting CT results into a hip socket versus standard fluoroscopy during total hip arthroplasty showed more accurate positioning of the implant and was faster and easier to use
Pelvis	Fotouhi et al.	2019	Developing spatially aware visualisations of an object in an AR environment can pave the way for improving surgical performance and visuo-motor coordination in fluoroscopy-guided surgery
Pelvis	Lei et al.	2019	Superimposing an anatomical holograph of a 3D image onto a pelvic structure during total hip arthroplasty was feasibly demonstrated
Pelvis	Andress et al.	2018	A head mounted device and c-arm relying on visual marker for registration is used to enable real-time AR and maintain accuracy whilst reducing radiation and procedure time
Pelvis	El-Hariri et al.	2018	CT and ultrasound data of bone structure is used to generated 3D visualisation of a surgical scene via a HUD which may improve operator ergonomics, reduce fatigue and simplify hand-eye coordination
Pelvis	Liu et al.	2018	Using AR assistance and an automatic registration and limb tracking system to locate and drill holes during hip surgery may provide a more natural surgical workflow and seamless computer assistance
Pelvis	Ogawa et al.	2018	AR system to view an acetabular cup (reconstructed from CT) superimposed on a surgical field provided more accurate information regarding acetabular cup placement angle than the conventional method
Pelvis	Oliveira et al.	2018	AR allows the surgeon to represent holographic patient-specific anatomical information and surgical instruments in the physical world
Pelvis	Unberath et al.	2018	Head mounted device used to reposition c-arm with clinically acceptable accuracy can reduce radiation exposure
Pelvis	Fischer et al.	2016	Comparing the performance of surgeons and their task load using three different mixed reality systems during K-wire placements showed 3D visualisation to improve accuracy and efficiency
Spine	Auloge et al.	2020	AR/AI-guided percutaneous vertebroplasty appears feasible, accurate and safe and reduced radiation exposure
Spine	Edstrom et al.	2020	Radiation exposure during pedicle screw placement surgery in a theatre with AR navigation and cone beam CT imaging showed significant reduction with use
Spine	Saylany et al.	2020	Using a HUD to view x-rays intraoperatively can improve efficiency in surgical workflow and decrease disruption of focus
Spine	Carl et al.	2019	A microscope-based AR environment was successfully implemented for spinal surgery
Spine	Carl et al.	2019	Reliable microscope-based AR support is possible because of automatic registration based on intraoperative imaging
Spine	Gibby et al.	2019	HUD used CT images to guide vertebroplasty in opaque lumbar models to enhance surgical navigation and visualisation
Spine	Yoo et al.	2019	Both AR and VR offer significant promise in the realm of spine surgery with simulation systems requiring further development
Spine	Agten et al.	2018	Augmented reality-guided facet joint injections are feasible and accurate without potentially harmful needle placement in an experimental setting
Spine	U-Thainual et al.	2014	MRI-guided vertebroplasty using AR image overlay navigation in human cadavers feasibly allows accurate access and cement deposition
Spine	Abe et al.	2013	AR guidance technology can become a useful assistive device during spine surgeries requiring percutaneous procedures
Spine	U-Thainual et al.	2013	Needle insertion on a model using MRI-overlay system (MR-IOS) guidance allows accurate needle placement and is technically efficacious
Spine	Traub et al.	2006	Trauma surgeons perform a drilling task using different visualizations of an AR system to good effect
Trauma	von Ruden et al.	2019	3D and AR techniques used in trauma surgery may improve surgical complex fracture care
Trauma	von de Heide et al.	2017	Camera augmented C-arm was compared with traditional C-arms showing reduction in radiation exposure whilst keeping similar surgical times
Trauma	Chimenti et al.	2015	HUD use resulted in reduced operating time and radiation exposure compared to traditional techniques
Trauma	Zheng et al.	2008	A 3D image is formed from c-arm data to create AR images which can be used for minimally invasive osteosynthesis
Trauma	Ortega et al.	2008	HUD use resulted in significant reduction in the number of times the surgeon left the attention of the operative field and/or may have been exposed to fluoroscopy radiation

3.1. Accuracy

Multiple papers reviewed reported an improvement in accuracy when AR/HUD technology was implemented in comparison to traditional surgical techniques.^{11, 12, 13, 14,22, 23, 24, 25, 26, 27, 28, 29, 30} This was attributed to superior, logical visual guidance ensuring correct placement of surgical tools for the operator. An example is a head-mounted device projecting a hologram of the exact position a needle would need to be placed for spinal needle placement. Gibby et al. utilized computed tomography data to superimpose the virtual path of needle placement through an optical see-through head mounted display.³¹ For studies that found that these novel advances were “non-superior” in accuracy, it was concluded that AR techniques remained non-inferior and within safe ranges for surgical practice.^13,23 Shah et al. showed only 1 of 40 AR-guided facet joint injections using a HUD performed in a single-center study³² was classified as incorrect even though it missed the facet joint space by 2 mm. However, the needle tip was inside the joint capsule and may have been classified as appropriate depending on the analyzer. As such, the potential for improving accuracy and safety of orthopaedic procedures appears to be a significant area of benefit in the adoption of such technology.^25,26

3.2. Time/efficiency

The procedure time significantly decreased when AR was used in several studies compared to traditional surgical methods.^{33, 34, 35} Alexander et al. showed AR-guided trocar deployment in percutaneous vertebroplasty took longer than standard fluoroscopy (642 s; range 300–963 with AR vs. 336 s; range 240–438; p = 0.001 with standard fluoroscopy). However, there was a considerable reduction between the first and final five time intervals suggesting that in this case increased confidence in utilization and reduced dependence on additional screening as the surgeon became more familiar with using AR.³⁶ This reduction in procedural time during AR technology use was also highlighted by Armstrong et al. showing that time taken for AR procedures decreased with experience.³⁷ Such findings highlight the importance of understanding the learning curve associated with new technology and the need for longer-term follow-up with larger case volumes.

3.3. Radiation exposure

A common aim of AR technology research focused on decreasing unnecessary radiation exposure to operating room personnel. Ponce et al. compared standard intraoperative fluoroscopy with using AR to give 3D surface reconstruction through digitally reconstructed radiographs showing a clear reduction of radiation exposure, whilst enabling real time visualisation without the need for continuous fluoroscopy.³⁸ Pauly et al. measured the radiation dose to operating room staff in a hybrid operating room using a ceiling-mounted c-arm with an AR Surgical Navigation system. Staff wore an active personal dosimeter on their chest, which measured their radiation exposure during 20 pedicle screw placement procedures, and this was compared to a reference ‘worse case’ radiation exposure level from a dosimeter attached to a c-arm. The staff-to-reference dose ratio for each procedure was 0.05% and after several procedures this decreased further to less than 0.01% again showing the improvement with experience using the technology.³⁹

3.4. Workload

Workload load questionnaires such as the validated Surgical Task Load Index and System Usability scores showed a significantly decreased workload using AR-assisted orthopaedic procedures.^38,40 Gavaghan et al. showed users rated the Surgical Task Load Index for their AR-technique to be less than half in comparison to the standard fluoroscopic technique (P = 0.01). The parameters on this rating scale included mental and physical demands, situational stress and distractions.⁴⁰

3.5. Educational benefit

The use of AR/HUD as an educational aid in orthopaedic surgery is a large potential growth area. Armstrong et al. showed how a HUD (Google Glass) was used for intraoperative education and communication between a junior operating surgeon, who wore the headset and a senior remote surgeon, who viewed the operator's surgical field via a desktop computer or mobile device using the Google Hangouts application.⁴¹ Once screen sharing the live recording from Google Glass, the operating surgeon was able to point to areas in their visual field and the remote surgeon could point using a mouse on a desktop computer. Senior surgeons were also able to use the screen share feature to give detailed descriptions of suture techniques that the operator could visualize. This application allowed for more time-efficient communication between the learner and the remote teacher. In addition, the learner was able to have more hands-on independence when carrying out surgery in a real clinical setting. Participants reported that the learner had more confidence in their independent techniques with the safety-net of the teacher's virtual presence.⁴²

A further application of HUD was described in being used to carry out a total shoulder arthroplasty. Google Glass was integrated with a Virtual Interactive Presence and Augments Reality (VIPAAR) system. VIPAAR allowed the operating surgeon to visualize the hands or tools of the assisting remote surgeon as a hybrid image combined with the surgical field through the HUD. The surgery was followed through with positive surgical outcomes and patient satisfaction on follow-up, however, the operative time of the surgeon increased by 45 min highlighting a potential disadvantage associated with this use of novel technology.⁴³

Logishetty et al. tracked 24 medical students allocated to four sessions of either one-on-one training from a hip arthroplasty surgeon or AR training using the Microsoft HoloLens headset was compared. Participants received surgical training to position an acetabular cup in 6 different orientations on an opaque hip model and the accuracy, trainee perceptions and potential training role of the AR headset was compared to hands-on expert training by a surgeon. During the training sessions, participants receiving AR training performed more accurate cup orientation with an error of 1° ± 6°, in comparison to the surgeon-trained group whose error of orientation were 6° ± 4°. However, at the final assessment, there was no significant difference in the error between the AR-trained and surgeon-trained groups (mean difference 1.2°, 95% CI -1.8 – 4.2°, P = 0.281). Post-training questionnaires showed that 11 of 12 participants would have preferred a combination of expert-guided teaching and AR-guided unsupervised learning.¹³

4. Discussion

The application of AR/HUD technology in orthopaedic surgery is still in its infancy and requires further modifications to justify its safety and efficacy for the clinical environment.^44,45 Several barriers have hindered the adoption of AR/HUD in trauma and orthopaedic surgery, such as unfamiliarity with technology and a convoluted overhaul of established clinical pathways.^46,47 The unprecedented pressure placed on healthcare systems during the COVID-19 pandemic has led to the rapid implementation of innovative new patient pathways that we have not had time to fully evaluate before using them and that are likely to persist after the pandemic is over. However, to ensure we have an evidence-based practice, we will evaluate the best-available evidence for the continued use of telemedicine in the post-COVID-19 era.

4.1. Technological shortcomings

Understandably, there is a learning curve associated with the novel use of AR technology and this was seen in multiple studies.^13,23,48 A range of calibration and registration techniques were used to align AR images with the anatomy of opaque models as well as cadavers.⁴⁹ These initial preclinical studies considered critical factors of real-time surgery such as the effects of patient respiration, occluding soft tissue and anatomical structures and unanticipated variances in anatomy.⁵⁰ In addition, some studies using HUD head mounted devices found that the projected images deviated slightly if the headset was moved.³¹ Although human error in surgical procedures is inevitable, any further error introduced through innovative technology use will always because for concern. Improvements including better software will be required to mitigate holographic instability, particularly in orthopaedic surgery where small positional errors in instrumentation can be catastrophic to patient outcomes.

4.2. Usability

Many of the studies used head-mounted devices such as Google Glass to visualize AR in orthopaedic surgery, which was associated with certain limitations. Firstly, several users reported visual discomfort after wearing the headset for a prolonged period. This was partly due to visual fatigue and vergence-accommodation conflict, where the user needed to accommodate their eyes to a fixed focal distance of 2 m away whilst the depth of the image varied on the application.¹⁹ Furthermore, some head-mounted devices still moved the user's attention away from the surgical field, if the AR images were not directly overlaid onto anatomical structures. This would require addressing as it could be distracting for the surgeon.^42,51

Concerns also exist about the clarity and contrast of AR images and whether they are disruptive in the surgical setting.⁵² Some AR images, such as image overlay systems and holographic projections are more difficult to see in darker environments, which can be a problem in operating rooms. More contrast and flexible colour assignments must be developed to ensure both actual reality and AR images are visible to the naked eye or through head-mounted devices as well as developing solutions for issues such as colour-blindness.¹⁹

Finally, in light of challenging environment faced by surgeons the uptake of such technology would be increased by purported or realisable benefits to them. Easing the workload of surgeons who frequently face emergency and high-risk operations is crucial to improve patient outcomes and surgeon well being.^38,40 Radiation exposure to senior orthopaedic trainees and surgeons is known to be high.⁵³ The utilization of technology to reduce operating time and fluoroscopy exposure is important, given the stagnation of innovation beyond use of radio-protective coverings.⁵⁴

4.3. Five-year forecast

Future studies should statistically quantify the overall cost benefit and liabilities of AR technology that has already been clinically implemented, as insufficient analysis has been carried out on this.⁵⁰ The reduction in use of such technology, with commercial units costing under 1000 US dollars, have shown to be implemented in surgical scenarios with the benefits of reduced disruption of focus, reduced surgical time, reduced physical strain and improved surgical satisfaction.⁵⁵ Despite this, the cost-effectiveness of routine utilization of such technology should be further investigated before wider adoption.

As several studies showed the learning curve effect with the use of AR, this merits further quantification.^55,56 Although these reports demonstrated an improvement in procedure times (even with small volumes), the long-term effect needs explored. Furthermore, the opportunity for such technology to improve the learning experience of orthopaedic trainees is one that should be investigated.³⁷

Since most of the studies focus on preclinical model and cadaveric scenarios the next steps would require higher fidelity options being used in the clinical setting. These would be expected to compare current techniques with the improvements to be gained through their use in reducing costs and operative time, improving accuracy and most importantly, patient outcomes to facilitate wider adoption. Also, input from the surgeon to understand their needs and challenges is needed in a fast-moving sector that ultimately requires optimized comfort and adaptation for the user to facilitate wide uptake.

It would be remiss to fail to acknowledge the limitation of this review being the heterogeneity of the studies that were included, which precluded a meta-analysis of results. As there were no standardised outcome measures, methodologies and even a mix of qualitative and quantitative data reporting this is a shortcoming and one that would hope to be better addressed in future research.

5. Conclusion

The current state of AR and HUD technology use in orthopaedic surgery offers significant potential benefits but is not a panacea to challenges in the operating theatre. Due to the rapidly evolving nature of technological advancement, it is likely that the literature search conducted could be outdated within a short period of time. Indeed, given the pace of change enforced by the COVID-19 pandemic it is conceivable that novel technology development and implementation will accelerate. With the large amount of potential benefit to patient safety as well as surgeon training, wellbeing, and safety it is inevitable that there will be significant interest in this area coupled with the commercial benefits stood to be gained from the bioengineering giants. Nevertheless, it will remain important to ensure that safety and improved outcomes underlie any such adoption of innovations and the pace of change is tempered with optimal outcomes in mind.