Augmented reality in neurosurgery

Abstract

Neurosurgery is a medical specialty that relies heavily on imaging. The use of computed tomography and magnetic resonance images during preoperative planning and intraoperative surgical navigation is vital to the success of the surgery and positive patient outcome. Augmented reality application in neurosurgery has the potential to revolutionize and change the way neurosurgeons plan and perform surgical procedures in the future. Augmented reality technology is currently commercially available for neurosurgery for simulation and training. However, the use of augmented reality in the clinical setting is still in its infancy. Researchers are now testing augmented reality system prototypes to determine and address the barriers and limitations of the technology before it can be widely accepted and used in the clinical setting.

Introduction

Neurosurgery is a medical specialty that deals with the surgical and non-surgical management of patients with medical problems affecting the brain, spine and peripheral nerves [1]. Examples of neurosurgery procedures include brain tumor resection, spinal fusion and cerebral aneurysms. Malone et al. [2] posited that this field of medicine has an ever-increasing complexity that has a very low threshold for errors. They stated that even minor errors during surgery carry dire consequences that could result in patient harm.

“Augmented reality (AR) is a new technology that superimposes three-dimensional virtual objects or text to real objects” [3]. Unlike virtual reality (VR), where the entire scene is completely synthetic and computer generated, AR shows both real and virtual objects, thus creating a semi-immersive environment for the user. Giglioli et al. [4] posited that AR enhances the user perception of reality by inserting virtual content into the real world and running it in the same presentation and in real time. They further explained that AR is a set of tools and techniques that add information to the physical reality.

One of the early adopters of AR technology in healthcare is nerosurgery. This medical specialty is heavily dependent on images for preoperative planning and intraoperative neuronavigation [5]. Pandya et al. [6] explained that the current neuronavigation system only displays two-dimensional images (axial, coronal and sagittal) on the computer screen. “The surgeon must perform the 2D (image) to 3D transformation in his mind and also project the envisioned data onto the view of the patient” [6]. However, every time a neurosurgeon has to perform this task, he/she has to constantly switch views between the surgical field and the computer screen, thereby creating constant interruptions and disrupting the surgical workflow.

Currently, AR technology is being used and tested in neuro-oncology, spinal surgery and neurovascular surgery. Neurosurgery uses AR for simulation and training. The application of AR technology in neurosurgery can boost the confidence of neurosurgeons in performing delicate neurosurgical procedures because they have a great tool that will guide and serve as a virtual map as they dissect and explore the patient’s anatomy.

Significance and purpose

Augmented reality technology is widely used and tested in various specialties in healthcare. Neurosurgery is one of the early adopters of this technology. However, the current adoption and accepted use of AR technology in neurosurgery is limited to simulation and training. This paper will investigate the state of science of AR technology application in various neurosurgery sub-specialties and examine the barriers and limitations, cost, equipment and the future uptake of clinicians in this novel technology.

Methods

In order to investigate the current application of AR technology in neurosurgery, a literature search was conducted from the following databases: Embase, Web of Science, CINAHL, Scopus, Google Scholar, PubMed and Psychinfo. The keywords used for the search were: Augmented reality and Neurosurgery. A total of 196 articles were retrieved, 25 of which were reviewed, and only 10 primary research papers were included in this review.

Included articles were grouped by neurosurgery sub-specialties: neuro-oncology, spinal surgery and neurovascular surgery. Articles that focus on the use of AR for training and simulation were also grouped.

Application of augmented reality by neurosurgery specialty

Augmented reality technology is utilized for simulation and training of neurosurgery residents and is applied in various neurosurgery subspecialties. Below we discuss in detail the application of AR in neurosurgery.

Neuro-oncology

Neuro-oncology is a subspecialty of neurosurgery that deals with the management and treatment of brain tumors. The American Brain Tumor Association [7] reported that the Central Brain Tumor Registry of the United States estimated that 69,720 new cases of primary brain tumors were diagnosed in 2013. According to Johns Hopkins Medicine (JHM) [8], the primary goal of surgery for metastatic brain tumors is to debulk and remove the entire tumor while maintaining neurological function.

The present practice involves the neurosurgeon relying heavily on image-guided neurosurgery (IGNS) to achieve this maximum resection of brain tumors. Deng et al. [9] explained that IGNS uses preoperative images of the patient to track the relative position of the tumor against the preoperative images during surgery. However, they posited that with the use of IGNS, every time the surgeon wants to check his relative position on the patient’s brain and the preoperative images, he/she has to switch views between the computer screen and the surgical field [9]. The preoperative image displayed on the computer screen is divided into three separate images (sagittal, coronal and axial), and the surgeon has to mentally merge these images in his/her brain to create a single three-dimensional composite image [6]. They claimed that the frequent switching of views during the surgery disrupts the surgeon’s workflow.

Several AR system prototypes are currently being tested specifically for the management of brain tumors. Inoue et al. [10] used an AR system prototype with a web camera to superimpose the image of the brain tumor in the patient’s skull and dura. Deng et al. [9] used a wireless tablet computer AR neuronavigation system for surgical planning and surgery of two clinical cases in Shanghai, China.

The AR system prototype was also used for preoperative planning to help surgeons in determining the “best approach” for the surgery [11]. Mahvash [12] tested an AR system that integrate the patient’s magnetic resonance (MR) and computed tomography (CT) images into the surgical field to improve the surgeon’s orientation and situational awareness during surgery.

Neurovascular surgery

Neurovascular surgery is a subspecialty of neurosurgery that evaluates, diagnoses and treats patients with cerebrovascular disease. Mitha et al. [5] explained that this neurosurgery subspecialty is strongly dependent on imaging, as surgeons rely on these images in their intraoperative decision making.

One example of cerebrovascular disease is arteriovenous malformation (AVM) [13]. According to JHM [13], AVM is a congenital disease that is characterized by an abnormal tangle of vessels in the brain or spinal cord. These vessels typically have a thin wall that could rupture and bleed any time, causing stroke and death.

The current and established management of AVM is microsurgical resection, which involves performing a craniotomy and clipping/resecting the AVM [13]. Kersten-Oertel et al. [14] posited that AVM surgery is often time consuming because the surgeon has to carefully dissect and identify all the involved blood vessels in the AVM including the feeder arteries, the nidus and the drainer veins. They explained that surgeons use a microscope to magnify the area of interest. However, the image from the microscope does not provide any information to help identify these blood vessels. They claimed that the complexity of this surgical procedure often results in longer operative time and a large surgical incision and skull opening to enable the surgical team to find and resect the AVM.

Kersten-Oertel et al. [14] developed and tested an AR system prototype that provides mixed reality visualization during AVM surgery for surgeons. This AR system incorporated chromadepth rendering and color coding of blood vessels and was interfaced with the surgical microscope in the operating room. The chromadepth feature provides information on the distance of the blood vessels from the brain surface, and the color coding feature helps surgeons in identifying blood vessels being visualized under the microscope. They found that the color coding feature of the AR system proved to be very useful during surgery since arterial and venous blood vessels in an AVM look similar because they all contain arterialized blood. The AR system has also the ability to control the alpha blending of the images, giving the user the control to adjust the translucency of the superimposed virtual image [14].

Spinal surgery

Spinal surgery is a subspecialty of neurosurgery that manages, evaluates and treats spinal disorders [15]. Spinal disorders may include degenerative conditions, deformities, spinal tumors and trauma/spinal injuries.

Abe et al. [16] tested a virtual protractor with an augmented reality (VIPAR) system for percutaneous vertebroplasty to 40 spine phantoms in the laboratory and 5 clinical patients in Enewa Hospital, Enewa, Hokkaido, Japan. The VIPAR system is designed to visualize the vertebroplasty needle trajectory in a three-dimensional space during the procedure.

Percutaneous vertebroplasty is a medical procedure to treat a fractured spinal vertebra that causes pain and loss of function [17]. During the procedure the surgeon injects medical grade cement into the fractured vertebra. This minimally invasive procedure heavily relies on the use of C-arm fluoroscopy to guide the neurosurgeons in determining the needle trajectory during the procedure. Abe et al. [16] explained that this is a relatively easy and safe procedure; however, inadequate needle placement has the potential to cause neurovascular injury and cement leakage. Johns Hopkins Medicine [17] also posited that risk factors associated with vertebroplasty includes hemorrhaging, rib or other surrounding bone fractures and leakage of cement outside the bone.

Abe et al. [16] found in their 40 spine phantom trials that the error of insertion angle of the vertebroplasty needle during the procedure was significantly improved compared to the current modality. In the clinical trial, all 5 VIPAR assisted percutaneous vertebroplasty procedures were successful, with no spinal pedicle breach or leakage of cement.

The VIPAR system was able to guide neurosurgeons in obtaining an ideal entry point and needle trajectory during a percutaneous vertebroplasty procedure and was superior to the current method of percutaneous vertebroplasty.

Neurosurgery training and simulation

In neurosurgery, AR technology is mostly applied and currently used for training and simulation of neurosurgery residents and surgeons. According to Malone et al. [2], “Computer-based surgical simulators create a no-risk virtual environment where surgeons can develop and refine skills through harmless repetition” (p. 1105). They explained that simulation-based training in neurosurgery has benefitted neurosurgeons since neurosurgery carries a very small margin of error during surgery.

Fargen et al. [18] posited that simulator-based training is an important adjunct to training on real patients among neurosurgery residents because of the latest restriction in the resident’s duty hours and expanding training requirements. The simulator-based training also provides neurosurgery residents a safe and realistic environment that may result in the improvement of their skills and performance during real surgery, which eventually improves patient outcome [19].

ImmersiveTouch (IT) is one of the few established companies that create AR simulation-based surgical training systems for healthcare and academic institutions. It provides high fidelity simulators with haptic technology that replicates a real surgery experience for residents [20]. The IT system can be loaded with applications for cranial and spinal procedures. The currently available IT cranial and spinal applications are ventriculostomy, bone drilling percutaneous treatment for trigeminal neuralgia, lumbar puncture, pedicle screw placement and vertebroplasty.

Equipment and setup

The basic equipment needed for AR setup in neurosurgery comprises personal computers (PC), a monitor display, a camera, tracking tools and three-dimensional image editing software [9–12, 14, 16]. Personal computers are used for preoperative editing of MR and CT images and for projection of the augmented image to the real scene. Monitors are used to display augmented images for the surgeon and the surgical team. Inoue et al. [10] used a 20-inch monitor to display navigational information and the augmented scene during skin incision and craniotomy. Mahvash and Tabrizi [12] used a projector to display the augmented brain tumor image on the skull of a head phantom, while Abe et al. [16] used a video see-through head mounted display (HMD) to visualize the vertebroplasty needle trajectory and entry point on the VIPAR system. Deng et al. [9] used a tablet computer (Samsung Galaxy note) to display the overlaid projected image directly on the patients’ intracranial anatomical structure.

Tracking tools are important components in a neurosurgery AR setup. Tracking tools are required for the proper alignment and orientation of the augmented image to the real scene. To achieve correct alignment and orientation, the camera, markers (e.g. ARToolKit, fiducials, spheres and reference frames) and the virtual image should be registered accurately before the start of surgery [12].

Limitations and barriers

Use of AR in neurosurgery revealed some limitations and barriers on the use of the technology. The identified limitations and barriers were system latency, image registration accuracy and alignment, and tissue and organ movement.

System latency

According to Wu et al. [21] a surgical procedure using AR technology may result in a delay in the display/projection of the images during AR neuronavigation, thereby providing inaccurate information to the surgeon and could cause the patient harm.

In the study of Deng et al. [9], the researchers noted that there was a delay in the projection image of the tablet screen when the tablet was being moved. They also noticed that the projection delay increased when they moved the tablet in a faster motion. However, they explained that in the real setting the tablet moves only at a low speed, giving plenty of time for the augmented images to align in real time, and the use of high performance wireless routers can increase data transfers and solve the problem.

Image alignment and accuracy

Augmented reality registration is the manual process of aligning tracking, imaging and user input devices with the real user surrounding or scene [21]. In neurosurgery, almost all AR systems and prototypes use passive markers for alignment and calibration. Examples of these markers are reflecting fiducial spheres and the ARToolKit.

Abe et al. [16] encountered image alignment problems when using the VIPAR system during a percutaneous vertebroplasty procedure. They posited that the VIPAR system does not allow accurate registration with CT-based navigation systems and therefore cannot be used for complicated spine deformities. The VIPAR system also failed to accurately show the position of the tip of the needle during the experiment. The team had to confirm the tip’s location by fluoroscopy. They explained that this is a big setback for the VIPAR system, since one major complication of percutaneous vertebroplasty is the leakage of cement outside of the vertebral body.

Deng et al. [9] stated that alignment errors can occur during registration, tracking and calibration of different devices. They also added that optical distortions from camera lenses can also contribute to the problem.

Tissue movement

One of the major challenges in using AR technology in healthcare is tissue and organ movement during surgery [21–23]. One reason why neurosurgery is one of the early adopters of AR technology is ecause most of their procedures require rigid fixation [23]. However, Shuhaiber [3] claimed that tissue and organ movement can still occur during a neurosurgery procedure due to cerebrospinal fluid leakage and gravity during intraoperative positioning changes. He explained that these small tissue movements can increase the AR alignment error intraoperatively.

Discussion

Neurosurgery is a medical specialty that embraced and adopted AR technology during its infancy stages. Even before AR, neurosurgeons extensively used image guidance technology to reduce the invasiveness, improve quality and shorten the operation time for neurosurgical procedures [8].

Presently, the application of AR in neurosurgery is mostly experimental. Researchers use AR prototypes to test the accuracy and usability of this novel technology in neurosurgery. Some of these AR prototypes are even made of consumer products, such as web cameras, projectors and tablet computers, which shows that AR technology prototypes can be made from inexpensive and consumer grade materials [8–10, 12]. Since the AR prototype can be built from consumer grade materials and free software (e.g. ARToolKit), designing and building these AR prototypes can be done at a relatively low cost.

Augmented reality simulator

Augmented reality technology is currently established and is commercially available for neurosurgery simulation and training.

Gelinas-Phaneuf et al. [24] stated that in order for a simulator to be integrated into formal training, it must demonstrate usefulness and appropriateness. This can be achieved by subjecting the simulator to different validation steps: namely face, content, construct and concurrent validity tests. Face and content validity show that the simulator is realistic and targets the training skill that is required. Construct validity measures the user scores obtained by the simulator to discriminate a novice from an expert. Concurrent validity measures the skills acquired by the trainee in the simulator, and this reflects the trainees’ performance during actual surgical procedures. The AR simulator used in the study has only demonstrated face and content validity at present for its neurosurgery procedure modules [3, 25]. Further research and testing are needed to establish AR simulators’ reliability and validity in neurosurgery training and simulation. Therefore, healthcare organizations should assess their priorities and resources before investing in these simulators for their neurosurgery residents and surgeons.

Uptake by practice and practitioners

Augmented reality technology shows a great potential for widespread adoption in all neurosurgery subspecialties. Several AR system prototypes that were tested resulted in positive views among neurosurgeons, and they are willing to integrate this novel technology into their practice. In order to integrate AR technology into the current neurosurgical workflow, Mahvash [12] suggested adding AR technology as an enhancement to the current neuronavigation system and surgical microscope.

Alaraj et al. [20] posited that AR simulation with haptic feedback holds great promise in providing a safe and consequence-free operative learning for neurosurgery residents. Deng et al. [9] and Abe et al. [16] stated that AR systems could potentially become routine for neurosurgery practice in the future.

Schaller et al. [26] in 2008 explained that molecular imaging and use of positron emission tomography (PET) is a novel technology that can identify both the physical nature and the most malignant part of the tumor based on the tumor’s microvascular structure and metabolic activity. They further explained that to make an accurate diagnosis stereotactic biopsies should be taken from the most malignant part of the tumor. The authors believe that further experiments should be done to exploit the possibility of using PET images in AR to improve tumor biopsy localization and resection.

Cost

Every time healthcare organizations plan to implement new and innovative technology in their setting, cost is most often a major factor to be considered. According to Callahan [27], medical technology is one of the leading reasons why healthcare costs continually rise. He posited that 40% to 50% of the annual cost of healthcare can be traced to implementation of new technologies in the clinical setting. He explained that healthcare organizations are sometimes pressured to adopt these new technologies with a questionable outcome because, “Patients expect it, doctors are primarily trained to use it, the medical industries make billions of dollars selling it, and the media loves to write about it” [27] (p. 80).

Evaluation whether AR is cost effective and cost beneficial to use in the clinical setting is difficult to assess at this time, because most of the clinical and experimental applications of AR in neurosurgery are using AR system prototypes. The current practice involves researchers using consumer grade, inexpensive equipment and materials to experiment with AR technology.

Implications for nursing informatics

Augmented reality technology will impact the way clinicians provide patient care in the future. Recently, Evena Medical released new AR computer-powered glasses named “Eyes-on” to help nurses in searching for and correctly piercing a patient’s vein during an intravenous (IV) line insertion [28]. They claim that the AR glasses will help in IV line insertion of patients who have undergone multiple and frequent IV puncture, and patients whose veins are miniscule or hard to find such as babies and heavy patients. Pelegrin [29] reported that Beth Israel Deaconness Medical Center recently tested the “Google glass” wearable AR computer in their emergency department to stream patients’ vital signs, lab results, problems and other data in the clinician’s field of vision. Sheffield Hallam University [30] uses a tablet computer AR system to superimpose videos of patients, played by actors, onto their training manikins to provide nursing and midwifery students real patient experience during their simulated patient interaction sessions.

The American Nurses Association [31] stated, “Wearable computing is a revolutionary paradigm that shatters myths about what computers are and how they should be used (p. 52)”. They also added that these devices would someday act as intelligent assistants for data collection and analysis. The example above clearly demonstrates the power of AR technology by providing clinicians the right information at the right time. It is therefore very important for the informatics nurse specialist and innovators to be involved in the early development, design and usability testing of these devices to ensure fast clinician acceptance of this novel technology once it is implemented in the clinical setting in the future.

Conclusions

Augmented reality technology holds a promising future and a big potential for application in neurosurgery. The body of literature shows that AR offers a new and more precise way of preoperative planning and intraoperative neuronavigation for neurosurgeons. It has also provided another avenue where neurosurgery residents can repeatedly practice in performing realistic simulation on various neurosurgical procedures in a safe environment.

There is also evidence that AR technology is spreading to other fields of healthcare, including nursing. We believe that informatics nurse specialists and innovators should involve themselves in the early design and development of these AR medical devices and systems to ensure the devices’ usability, which will significantly help in the fast and easy adaption of this novel technology as it is implemented in the clinical setting.

Neurosurgeons strongly suggest that AR technology should be added as an enhancement to the current technology (e.g. surgical microscope and neuronavigation system) in the operating room in order to fit with their current workflow.

The body of evidence also suggests that in order for AR technology to be successfully integrated into current neurosurgical practice, collaboration among academic researchers, industrial players and clinical users is needed to pull their knowledge, expertise and resources together in order to transition AR technology from the laboratory to the clinical setting.

Conflict of interest

The authors declare no conflict of interest.