Editorial: Challenges for the usability of AR and VR for clinical neurosurgical procedures

Abstract

There are a number of challenges that must be faced when trying to develop AR and VR-based Neurosurgical simulators, Surgical Navigation Platforms, and “Smart OR” systems. Trying to simulate an operating room environment and surgical tasks in Augmented and Virtual Reality is a challenge many are attempting to solve, in order to train surgeons or help them operate. What are some of the needs of the surgeon, and what are the challenges encountered (human computer interface, perception, workflow, etc). We discuss these tradeoffs and conclude with critical remarks.

The past two decades have seen a real attempt to use VR and AR interfaces to develop neurosurgical simulators and neuronavigational systems. Some utilize highly abstract OR-based scenarios, while others focus on low-level sensory-motor skils, which may even integrate haptic interfaces. The improvement of computer power has enabled researchers to use readily available commodity technology to design simulators. Therefore, a larger range of tools is at the surgeon's disposal, from expensive commercial large-scale neuronavigational systems to more adaptable platforms developed in-house. The design of such systems requires an open dialog between Neurosurgeons and Engineers; each important in the design effort to fully articulate the system use cases. Are you designing a tool to help train, plan or operate, or even to teach the patient?

Virtual reality can be used for training, planning or explaining a surgery to the patient; but obviously cannot be used to help in the real procedure, since it doesn't integrate real-time images. However, the “virtualization” of the environment can be attained whenever it is feasible to develop Augmented Reality modes – to augment the surgical field with important information that is not readily available to the eye of the surgeon, such as a segmented anatomy, a functional zone (for example areas that were found in fMRI), tracts (as seen with DTI).

Depending on what is the population using the device and it's goal, the technological developments associated to each simulator or navigation system, are not necessarily the same. For example, to train residents, one can use a generic model, while if you want to practice on a specific case, then the patient's own images need to be integrated. As the brain structures are not as easy to segment in an automated way (there are less contrasts than other anatomical areas in the body, and structures are intrinsically embedded to each other); therefore manual segmentation (or correction of a semi-automated way) is often required making it harder to have patient specific data readily available.

There are a number of good advances in VR for training and planning, such as a specific task trainers. For example: ventricular drain insertion trainers [1], simulators for training or planning the approach, such as the Dextroscope which has been used for planning surgeries for a decade now with good results [2, 3] and some simulators trying to train the whole procedure such as the neurotouch [4]. For intra-operative procedure planning and navigation, one of the most important success stories so far is the integration of the neuronavigation data into a microscope; which started 20 years ago, and has progressively been adopted for the last decade by multiple commercial vendors [5, 6]. However, since multiple surgeries are not done under the microscope (and even the procedures that require one only need it as a second step, after the planning of the craniotomy and the initial opening of the skin, the bone and the dura), it would be easier to have another device to help in planning surgery.

The use of goggles at the moment is impractical in the OR, since they are currently bulky. That will probably change with improved technology. The importance would be to have clear lenses, and not only a camera view, to be sure to always see real world if there is a technical problem. Other options that have been tried are a portable tablet or phone; or to project directly on the patient with web-cams [7–9]. However, there are unresolved challenges, the main ones being multiple points of view (the surgeon and assistants are not standing together, and therefore the point of view is different), depth perception and brain shift, among others. Studies looking at accuracy of AR focus on surface accuracy, and while some acknowledge the difficulty of depth perception, it isn't thoroughly evaluated.

In our experiments, we also have a good accuracy of surface [9, 10], but when assessing the targeting in depth we see that it is harder for the user [11]. In addition, there is one major issue that has not been solved in neuro-navigation, which is even more important when it is imported into AR, the brain shift occurring during a procedure. Even with small craniotomies under a microscope, the surgeons are seeing a shift or translation of the AR image compared to reality [5] therefore a model of brain deformation with some type of inexpensive intraoperative imaging for recalibration (cortical surface mapping with a camera, US image, or other modalities depending on the surgery) is necessary. Even with this problem ameliorated, if the anatomical structures are presented within an AR environment, the perceptual cues are not ‘absolute depth’, but rather, ‘relative depth’ with respect to other anatomical structures. Accordingly, when the user attempts to interact within the augmented environment, there are perceptual mis-matches induced by their own hands or tools in the augmented environment, corrupting the relative depth cues and over-riding the sense of absolute depth.

There are additional human factors issues that arise in conjunction with the development of such displays – in general we can categorize these as ‘issues for the integration of AR/VR into the surgical workflow’: A system that would be easy to use and not cumbersome, incorporating different points of view for multiple surgeons. Patient specific data should be easily imported and segmented (interactive segmentation enabling the surgeon's feedback). The accuracy in all planes (especially depth) not only on the surface, and a Brain deformation model to be incorporated with intraoperative reregistration. Last but not least, the ability to see real life directly and not through a camera (to be able to act if there is a technical issue, seconds are important if it is bleeding and can change the course of the operation).

Closing remark

The advances in Virtual and Augmented Reality in Neurosurgery are important in the last decades, and the collaboration of Neurosurgeons and Engineers in their development is key. One has to keep in mind that as long as a perfectly accurate system cannot be developed, it will be important to display the rendered workspace in a way which exposes the uncertainties and error bounds – as part of the workflow so the surgeon is aware of them and can take surgical decisions accordingly.