Augmented and virtual reality in surgery—the digital surgical environment: applications, limitations and legal pitfalls

Abstract

The continuing enhancement of the surgical environment in the digital age has led to a number of innovations being highlighted as potential disruptive technologies in the surgical workplace. Augmented reality (AR) and virtual reality (VR) are rapidly becoming increasingly available, accessible and importantly affordable, hence their application into healthcare to enhance the medical use of data is certain. Whether it relates to anatomy, intraoperative surgery, or post-operative rehabilitation, applications are already being investigated for their role in the surgeons armamentarium. Here we provide an introduction to the technology and the potential areas of development in the surgical arena.

Introduction

Surgeons are regularly on the lookout for technologies that will enhance their operating environment. They are often the early adopters of technologies that allow their field to offer a better surgical and patient experience. Examples of such innovations include fibre-optics which allowed for the advent of minimal access surgery and robotic surgery which lead to developments of systems such as the da Vinci robot over a decade ago (1). These tools have historically come at a considerable cost but over the last decade have become cheaper and increasingly available. New computational paradigms are emerging with rapid advancement and miniaturisation of real-time visualisation platforms. Smartphones are now commonplace with microprocessing powers rivalling desktop computers. The near ubiquitous use of smartphones by doctors has driven an increasing use of technology in healthcare (2). Medical applications (3) and instant access to web-based resources now guide clinical practice. This has allowed for the development of powerful wearable technologies that can provide high fidelity audio-visual data to the surgeon whilst operating. The recent emerging role and integration of virtual reality (VR) and augmented reality (AR) in healthcare is ripe for translation into this data rich field and warrants consideration for future applications to enhance the surgical experience. This viewpoint aims to review the applications, limitations and legal pitfalls of these devices across surgical specialties and imagines what the future surgical landscape may reveal.

Augmented reality

AR is the addition of artificial information to one or more of the senses that allows the user to perform tasks more efficiently. This can be achieved using superimposed images, video or computer generated models. Examples include the AccuVein (AccuVein Inc., NY, USA), a projector-like device that displays a map of the vasculature on the skin surface (Figure 1) (4) or Google Glass (GG) which is a head mounted display (HMD) with generated objects superimposed onto real-time images (Figure 2). The feedback can involve auditory augmentation, haptic feedback, smell and taste (5). Augmentation of reality has been used in surgery for many years especially in neurosurgery where stereotactic surgery has used the combination of radiographic scan data in stored or real time acquisition to allow accurate and safer “neuronavigation” (6). Traditionally this has been through a “head’s up” visualisation method (Figure 3) where the visualisation data is on a screen as commonly seen in smartphone based broadcasting and video games (7). Advances have been made in image registration and video tracking that allow for the displays to track points in the field that match the orientation and scale of the device to give accurate superimposition (8). This application is of greater benefit in real surgery compared with VR devices as the technology can be “see through”. The concept of a headset with superimposed display was introduced by Ivan Sutherland to the military in 1965 (9) and consisted of a head worn display and an image generation subsystem. These were cumbersome, heavy and expensive hence the future implementation of these technologies in healthcare need to be cost-effective, versatile and comfortable to ensure acceptance (10).

Figure 1.

Use of Accuvein to image veins on patient.

Figure 2.

Use of Google Glass in theatre.

Figure 3.

Difference between see through display and head’s up display.

A refreshed interest in accessible AR came after the announcement of GG which gained publicity up until January 2015 when production ceased (11). This technology benefited from a lightweight wearable superimposed viewing screen and high resolution video camera and features similar to that seen on a smartphone such as wireless and cloud accessibility (12). The open source development platform for the device allowed creative medical and surgical application and subsequently a number of other similar devices specific for surgical enhancement have been developed (13). Trials of GG in other surgical specialties have demonstrated good clinician satisfaction and great optimism for its integration into surgical practice (14). It is clear that there are many exciting opportunities and applications of AR to surgical healthcare.

Virtual reality

VR generates an immersive, completely artificial computer-simulated image and environment with real-time interaction. VR has been used for endoscopic training and assessment for more than a decade (15). One of the earliest platforms was Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR) (Mentice Medical Simulation, Gothenburg, Sweden) for endoscopic training (16). A recent meta-analysis of randomised controlled trials showed a reduction in operative time, error rate and accuracy when VR training is employed for new trainees with no prior experience or when supplemented with standard laparoscopic training (17). VR 3D technology is being incorporated into simulation-based training. This technology would surpass the fidelity of current distance learning packages and 2-dimensional videos with creation of a near true experience from the point of view of the operator. An example of this is work funded by the MOVEO Foundation which uses the Oculus Rift platform to put trainees into the virtual view of the operating surgeon. This provides the viewer with a first-person view of the procedure being performed. The MOVEO Foundation has produced videos in several orthopaedic procedures (18). Oculus Rift is one of the first mass-produced, recently released head-mounted VR devices that is particularly popular amongst developers and has a strong financial backing from Facebook (19).

Many other headsets are being developed which means user support will increase as the installer base is established (Table 1). Sony’s VR headset is likely to be one of the biggest platforms into the virtual world due to its low price point and preinstalled user base of PlayStation 4 owners. HTC vive offers a higher quality immersive environment and benefits from virtual space tracking which allows participants to move around a virtual room provided real living space is available. Most of the VR devices offer some tools to interact with the virtual environment and those familiar with endoscopic and robotic surgery may find these peripherals intuitive. The fidelity of the equipment may need to be tailored to surgical simulation to provide the most immersive experience. However, what is unique about the nature of this current vogue is that the consumer cost is accessible and therefore is in a good position to succeed. This may eventually form the cornerstone of future assessment, revalidation and continued training. Surgical training tools, like ‘Touch Surgery’ that are currently on smartphone and tablet may find themselves on a VR platform and be a significant change to how we learn and practice surgeries (20).

Table 1. Augmented and virtual reality devices.

Devices	Specifications
Augmented reality devices
Microsoft HoloLens	Windows 10, proprietary Microsoft Holographic Processing Unit, Intel 32 bit processor, 2 MP camera, four microphones, 2 GB RAM, 64 GB flash, Bluetooth, Wi-Fi
Sony SmartEyeglass	3 MP camera, Bluetooth 3.0, 802.11 b/g Wi-Fi, 2.5 battery life. Compass, gyroscope, accelerometer, brightness sensor
Epson Moverio BT-20	0.3 MP camera; Bluetooth 3.0; 802.11 b/g/n Wi-Fi; around 6 hours of battery life; 8 GB internal memory; built-in GPS, compass, gyroscope and accelerometer
Google Glass	5 MP camera with 720p video; Bluetooth; 802.11 b/g Wi-Fi; around 1 day of “typical use” on battery; 12 GB usable memory
Vuzix M100 Smart Glasses	5 MP camera; 1,080p video; Bluetooth; 802.11 b/g/n Wi-Fi; around 6 hours of battery life (display off); 4 GB internal memory; built-in GPS
Recon Jet	“HD” camera; Bluetooth 4.0; ANT+; 802.11 b/g/n Wi-Fi; around 4–6 hours battery life; 8 GB internal memory; built-in GPS; accelerometer, gyroscope, magnetometer, pressure sensor and infrared sensor
Optinvent Ora-1	5 MP camera; 1,080p video recording; Bluetooth 4.0; 802.11 b/g/n Wi-Fi; 4 GB internal memory; ambient light sensor; photochromic lenses
GlassUp	Android and iOS-friendly; Bluetooth LE; 1-day battery life; accelerometer, compass, ambient light sensor, precision altimeter
Virtual reality devices
Oculus Rift	2,160×1,200 resolution, 110-degree field of view, 90 Hz refresh rate, positional tracking, built-in mic and audio
Sony PlayStation VR	5.7-inch, 1,920×1,080 display; 120 fps refresh rate; 100-degree field of view; 360-degree tracking
HTC Vive	2,160×1,200 resolution, 90 fps refresh rate, 110-degree field of view, positional tracking, location tracking at up to 15×15 feet
Samsung Gear VR	2,560×1,440 resolution; 96 degree field of view; 60 Hz refresh rate. Supports head-mounted touch controls as well as Bluetooth controllers
FOVE VR	5.8-inch, 1,440p, display; a 100+ degree field of view; 90 fps frame rate; and eye-tracking measured at 120 fps
Avegant Glyph	1,280×800 resolution; 45 degree field of view; 120 Hz refresh rate

VR, virtual reality.

Many companies acknowledge that the healthcare arena is ripe for AR and VR and inevitably there will be an increase in the number of drivers of this technology into clinical practice.

Opportunities in surgical healthcare

Anatomical evaluation

The traditional teaching of anatomy usually involves use of an anatomical atlas, time spent in the dissection room and fixed prosections (21). AR and VR are being used to deliver a better appreciation of structures in virtual or real space to ease the transition from the learning environment to the clinical environment. An example of this is using Microsoft Kinect to produce an interactive ‘digital mirror’ of the leaner. This digital mirrored image is then augmented by anatomical datasets to visualise structures such as musculature, superimposed on the user’s own arm (Figure 4) (23). The technology is being applied to clothing with pre-patterned codes on fabric, which when detected by camera software will superimpose animated anatomical images of the internal organs on the subject (Figure 5). The values of these modalities enhance our appreciation of anatomy where cadaver resources have limited accessibility (24). The use of VR and AR allows interaction with anatomy in whole new ways where the only constraint is the computing power available. Dassault Systèmes are using computation modelling in relation to anatomy and VR which allow clinical scientists to immerse themselves into the patient’s anatomy to problem solve pathologies (25). Virtual vascular endoscopy can generate endoluminal views of blood vessels that may be useful in pre-operative planning (26), especially for patients with significant atherosclerosis or aberrant anatomy. VR devices would allow for 360-degree visualisation and the ability to view multiple images simultaneously. Similar benefits have been realised in the visualisation of complex hand and lower limb fractures following 3D CT or MRI.

Figure 4.

4D Anatomy Augmented reality app of the heart and human body (22). Available online: http://www.asvide.com/articles/1271

Figure 5.

Curiscope T-shirt with visualisation of animated internal organs.

AR is also being used to evaluate dynamic anatomy in real time through use of digital ultrasound (US) (27). This simple application takes a reference point on the Doppler probe and on a separate HMD superimposes the US display on the subject. This technology allows visualization of structures and blood flow that can enhance performing invasive procedures. AR can supplement anatomy learning by superimposing radiological (CT or MRI) images on to a body and creating a direct view of spatial anatomy for the learner (28). Additionally, the use of haptic technology alongside this AR application provides the user with tactile feedback for appreciating the tactile consistency to different tissue components (29). Overall, this represents an exciting area for VR and AR development in anatomical education.

Broadcasting and recording surgery

The first live global broadcast of the VR surgical environment has been successfully trialled at the Royal London Hospital and has gained much media attention in globalizing medicine (20). The experience itself provides full 360 view of the operating room from the head of the operating table and provides the viewer with a patients’ rather than a surgeons’ perspective. An increasing number of educational meetings incorporate live surgery as part of the programme. As the cost of VR can be kept low with technology like Google Cardboard, we may see this as a more immersive forum for surgical education (20). In practical terms, the ability to interact with real life and digital elements make AR the forerunner in usability for live surgery compared to full immersed VR. Both can be used in medical education to display surgical procedures but the simulated environments still fall short graphically from augmented images. According to the Lancet Commission on Global Surgery, 5 billion people do not have access to safe affordable surgery (30). Live operations using AR, have been broadcast to a global community with feasibility demonstrated for basic procedures both in Paraguay and Brazil (31). Virtual interactive presence and augmented reality (VIPAR) has developed a support solution that would allow a remote surgeon to project their hands into the display of another surgeon wearing a headset (32). Proximie is another platform that aims to introduce AR technology to surgeons in the developing world that will allow them to visualise real-time or recorded operations being performed by experts in other parts of the world, allowing for a breadth of surgical experience to be disseminated. Both complex visual and verbal communication enable long-distance intra-operative guidance (33). It is possible to combine this with triggers such as 3D printed props which then can launch specific instructional video of surgeries and dissection (Figure 6). In an era of collaboration and sub-specialisation, AR may provide the much-needed contribution to educational advancement in the future.

Figure 6.

3D printed haptic models triggered augmented operative videos (34). Available online: http://www.asvide.com/articles/1272

Such data rich resources require careful management of potential confidential information. Applications on these platforms will permit the capture of images and video with digital archiving and it is important that these are managed in ethical and secure ways. One possibility is to allow instantaneous upload of images or videos onto a secure encrypted health care server, which would improve ease and quality of out-of-hours documentation and reduce costs of medical illustration services. For example, in the acute assessment of burn injuries, these devices would enable the assessor to effectively evaluate total body surface area affected and simultaneously obtain photography for clinical documentation. Video recording capabilities can help in documenting surgical procedures and real-time range of motion, beneficial in monitoring hand therapy progress. Currently, the imaging quality of such devices are considerably inferior to that offered by full frame image capture (35) and data storage of images and video in this way would require systems with massive capacity. This would require a considerable investment in the health care information technology services.

An enhanced medical information infrastructure could see development of patient records into AR and mobile applications. MedRef (36) enables AR systems to retrieve medical records by patient facial recognition. Electronic records of blood results, wound swabs, clinic notes, ward documentation, and medications can all be reviewed in conjunction with live vital signs using systems such as drchrono (37). Paperless recording and a to-do list on display would allow busy units to keep track of important tasks. This has the potential to improve clinical effectiveness and patient safety.

Taking this concept further into surgery, holoportation, an application of the Microsoft Hololens allows the user to see a projected hologram of a person occupying a different ‘space’ as though they were in the same room. This application has promise in the operating theatre where an identical surgical space enables two surgeons to walk around a holographic representation in two different environments, facilitating interaction with surgeons worldwide. Recorded procedures in small, true magnified scale enables navigation from any angle or orientation. The potential for recording-enabled HMDs to serve as a medical ‘black box’, similar to the airline industry would have important medico-legal implications for clinical practice.

Telementoring and education

Live feed and recorded data could be used for a number of applications such as trainee-trainer interactions in the context of work-based assessments. This would allow greater flexibility to learn and interact on one’s own terms. Sharing information and techniques between teaching hospitals and across continents is potentially very powerful (38). This form of telementoring has already demonstrated its effectiveness by using GG to facilitate a real-time consultation with a live feed of the surgical view and relevant radiological imaging (39).

The use of technology to provide off-site out of hours specialist consults is now a reality and would be an obvious transition for AR and VR. Out of hours surgery in the United Kingdom is primarily manned by a single junior member of staff. Telementoring would enable juniors to share a real time view of complex cases with an off-site senior member of the team for immediate advice. Specialities whereby visual review is required would benefit from this such as injury assessment, patient monitoring and trauma management. Telementoring is a safe option for providing expert diagnosis and opinion which can reduce the likelihood of mismanagement and unnecessary patient transfer (40). Likewise, multidisciplinary team (MDT) management requires a careful coordination of different disciplines at one site and is often fraught with difficulties in accessibility (41). Using AR or VR in the teleconferencing set up has the potential to improve communication when specialists are off-site and combine specialist care from different centres.

Operative benefits

AR and VR have the potential to impact on surgery in a number of novel ways as discussed above, especially in the arena of surgical training in the virtual surgical environment (Figure 7). However, real-time enhancement of the surgical procedure remains a slightly tentative application. It is not yet validated that surgery can be enhanced with AR and in some instances, it could be distracting. Some features may be useful of systems like GG where with voice activation the operator could communicate beyond the theatre environment, retrieve images and test results without breaking scrub. Real-time updates regarding the progress of the trauma list would reduce unnecessary fasting of patients in the event of a delay in theatre.

Figure 7.

The virtual operating theatre.

Real-time augmentation of surgery usually involves the blending of acquired 3D imaging with surgical reference points. Novel applications of AR include use to project optimal port placement on the abdomen for laparoscopic surgery (42); using AR to identify the position of sentinel nodes with 3D freehand single photon emission computed tomography (43,44); and using this with near infra-red spectroscopy to provide visual guidance in lymph node dissection in cancer surgery (45). Specialised near infrared (NIR) devices have been developed for the detection of tissue vascularity using indocyanine green (ICG) dye (46). The use of ICG in lymphatic surgery is already well developed to help identify vessels and check for their patency hence the move from microscope to HMD is a likely future development (47). AR technology would also be able to seamlessly project diagnostic images intra-operatively for surgical planning to guide surgeons with optimal incisions and approach (39).

Several studies have demonstrated the use of AR to guide surgeons through intricate anatomy during minimally invasive surgery. Su et al. [2009] demonstrated the use of pre-operative imaging with intra-operative 3D overlay to guide robotic laparoscopic limited partial nephrectomy. Minimal access limited partial nephrectomy has been an area of interest for AR guided surgery (48). AR guidance allows for projection of 3D imaging onto the laparoscopic image to mark surgical incisions within the laparoscopic view (49).

AR has been used extensively in neurosurgical procedures. Use of pre-operative imaging to detect suitable vessels for extracranial-intracranial bypass allows for image injection into the operator’s microscope to guide intra-operative dissection. Similar techniques for AR have been utilised for intracranial arteriovenous malformation surgery (50,51).

Patient benefits

Companies are using the technology to provide patients with an augmented or virtual experience of what they can expect from surgery. Crisalix are promoting virtual aesthetics planning whereby after obtaining the patients attributes, the software can virtually demonstrate the likely changes in aesthetics such as breast enhancement (52). This improves documentation, communication, and education of clinicians which will have implications for quality of service and patient safety. In addition, a case study demonstrated that using Oculus Rift with distraction software reduced the level of pain experienced by a burns’ patient during occupational therapy (53). Other VR systems offered similar results for pain control in burns’ patients undergoing wound debridement (54).

Limitations of augmented reality

It is likely that AR will have an important role in image-based augmentation of the surgical environment. This will require increasingly powerful microcomputers to drive AR, which is currently limited but will improve with time. For the device to be a natural extension of the surgeon’s senses, it has be light, mobile, comfortable and functional for potentially long periods of time. Therein lies the limitations of the technology at present, where the battery life is limited, devices are large and the cables can be cumbersome. Such technology has to progress at present and eventually after several generations of development these tools will become as common as surgical loupes.

As with electronic patient records, confidentiality and data management will be a major hurdle in the integration of recordable HMDs into medical practice. As clinicians, we have a duty of care and we should remain mindful of our ethical and legal obligations when using technology to either review, store or transfer patient data. In 2004, the General Medical Council (GMC) highlighted the importance of appropriate security for electronically stored personal information (55). The holding organization and a ‘Caldicott guardian’ uphold and maintain the responsibility to enforce appropriate security of confidential information, protect all personal identifiable data. In the United Kingdom, we currently have a legal requirement determined by common law, the Data Protection Act 1998 and the Freedom of Information Act 2000. The common law duty of confidentiality states patient identifiable data should not be provided to third parties. Data that is regarded as confidential in nature, as witnessed during a doctor-patient relationship is protected under the notion that the confidant (doctor) can only disclose information with consent from the confider (patient) (56). The NHS is required to also abide by the NHS code of practice for confidentiality. There are now seven principles since March 2013 which are related to transferring and collecting identifiable information. All health professionals using and storing patient information would need to be aware of these seven principles. We must also consider the legal risk to healthcare trusts that do not make adequate attempts to prevent medical identity theft (57).

Encryption improves but does not guarantee prevention from data hijacking (58,59). It is important that whichever system is developed meets the standards for health care information governance (60). With so many different healthcare systems it is likely that there will be many different AR and VR systems in use, which will have varying degrees of compatibility. As such, the healthcare market will capitalise on developing accessible price sensitive software and hardware to market. It is anticipated the global AR and VR healthcare industry will be worth $641 million by 2018 (61).

Conclusions

HMDs with either VR or AR will have great potential in the field of surgery. Their functionality has the potential for benefit in a range of clinical settings across the MDT and in medical education. First generation devices like GG have given us a glimpse of what AR can provide and despite its demise our appetite for new head-mounted devices has not diminished. New innovations like Microsoft HoloLens and the emerging mass market of VR headsets would indicate that these technologies will become familiar to surgeons and inevitability we will find a way to integrate them into our day-to-day practice. The challenge of identifying compelling and valuable experiences for these modalities now begins along with validation of their benefits in all aspects of surgical care. The digital surgical environment is about to drastically change.