A Heads-Up Display for Diabetic Limb Salvage Surgery

Abstract

Although the use of augmented reality has been well described over the past several years, available devices suffer from high cost, an uncomfortable form factor, suboptimal battery life, and lack an app-based developer ecosystem. This article describes the potential use of a novel, consumer-based, wearable device to assist surgeons in real time during limb preservation surgery and clinical consultation. Using routine intraoperative, clinical, and educational case examples, we describe the use of a wearable augmented reality device (Google Glass; Google, Mountain View, CA). The device facilitated hands-free, rapid communication, documentation, and consultation. An eyeglass-mounted screen form factor has the potential to improve communication, safety, and efficiency of intraoperative and clinical care. We believe this represents a natural progression toward union of medical devices with consumer technology.

Augmented reality involves the overlay of digital imagery onto the real world in real time.^1,2 This has been used in a wide variety of applications and stems from virtual reality in which a user is able to interact with a virtual environment. Over the past decade, physicians and surgeons have adopted this technology by way of heads-up displays both in training and to employ patient-specific imaging to reference or guide intervention.^3,4

While physicians, surgeons and other health care providers have identified potential applications for this technology, devices developed have suffered from high cost, uncomfortable form factor, suboptimal battery life, or lack of an app-based developer ecosystem.¹ The recent, limited release of a wearable technology using an eyeglass form-factor (Google Glass, Mountain View, California) has begun to address some of these issues of use and specification.^5-7

The potential applications that we have identified for Glass include augmented reality, telecommunication, documentation and education. Augmented reality via Google Glass is a reality, allowing users to see GPS specific, web based information projected onto the real world. Where consumers see restaurants, we see patient-specific data. Our group is familiar with telecommunication in medicine, but as surgeons, the hands free aspect of this form factor is very attractive.⁵ Medical documentation is a hot topic as reimbursements are dependent on thorough, accurate records. Google Glass has the potential to revolutionize medical documentation. There is no need to dictate, when you can record the entire operation. Finally, medical education has been impacted by legislation and restricted work hours.⁸ If we are to ensure optimal medical education, Glass may improve the quality of education while working within the guidelines, thereby maximizing educational efficiency.

We are unaware of any reports in the medical literature detailing the use of this technology in the care of the high-risk extremity. Therefore, the purpose of this article was to describe the use of a wearable consumer-based augmented reality device to provide rapid communication, documentation, and consultation among clinicians.

Device

Google glass is a computer with an optical head mounted display comparable to a 25’’ HD television viewed from 8 feet. It uses bone conduction for audio, stores 16GB, possesses a 5MP camera, Wi-Fi, Bluetooth, voice recognition and has roughly 4 hours of continuous, active battery life. It is incorporated into a familiar eyeglass form factor (Figure 1). Connectivity is achieved via wireless local area network or Bluetooth from a standard 3G or 4G GSM/CDMA/LTE connection using Glass-Bluetooth-Phone. Remote parties were able to communicate from a mobile device or desktop computer using the Hangouts application (Google; Mountain View, CA) (Figure 1a).

Figure 1.

(a) Google Glass. One eye consists of a see-through monocle and integrated camera with bone-conducting or direct in-ear microphone. It connects to the Internet via Bluetooth and mobile phone or directly via Wi-Fi. (b) Connectivity is achieved via wireless local area network or Bluetooth connection using Glass linked to phone via Bluetooth. Remote parties are able to communicate from a mobile device or desktop computer using the Hangouts application (Google; Mountain View, CA).

Intraoperative Case Example

Without any prior planning except for short message service (SMS) text messaging by operating room staff, the surgeon consulted with another surgical colleague and discussed requirements for resection, and subsequent admixture of an antimicrobial bioactive implant to deliver into a previously infected bony defect following a combined vascular-soft tissue reconstructive limb salvage procedure (Figure 2). The Google “Hangouts” application was managed entirely by the operating surgeon using hands-free voice control. In addition, real-time diagrams and MRI measurements were developed as a picture in picture, permitting 2 colleagues to effectively consult without the surgeon’s eyes leaving the operative field to view intraoperative imaging along with surgical anatomy.

Figure 2.

Visible defect on main screen with consultant clinician in lower right (a), real-time photograph of admixture of antibiotic-demineralized bone matrix (b), and measured delivery into defect (c, d).

Clinic Case Example

A consultation for the above patient was continued the next day in clinic. The clinician performed a dressing change along with the virtual consultant, however, due to scheduling conflicts, the consultant was unable to be physically present for the first postoperative dressing change, but was still able to take part in postoperative care and management.

Intraoperative Education

In a separate patient at high risk for wound complications, we utilized Google Glass as an educational adjunct. A junior resident donned the Google Glass during a scheduled delayed primary closure of a plantar defect and was engaged in an interactive “screen share” feature which fed detailed descriptions on retention suture technique to the junior resident. This allowed for real-time visual instruction in collaboration with a senior attending surgeon (Figure 3). This approach maximized hands on experience and autonomy for the resident. This was performed using bandwidth from a standard 3G CDMA connection using Glass-Bluetooth-Phone. A similar procedure followed demonstrating compartmental anatomy and assisting in planning a surgical decompression of a limb-threatening infection (Figure 4). A helpful adjunct to education is the ability for the consultant to point to areas of interest within the operative field. As of now, pointing is possible via screen sharing of the live recording from Glass. The operative surgeon need only point to items in the visual field of Glass and the remote consultant can use a mouse to indicate via a desktop computer. Pointing performed by the consultant is not yet possible with a mobile device.

Figure 3.

(a, b) During a delayed primary closure of a high-risk plantar wound, real-time descriptions with instant “screen share” were fed through Glass to a junior resident during the surgical procedure to assist instruction by the senior attending surgeon.

Figure 4.

View during intraoperative consultation for plantar deep space infection to assist in incision planning, exploration, and decompression using senior author’s article’s figures as a case example.

Discussion

Surgeons have been employing augmented reality in medicine and surgery for the past 20 years.⁹ The key to successful introduction of new technologies is to seamlessly integrate them with existing technologies. The ability of oncologists to precisely eradicate a tumor using stereotactics, high definition imaging, and focused radiation, has had a significant impact on patient safety and survival.¹⁰ Many new technologies suffer from high cost and size, and therefore, wide spread application is not possible. Until recently, augmented reality neither had the correct form factor to make intraoperative and in clinic use practical, nor the efficacy to justify its application.^11,12

In addition to size and cost, accessibility is key to the proliferation of a new technology. The cost of development for a commercially available technology is spread over an enormous consumer base. Much new medical technology is slow to develop due to a relatively small consumer base—resulting in high costs—and proprietary restrictions. Google Glass is part of an app-based developer ecosystem, which greatly accelerates the maturation process as seen by mobile applications and the smart phone. Just as with smart phones a decade ago, the application ecosystem for Glass is rather small. However, the form factor and subsequent iterations of it allow for arguably enormous growth.

This technology has the potential to enhance communication. Google Glass may increase individual physician efficiency, especially for surgeons, so that more efficient patient care can be provided. Surgeons now have the ability to provide or receive consultation from their colleagues without having to leave the operating room, office, or clinic. Even intraoperative team communication could be improved. Each member of the operative team could be outfitted with Google Glass. The surgeon has heads-up real-time patient vitals, information and imaging. The anesthesiologist is able to administer critical medications, without ever losing sight of the operative field or communication with the surgeon. The surgical resident and medical students always have an ideal view of the operative field, which may help facilitate instruction. The operating room (OR) nurse knows exactly what tools the surgeon needs, and even while outside of the OR communication is never lost.

Concerning patient safety, many parallels might be drawn between the commercial airline industry and medicine. Surgical theaters have adopted checklists similar to pilots prior to take off, to make the journey safer. Google Glass may even serve as an early “black box” in medicine. If every action of a physician, surgeon, and nurse is recorded, it could impact litigation, but it could also provide critical insight into surgical decision-making, technique and safety. Furthermore, anatomic “no fly zones”^13,14 could be projected, like a road map, onto Glass, which could reduce the risk of intraoperative complications.

Immediate future works from our group include evaluating use of these “no fly zones” as well as merging with image overlay/anatomic registry to use techniques like indocyanin green angiography to identify tissue viability in real time.^15,16

While the above points seem laudable, we must consider that the quest for increased efficiency could paradoxically result in decreased quality of care. We have not yet investigated how the additional stimulus of augmented reality impacts performance or whether it might prove too distracting. We look forward to further works in this area that might seek the requisite balance.

Despite its advantages, Glass has limitations. The form factor though much improved from earlier head mounted interfaces, will still be incompatible with certain frames of prescription eye wear. However, prescription lenses can be integrated. Currently, the greatest utility of Glass lies in its connectivity and not necessarily Glass based applications. That being said, augmentation of the application “AR Glass for Wikipedia,” which projects GPS-specific, web-based information, could be very useful if all of your patients’ data were projected when in proximity to their room or by looking at the QR code on their identification bracelet. Google+ Hangouts are of great utility in keeping the surgical team (surgeon, resident, medical student, and nursing staff) connected and up to date. Google+ Hangouts would be more useful if it were integrated with the electronic medical record (EMR). Live links to keywords could show more detailed patient-specific data. The reliability of this technology has yet been tested. We do not anticipate any security issues as Glass works through a network that is already in use within the medical community. Patient privacy is a concern. Just as we discussed Glass as a potential black box for health care, widespread adoption of video-based communication would increase the amount of patient-specific images (data) if recorded. However, live videoconferencing via glass is not recorded and therefore is no more risky than discussing a patient over the telephone.

As this new technology finds its place in the consumer market, we continue to try to understand its place in medicine. Our hope is that with increased access to this technology that we can begin to objectively measure the effect that Glass has on our practice and patient care. We believe that the future in this area is promising. Enhancing communication, facilitating education, increasing safety, and improving care are not virtual goals, but are rather virtuous and ultimately realistic.