Skip to main content
PMC home page
Asian Journal of Andrology logoLink to Asian Journal of Andrology
. 2015 Nov 27;18(5):732–734. doi: 10.4103/1008-682X.166436

Augmented reality assisted surgery: a urologic training tool

Ryan M Dickey 1, Neel Srikishen 1, Larry I Lipshultz 1, Philippe E Spiess 2, Rafael E Carrion 2, Tariq S Hakky 1,✉
PMCID: PMC5000796  PMID: 26620455

Abstract

Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass® optical head-mounted display to train urology residents in how to place an inflatable penile prosthesis. We use the phrase Augmented Reality Assisted Surgery to describe this novel application of augmented reality in the setting of surgery. The application demonstrates the steps of the surgical procedure of inflatable penile prosthesis placement. It also contains software that allows for detection of interest points using a camera feed from the optical head-mounted display to enable faculty to interact with residents during placement of the penile prosthesis. Urology trainees and faculty who volunteered to take part in the study were given time to experience the technology in the operative or perioperative setting and asked to complete a feedback survey. From 30 total participants using a 10-point scale, educational usefulness was rated 8.6, ease of navigation was rated 7.6, likelihood to use was rated 7.4, and distraction in operating room was rated 4.9. When stratified between trainees and faculty, trainees found the technology more educationally useful, and less distracting. Overall, 81% of the participants want this technology in their residency program, and 93% see this technology in the operating room in the future. Further development of this technology is warranted before full release, and further studies are necessary to better characterize the effectiveness of Augmented Reality Assisted Surgery in urologic surgical training.

Keywords: andrology, augmented reality assisted surgery, Google glass, inflatable penile prosthesis, optical head mounted display, surgical education

INTRODUCTION

The earliest known surgical operation in humans was decompressive trepanation of the skull, performed in the Neolithic era.1 Surgery continued to evolve with the advent of anesthesia, antiseptics, and X-ray in the 19th century, allowing for more complex and precise procedures, as well as decreased mortality from sepsis.2,3 Novel use of technology has driven surgical advancement, from implantable technology such as the total artificial heart, cochlear implant, and inflatable penile prosthesis (IPP) to surgical techniques such as endoscopy and robotics.4,5,6,7,8 The current state of endoscopic and robotic surgery has rendered many once open procedures to minimally invasive techniques.4,5 As the surgical evolution continues, new technologies will become more ubiquitous in the operating room, for both patient and surgeon.

Augmented reality (AR) is defined as a “technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view.”9 AR is widely used in aeronautics to assist pilots in takeoff, maneuvers, and landing; some have suggested a similar embrace of this technology in the field of surgery.10 Recent use of AR in minimally invasive surgery has resulted in the creation of hybrid image-guided surgery using endoscopic and robotic video feeds. A dedicated institution has been developed around image-guided hybrid therapies.10 However, the delivery of AR in open surgery will require an alternative technology for the surgeon's interface.

Optical head-mounted displays (OHMDs) and specifically Google Glass's® wearable technology (Google Inc., Mountain View, California) have shown to be an effective tool in both the clinical and operative setting and can be used as an AR interface for surgery.11 OHMDs have demonstrated to help anesthesiologists maintain awareness of the patient's status in a distracting operating theater.12,13 AR is an emerging open surgical tool in education and is effective in orthopedic surgery in the creation of a composite view of student and teaching surgeon.14 Google Glass has been studied in other surgical fields including plastic surgery and Mohs surgery.15,16 In urology, a primitive form of AR has been implemented in robotic partial nephrectomy.17,18 To date, there is no study of the use of AR or Google Glass in open urologic surgery. In this pilot study, our aim is to determine the feasibility of implanting AR in the setting of an OHMD that could be built for open urologic surgery not only as a surgical assistant but also as a surgical training tool in the placement of IPP. This is a pilot study to look at the use of Google Glass as an Andrologic training tool. We use the phrase Augmented Reality Assisted Surgery (ARAS) to describe this novel application of AR in the setting of urologic surgery.

MATERIALS AND METHODS

After an IRB was obtained, Google Glass with software version XE 22.1 was purchased from Google during the beta testing phase. The device was loaded with a developed glass application, which contained a menu (Figure 1) with several options including a demonstration on how to place an IPP from start to finish in the penoscrotal approach, the most commonly used surgical approach for this procedure.19 It also contained software that allowed for detection of interest points using an optical display camera from the OHMD output feeds to allow faculty to interact with residents during placement of the penile prosthesis (Figure 2). In the operating room, the OHMD is worn as a pair of glasses, with a monocular refractive lens over the right eye projecting input to the surgeon. The application is launched via voice commands or by tapping a menu display on the side of the Google glass. Once the application has loaded, the user is presented with a menu. Menu options include: how to use Google Glass, a step-by-step placement video of IPP placement in the penoscrotal approach, an interactive video display, and a section about the authors and build of the application.

Figure 1.

Figure 1

Open in a new tab

ARAS menu screen shot from Google Glass application. ARAS: Augmented Reality Assisted Surgery.

Figure 2.

Figure 2

Open in a new tab

ARAS application screen shot demonstrating detection of interest points using an optical display camera from the OHMD output feeds with faculty interaction during IPP placement. ARAS: Augmented Reality Assisted Surgery; OHMD: optical head-mounted displays.

In the first component of the application, video footage corresponding to the sequential steps of the IPP procedure was projected onto the live view of the patient in real time. This allowed trainees to visualize specific steps prior to completing them. The second component of the application included detection software that streamed live OR footage to a remote attending physician. The attending physician could then interact with the surgical trainee by moving a cursor highlighting areas of interest to explain details of the operation in depth. Consent was obtained to film sequences of the operations from patients undergoing IPP placement.

Study participants included urology residents and faculty at two institutions who volunteered to experience the technology and application and provide feedback. Participants were shown how to navigate Google Glass using either voice-command or tapping gestures and were given time to wear the Google Glass in either the operative or perioperative setting of IPP placement. Participants were then given a feedback survey to complete regarding the effectiveness and interest in the device and software. Survey questions included both quantitative and free text options (Figure 3). Quantitative questions asked participations to scale their perceptions to survey questions from 1 to 10, with 1 being “not at all” and 10 being “always”. Other questions included a binary yes or no response with option to input free text. The surveys were then stratified into trainee and faculty categories and calculated to determine the mean rating for quantitative questions. Free text responses were analyzed for key words and grouped as closely as possible.

Figure 3.

Figure 3

Open in a new tab

ARAS Experience Survey.

RESULTS

A total of 30 participants volunteered to take part in the study. Of these, 10 were urology faculty members in academic urology programs, and 20 were either urology residents or fellows. Out of a 10-point rating scale, total participants rated the educational usefulness of the application as 8.6. Ease of navigation of the device and application was rated 7.6. The likeliness to use the technology if made available was rated 7.4. The technology was also rated if perceived as too distracting, with overall rating of 4.9 (Table 1). For the recommendation to implement the technology into respective training programs, a total of 26 responses were received (4 nonresponses), with 21/26 (81%) participants responding in the affirmative. For the belief that Google Glass has a role in the operating room, a total of 28 responses were received (2 nonresponses) with 26/28 (93%) responding in the affirmative. For the consideration of using hands-free Google Glass in future practice, a total of 28 responses were received (2 nonresponses) with 20/28 (71%) indicating the affirmative. For the belief that the application bettered understanding of the surgical procedure and anatomy during IPP placement, 16 responses were received with 9/17 (53%) responding in the affirmative (Table 2).

Table 1.

Mean results of ARAS experience survey from urologic faculty and trainees using 10-point rating scale

graphic file with name AJA-18-732-g004.jpg

Open in a new tab

Table 2.

Results of ARAS experience survey from urologic faculty and trainee respondents to yes-no questions

graphic file with name AJA-18-732-g005.jpg

Open in a new tab

When stratified between trainees and faculty, educational usefulness was found to be 8.7 versus 8.4, respectively. Ease of navigation was found to be 7.5 versus 7.9, respectively. Likelihood to use was found to be 7.4 versus 7.5, respectively. Distraction in the OR was found to be 4.7 versus 5.4, respectively (Table 1).

Specific comments given by participants recommended the inclusion of voice activation, patient and staff educational components for the procedure, video recording in the OR instead of streaming only, the ability to pull up patient imaging from while operating, and magnification.

Our novel study has several limitations. It is limited because the survey has not been validated and that we have a limited number of responses to the survey. Additionally, our surgical training tool was only useful in training one type of open urologic surgery. AR itself has limitations, as much of the surgical programming has yet to be done, and this technology is continuing to evolve. Another concern may be surgeon over reliance on AR, creating an intraoperative conflict as the surgeon's decision may not always be in line with the AR software. Limitations of the current Google Glass technology include limited battery life, overheating of the battery, and cumbersome software integration.

DISCUSSION

As technology continues to permeate into every aspect of our lives, we look to integrate technology into urologic surgery and surgical training. Surgeons currently perform procedures after years of education and human surgical training to safely navigate through anatomical planes of the human body. The concept of AR, where a live view of the environment, in this case, a surgical operation, is supplemented by computer-generated sensory input using an OHMD, is the next evolutionary step in open urologic surgery and is an important training tool. Our goal is to introduce the concept of AR to enhance the surgeon's perception of reality during a procedure, and to enrich the learning process during a surgical procedure.

Here, we successfully demonstrate the first integration of Google Glass in the surgical training of urology residents and faculty along with the implementation of ARAS in feature detection during penile prosthesis surgery. The overwhelming response from faculty is to have a proctoring platform, while trainees slightly favored the implementation of ARAS over their training counterparts. As expected, we noticed that trainees, who are younger, rated the application more educationally useful and less distracting in the operating room than faculty, but were surprised to find faculty rating ease of navigation and likelihood to use in the future higher than trainees. This may suggest that though there is more acceptance of newer technology from trainees, there is also a higher expectation of available features and intuitive design than faculty. Suggested future capabilities include remote proctoring and a less cumbersome design. As ARAS evolves, it will continue to innovate surgical procedures by adding graphics, sounds, and haptic feedback. Advantages of ARAS include real-time feedback of residents during surgery, superior visibility, and interaction between faculty and residents. ARAS is part of the confluence of change that will be vital in the future armamentarium of surgical education.

CONCLUSION

ARAS is a novel and effective urologic surgical training tool in this pilot study. Both faculty and trainees view this new technology in a positive light. Augmented reality in surgery represents a paradigm shift within surgery, and urologic training may benefit from further exploration of this technology within residency programs and practices alike. As future platforms of OHMDs become available, this technology will continue to change how medicine is both practiced and taught.

AUTHOR CONTRIBUTIONS

RMD wrote the manuscript, TSH developed the application and edited the manuscript, NS administered ARAS survey and collected data, LIL provided consultation, PES assisted with writing of the manuscript, REC developed the concept. All authors read and approved the final manuscript.

COMPETING INTEREST

All authors declared no competing financial interests.

ACKNOWLEDGMENTS

The authors would like to acknowledge American Medical Systems for providing the funding for the development of the software application, and Guil Zriel for creating the Google Glass application.

REFERENCES

  • 1.Bishop WJ. The Early History of Surgery. London: R Hale; 1960. p. 192. [Google Scholar]
  • 2.Gawande A. Two hundred years of surgery. N Engl J Med. 2012;366:1716–23. doi: 10.1056/NEJMra1202392. [DOI] [PubMed] [Google Scholar]
  • 3.Rontgen WC. On a new kind of rays. Science. 1896;3:227–31. doi: 10.1126/science.3.59.227. [DOI] [PubMed] [Google Scholar]
  • 4.Spaner SJ, Warnock GL. A brief history of endoscopy, laparoscopy, and laparoscopic surgery. J Laparoendosc Adv Surg Tech A. 1997;7:369–73. doi: 10.1089/lap.1997.7.369. [DOI] [PubMed] [Google Scholar]
  • 5.Pugin F, Bucher P, Morel P. History of robotic surgery: from AESOP(R) and ZEUS(R) to da Vinci(R) J Visc Surg. 2011;148:e3–8. doi: 10.1016/j.jviscsurg.2011.04.007. [DOI] [PubMed] [Google Scholar]
  • 6.Mobley DF. Early history of inflatable penile prosthesis surgery: a view from someone who was there. Asian J Androl. 2015;17:225–9. doi: 10.4103/1008-682X.140962. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Macherey O, Carlyon RP. Cochlear implants. Curr Biol. 2014;24:R878–84. doi: 10.1016/j.cub.2014.06.053. [DOI] [PubMed] [Google Scholar]
  • 8.Cooley DA, Liotta D, Hallman GL, Bloodwell RD, Leachman RD, et al. Orthotopic cardiac prosthesis for two-staged cardiac replacement. Am J Cardiol. 1969;24:723–30. doi: 10.1016/0002-9149(69)90460-3. [DOI] [PubMed] [Google Scholar]
  • 9.Stevenson A. New York: Oxford University Press; 2010. Oxford Dictionary of English; p. 2069. pxxii. [Google Scholar]
  • 10.Marescaux J, Diana M. Next step in minimally invasive surgery: hybrid image-guided surgery. J Pediatr Surg. 2015;50:30–6. doi: 10.1016/j.jpedsurg.2014.10.022. [DOI] [PubMed] [Google Scholar]
  • 11.Muensterer OJ, Lacher M, Zoeller C, Bronstein M, Kubler J. Google Glass in pediatric surgery: an exploratory study. Int J Surg. 2014;12:281–9. doi: 10.1016/j.ijsu.2014.02.003. [DOI] [PubMed] [Google Scholar]
  • 12.Sanderson PM, Watson MO, Russell WJ, Jenkins S, Liu D, et al. Advanced auditory displays and head-mounted displays: advantages and disadvantages for monitoring by the distracted anesthesiologist. Anesth Analg. 2008;106:1787–97. doi: 10.1213/ane.0b013e31817325cb. [DOI] [PubMed] [Google Scholar]
  • 13.Liu D, Jenkins SA, Sanderson PM, Watson MO, Leane T, et al. Monitoring with head-mounted displays: performance and safety in a full-scale simulator and part-task trainer. Anesth Analg. 2009;109:1135–46. doi: 10.1213/ANE.0b013e3181b5a200. [DOI] [PubMed] [Google Scholar]
  • 14.Lindeque BG, Ponce BA, Menendez ME, Oladeji LO, Fryberger CT, et al. Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Orthopedics. 2014;37:751–7. doi: 10.3928/01477447-20141023-05. [DOI] [PubMed] [Google Scholar]
  • 15.Davis CR, Rosenfield LK. Looking at plastic surgery through google glass: part 1. Systematic review of google glass evidence and the first plastic surgical procedures. Plast Reconst Surg. 2015;135:918–28. doi: 10.1097/PRS.0000000000001056. [DOI] [PubMed] [Google Scholar]
  • 16.Kantor J. Application of google glass to mohs micrographic surgery: a pilot study in 120 patients. Dermatol Surg. 2015;41:288–9. doi: 10.1097/DSS.0000000000000268. [DOI] [PubMed] [Google Scholar]
  • 17.Hughes-Hallett A, Mayer EK, Marcus HJ, Cundy TP, Pratt PJ, et al. Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology. 2014;83:266–73. doi: 10.1016/j.urology.2013.08.049. [DOI] [PubMed] [Google Scholar]
  • 18.Hung AJ, Shah SH, Dalag L, Shin D, Gill IS. Development and validation of a novel robotic procedure specific simulation platform: partial nephrectomy. J Urol. 2015;194:520–6. doi: 10.1016/j.juro.2015.02.2949. [DOI] [PubMed] [Google Scholar]
  • 19.Simon R, Hakky TS, Henry G, Perito P, Martinez D, et al. Tips and tricks of inflatable penile prosthesis reservoir placement: a case presentation and discussion. J Sex Med. 2014;11:1325–33. doi: 10.1111/jsm.12481. [DOI] [PubMed] [Google Scholar]

Articles from Asian Journal of Andrology are provided here courtesy of Editorial Office of AJA.

RESOURCES