Abstract
Objective
We report our first experience of using a case‐specific three‐dimensional (3D) hologram for tumor resection in otolaryngology to show the proof of concept. In addition, a questionnaire was administered to assess the usefulness of the mixed reality technique in otolaryngology.
Methods
A case‐specific 3D hologram was developed from enhanced images of dynamic computed tomography, with reference to contrast‐enhanced magnetic resonance images, and used for preoperative planning and intraoperative image reference. To evaluate the usefulness of the 3D hologram with head mount displays (HMDs), 18 attendings and resident otolaryngologists completed a questionnaire with the Likert scale.
Results
The case‐specific 3D hologram on HMDs was successfully used by means of easy gesture‐handling without any monitors preoperatively and intraoperatively. The experience of picturing the tumor localization and evaluating the surgical approach was statistically better using the 3D hologram on HMDs than using the computer images (P < .01). Similarly, the holograms were observed to be better for intraoperative application and surgical education than computer images (P < .01).
Conclusion
We demonstrated the use of a case‐specific 3D hologram for tumor resection in otolaryngology. The technology may be useful for preoperative planning and intraoperative image reference, especially for challenging cases, and surgical education.
Level of Evidence
NA.
Keywords: hologram, mixed reality, surgical education, three‐dimensional imaging, tumor resection
We demonstrated the use of a case‐specific three‐dimensional hologram for tumor resection in otolaryngology to show the proof of concept. The technology may be useful for preoperative planning and intraoperative image reference, especially for challenging cases, and surgical education.
1. INTRODUCTION
Virtual reality is a rapidly developing technology in the field of computer science and can contribute to surgical care and medical education in many ways. 1 , 2 Mixed reality (MR) is the merging of real and virtual worlds, where physical and digital objects co‐exist. 3 , 4 See‐through head mount displays (HMDs) are wearable devices for MR and provide the user with a three‐dimensional (3D) virtual model, called a hologram, in the real world. The 3D hologram can be made using the radiological images of the patient in each case. The application and advantages of the MR technique with HMDs in facilitating the surgeons' anatomical understanding have been reported in other surgical fields such as general surgery, 5 , 6 urology, 7 and neurosurgery. 8
In order to resect tumors in the otolaryngology field, surgeons need to be familiar with structures of complex shapes such as the facial bones and the temporal bone. In other words, surgeons must imagine the anatomy and tumor localization with a 3D understanding to decide the surgical approach. Radiological images displayed on a computer screen are usually used to discuss the surgical approach for tumors preoperatively or intraoperatively. Case‐specific 3D holograms could help surgeons to get a better instinctive picture of the tumor localization and allow all the involved surgeons to share the surgical approach more precisely. However, the MR technique with HMDs has not been used in the field of otolaryngology yet. Here, we report our first experience of using a case‐specific 3D hologram for tumor resection in otolaryngology to show the proof of concept. In addition, a questionnaire was administered to assess the usefulness of the MR technique in otolaryngology.
2. MATERIALS AND METHODS
2.1. Development of case‐specific 3D hologram
Case‐specific 3D holograms were created from enhanced images, including the arterial and venous phases obtained from dynamic computed tomography (CT) using a dual‐source CT scanner (SOMATOM Force; Siemens Healthineers, Erlangen, Germany) (Figure 1). First, using the two‐dimensional (2D) digital imaging and communications in medicine data from the CT images of the patient, the bone, artery, vein, and tumor were traced using the ZIOSTATION software (Zaiosoft, Inc, Tokyo, Japan) by a medical radiology technician (S.S.) along with a radiologist (N.K.), with the contrast‐enhanced magnetic resonance images of the patient used as a reference. The magnetic resonance imaging examinations were performed using a 3T scanner (MAGNETOM Skyra; Siemens Healthineers, Erlangen, Germany) according to the institutional scan protocol. Axial T1‐weighted imaging (WI) and T2‐WI at a slice thickness of 4 mm and axial and coronal gadolinium‐enhanced T1‐WI at a slice thickness of 0.7 or 4 mm were used as references. Following this, polygon (STL data) files were exported from the ZIOSTATION software, uploaded into the Holoeyes XR system (Holoeyes Inc., Tokyo, Japan), and converted into a 3D hologram for HMDs. Hololens 2 (Microsoft Corporation, Washington) was used as see‐through HMDs in this study. The bones, arteries, veins, and tumor were given specific colors on the Hololeyes XR system (bone: white, artery: red, vein: blue, tumor: green). The time between the data upload and generation of the hologram was approximately 15 minutes. This study was approved by the Institutional Review Board of the Ehime University (registration number: 2004012) and written informed consent was obtained from the patient.
FIGURE 1.
Development of case specific 3D hologram. After organ segmentation of images from dynamic CT with MRI images as reference, 3D polygon files were exported and converted to 3D holograms for the see‐through head mount displays. Hololens 2 (Microsoft Corporation, Washington) was used as see‐through head mount displays in this study. 3D: three‐dimensional, CT: computed tomography, MRI: magnetic resonance imaging
2.2. Questionnaire about the experience with 3D holograms on HMDs
To evaluate the usefulness of the case‐specific 3D hologram on HMDs, 18 attending and resident otolaryngologists were administered questionnaires with a Likert scale (Figure 2). They used the Hololens 2 and manually operated the case‐specific 3D hologram after being instructed on how to use the HMDs and view the hologram. The participants compared the 3D hologram with the standard CT and MRI images on a computer screen to evaluate the benefit. The participants were allowed to read any of the axial, coronal, and sagittal images on the computer screen.
FIGURE 2.
Questionnaire with Likert scale about the experience with case‐specific three‐dimensional holograms on head mount displays, compared to the regular radiological images viewed on a computer screen
2.3. Statistical analysis
All statistical analyses were performed and graphs were drawn using the GraphPad Prism software (Version 8.4; GraphPad Software, La Jolla, California). The box plot shows the median score, lower and upper quartile scores, and minimum and maximum scores. A Mann‐Whitney U test was used to assess the relationship between scores of the two groups in the questionnaire results. P values of <.05 were considered as statistically significant, and values are reported as mean ± SD.
3. RESULTS
3.1. Using the case‐specific 3D hologram preoperatively and intraoperatively
A case‐specific 3D hologram was developed in a patient with postoperative recurrence of maxillary squamous cell carcinoma. The scale and direction of the 3D hologram on Hololens 2 could be fully changed (Figure 3A,B). In addition, each structure could be removed if needed, and any slices of the hologram from any direction could be easily obtained using a slicer device (Figure 3C,D, and Video S1).
FIGURE 3.
The scale, A, and direction, B, of the three‐dimensional hologram on Hololens 2 could be fully changed. Each structure could be removed if needed, C, and any slices of the hologram from any direction could be easily obtained using a slicer device, D
In the preoperative meeting, the case‐specific 3D hologram was placed midair using the Hololens 2 and used to discuss the surgical approach for tumor resection (Figure 4A). In addition, the hologram and the Hololens 2 could be used in the operative room to verify the anatomy while performing partial maxillectomy (Figure 4B). The 3D hologram was successfully shared between the surgeons and could be viewed and moved through easy gesture‐handling without any monitors in the operative room (Video S2).
FIGURE 4.
A case‐specific three‐dimensional hologram on head mount displays was used during the preoperative meeting, A, and for intraoperative image reference, B. For a postoperative recurrence of left maxillary carcinoma, a partial maxillectomy was performed with a Weber‐Ferguson skin incision. The coronoid process of the mandible was removed to secure a field of view from the lateral side. Most of the pterygoid process was resected to ensure a sufficient margin. The inferior orbital wall was preserved. Arrows: coronoid process of mandible. Arrow heads: inferior orbital wall. Asterisk: pterygoid process
3.2. Usefulness of 3D hologram based on the questionnaire
Regarding the usefulness of case‐specific 3D hologram on HMDs, based on the questionnaire results, compared with the regular images viewed on a computer screen (Figure 5), the participants felt that the tumor‐related anatomy and the surgical approach could be better perceived and imagined using the 3D hologram, and the results were statistically significant (questions 1 to 3, P < .01). Similarly, the 3D hologram was observed to be better than the regular images for intraoperative analysis and surgical education (questions 4 and 5, P < .01). In particular, regarding its usefulness in surgical education, 94% (17 out of 18) of the participants gave a score of 4.5 or 5 (average 4.86) out of 5 (question 5).
FIGURE 5.
A, The results of the questionnaire. Three‐dimensional (3D) hologram on HMDs was statistically better than computer images for picturing the tumor and deciding the surgical approach, intraoperative image reference, and surgical education (questions 1 to 5, P < .01). HMDs: head mount displays
4. DISCUSSION
In this study, an initial experience of a case‐specific 3D hologram in otolaryngological tumor resection was reported, and the MR technique was successfully used preoperatively and intraoperatively. The questionnaire results suggest that a case‐specific 3D hologram on HMDs was more helpful in visualizing the tumor, determining the surgical approach, and surgical education than the regular radiological images viewed on a computer screen.
In our study, contrast‐enhanced MRI images were referred to identify the tumor extent and for organ segmentation of CT images, although the hologram was made from dynamic CT images. Consequently, the 3D hologram was easy to understand intuitively; it comprised not only the bone and vessels from dynamic CT but also the tumor from MRI. The 3D hologram could be viewed and moved by means of easy gesture‐handling without monitors in any places, including the operative room. Since the 3D hologram could be kept in the same position for more than one surgeon by see‐through HMDs in the room, the preoperative and intraoperative discussions were performed smoothly face‐to‐face. As reported by previous studies in other surgical fields, 5 , 6 , 7 , 8 the MR technique with HMDs was equally convenient and helpful for the intuitive understanding of surgeons in otolaryngological tumor resection.
In cases of tumor resections, surgeons should imagine the anatomy three‐dimensionally and adequately plan the dissection line. In this study, case‐specific 3D holograms helped surgeons determine the dissection line better based on the questionnaire results. The 3D hologram on HMDs has been previously utilized to reconstruct the mandible using a fibula in reconstruction surgery. 3 According to this report, 3D hologram of the bone can be easily created with a precise model, thus, indicating that the hologram especially proves beneficial in cases where the tumor is surrounded by a predominantly bony anatomy. In otolaryngological cases, tumors often exist near bony anatomy of complex shapes, such as the facial bone and the temporal bone, or important structures such as cranial nerves 9 , 10 and main vessels like carotid artery, 11 and the surgical approach for such tumors is difficult. Therefore, the intuitive understanding by case‐specific 3D holograms is considered helpful for such otolaryngological cases. Besides, the MR technique is useful for cases wherein the tumor has an atypical shape, which should be paid attention to ensure a sufficient margin, and cases with an atypical anatomy (eg, anatomical anomaly or postoperative change) because the resection of such tumors needs an atypical dissection line. In addition, the technique is equally helpful for salvage surgery for a recurrent case after initial therapy, such as surgery or radiation, since it is difficult to dissect the tissue and understand the tumor location because of the postoperative adhesion or post irradiation‐fibrosis. 12 , 13 In summary, the MR technique may help otolaryngology surgeons in cases: (a) wherein the tumor is difficult to locate for surgical approach; (b) with an atypical anatomy or tumor shape; and (c) with a recurrent tumor after initial surgery or radiation.
Regarding the usefulness of 3D holograms in surgical education, most participants gave a score of 4.5 or 5 out of 5. The 3D holograms can provide young surgeons with a more precise image to understand surgical procedures compared to surgical or anatomical textbooks, since the holograms are case‐specific. The effectiveness of using 3D stereoscopic models for medical students with low‐spatial ability to teach anatomy has been reported in medical education research, including the head and neck region. 14 Therefore, case‐specific 3D virtual imaging can be used as a new educational tool for young doctors to improve their competence.
The limitation of this study was that the usefulness of the 3D holograms was evaluated based on questionnaires. Therefore, additional research involving several cases is needed for an objective study of their utility in tumor resection. The 3D hologram on HMDs can be used as an intraoperative image reference and may have a potential as a new efficient surgical navigation system in the future. However, the issue of alignment between the 3D hologram and the surgical field still exists, if the model is to be fused to the patient's body for navigation. 15
5. CONCLUSION
Our study demonstrated the use of a case‐specific 3D hologram for tumor resection in otolaryngology. The technology may be useful in preoperative planning and for intraoperative image reference, especially for challenging cases, and surgical education.
CONFLICT OF INTEREST
All authors declare no conflict of interest.
Supporting information
Supplemental Video S1 A case‐specific three‐dimensional hologram on head mount displays. NOTE TO REVIEWERS: The video file is available in the reviewer score sheet, on the right side of the screen, under “Files.” Let the Editorial Office know if you need assistance viewing this file.
Supplemental Video S2 A three‐dimensional hologram in an operative room. NOTE TO REVIEWERS: The video file is available in the reviewer score sheet, on the right side of the screen, under “Files.” Let the Editorial Office know if you need assistance viewing this file.
ACKNOWLEDGMENTS
We thank Naoki Nishida, Masato Teraoka, Kaori Tanaka, Masahiro Okada, Hirofumi Sei, Taro Takagi, Naruhiko Kai, Takuya Kimura, Takahiro Hanari, Yudai Ohara, Ryota Takagi, and Seitaro Murakawa for participating in the questionnaire and Editage (www.editage.com) for English language editing. This work was supported by Japan Society for the Promotion of Science KAKENHI Grant Number JP20K18255.
Mitani S, Sato E, Kawaguchi N, et al. Case‐specific three‐dimensional hologram with a mixed reality technique for tumor resection in otolaryngology. Laryngoscope Investigative Otolaryngology. 2021;6:432–437. 10.1002/lio2.581
Funding information Japan Society for the Promotion of Science, Grant/Award Number: JP20K18255
BIBLIOGRAPHY
- 1. Davis CR, Rosenfield LK. Looking at plastic surgery through Google Glass: part 1. Systematic review of Google Glass evidence and the first plastic surgical procedures. Plast Reconstr Surg. 2015;135(3):918‐928. [DOI] [PubMed] [Google Scholar]
- 2. Sinkin JC, Rahman OF, Nahabedian MY. Google glass in the operating room: the plastic surgeon's perspective. Plast Reconstr Surg. 2016;138(1):298‐302. [DOI] [PubMed] [Google Scholar]
- 3. Tepper OM, Rudy HL, Lefkowitz A, et al. Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg. 2017;140(5):1066‐1070. [DOI] [PubMed] [Google Scholar]
- 4. Condino S, Turini G, Parchi PD, et al. How to build a patient‐specific hybrid simulator for orthopaedic open surgery: benefits and limits of mixed‐reality using the Microsoft HoloLens. J Healthc Eng. 2018;2018:5435097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Saito Y, Sugimoto M, Imura S, et al. Intraoperative 3D hologram support with mixed reality techniques in liver surgery. Ann Surg. 2020;271(1):e4‐e7. [DOI] [PubMed] [Google Scholar]
- 6. Devoto L, Muscroft S, Chand M. Highly accurate, patient‐specific, 3‐dimensional mixed‐reality model creation for surgical training and decision‐making. JAMA Surg. 2019;154(10):968‐969. [DOI] [PubMed] [Google Scholar]
- 7. Yoshida S, Sugimoto M, Fukuda S, Taniguchi N, Saito K, Fujii Y. Mixed reality computed tomography‐based surgical planning for partial nephrectomy using a head‐mounted holographic computer. Int J Urol. 2019;26(6):681‐682. [DOI] [PubMed] [Google Scholar]
- 8. Incekara F, Smits M, Dirven C, Vincent A. Clinical feasibility of a wearable mixed‐reality device in neurosurgery. World Neurosurg. 2018;118:e422‐e427. [DOI] [PubMed] [Google Scholar]
- 9. Terrell JE, Welsh DE, Bradford CR, et al. Pain, quality of life, and spinal accessory nerve status after neck dissection. Laryngoscope. 2000;110(4):620‐626. [DOI] [PubMed] [Google Scholar]
- 10. Yigit E, Dursun E, Omeroglu E, et al. The course of lower cranial nerves within the neck: a cadaveric dissection study. Eur Arch Otorhinolaryngol. 2018;275:2541‐2548. [DOI] [PubMed] [Google Scholar]
- 11. Chan JYK, Holsinger FC, Liu S, Sorger JM, Azizian M, Tsang RKY. Augmented reality for image guidance in transoral robotic surgery. J Robot Surg. 2020;14(4):579‐583. [DOI] [PubMed] [Google Scholar]
- 12. Mandapathil M, Roessler M, Werner JA, Silver CE, Rinaldo A, Ferlito A. Salvage surgery for head and neck squamous cell carcinoma. Eur Arch Otorhinolaryngol. 2014;271(7):1845‐1850. [DOI] [PubMed] [Google Scholar]
- 13. Mueller SA, Mayer C, Bojaxhiu B, et al. Effect of preoperative immunonutrition on complications after salvage surgery in head and neck cancer. J Otolaryngol Head Neck Surg. 2019;48(1):25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Cui D, Wilson TD, Rockhold RW, Lehman MN, Lynch JC. Evaluation of the effectiveness of 3D vascular stereoscopic models in anatomy instruction for first year medical students. Anat Sci Educ. 2017;10(1):34‐45. [DOI] [PubMed] [Google Scholar]
- 15. Mitsuno D, Ueda K, Hirota Y, Ogino M. Effective application of mixed reality device hololens: simple manual alignment of surgical field and holograms. Plast Reconstr Surg. 2019;143(2):647‐651. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Video S1 A case‐specific three‐dimensional hologram on head mount displays. NOTE TO REVIEWERS: The video file is available in the reviewer score sheet, on the right side of the screen, under “Files.” Let the Editorial Office know if you need assistance viewing this file.
Supplemental Video S2 A three‐dimensional hologram in an operative room. NOTE TO REVIEWERS: The video file is available in the reviewer score sheet, on the right side of the screen, under “Files.” Let the Editorial Office know if you need assistance viewing this file.