Enhanced Visualization: From Intraoperative Tissue Differentiation to Augmented Reality

Abstract

Background

Optimal visualization of the operative field and methods that additionally provide supportive optical information form the basis for target-directed and successful surgery. This article strives to give an overview of current enhanced visualization techniques in visceral surgery and to highlight future developments.

Methods

The article was written as a comprehensive review on this topic and is based on a MEDLINE search and ongoing research from our own group and from other working groups.

Results

Various techniques for enhanced visualization are described comprising augmented reality, unspecific and targeted staining methods, and optical modalities such as narrow-band imaging. All facilitate our surgical performance; however, due to missing randomized controlled studies for most of the innovations reported on, the available evidence is low.

Conclusion

Many new visualization technologies are emerging with the aim to improve our perception of the surgical field leading to less invasive, target-oriented, and elegant treatment forms that are of significant benefit to our patients.

Introduction

When performing surgery, the main sensory input is derived from vision as palpation can only identify larger structures such as organs, vessels, or solid tumors. Only by vision one can differentiate well perfused from ischemic bowel loops, identify distinct anatomic regions, and preserve small vessels and veins, and will immediately be alarmed if a cut is made too deeply resulting in acute bleeding. This holds true for conventional open surgery and is even more important in laparoscopic surgery with dramatically reduced palpatory feedback. This side effect of trauma reduction, the increasing dependence on visual guidance, is an attribute of all atraumatic modalities (also interventional radiology, endoscopy, etc.) and will play a decisive role in the pending dissemination of robotics.

However, not only trauma reduction contributes to the rising significance of vision in visceral surgery but also the general development in medicine and oncology in particular. For example, neoadjuvant treatment regimens in the majority of patients transform the well-palpable tumor mass of an esophageal or gastric cancer into a tiny scar the location of which can only be visually estimated on the basis of preoperative imaging and anatomical landmarks. Also screening programs and general public health consciousness result in an increasing percentage of small, impalpable, early tumors being detected. The concept of ‘non-touch isolation’ has become a paradigm in oncologic surgery, and cancer is nowadays treated by wide excision following germ layers and related anatomical compartments and by almost omitting any tactile exploration. This is best seen in mesorectal resection for rectal cancer where surgery follows a well-defined thin layer of the Waldeyer's fascia. Lastly, we have learned that cytoreductive surgery and resection of microscopically small tumor seeds can improve the survival of patients with metastatic cancer, and here again only the visual assessment of tissue is sensitive enough to identify tumor spots.

With the borders of surgery being pushed towards increased radicalness while reducing complication rates and overall trauma, enhanced visualization support plays a central if not pivotal role. This article gives insight into current developments and methods for enhanced visualization in surgery. A differentiation is made between the following topics:

• i) augmented reality (AR) that provides preoperative imaging in the operating room (OR);

• ii) staining techniques that change the appearance of tissue on the basis of biological or immunohistopathological characteristics;

• iii) imaging methods that use optical technologies for tissue differentiation in the absence of any markers.

Methods

A MEDLINE search was performed by applying the following search terms: ‘surgery’, ‘enhanced visualization’, ‘augmented reality’, ‘fluorescence’, ‘tissue differentiation’, ‘image-guided’, ‘visualization techniques’, ‘optical methods’, ‘optical navigation’.

Relevant articles from this search were selected by 2 independent reviewers after careful screening for significance and up-to-dateness, and are presented in this article. As most articles were case reports only or technical reports reporting on a variety of heterogeneous techniques, a systematic review on this topic was deemed unwarranted.

This article does not strive for a complete review of the current literature but intends to provide a comprehensive overview of relevant topics.

Augmented Reality

Since the very beginning of medical imaging, surgeons have asked to have preoperative data available for intraoperatively guidance. The ultimate wish of surgeons was to no longer be compelled to fuse all available information with the actual OR scenario in their brains but to simply blend this data over the operative field, thus augmenting the real world with computer-based knowledge (fig. 1). Especially in liver and brain surgery and in fields with adjacent vulnerable structures such as vessels and nerves, AR was to reduce the complication rate and offer an advanced surgical approach. AR was established to highlight vulnerable subsurface structures to make surgery safer and more goal-directed. To this end, preoperative image data is displayed with relevant anatomical structures being segmented and accented (e.g. by coloring or adaptation of thresholds). This processed data is then, anatomically correct, superimposed onto the operative field to provide additional information for the surgeon. Another goal in AR is the inclusion of preoperative planning data by displaying cutting trajectories, virtual tumor models, and access routes (e.g. optimized trocar sites) to guide the surgeon through the surgery [1].

Fig. 1.

a In augmented reality, preoperative imaging is processed (segmented, volume rendered, etc.) and b superimposed onto the current operative field. c The resulting image eases the identification and preservation of hidden structures (MITI, Technical University, Munich, Germany).

For visualization in AR, different displaying technologies can be used to augment the surgical field, of which a separate video screen represents the most often used solution. Although easy to install, it requires the surgeon to alternate his viewing direction between the screen and the monitor [2]. In laparoscopic surgery, the surgical monitor or a head-mounted display allow for fused visualization that can be instantly activated and adapted to the surgeon's needs. A more ergonomic option is described by Gavaghan et al. [3] who introduced a projection system to blend the computer data directly with the surgical field. More recently, navigated tablets have been developed that can be moved freely over the region of interest to augment the scenario with relevant information as shown by Kenngott et al. [4].

To provide intuitive guidance, AR requires alignment of the actual operative situation with preoperative imaging which is achieved by registration of representative landmarks and subsequent calibration of the data. To date, different techniques for registration have been described including point marks, (organ) surface registration, volume-based techniques, or simply manual registration. Depending on the applied registration technique and its accuracy, errors ranging from 2 to 8 mm can be incurred [5]. In contrast to ‘solid organ surgeries’ (e.g. ear/nose/throat (ENT), neurosurgery), in visceral surgery, organ deformation and shifting effects caused by manipulation and breathing movements interfere with the accuracy of this registration and hinder the broad application of this approach. Accordingly, to date, AR in visceral surgery is only used in select institutions and requires subsequent recalibration of the registration as well as remodeling of the applied data.

For this purpose, additional imaging modalities and especially ultrasonography are applied to continuously update existing data based on the current situation. In 2012, for example, Kleemann et al. [6] published a case report on navigated liver surgery where the orientation of the liver was registered by laparoscopic ultrasound facilitating repetitive recalibration of the system during the intervention (fig. 2) [6, 7].

Fig. 2.

Navigated liver surgery with ultrasonic guidance as performed at the University of Lübeck. The right upper image displays the laparoscopic view with the ultrasound probe and the right lower image the respective ultrasound image. By means of a navigation system, the probe is integrated in the preoperative imaging for augmented reality (left image) (Prof. Dr. M. Kleemann, University of Lübeck, Germany).

Continuous registration of organ surfaces by laser scanning technologies [8] or time-of-flight cameras [9] displays more sophisticated modalities to update both the location and the shape of visceral organs. These modalities will potentially overcome current limitations of AR in visceral surgery as they allow for ergonomic, automated, and continuous real-time registration that does not alter the workflow.

Besides the aforementioned techniques, different 3-dimensional (3D) registration methods are available today to be used for the establishment of AR [10] but also to supply surgeons with depth information that can be used to navigate mechatronic devices and prevent them from colliding. One of the most promising techniques in this field is structured light that, when blended onto the operative field, identifies the course of the outline of an organ and subsequently its 3D shape (fig. 3).

Fig. 3.

Structured light is a technology used to register depth information about a scenario. Colored stripe patterns are used to identify the outline of a 3D object. By projecting the respective patterns for short instances onto the operative field, the depth information is gained unperceived by the surgeon. a Upper image shows intraoperative application; b bottom pictures display the principle of structured light with an object in white light (left), during registration with structured light (middle), and the reconstruction (right) (MITI, Technical University, Munich, Germany).

Until these technologies are available, intraoperative imaging (computed tomography/magnetic resonance imaging (MRI)/fluoroscopy) with registration of firmly attached optical markers can alternatively be applied and is ideally implemented in a hybrid OR. The acquisition of intraoperative data increases the accuracy of imaging data as it can overcome errors caused by positioning of the patient and can be scheduled to a key step of an intervention (e.g. after mobilization of an organ and partial resection). However, scanners available for intraoperative image registration are of limited quality and in the case of MRI require suitable equipment. Furthermore, preoperative planning and segmentation of volume-rendered data is not available, lowering the level of aid provided.

Not only imaging data are of interest for AR in surgery but also any other data source such as patient parameters (e.g. heart rate, blood pressure) or surgery-related data (e.g. transmitted forces [11]) that can compensate for the lost tactile feedback.

Clinical Application of AR

Usage of AR in the clinical field was predominantly derived from neurosurgery and proved superior in guiding interventions as noted by Marcus et al. [12] who compared the effect of different virtual navigation systems in 50 novices showing that with AR and always-on wire mesh support the neurosurgical task was completed significantly faster. Evidence in this field also comes from Meola et al. [13] who summarized current literature on AR-guided neurosurgery and included 18 studies with a total of 195 successfully resected lesions. The authors found AR to be a reliable and versatile tool; however, prospective randomized trials are lacking, and the small patient numbers reported on do not allow for a reliable assessment.

In visceral surgery, available reports focus on hepatobiliary interventions, such as the publication by Okamoto et al. [14] who reported on 19 patients operated on for pancreatic and liver diseases and in whom AR was applied to define the resection line and identify hidden vessels. The authors concluded AR to be feasible but requiring substantial time for preparation (image segmentation) and set-up. Besides registration accuracy, difficulty was reported defining suitable outcome parameters to validate AR application, especially as it was preferentially applied in complex surgeries. Pessaux et al. [15] operated on 3 patients suffering from liver tumors using a robotic system and successfully performed AR-guided anatomical resection. AR in all patients allowed for a correct dissection of the tumor with precise and safe recognition of all major vascular structures and was easily implemented. No adverse events were registered. A comparable technique in open surgery was published by Tang et al. [16] for the resection of a hilar cholangiocarcinoma. They included AR to identify the hilar structures and to guide surgery. The list can be completed by reports from other surgical fields, mainly ENT, craniofacial surgery, and urology [17], stating that likewise AR is applicable in these fields; however, no randomized trials nor large clinical studies exist to prove the superiority of AR in comparison to standard techniques. Still, the accuracy, user friendliness, and time needed to prepare and implement the virtual data are being criticized and demand intelligent analytical tools such as random forests and Bayesian networks.

Tissue Staining

Marking of relevant regions by staining is a simple and reproducible approach to tissue differentiation and already well-established in visceral surgery. In this context, gastroenterologists have for years injected different kinds of dye (methylene blue, India ink, etc.) into the intestinal wall to visualize the location of tumors, or surgeons inject pigments into the skin for identification of sentinel lymph nodes [18]. Favorably, no alignment of preoperative data to the intraoperative situs is necessary as the demarcated site simply appears stained. Easily performed, this technique has several drawbacks such as pigment spillage (overstaining), non-localizability of markings due to injection in secluded regions (e.g. mesentery), bleaching effects, and dependence on correct injection [19].

To overcome these problems, researchers worldwide strive to develop new markers that provide superior marking quality (optical stability and contrast, tissue adherence, preservation of the unmarked tissue) and/or higher tissue specificity. The main focus here is on fluorescent dyes.

Unspecific (Non-Targeted) Markers

Indocyanine green (ICG) is the most popular marker as it is already approved for clinical application but also due to its favorable characteristics. ICG is a fluorophore that emits light in the range of the so-called near-infrared window (700–930 nm), a range of the light spectrum in which the absorption by blood, water, and oxygen is the lowest and the fluoroscopic signal is the strongest. To derive a signal, excitation of ICG with light of a wavelength similar to the adsorption range and registration of the emitted signal by dedicated camera systems are required [20]. ICG can be injected intravenously and is suitable for visualization of tissue perfusion, e.g. gastrointestinal (GI) anastomosis or transplants (fig. 4). In a recent publication, Boni et al. [21] evaluated the value of ICG perfusion analysis of lower rectal anastomoses and compared a group of 42 patients to a case-matched historical control group. Finding a reduced (albeit not significantly) leakage rate, the authors concluded this technique to be effective in reducing leakages after intestinal anastomosis. A systematic review on perfusion analysis completed by Degett et al. [22] summarizes the current literature on this topic and concludes perfusion analysis to be a promising tool to reduce anastomotic complications; however, evidence was low due to missing prospective comparative trials. Of note, even in the group of patients with assessed perfusion, almost 4% of patients developed an anastomotic leak. Also, in plastic surgery, quality of ICG perfusion correlated well with postoperative healing and survival of pediculated flaps [23, 24]. ICG is excreted via the liver and can visualize the bile duct with a high sensitivity even in obese patients [25, 26]. As liver tumors often affect intrahepatic bile ducts, accumulation of ICG in liver tumors can be used to identify metastasis and facilitate resection as shown by van der Vorst et al. [27] (so-called ‘enhanced permeability and retention effect’ [28]). With an adequate interval between application of ICG and intraoperative assessment, they could find a fluorescent rim surrounding colorectal liver metastases. ICG can also be injected locally and into separated vessels to demarcate anatomical liver segments and support their image-guided resection [29, 30]. Other applications of ICG angiography concern lymph vessels and nodes during sentinel lymph node mapping or the surgical treatment of lymphedema for which several studies have in principal shown the applicability of this approach [31]. Generally, ICG fluorescent imaging, due to its accumulation in certain tissues, allows for improved identification of tumor seeds also in peritoneal carcinomatosis, improving the surgical therapy [32].

Fig. 4.

a, b Bowel perfusion analysis by indocyanine green injection. The fluorescent signal indicates perfect perfusion of the anastomosis assuring its later healing process (MITI, Technical University Munich, Germany).

Unfavorably, the excitation and emission spectrum of ICG are closely adjoined, requiring sharp filters to achieve a suitable signal but also resulting in a weak signal. Especially if the object of interest is covered by surrounding tissue (e.g. fat), the weak signal can interfere with its identification as the signal cannot penetrate through tissue beyond 8–10 mm thickness [28, 33, 34].

Besides ICG, a small number of alternative fluorescent markers are approved for clinical application but have remained in their niche. Methylene blue for example is useful in the detection of insulinomas of the pancreas but besides this affinity has inferior characteristics compared to ICG [35]. Fluorescein, which is well known in ophthalmology, can also be applied for the identification of tumor deposits but is utilized almost exclusively in neurosurgery [36]. This applies also to 5-ALA, which additionally holds some indications in diagnostic laparoscopy, especially in gastric cancer [37].

Besides these traditional, mostly approved, markers, several fluorescent dyes have been described over the past years, but have not yet come into clinical practice. In this context, the group of cyano-dyes (e.g. Cy-5.5, Cy-7) and CW800 have to be mentioned [38], but also nanoparticles that offer superior optical characteristics and other favorable features (fig. 5) [39]. However, the pending approval for clinical application as well as some remaining skepticism concerning biocompatibility (toxic components, gene alteration, etc.) currently hamper their further development [20].

Fig. 5.

a, b As compared to traditional fluorophores such as indocyanine green or fluorescein, fluorescent nanoparticles (QDOTS) offer favorable attributes, e.g. higher quantum yield, high tissue contrast, and no bleaching. In the future, they could be used for marking intestinal lesions (MITI, Technical University, Munich, Germany).

All the above-mentioned dyes have in common a certain nonspecificity and demarcation of tissue according to pharmacokinetics rather than targeting specific tissue characteristics. This is the reason for low signal contrast and causes an interfering background noise (staining of all perfused tissue) which lowers the overall sensitivity. Only by considering the respective distribution times of the maker, a utilizable signal can be achieved.

Specific (Targeted) Markers

The mentioned problem of nonspecificity falls away when linking the (fluorescent) marker to a specific antibody so that the dye will bind to the respective epitope. With this, one can easily differentiate healthy from tumorous tissue, allowing for so-called ‘image-guided surgery’ which facilitates complete tumor resection while preserving most of the surrounding healthy tissue. First published in humans in 2011 for the treatment of peritoneal carcinomatosis from ovarian cancer when van Dam et al. [40] applied folate-labelled fluorescein isothiocynate (FITC) for image-guided surgery, this principle has attracted worldwide attention. Several conjugates have been described so far that adjust the intracorporal distribution of dyes to specific needs or that allow for specific demarcation of molecular cell surface properties [41]. However, all available markers have so far only been applied in small sample studies or animal models [42], although various human trials have been initiated [28]. As the introduction of disease-specific contrast agents has to first overcome regulatory hurdles, simple modification of current fluorophores that can also result in increased specificity might be a valuable alternative. In this context, identification of the lymph drainage was improved by conjugating ICG to polyethylene glycol (PEG) and enabled Proulx et al. [43] to identify dysfunctional lymph vessels in the case of metastatic tumor spread. Alternatively, tissue-inherent enzymes can activate modified fluorescent probes and thereby allow for specific marking as demonstrated by Wunderbaldinger et al. [44]. Targeting tumor-specific antigens with formulated dyes is currently the main focus of research and addresses all kinds of epitopes from carcinoembryonic antigen (CEA) to vascular endothelial growth factor. Already in 2008, Kaushal et al. [45] were able to identify residual tumor deposits of colorectal and pancreatic cancer by using CEA-labelled fluorophores; similarly, Sano et al. [46] used epidermal growth factor receptor/trastuzumab-labelled dyes to localize breast cancer cells. A list of ongoing studies is given in a consensus report published by Rosenthal et al. [47] in 2016. This article (same as [20]) also gives insight into the problem of approval of these new dyes and discusses potential solutions and recommendations.

Optical Methods

While in open surgery vision is gained directly through the surgeon's eyes, and enhancement is more or less limited to aspects like the illumination of the surgical field or the use of magnifying glasses, some promising modifications for visualization are available in minimally invasive surgery.

Nowadays, state-of-the-art laparoscopic optics provide high resolution and 3D vision, although their benefit is still debated and/or difficult to estimate. Notwithstanding, Pierre et al. [48] in 2009 found high-definition (HD) resolution to improve surgical performance and to ease the identification of fine structures as well as 3D spatial positioning. For 3D visualization, current publications almost exclusively demonstrate its superiority as compared to standard view. In a comprehensive evaluation, our group compared a state-of-the-art HD monitor with a comparable 3D system and additionally to an autostereoscopic 3D display and an artificial display that offered 3D vision of a quality that was almost comparable to real view [49]. Interestingly, not only novices profited from stereoscopic view but also experts with an improvement in task performance of almost 20%. In our investigation, we also addressed workload issues and objective parameters such as instrument-handling economy. Consistently, 3D vision showed advantages over standard view; however, it could also be demonstrated that the real not-camera-mediated view is still better and that there is room for further improvement. Our results are in line with other publications in this field such as those by Storz et al. [50] and Sorensen et al. [51] and emend the outdated assumption of inferiority of 3D vision that resulted from older publications [52].

Interestingly, and in contrast to the available ‘in-vitro’ evidence, studies that compare 3D vision to standard view during real-life operations generally fail to demonstrate an advantage. For example Curro et al. [53] compared the effect of 3D vision during laparoscopic cholecystectomy and could not find a significant advantage, similar to Agrusa et al. [54] who performed a case-control study for laparoscopic adrenalectomy which equally showed no benefit for 3D vision. This might be due to the fact that the stable situation in the laboratory facilitates the adjustment to 3D working conditions; also, the tasks performed during the investigations were less demanding (cholecystectomy and adrenalectomy), and most studies were undertaken with experienced surgeons well trained in 2D vision.

Spectral imaging is another feature that can potentially enhance laparoscopic vision. By selecting or filtering specific wave lengths of the visible light, a miscolored modified image of the operative field is generated that eases the recognition of different structures such as vessels or tumor deposits. Being widely applied in GI endoscopy for the identification of occult polyps [55], spectral view is at present only used in GI surgery, and here it is of questionable benefit. While Schnelldorfer et al. [56] could not prove any advantage if spectral imaging for the detection of tumor deposits was applied, Kikuchi et al. [57] at least detected that dilated vessels were more easily differentiated and hypothesized that this could potentially support tumor diagnosis. However, in combination with fluoroscopic markers and/or a better integration into the workflow, spectral imaging could offer additional support that could become helpful in the future.

More popular in photography and press, high dynamic range (HDR) imaging offers higher contrast and a more realistic reproduction of a scenery than standard view. Applications in medicine are scarce; however, our group could show that HDR imaging can also be used to erase fog and dust from cutting devices and to moderate blending effects (reflections, light) (fig. 6) [58].

Fig. 6.

a, b High dynamic range imaging is not only useful for extending the dynamic range but can also be supportive by erasing/filtering disturbing image elements like mist or blending effects (MITI, Technical University, Munich, Germany).

Another field that is currently in development and may potentially impact on surgery in the future are microscopic viewing technologies such as confocal laser microscopy (CLM), optoacoustic tomography, and optical coherence tomography (OCT). While in CLM a narrow laser-illuminated tissue portion is scanned with a confocal microscope, OCT analyses interferences between a measuring and a reference beam of coherent light with the measuring beam being reflected from the tissue of interest. Both techniques allow for the identification of cellular structures in vivo and the possibility of optical biopsies but will require intensive work and computer support before clinical application. While CLM is preferentially applied in flexible endoscopy to differentiate between benign and malignant alterations [59, 60], OCT is currently a highly contributing imaging modality in ophthalmology [60]. However, in surgery, the need for an in-vivo applicable near microscopic imaging modality also exists, and we already demonstrated that the differentiation between benign and malignant tissue is possible in that way (fig. 7) [61].

Fig. 7.

Optical coherence tomography (OCT) imaging of resected colon tissue in 2D (left lower and right images) and 3D view (left upper). While the 2D sections allow identification of cellular structures and intestinal layers, a clear differentiation between benign and malignant was possible only by 3D reconstruction (MITI, Technical University, Munich, Germany and Fraunhofer IPT, Aachen, Germany).

Discussion

Sight is one of our most important senses and the most significant one when performing minimally invasive surgery. Correspondingly, it is no wonder that improvements in the visual reproduction of the OR scenario are correlated with improvements in surgical performance. With 3D and HD resolution, current laparoscopic devices can now reproduce the operative field in a quality that is almost equal to real vision.

Notwithstanding, and as shown in this article, beyond the field of realistic and lifelike reproduction, the domain of enhanced visualization and AR has emerged which provides additional information for the surgeon by highlighting structures of interest, pointing out vulnerable structures, and guiding the surgeon through an intervention. As the integration of these new technologies is more easily achieved in laparoscopic surgery where camera systems have always been part of the armamentarium, we see a chance for this field to further evolve and to turn a former drawback into an advantage. At the same time, any new development is to also be implemented in open surgery, albeit with much more effort.

While some enhanced visualization modalities such as narrow band and HDR imaging are easily implemented, the application of new dyes and fluorescent markers is hindered by regulatory aspects that will delay clinical introduction and potentially prevent some promising developments from ever being used in humans. Surgeons, accordingly, are asked to announce their need for enhanced visualization techniques as these are key to further reducing the interventional trauma and improving our performance. As suggested by Tummers et al. [20], Rosenthal et al. [28], and de Boer et al. [47], we believe it to be permissible to approve new surgical markers based on criteria different to those applied for diagnostic markers as they are used in only a small number of well-indicated instances and will allow for new forms of therapy that will possibly improve the chances of curing cancer. Accordingly, we must always balance the resultant advantage versus the potential risks when examining new approaches.

In conclusion, many new visualization technologies are emerging which are supposed to improve our perception of the surgical field leading to less invasive, target-oriented, and elegant treatment forms that are of significant benefit to our patients.

Disclosure Statement

The authors did not provide a conflict of interest statement.