Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning

Abstract

According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.

INTRODUCTION

Gastrointestinal cancers are comprised of all cancers that occur in digestive tract organs, such as the stomach, large and small intestine, pancreas, colon, liver, rectum, anus, and biliary system. Collectively, they represent the first leading cause of cancer-related deaths. Of these, gastric cancer (GC) alone accounts for the fourth leading cause of cancer morbidity, at the global level.^1,2 Prevention is very important for avoiding GC, but the plethora of causes behind this severe pathology, together with their geographic and ethnic dependencies, yield significant challenges in implementing efficient primary prevention strategies able to reduce the current related mortality rates. Furthermore, a statistical model³ published by Tomasetti and Vogelstein in 2015, with conclusions consolidated in their follow-up article,⁴ brought evidence that two-thirds of the variation in adult cancer risk across tissues can be explained primarily by “bad luck,” which is related to the number of random mutations occurring in genes that can drive cancer growth. These aspects suggest that secondary prevention, obtainable through the early detection of precursor lesions or cues, should be the major focus for reducing the high mortality rates currently associated with GC. The three main steps that are of utmost importance for the proper endoscopic diagnosis of gastrointestinal cancers are “detection,” “characterization,” and “confirmation.”⁵ Diagnostic methods based on endoscopic imaging are currently regarded as being the most sensitive and specific in detecting neoplasia in patients suspected of harboring GC. In this perspective, we review relevant progress in three emerging technologies that could significantly boost the capabilities of next-generation endoscopes for GC diagnostics, and for gastrointestinal cancers in general: (i) biomimetic video systems (with special focus on compound eye cameras), (ii) non-linear optical microscopies, and (iii) artificial intelligence. We discuss as well some of the most prominent challenges that lie en route to translating these valuable technologies to the clinical practice and a potential configuration of a next-generation endoscope that would synergistically combine them.

CURRENT ENDOSCOPIC APPROACHES FOR GC DIAGNOSTICS

In the next paragraphs, we will briefly discuss the most popular approaches used for endoscopic diagnostics of GC, a part of which are presented in Fig. 1, highlighting some of their advantages and limitations.

FIG. 1.

Overview of traditional modalities for GC diagnostics.

Risk factors for GC development are gastric atrophy (GA) and intestinal metaplasia (IM), and surveillance is recommended for extended GA/IM.⁶ GC usually develops via low-grade and high-grade intestinal neoplasia. Early GC may be amenable to minimally invasive endoscopic therapy with endoscopic mucosal resection or endoscopic submucosal dissection, thus obviating the need for total gastrectomy. Endoscopic therapy requires high-resolution imaging to allow for the detection and characterization of lesions. While resolutions in the millimeter range can often be useful to identify well-developed tumors, and to define a lesion's margins, micron-level resolutions are required to probe microvascular, and micro surface patterns. Furthermore, sub-micron resolutions can be highly valuable for identifying dysplastic tissues with subtle cues, infiltrations of cancer cells in healthy tissues, and for spotting cell aspects characteristic of aneuploid cells. Resolutions at the nanometer level can provide access to important aspects, such as overexpression of cancer-related proteins, or to cell processes relevant for cancer genesis, such as cytokinetic abscission, nuclear envelope reassembly, nuclear pore complex formation, etc. Precancerous conditions, such as H. pylori-gastritis, and lesions, such as GA and IM, may be multifocal or diffuse, and the concept of field cancerization requires meticulous screening and surveillance of the complete gastric mucosa. Population-based and opportunistic screening for GC is performed in high-incidence areas in Asia, such as Japan and South Korea, and has been shown to significantly reduce mortality from GC.⁷ Diagnosis at an early stage is associated with improved survival,⁸ and endoscopic screening has resulted in approximately 40% risk reduction of GC mortality and seems superior to both no screening and radiographic screening in high-risk regions.⁹

A GC miss rate of around 10% has been reported for esophagogastroduodenoscopy (EGD)^10–12 (with some surveys including pre-HD-trials). Reasons for failure to adequately diagnose early GC may be due to a multitude of reasons, including non-detection, detection with inadequate assessment or biopsy protocol, or sampling/pathology error with inadequate follow-up.¹¹ The large surface of the stomach may put GCs at a particular risk of being missed during EGD. Therefore, systematic endoscopic mapping of the complete stomach has been recommended, with a recording of 22 endoscopic images of landmark areas to standardize reporting.¹³ Similarly, time spent with EGD¹⁴ and structured training¹⁵ has been positively linked to detection rates of gastric lesions. New endoscopic technologies that improve imaging of large surfaces of the gastrointestinal tract (GIT), in terms of clarity, depth-of-field, or postprocessing, have a high potential to lower the GC miss rate.

Endoscopic diagnosis of gastric neoplasia and precursor lesions follows a multistep approach, requiring the detection of suspicious areas, followed by endoscopic characterization and delineation for therapy or targeted biopsies. Gastroscopy with high definition (HD) white light endoscopy (WLE) is the standard of care. It has been augmented by two types of image enhancement during endoscopy, dye-based enhancement (chromoendoscopy), and equipment-based enhancement. The latter is sometimes referred to as “virtual chromoendoscopy” and includes both pre- and post-image-acquisition techniques such as narrow-band imaging (NBI), i-Scan and optical enhancement (OE) i-Scan, flexible spectral imaging color enhancement (FICE), and blue laser imaging (BLI), all of which are available and reversible by pressing a button on the endoscope's handpiece. Chromoendoscopy requires the physical application of a dye onto the mucosa, which is time- and material-consuming, but has the advantage of not relying on dedicated endoscopes. The ability of chromoendoscopy to enhance the detection of GC has been demonstrated by a meta-analysis using different dyes.¹⁶ For equipment-based image enhancement, diagnostic criteria for IM and GC have been validated for NBI.¹⁷ Once a lesion has been detected, magnification endoscopy was evaluated for on-site characterization based on the presence or absence of a clear demarcation line and irregular microvessel and/or microstructure patterns. The presence of these criteria was indicative of a GC lesion. Meta-analyses have confirmed the ability to predict the occurrence of neoplasia with high accuracy.^18–20

Another approach for enhanced endoscopic imaging has been to improve the limited field of view of standard forward-viewing endoscopes (usually between 140 and 170°) by wider-angle visualization. A prominent example of these efforts has been marketed under “full-spectrum endoscopy (FUSE),” permitting a panoramic 245° (gastroscope) to 330° (colonoscope) field of view (FOV). This has been shown to enable enhanced adenoma-detection rate during colonoscopy,^21,22 with increased absolute number of detected adenomas,²³ and lower adenoma miss-rate,²⁴ but has not been extensively studied in the upper GIT. At present, FUSE is no longer commercially available. Another emerging method for imaging inside the human body is capsule endoscopy,²⁵ which has been mainly used for evaluation of the mid-GIT. However, with a passive capsule, complete visualization is severely limited in the stomach for its capacious anatomy.²⁶ One approach under investigation to overcome this limitation has been the external steering of a magnetically assisted capsule, which can be navigated similarly to the movements of an endoscope in forward or backward direction, tilted, or rotated.^27,28 A series of pitfalls still pending to be addressed in the quest for more efficient diagnostics with capsule endoscopy are discussed in the recent review of S. H. Kim and H. J.Chun.²⁹

The ultimate confirmation of the diagnosis and evaluation of risk factors usually still relies on histologic assessment after biopsies, although the trials mentioned above can be indicative of mandating less or even no biopsies if a highly confident endoscopic diagnosis can be made. Excisional biopsy is impractical for screening in real-time high-risk patients who may have multiple lesions and endoscopic inspections are associated with a vast number of unnecessary biopsies, which are labor- and cost-intensive. To alleviate these problems, attempts to provide immediate microscopic analysis during endoscopy have been made with confocal laser endomicroscopy (CLE) and endocytoscopy.^30,31 The use of CLE has been found to result in a higher diagnostic yield of IM on a per-biopsy rather than a per-patient basis, in endomicroscopically targeted biopsies in comparison to WLE³² or FICE.³³ A recent meta-analysis has found CLE to be highly accurate in the diagnosis of gastric atrophy and IM.³⁴ In a large prospective trial, the establishment of CLE criteria and subsequent validation provided 98% accuracy for immediate GC diagnosis.³⁵ CLE has also been linked to fluorescence-based molecular detection of GC-associated antigens in preclinical models.^36,37 Although often evaluated separately within a trial setting, the different modalities of HD-WLE, dye- or equipment-based image enhancement, magnification, and microscopic endoscopy can be combined in a clinical approach.³⁸

EMERGING TECHNOLOGIES

Over the past decade, we have witnessed significant progress in multiple scientific and technological fields with the potential to enable a new generation of endoscopic imaging systems capable to exhibit unprecedented performance. Next we focus on reviewing a selected set of relevant advances in three of these fields and their potential importance in endoscopic diagnostics.

Biomimetic vision: Compound eye cameras, and others

A key characteristic of endoscopic imaging systems is their FOV, which dictates the extent of the scene that can be imaged at any given moment during the investigation procedure. Low FOV translates to the physician having to spend time repositioning the endoscope so that the regions of interest are in sight. Moreover, in standard forward-viewing endoscopes (SFV), areas that need to be observed many times can be visualized only tangentially, not in an optimal orientation and the procedure is thus prone to miss lesions. Higher FOV thus saves time, as instead of endoscope positioning it allows the physician to focus his/her attention on the scene that is imaged, which can lead to an increase in the lesion detection accuracy. Current flagship gastroscopes built on the traditional SFV architecture provide a FOV of ∼150°, and the Fuse^® system (Endochoice^®, USA), provides a 245° FOV for gastroscopic imaging. A multi-center tandem trial provided a convincing estimate on the massive advantages offered by increased FOV,³⁹ demonstrating 69% more colon adenomas detected using the higher FOV Fuse^® system in comparison to SFV endoscopes,³⁹ while other later studies shed further light on the advantages provided by this technology.^23,24

Light sensing organs specific to various animal and insect species have the potential to play in the near-future an important role in considerably extending the FOV of endoscopes while also providing other advantages that can result in reduced intervention time and enhanced success of the diagnostics procedure. Among these, compound eyes (CE), Figs. 2(f)–2(h), which can be found in arthropods (i.e., insects and crustaceans), have recently gained a massive focus of attention from the engineering communities preoccupied with optical imaging. These organs are made up of small, multiple optical units (i.e., ommatidia) per eye, while camera-type eyes in vertebrates have a single lens.^40,41 Compared with single-lens systems, CE offer wider FOV with negligible optical distortion, nearly infinite depth of field, and high temporal resolution.^40,42 Other characteristics, such as excellent photosensitivity in low-light environments, multi-spectral imaging, and polarization perception enhancement, can also be found in some CE.^40,42,43 To date, multiple noticeable attempts have been made to develop artificial CE in the form of Compound Eye Cameras (CEC), e.g., Refs. 44–51. The insightful reviews of Lee et al.,^43,52 Cheng et al.,⁵³ and Chung et al.⁵⁴ nicely present the progress that has been made to date in this field.

FIG. 2.

A selection of recent biomimetic imaging systems. (a) cylindrical vision sensor fabricated in columnar bar arrays,⁴⁶ with morphology and properties resembling those of primitive (f) and modern (g) arthropods; (b) hemispherical compound eye camera based on stretchable electronics technology,⁴⁴ inspired from the vision system of modern arthropods (g) and (c) ultrathin digital camera based on a sandwiched configuration of concave microprisms, microlenses, and pinhole arrays on a flat image sensor, resembling the compound eye of an endoparasite⁵¹ (h); inset: SEM images of concave microprism arrays filled with a black polymer; (d) multi-camera panoptic video system⁴⁷ developed for endoscopic imaging based on miniaturized standard video cameras; inset: hemispherical deployment of cameras (in red) for achieving biomimetic vision properties; (e) integrated device on a 3D structure for panoramic imaging featuring graded microlens arrays with flexible silicon photodiode arrays; (f), (g), (h), and (j) compound eyes of the (f) extinct trilobite Erbenochile erbeni, (g) Drosophila melanogaster (fruit fly), (h) Xenos peckii endoparasite, and (j) Uca arcuate bowed fiddler crab. The biomimetic cameras depicted on the top row replicate the vision capabilities of the corresponding natural models depicted on the bottom row. (i) Image collected in a human colon model with the biomimetic endoscopic video system depicted in (d) with 180° FOV (140° contained in the red square). (a), (f), and (g) Reproduced with permission from Floreano et al., Proc. Natl. Acad. Sci. U. S. A. 110(23), 9267–9272 (2013). Copyright 2013 PNAS.⁴⁶ (b) Reproduced with permission from Song et al., Nature 497(7447), 95 (2013). Copyright 2013 Macmillan Publishers Limited.⁴⁴ (c) and (h) Reproduced with permission from Keum et al., Light 7(1), 80 (2018).Copyright 2018 Authors, licensed under a Creative Commons Attribution 3.0 Unported License.⁵¹ (d) and (i) Reproduced with permission from Cogal and Y. Leblebici, IEEE Trans. Biomed. Circuits Syst. 11(1), 212–224 (2017). Copyright 2017 IEEE.⁴⁷ (e) and (j) Reproduced with permission from Lee et al., Nat. Electron. 5(7), 452–459 (2022). Copyright 2022 Springer Nature Limited.⁵⁷

A major challenge in the development of CE imaging systems with high FOV is the implementation of highly curved photodetector arrays on a hemispherical geometry and their integration with micro-optics. An important strategy to implement a CEC with wide FoV consisted in the use of a flexible printed circuit board with cylindrical shape,^46,55 Fig. 2(a). Imagers produced this way were shown to exhibit good cylindrical FOV (180° × 60°), and local adaptation to illumination. However, this unidirectional geometry did not provide a hemispherically wide FOV. One route toward achieving a CEC with omni-directional FOV relied on the assembly of polymer microlens arrays (MLAs) and stretchable electronic devices,⁴⁴ Fig. 2(b). The proof-of-concept images produced by this CEC architecture showed exceptionally wide FoV (160° × 160°),⁴⁴ without blurring or aberrations. Coupling two of these devices in back-to-back configurations can result in an FOV that almost fully covers all cardinal directions, which would represent a tremendous advantage for endoscopic imaging. Such configurations can be applied as well to other types of CE, such as refracting/reflecting superposition eyes and neural superposition eyes.⁴⁰ Another distinct type of CEC, taking the form of an ultrathin digital camera consisting of microprisms, microlenses, and pinhole arrays on a flat image sensor, was achieved by mimicking the vision principle of Xenos peckii, an endoparasite of paper wasps,⁵¹ Fig. 2(c). This unique optic configuration enables wide FoV, accompanied by high spatial resolution and sensitivity. Another notable progress in the field of biomimetic video system was recently reported by Liang et al.,⁵⁰ who demonstrated a single one-piece lens with curved hexagonal MLAs, which enables 92° FoV and an f-number of 1.68. This architecture succeeded in exploiting the principles of both the human eye as well as an insect's CE for achieving a robust device that benefits from advantages characteristic to both vision models.⁵⁰ In terms of image resolution and physical dimensions, the above-mentioned approaches need to be improved for them to become reliable solutions for enhanced endoscopic imaging. One way to overcome the current resolution limitations of these devices consists of the integration of discrete components onto a hemispherical surface. A prominent example for such potential approaches is the work of Cogal and Leblebici⁴⁷ who demonstrated 1.1-megapixel images under an FoV of 180° × 180° in a human colon model Fig. 2(h), using a 5 mm radius hemispherical shaped video system inspired from CE which is based on a distribution of multiple off-the-shelf cameras, Fig. 2(d). These approaches could eventually be augmented by employing latest hour single-lens biomimetic cameras mimicking fish vision, such as the Cichlid-inspired camera reported by Kim et al.,⁵⁶ which features a more convenient miniaturization factor compared to other biomimetic variants, and excellent technical performance, yielding wide FOV, low optical aberrations, wide depth of field, simple visual accommodation, and excellent light sensitivity. Another study⁵⁷ that carries important relevance for potential endoscopic imaging reported a biomimetic vision system inspired by the fiddler crab eye, featuring a flat microlens with a graded refractive index structure for suppressing the defocusing effect caused by changes in the external environment [see Fig. 2(e)]. The panoramic vision of the fiddler crab was replicated by integrating graded microlens arrays with flexible silicone photodiode arrays on a 3D spherical structure. The resulting vision capabilities were successfully demonstrated in both air and water.

Other CEC-related efforts that we find important to mention in this section are the development of CECs⁵⁸ capable to address the infrared domain, important for endoscopic imaging,⁵⁹ and the development of CECs capable of multispectral imaging in the visible domain,⁶⁰ which could potentially inspire the implementation of biomimetic modalities capable of virtual chromoendoscopy (e.g., NBI, BLI, etc.).

Additionally, the recent work proposing biomimetic cameras mimicking the cuttlefish-eye⁶¹ can turn out to be highly relevant for the field of endoscopy, where uneven illumination and low contrast⁶² represent two of the most critical factors affecting the quality of the medical act. In this design, the W-shaped pupil compensates for the uneven vertical light distribution by reducing incident lights from the top of its vertical FOV. In addition, the cylindrical silicon photodiode array, incorporating a high-density belt-like pixel region, enables high-acuity imaging in the region of interest. Another key feature of this camera is provided by a flexible carbon nanotube-polarizing film integrated into the surface of the photodiode array, which enables polarization-sensitive imaging. Another recent design, this time inspired by the characteristics of the human retina,⁶³ can also be considered as highly relevant for the field of endoscopy, given the utility of NBI imaging in this field. In addition to biomimetic cameras providing advantages in terms of optical parameters such as FOV, contrast, and others, recent work in the field of temperature-sensitive cameras can also turn out to be highly useful for the field of endoscopy, given the temperature differences between healthy tissues, malignant tumors, and benign tumors.^64,65 Building on the visual characteristics of snakes, which possess exceptional infrared perception conferred by pit organs, Ding et al. have proposed a biomimetic hemispherical thermoelectric sensor⁶⁶ based on a high-density array of ionic thermoelectric polymer nanowires, which exhibit high thermopower with subtle temperature difference. This sensor mimicking the thermal receptors in the pit organs of snakes provides an ultrawide FoV of 135°.

Non-linear optical microscopy

In clinical approaches for GC diagnostics, if an abnormal tissue region is identified during the endoscopic exam, a small fragment is extracted from it and then processed with fixation and staining for histopathological examination. The most important disadvantages of this approach are invasiveness, artifacts, sampling error, time consumption, high costs, interpretive variability, and failure to take therapeutic decisions in real-time.^67–69 As a result of their optical sectioning capabilities and ingenious contrast mechanisms, non-linear optical microscopies (NLOs) have emerged over the past couple of decades as very promising alternatives to traditional histology, taking into account their possibilities to alleviate the aforementioned drawbacks and offer label-free imaging. Such techniques are capable of characterizing in vivo, ex vivo, and in vitro tissue morphology, functionality, and biochemical composition^70–72 in a noninvasive way, with tissue penetration depths of several hundred microns⁷³ in routinely used configurations, and beyond 1 mm when used in association with long-wavelength excitation sources.⁷⁴ Furthermore, their important potential for virtual histology can be augmented by recent image processing techniques that can represent the collected images in color schemes familiar to histopathologists.^75–77

The use of NLO in diagnostic scenarios exploits endogenous optical signals generated by the tissues upon their interaction with a pulsed laser beam used for excitation. While tissue bioimaging applications based on NLO techniques have also been implemented in combination with contrast arising from exogenous agents,^78–81 the vast majority of NLO studies on tissues exploit intrinsic optical contrast, given their underpinning contrast mechanisms, corroborated with tissue composition and architecture. Strong emphasis has been placed to date on interrogating with NLO techniques autofluorescence corresponding to endogenous chromophores, e.g., NADH or FAD,⁸² and harmonic generations originated by non-centrosymmetric molecules present in the human tissues, such as myosin, tubulin or collagen, which are of interest with respect to pathology assessment.⁸³ The two most popular NLO modalities that exploit these optical signals are Two-Photon Excitation Fluorescence Microscopy (TPEF) and Second Harmonic Generation Microscopy (SHG), respectively. Their use has been demonstrated so far for exploring anatomical and physiological aspects that are relevant for assessing the state of various tissues and the progression of multiple pathologies.^84–86 The advantages offered by TPEF can also be exploited in the frame of Fluorescence Lifetime Imaging Microscopy (FLIM), a technique that measures how the fluorescence intensity decays in time following excitation (a.k.a fluorescence lifetime).⁸⁷ Coherent Anti-Stokes Raman Scattering Microscopy (CARS) and Stimulated Raman Scattering Microscopy (SRS) represent two other NLO techniques that have been demonstrated as being very useful for probing tissue states, based on the intrinsic vibrational contrast of endogenous biomolecular species, such as lipids.^88,89 Same as TPEF and SHG, Raman-based techniques can probe both soft,^90,91 and hard tissues.⁹² The advantages of NLO techniques in comparison with Confocal Laser Scanning Microscopy, such as superior optical penetration due to near-IR excitation,⁷³ label-free imaging,⁷¹ optical metabolic imaging based on coenzyme detection,⁸² and out-of-focus photodamage, have been thoroughly discussed.^70,85

To date, TPEF and SHG have been successfully employed for assessing the architecture of gastrointestinal tissues in both animal models and human biopsies in multiple studies^93–106 (see Fig. 3). In addition to the many studies reporting thorough characterization assays on ex vivo and fixed tissues, progress in miniaturization, enabled also in vivo imaging of the murine colon, for which SHG and TPEF were successful in probing tissue morphology changes during the time course of intestinal inflammation associated with ulcerative colitis.¹⁰⁷ These efforts demonstrate TPEF and SHG as solid alternatives for gastrointestinal tissue diagnostics, and their potential to address this problem is related to their ability to provide cellular information such as abnormal cell morphology or size variation of cell nuclei, which are important indicators for pathological or precursory cues. Other features that can be characterized with TPEF, such as tissue modifications linked to blood vessel hyperplasia, or inflammatory reactions, hold as well important potential for understanding in detail how the morphologies of the abnormal cells modify the structures of the gastrointestinal mucosa. Furthermore, the potential of FLIM imaging for the characterization of cancers has been demonstrated in a wide variety of experiments,^87,108,109 including a series of significant efforts addressing pathologies of the GIT^110,111 and accessory organs.^112,113 Its significant potential concerning this subject is connected to the fact that normal, dysplastic, and cancerous tissues exhibit different metabolic activity levels, different biochemical configurations, conformational states, ion concentrations, etc., which impact the decay rates of the endogenous fluorophores present in distinct tissue types. One of the fundamental applications of SHG with respect to the characterization of tissues and disease diagnostics consists of the imaging of collagen,¹¹⁴ the main structural protein in the extracellular matrix of animal tissues. Investigating collagen distribution with SHG in tissues^86,115–117 enables an accurate and noninvasive assessment of extracellular matrix modifications, which represent a hallmark of a wide range of pathologies, including cancer.⁸³ SHG complements the imaging potential of fluorescence-based NLO techniques for GC diagnostics through its superb possibilities to investigate collagen deposition and architecture changes in gastric tissues^118,119 in 2D and 3D, which are as well significant signatures of GC progression.^120,121 This was nicely demonstrated in the work of Makino et al.,¹²² who employed both TPEF and SHG, collectively known as Multiphoton Microscopy (MPM), for creating an atlas of images of the entire normal mouse GIT, including some accessory organs, Figs. 3(a)–3(c). This atlas together with TPEF/SHG images collected on mouse models of experimental colitis and colorectal neoplasia, as well as the images collected on human tissues from subjects with celiac sprue, inflammatory bowel disease, and colorectal neoplasia, showed that MPM can reliably identify a wide palette of relevant tissue substructures in the normal and the diseased human GIT. This study, together with others,^{93,95,107,120} stand as a convincing example that NLO can diagnose suspicious lesions within the mucosa, which is important for implementing appropriate therapeutic strategies in real-time, minimizing the biopsies of benign tissues, and coping with current GC diagnostics challanges.¹²³ Additional insight on the use of TPEF and SHG imaging for the study of gastric cancers is presented in the recent review of Wang et al.¹²⁴

FIG. 3.

Side-by-side comparison of images collected on unlabeled GIT tissues with non-linear optical microscopies and with conventional brightfield microscopy in the frame of traditional histopathology protocols based on H&E staining. (a) Images of the gastro-esophageal junction with a clear demarcation between the esophagus (a) and glandular stomach (b) are shown at low magnification¹²² (magnification 48× and 40× for the NLO and Brightfield Microscopy (BM) images, respectively). (b) High magnification images of the panels depicted in (a) show keratinized stratified squamous epithelium of the esophagus (a) and gastric glands of the stomach (b)¹²² (magnification 300× and 200× for the NLO and BM images, respectively). (c) Images collected on stomach tissue showing gastric glands (green in NLO) and the surrounding connective tissue (red in NLO). Insets show chief cells with basal nuclei (arrows) and large rounded parietal cells with central nuclei (arrowheads) lining a gastric gland;¹²² (magnification 300× and 200× for the NLO and BM images, respectively). (d) Images of early GC that has invaded into the submucosa. Blue arrow: tumor invasion, pink arrow: blood vessel.⁹⁸ (e) Image of full-thickness esophageal tissue with an intramural metastasis (IM) in the submucosa, which accompanies esophageal squamous cell carcinoma. Arrow: IM composed of abnormal cells positioned in the fibrous tissue. The structure of the esophageal mucosa, consisting of stratified squamous epithelium, lamina propria, and muscularis mucosae, is still maintained.¹⁰⁵ Scale bar: 200 μm; (f) and (g) images of human colorectal tissues of different histological statuses, normal colorectal mucosa (f), and colorectal cancer tissue (g); arrows in the NLO images depicted in (g) and arrowheads in (f) show the nuclei of epithelial cells. Scale bar: 50 μm.¹⁰³ The NLO images in (a)–(f), represent an overlay of TPEF and SHG signals. In (h) and (i), we show that an additional NLO modality, namely, CARS, enhances the information content in NLO images. An overlay of three NLO modalities, TPEF, SHG, and CARS is represented in the case of normal (h) and diseased (i) colon mucosa. The arrows depicted in the NLO image in (i) represent 1—lymphocytes, increase overall fluorescence, loss of crypt density; 2—crypt branching; 3—irregularities of crypts' shape; and 4—flattened epithelial layer facing the lumen. Scale bar 250 μm.¹⁰⁴ For all depicted micrographs, the source of TPEF signals are endogenous fluorophores; details can be found in the original publications. Details on color mapping: (a)–(c) blue: TPEF (550–650 nm), green: TPEF (420–490 nm), red: SHG; (d) red: TPEF (430–716 nm), green: SHG; (e) green: SHG, red: TPEF (430–710 nm); (f) and (g) blue: SHG + TPEF (387–447 nm), green: TPEF (460–500 nm), red: TPEF (601–657 nm); (h) and (i) blue: SHG, green: TPEF (426–490 nm), red: CARS (2850 cm⁻¹). (a)–(c) Reproduced with permission from Makino et al., Cancer Prevent. Res. 5(11), 1280–1290 (2012). Copyright 2012 American Association for Cancer Research.¹²² (d) Reproduced with permission from Li et al., BMC Cancer 19(1), 295 (2019).⁹⁸ Copyright 2019 Authors, licensed under a Creative Commons Attribution 3.0 Unported License. (e) Reproduced with permission from Xu et al. Biomed. Opt. Express 8(7), 3360–3368 (2017).¹⁰⁵ Copyright 2017 OSA License. (f) and (g) Reproduced with permission from Matsui et al., Sci. Rep. 7(1), 6959 (2017).¹⁰³ Copyright 2017 Authors, licensed under a Creative Commons Attribution 3.0 Unported License. (h) and (i) Reproduced with permission from Chernavskaia et al., Sci. Rep. 6, 29239 (2016). Copyright 2016 Authors, licensed under a Creative Commons Attribution 3.0 Unported License.¹⁰⁴

Even though Raman-based NLO techniques rely on more intricate illumination setups⁸⁹ compared to the NLO modalities probing fluorescent or higher-harmonic generation signals, these have reached nonetheless a sufficient level of technological maturity to be translated to in vivo clinical imaging.^91,125 Raman-based techniques provide complementary information to TPEF and SHG (see Fig. 3), via their ability to probe optical signals originating from the vibration energy of inter-atomic bonds in molecules, e.g., the C–H bond vibration, typically used for imaging lipids in tissues,⁸⁹ with lipids playing a key role in human health and diseases, being the major components of the cell membrane.¹²⁶ They can be thus highly useful to morphologically and quantitatively assess the architecture and composition of tissues, and identify subtle cues that can be specific to cancer onset, or to various cancer stages.^127–129 Although the amount of CARS and SRS applications addressing imaging of GIT components is considerably more limited compared to those reported for TPEF and SHG, the generic relevance of such techniques for imaging mammalian cells and tissues is widely acknowledged¹³⁰ and can be easily extrapolated to GIT tissues. Among the many uses of Raman-based NLO techniques for probing the tissues state, we feel noteworthy to highlight the ability of these techniques to probe inflammation-related aspects,¹³¹ with inflammation being known to promote all stages of tumorigenesis.¹³² In the same context, we find it noteworthy to highlight the complementarity of TPEF, SHG, and CARS for assessing tissue architectural aspects specific to inflammatory bowel diseases, as shown by Chernavskaia et al.,¹⁰⁴ Figs. 3(h)-3(i). While most studies addressing Raman-based NLO imaging on GIT tissues refer to endogenous contrast, recent breakthrough work⁸¹ has also demonstrated that combining such techniques with contrast agents holds as well important potential for probing disease-relevant metabolic processes, which represent a hallmark for cancer progression. We end this discussion on Raman-based NLO, by highlighting that recent work reported by Liu et al.¹³³ showed that by combining Raman-based NLO techniques with emerging Deep Learning methods for tissue diagnostics represents an exceptional avenue for gastric cancer diagnostics (Fig. 4).

FIG. 4.

Examples on the use of Deep Learning for assisting and augmenting endoscopy and non-linear optical microscopy imaging. (a) DL-based detection of polyps (highlighted in yellow) during colonoscopy. Reproduced with permission from Wang et al., Nat. Biomed. Eng. 2(10), 741 (2018). Copyright 2018 The Author(s), under exclusive licence to Springer Nature Limited.¹⁶⁵ (b) DL-based detection of gastric cancer in different stages in endoscopic images. The yellow rectangular frames in the top and bottom images were marked by the DNN as a possible lesion and to indicate the extent of a suspected gastric cancer lesion. An endoscopist manually marked the location of the cancer using a green rectangular frame. Reproduced with permission from Hirasawa et al., Gastric Cancer 21(4), 653–660 (2018). Copyright 2018 The International Gastric Cancer Association and The Japanese Gastric Cancer Association.¹⁷⁰ (c) DL-based prediction of gastric cancer in composite (falsely colored) SRS+SHG images; (i) schematics of DL predictions; (ii) examples on segmented tiles, classified by the DL model into non-cancer, differentiated cancer, and undifferentiated cancer classes; scale bars: 100 μm; (iii) (top) a typical endoscopic submucosal dissection (ESD) tissue and three test locations of intra-tumor, visual margin and ∼8 mm away from the margin, with (bottom) composite SRS-SHG images on the margin with predicted diagnostic segmentations of cancer (red) vs non-cancer (blue), and delineation of the tumor border. Scale bars: 200 μm. Reproduced with permission from Liu et al., Nat. Commun. 13(1), 4050 (2022). Copyright 2022 Authors, licensed under a Creative Commons Attribution 3.0 Unported License.¹³³ (d) Label-free multimodal NLO images of normal skin tissue. From the top to the bottom: raw NLO image (input), DL-enhanced image (network output), and ground-truth image. Scale bar: 100 μm; reproduced with permission from Shen et al., Light 11(1), 76 (2022). Copyright 2022 Authors, licensed under a Creative Commons Attribution 3.0 Unported License.²⁰⁰.

Deep Learning

Deep Learning (DL)¹³⁴ is a family of machine learning methods that employ neural networks with many layers, known as Deep Neural Networks (DNNs). These artificial intelligence methods currently lead machine learning benchmarks in computer vision, speech recognition, machine gameplay,^135,136 and various other fields. In medicine, a significant advantage of DL diagnostic approaches consists in their power to process vast amounts of data to provide a diagnostic with accuracies (at least) comparable to expert specialists.^137,138 DL frameworks for biomedicine can deal with any type of data^139,140 ranging from medical records,¹³⁷ behavioral traits,¹⁴¹ multiomics,¹⁴² or precise medical measurements, e.g., electrocardiograms¹⁴³ or medical images,¹⁴⁴ to successfully identify pathological cues or diverse health risks. This capacity is likely to completely change the current diagnostic routes and practices in the forthcoming years,^145–148 leading to the timely detection of silent but deadly pathologies and precursory signs, and hence to the implementation of more efficient therapeutic strategies.¹⁴⁹

The immense success of DL in image analysis applications is twofold: first, the accuracies achieved with DNNs are unmatched; second, DNNs are extremely effective at transfer learning (i.e., in training on one dataset and deploying on another, even across modalities¹⁵⁰). Considering this, over the past years, massive efforts have been placed on introducing DL to the realm of digital pathology,^{144,145,151–154} and recently, DL methods for automated medical image analysis provide these days results that are on par with trained experts,¹⁵⁵ or even exceed their performance in specific scenarios.¹⁵⁶ Automated dermatologist-level classification of skin cancers¹⁵⁵ with performance matching those of expert dermatologists is an illustrative example of the success of DL in biomedical image analysis. This experiment was regarded as a major breakthrough, since such methods can be used in association with mobile devices (e.g., smartphones), resulting in point-of-care diagnostics (PoC) devices, which could lead to outstanding clinical impact. Furthermore, combining the knowledge of specialists with the immense processing power of DL architectures has been shown to surpass the outputs of many traditional image analysis tasks.^157,158

Deep Learning and Gastroenterology

In the context of gastroenterology, DL methods can play multiple roles. One of the most important ones is to classify images collected on tissues with various instruments into different classes of interest, e.g., healthy, suspicious, and malignant. For example, Kather et al.¹⁵⁹ demonstrated the utility of DL in terms of predicting the clinical outcome from histological images of tissues extracted from patients presenting colorectal cancer (CRC), which was achieved by decomposing complex tissue into relevant constituent parts and exploiting their frequency. The authors showed that their method efficiently complements the current state of the art in two independent, multi-center, patient cohorts, where the prognostic score provided by their DL approach improved the survival prediction compared to the Union for International Cancer Control's staging system. Other experiments demonstrate as well various advantages offered by coupling DL with histological images of CRC.^160–163 Concerning GC diagnostics, the utility of DL has also been explored to date in diverse approaches. For example, previous studies reported DL frameworks capable of classifying GC in H&E stained histopathological whole slide images, while Garcia et al.¹⁶⁴ reported a method for the automated identification of tumor-infiltrating lymphocytes in immunohistochemistry images of GC.¹⁶⁴ The use of DL for GC/CRC diagnostics does not restrict to histopathological image analysis. Such approaches have also been successfully used in association with images collected during colonoscopy/gastroscopy sessions. For example, Wang et al.¹⁶⁵ have introduced a DL method compatible with real-time video analysis that can detect adenomatous polyps with per-image sensitivities and specificity of approximately 95%, Fig. 4(a), and Urban et al.¹⁶⁶ reported a DL framework capable of identifying polyps (in a set of 8641 colonoscopy images containing 4088 unique polyps) with a cross-validation accuracy of 96.4%. Several other strategies for polyp detection using DL are discussed in the review work of Kudo et al.¹⁶⁷ Furthermore, the recent systematic review and network meta-analysis of Spadaccini et al.¹⁶⁸ accurately highlights the advantages of DL-powered computer-aided diagnostics systems for colorectal neoplasia. The convergence between algorithms powered by DNNs and endoscopic imaging of the upper GIT¹⁶⁹ was also demonstrated in several relevant experiments,^170–172 including the landmark work of Hirasawa et al.¹⁷⁰ [Fig. 4(b)]. Two notable efforts in this direction are those of Meier et al.,¹⁷³ who successfully applied DL to identify cues collected on tissue microarrays indicating high risk of poor survival in GC images, enabling the stratification of patients in different risk groups, and Itoh et al.,¹⁷⁴ who achieved sensitivity/specificity of ∼87%, in the detection of Helicobacter pylori infection by applying a DNN based method to gastrointestinal endoscopy images collected from 139 patients. Also important to mention is the use of DNNs in ENDOANGEL, a framework developed to monitor in real-time endoscope slipping aspects, with a role in making endoscopists aware of blind spots associated with improper endoscope withdrawal.¹⁷⁵

Deep Learning and NLO

As discussed earlier in this article, virtual biopsies collected with NLO contain most essential aspects that histopathologists look for in the purpose of establishing diagnostics of GIT tissues. Hence, NLO images collected in vivo can potentially be evaluated by histopathologists for the purpose of tissue characterization and GC diagnosis and confirmation. However, diagnostic approaches in the frame in which human experts inspect tissue images still exhibit a series of drawbacks, irrespective of the modality by which these were acquired. Part of these drawbacks are caused by slow analysis speed in manual characterization assays, and by the fact that manual histopathological characterization of minute specimens can be highly subjective: for example, in previous studies, interobserver disagreements of up to >50% were reported for low-grade dysplasia.⁶⁸ While the use of DL-based methods for computer-aided diagnostics of the lower and upper GIT is gaining increasing momentum,¹⁶⁹ the use of artificial intelligence methods that can efficiently characterize NLO datasets is still in an early stage. However, a series of relevant efforts illustrate very well the possibilities derived from using in tandem NLO and DL. For example, Huttunen et al.¹⁷⁶ adapted four pre-trained DNNs: AlexNet, VGG-16, VGG-19, and GoogLeNet to classify images collected with TPEF and SHG on normal and cancerous murine ovarian tissues achieving sensitivity and specificity of 95% and 97%, respectively. What is important to highlight in this experiment is the reduced volume of the dataset, consisting of ∼200 images, while DL frameworks trained from scratch, i.e., without applying transfer learning, often require massive amounts of data, and carry a significant carbon footprint.¹⁷⁷ Their success was achieved by combining data augmentation strategies¹⁷⁸ (by which a large dataset can be simulated from a smaller one) with the superb capabilities of DNNs for transfer learning.^179,180 Such approaches allow addressing problems for which training data are expensive or difficult to collect (e.g., by using knowledge learned from natural images to aid in the classification of medical images). Later work of Huttunen et al.¹⁸¹ demonstrated the efficient use of GoogLeNet DNNs to distinguish healthy from dysplastic epithelial tissues by addressing composite TPEF/SHG images acquired on transversal sections containing the dermo-epidermal junction, a tissue region where many cancer-related processes originate. Accuracy, specificity, and sensitivity of ∼95% were demonstrated under a data augmentation scheme mixing generic augmentation approaches with a Gaussian blurring scheme, meant to help the DNNs to generalize for scale invariance. It is important to note that routine histopathology assays for epithelial cancer diagnostics typically evaluate transversally cut tissue sections, such as those evaluated in the above-mentioned study. While most NLO studies implemented in in vivo settings with tomographs fit for clinical use evaluate images collected on the xy plane,¹⁸² xz-plane images (vertical) are also possible to acquire.^183,184 The efficiency of NLO data augmentation by various Gaussian blurring schemes has also been demonstrated in the work of Anton et al.,¹⁸⁵ in the context of corneal edema diagnostics with SHG. In this work, the complementarity of Deep Learning models building on distinct architectures has also been shown by means of Gradient Classification Activation Mapping (Grad-CAM),¹⁸⁶ showing image regions that contribute most to the activation of network neurons. The authors argued that such complementarity can be exploited in majority vote schemes building on the outputs of different DL models. CAM was also used to show a correlation between image regions found most relevant by a DNN in its decision-making process, and features traditionally assessed by histopathologists, in the work of You et al.,¹⁸⁷ addressing DL classification of NLO images of breast tissues acquired with a portable intraoperative system. This work also demonstrated excellent diagnostics precision for a ResNet model: tile-level 95% area under receiver operating curve (AU-ROC); slide-level 100% (AU-ROC).¹⁸⁷ DL frameworks have also been implemented to address pathologies of the gastrointestinal system. For example, Lin et al.¹⁸⁸ used a DNN architecture based on the VGG-16 to automatically identify the differentiation stage of tissues affected by hepatocellular carcinoma, achieving a classification accuracy of over 90% with support of only 217 images. In a different experiment, Yu et al.¹⁸⁹ used a fully automated DL method based on the AlexNet architecture that resulted in AU-ROC values of up to 0.95 when addressing the problem of liver fibrosis staging. Maybe the highest profile work reported to date on the topic of GC diagnostic and NLO is the work reported by Liu et al.,¹³³ who used a U-Net DL model in combination with SRS+SHG images collected on freshly excised GC tissue, to achieve a GC diagnostic accuracy as high as 96.4%, and a level of concordance with the conventional histopathology exam of κ = 0.899. In addition to the automated diagnosis capabilities of U-Net-assisted SRS, the authors showed as well that DL can be efficiently used to assess the distribution of diagnostic groups and tumor subtypes, by semantic segmentation [Fig. 4(c)]. An additional highlight of their experiment was the successful use of DL-assisted SRS for evaluating the resection margins for endoscopic submucosa dissection [Fig. 4(c)]. This work extends previous efforts of the same group on the topic of automated diagnostics using DL-assisted NLO (SHG + SRS), where they used a ResNet34 network to differentiate between healthy and laryngeal squamous cell carcinoma, freshly excised, tissues, with 100% accuracy.¹⁹⁰ Other notable efforts on the topic of automated diagnostics by combined use of NLO and DL in the field of GIT imaging are the works of Jiang et al.,¹⁹¹ who reported a DL collagen classifier based on the Res-net 50 framework, testing NLO images collected on stage II–III colon cancer tissues, to predict disease-free survival and overall survival. Also, an important finding was the capability of the DL model to identify patients who are more likely to benefit from adjuvant chemotherapy.

In addition to experiments dealing with DL classification of NLO datasets into distinct classes of pathological relevance, DNNs have also successfully demonstrated their performance in other types of relevant biomedical image analysis applications. One example is the effort of Haft-Javaherian et al.,¹⁹² who developed DeepVess, a tool that exploits the power of DNNs to automate the segmentation of 3D in vivo NLO images of murine brain vasculature which is very laborious when performed manually; a related subject was also addressed by Damesh et al.¹⁹³ With a DL method developed to address another type of application, Liang et al.¹⁹⁴ tackled the problem of estimating the elasticity of collagenous tissues in a glutaraldehyde-treated bovine pericardium model imaged with SHG. Their method yielded a classification accuracy of 84% with very low regression errors in predicting the nonlinear anisotropic stress-strain curves, showing the potential of DL to extract physiological parameters that can be placed in correspondence with various health risks. The work of Wang et al.¹⁹⁵ presents as well an DL application relevant for the field of NLO bioimaging, consisting in a Generative Adversarial Network model capable to predict the evolution of the scar texture, allowing the subsequent extraction of potential features from SHG images at different scales. In other notable works, DL has successfully been used to: (i) compensate a low photon count,¹⁹⁶ (ii) help increase the acquisition speed^197,198 (which could be useful in the case of imaging moving tissues), or (iii) enable cross-modality imaging¹⁹⁹ (which, with respect to NLO-based diagnostics, would be important for simulating a type of image not attainable in vivo, e.g., forward SHG, based on an in vivo deployable modality, e.g., backscattered SHG). Noteworthy, a recently reported Deep Learning method was designed specifically for NLO techniques,²⁰⁰ and related bioimaging applications, providing high-speed, high-quality, and high-fidelity reconstruction for TPEF and SHG images based on dense generative adversarial networks [Fig. 4(d)]. This effort highlights the immense potential of combining Deep Learning with NLO, in order to overcome many of the bottlenecks that confined NLO bioimaging techniques to a limited number of applications and use cases over the past three decades, since they started to emerge as highly useful bio-imaging tools. Among such limitations stands also NLO data interpretation, which has long posed problems even for expert histopathologists, whose expertise is mainly associated with the analysis of stained tissues. Recent progress in virtual staining by Generative Adversarial Networks^76,77,201 came as a highly required solution to this problem, being likely to pave the way for considerably enhanced clinical penetration of NLO techniques, allowing access to NLO data in a color space and format with which histopathologists are highly familiar with.

TOWARD NEXT-GENERATION ENDOSCOPES ENABLING ENHANCED GC DIAGNOSTICS

We envision that a synergistic combination of the technologies discussed above, CEC-based biomimetic video systems, NLO, and DL could enable a next-generation endoscope, that we coin GastroAce, capable of massively improving the current state-of-the-art on detection, characterization, and confirmation of lesions in the gastrointestinal system. The main features of GastroAce, representing advances over the current state of the art, are discussed in Box 1. In the following paragraphs, we discuss some of the key challenges that lie en route to the development of such an ambitious next-generation endoscope.

Box 1: Advantages offered by GastroAce, a potential next-generation endoscope featuring biomimetic video systems, NLO and DL.

A potential configuration of GastroAce is presented in Fig. 5. Such a system would allow the entire mucosal surface of the GIT to be surveyed by using a video system comprising CEC that exploit the advantages of vision systems found in arthropods and other species, such as extended field-of-view,^44,46 very high depth of field,^44,50 great sensitivity,⁵¹ or capabilities for polarization resolved imaging.²⁰² The FOV enabled by such a video system would cover almost completely all cardinal directions by exploiting hardware coupling strategies.,⁴⁶ and panoramic image reconstruction methods.^47,203 This feature would result in significant reduction of the time required for the endoscopic session, which would be further augmented by the intrinsic depth of field capabilities of arthropod-inspired CECs, resulting in all imaged GIT areas being in focus. These features would allow the gastroenterologists to focus on identifying lesions, instead of spending time on repositioning the endoscope, thus enhancing the final output of the procedure. A second key feature of GastroAce would consist in capabilities for label-free in vivo characterization of suspicious lesions in the GIT by using an NLO endomicroscopy module inspired by recent proposals, e.g., Refs. 107 and 204, which could simultaneously collect NLO data at depths up to 1 mm, and even beyond, depending on the tissue composition, excitation wavelength and other imaging parameters.^73,74 Optical imaging with NLO could be performed based on both endogenous contrast (autofluorescence—TPEF, higher-harmonic generation—SHG, vibration energy of interatomic bonds in molecules—CARS/SRS), as in the vast majority of reported studies, but also based on exogenous agents specifically developed for NLO imaging.⁷⁸ In the design presented in Fig. 5, we propose the use of two NLO detection paths, one based on a Gradient-Index (GRIN) lens objective deployable via the functional channel of the endoscope, and a second one based on a liquid tunable lens and a rotating mirror, enabling the probing of NLO signals via a circular “window” on the endoscope body, allowing thus access to GIT lesions positioned in anatomically difficult locations, geometrically unavailable to the GRIN lens objective. The third important feature of GastroAce would stand in its capability for automated diagnostics. This feature would be provided by a DL framework capable of reliable automated identification of GIT lesions in the video streams captured by the biomimetic video system (or by an HD-WLE imaging system, which can be used in tandem)¹⁷¹ and automated interpretation^176,188 of NLO data collected at suspect tissue sites. The in-tandem use of DL for both the detection and characterization stages would significantly boost the overall efficiency of the endoscopic exam. GastroAce's DL platform would yield in real-time an automated diagnostic of the lesions imaged using the NLO module or could supply additional information helping the specialist performing an endoscopy session to validate/consolidate his/her diagnostic in computer-assisted assays.^157,158 NLO and DL could also help a surgeon to assess in real-time the tissue pathology, for accurately defining the resection margins of macroscopically invisible dysplasia or cancer.^75,102 Furthermore, besides diagnostic purposes, the DL framework could potentially be used to predict microsatellite instability, and thus the patient's response to immunotherapy.²⁰⁵ Last, but not least, the femtosecond lasers used for NLO imaging could also be used for selectively closing blood vessels via multiphoton photothermolysis,²⁰⁶ in cases when surgery/excisional biopsy is needed.

Toward compound eye video systems for GIT imaging

Two of the most important vision models that could stand behind future CEC-based video systems for endoscopy are the arthropod and the dragonfly eye, both providing powerful modes of perception. The first advantage of their hemispherical apposition design would consist of exceptionally wide FOV, without off-axis aberrations. This can allow GIT lesions at different angular positions to be imaged with comparable clarity, without anomalous blurring or aberrations. Equivalent imaging modes are difficult or impossible to obtain using planar detector technologies even with sophisticated fish-eye lenses, spherical mirrors, or other specialized optics. The second key attribute of such CECs is their nearly infinite depth-of-field that results from the short focal length of each microlens in the array and the nature of image formation. This attribute translates to the fact that when an object moves away from the camera, the size of the image decreases but remains in focus.⁴⁴ Such features would play a key role in reducing the time required for the endoscopic procedure, helping the specialist performing it to avoid time on focusing tasks aimed to deblur the images of features of interest or to readjust the position of the distal tip to image regions not contained in the FOV. Simultaneously, other valuable characteristics could be added to potential CEC-based video systems for endoscopy, such as polarization-resolved imaging, whose importance for the early diagnosis of flat cancerous lesions has been previously discussed.²⁰⁷ Such imaging approaches exploit the fact that the polarization state of light encodes various intrinsic properties of the visualized media or objects,⁴⁰ such as three-dimensional morphology or structural composition. Similar features are present in many animal species that exploit them for navigation, communication, or others, and recent works^202,208 have demonstrated artificial systems possessing such properties. Facile exploration of difficult anatomic sites in the GIT with wide-FOV CEC featuring various other biomimetic characteristics such as those mentioned above would offer important advantages for endoscopy, with direct implications for the procedure duration and the level of extracted information. However, a series of technological challenges still need to be addressed before we can witness such CECs in clinical applications for GIT imaging. The resolution and optical efficiency of current CECs are still not at the level required for commercial products. However, the current volume of efforts in the field is likely to generate soon important ideas for cost-effective mass production of such imaging systems with enhanced optical efficiency, capable of producing high-resolution images, and available in small form packages. For instance, Lee et al.²⁰⁹ has proposed a computational CEC (COMPU-EYE) solution that increases acceptance angles and uses a modern digital signal processing technique. The proposed COMPU-EYE yielded a fourfold improvement in spatial resolution, and more recent works showed that COMPU-EYE can also be modified to estimate depth, which is useful for assessing the distance between distinct objects contained in the FOV.²¹⁰ Recent progress on virtual super-resolution via Generative Adversarial Networks^211,212 represents as well a great promise to overcome resolution challenges related to the technical architectures of current biomimetic CECs. Other ways of overcoming the resolution challenge could rely on the integration of discrete components to replicate biomimetic CE models,⁴⁷ as presented in Fig. 2(d). Such approaches could even combine distinct biomimetic notions, for example, compound eye architectures (e.g., inspired from insect vision) with other biomimetic vision concepts.^61,66

A remaining challenge in translating emerging compound eye cameras to the field of endoscopy is miniaturization. While a small footprint is a mandatory prerequisite for using CECs in endoscopy applications, in many configurations, the resolution depends on the number, and diversity, of contained optical elements. The emerging field of micro-optics fabrication with femtosecond multiphoton direct laser writing²¹³ is highly relevant to mention in this regard, given the superb capabilities of such approaches to potentially enable the fabrication of ultracompact compound lens systems. Not only that these technologies hold significant potential for miniaturizing CECs deployable on the body of a conventional endoscope, but they can also be used to fabricate miniaturized endoscopes, sized <1 mm, realized by applying various fs laser fabricated micro-optical elements directly on the optical fiber tip.^214,215 These constructs could eventually be deployed via the functional channel of the conventional endoscope to access and probe anatomically difficult regions, such as hardly accessible tissue areas positioned between the stomach folds. This field of fs laser fabricated micro-optics is rapidly developing, and recent progress in tip-enhanced fs modification of the refractive index of optical materials, which allows the processing of regions sized well beyond the diffraction limit,²¹⁶ promises to consistently push forward the frontiers of this field in the coming years.

In addition to their technical capabilities, another challenge with respect to using CEC cameras in the GIT refers to maintaining their integrity while exposed to the corresponding harsh environment. For this, the advent of protective coatings that defend them from the acidic environment of the gastric fluids, and the agents used for system cleaning/sterilization is of utmost importance. Such coatings should possess properties such as high transparency, in order not to affect the signal intensity and consequently image contrast and brightness. Low surface roughness and low coefficient of friction should also be considered, for avoiding tissue damage, and finally, high hydrophobicity would be important for keeping the lenses clean during the procedure, and protecting them from the acidic liquid environments in the GIT. Intelligent bioinspired solutions²¹⁷ could represent a solid solution to address some of these problems.

Toward in vivo characterization of gastrointestinal tissues with NLO

Commercial NLO systems in the form of clinically-certified multiphoton tomographs have already been used in various hospital settings throughout Europe, Australia, China, Russia, and the US, to investigate diverse dermatological pathologies and disorders such as skin cancer or inflammation,^85,218,219 and even brain tumors.^85,220 NLO endomicroscopy, which could enable in vivo GC diagnostics as discussed earlier, has been demonstrated in several important studies that report ingenious configurations, e.g., Refs. 107, 204, and 221–225, and has recently been demonstrated in vivo inside the gastrointestinal system in an animal model,¹⁰⁷ but not yet in the human body. While the problem of miniaturization seems to be well addressed at the time being, an important challenge that still exists with respect to reliably collecting NLO images inside the GIT is represented by the pulsatory motion inherent to soft tissues inside the human body which makes the collection of artifact-free data difficult. If the tissue region that is being characterized changes position with respect to the objective during image acquisition, the collected image may either contain signal inhomogeneities or be entirely compromised. Such issues can be partially addressed with image processing techniques,^226,227 but more robust alternatives would be of great benefit. We hypothesize that such alternative solutions could consist in applying surface coatings to the optical element that collects the NLO signals (e.g., the GRIN objective), with properties custom designed to allow their temporary coupling to the tissue regions that need to be imaged. With such coupling, when the tissue moves during image acquisition, the optical element would move together in the same direction, while maintaining the focus. From a chemical engineering point of view, such coatings could potentially be achieved using materials that rearrange between hydrophilic/hydrophobic states.^228–230 Among the potential methods for controlled switching between such states, we consider that those based on excitation with laser pulses of specific properties in terms of intensity, wavelength, or pulse duration could be worth pursuing. Such strategies would allow for the hydrophilic state of the coating to be turned on via laser beam triggering before tissue imaging commences, and the objective coating to be similarly switched to a hydrophobic state when the imaging is completed, and the objective can disengage from the tissue site. Key attention in the design and synthesis of such coatings capable of photoactivated hydrophilic/hydrophobic switching should be given so that they do not interfere with the properties of the optical signals collected from the imaged region. Furthermore, as with any microscopy technique, another important challenge of NLO endomicroscopy is resolution. While the resolution of conventional benchtop or endomicroscopy NLO systems used to date for GIT imaging is limited by the light diffraction phenomenon to approximately half the excitation wavelength, superior resolutions would be useful for investigating subcellular details sized below this limit. Such resolutions could be achieved by exploiting recent imaging concepts for super-resolved laser scanning microscopy, such as the image scanning,^231,232 or re-scan²³³ strategies. With respect to the latter, a resolution advantage of 1.4 for re-scan SHG and 1.5 for re-scan TPEF over diffraction-limited SHG and TPEF, respectively, was recently demonstrated.²³⁴

An important advantage of nonlinear optical microscopies is their penetration depth, given the use of near-infrared wavelengths which are less scattered in tissues compared to visible radiation used in other bioimaging techniques, such as conventional confocal microscopy. However, the potential of NLO techniques is still biased by various beam aberrations caused by the propagation of a numerical aperture beam through the different epithelial layers. A potential workaround has been proposed in the recent work of Marini et al.,²³⁵ where instead of imaging via a high NA objective, tissue excitation is performed by microlenses, fabricated using a two-photon laser processing methodology, inspired from previous works^214,215 (discussed earlier in this article, also in the context of endoscopic CEC perspectives). A key application of such microlenses could stand in their direct coupling to micro scaffolds, implanted in vivo, serving as “windows” to specific tissue regions. Such developments would be highly important for better understanding diseases progression or therapeutic processes, by providing optical access to deep tissue layers.

Toward automated GC diagnostic with Deep Learning augmented NLO

To enable intelligent and automated diagnostics, next-generation endoscopes could exploit advanced artificial intelligence methods based on DL, capable to detect lesions in the video streams of standard HD cameras, e.g., Refs. 165–167 (or of future endoscopic CEC), and also capable of analyzing NLO datasets in real-time. This would enable automated identification of suspicious lesions in the GIT, followed by reliable tissue characterization, which is required for GC diagnostic (or GC risk assessment). A fast route toward achieving this second purpose can consist in using data collected in ex vivo assays to train the DNNs so that these can later perform classification using the NLO imaging modalities available in vivo (e.g., TPEF, SHG, FLIM or others). The utility of such potential approaches derives from the fact that DNNs are most effective when applied on large training sets, but the availability of such large NLO datasets collected in vivo from patients is difficult to expect in the forthcoming years. Therefore, to develop such a DL framework for automated GC diagnostics based on in vivo NLO imaging of GIT tissues, a route worthy to pursue could rely on the use of DNNs pre-trained with large scale NLO datasets collected with benchtop systems on fixed, or freshly excised, tissues, in relevant pathological states. An additional route for circumventing the problem of training data availability could consist in using transfer learning to exploit knowledge learned by the DNNs from images collected with conventional microscopes on fixed GC tissue fragments processed according to traditional histological protocols, e.g., Hematoxylin and Eosin or Masson's trichrome staining. Such approaches could potentially be merged, by using DNNs that exploit the information collected by using a wide variety of techniques and methodologies, capable to extrapolate what they learn from images collected on fixed or ex vivo tissue samples to the case of in vivo GC diagnostics, Fig. 6. The architecture of such DL methods developed for intelligent diagnostics should be flexible so that once NLO datasets collected in vivo will become available they can be easily included in the learning cores of the original DNNs, for their diagnostic capabilities to continuously evolve over time. In addition to spatial intensity information (images), the DL framework for automated GC diagnostics could also exploit other specific NLO information such as fluorescence spectra or SHG polarization signatures.¹¹⁶ Distinct DL methods included in the framework could independently address such distinct information categories, each providing its own diagnostic forecast. Further on, the outputs of such independent DL experiments could potentially be merged in majority vote schemes or other multiple expert decision strategies for enhanced diagnostic accuracy.

FIG. 6.

A potential DL framework for classification of images collected in vivo with non-linear optical microscopies on gastrointestinal tissues using transfer learning. In this architecture, each imaging modality has unique low-level feature extracting layers, and shared mid- and high-level layers; this modular structure of the DNNs would support both single-modal and multi-modal diagnosis. Such architectures could enable the accurate classification of datasets collected with any combination of NLO in vivo imaging modalities (e.g., TPEF, SHG, and FLIM, depicted here), based on a priori training with datasets collected with various other imaging modalities in the frame of in vivo, ex vivo, or in histology assays.

A key bottleneck in the further development of DL methods for NLO bioimaging stands in the limited availability of public datasets, with highly useful data usually being confined to the groups possessing expensive NLO instrumentation. Recent initiatives to publish curated NLO datasets^236–238 will hopefully inspire future similar efforts, which are highly necessary for expanding the field of DL-assisted NLO diagnostics.

Conclusions

A potential next-generation endoscopic system combining biomimetic CEC video systems, NLO and DL, could offer valuable new possibilities for GC prevention, diagnosis, and prognosis, leading toward more timely intervention and more efficient therapeutic approaches. Combining these technologies could also significantly lower the costs associated with a patient's gastrointestinal screening and diagnosis as a result of faster processing imaging procedures and lower requirements for highly specialized staff. All these could play an important role concerning the sustainability of healthcare systems across the globe in the forthcoming context of predicted population aging.

AUTHOR DECLARATIONS

Conflict of Interest

Professor Karsten Köning is co-founder and CEO of JenLab Gmbh, a German company that pioneered clinical multiphoton tomography. Professor Lior Wolf is CEO of Mentee Robotics. Dr. Paolo Bianchini is involved in Genoa Instruments, an Italian company focusing on super-resolved optical imaging. S. G. Stanciu, Y. M. Song, C. A. Charitidis, and M. Goetz do not have competing interest.

Author Contributions

Stefan G. Stanciu, Karsten König, Young Min Song, Lior Wolf, Costas A. Charitidis, Paolo Bianchini, and Martin Goetz contributed equally to this work.

Stefan G. Stanciu: Conceptualization (equal); Formal analysis (equal); Funding acquisition (equal); Methodology (equal); Writing – original draft (equal); Writing – review & editing (equal). Karsten König: Conceptualization (equal); Formal analysis (equal); Methodology (equal); Writing – original draft (supporting); Writing – review & editing (equal). Young Min Song: Conceptualization (equal); Formal analysis (equal); Methodology (equal); Writing – original draft (supporting); Writing – review & editing (equal). Lior Wolf: Conceptualization (equal); Formal analysis (equal); Methodology (equal); Writing – original draft (equal); Writing – review & editing (equal). Constantinos Charitidis: Conceptualization (equal); Formal analysis (equal); Methodology (equal); Writing – original draft (equal); Writing – review & editing (equal). Paolo Bianchini: Conceptualization (equal); Formal analysis (equal); Methodology (equal); Writing – original draft (equal); Writing – review & editing (equal). Martin Goetz: Conceptualization (equal); Formal analysis (equal); Methodology (equal); Writing – original draft (equal); Writing – review & editing (equal).

DATA AVAILABILITY

Data sharing is not applicable to this article as no new data were created or analyzed in this study.