Artificial Intelligence in Endoscopy

Abstract

Background

Owing to their rapid development, artificial intelligence (AI) technologies offer a great promise for gastroenterology practice and research. At present, AI-guided image interpretation has already been used with success for endoscopic detection of early malignant lesions. Nonetheless, there are complex challenges and possible shortcomings that must be considered before full implementation can be realized.

Summary

In this review, the current status of AI in endoscopy is summarized. Future perspectives and open questions for further studies are stressed.

Key Messages

The usage of AI algorithms for polyp detection in screening colonoscopy results in a significant increase in the adenoma detection rate, mainly attributed to the identification of diminutive polyps. Computer-aided characterization of colorectal polyps accompanies the detection, but further studies are needed to evaluate the clinical benefit. In contrast to colonoscopy, usage of AI in gastroscopy is currently rather limited. Regarding other fields of endoscopic imaging, capsule endoscopy is the ideal imaging platform for AI, due to the potential of saving time in the video analysis.

Introduction

The introduction of artificial intelligence (AI) using deep learning methods revolutionized the use of algorithms in the medical field. Deep learning neural networks are currently trained using a large amount of manually annotated datasets [1]. These datasets can be images like the ones that are taken during routine endoscopy or videos [2, 3]. However, not only images can be used to train AI. Sounds and even laboratory test results from medical health records can be used to generate an AI that intelligently guides decision making in endoscopy [4, 5]. Training is performed using powerful computers with multiple graphic cards specialized on the creation of neural networks. These are usually located in a server facility. Training and optimizing a neural network take time. A fully trained unmodifiable neuronal network is then usually applied using a computer with less computational resources, similar to the one that is located in the endoscopy room. One advantage of these AIs is that their application can be performed in real time due to the fact that images are mostly processed at a lower resolution than the image seen on the examination screen and thus require less computational power [6]. Efforts are made not only to develop neural networks within server facilities but also to apply the real-time application using cloud-based solutions [7].

AI in Colonoscopy

Prevention of colorectal cancer is one of the main aims of endoscopy [8]. Technological advances like high-definition images and virtual chromoendoscopy help identify and characterize lesions [9]. Adenoma detection rate (ADR) is one of the key performance measures for screening colonoscopy and is correlated to death due to colorectal cancer [10]. Guidelines recommend a clean bowel [11] and taking enough time for withdrawal to improve ADR [12]. Due to the fact that the diagnostic process depends only on the examiner looking for polyps on the endoscopic image, AI as a second pair of eyes like the ones of an observing nurse is suited to support the examiner [13, 14]. Algorithms to detect polyps in screening colonoscopy (computer-assisted detection, CADe) have been developed and tested in multiple prospectively randomized clinical trials with ADR as the primary endpoint [15, 16, 17, 18, 19, 20, 21] (Table 1). Polyps are usually highlighted with a colored square in the image (Fig. 1). Taken together, all of those studies present a significant increase in ADR [22]. Interestingly, the increase in ADR is mainly attributed to the identification of diminutive polyps. Yet the role of such lesions in the development of colorectal cancer is not clear [23]. Still, the removal of such lesions will increase costs and surveillance burden. However, there is a connection between identification of nonadvanced adenomas and the finding of advanced adenoma in the subsequent colonoscopy [24]. Another limiting factor might be that nearly all randomized studies are monocentric and have been performed in China. CADe algorithms have several other limitations. One is that they depend on a sufficiently inflated colonic lumen. Mucosal folds often lead to false-positive detections. Additionally, humans interpret the information presented on the screen in context. They easily recognize that a polyp moved from one side of the screen to another. CADe algorithms usually do not link the visible polyp on one image to the same one visible a second ago. They interpret each image for itself. Thus, changing the position of the endoscope often leads to discontinuation of polyp detection, although it is still visible on the screen. A solution for this problem might be linking of detections across image frames during a time, like the method described by Sabater et al. [25] called “Robust and efficient post-processing for Video Object Detection” (REPP).

Table 1.

Overview of the randomized controlled trials for the usage of CADe in colonoscopy using ADR as a primary outcome

Author	Year	Detection system	ADR percentage point increase	Country	Centers, n
Wang et al. [21]	2019	EndoScreener	8.8	China	1
Liu et al. [20]	2020	CADe	15.2	China	1
Wang et al. [16]	2020	EndoScreener	6	China	1
Repici et al. [18]	2020	Gl Genius	14.4	ltaly	3
Su et al. [17]	2020	ACQS	12.4	China	1
Gong et al. [15]	2020	ENDOANGEL	8	China	1
Liu et al. [19]	2020	EndoScreener	8.1	China	1

Listed increase in ADR percentage points is calculated by subtracting the ADR of examinations without CADe from the examinations with CADe support. ADR, adenoma detection rate; CADe, computer-assisted detection.

Fig. 1.

Comparison of a commercially available CADe system (left image, GI Genius, Medtronic) with a self-made CADe prototype system (right image, ENDOMIND). Polyp marked by a colored bounding box. CADe, computer-assisted detection.

Another important field of AI in colonoscopy is the computer-aided characterization (CADx) of polyps. High-definition endoscopy and virtual chromoendoscopy with magnification of mucosal areas are opening up new perspectives on surface details for examiners [26, 27]. CADx might help interpret these details and establish a diagnosis [28, 29]. Still, studies analyzing the effect of CAD systems do not have the same quality including a randomized prospective design as the ones examining the effect of CADe systems [30]. Additionally, vendors are grading their devices up by adding CADx to the already existing CADe modes [31]. Thus, examiners will start using CADx in practice and gather experience. Mainly one detail is limiting the usability of CADx systems. This detail is related to the data used to train CADx systems. Usually training data are composed of a correlation of the endoscopic image with a pathology report. Yet pathologists often do not agree on the differentiation of hyperplastic polyps and sessile serrated lesions [32, 33]. Thus, CADx interpretation of the image might be biased. Other limitations include the application of CADx to altered polyps, like granulation tissue next to a mucosal scar or dysplastic lesions with an inflamed mucosa surrounding the lesion as in ulcerative colitis. Another example is the submucosal injection of polyps using indigo carmine for staining. Usually bigger adenomatous polyps are removed this way. Hyperplastic polyps are removed with the cold snare technique without a submucosal injection. CADx trained with such images will tend to interpret bluish lesions as adenomatous, as one already commercially available CADx is presenting in clinical practice.

Last but not least, AI in colonoscopy may also include the measurement of key performance measures. One system provides a real-time evaluation method for bowel preparation [34]. Another system estimates the withdrawal speed in order to ensure that the examiner takes enough time to examine the colon segments [15]. Finally, reconstruction of the colonic surface might help identify areas that are not inspected and examine them for hidden polyps [35, 36].

AI in Gastroscopy

In contrast to colonoscopy, the use of AI in gastroscopy is currently rather limited. So far, there is no AI-based product commercially available. Nevertheless, promising data have been reported, and it can be expected that several products developed for upper GI may be available soon.

One of the greatest challenges in gastrointestinal endoscopy is to detect and delineate Barrett's neoplasia. Deep learning algorithms were first used by Ebigbo et al. [37] and de Groof et al. [38]. In the latter study, the CAD system had a higher accuracy for the detection of dysplastic Barrett's than general endoscopists. This highlights the aspect that when an AI is trained by experienced endoscopists, standard endoscopists take advantage of expert knowledge delivered by such a CAD system. A recent meta-analysis summarizing all kinds of upper GI-neoplasia confirms these data [39]. Besides the aid to detect lesions, there are CAD systems that help the examiner estimate the invasion depth of cancer lesions in the upper GI tract during endoscopy [6, 40].

Apart from detection and differentiation of neoplasia, further prototypic applications of AI in gastroscopy were to establish deep learning models for endoscopic classification of gastroesophageal reflux disease [41] and detection of H. pylori-associated gastritis [42].

Last but not least, similar to colonoscopy, measurement of key performance measures for gastroscopy is a potential target for AI. The quality of gastroscopy is directly related to the documented examination of certain anatomical landmarks starting from the hypopharynx to the descending duodenum. Since AI may control whether landmarks have been adequately visualized during an ongoing endoscopy [43], it is an ideal tool to assist the examiner.

AI in Other Fields of Endoscopic Imaging

In general, all imaging technologies may be used for AI. In endoscopy, these include endoscopic ultrasound (EUS) images, fluoroscopical images acquired during ERCP, and images from capsule endoscopy.

Clinical indications that have been observed with respect to EUS include assessment of quality measures for EUS imaging of the pancreas [44], diagnosis and differentiation of subepithelial tumors [45], and detection of pancreatic ductal adenocarcinoma [46]. For the latter 2 indications, an accuracy of 90% has been reported in these studies.

For ERCP, data are sparse. Here, apart from the diagnosis of diseases affecting the biliary system, an interesting approach of AI can be regarded as the adoption of a scoring system for ERCP difficulty, thereby guiding treatment [47]. This means that based on the ERCP image (and potentially other anamnestic or clinical data), AI may suggest whether a procedure is likely to bear an increased risk and experts should perform the examination rather than novices. Capsule endoscopy can be regarded as an ideal imaging platform for AI. Taking under consideration the time that is needed to evaluate capsule videos, adoption of AI for diagnostic assistance would be very attractive. It has been demonstrated that AI-guided detection of blood is almost 100% accurate [48]. Based on these excellent results and the time-consuming reading of images by capsule endoscopy, it can be expected that AI will soon be incorporated into clinical practice.

Current Problems and Perspectives

As mentioned, there are several categories of AI systems to be potentially used in endoscopy. The spectrum is large with many potentially useful clinical applications, such as improved detection of lesions, differentiation of lesions based on their mucosal or vascular pattern, risk stratification before/during therapy, and assessment of key performance measures. Overall, the main advantages of AI-assisted endoscopy should be regarded to reduce the workload and lead to greater accuracy. Hence, in the near future, AI will become more and more relevant in clinical practice. In addition, it might also be expected that AI will become relevant not only for endoscopic detection but also for more accurate AI-guided endoscopic resection.

Nonetheless, there are also several challenges yet to be addressed. AI in endoscopy has not been tested for large-scale clinical applications. Data that have been presented are obtained mainly from expert centers. For CADe systems applied during ongoing examinations, there is still the problem of false-positive detections that has been ignored in several studies. It remains unclear whether any activation should be regarded as a positive finding or whether only a constant signal over several frames/seconds should be regarded as a true positive. There is therefore a need to assess AI-based systems frame by frame and examine how robust a signal should be before it can be regarded a true positive. Furthermore, external validation through randomized trials should be regarded mandatory to judge on the true value of various commercially available systems.

Finally, AI should be regarded much more rather than solely image analysis. Other relevant data, such as anamnestic data or laboratory values, should be integrated in such deep learning systems, thereby helping the examiner diagnose and treat individual patients with maximum accuracy and minimal invasiveness.

Conflict of Interest Statement

A.M. received honorary as a consultant for Olympus and Ovesco.

Funding Sources

The author AH receives public funding from the state government of Baden-Württemberg, Germany (Funding cluster “Forum Gesundheitsstandort Baden-Württemberg”) to research and develop artificial intelligence applications for polyp detection in screening colonoscopy. The sponsor had no influence on drafting the manuscript.

Author Contributions

Conception and design, drafting of the manuscript, critical revision of the manuscript for important intellectual content, and final approval of the manuscript were done by both authors.