The application of artificial intelligence (AI) in colonoscopy is attracting a growing amount of attention because it has the potential to improve the quality of colonoscopy.1, 2 The main focuses of research in this field comprise automated polyp detection3, 4 and characterization5, 6 (ie, pathologic prediction), which may respectively contribute to a higher rate of adenoma detection and a reduction of the costs related to unnecessary polypectomy. However, there has not yet been any report of technology capable of simultaneous polyp detection and characterization, which is the optimal situation for fully automated colonoscopic observation.
Description of technology
We developed a novel AI system that enables automated polyp detection followed by immediate polyp characterization in a real-time fashion (Fig. 1) by use of an endocytoscope (CF-H290ECI; Olympus Corp, Tokyo, Japan). The endocytoscope has the special capability of 520-fold magnification power in addition to normal endoscope functions (Figs. 2 and 3).
Figure 1.
Real-time computer-aided diagnostic system designed for use with endocytoscopes (EB-00 and EB-01 prototypes from Cybernet System Corp, Tokyo, Japan).
Figure 2.
The endocytoscope (CF-H290ECI; Olympus Corp, Tokyo, Japan) used with the novel artificial intelligence system.
Figure 3.
The tip of the endocytoscope. It has the function of a contact microscopy system with 520-fold magnification power, and it also works as an ordinary high-definition colonoscope.
The software used for automated polyp detection (EB-00 prototype; Cybernet System Corp, Tokyo, Japan) was designed to analyze the nonmagnified white-light, high-definition, endoscopic video in real time based on a convolutional 3-dimensional network, which is a type of deep learning methodology.7 The AI system uses this algorithm to automatically detect polyps, and it indicates their presence by outputting a yellow color in the 4 corners of the displayed image and emitting a warning sound (Fig. 4). The automated detection system is able to recognize the presence of the polyps even if there are multiple lesions in the monitor. A pilot study showed that the AI system detected 94% of the test polyps (47 of 50), with a false-positive detection rate of 60% (51 of 85).7
Figure 4.
The artificial intelligence system indicates the automatic detection of a flat adenoma (3 mm, 0-IIa) in the ascending colon by outputting a yellow color in the 4 corners of the display screen and emitting a warning sound. The probability of the presence of the polyp is displayed in the upper-left corner of the endoscopic image.
The software used for automated polyp characterization (EB-01 prototype; Cybernet System Corp) is designed to predict the pathologic features of the detected polyp as either neoplastic or non-neoplastic after acquisition of the in vivo macroscopic image obtained by the endocytoscope. A probability of the predicted pathologic features is also outputted. The algorithm is based on texture analysis powered by a support vector machine, which is a type of hand-crafted analytic method.8 The AI system is able to analyze microvascular images of the polyp obtained by endocytoscopic observation (at 520-fold magnification) with the narrow-band imaging (NBI) mode (Olympus Corp). A large-scale prospective study8 showed that this software has negative predictive values for identifying diminutive rectosigmoid adenomas of 95.2% with the NBI mode, which exceeds the threshold required for the “diagnose-and-leave hyperplastic polyps” strategy.9
The newly developed, comprehensive AI system combines the abovementioned 2 algorithms and is directly connected to a standard endoscopy unit (Evis Lucera Elite; CV-290, Olympus Corp) that allows fully automated interpretation of the polyps encountered during ongoing colonoscopy. The latencies of the detection and characterization system are just 0.1 to 0.5 seconds after the polyp is detected, and 0.4 seconds after the capture button of the endocytoscope is pushed, which enables endoscopists to make real-time decisions regarding the necessity of polypectomy. However, the AI system has several limitations: it is not designed to detect dysplasia in patients with inflammatory bowel disease, and it is unable to diagnose sessile serrated adenomas/polyps, which might be misdiagnosed as non-neoplastic.
Regarding the clinical implementation of this technology, we are still evaluating the automated polyp detection system in an experimental setting, but we have submitted the regulatory approval application for the automated polyp characterization system in Japan. The endocytoscope is already commercially available.
Case presentations
These procedures were prospectively done after approval was obtained from the local ethical committee and informed consent was secured from the patients.
In patient 1, a diminutive adenoma (3 mm, 0-IIa) in the ascending colon was automatically detected by the AI system (Fig. 4). After detection of the polyp, the endoscopist obtained a microvascular image by contacting the polyp with the tip of the endocytoscope using the NBI mode. The endoscopist subsequently pushed the capture button of the endocytoscope, and the second algorithm was immediately activated and predicted the polyp pathologic features as neoplastic (Fig. 5) (Video 1, available online at www.VideoGIE.org).
Figure 5.
The artificial intelligence system automatically outputs the predicted pathologic features of the detected polyp as neoplastic with a probability of 90% immediately after the endoscopist pushes the capture button of the endocytoscope. The polyp shown in this figure is the same one described in Figure 4.
In patient 2, a diminutive hyperplastic polyp (3 mm, 0-IIa) in the rectum was automatically detected and characterized as non-neoplastic by the AI system (Figs. 6 and 7), by use of a technique similar to that described for patient 1.
Figure 6.
The artificial intelligence system indicates the automatic detection of a flat hyperplastic polyp (3 mm, 0-IIa) in the rectum by outputting a yellow color in the 4 corners of the display screen and emitting a warning sound.
Figure 7.
The artificial intelligence system automatically outputs the predicted pathologic features of the detected polyp as non-neoplastic with a probability of 89% immediately after the endoscopist pushes the capture button of the endocytoscope. The polyp shown in this figure is the same one described in Figure 6.
An additional 4 patients are presented in the video, as detailed in the legend for Video 1.
Disclosure
Drs Y. Mori, Kudo, and Misawa are recipients of speaking honoraria from Olympus and inventors’ premiums for the patent from Showa Univ. Dr K. Mori is the recipient of research funding from Cybernet.
Acknowledgments
We thank Dr Hayato Itoh, PhD, and Dr Masahiro Oda, PhD, of the Graduate School of Informatics, Nagoya University, for their invaluable contributions to the present study. We thank Dr Kelly Zammit, BVSc, of the Edanz Group (www.edanzediting.com/ac) for editing a draft of this manuscript. The present study was supported by the Project on Utilizing High-Definition Medical Imaging Data up to 8K Quality from the Japan Agency for Medical Research and Development (AMED) and Japan Society for the Promotion of Science (JSPS) Grant-in-Aid for Scientific for Research (KAKENHI) grant number 17H05305.
Supplementary data
One diminutive adenoma (patient 1) and 1 diminutive hyperplastic polyp (patient 2) were assessed in real time with the use of artificial intelligence (AI). Patient 1: a diminutive adenoma (3 mm, 0-IIa) in the ascending colon was automatically detected by the AI system. After the polyp was detected, the endoscopist obtained a microvascular image by contacting the polyp with the tip of the endocytoscope using the narrow-band imaging mode. The endoscopist then obtained the immediate prediction of the polyp pathology as neoplastic from the AI system by simply pushing the capture button of the endocytoscope. The time required for both detection and characterization of the polyp was approximately 30 seconds. Patient 2: a diminutive hyperplastic polyp (3 mm, 0-IIa) in the rectum was automatically detected and characterized as nonneoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 3: a diminutive hyperplastic polyp (3 mm, 0-IIa) in the sigmoid colon was automatically detected and characterized as nonneoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 4: a diminutive adenoma (4 mm, 0-IIa) in the sigmoid colon was automatically detected and characterized as neoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 5: a diminutive hyperplastic polyp (4 mm, 0-IIa) in the sigmoid colon was automatically detected and characterized as nonneoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 6: a small adenoma (8 mm, 0-Is) in the sigmoid colon was automatically detected and characterized as neoplastic by the AI system, by use of a technique similar to that described for patient 1.
References
- 1.Byrne M.F., Shahidi N., Rex D.K. Will computer-aided detection and diagnosis revolutionize colonoscopy? Gastroenterology. 2017;153:1460–1464e1. doi: 10.1053/j.gastro.2017.10.026. [DOI] [PubMed] [Google Scholar]
- 2.Mori Y., Kudo S.E., Berzin T.M. Computer-aided diagnosis for colonoscopy. Endoscopy. 2017;49:813–819. doi: 10.1055/s-0043-109430. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Fernandez-Esparrach G., Bernal J., Lopez-Ceron M. Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps. Endoscopy. 2016;48:837–842. doi: 10.1055/s-0042-108434. [DOI] [PubMed] [Google Scholar]
- 4.Urban G., Tripathi P., Alkayali T. Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy. Gastroenterology. 2018;155:1069–1078. doi: 10.1053/j.gastro.2018.06.037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Byrne M.F., Chapados N., Soudan F. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut. Epub 2017 Oct 24 doi: 10.1136/gutjnl-2017-314547. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Chen P.J., Lin M.C., Lai M.J. Accurate classification of diminutive colorectal polyps using computer-aided analysis. Gastroenterology. 2018;154:568–575. doi: 10.1053/j.gastro.2017.10.010. [DOI] [PubMed] [Google Scholar]
- 7.Misawa M., Kudo S.E., Mori Y. Artificial intelligence-assisted polyp detection for colonoscopy: initial experience. Gastroenterology. 2018;154:2027–2029 e3. doi: 10.1053/j.gastro.2018.04.003. [DOI] [PubMed] [Google Scholar]
- 8.Mori Y., Kudo S.E., Misawa M. Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study. Ann Intern Med. 2018;169:357–366. doi: 10.7326/M18-0249. [DOI] [PubMed] [Google Scholar]
- 9.Rex D.K., Kahi C., O'Brien M. The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on real-time endoscopic assessment of the histology of diminutive colorectal polyps. Gastrointest Endosc. 2011;73:419–422. doi: 10.1016/j.gie.2011.01.023. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
One diminutive adenoma (patient 1) and 1 diminutive hyperplastic polyp (patient 2) were assessed in real time with the use of artificial intelligence (AI). Patient 1: a diminutive adenoma (3 mm, 0-IIa) in the ascending colon was automatically detected by the AI system. After the polyp was detected, the endoscopist obtained a microvascular image by contacting the polyp with the tip of the endocytoscope using the narrow-band imaging mode. The endoscopist then obtained the immediate prediction of the polyp pathology as neoplastic from the AI system by simply pushing the capture button of the endocytoscope. The time required for both detection and characterization of the polyp was approximately 30 seconds. Patient 2: a diminutive hyperplastic polyp (3 mm, 0-IIa) in the rectum was automatically detected and characterized as nonneoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 3: a diminutive hyperplastic polyp (3 mm, 0-IIa) in the sigmoid colon was automatically detected and characterized as nonneoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 4: a diminutive adenoma (4 mm, 0-IIa) in the sigmoid colon was automatically detected and characterized as neoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 5: a diminutive hyperplastic polyp (4 mm, 0-IIa) in the sigmoid colon was automatically detected and characterized as nonneoplastic by the AI system, by use of a technique similar to that described for patient 1. Patient 6: a small adenoma (8 mm, 0-Is) in the sigmoid colon was automatically detected and characterized as neoplastic by the AI system, by use of a technique similar to that described for patient 1.