Artificial intelligence in colonoscopy

Abstract

Colorectal cancer remains a leading cause of morbidity and mortality in the United States. Advances in artificial intelligence (AI), specifically computer aided detection and computer-aided diagnosis offer promising methods of increasing adenoma detection rates with the goal of removing more pre-cancerous polyps. Conversely, these methods also may allow for smaller non-cancerous lesions to be diagnosed in vivo and left in place, decreasing the risks that come with unnecessary polypectomies. This review will provide an overview of current advances in the use of AI in colonoscopy to aid in polyp detection and characterization as well as areas of developing research.

Core Tip: The rapidly evolving field of artificial intelligence (AI) has found many applications in the field of colonoscopy. Specifically, we describe the technologies that have been developed to detect and characterize colonic polyps with the goal of real-time analysis as well as minimizing the risks of avoidable polypectomies. Additionally, we discuss some of the future directions of AI in this area including advancements in robotic technology.

INTRODUCTION

Colorectal cancer (CRC) is the third most common cancer and the second leading cause of cancer related death in men and women in the United States. The incidence of CRC has been declining for over 30 years due in part to screening colonoscopies that detect and remove pre-cancerous polyps[1].

The adenoma detection rate (ADR) is a metric used by endoscopists representing the percentage of time at least one adenoma is detected on screening colonoscopies[2]. Adenoma detection rates differ widely among endoscopists, between 7% to 52%, with higher ADRs associated with a decreased risk of CRC. It is now recommended that endoscopists target an ADR target ≥ 25%[3]. Artificial intelligence (AI), specifically computer-aided detection (CADe) software is being studied to detect polyps during colonoscopy with the goal of increasing adenoma detection rates[2,4,5].

Broadly, artificial intelligence (AI) relates to the ability of a computer program to obtain outside data (e.g., images) and to subsequently take independent actions towards a particular goal (e.g., pattern identification). Machine learning is a form of AI that relies on the analysis of large datasets in order to make predictions that can be used for decision making. Deep learning, a subtype of machine learning, uses an artificial neural network comprised of layers of interconnected “computing units” that mimic biological neural connections and allow for complex “understanding” of input data. This neural network allows the computer program to learn independently from unstructured input data. Many times a deep learning program can process a large number of photos, independently identify patterns among them, and then use that information to make predictions about new images. This powerful technology that has been used to train machines in image and sound recognition is now being applied in the medical field in the form of computer-aided diagnosis and detection, which applies AI and computer vision technologies to the diagnosis of various pathologies. The technology is rapidly expanding in areas like colonoscopy where there is significant room to mitigate human error in visual diagnosis.

Many polyps detected and resected during colonoscopies are diminutive polyps (≤ 5 mm), and a significant number of these are non-neoplastic. Polypectomy increases the risk of complications during colonoscopy, including the risk for bleeding and perforation[6]. Computer-aided diagnosis (CAD) technology has been studied to characterize the histology of polyps in vivo.

The American Society for Gastrointestinal Endoscopy’s Preservation and Incorporation of Valuable Endoscopic Innovations (PIVI) initiative has provided guidelines that aim to reduce the cost and need for pathological assessments in addition to reducing the risks associated with polypectomy. The PIVI’s first guideline is known as “resect and discard”, which entails resecting the diminutive polyp and discarding if the CAD technology used to characterize the polyp has a similar surveillance interval compared to the traditional pathology assessment (≥ 90%). The PIVI’s second guideline allows the endoscopist to leave hyperplastic diminutive polyps in place in the rectosigmoid area if the CAD technology has a NPV ≥ 90% for characterizing adenoma histology[7].

COLORECTAL POLYP DETECTION

The first study to use CADe to detect colorectal polyps was published in 2003[8-11]. Karkanis et al[8] used wavelet transformation technology to detect polyps with a sensitivity of 93.6% and specificity of 99.3%. Years later, deep learning networks were applied to CADe, which has paved the way for in vivo and real-time analysis studies.

Urban et al[2] was the first to use CADe for polyp detection in real-time. The study assessed 9 standard colonoscopy videos. At the time of the colonoscopies, 28 polyps were identified and removed. Using CADe, an additional 17 polyps were detected, compared to only an additional 8 that were identified by an expert endoscopist[2,9]. Klare et al[12] then applied CADe in real time and in vivo to 55 colonoscopies. The ADR of the CADe was similar to the endoscopists’ (29.1% and 30.9%, respectively). However, the CADe was inferior in detecting flat and small polyps.

More recently, there have been randomized control trials (RCTs) performed in real-time using colorectal polyp detection technology. Wang et al[4] included 1058 patients in a non-blinded study, 536 were randomized to colonoscopy and 522 were randomized to colonoscopy with CADe. The ADR was statistically superior in the colonoscopy with the CADe group compared to the control group (29.1% vs 20.3%). Moreover, CADe was better at detecting diminutive adenomas, but there was no difference in detection rate for polyps larger than 5 mm. Notably, the CADe did not miss any polyps, but had 39 false positives alarms for polyps[5]. Wang et al[4] then performed a double blinded RCT in which patients were randomized to a colonoscopy with sham system (n = 478) or a colonoscopy with CADe (n = 484) group. Results showed the ADR was 34% in the CADe group, which was superior to the control group, 28%. Gong et al[13] randomized 704 patients in a partially blinded RCT to CADe assisted colonoscopy or control standard colonoscopy. Similarly, the ADR was significantly better in the CADe group than the control group (16% compared to 8%, respectively). Repici et al[14] performed a similar nonblinded RCT and found an ADR of 54.8% in the colonoscopy with CADe group, which was significantly better than the ADR for the control group (40.4%). The CADe was also able to detect more adenomas that were < 10 mm in size compared to the control group. The authors also found that there was no significant difference in withdrawal time (excluding biopsy time) of the endoscope between the two groups. Liu et al[15] randomized 1026 patients to CADe or control groups. The ADR was significantly better in the CADe group (39%) than the control group (23%). The CADe did not miss any polyps and there were only 36 false positive alarms. It was also noted that the withdrawal times between groups were similar (CADe 6.16 minutes compared to control 6.11 min).

Su et al[16] created an automatic quality control system (AQCS) to improve aspects of colonoscopy using a deep learning model. They randomized 659 patients and found that the AQCS group had a superior ADR than the control group (28.9% vs 16.5%). However, they found the AQCS had a longer withdrawal time (excluding biopsy time) compared to the control group (7.03 ± 1.01 min vs 5.68 ± 1.26 min).

Recent meta-analyses have concluded that CADe was accurate at detecting adenomas[17,18]. Barua et al[17] included 5 RCTs (with a total of 4311 patients) and concluded the ADR was significantly better using CADe with colonoscopy (29.6%) than colonoscopy alone (19.3%), with a false positive alarm mean of 11.2%. Lui et al[18] analyzed 6 studies that used CADe and found the accuracy of CADe was 90% with sensitivity and specificity of 95% and 88% respectively. In both studies, colonoscopy with CADe improved detection of diminutive adenomas[17,18].

A significant limitation of CADe technology is the potential to have high false positive alarm rates. Even though Wang et al[5] and Liu et al[15] had low rates, Hassan et al[19] reported a total of 1092 false positive alarms, which averaged 27.3 per colonoscopy. Also, many of the studies assessing CADe had control groups with low ADRs between 8%-23%[5,13,15,16], which is below the recommended target ADR of ≥ 25%[3].

Table 1 summarizes the key recent studies in colorectal polyp detection.

Table 1.

Colorectal polyp detection

Ref.	Study design	Algorithm type	Dataset	Results
Karkanis et al[8]	Retrospective	CADe (Wavelet Decomposition)	180 images	Sensitivity: 93.6%
				Specificity: 99.3%
Urban et al[2]	Retrospective	CADe (DCNN)	8461 images &20 colonoscopy videos	Accuracy: 96.4%
				False Positive: 7%
Klare et al[12]	ProspectiveIn vivo	CADe	55 colonoscopies	ADR of: CAD 29.1% and Endoscopist 30.9%
Wang et al[5]	Non-blinded RCT	CADe using Shanghai Wision Al Co. Ltd. (DCNN)	Randomized 522 patients to CADe and 536 to control group	ADR of CAD 29.1% vs control 20.3%
Wang et al[4]	Double blinded RCT	CADe using EndoScreener (DCNN)	Randomized 484 patients to CAD and 478 to sham system	ADR of CAD 34% vs control 28%
Gong et al[13]	Partially blinded RCT	CADe using ENDOANGEL (DCNN)	Randomized 355 patients to CAD and 349 to control	ADR of CAD 16% vs control 8%
Repici et al[14]	Partially-blinded RCT	CADe using GI-Genius (CNN)	Randomized 341 patients to CAD and 344 to control	ADR of CAD 54.8% vs control 40.4%
Liu et al[15]	Non-blinded RCT	CADe using Henan Xuanweitang Medical Information Technology Co. Ltd (convolutional 3D network)	Randomized 508 patients to CAD and 518 control	ADR of CAD 39% vs control 23%
Su et al[16]	Partially blinded RCT	Automatic quality control system (ACQS)(DCNN)	Randomized 308 patients to AQCS and 315 to control	ADR of AQCS 28.9% vs control 16.5%

CADe: Computer-aided detection; CAD: Computer-aided diagnosis; DCNN: Deep convolutional neural network; ADR: Adenoma detection rate.

POLYP CHARACTERIZATION

Going beyond merely identifying polyps, AI has been applied to provide real-time in vivo diagnoses of neoplastic vs non-neoplastic lesions. A number of different modalities for this have been described over the years.

White light endoscopy

Traditional white light (WL) endoscopy is the most familiar modality used by endoscopists today. High-definition white light (HDWL) endoscopy is the most recent improvement in this area. Recent randomized controlled trials have demonstrated that HDWL is non-inferior to other modalities, specifically narrow band imaging (NBI) and chromoendoscopy[20-22]. In 2016, Rex et al[20] found that there was no statistically significant difference in the number of serrated lesions detected in 804 patients randomized to undergo either WL or NBI endoscopy. The same year, Klare et al[21] published the results of a trial that randomized 380 patients to either HDWL or NBI. They also found no statistically significant advantage of one modality over the other in distinguishing between neoplastic and non-neoplastic polyps. Yang et al[22] randomized 210 patients with ulcerative colitis to undergo colon cancer screening with HDWL or chromoendoscopy and found no significant difference in dysplasia detection rates between the two modalities (5.6% for HDWL vs 3.9% for chromoendoscopy).

In 2017, Komeda et al[23] designed and tested a CAD system based on a convolutional neural network to augment WL endoscopy. It functioned as an AI system that trained with previously collected colonoscopy images to assist endoscopists in detecting and diagnosing colon polyps during WL endoscopy. After training on 1200 images, their CAD-neural network system correctly differentiated between adenomatous and non-adenomatous polyps in 70% of newly presented cases[23].

This strategy of applying AI to the detection and classification of colorectal lesions using WL endoscopy has continued to be an area of active study. Researchers have shown that CAD, using both convolutional neural networks and deep learning models, promises to identify suspicious lesions and accurately classify them[24-26]. Zheng et al[24] developed a convoluted neural network (CNN) to be used with WL endoscopy. They trained their AI system with over 600 polyp-containing images from independent public databases and found that their diagnostic model had a sensitivity of 68.3% and a precision of 79.3% when applied to 196 new polyp-containing images[24]. However, they found significant variation in model performance depending on which image database was used to train the CNN, and they also only trained and tested their CNN on still images[24]. Going beyond polyp identification, Yang et al[26] developed a deep learning (DL) model to assist in classification of colorectal lesions during WL endoscopy. They trained their model on 3828 images and validated it on a set of 240 new images. When classifying lesions as neoplastic vs non-neoplastic, their model had a sensitivity of 95.4% and specificity of 30.1%[26]. Their model was also able to classify advanced lesions (high grade dysplasia and stages T1-T4 CRC vs non-advanced (tubular adenomas and non-neoplastic lesions) with a sensitivity of 80.0% and specificity of 91.3%[26].

A recent prospective crossover study conducted by Wang et al[25] compared traditional WL colonoscopy to CAD-assisted colonoscopy in 369 patients. Patients requiring colonoscopy underwent either traditional WL colonoscopy or CAD-assisted colonoscopy immediately followed by the other, such that each study participant underwent both methods. They found that the adenoma miss rate was 40% for those undergoing traditional colonoscopy first vs 14% in those undergoing CAD-assisted colonoscopy first[25]. Polyp detection followed a similar trend with a miss rate of 46% in those undergoing traditional colonoscopy first vs 13% in those undergoing CAD-assisted colonoscopy first[25]. Interestingly, of the adenomas missed, participants undergoing CAD-assisted colonoscopy were less likely to have polyps under 5mm and under 10 mm missed, suggesting that CAD is particularly helpful in identifying smaller lesions[25].

The key studies for polyp characterization using white light endoscopy are summarized in Table 2.

Table 2.

White light endoscopy

Ref.	Study design	Algorithm type	Dataset	Results
Komeda et al[23]	Diagnostic model development	CAD-neural network combination to assist WL endoscopy	1200 training images then tested on 10 new images	Cross-validation accuracy: 0.751
Zheng et al[24]	Diagnostic model development	WL endoscopy using YOLO (CNN)	196 WL images from an independent public database	Accuracy: 79.3%
				Sensitivity: 68.3%
Wang et al[25]	Prospective crossover study	Traditional WL endoscopy vs CAD colonoscopy	369 patients from a single hospital in China	Adenoma miss rate of 13.9% in the CAD group vs 40% in the traditional group, P < 0.0001
Yang et al[26]	Diagnostic model development	Validation of a deep learning model called “ResNet-152” to classify colorectal lesions	3828 WL colonoscopy images from 1339 patients	Mean model accuracy: 79.2% for advanced CRC, early CRC/HGD, TA, and non-neoplastic
				AUC: 0.818

CAD: Computer-aided diagnosis; WL: White light; CNN: Convoluted neural network.

Narrow band imaging

Narrow band imaging (NBI) is used to enhance visualization of vascular patterns in the epithelium of lesions to aid in the classification of polyps[27-29]. However, training and experience is needed to operate NBI; therefore, studies have applied computer-aided diagnosis to NBI[27,29-34].

Tischendorf et al[29] was the first to apply CAD to magnified NBI. The CAD evaluated 209 polyps and assessed for three vessel features on each NBI image, then a support vector machine was used to classify the polyp as neoplastic or non-neoplastic. It had an accuracy of 85.3%, sensitivity of 90% and specificity of 70.2%. When compared to the consensus of the investigators (accuracy 91.9%, sensitivity 93.8%, and specificity 85.7%), the CAD was inferior. Gross et al[27] performed a similar study with 434 polyps, but assessed for nine vessel features. The CAD had an accuracy of 93.1%, sensitivity of 95%, specificity of 90.3%, and NPV of 92.4%, which was comparable to the results of the expert endoscopists and superior to the novice endoscopists.

Years later, Byrne et al[30] and Chen et al[31] improved CAD by creating deep learning models to analyze NBI images and categorize polyp histology. Both studies only included diminutive polyps (≤ 5 mm). Chen et al[31]’s deep learning model assessed still NBI images of 284 polyps and classified polyps with an accuracy of 90.1%, sensitivity of 96.3%, specificity of 78.1%, PPV of 89.6%, and NPV of 91.5%. The authors also studied the diagnosis time, which was statistically faster for the deep learning model than both expert and novice investigators[31]. Byrne et al[30]’s deep learning model assessed 125 endoscopy NBI video segments of polyps. Of the classified polyps, the accuracy was 94% with sensitivity 98%, specificity 83%, PPV 90%, and NPV 97%. However, the model was not able to classify 15% of polyps due to a lack of confidence. All of the CAD with NBI studies named above with the exception of Tischendorf’s initial study met the PIVI criteria for resect and discard or diagnose and leave in situ[7,27,30,31].

Few prospective studies have been performed with CAD using NBI imaging to classify polyp histology. Kominami et al[32] evaluated 118 polyps with NBI. The CAD had an accuracy of 94.9%, sensitivity of 95.9%, specificity of 93.3%, PPV of 95.9%, and NPV 93.3%. The authors used this data to investigate colonoscopy surveillance interval which did not change in 38 of the 41 patients when using the CAD results to classify polyps. Mori et al[33] also performed a study assessing CAD when used with NBI on 466 diminutive polyps. The NPV for rectosigmoid neoplastic polyps ranged from 95.2% to 96.5% depending on the worst or best case scenario respectively.

Most recently, Song et al[35] created a CAD using a deep learning model and tested it in vivo by sending still NBI images during the colonoscopy to a computer. The CAD then categorized the histology in real time. The polyps were classified as serrated, benign adenoma, or deep submucosal cancer with an accuracy of 82.4%, which was superior to trainees (63.8%), but inferior to expert endoscopists (87.3%). The accuracy of trainee endoscopists improved with the addition of CAD to 82.7% showing that CAD can increase the accuracy in this group.

Table 3 provides a summary of the key recent studies of polyp characterization using narrow band imaging.

Table 3.

Narrow band imaging

Ref.	Study design	Algorithm type	Dataset	Results
Tischendorf et al[29]	Prospective Ex vivo	CAD – NBI (support vector machine)	209 polyp images	Accuracy: 85.3%
				Sensitivity: 90%
				Specificity: 70.2%
Gross et al[27]	Prospective Ex vivo	CAD – NBI (support vector machine)	434 polyp images	Accuracy: 93.1%
				Sensitivity: 95%
				Specificity: 90.3%
				NPV: 92.4%
Chen et al[31]	Retrospective	CAD – NBI (DCNN)	284 polyp images	Accuracy: 90.1%
				Sensitivity: 96.3%
				Specificity: 78.1%
				PPV: 89.6%
				NPV: 91.5%
Byrne et al[30]	Retrospective	CAD—NBI (DCNN)	125 polyp videos	Accuracy: 94%
				Sensitivity: 98%
				Specificity: 83%
				PPV: 90%
				NPV: 97%
Kominami et al[32]	Prospective	CAD –NBI (support vector machine)	118 polyps	Accuracy: 94.9%
				Sensitivity: 95.9%
				Specificity: 93.3%
				PPV: 95.9%
				NPV: 93.3%
Mori et al[33]	Prospective	CAD – NBI (support vector machine)	466 polyps	NPV: 95.2% to 96.5%
Song et al[35]	Prospective In vivo	CAD –NBI (DCNN)	363 polyps	Accuracy: 82.4%

CAD: Computer-aided diagnosis; NBI: Narrow band imaging; DCNN: Deep convolutional neural network.

Laser-induced fluorescence spectroscopy

Another strategy currently under investigation to optically diagnose lesions during endoscopy is laser-induced fluorescence spectroscopy. This diagnostic method relies on low-power laser radiation to induce fluorescence in tissues that can differentiate normal from neoplastic lesions[36]. In recent years, CAD systems have been developed to analyze the fluorescent spectra produced when tissues are exposed to a laser. These CAD systems take advantage of the differences between the fluorescence of normal and pathological tissue to predict the likelihood that a lesion is abnormal.

Kuiper et al[37] and Rath et al[38] studied a laser-induced fluorescence spectroscopy system designed to be used in real-time to help clinicians make decisions regarding biopsy and resection of concerning lesions. However, it is important to note that the accuracy of the algorithm used by Kuiper et al[37] was 73.4% and the NPV only 74.4%, falling short of the performance thresholds of the American Society for Gastrointestinal Endoscopy’s PIVI initiative for diminutive lesions. The pilot study conducted by Rath et al[38] was more promising, with an overall accuracy of 84.7%, sensitivity of 81.8%, specificity of 85.2%, and NPV of 96.1%. A 2017 randomized controlled trial by Min et al[39] was further able to demonstrate that use of linked color imaging technology (which enhances the colors produced by laser endoscopic modalities) improved overall polyp detection rate when compared to traditional white light endoscopy (polyp detection rate 73% for WL and 91% with linked color imaging).

The recent key studies of laser-induced fluorescence spectroscopy are summarized in Table 4.

Table 4.

Laser-induced fluorescence spectroscopy

Ref.	Study design	Algorithm type	Dataset	Results
Kuiper et al[37]	Diagnostic model development	Diagnostic performance of WavSTAT	87 patients	Accuracy: 73.4%
				NPV: 74.4%
Rath et al[38]	Diagnostic model development	Diagnostic performance of WavSTAT for predicting polyp histology	27 patients	Accuracy: 84.7%
				Sensitivity: 81.8%
				Specificity: 85.2%
				NPV: 96.1%
Min et al[39]	Randomized controlled trial	Linked color imaging with laser endoscopic system vs WL	141 patients from 3 hospitals in China	Polyp detection rate of 91% in the LCI group, 73% in the WL group, P < 0.0001

WL: White light.

Autofluorescence endoscopy

Autofluorescence imaging (AFI) is a form of image enhanced endoscopy that differentiates tissues based on their various abilities to capture and reflect fluorescent light[40]. Similar to laser-induced fluorescence spectroscopy, this method takes advantage of endogenous reflective properties of various tissues (fluorophores), but instead of using a laser emitting an exact wavelength of light, AFI uses incoherent light sources. This technology aims to visually highlight tumors, which have more heterogeneous fluorescence on their surface compared to normal colonic mucosa. This image-enhancement displays normally fluorescing mucosa as green and abnormally fluorescing mucosa as red/purple.

In a 2019 study of 802 patients randomized to undergo either AFI endoscopy or white light endoscopy, Takeuchi et al[41] found that using AFI during endoscopy increased the number of flat neoplasms detected overall compared to WL, especially in the ascending colon. However, the overall detection rate of advanced neoplasms was not significantly improved with AF compared to WL[41]. In a meta-analysis of 11 studies, Wanders et al[42] calculated that the sensitivity and specificity of autofluorescence imaging for the optical diagnosis of colonic lesions were 86.7% and 65.9% respectively. This led the authors to conclude that AFI is not as reliable as other methods for visual diagnosis of colonic neoplasms. This was further confirmed in a 2018 meta-analysis by Imperatore et al[43] which found no significant difference between the dysplasia detection rates between AFI and WL (OR = 1.42, 95%CI: 0.74-4.11) when combining the results of two randomized controlled trials representing 92 patients undergoing surveillance colonoscopy.

More recently, researchers have developed CAD systems that can further characterize the images obtained during AFI endoscopy using software that can calculate the green to red light ratios of various tissues encountered during colonoscopy[44-47]. Such developments may help differentiate lesions from normal mucosa in cases where the green to red variation is less obvious, improving on the results of the studies looking at the use of AFI without the use of CAD.

Arita et al[44] created a color-contrast index (CCI) for AFI. Their CCI was developed from 54 colorectal lesions found in 43 patients who underwent either WL or AFI endoscopy. They found that as the CCI increased (i.e., greater contrast between the lesion and the adjacent normal tissue), so did the malignant potential of the assessed lesion (i.e., carcinomas had higher CCIs on average compared to adenomas)[44]. Aihara et al[45] expanded on this idea by using color-analysis software to calculate red/green ratios (RGR) for 102 Lesions in 32 patients undergoing AFI endoscopy. In their study, they were able to differentiate neoplastic from non-neoplastic lesions with sensitivity of 94.2%, specificity of 88.9%, PPV of 95.6%, and NPV of 85.2%[45]. In a similar study, Inomata et al[46] also calculated RGRs to distinguish between non-neoplastic lesions, adenomas plus superficial cancers, and deep cancers. They were able to characterize hyperplastic polyps and neoplastic lesions, with sensitivity of 83.9%, specificity of 82.6%, PPV of 53.1%, and NPV of 95.6%[46]. Additionally, they were able to differentiate between adenomas plus superficial cancers and deep submucosal cancers with a sensitivity of 80.0%, specificity of 84.4%, PPV of 29.6%, and NPV of 98.1%[46].

In 2019, Horiuchi et al[47] developed software to calculate real-time RGR ratios with the specific goal of identifying diminutive neoplastic rectosigmoid polyps (≤ 5 mm). Using their CAD-assisted AFI endoscopy, they identified 429 diminutive polyps (258 rectosigmoid) in 95 patients. The endoscopists then confirmed whether the lesions identified with the CAD-assisted AFI were actually diminutive neoplastic polyps with trimodal imaging endoscopy (TME) combining findings of WL, AFI, and NBI. The CAD-assisted AFI software was able to identify diminutive neoplastic polyps with a sensitivity of 80.0%, specificity of 95.3%, PPV of 85.2%, and NPV of 93.4%[47].

Table 5 contains a summary of the key recent studies involving autofluorescence endoscopy.

Table 5.

Autofluorescence endoscopy

Ref.	Study design	Algorithm type	Dataset	Results
Arita et al[44]	Diagnostic model development	Calculation of a color-contrast index (CCI) for AFI	43 patients who underwent both WL and AF endoscopy	Sensitivity: 95.3%
				Specificity: 63.6%
Aihara et al[45]	Diagnostic model development	CAD-assisted AF	32 patients undergoing colonoscopy in a Japanese hospital	Sensitivity: 94.2%
				Specificity: 88.9%
				PPV: 95.6%
				NPV: 85.2%
Inomata et al[46]	Diagnostic model development	CAD-assisted AF	88 patients	Accuracy: 82.8%
				Sensitivity: 83.9%
				Specificity: 82.6%
				PPV: 53.1%
				NPV: 95.6%
Horiuchi et al[47]	Diagnostic model development	CAD-assisted AF	95 patients undergoing colonoscopy	Accuracy: 91.5%
				Sensitivity: 80.0%
				Specificity: 95.3%
				PPV: 85.2%
				NPV: 93.4%

AFI: Autofluorescence imaging; WL: White light; CAD: Computer-aided diagnosis; AF: Autofluorescence.

Magnifying chromoendoscopy

In the technique of magnifying chromoendoscopy, suspected colonic lesions are washed with proteinases and colored with an indigo carmine or crystal violet solution in order to allow better visualization of the surface under magnification up to 150 times[48]. In a systematic review, Brown et al[49] showed that chromoendoscopy significantly increased both the number of patients with polyps (OR = 1.87, 95%CI: 1.51-2.3) and neoplasms detected (OR = 1.53, 95%CI: 1.31-1.79). Kudo et al[48] demonstrated that certain pit patterns on magnifying chromoendoscopy are associated with malignancy, showing sensitivity of 97.8%, specificity of 91.4%, and accuracy of 97.1%. Kanao et al[50] demonstrated that magnifying chromoendoscopy can be used to differentiate severely irregular lesions from those with only mild irregularities, a key distinction as 56.1% of the former are associated with deep submucosal invasive adenomas vs only 6.7% of the latter.

Several automated computer-based systems have been developed for analysis of pit patterns. The system developed by Takemura et al[51] was able to accurately diagnose 132 out of 134 (98.5%) of images captured with magnifying chromoendoscopy demonstrating that artificial intelligence aided systems can reliably predict histological changes compared to endoscopists. Häfner et al[52] used texture analysis of magnified chromoendoscopic images to achieve accuracy as high as 99.59%. Recent work by Qi et al[53] further showed that artificial intelligence can be used to quantify colonic crypts and provide objective measures of area, density, eccentricity, solidity, straightness, and parallelism which in turn can be used to reduce variability compared to human observation.

Recent studies on the use of magnifying chromoendoscopy in polyp characterization are summarized in Table 6.

Table 6.

Magnifying chromoendoscopy

Ref.	Study design	Algorithm type	Dataset	Results
Takemura et al[51]	Partially blinded retrospective study	CAD using HuPAS	134 pit pattern images	Accuracy: 98.5%
Häfner et al[52]	Partially blinded retrospective study	CAD using Dual-Tree Complex Wavelet Transform	484 RGB pit pattern images	Accuracy: 99.59%
Qi et al[53]	Diagnostic model development	CAD using automated imaged analysis	79 colon samples (14 normal, 44 normal tissue adjacent to cancer, 21 malignant)	Automated segmentation achieved precision ratio of 0.69 and match ratio of 0.73

CAD: Computer-aided diagnosis.

Endocytoscopy

The technique of endocytoscopy uses a contact light microscope attached to a colonoscope to provide endoscopic images with ultra-magnification up to 520 times. The addition of staining allows for real-time histological diagnoses to be made. Studies have shown that the accuracy of this technique is comparable to traditional biopsy[54]. However, a major limitation is the need for expert experience in order to make real-time diagnoses. CAD has been developed in response to this shortcoming.

In 2015, Mori et al[54] described the use of an endocytoscopic imaging computer-aided diagnostic system. In this study, 39 non-neoplastic and 176 neoplastic small colorectal polyps less than 10 mm in size were analyzed by artificial intelligence software and compared to the results of both expert and trainee endoscopists. They showed comparable sensitivity (92% vs 92.7%) and accuracy (89.2% vs 92.3%) of the computer-aided diagnostic system compared to experts[54]. Moreover, the artificial intelligence program performed significantly better than trainee endoscopists who only had sensitivity of 81.8% and accuracy of 80.4%[54]. Takeda et al[55] described the development of a CAD system that used 5543 endocytoscopic images for machine learning. Following this, 188 images of a mix of adenomas and invasive cancers were analyzed by the CAD system and compared to pathological diagnoses with 89.4% sensitivity, 98.9% specificity, 98.8% accuracy, 98.8% PPV, and 90.1% NPV[55]. Moreover, the system used a support vector machine to calculate the probability of results being classified accurately. The study specifically looked at high-confidence diagnoses described as those having a ≥ 90 % probability of being correct. Out of 188 images analyzed, 134 fell into this category with 98.1% sensitivity, 100% specificity, 99.3% accuracy, 100% PPV, and 98.8% NPV[55].

In 2018, Mori et al[33] showed the efficacy of real-time endocytoscopy with CAD in detecting diminutive polyps ≤ 5 mm in size. 466 polyps were assessed with 98.1% pathologic prediction rate, 93.8% sensitivity, 90.3% specificity, and 94.1% PPV. Moreover, they were able to demonstrate overall negative predictive value of 96.4% which is significantly above the threshold for a “diagnose-and-leave” treatment strategy. However, the study only demonstrated 65.8% NPV for lesions proximal to the recto-sigmoid area[33].

In 2020, Kudo et al[56] performed a study to evaluate the efficacy of an AI system that uses endocytoscopic images to look at cell nuclei, crypt structure, and microvessels. It was able to identify malignant lesions with 96.9% sensitivity, 100% specificity, 98% accuracy, 100% PPV, and 94.6% NPV. Compared to expert endoscopists with 92.8% sensitivity, 94.3% specificity, and 93.9% accuracy as well as trainee endoscopists with 70.8% sensitivity, 65.7% specificity, and 69% accuracy, this CAD system significantly outperformed both groups[56].

Table 7 summarizes the key recent studies in endocytoscopy.

Table 7.

Endocytoscopy

Ref.	Study design	Algorithm type	Dataset	Results
Mori et al[54]	Pilot study	CAD using EC-CAD	176 colorectal polyps from 152 patients	Accuracy: 89.2%
				Sensitivity: 92%
				Specificity 79.5%
Takeda et al[55]	Retrospective study	CAD using EC-CAD	5543 endocytoscopy images for machine learning. 200 test images	Overall
				Accuracy: 94%
				Sensitivity: 89.4%
				Specificity: 98.9%
				PPV: 98.8%
				NPV: 90.1%
				High-confidence diagnosis
				Accuracy: 99.3%
				Sensitivity: 98.1%
				Specificity: 100%
				PPV: 100%
				NPV: 98.8%
Mori et al[33]	Single-group, open-label, prospective study	Real-time CAD during colonoscopy	466 diminutive polyps from 325 patients	Accuracy: 98.1%
				Sensitivity 93.8%
				Specificity 90.3%
				PPV 94.1%
				NPV 89.8%
Kudo et al[56]	Retrospective study	CAD using EndoBRAIN	100 polyps from 89 patients	Accuracy: 98%
				Sensitivity 96.9%
				Specificity 100%
				PPV 100%
				NPV 94.6%

CAD: Computer-aided diagnosis.

Confocal endomicroscopy

Anatomic variation can hinder accurate traditional endoscopic biopsy. Confocal endomicroscopy produces high resolution magnification of the mucosal layer of the gastrointestinal tract using laser illumination with simultaneous detection of light reflected from the tissue through a narrow pinhole. By filtering out light that is scattered from angles outside of layers corresponding to mucosa in question, this technique allows for high spatial resolution and real-time endoscopic evaluation of targeted areas of tissue. Compared to normal colonic mucosa with well-organized crypt structures, malignancy causes irregularities and interruptions[57].

André et al[58] designed software that used probe-based confocal laser endomicroscopy to automatically classify colonic polyps. 135 images with both neoplastic and nonneoplastic polyps were analyzed by the automated software with 92.5% sensitivity, 83.3% specificity, and 89.6% accuracy which was not significantly different compared to diagnosis by two expert endoscopists with 91.4% sensitivity, 85.7% specificity, and 89.6% accuracy. Ştefănescu et al[59] retrospectively analyzed 1035 endomicroscopic images processing them through a CAD system that allowed for feature identification via fractal analysis of glandular structures showing that homogeneity and feature number were significantly different in malignancy. In turn, this was used to design an artificial intelligence program with a diagnosis error rate of 15.5%[59].

One problem with early probe-based confocal endomicroscopic images is the need for high-level magnification which leads to a long learning-time for automated image interpretation. Taunk et al[60] showed the efficacy of a CAD algorithm that utilized a lower magnification method with a wider field-of-view necessitating less images. The algorithm demonstrated similar sensitivity, specificity, and accuracy compared to expert endoscopists (95% vs 98%, 94% vs 95%, and 94% vs 96% respectively) and significantly better performance than less experienced endoscopists who only had sensitivity of 60%, specificity of 85%, and accuracy of 73%[60].

The recent key studies on confocal endomicroscopy are summarized in Table 8.

Table 8.

Confocal endomicroscopy

Ref.	Study design	Algorithm type	Dataset	Results
André et al[58]	Diagnostic model development	CAD using content based image retrieval (CBIR) approach	135 polyps from 71 patients	Accuracy: 89.6%
				Sensitivity 92.5%
				Specificity 83.3%
Ştefănescu et al[59]	Diagnostic model development	CAD using NAVICAD and a two layer CNN	1035 endomicroscopy images including 725 for training, 155 for validation, and 155 for testing.	Testing decision accuracy error rate of 15.48% (24 out of 155 images)
Taunk et al[60]	Feasibility study	CAD using expectation-maximization algorithm	189 endomicroscopy images from 26 patient	Accuracy: 94.2%
				Sensitivity 94.8%
				Specificity 93.5%

CAD: Computer-aided diagnosis; CNN: Convoluted neural network.

FUTURE DIRECTIONS

Automated polyp detection and characterization

Many studies have been performed using AI to detect or classify the histology of colorectal polyps; however, little research has been done on simultaneous detection and classification of colorectal polyps using AI. Mori et al[33] combined previously studied CADe and CAD systems to develop AI technology that is able to detect then characterize the colorectal polyps[61,62]. White-light imaging was used to detect polyps with an accuracy of 94%[61,62]. Then, classification of the polyps was then performed using magnified NBI with a NPV of 95.2%[33,61]. Ozawa et al[34] created a deep convolutional neural network (DCNN) that detected polyps using white light or NBI with a sensitivity of 92% and a PPV of 86%. The DCNN then classified the polyps using either white light with an accuracy of 83% and NPV of 90% or NBI with an accuracy of 81% and NPV of 91%. Further research is needed in concurrent automated detection and characterization of colorectal polyps.

Robotics

Much of the work in robotics has centered around the use of self-propelling colonoscopes and less on polyp detection. Recent studies on the use of robotics in colonoscopy are summarized in Table 9.

Table 9.

Robotics

Ref.	Study design	Algorithm type	Dataset	Results
Eickhoff et al[63]	Prospective, nonrandomized, unblinded feasibility study	CAD using NeoGuide Endoscopy System	10 patients	100% cecal intubation rate. Median time to cecum 20.5 min. 0 complications or adverse effects reported at discharge, 48 h, and 30 d
Pullens et al[64]	Randomized control trial with crossover design	CAD using automated lumen centralization	8 expert endoscopists and 10 endoscopy-naïve novices performing endoscopy on a validated colon model with 21 polyps	Novice
				Accuracy: 88.1%
				Time to cecum: 8 min 56 s
				Experts
				Accuracy: 69%
				Time to cecum: 13 min 1 s

CAD: Computer-aided diagnosis.

Eickhoff et al[63] demonstrated the first use of a novel computer-assisted colonoscope in 2007 that would change shape at 16 different segments depending on insertion depth in an effort to decrease discomfort from colonoscope looping. While it required an endoscopist to steer the scope, the device used CAD to change shape as it was advanced. The device was able to intubate the cecum in 100% of patients with no complications at discharge, 48 h, and 30 d[63].

In 2016, Pullens et al[64] demonstrated the utility of colonoscopy with robotic steering and automated lumen centralization (RS-ALC). In a study of 18 endoscopists including 8 experts and 10 novices, the addition of RS-ALC significantly improved the time to intubate the cecum in novices (8 min 56 s compared to baseline 11 min 47 s without RS-ALC) as well as polyp detection rate (88.1% vs 78.6%)[64]. However, similar results were not seen with expert endoscopists whose time to intubate the cecum actually increased (13 min 1 s compared to baseline 2 min 9 s without RS-ALC) and polyp detection rate decreased (69% vs 80.9%)[64]. Slawinski et al[65] demonstrated the use of an in vivo autonomously controlled highly compliant magnetic flexible endoscope with diagnostic and therapeutic capability using an actuating permanent magnet in animal studies. They were able to conduct autonomous endoscopic retroflexion with 100% success in 30 attempts without perforation or trauma in pigs. However, diagnostic capability was worse than traditional endoscopy with an average detection miss rate of 21.7% and completion time of 575 s (compared to 5% and 257 s). When looking at lesion targeting alone, the robotic program took on average 251 s to identify lesions compared to only 32 s with traditional endoscopy[65].

In 2020, Formosa et al[66] showed the use of a sensor-enabled treaded robotic colonoscope with multiple degrees of freedom which was notable for containing all the functions of a traditional endoscope including direct visualization, channels for insufflation and irrigation as well as a tool port for endoscopy. It also had inertial measurement technology in addition to a magnetometer, motor encoders, and motor current sensors for future autonomous use. Ex-vivo porcine results showed locomotion ability up to 40 mm/s[66].

CONCLUSION

Traditional colonoscopy has been shown to reduce colon cancer incidence by more than 80%[67]. The application of artificial intelligence, computer-aided detection, and computer-aided diagnosis to this field provides possibilities for improving an already powerful tool. Namely, the possibility of combining these technologies for real-time endoscopic detection and analysis of lesions overall makes the procedure less operator dependent. With the ability to identify smaller diminutive lesions as non-cancerous, these techniques also offer time and resource savings. Given the widespread use of colonoscopy as a screening test and an aging world population, this potentially translates to billions of dollars in cost reductions and even the possibility of extending screening intervals. Multiple modalities have shown to increase adenoma detection rates and have negative predictive values > 90%, meeting the goals of the ASGE’s PIVI initiatives.

However, multiple challenges remain including a lack of large multicenter clinical trials and comparison of computer-aided detection and diagnosis modalities. Additionally, more widespread regulatory approval on government and payer levels is needed. There is still much room for clinical research following software development and more prospective studies evaluating the real-life application of these technologies in the endoscopy suite. Nonetheless, continued development of CADe, CAD and AI in colonoscopy offers patients the possibility of living longer and healthier lives.