How Artificial Intelligence Will Impact Colonoscopy and Colorectal Screening

INTRODUCTION

Colorectal cancer is a major cause of cancer-related death, and colonoscopy with complete resection of neoplastic lesions has been suggested to reduce both incidence and mortality of colorectal cancer.^1,2 Quality of colonoscopies in detecting malignancies varies, and outpatient colonoscopies in surgery centers cost the most for all procedures performed in ambulatory surgery centers, an estimated $3.2 billion in the United States in 2018.³ In the American Board of Internal Medicine’s “Choosing Wisely” campaign, one of the areas of endoscopic overuse is surveillance colonoscopy in individuals with low-risk polyps.⁴ Payers are motivated to decrease unnecessary costs associated with colonoscopy.

Artificial intelligence (AI) is poised to transform clinical practice through machine learning, a method that overlaps computer science and statistics to create software programs that directly learn from patterns found in data input.⁵

AI adds value to colorectal cancer screening and surveillance by enhancing the visual abilities of the endoscopist to find precancerous polyps and classify polyps that do not need to be removed. Currently, the state of AI in colonoscopy consists of software running supervised machine learning algorithms to specifically find polyps or to classify them. In the future, we anticipate a fully integrated system that will provide decision support for assessing depth of invasion, and possibly even integrating genotypic and other data to provide personalized risk assessments for colon cancer.

ARTIFICIAL INTELLIGENCE PROMOTES VALUE IN COLORECTAL CANCER SCREENING AND SURVEILLANCE

Value in health care has been defined as improvement in quality or decrease in cost.⁶ Colonoscopy as a preventive measure to completely remove neoplastic lesions holds value for public health in preventing colorectal cancer.¹ However, this is dependent on the quality of the procedure in finding all precancerous lesions, which can be reflected with adenoma detection rates (ADRs).⁷ One study has suggested that increase of 1% in ADR leads to a 3% decreased risk of interval colorectal cancer.⁷ AI in colonoscopy can potentially improve the quality of the procedure through computer-aided detection (CADe) to increase the adenoma detection rate.

Costs can be managed by expert judgment to avoid removing low-risk polyps unnecessarily, proposed in the Preservation and Incorporation of Valuable endoscopic Innovations (PIVI) by guidelines from the American Society for Gastrointestinal Endoscopy for optic biopsy of diminutive polyps.⁸ AI can decrease the costs related to unnecessary polyp removal through computer-aided diagnosis (CADx), as implemented by either a resect-and-discard or diagnose-and-leave strategy.

The technology that has advanced CADe into real-world implementation is deep learning through neural networks, a branch of machine learning that discovers features that are meaningful through large amounts of data. This contrasts with earlier methods, when algorithms were built by a human selecting features thought to be meaningful (recently summarized in a review by Ahmad and colleagues⁹).

For deep learning-driven CADe alone, there are 3 recent prospective validation studies.^10–12 (Table 1).

Table 1.

Deep learning computer-aided detection with validation on colonoscopy videos with histologically confirmed polyps

Study	Methods	Training Dataset	Validation Dataset	Sensitivity (per Image Frame), %	Specificity, %	Area Under the Receiver Operator Curve
Wang et al,10 2018	Convoluted Neural Networks	5545 images	138 videos	91.6
Misawa et al,11 2018	Convoluted Neural Networks	73 colonoscopy videos divided into short videos: training (411 videos)	Testing (135 videos)	90.0	63.3	0.87
Urban et al,12 2018	Convoluted Neural Networks	8641 images	9 videos	93.0	93.0

For CADx alone, machine learning models have been used in 6 prospective validation studies, which test different modalities: magnifying narrow band imaging (NBI),^13,14 endocytoscopy,¹⁵ laser-induced fluorescence spectroscopy,^16,17 and autofluorescence endoscopy.¹⁸ (Table 2).

Table 2.

Machine learning computer-aided diagnosis with validation on captured images of polyps

Study	Setting	Methods (with # Features)	Training Dataset	Validation Dataset, no. Polyps	Sensitivity, %	Specificity, %
Chen et al,14 2018	Magnifying NBI	Deep Neural Network	2157 polyps	284	96.3	78.1
Kominami et al,13 2016	Magnifying NBI	Support Vector Machines (SVM) (128 features)	2247 polyps at maximal magnifying NBI	118	95.9	93.3
Mori et al,15 2017	Endocytoscopy	SVM (312 features)	61,925 endocytoscopic images	466	91.3 – 93.8	88.7–91.0
Rath etal,16 2016	Laser-induced fluorescence spectroscopy	WavStat4 – Linear Discriminant Analysis	Not reported	137	81.8	85.2
Kuiper et al,17 2015	Laser-induced fluorescence spectroscopy	WavStat4 – Linear Discriminant Analysis	Not reported	207	85.3	58.8
Aihara et al,18 2013	Autofluorescence endoscopy	Unknown	103 polyps	102	94.2	88.9

Both CADe and CADx have been integrated for fully automated polyp detection and immediate characterization in real time in a recent study trained a convolutional neural network using a training set of 223 polyp videos (60,089 frames) and validated the algorithm on 40 videos (106 polyps) with sensitivity 98% and specificity of 83% for identifying adenomas.¹⁹

An obvious limitation of the technology is that the algorithm cannot detect polyps that are not in the visual field of the colonoscope and performance may vary based on quality of inspection. Furthermore, other factors such as the speed of withdrawal and bowel preparation can also affect algorithm performance. This of course is not unique to AI, but applies to other devices such as forms of virtual chromoendoscopy.

Finally, a group recently presented (in abstract form) a system that decreases documentation burden by automating the input of quality measures based on image recognition (time of insertion, cecal intubation time, withdrawal time, tool use and intervention via Current Procedural Terminology codes and preparation quality via Boston Bowel Preparation Scale).²⁰ They deployed an approach with 7 separate convoluted neural networks with live feedback during the procedure.

INTEGRATION INTO CLINICAL PROCESSES DIRECTLY AFFECTS ADOPTION OF MACHINE LEARNING PRODUCTS

Clinical integration is key, particularly in the fast-paced world of colonoscopy, and special considerations must be taken to ensure that the product enhances clinical care. As a baseline, machine learning products should provide a fully automate diagnostic workflow during colonoscopy that detects and characterizes polyps in real time. A recent model that does both was tested on 215 colonoscopy videos in which the model detected and tracked individual polyps while also providing an estimate for the histology via optical biopsy.²¹ In addition, the product should not unnecessarily prolong procedures, overwhelm practitioners with false positives, or become a distraction that adversely affects the ability of practitioners to perform the procedures well. These models function as clinical decision support, and must consider how explainable the model is to providers, the usability of the product during clinical care, how the information is delivered to the providers and how the product increases or decreases the time necessary to provide clinical care.²² The ultimate goal is to enhance the providers’ performance; for colonoscopy, improving efficiency for endoscopists is a priority for quality care.²³

Most proposed measures for efficiency include an element of time. This is particularly relevant in high-volume endoscopy centers, where the number of cases that can be performed directly affects the revenue of the center. Algorithms used must therefore seek to decrease the amount of time it takes for colonoscopies to be performed with the same level of quality (eg, with no decrease in the adenoma detection rate).

Successful integration also requires adequate study of the human factors and perception of the product, including robust educational programming and training, ongoing user feedback regarding confidence in the tool, and logistical issues regarding its use. For example, training should not only teach providers how to use the product but also educate providers on how to adjudicate recommendations that may not agree with their clinical experience or impression. An ongoing challenge specific to deep learning is the “black box” nature of the deep learning algorithms, which limit the ability to explain how the algorithm came up with specific recommendations.

MACHINE LEARNING PRODUCTS REQUIRE RIGOROUS VALIDATION

When working with deep learning or other machine learning products, it is essential to ensure that the models are appropriately tested, or validated.²⁴ Because the model learns from the available data, the data used to train the models should never be used to evaluate the model performance. Validation of the model using new data allows for the best measure of the model’s true performance, or generalizability. Ideally this would proceed in a well-designed, adequately powered prospective randomized controlled trial with real-time use of the machine learning models during colonoscopy versus colonoscopy without AI.^25–27 An alternative strategy proposed is a random tandem randomized controlled trial, which would theoretically be better because each patient would serve as his or her own control group.

The only randomized controlled trial to date for CADe was performed in China, an open, nonblinded trial of diagnostic colonoscopy with or without assistance of a real-time polyp detection system running a convoluted neural network in real time. In the trial, 1058 patients were included with 536 controls and 522 with the AI intervention. The intervention arm demonstrated significantly improved adenoma detection rate (29.1% vs 20.3%, P<.001) and significantly higher withdrawal time (6.9 vs 6.4 minutes, P<.001) that became equivalent after excluding biopsy time (6.18 vs 6.07 minutes, P = .15). They reported a total of 39 false alarms in the intervention arm, which averaged to 0.075 false alarms per colonoscopy.

Despite carefully designed experiments to test model performance, in real-world application deep learning models may be limited by model bias and model interoperability stemming from the limitations of the training data.²⁸ For example, if the training data for a polyp detection model were predominantly in East Asian patients with a specific brand of endoscope, the model may not be perform well in a white-predominant population with a different brand.

In addition, the issues of safety, reliability, and demonstrated improvement from the standard of care are key issues that go beyond validation on new datasets. To reassure providers that the tool is both safe and reliable, parallel processes should be in place to identify and deal with errors.²² This emphasizes the importance of continuous maintenance and monitoring, which has been addressed by the Food and Drug Administration (FDA) in a recent whitepaper detailing the regulatory framework for updating software as medical devices.

MACHINE LEARNING PRODUCTS FACE REGULATORY, LEGAL, AND ETHICAL CHALLENGES

Regulatory rules vary from country to country, but are required before implementation of these algorithms for clinical care. In the United States, the FDA has drafted a framework for streamlined review under the term software as medical devices (SaMD).^29–31 This is defined by the International Medical Device Regulators Forum as “software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device.” The proposed risk-based regulatory framework for SaMD outlines categories of regulation based on severity of health care condition or situation (critical, serious, and nonserious) and significance of information provided by the software to the health care decision (for treatment or diagnosis, to drive clinical management, or to inform clinical management). This framework is adopted from the International Medical Device Regulators Forum and shared by the European Union Medical Device Regulation and Canadian Draft Guidance Document for Software as a Medical Device.³² The 3 risk classes are as follows: Class I (low risk), Class II (divided into IIA and IIB), and Class III (highest risk) with exemptions for classes I and II from 510(k) requirements either by specific FDA exemption or if they were legally marketed before the enactment of Medical Device Amendments on May 28, 1976. Classes I and II require a 510(k), which is a premarket submission that demonstrates that the device is at least as safe and effective as an existing legally marketed device. For Class III, a premarket approval application to demonstrate safety and effectiveness, which includes clinical trial data with study protocols, safety and effectiveness data, adverse reactions and complications, device failures and replacements, patient information, complaints, all subject data, and all statistical analyses (Table 3).

Table 3.

Software as medical device (SaMD) category risk-based classification

State of Health Care Situation or Condition	Significance of Information Provided by SaMD to Health Care Decision
	Treat or Diagnose	Drive Clinical/Patient Management	Inform Clinical/Patient Management
Critical	III	III	I or IIb
Serious	II or IIIa	II or IIIa	I or IIb
Nonserious	I or IIb	I or IIb	I or IIb

^aClass III if an erroneous result could lead to immediate danger.

^bClass II if the software is intended to image or monitor a physiologic process or condition. Class I under Rule 12.

Furthermore, manufacturers must manage patient risks throughout the entire product lifecycle, mainly to ensure safety and effectiveness. The updated strategy includes SaMD Prespecifications, or “what” the algorithm is intended to learn, and an algorithm change protocol, or “how” the algorithm will learn while maintaining safety and efficacy.³⁰ An example for colonoscopy is as follows:

1. Polyp Detection and Classification SaMD for Adenomatous Polyps

Description of SaMD: An AI/ML application intended for average risk patients presenting for screening and surveillance colonoscopy. The images from the colonoscopy video are processed and analyzed to detect polyps in the visual field throughout the colon. When an image is captured, it will be classified to be adenomatous or nonadenomatous with a percentage risk generated to indicate the probability of being adenomatous. This SaMD application will improve polyp detection and help guide decisions for low-risk polyps to be resected and discarded or left in.

SaMD Prespecifications:

• Modify the algorithm to ensure consistent performance across different levels of bowel preparation

• Reduce false alarm rates while maintaining sensitivity to polyps

Algorithm Change Protocol:

For these modifications, the algorithm change protocol includes detailed methods for database generation, reference standard labeling, and comparative analysis along the performance requirements, including sensitivity and specificity, and statistical analysis plan. The manufacturer follows good machine learning practices.

In Japan, this has been studied by the Pharmaceuticals and Medical Devices Agency.³³ Of note, Japan’s regulatory agency has recently approved the first AI-assisted endocytoscopy product was approved without a prospective randomized controlled trial.

Legal

The framework for legal responsibilities has not been established, and in a sensitive field such as medicine, the possibility of error in health care decisions made in part with the help of algorithms may expose providers to unanticipated liability (eg, for misdiagnosis of a low-risk polyp that becomes interval cancer).²⁴ Modifications to malpractice insurance, informed consent for the use of AI, and contingency plans should be in place to mitigate risk. Unfortunately, due to the absence of real-world use of AI in health care, there is limited precedent to draw on to provide guidance for patients, providers, or health care systems.

Ethics and Data Security

Respect for autonomy

AI applications in colonoscopy require a large volume of images and videos to develop, validate, and maintain performance. Because of the insatiable appetite for data, patient privacy and data security are key issues that should be thoughtfully addressed.

Patient privacy has traditionally been maintained through the system of individual patient consents, which has been argued to be prohibitively restrictive for purposes of AI research. A proposed solution is a “broad consent” policy, in which patients allow multiple secondary uses of their fully anonymized health data.³⁴

Data security covers both maintenance of secure databases and also ensuring secure sharing and transfer across institutions and organizations. Data breaches could undermine patient privacy and the patient-physician relationship. Secure cloud computing solutions have been proposed and developed by commercial entities and electronic health record companies, but data harmonization and interoperability across systems remains a particular challenge.

Nonmaleficence

Deployment of AI applications in colonoscopy could result in mistakes that lead to interval cancers or overutilization by performing unnecessary biopsies. High-risk polyps could be wrongly classified as benign, and polyp detection could either miss a cancerous lesion or result in additional unnecessary biopsies of benign lesions. Furthermore, the phenomenon of “automation bias,” in which providers trust AI decisions even if they are incorrect, may lead to overreliance on the technology and deskilling of the endoscopists.³⁵ Proper education and training of providers on the limitations of the AI technology is critical before widespread deployment. Accountability also should be considered in the event that a mistake is made when using the AI application, possibly titrated to the degree of autonomy accorded to the application.

Beneficence

To ensure that AI applications in colonoscopy act for the best interest of the patient, its recommendations should be taken into clinical context. One suggestion has been a new workflow that may incorporate the use of multidisciplinary meetings to review lesions characterized by the AI system and discuss the best approach for treatment.³⁴

Justice

AI applications in colonoscopy may concentrate the benefits in wealthier systems and have data bias that do not include patients from low-resource settings. The expense of computational infrastructure to store data and deploy AI applications for colonoscopy may lead to a disparity in treatment between wealthy and poorer health systems. In addition, patients in low-resource settings may not have their data incorporated into the AI applications, leading to underrepresentation that may lead to poorer performance even if deployed for their care. To promote widespread uptake, funding of prospective trials should include earmarked funds to provide a basic level of infrastructure for every center. Furthermore, careful validation of the AI applications in multiple settings is important to ensure that the performance is consistent for all patients.

DATA LABELING IS A KEY LIMITATION TO DEVELOPING AND MAINTAINING MACHINE LEARNING PRODUCTS

The greatest limitation for the current models being developed is the need for large volumes of labeled datasets, commonly known as “ground truth.” In the case of colonoscopy, the ground truth is either human annotation that the polyp is present, or pathologic diagnosis of a polyp adjudicated by multiple graders. In particular, deep learning algorithms for imaging require an enormous quantity of data to capture the variety of real-world images.³⁶ These include both static images (still pictures) and video recordings of colonoscopies, considering positive images, and quality of labeling. The labeling quality is considered to be the most important aspect, simply because models area data dependent and operate under a “garbage in, garbage out” framework and require expert endoscopist evaluation.²⁴ Current models used in colonoscopy are “supervised” machine learning models, which requires that all data used to train the models have had “ground truth” or labels attached to each of the datapoints. However, the expense and effort required to label images and video frames is prohibitive, particularly due to the need for specialized expertise.

FUTURE DIRECTIONS

Enhancements or additional algorithms could include automatic polyp sizing, evaluation of polyp borders that could be used to gauge the depth of invasion for large polyps, and clinical decision support to guide method of tumor removal.

Endoscopic tissue resection in particular is an emerging area in which the depth of tumor invasion in the esophagus, stomach and colon may help guide management decisions. Several deep learning–based CADx tools have been developed to detect very low risk lesions for esophageal squamous cell, early gastric, and colorectal cancer with similar criteria that have very low risk for lymph node spread.³⁷ In particular, computer-aided diagnosis platforms can help determine whether squamous cell carcinoma of the esophagus is classified as confined to the submucosa (SM1). SM1 lesions have very low risk for spread to lymph nodes and in the absence of high-risk features should undergo curative endoscopic resection.³⁸ Further work could provide real-time guidance that determines the extent of intervention needed (for example, endoscopic mucosal resection vs endoscopic submucosal dissection vs surgical intervention).

AI also has the potential to adjudicate when endoscopic assessment of diminutive colorectal polyps do not agree with pathologic diagnosis. A recent study provides preliminary evidence that an AI clinical decision support solution; for discordant diagnoses, the clinical decision support services provided additional analysis to suggest that endoscopic diagnoses were correct (90.3% of lesions).³⁹ The role for AI in clinical decision support to adjudicate discordant endoscopic and pathologic diagnosis could help future planning for surveillance colonoscopies.

The potential for misclassification rate for deep learning models in CADe exists with a limited understanding of the neural network architecture. Increased interpretability of neural architectures could be achieved through unsupervised machine learning techniques that examine the presence of important or redundant images. By understanding how the neural network classifies polyps, a misclassified polyp image could be compared to see if there is a deficiency in the dataset or architecture that may have accounted for the error. Another limitation to deep learning–based model is the bottleneck of labeled images. Emerging work in unsupervised machine learning techniques to create autoencoders that could be used to generate synthetic datasets for training deep learning models may augment existing datasets to improve model performance.

In the future, imaging data could be integrated in context with other biological and electronic health record data to create a more nuanced and personalized risk assessment. For patients at particular risk (eg, patients with inflammatory bowel disease), the integration of other clinical risk scores could be used to modify the threshold accordingly for polyp detection.

KEY POINTS.

• Artificial intelligence can improve colonoscopy quality by detecting adenomatous polyps and decrease associated costs by classifying low-risk polyps that do not need removal.

• An integrated system with artificial intelligence can improve the provider experience by decreasing documentation burden through automating data entry for quality measures.

• Important limitations of the technology must be recognized, including the dataset features for algorithm development and integration into endoscopist workflow.

• Future directions include automatic assessment of polyp size and borders, histologic staging and personalized risk stratification.