Deep learning and capsule endoscopy: automatic panendoscopic detection of protruding lesions

Abstract

Objective

Capsule endoscopy (CE) provides a minimally invasive exam modality for panendoscopic evaluation of the entire gastrointestinal (GI) tract. However, conventional reading methods can be time-consuming and error-prone. Protruding lesions are a relatively common entity that can be found with a variable incidence and different pathological significance throughout the GI tract. The aim of this study was to develop and test a convolutional neural network (CNN)-based algorithm for panendoscopic automatic detection of protruding lesions on CE exams.

Methods

A multicentric retrospective study was conducted, based on 1245 CE exams. We used a total of 191 455 frames, from six types of CE devices, of which 52 717 had protruding lesions (polyps, epithelial tumours or subepithelial lesions) after triple validation. Data were divided into a training and test set (90% vs 10%), in an exam-split design. During the training stage, we performed a fivefold cross-validation. Our outcome measures were sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (NPV), and areas under the conventional receiver operating characteristic curve (AUC-ROC) and the precision-recall curve (AUC-PR).

Results

In the test set, the sensitivity was 79.7% and the specificity was 96.5%. The PPV and NPV were 81.5% and 96.0%, respectively. The global accuracy was 93.7%.

Conclusion

This study aims to address a gap in artificial intelligence (AI)-enhanced capsule panendoscopy by reporting the development of the first CNN for the detection of protruding lesions across the GI tract. AI’s improvement of CE’s diagnostic accuracy, along with the growing interest in minimally invasive procedures, may contribute to increasing access to this diagnostic tool. Further multicentric and prospective studies are needed to validate our preliminary results to ultimately introduce deep learning models into clinical practice.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• artificial intelligence (AI) models have previously shown success in automatically detecting protruding lesions both in small bowel and colon but have not yet been applied effectively to proximal gastrointestinal (GI) tract topographies in a panendoscopic way.

WHAT THIS STUDY ADDS

• This study is the first to evaluate the use of AI deep learning models for the automatic detection of protruding lesions in a panendoscopic approach, extending the detection range across all GI topographies.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The results of this study emphasise the need for further development of these algorithms in order to make minimally invasive panendoscopy by capsule endoscopy a feasible option in the near future.

Introduction

The advent and widespread adoption of capsule endoscopy (CE) at the outset of this century marked the beginning of a new era for managing a wide spectrum of small bowel (SB) pathologies traditionally beyond the reach of conventional endoscopy.¹ Since then, the introduction of dual-camera capsules had enabled colonoscopy to be performed using colon capsule endoscopy (CCE), which has now become a valuable diagnostic tool in colorectal cancer screening, mainly in patients refusing or intolerant to conventional colonoscopy (CC).² Among the currently available non-invasive screening methods, CCE seems to be the most accepted, displaying a polyp detection rate equivalent to that of CC and superior to that of CT colonography.³,⁵ Moreover, CCE has a pivotal role in the panenteric assessment of patients with Crohn’s disease.^{6 7}

However, the role of CCE as an alternative to CC for diagnostic purposes continues to present some technical difficulties, greatly due to its demanding reading time, increasing physician’s fatigability and possibility of missing crucial frames for lesion identification. By introducing artificial intelligence (AI) as a video reading assistance, such disadvantages could be greatly mitigated, further supporting the smooth introduction of AI-assisted CE in daily clinical practice.⁸

Convolutional neural networks (CNNs) are deep learning algorithms modelled after the interconnected neural architecture of the human cortex that have been deeply transforming image pattern recognition.⁹ Extensive literature has been published using these AI algorithms in different image-based procedures, including CE. Indeed, there are already published Deep learning (DL) models for AI-enhanced CE automatic detection of protruding lesions (polyps, sub and epithelial lesions) both in SB and colon, presenting a high overall accuracy.^{10 11}

Nonetheless, since CE also opportunistically captures the proximal segments of oesophagus and stomach during its descending course, it would be useful to train AI algorithms to detect such lesions comprehensively across the entire gastrointestinal (GI) tract, similarly to what was previously described for vascular lesions.¹

The aim of this study was to develop and test a CNN-based algorithm for panendoscopic automatic detection of protruding lesions during CE exams.

Methods

Study design and setting

This retrospective cohort study was performed at two centres (Centro Hospitalar Universitário de São João and ManopH Gastroenterology Clinic) in Porto, Portugal. A total of 1245 CE and CCE videos from between June 2021 and August 2023 were extracted to constitute the patient cohort.

Patients were independently handled, as the study was developed without active intervention. In order to ensure patient identity protection, all personal data were excluded, and a random number allocation was implemented. As no personally identifiable information was collected, informed consent was waived by the Ethics Committee of Hospital São João. Data protection was further guaranteed in terms of non-traceability and adherence to the General Data Protection Regulation by a legal team certified as a Data Protection Officer (Maastricht University).

The reporting of this study is in accordance with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) guidelines. A complete TRIPOD checklist is provided in the online supplemental material.

CE protocol

Regarding CE procedures, six distinct CE devices were used: PillCam COLON 1 (Medtronic Corp., Minneapolis, Minnesota, USA), PillCam COLON 2 (Medtronic Corp., Minneapolis, Minnesota, USA), PillCam Crohn’s Capsule (Medtronic Corp., Minneapolis, Minnesota, USA), PillCam SB3 (Medtronic Corp., Minneapolis, Minnesota, USA), OMOM HD Capsule (JINSHAN, Yubei, Chongqing, China) and Olympus Endocapsule 10 (Olympus Corp., Tokyo, Japan). Images from PillCam Colon 1, PillCam Crohn’s and PillCam SB3 were analysed using PillCam Software V.9 (Medtronic, Minneapolis, Minnesota, USA), whereas OMON HD was examined with Vue Smart Software (Jinshan Science & Technology Co, Chonqing, Yubei, China) and Olympus with EC-10 System (Olympus).

Image processing techniques were employed to remove personal information (particularly name, procedure data and operation number) and preserve patient anonymity. A sequential number was then assigned to each frame.

Regarding bowel preparation, the protocol followed the recommendations by the European Society of Gastrointestinal Endoscopy.¹² A clear liquid diet was to be complied with on the day preceding capsule ingestion and refraining from eating was required the night before the exam. Before capsule ingestion, the patient completed bowel preparation, which entailed consuming 2 L of polyethylene glycol (PEG) solution. In the case of PillCam Crohn’s capsule, patients received an additional 2 L of PEG solution in the evening before the exam. Simethicone, an antifoaming agent, was systematically administered, and in case the capsule persisted in the stomach for over an hour, a prokinetic, namely domperidone 10 mg, was administered.

Extraction and categorisation of lesions

Regarding image extraction, CE videos were decomposed into individual frames at predefined intervals to avoid excessive redundancy among consecutive images. From this pool, a random selection of frames was made, ensuring a broad representation of cases, without prior filtering based on clinical diagnoses.

Each frame was then examined for the presence of protruding lesions (polyps and subepithelial lesions), allowing for image organisation into two categories: those containing normal mucosa and those with protruding lesions. A consensus between three experienced gastroenterologists in CE was required for the final inclusion of each frame.

CNN was developed using a total of 191 455 frames retrieved from six different types of CE devices, of which 52 717 had pleomorphic protruding lesions.

Development of the CNN and performance analysis

A deep learning CNN was constructed to automatically detect protruding lesions across the entire GI tract. Its construction involved a two-step process. Initially, 90% of the data set was used for a fivefold validation on the training set in order to evaluate the CNN’s overall performance. Second, the remaining 10% of the data set was allocated for testing with the average model derived from the previous five training sessions. The test set was used to analyse potential fluctuations in the CNN’s performance. A graphical flow chart of study design is represented in figure 1.

Figure 1. Flow chart illustrating the study design. AUC-PR, area under the precision-recall curve; AUC-ROC, area under the conventional receiver operating characteristic curve; CE, capsule endoscopy; CCE, colon capsule endoscopy; NPV, negative predictive value; PPV, positive predictive value.

The ResNet model served as the construct to our CNN. ImageNet, an image data set specially formulated for object software recognition, was used to train the weights between units in the network. To incorporate its learning into our model, we retained its convolutional layers, while removing the final fully connected layers of dense and dropout layers in our own classifier. Each of the two blocks we used was initiated by a fully connected layer. A dense layer was then added, the dimensions of which determined the number of classification groups (normal or protruding lesions). Trial and error guided the selection of the learning rate, batch size and the number of epochs.

PyTorch and scikit libraries were chosen to prepare the model. Standard augmentation techniques, such as image rotation and mirroring, took place during the training stage. Processing was performed on a system equipped with a dual NVIDIA Quadro RTX 80000 graphics processing unit (NVIDIA Corp., Santa Clara, California, USA) and a 2.1 GHz Intel Xeon Gold 6130 Central Processing Unit (CPU) (Intel, Santa Clara, California, USA).

The algorithm computed the probability of each frame being classified as normal or having a protruding lesion. Each frame was then assigned to the category with highest probability (figure 2). Heatmaps were generated to identify the features that had the greatest contribution to the model’s prediction (figure 3).

Figure 2. Each bar represents the algorithm’s estimated probabilities for both normal mucosa and the presence of a protruding lesion, with each frame being classified into one of the aforementioned categories based on the highest probability. The corresponding evaluation from the three experts, considered the gold standard, is shown in the lower right corner under the image.

Figure 3. Examples of heatmaps illustrating the algorithm’s process of identifying a protruding lesion. 1. Oesophagus; 2. Stomach; 3. Small bowel; 4. Colon.

The algorithm’s final output was compared against the gold standard, which entailed the correspondent consensus evaluation by three gastroenterologists with expertise in CE.

Training and validation data set

To evaluate the model’s overall diagnostic performance, we conducted a fivefold validation. We partitioned 90% of the total data set (n=174 670) into five folds of equal size. We employed the StratifiedGroupKFold approach for this division, with the aim of maintaining two key aspects within the cross-validation process. First, we ensured images from the same procedure were grouped together. Second, we prioritised a balanced distribution of lesions across each fold. Table 1 demonstrates the number of frames, patients, devices, regions (oesophagus, stomach, SB and colon) and protruding lesions contained in each fold. Online supplemental table 1 presents the number of frames, devices and protruding lesions across each GI segment. Online supplemental table 2 demonstrates the number of exams and corresponding number of frames for each device in each fold and the test set.

Table 1. Number of frames, patients, devices, regions (oesophagus, stomach, small bowel and colon) and protruding (PT) lesions presented within each fold and in the test set.

	Frames (n)	Patients (n)	Devices (n)	Regions (n)	PT lesions (n)
Fold 1	43 218	188	5	4	21 160
Fold 2	28 216	198	5	4	2890
Fold 3	45 943	280	4	4	16 397
Fold 4	32 036	268	5	4	4645
Fold 5	25 257	186	5	4	4865
Test set	16 785	125	4	4	2760

Then, we computed mean sensitivity, specificity, accuracy, positive predictive value (PPV) and negative predictive value (NPV) for each of the categories: normal mucosa and both protruding lesions P1 and P2. In addition to mean area under the conventional receiver operating characteristics (AUC-ROC) curve, we also calculated area under the precision-recall curve (AUC-PR), since the overrepresentation of normal mucosa (true negatives) compared with protruding lesions (true positives) could potentially lead to a misinterpretation of the ROC curve.

Test data set

In a subsequent phase, we evaluated the generalisability on a previously unseen subset of the data. The remaining 10% of the data set (n=16 785) was reserved for this independent test set. We ensured that the initial division of 90% vs 10% data followed an exam-split division, eliminating the possibility of encountering similar frames from the same procedure in both training/validation and testing data sets.

At this point, we calculated mean sensitivity, specificity, accuracy, NPV, PPV, and both AUC-ROC and AUC-PR for the aforementioned categories. For computational performance analysis, we measured the processing time of the entire test set. Statistical analysis was conducted using Sci-kit Learn V.0.22.2.

Results

A total of 1245 capsule exams were included for the development and testing of the CNN, from which 1038 correspond to SB CE (PillCam SB3, n=905; OMOM HD capsule, n=132; Olympus endocapsule, n=1) and 208 to devices allowing colon CE (PillCam Crohn’s capsule, n=184; PillCam COLON 1, n=17; PillCam COLON 2, n=7). A total of 191 455 images were retrieved from these exams and ultimately validated and incorporated into the data set. From those, 52 717 images contained protruding lesions (polyps, epithelial tumours or subepithelial lesions).

Training set

Table 2 displays the results obtained for each of the categories on the fivefold validation process and testing set. Regarding the overall performance for protruding lesions detection (normal vs protruding lesions), the mean sensitivity was 72.2% (95% CI 57.3% to 87.1%) and mean specificity was 97.6% (95% CI 96.4% to 98.8%). The mean PPV and NPV were 89.1% (95% CI 82.3% to 95.6%) and 93.6% (95% CI 91.8% to 95.3%), respectively. Mean accuracy was 93.3% (95% CI 91.8% to 94.7%). The mean AUC-ROC was 0.94 (95% CI 90.3 to 96.9) while mean AUC-PR was 0.85 (95% CI 75.1 to 95.7) (online supplemental figure 1).

Table 2. Summary of results.

	Sensitivity, %	Specificity, %	PPV, %	NPV, %	Accuracy, %	AUC-ROC	AUC-PR
Fold 1	94.8	95.1	95.1	95.1	95.1	0.98	0.98
Fold 2	60.7	98.3	80.2	95.6	94.4	0.89	0.72
Fold 3	85.8	97.8	95.5	92.6	93.5	0.97	0.97
Fold 4	63.0	97.7	82.0	94.0	92.6	0.92	0.77
Fold 5	56.9	98.9	92.5	90.6	90.8	0.92	0.83
Test set	79.7	96.5	81.5	96.0	93.7	–	–

AUC-PR, area under the precision-recall curve; AUC-ROC, area under the conventional receiver operating characteristic curve; NPV, negative predictive value; PPV, positive predictive value.

Test set

The testing data set comprised 10% of the frames from the full data set. The CNN’s sensitivity and specificity were 79.7% and 96.4%, respectively. The PPV and NPV were 81.5% and 96.0%, respectively. The model’s overall accuracy was 93.7%.

The performance of the model was further analysed by the GI segment. Table 3 presents the confusion matrix for the test set, stratified by segment, showing the distribution of correctly and incorrectly classified frames as either normal or protruding lesions. Overall, the model demonstrated robust performance across all segments, with particularly high detection rates in the stomach and colon.

Table 3. Confusion matrix of the test set results, stratified by the gastrointestinal segment.

Actual	Predicted
	Normal	Protruding
Oesophagus
Normal	201	7
Protruding	16	28
Stomach
Normal	0	0
Protruding	8	380
Small bowel
Normal	4196	97
Protruding	394	1356
Colon
Normal	9130	394
Protruding	142	436

The table compares actual and predicted classifications (normal vs protruding lesions) across the oesophagus, stomach, small bowel and colon. Values represent the number of frames per class.

A more detailed breakdown of sensitivity, specificity and accuracy per GI segment is provided in table 4. Notably, the model achieved the highest sensitivity and accuracy in the stomach segment (97.9%), although specificity could not be calculated due to the absence of negative cases. Performance in the oesophagus was more modest in terms of sensitivity (63.6%) but still maintained high specificity (96.6%) and accuracy (90.9%).

Table 4. Per-segment performance metrics of the model on the test set, including sensitivity, specificity and accuracy.

	Sensitivity, %	Specificity, %	Accuracy, %
Oesophagus	63.6	96.6%	90.9
Stomach	97.9	NA	97.9
Small bowel	77.5	97.7	91.9
Colon	75.4	95.9	94.7

Metrics are reported separately for the oesophagus, stomach, small bowel and colon. ‘NA’ indicates that specificity could not be calculated due to the absence of negative cases in the stomach segment.

Two major reasons for the model’s incorrect predictions were the existence of large air bubbles/presence of an air-water interface and inadequate cleansing during CE (online supplemental figure 2).

Discussion

The present study is the first, to our knowledge, to evaluate the application of AI deep learning models in automatic detection of protruding lesions in a panendoscopic way, extending the lesion detection range to each GI topography. This model solves a crucial technological interoperability barrier by achieving high diagnostic accuracy across several device brands, allowing its implementation in a multitude of technological platforms.

The development of AI-enhanced algorithms capable of detecting and diagnosing lesions is increasing exponentially in various gastroenterology fields. The need for developing this type of technology is demonstrated in recent guidelines, with the primary goal of being supportive rather than substitutive. One of the most mature applications of AI in Gastroenterology is in colonoscopy practice, in which there is some evidence indicating that the use of AI software increases both the adenoma detection rate and the number of adenomas per procedure, independently of the size and morphology of the lesion.^{13 14} Nevertheless, it remains an invasive procedure with non-despicable risks.

CCE presents a well-tolerated, minimally invasive option for colorectal cancer screening with minimal contraindications, making it a valuable tool to consider. Its high tolerability suggests potential for increased patient adherence, which could have a favourable influence on the health of the population. Nonetheless, limitations exist as bowel preparation protocols are rigorous and, more importantly, interpreting the findings remains time-consuming. Moreover, while CE allows for the assessment of the entire digestive tract (therefore often referred to as capsule panendoscopy (CPE)), interpreting its findings takes even greater time and is more susceptible to errors. Therefore, for both CCE and CPE to attain cost-effectiveness, the incorporation of AI software is likely necessary, assuming adequate bowel cleansing is ensured.

Protruding lesions are a relatively common entity that can be found throughout the GI tract, with a variable incidence and malignant potential according to digestive segment. Their automatic detection using AI models has been the aim of extensive research in recent years, particularly in SB and colon. Indeed, there are published deep learning methods that can automatically detect protruding lesions and also provide a morphological assessment regarding their bleeding potential, in the aforementioned regions. Specifically, the colon is the segment that has gathered most attention from researchers with the development of AI models with overall good diagnostic performance not only for polyps’ detection,^{15 16} but also stages of colorectal neoplasia.¹⁶ While there are existing models for detecting protruding lesions in the SB, colon and both regions combined, these models were only trained using data from these regions.¹⁷,¹⁹ Consequently, we do not know how these models would perform in the presence of a protruding lesion in proximal GI tract areas.

To the best of our knowledge, there are no studies of AI-assisted diagnosis systems for the detection of protruding lesions in a true panendoscopic evaluation, encompassing the entire digestive tract mucosa rather than just the SB and colon. This could hold special significance in populations with an increased incidence of both gastric and colorectal cancers, such as ours, as a single exam could potentially screen for both pathologies, expediting the diagnosis and improving treatment strategies of these patients.

Our results reveal distinct performance profiles of the model across different GI segments, reflecting the anatomical and visual heterogeneity of the GI tract. Notably, the highest sensitivity and accuracy were observed in the stomach and colon, where protruding lesions tend to be more prominent and morphologically consistent. In contrast, performance in the oesophagus was lower, particularly in terms of sensitivity. This is not unexpected, as the oesophageal segment poses some inherent challenges: the transit time through the oesophagus is typically faster, which reduces the number of high-quality frames available for analysis. Moreover, technical precautions that can improve image acquisition in this segment—such as positioning the patient in a recumbent position—were likely not consistently applied during the procedures included in our data set. These factors may have contributed to the reduced performance in this segment. Overall, while a pan-GI model appears feasible, these findings underscore the need for segment-specific considerations in both data acquisition and model development to ensure consistent performance across all regions of the GI tract.

This study has some highlights that deserve to be mentioned. First, the study’s exam-split design ensured that frames from each exam were assigned to a single fold both in the cross-validation experiment and in the subsequent phase of global performance assessment, thereby reducing the risk of overfitting. Overfitting is an undesirable consequence that arises when the model is too tuned for the training data and unable to generalise to all possible input data. Not opting for an exam-split design would have increased the probability of finding similar images on both data sets, therefore increasing the chances of this phenomenon occurring and potentially compromising the study’s external validity. Second, the model consistently exhibited notable diagnostic performance metrics throughout the fivefold cross-validation experiment, demonstrating its consistency across different patient and device distributions, further supporting its external validity. This underscores its potential for widespread use in clinical settings and suggests that its efficacy remains unaffected by the specific CE device used. The successful development of this deep learning brand-spanning algorithm (five in total) is a notable achievement, not only because those are rare at the moment, but also because it overcomes an interoperability challenge that has hindered the technological readiness level (TRL) of CE-related AI algorithms. Additionally, there was a concern to also present the number of protruding lesions across each GI segment to understand the contribution of each region to the overall distribution of protruding lesions and its impact on diagnostic performance. Findable, Accessible, Interoperable and Reusable (FAIR) principles were followed in the development of this model. Each patient was assigned a unique and confidential identifier, ensuring data traceability while protecting privacy (findable). Study researchers could access individual video frames for analysis, complying with data confidentiality and Institutional Review Board (IRB) regulations (accessible). The model incorporated data from six different CE and CCE devices, enhancing its generalisability across various technologies (interoperable). A portion of data was from a previous data set, and we plan to use the entire data set for future research, maximising its long-term value (reusable). Lastly, the findings of this study are reported in accordance with the TRIPOD guidelines, to ensure transparency, methodological rigour and reproducibility.

Nevertheless, the study has some drawbacks. The retrospective nature of the study could introduce a selection bias as the study sample may not accurately represent the entire population to which the study is intended. Second, the study only evaluated the performance of the CNN at the still frame level and, despite achieving good performance results, it may not ensure equivalent performances with full-length videos. Therefore, further multicentric and prospective studies, with a higher statistical value, are deemed to validate the clinical significance of our results and properly introduce deep learning models in clinical practice. Third, in a subjective assessment of frames where our model generated incorrect projections, it became evident that suboptimal bowel cleansing quality with presence of faeces or air bubbles hindered the CNN’s performance. Ensuring a proper level of bowel cleansing quality is deemed to instil confidence in physicians regarding both anomaly detection and, more crucially, its exclusion during CCE exams. Fourth, the absence of patient’s clinical information due to General Data Protection Regulation (GDPR) restrictions may also introduce selection bias. Nevertheless, the value of this study remains as a proof of concept with a lower TRL, which primarily focuses on the technical feasibility of the proposed method rather than detailed clinical characterisation by region or lesion. Finally, for the stomach topogrphy, only frames displaying protruding lesions were included in the analysis, with no inclusion of normal (negative) frames from the same region. This selective inclusion may have led to an overestimation of the model’s diagnostic performance, as it does not fully reflect real-world conditions where both normal and abnormal findings coexist. For a more reliable and generalisable validation of AI tools in clinical practice, future studies should be carefully designed to include a representative balance of both positive and negative cases.

Conclusion

The concept of a minimally invasive panendoscopy by CE is still in its early stages, yet it has already generated great enthusiasm over the advance towards the ultimate goal of applying AI in CE: enabling the simultaneous detection of different lesions while performing a single exam of the entire GI tract in a highly accurate and cost-effective manner. The convergence of both an enhanced diagnostic efficacy offered by AI in CE and an increased interest in minimally invasive procedures by today’s society holds the potential to improve the performance and further access to this diagnostic modality.

Further multicentric and prospective studies are needed to validate our preliminary results to ultimately introduce deep learning models into clinical practice.

Data availability statement

No data are available.