A ubiquitous and interoperable deep learning model for automatic detection of pleomorphic gastroesophageal lesions

Abstract

In recent years, artificial intelligence (AI) has been widely explored to enhance capsule endoscopy (CE), with the goal of improving the efficiency of the reading process. While most AI models have been developed for small bowel and colon analysis, the development of esophagogastric (E-G) models has been limited due to the scarcity of frames captured during the procedure, making it challenging to develop a robust model. This study aims to develop an interoperable ubiquitous model capable detecting pleomorphic lesions in the E-G tract. We included 59,482 E-G frames, from 774 CE procedures of 5 centers, to develop a Convolutional Neural Network (CNN). The dataset was divided following an exam-based split, with 90% allocated for training – including a 5-fold cross validation – while the remaining was used for testing. The primary outcomes were: sensitivity, specificity, accuracy and area under the curve (AUC). During training, the CNN achieved mean sensitivity of 85.0% (IC95% 76.5–93.5), specificity of 96.3% (IC95% 93.9–98.7), accuracy of 93.6% (IC95% 91.7–95.4), and AUC-ROC of 0.98 (IC95% 0.97–0.98). During testing, the sensitivity, specificity and accuracy of CNN were 92.2%, 95.1% and 94.6%, respectively. CE-AI models capable of assessing E-G are crucial step toward developing a truly robust panendoscopic model. This ubiquitous and interoperable model capable of lesion detection in both esophagus and stomach, achieved good overall accuracy. However, prospective real-world studies comparing its performance with standard upper endoscopy are still needed to validate its clinical applicability.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-03397-7.

Introduction

Capsule endoscopy (CE) was designed as minimally invasive method for endoscopic evaluation of the small bowel. It is now established as the first line procedure for assessing potential small bowel involvement. The two most common indications include excluding mid-gastrointestinal bleeding in cases of overt GI bleeding or iron-deficiency anemia and diagnosis or monitoring patients with suspected or confirmed small bowel involvement in Crohn’s disease¹.

Over time, CE devices have evolved from single-camera to dual-camera capsules, significantly expanding the field of view, which was critical to increase its diagnostic yield in the colon. Consequently, with the introduction of dual-camera capsules, CE has not only facilitated the assessment of colonic involvement in Crohn’s disease but also emerged as an additional diagnostic tool for colorectal cancer screening¹.

In fact, CE is the only endoscopic exam capable of assessing the entire gastrointestinal (GI) tract, from the esophagus and stomach, to the small bowel and colon – all in a single examination. The concept of capsule panendoscopy (CPE) is gaining increasing attention, as its minimally invasive nature could offer significant advantages not only in emergent settings (e.g. GI bleeding), but also as a screening tool (e.g. GI tract neoplasia screening using a pill-sized device)^2,3.

Despite its growing role in GI tract assessment, CE has several inherent limitations that must be acknowledge. One of its primary constrains is the inability to optimize imaging conditions, perform insufflation or carry out therapeutic interventions. Additionally, its reliance on manual interpretation makes the process the time-intensive and error-prone⁴. Furthermore, when it comes to esophagogastric (E-G) evaluation, two key aspects should be considered. On one hand, esophageal imaging is challenged by rapid transit time, which reduces the number of esophageal captured frames per procedure. On another hand, the inability to control CE’s movements can result in incomplete visualization of certain areas, particularly the proximal gastric region.

Nonetheless, given the minimally invasiveness nature and excellent tolerability for most patients, CE could also play a significant role in upper GI tract assessment, especially when considering its potential within the context of panendoscopy. This is supported by current literature which emphasize its diagnosing esophageal conditions such as esophagitis, Barrett’s esophagus and E-G varices^5–7. Moreover, CE has been used to detect gastric lesions, including ulcers, erosions and polyps, and may even aid in assessing helicobacter pylori infection status^8–10.

The development of artificial intelligence models in CE is increasing exponentially, driven by the potential to significantly reduce reading-time by focusing only frames with a high probability of containing lesions. This could have significant impact not only on small bowel assessment but even more so if applied to panenteric or panendoscopic evaluation of the entire GI tract. Although most AI models have been developed for small bowel and colon analysis, the development of E-G models has been limited due to the scarcity of frames captured during the procedure, making it challenging to develop a robust model^11–13. This underscores the need for a ubiquitous model capable of accurately detecting lesions not only in the esophagus, but also in the stomach. Expanding detection capabilities in these regions comprehensively would significantly increase the available data for model training, ultimately accelerating improvements in model performance and accuracy.

The aim of the study is to develop an interoperable ubiquitous model capable detecting pleomorphic lesions in the E-G tract.

Methods

In this multicentric study, we included data from 5 centers (São João University Hospital [Porto, Portugal], ManopH Gastroenterology Clinic [Porto, Portugal], Nossa Senhora de Oliveira Hospital [Guimarães, Portugal], Puerta del Hierro University Hospital [Madrid, Spain] and La Princesa University Hospital [Madrid, Spain]). A total of 774 of CE procedures, performed between June 2021 and May 2023, were collected and reviewed.

CE procedures in all centers adhered to recommendations of the European Society of Gastrointestinal Endoscopy. Patients followed a clear liquid diet the day before the procedure, fasted overnight, and underwent bowel preparation with 2 L of polyethylene glycol (PEG) before ingesting single-headed capsules. For two-headed capsules, the regimen included 2 L of PEG the night before, followed by an additional 2 L of PEG prior to ingestion. CE data collection did not have any implication on patient management. Pseudonymization of personal data assured to be compliant with General Data Protection Regulation (GDPR), minimizing data traceability.

Image data from both one and two-headed capsule devices was collected, including PillCam™ SB3 (Medtronic Co., USA), PillCam™ Crohn’s Capsule (Medtronic Co., USA), OMOM^® HD Capsule (Jinshan Co., China) and Olympus Endocapsule 10 (Olympus Corp., Japan). We processed the videos using each device specific software. From the total set of images in each case, we included still frames specifically from the esophageal and gastric assessment of GI tract.

Two CE expert physicians independently reviewed esophageal and gastric frames from each procedure. The frames were classified as either normal or containing a clinically relevant lesion (vascular lesions, ulcers and erosions, protruding lesions and hematic residues). Only frames where both physicians reached the same classification were included in the final dataset. The final dataset contained 59,482 frames, including 11,925 containing lesions, which were used to develop an AI model.

We used a XCiT (Cross-Covariance Image Transformer) to develop a model of automatic detection and classification of E-G CE still frames into two categories: normal vs. clinically relevant lesion. The dataset was divided in two groups (Fig. 1): training/validation (90% procedures) and testing (10% procedures), following an exam-based split (frames from the same procedure were only included in one group, ensuring they were never included in both). The training/validation phase was used to assess robustness of the model and to fine-tune hyperparameters, while the testing phase was used to independently test the diagnostic performance of the model. The training/validation phase included a 5-fold cross validation, where the designated subset of dataset was divided in five equal folds. The dataset was processed 5 times, each run using a different conjugation of training and validation folds (4 for training, 1 for validation, different combinations of CE procedures) and different fine hyperparameters. The model that achieved the highest F1-score during 5-fold cross validation phase was then selected and evaluated on the independent test set.

Fig. 1.

The study design involves an exam-based split, dividing data into training/validation and testing sets.

First, we initialize the XCIT-Tiny-24-P8-224 with pretrained weighted images from Image Net¹⁴. We preserved the convolutional layers to transfer their learned features to our model, except for final layers, which were modified the classified head by replacing them with a fully connected layers tailored for binary classification (normal vs. lesion). We also incorporated drop out layers (0.2) between fully connected layers to reduce overfitting. Key hyperparameter were set including a learning rate of 0.0001 and a batch size of 64, while optimizing the number of epochs through experimentation. We also applied image rotation and rotation during the training phase. The training was conducted using a high-performance computing setup, including a NVIDIA RTX A6000 graphic processing unit (NVIDIA Corp, Santa Clara, CA, USA) and dual AMD EPYC 7282 16-Core processor (AMD, Santa Clara, CA, USA)¹⁵.

For each frame, the AI model calculated a probability of being classified as normal or lesion. The classification with highest probability was considered the AI’s predicted outcome. Each AI-predicted outcome was evaluated against expert classification and categorized as true positive, true negative, false positive and false negative, which was used to calculate the primary outcomes (Fig. 2): sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (NPV). Additionally, we also calculated the Receiver Operating Characteristic (AUC-ROC) curve to assess global discriminatory capacity of the model. These metrics were evaluated both during the 5-fold cross validation in training/validation phase and in the testing phase. Perfomance metrics of our model were compared with common Non-ViT Architectures (ResNet10, EfficientNet, ConvXNet, DenseNet, MobileNet). The 5-fold cross-validation splits for the non-ViT architectures were generated using a common fixed random seed. We used Sci-kit Learn v0.22.2 to perform statistical analysis¹⁵.

Fig. 2.

Each frame is assessed by two CE expert physicians, comparing them to model’s prediction. AI predicted outcome are classified as true positive, true negative, false positive and false negative, forming the basis for primary outcome calculations.

Results

A total of 774 CE exams were utilized for training and testing this model. From these exams, a total of 59,482 frames were extracted, of which 11,925 contained lesions. Table 1 describes the dataset distribution across different stagings of AI model development and validation. The dataset included 96 PillCam Crohn’s, 578 PillCam SB3, 69 OMON, and 31 Olympus CE procedures. Of the total dataset, 8317 were from the esophagus, while the remaining 51,165 were from the stomach.

Table 1.

Dataset distribution of frames, CE procedures and CE devices across folds and between training or testing.

	Categories	Training/validation					Testing
		Fold 1	Fold 2	Fold 3	Fold 4	Fold 5
Frames	Total (n)	13,794	14,235	8017	6063	12,477	4896
	Lesions (n)	2422	2971	3056	810	1835	831
	Normal (n)	11,372	11,264	4961	5253	10,642	4065
	PillCam Crohn’s (n)	523	1236	1052	289	532	420
	PillCam SB3 (n)	4612	2032	4815	1820	3982	3079
	OMON (n)	990	2788	13	1229	2297	93
	Olympus (n)	7669	8179	2137	2725	5666	1304
Procedures	Total (n)	145	132	140	141	138	78
	PillCam Crohn’s (n)	18	16	17	21	16	8
	PillCam SB3 (n)	110	94	112	101	100	61
	OMON (n)	10	17	7	14	16	5
	Olympus (n)	7	5	4	5	6	4

Training set

Regarding the training set, Table 2 presents the results obtained from the 5-fold cross validation. In terms of overall performance for detecting E-G lesions the CNN demonstrated a mean sensitivity of 85.0% (IC95% 76.5–93.5), a mean specificity of 96.3% (IC95% 93.9–98.7), a mean PPV of 83.6% (IC95% 69.7–97.6), a mean NPV of 95.5% (IC95% 90.5–100.0) and a mean accuracy of 93.6% (IC95% 91.7–95.4). Additionally, the mean AUC-ROC was 0.98 (IC95% 0.97–0.98), as represented in Fig. 3. Supplementary Table 1 benchmarks the performance metric of our model against common Non-ViT Architectures.

Table 2.

Five-fold cross validation conducted during training validation.

	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Accuracy (%)	AUC-ROC	AUC-PR
Fold 1	92.7	95.8	82.3	98.4	95.2	0,98	0,92
Fold 2	75.6	98.9	94.7	93.9	94.0	0,98	0,95
Fold 3	80.7	97.7	95.6	89.1	91.2	0,97	0,96
Fold 4	89.3	95.1	73.6	98.3	94.3	0,98	0,9
Fold 5	86.8	94.1	71.9	97.6	93.1	0,98	0,9

Fig. 3.

For 5-fold cross-validation, the area under the receiver operating characteristic (AUC-ROC) curve is shown for each fold, with the average curve highlighted in blue.

Testing set

For the testing set, which accounted for 10% of patients in the dataset, the model achieved a sensitivity of 92.2%, a specificity of 95.1%, a PPV of 79.3%, an NPV of 98.3%, and an overall accuracy of 94.6%.

Discussion

The present study is one of the first studies to develop and validate an AI model for the detection of both esophageal and gastric lesions using CE, representing a significant step toward our ultimate goal—making the concept of capsule panendoscopy more feasible and robust in clinical practice.

Detecting E-G lesions remains a complex challenge. While some studies have explored AI applications in this area, research is still in its early stages, making direct comparisons difficult. However, findings from other domains suggest that Vision Transformer (ViT) models offer distinct advantages, particularly in terms of enhanced efficiency in generating predictions^16,17. This advantage is particularly relevant in CE, where reducing reading time can make AI-assisted analysis faster and more cost-effective.

This is a proof-of-concept model, considering its ubiquitous nature, that was developed and validated to detect various types clinically relevant lesions in both the esophagus and stomach. The limited availability of E-G frames per procedure, particularly for the esophagus, difficult the development of robust AI models. To partially mitigate this limitation, we can address this is by developing ubiquitous and interoperable models like this AI model, which improve performance confidence by increasing data training, not only by increasing the absolute data volume but also by adding heterogeneity and population diversity. The inclusion of different high-volume CE centers, as well as, different CE devices, approximates the adherence to FAIR principles, promoting greater accessibility and interoperability, key factors in bridging technology development to real clinical practice.

Moreover, incorporating a variant of a Vision Transformer (ViT) model – traditionally associated in natural language processing tasks but increasingly explored for classification and detection of complex visual patterns – could be a promising technological pathway. Given its cross-variation attention mechanism, this type of AI model has the potential to be more efficient, reducing computational burden and costs¹⁴. Benchmarking its performance with other common non-ViT deep learning models (Supplementary Table 1), suggests a more robust performance, particularly when assessing the cross-validation phase. This efficiency could be particularly relevant in clinical setting, particularly when aiming to develop a ubiquitous model, which, in turn, could play a key role in advancing AI-enhanced CPE.

However, it is important to highlight the limitations inherent to this study. First, this AI model was developed solely for lesion detection and lacks the ability to differentiate between lesions. This differentiation capacity is more challenging to develop given the number of total frames. We should also acknowledge that the dataset is not fully balanced, both in the proportion of esophageal versus gastric frames and in the distribution of normal versus lesion-containing frames, increasing the risk of overfitting of the model. Moreover, due to lack of access to clinical data, we cannot fully characterize the population from which these CE procedures originate. The GPRD restrictions, further limit our ability to obtain patient-specific information, neither to correlate with findings in conventional upper endoscopy. All these considerations affect the generalizability of the results. Nonetheless, we believe this AI model is valuable to share and should be viewed as a proof-of-concept since it is the first to leverage its ubiquitous nature to enhance model’s overall robustness and efficiency, further strengthen by the use of transformer model.

As far as the impact of AI in detection of gastric lesions during CE goes, early AI models focused on magnetic controlled capsule endoscopy (MCE). These AI models were first developed using CE still images (sensitivity 96%, specificity 76%) and later advanced to real-time detection (sensitivity 99%)^18,19. While MCE offers the advantage of controlling capsule movements, unlike conventional wireless capsule endoscopy (WCE), its accessibility remains limited. Therefore, the AI application in MCE may not lead to a disruptive breakthrough, given that MCE remains a specialized technique with limited adoption. Conversely, AI models designed for WCE have also demonstrated good performance metrics in detecting gastric lesions (sensitivity 97%, specificity 96%), offering not only promising accuracy but also greater reproducibility to wide variety of clinical contexts²⁰.

Currently, there are published AI models than can detect lesions in the stomach and small bowel²¹. Moreover, panendoscopic AI-enhanced detection models have already been developed and published, capable of detecting lesions not only in the small bowel and colon, but also in the stomach and esophagus (in the esophagus: sensitivity 76%, specificity 86%; in the stomach: sensitivity 97%, specificity 96%; in the small bowel: sensitivity 78%, specificity 98%; in the colon: sensitivity 86%, specificity 94%)²². However, a potential limitation of these models is that their dependence on separate, sequential regional AI systems, rather than a unique ubiquitous approach.

Our model was capable of detecting lesions in esophageal and gastric still CE frames with sensitivity of 92% and specificity of 95%, showing similar performance to the average metrics obtained during the five-fold cross validation. This consistency suggest that the model is robust, regardless of frame or patient distribution, ensuring interoperability and broader clinical applicability.

In conclusion, CE has revolutionized minimally invasive gastrointestinal imaging, initially focusing on the small intestine but later expanding to the esophagus and stomach. Despite its advantages, CE faces challenges in upper GI assessment, such as limited esophageal frames and the absence of therapeutic capabilities. AI has shown promise in improving lesion detection, but its application to the upper GI tract remains limited due to dataset constraints.

This study represents a significant milestone as the first multicenter investigation to develop and validate an AI model for detecting esophageal and gastric lesions using CE, with great performance metrics. The inclusion of multiple CE devices and multicentric datasets improves real-world applicability. Future efforts should focus on prospective validation in larger, more diverse cohorts. By doing so, this technology may pave the way for capsule panendoscopy to become a viable, minimally invasive alternative for comprehensive gastrointestinal assessment.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

M.M and M.J.M: equal contribution in study design, image extraction, drafting of the manuscript, and critical revision of the manuscript.T.R, JA.: bibliographic review, image extraction, critical revision of the manuscript.P.C, F.M, M.J.A, J.M: bibliographic review, drafting of the manuscript, critical revision of the manuscript.JF: construction and development of the VIT model, statistical analysis, critical revision of the manuscript.P.A, H.C, M.M.S, G.M: study design, critical revision of the manuscript.All authors approved the final version of the manuscript.

Funding

The authors recognize NVIDIA support by providing the graphic units used in this study.

Data availability

Raw data were generated at the Faculty of Medicine of the University of Porto, PT. Derived data supporting the findings of this study are available from the corresponding author upon request.

Declarations

Statement of ethics

Ethical committee permission was acquired prior to initiation of the study by the ethics committee of Centro Hospitalar Universitário São João (No. CE 407/2020) and conducted in accordance with declaration of Helsinki. The informed consent requirement was waived by the ethics committee of Centro Hospitalar Universitário São João for this retrospective study.

Competing interests

The authors declare no competing interests.