A deep learning–based system to identify originating mural layer of upper gastrointestinal submucosal tumors under EUS

ABSTRACT

Background and Objective

EUS is the most accurate procedure to determine the originating mural layer and subsequently select the treatment of submucosal tumors (SMTs). However, it requires superb technical and cognitive skills. In this study, we propose a system named SMT Master to determine the originating mural layer of SMTs under EUS.

Materials and Methods

We developed 3 models: deep convolutional neural network (DCNN) 1 for lesion segmentation, DCNN2 for mural layer segmentation, and DCNN3 for the originating mural layer classification. A total of 2721 EUS images from 201 patients were used to train the 3 models. We validated our model internally and externally using 283 images from 26 patients and 172 images from 26 patients, respectively. We applied 368 images from 30 patients for the man-machine contest and used 30 video clips to test the originating mural layer classification.

Results

In the originating mural layer classification task, DCNN3 achieved a classification accuracy of 84.43% and 80.68% at internal and external validations, respectively. In the video test, the accuracy was 80.00%. DCNN1 achieved Dice coefficients of 0.956 and 0.776 for lesion segmentation at internal and external validations, respectively, whereas DCNN2 achieved Dice coefficients of 0.820 and 0.740 at internal and external validations, respectively. The system achieved 90.00% accuracy in classification, which is comparable with that of EUS experts.

Conclusions

Our proposed system has the potential to solve difficulties in determining the originating mural layer of SMTs in EUS procedures, which relieves the EUS learning pressure of physicians.

INTRODUCTION

Submucosal tumors (SMTs) represent a class of protruding lesions that can arise from any layer of the gastrointestinal (GI) tract wall.^[1,2] Compared with computed tomography and magnetic resonance imaging,^[3] EUS is considered the most accurate procedure for detecting and diagnosing SMTs owing to its unique ability to examine tumors with high proximity.^[4–9] Besides, EUS is the only imaging technique that can delineate the separate histologic layers of the GI tract.^[10,11]

Recently, several endoscopic techniques have been proven useful in managing SMTs. Based on the experts' opinions and published case reports, when it comes to tumors arising from mucosal and submucosal layers, the procedures that can be performed include endoscopic mucosal resection (EMR), ligation device–assisted EMR,^[12–17] transparent cap-assisted EMR,^[18–22] and endoscopic submucosal dissection (ESD). As for tumors originating from muscularis propria, feasible procedures include endoscopic submucosal enucleation,^[23,24] ESD,^[25–30] endoscopic full-thickness resection,^[31,32] endoscopic submucosal tunneling dissection,^[33–36] and endoscopic ligation.^[37–39] Therefore, it is essential to achieve an accurate identification of the originating mural layer of SMTs to select the endoscopic treatment.

However, EUS is an operator-dependent, technically challenging procedure that requires unique technical and cognitive skills. The accuracy of EUS in identifying mural layers among different endoscopists has been reported to range from approximately 59% to 94%.^[40–50] This varying ability of endoscopists to distinguish mural layers leads to frequent distractors of the follow-up clinical decision, seriously impacting the effect of diagnosis and treatment. Ideally, a computer-aided mural layer annotation system could improve the ability of ultrasonic endoscopists to define the originating mural layer in SMTs.

Recent years brought a tremendous advancement of artificial intelligence (AI) in the medical field.^[51] AI and EUS have been successfully combined for the identification of SMTs, and the feasibility of AI-assisted diagnosis of SMTs on ultrasound images has been preliminarily confirmed. However, these studies have been limited to the nature of SMTs and did not involve the recognition of mural layers.^[52–54] Selecting follow-up endoscopic treatment methods for SMTs is still an unsolved challenge.

In this work, we constructed a deep learning–based system, called SMT Master, to annotate mural layers and identify the originating layer of upper GI SMTs under EUS. This system was evaluated with internal and external validations on images and subsequently compared with the performance of EUS endoscopists. To the best of our knowledge, this is the first study using deep learning to distinguish mural layers and achieve mucosal visualization and the first study to define the originating mural layer of SMTs. It could be a valuable assistance in the diagnosis and treatment of SMTs and relieve the learning pressure of ultrasonic endoscopists.

MATERIALS AND METHODS

System framework

We included 3 deep convolutional neural network (DCNNs) models into the SMT Master system to achieve 2 main functions. The first was the segmentation module, which could annotate the regions of the lesions and the mural layers adjacent to them and contained 2 DCNN models: DCNN1 to segment SMTs and DCNN2 to segment the mural layers of the upper GI tract wall. The second was the classification module, DCNN3, to output and classify the originating mural layer of the observed SMTs.

Two models of the first function, especially DCNN2, were constructed to enhance the performance of the classification model, DCNN3. In particular, after the original EUS images went through DCNN1 and DCNN2, the system edited the EUS images centered on the lesion. It made 2 tangents of the lesion starting from the central point of the EUS image in white and extending for 45 degrees to the left and right sides, separately, to make 2 auxiliary lines in red (Supplementary Figure S1, http://links.lww.com/ENUS/A344). Finally, the EUS images with labeling of the lesions and mural layers within a certain range around the lesions were provided as input to DCNN3. Based on the marker information of the previous module, DCNN3 recognized the originating mural layers of SMTs (Supplementary Figure S2, http://links.lww.com/ENUS/A345). The workflow chart of SMT Master is shown in Figure 1.

Figure 1.

The framework of BP MASTER. DCNN1 was applied to segment SMTs. DCNN2 was applied to segment mural layers. DCNN was applied to classify the originating mural layers of SMTs. DCNN: deep convolutional neural network; SMT, submucosal tumor.

Data sets, data classification, and sample distribution

A total of 2721 images from 201 EUS procedures from Wuhan Renmin Hospital during January 2019 to June 2021 were used to train the 3 models (DCNN1–3). The average age of patients was 53.8 years (SD of 11.7 years), and 69 were men (34.3%). For internal validation, 283 images from 26 EUS procedures from Renmin Hospital of Wuhan University during July 2021 to November 2021 were used. The average age was 58.0 years (SD of 10.8 years), and 11 were men (42.3%). Finally, a testing data set containing 172 images from 26 examinations of Wuhan Union Hospital was collected for external validation. The average age was 53.7 years (SD of 9.6 years), and 12 were men (46.2%). The internal and external test sets contained 97 and 69 “transition zone” images, respectively. All selected patients had previously undergone endoscopic resection, and the lesions were confirmed to be SMTs according to definite pathological findings.

Two experts were invited to mark the images, by marking the outline of the lesion and the recognizable mural layers on the lesion side. To distinguish the 5 mural layers, they marked them with different colors. Two experts classified the originating mural layer of SMT in each case according to the ESD procedure images and pathological results of the corresponding lesion. We only included EUS images from the dissenting cases if they were rediscussed by 2 experts and a unified conclusion was reached, which was then approved by 2 other experts.

All 3 models were trained and validated using the same set of images from the same group of patients. EUS images with artificial labels were used to train the DCNN1 and DCNN2 models, whereas EUS images marked by segmentation modules were used to train and test DCNN3.

A total of 368 images from 30 EUS procedures in Renmin Hospital of Wuhan University during July 2021 to December 2021 were used to compare the performance of DCNN3 with that of EUS experts (man-machine contest). The average age of the patients was 54.7 years (SD of 11.0 years), and 11 were men (36.7%). For video test, 30 video clips from 30 EUS procedures from Renmin Hospital of Wuhan University during July 2021 to November 2021 were used. The average age of patients was 56.8 years (SD of 10.8 years), and 13 were men (43.3%). These video clips included the whole process of SMT scanning, including clips before SMTs appeared in view, while scanning and after these disappeared from view; that is, there are so-called transition zones.

The sample distribution for each data set is shown in Table 1. Images from the same person were not split among the data sets. The procedures were performed using Olympus EU-ME1 and EU-ME2 (Olympus Medical Systems Co, Tokyo, Japan) processors and adapted endoscopes.

Table 1.

Sample distribution.

	Training	Testing
		Internal test data set	External test data set	Man-machine contest set	Video test data set
Patient (n)	201	26	26	30	30
Age (SD), y	53.8 (11.7)	58.0 (10.8)	53.7 (9.6)	54.7 (11.0)	56.8 (10.8)
Male, n (%)	69 (34.3)	11 (42.3)	12 (46.2)	11 (36.7)	13 (43.3)
Pathological diagnosis, n (%)
GIST	101 (50.25)	11 (42.31)	10 (38.46)	15 (50.00)	8 (26.67)
Leiomyoma	62 (30.85)	14 (53.85)	12 (46.15)	10 (33.33)	16 (53.33)
Aberrant pancreas	22 (10.95)	0 (0.00)	0 (0.00)	3 (10.00)	2 (6.67)
Lipoma	8 (3.98)	0 (0.00)	1 (3.85)	1 (3.33)	1 (3.33)
NET	4 (1.99)	1 (3.85)	3 (11.54)	1 (3.33)	3 (10.00)
Lymphangioma	2 (0.10)	0 (0.00)	0 (0.00)	0 (0.00)	0 (0.00)
Glomus tumor	2 (0.10)	0 (0.00)	0 (0.00)	0 (0.00)	0 (0.00)
Image no.	2721	283	172	368

SD: standard deviation, GIST: gastrointestinal stromal tumor, NET: neuroendocrine tumor.

Training of the DCNN models

We used UNet++ for image segmentation and ResNet for image classification. Both networks were trained on a NIVIDIA GeForce GTX 2080. The technical details and neural network architecture are illustrated in supplementary materials, http://links.lww.com/ENUS/A346. To train DCNN3, we used ResNet-50, which is an amateur DCNN architecture pretrained on data from ImageNet (1.28 million images from 1000 object classes). The final classification layer was replaced with another fully connected layer using transfer learning and retrained using our data sets, and the parameters were fine-tuned to fit our needs. The data set was randomly divided into 5 subsets, and each subset was individually validated with the remaining for training in Google's Tensor Flow.^[55] Three methods were used to minimize the overfitting risk: dropout,^[56] data augmentation,^[57] and early stopping.^[58] We implemented UNet++, a novel and robust architecture for medical image segmentation, for lesions and mural layers annotation to develop DCNN1 and DCNN2.^[59,60] Using the original EUS image as input with the resolution of 256 × 256 and the expert-marked map as output, UNet++ was used to train and test DCNN1 and DCNN2 in an image-to-image manner in Keras. Based on the result of internal validation, we obtained the best lesion segmentation and the first to fifth mural layer segmentation thresholds by increasing 2 each time, and the threshold was set as 50, 30, 50, 30, 50, and 20 (Supplementary Figure 3, http://links.lww.com/ENUS/A347).

Man-machine contest

To evaluate the difference in the accuracy between SMT Master and endoscopists in determining the originating mural layer of SMTs, we chose 368 EUS images from 30 patients as a test data set. The endoscopists participating in this evaluation were split into 3 groups according to their experience performing EUS: 4 novices (<1 year), 3 seniors (1–3 years), and 2 experts (>3 years). Then, we asked them to carefully examine all the images and come up with a result of the SMT originating mural layer for each case. Because the choice of endoscopic treatment depended on whether SMT was located at layers 2 and 3 or layer 4, we conducted statistical analysis in 2 ways: method A, to calculate the accuracy of classifying layer 2 or layer 3, or layer 4, and method B, to estimate the accuracy of classifying layers 2 and 3 or layer 4.

Statistical analysis

To evaluate the classification of the originating mural layer, we used accuracy as a metric, which is defined as the number of correctly classified images divided by the total number of images. To evaluate the segmentation, including lesion segmentation and mural layers segmentation, intersection over union (IoU) was used, defined as the relative area of overlap between the predicted bounding box (A) and ground-truth (B) bounding box, which was labeled by the experts. IOU could be calculated as follows:

IoU = $\frac{\left|\mathrm{A\cap }\mathrm{B}\right|}{\left|\mathrm{A|\cup }\right|\left.\mathrm{B}\right|}$

If IoU>threshold, the prediction is true positive, whereas if IoU<threshold, the prediction is false positive. If the model segmentation area = 0, it is false negative. We also used the Dice coefficient, precision and recall, defined as follows, respectively:

$$\text{Dice}\left(\text{F1 score}\right)=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

$$\text{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

$$\text{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

Interobserver and intraobserver agreements of the endoscopists were appraised using Cohen κ coefficient. All analyses were performed using the IBM SPSS software (version 20; IBM Corp, Armonk, NY).

Human subjects

This study was approved by the Renmin Hospital Ethics Committee of Wuhan University, and the participating endoscopists provided informed consent.

Role of the funding source

The funder had no role in the study design, data collection, data analysis, data interpretation, or report writing. The corresponding author had full access to all the data in the study and was responsible for the decision to submit the work for publication.

RESULTS

Segmentation performance

The DCNN1 model achieved Dice of 0.956 in the internal testing data set, with precision and recall at 50% IoU of 92.5% and 98.9%, respectively. In the external testing data set, the Dice was 0.776, with precision and recall at 50% IoU of 67.3% and 91.7%, respectively [Table 2]. For the second, third, and fourth layers, the DCNN2 model achieved Dice values of 0.871, 0.750, and 0.840 in the internal testing data set and 0.727, 0.710, and 0.784 in the external data set, respectively [Table 3].

Table 2.

DCNN1 segmentation performance.

Internal test data set			External test data set
Dice	Precision at 50% IoU	Recall at 50% IoU	Dice	Precision at 50% IoU	Recall at 50% IoU
95.6	92.5	98.9	77.6	67.3	91.7

All results are given as a percentage.

DCNN: deep convolutional neural network; IoU: intersection over union.

Table 3.

DCNN2 segmentation performance

		Internal test data set			External test data set
	Dice	Precision at 50% IoU	Recall at 50% IoU	Dice	Precision at 50% IoU	Recall at 50% IoU
Layer 2	87.1	78.7	97.4	72.7	61.5	88.9
Layer 3	75.0	69.2	81.8	71.0	57.9	91.7
Layer 4	84.0	72.4	100.0	78.4	64.4	100.0
Average	82.0	73.4	93.1	74.0	61.3	93.5

All results are given as a percentage.

DCNN: deep convolutional neural network; IoU: intersection over union.

Classification performance

Table 4 shows the results of DCNN3 for classifying the 3 mural layers. It achieved an accuracy of 84.4% in the internal testing data set and 80.7% in the external one, and the accuracy was 80.0% in the video test. For the transition zone images in the internal and external test sets, DCNN3 achieved an accuracy of 82.5% and 81.2%, respectively. In addition, it achieved an accuracy of 80.0% when tested in the internal test set of images without mural layers marked by DCNN2, which was lower than the test set with the participation of both DCNN1 and DCNN2 (Table S1, http://links.lww.com/ENUS/A346).

Table 4.

DCNN3 accuracy of classifying originating mural layer in internal and external test as well as the video test

	Internal Test	External Test	Video Test
Layer 2	87.3	80.7	77.8
Layer 3	91.6	77.3	66.7
Layer 4	81.5	81.4	86.7
Total	84.4	80.7	80.0

All results are given as a percentage.

DCNN: deep convolutional neural network.

Man-machine contest

Compared with the SMT Master performance, the endoscopists' performance was slightly inferior. In the testing data set for the man-machine contest, DCNN3 correctly classified the originating mural layer with an accuracy of 90.0% using both methods A and B. Meanwhile, the 2 expert, 3 senior, and 4 novice endoscopists obtained accuracy values of 83.3%, 77.8%, and 60.0% using method A and 88.3%, 84.5%, and 67.5% using method B, respectively. There were no differences among the experienced endoscopists [Table 5, Figure 2]. Thus, SMT Master showed higher accuracy than all endoscopists. The interobserver agreement between DCNN3 and the experts is shown in Table S2, http://links.lww.com/ENUS/A346.

Table 5.

Performance of SMT Master and endoscopists in judging the accuracy of originating mural layer of SMTs

	SMT Master	Experts (n = 2)	Seniors (n = 3)	Novices (n = 4)
Layer 2 accuracy	83.3	83.3	77.8	70.8
Layer 3 accuracy	100.0	70.0	60.0	45.0
Layer 4 accuracy	89.5	86.8	82.5	60.5
Total accuracy in method A	90.0	83.3	77.8	60.0
Total accuracy in method B	90.0	88.3	84.5	67.5

All results are given as a percentage.

DCNN: deep convolutional neural network.

Figure 2.

The accuracy in the man-machine contest. A, The accuracy of SMT Master and endoscopists of method A, method B, layer 2, layer 3, and layer 4. B, The accuracy of SMT Master and endoscopists of method A and method B. SMT, submucosal tumor

DISCUSSION

In this study, we developed an AI system named SMT Master to segment SMTs and surrounding mural layers of the upper GI tract with an excellent Dice and distinguish the originating mural layer of SMTs with high accuracy in retrospectively images and videos. We evaluated the system in both internal and external validations, and it achieved high performance. In the man-machine competition, the accuracy of SMT Master in classifying originating mural layers even exceeded the average of expert physicians. This system could serve as a valuable tool to assist the diagnosis and treatment of SMTs and reduce the pressure on EUS performers and learners.

Compared with computed tomography and magnetic resonance imaging, EUS is the first choice for the detection and diagnosis of SMTs in the upper GI tract.^[61] Before treating SMTs, it is necessary to use EUS to detect the location of SMTs in the GI wall.^[8] Endosonographically, the upper GI tract wall comprises 5 layers of alternating echogenicity. The first layer represents the superficial layer of the mucosa, and it is hyperechoic. The second layer constitutes the deep layer of the mucosa, including the muscularis mucosae, and it is hypoechoic. The third layer is called submucosa, and it is hyperechoic. The fourth layer is hypoechoic, called the muscularis propria, and the fifth layer is hyperechoic and called the serosa/adventitia.^[62,63] By determining the location, especially the originating mural layer of SMT, EUS endoscopists can indicate the appropriate type of endoscopic resection. For example, when SMTs are limited to the muscularis mucosa or submucosa, then the standard snare polypectomy, strip biopsy, endoscopic submucosal resection with a ligation device, and endoscopic SMT resection with a transparent cap methods are valid options. On the other hand, sessile SMTs that extend into the proper muscle layer can be removed by endoscopic enucleation using a snare, a cutting knife, or an insulated-tip electrosurgical knife.^[5]

Although EUS is a powerful tool to diagnose SMTs, most endoscopists still cannot use it properly because of its steep learning curve and overdependence on the operator.^[64] Different endoscopists have different abilities to identify mural layers, which was also reflected in the results of the man-machine competition in our study. Previous studies have reported the accuracy of EUS for the identification of mural layers to range from approximately 59% to 94%.^[40–50] This is a crucial clinical problem that has negatively impacted the diagnosis and treatment of patients.

Recent years have witnessed medical integration with AI and SMT-related research using AI. Minoda et al.^[54] developed a system named EUS-AI, which can make a differential diagnosis of gastrointestinal stromal tumors (GISTs) from non-GIST for subepithelial lesions ≥20 mm with an accuracy of 86.3%. The work of Seven et al.^[53] reported that a deep learning algorithm could predict the malignant potential of gastric GISTs on EUS images with an accuracy of 99.6%. However, previous studies have been limited to the nature of SMTs.

In this study, we developed 2 functions: one to mark SMTs and the surrounding mural layers and one to indicate the originating mural layer. The system provides direct and indirect help for ultrasonic endoscopists to determine the location of SMTs. Endoscopists can agree with the AI's results or make other judgments based on the markers, eliminating the problem that using AI limits the physician's ability to judge. Our results showed that the accuracy of DCNN3 in classifying the EUS images with DCNN1 and DCNN2 markers was higher than that of images with only DCNN1 markers (Supplementary Figure S1, http://links.lww.com/ENUS/A344). The segmentation and labeling of mural layers by DCNN2 enhance the SMT location-related features in EUS images, thus enabling the performance of the DCNN3 classification model. Marking the local features of images first and then classifying them can overcome the existing technical bottleneck and provide a better choice for designing other medical AI schemes.

In the man-machine competition, the classification accuracy of DCNN3 was slightly higher than that of expert endoscopists (90.0% vs. 83.3%, respectively). Experts will generally judge the origin level of lesions according to the continuity of the mural layer, location relationship between lesions and mural layers, presence or absence of trumpet-shaped openings in the mural layer, or other features. Besides, the similar accuracy in classification to the experts implied that the model itself could also summarize the characteristics, showing the same superiority of classifying the originating mural layer of SMTs in radial EUS images. The application of SMT Master in clinical practice should be verified in prospective clinical trials in the future.

The accuracy of the external test set was not as good as the internal test. The reason might be that the parameters of the radial EUS in Wuhan Union Medical College Hospital were slightly different from ours, and the image collection style of the endoscopists in various hospitals was somewhat different, which resulted in differences in the images display. Nevertheless, SMT Master still achieved high accuracy in the external test set, which indicates that it can be applied among different hospitals.

Our study has several limitations to consider. First, although high accuracy was obtained in the external verification, the model was developed from cases and obtained at a single center. Future multicenter data collection will be necessary to enhance the generalization of our models. Second, deep learning model training using video has been proven to be practicable.^[65] Although the image training–based model has achieved excellent performance in both image and video validation, the extra benefit of using video for mode training is worth to be explore in the future.

CONCLUSIONS

In conclusion, we constructed a system named SMT Master to effectively solve the problematic identification of mural layers under EUS. Our system can assist ultrasonic endoscopy physicians with the diagnosis and treatment of SMTs.

Supplementary Material

eusj-12-465-s001.docx ^{(110.1KB, docx)}

eusj-12-465-s002.docx ^{(17.3KB, docx)}

Funding

This work was supported by the Innovation Team Project of Health Commission of Hubei Province (grant no. WJ202C003) (to Honggang Yu).

Conflicts of interest

None.

Author Contributions

All authors contributed to the study conception and design. The original idea was generated by Jun Zhang and Liwen Yao. Material preparation, data collection, and analysis were performed by Xun Li, Chenxia Zhang, and Kun Zhang. The first draft of the manuscript was written by Xun Li, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.