Artificial intelligence-based diagnosis of standard endoscopic ultrasonography scanning sites in the biliopancreatic system: a multicenter retrospective study

Abstract

Background:

There are challenges for beginners to identify standard biliopancreatic system anatomical sites on endoscopic ultrasonography (EUS) images. Therefore, the authors aimed to develop a convolutional neural network (CNN)-based model to identify standard biliopancreatic system anatomical sites on EUS images.

Methods:

The standard anatomical structures of the gastric and duodenal regions observed by EUS was divided into 14 sites. The authors used 6230 EUS images with standard anatomical sites selected from 1812 patients to train the CNN model, and then tested its diagnostic performance both in internal and external validations. Internal validation set tests were performed on 1569 EUS images of 47 patients from two centers. Externally validated datasets were retrospectively collected from 16 centers, and finally 131 patients with 85 322 EUS images were included. In the external validation, all EUS images were read by CNN model, beginners, and experts, respectively. The final decision made by the experts was considered as the gold standard, and the diagnostic performance between CNN model and beginners were compared.

Results:

In the internal test cohort, the accuracy of CNN model was 92.1–100.0% for 14 standard anatomical sites. In the external test cohort, the sensitivity and specificity of CNN model were 89.45–99.92% and 93.35–99.79%, respectively. Compared with beginners, CNN model had higher sensitivity and specificity for 11 sites, and was in good agreement with the experts (Kappa values 0.84–0.98).

Conclusions:

The authors developed a CNN-based model to automatically identify standard anatomical sites on EUS images with excellent diagnostic performance, which may serve as a potentially powerful auxiliary tool in future clinical practice.

Introduction

Highlights

• Endoscopic ultrasonography (EUS) is an essential diagnostic tool for biliopancreatic diseases, there are challenges for beginners to identify standard biliopancreatic system anatomical sites on EUS images.

• Artificial intelligence (AI) has excellent performance in gastrointestinal images recognition, but there are few studies on the use of AI on the recognition of images from standard anatomical sites of the biliopancreatic system.

• We have developed an AI model that can automatically identify standard anatomic sites on EUS images with excellent diagnostic performance and can be used as a potentially powerful auxiliary tool in future clinical practice.

Definitive diagnosis of diseases of the biliopancreatic system (BPS) is difficult because of the complex anatomy^1,2 and wide varieties of diseases^3,4. Linear-array endoscopic ultrasonography (EUS) allows clear visualization of the anatomical structures of the BPS, and EUS-guided puncture can obtain pathological data to help distinguish benign from malignant lesions^5–7. Moreover, EUS-guided puncture can play further therapeutic roles, such as bile drainage, drug injection, and even hemostasis⁸.

EUS requires the endoscopist to be very familiar with the anatomy of the BPS, and endoscopist need to gain a certain amount of experience in operating EUS devices before they can perform this task according to the relevant recommendations^9,10. However, endoscopist have major difficulties in mastering the skills to identify standard BPS anatomical sites on EUS images^11,12. Therefore, there is an urgent need to resolve this inability to identify standard BPS sites in the field of vision.

In recent years, deep convolutional neural networks (CNNs) have been applied in the recognition of digestive system diseases using examination images. Artificial intelligence (AI) is a valuable and practical technology for identifying benign and malignant liver lesions, staging liver fibrosis^13,14, and performing real-time detection in gastrointestinal endoscopy, aiding the operator in finding sites of inflammation, polyps, ulcers, tumors, etc., without blind spots^15,16. Few studies have been published on the recognition of images from standard anatomical sites of the BPS by AI. In this study, we explored and validated a novel computer-aided diagnosis system with a CNN model for analyzing linear-array EUS standard scan images of the BPS to help operators locate ‘home base’ anatomical landmarks^17,18.

The aim of this study was to investigate whether a high-performance, automatic, standard BPS anatomical site detection system could be developed for future training and quality control.

Methods

Study design

In this study, we developed an CNN model to identify standard BPS anatomical sites on EUS images and validated its performance in internal and external validation sets. In the training phase and the internal validation, data were collected retrospectively from two centers. In external validation, we retrospectively collected data from 16 other centers. All the patients included in this study had no biliopancreatic disease and had normal biliary pancreatic system structure under EUS. Patients with any of the following conditions will be excluded: patients with pancreatic and/or bile duct lesions; patients who have undergone surgical treatment that result in changes in the pancreatic and/or bile duct anatomy; patients with pancreatic and/or bile duct anatomical changes due to lesions of other organs in the chest and/or abdomen. Patient data were anonymized, and any personally identifying information was omitted. The work has been reported in line with the strengthening the reporting of cohort, cross-sectional and case–control studies in surgery (STROCSS) criteria¹⁹ (Supplemental Digital Content 1, http://links.lww.com/JS9/B543).

The anatomical structures of the gastric and duodenal regions observed by EUS was divided into 14 sites, as shown in Figure 1. Study flow charts of the training and validation phases were shown in Figure 2. In the training phase, 6230 EUS images were selected from 66 508 images of 1812 patients to construct the CNN model. The number of EUS images used in the training phase at each anatomical site was shown in Supplementary Table 1 (Supplemental Digital Content 2, http://links.lww.com/JS9/B544).

Figure 1.

Standard structures observed from each scanning site.

Figure 2.

Study flow charts of the training and validation phases.

In the validation phase, we included both internal and external validation. A total of 1569 standard site images were obtained from 47 patients from two centers were internally tested, and 85 322 EUS images (80 488 images with standard anatomical sites, 4834 images without standard anatomical sites) collected from 131 patients from 16 other centers were externally tested (Supplementary Table 1, Supplemental Digital Content 2, http://links.lww.com/JS9/B544). During the external validation, all EUS images were read by CNN model, beginner and expert respectively. Beginners are defined as those who have more than 1 year of experience in gastroenteroscopy, but do not have EUS experience or training. Prior to the start of the study, beginners were required to complete reading 30 typical images of each standard anatomical site in advance during the first month of EUS training. The expert panel was composed of three experts who had performed EUS for more than 10 years. The diagnosis results of the beginner group were made by discussions among the beginners. The final decision made by the experts was considered as the gold standard, and the diagnostic performance between CNN model and beginners were compared.

Training

In the training phase, we trained the CNN model to identify standard anatomical sites on EUS image. A total of 66 508 BPS linear EUS images were collected from 1812 patients examined between January 2015 and October 2019 at two medical centers. All the EUS Images were acquired using Olympus Aloka, EU-ME1 and EU-ME2 processors (Olympus Medical Systems Co.), Fujinon SU-9000 processors (Fujinon Toshiba ES Systems Co., Ltd.) and adapted endoscopes.

In the CNN-based model, 6230 EUS images with standard anatomical sites were selected from 66 508 EUS images (up to five EUS images from each standard anatomical site were selected into the training set for each patient, and up to five images of different standard anatomical sites of each patient were collected for model training), and fivefold stratified cross-validation was used for training. Since EUS images are usually greyscale and mainly contain information on the morphological structure of the imaged parts of the body, we adopted deep CNNs with an attention module to better recognize these morphological structures within the EUS images. First, we cut off the frame of the images, keeping only the effective information part of the images, and grayed out 192 images with color Doppler. Then, the training set was divided into five groups; four were used for training, and the remaining group was used for validation. In each iteration, the model with the best validation accuracy was kept. The k-fold cross-validation method was used for training (k-value was taken as 5), the learning rate attenuation method was used to tune the model, an Adam optimizer was used for model training, and the loss function used focal loss. The training data were augmented, and brightness and contrast enhancement methods were used. Hyperparameters: the initial learning rate was 0.01, the batch size was 16, and the learning rate decay was set to 10 times every 10 epochs. Finally, classification models for analyzing linear endoscopic EUS images in the gastric region and duodenal region were obtained (Fig. 3A), The residual learning structure of the CNN-based system is shown in Figure 3B.

Figure 3.

CNN-based algorithm reading model. A. Data flow from left to right: EUS images are subsequently processed with the CNN model, and the recognition result was output. B. A detailed description of the attention block in A. The squares of F1… Fn represent the feature tensor extracted by the deep learning model, + represents the tensor addition operation, and × represents the tensor product operation.

Validation

This study included both internal and external validation. In internal validation, 1569 EUS images with standard anatomical sites selected by experts from 47 patients from two centers were used to test the diagnostic performance of the CNN model. At least 100 EUS images were selected for each site. In external validation, a total of 85 322 EUS images from 131 patients from 16 other centers were read by beginners, CNN model, and experts, respectively. Using expert diagnosis as the gold standard, and the diagnostic performance between CNN model and beginners were compared.

Statistical analysis

All relevant data were entered into a customized database and then analyzed with SPSS software, version 21.0 (IBM). The sensitivity, specificity, Youden index, positive likelihood ratio, positive predictive value, Kappa value and confusion matrix were used to evaluate the diagnostic performance. The χ²-test was used to analyze the differences in the sensitivity, specificity, Youden index, positive likelihood ratio, positive predictive value, consistency rate, and Kappa value between groups. An independent-sample t-test was used to compare the total time spent to evaluate the EUS images among beginners, experts and the CNN model. Sensitivity, specificity, positive predictive values, and negative predictive values are described as percentages with 95% CIs. A value of P<0.05 was considered statistically significant.

Results

Internal validation

In the internal validation, a total of 1569 EUS images with standard anatomical sites were processed using the AI model. The internal validation results showed that the accuracy of the AI model was 99.2% for liver and hepatic vein, 98.3% for abdominal aorta, 97.3% for pancreatic body and the splenic artery and vein, 99.1% for pancreatic body and left kidney, 99.2% for left adrenal gland, 100% for splenic hilum and spleen, 99.0% for confluence of the portal intestine, superior mesangial vein and splenic vein, 97.5% for first hepatic portal, 97.2% for branch of portal vein, 92.1% for gallbladder, 98.4% for portal vein and bile duct, 99.2% for pancreatic head, 94.3% for confluence of the portal intestine, superior mesangial vein and splenic vein, and 95.9% for papilla (Fig. 4).

Figure 4.

Confusion matrix in internal validation.

High sensitivity of AI model in recognition of standard EUS anatomical sites in external validation

The sensitivity of the AI model were 94.90% (95% CI: 93.85–95.81) for liver and hepatic vein, 99.58% (95% CI: 99.39–99.71) for abdominal aorta, 97.23% (95% CI: 96.93–97.51) for pancreatic body and the splenic artery and vein, 96.34% (95% CI: 95.78–96.83) for pancreatic body and left kidney, 98.97% (95% CI: 98.68–99.20) for left adrenal gland, 90.40% (95% CI: 89.57–91.17) for splenic hilum and spleen, 96.15% (95% CI: 95.71–96.54) for confluence of the portal intestine, superior mesangial vein, and splenic vein (gastric region), 89.70% (95% CI: 88.60–90.71) for first hepatic portal, 96.40% (95% CI: 95.77–96.93) for branch of portal vein, 93.56% (95% CI: 92.57–94.42) for gallbladder, 97.13% (95% CI: 96.69–97.50) for portal vein and bile duct, 99.92% (95% CI: 99.77–99.97) for pancreatic head, 98.28% (95% CI: 97.94–98.56) for confluence of the portal intestine, superior mesangial vein, and splenic vein (duodenal region), and 89.45% (95% CI: 88.78–90.07) for papilla, respectively (Table 1). And the AI model also showed high specificity, ranging from 93.35% (95% CI: 93.04–93.66) to 99.79% (95% CI: 99.75–99.83) (Table 1). In addition, there was high agreement between CNN model and experts for 14 sites (Kappa scores of 0.84 to 0.98) (Table 1).

Table 1.

Diagnostic performance of the CNN model in the external validation.

Region	Scanning position	Sensitivity % (95% CI)	Specificity % (95% CI)	Youden index	Positive likelihood ratio	Positive predictive value	Consistency rate %	Kappa value
Stomach	1	94.90 (93.85–95.81)	99.65 (99.60–99.70)	0.95	274.21	0.91	99.48	0.93
	2	99.58 (99.39–99.71)	98.37 (98.25–98.48)	0.98	61.06	0.90	98.52	0.94
	3	97.23 (96.93–97.51)	98.38 (98.26–98.50)	0.96	60.08	0.95	98.13	0.95
	4	96.34 (95.78–96.83)	99.79 (99.75–99.83)	0.96	469.87	0.98	99.48	0.97
	5	98.97 (98.68–99.20)	99.61 (99.55–99.66)	0.99	254.83	0.97	99.54	0.98
	6	90.40 (89.57–91.17)	99.43 (99.36–99.49)	0.90	157.69	0.94	98.56	0.92
	7	96.15 (95.71–96.54)	98.22 (98.10–98.34)	0.94	54.15	0.91	97.90	0.92
	8	89.70 (88.60–90.71)	99.70 (99.65–99.74)	0.89	299.79	0.95	99.11	0.92
	9	96.40 (95.77–96.93)	98.62 (98.51–98.72)	0.95	69.80	0.85	98.45	0.89
Duodenal	10	93.56 (92.57–94.42)	97.60 (97.42–97.77)	0.91	39.01	0.97	97.26	0.84
	11	97.13 (96.69–97.50)	96.07 (95.82–96.30)	0.93	24.71	0.96	96.29	0.89
	12	99.92 (99.77–99.97)	98.30 (98.14–98.45)	0.98	58.85	0.98	98.50	0.93
	13	98.28 (97.94–98.56)	93.35 (93.04–93.66)	0.92	14.79	0.94	94.43	0.85
	14	89.45 (88.78–90.07)	97.43 (97.22–97.63)	0.87	34.86	0.95	95.23	0.88

The sensitivity of the beginners were 82.53% for liver and hepatic vein, 89.87% for abdominal aorta, 76.65% for pancreatic body and the splenic artery and vein, 96.49% for pancreatic body and left kidney, 76.23% for left adrenal gland, 92.75% for splenic hilum and spleen, 87.98% for confluence of the portal intestine, superior mesangial vein and splenic vein (gastric region), 86.05% for first hepatic portal, 47.79% for branch of portal vein, 92.52% for gallbladder, 77.61% for portal vein and bile duct, 80.16% for pancreatic head, 86.72% for confluence of the portal intestine, superior mesangial vein and splenic vein (duodenal region), and 72.01% for papilla, respectively (Table 2). The specificities of the beginners ranged from 98.18 to 99.98%. There was no significant difference between AI model and beginners in the recognition of left kidney, spleen, and gallbladder sites (P>0.05), but the sensitivity of AI model to the other 11 sites was significantly higher than that of beginners (P<0.01).

Table 2.

Sensitivity, specificity comparison in recognition of standard BPS anatomical sites on EUS images between beginners and the CNN model.

		Beginners		CNN model
Region	Scanning position	Sensitivity, % (95% CI)	Specificity, % (95% CI)	Sensitivity, % (95% CI)	Specificity, % (95% CI)
Stomach	1	82.53 (80.77–84.15)	99.98 (99.96–99.99)	94.90 (93.85–95.81)	99.65 (99.60–99.70)
	2	89.87 (89.12–90.57)	99.96 (99.93–99.97)	99.58 (99.39–99.71)	98.37 (98.25–98.48)
	3	76.65 (75.88–77.39)	99.67 (99.61–99.72)	97.23 (96.93–97.51)	98.38 (98.26–98.50)
	4	96.49 (95.94–96.97)	99.90 (99.87–99.93)	96.34 (95.78–96.83)	99.79 (99.75–99.83)
	5	76.23 (75.12–77.31)	99.96 (99.94–99.98)	98.97 (98.68–99.20)	99.61 (99.55–99.66)
	6	92.75 (92.01–93.43)	99.22 (99.14–99.29)	90.40 (89.57–91.17)	99.43 (99.36–99.49)
	7	87.98 (87.27–88.66)	99.53 (99.47–99.59)	96.15 (95.71–96.54)	98.22 (98.10–98.34)
	8	86.05 (84.81–87.21)	98.18 (98.06–98.29)	89.70 (88.60–90.71)	99.70 (99.65–99.74)
	9	47.79 (46.25–49.34)	99.25 (99.17–99.32)	96.40 (95.77–96.93)	98.62 (98.51–98.72)
Duodenal	10	92.52 (90.41–92.51)	99.92 (99.88–99.94)	93.56 (92.57–94.42)	97.60 (97.42–97.77)
	11	77.61 (76.53–78.60)	98.38 (98.22–98.53)	97.13 (96.69–97.50)	96.07 (95.82–96.30)
	12	80.16 (68.68–81.39)	99.44 (99.35–99.52)	99.92 (99.77–99.97)	98.30 (98.14–98.45)
	13	86.72 (85.90–87.50)	99.76 (99.65–99.78)	98.28 (97.94–98.56)	93.35 (93.04–93.66)
	14	72.01 (71.06–72.95)	99.69 (99.61–99.75)	89.45 (88.78–90.07)	97.43 (97.22–97.63)

BPS, biliary and pancreatic system; CNN, convolutional neural networks.

High time-efficiency of AI model in recognition of standard EUS anatomical sites in external validation

It took an average of 520±66 min for the experts and 960±136 min for the beginners to read all the EUS images and classify them, respectively, while the AI model took only 109±2 min (P<0.001, Fig. 5).

Figure 5.

Average reading time of beginners (blue), experts (red) and CNN model (yellow). ***P < 0001.

Discussion

In this study, we explored a CNN model to recognition of standard BPS anatomical site images from linear-array EUS and successfully validated it with multicenter data. Several striking features and fundamental characteristics should be emphasized in our study. First, to the best of our knowledge, the validation data used in this study was obtained from the three EUS different facilities, and the model was validated with high sensitivity and excellent adaptability. This design is also in line with the construction of high-quality annotation datasets, which have become the key foundation for establishing and optimizing CNN model platforms^20,21. Second, this study involves all normal anatomical sites of the BPS investigated during linear-array EUS scans; hence, we consider the datasets we developed to accurately represent real-world changes in the standard anatomy of the BPS without blind spots because such a meticulous design meets practical guidelines^22,23. Third, we built a CNN-based algorithm with high sensitivity and specificity; the model had high accordance with experts, and its reading time was obviously shorter than that of the beginners and experts.

Qualified digestive endoscopy must cover at least two important criteria: first, we must be able to accurately identify the information that appears on the screen; second, and more importantly, we must be able to check every part without leaving blind spots²². In clinical practice, however, achieving these two goals can be difficult. The differences between humans and machines are that humans can suffer from visual fatigue and become distracted by external stimuli while sifting through similar images, leading to some low-level errors, such as neglect or missed diagnosis, even among experienced physicians^24,25. AI, in contrast to humans, can alert the operator and prevent low-level human errors in clinical practice; these advantages have been well verified in the discovery of small bowel diseases and colon polyps^26,27.

Due to the anatomical proximity and common embryological origin of the bile duct system and pancreas, they share many similarities in the spectrum of diseases. EUS plays a vital role in the diagnosis of BPS disease, but it is difficult for beginners to recognize the standard BPS anatomical site on the EUS image. Even for experienced operators, it cannot be guaranteed that no anatomical site will be missed in each EUS operation. Zhang et al.²⁸ constructed a pancreas segmentation and station recognition system (six stations) based on deep learning in EUS, and the accuracy of internal and external station classification was 94.2 and 82.4%, respectively. Yao et al.²⁹ trained a deep learning-based system for bile duct annotation and station recognition in linear EUS for classification (four stations), and the model achieved an accuracy of 93.3% in image set and 90.1% in video set. While these two studies have verified the use of AI in standard scans at BPS sites with good results, they focused more on individual organs of the pancreas or biliary tract and the surrounding anatomy, whereas our study evaluated scan sites that were seamlessly linked across the BPS and included all 14 sites of BPS. Compared with the above two studies, ours included four new sites, including the liver vein and portal vein from the first hepatic portal to the superior mesenteric vein, as these are important sites for standard BPS scans because pancreatic and bile duct cancers often metastasize to the portal vein system.

The CNN model we trained achieved an excellent diagnostic performance. In the internal validation, the accuracy ranged from 92.1 to 100.0%. In the external validation, the sensitivity of the CNN model ranged from 89.45 to 99.92% and the specificity ranged from 93.35 to 99.79%. For beginners, the sensitivity was 47.79–96.49%, and the specificity was 98.18–99.98%. The CNN model had significant higher sensitivity in the recognition of 11 sites, except for left kidney, spleen, and gallbladder sites. This result may be due to the fact that these three anatomical sites are easier for beginners to identify. In addition, there was a great diagnosis agreement between CNN model and experts for 14 sites (Kappa scores of 0.84–0.98).

Several limitations of this study should be noted. First, the external validation was obtained from recorded video clips, and the video may contain flashing structures that were not captured by the computer. Second, the training and validation images come from patients with normal anatomy, and the ability to identify anatomy with abnormal lesions has not been verified. Third, well-designed multicenter prospective investigations should be conducted with larger datasets to verify the results and refine the model before clinical use. Fourth, the effect of patients’ personal information characteristics (such as sex, BMI, etc.) on the identification of anatomical sites was not taken into account, and the distribution of diagnoses made among beginners was not assessed.

Conclusions

In summary, we established a CNN model for identifying images of standard BPS sites obtained with linear-array EUS through a multicenter retrospective study. First, we determined that the whole process of true standard scanning does not leave blind spots, and that the model shows a high degree of agreement with experts in recognizing standard EUS images. We believe that the CNN model proposed in our study may be an important BPS linear-array EUS reading model and may become an effective auxiliary system.

Ethical approval

The study was approved by the Ethics Committee of Tongji Medical College, Huazhong University of Science and Technology (IORG No. IORG0003571).

Consent

Written informed consent was obtained from all patients. Patient data were anonymized, and any personally identifying information was omitted.

Sources of funding

Supported by the National Natural Science Foundation of China (Nos. 81974068 and 82170571), the National key research and development program of China (No. 2017YFC0110003), the Natural Science Foundation of Hubei Province (No. 2017CFA061), the Non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (No. 2020-PT330-003). The funding body had no part in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Author contribution

T.-S.X. and S.-H.Y.: project design, analyzed and interpreted data, drafted and edited the manuscript; C.-W.G., L.-S.J., H.-C.Q., D.-F., and W.-W.J.: collected data and analyzed data; L.-S.J., W.-H.X., L.-Y.L., D.-L., T.-J., Z.-J.J., L.-J.J., S.-L., N.-B., Z.-K., M.-J.R., W.-G.B., H.-H., H.-X.X., K.-W.J., and J.-X.J.: provided verification data; D.-Z. and L.-R.: project design, supervision, and manuscript revision.

Conflicts of interest disclosure

There are no conflicts of interest.

Research registration unique identifying number (UIN)

This study was registered in the Chinese Clinical Trial Registry (Registration No. ChiCTR2100048349).

Guarantor

Rong Lin.

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Provenance and peer review

Not commissioned, externally peer-reviewed.