Explainable deep learning and radiomics integration for differentiating insulinomas from NF-PNETs in EUS imaging

Abstract

Objectives

This study aimed to develop interpretable deep learning (DL) and radiomics models using endoscopic ultrasound (EUS) images to differentiate insulinomas from nonfunctional pancreatic neuroendocrine tumors (NF-PNETs).

Methods

The retrospective analysis comprised 115 patients, including 61 with insulinomas and 54 with NF-PNETs, all confirmed through pathological examination. The patient cohort was divided into training and test groups. From standardized EUS images, a total of 512 DL features and 107 radiomics features were extracted. LASSO regression was employed to identify non-zero coefficient features from both the DL and radiomics datasets. Subsequently, four machine learning algorithms were utilized to construct predictive models. The optimal DL and radiomics models were then integrated into a nomogram for enhanced predictive capability. Gradient-weighted Class Activation Mapping (Grad-CAM) and Shapley Additive Explanations (SHAP) provided model interpretability.

Results

The ExtraTrees DL and radiomics models demonstrated exceptional performance. The integrated nomogram yielded AUC values of 0.978 for the training group and 0.842 for the test group. Calibration curves and decision curve analysis corroborated the high accuracy and clinical utility of the models. Grad-CAM identified tumor margins and heterogeneity as significant features in the DL model, whereas SHAP analysis highlighted texture patterns in the radiomics data. The nomogram effectively facilitated visual simplification of risk stratification.

Conclusions

The interpretable DL-radiomics nomogram exhibited significant potential in differentiating insulinomas from NF-PNETs using EUS. This methodology improves diagnostic accuracy and informs clinical decision-making.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-026-03388-2.

Introduction

Pancreatic neuroendocrine tumors (PNETs) are the second most prevalent malignant pancreatic tumors, with increasing incidence [1]. Based on the evidence of hormone-producing and clinical symptoms, PNETs can be classified into nonfunctional (NF-PNETs) and functional types (F-PNETs) [2, 3]. PNETs are diverse in nature, exhibiting unique clinical and histomorphology characteristics, and their prognosis varies [4]. F-PNETs can secrete hormones like insulin, gastrin, vasoactive intestinal peptide, glucagon, and somatostatin, leading to specific symptoms. Insulinomas, the most common F-PNET subtype, cause recurrent hypoglycemia due to continuous high insulin levels [5, 6]. Diagnosing insulinomas is challenging due to varied symptoms, nonspecific tests, and the absence of a specific diagnostic model, often resulting in prolonged misdiagnosis [7]. The symptoms of insulinoma are insidious and deceptive, leading to many misdiagnoses as neurological disorders [8]. Historically, Whipple’s triad has been used to diagnose insulinoma in human medicine. However, these criteria are not definitive, as patients with other causes of hypoglycemia can also meet them. Furthermore, insulinoma can’t be diagnosed with complete accuracy using clinicopathologic tests like serum insulin levels, as these tests lack full sensitivity and specificity [9]. Recurrent hypoglycemia can manifest in individuals with diabetes, endocrine disorders, malignant neoplasms, malnutrition, and renal insufficiency, among other conditions [10]. Additionally, patients with obesity and metabolic syndrome may exhibit marked hyperinsulinemia [11].

Currently, surgery is the sole cure for PNETs. While the WHO’s tumor grade classification for PNETs is the best predictor of clinical outcomes, treatments vary in effectiveness even for patients with tumors of the same grade or those undergoing similar therapies [12]. European Neuroendocrine Tumor Society (ENETS) guidelines recommend monitoring asymptomatic NF-PNETs under 2 cm without surgery [13]. Conversely, insulinomas typically require surgical intervention, and somatostatin analogs are becoming a more popular treatment for well-differentiated, low-grade F-PNETs [14–16]. Accurate and prompt diagnosis and prognosis of NF-PNETs and insulinomas are crucial for effective treatment planning.

EUS is now widely used for diagnosing PNETs due to its superior high-resolution imaging compared to multidetector computerized tomography (MDCT) and magnetic resonance (MR), particularly in the case of small lesions [17–19]. It is also the preferred method when other non-invasive techniques fail, as per the 2023 ENETS guidelines [20]. However, in addition to lesion-related factors, operator-related factors influence EUS accuracy. The most important operator-related factor is the amount of EUS experience [19]. For this reason, reducing the anthropogenic factors is extremely important for improving pancreatic lesion diagnosis.

Advancements in computer-aided detection and artificial intelligence (AI) have spurred radiomics [21, 22]. These technologies aid decision-making in managing pancreatic diseases, with EUS imaging-based radiomics gaining traction recently [23, 24]. In prior research, we constructed radiomics models based on EUS imaging, employing machine learning techniques to differentiate PNETs from pancreatic adenocarcinoma, F-PNETs, and NF-PNETs, as well as to predict their pathological grades. Nonetheless, these models exhibit a deficiency in visual interpretability, thereby constraining their applicability and practical utility [7, 25, 26].

The deep learning (DL) algorithm represents a specialized form of machine learning that integrates neural networks into its AI framework [27]. Unlike traditional radiomics, DL-based approaches leverage the inherent non-linearity of deep neural networks to autonomously learn relevant features [28]. Additionally, recent advancements in DL have shown that features can be independently extracted by neural networks without human intervention, resulting in improved predictive performance [29]. Transfer learning is a new DL method where a model designed for one task is repurposed as the starting point for another task [30]. In the context of transfer learning, knowledge acquired from source tasks is utilized and adapted to enhance the performance of models on target tasks [31].

Many studies have demonstrated that enhanced CT, ultrasound, and MR images combined with DL algorithms can accurately predict postoperative recurrence, invasiveness, and pathological grading of PNETs [32–34]. Nevertheless, the interpretability of these models is hindered by the absence of visualization techniques. Although EUS demonstrates superior efficacy in detecting PNETs, there is a paucity of research concerning predictive models for insulinomas that employ EUS-based radiomics integrated with deep transfer learning (DTL). A notable deficiency persists in the interpretation and visualization of these models.

This study evaluated the effectiveness of combining radiomics and DTL features from standard EUS images to distinguish insulinomas from NF-PNETs. We also used Grad-CAM and SHAP to explain and visualize the model outputs. We hypothesize that the combined radiomics-DL models, enhanced with SHAP, can accurately and interpretably differentiate insulinomas from NF-PNETs.

Materials and methods

Clinical data

In this retrospective study, the ethics committee of our institution approved the protocol (No. 2023-K346-01), exempting the need for patient consent or signed informed consent. These criteria were used to determine eligibility: (1) undergo a meticulous EUS scan of the entire pancreas; (2) have proven pathological outcomes; (3) have complete, clear EUS images before preoperative or pathological biopsies; (4) chemotherapy or radiotherapy couldn’t be administered before EUS. It was excluded from the study patients who had tumors of other types, motion artifacts, or noise, or whose images did not show the whole lesion.

Ultimately, a cohort of 115 participants was recruited for this study, comprising 61 individuals diagnosed with insulinomas and 54 patients with NF-PNETs. These participants underwent either pancreatic surgery or EUS-guided fine-needle aspiration/biopsy (EUS-FNA/B) at our institution between October 2012 and October 2023. Fig. 1 depicts the randomization process, which allocated the participants into training and test groups in a 7:3 ratio. Additionally, we conducted an analysis of various clinical parameters, including age and gender.

Fig. 1.

Flowchart of the study population enrolled

EUS examination and image acquisition

Preoperative or pre-biopsy pancreatic EUS examinations were performed on all enrolled patients using FUJIFILM SU-9000 and Olympus EU-ME2 equipment. An EUS specialist with more than 10,000 EUS procedures under his belt thoroughly examined the pancreatic area and obtained detailed images of the masses. In these images, a grayscale level of 125 values and a grayscale window of 250 values were consistently used. Our institution’s Picture Archive and Communication System (PACS) was used to obtain the imaging data.

Region of interest segmentation

Two EUS specialists, each with over seven years of experience and unaware of the histopathological diagnoses, reviewed the EUS images of the enrolled patients during the study. A region of interest (ROI) is manually outlined using the open-source software ITK-SNAP (version 3.8.1, http://www.itksnap.org). In traditional EUS imaging, lesions were meticulously outlined along their peripheries, ensuring the exclusion of surrounding normal tissues, blood vessels, bile ducts, and pancreatic ducts from the delineation. Discrepancies in the delineations were addressed and resolved through collaborative discussion and consensus among the specialists. An overview of the situation is provided in Fig. 2.

Fig. 2.

The workflow for the whole study

To ensure reproducibility, standardization procedures were implemented in the preprocessing of images and data. The intraclass correlation coefficient (ICC) was utilized to assess both intraobserver and interobserver reproducibility. A cohort of 32 patients was randomly selected, and after a one-month interval, the same EUS specialists performed the ROI segmentation once more. This process was repeated until both the intraobserver and interobserver ICC exceeded 0.80, a threshold considered indicative of satisfactory agreement.

Radiomics feature extraction

To eliminate confounding factors, EUS images were resampled to a voxel size of 1 × 1 mm² before feature extraction. The categorization of handcrafted features can be delineated into three discrete groups, namely geometric, intensity, and textural. Geometric features are concerned with the three-dimensional morphological characteristics of tumors. Intensity features encompass the statistical dispersion of voxel intensities within the tumor in the first order. Conversely, textural features elucidate patterns and higher-order spatial distributions of intensities. This article utilized multiple methodologies, reported previously in the literature [7], to extract texture features. The extraction and screening of radiomics features were performed using PyRadiomics, an internal feature analysis program, which facilitated the extraction of all handcrafted features. The processes of extracting radiomics features followed the Image Biomarker Standardization Initiative (IBSI). A Z-score method was used to standardize the radiomics features.

Deep transfer learning features extraction

Transfer learning was used to extract DL features from pre-trained convolutional neural networks. The Resnet18 model was trained on ImageNet, which includes 1.2 million images classified into 1000 categories and may not necessarily apply to our task [35]. A pre-trained ResNet18 neural network model was re-trained on all EUS images from the training group beforehand. For each patient, the image with the largest cleft was selected, and the gray values were normalized using the min-max transformation. Subsequently, each cropped subregion image was resized to 224 × 224 pixels using nearest-neighbor interpolation. Using the obtained images as input, the DTL features were sized at 512. To assess the areas emphasized by DTL, we utilized the Grad-CAM method to generate saliency maps for every instance of pancreatic mass. A Z-score method was used to standardize the DL features and mean and variance (standard deviation) were calculated for each column.

Radiomics and deep learning signatures building

Following the comparison of training and test groups, Mann-Whitney U tests were conducted. Subsequently, feature selection was performed, retaining only those radiomics and DL features that exhibited significance levels of p < 0.05 for further analysis. The Spearman rank correlation coefficient was employed to assess the interrelationships among the features. Features exhibiting a correlation coefficient exceeding 0.9 were retained through random sampling. To further refine the feature representation, a greedy recursive deletion strategy was implemented, incorporating the L1 regularization term to constrain the model parameters. This approach involved iteratively eliminating the most redundant features within the current set. The application of the Least Absolute Shrinkage and Selection Operator (LASSO) algorithm facilitated the derivation of the optimal fusion feature subset. The radiomics and DL features with nonzero coefficients were definitively retained. The radiomics and DL signatures were constructed by integrating features with nonzero coefficients, employing a 5-fold cross-validation approach and utilizing widely recognized supervised machine learning algorithms, including random forest (RF), light gradient boosting machine (LightGBM), extreme gradient boosting (XGBoost), and Extra trees, simultaneously. Early stopping is used during model training to prevent overfitting.The model exhibiting superior, reasonable, and consistent performance between radiomics and DL models was identified and regarded as radiomics and DL signatures. Subsequently, the SHAP value of each retained radiomics and DL feature was computed to enhance the interpretability of the predictions generated by the signature. Finally, various metrics, such as area under the curve (AUC), accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were utilized to assess the diagnostic performance of the radiomics and DL signatures, separately. Decision curve analysis (DCA) quantified the radiomics and DL signatures’ net benefit in identifying insulinomas and NF-PNETs.

Nomogram establishment and assessment

The R rms package was utilized to develop a nomogram for effectively distinguishing insulinomas from NF-PNETs integrating DL and radiomics signatures. Calibration was confirmed with a calibration curve and the Hosmer-Lemeshow (H-L) test, aiming to evaluate the calibration capability of the nomogram. To compare and assess the effectiveness of the radiomics signature, DL signature, and nomogram, the Delong test was conducted. Finally, the DCA was applied to assess the nomogram’s net benefit and predictive performance.

Statistical analysis

Participants’ clinical parameters and DL features were compared using appropriate statistical tests such as independent sample t-tests, Mann-Whitney U tests, or X² tests. Statistical significance was determined by P < 0.05. Several metrics were used to evaluate prediction performance, including AUC, specificity, sensitivity, accuracy, and PPV. AUC was compared using Delong’s test. Fig. 2 summarizes the comprehensive methodology for this study.

Results

Population characteristics statistics

A total of 115 patients, comprising 72 women and 43 men, were included in this retrospective study. These patients were randomly allocated into two groups: a training group (N = 80) and a testing group (N = 35). The baseline characteristics of the entire patient cohort are presented in Table 1. The results indicated no statistically significant differences in age and gender between the insulinoma and NF-PNETs groups.

Table 1.

Clinical characteristics between NF-PNETs and insulinomas

Variable	NF-PNETs (N = 54)	Insulinomas (N = 61)	P-value
Age	48.63 ± 13.58	46.49 ± 13.31	0.342
Gender			0.096
Female	29(53.70%)	43(70.49%)
Male	25(46.30%)	18(29.51%)

Radiomics feature extraction and screening

107 manually derived radiomic features were collected encompassing seven categories. These features include 18 first-order features, 14 shape features, and a variety of texture features. These handcrafted features have previously been defined in previous articles [36]. The complete series of radiomic features, along with their corresponding p-values, is presented in Supplementary Fig. 1. A total of six radiomic features with nonzero coefficients were retained following the feature selection process utilizing LASSO logistic regression. The coefficients and mean standard errors (MSEs) obtained from the 10-fold cross-validation are illustrated in Fig. 3A and B, while the retained radiomic features along with their corresponding coefficients are presented in Fig. 3C.

Fig. 3.

Radiomics feature selection with the LASSO regression model. (A) The LASSO model’s tuning parameter (λ) selection used 10-fold cross-validation via minimum criterion. (B) LASSO coefficient profile plot with different log (λ) was displayed. (C) The bar graph of 6 radiomics features with their nonzero coefficients

Construction of radiomics signature

As illustrated in Fig. 4A and B, the ROC curves and AUCs for each radiomics model, derived from four widely utilized machine learning algorithms, are presented for both the training and test groups. A detailed summary of the results is provided in Table 2. It is noteworthy that the RF and XGBoost models tend to overfit. In contrast, the ExtraTrees model demonstrated superior performance and exhibited greater consistency between the training (AUC = 0.903, 95% CI 0.8359–0.9704) and test (AUC = 0.855, 95% CI 0.7339–0.9766) groups, underscoring its efficacy as a radiomics model. The ExtraTrees-based radiomics model achieved an accuracy of 0.743, sensitivity of 0.789, specificity of 0.687, PPV of 0.750, and NPV of 0.733 in the test group (Table 2). This model was selected as the radiomics signature for further analyses. It showed superior clinical net benefits for preoperative prediction of insulinomas, as demonstrated by DCA (Fig. 4C and D). Prediction scores from the model are shown in Fig. 4E and F. The weights of all retained radiomics features involved in modeling the radiomics signature are shown in Fig. 5.

Fig. 4.

The performance of different radiomics-based machine learning models. The ROC curves of different radiomics models in the training(A) and test group (B). The DCA curve for the ExtraTrees-based radiomics model in the training (C) and test groups (D). ExtraTrees-based prediction scores of the radiomics model in the training (E) and test (F)groups. (“label = 0” means “NF-PNETs”; “label = 1” means “Insulinomas”)

Table 2.

Diagnostic performance of different radiomics models for predicting insulinomas in training and test groups

Model	Group	AUC(95% CI)	Accuracy	Sensitivity	Specificity	PPV	NPV
RF	Training	0.980(0.9582–1.0000)	0.925	0.952	0.895	0.909	0.944
	Test	0.812(0.6611–0.9639)	0.771	0.632	0.937	0.923	0.682
XGBoost	Training	0.996(0.9874–1.0000)	0.963	0.952	0.974	0.976	0.949
	Test	0.781(0.6234–0.9391)	0.743	0.789	0.687	0.750	0.733
LightGBM	Training	0.914(0.8558–0.9743)	0.825	0.738	0.921	0.912	0.761
	Test	0.803(0.6563–0.9490)	0.771	0.632	0.937	0.923	0.682
ExtraTrees*	Training	0.903(0.8359–0.9704)	0.825	0.857	0.789	0.818	0.833
	Test	0.855(0.7339–0.9766)	0.743	0.789	0.687	0.750	0.733

*Represents selected models for radiomics signature

RF, random forest; LightGBM, light gradient boosting machine; XGBoost, extreme gradient boosting; CI, credibility interval

Fig. 5.

The weights of all retained radiomics features involved in the ExtraTrees-based radiomics model

Explanation and visualization of the radiomics signature

Utilizing the radiomics signature, we applied interpretable machine-learning techniques through the SHAP method. The significance of each feature within the model was assessed using the SHAP approach. Fig. 6A illustrates the outcomes of the radiomics feature importance analysis, with the most significant features positioned at the top and the less significant features at the bottom. The majority of the retained radiomics characteristics exhibited either positive or negative correlations with the prediction outcomes. SHAP summary plots were utilized to visually represent the significance and influence of radiomic features on the model’s output. Features were ranked according to their global importance, with each dot signifying an individual patient’s SHAP value for a specific feature. These dots were plotted horizontally and stacked vertically to illustrate density and were color-coded from blue (indicating low values) to red (indicating high values) based on the feature value. Our analysis identified “original_shape_ VoxelVolume” as the principal feature for distinguishing between insulinoma and NF-PNETs classifications. The density plot revealed a range of SHAP values for this feature, demonstrating that the model’s output increased as the feature’s value decreased.

Fig. 6.

(A) SHAP summary plots of ExtraTrees-based radiomics model. The predicted diagnosis of these pancreatic lesions was insulinoma (B) and NF-PNETs (C) via the SHAP force plots, respectively

The force plot (Fig. 6B and C) provides a detailed assessment of an individual patient’s data by depicting each feature’s SHAP value as a force that either increases or decreases the prediction, relative to the base value, which is the mean SHAP value. The length of the arrows represents the percentage contribution of each feature, and the color coding indicates the nature of the contribution: red for positive and blue for negative. As demonstrated in Fig. 6B, the SHAP value for this patient was 0.73, surpassing the baseline value, thereby indicating that this patient may be classified within the insulinoma group. In contrast, another patient presented a SHAP value of 0.26, which is below the baseline value. Therefore, this patient may be categorized under the NF-PNETs classification, as illustrated in Fig. 6C.

Deep learning feature extraction and selection

In the course of this study, we utilized the pre-trained ResNet18 model to extract 512 DL features. Among these, a total of 239 features demonstrated statistically significant differences between the insulinoma and NF-PNETs. We subsequently conducted a comparative analysis and visualization of the correlation coefficients for these DL features, ultimately retaining 239 features for further examination (Supplementary Fig. 2). Our results indicated weak collinearity among the DL features, suggesting that the pre-trained ResNet18 model effectively captured the distinctions between the groups. To examine the interpretability of the deep learning regressor (DLR), we utilized Grad-CAM to visualize the network’s decision-making process. This technique generates a coarse localization map that emphasizes regions of significance pertinent to the classification objective. For this analysis, we rendered the last convolutional layer of the final residual block transparent (Fig. 7). From the DL features, 12 features with non-zero coefficients were selected using a LASSO logistic regression model applied to the training group. The coefficients, mean standard error from 10-fold cross-validation, and the values of the coefficients for the finally selected non-zero DL features are presented in the accompanying Fig. 8.

Fig. 7.

Grad-CAM visualization from two different patients with insulinoma (A) and NF-PNETs (B), displaying the importance of different image regions to the network decision of identifying mass classification

Fig. 8.

DL feature selection with the LASSO regression model. (A)The LASSO model’s tuning parameter (λ) selection used 10-fold cross-validation via minimum criterion. (B) LASSO coefficient profile plot with different log (λ) was displayed. (C) The bar graph of 12 DL features with their nonzero coefficients

Deep learning signature and visualization

As shown in Fig. 9A and B, the ROC curves and AUCs of each DL model derived from the same four machine learning algorithms are shown for the training and test groups. A detailed summary of these results is provided in Table 3. It is important to note that the XGBoost models tend to overfit. Compared to other models, the ExtraTrees model demonstrated superior performance, with AUCs of 0.979 (95% CI: 0.9572-1.0000) and 0.737 (95% CI: 0.5563–0.9174) for the training and test groups, respectively. Consequently, the ExtraTrees model was designated as the DL signature, similarly. However, we noticed that the accuracy of the DL signature in the test group was significantly lower than that in the training group, indicating the possibility of overfitting. To further enhance the performance of the model on the test group, we integrated the DL signature with the radiomics signature.

Fig. 9.

The performance of different DL-based machine learning models.The ROC curves of different DL models in the training (A) and test (B) group. The DCA curve for the ExtraTrees-based DL model in the training (C) and (D) test groups. ExtraTrees-based prediction scores of the DL model in the training (E) and test (F) groups. (“label = 0” means “NF-PNETs”; “label = 1” means “Insulinomas”)

Table 3.

Diagnostic performance of different deep learning models for predicting insulinomas

Model	Group	AUC(95% CI)	Accuracy	Sensitivity	Specificity	PPV	NPV
RF	Training	0.988(0.9732–1.0000)	0.938	0.952	0.921	0.930	0.946
	Test	0.750(0.5798–0.9202)	0.743	0.895	0.562	0.708	0.818
XGBoost	Training	1.000(1.0000–1.0000)	0.988	0.976	1.000	1.000	0.974
	Test	0.676(0.4846–0.8674)	0.657	0.597	0.750	0.733	0.600
LightGBM	Training	0.975(0.9470–1.0000)	0.925	0.929	0.921	0.929	0.921
	Test	0.702(0.5225–0.8821)	0.657	0.632	0.687	0.706	0.611
ExtraTrees*	Training	0.979(0.9572–1.0000)	0.912	0.905	0.921	0.927	0.897
	Test	0.737(0.5563–0.9174)	0.743	0.895	0.562	0.708	0.818

*Represents selected models for deep learning signature

RF, random forest; LightGBM, light gradient boosting machine; XGBoost, extreme gradient boosting; CI, credibility interval

The study further demonstrated superior clinical net benefits for preoperative prediction of insulinomas, as evidenced by DCA (Fig. 9C and D). The prediction scores derived from this model are illustrated in Fig. 9E and F. Additionally, the weights of all retained DL features contributing to the construction of the DL signature are visualized in Fig. 10.

Fig. 10.

The weights of all retained DL features involved in the ExtraTrees-based DL model

Similar to the previous radiomics signature, we utilized the SHAP method to interpret this DL machine-learning signature, the outcomes are shown in Fig. 11. We found that DL_73 was the key feature for classifying insulinoma and NF-PNETs. The density plot illustrated a spectrum of SHAP values for this feature, indicating that the model’s output increased as the feature’s value decreased (Fig. 11A). This finding aligns with the radiomics signature depicted in Fig. 11B, where the SHAP value for the same patient was 0.80, exceeding the baseline value and suggesting classification within the insulinoma group. Conversely, another identical patient with identical characteristics exhibited a SHAP value of 0.33, which is below the baseline value. Therefore, this patient may be categorized under the NF-PNETs classification, as illustrated in Fig. 11C. Therefore, the DL and radiomics signatures predict highly similar performance in the tasks of this study.

Fig. 11.

(A) SHAP summary plots of ExtraTrees-based DL model. The predicted diagnosis of these pancreatic lesions was insulinoma (B) and NF-PNETs (C) via the SHAP force plots, respectively

Construction and validation of the nomogram

Given that radiomics and DL extract features from different dimensions, their respective signatures were integrated to develop a nomogram. This comprehensive nomogram was constructed using logistic regression analysis of the DL and radiomics signatures, facilitated by the R rms package (Fig. 12). The resulting nomogram achieved an AUC of 0.978 in the training group and an AUC of 0.842 in the test group, outperforming the radiomics signature in the training group and the DL signature in the test group, respectively (Fig. 13A and B). Meanwhile, the AUCs of the nomogram were not inferior to those of the DL signature in the training group and the radiomics signature in the test group, as demonstrated by the Delong test and presented in Table 4. These findings suggest that the nomogram may provide a superior classification of insulinomas and NF-PNETs compared to the application of DL or radiomic signatures alone. A H-L test for the calibration curves of the nomogram revealed no significant difference between the predicted and observed incidences of insulinomas in both the training and testing cohorts (Table 4). The calibration curves for the training and testing groups are depicted in Fig. 13C and D, respectively. Moreover, the DCA revealed that the ‘Nomogram’ curve exhibited superior values and net benefits within the threshold range of approximately 0.4 to over 0.8 in both the training and test groups (Fig. 13E and F), indicating that patients whose risks are within this range may benefit from treatment for insulinoma. These findings suggest that the integration of the DL and radiomics signature could significantly enhance the prediction of insulinomas.

Fig. 12.

The nomogram predicts insulinomas based on DL signature (abbreviated “DL_Sig”) and radiomics signature (abbreviated “Rad_Sig”) simultaneously

Fig. 13.

(A) The ROCs and AUCs of radiomics signature (abbreviated “Rad Signature”), DL signature, and nomogram for predicting insulinomas in the training (A) and test (B) groups. Calibration curves for the radiomics signature, DL signature, and nomogram in the training (C) and test (D) groups. The DCA curves for the DL signature, radiomics signature, and nomogram in the training (E) and test (F) groups

Table 4.

The results of the Delong test and Hosmer-Lemeshow test

Model	P-value
	Training Group	Test Group
Delong test
Nomogram vs. Radiomics signature	0.008	0.761
Nomogram vs. Deep learning signature	0.765	0.048
Hosmer-Lemeshow test
Radiomics signature	0.335	0.283
Deep learning signature	0.237	0.248
Nomogram	0.056	0.164

Discussion

This study created models to distinguish insulinomas from NF-PNETs by combining EUS-based radiomics and DL features with four machine-learning algorithms using ROI data. Results show that integrating radiomics-DL features with machine learning significantly improves prediction accuracy for insulinomas. The ExtraTrees model performed best, with AUCs of 0.903 (95% CI: 0.8359–0.9704) and 0.855 (95% CI: 0.7339–0.9766) in the radiomics model, and AUCs of 0.979 (95% CI: 0.9572-1.0000) and 0.737 (95% CI: 0.5563–0.9174) in the DL model. Furthermore, the DL and radiomics signatures were used to create a nomogram for predicting insulinomas, showing high accuracy in both training (AUC = 0.978) and test (AUC = 0.842) groups. ROC curves, calibration curves, and DCA confirmed its effectiveness. Grad-CAM and SHAP values were applied to clarify and visualize the DL and machine learning model outputs, improving interpretability. Consequently, it was deemed a reliable and valid instrument for differentiating insulinomas from NF-PNETs and informing treatment decisions.

PNETs exhibit significant heterogeneity, encompassing a wide range of clinical and biological characteristics [37]. The majority of NF-PNETs, accounting for approximately 80% of cases, whereas insulinomas represent the most prevalent form of F-PNETs [38, 39]. Additionally, patients diagnosed with NF-PNETs demonstrate markedly lower overall survival rates in comparison to those with F-PNETs [40]. On the contrary, insulinomas are predominantly benign and solitary, with 87% being singular and benign, 7% multiple and benign, and 6% malignant with metastasis [41, 42]. Insulinoma is characterized by uncontrolled insulin production and hypoglycemia even at an early stage [43]. Recurrent hypoglycemia due to abnormal endogenous hyperinsulinism is a key sign of insulinomas. Although excessive insulin secretion is crucial for diagnosis, delayed or incorrect detection of hypoglycemia and other symptoms often leads to severe outcomes and increased mortality [6]. Insulinoma commonly presents with Whipple’s triad, characterized by hypoglycemia, hyperinsulinemia, a reduction in blood glucose levels to 50 mg/dL, and the normalization of hypoglycemia following the administration of glucose. However, these criteria are not definitive, as patients with other etiologies of hypoglycemia may also meet these conditions. During hypoglycemic episodes, insulinoma is frequently misdiagnosed as a primary neurological or psychiatric disorder [8]. In fact, in instances of insulinomas, it can be challenging to ascertain whether hypoglycemia is the precipitating factor for the seizure or if the hypoglycemia results from augmented skeletal muscle glucose utilization secondary to seizure activity [9]. Furthermore, blood glucose levels may fluctuate within and outside the normal range due to counter-regulatory mechanisms and the effects of feeding [44]. Many patients are reluctant to undergo the 72-hour fasting test because they cannot tolerate the discomfort associated with prolonged hunger, thereby hindering the accurate diagnosis of insulinomas [45]. Unfortunately, when the patient’s history, clinical signs, and insulin concentration results in the context of hypoglycemia suggest the presence of an insulinoma, contrast-enhanced CT scans often fail to detect pancreatic masses, particularly small lesions [46, 47]. Consequently, patients with insulinoma often do not receive timely diagnostic evaluations, resulting in an average delay of 3.8 years before optimal treatment is administered [48]. Therefore, it is imperative to seek innovative ways to distinguish NF-PNETs and insulinomas accurately.

Compared to CT and MR, EUS is regarded as highly accurate for diagnosing pancreatic diseases, offering high-definition images and a sensitivity range of 57% to 94%. Numerous studies have consistently shown that EUS detects small pancreatic tumors more accurately than CT and MR [17]. A meta-analysis of ten studies involving 261 participants indicated that EUS has an average predictive accuracy of 90% (ranging from 77% to 100%) for diagnosing PNETs [49]. EUS can provide a detailed pre-surgical assessment of pancreatic lesions and their proximity to bile ducts, arteries, and veins, thereby aiding in surgical planning and decision-making [26]. However, EUS is valuable for detecting pancreatic masses but relies heavily on the examiner’s experience, leading to observer bias [50]. While cost-effective for detecting PNETs, its diagnostic accuracy varies across various previous studies [51]. Computer-aided diagnosis and artificial intelligence technology can improve the ability of image classification and recognition [52]. The main methods of radiomics analysis today include radiomic AI, machine learning, convolutional neural networks (CNN), as well as other DL approaches [53]. Currently, radiomics and DTL are the most studied techniques in medical imaging [54].

Radiomics facilitates the detection of subtle changes that are imperceptible to the human eye and enhances the extraction of high-quality quantitative data from images, thereby surpassing traditional imaging modalities [55]. The application of DL for semantic segmentation is increasingly becoming mainstream due to its robust feature extraction capabilities [56]. DL algorithms are regarded as superior in learning abstract features from basic ones, which can be particularly beneficial for the development of AI models [57]. The transfer learning approach is employed to expedite the training process in DL and machine learning [58]. In this context, feature extraction or classification is primarily achieved through the transfer learning of a pre-trained deep neural network [59]. The integration of DL with radiomics is anticipated to yield significant performance improvements, as DL has the potential to uncover previously unknown patterns [60].

Previous studies have demonstrated that integrating peritumoral and intratumoral data with a nomogram model, utilizing DL contrast-enhanced ultrasound and clinical features, effectively identifies preoperative aggressiveness in PNETs [33]. Similarly, we have developed and validated a successful EUS-based radiomics model that combines clinical-ultrasound and radiomics features to predict the pathological grading of PNETs [25]. Moreover, many studies have confirmed that using radiomics, machine learning, and DL techniques with EUS imaging effectively predicts gastrointestinal stromal tumors and pancreatic ductal adenocarcinoma [7]. However, our study was the first to demonstrate that combined EUS imaging-based DL and radiomics models can accurately predict NF-PNETs and insulinomas. The nomogram we developed demonstrated superior performance compared to the radiomics signature in the training cohort and the DL model in the testing cohort. This indicates that the integration of features from diverse sources has the potential to enhance model performance and broaden clinical application prospects. Nonetheless, it is important to acknowledge that the nomogram did not surpass the radiomics model in the testing cohort. Consequently, the observed improvement in efficacy may be relatively modest, necessitating further validation through large-sample studies or external center samples.

CNN is a key DL technology extensively used in medical image analysis [61, 62]. Deep Residual Networks (Resnet), very deep CNN architectures, excel in image recognition and object detection [63]. ResNet and similar models represent the forefront of image processing advancements [64]. ResNet’s superior performance effectively addresses gradient disappearance in DL [65]. Variants like ResNet18, ResNet34, and ResNet50 differ in layer count, with ResNet18 having the fewest and ResNet50 the most [66]. In small-sample DL applications, complex ResNet models often overfit, as many studies have shown [67]. This phenomenon is mainly due to the complex structure and large parameter size of ResNet models, which tend to memorize training data details instead of learning general features, especially with small datasets [68]. Thus, in our single-center study with limited data, we opted for the simpler ResNet18 architecture to reduce overfitting risk. Additionally, the training time can be reduced by using a pre-trained ResNet18, which excels in medical image recognition and prediction [69, 70]. Consequently, the transfer learning based on pre-trained Resnet18 was chosen as the foundational model for this training framework.

Our research extracted 512 DL features from EUS imaging using the pre-trained ResNet18 model. Through t-tests, correlation analysis, and LASSO regression, we identified 12 significant DL features associated with insulinomas. Utilizing Grad-CAM, AI can delineate regions of interest within images [71]. We used Grad-CAM to visually explain the inferential processes behind the original images. This helped us understand the classification of correctly identified PNETs photos and validated the key extracted DL features by tracing their origins.

Numerous clinical prediction models have recently been developed utilizing machine learning methodologies [72]. Integrating radiomics, DL, and machine learning techniques has demonstrated substantial prognostic accuracy in oncology [73, 74]. Numerous studies have shown that combining machine learning and radiomics is effective for diagnosing and predicting PNETs [75, 76]. To overcome the limitations of single algorithms, multiple mainstream machine-learning algorithms were used to create an optimal model for distinguishing insulinomas from NF-PNETs. The ExtraTrees algorithm demonstrated the highest accuracy and consistency, making it the choice for further model refinement.

Our findings demonstrated that both the DL and radiomics signatures attained commendable AUC values when employing the ExtraTrees algorithm and exhibited significant performance. Nevertheless, the limited interpretability of these machine learning models has restricted the integration of radiomics-based studies into clinical practice. Consistent with prior literature [26, 75, 76], machine learning algorithms frequently produce difficult results, thereby impeding clinicians’ ability to incorporate these solutions into their practice effectively.

Recent research uses SHAP values to overcome machine learning model limitations [77]. SHAP assigns importance to each feature, with positive values indicating a higher likelihood of a class and negative values indicating a lower likelihood [78]. A recent study successfully used SHAP in a CT radiomics-based model to non-invasively predict the pathological grading of PNETs [79]. We used SHAP values to visualize the impact of nonzero features in ExtraTrees models for individual patients. Summary plots highlighted the importance of DL and radiomics features, explaining the predicted outcomes. Thus, beyond the high accuracy of the EUS-based DL and radiomics models, their interpretability is a significant contribution. To our knowledge, this is the first study to show that a new combined DL-radiomics model using EUS imaging can accurately distinguish insulinomas from NF-PNETs. Furthermore, a visual nomogram combining radiomics and DL signatures was developed to predict insulinomas, showing high efficacy and accuracy in both training and testing groups, as confirmed by calibration, DCA, and ROC curves. It is thus a reliable tool for predicting insulinomas and aiding treatment decisions.

This study has several limitations. The data were obtained from a single medical center, and the manual segmentation process may introduce additional bias in image segmentation. Furthermore, the low incidence of insulinoma, combined with the small sample size of patients who have undergone either preoperative biopsy or surgical resection, results in insufficient training data. This inadequacy poses a risk of developing evaluation models with potentially inappropriate PPV and NPV. Despite the application of various technical methodologies, this inadequacy continues to cause the model to exhibit overfitting within the training group. This represents a significant limitation of the study, potentially constraining the generalizability of the findings. Consequently, forthcoming research on EUS-based DL for the prediction of insulinomas should incorporate multicenter studies, larger sample sizes, and prospective study designs. Furthermore, it is advisable for future investigations to explore the integration of automatic image segmentation technology for EUS images. Ultimately, the precise biological mechanisms and histological changes reflected by the radiomics and DL features employed in modeling remain unclear. This study proposes that integrating DL and radiomics with genomics and pathological omics could potentially enhance biological understanding, establish connections between imaging features and biological mechanisms, and improve model interpretability in differentiating insulinomas from NF-PNETs.

Conclusion

In summary, a novel interpretable DL model and nomogram were developed and validated using EUS images, in conjunction with machine learning algorithms. This approach exhibits substantial potential for augmenting the clinical utility of EUS in differentiating insulinomas from NF-PNETs.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

SM, HC, RJ, and HX designed this study and drafted the manuscript, these authors contributed equally to this work and shared the first authorship. SM and HC performed the statistical analyses. RJ and HX carried out the clinical data. NL and SQ provided funding support, and these authors contributed equally to this work and shared senior authorship. QH, BH, YZ, YW and CH performed the statistical analyses and reviewed the manuscript. All authors contributed to the article and approved the submitted version. All authors read and approved the final manuscript.

Funding

This research received no external funding.

Data availability

Detailed contributions to the study are included in the article/supplements; corresponding authors can be contacted for further inquiries. The inquiry of the original code for this manuscript can be directed to the corresponding authors.

Declarations

Ethical approval and consent to participate

This retrospective study was approved by the Medical Ethics Committee of The First Affiliated Hospital of Guangxi Medical University (No. 2023-K346-01, 2023-12-29) and conformed to the Declaration of Helsinki. Considering the retrospective nature of the study, informed consent was not required.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.