Comparison of radiomics handcrafted features and deep features in ¹⁸F-FDG PET/CT for differentiating benign and malignant persistent pulmonary ground-glass nodules

Abstract

Background

The differentiation between benign and malignant persistent pulmonary ground-glass nodules (GGNs) remains challenging, and the relative value of radiomics handcrafted features and deep features derived from ¹⁸F-FDG PET/CT in this setting requires further comparison. This study aimed to develop and validate diagnostic models using radiomics handcrafted features and deep features extracted from ¹⁸F-FDG PET/CT for differentiating benign and malignant persistent pulmonary GGNs.

Methods

Data from 173 patients (184 GGNs) across three PET/CT centers were retrospectively analyzed. Patients underwent ¹⁸F-FDG PET/CT and breath-hold chest CT, with diagnoses confirmed by pathology or follow-up. Models were developed using clinical features, handcrafted features, and deep features extracted via pretrained convolutional neural networks (VGG19 and ResNet50). The SUTAH dataset was used for model training and validation, and CZ2PH and CZCH datasets were used as external test sets. Single-modality and dual-modality diagnostic models were developed based on clinical/conventional imaging features, radiomics handcrafted features, and deep features. Deep features were extracted from PET and CT images using the pretrained convolutional neural networks VGG19 and ResNet50. Model performance was evaluated using the AUC and its 95% CI, while decision curve analysis (DCA) and the Brier score were used to assess net benefit and probability prediction error, respectively.

Results

In the internal validation set, the CT_HF and PET_HF models achieved AUCs of 0.889 (95% CI: 0.756–0.964) and 0.903 (95% CI: 0.774–0.972), respectively, which were numerically higher than that of the reference model (AUC = 0.864, 95% CI: 0.725–0.949). In the external test set, the PET_ResNet50/CT_VGG19 combined model achieved the highest AUC of 0.927 (95% CI: 0.802–0.984), compared with 0.824 (95% CI: 0.676–0.924) for the reference model and 0.884 (95% CI: 0.747–0.962) for PET_ResNet50; however, the differences among the three models were not statistically significant (P = 0.101–0.464). DCA showed that this combined model had a relatively high net benefit when the threshold probability exceeded 0.5, and its Brier score in the external test set was 0.122, which was comparable to that of the reference model.

Conclusion

The integrated ¹⁸F-FDG PET/CT model with PET_ResNet50 and CT_VGG19 deep features achieved favorable discrimination and modest net clinical benefit in the external test set. However, its probability calibration and generalizability await validation on larger multicenter datasets due to insufficient benign nodules in the external test set.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12885-026-16516-y.

Background

With the widespread use of low-dose chest CT and the increasing prevalence of lung cancer screening, the detection rate of pulmonary ground-glass nodules (GGNs) has significantly increased. GGNs represent nonspecific imaging findings, encompassing pure ground-glass nodules (pGGNs) and mixed ground-glass nodules (mGGNs). Their pathological types range from benign lesions, such as focal fibrosis, hemorrhage, inflammation, and granulomas, to malignant lesions, such as lung adenocarcinoma. Persistent mGGNs, in particular, are associated with a higher likelihood of malignancy. However, the ELCAP study [1] revealed that the malignancy rate for persistent pGGNs is only 18%, while it is 63% for mGGNs, highlighting that a considerable proportion of GGNs are benign. Medicare data show that over 40% of lung cancer management costs are due to invasive procedures on benign nodules [2]. Accurate differentiation of benign and malignant persistent GGNs is crucial for formulating appropriate treatment and follow-up strategies. Specifically, reducing follow-up frequency for benign GGNs can alleviate patient anxiety and effectively prevent overtreatment.

Nonsurgical biopsies, such as percutaneous lung and bronchoscopic biopsies, are commonly used to obtain tissue samples for distinguishing benign from malignant GGNs [3]. These techniques, however, often yield false negatives for small lesions or GGNs with minimal solid components and carry significant risks of pneumothorax (26%) and bleeding (16%) [4, 5]. High-resolution CT (HRCT) can depict the morphology, density, and edge characteristics of GGNs, while PET/CT provides both functional metabolic and anatomical information. Previous studies have developed various models or probability calculation tools, such as the Brock and Herder models, based on clinical data and traditional imaging parameters to assist clinicians in diagnosis and decision-making [6]. However, these models were primarily developed using data from Western populations, incidentally detected nodules, or solid nodules [3, 6], and models specifically tailored for differentiating benign and malignant persistent GGNs remain scarce.

Radiomics, a high-throughput method for extracting and modeling imaging features, can transform raw imaging data within regions of interest into high-dimensional features difficult for the human eye to recognize or quantify. Radiomics has shown promise in diagnosing benign versus malignant lung nodules, classifying invasive subtypes and pathological types of lung adenocarcinoma, and predicting prognosis [7]. Deep learning (DL), based on convolutional neural networks, excels at extracting meaningful features and patterns from complex datasets with high performance and reliability [8]. It has demonstrated potential in distinguishing benign from malignant lung nodules [9, 10]. However, most current deep learning models are based on CT images or public datasets dominated by solid nodules [9]. PET/CT, as a dual-modality imaging technique, simultaneously provides structural and metabolic information about GGNs. Previous studies have shown that incorporating PET metabolic parameters (e.g., SUV) into CT radiomics models can improve the differentiation of benign and malignant GGNs [11]. Diagnostic models integrating PET and CT imaging features through radiomics or deep learning algorithms could theoretically achieve higher diagnostic performance. However, few studies have compared radiomics- and deep learning-based diagnostic models for GGNs or examined the impact of features extracted using different deep learning algorithms on model performance. This study aims to develop ¹⁸F-FDG PET/CT diagnostic models based on radiomics handcrafted features and deep features and to validate and compare their performance in distinguishing benign and malignant persistent GGNs, thereby elucidating their clinical value.

Methods

Study Subjects

A retrospective analysis was conducted on patients with persistent pulmonary ground-glass nodules (GGNs) who underwent ¹⁸F-FDG PET/CT, and breath-hold chest CT scans at three PET/CT centers. The inclusion criteria were: 1). Persistent pulmonary GGNs present for over 3 months. 2). Underwent both PET/CT and breath-hold chest CT scans. 3). Completed surgery with full pathological data available within 1 month after PET/CT or followed up until nodule resolution. 4). Malignant GGNs are classified according to the 2011 IASLC/ATS/ERS classification of lung adenocarcinoma subtypes. 5). No history of severe liver disease, diabetes, or other malignancies. Exclusion criteria included: 1). Previous anti-tumor therapy. 2). Fasting blood glucose > 11.1 mmol/L. 3). Severe liver or kidney dysfunction. 4). Poor image quality affecting nodule measurement. Clinical data for the enrolled patients, including age, sex, smoking history, and fasting blood glucose, were recorded. This study adhered to the principles of the Declaration of Helsinki and was approved by the institutional ethics committee (No. [2020] KD 075). The patient enrollment process is detailed in Fig. 1.

Fig. 1.

Flowchart of the patient enrollment process

Based on the inclusion criteria, 173 patients with 184 pulmonary GGNs (43 benign and 141 malignant) were enrolled from three centers. Among them, 142 GGNs from the Third Affiliated Hospital of Soochow University (SUTAH dataset) were designated the development set (including the training and internal validation sets) for model development and optimization.

The SUTAH dataset was acquired using a Siemens Biograph mCT (64) PET/CT scanner and included 136 patients with 142 nodules, comprising 36 benign nodules and 106 malignant nodules. This dataset included 106 malignant nodules, comprising 1 pre-invasive lesion (adenocarcinoma in situ, AIS, 0.9%), 5 minimally invasive adenocarcinomas (MIA, 4.7%), and 100 invasive adenocarcinomas (IAC, 94.3%). The 36 benign nodules consisted of 20 inflammatory lesions or inflammatory pseudotumors, 6 granulomatous inflammations, 5 fungal infections (1 aspergillosis, 1 cryptococcosis, and 3 other fungal infections), 1 bronchiolar metaplasia, 1 sclerosing pneumocytoma, and 3 nodules that resolved during follow-up. Additionally, 42 GGNs from Changzhou No. 2 People’s Hospital (CZ2PH dataset) and Changzhou Cancer Hospital (CZCH dataset) were used as external test set to evaluate model generalization performance. The CZ2PH dataset was acquired using a Siemens Biograph mCT Flow PET/CT scanner and included 19 patients with 20 nodules, comprising 3 benign nodules and 17 malignant nodules. The CZCH dataset was acquired using a United Imaging UMI780 PET/CT scanner and included 18 patients with 22 nodules, comprising 4 benign nodules and 18 malignant nodules. These nodules included 35 malignant lesions, comprising 1 pre-invasive lesion (AIS, 2.9%), 4 MIAs (11.4%), and 30 IACs (85.7%). The 7 benign nodules included 4 inflammatory lesions or inflammatory pseudotumors, 1 fungal infection, and 2 nodules that resolved during follow-up. The numbers of nodules from the two external test centers were similar (20 vs. 22 nodules). However, the overall distribution across the three scanners was not fully balanced because scanner type corresponded to the data source center.

PET/CT imaging, breath-hold chest CT imaging, image reconstruction, and analysis

In this study, the three PET/CT centers used a Siemens Biograph mCT (64) PET/CT scanner, a Siemens Biograph mCT Flow PET/CT scanner, and a United Imaging UMI780 PET/CT scanner, respectively. The corresponding scanner and case distributions have been described in the Study Subjects section. All patients were required to fast for 4–6 h before PET/CT imaging. After confirming that fasting blood glucose levels were below 11.1 mmol/L, an intravenous injection of ¹⁸F-FDG (radiochemical purity > 95%) was administered at a 3.7–4.44 MBq/kg dose. Patients rested in a quiet, warm, and dimly lit environment for 45–60 min before undergoing bladder emptying and PET/CT scanning. Immediately following the PET/CT scan, a breath-hold chest CT was performed using the same equipment. Thin-slice reconstructions were conducted for the regions containing ground-glass nodules (GGNs). The acquisition and reconstruction parameters were as follows: Tube current: Adjusted automatically based on anatomical structure and tissue density; Tube voltage: 140 kV; Rotation time: 0.5 s; Pitch: 0.6; Matrix size: 512 × 512; Reconstruction slice thickness: 3.0 mm; Lung window settings: Window width of 1200 HU and window level of -600 HU.

Two nuclear medicine physicians, each with over five years of experience, independently and blindly analyzed the images to obtain the CT semantic features, quantitative parameters, and PET functional parameters of the GGNs. (1) CT Morphological Features: GGN number: (Single or multiple); Type (Pure ground-glass nodule (pGGN) or mixed ground-glass nodule (mGGN)); Location: Central or peripheral; Shape: Round/oval or irregular; Edge characteristics: Smooth or lobulated; Additional features: Bronchus abnormality, vacuole sign, pleural indentation, and vascular convergence sign. (2) CT Quantitative Parameters: Maximum nodule diameter (D_GGN, the longest diameter of the GGN measured in the lung window’s axial section); Maximum diameter of the solid component (D_solid, the longest diameter of the solid component within the nodule, measured in the lung window’s axial section); Attenuation value of the ground-glass component (CT_GGO, reflecting the density of the ground-glass portion); Consolidation-to-tumor ratio (CTR, the ratio of D_solid to D_GGN). (3) PET Functional Parameters: SUVmax (maximum standardized uptake value of the GGN); SUVindex (the ratio of the GGN’s SUVmax to the mean SUV of the liver, SUVmax_GGNs/SUVmean_Liver). To evaluate interreader agreement in image interpretation, Cohen’s κ was used for categorical variables, and intraclass correlation coefficients (ICCs) were used for continuous quantitative parameters. Discrepancies in categorical variables between the two readers were resolved by consensus after joint review. For continuous quantitative parameters, the mean values of the two readers’ measurements were used for subsequent analysis.

Development of diagnostic models for differentiating malignant and benign GGNs based on radiomic handcrafted and deep features

Lesion segmentation

Two nuclear medicine physicians, each with over five years of experience, used the 3D Slicer software (version 4.11.2, http://www.slicer.org) to perform semi-automatic, slice-by-slice segmentation of regions of interest (ROIs) on PET and CT images in a blinded manner. One physician repeated the semi-automatic segmentation process on PET and CT images four weeks later to validate segmentation consistency. Intra-class correlation coefficients (ICCs) were calculated to evaluate inter-observer and intra-observer consistency, ensuring the stability and reliability of the segmentation results.

Image preprocessing

The image preprocessing steps included intensity normalization to minimize variability caused by differences in imaging devices and resampling to ensure isotropy and rotation invariance, using the sitkBSpline algorithm (third-order B-spline interpolation) to resample CT images to a voxel spacing of 1 × 1 × 1 mm³ and PET images to 3 × 3 × 3 mm³. Fixed bin-width discretization was applied to pixel intensities, with a bin width 25 for CT images and 0.313 for PET images. Multi-scale feature sets were generated using image enhancement techniques, including Laplacian of Gaussian (LOG) filtering with a sigma range of 0.5 to 5 (incremented by 0.5) and wavelet transforms, which applied high-pass and low-pass filters in all three dimensions to produce eight decomposition images (HHH, HHL, HLH, HLL, LHH, LHL, LLH, and LLL). Extracted intensity, histogram, and texture features underwent further discretization, logarithmic transformation, and wavelet processing to optimize feature representation and improve model classification performance.

Extraction of radiomic handcrafted features

Using Pyradiomics software (version 3.0.1, http://www.radiomics.io/pyradiomics.html), a total of 1,781 radiomic features were extracted from CT and PET images based on 3D masks. The feature categories included:

Shape features (14 features), first-order statistics (18 features), Gray Level Co-occurrence Matrix (GLCM, 24 features), Gray Level Dependence Matrix (GLDM, 14 features), Gray Level Run Length Matrix (GLRLM, 16 features), Gray Level Size Zone Matrix (GLSZM, 16 features), and Neighboring Gray Tone Difference Matrix (NGTDM, 5 features).

Extraction of deep features

To extract deep learning features, CT and PET images were preprocessed to isolate lesion regions by combining images with their corresponding masks. CT images were cropped to 64 × 64 × 64 voxels and normalized to a range of [-1000 HU, 400 HU]. PET images were cropped to 32 × 32 × 32 voxels and normalized to a range of [0, 15]. The CT images were converted to pseudo-RGB format by duplicating each slice into three channels, creating 64 pseudo-RGB images (size 64 × 64 × 3) for input. Two pre-trained convolutional neural networks, VGG19 and ResNet50, trained on the ImageNet dataset, were used to extract deep features. These networks processed each pseudo-RGB image independently, generating flattened feature maps into one-dimensional vectors. The features from all slices were concatenated into a single feature set. Finally, the features extracted based on VGG19 include 131,072 for CT images and 16,384 for PET images; the features extracted based on ResNet50 include 524,288 for CT images and 65,536 for PET images. This process enabled the combination of spatial and contextual information from lesion regions for enhanced model performance.

Feature selection

To avoid model overfitting, a multi-stage feature selection process was applied to both handcrafted and deep features. First, an unsupervised variance filtering method was used to remove features with low variance, with the threshold set at 0.19; this step did not use the benign/malignant labels of the nodules. Subsequently, the development set was stratified and randomly sampled in a 7:3 ratio to form the training set (n = 99, including 25 benign nodules, accounting for 25.3%) and the internal validation set (n = 43, including 11 benign nodules, accounting for 25.6%). In the training set, the Mann-Whitney U test was used to identify candidate features associated with nodule malignancy (P < 0.1). The MinMaxScaler normalization parameters were then fitted using only the training set and were subsequently applied to the internal validation set and external test set. After normalization, the least absolute shrinkage and selection operator (LASSO) algorithm was used to further select the optimal predictive features [12]. The LASSO algorithm reduced the risk of overfitting through L1 regularization, and 5-fold cross-validation was used to select the regularization parameter. Finally, a Spearman correlation matrix was calculated based on the features selected in the training set to remove highly correlated redundant features (r > 0.85). The fixed feature set determined in the training set was then used for model construction, internal validation, and external testing. It should be noted that the 5-fold cross-validation in LASSO was used for regularization parameter selection within the training set, and the complete feature selection pipeline was not repeated independently within each cross-validation fold of the subsequent SVM grid search.

Classifier and modeling

A support vector machine (SVM) was selected as the classifier for this study. Feature selection and model construction were performed using the open-source Python library Scikit-Learn [13]. To address class imbalance, the SVM classifier was configured with class_weight=’balanced’, which automatically adjusts class weights inversely proportional to class frequencies. During training, stratified 5-fold cross-validation combined with grid search was performed within the training set based on the fixed feature set to optimize the SVM hyperparameters and determine the optimal classifier configuration, while maintaining the original class distribution in each cross-validation fold. This cross-validation procedure was mainly used for classifier hyperparameter optimization rather than for repeating the complete feature selection pipeline. After the final model was determined in the training set, it was applied to the internal validation set and external test set for performance evaluation. Model performance was assessed using the area under the curve (AUC), with particular attention to metrics suitable for imbalanced classification tasks.

Statistical analysis

Quantitative data following normal distribution were expressed as mean ± standard deviation (SD), while non-normally distributed quantitative data were expressed as median (interquartile range) [M (P25, P75)]. Qualitative data were presented as frequency and percentage. Between-group differences were analyzed using an independent samples t-test or Mann-Whitney U test for quantitative data and the χ² test or Fisher’s exact test for qualitative data. The performance of the radiomics-based machine learning classifiers was evaluated using metrics such as area under the curve (AUC), accuracy, precision, sensitivity (recall), and specificity. The 95% CI for AUC was calculated using MedCalc statistical software (version 19.0.7, Ostend, Belgium) [14]. Comparisons of the AUC of receiver operating characteristic (ROC) curves were performed using the DeLong test [15]. The 95% CIs for accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) in the training set and internal validation set were calculated using the Clopper-Pearson exact binomial method. Given the limited number of benign nodules in the external test set, we adopted Bootstrap resampling with 1000 replicates for CI calculation. The calibration performance of model probability outputs was quantified by the Brier score, and calibration curves were plotted accordingly. A lower Brier score indicates a closer match between predicted probabilities and actual outcomes. Decision curve analysis (DCA) was performed to assess the net benefit of each model at different threshold probabilities for clinical decision-making. Statistical analyses were conducted using the “scikit-learn,” “SciPy,” and “math” libraries in the Python programming language. All statistical tests were two-tailed, and a P-value < 0.05 was considered statistically significant.

This study was reported in accordance with the TRIPOD reporting guideline and its AI extension for prediction model studies involving artificial intelligence and machine learning. The completed TRIPOD + AI checklist is provided as Supplementary material 1.

Results

General characteristics of patients in the development and external test sets

The development set included 136 patients (142 pulmonary GGNs) with an average age of 62.5 ± 10.0 years (range: 31.0–82.0 years). The proportion of males was 36.0% (49/136), and the proportion of smokers was 20.6% (28/136). Among these patients, 77.2% (105/136) were diagnosed with malignant nodules based on postoperative pathology. There were no cases where both benign and malignant GGNs coexisted in the same patient. Further analysis revealed that patients with malignant GGNs were significantly older than those with benign GGNs (63.9 ± 9.3 years vs. 57.5 ± 10.7 years, P = 0.002), and the proportion of females was higher among patients with malignant GGNs (70.5% vs. 41.9%, P = 0.004).

The external test set included 37 patients (42 pulmonary GGNs) with an average age of 63.7 ± 8.6 years (range: 44.0–79.0 years). The proportion of males was 43.2% (16/37). Among these patients, 83.7% (31/37) were diagnosed with malignant nodules based on postoperative pathology.

Comparison of routine PET/CT parameters between benign and malignant GGNs in the development set

Before comparing conventional PET/CT parameters between benign and malignant GGNs, two physicians independently evaluated the imaging semantic features and quantitative parameters of all included GGNs. The Cohen’s κ values for CT morphological categorical variables ranged from 0.75 to 0.92, indicating good to excellent interreader agreement. For continuous quantitative parameters, including D_GGN, D_solid, and SUVmax, the ICC values ranged from 0.81 to 0.93, indicating high interreader agreement. The development set included 142 pulmonary GGNs, of which 74.6% (106/142) were malignant and 25.4% (36/142) were benign. Significant differences were observed in routine PET/CT parameters between benign and malignant GGNs (see Table 1). Among malignant GGNs, the proportion of mixed ground-glass nodules (mGGNs) was significantly higher compared to benign GGNs (89.6% vs. 75.0%, P = 0.029), and malignant GGNs were more frequently associated with lobulation (71.7% vs. 27.8%, P < 0.001), bronchus sign (91.5% vs. 72.2%, P = 0.003), and pleural indentation (79.2% vs. 27.8%, P < 0.001). Furthermore, the lesion diameter (D_GGN) and solid component diameter (D_solid) of malignant GGNs were significantly larger than those of benign GGNs (P = 0.041 and P = 0.002, respectively). Other parameters, such as CT_GGO, SUVmax, and SUVindex, showed no significant differences between the two groups.

Table 1.

Comparison of Routine PET/CT Parameters Between Benign and Malignant GGNs in the Development Set

Parameter	Benign (n = 36)	Malignant (n = 106)	P-value
GGNs type			0.029*
pGGN	9 (25.0%)	11 (10.4%)
mGGN	27 (75.0%)	95 (89.6%)
GGNs location			0.607
Top right	15 (41.7%)	40 (37.7%)
Middle right	1 (2.8%)	7 (6.6%)
Bottom right	7 (19.4%)	13 (12.3%)
Top left	10 (27.8%)	30 (28.3%)
Bottom left	3 (8.3%)	16 (15.1%)
Shape			0.385
Round/Oval	16 (44.4%)	56 (52.8%)
Polygonal/Irregular	20 (55.6%)	50 (47.2%)
Lobulation sign	10 (27.8%)	76 (71.7%)	< 0.001*
Spiculation sign	24 (66.7%)	71 (67.0%)	0.972
Bronchus sign	26 (72.2%)	97 (91.5%)	0.003*
Cavitation sign	8 (22.2%)	26 (24.5%)	0.779
Pleural indentation sign	10 (27.8%)	84 (79.2%)	< 0.001*
DGGN (mm)	22.6 ± 11.3	26.5 ± 9.3	0.041*
DSolid (mm)	7.6 (2.7–12.0)	13.3 (6.0-17.8)	0.002*
CTR	0.4 (0.1–0.6)	0.5 (0.3–0.7)	0.117
CTGGO (HU)	-483.2 ± 138.0	-480.9 ± 115.6	0.921
SUVmax	3.5 (2.2–8.1)	3.1 (2.1–4.4)	0.228
SUVindex	1.4 (0.8–3.4)	1.2 (0.8–1.7)	0.165

Results are represented as Mean ± SD / Median (Q1-Q3) / N(%), *Indicates statistically significant differences. pGGN pure ground-glass nodule, mGGN mixed ground-glass nodule, D_GGN diameter of the GGN, D_solid diameter of the solid component, CT_GGO attenuation value of the GGO component on CT, CTR  D_solid/D_GGN, SUVindex SUVmax_GGNs/SUVmean_Liver

Diagnostic reference model for benign and malignant GGNs based on PET/CT clinical and imaging features

After feature selection, 9 clinical indicators and routine PET/CT parameters with potential significance for distinguishing benign and malignant pulmonary GGNs were identified in the training set. These included age, gender, smoking history, GGN type, lobulation, bronchus sign, pleural indentation, D_GGN, and D_solid. Through differential testing and LASSO regression analysis, 5 key features (age, gender, lobulation, pleural indentation, and D_solid) were retained for constructing the reference model. Spearman correlation analysis confirmed low correlations among the 5 features (all r < 0.85), indicating no significant redundancy. The reference model achieved AUCs of 0.871 (95% CI: 0.788–0.903), 0.864 (95% CI: 0.725–0.949), and 0.824 (95% CI: 0.676–0.924) in the training, internal validation, and external test sets, respectively (Fig. 2 and Table S1 in Supplementary material 2), indicating that it maintained acceptable diagnostic performance in the external test set. The accuracy, sensitivity, specificity, PPV, and NPV of the model in the external test set, along with their 95% CIs, are presented in Table 2.

Fig. 2.

ROC curves and AUC values of the reference model on the training, internal validation, and external test sets

Table 2.

Comparison of model performance on the external test set

Model	AUC (95%CI)	Cutoff	Accuracy (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	B: TN-FP	M: TP-FN
Reference model	0.824 (0.676–0.924)	0.397	0.905 (0.810–0.976)	0.943 (0.857-1.000)	0.714 (0.429-1.000)	0.943 (0.886-1.000)	0.714 (0.444-1.000)	5 − 2	33 − 2
PET_ResNet50	0.884 (0.747–0.962)	0.784	0.762 (0.643–0.881)	0.714 (0.571–0.857)	1.000 (1.000–1.000)	1.000 (1.000–1.000)	0.412 (0.318–0.583)	7 − 0	25 − 10
PET_ResNet50/CT_VGG19	0.927 (0.802–0.984)	0.728	0.881 (0.786–0.976)	0.886 (0.771–0.971)	0.857 (0.571-1.000)	0.969 (0.912-1.000)	0.600 (0.400-0.875)	6 − 1	31 − 4

The 95% CIs for accuracy, sensitivity, specificity, PPV and NPV in the external test set were calculated using Bootstrap resampling with 1000 replicates. B Benign, M Malignant

Development of benign and malignant pulmonary GGNs diagnostic model based on radiomic handcrafted features and deep features

Based on the five clinical/conventional imaging features included in the reference model (age, sex, lobulation sign, pleural indentation sign, and D_solid), PET- or CT-derived radiomics handcrafted features and deep features were further incorporated to develop six single-modality combined diagnostic models integrating clinical/conventional imaging features with image-derived features. These models included CT_HF, CT_VGG19, CT_ResNet50, PET_HF, PET_VGG19, and PET_ResNet50. The number of features included in each model was as follows: CT_HF: 20 features; CT_VGG19: 3 features; CT_ResNet50: 33 features; PET_HF: 9 features; PET_VGG19: 3 features; PET_ResNet50: 5 features. The feature heatmaps, feature weights, and predictive performance of each model are provided in Supplementary material 2, Figures S1-S6.

The AUC values of each model in the training, internal validation, and external test sets are shown in Fig. 3, and the AUCs with their 95% CIs are presented in Supplementary material 2, Table S1. In the internal validation set, the CT_HF and PET_HF models achieved AUCs of 0.889 (95% CI: 0.756–0.964) and 0.903 (95% CI: 0.774–0.972), respectively, which were numerically higher than that of the reference model (AUC = 0.864, 95% CI: 0.725–0.949). In the external test set, CT_VGG19, PET_HF, and PET_ResNet50 achieved AUCs of 0.851 (95% CI: 0.707–0.942), 0.833 (95% CI: 0.686–0.930), and 0.884 (95% CI: 0.747–0.962), respectively, which were numerically higher than that of the reference model (AUC = 0.824, 95% CI: 0.676–0.924). Among the deep feature-based models, CT_ResNet50 achieved an AUC of 0.821 (95% CI: 0.674–0.921) in the internal validation set, which was numerically higher than that of CT_VGG19 (AUC = 0.793, 95% CI: 0.642–0.901). However, in the external test set, CT_VGG19 achieved a higher AUC than CT_ResNet50 (0.851, 95% CI: 0.707–0.942 vs. 0.551, 95% CI: 0.390–0.705). PET_ResNet50 achieved AUCs of 0.838 (95% CI: 0.694–0.933) and 0.884 (95% CI: 0.747–0.962) in the internal validation and external test sets, respectively, both of which were numerically higher than those of PET_VGG19 (0.814, 95% CI: 0.666–0.916 and 0.767, 95% CI: 0.611–0.884, respectively).

Fig. 3.

AUC values of each model for diagnosing lung GGNs across the training, internal validation, and external test sets

To further improve the diagnostic performance for differentiating benign and malignant pulmonary GGNs, handcrafted features or deep features from dual-modality imaging were further incorporated based on the five clinical/conventional imaging features included in the reference model. Accordingly, two dual-modality combined diagnostic models, PET_HF/CT_HF and PET_ResNet50/CT_VGG19, were developed. These models included PET_HF/CT_HF and PET_ResNet50/CT_VGG19, containing 5 and 6 features, respectively (feature heatmaps, weights, and predictive performance of each model are provided in Supplementary material 2, Figures S7-S8). Since CT handcrafted features were not included in the dual-modality handcrafted feature model, the PET_HF/CT_HF model achieved an AUC of 0.906 (95% CI: 0.778–0.974) in the internal validation set, showing performance comparable to that of the single-modality PET_HF model. In the external test set, the PET_HF/CT_HF model achieved an AUC of 0.849 (95% CI: 0.705–0.941), whereas the PET_ResNet50/CT_VGG19 model achieved an AUC of 0.927 (95% CI: 0.802–0.984). These findings suggest that, in the external test set of this study, the model integrating PET_ResNet50 and CT_VGG19 deep features showed relatively high discriminatory ability. The statistical differences between this model and the main comparison models, as well as other diagnostic metrics, are presented in Sect.  5 and Table 2.

Comparative analysis of diagnostic models for benign and malignant pulmonary GGNs

Based on the generalization performance in the external test set, the two models with the highest AUC values (PET_ResNet50 and PET_ResNet50/CT_VGG19) were compared with the reference model in terms of overall performance (see Table 2; Fig. 4). The three models had similar feature counts: 5 for the reference model, 5 for PET_ResNet50, and 6 for PET_ResNet50/CT_VGG19. In the external test set, the PET_ResNet50/CT_VGG19 model achieved an AUC of 0.927 (95% CI: 0.802–0.984), compared with 0.824 (95% CI: 0.676–0.924) for the reference model and 0.884 (95% CI: 0.747–0.962) for PET_ResNet50. However, the differences among the three models were not statistically significant (P = 0.101–0.464; see Fig. 4A). Meanwhile, the accuracy, sensitivity, specificity, PPV, and NPV of the three models in the external test set, along with their 95% CIs, are listed in Table 2.

Fig. 4.

Performance comparison of PET_ResNet50, PET_ResNet50/CT_VGG19, and the reference model on the external test set. A Comparison of ROC curves; B Decision curve analysis. The basic model shown in the figure refers to the reference model constructed using clinical/conventional imaging features

From the DCA, the PET_ResNet50/CT_VGG19 model showed the relatively highest net benefit when the threshold probability exceeded 0.5 (Fig. 4B). To complement the calibration curve analysis (Supplementary material 2, Figures S9-S11), we further calculated the Brier score as a quantitative calibration-related metric (Supplementary material 2, Table S2). In the external test set, the Brier scores of the reference model, PET_ResNet50, and PET_ResNet50/CT_VGG19 were 0.122, 0.119, and 0.122, respectively. These results indicate that, while the PET_ResNet50/CT_VGG19 model showed relatively high discriminatory ability and decision net benefit, its overall probability prediction error was comparable to that of the reference model. Given the limited number of benign nodules in the external test set, the probability calibration, generalizability, and clinical interpretation of the model should still be interpreted cautiously and require further validation in larger external cohorts.

Post-hoc analysis

To further validate the interpretability of deep features and explore the distribution patterns of handcrafted and deep features in benign and malignant nodules, the study utilized features selected through LASSO. These included 16 CT handcrafted features, 1 deep feature from VGG19, 5 PET handcrafted features, 3 deep features from ResNet50, and select clinical variables (age, sex, lobulation sign, pleural indentation sign, D_solid). A feature visualization heatmap was generated using the development set (Fig. 5) alongside unsupervised clustering analysis.

Fig. 5.

Heatmap of radiomics handcrafted and deep feature expressions with unsupervised clustering

The analysis revealed that all features clustered into two major categories based on similarity, as illustrated by the dendrogram on the left of Fig. 5. The first category primarily comprised most CT handcrafted features, such as CT_wavelet-LHL_gldm_DependenceVariance and CT_wavelet-LHL_glrlm_LongRunEmphasis. The “hot zones” of these features were predominantly concentrated in samples of malignant nodules. The second category included some handcrafted features (e.g., PET_wavelet-HHL_glrlm_LongRunLowGrayLevelEmphasis) and all deep features (e.g., PET29902, PET35219, PET36534, and CT74230). The deep features were located adjacent to certain handcrafted features in the clustering. These two major categories’ clustering patterns are strongly associated with the sample labels (benign/malignant). Additionally, correlation analysis (Fig. 6) indicated a moderate correlation among deep features (e.g., PET35219 and PET36534, r = 0.53) but weak correlations between deep features and handcrafted and clinical features.

Fig. 6.

Correlation coefficient matrix of clinical features, radiomics handcrafted features, and deep features

Discussion

This study developed diagnostic models for differentiating between benign and malignant persistent pulmonary GGNs based on conventional imaging features, radiomics handcrafted features, and deep features extracted from ¹⁸F-FDG PET/CT. The results showed that the combined model integrating PET_ResNet50 and CT_VGG19 deep features achieved the highest AUC in the external test set (0.927, 95% CI: 0.802–0.984) and showed a relatively high net benefit in DCA. The Brier score results indicated that the overall probability prediction error of this combined model was comparable to that of the reference model. These findings suggest that the combined model had relatively high discriminatory ability; however, its probability calibration, generalizability, and clinical applicability require further validation in larger external cohorts.

The malignancy risk assessment of pulmonary nodules is a key factor influencing nodule management guidelines, typically relying on clinical expertise [6]. Despite advancements in artificial intelligence and computer-aided diagnosis (CAD) technologies, physicians in real clinical settings predominantly base their evaluations on clinical features (e.g., age, sex) and semantic imaging characteristics (e.g., lesion shape, margins, and distribution) when determining the benign or malignant nature of pulmonary nodules. This study constructed a reference model based on clinical features and conventional PET/CT imaging parameters to compare the performance of different imaging modalities and feature types (conventional parameters, radiomics handcrafted features, and deep features) in diagnosing pulmonary GGNs. During the development of the reference model, the study found that only conventional CT parameters (e.g., lobulation sign, pleural indentation sign, and D_solid) were incorporated into the model, while conventional PET metabolic parameters (e.g., SUVmax and SUVindex) were excluded. This exclusion may be attributed to the role of ¹⁸F-FDG in both tumor and inflammation imaging. Inflammatory conditions often result in abnormal radiotracer uptake due to the aggregation of inflammatory cells and increased glucose metabolism. Many pulmonary diseases, such as tuberculosis, cryptococcosis, and organizing pneumonia, exhibit significantly elevated tracer uptake [16]. In contrast, adenocarcinoma-related pulmonary GGNs, characterized by low cellular atypia, fewer tumor cells, and slower proliferation rates, generally display lower tracer uptake, consistent with previous studies [17, 18]. Moreover, the study observed a notable decline in accuracy for the reference model based on conventional imaging parameters when applied to the external test set, highlighting the limited generalization capability of models relying solely on traditional imaging parameters.

Numerous studies have demonstrated that CT radiomics features can effectively differentiate benign from malignant pulmonary nodules, with sensitivity ranging from 76.2% to 92.85% and specificity ranging from 72.73% to 96.1% [7]. In this study, the diagnostic model developed based on CT radiomics features achieved a slightly higher AUC than the reference model in the training set (0.881 vs. 0.871) but a lower AUC in the external test set (0.808 vs. 0.824), indicating limitations in the robustness and generalizability of this model. Research by Niu et al. [11] highlighted that incorporating FDG PET metabolic parameters (e.g., SUVmax) into CT radiomics models significantly improved the diagnostic performance for differentiating benign from malignant pulmonary GGNs, with the AUC increasing from 0.704 to 0.940. This suggests that integrating multimodal imaging features may enhance diagnostic efficacy. Kang et al. [19] further emphasized that diagnostic models combining CT and FDG PET radiomics features outperformed single-modality models for solitary pulmonary nodule diagnosis (combined: AUC 0.929; CT: AUC 0.819; PET: AUC 0.833). Similarly, this study found that while conventional PET metabolic parameters had limited diagnostic value for pulmonary GGNs, the diagnostic models based on PET radiomics handcrafted features showed numerically higher AUCs than the reference model in both the internal validation set and external test sets. This may be because PET radiomics features reflect tumor metabolic information and capture characteristics related to tumor phenotype, microenvironment, and other attributes [20]. Furthermore, studies suggest that specific imaging features can reveal subvisual pathological information and may even correlate with underlying genotypic variations [21].

As a class of multi-layer feedforward neural networks, deep learning networks can take images as input and utilize supervised learning to perform end-to-end training, extracting highly discriminative image features [22]. Compared to traditional methods, these networks can capture a greater number and higher-order imaging features [23], playing pivotal roles in applications such as pulmonary GGN image reconstruction [24], lesion segmentation [25], benign-malignant classification [26], and invasiveness assessment [27]. VGG19 and ResNet50 are widely used convolutional neural network architectures in deep learning. However, prior research has mainly focused on their application in single-modal imaging, with relatively limited exploration of their potential in multimodal imaging, particularly PET/CT. This study utilized pre-trained VGG19 and ResNet50 models and adapted three-dimensional medical images into a two-dimensional format suitable for convolutional networks through slice extraction and pseudo-RGB conversion. This approach conserved computational resources and effectively captured high-level semantic features. Extracting many deep features laid the foundation for subsequent feature selection and modeling. Wang et al. [9] demonstrated that deep learning diagnostic models based on CT images, as well as combined models (integrating deep features from CT images and clinical characteristics), exhibit superior diagnostic performance in distinguishing benign from malignant pulmonary GGNs, aiding clinical decision-making. Similarly, Zhang et al. [24] found that models extracting deep features using ResNet architecture achieved high diagnostic efficacy in differentiating lung cancer from tuberculosis. This study found that some deep feature-based models showed relatively high AUCs in the external test set. In the external test set, CT_VGG19 and PET_ResNet50 models showed numerically higher AUCs than the reference model in diagnosing pulmonary GGNs (AUCs: CT_VGG19 = 0.851, PET_ResNet50 = 0.884). Moreover, the combined model constructed using deep features from both imaging modalities achieved the highest AUC in the external test set and showed a relatively high net benefit in DCA. However, its clinical applicability should be further validated in larger and more balanced external cohorts.

This study further explored the interpretability of deep features and the distribution patterns of handcrafted and deep features in benign and malignant pulmonary nodules. The study revealed the clustering relationships among different feature categories and their correspondence to sample labels by generating feature visualization heatmaps and conducting unsupervised clustering analysis. The exploratory clustering and correlation analyses suggested that deep features were not simple duplicates of handcrafted or clinical features. Some deep features clustered near selected PET/CT handcrafted features but showed relatively weak correlations with most clinical and handcrafted features, indicating potential complementarity among different feature types. These findings may partly explain the performance of the combined deep-feature model. However, the biological meaning of these deep features remains uncertain and requires further validation in larger cohorts with pathological or molecular correlates.

Limitations

This study has several limitations: First, the study population consisted of patients with suspected malignant persistent pulmonary GGNs undergoing PET/CT, resulting in a relatively small and imbalanced dataset. In particular, the external test set included only seven benign nodules, which may lead to unstable estimates of specificity, predictive values, calibration, and generalizability, and may increase the risk of model overfitting. Second, because this was a retrospective multicenter study using PET/CT data acquired from three different scanners, potential scanner-related variability cannot be fully excluded. Although standardized preprocessing, intensity normalization, resampling, and fixed bin-width discretization were applied, formal scanner-level harmonization, such as ComBat correction, and scanner-stratified sensitivity analysis were not performed. Third, deep features were extracted using VGG19 and ResNet50 models pre-trained on ImageNet without fine-tuning on pulmonary nodule-specific data, potentially reducing the specificity of the features for GGNs. Fourth, converting 3D images into pseudo-RGB format for 2D convolutional networks may lead to the loss of spatial information in the depth dimension, while the simple replication of channels during processing may introduce irrelevant information, limiting the effective utilization of PET metabolic features. Fifth, the high-dimensional deep learning features may contain redundancy, increasing computational complexity and the risk of overfitting, which restricts the model’s applicability in clinical practice. Finally, this study developed and compared multiple single-modality and multimodal diagnostic models, but Bonferroni correction or false discovery rate (FDR) correction was not applied to all possible pairwise model comparisons. Therefore, the statistical interpretation of performance differences among models should be made with caution, and these results should be regarded primarily as exploratory comparisons for model comparison and selection. In addition, this study did not further construct a traditional logistic regression-based nomogram model. Future studies with larger multicenter external validation cohorts may further develop and validate visual clinical decision-support tools and incorporate multiplicity adjustment to assess the statistical significance of performance differences among models.

Conclusion

This study systematically compared diagnostic models for differentiating benign and malignant persistent pulmonary GGNs developed using conventional imaging features, radiomics handcrafted features, and deep features derived from ¹⁸F-FDG PET/CT. The results showed that the combined model integrating PET_ResNet50 and CT_VGG19 deep features had relatively high discriminatory ability in the external test set and showed a relatively high net benefit in DCA. However, given the limited number of benign nodules in the external test set and the need for further validation of model probability calibration and generalizability, the findings of this study should be considered an auxiliary reference for risk stratification and clinical decision-making in pulmonary GGNs, rather than a replacement for comprehensive clinical judgment by physicians.

Abbreviations

GGNs

ground-glass nodules

pGGNs

Pure ground-glass nodules

mGGNs

mixed ground-glass nodules

HRCT

High-resolution CT

DL

Deep learning

ROIs

Regions of interest

ICCs

Intra-class correlation coefficients

LOG

Laplacian of Gaussian

LASSO

Least absolute shrinkage and selection operator

SVM

Support vector machine

AUC

Area under the curve

SD

Standard deviation

ROC

Receiver operating characteristic

DCA

Decision curve analysis

PPV

Positive predictive value

NPV

Negative predictive value

Authors’ contributions

YS and XLS were responsible for the literature search and manuscript writing. YS and XLS contributed to content planning and editing. JG, XG, SJY, and RN participated in data collection and analysis. JZ, XBM, and XNS proposed the study concept, supervised the project, and revised the manuscript. YW provided overall guidance and critical review. All authors have read and approved the final manuscript for publication.

Funding

This study was supported by Top Talent of Changzhou “The 14th Five-Year Plan” High-Level Health Talents Training Project (2022CZBJ037); Top Talent of Changzhou “The 14th Five-Year Plan” High-Level Health Talents Training Project (2024CZBJ008); Changzhou Clinical Medical Center (Nuclear Medicine) (CZZX202204); Postgraduate Research & Practice Innovation Program of Jiangsu Province (SJCX24_1820).

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

This retrospective multicenter study was conducted in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of the Third Affiliated Hospital of Soochow University (No. [2020] KD 075). Given the retrospective nature of this study, the requirement for informed consent was waived by the Ethics Committee of the Third Affiliated Hospital of Soochow University.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.