Pseudo-siamese network combined with dosimetric and clinical factors, radiomics features, CT images and 3D dose distribution for the prediction of radiation pneumonitis: A feasibility study

Highlights

• •We firstly proposed a novel method that combined dosimetric and clinical factors, radiomics feature, 3D CT images and dose distribution for predicting RP.
• •A pseudo-siamese network were developed for combination of 1D and 3D information.
• •The used deep learning frame showed a good accuracy (81.8%) for the prediction of RP.

Abstract

Purpose

Radiation pneumonitis (RP)(grade ≥ 2) can have a considerable impact on patient quality-of life. In previous studies, the traditional method commonly used radiomics and clinical factors for RP prediction. This study aims to develop and evaluate a novel pseudo-siamese network (PSN) to assist radiologists predict RP before radiotherapy based on combination of dosimetric and clinical factors, radiomics features, CT (computed tomography) images, and dose distribution (hybrid model).

Method

One hundred and ten patients with lung cancer (19 RP ≥ 2) who received radiotherapy between 2016 and 2020 were retrospectively enrolled in this study. Dosimetric factors were calculated from DVH (dose-volume histogram), such as lung mean dose, lung V₅, and prescription dose. Clinical characteristics were recorded, such as age, sex, smoking status, TN stage, and overall stage. A total of 1419 radiomics features were extracted. Cluster analysis was used for detecting radiomics features that associated with RP. Patients were randomly split into a training set (90 %, 85 non-RP, and 14 RP) and a validation set (10 %, 6 non-RP, and 5 RP). A PSN architecture was designed for combining 1D (dosimetric and clinical factors, radiomics) and 3D (CT images, 3D dose distribution) features. 5-fold cross-validation procedure for estimating the skill of the model on new data.

Results

For cluster analysis, totally of 106 radiomics features with high correlation were selected. The accuracy was 0.727, 0.636, 0.545, and 0.727 for input dosimetric and clinical factors, dose distribution, CT images, and radiomics features, respectively. The accuracy of hybrid model was 0.818. The sensitivity of hybrid model was 0.800 (95 % confidence interval (CI) [0.299, 0.989]), and specificity was 0.833(95 % CI [0.364, 0.991]). The areas under the receiver operating characteristic curves (AUCs) result in 5-fold cross-validation was 0.77–0.90(mean AUC ± std was 0.85 ± 0.05).

Conclusion

This study firstly propose method that the combination of high dimensional and low dimensional features for RP prediction. The results confirm the feasibility of multi-dimensional features predict RP.

Introduction

Lung cancer is the most frequently occurring cancer, and leading cause of cancer death (18.0 % of the total cancer deaths) [1]. Radiation therapy (RT) is an importance treatment strategy that plays a crucial role in the lung cancer. Local tumor control and overall survival can be improved by high-dose radiation [2], [3]. Nevertheless, the risk of radiation pneumonitis (RP) may simultaneously increase. RP is the most common dose-limiting complication of lung radiation, which can have a considerable impact on patient quality-of-life and respiratory function [4].

Previous studies report that RP have been proved associated with dosimetric factors that were calculated from the dose-volume histogram (DVH), such as plan target volume (PTV) D₉₅ (means dose that covers 95 % of the PTV), lung V₂₀(means lung volume receiving ≥ 20 Gy) and lung mean dose, et al. [5], [6], [7]. Krafft et al. [8] developed clinical and dosimetric factors-based model for RP prediction, the result were shown that the areas under receiver operating characteristic curves (AUC) were only 0.68. However, the ideal DVH metric with outperformer operating characteristics, either alone or with other predictive variable in a model, for RP risk prediction before radiotherapy has not yet been identified [4]. Radiomics extraction can provide a large number of quantitative images features, which tends to be hard for eyes to recognize [9], [10], [11]. Luo et al. [12] extracted radiomics features from positron emission tomography and combined with clinical and dosimetric factors, the AUC for predicting RP achieved 0.77. Jiang et al. [13] combine dosimetric factors and multi-target volume radiomics features from planning CT images for the prediction of symptomatic RP, the AUC has improved significantly comparing with previous studies.

In addition, the above studies were combined radiomics and dosimetric factors as input, and machine learning algorithm were commonly used. Due to dosimetric factors and radiomics feature are one-dimensional (1D) and discrete variables, some features associated with RP (such as three-dimensional(3D) CT image and dose distribution) were neglected. These factors assume a uniform distribution of pulmonary function and therefore a homogeneous dose response for sub-volume [14]. On the contrary, the true dose distribution and gradient were heterogeneous yet. Moreover, the main difference between deep learning and traditional machine learning is that its performance increases as the data size increases [15]. In terms of practical application, many studies have that deep learning can reach or exceed that of traditional machine learning if data adequate [16]. Deep learning radiomics, a novel developed method, can support quantitative and high-throughput features from images by convolution [11], [17]. And deep learning algorithm were able to extract spatial features from high-dimensional data. Therefore, 3D dose distribution and CT image may able to improve the performance of RP prediction.

Here, we proposed a novel network architecture that combination of 1D radiomics features, dosimetric and clinical factors, and 3D CT image and dose distribution features for predicting the incidence of RP after radiotherapy.

Methods

Patients and CT image acquisition

This study retrospectively 110 patients with stage I-IV lung cancer and received radiotherapy, including 66 IMRT (Intensity modulated radiation therapy) and 44 VMAT (Volumetric modulated arc therapy), at our institution between 2016 and 2020. The patient characteristics were shown in Table 1. A total of 110 patients were enrolled based on the criteria (Table 1). The dosimetric factors include prescription dose, fraction dose, PTV, lung V₅, V₁₀, V₂₀, V₃₀, V₄₀, lung Dean dose. National Cancer Institute’s Common Terminology Criteria for Adverse Events (vision 4) report that RP were graded from 1 (asymptomatic) to 5 (death) [18]. Tumor stage was performed based on the clinical staging of the 8th edition of the American Joint Committee on Cancer (AJCC) TNM staging system, including locally advance disease (stage III), inoperable stage I and II disease. Patients were selected if they received a prescription dose of 40–60 Gy in 1.5–3.5 Gy-daily fractions using 6 MV photons. The endpoint was grade ≥ 2. The RP were determined by a radiologist with 10 years of clinical experience by using follow-up CT images and clinical records. In addition, the general region of interests (ROIs) were segmented, such as Lung, heart, spine et al. All contours were double-checked by physicist to improve the performance of the deep-learning model.

Table 1.

Patient characteristics.

Clinical characteristics	Data
RP(grade ≥ 2)	19
Non-RP	91
Sex
Male	60
Female	50
T stage
0	2
1	11
2	29
3	23
4	45
N stage
0	14
1	7
2	48
3	41
Overall stage
I	4
II	7
III	68
IV	31
Age
Median	62
Average	63
Range	39–92
Smoking status
Yes	45
No	65

Patients underwent a free-breathing CT scan for radiotherapy planning using Brilliance-16 system (Philips Medical Systems, Inc., Cleveland, OH). The imaging parameters as follows: tube voltage (120kVp), tube current (182–320 mA), pixel size (1.06 mm × 1.06 mm), reconstruction increment (3.00 mm), slice thickness (3.00 mm), and matrices of images (X:512, Y:512, Z:79–150).

Radiomics features extraction and analysis

The workflow was shown in Fig. 1. This study can be divided into two parts. For Phase 1, patients radiomics features were extracted from CT images. Totally 1419 features were extracted from Dicom (digital imaging and communications in medicine) CT files. The image types include original, wavelet, LoG (Laplacian of Gaussian filter), and available feature classes were first order statistic, shape-based(3D), GLCM (gray level cooccurrence matrix), GLRLM (gray level run length matrix), GLSZM (gray level size zone matrix), NGTDM (Neighbouring gray tone difference matrix), GLDM (gray level dependence matrix). The program was development with Python (version 3.6). Pyradiomics package were used for feature extraction. This is an open-source package for the extraction of radiomics from medical imaging. With this package we could to establish a reference standard for radiomic analysis, and provide a tested and maintained open-source platform for easy and reproducible radiomic feature extraction.

Fig. 1.

The workflow of data process and data analysis performed in this study.

Radiomics features were selected based on significance, correlation, and cluster result. Manhattan distance was widely used for the clustering of multi-variable datasets in computational biology [19]. it was used for selecting features that correlation with RP.

Dataset and CT image preprocessing

Thereafter, radiomics features extraction, a label (RP or non-RP) was assigned to each feature and data were randomly divided into training set and test set, with 90 patients in training set included 85 non-RP and 14 RP. The test set included 6 non-RP and 5 RP, total 11 patients.

For each endpoint combined radiomics features, dosimetric and clinical factors and 3D CT-dose images (Fig. 2). To investigate the incremental prediction value of high-dimensional features, five prediction models were designed using the same method: (1) based on clinical and dosimetric factors; (2) based on 3D dose distribution; (3) based on radiomics features; (4) based on 3D CT images; (5) hybrid (1)-(4) features.

Fig. 2.

Overall preprocess workflow of feature extraction: CT image and dose distribution data for PSN sub-network 3D input channel; dosimetric and clinical factors and radiomics features for PSN sub-network 1D input channel.

Architecture of deep-learning network

For phase 2 in Fig. 1, Fig. 3D CT images and dose distribution were exported as dicom file format for image process. Fig. 3 was shown the detailed prediction network architecture. The framework backbone was residual block, for processing different dimension of data, we developed a pseudo-siamese network. This network can be set as single mode when input only 1D or 3D.

Fig. 3.

Pseudo-siamese network, 3D Conv, 3D convolutional layer;1D Conv, 1D convolutional layer; Batch norm, batch normalization; Maxpool, max-pooling layer; GlobalAvgpool, global average pooling layer; Linear, linear layer.

The input images were resized to 96 × 96 × Z pixel. Random flipping, rotation, contrast adjustment, color jitter, and affine transform were used for data augmentation. All images pixel value were normalized within range of 0 to 1. The radiomics feature value were normalized within range of −10 to 10. The batch size for training set was 16 and test set was 1. For radiomics feature input, a single 1D Resnet feature extractor were designed for filter useful signature. 1D features and 3D features are finally flattened to assemble by fully connected layers. Modified network layer can improved model performance and avoid overfitting [20] (i.e., reduce the number of residual blocks and network layer appropriately). The loss function was cross-entropy, and Adam was used as the optimizer. Epochs were set 1000 with the 1e−4 learning rate. Weight decay was 0.1. An Intel 8700 k CPU and Nvidia TITAN GPU were used for model training.

The impact of different features for predicting radiation pneumonitis

To investigate different kind of data for predicting RP, 5 classifiers were built by using (a)Dosimetric and clinical factors, (b)3D dose distribution, (c)3D CT images, (d)Radiomics features, (e)the combination of dosimetric and clinical factors, radiomics features, 3D dose distribution, and 3D CT images (hybrid model). Each dataset was trained separately until the model converged. 5-fold cross-validation were used for testing the model’s ability to predict data that was not used in estimating it, in order to reduce problems like overfitting or selection bias [21].

Performance evaluation

The receiver operating characteristic (ROC) curve was a commonly diagnostic tool for evaluating the performance of binary classifiers. This study exist class imbalance may introduce bias to the non-RP group. Cross validation were introduced for verifying the generalization ability of the model.

Results

Clustering analysis

The clustering results were shown as feature heatmap in Fig. 4, with 110 patients on the x-axis and the expression of 106 radiomics features on the y-axis. The heatmap that patients within the same group cluster displayed similar value patterns. 106 radiomics features were selected from 1419 radiomics features.

Fig. 4.

The radiomics features are shown as the heatmap, with 110 patients on the x-axis and 107 radiomics features on the y-axis. The top legend shows the patient RP situation, 1 and 0 represent RP and non-RP, respectively.

Performance and index estimation

Fig. 5 shows confusion matrix of the different dataset, each element in the confusion matrix means the number of samples with the true class y that was identified as being in class x. To explore the value of different characteristic of data, the single type dataset was used for training (a-d). Dosimetric and clinical factor model were shown in Fig. 5(a), totally 8 samples were truly predicted. 3D dose distribution and CT image model were shown in Fig. 5(b) and (d), respectively. 3D dose distribution model was shown poor sensitivity than (a). RP and non-RP pattern cannot be identified by only using CT images dataset. Radiomics feature model were shown in Fig. 5(c), the number of true predictions the same to the dosimetric and clinical factors. The sensitivity, however, were poor than dosimetric and clinical factor model. The hybrid model result was shown outperformance than other models in Fig. 5(e).

Fig. 5.

The confusion matrix for prediction of non-RP patient and patients with RP.

To summarize, the index that evaluate the performance of model were shown in Table 2 (for example, accuracy, precision, recall, and f1-score). The accuracy of dosimetric and clinical factor model and radiomics feature model both 0.727, while dosimetric and clinical factor model was shown good performance in recall (0.800) and F1-score (0.727) than radiomics feature model. Precision of dosimetirc and clinical factor and radiomics feature model were 0.667 and 0.750, respectively. The accuracy of dose distribution model and CT image model was 0.636 and 0.545, respectively. Recall, precision, and F1-score were 0.600 in dose distribution model, and 0.600, 0.500 and 0.545 in CT image model.

Table 2.

Summarizes the accuracy, precision, recall, and f1-score in test dataset.

Input	Accuracy	Recall	Precision	F1-score
Dosimetric and clinical factors	0.727	0.800	0.667	0.727
Dose distribution	0.636	0.600	0.600	0.600
CT images	0.545	0.600	0.500	0.545
Radiomics features	0.727	0.600	0.750	0.667
Hybrid model	0.818	0.800	0.800	0.800

Model validation

Fig. 6 shows the results of 5-fold cross-validation for hybrid model. The mean AUC was 0.85, For 1–5-fold cross-validation, AUC was 0.77–0.90, the mean AUC was 0.85 ± 0.05. The sensitivity and specificity for different model was listed in Table 3. Hybrid model and dosimetric and clinical factor model have the same sensitivity 0.800 (95 % CI [0.299, 0.989]), while the sensitivity for dose distribution, CT image, and radiomics feature model were 0.600 [0.170, 0.927]. For specificity, CT image model was 0.500 [0.139, 0.860], dosimetric and clinical factor model and dose distribution model are 0.667 [0.241, 0.940]. Radiomics feature model and hybrid model are 0.833 [0.364, 0.991]. The predictive effect of the hybrid model showed a stable performance in both the sensitivity and specificity.

Fig. 6.

ROC for hybrid model with 5-fold cross-validation. Note: the mean value and standard deviation are shown.

Table 3.

Summarizes the sensitivity and specificity in test dataset.

Input	Sensitivity [95 % CI]	Specificity [95 % CI]
Dosimetric and clinical factors	0.800 [0.299, 0.989]	0.667 [0.241, 0.940]
Dose distribution	0.600 [0.170, 0.927]	0.667 [0.241, 0.940]
CT images	0.600 [0.170, 0.927]	0.500 [0.139, 0.860]
Radiomics features	0.600 [0.170, 0.927]	0.833 [0.364, 0.991]
Hybrid model	0.800 [0.299, 0.989]	0.833 [0.364, 0.991]

Discussion

In this study, we firstly proposed a novel method extracted dosimetric and clinical factors, radiomics features combined with 3D CT images and dose for predicting RP occurrence before delivery. The results were shown the feasibility of the model for predicting RP incidence.

Previous studies have found clinical markers associated with RP. Zhang et al reported that the multivariate analysis of risk factors predicting grade ≥ 2, higher CD8⁺ T cell count was shown a risk factor associated with RP (p < 0.05) [22]. There are many considerable and novel characteristics were not enrolled in this study, such as CD4/8⁺ T cell count, CD4/8⁺ T cell ratio, B cell, etc. The useful cell statistics information associate with RP able be collected only after radiotherapy [23]. Therefore, we have been excluded these parameters. The advantage of our model is that RP would be predicted before the first delivery. Dosimetric factors were proved considerable for predicting RP in previous study, such as lung V₂₀, mean lung dose, PTV [24]. Since 3D dose distribution as input, this study hasn’t compared these dosimetric parameter which have high correlation with RP. These dosimetric characteristic, whereas, still as deep learning model input for improving performance. In our implementation of network, we compared the efficacies of multi-input model, the experiment shows that dosimetric and clinical characteristic model have the same sensitivity than hybrid model while specificity is lower.

Radiomics features were traditionally combined with machine learning algorithm for the RP prediction. Radiomics features were extracted from different scale ROI in CT images. Although radiomics feature extraction shows a good performance in predicting RP, the high-dimensional information from radiotherapy plan useful for prediction. On the other hand, there is no consensus yet whether the design of radiotherapy plan, beam angle, target, etc, may affect the arise of RP. Deep learning may detect heterogeneity in medical images from high dimensional data point, resulting in greater predictive performance [25], [26]. In terms of this, we consider the complete 3D CT images/dose distribution contain potential features which can be extracted by using convolution neural network. To mitigate the limitations of the traditional radiomics predict RP techniques, our approach introduced the 3D CT images and dose distribution, which including the information of the anatomic status, beam angle, and radiotherapy plan could as supplement for model prediction.

CT has become the main imaging protocol for lung diagnosis because there are many alveoli in the lung, which were filled with gas. In CT image preprocessing, the dose distribution and CT images should be registered. To reduce redundant information, the table was removed from CT image (Fig. 2). Lung is crucial structure for prediction RP, therefore, the radiomics features were extracted from lung. Traditional radiomics feature can only reflect texture information. In this study, we also use single feature, like 3D CT image CNN feature or radiomics features, to build model. The results were shown that radiomics model has better performance than 3D CT image model, therefore, we consider radiomics feature still valuable be added in. Using 3D CT images as CNN model input combine with 3D dose map can make them spatial coordinate matching.

Compared with the previous study, our research yielded a better predict performance by concentrating on the 3D images, dose feature combined deep learning algorithm, which can complement 3D feature with more information [9], [17]. Experiments were shown that combine radiomics features and deep learning features can improve performance for predicting RP. Liang et al. [14] designed a feature filter and developed a dual-omics (original dose and ventilation images) prediction model for RP, the AUC was 0.874. The ventilation images, however, should be derived from 4-dimensional CT, this approach certainly raises the cost of diagnosis. The advantage of our approach is that we can used more information which can be collected easily from clinical routine without affiliated step.

For the training model, we selected PSN for this task, the residual block as the backbone to prevent the gradient disappearance and minimize the degradation problem after many iterations [27], [28], [29]. Dual channel solved the data dimensional match problem.

For the feature standardization, our study was similar to the IBSI (Image Biomarker Standardization Initiative) consortium to processed radiomics features [30]. The image date was processed by conversion to standardized uptake values, segmentation and create ROI mask. Finally, the radiomics feature were computed via ROI masks and images. PSN network can be reproduced easily refer our architecture.

This study has several limitations. First, only single institution with a small cohort were enrolled in this retrospective study. We did not validate this model with an external dataset, which could have been valuable in demonstrating the reliable performance of the model. We did many times of 10 folds cross validation to prevent the model overfitting (appendix A). We also compared a pre-trained model for feature extraction. However, the results were no better than our proposed model (appendix B). Therefore, we think our method may suitable for specific medical image analysis area. In this study, we proposed model is not a large CNN network (like Resnet 50, 100, 150 or Dense net-101). It just used the residual blocks. The model parameter was lightweight. The number of layers isn’t too deep. Second, we combined dosimetric and clinical factors, radiomics, CT images and dose which have not involved parameter in feature. For example, in radiomics extraction, the resample pixel spacing, bin width, and sigma should be defined (The default in this study we set 0.5 mm per pixel, bin width was 25, sigma was (1,3)). The parameter setting have many method (e.g. SquartRoot, Sturges, Freedman-Diaconis) that have not be consideration in this study. Third, the positive sample in this study was small (only 19 patients), therefore, this study merely explored the feasibility of the PSN model for predicting RP with a few dataset.

A large-scale, multicenter validation and study are required for full evaluation of model in the future. We will enroll more patients and join multi-scale parameter in this model for assisting physician and physicist design radiotherapy plan.

Conclusion

In this study, we firstly propose a novel method that combining dosimetric and clinical factors, radiomics features, 3D CT images, and dose maps for RP prediction. The result shows that the feasibility of the proposed method, which introducing high dimensional features and a novel deep-learning frame. Nevertheless, future multi-institution larger-scale studies are required for validation.

Funding

This work was supported by the Integrated innovation and application of key technologies for precision prevention and treatment of primary lung cancer [Grant No 2019ZX002]; Research and application of FACE radiotherapy management system in patients with malignant tumor [grant number 2022DBXM005].

Conflict of interest

The authors declare that they have no conflict of interest and comply with the ethical standards.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: