Deep Learning-Based Computed Tomography Image Standardization to Improve Generalizability of Deep Learning-Based Hepatic Segmentation

Abstract

Objective

We aimed to investigate whether image standardization using deep learning-based computed tomography (CT) image conversion would improve the performance of deep learning-based automated hepatic segmentation across various reconstruction methods.

Materials and Methods

We collected contrast-enhanced dual-energy CT of the abdomen that was obtained using various reconstruction methods, including filtered back projection, iterative reconstruction, optimum contrast, and monoenergetic images with 40, 60, and 80 keV. A deep learning based image conversion algorithm was developed to standardize the CT images using 142 CT examinations (128 for training and 14 for tuning). A separate set of 43 CT examinations from 42 patients (mean age, 10.1 years) was used as the test data. A commercial software program (MEDIP PRO v2.0.0.0, MEDICALIP Co. Ltd.) based on 2D U-NET was used to create liver segmentation masks with liver volume. The original 80 keV images were used as the ground truth. We used the paired t-test to compare the segmentation performance in the Dice similarity coefficient (DSC) and difference ratio of the liver volume relative to the ground truth volume before and after image standardization. The concordance correlation coefficient (CCC) was used to assess the agreement between the segmented liver volume and ground-truth volume.

Results

The original CT images showed variable and poor segmentation performances. The standardized images achieved significantly higher DSCs for liver segmentation than the original images (DSC [original, 5.40%–91.27%] vs. [standardized, 93.16%–96.74%], all P < 0.001). The difference ratio of liver volume also decreased significantly after image conversion (original, 9.84%–91.37% vs. standardized, 1.99%–4.41%). In all protocols, CCCs improved after image conversion (original, -0.006–0.964 vs. standardized, 0.990–0.998).

Conclusion

Deep learning-based CT image standardization can improve the performance of automated hepatic segmentation using CT images reconstructed using various methods. Deep learning-based CT image conversion may have the potential to improve the generalizability of the segmentation network.

INTRODUCTION

The development and application of artificial intelligence (AI) in the field of radiology have made significant progress in automated image analysis [1]. In radiology, automated tools using deep learning have been developed for image classification, lesion detection, and image segmentation [2]. Automatic image analysis with a deep learning algorithm can improve workflows in radiology by eliminating time-consuming workloads. In addition, the reported accuracies of many deep learning algorithms are beginning to match or even exceed those of radiologists [3,4,5].

One substantial barrier to the development of deep learning algorithms is securing large-scale annotated data [6,7]. Large and heterogeneous datasets with high-quality images from multiple institutions and different geographic areas are essential for training and developing high-quality deep learning algorithms with general applications [8]. However, curating large datasets is challenging owing to the limited availability of radiologists and tedious annotation processes [9]. Therefore, in most cases, the development of deep learning algorithms has been achieved using a limited dataset that represents the characteristics of certain distributions in the research population, and the developed algorithms may suffer from overfitting, which results in poor generalizability [9,10,11].

According to a recent review of the external validation of deep learning algorithms for radiologic diagnosis, the vast majority of algorithms demonstrated diminished performance on the external dataset, with some reporting a substantial performance decrease [12]. A few methods have been suggested to improve the generalizability of deep learning algorithms, including transfer learning [10]; however, transfer learning has the disadvantage of additional data augmentation, which is required whenever the characteristics of the input images are changed.

A recent study reported a deep learning algorithm capable of converting various computed tomography (CT) images derived from diverse CT protocols into target CT images [13]. This study used a generative adversarial network (GAN) to standardize CT images to improve the reproducibility of radiomics features. The developed deep learning algorithm shows potential for the standardization of medical image data. Another study reported that data normalization and augmentation improved the generalizability of neural network-based cardiac magnetic resonance image segmentation methods [14]. Therefore, we aimed to investigate whether image standardization using deep learning-based CT image conversion improves the performance of deep learning-based automated hepatic segmentation across various reconstruction methods.

MATERIALS AND METHODS

The institutional review boards of the Seoul National University Hospital (IRB No. 2202-096-1301) approved this retrospective study and waived the requirement for informed patient consent.

Study Population

For the training and tuning datasets, we collected data from 117 patients who underwent 142 contrast-enhanced abdominal CT examinations with dual-energy (DE) scans at a single tertiary hospital between March 2021 and July 2021 (Table 1). The collected data were divided into training and tuning datasets at a ratio of 9:1 (i.e., 128 and 14 examinations, respectively). For the test dataset, we separately collected 43 contrast-enhanced abdominal CT examinations in 42 patients without focal hepatic lesions obtained using DE scans at the same tertiary center between December 2021 and January 2022 (Table 1).

Table 1. Characteristics of the Datasets.

	Training & Tunning Dataset	Test Dataset
Number of patients (male:female)	117 (57:60)	42 (19:23)
Age* (range)	8.7 ± 5.5 years (2 months–19 years)	10.1 ± 8.7 years (7 months–49 years)
BMI* (range)	18.2 ± 4.6 (10.5–33.4)	18.5 ± 5.0 (12.5–35.2)
Reason for examination	Abdominal pain (n = 15, 12.8%) Tumor follow-up (n = 69, 59.0%), Others (n = 33, 28.2%)	Abdominal pain (n = 3, 7.1%), Tumor follow-up (n = 33, 78.6%), Others (n = 6, 14.3%)
Number of CT scans	142	43
CT machines	SOMATOM Force (Siemens)	SOMATOM Force (Siemens)
Tube voltage	70 kVp and 150 kVp	70 kVp and 150 kVp
Reference tube current	370 mAs for the 70 kVp tube 93 mAs for the 150 kVp tube	370 mAs for the 70 kVp tube 93 mAs for the 150 kVp tube
CT slice thickness	3 mm	3 mm
Scan timing	Portal phase†	Portal phase†

^*Values represent mean±standard deviation, ^†65 s after the initiation of contrast agent injection. BMI = body mass index, CT = computed tomography

The images were acquired using a SOMATOM Force CT scanner (Siemens) in DE mode. The scanned data were reconstructed using filtered back projection (FBP), iterative reconstruction (IR) with a strength of 3 (SAFIRE), and virtual mono-energetic images with 40 keV (M40), 60 keV (M60), 80 keV (M80), and optimum contrast (OPT). The DECT parameters are listed in Table 1.

Architecture of the Deep Neural Network for Image Standardization

Lee et al. [13] proposed a GAN to improve the reproducibility of CT-based radiomics features. We employed a generator network (G) architecture, including a hierarchical feature synthesis module from a prior study [13]. We modified the first layer of the generator network to have a pixel-unshuffle layer instead of a spatial average pooling layer, to reduce the dimensions of the input image to a quarter. The pixel unshuffle is the reverse operation of the pixel-shuffle [15], which helps computational efficiency by reducing the input dimension without pixel loss. We employed a U-Net discriminator (D) for spectral normalization [16]. The U-Net discriminator network was proposed for the image super-resolution task and can provide per-pixel feedback to the generator. The proposed GAN architecture is illustrated in Figure 1. We employed a Real-ESRGAN training strategy [16], which is divided into two parts. First, we train a generator network without a discriminator network. Therefore, the generator network was trained with an L1 loss between the generated image and the ground-truth image and exhibited a peak signal-to-noise ratio (PSNR)-oriented performance. The equation for the L1 loss is

Fig. 1. Architecture of the generator and discriminator. In the generator, the pixel unshuffle is the reverse operation of the pixel shuffle to reduce the input dimension without pixel loss. The discriminator is a U-Net architecture with spectral normalization. Where n is the number of output feature maps of the convolution, k is the convolutional kernel size, and s is the stride of the convolution along the height and width. Conv = convolution, Norm = normalization.

$${\text{L}}_{\text{l}1}=\frac{1}{\text{hw}}{‖\text{G}\left(\text{x}\right)-\text{x}‖}_F^1$$ Eq. (1)

where ‖·‖_F denotes the Frobenius norm, and h and w denote the height and width of the 2D image, respectively. x is the given source image and G(x) is the generated image. Second, the PSNR-oriented model was used for initialization and then trained using a discriminator network. A combination of L1 loss, perceptual loss [17], and GAN loss [18,19] was used to train the generator network. The perceptual loss equation is as follows:

$${\mathrm{L}}_{\text{perceptual}}=\frac{1}{\text{hwd}}{‖\Phi \left(\mathrm{G}\left(\mathrm{x}\right)\right)-\Phi \left(\mathrm{x}\right)‖}_F^2$$ Eq. (2)

where Φ is the feature extractor, and h, w, and d represent the height, width, and depth of the feature space, respectively. The feature extractor is the 16th convolution layer of the well-known pre-trained VGG-19 network [20]. The GAN loss is defined as follows:

$${\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}\mathrm{}}={\mathrm{I}\mathrm{E}}_{{\text{x}~p}_{data}\left(\text{x}\right)}\left[-\text{log D}\left(\mathrm{G}\left(\mathrm{x}\right)\right)\right]$$ Eq. (3)

where IE(·) is the expectation operator and p_data denotes the probability distributions of the source image. Combining equations Eqs. (1-3), the total loss function is obtained as follows:

$${\mathrm{L}}_{\text{total}}={\lambda }_1\times {\mathrm{L}}_{\mathrm{l}1}+{\lambda }_2\times {\mathrm{L}}_{\text{perceptual}}+{\lambda }_3\times {\mathrm{L}}_{\text{GAN}}$$ Eq. (4)

where λ₁, λ₂, and λ₃ are the parameters used to adjust the balance of L_l1, L_perceptual, and L_GAN in L_total· In this study, we set λ₁ = 10, λ₂ = 1, and λ₃ = 1 in equation Eq. (4).

Training of the Deep Neural Network for Image Standardization

In this study, the network input comprised six different reconstructed images: FBP, IR, M40, M60, M80, and OPT, and the target was M80. We randomly sampled a 256 x 256 local patch from the same location in all the reconstructed images. Min-max normalization was applied to rescale the CT values to “0, 1”. A local patch was used as an input to the generator network, and the generated image had the same size. The discriminator network provides pixel-wise feedback over the local patch. All parameters of the generator and discriminator networks were optimized using an adaptive moment estimation optimizer (hyper-parameters α = 1 × [10]^(-4), β_1 = 0.9, and β_2 = 0.99) [21]. The networks were trained for 200 epochs, and the learning rate decayed by 95% of the decay rate after 100 epochs. We implemented this deep learning network using PyTorch 1.10. All experiments were performed on a personal computer (Intel i7 9770 [Intel] with 32 GB of memory) and accelerated using an NVIDIA RTX 2080 Ti GPU (NVIDIA) with 11 GB of memory.

Liver Segmentation

We used a previously trained 2D U-NET in a commercially available segmentation software program (MEDIP PRO v2.0.0.0, MEDICALIP Co. Ltd.) to create liver segmentation masks and calculated the liver volume for the test dataset. A pre-test was performed with five CT examinations from the training dataset to assess the performance of segmentation in various protocols. According to the pre-test results, we selected the protocol with the highest segmentation performance among the different images as the ground truth for image conversion. Two radiologists assessed the liver segmentation masks for all protocols, and the original M80 protocol, which showed nearly perfect segmentation, was selected as the ground truth image (Supplementary Fig. 1).

Evaluation of Liver Segmentation Performance

To evaluate the performance of the automated segmentation, the masks that were generated on each protocol image were compared with the reference segmentation mask of the ground truth image (i.e., originalM80 images). We calculated the Dice similarity coefficients (DSCs) by comparing two segmentation masks. Segmentation performance was obtained for the original CT images and standardized CT images that were synthesized using the deep learning algorithm for image standardization. Moreover, we calculated the absolute difference between the segmented volume and ground truth liver volume (i.e., the original M80 images) as well as the ratio of the liver volume difference to the ground truth volume.

Statistical Analyses

We used a paired t-test to compare the segmentation performance in DSC before and after the image conversion using the deep learning-based standardization algorithm. We also compared the absolute volume difference from the ground truth volume and the ratio of the liver volume difference to the ground truth volume between the original and standardized images using a paired t-test. Agreement between the segmented liver volume and ground truth volume was analyzed using concordance correlation coefficients (CCCs) [22] and Bland–Altman analysis.

RESULTS

Patient Demographics

For training and tuning, we used 142 CT examinations (128 and 14, respectively) of 117 patients. For testing, 43 CT scans from 42 patients were included in the test dataset. The mean age of the test population was 10.1 years (range, 7–49 years). Detailed demographics of the test datasets are summarized in Table 1.

Comparison of Segmentation Performance

The DSCs of the automated liver segmentation in the original and standardized images are summarized in Table 2. According to the protocols, the original images showed variable and poor segmentation performances for the liver. The standardized images that were obtained using deep neural networks achieved significantly higher DSCs for liver segmentation than the original images (all P < 0.001). M40 had the lowest DSC in the original images, and the largest performance increase with the image standardization. Representative images are shown in Figures 2 and 3.

Table 2. Comparison of Dice Similarity Coefficients between the Original and Standardized Images in Various Protocols.

	DSC (%)
	Original	Standardized	P
FBP	90.79 ± 5.57	96.74 ± 2.43	< 0.001
IR	91.27 ± 5.11	96.57 ± 2.45	< 0.001
M40	5.40 ± 18.06	93.16 ± 4.19	< 0.001
M60	87.55 ± 7.91	95.88 ± 2.61	< 0.001
OPT	83.63 ± 12.50	96.27 ± 2.33	< 0.001

Values represent the mean ± standard deviation. DSC = Dice similarity coefficient, FBP = filtered back projection, IR = iterative reconstruction, M40 = mono-energy 40 keV, M60 = mono-energy 60 keV, OPT = optimal contrast

Fig. 2. A representative case of automated liver segmentation of a 14-year-old girl with various protocols. The images show the three-dimensional volume rendering of the liver segmentation. A: Liver segmentation at ground truth. B: Liver segmentation at a mono-energy of 40 keV (M40). C: Liver segmentation using standardized M40. D: Liver segmentation at a mono-energy of 60 keV (M60). E: Liver segmentation using standardized M60. All standardized images exhibited better segmentation performance than the original images.

Fig. 3. A representative case of automated liver segmentation of a 14-year-old girl with various protocols. The images show the three-dimensional volume rendering of the liver segmentation. A: Liver segmentation using filtered back projection (FBP). B: Liver segmentation using standardized FBP. C: Liver segmentation using hybrid iterative reconstruction (IR). D: Liver segmentation using standardized IR. E: Liver segmentation with optimal contrast (OPT). F: Liver segmentation using standardized OPT. All standardized images exhibited better segmentation performance than the original images.

Comparison of Liver Volume

Compared with the ground truth liver volume, the absolute difference and difference ratio of the liver volume in the original and standardized images are presented in Table 3. The absolute difference in liver volume decreased after image conversion (all P < 0.001). The difference ratio of the liver volume also decreased significantly after the image conversion (all P < 0.001).

Table 3. Comparison of Liver Volume between Original and Standardized Images Using Various Protocols.

	Absolute Liver Volume Difference (mL)			Difference Ratio of Liver Volume (%)
	Original vs. GT	Standardized vs. GT	P	Original vs. GT	Standardized vs. GT	P
FBP	949.7 ± 784.1	154.0 ± 223.1	< 0.001	11.5 ± 9.0	2.0 ± 2.9	< 0.001
IR	808.2 ± 732.7	159.3 ± 208.0	< 0.001	9.8 ± 8.4	2.0 ± 2.8	< 0.001
M40	8684.8 ± 4371.7	356.8 ± 414.1	< 0.001	91.0 ± 19.3	4.4 ± 5.1	< 0.001
M60	8587.8 ± 3652.5	324.7 ± 222.4	< 0.001	91.4 ± 1.4	4.3 ± 3.8	< 0.001
OPT	1917.9 ± 1789.5	192.8 ± 299.8	< 0.001	21.6 ± 18.4	2.4 ± 3.3	< 0.001

Values represent the mean ± standard deviation. FBP = filtered back projection, IR = iterative reconstruction, M40 = monoenergy 40 keV, M60 = mono-energy 60 keV, OPT = optimal contrast, GT = ground truth

Agreement of Segmented Liver Volume with Ground Truth Volume

The agreement of the segmented liver volume results with the ground truth images, in terms of CCC, for the original images and standardized CT images in various protocols is summarized in Table 4. In all protocols, CCCs improved after the image conversion. In the original images, M40, M60, and OPT showed poor agreement. After image conversion, all protocols showed almost perfect agreement.

Table 4. Concordance Correlation Coefficients of Liver Volume in the Original and Standardized Images with the Ground-Truth Images in Various Protocols.

	Original	Standardized
FBP	0.955 (0.923–0.973)	0.998 (0.996–0.999)
IR	0.964 (0.939–0.979)	0.998 (0.996–0.999)
M40	-0.006 (-0.046–0.035)	0.990 (0.983–0.995)
M60	0.031 (0.018–0.044)	0.995 (0.991–0.997)
OPT	0.809 (0.696–0.883)	0.996 (0.993–0.998)

Values in parentheses are 95% confidence interval. FBP = filtered back projection, IR = iterative reconstruction, M40 = monoenergy 40 keV, M60 = mono-energy 60 keV, OPT = optimal contrast

Figures 4 and 5 show the Bland–Altman plot of the liver volume difference ratio against the ground truth volume. For FBP, IR, and OPT, prior to applying image conversion, the plot exhibited a negative bias (mean difference: FBP -11.5%, IR -9.8%, OPT -21.5%) with wide 95% limits of agreement (LOAs) (FBP -29.2%–6.2%, IR -26.3%–6.7%, OPT -58.0%–15.0%), and there was no systematic trend in bias with liver volume at ground truth. After image conversion, the plot showed a small negative bias (mean difference: FBP -1.5%, IR -1.0%, OPT -2.0%) with narrow LOAs (FBP -7.7% – 4.8%, IR -7.4%–5.4%, OPT -8.9%–5.0%). For M40, the original image exhibited a large negative bias (mean difference, -91.0%) with a wide LOA (-128.8% – -53.2%). In contrast, the standardized image showed a small positive bias (0.8%) with a narrow LOA (-12.3%–13.9%). The original M60 images showed a narrow LOA (-94.0% – -88.7%) but had the largest negative bias (mean difference, -91.4%). After image conversion, the plot exhibited a small positive bias (mean difference, 3.7%) with a narrow LOA (-4.9%–12.3%).

Fig. 4. Bland–Altman curve of liver volume between various protocols and ground truth. A: Bland–Altman plot between mono-energy 40 keV (M40) and ground truth (GT) (mean difference = -91.0%, 95% limits of agreement [LOA] = -128.8% – -53.2%). B: Bland–Altman plot between standardized M40 and GT (mean difference = 0.8%, 95% LOA = -12.83%–13.9%). C: Bland–Altman plot between mono-energy 60 keV (M60) and GT (mean difference = -91.4%, 95% LOA = -94.0% – -88.7%). D: Bland–Altman plot between standardized M60 and GT (mean difference = 3.7%, 95% LOA = -4.9%–12.3%). s = standardized, SD = standard deviation.

Fig. 5. Bland–Altman curve of liver volume between various protocols and ground truth. A: Bland–Altman plot between filtered back projection (FBP) and ground truth (GT) (mean difference = -11.5%, 95% limits of agreement [LOA] = -29.2%–6.2%). B: Bland–Altman plot between standardized FBP and GT (mean difference = -1.5%, 95% LOA = -7.7%–4.8%). C: Bland–Altman plot between hybrid iterative reconstruction (IR) and GT (mean difference = -9.8%, 95% LOA = -26.3%–6.7%). D: Bland–Altman plot between standardized IR and GT (mean difference = -1.0%, 95% LOA = -7.4%–5.4%). E: Bland–Altman plot between optimal contrast (OPT) and GT (mean difference = -21.5%, 95% LOA = -58.0%–15.0%). F: Bland–Altman plot between standardized OPT and GT (mean difference = -2.0%, 95% LOA = -8.9% – -5.0%). s- = standardized, SD = standard deviation.

DISCUSSION

This study demonstrated that deep learning-based image analysis tools without training data augmentation showed poor performance when medical images that were different from the training dataset were used as input data. Liver segmentation performance showed variable and poor performance according to the image protocols, despite the data being obtained from the same patient simultaneously. After image conversion using the developed deep neural network algorithms, organ segmentation performance was significantly improved.

AI is a rapidly growing field in medical imaging. A considerable number of articles submitted and published in the field of radiology are related to AI, and many AI-based medical image analysis software programs are being developed [23]. As the medical imaging volume increased, radiologist loading increased. The application of AI to medical imaging helps various tasks of radiologists [1]. Image analysis using AI extends beyond lesion detection and enables lesion classification, treatment response prediction, image conversion, synthetic image generation, and various quantitative analyses. AI helps improve accuracy and efficiency in the detection of lesions and reduces measurement and perceptual errors [24,25,26].

AI is a promising technology for medical imaging; however, there are several obstacles to its application in daily clinical practice. The biggest limitation in AI models that can be applied to clinical practice is securing generalizability [11,27]. Developed and tested deep learning-based algorithms, showing excellent performance in external tests, may demonstrate degraded performance in real clinical practice [10,28,29,30]. In this study, liver segmentation performance using automated deep learning-based algorithms showed variable results across different reconstruction methods. The lower the similarity with the ground truth image, the lower the segmentation performance. In the training of the deep learning-based algorithm, image characteristics, including image noise, contrast, artifacts, and texture, were trained. In CT images, various CT parameters (e.g., tube voltage or tube current), patient body size, reconstruction method, machine, and vendor type affect the image characteristics. Deploying deep learning-based algorithms to individual institutions in real clinical practice encounters new datasets that are not identical to the training dataset. This heterogeneity of medical images from individual institutions degrades the performance of deep learning-based algorithms. A few studies have attempted to improve the generalizability of deep learning-based algorithms with data augmentation [10,11,31,32]. Rauschecker et al. [10] used a limited additional local training dataset to overcome generalization issues. The major advantage of transfer learning is that it can improve the performance of the developed algorithm using only a modest amount of data that is acquired by the institution that uses the developed algorithm. Transfer learning may be a solution for applying the algorithm to external institutions. However, this method has several limitations. Additional data augmentation is required for each institution to use this algorithm. Additional transfer learning is required whenever the environment for the acquisition of medical images is changed (e.g., changing the imaging protocols or CT machine). Moreover, there are no specific guidelines for the number of local datasets that are required to restore the internally tested performance of the algorithm.

Eche et al. [11] attempted to secure generalizability using computational stress testing. Generating various datasets by artificially modifying images can improve the generalizability of stress tests. In radiology, various image datasets can be generated by adjusting image noise, contrast, section thickness, or artifacts. Training and validation using a large heterogeneous dataset can improve generalizability. In the case of a model undergoing stress testing to obtain generalizability, the average performance of several test datasets may be high, but the performance of a specific dataset that is evenly distributed with the training set can be rather low [11].

However, image conversion and standardization have several advantages over the other methods. First, it can be a useful means of securing the generalizability of deep learning models regardless of the application environment. If the converted image data are close to the training set, they can show ideal performance regardless of the CT machine, vendor, and reconstruction method that is used in individual institutions. Second, there is no need to collect the dataset for transfer learning or have a stress test dataset, which saves time and effort for the deployment of the algorithm.

Our study had some limitations. First, the data used as the ground truth for image conversion were not used to train the segmentation algorithm, which may be the reason for the slightly low segmentation performance. However, when we evaluated each segmentation mask from the ground truth, no substantial errors were observed (e.g., the segmentation of other organs). Second, we used a single CT machine for the training and validation. Other CT machines and vendors should be evaluated to expand the use of the developed model in clinical practice.

In conclusion, we suggest that deep learning-based CT image standardization can improve the performance of automated segmentation of the liver using CT images that are reconstructed with various methods. Deep learning-based CT image conversion may have the potential to improve the generalizability of the segmentation network.

Availability of Data and Material

The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.

Supplement

The Supplement is available with this article at https://doi.org/10.3348/kjr.2022.0588.