Image synthesis of effective atomic number images using a deep convolutional neural network-based generative adversarial network

Abstract

Background

The effective atomic numbers obtained from dual-energy computed tomography (DECT) can aid in characterization of materials. In this study, an effective atomic number image reconstructed from a DECT image was synthesized using an equivalent single-energy CT image with a deep convolutional neural network (CNN)-based generative adversarial network (GAN).

Materials and methods

The image synthesis framework to obtain the effective atomic number images from a single-energy CT image at 120 kVp using a CNN-based GAN was developed. The evaluation metrics were the mean absolute error (MAE), relative root mean square error (RMSE), relative mean square error (MSE), structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and mutual information (MI).

Results

The difference between the reference and synthetic effective atomic numbers was within 9.7% in all regions of interest. The averages of MAE, RMSE, MSE, SSIM, PSNR, and MI of the reference and synthesized images in the test data were 0.09, 0.045, 0.0, 0.89, 54.97, and 1.03, respectively.

Conclusions

In this study, an image synthesis framework using single-energy CT images was constructed to obtain atomic number images scanned by DECT. This image synthesis framework can aid in material decomposition without extra scans in DECT.

Introduction

In a conventional single-energy computed tomography (SECT) image, the pixel value represents the photon attenuation of the tissue. Materials with similar absorbance have the same CT numbers and are difficult to distinguish [1].

Dual-energy CT (DECT) uses two different energy levels, which can determine the ratio of the photoelectric effect components and Compton scattering [2]. It has been used to distinguish between tissues and characterize materials. DECT can obtain a variety of data, including an effective atomic number (Z_eff) and iodine- and calcium-enhanced maps [3]. Revolution CT (GE Healthcare, Milwaukee, WI, USA) reconstructs 120 kVp equivalent images and Z_eff using the Gemstone Spectral Imaging (GSI) technique [4]. Z_eff decomposition analysis can aid in the characterization of materials. Mileto et al. used Z_eff data to distinguish between non-enhancing renal cysts and enhancing masses [5]. Determining the electron density and effective atomic number is important to better understand the interaction of radiation and to accurately estimate the absorbed dose. For proton and carbon treatment planning, CT values are commonly converted into stopping power ratio (SPR_w) values using a conversion table for dose calculation [6]. However, this approach is restricted to specific human tissue compositions. Z_eff is useful for estimating the SPR_w for human tissues in complex anatomy [7]. However, it increases the radiation dose, scan time, and cost.

Convolutional neural networks (CNN) have been successfully applied to image processing and synthesis. Previous studies have developed a deep learning approach using a CNN to perform DECT imaging using standard SECT data. These studies have focused on noise reduction from scanned and synthesized DECT images [8, 9]. Generative adversarial networks (GANs) have two different networks: a generator network that synthesizes images and a discriminator network that distinguishes between the reference and synthesized images [10]. Kida et al. adapted CycleGAN to synthesize PlanCT-like images from CBCT images to improve the quality of CBCT images [11]. Charyyev et al. proposed image synthesis of DECT from SECT and reconstructed the SPR map [12]. The corresponding SPR maps synthesized from DECT reduced the artifacts and noise levels compared with those from the original DECT. In our previous study, we proposed an image synthesis framework that uses single-energy CT images at 120 kVp to obtain fat-water and bone-water images [13]. These studies demonstrated that an image synthesis network with GAN could synthesize DECT images from SECT images.

Herein, we propose an image synthesis approach to obtain effective atomic number images reconstructed from DECT based on GAN architectures.

Materials and methods

Data acquisition

A total of 18,862 images from 29 patients approved by the institutional review board were used for the analysis. The DECT images for each patient were acquired using a Revolution DECT scanner (GE Healthcare, Princeton, NJ, USA). DECT was performed at tube voltages of 80 and 140 kV and exposure of 560 mA. The scanning parameters were rotation time of 1.0 s, slice thickness (ST) of 5 mm, and field of view of 360 mm. The Z_eff and equivalent SECT images were reconstructed using the GSI technique.

Deep learning model

An overview of the comparison between the synthesized and reference Z_eff images is shown in Figure 1. The Z_eff image was synthesized using a GAN. The GAN framework is illustrated in Figure 2. The 16-bit DICOM image was converted into an 8-bit RGB portable network graphics (PNG) image. The output 8-bit RGB PNG images synthesized from the two-dimensional (2D) CNN model were also converted into 16-bit DICOM images [14]. The range of pixel numbers in the effective atomic number images was 0–255. Thus, the unused pixel values (255–65,356) were eliminated in the 16-bit (0–65,536) images and converted into 8-bit images. The SECT and DECT images were rescaled using RescaleIntercept and RescaleSlope from the DICOM header as follows:

Figure 1.

Comparison of synthesized and reference Z_eff images; the Z_eff image was synthesized from single-energy computed tomography (SECT) obtained from dual-energy computed tomography (DECT) with deep learning. The reference Z_eff image was obtained from the DECT; GAN — generative adversarial network

Figure 2.

Generative adversarial network (GAN) architecture for the image synthesis of Z_eff images from single-energy computed tomography (SECT) images; for gradient conversion, 16-bit DICOM images were converted to 8-bit PNG images

$$Image\hspace{0.17em}Data=(Image\hspace{0.17em}Data)\times RescaleSlope+RescaleIntercept+1000$$ (1)

The proposed 2D CNN model with GAN includes a generator to estimate the Z_eff image and a discriminator to distinguish between the reference and synthesized Z_eff images. These generator and discriminator networks were trained simultaneously by evaluating. The generator consists of an encoder and a decoder. The encoding maps used eight convolutional layers each, followed by batch normalization and leaky-ReLU activation functions. The number of convolutional and deconvolution filters is shown in Figure 2. The stride was 2 and the kernel size was 4 × 4. The discriminator used seven convolution layers to extract features from the input image and produce the output image. The input images (x) to generator G were SECT images, and the target images (y) were the corresponding Z_eff images. Discriminator D was trained to return the loss to which the given image was synthesized. The loss was calculated as follows:

$${ℒ}_{\text{GAN}}(\text{G},\text{D})={\mathbb{E}}_{\text{y}}[\text{log\hspace{0.17em}D}(\text{y})]+{\mathbb{E}}_{\text{x},\text{y}}[\text{log}(1-\text{D}(\text{G}(\text{x}))]$$ (2)

where G is the generator network and is the expected value dependent on both the SECT images (x) and target images (y). Moreover, it includes an additional loss, based on the absolute difference between the input and synthesized images (L1 norm loss). The L1 norm loss is calculated as follows:

$${ℒ}_{\text{L}1}(\text{G})={\mathbb{E}}_{\text{x},\text{y}}({\mid \text{y}-\text{G}(\text{x})\mid }_1)$$ (3)

Adversarial loss is calculated using the binary cross-entropy cost function. The final cost function is calculated as follows:

$${\theta }_{\text{G},\text{D}}=\text{arg}\hspace{0.17em}\underset{G}{\underbrace{min}}\hspace{0.17em}\underset{D}{\underbrace{max}}\hspace{0.17em}({ℒ}_{\text{L}1}(\text{G})+{ℒ}_{\text{GAN}}(\text{G},\text{D}))$$ (4)

The proposed image synthesis model was implemented using TensorFlow packages (V1.7.0, Python 2.7, CUDA 10.0) on an Ubuntu 16.04 LTS system. The number of epochs was 300. The dataset consisted of 18,826 DECT images scanned from the chest to the pelvis of 29 patients. The data were split into two sets: 16,726 images (21 patients) for training the models and 2100 images (8 patients) for testing.

Evaluation

The prediction accuracy of the model for the reference and synthesized Z_eff images was evaluated based on six metrics. The relative mean absolute error (MAE) and mean absolute percentage error (MAPE) were derived as follows:

$$MAE=\frac{1}{n_x{n}_y}\sum _{i,j}^{n_x{n}_y}\mid r(i,j)-t(i,j)\mid$$ (5)

$$MAE=\frac{1}{n_x{n}_y}\sum _{i,j}^{n_x{n}_y}\frac{\mid r(i,j)-t(i,j)\mid }{r(i,j)}$$ (6)

where and are the values of pixels in the reference Z_eff and target Z_eff images, respectively, and is the total number of pixels. The relative root mean square error (RMSE) is defined as

$$RMSE=\sqrt{\frac{1}{n_x{n}_y}{\sum _{i,j}^{n_x{n}_y}(r(i,j)-t(i,j))}^2}$$ (7)

The structural similarity index (SSIM) considers luminance, structure, and contrast between two images. The SSIM between two images and can be computed as

$$SSIM(\overrightarrow{x},\overrightarrow{y})=\frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({{\mu }_x}^2+{{\mu }_x}^2+C_1)({{\sigma }_x}^2+{{\sigma }_y}^2+C_2)}$$ (8)

$$C_1={(k_{1}Q)}^2,\hspace{0.17em}{k}_1=0.01$$ (9)

$$C_2={(k_{2}Q)}^2,\hspace{0.17em}{k}_2=0.03$$ (10)

where and are constants used to prevent a zero denominator. is the maximum Z_eff value for the reference and synthesized Z_eff images. The values of and are typically obtained from [15], and is an estimate in discrete form as follows:

$${\sigma }_x={\left(\frac{1}{N-1}\sum _{i=1}^N{(x_i-{\mu }_x)}^2\right)}^{1/2}$$ (11)

The correlation coefficient between and is defined as, and is given by

$${\sigma }_{xy}=\frac{1}{N-1}\sum _{i=1}^N(x_i-{\mu }_x)\left(y_i-{\mu }_y\right)$$ (12)

and is the mean intensity, which is given by

$${\mu }_x=\frac{1}{N}\sum _{i=1}^N{x}_i$$ (13)

The peak signal-to-noise ratio (PSNR) is calculated as

$${PSNR}_{GL}=10\times {log}_{10}\hspace{0.17em}\left(\frac{{(MAX)}^2}{MSE}\right)$$ (14)

The mutual information (MI) [16] is calculated as

$$I(r:t)=\sum _{m\in I_r}\sum _{n\in I_t}p(m,n)log\hspace{0.17em}\left(\frac{p(m,n)}{p(m)p(n)}\right)$$ (15)

where m and n are the intensities in the reference I_r and synthesized Z_eff images, and the predicted I_t and Z_eff image, respectively. p(m) and p(n) are the marginal densities, and p(m, n) is the joint probability density of I_r and I_t. Moreover, the difference between the reference and synthesized Z_eff images in the region of interest (ROI) was evaluated for several slices in the images from the chest to pelvis, as shown in Figure 3.

Figure 3.

Measurement region in the evaluation of the Z_eff value from pelvis to chest slices; the average and standard deviation values of Z_eff were measured by creating a circular region of interest (ROI) of 2 cm in diameter

Results

The losses of the generator, discriminator, and L1 norm are shown in Figure 4. The training time was approximately 154.8 ± 3.2 h. The times to synthesize the Z_eff images using the trained models were approximately 7.8–8.2 images/s.

Figure 4.

Average training losses in the discriminator, generator, and L1 norm for the training model

Figures 5 and 6 show samples of the synthetic Z_eff images at the pelvic and chest levels. A difference between the reference and synthetic Z_eff images was found on the body surface and at the edge of the heart. Table 1 presents the numerical and percentage differences in the Z_eff values between the synthetic and reference Z_eff images. The numerical and percentage differences of the Z_eff value were within 0.86 and 9.5%, respectively, in all ROIs. Table 2 lists the average MAE, MSE, RMSE, PSNR, and MI computed over multiple slices from the pelvis to the chest slices. The standard deviation (SD) from the pelvis to the chest slices was significantly smaller for all evaluation items.

Figure 5.

Samples of cross-modality Z_eff image generation results at pelvic level: input, output, and reference are the equivalent SECT image at 120 kVp, synthetic Z_eff images, and reference Z_eff images, respectively. The absolute error was calculated using the synthetic and reference Z_eff images

Figure 6.

Samples of cross-modality Z_eff image generation results at chest level: input, output, and reference are the equivalent single-energy computed tomography (SECT) image at 120 kVp, synthetic Z_eff images, and reference Z_eff images, respectively. The absolute error was the calculated using the synthetic and reference Z_eff images

Table 1.

Numerical (Δ) and percentage differences of the Z_eff value between synthetic and reference Z_eff images. The numerical and percentage differences of the Z_eff value were within 0.86 and 9.5% in all ROIs from the chest to pelvis

Measurement region	Δ	%
1	0.69	8.77
2	0.72	9.19
3	0.41	5.98
4	0.21	2.74
5	0.19	2.47
6	0.53	6.41
7	−0.11	−1.47
8	0.73	8.89
9	0.77	9.58
10	0.79	9.42
11	0.79	9.75
12	0.76	9.46
13	−0.06	−0.73
14	0.29	8.04
15	0.12	7.90
16	0.76	8.13
17	0.82	9.54
18	0.84	9.95
19	0.86	9.84
20	0.02	−0.26
21	−0.01	8.12
22	0.06	0.70
23	−0.08	−0.80

Table 2.

Evaluation metrics of Z_eff image synthesis from pelvis to chest slice

MAE		MAPE		MSE		RMSE		PSNR		SSIM		MI
Avg	SD	Avg	SD	Avg	SD	Avg	SD	Avg	SD	Avg	SD	Avg	SD
0.09	0.01	1.16	0.14	0.21	4.2E–03	0.45	4.6E–03	54.97	0.09	0.89	0.01	1.03	0.12

MAE — mean absolute error; MAPE — mean absolute percentage error; MSE — relative mean square error; RMSE — relative root mean square error; PSNR — peak signal-to-noise ratio; SSIM — structural similarity index; MI — mutual information; Avg — average; SD — standard deviation

Discussion

In this study, an image synthesis model for Z_eff images from SECT images using a deep learning approach was proposed. The numerical difference between the Z_eff values of synthesized and reference Z_eff images was within 9.95% in some regions, from the pelvis to chest slice. Mitchell et al. evaluated the Z_eff values obtained from DECT by comparing them with theoretical Z_eff values. The Catphan phantom (The Phantom Laboratory, Salem, NY, USA) had a Z_eff value accuracy of 15% when no lung inserts were used [17]. This suggests that the synthesized Z_eff image was in good agreement with the reference image within the uncertainty of the Z_eff image obtained from DECT.

The SD of the Z_eff values in the lung region of the Z_eff images was larger than that of other regions. This is because the lungs have a non-uniform structure. A previous study also showed that the measured Z_eff values of the inhaled lung insert in the CIRS 062M phantom were significantly different from the theoretical Z_eff values [18]. Thus, an accurate Z_eff image reconstructed from DECT is an essential input for deep learning. Further studies are needed to synthesize Z_eff values in the lung region using high-quality DECT images.

Schaeffer evaluated the accuracy of the Z_eff between the theoretical and measurement Z_eff from DECT. The MAPE was 6.3% for the body phantom and 3.2% for the head phantom [19]. The current study showed that the MAPE of the Z_eff was 1.16% ± 0.14 % with the GAN method. Moreover, Garcia et al. proposed a method of the extraction the Z_eff for the DECT image based on an Karhunen-Loeve expansion of the atomic cross section per electron [20]. The MAPE between the theoretical and calculated value was 4.1% ± 0.3%. Schaeffer et al. evaluated the accuracy of Z_eff from DECT. It suggests that the synthesized Zeff image showed a good agreement within the uncertainty of the Z_eff image obtained from DECT and the accuracy of the estimation for the Zeff was superior to the conventional method. Although the other evaluation metrics were used in the image synthesis study, it has never been used for the Z_eff image synthesis. These results of the evaluation metrics would be important in the image synthesis or conversion to the Z_eff from the DECT image in further studies.

An equivalent SECT image was used in this study. Kamiya et al. compared equivalent and conventional SECT images [21]. Although the radiation dose was reduced for the equivalent SECT image, the image quality was equivalent in both the quantitative and qualitative evaluations. Thus, the proposed model can be applied to conventional SECT images.

Zhao et al. proposed an image synthesis method to map low-energy to high-energy images using a two-stage CNN [16]. Zhao et al. evaluated virtual non-contrast imaging using DECT from SECT [16]. This might contribute to the prediction of perfusion imaging, urinary stone characterization, cardiac imaging, and angiography from SECT images. Our model extends the possibility of predicting the DECT image from the SECT image and contributes to the material decomposition with the predicted DECT image. Thus, the proposed image synthesis model can significantly simplify the DECT system design and reduce scanning and imaging costs. For radiation diagnosis, the Z_eff image should assist in lesion detection. The current study showed the possibility of efficient image synthesis of Z_eff images for material decomposition from a simple analysis. Further studies will be performed to evaluate the detectability of the lesions.

Conclusion

In this study, an image synthesis framework using single-energy CT images to generate atomic number images scanned by DECT was proposed. This image synthesis framework can aid in determining material decomposition without extra scans in DECT.