Multi-stage cascade GAN for synthesis of contrast enhancement CT aorta images from non-contrast CT

Abstract

Recently in diagnosis of Aortic dissection (AD), the synthesis of contrast enhanced CT (CE-CT) images from non-contrast CT (NC-CT) images is an important topic. Existing methods have achieved some results but are unable to synthesize a continuous and clear intimal flap on NC-CT images. In this paper, we propose a multi-stage cascade generative adversarial network (MCGAN) to explicitly capture the features of the intimal flap for a better synthesis of aortic dissection images. For the intimal flap with variable shapes and more detailed features, we extract features in two ways: dense residual attention blocks (DRAB) are integrated to extract shallow features and UNet is employed to extract deep features; then deep features and shallow features are cascaded and fused. For incomplete flaps or lack of details, we use spatial attention and channel attention to extract key features and locations. At the same time, multi-scale fusion is used to ensure the continuity of the intimal flap. We perform the experiment on a set of 124 patients (62 with AD and 62 without AD). The evaluation results show that the synthesized images have the same characteristics as the real images and achieves better results than the popular methods.

Introduction

Aortic dissection (AD) is a critical case in cardiovascular disease that is even fatal if left untreated¹. At present, contrast-enhanced computed tomography (CE-CT) serves as the predominant imaging technique for medical diagnostics, providing physicians with a wealth of diagnostic information². However, the obtain of CE-CT need to injected a kind of contrast agent. It is interventional, invasive and with lots of side effects. According to relevant literature reports³, non-ionic iodine contrast agents have varying degrees of impact on various organs, especially the skin and its accessories, immune system, digestive system, urinary system and nervous system during use. Even some people have allergic reactions after injecting contrast agent. Consequently, in clinical practice, non-contrast CT (NC-CT) is typically utilized for preliminary patient diagnosis, with CE-CT reserved for subsequent examination in cases where aortic dissection is suspected³. In addition, the accessibility of CE-CT is limited during specific times, such as nighttime or holidays, and in regions where medical resources are scarce. Unfortunately, there is a risk of delay from preliminary diagnosis to subsequent examination, which may lead to fatal events. Therefore, finding a diagnostic method that combines the high clarity of CE-CT with the low risk of NC-CT is important to improve the diagnostic efficiency and patient safety of AD using NC-CT.

Cross-modal medical image synthesis provides a promising avenue for addressing the above problems in AD diagnosis from NC-CT^4,5. Despite the limited visibility of AD in NC-CT, valuable features such as intimal calcification displacement and the presence of a highly attenuated false lumen can be discerned, underscoring the potential of synthesizing CE-CT from NC-CT⁶. In recent years, generative adversarial networks (GANs)⁷ and their variants^8,9 have made breakthroughs in the field of medical image synthesis. Xiong et al.¹⁰ used pix2pixHD to synthesize 2D slice images from NC-CT into CE-CT images with some success, but the results can be improved, and then the author constructed 3D multitasking network¹¹ to synthesize non-contrast CT images into enhanced CT images for the detection of aortic coarctation, but the model is large and has big memory cost. There is no proven CE-CT image synthesis method for aortic dissection.

The challenge of synthesizing CE-CT from NC-CT due to the intricate nature of its feature distribution. as shown in Fig. 1. The first row of the figure is the NC-CT scans and the second row is the corresponding CE-CT scans. The figure illustrates the following scenarios: (a) the indistinguishable image intensities between the true and false lumen in aortas with complete intimal flaps; (b) the presence of incomplete intimal flaps in some cases; (c) the stark contrast in image intensities within the true and false lumen in the presence of thrombus; and (d) non-aortic dissection image. It can be seen that an intimal flap dividing the aortic lumen into two distinct regions is crucial to the successful synthesis of CE-CT images in patients with AD. However, the intimal flap is thinness and indistinctness, coupled with the great variability in its morphological characteristics across different pathological cases, renders its identification and synthesis arduous.

Fig. 1.

Paired NC-CT and CE-CT scans.

To solve the above challenge, this paper proposes a multi-stage cascaded generate adversarial network (MCGAN) to synthesize CE-CT from NC-CT. The generator is composed of three parts: feature extraction, feature mapping, and feature fusion. Due to different structures, shapes, and detailed information of the intimal flaps, multiple convolution operations during feature extraction may result in loss of image details as the network deepens¹². In order to capture shallow features, a set of dense residual attention blocks (DRAB) is used instead of the original single convolutional kernel structure. DRAB can obtain feature combinations with different numbers and receptive field sizes, increase the receptive field of each layer of the network, and keep richer contextual and detailed information. Deep features can exploit the global information for the overall synthesis of CE-CT images. Deep and shallow features are cascaded in the feature mapping section to improve the extraction of detailed flaps; moreover, spatial attention and channel attention modules are used to enhance the result of feature extraction. Spatial attention focuses on the edges, texture, and shape changes of the intimal flap for a better discrimination between true and false lumens. Frequency-domain attention can detect small changes and abnormalities in the intimal flap that is hard to observe in the spatial domain. These two attentions mechanism complement each other to improve the recognition and synthesis accuracy of true and false lumen and intimal flaps. For incomplete flaps and small edges, multi-scale fusion is used to reduce information loss and integrate multi-scale characteristics, so as to ensure the continuity of flap information¹³. All these modules work together to boost the system performance.

The contributions of this work are the following:

1. In order to learn the shape-varying intimal flap features, we designed a new GAN model to synthesize aortic CE-CT images from NC-CT images, which extracts the deep and shallow features of the image separately, and then cascades them to ensure the adequate preservation of the feature.

2. In order to synthesize the location information of the true and false lumen, we propose an image mapping and fusion module for feature decoding that combines spatial and channel attention as well as multi-scale fusion to ensure the clarity and continuity of the intimal flap in the synthetic image.

Related work

Imaging studies of aortic dissection based deep learning

There is an increasing number of imaging studies for aortic dissection. Gayhart and Arisawa et al. developed a system for automatic assessment of aortic dissection disease on enhanced CT images, and they were among the first to investigate the problem of aortic dissection detection¹⁴. Compared with traditional methods, the deep learning algorithms can extract shallow biological structural features in medical images of AD. Akinori Hata et al.¹⁵ designed a DL algorithm for the detection of AD on non-enhanced CT images, which focuses on classification in terms of the statistics of the slices. Chen et al.¹⁶ developed a deep learning algorithm for segmentation and fast and automatic detection of aortic coarctation using contrast-enhanced computed tomography (CT) images, which focuses on detecting non-contrast CT on the morphological features for statistical analysis rather than visualization. Singh et al.¹⁷ constructed a deep learning model for automatic assessment of aortic dissection on chest CT, their study demonstrated the effectiveness of deep convolutional neural networks for aortic dissection image recognition.

The above methods have made substantial progress in the study of automatic segmentation and detection of aortic dissection, and they have laid foundation for AD image synthesis.

Medical image synthesis

In the field of medical imaging, challenges and costs associated with acquisition of different modalities have brought great chances for the synthesis of cross-modal medical images. There are differences in the visualization structure and clarity of these modalities, and therefore they have different roles in medical diagnosis and treatment. Multiple surveys have reported that generative adversarial networks (GANs) are capable of synthesizing realistic medical images for clinical applications^18,19. Hiasa et al.²⁰ extended the loss function of CycleGAN to generate CT images from MRI images, thus eliminating the need for radiation exposure during CT scans. Santini et al.²¹ used CNN to extract relevant features from non-contrast CT images and remap them onto typical features of CE-CT to synthesize virtual iodine-enhanced heart images. To minimize the use of gadolinium-based contrast agents, Gong et al.²² developed an encoder-decoder model that can generate contrast-enhanced MRI images from low-contrast MRI images. Chen et al.²³ designed a 3D fully convolutional network to synthesize contrast-enhanced MRI images from non-contrast-enhanced MRI images. Olut et al.²⁴ introduced a manipulable GAN for synthesizing MRI angiography images using multi-contrast MRI images. Chandrashekar et al.²⁵ utilized a CycleGAN network to identify subtle differences between soft tissue components in CT images without contrast agent. To avoid the adverse effects of fluorescein angiography, Fang et al.²⁶ proposed a cGAN network to generate high-contrast angiographic images of the retina using non-contrast images. A fully convolution network containing an encoding path and a decoding path is designed to synthesize cardiac contrast enhanced CT image, with the purpose of calculating the left cardiac chambers volume and evaluating the contrast dynamics ²⁷.

However, these methods are associated with inconsistent data, variable degrees of disease-related regions. Highly nonlinear relationships between different imaging modalities makes it challenging to learn the mapping from one modality to another. Consequently, the synthesized medical images often exhibit discrepancies in fine-scale structure when compared to the real ones.

Method

We propose multi-stage cascade GAN (MCGAN) for CE-CT image synthesis. Figure 2 illustrates the pipeline of the proposed synthetic framework. The framework is composed of two parts: generator and discriminator. In the generator, NC-CT images are used as input to synthesis enhanced CT images. It consists of three parts operations: feature extraction (Fig. 2a), feature mapping (Fig. 2b) and feature fusion (Fig. 2c). In the discriminator, the synthetic image and the original real image together are used as input to discriminate the real from the fake. We use the multi-scale discriminator in pix2pixHD to evaluate the output of the generator. For the synthetic image Y generated by the generator, the first discriminator D1 takes the original image X and the synthetic image Y as inputs, while the second discriminator D2 takes the downsampled X/2 and Y/2 of the original and synthetic images as inputs. The details of the generator are introduced in the following sub-sections.

Fig. 2.

Network architecture of MCGAN.

Feature extraction

Shallow features usually have a smaller receptive field, focus on detail information, and can capture more fine-grained features such as edges and intimal flaps. Deep features have larger receptive fields, contain rich semantic information, focus more on global information, and provide the overall structure of the image. In order to extract these features adequately, feature extraction is performed in two ways, DenseNet downsampling for shallow features and a global generator for deep features, showed as Fig. 2a. DenseNet is composed of iterative dense residual attention blocks (DRAB) with different output dimensions. Dense connections have obvious advantages. On one hand, the information loss in the feedforward of CNNs can be reduced, ensuring maximum utilisation of information, the problem of vanishing gradient can be alleviated, feature propagation can be improved. On the other hand, dense connections have a regularizing effect, which reduces the risk of over-fitting on tasks with smaller training sets. Moreover, in order to adequately extract the aortic intima flap features in the NC-CT images, the network retains the shallow features during the DRAB, while the deep features are obtained through a global generator. The global generator is a UNet with two convolution downsampling layers and two deconvolution layers. The extracted features from top to bottom are of size 256 × 256 × 32, 128 × 128 × 64, 64 × 64 × 128, 32 × 32 × 256. The output of the global generator has the same dimension as the input, which is of size 32 × 32 × 256.

Figure 3a exhibits the structure of DRAB, which is made up of three RABs, a convolution layer, six skip connections, three concatenation layers and an identity connection. Dense connections are used to ensemble the features from different layers for strengthening feature propagation. The 1 × 1 convolution is adopted to decrease network width while retaining salient features and thus improve computational efficiency. Dense connections have significant advantages. Firstly, they can reduce information transfer loss in feedforward CNNs, alleviate the problem of vanishing gradient, and enhance feature propagation. Additionally, dense connections exhibit a regularizing effect, which reduces the risk of overfitting on tasks with relatively small training sets.

Fig. 3.

The details of modules.

Figure 3b illustrates the structure of residual attention block (RAB). In each RAB, one BN-ReLU- conv(1 × 1) and one BN-ReLU-conv(3 × 3) are stacked, followed by a full-spectrum channel attention module (FCA). The channel attention is introduced into DenseUNet, so that the network can focus more on critcal information when extracting features. The details of the CA are introduced in the following sub-sections.

Local feature mapping

After the feature extraction, the shallow features at different scales and the deep global features are transferred to the local feature mapping part. In this part, the shallow features and deep features are cascaded four times, as shown in Fig. 2b. In order to extract the key features of the images, this part contains two attention modules: residual attention block (RAB) and spatial attention module (SA). Channel attention is embedded in the RAB. Complementary information extracted by two attention models is employed to improve feature mapping.

The spatial attention module

In shallow features, the region of true and false lumen obstructed by the intimal flap should be focused on. The position of intimal flap should be determined. Inspired by the spatial attention proposed in CA-Net²⁸, we use the dual-pathway spatial attention (SA) using high-level features to guide low-level dense features. The spatial attention module is utilized to strengthen the effective feature regions, while suppressing the irrelevant parts. The module automatically learns to focus on target structures of different shapes and sizes, achieving the target of feature localization. The details about SA are shown in Fig. 3d, e.

The inputs of the attention module are two vectors x^d and x^s with same dimension. x^s respect a shallow feature map from DenseNet feature extraction and x^d respect the deepest feature map from global generator. They are compressed to the same output channels of C by a 1 × 1 convolution. The outcomes are summed and subsequently processed through a ReLU activation function. The ReLU-generated feature map is subsequently inputted into another 1 × 1 convolutional layer with a single output channel, which is then passed through a sigmoid function to yield a pixel-by-pixel attention coefficient α, ranging from 0 to 1 and sized H×W. Subsequently, x^s undergoes calibration by being multiplied with α. In our SA, the spatial attention maps for both pathways are represented by α and αʹ, respectively. The output of SA can be expressed as:

1

where is the 1 × 1 convolution with output channel C. Here C is 256, 128, 64 and 32 from bottom to top in feature mapping part. It is worth noting that, in the deepest layer of the feature mapping, x^s is the input of the global generator and x^d is the output of the global generator, the two features cascaded first time into the SA. Subsequently, x^s and x^d cascaded three times into the SA, and x^d is the deconvolution of the deep features.

The channel attention module

In aortic dissection, the intraluminal membrane of the aorta at the diseased site is ruptured, and the visual characteristics of the boundary between the true and false cavities are not obvious on the CT image. They are mostly manifested in the form of dissection tear lines, with small slits. However, because the sharp changes in the image gray value at the edge of the aortic tear belong to high-frequency information, the frequency domain is very different. Therefore, in order to improve the feature expression ability, the frequency channel attention module is used to further filter the features of different channels in the feature map to improve the feature expression ability of key channels, and further guide the network to focus attention to areas containing key information.

Inspired by ECANet²⁹, FCANet³⁰, and channel attention in CAN³¹, we extend the channel attention and proposed a light full-spectral channel attention called FCA. The details about FCA are shown in Fig. 3c. First, the input feature is divided into n parts by channel dimension, where n must be divisible by the number of channels. For each part, the corresponding 2D DCT frequency component is computed, and the result can be used as a preprocessing result for channel attention. Here, 2D DCT frequency component have included all the spectral information instead of global average pooling in SENet³², so it retains more useful information. Immediately after that, we concatenate the frequency components of the parts. Then, we performed a multi-layer perceptron to obtain the cross-channel interaction information and a convolution to adjust the channel number. Finally, we computed the attention values α using a Sigmoid function, the normalized weights are multiplied channel-by-channel with the original input feature map to generate the weighted feature map. The FCA can be expressed as:

2

where is the 1 × 1 convolution with output channel C. Here C is 256, 128, 64 and 32 from bottom to top in feature mapping part. The attention mechanism is introduced into the feature mapping, and the feature of the positioning is more accurate and the details are richer.

Multi-scale feature fusion

Our generator obtains four different scale features in feature maping. To enhance the retention of spatial information, these features undergo interpolation and subsequent upsampling. This process serves to redistribute the existing information across a larger spatial extent. We then integrate these features using concatenation and standard convolutional layers to construct the final synthesized image, as illustrated in Fig. 2c. This approach leverages the multi-scale information to enrich the feature maps, allowing our network to capture and synthesize finer image details, thus generating more realistic images.

Loss fuction

In order to be able to better form aortic dissection images, multifaceted constraints are introduced in the loss function, adversarial loss, feature matching loss and perceptual loss. These constraints can help the generator to better learn the structure and content of the aortic dissection images. The main reason for using feature matching loss and perceptual loss instead of MSE is the excessive smoothness and lacking in details of the generated images by MSE loss. The feature matching loss and the perceptual loss ensure that the content is consistent and the details are learned by the adversarial loss.

The adversarial loss, feature matching loss and perceptual loss are expressed in formulas (3), (4), (5) and the total loss is in formulas (6).

3

4

where, n is the number of layers and N_i is the number of elements per layer. (e.g., convolution, normalization, leaky-ReLU implies three elements). Minimizing this loss ensures that each layer extracts the same features from pairs of images.

5

where, and denote their feature representations at layer i in the pre-trained neural network, respectively, and N is the number of feature layers. The pre-trained neural network in the experiment is VGG19 network, which utilizes perceptual loss to extract features from paired images. The total loss function is defined in formulas (6).

6

Experiments and results

Dataset

Data acquisition

The dataset used in this study consisted of 124 different patients, including 62 each of aortic dissection patients and non-aortic dissection patients. The dataset was collected and manually annotated by specialized imaging physicians. The study was approved by the institutional review board. Each subject consists of a NC-CT scan and a corresponding CE-CT scan. The paired scans were collected at the end of respiration, and share same spatial resolution that range from 0.549 × 0.549 × 0.625 to 0.977 × 0.977 × 1.250mm³. The tube voltage of the collection device is 100–120KVp. After obtaining the original data, the personal information of each subject was desensitized.

Data preprocessing

Before feeding the data into the model, some preprocessing operations are required. The cross-sectional images obtained from CE-CT and NC-CT images are sharper than sagittal and coronal images, with higher density resolution. Thus, the following pre-processing procedure is performed on the paired cross-section as shown in Fig. 4.

Fig. 4.

Data preprocessing flow. NC-CT (the first row) and corresponding CE-CT (the second row).

For the entire scanned CE-CT and NC-CT images, in order to highlight the abdominal region, the entire image was cropped to contain only the abdomen, removing a large amount of background. In addition, affected by the body size of different subjects, the number of scan layers is not consistent. We crop all the image layers to keep the number of images slices the same, and all of them contain the aorta. The progress shown in Fig. 4a to b, and Fig. 4b respect the paired cropped image.

To cope with some misalignments in the images caused by respiratory motion, a rigid registration on NC-CT and CE-CT image pairs is performed, using the NC-CT images as references and the mutual information as cost function to minimize. The progress shown in Fig. 4b to c, and Fig. 4c respect the paired registration image.

The resolution of different subject is not consistent, due to different acquisition equipment, different scanners, and different parameter settings. For such reason, all volumes of aortic region are resampled to most frequent resolution 0.7031 × 0.7031 × 0.625 mm³ by trilinear interpolation. The progress shown in Fig. 4c to d, and Fig. 4d respect the paired resampled image.

Affected by metal instruments during scanning, the NC-CT and CE-CT slices have metal outliers on the periphery. Outliers were eliminated by threshold method to avoid artifacts after image synthesis, and then entire image is normalized to [0,1]. All normalized images were matched by histogram matching to ensure that all matched training images have similar contrast ranges. The progress shown from Fig. 4d to e, and Fig. 4e respect the paired images with singularities eliminated.

Making data sets

During the synthesis of CE-CT images from NC-CT images for aortic dissection, the aorta constitutes a relatively small proportion of the extensive field of view encompassing the chest and abdomen. Consequently, to mitigate the confounding effects of the background and other anatomical structures, we extract the aortas from paired NC-CT and CE-CT volumes before aortic synthesis, as shown in Fig. 4f. Initially, professionals manually delineated the aorta in CE-CT images of healthy subjects using ITK-Snap software. Then, these manual annotations served as labels for training a 3D nnU-Net segmentation network³³, which automated the extraction of aortic masks from the CE-CT images. And finally, postprocessing techniques were applied to refine the mask borders, and the resulting masks were aligned with their respective NC-CT and CE-CT image pairs to isolate the aortic regions effectively. For the segmentation of the aortic regions in our dataset, we employed ITK-Snap software, version 3.8.0, an advanced tool for medical image segmentation and visualization. More information about ITK-Snap, including download access, can be found at http://www.itksnap.org.

After segmentation, our dataset was randomly selected, with 100 pairs allocated for training the aorta synthesis model and a separate 24 pairs reserved for validation. The training set consisted of 50 healthy pairs and 50 aortic dissection pairs. The NC-CT and CE-CT volumes are sliced to obtain a series of 2D images perpendicular to the aortic layers. Axial slices that contained no aorta information were discarded. The volumes were subsequently cropped to size of 256 × 256 × 420 pixels. Finally, coordinate transformation was applied in all volumes to get three 2D datasets, including axial, sagittal and coronal, shown as Fig. 5.

Fig. 5.

Example of the dataset before and after segmentation on axial, sagittal and coronal. (a) Without aortic dissection, (b) with aortic dissection.

Implementation details

All experiments were implemented with Pytorch. We trained the model on two NVIDIA GeForce RTX 2080 Ti with 11GB memory each, for 200 epochs. We set the intimal learning rate 2 × 10⁻⁴ and use Adam optimizer with beta1 0.5, decay rate of learning 1 × 10⁻⁷. for feature matching loss is set to 10.

Experimental comparison

We compared the results of MCGAN with the classic image synthesis models of Pix2pix³⁴, CycleGAN³⁵ and Pix2pixHD³⁶, as well as recent literature on AD synthesis Cascaded Deep Learning Framework(CDLF)¹⁰ and 3D MTGA[11]. This comparative analysis was conducted on both qualitative and quantitative fronts, with a focus on the PSNR and SSIM metrics to evaluate the synthesized images’ quality relative to ground truth.

Table 1 shows the qualitative results. The comparative analysis reveals that MCGAN outperforms in both PSNR and SSIM metrics, with a PSNR of 32.85 and an SSIM of 0.9899. While Pix2pixHD and CDLF show competitive results, MCGAN demonstrates a slight improvement, suggesting its enhanced ability to capture fine details and structural integrity in synthesized images. The performance of CycleGAN and Pix2pix is slightly lower, indicating the superior of multi-stage cascade structure, which is particularly beneficial for applications requiring high-fidelity image generation.

Table 1.

PSNR and SSIM (mean ± standard deviation) of several algorithms.

Method	PSNR	SSIM
CycleGAN	31.23 ± 3.356	0.9841 ± 0.002
Pix2pix	31.50 ± 2.867	0.9846 ± 0.002
Pix2pixHD	31.78 ± 3.225	0.9879 ± 0.003
CDLF	32.12 ± 3.356	0.9882 ± 0.003
3D MTGA	31.89 ± 3.356	0.9880 ± 0.004
MCGAN	32.85 ± 3.048	0.9899 ± 0.002

Significant values are given in bold.

Besides quantitative evaluation, Fig. 6 shows a visual comparison between five methods and our framework. The images of each view contain a lesion area and its magnification. The first column is the input NC-CT image and the last column is the real CE-CT image as the ground truth. As can be seen from the first column of Fig. 6, CycleGAN is completely unable to synthesize the intimal flap in the yellow box in all three views of the aorta. This leads to erroneous synthesis results, synthesizing cases with AD as without AD. Pix2pix is a bit better than CycleGAN in terms of brightness, but it only synthesizes some noise in the lesion area and does not synthesize the intimal flap. Pix2pixHD generates images with higher resolution and brightness, but also lacks grey scale detail of the flap. We observe that the CDLF generates intimal flaps with a degree of blurriness. Despite the lack of sharpness, the results achieved by CDLF are superior to those produced by the 3DMTGA. This may be attributed to the insufficiency of a small dataset to train a robust 3DMTGA model. Compared with these methods, our MCGAN provides smooth, clear and correct synthesis results for all three views. MCGAN is able to synthesize the intimal flap of aortic dissection close to the ground truth in all three views. Especially on axial, the features of the curved and irregular intimal flap are also well preserved, thus MCGAN captures subtal image details and improves the overall performance, indicating that MCGAN achieves better visual effect and less distortion than other methods.

Fig. 6.

Comparative visualization of synthesized CE-CT: axial, sagittal, and coronal views with zoom-in.

Ablation study

Effects of cascade structure and number

In our method, when extracting NC-CT image features, shallow features and deep features are extracted by different methods, and fuses them in a multi-stage cascade. To verify the benefit of the multi-stage cascades, we also compared MCGAN with single-stage, two-stage and three-stage cascade networks by choosing the same parameters.

Table 2 shows the results of experiments by different number of cascades. The first column shows the number of cascades, once, twice, three times and four times. The more the number of cascades, the more the downsampling module of the local generator increases while the number of downsampling layers of the global generator decreases. The local generator of MCGAN has three dense downsampling modules, the global generator has two convolutional downsampling layers, and the local and global features are cascaded four times. Cascading three times means that the local generator has two dense downsampling modules and the global generator has three convolutional downsampling layers. Thus, cascading once means that no local dense downsampling module is used, and the downsampling process consists entirely of the five convolutional downsampling layers of the global generator. The second and third columns of Table 2 show the values of the evaluation indicators PSNR and SSIM, respectively. It can be seen that our four times cascade method gives better results than all other methods. Cascade once attained the worst performance in all evaluation metrics while cascade twice and three times had comparable performance. The reason for this is that cascade once did not use dense connectivity module for downsampling in feature extraction, which resulted in feature propagation loss. Also, the attention module was not used in feature mapping, resulting in a semantic gap between the encoder and decoder.

Table 2.

PSNR and SSIM (mean ± standard deviation) of different cascade times.

Cascade times	PSNR	SSIM
One	31.61 ± 2.953	0.9852 ± 0.003
Two	32.13 ± 3.465	0.9882 ± 0.004
Three	32.54 ± 2.857	0.9891 ± 0.003
Four	32.85 ± 3.048	0.9899 ± 0.002

Significant values are given in bold.

Significant values are given in bold.

Figure 7 shows views of axial, sagittal and coronal synthesized CE-CT images for a randomly selected subject with AD. When cascading once, the synthesized image as a whole looks noisy and not smooth enough, the aortic edges on the axial view are not clear enough, and the synthesis of the intimal flap is incomplete. In the coronal view, the synthesized intimal flap is not continuous. Even in the sagittal view, no intimal flap is synthesized. When cascaded twice, the synthesized aortic valve is distorted on the axial view and blurred on the coronal view. When cascaded three times, intimal flap features were extracted in all three views, but compared with ground truth, the features were discontinuous in the axial view, the intimal flap was unclear in the coronal view, and no intimal flap is synthesized in the sagittal view. Here, as can be seen from ground truth images, the second row of coronal view, there is inconsistency in the intensity values of the true and false lumen caused by the thrombus. As a result, noise and artefacts in the synthesizes image increase, interfering with the extraction of intima features. In the third row of the sagittal view, the intimal flap in the lesion area is very thin and difficult to distinguish from the surrounding tissue, and noise may interfere with intimal recognition and synthesis. Compared to cascade once, twice and three times, our method with cascade four times synthesizes images closer to the true value in three views. It is evident that features are extracted both shallowly and deeply, and then integrated in multi-stage cascades. This approach ensures the retention of both global and detailed features. This indicates the necessity and superiority of cascade.

Fig. 7.

Ablation visualization of cascade times: axial, sagittal, and coronal views with zoom-in.

Effectiveness of the attention module

Our method uses the DRAB and UNet to obtain four different scale features, and then implement the SA and CA to perform the synthesis. Therefore, these attentions play an important role in our method. To verify the necessity of the spatial attention and channel attention modules in local feature mapping, we separately synthesized CE-CT images from the same set of NC-CT images without attention, with spatial attention only and channel attention only, respectively.

The first column of Table 3 shows the module of the attention. No attention implies direct cascade fusion of global and local features of the feature mapping part, similar to a direct skip-connection between an encoder and a decoder, at each time. Only spatial attention means that the features at each scale obtained from downsampling are extracted by spatial attention and then cascaded with the global features. Only channel attention means that the downsampled features at each scale are directly cascaded with the global features, which are fused by channel attention before being upsampled. The second and third columns of Table 3 show the values of the evaluation metrics PSNR and SSIM, respectively.

Table 3.

PSNR and SSIM (mean ± standard deviation) of different attention combinations.

Attention	PSNR	SSIM
Without attention	31.59 ± 3.811	0.9852 ± 0.003
Channel attention only	31.90 ± 2.283	0.9893 ± 0.004
Spatial attention only	32.56 ± 3.536	0.9899 ± 0.002
Double attention (ours)	32.85 ± 3.048	0.9899 ± 0.002

Significant values are given in bold.

It shows that using dual attention in our method are better than the results of any other method. Without attention shows no good performance in all evaluation metrics. Using only channel attention has better results than without attention. Spatial attention only had better performance than channel attention only. The reason for this is that the failure to use attention to directly cascade global and local features without paying sufficient attention to the key information makes the synthesis less effective. In CT images, different channels can correspond to distinct structures, and aortic dissection can exhibit unique characteristics in specific channels. Through the channel attention mechanism, the model can adaptively adjust the weights of different channels to emphasize the features related to intimal flap and suppress other irrelevant features. With the spatial attention mechanism, the model can pay better attention to the location and shape variations of the aortic coarctation and accurately synthesize the corresponding regions in the enhanced CT images. The combination of the two attention improves the accuracy of the synthesis, resulting in higher values of PSNR and SSIM.

Figure 8 shows axial, sagittal and coronal views of synthesized CE-CT images. In the case of no attention, endocardial slices were blurred or even absent. With spatial attention alone, the generated image endomembrane slices are incomplete and unclear. With channel attention alone, the generated image inner membrane slice is not continuous enough. The whole image generated by using double attention is more uniform, and the continuity of the length of the inner membrane slice is also better. Compared with no attention, spatial attention only and channel attention only, MCGAN with double attention makes great contributions to performance improvement of image synthesis. This indicates the necessity and superiority of double attention.

Fig. 8.

Ablation visualization of attention combinations: axial, sagittal, and coronal views with zoom-in.

Effectiveness of multiscale fusion

The four scale features obtained from the cascade of shallow feature and deep feature of MCGAN generator are multi-scale fusion. To evaluate the superiority of feature fusion, we perform the GAN with and without feature fusion separately.

Without multiscale feature fusion means that the feature mapping process has only one output, and each layer of features is up-sampled until it reaches the input size and then output, i.e., the synthetic image. Unlike the proposed MCGAN network in which there are four different scales of feature outputs in the feature mapping process, containing multiscale fusion i.e. as described in the above section. Table 4 shows the experimental results of fusion effectiveness. The second and third columns of Table 4 show the values of the evaluation metrics PSNR and SSIM respectively. Multi-scale fusion can comprehensively utilize local and global features at different scales to improve the richness and diversity of features. It can be seen that the fused synthetic evaluation results are better than those without fusion.

Table 4.

PSNR and SSIM (mean ± standard deviation) of multiscale fusion.

Methods	PSNR	SSIM
No fusion	32.29 ± 2.611	0.9890 ± 0.002
Fusion (ours)	32.85 ± 3.048	0.9899 ± 0.002

Significant values are given in bold.

Figure 9 shows three views of synthesized CE-CT images. Without feature fusion, the synthesized image has a grid effect and overall lack of clarity. And the synthesized image after performing multi-scale fusion is richer in detail information. Multi-scale feature fusion allows for better understanding and reconstruction of shape, edge, and texture information of the intimal flap, which improves the accuracy of the synthesis.

Fig. 9.

Ablation visualization of multiscale fusion: axial, sagittal, and coronal views with zoom-in.

Model analysis

Feature maps and attention maps in different scales

In our MCGAN generator, dense residual attention blocks (DRAB) are employed during the feature extraction downsampling phase, ensuring the propagation of early-layer features directly to subsequent layers, which aids in retaining spatial information. The visualization of feature and attention maps at various scales is depicted in Fig. 10. The upper left portion adjacent to the dashed line in Fig. 10. signifies the NC-CT image, with the lower representing the CE-CT image. The upper row to the right of the dashed line showcases the feature maps following each downsampling iteration. It is evident that the network can effectively capture and preserve key aortic features, even when the aortic region is considerably diminished. Furthermore, the incorporation of channel and spatial attention in the MCGAN’s feature mapping is demonstrated in the lower row to the right of the dashed line in Fig. 10, where attention maps at different scales are presented. These attention mechanisms significantly bolster the network’s capacity to discern critical aortic features, underscoring the role of attention in refining feature identification within the context of medical image synthesis.

Fig. 10.

Feature maps and attention maps in different scales.

Computational complexity

In Table 5, we delineate the floating-point operations (FLOPs) and average inference times for the models under comparison. The MCGAN exhibits a computational complexity of 122.96 GFLOPS and inference time is extended to 0.0092 s, substantially higher than its counterparts. Despite the increased computational expense, the MCGAN’s architectural sophistication has been empirically demonstrated to enhance the quality of synthesized images significantly, capturing and rendering image key features with greater finesse. Moreover, we argue that the increased FLOPs and inference time are a justifiable trade-off for the performance gains achieved by MCGAN. Given the ongoing advancements in hardware capabilities, especially in the realms of parallel processing and specialized hardware acceleration, the computational demands of MCGAN are manageable and are projected to diminish over time.

Table 5.

Comparison of FLOPs and averaged inference time for various models.

Models	FLOPs (G)	Inference time (s)
CycleGAN	50.05	0.0044
Pix2pix	16.03	0.0016
Pix2pixHD	21.44	0.0019
CDLF	24.58	0.0028
3D MTGA	56.95	0.0051
MCGAN	122.96	0.0092

Conclusions

The synthesis of CE-CT images from NC-CT images is an important technique in AD diagnosis. The NC-CT images exhibit numerous artefacts and noise, resulting in small density variations that hinder the precise discrimination of intricate vascular structures. Additionally, the dynamic alterations in the shape and position of the true and false lumens pose a significant challenge for accurate imaging capture. MCGAN is proposed in this paper to overcome these difficulties. Firstly, the aortic regions of interest in paired NC-CT and CE-CT were segmented, using the nnU-Net network and post processing. Then, the generator of MCGAN takes segmented aortic NC-CT slices as inputs to produce synthetic CE-CT images, and the discriminator takes real CE-CT slices and synthetic CE-CT images together as inputs, to guide the generator to synthesize more realistic CE-CT slices. The generator of this network consists of three parts, feature extraction, feature mapping and feature fusion. Feature extraction is performed in two ways, DRAB for shallow features and UNet for deep features. This approach not only captures local features of the image, such as texture and thrombus, but also provides more global structural information, such as direction and branching of aorta. The feature maps cascade and fuse shallow and deep features, and exploit channel attention and spatial attention. By complementing spatial and frequency domain information, the network focuses on the morphology and changes of the intimal flaps, improving the synthesis accuracy of true and false lumen and intimal flaps. The feature fusion achieves multi-scale fusion of the four scales obtained by cascading shallow and deep features. This approach better captures the richness and diversity of intimal flap features, helping to improve the quality and authenticity of synthetic images, making the synthesis of true and false luminal intimal flaps clearer and more accurate.

The PSNR and SSIM values of the synthesized image and the original image were calculated through the comparison experiment. The results demonstrated that the proposed framework generally outperforms baseline methods both quantitatively and qualitatively. The benefit of the cascade strategy, attention modules and multi-scale fusion is also proved by the convincing results. To assess the proposed method comprehensively, we will research on the follow-up quantitative analysis (e.g., image segmentation and detection) of generated CE-CT images in future.

Author contributions

Juanjuan YIN and Huijuan Tu designed the study. Juanjuan Yin and Jinye Peng collected the clinical data and designed the experiment. Juanjuan YIN performed the experiments and wrote the first draft of the manuscript. Xiaohui LI, Jianguo JU, Jun Wang and Huijuan Tu revised the manuscript. All authors reviewed the final manuscript.

Funding

This work was partially supported by the Kunshan City Traditional Chinese Medicine (TCM) Science and Technology Development special fund (KZYY202302), and the Suzhou Medical Association “Imaging Medical Star” general project (2023YX-M04).

Data availability

The validation datasets generated and/or analysed during the current study are not publicly available due to the ethical and patient privacy regulations, but are available from the first author (Juanjuan Yin) on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Ethical and informed consent for data used

This study was approved by institutional review boards of Beijing Anzhen Hospital, Capital medical university, and Fujian Province Hospital. Informed consent was obtained from all participants. The study was conducted in accordance with relevant guidelines and regulations.