Synthetic computed tomography generation for abdominal adaptive radiotherapy using low-field magnetic resonance imaging

Highlights

• •Deep learning generation of synthetic Computed could enable in-bore abdominal adaptive radiotherapy.
• •Full scale synthetic Computed Tomography images were generated using U-Net and Generative Adversarial Network architectures.
• •Image fidelity and dose accuracy metrics show feasibility for synthetic Computed Tomography use in adaptive radiotherapy.

Abstract

Background and Purpose

Magnetic Resonance guided Radiotherapy (MRgRT) still needs the acquisition of Computed Tomography (CT) images and co-registration between CT and Magnetic Resonance Imaging (MRI). The generation of synthetic CT (sCT) images from the MR data can overcome this limitation. In this study we aim to propose a Deep Learning (DL) based approach for sCT image generation for abdominal Radiotherapy using low field MR images.

Materials and methods

CT and MR images were collected from 76 patients treated on abdominal sites. U-Net and conditional Generative Adversarial Network (cGAN) architectures were used to generate sCT images. Additionally, sCT images composed of only six bulk densities were generated with the aim of having a Simplified sCT.

Radiotherapy plans calculated using the generated images were compared to the original plan in terms of gamma pass rate and Dose Volume Histogram (DVH) parameters.

Results

sCT images were generated in 2 s and 2.5 s with U-Net and cGAN architectures respectively.

Gamma pass rates for 2%/2mm and 3%/3mm criteria were 91% and 95% respectively. Dose differences within 1% for DVH parameters on the target volume and organs at risk were obtained.

Conclusion

U-Net and cGAN architectures are able to generate abdominal sCT images fast and accurately from low field MRI.

1. Introduction

In the Radiotherapy (RT) workflow, the use of Magnetic Resonance Imaging (MRI) allows for excellent soft tissue identification, helping on the localization of tumors and organs at risk. The development of MR-LINACs that couple an MR scanner with a Linear Accelerator is redesigning the importance of MRI in RT. The status of MRI is changing from a planning only tool into an active monitoring role in day-to-day imaging, allowing for on-table adaptive RT. Nevertheless, Computer Tomography (CT) remains necessary for dose calculation.

The current workflow for Magnetic Resonance guided Radiotherapy (MRgRT) requires the acquisition of two imaging modalities, MRI and CT. The MR image is used to contour the target volume and organs at risk by the radiation oncologist. The CT, quantified as Hounsfield Units (HU), is used to obtain an electron density map for dose calculation. Both imaging modalities are registered, resulting in the deformation of the CT image. Nevertheless, this fusion process can still introduce systematic errors on the planned dose [1].

This issue persists in Adaptive Radiotherapy (ART), where daily MR images are registered to a single CT acquired during planning. This could affect the performance of the RT treatment based on this image registration.

A possible solution is the generation of synthetic CT (sCT) images which can provide the electron density information necessary for dose estimation. This has been demonstrated by atlas-based methods [2], [3], bulk density methods [4], [5] and Deep Learning (DL) techniques [6]. This process remains a challenging task considering the diversity of tissues in the human body. In this context, bulk methods for sCT image generation simplified this task by assigning homogeneous electronic densities to similar tissues. The main advantage here stands in their simplicity for image generation. However, they are hardly competitive to DL methods in terms of image quality and dose calculation. Additionally, they often rely on the manual segmentation of tissues, which makes them user-dependent and time consuming. Nevertheless, automatic segmentation methods could be used to improve in this regard. The number of different electronic densities used for the bulk generation of sCT changes with some studies focusing on three or four considering soft tissue, bone, air and lungs [5], [7], [8]. Others using up to five densities: soft tissue, bone, air, lung, and fat [9], [10].

DL techniques for the generation of sCT images from MRI have shown promising results for pelvis and brain imaging [26], [11], [12], [13], [14], [15], [16], [17]. However, the use of these methods for the abdomen has been scarcely studied [25], [18], [19], [20], precisely in a region for which ART can be critical. Regardless of the application, the most commonly used neural network architectures, U-Net and Generative Adversarial Networks (GANs) [6], [22], are expected to be relevant for abdominal sCT.

In this study we aimed to generate sCT images using low-field MR images with U-Net and GAN neural network architectures. We assessed the performance of each network in terms of image similarity metrics and dose accuracy. Additionally, we generated bulk density images containing only six electronic densities with a DL-based approach.

2. Methods

2.1. Data

For this study, we used abdominal MRI and CT image pairs from 100 patients undergoing treatment with the MRIdian MR-LINAC at Institute Paoli-Calmettes in Marseille, France. The project was conducted in accordance with the Declaration of Helsinki and French Good Clinical Practice, with the agreement of an ethic committee. 24 patients with metallic prostheses, stents and atypical anatomies were excluded from selection to avoid the presence of artifacted CT and MR images on the training and testing cohorts. From the remaining 76 patients, 35 were diagnosed with pancreatic cancer (18 female, 17 male), 29 with liver cancer (13 female, 16 male) and 12 had other types of diagnosis localized on the abdomen (2 female, 10 male). 43 patients were male and 33 females with a mean age of 63.5 years.

MR images were acquired on the MRIdian at 0.35 T with the patients immobilized in treatment position in inspiration breath hold, following the planning image acquisition protocol. A TrueFISP sequence was used with an acquisition time of 17 s and FOV 45 × 45 × 24 cm³. In-plane resolution was 1.63 × 1.63 mm² with slice thickness of 3 mm, matrix size of 276 × 276 × 80, TE 1.62 ms, TR 3.83 ms and FA 60°.

CT images were acquired on a GE LightSpeed 580RT16 scanner on the same day as the MRI acquisition and with the same position setup, following the planning image acquisition protocol. CT slice thickness was 2.5 mm with in-plane resolution of 1.27 × 1.27 mm², 120 kVp, 920 ms exposure time, 300 mA tube current.

MR and CT images were registered using the MRIdian TPS deformable image registration tool. The image registration and anatomical correspondence between the CT-MRI pairs was validated by a radiation oncologist and a medical physicist, so that images in which strong differences in terms of air pockets existed were discarded from the database.

2.1.1. Simplified sCT

A DL-generated homogeneous density image can overcome the time needed for bulk sCT generation, while keeping a fast and simple approach to MR-only RT. CT images were thresholded into six gray-levels corresponding to the HU of: air (-1024), soft tissue (0), bone (200, 600) and lung (-800, −600). The threshold values were chosen based on the specific CT scanner density curves with the following intervals: Air < −910 HU, Lung2 −910 to −800 HU, Lung1 −800 to −600 HU, Water/Soft Tissue −600 to 150 HU, Bone1 150 to 400 HU and Bone2 > 400 HU. These images defined as Simplified CT, were then used as reference for the generation of Simplified synthetic-CT (SsCT) images.

MRI slices closer to the head and feet were removed to avoid distortion, the corresponding CT slices were discarded to match the volume data. The database consisted on 5486 pairs of MR-CT images from 76 patients. HU values and MRI intensities were rescaled to [0,1]. All the images were downsampled from 276 × 276 to 256 × 256 with bilinear interpolation. From these, 3966 pairs of images corresponding to 57 patients were used for training and validation and 1520 pairs (from the remaining 19 patients) for testing. The same data splitting was followed for the SsCT data.

2.2. Network architectures

Two main network architectures were trained to generate sCT and SsCT images: U-Net [6] and conditional-GAN (cGAN) [19]. These types of architectures have been chosen since they allow for the generation of images with the limited database available while still retaining structural features. Each network was used to generate sCT and SsCT images from MRI in an independent manner.

2.2.1. U-Net

The U-Net consists of an encoder and a decoder side with skip-connections between them [6]. The encoder is divided into eight blocks, each of which is formed by 2D Convolution, Batch Normalization and LeakyReLu layers. Each step of the encoder downsamples the image by a factor of two until achieving a latent space representation of 1 × 1 × 512. The decoder consists of 8 blocks with 2D Transposed Convolution, Batch Normalization, Dropout and ReLu layers. Each step of the decoder upsamples the image by a factor of two and concatenates the high resolution features from the encoder with the decoder layer. The last layer performs a 2D Transposed Convolution with a Sigmoid activation function reaching the initial image size 256 × 256 × 1. Fig. 1 shows a diagram of the U-Net with the channels/filters used on each step. The L1 loss function was used during the optimization of the network destined to the generation of sCT. While a sparse categorical cross entropy loss function was used in the case of SsCT.

Fig. 1.

U-Net and cGAN network architectures.

2.2.2. conditional-GAN

The cGAN trained was based on the pix2pix network proposed by Isola [21], which performs Image-to-Image translation with a conditional input. In our case this means generating sCT images from MR images using the registered CT as conditional input during training. This architecture is characterized by the use of a generator and a discriminator network that are trained in competition between them.

Fig. 1 shows a diagram of the cGAN architecture. The generator G is by itself a U-Net network with the same characteristics as the one used in this study. The discriminator D is a PatchGAN classifier that evaluates whether the generated image is real or not by looking at patches of 70x70 pixels. The generator part of the cGAN network trains with a loss function that is dependent on the structure of the target image. This loss function takes the form:

$$L=L_{\mathit{GAN}}+\lambda L_1$$

where $L_{\mathit{GAN}}$ is the cross-entropy adversarial loss, $\lambda$ is a regularization parameter set to 100 and $L_1$ is the Mean Absolute Error between the target image and the generated image [21].

The networks were trained for 100 epochs with batch size of 10 with Adam optimizer and learning rate of 0.0002. K-fold cross validation was used with k = 5. They were implemented using Python 3.7.6, Keras 2.2.5 and Tensorflow 1.14.0 on a double Intel Xeon Silver 4114 2.2 GHz processor with 128 Gb RAM and two GPUs Nvidia Quadro P5000 16 Gb.

2.3. Evaluation

sCT and SsCT images were generated from the test dataset MR images using the trained U-Net and the generator network of the cGAN. The generated images were encoded as DICOMs and imported into the MRIdian Treatment Planning System (TPS). sCT and SsCT images were compared in terms of Image and Dose metrics with the reference CT images.

2.3.1. Image metrics

The Mean Error (ME) and Mean Absolute Error (MAE) in Hounsfield Units (HU) were used as 2D image evaluation metrics. ME and MAE were calculated between the reference CT and the generated sCT images. Following:

$$\mathit{ME}=\frac{{\sum }_{i=0}^M{\sum }_{j=0}^N\left(CT\left(i,j\right)-sCT\left(i,j\right)\right)}{M\ast N}$$

$$\mathit{MAE}=\frac{{\sum }_{i=0}^M{\sum }_{j=0}^N\left|CT\left(i,j\right)-sCT\left(i,j\right)\right|}{M\ast N}$$

where $M$ and $N$ are the image dimensions. For the simplified case, SsCT generated images were compared with the reference Simplified CT using the same criteria.

These two metrics were calculated only for the HU values found within the body contours in a patient-wise manner to avoid the impact of the image background on the comparison. These are reported as ME_BODY and MAE_BODY and corresponded exclusively to the pixels within the ROI of the body. Additionally, Bland-Altman plots were used for comparison with respect to the reference CT.

2.3.2. Dose

Dose comparison was done between RT treatment plans calculated using the MRIdian TPS on the original CT image and on generated sCT and SsCT images. The Monte-Carlo MRIdian TPS computation of dose distributions takes into account the presence of the 0.35 T magnetic field on a dose grid of 2 mm³. Target volumes and organs at risk were contoured by the radiation oncologist on the MRI. The same contours were used for the sCT and SsCT dose plans, except for the external body contour which was redefined for the generated images. Gamma index analysis and Dose Volume Histogram (DVH) parameters were used to compare dose distributions.

Global gamma index analysis was performed with SNC Patient (Sun Nuclear Corporation, Melbourne, FL, USA) comparing whole dose distributions at: 2%/2mm and 3%/3mm with a dose threshold of 10%. The gamma passing rate (GPR) was averaged between the test patients for each network and each type of image (i.e. sCT and SsCT).

For the DVH analysis, D98%, D50% and D2% parameters were obtained for the Plannint Target Volume (PTV), Liver and Stomach on each RT plan. The average dose difference (%) of each generated plan with respect to the original plan was compared. Additionally, a Two One-Sided Test (TOST) for Equivalence was performed to evaluate the statistical significance of the dose differences found. Matlab (The MathWorks Inc, Natick, MA, USA) was used for the TOST test with a significance level of 0.05 and equivalence interval of (-1%,1%).

3. Results

sCT and SsCT images were generated using the U-Net and GAN architectures for the 19 patients used as testing cohort. Each patient sCT volume, consisting of 80 2D-images was generated in an average of 2 s with the U-Net architecture and 2.5 s with GAN.

Table 1 shows the average ME_BODY and MAE_BODY for the test dataset which are obtained taking only into account the values within the body contour of the images. The U-Net generated sCT images achieved a ME_BODY of −7.4 HU and a MAE_BODY of 28 HU. The GAN based sCT images showed a lower MAE_BODY with 25.9 HU and a higher ME_BODY with −11.4 HU.

Table 1.

ME and MAE calculated between CT vs sCT and Simplified CT vs SsCT taking into account only the body contour.

	MEbody (HU)	MAEBody (HU)
CT vs U-Net sCT	−7.4 ± 21.8	28 ± 14.7
CT vs GAN sCT	−11.4 ± 17.4	25.9 ± 13.4
Simplified CT vs U-Net SsCT	−2 ± 12.7	11.6 ± 16.3
Simplified CT vs GAN SsCT	−42.9 ± 37.3	52.9 ± 31.6

Regarding the 6-density case, the U-Net generating SsCT images showed the best performance with −2 HU and 11.6 HU for ME_BODY and MAE_BODY respectively compared against the Simplified CT. GAN SsCT produced the worse results for ME_BODY and MAE_BODY with −42.9 HU and 52.9 HU.

Fig. 2 shows the Bland-Altman plots comparing the reference CT and the generated sCT and SsCT images for the test dataset. The mean differences for the sCT images generated with both U-Net and GAN were of −9.7 HU and −12.4 HU. The GAN SsCT method showed a mean difference of −53 HU.

Fig. 2.

Bland-Altman plots comparing CT and generated sCT and SsCT images.

Fig. 3 shows the axial view of a patient’s MRI, CT, U-Net generated sCT and GAN generated sCT as well as the difference between the CT and each sCT. Additionally, the Simplified CT and each generated SsCT (U-Net and GAN) are presented with the respective image difference (Simplified CT-SsCT). The differences are mainly located on the interfaces between different tissues where the density changes.

Fig. 3.

MRI, CT, simplified CT, generated sCT and SsCT and Difference.

In Table 2, we report the average gamma passing rate for the RT plans calculated using the generated images. The plans obtained with sCT images for both architectures show similar results with pass rates of 91% and 95% for 2%/2mm and 3%/3mm respectively. GAN SsCT plans showed the worse performance with only 82.6% for 2%/2mm and 92.7% for 3%/3mm.

Table 2.

Average gamma passing rate (%). Mean ± SD.

	2%/2mm	3%/3mm
U-Net sCT	91.4 ± 6	95.3 ± 5.1
GAN sCT	91.6 ± 6	95.4 ± 4.9
U-Net SsCT	89.6 ± 6.9	94.1 ± 5.4
GAN SsCT	82.6 ± 11.4	92.7 ± 5.6
Simplified CT	90.8 ± 7	94.6 ± 5.4

The average difference (%) of DVH parameters with respect to the original CT plan for the PTV, Liver and Stomach is shown on Table 3. Additionally, p-values corresponding to the paired sample t-test are shown. The U-Net sCT method shows the least difference with respect to the original plan with only 0.8% for the PTV-D98%, 0.8% for the PTV-D50% and −0.08% for PTV-D2%. The GAN sCT follows closely with 1.3%, 1.6% and 1.3% differences for D98%, D50% and D2% on the PTV. The U-Net SsCT and GAN SsCT show stronger differences up to 2.3% and 4.7% for the PTV-D50% respectively. The same trend is shown on the Liver and Stomach, with the U-Net sCT outperforming the other methods.

Table 3.

Mean difference (%) of PTV, Liver and Stomach DVH parameters for the generated image RT plans with respect to the reference plan. Mean ± SD.

		U-Net sCT		GAN sCT		U-Net SsCT		GAN SsCT
			P-values		P-values		P-values		P-values
PTV	D98	0.8 ± 1.12	<0.001	1.3 ± 0.77	0.007	1.9 ± 1.21	0.021	4.1 ± 3.27	0.109
	D50	0.8 ± 1.41	0.010	1.6 ± 0.82	0.055	2.3 ± 1.29	0.102	4.7 ± 3.59	0.172
	D2	−0.08 ± 1.68	0.035	1.3 ± 1.15	0.119	2.3 ± 1.32	0.199	4.7 ± 3.59	0.154
LIVER	D98	−0.02 ± 0.11	<0.001	−0.03 ± 0.10	<0.001	−0.05 ± 0.09	<0.001	−0.03 ± 0.10	<0.001
	D50	0.2 ± 0.61	0.211	0.2 ± 0.62	0.208	0.2 ± 0.64	0.225	0.3 ± 0.80	0.256
	D2	0.7 ± 1.35	0.401	1.1 ± 0.89	0.335	1.4 ± 1.03	0.399	3 ± 3.02	0.334
STOMACH	D98	−0.1 ± 0.28	<0.001	−0.1 ± 0.28	<0.001	−0.1 ± 0.28	<0.001	−0.1 ± 0.25	<0.001
	D50	0.4 ± 0.95	0.318	0.5 ± 0.97	0.327	0.5 ± 0.97	0.334	0.9 ± 1.55	0.338
	D2	−0.02 ± 0.11	<0.001	0.7 ± 1.23	0.347	1.1 ± 1.45	0.386	2.4 ± 2.96	0.427

4. Discussion

In this study we have generated sCT images from low-field MR images using two different DL-architectures: U-Net and GAN. Additionally, we proposed the generation of images with only six electronic densities denoted as Simplified sCT. We have focused on the abdominal area, where few studies have been published in comparison to pelvic or brain imaging. We evaluated the performance of these methods with image evaluation metrics and dose plan comparisons.

Although the network training required 25hrs for the GAN architecture and 5hrs for the U-Net, the generation of 80 images for a single patient took only 2.5 s and 2 s respectively. This establishes the utility of our method in the current workflow of ART, where the patient remains on table during the daily plan re-optimization.

In terms of image metrics, the results obtained are in accordance with the studies done on abdominal sCT image generation with low-field MR images [19], [20], [25]. Cusumano [19] which used a cGAN architecture reports ME of 10.8 HU and MAE of 78.7 HU. Fu [20] using a cGAN and cycleGAN architectures shows an MAE of 89.9 HU and 94.1 HU respectively. When comparing our proposed SsCT we see that an important difference is seen, specifically with the U-Net architecture with ME and MAE values of −2.0 HU and 11.6 HU within the body contours. This is also shown with the Bland-Altman plots, were strong differences and bias are seen on SsCT images, compared to the whole range images.

Regarding the dose accuracy, we report mean gamma pass rates of 95% for sCT and 92% for SsCT images on 3%/3mm. The reported values in [19], [20] show gamma pass rates of 99% for 3%/3mm, however these values were calculated within specific dose volumes. This could explain the difference with our results which were obtained on the whole dose distribution volume. Nevertheless, our results are in line with those reported on [6], [23].

The statistical DVH analysis showed that with the U-Net sCT method we could achieve a difference of < 1% on the PTV, Liver and Stomach. GAN sCT followed with values of < 1.6% on the PTV and < 1.1% on the Liver. The SsCT based methods showed differences up to 2% and 4% for the U-Net and GAN with no statistical equivalence to the original dose distribution. This can be caused by the effect of the poor image reconstruction shown on the edges of organs and close to tissue interfaces, where a strong change of density occurs.

Although the use of bulk homogenous density images for RT plans is not new [24], we have explored their use in combination with a DL approach which has not been done before. The generated SsCT images composed of only six density levels, showed results within 2% of the dose of whole range image plans. Therefore, these types of images do not represent an advantage over the generation of whole range sCT images.

This study showed that sCT images can be generated by using low field abdominal MR images with U-Net and GAN architectures. The generated images are comparable to other studies in terms of image quality metrics and dose statistics. These methods may be suitable for integration into the ART workflow in terms of image generation time. However, it remains to confirm the accuracy on a larger patient cohort with the inclusion of cases in which air pockets in the abdomen may be assessed to ensure a generalizable approach with dosimetric accuracy. Additionally, the use of images containing data augmentation during training has to be explored as this could also have an impact on the performance of the generation of images.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.