Synthetic CT Generation from 0.35T MR Images for MR-only Radiation Therapy Planning Using Perceptual Loss Models

Abstract

Aim:

Magnetic resonance imaging (MRI) provides excellent soft tissue contrast which makes it useful for delineating tumor and normal structures in radiotherapy planning, but MRI cannot readily provide electron density for dose calculation. CT is used but introduces registration uncertainty between MRI and CT. Previous studies have demonstrated synthetic CTs (sCTs) generated directly from MRI images with deep learning methods. However, mainly high-field MRI images have been validated. This study is to test whether acceptable sCTs for MR-only radiation therapy planning can be synthesized using an integrated MR-Linac at 0.35T, using MRI images and treatment plans in the liver region.

Materials and methods:

Two models were investigated in this study, a Unet with conventional mean square error (MSE) loss and a Unet utilizing a secondary VGG16 network for perceptual loss. 37 cases were utilized in this study with ten-fold cross validation. 37 treatment plans were generated and evaluated for target coverage and dose to organs at risk (OARs) on the MSE loss model, perceptual loss model, and original CT.

Results:

The sCTs predicted by the perceptual loss model had improved subjective visual quality compared to the MSE loss model, but both were similar in MAE, PSNR, or NCC. The MAE, PSNR, and NCC for perceptual loss model were 35.64, 24.11, and 0.9539 while those for MSE loss model were 35.67, 24.36, and 0.9566. No significant differences in target coverage and dose to OARs were found between the sCT predicted by perceptual loss model or by MSE model and the original CT.

Conclusion:

This study indicates that a Unet with both MSE loss and perceptual loss models can be used for generating sCT images from a 0.35T integrated MR-Linac.

1. Introduction

Magnetic resonance imaging (MRI) has been increasingly incorporated in the planning and delivery of radiation treatment (RT) due to its excellent soft tissue contrast which aids in contouring organs at risk (OARs) and tumors[1]. Additionally, recent advances have resulted in integrated magnetic resonance image guided radiation therapy (MRgRT) devices that combine MRI scanners with 60Co or linear accelerators[2], which make it much easier for MRI images to be directly applied to therapy planning. The ViewRay MRIdian Linac (ViewRay Inc, Oakwood Village, OH) and Elekta Unity (Elekta AB, Stockholm, Sweden) are two newly developed and commercially-available MR linear accelerator machines currently in use. The MRIdian system integrates a low-field 0.35T MRI scanner and the Unity integrates a 1.5 T MRI scanner.

Although MRI images have been widely used in clinical application of radiotherapy planning, one limitation for MRI images is that it cannot provide electron density information required for dose calculation. A common solution is to obtain kilovoltage computed tomography (kVCT) images and spatially co-register the MRI images to the planning CT, but this introduces two problems at the same time, extra ionizing radiation dose for patients and registration uncertainties[3]. Thus, an active area of research is to obtain synthetic CT (sCT) images directly from MRI images. Several types of sCT approaches have been proposed and can be divided into three categories: bulk density assignments, atlas-based techniques, and voxel-based techniques[4]. Bulk density assignments try to separate tissues from MRI images into several classes, like bones, air, soft tissue, and fat, and then assign each class an electron density or Hounsfield unit (HU) value[4]. The bulk density assignment approach is very simple, but it is not very accurate[5]. In contrast, atlas-based methods and voxel-based methods are more clinically useful[4]. Atlas-based techniques try to obtain the CT number based on its corresponding MRI voxel position through registration of an MRI image to an atlas space, and resulting assignment of CT number based on pixel location[6]. This method has been applied to obtain acceptable sCT images for the prostate[7]. Voxel-based approaches primarily compute the electron densities of CT images directly from the intensities of MRI images[6].

Recently, a number of studies have utilized deep learning and convolutional neural networks (CNNs) to create sCT images [8]–[10]. Among the many kinds of CNNs, the Unet[11] has demonstrated outstanding performance in medical image segmentation and synthesis. More recently, generative adversarial networks (GANs)[12] have become popular in creating realistic synthetic images. sCT images of pelvic, liver, brain, head and neck regions have been produced by GANs and its variants [13]–[17]. However, GANs are known to be unstable and can be difficult to train. In this study we implement a Unet with a perceptual loss model, which enjoys GAN-like capability in synthesizing realistic outputs, but utilizes a fixed discriminator and is generally considered to be more stable [18].

Prior studies have mainly focused on converting high-field (diagnostic) MRI images to sCT images. The purpose of this study is to get sCT images directly from low-field, 0.35 T, MRI images in the liver region using Unet with perceptual loss for improved output image quality, and to verify whether the sCT images are accurate enough for MR-only radiation therapy planning.

2. Materials and methods

2.1. Dataset description

For this study, 37 liver cases were obtained using a retrospective IRB protocol for patients undergoing planning and therapy on ViewRay 0.35 T MR-Linac. The clinical scans were acquired with a steady-state free precession sequence (TrueFISP) on a 0.35T MRIdian system. Each scan had a slice thickness of 3 mm, 1.5 ×1.5 mm² in-plane resolution, and field of view (FOV) 40×43×40 acquired in 25 seconds. Patients were positioned headfirst supine with right arm up, left arm down, laying on a half mattress with a large knee cousin. All scans were acquired in maximum inhale breath hold position (MIBH). A CT scan with patient setup similar to MRI was acquired on Siemens SOMATOM Definition Edge (Siemens Healthineers, Malvern, PA).

Before training, several preprocessing steps were applied to the MRI and CT images. Due to differences in the data size first, MRI images (256×256) were up-sampled to match the resolution of CT data (512×512). Next, rigid registration for each images set was performed in MIM (version 6.6.11, MIM Software Inc., Cleveland, OH). Slices outside of the liver region were not considered in this study. As a result, there were 2561 slices from the 37 patients in this study. Additionally, any signal outside the patient’s body cavity in the MR images was eliminated manually. To compensate for non-uniform intensity in MR images, each MR input slice was multiplied with scalar intensity maps as a method of data augmentation to help the network learn to ignore non-uniform image intensity. Equation 1 shows how the scalar intensity maps were randomly generated[19].

$$z_{x,y}=\frac{\sin \left(w_{1}x+{\phi }_1\right)+\sin \left(w_{2}y+{\phi }_2\right)+8}{8}$$ (1)

Where x and y are the coordinates for a single image, ranging from 1 to 512 according to the image size 512×512. w₁ and w₂ ranges from 0 to 3e⁻² while φ₁ and φ₂ ranges from 0 to 1. Then the range of z_(x,y) could be from 0.75 to 1.25. For each slice, since w₁, w₂, φ₁, and φ₂ were generated randomly and separately, the scalar intensity maps were different for different MR slice images. Next, normalization was applied to both MRI and CT images, using equation 2. For the CT images, Hounsfield Unit (HU) values were constrained between −1000 to 2000 before normalization by changing the values larger than 2000 to be 2000 and those smaller than −1000 to be −1000, which means x_min and x_max are −1000 and 2000 respectively for CT images. As a result, the input data ranged from 0 to 1 after normalization but before training. Finally, data augmentation methods, such as rotation, flip, and shift, were utilized to increase the robustness of the network in real-time during model training.

$$x_{new}=\frac{x-x_{min}}{x_{max}-x_{min}}$$ (2)

2.2. Deep learning workflow

A four-layer Unet with 64, 128, 256, and 512 filters at each layer was used as the main model for converting MRI images to CT images in this study. The structure of a Unet is symmetric and composed of two stages, encoding stage and decoding stage, (Fig. 1A). The encoding stage included regular convolution layers and max pooling layers. In this stage, the MRI images size decreased while the depth increased to extract effective feature maps. In contrast, the decoding stage was designed for precise localizing and generating high-resolution CT images. In the decoding stage, the feature map size increased and its depth decreased. Furthermore, the feature maps from the encoder stage were concatenated with the same level output of the decoding stage, providing more information and recovering CT images in a more precise way.

Fig. 1.

(A) A four-layer Unet structure with start filter 64 including encoding and decoding stage; (B) The VGG16 model structure used for discriminating MRI and CT in the first round of ten cases while sCT and real CT in other rounds; (C) The general pipeline shows how data was prepared and used for training. Processed MRI images were the input of Unet. The output of Unet and prepared CT were used to compute perceptual loss with VGG16 to improve Unet model.

VGG16[20] also played an important role in the pipeline, (Fig. 1B). With its high accuracy in performing image classification tasks, VGG16, with initial weights learned from ImageNET, was used for measuring the difference between real CT images and the sCT images generated by the Unet. Initially, the VGG16 network was trained to discriminate MRI and CT images. However, during ongoing training, it was retrained to discriminate between CT and sCT images so that the differences between CT and sCT images could be measured more accurately.

The general pipeline used in this study is represented in Fig. 1C. MRI images were the input into the Unet. During the training process, the output of Unet was also the input to VGG16 along with the co-registered CT images. Perceptual loss[18] was computed, which improved Unet performance. As a test of a baseline model for comparison, a Unet model with conventional mean square error (MSE) loss was also constructed.

2.3. Training details

2.3.1. Perceptual loss model

In the field of computer vision, converting an MRI into a CT could be seen as a task of “style transfer,” where the subject matter of the image (i.e., the abdomen) remains the same, but the visual appearance changes (i.e., MRI vs. CT modality). In this context, we utilized a perceptual loss approach which has been previously studied for style transfer applications. Perceptual loss consists of feature reconstruction loss and style reconstruction loss which measure high-level perceptual and semantic differences between images[21], [22]. Feature reconstruction loss focuses on the image content and the overall spatial structure while style reconstruction loss looks into texture, color, common patterns. To compute this loss, a VGG16 loss network as depicted in Fig. 2A was used. The input of the loss network was composed of three kinds of images: the generated, the style target, and the content target. The generated image was the output image synthesized by the Unet model. The style target aimed at providing the style information which refers to CT modality. Similarly, the content target offered some content information, for example, indicating the abdominal region instead of another body region such as the head or pelvis. Since the sCT image is expected to be similar to the real CT image no matter in the aspect of content or style, the content target and the style target were both set as the real CT images here. The content representation was taken from the layer relu3_3 and the style representations are taken from the following layers: relu1_2, relu2_2, relu3_3, and relu4_3. As mention earlier, the Unet model learned to convert MRI images into CT images with the help of perceptual loss computed by the fixed VGG16 network. To prove the perceptual loss was effective and efficient, MSE loss was also used to train the same Unet for comparison.

Fig. 2.

(A) The explanation of perceptual loss computation. Three kinds of images were involved, including sCT, style target and content target. sCT was generated by Unet. Style target and content target was the real CT. By comparing the outputs of VGG16 in different layers for the sCT and real CT, perceptual loss can be got; (B) The training process of perceptual loss model. There are three phases in total and two rounds are included in each phase. The VGG model in the first round of phase was trained to discriminate MRI and CT images while that in other rounds was trained to discriminate sCT and CT images.

2.3.2. Model implementation

The model was implemented with the Keras package (https://keras.io). RAdam was used as the optimization method to minimize the loss function described above [23]. The parameters of RAdam were set as the default, except the learning rate, which was changed to 0.0001. The split ratio between training and validation dataset was set as 0.3, which means thirty percent of the images were chosen for validation and the rest for training. At each iteration, a minibatch of 4 slices were randomly selected from the training set due to limitations in GPU memory. An NVIDIA Titan RTX GPU (NVIDIA Corp., Santa Clara, CA) was utilized for all deep learning training.

2.3.4. Training the perceptual loss model

According to preliminary experiments conducted with the same code and data, the training and validation loss was higher when the model was trained with 34 cases at one time, compared to the loss when the model was trained by adding cases step by step. Therefore, data was fed in three phases and each phase involved two rounds of approximately ten more cases, just as Fig. 2B shows.

In phase one, 10 cases were used. In the first round of this phase, the VGG16 was trained with MRI and co-registered CT images. Once the classification accuracy for VGG16 reached to 99% and became steady, the model was considered converged, and training was stopped. Then this well-trained VGG16 was used to compute the perceptual loss during the training of Unet 1_1. The same MRI and CT images were used for training the Unet 1_1 as well. Once the training loss and validation loss was observed to fluctuate within a small range, like the variation accounting for no more than two percentage of the loss value for this project, the model was considered converged. Then in the second round of the first phase, the well-trained Unet 1_1 was used for predictions and the sCT images were saved. A new VGG16 was trained to tell the difference between real CT images and sCT images. Similar to the previous round, the VGG16 model was involved in the training of Unet 1_2 which was fine-tuned based on the previous Unet 1_1. In phase two, additional ten cases were added so there were 20 cases in total. The process of training the subsequent 20 cases was analogous to the 2^nd round of phase one. In phase three, 33 or 34 cases were used for the training, validation, and making predictions while the remaining 4 or 3 cases were used for testing. The training of phase three is very similar to that of phase two, but the difference lies in that there is one more fine-tuning process with the combined loss function, including previous perceptual loss and mean square error (MSE).

2.4. Evaluation of model and sCT

Ten-fold-cross-validation was performed to evaluate model performance for both the MSE loss and perceptual loss models. The 37 patients were randomly divided into ten groups, seven groups of four patients and three groups of three cases. For each cross-validation fold, nine groups were used for training and validation, and one group was left for testing. Finally, the synthesized sCT images were generated case by case and compared with the original CT images. Three measures were utilized to evaluate the difference and similarity between CT and sCT images: mean absolute error (MAE), peak-signal-to-noise-ratio (PSNR), and normalized cross-correlation (NCC). The formulas are shown as follows.

$$MAE=\frac{1}{N}\sum _{i=1}^N\left|sC{T}_i-C{T}_i\right|$$ (3)

$$PSNR=20lo{g}_{10}\frac{\max \left(CT\right)}{\sqrt{\frac{1}{N}{{\sum }_{i=1}^N\left(sC{T}_i-C{T}_i\right)}^2}}$$ (4)

$$NCC=\frac{{\sum }_{i=1}^N(C{T}_i-\frac{1}{N}{\sum }_{j=1}^{N}C{T}_i)(C{T}_i-\frac{1}{N}{\sum }_{j=1}^{N}C{T}_j)}{\sqrt{({\sum }_{i=1}^N{(C{T}_i-\frac{1}{N}{\sum }_{j=1}^{N}C{T}_j)}^2)({\sum }_{i=1}^N{(sC{T}_i-\frac{1}{N}{\sum }_{j=1}^{N}sC{T}_j)}^2)}}$$ (5)

Where N is the total number of pixels within one test case and max(CT) stands for the largest HU value within one case. Generally, high prediction accuracy accompanies lower MAE, higher PSNR, and higher NCC.

2.5. Treatment planning and Evaluation

The feasibility of utilizing the proposed sCT images for treatment planning was performed for each of the 37 patients in the 10 different models with 3 or 4 patient plans evaluated per model. The perceptual loss model sCT, the MSE loss sCT, and the original kVCT scans were exported to MIM for segmenting target volume or OARs. The clinical target volume was exported from the kVCT planning scan to the perceptual loss and MSE loss sCT images. OARs included: spinal cord, bowel loops (small and large), stomach, and chest wall. The planning objectives used for planning are listed in Table 1. All plans were prescribed to 50Gy in 5 fractions and generated with 6 MV photon beam energy with 3 to 4 arcs using the Volumetric Modulated Arc Therapy (VMAT) planning technique in RayStation (RaySearch Laboratories AB, Stockholm, Sweden) using the collapsed cone version 5 dose calculation algorithm. A 0.2×0.×0.2 cm/voxel grid was used for dose calculation. All plans were normalized such that 95% of the PTV receives the prescribed dose. Mean dose, maximum dose and the Wilcoxon signed rank test was used to compare clinically relevant dosimetric data for perceptual loss, MSE loss and original kVCT plans for PTV and OARs.

Table 1.

Treatment Planning Objectives

Structure	Dose Objectives
Target
PTV	V50Gy≥95%
PTV Conformality Index	0.95–1.2
OARs
Small and Large Bowel	V30Gy ≤0.5cc
Spina Cord	V25Gy ≤0.5cc
Stomach	V30Gy ≤0.5cc
Chest wall	Dmax≤<=44Gy
Skin	Dmax(1cc)≤40Gy

3. Results

It took about 113 hours to train one individual cross-validation fold for the Unet with the perceptual loss model and 37.5 hours to train the Unet with the traditional MSE loss model. Overall, the sCT images generated by the perceptual loss model were more visually similar to kVCT planning images compared to these generated by MSE model. An example of 0.35 T MRI, real CT and the sCT images generated by Unet with perceptual loss model and Unet with MSE loss model respectively for one example is shown in Fig. 3. Additionally, although the perceptual loss model performed well in dose estimation for the liver, air, stomach, and kidney, it was not successful in generating all fine structures, such as the ribs. The absolute difference maps of both the two models demonstrate that the biggest difference between the sCT and real CT images was the body contour of patients, outside the region of interest, this might be because MRI and CT images were obtained 15 to 20 mins apart and the patients moved between scanners.

Fig. 3.

Example sCT images from a 65-year-old male patient. Multiple slices from (a) 0.35 T MRI images, (b) the corresponding real CT images, (c) the sCT images generated by the perceptual loss model, (d) the sCT images generated by the MSE loss model, (e) the absolute difference map for perceptual loss model, and (f) the absolute difference map for MSE loss model.

The evaluation of sCT images from ten-fold cross validation, 37 patients in total, has been computed. For the perceptual loss model, the mean (standard deviation) of MAE, PSNR, and NCC are 35.64 (12.66) HU, 24.11 (2.47) dB, and 0.9539 (0.0227); for the MSE loss model, those are 35.67 (12.70) HU, 24.36 (2.45) dB, and 0.9566 (0.0215). In general, the perceptual loss model and traditional MSE model had very similar results.

All plans were evaluated in MIM for PTV coverage and treatment planning object for OARs, using a dose volume histogram (DVH) and isodose distribution in 3 planes. PTV volume ranged from 31cc-273 cc. All plans using the sCT images met the planning objective and maximum dose were less than 110%. An example isodose distribution in transverse, sagittal and coronal plane is shown in Fig. 4. The DVH for the same case is shown in Fig. 5. Overall dose distribution was similar for perceptual and MSE for all model (p=0.02). Overall average difference in PTV and OARs in perceptual and MSE model were clinically insignificant (<0.03%). It was noted that the low dose distribution was higher in the original scan compared to sCT plans and this may be due to the deviation of sCT from the original CT which was clinically insignificant.

Fig. 4.

Comparing dose distribution in planning CT, Perceptual loss sCT, MSE loss sCT, and MRI images in sagittal, coronal and transverse planes. The red, yellow, orange, green, and blue areas represent the 50 Gy, 45 Gy, 40 Gy, 30 Gy, and 20 Gy isodose lines, respectively.

Fig. 5.

Dose volume histogram for PTV, Rt kidney and AAA. Original CT is presented with the solid line, thin and thick dashed lines are for Perceptual loss and MSE loss derived plan. The Perceptual loss and MSE loss are nearly identical, the DVH for the PTV and OARs are overlapping.

4. Discussion

In this study, Unet models utilizing MSE loss and perceptual loss was trained to convert low field 0.35T MR images from an MR-Linac to CT images. From the treatment planning results, no significant differences were found between the original CT and the sCT images regardless of whether they were generated by the perceptual loss model or the MSE model, suggesting that the sCT images predicted by both models from low field MR images are reasonable. The evaluation of sCT images shows that the sCT images from perceptual loss model and MSE model were very similar to each other in the aspect of MAE, PSNR, and NCC. However, two loss functions can be considered to have their own advantages. The sCT images from perceptual loss model had better subjective image quality and organs could be better observed, but took much longer to train. The time necessary for training the MSE loss model was only one third of that for training perceptual loss model.

For the MSE loss model, a small MAE was achieved but yielded blurry sCT images. This is expected for an MSE loss approach because the model tends to predict the average CT value to make the overall loss as small as possible, hence images tend to appear blurred [24]. In contrast, perceptual loss models have shown powerful capability in applications such as style transfer and image super-resolution [18]. Therefore, a model trained with a perceptual loss function would be expected to generate sCTs with better subjective image quality. However, perceptual loss focuses on the image’s content and style information rather than just the intensity of a given pixel value, so the magnitude of the differences between the perceptual loss sCT and the original CT can be larger than those between the MSE loss sCT and the original CT. To take advantage of both the MSE loss function and the perceptual loss function, finetuning with a combined loss function, which included both MSE loss and perceptual loss improved the quantitative accuracy of the generated sCTs. Even though the perceptual loss sCT appears to have better visual quality upon inspection, it appears that the MSE loss model sCT performs adequately well for treatment planning.

Most prior work developing sCT images in the abdominal region have focused on high-field MRI images (1.5T or 3T). For deep learning sCT approaches a Cycle GAN with an integrated dense block was trained with 1.5T and 3T MRI scans in the liver region yielded acceptable sCT images (MAE: 72.87 HU; PSNR: 22.65 dB; NCC: 0.92) [25]. Other recent studies along with our study have demonstrated that low-field, 0.35T, MRI images can also be utilized in the abdomen and pelvis [26]–[28]. In these studies, conditional GANs and Cycle GANs were utilized. The MAE (PSNR) for the sCT from the conditional GAN was 89.8 HU (27.4 dB) while that from the Cycle GAN was 94.1 HU (27.2 dB) [28]. Compared with the studies with GAN and its variants, our approach shown the possibility of obtaining sCT images with improved performance (MAE: 35.64 HU; PSNR: 24.11 dB; NCC: 0.9539) without utilizing a GAN approach. Additionally, the sCT predicted from the simpler Unet trained using a MSE loss function has similar performance (MAE: 35.67 HU; PSNR: 24.36 dB; NCC: 0.9566). Both sCT approaches provided sufficiently good plan quality.

This study has several limitations. First, only 33 or 34 cases were used for training each model, which means the dataset is a relatively small to train a deep learning model, yet comparable in size to other studies. Note that we utilized cross validation to mitigate this limitation. Second, the batch size, an important training parameter that also affects learning rate, was set as four due to GPU memory limitations. If multiple GPUs are available, the model training performance could be improved further. Third, the currently trained models utilized a 2D input. Future work should utilize 3D inputs for improved 3D context in medical images. Finally, MRI and CT were acquired on separate scanners up to 30 minutes apart. Spatial registration was required which may be challenging, though it did not appear to be a major limitation in this study. Note that an advantage of the Cycle GAN approach is that it can utilize unpaired data, and thus does not require image registration as a pre-processing step. It remains to be determined in future work if this improves the capability of resolving small features such as the ribs.

In this study, we have shown that 0.35T MRI images could be used for generating clinically acceptable sCT images for MR-only radiotherapy in the liver utilizing a simple and stable Unet model. In the aspect of dose calculation, there was no difference between the sCT images generated by a perceptual loss model and a MSE loss model. In future work, more cases and multi-GPUs will be involved in the training and networks will be revised to improve the image quality and potentially obtain more spatial fine detail.

5. Conclusion

We present a straightforward Unet model that can generate sCT images from 0.35 T MRI images for liver with state-of-the-art performance. Our study also demonstrates that low-field MRI images can be used for training a deep learning model and making precise predictions without needing high field MRI images. Furthermore, quantitative and clinical evaluation support that the generated sCT images are clinically acceptable for dose calculation, reducing the need for a potentially unnecessary CT, and simplifying the whole workflow for MR only RT planning, which indicates the promising future for MR-only radiotherapy.

[Data Availability Statement for this Work]

Research data are not available but source code and well-trained models are available at https://github.com/mimrtl/MRI-to-CT-Generation/tree/master.