Noise Controlled CT Super-Resolution with Conditional Diffusion Model

Abstract

Improving the spatial resolution of CT images is a meaningful yet challenging task, often accompanied by the issue of noise amplification. This article introduces an innovative framework for noise-controlled CT super-resolution utilizing the conditional diffusion model. The model is trained on hybrid datasets, combining noise-matched simulation data with segmented details from real data. Experimental results with real CT images validate the effectiveness of our proposed framework, showing its potential for practical applications in CT imaging.

I. Introduction

Medical imaging, especially Computed Tomography (CT), is crucial for diagnosing and treating health conditions. Achieving higher resolution in CT scans is an ongoing challenge, and super-resolution techniques play a vital role in enhancing spatial resolution. This improvement promises more detailed information for clinicians, leading to better diagnostic accuracy and improved patient care. Diffusion models [1], [2], rooted in probabilistic modeling and diffusion processes, offer significant advancements in natural imaging tasks, including super-resolution [3]–[5]. Their stability, unlike Generative Adversarial Networks (GANs), ensures reliable image generation without the pitfalls of mode collapse or unrealistic artifacts. In the context of CT super-resolution, diffusion models stand as a source of inspiration, paving the way for novel developments in medical imaging.

Achieving super-resolution in CT images presents a complex challenge in noise control, particularly when compared to natural images. The training of conditional diffusion models for super-resolution necessitates paired sets of high resolution (HR) and low resolution (LR) CT images. Achieving super-resolution without noise amplification requires meticulous matching of noise levels in HR and LR CT images. The scarcity of noise-matched pairs, prompted by radiation exposure, has led certain methodologies to resort to downsampling HR CT images in either the image domain or projection domain to obtain corresponding LR CT images for model training [6]. However, these approaches inadvertently exacer-bate noise while improving spatial resolution. Some techniques introduce Gaussian noise to LR CT images [7] before or after downsampling to harmonize with the noise level, yet their efficacy in super-resolution may waver when applied to real data due to the introduction of unrealistically distributed noise.

In this article, we present an innovative framework for noise-controlled CT super-resolution, utilizing a conditional diffusion model trained on hybrid datasets. Numerical phantoms are employed to generate noise-matched simulation pairs of HR and LR CT images. Furthermore, details absent in numerical phantoms are segmented from noise-unmatched real pairs and integrated into the training process. Testing using real CT images validates the effectiveness of the proposed framework in real-world scenarios.

II. Method

A. Conditional Diffusion Model

In our article, we employ the Conditional Denoising Diffusion Probabilistic Model (DDPM) [4], [5] for super-resolution in CT images. Here, the set of LR images serves as the condition $y$, and the set of HR images forms the generation target $x_0$. The Conditional DDPM contains a forward and reverse processes. The forward process, a Markov chain, gradually introduces noise into the image until it becomes a standard Gaussian noise image, with the following process:

$$q\left(x_t\mid x_{t-1}\right)=N\left(x_t\mid \sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_{t}I\right),$$ (1)

where $t$ is the diffusion time step from 0 to T, and ${\beta }_t$ is a small positive hyperparameter determining the speed of the diffusion. $x_0$ is the original image and $x_T$ approximately follows the standard normal distribution.

From (1), we can derive $q\left(x_t\mid x_0\right)$ and $q\left(x_{t-1}\mid x_0,x_t\right)$:

$$q\left(x_t\mid x_0\right)=N\left(x_t\mid \sqrt{{\gamma }_t}{x}_0,\left(1-{\gamma }_t\right)I\right),$$ (2)

$$q\left(x_{t-1}\mid x_t,x_0\right)=N\left(x_{t-1}\mid {\tilde{\mu }}_t\left(x_t,x_0\right),{\tilde{\beta }}_{t}I\right),$$ (3)

where

$${\gamma }_t=\prod _{s=1}^t{\alpha }_s,$$ (4)

$${\alpha }_t=1-{\beta }_t,$$ (5)

$$\widetilde{{\mu }_t}\left(x_t,x_0\right)=\frac{\sqrt{{\gamma }_{t-1}}{\beta }_t}{1-{\gamma }_t}{x}_0+\frac{\sqrt{{\alpha }_t}\left(1-{\gamma }_{t-1}\right)}{1-{\gamma }_t}{x}_t,$$ (6)

and

$${\tilde{\beta }}_t=\frac{1-{\gamma }_{t-1}}{1-{\gamma }_t}{\beta }_t.$$ (7)

The reverse process, a Markov chain aimed at generating $x_0$ from $x_T~N(0,I)$, is modeled as a Gaussian process $p_{\theta }$:

$$p_{\theta }\left(x_{t-1}\mid x_t,y\right)=N\left(x_{t-1}\mid {\mu }_{\theta }\left(x_t,y,t\right),{\sigma }_t^2\right),$$ (8)

where ${\sigma }_t^2$ is a hyperparameter and is usually set as ${\tilde{\beta }}_t$, and ${\mu }_{\theta }$ is trained to match $\widetilde{{\mu }_t}$ to minimize the Kullback–Leibler (K-L) divergence between $p_{\theta }\left(x_{t-1}\mid x_t,y\right)$ and $q\left(x_{t-1}\mid x_t,x_0\right)$:

$${\theta }^{*}=\underset{\theta }{\text{argmin}}{E}_q\left\{\frac{1}{2{\sigma }_t^2}‖{\mu }_{\theta }\left(x_t,y,t\right)-{\tilde{\mu }}_t\left(x_t,x_0\right)‖\right\}.$$ (9)

To simplify the training, we can parameterize ${\mu }_{\theta }\left(x_t,y,t\right)$ as:

$${\mu }_{\theta }\left(x_t,y,t\right)={\tilde{\mu }}_t\left(x_t,{\widehat{x}}_{0,\theta }\left(x_t,y,t\right)\right),$$ (10)

where

$${\widehat{x}}_{0,\theta }\left(x_t,y,t\right)=\frac{1}{\sqrt{{\gamma }_t}}\left(x_t-\sqrt{1-{\gamma }_t}{ϵ}_{\theta }\left(x_t,y,t\right)\right).$$ (11)

It leads to our training loss function

$${\theta }^{*}=\underset{\theta }{\text{argmin}}{E}_{x_0,y}{E}_{t,ϵ}{‖{ϵ}_{\theta }\left(x_t,y,t\right)-ϵ‖}_2^2,$$ (12)

where

$$x_t=\sqrt{{\gamma }_t}{x}_0+\sqrt{1-{\gamma }_t}ϵ,ϵ~N\left(0,I\right).$$ (13)

After the network ${ϵ}_{\theta }$ is trained, one can predict $x_0$ from $x_T~N(0,I)$ and the condition $y$ following (8).

B. Framework of Noise-Controlled CT Super Resolution

The framework for noise-controlled CT super-resolution is illustrated in Fig. 1. The HR images are the labels $x_0$, and the corresponding LR images are the conditions $y$. Instead of directly training the conditional diffusion model using noise-unmatched real data, which is prone to amplifying noise while enhancing spatial resolution, our approach integrates training with hybrid datasets. These datasets comprise both noise-matched simulation data and segmented bones from real data.

Fig. 1.

Framework of Noise Controlled CT Super-Resolution

Numerical phantoms are utilized to simulate CT scans with the same geometry parameters as real CT images, yielding noise-free projections. Assuming that the quantum noise is Gaussian, the following equations are used to inject correlated noises into the HR and LR images:

$${\widehat{p}}_{HR}=p_{HR}+\frac{k_{HR}}{\sqrt{N_{HR}}}z$$ (14)

$${\widehat{p}}_{LR}=p_{LR}+\frac{k_{LR}}{\sqrt{N_{LR}}}\mathit{\text{downsample}}\left(z\right),$$ (15)

where $z~N(0,I).\widehat{p}$ and $p$ are the noisy and noiseless post-log projections, respectively. $N$ is the number of photons per ray that reached the detector, which is different for each detector. $k_{HR}$ and $k_{LR}$ are hyperparameters to match the noise levels of both simulated HR and LR images to that of the real LR image.

Because of the lack of details such as trabecular bone structures in the simulation phantoms, models trained with the simulation data alone would lead to significant oversmooth of the bony structures when applied to the real data. The bone regions from the real data are segmented using threshold followed by hole filling algorithms and opening and closing morphological operations, and serve as part of the training data, as demonstrated in Fig. 1. During testing, the trained model is used on both the original LR image and segmented bones. The pixels within the bone mask on the non-segmented super-resolved image are replaced by those in the bone-only super-resolution image.

In summary, the noise-matched simulated images provide the model the capability of super-resolution without amplification of the noise. Meanwhile, the segmented HR and LR bone pairs encourage the preservation of the detailed bony structures in reality without promoting noise amplification too much. By adopting both training data, the model is trained to enhance spatial resolution without noise amplification and good preservation of the bony structures.

C. Implementation Details

The real training and testing data were acquired using the OmniTom PCD portable photon counting CT system (Neurologica, Danvers USA) at Massachusetts General Hospital. Acquisition parameters included 120 kVp and 40 mAs for $1\times 1$ binning data, with an effective detector size of $0.12\times 0.14{\text{mm}}^2$ at the isocenter. The projection data were rebinned to $3\times 3$ and $6\times 5$ to generate paired HR and LR real data. The real training dataset comprises 64 slices of a cadaver head, and the real testing dataset includes 32 slices of a separate temporal bone.

Simulation data were generated using the XCIST simulation tool [8], utilizing head parts from five whole-body NURBS phantoms it provides, resulting in a total of 1040 slices. The hyperparameters $k_{HR}$ and $k_{LR}$ in equations (14) and (15) are set to 0.12 and 0.60, respectively, to match the noise level of the simulation data to that of the real LR image.

Both real and simulation data were reconstructed into $0.3\times 0.3{\text{mm}}^2$ HR images and $0.6\times 0.6{\text{mm}}^2$ LR images using edge-enhancing bone filter. To align their sizes for the conditional diffusion model training, the LR images were upsampled by two times using sinc-interpolation before being fed into the network as condition $y$ [9].

The architecture of the conditional noise predictor we employed is illustrated in Fig. 2. The core structure of the conditional noise predictor is a U-Net, similar to the one utilized in the DDPM [2]. $y$ and $x_t$ undergo a convolution layer separately and are then concatenated in the channel dimension. This structure is propagated through skip and residual connections across the entire U-Net, allowing the conditional noise predictor to extract information from $y$ and $x_t$ both separately and jointly.

Fig. 2.

Architecture of the conditional noise predictor. The content in parentheses (c, 2c, 4c, 8c and 16c) after the block name indicates the number of output channels of each block, and in our implementation c is set to 32.

We used $T=1,000$ as the total number of diffusion steps, and ${\beta }_t$ was selected according to the sigmoid schedule [10]. The model was trained on randomly cropped patches of $128\times 128$, and tested on the full image. A batch size of 64 was used during the training, with 48 slices from the noise-matched simulation and 16 from the real segmented bone images. The network was trained by the Adam algorithm with a learning rate of $8\times {10}^{-5}$ for 100,000 iterations.

III. Result

The representative results of noise-controlled super resolution from our proposed framework on LR CT images of a temporal bone are presented in Fig. 3, with quantitative analyses in Table I. The proposed method was further compared to two other methods: M1, which was trained with simulation data only; and M2, which was similar to the proposed method but without matched noise levels in the simulation.

Fig. 3.

Results on LR CT images of a temporal bone using our proposed framework and two comparison methods M1 and M2.

TABLE I.

Quantitative analysis for all tested methods on cases and ROIs in Fig. 3.

Method	STD (HU)		Haralick featuredistance		PSNR (dB)
	ROI 1	ROI 2	ROI 3	ROI 4	Case 1	Case 2
HR	60.4	61.2	N/A	N/A	N/A	N/A
LR	45.7	38.1	1367	1148	23.88	24.30
Proposed	41.5	39.9	325	535	23.79	24.33
M1	43.3	40.3	857	976	21.46	21.62
M2	54.1	51.9	324	550	22.97	24.26

Regions of interests (ROIs) are selected for quantitative analyses and marked in Fig. 3 using yellow ellipses and rectangles. Uniform ROIs 1 and 2 are used to calculate the standard deviation for evaluating the noise level, and ROIs 3 and 4, which contain intricate details, are used to calculate Haralick feature [11] distances to evaluate the similarity of the textures. Peak signal-to-noise ratio (PSNR) is also calculated to assess the consistency of the generated HR images.

Notably, M1 oversmoothes the trabecular bone structures due to the lack of such structures in the training data. Due to the oversmoothing, it has the worst Haralick feature distances in the bone ROIs (3 and 4) and the worst PSNR compared to the proposed method and M2. M2 has the similar enhancement of the spatial resolution and bony details compared to the proposed method, but it amplifies the noise as shown by the increased standard deviations in ROIs 1 and 2, leading to slightly worse PSNR compared to the proposed method. The proposed method achieved spatial resolution improvement without amplification of the noise. Both M2 and the proposed method had similar PSNR with the LR image, demonstrating that they are not drastically changing the structures as in M1. No significant improvement of the PSNR was observed due to the presence of noise in the HR images.

IV. Conclusion and Discussion

In conclusion, our proposed framework for noise controlled CT super resolution, leveraging the Conditional DDPM, presents a promising approach to enhance the spatial resolution of CT images while effectively controlling noise. By incorporating segmented details from real data and ensuring a matched noise level in simulation data during training, our model achieved image super-resolution without noise amplification. These results show its potential for practical applications in CT imaging, contributing to advancements in medical imaging technology.

One major limitation of the current method is that the proposed model depends on the precision of details segmentation, and errors in this process may introduce artifacts or result in the loss of details in the final output. Additionally, the training using segmented details featuring a completely black background may pose challenges to the performance of the conditional diffusion model. In our future work, we aim to investigate more robust and effective approaches to seamlessly integrate details unique to real data with noise-matched simulation data, thereby further enhancing the efficacy of noise-controlled CT super-resolution.