Abstract
Deep convolutional neural network (DCNN)-based noise reduction methods have been increasingly deployed in clinical CT. Accurate assessment of their spatial resolution properties is required. Spatial resolution is typically measured on physical phantoms, which may not represent the true performance of DCNN in patients as it is typically trained and tested with patient images and the generalizability of DNN to physical phantoms is questionable. In this work, we proposed a patient-data-based framework to measure the spatial resolution of DCNN methods, which involves lesion- and noise-insertion in projection domain, lesion ensemble averaging, and modulation transfer function measurement using an oversampled edge spread function from the cylindrical lesion signal. The impact of varying lesion contrast, dose levels, and CNN denoising strengths were investigated for a ResNet-based DCNN model trained using patient images. The spatial resolution degradation of DCNN reconstructions becomes more severe as the contrast or radiation dose decreased, or DCNN denoising strength increased. The measured 50%/10% MTF spatial frequencies of DCNN with highest denoising strength were (−500 HU:0.36/0.72 mm−1; −100 HU:0.32/0.65 mm−1; −50 HU:0.27/0.53 mm−1; −20 HU:0.18/0.36 mm−1; −10 HU:0.15/0.30 mm−1), while the 50%/10% MTF values of FBP were almost kept constant of 0.38/0.76 mm−1.
Keywords: Computed tomography (CT), Deep convolutional neural network (DCNN), Image quality
1. Introduction
Deep convolutional neural network (DCNN)-based methods have been deployed on clinical CT for noise reduction, either using CT vendor-developed methods directly available on the scanner or vendor-neutral image-based denoising methods [1, 2]. It is important to understand the fundamental spatial resolution performance of these DCNN methods as they are highly non-linear and sometimes difficult to interpret. Similar to iterative reconstruction (IR), spatial resolution of DCNN has been measured in physical phantoms and found to be contrast-dependent [3]. However, the performance measured on physical phantoms may not represent the true performance of DCNN in patients as DCNN is trained mostly on patient images and the generalizability to physical phantoms is questionable. In addition, precise measurement of edge responses at low-contrast levels (e.g., <30 HU) has been extremely challenging.
To overcome these challenges, in this work we aim to develop a patient-data-based framework using ensemble averaging over a large number of noise realizations of lesion-present and lesion-absent images. The impact of contrast level, noise level, and denoising strength on spatial resolution of a DCNN noise reduction method was also evaluated.
2. Methods
The proposed framework is illustrated in Fig. 1, which includes four major components: (1) projection-space lesion insertion; (2) projection-space noise insertion; (3) generation of mean lesion signal using ensemble averaging and subtraction; (4) calculation of modulation transfer function (MTF).
Figure 1.
A framework for measuring spatial resolution using patient data, which involves 4 major components: projection-space lesion insertion, noise insertion, generation of mean lesion signal using ensemble averaging and subtraction, and MTF calculation.
2.1. Projection-space lesion and noise insertion:
Cylindrically shaped low-contrast lesions were digitally generated. Each lesion was forward projected to the same locations using the same CT acquisition geometry as used in patient scans. The projected lesion data were combined with the patient raw data to create lesion-inserted projection data, using our previously validated projection-based lesion insertion tool [4].
Poisson noise was inserted into patient raw projection data to simulate chest CT exams acquired at lower radiation dose levels, using our previously validated projection-domain noise insertion tool [5]. Due to noise contamination, as well as non-linear responses of DCNN to each condition, it is challenging to obtain accurate edge information of the lesions. Therefore, for each experimental condition, independent noise realizations were inserted 600 times to generate ensembles of projection data.
2.2. Ensemble average of lesion signals and MTF calculation:
After image reconstruction and/or DCNN denoising, mean lesion signal was obtained by subtracting the ensemble average of the reconstructed images without lesion insertion from that with lesion insertion. Thereafter, oversampled edge spread function (ESF) was measured across the edges of the center slices of the ensemble averaged cylindrical lesions [6]. A logistic function (Fermi) [7] is fitted on the resulting oversampled ESF, and modulation transfer function (MTF) was calculated using the gradient of the ESF.
2.3. Using the proposed framework to evaluate a DCNN method:
Using the proposed framework, we evaluated the impact of lesion contrast, noise level and denoising strength on the spatial resolution of an example DCNN-based CT image denoising algorithm. The DCNN method was based on the ResNet model developed in our institution [8], trained with the Mayo/AAPM Grand Challenge cases [9], which has been used for many clinical applications [10, 11]. Multiple experimental conditions were created, including five lesion contrasts (−10, −20, −50, −100, −500 HU), five reconstruction/noise reduction types (FBP, SAFIRE-3, CNN-Weak, CNN-Medium, CNN-Strong) and three radiation dose levels (12.5%, 25%, and 50% of routine dose). SAFIRE-3 indicates Sinogram Affirmed Iterative Reconstruction with the strength setting of 3. The DCNN method was applied to the original FBP-reconstructed images. The CNN-Weak indicates the CNN denoising at matched noise level with SAFIRE-3, which is realized by adding scaled noise map to the predicted CNN denoising output. CNN-Strong indicates the CNN denoising without adding noise map back to the prediction, while CNN-Medium is in between CNN-Weak and CNN-Strong.
3. Results
Figure 2 shows one example of DCNN denoised images without or with lesions, with one noise realization and with ensemble averaging over the 600 noise realizations. The lesions after ensemble had much better-defined boundaries. The mean lesion signal at each contrast and noise level was obtained by subtracting the ensemble average of DCNN denoised images with lesion present from that without lesion (third row of Figure 2). The same procedure was applied for all the reconstruction methods (FBP, SAFIRE-3, CNN-Medium, CNN-Weak, CNN-Strong).
Figure 2.
Example patient images after DCNN denoising, without lesion (left column) and with lesions at different contrast levels (−10, −20, −50, −100, −500 HU). The top row shows an example of one single noise realization and the middle row shows the ensemble average of 600 realizations after DCNN denoising. The mean lesion signal (third row) was obtained by subtracting the ensemble average with lesion from that without lesion. The yellow arrows point to an inserted liver lesion. The display window is (WL, WW) = (40, 400) HU for patient image, and (−30,100) HU for lesion signal.
Figure 3 shows the ESF and MTF for the four contrast levels (−10, −20, −50, −100, and −500 HU), respectively, at the 25% of routine dose, for the five reconstruction algorithms, including FBP, SAFIRE-3 and three CNN denoising algorithms (CNN-Strong, CNN-Medium, and CNN-Weak).
Figure 3.
ESF and MTF measured from the ensemble averaged images acquired at 25% routine dose level for five different contrast levels (−10, −20, −50, −100, and −500 HU)
Figure 4 compares the ESF and MTF at the three dose levels, 12.5%, 25%, and 50% of routine dose respectively, at a contrast level of −100 HU, for the five reconstruction algorithms. A summary of all the results is provided in Tables 1 and 2, which compare the spatial resolution (quantified as the spatial frequency at 50% and 10% MTF) for the five reconstruction algorithms as a function of contrast level (Table 1) or dose levels (Table 2).
Figure 4.
ESF and MTF measured from the ensemble averaged images at three different dose levels (12.5%, 25%, and 50% of routine dose), at a contrast level of −100 HU
Table 1.
The measured 50%/10% MTF spatial frequencies at 5 different contrast levels for 5 different reconstructions (dose level was fixed at 25% of the routine dose level)
| −10HU | −20HU | −50HU | −100HU | −500HU | |
|---|---|---|---|---|---|
| FBP | 0.37/0.75 | 0.38/0.76 | 0.38/0.76 | 0.38/0.76 | 0.38/0.76 |
| SAFIRE-3 | 0.28/0.55 | 0.28/0.56 | 0.31/0.62 | 0.33/0.66 | 0.37/0.75 |
| CNN-Weak | 0.31/0.63 | 0.32/0.65 | 0.35/0.70 | 0.36/0.73 | 0.37/0.75 |
| CNN-Medium | 0.22/0.45 | 0.24/0.49 | 0.30/0.61 | 0.34/0.68 | 0.37/0.74 |
| CNN-Strong | 0.15/0.30 | 0.18/0.36 | 0.27/0.53 | 0.32/0.65 | 0.36/0.72 |
Table 2.
The measured 50%/10% MTF spatial frequencies at 3 different dose levels for 5 different reconstructions (contrast level was fixed at −100 HU)
| 12.5% dose | 25% dose | 50% dose | |
|---|---|---|---|
| FBP | 0.38/0.76 | 0.38/0.76 | 0.38/0.76 |
| SAFIRE-3 | 0.32/0.64 | 0.33/0.66 | 0.37/0.74 |
| CNN-Weak | 0.35/0.71 | 0.36/0.73 | 0.37/0.74 |
| CNN-Medium | 0.32/0.63 | 0.34/0.68 | 0.36/0.73 |
| CNN-Strong | 0.29/0.58 | 0.32/0.65 | 0.35/0.71 |
The spatial resolution of DCNN reconstructions, showed a strong dependency on both contrast and dose levels. At high contrast (−500HU), all three CNN denoising algorithms have almost identical MTFs as FBP. The degradation of MTF by DCNN compared to FBP becomes more severe as the contrast decreased. For example, the measured 50% MTF values of CNN-Strong decreased to 84.2%, 71.1%, 47.4%, 40.5% of the corresponding 50% MTF values of FBP at −100, −50, −20, and −10 HU, respectively. With the decrease of dose level, the degradation of the MTF of DCNN reconstructions relative to FBP become more severe. At the contrast level of −100 HU, the 50% MTF values of CNN-Strong decrease from 92.1% at 50% routine dose level to 84.2% and 76.3% at 25% and 12.5% routine dose levels, respectively, with MTF by FBP as reference. With the decrease of DCNN denoising strength, the degradation of MTF by DCNN is alleviated at the cost of higher noise level in the resulting images. With matched noise level, CNN-Weak can obtain higher MTF than SAFIRE-3 for low contrast lesions.
4. Conclusions
Quantifying spatial resolution at low contrast is important to understand the quality of DCNN noise reduction methods that have been deployed on clinical CT scanners. A methodology was developed to measure the spatial resolution after DCNN noise reduction at low contrast levels using real patient images. As with other non-linear image reconstruction and post-processing techniques, the evaluated DCNN method degraded the spatial resolution at low contrast levels, low radiation dose, and high denoising strength.
Acknowledgement
Research reported in this publication was supported in part by the National Institutes of Health under award numbers U24 EB028936 and R01 EB017095, and by the Department of Radiology Research Fellow Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Dr. McCollough receives industry grant support from Siemens.
References
- 1.Hsieh J, et al. , New Era of Image Reconstruction: TrueFidelity™. Technical White Paper on Deep Learning Image Reconstruction, GE Healthcare, 2019. [Google Scholar]
- 2.Boedeker K, AiCE Deep Learning Reconstruction: Bringing the power of Ultra-High Resolution CT to routine imaging. Technical white paper on deep learning reconstruction Canon Medical system,Canon Medical Systems Corporation, 2019. [Google Scholar]
- 3.Solomon J, et al. , Noise and spatial resolution properties of a commercially available deep learning-based CT reconstruction algorithm. Medical Physics, 2020. 47(9): p. 3961–3971. [DOI] [PubMed] [Google Scholar]
- 4.Chen BY, et al. , Lesion insertion in the projection domain: Methods and initial results. Medical Physics, 2015. 42(12): p. 7034–7042. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Yu LF, et al. , Development and Validation of a Practical Lower-Dose-Simulation Tool for Optimizing Computed Tomography Scan Protocols. Journal of Computer Assisted Tomography, 2012. 36(4): p. 477–487. [DOI] [PubMed] [Google Scholar]
- 6.Friedman SN, et al. , A simple approach to measure computed tomography (CT) modulation transfer function (MTF) and noise-power spectrum (NPS) using the American College of Radiology (ACR) accreditation phantom. Medical Physics, 2013. 40(5). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Pisana F, et al. , Noise reduction and functional maps image quality improvement in dynamic CT perfusion using a new k-means clustering guided bilateral filter (KMGB). Medical Physics, 2017. 44(7): p. 3464–3482. [DOI] [PubMed] [Google Scholar]
- 8.Huber NR, et al. , Evaluating a Convolutional Neural Network Noise Reduction Method When Applied to CT Images Reconstructed Differently Than Training Data. Journal of Computer Assisted Tomography, 2021. 45(4): p. 544–551. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.McCollough CH, et al. , Low-dose CT for the detection and classification of metastatic liver lesions: Results of the 2016 Low Dose CT Grand Challenge. Medical Physics, 2017. 44(10): p. e339–e352. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Huber N, et al. , Clinical evaluation of a phantom-based deep convolutional neural network for whole-body-low-dose and ultra-low-dose CT skeletal surveys. Skeletal Radiology, 2022. 51(1): p. 145–151. [DOI] [PubMed] [Google Scholar]
- 11.Huber N, et al. , Random search as a neural network optimization strategy for Convolutional-Neural-Network (CNN)-based noise reduction in CT. SPIE Medical Imaging. Vol. 11596. 2021: SPIE. [DOI] [PMC free article] [PubMed] [Google Scholar]