Anatomically aided PET image reconstruction using deep neural networks

Abstract

Purpose:

The developments of PET/CT and PET/MR scanners provide opportunities for improving PET image quality by using anatomical information. In this paper, we propose a novel co-learning 3D convolutional neural network (CNN) to extract modality-specific features from PET/CT image pairs and integrate complementary features into an iterative reconstruction framework to improve PET image reconstruction.

Methods:

We used a pre-trained deep neural network to represent PET images. The network was trained using low-count PET and CT image pairs as inputs and high-count PET images as labels. This network was then incorporated into a constrained maximum likelihood framework to regularize PET image reconstruction. Two different network structures were investigated for the integration of anatomical information from CT images. One was a multi-channel CNN, which treated PET and CT volumes as separate channels of the input. The other one was multi-branch CNN, which implemented separate encoders for PET and CT images to extract latent features and fed the combined latent features into a decoder. Using computer-based Monte Carlo simulations and two real patient datasets, the proposed method has been compared with existing methods, including the maximum likelihood expectation maximization (MLEM) reconstruction, a kernel-based reconstruction and a CNN-based deep penalty method with and without anatomical guidance.

Results:

Reconstructed images showed that the proposed constrained ML reconstruction approach produced higher quality images than the competing methods. The tumors in the lung region have higher contrast in the proposed constrained ML reconstruction than in the CNN-based deep penalty reconstruction. The image quality was further improved by incorporating the anatomical information. Moreover, the liver standard deviation was lower in the proposed approach than all the competing methods at a matched lesion contrast.

Conclusions:

The supervised co-learning strategy can improve the performance of constrained maximum likelihood reconstruction. Compared with existing techniques, the proposed method produced a better lesion contrast vs. background standard deviation trade-off curve, which can potentially improve lesion detection.

I. Introduction

Positron emission tomography (PET) is a widely used clinical imaging modality for cancer diagnosis, staging and treatment response assessment¹. Due to the potential radiation risk associated with the radioactive tracers, the injected activity in each PET scan is limited. As a result, PET data have low count statistics. Recent advances in PET instrumentation have focused on improving the photon detection sensitivity by measuring time-of-flight (TOF) information² and extending the axial field of view (FOV)³.

Noise in PET data can be further amplified in the image reconstruction process, resulting in noisy reconstructed images. In order to improve image reconstruction performance, prior information has been introduced to regularize the reconstruction. Instead of maximizing the likelihood function directly, the algorithm maximizes the posterior distribution. A common prior model is Tikhonov regularization, which assumes the images are smooth and continuous and consequently blurs sharp edges during reconstruction. To address this issue, other penalty terms, such as total variation (TV)⁴ and patched based penalties⁵ have been proposed to reduce the extent of blurring. However, these techniques still have limited ability to distinguish an edge from noise.

Since the spatial distribution of a radioactive tracer should be associated with anatomical structures, anatomical information from a co-registered CT or MRI can be applied to build an appropriate prior model. One challenge in anatomy-guided PET reconstruction is the existence of mismatches between different modalities that measure different morphological or functional information. A variety of methods have been developed to solve the mismatch issues, such as segmentation based A-MAP^6,7, Bowsher’s similarity method⁸, the joint entropy prior⁹, and level-set prior¹⁰. Hutchcroft et al¹¹ applied the kernel method¹² to anatomy-guided PET reconstruction using an anatomical image derived kernel matrix. This method has been extended to incorporate both PET and MRI derived kernel matrices based on the structural similarity metric¹³. Other interesting methods and comparisons between them can be found in^14,15,16. All of the above methods try to find an appropriate similarity measure to weigh PET and MRI features differently. Users can also manually adjust hyper-parameters, such as a predefined neighborhood size or the tolerance for relative differences, to emphasize modality-specific features. But the complexity of these adjustments introduces another new problem: lack of reproducibility.

Deep learning¹⁷, due to its unprecedented success in tasks such as image classification, segmentation, inpainting, and denoising, has found applications in medical image reconstruction. The convolutional neural network (CNN) is capable of implicitly fusing anatomical and functional features from multi-modality image data with proper training. Studies^18,19 have shown that co-learning features from PET-CT significantly increase tumor segmentation accuracy, compared with the use of a single modality. For image denoising, Lei et al.²⁰ proposed a deep learning architecture to estimate the high-quality PET images from the combination of low-count PET and T1-weighted MRI. Liu and Qi²¹ trained three modified U-Nets to improve the PET signal-to-noise ratio (SNR) with PET/MRI images. Similarly, Costa-Luis and Reader²² used a low-complexity CNN as an MR-aided image processing step to reduce noise and reconstruction artefacts. Subsequently, Cui et al.²³ presented a deep image prior²⁴ denoising method to leverage CT/MR information without using a high-count PET label. The cycle consistent generative adversarial network (GAN) was also utilized in PET denoising to predict diagnostic quality PET images with^25,26 and without anatomical aids^27,28. The cycle GAN algorithm not only learns a transformation to synthesize high-count PET images using low-count images but also learns an inverse transformation to ensure that pseudo low-count PET images generated from synthetic high-count PET are close to the true low-count PET images. Despite the fact that these approaches improve the model’s prediction, pathological abnormalities and patient-to-patient variations may reduce the robustness of these end-to-end denoising methods. Some instability phenomena have been reported in²⁹.

In order to safely use deep learning methods in medical imaging, data consistency needs to be taken into account. Besides the post-processing reconstructed images, deep learning has emerged as a new tool in image reconstruction^30,31,32. The various types of methods were introduced in a very short period of time, such as direct inversion^33,34, image regularization³⁵, and unrolling approaches³⁶. The approach of using a deep image prior²⁴ has also been applied to anatomically aided PET reconstruction³⁷.

Our aim in this paper is to improve the fusion of the complementary information in PET/CT or PET/MRI images for anatomy-aided reconstruction. In particular, we focus on the network-constrained image reconstruction method^38,39, where a denoising CNN, trained by high-quality images, is used to represent feasible PET images in a constrained maximum likelihood reconstruction framework. The low-count PET and CT images are used as inputs to train a 3D denoising CNN with high-count PET images as labels. The unknown PET image during a reconstruction is represented as the output of a pre-trained CNN in a constrained maximum likelihood framework. We investigated two different co-learning strategies. One is a multi-channel input CNN^20,40, which treats PET and CT volumes as two channels of the input. The other is a multi-branch CNN^19,41, which implements separate encoders for PET and CT images to extract features and concatenates the latent features before the decoder. By using computer-based Monte Carlo simulations and two real patient datasets, we compared the proposed method with existing methods, including EM reconstruction with Gaussian filtering, kernel-based reconstruction, and the CNN-based deep penalty method with and without anatomical guidance.

II. Methods and materials

II.A. Theoretical framework

The data model used in this work considers the measured PET data y as a collection of independent Poisson random variables. For a typical PET system with M number of lines of response (LOR) and N number of image voxels, the expectation of measured counts, $\overline{y}\in {ℝ}^{M\times 1}$, can be expressed by

$$\overline{\mathit{y}}=\mathit{P}\mathit{x}+\overline{\mathit{s}}+\overline{\mathit{r}},$$ (1)

where $x\in {ℝ}^{N\times 1}$ is the unknown radioactive tracer distribution, and $P\in {ℝ}^{M\times N}$ is the detection probability matrix. The (i, j)^th element of P, p_i,j, represents the probability that an emission originated in the j^th voxel is detected by the i^th LOR. $\overline{s}\in {ℝ}^{M\times 1}$ and $\overline{r}\in {ℝ}^{M\times 1}$ are the expected number of scattered and randoms coincidences, respectively. The Poisson log-likelihood function is given by

$$L(y\mid x)=\sum _{i=1}^M\left(y_i\right)\text{log}\left({\overline{y}}_i\right)-{\overline{y}}_i-\text{log}\left(y_i!\right).$$ (2)

Then the unknown image x can be estimated by the maximum likelihood estimator:

$$\widehat{x}=\text{arg}\underset{x\ge 0}{\text{max}}L(y\mid x).$$ (3)

To improve the image quality and reduce noise, we adopt a constrained maximum likelihood estimation framework. Basically, the unknown PET image x is represented by the output of a pre-trained CNN with multi-modality images, such as PET/CT or PET/MRI, as the input:

$$x=CNN\left({\alpha }_{CT},{\alpha }_{PET}\right)\text{or}x=CNN\left({\alpha }_{MRI},{\alpha }_{PET}\right),$$ (4)

where $CNN:{ℝ}^{N\times 1}\to {ℝ}^{N\times 1}$ denotes the pre-trained denoising neural network and α denotes the inputs to the network. Compared with previous work^38,39, the network inputs in this study not only include a noisy PET image (α_PET) but also a co-registered anatomical image (α_CT or α_MRI). Here we will focus on using CT images as an application example. Then the PET image is estimated by solving the following constrained maximum likelihood estimation problem:

$$\widehat{x}=\underset{x}{\text{arg}\text{max}}L(y\mid x),\mathit{\text{s}}.\mathit{\text{t}}.,x=CNN\left({\alpha }_{CT},{\alpha }_{PET}\right).$$ (5)

The network is trained off-line by using ML-EM reconstructions of high-count PET data as labels. When we only use PET images as the input to the network, the formulation of equation (5) simplifies to the previous method³⁸. We apply the augmented Lagrangian method to the constrained optimization problem in (5) and obtain

$$L_{\rho }=L(y\mid x)-\frac{\rho }{2}{\Vert x-CNN\left({\alpha }_{CT},{\alpha }_{PET}\right)+\mu \Vert }^2+\frac{\rho }{2}\Vert \mu {\Vert }^2,$$ (6)

where $\mu \in {ℝ}^{N\times 1}$ represents the vector of Lagrange multipliers and ρ > 0 is a hyperparameter, which affects the convergence speed but does not change the final solution at convergence. When the algorithm is stopped before convergence, a larger ρ generally results in a smoother image. In this study, we empirically chose the value of ρ based on the reconstruction of the validation data. Equation (6) can be maximized by the alternating direction method of multipliers (ADMM) algorithm⁴² in three steps:

$$x^{n+1}=\underset{x}{\text{arg}\text{max}}L(y\mid x)-\frac{\rho }{2}{\Vert x-CNN\left({\alpha }_{CT},{\alpha }_{PET}^n\right)+{\mu }^n\Vert }^2,$$ (7)

$${\alpha }_{PET}^{n+1}=\underset{{\alpha }_{PET}}{\text{arg}\text{min}}{\Vert CNN\left({\alpha }_{CT},{\alpha }_{PET}\right)-\left(x^{n+1}+{\mu }^n\right)\Vert }^2,$$ (8)

$${\mu }^{n+1}={\mu }^n+x^{n+1}-CNN\left({\alpha }_{CT},{\alpha }_{PET}^{n+1}\right).$$ (9)

We solve equation (7) by using the optimized transfer method⁵:

$${\widehat{x}}_j^{n+1}=\frac{1}{2}\left[CNN\left({\alpha }_{PET}^n,{\alpha }_{CT}\right)j-{\mu }_j^n-\frac{p_j}{\rho }-\sqrt{{\left(CNN\left({\alpha }_{PET}^n,{\alpha }_{CT}\right)j-{\mu }_j^n-\frac{p_j}{\rho }\right)}^2+\frac{4x_{j,EM}^{n+1}{p}_j}{\rho }}\right],$$ (10)

where $p_j={\sum }_{i=1}^M{p}_{ij}$ and $x_{j,EM}^{n+1}$ is obtained by the MLEM update:

$$x_{j,EM}^{n+1}=\frac{x_j^n}{p_j}\left[\sum _{i=1}^M\frac{p_{ij}{y}_i}{{\sum }_{j=1}^N\left(p_{ij}{x}_j^n\right)+s_i+r_i}\right].$$ (11)

The subproblem (8) can be solved using a gradient ascent algorithm:

$${\left({\alpha }_{PET}^{n+1}\right)}_j={\left({\alpha }_{PET}^n\right)}_j-S\left[\sum _{m=1}^N\frac{\partial CNN{\left({\alpha }_{PET}^n,{\alpha }_{CT}\right)}_m}{\partial {\left({\alpha }_{PET}\right)}_j}\rho \left[x_m^{n+1}+{\mu }_m^n-CNN{\left({\alpha }_{PET}^n,{\alpha }_{CT}\right)}_m\right]\right],$$ (12)

We use ‘tf.GradientTape’ API of TensorFlow for automatic differentiation. As the intensity of network input is normalized, once a proper step size S is found, it can be used for different datasets. In this work, we use S = 0.01 for all three different datasets. It should be noted that, there is no update on the anatomical input, i.e., α_CT remains the same during iterations.

II.B. Network architecture

In this work, we investigated two co-learning strategies and compared the results with those from the single-modality (PET only) CNN. Fig. 1 shows the structure of the three networks. For combining functional (PET) image and structural (CT) images, we start with a multi-channel input CNN modified from the U-net⁴³ as shown in Fig.1 (b), which treats PET/CT images as two separate input channels. Except the first convolutional layer, multi-channel input CNN has the same architecture as the PET-only input CNN shown in Fig.1 (a). Specifically, the first convolutional layer contains 16 filters of support 3×3×3×c, where c is the number of input channels (with c=1 for PET-only CNN and c=2 for multi-channel input CNN), and 3×3×3 denotes the spatial size of the filter. The second co-learning architecture is a multi-branch CNN shown in Fig.1 (c), which implements separate encoders for PET and CT images. The modality-specific features generated by two encoders are concatenated channel-wise. In this work, we also include a self-attention module to learn the optimal weights of features in the latent space^44,45. Essentially, the self-attention module reweighs features in a certain layer of the network according to inter-dependencies of different features. Similar to a non-local means filter, the self-attention module tries to capture long-range spatial dependencies. In our previous work⁴⁶, we demonstrated that the U-net with a self-attention module outperforms the U-net without a self-attention module. For multi-branch input network, the self-attention module enables to weigh modality-specific features at different locations, which is expected to efficiently fuse features from the two encoders. All three networks have the same decoder for synthesizing high-count PET images.

Figure 1:

The architecture of three 3D deep neural networks with (a) PET input only, (b) PET/CT multi-channel input and (c) PET/CT multi-branch input, respectively. The input to each modality-specific encoder is a 3D patch of the corresponding modality.

We train the networks using the adaptive moment estimation (Adam) optimizer⁴⁷ with following loss function

$$L=\Vert CNN\left({\alpha }_{CT},{\alpha }_{PET}\right)-\mathit{\text{x}}{\Vert }^2\text{or}L=CNN\left({\alpha }_{PET}\right)-\mathit{\text{x}}{\Vert }^2$$ (13)

where α_CT, α_PET and x denote co-registered CT images, low and high count PET images, respectively. The training was implemented using the open-source library Keras 2.2.5 with Tensorflow backbone and trained on an NVIDIA GTX 1080 Ti GPU. Considering limited graphics processing unit (GPU) memory, 3D patches (64×64×48) from input images were extracted for training by sliding a window on each image with a stride of 32. We empirically set the mini-batch size to 8, the number of epochs to 1000, and the decayed learning rate schedule as follows: initial value was set to 0.004 and decay of rate 0.5 every 400 epochs. The L2 losses of the training and validation datasets for each epoch were saved and are shown in Fig. 2.

Figure 2:

The training and validation losses as a function of epoch.

II.C. Data pre-processing

Prior to training, the CT image was downsampled using linear interpolation to match the PET pixel size and global registration was performed. In order to enhance soft-tissue contrast in CT images, we used the histogram equalization technique, which was performed using the function “histeq” in MATLAB. Essentially, the histogram equalization process finds a monotonic gray scale transformation such that the pixel intensities in the transformed image are uniformly distributed across the full gray scale. The image after histogram equalization would have a linearized cumulative distribution across the value range. Typical Hounsfield unit (HU) values of CT images are ranging from −1000 to 1000. In this paper, we add 1000 to the CT images and therefore the gray scale ranges from 0 to 2000. As shown in Fig.3, histogram equalization enhances soft-tissue contrast. Additionally, the intensity of CT and PET images are normalized by their own standard deviation. For a fair comparison, we use histogram equalized CT images in all anatomically aided reconstruction methods.

Figure 3:

CT images and corresponding cumulative histogram (a) before and (b) after histogram equalization.

II.D. Reference Methods

In this study, we compared the proposed method with the kernel based reconstruction¹², a state-of-the-art CNN-based regularized reconstruction^48,49, and the iterative CNN constrained method published previously³⁸.

Kernel method:

We used the CT image to generate a kernel matrix using a radial Gaussian function

$$K_{CT}(i,j)=\exp \left(-\frac{{\Vert \mathit{\mu }}_i-{\mathit{\mu }}_j\Vert {}^2}{2N_f^{CT}{\sigma }^2}\right),$$ (14)

where μ_i and μ_j are 3×3×3 local patches centered at voxel i and voxel j, respectively, in the CT image and N_f = 27 is the number of voxels in the patch. For efficient computation, all kernel matrices were constructed using a k-Nearest-Neighbor (kNN) search¹³ in a 7×7×7 search window with k = 30. Before feature extraction, CT images were normalized so that the image standard variation (σ) was equal to 1. The reconstructed image is then obtained by

$$\widehat{x}=\underset{\alpha }{\text{arg}\text{max}}L\left(y\mid K_{CT}\alpha \right)$$ (15)

which is solved by the kernel-EM algorithm¹². This method is referred to as KEMCT.

Non–local meanpost–filtering method:

We applied non-local means (NLM) post-filtering to MLEM reconstruction. The filter kernel was built using the histogram-equalized CT images as described in equation (14). This method is referred to as NLM.

Deep penalty method:

We use the pre-trained CNN in the penalty function of a penalized ML reconstruction^48,49. The reconstruction of the image x is formulated as the following optimization problem:

$$\widehat{x}=\underset{x}{\text{arg}\text{max}}L(y\mid x)-\beta \Vert x-CNN{\left({\mathit{x}}_0\right)\Vert }^2$$ (16)

where x₀ denotes an initial noisy reconstruction. The networks used in equation (16) are the same networks shown in Fig. 1 with either PET images only or PET/CT multi-modality images as the input. By using different networks, this method is denoted as DP-PET-only, DP-MC and DP-MB, respectively. Similar to equation (7), equation (16) with the quadratic prior can be solved by the optimization transfer method⁵.

Previous CNN constrained method:

This method represents the unknown PET image x by the output of a pre-trained CNN with PET only input without using anatomical information:

$$\widehat{x}=\underset{x}{\text{arg}\text{max}}L(y\mid x),\mathit{\text{s}}.\mathit{\text{t}}.,x=CNN\left({\alpha }_{PET}\right)$$ (17)

Similar to our proposed method, equation (17) is solved by ADMM³⁸. This iterative CNN constrained method is denoted as ADMM-PET-only.

III. Experimental evaluations

III.A. Simulation Study

The computer simulation was used to evaluate the proposed method based on the geometry of a GE 690 scanner⁵⁰. We simulated twenty XCAT⁵¹ phantoms (18 for training, 1 for validation and 1 for testing) with different organ sizes and genders. The image dimension and voxel size were 128 × 128 × 49 and 3.27 × 3.27 × 3.27 mm³, respectively. In the training datasets, 30 hot spheres were inserted in the lung region with diameters ranging from 12.8 mm to 22.1 mm. Similarly, we inserted two spherical lesions with diameters of 12.8 mm in the test phantom. The tissue specific time-activity curves were obtained using a compartmental model with kinetic parameters mimicking an ¹⁸F-FDG scan. Details of the simulation setup were described in our previous work³⁸. The neural networks were trained with low-count reconstructions (4-min scans) as inputs and high-count reconstructions (40-min scans) as labels. Afterward, we quantitatively evaluated the performance of each method using the test phantom. The contrast recovery (CR) for the lesions vs. the background standard deviation (STD) curves were plotted. The CR was determined by

$$\text{CR}=\frac{1}{R}\sum _{r=1}^R\frac{{\overline{l}}_r}{{\overline{l}}^{\mathit{\text{true}}}}$$ (18)

where ${\overline{l}}_r$ is the uptake of the lung lesion in realization r, and ${\overline{l}}^{\mathit{\text{true}}}$ is the true lesion uptake value. The total number of realizations is R = 10. The background STD was calculated as

$$\text{STD}=\frac{1}{K_b}\sum _{k=1}^{K_b}\frac{\sqrt{\frac{1}{R-1}{\sum }_{r=1}^R{\left(b_{r,k}-{\overline{b}}_k\right)}^2}}{{\overline{b}}_k},{\overline{b}}_k=1/R\sum _{r=1}^R{b}_{r,k}$$ (19)

where b_r,k is the average value of the k^th background ROI in the r^th realization and K_b = 20 is the number of 2D circular ROIs in each image. The background ROIs were manually drawn in the liver region across multiple axial slices.

III.B. Real patient studies

III.B.1. Hybrid lesion datasets

The hybrid lesion dataset consisted of 6 female patients (5 for training and 1 for testing), who received one-hour ¹⁸F-FDG dynamic scans on the GE Discovery 690 TOF-PET/CT scanner⁵⁰ (GE Healthcare, Waukesha, WI, USA). The study was approved by the UC Davis Institutional Review Board. More patient information can be found in our previous work³⁹. We generated high-count data using the measured events from 20 to 60 minutes post injection. Ten realizations of low-count data were obtained by randomly down-sampling the high-count data to 1/10th events. The reconstruction image dimension and voxel size were 180 × 180 × 49 and 3.27 × 3.27 × 3.27 mm³, respectively. For each realization, image augmentation was used by rotating the image with an angle randomly sampled between 0 and 90 degree³⁹. The neural networks were trained with low-count reconstructions as inputs and high-count reconstructions as labels.

For quantitative analysis, spherical lesions with diameters of 13.1 mm and 16.4 mm were manually inserted into the testing data, but not into the training data. Lesion CR vs. background STD curves were obtained using the procedure described in Section III.A..

III.B.2. Lung cancer datasets

The lung cancer dataset included seven ¹⁸F-FDG patient scans (5 for training, 1 for validation and 1 for testing) from a Canon Celesteion TOF-PET/CT scanner⁵² (Canon Medical Corporation, Tochigi, Japan). Two 50% overlapping bed positions were acquired with 14 minutes per bed. The body weights distribution of patients was in a moderate range from 45 to 74 kg and the injected activity ranged from 160 to 250 MBq. The 14-minute scans were used as the high-count datasets and the low-count datasets were generated by downsampling the high-count datasets to 1/7th events. The same image augmentation procedure described above was used³⁹. The reconstructed image dimension and voxel size were respectively 272 × 272 × 141 and 2.0 × 2.0 × 2.0 mm³. For quantitative analysis, liver ROIs were placed in a uniform area to calculate the STD. Although true lesion uptakes in real patient datasets were not known, we computed the CR based on the high-count reconstruction (100 MLEM iterations), which can serve as a reasonable surrogate of the ground truth.

IV. Results

IV.A. Simulation data

Fig. 4 shows transverse (top), coronal (middle), and sagittal (bottom) views of different reconstructed images through the lesion of interest. All of the methods are compared at a matched lesion contrast, indicated by the dashed black line in Fig. 5 (a). Both the NLM filter and the CT-aided kernel reconstruction demonstrated reduced noise and reduced partial volume effects (PVE), in comparison with the commonly used MLEM. Compared with the NLM filter, KEMCT can generate clearer myocardial structures with the help of the data consistency. Comparing images in fig 4(d)–(f) with those in fig 4(g,h,j), we can see that the results of CNN constrained reconstructions (ADMM-PET-only, ADMM-MC, ADMM-MB) have lower noise in the uniform region compared to their deep penalty counterparts (DP-PET-only, DP-MC, DP-MB). Comparing Fig. 4(i) with Fig. 4(j), we found that with histogram equalization, the myocardium boundary is more continuous and the liver region is smoother. We also computed the mean squared error (MSE) of the reconstructed test image with respect to the high-count reference image (HC-MLEM) as a function of iterations. The MSE curves in Fig. 5 (b) indicated that ADMM-MB has the lowest MSE.

Figure 4:

Reconstructed low-count images using different methods for the simulated test dataset. ADMM-MB-histequ denotes our proposed method method that uses the multi-branch input CNN constrained reconstruction with histogram equalized CT image. HC-MLEM represents MLEM reconstruction of high-count data for reference. Our proposed method is marked in bold face.

Figure 5:

(a) The contrast recovery (CR) versus background standard derivation (STD) curves for the simulated test dataset. (b) The MSE as a function of iteration number for the simulated test dataset. Markers are plotted for every 10 iterations. Deep learning based methods were initialized with the 30^th MLEM iteration.

From the CR vs. STD curves shown in Fig. 5, we see that the proposed iterative CNN constrained reconstruction with the multi-branch input outperforms all other methods with the highest CR at any matched STD level. In contast to the NLM method, both KEMCT and deep learning based reconstruction can better preserve lesions uptake. We can also see that both multi-channel input and multi-branch input have lower liver STD than the PET-only input result, although the difference between the CNN constrained reconstructions is relatively small. This finding indicates that the use of CT information in these methods can reduce noise in uniform soft-tissue regions. Including histogram equalization (ADMM-MB-histequ) also improves the convergence speed over using the CT image directly (ADMM-MB).

IV.B. Hybrid lesion datasets

Fig. 6 shows three orthogonal views of different reconstructed images through the lesion of interest. Similar to the simulation test dataset, all of the methods were compared at a matched lesion contrast, as indicated by the dashed black line in the CR vs. STD curves (Fig. 7). The CT-aided kernel reconstruction reduced noise and partial volume effects (PVE), in comparison with the commonly used MLEM. Compared with the deep penalty approaches, the CNN constrained reconstruction framework can recover more image details. Fig. 6 (c), (d), and (e) show that the deep penalty methods resulted in lower lung lesion contrast due to an increased smoothing across the images. The multi-channel network loses some fine details around the spine, as shown in Fig. 6 (d) and (g). Similar to KEMCT, the high activity region inside the bone marrow cannot be successfully identified.

Figure 6:

Reconstructed low-count images using different methods for the hybrid lesion test dataset. Our proposed method is marked in bold face. HC-MLEM denotes MLEM reconstruction of high-count data for reference.

Figure 7:

The contrast recovery (CR) versus background standard derivation (STD) curves for the hybrid lesion test dataset. Markers are plotted for every 10 iterations. Deep learning based methods were initialized with the 30^th MLEM iteration.

In comparison, the multi-branch CNN constrained reconstruction holds the best CR-STD trade-off among all the methods. Fig. 7 demonstrates the proposed anatomy-aided CNN constrained reconstruction (ADMM-MB) can achieve better performance than KEMCT. After introducing histogram equalization, ADMM-MB-histequ converges faster and achieves a slightly better performance than ADMM-MB. Finally, compared with methods without using anatomical information, such as ADMM-PET-only and MLEM, our method can obtain 41% and 86% noise reduction, respectively, at matched CR levels.

IV.C. Lung cancer datasets

Fig. 8 shows the representative slices of reconstructed images and Fig. 9 shows the lung lesion contrast vs. liver STD curves for the test dataset. Using the KEMCT method (Fig. 8(b)), the noise has been suppressed but the lesion contrast is reduced. This is due to the imperfect registration between CT and PET. From our observations, there are some local mis-registrations between the CT and the PET, possibly due to repiratory motion. Compared with the KEMCT method, CNN constrained reconstruction achieved a higher lesion contrast while keeping a lower noise level. Moreover, ADMM-MB produces higher lesion contrast and has better noise tolerance than ADMM-PET-only. Similar to the results of hybrid lesion dataset, we also find that the liver is smoother in the anatomy-aided reconstructions compared to PET-only results (indicated by the black arrow). By visual inspection, our proposed method (ADMM-MB) has similar texture pattern compared to the high-count ML-EM reference. For instance, we can see that tumor activity in the lung is more uniformly distributed in the ADMM-MB method (Fig. 8(e)) compared to the ADMM-MB-w/o-histequ image (Fig. 8 (d)) and the ADMM-PET-only image (Fig. 8(c)).

Figure 8:

Reconstructed low-count images using different methods for the lung cancer test dataset. Our proposed method is marked in bold face. HC-MLEM denotes MLEM reconstruction of high-count data for reference.

Figure 9:

The contrast recovery (CR) versus background standard derivation (STD) curves for the lung cancer test dataset. Markers are plotted for every 10 iterations. Deep learning based methods were initialized with the 30^th MLEM iteration.

V. Discussion

In this paper, we investigated two co-learning strategies for regularized PET image reconstruction. Our experimental results showed that the ADMM-MB-histequ outperformed existing methods, such as PET-only CNN-based reconstruction³⁸ and anatomy-aided kernel reconstruction¹¹. Although the KEMCT method has demonstrated reduced noise and reduced PVE properties, in practice, CT(only)-aided method is easily influenced by the mis-registration introduced by respiratory motion or the registration algorithm, as shown in the lung cancer dataset. In comparison, a deep network trained using real PET/CT images learns the mis-registration between low-count PET and CT inputs. Our experimental results demonstrated that the co-learning methods could compensate mismatch effects at least to some extent compared with KEMCT, as shown in Fig. 8.

Similar to previously reported methods¹⁶, the anato-functional methods which combine both anatomical and functional information, perform better than anatomy-aided only methods. It also should be noted that two co-learning strategies behave differently in terms of recovering small lesions, while both MC and MB have similar noise reduction ability. Without the help of modality-specific encoders, the multi-channel CNN combines both modalities in the first convolutional layer. However, as indicated by the metrics in Fig. 7 and the images in Fig. 6, this co-learning strategy tends to prioritize information from CT images at the expense of information from the PET data. A clear example is in the bone marrow, and only part of the bone marrow can be separated in DP-MC and ADMM-MC, because the tracer uptakes in the bone marrow are better separated in the PET image than in the CT image. Also the multi-channel network may not be able to fully learn the correlation between features in PET and CT images. To show that the improvement in performance was due to the change in network architecture, not simply by increasing the number of trainable parameters, we also trained a multi-branch network with 0.54 million parameters (8 root features). The new multi-branch network still outperformed the multi-channel network with 0.8 million parameters.

Compared with deep image prior based³⁷ regularized reconstruction, our pre-trained network is capable of incorporating both inter-patient information and intra-patient information by learning from high-quality training samples. Additionally, the deep image prior method requires early stopping and the optimal iteration number is difficult to determine. In this paper, we do not introduce an extra likelihood function that encourages the network input to be similar to the unregularized ML reconstruction as we did in³⁹, because we observed monotonic increase of L(y|α)) after incorporating the anatomical information. This indicates that with the introduction of anatomical information, pre-trained networks are less likely to generate undesirable outputs.

Another popular way of deep learning based reconstruction is the unrolling approach, which unrolls an iterative image reconstruction algorithm into a layer-wise structure and replaces certain components by neural networks. It often results in better interpretability. An unrolled network consists of series of modules, each having a regularization and data consistency unit. The number of trainable parameters increases with the number of modules if the network parameters are not shared among the modules⁵³, and the GPU memory usage of the whole unrolled network can be huge. Furthermore, because the forward and back projections are embedded inside each module, the computational cost for an end-to-end training is extremely high. In comparison, we incorporate a pre-trained network in a constraint iterative reconstruction framework. The GPU memory usage is independent of iteration number, so we can use a deeper network. Also because the training does not involve any forward and back projection operations, the computational cost for training is much lower. Compared with unrolled methods, the performance of the ADMM in our method is sensitive to the hyperparameter ρ, which was manually adjusted in this work. How to address and alleviate this issue is of interest in our future studies.

One challenge in anatomically guided PET reconstruction is the potential mismatch between PET and CT images. We observed an example of a PET-unique lesion case in the validation data of the lung cancer dataset. As shown in Fig.10, the lesion can be barely identified in the CT image. Fig. 10(b) shows that KEMCT, which only uses CT information, has the worst lesion contrast. ADMM-PET-only performs better than ADMM-MB in terms of lesion contrast. However, ADMM-MB has a lower STD in the liver region at matched contrast level, which is mainly attributed to the much smoother background in CT images.

Figure 10:

Reconstructed low-count images using different methods for the lung cancer validation dataset. Our proposed method is marked in bold face. HC-MLEM denotes MLEM reconstruction of high-count data for reference.

In this study, we trained two CNNs with datasets acquired on GE and Canon scanners separately. In a previous work⁵⁴, we also tried to directly deployed the network trained using GE dataset to the Canon dataset. The results indicated a reasonable generalizability of the pre-trained network between inter-vendor scanners. Furthermore, integrating the deep neural network in iterative reconstruction can improve the image quality and reduce the variation between different networks. While we have only studied FDG PET scans so far, our reconstruction framework is applicable to other PET tracers. However, to achieve the best performance, additional fine tuning of the network parameters may be necessary for some PET tracers.

VI. Conclusion

In this work, we proposed an anatomy-aided PET image reconstruction method. By co-learning PET/CT information, the pre-trained deep neural network is used to represent PET images and to incorporate inter- and intra-patient information into PET image reconstruction. Evaluations using computer simulations and real datasets indicated that the proposed reconstruction method improved image quality and produced a better lesion contrast vs. background standard deviation trade-off curve compared existing methods, such as ML-EM reconstruction, kernel-based reconstruction and CNN-based penalized reconstruction with and without anatomical guidance.

Table 1:

The acronyms of different methods.

	Methods	Abbreviation	Histequ preprocessing
Low count
	High count MLEM	HC-MLEM	--
	Low count MLEM	MLEM	--
	Non-local mean post processing based on CT kernel	NLM	w/
	CT kernel base reconstruction	KEMCT	w/
	Deep penalty reconstruction base on PET-only input network	DP-PET-only	--
	Deep penalty reconstruction base on multi-channel input network	DP-MC	w/
	Deep penalty reconstruction base on multi-branch input network	DP-MB	w/
	Deep penalty reconstruction base on PET-only input network	DP-PET-only	--
	PET-only input network-based iterative reconstruction with input constrain	ADMM-PET-only	--
	Multi-channel input network-based iterative reconstruction with input constrain	ADMM-MC	w/
	Multi-branch input network-based iterative reconstruction with input constrain	ADMM-MB	w/
	Multi-branch input network-based iterative reconstruction with input constrain	ADMM-MB-w/o-hisequ	w/o

-- : The inputs do not include CT.

Table 2:

The summary of three datasets.

		Dataset	Scan time	Numbers of subject	Image dimension	Voxel size (mm3)
Simulation	GE 690 geometry	Low count (20th, 30th, 50th MLEM iteration)	4 min	20 phantoms: 18 for training, 1 for validation, 1 for testing	128 × 128 × 49	3:27 × 3:27 × 3:27
		High count (50th MLEM iteration)	40 min
Real patient	GE 690	Low count (20th, 30th, 50th MLEM iteration)	4 min	6 patients: 5 for training, 1 for testing	128 × 128 ×49	3:27 × 3:27 × 3:27
		High count (50th MLEM iteration)	40 min
	Canon Celesteion	Low count (20th, 30th, 50th MLEM iteration)	2 min	7 patients: 5 for training, 1 for validation, 1 for testing	272 × 272 × 141	2:0 × 2:0 × 2:0
		High count (50th MLEM iteration)	14 min

Data Availability Statement

The authors are unable to share patient data at this time.