PET Image Reconstruction Using Deep Image Prior

Abstract

Recently deep neural networks have been widely and successfully applied in computer vision tasks and attracted growing interests in medical imaging. One barrier for the application of deep neural networks to medical imaging is the need of large amounts of prior training pairs, which is not always feasible in clinical practice. This is especially true for medical image reconstruction problems, where raw data are needed. Inspired by the deep image prior framework, in this work we proposed a personalized network training method where no prior training pairs are needed, but only the patient’ own prior information. The network is updated during the iterative reconstruction process using the patient specific prior information and measured data. We formulated the maximum likelihood estimation as a constrained optimization problem and solved it using the alternating direction method of multipliers (ADMM) algorithm. Magnetic resonance imaging (MRI) guided Positron emission tomography (PET) reconstruction was employed as an example to demonstrate the effectiveness of the proposed framework. Quantification results based on simulation and real data show that the proposed reconstruction framework can outperform Gaussian post-smoothing and anatomically-guided reconstructions using the kernel method or the neural network penalty.

I. Introduction

Over the past several years, deep neural networks have been widely and successfully applied to various imaging tasks such as segmentation [1], object detection [2] and image synethesis [3], by demonstrating better performance than state-of-the-art methods when large amounts of data sets are available. For medical imaging tasks such as lesion detection and region-of-interest (ROI) quantification, obtaining high quality diagnostic images is essential. Recently the neural network method has been applied to transform low-quality images into the images with improved signal-to-noise ratio (SNR).

During network training, images reconstructed from high dose or long-scanned duration are required to be used as the training labels. In some cases collecting large amounts of training labels is easy, such as static magnetic resonance (MR) reconstruction. However, this is not an easy task for some other cases: high-dose computed tomography (CT) has potential safety concerns; long-scanned dynamic positron emission tomography (PET) is not employed in routine clinical practice; in cardiac MR applications, it is impossible to acquire breath-hold fully sampled 3D images. With limited amounts of high-quality patient data sets available, overfitting can be a potential pitfall: if new patient data does not lie in the training space due to population difference, the trained network cannot accurately recover unseen structures. In addition, low-quality images are often simulated by artificially downsampling the full-dose/high-count data, which may not reflect the real physical condition of low-dose imaging. This mismatch between training and the real clinical environment can reduce the network performance.

Apart from using training pairs to perform supervised learning, a lot of prior arts focus on exploiting prior images acquired from the same patient to improve the image quality. The priors can come from temporal information [4], [5], different physics settings [6], or even other imaging modalities [7]. They are included into the maximum posterior estimation or sparse representation framework using pre-defined analytical expressions or pre-learning steps. The pre-defined expressions might not be able to extract all the useful information, and the pre-learnt model might not be optimal for the later reconstruction as no data consistency constraint is enforced during pre-learning. Ideally the learning process should be included inside the reconstruction framework.

Recently, the deep image prior (DIP) framework proposed in [8] shows that convolutional neural networks (CNNs) have the intrinsic ability to regularize a variety of ill-posed inverse problems without pre-training. No prior training pairs are needed and random noise can be employed as the network input to generate denoised images. The ability of CNN to learn the structure information is also revealed in the deep convolutional generative adversarial network (GAN) [9] where the generator is fully convolutional, and the network can generate various distributions based on random noise input. Furthermore, it has been presented in conditional GAN works [10]–[12] that when the input is not random noise, but the associated prior information, the prediction results can be improved. This shows that for the DIP framework, its conditional version using prior information as input, instead of random noise, may generate better results.

Inspired by the prior arts, we proposed a personalized image reconstruction framework based on the conditional DIP method, called DIPRecon. The input is not random noise, but prior images of the same patient. Instead of calculating the mean squared error (MSE) between the network output and the original corrupted image, we formulated the training objective function based on maximum likelihood estimation derived from imaging physics. Modified 3D U-net [13] was employed as the neural network structure. To stabilize the network training, the limited memory BFGS (L-BFGS) algorithm [14] was used instead of adaptive stochastic gradient descent (SGD) methods.

PET is a molecular imaging modality widely used in neurology studies. The image resolution of current PET scanners is still limited by various physical degradation factors [15]. Improving PET image resolution is essential for a lot of applications, such as dopamine neurotransmitter imaging, brain tumor staging and early diagnosis of Alzheimer’s disease. For the past decades, various efforts are focusing on using MR or CT to improve PET image quality [16]–[23]. In this work, the anatomically-aided PET image reconstruction problem was presented as an example to demonstrate the effectiveness of the proposed DIPRecon framework. Compared with the state-of-the-art kernel-based method and the penalized reconstruction based on a neural network penalty, DIPRecon shows superior performance both visually and quantitatively in simulation and real data experiments.

This paper is organized as follows. Section 2 describes the related works. Section 3 introduces the DIPRecon framework and implementation details. Section 4 describes the simulations and real data used in the evaluation. Experimental results are shown in section 5, followed by discussions in section 6. Finally, conclusions are drawn in Section 7.

II. Related Works

A. Deep neural network with training pairs

When large amounts of training pairs are available, a neural network can be trained by

$$\widehat{\mathit{\theta }}=\arg \underset{\mathit{\theta }}{\min }\sum _i\Vert {\mathit{x}}_{\text{label}}^i-f\left(\mathit{\theta }|{\mathit{z}}^i\right)\Vert ,$$ (1)

where $f:ℝ\to ℝ$ represents the neural network, $\mathit{\theta }\in {ℝ}^L$ indicates the trainable variables, ${\mathit{z}}^i\in {ℝ}^N$ is the network input for the ith training pair, and ${\mathit{x}}_{\text{label}}^i\in {ℝ}^N$ denotes the ith training label. For CNN, θ contains the convolutional filters and the bias items from all layers. The trained network can be directly applied to image denoising [24]–[27]. Apart from image denoising, some efforts focus on including the trained neural network into an iterative reconstruction framework, using penalty design or variable reparameterization methods [24], [28]–[30]. Unrolled neural network methods have also been proposed, which unfold the iterative update steps and use neural networks to implicitly represent some modules [31]–[36]. Compared to denoising approaches, combining the neural network with iterative reconstruction framework takes the data consistency into consideration and can recover more image details. All of the above approaches require a large number of training pairs to learn the network parameters θ. Even for GAN, high-quality reference images are still needed in the discriminative network.

B. Train-data-free approaches

In this category, the high-quality training labels are not needed. The model is learned from the measurement data or prior images from other resources.

1). Adaptive dictionary learning

Dictionary learning is an effective method in image denoising [37]. It constructs an overcomplete dictionary and uses the estimated dictionary to approximate the distorted image. Similar to the pre-mentioned neural network method, high-quality reference images can be used to train a global dictionary. When high-quality reference images do not exist, adaptive dictionary learning can be applied where the training and denoising are implemented in an alternating iterative process. Apart from denoising, adaptive dictionary learning has also been applied to image reconstruction by combining the data fidelity item with the constraint from sparse representations [38]–[40].

2). Deep image prior

In [8], the authors proposed the DIP method, where no prior-learning was performed before applied to image restoration/denoising. The general idea is similar to the adaptive dictionary learning approach. Supposing ${\mathit{x}}_0\in {ℝ}^N$ is the distorted image, the training process is characterized as

$$\widehat{\mathit{\theta }}=\arg \underset{\mathit{\theta }}{\min }\Vert {\mathit{x}}_0-f(\mathit{\theta }|\mathit{z})\Vert ,\widehat{\mathit{x}}=f(\widehat{\mathit{\theta }}|\mathit{z}),$$ (2)

where $\widehat{\mathit{x}}\in {ℝ}^N$ is the final corrected image output, and the network input $\mathit{z}\in {ℝ}^N$ is random noise. No training pairs are needed and f(θ|z) is updated from scratch. Results from this work indicate that the network structure of a CNN can function as a regularizer and the network weights are the parameters representing $\widehat{\mathit{x}}$.

3). Kernel method

The kernel method is a method developed for PET image reconstruction using temporal or anatomical priors [5]. It is based on the assumption that the unknown image $\mathit{x}\in {ℝ}^N$ can be represented as

$$\mathit{x}={\mathit{K}}_{\mathit{z}}\mathit{\theta },$$ (3)

where ${\mathit{K}}_{\mathit{z}}\in {ℝ}^{N\times N}$ is the kernel matrix calculated from the prior image $\mathit{z}\in {ℝ}^N$ with a pre-defined kernel basis function. $\mathit{\theta }\in {ℝ}^N$ is the kernel coefficients. The kernel representation shown in (3) is included in the iterative reconstruction framework to estimate the kernel coefficients θ. From the network point of view, the kernel method can be considered as a twolayer network: the prior image z is the network input, K_z is a non-local feature extraction layer, θ is the convolutional filter with spatial size 1 × 1 and N feature channels, and x is the network output. The concept of non-local feature extraction layer is similar to the recently proposed non-local neural networks [41]. Thus, the kernel based iterative reconstruction can be treated as a two-layer neural network training process.

III. Methodology

A. Background

In inverse problems, such as image deblurring and image reconstruction, the measured data $\mathit{y}\in {ℝ}^M$ can be assumed as a collection of independent random variables and its mean $\overline{\mathit{y}}\in {ℝ}^M$ is assumed to be related to the original image $\mathit{x}\in {ℝ}^N$ through an affine transform

$$\overline{\mathit{y}}=\mathit{A}\mathit{x}+\mathit{s},$$ (4)

where $\mathit{A}\in {ℝ}^{M\times N}$ is the transformation matrix and $\mathit{s}\in {ℝ}^M$ is a known additive term. Supposing the measured random variable y_i follows a distribution of p(y_i|x), the log likelihood for the measured data y can be written as

$$L(\mathit{y}|\mathit{x})=\sum _{i=1}^M\log p\left(y_i|\mathit{x}\right).$$ (5)

B. Proposed framework

In this proposed DIPRecon framework, the unknown image x is represented as

$$\mathit{x}=f(\mathit{\theta }|\mathit{z}),$$ (6)

where f represents the neural network, θ are the unknown parameters of the neural network, z denotes the prior image and is the input to the neural network. When substituting x with the neural network representation in (6), the original data model shown in (4) can be rewritten as

$$\overline{\mathit{y}}=\mathit{A}f(\mathit{\theta }|\mathit{z})+\mathit{s}.$$ (7)

Replacing x by (6), we can express the log likelihood using θ as

$$L(\mathit{y}|\mathit{\theta })=\sum _{i=1}^M\log p(y_i|f(\mathit{\theta }|\mathit{z})).$$ (8)

The maximum likelihood estimate of the unknown image x can be calculated in two steps as

$$\widehat{\mathit{\theta }}=\arg \max L(\mathit{y}|\mathit{\theta }),$$ (9)

$$\widehat{\mathit{x}}=f(\widehat{\mathit{\theta }}|\mathit{z}).$$ (10)

C. Optimization

The objective function in (9) is difficult to solve due to the coupling between the likelihood function and the neural network. Here we transfer it to the constrained format as below

$$\begin{gathered} \max L(\mathit{y}|\mathit{x}) \\ \text{s.t.}\mathit{x}=f(\mathit{\theta }|\mathit{z}). \end{gathered}$$ (11)

We use the augmented Lagrangian format for the constrained optimization problem in (11) as

$$L_{\rho }=L(\mathit{y}|\mathit{x})-\frac{\rho }{2}\Vert \mathit{x}-f(\mathit{\theta }|\mathit{z})+\mathit{\mu }{\Vert }^2+\frac{\rho }{2}\Vert \mathit{\mu }{\Vert }^2,$$ (12)

which can be solved by the ADMM algorithm iteratively in three steps

$${\mathit{x}}^{n+1}=\underset{\mathit{x}}{\arg \max }L(\mathit{y}|\mathit{x})-\frac{\rho }{2}{\Vert \mathit{x}-f\left({\mathit{\theta }}^n|\mathit{z}\right)+{\mathit{\mu }}^n\Vert }^2,$$ (13)

$${\mathit{\theta }}^{n+1}=\underset{\mathit{\theta }}{\arg \min }{\Vert f(\mathit{\theta }|\mathit{z})-\left({\mathit{x}}^{n+1}+{\mathit{\mu }}^n\right)\Vert }^2,$$ (14)

$${\mathit{\mu }}^{n+1}={\mathit{\mu }}^n+{\mathit{x}}^{n+1}-f\left({\mathit{\theta }}^{n+1}|\mathit{z}\right).$$ (15)

1). Solving subproblem (13)

In this work, we use PET image reconstruction as an example. In PET image reconstruction, A is the detection probability matrix, with A_ij denoting the probability of photons originating from voxel j being detected by detector i [42]. $\mathit{s}\in {ℝ}^M$ denotes the expectation of scattered and random events. M is the number of lines of response (LOR). Assuming the measured photon coincidences follow Poisson distribution, the log-likelihood function L(y|x) can be explicitly written as

$$L(\mathit{y}|\mathit{x})=\sum _{i=1}^M\log p\left(y_i|\mathit{x}\right)=\sum _{i=1}^M{y}_i\log {\overline{y}}_i-{\overline{y}}_i-\log y_i!.$$ (16)

Though the measurement data may follow different distributions for other inverse problems, only subproblem (13) needs to be reformulated and the whole DIPRecon framework keeps unchanged. Here subproblem (13) is a penalized image reconstruction problem, and the optimization transfer method [43] was chosen to solve it. As x in L(y|x) is coupled together, we first construct a surrogate function Q_L(x|xⁿ) for L(y|x) to decouple the image pixels so that each pixel can be optimized independently. Q_L(x|xⁿ) is constructed as follows

$$Q_L\left(\mathit{x}|{\mathit{x}}^n\right)=\sum _{j=1}^{n_j}{A}_{.j}({\widehat{x}}_{j,\text{EM}}^{n+1}\log x_j-x_j),$$ (17)

where $A_{.j}={\sum }_{i=1}^M{A}_{ij}$ and ${\widehat{x}}_{j,\text{EM}}^{n+1}$ is calculated by

$${\widehat{x}}_{j,\text{EM}}^{n+1}=\frac{x_j^n}{A_{.j}}\sum _{i=1}^M{A}_{ij}\frac{y_i}{{\left[\mathit{A}{\mathit{x}}^n\right]}_i+s_i}.$$ (18)

It can be verified that the constructed surrogate function Q_L(x|xⁿ) fulfills the following two conditions:

$$Q_L\left(\mathit{x}|{\mathit{x}}^n\right)-Q_L\left({\mathit{x}}^n|{\mathit{x}}^n\right)\le L(\mathit{y}|\mathit{x})-L\left(\mathit{y}|{\mathit{x}}^n\right),$$ (19)

$$\nabla Q_L\left({\mathit{x}}^n|{\mathit{x}}^n\right)=\nabla L\left(\mathit{y}|{\mathit{x}}^n\right).$$ (20)

After getting this surrogate function, subproblem (13) can be optimized pixel by pixel. For pixel j, the surrogate objective function for subproblem (13) is

$$\begin{gathered} P\left(x_j|{\mathit{x}}^n\right)=A_{.j}\left({\widehat{x}}_{j,\text{EM}}^{n+1}\log x_j-x_j\right) \\ -\frac{\rho }{2}{\left[x_j-f{(\mathit{\theta }|\mathit{z})}_j^n+{\mu }_j^n\right]}^2. \end{gathered}$$ (21)

The final update equation for pixel j after maximizing (21) is

$$\begin{gathered} x_j^{n+1}=\frac{1}{2}\left[f{\left({\mathit{\theta }}^n|\mathit{z}\right)}_j-{\mu }_j^n-A_{.j}/\rho \right] \\ +\frac{1}{2}\sqrt{{\left[f{\left({\mathit{\theta }}^n|\mathit{z}\right)}_j-{\mu }_j^n-A_{.j}/\rho \right]}^2+4{\widehat{x}}_{j,\text{EM}}^{n+1}{A}_{.j}/\rho }. \end{gathered}$$ (22)

2). Solving subproblem (14)

Subproblem (14) is a nonlinear least square problem and it has the same format as the network training problem shown in (1), with x_label replaced by xⁿ⁺¹ + μⁿ. Currently network training is mostly based on first order methods, such as the Adam algorithm [44] and the Nesterov’s accelerated gradient (NAG) algorithm [45]. The L-BFGS algorithm is a quasi-newton method, combining a history of updates to approximate the Hessian matrix. It is not widely used in network training as it requires large batch size to accurately calculate the descent direction, which is less effective than first order methods for large-scale applications. In this proposed framework, as only the patient’s own prior images are employed as the network input, the data size is much smaller than the traditional network training. In our case, the L-BFGS method is preferred to solve subproblem (14) due to its stability and better performance observed

Algorithm 1.

Algorithm for the proposed DIPRecon method.

Input: Maximum iteration number MaxIt, sub-iteration number SubIt1, sub-iteration number SubIt2, network initialization θ0, Prior image z

1: x0,SubIt1 = f(θ0|z)

2: μ0 = 0

3: for n = 1 to MaxIt do

4:  xn,0 = xn−1,SubIt1

5:  for m = 1 to SubIt1 do

6:   ${\widehat{x}}_{j,\text{EM}}^{n,m}=\left[x_j^{n,m-1}/A_{.j}\right]{\sum }_{i=1}^M{A}_{ij}\frac{y_i}{{\left[\mathit{A}{\mathit{x}}^{n,m-1}\right]}_i+s_i},$ where $A{.}_j={\sum }_{i=1}^M{A}_{ij}$

7:   $$\begin{gathered} x_j^{n,m}=\frac{1}{2}\left[f{\left({\mathit{\theta }}^{n-1}|\mathit{z}\right)}_j-{\mu }_j^{n-1}-A_{.j}/\rho \right] \\ +\frac{1}{2}\sqrt{{\left[f{\left({\mathit{\theta }}^{n-1}|\mathit{z}\right)}_j-{\mu }_j^n-A_{.j}/\rho \right]}^2+4{\widehat{x}}_{j,\text{EM}}^{n,m}{A}_{.j}/\rho } \end{gathered}$$

8:  end for

9:  ${\mathit{x}}_{\text{label}}^n={\mathit{x}}^{n,\text{SubIt}1}+{\mathit{\mu }}^{n-1}$

10:  Running L-BFGS algorithm SubIt2 iterations to train the network, get ${\mathit{\theta }}^n=\arg {\min }_{\mathit{\theta }}{\Vert f(\mathit{\theta }|\mathit{z})-{\mathit{x}}_{\text{label}}^n\Vert }^2$

11:  μn = μn−1 + xn,SubIt1 − f(θn|z)

12: end for

13: return $\widehat{\mathit{x}}=f\left({\mathit{\theta }}^{\mathrm{MaxIt}}|\mathit{z}\right)$

in the experiments. The comparing results are presented in Section. V-A.

In the iterative framework, we run two iterations to solve subproblem (13) and ten iterations to solve subproblem (14). The overall algorithm flowchart is presented in Algorithm 1.

D. Network structure

The network structure employed in this work is based on the modified 3D U-net [13]. The overall network architecture is summarized in Fig. 1. It consists of repetitive applications of 1) 3×3×3 3D convolutional layer, 2) batch normalization (BN) layer, 3) Leaky rectified linear unit (LReLU) layer, 4) 3 × 3 × 3 3D convolutional layer with stride 2 × 2 × 2 for down-sampling, 5) 2 × 2 × 2 bilinear interpolation layer for up-sampling, and 6) identity mapping layer that adds features from left-side encoder path to the right-side decoder path. In our implementation, there are several modifications compared to the original 3D U-net:

1. using convolutional layer with stride 2 to down-sample the image instead of using max pooling, to construct a fully convolutional network;

2. using skip connections to link the encoder path and decoder path instead of concatenating, to reduce the number of training parameters;

3. using the bi-linear interpolation instead of the deconvolution upsampling, to reduce the checkboard artifact;

4. using leaky ReLU instead of ReLU.

Through the experiments it was found that using 3D convolution for 3D image reconstruction is better than using 2D convolution with multiple axial slices in the input channels. To enable the non-negative constraint on the reconstructed image, a ReLU layer was added before the output. In addition, we found that compared to the network without encoder and decoder path, the U-net structure could save more GPU memory because the spatial size is reduced due to the encoder path.

Fig. 1:

The schematic plot of the Network structure used in this work. The spatial input size for each layer is based on the real data experiment described in IV-B.

E. Implementation details and reference methods

To stabilize the network training, before we run the networking training, the intensity of PET images were scaled to the range [0,1]. Due to the non-convexity of neural network training, it is better to assign good initials to the network parameters before trained inside the iterative reconstruction loop. Comparisons between the results with and without pretraining are shown in the supplemental materials. In our implementation, we first ran ML EM algorithm for 60 iterations, and then used it as x_label to train the network based on (1) with MR images as network input. This pre-training was run 300 epochs using the L-BFGS algorithm based on minimizing (1). As the penalty parameter has a large impact on the convergence speed, we examined the convergence of the log-likelihood $L(\mathit{y}|f(\widehat{\mathit{\theta }}|\mathit{z}))$ to determine the penalty parameter used in practice. As an example, Fig. 2 shows the log-likelihood curve for different penalty parameter ρ for the simulation study mentioned in Section. IV-A. Considering the convergence speed and stability of the likelihood, ρ = 3×10⁻³ was chosen.

Fig. 2:

The effect of penalty parameter ρ on the likelihood $L(\mathit{y}|f(\widehat{\mathit{\theta }}|\mathit{z}))$ based on the simulation data set described in Section IV-A.

We compared the DIPRecon method with the Gaussian postfiltering method, the penalized reconstruction method based on including a pre-trained neural network in the penalty [24], and the kernel method [5]. One popular and intuitive way of incorporating the trained network into iterative reconstruction framework is include it in the penalty item as first shown in [24]. The objective function is

$$\widehat{\mathit{x}}=\underset{\mathit{x}}{\arg \max }L(\mathit{y}|\mathit{x})-\beta {\Vert \mathit{x}-f\left({\mathit{\theta }}_0|\mathit{z}\right)\Vert }^2.$$ (23)

Here f(θ₀|z) is the pre-trained network using noisy PET images as labels and the pre-trained network parameters θ₀ keeps fixed during the reconstruction process. This method is denoted as CNNPenalty method. Interestingly, the objective function shown in (23) becomes the same as subproblem (13) by replacing f(θ₀|z) with f(θⁿ|z)+μⁿ. In our implementation, the difference between the CNNPenalty method and the proposed DIPRecon method is that μ was fixed to 0 and the network updating step (14) was skipped. By this setting, we can understand the effects of updating the network parameters inside the reconstruction loop. For the kernel method, the (i, j)th element of the kernel matrix K_z is

$$k_{ij}=\exp \left(-\frac{{\Vert {\mathit{f}}_i-{\mathit{f}}_j\Vert }^2}{2N_f{\sigma }^2}\right),$$ (24)

where ${\mathit{f}}_i\in {ℝ}^{N_f}$ and ${\mathit{f}}_j\in {ℝ}^{N_f}$ represents the feature vectors of voxel i and voxel j from the MR prior image z, σ² is the variance of the prior image and N_f is the number of voxels in a feature vector. For efficient computation, the kernel matrix was constructed using a K-Nearest-Neighbor (KNN) search in a 7×7×7 search window with K = 50. A 3×3×3 local patch was extracted for each voxel to form the feature vector. During image reconstruction, the linear coefficients θ were computed using iterative update as [46]

$${\mathit{\theta }}^{n+1}=\frac{{\mathit{\theta }}^n}{{\left(\mathit{A}{\mathit{K}}_{\mathit{z}}\right)}^T{\mathbf{1}}_M}\left[{\left(\mathit{A}{\mathit{K}}_{\mathit{z}}\right)}^T\frac{\mathit{y}}{\left(\mathit{A}{\mathit{K}}_{\mathit{z}}\right){\mathit{\theta }}^n+\mathit{s}}\right],$$ (25)

and the final output image is $\widehat{\mathit{x}}={\mathit{K}}_{\mathit{z}}\widehat{\mathit{\theta }}$. The kernel method is denoted as KMRI in later comparisons.

Currently the code is not optimized and the network training was done in GPU (NVIDIA GTX 1080 Ti) and the image reconstruction was implemented in CPU with 16 threads (Intel Xeon E5–2630 v3). The neural network was implemented using TensorFlow 1.4. The LBFGS algorithm was implemented using the TensorFlow “ScipyOptimizerInterface” to call “L-BFGS-B” method from scipy libary running in CPU mode. 10 previous iterations were used in L-BFGS-B to approximate the Hessian matrix. For the simulation setting, the CPU memory usage for L-BFGS is 2GB and for the real data setting, the CPU memory usage is 7 GB. For the simulation study described in Section. IV-A, the running time is 20 seconds per iteration for the EM method, 23 seconds per iteration for the CNNPenalty method, 30 seconds per iteration for the KMRI method (averaging the kernel matrix calculation time), and 78 seconds per iteration for the DIPRecon method. If the L-BFGS is implemented in GPU mode, the computation time for DIPRecon can be further reduced.

IV. Experimental setup

A. Brain phantom simulation

A 3D brain phantom from BrainWeb [47] was used in the simulation. Corresponding T1 weighted MR image was used as the prior image. The voxel size is 2×2×2 mm³ and the phantom image size is 128×128×105. The input to the network was cropped to 128×128×96 to reduce GPU memory usage. To simulate mismatches between the MR and PET images, twelve hot spheres of diameter 16 mm were inserted into the PET image as tumor regions, which are not visible in the MR image. The time activity curves of blood, gray matter, white matter and tumor were simulated based on a two-tissue compartment model, with kinetic parameters the same as those used in [48]. The computer simulation modeled the geometry of a Siemens mCT scanner [49], and the system matrix was generated using the multi-ray tracing method [50]. In this experiment, the last 5 min frame of a one-hour FDG scan was used as the ground-truth image. Noise-free sinogram data were generated by forward-projecting the ground-truth image using the system matrix and the attenuation map. Poisson noise was then introduced to the noise-free data by setting the total count level to be equivalent to last 5 min scan with 5 mCi injection. Uniform random events were simulated and accounted for 30 percent of the noise-free data. Scatters were not included. For quantitative comparison, contrast recovery coefficient (CRC) vs. the standard deviation (STD) curves were plotted based on reconstructions of twenty independent and identically distributed (i.i.d) realizations. The CRC was computed between selected gray matter regions and white matter regions as

$$\text{CRC}=\frac{1}{R}\sum _{r=1}^R\left(\frac{{\overline{a}}_r}{{\overline{b}}_r}-1\right)/\left(\frac{a^{\text{true}}}{b^{\text{true}}}-1\right).$$ (26)

Here R is the number of realizations and is set to 20, ${\overline{a}}_r=1/K_a{\sum }_{k=1}^{K_a}{a}_{r,k}$ is the average uptake of the gray matter over K_a ROIs in realization r. The ROIs were drawn in both matched gray matter regions and the tumor regions. For the case of matched gray matter, K_a = 10. For the tumor regions, K_a = 12. When choosing the matched gray matter regions, only those pixels inside the predefined 20mm-diameter spheres and containing 80% of gray matter were included. ${\overline{b}}_r=1/K_b{\sum }_{k=1}^{K_b}{b}_{r,k}$ is the average value of the background ROIs in realization r, and K_b = 37 is the total number of background ROIs. The background ROIs were drawn in the white matter. The background STD was computed as

$$\text{STD}=\frac{1}{K_b}\sum _{k=1}^{K_b}\frac{\sqrt{\frac{1}{R-1}{\sum }_{r=1}^R{\left(b_{r,k}-{\overline{b}}_k\right)}^2}}{{\overline{b}}_k},$$ (27)

where ${\overline{b}}_k=1/R{\sum }_{r=1}^R{b}_{r,k}$ is the average of the background ROI means over realizations.

The network structure for the simulated data set is the same as the network structure shown in Fig. 1, except input and output size are 128 × 128 × 96.

B. Real brain data sets

A 70-minutes dynamic PET scan of a human subject acquired on a Siemens Brain MR-PET scanner after 5 mCi FDG injection was employed in the real data evaluation. The data were reconstructed with an image array of 256×256×153 and a voxel size of 1.25 × 1.25 × 1.25 mm³. The input to the network was cropped to 192×192×128 to reduce GPU memory usage. A simultaneous acquired T1-weighted MR image having the same image array and voxel size as the PET image was used as the prior image. Correction factors for randoms, scatters were estimated using the standard software provided by the manufacturer and included during reconstruction. The motion correction was performed in the line-of-response (LOR) space based on the simultaneously acquired MR navigator signal [51]. Attenuation was derived from T1-weighted MR image using the SPM based atlas method [52]. To generate multiple realizations for quantitative analysis, the last 40 minutes PET data were binned together and resampled with a 1/8 ratio to obtain 20 i.i.d. datasets that mimic 5-minutes frames. As the ground truth of the regional uptake is unknown, a hot sphere with diameter 12.5 mm, mimicking a tumor, was added to the PET data (invisible in the MRI image). It simulates the case where MRI and PET information does not match. The TAC of the hot sphere was set to the TAC of the gray matter, so the final TAC of the simulated tumor region is higher than that of the gray matter because of the superposition. The simulated tumor image of the last 40 minutes was forward-projected to generate a set of noise-free sinograms, including detector normalization and patient attenuation. Randoms and scatters from the inserted tumor were not simulated as they would be negligible compared with the scattered and random events from the patient background. Poisson noise was then introduced and finally the tumor sinograms were added to the original patient sinograms to generate the hybrid real data sets. For tumor quantification, images with and without the inserted tumor were reconstructed and the difference was taken to obtain the tumor only image and compared with the ground truth. The tumor contrast recovery (CR) was calculated as

$$\text{CR}=\frac{1}{R}\sum _{r=1}^R{\overline{l}}_r/l_{\text{true}},$$ (28)

where ${\overline{l}}_r$ is the mean tumor uptake inside the tumor ROI, l_true is the ground truth of the tumor uptake, and R is the number of the realizations. For the background, 11 circular ROIs with a diameter of 12.5 mm were drawn in the white matter and the standard deviation was calculated according to (27). The network structure for the real data set is shown in Fig. 1.

V. Results

A. Effect of network settings

To test the effectiveness of the conditional DIP framework, an experiment was performed by using either the uniform random noise or the patient’s MR prior image as the network input. ML EM reconstruction of the real brain data at 60^th iteration was treated as the label image. 300 epochs were run for network training using the L-BFGS algorithm. Fig. 3(a,b) shows one coronal view of the network output. When the input is random noise, the image is smooth, but some cortex structures cannot be recovered. When the input is MR image, more cortex structures show up. This comparison demonstrates the benefits of including the patient’s prior image as the network input. To show the benefit of including the neural network into reconstruction framework, compared to image denoising as shown in the original DIP paper [8], we have also presented the reconstructed image using the proposed DIPRecon framework in Fig. 3(c). Clearly, DIPRecon can recover more cortex details and generate higher contrast between the white matter and gray matter as compared to the original DIP framework.

Fig. 3:

(a,b) One coronal view of the network output according to (a) the DIP framework using random noise as network input, (b) conditional DIP framework using the Patient’s own MR image as network input and (c) the proposed DIPRecon framework using the Patient’s MR image as network input. Images shown are based on the real data set described in Section IV-B.

We also compared the behaviors of different optimization algorithms under the conditional DIP framework. The Adam, NAG, and L-BFGS algorithms were compared regarding the pre-training process using the real brain data. When comparing different algorithms, we computed the normalized cost value, which is defined as

$$L_n=\frac{{\varphi }_{\text{Adam}}^{\text{ref}}-{\varphi }^n}{{\varphi }_{\text{Adam}}^{\text{ref}}-{\varphi }_{\text{Adam}}^1},$$ (29)

where ${\varphi }_{\text{Adam}}^{\text{ref}}$ and ${\varphi }_{\text{Adam}}^1$ is the cost value defined in (1) after running Adam for 700 iterations and 1 iteration, respectively. Fig. 4 plots the normalized cost value curves for different algorithms. The L-BFGS algorithm is monotonic decreasing while the Adam algorithm is not due to the adaptive learning rate implemented. The NAG algorithm is slower than the other two algorithms. The reason why L-BFGS is faster is due to its using the approximated Hessian matrix, which makes it closer to a second-order optimization algorithm, while both the Adam and NAG methods are first-order methods. The monotonic property is another good advantage of L-BFGS algorithm. Due to the monotonic property, the network output using the L-BFGS algorithm is more stable and less influenced by the image noise when running multiple realizations. Faster convergence speed and better quantitative results are the reasons we chose L-BFGS algorithm to solve subproblem (14) and perform the initial network training.

Fig. 4:

Comparison of the normalized likelihood for the Adam, Nesterov’s accelerated gradient (NAG) and L-BFGS algorithms based on the real brain data set described in Section IV-B.

B. Simulation results

Fig. 5 shows three orthogonal views of the reconstructed images using different methods for the simulated dataset. The kernel method and the DIPRecon method both reveal more cortex structures and have lower noise compared to the EM-plus-filter method and the CNNPenalty method. Compared with the kernel method, the proposed DIPRecon method can recover even more details of the cortices and the white matter regions are cleaner. Furthermore, compared to the kernel method, the tumor uptake using DIPRecon is higher and the tumor shape is closer to the ground truth. In this simulation set-up, there are no tumor signals in the prior MRI image, and the DIPRecon method can still recover PET signals, which is a sign of robustness to potential mismatches between PET and prior images. Besides, by comparing the CNNPenalty method and the proposed DIPRecon method, we can also see that updating the network parameters inside the reconstruction loop can recover more brain structures and reduce the image noise. Fig. 6 shows the CRC-STD curves for different methods. For both the gray matter region and the tumor region, we can see that at fixed STD, the CRC of the DIPRecon method is higher than other methods. This observation quantitatively shows that the DIPRecon method out-performs other methods.

Fig. 5:

Three orthogonal slices of the reconstructed image using different methods for the simulation dataset. The iteration number is 100 for all methods. The first column is the corresponding MR prior image.

Fig. 6:

CRC-STD curves at the gray matter region (left) and the tumor region (right) for the reconstruction using the simulation dataset. Markers are plotted every ten iterations. The lowest point corresponding to the 20^th iteration.

C. Real data results

Fig. 7 shows the reconstructed images and the corresponding MR prior images of the real brain dataset with inserted lesion using different methods. Reconstructed images from the real dataset without inserted lesion are shown in the supplemental material. The high-count images were reconstructed from the combined 40-min scanning for reference. Compared to the EM-plus-filter method and the CNNPenalty method, the kernel method and the DIPRecon method can recover more cortex details and the image noise in the white matter is much reduced. The cortex shape using the DIPRecon method is clearer than the kernel method. For the tumor region which is unobserved in the MR image, the uptake is higher in the DIPRecon method compared with the kernel method. Fig. 8 shows the CR-STD curves for different methods. Clearly the DIPRecon method has the best CR-STD trade-off compared with the other methods.

Fig. 7:

First three rows show three orthogonal slices through the simulated-tumor region for the real brain dataset. Arrows in the high-count image (40 min scanning) indicate the simulated tumor. The last row shows one coronal view which contains more cortex structures. The first column is the corresponding MR prior image. The iteration number is 100 for EM-plus-filter and the proposed DIPRecon method, 50 for the CNNPenalty method, and is 120 for the KMRI method.

Fig. 8:

The CR-STD plot of the tumor uptake in the real brain data set. Markers are generated for every ten iterations for CNNPenalty method and every twenty iterations for other methods. For EM-plus-filter and DIPRecon method, the lowest point corresponding to the 40^th iteration. For KMRI method, the lowest point corresponding to the 60^th iteration. For CNNPenalty method, the lowest point corresponding to the 10^th iteration.

VI. Discussion

The number of training variables for the U-net structure implemented in this work is around 1.5 million. For both simulation and real data sets, the number of voxels for the network input is more than 1.5 million. Compared to the traditional iterative reconstruction, the number of unknowns is reduced in our proposed framework. During our implementation, we replaced the feature concatenation with feature adding to reduce the network training parameters, performed bi-linear interpolation to replace the deconvolution upsampling, and finally used the L-BFGS algorithm to train the network. All these three changes improved the network performance in our experiments. Finding a network structure with less trainable parameters but stronger representation power is our on-going work.

In this paper we used the PET image reconstruction as an example to demonstrate the effectiveness of the proposed DIPRecon framework. Subproblem (13) was solved using the optimization transfer method, and subproblem (14) was treated as the traditional network training with MSE-based loss function. This proposed framework can also be applied to other image reconstruction problems where a patient-specific image prior is available. All we need to change is the imaging model and likelihood function of the measured data. Here the initial network training was performed based on one single patient dataset. We expect that if the network was pre-trained from other patient data sets, and later on fine-tuned in the DIPRecon framework, the results can be better as the variables in this population-based initial network is more robustly estimated.

Benefits of employing patient’s prior images as network input can be observed comparing Fig. 3(a) and Fig. 3(b). Currently we only use one T1-weighted MR as the anatomical prior. More prior information, such as multiple MR images using different sequences and the temporal information from dynamic PET, can be included by adding more input channels. As no pre-defined weighting is needed when combing multiple priors, this proposed DIPRecon framework can make use of the patient’s prior information more efficiently. Both KMRI and DIPRecon are constructed based on representation unknown image x using the kernel matrix or CNN. We have calculated likelihood values for the EM, KMRI and DIPRecon methods regarding images shown in Fig. 5. From high to low, the likelihood values are −7.9588×10⁷ (EM), −7.9649×10⁷ (DIPRecon) and −7.9654 × 10⁷ (KMRI). We can see that DIPRecon method can better fit the data compared with the KMRI method, which is based on the strong approximation ability of CNN.

As for the ADMM algorithm used, there is no convergence theory due to non-convexity of the neural network. The reason to use ADMM is to de-couple the image update step and the network training step. The network training step ( subproblem 14) needs more update steps than the iterative reconstruction step (subproblem 13). If they are coupled together, every time updating the network parameters, we need to compute the forward and backward projections. As the computation of projections are very time-consuming for fully-3D PET, decoupling these two steps will save more computational time. Furthermore, after using the ADMM algorithm, subproblem 13 is a traditional penalized reconstruction and can reuse current image reconstruction packages, which makes the DIPRecon method adaptable to current iterative frameworks. Compared to the DIP framework, the proposed DIPRecon framework uses raw sinogram data and the objective function is constructed from imaging physics, which exploits more useful information compared with using noisy images as the training labels. This can be observed comparing Fig. 3(b) and Fig. 3(c).

Compared to the simulation results shown in Fig. 5, the real data results of the DIPRecon method shown in Fig. 7 are a little blurred. Two possible reasons are: (1) the number of voxels for the real data is larger than the simulation study and more iterations are needed to achieve the similar sharpness as the simulation data; (2) in the simulation, the MR boundary are perfectly matched with PET images. For real datasets, there might be boundary mismatches between T1 MR and PET images. Currently we only tested our methods on simultaneous PET-MR scanning, but does not perform experiments based on data sets acquired during different scans, in which cases large registration errors might arise. One possible solution is to couple a registration network, such as the spatial transformer network [53], to the U-net to perform additional registration. Testing and adjusting the proposed framework regarding large registration errors are our future research direction. Finally, our evaluations are based on FDG tracers, and the FDG brain uptake has similar patterns as the T1 MR images. Further evaluations based on data sets with other tracers are needed to test the robustness of the proposed framework.

VII. Conclusion

In this work, we propose a framework to include the personalized deep neural network into the iterative reconstruction method. Prior training image pairs are not needed in this process, but only the patient’s own prior images. PET image reconstruction was employed to demonstrate the effectiveness of the proposed DIPRecon framework. Both simulation and real brain data sets show that this proposed personalized medical imaging reconstruction framework performs better than the Gaussian filter, the penalized reconstruction using a neural network penalty, and the kernel method, in terms of contrast recovery vs. noise trade-offs. Future work will focus on finding better network structures, evaluations using more clinical data sets with different tracers, and testing the robustness of the proposed method to registration errors.