Artificial Neural Network Enhanced Bayesian PET Image Reconstruction

Abstract

In PET image reconstruction, the Bayesian framework with various regularization terms has been implemented to constrain the radio tracer distribution. Varying the regularizing weight of a maximum a posteriori (MAP) algorithm specifies a lower bound of the tradeoff between variance and spatial resolution measured from the reconstructed images. The purpose of this study is to build a patch-based image enhancement scheme to reduce the size of the unachievable region below the bound and thus to quantitatively improve the Bayesian PET imaging. We cast the proposed enhancement as a regression problem which models a highly nonlinear and spatial-varying mapping between the reconstructed image patches and an enhanced image patch. An artificial neural network (ANN) model named multilayer perceptron (MLP) with backpropagation was used to solve this regression problem through learning from examples. Using the BrainWeb phantoms, we simulated brain PET data at different count levels of different subjects with and without lesions. The MLP was trained using the image patches reconstructed with a MAP algorithm of different regularization parameters for one normal subject at a certain count level. To evaluate the performance of the trained MLP, reconstructed images from other simulations and two patient brain PET imaging datasets were processed. In every testing cases, we demonstrate that the MLP enhancement technique improves the noise and bias tradeoff compared with the MAP reconstruction using different regularizing weights thus decreasing the size of the unachievable region defined by the MAP algorithm in the variance/resolution plane.

I. Introduction

Positron emission tomography (PET) imaging is the leading example of molecular imaging procedures enabling quantitative measurements of physiological and biochemical processes in vivo. In order to obtain accurate estimation of the radiotracer distribution in an object from the detected emission photons, various image reconstruction methods have been proposed, either based on analytical or iterative approaches [1]. Analytical reconstruction methods use a simplified imaging model assuming that the PET data is deterministic without statistical noise. In contrast, iterative reconstruction methods modeling the uncertainties associated with several aspects of PET physics yield improved performance over analytical methods. Describing the data probability distribution with the Poisson model [2], [3], statistical iterative methods are now applied in commercial scanners for clinical studies. The expectation-maximization (EM) algorithm seeking the maximum likelihood (ML) solution provides an asymptotically unbiased estimator of the true activity distribution [3]. However, achieving the low-bias estimate requires a large number of iteration steps, which inevitably increases the estimate variance and results in highly noisy images [4], [5]. This degradation in the reconstructed images affects qualitative interpretation in a clinical context.

To suppress the undesirable noise propagation and to generate smooth images for clinical usage, the natural way is to stop the iteration process early using criteria based on statistical hypothesis tests or cross-validation [6]- [9] and postfilter the ML EM reconstructed images afterward [10], [11]. Alternative approaches to avoid noise amplification are offered by maximum a posteriori (MAP) reconstruction methods. In the MAP reconstruction process, various regularization terms are incorporated in the Bayesian framework to constrain the radiotracer distribution [12]-[20]. In particular, higher resolution anatomical information measured from other imaging modalities such as magnetic resonance imaging (MRI) or computed tomography (CT) has been used to define the prior function for MAP reconstruction [21]-[29].

It is customary to assess the quantitative accuracy of the MAP reconstructed PET images in terms of the inherent tradeoff between their variance and spatial resolution [30]. Assuming that the iterative algorithm seeking the MAP solution converges to a unique and stable point, we can compute the image quality metrics at the point of convergence, independent of the iteration number [31], [32]. Based on the assumption, Hero et al. derived asymptotic formulas for variance and spatial resolution of the quadratic penalized MAP reconstruction [33]. As the convergence point of a quadratic MAP reconstruction depends on the choice of the regularization parameter, varying the regularizing weight in the parametric descriptions of variance and resolution traces out a curve in the variance/resolution plane. Hero et al. showed that this curve specified a bound of the variance/resolution tradeoff since lower variance can only be bought at the price of coarser resolution and vice versa. The region below the curve defines an “unachievable region” where the MAP estimator can not reach [33]. The goal of our study is to develop a patch-based image enhancement technique for the MAP reconstructed PET images using an artificial neural network (ANN). This technique is expected to enhance the MAP reconstruction from different weighting parameters to lower the bound of the variance/resolution tradeoff and thus to decrease the size of the unachievable region in the variance/resolution plane.

Taking advantage of their learning and generalization capabilities, ANNs are impressive information processing systems. They have attracted a lot of attention and are very successful in solving image processing problems [34]- [38]. A feedforward ANN with supervised learning can adaptively approach any non-linear mapping function, which models the complex relationships between inputs and outputs [39]. Freeman et al. and Dong et al. developed image super-resolution algorithms based on the feedforward ANNs through learning a mapping between low- and high-resolution image patches [40], [41]. Their algorithms estimate missing high-resolution details that are not present in the low-resolution images. Feedforward ANNs have also been applied for natural and medical image enhancement. Madani et al. proposed to use the radial basis function (RBF) network, which is an ANN that uses RBF as an activation function, to achieve noise reduction, feature enhancement, and removal of inhomogeneous background for the degraded images [42]. The multilayer perceptron (MLP), an ANN with all layers fully-connected, was used by Burger et al. to remove noise while preserving the edges and fine details of the original images [43]. A similar image enhancement idea was applied by Boublil et al. to improve the tradeoff between noise and bias of the reconstructed CT images [44].

Despite different architectures, the ANNs used in these methods construct an end-to-end mapping from corrupted (noisy/blurry) image patches to the clean image patches. In detail, an ANN is trained to memorize small patches of a corrupted image and associate each patch with the corrected pixel values, which are those of the corresponding patch from the clean image. After the learning phase, all the patches of the unlearned corrupted image are compared one after another with the memorized examples. For each patch of the unlearned corrupted image, it is replaced with the corrected pixel values of the closest example from the training database. When we look at images through small windows, there exist different kinds of shapes that are not identifiable due to their proximity and high gradient. The number of existing shapes that can be seen by human eyes is limited [42]. The ANN trained with as many shapes as possible that exist in an image can identify the similar shapes from other images not related to the learned one.

In this paper, we propose to develop an MLP model for the purpose of enhancing MAP reconstructed PET images. The MLP model will be used to serve as a nonlinear multivariate regression function that forms a mapping from image patches in the MAP reconstructed images to the corresponding patches in the expectantly enhanced image. In our previous study on dictionary learning (DL) based PET image reconstruction, DL was used to efficiently capture local image features from patches of a brain PET image after which sparse coding (SC) was applied to represent each patch as a linear combination of the learned image features [20]. We demonstrated that the dictionary learned from image patches of one simulated subject can be used to represent the image patches of other simulated subjects and, more importantly, of various patient images. The DL followed by SC procedure forms a patch-to-patch mapping as what an MLP also achieves. Therefore, we expect that the MLP model to be learned from the simulated patches of one subject image can also be applied to the patches of other subject images and of patient PET images as well.

This paper is organized as follows: In Section II, we describe the MLP-based MAP reconstruction enhancement method. In Section III the proposed method is implemented on PET images reconstructed from various simulated brain PET data and two patient PET datasets. The results of the quantitative evaluations for simulation and patient studies are presented in Section IV. We further discuss the results in Section V, and the conclusion of this work is formulated in Section VI.

II. Methods

We first describe a MAP EM image reconstruction algorithm that assumes the Gibbs prior [12] with the commonly used Green ‘smoothness’ energy function [16]. Then we introduce the three steps to achieve the MAP reconstruction enhancement by solving a nonlinear and spatial-varying mapping based on the MLP model. The implementation setup for the proposed enhancement scheme are presented at the end of this section.

A. MAP Reconstruction

For PET image reconstruction, the measured emission sinogram data g is modeled as a collection of independent Poisson random variables {g_i}, with i as the individual bin number. The ML estimate of the tracer activity distribution $\widehat{\mathbf{f}}$ is the solution that maximizes the log likelihood function

$$\log P\left(\mathbf{g}|\mathbf{f}\right)={\displaystyle \sum _i{g}_i\log \left({\overline{g}}_i\right)-{\overline{g}}_i-\log \left(g_i!\right)},$$ (1)

where ${\overline{g}}_i$ is the expected value of g_i and P(·) computes the probability of the variable enclosed in parentheses [2].

From the statistical perspective, the activity image can also be estimated by maximizing a posteriori of the function P (f|g). According to the Bayesian rule, we have

$$P\left(\mathbf{f}|\mathbf{g}\right)=\frac{P\left(\mathbf{g}|\mathbf{f}\right)P\left(\mathbf{f}\right)}{P\left(\mathbf{g}\right)},$$ (2)

where P(f) carries the prior knowledge of the activity image serving as the regularization in a MAP reconstruction algorithm. The general form of the prior follows Gibbs distribution

$$P\left(\mathbf{f}\right)=\exp \left(-\beta V\left(\mathbf{f}\right)\right)/W,$$ (3)

in which W is a normalization factor, and βV(f) is the weighted Gibbs energy function [12].

Seeking the solution to maximize (2) is equivalent to finding $\widehat{\mathbf{f}}$ that maximizes the log-posterior probability

$$\log P\left(\mathbf{g}|\mathbf{f}\right)-\beta V\left(\mathbf{f}\right)+k,$$ (4)

in which the constant k represents the contribution of the W term in (3) and the denominator in (2). Applying the Poisson statistics to the first term in (4) as in (1), the iterative procedure is derived as what was done in [16]:

$$f_j^{\mathrm{new}}=\frac{f_j^{\mathrm{old}}}{{\displaystyle {\sum }_i{c}_{ji}+\beta \frac{\partial V\left(\mathbf{f}\right)}{\partial f_j}|f_j=f_j^{\mathrm{old}}}}{\displaystyle \sum _i\frac{c_{ji}{g}_i}{{\displaystyle {\sum }_j{c}_{ji}{f}_j^{\mathrm{old}}}},}$$ (5)

where the new estimate of voxel j in the activity image is updated from the old estimate and c_ji represents an element of the projection matrix modeling the contribution of voxel j to projection bin i.

In this work, we study the enhancement of the MAP reconstruction with the commonly used Green ‘smoothness’ prior, which penalizes inter-voxel intensity variations of the reconstructed PET image

$$V\left(\mathbf{f}\right)={\displaystyle \sum _j{\displaystyle \sum _{k\in N_j}{w}_{jk}\log \cosh \left(\frac{f_j-f_k}{\delta }\right)},}$$ (6)

where w_jk is a weight reflecting the closeness between voxel j and voxel k in the neighborhood N_j, and δ is a scaling factor [16]. The weight w_jk is set to 1 if j and k are orthogonal nearest neighbors, to $\sqrt{1/2}$ for diagonal neighbors, and to 0 otherwise. The parameter δ is set to 1/20 of the maximum intensity of the updated PET image in each reconstruction iteration. It is chosen empirically based on the experience from working with patient data.

It is worth mentioning that the proposed enhancement method is not limited to this specific MAP algorithm. It is open to applications to images reconstructed by other MAP algorithms with different types of regularization. On a side note, varying the cutoff frequency leads to different variance/resolution tradeoffs from post-filtered ML reconstruction. The MLP enhancement is applicable to filtered ML reconstructed images as well.

B. MLP-based MAP Reconstruction Enhancement Scheme

Assume that there exists an intrinsic mapping function that maps image patches in the MAP reconstructed images to the corresponding patches in the expectantly enhanced image. The goal of our MAP reconstruction enhancement scheme is to train an approximate function using the MAP reconstructed images and the reference image i.e. the true phantom activity image so that this function can be applied to enhance the unlearned MAP reconstruction. This highly non-linear and spatial-varying mapping function can be approached by the MLP, one of the most commonly used feedforward ANN model [45], through learning from the patches extracted from the training images.

Capturing useful features from the training image patches is essential for the MLP to associate a new image patch with the closest learned example and then compute the desired output. MAP reconstructed images of different weighting parameters carry different tradeoffs of noise and bias, which would provide different structural features for generalizing the MLP model. Instead of training the MLP from image patches corresponding to a specific weighting parameter, we propose to pair each reference image patch with a set of patches reconstructed with different regularizing weights. Following the MLP learning phase to build the mapping function, image patches of the unlearned MAP reconstruction can be processed by the MLP to yield the corresponding output image patches with corrected intensity values.

Let $E=\langle {\left\{{\mathbf{f}}^k\right\}}_{k=1}^K,\mathbf{u}\rangle$ be an image pair, where ${\left\{{\mathbf{f}}^k\right\}}_{k=1}^K$ are images reconstructed using the MAP algorithm with different regularizing weights and u is the corresponding reference image. The mapping function F maps each set of small patches at the same location from ${\left\{{\mathbf{f}}^k\right\}}_{k=1}^K$ to the corresponding patch in u. To train the approximate function $\tilde{F}$ using E and apply $\tilde{F}$ to enhance a new image set ${\left\{{\mathbf{z}}^k\right\}}_{k=1}^K$, the specific steps of the proposed MLP-based MAP reconstruction enhancement are presented as below.

1). Generating the training data:

Considering that the intensity ranges of the given image set ${\left\{{\mathbf{f}}^k\right\}}_{k=1}^K$ can be largely different from those of the testing image sets to be processed, we normalize the intensity values of ${\left\{{\mathbf{f}}^k\right\}}_{k=1}^K$ to the range of [0,1] using their maximum intensity. For a voxel location in the normalized training images, K of 3D patches are extracted, where K is the total number of training images. After subtracting the mean value from each patch, the voxels of these K patches are stacked into one vector to compose a training input vector $\mathbf{p}={\left\{p_j\right\}}_{j=1}^{K\times L}$ where L is the patch size. The patch mean value is subtracted to make all the image patches have zero mean, which improves the efficiency of the training procedure as the MLP learns the relationships of intensity variations between the training and the label image patches regardless of the absolute mean values. We generate the associated desired output vector $\mathbf{r}={\left\{r_j\right\}}_{j=1}^L$ from the corresponding patch in the reference image u similarly.

Adopting maximum overlapping between two adjacent patches (with one voxel shift in one direction), all possible image patches covering the full region of the training images and the reference image are extracted. These patches compose a very large number of input vectors and their desired outputs. In order to select a representative subset of the vector pairs, we calculate the variance of each desired output which describes the complexity of the contained structures and rank the corresponding input vectors in the order of the variance. Assuming N is the total number of the selected training patch pairs, N/2 of vector pairs having the larger variances are picked together with another N/2 randomly selected ones.

The picked input and the corresponding output vectors form two matrices $\mathbf{P}\in {\Re }^{KL\times N}$ and $\mathbf{R}\in {\Re }^{L\times N}$, where each column of the matrix is an input vector or a desired output vector. To equally distribute importance of each dimension of the input matrix, we scale the intensities of each row to the range of [–1, 1] by using the min-max scaling method, p_scaled = 2(p − p_min)/(p_max − p_min) − 1, where p is an element of a specific row in the matrix, p_min and p_max are the minimum and the maximum of that row. Each row of the desired output matrix is scaled similarly. Thus the generated input matrix and the desired output matrix constitute the training data.

2). Training the mapping Function:

Given P, we formulate $\tilde{F}$ as a parametric function $\tilde{F}\left(\mathbf{W},{\mathbf{p}}^i\right)$, where W represents the parameters and pⁱ is the ith column of P. With this formulation, training the function $\tilde{F}$ using P and R becomes computing the parameters W from the training data through nonlinear regression. We learn $\tilde{F}\left(\mathbf{W},{\mathbf{p}}^i\right)$ by solving the following least squares minimization problem defined over all training patch pairs sampled from P and R:

$$\underset{\mathbf{W}}{\arg \min }{\displaystyle \sum _{i=1}^N{\left(\tilde{F}\left(\mathbf{W},{\mathbf{p}}^i\right)-{\mathbf{r}}^i\right)}^2,}$$ (7)

where rⁱ is the ith column of the desired output matrix R, and N is the total number of the columns in P or R. In this work, we use an MLP model with one hidden layer to represent $\tilde{F}\left(\mathbf{W},{\mathbf{p}}^i\right)$.

The architecture of the MLP used in our work is shown in Fig. 1, where each node represents a neuron and neurons are organized in a number of layers. An input layer, one hidden layer, and an output layer are included in the MLP. The input and output layers directly map to the input and output vectors, which are pⁱ and $\tilde{F}\left(\mathbf{W},{\mathbf{p}}^i\right)$ in our case. Therefore, the number of neurons in the input layer is K × L and the output layer consists of L neurons. Each neuron within a hidden layer or the output layer is fed with the responses from all the neurons in the preceding layer. Each connection between a pair of neurons is associated with a weight.

Fig. 1.

MLP architecture for the MAP reconstruction enhancement. The neurons above the dash line are included to compute the loss function in (7). During training, loss backpropagation starts from the output layer since the connection weights above the dash line are fixed.

Let $x_L^l$ be the output of the Lth neuron in the lth layer. $x_L^l$ is computed as

$$x_L^l=g(w_{L0}^l{b}^{l-1}+{\displaystyle \sum _m{w}_{Lm}^l{x}_m^{l-1}}),$$ (8)

where $w_{Lm}^l$ is the weight associated with the connection between the Lth neuron in the lth layer and the mth neuron in the (l – 1)th layer, b^l−1 is the value stored in the 0th bias neuron in the (l − 1)th layer, and g(·) is a nonlinear activation function. The hyperbolic tangent (tanh), g(z) = tanh(z) = 2/(1 + exp(−2z)) – 1, is used as the activation function in our network. Different from other widely used activation functions, such as the rectified linear unit (ReLU), g(z) = max(0, z), or the sigmoid, g(z) = 1/(1 + exp(−z)), tanh is symmetric regarding to 0. Since the mean value subtracted input and output vectors are also symmetric regarding to 0, choosing tanh is intuitive. The output of each neuron in the output layer is only the linear combination of its inputs from the preceding layer as there is no nonlinear activation function for neurons in the output layer. The two extra layers above the output layer in Fig. 1 are used to compute the loss function in (7).

Given the training data, we use the classic loss backpropagation algorithm called stochastic gradient descent (SGD) [46] to update the weight of every connection between the hidden layer and the input layer, and between the output layer and the hidden layer. Specifically, a batch of the input vectors randomly selected from the training input matrix P and the associated desired output vectors from R are fed to the MLP. By computing the average gradient for those examples, the weights are modified to reduce the loss calculated in (7). This process is repeated for many small batches of the input vectors until the average of the loss function stops decreasing.

3). Processing the testing image set:

For the purpose of computing an expectantly enhanced image ẑ from the testing image set ${\left\{{\mathbf{z}}^k\right\}}_{k=1}^K$ with the trained MLP model, we first normalize ${\left\{{\mathbf{z}}^k\right\}}_{k=1}^K$ to the range of [0,1] using the maximum intensity value. Such a normalization procedure, which is also applied to the training data creation, is necessary to generalize the trained MLP model for patient images as the intensity ranges of training and testing image sets could be very different. For each location in ẑ, the testing input vector is generated by extracting the patches from the K testing images, removing their mean values, stacking the voxels from K patches into a vector, and scaling each entry of the vector to [−1,1] using the maximum and minimum of the corresponding row in the training input matrix P. This process reinforces the generalization from training image patches to the unlearned testing patches. We feed the testing input vector to the trained MLP to compute an output vector that is then descaled by the parameters (maximum and minimum of each row) from the desired output matrix R.

The patch mean value from image z^k reconstructed with the smallest β, which is less biased than the other versions, is added to the output vector. We arrange back the output vector to an image patch, and position it into the output image. Since each voxel in the output image is covered by L patches, where L is the patch size as defined before, the final estimation of a voxel is computed by averaging all the L contributions. After denormalizing the output image using the maximum intensity value, we obtain the MLP processed image ẑ. As the operations include the straightforward one of adding back the mean patch value of the smallest β regularized MAP reconstruction, the resulting variance/resolution property in the MLP processed image would be naturally related to those in the input testing image set. That is consistent with the purpose of the proposed method, which is to enhance the measure of noise/bias tradeoff base on certain given MAP reconstructed images.

C. Implementation setup

In our work, the number of image versions K is set to 3, which means images reconstructed from the MAP algorithm corresponding to three different β values are used to train the MLP model. The smallest β value is selected to obtain a MAP reconstruction reaching the bias close to what the ML algorithm achieves, while the largest β value is selected to suppress the noise in the reconstructed image as much as possible. The other β is an average between the smallest and the largest β values. Since different β values result in images with different local features, extracting image patches from them enables us to take advantage of those features and to train the MLP sufficiently.

We set the 3D patch size L to 4 × 4 × 4 according to our previous study [20]. A smaller patch size could not catch enough structures in an image while a larger patch size would increase the computational cost. Therefore, the size of an input vector is 192 × 1, and the size of an output vector is 64 × 1. The number of neurons in the hidden layer of the MLP model is set to 128 empirically, which computes a lower level abstraction from the input data by suppressing its redundant variations.

Employing Caffe [47], which is a widely used practical framework for machine learning, our MLP is trained by the SGD method with the number of training patch pairs N set to 200,000. Such a large-scale training samples can largely eliminate the risk of overfitting. Referring to the training setup for the models in Caffe, we conduct our training procedure with the learning rate initialized at 0.01, and the inverse decay policy is applied to decrease the learning rate. A total of 100,000 iterations is implemented to train our MLP model. We will discuss the convergence behavior of our MLP training in the Discussion section.

III. Experiments and Evaluation

To study the performance of the proposed MLP enhancement scheme, we simulated realistic brain PET imaging data at various count levels for different subjects with and without lesions using known activity. The MLP was trained by images reconstructed from the Green ‘smoothness’ MAP algorithm with different regularizing weights for a normal subject at one count level. Once the MLP model has been trained, reconstructed images with lesions, of other count levels, and of other subjects were processed by the network. We compared the MLP processed images with those reconstructed from the MAP algorithm of different weights using the noise versus bias/contrast tradeoff. In addition to the simulation study, we applied this MLP model trained using the simulated dataset to process the MAP reconstructed patient brain PET images.

A. Simulation Study

1). Emission data generation:

We used anatomical models from the BrainWeb database to create PET phantom images for the simulation study. BrainWeb is a realistic database containing high-resolution, volumetric, anthropomorphic, digital brain phantoms built for tomographic image simulation [48]- [50]. Each subject model consists of a set of 3D fuzzy tissue membership volumes including 11 tissue classes of gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF), fat, muscle, muscle/skin, skull, blood vessels, connective, dura matter, and bone marrow. The voxel values in these volumes reflect the proportion of that tissue presented in that voxel. Anatomical models corresponding to three subjects were used to create the activity images from which the PET imaging data was simulated.

The PET activity distribution was created by assigning a clinically realistic activity of 12500 Bq/ml in GM, 3250 Bq/ml in WM, 0 Bq/ml in air, CSF and bone, and 1000 Bq/ml in all other tissues [27]. The attenuation map was generated using the attenuation coefficient of 0.146 cm⁻¹ for bone, 0 cm⁻¹ for air, and 0.096 cm⁻¹ for other tissues. Besides the PET activity image with normal uptake, we also created an activity image with hypointense GM lesions for the subject 1. Five spherical lesions with a 20 mm diameter were created by reducing the GM tracer uptake by 25% inside each sphere. The activity of the other classes was kept and the actual volume varied among the lesions. The simulated PET activity images with normal uptake and lesions of subject 1, and the normal subject 2 (all of the central transaxial slice) are shown in Fig. 2.

Fig. 2.

Transaxial slices through the BrainWeb phantoms (in Bq/ml) simulating the subject 1 (a) normal PET activity, (b) PET activity with lesions, and (c) the subject 2 normal PET activity.

The PET data was simulated for the geometry of the high resolution research tomograph (HRRT) scanner [51]. The PET image was interpolated to create the voxel size of (1.22mm)³ and the image dimension of 256 × 256 × 207. We used a finer grid brain phantom with a reduced voxel size by a factor of 2 in each dimension to model the effect of limited spatial resolution. Noise-free emission data and the attenuation coefficient projection data, each with the dimension of 256 × 288 × 2209, were generated by forward-projecting the phantom activity image and the attenuation map using the symmetry and SIMD-based projection algorithm, respectively [52], [53]. The normalization coefficients were obtained using the component-based method [54]. Uniform random events were simulated summing in total a 30% of the simulated counts and the effect of scatter was simulated analytically using the single scatter simulation formula [55]. Poisson noise was introduced to simulate three count levels for normal subject 1, corresponding to 10 min, 5min, and 3min FDG PET measurement. The three count levels were ~ 1.8 times apart in terms of the total counts. For the abnormal subject 1, we generated data of the count level corresponding to 10 min acquisition. For the normal subjects 2 and 3, emission data corresponding to 3 min acquisition was generated.

2). Training and testing image creation:

The PET image set for training is shown in Fig. 3. They were reconstructed from the 10 min simulation of the normal subject 1, using the MAP algorithm with three different β values at the 10th iteration. The smallest β keeps more details in the reconstruction while the largest β significantly suppresses the image noise. The implementation of the MAP algorithm used the ordered-subset (OS) EM [56] with 16 subsets in each iteration. Normalization, attenuation, randoms, and scatter corrections were incorporated in the reconstruction process [52].

Fig. 3.

Transaxial slices through the training images (in Bq/ml) reconstructed from 10 min simulation of the normal subject 1 using the MAP algorithm with the β values of (a) 0.0015, (b) 0.0025, and (c) 0.0035. The maximum gray level of the images is set as the maximum value in the phantom image.

To test the performance of the MLP image enhancement scheme, we first applied the trained model to the MAP reconstructed 10 min abnormal subject 1 images. The β values used were the same as those for the creation of the training images. Then the reconstructed normal subject 1 images corresponding to 5 min and 3 min acquisition were tested. At last, we tested the reconstructed normal subject 2 and subject 3 images of 3 min acquisition, respectively. For each of the testing cases, we selected an appropriate set of β values to create the testing images with the smallest β keeping more details and the largest β reducing the image noise better. The specific β values in each testing case will be presented with the testing results in the Result section. To study the effect of β selection on the performance of the proposed method, we used multiple sets of β values for the reconstruction of the 3 min normal subject 2 data. Comparisons were made among the processed results.

3). Evaluation metrics:

To evaluate the resulting images from the MLP enhancement, we compare them with images reconstructed from the MAP algorithm and the ML algorithm (also implemented using OS EM with 16 subsets) with and without Gaussian post-smoothing [10], [11]. As the proposed method could be considered as a fusion technique, we include a wavelet based image fusion of the multiple MAP reconstructed images in the comparison [57]. The tradeoff between the normalized mean squared error (NMSE), as a measure of bias, and the normalized standard deviation (NSD), as a measure of noise, on the GM region of a brain is used for image evaluation. The GM region is defined to include the voxels of which more than 95% belong to the GM, based on the fuzzy class model.

The NMSE for a region is calculated using

$$\mathrm{NMSE}=\frac{1}{n}{{\displaystyle \sum _{j=1}^n\left(\frac{f_j-\overline{u}}{\overline{u}}\right)}}^2,$$ (9)

where f_j is the jth voxel activity value, ū is the reference true mean activity value, and n denotes the number of voxels in the region under consideration. The NMSE bias value is plotted against the NSD noise value calculated by

$$\mathrm{NSD}=\frac{\sqrt{\frac{1}{n-1}{\displaystyle {\sum }_{j=1}^n{\left(f_j-\overline{f}\right)}^2}}}{\overline{f}},$$ (10)

where f_j and n are defined as those in (9). $\overline{f}=\frac{1}{n}{\displaystyle {\sum }_{j=1}^n{f}_j}$ represents the regional mean activity value.

To quantify the ability of recognizing reduced activity from normal activity in the reconstructed images and the images processed by the Gaussian filter, the wavelet fusion, and the MLP enhancement, we calculate the contrast between the normal and abnormal activities on the regions of interest (ROIs). The defect ROI covers the GM voxels where the defects are defined. The normal ROI is defined on the normal GM covering the same number of voxels as the defect ROI. The contrast is defined by

$$\mathrm{Contrast}=\frac{{\overline{f}}_{\mathrm{N}}-{\overline{f}}_{\mathrm{D}}}{{\overline{f}}_{\mathrm{N}}+{\overline{f}}_{\mathrm{D}}},$$ (11)

where ${\overline{f}}_{\mathrm{N}}$ and ${\overline{f}}_{\mathrm{D}}$ are the averaged activity values from the normal (N) and defect (D) ROIs, respectively.

B. Patient Study

We applied the MLP model trained by images from the simulation study to process images reconstructed from two patient PET imaging datasets with tracers for different diagnostic purposes. One set of PET data was acquired using the tracer [11C] DPA-713, which is a radioligand that binds to the translocator protein (TSPO) and serves as a marker of the neuroinflammatory burden [58]. The other tracer was [18F] Florbetapir, which is an FDA approved amyloid-beta imaging agent for clinical evaluation of late-life cognitive impairment in Alzheimer’s disease [59]. PET scans were performed on the second-generation Siemens HRRT, an LSO-based, 2.5 mm- resolution, dedicated 3D brain PET scanner [51]. The data was reconstructed with correction for attenuation, normalization, deadtime, scatter, and randoms [52]. The PET image space consisted of cubic voxels with a size of (1.22 mm)³, spanning dimensions of (31 cm)² transaxially and 25 cm axially. We studied the DPA-713 data with accumulation from 0 to 30 min post-injection while for the Florbetapir we worked on data with the time span from 50 to 70 min. The number of counts in the studied DPA-713 data is ~ 3.8 times of that in the studied Florbetapir data.

To evaluate the performance of the proposed MLP enhancement and compare it with the MAP algorithms and the post processing methods, we calculated the noise in the ROIs using (10). The ROIs were drawn using an in-house software on the corresponding MR image and applied to the reconstructed and post processed PET images. As the true intensity values of patient experiments are not available, we plotted the regional mean value (instead of bias) versus the noise along with the iteration on the specified ROIs. The size of each ROI will be presented with the patient study results in the Result section.

IV. RESULTS

A. Simulation Study Results

For the 10 min simulation of the subject 1 with lesions, we plot the ROI NSD versus NMSE/Contrast for images reconstructed from the ML algorithm with and without Gaussian post-filtering, the MAP algorithm with different weights, the wavelet fusion, and the MLP enhancement in Fig. 4. For straightforward comparison, we choose the cutoff frequency of the Gaussian filter to result in similar end-iteration noise to that reached by the MLP enhancement. As to wavelet fusion, we calculated the wavelet decompositions of the MAP reconstructed images at level 5 using the widely used wavelet sym4. The decompositions were then merged by taking the mean values of the details coefficients and the approximations coefficients.

Fig. 4.

Regional noise versus (a) bias and (b) contrast plots (true contrast indicated) comparing the MLP enhancement with the ML, the filtered ML, the MAP, and the wavelet fusion algorithms from 10 min data reconstruction of the subject 1 with lesions.

The NSD and NMSE are computed on the defect ROI in Fig. 4(a), while in Fig. 4(b) the NSD is the average of the normal and defect ROI NSD values. As demonstrated, the MLP processed images reach similar noise measures to those from the largest β regularized MAP algorithm. Furthermore, the MLP enhanced end-iteration image has a contrast closer to that reached by the ML algorithm, demonstrating the advantage of the MLP enhancement on recovering the lesions over the MAP algorithm with different weights. On the contrary, the post-smoothed ML has to sacrifice contrast to achieve low noise. The wavelet fused images are similar to the MAP reconstructed images with the middle beta.

The GM NSD versus NMSE plots for the normal subject 1 of the 5 min and 3 min simulations are shown in Fig 5. In both cases, the MLP enhanced images result in better noise versus bias tradeoffs compared with those from the MAP algorithm with three β values and from other post processing methods. That is, the MLP model trained from images of longer acquisition applies well to images of shorter acquisition time. Moreover, Fig. 6 shows the results from the normal subjects 2 and 3 of 3 min acquisition. The improved noise versus bias tradeoffs demonstrate the effectiveness of applying the trained MLP model from images of a randomly selected subject to images of other subjects.

Fig. 5.

GM noise versus bias plots comparing the MLP enhancement with the ML, the filtered ML, the MAP, and the wavelet fusion algorithms from (a) 5 min data and (b) 3 min data reconstruction of the normal subject 1.

Fig. 6.

GM noise versus bias plots comparing the MLP enhancement with the ML, the filtered ML, the MAP, and the wavelet fusion algorithms from 3 min data reconstruction of (a) the subject 2 and (b) the subject 3.

The testing images reconstructed using the post-smoothed ML reconstruction, the MAP algorithm with different β values (the smallest and the largest), the wavelet fusion results, and the MLP processed images from the 10 min simulation of the subject 1 with lesions, the 3 min simulation of the normal subject 1, and the 3 min simulation of the normal subject 2 are shown in Fig. 7. The maximum gray levels of the images are set as the maximum value of the phantom images so that it is easier to compare the images from different reconstruction algorithms and from different count levels. The MLP enhanced images are obviously less noisy than the low-β regularized MAP reconstruction and the wavelet fusion results. They also display more details i.e. are less blurry than the MAP reconstructed images with heavy regularization and the images processed by the Gaussian filter.

Fig. 7.

Transaxial slices through the testing images (in Bq/ml) reconstructed from (a) the 10 min simulation with lesions of the subject 1, (b) the 3 min simulation of the subject 1, and (c) the 3 min simulation of the subject 2 using (1) the filtered ML algorithm, the MAP algorithm with (2) the smallest β value, and (3) the largest β value, (4) the wavelet fusion of the MAP reconstructed images, and (5) the MLP enhancement of the MAP reconstructed images.

B. Patient Study Results

In Fig 8, we plot the regional noise versus mean value along with the iteration number for different ROIs in the DPA-713 patient images reconstructed from using the ML algorithm, the post-smoothed ML algorithm, the MAP algorithm with different β values, the wavelet fusion, and the MLP enhancement. In most of the regions, the MLP enhancement suppresses noise as low as the MAP algorithm with large weighting factor while arriving at mean regional values close to what the ML algorithm achieves. Its improvement on noise versus mean value tradeoff over the MAP algorithm and the other two post processing methods is well demonstrated.

Fig. 8.

Regional noise versus mean value (in counts) plots comparing the MLP enhancement with the ML, the filtered ML, the MAP, and the wavelet fusion algorithms, for the DPA-713 patient study. The ROI sizes (in voxels) are 61299, 74144, 45274, 35979, 18356, 7524, 4574, 1764, 9017, and 1958, respectively.

We plot the regional noise versus mean value along with iteration number for different ROIs reconstructed from 50–70 min Florbetapir data using the ML algorithm with and without post-smoothing, the MAP algorithm with different weights, the wavelet fusion, and the MLP enhancement in Fig. 9. As shown, the MLP processed images have similar noise measures to those from the large β regularized MAP algorithm while achieving regional mean values close to those of the ML algorithm. The MLP enhanced image demonstrates improvement over the MAP algorithm and other post processing algorithms on the noise versus regional mean tradeoff in this Florbetapir patient study.

Fig. 9.

Regional noise versus mean value (in counts) plots comparing the MLP enhancement with the ML, the filtered ML, the MAP, and the wavelet fusion algorithms, for the 50–70 min Florbetapir patient study. The ROI sizes (in voxels) are 11111, 60357, 13637, 32263, 29078, 9080, 38935, and 1425, respectively.

Fig 10 presents the transaxial slices of the reconstructed images using the MAP algorithm and the proposed MLP enhancement for the DPA-713 and the Florbetapir studies. Resulting images of post-smoothing and the wavelet fusion are not shown to save space. For each tracer, the reconstructed images are shown with the same maximum gray level for a straightforward comparison. The MLP enhanced images are noticeably less noisy than the small β regularized MAP reconstruction and the textures in the enhanced images are clearer compared with the large β regularized MAP reconstruction.

Fig. 10.

Transaxial slices through the reconstructed PET images (in counts) from (a) the 0–30 min DPA-713 patient data and (b) the 50–70 min Florbetapir patient data using the MAP algorithm with (1) the smallest β value, (2) the largest β value, and (3) the MLP enhancement of the MAP reconstructed images.

V. DISCUSSION

A. Convergence Behavior and Run-Time

As mentioned in the MLP enhancement scheme description, we implemented the SGD method with 100,000 iterations for the backpropagation of the network. Although the SGD is actually a gradient method, the researchers involved in the backpropagation applications believe that this method often leads to a global minimum, or at least makes it possible to meet practical stopping criteria [60]. To illustrate the convergence behavior of our MLP training, we plot the average of the loss calculated using (7) and the learning rate along with the training iteration number in Fig. 11. During the training phase, the learning rate drops continuously from the initialization of 0.01. Meanwhile, the loss averaged on a batch of training input vectors decreases dramatically at first, followed by small reductions. When we stop the training procedure at the maximal iteration, the loss reduction is less than 6 × 10⁻⁵, which is small enough to ensure convergence of the gradient method.

Fig. 11.

Convergence behavior of the averaged objective function.

In terms of run-time, the training procedure in our experiment takes ~ 20 minutes on a 3.2 GHz CPU. Once we have trained the network, it needs ~ 3 minutes to process a testing image set with each image of a size 256 ×256 ×207 on the same CPU. As the patch extraction and combination is the most time-consuming operation that slows down the process, parallel patch operation with GPUs will highly decrease the time for testing. If a deeper MLP were used in our experiment, more running time would be required either to make the training procedure converge or to process a testing image set using the trained model. Moreover, there may exist the risk of overfitting and the trained model would not be able to generalize well on the testing images [37]. Therefore, we choose the MLP with one hidden layer that has been shown sufficient to represent the mapping function in our work.

B. Effect of β Selection

Based on the simulation study, we demonstrate that the proposed MLP model trained with image patches reconstructed from the simulated data of one subject at one count level applies well to image patches from the same count level with lesions and other count levels of the same and other subjects. To examine how the selection of the β values used to create the testing images affects the MLP enhancement result, we applied the trained MLP to the testing images reconstructed with different sets of β values.

1). Multi-β enhancement:

Taking the 3 min simulation of the normal subject 2 as an example, different from previously tested β set {0.001, 0.004, 0.007} with a large range, two image sets reconstructed using smaller ranges of β values {0.001, 0.0025, 0.004} and {0.004, 0.0055, 0.007} were processed, respectively. The NSD versus NMSE tradeoffs of the images reconstructed from the ML algorithm, the MAP algorithm with different β values, and the MLP enhancement are shown in Fig. 12(a), where the results in Fig. 6(a) are plotted again in red for comparison. As demonstrated, for each smaller range β set, the MLP processed images reach relatively lower noise and smaller bias compared with the three associated input MAP reconstructions.

Fig. 12.

GM noise versus bias plots from the 3 min data reconstruction of the subject 2 comparing the MLP enhancement with the ML and the MAP algorithms with (a) two groups of β values and (b) two single β values.

The end iteration image enhanced from using the small set of β values has higher noise level but smaller bias compared with the enhanced result from the large β set, which agrees with that large β suppresses noise more while sacrificing the spatial resolution. Connecting the tradeoffs of the MAP reconstruction from different regularizing weights at the end iteration in Fig. 12(a), we specify a lower bound of the variance/bias tradeoff as described in [33]. The tradeoffs of the MLP processed images from both small range and large range β sets are obviously located in the unachievable region of the MAP reconstruction. Therefore, we have achieved an improvement of the tradeoff between variance and bias using the MLP-based enhancement, which cannot be obtained by tuning the weighting parameters of the MAP algorithm.

2). Single-β enhancement:

We further tested the trained MLP on a single image reconstructed using the ML algorithm or the MAP algorithm with β as 0.005. In this case, three same PET images were processed by the trained MLP model. Fig. 12(b) shows the noise versus bias tradeoffs of the ML algorithm, the MLP processed ML, the MAP algorithm, and the MLP processed single MAP. It is interesting to see that the MLP enhancement reaches lower noise measure with reduced bias compared with the single ML/MAP reconstructed image. Since the MLP enhancement applies to one ML/MAP reconstructed image, it does not necessarily require three MAP reconstruction processes in case that the computation time is of a concern.

As the MAP algorithm significantly suppresses the noise of the ML reconstruction, the MLP processed MAP reconstruction results in lower noise level but larger bias compared with the MLP processed ML reconstruction. Considering the ML reconstruction as a special case of the MAP reconstruction with the weighting parameter set to 0, we can specify two rough lower bounds of the tradeoff between variance and bias for the MAP reconstruction and the enhanced MAP reconstruction, respectively. It is obvious that the proposed MLP enhancement technique significantly reduced the size of the unachievable region of the MAP reconstruction.

To summarize, the proposed MLP enhancement technique is robust to the selection of the weighting parameters used to create the testing images. For both the multi-version enhancement and the single image enhancement, the MLP-based processing scheme moves the the noise/bias tradeoff bound of the MAP algorithm towards the lower variance and smaller bias directions. That is to say that the MLP enhancement provides noise reduction with less spatial resolution compromise compared with the MAP algorithm.

C. Generalization Capability

1). Application to other reconstruction methods:

As stated earlier, the proposed MLP enhancement is not limited to the Green ‘smoothness’ regularized MAP algorithm. It should be applicable to other regularized image reconstruction. To demonstrate the generalization capability of this method on other MAP algorithms, we applied the trained MLP to process images reconstructed using the quadratic regularization and an edge-preserving TV regularization [15], [17]. Taking the 3 min simulation of the normal subject 2 as an example, the results from enhancing the reconstructed images of these two MAP methods are presented in Fig. 13. As shown, the enhancement effect in these cases is similar to that for the Green MAP reconstructed images presented in Fig. 6(a).

Fig. 13.

GM noise versus bias plots from 3 min data reconstruction of the subject 2 comparing the MLP enhancement with the ML and the MAP algorithms with (a) the quadratic regularization and (b) a TV regularization.

Similar to the images reconstructed using MAP algorithms with different β values, the ML reconstructions with post filtering of different cutoff frequencies result in different tradeoffs of noise and bias as well. Thus we further evaluated the proposed method for the Gaussian post-filtered images [10], [11]. Fig. 14 shows the MLP enhancement results compared with the post-filtered images with different cutoff frequencies for the case of the 3 min data reconstruction of the subject 2. As demonstrated, the MLP enhanced image achieves similar noise measure as the high cutoff frequency gets while reducing the bias compared with that from the low cutoff frequency filtered ML reconstruction. The noise versus bias tradeoff curve from using the MLP enhancement obviously reaches the unachievable region of the filtered ML algorithm.

Fig. 14.

GM noise versus bias plots from 3 min data reconstruction of the subject 2 comparing the MLP enhancement with the ML algorithm, the filtered ML algorithm with cutoff frequencies of 0.3 cycle/pixel, 0.225 cycle/pixel, and 0.15 cycle/pixel, and the MLP enhancement.

2). Application to other PET images:

In our simulation study, the training and testing subjects carry the same uptake values in their individual tissue classes. This leads to apparently idealized testing situations. However, it is worth noting that the BrainWeb subject models consist of fuzzy tissue membership volumes. Therefore, the voxels on tissue borders have different combinations of tissue uptake values. The MLP model learns the relationships of all kinds of intensity variations between the training and label image patches and is potentially open to process PET images with activity distributions different from the training image. In patient study, we further verified that the proposed MLP model trained using image patches reconstructed from the simulated FDG brain PET data also applies to the image patches reconstructed from the patient brain datasets with the tracers DPA-713 and Florbetapir.

This again confirms that the MLP model is able to efficiently capture local features of the 3D image patches containing structural patterns of the brain PET images, for example, edges, smooth regions, and textures regardless of activity distribution. Since all possible images can be constructed using these structural patterns [61], this work that has so far studied brain images with a few tracers will be continued with images of other organs taken with more varieties of tracers. Besides, we could also finetune our MLP model trained by the specific brain image pair through learning from more other image pairs. With improved generalization capability of this network, we expect to further broaden the applicability of the proposed ANN based PET image enhancement technique.

VI. Conclusion

We developed a patch-based MLP enhancement scheme to improve PET images reconstructed using a MAP algorithm with different regularizing weights. The MLP was trained using image patches reconstructed from the simulated brain PET data of a normal subject at a certain count level. The trained MLP model was tested on image patches of the same subject with lesions, at other count levels, and of two other subjects. In addition, the trained MLP was also tested on image patches reconstructed from two patient PET image datasets. In every testing experiments, the MLP processed image reaches the noise level as low as or even lower than the MAP reconstruction achieves at heavy regularization while significantly reducing bias or increasing the lesion contrast. Showing its robustness in processing images reconstructed with different sets of weighting parameters, the proposed MLP enhancement technique achieves an improvement on the tradeoff between noise and bias of the MAP reconstruction, which can not be obtained by tuning the weights of the MAP algorithm. We conclude that the proposed MLP enhancement method has the potential to quantitatively improve clinical PET imaging.