PET Image Super-Resolution Using Generative Adversarial Networks

Abstract

The intrinsically low spatial resolution of positron emission tomography (PET) leads to image quality degradation and inaccurate image-based quantitation. Recently developed supervised super-resolution (SR) approaches are of great relevance to PET but require paired low- and high-resolution images for training, which are usually unavailable for clinical datasets. In this paper, we present a self-supervised SR (SSSR) technique for PET based on dual generative adversarial networks (GANs), which precludes the need for paired training data, ensuring wider applicability and adoptability. The SSSR network receives as inputs a low-resolution PET image, a high-resolution anatomical magnetic resonance (MR) image, spatial information (axial and radial coordinates), and a high-dimensional feature set extracted from an auxiliary CNN which is separately-trained in a supervised manner using paired simulation datasets. The network is trained using a loss function which includes two adversarial loss terms, a cycle consistency term, and a total variation penalty on the SR image. We validate the SSSR technique using a clinical neuroimaging dataset. We demonstrate that SSSR is promising in terms of image quality, peak signal-to-noise ratio, structural similarity index, contrast-to-noise ratio, and an additional no-reference metric developed specifically for SR image quality assessment. Comparisons with other SSSR variants suggest that its high performance is largely attributable to simulation guidance.

1. Introduction

Positron emission tomography (PET) is a 3D functional imaging modality that uses exogenous radio-labeled contrast agents, referred to as radiotracers, to map the biodistribution of physiological processes and parameters. One of the key hurdles toward accurate quantitation using PET is its intrinsically poor spatial resolution [1]. A number of remedial strategies have been proposed to address this challenge. Of these, some model the point spread function (PSF) of the PET scanner within the image reconstruction framework [2, 3], while others perform image-domain corrections either set up as a partial volume correction problem [4, 5, 6] or an image deblurring problem [7, 8, 9, 10]. Recent advances in deep learning, particularly convolutional neural networks (CNNs), have ushered in a wide variety of novel super-resolution (SR) imaging techniques. SR imaging refers to the task of converting a low-resolution (LR) image to a high-resolution (HR) one. Seminal CNN-based SR models featured lightweight (shallower) networks [11] with some architectures performing the upsampling from the LR to the HR domain via a convolutional layer [12, 13]. Later efforts involving deeper CNN architectures, such as the very deep SR (VDSR) [14], led to substantial performance gains. While these CNN-based SR techniques tend to outperform classical deblurring and SR methods by large margins, it should be noted that they are based on supervised learning and consequently require matched LR-HR image pairs for training.

Our previous work on the resolution recovery of PET images includes both a (classical) penalized deconvolution approach based on joint entropy [15, 10] and a deep supervised learning approach inspired by the VDSR [9, 16]. An attractive feature of this latter approach is that it enables mapping from the image domain of legacy scanners to that of newer state-of-the-art scanners that exhibit greatly-enhanced image resolution. However, due to its supervised learning format, it requires paired LR and HR training samples with both images corresponding to the same human subject under similar scan settings. PET scans involve substantial overhead in terms of scan time, cost, and patient irradiation. This need for dual scans on LR and HR scanners is a major bottleneck in clinical settings and greatly limits the practical utility of VDSR and similar supervised SR techniques.

The need for paired LR-HR images for training could be a bottleneck even in the natural image processing domain as it necessitates the capture of the same image at different resolution scales. This has motivated a pursuit for unsupervised SR (USR) or self-supervised SR (SSSR) models, which are better equipped to deal with most real-world scenarios. The zero-shot SR network [17], for example, estimates a degradation kernel from a single test image. However, its requirement for network retraining for individual test images poses a computational burden in the validation and application phases. With the advent of state-of-the-art generative adversarial network (GAN) architectures, there is increasing interest in SR techniques that rely on weak supervision, i.e., using unpaired LR and HR images. One such GAN-based technique uses sub-pixel convolutions [18]. Following the success of image-to-image translation using the CycleGAN [19], which uses two generator and discriminator pairs, newer SSSR techniques have used two cycleGAN modules to learn the LR-HR mapping for natural images [20, 21]. Specific to the domain of facial landmark localization, a GAN-based approach was developed to simultaneously super-resolve facial images and detect landmarks from them [22]. These new networks are promising and tend to outperform classical single-image SR techniques [23, 24, 25].

For clinical imaging applications, it is usually infeasible to obtain the “ground-truth” HR counterparts for LR PET scans of human subjects. While paired LR-HR images are difficult to acquire, there is wider availability of unpaired LR and HR images (e.g., populations derived from distinct cohorts, with the same scanner, image acquisition, and image reconstruction settings being used for the same cohort). Additionally, each individual imaging modality leads to unique image characteristics that tend to be quite distinct from those of natural images, thereby hindering the direct application of image processing techniques developed for and validated on natural images. This has led to the emergence of domain-specific USR or SSSR techniques for medical modalities such as computed tomography (CT) [26], magnetic resonance imaging (MRI) [27], and ultrasound [28]. This paper addresses challenges unique to PET imaging and presents an SSSR technique based on dual GANs that is able to perform PET resolution recovery without requiring paired training samples.

The key methodological novelties of this work are as follows: (1) simulation guidance to boost SR performance in clinical datasets, (2) spatial inputs to account for the spatially-variant nature of image resolution in PET, (3) anatomical inputs to incorporate HR anatomical information, and (4) HR-to-LR mapping via generator pretraining which offers additional feature guidance for SR. The significance of our method lies in its immense value for data harmonization without requiring ground-truth training data and without sacrificing resolution in large multi-center studies [29], e.g., those connected to vast image repositories such as the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and The Cancer Imaging Archive (TCIA). Without SR capabilities, data harmonization is typically performed by image degradation (e.g., by blurring HR images to match the resolution of the lowest-resolution images in the dataset), which is a suboptimal approach. To our knowledge, PET image mapping across two scanner domains based on unpaired training data is a previously unexplored area in the PET imaging field.

In Section 2, we present the underlying network architecture. In Section 3, we describe the clinical and simulation datasets and the training and validation steps. In Section 4, we present clinical results comparing the performance of the multiple SSSR variants with a range of reference approaches. Finally, in Section 5, we summarize this work and discuss its strengths and limitations.

2. Theory

2.1. Overview

The CycleGAN architecture [19], which consists of two generator-discriminator pairs, is particularly effective at image-to-image translation in an unpaired setting for training. Our SSSR network architecture for PET is broadly inspired by the CycleGAN in that it consists of two coupled GANs with one generator-discriminator pair performing LR-to-HR mapping and another generator-discriminator pair responsible for the reverse mapping. CycleGAN, however, is not directly applicable to SR imaging. Previously reported SR imaging techniques that rely on unpaired data have used dual CycleGANs leading to substantially higher computational burden [20, 21]. A key distinguishing aspect of our work is the use of a separately-trained VDSR network based on paired simulation datasets [16]. This simulation guidance, as we will demonstrate subsequently, enables us to generate SR images using dual GANs without requiring more expensive bi-cycle [21] or cycle-in-cycle structures [20].

Unlike uniformly-blurred natural images, PET images are blurred in a spatially-variant manner [2, 30]. We, therefore, accommodate spatial location details as inputs to assist the SR process. In light of the cylindrical symmetry of PET scanners, two spatial input channels are used, one containing the axial coordinates and the other containing the radial coordinates. Most PET scanners come equipped with anatomical imaging capabilities in the form of CT or MR to complement the functional information in PET with structural information. The anatomical counterparts of PET images typically have much higher resolution. To facilitate SR image recovery, we utilize HR features from MR images derived from an additional anatomical input channel. These additional inputs are vital for super-resolving PET images. The spatially-variant nature of spatial resolution in PET makes the resolution recovery problem quite different from that for MR or CT. Additionally, our network learns structural similarities between PET and MR images and uses this information to assist the SR image recovery.

As illustrated in Figure 1, the first GAN contains a generator G₁ : LR → HR with three subnetworks. The subnetwork (G₁₁) accepts as inputs the LR PET image, an HR MR image, and radial and axial location information. The subnetwork (G₁₂) receives as input the output of the second last layer of the VDSR network (a 20-layer CNN architecture for SR), which plays the role of a fixed feature extractor. The VDSR network is separately trained using paired simulation data based on supervised learning. It receives the same input as G₁₁. The outputs of G₁₁ and G₁₂ are concatenated and then fused through a third subnetwork G₁₃, which produces the final SSSR image.

Figure 1:

GAN architecture for SR imaging with patch-level multi-channel inputs (LR PET, HR MR, and radial and axial location details for voxels) which outputs an SR PET image patch.

Since the VDSR network is trained using only simulation data, its output for clinical data inputs is expected to be artifact-ridden. Utilizing the output of this 20-layer CNN’s penultimate layer (a higher-dimensional feature set) instead of its final image as an auxiliary input to the SSSR network ensures more effective information capture. Additionally, while the SSSR network utilizes information derived from the imperfect VDSR network, it uses this in conjunction with features derived directly from the clinical data.

The discriminator D₁ verifies the SR output ($\hat{\mathit{x}}$) by comparing it with an unpaired HR image sample ($\tilde{\mathit{x}}$). Similar to the CycleGAN architecture, the SR image is mapped back to the LR domain using a second generator G₂ : HR → LR. A second discriminator D₂ verifies this blurry image, denoted in Figure 1 as LR′ and mathematically as $\hat{\mathit{b}}$, by comparing it with the original LR image, $\tilde{\mathit{x}}$. G₂ is pretrained using LR and HR simulation image pairs and spatial information, with the LR images generated by applying the measured spatially-variant point spread function (SVPSF) of the scanner [10] to the HR images.

While LR-to-HR mapping is an ill-posed problem, HR-to-LR mapping in PET involves a spatially-variant blurring operation that can be measured and modeled in either image or sinogram domains. As part of our previous work, we have modeled and measured the spatially-variant point spread function (SVPSF) for individual scanners [10]. We use this knowledge to guide the training of the generator G₂. Though it is possible to entirely replace G₂ with the SVPSF blurring operator. However, our experiments clearly suggested that despite the feasibility of the latter approach, using a generator neural network for LR-to-HR mapping offers distinct advantages. Specifically, G₂ serves to do more than simple LR-to-HR mapping and provides additional features guidance to the overall network as will be demonstrated via comparisons with an alternative network where G₂ is replaced by an SVPSF blurring operator. We consider G₂ pretraining using domain-specific knowledge an additional contribution of this paper.

2.2. Loss Functions

The overall loss function for training the generator G₁ includes two adversarial losses (${\mathcal{L}}_{adv}^1$ and ${\mathcal{L}}_{adv}^2$), a cycle consistency loss $\left({\mathcal{L}}_{cyc}\right)$, and a total variation penalty $\left({\mathcal{L}}_{tv}\right)$.

$${\mathcal{L}}_{GAN}={\mathcal{L}}_{adv}^1+{\lambda }_a{\mathcal{L}}_{adv}^2+{\lambda }_c{\mathcal{L}}_{cyc}+{\lambda }_t{\mathcal{L}}_{tv}.$$

Here λ_a, λ_c, and λ_t are hyperparameters. These loss functions are described below.

2.2.1. Adversarial Loss

The adversarial losses are utilized for marginal matching [31] and incentivize the generated images to obey empirical distributions in the source and target domains. We employed two adversarial losses (${\mathcal{L}}_{adv}^1$ and ${\mathcal{L}}_{adv}^2$) which were adopted from the Least Squares GAN [32]. These least-squares adversarial losses tend to be free of the vanishing gradient problem that often accompanies regular GANs with sigmoid cross-entropy loss functions. While training the generator G₁, we express the adversarial loss functions as:

$${\mathcal{L}}_{adv}^1={\left(D_1(\hat{\mathit{x}})-c\right)}^2,$$ (1)

$${\mathcal{L}}_{adv}^2={\left(D_2(\hat{\mathit{b}})-c\right)}^2,$$ (2)

where $\hat{\mathit{x}}$ and $\hat{\mathit{b}}$ are the SSSR (synthetic HR) and blurry synthetic LR images respectively and c is the value that the generator sets for the discriminator to accept the SSSR images.

While training the discriminators D₁ and D₂, we express the adversarial loss functions as:

$${\mathcal{L}}_{adv}^1={\left(D_1(\tilde{\mathit{x}})-a\right)}^2+{\left(D_1(\hat{\mathit{x}})-b\right)}^2,$$ (3)

$${\mathcal{L}}_{adv}^2={\left(D_2(\tilde{\mathit{b}})-a\right)}^2+{\left(D_2(\hat{\mathit{b}})-b\right)}^2,$$ (4)

where $\tilde{\mathit{x}}$ and $\tilde{\mathit{b}}$ are unpaired HR and LR PET images respectively, and a and b are the labels for the real ($\tilde{\mathit{x}}$ and $\tilde{\mathit{b}}$) and fake ($\hat{\mathit{x}}$ and $\hat{\mathit{b}}$) images respectively. It should be noted that the $\hat{\mathit{x}}$ and $\hat{\mathit{b}}$ are produced by the generator (hence considered “fake” in GAN terminology) while $\tilde{\mathit{x}}$ and $\tilde{\mathit{b}}$ belong to the datasets (and thus considered “real”).

For a least-squares loss function, the parameters, a, b, and c are selected such that the Pearson χ² divergence between the probability distribution from the generator and the probability distribution from the data is minimized. Though these values are not uniquely determined, it has been shown [32] that they are constrained by the equations a−c = 1 and a−b = 2. In our work, we vary the range of a from 0.7 to 1.2, thus effectively constraining b and c.

2.2.2. Cycle Consistency Loss

The cycle consistency loss $\left({\mathcal{L}}_{cyc}\right)$ computes an L₂-norm error in the LR domain. The cycle consistency loss is critical as previous works [33], [19] have found that the adversarial loss alone is insufficient to ensure that a learned function is able to transform an input source image to a target output image. The cycle consistency loss imposes shared latent space constraints and thus urges the source content to be preserved during cycle reconstruction mapping, i.e., $G_2(G_1(\tilde{\mathit{b}})\approx \tilde{\mathit{b}}$. This loss is computed as:

$${\mathcal{L}}_{cyc}=\Vert G_2\left(G_1(\tilde{\mathit{b}})\right)-\tilde{\mathit{b}}{\Vert }_2.$$ (5)

2.2.3. Total Variation Penalty

The total variation loss has a denoising impact on image deblurring algorithms but with an edge preservation [34] effect and has been shown to alleviate conspicuous artifacts that may occur in GAN-synthesized images. The total variation penalty is computed as:

$${\mathcal{L}}_{tv}={\left|\nabla G_1(\tilde{\mathit{b}})\right|}_1.$$ (6)

3. Methods

3.1. Network Architecture

The SSSR network is implemented and trained on PyTorch using an NVIDIA GTX 1080 Ti graphics card. In the generators G₁ and G₂, each convolutional layer, except for the last ones, is followed by a leaky rectified linear unit (ReLU). The subnetworks G₁₁ and G₁₂ have only 2 convolutional layers with their outputs concatenated and fused through the third subnetwork G₁₃, which has 6 convolutional layers. A detailed representation of G₁ and G₂ is shown in Figure 2. The original VDSR network contains 20 convolutional layers. Each layer is followed by a leaky ReLU. The SSSR network removes the last layer and uses the output of the 19th layer from the VDSR network. D₁ and D₂ contain 9 convolutional and 3 fully-connected layers. Each convolutional layer is followed by a batch normalization layer (BN), a dropout layer (to prevent overfitting), and a leaky ReLU. Each fully-connected layer, except for the last one, is followed by a dropout layer and a leaky ReLU. To avoid sparse gradients, convolutional layers with stride 2 were adopted instead of using max-pooling layers [35]. A detailed representation of D₁ and D₂ is shown in Figure 3.

Figure 2:

Network architecture of G₁ and G₂. Each convolutional layer box shows the parameters in the format Conv(number of filters, filter size, stride, padding).

Figure 3:

Network architecture of D₁ and D₂. Each convolutional layer box shows the parameters in the format Conv(number of filters, filter size, stride, padding). In each fully-connected (FC) layer box, the only parameter is the number of filters.

3.2. Network Inputs

As mentioned in Section 2, the multi-channel input consists of an LR PET image, an HR MR image, and spatial (radial and axial) coordinates. We trained the SSSR network using 2D transverse patches from 3D brain images. To do this, we split the 3D images of size 192 × 192 × 128 × 4 into 128 192 × 192 × 4 slices (4 input channels). The input intensities were normalized into the range [0,1]. Data augmentation was achieved by randomly rotating the inputs by 1 to 360^◦ and randomly cropping them to the size of 96 × 96 × 4.

3.3. Network Training

To compute the network weights, training was performed for 1000 epochs using the following sequential order: {G₁,D₁,D₂}. The loss function for G₁ was minimized by applying the Adam solver [36], while that for D₁ and D₂ was minimized using stochastic gradient descent. G₁, D₁, and D₂ shared the same learning rate, which was set to 0.0002 and decreased by 1.5 × 10⁻⁷ at every epoch. The batch sizes of G₁ and G₂ were set to 10, and the batch sizes of D₁ and D₂ were set to 20.

3.4. Data Description

The SSSR network was trained using unpaired clinical LR and HR PET images. The HR PET images used are all based on the Siemens HRRT, a high-resolution dedicated brain PET scanner. The LR PET images corresponded to the Siemens ECAT EXACT HR+, an older but popular scanner model [37].

3.4.1. Clinical Data for SSSR Training and Validation

A total of 30 HRRT scans were obtained from the ADNI database (http://adni.loni.usc.edu/), of which 18 were used for training the SSSR network, while the remaining 12 were used for validation. The size of all HRRT PET images used were 192 × 192 × 128.

Similar to CycleGAN, the training data were equally split into two new training datasets. Each of the two training datasets had 9 subjects, while the validation dataset had 12 subjects. For the HR PET image samples, the 30 HRRT scans were all smoothed by a 2.4 mm full width at half maximum (FWHM) Gaussian filter. The LR PET images were computed by blurring the HRRT scans by the HR+ spatially-variant PSF measured in the image domain by placing point sources at multiple radial and axial locations inside the scanner bore [10]. G₁ was trained using the LR images from the first training dataset. D₁ was trained using the HR images from the second training dataset and the SR images that G₁ generated. D₂ was trained with the LR images from the first training dataset and the synthetic LR (LR’) images generated by G₂. For validation, G₁ was tested using the paired LR and HR images from the validation dataset.

3.4.2. Simulation Data for VDSR and G₂ Pretraining

The VDSR and G₂ networks were pretrained using paired, simulated LR and HR PET images. While VDSR performs LR-to-HR mapping, G₂ performs the reverse (HR-to-LR) mapping. The input and target image sets were therefore distinct for training the two networks. All pretraining was based on the BrainWeb digital phantom (http://brainweb.bic.mni.mcgill.ca/brainweb/). 20 distinct human brain atlases were used to generate a set of simulated HR PET images, one of which is shown in Figure 4. Further details on the simulations may be found in our previous publication [16]. The geometric models of the HRRT and HR+ scanners were used to generate sinograms, which were corrupted by Poisson noise. Image reconstruction was performed using ordered subsets expectation maximization (OSEM). The HRRT reconstruction results were smoothed using a 2.4 mm FWHM Gaussian filter to generate the HR PET images. The reconstructed HR+ PET images were blurred using the spatially-variant HR+ PSF to generate the LR PET images. These paired images along with spatial and anatomical inputs were then used for VDSR and G₂ pretraining.

Figure 4:

Transverse image slices from one of the 20 BrainWeb digital phantoms showing: A) HR PET, B) LR PET, C) HR MR, D) SR PET from the VDSR network.

4. Results

4.1. Reference Approaches

To evaluate the proposed SSSR framework, we compared the performance of the following techniques, which span partial volume correction, penalized deconvolution, supervised SR, and SSSR (multiple variants):

1. Region-based voxel-wise (RBV) correction: The RBV technique [6] is a voxel-based extension of the popular geometric transfer matrix (GTM) method [4], a popular partial volume correction technique used in PET. It computes the corrected image $\mathit{x}\in {ℝ}^N$ from an upsampled blurry image of the same size, $\stackrel{⌣}{\mathit{b}}\in {ℝ}^N$ as follows:

   $$\begin{gathered} \mathit{x}=\stackrel{⌣}{\mathit{b}}\odot [\mathit{s}\oslash (\mathit{s}\otimes \mathit{h})], \\ \mathit{s}=\sum _{i=1}^{N_{ROI}}{g}_i{\mathit{p}}_i. \end{gathered}$$ (7)

   Here $\mathit{h}\in {ℝ}^N$ is the approximate, spatially-invariant PSF, ⊙ and ⊘ represent Hadamard (entrywise) multiplication and division respectively, ⊗ represents 3D convolution, N_ROI = 2 represents the two gray and white matter anatomical regions-of interest (ROIs), p_i is a binary mask for the ith ROI (p_ij = 1 if voxel j belongs to ROI i, 0 otherwise), and g_i is the GTM-corrected intensity for the ith ROI[4].
2. Spatially-variant deconvolution with a total variation (TV) penalty: For an unknown SR PET image vectorized as $\mathit{x}\in {ℝ}^N$ and a scanner-reconstructed LR PET image $\mathit{b}\in {ℝ}^n$, the least-squares cost function for data fidelity is:

   $${\Phi }_{\text{PSF}}(\mathit{x}|\mathit{b})=\frac{1}{2}{(\mathit{A}\mathit{x}-\mathit{b})}^T(\mathit{A}\mathit{x}-\mathit{b}).$$ (8)

   Here N and n are the respective numbers of voxels in the SR/HR and LR images and $\mathit{A}\in {ℝ}^{n\times N}$ represents a blurring and downsampling operation based on the measured spatially-variant PSF of the scanner [10]. The two deconvolution techniques used as reference in this paper seek to minimize a composite cost function, Φ_PSF + βΦ_reg, where β is a regularization parameter and Φ_reg is a regularization penalty function. The TV penalty is defined as:

   $${\Phi }_{\text{reg}-\text{TV}}(\mathit{x})=\left(\Vert {\Delta }_1\mathit{x}{\Vert }_1+\Vert {\Delta }_2\mathit{x}{\Vert }_1+\Vert {\Delta }_3\mathit{x}{\Vert }_1\right),$$ (9)

   where Δ_k (k = 1,2, or 3) are finite difference operators along the three Cartesian coordinate directions [10].
3. Spatially-variant deconvolution with an MR-based joint entropy (JE) penalty: This uses the deconvolution framework in (8). Given an HR MR image, denoted by a vector $\mathit{y}\in {ℝ}^N$, the JE penalty is defined as [10]:

   $${\Phi }_{\text{reg}-\text{JE}}(\mathit{x}|\mathit{y})=-\sum _{i=1}^M\sum _{j=1}^M\delta u\delta vp\left(u_i,v_j\right)\text{log}p\left(u_i,v_j\right).$$ (10)

   Here $\mathit{u}\in {ℝ}^M$ and $\mathit{v}\in {ℝ}^M$ are intensity histogram vectors based on the PET and MR images respectively and M is the number of intensity bins [10].

4. VDSR: This is a previously validated, CNN-based SR model that is trained in a supervised manner with paired training samples [16]. The model is a modified version on the VDSR network [14] with anatomical and spatial inputs supplementing the LR PET input channel.

5. Lin et al. [18]: This is a state-of-the-art SR technique that uses unpaired training samples like our method but developed for and validated on natural images. Our initial experiments with this approach showed poor performance of this method in its original, unmodified form which uses a Lanczos kernel for HR-to-LR mapping. To make it more competitive and to ensure a fair comparison, we made two major changes to this technique: (1) we replaced the inbuilt Lanczos interpolation step by our pretrained generator network G₂ for HR-to-LR mapping and (2) we used 4-channel inputs similar to the proposed method.

6. SSSR network with SVPSF used for HR-to-LR mapping (SSSR-SVPSF): This variant of the proposed SSSR approach was implemented so as to be able to assess the contribution of using a pretrained generator G₂ for HR-to-LR mapping. The SSSR-SVPSF network is similar to the proposed SSSR network but with G₂ replaced by a spatially-variant 2D convolution step based on the measured image-domain SVPSF of the PET scanner [10].

7. SSSR without simulation guidance (SSSR-NoSim): This variant of the proposed SSSR approach was implemented so as to be able to assess the contribution of simulation guidance to overall SR performance. This SSSR-NoSim network is similar to the proposed SSSR network but lacks the extra inputs based on the simulation-based VDSR network.

8. SSSR network with simulation guidance (SSSR-Sim): This is the proposed SSSR network that features additional inputs from a VDSR network separately trained using simulation data generated using the BrainWeb phantom.

4.2. Evaluation Metrics

Conventional metrics for image-quality assessment rely on the knowledge of the “ground truth” used as a reference. This could be pose challenges for medical applications, particularly for self-supervised scenarios, where the ground truth may not be available for the validation dataset. To evaluate the SSSR technique, we therefore uses a mix of reference-based metrics and reference-free metrics described below:

4.2.1. Reference-Based Evaluation

We assess the image quality of the super-resolved PET images via two conventional and popular metrics in image processing and computed vision, namely the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM). In the metric definitions provided below, we denote the true and estimated images as x and $\hat{\mathit{x}}$ respectively. We use the notation μ_x and σ_x respectively for the mean and standard deviation of x. The PSNR is the ratio of the maximum signal power to noise power and is defined as:

$$\text{PSNR}(\hat{\mathit{x}},\mathit{x})=20{\text{log}}_{10}\left(\frac{\text{max}(\hat{\mathit{x}})}{\text{RMSE}(\hat{x},\mathit{x})}\right),$$ (11)

where the root-mean-square error (RMSE) is defined as:

$$\text{RMSE}(\hat{\mathit{x}},\mathit{x})=\sqrt{\frac{1}{N}\sum _k{\left({\hat{x}}_k-x_k\right)}^2}.$$ (12)

The SSIM is a well-accepted measure of perceived image quality and is defined as:

$$\text{SSIM}(\hat{\mathit{x}},\mathit{x})=\frac{\left(2{\mu }_x{\mu }_{\hat{x}}+c_1\right)\left(2{\sigma }_{x\hat{x}}+c_2\right)}{\left({\mu }_x^2+{\mu }_{\hat{x}}^2+c_1\right)\left({\sigma }_x^2+{\sigma }_{\hat{x}}^2+c_2\right)}.$$ (13)

Here c₁ and c₂ are parameters stabilizing the division operation.

4.2.2. Reference-Free Evaluation

As the rationale for constructing an SSSR methodology is the dearth of available reference ground-truth images, we also evaluate SR images via two reference-free metrics. The first of these is the contrast-to-noise ratio (CNR), a widely used figure-of-merit for PET image quality assessment. The CNR for a target ROI $\mathcal{R}$ and a reference ROI ${\mathcal{R}}_{\text{ref}}$ is defined as:

$$\text{CNR}=\frac{\left|{\mu }_{\hat{\mathit{x}}}^{\mathcal{R}}-{\mu }_{\hat{\mathit{x}}}^{{\mathcal{R}}_{\text{ref}}}\right|}{\sqrt{[{({\sigma }_{\hat{\mathit{x}}}^{\mathcal{R}})}^2+{({\sigma }_{\hat{\mathit{x}}}^{{\mathcal{R}}_{\text{ref}}})}^2]}}.$$ (14)

In this work, the gray matter ROI is treated as the target and the white matter ROI as the reference.

In addition, we employed a second reference-free metric for SR images (NRM-SR) which was specifically developed for SR image evaluation with the intent of detecting potential SR-based artifacts [38]. NRM-SR exploits three statistical features to quantify artifacts and evaluate the quality of SR images. These features capture local and global frequency as well as spatial frequency features. The features are then modeled on independent regression forests and regressed on perceptual scores to predict SR image quality.

4.3. Performance Comparison

Figure 5 and Figure 6 show transverse slices and corresponding magnified areas of these slices derived from the HR MR, LR PET, HR PET, and the outputs of the eight methods described in Section 4.1. The PSNR, SSIM, CNR, and NRM-SR, which are the quantitative figures-of-merit used in this study, are presented in Table 1.

Figure 5:

Transverse image slices from a human subject showing: A) HR MR, B) LR PET, C) HR PET, D) RBV, E) TV, F) JE, G) VDSR, H) Lin et al. [18], I) SSSR-SVPSF (G₂ replaced by SVPSF), J) SSSR-NoSim (no simulation guidance), and K) SSSR-Sim (the proposed method with simulation guidance). The red box in the MR image indicates the region that is magnified for closer inspection in Figure 6.

Figure 6:

Magnified subimages corresponding to the red box shown in Figure 5: A) HR MR, B) LR PET, C) HR PET, D) RBV, E) TV, F) JE, G) VDSR, H) Lin et al. [18], I) SSSR-SVPSF (G₂ replaced by SVPSF), J) SSSR-NoSim (no simulation guidance), and K) SSSR-Sim (the proposed method with simulation guidance).

Table 1:

Comparison of different SR imaging methods for PET

Metric	LR	HR	RBV	TV	JE	VDSR	Lin et al. [18]	SSSR-SVPSF	SSSR-NoSim	SSSR-Sim
PSNR	29.65	∞	30.52	33.26	33.47	37.21	34.13	34.17	34.70	35.57
SSIM	0.873	1.000	0.712	0.915	0.917	0.961	0.931	0.926	0.933	0.941
CNR	0.154	0.455	0.236	0.262	0.238	0.405	0.306	0.299	0.302	0.341
NRM-SR	2.302	2.676	2.6693	2.449	2.616	2.743	2.581	2.547	2.553	2.642

As expected, VDSR, a supervised technique, steadily outperformed all methods from the self-supervised family. The latter approaches, however, exhibited substantial improvement over classical deblurring and partial volume correction not only in terms of image quality but also in terms of quantitation via the reference-based metrics PSNR and SSIM. Within the self-supervised family of methods tested (including all SSSR variants and Lin et al. [18]), SSSR-Sim is visually the sharpest and also leads to higher PSNR and SSIM. Of note are its clear qualitative and quantitative advantages over the variants SSSR-SVPSF and SSSR-NoSim. This clearly indicates that the incorporation of simulation-based VDSR features and G₂ pretraining greatly increase the proposed network’s accuracy. Overall, among the self-supervised family of methods, Lin et al. [18] and SSSR-SVPSF exhibited comparable performance. SSSR-NoSim is a more competitive approach than Lin et al. [18] and SSSR-SVPSF, while SSSR-Sim is the clear winner in the self-supervised category.

Among the reference-free metrics, CNR leads to relative performance trends across all the compared methods that are broadly similar to the PSNR trends. NRM-SR, however, is less predictable. While it suggests the strongest performance for VDSR, RBV, JE, and SSSR-Sim are close contenders for second place. This can be attributed to the fact that the NRM-SR measure is higher for methods that more effectively incorporate anatomical information. While all other metrics seem to suggest that network-based methods vastly outperform non-network-based ones, the NRM-SR outcome is an aberrant one. It should be noted that a reference-free metric such as NRM-SR is unconstrained by a reference image and may lead to the SR image earning a score higher than HR image as is the case with VDSR image.

5. Discussion

We have developed a self-supervised technique for super-resolving PET images which obviates the need for paired inputs for training and therefore promises wide practical utility. The method was validated using clinical data. While PSNR and SSIM are the predominantly used image quality assessment metrics for image processing and computer vision, these metrics are not always well-aligned with human perceptual assessment [39]. They tend to be sensitive to noise and thus perhaps are more suggestive of denoising performance than deblurring performance, the latter being our aim. As the rationale for designing a self-supervised methodology is the absence of ground-truth images, we extend the image quality assessment to include two reference-free metrics. Not unexpectedly, supervised learning using VDSR led to the best performance for all metrics. But our proposed algorithm SSSR-Sim was a clear winner among all classical and self-supervised approaches, including partial volume correction, penalized deconvolution, and a state-of-the-art SSSR technique developed for and validated on natural images. SR is particularly important for a functional/physiological imaging modality such as PET, for which the resolution limitations tend to be more severe than anatomical modalities such as MR or CT. The ability to generate SR images without paired training datasets makes our method particularly useful for clinical practice.

Self-supervised learning is a fundamentally difficult problem. A key characteristic of our network is the incorporation of high-dimensional features from a VDSR network trained using simulation data. To test the utility of this specific modification, we compared our network (SSSR-Sim) with a simpler network that does not use simulation-based features (SSSR-NoSim). The latter resembles the CycleGAN architecture. Our results show that the simulation-based features are vital for ensuring high performance in self-supervised mode. Simulation guidance to achieve SR is, therefore, an important contribution of this work.

Another distinctive feature of the proposed method is LR-to-HR mapping via generator pretraining using simulation data prepared via blurring with the measured SVPSF of the PET scanner. To determine the relative efficacy of a generator network over a classical blurring operator, we compared our network (SSSR-Sim) with a similar network that replaces the generator G₂ by SVPSF blurring (SSSR-SVPSF). Our results show that the pretrained generator serves to do more than simple LR-to-HR mapping and provides additional features guidance to the overall network. G₂ pretraining using domain-specific knowledge is therefore yet another important and novel contribution of this paper.

Our evaluations using the reference-free metric NRM-SR led to an interesting outcome. Our results showed that methods like RBV and JE that rely heavily on MR guidance tend to benefit disproportionately and outperform several network-based methods when adjudicated via NRM-SR. This result is suggestive of the inadequacy of any single metric to adjudge specific algorithms and the need for designing clinically meaningful task-based metrics. The development of such a metric for SR PET imaging is our intended future task.

It should also be noted that the demonstrated SSSR performance is specific to inputs with noise and blur levels similar to those used in training. Our future work will characterize the performance of these networks when input noise levels are varied. Due to the computational overhead of training two GANs, we resorted to 2D convolutions instead of 3D, a scenario that is sub-optimal. Finally, this proof-of-concept study was based on a small cohort. As future work, we will perform more extensive validation using a larger dataset. Despite these limitations, given the fundamental need for self-supervised approaches while dealing with clinical datasets, we envision wide applicability of our method in the clinical domain.