A Pilot Study: PET/CT Cross-Modal-Based Multi-Head Fusion Attention Generative Adversarial Network (MHFA-GAN) for PET Image Super-Resolution

Abstract

Positron emission tomography (PET) is a valuable imaging technique for assessing cellular metabolic activity and is widely used for early disease detection and diagnosis. While high-resolution (HR) PET images offer richer diagnostic information, achieving such quality often requires expensive equipment and time-intensive reconstruction processes. To address these challenges without hardware upgrades, this paper introduces a novel multi-head fusion attention generative adversarial network (MHFA-GAN), designed to enhance the spatial resolution and reconstruction efficiency of PET images through super-resolution (SR) techniques. The MHFA-GAN employs a multi-head mixed convolution mechanism to extract complementary features from PET and computed tomography (CT) images. An enhanced multi-head fusion attention module (EMHFA) is incorporated to adaptively adjust feature weights and align cross-modal information, effectively minimizing the impact of sub-pixel displacement while integrating metabolic and anatomical details. During the reconstruction phase, a global residual block (GRB) is utilized to balance local high-frequency details with global contextual information. Experiments conducted on small animal (SA), phantom, capillary, and ²²Na point source datasets demonstrate that MHFA-GAN outperforms current state-of-the-art methods in both qualitative and quantitative evaluations. Additionally, the model also shows significant results in SR tasks on different device datasets, further verifying its cross-device applicability and robustness. Using the full width at half maximum (FWHM) metric to assess spatial resolution, the SR images from the capillary dataset achieved a FWHM value of 0.625 mm, approaching the physical limit of the SA PET scanner (0.6 mm) and significantly outperforming the HR images (0.871 mm). These results highlight the effectiveness of MHFA-GAN in improving PET image quality, offering a cost-effective solution to enhance spatial resolution and support clinical diagnostics.

Introduction

PET is a highly sensitive, non-invasive imaging technique that provides valuable insights into the metabolic and physiological tissue functions using radioactive tracers. PET detects dual gamma rays emitted during the interaction of positrons and electrons as isotopes decay, generating functional images after reconstruction. Clinically, it is widely used for diagnosis and post-treatment evaluation due to its ability to capture metabolic activity directly [1, 2]. However, PET images generally have lower spatial resolution than CT or magnetic resonance imaging (MRI) [3], constrained by the physical limitations of positron emission and hardware [4]. It is well known that spatial resolution is essential for PET’s diagnostic accuracy, enabling clear visualization of tissue structure and function. HR PET enhances lesion detection, particularly for small or early-stage abnormalities.

Figure 1 displays CT and PET images from SA PET/CT scans of a mouse, highlighting the markedly lower resolution of PET images compared to CT images. Both the low-resolution (LR) PET and HR PET images were reconstructed using the ordered subset expectation maximization (OSEM) method, differing only in iteration counts. While fewer iterations reduced reconstruction time, the visual quality of the resulting PET images was significantly inferior to those reconstructed with the standard iteration number. PET resolution is constrained by the physical and technical limitations of imaging equipment, affecting the accurate detection of small lesions and biological processes. Moreover, achieving high-quality PET images often involves lengthy reconstruction times, potentially delaying clinical diagnoses. Therefore, optimizing imaging systems and refining reconstruction parameters is essential to enhance spatial resolution and expedite reconstruction, improving clinical efficiency and supporting accurate diagnosis and treatment planning.

Fig. 1.

From left to right, the images display a small animal: CT data, HR PET images reconstructed with five iterations, and LR PET images reconstructed with one iteration

To address the problem, two primary strategies can be considered. The first involves upgrading to advanced imaging equipment, which is often costly and impractical for widespread clinical adoption. The second approach leverages computer vision and image processing techniques, specifically SR algorithms. These algorithms reconstruct HR images from LR data, effectively overcoming the physical and technical limitations of existing imaging systems without requiring hardware upgrades. This method significantly enhances image quality and detail recovery, making it a promising post-processing solution for improving the spatial resolution of medical images, particularly PET, in clinical applications.

With the rapid advancement of deep learning (DL) [5], neural networks have become widely used in image SR tasks. Early convolutional neural networks (CNNs), such as those in [6–10], showed some success in enhancing resolution but were limited by their inability to fully capture global information across layers and their restricted receptive fields. These limitations made it challenging to capture complex, long-range relationships, and fine details effectively. To overcome this, Zhang et al. [11] introduced a residual dense network (RDN), which builds on the principles of residual learning [12] and dense connections [13]. This architecture integrates multi-scale information and captures global dependencies by utilizing the hierarchical features of deep models, resulting in significant improvements in detail restoration. Building on this concept, Behjati et al. developed OverNet [14], employing cascaded residual dense blocks to extract multi-scale features and enrich contextual information. OverNet also introduced wide low-rank convolutions, and combined local and global dense groups to aggregate features across different scales effectively. However, its focus on long-range feature aggregation limited its ability to independently handle local high-frequency details, affecting performance in restoring fine textures. To address this limitation, the hybrid attention transformer (HAT) model combines channel attention with window-based self-attention mechanisms, utilizing overlapping cross-attention modules to enhance information exchange between adjacent windows [15]. This design optimizes spatial correlations and improves the network’s ability to process high-frequency details, making it outperform previous state-of-the-art SR methods.

Despite significant advancements in DL for SR in natural images, these models often struggle with medical imaging due to its complex textures and higher resolution demands. Therefore, further development of SR methods tailored for medical applications is essential [16, 17]. For example, Chen et al. [18] introduced a 3D densely connected SR network (DCSRN) for enhancing brain MR images, significantly improving image quality. Similarly, Shi et al. [19] developed a progressive wide residual network (FSCWRN) with skip connections to improve MRI reconstruction, achieving both global consistency and local detail preservation. Other notable efforts include Hatvani et al.’s [20] CNN architectures for CT resolution enhancement and Zhang et al.’s [21] squeeze and excitation reasoning attention network (SERAN) for MRI. However, these approaches focus solely on single-modality images, limiting the potential of SR improvements by not incorporating complementary information from other modalities. Single-modality SR methods, particularly for PET, often face challenges in detecting subtle lesions. By integrating CT or MRI data, the resolution of PET images can be significantly improved. CT and MRI provide detailed anatomical structures, while PET supplies metabolic information, allowing more accurate lesion localization and analysis [22, 23]. For instance, Song et al. [24] advanced PET SR by incorporating anatomical data using CNNs, but this approach struggled to recover fine details. Song et al. [25] later introduced a self-supervised PET SR method using dual GANs with MRI priors to enhance detail recovery. Further progress was made by Bi et al. [22], who developed a multi-channel GAN (M-GAN) that processes multimodal inputs with independent channels and uniform convolution kernels. LA-GANs (local adaptation generative adversarial networks) [26] similarly integrate MR images to reduce noise, extract input modality features with a 1 × 1 × 1 convolution kernel, and apply a local adaptive network for modality fusion. However, many multimodal approaches rely on basic image concatenation [27], which fails to account for the inherent differences between modalities, limiting their effectiveness. Our work aims to address this limitation by fully leveraging the metabolic information from LR PET and the anatomical structure provided by CT images. Recent approaches, such as the multimodal multi-head convolutional attention mechanism (MMHCA) [28], have improved reconstruction quality by using convolution kernels of varying sizes to capture multimodal features. However, challenges remain in handling complex, long-range relationships, and addressing sub-pixel misalignments between modalities, which can result in artifacts or blurring. Minimizing the impact of these misalignments on multimodal feature extraction has become a key motivation for our work.

To address these aforementioned challenges, we propose a multi-head feature attention module-based multimodal GAN (MHFA-GAN) to directly reconstruct SR PET images from LR PET and CT data. Our goal is to improve reconstruction quality, restore image details accurately, and enhance the spatial resolution, even surpassing that of HR PET images. By incorporating CT images as auxiliary inputs, we leverage additional anatomical information to enhance reconstruction accuracy. The MHFA-GAN generator employs a multi-head hybrid convolution mechanism (see Fig. 2). To focus on metabolic features and minimize interference from spatial details, we use a convolution operation with a kernel size of 1 for PET images. For CT images, we utilize a multi-scale feature extraction approach with kernel sizes of 3 and 5 to capture complex edges and textures. To better fuse features from different modalities, we introduce an enhanced multi-head fusion attention (EMHFA) module, which dynamically adjusts feature weights and aligns cross-modal information, addressing sub-pixel misalignments and improving spatial consistency. In the reconstruction phase, we include a plug-and-play GRB that combines residual structures with attention mechanisms, enhancing the model’s capability to restore high-frequency details while retaining global information. For the discriminator, we adopt a U-Net architecture to enhance GAN stability, effectively integrating global and local information while preserving HR features. We evaluated the MHFA-GAN model on four datasets: SA, phantom, capillary, and ²²Na point source. The evaluation employed both full-reference and no-reference metrics to provide a comprehensive analysis of the model's performance.

Fig. 2.

The multi-head hybrid convolution mechanism in MHFA-GAN

In summary, our work presents four key innovations:

1. This study proposes an innovative MHFA-GAN, which effectively integrates the metabolic information from PET images with the anatomical structure provided by CT images. By overcoming the hardware and resolution limitations of traditional PET image reconstruction methods, this model achieves a significant enhancement in spatial resolution while reducing reconstruction time. It offers a cost-effective and efficient solution for improving PET image quality.

2. We developed a MHFAM to overcome the drawbacks of single-modal features and the difficulties in extracting and fusing multimodal or multi-scale data. This module addresses differences in multimodal features and mitigates sub-pixel displacement effects by dynamically adjusting feature weights and adaptively aligning cross-modal information.

3. Acknowledging the limitations of traditional residual networks in capturing local high-frequency details, we designed a plug-and-play GRB. This module effectively fuses global and local information, significantly improving the model’s ability to process high-frequency details while preserving overall context.

4. To validate the broad applicability of the MHFA-GAN model, SR experiments were conducted on datasets from various devices. The results demonstrate its robustness and adaptability across platforms. Even in the absence of multimodal data, the single-modality MHFA-GAN model significantly outperforms traditional filtered back projection (FBP) reconstruction, showing substantial improvements in spatial resolution while ensuring computational efficiency and practical applicability.

Methods

Mathematical Model

Enhancing the spatial resolution of PET images can be understood as applying SR techniques to LR PET images to make them comparable to HR images. The aim of this study is to identify a function f that maps the distribution $D_{\mathit{LR}}^{\mathit{PET}}$ of LR PET images $I_{\mathit{LR}}^{\mathit{PET}}$ to the distribution $D_{\mathit{HR}}^{\mathit{PET}}$ of HR PET images $I_{\mathit{HR}}^{\mathit{PET}}$, as shown in Eq. (1).

$$f(D_{\mathit{LR}}^{\mathit{PET}})=D_{\mathit{HR}}^{\mathit{PET}}$$ 1

When constructing the GAN network for the multimodal PET image SR task, we need to optimize a generator $G$ with parameters ${\theta }_G$ that can simulate f described in Eq. (1) and a discriminator $D$ with parameters ${\theta }_D$, so that the generated SR images closely resemble the true HR images $I_{\mathit{H}R}^{\mathit{PET}}$. Assume that the generator $G$ of this task receives $I_{\mathit{LR}}^{\mathit{PET}}$ and corresponding CT images $I^{\mathit{C}T}$ as input and generates SR PET images $G_{{\theta }_G}\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{C}\mathit{T}}\right)$ after feature extraction. The generator’s objective is to minimize the discrepancy between $G_{{\theta }_G}\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{C}\mathit{T}}\right)$ and $I_{\mathit{H}R}^{\mathit{PET}}$, as defined by the loss function l in Eq. (2).

$$\mathit{arg}\underset{{\theta }_G}{\mathit{min}}(\sum l(G_{{\theta }_G}(I_{\mathit{LR}}^{\mathit{PET}},I^{\mathit{CT}}),I_{\mathit{HR}}^{\mathit{PET}}))$$ 2

The discriminator, on the other hand, seeks to maximize its ability to differentiate between $I_{\mathit{H}R}^{\mathit{PET}}$ and generated images, as detailed in Eq. (3).

$$\mathit{arg}\underset{{\theta }_D}{\mathit{max}}\left(l,o,g,D_{{\theta }_D},\left(I_{\mathit{HR}}^{\mathit{PET}}\right)\right)$$ 3

Overall, the optimization objective for using GANs in LR PET image SR is shown in Eq. (4).

$$\underset{{\theta }_G}{\mathit{min}}\underset{{\theta }_D}{\mathit{max}}{E}_{I_{\mathit{HR}}^{\mathit{PET}}\sim p_{\mathit{data}}(I_{\mathit{HR}}^{\mathit{PET}})}[log{D}_{{\theta }_D}(I_{\mathit{HR}}^{\mathit{PET}})]+E_{\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{CT}}\right)\sim {p_{\theta }}_G\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{CT}}\right)}[\mathit{log}(1-D_{{\theta }_D}({G_{\theta }}_G(I_{\mathit{LR}}^{\mathit{PET}},I^{\mathit{CT}}))]$$ 4

Network Architecture

In this section, a MHFA-GAN based on a multi-head feature attention module for PET image SR is proposed by learning fusion features from PET and CT images. As shown in Fig. 3, the model is composed of two main parts, and details of each part in MHFA-GAN are given as follows.

Fig. 3.

The architecture of the proposed MHFA-GAN

Generator

Our generator consists of two key modules: feature extraction and reconstruction, which aim to capture and reconstruct detailed image features effectively. In the feature extraction stage, we designed MHFAM (multi-head feature attention module), which includes a multi-head feature extractor and an enhanced multi-head fusion attention (EMHFA) module. The multi-head extractor captures shallow features from different input modalities, providing the necessary inputs for efficient fusion in the EMHFA module. These features serve as the query (Q), key (K), and value (V) at various levels, facilitating precise and adaptable feature integration across modalities. In the reconstruction phase, we build on the DRRN [29] architecture and introduce an innovative residual-in-residual structure called GRB. Unlike the DRRN’s DRRB module, the GRB incorporates deeper, hierarchical residual connections, enabling the model to perform more nuanced, layered learning. This structure strengthens feature representation and significantly enhances the SR reconstruction performance.

A. Multi-Head Feature Attention Module

To reconstruct detailed features, we designed a multi-head feature extraction module that captures features from different modalities of the input image, as shown in Figs. 2 and 3 (a). Each head within this module focuses on extracting unique characteristics, supporting both multimodal and single-modal images and enabling multi-scale feature extraction. This flexible design allows the integration of any neural architecture layer, making the module universally adaptable. In our experiments, we applied an ablation configuration to demonstrate the advantages of multimodal input over single-modal input. The feature extraction module consists of multiple heads, denoted as $h_j$ for $j\in \{1,2,\cdots ,h\}$, where $h$ represents the total number of heads. This structure provides a robust framework for effectively capturing rich, detailed features across different image modalities. Each head uses convolution kernels of varying sizes, which include convolutional, spectral normalization (SN), activation, and deconvolutional layers. The convolutional layers extract multi-scale local features by compressing the input tensor in both spatial and channel dimensions, while the deconvolutional layers restore the original feature map size to preserve spatial details. For the $j$-th convolutional head $h_j$, we set the kernel size $k_j$ to $2\bullet \left(j,-,1\right)+1$. This means that the first head has a $1\times 1$ receptive field, the second head has a $3\times 3$ receptive field, and so forth. We apply a stride of 1 and set padding to 0 for both convolutional and deconvolutional layers. The output of the $j$-th head, $h_j$, is defined as

$$h_j={\mathit{Deconv}}_{k_j}\left(A,F,\left(S,N,\left({\mathit{Conv}}_{k_j},\left(x_j\right)\right)\right)\right),$$ 5

where $x_j$ is the $j$-th input image, ${\mathit{Conv}}_{k_j}$ and ${\mathit{Deconv}}_{k_j}$ represent the convolution and deconvolution operations with a $k_j\times k_j$ kernel, $\mathit{SN}$ denotes the spectral normalization layer, and $\mathit{AF}$ is the LeakyReLU activation function with $\alpha =0.2$. The feature extraction module’s output for each head can then be expressed as:

$$F_j={\mathit{Conv}}_1\left(A,F,\left(S,N,\left({\mathit{Conv}}_3,\left(h_j\right)\right)\right)\right),$$ 6

where $F_j$ represents the output of $3\times 3$ and $1\times 1$ convolutions with spectral normalization and LeakyReLU activation applied sequentially. In summary, the output of the convolutional head $H_j$ in the multi-head feature extraction module is calculated as

$$H_j=F_j(h_j(x_j))$$ 7

This configuration enables each head to effectively extract distinct features, contributing to robust multi-scale feature extraction across both single-modal and multimodal inputs.

In the EMHFA module, shallow features $h_1$, $h_2$, and $h_3$ obtained from the multi-head feature extraction stage are mapped to query (Q), key (K), and value (V) representations, enabling feature fusion across modalities. The multi-head attention mechanism [30] computes a cross-correlation matrix among these tensors to capture inter-modal relationships. Next, the soft-attention module refines these interactions by processing the key features with the cross-correlation matrix, calculating similarity scores, and normalizing them via the softmax function to obtain weight. These weights are used to perform a weighted summation over the value features, producing a representation that captures the association between the CT and LR PET images. Through this process, the soft-attention module adaptively highlights important features, facilitating effective fusion of information from different modalities. Finally, the resulting association matrix is combined with the query tensor, and after sigmoid activation, connected to the original query features via a residual connection. This fused representation enhances cross-modal feature interaction, supporting the SR reconstruction of PET images.

B. Global Residual Block

The global residual block (GRB) is designed to enhance feature representation and strengthen the reconstruction of feature information through residual learning. As shown in Fig. 3, the GRB consists of three residual groups (RGs), a convolutional layer, and a convolutional block attention module (CBAM) [31]. Each RG includes nine serially connected residual blocks (RBs) and a $1\times 1$ convolutional layer, performing multi-level feature extraction through the sigmoid function. The output information $f_{g,RB}$ of the $g$-th RB module can be defined as follows:

$$f_{g,RB}={(H}_{g,RG}(f_{g-1,RB}))=H_{g,RB}(H_{g-1,RB}(\cdots H_{1,RB}(f_{0,RB}))),$$ 8

where $H_{g,RB}$ represents the operation in the $g$-th RB and $x$ is the input vector. Our innovative RB module consists of two layers of $3\times 3$ convolutions, each followed by a LeakyReLU activation function to improve the network’s nonlinear expression capability. To further refine feature selection, we introduce the squeeze-and-excitation (SE) module [32] after the second convolutional layer in the RB. The SE module applies a “squeeze and excitation” mechanism that adaptively weights features along the channel dimension, enabling the network to focus on essential feature channels. This mechanism compensates for the lack of inter-channel relationship modeling in traditional convolutional operations. Finally, the SE module output is added to the initial input to form a residual connection. This result is then processed by a $1\times 1$ convolutional layer to yield the final output $f_{\mathit{RB}}$ for the RB module, as illustrated by

$$f_{\mathit{RB}}={\mathit{Conv}}_1{[SE(AF(Conv}_3(AF({\mathit{Conv}}_3(x)))))+x]$$ 9

Discriminator

This study adopts a U-Net architecture [33] for the GAN discriminator, diverging from conventional GAN discriminators to enhance discriminative capacity and training stability in PET image SR. The U-Net’s multi-scale feature fusion provides nuanced feedback to the generator, aiding in robust convergence. Key components include a $3\times 3$ convolutional layer, batch normalization, dropout, and LeakyReLU activation. The encoding–decoding structure allows progressive extraction of high-level features while preserving detail. During downsampling, convolutions reduce spatial resolution and increase channels, effectively capturing rich local features and edges. Residual connections (green arrows in Fig. 3 (b)) merge downsampled and upsampled features, preserving low-level details and enriching multi-scale representation. The upsampling phase restores spatial resolution and incorporates multi-scale features to retain detail and improve discrimination accuracy. The discriminator outputs a single authenticity probability, guiding the generator to iteratively refine image realism and detail quality.

Loss Function

According to Eqs. (1) and (2), the proposed model defines a mapping function from $I_{\mathit{LR}}^{\mathit{PET}}$ to $I_{\mathit{HR}}^{\mathit{PET}}$. In this framework, the generator $G$ is optimized by minimizing the image loss $L_{\mathit{img}}$ between the ground truth (GT) images $I_{\mathit{HR}}^{\mathit{PET}}$ and estimated images $I_{\mathit{SR}}^{\mathit{PET}}$. Here $I_{\mathit{HR}}^{\mathit{PET}}=\{I_{\mathit{HR}}^{PET1},I_{\mathit{HR}}^{PET2},\cdots ,I_{\mathit{HR}}^{\mathit{PETn}}\}$, $I_{\mathit{SR}}^{\mathit{PET}}=\{G\left(I_{\mathit{LR}}^{PET1},,,I^{CT1}\right),G\left(I_{\mathit{LR}}^{PET2},,,I^{CT2}\right),\cdots ,G\left(I_{\mathit{LR}}^{\mathit{PETn}},,,I^{\mathit{CTn}}\right)\}$. To quantify this difference, we employ the mean squared error (MSE) loss function to compute the pixel-wise discrepancy, as formulated in Eq. (10).

$$L_{\mathit{img}}=\frac{1}{n}{\Vert I_{\mathit{SR}}^{\mathit{PET}}-I_{\mathit{HR}}^{\mathit{PET}}\Vert }_F^2$$ 10

We define a perceptual loss function $L_{\mathit{per}}$ to enhance the visual fidelity of $I_{\mathit{SR}}^{\mathit{PET}}$ by leveraging high-level features extracted from a pre-trained VGG16 network.

Specifically, we use the feature extraction layers of VGG16 up to the 30th layer, which capture complex and semantically rich representations of image content. The $L_{\mathit{per}}$ is calculated as the MSE between the feature maps of the $I_{\mathit{HR}}^{\mathit{PET}}$ and $I_{\mathit{SR}}^{\mathit{PET}}$. By aligning feature representations at this high-level semantic stage, $L_{\mathit{per}}$ guides the SR model to produce outputs that are not only close to HR images but also perceptually consistent in structural details and texture. The $L_{\mathit{per}}$ is defined in Eq. (11), where $\varnothing$ represents the output from the 30th layer of the pre-trained VGG16 network.

$$L_{\mathit{per}}=\frac{1}{n}{\Vert \varnothing (I_{\mathit{SR}}^{\mathit{PET}})-{\varnothing (I}_{\mathit{HR}}^{\mathit{PET}})\Vert }_F^2$$ 11

Then, we define the adversarial loss $L_{\mathit{adv}}$, as shown in Eq. (12).

$$L_{\mathit{adv}}=E_{\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{CT}}\right)\sim {p_{\theta }}_G\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{CT}}\right)}[1-{G_{\theta }}_G(I_{\mathit{LR}}^{\mathit{PET}},I^{\mathit{CT}}))]$$ 12

Finally, the generator loss $L_G$ can be expressed as Eq. (13), where ${\alpha }_{\mathit{img}}$, ${\alpha }_{\mathit{per}}$, and ${\alpha }_{\mathit{adv}}$ are weighted-factors of different loss functions.

$$L_G={\alpha }_{\mathit{img}}{L}_{\mathit{img}}+{{\alpha }_{\mathit{per}}L}_{\mathit{per}}+{{\alpha }_{\mathit{adv}}L}_{\mathit{adv}}$$ 13

Inspired by the Hinge loss function [34], we propose an improved discriminator loss function $L_D$ to enhance the performance of the GAN-based PET SR model. Specifically, the discriminator loss function is implemented as Eq. (14).

$$L_D=E_{I_{\mathit{HR}}^{\mathit{PET}}\sim p_{\mathit{data}}(I_{\mathit{HR}}^{\mathit{PET}})}[ReLU(1-{D_{\theta }}_D(I_{\mathit{HR}}^{\mathit{PET}})]+E_{\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{CT}}\right)\sim {p_{\theta }}_G\left(I_{\mathit{LR}}^{\mathit{PET}},,,I^{\mathit{CT}}\right)}[ReLU(1+{{{D_{\theta }}_D(G}_{\theta }}_G(I_{\mathit{LR}}^{\mathit{PET}},I^{\mathit{CT}})))]$$ 14

This loss function employs the ReLU activation rather than strict boundary constraints, which reduces the discriminator’s sensitivity to minor score fluctuations and improves training stability. Even when predictions approach the target boundary, the Hinge loss maintains gradient flow, mitigating the vanishing gradient problem. As a result, the loss function enhances both the robustness of training and the discriminator’s effectiveness in handling PET image data.

Evaluation Metrics

To evaluate the performance of the proposed models on SA, we employed three widely used metrics: PSNR, RMSE, and SSIM. These metrics were selected to quantify the differences between the ground-truth $I_{\mathit{HR}}^{\mathit{PET}}$ and estimated images $I_{\mathit{SR}}^{\mathit{PET}}$. PSNR measures pixel-level fidelity using the peak intensity of the image (MAX) and the mean square error (MSE), expressed in decibels (dB). RMSE reflects the average error magnitude as the square root of the MSE. SSIM assesses image similarity by considering luminance, contrast, and structural information.

In this study, we utilized the FWHM to assess spatial resolution in phantom, capillary, and ²²Na datasets, where smaller FWHM values indicate higher spatial resolution. The choice of fitting function for FWHM calculation was tailored to the intensity distribution characteristics observed in the phantom, capillary, and ²²Na datasets. For the phantom dataset, the pixel intensity along the vertical line through the highlighted point exhibited an asymmetric distribution (as shown in Fig. 4). This asymmetry, characterized by extended tails in the intensity profile, cannot be accurately modeled by symmetric distributions such as the Gaussian function. To precisely capture this asymmetric and extended intensity distribution, the Lorentzian function, which is known for its long tails and symmetric peak structure, was employed for fitting. Conversely, in the capillary and ²²Na datasets, the intensity distribution for individual points was symmetrical and localized, a common pattern for point sources with a well-defined central peak. Given this symmetry and localized nature of the intensity distribution, the Gaussian function, known for its ability to fit symmetric, bell-shaped profiles, was ideal for accurately capturing this pattern and ensuring precise FWHM measurement.

Fig. 4.

The left and right images are the pixel intensity distribution observed in the HR phantom image and HR capillary image, respectively

This targeted fitting method ensures that the fitting function aligns closely with the actual intensity distribution characteristics in each dataset, thereby significantly improving the accuracy and reliability of the FWHM calculation. By employing the Lorentzian function for the asymmetric phantom dataset and the Gaussian function for the symmetric capillary and ²²Na datasets, we achieved optimal fitting for each case, leading to more precise spatial resolution assessments across different imaging conditions.

Experiments

Experimental Settings

Training Details

Our proposed MHFA-GAN model was implemented using the PyTorch 1.9 framework. Network training and testing were conducted on a single NVIDIA 3080 Ti GPU. We used the Adam [35] optimizer to facilitate efficient training, setting the total number of training epochs to 100. The initial learning rate was set to 8e − 5 and decayed exponentially throughout training. Specifically, the learning rate was reduced by approximately 3% every 10 epochs by multiplying it by a decay factor of 0.97 at each interval. To enhance stability during training, image data were normalized to a range of [− 1, 1]. ${\alpha }_{\mathit{img}}$, ${\alpha }_{\mathit{per}}$, and ${\alpha }_{\mathit{adv}}$ are, respectively, 1e − 1, 8e − 4, and 1e − 4. For experiments on the SA dataset, the batch size was set to 8, and the patch size for training data was $64\times 64$. Validation and test datasets used the full image size of $257\times 257$.

Datasets

The SA, phantom, and capillary datasets in this study were acquired using the Metis™ Small Animal PET/CT system. PET images were reconstructed with the OSEM algorithm using 20 iterations and a matrix size of $257\times 257\times 389$. The scanning settings of CT images were 70 kVp tube voltage and 5 mA tube current. The SA dataset was used for training, validation, and testing to assess the performance of SR models in enhancing PET image quality. The SA PET dataset included scans from ten mice, anesthetized with chloral hydrate, injected with 500 µCi 18F-FDG, and allowed a 10-min uptake period before undergoing 20 min of scanning. For model training, data from seven mice were used, with one mouse reserved for validation and two for testing. In multi-modality PET and CT input, the data volume of both modalities remains consistent. To evaluate the model’s ability to improve spatial resolution, phantom and capillary datasets were included as test datasets. Before scanning, the phantom was injected with 400 µCi of ¹⁸F-FDG, while the 0.1 mm capillary was injected with 200 µCi of ¹⁸F-FDG, and both were scanned for 20 min. HR images for all datasets were reconstructed using seven subsets with five iterations per subset. For LR images, three cases were defined based on the reconstruction settings: case 1 used images reconstructed with seven subsets of one iteration, case 2 used images reconstructed with seven subsets and two iterations, and case 3 used images reconstructed with seven subsets and three iterations.

To evaluate the effectiveness of our model in SR tasks across different devices, we used ²²Na (sodium-22) point source data acquired from a small animal 7 T PET/MRI system as the test set. Spatial resolution was assessed by performing FWHM measurements. The system was manufactured by Bruker BioSpin MRI GmbH in Ettlingen, Germany. In our experiments, a point source with a diameter of 0.25 mm was enclosed in an acrylic cube with a volume of 1 cm³. Before initiating the measurements, the radioactivity of the point source was calibrated to a value of 0.379 MBq. HR images and LR images were reconstructed using two distinct algorithms. The 3D maximum likelihood expectation maximization (3D-MLEM) algorithm was employed for HR reconstruction, while the FBP algorithm was used for LR reconstruction. The matrix size of $360\times 360\times 600$ was kept consistent across both reconstructions.

Ablation Study

This section evaluates the effectiveness of each novel component in the proposed network and highlights the advantages of incorporating CT information as additional knowledge. All methods were optimized with PSNR-oriented, ensuring a fair comparison of their performance. Ablation experiments were conducted in cases 1 and 2 to rigorously assess the proposed method, where the LR image was reconstructed using seven subsets of one iteration. The quantitative results are presented in Table 1, while visual comparisons are illustrated in Fig. 5 (a). To further validate the method, statistical correlation analyses were performed between the SR PET images and the HR PET images using various approaches. Pearson correlation coefficient (PCC) [36] and regression curves were calculated to quantify the correlations, as shown in Fig. 5 (b). Additionally, Bland–Altman plots [37] were employed to analyze the differences in the regions of interest (ROIs) generated by different algorithms, illustrated in Fig. 5 (c).

Table 1.

Quantitative comparison results (mean of PSNR, SSIM, and RMSE) for case 1 and case 2 were calculated using all testing PET slices. The table evaluates the improvements in PET SR achieved by incorporating the CT modality and emphasizes the contributions of each proposed module. The best-performing results are presented in bold

Method	Case 1 PSNR↑/SSIM↑/RMSE↓	Case 2 PSNR↑/SSIM↑/RMSE↓
LR	31.61/0.917/0.027	35.70/0.954/0.017
Multimodal multi-head (method A)	36.50/0.960/0.015	33.99/0.950/0.020
Method A + EMHFA (method B)	36.65/0.956/0.015	34.56/0.955/0.019
Method A + GRB (method C)	37.26/0.947/0.014	35.94/0.941/0.016
Method A + EMHFA + GRB (proposed—MFHA-GAN)	37.31/0.955/0.014	36.06/0.949/0.016
Single-modal multi-head + EMHFA + GRB (method D)	37.23/0.950/0.014	35.16/0.941/0.018
Single-modal single-head + GRB (method E)	36.71/0.957/0.015	34.49/0.958/0.019

Fig. 5.

SR results and statistical analysis for ablation studies. (a) Rows 1 and 2 present qualitative visual comparisons of different PET SR methods on the SAPET dataset for case 1 and case 2. Case 1 and case 2 refer to SR images obtained from LR reconstruction using seven subsets with one iteration and seven subsets with two iterations, respectively. (b) A scatterplot analysis is conducted to assess the correlation between SR PET and HR PET images for case 1 and case 2 shown in (a). The reference line in each scatterplot is represented by y = ax + b, with the yellow line showing the fitted line from linear regression. The PCC and the equation of the fitted line are displayed in the upper-left corner of each scatterplot, providing a quantitative evaluation of image correlation. (c) Bland–Altman consistency analysis for the ROIs marked by red bounding boxes in case 1 and case 2 from (a)

In Table 1, method A uses PET and CT as multimodal inputs and applies multi-head feature extraction for PET image SR, serving as the baseline. As illustrated in Fig. 5 (a), while method A enhances resolution compared to LR images, it lacks detailed texture and edge information compared to HR images. Method B builds on method A by incorporating the EMHFA module, significantly improving edge artifact handling and texture recovery, particularly in case 2. This method effectively leverages the complementary strengths of multimodal features during extraction. Method C extends method A by incorporating the GRB, which facilitates deeper reconstruction and enhances detail recovery, as highlighted in the red-circled regions of Fig. 5. The proposed MFHA-GAN combines the innovations of methods B and C with method A, achieving superior resolution and restoring details overlooked by other methods, as indicated by the yellow arrows in Fig. 5. Method D represents a single-modal variation of MFHA-GAN, relying solely on PET inputs. While its SR performance is slightly lower than the multimodal MFHA-GAN, it surpasses methods A and B in detail recovery. Additionally, method D demonstrates scalability by accommodating both single-modal and multimodal inputs without altering the network architecture. Method E only uses a single-modal PET input and performs a simple single-head feature extraction (head 2) instead of multi-head feature extraction and feature fusion, resulting in weaker edge artifacts and less effective detail recovery. These ablation studies highlight the importance of multi-scale feature extraction and robust feature fusion, even for single-modal SR tasks.

Figure 5 (b) depicts the pixel-wise similarity between the SR PET and HR PET images. The red line indicates perfect agreement, where each pixel value in the SR image matches its corresponding value in the HR image. Our proposed method achieved the highest PCC of 0.994 in case 1. In case 2, method D, utilizing the single-modal MFHA-GAN, attained the highest PCC of 0.9927. Compared to the widely scattered pixel values in LR PET images, the progressively concentrated scatter plots and increasing PCC values from method A to the proposed method demonstrate the effectiveness of each network component. Figure 5 (c) presents Bland–Altman plots analyzing the ROIs highlighted in Fig. 5 (a), showing the 95% agreement bounds between the SR PET images and the GT HR images. In cases 1 and 2, our proposed method exhibited minimal biases (0.05 and − 0.00), whereas method E exhibited larger biases (0.06 and − 0.02). These findings indicate that relying solely on PET single modality data or basic feature extraction is inadequate for optimal PET image restoration.

Super-Resolution Result Comparison

Results on the SA Dataset

Table 2 presents the average PSNR, SSIM, and RMSE values for various SR methods applied to the SA dataset in both case 1 and case 2. The results demonstrate that the proposed method consistently outperforms traditional approaches across most metrics. Specifically, the proposed method significantly enhances the quality of LR PET images, surpassing all compared methods. This indicates that our model effectively integrates complementary information from multiple modalities, leading to substantial improvements in SR performance. Notably, compared to the original LR PET images, our MFHA-GAN model improves PSNR by 5.7 dB in case 1 and 0.36 dB in case 2, underscoring the effectiveness of the proposed generative network. To assess whether the improvements in PSNR, SSIM, and RMSE achieved by the different SR methods compared to the LR images are statistically significant, a t-test was performed with a significance level of p < 0.01. The results show that all metrics demonstrated statistical significance, except for the PSNR improvement of the RDN method in case 2.

Table 2.

Quantitative comparison with five state-of-the-art image reconstruction methods in terms of PSNR, SSIM, and RMSE on the SA datasets of case 1 and case 2. Red fonts indicate the best results, and blue fonts indicate the second-best results. * indicates p < 0.01

Method	Case 1 PSNR↑/SSIM↑/RMSE↓	Case 2 PSNR↑/SSIM↑/RMSE↓
LR	31.61/0.917/0.027	35.70/0.954/0.017
SRGAN	24.52*/0.347*/0.062*	24.83*/0.345*/0.060*
SRCNN	31.05*/0.905*/0.029*	31.73*/0.907*/0.027*
VDSR	34.24*/0.921*/0.020*	33.72*/0.936*/0.021*
RDN	32.45*/0.886*/0.024*	35.75/0.924*/0.070*
DRRN	32.64*/0.947*/0.024*	33.16*/0.948*/0.224*
Proposed	37.31*/0.955*/0.014*	36.06*/0.949*/0.016*

To provide a more comprehensive evaluation of the models’ performance, we use box plots (Fig. 6) to visualize the results from various SR methods. Overall, the proposed method outperforms state-of-the-art methods, consistently achieving the highest PSNR values in both cases. Its enhanced stability and efficacy are seen in case 1, where it displays noticeably greater upper and lower limits together with a higher median PSNR. The great generalization capacity and consistency of our model are demonstrated by the close performance between the two cases. However, method D ranks second, but its PSNR lower bound in case 2 is slightly lower than that of the proposed method, showing slightly worse robustness. In contrast, RDN exhibits a wide PSNR distribution gap between the two cases, indicating poor adaptability to varying scenarios. Other methods, such as SRGAN and SRCNN, display more outliers, suggesting inconsistencies in certain image reconstruction scenarios. VDSR and DRRN demonstrate higher SSIM values and closer medians compared to SRGAN and SRCNN, showing better performance in structural reconstruction. Notably, the proposed method achieves the best SSIM performance, with values close to 1 in both cases, demonstrating superior stability and reliability. Additionally, its RMSE is the lowest among all methods, with minimal outliers, further reinforcing its quantitative superiority.

Fig. 6.

Boxplot visualization of PSNR, SSIM, and RMSE results for LR and SR methods on the SA dataset

Beyond quantitative metrics, we provide visual comparisons of the SR images generated by different methods in Fig. 7. The ROI regions (highlighted in red) illustrate that our method produces SR PET images with finer details and sharper edges compared to others. Furthermore, the error maps are lighter in color, indicating reduced discrepancies with the HR images, particularly in case 1.

Fig. 7.

Rows 1 and 3 show qualitative comparisons between various state-of-the-art SR methods and our proposed method in case 1 and case 2, respectively. Row 2 and row 4 indicate the corresponding error maps. The color brightness denotes the degree of difference between each result and these HR images

Results on the Phantom Dataset

To evaluate the effectiveness of the proposed method in improving spatial resolution, we tested it on the phantom dataset under three experimental scenarios, and the actual photograph of the phantom was shown in Fig. 8. In case 1, SR reconstruction was performed on LR images generated using seven subsets and one iteration. For case 2 and case 3, the reconstructions were based on LR images obtained with seven subsets and two and three iterations, respectively. As shown in Fig. 8, two holes of different diameters (x₁ = 1.5 mm, x₂ = 1.2 mm) were selected for evaluation to comprehensively compare the performance of various SR methods in vertical directions.

Fig. 8.

The left picture is a photograph of the phantom. The right image is the HR image, where the green and red dashed line is the dashed line drawn at the brightest ROI selected

The proposed one produced the lowest FWHM value of 1.732 mm in case 3, which is 0.38 mm better than the HR, for the 1.5 mm hole in the vertical direction, as seen in Table 3. Method D achieved the second-best resolution in case 2 with a FWHM of 1.880 mm. For the 1.2 mm hole, method D and the proposed achieved the best and second-best resolution in case 3, with FWHM values of 1.245 mm and 1.268 mm, respectively. Notably, the FWHM values of both methods in case 2 and case 3 were better than those of the HR images. For the 1.5 mm hole, the FWHM value was reduced from 2.112 mm (HR image) to 1.732 mm, while for the 1.2 mm hole, it was reduced from 1.590 mm to 1.245 mm.

Table 3.

Quantitative comparison of vertical FWHM on representative slices from cases 1–3 (x₁ and x₂) of the phantom dataset across five state-of-the-art SR and reconstruction methods. Values in red indicate the best result, and values in blue indicate the second-best result

Method	Case 1 x1 = 1.5 mm/x2 = 1.2 mm FWHM↓(mm)	Case 2 x1 = 1.5 mm/x2 = 1.2 mm FWHM↓(mm)	Case 3 x1 = 1.5 mm/x2 = 1.2 mm FWHM↓(mm)
HR	2.112/1.590	2.112/1.590	2.112/1.590
LR	2.486/1.728	2.266/1.690	2.155/1.632
SRGAN	2.363/1.716	2.170/1.630	2.170/1.564
SRCNN	3.988/1.760	2.356/1.718	2.207/1.650
VDSR	2.444/1.765	2.122/1.654	2.018/1.528
RDN	2.428/1.723	2.166/1.603	2.017/1.500
DRRN	2.271/1.678	2.014/1.593	1.902/1.519
Method D	2.226/1.656	1.880/1.408	1.738/1.245
Proposed	2.309/1.655	1.899/1.434	1.732/1.268

Figure 9 demonstrates that the proposed MHFA-GAN method consistently produces images with clearer hole structures across all cases, achieving the most visually accurate results. Notably, in the generated phantom images, even the 1.0 mm hole near the center is distinguishable. Compared to other methods, the images generated by MHFA-GAN exhibit significantly fewer edge artifacts, highlighting its effectiveness in artifact reduction. Furthermore, the method demonstrates superior focus on the ROI, enabling the recovery of finer structural details and enhancing image quality.

Fig. 9.

Rows 1, 2, and 3 show qualitative comparisons between various state-of-the-art SR methods and our proposed method in case 1, case 2, and case 3, respectively

We also measured the reconstruction time for LR, HR, and SR images, as shown in Table 4. Reconstructing a set of HR phantom images took 253 s, while reconstructing LR images required 226 s, 230 s, and 239 s for case 1, case 2, and case 3, respectively. Both high- and low-iteration image reconstructions took over 200 s. In contrast, generating each set of SR phantom images using DL methods took just over 20 s, significantly improving efficiency and reducing reconstruction time. These results demonstrate the substantial time advantage of DL technology in SR tasks.

Table 4.

Reconstruction time on the full phantom dataset for HR images, LR images, and images generated by various SR methods

Method	HR	LR (case 1/case 2/case 3)	SRGAN	SRCNN	VDSR	RDN	DRRN	Method D	Proposed
Time(s)	253	226/230/239	25	22	23	38	24	27	29

Results on the Capillary Dataset

To assess the proposed method’s performance across various datasets, we conducted a spatial resolution test using the capillary dataset as shown in Fig. 10, with the results in the vertical and horizontal direction are presented in Tables 5 and 6. Our proposed method outperformed other methods, achieving the best outcomes. Notably, in case 2 and case 3, for capillaries with a diameter of 0.1 mm, the proposed method achieved a FWHM value of 0.625 mm, as shown in Table 5. This is significantly better than the FWHM value of HR images (0.871 mm) and approaches the physical limit of the PET/CT equipment (0.6 mm). These findings highlight the method’s effectiveness in enhancing the spatial resolution of PET images. Furthermore, the single-modality method (method D) demonstrated strong performance, achieving FWHM values second only to the multimodal approach and surpassing those of HR images in all cases. This indicates that even without multimodal inputs, the single-modality MFHA-GAN model can substantially improve PET image spatial resolution, outperforming real HR images and showcasing excellent SR images. As shown in Table 6, the advantages of single-modal MFHA-GAN are further demonstrated. The FWHM measured in the horizontal direction of method D has the smallest value in all cases, and the spatial resolution is significantly better than other comparison methods and is close to HR.

Fig. 10.

The left picture is a photograph of the capillary. The right image is the HR image, where the green line is the dotted line drawn at the selected ROI

Table 5.

Quantitative comparison of FWHM in the vertical direction is conducted on representative slices from the capillary dataset (case 1–3) across five state-of-the-art SR and image reconstruction methods

Method	Case 1 X = 0.1 mm FWHM↓(mm)	Case 2 X = 0.1 mm FWHM↓(mm)	Case 3 X = 0.1 mm FWHM↓(mm)
HR	0.871	0.871	0.871
LR	1.430	1.425	1.446
SRGAN	1.821	1.865	1.727
SRCNN	1.844	1.902	1.941
VDSR	1.313	1.315	1.329
RDN	1.119	1.134	1.155
DRRN	1.097	1.127	1.193
Method D	0.671	0.665	0.681
Proposed	0.633	0.625	0.625

Table 6.

Quantitative comparison of horizontal FWHM on representative slices from the capillary dataset (case 1–3) across five state-of-the-art SR and image reconstruction methods

Method	Case 1 X = 0.1 mm FWHM↓(mm)	Case 2 X = 0.1 mm FWHM↓(mm)	Case 3 X = 0.1 mm FWHM↓(mm)
HR	0.957	0.957	0.957
LR	2.085	2.052	2.056
SRGAN	2.215	2.242	2.216
SRCNN	2.425	2.548	2.623
VDSR	2.101	2.063	2.064
RDN	1.564	1.505	1.503
DRRN	1.863	1.859	1.890
Method D	1.114	1.077	1.115
Proposed	1.397	1.374	1.414

The decision to measure FWHM in only the vertical direction for the phantom dataset, while evaluating both vertical and horizontal directions in the capillary dataset, is based on the structural characteristics of the datasets. The phantom dataset, with evenly distributed holes, has relatively simple structures, making vertical resolution measurements sufficient for overall evaluation. In contrast, the capillary dataset contains finer, more intricate structures that are sensitive to resolution in both directions. Measuring FWHM in both directions enables a more comprehensive assessment of the spatial resolution, capturing potential variations in resolution due to the orientation of these detailed features.

Results on the ²²Na Point Source Dataset

We conducted SR experiments on ²²Na point source data acquired using a small animal 7T PET/MRI system and measured the FWHM values to assess the performance of SR methods in improving spatial resolution cross-device (Fig. 11). The results, summarized in Table 7, demonstrate that our single-modal MHFA-GAN method achieved superior spatial resolution in both horizontal and vertical directions compared to the LR FBP algorithm. Notably, in the vertical direction, MHFA-GAN outperformed all other methods, achieving the best spatial resolution with an FWHM of 1.077 mm, which closely approximates the HR 3D-MLEM algorithm’s FWHM of 1.050 mm.

Fig. 11.

a The photograph of the ²²Na point source used in the model testing process to calculate the spatial resolution values of outputs from various methods. b The photograph of the container designed to hold the ²²Na source

Table 7.

Quantitative comparison of vertical and horizontal FWHM on a representative slice from the ²²Na point source dataset across five state-of-the-art SR and image reconstruction methods

Method	Vertical direction FWHM↓(mm)	Horizontal direction FWHM↓(mm)
HR	0.992	1.050
LR	1.629	1.406
SRCNN	1.832	1.670
VDSR	1.629	1.406
RDN	1.182	1.150
DRRN	1.476	1.213
Method D	1.216	1.077

Discussion

This work aims to enhance PET spatial resolution and accelerate reconstruction to better support clinical decision-making. While many DL-based SR methods have been proposed, they often fail to recover high-frequency details and typically rely on single-modality inputs, limiting their ability to exploit complementary information. To address these limitations, we propose MHFA-GAN, a multimodal GAN framework that reconstructs HR PET images from LR PET and CT data. The model integrates multi-head convolution with variable kernels for edge and texture detail recovery, an EMHCA module for dynamic cross-modal alignment and sub-pixel misalignment correction, and a plug-and-play GRB for high-frequency restoration. A U-Net-based discriminator further provides multi-scale feedback, improving generator optimization. Collectively, these innovations enable MHFA-GAN to deliver superior PET image quality and resolution compared with conventional SR methods.

Extensive experiments demonstrated the superiority of MHFA-GAN across multiple datasets. Compared with conventional iterative methods, the proposed framework reduced reconstruction time nearly tenfold (Table 4), achieving inference in 20–30 s per case on an NVIDIA RTX3080Ti GPU without requiring specialized hardware. This improvement demonstrates that the method can accelerate image generation while maintaining moderate computational requirements. Since the framework does not depend on specialized hardware beyond a standard GPU, integration into routine preclinical workflows is technically feasible. With future validation on clinical PET/CT datasets, the approach may also be adapted for incorporation into standard clinical pipelines. In multimodal settings, MHFA-GAN improved spatial resolution close to the physical limit of PET/CT (0.625 mm FWHM in the capillary dataset). Furthermore, in cross-device single-modality experiments, it surpasses the performance of traditional FBP algorithms and approaches that of 3D-MLEM algorithms. These findings highlight the versatility and potential of MHFA-GAN in both multimodal and constrained single-modal scenarios.

Beyond improvements in FWHM and image similarity metrics, enhanced PET spatial resolution has clear clinical implications. By reducing blurring and partial volume effects, super-resolution methods can improve the visibility of small or low-contrast lesions, enabling more reliable delineation and potentially earlier detection. These benefits are particularly relevant in oncology, where accurate lesion localization and characterization directly guide staging and treatment planning. Although this study did not assess quantitative indices such as SUV, future work is needed to verify that resolution gains do not compromise quantitative accuracy in clinical datasets. Due to the scientifically unreliable FWHM values obtained with SRGAN (Table 7), its results were excluded from analysis.

In the “Ablation Study” section, ablation experiments confirmed that each component of the proposed framework contributes to SR PET image quality. As shown in Table 1, incorporating the GRB consistently increased PSNR but slightly reduced SSIM in both method A and method B. This trade-off likely stems from GRB’s emphasis on high-frequency detail, which sharpens edges but introduces subtle structural variations that penalize SSIM. Nevertheless, visualization results (Fig. 5 (a)) show that method C produces clearer, sharper edges than Method B, appearing closer to the HR reference. These findings suggest that GRB enhances edge fidelity despite minor structural alterations. Future work will focus on refining its attention mechanism to better balance high-frequency enhancement with structural integrity, thereby improving both PSNR and SSIM.

In this study, we benchmarked our method against representative CNN- and GAN-based SR approaches, including SRCNN, VDSR, RDN, DRRN, and SRGAN, which are widely applied in medical imaging for their ability to recover fine details with stable performance. Consistent with prior findings, GAN-based models enhanced perceptual quality, while CNN-based residual networks yielded more stable quantitative gains. Our model outperformed all baselines across multiple metrics, demonstrating its effectiveness in improving PET spatial resolution. Beyond these methods, emerging transformer- and diffusion-based SR approaches offer promising alternatives: transformers capture long-range dependencies to preserve global structures, whereas diffusion models iteratively refine images to reconstruct high-frequency details with fewer artifacts. Although not included in our current experiments, these paradigms represent valuable directions for future work to broaden the evaluation of PET SR techniques.

In clinical PET imaging, limited spatial resolution remains a major challenge, producing blurring and partial volume effects that obscure fine details and compromise the detection of small or low-contrast lesions. By enhancing resolution, SR methods improve lesion conspicuity and boundary delineation, enabling earlier and more reliable detection. These advantages are particularly significant in oncology, where precise lesion characterization directly informs staging and treatment planning. The proposed MHFA-GAN framework is designed to address these limitations and shows strong potential for clinical translation.

To systematically evaluate the proposed framework, we first conducted experiments on preclinical PET/CT datasets. These datasets exhibit more pronounced resolution constraints and provide controlled conditions for quantitative validation. Although carried out in a preclinical setting, the results partially simulate clinical scenarios and, combined with cross-device validation on PET/MR data, demonstrate the translational relevance and generalizability of MHFA-GAN. Future work will extend evaluation to large-scale clinical PET/CT datasets to assess robustness and practical feasibility in patient imaging.

While our proposed method demonstrates significant advances, there remain opportunities for improvement. Future work will focus on improving the SSIM indicator by refining the GRB module and exploring multi-scale loss functions to preserve structural integrity while enhancing high-frequency details. Moreover, we aim to improve the multimodal feature alignment mechanism and incorporate additional modalities such as MRI to further enhance performance. To evaluate clinical applicability, we plan to validate the model on larger, multi-center clinical datasets and expand to dynamic PET imaging. These efforts aim to deliver more robust, efficient, and clinically viable solutions for high-quality PET image reconstruction without hardware modifications.

Conclusions

In this study, we proposed the MHFA-GAN model, a novel multimodal GAN approach for reconstructing high-resolution PET images from low-resolution PET and CT images. By leveraging a multi-head convolution mechanism and an EMHCA module, MHFA-GAN effectively captures detailed features from both modalities and minimizes sub-pixel misalignment. Additionally, a plug-and-play GRB module enhances the recovery of high-frequency details. Extensive experiments across various datasets demonstrate that MHFA-GAN significantly improves spatial resolution, approaching the physical limits of PET/CT systems. Single-modality experiments further validate its effectiveness, with results approaching those of traditional methods like 3D-MLEM. MHFA-GAN provides a reliable solution for PET image super-resolution, offering substantial improvements without the need for additional hardware, and shows strong potential for clinical applications.

Author Contribution

Xin Tian: Conceptualization; methodology; software; writing, original draft; visualization; formal analysis. Xuetai Chen: Writing, original draft; visualization; formal analysis; resources. Kishore Krishnagiri Manoj Doss: Data curation, writing—review and editing. Shijie Chen: Data curation, visualization. Haitao Ge: Visualization, investigation. Yifeng Pan: Formal analysis. Jyh-Cheng Chen: Writing, review and editing; funding acquisition; supervision. Jie Zhao: Project administration, Supervision, Validation.

Funding

This work was supported by Xuzhou Medical University–Research Cooperation Project under grant no. KY17012004 and the Excellent Talents Project of Xuzhou Medical University under grant no. 53681942.

Declarations

Ethical Approval

No ethics approval was required.

Institutional Review Board Statement

The animal studies were approved by the Laboratory Animal Ethics Committee of Xuzhou Medical University (process number for animal experiments: 202405T024).

Competing interests

The authors declare no competing interests.