Attention-driven complementary information fusion network for sparse photoacoustic image reconstruction

Abstract

Photoacoustic tomography (PAT) is an emerging biomedical imaging modality that uniquely combines high spatial resolution with deep tissue penetration in a non-invasive manner, holding significant promise for diverse applications. However, image reconstruction quality in PAT severely degrades under limited-view data acquisition scenarios, such as those imposed by the physical constraints of intracavitary imaging. Conventional reconstruction methods (e.g., Delay-and-Sum, DAS) under these conditions typically yield images plagued by severe artifacts and loss of fine structural details. While deep learning (DL) approaches offer some improvement, existing post-processing methods still struggle to accurately recover intricate anatomical features from severely undersampled, limited-view data, often resulting in blurred details or persistent artifacts. To address these critical limitations, we propose DUAFF-Net, a novel dual-stream deep learning architecture. DUAFF-Net uniquely processes two complementary input representations in parallel: 1) conventional DAS reconstructions, and 2) pixel-wise interpolated raw data. The network employs a sophisticated two-stage feature fusion strategy to maximize information extraction and synergy. In the first stage, the Multi-scale Information Aggregation and Feature-refinement Module (MIAF-Module) enables early-stage cross-modal information complementarity and feature enhancement. Subsequently, the Global Context and Deep Fusion Module (GCDF-Module) focuses on holistic feature optimization and deep integration across the streams. These modules work synergistically to progressively refine the reconstruction. Extensive experiments on simulated PAT datasets of retinal vasculature and complex brain structures, as well as an in vivo mouse abdomen dataset, demonstrate that DUAFF-Net robustly generates high-quality images even under highly incomplete data conditions. Quantitative evaluation shows that DUAFF-Net achieves substantial improvements over the standard DAS algorithm, with gains of ∼18.38 dB in Peak Signal-to-Noise Ratio (PSNR) and ∼0.69 in Structural Similarity Index (SSIM). Furthermore, DUAFF-Net consistently outperforms other state-of-the-art DL-based reconstruction models across multiple metrics, demonstrating its superior capability in preserving fine details and suppressing artifacts, thereby establishing comprehensive performance advantages for limited-view PAT reconstruction.

1. Introduction

Photoacoustic tomography (PAT), an emerging hybrid imaging modality, uniquely combines the advantages of high optical contrast and high ultrasonic resolution. It enables detailed structural and functional imaging of biological tissues at penetration depths ranging from several millimeters to centimeters,demonstrating immense potential for clinical applications [1], [2], [3]. However, reconstructing the internal initial pressure distribution from photoacoustic signals detected at the tissue boundary is an inherently challenging and ill-posed inverse problem, the solution of which is highly sensitive to data incompleteness. Conventionally, this problem has been addressed using non-iterative algorithms such as Time Reversal (TR) [4], [5] and Delay-and-Sum (DAS) [6], [7].

When the sensor array is equipped with a sufficient number of transducer elements and can achieve full-angle, dense sampling of the target tissue, traditional algorithms can reconstruct high-quality images [8]. However, in practical biomedical imaging applications, physical and geometric constraints—including transducer cost, size limitations, and restricted acoustic windows for complex or deep-seated tissues—often make the construction of such ideal acquisition systems infeasible. These real-world limitations typically lead to incomplete photoacoustic signal acquisition, manifesting primarily as sparse spatial sampling and a limited data acquisition aperture (i.e., the limited-view problem) [9], [10], [11]. This inherent data deficiency is a critical bottleneck that compromises the fidelity of the subsequent image reconstruction.

To address the aforementioned shortcomings of traditional reconstruction methods, deep learning (DL) has emerged as a key paradigm for enhancing PAT image quality, owing to its powerful data-driven modeling capabilities [12], [13], [14]. Currently, DL-based PAT reconstruction strategies can be broadly categorized according to their primary role in the workflow: signal-domain pre-enhancement [15], [16], end-to-end direct reconstruction [17], [18], [19], and image-domain post-processing enhancement [20], [21], [22], [23].

In end-to-end direct reconstruction (direct mapping from sensor signals to an image), researchers have explored various network architectures to recover the initial pressure distribution from incomplete or sparse acquisitions. For instance, early work by Waibel et al. [24] proposed a modified U-Net architecture, validating the feasibility of learning reconstruction directly from acquired data and laying the groundwork for using neural networks to overcome PAT data limitations. Subsequent research, such as the work by Feng et al. [25], further improved reconstruction quality by introducing residual learning into the U-Net (ResUNet). Additionally, to learn a more robust signal-to-image mapping, Tong et al. [26] proposed the Feature Projection Network (FPNet), which innovatively integrated photoacoustic signals and their temporal derivatives as a dual-channel input. However, deep learning networks that directly reconstruct from time-domain signals often face challenges when processing the high-dimensional, asymmetric nature of PAT data. The use of fully connected layers or large-receptive-field convolutions can lead to a prohibitive number of model parameters, imposing significant computational and memory demands and creating challenges for training and generalization.

Image-domain post-processing enhancement (image-to-image) is another extensively studied strategy, which utilizes deep neural networks to optimize images initially reconstructed by traditional algorithms like DAS.The Fully Dense U-Net (FDUNet) proposed by Guan et al. [27] is a typical example, designed to remove sparse-sampling artifacts from these preliminary images. Although such methods can effectively improve the visual appearance of images, their ultimate performance is highly dependent on the quality of the input image. If the initial image already exhibits a severe loss of structural information or is plagued by strong artifacts, the network faces an inherent limitation in recovering fine features and supplementing missing details, making it difficult to achieve the desired high-quality imaging standards [28], [29].

To overcome the limitations of single-source information or single-stage processing—such as the artifact susceptibility and detail loss of post-processing and the lower robustness of direct-processing [30], [31]—researchers have begun to explore more advanced hybrid-input and feature-fusion strategies. Such dual-domain approaches, which leverage complementary information from both the image and signal (sinogram) domains, have proven highly effective in related reconstruction tasks like sparse-view and low-dose CT [32], [33], [34]. Similarly, in the field of PAT, the Y-Net architecture proposed by Lan et al. [30] demonstrated the potential of multi-source information fusion by concurrently processing raw sensor data and DAS-reconstructed images. However, Y-Net's use of large-sized convolution kernels to handle high-dimensional raw sensor data posed challenges to computational resources and training efficiency. To mitigate such issues, the AS-Net designed by Guo et al. [35] introduced a parameter-efficient folding transform to process time-series data, making progress in enhancing parameter efficiency. Yet, this operation, which reshapes the 1D time series and transfers some temporal information to the channel dimension, may face challenges in optimally preserving critical long-range temporal dependencies within the original signal. Although existing fusion methods have advanced reconstruction quality, effectively mining and fusing the complementary advantages of different data sources (e.g., structural information from the image domain and detailed information from the signal domain) remains a core challenge and a critical research direction. This is especially true under conditions of extreme data incompleteness, such as combined limited-view and sparse-sampling scenarios, where achieving a synergistic, high-fidelity recovery of both structure and detail while suppressing artifacts remains a significant challenge.

To address these challenges, this study proposes a novel deep learning framework named DUAFF-Net (Dual-stream U-Net with Attentive Feature Fusion Network). The core innovation of DUAFF-Net lies in its unique dual-input stream design and its staged, multi-level attentive feature fusion mechanism. First, it concurrently processes a traditional DAS image and a multi-channel interpolated dataset, which is pre-processed based on physical principles to preserve independent multi-sensor perspectives. Simultaneously, an MSRA-Module is designed for the DAS image stream to deeply mine and reinforce its macro-structural prior information. Second, a novel MIAF-Module is introduced to achieve effective interaction and preliminary fusion of heterogeneous features (structural information from DAS and detail information from the interpolated data) at the shallow stages of the network, promoting early-stage information complementarity. Finally, a unique GCDF-Module is constructed to efficiently integrate and enhance the deep features from both parallel paths at the deeper layers of the network, ensuring a comprehensive synergy between global structure and local detail to elevate the final reconstruction quality. To validate the efficacy of DUAFF-Net, we conducted comprehensive training and evaluation on both computer-simulated photoacoustic datasets (retinal vasculature and brain MRI) and an in vivo mouse abdomen dataset. These experiments specifically focused on the model's performance under limited-view and sparse-data conditions, as well as its robustness in real-world biological imaging scenarios.

2. Method

2.1. Imaging model and network input generation

Assuming a two-dimensional space, the relationship between the initial photoacoustic pressure distribution ${\mathrm{p}}_0$ and the measurement data $\mathrm{b}$ detected by the sensors can be expressed as:

$$A{p}_0=b$$ (1)

where $\mathrm{A}$ represents the forward system matrix determined by the sensor geometry and acoustic wave propagation laws. In limited-view scenarios, $\mathrm{A}$ is severely ill-conditioned, making direct inverse reconstruction extremely unstable.

To generate the network input, we employ the Delay-and-Sum (DAS) algorithm to transform the measurement data $\mathrm{b}$ into the image domain. The reconstruction value at pixel $,\mathrm{r}$ is calculated as follows:

$$I_{\mathit{DAS}}\left(r\right)=\sum _{i=1}^N{b}_i\left(t-{\tau }_{i,r}\right)$$ (2)

where ${\mathrm{b}}_{\mathrm{i}}$ is the signal received by the $\mathrm{i}$-th sensor, and ${\mathrm{\tau }}_{\mathrm{i},\mathrm{r}}$ is the time-of-flight for the sound wave to propagate from position $\mathrm{r}$ to the $\mathrm{i}$-th sensor. This ${\mathrm{I}}_{\mathrm{DAS}}$ image serves as the initial structural input for the subsequent deep learning model.

2.2. Pixel-wise interpolation

To construct an input stream that is information-complementary to the DAS image and that preserves the fine details and multi-view characteristics of the original signals, we employed a pixel-wise interpolation technique for pre-processing, based on the work of Guan et al. [29] The core of this technique is to take the one-dimensional time-domain signals captured by each of the N sensors (system configuration shown in Fig. 1(a), with raw data shown in Fig. 1(b)) and, based on the acoustic Time-of-Flight (TOF), map them individually onto a pre-defined H×W reconstruction grid.

Fig. 1.

Photoacoustic imaging process. (a) Schematic of the imaging system with 64 detectors (yellow dots). (b) Raw signal data from the 64 sensors. (c) Spatial mapping of interpolated data from a single sensor located at coordinates (-10, 0) mm. The plot displays the data in a two-dimensional Cartesian coordinate system, where the time-series signal amplitudes are mapped to corresponding spatial locations based on acoustic propagation distance, resulting in concentric ring textures centered at the sensor position.

Ultimately, these N generated, independent two-dimensional spatial maps (an example is shown in Fig. 1(c)) are stacked along the channel dimension to form an N × H×W multi-channel data matrix. This matrix effectively preserves the independent perspective of each sensor, providing a detail-rich input for the subsequent Pixel-level Feature Extraction Network (PFE-UNet). To simplify computation, the TOF estimation in this study adopts a uniform speed of sound assumption. Although this assumption may introduce certain geometric deviations, the powerful end-to-end learning capability of DUAFF-Net can effectively compensate for and correct such deviations.

2.3. Dual-stream U-net with attentive feature fusion network (DUAFF-Net)

2.3.1. Overall network architecture

To achieve high-quality image reconstruction from incomplete photoacoustic signals, this study proposes a novel dual-stream deep learning framework, termed DUAFF-Net.The overall architecture of DUAFF-Net is illustrated in Fig. 2. Its core design philosophy lies in an information-complementary strategy using dual input streams and a multi-stage, multi-level feature fusion mechanism.

Fig. 2.

An overview of the DUAFF-Net photoacoustic reconstruction framework proposed in this paper. The framework employs a dual-input streaming strategy: one branch inputs a DAS image into the SFE-UNet; the other inputs spatially mapped raw data into the PFE-UNet. The two modified UNet-like subnetworks interact and integrate information through a two-stage feature fusion module (FFM, which includes the MIAF-Module and GCDF-Module), ultimately outputting a high-quality reconstructed photoacoustic image.(Note: Tensor dimensions in the diagram are denoted as Channels $\times$ Height $\times$ Width.).

Specifically, DUAFF-Net processes two information sources in parallel: The first stream is the image initially reconstructed by the conventional Delay-and-Sum (DAS) algorithm, which aims to provide the network with macro-structural and contour priors of the target. The second stream consists of the multi-channel spatial map data formed by pre-processing the raw sensor data with pixel-wise interpolation, intended to preserve the rich micro-structural details and multi-view features of the original signal. These two input streams are processed by two dedicated U-Net-based subnetworks, SFE-UNet (see Section 2.3.2) and PFE-UNet (see Section 2.3.3), for deep feature extraction.

To efficiently integrate these two complementary information streams, DUAFF-Net employs a feature fusion mechanism that runs throughout the network. This primarily includes the MIAF-Module (Multi-level Inter-stream Attention Fusion Module, see Section 2.3.4(1)) for early information interaction between corresponding levels of the subnetworks, and the GCDF-Module (Global Context and Dense Feature Fusion Module, see Section 2.3.4(2)) for deep integration of high-level features at the end of the network. This design enables DUAFF-Net to synergistically leverage the advantages of different information sources, ultimately generating high-quality reconstructed photoacoustic images.

2.3.2. Structural feature extraction UNet(SFE-UNet)

SFE-UNet is the critical branch in DUAFF-Net responsible for processing the DAS image. Its core objective is to extract robust and information-rich macro-structural features from the preliminary reconstructed images, which often contain significant artifacts and blurred structures. Although the DAS image may lack fine details, it preserves the general contour and position of the target, serving as a valuable structural prior. To effectively utilize this prior while suppressing its inherent defects, SFE-UNet incorporates the following key improvements upon the classic U-Net architecture:

First, to enhance the network's multi-scale feature representation capabilities,we have comprehensively replaced the standard convolutional layers in both the encoder and decoder with a novel Multi-scale Residual Attention Module (MSRA-Module). As the artifacts and target structures in DAS images vary in size, the MSRA-Module, by processing branches with different receptive fields in parallel, can simultaneously capture this multi-scale contextual information. This is crucial for identifying and separating artifacts from true structures.

Second, to optimize feature fusion, we integrated the lightweight ECA-Attention [36] into the skip connections, enabling adaptive channel refinement and suppression of redundant features with negligible computational cost.

Furthermore, the decoder of SFE-UNet also performs early cross-stream fusion with features from PFE-UNet at multiple levels via the MIAF-Module (detailed in Section 2.3.4). This serves to continuously guide and correct the feature learning process.

The MSRA-Module is the core building block for efficient feature extraction in SFE-UNet, and its detailed structure is shown in Fig. 3(a).

Fig. 3.

Presents the three core modules of our model: (a) MSRA-Module, (b) MIAF-Module, and (c) GCDF-Module.

Unlike generic multi-scale modules, the MSRA-Module is a physics-driven adaptation designed specifically for scale-aware artifact decoupling. As observed in preliminary DAS reconstructions, streak artifacts and vascular structures are spatially intertwined but possess distinct scale characteristics. Standard convolutional layers, with fixed receptive fields, struggle to effectively separate these aliased signals. To address this, we integrated parallel depth-wise separable convolutions (3$\times$3 and 5$\times$5) with the CBAM attention mechanism [37]. In this design, the depth-wise convolutions capture features at different receptive fields with minimal parameter overhead, achieving scale separation of signal components. Simultaneously, CBAM acts as a signal selector, dynamically re-weighting features to suppress artifact-dominated channels while enhancing those containing valid structural priors. Finally, residual learning is introduced to facilitate gradient flow and maintain structural integrity. This module effectively serves as a robust feature filter that disentangles artifacts from true structures, as validated in subsequent experiments.

2.3.3. Pixel-level feature extraction UNet(PFE-UNet)

PFE-UNet is the parallel branch in DUAFF-Net responsible for processing the multi-channel, pixel-wise interpolated data. Its core task is to extract the fine-grained features crucial for high-quality photoacoustic image reconstruction from this detail-rich data, which preserves the independent perspectives of each sensor, thereby effectively supplementing the macro-structural information provided by the SFE-UNet branch.

The architecture of PFE-UNet is also based on U-Net, but it incorporates the following key optimizations to efficiently process its unique input data:

In the encoder stage, we replace the standard convolutions with a combination of Selective Kernel Attention (SK-Attention) [38] and depthwise separable convolution. Each channel of the pixel-wise interpolated data corresponds to the perspective of a single sensor, and the feature scales and patterns they contain can vary significantly. SK-Attention dynamically and adaptively adjusts the receptive field size of the convolution kernel based on the input features, which enables PFE-UNet to more flexibly capture and fuse diverse information from different perspectives and scales.

To maximize the extraction of contextual information at the deepest layer of the network, the bottleneck layer integrates dilated convolution and a self-attention [39] mechanism. Dilated convolution effectively expands the receptive field without increasing parameters or computational cost to capture multi-scale contextual features. Subsequently, the self-attention mechanism models global relationships among these features, further enhancing their expressive power and the ability to capture long-range dependencies.

Similar to SFE-UNet, PFE-UNet also integrates ECA-Attention modules at its skip connections. This is intended to optimize the flow of low-level detail information from the encoder to the decoder by selectively enhancing information-rich feature channels.

The deep features extracted by PFE-UNet are fed, along with the features from SFE-UNet, into the subsequent feature fusion modules (MIAF-Module and GCDF-Module) to achieve synergistic integration of the two information streams.

2.3.4. Feature fusion

The success of DUAFF-Net lies in its innovative multi-stage collaborative feature fusion strategy, which is designed to intelligently integrate the heterogeneous features from the two parallel branches, SFE-UNet and PFE-UNet. Unlike conventional strategies that perform a single fusion only at the end of the network, DUAFF-Net adopts a two-stage fusion mechanism that combines "early guidance with late-stage integration." This mechanism is collaboratively implemented by two core modules: the Multi-level Inter-stream Attention Fusion Module (MIAF-Module) and the Global Context and Dense Feature Fusion Module (GCDF-Module), whose structures are shown in Fig. 3(b) and 3(c), respectively.

• 1)Multi-level Inter-stream Attention Fusion (MIAF-Module)

The core design philosophy of the MIAF-Module is to use the rich, detailed features extracted by PFE-UNet from the pixel-wise interpolated data to continuously guide and supplement the feature learning process of SFE-UNet as it handles the DAS image. This enables the generation of more comprehensive intermediate feature representations with fewer artifacts in the early and middle stages of the network, thereby avoiding the potential issues of feature loss or cumulative deviation that can occur in a single network path.

To achieve this goal, the MIAF-Module performs collaborative feature fusion at the bottleneck layer and at multiple corresponding levels of the decoders of SFE-UNet and PFE-UNet. At the bottleneck layer, the global semantic contextual features from PFE-UNet are fused with the deep features of SFE-UNet to correct structural deviations in the DAS reconstruction at a macro level. Subsequently, in the decoder path, the fusion strategy is adapted for different resolution levels: at lower-resolution fusion points, contextual information from PFE-UNet is used to optimize the fundamental structure in SFE-UNet; at higher-resolution fusion points, high-frequency spatial details, edges, and texture features from PFE-UNet are injected into and reinforce the SFE-UNet feature stream to accurately recover minute structures that are blurred or lost in the DAS image.

Distinct from standard symmetric fusion mechanisms [40], the core innovation of the MIAF-Module lies in its physics-prior-guided asymmetric retrieval strategy. As illustrated in Fig. 3(b), we designed an asymmetric assignment where the SFE-UNet features (${\text{F}}_{\text{DAS}}$) are designated as the Query, while the PFE-UNet features (${\text{F}}_{\text{pixel}}$) serve as the Key and Value. This assignment is physically motivated: the DAS features provide a reliable structural skeleton, whereas the interpolated features contain rich but cluttered textural details. By projecting ${\text{F}}_{\text{DAS}}$ as the Query, the network uses the reliable structural skeleton as an index to 'retrieve' missing high-frequency details from the interpolated stream.Mathematically, for the $\mathrm{i}$-th attention head, this asymmetric projection is defined as:

$${\mathit{head}}_i=\mathit{Attention}\left(F_{\mathit{DAS}}{W}_{\mathit{qi}}\mathrm{，}{F}_{\mathit{pixel}}{W}_{\mathit{ki}}\mathrm{，}{F}_{\mathit{pixel}}{W}_{\mathit{vi}}\right)$$ (3)

where ${\mathrm{W}}_{\mathrm{qi}}$, ${\mathrm{W}}_{\mathrm{ki}}$, and ${\mathrm{W}}_{\mathrm{vi}}$ denote the learnable projection matrices for the Query, Key, and Value of the $\mathrm{i}$-th head, respectively. These matrices map the input features into subspaces of dimension $d_k$ (for Query/Key) and $d_v$ (for Value).

The attention weights are computed to filter the interpolated details based on the DAS structure:

$$\mathit{Attention}\left(Q_i\mathrm{，}{K}_i\mathrm{，}{V}_i\right)=\mathit{softmax}\left(Q_i{K_i}^{{\rm T}}\right)/\sqrt{d_k}{V}_i$$ (4)

Here, the scaling factor $\frac{1}{\sqrt{d_k}}$ is applied to prevent gradient vanishing in the softmax function.

The final output, representing the structure-guided details, is projected and fused back into the DAS stream via a residual connection:

$$\mathit{Att}{n}_{\mathit{output}}=\mathit{Concat}\left({\mathit{head}}_1,\cdots ,{\mathit{head}}_h\right)W_o$$ (5)

$$F_{\mathit{fused}}=F_{\mathit{DAS}}+\mathit{Attn}\mathrm{\_}\mathit{output}$$ (6)

where $\mathit{Concat}\left(\bullet \right)$ denotes the concatenation operation across the channel dimension, and ${\mathrm{W}}_{\mathrm{o}}$ is the output linear projection matrix.

This residual addition ensures that the retrieved details specifically reinforce the structural backbone, achieving precise guidance and avoiding artifact hallucination.

• 2)Global Context and Dense Feature Fusion Module(GCDF-Module)

Located at the terminal end of the DUAFF-Net architecture, the GCDF-Module employs a tailored terminal dense fusion strategy to integrate the final high-level features from the SFE-UNet and PFE-UNet outputs. This design is intended to replace simple shallow concatenation, which often leads to information fragmentation, and is specifically used to bridge the semantic gap between heterogeneous features. Its core task is to generate a unified feature representation that possesses both a global field of view and preserves complementary information from both streams, ensuring high-quality image reconstruction. Specifically, this strategy comprises three cascaded stages: First, we utilize a GCBlock to model global context on the concatenated features [41]. By recalibrating channel weights, it enhances responsiveness to long-range structural dependencies, effectively mitigating macro-vascular discontinuities caused by limited local receptive fields. Second, cascaded DenseBlocks focus on facilitating deep interaction between the dual-stream features [42]. This design ensures that the structural skeleton from DAS and the textural details from interpolated data achieve synergistic fusion, significantly outperforming simple linear concatenation in artifact suppression. Finally, we employ a convolutional bottleneck layer as a feature distiller to compress and map the high-dimensional dense features back into the residual space. This step effectively aligns the semantic features of the two data streams, ensuring the final output features are highly compact and discriminative.

Through its multi-level feature integration strategy, the GCDF-Module synergistically enhances the fidelity of micro-structural details in the image and effectively suppresses background artifacts caused by signal incompleteness, thereby significantly improving the overall final imaging quality.

2.4. Loss function

To achieve higher-quality photoacoustic image reconstruction, we employ a composite loss function to optimize the performance of the dual-stream parallel feature fusion network. This composite loss function is composed of the Smooth L1 Loss and the Mean Squared Error (MSE) Loss.

The Smooth L1 loss, through its robustness to large deviations and its precision in penalizing subtle differences, effectively guides SFE-UNet to learn how to recover a high-fidelity image that is closer to the ground truth from a preliminary reconstructed image containing artifacts.

$$\begin{array}{l}{L}_{SFE}=smoothL1\left(y-y_{sfe}\right)\\ =\left\{\begin{array}{l}0.5x^2\\ \left|x\right|-0.5\end{array}\genfrac{}{}{0ex}{}{if\left|x\right|<1}{otherwise}\right)\end{array}$$ (7)

where $\mathrm{y}$ represents the ground truth image and ${\mathrm{y}}_{\mathrm{sfe}}$ represents the output reconstructed by the SFE-UNet network.

The Mean Squared Error (MSE) loss, by comparing the squared differences pixel by pixel, guides PFE-UNet to learn how to recover a photoacoustic image that is faithful to the ground truth in terms of pixel values, details, and overall structure from interpolated data that may contain incomplete or distorted information.

$$L_{\mathit{FPE}}=\frac{1}{N}\sum {‖y-y_{\mathit{fpe}}‖}_2^2$$ (8)

where $\mathrm{y}$ represents the ground truth image and ${\mathrm{y}}_{\mathrm{fpe}}$ represents the output reconstructed by the PFE-UNet network.

Finally, the composite loss function is defined as:

$$\begin{array}{l}{L}_{\mathit{total}}={\gamma L}_{\mathit{output}}+{\alpha L}_{\mathit{SFE}}+\beta L_{\mathit{FPE}}\end{array}$$ (9)

In the total loss function, we assign different weights to each loss term: the weight for $L_{\mathit{output}}\left(\gamma \right)$ is set to 1, while the weight for $L_{\mathit{SFE}}\left(\alpha \right)$ is 0.1, and the weight for $L_{\mathit{FPE}}\left(\beta \right)$ is 0.05. These weights were determined empirically to balance the contributions of each term, enabling the model to generate reconstructed images with the highest visual plausibility and quality. $L_{\mathit{output}}$, the loss on the final network output, is considered the primary optimization target; therefore, its weight is set to 1 to make it the main driver of the optimization. $L_{\mathit{SFE}}$ and $L_{\mathit{FPE}}$ are auxiliary losses for the specific branches. Setting their weights to the relatively small values of 0.1 and 0.05 ensures that they effectively guide the learning of their corresponding branches without overly dominating or interfering with the learning process for the primary objective.

3. Experiments

3.1. Dataset

3.1.1. Numerical simulation datasets

To evaluate the performance and generalization capability of the proposed DUAFF-Net, we conducted comprehensive experiments using publicly available datasets, including a retinal fundus vasculature dataset [43] and a brain MRI dataset [44]. The images from these datasets were used to define the initial pressure distributions in our photoacoustic tomography (PAT) simulations. Subsequently, the required photoacoustic signal data for the experiments were generated through simulations constructed using the k-Wave toolbox [45].

Specifically, the key parameters for the k-Wave simulation environment were configured as follows: The simulation employed a two-dimensional (2D) environment, with a total grid size of 128 × 128 pixels, corresponding to a computational domain of 20.8 × 20.8 mm. The number of ultrasonic transducers in the region of interest was determined by the experimental requirements. To simulate the restricted acoustic windows commonly encountered in clinical scenarios, we employed a 64-element transducer array arranged in a 180-degree semi-circle, as shown in Fig. 1(a). This specific geometry was selected to introduce realistic limited-view and sparse-sampling reconstruction challenges. The transducers were placed at a radius of 10 mm from the center of the grid. The surrounding medium was water with a density of 1000 kg/m³ and a speed of sound set to 1500 m/s. Additionally, in this study, we simulated an ideal noise-free acoustic environment. This choice was made to explicitly isolate the geometric artifacts induced by the limited-view configuration from noise-related signal degradation, allowing us to focus specifically on structural restoration.

As the number of samples in the public retinal vasculature dataset was insufficient to support our experiments, we performed data augmentation to expand the training set to 1600 samples and the test set to 400 samples.

3.1.2. In vivo mouse abdomen dataset

To further validate the clinical applicability and robustness of DUAFF-Net in real-world scenarios, we utilized the publicly available in vivo mouse abdomen dataset (MSOT-Abdomen) open-sourced by Tong et al. [26] This dataset comprises 699 cross-sectional photoacoustic scans of the mouse abdomen acquired using an MSOT inVision 128 system. To maintain consistency with the sparse sampling configuration used in our simulation experiments, the raw data was uniformly downsampled from 128 channels to 64 channels (i.e., utilizing every other transducer element). Beyond the requisite alignment of physical specifications, the preprocessing of the in vivo dataset remained entirely consistent with the methodology established for the numerical simulations.

Following the standard data partition provided in the reference, we allocated 575 samples for training and 124 samples for testing. Given the high acquisition cost and relatively limited sample size of real biological data, we adopted a transfer learning strategy to effectively facilitate the transfer and adaptation of model features from the simulation domain to the real in vivo imaging domain.

3.2. Parameter settings

Numerical simulations and in vivo experiments were conducted in this study to validate the performance of the proposed reconstruction method. The proposed method was implemented using the PyTorch framework and run on a workstation equipped with four NVIDIA TITAN Xp GPUs. To ensure a fair comparison, all models used the exact same training and test sets, and were trained and inferred on the same hardware platform.

For the numerical simulations, the network was trained for 500 epochs using the Adam optimizer. The batch size was set to 4 to accommodate the memory constraints of the GPU, and the learning rate was initialized at 0.0005 to ensure the stable and efficient convergence of the training loss.All baseline methods strictly followed the hyperparameter settings from their respective original papers.

For the in vivo validation, we applied a consistent transfer learning strategy to both DUAFF-Net and all baseline models to bridge the domain gap and ensure rigorous fairness. Specifically, all models were initialized with their respective weights pre-trained on the simulated Brain MRI dataset and subsequently fine-tuned on the sparsely sampled in vivo data. This fine-tuning process was universally conducted for 200 epochs with a learning rate of 0.0001, ensuring stable convergence and optimal cross-domain adaptation while maintaining high physical fidelity in the reconstructed images.

3.3. Performance evaluation

To objectively evaluate the performance of the image reconstruction, this study employed two widely used image quality assessment metrics: the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM). PSNR measures the degree of image distortion by calculating pixel-wise errors, while SSIM focuses more on assessing the similarity in brightness, contrast, and structural information between the images.

$$\begin{array}{l}\mathit{SSIM}=\frac{\left(2{\mu }_y{\mu }_{\hat{y}}+c_1\right)\left({2\sigma }_{y\hat{y}}+c_2\right)}{\left({\mu }_y^2+{\mu }_{\hat{y}}^2+c_1\right)\left({\sigma }_y^2+{\sigma }_{\hat{y}}^2+c_2\right)}\end{array}$$ (10)

where $\mathrm{y}$ and $\hat{\mathrm{y}}$ represent the ground truth and reconstructed images, respectively.${\mathrm{\mu }}_{\mathrm{y}}$ and ${\mathrm{\mu }}_{\hat{\mathrm{y}}}$ are the mean values of the ground truth and reconstructed images,${\mathrm{\sigma }}_{\mathrm{y}}$and ${\mathrm{\sigma }}_{\hat{\mathrm{y}}}$ are their variances, and ${\mathrm{\sigma }}_{\mathrm{y}\hat{\mathrm{y}}}$ is their covariance.${\mathrm{c}}_1$ and ${\mathrm{c}}_2$ are two constants to stabilize the division with a weak denominator.

$$\begin{array}{l}\mathit{PSNR}=10{\log }_{10}\frac{{\max }^2\left(y,\hat{y}\right)}{\frac{1}{N}\sum {∥y-\hat{y}∥}_2^2}\end{array}$$ (11)

where $\mathrm{y}$ and $\hat{\mathrm{y}}$ represent the ground truth and reconstructed images, respectively, MAX is the maximum possible pixel value of the image, where N is the total number of pixels in the image.

3.4. Results comparison

To validate the effectiveness of the proposed method, we compared it against the conventional Delay-and-Sum (DAS) algorithm as well as a series of deep learning models. Specifically, the compared deep learning methods include:U-Net, which takes the initially reconstructed DAS image as input.Y-Net, a hybrid architecture that simultaneously utilizes the DAS reconstructed image and the raw time-series data as dual-channel inputs and generates the final reconstruction through a shared decoder.AS-Net, a network that employs a multi-feature fusion strategy, fusing features extracted from the DAS image with folded time-series data, and then fusing them again at the final stage to output the reconstructed image. In addition, we also included SwinIR [46] for comparison, a recent Transformer-based model that has demonstrated powerful restoration capabilities in the field of photoacoustic reconstruction.

3.4.1. Simulated vasculature

In this section, we qualitatively and quantitatively evaluate the reconstruction performance of DUAFF-Net and various competing methods on the retinal vasculature dataset. As shown in Fig. 4, DUAFF-Net demonstrates visual results that most closely resemble the ground truth (GT) image in terms of detail reconstruction, accuracy, and artifact suppression for the vascular network. The results from the conventional DAS method are blurry, with an almost complete loss of vascular structural information. As an improvement, although U-Net and Y-Net can recover the main vascular contours,

Fig. 4.

Comparison of reconstruction results from different methods on two representative retinal vascular image samples. The first row shows the results for sample one, and the second row for sample two. The columns from left to right are: Ground Truth (GT), DAS, U-Net, SwinIR, Y-Net, AS-Net, and the proposed DUAFF-Net.

they exhibit clear and consistent deficiencies in fine structures. This issue is particularly prominent in the regions marked by red boxes in Fig. 4. In sample one, the vessel edges reconstructed by U-Net are blurred and the terminals are discontinuous, while in the Y-Net result, fine vessels are nearly invisible due to severely insufficient contrast. The same defects reappear in sample two, where U-Net fails to reconstruct the complete microvascular network, leading to structural disconnections and information loss, while the Y-Net result is even dimmer, with an almost complete loss of detail in the region. In contrast, DUAFF-Net clearly and completely reconstructs the vascular branches and sharp edges in both of these critical regions, showcasing its significant advantage in detail fidelity.

Compared to advanced models like SwinIR and AS-Net, the advantage of DUAFF-Net lies in its higher fidelity and accuracy. In the blue-boxed region of sample one, SwinIR introduces faint line artifacts not present in the GT. In the yellow-boxed region of sample two, AS-Net not only fails to fully recover the microvasculature but also erroneously generates non-existent vascular connections. DUAFF-Net, however, achieves accurate reconstruction in these key areas, preserving fine structures while avoiding such erroneous generations.

The significant visual advantages are strongly supported by objective evaluation metrics. Looking at specific samples, DUAFF-Net achieves the highest SSIM and PSNR values on both sample one (SSIM: 0.877, PSNR: 21.109 dB) and sample two (SSIM: 0.912, PSNR: 21.216 dB). To evaluate the overall performance and robustness of the models, Table 1 summarizes the average metrics for all methods on the entire vasculature test set. The data clearly show that DUAFF-Net's average SSIM (0.9014 ± 0.0379) and average PSNR (20.6815 ± 2.5049 dB) are significantly superior to all competing methods. This series of qualitative and quantitative results collectively demonstrates the superior performance of DUAFF-Net in the task of limited-view photoacoustic image reconstruction of vasculature.

Table 1.

Quantitative comparison of different methods on the vessel test set (mean ± std).

Algorithm	SSIM	PSNR
DAS	0.100 ± 0.033	12.030 ± 1.957
U-Net	0.782 ± 0.060	18.154 ± 2.432
Y-Net	0.784 ± 0.056	17.371 ± 1.874
SwinIR	0.857 ± 0.048	20.397 ± 2.455
AS-Net	0.888 ± 0.037	19.705 ± 2.423
DUAFF-Net	0.901 ± 0.037	20.681 ± 2.504

3.4.2. Simulated brain structure

In the more challenging task of brain structure reconstruction, DUAFF-Net again demonstrates comprehensive performance advantages, with the evaluation results shown in Fig. 5 and Table 2. The visual quality assessment (Fig. 5) intuitively reveals the performance hierarchy of the different methods. The conventional DAS method is almost unusable due to severe radial artifacts. Although U-Net and Y-Net can suppress some artifacts, they do so at the cost of severe image blurring and the loss of key anatomical details. SwinIR and AS-Net perform better, reconstructing relatively clear brain contours, but there is still a gap compared to the ground truth (GT) image.

Fig. 5.

Comparison of reconstruction results from different methods on brain structures. From left to right: Ground Truth (GT), DAS, U-Net, SwinIR, Y-Net, AS-Net, and the proposed DUAFF-Net.

Table 2.

Quantitative comparison of different methods on the brain test set (mean ± std).

Algorithm	SSIM	PSNR
DAS	0.174 ± 0.030	8.650 ± 1.957
U-Net	0.755 ± 0.052	24.004 ± 1.697
Swin-IR	0.788 ± 0.041	25.662 ± 2.020
Y-Net	0.792 ± 0.051	24.581 ± 1.415
AS-Net	0.834 ± 0.047	26.988 ± 2.348
DUAFF-Net	0.861 ± 0.044	27.025 ± 2.078

Among all methods, the reconstruction result from DUAFF-Net is visually closest to the GT. It not only exhibits the highest image clarity but also accurately restores natural fine structures. Particularly in terms of clear gray-white matter boundaries and effective artifact suppression, its visual results are significantly superior to all competing methods.

This visual superiority is strongly supported by the quantitative evaluation results (Table 2). DUAFF-Net achieves the highest average SSIM (0.861 ± 0.044) and PSNR (27.025 ± 2.078 dB). Notably, its SSIM value is significantly higher than that of the suboptimal AS-Net (0.834), which strongly demonstrates DUAFF-Net's superior ability to accurately preserve the complex anatomical structures of the brain—a finding that is perfectly consistent with the clear boundaries and structural details we observed in Fig. 5. In terms of PSNR, both DUAFF-Net and AS-Net (26.988 dB) delivered top-tier performance, far surpassing the other deep learning models and indicating their excellent performance in high signal fidelity and noise suppression.

As shown in Table 3, although our DUAFF-Net (29.81 M) has more parameters than SwinIR and AS-Net, it demonstrates a significant advantage in training efficiency. Our training time (4.45 h) is markedly faster than that of SwinIR (10.97 h) and AS-Net (5.99 h), which suggests our model converges more quickly.

Table 3.

Computational cost comparison of different methods.

Method	U-Net	Y-Net	SwinIR	AS-Net	DUAFF-Net
ParaMeters (M)	1.96	11.24	16.41	5.89	29.81
Training Time (h)	0.12	3.48	10.97	5.99	4.45
Inference Time (ms)	6.25	11.98	86.15	28.15	62.13

Furthermore, while the inference time of DUAFF-Net is not as fast as AS-Net's, our model is superior to AS-Net in terms of both performance metrics (as shown in Table 1 and Table 2) and training time. This demonstrates that our proposed architecture strikes an excellent balance between SOTA performance and training efficiency.

3.4.3. In vivo mouse abdomen

To validate the generalization capability of DUAFF-Net in real-world biological scenarios, we conducted a cross-domain evaluation on the in vivo mouse abdomen dataset, employing the transfer learning strategy detailed in Section 3.2. As presented in Fig. 6 and Table 4, while the conventional DAS reconstruction is degraded by sparse-sampling artifacts and baseline deep learning models generally suffer from over-smoothing, DUAFF-Net demonstrates superior performance. Visually, it effectively eliminates streak artifacts and recovers the clearest anatomical boundaries; quantitatively, it achieves the highest metrics (SSIM: 0.9077, PSNR: 29.67 dB). These results confirm that, leveraging the transfer learning strategy, DUAFF-Net successfully learns robust physical features transferable to the complex acoustic environments of real biological tissues.

Fig. 6.

Comparison of reconstruction results from different methods on the in vivo mouse abdomen dataset. From left to right: Ground Truth (GT), DAS, U-Net, SwinIR, Y-Net, AS-Net, and the proposed DUAFF-Net.

Table 4.

Quantitative comparison of different methods on the in vivo mouse abdomen dataset.

Algorithm	DAS	U-Net	SwinIR	Y-Net	AS-Net	DUAFF-Net
SSIM	0.1851	0.8505	0.8639	0.8704	0.8882	0.9077
PSNR	8.00	27.39	28.08	28.81	24.99	29.67

3.5. Ablation study

To evaluate the contributions of the key components in our network architecture, we conducted systematic ablation studies, focusing on the two core parts: feature extraction and feature fusion.

To evaluate the effectiveness of the Structural Feature Extraction Network (SFE-UNet) and the Pixel-level Feature Extraction Network (PFE-UNet), we established two baseline U-Nets for comparison. The first baseline (U-Net I) uses a standard U-Net architecture to directly process the raw Delay-and-Sum (DAS) images. The second baseline (U-Net II) also uses a standard U-Net architecture, but its input is the data pre-processed with pixel-wise interpolation.This baseline was specifically designed to evaluate how a standard network performs when processing only the stack of pixel-wise interpolated data. Through these comparisons, we aim to clearly demonstrate the advantages of SFE-UNet and PFE-UNet in their specific feature representation capabilities.

To dissect our proposed two-stage fusion strategy, we designed systematic comparative experiments. We established a baseline configuration called End Simple Fusion (ESF), which only performs a terminal fusion of the dual-stream features via simple concatenation and a 1 × 1 convolution, to measure the baseline performance of the dual-stream architecture. Second, we individually evaluated the independent contributions of the two core modules: one setup replaced ESF with only our designed GCDF-Module for advanced terminal fusion; another setup introduced the MIAF-Module on top of the ESF baseline to achieve early feature interaction. Finally, we combined both to form the complete DUAFF-Net, to validate the final effect of the MIAF and GCDF modules working in synergy.

The experimental results in Table 5 clearly demonstrate the effectiveness of the improved design of SFE-UNet. Compared to U-Net I, which only uses a standard U-Net to process the DAS input, SFE-UNet, which integrates the MSRA-Module and ECA-Attention, achieves a significant improvement in the SSIM metric, reaching 0.837 ± 0.047, while the PSNR also increased to 18.952 ± 2.371. The significant increase in SSIM indicates that SFE-UNet can more accurately reconstruct the structural information of the vasculature. The visual comparison in Fig. 7 further validates this point; the vessels reconstructed by SFE-UNet exhibit superior overall structural clarity and continuity compared to U-Net I, more closely resembling the ground truth image. Similarly, in processing the pixel-wise interpolated data, PFE-UNet achieves a substantial performance leap in both SSIM and PSNR compared to U-Net II. This fully demonstrates that the specific designs within PFE-UNet, such as SK-Attention, the enhanced bottleneck layer, and ECA-Attention, are crucial for effectively learning and extracting fine-grained features from the multi-channel mapped data. The visual results in Fig. 7 show that PFE-UNet can initially recover vascular structures from the pixel-wise interpolated data, with a quality far superior to that of U-Net II, which failed to generate a coherent image, showcasing its powerful feature learning capabilities.

Table 5.

Ablation study on the performance of feature extraction networks (mean ± std).

Model	SSIM	PSNR
U-NetⅠ	0.782 ± 0.060	18.154 ± 2.432
U-NetⅡ	0.442 ± 0.112	12.776 ± 2.300
SFE-UNet	0.837 ± 0.047	18.952 ± 2.371
PFE-UNet	0.7424 ± 0.073	16.317 ± 2.103

Fig. 7.

Impact of feature extraction networks on vascular reconstruction in ablation experiments. From left to right: Ground Truth (GT), U-NetⅠ, SFE-U-Net, U-NetⅡ, PFE-UNet, and the proposed DUAFF-Net.

Next, we further evaluate the effectiveness of different fusion strategies through Table 6 and Fig. 8. First, a key observation is that even with the simplest fusion strategy, ESF (second row of Table 6), combining the features extracted by SFE-UNet and PFE-UNet yields SSIM and PSNR metrics that are significantly superior to any single-branch network shown in Table 5. This strongly validates that our proposed dual-source input itself is a highly promising strategy that provides a fundamental performance boost. Compared to ESF, using only the GCDF-Module for terminal deep fusion (first row of Table 6) increases the SSIM to 0.8770 and the PSNR to 19.7324 dB. This indicates that GCDF can more effectively integrate high-level features through its global context modeling and dense feature refinement. Subsequently, when we introduce the MIAF-Module between SFE-UNet and PFE-UNet for early multi-level feature interaction (supplemented by ESF for terminal fusion), the SSIM significantly improves to 0.8904 and the PSNR to 20.2950 dB. This highlights the critical role of the MIAF-Module in promoting early information complementarity and guidance between the parallel branches, providing higher-quality features for subsequent fusion. Finally, our proposed complete DUAFF-Net architecture, which integrates the early interaction of the MIAF-Module with the terminal deep fusion of the GCDF-Module, achieves the best performance among all configurations, as shown in the fourth row of Table 6. As depicted in Fig. 8, the visual reconstruction results of DUAFF-Net are optimal in terms of the clarity, completeness, and artifact suppression of the vascular network. This fully demonstrates the superiority of the synergistic work of the MIAF and GCDF modules, which effectively maximizes the complementary advantages of the dual-information streams.

Table 6.

Ablation study on the effectiveness of different feature fusion strategies.

MIAF	ESF	GCDF	SSIM	PSNR
×	×	√	0.8770	19.7324
×	√	×	0.8658	19.6524
√	√	×	0.8904	20.2950
√	×	√	0.9014	20.6815

Fig. 8.

Impact of feature fusion strategies on vascular reconstruction in ablation experiments. From left to right: Ground Truth (GT), End Simple Fusion (ESF), GCDF-Module, MIAF-Module + ESF, and the proposed DUAFF-Net.

To further isolate the independent contributions of our proposed modules, we conducted an ablation study utilizing a simple U-Net backbone. As detailed in Table 7, the baseline U-Net I (processing a single DAS input) serves as the standard U-Net. U-Net+MSRA, which added the MSRA-Module (a single-stream module) on top of it, yielded respective SSIM and PSNR gains of 5.9 % and 3.8 %.

Table 7.

Ablation study on the efficacy of different modules applied to a simple U-Net backbone.

Method	U-NetⅠ	U-Net+MSRA	U-Net+MIAF	U-Net+GCDF
SSIM	0.782	0.828	0.852	0.844
PSNR	18.154	18.850	19.501	19.137

In contrast, our proposed MIAF and GCDF are fusion modules designed to intelligently fuse dual-stream data (DAS and pixel-wise interpolation). To test them fairly in a simplified environment, we constructed a parallel dual U-Net backbone—one U-Net processing the DAS input and the other processing the pixel-wise interpolation input—and then used our proposed modules to fuse the features from these two streams.

As shown in Table (6), the SSIM and PSNR for the U-Net+MIAF fusion strategy increased substantially by 9.0 % and 7.4 % compared to the U-Net I baseline, while the U-Net+GCDF strategy also yielded gains of 7.9 % and 5.4 %. This result objectively shows that, even on a simplified U-Net backbone, the gains on key metrics from our two proposed dual-stream fusion strategies (MIAF and GCDF) are also significantly higher than those from the single-stream enhancement module (MSRA).

The visual results in Fig. 9 intuitively verify this point. Compared with the baseline U-Net I and U-Net + MSRA, our U-Net + GCDF and U-Net + MIAF can reconstruct clearer and more continuous vascular networks, and effectively suppress artifacts, which is consistent with the quantitative results in Table 7.

Fig. 9.

Visual comparison of the module ablation study on the U-Net backbone. From left to right: Ground Truth (GT); Baseline U-Net I (single DAS input); U-Net + MSRA (single-stream comparison module); U-Net + GCDF (dual-stream fusion module); U-Net + MIAF (dual-stream fusion module).

4. Discussion

The comprehensive experimental evaluation results of this study clearly indicate that the proposed DUAFF-Net demonstrates significant superiority over the conventional DAS algorithm and several advanced deep learning models (including Y-Net, SwinIR, and AS-Net). This advantage is evident in

both key quantitative evaluation metrics (Table 1, Table 2, Table 4) and visual reconstruction quality (Fig. 4, Fig. 5, Fig. 6) for the tasks of retinal vasculature, brain MRI photoacoustic reconstruction., and in vivo mouse abdomen imaging.The superior performance of DUAFF-Net is primarily attributed to several key innovative designs in its architecture, the effectiveness of which has been further validated through systematic ablation studies (see Table 5, Table 6, and Fig. 7, Fig. 8).

Furthermore, this ablation study in Table 7 and Fig. 9 successfully disentangles module contributions from the backbone design. It first validates MSRA as an effective single-stream enhancement strategy (outperforming U-Net I), and then compellingly demonstrates that our MIAF/GCDF fusion modules represent a superior dual-stream strategy. The results confirm that, even on a simple U-Net backbone, our fusion modules provide performance gains that significantly surpass both the standard baseline and the single-stream MSRA module.

First, the core MSRA-Module within SFE-UNet is crucial for enhancing the feature representation capability for DAS images. As shown in Table 5, SFE-UNet, which employs the MSRA-Module, achieves a significant improvement in the SSIM metric from 0.782 to 0.837 compared to the standard U-Net I. The core advantage of the MSRA-Module lies in its multi-scale design, which can effectively decouple artifacts and true structures of different sizes within the DAS image, while the CBAM attention mechanism guides the network to focus on information-rich regions. This provides high-quality feature maps containing reliable macro-structural priors for the subsequent feature fusion, which is fundamental to DUAFF-Net's ability to accurately reconstruct brain contours and the paths of major vessels.

Second, the most innovative component of our framework is the MIAF-Module, which establishes an information pathway between the two streams early in the network. The ablation study (Table 6) confirms that introducing the MIAF-Module brings a significant performance gain (SSIM improving from 0.8658 to 0.8904). Its success can be attributed to two points: First, at the bottleneck layer, it injects the global context-rich features from PFE-UNet into SFE-UNet, effectively correcting macro-structural deviations caused by the limited-view acquisition. Second, in the decoder, MIAF enhances reconstruction fidelity and effectively avoids artifact generation by injecting high-frequency details from PFE-UNet into SFE-UNet, which also improves image clarity. The green-boxed region in sample one of Fig. 4 clearly illustrates this, where several competing methods erroneously reconstruct artifacts not present in the GT, a problem DUAFF-Net does not exhibit. This fully demonstrates that the MIAF-guided feature fusion allows the model to more robustly learn true structures rather than hallucinating details.

At the end of the network, the GCDF-Module is responsible for the final intelligent integration of the deeply optimized features from both streams. Compared to simple feature concatenation, the application of the GCDF-Module brings a clear performance improvement (Table 6, SSIM increasing from 0.8658 to 0.8770). We believe that its built-in GCBlock ensures the overall consistency of the reconstructed structures by establishing global dependencies, while the DenseBlock performs deep refinement of the fused features through dense feature reuse. As shown in the yellow-boxed region of sample two in Fig. 4, DUAFF-Net is able to avoid the erroneous connections generated by AS-Net, which intuitively demonstrates the superiority of the GCDF-Module in final feature arbitration and refinement, thereby ensuring the high fidelity of the output image.

To address the physical domain gap between 2D simulations and real 3D tissues, and to mitigate the risk of overfitting, we conducted a rigorous cross-domain validation using in vivo mouse experiments (Section 3.4.3). By employing the specific parameter configurations and transfer learning strategy tailored for the in vivo data (Section 3.2), DUAFF-Net quantitatively outperformed baseline methods on the tested in vivo dataset. Importantly, this result does not imply that 2D models are physically equivalent to 3D modeling; rather, it empirically demonstrates that transfer learning adaptation serves as a feasible and effective pathway to bridge the simulation-to-reality gap.

However, we acknowledge that limitations remain. Although incorporating real noise distributions via transfer learning partially mitigates the inherent risk of deep learning hallucination, future clinical deployment will require integrating uncertainty quantification techniques. Furthermore, while the current work focuses on cross-sectional tomography, extending the DUAFF-Net architecture to support native 3D reconstruction remains a critical direction to fully capture 3D wave propagation characteristics (e.g., geometric attenuation). Future work will also focus on verifying the method on larger-scale clinical datasets with greater pathological diversity to ensure broad generalizability.Finally, the computational complexity of DUAFF-Net may be higher compared to more lightweight models. Exploring model compression or acceleration methods will be an important direction for subsequent research.

5. Conclusion

Focusing on the core challenges in the field of photoacoustic tomography (PAT) reconstruction, this study successfully proposes and validates a novel deep learning framework, DUAFF-Net, which aims to significantly enhance the reconstruction fidelity and detail clarity of photoacoustic images. The core of DUAFF-Net lies in its innovative feature fusion mechanism, which effectively combines the macro-structural information provided by the initial reconstruction from the conventional Delay-and-Sum (DAS) algorithm with fine-grained features deeply extracted from pixel-wise interpolated data, thereby achieving a synergistic capture of both macro-structures and micro-details.

Extensive experiments and systematic ablation studies based on simulated photoacoustic data of retinal vasculature and brain structures, as well as real-world in vivo mouse abdomen data, have strongly demonstrated that, especially under challenging conditions such as limited-view and sparse data acquisition, the proposed DUAFF-Net is significantly superior to the conventional DAS algorithm and several advanced deep learning models, including Y-Net and SwinIR, in terms of both reconstruction quality and various performance metrics (such as SSIM and PSNR).Notably, the in vivo results empirically verify that, empowered by transfer learning, DUAFF-Net possesses strong generalization capabilities to bridge the domain gap between simulation and real biological tissues. Furthermore, systematic ablation studies have verified the effectiveness of the core innovative designs proposed in DUAFF-Net. The results clearly indicate that all components—including the multi-scale SFE-UNet optimized for DAS, the PFE-UNet for processing specialized interpolated data, and the key fusion modules like the MIAF for early interaction and the GCDF for deep integration—have made clear and positive contributions to improving the quality of the final reconstructed image. The synergy of these components resulted in significant performance gains over the single-stream baseline (U-Net I), with cumulative improvements of up to 14 % in PSNR and 15 % in SSIM. This fully validates the advanced nature of our proposed method's overall architecture and the soundness and efficiency of each module's design.

In summary, DUAFF-Net stands as an efficient and robust framework for photoacoustic image reconstruction, offering an effective solution to overcome common challenges in the current PAT field, such as artifacts and loss of detail. This research provides a valuable reference for the development of high-performance photoacoustic imaging technology and its related applications. In the future, we plan to further validate the generalization capability of DUAFF-Net on a broader range of diverse pathological clinical data, extend the DUAFF-Net framework to a 3D architecture, and explore corresponding model lightweighting methods to enhance its practical utility.

CRediT authorship contribution statement

Yixin Lai: Writing – review & editing, Writing – original draft, Visualization, Methodology, Investigation, Formal analysis, Conceptualization. Zhengnan Yin: Writing – review & editing, Validation, Methodology, Conceptualization. Qiong Zhang: Writing – review & editing, Supervision, Resources, Project administration, Funding acquisition, Conceptualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Biographies

Yixin Lai received his bachelor's degree from Lingnan Normal University in 2023. Currently, he is pursuing a Master's degree in Electronic Information at Shantou University. His main research interests include photoacoustic imaging and image reconstruction.

Qiong Zhang received her Ph.D. degree in biomedical engineering from the University of Science and Technology of China in 2012. Currently she works as associate professor in Shantou University Medical College. Her research interests include medical image processing, ultrasound signal processing, photoacoustic imaging.

Zhengnan Yin received an M.S. degree in physics from Shantou University, Guangdong, China, in 2024. His research interests include low level computer vision, multimodal interaction and learning.

Data availability

Data will be made available on request.