PA-Fuse: deep supervised approach for the fusion of photoacoustic images with distinct reconstruction characteristics

Abstract

The methods available for solving the inverse problem of photoacoustic tomography promote only one feature–either being smooth or sharp–in the resultant image. The fusion of photoacoustic images reconstructed from distinct methods improves the individually reconstructed images, with the guided filter based approach being state-of-the-art, which requires that implicit regularization parameters are chosen. In this work, a deep fusion method based on convolutional neural networks has been proposed as an alternative to the guided filter based approach. It has the combined benefit of using less data for training without the need for the careful choice of any parameters and is a fully data-driven approach. The proposed deep fusion approach outperformed the contemporary fusion method, which was proved using experimental, numerical phantoms and in-vivo studies. The improvement obtained in the reconstructed images was as high as 95.49% in root mean square error and 7.77 dB in signal to noise ratio (SNR) in comparison to the guided filter approach. Also, it was demonstrated that the proposed deep fuse approach, trained on only blood vessel type images at measurement data SNR being 40 dB, was able to provide a generalization that can work across various noise levels in the measurement data, experimental set-ups as well as imaging objects.

1. Introduction

Photoacoustic (PA) imaging is a non-invasive imaging modality which renders optical resolution at ultrasonic depth inside the biological tissue [1–4]. In photoacoustic tomography (PAT), a short (nano-second) laser pulse irradiates the biological tissue under investigation. The incident optical energy is absorbed by the biological tissue, which results in the rise in temperature, generating pressure waves because of thermoelastic expansion and photon absorption. The ultrasonic detectors record the pressure waves at the imaging domain boundary. These acquired pressure waves are utilized in an inverse fashion for computing the initial pressure distribution. The information about the tissue functional parameters, such as hemoglobin, water, oxyhemoglobin, bilirubin, melanin [5] is given by the optical absorption. The applications of PA imaging includes ophthalmology [6], oncology [7,8], neurosciences [9], dermatology [10], cardiology [11], and monitoring tissue health conditions.

Photoacoustic tomography involves solving the initial problem by utilizing the data acquired at the boundary at time “t” to compute the initial pressure distribution at time “t” = 0. This initial value problem is solved using several reconstruction techniques. These include analytical algorithms (delay and sum, and back-projection (BP)) as well as time-reversal (TR) algorithms [12, 13]. These are computationally effective, but need large amount of data for providing meaningful results as compared to the model based methods. The requirement of huge data result in prolonged scan time or costly experimental setups. Also, the setups utilized for capturing the pressure waves generally cover only an aperture resulting in limited data scenarios [14–16]. Often, reconstruction using analytical methods or time reversal method for limited data cases result in poor image quality and poor contrast. The model-based methods generally have improved quantitative accuracy in these limited data cases [17–20]. Since model based methods are iterative, they are more robust to noise and are used for real time reconstructions [12, 21]. These methods are typically computationally expensive as compared to analytical methods and require careful choice of reconstruction (regularization) parameters. The reconstructed image characteristics obtained using these methods are typically complimentary as compared to the analytical methods [22–24].

The post-processing of reconstructed images is a standard procedure in the field of medical imaging. The reconstructed images are generally histogram normalized or a filter is used to enhance the image characteristics. In PA imaging, there have been previous attempts to enhance the resolution for Optical Resolution Photoacoustic Microscopy [25]. The method involves the use of blind deconvolution to estimate the blur kernel using Wiener filter [25]. Similar to these efforts, other techniques were developed which consists of two steps. The model based image reconstruction is the first step and utilizing the model resolution matrix to deblur the image is the second step [22]. The deblurring step is computationally expensive and requires computational time comparable with the first step of model based reconstruction [22,26].

Computer vision community has recently used fusion methods to combine the best features from two different images to get a single image that is improved version compared to the individual input images [27]. Especially in areas of multi-focus and multi-camera fusion, the use of fusion based methods have shown to produce superior results [28]. Recently introduced modified guided filtering [29] based fusion method in PA imaging was shown to improve the reconstructed images. It uses a set of regularization parameters, which need to be selected carefully, to obtain the final image output. In this work, a convolutional neural network (CNN) based architecture for fusion is proposed, to combine the best features from two input images obtained using distinct reconstruction methods. Firstly, the reconstructions are obtained using two different methods (example being Linear Back Projection and Lanczos Tikhonov). These reconstructed images were used as input and the network was trained using the target phantom as the output image. The proposed technique was compared with standard reconstruction techniques, such as total variation (TV) regularization and Lanczos Tikhonov as well as contemporary fusion methods, such as modified guided filtering. It was shown to out perform these methods in a systematic evaluation involving numerical as well as experimental studies. The proposed method (termed as PA-Fuse) was evaluated at different noise levels to show the robustness of the architecture. It was also assessed with inputs other than the network was trained for and shown that the reconstruction results obtained were conforming to broad utility. The experiments were performed with structures that were not seen by the network and also on an experimental phantom data and shown to perform better than contemporary methods. Also, to show the universal utility of the proposed PA-Fuse method, reconstructed results from different experimental setups were utilized to improve the reconstruction results without retraining the network.

2. Photoacoustic image reconstruction

The forward model of PA imaging involves the computation of pressure waves collected by the ultrasonic detectors placed at the imaging domain boundary. The propagation of PA waves inside the tissue is governed by [4]

$${\nabla }^{2}P(x,t)-\frac{1}{c^2}\frac{{\partial }^{2}P(x,t)}{\partial t^2}=\frac{-\beta }{C_p}\frac{\partial H(x,t)}{\partial t},$$ (1)

Here, the pressure at time t and position x is denoted by P(x, t), the speed of sound is denoted by c, specific heat capacity is denoted by C_p, the thermal expansion coefficient is denoted by β, and the energy deposited per unit time per unit volume is denoted by H(x, t).

2.1. System matrix building

The setup used here for numerical experiments is similar to the ones used in Refs. [23,24,30]. It is summarized here briefly for completeness. The dimension of the imaging domain is n × n. It is reshaped into a vector by resizing it to a vector of size n² × 1 and denoted as x. A (dimension of m × n²) denotes the system matrix. Here m denotes the product of number of time samples acquired by the detectors and the total number of detectors present at the boundary. Impulse response is calculated for each entry of the image. This impulse response is stacked as columns of the system matrix. The data denoted as b of size m × 1 denotes the data collected by the detectors at the imaging domain boundary [23,24]. The forward model is given as

$$\mathbf{A}x=b,$$ (2)

Here, b represents the measurement vector (m × 1 (m = 512 × 100)) and x represents the solution vector (n² × 1 (n = 201)). The Linear Back-Projected (LBP), which is the simplest image reconstruction method, image can be obtained as [22,31]

$$x_{\mathit{LBP}}={\mathbf{A}}^{\mathbf{T}}b,$$ (3)

with T representing the transpose. The Back-projection (BP) methods are computationally efficient as they are single step in nature [21]. The BP based methods are not used for quantitative imaging as they provide only qualitative results. The starting vector for the iterative reconstruction techniques is generally the BP image. [20,24]. Here, the BP image (x_LBP) was used as the input image for the fusion based methods.

2.2. Lanczos Tikhonov regularization method

Since the photoacoustic reconstruction is ill-conditioned for the limited data cases, Tikhonov regularization is a widely used technique for the regularization of these ill-conditioned problems. This technique minimizes the residue of the equations forming the system matrix combined with a regularization term. The objective function is given as

$$\mathrm{\Omega }={\Vert \mathbf{A}x-b\Vert }_2^2+\delta {\Vert x\Vert }_2^2$$ (4)

with δ being the regularization parameter. Higher value of regularization parameter over-smooth the image, while a lower value magnifies the noise level in the image. The minimized function and the l₂ norm are represented as Ω and ‖.‖₂ respectively. Minimizing this equation results in

$$\left({\mathbf{A}}^{\mathbf{T}}\mathbf{A}+\delta \mathbf{I}\right)x_{\mathit{Tikhonov}}={\mathbf{A}}^{\mathbf{T}}b$$ (5)

Solving Eq. 5 takes O(n⁶) operations and thus computationally expensive. Thus, Lanczos bidiagonalization has been employed here to reduce the dimensionality of the problem at hand, which reduces the memory requirement and also reduces the number of operations [23]. The dimensionality reduction is performed using the Lanczos bidiagonalization method on the system matrix (A) [32]. The Lanczos matrices (E_k - left Lanczos matrix, F_k - right Lanczos matrix), the bidiagonal matrix (B_k) and the system matrix (A) are associated as [23,30]:

$${\mathbf{E}}_{\mathbf{k}+\mathbf{1}}({\gamma }_0{e}_1)=b$$ (6)

$$\mathbf{A}{\mathbf{F}}_{\mathbf{k}}={\mathbf{E}}_{\mathbf{k}+\mathbf{1}}{\mathbf{B}}_{\mathbf{k}}$$ (7)

$${\mathbf{A}}^{\mathbf{T}}{\mathbf{E}}_{\mathbf{k}+\mathbf{1}}={\mathbf{F}}_{\mathbf{K}}{\mathbf{B}}_{\mathbf{k}}^{\mathbf{T}}+{\delta }_{k+1}{n}_{k+1}{e}_{k+1}^T$$ (8)

The l₂ norm of b is represented by γ₀ and e_k denotes the unit vector of dimension k (which is 1 at the k’th position and 0 at other places). The dimension of F_k and E_k are (n² × k) and (m × k), correspondingly, where k denotes the number of bidiagonalization iterations. Utilizing Eqns. (6,7,8), the residual can be written as:

$$b-\mathbf{A}x={\mathbf{E}}_{\mathbf{K}+\mathbf{1}}\left({\gamma }_0{e}_1-{\mathbf{B}}_{\mathbf{K}}{x}^{(k)}\right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}x={\mathbf{F}}_{\mathbf{K}}{x}^{(k)}$$ (9)

x^(k) denotes the lower dimensional representation of x when k << n². Utilizing the property ${\mathbf{E}}_{\mathbf{K}+\mathbf{1}}^{\mathbf{T}}{\mathbf{E}}_{\mathbf{K}+\mathbf{1}}=\mathbf{I}$ and substitution of Eq. (9) in Eq. (4) and, the new cost function is

$$\tilde{\mathrm{\Omega }}={\Vert {\gamma }_0{e}_1-{\mathbf{B}}_{\mathbf{K}}{x}^{(k)}\Vert }_2^2+\delta {\Vert x^{(k)}\Vert }_2^2$$ (10)

The new solution becomes

$$x_{\mathit{estimate}}^{(k)}={\left({\mathbf{B}}_{\mathbf{K}}^{\mathbf{T}}{\mathbf{B}}_{\mathbf{K}}+\delta \mathbf{I}\right)}^{-1}{\gamma }_0{\mathbf{B}}_{\mathbf{K}}^{\mathbf{T}}{e}_1$$ (11)

$$x_{\mathit{estimate}}={\mathbf{F}}_{\mathbf{K}}{x}_{\mathit{estimate}}^{(k)}$$ (12)

using the first order condition. For a fixed k and regularization parameter δ [33], the $x_{\mathit{estimate}}^{(k)}$ is an estimate of x^(k). For a given value of the regularization parameter δ and k = n², x_estimate = x_Tikhonov. x_estimate is an approximation of x_Tikhonov for all other cases. There exist an optimal value of k which decreases the error between x_estimate and x_Tikhonov [30,32]. The quality of reconstructed image is highly dependent on choice of k and δ, the contemporary method for an automated choice of these parameters is based on error estimate, which provides optimal solution. For more details of the error estimate method, the reader can refer to Ref. [34]. The solution obtained via optimal choice of k and δ using error estimate method will be refereed as x_LTO, with LTO referring to Lanczos Tikhonov solution obtained via Optimal choice of reconstruction parameters.

2.3. Total variational regularization method

The smooth nature of the solution in Tikhonov regularization (Eq. 4) is promoted because of the l₂ norm present in the objective function. Another method to solve this ill-conditioned problem is by proposing a term minimizing the variation in the solution, and called as total variation (TV) regularization method [35,36]. The cost minimization function can be given as

$$\mathrm{\Gamma }={\Vert \mathbf{A}x-b\Vert }_2^2+\eta {\Vert x\Vert }_{\mathit{TV}},$$ (13)

η denotes the regularization parameter and ‖.‖_TV represents the isotropic total variation [37]. Alternating Direction Method of Multipliers (ADMM) framework is utilized for TV regularization which is given in Ref. [38]. The Split Augmented Lagrangian shrinkage Algorithm (SALSA) (given in Algorithm 1), was utilized in this work. The LBP image (Eq. 3) was used as the initialization of x for the SALSA algorithm. The variables f₀ and u₀ were initialized to zero. The reconstructed PA image obtained using TV regularization is represented by x_TV.

Algorithm 1.

SALSA Algorithm

Input: A, η, b, k = 0, u0, f0

Output: x k+1

Repeat till convergence is satisfied

$x_{k+1}=\text{arg}\hspace{0.17em}{\mathit{min}}_x{\Vert \mathbf{A}x-b\Vert }_2^2+\eta {\Vert x-u_k-f_k\Vert }_2^2$

$u_{k+1}=\text{arg}\hspace{0.17em}{\mathit{min}}_u\tau \varphi (u)+(\eta /2){\Vert x_{k+1}-u-f_k\Vert }_2^2$

fk+1 = fk − (xk+1 − uk+1)

k = k + 1

2.4. Image guided filter based fusion

Recently proposed guided filter based method combines the feature from two different reconstructions techniques and was shown to perform better as compared to the contemporary methods for photoacoustic imaging [29]. The cartoon of the fusion process for PA imaging is given in Fig. 1. For complete information the reader can refer to Ref. [29]. The modified guided filtering algorithm is given here (Algorithm 2) for completeness.

Fig. 1.

Flowchart of the fusion process. The typical computational times for each process is given below each block.

Algorithm 2.

Modified Guided Filter(x_I, x_G, w, ∊, α, β), g_o(.) represents performing operation ‘o’ on the arguments

Input: xI − Input Image to be filtered (xLTO or xTV)

xGI − Guidance Image (xLBP)

w − Radius of the Window (patch size)

∊, α, β − Regularization Parameters

Output xF − Output Image

meanxGI = gmean(xGI); meanxI = gmean(xI)

corrxGI = gmean(xGI. * xGI); corrxGI−I = gmean(xGI. * xI)

varxGI = corrxGI − meanxGI. * meanxGI; covxGI−I = corrxGI−I − meanxGI. * meanxI

a = (covxGI−I./(varxGI + ∊))α; b = meanxI − β * a. * meanxGI

meanb = gmean(b); meana = gmean(a)

xF = meana. * xGI + meanb

2.5. PA-Fuse (proposed method)

Network architecture: With I₁ and I₂ as input (obtained from two reconstruction methods as discussed in above sections), the objective is to fuse them together to generate a final image that has combined features of I₁ and I₂. This is objective of the proposed PA-Fuse (𝒩) model (the fusion process is depicted in Fig. 1). This model has been motivated from the DeepFuse model proposed by Prabhakar et al. [39]. The network architecture of the proposed PA-Fuse model is shown in Fig. 2. In here, Siamese network was utilized to extract similar features for both I₁ and I₂. The mathematical description of operations in the proposed PA-Fuse network are given below. Let us denote the feature map of Image j at layer i as $F_i^j$, convolutional filters at layer c as f_c, and convolution operation on d with filter e as d ⊛ e. Then, the series of operations will be as follows:

$$\begin{array}{ll}\hfill F_1^1& =I_1⊛f_1;F_1^2=I_2⊛f_1\hfill \\ \hfill F_2^1& =F_1^1⊛f_2;F_2^2=F_1^2⊛f_2\hfill \\ \hfill F_{\mathit{fused}}& =F_2^1+F_2^2\hfill \\ \hfill F_3& =F_{\mathit{fused}}⊛f_3\hfill \\ \hfill F_4& =F_{l_3}⊛f_4\hfill \\ \hfill F_5& =F_4⊛f_5\hfill \\ \hfill \mathit{Output}& =F_5⊛f_6\hfill \end{array}$$ (14)

The similar features are extracted by applying same convolutional filters f₁ for layer 1 (and f₂ for layer 2) to both images I₁ and I₂. As filter sizes are same, the extracted feature types will be same. Thus, they can be merged directly.

Fig. 2.

Network architecture of proposed PA-Fuse model. It consists of three stages: feature encoder to extract salient features, feature aggregator to combine them, and feature decoder to reconstruct fused image. In the feature encoder section, feature extraction of input images I₁ and I₂ using two convolutional layers with tied weights were achieved. The extracted features are fused using sum operation and fused image was reconstructed using four convolutional layers.

The proposed model is sufficient enough to handle increase in training samples. Hence, network parameters updating is not necessary. However, there is a need to adjust training strategy to stabilize the training. This could be achieved by annealing the learning rate parameter over the training epochs. One of the widely followed annealing strategy is to reduce the learning rate by a fixed factor every few epochs (which was not needed here).

In this part, two convolutional layers that has 32 filters with kernel size of 3×3 and 64 filters with kernel size being 5×5 were deployed. Both input images (size of h × w) are passed into the Siamese network to extract feature blobs of size h × w × 1 × 64. These layers have 51,584 trainable parameters. As each feature in the extracted feature blob represent similar feature type, they can be combined by sum operation. The combined feature map is passed onto four convolutional layers to generate a single channel fused image (Ô). The network 𝒩 was trained with ℒ₂ loss computed between Ô and ground truth O_gt:

$$\mathit{Loss}={\Vert 𝒩(I_1,I_2)-O_{\mathit{gt}}\Vert }_2$$ (15)

These four layers have 265,601 parameters to be trained beyond the combined feature map (sum operation in Fig. 4). The training images were extracted from databases STARE [40], DRIVE [41], and CHASE [42]. The total number of images used were 4600 with 201 × 201 dimension. These images were used to generate the experimental data (b) using the k-Wave toolbox [43]. The data was then added with a white Gaussian noise resulting in Signal-to-Noise-Ratio (SNR) of 40 dB. This data was utilized in an inverse framework to generate the images using the Back-Projected (x_LBP) and Lanczos Tikhonov Optimal (x_LTO) reconstruction techniques. The total patches (having size 77 × 77) numbering 1,00,000 out of which 80,000 were used for training the network while 20,000 were used for validation of the network. The code was written in Keras [44] using Tensorflow [45] as the backend for training and testing the network. The ‘Mean Square Error’ was utilized as loss function and Adam optimizer for training the network [46]. The learning rate, the first momentum, and the second momentum were set to be 1 × 10⁻³, 0.9, and 0.999 respectively.

Fig. 4.

Target blood vessel (BV) phantom is shown in (a). The reconstructed initial pressure distribution with SNR of measurement data being 40 dB using (b) LBP, (c) LTO, and (d) TV regularization. The results obtained using modified guided filter with (b) as guiding image and to be guided image being (c) and (d) are correspondingly given in (e) and (g). The results using the PA-Fuse with one input image being (b) and another being (c) and (d) are correspondingly given in (f) and (h).The RMSE and CNR of these images are correspondingly presented in Table-1.

The computations performed in this work, including the training, was on a workstation having Dual Intel Skylake Xeon 4116 (24 core) with 2.10 GHz clock speed having two NVIDIA Tesla P100 12GB GPU having 64GB RAM. The total training time was approximately 8.27 hours.

3. Figures of merit

To compare the quality of the PA image obtained using till now discussed methods, the image metrics, such as Signal-to-Noise Ratio (SNR), Root Mean Square Error (RMSE), and Contrast-to-Noise Ratio (CNR) were used. For numerical phantoms, RMSE and CNR were employed as the target initial pressure was known for these, while for experimental phantoms SNR was used as the evaluation criteria.

3.1. Root mean square error (RMSE)

RMSE is generally used to compare the reconstruction quality as an absolute metric and is defined as [47,48]:

$$\mathit{RMSE}\left(x^{\mathit{target}},x^{\mathit{recon}}\right)=\sqrt{\frac{\mathrm{\Sigma }(x^{\mathit{target}}-x^{\mathit{recon}})}{N}}$$ (16)

where target initial pressure distribution is denoted by x^target, the reconstructed initial pressure distribution is denoted by x^recon, and the total number of pixels by N. The lower the value, the closer it is to the reconstructed image quantitatively.

3.2. Contrast-to-Noise Ratio (CNR)

CNR is defined as follows [49–52]:

$$\mathit{CNR}=\frac{m_{\mathit{roi}}-m_{\mathit{back}}}{{(v_{\mathit{roi}}^2{a}_{\mathit{roi}}+v_{\mathit{back}}^2{a}_{\mathit{back}})}^{\mathrm{1/2}}}$$ (17)

where v and m denotes the variance and the mean for the background (back) and the region of interest (roi) for the initial pressure distribution reconstruction. The $a_{\mathit{roi}}=\frac{A_{\mathit{roi}}}{A_{\mathit{total}}}$ and $a_{\mathit{back}}=\frac{A_{\mathit{back}}}{A_{\mathit{total}}}$ represents the area ratio. A_roi denotes the area of the region of interest and A_back denotes the area of the background. Note that the initial pressure values being greater than zero is treated as the roi and rest becomes the background. Higher value of CNR represents improved contrast recovery of the reconstruction method.

3.3. Signal-to-noise ratio (SNR)

SNR is defined as [53]:

$${\mathit{SNR}}_r(\mathit{in}\hspace{0.17em}\mathit{dB})=20\times {\mathit{log}}_{10}\left(\frac{P_{\mathit{Pressure}}}{n_I}\right)$$ (18)

where P_Pressure represents the peak initial pressure distribution and n_I represents the standard deviation of the image. Higher SNR represents improved quality and less noise in the reconstructed image. The term SNR_r was used to differentiate it from the SNR in the measurement vector (b).

4. Numerical and experimental studies

Three different numerical phantoms with distinct characteristics were used in this work for comparing the performance of discussed methods. The numerical phantoms will be helpful in objectively assessing the performance of the reconstruction as the measurement of the actual initial pressure distribution is challenging in the real experimental scenarios, including phantoms. The ultrasonic detectors are placed at a radius of 22 mm and the dimension of the computational grid is 501 × 501 (0.1 mm/pixel). The experimental data generation was performed on a high dimensional grid (401 × 401) and the reconstruction was done on a lower dimensional grid (201 × 201). The Fig. 3 depicts these grids in a schematic manner, including the position of 100 acoustic detectors. White Gaussian noise was added to the data generated to result in SNR levels of the measured data being 30 dB to 50 dB for testing the PA-Fuse model. An open source MATLAB based toolbox k– WAVE [43] was deployed for generation of the data. The data collection was performed at a sampling rate of 20 MHz and time samples were 512. Hundred ultrasonic transducers with 70 % bandwidth and a center frequency of 2.25 MHz placed at equi-distance on the boundary were utilized to collect the boundary data (b). Thus, the system matrix formed had a dimension of 51200 × 40401. The simulation was performed assuming speed of sound in medium as 1500 m/s, along with the assumption that medium is non-absorptive and non-dispersive.

Fig. 3.

Schematic diagram of photoacoustic data acquisition geometry along with depiction of position of 100 acoustic detectors (shown by dots) around the imaging domain. The data generation grid had a dimension of 401×401 and reconstruction grid was 201×201.

The maximum initial pressure distribution of 1 kPa was set for these numerical phantoms. The Blood vessel (BV) phantom consists of thin and thick blood vessels imitating the real blood vessel structures (shown in Fig. 4(a)). The modified Derenzo phantom was also used which consists of circular objects having varying radius to evaluate the performance of the proposed method in terms of varying size (shown in Fig. 6(a)). Another phantom was made which consists of the letters ‘PAT’ (shown in Fig. 7(a)) was utilized to show the superiority of the reconstruction performance of the proposed PA-Fuse method in terms of recovering the sharp edges.

Fig. 6.

Target modified Derenzo phantom is shown in (a). The reconstructed initial pressure distribution with SNR of measurement data being 40 dB using (b) LBP, (c) LTO, and (d) TV regularization. The results obtained using modified guided filter with (b) as guiding image and to be guided image being (c) and (d) are correspondingly given in (e) and (g). The results using the PA-Fuse with one input image being (b) and another being (c) and (d) are correspondingly given in (f) and (h). The RMSE and CNR of these images are correspondingly presented in Table-1. The red arrows in the TV (reconstructed or fused) images show non-uniform background induced artifacts.

Fig. 7.

Target PAT phantom is shown in (a). The reconstructed initial pressure distribution with SNR of measurement data being 40 dB using (b) LBP, (c) LTO, and (d) TV regularization. The results obtained using modified guided filter with (b) as guiding image and to be guided image being (c) and (d) are correspondingly given in (e) and (g). The results using the PA-Fuse with one input image being (b) and another being (c) and (d) are correspondingly given in (f) and (h).The RMSE and CNR of these images are correspondingly presented in Table-1.

To validate the proposed method experimentally, a horse hair phantom data having triangular-shape and in vivo rat brain data obtained using different image acquisition set ups with Nd:YAG laser and pulsed laser diode (PLD) laser were utilized respectively. The details about the experimental set up and data acquisition for triangular-shaped horse hair phantom data was provided in Refs. [54] & [55]. The in vivo rat brain data collection details were provided in Ref. [55]. Note that all animal experiments conducted as part of this work followed the guidelines and regulations accepted by the institutional Animal Care and Use committee of Nanyang Technological University, Singapore (Animal Protocol Number ARF-SBS/NIE-A0263).

5. Results and discussion

The initial pressure distribution reconstruction for the Blood Vessel (BV) phantom using the LBP is given in Fig. 4(b). The reconstruction using the Lanczos Tikhonov (LTO) method and TV based regularization were shown in Fig. 4(c) and Fig. 4(d) respectively. The optimal parameters for the Lanczos method are calculated by utilizing the error estimate method. The fused result using the guided filter based approach using the LBP reconstruction as the guiding image and the input image being x_LTO, and x_TV were shown in Fig. 4(e) and Fig. 4(g) respectively. The reconstruction results using the proposed PA-Fuse method with LBP as one input image and other being x_LTO and x_TV were presented in Fig. 4(f) and Fig. 4(h) respectively. Table-1 presents the quantitative comparison of these results. The improvement obtained with proposed PA-Fuse method using the x_LTO as another input (x_LBP was always one of the standard input) is 85.41% in RMSE and 147.46% in CNR as compared to the guided filter based fusion method with the same input. In case of measurement data SNR level being 30 dB, the improvement using PA-Fuse in terms of RMSE and CNR was 41.68% and 214.42% respectively and similar improvement was observed in case of 50 dB SNR level. Note that the training of the PA-Fuse network was performed with measurement data SNR being 40 dB, but the network was able to generalize and provide improved performance at other SNR levels as well. On similar note, even though the network was trained with x_LTO, from results it is clear that the proposed PA-Fuse model was able to improve the TV based reconstruction result as well.

Table 1.

The figures of merit, RMSE and CNR, for the resultant images using reconstruction methods examined here with SNR in measurement data being 30 dB, 40 dB, and 50 dB utilizing three numerical phantoms (blood vessel (BV), Derenzo, and PAT). For the case of guided filter, guiding image was always LBP and other input being given being given as argument. The results of the proposed method, PA-Fuse, are given in bottom two rows with one input being LBP image and another image given as argument. The highest RMSE and CNR in each column are marked in bold.

Phantom	Modified Derenzo						Blood Vessel (BV)						PAT
SNR	30 dB		40 dB (Fig. 6)		50 dB		30 dB		40 dB (Fig. 4)		50 dB		30 dB		40 dB (Fig. 7)		50 dB
Metric	RMSE	CNR	RMSE	CNR	RMSE	CNR	RMSE	CNR	RMSE	CNR	RMSE	CNR	RMSE	CNR	RMSE	CNR	RMSE	CNR
LBP	0.1646	0.17	0.1666	0.17	0.1675	0.17	0.2373	0.74	0.2383	0.74	0.2380	0.74	0.2242	1.01	0.2202	1.02	0.2192	1.02
LTO	0.1260	0.73	0.1249	1.12	0.1534	1.14	0.2031	1.59	0.1901	1.99	0.1904	1.95	0.2614	1.86	0.2172	2.83	0.2256	3.09
TV	0.2061	0.95	0.1231	1.48	0.1166	1.49	0.1499	2.47	0.1505	2.60	0.1695	2.48	0.2077	2.80	0.1335	3.99	0.1393	4.08
Guided Filter (LTO)	0.0902	1.15	0.0689	2.57	0.0499	4.20	0.1771	2.02	0.1310	3.75	0.1047	4.74	0.2442	2.24	0.1852	4.29	0.0899	8.07
Guided Filter (TV)	0.0726	2.63	0.0802	2.40	0.0783	2.66	0.1406	3.19	0.1368	3.50	0.1312	3.78	0.1007	6.54	0.1154	6.25	0.1154	7.12
PA-Fuse (LTO)	0.0181	31.55	0.0063	159.82	0.0079	93.75	0.0610	10.03	0.0229	9.28	0.0192	8.83	0.0208	14.92	0.0052	20.34	0.0057	21.49
PA-Fuse (TV)	0.0542	1.55	0.0294	2.33	0.0168	4.19	0.0291	9.03	0.0191	9.02	0.0310	8.77	0.0229	7.38	0.0060	16.75	0.0053	16.70

The LBP image (Fig. 4(b)) is smooth, has many streak artifacts, and also lacks sharp features as available data is limited. The LTO/TV regularized reconstruction gives sharp features with caveat that they enhance the noise present in the boundary data. These features (less noise in the LBP image and sharp features present in the LTO/TV image) are combined to provide a fused output that is less noisy with reduced streak artifacts. The frequency spectrum plots of corresponding to reconstructed images of the blood vessel phantom given in Fig. 4 were shown in Fig. 5. The low frequency components are at the center of these plots while the edges represents the high frequency components. The resulting fused images Fig. 4(e–h) significantly changes the spectrum of the frequency components (correspondingly in Fig. 5(e–h)). The fused image contains low-frequency components of the LBP and high-frequency components present in the LTO/TV. The fused image filters/excludes high frequency components, which is typically attributed to the noise in the reconstructed image, while retaining the other components which contains the useful information. Thus fused images figures of merit are significantly improved compared to individual images (refer to Table-1).

Fig. 5.

The logarithm of amplitude of the Fourier spectrum of results pertaining to (a) Target BV Phantom (spatial distribution is given in Fig. 4(a)), (b) Linear Back-Projection (LBP) (spatial distribution is given in Fig. 4(b)), (c) Lanczos-Tikhonov-Optimal (LTO) (spatial distribution is given in Fig. 4(c)), (d) Total Variation (TV) (spatial distribution is given in Fig. 4(d)), (e) Modified guided filter based fused output with LBP and LTO as inputs (spatial distribution is given in Fig. 4(e)), (f) Proposed PA-Fuse output with LBP and LTO as inputs (spatial distribution is given in Fig. 4(f)), (g) Modified guided filter based fused output with LBP and TV as inputs (spatial distribution is given in Fig. 4(g)), and (h) Proposed PA-Fuse output with LBP and TV as inputs (spatial distribution is given in Fig. 4(h)).

The reconstructed initial pressure distribution for the modified Derenzo phantom using the LBP is given in Fig. 6(b). The reconstruction using the Lanczos Tikhonov Optimal (LTO) method and total variation (TV) based methods are shown in Fig. 6(c) and Fig. 6(d) respectively. The fused result using the guided filter based approach using the LBP reconstruction as the guiding image and the input image being x_LTO, and x_TV were presented in Fig. 6(e) and Fig. 6(g) respectively. The proposed PA-Fuse method results with one input image being x_LBP and another being x_LTO and x_TV were shown in Fig. 6(f) and Fig. 6(h) respectively. Table-1 presents the RMSE and CNR (figures of merit) for the methods discussed here. The improvement obtained with the proposed PA-Fuse using the x_LTO as input is 90.85% in RMSE and 6118.67% in CNR as compared to the guided filter based fusion method with the same input for measurement data SNR being 40 dB. For case SNR being 30 dB, the improvement obtained in terms of RMSE and CNR was 75% and 1099.61% respectively using the proposed PA-Fuse compared to guided filter based approach and similar improvement was observed when measurement data SNR being 50 dB. It is also important to observe that the PA-Fuse was trained with images consisting of blood vessel networks, but the network was able to generalize to provide improved fusion even for case of circular structures as present in this numerical phantom case. Note that the artifacts observed in Fig. 6(h) are mainly due to the TV based approach reconstruction result not being optimal for these circular objects and the same artifacts can be seen in the original reconstructed image (Fig. 6(d)) as well. The artifacts are due to non-uniform background and presence of black patches in the Fig. 6(d) (indicated by red arrows). These non-uniformities were not present in the LTO image (Fig. 6(c)) and hence the final output was not affected for the fusion techniques. Even Figs. 6(g) and (h) has these artifacts manifested as one of the input image was Fig. 6(d), with guided filtered fusion (Fig. 6(g)) providing more optimal result in this case. Note that the proposed PA-Fuse network was not trained on these type of images with circular objects, thus provides only sub-optimal result. The network can be retrained with images having these features to eliminate the observed black patches in the fused result (indicated by red arrows in Fig. 6(h)).

The reconstructed initial pressure distribution for the ‘PAT’ phantom using the LBP is given in Fig. 7(b). The reconstruction using the Optimal Lanczos method and total variation based method were given in Fig. 7(c) and Fig. 7(d) respectively. The guided filter based fusion results using the LBP reconstruction as the guiding image and the input image being x_LTO, and x_TV were shown in Fig. 7(e) and Fig. 7(g) respectively. Results with the proposed PA-Fuse method with one input image being LBP and another input image being x_LTO, and x_TV are shown in Fig. 7(f) and Fig. 7(h) correspondingly. The RMSE and CNR, similar to earlier results were presented in Table-1. The improvement obtained with proposed PA-Fuse using the x_LTO as the input is 95.49% in RMSE and 225.44% in CNR as compared to the guided filter based fusion method. In case of 30 dB SNR level of the measurement data, the improvement obtained in terms of RMSE and CNR was 79.34% and 128.13% respectively and similar improvement was observed in case of 50 dB SNR level. Again, the PA-Fuse network has not been trained with images containing any straight/sharp edges, it was able to provide improved fusion for this numerical phantom as well.

Also from Table-1, it can been seen that for the proposed PA-Fuse method with input image obtained using Lanczos Tikhonov (LTO) consistently resulted in better figures of merit. This can be attributed to optimal choice of reconstruction parameters that were chosen based on error estimation technique utilized in LTO, which results in better quality image.

The initial pressure distribution reconstruction for the experimental horse hair phantom, data collected with utilization of Nd:YAG laser, using the LBP method is shown in Fig. 8(a). The PA images reconstructed using the Optimal Lanczos method and total variation based methods were shown in Fig. 8(b) and Fig. 8(c) respectively. The guided filter based fusion results using the LBP reconstruction as the guiding image and the input image being x_LTO, and x_TV were presented in Fig. 8(d) and Fig. 8(f) respectively. Results with the proposed PA-Fuse method with one input image being LBP and another input image being x_LTO, and x_TV are shown in Fig. 8(e) and Fig. 8(g) respectively. The improvement obtained with proposed PA-Fuse using the x_LTO as the input is 7.11 dB in SNR_r as compared to the guided filter based fusion method. The background artifacts, even in these experimental scenario for which the PA-Fuse was trained, were completely eliminated in the reconstructed image using the proposed PA-Fuse method with a clear improvement in the contrast.

Fig. 8.

The initial pressure distribution reconstruction for the experimental horse hair phantom data obtained using Nd:YAG laser with LBP, LTO, and TV regularization methods are given in (a), (b), and (c) respectively. The results obtained using modified guided filter with (a) as guiding image and to be guided image being (b) and (c) are correspondingly given in (d) and (f). The results using the PA-Fuse with one input image being (a) and another being (b) and (c) are correspondingly given in (e) and (g). SNR_r (in dB), of each PA image is given below.

To further evaluate the robustness of the proposed method, experiment involving data acquired using PLD laser was conducted. In here, the SNR of the measurement data is expected to be much lower compared to the case of utilizing Nd:YAG laser. The reconstructed initial pressure distribution in this case using the LBP method is shown in Fig. 9(a). The reconstructed PA images using the LTO and TV based methods were presented in Fig. 9(b) and Fig. 9(c) respectively. The guided filter based fusion results using the LBP reconstruction as the guiding image and the input image being x_LTO, and x_TV were shown in Fig. 9(d) and Fig. 9(f) respectively. Results with the proposed PA-Fuse method with one input image being LBP and another input image being x_LTO, and x_TV are shown in Fig. 9(e) and Fig. 9(g) respectively. The SNR of these reconstructed images were given below. The improvement obtained with proposed PA-Fuse using the x_TV as the input is 7.77 dB in SNR_r in comparison to modified guided filter based fusion method, in agreement to the earlier case (Fig. 8). This experiment showed the versatility of the proposed PA-Fuse working across experimental set-ups without the additional need for retraining.

Fig. 9.

The initial pressure distribution reconstruction for experimental horse hair phantom data obtained using PLD laser with LBP, LTO, and TV regularization methods are given in (a), (b), and (c) respectively. The results obtained using modified guided filter with (a) as guiding image and to be guided image being (b) and (c) are correspondingly given in (d) and (f). The results using the PA-Fuse with one input image being (a) and another being (b) and (c) are correspondingly given in (e) and (g). SNR_r (in dB), of each PA image is given below.

One of the PA imaging important application is in the area of pre-clinical imaging. The reconstruction results obtained using in vivo rat brain were presented in Fig. 10. The reconstruction is shown in Fig. 10(a) using the LBP method. The PA images obtained using LTO and TV reconstruction methods were given in Fig. 10(b) and Fig. 10(c) respectively. The guided filter based fusion results using the LBP reconstruction as the guiding image and the input image being x_LTO, and x_TV were shown in Fig. 10(d) and Fig. 10(f) respectively. Results with the proposed PA-Fuse method with one input being LBP and another input image being x_LTO, and x_TV are shown in Fig. 10(e) and Fig. 10(g) respectively. The improvement obtained with proposed PA-Fuse using the x_LTO as the input is 3.44 dB as compared to the guided filter based fusion method. Again, the utility of PA-Fuse method even in this in vivo case was clearly demonstrated through this experiment. Another set of experiments were performed where the reconstruction obtained using Delay and Sum (DAS) was used instead of LBP as the input image. The initial pressure distribution obtained using Delay and Sum (DAS) [54] method was given in Fig. 10(b). It was used as one input for the PA-Fuse method instead of the LBP image and the experiments were performed for the same set of LTO and TV images. There is an improvement obtained in SNR_r for the fused result as compared to the LTO and TV image (last row of Fig. 10). Since the network was not trained for these DAS images, the improvement obtained was not optimal. Even though the SNR_r obtained using DAS compared to rest reconstruction methods (including LBP, LTO, and TV) is significantly better, as DAS is only a analytical technique, it does not provide any quantitative results (the image units are arbitrary). This experiment also confirms that the proposed network including guided filter based methods needs to be trained/tailored for the input images. Other reconstruction results (other than those trained/tailored) may not provide improved fused output.

Fig. 10.

The initial pressure distribution reconstruction for the rat brain in-vivo data with LBP, Delay-and-Sum (DAS), LTO, and TV regularization methods are given in (a), (b), (c), and (d) respectively. The results obtained using the modified guided filter with (a) as guiding image and to be guided image as (c) and (d) are shown in (e) and (g) respectively. The results using the PA-Fuse with one input image being (a) and another being (b) and (c) are correspondingly given in (f) and (h). The results obtained using the modified guided filter with (b) as guiding image and to be guided image as (c) and (d) are shown in (i) and (k) respectively. The results using the PA-Fuse with one input image being (b) and another being (c) and (d) are correspondingly given in (j) and (l). SNR_r (in dB), of each PA image is given below.

Image post-processing techniques are commonly used for improving the reconstructed images across imaging modalities. In this work, fusion based approach was utilized as a post-processing step for improving the photoacoustic image reconstruction. Unlike, multi-focus experiments conducted in computer vision, here the input images were obtained using distinct reconstruction methods. One standard input that was utilized for these fusion approaches is linear back projected image, which is easy to obtain in real-experiments and provide good qualitative assessment of acquired data. Also, the fusion methods that were presented here (both guided filter as well as proposed PA-Fuse) are highly computationally efficient with their run times being as low as 1 milli-second. Through systematic evaluation via numerical, experimental, and in-vivo cases, it was clearly demonstrated that the PA-Fuse network outperformed the guided filter based approach and is generalizable across various measurement noise levels, experimental conditions as well structures present in the target/expected photoacoustic image. The PA-Fuse model performed extremely well with the suppression of background noise (streaking artifacts), improvement of the reconstructed image and generalization to unseen phantoms. Here LTO was used as one of the input image to train the network, while the testing was done even on the TV images, which also gave complimentary features as compared to the LBP based image. Hence the method generalizes even on images reconstructed using different techniques showing the superiority of the proposed method. Overall, the deep fusion based method show very good promise in terms of improving the reconstructed photoacoustic images and will surely pave a way for making photoacoustic imaging being main stream modality to image the soft tissue vascularity.

6. Conclusions

A deep learning based fusion model was proposed in this work to improve the model-based reconstruction for photoacoustic tomography. The proposed PA-Fuse model was shown to be a generic model working well across different measurement noise levels, structures in the target image, as well as experimental set-ups. It was also demonstrated that the input image for the fusion model from other reconstruction methods (than the trained one) can also be utilized for achieving improved performance. The improvement in the SNR of the reconstructed image as compared to the state-of-the-art fusion method is as high as 7.77 dB for in-vivo rat brain, making it very attractive in real-time. Moreover, as opposed to the state-of-the-art fusion methods, the proposed PA-Fuse model being fully an automated approach that is driven by the data, it does not require any choice of reconstruction parameters and thus eliminates the bias associated with careful choice of these parameters.

Funding

DST-ICPS division under Data Science (DS) research program (Proposal No: T-851).

Disclosures

The authors declare that there are no conflicts of interest related to this article.