Multi-branch attention prior based parameterized generative adversarial network for fast and accurate limited-projection reconstruction in fluorescence molecular tomography

Abstract

Limited-projection fluorescence molecular tomography (FMT) allows rapid reconstruction of the three-dimensional (3D) distribution of fluorescent targets within a shorter data acquisition time. However, the limited-projection FMT is severely ill-posed and ill-conditioned due to insufficient fluorescence measurements and the strong scattering properties of photons in biological tissues. Previously, regularization-based methods, combined with the sparse distribution of fluorescent sources, have been commonly used to alleviate the severe ill-posed nature of the limited-projection FMT. Due to the complex iterative computations, time-consuming solution procedures, and less stable reconstruction results, the limited-projection FMT remains an intractable challenge for achieving fast and accurate reconstructions. In this work, we completely discard the previous iterative solving-based reconstruction themes and propose multi-branch attention prior based parameterized generative adversarial network (MAP-PGAN) to achieve fast and accurate limited-projection FMT reconstruction. Firstly, the multi-branch attention can provide parameterized weighted sparse prior information for fluorescent sources, enabling MAP-PGAN to effectively mitigate the ill-posedness and significantly improve the reconstruction accuracy of limited-projection FMT. Secondly, since the end-to-end direct reconstruction strategy is adopted, the complex iterative computation process in traditional regularization algorithms can be avoided, thus greatly accelerating the 3D visualization process. The numerical simulation results show that the proposed MAP-PGAN method outperforms the state-of-the-art methods in terms of localization accuracy and morphological recovery. Meanwhile, the reconstruction time is only about 0.18s, which is about 100 to 1000 times faster than the conventional iteration-based regularization algorithms. The reconstruction results from the physical phantoms and in vivo experiments further demonstrate the feasibility and practicality of the MAP-PGAN method in achieving fast and accurate limited-projection FMT reconstruction.

1. Introduction

Fluorescence molecular tomography (FMT) is a promising optical imaging technique with high sensitivity and depth-resolution capability, which can quantitatively detect the three-dimensional (3D) distribution information of fluorescent probes inside biological tissues [1–3]. Compared with other structural imaging techniques such as CT and MRI, FMT can detect the occurrence of diseases at the molecular or cellular level in advance. Over the past few decades, FMT has become popular and attractive in numerous physiological and clinical studies and has been extensively applied in various small animal model-based studies, including tumor diagnosis [4–6], drug development [7,8], therapy assessment [9,10], etc.

However, due to the high scattering properties of photons in biological tissues and insufficient surface fluorescence measurement data, the reconstruction process of FMT is viewed as a severely ill-posed inverse problem. To improve the reconstruction performance of FMT, researchers have made great efforts in both imaging algorithms and hardware systems. Generally, anatomical information (provided by XCT [10,11] or MRI [12,13]) can be exerted in the forward model of photon propagation, or further inserted into the inverse problem in a priori form to alleviate the ill-posedness. Regularization technology is another universal and effective method used to mitigate the ill-posed nature of the inverse problem of FMT [14–23]. So far, a variety of regularization algorithms, including Tikhonov regularization [14–16], sparse-promoting regularization (L0, L1, L1/2, L2,1, Lasso) [17–21], total variation regularization (TV) [22,23], reweighted L2 and L1 regularization [24], etc., have been developed to improve the reconstruction performance of FMT. Furthermore, to obtain better reconstruction results, these algorithms typically employ full-angle projection data to recover the distribution of fluorescent probes. In a full-angle FMT imaging system, it is generally necessary to acquire 36-72 projections, or even more projections, to obtain a higher reconstructed image resolution. However, the acquisition of high spatial sampling data usually takes a long time (about 5 min-45 min, [25]), which is not suitable for rapid visualization of biological processes in vivo. Therefore, it is necessary to develop a new reconstruction method or system that can achieve fast and accurate FMT reconstruction using limited projection data.

Over the past decades, some researchers have proposed several limited-projection algorithms to achieve fast FMT reconstruction by shortening the experimental data acquisition time [26–31]. Bai’s research group [26] firstly explored the influence of insufficient measurements on the FMT reconstruction and then analyzed the limited-projection reconstruction through numerical simulation and phantom experiments, and finally verified the conclusion by in vivo experiments. Later, Cao et al. [27] proposed a reconstruction method based on projected restarted conjugate gradient normal residual to achieve limited-projection FMT reconstruction. Xie et al. [17] proposed a reconstruction method based on iterative reweighted L1 (IRL1) regularization for reducing artifacts with limited measurements. He et al. [28] proposed a feasible region extraction strategy based on a double mesh for limited-projection FMT. Later, their group [29] proposed a limited-projection reconstruction method combining smoothing L0 norm (SL0) and feasible regions. Although these methods indeed improve the reconstruction performance, they were, after all, based on iterative regularization methods, which involved complex iterative computations and time-consuming solution procedures that are not conducive to the fast dynamic reconstruction of biological processes. In addition, some researchers added anatomical information to increase axial resolution and localization accuracy of limited projections [30,31]. For example, Karin Radrich [30] improved the limited-projection reconstruction performance by using a co-registered x-ray computed tomography (x-CT) scan. Later, they [31] further investigated multimoiety imaging in the context of a limited-projection hybrid FMT, and x-CT implementation and the further registration with positron emission tomography (PET) data. Although the reconstruction performance was improved to a degree, the complex hardware system setup and registration posed a great challenge to achieving the fast FMT reconstruction.

Recently, deep learning technology has shown great advantages in achieving fast and accurate full-angle FMT reconstruction [32–37]. For example, our group [32] proposed a 3D-En-Decoder network, which significantly improves the reconstruction speed. Afterward, our research group [33] further proposed a UHR-DeepFMT network, which achieved ultra-high spatial resolution in an extremely short time and enormously improved the reconstruction accuracy. Besides, some other research groups have also presented learning-based reconstruction network models, such as DGMM [34], ResGCN [35], SAE [36], and KNN [37], which have also effectively accelerated the reconstruction speed and accuracy. However, these deep learning models generally use mean square error as the loss function. Although the reconstruction quality and efficiency are improved compared with traditional reconstruction methods, they tend to lose edge details and have relatively poor robustness. Therefore, it is necessary to continuously strive to develop more efficient deep learning methods for limited-projection FMT.

The deep learning structure framework based on generative adversarial network (GAN) is a generative model that has emerged in recent years and has been widely used in the field of image reconstruction for various imaging modalities [38–40], and has achieved good reconstruction performance. In this work, inspired by the great advantages of GAN in image reconstruction, we propose a novel multi-branch attention prior based parameterized generative adversarial network (MAP-PGAN) for the limited-projection reconstruction of FMT. We specifically design a parameterized multi-branch attention prior mechanism for the generator network in MAP-PGAN. First, the multi-branch attention presents the feature representations extracted from the decoder to the encoder module to provide weighted sparse a priori information for the fluorescent sources. In addition, the parameterized skip connections are set as learnable parameters of the network to enhance the nonlinear capability and generalization performance of the network. Collectively, the parameterized multi-branch attention prior mechanism extracts rich edge and morphological features and generates spatial attention maps to guide the limited-projection reconstruction of FMT. Cases in different situations are designed to fully demonstrate the effectiveness of the MAP-PGAN method in reconstructing fluorescent targets within different limited projections. The proposed MAP-PGAN method is demonstrated through numerical simulations, physical phantoms, and in vivo experiments. Furthermore, we also compared the traditional L2-regularized, L1-regularized, L0-regularized methods, and cutting-edge learning-based 3D-En-Encoder and UHR-DeepFMT methods to prove the feasibility and superiority of the proposed MAP-PGAN method.

2. Method

This section starts with an introduction of the deep learning framework for limited-projection FMT, followed by the proposed MAP-PGAN method, specifically including the design of the parameterized generator, discriminator and the total loss function. The evaluation index to measure the reconstruction performance of reconstruction algorithms is described finally.

2.1. Deep learning framework for FMT

Limited-projection FMT implies that fewer projections are employed to reconstruct the 3D distribution of fluorescent targets inside biological tissues. Unlike the previous conventional reconstruction process involving forward physical modeling and inverse problem solving [26–31], the learning-based limited-projection FMT reconstruction aims to directly establish the nonlinear mapping relationship between surface fluorescence measurements and internal fluorescent sources. The deep learning framework for limited-projection FMT is as follows,

$$\mathrm{\Phi }=f_{\mathrm{\Theta }}(x)$$ (1)

where $\mathrm{\Phi }$ denotes the surface fluorescence measurements, which is the input of the deep neural network; x represents the distribution of internal fluorescent sources to be reconstructed, which is the output of the deep neural network. Solving the inverse problem of (1) with deep learning can be phrased as seeking a nonlinear mapping function $f_{\mathrm{\Theta }}$ that satisfies the pseudo-inverse property,

$$x_{rec}={f_{\mathrm{\Theta }}}^{-1}(\mathrm{\Phi })$$ (2)

where $x_{rec}$ denotes the distribution of the fluorescent sources to be reconstructed.

The main goal of the deep learning-based FMT reconstruction method is to use a deep neural network to fit a nonlinear mapping function $f_{\mathrm{\Theta }}$ to achieve the smallest errors between the reconstructed and true results. Therefore, the inverse problem of the limited-projection FMT can be solved by (3),

$$\underset{\mathrm{x}\ge 0}{\mathrm{arg\; min}}{||x_{rec}(\mathrm{\Phi })-x||}_2^2$$ (3)

where $\mathrm{\Phi }$ represents the weight parameters to be updated, x represents the true results. The weight parameters of the deep network are updated iteratively by minimizing the mean square error between x and $x_{rec}(\mathrm{\Phi })$ . From Eqs. (1–3), it is clear that the deep learning-based FMT reconstruction framework is a data-driven approach, and its performance depends mainly on the well-designed network and the optimization scheme for network training.

2.2. MAP-PGAN reconstruction network

The proposed MAP-PGAN architecture is shown in Fig. 1, which mainly consists of a parameterized generator, a discriminator, and a specially designed multi-branch attention prior module. The parameterized generator aims to synthesize high-quality FMT images using parameterized skip connections combined with hierarchical features of limited-projection FMT images, the discriminator guarantees the estimated FMT images are close to the true FMT images by way of feedback to train the generator, and the multi-branch attention prior module is introduced into the generator to provide parameterized weighted sparse prior information for the fluorescent sources.

Fig. 1.

The topological framework of the proposed MAP-PGAN network. (a) The MAP-PGAN network structure. (b) The parameterized generator network. (c) The multi-branch attention prior module.

2.2.1 Parameterized generator

In MAP-PGAN, the generator is composed of a U-shaped topology structure and a parameterized skip connection between the encoder and the decoder. In the generator, we specially design a multi-branch attention prior module, as shown in Fig. 1(c). The topology of the proposed multi-branch attention prior module is that the feature representations extracted from the network encoder are provided to the decoder module. Due to the different spatial sizes of feature maps, linear interpolation is first performed to upsample the feature maps. The features are then compressed to suppress irrelevant information in the channel dimension and reduce computation. Next, the compressed features are concatenated respectively in the channel dimension for feature fusion. Finally, the four feature maps are fused together to obtain the final prior results.

We define the feature map extracted from the network encoder as $X_i\in {\mathbb{R}}^{D\times C_i\times H_i\times W_i}$ , where D denotes the depth (i.e., the number of FMT projections), C denotes the number of channels, and H, W denote the height and width of the FMT projections or feature maps, respectively. Feature compression and fusion are calculated as follows,

$$Z_5=W_{c5}^T{X}_5\oplus X_4$$ (4)

$$Z_4=(W_{c4}^T(W_4^T{Z}_5))\oplus X_3$$ (5)

$$Z_3=(W_{c3}^T(W_3^T{Z}_4))\oplus X_2$$ (6)

where $Z_5\in {\mathbb{R}}^{D\times C_4\times H_4\times W_4}$ represents the fused features of X₅ and X₄, $Z_4\in {\mathbb{R}}^{D\times C_3\times H_3\times W_3}$ represents the fused features of X₄ and X₃, $Z_3\in {\mathbb{R}}^{D\times C_2\times H_2\times W_2}$ represents the fused features of X₃ and X₂, $W_{c5}\in {\mathbb{R}}^{D\times C_5\times C_4}$ , $W_{c4}\in {\mathbb{R}}^{D\times C_4\times C_3}$ and $W_{c3}\in {\mathbb{R}}^{D\times C_3\times C_2}$ represent the corresponding compressed convolutions, $W_4\in {\mathbb{R}}^{D\times C_4\times C_4}$ represents the fused convolution of X₅ and X₄, ${\mathrm{W}}_3\in {\mathbb{R}}^{D\times C_3\times C_3}$ represents the fused convolution of X₄ and X₃, and $\oplus$ represents the feature concatenation. The final output of the multi-branch attention prior module is calculated as follows:

$$Y=W_{out}^T(W_2^T{Z}_3)$$ (7)

where $W_2\in {\mathbb{R}}^{D\times C_2\times C_2}$ represents the fused convolution of X₃ and X₂, and $W_{out}\in {\mathbb{R}}^{D\times C_2\times 1}$ represents the fused output convolution.

To further improve the reconstruction performance, we introduce a parameterized skip connection mechanism. Compared with conventional skip connection, the parameterized magnitude factor of the skip connection path α_i, i $\in$ (1, 2, …, 5) is set as a learnable parameter of the network, which is updated during the back propagation. Such settings could add extra non-linearity capacity of the network and enhance the effectiveness of skip connections.

We use F_i, i $\in$ (1, 2, 3, 4, 5) to represent the feature maps from the encoder, and Y to denote the prior attention map, the parameterized feature map F_pi, i ∈ (1, 2, …, 5) as follows:

$$F_{pi}=\alpha F_i+Y\cdot F_i,i\in (i=1,2,\dots ,5)$$ (8)

Finally, we introduce the attention prior module into multiple branches in the parameterized skip connections of the MAP-PGAN network to extract the multi-branch attention prior information. The multi-branch attention prior module extracts rich edge and morphology features from the encoder and generates spatial attention maps to guide the limited-projection reconstruction of FMT.

2.2.2. Discriminator

The discriminator uses a ResNet structure with residual connections, including four ResBlock modules. Each DenseBlock module contains a convolutional kernel, a layer norm layer, and a Relu activation function. Each ResBlock module is followed by a Max-pooling operation, and then a discriminant score is obtained using the average pooling layer and the FC fully connected layer.

2.2.3 Total loss function

The proposed MAP-PGAN network adopts the Wasserstein distance as an optimization objective for network training. The total loss of the generator is divided into three parts: the WGAN loss prevents the blurring of the FMT images and reduces sharp parts of the FMT images, the L1 loss focuses on image details, and the attention prior loss is used to constrain fluorescent source items for better limited-projection reconstruction. The total loss function of MAP-PGAN is shown in (9)-(10),

$$\ell (G)=-{\mathrm{E}}_{x\sim p_e}[D(x)]+{\lambda }_2{\ell }_{L1}(G)+{\lambda }_3{\ell }_{prior}(G)$$ (9)

$$\ell (D)=-{\mathrm{E}}_{x\sim p_r}[D(x)]+{\mathrm{E}}_{x\sim p_e}[D(x)]+{\lambda }_1{\mathrm{E}}_{x\sim p_{\hat{x}}}{[||{\mathrm{\nabla }}_{x}D(x)|{|}_p-1]}^2$$ (10)

where $\ell (G)$ and $\ell (D)$ denote the generator and discriminator loss function, E (·) denotes the expectation operator, $\mathrm{\nabla }$ denotes the gradient operator, P_e denotes the distribution of the estimated FMT images, P_r denotes the distribution of the real FMT images, P_x is uniformly sampled along the straight lines connecting the paired estimated FMT images and the true FMT images, and λ_i (i = 1, 2, 3) are the weighted penalty parameters.

2.3. Evaluation index

To evaluate the reconstruction performance of the proposed MAP-PGAN method, several quantitative indexes, including contrast-to-noise ratio (CNR) [21, 23, 32, 33], positioning error (PE) [18, 23, 32, 33], and intersection over union (IOU) [19, 23] were calculated.

CNR is used to measure whether the localized features of the reconstructed image are well recovered in the background noise, and higher values are preferred. For a volume of interest (VOI) containing the fluorescent target, the CNR is calculated as,

$$\mathrm{C}\mathrm{N}\mathrm{R}=\frac{m_{VOI}-m_{BG}}{\sqrt{(w_{VOI}{\sigma }_{VOI}^2+w_{BG}{\sigma }_{BG}^2)}}$$ (11)

where $m_{VOI}$ and $m_{BG}$ represents the mean concentration values in the VOI and background, respectively; ${\sigma }_{VOI}^2$ and ${\sigma }_{BG}^2$ represents the standard deviations in the VOI and background, respectively; $w_{VOI}$ and $w_{BG}$ represents weight parameters used to adjust the relative volumes of VOI and background, respectively.

PE is used to calculate the distance variation between the centroids of the true fluorescent target and the reconstructed fluorescent target, and is defined as follows:

$$\mathrm{P}\mathrm{E}={||P_{rec}-P_{true}||}_2$$ (12)

where P_rec denotes the centroid coordinates of the reconstructed region, and P_true denotes the centroid coordinates of the true fluorescence region.

The IOU is a metric used to evaluate the performance of morphological reconstruction and is defined as,

$$\mathrm{I}\mathrm{O}\mathrm{U}=\frac{\mathrm{x}\cup \hat{\mathrm{x}}}{\mathrm{x}\cap \hat{\mathrm{x}}}$$ (13)

where $\mathrm{x}$ and $\hat{\mathrm{x}}$ represent the true and reconstructed fluorescent target, respectively.

3. Experiments and results

In this section, numerical simulations, physical phantoms and in vivo experiments were conducted. We set three different cases with different projections (i.e., 3, 6, 10) to fully investigate the effectiveness and practicality of the proposed MAP-PGAN method. Moreover, the commonly used algorithms, including the L2-regularized, L1-regularized, L0-regularized methods and the cutting-edge learning-based 3D-En-Decoder and UHR-DeepFMT methods, were compared in this study. The L2-regularized method used analytical solution with the optimal regularization parameter determined by the L-curve method [41]. The optimal regularization parameters of the L1-regularized and L0-regularized methods were determined by the generalized cross validation (GCV) method [42].

All the processes of the L2-regularized, L1-regularized, and L0-regularized methods were executed on a desktop computer with a 3.20 GHz Intel Core i7-8700 CPU and 32 GB RAM. The three deep network models (i.e., 3D-En-Decoder, UHR-DeepFMT, and MAP-PGAN) were implemented by PyTorch and Python 3.8 on Ubuntu 20.04 system, and all procedures for training were operated on an NVIDIA GeForce GTX 2080Ti with 11GB memory. The test was conducted on another computer with a 3.20 GHz Intel Core i7-8700 CPU and 32 GB RAM.

3.1. Numerical simulation experiments and results

We performed numerical simulations on a cylindrical phantom (30 mm in diameter and 15 mm in height) with two fluorescent targets (3 mm in diameter and 3 mm in height) located at a symmetric center of the X-axis with an edge-to-edge distance (EED) of 3 mm (Case 1, displayed in (a) and (b)), at depths of 0 and 3 mm from the center of the X-axis (Case 2, displayed in (c) and (d)) and at a depth of 5 mm from the X-axis with an offset of 2.1 mm from the Y-axis (Case 3, displayed in (e) and (f)), as shown in Fig. 2. The 3D coordinates of the fluorescent targets for the three different cases are shown in Table 1. The bottom of the cylindrical model was set to Z = 0 cm. The center of the fluorescent target was located at the height of Z = 0.75 cm. The excitation source was also located at the height of Z = 0.75 cm, and the emitted light was detected within a field of view (FOV) of 160° on the other side of the cylindrical surface (displayed in (g)), and the detector sampling distance was set to 0.2 cm. The absorption and reduced scattering coefficients were set to µ_a = 0.02 cm^-1, µ_0s = 10.0 cm^-1 [25]. The fluorescence field was set to 1 and assumed to be constant during the imaging process. To simulate the real experimental situation, 5% Gaussian noise was added to the measured data.

Fig. 2.

Numerical simulation experimental setups. (a), (c), (e) The view of XY plane of the three cases. (b), (d), (f) The view of the XZ plane of the three cases. Two fluorescent targets were placed inside a cylinder phantom, where Case1 is the centrosymmetric two targets, and Case 2 and Case 3 are the non-centrosymmetric two targets with different spatial distributions. (g) The top view of the distribution of sources when capturing three projections. The red dot S1 denotes the position of the point source in the first projection. And pink rectangle distributed among the 160° FOV indicates the detector (i.e., EMCCD) corresponding to S1. The red dots S2 and S3 represent the source positions in the second and third projection, respectively.

Table 1. The 3D coordinates of the fluorescent targets in three different cases.

Target	Case 1	Case 2	Case 3
Target 1 (cm)	(0, 0, 0.30)	(0, 0, 0.75)	(0.5, 0.36, 0.75)
Target 2 (cm)	(0, 0, -0.30)	(0, -0.6, 0.75)	(0.5, -0.36, 0.75)

To explore the influence of different projections on the reconstruction results, we used different projections for FMT reconstruction. Figure 3 shows the reconstruction results of the proposed MAP-PGAN method using 3, 6, and 10 projections. It can be found that the proposed MAP-PGAN method can achieve excellent reconstruction in terms of localization accuracy and morphological reconstruction for the centrosymmetric fluorescent targets (i.e., Case 1) and non-centrosymmetric fluorescent targets (i.e., Case 2 and Case 3). It is worth mentioning that the proposed method can effectively achieve reconstruction using only three projections, and as the number of projections increases (i.e., from 3 to 10 projections), the shape and edge of the fluorescent target are recovered better and better.

Fig. 3.

The numerical simulation reconstruction results of the proposed MAP-PGAN method using different projections. (a)-(c) for Case 1, (d)-(f) for Case 2 and (g)-(i) for Case 3. Each row shows the 2D/3D rendering results of different cases. Each column shows the 2D/3D rendering results of different projections. The true locations of the two targets are depicted by the white circles, and the phantom boundary is depicted by the outer red circle.

To quantitatively evaluate the effect of different projections on the proposed MAP-PGAN method, we plot the quantitative analysis curves between the reconstruction performance and the number of projections for the three different cases, as shown in Fig. 4. Consistent with the results of qualitative analysis, PE1 and PE2 decrease with the increase of the number of projections, indicating that the localization accuracy becomes better with the increasing number of projections. Conversely, CNR and IOU increase with the increase of the number of projections, indicating that the image quality and reconstruction similarity get better with the increasing number of projections.

Fig. 4.

Qualitative analysis curves between the reconstruction performance and the number of projections of the proposed MAP-PGAN method for the three different cases in numerical simulations.

Furthermore, to further demonstrate the superiority of the proposed MAP-PGAN method in the limited-projection reconstruction of FMT. We compare five reconstruction methods, including the classical L2-regularized, L1-regularized, and L0-regularized methods and cutting-edge learning-based 3D-En-Decoder and UHR-DeepFMT methods. Taking Case 1 as an example, the results of different reconstruction algorithms using three projections are shown in Fig. 5. The L2-regularized method obtains over-smoothed reconstruction results and fails to reconstruct two fluorescence targets. The iteration-based L1-regularized and L0-regularized methods are severely disturbed by noise and fail to accurately reconstruct the positions and shapes of the two fluorescent targets. The learning-based 3D-En-Decoder method improves the peak signal-to-noise ratio of the reconstructed images, but it only reconstructs one of the two fluorescent targets well. In contrast, the UHR-DeepFMT and MAP-PGAN methods can reconstruct two fluorescent targets, but the reconstruction similarity and shape recovery of the UHR-DeepFMT method is slightly worse than those of the MAP-PGAN method. Therefore, it can be concluded that the MAP-PGAN method achieves the best performance in limited-projection FMT reconstructions.

Fig. 5.

The numerical simulation reconstruction results of six different methods using three projections. The 2D and 3D displays of the L2-regularized ((a1)-(a6)), L1-regularized ((b1)-(b6)), L0-regularized ((c1)-(c6)), 3D-En-Decoder ((d1)-(d6)), UHR-DeepFMT ((e1)-(e6)) and MAP-PGAN ((f1)-(f6)) methods. The true positions of the two targets are indicated by white circles, the phantom boundaries are indicated by red outer circles.

Likewise, we quantitatively evaluate the reconstruction performance of different algorithms, and the results are shown in Table 2. Consistent with the results of qualitative analysis, the proposed MAP-PGAN method achieves the highest CNR and the largest IOU for the three different cases, indicating that the MAP-PGAN method achieves the best peak signal-to-noise ratio and reconstruction similarity. In addition, the PE1 and PE2 obtained by the MAP-PGAN method are the smallest, indicating that the proposed method has the highest localization accuracy among all reconstruction algorithms.

Table 2. Quantification analysis using three projections for the three different cases in the numerical simulation.

Cases	Methods	CNR	IOU	PE (cm)
				PE1	PE2
Case 1	L2-regularized	13.0	0.30	0.27	0.26
	L1-regularized	6.79	0.45	0.12	0.11
	L0-regularized	11.2	0.43	0.14	0.09
	3D-En-Decoder	25.5	0.49	0.21	0.08
	UHR-DeepFMT	24.3	0.65	0.10	0.12
	MAP-PGAN	25.2	0.72	0.07	0.09
Case 2	L2-regularized	15.3	0.28	0.26	0.25
	L1-regularized	9.15	0.42	0.16	0.18
	L0-regularized	13.7	0.51	0.11	0.10
	3D-En-Decoder	21.3	0.55	0.07	0.10
	UHR-DeepFMT	22.1	0.61	0.09	0.09
	MAP-PGAN	22.7	0.67	0.08	0.08
Case 3	L2-regularized	14.4	0.32	0.25	0.24
	L1-regularized	5.06	0.36	0.10	0.16
	L0-regularized	15.7	0.52	0.12	0.13
	3D-En-Decoder	16.4	0.45	0.15	0.17
	UHR-DeepFMT	15.8	0.54	0.14	0.11
	MAP-PGAN	17.6	0.65	0.10	0.10

To further verify the superiority and generalization of the MAP-PGAN network, the ablation experiments and the three-target experiments were also performed and the results were shown in Supplement 1 ^{(1.4MB, pdf)} . Quantitative (Fig. S1 and Fig. S2) and qualitative analysis (Table. S1 and Table. S2) show that the MAP-PGAN network was more accurate and stable in experiments.

3.2. Physical phantom experiments and results

To demonstrate the effectiveness of the proposed MAP-PGAN method, we further executed the physical phantom experiments. The phantom experimental platform was built by our laboratory, as shown in Fig. 6. A continuous wave (CW) laser (LD-808/8-750/2WG, Zhongshan Dingshuo Optoelectronics Technology Co., Ltd., China) was used as the excitation light source, provided by excitation light coupled to an optical fiber through a bandpass filter at 780 ± 10 nm. A high-sensitivity electron-multiplying charge-coupled device (EMCCD) camera (iXon DU-897, Andor Technologies, Belfast, Northern Ireland) with a Nikkor 60 mm, f/2.8D lens (Nikon, Melville, NY, USA) was placed on the opposite side of the imaged phantom. The EMCCD was cooled to -70 °C to reduce dark noise. During the data acquisition process, the fluorescence projection data were acquired for reconstruction, and an 840 ± 6 nm emission filter was placed in front of the CCD camera to filter out the excitation light. The exposure time for acquiring a 1 × 1 fluorescence projection was 1.5 s. Finally, 3, 6, and 10 fluorescence projections were acquired for reconstruction.

Fig. 6.

Sketch of the FMT imaging system.

The experimental setup of the phantom experiments is the same as that of Case 1 in the numerical simulations. The phantom model is a transparent glass cylinder with a diameter of 3.0 cm and a height of 8.0 cm, filled with 1% intralipid and water. Two small clear glass-like tubes (0.3 cm in diameter) containing 20 µL of indocyanine green (ICG, Sigma-Aldrich, St. Louis, MO, USA, at a concentration of 1.3 µM) were implanted into the phantom to be used as fluorescent targets.

Figure 7 shows the physical phantom experimental reconstruction results using the different reconstruction methods. Consistent with the numerical simulation results, the proposed MAP-PGAN method performs well in terms of image contrast, target recovery and reconstruction similarity, and achieves the best reconstruction results. Similarly, we also quantitatively evaluate the effect of different projections on different reconstruction methods and plot the bar graphs shown in Fig. 8. Consistent with the qualitative analysis results, the MAP-PGAN method obtains the largest CNR, the smallest PE, and the largest IOU coefficient, indicating that the MAP-PGAN method performs the best among all reconstruction methods, and as the number of projections increases, the PE1 and PE2 values obtained by the proposed method decrease, and the CNR and IOU values increase, indicating that the proposed method can robustly and accurately achieve the limited-projection reconstruction of FMT.

Fig. 7.

The physical phantom experimental reconstruction results of different algorithms using different projections. The 2D and 3D displays of the L2-regularized ((a1)-(a6)), L1-regularized ((b1)-(b6)), L0-regularized ((c1)-(c6)), 3D-En-Decoder ((d1)-(d6)), UHR-DeepFMT ((e1)-(e6)) and MAP-PGAN ((f1)-(f6)) methods. The true positions of the two targets are indicated by white circles, the phantom boundaries are indicated by red outer circles, and the height of the cross-section is 0.75 cm.

Fig. 8.

Quantitative analysis of different reconstruction methods under different projections in the physical phantom experiments.

Additionally, the reconstruction times of different methods are calculated to evaluate the reconstruction efficiency of MAP-PGAN method, as shown in Table 3. For FMT reconstruction, both iteration-based conventional methods and learning-based deep learning methods require much preparation time. However, FMT reconstruction using a well-trained network is faster than the iterative regularization-based methods and is more suitable for 3D visualization of biological tissues. This study uses only the test time of the deep learning methods and iterative computation time of the traditional regularization methods for comparison. The Mean ± SD of the reconstruction time of 3D-En-Decoder (0.187 ± 0.0153s), UHR-DeepFMT(0.20 ± 0.01s), and MAP-PGAN network (0.21 ± 0.0153s) is close, significantly shorter than L2-regularized (20.33 ± 0.58s), L1-regularized (257.33 ± 1.53s) and L0-regularized (437 ± 12.5s) methods, showing the advantages of deep learning methods in fast FMT reconstruction.

Table 3. Quantification analysis of the physical phantom experiments.

Methods	3-projection	6-projection	10-projection	Mean ± SD (s)
L2-regularized	20	21	20	20.33 ± 0.58
L1-regularized	256	259	257	257.33 ± 1.53
L0-regularized	433	427	451	437 ± 12.5
3D-En-Decoder	0.17	0.19	0.20	0.187 ± 0.0153
UHR-DeepFMT	0.19	0.20	0.21	0.20 ± 0.01
MAP-PGAN	0.18	0.19	0.18	0.1833 ± 0.0058

3.3. In vivo experiments and results

To ensure the feasibility of our proposed MAP-PGAN method in practical applications, we further performed in vivo experiments with an 8-week-old female mouse. We used two transparent glass tubes (0.4 cm inner diameter and 0.6 cm height) filled with ICG diluted in DMSO as fluorescent targets. The two tubes were then fixed together with an EED of 0.6 cm and surgically buried into the mouse's abdominal cavity. The absorption coefficient of the mouse abdomen in vivo data was set as 0.3 cm^-1, and the reduced scattering coefficient was set as 10 cm^-1. The acquisition of in vivo experimental data mainly includes the collection of fluorescence and CT projections. The fluorescence projections were acquired to reconstruct the 3D distribution of the fluorescent targets, and the CT projections were obtained to reconstruct the anatomical structure of the targets. Before the in vivo mouse experiments, the mouse was first anesthetized by intraperitoneal injection of 2% sodium pentobarbital solution at a dose of 0.3 ml/100 g body weight. The mouse was then placed in a supine position and fixed on a mouse plate. The abdominal skin was wiped with iodine and alcohol for routine disinfection. The abdomen of the mice was cut along the mid-abdominal line using surgical scissors, and the target tubes were placed inside the abdomen of the mice. After inoculation, the abdominal incision was closed with surgical sutures, and gentamycin eye ointment was applied to the suture to prevent infection of the incision.

Similarly, we compared five reconstruction algorithms to prove the superiority of the proposed MAP-PGAN method in the limited-projection reconstruction of FMT. In vivo reconstruction results are shown in Fig. 9. The L2-regularized method obtains over-smoothed results and fails to reconstruct the correct positions of the two fluorescent targets. The L1-regularized and L0-regularized methods are severely disturbed by noise and cannot reconstruct two fluorescent targets as well. Compared with the iteration-based methods, the deep learning-based methods are clearly less affected by noise. The 3D-En-Decoder method can reconstruct only one fluorescent target, although it obtains higher signal-to-noise ratio. In contrast, the UHR-DeepFMT and MAP-PGAN methods can reconstruct the two fluorescent targets, but the MAP-PGAN method has higher localization accuracy and reconstruction similarity compared to the UHR-DeepFMT method. Therefore, it can be concluded that the MAP-PGAN method performs the best in the limited-projection FMT reconstruction.

Fig. 9.

The reconstruction results of in vivo mouse experiments using three projections. The 2D and 3D diaplays of the L2-regularized ((a1) - (a2)), L1-regularized ((b1) - (b2)), L0-regularized ((c1) - (c2)), 3D-En-Decoder ((d1) - (d2)), UHR-DeepFMT ((e1) - (e2)) and MAP-PGAN ((f1) - (f2)) methods. The true positions of the two targets are indicated by white circles, the phantom boundaries are indicated by red outer circles.

4. Discussion and conclusions

As a promising optical molecular imaging technique, FMT has played a significant role in the 3D visualization of in vivo imaging. However, the image quality is relatively poor due to the severe ill-posedness of the FMT inverse problem. Especially when limited projections are used, the quality of the reconstructed images is even worse, which limits the practical application of FMT, especially for in vivo fast dynamic imaging. To address these challenges, in this work, we propose a novel MAP-PGAN reconstruction method based on multi-branch attention prior mechanism to achieve the limited-projection reconstruction of FMT. Firstly, the proposed MAP-PGAN significantly improves the reconstruction efficiency and achieves excellent reconstruction performance in terms of localization accuracy and morphological reconstruction. Secondly, the proposed MAP-PGAN method helps to simplify the FMT hardware system, enabling fast and accurate FMT reconstruction by collecting only three projections.

The reconstruction results from numerical simulations and phantom experiments show that the proposed MAP-PGAN method can reconstruct the 3D distribution of the fluorescent targets well using only three projections (Fig. 5 and Fig. 7). Quantitative analysis (Table 2, Fig. 4, and Fig. 8) also shows that the proposed MAP-PGAN method achieves the highest localization accuracy, image contrast, and the most similar target shape recovery compared to the classical iteration-based L2-regularized, L1-regularized, L0-regularized methods and the state-of-the-art learning-based 3D-En-Decoder and UHR-DeepFMT methods. These results show that the MAP-PGAN method has great potential to improve the reconstruction performance of limited-projection FMT.

To our knowledge, this is the first study to apply the deep learning approach to achieve the accurate and fast limited-projection reconstruction of FMT. This makes FMT a more powerful molecular imaging tool in the field of rapid and dynamic studies of tumors in small animals. Our in vivo experiments (Fig. 9) fully demonstrate the feasibility of the MAP-PGAN method for the 3D visualization of biological processes. Furthermore, for a well-trained MAP-PGAN model, it takes only ∼0.18 s to achieve the limited-projection reconstruction of FMT, providing a new limited-projection method for FMT to visualize this difference in vivo with reliable localization accuracy, allowing further insights into biochemical events at the cellular and molecular levels in mouse models.

In conclusion, we propose a MAP-PGAN method for limited-projection FMT. The proposed method achieves more accurate reconstruction results in terms of localization, spatial resolution, and morphological recovery of fluorescent probe distribution compared with conventional methods and state-of-the-art deep learning methods. We believe that this method pushes limited-projection FMT to a new level in the pursuit of higher accuracy and brings great promise for the application of FMT in the visualization of molecular dynamics in vivo in mouse models.

Funding

National Key Research and Development Program of China10.13039/501100012166 (2017YFA0700401); National Natural Science Foundation of China10.13039/501100001809 (61871022); Beijing Municipal Natural Science Foundation10.13039/501100005089 (7202102); 111 Project10.13039/501100013314 (B13003); Fundamental Research Funds for the Central Universities10.13039/501100012226; Academic Excellence Foundation of BUAA for PHD Students10.13039/501100012240.

Disclosures

The authors declare that there are no conflicts of interest related to this article.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Supplemental document

See Supplement 1 ^{(1.4MB, pdf)} for supporting content.