Accelerating vasculature imaging in tumor using mesoscopic fluorescence molecular tomography via a hybrid reconstruction strategy

Abstract

Mesoscopic fluorescent molecular tomography (MFMT) enables to image fluorescent molecular probes beyond the typical depth limits of microscopic imaging and with enhanced resolution compared to macroscopic imaging. However, MFMT is a scattering-based inverse problem that is an ill-posed inverse problem and hence, requires relative complex iterative solvers coupled with regularization strategies. Inspired by the potential of deep learning in performing image formation tasks from raw measurements, this work proposes a hybrid approach to solve the MFMT inverse problem. This methodology combines a convolutional symmetric network and a conventional iterative algorithm to accelerate the reconstruction procedure. By the proposed deep neural network, the principal components of the sensitivity matrix are extracted and the accompanying noise in measurements is suppressed, which helps to accelerate the reconstruction and improve the accuracy of results. We apply the proposed method to reconstruct in silico and vascular tree models. The results demonstrate that reconstruction accuracy and speed are highly improved due to the reduction of redundant entries of the sensitivity matrix and noise suppression.

1. Introduction

The proliferation and metastasis of cancerous tissue highly depend on the generation of neovascular networks. Many studies have proved that angiogenesis plays a pivotal role in the occurrence, invasion, and metastasis of cancer [1]. To detect the abnormality of vascular networks early many studies have shifted from traditional structural imaging to molecular scale imaging [2]. By injecting specific molecules (fluorescent dyes) into the tissues, utilizing the different absorption capability of the diseased tissues and healthy tissues for the fluorescent dyes to detect biological tissues at the molecular level and obtain internal information [3]. Among multiple forms of fluorescence molecular tomography (FMT), mesoscopic FMT (MFMT) can be viewed as a special kind of FMT with millimeter-depth differentiation and high resolution, which fills the vacuum zone of microscopic and macroscopic imaging spatial resolution. Comparing with FMT, MFMT can reach achieve 100 μm–200 μm resolution at depths of 3–5 mm, and retrieve the concentration and distribution of fluorophores along the vessel inside the tissue from the weak two-dimensional fluorescence intensity measurements collected from the surface of the tissue [4]. However, due to insufficient measurements compared to the unknowns and complex propagation process of photons such as absorption, scattering, and reflection of light in tissue, MFMT reconstruction is an intrinsically ill-posed inverse problem.

As a promising data-driven approach, deep learning [5] has verified its advantages in terms of solving the ill-posed inverse problem in medical fields such as computer tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), diffuse optical tomography (DOT). Sinha et al. [6] employed deep neural network (NN) (U-net) to recover phase objects. Wu et al. [7] employed artificial NN to quickly obtain a fluorescence lifetime image. Yoo et al. [8] designed a deep convolutional neural network to invert the Lippman-Schwinger integral equation to solve the inverse problem for DOT. Zhang et al. [9] developed an end-to-end three-dimensional deep encoder-decoder network to directly establish a nonlinear mapping relationship between the distribution of internal fluorescent sources and boundary fluorescent signals. However, the major drawback of applying deep NN as shown above is it obviously covered the clear physical relationship between measurements and reconstructions, thus, turned the reconstruction procedure into a purely data-driven process, which is widely known to be possibly misled by the training datasets [10], especially when the datasets are flawed or lacked any appropriate physical or mathematical explanations of the trained network.

As a result, compared with other optical imaging technologies, there is little study on the emerging technology MFMT, and the application of deep learning to this background is even rarer. Therefore, inspired by the above-mentioned various optical imaging algorithms, in view of the problems of enlargement of solution space, lengthy iteration time, and lower reconstruction accuracy caused by the large-scale sensitivity matrix generated in the reconstruction process of multi-excitation point MFMT, we proposed a hybrid reconstruction strategy combining deep convolutional symmetric network and simultaneous algebraic reconstruction technique (SART) [11], and is verified that the hybrid strategy has better processing effects and faster reconstruction speed than the conventional method by a series of computer simulation experiments to retrieve the three-dimensional distribution of the vascular system.

2. Materials and methods

2.1. Imaging platform

The proposed reconstruction strategy is performed on our 2nd generation MFMT system described in detail in the previous works [12]. Herein, we elaborate the influence of the significant configurations and parameters of the optical system related to the inverse problems. Briefly, the system is a multi-wavelength MFMT platform which is powered by an electron-multiplying CCD camera (EMCCD) (iXonEM + DU-897 backlight, Andor Technology) acquiring data in a descanned mode.

2.2. Forward model and inverse problem

In our MFMT application both dense sampling and fine discretization are employed to acquire more measurements to achieve mesoscopic spatial resolution. However, it is challenging for some traditional methods to accurately imitate the photon propagation in tissue with a high scattering property of light. The diffusion equation is not accurate in the mesoscopic regime as it cast the problem as an isotropic scattering one. Moreover, at short source detector separation, one may collect photons with only a very few scattering events [13]. Consequently, we apply the Monte Carlo (MC) simulation, a golden standard based on stochastic method, to construct a forward model for photon propagation in biological tissue by combining the optical properties of the tumor [14]. Given the light source is at r_s and the detector is at r_d, the photon weight [15] of the excitation light detected on the detector is expressed as equation (1):

$$W\left(r_s,r_d,r\right)=G^x\left(r_s,r\right)\times G^m\left(r,r_d\right)$$ (1)

where G^x and G^m are the Green’s functions of excitation and emission, respectively [16]. r_s, r_d, and r are locations of source, detector, and the voxel in the image space, respectively. The measurements Y of fluorescence intensity for the corresponding source-detector pair U(r_s, r_d) can be formulated as the integral equation (2) [17]:

$$U\left(r_s,r_d\right)=\underset{\Omega }{\int }W\left(r_s,r_d,r\right)\eta (r)dr$$ (2)

where η(r) is the 3D distribution of the fluorophore effective quantum yield. With m source-detector pairs and n discretized voxels of the Region of Interest (ROI), the ill-posed inverse problem for MFMT then can be cast as the following equation (3):

$$\mathit{A}\mathit{X}+\mathit{v}=\mathit{Y}$$ (3)

where A∈R^m×n is the sensitivity matrix, X∈R^n×1 represents the effective quantum yield, v is the ever-present noise, and Y∈R^m×1 is the detector readings at all scanning spots. Because of the ill-posedness of the inverse problem in MFMT, we set up a least square optimization problem with the weighted l₁-norm [18] minimizing an objective function given by equation (4):

$$\underset{x}{\arg \min }\left\{\frac{1}{2}\Vert \mathbf{A}\mathbf{X}-\mathbf{Y}{\Vert }_2^2+\lambda \Vert \mathbf{\alpha }\mathbf{X}{\Vert }_1\right\}$$ (4)

where α is a weight, which is continuously updated with the number of iterations in the subsequent reconstruction algorithm. λ is a spatially variant regularization parameter to compensate for the spatial dependence of the contrast and resolution in the reconstruction [19], and is set to 1.9 according to our experiments.

The objective function (4) is a nondifferentiable convex function resulting in it being difficult to solve analytically. Herein, we employ SART to quickly approximate the sparse solution:

$${\mathit{X}}^{(k+1)}={\mathit{X}}^{(k)}+\lambda \frac{1}{a_j}\sum _{i=1}^{r_i}\frac{a_{ij}}{a_i}\left(y_i-y_i\left({\mathit{X}}^{(k)}\right)\right)$$ (5)

where $a_i=\sum _{i=1}^{r_i}{a}_{ij}$, $a_j=\sum _{j=1}^n{a}_{ij}$, and a_ij are the entries of the condensed sensitivity matrix A and y_i is the principal components of the measurements Y. k is the iteration number. With the help of the simultaneous iterative algorithm, each row of the augmented matrix can simultaneously be applied to the iterative procedure, which permits the achievement of higher computational efficiency. Then, the 3D structure and spatial distribution of the vessel inside the tumor can be retrieved by the solution of equation (5).

2.3. Data preprocessing

To generate computer simulation measurements, we copied the imaging system configuration used in the actual experiment: 441 illumination scan locations and 48 detectors per source location produce a Jacobian determinant (sensitivity matrix A) with a size of 21168 by 6615 elements, the light intensity measurements Y is 21168 × 1. As a result of the large scale of experimental data due to intensive sampling, directly reading will lead to computer memory overflow. Therefore, in the application, a targeted preprocessing plan was developed on the premise of choosing to compress large-scale data to facilitate network reading data.

Data preprocessing is a critical step for implementation of training. However, the preprocessing method adopted highly depends on specific scenarios. In our application, we take four steps to produce the training datasets and testing datasets, as shown in Fig. S1. First, we combine the sensitivity matrix A and the measurements Y into an augmented matrix ${\overline{\mathit{A}}}_{21168\times 6616}$ and add 8 different levels of Gaussian noise with SNR [13] ranging from 2 to 30 into the augmented matrix. Each group of the synthesized noisy matrix is then transposed and divided into 5 subgroups with size of 1324 × 21168. Note that 4 rows are padded in the last subgroup to form the same size as previous subgroups. We also pad two zero matrices with size of 1324 × 14504 besides each subgroup to form a matrix of 1324 × 50176 in third step to ensure that the input data size is convenient for multiple convolution operations. In the final stage the subgroups of datasets are normalized and reshaped into 52960 training datasets and 6616 testing datasets with size of 224 × 224 × 1 for training and prediction.

2.4. Deep convolutional symmetric network

Some deep neural network architectural choices for biomedical imaging tasks have offered considerable insight as to how these deep learning tools can accelerate the solution of ill-posed inverse problem [20]. Herein, a scenario-specific deep neural network is engineered to map the relationship between the input data and its low-dimensional hidden variables, which belongs to unsupervised learning. The entire network comprises two parts: an encoder and a decoder. In the encoder, multi-layer convolution is used to extract the features of the input signal and capture the abstract information in the original data, thereby obtaining low-dimensional hidden variables. The decoder, a mirror image relationship, uses multi-layer deconvolution to restore data details while retaining abstract information and suppressing noise. At the same time, by imposing certain constraints on intermediate signal, the network can learn more meaningful hidden variables. To avoid the loss of information details caused by multiple pooling layers in the stage of restoring the underlying information, the designed network only performs a pooling operation. In addition, skip connection is introduced to prevent the gradient from vanishing and expedite the back-propagation of the gradient in the training phase.

As shown in Fig. S2, the encoder and decoder contain 6 convolutional blocks and 6 deconvolutional blocks, respectively. Each block comprises different numbers of convolutional layers with activation function ReLU. Note that the operation of the deconvolutional blocks is like that of the convolutional block which are painted in different colors representing different operations. Eventually, a single matrix is output through the convolutional layer with a kernel size of 1 × 1 connected with a nonlinear activation function, which is the weighted sum of its input channels. On the premise of keeping the size of the feature map constant (that is, without loss of resolution), nonlinear features are greatly increased. In our application, the network training epoch and batch size are set to 5 and 2, respectively.

Mean square error is used as the loss function to keep the predictive value close to the true one. Meanwhile, the standard stochastic gradient descent with momentum is employed to optimize the backpropagation process, where initial learning rate, momentum, and weight decay are preset to 0.01, 0.9, and 0.0005, respectively. The weight matrix w and the bias b are updated in each training process until it stops when the convergence or the preset iteration number is reached. Then, the testing datasets are predicted using the encoder, in which w and b were finally determined. The abstract representations of the augmented sensitivity matrix in the low-dimensional space are calculated by the following equation (6):

$$Outpu{t}_{r_i\times r_i}=f\left(\mathit{w}Inpu{t}_{224\times 224}+\mathit{b}\right)$$ (6)

It gives a low-dimensional representation for the invertible linear matrix equation as equation (7):

$${\mathit{A}}_{r_i{}^2}\mathit{X}+\mathit{v}={\mathit{Y}}_{r_i{}^2}$$ (7)

where Xrepresents the 3D distribution of the fluorophore to be reconstructed. ${\mathit{A}}_{r_i{}^2}$ and ${\mathit{Y}}_{r_i{}^2}$ are low-dimensional spatial representations of the sensitivity matrix and measurements with zero padding to achieve the same size as before. v is the ever-present noise. Compared to the original 224²-dimensional data, the reduced r_i²-dimensional data is greatly condensed but retains the principal components of sensitivity matrix and measurements.

The training model is set up in the Tensorflow-gpu environment. We perform the proposed network architecture and solving algorithm on an NVIDIA GeForce GTX 1080 Ti GPU.

2.5. Numerical phantom experiment settings and synthetic vascular

In the numerical phantom experiment, a vascular bed model mimicking a vein region in tumor is designed to evaluate the performance of the proposed method. The vessel tree with a trunk and branches was simulated, in which the diameter of the main trunk and branches are 400 μm and 200 μm, and the separation between two adjacent shoots is set to 100 μm as shown in Fig. S3 A. The imaging volume containing the imitation model has a size of 2.1 mm × 2.1 mm × 1.5 mm and is divided into 21 × 21 × 15 voxels with a discretization of 100 μm. The fluorophore concentration is assumed to be uniform in the vessel with effective quantum yield set to 1. The optical properties of domain are assumed to be homogeneous at the excitation wavelength with absorption coefficient μ_a, scattering coefficient μ_s’, index of refraction n, and anisotropy factor g equal to 0.02 mm⁻¹, 1 mm⁻¹, 1.34, and 0.81, respectively. These values are derived from the collagen scaffold typically employed in our bio-printing application at 6–9 mg/ml density [4,21].

To mimic the real applications more closely, two vascular beds with different 3D structure and different depth distribution as shown in Fig. S3 C and Fig. S3 D, named as T1 and T2, are simulated based on the segmentation of the VascuSynth sample dataset depicted in Fig. S3 B [22]. The dataset is typically to produce 3D volumetric images with oxygen demand maps and a few physical parameters about the bifurcation locations, branch radii and tree hierarchy.

2.6. Quantitative metrics to evaluate reconstruction results

We employ the same metrics as our previous works to quantitatively evaluate the performance of the hybrid reconstruction strategy such as nSSD, nSAD, nR and nRMSE. Herein nSSD, nSAD, and nR have the same trend with a higher value approaching 1 indicating a higher similarity between the reconstructed vessel and the ground truth [14]. As for nRMSE, a value close to zero means a high-quality reconstruction as defined in equation (8):

$$\text{nRMSE}=\frac{{\Vert C_{rec}-C_{org}\Vert }_2}{{\Vert C_{org}\Vert }_2}$$ (8)

where C_rec is the reconstructed fluorescence vessel concentration, C_org is the original fluorescence distribution in tumor.

3. Results

3.1. Effect of different number of principal components of sensitivity matrix on reconstruction

The proposed network contains different levels of extractors, which can extract different principal components of the sensitivity matrix via different hidden layers. We explore the effect that the number of principal components extracted from the sensitivity matrix by different convolutional layers of the encoder have on the reconstruction quality. Fig. 1 A–E show the reconstructions from the reduced sensitivity matrix with SNR of 30 after 400 iterations. As can be seen from these 3D reconstructions, the sensitivity matrix with 6400 top principal components extracted by the proposed network can achieve higher reconstruction performance. Fig. 1 F plots the changes of five metrics with different size sensitivity matrices such as 12544, 10000, 6400, 4096 and 3136. As displayed in the figure, nSSD, nSAD, nR, and nRMSE achieve best scores and the reconstruction time significantly become shorter when the size of the sensitivity matrix is compressed to 6400 by 6615. Fig. 1 G further reveals a gradual increase of the value of metrics before the plateau based on the optimal sensitivity matrix size of 6400 by 6615 when the number of iterations equal to 400.

Fig. 1.

Visual reconstruction results and the plots of the metrics. (A–E) The 3D visual reconstruction results with the size of sensitivity matrix are 12544 × 6615, 10000 × 6615, 6400 × 6615, 4096 × 6615, and 3136 × 6615, respectively. (F) The metrics with different size of sensitivity matrix. (G) The metrics with different iteration numbers when the size of the sensitivity matrix is 6400 × 6615. (H–L) The reconstruction results by applying different methods.

3.2. Comparative experiments with different reconstruction methods

To verify that the designed deep convolutional network has more advantages than commonly used dimension reduction techniques, and can effectively accelerate the reconstruction of complex fluorescent vascular structures. Simultaneously, to prove that the hybrid strategy has better processing effect and faster reconstruction speed than the conventional method. Firstly, two traditional iterative algorithms, conjugate gradient (CG) and SART, are applied to the vascular reconstruction. To reflect the effective comparison of the experiments, the number of iterations is set to 400. As shown in Fig. 1 H and Fig. 1 I, reconstruction performance of SART is better than CG though they are both not effective enough to resolve the fine structure of the model. Then we compare the proposed network with two conventional methods used to dimensionality reduction, PCA and AE. To make a fair comparison, these methods are employed to compress the sensitivity matrix into a smaller size, such as 6400 × 6615 in our application. Based on the three reduced sensitivity matrix the SART reconstruction algorithm is utilized to reconstruct the vascular tree. The visual reconstruction results of the hybrid strategy have relatively higher fidelity, minimal noise interference, and more similarity to ground truth, as shown in Fig. 1 J–L. Compared with its counterparts, the proposed method can retrieve the fine structure and spatial distribution of vascular tree. The quantitative results in Table 1 equally confirm the advantages of the proposed method in terms of the accuracy and reconstruction consuming time. Other methods will undoubtedly increase the number of iterations to obtain better reconstruction results, which also means an increase in reconstruction time. Therefore, under the condition that the control variable is 400 iterations, Ours + SART accelerates the reconstruction of the fluorescent vascular structures within the acceptable reconstruction time range.

Table 1.

Quantification results of five methods for in silico model reconstruction.

Methods	Size of sensitivity matrix	nSSD	nSAD	nR	nRMSE	Reconstruction time (s)
CG	21168 × 6615	0.202	0.082	0.196	0.213	251.475
SART	21168 × 6615	0.216	0.139	0.392	0.120	353.896
PCA + SART	6400 × 6615	0.594	0.338	0.709	0.105	566.901
AE + SART	6400 × 6615	0.640	0.538	0.840	0.089	489.736
Ours + SART	6400 × 6615	0.941	0.803	0.978	0.033	507.461

3.3. Comparison of reconstruction results under different noise levels

MFMT reconstruction for tumor vasculature, as an ill-posed system of equations inherently, is susceptible to noise. Therefore, the proposed network is engineered to suppress the noise of measurements in addition to compressing the sensitivity matrix. To verify the noise suppression performance of the proposed network, comparative experiments are performed on the measurements with different levels of noise. As shown in Fig. 2 A, the measurements with a higher SNR contribute to achieve high quality reconstructions for all the methods. Meanwhile, the proposed network demonstrates a higher performance than its counterparts in terms of noise suppression and reconstruction quality. Furthermore, the reconstruction time at different noise levels reveals the proposed method is robust to noise as shown in Fig. 2 B.

Fig. 2.

Comparison of reconstruction metrics at different noise levels by the different methods.

3.4. Synthetic vascular network experiments

Fig. 3 displays the reconstruction results of VascuSynth sample by different methods. As illustrated by Fig. 3 B and G, the reconstruction results without any dimensionality reduction strategy using the SART only retrieve the rough contour of the vascular beds though the reconstruction results are better than those from CG (Fig. 3 A and F). Meanwhile, the results of the PCA shown in Fig. 3 C and H can reconstruction the larger branches of the vascular beds whereas AE provides more accurate reconstructions but still fails to recover more complex distributions, as shown in Fig. 3 D and I. Fortunately, the proposed method provides more accurate rendering of the simulated vasculatures. As illustrated in Fig. 3 E and J, in both cases the structures of the vascular beds are retrieved with higher fidelity at all depths with a resolution of 100 μm. Fig. 3 K and L shows the comparison of the metrics scores for T1 and T2 vasculatures under five reconstruction strategies.

Fig. 3.

Reconstruction results and the stacked bar chart of the metrics scores for two complex vasculatures by five methods.

4. Discussion

Herein, we propose a hybrid method combining the engineered deep neural network and SART to image vasculature in tumor via mesoscopic fluorescence molecular tomography, which provides a tool to remodel and visualize the vasculature and would enable a new option in tumor therapy. This approach is featured by the following advantages. Firstly, the proposed optical imaging modality has the potential to visualize the distribution of vascular network in tumor with the mesoscopic resolution (100–200 μm). To expedite the speed of vessel reconstruction in tumor for clinical application a well-designed deep neural network is employed to extract the principal components from the redundant sensitivity matrix and suppress the ever-present noise in measurements as well. Moreover, in all cases investigated, both for numerical phantom and vascular tree experiments, the reconstruction based on the sensitivity matrix reduction strategy outperforms their counterparts in terms of retrieving the structure of the inclusions with high fidelity and recovering quantitatively the distribution of the effective quantum yield. The proposed approach demonstrates a higher resolution, more accuracy, and more robustness against noise, which suggests that the combination of deep neural network and traditional reconstruction techniques are suitable for fluorescence tomographic imaging in the mesoscopic regime. However, as one refines the discretization of the imaging space, solving the ill-posed inverse problem becomes more challenging due to the sheer size of the matrices/vectors to be manipulated. In our current implementation, the spatial resolution attainable for vasculature reconstruction is 100 μm. In the future, we plan to perform more experiments and engineer artificial intelligent methods to improve the spatial resolution and reconstruction speed of imaging the vasculature in tumor.