A parallel adaptive finite element simplified spherical harmonics approximation solver for frequency domain fluorescence molecular imaging

Abstract

Fluorescence molecular imaging/tomography may play an important future role in preclinical research and clinical diagnostics. Time- and frequency-domain fluorescence imaging can acquire more measurement information than the continuous wave (CW) counterpart, improving the image quality of fluorescence molecular tomography. Although diffusion approximation (DA) theory has been extensively applied in optical molecular imaging, high-order photon migration models need to be further investigated to match quantitation provided by nuclear imaging. In this paper, a frequency-domain parallel adaptive finite element solver is developed with simplified spherical harmonics (SP_N) approximations. To fully evaluate the performance of the SP_N approximations, a fast time-resolved tetrahedron-based Monte Carlo fluorescence simulator suitable for complex heterogeneous geometries is developed using a convolution strategy to realize the simulation of the fluorescence excitation and emission. The validation results show that high-order SP_N can effectively correct the modeling errors of the diffusion equation, especially when the tissues have high absorption characteristics or when high modulation frequency measurements are used. Furthermore, the parallel adaptive mesh evolution strategy improves the modeling precision and the simulation speed significantly on a realistic digital mouse phantom. This solver is a promising platform for fluorescence molecular tomography using high-order approximations to the radiative transfer equation.

1. Introduction

Imaging of the biological actions of therapeutics in vivo is paramount for the advancing field of molecular medicine (Weissleder 1999, Massoud and Gambhir 2003, Herschman 2003, Cherry 2004). Currently, nuclear imaging techniques of positron emission tomography (PET), single photon emission computed tomography (SPECT), or gamma scintigraphy are the clinical ‘gold-standards’ of molecular imaging. These techniques depend upon the collection of high energy photons that emanate from the relaxation of a radionuclide that is targeted to a specific disease marker in order to generate either a planar image (gamma scintigraphy) or a tomographic image (PET and SPECT). Preclinical small animal studies using these techniques are commonplace in the drug discovery field.

Optical molecular imaging is akin to nuclear imaging in that it also involves the collection of photons, albeit low energy photons that have significant attenuation due to scatter and absorption. In contrast to bioluminescence (Contag and Bachmann 2002, Wang et al 2004, Alexandrakis et al 2005, Chaudhari et al 2005, Cong et al 2005, Lv et al 2006, Dehghani et al 2006, Kuo et al 2007, Lu et al 2009a, Klose et al 2010) or emerging Cherenkov imaging (Cho et al 2009, Robertson et al 2009) approaches, fluorescence imaging has the capability for higher photon count rate for both planar and tomographic imaging as the fluorescent gene product or fluorescent exogenous agent can be repeatedly excited by tissue surface illumination (Sevick-Muraca et al 2002). Although most gene reporters are excited and emit in the visible regime where limited tissue penetration and autofluorescence can confound signals, new fluorescent gene reporters such as IFP1.4 (Shu et al 2009) are excited in the red range (>690 nm) with reduced autofluorescence and greater tissue penetration, opening up interesting possibilities for quantitative tomography for preclinical drug discovery. Exogenous agents with near-infrared (NIR) excitation (>780 nm) have minimal tissue autofluorescence thereby offering the lowest background for fluorescence planar and tomography techniques. Indeed, using indocyanine green (ICG), a dim and poor fluorophore with a low quantum efficiency of 0.016, planar human imaging has been conducted even at microdosages of <100 μg and millisecond data acquisition rates (Sevick-Muraca et al 2008, Rasmussen et al, 2009) to image structures in shallow tissues as well as tissue at depths up to 4–5 cm (Sevick-Muraca et al 2008).

Given the high photon count rates of planar fluorescence imaging for both fluorescent gene products and exogenous agents, tomographic reconstruction for quantification of expression/deposition should be possible given an accurate forward model for propagation of light in tissues. For quantification, the forward model must be suitable for visible and red wavelength-excitable gene products in preclinical studies, as well as NIR excitable exogenous agents in preclinical as well as clinical studies. While significant work has been accomplished using the diffusion approximation to the radiative transport equation (RTE) to model light transport in biological tissues (Gibson et al 2005), the high tissue absorption associated with visible and red-excited wavelengths, the small tissue volumes of preclinical models, or the shallow depths of human imaging studies obviate its use. A direct solution of the RTE is an alternative (Lewis and Warren 1984, Klose et al 2005, Rasmussen et al 2006, Joshi et al 2008), but with an expense of solving N (N + 2) and (N + 1)²-coupled partial differential equations corresponding to discrete ordinates (S_N) and spherical harmonics (P_N) methods, where N depicts the approximation degree. High-order approximations have been recently attempted to obtain sufficient performance for model-based iterative image reconstructions with less memory requirement and time cost compared with direct RTE solvers (Klose and Larsen 2006, Cong et al 2007, Yuan et al 2009, Chu et al 2009). However, in order to realize quantitative tomographic reconstruction in complex heterogeneous geometries, such as a mouse, we need to fully evaluate and validate the performance of those high-order approximations.

In addition, time-dependent measurements, i.e. time- or frequency-domain measurements involving pulsed or modulated illumination of excitation light, have been shown to improve reconstructions of embedded fluorescent tissues by providing more information than continuous wave or time-independent methods (Godavarty et al 2003), but few, if any, studies validate higher order approximation model predictions of both excitation and fluorescence measurements in both time and 3D space. Owing to the complex and curvilinear geometries associated with biological tissues, finite element methods (FEM) become necessary for accurate quantification especially in small volumes where gradients of excitation and emission fluence distribution contribute substantially to discretization errors. The adaptive finite element method uses local mesh evolution strategy not only to accelerate the simulation with fewer discretized points but also to improve upon the discretization errors, forward solution accuracy and therefore reconstruction quality (Joshi et al 2004, Lv et al 2006). When the simulation region is decomposed into several subdomains, parallel simulation implementation can effectively accelerate the adaptive FEM simulation (Gu et al 2009, Lu and Chatziioannou 2009), enabling fully parallel adaptive finite element photon migration simulation with high-order RTE approximations that are suitable for use within inverse image reconstruction algorithms (Lu et al 2009b).

Herein, the forward solution for frequency domain, fluorescent photon migration in biological tissues, is realized with fully parallel adaptive finite element methods. The simplified spherical harmonics (SP_N) approximations are validated using a unique time-domain tetrahedron-based Monte Carlo (MC) fluorescence simulator utilizing the convolution method (Swartling et al 2003) for improvement in computational time. Compared with diffusion approximation, computational validation shows the preferable performance of the SP_N approximations especially in cases of high absorption and high modulation frequency. Simulation performance is then evaluated using a whole-body digital mouse to document improvement arising from parallel adaptive finite element strategies for fluorescence molecular imaging.

In the following section, the proposed simulation framework that incorporates parallel adaptive strategy and SP_N approximations for frequency-domain fluorescence molecular imaging is introduced. The validation and comparative simulation results of the SP_N approximations and MC methods are demonstrated in section 3. The performance and simulation improvements from parallel adaptive mesh evolution are represented in section 4. The conclusion and relevant discussions are provided in section 5. The forward simulation model presented herein represents a platform pertinent to time-dependent and -independent measurements, excitation and emission measurements, as well as emission measurements employing bioluminescence and Cherenkov emission tomography in small animals for drug discovery purposes.

2. Methods

2.1. The simplified spherical harmonics (SP_N) approximations for frequency domain fluorescence molecular imaging

The frequency domain RTE for fluorescence molecular imaging in 3D are (Rasmussen et al 2006)

$$\{\begin{array}{l}\widehat{\mathbf{s}}·\nabla {\psi }^x(\mathbf{r},\widehat{\mathbf{s}},\omega )+\left({\mu }_t^x(\mathbf{r})+\frac{\text{i}\omega }{c_b(\mathbf{r})}\right){\psi }^x(\mathbf{r},\widehat{\mathbf{s}},\omega )={\mu }_s^x(\mathbf{r}){\int }_{4\pi }p(\widehat{\mathbf{s}},{\widehat{\mathbf{s}}}^{\prime }){\psi }^x(\mathbf{r},{\widehat{\mathbf{s}}}^{\prime },\omega )\text{d}\widehat{\mathbf{s}}\hfill \\ \widehat{\mathbf{s}}·\nabla {\psi }^m(\mathbf{r},\widehat{\mathbf{s}},\omega )+\left({\mu }_t^m(\mathbf{r})+\frac{\text{i}\omega }{c_b(\mathbf{r})}\right){\psi }^m(\mathbf{r},\widehat{\mathbf{s}},\omega )={\mu }_s^m(\mathbf{r}){\int }_{4\pi }p(\widehat{\mathbf{s}},{\widehat{\mathbf{s}}}^{\prime }){\psi }^m(\mathbf{r},{\widehat{\mathbf{s}}}^{\prime },\omega )\text{d}{\widehat{\mathbf{s}}}^{\prime }+\frac{Q{\mu }_a^{xf}}{1+\iota \omega \tau }{\psi }^x\hfill \end{array}$$ (1a) (1b)

where the modulation frequency is ω; ψ^x,m are the excitation and emission radiances; c_b(r) is the light speed in the domain Ω; ${\mu }_t^{x,m}={\mu }_s^{x,m}+{\mu }_a^{x,m},{\mu }_s^{x,m}$ and ${\mu }_a^{x,m}$ are the scattering and absorption coefficients, respectively; ${\mu }_a^{xf}$ is the absorption coefficient of the fluorophore; Q and τ are the quantum efficiency and lifetime of the fluorophore, respectively; p (ŝ, ŝ′) is the scattering phase function giving the probability of a photon anisotropically scattering from the direction ŝ′ to direction ŝ. Generally, the Henyey–Greenstein (HG) phase function is often used to characterize this probability (Ishimaru 1997):

$$p(cos\theta )=\frac{1-g^2}{4\pi {(1+g^2-2gcos\theta )}^{3/2}},$$ (2)

where g is the anisotropy parameter; cos θ denotes the scattering angle that is equal to ŝ · ŝ′. The HG phase function is easily expanded by the Legendre polynomial and is therefore convenient for numerical computation. The angle-independent fluence φ^x,m is further defined as

$${\phi }^{x,m}={\int }_{4\pi }{\psi }^{x,m}\text{d}\mathrm{\Omega }.$$ (3)

At the tissue surface, r ∈ ∂Ω, we assume that some photons are internally reflected and do not escape from the domain because of the mismatch between the refractive indices n_b of the domain Ω and n_m of the surrounding external medium. Therefore, the partially reflecting boundary condition for ψ^x,m is described by (Klose and Larsen 2006)

$$\{\begin{array}{l}{\psi }^x(\mathbf{r},\widehat{\mathbf{s}},\omega )=S(\mathbf{r},\widehat{\mathbf{s}},\omega )+R({\widehat{\mathbf{s}}}^{\prime }·\mathbf{v}){\psi }^x(\mathbf{r},{\widehat{\mathbf{s}}}^{\prime },\omega )\hfill \\ {\psi }^m(\mathbf{r},\widehat{\mathbf{s}},\omega )=R({\widehat{\mathbf{s}}}^{\prime }·\mathbf{v}){\psi }^m(\mathbf{r},{\widehat{\mathbf{s}}}^{\prime },\omega ),\hfill \end{array}$$ (4a) (4b)

where v is the unit outer normal vector; the incidence angle θ_b from the volume is not beyond the critical angle θ_c (θ_c = arcsin(n_m/n_b) based on Snell’s law), and where the reflectivity R(cos θ_b) is given by (Haskell et al 1994)

$$R(cos{\theta }_b)=\frac{1}{2}\left[\frac{{sin}^2({\theta }_b-{\theta }_m)}{{sin}^2({\theta }_b+{\theta }_m)}+\frac{{tan}^2({\theta }_b-{\theta }_m)}{{tan}^2({\theta }_b+{\theta }_m)}\right],$$ (5)

where θ_m is the transmission angle. The exiting partial current J⁺(r) at each point r on the volume surface (where the illumination S(r, ŝ, ω) is not considered) is given as (Klose and Larsen 2006)

$$J_{x,m}^{+}(\mathbf{r},\omega )={\int }_{\widehat{\mathbf{s}}·\mathbf{v}>0}[1-R(\widehat{\mathbf{s}}·\mathbf{v})](\widehat{\mathbf{s}}·\mathbf{v}){\psi }^{x,m}(\mathbf{r},\widehat{\mathbf{s}},\omega )\text{d}\widehat{\mathbf{s}}.$$ (6)

After a series of deductions with the P_N method, the excitation SP_N approximations are obtained similar to that described by Klose and Larsen (2006)

$$-\left(\frac{n+1}{2n+1}\right)\nabla ·\frac{1}{{\mu }_{a,n+1}^x}\nabla \left(\left(\frac{n+2}{2n+3}\right){\gamma }_{n+2}^x+\left(\frac{n+1}{2n+3}\right){\gamma }_n^x\right)-\left(\frac{1}{2n+1}\right)\nabla ·\frac{1}{{\mu }_{a,n-1}^x}\nabla \left(\left(\frac{n}{2n-1}\right){\gamma }_n^x+\left(\frac{n-1}{2n-1}\right){\gamma }_{n-2}^x\right)+{\mu }_{a,n}^x{\gamma }_n^x=0,$$ (7)

where ${\mu }_{a,n}^x={\mu }_a^x+{\mu }_s^x(1-g^n)+\iota \omega /c_b$ (Chu et al 2009); when ψ^x is expanded by the P_N approximation, ${\gamma }_n^x$ are the Legendre moments of ψ^x (2 ≤ n ≤ N, N is an odd positive integer). The (N + 1)/2 boundary conditions for equation (7) can be obtained (Klose and Larsen 2006). When we define ${\varphi }_n^x$ as the composite moments of ${\gamma }_n^x$,

$$\begin{array}{l}{\varphi }_1^x={\gamma }_0^x+2{\gamma }_2^x,\\ {\varphi }_2^x=3{\gamma }_2^x+4{\gamma }_4^x,\\ \cdots ,\\ {\varphi }_n^x=(2n-1){\gamma }_{2n-2}^x+(2n){\gamma }_{2n}^x,\\ \cdots ,\\ {\varphi }_{(N+1)/2}^x=N{\gamma }_{N-1}^x;\end{array}$$ (8)

the general equations of the SP_N approximations and its boundary conditions for excitation wavelength can be subsequently written as

$$\{\begin{array}{l}-\nabla ·{\mathcal{C}}_{i,\nabla {\varphi }_i}^x\nabla {\varphi }_i^x+{\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,{\varphi }_j}^x{\varphi }_j^x=0\hfill \\ {\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,\nabla {\varphi }_j}^{b,x}\mathbf{v}·{\varphi }_j^x={\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,{\varphi }_j}^{b,x}{\varphi }_j^x+{\mathcal{C}}_{i,S}^x{S}_{i}i\in [1,(N+1)/2],\hfill \end{array}$$ (9a) (9b)

where ${\mathcal{C}}_{i,\nabla {\varphi }_i}^x,{\mathcal{C}}_{i,{\varphi }_j}^x,{\mathcal{C}}_{i,\nabla {\varphi }_j}^{b,x},{\mathcal{C}}_{i,\nabla {\varphi }_j}^{b,x}$ and ${\mathcal{C}}_{i,S}^x$ can be calculated for SP₁ to SP₇ (Klose and Larsen 2006). The corresponding approximation and boundary conditions for emission wavelength can be similarly defined as

$$\{\begin{array}{l}-\nabla ·{\mathcal{C}}_{i,\nabla {\varphi }_i}^m\nabla {\varphi }_i^m+{\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,{\varphi }_j}^m{\varphi }_j^m=\frac{Q{\mu }_a^{xf}}{1+\iota \omega \tau }{\varphi }_i^x\hfill \\ {\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,\nabla {\varphi }_j}^{b,m}\mathbf{v}·{\varphi }_j^m={\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,{\varphi }_j}^{b,m}{\varphi }_j^{m}i\in [1,(N+1)/2].\hfill \end{array}$$ (10a) (10b)

2.2. The parallel adaptive finite element methods for the SP_N approximations

For the finite element analysis, a general weak formulation for the SP_N approximations can be written as (Rao 1999)

$$\{\begin{array}{l}{\int }_{\mathrm{\Omega }}\left\{{\mathcal{C}}_{i,\nabla {\varphi }_i}^x\nabla {\varphi }_i^x·\nabla \upsilon +{\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,{\varphi }_j}^x{\varphi }_j^x·\upsilon \right\}\text{d}\mathrm{\Omega }-{\int }_{\partial \mathrm{\Omega }}{\mathcal{C}}_{i,\nabla {\varphi }_i}^x{\sum }_{j=1}^{(N+1)/2}{f}_{\mathbf{v}·{\varphi }_i^x}({\varphi }_j^x)·\upsilon \text{d}\partial \mathrm{\Omega }={\int }_{\partial \mathrm{\Omega }}{\mathcal{C}}_{i,\nabla {\varphi }_i}^x{f}_{S_i}(S_i)·\upsilon \text{d}\partial \mathrm{\Omega }\hfill \\ {\int }_{\mathrm{\Omega }}\left\{{\mathcal{C}}_{i,\nabla {\varphi }_i}^m\nabla {\varphi }_i^m·\nabla \upsilon +{\sum }_{j=1}^{(N+1)/2}{\mathcal{C}}_{i,{\varphi }_j}^m{\varphi }_j^m·\upsilon \right\}\text{d}\mathrm{\Omega }-{\int }_{\partial \mathrm{\Omega }}{\mathcal{C}}_{i,\nabla {\varphi }_i}^m{\sum }_{j=1}^{(N+1)/2}{f}_{\mathbf{v}·{\varphi }_i^m}({\varphi }_j^m)·\upsilon \text{d}\partial \mathrm{\Omega }={\int }_{\mathrm{\Omega }}\frac{Q{\mu }_a^{xf}}{1+\iota \omega \tau }{\varphi }_i^x\mathrm{\Omega }.\hfill \end{array}$$ (11a) (11b)

To avoid the processing of $\mathbf{v}·{\varphi }_i^{x,m}$ in boundary integration, we assume that $\mathbf{v}·{\varphi }_i^{x,m}$ are unknown variables in the boundary equations ((9b) and (10b)). We obtain $f_{\mathbf{v}·{\varphi }_i}^{x,m}(·)$ and f_Si (·)by solving a set of first-order equations to obtain the boundary conditions.

After the simulation domain Ω is discretized into the volumetric mesh , the mesh is next partitioned into N_c mesh subdomains (1 ≤ ≤ N_c), where N_c is the number of the utilized CPUs. For the finite element implementation, the space of linear finite elements is introduced on . ${\varphi }_i^{x,m}$ are approximated as

$${\varphi }_i^{l,x,m}(\mathbf{r},\omega )\approx \sum _{p=1}^{N_{\mathcal{P}}}{\varphi }_{i,p}^{l,x,m}(\omega ){\upsilon }_p^l(\mathbf{r}),$$ (12)

where ${\varphi }_{i,p}^{l,x,m}(\omega )$ are the discretized values at a discretized point p at the lth adaptive mesh for excitation and emission wavelengths when using the basis function υ_p (r); is the total number of the discretized points over the whole domain. Upon combining equations (11a),(11b), and (12), one obtains

$$\sum _{e=1}^{N_{{\mathcal{T}}_c}}\left[\begin{array}{cccc}{m}_{1{\varphi }_1}^{l,x,m}& m_{1{\varphi }_2}^{l,x,m}& \cdots & m_{1{\varphi }_{(N+1)/2}}^{l,x,m}\\ m_{2{\varphi }_1}^{l,x,m}& m_{2{\varphi }_2}^{l,x,m}& \cdots & m_{1{\varphi }_{(N+1)/2}}^{l,x,m}\\ ⋮& ⋮& ⋮& ⋮\\ m_{(N+1)/2{\varphi }_1}^{l,x,m}& m_{(N+1)/2{\varphi }_2}^{l,x,m}& \cdots & m_{(N+1)/2{\varphi }_{(N+1)/2}}^{l,x,m}\end{array}\right]\left[\begin{array}{c}{\varphi }_{1,{\tau }_e}^{l,x,m}\\ {\varphi }_{2,{\tau }_e}^{l,x,m}\\ ⋮\\ {\varphi }_{(N+1)/2,{\tau }_e}^{l,x,m}\end{array}\right]=\sum _{e=1}^{N_{{\mathcal{T}}_c}}\left[\begin{array}{c}{b}_{1,{\tau }_e}^{l,x,m}\\ b_{2,{\tau }_e}^{l,x,m}\\ ⋮\\ b_{(N+1)/2,{\tau }_e}^{l,x,m}\end{array}\right],$$ (13)

where is the number of the total discretized elements in subdomain ,

$$m_{i{\varphi }_j}^{l,x,m}=\{\begin{array}{lll}{\int }_{{\tau }_e}\{{\mathcal{C}}_{i,\nabla {\varphi }_i}^{x,m}\nabla {\upsilon }_p^l·\nabla {\upsilon }_q^l+{\mathcal{C}}_{i,{\varphi }_i}^{x,m}{\upsilon }_p^l{\upsilon }_q^l\}\text{d}\mathbf{r}-{\int }_{\partial {\tau }_e}{\mathcal{C}}_{i,\nabla {\varphi }_i}^{x,m}{f}_{\mathbf{v}·{\varphi }_i^{x,m}}({\upsilon }_p^l){\upsilon }_q^l\text{d}\mathbf{r}\hfill & \text{if}\hfill & i=j\hfill \\ {\int }_{{\tau }_e}{\mathcal{C}}_{i,{\varphi }_j}^{x,m}{\upsilon }_p^l{\upsilon }_q^l\text{d}\mathbf{r}-{\int }_{\partial {\tau }_e}{\mathcal{C}}_{i,\nabla {\varphi }_i}^{x,m}{f}_{\mathbf{v}·{\varphi }_i^{x,m}}({\upsilon }_p^l){\upsilon }_q^l\text{d}\mathbf{r}\hfill & \text{if}\hfill & i\ne j\hfill \end{array}$$

and

$$\{\begin{array}{l}{b}_{i,{\varphi }_i}^{l,x}={\int }_{\partial {\tau }_e}{\mathcal{C}}_{i,\nabla {\varphi }_i}^x{f}_{S_i}(S_i){\upsilon }_p^l\text{d}\mathbf{r}\hfill \\ b_{i,{\varphi }_i}^{l,m}={\int }_{{\tau }_e}\frac{Q{\mu }_a^{xf}}{1+\iota \omega \tau }{\varphi }_i^{l,x}{\upsilon }_p^l\text{d}\mathbf{r}\hfill \end{array}.$$

τ_e and ∂τ_e are the basic volumetric and surface elements of the mesh and belong to the respective subdomain in parallel implementation. After assembling all the submatrices on element τ_e, we obtain

$$\left[\begin{array}{cccc}{M}_{1{\varphi }_1}^{{\mathcal{T}}_c,l,x,m}& M_{1{\varphi }_2}^{{\mathcal{T}}_c,l,x,m}& \cdots & M_{1{\varphi }_{(N+1)/2}}^{{\mathcal{T}}_c,l,x,m}\\ M_{2{\varphi }_1}^{{\mathcal{T}}_c,l,x,m}& M_{2{\varphi }_2}^{{\mathcal{T}}_c,l,x,m}& \cdots & M_{2{\varphi }_{(N+1)/2}}^{{\mathcal{T}}_c,l,x,m}\\ ⋮& ⋮& ⋮& ⋮\\ M_{(N+1)/2{\varphi }_1}^{{\mathcal{T}}_c,l,x,m}& M_{(N+1)/2{\varphi }_2}^{{\mathcal{T}}_c,l,x,m}& \cdots & M_{(N+1)/2{\varphi }_{(N+1)/2}}^{{\mathcal{T}}_c,l,x,m}\end{array}\right]\left[\begin{array}{c}{\varphi }_1^{{\mathcal{T}}_c,l,x,m}\\ {\varphi }_2^{{\mathcal{T}}_c,l,x,m}\\ ⋮\\ {\varphi }_{(N+1)/2}^{{\mathcal{T}}_c,l,x,m}\end{array}\right]=\left[\begin{array}{c}{b}_1^{{\mathcal{T}}_c,l,x,m}\\ b_2^{{\mathcal{T}}_c,l,x,m}\\ ⋮\\ b_{(N+1)/2}^{{\mathcal{T}}_c,l,x,m}\end{array}\right].$$ (14)

In contrast to fixed or uniformly refined meshes, adaptively refined meshes require an a posteriori error estimation and the processing of hanging nodes. A posteriori error estimators decide which element should be refined or coarsened based upon the current solution. In parallel mode, the local error indicators are suitable because they reduce communications between processors. In this application, the gradient-jump error indicator is applied (Kelly et al 1983). For the SP_N, the local jump indicator for element τ_e is defined as (Kirk 2007)

$${\eta }_{{\tau }_e}^{x,m}=\sum _{i=1}^N{w}_i{\left(\frac{h_{{\tau }_e}}{24}{\int }_{\partial {\tau }_e}{\mid R_{i,{\tau }_e^{x,m}}\mid }^2\text{d}\mathbf{r}\right)}^{1/2},$$ (15)

where w_i is a weighted factor corresponding to ϕ_i, and R_{i, τe} is the local residual for ϕ_i, which is defined by

$$R_{i,{\tau }_e}^{x,m}=\{\begin{array}{lll}{f}_{\mathbf{v}·{\varphi }_i^x}(S)-{\mathbf{v}}_{{\tau }_e}·\nabla {\varphi }_i^{l,x,m}\hfill & \text{if}\hfill & \partial {\tau }_e\in \partial \mathrm{\Omega }\hfill \\ \frac{1}{2}{\mathbf{v}}_{{\tau }_{\mathbf{e}}}(\nabla {\varphi }_i^{l,x,m}\mid {\tau }_{e_q}-\nabla {\varphi }_i^{l,x,m}\mid {\tau }_{e_q})\hfill & \text{if}\hfill & \partial {\tau }_e\in {\tau }_{e_p}\cap {\tau }_{e_p}\hfill \end{array}.$$ (16)

v_τe is the outward unit normal for element τ_ep and h_τe is the diameter of τ_ep. Using ${\eta }_{{\tau }_e}^{x,m}$, a statistical strategy is used to decide the proportion of refined and coarsened elements (Carey 1997) by computing the mean (m) and standard deviation (σ) of the distribution as an indictor for adaptive refinement or coarsening. The refinement and coarsening ratios (γ_r and γ_c) are set to select elements. By judging whether the log-error of one element is higher than (m + γ_r σ), or lower than (m − γ_cσ), the element is either refined or coarsened. When coarsening an element, only the refined elements are operated upon.

Since the hanging node constraint needs to consider specific elements in geometric method processing, the algebraic constraint method is selected to deal with hanging nodes (Carey 1997). Adaptive mesh evolution brings new challenges for balancing computational burden across partitioned domains. Imbalance deteriorates parallel performance and dynamic repartitioning of the mesh is indispensable for balancing computational burden. A k-way partitioning method is used to perform the partitioning after mesh refinement (Karypis and Kumar 1998). The reader is referred to Lu and Chatziioannou (2009) for details.

2.3. Time-resolved tetrahedron-based Monte Carlo fluorescence simulation

MC methods provide ground truth to evaluate and validate photon migration simulations. The need to simulate illumination patterns that have been demonstrated to improve reconstructions (Joshi et al 2006) and incorporate time dependence (Sevick-Muraca et al 1997, Li et al 1998, Godavarty et al 2004, Patwardhan et al 2005) while providing comparable domains for validating FEM solutions of complex geometries requires new MC solvers. To fully validate the SP_N approximations for our parallel FEM solver, we develop a time-resolved MC fluorescence simulator (TR-TIM-OS) based on a tetrahedron-based inhomogeneous MC optical simulator (TIM-OS) (Shen and Wang 2010).

2.3.1. Arbitrary shaped source settings and photon sampling

Triangular and tetrahedral elements have become popular in numerical computation, especially in numerical computation, because of their ability to describe complex geometries. Tetrahedron-based MC photon simulation facilitates validation of complex geometries and patterned illumination sources that require triangular and tetrahedral elements (Shen and Wang 2010). In fluorescent photon MC, there are two manners to initiate photon trajectories: (i) through external illumination of excitation photons and (ii) through internal generation of emission photons.

On the boundary of a domain, a triangle is the basic element describing the surface. In order to set arbitrary-shaped area illumination E_ai, a group of triangles ( $G_{\partial \tau }^x=\{\partial {\tau }_1^x,\dots ,\partial {\tau }_i^x,\dots ,\partial {\tau }_{N_{\partial \tau }^x}^x\}$) at any position of the surface can be selected. In order to simulate the spatially uniform photon illumination when the area of the selected triangles is calculated, a uniform random number generator can be used to randomly select one from all the triangles depending on their areas. In a selected triangle, the position of the sampled point obeying the uniform random distribution is given by (Turk 1990)

$$p_{\partial {\tau }_i}^{x,rd}=p_{\partial {\tau }_i}^1\ast {rd}_1+p_{\partial {\tau }_i}^2\ast {rd}_2+p_{\partial {\tau }_i}^3\ast {rd}_3,$$ (17)

where rd₁ and rd₂ are the random number obeying a uniform distribution. If the sum of rd₁ and rd₂ is greater than ‘1.0’, two new random numbers for them are regenerated. Conversely, rd₃ = 1 − rd₁ − rd₂. The emitting direction ${\mathbf{v}}_{\partial {\tau }_i}^{rd}({\mathbf{v}}_{\partial {\tau }_i}^{rd}·{\mathbf{v}}_{\partial {\tau }_i}<0)$ of the photon can be uniformly sampled.

Since the volumetric domain is discretized using tetrahedral elements, a group of tetrahedra ( $G_{\tau }^{om}=\{{\tau }_1^{om},\mathrm{..},{\tau }_i^{om},\dots ,{\tau }_{N_{\tau }^{om}}^{om}\}$) can be selected as volumes containing fluorophores of uniform distribution that can be sites for generation of fluorescent photons. Similar to the method for simulating excitation area illumination, any ${\tau }_i^{om}$ can be selected randomly. The position $p_{{\tau }_i}^{m,rd}$ of the generated emission photon can be obtained randomly and quickly using the algorithm in Rocchini and Cignoni (2000). The emitting direction of the emission photon can then be uniformly sampled over the whole solid angle.

2.3.2. Photon tracking

Once the initial excitation or emission photon generation is simulated as described above, trajectories are simulated using conventional MC techniques (Wang et al 1995, Boas et al 2002) where the weight of each photon is set to ‘1.0’ and attenuated depending upon the length of a step; each step length is sampled from

$$s^{x,m}=\frac{-ln{rd}_s}{{\mu }_a^{x,m}+{\mu }_s^{x,m}},$$ (18)

where rd_s is a uniform random number in [0, 1]. The corresponding time traveled is calculated from

$$\mathrm{\Delta }{t}^{x,m}=\frac{s^{x,m}}{c_b}.$$ (19)

When an emission photon is generated, the photon travel time is increased by (− ln(rd_l )τ) due to the fluorescence lifetime, where rd_l is similar to rd_s. At each step, the position, weight and accumulated travel time are recorded. In MC simulation using tetrahedron-based discretized domains, a data structure is designed to record the element adjacent information. When the photon trajectory occurs over two or even several tetrahedra, the photon trajectory can also be quickly localized through the established element adjacent information. Preliminary tests have shown that tetrahedron- and analytic geometry-based MC simulation have comparable simulation time (Shen and Wang 2010), although for validation work, simulation time is not as critical as is for FEM simulation. After setting the time step Δt_MC and the total photon traveling time $T_{\text{MC}}^x$ and $T_{\text{MC}}^{om}$ for excitation and emission procedures, respectively, the absorbed photon weight over time can be recorded before it is terminated.

2.3.3. Convolution-based emission simulation

In the MC simulation employed, simulation of excitation and emission photon distributions are performed separately. After simulating the temporal excitation, the fluence distribution over time is obtained within the whole simulation domain. ${\phi }_{{\tau }_i}^x(t_p),t_p\in [1,N_t^x]$ is used to describe the fluence in the tetrahedral element ${\tau }_i^{om}$. If $G_{\partial \tau }^m=\{\partial {\tau }_1^m,\dots ,\partial {\tau }_j^m,\dots ,\partial {\tau }_{N_{\partial \tau }^m}^m\}$ describes a group of triangles for the surface measurement domain, then to each ${\tau }_i^{om}$ and $\partial {\tau }_j^m$, the time-resolved emission surface fluence ${\phi }_{{\tau }_i^{om}\to \partial {\tau }_j^m}^{om}(t_q),t_q\in [1,N_t^{om}]$ is obtained in the emission simulation. The final emission fluence ${\phi }_{\partial {\tau }_j^m}^m(t_r),t_r\in [1,N_t^x+N_t^{om}-1]$ on the surface triangle $\partial {\tau }_j^m$ can be computed from

$${\phi }_{\partial {\tau }_j^m}^m(t_r)=\sum _{t_p=1}^{N_t^x}{\phi }_{{\tau }_i}^x(t_p){\phi }_{{\tau }_i^{om}\to \partial {\tau }_j^m}^{om}(t_r-t_p).$$ (20)

With ${\phi }_{{\tau }_i}^x(t_p),{\phi }_{{\tau }_i^{om}\to \partial {\tau }_j^m}^{om}(t_q)$ and ${\phi }_{\partial {\tau }_j^m}^m(t_r)$, the corresponding amplitude and phase shift can be determined from the discrete Fourier transform of ${\phi }_{\partial {\tau }_j^m}^m(t_r)$ when the specific modulated frequency ω is selected (Pan et al 2007). The convolution approach dramatically speeds upon time-dependent MC simulation of emission.

3. Simulations

In order to demonstrate the performance of the SP_N approximations, the in-house code was evaluated (SP₁(i.e. the diffusion approximation), SP₃ and SP₇) for performance and compared to MC for validation. Currently, many open-source, high-quality software packages have been developed to meet the need of FEM-based simulation. In our application, libMesh was selected as the basis development environment (Kirk et al 2006). LibMesh provides almost all of the components used in parallel PDE-based simulation with unstructured discretization. We used PETSc developed by Argonne National Laboratory (ANL) to solve the linear systems in parallel mode (Balay et al 2001) and METIS (Karypis and Kumar 1998) to dynamically partition the whole domain in libMesh using the k-way partitioning algorithm. The time-resolved tetrahedron-based MC program, that is TR-TIM-OS, was written in C/C++ (Intel MKL 2009). All the simulations were performed on a cluster of eight nodes (eight CPU cores of 3.0 GHz and 16 GB RAM at each node).

3.1. Validation of SP_N approximations with Monte Carlo simulation

The validation studies were performed using a simulated cylindrical phantom of 12.5 mm radius and 25.0 mm height. The phantom was simulated to contain a single spherical fluorescent heterogeneity and a second non-fluorescent inclusion (figures 1(a) and (b)). The center of the phantom is located at the origin of the Cartesian coordinate. The phantom was discretized into a tetrahedron-based volumetric mesh with approximately 5900 nodes and the average element diameter of 1.2 mm. This mesh was used in the SP_N- and MC-based simulations. The range of the launched photon numbers was from 10⁷ to 10⁸. With the increase of the absorption coefficient, the number of launched photons was increased to reduce simulated Poisson noise. A fluorescent spherical volume with 1.0 mm radius was simulated at the half-radius position (the center: (− 6.25, 0.0, 0.0)) as the single fluorescent domain. An area located at the height of 4.0 mm along the Z-axis subtending a circumferential angle of 11.2° was used as the uniform illumination area with its center along the negative x-axis. The absorption and scattering coefficients of the phantom were set to 0.005 mm⁻¹ and 5.0 mm⁻¹, respectively. The anisotropic factor and refractive index were 0.9 and 1.33, respectively. The absorption coefficient, lifetime and quantum efficient (QE) of the fluorophore were set to 0.0147 mm⁻¹, 1.0 ns, and 0.016 consistent with available near-infrared fluorophores. The goal of fluorescence molecular imaging/tomography is to acquire the quantitative information of the fluorophores in the in vivo animal and human body; the two parameters collected were the excitation photon fluence distribution (φ^x(r), r ∈ Ω) in the body and the outgoing emission photon fluence ( $J_m^{+}(\mathbf{r})$, r ∈ ∂Ω).

Figure 1.

The discretized meshes of the used phantoms. The black circles and lines are for comparisons of the exiting partial current $J_m^{+}$ and the fluence φ^x. The red solid spheres represent fluorescent heterogeneities and the cylindrical inclusion is also represented.

The comparison results of $J_m^{+}$ are shown in figure 2 for a modulation frequency of 100 MHz. The amplitudes obtained from SP_N approximations and MC simulation were normalized with the average values of SP₇ and MC methods, respectively. The absolute phase shift was used for the comparison. In order to demonstrate the difference between SP_N approximations and MC simulation, their amplitude ratios and phase shift errors were calculated using ( $A_{{SP}_N}^{\text{norm}}/A_{\text{MC}}^{\text{norm}}$) and (P_SPN − P_MC), respectively. $A_{{SP}_N}^{\text{norm}}$ and $A_{\text{MC}}^{\text{norm}}$ are the normalized SP_N and MC amplitude; P_SPN and P_MC are the corresponding absolute phase shift. Due to the effect of the Poisson noise of the MC simulation in $J_m^{+}$, the values of the points far from the fluorophore have somewhat larger errors than those close to the fluorophore. However, the amplitude ratios and phase errors between several SP_N approximations and MC are close, showing the accuracy of the in-house codes.

Figure 2.

Emission comparisons between the diffusion equation, Monte Carlo method and SP_N approximations in the phantom (figure 1(a)) where μ_a = 0.005 mm⁻¹, μ_s = 5.0 mm⁻¹, g = 0.9 and n_b = 1.33. The modulation frequency is 100 MHz. ‘A * to MC’ and ‘P * to MC’ denote the normalized amplitude ratios and the errors of the absolute phase shift between SP_N and MC. Two dash-dot lines are 0 degree (top) and the ratio of ‘1’ (bottom).

3.2. Performance of SP_N approximations

3.2.1. With varying ratios of ${\mu }_s^{\prime }$ to μ_a

The performance evaluation of the SP_N approximations in fluorescence molecular imaging was first performed with different ratios of ${\mu }_s^{\prime }$ to μ_a. When the absorption coefficient of the phantom was adjusted to 0.05 mm⁻¹ with other parameters unchanged from those used in the accuracy validation, the comparison of $J_m^{+}$ and φ^x between several SP_N approximations and MC was computed as shown in figures 3(a) and (b), respectively. At the ratio of ${\mu }_s^{\prime }$ to μ_a of 10, the difference of the amplitude ratios and phase errors between SP₁, SP₃ and SP₇ becomes distinct as compared to when the ratio was set to 100 (figure 2). With the increase of the distance from the illumination area, the phase errors of φ^x gradually become large (~1.0°). Note that the simulated amplitudes near the source domain from all SP_N approximations are inaccurate, resulting in the inaccuracy of the normalized amplitude ratios. The distance from the illumination area where this inaccuracy predominates extends about 3.0 mm in this case. When the amplitudes were re-normalized using the average values of φ^x that did not include values within this distance from the source, better accuracy of the SP_N-based amplitudes occurs (figure 3(b)). In these simulations, the errors arise from both model mismatch and discretization. To further confirm the performance of SP_N approximation in the near-source domain, a fine mesh consisting of approximately 46 000 nodes and having an average element length of 0.55 mm was used. Upon setting the ratios of ${\mu }_s^{\prime }$ to μ_a to 10.0 and 2.0 and launching 10⁸ and 3 × 10⁸ photons, respectively, we observe that the errors are reduced as shown in figures 4(a) and (b). Although the phase shift errors are reduced and the amplitude ratio ranges are closer to ‘1.0’, the error and ratio trends are consistent with those using the coarse mesh. Furthermore, the inaccuracy of SP_N approximation in the near-source domain remains.

Figure 3.

Comparisons of $J_m^{+}$ ((a) and (c)) and φ^x ((b) and (d)) between the diffusion equation, Monte Carlo method and SP_N approximations in the homogeneous phantom when μ_s = 5.0 mm⁻¹, g = 0.9 and n_b = 1.33. The ${\mu }_s^{\prime }/{\mu }_a$ ratios of ((a) and (b)) and ((c) and (d)) are 10.0 and 2.0, respectively. The display settings are the same as those in figure 2.

Figure 4.

Comparisons of φ^x between the diffusion equation, Monte Carlo method and SP_N approximations on the fine mesh of the homogeneous phantom when μ_s = 5.0 mm⁻¹, g = 0.9 and n_b = 1.33. The ${\mu }_s^{\prime }/{\mu }_a$ ratios of (a) and (b) are 10.0 and 2.0, respectively. The modulation frequency is 100 MHz. The display settings are the same as those in figure 2.

When the absorption coefficient of the phantom was set to 0.25 mm⁻¹ (and the ratio of ${\mu }_s^{\prime }$ to μ_a is 2.0), the advantages of the high-order SP_N approximations become more distinct as shown in figures 3(c) and (d). In this case, the amplitude and phase shift of $J_m^{+}$ and φ^x, computed from SP₃ and SP₇, contain the maximal amplitude ratio and phase errors that differ from MC by a factor of 0.4 and 2.0°, respectively. The comparable errors between SP₁ and MC are 0.9 and 4.0°. The corresponding differences of φ^x between DA and high-order SP_N approximations are also distinct as shown in figure 3(d). The inaccuracy becomes worse regarding SP_N approximations with the increased distances from the fluorophore and illumination area. Note that near the source, the distance in which φ^x inaccuracy occurred did not change when the absorption coefficient of the phantom was adjusted from 0.05 mm⁻¹ to 0.25 mm⁻¹.

3.2.2. With varying modulation frequencies

Although 100 MHz is the optimal modulation frequency for experimental measurements (Thompson and Sevick-Muraca 2003), higher modulation frequencies of 500 and 1000 MHz were considered to evaluate the performance of high-order SP_N approximations. Figure 5 shows the comparison of SP_N approximations and MC simulations when the absorption and scattering coefficients of the phantom were fixed to 0.05 and 5.0 mm⁻¹, respectively. Similar accuracies of the excitation and emission amplitude at 100 MHz cases were obtained. However, the diffusion equation has inaccurate excitation and emission phase when compared with high-order SP_N approximations. With the increase of the modulation frequency, the diffusion inaccuracy becomes worse. For $J_m^{+}$, the maximal phase errors of SP₃ and SP₇ relative to MC are 2.0 and 4.0° at 500 and 1000 MHz, respectively. However, the counterparts of the diffusion equation reach 6.0 and 12.0°.

Figure 5.

Comparisons of $J_m^{+}$ ((a) and (c)) and φ^x ((b) and (d)) between the diffusion equation, Monte Carlo method and SP_N approximations in the homogeneous phantom when μ_a = 0.05 mm⁻¹, μ_s = 5.0 mm⁻¹, g = 0.9 and n_b = 1.33. The modulation frequencies are 500 MHz for (a) and (b) and 1000 MHz for (c) and (d). The display settings are the same as those in figure 2.

3.2.3. With inclusions of high absorption and void-like optical characteristics

In practice, small animal fluorescence molecular imaging encounters organs of varying optical properties. The performance evaluation in heterogeneous domains is important. A cylindrical phantom with 2.0 mm radius and 20.0 mm height was simulated within the cylindrical homogeneous phantom. Its center was co-located with that of the homogeneous phantom. The discretized volumetric mesh is shown in figure 1(b), consisting of 6162 nodes. The same fluorophore properties with the validated homogeneous phantom case were used in this evaluation. The optical properties of the surrounding continuous volume were set to 0.05 (μ_a) and 5.0 mm⁻¹ (μ_s), respectively. The optical properties of the inclusion were set to 0.25 (μ_a) and 5.0 mm⁻¹ (μ_s), to simulate the high-absorption domain. As shown in figures 6(a) and (b), there is little effect of the high-absorption domain in values of $J_m^{+}$ computed from the SP_N approximations. However, φ^x of several SP_N approximations are inaccurate in the simulated phantom containing the high-absorption region, significantly affecting the quantitative information acquisition of the fluorophore. Note that there is little effect in the phase shift compared with the emission amplitude.

Figure 6.

Comparisons of $J_m^{+}$ ((a) and (c)) and φ_x ((b) and (d)) between the Monte Carlo method and SP_N approximations in phantoms containing high absorption and void-like inclusions. μ_a, μ_s, g, and n_b of the surrounding continuous volume of the phantom are fixed to 0.05 mm⁻¹, 5.0 mm⁻¹, 0.9, and 1.33, respectively. μ_a and μ_s of the small cylindrical inclusion are set to 0.25 mm⁻¹ and 5.0 mm⁻¹ (high absorption), and 0.005 mm⁻¹ and 0.1 mm⁻¹ (void-like), respectively. The modulation frequency is 100.0 MHz. The display settings are the same as those in figure 2.

For the void-like case, the simulated inclusion was set as a low-scattering domain with a scattering coefficient of 0.1 mm⁻¹ and the high ratio of ${\mu }_s^{\prime }$ and μ_a equal to 20.0. The comparison of MC and SP_N approximations of $J_m^{+}$ and φ^x is shown in figures 6(c) and (d). Similar to the high-absorption inclusion, low-scattering inclusions appreciably impact $J_m^{+}$ even though φ^x is inaccurate compared to MC simulation.

4. Evaluation of the parallel adaptive finite element strategy on a simulated mouse

As the final evaluation of the proposed algorithm, the microMRI-based mouse volume (MOBY) (Segars et al 2004) was investigated for accuracy and computational performance. Amira 5.2 (Visage Imaging, Inc., San Diego, CA) was used to obtain the discretized mesh of MOBY. The coarse and fine meshes, consisting of approximately 21 000 and 230 000 discretized points, respectively, were generated for adaptive mesh evolution and the acquisition of the precise solution. Most organs were depicted in the discretized mesh as described along with assigned optical properties in table 1, reproduced from Alexandrakis et al (2005). To represent a fluorescent heterogeneity, a cubic domain with volume of 15.6 mm³ was simulated within the liver. The optical parameters of the fluorophore were the same with those used in the cylindrical phantom. The illumination area was simulated as a triangular element near the x-axis on the mouse surface. In order to observe the performance of the adaptive mesh evolution as a function of SP_N, the solutions were obtained from the SP₃ and SP₇ approximations of the excitation light only. Because emission and excitation solutions are solved on independently adapted meshed and require accurate interpolation schemes (for example see Lee et al (2008)), we focus only on the effect of SP_N approximation order on adaptive meshes in the simulation precision. Future work will address independently adapted meshing.

Table 1.

Optical properties of organs of the MOBY mouse phantom at 700 nm.

	μa (mm−1)	μs (mm−1)	g	n
Heart	0.038	9.05	0.9	1.33
Liver	0.23	6.48	0.9	1.33
Lung	0.13	21.24	0.9	1.33
Stomach	0.0077	13.77	0.9	1.33
Pancreas	0.043	21.09	0.9	1.33
Kidney	0.043	21.09	0.9	1.33
Spleen	0.23	6.48	0.9	1.33
Intestine	0.0078	10.88	0.9	1.33
Bone	0.039	23.40	0.9	1.33
Brain	0.0027	11.80	0.9	1.33
Bladder	0.0027	11.80	0.9	1.33
Muscle	0.0027	11.80	0.9	1.33

The dynamic mesh partitioning and adaptive mesh evolution are shown in figure 7 when ten domains and ten CPUs were used in the parallel adaptive simulation. To guarantee the same number of the final discretized points in SP₃- and SP₇-based simulation, the refinement ratio γ_r was set to 99.9% and 99.91%, respectively, and both of the coarsening ratios γ_c were set to 0.0%. Upon comparing the initial partitioned mesh and the counterparts after the first and third adaptive mesh refinements (figure 7), one finds that the subdomains for CPUs were almost totally different due to the adaptive mesh evolution. If the initial partitioning mesh was used for the entire simulation, the load imbalance would significantly impair the simulation performance. It is noteworthy that because the larger errors exist in the proximity of the illumination area, mesh refinements were performed primarily in the vicinity of the triangular surface element.

Figure 7.

Dynamic mesh partitioning and evolution and the fine mesh partitioning.

After the SP₃-, SP₇- and MC-based simulation results were acquired on the fine mesh of the MOBY, the amplitude and phase shift profiles of φ^x along the X-axis transecting the fluorophore domain were obtained by the interpolation. The relative errors of the normalized amplitude and phase shift were calculated using $\mid A_{{SP}_N}^{\text{norm}}-A_{MC}^{\text{norm}}\mid /A_{MC}^{\text{norm}}$ and |P SP_N − P_MC |/|P_MC | as shown in figure 8. To remove the effect of the inaccuracy of SP₃ and SP₇ in the near-source domain, the φ^x values within the range of 3.0 mm far from the illumination were not considered in the comparisons. Compared to the solution on the fine mesh, the solutions of the excitation amplitude and phase shift of SP₃ and SP₇ close to the illumination are significantly improved due to the adaptive mesh evolution. Note that there are several spikes in the comparisons. We believe the spikes are from the poor discretization and element quality since they are not observed from the solution on the fine mesh.

Figure 8.

Comparisons of φ^x between SP₃, SP₇ and the Monte Carlo method in the digital mouse phantom (MOBY). The modulation frequency is 100 MHz. ‘A SP_3,7 (*) to MC’ and ‘P SP_3,7 (*) to MC’ denote the relative errors of the excitation amplitude and phase shift on different fixed and adaptively refined meshes. The dash-dot rectangles are used to show the part of the significant error reduction due to the adaptive mesh evolution.

To further observe the performance of the proposed algorithm, the computational time as a function of CPUs is shown in figure 9. For the SP₃- and SP₇-based simulations on the adaptive and fixed fine meshes, the parallel implementation significantly improves the simulation speed although the simulation is not always accelerated with the increasing CPU number. The maximal acceleration ratios for fixed fine meshes were 1.27 and 2.31 for SP₃ and SP₇, respectively. The maximal acceleration ratios for adaptively refined meshes were 2.93 and 4.13 for SP₃ and SP₇, respectively. These results show that the acceleration ratio is more positively impacted by adaptive mesh evolution for SP₇ than SP₃. However, higher order approximations take more CPUs (10 and 18 CPUs for SP₃ and SP₇, respectively).

Figure 9.

Time analysis of SP₃- and SP₇-based simulation using the MOBY phantom. (a) The total cost time of the SP₃ and SP₇ approximations. (b) (SP₃) and (c) (SP₇) denote the cost time of the submodules of the parallel adaptive finite element simulation. (d) The time percentage of the submodules in the total cost time for SP₃ and SP₇.

The computational time of several modules that perform specific computational tasks, matrix assembly, mesh communication, mesh partitioning, mesh adaptive, error estimation and solver, were considered to analyze performance bottleneck. Figures 9(b) and (c) illustrate the computational time as a function of number of CPUs for SP₃ and SP₇, respectively. Although the time costs of several modules (matrix assembly, error estimation and solver) are quickly reduced at the beginning, the improvement of the time cost decreases after an optimal CPU number. In contrast, mesh communication becomes worse with the increased number of CPU due to the more frequent communication between CPUs. Furthermore, figure 9(d) shows the percentage of several modules in the total simulation time. Mesh communication and solver are the major time-consuming modules especially with the increase of the approximation degree. Significant improvement can be obtained by the acceleration of solver.

5. Summary and conclusion

Herein, a parallel adaptive finite element simulation algorithm is developed with the radiative-transfer-based model for frequency domain fluorescence molecular imaging. To demonstrate the performance of high-order SP_N approximations, the time-resolved tetrahedron-based Monte Carlo fluorescence simulator is developed, suitable for complex heterogeneous geometries and arbitrary-shaped illumination. The comparison results show that high-order SP_N approximations may obtain more precise simulation in the high-absorption and high-frequency cases compared with the popular diffusion approximation. The strategy of the parallel adaptive mesh evolution further accelerates the simulation and improves the simulation precision especially when higher order SP_N approximations are applied. The proposed simulation algorithm provides a possible solution for fast and quantitative fluorophore reconstruction for fluorescence molecular tomography at the whole-body small animal level.

Although the radiative transfer equation can provide more accurate solutions, the cost may prevent its use within iterative inverse solutions. For accurate quantitation in drug discovery, suitable high-order approximations may be necessary. As a consequence, optimizing performance of higher order approximations become necessary in order to realize reconstruction accuracy. This paper systematically demonstrates accuracy improvements compared with the diffusion equation and developed Monte Carlo methods. The time cost of SP₇ approximation makes the SP₃ approximation an optimal choice for conditions in which small animal imaging is conducted. However, the poor performance of SP_N approximations in near-source and void-like regions makes the development and investigation of more advanced high-order approximations necessary.

Precise and fast simulation can be obtained using a fully parallel adaptive mesh evolution strategy. The performance evaluation shows that the time improvement of higher order SP_N approximations obtained from the proposed algorithm becomes more distinct compared with sequential implementation, providing a powerful computational platform for radiative-transfer-based simulation. However, the distributed memory-based parallel simulation exhibits a performance bottleneck especially when a great number of CPUs are used. With the increase of the approximation degree of radiative-transfer-based model, it is necessary to obtain optical simulation performance by increasing the CPU number. The development of current multi-core techniques makes the hybrid threaded-MPI parallel mode possible, potentially improving the simulation performance for the high-order radiative-transfer-based model. Module solver becomes a time-consuming phase especially for higher order approximation. Note that the acceleration becomes slow when the CPUs reach a certain number. To improve the parallel simulation performance, more advanced preconditioners and iterative techniques need to be investigated.

In conclusion, we have developed a fully parallel adaptive finite element algorithm for fluorescent photon propagation simulation using high-order radiative-transfer-based models. The fully performance evaluations show the advantages and limits of SP_N approximations and the developed parallel simulation framework. Optimal improvements can be further investigated in future research. Currently, using dual-labeled near-infrared fluorescence and PET traces, we are developing methods to assess the quantitative algorithm by comparing it to established PET.

Nomenclature

Ω, ∂Ω

domain and its boundary

r

position vector

ŝ, ŝ′

outgoing and incoming directions of photons

ω

modulation frequency

ψ^x,m

excitation (^x) and emission (^m) radiances

${\mu }_a^{x,m},{\mu }_s^{x,m}$, g

absorption and scattering coefficients, and anisotropy factor of the domain at excitation and emission wavelengths

${\mu }_a^{xf}$

absorption coefficient of the fluorophore

c_b

light speed in the domain Ω

p(·)

scattering phase function

Q,τ

quantum efficiency and lifetime of the fluorophore

cos θ

scattering angle

φ^x,m

excitation and emission fluence

n_b, n_m

refractive indices of the domain Ω and the medium

S

illumination source

R(·)

reflectivity ratio

θ_b, θ_m, θ_c

incidence, transmission and critical angle

$J_{x,m}^{+}$

exiting partial current at excitation and emission wavelengths

γ_n,ϕ_n

Legendre moments of ψ and composite moments of γ_n

${\mathcal{C}}_{i,\nabla {\varphi }_i}^{x,m},{\mathcal{C}}_{i,{\varphi }_j}^{x,m}$

coefficients of SP_N approximations at excitation and emission wavelengths

${\mathcal{C}}_{i,S}^x$

coefficients of illumination source of SP_N approximations

${\mathcal{C}}_{i,\nabla {\varphi }_j}^{b,x,m},{\mathcal{C}}_{i,\nabla {\varphi }_j}^{b,x,m}$

coefficients of boundary conditions at excitation and emission wavelengths

,

volumetric mesh and its subdomain

N_c, ,

number of CPUs, nodes of the mesh, elements in

τ_e,∂τ_e

volumetric and surface elements

${\varphi }_{i,p}^{l,x,m}$

value at node p at the l-th mesh

${\upsilon }_p^l$

basis function in FEM

$m_{i{\varphi }_j}^{l,x,m},b_{i,{\varphi }_i}^{l,x,m}$

submatrix and subvector at τ_e

$M_{i{\varphi }_j}^{{\mathcal{T}}_c,l,x,m},b_i^{{\mathcal{T}}_c,l,x,m}$

submatrix and subvector at

${\eta }_{{\tau }_e}^{x,m}$

error indicator

w_i, $R_{i,{\tau }_e}^{x,m}$

weighted factor and local residual

γ_r, γ_c

refinement and coarsening ratios

$G_{\partial \tau }^x,G_{\partial \tau }^m,G_{\tau }^{om}$

group of triangles for illumination and emission measurements, group of tetrahedra for fluorephore

rd_i

uniformly random number

$p_{\partial {\tau }_i}^{x,rd}$

sampled point on the surface for excitation

s^x,m

step size of photon trajectory

Δt^x,m

traveling time corresponding to s^x,m

$T_{MC}^x,T_{MC}^{om}$

simulation time for excitation and emission wavelengths

Δt_MC

Δt^x,m, time step and travel time of photons

${\phi }_{{\tau }_i}^x(·),{\phi }_{\partial {\tau }_j^m}^m(·)$

time-resolved excitation and emission fluences

${\phi }_{{\tau }_i^{om}\to \partial {\tau }_j^m}^{om}(·)$

time-resolved artificial emission fluence