Comparison of tomographic fluorescence spectral and lifetime multiplexing

Abstract

Multispectral and lifetime imaging in turbid media can be mathematically described in two steps, involving spectral or temporal mixing of the fluorophores and the diffuse light transport in the turbid medium. We show that the order of fluorophore mixing and diffuse propagation is reversed in spectral and lifetime multiplexing, resulting in a fundamental difference in their multiplexing capabilities, regardless of the measurement conditions. Using the resolution matrix to define a quantitative measure for inter-fluorophore cross-talk, we show that lifetime multiplexing, using the asymptotic time domain approach, provides zero cross-talk, while spectral multiplexing can achieve zero cross-talk under special conditions. We also compare the performance of spectral and lifetime multiplexing for tomographic inversion of two overlapping fluorophores in a heterogeneous digital mouse atlas.

The quantitative separation of multiple fluorophores embedded deep in tissue with spectral and lifetime contrast (“multiplexing”) would enable visualization of parallel biological processes in vivo, and the detection of molecular interactions such as enzyme activation [1] and fluorescence resonance energy transfer [2]. Fluorescence lifetime and multispectral methods have commonly been used in microscopy to unmix multiple fluorophores in thin tissue samples using their unique fluorescence lifetime and spectral signatures [3,4]. Both types of contrast have also been successfully applied to distinguish fluorophores of interest from tissue autofluorescence [5,6]. In microscopy, spectral or lifetime unmixing can be performed directly at each pixel using various fitting techniques [3]. However, in the case of thick turbid samples such as biological tissue, the measurement at the surface of the sample is affected by light propagation through the medium. Therefore, tomographic multiplexing in macroscopic samples requires the consideration of diffuse light transport, in addition to spectral or lifetime unmixing.

In this Letter, we recast the forward problems for multispectral fluorescence tomography (MSFT) and tomographic fluorescence lifetime multiplexing (TFLM) in mathematical forms that explicitly show the reverse order of fluorophore mixing and diffuse light propagation steps in the two techniques. Using the model resolution matrix, we show that this reversal implies a distinct cross-talk performance between MSFT and TFLM. While TFLM can provide zero cross-talk solutions, MSFT can achieve zero cross-talk under two general conditions, regardless of the measurement geometry or other experimental considerations. High cross-talk has been shown to lead to errors in localization, identification, and relative quantification of multiple fluorophores [7,8]. Here, we illustrate the distinct performance of MSFT and TFLM using a simulation model consisting of near-infrared (NIR) fluorophores in a heterogeneous digital mouse atlas. We show that for realistic tissue optical properties, TFLM achieves accurate relative quantitation of the fluorophores, whereas MSFT distorts the relative quantitation due to significant cross-talk.

Consider a turbid medium containing N fluorophores with distinct excitation or emission spectra, b_n(λ) and lifetimes ${\tau }_n$, n = 1 … N. The forward problem for MSFT and TFLM takes the following matrix form for V medium voxels, M measurement pairs (number of sources × detectors), and either K wavelengths for MSFT or L time gates for TFLM:

$$y=W_{S}c,$$ (1)

$$y=W_{\mathrm{TD}}c,$$ (2)

where $W_S=[W_{S_1},\dots ,W_{S_N}]$ is the (KM × NV) spectral weight matrix, $W_{\mathrm{TD}}=[W_{{\mathrm{TD}}_1},\dots ,W_{{\mathrm{TD}}_N}]$ is the (LM × NV) TD weight matrix, y is a measurement vector with dimensions (LM × 1) for TD and (KM × 1) for spectral data, and $c={[c_1,\dots ,c_N]}^T$ is a (NV × 1) parameter vector containing the unknown concentrations for each fluorophore.

Both the spectral and lifetime weight matrices can be factorized into a product of a basis matrix containing either the spectral or lifetime basis functions alone, and a spatially varying matrix representing the diffuse propagation in the medium. For MSFT, the factorized form is

$$W_S={\overline{W}}_S{A}_S,$$ (3)

where $A_S=[b_1(\lambda )\otimes I,\dots ,b_N(\lambda )\otimes I]$ is a (KV × NV) spectral basis matrix containing Kronecker products $(\otimes )$ of the excitation or emission spectra and the (V × V) identity matrix, I, and ${\overline{W}}_S=\text{diag}({\overline{W}}_{S_1},\dots ,{\overline{W}}_{S_k})$ is a (KM × KV) block diagonal matrix containing continuous wave (CW) weight matrices for each wavelength. A similar factorization also occurs in TFLM in the decaying or the asymptotic regime $(t\gg {\tau }_D)$ [7,8] of the TD fluorescence signal, assuming that ${\tau }_n>{\tau }_D$, where τ_D is the intrinsic diffusion time scale for the light transport in the medium [7]:

$$W_{\mathrm{TD}}=A_{\mathrm{TD}}{\overline{W}}_{\mathrm{TD}}.$$ (4)

Here, $A_{\mathrm{TD}}=[\exp (-t/{\tau }_1)\otimes I,\dots ,\exp (-t/{\tau }_N)\otimes I]$ is a (LM × NM) temporal basis matrix containing Kronecker products of exponential decay functions and the (M × M) identity matrix, I, and $\overline{W}=\text{diag}({\overline{W}}_{{\mathrm{TD}}_1},\dots ,{\overline{W}}_{{\mathrm{TD}}_N})$ is a (NM × NV) block diagonal matrix containing reduced absorption CW weight matrices for each lifetime component. A fundamental difference between the spectral and TD forward problems is immediately clear from Eqs. (3) and (4), as schematically represented in Fig. 1. For spectral multiplexing, the mixing of the unknown fluorophore concentrations occurs at the location of the individual voxels (through A_S). The mixed concentrations are then propagated through the medium by wavelength-dependent CW weight matrices $({\overline{W}}_S)$. For lifetime multiplexing, the individual concentrations are first propagated through the medium by reduced absorption CW matrices $({\overline{W}}_{TD})$ followed by mixing with temporal basis functions A_TD).

Fig. 1.

Schematic showing the fundamental difference in the order of the mixing and diffuse propagation steps in TFLM and MSFT forward problems for three fluorophores with concentrations c₁, c₂, c₃, lifetimes τ₁, τ₂, τ₃, and spectral basis functions b₁, b₂, b₃.

We focus on linear inverse operators $\widehat{W}$ which allow the reconstructed concentration $\widehat{c}$ to be expressed as

$$\widehat{c}={\widehat{W}}_y=Rc+{\widehat{W}}_n,$$ (5)

where we have introduced the (NV × NV) model resolution matrix, $R=\widehat{W}W$, and n represents the additive noise. We focus on the bias term, Rc [9], assuming that the noise sensitivity term, ${\widehat{W}}_n$, can be minimized with appropriate regularization. We first calculate the resolution matrices R of the TD and spectral methods based on both their direct forward problems [Eqs. (1) and (2)] and the corresponding factorized forms [Eqs. (3) and (4)]. Consider first an inversion of the spectral forward problem in Eq. (1), called the direct spectral (DS) method [10,11]. Using Tikhonov regularization [9] for inverting W_S and substituting Eq. (3), we get the following resolution matrix for the direct spectral case:

$$R_{\mathrm{DS}}={\widehat{W}}_S{W}_S=A_S^T{\overline{W}}_S^T{({\overline{W}}_S{A}_S{A}_S^T{\overline{W}}_S^T+\lambda I)}^{-1}[{\overline{W}}_S{A}_S],$$ (6)

where λ is the Tikhonov regularization parameter. Alternately, Eq. (3) can be used to invert the spectral data in two stages [10,11]. First, ${\overline{W}}_S$ is inverted using its Tikhonov-regularized inverse matrix, ${\widehat{\overline{W}}}_S$. Next, due to the well-conditioned nature of A_S, it is inverted without regularization by multiplication with its Moore–Penrose pseudoinverse, $A_S^{†}$. The latter step is equivalent to performing a linear fit to the spectral basis functions, b_n(λ), at each voxel. The resolution matrix for this indirect spectral (IS) method is given by

$$R_{\mathrm{IS}}=A_S^{†}{\widehat{\overline{W}}}_S{W}_S=A_S^{†}[{\overline{W}}_S^T{({\overline{W}}_S{\overline{W}}_S^T+\lambda I)}^{-1}]{\overline{W}}_S{A}_S,$$ (7)

where we have again used Eq. (3) for W_S.

The inversion of the TD data can also proceed using a direct or a two-step approach. A direct inversion of the TD data in Eq. (2) using Tikhonov regularization and substitution of Eq. (4) leads to the direct TD (DTD) approach, with a resolution matrix [12]:

$$R_{\mathrm{DTD}}={\overline{W}}_{\mathrm{TD}}^T{({\overline{W}}_{\mathrm{TD}}{\overline{W}}_{\mathrm{TD}}^T+\lambda {(A_{\mathrm{TD}}^T{A}_{\mathrm{TD}})}^{-1})}^{-1}{\overline{W}}_{\mathrm{TD}}.$$ (8)

If we exploit the factorization in Eq. (4), the Moore–Penrose pseudoinverse of A_TD can first be applied to the time points in the asymptotic regime resulting in the decay amplitudes, $a=A_{\mathrm{TD}}^{†}y$. This step is equivalent to a linear fit of the TD data with exponential decay basis functions. Next, the amplitudes a are inverted using the Tikhonov inverse of ${\overline{W}}_{\mathrm{TD}}$, resulting in the asymptotic TD (ATD) resolution matrix [12]

$$R_{\mathrm{ATD}}={\widehat{\overline{W}}}_{\mathrm{TD}}{A}_{\mathrm{TD}}^{†}{W}_{\mathrm{TD}}={\overline{W}}_{\mathrm{TD}}^T{({\overline{W}}_{\mathrm{TD}}{\overline{W}}_{\mathrm{TD}}^T+\lambda I)}^{-1}{\overline{W}}_{\mathrm{TD}},$$ (9)

where we have used Eq. (4). A key step in the derivation of the TD resolution matrices is the use of the identity $A_{\mathrm{TD}}^{†}{A}_{\mathrm{TD}}=1$. A similar identity cannot be used in the spectral case due to the reverse order of A_S and ${\overline{W}}_S$ in Eq. (3).

The cross-talk performance of the direct and indirect MSFT and TFLM problems can now be quantitatively compared using their resolution matrices in Eqs. (6)–(9) [12]. Let the resolution matrix R be divided into N × N blocks, each of the size V × V. Then, the on-diagonal blocks ($[R_{ii}],i=1\dots N$) contain the point spread functions for individual fluorophores, while the off-diagonal blocks ($[R_{ij}],i,j=1\dots N,i\ne j$) represent the cross-talk between fluorophores with distinct spectra or lifetimes. First, it is clear that all matrices in Eq. (9) are block diagonal since ${\overline{W}}_{\mathrm{TD}}$ is block diagonal, so that R_ATD is also block diagonal. This implies that ATD solutions provide zero cross-talk between the c_n’s. In the DTD case, the covariance matrix term, ${(A_{\mathrm{TD}}^T{A}_{\mathrm{TD}})}^{-1}$, in Eq. (8), results in off-diagonal terms in R_DTD, leading to solutions with non-zero cross-talk.

For MSFT, the presence of the non-diagonal spectral basis matrices, A_S and $A_S^{†}$, in both Eqs. (6) and (7), implies that, in general, both R_DS and R_IS are not block diagonal. However, an inspection of Eqs. (6) and (7) reveals that R_DS and R_IS are block diagonal under certain conditions. Considering the DS case [Eq. (6)] first, and switching to the equivalent overdetermined form of the inverse matrix, we have [using Eq. (3)]

$$R_{\mathrm{DS}}={(A_S^T{\overline{W}}_S^T{\overline{W}}_S{A}_S+\lambda I)}^{-1}(A_S^T{\overline{W}}_S^T{\overline{W}}_S{A}_S).$$ (10)

When the basis functions b_j(λ) are nonoverlapping [i.e., $b_l({\lambda }_j)b_m({\lambda }_j)=0\forall j$, for all fluorophore pairs (l, m)], it is clear that the off-diagonal blocks ${\Sigma }_j{b}_l({\lambda }_j)b_m({\lambda }_j){\overline{W}}_{S_j}^T{\overline{W}}_{S_j}$ of the term $A_S^T{\overline{W}}_S^T{\overline{W}}_S{A}_S$ are zero. Hence, R_DS also becomes block diagonal. For the IS case, we can rewrite Eq. (7) as

$$R_{\mathrm{IS}}={(A_S^T{A}_S)}^{-1}{A}_S^T\tilde{W}{A}_S,$$ (11)

where $\tilde{W}=[{\overline{W}}_S^T{({\overline{W}}_S{\overline{W}}_S^T+\lambda I)}^{-1}]{\overline{W}}_S=\text{diag}({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_K)$ is a block diagonal matrix with the V × V matrices α_j along the diagonal blocks, and we have applied the definition of the Moore–Penrose pseudoinverse for full-column rank matrices, $A_S^{†}={(A_S^T{A}_S)}^{-1}{A}_S^T$. When the basis functions are nonoverlapping, ${(A_S^T{A}_S)}^{-1}$ becomes block diagonal and $A_S^T\tilde{W}{A}_S$ whose off-diagonal blocks are equal to ${\Sigma }_j{b}_l({\lambda }_j)b_m({\lambda }_j){\alpha }_j$ is also block diagonal. Hence, R_IS also becomes block diagonal. Additionally, when the optical properties are wavelength independent, we have ${\overline{W}}_{S_1}={\overline{W}}_{S_2}\dots ={\overline{W}}_{S_K}({\alpha }_1={\alpha }_2=\dots =\alpha )$, and R_IS can be simplified to $R_{\mathrm{IS}}={(A_S^T{A}_S)}^{-1}{A}_S^T{A}_S\text{diag}(\alpha ,\dots ,\alpha )=\text{diag}(\alpha ,\dots ,\alpha )$, which is a NV × NV block diagonal matrix. We can thus summarize the conditions for zero cross-talk in spectral multiplexing as follows:

(1) R_IS becomes block diagonal when the blocks along the diagonal of ${\overline{W}}_S$ are equal. This occurs when the tissue optical properties (and the ${\overline{W}}_{S_j}$.) are wavelength independent.

(2) Both R_IS and R_DS become block diagonal when the spectral basis functions are nonoverlapping. For biomedical applications, which involve strong wavelength-dependent optical properties and NIR fluorophores that typically exhibit broad and overlapping spectra, both conditions are hard to satisfy.

We note that while the above results employed the Tikhonov form of inversion, the non-existence of a general zero cross-talk estimator for MSFT can be more generally proved based on a recently derived nullity condition for multiplexing (NCM) [12]. The NCM requires that the transpose of the weight matrix be of non-zero nullity for the existence of zero cross-talk estimators. It can be shown that the nullity of the spectral weight matrix, $W_{S_j}^T$, is generally zero, implying that zero cross-talk solutions do not exist for arbitrary conditions. However, the nullity can become non-zero when the optical properties are wavelength independent or the spectra are nonoverlapping, in agreement with the zero cross-talk conditions specified above. On the other hand, the TD weight matrix [Eq. (4)] has non-zero nullity in the asymptotic region [12], thereby satisfying the NCM and ensuring the existence of zero cross-talk estimators (ATD).

We numerically illustrate these conditions and compare the performance of MSFT with TFLM below. To proceed, we define a measure for cross-talk in terms of the resolution matrix. Let j be the linear index corresponding to voxel (x, y, z). Note that the jth column of R₂₁ represents the cross-talk into the fluorophore 2 channel due to fluorophore 1 located at (x, y, z). The cross-talk of fluorophore 2 into fluorophore 1 can also be defined similarly using the columns of R₁₂. The total cross-talk C_j for the voxel j can be expressed as:

$$C_j=\sum _{i=1}^V|{[R_{21}]}_{\mathit{ij}}|/\sum _{i=1}^V|{[R_{11}]}_{ij}|+\sum _{i=1}^V|{[R_{12}]}_{ij}|/\sum _{i=1}^V|{[R_{22}]}_{ij}|.$$ (12)

For the simulations, we consider the recently developed NIR fluorescent proteins (iRFP670, iRFP702, and iRFP720) [13] with emission spectra in Fig. 3(a) and lifetimes of 0.68, 0.78, and 0.93 ns [6]. Consider a rectangular slab of size 2 cm × 2 cm × 2 cm with a 1 mm³ voxel size, with 42 sources and 42 detectors located at the z = 0 cm and z = 2 cm planes, respectively. For TD methods, the bulk absorption and reduced scattering were set to μ_a = 0.6 cm⁻¹ and μ_S′ = 10 cm⁻¹, respectively. For spectral methods, μ_a was assumed to linearly increase from 0.6 cm⁻¹ to 1.2 cm⁻¹ across the wavelength range simulated (650–800 nm), while ${\mathrm{\mu }}_s^{\prime }$ was kept constant at 10 cm⁻¹. The TD data were simulated for 25 time gates separated by 100 ps, while the spectral data consisted of eight wavelengths separated by 20 nm. The TD and the spectral Green’s functions were generated using a Monte Carlo model [8].

Fig. 3.

Dependence of cross-talk of the spectral reconstruction methods on the net variation in the bulk absorption across the full spectral range (Δμ_a). (a) Fluorescence emission spectra for three NIR fluorescent proteins (iRFP670 in blue, iRFP702 in green, and iRFP720 in red) [13]. Total cross-talk [Eq. (12)] for each pair of iRFPs co-localized at the center of a 2 cm thick medium, plotted as a function of Δμ_a using the (b) DS and (c) IS methods.

Figure 2 illustrates the form of the resolution matrix for TD and spectral methods in Eqs. (6)–(9), for a pair of fluorophores, iRFP702 and iRFP720. To aid in visualization, the rows and columns were binned by a factor of 80. Figure 2 shows that the cross-talk terms for R_DS are positive and generally symmetric between the two fluorophores. On the other hand, R_IS shows more asymmetry between the two fluorophores with a major portion of the cross-talk being negative. For the TD case, while R_DTD contains off-diagonal terms as expected, R_ATD is a block diagonal matrix with zero cross-talk.

Fig. 2.

Resolution matrices for spectral and lifetime tomography, R_DS, R_IS, R_DTD, and R_ATD, as given in Eqs. (6)–(9). Each resolution matrix was generated for a 2 cm thick diffuse medium for multiplexing of two NIR fluorophores (iRFP702 and iRFP720). The off-diagonal blocks of each resolution matrix (outlined in red) represent cross-talk between different fluorophores.

We next study the effect of wavelength dependence of the optical properties and the fluorescence spectral overlap on the cross-talk for the MSFT methods, using the same simulation parameters as Fig. 2. Note that the cross-talk in the ATD method is not affected by the optical properties of the medium as the ATD resolution matrix remains block diagonal, regardless of the optical properties. The cross-talk [Eq. (12)] for a voxel at the center of the medium is plotted as a function of the net change in optical absorption, Δμ_a, across the entire spectral range, for all pairs of the three iRFPs. Δμ_a was varied from 0% to 200% (from a baseline of μ_a = 0.6 cm⁻¹), while the bulk scattering was kept constant. Figure 3(c) shows that the cross-talk for IS is zero for all fluorophore pairs when Δμ_a = 0 and sharply increases for larger Δμ_a. For the DS method, the cross-talk is significant, even when Δμ_a = 0 and shows a gradual increase for larger Δμ_a. Furthermore, the cross-talk increases with increasing spectral overlap for both the IS and DS methods, with the iRFP670/iRFP720 pair showing the least cross-talk. These results are in agreement with the general conditions for zero-cross talk in MSFT presented above. The cross-talk in the IS approach depends on both the spectral overlap and the optical property variation whereas, in the DS case, the cross-talk depends mainly on the spectral overlap.

We next compare the effect of cross-talk on tomographic imaging with MSFT and TFLM (ATD) for a more realistic imaging geometry, using a digital mouse atlas [14] with heterogeneous optical property values [15]. Seventy-two sources were placed beneath, and 72 detectors were placed above a region of the torso covering parts of the skeleton, lung, heart, liver, and kidneys. A fluorescent inclusion (1 mm³) was placed at the centroid of the liver. A 2% shot noise was added to all measurements, and regularization was chosen so that the resolution, measured as FVHM^1/3 (FVHM, full volume at half-maximum), reached 5 mm for MSFT and 6 mm for ATD. Figure 4 shows the tomographic reconstructions for the case where the inclusion contained either iRFP670 or iRFP720 alone. While ATD shows negligible cross-talk, the DS shows significant cross-talk for both cases. The IS method showed strong cross-talk into iRFP720 when iRFP670 was used, while the cross-talk from iRFP720 into iRFP670 was minimal. This one-sided cross-talk (also seen in Fig. 2 for the IS case) can be attributed to the redshift of the fluorophore emission spectra due to diffuse propagation [16], which causes the iRFP670 signals to leak into the iRFP720 channel. Finally, we considered the case where both iRFP670 and iRFP720 were simultaneously present (overlapping) in the inclusion, and estimated the ratio of the relative concentration of iRFP670 to iRFP720 using both the MSFT and TFLM methods. The ATD approach resulted in an error of 6.3% in the relative concentration, while the DS and IS methods resulted in significantly higher error rates of 40.5% and 77.9%, respectively, due to cross-talk.

Fig. 4.

Tomographic reconstructions to compare the cross-talk performance of the DS (top row), IS (middle row), and ATD (bottom row). A single inclusion with either iRFP670 (left two columns) or iRFP720 (right two columns) is located at the centroid of the liver in a digital mouse atlas with heterogeneous optical properties. The images show X–Y slices of the reconstructed concentration averaged over a depth of 6 mm around the inclusion for both iRFP670 and iRFP720 when either alone is present in the inclusion.

In summary, we have presented a comparison of tomographic spectral and lifetime multiplexing using a rigorous mathematical framework based on the model resolution matrix. The actual amount of cross-talk and quantitation error for spectral methods will depend on specific experimental conditions, including spatial and wavelength dependence of the optical properties, the accuracy in optical property estimation, and the degree of spectral overlap. However, the general results of this Letter, namely the superior cross-talk performance of ATD compared to the MSFT methods (due to measurement-level versus voxel-level mixing) and the conditions for zero cross-talk in MSFT are valid, regardless of experimental conditions.