Multiresolution Localization with Temporal Scanning for Super-Resolution Diffuse Optical Imaging of Fluorescence

Abstract

A super-resolution optical imaging method is presented that relies on the distinct temporal information associated with each fluorescent optical reporter to determine its spatial position to high precision with measurements of heavily scattered light. This multiple-emitter localization approach uses a diffusion equation forward model in a cost function, and has the potential to achieve micron-scale spatial resolution through centimeters of tissue. Utilizing some degree of temporal separation for the reporter emissions, position and emission strength are determined using a computationally efficient time stripping multiresolution algorithm. The approach circumvents the spatial resolution challenges faced by earlier optical imaging approaches using a diffusion equation forward model, and is promising for in vivo applications. For example, in principle, the method could be used to localize individual neurons firing throughout a rodent brain, enabling direct imaging of neural network activity.

I. Introduction

Fluorescence imaging has become a standard tool in biomedical research because modulation of fluorescence intensity in space and time can provide information on biochemical processes [1]. Methods such as confocal microscopy [2] and multi-photon imaging [3] have enabled high-resolution fluorescence imaging near the surface of biological tissue (referred to hereafter simply as “tissue”). However, fluorescence imaging in deep tissue, where the propagation direction of light becomes randomized, presents a major challenge in optical imaging. Information is lost due to scattering and absorption, which hinders image formation. In this case, cost-function-based imaging methods with a forward model can be used to form images of fluorescence. In such a computational imaging framework, the use of a simple diffusion model to describe the photon transport has been shown to capture the mean intensity of light under suitable conditions in the diffusive regime that can be found in tissue with visible and near-infrared light. This has led to substantial interest in diffuse optical tomography (DOT, or optical diffusion tomography, ODT) [4], [5], and fluorescence diffuse optical tomography (FDOT or FODT) [6], [7]. Imaging of fluorescence resonance energy transfer (FRET) parameters, providing a spatial map of the nanometer FRET distance that is important in molecular biology, has also been shown [8], [9]. While much progress has been made, diffuse optical imaging methods continue to suffer from low spatial resolution due to the loss of high-frequency information as light propagates through scattering media [10]. This attenuation occurs whether the media is lossy (with absorption) or lossless (without absorption).

We present a deep-tissue optical imaging method that relies on treating fluorescent emitters whose signals vary in time as a set of point sources, and uses measurements of multiply scattered light as a function of time to find their locations to high precision. Our approach uses an optimization method with a forward model for the participating medium. It appears that, in the diffusive regime where the diffusion approximation is valid, a resolution on the order of ten microns through one centimeter or more of tissue should be possible (see the experimental demonstration for finding the location of a single point fluorophore [10]). Consequently, the methods have the potential to directly image neurons throughout the brain of a small mammal, and could provide new understanding for neuroscience. The work should also allow for imaging in other dynamic source situations in a cluttered environment.

Of particular significance is the prospect of imaging calcium (Ca²⁺) channel activity in neurons throughout the brain. Neuron activity involves Ca²⁺ transport through channels in the cell body and then, after signal transfer along the axon, regulation of the neurotransmitter across the synaptic cleft in communication with another neuron. Monitoring Ca²⁺ signaling thus presents a direct picture of neuron activity [11]. If it were possible to image Ca²⁺ channel activity across the brain, the resulting maps would provide new understanding about how the brain works, and also answer fundamental questions about health and disease. Fortunately, Ca²⁺ fluorescence and FRET sensors have been developed for in vitro studies [11]. The principle is that Ca²⁺ modulates the fluorescence emission or the energy transfer from the donor to the acceptor in the case of FRET, thereby encoding concentration information in the fluorescence that can be detected. The temporal modulation of the fluorescence therefore reports on calcium channel activity. However, with optical excitation using a laser, both the excitation light and the emitted fluorescence are multiply scattered. Computational imaging of fluorescence and FRET is possible with a forward model, such as the diffusion equation, using a suitable cost function [6], [8], [9], but the spatial resolution achieved precludes being able to resolve individual neurons in deep tissue. By representing the local Ca²⁺ fluorescent reporter as a time-varying point emitter or point absorber (the donor emission can reduce due to FRET), we present a framework suitable for high-resolution localization using a forward model.

None of the available in vivo methods provide direct access to neurons, i.e., they measure secondary parameters. For example, brain imaging with fMRI has been developed using the blood-oxygen-level dependent (BOLD) contrast [12], and this has become an important tool for brain science. However, blood oxygen levels provide low resolution and a slow and remote measure of brain signaling. Likewise, whole-brain imaging of manganese (Mn²⁺) has been developed as manganese-enhanced MRI (MEMRI), and there is evidence that increased Mn²⁺ concentration develops in the region of the brain associated with activity over long time scales, and this method has been used in neuroscience studies (see, for example, [13]). Under certain conditions, Mn²⁺ is thought to enter neurons through Ca²⁺ channels [14] and hence be a possible indicator of neuron circuitry. Optical methods such as two-photon microscopy [3] offer more direct access to a wider range of neurobiological information through optical contrast agents, but only provide information about the cortical surface of the brain visible through the use of cranial windows [15].

We present a super-resolution localization imaging method that is an extension of earlier localization concepts [10], [16] and is based on spatiotemporal data. There has been substantial prior work done on imaging using a diffusion forward model, but without the point emitter constraint. These methods were developed to overcome the detrimental effects of scattering, enabling deep-tissue fluorescence imaging [1]. In FDOT [6], [7], [17], computational imaging allows the formation of volumetric images of the absorption, scattering, and fluorescence from boundary measurements. Fundamentally, these methods detect what have become known as diffuse photon density waves (DPDWs) emanating from excitation or fluorescent sources [18]. For typical measurements and tissues, the wavelength of a DPDW is a few centimeters, and measurements are made in the near field. Considering the angular spectrum of DPDWs [18], all the spatial frequencies propagate to the detector, however they are highly attenuated, even without absorption. This causes a severe reduction in spatial resolution with depth into the medium. Consequently, the position of mm-scale regions can be found at cm depths in tissue-like scatter using an equivalent point representation [10], [19], [20], and likewise the Ca²⁺ channel fluorescence reporter position can be treated in this fashion. The dependence of the spatial resolution on depth is nonlinear, but for typical tissues, measurement geometries, and beyond a depth of about 1 cm, spatial resolutions of about depth/2 have been achieved [10], [18]. The temporal localization method we present circumvents the earlier resolution limits. However, the object is modeled as a set of points, so other geometrical information is lost.

We develop a model describing the simulated measured fluorescence intensity due to multiple emitters within a highly scattering medium in Section II. We then use the model to localize fluorescent sources separated in space and distributed in time in Section III, and the results are presented in Section IV. We compare our method to existing super-resolution imaging methods and discuss potential applications in Section V, and conclude in Section VI.

II. Model

A. Physical Problem

The primary application we address is the direct in vivo imaging of neuron activity through optical fluorescence reporting of Ca²⁺ concentration. An optical experiment to measure Ca²⁺ signaling would involve using a pulsed laser source to excite the fluorescent reporter and time-gated measurements of multiply scattered light at a number of positions around the periphery of the domain to be imaged. That domain needs to be defined as a surface, and this could be achieved a variety of ways, such as the camera imaging of a laser line scanned over the subject with a transform to map from the subject to the camera plane [21]. The temporal sampling could be achieved with a time-gated camera or an array of avalanche photodiodes (APD) or photomultiplier tubes (PMTs). Regardless of the specific experimental implementation, we assume that the necessary temporal information at a number of points around the image domain are available. The question we address here is how one could image blinking fluorescence contrast sources that are physically optical reporters of Ca²⁺ channel activation. In principle, by treating the fluorescent Ca²⁺ reporter in each neuron as a point emitter in space having temporal characteristics attributable to the neuron response and the fluorescence lifetime, it becomes possible to use a mathematical framework where each neuron can be modeled in space as a single emitter. This is called localization.

B. Coupled Diffusion Equations

We use the diffusion approximation to the radiative transfer equation to describe the propagation of light in a highly scattering medium such as tissue [6], [17]. This is an incoherent picture that has proven useful in describing the mean optical intensity when there is much more scatter than absorption, which is the case for most tissue having millimeter thickness or more and red or near-infrared light. The coupled diffusion model in the time domain is given by [22]

$$\begin{gathered} \frac{1}{v}\frac{\partial }{\partial t}{\varphi }_x(\mathbf{r},t)-\nabla \cdot \left[D_x(\mathbf{r})\nabla {\varphi }_x(\mathbf{r},t)\right]+{\mu }_{a_x}(\mathbf{r}){\varphi }_x(\mathbf{r},t) \\ =S_x(\mathbf{r};t) \end{gathered}$$ (1)

$$\begin{gathered} \frac{1}{v}\frac{\partial }{\partial t}{\varphi }_m(\mathbf{r},t)-\nabla \cdot \left[D_m(\mathbf{r})\nabla {\varphi }_m(\mathbf{r},t)\right]+{\mu }_{a_m}(\mathbf{r}){\varphi }_m(\mathbf{r},t) \\ ={\varphi }_x(\mathbf{r},t)\ast S_f(\mathbf{r};t), \end{gathered}$$ (2)

where r denotes the position, ϕ (W/mm²) is the photon flux density, μ_a (mm⁻¹) is the absorption coefficient, $D=1/\left[3\left({\mu }_a+{\mu }_s^{\prime }\right)\right]$ (mm) is the diffusion coefficient, ${\mu }_s^{\prime }$ (mm⁻¹) is the reduced scattering coefficient, v = c/n is the speed of light in the medium, where c is the speed of light in free space and n is the refractive index, the subscripts x and m, respectively, denote parameters at the excitation and emission wavelengths, λ_x and λ_m, S_x (W/mm³) is the excitation source term, S_f (s⁻¹/mm) is the fluorescence source term, and * signifies a temporal convolution. The spatially-dependent fluorescence source in (2), assuming a single lifetime at each point in space and Dirac delta excitation at t = 0, can be written as

$$S_f(\mathbf{r};t)=\frac{\eta (\mathbf{r})}{{\gamma }_f(\mathbf{r})}\exp \left(\frac{-t}{{\gamma }_f(\mathbf{r})}\right),$$ (3)

where $\eta ={\eta }_f{\mu }_{a_f}$ is the fluorescent yield, with ${\mu }_{a_f}$ the fluorophore absorption coefficient at λ_x and η_f the fluorophore quantum yield, and γ_f is the fluorescence lifetime. With a temporal Fourier transform, resulting in the frequency domain form of (1) and (2), and considering homogeneous D and μ_a, the resulting scalar wave equations for ϕ describe the propagation of DPDWs.

Throughout this paper, we present results in terms of the mean free path, l* = 3D. However, the results can be scaled to geometries of different size and amount of scatter, according to the scaling principles of Appendix A. This scaling argument allows physical problems as diverse biological tissue, atmospheric scatter and snow to be considered.

We show the spatial resolution with an example FDOT reconstruction in Fig. 1, where we numerically reconstruct the fluorescence yield (η) in 3-D space using (1) and (2), as we have described previously [6]. We assume optical properties similar to tissue at wavelengths of red and nearinfrared light [23] (${\mu }_s^{\prime }=2{\text{mm}}^{-1}$, μ_a = 0.02 mm⁻¹), and that measurements are made with a signal-to-noise ratio (SNR) of 30 dB at the detector, with Gaussian noise that is based on a shot-noise model (as described in Section II-E). The 3D geometry is a cube of side length 32l*, where sources and detectors were placed on each side of the cubic geometry, and the point fluorescent emitters are near the center. There is 1 source and 1 detector on each of the six faces of the cubic scattering medium, each placed equidistant from the centroid of the face and one of its edges. Figure 1(a) shows a 2-D slice of the target image, η, and Fig. 1(b) shows the corresponding 2-D slice isoimage of the estimated or reconstructed image, $\widehat{\eta }$. It is clear from Fig. 1(b) that the point emitters in Fig. 1(a) are not resolvable with FDOT, under the conditions considered.

Fig. 1.

3D FDOT reconstruction of four point fluorescent emitters with 30 dB SNR. The optical properties are similar to tissue, where ${\mu }_s^{\prime }=2{\text{mm}}^{-1}$, μ_a = 0.02 mm⁻¹, and n = 1.33, giving a mean free path length l* = 3D = 0.5 mm. The geometry is a cube of side length 32l* with the point fluorescent emitters near the center. 6 sources and detectors are placed at the boundary of the cube, one source (red) and one detector (blue) at each face. (a) Target η xz image slice, where the fluorescent emitters are located at depths of about 17l*, and are separated by about 2l*. (b) Estimated $\widehat{\eta }$ reconstructed with FDOT. An isoimage with $\widehat{\eta }=0.003{\text{mm}}^{-1}$ threshold was formed and the same xz slice as in (a) is shown. Note that it is a 3D isoimage, and the shading is meant to indicate its shape. In this case, the separation of the fluorescent emitters is much less than depth/2 and they cannot be resolved.

C. Forward Model for a Single Fluorescent Emitter

Equations (1) and (2) can be used to form a model of the detected power at λ_x and λ_m, respectively, at a set of locations around the periphery of the scattering medium. We consider the case of a set of point-emitting fluorophores, each with a differing temporal response due to the physical situation, so that the response can be attributed to a sequence of single point fluorophores. We start by describing the response due to a single fluorescent source, such as would be the case for a Ca²⁺ reporter, embedded in a highly scattering medium.

The problem geometry is shown in Fig. 2. The incident laser light at wavelength λ_x can be modeled as an (isotropically emitting) point source for S_x in (1) located l* into the scattering medium (the brain). Point excitation sources represent a set of laser beam excitation locations at known positions r_s. We therefore assume the excitation source term is given by S_x(r; t) = S_oδ(r − r_s, t), where S_o is the excitation power (W/mm³) and δ(·) is the Dirac delta function. The implication is that the temporal excitation laser source is short relative to the other time constants involved. In this work, we let S_o = 1 for simplicity. Additionally, point detectors that collect light at λ_m are assumed to be placed at known positions r_d. Finally, a fluorescent emitter (at λ_m) that is excited at λ_x is assumed to be located at the unknown position r′. Using the domain geometry in Fig. 2, we define g_x(r_s, r′, t) as the Green’s function from (1) at λ_x and g_m(r′, r_d, t) as the Green’s function from (2) at λ_m. The fluorescence emission photon flux density at the detector is then

$$\begin{gathered} {\varphi }_m\left({\mathbf{r}}_s,{\mathbf{r}}_d,t\right)= \\ \int g_m\left({\mathbf{r}}^{\prime },{\mathbf{r}}_d,t\right)\ast S_f\left({\mathbf{r}}^{\prime };t\right)\ast g_x\left({\mathbf{r}}_s,{\mathbf{r}}^{\prime },t\right)d{\mathbf{r}}^{\prime }, \end{gathered}$$ (4)

where * is the temporal convolution.

Fig. 2.

Model geometry with position vector r = (x,z). Excitation sources at λ_x (red) are placed at known positions r_s, point fluorescence emission locations are assumed to be r_f (green), and detectors at λ_m are placed at known positions r_d (blue).

We assume a point fluorophore is located at r_f, as seen in Fig. 2, and from (3),

$$S_f(\mathbf{r};t)=\frac{{\eta }_f{\mu }_{a_f}}{{\gamma }_f}\exp \left(\frac{-t}{{\gamma }_f}\right)\delta \left(\mathbf{r}-{\mathbf{r}}_f\right).$$ (5)

For simplicity, we assume that D and μ_a are constant, and that v is the same at λ_x and λ_m. Using the Green’s function solution to the diffusion equation [22], we find from (4) that

$$\begin{gathered} {\varphi }_m\left({\mathbf{r}}_s,{\mathbf{r}}_f,{\mathbf{r}}_d,t\right)= \\ \frac{v}{{\left(4\pi D_{m}vt\right)}^{3/2}}\exp \left(\frac{-{\left|{\mathbf{r}}_d-{\mathbf{r}}_f\right|}^2}{4D_{m}vt}-{\mu }_{a_m}vt\right) \\ *\frac{\eta }{{\gamma }_f}\exp \left(\frac{-t}{{\gamma }_f}\right) \\ *\frac{v}{{\left(4\pi D_{x}vt\right)}^{3/2}}\exp \left(\frac{-{\left|{\mathbf{r}}_f-{\mathbf{r}}_s\right|}^2}{4D_{x}vt}-{\mu }_{a_x}vt\right). \end{gathered}$$ (6)

We also assume that ${\mu }_{a_x}={\mu }_{a_m}={\mu }_a$ and D_x = D_m = D. The detected power through an aperture is described by the current density J (W/mm²), with a diffusion framework, and we assume pointwise measurements

$$\mathbf{J}\left({\mathbf{r}}_s,{\mathbf{r}}_f,{\mathbf{r}}_d,t\right)=-D\nabla {\varphi }_m\left({\mathbf{r}}_s,{\mathbf{r}}_f,{\mathbf{r}}_d,t\right).$$ (7)

For laser excitation source locations ${\mathbf{r}}_{{\mathbf{s}}_q}$ with q ∈ [1, …, Q] and fluorescence detector locations ${\mathbf{r}}_{d_m}$ with m ∈ [1, …, M], we write the detected fluorescent photon current density in compact form as ${\mathbf{J}}_{qm}\left({\mathbf{r}}_f,t\right)=\mathbf{J}\left({\mathbf{r}}_{s_q},{\mathbf{r}}_f,{\mathbf{r}}_{d_m},t\right)$, where we emphasize the dependence on r_f because it is estimated in Section III. The pointwise detected fluorescence is then $G_{qm}\left({\mathbf{r}}_f,t\right)=\left|{\mathbf{J}}_{qm}\left({\mathbf{r}}_f,t\right)\cdot \widehat{\mathbf{n}}\right|$, where $\widehat{\mathbf{n}}$ is the unit vector normal to the detector surface and G signifies the (diffusion equation) Green’s function basis. For simplicity we use the analytical solution for G_qm(r_f, t) in this work, but the model could also be solved using finite element or other numerical methods [5]. These analytical solutions to the diffusion equation subject to boundary conditions have been shown to match Monte Carlo simulations [24] and experiments of photon propagation in highly scattering media [19]. It has been shown that small fluorescent inhomogeneities can be well approximated as point emitters because of the rapid attenuation of high spatial frequencies in scattering media [10], [19], [20], [25]. For this reason, imaging based on a point representation is possible.

D. Forward Model for Multiple Fluorescent Emitters

In the case where multiple fluorescent emitters are present, the forward model must be modified. We assume each fluorescent emitter can be represented by a point that fluoresces with a unique and increasing delay τ from t = 0. We let K fluorescent emitters be located at positions ${\mathbf{r}}_{f_k}$, where k is an index from 1 to K. Each fluorescent emitter has a different yield, η_k, and fires with different delay, τ_k. Then, the detected fluorescence for a single source-detector pair measured over the temporal support w starting at time t_o is

$$\begin{gathered} {\tilde{G}}_{qm}(\mathbf{R},\tau ,t)=\mathrm{rect}\left(\frac{t-0.5w-t_o}{w}\right) \\ \times \sum _{k=1}^K\delta \left(t-{\tau }_k\right)\ast G_{qm}\left({\mathbf{r}}_{f_k},t\right), \end{gathered}$$ (8)

where for general x, rect(x) is 1 when |x| < 0.5 and zero otherwise, the vector _τ = [τ₁ …, τ_K]^T corresponds to the delays τ_k, and the vector R contains the K positions ${\mathbf{r}}_{f_k}$.

We assume the detected fluorescence from (8) is sampled with sampling interval T, and we discretize the domain such that V_vox is the volume of a single voxel, giving the forward measurements

$$f_{qmn}(\mathbf{R},\tau )=V_{vox}\left[{\tilde{G}}_{qm}(\mathbf{R},\tau ,t){|}_{t=nT}\right]$$ (9)

where n is an index from 1 to N, and t_max = NT. We emphasize the dependence on R and _τ because these parameters are estimated in Section III. We can now write the fluorescence data vector expected from the diffusion model as f(R, _τ), which is

$$\begin{gathered} \mathbf{f}(\mathbf{R},\mathbf{\tau })=[f_{111},\dots ,f_{11N},f_{121},\dots ,f_{12N},\dots ,f_{1M1}, \\ \dots ,f_{1MN},f_{211},\dots ,f_{QMN}{]}^T. \end{gathered}$$ (10)

Considering (6), each f_qmn is linear with respect to η_k, allowing us to write

$$\mathbf{f}(\mathbf{R},\mathbf{\tau })=\mathbf{F}(\mathbf{R},\mathbf{\tau })\eta ,$$ (11)

where _η = [η₁ …, η_K]^T is a vector containing the fluorescent yields (η_k) and F(R, _τ) is a matrix of dimensions [QMN, K] that contains the scaled forward measurements. The matrix F(R, _τ) can be calculated from (8) by setting all η_k equal to 1. The vector multiplication in (11) is equivalent to the superposition of the K fluorescent responses, which, after scaling, make up the columns of F(R, τ).

E. Detector Noise

We use a Gaussian noise model, for which the statistics are based on a shot-noise-limited measurement system [26], [27]. We let y be the noisy measurement vector corresponding to f, such that both have dimensions of [QMN, 1]. We assume that the noise is independent, zero mean, and Gaussian with covariance matrix ϒ, where

$${[\mathrm{\Upsilon }]}_{ii}=\alpha \left|{\mathbf{y}}_i\right|,$$ (12)

i = [1, …, QMN] is the data index, and α is a scalar parameter that is dependent on the noise process and detector physics [26]. The SNR in dB for a single source-detector pair depends on α according to

$$\text{SNR}=10\log \left[\frac{1}{\alpha }{f}_{qmn}(\mathbf{R},\tau )\right].$$ (13)

We generate simulated noisy measurements y for a specified SNR by calculating α from (13).

III. Localization for Super-Resolution Imaging

We describe a localization method that uses temporal information to enhance spatial resolution and yields the position of each fluorescent emitter (R) and the yield (η). The framework for extracting this information is presented, along with a multiresolution approach. We assume an unbounded homogeneous medium and that ${\mu }_s^{\prime }$, μ_a, and γ_f are known or extracted from a previous characterization step. The temporal form of the measured multiply scattered fluorescence is used to obtain high-spatial-resolution information for a set of point emitters under the assumption that the fluorescence from a single emitter can be measured within a temporal window w.

Compared to diffusive imaging with FDOT, fluorescence localization is a simpler problem [10], [19]. Biomedical applications of localization have included determining the location of a fluorescing tumor in a mouse [28] or the location of a targeted fluorophore embedded in the tongue of a live mouse [29]. Here, we describe the localization of multiple fluorescent emitters for the formation of super-resolution images. Our method assumes there is a significant temporal separation τ_k such that the fluorescent impulse responses can be separated, and we explore this assumption in Section III-C. By exploiting this information as prior knowledge, higher-resolution imaging compared to FDOT is achieved.

A. Localization of Multiple Fluorescent Emitters

In order to localize K fluorescent emitters inside a highly scattering medium, the positions ${\mathbf{r}}_{f_k}$ must be estimated. To further characterize the fluorescent emitters, the yields η_k, which are proportional to the fluorophore concentration, can also be estimated. We form the maximum likelihood (ML) estimation as [10], [19], [20], [25]

$$\widehat{\theta }=\arg \underset{\theta }{\max }{p}_{1,\theta }(\mathbf{y}),$$ (14)

with p_1,θ(y) is the probability distribution given by

$$p_{1,\theta }(\mathbf{y})=\frac{1}{\sqrt{{(2\pi )}^P\left|\mathrm{\Upsilon }\right|}}\exp \left(-\frac{1}{2}{\Vert \mathbf{y}-\mathbf{F}(\mathbf{R},\tau )\eta \Vert }_{{\mathrm{\Upsilon }}^{-1}}^2\right),$$ (15)

where P = QMN is the number of measurements, for an arbitrary vector u, $\Vert \mathbf{u}{\Vert }_{\mathbf{V}}^2={\mathbf{u}}^H\mathbf{Vu}$, H being the Hermitian transpose, and θ corresponds to the parameters of interest for localization, which are R and _η. After taking the logarithm, (14) is equivalent to minimizing

$$c(\mathbf{R},\tau )=\underset{\eta }{\min }\Vert \mathbf{y}-\mathbf{F}(\mathbf{R},\mathbf{\tau })\eta {\Vert }_{{\mathrm{\Upsilon }}^{-1}}^2.$$ (16)

Minimizing (16) could be accomplished using matching pursuit [30], where a dictionary of basis functions is formed using (8) (with K = 1) for all possible ${\mathbf{r}}_{f_k}$ and τ_k. The weight of each basis function is then estimated by finding the maximum inner products between the measured data y and the dictionary basis functions. For (16), the basis function weight would be η_k. However, if high temporal and spatial resolution is desired, as is the case in this work, forming the dictionary and calculating the inner products becomes computationally difficult, which we show in Section IV-B.

We propose a method to minimize (16) that takes advantage of the causality of the problem. The approach allows us to avoid calculation of the basis functions at each τ_k and their inner products, alleviating the computational burden. We assume that each τ_k is unique, allowing temporal separation of the fluorescent impulse responses, as shown in Fig. 3 for the case of two emitters. In Fig. 3(a) the temporal responses from two point fluorophore locations in a scattering domain are measured to be displaced in time but overlapping, whereas in Fig. 3(b) they are distinct. We consider the general case where the delays τ_k are not large enough for the fluorescence to decay to noise before the start of the next response, as shown in Fig. 3(a). The discretized measurement data leads to a temporal sampling period T and we consider a temporal window w, illustrated in Fig. 3, consisting of an integer number of samples separated in time by T.

Fig. 3.

Typical fluorescence temporal responses for one source and seven detectors (Q = 1, M = 7) using the same optical properties as Fig. 1. The 7 different symbols and corresponding colors represent different sourcedetector measurement pairs. The time axis is a discrete set of points t₁,…,t_N, with T between sample points. (a) The delay τ₂ is short, causing substantial overlap due to superposition. (b) The delay τ₂ is long such that the detected fluorescence decays substantially before the next fluorescence response. We show that localization of the fluorescence emitters is possible in both cases.

Considering (8), we see that η_k and ${\mathbf{r}}_{f_k}$ could be estimated using data within a temporal window starting at t_o = τ_k with w < τ_k+1 − τ_k. Thus, we assume that the data within the temporal window w comes from a single fluorophore at ${\mathbf{r}}_{f_k}$. A simplified cost function based on (16) can then be written as

$$c_k\left({\mathbf{r}}_{f_k},{\tau }_k\right)=\underset{{\eta }_k}{\min }{\Vert {\mathbf{y}}_k-{\eta }_k{\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)\Vert }_{{\mathrm{\Upsilon }}_k^{-1}}^2,$$ (17)

where ${\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)$ is derived from f(R, _τ) in (10) with t_o = τ_k and w < τ_k+1 − τ_k, y_k is the corresponding windowed measurement vector, and ϒ_k is the noise covariance matrix for y_k. In this description, y_k contains the subset of measurements from y at all detectors that are within w. Note that w is being used here as a model parameter to identify the response of a single point fluorophore, and that the measured data y has a temporal support that is much larger than w.

Forming (17) requires that τ_k is known. We estimate τ_k using the generalized likelihood ratio test (GLRT) [19], which is described in Appendix B. The GLRT uses a binary hypothesis to calculate a threshold for determining whether a point fluorophore is detectable or “localizable.” We apply the GLRT at each time index starting at t = 0 and progressing in positive time in increments of T. The time index where fluorescence is first detected is then our estimate for τ_k. Once detectability is established, (17) can be minimized using the estimated τ_k and a preselected w. To detect the next emitter, in general the estimated forward solution, ${\widehat{\mathbf{y}}}_k={\widehat{\eta }}_k\mathbf{f}\left({\widehat{\mathbf{r}}}_{f_k}\right)$, must be subtracted from y before moving the temporal window w and calculating the GLRT at the next time point. This removes the influence of one reporter’s response from the responses following it, and is valid by superposition. A similar procedure is used in matching pursuit [30]. The robustness of the method can be improved by adding positive noise to ${\widehat{\mathbf{y}}}_k$ before subtraction, where values that are negative after subtraction are set to zero.

We also consider the special case where w can be large compared to the decay time of the fluorescence responses. This is shown in Fig. 3(b), demonstrating that the subtraction step is not needed if the time scan is resumed at t = τ_k + w. However, we use the general case which allows extraction of the response of each emitter from the signals in Fig. 3(a) or (b).

To minimize (17), we use the method developed by Milstein et al. [19], which is a two-step procedure described by

$${\tilde{\eta }}_k\left({\mathbf{r}}_{f_k}\right)=\frac{{\mathbf{f}}_k^T\left({\mathbf{r}}_{f_k}\right){\mathrm{\Upsilon }}_k^{-1}{\mathbf{y}}_k}{{\mathbf{f}}_k^T\left({\mathbf{r}}_{f_k}\right){\mathrm{\Upsilon }}_k^{-1}{\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)}$$ (18)

$$c_k\left({\mathbf{r}}_{f_k},{\tau }_k\right)={\Vert {\mathbf{y}}_k-{\tilde{\eta }}_k\left({\mathbf{r}}_{f_k}\right){\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)\Vert }_{{\mathrm{\Upsilon }}_k^{-1}}^2.$$ (19)

Equation (18) was found by setting the derivative of $\Vert {\mathbf{y}}_k-{\eta }_k{\mathbf{f}}_k\left({\mathbf{r}}_k\right){\Vert }_{{\mathrm{\Upsilon }}_k^{-1}}^2$ with respect to η_k equal to zero. Equations (18) and (19) are then solved at positions ${\mathbf{r}}_{f_k}$ of interest (within the time window w), and the kth fluorescent emitter’s position and yield are estimated by

$${\widehat{\mathbf{r}}}_{f_k}=\arg \underset{{\mathbf{r}}_{f_k}}{\min }{c}_k\left({\mathbf{r}}_{f_k},{\tau }_k\right)$$ (20)

$${\widehat{\eta }}_k={\tilde{\eta }}_k\left({\widehat{\mathbf{r}}}_{f_k}\right).$$ (21)

In an experiment, where calibration scale factors are unknown, ${\widehat{\eta }}_k$ can be considered a nuisance parameter [19], where its estimate is no longer quantitative. Interestingly, if ϒ⁻¹ is the identity matrix, (18) would be equivalent to calculating the basis function weight in matching pursuit. Because of this, the localization method presented here could be considered a constrained version of matching pursuit that incorporates a noise model.

We demonstrate localization of a single fluorescent emitter using optical properties similar to tissue in Fig. 4. For simplicity, we assume a two-dimensional geometry and a region of interest of length 32×l* along x and of length 32×l* along z, where the mean free path length l* = 3D. Extension to three dimensions is unnecessary to demonstrate the method, but it is straightforward. We place a single source and seven detectors along the boundary of the region of interest, as seen in Fig. 4. We use a discretized grid with N_x = 64 points in the x dimension and N_z = 64 points in the z dimension. The position with lowest cost in Fig. 4(b) is ${\widehat{\mathbf{r}}}_f$, and the value of ${\tilde{\eta }}_k$ at ${\widehat{\mathbf{r}}}_{f_k}$ in Fig. 4(a) is ${\widehat{\eta }}_k$. Here, ${\widehat{\mathbf{r}}}_{f_k}=(x,z)=(15.24,15.75)\times l^{*}$ and $\widehat{\eta }=0.0999{\text{mm}}^{-1}$, which are close to the true values given in the Fig. 4 caption. We calculated the localization error as $\left[\left({\mathbf{r}}_{f_k}-{\widehat{\mathbf{r}}}_{f_k}\right)/{\mathbf{r}}_{f_k}\times 100\right]\%$. The results can be extrapolated to three dimensions, and the problem is scalable in the amount of scatter.

Fig. 4.

Localization of a single fluorescent emitter (K = 1) using an (x, z) coordinate system. 1 source (green) and 7 detectors (red) are placed at the boundary of a square of side length 32l*. The optical properties are the same as in Fig. 1, where l* = 0.5 mm, and we assume an SNR of 30 dB. A fluorescent emitter with η1 = 0.1 mm⁻¹ was placed at x = 15.26 × l* and z = 15.56 × l*. The simulated noisy data is the same as the first set of curves at τ1 shown in Fig. 3(b). (a) Yield ${\tilde{\eta }}_k\left({\mathbf{r}}_{f_k}\right)$ from (18) plotted over the region of interest. (b) Cost $c_k\left({\mathbf{r}}_{f_k},{\tau }_k\right)$ from (19) plotted over the region of interest. The position with lowest cost in (b) is ${\widehat{\mathbf{r}}}_{f_k}$, and the value of ${\tilde{\eta }}_k$ at ${\widehat{\mathbf{r}}}_{f_k}$ in (a) is ${\widehat{\eta }}_k$ Here, ${\widehat{\mathbf{r}}}_{f_k}=(x,z)=(15.24,15.75)\times l^{*}$ and ${\widehat{\eta }}_k=0.0999{\text{mm}}^{-1}$. The percent errors in the estimated x and z positions are 0.143% and 1.196%, respectively. The discretization of the region of interest is a primary contributor to the estimation error.

B. Localization with Multiresolution Analysis

We introduce a method to simultaneously reduce the computation time and improve the accuracy of the localization by incorporating a hierarchy of discretization grids into the localization. We refer to this as a multiresolution analysis (MRA) method [27], [31], and it is demonstrated in Fig. 5. First, $c_k\left({\mathbf{r}}_{f_k},{\tau }_k\right)$ is calculated using (18) and (19) on a coarse grid over the entire region of interest. Then, $c_k\left({\mathbf{r}}_{f_k},{\tau }_k\right)$ is iteratively calculated on successively smaller and finer grids, such that each new grid contains the point of minimum cost of the current iteration. This procedure is repeated until convergence, which, in our case, was two successive grids where the change in the minimum cost was less than 1%, but not equal to zero. Importantly, not converging when the change in cost is zero avoids the situation where the same point happens to have the lowest cost on two grids.

Fig. 5.

Localization with MRA of the single fluorescent emitter in Fig. 4. The cost is calculated using (18) and (19) on progressively finer grids, where each new grid contains the region of smallest cost. Here, ${\widehat{\mathbf{r}}}_{f_k}=(x,z)=(15.27,15.57)\times l^{*}$ and ${\widehat{\eta }}_k=0.1003{\text{mm}}^{-1}$. The percent errors in the estimated x and z positions are 0.037% and 0.066%, respectively. The discretization error has been reduced, especially for the z coordinate. The number of positions where the cost must be calculated has also been reduced, decreasing the computation time. The reduction is even greater when extrapolated to 3D.

Pseudocode demonstrating the localization of multiple emitters is shown in Fig. 6 for a window length of w = 3T. The MRA method is used to calculate ${\widehat{\theta }}_k$, where the number of multiresolution iterations is I. We found that the full MRA was not needed to calculate ${\widehat{\theta }}_{\text{temp}}$ in order to determine $\tilde{q}$; calculating ${\widehat{\theta }}_{\text{temp}}$ on only the first coarse grid was sufficient, saving significant computation time.

Fig. 6.

Pseudocode for the temporal localization algorithm using MRA and a window length of w = 3T. Note that the detection step can be very fast since $\tilde{q}$ is calculated using only the first coarse grid. For the definitions of $\tilde{q}$, $k_{P_F}$, and h, see Appendix B.

C. Probability of Successful Localization

Our method assumes that there is significant temporal separation τ_k between fluorescence signals. Here, we explore that assumption and develop a stochastic model to estimate the probability that it holds.

In order to be able to distinguish between the fluorescence impulse responses from two fluorophores, their peaks must be separated by a significant enough amount of time τ_k. We developed a framework to estimate a conservative value of τ_k such that, if all impulse responses are separated by this amount, the algorithm will run correctly. For now, we assume a geometry described in Fig. 7. As depicted there, R_f is the farthest that a fluorophore (for example, f₁ in the figure) can be from the center of the detector array, and R_f+R_d is the longest possible geometrical distance between a fluorophore in the region of interest and a detector in the array. If photons travel at an average speed of ⟨v⟩ through the scattering medium, then it would take, on average, $\frac{1}{\langle v\rangle }\left(R_f+R_d\right)$ seconds to guarantee that a fluorophore’s impulse response reaches all detectors.

Fig. 7.

Geometry used for our stochastic analysis of the requirements on the temporal separation τ_k. We assume a circular region of interest, with radius R_f. We also assume that all of the detectors are gathered in the center of the region of interest, in a circular array of radius R_d. Two example fluorophores are labeled: f₁, which is far from the detector array, and f₂, which is close to it. Let us denote the number of total fluorophores in the region of interest as N_f. That allows us to define the average fluorophore density ρ_f with the equation $N_f={\rho }_f\pi R_f^2$.

However, if another impulse response was to begin after this amount of time, for example fluorophore f₂ in Fig. 7, the peaks of the two impulse responses might still coincide. If we were to wait an additional amount of time equal to the full-width-at-half-max (FWHM) of the widest-possible impulse response, then the peaks would be sufficiently separated in time that our algorithm could work properly, no matter the locations of the two fluorophores in question. Let us denote this additional amount of time as Δt_FWHM,max This gives us the conservative value of

$${\tau }_k=\frac{1}{\langle v\rangle }\left(R_f+R_d\right)+\mathrm{\Delta }{t}_{\text{FWHM},\text{max}}.$$ (22)

Next, we develop a model for estimating the probability of achieving this temporal separation τ_k. Let us model the fluorophore emission times as independent and identically distributed exponential random variables, with rate parameter λ_f emissions per second. This means the fluorophore emissions can be modeled as independent Poisson processes, which is consistent with the shot-noise-limited measurement model described in Section II-E. If emissions are measured over a time interval Δt, then the number of emissions that any given fluorophore gives off is distributed as a Poisson random variable, with rate parameter λ_fΔt. The probability mass function for the number of emissions in this time interval is then

$$\mathrm{Poisson}\left({\lambda }_f\mathrm{\Delta }t\right)~\frac{{\left({\lambda }_f\mathrm{\Delta }t\right)}^n}{n!}{e}^{-\left({\lambda }_f\mathrm{\Delta }t\right)}.$$ (23)

Now, suppose there are N_f fluorophores in the region of interest, as described in Fig. 7. By merging these N_f independent and identically distributed Poisson processes, we have that the number of total fluorophore emissions during some time interval Δt is distributed as a Poisson random variable, with rate parameter N_fλ_fΔt. Let us denote this total number of emissions as F(Δt). Then, if we denote as F₁ the event that no more than one fluorophore emission occurs in the time interval Δt, its probability is

$$\begin{gathered} P\left\{F_1\right\}=P\{F(\mathrm{\Delta }t)=0\cup F(\mathrm{\Delta }t)=1\} \\ =\left(1+N_f{\lambda }_f\mathrm{\Delta }t\right)e^{-\left(N_f{\lambda }_f\mathrm{\Delta }t\right)}. \end{gathered}$$ (24)

The value given in (22) for τ_k is a conservative one; having fluorophore emissions separated by this amount of time is a sufficient condition for the algorithm to work correctly, but not a necessary condition. Still, suppose we require enough temporal separation of fluorophore emissions such that each measured time interval contains at most one event. That is, let us conservatively assign Δt = τ_k. If S is the event that the algorithm is able to work correctly, then we can bound the probability of this event as

$$P\left\{S\right\}\ge \left(1+{\lambda }_F\right)e^{-{\lambda }_F},$$ (25)

where

$$\begin{gathered} {\lambda }_F=N_f{\lambda }_f\mathrm{\Delta }t \\ =N_f{\lambda }_f{\tau }_k \\ ={\rho }_f\pi R_f^2{\lambda }_f\left(\frac{R_f+R_d}{\langle v\rangle }+\mathrm{\Delta }{t}_{\text{FWHM},\text{max}}\right). \end{gathered}$$ (26)

Equation (25) shows that we can increase the probability of successful localization by decreasing λ_F, as defined in (26). Of course, we are free to choose a measurement interval Δt < τ_k, which would increase P{S} further by decreasing λ_F. Indeed, the lower bound on Δt is limited only by one’s detection system and the noise involved, thus linking P{S} to the SNR that one is able to achieve. Additionally, if decreasing the average fluorophore emission rate λ_f is too difficult for one’s application, it may be easier to decrease the emitter density ρ_f instead. The average speed ⟨v⟩ is material- and geometry-dependent, so that may be harder to affect. Importantly, λ_F depends cubically on the radius R_f of the region of interest, but this is often simple to adjust. While (25) and (26) only provide an approximation for the probability of success of our localization algorithm, they are nonetheless useful for analyzing it and how it relates to the experimental setup.

IV. Results

We show localization of four different fluorescent emitters separated by varying delays. We use the same geometry and optical parameters as in Fig. 4, except we place four emitters within the medium instead of one.

Figure 8(a) shows the problem geometry as in Fig. 4, where the positions of the excitation source (green), detectors (red), and fluorophores (cyan) are shown. Figure 8(b) shows the simulated noisy data from the four emitters with delays of 0.9, 3.75, 7.5, and 8.6 ns. The first two emitters are temporally separated, while the second two have a significant temporal overlap. Figure 8(c) plots the actual and predicted locations of the emitters, which are separated by 0.2×l*. Figure 8(d) shows the yield η (proportional to concentration) experimental errors, calculated as $\left[\left({\eta }_k-\widehat{\eta }k\right)/{\eta }_k\times 100\right]$. For the multiresolution operation, we used N_x = N_z = 5 equally spaced points for each grid, the same as what was used in Fig. 5. The first coarse grid was defined over the entire square region of interest from Fig. 4, and each finer square grid was centered at the estimate given by (20) and extended a distance slightly greater than the previous grid spacing along the x and z directions (similar to Fig. 5). Both the position and η were predicted with high accuracy, even for the case when there is overlap between the temporal signals. The accuracy of the localization is the focus of Section IV-A.

Fig. 8.

Localization of four fluorescent emitters at different positions with different delays and yields using the same optical parameters as in Fig. 1 and the same geometry as in Fig. 4. The true parameters describing the fluorescent sources are (x, z, η, τ) = (15l*, 15l*, 0.1, 5T), (15l*, 15.2l*, 0.15, 20T), (15.2l*, 15.0l*, 0.075, 40T), and (15.2l*, 15.2l*, 0.05, 46T), where l* = 0.5 mm and T = 0.19 ns. All of these parameters are estimated by the algorithm. We assume the fluorescence lifetime γ_f is known. (a) Problem geometry as in Fig. 4, where the positions of the excitation source (green), detectors (red), and fluorescent emitters (cyan) are plotted. (b) Detected fluorescence temporal profile. One source (Q = 1) and seven detectors (M = 7) give 7 measurements y_qm. The 7 different symbols and their corresponding colors represent different source-detector measurement pairs. The data was generated using the true parameters with simulated noise for an SNR of 30 dB. (c) True positions of the fluorescent emitters r_k and the estimated positions ${\widehat{\mathbf{r}}}_{f_k}$ determined by the localization algorithm. Note the accuracy of the estimated positions. (d) Yield η_k experimental errors. Labels one to four correspond to delays from shortest to longest. Each fluorescent emitter was successfully localized, even for the case when there is overlap between the temporal signals.

A. Localization for High Spatial Resolution

We assume insignificant background signal and that the fluorophores do not diffuse or change positions significantly during the integration time. In this case, the localization uncertainty is dominated by measurement noise and each ${\widehat{\mathbf{r}}}_{k_f}$ coordinate x and z has a Gaussian probability distribution characterized by the standard deviations σ_x and σ_z [32]. These standard deviations are commonly used to characterize the spatial precision of localization [33]. Here, we show the capability of localization to extract high-spatial-resolution information by generating these statistics from numerical calculations.

We iteratively localize the single emitter in Fig. 4 using MRA, and the results are shown in Fig. 9. During each iteration, random noise was added to the forward calculation in order to generate independent simulated measurements. Both σ_x and σ_z were then calculated from the localized positions and used to plot ellipses with major and minor axis equal to 4σ_x or 4σ_z. The MRA method described in Section III-B was important in reducing the time required to compute these statistics, as discussed in Sect. IV-B.

Fig. 9.

Localization uncertainty of a single fluorescent emitter using the same optical parameters as in Fig. 1 and the same geometry as in Fig. 4 with Q = 1. The fluorescent emitter location was estimated 150 times using noisy simulated independent data sets. The true location is the black point. The ellipses have major and minor axes of length 4σ_x or 4σ_z, such that they contain 95% of the $\widehat{x}$ and $\widehat{z}$ positions. The center points of the ellipses are the mean of the $\widehat{x}$ and $\widehat{z}$ positions. (a) Localization uncertainty for different SNR with M = 7 and w = 3T. Blue, green, and red correspond to SNR of 30, 20, and 10 dB, respectively. (b) Enlarged version of (a) to show the mean values. (c) Localization uncertainty for different numbers of detectors M with an SNR of 30 dB and w = 3T. Red, green, and blue correspond to M = 7, M = 31, and M = 50. (d) Enlarged version of (c) to show the mean values. (e) Localization uncertainty for different window lengths w, 30 dB SNR, and M = 7. Red, green, and blue correspond to windows w = 32T, 17T, and 2T, where T = 0.19 ns and t_max = 64T. (f) Enlarged version of (e) to show the mean values. The ellipses are not circles because the fluorescent emitter is not located at the center of the medium and equidistant to all detectors. Note that the fluorescent emitter can be accurately localized even with low SNR, few detectors, and a short window w.

Figures 9(a) and (b) show localization uncertainly statistics for different SNR, where the noise was added as described by (12) [26]. Even for a low SNR of 10 dB, the location of the fluorescent emitter can be estimated with much higher accuracy compared to traditional diffusive imaging methods, as seen for example in Fig. 1. Figure 9(c) and (d) show localization uncertainly statistics for different numbers of detectors M. The detectors were distributed as in Fig. 4, and the statistics depend on the spatial support. Compared to volumetric image reconstruction with FDOT, little information, or fewer detectors are needed to localize the fluorophore.

Considering Fig. 6, it is not clear how or if the localization uncertainty depends on the window length w. To study this, we build up localization uncertainly statistics for different w with a constant SNR of 30 dB and M = 7. The results are shown in Fig. 9(e) and (f). For all results in this figure (and also in Fig. 8), the time axis starts at t = 0 and continues in increments of T = 0.19 ns for 64 increments, giving t_max = 12.16 ns. As can be seen in Fig. 9(e) and (f), the localization statistics change slightly with window size, but they are still highly accurate even for the shortest window of length 2T. As indicated in Fig. 9(e) and in enlarged form in Fig. 9(f), increasing the window length allows more data points over time to be used and provides a reduced localization uncertainty.

The localization uncertainty of the emitters in Fig. 8(c) is described by the blue ellipse in Fig. 9(b). The extension to imaging is straightforward. If we consider an image which is made up of voxels, and each voxel can be individually localized, then Fig. 9 describes the resolution limit of the image. Comparing to Fig. 1, where the resolution limit is less than about 8.5l*, the resolution limit from Fig. 9(b), or the minimum distance between two emitters such that they can both be accurately localized, is about 0.1l* or 50 μm. This is a remarkable spatial resolution for deep-tissue optical imaging.

B. Run Time Analysis

We investigate the computational advantage of the MRA localization algorithm shown in Fig. 6 compared to matching pursuit, an established method for sparse approximation. The run time of matching pursuit is (number of iterations) × (time per iteration) [30], where on average there are K iterations. We let $N_{x_{MP}}$ and $N_{z_{MP}}$ be the number of grid points along the x and z dimensions for matching pursuit, where $N_{x_{MP}}$ and $N_{z_{MP}}$ must be very large in order to achieve high spatial resolution. To minimize (16), matching pursuit must calculate $\mathcal{O}\left(N_{x_{MP}}{N}_{z_{MP}}N\right)$ inner products since t_max = NT, and each inner product involves $\mathcal{O}(QMN)$ multiply-add operations. Therefore, the expected run time of matching pursuit is $\mathcal{O}\left(QM{N}_{x_{MP}}{N}_{z_{MP}}{N}^{2}K\right)$. For the algorithm in Fig. 6, both the threshold tests and the localizations require repeated evaluation of grids, each representing progressively finer resolution. Each of these grids has the same number of points as the first (lowest-resolution) one. Suppose this grid has N_MG points. Since each of these grid points could potentially be the one visited in a second iteration of MRA, there are $N_{MG}^2$ points represented in a two-iteration MRA algorithm. Since we perform I multiresolution iterations and eventually reach $N_{x_{MP}}{N}_{z_{MP}}$ points, it must be that $N_{MG}^I=N_{x_{MP}}{N}_{z_{MP}}$, or $N_{MG}={\left(N_{x_{MP}}{N}_{z_{MP}}\right)}^{(1/I)}$. So, the time for the scanning threshold tests is $\mathcal{O}\left[QM{\left(N_{x_{MP}}{N}_{z_{MP}}\right)}^{(1/I)}N\right]$. We expect there to be K localizations on average, so the time spent for localization is $\mathcal{O}\left[KQM{\left(N_{x_{MP}}{N}_{z_{MP}}\right)}^{(1/I)}I\right]$, using similar analysis as before for the number of points in each multiresolution iteration. Therefore, the expected run time of the Fig. 6 algorithm is $\mathcal{O}\left[QM{\left(N_{x_{MP}}{N}_{z_{MP}}\right)}^{(1/I)}(N+KI)\right]$. We summarize these results in Table I.

TABLE I.

Run time comparison with matching pursuit

	Run Time
Matching Pursuit	$\mathcal{O}\left(QM{N}_{x_{MP}}{N}_{z_{MP}}{N}^{2}K\right)$
Fig. 6 Algorithm	$\mathcal{O}[QM{\left(N_{x_{MP}}{N}_{z_{MP}}\right)}^{(1/I)}(N+KI)]$

There are some clear advantages to the MRA algorithm presented in Fig. 6. First, the run time is linear in N instead of quadratic. Second, the grid size $\left({\left(N_{x_{MP}}{N}_{z_{MP}}\right)}^{(1/I)}\right)$ is much smaller because of the MRA method. In situations where a high-resolution image is desired, $N_{x_{MP}}$, $N_{z_{MP}}$, and N must all be very large, severely limiting the practicality of matching pursuit. We show example run times in Fig. 10 using a 3.47 GHz Intel X5690 with 96 GB RAM. It is clear from the plot that for large N, matching pursuit is quadratic in N while the new localization method remains linear. For matching pursuit, the amount of memory needed to store the large dictionary is $8\times \left(QM{N}_{x_{MP}}{N}_{z_{MP}}{N}^2\right)$ bytes, and using the parameters given in Fig. 10, this gives an ideal maximum value of N = 855. In practice, more than the dictionary must be saved in RAM, and in Fig. 10 the maximum value of N possible for matching pursuit was found to be 481.

Fig. 10.

Run times of matching pursuit (red) and the Fig. 6 algorithm (blue) for varying N, where tmax = NT. We use the same optical properties as Fig. 1, and the same geometry as Fig. 4. The computation time required to generate and load the dictionary by matching pursuit is not included. Including this almost triples the run times shown in the plot.

V. Discussion

We have shown that temporal fluorescence data can be used to precisely localize a set of point fluorescent emitters in a homogeneous heavily scattering environment. For an inhomogeneous domain, the background scattering parameters could be estimated from prior measurements using established reconstruction methods [6]. If substantial background signal is present, for example, from tissue autofluorescence, it could be subtracted from the measured data in a calibration step, or incorporated into the forward model, (11), through the addition of a background source term. We assumed a fixed and known fluorescence lifetime for all reporters. While this may be appropriate for some applications, if there should be variations in lifetime because of the local (chemical) environment, lifetime could be estimated along with position, in a revised interpretation of the cost function in (16).

We have analyzed several requirements of being able to successfully detect and localize fluorescent emitters. Appendix B develops a method for detecting individual emitters during a temporal scan, and its success depends on the yield, η, as well as the SNR. For this detection method to work, fluorescent emissions must be separated in time by at least the window duration, w, a requirement that is explored in Section III-C. It is important to note that this requirement for a separation in time places an implicit constraint on the sampling period. Namely, the sampling period must be short enough so as to not hide this existing temporal separation. Furthermore, for such emitters to be fully distinguishable, their spatial locations must be separated by at least the localization uncertainty described in Section IV-A, otherwise, while they may be detectable, they will be mistaken for the same emitter. If all of these conditions are met, then all fluorescent emitters will be detectable and distinguishable from one another. In the case where the same emitter is localized several times because its light is emitted at multiple delays separated by at least the window length, the emitter could be identified through calculation of its localization uncertainty.

We compare our method to Matching Pursuit, and extensions to that method do exist, one being Orthogonal Matching Pursuit [30] (OMP). Rather than calculating basis function weights once, OMP recalculates them upon every iteration, thereby mitigating error accumulation. Taking inspiration from OMP, we could reduce potential error accumulation in the estimated η_k by recalculating all of their values upon each successive detection of a fluorescent emitter.

Other super-resolution methods have been developed for improving the spatial resolution beyond the diffraction limit in microscopy. Fundamentally, imaging methods can surpass resolution limits with the addition of prior information to compensate for the information that is lost due to attenuation or randomization of the signal. For example, structured illumination microscopy (SIM) breaks the resolution limit through spatial modulation of the excitation light, and can be further improved by leveraging the nonlinear dependence of the fluorescent emission rate in the saturation regime [34]. Stimulated emission depletion (STED) [35] forms a smaller effective point spread function (PSF) by saturating fluorophores at the periphery of the focal point. Other techniques such as photoactivated localization microscopy (PALM) [36], fluorescence PALM (fPALM) [37], and stochastic optical reconstruction microscopy (STORM) [38] are able to localize switchable fluorescent molecules by distinguishing the emission between their fluorescent and non-fluorescent states. These are all super-resolution methods that achieve position information beyond that directly represented by the point spread function. The shared use of temporal switching of fluorescence states in fluorescent molecules makes our method analogous to PALM, fPALM, and STORM. However, our incorporation of a forward model, such as the coupled diffusion model for scattered light we use here, allows imaging through scatter. Moreover, our super-resolution imaging method can be applied to a wider variety of applications that employ forward models, such as photoacoustic tomography [39], electrical impedance tomography [40], and microwave imaging [41].

VI. Conclusion

We have presented an approach for the fast localization of multiple fluorescent emitters within a highly scattering medium that takes advantage of variations in temporal delays between responses. The method allows formation of super-resolution optical images from highly scattered light. We demonstrated through simulations that MRA and temporal scanning can significantly reduce the computational burden compared to matching pursuit. The geometries can be scaled and other forward models can be used, allowing a broad range of applications, most notably, those related to in vivo imaging. While earlier experimental work with a small single fluorescence inhomogeneity led to the position that a spatial resolution of ten microns or so through centimeters of tissue is possible, the influence of practical issues needs to be appraised, including variations in fluorescence lifetime with position and an inhomogeneous background medium, as would exist in whole-brain imaging due to the different optical properties of the blood. An experimental study relative to the pursuit of an application would be the next step that is motivated by this work.

One important potential application of our method is brain imaging, because the response of fluorescent contrast agents due to neurons or groups of neurons firing may follow a model similar to in (8) [42], [43]. If this is found to be true, and a reasonable SNR can be achieved, then it would be possible to form images of the whole-brain or brain surface with high spatial and temporal resolution. These images could be used to form correlation maps of brain activity, useful for diagnosing and studying neurological diseases such as Parkinson’s or Alzheimer’s disease, and developing treatments [44]. The experiment could be performed in the time domain with fast and sensitive detectors, or with an integration time equal to w if the delays τ_k are sufficiently large. Data captured with a time-gated intensified camera over an integration time of w could be used to temporally separate overlapping neuron responses in both time and space. It is possible to control the switching of a neuron using optical [45] or electrical [46] stimulation, and this could allow initiation control and aid experimental validation of our localization approach. Pace-making neurons, which fire periodically at 2–3 Hz [47], may also prove useful for verification or for further constraining the methods. Absorption instead of fluorescent emitters could be localized [20], and the forward model could incorporate multiple point absorbers or emitters instead of just one [25]. There are other biomedical applications, such as the extraction of kinetic information [48], [49], as well as other physical problems where fast localization benefits the extraction of physical information.

Semiconductor quantum dots (QDs) are popular in fluorescence imaging due to their convenient absorption and emission spectrum properties and greater brightness compared to most organic dyes. However, they commonly exhibit an inherent intermittence in their emission (blinking) [50], a property that is often considered a nuisance, and work has been done to suppress it [51], [52]. This blinking could be exploited, and it has been shown that blinking can be affected by variables such as the excitation power, the shell material about the QD, and environment conditions such as the temperature and surrounding medium [53]. Such inherent blinking of QDs could allow our localization method to be used in applications where the required temporal separation of fluorescent emissions would not otherwise occur.

Biography

Brian Z. Bentz received his B.S. and Ph.D. in 2011 and 2017, respectively, from Purdue University, West Lafayette, Indiana, both in electrical and computer engineering. He was a Research Assistant with the School of Electrical and Computer Engineering at Purdue University from 2012 to 2017. In 2017, he joined Sandia National Laboratories, Albuquerque, New Mexico.

Dergan Lin received his M.S. in 2012 from Purdue University in electrical and computer engineering. He is currently wrapping up his Ph.D. thesis while working as an independent video game developer with his brother.

Justin A. Patel received his B.S. in 2016 from the University of California, Berkeley in electrical engineering and computer science. He currently studies at Purdue University in West Lafayette, Indiana, where he is working towards a Ph.D. in electrical and computer engineering, and works there as a Research Assistant with the School of Electrical and Computer Engineering.

Kevin J. Webb is a professor in the School of Electrical an Computer Engineering at Purdue University, West Lafayette, Indiana, where he has been on the faculty since 1990. Prior to this, he was on the faculty of the University of Maryland, College Park, following completion of his Ph.D. at the University of Illinois, Urbana-Champaign, in 1984. He is a Fellow of the IEEE, the OSA, and APS.

Appendix A

Scaling the Diffusion Equation

In this Appendix we show scaling of the spatial dimensions of the diffusion equation. Considering (1) and (2) along with (6), it is clear that the dimensions of the diffusion problem can be scaled. We let μ_a = 1/l_a, where l_a is the distance between absorption events, and μ_s = 1/l_s where l_s is the distance between scattering events. Then these parameters can be scaled by a parameter α_s, resulting in $\tilde{\mathbf{r}}={\alpha }_s\mathbf{r}$, ${\tilde{\mu }}_a={\mu }_a/{\alpha }_s$, ${\tilde{\mu }}_s={\mu }_s/{\alpha }_s$, $\tilde{g}=g$, and $\tilde{c}={\alpha }_{s}c$, giving $\tilde{D}={\alpha }_{s}D$. This scaling allows a material with widely spaced scatterers to be modeled by a more densely packed material or vice versa. Significant scaling of the velocity or refractive index is not realistic, however in the case of unmodulated light (ω = 0 in the frequency domain) this is not a problem. Alternatively, the time axis can be scaled as $\tilde{t}={\alpha }_{s}t$ instead of scaling the velocity. Figure 11 shows scaling of (7) with α_s = 1/1000.

Fig. 11.

Scaling of the current density from (7) detected through cloud. In (a), the cloud is 1 km thick and μa = 0.1 km⁻¹, ${\mu }_s^{\prime }=10{\text{km}}^{-1}$, and n = 1.5, which are typical measured values [54]. In (b), α_s = 1/1000 scaling is applied with the result that ${\tilde{\mu }}_a=0.1{\text{m}}^{-1}$, ${\tilde{\mu }}_s^{\prime }=10{\text{m}}^{-1}$, a thickness of 1 m, and we keep n = 1.5. Note that the time axis is scaled by αs and the current density J is scaled by ${\alpha }_s^{-3}$. If we had scaled the refractive index as $\tilde{n}=n/{\alpha }_s$, then the current density J would be scaled by ${\alpha }_S^{-2}$ and the time axis would not be scaled.

Appendix B

Detection

In this Appendix we develop the method used to detect a single fluorescent emitter. The model is used to determine the threshold needed for a decision to localize an emitter during the temporal scan. Following Milstein et al. [19], a fluorescent emitter is detected for a specified false-alarm rate P_F. We let hypothesis H₀ correspond to the absence of a fluorescent emitter, and we let $H_{1,{\theta }_k}$ correspond to the presence of a fluorescent emitter that is parameterized by the vector ${\theta }_k={\left[\begin{array}{cc}{\mathbf{r}}_{f_k}& {\eta }_k\end{array}\right]}^T$. Then the densities for measurement vector y_k of length P under both hypotheses are given by

$$p_0\left({\mathbf{y}}_k\right)=\frac{1}{\sqrt{{(2\pi )}^P\left|{\mathrm{\Upsilon }}_k\right|}}\exp \left(-\frac{1}{2}{\Vert {\mathbf{y}}_k\Vert }_{{\mathrm{\Upsilon }}_k^{-1}}^2\right)$$ (27)

$$p_{1,{\theta }_k}\left({\mathbf{y}}_k\right)=\frac{1}{\sqrt{{(2\pi )}^P\left|{\mathrm{\Upsilon }}_k\right|}}\exp \left(-\frac{1}{2}{\Vert {\mathbf{y}}_k-{\eta }_k{\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)\Vert }_{{\mathrm{\Upsilon }}_k^{-1}}^2\right).$$ (28)

Then, we form the log likelihood ratio

$$L\left({\mathbf{y}}_k,{\theta }_k\right)=\ln \frac{p_{1,{\theta }_k}\left({\mathbf{y}}_k\right)}{p_0\left({\mathbf{y}}_k\right)}={\mathbf{h}}^T\left({\theta }_k\right){\mathbf{y}}_k-c\left({\theta }_k\right),$$ (29)

where ${\mathbf{h}}^T\left({\theta }_k\right)={\eta }_k{\mathbf{f}}_k^T\left({\mathbf{r}}_{f_k}\right){\mathrm{\Upsilon }}_k^{-1}$ and $c\left({\theta }_k\right)=\frac{1}{2}{\eta }_k^2{\mathbf{f}}_k^T\left({\mathbf{r}}_{f_k}\right){\mathrm{\Upsilon }}_k^{-1}{\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)$. By the Neyman-Pearson lemma, the likelihood ratio test (LRT) produces the highest probability of detection for a specified false alarm rate of P_F. For this paper, we used P_F = 0.03 to generate all related results. We declare an emitter present when

$$L\left({\mathbf{y}}_k,{\theta }_k\right)>{\tilde{k}}_{P_F}\left({\theta }_k\right).$$ (30)

where ${\tilde{k}}_{P_F}\left({\theta }_k\right)$ is a threshold determined by P_F. The LRT suggests that if the correlation between the data y_k and the model ${\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)$ is above a threshold, an emitter exists.

We define $\tilde{q}={\mathbf{h}}^T\left({\theta }_k\right){\mathbf{y}}_k$ as the decision statistic, and we find that it has a normal distribution under each of the two hypotheses, such that $\left(\tilde{q}|H_0\right)=N\left(0,{\sigma }_{\tilde{q}}^2\right)$ and $\left(\tilde{q}|H_{1,{\theta }_k}\right)=N\left({\mu }_{\tilde{q}},{\sigma }_{\tilde{q}}^2\right)$, where the mean ${\mu }_{\tilde{q}}={\eta }_k^2{\mathbf{f}}_k^T\left({\mathbf{r}}_{f_k}\right){\mathrm{\Upsilon }}_k^{-1}{\mathbf{f}}_k\left({\mathbf{r}}_{f_k}\right)$ and the variance ${\sigma }_{\tilde{q}}^2={\text{h}}^T\left({\theta }_k\right){\mathrm{\Upsilon }}_k\text{h}\left({\theta }_k\right)$. Equation (30) is then equivalent to $\tilde{q}>k_{P_F}\left({\theta }_k\right)$, where $k_{P_F}\left({\theta }_k\right)$ is a modified threshold that we calculate below.

By definition, the false alarm rate P_F is given by

$$P_F={\int }_{k_{P_F}\left({\theta }_k\right)}^{\infty }{p}_0(\tilde{q})d\tilde{q}=\frac{1}{2}\mathrm{erfc}\left(\frac{k_{P_F}\left({\theta }_k\right)}{{\sigma }_{\tilde{q}}}\right),$$ (31)

where we used the substitution $u=1/\left(\sqrt{2}{\sigma }_{\tilde{q}}\right)$ and

$$\mathrm{erfc}(\mathbf{r})={\int }_{\mathbf{r}}^{\infty }\frac{2}{\sqrt{\pi }}\exp \left(-u^2\right)du,$$ (32)

is the complementary error function. The threshold $k_{P_F}\left({\theta }_k\right)$ can then be calculated as

$$k_{P_F}\left({\theta }_k\right)=\sqrt{2}{\sigma }_{\tilde{q}}{\mathrm{erfc}}^{-1}\left(2P_F\right),$$ (33)

where erfc⁻¹(·) is the inverse complementary error function. Using this threshold, the probability of detection P_D can be computed as

$$P_D={\int }_{k_{P_F}\left({\theta }_k\right)}^{\infty }{p}_{1,{\theta }_k}(\tilde{q})d\tilde{q}=\frac{1}{2}\mathrm{erfc}\left(\frac{k_{P_F}\left({\theta }_k\right)-{\mu }_{\tilde{q}}}{\sqrt{2}{\sigma }_{\tilde{q}}}\right).$$ (34)

The LRT assumes θ_k is known, but in practice it must be estimated. We use the generalized likelihood ratio test (GLRT), where θ_k in the LRT is replaced by the ML estimate ${\widehat{\theta }}_k=\arg {\max }_{{\theta }_k}{p}_{1,{\theta }_k}(\mathbf{y})$. Calculating ${\widehat{\theta }}_k$ constitutes localization.