Evaluating online filtering algorithms to enhance dynamic multispectral optoacoustic tomography

Graphical abstract

Highlights

• •Simple $\alpha$ and $\alpha \beta$ filters effectively suppress noise in dynamic spectral images.
• •A multispectral extension of the Shepp-Logan phantom was developed.
• •Practical application is demonstrated in distinct renal cell carcinoma models.
• •$\alpha \beta$ filters enabled effective separation of gas breathing challenge response.

Abstract

Multispectral optoacoustic tomography (MSOT) is an emerging imaging modality, which is able to capture data at high spatiotemporal resolution using rapid tuning of the excitation laser wavelength. However, owing to the necessity of imaging one wavelength at a time to the exclusion of others, forming a complete multispectral image requires multiple excitations over time, which may introduce aliasing due to underlying spectral dynamics or noise in the data. In order to mitigate this limitation, we have applied kinematic $\alpha$ and $\alpha \beta$ filters to multispectral time series, providing an estimate of the underlying multispectral image at every point in time throughout data acquisition. We demonstrate the efficacy of these methods in suppressing the inter-frame noise present in dynamic multispectral image time courses using a multispectral Shepp-Logan phantom and mice bearing distinct renal cell carcinoma tumors. The gains in signal to noise ratio provided by these filters enable higher-fidelity downstream analysis such as spectral unmixing and improved hypothesis testing in quantifying the onset of signal changes during an oxygen gas challenge.

1. Introduction

Imaging has become a powerful tool in diverse fields such as medical diagnostics, high-throughput screening, and remote sensing [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11]]. The imaging toolkit is becoming progressively more sophisticated, moving beyond spatial discrimination with additional dimensions: Spectral imaging allows the capture of multiple imaging channels, in the hope of using the spectral fingerprints of underlying components to learn more about the system under study [1,[7], [8], [9], [10],[12], [13], [14], [15], [16], [17], [18], [19]]; and dynamic imaging captures many single images over time in order to resolve changes in the observed system [12,13,[20], [21], [22], [23], [24]]. Imaging methods are subject to inherent trade-offs between spatial, spectral, and temporal resolution, as well as the discrimination of signal versus noise. The quality of data is fixed once the data have been acquired by instrument hardware, but it is possible to take advantage of the inherent coherence and structure within a dataset in order to mitigate the effects of noise and finite sampling.

Multispectral optoacoustic tomography (MSOT) is an imaging modality, characterized by the use of optical excitation and ultrasonic detection, which is able to capture spectral images at high spatial and temporal resolution [7,17,20,22]. MSOT intrinsically acquires data based on sequential wavelength excitation such that images are captured one channel at a time. There may therefore be substantial spectral distortion due to aliasing in imaging scenarios where the rate of spectral dynamics is comparable to the multispectral sampling rate. Further exacerbating aliasing is the presence of noise, which may vary spatially, temporally, and spectrally, resulting in the reduction of measurement fidelity.

One experimental motif which exemplifies these difficulties is the gas-breathing challenge, wherein the inhaled gas of a subject under observation is varied in order to induce a change in the subject’s blood oxygen saturation (SO₂) [25,26]. In order to determine or estimate the oxygen saturation by MSOT, images must be acquired at multiple wavelengths and spectrally unmixed to provide oxy- and deoxy-hemoglobin images, whereupon the ratio of oxy-hemoglobin to total hemoglobin provides an estimate of the oxygen saturation of blood. The propagation of image noise through this process can result in extremely noisy SO₂ estimates, limiting the amount of useful information which can be extracted from noisy data.

It is therefore desirable to devise a scheme to mitigate the corrupting effects of noise while still accurately capturing dynamic phenomena. Similarly, it is desirable to be able to perform such filtering in an online manner, as data are acquired, such that the filtering may be implemented for real-time imaging or efficient post-acquisition analysis. This last criterion imposes a strong restriction on the complexity of any algorithm, since many photoacoustic imaging systems gather data at rates exceeding 10 Hz ($\text{Δ}t<0.1\text{s})$.

We briefly establish general notation for equations throughout this work: $k$ is the current time index, $x$ refers to the states of interest, here the pixel values for the spectral image; $z_k$ is the acquired data at time $k$, $x_{k|k-1}$ is the estimate of $x$ at time k considering all data up to and including the data at time $k-1$, $r_k$ is the so-called prior residual, and $x_{k|k}$ is the estimate of $x$ after integration of the data acquired at time point $k$.

A simple, efficient approach to recursively estimating the state of a multispectral image is to perform online averaging on a per-channel basis according to the following update equations:

$$r_k=z_k-x_{k|k-1}$$ (1)

$$x_{k|k}=x_{k|k-1}+\left(\frac{1}{k}\right)r_k$$ (2)

For stationary distributions, online averaging performs exactly as the usual average, converging to the true mean as $\mathcal{O}\left(\sqrt{N}\right)$, i.e., the variance of sampling noise is proportional to the square root of the number of samples averaged [27]. For non-stationary distributions such as those encountered in dynamic imaging, $x_{k|k}$ will not, in general, converge to the true mean even after acquisition of an infinite number of samples. In the context of a gas challenge, $x_{k|k}$ will be heavily biased towards the initially acquired data and will not effectively follow dynamics.

To accommodate nonstationary distributions, the $\frac{1}{k}$ term may be replaced with a constant $\alpha \in \left[\mathrm{0,1}\right]$. This gives the averaging process a time-invariant form which may be analyzed as a recursive filter, often referred to as the exponentially weighted moving average or $\alpha$ filter:

$$x_{k|k}=x_{k|k-1}+\alpha r_k$$ (3)

Observation indicates that varying $\alpha$ between 0 and 1 biases the estimate of $x$ towards the previous estimate and the new data, respectively. $\alpha \to 0$ converges to the same structure as the online average as the sampling time tends to infinity. Conversely, $\alpha \to 1$ ignores all previous data in favor of the new data, but without any noise rejection. This trade-off between noise rejection and dynamical sensitivity is irrelevant if there are no underlying dynamics, but for values of $\alpha <1$ the estimate will always lag the true mean of dynamic systems [28]. The limiting case of $\alpha =1$ results in the filter taking on the form of a sliding-window (SW) filter, where old data is entirely overwritten by new data as it is acquired:

$$x_{k|k}=z_k$$ (4)

Clearly, a simple $\alpha$ filter is unable to fully accommodate nonstationary data and is therefore generally inappropriate for dynamic imaging. Many more sophisticated algorithms such as the broad class of Bayesian filters can incorporate large amounts of prior knowledge, including the presence of dynamics, and have exceptional performance in tracking the state of dynamical systems [28,29]. However, Bayesian filters can require large amounts of data to undergo computationally intensive processing and are particularly challenging to apply to spectral images due to the large size of associated datasets.

Though full Bayesian filters are computationally prohibitive, they may be simplified by assuming Gaussian noise statistics and linear dynamics. These so-called Kalman filters operate in a two-stage prediction-correction cycle, wherein the prediction phase propagates the state estimate in time according to the dynamical model and the correction phase incorporates new data observations into the state estimate. Kalman filters have been shown to be useful in integrating multi-domain measurements, including multi-channel images, due to their inherently dynamical nature [[30], [31], [32], [33], [34]]. Nevertheless, the size of the covariance matrices necessary for the operation of Kalman filters requires offline processing or substantial computing resources.

If the assumption is made that the Kalman filter is operating at steady state, it degenerates to a form which is temporally invariant. Further assumption that the state of interest has a known second-order process variance (i.e., that the process noise is in the acceleration term of a Newtonian kinematic model) yields a family of recursive filters indexed by the order of their Taylor series expansion, with associated constant coefficients [35,36]. Here, we describe the form of the 1st and 2nd order filters, the former of which is the previously-described $\alpha$ filter and the latter correspondingly named the $\alpha \beta$ filter.

As before, the state of interest is denoted by $x$, with first temporal derivative given by $\dot{x}$. The measurement noise variance, describing the ‘noisiness’ of each measurement, is denoted by ${\sigma }_n,$ with units determined by the measurement device (here arbitrary units a.u.), and the process noise variance, describing the volatility of the underlying dynamics, as ${\sigma }_w$ with units (a.u./s²). We assume that each pixel evolves independently and thus the filter is applied to each pixel of the acquired image separately.

The filter coefficients may be optimally found using prior knowledge of $\text{Δ}t$, ${\sigma }_n$, and ${\sigma }_w$ via a single scalar parameter, the tracking index:

$$\text{Λ}=\frac{{\sigma }_w{\left(\text{Δ}t\right)}^2}{{\sigma }_n}$$ (5)

We assume that the values of $\text{Δ}t$, ${\sigma }_n$, and ${\sigma }_w$ are homogenous across the entire image such that each image has a single tracking index. In practice, these parameters may vary between channels and over time, which we address in Section 4.

The filtering equations for an $\alpha$ filter are identical to the exponentially weighted moving average described in Eqs. 1 and 3. As the $\alpha$ filter tracks only position information, the prediction step is degenerate and only the correction phase updates the state estimate:

Prediction

$$x_{k|k-1}=x_{k-1|k-1}$$ (6)

Correction

$$x_{k|k}=x_{k|k-1}+{\alpha }_k{r}_k$$ (7)

The filtering equations for an $\alpha \beta$ filter propagate the state estimate over time by assuming constant-velocity dynamics and are:

Prediction

$$x_{k|k-1}=x_{k-1|k-1}+{\dot{x}}_{k-1|k-1}\text{Δ}t$$ (8)

$${\dot{x}}_{k|k-1}={\dot{x}}_{k-1|k-1}$$ (9)

Correction

$$x_{k|k}=x_{k|k-1}+{\alpha }_k{r}_k$$ (10)

$${\dot{x}}_{k|k}={\dot{x}}_{k|k-1}+\frac{{\beta }_k}{\text{Δ}t}{r}_k$$ (11)

The tracking index formalism enables analytic determination of the $\alpha$ and $\alpha \beta$ coefficients (Appendix A) in an optimal manner, and requires knowledge of only three parameters $\text{Δ}t$, ${\sigma }_n$, and ${\sigma }_w$, which may themselves be determined from prior knowledge of the system, or updated recursively by examining the prior and posterior residuals of the filtering process [37]. In environments where the sampling rate is inhomogeneous, such as when oversampling each channel before sampling another, $\text{Δ}t$ is variable. This requires that the tracking index and thus the $\alpha$ and $\beta$ coefficients must be updated at each time point.

We compare the $\alpha$ and $\alpha \beta$ filters against the sliding window (SW) filter equivalent to $\alpha =1$: A circular buffer of data is maintained and updated on a per-channel per-time-point basis as data are acquired, following a ‘sample and hold’ pattern. There is thus no incorporation of previous data into the current estimate, nor is there any accommodation for dynamic evolution of the state.

2. Methods

2.1. Numerical phantom

We developed a dynamic, multispectral numerical phantom to act as a known ground-truth for testing the ability of the various filters to track spectral changes while rejecting noise. The full process of creating the phantom is described in Appendix B and illustrated in Fig. 1. Briefly, a Shepp-Logan phantom [38] was modified to include time-varying concentrations of endmembers with known spectra. Sampling the phantom image at specific wavelengths at each point in time provided a sample train to emulate the acquisition of one channel at a time. We directly sampled the phantom, rather than simulating the photoacoustic imaging process, to limit the possibility of introducing implementation and modeling errors. After sampling, zero-mean white Gaussian noise with standard deviation $\sigma =500a.u.$ was added to each image, corresponding to an SNR of approximately 20. The dynamics of the phantom were designed to mimic a gas challenge due to the variety of states and rates of change which can be present in a single time course.

Fig. 1.

Workflow for creating numerical phantom. A) A Shepp-Logan phantom was created with ellipses defined by various constant values for total hemoglobin [Hb_tot] and time-varying SO₂. The time-varying images of [Hb] and [HbO₂] were then calculated (B) and their spectra used to create time-varying spectral images (C). The spectral image was then sampled at predetermined wavelengths to create a time course of single-wavelength images (D), which were used as input for the filtering methods described in this work.

2.2. Experimental data

Three mice from ongoing studies were used for in vivo validation, with all data acquired using an MSOT InVision 256-TF photoacoustic imaging system [39] (iThera Medical GmbH, Munich, Germany). The water bath used for imaging was maintained at a temperature of 35$°$ C for all experiments. All animal work was conducted in accordance with the UT Southwestern Institutional Animal Care and Use Committee guidelines as well as all superseding federal guidelines. Two distinct wavelength sampling patterns were used to demonstrate the applicability of the filters to general acquisition conditions.

The first mouse was a female BALB/c mouse (UT Southwestern Animal Breeding Core) with a RENCA-luc [40] mouse kidney tumor implanted in the right kidney following published protocols [41], with data acquired 28 days after implantation. Prior to imaging, the mouse was shaved and any remaining hair was removed from the animal using depilation cream. Five wavelengths were sampled [λ = 700, 730, 756, 800, 850 nm] to correspond with major features of the hemoglobin spectrum, including local spectral peaks and the isosbestic point. Each wavelength was oversampled 5 times and averaged prior to image reconstruction, leading to an overall acquisition time of 2.5 s per multispectral sampling cycle. All images were acquired at the same axial cross-section. During imaging, the anesthetized mouse underwent a gas challenge from air-oxygen-air, with 10.5 min on air, 10 min on oxygen, followed by another 5 min on air, while breathing 2.0 % isoflurane at a rate of 2 L/min.

The remaining two mice were female Hsd:Athymic Nude-Foxn1^nu mice (Envigo) with XP373 human renal cell carcinoma tumor xenografts [42] implanted in the right kidney following published protocols [41], with two imaging sessions acquired for Mouse #1 at 31 and 36 days, and one session at for Mouse #2 at 31 days after tumor implantation. Prior to imaging, hair was removed from the animal using depilation cream. Seven wavelengths were sampled sequentially [λ = 680, 734, 757, 798, 850, 922, 980 nm] to correspond with major features of the hemoglobin spectrum, including local spectral peaks and the isosbestic point. Each wavelength was sequentially oversampled 3 times and averaged prior to image reconstruction, leading to an overall acquisition time of 2.1 s per multispectral sampling cycle. All images were acquired at the same axial cross-section. During imaging, each anesthetized mouse underwent a gas breathing challenge from air-oxygen-air, with 5 min on each gas, while breathing 2.0 % isoflurane at a rate of 2 L/min.

2.3. Image reconstruction

After acquisition, each single-wavelength image was preprocessed by correcting for laser power variation and deconvolving the signal impulse response using Wiener deconvolution with a regularization constant of 0.001. Each image was then reconstructed using a curve-driven model matrix as a forward model [43] and a non-negative accelerated projected conjugate gradient method as the inverse solver [44].

2.4. Multispectral state filtering

All analysis was performed using code developed in MATLAB 2017b, with production code available at https://git.biohpc.swmed.edu/PIRL/kalata-filters-matlab. Archived version is available at https://doi.org/10.5281/zenodo.3836682.

The single-wavelength images from the numerical and experimental datasets were used as input to each of the SW, $\alpha$, and $\alpha \beta$ filters. For each method, an $N_y\times N_x\times N_c$ multispectral image with $N_y\times N_x$ pixels per channel and $N_c$ channels was maintained, to store each filter’s estimate of the multispectral image. At each point in time, all channels of the entire multispectral image were updated according to the prediction step for each method, using either Eq. 6 for the SW and $\alpha$ filters, or Eqs. 8 and 9 for the $\alpha \beta$ filter. Following the prediction step, a single-channel image, acquired at wavelength ${\lambda }_k$, was loaded into memory, corresponding to $z_k$ in Eq. 1. This image was used to update channel ${\lambda }_k$ in each filter’s multispectral state estimate, using either Eq. 7 for the SW and $\alpha$ filters or Eqs. 10 and 11 for the $\alpha \beta$ filter.

The numerical phantom data were filtered using ${\sigma }_w=300\frac{a.u.}{s^2}$, ${\sigma }_n=700a.u.$, the RENCA-luc data with ${\sigma }_w=2\frac{a.u.}{s^2}$, ${\sigma }_n=70a.u.$, and the XP373 data with ${\sigma }_w=1\frac{a.u.}{s^2}$, ${\sigma }_n=120a.u.$

2.5. Post-processing

At each time point, the multispectral images estimated by each filtering method were spectrally unmixed using an unconstrained linear model to yield images of oxy- and deoxy-hemoglobin. The total hemoglobin and oxygen saturation in each pixel were estimated as:

$$S{O}_2^{MSOT}=\frac{{\left[Hb{O}_2\right]}^{MSOT}}{{\left[H{b}_{tot}\right]}^{MSOT}}=\frac{{\left[Hb{O}_2\right]}^{MSOT}}{{\left[Hb\right]}^{MSOT}+{\left[Hb{O}_2\right]}^{MSOT}}$$ (12)

We note that the resultant values are estimates, and are thus indicated by the superscript MSOT as used by others [26]; truly quantitative measures of endmember concentration would require additional processing using, e.g., fluence correction [18,19,45,46] or Bayesian methods [47]. In general, the filtering methods presented here cannot correct for errors in quantitation due to modelling errors.

2.6. Quantification methods

2.6.1. Image quality

The deviation of each estimated multispectral image from the known ground truth image was calculated using two metrics: the mean-squared error (MSE) and the structural similarity index (SSIM).

The MSE was used to examine the effects of each approach on noise rejection, but includes some deviation due to differences in the image mean. It is calculated as:

$$MSE=\frac{{{\displaystyle \sum }}_{i,j}{\left(x_{ij}-{\hat{x}}_{ij}\right)}^2}{N_{pixels}}$$ (13)

The SSIM was used to examine images on a holistic basis, accounting for structure, noise, and constant value offsets. It is calculated as:

$$SSI{M}_{x\hat{x}}=\frac{\left(2{\mu }_x{\mu }_{\hat{x}}+C_1\right)\left(2{\sigma }_{x\hat{x}}+C_2\right)}{\left({\mu }_x^2+{\mu }_{\hat{x}}^2+C_1\right)\left({\sigma }_x^2+{\sigma }_{\hat{x}}^2+C_2\right)}$$ (14)

where ${\mu }_x$ and ${\mu }_{\hat{x}}$ represent the mean of each image, ${\sigma }_x^2$ and ${\sigma }_{\hat{x}}^2$ represent the variance of each image, and ${\sigma }_{x\hat{x}}$ represents the covariance between the two images. $C_1$ and $C_2$ are positive constants chosen to improve numerical stability and are related to the dynamic range of the images.

2.6.2. ROI analysis

For the phantom data, a 5 × 5 region of interest (ROI) was defined in the same location for all images within ellipse (e) (Appendix B, Table B1), and the average value within that region determined for all time points for each channel. This was performed for all filtering methods, for each channel of the multispectral image, as well as for each of the [Hb]^MSOT, [HbO₂]^MSOT, [Hb_tot]^MSOT, and SO₂ ^MSOT images, to yield time courses of each parameter.

Manually selected ROIs were defined over the tumor and spine for each mouse and used as the ROIs for both single-channel data and the unmixed parameters. Within each ROI, for each filter, for each parameter, at each time point, a histogram was calculated using 256 equally-spaced bins, with lower and upper limits defined by the 0.1st and 99.9th percentile of the values for that parameter in that ROI for all time points. Each one-dimensional histogram was then used as a column of pixels in a dynamic histogram image. Three time points for each tumor type were chosen prior to, during, and after the transition period from air to oxygen. For each condition, the histograms from 11 time points centered on those selected time points were averaged together and used as representative time slices of the dynamic histogram image to improve visualization.

Further analysis was performed on each filter to compare the statistical effects of each, and this is described in Appendix C.

3. Results

3.1. Phantom data

When applied to the simulated data, both $\alpha$ and $\alpha \beta$ filters effectively suppressed noise in the multispectral images as compared to the SW filter. As seen in Fig. 2, the $\alpha$ filter rejected a large amount of noise from the signal, but failed to follow the rapid dynamics of the simulated gas challenge, causing visible artifacts (Fig. 2A, bottom right). By comparison, the $\alpha \beta$ filter both rejected noise and accurately tracked dynamic changes throughout the image. The relative performance of each of the filtering methods can be seen in the single-channel ROI time course and quality metric time courses. Overall, the $\alpha$ filter performed extremely well unless challenged by sudden dynamic changes. Due to the $\alpha$ filter’s intrinsic time lag, there was a substantial resettling time before it returned to pre-challenge performance. The properties of each filter are retained upon spectral unmixing of the multispectral images, and still further when calculating the derived [Hb_tot]^MSOT and SO₂^MSOT images as seen in Fig. 3. The $\alpha$ filter demonstrated superior performance on the unchanging [Hb_tot]^MSOT channel, but once again suffered from a time lag following dynamic changes. These observations are validated by the time courses of error metrics in Fig. 4. Notably, the unmixed images using the SW were extremely noisy, with the [HbO₂]^MSOT SSIM dropping below 0.8 towards the end of the challenge. The $\alpha$ filter is able to achieve excellent noise rejection at baseline, yielding small MSE and SSIM very close to 1, but during transitions its performance fell below that of the SW.

Fig. 2.

Effect of each filter on dynamic images in numerical phantom. A) Single-wavelength images (left) and their differences (right) from the known ground truth image (left, top) at the final time point of the simulated gas challenge. A clear reduction in noise is seen for both the $\alpha$ and $\alpha \beta$ filters when compared to the SW filter. The $\alpha$ filter suffers from a visible artifact due to its intrinsic time lag, as indicated by the yellow arrow. B) Image showing ROI for signal intensity calculation in C. C) Mean signal intensity over the yellow ROI in B for each time point during challenge. Conspicuous lag is visible in the $\alpha -$ filtered data. Red: SW, blue: $\beta$, green: $\alpha ,$ black: Ground truth. D) Mean squared error of entire multispectral image at each point in time. Degradation of $\alpha$ filter performance is clearly seen accompanying gas challenge changes. E) Structural similarity of entire multispectral image at each point in time, matching D.

Fig. 3.

Effect of each filter on time courses of derived parameters. A) Derived images from filtered data for each filter. The effects of noise in the multispectral images are magnified in the unmixed data. Filtering prevents noise from being propagated to this stage of analysis, leading to qualitatively clearer images for both the $\alpha$ and $\alpha \beta$ filters. B) Difference of each derived parameter image from the corresponding ground truth (A, top row). Both $\alpha$ and $\alpha \beta$ demonstrate decreased noise relative to SW, but the $\alpha$ filter’s lag has resulted in an inaccurate unmixing result, overestimating [HbO₂] and underestimating [Hb] as indicated by yellow arrows. C) Time courses of each parameter for different filters. Red: SW, blue: $\beta$, green: $\alpha ,$ black: Ground truth. Though dynamic changes cause the $\alpha$ filter to deviate from ground truth, it has excellent performance in filtering the constant [Hb_tot] channel.

Fig. 4.

Image quality metrics over time for unmixed parameter images in phantom. Red: sliding window filter, green: $\alpha$, blue: $\alpha \beta$. As in the multispectral case, the $\alpha$ filter achieves excellent noise reduction in steady-state contexts, such as the initial baseline or the entirety of the [Hb_tot] time course. Overall performance of each filter in the unmixed data is similar to that seen in the multispectral case (Fig. 2).

3.2. Experimental data

The αβ filter effectively captured transient spectral dynamics while rejecting noise in both tumor models (Fig. 5). This is in contrast to the SW’s lack of noise rejection, illustrated by the highly variable baseline signal, and the α’s lack of dynamic sensitivity, seen in the rounding-off of the gas challenge transition in the 700 nm channel (Fig. 5B, D: green line).

Fig. 5.

Time courses of MSOT spine signal for representative channels in RENCA-luc mouse. A) Sum image of all channels at final time point, with ROIs used for analysis indicated. T: tumor, S: spine. B, C, D) Time courses of signal in selected channels based on respective filters. B) 700 nm, C) 800 nm, D) 850 nm. Red: sliding window filter, green: $\text{α}$, blue: $\text{α}\text{β}$. The $\text{α}\text{β}$ filter removed a substantial proportion of the noise from the dynamic signal while still accurately following the gas challenge. All channels are shown in Supp. Figures S1 (image comparisons) and S2 (time courses).

The dynamic performance of each filter for the RENCA-luc data is shown in Fig. 6, Fig. 7: Whereas the sliding window data are extremely noisy, preventing straightforward characterization of the response to gas challenge, the α filter clearly separated distinct subpopulation time courses within each ROI. However, the α filter introduced a substantial response time, taking minutes to reach steady state after each gas change, and consequently destroying quantitative information regarding the rate constant of the transition, as well as the transition time itself. In comparison, the αβ filter revealed conspicuous differences in both the rates and degree of response to gas challenge across both simulated and experimental data. These effects are more obvious in the transition region highlighted in Fig. 7: There was a dramatic decrease in the amount of noise in both the α and αβ filtered data, but only the $\alpha \beta$ filter preserved the rate and degree of response at each time point.

Fig. 6.

Dynamic histogram images and selected histograms for each parameter-filter combination in RENCA-luc tumor-bearing mouse. ROIs defined in Fig. 5A. Upper panels: spine ROI. Lower panels: Tumor ROI. Each panel: Dynamic histogram images (left column) and corresponding histograms at selected time points (right column) for sliding window (top), $\alpha$ (middle), and $\alpha \beta$ (bottom) filters. For all parameters, the use of either the $\alpha$ or $\alpha \beta$ filters reduced the noise in the dynamic histograms, but the $\alpha$ filter lagged behind across the entire ROI, as seen by the misalignment in the red and green histograms highlighted by the black arrows.

Fig. 7.

Expanded time courses of air-oxygen transition in RENCA-luc tumor-bearing mouse. ROIs defined in Fig. 5A. Upper panels: Spine ROI. Lower panels: Tumor ROI. Left panels: [HbO₂]^MSOT. Right panels: [SO₂]^MSOT. A conspicuous reduction in noise is seen between the sliding window (each panel, top) and $\alpha \beta$ (each panel, bottom) time courses, allowing one to resolve the distinct evolution of different subpopulations within the ROI.

When applied to the XP373 tumor (Fig. 8, Fig. 9), the $\alpha \beta$ filter reveals heterogenous response to the gas challenge which is not visible in the sliding window data. Indeed, it is clear in the upper-left panel of Fig. 9 that there are both responsive and nonresponsive subpopulations of pixels within the spine ROI. The αβ filter also performed well on relatively constant data such as the [HbO₂]^MSOT channel in the tumor ROI, delineating a bimodal distribution which was obscured by noise in the SW filter. These general between-filter patterns are sustained when applying the filters to additional XP373 gas challenge datasets, as seen in Supplementary Figs. S3–5.

Fig. 8.

Dynamic histogram images and selected histograms for each parameter-filter combination in XP373 tumor-bearing Mouse #1. ROIs defined in Supp. Fig. 3A. Upper panels: Spine ROI. Lower panels: Tumor ROI. For all derived parameters, the use of either the $\alpha$ or $\alpha \beta$ filters reduced the amount of noise in the dynamic histograms, though the $\alpha$ filter lagged behind across the entire ROI, as seen by the misalignment in the red and green histograms highlighted by the black arrows.

Fig. 9.

Expanded time courses of air-oxygen transition in XP373 tumor-bearing mouse (Fig. 8). Upper panels: Spine ROI. Lower panels: Tumor ROI. Left panels: [HbO₂]^MSOT. Right panels: [SO₂]^MSOT. A conspicuous reduction in noise can be seen between the sliding window (each panel, top) and $\alpha \beta$ (each panel, bottom) time courses, allowing one to resolve the distinct evolution of different subpopulations within the ROI.

Both the $\alpha$ and $\alpha \beta$ filters provided improved statistical performance in assessing whether an oxygen-induced change is significantly different from the air baseline, in both spatial and temporal dimensions (Fig. 10, Table 1). Areas where the signal to noise ratio is low are deemed as insignificant changes under analysis of the SW-filtered data, but both the $\alpha$ and $\alpha \beta$ filters resolve those same areas as significant (Supplementary Fig. S6). In examining the transition over time from insignificant to significant, fitting of a logistic function to the binary classification time course yields an effective transition time. The analysis provided by the SW filter provided inconsistent fits for transition times, which were highly variable depending on the SNR of the local region. Though the $\alpha$ filter provided a much sharper transition from non-significant to significant, it occurred at a substantial delay from the known switching time of 10.5 min. The $\alpha \beta$ filter yields consistent and sensible transition times between all ROIs.

Fig. 10.

Significance testing of HbO₂^MSOT change due to gas challenge in RENCA-luc tumor-bearing mouse. Left: 4-pixel ROIs used. ROIs 1 and 2 are significant via the pixel-wise analysis for all filters, while ROI 3 is not significant for the SW filter (Table 1, Supplementary Fig. S2). Center: log₁₀(p) values over time for each filter in each ROI. Right: Binary classification of change as significant (s.) or not (n.s.), as well as logistic curve fit and indicated switching time. The SW filter is conspicuously noisy throughout time for all ROIs, but particularly so in ROI 3, causing an unreasonable fit of the switching time. The $\alpha$ filter denoises effectively and has a very sharp transition from non-significant to significant, but incurs a 0.5 min delay in the switching time. The $\alpha \beta$ filter denoises effectively while preserving the dynamic transition, giving a consistent switching time for all ROIs (see Table 1).

Table 1.

Comparison of filter performance based on spatial and temporal analyses. The spatial analysis refers to the significance of the [HbO₂]^MSOT difference before and after the gas challenge transition within each ROI inFig. 10. The temporal analysis refers to the significance of the difference between a single [HbO₂]^MSOT image at a particular time point before the gas challenge transition and each successive [HbO₂]^MSOT image during the gas challenge transition.

ROI	Filter	Spatial analysis			Temporal analysis
		p-value	Scaled p-value	Significant	Mean p-value of $t_{1695}:t_{1700}$	tswitch (min)
1	SW	$1.3\times {10}^{-8}$	$7.8\times {10}^{-4}$	Yes	$7.1\times {10}^{-6}$	11.53
	$\alpha$	$5.6\times {10}^{-24}$	$3.5\times {10}^{-19}$	Yes	$2.2\times {10}^{-6}$	11.95
	$\alpha \beta$	$3.9\times {10}^{-13}$	$2.4\times {10}^{-8}$	Yes	$1.6\times {10}^{-6}$	11.41
2	SW	$5.2\times {10}^{-9}$	$3.2\times {10}^{-4}$	Yes	$1.1\times {10}^{-6}$	11.77
	$\alpha$	$2.2\times {10}^{-26}$	$1.4\times {10}^{-21}$	Yes	$1.9\times {10}^{-6}$	12.07
	$\alpha \beta$	$1.6\times {10}^{-14}$	$9.9\times {10}^{-10}$	Yes	$8.5\times {10}^{-7}$	11.66
3	SW	$2.1\times {10}^{-6}$	$0.13$	No	$5.2\times {10}^{-6}$	13.05
	$\alpha$	$1.6\times {10}^{-20}$	$1.0\times {10}^{-15}$	Yes	$9.3\times {10}^{-7}$	12.03
	$\alpha \beta$	$9.4\times {10}^{-11}$	$5.9\times {10}^{-6}$	Yes	$3.5\times {10}^{-7}$	11.45

4. Discussion

The $\alpha \beta$ filter provides an effective balance between fidelity to spectral dynamics and noise rejection in a wide variety of contexts and under different spectral sampling patterns. Notably, the filter is robust to variations in the parameters ${\sigma }_w$ and ${\sigma }_n$, and thus requires little tuning in order to achieve substantially improved results.

In scenarios where no appreciable change is expected, or where the dynamics are very slow relative to the sampling rate, the $\alpha$ filter provides superior performance over the $\alpha \beta$ filter due to its effective noise-suppression. This may be used to accommodate piecewise-stationary statistics wherein large changes in the underlying signal occur in between periods of steady-state behavior, or when the noise power is large relative to the true signal variation.

Accommodating high-noise, low-variation environments may be accomplished by setting the measurement noise variance high relative to a lower process noise variance. This causes the filters to bias themselves towards previous estimates, providing benefits for such contexts as measuring hemodynamic cycling. Other dynamic imaging modalities such as real-time functional MRI [48,49] or calcium imaging [50] may benefit from the use of these filters as well, as experimental motifs such as neurofeedback require reduced data noise without incurring substantial analysis delays [51].

Due to the ability of the $\alpha$ and $\alpha \beta$ filters to incorporate information from previous samples, it may be desirable to choose sampling patterns which acquire one image at each wavelength, rather than oversampling each wavelength sequentially. Although the latter case enables averaging to improve signal to noise ratio, it inherently degrades time resolution and can introduce substantial blurring due to motion [24]. Further, this work only explores the application of these filters to reconstructed images – it is probable that image quality itself could be improved by applying the methods to raw photoacoustic data, in order to better condition the reconstruction process itself, and we will investigate this matter in a forthcoming publication.

Motion may present a challenge for these filters due to their recursive structure. As the $\alpha$ and $\alpha \beta$ filters are infinite impulse response (IIR) filters, an arbitrarily extreme observation or outlier will affect the output data at any later time. Rejection of spurious data could be accomplished by identifying frames which are misregistered and down-weighting the associated residuals. This identification process may be accomplished via a number of methods, ranging from simple differences of sequential frames for online analysis [24] to sophisticated clustering methods when post-processing is an option [[52], [53], [54]].

The work presented here assumes that ${\sigma }_w$, ${\sigma }_n,$ and $\text{Δ}t$ are constant through time, as well as being homogeneous throughout the image and across channels. As measurement noise may vary between imaging channels and across a field of view, and different regions may have distinct dynamics, a single global tracking index may not be optimal. The $\alpha \beta$ filter performs well even with the use of global parameters, but it may nevertheless be desirable to enable these parameters to vary throughout the temporal, spatial, and spectral dimensions.

Although other methods such as Bayesian or Kalman filters might provide better quantitative and qualitative results, the filters described here are very efficient both in terms of computational complexity and required storage, and can be readily implemented in any processing or acquisition pipeline with minimal performance impact. Indeed, the SW and $\alpha$ filters each require only a single copy of the spectral image, while the $\alpha \beta$ filter only requires the additional storage occupied by the $\dot{x}$ image for substantially improved performance.

As these filters are extremely efficient, it is possible to simultaneously maintain several filters with different parameters, using the data likelihood for each in order to provide an improved estimate of the state [28,29]. This, in turn, may be used to modify the parameters for each filter, giving an efficient adaptive update scheme.

Table 2 shows the time complexities and storage requirements for each of the presented methods, as well as comparisons against the full Kalman filter (KF). We assume an overall multispectral image size of $N$ pixels over $M$ channels, a measured image size of N, and the use of global parameters for both $\alpha$ and $\alpha \beta$ filters. For the KF, we assume non-sparse matrices and no control inputs. Timing for the various filters was performed using a single thread on an Intel Xeon E5-2680 v3 processor operating at 2.5 GHz. The algorithm-specific memory footprint was less than 5 MB for all filters tested.

Table 2.

Comparison of filter performance for various filtering methods.

	Time Complexity	Storage Requirement	Time/frame (s)
SW	O(N)	N*M	8.41 ($\pm$ 9.54) x 10−5
$\alpha$	O(N)	N*M + 3	6.83 ($\pm$2.26) x 10−4
$\alpha \beta$	O(N)	2*N*M + 3	1.3 ($\pm$ 0.42) x 10−−3
Kalman	O(N3M3)	4N2M2 + N2M + 2N2 +NM+2N	Not tested

The SW retains only the copy of the multispectral image, and overwrites the values in a given imaging channel with the measured values. It therefore has a time complexity of O(N) and a storage requirement of N*M. The $\alpha$ filter retains a copy of the multispectral image as well as a single floating-point value to store the tracking index, or three values to represent the ${\sigma }_w$, ${\sigma }_n$, and $\text{Δ}t$ values. The calculations are performed elementwise, and so the calculation has a time complexity of O(N), and a storage of N*M + 3. The $\alpha \beta$ filter retains a copy of the multispectral image as well as the dx image, giving it a storage requirement of 2*N*M + 3. The calculations still operate using basic arithmetic, and so the overall cost of an iteration is O(N).

The Kalman filter requires the storage of a N*M x N*M state-transition matrix F, a N*M x N*M covariance matrix P, a N*M x N*M process noise covariance matrix Q, measurement matrices H for each channel, yielding M * (N x N*M), a measurement noise covariance matrix R, of size N*N, the innovation covariance S, of size N*N, and must have storage for the Kalman gain matrix of size N*M x N. There must also be storage for the previous estimate, the measurement itself, and the innovation residuals, adding another M*N+2*N elements. The time complexity is dominated by matrix multiplications and the finding of a matrix inverse, and exceeds O(N³M³). Various optimizations may be performed in order to limit the complexity of the KF, but many induce their own overhead and do not lower the complexity to the linear time seen in these filters [55].

5. Conclusion

The $\alpha$ and $\alpha \beta$ filters, originally developed for target tracking and now applied to spectral imaging, are able to effectively preserve the spatial, spectral, and temporal structure of MSOT data while improving the signal to noise ratio, and are able to do so in a highly efficient, online fashion. These filters are simple and easily implemented, theoretically optimal, and practically effective. The resultant increases in image quality enable new capabilities and better insight into highly dynamic biological systems, allowing improved hypothesis testing and decision making.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Biographies

Devin O’Kelly is currently a PhD student at the University of Texas Southwestern Medical Center (UTSW) pursuing a degree in Biomedical and Molecular Imaging as well as Computational and Systems Biology. He received his B.S. in Biomedical Engineering with high honors from the University of Texas at Austin. His major research interest is the development and application of experimental techniques and numerical methods in order to probe complex biological systems using high-information imaging. He is a Biomedical High-Performance Computing Fellow at the UTSW BioHPC, developing computational frameworks to enable scalable analyses of massive imaging datasets.

Dr. Yihang Guo is currently a visiting fellow of the Small Animal Imaging Resource at the University of Texas Southwestern Medical Center (UTSW), as well as a surgical doctor of the Department of Gastrointestinal Surgery at The Third XiangYa Hospital of Central South University in China. He received his M.D and Ph.D in Clinical Medicine from Central South University. His major research interest is the development and application of molecular animal imaging for predicting optimal cancer therapy and assessing early response to treatment. He has extensive experience with building multiple animal orthotopic tumor models in kidney, lung, colon, pancreas and tibia.

Dr. Ralph Mason, a chemist by training, has over 25 years’ experience in cancer imaging, therapy, and tumor pathophysiology. He is Professor of Radiology and directs the Small Animal Imaging Resource. His primary research interest is prognostic radiology - developing and implementing methods for predicting optimal cancer therapy and assessing early response to treatment ("precision medicine"). Dr. Mason has experience with diverse imaging modalities and has published extensively regarding technology development and applications to assessing hypoxia, radiation therapy and the effects of vascular disrupting agents.

Appendix A

Calculation of coefficients for $\alpha$ and $\alpha \beta$ filters

The calculation of the $\text{α}$ and $\text{β}$ coefficients for the tracking filters depends on the order of the filter; the $\text{α}$ calculated for a 1st order filter ($\text{α}$ filter) is different from that calculated for a 2nd order filter ($\text{α}\text{β}$ filter). The equations here were derived in the paper by Kalata [36].

In each case, the value of the tracking index is the same and is calculated using Eq. 5.

(A.1) $\alpha$ Filter

$$\alpha =\frac{-{\text{Λ}}^2+\sqrt{{\text{Λ}}^4+16{\text{Λ}}^2}}{8}$$ (A.1)

(A.2) $\text{α}\text{β}$ Filter

$$R=\frac{4+\text{Λ}-\sqrt{8\text{Λ}+{\text{Λ}}^2}}{4}$$ (A.2)

$$\alpha =1-R^2$$ (A.3)

$$\beta =2\left(2-\alpha \right)-4\sqrt{1-\alpha }$$ (A.4)

Appendix B

Dynamic Spectral Phantom

In order to provide a test bench for quantifying the efficacy of the filters, we created well-defined datasets with specific properties to present challenges for filtering in spatial, spectral, and temporal dimensions.

We used a modified form of the Shepp-Logan phantom with all-positive entries, built up from the sum of non-negative parts. This is done as imaging is usually concerned with measuring quantities that are strictly positive. Each of the ellipses which comprise the phantom was given its own time course of spectral components. This allowed us to test the effects of multiple basis elements j present in a given pixel, all of which may follow different dynamics.

$$S\left(\lambda ,t\right)={\displaystyle \sum }_j{C}_j\left(t\right){\mu }_{a,j}\left(\lambda \right)$$ (B.1)

To test the dynamics, $C(t)$ was created as a pair of step functions convolved with a particular exponential kernel for each ellipse.

$$c\left(t\right)=c_0+c_{1}H\left(t-t_1\right)-c_{2}H\left(t-t_2\right)$$ (B.2)

$$K\left(t\right)=\exp \left(-kt\right)$$ (B.3)

$$C\left(t\right)=c\left(t\right)\otimes K\left(t\right)$$ (B.4)

Where $\otimes$ denotes convolution and H is the Heaviside step function. For this work, t₁ and t₂ were defined to be 1/3 and 2/3 of the total simulation time, respectively.

Definition of oxy-deoxy hemoglobin time courses was performed by first defining the time courses of total hemoglobin (Hb_tot) and oxygen saturation (SO₂) and then multiplying each concentration time course by the corresponding absorption spectrum as in Eq. B.1

$$C_{Hb}\left(x,y,t\right)=C_{THb}\left(x,y,t\right)*\left(1-S{O}_2\left(x,y,t\right)\right)$$ (B.5)

$$C_{Hb{O}_2}\left(x,y,t\right)=C_{THb}\left(x,y,t\right)*S{O}_2\left(x,y,t\right)$$ (B.6)

Total hemoglobin concentration was defined to be constant within each ellipse for all time.

Based on the ground-truth multispectral images, one single-wavelength image was sampled at each point in time to provide a sampled multispectral phantom. The phantom created for this paper used the parameters in Table B1.

Table B1.

Parameters for dynamic multispectral phantom.

Ellipse	Center (x,y)	Major Axis	Minor Axis	Theta	Hbtot	c0	c1	c2	kresponse
a	(0,0)	0.69	0.92	0	1	0.4	0.6	0.4	0.2
b	(0, −0.0184)	0.6624	0.874	0	0.8	1.0	0.5	0.0	0.3
c	(0.22,0)	0.11	0.31	−18°	0.2	0.2	0.5	0.1	0.02
d	(−0.22, 0)	0.16	0.41	18°	0.2	0.5	0.5	0.6	0.04
e	(0, 0.35)	0.21	0.25	0	0.1	0.2	0.9	0.2	0.2
f	(0, 0.1)	0.046	0.046	0	0.1	0.1	0.1	0.9	0.1
g	(0, −0.1)	0.046	0.046	0	0.1	0.9	0.8	0.7	0.05
h	(−0.08, −0.605)	0.046	0.023	0	0.1	1.0	1.0	1.0	0.4
i	(0, −0.605)	0.023	0.023	0	0.1	0.7	0.6	0.2	0.2
j	(0.06, −0.605)	0.023	0.046	0	0.1	0.9	0.2	0.9	0.1

Appendix C

Statistical Comparisons of Different Filtering Approaches

In order to compare the efficacy of the different filtering approaches, we considered the onset of the air-oxygen gas challenge and the quantification of the significance of a given change on both a pixel-wise basis and on an ROI-wise basis. Both analyses were done using the [HbO₂]^MSOT image time courses from the RENCA-luc mouse data; the XP373 data was analyzed using the pixel-wise approach only.

Pixel-wise comparison

Two image populations from the RENCA-luc mouse image time-series were considered for each filter: $P_1$, consisting of the [HbO₂]^MSOT images from filtered frames 1297–1303, such that $N_1=7$; and $P_2$, consisting of the [HbO₂]^MSOT images from filtered frames 1697–1703. These time points were chosen to correspond with the blue and green histograms considered in Fig. 6, Fig. 7, and assuming that the dynamic changes over each population’s time range are small relative to the level of noise.

For each pixel $\left(x_i,y_i\right)$, we performed a one-tailed T-test with a null hypothesis $H_0$ that the difference in the means of $P_2\left(x_i,y_i\right)$ and $P_1\left(x_i,y_i\right)$ is positive: $H_0:{\mu }_2\left(x_i,y_i\right)-{\mu }_1\left(x_i,y_i\right)>0$. This test was chosen as we expect the level of oxy-hemoglobin to increase when transitioning from air to oxygen. We assume that issues of spectral coloring are negligible. As the test is repeated across every pixel in the $250\times 250$ RENCA tumor images, the number of hypotheses tested $M_H=250\times 250=62500$. The Bonferroni-corrected significance threshold ${\alpha }_{space}$ is thus $\alpha =\frac{0.05}{62500}=8\times {10}^{-7}$. Repeating this analysis for each pixel results in p-value maps which are used to assign a binary classification of significant if the p-value laid below ${\alpha }_{space}$, and non-significant otherwise. Results are depicted in Supplementary Fig. S6.

For each of ${ROI}_1$, $RO{I}_2$, and ${ROI}_3$ as described below, the median significance classification and mean p-value were calculated and used as figures of merit for each filtering method.

The same approach was repeated for each of the XP373 mice, with $P_1$ consisting of the [HbO₂]^MSOT images from filtered frames 2597–2603, such that $N_1=7$; and $P_2$, consisting of the [HbO₂]^MSOT images from filtered frames 3997–4003. Results are depicted in Panel A of Supplementary Figs. S3–S5.

ROI-wise comparison

Three ROI populations within each image were considered for each filter: ${ROI}_1$ was chosen to lie in the spine, in a set of pixels whose change was deemed significant for all methods by the pixel-wise comparison. ${ROI}_2$ was chosen to lie in the tumor bulk, in a set of pixels whose change was significant for all methods by the spatial comparison. ${ROI}_2$ was chosen to lie in the tumor bulk, in a set of pixels deemed significant by the spatial analysis using both the $\alpha$ and $\alpha \beta$ filters, but was non-significant using the SW filter. ${ROI}_1$ consisted of the pixels $(x_i,y_i:x\in \left[\mathrm{125,126}\right],y\in \left[\mathrm{175,176}\right])$, ${ROI}_2$ consisted of the pixels $(x_i,y_i:x\in \left[\mathrm{170,171}\right],y\in \left[\mathrm{150,151}\right])$, and ${ROI}_3$ consisted of the pixels $(x_i,y_i:x\in \left[\mathrm{150,151}\right],y\in \left[\mathrm{120,121}\right])$.

For each ROI and each time-point $t_k$ between 1300 and 1700, we performed a one-tailed T-test with a null hypothesis $H_0$ that the difference in the means of $R\left(t_k\right)$ and $R\left(t_{1300}\right)$ is positive: $H_0:{\mu }_R\left(t_k\right)-{\mu }_R\left(t_{1300}\right)>0$. As the test is repeated for every time-point between 1300 and 1700, the number of hypotheses tested $M_H=400$. The Bonferroni-corrected significance threshold $\alpha$ is thus $\alpha =\frac{0.05}{400}=1.25\times {10}^{-4}$.

The classification of the difference in means for each ROI was assigned a value of 1 if the null hypothesis was rejected, and 0 if it was not. This yielded a time course of binary classification for each method for each ROI. A logistic function was then fit to these classification time courses using the equation

$$H\left(t\right)=\frac{1}{1+\text{exp}\left(-c\left(t-t_{switch}\right)\right)}$$

and t_switch, the time coordinate of the half-max of the logistic function, used as the indicated switching time.

The mean p-value of the last 6 time points was used to report the degree of significance revealed by each of the filtering methods.

Appendix D. Supplementary data

The following is Supplementary data to this article: