Deep Learning Approach for Dynamic Sparse Sampling for High-Throughput Mass Spectrometry Imaging

Abstract

A Supervised Learning Approach for Dynamic Sampling (SLADS) addresses traditional issues with the incorporation of stochastic processes into a compressed sensing method. Statistical features, extracted from a sample reconstruction, estimate entropy reduction with regression models, in order to dynamically determine optimal sampling locations. This work introduces an enhanced SLADS method, in the form of a Deep Learning Approach for Dynamic Sampling (DLADS), showing reductions in sample acquisition times for high-fidelity reconstructions between ~ 70–80% over traditional rectilinear scanning. These improvements are demonstrated for dimensionally asymmetric, high-resolution molecular images of mouse uterine and kidney tissues, as obtained using Nanospray Desorption ElectroSpray Ionization (nano-DESI) Mass Spectrometry Imaging (MSI). The methodology for training set creation is adjusted to mitigate stretching artifacts generated when using prior SLADS approaches. Transitioning to DLADS removes the need for feature extraction, further advanced with the employment of convolutional layers to leverage inter-pixel spatial relationships. Additionally, DLADS demonstrates effective generalization, despite dissimilar training and testing data. Overall, DLADS is shown to maximize potential experimental throughput for nano-DESI MSI.

Introduction

Background

Traditional scans in Mass Spectroscopy Imaging (MSI), among other forms of spectroscopy and microscopy, move a probe along a rectilinear grid (e.g. raster scanning), with measurements of each scanning location requiring up to ~ 5 seconds for high-resolution acquisitions. This results in a high inefficiency for sample processing and throughput, possible sample deterioration in scenarios requiring the employment of high-energy particle streams, and difficulties when studying time-sensitive chemical/biological phenomena. While there are other sampling methods that use predetermined locations, (including uniform, random, and low-discrepancy approaches [1, 2, 3]), or instances where a sample geometry is sufficiently defined [4], typically they either lack sufficient context to effectively approximate the ground-truth, or the needed flexibility for widespread adoption.

Alternative dynamic sampling methods were developed to determine optimal scan locations based on information obtained during acquisition [5, 6, 7]. A particularly successful implementation, Supervised Learning Approach for Dynamic Sampling (SLADS) [8, 9], seeks to only acquire data necessary to generate a low-error reconstruction of the ground-truth, through weighted mean interpolation. Measurement locations are therefore chosen by which minimize the entropy within the reconstruction, as estimated through a regression model. Training this model incorporates matched sets of extracted features and their corresponding entropy reductions for potential measurement locations. Since it only performs measurements in regions likely to contain information of interest and requires minimal post-processing to produce high-fidelity reconstructions, SLADS minimizes computational expense and maximizes experimental throughput.

The SLADS framework provides a significant reduction to the number of measurements for many pointwise scanning scenarios. SLADS achieved below 10⁻⁵ Normalized Distortion (ND) levels with only 6.9% of scanned locations in Electron Back Scatter Diffraction (EBSD) microscopy [8, 9]. In X-Ray crystallography of proteins, only 5% of a sample was required for a ND level of ~ 10⁻³% (a ~ 20-fold reduction in X-ray exposure) [10]. In confocal Raman microscopy, SLADS yielded a 6-fold reduction in the number of measurements needed for a 0.1% image difference [11]. Besides the original SLADS least-squares model, variations using a neural network [12], Gaussian mixture models [13], and vector measurements [14] have also previously been employed. The neural network implementation, designated as SLADS-Net, provided evidence of improved generalization when employed with dissimilar training and testing content.

DLADS Approach

Existing SLADS implementations have utilized synthetic, symmetric data with relatively low resolutions, and only operated in the pointwise domain. This work’s architecture, Deep Learning Approach for Dynamic Sampling (DLADS), was engineered for real-world, high-resolution images of biological tissues. Mouse uterine and kidney ion images, seen in Figure 1, were acquired using Nanospray Desorption ElectroSpray Ionization (nano-DESI) MSI [15], to examine DLADS generalizability and performance compared with prior SLADS approaches.

Figure 1:

Ion images of training, validation, and testing samples formed by averaging handpicked nano-DESI MSI density visualizations.

Rather than using the least-squares model of SLADS, or the multi-layer perceptron network of SLADS-Net, DLADS employs a Convolutional Neural Network (CNN) to leverage inter-pixel relationships in a two dimensional space. Additional advancements were made to the reconstruction and entropy estimation methods, compensating for input image asymmetry, modifying previously static hyperparameters, and removing the need for feature extraction. Given the potential time expense for hardware alignment, as present in nano-DESI MSI, a group-wise selection implementation was also developed. This line-bounded mode, limits potential scanning locations to specific coordinate ranges. While the original SLADS’s least-square, SLADS-Net’s neural network, and DLADS’s CNN all resulted in throughput improvements in this study, DLADS produced the most significant advancements.

Methods

Pointwise Acquisition

Ω consists of all locations within an N ×M image X. The set S contains k pixel locations obtained (S = {s⁽¹⁾,s⁽²⁾, …,s^(k)}), with their values denoted by X^(S). Similarly, the set T contains the q unmeasured locations (T = {t⁽¹⁾,t⁽²⁾, …,t^(q)}). Within pointwise acquisition, groups of locations U containing m pixels are greedily selected from T (U = {u⁽¹⁾,u⁽²⁾, …,u^(m)}). When scanned, X^(U) are set in the reconstruction: ${\widehat{X}}^{(U)}$. Values ${\widehat{X}}^{(T)}$ are then determined through weighted mean interpolation where DLADS applies a penalty (derived from the input sample’s aspect ratio) on Ω, preventing the development of stretching artifacts.

Overall, this selection process equivocates to the maximization of a Reduction in Distortion (RD) entropy metric: R, where D(·,·) is absolute difference between two images. The RD of a measurement is described by the difference between the distortion in the reconstructions with and without that measurement (Eq (1)). Since X cannot be known during implementation, an Estimated RD (ERD) for T must be found with Eq (2).

$$R^{(U)}=D\left(X,{\widehat{X}}^{(S)}\right)-D\left(X,{\widehat{X}}^{(S+U)}\right)$$ (1)

$${\widehat{R}}^{(T)}=\mathbb{E}\left[R^{(T)}\mid X^{(S)}\right]$$ (2)

Line-Bounded Acquisition

The set of horizontal lines in X, L = {l⁽¹⁾,l⁽²⁾, …,l^(M)} (with pixel locations z), is comprised of a set of lines not visited: T ⊂ L = {l ∈ L : (∀z ∈ l) ∉ S}, and the set of visited lines: S ⊂ L = L\T. For line-bounded acquisition, the ERD for ${\widehat{R}}^{(l\in T)}$, is defined by the sum of ${\widehat{R}}^{(z\in l)}$ (Eq (3)). The next line to scan is selected by finding the line where {${\widehat{R}}^{(l\in T)}$} is maximized. Data acquisition was further refined to measure only 40% of the chosen line’s pixels, those with the highest ERD.

$${\widehat{R}}^{(l\in T)}=\sum _{z\in (l\in T)}{\widehat{R}}^{(T)}(z)$$ (3)

SLADS

In SLADS, the ERD for unmeasured locations: ${\widehat{R}}^{(T)}$, may be described by a least-squares predictor (Eq (4)), with the product of a ρ-dimensional vector V_S extracted from $\widehat{X}$ (describing measures of the data gradient, standard deviation, and density), and a parameter set $\widehat{\theta }$, approximated through least-squares (Eq (5)).

$${\widehat{R}}^{(T)}=V_S\widehat{\theta }$$ (4)

$$\widehat{\theta }=arg\underset{\theta \in {ℝ}^{\rho }}{min}{\Vert \left[\begin{array}{c}{R}^{s^{(1)}}\\ ⋮\\ R^{s^{(k)}}\end{array}\right]-\left[\begin{array}{c}{V}_{s^{(1)}}\\ ⋮\\ V_{s^{(k)}}\end{array}\right]\theta \Vert }^2$$ (5)

SLADS-Net

SLADS-Net’s regression (Eq (6)) uses a function g(·), to denote a deep neural network (50 neurons in 5 layers). Its layer weights w are optimized using an Adam solver (learning rate of 1e−3) to minimize a squared loss function (Eq (7)). As in the original publications, in order to reduce computational overhead during operation, both SLADS and SLADS-Net only update the ERD for pixels in neighboring regions (size determined during operation) of newly scanned points.

$${\overline{R}}^{(s)}=g^w\left(V_s\right)$$ (6)

$$\mathit{\text{Loss}}=\frac{1}{2}\sum _{i=1}^n{\left(R_{s^{(i)}}-g^w\left(V_{s^{(i)}}\right)\right)}^2$$ (7)

DLADS

DLADS replaces g(·) with a CNN consisting of 4 sequential ResNet identity blocks, using layer normalization, and He weight initialization (Figure 2). A Nadam optimizer (learning rate of 1e−3) is used to minimize a Mean-Squared Error (MSE) loss (Eq (8)) with two inputs: the known measured values (X^(k)) and the reconstruction values for unmeasured locations (${\widehat{X}}^{(q)}$). After training for b epochs, if the training loss falls below that of a validation set, and the average PSNR of that set’s ERD does not improve within e epochs, training will terminate.

$$\mathit{\text{Loss}}=\frac{1}{n}\sum _{i=1}^n{\left(R_{s^{(i)}}-g^w\left(\left[X^{(S)},{\widehat{X}}^{(T)}\right]\right)\right)}^2$$ (8)

Figure 2:

DLADS CNN model (right) with the component identity block architecture (left).

Optimization Parameter c

In all of the models, each of the training samples are measured at h densities: P = {p₁, p₂, …, p_h} to simulate data patterns encountered during operation. As the actual ground-truth RD is computationally prohibitive to determine, an approximation is formed with a parameter c used to limit the regional effects of each unmeasured location’s determined reconstruction value.

In a scenario with two locations, one measured: i and another unmeasured: j, there exists a weighted distance σ_i (Eq (9)). The RD at j, given i: R^(j)(i) can be estimated according to σ_i applied to the estimated distortion (Eq (10)). As i becomes more distant from j, its impact decreases up to the bounds of a window W.

$${\sigma }_i=\frac{{\text{min}}_{f\in S}\Vert i-f\Vert }{c}$$ (9)

$$R^{(j)}(i)\approx {\widehat{R}}^{(j)}(i)=\text{exp}\left\{-\frac{1}{2{\sigma }_i^2}\Vert j-i{\Vert }^2\right\}D\left(X^{(j)},{\widehat{X}}^{(j)}\right)$$ (10)

In SLADS and SLADS-Net, W is set to a static, symmetric value. DLADS uses a dynamic window set on a pixel-by-pixel basis as 3σ, additionally applying the sample aspect-ratio penalty to account for asymmetry. Further, while the ground-truth RD values in the original SLADS and SLADS-Net publications were only calculated for a maximum of half of the unmeasured locations in order to reduce training times, DLADS leverages multi-processing to enable calculation for all locations.

The c parameter still needs to be optimized for each application with o possible values: C = {c₁,c₂, …,c_o}. SLADS and SLADS-Net generate a model for each possible c, simulate those models’ employment, and select the optimal c value: c*, as which best minimizes $D(X,\widehat{X})$ over all measurement iterations. DLADS uses the approximated ground-truth RD in place of a model generated ERD, so only a single model must be trained. DLADS then chooses c* based on which maximizes the PSNR for all of the training and validation samples, over a simulated scan.

Experimental

Parameters

Visualizations of the employed data, may be seen in Figure 1. The training set consisted of 6 mouse uterine ion images, each formed from 10 handpicked visualized mass ranges, averaged together to highlight biological structures. The validation set for DLADS was similarly constructed from 2 uterine samples. Testing data included another 2 mouse uterine and alternately 4 kidney samples. All tissues were scanned with nano-DESI MSI [15], with a spatial resolution of ~ 10μm. The resultant images possessed average widths/heights of 1000/60 and 2000/49 pixels, and average pixel values of 7.573 and 0.008 for uterine and kidney respectively. The images’ dimensional asymmetry resulted from a linear interpolation between rows, made necessary by the use of automated gain control during data acquisition. Additional parameters were empirically chosen as follows: C = {1, 2, 4, 8, 16, 32, 64, 128}; P = {1, 2, …40}; m = 0.01(N *M); W = [15 × 15] (for SLADS and SLADS-Net); b = 50 and e = 50 (for DLADS).

Simulated nano-DESI MSI

Pointwise selection initialized after a 1% random acquisition, while line-bounded selection began after scanning 40% random locations in the central horizontal line. Each simulated scan was terminated after ≥ 40% of all pixels were obtained. Subsequent evaluations were conducted with PSNR.

Pointwise

For qualitative evaluation, matched measurement masks and reconstructed images for uterine and kidney test samples, are shown in Figure 3a and Figure 4a, with 10%, 20%, and 40% of all pixels scanned. SLADS and SLADS-Net had a high coherence to the most prominent structures, but failed to obtain sufficient contextual information for low-error reconstructions. DLADS measurements tended to be more diverse, owing to the spatial awareness afforded by the convolutional layers. Further, while vertical artifacts may be seen in both SLADS and SLADS-Net results, these are absent in those of DLADS, due to the modifications made for asymmetry. These trends may also be seen throughout the scanning progressions, visualized in Figure 5, where DLADS outperformed both of the SLADS architectures.

Figure 3:

Binary measurement masks and reconstructions of a uterine test sample across simulated scans, with all models trained on uterine data. SLADS and SLADS-Net exhibit visual artifacts and clustered measurements. DLADS avoids these behaviors, allowing for effective reconstructions at sampling densities as low as 10%.

Figure 4:

Binary measurement masks and reconstructions of a kidney test sample across simulated scans, with all models trained on uterine data. SLADS and SLADS-Net have similar issues to those exhibited in uterine testing and additionally fail to generalize for line-bounded acquisition. DLADS was able to produce high-fidelity reconstructions, despite dissimilar training and testing content.

Figure 5:

PSNR scanning progression for models with varied acquisition modes and training/testing sets. DLADS shows notable improvements in pointwise acquisition over prior SLADS implementations, with better generalization exhibited when testing with dissimilar data. Scans conducted with a line-bounded constraint were able to produce comparable reconstructions to those of pointwise.

These observations are borne out by the final quantitative results in Table 1a. Using uterine and kidney samples for testing, DLADS provided average improvements of 25.0% and 45.4% over SLADS. Similarly, DLADS had corresponding improvements of 25.4% and 45.9% over SLADS-Net. Neither SLADS, nor SLADS-Net was able to achieve a high-fidelity 40 dB reconstruction, though DLADS was able to with only 20.5% of the sample measurement locations for uterine and 30.8% for kidney.

Table 1:

PSNR (dB) at 40% acquisition of pixels for uterine and kidney testing samples with models trained on uterine data.

	Uterine	Kidney
SLADS-LS	38.4 ± 0.4	27.0 ± 1.6
SLADS-Net	38.2 ± 0.2	26.9 ± 1.7
DLADS	49.3 ± 0.8	42.8 ± 0.8
(a) Pointwise Acquisition
	Uterine	Kidney
SLADS-LS	30.1 ± 0.3	22.1 ± 1.3
SLADS-Net	29.5 ± 0.7	22.2 ± 1.4
DLADS	33.8 ± 0.6	25.8 ± 1.9
(b) Line-Bounded Acquisition

Line-Bounded

Line-bounded simulations were able to produce comparable reconstructions to those of pointwise operation, seen in Figure 3b and Figure 4b. SLADS and SLADS-Net lacked sufficient spatial awareness to select effective measurement locations, while DLADS was better able to determine the underlying geometries. This is evidenced by DLADS having performed fewer measurements in background regions, as well as the lack of visual artifacts in its produced reconstructions. Final PSNR scores for line-bounded simulations are tabulated in Table 1b, where DLADS improved on SLADS by 11.5% and 15.2%, as well as SLADS-Net by 13.7% and 14.7% for uterine and kidney testing sets. The PSNR progression is visualized in Figure 5, where DLADS was the only model able to reach a reasonable approximation of the ground-truth, achieving 30 dB at 30.0% for uterine testing.

Discussion

DLADS provides a demonstrable improvement over prior SLADS implementations of ~ 11 dB and ~ 16 dB for similar and dissimilar tissues when using pointwise acquisition. It further showed similar gains of ~ 4 dB in line-bounded scenarios irregardless of varied training/testing tissues. This exhibits a 70–80% advancement in sample throughput compared to traditional rectilinear scanning, depending on acceptable PSNR levels.

Designing and training the DLADS model sufficiently for reliable improvements over existing SLADS implementations was particularly difficult given the low availability of nano-DESI MSI data. As a result, this study could not determine the limit of DLADS ability to generalize and learn specific biological structures. Therefore, simulating and training DLADS models with a wider assortment and quantity of tissue visualizations remains advisable prior to general implementation or integration with physical equipment. More advanced methods should be considered for reconstruction to further improve throughput. While DLADS averages together handpicked visualized mass ranges to generate the ground-truth sample and RD images, this process may obscure pertinent structural information. Instead, the correlations between and the information within all mass ranges should be utilized for RD generation and ERD estimates. This can be more easily accomplished with DLADS, where each mass range can simply be appended to the network’s input. Further adjustments may also allow for semantic segmentation (multi-label classification for all pixel locations) during active scans, thereby enabling researchers to further refine what specific data should be obtained.

Conclusion

Pointwise SLADS and SLADS-Net methodologies and models were adapted within this work to perform pointwise and line-bounded acquisitions for dimensionally asymmetric molecular images of real-world biological samples, additionally forming the DLADS architecture. Simulations of these models were shown to significantly improve nano-DESI MSI throughput, allowing for the reduction in the number of required measurements by 70–80%. DLADS additionally allows for better inter-tissue generalization and demonstrated a ~ 14 – 46% improvement over prior SLADS variations. Further work should be conducted to make this method more practical for high-resolution scanning applications, utilizing the entirety of available samples’ mass spectra and employing more advanced reconstruction methods.