The AGNOSTIC MRS Benchmark Dataset: Deep Learning for Out-of-voxel Artifacts

Abstract

Purpose:

Neural networks are potentially valuable for many challenges associated with MRS data. The purpose of this manuscript is to describe the AGNOSTIC dataset, which contains 259,200 synthetic MRS examples. To demonstrate the utility, we use AGNOSTIC to train two Convolutional Neural Networks (CNNs) to address out-of-voxel (OOV) echoes. A Detection Network was trained to identify the point-wise presence of OOV echoes, providing proof of concept for real-time detection. A Prediction Network was trained to reconstruct OOV echoes, allowing subtraction during post-processing.

Methods:

AGNOSTIC was created using 270 basis sets that were simulated across 18 field strengths and 15 echo times. The synthetic examples were produced to resemble in vivo brain data with combinations of metabolite, macromolecule, and residual water signals, and noise.

Complex OOV signals were mixed into 85% of synthetic examples to train two separate U-net CNNs for the detection and prediction of OOV signals.

Results:

AGNOSTIC is available through Dryad and all Python 3 code is available through GitHub. The Detection network was shown to perform well, identifying 95% of OOV echoes. Traditional modeling of these detected OOV signals was evaluated and may prove to be an effective method during linear-combination modeling. The Prediction Network greatly reduces OOV echoes within FIDs and achieved a median ${\text{log}}_{10}$ normed-MSE of −1.79, an improvement of almost two orders of magnitude.

Conclusion:

The AGNOSTIC benchmark dataset for MRS is introduced and various dataset features are described. As an exemplar use of AGNOSTIC, two CNNs were developed to detect and predict OOV echoes.

1. Introduction:

Proton (¹H) magnetic resonance spectroscopy (MRS) non-invasively measures levels of endogenous neurometabolites. MRS-visible metabolites are present at millimolar concentrations in the brain, yielding detectable signals with relatively low signal-to-noise ratio (SNR) which mutually overlap. In vivo spectra suffer from several artifacts that complicate modeling and interpretation of the data, including eddy current effects and out-of-voxel (OOV) echoes¹. While there is some degree of standardization and consensus around pre-processing, modeling, and quantification of MRS data^2–5, this is an evolving field that lacks a single ideal solution due to the complexity of the problem, and which therefore is likely to benefit from recent advances in machine learning.

Deep learning (DL) uses a network consisting of a series of computational layers to process information⁶. Iterative training allows features of the data to be identified and weighted to estimate a final function which predicts a desired output based on a given input⁷. Supervised learning involves training the network based on a pre-defined target, associating ground-truth parameters with each input. An extensive, balanced, and diverse dataset is preferred to increase the generalizability of the DL outcome. High-dimensional data, such as medical images or time series, are demonstrated to be the most beneficial set of data for several computer vision tasks, such as classification, registration, segmentation, reconstruction, and object detection^8,9.

DL has been developed for MRS data as a proof-of-concept in many applications, including metabolite quantification^10–14, phase and frequency correction^15–17, reconstruction of missing data¹⁸, accelerated post-processing^19,20, denoising²¹, super-resolution²², artifact removal²³, and anomaly detection²⁴. Despite the potential, these methods have yet to be shown to generalize outside of small datasets with a single fixed acquisition protocol. Before these methods can be adopted more broadly, appropriate tools must be developed to evaluate performance. A key barrier is the lack of a benchmark dataset to play the role that MNIST and ImageNet have played in the field of Computer Vision^25,26. Such a dataset lowers the barrier to entry for neural network development in MRS and allows performance comparisons between models.

OOV echoes are a substantial issue for in vivo MRS, and an under-studied potential DL application. MRS voxel localization is achieved via a combination of RF pulses and magnetic field gradients, with the intended coherence transfer pathway selected both by phase cycling and dephasing “crusher” gradient scheme²⁷. OOV signals arise from gradient echoes – signals from outside the shimmed voxel of interest are refocused by evolution in local field gradients that are either inherent (from air-tissue-bone interfaces) or arising from second-order shim terms²⁸. Therefore, brain regions close to air cavities (e.g., medial prefrontal cortex) or which require stronger shim gradients (e.g., thalamus, hippocampus, etc.) most commonly exhibit OOV artifacts²⁸. OOV echoes seldom occur at the time of the primary echo, so they manifest in the spectrum as broad peaks with strong first-order phase “ripple” that can obscure metabolite resonances. While acquisition strategies can mitigate OOV echoes to some extent, by careful consideration of crusher schemes or voxel orientation^29,30, post-processing strategies remain valuable where complete elimination is not possible.

This manuscript develops Adaptable Generalized Neural-Network Open-source Spectroscopy Training dataset of Individual Components (AGNOSTIC), a dataset consisting of 259,200 synthetic MRS examples. AGNOSTIC spans a range of field strengths, echo times, and clinical profiles, representing metabolite signals, macromolecule (MM) background signals, residual water signals, and Gaussian noise as separate components. To date, DL applications to MRS have relied upon narrow in-house-generated training datasets that limit the generalizability of the solutions developed and comparisons between tools; AGNOSTIC is proposed as a benchmark dataset to fill this gap. In order to demonstrate the utility of this resource, we then illustrate a specific augmentation of the AGNOSTIC dataset to train neural networks for the detection and prediction of OOV echoes.

2. Methods:

2.1. AGNOSTIC Synthetic Dataset

The parameter space that AGNOSTIC spans is deliberately broad, comprising: 18 field strengths; 15 echo times; broad distributions of metabolite, MM, and water amplitudes; and densely sampled time-domain to allow down-sampling. Calculations were carried out using an in-house Python 3³¹ programming script using NumPy³². The dataset is structured as a zipped NumPy³² archive file (.npz) and can be opened as a Python 3³¹ dictionary object. This dictionary contains complex-valued NumPy³² arrays of time-domain data corresponding to the metabolite, macromolecule, water, and noise components which can be combined in different ways depending on the application. For instance, a denoising model may want to target the combined metabolite, MM, and water signal without noise. Within the file, all the acquisition parameters (field strength, echo time, spectral width, etc.), simulation parameters (signal to noise, full-width half-max, concentrations, T₂ relaxation, etc.), and data augmentation options are specified as detailed below.

2.1.1. Basis Set Simulation:

Metabolite spectra are based upon density-matrix-simulated basis functions^33–36. A total of 270 basis sets were created across 18 field strengths (1.4 T – 3.1 T in steps of 0.1 T) and 15 echo times (10 ms – 80 ms in steps of 5 ms). The Point RESolved Spectroscopy (PRESS) pulse sequence³⁷ was simulated using ideal pulses. The simulated spectral width, centered on 4.7 ppm, was 63.62 ppm for all field strengths (e.g., 8 kHz at 3 T; 4 kHz at 1.5 T). The simulated “acquisition window” was started immediately after the last pulse to generate points before the echo. Each metabolite signal was output as a N × 16684 NumPy³² array, where N is the number of spins for a given metabolite and 16684 is the fixed length of complex time points (300 points before the echo maximum, with an appropriate padding number of zeros and followed by the simulated pre-echo signal, and 16384 points after the echo).

39 brain metabolite basis functions were simulated: Adenosine Triphosphate (ATP); Acetate (Ace); Alanine (Ala); Ascorbate (Asc); Aspartate (Asp); β-hydroxybutyrate (bHB); β-hydroxyglutarate (2HG); Citrate (Cit); Cysteine (Cys); Ethanolamine (EA); Ethanol (EtOH); Creatine (Cr); y-Amino-Butyric Acid (GABA); Glucose (Glc); Glutamine (Gln); Glutamate (Glu); Glycerophosphocholine (GPC); Glutathione (GSH); Glycerol (Glyce); Glycine (Gly); Water (H₂O); Homocarnosine (HCar); Histamine (Hist); Histidine (His); Lactate (Lac); Myo-Inositol (mI); N-Acetyl-Aspartate (NAA); N-Acetyl-Aspartate-Glutamate (NAAG); Phenylalanine (Phenyl); Phosphocholine (PCho); Phosphocreatine (PCr); Phosphoethanolamine (PE); Scyllo-Inositol (sI); Serine (Ser); Taurine (Tau); Threonine (Thr); Tryptophan (Tryp); Tyrosine (Tyr); and Valine (Val). GABA was separately simulated using two different spin-system enumerations^38,39. Both α-glucose alpha and β-glucose were simulated.

2.1.2. Assembly of Metabolite Component:

Individual metabolite basis functions were linearly combined to give a metabolite spectral component, weighted by metabolite concentrations sampled from distributions defined by our recent meta-analysis⁴⁰, including both healthy and clinical cohort ranges. From the full basis sets, 22 metabolites were selected which had defined concentration ranges available in our recent meta-analysis⁴⁰. Concentrations were selected with equal probability from a range defined by ±2.5 standard deviations from the meta-analysis mean of each cohort⁴⁰.

Transverse decay of time-domain data was simulated with exponential and Gaussian components. Intrinsic T₂ relaxation times were based upon the 1.5 T results from a relaxation meta-regression⁴⁰ and modeled as an exponential decay⁴¹. Additional “T₂*” contributions were modeled to achieve a full-width half-maximum (FWHM) linewidth of the NAA singlet between 3 Hz and 18 Hz with a uniform distribution by applying appropriate amounts of Gaussian decay. A small amount of jitter (between 20 s⁻² and 100 s⁻²) was added to the gaussian decay rate such that each metabolite would undergo a similar, but not identical, amount of Gaussian decay to better replicate the variability observed for in vivo data.

2.1.3. Macromolecular Component:

Fourteen MM signals were modeled at: 0.92 ppm; 1.21 ppm; 1.39 ppm; 1.67 ppm; 2.04 ppm; 2.26 ppm; 2.56 ppm; 2.70 ppm; 2.99 ppm; 3.21 ppm; 3.62 ppm; 3.75 ppm; 3.86 ppm; and 4.03 ppm^42,43. MM chemical shifts were jittered by ± 0.03 ppm. Each MM signal was simulated as a singlet with exponential decay rate sampled uniformly from a range specified by literature of MM T₂ time constants⁴⁴ and additional Gaussian decay to reach published linewidths^43,44. MM amplitudes were sampled uniformly from within published ranges^43,44.

2.1.4. Noise Component:

Noise was generated from a normal distribution, with independent random real and imaginary points. The noise was scaled such that the signal-to-noise ratio of the NAA singlet (SNR_NAA, defined as NAA height divided by the standard deviation of the noise) was between 5 and 80, uniformly sampled. The noise amplitude values are also stored within the archive file.

2.1.5. Residual Water Component:

The residual water basis signal was simulated as a singlet at 4.7 ppm. In order to simulate imperfect water suppression, this residual water signal was used to model up to 5 unique water signals with variable ppm locations, phases, and amplitudes⁴⁵. The ranges for these parameters are listed in Table 1. The final water component was scaled to be between 1× and 20× the maximum value of the frequency-domain metabolite spectrum. The water components used, along with their corresponding parametrizations, are stored within the NumPy³² archive file.

Table 1.

Parametrization of the residual water signal

Component	Location		Phase / deg		Amplitude
1	4.679	4.711	−10	10		1.00
2	4.599	4.641	15	45	.35	.55
3	4.801	4.759	−60	−30	.35	.55
4	4.449	4.541	45	−70	.10	.25
5	4.859	4.901	105	135	.10	.25

2.1.6. Frequency and Phase Shifts:

Within the NumPy³² archive file, frequency and phase shifts are specified for each entry in the dataset, but not applied to the time-domain components. Frequency shifts were sampled uniformly from the range −0.313 ppm to +0.313 ppm. Zero-order phase shifts were sampled uniformly from the range −1.57 radians to +1.57 radians. First-order phase shifts were sampled uniformly from the range −0.34 to +0.34 radians per ppm.

2.2. Exemplar Application to AGNOSTIC: Machine Learning for Out-Of-Voxel Artifacts:

The primary motivation for the AGNOSTIC dataset is as a training resource for the development of processing, modeling, and analysis tools for MRS. Synthetic spectra with known ground truths are valuable in a range of applications, from the development and validation of traditional linear combination modeling algorithms to training DL models.

In order to demonstrate the utility of the dataset, an exemplar application is presented, in which the AGNOSTIC dataset is supplemented by simulated artifacts (in this case out-of-voxel OOV echoes) and used to train DL models to detect and predict the artifact signals. The AGNOSTIC dataset was developed as building blocks which can be combined to train a variety of different models. A strength of this dataset is that custom user-defined components can be utilized. We demonstrate this point here by building an OOV dataset to train and evaluate a DL model to identify and suppress OOV artifacts.

2.2.1. Simulation of Out-Of-Voxel Echoes:

OOV artifacts were defined as complex time-domain signals with a time point $\left({\mathrm{\tau }}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$, width $\left({\mathrm{W}}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$, frequency $\left({\mathrm{\omega }}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$, phase $\left({\mathrm{\varphi }}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$ and amplitude $\left({\mathrm{a}}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$ as shown in Figure 1. ${\mathrm{\tau }}_{\mathrm{O}\mathrm{O}\mathrm{V}}$ describes the timepoint of the top of the OOV echo and was sampled randomly from a uniform distribution between 10 ms and 400 ms. ${\mathrm{W}}_{\mathrm{O}\mathrm{O}\mathrm{V}}$ describes the Gaussian decay rate and was sampled randomly from a uniform distribution between 500 s⁻² and 8000 s⁻², resulting in a FWHM echo duration between 18 ms and 74 ms. ${\mathrm{\omega }}_{\text{OOV}}$ describes the offset in the frequency domain, and was sampled randomly from a uniform distribution in order to produce OOVs that occur between 1 ppm and 4 ppm.

Figure 1. Simulation of OOV echoes and OOV-corrupted synthetic data:

OOV echoes were simulated as complex time-domain signals with a center timepoint $\left({\mathrm{\tau }}_{\text{OOV}}\right)$, width $\left({\mathrm{W}}_{\text{OOV}}\right)$, frequency $\left({\mathrm{\omega }}_{\text{OOV}}\right)$, phase $\left({\mathrm{\varphi }}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$, amplitude $\left.{(\mathrm{a}}_{\mathrm{O}\mathrm{O}\mathrm{V}}\right)$. OOV echoes were added to 85% of synthetic data to create datasets for training and evaluation.

$$OutofVoxelEcho=a_{OOV}\left(e^{-W_{OOV}{\left(t-{\tau }_{OOV}\right)}^2}\right)\left(e^{-i\omega t}\right)\left(e^{-i\Phi }\right)$$ [1]

2.2.2. Integration of OOV Echoes into AGNOSTIC for the Training Dataset:

To build the training dataset, we combined metabolite, water, MM, and noise components from the AGNOSTIC dataset. We then added OOV signals to 85% of the dataset and a complex zeros array in the remaining 15%. Finally, we applied the included frequency and phase shifts specified within the AGNOSTIC dataset. The network input consisted of the combined metabolite, water, MM, noise, and OOV signals as a complex time-domain signal. This input was normalized so that the absolute maximum among the real and imaginary values was 1. Finally, training data were converted to a TensorFlow Dataset⁴⁶.

2.2.3. Detection Network:

The first exemplar network is designed to detect OOV echoes within time-domain data by identifying the points in the spectra that have been contaminated by OOV echoes. This Detection Network is a fully Convolutional Neural Network (CNN) designed using TensorFlow2⁴⁶ with Keras⁴⁷ in a Python 3³¹ environment. The network consists of contracting encoding layers and expanding decoding layers with a total of 1.543 million parameters, as shown in Figure 2. Each layer was initialized (kernel_initializer) with “he_normal”⁴⁸. Each convolutional layer (except the output layer which uses a sigmoid activation) includes batch normalization and a leaky rectified linear unit (ReLu) activation function⁴⁹. The network is designed to receive a time-domain input signal and return a binary mask of the same size as the input with ones placed in OOV-detected regions and zeros elsewhere. A ground-truth binary mask was determined as the 5% level of the Gaussian OOV kernel. For training, the input and output of this network is a 60 × 2048 × 2 × 1 tensor, where 60 is the batch number of input examples, 2048 is the number of time points, 2 is the real/imaginary dimension, and 1 is the channel dimension.

Figure 2. Convolutional Neural Network Architecture, Input, and Output:

A) Fully convolutional neural network architecture used for both the Detection and Prediction Network. Convolutional strides, batch normalization, and Leaky ReLu activation functions are denoted by a colored line. Dark gray blocks represent complex data with the 2nd dimension representing real and imaginary components, while white blocks represent the network abstracted single dimension. Arrows show residual connections. Note, inputs and outputs are all time-domain signals; Frequency-domain is shown for convenient visualization. B) OOV-corrupted synthetic example and the isolated OOV. The complex OOV-corrupted data was used as the Detection and Prediction Network input. The target Output is the isolated OOV.

The Dice coefficient^50–52 of the overlap between the network output and the correct binary OOV location vector was used as a training loss function, calculated as 2x the intersection divided by the union plus 1; where 1 was used to avoid division by 0. The Adam optimizer was used with a learning rate of 0.0003. Training was performed using an 8 GB NVIDIA GeForce RTX 3070 GPU. A clustering algorithm was applied to the final network output, which zeroed any group of time points in which the network detected OOV echo that was smaller than 5 consecutive time points, to dampen spurious output.

2.2.4. Modeling:

Modeling of the OOV echoes was performed as an optimization problem and solved with SciPy⁵³ minimization routines. Here, the non-gradient Powell^54,55 optimizer was used to determine the five OOV parameters (${\mathrm{\tau }}_{\mathrm{O}\mathrm{O}\mathrm{V}},{\mathrm{W}}_{\mathrm{O}\mathrm{O}\mathrm{V}},$ ${\mathrm{\omega }}_{\mathrm{O}\mathrm{O}\mathrm{V}},$ ${\mathrm{\varphi }}_{\mathrm{O}\mathrm{O}\mathrm{V}}$, and ${\mathrm{a}}_{\mathrm{O}\mathrm{O}\mathrm{V}}$), minimizing the mean squared error (MSE) between the model and the data within the time window identified by the Detection Network. Initial values for ${\mathrm{\tau }}_{\mathrm{O}\mathrm{O}\mathrm{V}},$ ${\mathrm{W}}_{\mathrm{O}\mathrm{O}\mathrm{V}}$, and the ${\mathrm{a}}_{\mathrm{O}\mathrm{O}\mathrm{V}}$ are inferred from the Detection Network’s output center timepoint, the detection duration, and the standard deviation of the target signal within the detected region.

Optimization was performed as three sequential optimization steps. The first optimization is used to determine ${\mathrm{\tau }}_{\mathrm{O}\mathrm{O}\mathrm{V}},{\mathrm{W}}_{\mathrm{O}\mathrm{O}\mathrm{V}}$, and the ${\mathrm{a}}_{\mathrm{O}\mathrm{O}\mathrm{V}}$ by minimizing the MSE between the absolute values of the model and the data (i.e., removing frequency and phase from the model) in the time domain. The second optimization determines ${\omega }_{\text{OoV}}$ by absolute-mode minimization in the frequency domain. The third optimization refines the values determined by optimizations 1 and 2 and determines ${\mathrm{\varphi }}_{\mathrm{O}\mathrm{O}\mathrm{V}}$ by complex optimization in the time domain.

2.2.5. Prediction Network:

The second exemplar network is designed to predict the OOV echoes found within time-domain data. This prediction network is also a CNN designed using TensorFlow2⁴⁶ with Keras⁴⁷ in a Python 3³¹ environment, with the same architecture as the detection network (as shown in Figure 2). The network is designed to receive a time-domain input signal containing a combination of the “correct” time-domain signal and the OOV artifact and return a time-domain output signal that only contains the OOV signal, amplified 10x, where the amplification served to globally weight the entire echo. For training, a weighted mean squared error was used as a loss function with an ADAM⁵⁶ optimizer and learning rate of 0.0003. Training was again performed on an 8 GB NVIDIA GeForce RTX 3070 GPU.

2.2.6. Evaluating the Performance of Networks and Modeling:

In the final testing set, OOV artifacts were present in 6,137 of the total 7,200 examples (85.2%). The detection network was evaluated using the Dice coefficient^50–52, the overlap between the ground-truth binary OOV mask and the cluster-thresholded network output. As well as computing global success, the dependence of detection success on various attributes of the OOV echo and the underlying spectrum were also investigated.

Both modeling and the prediction network return a pure OOV signal, and in both cases, the MSE between the prediction/model and the ground-truth OOV echo is used for evaluation. If the ground-truth echo datapoints are ${\mathrm{E}}_{\mathrm{i}}$ and the model or echo prediction is ${\mathrm{M}}_{\mathrm{i}}$, we calculate the fractional remaining OOV amplitude as:

$$\text{Fractional}\text{OOV}\text{Remaining}=\frac{\sum {|M_i-E_i|}^2}{\sum {|E_i|}^2}$$ [2]

where the bars represent the complex amplitude. The sum is taken over the ground-truth range of the OOV echo. In order to visualize a wide range of success and failure, we take the ${\text{log}}_{10}$ of this quantity for plotting (i.e., a ${\text{log}}_{10}$ value of 1 is no change, anything positive is a manipulation that is worse than doing nothing, and a negative value show the order of magnitude of improvement). Note that $E_i$ is the ground-truth echo signal, not the signal from which the echo is being removed which also contains metabolite, macromolecule, and noise components.

The timing of the OOV was found to be a key parameter determining the success of detection and prediction, and as a result, the evaluation metrics were calculated for the following time-bins (based on the known value of ${\mathrm{\tau }}_{\mathrm{O}\mathrm{O}\mathrm{V}}$): 10–20 ms; 20–40 ms; 40–60 ms; 60–80 ms; 80–120 ms; 120–200 ms; 200–300 ms; 300–400 ms.

2.2.7. In Vivo Proof-of-Principle

As a proof-of-principle demonstration of this exemplar use of the AGNOSTIC dataset, the network was applied to 256 transients of in vivo data, selected because they contain prominent OOV echoes and were excluded during quality assessment in a recent study⁵⁷. These data were collected on a 2.89 T Siemens scanner using the MEGA-PRESS^58,59 pulse sequence with a TE of 68 ms and TR of 1.75 s, and a spectral width of 2.4 kHz. Note that this challenges the generality of the training because the network has never seen data acquired at 2.89 T, nor at 2.4 kHz spectral width, nor at TE 68 ms, nor with MEGA-Editing, nor with actual real RF pulses. Raw data from a 25 × 25 × 25 mm³ voxel in the cerebellum were loaded and coil combined in Osprey⁶⁰. Time-domain data were saved as a MATLAB⁶¹ .mat file and loaded as a Python 3³¹ object using SciPy⁵³. The data were normalized (as above with training data) to be used as input for the neural networks.

One challenge of in vivo data (and the reason that this network demonstration focuses substantially on synthetic data) is that no ground truth is available. Therefore, the degree of success in removing OOV echo signals from time-domain data $D_i$ is:

$$\text{Fractional}\text{Reduction}\text{in}\text{standard}\text{deviation}=1-\frac{\sigma \left(M_i-D_i\right)}{\sigma \left(D_i\right)}$$ [3]

where $\mathrm{\sigma }$ denotes the standard deviation. Note that, in contrast to the metric used for synthetic data in Equation 1, only $D_i$ is available, not the ground truth $E_i$, which substantially changes the ceiling of success. It is still expected that substantial signal variance remains after OOV removal, since $D_i$ contains metabolite signals and noise. The range over which this standard deviation is calculated is the 50% level of the normalized histogram of the detection network’s output across the 128 averages.

3. Results:

3.1. AGNOSTIC Synthetic Dataset:

The AGNOSTIC dataset contains 259,200 examples, consisting of 960 examples from each of the eighteen field strengths and fifteen echo times (i.e., $960\times 18\times 15=\mathrm{259,200}$). A representative set of ten spectra are shown in Figure 3, illustrating the diversity of field strengths, TEs, SNR, and linewidth within the dataset.

Figure 3. AGNOSTIC synthetic dataset:

10 representative spectra from the AGNOSTIC dataset. The 10 examples show the diversity of field strength, TE, linewidths, and residual water signal present among the data. Note, examples are shown here in the frequency-domain to better illustrate the heterogeneity, but the dataset provides time-domain examples.

One challenge with making this dataset available is its size — 75 GB — but we do make it freely available on Dryad. The basis sets from which these are constructed are more manageable — 9 GB — and can also be accessed through Dryad. Code for generating the AGNOSTIC dataset locally is available at: https://github.com/agudmundson/agnostic.

3.2. Exemplar Application to AGNOSTIC: Machine Learning for Out-Of-Voxel Artifacts:

3.2.1. Detection Network:

Of the 6,137 examples where OOV artifacts were present, the Detection Network correctly identified 5,827 (94.9%) with a median Dice score of 0.974 (0.941–0.985 interquartile range) and missed 310 (5.05%) with a Dice score of 0.00. In the 1063 examples that did not include OOV artifacts, the network correctly ignored 912 (85.8%) and falsely detected OOV echoes in 151 (14.2%). Figure 4 shows the Detection Network’s output for a synthetic OOV-corrupted example.

Figure 4. OOV-corrupted example:

OOV-corrupted synthetic example and the isolated OOV. Results from Detection Network (green), Model (orange), and Prediction Network (blue) are shown below the ground truth OOV-corrupted and OOV. OOV residuals are shown for the Model (orange) and Prediction Network (blue) demonstrating remaining signal after subtraction. Note, frequency-domain is shown for convenient visualization, but the Detection Network, Modeling, and Prediction Network all operate on time-domain signals.

Analysis of the factors that determined success indicated that the time at which OOV signals occur is most critical. Therefore, OOV echoes were further broken down into eight time-bins, and the Dice score plotted in Figure 5. The median Dice scores — 0.165, 0.858, 0.892, 0.934, 0.960, 0.974, 0.978, and 0.978 — are poor in the first bin and improve thereafter. Note that these bins are not spaced equally to emphasize poor performance extremely early. The number of examples in each bin is 161, 289, 282, 329, 622, 1256, 1565, and 1633, respectively.

Figure 5. Evaluation of Detection Network, Modeling, and Prediction Network:

A testing set with 7200 (2400 examples with 3 different OOV echoes) unseen examples was used to evaluate the A) Detection Network and B) Modeling and Prediction Network. Performance across the whole test set is shown on the left-hand side. Performance across the binned center timepoint $\left(\mathrm{\tau }\mathrm{O}\mathrm{O}\mathrm{V}\right)$ is shown across the right-hand side.

3.2.2. Modeling

The modeling optimization converged in 5,824 of the 5,827 examples where the detection network detected OOV artifacts and provided initial values. Across this subset of the examples, the modeling achieved a median ${\text{log}}_{10}$ (fractional OOV remaining) of −2.19 (−2.90 – −1.19 inter-quartile range), i.e., a median reduction of more than two orders of magnitude. Figure 5 shows the resulting model for a synthetic OOV-corrupted example.

These values — broken down into 8 time-bins — are shown in Figure 5. The median ${\text{log}}_{10}$(fractional OOV remaining) decreases across the time bins: 1.663, 1.324, 0.680, 0.223, −1.586, −2.276, −2.491, and −2.567.

3.2.3. Prediction Network:

In the 6,137 examples where OOV artifacts were present, the prediction network achieved a median ${\text{log}}_{10}$ normed-MSE of −1.79 (−2.21 – −1.11 inter-quartile range). In the 5,824 examples where OOV artifacts were successfully modeled, the prediction network achieved a median ${\text{log}}_{10}$ normed-MSE of −1.85 (−2.24 – −1.24 inter-quartile range). Figure 5 shows the Prediction Network’s output for a synthetic OOV-corrupted example. OOVs were further broken down into 8 time-bins (Figure 5) early — the number of examples in each bin is 86, 226, 261, 312, 592, 1208, 1538, 1601. The median ${\text{log}}_{10}$(fractional OOV remaining) decreases across the time bins: −0.207, −0.583, −0.862, −1.250, −1.577, −1.878, −2.005, and −2.052.

3.2.4. In Vivo Proof-of-Principle:

The Detection Network identified an OOV in 243 of 256 transients (94.9%). In these 243 OOV-detected transients, the modeling achieved a median reduction in standard deviation of 71.0 % (60.2 −75.3% inter-quartile range). The Prediction Network achieved a median reduction in standard deviation of 69.65% (66.33 %/72.7 % inter-quartile range) in this subset. In the full set of 256 transients, the Prediction Network achieved a median 69.4 % (65.3 – 72.6 % inter-quartile range) reduction in standard deviation. The standard deviation of the noise floor was found to account for a median of 10.3% (9.35–11.6 % inter-quartile range) of the standard deviation of signal within the time window for the 256 averages. A representative in vivo example is shown in Figure 6.

Figure 6. In Vivo MEGA-PRESS Example:

In vivo OOV-corrupted example. Results from Detection Network (green), Model (orange), and Prediction Network (blue) are shown below. Note, frequency-domain is shown for convenient visualization, but the Detection Network, Modeling, and Prediction Network all operate on time-domain signals.

4. Discussion:

AGNOSTIC is a benchmark MRS dataset for training and evaluating performance across various models. In order to make these synthetic data representative of in vivo brain MRS datasets, a total of 22 brain metabolites and 14 MM peaks were simulated within 270 basis sets, spanning field strengths from 1.4 T to 3.1 T and TEs from 10 to 80 ms. Parameterized water residual and noise were included. SNR and linewidths were assigned at random, independent of B₀ or TE. The broad span of the dataset is key in training networks that generalize. While AGNOSTIC is broad in these dimensions, it does only represent simulated data for PRESS³⁷ acquisitions, and may benefit from expansion to include other pulse sequences, such as STEAM⁶², SPECIAL^63,64, LASER⁶⁵, and semi-LASER^66,67, and edited versions including MEGA^58,59 and Hadamard-encoded^68–71 schemes. AGNOSTIC simulations used ideal pulses, and thus fail to capture effects associated with spatially heterogeneous coupling evolution. The extent to which these limitations matter will depend on the applications that AGNOSTIC synthetic data are being used for.

The Detection network was highly successful, identifying 94.9% of the testing set where OOV artifacts were present. The precise value of this success metric is obviously impacted by the parameters of the OOVs – a later minimum OOV time would tend to increase performance, and earlier would degrade it. It is noteworthy that, although the training datasets never contained more than one OOV echo, the detection and prediction networks were able to handle more than one OOV echo in vivo data, presumably because CNNs operate locally within the FID. It is also encouraging that the networks generalized well to the in vivo data (Figure 6), which was collected with unseen acquisition parameters, i.e., edited MEGA-PRESS^58,59 data acquired at 2.89 T with a TE of 68 ms, and 2.4 kHz spectral width.

In the exemplar OOV application, the success of the networks depended heavily on the timing of the OOV signal. The earliest OOV echoes were most challenging, unsurprisingly since such signals are broad Gaussian resonances that are indistinguishable from within-voxel MM and baseline signals. Indeed, the only feature that differentiates OOV signals from other broad components of the model is timing. It is conceptually helpful to consider this in the Fourier domain, even though all network processing is performed in the time domain. In the frequency domain, a mismatch between the echo-top and the acquisition start is represented as a first-order phase error of the signal associated with that echo. Where insufficient first-order phase exists to be represented within the linewidth of the signal in question, (which in the time domain corresponds to substantial truncation of the lefthand side of the echo), the network struggles to identify OOV signals.

In the context of this study, modeling and prediction are treated as two alternative approaches to OOV characterization. For early OOV signals, the modeling approach tended to mis-attribute non-artifact signal as OOV signal, a result that the metric scored as worse than no intervention. The median performance of the Prediction network, even for very early OOV signals, was close to zero. Both modeling and prediction performance improve as the OOV moves later in the acquired signal, with modeling improving faster than the network, and performing better than prediction beyond 120 ms. This strong performance of the model at least in part reflects the exact match between the generative model of the synthetic OOV artefacts and the model that is being used to extract them. More moderate performance might be expected for real in vivo examples – but the same may also be true for networks which have been trained with the same synthetic data and may have learned specifically to identify OOV signals that have a Gaussian kernel.

One key difference between most DL applications and applications in MRS, is the strict requirement to preserve amplitude fidelity in network outputs. A common approach to artifacts in DL is to return an artifact-free version of the network input. In contrast, the approach taken here is to return the artifact, which has the following benefits: it avoids networks over-learning the formulaic pattern of typical spectra; it reduces the impact of the lack of sequence diversity within the AGNOSTIC dataset; and it is less likely to impact the amplitudes of metabolite signals.

The ultimate goal of this work is to extract metabolite levels from MRS data that are not impacted by OOV artifacts. This problem can be addressed at several points: either by not acquiring data that contain OOV artifacts; by removing OOV artifacts post-acquisition; and by incorporating appropriate OOV model components into quantification model so that the impact of OOV is minimized. While the work presented here focuses primarily on the second context, it raises important potential applications in the other contexts. One motivator for developing the Detection network is the possibility of real-time deployment during sequence acquisition to trigger sequence changes when OOV artifacts are detected. The modeling applied here was time-restricted to a given window and ignored other components of the spectrum, but demonstrates potential for future integration within a full linear-combination model.

In conclusion, we have presented the AGNOSTIC dataset for deep learning in MRS and demonstrated an exemplar use case to develop CNNs to detect and predict out-of-voxel artifacts.