Input layer regularization for magnetic resonance relaxometry biexponential parameter estimation

Abstract

Many methods have been developed for estimating the parameters of biexponential decay signals, which arise throughout magnetic resonance relaxometry (MRR) and the physical sciences. This is an intrinsically ill-posed problem so that estimates can depend strongly on noise and underlying parameter values. Regularization has proven to be a remarkably efficient procedure for providing more reliable solutions to ill-posed problems, while, more recently, neural networks have been used for parameter estimation. We re-address the problem of parameter estimation in biexponential models by introducing a novel form of neural network regularization which we call input layer regularization (ILR). Here, inputs to the neural network are composed of a biexponential decay signal augmented by signals constructed from parameters obtained from a regularized nonlinear least-squares estimate of the two decay time constants. We find that ILR results in a reduction in the error of time constant estimates on the order of 15%–50% or more, depending on the metric used and signal-to-noise level, with greater improvement seen for the time constant of the more rapidly decaying component. ILR is compatible with existing regularization techniques and should be applicable to a wide range of parameter estimation problems.

1 ∣. INTRODUCTION

1.1 ∣. Background: Biexponential analysis

Biexponential models occur throughout magnetic resonance relaxometry and related experiments^[1-5] as well as in chemistry,^[6-8] biomedicine,^[9-14] population dynamics,^[15] and physics.^[16-18]

This model is of the form:

$$s(t;T_{2,1},T_{2,2},c_1,c_2)=c_1{e}^{-t∕T_{2,1}}+c_2{e}^{-t∕T_{2,2}}+\eta (t),$$ (1)

with $s(t)$ indicating signal amplitude at measurement time $t$. In other settings, the data acquisition variable may represent an indirectly detected time, or a diffusion $b$ value. The model is linear in the initial population fractions, $c_1$, $c_2>0$, but nonlinear with respect to the corresponding decay constants 0 ms $<T_{2,1}<T_{2,2}$. The function $\eta (t)$ is a random process representing noise. We will use biomedical magnetic resonance relaxometry (MRR), in which $T_{2,1}$ and $T_{2,2}$ are transverse relaxation times, as our illustrative example^[5]; the two components then represent two distinct water compartments in tissue. We further restrict our analysis to the two-parameter problem of known $c_1$ and $c_2$ and unknown $T_{2,1}$ and $T_{2,2}$. This is somewhat restrictive but suffices to illustrate the current status of our developments.

The noiseless, discretized version of Equation (1) is

$$\mathbf{G}(\mathbf{p})\stackrel{\text{def}}{=}(c_1{e}^{-t_n∕T_{2,1}}+c_2{e}^{-t_n∕T_{2,2}}{)}_{n=1}^N,$$ (2)

indicating signal acquisition at times $\{t_n{\}}_{n=1}^N$ where $\mathbf{p}=(T_{2,1},T_{2,2},c_1,c_2)$ is the vector of model parameters to be estimated. We take $t_1=0$, although transverse relaxometry acquisition generally starts at a minimum echo time $t_1=TE$. The model $\mathbf{G}$ is an explicit function of the vector of model parameters; its dependence on measurement times is understood but often omitted in what follows.

Many approaches have been developed to estimate $\mathbf{p}$,^[19-21] with nonlinear least-squares^[22] (NLLS) remaining very popular. Incorporating constraints on the model's parameters, the problem is

$$\begin{gathered} {\mathbf{p}}^{\ast }\stackrel{\text{def}}{=}\underset{\mathbf{p}\in F}{\text{argmin}}\Vert \mathbf{G}(\mathbf{p})-\mathbf{s}{\Vert }_2^2 \\ =\underset{\mathbf{p}\in F}{\text{argmin}}\left[\sum _{n=1}^N(c_1{e}^{-t_n∕T_{2,1}}+c_2{e}^{-t_n∕T_{2,2}}-s_n{)}^2\right]. \end{gathered}$$ (3)

Above, $\mathbf{s}=\{s_n{\}}_{n=1}^N$ with $s_n=s(t_n)$ is a vector containing the experimental signal. Also, $\Vert \mathbf{a}{\Vert }_p\equiv {\left({\sum }_{n=1}^N\mid a_n{\mid }^p\right)}^{1∕p}$ for p ≥ 1 is the ${\ell }_p$-norm of an $N$-vector $\mathbf{a}$. In NLLS, the estimate of the true parameters ${\mathbf{p}}^{\ast }$ is one that minimizes the sum of squared residuals $r_n^2=(c_1{e}^{-t_n∕T_{2,1}}+c_2{e}^{-t_n∕T_{2,2}}-s_n{)}^2$ over a feasible set of parameters $F$. The choice of $F$ permits the introduction of prior information, such as nonnegativity bounds. For MRR, we define our estimated $T_{2,1}$ and $T_{2,2}$ values as $T_{2,1}^{est}$ and $T_{2,2}^{est}$ respectively.

More recently, neural networks (NNs) have been used for parameter estimation, including for biexponential models.^[23-27] Training requires a large set of biexponential curves with known parameters; this is generally not available experimentally, but may readily be simulated in the case of a known signal model with well-understood noise properties. A choice must also be made for the distribution of the noise $\eta (t)$, which can be taken as Gaussian for MRR obtained in a spectroscopic mode, as in this work, or Rician or higher-order chi-squared distributions for multi-coil image acquisition. Conveniently, knowledge of reasonable parameter ranges may be readily incorporated into construction of the training data.

Estimating the parameters of a biexponential model is inherently difficult, with distinct sets of parameters yielding nearly indistinguishable decay curves; this is one hallmark of an ill-posed inverse problem for parameter estimation. This results in a high degree of instability of derived solutions with respect to even small amounts of noise; stabilization of ill-posed inverse problems is therefore a major topic of investigation.^[28] An important strategy is to replace the original parameter estimation problem with a related but more stable problem; this is one definition of regularization. Available methods exhibit a trade-off between estimation variance, which is decreased by the regularization procedure, and bias, which is increased.^[29]

1.2 ∣. Regularization

Tikhonov regularization is accomplished through the addition of a penalty term to the objective function that is to be minimized, so that Equation (3) is replaced by the regularized nonlinear least-squares problem (R-NLLS):

$${\mathbf{p}}_{\lambda }^{\ast }\stackrel{\text{def}}{=}\underset{\mathbf{p}\in {\mathbb{R}}_{+}^2}{\text{argmin}}(\Vert \mathbf{G}(\mathbf{p})-\mathbf{s}{\Vert }_2^2+{\lambda }^2\Vert \mathbf{p}{\Vert }_2^2),$$ (4)

where $\lambda >0$ is called the regularization parameter. This balances the size of the ${\ell }_2$-norm of the solution against the objective function, avoiding a fine-structured solution that fits the noisy data well but generalizes poorly to additional data sets.^[30] Qualitatively, increasing $\lambda$ biases the length of the parameter vector $\Vert \mathbf{p}{\Vert }_2$ to be small. In the limit $\lambda \to \infty$ the penalty term dominates, and the solution satisfies ${\mathbf{p}}_{\lambda }^{\ast }\to 0$. Conversely, decreasing $\lambda$ diminishes the impact of the penalty term, and in the limit $\lambda \to 0$ the solution of Equation (4) converges to the solution of (3), that is, ${\mathbf{p}}_{\lambda }^{\ast }\to {\mathbf{p}}_0^{\ast }={\mathbf{p}}^{\ast }$. A number of methods are available to analogously regularize NNs.^[31-33]

Equation (4) is a conventional expression for Tikhonov regularization of a nonlinear inverse problem. However, it can be difficult to ascribe physical significance to the penalty term; for example, in our case, there is no physical basis for limiting the size of $\Vert \mathbf{p}{\Vert }_2^2=\mid T_{2,1}^{est}{\mid }^2+\mid T_{2,2}^{est}{\mid }^2$. This is in contrast to the common case in MR in which the recovered $\mathbf{p}$ represents a probability distribution function, describing in effect the number of spins relaxing at a given $T_2$ over a grid of possible values. In that case, it is physically meaningful to impose properties such as sparsity or smoothness on the solution. Limitations on solution norm or total variation may also be imposed. These considerations do not hold for the problem at hand or for many nonlinear inverse problems with essentially uncorrelated parameters, but it remains true that regularization decreases parameter instability with respect to noise at the expense of introducing additional bias into parameter estimation. In effect, here we are incorporating a penalty term for mathematical rather than for physical reasons.

Our present work involves the incorporation of two separate concepts. First, we establish a conventional NN framework for decay time constant estimation in the biexponential model.^[23] Second, we provide a novel means of regularization of the NN; we include a regularized version of the biexponential data to the input layer to improve stability.

An important perspective on our method is the recognition that NNs provide a natural means for combining distinct sources of information; rather than replacing the original ill-posed problem by one that is related but more well-posed, we instead train a NN that maps a concatenated pair of time series $(\mathbf{s},\mathbf{G}({\mathbf{p}}_{\lambda }^{\ast }))$, respectively the noisy signal decay and a noiseless decay resulting from regularized parameter estimation, to new parameter estimates. This augments, rather than replaces, the original problem. We will show that this leads to an improvement in parameter estimation in comparison to a NN that is trained using only the noisy decay $\mathbf{s}$ as input.

2 ∣. METHODS

The workflow for the construction of synthetic data sets, training and validation of NNs, and testing is illustrated in Figure 1, with a more detailed description found in Section S1 of the supporting information (SI).

FIGURE 1.

Our procedure consists of three steps: data set generation, NN training and validation, and NN testing. A conventional neural network for parameter estimation takes as input the biexponential decay curve. We call such a network (ND, ND), where ND stands for “noisy decay.” Our input layer regularized network is denoted (ND, Reg)

2.1 ∣. Neural network architectures

To demonstrate ILR, we separately trained and compared the performance of two NNs with the same architectures but differing in their input, denoted $\mathbf{X}$. The first, which we call (ND, ND), takes as input a concatenated vector of two identical time series $\mathbf{X}=(\mathbf{s},\mathbf{s})$, where $\mathbf{s}$ is a synthetic noisy decay (ND). The second, called (ND, Reg), takes as input a concatenated vector of two different time series $\mathbf{X}=(\mathbf{s},\mathbf{G}({\mathbf{p}}_{\lambda }^{\ast }))$, where $\mathbf{G}({\mathbf{p}}_{\lambda }^{\ast })$ is generated from parameters estimated with R-NLLS, that is, Equation (4). Thus, the input layers of (ND, Reg) and (ND, ND) have the same length, $2N$, identical elements in the first $N$ slots, and different elements in the second $N$ slots.

Both networks, (ND, ND) and (ND, Reg), were structured with an input layer of 128 nodes, four hidden layers of 32, 256, 256, and 32 nodes, and an output layer with two nodes defining the parameter estimates $T_{2,1}^{est}$ and $T_{2,2}^{est}$. This architecture was based on past work^[25,34] that used NNs for related problems.

We compared (ND, Reg) with (ND, ND) rather than comparing (ND, Reg) with a network, called (ND), with input layer $\mathbf{X}=\mathbf{s}$, in order to avoid comparing NN's with different input layer widths. Nevertheless, we have found that (ND, ND) performs virtually identically to (ND).

We consider (ND, ND) to be the baseline NN for parameter estimation since it directly takes noisy experimental data as input and yields an estimate of the decay constants $(T_{2,1}^{est},T_{2,2}^{est})$.^{[23,24,26,34,35]} In contrast, (ND, Reg) represents a novel NN input layer architecture, and comparison of this with (ND, ND) is the topic of this manuscript.

2.2 ∣. Training and validation

Details regarding training and validation can be found in Section S1.3 of the SI. We note that we have selected $c_1=0.6$ and $c_2=0.4$ throughout, although other values yielded very similar results (not shown).

2.3 ∣. Testing

We evaluated the performance of each trained NN using four distinct metrics calculated from its predictions using the pairs of input and target samples $(\mathbf{X},\mathbf{y})$ in the testing data set $\mathbb{T}$. These were: mean percent error (MPE), bias, variance, and mean squared error (MSE). Each of these provides independent insight into performance. Mean percent error is an easily-interpreted metric of average performance expressed in relative terms. Bias and variance are essential elements in the analysis of regularization techniques, providing respectively a measure of accuracy and precision, with $MSE=bia{s}^2+variance$. We computed two versions of each of these four metrics. The first version is a function of two variables; it depends on the pair of target decay time constants ($T_{2,1}^{true}$, $T_{2,2}^{true}$). The second version is a function of one of these variables, obtained by averaging over the other. Such averaged results are presented in the SI for MPE and MSE. The functions of both $T_{2,1}^{true}$ and $T_{2,2}^{true}$ are denoted with uppercase letters. We use a subscript, 1 or 2, on the function name to indicate whether it describes $T_{2,1}^{est}$ or $T_{2,2}^{est}$. The precise definitions of these metrics are found in Section S1.4 of the SI.

3 ∣. RESULTS

Comparisons of (ND, Reg) with (ND, ND) will be based on the four metrics defined above. The main results can be found below, with further details presented in Section S2 of the SI.

3.1 ∣. Mean percent error performance

We wish to compare the two NNs based on their combined performance in estimating $T_{2,1}$ and $T_{2,2}$; accordingly, we defined a metric $MP{E}_{diff}$:

$$\begin{gathered} MP{E}_{diff}(T_{2,1}^{true},T_{2,2}^{true})\stackrel{\text{def}}{=}[MP{E}_1^{(\text{ND},\text{ND})}(T_{2,1}^{true},T_{2,2}^{true})\phantom{]} \\ +\phantom{[}MP{E}_2^{(\text{ND},\text{ND})}(T_{2,1}^{true},T_{2,2}^{true})] \\ -[MP{E}_1^{(\text{ND},\text{Reg})}(T_{2,1}^{true},T_{2,2}^{true})\phantom{]} \\ +\phantom{[}MP{E}_2^{(\text{ND},\text{Reg})}(T_{2,1}^{true},T_{2,2}^{true})], \end{gathered}$$ (5)

where $MP{E}_1$ and $MP{E}_2$ are defined by Equation (S4).

Figure 2 shows results for the metric defined in Equation (5) as a heat map. Blue color indicates superiority of (ND, Reg), while red indicates superiority of (ND, ND). Each pixel represents an average over $N_{\omega }=20$ noise realizations. Finally, note that the convention $T_{2,2}^{true}\ge T_{2,1}^{true}$ leads to results being generated only for the upper left half of the parameter space defined by the heat map axes. One region of particular interest is on the left, where $T_{2,1}^{true}$ values are small and hence difficult to estimate due to the rapid decay of the corresponding signal component. The diagonal region, where $T_{2,1}^{true}\approx T_{2,2}^{true}$, also represents a zone of parameters where estimation is difficult.^[36]

FIGURE 2.

Heat maps defining the relative mean percent error ($MPE$) performance of the two NNs, according to Equation (5) with averaging over $N_{\omega }=20$ noise realizations, as a function of ($T_{2,1}^{true}$, $T_{2,2}^{true}$). Blue regions indicate superiority of (ND, Reg) and red the superiority of (ND, ND). Results are shown for SNR = 1 (left), where neither method performs well, and SNR 30 (center) and 100 (right), where (ND, Reg) outperforms (ND, ND) over the majority of the heat map region. The regularization parameter was fixed at $\lambda =1.6\times {10}^{-4}$.

At this stage of development, a fixed $\lambda$ is used throughout all of parameter space, and so represents a compromise to achieve the overall best results over the entire heat map. With this, it is clearly seen that the blue regions dominate the red regions for moderate-to-high SNR values, with the exception of certain regions near the diagonal. This indicates the fact that tuning the regularization parameter $\lambda$ to this region would be particularly important. The results for SNR = 1 indicate that neither method can perform well in that regime.

3.2 ∣. Bias-variance performance

We examine the accuracy, represented by bias, and precision, represented by variance, of (ND, Reg) and (ND, ND) as estimators of the decay constants. Smaller variance indicates less sensitivity to noise.

Figure 3 shows results for $T_{2,1}^{est}$ computed via Equations (S6) and (S7), respectively. Three SNR levels are shown with a fixed $\lambda =1.6\times {10}^{-4}$. For low SNR, (ND, ND) and (ND, Reg) perform virtually identically and poorly. However, for larger SNR, (ND, Reg) is very effective at limiting bias compared with (ND, ND) across the full range of target $T_{2,1}^{true}$ values. Variance, on the other hand, is similar between the two NNs. We attribute these results to the fact that while the introduction of regularization generally leads to decreased variance at the expense of increased bias,^[30] in the present analysis the NNs are, in effect, trained on bias as the loss function. Comparable results were obtained for bias and variance in the estimation of $T_{2,1}^{true}$, although results were more comparable between the two NNs. See SI, Figure S5.

FIGURE 3.

Bias ($b_1$) (left) and variance ($va{r}_1$) (right) in $T_{2,1}^{est}$ as defined by Equations (S6) and (S7) for (ND, ND) (green) and (ND, Reg) (blue) for three levels of SNR. The regularization parameter is fixed at $\lambda =1.6\times {10}^{-4}$. The location of the maximum error is indicated with a vertical line and the value of the and mean error is indicated by a horizontal line

3.3 ∣. Mean squared error performance

3.3.1 ∣. Mean squared error as a function of one target decay constant

Figure 4 shows the comparison of (ND, Reg) and (ND, ND) in terms of average mean squared error in $T_{2,1}^{est}$ and $T_{2,2}^{est}$. As was also seen in Figure S2, both methods fail at the lowest values of SNR. For larger SNR, (ND, Reg) consistently outperforms (ND, ND), with improvements in $T_{2,1}^{est}$ of ~20% for SNR = 10, ~50% for SNR = 30, and ~75% for SNR = 100. More modest, but nevertheless substantial, improvements were seen for $T_{2,2}^{est}$, of ~15% for SNR = 30 and ~ 33% for SNR = 100.

FIGURE 4.

Average mean squared error ($mse$) for $T_{2,1}^{est}$ (top) and $T_{2,2}^{est}$ (bottom) defined by Equation (S9) across five SNR values from 1 to 100. The regularization parameter was fixed at $\lambda =1.6\times {10}^{-4}$. Values on the blue (for (ND, Reg)) and green (for (ND, ND)) curves (left-side ordinate) correspond to the average value of mse₁ (upper panel) over all target values of $T_{2,1}^{true}$ or $ms{e}_2$ (lower panel) over all target values of $T_{2,2}^{true}$ in the testing set. Average percent improvement ($api$) as defined by Equation (S10) of (ND, Reg) compared with (ND, ND) is displayed on the right-side ordinate. We see that $MSE$ of (ND, Reg) is lower than that of (ND, ND), indicating improvement through ILR. Greater improvements are seen for larger values of SNR (red curve). Overall, the mean squared error metric shows results similar to those shown in Figure S2 for average $mpe$

3.2.2 ∣. Heat map of the sum of mean squared error over target decay constants

Analogous to Equation (5) in Section 3.1, we define a mean squared error difference metric $MS{E}_{diff}$ to quantify the difference between the performance of the two NNs:

$$\begin{gathered} MS{E}_{diff}(T_{2,1}^{true},T_{2,2}^{true})\stackrel{\text{def}}{=}[MS{E}_1^{(\text{ND},\text{ND})}(T_{2,1}^{true},T_{2,2}^{true})\phantom{]} \\ +\phantom{[}MS{E}_2^{(\text{ND},\text{ND})}(T_{2,1}^{true},T_{2,2}^{true})] \\ -[MS{E}_1^{(\text{ND},\text{Reg})}(T_{2,1}^{true},T_{2,2}^{true})\phantom{]} \\ +\phantom{[}MS{E}_2^{(\text{ND},\text{Reg})}(T_{2,1}^{true},T_{2,2}^{true})] \end{gathered}$$ (6)

where $MS{E}_1$ is the mean squared error in $T_{2,1}^{est}$ and $MS{E}_2$ is the mean squared error in $T_{2,2}^{est}$ computed via Equation (S8), and the NN used is indicated by a superscript.

Corresponding heat maps as a function of ($T_{2,1}^{true}$, $T_{2,2}^{true}$) for three values of SNR are shown in Figure 5. At low SNR, both methods perform poorly and are comparable. At more realistic SNR, (ND, Reg) clearly outperforms (ND, ND) over most of the range of targets. However, (ND, Reg) performs worse than (ND, ND) over much of the diagonal region for SNR = 100. We attribute this to the fact that $\lambda$ is fixed for all target values, and this region likely is over-regularized; see SI Figure S5. Similar results were seen in Figure 2.

FIGURE 5.

Heat maps defining the relative mean squared error ($MSE$) performance of the two NNs, $MS{E}_{diff}$ according to Equation (6) with averaging over $N_{\omega }=20$ noise realizations. Blue regions indicate superiority of (ND, Reg) and red the superiority of (ND, ND). Results are shown for SNR = 1 (left), where neither method performs well, and SNR 30 (center) and 100 (right), where (ND, Reg) outperforms (ND, ND) over the majority of the heat map region. The regularization parameter was fixed at $\lambda =1.6\times {10}^{-4}$. Note that the scale of the color bars varies between the three subplots

3.4 ∣. Application to myelin ${\mathit{T}}_2$ mapping of the human brain

We apply ILR to pixel-wise analysis of a sequence of $T_2-$ weighted human brain images. Each pixel represents the amplitude of a decaying biexponential as a function of echo time and was processed independently with an (ND, ND) NN and an (ND, Reg) NN. The networks were trained with an SNR of 241, which is the average of the SNR over the brain parenchyma in shortest echo-time image. Noise was added pixelwise to construct an additional data set with mean SNR = 30 for comparison purposes, and additional (ND, ND) and (ND, Reg) NNs were trained at this SNR. Further acquisition and processing details are found in Section S1.5.1 and Section S1.5.2, respectively, of the SI. Results are shown in Figure 6.

FIGURE 6.

$T_{2,1}$ and $T_{2,2}$ maps generated pixel-wise with (ND, ND) and (ND, Reg). Values for $c_1$ and $c_2$ were taken as the average of the estimates from Bi et al.^[37] Comparision is made between maps obtained at high (left; mean SNR = 241) and low (center; noise added to achieve mean SNR = 30) SNR. The right-hand panel compares the sensitivity to noise of the two NNs, as defined by Equation (7). As seen, the (ND, Reg) NN maintains its performance to a substantially greater extent than (ND, ND) as SNR decreases

There is no gold standard for these in vivo images, so that comparison of bias is problematic. However, we can instead compare (ND, ND) and (ND, Reg) with respect to sensitivity to noise and evaluate whether the introduction of ILR improves the stability of the parameter estimation problem. We quantify this by determining, for each of these two NN separately, the absolute difference in the images obtained with SNR = 241 and SNR = 30. A smaller difference, as measured by the difference between these two difference maps, indicates a desirable lower sensitivity to noise for (ND, Reg). The expression used is

$$\begin{gathered} s_{diff}(i,j)\stackrel{\text{def}}{=}\left[s_1^{(\text{ND},\text{Reg})}(i,j)+s_2^{(\text{ND},\text{Reg})}(i,j)\right] \\ -\left[s_1^{(\text{ND},\text{ND})}(i,j)+s_2^{(\text{ND},\text{ND})}(i,j)\right], \end{gathered}$$ (7)

with $s_1(i,j)$ indicating the noise sensitivity of an estimate of $T_{2,1}$ from either (ND, Reg) or (ND, ND) as defined by Equation (S11) in Section S1.5.2 of SI, and $s_2(i,j)$ the corresponding expression for $T_{2,2}$; superscripts indicate the neural network that was used. A map of this quantity is displayed in the right-hand panel of Figure 6, with the negative values seen demonstrating that (ND, Reg) is less sensitive to noise than (ND, ND) across the majority of the figure. We note the obvious difference in the mean value of $T_{2,1}$ for the two SNR levels; we attribute this to the introduction of additional bias for SNR = 30 due to the nonlinear estimation implemented by the NNs,^[38] with $T_{2,2}$ evidently being less sensitive to this effect.

4 ∣. DISCUSSION

The applicability of NNs to parameter estimation has been long-recognized,^[23] and they are now widely used for this purpose.^{[24,26,35,39]} Applications of ILR to functional forms other than the biexponential model studied here is straightforward. Models of current interest in MR include kurtosis,^[40] the stretched exponential,^[4,41] and mcDESPOT.^[42,43] Multiexponential models with more than two components may also be evaluated with, for example, six independent parameters estimated for a three-component model. However, we expect such an extension to present substantial problems both in terms of training over a large hypercube in parameter space and due to the extreme ill-conditioning of such functional forms. The current analysis is also applicable to components defined by $T_1$ values, which are less-frequently encountered due to the timescale of intercompartment exchange but still of importance. The constant offset to the exponential term in this signal model does not affect noise amplification,^[36] the calculation of which involves derivatives,^[5] so the behavior is expected to follow closely that of the biexponential $T_2$ model investigated here. Further, we expect ILR to be of use in areas outside of MR.^[44-49]

The biexponential model incorporates four unknown parameters, namely, compartment sizes and decay time constants. More practical applications would require estimation of all of these, or a three-parameter analysis when the data can be normalized. Nevertheless, the improved estimation of the two decay constants as presented here shows the potential value of the approach, and may itself have applications.^[50-53] We note that we have performed simulations for $c_1$ values ranging from 0.05 to 0.7, finding that in all cases (ND, Reg) outperforms (ND, ND) to roughly the same extent as in the results presented in detail.

The extension of our work to the estimation of all four parameters in the biexponential model, ($T_{2,1}$, $T_{2,2}$, $c_1$, $c_2$), is straightforward. The only step that changes in an essential way is data set generation, since all four parameters would require estimation by R-NLLS. Because of the disparate dimensions and scales of the time constants and component fractions, it may be worthwhile to introduce a diagonal matrix of weights to the penalty term of R-NLLS so that the magnitude of the parameters in the penalty term contribute in a roughly equal manner. A similar adjustment to the loss function during NN training may also be implemented. Similar comments apply to parameter estimation for other models.

ILR may also be useful for classification problems. In image classification, for example, a NN is trained on and applied to vectorized versions of the input image. ILR should improve the accuracy of matching this data structure to an output class or to output classes in the presence of noise.^[54]

Overfitting, viewed as creation of a NN model with weights that are finely-tuned to a particular data set, is a major potential pitfall in NN design and leads to models that are not generalizable to new data sets. The most straightforward solution to this problem is to provide the training step with more data; since this can be the most problematic aspect of NN development, a variation on this, in which the input data is re-used following transformations, may instead be implemented.^[55]

Regularization represents another approach to avoid overfitting, where the goal is to penalize models that have excessive complexity. Weight decay introduces a modified loss function that incorporates a term proportional to the ${\ell }_2$-norm of the set of weights.^[31] This is similar to Tikhonov regularization of conventional parameter estimation problems in that it penalizes large weights, which are characteristic of over-fit models. The ${\ell }_1$-norm may instead be used to promote sparsity in the weight amplitudes.^[56] Weight dropout regularization is unique to NN methods; with each training cycle of forward and backward propagation, a different random subset of weights within a single or multiple layers is set to zero with a pre-established probability.^[32] This reduces the influence of any particular weight, acting to ensure substantial amplitude in all weights. Early stopping is also an effective form of regularization, and was employed in the present work. Here, NN performance is evaluated at each training epoch, with overfitting being characterized by an increase in error rate over a validation set.^[33] As an alternative, performance may first improve and then become stable as a function of epoch number, with this stability then defining a suitable epoch number for training.

All of the regularization methods described above have in common that they act on NN weights. ILR, in contrast, alters the input layer by augmenting the raw data with data that is, in some sense, regularized. It therefore bears some similarity to input data transformations; there is no direct effect on weights, and the weight amplitudes resulting from this procedure remains to be investigated. ILR and the other methods outlined above may be combined; a systematic study of this also remains to be explored.

We based our hyperparameter choices on previous literature^[25,34] and our empirical observations. It is likely that performance can be further improved by implementing a more systematic approach to network architecture and hyperparameter optimization.

The statistical distribution of the data set chosen for training, validation, and testing a NN should correspond to that of the real-world data to which it will be applied. In the present case, with synthetic data, we can incorporate prior knowledge by setting bounds on the two estimated time constants. For myelin mapping in the brain, for example, we may take 20ms $\le T_{2,1}\le 50$ ms and $50\text{ms}\le T_{2,2}\le 500$ ms, with $T_{2,2}$ considered to be the transverse relaxation time of brain parenchyma. In our application, we found it to be very useful to extend the time constant values in the training set beyond those in the testing set to accommodate the NLLS recovery of values outside this a priori range from the noisy data; this improved the performance of both the (ND, ND) and (ND, Reg) NN. Results for both $MP{E}_1$ and $MS{E}_1$ were degraded when the target ($T_{2,1}$, $T_{2,2}$) pair was near the boundary of the range of time constants in the training set. This behavior is independent of the use of ILR. Finally, the spacing between the values of $T_{2,1}$ and $T_{2,2}$ is determined by the required precision of the estimates, with the tradeoff of the increased training time required to maintain training set density.^[57]

ILR as described requires knowledge of the noise level in the signal. In the present case, the noise level may be accurately estimated from data collected from the signal tail, but more general signal forms would present a more difficult challenge. One possibility would be to greatly expand the training set to incorporate a range of plausible SNR values; the effectiveness of this approach remains to be investigated.

In settings in which a discretized probability distribution function (PDF) is sought to define model parameters,^[56] addition of a regularization term serves to e.g., reduce outliers (${\ell }_2$-norm penalty) or promote sparsity (${\ell }_1$ penalty) or promote smoothness (derivative or total variation penalty).^[58,59] However, regularization of the NLLS problem according to Equation (4) does not have an apparent physical basis. The effects noted above, relevant for a PDF, make little sense when estimating parameters that are more-or-less independent. Thus, we have introduced regularization for the mathematical goal of reducing variance rather than on physical grounds. We have not formally studied the effect of implementing alternative penalties for the construction of ILR.

There are two particularly time consuming steps in implementing ILR. The first is the creation of the training set, which requires the solution of many R-NLLS problems. While this can be mitigated by vectorizing our computations to the greatest extent possible, for example in the construction of noisy decays and the reconstruction of discrete signals from their recovered time constants during training, it is less clear how to vectorize multiple R-NLLS problems. Reasonable alternatives include the use of OpenMP for multi-threading the solution of R-NLLS problems over noise realizations.^[60] In addition, a message passing interface (MPI) implementation could partition the R-NLLS problems and assign them to separate nodes on a computing cluster.

The second time-intensive task is NN training. It may be possible to accelerate this through a systematic exploration of the minimal number of training samples that provides a high-performing and generalizable network. In addition, careful selection of the grid spacing of target time constants may serve to reduce training time by avoiding unnecessary precision in the predicted values. One potential aproach would be to increase the density of samples in regions of ($T_{2,1}$, $T_{2,2}$)-space where NLLS problems are known to be more ill-posed while decreasing density in better-conditioned regions.

There are several advantages to using a NN to estimate time constants as compared with classical methods. For example, Prony's method^[20] requires sampling times to be equally spaced, and is known to be highly sensitive to noise.^[61] In contrast, NN approaches have no requirement on the spacing of acquisition times, and can perform well even with limited SNR (Table S2). For curve stripping,^[62,63] an estimate is required for the time constant of the more rapidly-decaying signal, and the underlying time constants $T_{2,1}$, $T_{2,2}$ should be separated by at least a factor of roughly three.^[63] Neither of these limitations apply to NN estimation. Bayesian methods^[64] yield a joint probability distribution for recovered parameters, with highdimensional marginalization often implemented to define the PDF for a single variable. In contrast, NN analysis provides direct parameter estimates. Further important advantages of the NN approach include the fact that once a network is trained, parameter estimation is extremely fast, as well as the ability to incorporate prior knowledge through the design of training data sets.

It is well-known that the efficacy of classical methods for biexponential analysis depend on separation of the underlying time constants and SNR, so it is natural to examine the dependence of ILR in terms of these quantities. While NNs can perform well even when decay constants are not well-separated, larger separation does decrease mean percent error, and to a greater extent in (ND, Reg) as compared with (ND, ND). Performance becomes less sensitive to the difference in time constants as SNR increases to the moderate regime (cf. Figures 2 and 5); we interpret this as indicating that the variability in the estimates is already fairly low, so that the dominant effect of regularization is to increase bias (cf. the bottom left panel in Figure 3), although this increase is somewhat ameliorated through use of the (ND, Reg) network as compared with (ND, ND) (cf. Figure 3) for a reasonable choice of the regularization parameter.

In this initial implementation of ILR, a single regularization parameter $\lambda$ has been used for the entire training data set. This value will not be optimal for all pairs of relaxation times; for example, less regularization may be needed for widely-separated values of $T_{2,1}$ and $T_{2,2}$ than for closely-spaced values. In applications, the underlying relaxation time constants will be unknown, so that this approach is reasonable. Moreover, the optimal value of $\lambda$ will change from noise realization to noise realization even for given values of the time constants. Training a NN to estimate an optimal regularization parameter for each individual input signal, as has been proposed for the introduction of regularization into a NN loss function,^[65] may be effective for ILR as well.

In addition to demonstrating the performance advantages of ILR on a large range of simulated data, we have also demonstrated its applicability to in vivo brain imaging. In this case, we do not have a gold standard to assess accuracy, but we were able to demonstrate the greater resistance to noise in parameter estimation with (ND, Reg) as compared with (ND, ND); see Figure 6. This desirable property is one of the hallmarks of regularization in general. Additionally, only a single average SNR value was used when training (ND, Reg). In effect, this assumes that the $T_2$ decay from each pixel exhibits the same SNR, while in fact the SNR map yielded after NESMA filtering (see S1.5 of SI) indicates there are decays with SNRs between 200 and 800. This may be addressed by training several (ND, Reg) networks for a variety of SNRs and analyzing the decay from a given pixel using the appropriate NN. Moreover, we developed NNs with $c_1=0.1$, the average value across the $c_1$ map obtained in a different study.^[37] This is essentially required in our current implementation, in which we estimate only the two decay constants. Ongoing work is concerned with extending our analysis to a three- or four-parameter model that includes estimates of component fractions. Nevertheless, the provided experimental results serve to illustrate the application of ILR to brain imaging.

5 ∣. CONCLUSION

We have achieved improvements in performance of 15%-50% or more, depending on the metric used and SNR, for a NN designed to estimate relaxation constants of biexponential decay by incorporating a new type of regularization, which we call input layer regularization (ILR). Our studies were performed at SNR levels typical for biomedical magnetic resonance relaxometry. This new approach can readily be expanded to a greater number of parameters and to other signal models. Further, it is compatible with conventional methods for regularizing NN. Finally, we demonstrated the application of ILR on human brain relaxometry data by mapping $T_{2,1}$, attributed to the transverse relaxation time of myelin water.