A Theoretical Framework for Self-Supervised MR Image Reconstruction Using Sub-Sampling via Variable Density Noisier2Noise

Abstract

In recent years, there has been attention on leveraging the statistical modeling capabilities of neural networks for reconstructing sub-sampled Magnetic Resonance Imaging (MRI) data. Most proposed methods assume the existence of a representative fully-sampled dataset and use fully-supervised training. However, for many applications, fully sampled training data is not available, and may be highly impractical to acquire. The development and understanding of self-supervised methods, which use only sub-sampled data for training, are therefore highly desirable. This work extends the Noisier2Noise framework, which was originally constructed for self-supervised denoising tasks, to variable density sub-sampled MRI data. We use the Noisier2Noise framework to analytically explain the performance of Self-Supervised Learning via Data Undersampling (SSDU), a recently proposed method that performs well in practice but until now lacked theoretical justification. Further, we propose two modifications of SSDU that arise as a consequence of the theoretical developments. Firstly, we propose partitioning the sampling set so that the subsets have the same type of distribution as the original sampling mask. Secondly, we propose a loss weighting that compensates for the sampling and partitioning densities. On the fastMRI dataset we show that these changes significantly improve SSDU’s image restoration quality and robustness to the partitioning parameters.

I. Introduction

The data acquisition process in Magnetic Resonance Imaging (MRI) consists of traversing a sequence of smooth paths through the Fourier representation of the image, referred to as “k-space”, which is inherently time-consuming. Images can be reconstructed from accelerated, sub-sampled acquisitions by leveraging the non-uniformity of receiver coil sensitivities, referred to as “parallel imaging” [1], [2], [3], [4]. Compressed sensing [5], [6], which uses sparse models to reconstruct incoherently sampled data, has also been widely applied to MRI [7], [8], [9].

There has been significant research attention in recent years on methods that reconstruct sub-sampled MRI data with neural networks [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24]. The majority of these works use fully-supervised training. To train a network in a fully-supervised manner, there must be a dataset comprised of fully sampled k-space data y_0,t ∈ ℂ^N, where N is the dimension of k-space multiplied by the number of coils, and paired sub-sampled data $y_t=M_{{\Omega }_t}{y}_{0,t}$. Here, t indexes the training set and $M_{{\Omega }_t}\in {ℝ}^{N\times N}$ is a sub-sampling mask with sampling set Ω_t, so that the jth diagonal of $M_{{\Omega }_t}$ is 1 if j ∈ Ω_t and zero otherwise. Then a network f_θ with parameters θ is trained by seeking a minimum of a non-convex loss function:

$$\widehat{\theta }=\underset{\theta }{\arg \min }{\displaystyle \sum _{t}L\left(f_{\theta }\left(y_t\right),y_{0,t}\right),}$$ (1)

which could be, for example, an ${\ell }_p$ norm in the image domain after coil combination [25]. The network $f_{\widehat{\theta }}$ estimates the ground truth in the image domain or k-space depending on the choice of loss function. For a k-space to k-space network, y_0,s can be estimated with ${\widehat{y}}_s=f_{\widehat{\theta }}\left(y_s\right)$, where s indexes the test set.

Given sufficient representative training data, fully-supervised networks can yield substantial reconstruction quality gains over sparsity-based compressed sensing methods. There are a number of large datasets available for fully supervised training, such as the fastMRI knee and brain data [25]. However, for many contrasts, orientations, or anatomical regions of interest, fully sampled datasets are not publicly available. Fully sampled data is rarely acquired as part of a normal scanning protocol, so acquiring sufficient training data for a specific application is highly resource intensive. In some cases, it may not even be technically feasible to acquire such data [26], [27], [28]. Therefore, for MRI reconstruction with deep learning to be applicable to datasets acquired using only standard protocols, a training method that uses solely sub-sampled data is required.

There have been several attempts to train networks with only sub-sampled MRI data [29], [30], [31], [32], [33], [34], [35], [36], [37], some of which are based on methods from the denoising literature [38], [39], [40], [41], [42], [43], [44].One such approach is Noise2Noise [38]. Rather than mapping y_t to y_0,t, Noise2Noise trains a network to map y_t to another sub-sampled k-space $y_T=M_{{\Omega }_t}{y}_{0,T}$ where Ω_T and Ω_t are independent and y_0,T =y_0,t when t = T [31]. A limitation of Noise2Noise is that it requires paired data, so the dataset must contain two independently sampled scans of the same k-space [14], which is not part of standard protocols. Further, unless compensated for [45], any motion and phase drifts between scans would cause the paired data to be inconsistent, violating the central assumption that underlies the method.

SSDU [33] is a recently proposed method for ground-truth free training that does not require paired data. SSDU partitions the sampling set Ω_t into two disjoint sets: ${\Omega }_t=A_t\cup B_t$, where $A_t\cap B_t=\varnothing$. Then the network is trained to recover $M_{A_t}{y}_t$ from $M_{B_t}{y}_t$:

$$\widehat{\theta }=\underset{\theta }{\arg \min }{\displaystyle \sum _{t}L\left(M_{A_t}{f}_{\theta }\left(M_{B_t}{y}_t\right),M_{A_t}{y}_t\right)}.$$ (2)

At inference, the estimate $f_{\widehat{\theta }}\left(y_s\right)$ is used. With a physics-guided network architecture, SSDU was found to have a reconstruction quality comparable with fully supervised training given certain empirically selected choices of A_t and B_t. However, it was presented without theoretical justification. Although SSDU has similarities with Noise2Self [40], Noise2Self’s analysis has a strong requirement on independent noise, so do not apply to k-space sampling in general.

A. Contributions

This article considers the recently proposed Noisier2Noise framework [41], which was originally constructed for denoising problems. We modify Noisier2Noise so that it can be applied to variable density sub-sampled MRI data. To our knowledge, this is the first work that applies Noisier2Noise to image reconstruction. Like SSDU, the proposed modification of Noisier2Noise does not require paired data, and involves training a network to map from one subset of Ω_t to another. While SSDU recovers one disjoint set from the other, Noisier2Noise applies a second sub-sampling mask to the data, ${\tilde{y}}_t=M_{{\Lambda }_t}{y}_t=M_{{\Lambda }_t}{M}_{{\Omega }_t}{y}_{0,t}$, and the network is trained to recover y_t from ${\tilde{y}}_t$ with an ${\ell }_2$ loss. Then, at inference, the fully sampled data is estimated via a correction term based on the distributions of Λ_t and Ω_t that ensures that the estimate is correct in expectation.

Despite their superficial differences, we show that, in fact, SSDU and Noisier2Noise are closely related. Specifically, we demonstrate that SSDU is a version of Noisier2Noise with a particular loss function modification that removes the need for the correction term at inference. The primary contribution of this article is the use of Noisier2Noise to theoretically explain SSDU’s excellent empirical performance. Specifically, we show that SSDU with an ${\ell }_2$ loss correctly estimates fully sampled k-space in expectation: see Section II-D.

The second contribution of this article is the proposal of two modifications of SSDU that significantly improve its reconstruction quality and robustness to the parameters of $M_{{\Lambda }_t}$, both of which arise as a consequence of SSDU’s connection to Noisier2Noise. Firstly, we use Noisier2Noise to inform SSDU’s sampling set partition: we show that SSDU’s performance improves when B_t has the same type of distribution as the original mask Ω_t, but not necessarily with the same parameters. Secondly, we show that SSDU’s performance improves when a particular weighting is employed in the loss function. This non-trivial weighting, which arises as a consequence of the novel theoretical analysis of SSDU, depends on the distributions of Λ_t and Ω_t and has minimal additional computational cost: see Section II-F.

Although this paper focuses on MRI reconstruction, we emphasize that none of the theoretical developments are specific to k-space. This framework is therefore applicable to any image reconstruction problem with a forward model that involves random sub-sampling, such as low dose x-ray computed tomography [46] or astronomical imaging [47].

II. Theory

This section describes how the Noisier2Noise framework can be applied to sub-sampled data. Additive and multiplicative noise versions of Noisier2Noise are proposed in [41]. Based on the observation that a k-space sub-sampling mask can be considered as multiplicative “noise”, we extend Noisier2Noise to image reconstruction by modifying the latter. It is standard practice in MRI to sub-sample k-space with variable density, so that low frequencies, where the spectral density is larger, are sampled with higher probability [7]. Since the multiplicative noise version of standard Noisier2Noise assumes uniformity, this requires a modification of the framework to variable density sampling.

A. Variable Density Noisier2Noise for Reconstruction

The terms in the measurement model $y_t=M_{{\Omega }_t}{y}_{0,t}$ can be considered as instances of random variables. We denote Y = M_ΩY₀, where Y, M_Ω and Y₀ are the random variables corresponding to y_t, $M_{{\Omega }_t}$, and y_0,t respectively. Now consider the multiplication of Y by a second mask represented by the random variable M_Λ,

$$\tilde{Y}=M_{\Lambda }Y=M_{\Lambda }{M}_{\Omega }{Y}_0,$$

so that $\tilde{Y}$ is a further sub-sampled random variable. The following result states how the expectation of Y₀ can be computed from $\tilde{Y}$ and Y. Here, and throughout this article, 𝔼[·] is used to denote the expectation over all random variables within the brackets.

Claim 1: When $\text{𝔼}\left[M_{\Omega ,jj}\right]=p_j>0$ and $\text{𝔼}\left[M_{\Lambda ,jj}\right]={\tilde{p}}_j<1$ for all j, the expectation of Y₀ given $\tilde{Y}$ is

$$\text{𝔼}\left[Y_0|\tilde{Y}\right]={\left(\mathrm{𝟙}-K\right)}^{-1}\left(\text{𝔼}\left[Y|\tilde{Y}\right]-K\tilde{Y}\right),$$ (3)

where K is a diagonal matrix defined as

$$K={\left(\mathrm{𝟙}-\tilde{P}P\right)}^{-1}\left(\mathrm{𝟙}-P\right)$$ (4)

for P = 𝔼[M_Ω] and $\tilde{P}=\text{𝔼}\left[M_{\Lambda }\right]$.

Proof: See Section A of the Appendix, which is based on the proof given in Section III.D of [41].

Equation (3) generalizes the version of Noisier2Noise pro-posed for uniform, multiplicative noise in [41] to variable density sampling. The difference between the uniform and variable density versions is the matrix K, which is a scalar in [41]. For the special case where M_Ω and M_Λ are uniformly random sub-sampling masks, P, $\tilde{P}$ and therefore K are proportional to the identity matrix, and (3) simplifies to the uniform version. The mathematical requirement that p_j > 0 and ${\tilde{p}}_j<1$ for all j simply ensures that (𝟙 − K) is invertible: see Section A of the Appendix.

Equation (3) implies that $\text{𝔼}\left[Y_0|\tilde{Y}\right]$ can be estimated without fully sampled data by training a network to estimate $\text{𝔼}\left[Y|\tilde{Y}\right]$. To do this, a network can be trained to minimize

$$\theta *=\underset{\theta }{\arg \min }\text{𝔼}\left[\Vert W\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}|\tilde{Y}\right]$$ (5)

for a full-rank matrix W. The minimum occurs when the gradient with respect to θ is zero:

$${\nabla }_{\theta }\text{𝔼}\left[\Vert W\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}|\tilde{Y}\right]=\text{𝔼}\left[2J{W}^{H}W\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)|\tilde{Y}\right]=0,$$

where J is the Jacobian matrix with entries $J_{ij}=\partial f_{\theta }{\left(\tilde{Y}\right)}_j/\partial {\theta }_i$. The number of parameters is typically much greater than N, so J has far more rows than columns. Assuming that the rows of J are maximally linearly independent, so the row space is N-dimensional, the only solution is

$$\text{𝔼}\left[W^{H}W\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)|\tilde{Y}\right]=0.$$ (6)

If W is full-rank, W^H W is also full rank, so left-multiplying by (W^H W)⁻¹ and using $\text{𝔼}\left[f_{\theta }\left(\tilde{Y}\right)|\tilde{Y}\right]=f_{\theta }\left(\tilde{Y}\right)$,

$$f_{\theta }\left(\tilde{Y}\right)=\text{𝔼}\left[Y|\tilde{Y}\right].$$

Therefore, by (3), a candidate for estimating fully sampled k-space with sub-sampled data only is

$$\text{𝔼}\left[Y_0|\tilde{Y}\right]={\left(\mathrm{𝟙}-K\right)}^{-1}\left(f_{\theta *}\left(\tilde{Y}\right)-K\tilde{Y}\right).$$

This expression does not use Y, so does not use all available data. Two candidate approaches for using all available data at inference are considered in this article. Firstly, one can overwrite known entries of the network output with Y:

$$\begin{array}{c}{\widehat{Y}}^{dc}=\left(\mathrm{𝟙}-M_{\Omega }\right)\text{E}\left[Y_0|\tilde{Y}\right]+Y\\ =\left(\mathrm{𝟙}-M_{\Omega }\right){\left(\mathrm{𝟙}-K\right)}^{-1}\left(f_{\theta *}\left(\tilde{Y}\right)-K\tilde{Y}\right)+Y\\ =\left(\mathrm{𝟙}-M_{\Omega }\right){\left(\mathrm{𝟙}-K\right)}^{-1}{f}_{\theta *}\left(\tilde{Y}\right)+Y,\end{array}$$

where the final step uses $\left(\mathrm{𝟙}-M_{\Omega }\right)\tilde{Y}=\left(\mathrm{𝟙}-M_{\Omega }\right)M_{\Lambda }{M}_{\Omega }{Y}_0=0$. Here, the superscript refers to “data consistent”, since the estimate is exactly consistent with Y. We emphasize that ${\widehat{Y}}^{dc}$ is consistent with all available data Y, not just the data in $\tilde{Y}$. Alternatively, similar to the approaches suggested in both SSDU [33] and the additive noise examples in Noisier2Noise [41], one can use singly sub-sampled k-space Y as the network input at inference:

$$\widehat{Y}={\left(\mathrm{𝟙}-K\right)}^{-1}\left(f_{\theta *}\left(Y\right)-KY\right)$$ (7)

Since Claim 1 applies to $f_{\theta *}\left(\tilde{Y}\right)$, not $f_{\theta *}\left(Y\right)$, (7) is not guaranteed to be correct in expectation. However, it has the advantage that all available data is used by the network. Hence, despite deviating from strict theory, we have found that it performs well in practice: see Section IV.

This suggests the following procedure, illustrated in Fig. 1, for training a network without fully-sampled data. For each sub-sampled k-space in the training set $y_t=M_{{\Omega }_t}{y}_{0,t}$, generate a further sub-sampled k-space ${\tilde{y}}_t=M_{{\Lambda }_t}{y}_t=M_{{\Lambda }_t}{M}_{{\Omega }_t}{y}_{0,t}$, where $M_{{\Lambda }_t}$ is an instance of M_Λ. Then, approximate (5) by training a network to minimize the loss function

$$\widehat{\theta }=\underset{\theta }{\arg \min }{\displaystyle \sum _t\Vert W\left(f_{\theta }\left({\tilde{y}}_t\right)-y_t\right)\Vert \begin{array}{c}2\\ 2\end{array}},$$ (8)

for some full-rank matrix W. During inference, estimate fully-sampled k-space with either

$${\widehat{y}}_s^{dc}=\left(\mathrm{𝟙}-M_{{\Omega }_s}\right){\left(\mathrm{𝟙}-K\right)}^{-1}{f}_{\widehat{\theta }}\left({\tilde{y}}_s\right)+y_s$$ (9)

or

$${\widehat{y}}_s={\left(\mathrm{𝟙}-K\right)}^{-1}\left(f_{\widehat{\theta }}\left(y_s\right)-K{y}_s\right),$$ (10)

where s indexes the test set.

Fig. 1.

Schematic of the self-supervised training methods in this article. If the loss weighting W is full rank, the training method is variable density Noisier2Noise, as proposed in Section II-A, whereas if $W=\left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$ the training method is SSDU: see Section II-D.

In other words, we train a network to estimate the “singly” sub-sampled k-space y_t from “doubly” sub-sampled k-space ${\tilde{y}}_t$ and then, during inference, apply a correction based on the diagonal matrix K to estimate the fully sampled data. The correction term only needs to be applied during inference and has minimal computational cost.

In [41], only the version with W = 𝟙 was presented. Here we present a version with non-trivial W because it provides a theoretical link to SSDU; Section II-D shows that Noisier2Noise with the rank-deficient W = (𝟙 − M_Λ)M_Ω is SSDU exactly.

Noisier2Noise and SSDU work because the network cannot deduce from ${\tilde{y}}_t$ which entries of y_t are non-zero [41]. Therefore, the loss is minimized when the network learns to recover all of k-space: see Section V for a detailed discussion.

B. Choice of Mask Distributions

The only condition on the first mask M_Ω from Claim 1 is that p_j > 0 for all j. In other words, the guarantee only applies when there is a non-zero probability that there are sampled examples of all k-space locations in the training set.

Claim 1 also states that the second mask M_Λ must obey ${\tilde{p}}_j<1$ for all j. This ensures that there is a non-zero probability that any entry of $\tilde{Y}$ is masked. Unlike M_Ω, whose distribution is determined by the acquisition protocol, the M_Λ is chosen freely during training. Following [41], we suggest using a distribution of M_Λ that is the same type as M_Ω, but not necessarily with the same parameters. For instance, if M_Ω is column-wise sampling with variable density, such as in Fig. 1, an appropriate M_Λ is one that is also column-wise, but possibly with a different variable density distribution.

C. Choice of Network

Noisier2Noise is agnostic to the network architecture. We have found that using the data consistent function

$$f_{\theta }\left({\tilde{y}}_t\right)=\left(\mathrm{𝟙}-M_{{\Lambda }_t}{M}_{{\Omega }_t}\right)g_{\theta }\left({\tilde{y}}_t\right)+{\tilde{y}}_t,$$ (11)

where $g_{\theta }\left({\tilde{y}}_t\right)$ is a network with arbitrary architecture, may improve the performance of Noisier2Noise. This is because the $g_{\theta }\left({\tilde{y}}_t\right)$ in (11) only recovers regions of k-space that are not already sampled in ${\tilde{y}}_t$, so the network does not need to learn to map sampled k-space locations to themselves. We emphasize that (11) ensures that $f_{\theta }\left({\tilde{y}}_t\right)$ is consistent with ${\tilde{y}}_t$, while (9) ensures the estimate ${\widehat{y}}_s^{dc}$ is consistent with y_s, which is only applied at inference and cannot be part of the network architecture when ${\tilde{y}}_s$ is used as the input.

Many popular network architectures for MRI reconstruction are based on a sequence of “unrolled” iterations of a optimization algorithm [48] such as the Iterative Shrinkage Thresholding Algorithm (ISTA) [49] or the Alternating Direction Method of Multipliers (ADMM) [50]. These are variously known as “physics-guided”, “physics-based” or “model-based” methods due to their explicit use of the MRI forward model. These architectures typically alternate between a module that recovers missing k-space entries by removing aliasing in the image domain and a module that ensures consistency with the k-space data. This implies that (11), or possibly a “soft” version of it where the data is not forced to be exactly consistent, may already be implemented as part of the network architecture. In the experimental evaluation of the methods in this article we used the Variational Network (VarNet) [12], [51], which is one such architecture where (11) is not necessary. However, in preliminary studies not presented in this article we found that a U-net [52], which does not already employ data consistency, benefited considerably from (11).

D. Relationship to SSDU

This section shows that SSDU [33] with an ${\ell }_2$ loss is a version of Noisier2Noise with a particular rank-deficient loss weighting matrix W.

To see the connection between SSDU and Noisier2Noise, it is instructive to see the relationship between Noisier2Noise’s Λ_t and SSDU’s disjoint subsets A_t and B_t. Disjoint subsets of Ω_t can be formed in terms of Ω_t and Λ_t by setting A_t =Ω_t \ Λ_t and B_t =Ω_t ∩ Λ_t. The distribution of A_t and B_t are defined by the distributions of Ω_t and Λ_t and always satisfy A_t ∪ B_t = Ω_t and A_t ∩ B_t = ∅ as required. In terms of sampling masks, this is written as $M_{A_t}=\left(1-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$ and $M_{B_t}=M_{{\Lambda }_t}{M}_{{\Omega }_t}$. Therefore, SSDU’s loss (2) with a squared ${\ell }_2$ norm is

$${\displaystyle \sum _t\Vert M_{A_t}{f}_{\theta }\left(M_{B_t}{y}_t\right)-M_{A_t}{y}_t\Vert \begin{array}{c}2\\ 2\end{array}}={\displaystyle \sum _t\Vert \left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)\cdot M_{{\Omega }_t}\left(f_{\theta }\left({\tilde{y}}_t\right)-y_t\right)\Vert \begin{array}{c}2\\ 2\end{array}},$$

so is exactly Noisier2Noise with $W=\left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$. In other words, while Noiser2Noise’s loss is computed over all k-space, SSDU’s loss is computed only on indices that are in Ω_t but not in Λ_t.

SSDU’s weighting ensures that any indices not sampled in Y are ignored in the loss. One might think that the correct choice for this goal would be $W=M_{{\Omega }_t}$. However, if a data consistent network is employed, as in (11), the contribution to the loss from indices in both Ω_t and Λ_t would be zero because they are consistent by construction. Therefore the loss for $W=M_{{\Omega }_t}$ and $W=\left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$ would be identical. A similar idea was presented for fully supervised learning in [53], where a mask is applied to the training data multiple times.

E. Proof of SSDU

This section shows that SSDU’s loss weighting causes the correction (𝟙 − K)⁻¹ at inference to no longer be necessary. When the weighting matrix W is the random variable (𝟙 − M_Λ)M_Ω, the network parameters are trained to seek a minimum of

$$\theta *=\underset{\theta }{\arg \min }\text{𝔼}\left[\Vert \left(\mathrm{𝟙}-M_{\Lambda }\right)M_{\Omega }\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}|\tilde{Y}\right].$$ (12)

Unlike Noisier2Noise, W =(𝟙 − M_Λ)M_Ω is not full-rank, so $f_{\theta *}\left(\tilde{Y}\right)\ne \text{E}\left[Y|\tilde{Y}\right]$. The usual theoretical goal for self-supervised methods is to prove that the network is correct in expectation [38], [39], [40], [41], [42], [43], [44], as in Claim 1 for variable density Noisier2Noise. In the following we state, to our knowledge, the first similar result for SSDU.

Claim 2: A network with parameters that minimizes (12) satisfies

$$\left(\mathrm{𝟙}-K\right)\left(\mathrm{𝟙}-M_{\Lambda }{M}_{\Omega }\right)\left(f_{\theta *}\left(\tilde{Y}\right)-\text{𝔼}\left[Y_0|\tilde{Y}\right]\right)=0.$$ (13)

Proof: See Section B of the Appendix.

If 𝟙 − K is invertible, which holds when p_j > 0 and ${\tilde{p}}_j<1$ for all j,

$$\left(\mathrm{𝟙}-M_{\Lambda }{M}_{\Omega }\right)f_{\theta *}\left(\tilde{Y}\right)=\left(\mathrm{𝟙}-M_{\Lambda }{M}_{\Omega }\right)\text{𝔼}\left[Y_0|\tilde{Y}\right].$$

Therefore, in general, $f_{\theta *}\left(\tilde{Y}\right)$ is correct in expectation, but only in regions of k-space that are not sampled in $\tilde{Y}$. This contrasts with the variable density Noisier2Noise method presented in Section II-A, which is correct in expectation for all k-space indices. However, as described in the following, this apparent shortcoming can easily be circumvented by using all available data at inference.

Similarly to Noisier2Noise’s (9) and (10), we consider two options for the k-space estimate at inference, both of which use all available data. Firstly, similarly to (9), the data consistent estimate

$${\widehat{Y}}^{dc}=\left(\mathrm{𝟙}-M_{\Omega }\right)f_{\theta *}\left(\tilde{Y}\right)+Y$$ (14)

can be used, which is correct in expectation everywhere in k-space for any network architecture. Alternatively, the SSDU paper [33] suggests using

$$\widehat{Y}=f_{\theta *}\left(Y\right)$$ (15)

and a physics-guided network architecture. Like (10) for Noisier2Noise, the network input for (15) is singly sub-sampled, so Claim 2 does not apply and the estimate is not guaranteed to be correct in expectation. Nonetheless, it has the advantage over (14) that it uses all available data in the input to the network. As in [33], we have found that (15) performs well in practice when the network architecture includes a data consistency module: see Section IV.

We emphasize that unlike Noisier2Noise, SSDU does not require the correction term (𝟙 − K)⁻¹ at inference. This implies that SSDU is less sensitive to inaccuracies in $f_{\theta *}\left(\tilde{Y}\right)$, and we have found that SSDU outperforms Noisier2Noise in general: see Section IV.

F. K-Weighted SSDU

Since we train on a finite number of instances of the random variables Y, $\tilde{Y}$, Ω and Λ, the network parameters we obtain in practice, which we denote $\widehat{\theta }$, are an approximation of the ideal θ^∗ from (12). In this case, the right-hand-side of (13) is not exactly zero. Rather,

$$\left(\mathrm{𝟙}-K\right)\left(\mathrm{𝟙}-M_{\Lambda }{M}_{\Omega }\right)\left(f_{\widehat{\theta }}\left(\tilde{Y}\right)-\text{𝔼}\left[Y_0|\tilde{Y}\right]\right)=\epsilon ,$$ (16)

where ε is a vector random variable. The vector ε characterizes the difference between a true expectation and the network’s estimate of it, which is non-zero for finite data. In other words, ε is a statistical error due to finite sampling. The difference between the trained network’s output and the expectation of interest, $\text{𝔼}\left[Y_0|\tilde{Y}\right]$, is (𝟙 − K)⁻¹ε. This implies that the network is more sensitive to errors in k-space locations where (𝟙 − K)⁻¹ is large.

To compensate for this, we propose minimizing the following weighted version of SSDU’s loss as an alternative to (12):

$$\underset{\theta }{\arg \min }\text{𝔼}\left[\Vert {\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}\left(\mathrm{𝟙}-M_{\Lambda }\right)M_{\Omega }\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}|\tilde{Y}\right].$$

Introducing ${\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}$ in the loss cancels the 𝟙 − K in (16), so mitigates the error amplification caused by θ* approximation. We find that this version of SSDU, which we refer to as “K-weighted SSDU” throughout the remainder of this article, substantially improves the image restoration quality and robustness to training hyperparameters: see Section IV. We chose the power ${\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}$ because it exactly cancels the 𝟙 − K on the left-hand-side of (16) when the squared ${\ell }_2$ loss is used; we also tried power (𝟙 − K)⁻¹ and found that, as expected, it did not perform as well in practice.

G. Understanding the Need for Correction

This section intuitively explains why Noisier2Noise requires correction at inference but SSDU does not. We can write the weighted loss as

$$\Vert W\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}=\Vert W\left[M_{\Omega }{M}_{\Lambda }+\left(\mathrm{𝟙}-M_{\Lambda }\right)M_{\Omega }+\left(\mathrm{𝟙}-M_{\Omega }\right)\right]\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array},$$

where we have used that the term is square brackets equals the identity matrix. When $f_{\theta }\left(\tilde{Y}\right)$ is consistent with $\tilde{Y}$, such as in (11), $M_{\Omega }{M}_{\Lambda }\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)=0$. Therefore

$$\Vert W\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}=\Vert W\left(\mathrm{𝟙}-M_{\Lambda }\right)M_{\Omega }\left(f_{\theta }\left(\tilde{Y}\right)-Y\right)\Vert \begin{array}{c}2\\ 2\end{array}+\Vert W\left(\mathrm{𝟙}-M_{\Omega }\right)f_{\theta }\left(\tilde{Y}\right)\Vert \begin{array}{c}2\\ 2\end{array},$$ (17)

where we have used (𝟙 − M_Ω)Y = 0. In (17) is SSDU’s loss function (12) plus a contribution from all $j\in {\Omega }_t^c$.

Intuitively, the second term on the right-hand-side of (17) causes the proposed method to underestimate regions of k-space with index $j\in {\Omega }_j^c$. This underestimation is compensated for with (𝟙 − K)⁻¹ at inference. For SSDU, where W =(𝟙 − M_Λ)M_Ω, the second term on the right-hand-side of (17) is zero, k-space is not underestimated anywhere, and there is no need for a correction term at inference.

III. Experimental Method

A. Description of Data

We used the multi-coil brain and knee data from the fastMRI dataset [25], which is comprised of multi-channel raw k-space MRI data. The reference fastMRI test set data is magnitude images only, without fully sampled k-space data. Since we also require phase, we discarded the data allocated for testing and generated our own partition into training, validation and test sets. For the brain data, we only used data that was acquired on 16 coils, and used training, validation and test set sizes of 127, 19 and 14 volumes (2020, 302, and 224 slices) respectively. For the knee data, the training, validation and test sets consisted of 166, 19 and 14 volumes (5977, 665, and 493 slices) respectively. We set the network output to be zero in regions of k-space where the reference data had zero padding.

B. Network Architecture

For f_θ, we used the variant of the VarNet [12] that estimates coil sensitivities on-the-fly [51], which performs competitively on the fastMRI leaderboard and is available as part of the fastMRI package.¹ After a coil sensitivity estimation module, VarNet uses multiple repetitions of a module based on gradient descent, which is comprised of a data consistency term in k-space and a prior based on a U-net [52] that acts in the image domain after an inverse Fourier transform and coil combination. The output of the neural network was in k-space. We used 6 repetitions of the main module, so that our model had around 1.5×10⁷ parameters. Note that in [25], the Structural Similarity Index (SSIM) [54] was used as the loss, while in this article we use an ${\ell }_2$ loss.

The only additional operations SSDU and Noisier2Noise require compared to fully-supervised training are simple entry-wise masks, so all methods had similar memory requirements and training time. We trained for 50 epochs, which took around 17 hours on a GTX 1080 Ti GPU with 11 GB of RAM for the brain data. For all methods we used the Adam optimizer [55] with a fixed learning rate of 10⁻³. Our PyTorch implementation is publicly available on GitHub.²

C. Distribution of Masks

So that the distribution of the sampling masks were known exactly, we generated our own masks rather than using those suggested in fastMRI. Unless stated otherwise, the distribution of the first mask M_Ω was 1D column-wise. We fully sampled the central 10 columns and sampled the remainder with polynomial variable density. We used polynomial order 8, and scaled the probability density P so that it matched a desired acceleration factor. We ran each method with R_Ω ∈ {4, 8}, where $R_{\Omega }=N/{\displaystyle {\sum }_j{p}_j}$ is the expected acceleration factor. An example at R_Ω = 4 is shown in Fig. 2(a).

Fig. 2.

Example of the singly sub-sampled mask $M_{{\Omega }_t}$, and doubly sub-sampled $M_{{\Lambda }_t}{M}_{{\Omega }_t}$ with two M_Λ distribution types. Here, the acceleration factor of the first mask is R_Ω = 4 and the second is R_Λ = 2.

In [41], it is suggested that the distribution of Noisier2Noise’s second random variable is the same as the first, but not necessarily with the same distribution parameters. Therefore, for Noisier2Noise’s second mask M_Λ, we used the same type of distribution as M_Ω with a different variable density. An example with R_Ω = 4 and $R_{\Lambda }=N/{\displaystyle {\sum }_j{\tilde{p}}_j=2}$ is shown in Fig. 2(b). Concretely, we define two masks as having the same ‘type’ of distribution when the conditional dependence of the sampling set indices is the same. Let $p_{j|k}=\text{P}\left[j\in \Omega |k\in \Omega \right]$. If p_j|k = p_j for all j and k, the entries are independent and the mask is the type ‘2D Bernoulli’. If p_j|k = 1 when j and k are in the same k-space column and p_j|k = p_j otherwise, the mask is the type ‘1D column-wise’. The experiments in this article focus on these two types of masks; other types are discussed in Section V. We emphasize that constraining a mask to a type does not constrain the p_js, which define the variable sampling density.

To ensure that ${\tilde{p}}_j<1$ everywhere, we set ${\tilde{p}}_j<1-ϵ$ in the central 10 columns of k-space, where ϵ is a small real constant. The network architecture ensures that the central region is consistent with the input, so ϵ can be small without penalty. We used ϵ = 10⁻³.

In order to be a realistic simulation of prospectively sub-sampled data, the sampling set Ω_t must be fixed for all epochs. However, Λ_t need not be. Therefore, we re-generated $M_{{\Lambda }_t}$ from the distribution of M_Λ once per epoch. Since the network sees more samples from the distribution of M_Λ, the loss function is closer to (5), so $f_{\widehat{\theta }}$ is expected to be a more accurate approximation of $\text{𝔼}\left[Y|\tilde{Y}\right]$. This has similarities with training data augmentation, as each slice is used to generate several inputs to the network [56].

D. Comparative Methods

We trained Noisier2Noise using different weightings of the ${\ell }_2$ loss stated in (8). For each self-supervised method, we considered two possible estimates at inference: one with the doubly sub-sampled ${\tilde{y}}_s$ as the network input and the other with the singly sub-sampled y_s. The methods and their two estimates at inference are summarized in Table I.

TABLE 1. The Self-Supervised Methods Evaluated in This Paper.

Name	Loss weighting W	MΛ distribution	Estimate with ${\tilde{y}}_s$ input	Estimate with ys input
Unweighted Noisier2Noise	𝟙	ID column-wise	$\left(\mathrm{𝟙}-M_{{\Omega }_s}\right){\left(\mathrm{𝟙}-K\right)}^{-1}{f}_{\widehat{\theta }}\left({\tilde{y}}_s\right)+y_s$	${\left(\mathrm{𝟙}-K\right)}^{-1}\left(f_{\widehat{\theta }}\left(y_s\right)-K{y}_s\right)$
2D partitioned SSDU	$\left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$	2D Bernoulli	$\left(\mathrm{𝟙}-M_{{\Omega }_s}\right)f_{\widehat{\theta }}\left({\tilde{y}}_s\right)+y_s$	$f_{\widehat{\theta }}\left(y_s\right)$
1D partitioned SSDU	$\left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$	1D column-wise	$\left(\mathrm{𝟙}-M_{{\Omega }_s}\right)f_{\widehat{\theta }}\left({\tilde{y}}_s\right)+y_s$	$f_{\widehat{\theta }}\left(y_s\right)$
K-weighted 1D partitioned SSDU	${\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}\left(\mathrm{𝟙}-M_{{\Lambda }_t}\right)M_{{\Omega }_t}$	1D column-wise	$\left(\mathrm{𝟙}-M_{{\Omega }_s}\right)f_{\widehat{\theta }}\left({\tilde{y}}_s\right)+y_s$	$f_{\widehat{\theta }}\left(y_s\right)$

Here, and throughout this paper, the subscripts t and s index the training and test sets respectively. Examples of $M_{{\Lambda }_t}{M}_{{\Omega }_t}$ for 2D bemoulli and ID column-wise $M_{{\Lambda }_t}$ are shown in fig. 2.

We trained with W = 𝟙, referred to as “Unweighted Nois-ier2Noise”. By Claim 1, Unweighted Noisier2Noise requires (𝟙 − K)⁻¹ correction at inference: see Table I. We have found that the need for correction substantially reduces the image quality compared to SSDU, so do not recommend using Unweighted Noisier2Noise in practice. Nonetheless, we include some Unweighted Noisier2Noise results to illustrate the value of SSDU’s loss weighting.

We also trained Noisier2Noise with W = (𝟙 −M_Λ)M_Ω which, based on the relationship described in Section II-D, we refer to as “SSDU”, despite some differences between our implementation and [33]. In [33], a mixture of an ${\ell }_1$ and ${\ell }_2$ loss was used, whereas here, so that it can be directly compared with Unweighted Noisier2Noise, we used an ${\ell }_2$ loss. We also used a different M_Ω distribution, dataset and network architecture to [33].

SSDU [33] was originally applied to an architecture that requires pre-computed sensitivity maps. It was suggested that $M_{B_t}$ has a fully sampled 4 × 4 central region and 2D Gaussian variable density otherwise, so that high frequencies are sampled with higher probability. For the architecture considered in this article, which has a coil sensitivity estimation module, we found that increasing the size of the fully sampled central region considerably improved the method’s performance. Since M_Ω has 10 fully sampled central columns, we increased the size of the central region of M_Λ to 10 × 10.

As the probability of sampling each location in k-space is independent, the sampling set partition proposed in [33] is equivalent to a 2D variable density Bernoulli M_Λ distribution. To estimate their variable density distribution $\tilde{P}$ we ran the SSDU authors’ set partitioning code³ 1000 times on a fully sampled mask and averaged the result. We trained SSDU using a distribution of M_Λ of this type, referred to as “2D partitioned SSDU”, illustrated in Fig. 2(c). We also trained SSDU using the same distribution type of M_Λ as M_Ω, as in Fig. 2(b). We refer to this method as “1D partitioned SSDU”, or “K-weighted 1D partitioned SSDU” when a ${\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}$ weighting is used in the loss as described in Section II-F. Like Unweighted Noisier2Noise, $M_{{\Lambda }_t}$ was re-generated once per epoch [56]. We emphasize that although 2D partitioned SSDU has a similar M_Λ distribution as in [33], the distribution of M_Ω here is random variable density columns, not equidistant columns as in [33]. Therefore, 2D partitioned SSDU is not necessarily expected to perform as well as SSDU in [33].

As a best-case target, we also trained using a fully supervised method with an (unweighted) ${\ell }_2$ loss. All deep learning methods had the same network architecture and training hyperparameters, as described in III-B.

Finally, as a comparative method that does not use deep learning, we ran a compressed sensing algorithm with a sparse model on wavelet coefficients, which we implemented via the Berkeley Advanced Reconstruction Toolbox (BART) [57].We used BART’s default settings with fourth-order Daubechies wavelets and a sparse weighting of λ = 2 × 10⁻³.

E. Quality Metrics

To evaluate the reconstruction quality, we computed the Normalized Mean Squared Error (NMSE) in k-space on the test set: $\Vert {\widehat{y}}_s-y_{0,s}\Vert \begin{array}{c}2\\ 2\end{array}/\Vert y_{0,s}\Vert \begin{array}{c}2\\ 2\end{array}$. We also computed the image-domain root-sum-of-squares (RSS), ${\widehat{x}}_s={\left({\displaystyle {\sum }_c{\left|F^H{y}_{s,c}\right|}^2}\right)}^{1/2}$ where y_s,c is the k-space entries on coil c and F is the discrete Fourier transform, cropped the RSS estimate to a central 320×320 region and computed the SSIM, as suggested in fastMRI [25].

IV. Results

For brevity, the results presented here focus on R_Ω = 8. Similar results for the brain data at R_Ω = 4 are shown in the supplementary material: see Figs. S1-S4.

For the brain data, we evaluated the dependence of the methods’ performance on the distribution of M_Λ by varying the parameters so that the sub-sampling factor R_Λ changed. We trained with R_Λ ∈ {1.2, 1.6, 2, 4, 6}, except for 2D partitioned SSDU, which we found needed finer tuning and a smaller R_Λ for the best performance, so we trained with R_Λ ∈ {1.1, 1.2, …, 2, 3, 4, 6}.

A. Performance With Tuned R_Λ

This section focuses on the case where R_Λ has been tuned to minimize the ground truth test set NMSE. Figs. 3 and S1 show bar charts of the percentage difference between fully supervised training and each method: (μ−μ_full)/μ_full where μ and μ_full are the mean NMSE of interest and mean NMSE of fully supervised training respectively. The best performance was for K-weighted 1D partitioned SSDU with a y_s input; its mean NMSE was only 1.1% and 0.8% larger than fully supervised for R_Ω = 8, 4 respectively. Figs. 4 and S2 show box plots of the NMSE of each method for R_Ω = 8 and R_Ω = 4 respectively: see Table S1 of the supplementary material for the numerical values.

Fig. 3.

Mean test set NMSE percentage difference between fully supervised and each methods at R_Ω = 8 and a 1D distributed M_Ω, where R_Λ has been tuned to minimize the test set NMSE. Fig. S1 shows a similar plot for R_Ω = 4.

Fig. 4.

NMSE for all methods at R_Ω = 8 and a 1D distributed M_Ω, where R_Λ has been tuned to minimize the test set NMSE. Fig. S2 shows a similar plot for R_Ω = 4 and the exact numerical values are in Table S1.

To evaluate whether the proposed changes to SSDU were statistically significant, we performed a one-sided Wilcoxon signed-rank test with p-value 0.01 on the test set NMSEs. For both the y_s and ${\tilde{y}}_s$ inputs, we found that there was a significant statistical difference between 2D and 1D partitioned SSDU. We also found that the difference between 1D partitioned SSDU and K-weighted 1D partitioned SSDU was statistically significant.

Figs. 5 and S3 show RSS estimates from the test set at R_Ω = 8 and R_Ω = 4 respectively. Qualitatively, K-weighted 1D partitioned SSDU performs the most similarly to fully supervised training. Although 2D partitioned SSDU has a competitive quantitative score for the estimate with ${\tilde{y}}_s$ input, it exhibits some streaking artifacts.

Fig. 5.

Reconstruction example with a 1D sub-sampled M_Ω and R_Ω = 8, with a R_Λ tuned to minimize the test set NMSE. A similar figure for R_Ω = 4 is in the supplementary material, Fig. S3.

Unweighted Noisier2Noise’s performance was substantially worse than SSDU. Therefore we compare SSDU and its modifications only in the remainder of this article.

B. Robustness to R_Λ

For actual, prospectively sampled data, it would not be possible to tune R_Λ on the ground truth test set NMSE. The practicality of SSDU therefore depends greatly on the robustness to R_Λ. Figs. 6 and S4 show the dependence of the mean test set NMSE on R_Λ for R_Ω = 8 and R_Ω = 4 respectively. K-weighted 1D partitioned SSDU was the most robust to the tuning of R_Λ. 2D partitioned SSDU was the least robust, especially for the estimate with y_s input. This is visualized in Fig. 7, which shows reconstruction examples for a number of R_Λs. K-weighted 1D partitioned SSDU performs very similarly for all R_Λs between 1.6 and 6, while 2D partitioned SSDU’s restoration quality significantly degrades qualitatively and quantitatively for mistunings as small as 0.1.

Fig. 6.

Dependence of the test set NMSE on the acceleration factor of the second mask M_Λ, denoted as R_Λ, at R_Ω = 8 for both outputs. 1D partitioned SSDU is far more robust to the tuning of R_Λ than 2D partitioned SSDU. Fully supervised learning does not use a second mask M_Λ, so has the same performance for all R_Λ. A similar figure for R_Ω = 4 is in the supplementary material, Fig. S3.

Fig. 7.

Robustness to R_Λ, where the blue box highlights the case where R_Λ is tuned. K-weighted 1D partitioned SSDU is very robust to R_Λ, with very similar restoration quality for all R_Λ between 1.6 and 6. 2D partitioned SSDU is far more sensitive, with substantial degradation in image quality for mistunings as small as 0.1. Here, we show the estimate with y_s input only.

C. Performance on 2D Sampled Brain Data

To further evaluate the role of the partitioning distribution, we also ran 1D and 2D partitioned SSDU on the brain data with a 2D Bernoulli sampled M_Ω. In this case, the type matching of the second mask to M_Ω is switched: 2D partitioned SSDU’s second mask has the same type of distribution as the first, while 1D partitioned SSDU has a different type. For M_Ω, we used a fully sampled 10 × 10 central region and a polynomial variable density that samples low frequencies with higher probability otherwise. We used R_Λ = 1.2 and R_Λ = 4 for 2D and 1D partitioned SSDU respectively. All other hyperparameters and network specifics were unchanged.

In this case, the best performance was 2D partitioned SSDU, which performed very similarly to fully supervised training: see Fig. 8. The ${\tilde{y}}_s$ input had a mean test set NMSE of 0.141 and 0.144 for 2D and 1D partitioned SSDU respectively, and the y_s input had 0.141 and 0.145, compared with 0.139 for fully supervised training. Although not shown in Fig. 8 for brevity, we also trained 2D partitioned SSDU with a ${\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}$ loss weighting. As for 1D partitioned SSDU in Section IV-A, we found that this reduced the mean NMSE further to 0.140 for both the y_s and ${\tilde{y}}_s$ input.

Fig. 8.

Reconstruction example from the brain fastMRI dataset with a 2D Bernoulli distributed M_Ω and R_Ω = 8. Compared to Fig. 5, the comparative performance of the SSDU algorithms are switched: here, 2D partitioned SSDU performs similarly to fully supervised training, while 1D partitioned SSDU suffers from streaking artifacts.

D. Performance on 1D Sampled Knee Data

We also trained K-weighted 1D partitioned SSDU on the fastMRI knee data with the same network architecture, training hyperparameters, and a 1D distributed M_Ω. The sub-sampling factor of the first and second masks were R_Ω = 8 and R_Λ = 2 respectively. The mean test set NMSE was 0.233 and 0.231 for the estimates with ${\tilde{y}}_s$ and y_s inputs respectively, compared with 0.230 for fully supervised training. Fig. 9 shows two example reconstructions from the test set, demonstrating competitive performance with fully supervised training qualitatively.

Fig. 9.

Two reconstruction examples of K-weighted 1D partitioned SSDU from the knee fastMRI dataset, where M_Ω is 1D. As in Fig. 5, K-weighted 1D partitioned SSDU’s restoration quality is very similar to fully supervised training.

V. Discussion

Due to its need for correction at inference, Unweighted Noisier2Noise had consistently the worst score. We therefore do not recommend using Unweighted Noisier2Noise in practice. Rather, we suggest using a variant of SSDU, which has a loss weighting that removes the need for such a correction.

The hierarchy of 1D and 2D partitioned SSDU depends on the distribution of M_Ω. In particular, the best performance was when they are both 1D or both 2D. It is conventional wisdom that better reconstruction quality is possible when k-space is randomly sub-sampled in both spatial dimensions (see, for instance, [58]). This is because the image-domain aliasing is incoherent in both dimensions, so is easier to remove. The superior performance of 1D partitioned SSDU compared with 2D partitioned SSDU when M_Ω is 1D shows that it is not necessarily true that the sampling set partition should also ideally be two-dimensional. Rather, better performance is possible when the distribution of M_Ω and M_ΛM_Ω are of the same type.

To see why, consider the nature of the aliasing caused by sub-sampling and further sub-sampling k-space, focusing on the example of a random 1D column sampled M_Ω. Such sampling causes the image-domain aliasing to be horizontally incoherent and vertically coherent. With a 1D column-wise Λ_t, further horizontal aliasing is introduced. Since the network cannot distinguish between the horizontal aliasing caused by Ω_t or Λ_t, the loss is minimized when the aliasing due to both is removed. On the other hand, a 2D Λ_t introduces some aliasing that is orthogonal to the original aliasing, which is distinguishable in principle. In this case, the loss is minimized when the network removes the aliasing caused by Λ_s, but not necessarily the original aliasing caused by Ω_s. This is visible in Figs. 5 and 8, where SSDU fails to completely remove artifacts caused by M_Ω when M_Λ does not have the same type of distribution.

This implies that, in general, better performance is possible when the distribution of the aliasing of ${\tilde{y}}_t$ and y_t are of the same type. For both the independent 1D column sampling and 2D Bernoulli sampling considered here, this can be achieved by choosing a M_Λ with the same type of distribution as M_Ω. Recently, in [59], this was also observed empirically for SSDU with random spoke sampling. However, such a procedure does not always achieve this goal. For instance, while the SSDU paper [33] considers a fully sampled central region and equidistant column sampling, recovery of images with regular under-sampling is not currently considered in the proposed framework. In this case, a Λ_t of the same type would not give a ${\tilde{y}}_t$ with the same aliasing type as y_t. The 2D Gaussian variable density partition employed in this article was originally constructed to handle such sampling patterns, and was found to perform very well in this context. Future work includes establishing the correct sampling set partitions for M_Ω distributions not in [33] or covered by the approach suggested here.

We found that K-weighted SSDU further improved the image quality and robustness to R_Λ. Consider the jth entry of the (squared) weighting (𝟙 − K)⁻¹ in terms of sampling probabilities:

$${\left(1-k_j\right)}^{-1}=\frac{1-{\tilde{p}}_j{p}_j}{p_j\left(1-{\tilde{p}}_j\right)}=\frac{\text{P}\left(j\notin \Lambda \cap \Omega \right)}{\text{P}\left(j\in \Omega \backslash \Lambda \right)}.$$

This leads to the following intuitive interpretation of the proposed loss weighting as compensation for the variable density of Ω and Λ. A smaller denominator ℙ(j ∈ Ω \ Λ) implies that the jth location occurs less frequently in the loss, which is compensated for by an increased weighting. A smaller numerator $\text{ℙ}\left(j\notin \Lambda \cap \Omega \right)$ implies that the jth location is estimated by the network less frequently, so has a decreased weighting.

The benefit of the (𝟙 − K)⁻¹ weighting highlights and addresses a general challenge of self-supervised learning with variable density sampling: regions of k-space sampled with lower probability are underrepresented in the loss. This issue has been noted in other works. For instance, for variable density reconstruction with Noise2Noise, [60] suggests weighting the loss function by the sampling density. An alternative approach was suggested in [61], which suggests passing the training target through the network before it is employed in the loss function. We note that if the sampling and partitioning had uniform density, such as in [56], K would also be uniform, so the proposed weighting would not be required. This may explain in part the empirical performance observed in [56].

When M_Ω was 1D, with the exception of 2D partitioned SSDU, Fig. 6 shows that the estimate with y_s input performed similarly or better than with ${\tilde{y}}_s$ input when R_Λ is tuned. This indicates that, for these methods, the advantage of using all the data in the input to the network outweighs the disadvantage that the input data has a different sampling distribution to the training data so is not guaranteed by Claim 1 or 2 to be correct in expectation. Heuristically, when M_Ω and M_ΛM_Ω are both variable density column-wise sampled, a network trained on doubly sub-sampled data is likely to also be able to handle singly sub-sampled data. However, for 2D partitioned SSDU, M_ΛM_Ω is no longer column-wise, see Fig. 2(c). Accordingly, 2D partitioned SSDU was the only method that had a higher NMSE for the y_s input compared to the ${\tilde{y}}_s$ input.

The best R_Λ for 2D partitioned SSDU was lower than competing methods: R_Λ = 1.8 and R_Λ = 1.2 for the y_s and ${\tilde{y}}_s$ inputs respectively. In [33], the sampling set partition was quantified in terms of the ratio ρ = |A_t|/|B_t|, and it was found that ρ = 0.4 offered the best performance. Since the M_Ω distributions are different here, the optimal ρ is not expected to necessarily be the same. For 2D partitioned SSDU R_Λ = 1.8 and R_Λ = 1.2 corresponds to ρ = 0.52 and ρ = 0.21 respectively, while for the other methods’s best performance at R_Λ = 4 corresponded to ρ = 0.57. Therefore the ρ were reasonably similar despite the substantial difference in R_Λ.

Since the network architecture uses ${\tilde{y}}_t$ in its coil sensitivity estimation module, not y_t, it is plausible that the differences between 1D and 2D partitioning could be due to poorer coil sensitivity estimation rather than an intrinsic property of the partition change. To examine this, we re-trained tuned 1D and 2D partitioned SSDU on the 1D sampled brain data with k-space masked to a central 10 × 10 region in the coil sensitivity estimation module. We found that the test set NMSE was within 1% of the usual approach. This verifies that the performance improvement was indeed a consequence of the partition change, not simply a consequence of specifics of the architecture.

Unweighted Noisier2Noise’s correction at inference (𝟙 − K)⁻¹ is only valid when an ${\ell }_2$ loss is used; we have found that other loss functions do not perform well in practice. This loss leads to smoothing artifacts, even for fully supervised training. For SSDU, since there is no correction term, loss functions other than ${\ell }_2$ are possible. For instance, in [33], a mixture of ${\ell }_2$ and ${\ell }_1$ was used. Better visual quality may be achievable when SSDU is implemented with a different loss; we do not suggest using an ${\ell }_2$ loss in general, it is only required here so that it can be compared directly with Noisier2Noise.

For all self-supervised methods in this work, we re-generated Λ_t once per epoch. This has similarities to the multi-mask SSDU approach proposed in [56]. However, in [56], a fixed number n_Λ of Λ_ts were generated for each Ω_t, each of which were treated as an additional member of the training set. Therefore, unlike in this article, each epoch was n_Λ times as long. Future work includes establishing whether it is also advantageous to limit the number of unique Λ_ts per Ω_t for the approach considered in this article.

All methods in this article were trained without taking measurement noise into account [62], [63]. Recent work by the present authors has shown that the additive and multiplicative versions of Noisier2Noise can be combined to recover higher fidelity images than SSDU in the presence of noise [64].

VI. Conclusions and Future Work

Based on the observation that SSDU is a version of Noisier2Noise with a particular rank-deficient loss weighting, we proved that SSDU correctly estimates Y₀ in expectation. This analysis led to two proposals that we found significantly improved SSDU’s performance in practice. Firstly, we propose employing a distribution of M_ΛM_Ω that is the same type as the original mask M_Ω. Secondly, we propose introducing a weighting of ${\left(\mathrm{𝟙}-K\right)}^{-\frac{1}{2}}$ in SSDU’s loss. We found that that each of these modifications significantly improved SSDU’s test set NMSE and robustness to R_Λ.

There are a number of other self-supervised learning methods that also use sampling set partitioning [37], [56], [65], some of which are variants of SSDU. For instance, [37], [65], [66] propose training two networks in parallel, one for each sampling subset, with a loss function that includes the difference between the outputs of the two networks. Another recent development is zero-shot SSDU [67], which shows that sampling set partitioning can also be applied to recover images without a training dataset [68]. Future work includes determining whether the theoretical and practical developments of this article can be extended to these methods.

Biographies

Charles Millard received the M.Sc. degree in physics from Imperial College London, London, U.K. and the doctorate degree in mathematics with biomedical imaging from the University of Oxford, Oxford, U.K. He is currently a Postdoctoral Researcher with the Wellcome Centre for Integrative Neuroimaging, University of Oxford. His research focuses on methods for reconstructing accelerated magnetic resonance imaging acquisitions with compressed sensing and deep learning.

Mark Chiew received the B.ASc. degree in engineering physics from the University of British Columbia, Vancouver, BC, Canada, and the Ph.D. degree in medical biophysics from the University of Toronto, Toronto, ON, Canada. From 2012 to 2022, he was a Postdoctoral Researcher and then the Royal Academy of Engineering Research Fellow with the University of Oxford, Oxford, U.K. Since 2022, he has been an Associate Professor with the University of Toronto, and the Scientist with Sunnybrook Research Institute, Toronto, ON. His research interests include the development of acquisition and image reconstruction strategies for magnetic resonance imaging.