Multi‐mask self‐supervised learning for physics‐guided neural networks in highly accelerated magnetic resonance imaging

Abstract

Self‐supervised learning has shown great promise because of its ability to train deep learning (DL) magnetic resonance imaging (MRI) reconstruction methods without fully sampled data. Current self‐supervised learning methods for physics‐guided reconstruction networks split acquired undersampled data into two disjoint sets, where one is used for data consistency (DC) in the unrolled network, while the other is used to define the training loss. In this study, we propose an improved self‐supervised learning strategy that more efficiently uses the acquired data to train a physics‐guided reconstruction network without a database of fully sampled data. The proposed multi‐mask self‐supervised learning via data undersampling (SSDU) applies a holdout masking operation on the acquired measurements to split them into multiple pairs of disjoint sets for each training sample, while using one of these pairs for DC units and the other for defining loss, thereby more efficiently using the undersampled data. Multi‐mask SSDU is applied on fully sampled 3D knee and prospectively undersampled 3D brain MRI datasets, for various acceleration rates and patterns, and compared with the parallel imaging method, CG‐SENSE, and single‐mask SSDU DL‐MRI, as well as supervised DL‐MRI when fully sampled data are available. The results on knee MRI show that the proposed multi‐mask SSDU outperforms SSDU and performs as well as supervised DL‐MRI. A clinical reader study further ranks the multi‐mask SSDU higher than supervised DL‐MRI in terms of signal‐to‐noise ratio and aliasing artifacts. Results on brain MRI show that multi‐mask SSDU achieves better reconstruction quality compared with SSDU. The reader study demonstrates that multi‐mask SSDU at R = 8 significantly improves reconstruction compared with single‐mask SSDU at R = 8, as well as CG‐SENSE at R = 2.

The proposed multi‐mask self‐supervised learning via data undersampling applies a holdout masking operation on the acquired measurements to split them into multiple pairs of disjoint sets for each training sample, while using one of these pairs for data consistency units and the other for defining loss, thereby more efficiently using the undersampled data.

Abbreviations

CG

conjugate gradient

CS

compressed sensing

DC

data consistency

DL

deep learning

PG‐DL

physics‐guided deep learning

RB

residual block

SNR

signal‐to‐noise ratio

SSDU

self‐supervision via data undersampling

1. INTRODUCTION

Data acquisition is lengthy in many magnetic resonance imaging (MRI) examinations, creating challenges for improving resolution and coverage, hence making accelerated MRI reconstruction an ongoing research topic. Parallel imaging ¹ , ² , ³ and compressed sensing ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ are two commonly used acceleration methods, with the former being the clinical gold standard for fast MRI, and the latter providing additional acceleration in a number of scenarios. However, acceleration rates remain limited as reconstructed images may suffer from noise amplification ¹⁰ or residual artifacts ¹¹ , ¹² in parallel imaging and compressed sensing, respectively. Recently, deep learning (DL) has emerged as an alternative to accelerated MRI because of its improved reconstruction quality compared with conventional approaches, especially at higher acceleration rates. ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸

Among DL methods, physics‐guided DL approaches, which incorporate the MRI‐encoding operator to solve a regularized inverse problem, have attracted interest because of their robustness. ¹⁸ , ¹⁹ Physics‐guided DL approaches unroll an iterative process that alternates between data consistency (DC) and regularization for a certain number of iterations. They are trained end‐to‐end, typically in a supervised manner, by minimizing the difference between network output and a ground truth reference obtained from fully sampled data. ¹⁴ , ²⁰ , ²¹ , ²² However, the acquisition of fully sampled data, especially on large patient populations, is either challenging or impossible in many practical scenarios. ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷

As supervised training becomes inoperative in the absence of fully sampled data, several methods have been proposed to train networks without fully sampled data. ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ Among these approaches, self‐supervision via data undersampling (SSDU) trains physics‐guided neural networks by utilizing only the acquired subsampled measurements. ³¹ In SSDU, the available measurements are split into two disjoint sets by a masking operation, which reduces the sensitivity to overfitting and is central for reliable performance. One of these sets is used in the DC units of the network, and the other is used to define the loss function in k‐space. For moderately high acceleration rates, the networks trained using SSDU match the performance of those from supervised learning. While SSDU demonstrated that the splitting of acquired points into two sets was sufficient for training a neural network for reconstruction from undersampled data, a strategy that augments the use of the subsampled data to improve the performance is essential for higher acceleration rates.

In this study, we sought to improve the performance of SSDU with multiple masks. The proposed multi‐mask SSDU splits acquired measurements into multiple pairs of disjoint sets for each training slice, while using one of these sets in a pair for DC units and the other for defining loss, similar to the original SSDU. The proposed multi‐mask SSDU approach is applied on fully sampled 3D knee MRI datasets from mridata.org, ³⁴ as well as a prospectively undersampled high‐resolution 3D brain MRI dataset, and is compared with parallel imaging, SSDU with a single mask, ³¹ and supervised DL‐MRI when fully sampled data are available. The results show that the proposed multi‐mask SSDU approach at high acceleration rates significantly improves upon SSDU and performs as well as supervised DL‐MRI, while the reader studies indicate that the proposed multi‐mask approach also outperforms the supervised DL‐MRI approach in terms of signal‐to‐noise ratio (SNR) improvement and aliasing artifact reduction.

2. METHODS

2.1. Supervised training of physics‐guided DL‐MRI reconstruction

Let ${\mathbf{y}}_{\mathbf{\Omega }}$ be the acquired subsampled measurements, with Ω denoting the subsampling pattern and x the image to be recovered. The forward model for encoding is

$${\mathbf{y}}_{\mathbf{\Omega }}\mathbf{=}{\mathbf{E}}_{\mathbf{\Omega }}\mathbf{x}\mathbf{+}\mathbf{n},$$ (1)

where ${\mathbf{E}}_{\mathbf{\Omega }}:{ℂ}^M\to {ℂ}^P$ is the encoding operator including the coil sensitivities and a partial Fourier matrix sampling the locations specified by Ω, and $\mathbf{n}\mathbf{\in }{ℂ}^P$ is measurement noise. For sub‐Nyquist sampling at high rates, the forward model may be ill‐conditioned, necessitating the use of regularization, leading to an inverse problem for image reconstruction:

$$\arg \underset{x}{\min }{‖{\mathbf{y}}_{\mathbf{\Omega }}\mathbf{-}{\mathbf{E}}_{\mathbf{\Omega }}\mathbf{x}‖}_2^2+\mathcal{R}\left(\mathit{x}\right),$$ (2)

where the first term represents DC and the second term, ℛ(·), is the regularizer. Several approaches may be used to iteratively solve the above optimization problem. ³⁵ In this work, we use variable splitting via the quadratic penalty method, ²¹ , ³¹ , ³⁵ which decouples DC and regularizer operations:

$${\mathbf{z}}^{\left(i-1\right)}=\arg \underset{\mathbf{z}}{\min }\mu {‖{\mathbf{x}}^{\left(i-1\right)}\mathbf{-}\mathbf{z}‖}_2^2+\mathcal{R}\left(\mathit{z}\right)\mathit{,}$$ (3)

$${\mathbf{x}}^{\left(i\right)}=\arg \underset{\mathbf{x}}{\min }{‖{\mathbf{y}}_{\mathbf{\Omega }}\mathbf{-}{\mathbf{E}}_{\mathbf{\Omega }}\mathbf{x}‖}_2^2+\mu {‖\mathbf{x}\mathbf{-}{\mathbf{z}}^{\left(i-1\right)}‖}_2^2,$$ (4)

where μ is the quadratic penalty parameter, x ⁽ⁱ⁾ is the network output at iteration i, z ⁽ⁱ⁾ is an intermediate variable, and x ⁽⁰⁾ is the initial image obtained from zero‐filled undersampled k‐space data. In physics‐guided DL, this iterative optimization is unrolled for a fixed number of iterations. Equation (3) corresponds to a regularizer, which is implicitly solved by a neural network, whereas Equation (4) has a closed form solution ³¹ that can be solved by gradient descent methods such as conjugate gradient. ²⁰

In traditional DL‐MRI approaches, training datasets contain pairs of undersampled k‐space/image and fully sampled k‐space/ground truth image. ¹⁴ , ²⁰ , ²¹ , ³⁶ Let ${\mathbf{y}}_{\mathit{ref}}^i$ be the fully sampled reference k‐space data for subject i, and $f\left({\mathrm{y}}_{\mathrm{\Omega }}^i,{\mathbf{E}}_{\mathrm{\Omega }}^i;\mathit{\theta }\right)$ denote the output of the unrolled network that is parametrized by θ for subsampled k‐space data ${\mathbf{y}}_{\mathbf{\Omega }}^{\mathit{i}}$ and the corresponding encoding matrix ${\mathbf{E}}_{\mathrm{\Omega }}^i$ of the same subject i. The supervised PG‐DL training is performed by defining the loss function in image domain or k‐space. ³¹ Training can be performed by minimizing a k‐space loss function as

$$\underset{\mathit{\theta }}{\min }\frac{1}{N}\sum _{i=1}^N\mathcal{L}\left({\mathbf{y}}_{\mathit{ref}}^i,{\mathbf{E}}_{\mathit{\text{full}}}^i\left(f\left({\mathrm{y}}_{\mathrm{\Omega }}^i,{\mathbf{E}}_{\mathrm{\Omega }}^i;\mathit{\theta }\right)\right)\right),$$ (5)

where N is the number of fully sampled training data in the database, ${\mathbf{E}}_{\mathit{\text{full}}}^i$ is the fully sampled encoding operator that transforms network output to k‐space, and $\mathcal{L}\left(.,.\right)$ is the loss between the fully sampled and reconstructed k‐spaces. The sampling locations, Ω, may vary per subject in a more general setup (i.e., indexed by i). However, this was not included for simplicity of notation.

2.2. Self‐supervision via data undersampling

To enable training without fully sampled datasets, SSDU has been proposed, ³¹ where the acquired subsampled data indices, Ω, from each scan are divided into two disjoint sets, Θ and Λ. Θ is used in DC units in the unrolled network and Λ is used to define the loss function, and the following self‐supervised loss function is minimized:

$$\underset{\mathit{\theta }}{\min }\frac{1}{N}\sum _{i=1}^N\mathcal{L}\left({\mathbf{y}}_{\mathrm{\Lambda }}^i,{\mathbf{E}}_{\mathrm{\Lambda }}^i\left(f\left({\mathrm{y}}_{\mathrm{\Theta }}^i,{\mathbf{E}}_{\mathrm{\Theta }}^i;\mathit{\theta }\right)\right)\right).$$ (6)

Unlike the supervised approach, only a subset of measurements, Θ, are used as the input to the unrolled network. The network output is transformed to k‐space, where the loss is calculated only at unseen k‐space indices, Λ. After training is completed, testing is performed on unseen data using all available measurements, Ω.

2.3. Proposed multi‐mask SSDU

SSDU reconstruction quality degrades at very high acceleration rates due to higher data scarcity, arising from the splitting into Θ and Λ. To tackle this issue, we propose a multi‐mask SSDU approach, which retrospectively splits acquired indices Ω into disjoint sets Θ and Λ multiple times, as shown in Figure 1. Formally, we split available measurements multiple times for each subject i such that for each partition $\mathrm{\Omega }={\mathrm{\Theta }}_j\cup {\mathrm{\Lambda }}_j$, for j = 1, …, K, denoting the number of partitions for each scan. Similar to SSDU, each pair of sets in each scan was disjoint (i.e., ${\mathrm{\Lambda }}_j=\mathrm{\Omega }\backslash {\mathrm{\Theta }}_j$ for j ∈ 1, …, K). In other words, Ω is retrieved by the union of each pair of ${\mathrm{\Lambda }}_j$ and ${\mathrm{\Theta }}_j$ for any j ∈ 1, …, K. Hence, the loss function to minimize during training becomes

$$\underset{\mathit{\theta }}{\min }\frac{1}{N·K}\sum _{i=1}^N\sum _{j=1}^K\mathcal{L}\left({\mathbf{y}}_{{\mathrm{\Lambda }}_j}^i,{\mathbf{E}}_{{\mathrm{\Lambda }}_j}^i\left(f\left({\mathrm{y}}_{{\mathrm{\Theta }}_j}^i,{\mathbf{E}}_{{\mathrm{\Theta }}_j}^i;\mathit{\theta }\right)\right)\right).$$ (7)

The proposed multi‐mask approach enables efficient use of available data by ensuring a higher fraction of low and high frequency components are utilized in the training and loss masks. Such utilization was inherently limited in the original SSDU approach, because each acquired k‐space point was either used in training or loss masks only once.

FIGURE 1.

The multi‐mask self‐supervised learning scheme to train physics‐guided deep learning without fully sampled data. The acquired subsampled k‐space measurements for each scan, Ω, are split into multiple disjoint sets, ${\mathrm{\Theta }}_j$ and ${\mathrm{\Lambda }}_j$, in which $\mathrm{\Omega }={\mathrm{\Theta }}_j\cup {\mathrm{\Lambda }}_j$, for j ∈ 1, …, K. The first set of indices, ${\mathrm{\Theta }}_j$, is used in the data consistency (DC) unit of the unrolled network, while the latter set, ${\mathrm{\Lambda }}_j$, is used to define the loss function for training. During training, the output of the network is transformed to k‐space, and the available subset of measurements at ${\mathrm{\Lambda }}_j$ are compared with the corresponding reconstructed k‐space values. Based on this training loss, the network parameters are subsequently updated

2.4. 3D imaging datasets

Fully sampled 3D knee datasets were obtained from mridata.org, ³⁴ and were acquired with approval from the local institutional review board on a 3‐T GE Discovery MR 750 system with an eight‐channel knee coil array using a fast spin‐echo sequence. The relevant imaging parameters were: field of view (FOV) = 160 × 160 × 154 mm³, resolution = 0.5 × 0.5 × 0.6 mm³, and matrix size = 320 × 320 × 256.

Brain imaging was performed on healthy subjects using a standard Siemens 3D magnetization‐prepared rapid gradient echo imaging (MPRAGE) sequence on a 3‐T Siemens Magnetom Prisma (Siemens Healthcare, Erlangen, Germany) system using a 32‐channel receiver head coil‐array. ³¹ The imaging protocols were approved by the local institutional review board, and written informed consent was obtained from all participants before each examination for this HIPAA‐compliant study. The relevant imaging parameters were: FOV = 224 × 224 × 157 mm³, resolution = 0.7 × 0.7 × 0.7 mm³, matrix size = 320 × 320 × 224, prospective acceleration R = 2 (uniform in k_y) and autocalibration signal (ACS) lines = 32. ³¹

The 3D k‐space datasets were inverse Fourier‐transformed along the read‐out direction, and these slices were processed individually. The knee and brain datasets were retrospectively undersampled to R = 8 using a uniform sheared 2D undersampling pattern. ³⁷ Additionally, for the knee datasets, where a fully sampled reference was available, further undersampling was performed at R = 8 using uniform 1D and 2D (k_y‐k_z) random, and 1D and 2D and Poisson undersampling masks. The undersampling masks are provided in Figure  S1 . Finally, knee datasets were also undersampled to R = 12 using 2D random and Poisson undersampling masks. A 24 × 24 and 32 × 32 ACS region in the k_y‐k_z plane was kept fully sampled for the knee and brain datasets, respectively. ³¹ The training sets for both knee and brain datasets consisted of 300 slices from 10 subjects, formed by taking 30 slices from each subject. For knee MRI, two different subjects with 200 slices were used for validation in multi‐mask hyperparameter tuning, and eight other different subjects were used for the testing of the final method. For the brain dataset, the testing was performed on nine different subjects. The proposed multi‐mask SSDU approach was compared with: the parallel imaging method, CG‐SENSE; the blind compressed sensing method (Blind‐CS) ³⁸ ; the zero‐shot learning approach, deep image prior (DIP) ³⁹ , ⁴⁰ ; the state‐of‐the‐art self‐supervised learning approach (SSDU) ³¹ ; and the state‐of‐the‐art supervised DL method when applicable. ²⁰ For Blind‐CS and DIP, we used publicly available implementations from github.com/wenbihan/FRIST_ivp2017 and github.com/MLI‐lab/ConvDecoder, respectively. We tuned the number of epochs for DIP for maximal performance.

2.5. Choice of multi‐mask hyperparameters

There are several tunable hyperparameters in multi‐mask SSDU, including the number of partitions, K in Equation (7), as well as the distribution and size of Λ as in SSDU. A variable‐density Gaussian distribution was used for Λ in ³¹ for a single mask. In this study, we used a uniformly random distribution for the proposed approach, as the benefits of a variable density distribution diminish with multiple masks (Figure  S2 ). In, ³¹ the size of Λ was optimized to ρ = |Λ|/|Ω| = 0.4, which is also the optimal choice for the distribution considered here (Figure  S3 ). After these two hyperparameters were set, the number of partitions of each scan, K, was varied among 3, 5, 6, 7, 8, and 10 to optimize the remaining distinct hyperparameter of the multi‐mask SSDU.

2.6. Network and training details

The iterative optimization problem in Equations (3) and (4) was unrolled for T = 10 iterations. Conjugate gradient descent was used in DC units of the unrolled network. ²⁰ , ³¹ The proximal operator corresponding to the solution of Equation (3) employs the ResNet structure used in SSDU. ³¹ It comprises input and output convolution layers and 15 residual blocks (RBs) each containing two convolutional layers, where the first layer is followed by a rectified linear unit (ReLU) and the second layer is followed by a constant multiplication layer. All layers had a kernel size of 3 × 3, 64 channels. The unrolled network, which shares parameters across the unrolled iterations, had a total of 592,129 trainable parameters. As in SSDU, a ResNet structure was used for the regularizer in Equation (3), where the network parameters were shared across the unrolled network. ³¹ Coil sensitivity maps were generated from 24 × 24 center of k‐space using ESPIRiT. ⁴¹

As a preprocessing step, the maximum absolute value of the k‐space for each slice in the datasets was normalized to 1 in all cases. The real and imaginary parts of the complex MRI dataset were concatenated as two channels prior to inputting into the network. Separate networks for knee and brain datasets were trained using the Adam optimizer with a learning rate of 5 x 10⁻⁴, by minimizing a normalized ℓ₁ – ℓ₂ loss function defined as $\mathcal{L}\left(\mathit{u},\mathit{v}\right)=\frac{{‖\mathit{u}-\mathit{v}‖}_2}{{‖\mathit{u}‖}_2}+\frac{{‖\mathit{u}-\mathit{v}‖}_1}{{‖\mathit{u}‖}_1}$, ³¹ with a batch size of 1 over 100 epochs. All training was performed using Tensorflow in Python, and was processed on a workstation with an NVIDIA Tesla V100 GPU with 32 GB of memory. The implementation of this method will be provided online (https://github.com/byaman14/SSDU).

2.7. Image evaluation

Quantitative assessment of the experimental results was performed using normalized mean square error (NMSE) and structural similarity index (SSIM) when fully sampled data were available as reference. Moreover, qualitative assessment of the image quality from different reconstruction methods was performed by an experienced radiologist. For knee MRI, the proposed multi‐mask SSDU was compared with ground truth obtained from fully sampled data, SSDU, and the parallel imaging method CG‐SENSE, all at R = 8 with 2D uniform undersampling. For brain MRI, the proposed multi‐mask SSDU was compared with SSDU at R = 8. Additionally, the CG‐SENSE approach at the acquisition acceleration R = 2 was evaluated to serve as the clinical baseline. The reader, with 15 years of experience of musculoskeletal and neuro imaging, was blinded to the reconstruction method, which were shown in a randomized order to avoid bias, except for the knowledge of the reference image in the knee MRI dataset. Evaluations were based on a four‐point ordinal scale, adopted from ¹⁴ for blurring (1: no blurring, 2: mild blurring, 3: moderate blurring, 4: severe blurring), SNR (1: excellent, 2: good, 3: fair, 4: poor), aliasing artifacts (1: none, 2: mild, 3: moderate, 4: severe), and overall image quality (1: excellent, 2: good, 3: fair, 4: poor). Wilcoxon signed‐rank test was used to evaluate the scores with a significance level of p less than 0.05.

3. RESULTS

3.1. Number of partitions for multi‐mask SSDU

Figure 2 shows the effect of the proposed multi‐mask self‐supervised network training at R = 8 using 2D uniform sheared undersampling with different numbers of masks (K = 3, 5, 6, 7, 8, and 10), as well as the ground truth reference and the zero‐filled undersampled data. The multi‐mask SSDU approach suppresses residual artifacts as K increases from 3 to 6. At K = 7, the visible residual artifacts are removed completely. When K is further increased to 8 and 10, residual artifacts reappear. The quantitative assessment of the validation dataset further confirms this qualitative assessment. The median and interquartile range of SSIM values on the validation set were 0.8256 [0.7980, 0.8507], 0.8260 [0.8002, 0.8516], 0.8264 [0.8016, 0.8527], 0.8267 [0.8027, 0.8537], 0.8263 [0.8007, 0.8519], and 0.8257 [0.7989, 0.8511], and the NMSE values were 0.0138 [0.0121, 0.0158], 0.0135 [0.0119, 0.0158], 0.0135 [0.0119, 0.0157], 0.0134 [0.0118, 0.0156], 0.0135 [0.0119, 0.0158], and 0.0137 [0.0121, 0.0159] using Gaussian selection for K ∈ 3, 5, 6, 7, 8, and 10, respectively. Hence, K = 7 was used for the remainder of the study.

FIGURE 2.

A representative test slice showing the reconstruction results for different numbers of partitions K using a 2D uniform sheared R = 8 undersampling mask. Red arrows mark residual artifacts for K ≤ 6 and K ≥ 8. These artifacts are suppressed at K = 7, which is used for the remainder of the study. NMSE, normalized mean square error; SSIM, structural similarity index

3.2. 3D imaging datasets

Figure 3 depicts representative reference and reconstruction results of the 3D knee dataset using CG‐SENSE, Blind‐CS, DIP, supervised DL‐MRI, SSDU, and the proposed multi‐mask SSDU, as well as the difference images of these methods with respect to the reference for the 2D uniform sheared R = 8 undersampling mask. Red arrows indicate that CG‐SENSE suffers from highly visible residual artifacts and noise amplification. Blind‐CS and DIP severely suffer from the blurring artifacts. SSDU alleviates these artifacts substantially, although residual artifacts remain. The proposed multi‐mask SSDU further removes these artifacts for both slices shown, while achieving similar reconstruction quality to supervised DL‐MRI for the first slice, and further reducing the residual aliasing artifacts visible in the supervised DL‐MRI approach for the second slice. Quantitative metrics and difference images in the figure further confirm that multi‐mask SSDU outperforms SSDU, while performing similarly to supervised DL‐MRI. Additional experimental results on the 3D knee dataset using different undersampling patterns at R = 8 and R = 12 are provided in Figures S4, S5, and S6. In all of these experiments, multi‐mask SSDU visibly and quantitatively outperformed SSDU, further reducing the gap between self‐supervised learning and supervised DL‐MRI.

FIGURE 3.

(A, B) Representative test slices from the 3D fast spin‐echo knee magnetic resonance imaging (MRI) dataset showing the reconstruction results for the proposed multi‐mask self‐supervised deep learning (DL)‐MRI (multi‐mask SSDU), self‐supervised DL‐MRI (SSDU), supervised DL‐MRI, DIP, Blind‐CS, and CG‐SENSE approaches for retrospective 2D uniform undersampling R = 8, as well as the error images with respect to the fully sampled reference. CG‐SENSE suffers from substantial residual artifacts that are shown with red arrows for both slices. Blind‐CS and DIP suffer severely from blurring artifacts. DL‐MRI with SSDU learning suppresses a large portion of these artifacts, but still exhibits visible residual artifacts in both scenarios. The proposed multi‐mask SSDU successfully suppresses these artifacts further for both slices; in (A) it closely matches the performance of supervised DL‐MRI, and in (B) it reduces residual aliasing further compared with supervised DL‐MRI. Blind‐CS, the blind compressed sensing method; CG‐SENSE, the parallel imaging method; DIP, deep image prior; NMSE, normalized mean square error; SSDU, self‐supervision via data undersampling; SSIM, structural similarity index

Table 1 summarizes the median and interquartile ranges for NMSE and SSIM value metrics for different undersampling masks and acceleration rates across the whole knee MRI test datasets. In all cases, CG‐SENSE reconstruction is significantly outperformed by all the DL approaches. Among the DL approaches, supervised DL‐MRI outperforms self‐supervised learning methods, while multi‐mask SSDU quantitatively improves upon SSDU.

TABLE 1.

The median and interquartile ranges for NMSE and SSIM metrics for different undersampling masks and acceleration rates. Note that due to the different size of the ACS data, 1D masks correspond to an effective acceleration rate of 5.2, while the 2D masks yield an effective acceleration rate of 7.7

		CG‐SENSE	DIP	Blind‐CS	Supervised DL‐MRI	SSDU	Multi‐mask SSDU
Uniform 1D R = 8	NMSE	0.0994 [0.0949, 0.1419]	0.0457 [0.0316, 0.0627]	0.0192 [0.0172, 0.2274]	0.0122 [0.0108, 0.0139]	0.0166 [0.0147, 0.0191]	0.0140 [0.0124, 0.0161]
	SSIM	0.4698 [0.4193, 0.5353]	0.7224 [0.6819, 0.7435]	0.7945 [0.7648, 0.8332]	0.8505 [0.8306, 0.8751]	0.8211 [0.7947, 0.8527]	0.8315 [0.8084, 0.8600]
Uniform 2D R = 8	NMSE	0.1475 [0.1291, 0.1779]	0.0438 [0.0297, 0.0622]	0.0187 [0.0168, 0.0218]	0.0124 [0.0112, 0.0143]	0.0164 [0.0148, 0.0189]	0.0135 [0.0123, 0.0155]
	SSIM	0.4411 [0.3797, 0.4976]	0.7124 [0.6704, 0.7452]	0.7884 [0.7526, 0.8126]	0.8421 [0.8201, 0.8662]	0.8150 [0.7877, 0.8426]	0.8298 [0.8067, 0.8560]
Random 1D R = 8	NMSE	0.0994 [0.0805, 0.1236]	0.0392 [0.0216, 0.0524]	0.0179 [0.0162, 0.0205]	0.0121 [0.0107, 0.0139]	0.0156 [0.0135, 0.0177]	0.0137 [0.0121, 0.0155]
	SSIM	0.4886 [0.4305, 0.5571]	0.7326 [0.7005, 0.7715]	0.8032 [0.7789, 0.8407]	0.8524 [0.8314, 0.8756]	0.8328 [0.8089, 0.8615]	0.8367 [0.8144, 0.8633]
Random 2D R = 8	NMSE	0.1473 [0.1301, 0.1759]	0.0412 [0.0256, 0.0567]	0.0197 [0.0178, 0.0224]	0.0130 [0.0117, 0.0149]	0.0173 [0.0155, 0.0199]	0.0145 [0.0131, 0.0165]
	SSIM	0.4239 [0.3631, 0.4766]	0.7286 [0.6957, 0.7592]	0.7914 [0.7648, 0.8229]	0.8379 [0.8164, 0.8637]	0.8123 [0.7853, 0.8417]	0.8224 [0.8002, 0.8509]
Poisson R = 8	NMSE	0.1035 [0.0937, 0.1206]	0.0358 [0.0182, 0.0497]	0.0158 [0.0134, 0.0178]	0.0101 [0.0091, 0.0112]	0.0131 [0.0118, 0.0149]	0.0108 [0.0098, 0.0121]
	SSIM	0.4885 [0.4394, 0.5397]	0.7325 [0.6914, 0.7629]	0.8075[0.7800, 0.8370]	0.8554 [0.8365, 0.8793]	0.8312 [0.8066, 0.8585]	0.8421 [0.8212, 0.8679]
Random 2D R = 12	NMSE	0.1331 [0.1207, 0.1556]	0.0542 [0.0386, 0.0785]	0.0247 [0.0218, 0.0287]	0.0157 [0.0141, 0.0179]	0.0221 [0.0198, 0.0254]	0.0185 [0.0167, 0.0208]
	SSIM	0.4325 [0.3756, 0.4796]	0.6834 [0.6576, 0.7182]	0.7614 [0.7264, 0.7916]	0.8148 [0.7916, 0.8431]	0.7809 [0.7517, 0.8151]	0.7982 [0.7722, 0.8288]
Poisson R = 12	NMSE	0.0876 [0.0795, 0.1012]	0.0491 [0.0342, 0.0673]	0.0173 [0.0151, 0.0192]	0.0119 [0.0107, 0.0133]	0.0151 [0.0136, 0.0169]	0.0129 [0.0117, 0.0145]
	SSIM	0.5119 [0.4638, 0.5597]	0.7014 [0.6632, 0.7376]	0.7753 [0.7385, 0.8164]	0.8362 [0.8156, 0.8625]	0.8133 [0.7862, 0.8442]	0.8326 [0.8098, 0.8609]

Abbreviations: ACS, autocalibration signal; Blind‐CS, the blind compressed sensing method; CG‐SENSE, the parallel imaging method; DIP, deep image prior; DL‐MRI, deep learning magnetic resonance imaging; NMSE, normalized mean square error; SSDU, self‐supervision via data undersampling; SSIM, structural similarity index.

Figure 4 demonstrates CG‐SENSE reconstruction of a slice of the 3D‐MPRAGE dataset at prospective acceleration R = 2, as well as CG‐SENSE, Blind‐CS, DIP, SSDU, and the proposed multi‐mask SSDU approach at retrospective acceleration R = 8 using the 2D uniform sheared undersampling mask. Blind‐CS and DIP reconstructions significantly suffer from blurring artifacts. SSDU at high acceleration R = 8 achieves a similar reconstruction quality to CG‐SENSE at acquisition acceleration R = 2. Multi‐mask SSDU further improves the reconstruction quality by suppressing the noise evident in SSDU and CG‐SENSE.

FIGURE 4.

Reconstruction results from prospectively two‐fold equispaced undersampled brain magnetic resonance imaging (MRI). SSDU, multi‐mask SSDU, DIP, Blind‐CS, and CG‐SENSE are applied at further retrospective acceleration rates of 8 with equispaced sheared k_y‐k_z undersampling patterns, while CG‐SENSE is also used at the acquisition rate of 2, which serves as the clinical baseline. CG‐SENSE suffers from visibly higher noise amplification at R = 8. Blind‐CS and DIP reconstructions significantly suffer from blurring artifacts. SSDU DL‐MRI performs successful reconstruction at R = 8, while achieving similar image quality to CG‐SENSE at R = 2. The proposed multi‐mask SSDU DL‐MRI further enhances the SSDU DL‐MRI performance by achieving a lower noise level in reconstruction results. Blind‐CS, the blind compressed sensing method; CG‐SENSE, the parallel imaging method; DIP, deep image prior; DL, deep learning; SSDU, self‐supervision via data undersampling

3.3. Image evaluation scores

Figure 5A,B summarize the reader study results for knee and brain datasets using the 2D uniform sheared R = 8 undersampling mask, respectively. For knee MRI, the proposed multi‐mask SSDU was rated highest in terms of SNR, with a statistically significant improvement over all other methods, except for supervised DL‐MRI. For blurring, ground truth data was rated better than all the other methods. In terms of aliasing artifacts and overall image quality, the proposed multi‐mask SSDU approach was rated best compared with the other methods and ground truth. In terms of these two evaluation criteria, all DL‐MRI approaches and the reference showed similar statistical behavior, except SSDU was statistically worse than the proposed multi‐mask SSDU and supervised approach in terms of aliasing artifacts. A more comprehensive comparison also containing reader scores for CG‐SENSE is presented in Figure S7.

FIGURE 5.

(A) Reader study for knee magnetic resonance imaging (MRI) for reconstructions using uniform 2D undersampling masks at R = 8. Bar‐plots show average reader scores and their standard deviation across the test subjects. Statistical testing was performed by one‐sided Wilcoxon single‐rank test, with * showing a significant statistical difference with p < 0.05. In terms of SNR, the proposed multi‐mask SSDU was rated highest, and statistically better than all approaches, except for supervised DL‐MRI. For blurring, ground truth data was rated statistically better than all the other methods, except for the proposed multi‐mask SSDU. In terms of aliasing artifacts and overall image quality, the proposed multi‐mask SSDU approach was rated best compared with the other methods and ground truth. In terms of these two evaluation criteria, all DL‐MRI approaches and the reference showed similar statistical behavior, except SSDU was statistically worse than the proposed multi‐mask SSDU and supervised approach in terms of aliasing artifacts. (B) Reader study for brain MRI. CG‐SENSE at R = 2, and the proposed multi‐mask SSDU and SSDU at R = 8 using a uniform 2D undersampling mask, were in good agreement in terms of SNR and blurring. In terms of aliasing artifacts and overall image quality, the proposed multi‐mask SSDU approach received the best scores, while CG‐SENSE at R = 2 was rated lowest and showed a significant statistical difference with the proposed multi‐mask SSDU in terms of both evaluation criteria and SSDU in terms of overall image quality. The proposed multi‐mask SSDU was also rated statistically better than SSDU in terms of aliasing artifacts. CG‐SENSE, the parallel imaging method; DL, deep learning; FSE, fast spin‐echo; MPRAGE, magnetization‐prepared rapid gradient echo imaging; SNR, signal‐to‐noise ratio; SSDU, self‐supervision via data undersampling

For brain MRI, DL‐MRI reconstructions trained using the proposed multi‐mask SSDU and SSDU approach at an acceleration rate of 8 performed similar to CG‐SENSE at acquisition R = 2 in terms of SNR and blurring. However, in terms of aliasing artifacts, the proposed multi‐mask SSDU significantly outperformed its counterparts. In terms of overall image quality, both SSDU methods at R = 8 showed a statistically significant improvement over CG‐SENSE at R = 2, while the proposed multi‐mask SSDU achieved the best performance.

4. DISCUSSION

In this work, we extended our earlier work on self‐supervision via data undersampling, which trains a physics‐guided neural network without fully sampled data, to a multi‐mask setting where multiple pairs of disjoint sets were used for each training slice in the dataset. The training of physics‐guided DL‐MRI reconstruction without ground truth data remains an important topic, because the acquisition of fully sampled data is either impossible or challenging in a number of scenarios. ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ Among multiple methods proposed for this goal, ²⁸ , ²⁹ , ³⁰ , ⁴² , ⁴³ self‐supervision directly uses the acquired data without relying on generative models or intermediate estimates. Our work makes several contributions to these approaches. The main contribution of the proposed multi‐mask self‐supervised learning approach is to use the available undersampled data more efficiently to enable physics‐guided DL training, by retrospectively splitting these data into multiple 2‐tuple of sets for the DC units during training and for defining loss. Another resulting contribution of the proposed multi‐mask SSDU is an alternative data augmentation strategy for DL‐MRI reconstruction, via the retrospective holdout masking of the acquired measurements multiple times, with potential applications even beyond self‐supervised learning. ⁴⁴ Finally, we applied the proposed multi‐mask approach on knee and brain MRI datasets using different undersampling and acceleration rates, showing its improved reconstruction performance compared with a single‐mask SSDU approach. Specifically, the extensive experimental results using different subsampling patterns on the retrospectively subsampled 3D knee dataset at R = 8 and R = 12 show that the proposed multi‐mask SSDU consistently outperforms SSDU, while performing similarly to the supervised DL‐MRI approach. Similarly, on prospectively subsampled brain MRI, multi‐mask SSDU at R = 8 enhances the reconstruction quality of SSDU, while achieving lower noise level compared with SSDU at R = 8 and CG‐SENSE at the acquisition R = 2.

As mentioned earlier, the proposed multi‐mask SSDU approach can be interpreted as an alternative technique for data augmentation in DL‐MRI reconstruction. With the proposed multi‐mask data augmentation, self‐supervised training was rated higher than supervised training in the reader study for knee imaging by a musculoskeletal expert reader in terms of noise and aliasing artifacts. Furthermore, Figure 3 shows example slices, where multi‐mask self‐supervised learning displays a better performance in handling artifacts compared with supervised DL‐MRI, despite having lower SSIM and NMSE values. While these observations may seem surprising at first, they are consistent with recent studies that show quantitative metrics may not always align with the reconstruction performance. ⁴⁵ Additionally, there are recent studies that show self‐supervised DL approaches outperforming their supervised counterparts in various applications. ⁴⁶ , ⁴⁷ These and other studies suggest that supervised learning may preclude discovery, hence it may not generalize well on unseen data or may not be as robust as self‐supervised learning techniques. ⁴⁸ Another interesting finding from the reader study on knee data was the worse scores given to the fully sampled ground truth compared with DL‐MRI methods. The expert reader noted the low SNR of the fully sampled acquisition, due to the high acquisition resolution compared with conventional clinical scans, which was substantially improved visually using the inherent noise reduction of DL‐MRI reconstruction.

While the proposed multi‐mask SSDU approach enhances the SSDU performance, it also has a longer training time by a factor of K compared with SSDU due to the increased size of the training dataset. Due to these lengthy training times, holdout cross‐validation was used for the hyperparameter selection substudy for optimizing K instead of n‐fold cross‐validation. Furthermore, while the proposed multi‐mask approach enables data augmentation, helping to overcome data scarcity and enhance reconstruction quality, it also bears the risk of overfitting. In a broader context, it is understood that data augmentation can lead to massive datasets, but when this idea is applied to augment initially limited datasets, it may result in overfitting. ⁴⁹ This phenomenon was also observed in our study as the reconstruction quality does not monotonically improve with increasing K, and residual artifacts reappear for K of 8 or higher. The problem of choosing the optimal size of the postaugmented dataset, which corresponds to K in our setup, remains an open problem in the broader machine learning community and this challenge has been highlighted in a recent survey on data augmentation as “There is no consensus as to which ratio of original to final dataset size will result in the best performing model”. ⁴⁹ Hence, while we have optimized K, this was achieved using the same experimental settings of. ³¹ We note that the optimal value for K may differ based on the selection of hyperparameters, such as the number of epochs or the learning rate. Nonetheless, our results readily show that multi‐mask SSDU improves upon the single‐mask SSDU in terms of quantitative metrics for any choice of K more than 1, while also suggesting that it is not advantageous to choose a very high value of K, both from a performance perspective, and from a practical viewpoint, because of the increased training time.

In this study, we have split measurements into training and loss sets in a disjoint manner as SSDU has shown that overlaps between these sets lead to degraded reconstruction performance. ³¹ A uniformly random selection of masks was used in the multi‐mask SSDU. This was motivated by the issue that splitting Ω based on a Gaussian random selection leads to selecting mostly low‐frequency components from scarce data, especially at high acceleration rates. With a Gaussian selection of Λ, a multi‐mask approach still tends to select low‐frequency components for each mask. Figure S2 shows that using uniformly random selection in combination with multi‐mask selection may circumvent this issue, because this will ensure that both low and high frequency are contained in the loss masks of each scan.

Although we concentrated on random selection of masks, another special type of multi‐mask SSDU may be based on using a cyclic selection that ensures all acquired measurements are used for both training and loss. Figure S8 shows a comparison between the cyclic multi‐mask and multi‐mask SSDU approach, with multi‐mask SSDU showing better reconstruction quality. Multi‐mask SSDU does not impose any bounds on K, allowing K × |Λ| ≥ |Ω|, while cyclic multi‐mask SSDU strictly imposes K × |Λ| = |Ω|. Hence, although cyclic multi‐mask SSDU ensures every point in Ω is eventually used in both DC units and for defining the k‐space loss, it inherently limits the number of masks for training, which in turn hinders the amount of improvement in reconstruction quality.

The multi‐masking approach proposed in this work may also be adapted to the supervised learning setting by using multiple random Θ_j in the DC units of the unrolled network, while calculating the loss on the fully sampled k‐space. This extension to supervised training, which introduces an additional degree of randomness to the training process, was recently shown to improve performance over a conventional supervised DL‐MRI approach. ⁵⁰ However, we note that because this is a new extension arising from this work, comparisons in this study were made with the conventional DL‐MRI approach that is used in the literature. ²⁰

The proposed multi‐mask SSDU approach also shares similarities with bootstrap aggregation mechanisms. In bootstrap approaches, multiple subdatasets are generated by randomly sampling from a main dataset. The final prediction is performed by averaging outputs from each of these subdatasets to reduce the variance among trained models. However, in multi‐mask SSDU, each sample in the main dataset is subsampled multiple times by retrospectively splitting its measurements into disjoint sets. As a result, an aggregated large dataset that contains the measurements of each sample in the main dataset multiple times is obtained and used for training. Unlike bootstrapping approaches, the proposed approach performs final prediction by directly using the model trained on the aggregated large dataset.

The current study has some limitations. In the proposed multi‐mask SSDU, we optimized the hyperparameters ρ and K independently for two main reasons: (1) if the joint optimization led to a different ρ value, then this would create a confounding variable for the direct comparison with the single‐mask scenario; and (2) optimizing over the two hyperparameters jointly leads to a large number of trainings for marginal gain. Such a large number of trainings, which does not have a substantial implication in terms of the perspective of the study, may also come at an environmental cost, as the training of DL models has been shown to lead to a considerable amount of carbon emissions. ⁵¹ Thus, in the current study, we concentrated on the individual optimization of the K parameter for the fixed ρ value that works best for single‐mask SSDU, ³¹ as this provides a fairer comparison. Additionally, this study focused on methodological development to improve the performance on single‐mask SSDU without a specific application that may leverage large stores of existing undersampled data. With the methodology in place, such applications are being pursued in subsequent studies, both in late gadolinium enhancement (LGE) cardiac and brain MRI applications. ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ Moreover, the proposed multi‐mask SSDU may also be synergistically combined with dynamic cardiac MRI applications, in which spatio‐temporal information can be leveraged to boost the performance at high acceleration rates. ⁵⁶ , ⁵⁷ , ⁵⁸

More recently, zero‐shot learning approaches have attracted interest as they enable training using only a single slice. ⁵⁹ , ⁶⁰ , ⁶¹ Thus, zero‐shot learning approaches do not require any external dataset for training, unlike database DL approaches such as SSDU and the proposed multi‐mask SSDU. We note that Blind‐CS methods, ⁹ , ³⁸ , ⁶² which predate DL methods, are also learning‐based techniques trained on a single slice, requiring no external database. Furthermore, recent Blind‐CS methods, which simultaneously combine the sparsity with low rankness, have shown great promise for solving inverse problems, ⁶² which may further improve upon the performance of FRIST reported in this manuscript. Blind‐CS methods typically include hand‐tuned parameters for weighing data consistency and regularization terms in their objective functions, ³⁸ , ⁶² which are also learnable in some of the more contemporary zero‐shot DL approaches. ⁶⁰ DIP is the pioneering zero‐shot DL approach, but it requires a heuristic early stopping criterion. Recent work on zero‐shot learning has built on the multi‐mask concept of the proposed approach to develop a rigorous stopping criterion. ⁶⁰ While zero‐shot learning enables training from a single slice, the reconstruction times are much longer compared with database learning approaches.

5. CONCLUSION

The proposed multi‐mask SSDU approach enables training of physics‐guided neural networks without fully sampled data, while significantly outperforming single‐mask SSDU at high acceleration rates through the efficient use of the acquired undersampled data with multiple masks.

Supporting information

Figure S1. Undersampling masks used in the study. Note that due to the different size of the ACS data, 1D masks correspond to an effective acceleration rate of 5.2, while the 2D masks yield an effective acceleration rate of 7.7.

Figure S2. Reconstruction results from SSDU, and multi‐mask SSDU at R = 8 using 2D uniform undersampling mask with uniform random selection and variable‐density Gaussian selection for K = 5 and ρ = 0.4. Multi‐mask SSDU with Gaussian random selection fails to remove the artifacts apparent in SSDU, whereas multi‐mask SSDU with uniformly random selection significantly suppresses these artifacts. Difference images show that multi‐mask SSDU with uniformly random selection shows fewer residual artifacts compared to its multi‐mask Gaussian counterpart. The median and interquartile range of SSIM values across the validation dataset were 0.7974 [0.7723, 0.8293], 0.8009 [0.7789, 0.8313], 0.8260 [0.8002, 0.8516], and NMSE values were 0.0166 [0.0142, 0.0202], 0.0159 [0.0139, 0.0191], 0.0135 [0.0119, 0.0157] for SSDU, multi‐mask SSDU with Gaussian selection and uniformly random selection, respectively.

Figure S3. Reconstruction results from SSDU with uniform random selection of Λ for ρ ∈ 0.1, 0.2, 0.3, 0.4, 0.5, 0.6 using 2D uniform undersampling mask at R = 8. SSDU reconstructions suffers from residual artifacts for low ρ values of 0.1, 0.2 and 0.3. The best reconstruction quality is achieved at ρ = 0.4. Residual artifacts start to reappear after ρ = 0.5, becoming more pronounced as ρ increases. The quantitative assessment from hold‐out validation set align with these qualitative assessments. The median and interquartile range of SSIM values were 0.8166 [0.7875, 0.8408], 0.8208 [0.7928, 0.8451], 0.8230 [0.7967, 0.8486], 0.8236 [0.7964, 0.8494], 0.8229 [0.7960, 0.8499], 0.8192 [0.7937, 0.8473], and NMSE values were 0.0149 [0.0136, 0.0175], 0.0143 [0.0128, 0.0167], 0.0141 [0.0123, 0.0163], 0.0140 [0.0122, 0.0161], 0.0145 [0.0125, 0.0168], 0.0145 [0.0127, 0.0169] using uniformly random selection for ρ ∈ 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, respectively.

Figure S4. Reconstruction results using 2D a) random and b) Poisson undersampling masks at R = 8. CG‐SENSE suffers from noise and incoherent residual artifacts. Blind‐CS and DIP suffers from blurring artifacts. All other DL approaches achieve artifact‐free and improved reconstruction quality.

Figure S5. Reconstruction results at R = 8 using 1D a) random and b) uniform undersampling masks. CG‐SENSE suffers from noise and residual artifacts for both of these undersampling masks. Blind‐CS and DIP severely suffers from blurring artifacts. All other DL reconstructions achieve artifact‐free reconstruction with random undersampling. In uniform undersampling, SSDU suffers from residual artifacts shown with red arrows, whereas multi‐mask SSDU improves upon SSDU and achieve similar reconstruction quality with supervised DL‐MRI.

Figure S6. Reconstruction results at R = 12 using 2D a) random and b) Poisson undersampling masks. CG‐SENSE suffers from noise and incoherent residual artifacts. Blind‐CS and DIP suffers from visible blurring artifacts. All other DL approaches achieve artifact‐free and improved reconstruction quality.

Figure S7. The image reading results from the clinical reader study for the 3D FSE knee dataset using 2D uniform sheared R = 8 undersampling mask. CG‐SENSE was consistently rated lowest in terms of all evaluation criteria. CG‐SENSE was significantly worse than all other methods and ground truth in terms of SNR, aliasing artifacts and overall image quality. For blurring, it was only statistically different than the ground truth.

Figure S8. Reconstruction results from SSDU, multi‐mask SSDU and multi‐mask cyclic SSDU for K = 5 using 2D uniform undersampling masks at R = 8. In multi‐mask SSDU ρ = 0.4 for K = 5, whereas multi‐mask cyclic SSDU approach enforces ρ to be 0.2 for K = 5. Multi‐mask SSDU successfully removes artifacts in SSDU, whereas multi‐mask cyclic SSDU suffers from residual artifacts. Difference images further confirm these observations. In this setting, the median and interquartile range of SSIM values across the validation dataset were 0.7974 [0.7723, 0.8293], 0.8249[0.7968, 0.8497], 0.8260 [0.8002, 0.8516], and NMSE values were, 0.0166 [0.0142, 0.0202], 0.0137 [0.0121, 0.0161], 0.0135 [0.0119, 0.0157] for SSDU, multi‐mask cyclic SSDU and multi‐mask SSDU, respectively.

Yaman B, Gu H, Hosseini SAH, et al. Multi‐mask self‐supervised learning for physics‐guided neural networks in highly accelerated magnetic resonance imaging. NMR in Biomedicine. 2022;35(12):e4798. doi: 10.1002/nbm.4798