A review and experimental evaluation of deep learning methods for MRI reconstruction

Abstract

Following the success of deep learning in a wide range of applications, neural network-based machine-learning techniques have received significant interest for accelerating magnetic resonance imaging (MRI) acquisition and reconstruction strategies. A number of ideas inspired by deep learning techniques for computer vision and image processing have been successfully applied to nonlinear image reconstruction in the spirit of compressed sensing for accelerated MRI. Given the rapidly growing nature of the field, it is imperative to consolidate and summarize the large number of deep learning methods that have been reported in the literature, to obtain a better understanding of the field in general. This article provides an overview of the recent developments in neural-network based approaches that have been proposed specifically for improving parallel imaging. A general background and introduction to parallel MRI is also given from a classical view of k-space based reconstruction methods. Image domain based techniques that introduce improved regularizers are covered along with k-space based methods which focus on better interpolation strategies using neural networks. While the field is rapidly evolving with plenty of papers published each year, in this review, we attempt to cover broad categories of methods that have shown good performance on publicly available data sets. Limitations and open problems are also discussed and recent efforts for producing open data sets and benchmarks for the community are examined.

1. Introduction

Magnetic Resonance Imaging (MRI) is an indispensable clinical and research tool used to diagnose and study several diseases of the human body. It has become a sine qua non in various fields of radiology, medicine, and psychiatry. Unlike computed tomography (CT), it can provide detailed images of the soft tissue and does not require any radiation, thus making it less risky to the subjects. MRI scanners sample a patient’s anatomy in the frequency domain that we will call “k-space”. The number of rows/columns that are acquired in k-space is proportional to the quality (and spatial resolution) of the reconstructed MR image. To get higher spatial resolution, longer scan time is required due to the increased number of k-space points that need to be sampled (Fessler, 2010). Hence, the subject has to stay still in the MRI scanner for the duration of the scan to avoid signal drops and motion artifacts. Many researchers have been trying to reduce the number of k-space lines to save scanning time, which leads to the well-known problem of “aliasing” as a result of the violation of the Nyquist sampling criteria (Nyquist, 1928). Reconstructing high-resolution MR images from the undersampled or corrupted measurements was a primary focus of various sparsity promoting methods, wavelet-based methods, edge-preserving methods, and low-rank based methods. This paper reviews the literature on solving the inverse problem of MR image reconstruction from noisy measurements using Deep Learning (DL) methods, while providing a brief introduction to classical optimization based methods. We shall discuss more about this in Sec. 1.1.

A DL method learns a non-linear function $f:\mathcal{Y}\to \mathcal{X}$ from a set of all possible mapping functions $\mathcal{F}$. The accuracy of the mapping function can be measured using some notion of a loss function $l:\mathcal{Y}\times \mathcal{X}\to [0,\infty )$. The empirical risk (Vapnik, 1991), $\widehat{L}(f)$, can be estimated as $\widehat{L}(f)=\frac{1}{2}{\sum }_{i=1}^{m}l\left(f\left(y_i\right),x_i\right)$ and the generalization error of a mapping function f(·) can be measured using some notion of accuracy measurement. MR image reconstruction using deep learning, in its simplest form, amounts to learning a map f from the undersampled k-space measurement $\mathcal{Y}\in {ℂ}^{N_1\times N_2}$, or $\mathcal{Y}\in {ℝ}^{N_1\times N_2\times 2}$ to an unaliased MR image $\mathcal{X}\in {ℂ}^{N_1\times N_2}$, or $\mathcal{Y}\in {ℝ}^{N_1\times N_2\times 2}$, where N₁, N₂ are the height and width of the complex valued image. In several real-world cases, higher dimensions such as time, volume, etc., are obtained and accordingly the superscripts of $\mathcal{Y}$ and $\mathcal{X}$ change to ${ℂ}^{N_1\times N_2\times N_3\times N_4\times \cdots }$. For the sake of simplicity, we will use assume $\mathcal{Y}\in {ℂ}^{N_1\times N_2}$ and $\mathcal{X}\in {ℂ}^{N_1\times N_2}$.

In this survey, we focus on two broad aspects of DL methods, i.e. (i) generative models, which are data generation processes capturing the underlying density of data distribution; and (ii) non-generative models, that learn complex feature representations of images intending to learn the inverse mapping from k-space measurements to MR images. Given the availability and relatively broad access to open-source platforms like Github, PyTorch (Paszke et al., 2019), and TensorFlow (Abadi et al., 2015), as well as large curated datasets and high-performance GPUs, deep learning methods are actively being pursued for solving the MR image reconstruction problem with reduced number of samples while avoiding artifacts and boosting the signal-to-noise ratio (SNR).

In Sec. 1.1, we briefly discuss the mathematical formulation that utilizes k-space measurements from multiple receiver coils to reconstruct an MR image. Furthermore, we discuss some challenges of the current reconstruction pipeline and discuss the DL methods (in Sec. 1.2) that have been introduced to address these limitations. We finally discuss the open questions and challenges to deep learning methods for MR reconstruction in sections 2.1, 2.2, and 3.

1.1. Mathematical Formulation for Image Reconstruction in Multi-coil MRI

Before discussing undersampling and the associated aliasing problem, let us first discuss the simple case of reconstructing an MR image, $x\in {ℂ}^{N_1\times N_2}$, from a fully sampled k-space measurement, $y^{\mathit{\text{full}}}\in {ℂ}^{N_1\times N_2}$, using the Fourier transform $\mathcal{F}(\cdot )$:

$$y^{\mathit{\text{full}}}=\mathcal{F}x+\eta ,$$ (1)

where $\eta ~\mathcal{N}(0,\Sigma )$ is the associated measurement noise typically assumed to have a Gaussian distribution (Virtue and Lustig, 2017) when the k-space measurement is obtained from a single receiver coil.

Modern MR scanners support parallel acquisition using an array of overlapping receiver coils n modulated by their sensitivities S_i. So Eqn. 1 changes to: $y_i^{\mathit{\text{full}}}=\mathcal{F}{S}_{i}x+\eta$, where i = {1, 2, ⋯, n} is the number of receiver coils. We use y_i for the undersampled k-space measurement coming from the i^th receiver coil. To speed up the data acquisition process, multiple lines of k-space data (for cartesian sampling) are skipped using a binary sampling mask $\mathcal{M}\in {ℂ}^{N_1\times N_2}$ that selects a subset of k-space lines from y^full in the phase encoding direction:

$$y_i=\mathcal{M}\odot \mathcal{F}{S}_{i}x+\eta .$$ (2)

An example of y^full, y, $\mathcal{M}$ is shown in Fig 1.

Figure 1:

(Left to Right) An example of a fastMRI (Zbontar et al., 2018) knee image x, fully sampled k-space y^full, corresponding reconstructed image x^full from y^full, the sampling mask $\mathcal{M}$ of fastMRI that we apply on fully sampled k-space y^full, the sampled k-space y, and the corresponding aliased reconstructed image x^aliased.

To estimate the MR image x from the measurement, a data fidelity loss function is typically used to ensure that the estimated data is as close to the measurement as possible. A typical loss function is the squared loss, which is minimized to estimate x:

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\sum _i{\Vert y_i-\mathcal{M}\odot \mathcal{F}{S}_{i}x\Vert }_2^2=\text{arg}\underset{x}{\text{min}}\Vert y-Ax{\Vert }_2^2.$$ (3)

We borrow this particular formulation from (Sriram et al., 2020a; Zheng et al., 2019). This squared loss function is quite convenient if we wish to compute the error gradient during optimization.

However, the formulation in Eqn. 3 is under-determined if data is undersampled and does not have a unique solution. Consequently, a regularizer $\mathcal{R}(x)$ is typically added to solve such an ill-conditioned cost function: ¹

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\Vert y-Ax{\Vert }_2^2+\sum _i{\lambda }_i{\mathcal{R}}_i(x).$$ (4)

Please note that each ${\mathcal{R}}_i(x)$ is a separate regularizer, while the λ_is are hyperparameters that control the properties of the reconstructed image $\widehat{x}$ while avoiding over-fitting. Eqn. 3 along with the regularization term can be optimized using various methods, such as (i) the Morozov formulation, $\widehat{x}=\text{min}\{\mathcal{R}(x);\text{ such that},\Vert Ax-y\Vert \le \delta \}$; (ii) the Ivanov formulation, i.e. $\widehat{x}=\text{min}\{\Vert Ax-y\Vert ;\text{ such that},\mathcal{R}(x)\le ϵ\}$; or (iii) the Tikonov formulation, $\widehat{x}=\text{min}\{\Vert Ax-y\Vert +\lambda \mathcal{R}(x)\}$, discussed in (Oneto et al., 2016).

In general, the Tikonov formulation can be designed using a physics based, sparsity promoting, dictionary learning, or a deep learning based model. But there are several factors that can cause loss in data quality (especially small anatomical details) such as inaccurate modeling of the system noise, complexity, generalizability etc. To overcome these limitations, it is essential to develop inverse mapping methods that not only provide good data fidelity but also generalize well to unseen and unexpected data. In the next section, we shall describe how DL methods can be used as priors or regularizers for MR reconstruction.

1.2. Deep Learning Priors for MR Reconstruction

We begin our discussion by considering DL methods with learn-able parameters θ. The learn-able parameters θ can be trained using some notion of a learning rule that we shall discuss in Sec. 3. A DL training process helps us to find a function G_DL(·) that acts as a regularizer to Eqn. 4 with an overarching concept of an inverse mapping, i.e; (please note that, we shall follow (Zheng et al., 2019) to develop the optimization formulation

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\Vert y-Ax{\Vert }_2^2+\lambda {\mathcal{L}}_{\theta };\text{ where},{\mathcal{L}}_{\theta }=\text{arg}\underset{\theta }{\text{min}}\sum _j{\Vert x_j-G_{DL}(x\mid z,c,\theta )\Vert }_2^2$$ (5)

and z is a latent variable capturing the statistical regularity of the data samples, while c is a conditional random variable that depends on a number of factors such as: undersampling of the k-space (Shaul et al., 2020; Oksuz et al., 2019a; Shitrit and Raviv, 2017), the resolution of the image (Yang et al., 2017; Yuan et al., 2020), or the type of DL network used (Lee et al., 2019). Based on the nature of the learning, there are two types of DL methods known as generative models, and non-generative models. We shall start with a basic understanding of DL methods to a more in-depth study of different architectures in Secs. 4 and 5.

In generative modeling, the random variable z is typically sampled from a noisy Gaussian distribution, $z~\mathcal{N}(0,I)$ with or without the presence of the conditional random variable, i.e.;

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\Vert y-Ax{\Vert }_2^2+\lambda {\mathcal{L}}_{{\theta }_g};{\mathcal{L}}_{{\theta }_g}=\text{arg}\underset{{\theta }_g}{\text{min}}{\mathbb{E}}_{z~\mathcal{N}(0,I)}\frac{1}{2}{\Vert x_j-G_{GEN}\left(x\mid z,c,{\theta }_g\right)\Vert }_2^2$$ (6)

There are various ways to learn the parameters of Eqn. 6. For instance, the Generative Adversarial Network (GAN) (Goodfellow et al., 2014) allows us to learn the generator function G_GEN(·) using an interplay between two modules, while the Variational Autoencoders (VAEs) (Kingma and Welling, 2013) learns G_GEN(·) by optimizing the evidence lower bound (ELBO), or by incorporating a prior in a Bayesian Learning setup as described in Section 4.2. It is shown in the literature that a generative model can efficiently de-alias an MR image that had undergone a 4× or 8× undersampling in k-space (Zbontar et al., 2018).

The non-generative models on the other hand do not learn the underlying latent representation, but instead learn a mapping from the measurement space to the image space. Hence, the random variable z is not required. The cost function for a non-generative model is given by:

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\Vert y-Ax{\Vert }_2^2+\lambda {\mathcal{L}}_{{\theta }_g};{\mathcal{L}}_{{\theta }_g}=\text{arg}\underset{{\theta }_g}{\text{min}}{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\frac{1}{2}{\Vert x_j-G_{NGE}\left(x\mid c,{\theta }_g\right)\Vert }_2^2.$$ (7)

The function G_NGE(·) is a non-generative mapping function that could be a Convolutional Neural Network (CNN) (Zheng et al., 2019; Akçakaya et al., 2019; Sriram et al., 2020b), a Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), or any other similar deep learning model. The non-generative models show a significant improvement in image reconstruction quality compared to classical methods. We shall describe the generative and the non-generative modeling based approaches in detail in Secs. 4 and 5 respectively. Below, we give a brief overview of the classical or the non-DL based approaches for MR image reconstruction.

1.3. Classical Methods for MR Reconstruction

In the literature, several approaches can be found that perform an inverse mapping to reconstruct the MR image from k-space data. Starting from analytic methods (Fessler and Sutton, 2003; Laurette et al., 1996), there are several works that provide ways to do MR reconstruction, such as the physics based image reconstruction methods (Roeloffs et al., 2016;Tran-Gia et al., 2016; Maier et al., 2019;Tran-Gia et al., 2013; Hilbert et al., 2018; Sumpf et al., 2011; Ben-Eliezer et al., 2016; Zimmermann et al., 2017; Schneider et al., 2020), the sparsity promoting compressed sensing methods (Feng and Bresler, 1996; Bresler and Feng, 1996; Candès et al., 2006), and low-rank based approaches (Haldar, 2013). All these methods can be roughly categorized into two categories, i.e. (i) GRAPPA-like methods: where prior assumptions are imposed on the k-space; and (ii) SENSE-like methods: where an image is reconstructed from the k-space while jointly unaliasing (or dealiasing) the image using sparsity promoting and/or edge preserving image regularization terms.

A few k-space methods estimate the missing measurement lines by learning kernels from an already measured set of k-space lines from the center of the k-space (i.e., the autocalibration or ACS lines). These k-space based methods include methods such as SMASH (Sodickson, 2000), VDAUTOSMASH (Heidemann et al., 2000), GRAPPA and its variations (Bouman and Sauer, 1993; Park et al., 2005; Seiberlich et al., 2008). The k-t GRAPPA (Huang et al., 2005) takes advantage of the correlations in the k-t space and interpolates the missing data. On the other hand, sparsity promoting low rank based methods are based on the assumption that, when the image reconstruction follows a set of constraints (such as sparsity, smoothness, parallel imaging, etc.), the resultant k-space should follow a structure that has low rank. The low rank assumption has been shown to be quite successful in dynamic MRI (Liang, 2007), functional MRI (Singh et al., 2015), and diffusion MRI (Hu et al., 2019). In this paper we give an overview of the low-rank matrix approaches (Haldar, 2013;Jin et al., 2016; Lee et al., 2016; Ongie and Jacob, 2016; Haldar and Zhuo, 2016; Haldar and Kim, 2017) in Sec. 2.1. While in k-t SLR (Lingala et al., 2011), a spatio-temporal total variation norm is used to recover the dynamic signal matrix.

The image space based reconstruction methods, such as, the model based image reconstruction algorithms, incorporate the underlying physics of the imaging system and leverage image priors such as neighborhood information (e.g. total-variation based sparsity, or edge preserving assumptions) during image reconstruction. Another class of works investigated the use of compressed sensing (CS) in MR reconstruction after its huge success in signal processing (Feng and Bresler, 1996; Bresler and Feng, 1996; Candès et al., 2006). Compressed sensing requires incoherent sampling and sparsity in the transform domain (Fourier, Wavelet, Ridgelet or any other basis) for nonlinear image reconstruction. We also describe dictionary learning based approaches that are a spacial case of compressed sensing using an overcomplete dictionary. The methods described in (Gleichman and Eldar, 2011; Ravishankar and Bresler, 2016; Lingala and Jacob, 2013;Rathi et al., 2011; Michailovich et al., 2011) show various ways to estimate the image and the dictionary from limited measurements.

1.4. Main Highlights of This Literature Survey

The main contributions of this paper are:

• We give a holistic overview of MR reconstruction methods, that includes a family of classical k-space based image reconstruction methods as well as the latest developments using deep learning methods.

• We provide a discussion of the basic DL tools such as activation functions, loss functions, and network architecture, and provide a systematic insight into generative modeling and non-generative modeling based MR image reconstruction methods and discuss the advantages and limitations of each method.

• We compare eleven methods that includes classical, non-DL and DL methods on fastMRI dataset and provide qualitative and quantitative results in Sec. 7.

• We conclude the paper with a discussion on the open issues for the adoption of deep learning methods for MR reconstruction and the potential directions for improving the current state-of-the-art methods.

2. Classical Methods for Parallel Imaging

This section reviews some of the classical k-space based MR image reconstruction methods and the classical image space based MR image reconstruction methods.

2.1. Inverse Mapping using k-space Interpolation

Classical k-space based reconstruction methods are largely based on the premise that the missing k-space lines can be interpolated (or extrapolated) based on a weighted combination of all acquired k-space measurement lines. For example, in the SMASH (Sodickson, 2000) method, the missing k-space lines are estimated using the spatial harmonics of order m. The k-space signal can then be written as:

$$y\left(k_1,k_2\right)=\sum _{i=1}^n{w}_i^m{y}_i\left(k_1,k_2\right)=\iint d{n}_{1}d{n}_2\sum _{i=1}^n{w}_i^m{S}_{i}x{e}^{-j{k}_1{n}_1-j\left(k_2+m\Delta k_2\right)n_2}$$ (8)

where, $w_i^m$ are the spatial harmonics of order m, $\Delta k_2=\frac{2\pi }{FOV}$ is the minimum k-space interval (FOV stands for field-of-view), and y(k₁, k₂) is the k-space measurement of an image x, (k₁, k₂) are the co-ordinates in k-space along the phase encoding (PE) and frequency encoding (FE) directions, and j represents the imaginary number. From Eqn. 8, one can note that the m^th-line of k-space can be generated using m-number of spatial harmonics and hence we can estimate convolution kernels to approximate the missing k-space lines from the acquired k-space lines. So, in SMASH a k-space measurement x, also known as a composite signal in common parlance, is basically a linear combination of n number of component signals (k-space measurements) coming from n receiver coils modulated by their sensitivities S_i, i.e.

$$y=\sum _{i=1}^n{w}_i\mathcal{M}\odot \mathcal{F}{\mathcal{S}}_{i}x+\eta .$$ (9)

We borrow the mathematical notation from (Sodickson, 2000) and represent the composite signal in Eqn. 9 as follows:

$$y\left(k_1,k_2\right)=\sum _{i=1}^n{w}_i{y}_i\left(k_1,k_2\right)=\iint d{n}_{1}d{n}_2\sum _{i=1}^n{w}_i\mathcal{M}{\mathcal{S}}_{i}x{e}^{-j{k}_1{n}_1-j{k}_2{n}_2}.$$ (10)

However, SMASH requires the exact estimation of the sensitivity of the receiver coils to accurately solve the reconstruction problem.

To address this limitation, AUTO-SMASH (Jakob et al., 1998) assumed the existence of a fully sampled block of k-space lines called autocalibration lines (ACS) at the center of the k-space (the low frequency region) and relaxed the requirement of the exact estimation of receiver coil sensitivities. The AUTO-SMASH formulation can be written as:

$$\begin{gathered} y\left(k_1,k_2+m\Delta k_2\right)= \\ \sum _{i=1}^n{w}_i^0{y}_i^{ACS}\left(k_1,k_2+m\Delta k_2\right)=\iint d{n}_{1}d{n}_2\sum _{i=1}^n{w}_i^0{S}_{i}x{e}^{-j{k}_1{n}_1-j\left(k_2+m\Delta k_2\right)n_2} \end{gathered}$$ (11)

We note that the AUTO-SMASH paper theoretically proved that it can learn a kernel that is, ${\sum }_{i=1}^n{w}_i^m{y}_i\left(k_1,k_2\right)={\sum }_{i=1}^n{w}_i^m{y}_i^{ACS}\left(k_1,k_2+m\Delta k_2\right)$, that is, it can learn a linear shift invariant convolutional kernel to interpolate missing k-space lines from the knowledge of the fully sampled k-space lines of ACS region. The variable density AUTO-SMASH (VD-AUTO-SMASH) (Heidemann et al., 2000) further improved the reconstruction process by acquiring multiple ACS lines in the center of the k-space. The composite signal x is estimated by adding each individual component y_i by deriving linear weights $w_i^m$ and thereby estimating the missing k-space lines. The more popular, generalized auto calibrating partially parallel acquisitions (GRAPPA) (Bouman and Sauer, 1993) method uses this flavour of VD-AUTO-SMASH, i.e. the shift-invariant linear interpolation relationships in k-space, to learn the coefficients of a convolutional kernel from the ACS lines. The missing k-space are estimated as a linear combination of observed k-space points coming from all receiver coils. The weights of the convolution kernel are estimated as follows: a portion of the k-space lines in the ACS region are artificially masked to get a simulated set of acquired k-space points $y^{AC{S}_1}$ and missing k-space points $y^{AC{S}_2}$. Using the acquired k-space lines $y^{AC{S}_1}$, we can estimate the weights of the GRAPPA convolution kernel K by minimizing the following cost function:

$$\widehat{K}=\text{arg}\underset{K}{\text{min}}{\Vert y^{AC{S}_2}-K⊛y^{AC{S}_1}\Vert }_2^2$$ (12)

where ⊛ represents the convolution operation. The GRAPPA method has shown very good results for uniform undersampling, and is the method used in product sequences by Siemens and GE scanners. There are also recent methods (Xu et al., 2018; Chang et al., 2012) that show ways to learn non-linear weights of a GRAPPA kernel.

The GRAPPA method regresses k-space lines from a learned kernel without assuming any specific image reconstruction constraint such as sparsity, limited support, or smooth phase as discussed in (Kim et al., 2019). On the other hand, low-rank based methods assume an association between the reconstructed image and the k-space structure, thus implying that the convolution-structured Hankel or Toeplitz matrices leveraged from the k-space measurements must show a distinct null-space vector association with the kernel. As a result, any low-rank recovery algorithm can be used for image reconstruction. The simultaneous autocalibrating and k-space estimation (SAKE) (Shin et al., 2014) algorithm used the block Hankel form of the local neighborhood in k-space across all coils for image reconstruction. Instead of using correlations across multiple coils, the low-rank matrix modeling of local k-space neighborhoods (LORAKS) (Haldar, 2013) utilized the image phase constraint and finite image support (in image space) to produce very good image reconstruction quality. The LORAKS method does not require any explicit calibration of k-space samples and can work well even if some of the constraints such as sparsity, limited support, and smooth phase are not strictly satisfied. The AC-LORAKS (Haldar, 2015) improved the performance of LORAKS by assuming access to the ACS measurements, i.e.:

$$y=\text{arg}\underset{y}{\text{min}}\Vert y-\mathcal{P}(\mathcal{M}Ax)N{\Vert }_2^2$$ (13)

where $\mathcal{P}(\cdot )$ is a mapping function that transforms the k-space measurement to a structured low-rank matrix, and the matrix N is the null space matrix. The mapping $\mathcal{P}(\cdot )$ basically takes care of the constraints such as sparsity, limited support, and smooth phase. In the PRUNO (Zhang et al., 2011) method, the mapping $\mathcal{P}(\cdot )$ only imposes limited support and parallel imaging constraints. On the other hand, the number of nullspace vectors in N is set to 1 in the SPIRiT method (Lustig and Pauly, 2010). The ALOHA method (Lee et al., 2016) uses the weighted k-space along with transform-domain sparsity of the image. Different from them, the method of (Otazo et al., 2015) uses a spatio-temporal regularization.

2.2. Image Space Rectification based Methods

These methods directly estimate the image from k-space by imposing prior knowledge about the properties of the image (e.g., spatial smoothness). Leveraging image prior through linear interpolation works well in practice but largely suffers from sub-optimal solutions and as a result the practical cone beam algorithm (Laurette et al., 1996) was introduced that improves image quality in such a scenario. The sensitivity encoding (SENSE) method (Pruessmann et al., 1999) is an image unfolding method that unfolds the periodic repetitions from the knowledge of the coil. In SENSE, the signal in a pixel location (i, j) is a weighted sum of coil sensitivities, i.e.;

$$I_k(i,j)=\sum _{k=1}^{N_c}\sum _{j=1}^{N_2}{S}_{kj}x(i,j),$$ (14)

where N₂ is the height of image $x\in R^{N_1\times N_2}$, N_c is the number of coils, and S_k is the coil sensitivity of the k^th coil. The I_k is the k^th coil image that has aliased pixels at a certain position, and i is a particular row and j = {1, ⋯, N₂} is the column index counting from the top of the image to the bottom. The S is the sensitivity matrix that assembles the corresponding sensitivity values of the coils at the locations of the involved pixels in the full FOV image x. The coil images I_k, the sensitivity matrix S, and the image x in Eqn. 14 can be re-written as;

$$I=Sx.$$ (15)

By knowing the complex sensitivities at the corresponding positions, we can compute the generalized inverse of the sensitivity matrix:

$$x={\left({\widehat{S}}^H\widehat{S}\right)}^{-1}{\widehat{S}}^H\cdot I.$$ (16)

Please note that, I represents the complex coil image values at the chosen pixel and has length N_c. In k-t SENSE and k-t BLAST (Tsao et al., 2003) the information about the spatio-temporal support is obtained from the training dataset that helps to reduce aliasing.

The physics based methods allow statistical modeling instead of simple geometric modeling present in classical methods and reconstruct the MR images using the underlying physics of the imaging system (Roeloffs et al., 2016;Tran-Gia et al., 2016; Maier et al., 2019;Tran-Gia et al., 2013; Hilbert et al., 2018; Sumpf et al., 2011; Ben-Eliezer et al., 2016; Zimmermann et al., 2017; Schneider et al., 2020). These types of methods sometimes use very simplistic anatomical knowledge based priors (Chen et al., 1991; Gindi et al., 1993; Cao and Levin, 1997) or “pixel neighborhood” (Szeliski, 2010) information via a Markov Random Field based regularization (Sacco, 1990; Besag, 1986).

A potential function based regularization takes the form $\mathcal{R}(x)={\sum }_{i=2}^n\psi \left(x_i-x_{i-1}\right)$, where the function, ψ(·), could be a hyperbolic, Gaussian (Bouman and Sauer, 1993) or any edge-preserving function (Thibault et al., 2007). The Total Variation (TV) could also be thought of as one such potential function. The (Rasch et al., 2018) shows a variational approach for the reconstruction of subsampled dynamic MR data, which combines smooth, temporal regularization with spatial total variation regularization.

Different from Total Variation (TV) approaches, (Bostan et al., 2012) proposed a stochastic modeling approach that is based on the solution of a stochastic differential equation (SDE) driven by non-Gaussian noise. Such stochastic modeling approaches promote the use of nonquadratic regularization functionals by tying them to some generative, continuous-domain signal model.

The Compressed Sensing (CS) based methods impose sparsity in the image domain by modifying Eqn. 2 to the following:

$$\widehat{x}=\underset{x}{\text{min}}\frac{1}{2}\Vert y-Ax{\Vert }_2^2+\lambda \Vert \Gamma x{\Vert }_1,$$ (17)

where Γ is an operator that makes x sparse. The l₁ norm is used to promote sparsity in the transform or image domain. The l₁ norm minimization can be pursued using a basis pursuit or greedy algorithm (Boyd et al., 2004). However, use of non-convex quasinorms (Chartrand, 2007; Zhao and Hu, 2008; Chartrand and Staneva, 2008; Saab et al., 2008) show an increase in robustness to noise and image non-sparsity. The structured sparsity theory (Boyer et al., 2019) shows that only $\mathcal{O}(M+M\text{log}N)$ measurements are sufficient to reconstruct MR images when M-sparse data with size N are given. The kt-SPARSE approach of (Lustig et al., 2006) uses a spatio-temporal regularization for high SNR reconstruction.

Iterative sparsity based methods (Ravishankar and Bresler, 2012; Liu et al., 2015, 2016) assume that the image can be expressed as a linear combination of the columns (atoms) from a dictionary Γ such that x = Γ^T h and h is the coefficient vector. Hence Eqn. 4 becomes:

$$\widehat{x}=\underset{x}{\text{min}}\frac{1}{2}\Vert y-Ax{\Vert }_2^2+\lambda \mathcal{R}(x);\mathcal{R}(x)=\underset{h}{\text{min}}\frac{1}{2}{\Vert x-{\Gamma }^{T}h\Vert }_2^2+\alpha \Vert h{\Vert }_1.$$ (18)

The SOUP-DIL method (Bruckstein et al., 2009) uses an exact block coordinate descent scheme for optimization while a few methods (Chen et al., 2008; Lauzier et al., 2012; Chen et al., 2008) assume to have a prior image x₀, i.e. $\mathcal{R}\left(x-x_0\right)$, to optimize Eqn. 4. The method in (Caballero et al., 2014) optimally sparsifies the spatio-temporal data by training an overcomplete basis of atoms.

The method in (Hongyi Gu, 2021) shows a DL based approach to leverage wavelets for reconstruction.

The transform based methods are a generalization of the CS approach that assumes a sparse approximation of the image along with a regularization of the transform itself, i.e., Γx = h + ϵ, where h is the sparse representation of x and ϵ is the modeling error. The method in (Ravishankar and Bresler, 2015) proposed a regularization as follows:

$$\mathcal{R}(x)=\underset{\Gamma ,h}{\text{min}}\frac{1}{2}{\Vert x-{\Gamma }^{T}h\Vert }_2^2+\alpha Q(\Gamma ),$$ (19)

where Q(Γ) = −log |detΓ| + 0.5∥Γ∥² is the transform regularizer. In this context, the STROLLR method (Wen et al., 2018) used a global and a local regularizer.

In general, Eqn. 5 is a non-convex function and cannot be optimized directly with gradient descent update rules. The unrolled optimization algorithm procedure decouples the data consistency term and the regularization term by leveraging variable splitting in Eqn 4 as follows:

$$\underset{x,z}{\text{min}}\Vert Ax-y{\Vert }_2^2+\mu \Vert x-h{\Vert }_2^2+\mathcal{R}(h),$$ (20)

where the regularization is decoupled using a quadratic penalty on x and an auxiliary random variable z. Eqn 20 is optimized via alternate minimization of

$$h_i=\underset{h}{\text{min}}\lambda {\Vert x_{i-1}-h\Vert }_2^2+\mathcal{R}(h)$$ (21)

and the data consistency term:

$$x_i=\underset{x}{\text{min}}{\Vert A{x}_{i-1}-y\Vert }_2^2+\lambda {\Vert h_i-x\Vert }_2^2$$ (22)

where the h_i, h_i−1 are the intermediate variables at iteration i. The alternating direction method of multiplier networks (ADMM net) introduce a set of intermediate variables, h₁, h₂, ⋯, h_n, and eventually we have a set of dictionaries, Γ₁, Γ₂, ⋯, Γ_n, such that, h_i = Γ_ix, collectively promote sparsity. The basic ADMM net update (Yang et al., 2018) is as follows:

$$\begin{gathered} \text{arg}\underset{x}{\text{min}}\frac{1}{2}\Vert Ax-y{\Vert }_2^2+\sum _i{\Vert {\Gamma }_{i}x+{\beta }_i-z_i\Vert }_2^2;\text{arg}\underset{z_i}{\text{min}}\sum _i\left[{\lambda }_{i}g\left(h_i\right){\Vert {\Gamma }_{i}x+{\beta }_i-h_i\Vert }_2^2\right] \\ {\beta }_i\leftarrow {\beta }_{i-1}+{\alpha }_i\left({\Gamma }_{i}x-h_i\right) \end{gathered}$$ (23)

where g(·) can be any sparsity promoting operator and β is called a multiplier. The iterative shrinkage thresholding algorithm (ISTA) solves this CS optimization problem as follows:

$$h_{i+1}=x_i-{\Phi }^H\left(\Phi x_i-y\right);x_{i+1}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}{\Vert x-h_{i+1}\Vert }_2^2+\lambda \Vert \Gamma x{\Vert }_1.$$ (24)

Later in this paper, we shall show how ISTA and ADMM can be organically used within the modern DL techniques in Sec. 5.3.

3. Review of Deep Learning Building Blocks

In this section, we will describe basic building blocks that are individually or collectively used to develop complex DL methods that work in practice. Any DL method, by design, has three major components: the network structure, the training process, and the dataset on which the DL method is trained and tested. We shall discuss each one of them below in detail.

3.1. Various Deep Learning Frameworks

Perceptron:

The journey of DL started in the year 1943 when Pitts and McCulloch (McCulloch and Pitts, 1943) gave the mathematical model of a biological neuron. This mathematical model is based on the “all or none” behavioral dogma of a biological neuron. Soon after, Rosenblatt provided the perceptron learning algorithm (Rosenblatt, 1957) which is a mathematical model based on the behaviour of a neuron. The perceptron resembles the structure of a neuron with dendrites, axons and a cell body. The basic perceptron is a binary classification algorithm of the following form:

$$f(x)=\{\begin{array}{ll}1\hfill & \text{if }w\cdot x+b>0,\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}$$ (25)

where x_i’s are the components of an image vector x, w_i’s are the corresponding weights that determine the slope of the classification line, and b is the bias term. This setup collectively resembles the “all or none” working principle of a neuron. However, in the famous book of Minsky and Papert (Minsky and Papert, 2017) called “Perpetron” it was shown that the perceptron can’t classify non-separable points such as an exclusive-OR (XOR) function.

Multilayer Perceptron (MLP):

It was understood that the non-separability problem of perceptron can be overcome by a multilayer perceptron (Minsky and Papert, 2017) but the research stalled due to the unavailability of a proper training rule. In the year 1986, Rumelhart et al. (Rumelhart et al., 1986) proposed the famous “backpropagation algorithm” that provided fresh air to the study of neural network. A Multilayer Perceptron (MLP) uses several layers of multiple perceptrons to perform nonlinear classification. A MLP is comprised of an input layer, an output layer, and several densely connected in-between layers called hidden layers:

$$\begin{gathered} h^1={\psi }^1\left(\sum W^{1}x+b^1\right);h^i={\psi }^i\left(\sum W^i{h}^{i-1}+b^i\right),\text{ where},i\in \{2,\cdots ,n-1\} \\ y={\psi }^n\left(\sum W^n{h}^{n-1}+b^{n-1}\right). \end{gathered}$$ (26)

Along with the hidden layers and the input-output layer, the MLP learns features of the input dataset and uses them to perform classification. The dense connection among hidden layers, input and output layers often creates a major computational bottleneck when the input dimension is very high.

Neocognitron or the Convolutional Neural Network (CNN):

The dense connection (also known as global connection) of a MLP was too flexible a model and prone to overfitting and sometimes had large computational overhead. To cope with this situation, a local sliding window based network with shared weights was proposed in early 1980s called neocognitron network (Fukushima and Miyake, 1982) and later popularized as Convolutional Neural Network (CNN) in the year 1998 (LeCun et al., 1989). Similar to the formulation of (Liang et al., 2020a), we write the feedforward process of a CNN as follows:

$$\begin{gathered} C_0=x \\ {C}_i={\psi }_{i-1}\left(K_{i-1}\star C_{i-1}\right);i\in \{1,\cdots ,n-1\},C_n={\psi }_{n-1}\left(K_{n-1}\star C_{n-1}\right), \end{gathered}$$ (27)

where C_i ∈ R^h×w×d is the i^th hidden layer comprised of d-number of feature maps each of size h × w, K_i is the i^th kernel that performs a convolution operation on C_i, and ψ_i(·) are activation functions to promote non-linearity. We show a vanilla kernel operation in Eqn. 27. Please note that, the layers of the CNN can either be a fully connected dense layer, a max-pooling layer that downsizes the input, or a dropout layer to perform regularization that is not shown in Eqn. 27.

Recurrent Neural Networks (RNNs):

A CNN can learn hidden features of a dataset using its inherent deep structure and local connectivities through a convolution kernel. But they are not capable of learning the time dependence in signals. The recurrent neural network (RNN) (Rumelhart et al., 1986) at its basic form is a time series neural network with the following form:

$$\begin{gathered} h_0=x \\ {h}_t={\psi }_t\left(W_{hh}{h}_{t-1}+W_{xh}{x}_t\right),t\in \{1,\cdots ,n-1\};h_n={\psi }_n\left(W_{hy}{h}_{n-1}\right), \end{gathered}$$ (28)

where t is the time and the RNN takes the input x in a sequential manner. However, the RNN suffers from the problem of “vanishing gradient”. The vanishing gradient is observed when gradients from output layer of a RNN trained with gradient based optimization method changes parameter values by a very small amount, thus effecting no change in parameter learning. The Long Short Term Memory (LSTM) network (Hochreiter and Schmidhuber, 1997) uses memory gates, and sigmoid and/or tanh activation function and later ReLU activation function (see Sec. 3.2 for activation functions) to control the gradient signals and overcome the vanishing gradient problem.

Transformer Networks:

Although the LSTM has seen tremendous success in DL and MR reconstruction, there are a few problems associated with the LSTM model (Vaswani et al., 2017) such as: (i) the LSTM networks perform a sequential processing of input; and (ii) the short attention span of the hidden states that may not learn a good contextual representation of input. Such shortcomings are largely mitigated using a recent advancement called the Transformer Network (Vaswani et al., 2017). A transformer network has a self-attention mechanism², a positional embedding and a non-sequential input processing setup and empirically this configuration outperforms the LSTM networks by a large margin (Vaswani et al., 2017).

3.2. Activation Functions

The activation function, ψ(·), operates on a node or a layer of a neural network and provides a boolean output, probabilistic output or output within a range. The step activation function was proposed by McCulloch & Pitts in (McCulloch and Pitts, 1943) with the following form: ψ(x) = 1, if, x ≥ 0.5 and 0 otherwise. Several initial works also used the hyperbolic tangent function, ψ(x) = tanh(x), as an activation function that provides value within the range [−1, +1]. The sigmoid activation function, $\psi (x)=\frac{1}{1+e^{-x}}$, is a very common choice and provides only positive values within the range [0, 1]. However, one major disadvantage of the sigmoid activation function is that its derivative, ψ′ (x) = ψ(x)(1 − ψ(x)), quickly saturates to zero which leads to the vanishing gradient problem. This problem was addressed by adding a Rectified Linear Unit (ReLu) to the network (Brownlee, 2019), ψ(x) = max(0, x) with the derivative ψ′ (x) = 1, if, x > 0 or 0 elsewhere.

3.3. Network Structures

The VGG Network:

In late 2015, Zisserman et al. published their seminal paper with the title “very deep convolutional networks for large-scale image recognition” (VGG) (Simonyan and Zisserman, 2014) that presents a 16-layered network called VGG network. Each layer of the VGG network has an increasing number of channels. The network was shown to achieve state-of-the-art level performance in Computer Vision tasks such as classification, recognition, etc.

The ResNet Model:

A recently developed model called Residual Networks or ResNet (He et al., 2016), modifies the layer interaction shown in Eqn. 27 to the following form, C_i = ψ_i−1(K_i−1 ⋆ C_i−1) + C_i−2 where, i ∈ {2, ⋯, n − 1}, and provides a “shortcut connection” to the hidden layers. The identity mapping using the shortcut connections has a large positive impact on the performance and stability of the networks.

UNet:

A UNet architecture (Ronneberger et al., 2015) was proposed to perform image segmentation task in biomedical images. The total end-to-end architecture pictorially resembles the english letter “U” and has a encoder module and a decoder module. Each encoder layer is comprised of unpadded convolution, a rectified linear unit (ReLU in Sec. 3.2), and a pooling layer, which collectively downsample the image to some latent space. The decoder has the same number of layers as the encoder. Each decoder layer upsamples the data from its previous layer until the input dimension is reached. This architecture has been shown to provide good quantitative results on several datasets.

Autoencoders:

The autoencoders (AE) are a type of machine learning models that capture the patterns or regularities of input data samples in an unsupervised fashion by mapping the target values to be equal to the input values (i.e. identity mapping). For example, given a data point x randomly sampled from the training data distribution p_data(x), a standard AE learns a low-dimensional representation z using an encoder network, z ~ D(z|x, θ_d), that is parameterized by θ_d. The low-dimensional representation z, also called the latent representation, is subsequently projected back to the input dimension using a decoder network, $\tilde{x}~G_{GEN}\left(x\mid z,{\theta }_g\right)$, that is parameterized by θ_g. The model parameters, i.e. (θ_d, θ_g), are trained using the standard back propagation algorithm with the following optimization objective:

$${\mathcal{L}}_{{\theta }_d,{\theta }_g}=\text{arg}\underset{{\theta }_g,{\theta }_d}{\text{min}}{\mathbb{E}}_{z~D\left(x,{\theta }_d\right)}\frac{1}{2}{\Vert x-G_{GEN}\left(x\mid z,{\theta }_g\right)\Vert }_2^2.$$ (29)

From a Bayesian learning perspective, an AE learns a posterior density, p(z|x), using the encoder network, G_GEN(z|x, θ_g), and a decoder network, D(x|z, θ_d). The vanilla autoencoder, in a way, can be thought of as a non-linear principal component analysis (PCA) (Hinton and Salakhutdinov, 2006) that progressively reduces the input dimension using the encoder network and finds regularities or patterns in the data samples.

Variational Autoencoder Networks:

Variational Autoecoder (VAE) is basically an autoencoder network that is comprised of an encoder network G_GEN(z|x, θ_g) that estimates the posterior distribution p(z|x) and the inference p(x|z) with a decoder network D(z, θ_d). However, the posterior p(z|x) is intractable³ and several methods have been proposed to approximate the inference using techniques such as the Metropolis-Hasting (Metropolis et al., 1953) and variational inference (VI) algorithms (Kingma and Welling, 2013). The main essence of VI algorithm is to estimate an intractable probability density, i.e. p(z|x) in our case, from a class of tractable probability densities, $\mathcal{Z}$, with an objective to finding a density, q(z), almost similar to p(z|x). We then sample from the tractable density q(z) instead of p(z|x) to get an approximate estimation, i.e.

$$q^{*}(z)=\text{arg}\underset{q(z)\in \mathcal{Q}}{\text{min}}{D}_{KL}(q(z)\Vert p(z\mid x)).$$ (30)

Here, D_KL is the KL divergence (Kullback and Leibler, 1951). The VI algorithm, however, typically never converges to the global optimal solution but provides a very fast approximation. The VAE consists of three components: (i) the encoder network $E_{{\theta }_e}(z\mid x)$ that observes and encodes a data point x from the training dataset $\mathcal{D}(x)$ and provides the mean and variance of the approximate posterior, i.e. $\mathcal{N}\left({\mu }_{\theta }\left(x_i\right),{\sigma }_{\theta }\left(x_i\right)\right)$ from a batch of n data points ${x_i|}_{i=1}^n$; (ii) a prior distribution G(z), typically an isotropic Gaussian $\mathcal{N}(0,I)$ from which z is sampled, and (iii) a generator network $G_{{\theta }_g}(x\mid z)$ that generates data points given a sample from the latent space z. The VAE, however, cannot directly optimize the VI, i.e. $q^{*}(z)=\text{arg}{\text{min}}_{q(z)\in \mathcal{Q}}{D}_{KL}(q(z)\Vert p(z\mid x))$, as the KL divergence requires an estimate of the intractable density p(x), i.e. $D_{KL}(q(z)\Vert p(z\mid x))=\mathbb{E}[\text{log}q(z)]-\mathbb{E}[\text{log}p(z,x)]+\mathbb{E}[\text{log}p(x)]$. As a result, VAE estimates the evidence lower bound (ELBO) that is similar to KL divergence, i.e.:

$$\begin{gathered} \mathit{\text{ELBO}}(q)=\mathbb{E}[\text{log}p(z,x)]-\mathbb{E}[\text{log}q(z)]=\mathbb{E}[\text{log}p(z)]+\mathbb{E}[\text{log}p(x\mid z)]-\mathbb{E}[\text{log}q(z)] \\ =\mathbb{E}[\text{log}p(x\mid z)]-D_{KL}(p(z\mid x)\Vert q(z))={\mathbb{E}}_{z~p(z\mid x)}[\text{log}p(x\mid z)]-D_{KL}(p(z\mid x)\Vert q(z)) \end{gathered}$$ (31)

Since ELBO(q) ≤ log p(x), optimizing Eqn. (31) provides a good approximation of the marginal density p(x). Please note that, Eqn. 31 is similar to Eqn. 4 where the first term in Eqn. 31 is the data consistency term and the KL divergenece term acts as a regularizer.

Generative Adversarial Networks:

A vanilla Generative Adversarial Networks (GAN) setup, by design, is an interplay between two neural networks called the generator, that is G_GEN(x|z, θ_g), and the discriminator $D_{{\theta }_d}(\cdot )$ parameterized by θ_g and θ_d respectively. The G_GEN(x|z, θ_g) samples the latent vector $z\in {ℝ}^{n\times 1\times 1}$ and generates x^gen. While the discriminator, on the other hand, takes x (or x^gen) as input and provides a {real, fake} decision on x being sampled from a real data distribution or from G_GEN(x|z, θ_g). The parameters are trained using a game-theoretic adversarial objective, i.e.:

$${\mathcal{L}}_{{\theta }_d,{\theta }_g}=-{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\left[\text{log}D\left(x\mid {\theta }_d\right)\right]-{\mathbb{E}}_{z~\mathcal{N}(0,I)}\left[\text{log}\left(1-D\left(G_{GEN}\left(x\mid z,{\theta }_g\right),{\theta }_d\right)\right)\right].$$ (32)

As the training progresses, the generator G_GEN(z|θ_g) progressively learns a strategy to generate realistic looking images, while the discriminator D(x, θ_d) learns to discriminate the generated and real samples.

3.4. Loss Functions

In Sec. 1.1, we mentioned the loss function, L, that estimates the empirical loss and the generalization error. Loss functions that are typically used in MR image reconstruction using DL methods are the Mean Squared Error (MSE), Peak Signal to Noise Ratio (PSNR) and the Structural Similarity Loss function (SSIM), or an l₁ loss to optimize Eqns. 3, 4, 5. The MSE loss between an image x and its noisy approximation $\widehat{x}$ is defined as, $MSE(x,\widehat{x})=\frac{1}{n}{\sum }_{i=1}^n{\left({\widehat{x}}_i-x_i\right)}^2$, where n is the number of samples. The root MSE (RMSE) is essentially the squared root of the MSE, $\sqrt{\text{MSE}(x,\widehat{x})}$. The l₁ loss, $l_1(x,\widehat{x})=\frac{1}{n}{\sum }_{i=1}^n\left|{\widehat{x}}_i-x_i\right|$, provides the absolute difference and is typically used as a regularization term to promote sparsity. The PSNR is defined using the MSE, $\text{PSNR}(x,\widehat{x})=20{\text{log}}_{10}\left(MA{X}_x-10{\text{log}}_{10}(\text{MSE}(x,\widehat{x})\right)$, where MAX_x is the highest pixel value attained by the image x. The PSNR metric captures how strongly the noise in the data affects the fidelity of the approximation with respect to the maximum possible strength of the signal (hence the name peak signal to noise ratio). The main concern with $\text{MSE}(x,\widehat{x})$ and $\text{PSNR}(x,\widehat{x})$ is that they penalize large deviations much more than smaller ones (Zhao et al., 2016) (e.g., outliers are penalized more than smaller anatomical details). The SSIM loss for a pixel i in the image x and the approximation $\widehat{x}$ captures the perceptual similarity of two images: $\text{SSIM}\left(x_i,{\widehat{x}}_i\right)=\frac{2{\mu }_x{\mu }_{\widehat{x}}+c_1}{{\mu }_x^2+{\mu }_{\widehat{x}}^2+c_1}\frac{2{\sigma }_{x\widehat{x}}+c_2}{{\sigma }_x^2+\sigma {\widehat{x}}^2+c_2}$, here c₁, c₂ are two constants, and μ is the mean and σ is the standard deviation.

The VGG loss:

It is shown in (Johnson et al., 2016) that the deeper layer feature maps (feature maps are discussed in Eqn. 27) of a VGG-16 network, i.e. a VGG network that has 16 layers, can be used to compare perceptual similarity of images. Let us assume that, the L^th layer of a VGG network has distinct N_L feature maps each of size M_L × M_L. The matrix $F^L\in {ℝ}^{N_L\times M_L}$, stores the activations $F_{i,j}^L$ of the i^th filter at position j of layer L. Then, the method computes feature correlation using: $C_{i,j}^L={\sum }_k{F}_{i,k}^L{F}_{j,k}^L$, where any $F_{n,o}^m$ conveys the activation of the n^th filter at position o in layer m. The correlation $C_{i,j}^L$ is considered as a VGG-loss function.

4. Inverse Mapping using Deep Generative Models

Based on how the generator network, G_GEN(x|z, θ_g), is optimized in Eqn. 6, we get different manifestations of deep generative networks such as Generative Adversarial Networks, Bayesian Networks, etc. In this section, we shall discuss specifics on how these networks are used in MR reconstruction.

4.1. Generative Adversarial Networks (GANs)

We gave a brief introduction to GAN in Sec. 3.3 and in this section we shall take a closer look on the different GAN methods used to learn the inverse mapping from k-space measurements to the MR image space.

Inverse Mapping from k-space:

The current GAN based k-space methods can be broadly classified into two categories: (i) methods that directly operate on the k-space y and reconstruct the image x by learning a non-linear mapping and (ii) methods that impute the missing k-space lines y^missing in the undersampled k-space measurements. In the following paragraphs, we first discuss the recent GAN based direct k-space to MR image generation methods followed by the undersampled k-space to full k-space generation methods.

The direct k-space to image space reconstruction methods, as shown in Fig. 3 (a), are based on the premise that the missing k-space lines can be estimated from the acquired k-space lines provided we have a good non-linear interpolation function, i.e.;

$${\mathcal{L}}_{{\theta }_d,{\theta }_g}=-{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\left[\text{log}D\left(x\mid {\theta }_d\right)\right]-{\mathbb{E}}_{z~\mathcal{N}(0,I)}\left[\text{log}\left(1-D\left(G_{GEN}\left(x\mid y,{\theta }_g\right),{\theta }_d\right)\right)\right].$$ (33)

This GAN framework was used in (Oksuz et al., 2018) for correcting motion artifacts in cardiac imaging using a generic AUTOMAP network⁴. Such AUTOMAP-like generator architectures not only improve the reconstruction quality but help in other downstream tasks such as MR image segmentation (Oksuz et al., 2019a, 2020, 2019b). However, while the “AUTOMAP as generator” based methods solve the broader problem of motion artifacts, but they largely fail to solve the banding artifacts along the phase encoding direction. To address this problem, a method called MRI Banding Removal via Adversarial Training (Defazio et al., 2020) leverages a perceptual loss along with the discriminator loss in Eqn. 32. The perceptual loss ensures data consistency, while, the discriminator loss checks whether: (i) the generated image has a horizontal (0) or a vertical (1) banding; and (ii) the generated image resembles the real image or not. With a 4x acceleration, a 12-layered UNet generator and a ResNet discriminator, the methodology has shown remarkable improvements (Defazio et al., 2020) on fastMRI dataset.

Figure 3: Deep Learning Models:

We visually demonstrate various DL models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Autoencoders, Residual Layer, Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) discussed in Sec. 3. The Convolutional Neural Network is comprised with many layers {C₀, C₁, ⋯, C_n} and the network gets input for layer C_i+1 from its previous layer C_i after imposing non-linearity, i.e. C_i+1 = ψ_i(K_i ⋆ C_i), using a non-linear function ψ_i. The Recurrent Neural Network is a time varying memory network. On the other hand, the Autoencoders perform non-linear dimensionality reduction by projecting the input x to a lower dimensional variable z using an encoder network and projects back z to the image space x using a decoder network. The ResNet uses the Residual Layer to solve problems like vanishing gradient, slow convergence, etc. The Generative Adversarial Network and the Variational Autoencoder are generative models that are discussed in Sec. 4.

Instead of leveraging the k-space regularization within the parameter space of a GAN (Oksuz et al., 2018, 2019a), the k-space data imputation using GAN directly operates on the k-space measurements to regularize Eqn. 32. To elaborate, these type of methods estimate the missing k-space lines by learning a non-linear interpolation function (similar to GRAPPA) within an adversarial learning framework, i.e.

$${\mathcal{L}}_{{\theta }_d,{\theta }_g}=-{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\left[\text{log}D\left(x\mid {\theta }_d\right)\right]-{\mathbb{E}}_{z~\mathcal{N}(0,I)}\left[\text{log}\left(1-D\left(\mathcal{F}{G}_{GEN}\left(y^{\mathit{\text{full}}}\mid y,{\theta }_g\right),{\theta }_d\right)\right)\right].$$ (34)

The accelerated magnetic resonance imaging (AMRI) by adversarial neural network method (Shitrit and Raviv, 2017) aims to generate the missing k-space lines, y^missing from y using a conditional GAN, y^missing ~ G(y^missing|z, c = y). The combined y, y^missing is Fourier transformed and passed to the discriminator. The AMRI method showed improved PSNR value with good reconstruction quality and no significant artifacts as shown in Fig. 4. Later, in subsampled brain MRI reconstruction by generative adversarial neural networks method (SUBGAN) (Shaul et al., 2020), the authors discussed the importance of temporal context and how that mitigates the noise associated with the target’s movement. The UNet-based generator in SUBGAN takes three adjacent subsampled k-space slices y_i−1, y_i, y_i+1 taken at timestamps t_i−1, t_i, t_i+1 and provides the reconstructed image. The method achieved a performance boost of ~ 2.5 in PSNR with respect to the other state-of-the-art GAN methods while considering 20% of the original k-space samples on IXI dataset (Rowland et al., 2004). We also show reconstruction quality of SUBGAN on fastMRI dataset in Fig. 5. Another method called multi-channel GAN (Zhang et al., 2018b) advocates the use raw of k-space measurements from all coils and has shown good k-space reconstruction and lower background noise compared to classical parallel imaging methods like GRAPPA and SPIRiT. However, we note that this method achieved ~ 2.8 dB lower PSNR than the GRAPPA and SPIRiT methods.

Figure 4: Reconstruction using k-space Imputation:

We show qualitative comparison of a few methods with the AMRI method (Shitrit and Raviv, 2017). The first row are the reconstructed images and the second row images are zoomed-in version of the red boxes shown in first row. The Zero-filled, compressed sensing MRI (CS-MRI), CNN network trained with l₂ loss (CNN-L2) have lesser detail as compared to the AMRI (proposed) method. The images are borrowed from AMRI (Shitrit and Raviv, 2017) paper.

Figure 5: Comparison of GAN Methods:

Qualitative comparison of (i) k-space interpolation based method, i.e. SUBGAN; (ii) Image space rectification method, i.e. RefineGAN; and (iii) the combined k-space and image space rectification method, i.e. SANTIS

Despite their success in MR reconstruction from feasible sampling patterns of k-space, the previous models we have discussed so far have the following limitations: (i) they need unaliased images for training, (ii) they need paired k-space and image space data, or (iii) the need fully sampled k-space data. In contrast, we note a recent work called unsupervised MR reconstruction with GANs (Cole et al., 2020) that only requires the undersampled k-space data coming from the receiver coils and optimizes a network for image reconstruction. Different from AutomapGAN (Oksuz et al., 2019a), in this setup the generator provides the undersampled k-space (instead of the MR image as in case of AutomapGAN) after applying Fourier transform, sensitivity encoding and a random-sampling mask ${\mathcal{M}}_1$ on the generated image, i.e. $y^{gen}={\mathcal{M}}_1\mathcal{F}\mathcal{S}\left(G\left(x\mid y,{\theta }_g\right)\right)$. The discriminator takes the k-space measurements instead of an MR image and provides thee learned signal to the generator.

Image space Rectification Methods:

Image space rectification methods operate on the image space and learn to reduce noise and/or aliasing artifacts by updating Eqn. 4 to the following form:

$$\begin{gathered} \widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}{\Vert x-G_{\mathit{\text{GEN}}}\left(x\mid x^{\mathit{\text{low}}},{\theta }_g\right)\Vert }_2^2+\lambda {\Vert y^t-\mathcal{F}{G}_{\mathit{\text{GEN}}}\left(x\mid x^{\mathit{\text{low}}},{\theta }_g\right)\Vert }_2^2+\zeta {\mathcal{L}}_{{\theta }_d,{\theta }_g}; \\ {\mathcal{L}}_{{\theta }_d,{\theta }_g}=-{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\left[\text{log}D\left(x\mid {\theta }_d\right)\right]-{\mathbb{E}}_{z~\mathcal{N}(0,I)}\left[\text{log}\left(1-D\left(G_{\mathit{\text{GEN}}}\left(x\mid x^{\mathit{\text{low}}},{\theta }_g\right),{\theta }_d\right)\right)\right]. \end{gathered}$$ (35)

The GAN framework for deep de-aliasing (Yang et al., 2017) regularizes the reconstruction by adopting several image priors such as: (i) image content information like object boundary, shape, and orientation, along with using a perceptual loss function: $\frac{1}{2}{\Vert x-x^{\mathit{\text{low}}}\Vert }_2^2$, (ii) data consistency is ensured using a frequency domain loss $\frac{1}{2}{\Vert y^t-y^u\Vert }_2^2$, and (iii) a VGG loss (see Sec. 3.4) which enforces semantic similarity between the reconstructed and the ground truth images. The method demonstrated a 2dB improvement in PSNR score on IXI dataset with 30% undersampling. However, it was observed that the finer details were lost during the process of de-aliasing with a CNN based GAN network. In the paper called self-attention and relative average discriminator based GAN (SARAGAN) (Yuan et al., 2020), the authors show that fine details tend to fade away due to smaller size of the convolution kernels, leading to poor performance. Consequently, the SARAGAN method adopts a relativistic discriminator (Jolicoeur-Martineau, 2018) along with a self-attention network (see Sec. 3.1 for self-attention) to optimize the following equation that is different from Eqn. 35:

$${\mathcal{L}}_{{\theta }_d,{\theta }_g}=-{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\left[\mathit{\text{sigmoid}}\left(D(x)-D\left(G\left(x^{\mathit{\text{low}}}\right)\right)\right)\right]-{\mathbb{E}}_{z~\mathcal{N}(0,I)}\left[\mathit{\text{sigmoid}}\left(D\left(G\left(x^{\mathit{\text{low}}}\right)\right)-D(x)\right)\right]$$ (36)

where, sigmoid(·) is the sigmoid activation function discussed in Sec. 3.2. This method showed excellent performance in the MICCAI 2013 grand challenge brain MRI reconstruction dataset and got an SSIM of 0.9951 and PSNR of 45.7536 ± 4.99 with 30% sampling rate. Among other methods, sparsity based constraints are imposed as a regularizer to Eqn. 35 in compressed sensing GAN (GANCS) (Mardani et al., 2018a), RefineGAN (Quan et al., 2018), and the structure preserving GAN (Deora et al., 2020; Lee et al., 2018) methods. Some qualitative results using the RefineGAN method are shown in Fig. 5. On the other hand, methods like PIC-GAN (Lv et al., 2021) and (MGAN) (Zhang et al., 2018a) use a SENSE-like reconstruction strategy that combines MR images reconstructed from parallel receiver coils using a GAN framework. Such methods have also shown good performance with low normalized mean squared error in the knee dataset.

Combined k-space and image space methods:

Thus far, we have discussed k-space (GRAPPA-like GAN methods) and image space (SENSE-like GAN methods) MR reconstruction methods that work in isolation. However, both these strategies can be combined together to leverage the advantages of both methods. Recently, a method called sampling augmented neural network with incoherent structure for MR image reconstruction (SANTIS) (Liu et al., 2019) was proposed that leverages a cycle consistency loss, ${\mathcal{L}}_{cyc}$ in addition to a GAN loss, ${\mathcal{L}}_{{\theta }_d,{\theta }_g}$ i.e.,

$${\mathcal{L}}_{\mathit{\text{full}}}={\mathcal{L}}_{{\theta }_d,{\theta }_g}+{\mathcal{L}}_{cyc}={\lambda }_1\mathbb{E}\left[{\Vert x-G\left(x^{\mathit{\text{low}}},{\theta }_g\right)\Vert }_2^2\right]+{\lambda }_{GAN}(\mathbb{E}\left[\text{log}D\left(x,{\theta }_d\right)\right]+\mathbb{E}\left[\text{log}\left(1-D\left(G\left(x^{\mathit{\text{low}}},{\theta }_g\right),{\theta }_d\right)\right)\right])+{\lambda }_2\mathbb{E}\left[{\Vert y-F\left(G\left(x^{\mathit{\text{low}}},{\theta }_g\right),{\theta }_f\right)\Vert }_2^2\right],$$ (37)

where the function F(·) is another generator network that projects back the MR image to k-space. The method achieved an SSIM value of 91.96 on the 4x undersampled knee. FastMRI dataset (see Fig. 5 and Table 1). In the collaborative GAN method (CollaGAN) (Lee et al., 2019), instead of cycle consistency between k-space and the image domain from a single image, they consider a collection of domains such as T1-weighted and T2-weighted data and try to reconstruct the MR images with cycle consistency in all domains. The InverseGAN (Narnhofer et al., 2019) method performs cycle consistency using a single network that learns both the forward and inverse mapping from and to k-space.

Table 1:

Normalized mean squared error (NMSE), PSNR and SSIM over the test data with 1000 samples using eight different methods at 4-fold and 8-fold acceleration factors.

Acceleration	Model	NMSE	PSNR	SSIM	Acceleration	Model	NMSE	PSNR	SSIM
4-fold	Zero Filled	0.0198	32.51	0.811	8-fold	Zero Filled	0.0352	29.60	0.642
	SENSE (Pruessmann et al., 1999)	0.0154	32.79	0.816		SENSE (Pruessmann et al., 1999)	0.0261	31.65	0.762
	GRAPPA (Griswold et al., 2002)	0.0104	27.79	0.816		GRAPPA(Griswold et al., 2002)	0.0202	25.31	0.782
	RfineGAN(Quan et al., 2018)	0.0138	34.00	0.901		RefineGAN(Quan et al., 2018)	0.0221	32.09	0.792
	DeepADMM (Sun et al., 2016)	0.0055	34.52	0.895		DeepADMM(Sun et al., 2016)	0.0201	36.37	0.810
	LORAKI (Kim et al., 2019)	0.0091	35.41	0.871		LORAKI (Kim et al., 2019)	0.0181	36.45	0.882
	VariationNET (Sriram et al., 2020a)	0.0049	38.82	0.919		VariationNET (Sriram et al., 2020a)	0.0211	36.63	0.788
	GrappaNet (Sriram et al., 2020b)	0.0026	40.74	0.957		GrappaNet (Sriram et al., 2020b)	0.0071	36.76	0.922
	J-MoDL (Aggarwal and Jacob, 2020)	0.0021	41.53	0.961		J-MoDL (Aggarwal and Jacob, 2020)	0.0065	35.08	0.928
	Deep Decoder (Heckel and Hand, 2018)	0.0132	31.67	0.938		Deep Decoder (Heckel and Hand, 2018)	0.0079	29.654	0.929
	DIP (Ulyanov et al., 2018)	0.0113	30.46	0.923		DIP (Ulyanov et al., 2018)	0.0076	29.18	0.912

4.2. Bayesian Learning

Bayes’s theorem expresses the posterior p(x|y) as a function of the k-space data likelihood, p(y|x) and the prior p(x) with the form p(x|y) ∝ p(y|x)p(x) also known as the “product-of-the-experts” in DL literature. In (Tezcan et al., 2018), the prior is estimated with a Monte Carlo sampling technique which is computationally intensive. To overcome the computational cost, several authors have proposed to learn a non-linear mapping from undersampled k-space to image space using VAEs. In these VAE based methods (Tezcan et al., 2019; Gaillochet et al., 2020; Van Essen et al., 2012), the networks are trained on image patches m_j obtained from k-space measurement y_i and the VAE network is optimized using these patches with the following cost function: ${\sum }_{j=1}^N\mathit{\text{ELBO}}\left(m_j\right)$,

$$\text{arg}\underset{m}{\text{min}}\left[\parallel Ex-y{\parallel }_2^2-\sum _{x\in \Omega (m)}\mathit{\text{ELBO}}(x)\right];\mathit{\text{ELBO}}(x)={\mathbb{E}}_{D_{{\theta }_d}(z\mid x)}\left[\text{log}{G}_{{\theta }_g}(x\mid z)+\text{log}\frac{p(z)}{G_{{\theta }_g}(x\mid z)}\right].$$ (38)

These methods have mainly been evaluated on the Human Connectome Project (HCP) (Van Essen et al., 2012) dataset and have shown good performance on 4x undersampled images (see Fig 6).

Figure 6: MR Reconstruction using VAE Based Methods:

Qualitative comparison of (a) original; (b) zero-filled; and (c) VAE based DDP method (Tezcan et al., 2019). Also shown are the difference maps in (d) between DDP and the original images. These results are borrowed from the DDP (Tezcan et al., 2019) paper.

Different from them, PixelCNN+ considers each pixel as random variable and estimates the joint distribution of pixels over an image x as the product of conditional distribution, i.e. $p(x)={\prod }_{i=1}^{n^2}p\left(x_i\mid x_1,x_2,\cdots ,x_{i-1}\right)$ The method proposed in (Luo et al., 2020) considers a generative regression model called PixelCNN+ (Oord et al., 2016) to estimate the prior p(x). This method demonstrated very good performance, i.e. they achieved more than 3 dB PSNR improvement than the current state-of-the-art methods like GRAPPA, variational networks (see Sec. 5.3) and SPIRIT algorithms. The Recurrent Inference Machines (RIM) for accelerated MRI reconstruction (Lonning et al., 2018) is an general inverse problem solver that performs a step-wise reassessments of the maximum a posteriori and infers the inverse transform of a forward model. Despite showing good results, the overall computational cost and running time is very high compared to GAN or VAE based methods.

4.3. Active Acquisition Methods

Combined k-space and image methods:

All of the above methods consider a fixed k-space sampling that is predetermined by the user. This sampling process is isolated from the reconstruction pipeline. Recent works have investigated if the sampling process itself can be included as a part of the reconstruction optimization framework. A basic overview of these works can be described as follows:

• The algorithm has access to the fully sampled training MR images {x₁, x₂, ⋯, x_N}

• The encoder, $G_{{\theta }_g}(\cdot )$, learns the sampling pattern by optimizing parameter θ_g.

• The decoder, $D_{{\theta }_d}(\cdot )$, is the reconstruction algorithm that is parameterized by θ_d

• The encoder $G_{{\theta }_g}(\cdot )$ is optimized by minimizing the empirical risk on the training MR images, $\frac{1}{N}{\sum }_{i=1}^N{L}_q\left(x_i,D_{{\theta }_d}\left(G_{{\theta }_g}(\mathcal{F}(x))\right)\right)$, where L_q is some arbitrary loss of the decoder.

This strategy was used in LOUPE (Bahadir et al., 2019) where a network was learnt to optimize the under-sampling pattern such that $G_{{\theta }_g}(\cdot )$ provided a probabilistic sampling mask $\mathcal{M}(\cdot )$ assuming each line in k-space as an independent Bernoulli random variable by optimizing:

$$\text{arg}\underset{{\theta }_g,{\theta }_d}{\text{min}}{\mathbb{E}}_{\mathcal{M}~G_{{\theta }_g}}\left[{\Vert D_{{\theta }_d}(\mathcal{M}\mathcal{F}x)-x\Vert }_1+\lambda \mathcal{M}\right],$$ (39)

where $D_{{\theta }_d}$ is an anti-aliasing deep neural network. Experiments on T₁-weighted structural brain MRI scans show that the LOUPE method improves PSNR by ~ 5% with respect to the state-of-the-art methods that is shown in Fig. 7, second column. A follow-up work to LOUPE (Bahadir et al., 2020) imposed a hard sparsity constraint on $G_{{\theta }_g}(\cdot )$ to ensure robustness to noise. In the deep active acquisition method (Zhang et al., 2019b), $G_{{\theta }_g}(\cdot )$ is termed the evaluator and $D_{{\theta }_d}(\cdot )$ is the reconstruction network. Given a zero-filled MR image, $D_{{\theta }_d}\left(x^{ZF}\right)$ provides the reconstructed image and the uncertainty map. The evaluator $G_{{\theta }_g}(\cdot )$ decomposes the reconstructed image and ground truth image into spectral maps and provides a score to each k-space line of the reconstructed image. Based on the scores, the methodology decides to acquire the appropriate k-space locations from the MR scanner. The Deep Probabilistic Subsampling (DPS) method in (Huijben et al., 2019) develops a task-adaptive probabilistic undersampling scheme using a softmax based approach followed by MR reconstruction. On the other hand, the work on joint model based deep learning (J-MoDL) (Aggarwal and Jacob, 2020) optimized both sampling and reconstruction using Eqns 21 and 22 to jointly optimize a data consistency network and a regularization network. The data consistency network is a residual network that acts as a denoiser, while the regularization network decides the sampling scheme. The PILOT (Weiss et al., 2021) method also jointly optimizes the k-space sampling and the reconstruction. The network has a sub-sampling layer to decide the importance of a k-space line, while the regridding and the task layer jointly reconstruct the image. The optimal k-space lines are chosen either using the greedy traveling salesman problem or imposing acquisition machine constraints. Joint optimization of k-space sampling and reconstruction also appeared in recent methods such as (Heng Yu, 2021; Guanxiong Luo, 2021).

Figure 7: MR Reconstruction using Active Acquisition Method:

Variable mask learned by the active acquisition method LOUPE (Bahadir et al., 2019) can improve the image quality of a UNet based MR image reconstruction task. The leftmost image is the ground truth, the second image shows the reconstruction using an optimized mask, the next image is the variaable mask, that is followed by the uniformly random and Cartesian mask. The images are borrowed from the LOUPE (Bahadir et al., 2019) paper.

5. Inverse Mapping using Non-generative Models

In this section we discuss non-generative models that use the following optimization framework:

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\parallel y-Ax{\parallel }_2^2+\lambda {\mathcal{L}}_{{\theta }_g};{\mathcal{L}}_{{\theta }_g}=\text{arg}\underset{{\theta }_g}{\text{min}}{\mathbb{E}}_{x~p_{\mathit{\text{data}}}}\frac{1}{2}\parallel x_j-G_{NGE}{\left(x\mid c,{\theta }_g\right)\parallel }_2^2.$$ (40)

The non-generative models also have a data consistency term and a regularization term similar to Eqn. 4. As discussed earlier in section 1.2, however, the non-generative models do not assume any underlying distribution of the data and learn the inverse mapping by parameter optimization using Eqn. 7. Below, we discuss the different types of non-generative models.

5.1. Perceptron Based Models

The work in (Kwon et al., 2017; Cohen et al., 2018) developed a multi-level perceptron (MLP) based learning technique that learns a nonlinear relationship between the k-space measurements, the aliased images, and the desired unaliased images. The input to an MLP is the real and imaginary part of an aliased image and the k-space measurement, and the output is the corresponding unaliased image. We show a visual comparison of this method (Kwon et al., 2017) with the SPIRiT and GRAPPA methods in Fig. 8. This method showed better performance with lower RMSE at different undersampling factors.

Figure 8: Visual Comparison with Perceptron, SPIRiT and GRAPPA methods.

We note the dealiasing capability of the perceptron based method at different sampling rates, i.e. row (a) and row (b), for a T₂-weighted image are visibly better than the classical methods. The figure is borrowed from (Kwon et al., 2017) paper.

5.2. Untrained Networks

So far, we have talked about various deep leraning architectures and their training strategies using a given training dataset. The most exciting question one can ask “is it always necessary to train a DL network to obtain the best result at test time?”, or “can we solve the inverse problem using DL similar to classical methods that do not necessarily require a training phase to learn the parameter priors”? We note several state-of-the-art methods that uses ACS lines or other k-space lines of the k-space measurement y to train a DL network instead of an MR image as a ground truth. The robust artificial neural network for k-space interpolation (RAKI) (Akçakaya et al., 2019) trains a CNN by using the ACS lines. The RAKI methodology shares some commonality with GRAPPA. However, the main distinction is the linear estimation of the convolution kernel in GRAPPA which is replaced with a non-linear kernel in CNN. The CNN kernels are optimized using the following objective function:

$$\widehat{x}=\text{arg}\underset{x}{\text{min}}\frac{1}{2}\parallel y-Ax{\parallel }_2^2+\lambda {\mathcal{L}}_{{\theta }_g};{\mathcal{L}}_{{\theta }_g}=\parallel y^{ACS}-G_{NGE}{\left({\tilde{y}}^{ACS};{\theta }_g\right)\parallel }_F^2$$ (41)

where, y^ACS are the acquired ACS lines, and G_NGE(·) is the CNN network that performs MRI reconstruction. The RAKI method has shown 0%, 0%, 11%, 28%, and 41% improvement in RMSE score with respect to GRAPPA on phantom images at {2x, 3x, 4x, 5x, 6x} acceleration factors respectively. A followup work called residual RAKI (rRAKI) (Zhang et al., 2019a) improves the RMSE score with the help of a residual network structure. The LORAKI (Kim et al., 2019) method is based on the low rank assumption of LORAKS (Haldar, 2013). It uses a recurrent CNN network to combine the auto-calibrated LORAKS (Haldar, 2013) and the RAKI (Akçakaya et al., 2019) methods. On five different slices of a T₂-weighted dataset, the LORAKI method has shown good improvement in SSIM scores compared to GRAPPA, RAKI, AC-LORAKS among others. Later, the sRAKI-RNN (Hosseini et al., 2019b) method proposed a unified framework that performs regularization through calibration and data consistency using a more simplified RNN network than LORAKI.

Deep Image Prior (DIP) and its variants (Ulyanov et al., 2018; Cheng et al., 2019; Gandelsman et al., 2019) have shown outstanding results on computer vision tasks such as denoising, in-painting, super resolution, domain translation etc. A vanilla DIP network uses a randomly weighted autoencoder, $D_{{\theta }_d}\left(G_{{\theta }_g}(z)\right)$, that reconstructs a clean image x ∈ R^W×H×3 given a fixed noise vector z ∈ R^W×H×D. The network is optimized using the “ground truth” noisy image $\widehat{x}$. A manual or user chosen “early stopping” of the optimization is required as optimization until convergence overfits to noise in the image. A recent work called Deep Decoder (Heckel and Hand, 2018) shows that an under-parameterized decoder network, $D_{{\theta }_d}(\cdot )$, is not expressive enough to learn the high frequency components such as noise and can nicely approximate the denoised version of the image. The Deep Decoder uses pixel-wise linear combinations of channels and shared weights in spatial dimensions that collectively help it to learn relationships and characteristics of nearby pixels. It has been recently understood that such advancements can directly be applied to MR image reconstruction (Mohammad Zalbagi Darestani, 2021). Given a set of k-space measurements y₁, y₂, ⋯, y_n from receiver coils, an un-trained network G_θ(z) uses an iterative first order method to estimate parameters $\widehat{\theta }$ by optimizing;

$$\text{min}{\mathcal{L}}_{\theta }=\frac{1}{2}{\Vert y_i-\mathcal{M}\mathcal{F}G(z;\theta )\Vert }_2^2.$$ (42)

The network is initialized with random weight θ₀ and then optimized using Eqn. 42 to obtain $\widehat{\theta }$. The work in (Dave Van Veen, 2021) introduces a feature map regularization: $\frac{1}{2}{\sum }_{j=1}^L{\Vert D_j{y}_i-\mathcal{M}\mathcal{F}{G}_{j,i}(z;\theta )\Vert }_2^2$, in Eqn. 42 where D_j matches the features of j^th layer. This term encourages fidelity between the network’s intermediate representations and the acquired k-space measurements. The works in (Heckel, 2019; Heckel and Soltanolkotabi, 2020) provide theoretical guarantees on recovery of image x from the k-space measurements. Recently proposed method termed “Scan-Specific Artifact Reduction in k-space” or SPARK (Arefeen et al., 2021) trains a CNN to estimate and correct k-space errors made by an input reconstruction technique. The results of this method are also quite impressive given that only ACS lines are used for training the CNN. Along similar lines, the authors in (Yoo et al., 2021) used the Deep Image Prior setup for dynamic MRI reconstruction.

In the self supervised approach, a subset of the undersampled k-space lines are typically used to validate the DL network in addition to the acquired undersampled k-space lines that are used to optimize the network. Work in this direction divides the total k-space lines into two portions:(i) k-space lines for data consistency y^dc and (ii) k-space lines y^loss for regularization. In (Yaman et al., 2020), the authors use a multi-fold validation set of k-space data to optimize the DL network. Other methods such as SRAKI (Hosseini et al., 2020, 2019a) use the self-supervision to reconstruct the images. A deep Reinforcement learning based approach is studied in (Jin et al., 2019) that deploys a reconstruction network and an active acquisition network. Another method such as (Yaman et al., 2021b) provides an unrolled optimization algorithm to estimate the missing k-space lines. Other methods that fall under this umbrella include the transformer based method (Korkmaz et al., 2021) and a scan specific optimization method (Yaman et al., 2021a; Tamir et al., 2019).

5.3. Convolutional Neural Networks

Spatial models are mostly dominated by the various flavours of CNNs such as complex-valued CNN (Wang et al., 2020b;Cole et al., 2019), unrolled optimization using CNN (Schlemper et al., 2017), variational networks (Hammernik et al., 2018), etc. Depending on how the MR images are reconstructed, we divide all CNN based spatial methods into the following categories.

Inverse mapping from k-space:

The AUTOMAP (Zhu et al., 2018) learns a reconstruction mapping using a network having three fully connected layers (3 FCs) and two convolutional layers (2 Convs) with an input dimension of 128 × 128. Any image of size more than 128 × 128 is cropped and subsampled to 128 × 128. The final model yielded a PSNR of 28.2 on FastMRI knee dataset outperforming the previous validation baseline of 25.9 on the same dataset. Different from these methods, there are a few works (Wang et al., 2020b;Cole et al., 2019) that have used CNN networks with complex valued kernels to reconstruct MR images from complex valued k-space measurements. The method in (Wang et al., 2020a) uses a complex valued ResNet (that is a type of CNN) network and is shown to obtain good results on 12-channel fully sampled k-space dataset (see Fig. 9 for a visual comparison with other methods). Another method uses a Laplacian pyramid-based complex neural network (Liang et al., 2020b) for MR image reconstruction.

Figure 9: Comparison of a Complex-valued CNN with other state-of-the-art methods:

We show visual comparison of SPIRiT (Lustig and Pauly, 2010), GRAPPA (Bouman and Sauer, 1993), VariationalNet, and the ComplexCNN. We note from the difference map that the ComplexCNN performed well with respect to the other state-of-the-art methods. The PSNR and SSIM values are given at the bottom for each method.

Inverse Mapping for Image Rectification:

In CNN based sequential spatial models such as DeepADMM net models (Sun et al., 2016; Schlemper et al., 2017) and Deep Cascade CNN (DCCNN) (Schlemper et al., 2017), the regularization is done in image space using the following set of equations:

$$\begin{gathered} \text{arg}\underset{x}{\text{min}}\frac{1}{2}\parallel Ax-y{\parallel }_2^2+\parallel x+\beta -z{\parallel }_2^2;\text{arg}\underset{z_i}{\text{min}}\sum _i\left[{\lambda }_{i}g\left({\Gamma }_i{z}_i\right)\parallel x+\beta -z{\parallel }_2^2\right] \\ \beta \leftarrow \beta +\alpha (x-z). \end{gathered}$$ (43)

Here, α and β are Lagrange multipliers. The ISTA net (Zhang and Ghanem, 2018) modifies the above image update rule as follows, $x_i=\text{arg}{\text{min}}_x\frac{1}{2}{\Vert C(x)-C\left(z_i\right)\Vert }_2^2+\lambda \parallel C(x){\parallel }_1$, using a CNN network C(·). Note that the DeepADMM network demonstrated good performance when the network was trained on brain data but tested on chest data. Later, MODL (Aggarwal et al., 2019) proposed a model based MRI reconstruction where they used a convolution neural network (CNN) based regularization prior. Later, a dynamic MRI using MODL based deep learning was proposed by (Biswas et al., 2019). The optimization, i.e. $\text{arg}{\text{min}}_x\frac{1}{2}\parallel Ax-y{\parallel }_2^2+\lambda \parallel C(x){\parallel }_2^2,$, denoises the alias artifacts and noise using a CNN network C(·) as a regularization prior, and λ is a trainable parameter. To address this concern, a full end-to-end CNN model called GrappaNet (Sriram et al., 2020b) was developed, which is a nonlinear version of GRAPPA set within a CNN network. The CNN network has two sub networks; the first sub network, f₁(y), fills the missing k-space lines using a non-linear CNN based interpolation function similar to GRAPPA. Subsequently, a second network, f₂, maps the filled k-space measurement to the image space. The GrappaNet model has shown excellent performance (40.74 PSNR, 0.957 SSIM) on the FastMRI dataset and is one of the best performing methods. A qualitative comparison is shown in Fig. 10. Along similar lines, a deep variational network (Hammernik et al., 2018) is used to MRI reconstruction. Other works, such as (Wang et al., 2016;Cheng et al., 2018; Aggarwal et al., 2018) train the parameters of a deep network by minimizing the reconstruction error between the image from zero-filled k-space and the image from fully sampled k-space. The cascaded CNN network learns spatio-temporal correlations efficiently by combining convolution and data sharing approaches in (Schlemper et al., 2017). The (Seegoolam et al., 2019) method proposed to use a CNN network to estimate motions from undersampled MRI sequences that is used to fuse data along the entire temporal axis.

Figure 10:

GrappaNet Comparison with VariationalNET, UNet methods

5.4. Recurrent Neural Networks

Inverse mapping from k-space:

We note that a majority of the iterative temporal networks, a.k.a the recurrent neural network models, are k-space to image space reconstruction methods and typically follow the optimization described in Section 5.3. The temporal methods, by design, are classified into two categories, namely (i) regularization methods, and (ii) variable splitting methods.

Several state-of-the-art methods have considered temporal methods as a way to regularize using the iterative hard threshold (IHT) method from (Blumensath and Davies, 2009) that approximates the l₀ norm. Mathematically, the IHT update rule is as follows:

$$x_{t+1}=H_k\left[x_t-\alpha {\Phi }^T\left(\Phi x_t-y\right)\right],$$ (44)

where α is the step-size parameter, H_k[·] is the operator that sets all but k-largest values to zero (proxy for l₀ operation), and the dictionary Φ satisfies the restricted isometry property (RIP)⁵. The work in (Xin et al., 2016) shows that this hard threshold operator resembles the memory state of the LSTM network. Similar to the clustering based sparsity pattern of IHT, the gates of LSTM inherently promotes sparsity. Along similar lines, the Neural Proximal Gradient Descent work (Mardani et al., 2018b) envisioned a one-to-one correspondence between the proximal gradient descent operation and the update of a RNN network. Mathematically, an iteration of a proximal operator P_f given by: x_t+1 = P_f(x_t +αΦ^H(y − Φx)), resembles the LSTM update rule:

$$s_{t+1}=g\left(x_t;y\right);x_{t+1}=P_f\left(s_{t+1}\right),$$ (45)

where g(x_t; y) = x_t + αΦ^H(y − Φx) is the update step, s_t+1 is the hidden state and Φ is a dictionary. Different from these, a local-global recurrent neural network is proposed in (Guo et al., 2021) that uses two recurrent networks, one network to capture high frequency components, and another network to capture the low frequency components. The method in (Oh et al., 2021) uses a bidirectional RNN and replaces the dense network structure of (Zhu et al., 2018) while removing aliasing artifacts in the reconstructed image.

The Convolutional Recurrent Neural Networks or CRNN (Qin et al., 2018) method proposed a variable splitting and alternate minimisation method using a RNN based model. Recovering finer details was the main challenge of the PyramidRNN (Wang et al., 2019) that proposed to reconstruct images at multiple scales. Three CRNNs are deployed to reconstruct images at different scales, i.e. x⁰ = CRNN(x^zero, y, θ₁), x¹ = CRNN(x⁰, y, θ₂), x² = CRNN(x¹, y, θ₃), and the final data consistency is performed after x⁰, x¹, x² are combined using another CNN. The CRNN is used as a recurrent neural network in the variational approach of VariationNET (Sriram et al., 2020a), i.e.

$$x_{t+1}=x_t-\alpha {\Phi }^H(y-\Phi x)+D\left(x_t\right),$$ (46)

where D(·) is a CRNN network that provides MR reconstruction. In this unrolled optimization method, the CRNN is used as a proximal operator to reconstruct the MR image. The VariationNET is a followup work of Deep Variational Network of (Hammernik et al., 2018) that we discussed in Sec. 5.3. The VariationalNET unrolls an iterative algorithm involving a CRNN based recurrent neural network based regularizer, while the Deep Variational Network of (Hammernik et al., 2018) unrolls an iterative algorithm involving a receptive field based convolutional regularizer.

5.5. Hypernetwork Models

Hypernetworks are meta-networks that regress optimal weights of a task network (often called as data network (Pal and Balasubramanian, 2019) or main network (Ha et al., 2016)). The data network G_NGE(x^low, θ_g) performs the mapping from aliased or the low resolution images x^low to the high resolution MR images x^gen. The hypernetwork H(α, θ_hyp) estimates weights θ_g of the network G_NGE(θ_g) given the random variable α sampled from a prior distribution α ~ p(α). The end-to-end network is trained by optimizing:

$$\text{arg}\underset{\psi }{\text{min}}{\mathbb{E}}_{{\theta }_g~H\left(\alpha ,{\theta }_{hyp}\right)}\left[\parallel G_{NGE}\left(x^{\mathit{\text{low}}},{\theta }_g\right)-x{\parallel }_2^2+\mathcal{R}\left(G_{NGE}\left(x^{\mathit{\text{low}}},{\theta }_g\right)\right)\right].$$ (47)

In (Wang et al., 2021), the prior distribution p(α) is a uniform distribution $\mathcal{U}[-1,+1]$ (and the process is called Uniform Hyperparameter Sampling) or the sampling can be based on the data density (called data-driven hyperparameter sampling). Along similar lines, the work in (Ramanarayanan et al., 2020) trained a dynamic weight predictor (DWP) network that provides layer wise weights to the data network. The DWP generates the layer wise weights given the context vector γ that comprises of three factors such as the anatomy under study, undersampling mask pattern and the acceleration factor.

6. Comparison of state-of-the-art methods

Given the large number of DL methods being proposed, it is imperative to compare these methods on a standard publicly available dataset. Many of these methods have shown their effectiveness on various real world datasets using different quantitative metrics such as SSIM, PSNR, RMSE, etc. There is, however, a scarcity of qualitative and quantitative comparison of these methods on a single dataset. While the fastMRI challenge allowed comparison of several methods, yet, several recent methods from the categories discussed above were not part of the challenge. Consequently, we compare a few representative MR reconstruction methods both qualitatively and quantitatively on the fastMRI knee dataset (Zbontar et al., 2018). We note that, doing a comprehensive comparison of all the methods mentioned in this review is not feasible due to non-availability of the code as well as the sheer magnitude of the number of methods (running into hundreds). We compared the following representative models:

• Zero filled image reconstruction method

• Classical image space based SENSE method (Pruessmann et al., 1999)

• Classical k-space based GRAPPA method (Griswold et al., 2002)

• Unrolled optimization based method called DeepADMM (Sun et al., 2016)

• Low rank based LORAKI (Kim et al., 2019)

• Generative adversarial network based RefineGAN (Quan et al., 2018) network

• Variational network called VariationNET (Sriram et al., 2020a)

• The deep k-space method GrappaNet (Sriram et al., 2020b)

• Active acquisition based method J-MoDL (Aggarwal and Jacob, 2020)

• Untrained network model Deep Decoder (Heckel and Hand, 2018) and

• Deep Image Prior DIP (Ulyanov et al., 2018) method.

The fastMRI knee dataset consists of raw k-space data from 1594 scans acquired on four different MRI machines. We used the official training, validation and test data split in our experiments. We did not use images with a width greater than 372 and we note that such data is only 7% of the training data split. Both the 4x and 8x acceleration factors were evaluated.

We used the original implementation⁶ of GrappaNet, VariationaNET, SENSE, GRAPPA, DeepADMM, Deep Decoder, and DIP method. Similar to GrappaNET, we always use the central 30 k-space lines to compute the training target. Treating the real and imaginary as two distinct channels, we dealt with the complex valued input, i.e. we have 30 channel input k-space measurements for the 15-coil complex-valued k-space. Where applicable, the models were trained with a linear combination of L₁ and SSIM loss, i.e.

$$J(\widehat{x},x)=-\mathit{\text{SSIM}}(G(y),x)+\lambda \parallel G(y)-x{\parallel }_1$$ (48)

where λ is a hyperparameter, G(y) is the model prediction, and x is the ground truth.

Quantitative results are shown in Table 1 for several metrics such as NMSE, PSNR, and SSIM scores. We observe that GrappaNET, J-MoDL and VariationNET outperformed the baseline methods by a large margin. We note that the zero-filled and SENSE reconstructions in Fig 11 (a), (b) show a large amount of over-smoothing. The reconstruction of SENSE and zero-filled model also lack a majority of the high frequency detail that is clinically relevant, but fine details are visible in case of GrappaNET, VariationNET, J-MoDL, and RefineGAN methods. The comparable performance of Deep Decoder and DIP advocates the importance of letting untrained neural network figure out how to perform k-space to MR image reconstruction. The J-MoDL method makes heavy use of training data and the joint optimization of k-space lines and the reconstruction of MR images to get good results both for 4× and 8× as shown in Table 1. On the other hand, the Deep Decoder and DIP methods achieve good performance using untrained networks as discussed in Sec. 5.2, which is advantageous as it generalizes to any MR reconstruction scenario.

Figure 11: Qualitative Comparison:

Comparison for 8x acceleration using “Zero Filled”, SENSE, DeepADMM (Sun et al., 2016), LORAKI (Kim et al., 2019), RefineGAN (Quan et al., 2018), VariationNET (Sriram et al., 2020a), GrappaNet (Sriram et al., 2020b), J-MoDL (Aggarwal and Jacob, 2020) results on fastMRI dataset. We show qualitative results for (a) knee and (b) brain datasets and also report the corresponding SSIM scores.

7. Discussion

In this paper, we discussed and reviewed several classical reconstruction methods, as well as deep generative and non-generative methods to learn the inverse mapping from k-space to image space. Naturally, one might ask the following questions given the review of several papers above: “are DL methods free from errors?”, “do they always generalize well?”, and “are they robust?”. To better understand the above mentioned rhetorical questions, we need to discuss several aspects of the performance of these methods such as (i) correct reconstruction of minute details of pathology and anatomical structures; (ii) risk quantification; (iii) robustness; (iv) running time complexity; and (v) generalization.

Due to the blackbox-like nature of DL methods, the reliability and risk quantification associated with them are often questioned. In a recent paper on “risk quantification in Deep MRI reconstruction” (Edupuganti et al., 2020), the authors strongly suggest for quantifying the risk and reliability of DL methods and note that it is very important for accurate patient diagnoses and real world deployment. The paper also shows how Stein’s Unbiased Risk Estimator (SURE) (Metzler et al., 2018) can be used as a way to assess uncertainty of the DL model:

$$\mathit{\text{SURE}}=-n{\sigma }^2+\parallel \widehat{x}-x{\parallel }^2+{\sigma }^2\mathit{\text{trace}}\left(\frac{\partial \widehat{x}}{\partial x}\right),$$ (49)

where the second term represents the end-to-end network sensitivity to small input perturbations. This formulation works even when there is no access to the ground truth data x. In this way, we can successfully measure the risk associated with a DL model. In addition to the SURE based method, there are a few other ways to quantify the risk and reliability associated with a DL model. The work “On instabilities of deep learning in image reconstruction” (Antun et al., 2019) uses a set of pretrained models such as AUTOMAP (Zhu et al., 2018), DAGAN (Yang et al., 2017), or Variational Network (Sriram et al., 2020a) with noisy measurements to quantify their stability. This paper as well as several others (Narnhofer et al., 2021; Antun et al., 2019) have discussed how the stability of the reconstruction process is related to the network architecture, training set and also the subsampling pattern.

Whether a DL method can capture high frequency components or not is also another area of active research in MR reconstruction. The robustness of a DL based MR reconstruction method is also studied in various papers such as (Raj et al., 2020; Cheng et al., 2020a; Calivá et al., 2020; Zhang et al., 2021). For example, the (Cheng et al., 2020a; Calivá et al., 2020; Cheng et al., 2020b) works perceived adversarial attack as a way to capture minute details during sMR reconstruction thus showing a significant improvement in robustness compared to other methods. The work proposed to train a deep learning network with a loss that pays special attention to small anatomical details. The methodology progressively adds minuscule perturbation δ to the input x not perceivable to human eye but may shift the decision of a DL system. The method in (Raj et al., 2020) uses a generative adversarial framework that entails a perturbation generator network G(·) to add minuscule distortion on k-space measurement y. The work in (Zhang et al., 2021) proposed to incorporate fast gradient based attack on a zero filled input image and train the deep learning network not to deviate much under such attack. The FINE (Zhang et al., 2020) methodology, on the other hand, has used pre-trained recon network that was fine-tuned using data consistency to reduce generalization error inside unseen pathologies. Please refer to the paper (Darestani et al., 2021) which provides a summary of robustness of different approaches for image reconstruction.

Optimizing a DL network is also an open area of active research. The GAN networks suffer from a lack of proper optimization of the structure of the network (Goodfellow et al., 2014). On the other hand, the VAE and Bayesian methods suffer from the large training time complexities. We see several active research groups and papers (Salimans et al., 2016; Bond-Taylor et al., 2021) in Computer Vision and Machine Learning areas pondering upon these questions and coming up with good solutions. Also, the work in (Hammernik et al., 2017) has shown the effectiveness of capturing perceptual similarity using the l₂ and SSIM loss to include local structure in the reconstruction process. Recently, work done in (Maarten Terpstra, 2021) shows that the standard l₂ loss prefers or biases the reconstruction with lower image magnitude and hence they propose a new loss function between ground truth x^gt and the reconstructed complex-valued images x, i.e. L_perp = P(x, x^gt) + l₁(x, x^gt) where P(x, x^gt) = |Real(x)Imaginary(x^gt) − Real(x^gt)Imaginary(x)|/|x^gt| which favours the finer details during the reconstruction process. The proposed loss function achieves better performance and faster convergence on complex image reconstruction tasks.

Regarding generalization, we note that some DL based models have shown remarkable generalization capabilities, for example: the AUTOMAP work was trained on natural images but generalized well for MR reconstruction. However, current hardware (memory) limitations preclude from using this method for high resolution MR reconstruction. On the other hand, some of the latest algorithms that show exceptional performance on the knee dataset have not been extensively tested on other low SNR data. In particular, these methods also need to be tested on quantitative MR modalities to better assess their performance.

Another bottleneck for using these DL methods is the large amount of training data required. While GAN and Bayesian networks produce accurate reconstruction of minute details of anatomical structures if sufficient data are available at the time of training, it is not clear as to how much training data is required and whether the networks can adapt quickly to change in resolution and field-of-view. Further, these works have not been tested in scenarios where changes in MR acquisition parameters such as relaxation time (TR), echo time (TE), spatial resolution, number of channels, and undersampling pattern are made at test time and are different from the training dataset.

Most importantly, MRI used for diagnostic purposes should be robust and accurate in reconstructing images of pathology (major and minor). While training based methods have demonstrated their ability to reconstruct normal looking images, extensive validation of these methods on pathological datasets is needed for adoption in clinical settings. To this end, the MR community needs to collaborate and collect normative and pathological datasets for testing. We specifically note that, the range of pathology can vary dramatically in the anatomy being imaged (e.g., the size, shape, location and type of tumor). Thus, extensive training and unavailability of pathological images may present significant challenges to methods that are data hungry. In contrast, untrained networks may provide MR reconstruction capabilities that are significantly better than the current state-of-the-art but do not perform as well as highly trained networks (but generalize well to unknown scenarios).

Finally, given the exponential rate at which new DL methods are being proposed, several standardized datasets with different degrees of complexity, noise level (for low SNR modalities) and ground truth availability are required to perform a fair comparison between methods. Additionally, fully-sampled raw data (with different sampling schemes) needs to be made available to compare for different undersampling factors. Care must be taken not to obtain data that have already been “accelerated” with the use of standard GRAPPA-like methods, which might bias the results (Efrat Shimron, 2021).

Nevertheless, recent developments using new DL methods point to the great strides that have been made in terms of data reconstruction quality, risk quantification, generalizability and reduction of running time complexity. We hope that this review of DL methods for MR image reconstruction will give researchers a unique viewpoint and summarize in succinct terms the current state-of-the-art methods. We however humbly note that, given the large number of methods presented in the literature, it is impossible to cite and categorize each one of them. As such, in this review, we collected and described broad categories of methods based on the type of methodology used for MR reconstruction.

Figure 2: MRI Reconstruction Methods:

We shall discuss various MR reconstruction methods with a main focus on Deep Learning (DL) based methods. Depending on the optimization function, a DL method can be classified into a Generative (discussed in Sec. 4) or a Non-Generative model (discussed in Sec. 5). However, for the sake of completeness, we shall also discuss classical k-space (GRAPPA-like) and image space based (SENSE-like) methods.