Learning a Variational Network for Reconstruction of Accelerated MRI Data

Abstract

Purpose

To allow fast and high-quality reconstruction of clinical accelerated multi-coil MR data by learning a variational network that combines the mathematical structure of variational models with deep learning.

Theory and Methods

Generalized compressed sensing reconstruction formulated as a variational model is embedded in an unrolled gradient descent scheme. All parameters of this formulation, including the prior model defined by filter kernels and activation functions as well as the data term weights, are learned during an offline training procedure. The learned model can then be applied online to previously unseen data.

Results

The variational network approach is evaluated on a clinical knee imaging protocol for different acceleration factors and sampling patterns using retrospectively and prospectively undersampled data. The variational network reconstructions outperform standard reconstruction algorithms, verified by quantitative error measures and a clinical reader study for regular sampling and acceleration factor 4.

Conclusion

Variational network reconstructions preserve the natural appearance of MR images as well as pathologies that were not included in the training data set. Due to its high computational performance, i.e., reconstruction time of 193 ms on a single graphics card, and the omission of parameter tuning once the network is trained, this new approach to image reconstruction can easily be integrated into clinical workflow.

INTRODUCTION

Imitating human learning with deep learning (1, 2) has become an enormously important area of research and development, with a high potential for far-reaching application, including in the domain of Computer Vision. Taking encouragement from early successes in image classification tasks (3), recent advances also address semantic labeling (4), optical flow (5) and image restoration (6). In medical imaging, deep learning has also been applied to areas like segmentation (7, 8), q-space image processing (9), and skull stripping (10). However, in these applications, deep learning was seen as a tool for image processing and interpretation. The goal of the current work is to demonstrate that the concept of learning can also be used at the earlier stage of image formation. In particular, we focus on image reconstruction for accelerated MRI, which is commonly accomplished with frameworks like Parallel Imaging (PI) (11–13) or Compressed Sensing (CS) (14–16). CS in particular relies on three conditions to obtain images from k-space data sampled below the Nyquist rate (17,18).

The first CS condition requires a data acquisition protocol for undersampling such that artifacts become incoherent in a certain transform domain (14, 15). In MRI, we usually achieve incoherence by random (16) or non-Cartesian sampling trajectories (19). The second requirement for CS is that the image to be reconstructed must have a sparse representation in a certain transform domain. Common choices are the Wavelet transform (16, 20) or Total Variation (TV) (19, 21–23). In these transform domains, the l₁ norm is commonly applied to obtain approximate sparsity. The third CS condition requires a non-linear reconstruction algorithm that balances sparsity in the transform domain against consistency with the acquired undersampled k-space data.

Despite the high promise of CS approaches, most routine clinical MRI examinations are still based on Cartesian sequences. Especially in the case of 2D sequences, it can be challenging to fulfill the criteria for incoherence required by CS (24). One other obstacle to incorporation of CS into some routine clinical routine examinations is the fact that the sparsifying transforms employed in CS applications to date may be too simple to capture the complex image content associated with biological tissues. This can lead to reconstructions that appear blocky and unnatural, which reduces acceptance by clinical radiologists. A further drawback, not only for CS but for advanced image acquisition and reconstruction methods in general, is the long image reconstruction time typically required for iterative solution of non-linear optimization problems. A final challenge concerns the selection and tuning of hyper-parameters for CS approaches. A poor choice of hyper-parameters leads either to over-regularization, i.e., excessively smooth or unnatural-looking images, or else to images that still show residual undersampling artifacts. The goal of our current work is to demonstrate that, using learning approaches, we can achieve accelerated and high-quality MR image reconstructions from undersampled data which do not fulfill the usual CS conditions, which we adress with both quantitative error measures and a clinical reader study.

With current iterative image reconstruction approaches, we treat every single exam and resulting image reconstruction task as a new optimization problem. We do not use information about the expected appearance of the anatomy, or the known structure of undersampling artifacts, explicitly in these optimization problems, which stands in stark contrast to how human radiologists read images. Radiologists are trained throughout their careers to look for certain reproducible patterns, and they obtain remarkable skills to “read through” known image artifacts (24). Translating this conceptual idea of human learning to deep learning allows us to shift the key effort of optimization from the online reconstruction stage to an up-front offline training task. In other words, rather than solving an inverse problem to compute, for each new data set, a suitable transform between raw data and images, we propose to learn the key parameters of that inverse transform in advance, so that it can be applied to all new data as a simple flow-through operation.

In this work, we introduce an efficient trainable formulation for accelerated PI-based MRI reconstruction that we term a variational network (VN). The VN embeds a generalized CS concept, formulated as a variational model, within a deep learning approach. Our VN is designed to learn a complete reconstruction procedure for complex-valued multi-channel MR data, including all free parameters which would otherwise have to be set empirically. We train the VN on a complete retrospectively undersampled clinical protocol for musculoskeletal imaging, evaluating performance for different acceleration factors, and for both regular and pseudo-random Cartesian 2D sampling. Using both retrospectively and prospectively undersampled clinical patient data, we investigate the applicability of our proposed VN approach for clinical routine examination, including improved image quality and preservation of unique pathologies that are not included in the training data set.

THEORY

From Linear Reconstruction to a Variational Network

In MRI reconstruction, we naturally deal with complex numbers. Here, we introduce a mapping to real-valued numbers that we will use throughout our manuscript. We define complex images $\stackrel{\mathbf{\sim }}{\mathbf{u}}$ of size n_X × n_Y = N as equivalent real images u as follows:

$$\stackrel{\mathbf{\sim }}{\mathbf{u}}={\mathbf{u}}_{\text{RE}}+\mathrm{j}{\mathbf{u}}_{\text{IM}}\in {\mathrm{ℂ}}^N\iff \mathbf{u}=({\mathbf{u}}_{\text{RE}},{\mathbf{u}}_{\text{IM}})\in {\mathrm{ℝ}}^{2N}.$$

We consider the ill-posed linear inverse problem of finding a reconstructed image u ∈ ℝ^2N that satisfies the following system of equations

$$\mathbf{A}\mathbf{u}=\widehat{\mathbf{f}},$$ [1]

where $\widehat{\mathbf{f}}\in {\mathrm{ℝ}}^{2NQ}$ is the given undersampled k-space data, where missing data are padded by zeros. The linear forward sampling operator A implements point-wise multiplications with Q coil sensitivity maps, Fourier transforms, and undersampling according to a selected sampling pattern. Originally, the operator A is defined by the mapping ${\mathrm{ℂ}}^N\mapsto {\mathrm{ℂ}}^{NQ}$, but embedding it in our real-valued problem changes the mapping to ℝ^2N ↦ ℝ^2NQ. Since the system in Eq. 1 is ill-posed, we cannot solve for u explicitly. Therefore, a natural idea is to compute u by minimizing the least squares error

$$\underset{\mathbf{u}}{min}\frac{1}{2}{\Vert \mathbf{A}\mathbf{u}-\widehat{\mathbf{f}}\Vert }_2^2.$$ [2]

In practice we do not have access to the true $\widehat{\mathbf{f}}$ but only to a noisy variant f satisfying

$${\Vert \widehat{\mathbf{f}}-\mathbf{f}\Vert }_2\le \delta$$

where δ is the noise level. The idea is to perform a gradient descent on the least squares problem Eq. 2 that leads to an iterative algorithm, which is known as the Landweber method (25). It is given by choosing some initial u⁰ and performing the iterations with step sizes α^t

$${\mathbf{u}}^{t+1}={\mathbf{u}}^t-{\alpha }^t{\mathbf{A}}^{\ast }(\mathbf{A}{\mathbf{u}}^t-\mathbf{f}),t\ge 0$$ [3]

where A* is the adjoint linear sampling operator. To prevent over-fitting to the noisy data f, it is beneficial to stop the Landweber iterative algorithm early (26), i.e., after a finite number of iterations T.

Instead of early stopping, we can also extend the least squares problem by an additional regularization term $\mathcal{R}(\mathbf{u})$ to prevent over-fitting. The associated (variational) minimization problem is given by

$$\underset{\mathbf{u}}{min}\left\{\mathcal{R}(\mathbf{u})+\frac{\lambda }{2}{\Vert \mathbf{A}\mathbf{u}-\mathbf{f}\Vert }_2^2\right\}.$$

The minimizer of the regularized problem depends on the trade-off between the regularization term and the least squares data fidelity term controlled by λ > 0. One of the most influential regularization terms in the context of images is the TV semi-norm (21), which is defined as

$$\mathcal{R}(\mathbf{u})={\Vert (\mathbf{D}{\mathbf{u}}_{\text{RE}},\mathbf{D}{\mathbf{u}}_{\text{IM}})\Vert }_{2,1}={\displaystyle \sum _{l=1}^N\sqrt{|\mathbf{D}{\mathbf{u}}_{\text{RE}}{|}_{l,1}^2+|\mathbf{D}{\mathbf{u}}_{\text{IM}}{|}_{l,1}^2+|\mathbf{D}{\mathbf{u}}_{\text{RE}}{|}_{l,2}^2+|\mathbf{D}{\mathbf{u}}_{\text{IM}}{|}_{l,2}^2}}$$

where D : ℝ^N ↦ ℝ^N×2 is a finite differences approximation of the image gradient, see for example (27). The main advantage of TV is that it allows for sharp discontinuities (edges) in the solution while being a convex functional enabling efficient and global optimization. From a sparsity point of view, TV induces sparsity in the image edges and hence, favors piecewise constant solutions. However, it is also clear that the piecewise-constant approximation is not a suitable criterion to describe the complex structure of MR images and a more general regularizer is needed.

A generalization of the TV is the Fields of Experts model (28)

$$\mathcal{R}(\mathbf{u})={\displaystyle \sum _{i=1}^{N_k}\langle {\mathbf{\Phi }}_i({\mathbf{K}}_i\mathbf{u}),\mathbf{1}\rangle }.$$ [4]

Here, the regularization term is extended to N_k terms and 1 denotes a vector of ones. The linear operator K = (K_RE, K_IM) : ℝ^2N ↦ ℝ^N models convolutions with filter kernels k ∈ ℝ^s×s×2 of size s, which is expressed as

$$\mathbf{K}\mathbf{u}={\mathbf{K}}_{\text{RE}}{\mathbf{u}}_{\text{RE}}+{\mathbf{K}}_{\text{IM}}{\mathbf{u}}_{\text{IM}},\mathbf{u}\in {\mathrm{ℝ}}^{2N}\iff u\ast k=u_{\text{RE}}\ast k_{\text{RE}}+u_{\text{IM}}\ast k_{\text{IM}},u\in {\mathrm{ℝ}}^{n_{\mathrm{X}}\times n_{\mathrm{Y}}\times 2}.$$

The non-linear potential functions Φ(z) = (ϕ(z₁),…, ϕ(z_N))^⊤ : ℝ^N ↦ ℝ^N are composed by scalar functions ϕ. In the Fields of Experts model (28), both convolution kernels and parametrization of the non-linear potential functions, such as student-t functions, are learned from data.

Plugging the Fields of Experts model Eq. 4 into the Landweber iterative algorithm Eq. 3 yields

$${\mathbf{u}}^{t+1}={\mathbf{u}}^t-{\alpha }^t\left({\displaystyle \sum _{i=1}^{N_k}{({\mathbf{K}}_i)}^{\top }{\mathbf{\Phi }}_i^{ʹ}({\mathbf{K}}_i{\mathbf{u}}^t)+\lambda {\mathbf{A}}^{\ast }(\mathbf{A}{\mathbf{u}}^t-\mathbf{f})}\right)$$ [5]

where ${\mathbf{\Phi }}_i^{ʹ}(\mathbf{z})=\text{diag}({\varphi }_i^{ʹ}(z_1),\dots ,{\varphi }_i^{ʹ}(z_N))$ are the activation functions defined by the first derivative of potential functions Φ_i. Observe that the application of the tranpose operation (K_i)^⊤ can be implemented as a convolution with filter kernels k_i rotated by 180°. Chen et al. (6) introduce a trainable reaction-diffusion approach that performs early stopping on the gradient scheme Eq. 5 and allows the parameters, i.e., filters, activation functions and data term weights, to vary in every gradient descent step t. All parameters of the approach are learned from data. This approach has been successfully applied to a number of image processing tasks including image denoising (6), JPEG deblocking (6), demosaicing (29) and image inpainting (30). For MRI reconstruction, we rewrite the trainable gradient descent scheme with time-varying parameters ${\mathbf{K}}_i^t$, ${\mathbf{\Phi }}_i^{tʹ}$, λ^t as

$${\mathbf{u}}^{t+1}={\mathbf{u}}^t-{\displaystyle \sum _{i=1}^{N_k}{({\mathbf{K}}_i^t)}^{\top }{\mathbf{\Phi }}_i^{tʹ}}({\mathbf{K}}_i^t{\mathbf{u}}^t)-{\lambda }^t{\mathbf{A}}^{\ast }(\mathbf{A}{\mathbf{u}}^t-\mathbf{f}),0\le t\le T-1.$$ [6]

Additionally, we omit the step size α^t in Eq. 5 because it is implicitly contained in the activation functions and data term weights.

By unfolding the iterations of Eq. 6, we obtain the variational network (VN) structure as depicted in Figure 1. Essentially, one iteration of an iterative reconstruction can be related to one step in the network. In our VN approach, we directly use the measured raw data as input. Coil sensitivity maps are pre-computed from the fully sampled k-space center. A zero filled solution is computed from the undersampled k-space data by applying the adjoint operator A*. The measured raw data and sensitivity maps, together with the zero filled initializations, are fed into the VN as illustrated in Supporting Figure S1. The sensitivity maps are used in the operators A, A*, which perform sensitivity-weighted image combination and can also implement other processing steps such as the removal of readout oversampling. While both raw data and operators A, A* are required in every iteration of the VN to implement the gradient of the data term, the gradient of the regularization is only applied in the image domain as depicted in Figure 1.

Figure 1.

Structure of the variational network (VN). The VN consists of T gradient descent steps. To obtain a reconstruction, we feed the undersampled k-space data, coil sensitivity maps and the zero filling solution to the VN. Here, a sample gradient step is depicted in detail. As we are dealing with complex-valued images, we learn separate filters $k_i^t$ for the real and complex plane. The non-linear activation function ${\varphi }_i^{tʹ}$ combines the filter responses of these two feature planes. During a training procedure, the filter kernels, activation functions and data term weights λ^t are learned.

METHODS

Variational Network Parameters

The VN defined by Eq. 6 and illustrated in Figure 1 contains a number of parameters: Filter kernels $k_i^t$, activation functions ${\mathbf{\Phi }}_i^{tʹ}$, and data term weights λ^t. We first consider the filter kernels which requires us to introduce a vectorized version ${\mathbf{k}}_i^t\in {\mathrm{ℝ}}^{2s^2}$ of the filter kernel $k_i^t$. We constrain the filters to be zero-mean which is defined as ${\mathbf{\xi }}_{\text{RE}}^{\top }{\mathbf{k}}_i^t=0$, ${\mathbf{\xi }}_{\text{IM}}^{\top }{\mathbf{k}}_i^t=0$, where ${\mathbf{\xi }}_{\text{RE}}^{\top }{\mathbf{k}}_i^t$, ${\mathbf{\xi }}_{\text{IM}}^{\top }{\mathbf{k}}_i^t$ estimate the individual means of the filter kernel on the real and imaginary plane, respectively. Additionally, the whole kernel is constrained to lie on the unit-sphere, i.e., ${\Vert {\mathbf{k}}_i^t\Vert }_2=1$, for to avoid a scaling problem of the activation functions. To learn the activation functions, we require a suitable function parametrization. A standard choice to smoothly approximate any functions are Gaussian radial basis functions (RBFs). We define the scalar activation functions ${\varphi }_i^{tʹ}$ as a weighted combination of N_w RBFs with nodes μ and standard deviation $\sigma =\frac{2I_{\text{MAX}}}{N_w-1}$

$${\varphi }_i^{tʹ}(z)={\displaystyle \sum _{j=1}^{N_w}{w}_{ij}^t\exp \left(-\frac{{(z-{\mu }_j)}^2}{2{\sigma }^2}\right)}.$$

The nodes are distributed in an equidistant way in [−I_MAX, I_MAX] which allows us to achieve the same resolution over the whole defined range. Note here that μ, σ depend on the maximum estimated filter response I_MAX. The final parameters that we consider are the data term weights λ^t, which are constrained to be non-negative (λ^t > 0). During training, all constraints on the parameters are realized based on projected gradient methods.

Variational Network Training

During the offline training procedure illustrated in Figure 2, the goal is to find an optimal parameter set θ = {θ⁰, …, θ^T−1}, ${\mathit{\theta }}^t=\{w_{ij}^t,{\mathbf{k}}_i^t,{\lambda }^t\}$ for our proposed VN in Eq. 6. To set up the training procedure, we minimize a loss function over a set of images S with respect to the parameters θ. The loss function defines the similarity between the reconstructed image u^T and a clean, artifact-free reference image g. A common choice for the loss function is the mean-squared error (MSE)

$$\mathcal{L}(\mathit{\theta })=\underset{\mathit{\theta }}{min}\frac{1}{2S}{\displaystyle \sum _{s=1}^S{\Vert {\mathbf{u}}_s^T(\mathit{\theta })-{\mathbf{g}}_s\Vert }_2^2}.$$

Figure 2.

Variational network training procedure: We aim at learning a set of parameters θ of the VN during an offline training procedure. For this purpose, we compare the current reconstruction of the VN to an artifact-free reference using a similarity measure. This gives us the reconstruction error which is propagated back to the VN to compute a new set of parameters.

As we are dealing with complex numbers in MRI reconstruction and we typically assess magnitude images, we define the MSE loss of (ε-smoothed) absolute values

$$\mathcal{L}(\mathit{\theta })=\underset{\mathit{\theta }}{min}\frac{1}{2S}{\displaystyle \sum _{s=1}^S{\Vert |{\mathbf{u}}_s^T(\mathit{\theta }){|}_{\epsilon }-|{\mathbf{g}}_s{|}_{\epsilon }\Vert }_2^2},|\mathbf{x}{|}_{\epsilon }=\sqrt{{\mathbf{x}}_{\text{RE}}^2+{\mathbf{x}}_{\text{IM}}^2+\epsilon }$$

where |·|_ε is understood in a point-wise manner. To solve this highly non-convex training problem, we use the Inertial Incremental Proximal Gradient (IIPG) optimizer which is related to the Inertial Proximal Alternating Linearized Minimization (IPALM) algorithm (31). For algorithmic details on IIPG refer to Appendix A and (32). First-order optimizers require both the loss function value and the gradient with respect to the parameters θ. This gradient can be computed by simple back-propagation (33), i.e., applying the chain rule

$$\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathit{\theta }}^t}=\frac{\partial {\mathbf{u}}^{t+1}}{\partial {\mathit{\theta }}^t}\cdot \frac{\partial {\mathbf{u}}^{t+2}}{\partial {\mathbf{u}}^{t+1}}\cdots \frac{\partial {\mathbf{u}}^T}{\partial {\mathbf{u}}^{T-1}}\cdot \frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathit{\theta }}^T}.$$

The derivation of the gradients for the parameters is provided in Appendix B. After training, the parameters θ are fixed and we can reconstruct previously unseen k-space data efficiently by forward-propagating the k-space data through the VN.

Data Acquisition

A major goal of our work was to explore the generalization potential of a learning-based approach for MRI reconstruction. For this purpose, we used a standard clinical knee protocol for data acquisition with a representative patient population that differed in terms of anatomy, pathology, gender, age and body mass index. The protocol consisted of five 2D turbo spin echo (TSE) sequences that differed in terms of contrast, orientation, matrix size and signal-to-noise ratio (SNR). For each sequence, we scanned 20 patients on a clinical 3T system (Siemens Magnetom Skyra) using an off-the-shelf 15-element knee coil. All data were acquired without acceleration, and undersampling was performed retrospectively as needed. In addition, we acquired prospectively accelerated data for one case. The number of acquired slices was chosen individually for each clinical patient exam. The study was approved by our institutional review board. Sequence parameters were as follows:

• Coronal proton-density (PD): TR=2750ms, TE=27ms, turbo factor/echo train length TF=4, matrix size 320 × 288, in-plane resolution 0.49 × 0.44mm², slice thickness 3mm, 35–42 slices, 5 female / 15 male, age 15–76, BMI 20.46-32.94

• Coronal fat-saturated PD: TR=2870ms, TE=33ms, TF=4, matrix size 320×288, in-plane resolution 0.49 × 0.44mm², slice thickness 3mm, 33–44 slices, 10 female / 10 male, age 30–80, BMI 19.76-33.87

• Axial fat-saturated T₂: TR=4000ms, TE=65ms, TF=9, matrix size 320 × 256, in-plane resolution 0.55 × 0.44mm², slice thickness 3mm, 33–41 slices, 10 female / 10 male, age 20–70, BMI 19.20-35.69

• Sagittal fat-saturated T₂: TR=4300ms, TE=50ms, TF=11, matrix size 320×256, in-plane resolution 0.55 × 0.44mm², slice thickness 3mm, 31–40 slices, 11 female / 9 male, age 12–73, BMI 18.16-37.31

• Sagittal PD: TR=2800ms, TE=27ms, TF=4, matrix size 384 × 307, in-plane resolution 0.46 × 0.36mm², slice thickness 3mm, 31–38 slices, 11 female / 9 male, age 15–94, BMI 18.69-35.15

Coil sensitivity maps were precomputed from a data block of size 24 × 24 at the center of k-space using ESPIRiT (34). For both training and quantitative evaluation, each network reconstruction was compared against a gold standard reference image. We defined this gold standard as the coil-sensitivity combined, fully sampled reconstruction. The fully sampled raw data were retrospectively undersampled for both training and testing.

Experimental Setup

Our experiments differed in contrast, orientation, acceleration factor and sampling pattern. For all our experiments, we pre-normalized the acquired k-space volumes with n_SL slices by $\frac{\sqrt{n_{\text{SL}}}10000}{{\Vert \mathbf{f}\Vert }_2}$. We trained an individual VN for each experiment and kept the network architecture fixed for all experiments. The VN consisted of T = 10 steps. The initial reconstruction u₀ was defined by the zero filled solution. In each iteration N_k = 48 real/imaginary filter pairs of size 11×11 were learned. For each of the N_k filters, the corresponding activation function was defined by N_w = 31 RBFs equally distributed between [−150,150]. Including the data term weight λ^t in each step, this resulted in a total of 131,050 network parameters.

For optimization, we used the IIPG optimizer described in Appendix A. The IIPG optimizer allows handling the previously described constraints on the network parameters. We generated a training set for each of the five knee datasets. In each experiment, we used 20 image slices from 10 patients with the same contrast weighting and orientation, which amounts to 200 images, as the training set. For each patient, the central 20 slices were used for training. In fact, each single pixel of these training images provides a training example. In the case of a 320×320 matrix, this results in more than 20 million pixels which is orders of magnitudes larger than the number of network parameters. The training set was split into mini batches of size 10. Optimization was performed for 1000 epochs with a step size of η = 10⁻³.

Experiments

In the first step, we investigated whether the learning-based VN approach actually benefits from structured undersampling artifacts due to regular undersampling, or if it performs better with incoherent undersampling artifacts as are typically present in CS applications. We used a regular sampling scheme with fully-sampled k-space center consisting of 24 auto-calibration lines, identical to the vendor implementation of an accelerated TSE sequence on an MR-system. To introduce randomness, we also generated a variable-density random sampling pattern according to Lustig et al. (16). Both sampling patterns have the same fully-sampled k-space center and same number of phase encoding steps. We evaluated the acceleration factors R ∈ {3, 4} for two sequences which differ in contrast and SNR. The second step was to explore the generalization potential with respect to different contrasts and orientations of a clinical knee protocol. In a third step, we performed an experiment with prospectively accelerated data.

Evaluation

We tested our algorithm on data from 10 clinical patients per sequence and reconstructed the whole imaged volume for each patient. These cases were not included in the training set, and they also contained pathology not represented in the training set. It is worth noting that the number of slices was different for each patient, depending on the individual optimization of the scan protocol by the MR technologist.

We compared our learning-based VN to the linear PI reconstruction method CG SENSE (12) and a combined PI-CS non-linear reconstruction method based on Total Generalized Variation (TGV) (22, 35). Additionally, we compared our qualitative results to dictionary learning (36) and provide quantitative measures for the selected cases. However, a full comparison to dictionary learning for all cases is out of scope of this work due to the long runtime requirements (approximately one hour per slice). The forward and adjoint operators for all three reference methods, in particular the coil sensitivity maps, were consistent with our VN approach. All hyper-parameters for CG SENSE and PI-CS TGV such as the number of iterations and regularization parameters were estimated individually by grid search for each sampling pattern, contrast and acceleration factor, such that the MSE of the reconstruction to the gold standard reconstruction was minimized. For dictionary learning, we used the standard parameters as in (36) and estimated the regularization parameter by grid search such that the MSE of the depicted slices was minimized. We assessed the reconstruction results quantitatively in terms of MSE, Normalized Root Mean Square Error (NRMSE), and Structural Similarity Index (SSIM) (37) with σ = 1.5 on the magnitude images.

In addition to the qualitative and quantitative evaluation, we performed a reader study that compared results from the proposed VN method with results from PI-CS TGV. The 50 test cases from all five sequences were independently reviewed by two fellowship trained musculoskeletal radiologists who were blinded to the MRI reconstruction method. Cases were reviewed in two different sessions, separated by 2 weeks to minimize recall bias. Each session consisted of a random selection of 25 learning and 25 TGV reconstructions. Using a 4-point ordinal scale, reconstructed images were evaluated for sharpness (1: no blurring, 2: mild blurring, 3: moderate blurring, 4: severe blurring), SNR (1: excellent, 2: good, 3: fair, 4: poor), presence of aliasing artifacts (1: none, 2: mild, 3: moderate, 4: severe) and overall image quality (1: excellent, 2: good, 3: fair, 4: poor). Comparisons in terms of image quality scores, averaged over the two readers, were made using a one-sided Wilcoxon signed-rank test. The null hypothesis that PI-CS TGV reconstruction results are equal or better than VN-based results is rejected at significance level α = 0.05 if the resulting P-value of the test is lower than the significance level α.

Implementation Details

The VN approach as well as the reference methods were implemented in C++/CUDA with CUDNN support. We provide Python and Matlab interfaces for testing. Experiments were performed on a system equipped with an Intel Xeon E5-2698 Central Processing Unit (CPU) (2.30GHz) and a single Nvidia Tesla M40 Graphics Processing Unit (GPU). For dictionary learning, we used the Matlab implementation provided by the authors (36) and extended their formulation to be used with our multi-coil sampling operator. This requires to solve Eqn. 7 in their work using the conjugate gradient method which additionally increases runtime. Source code and data are available online¹.

RESULTS

Retrospective Variational Network Reconstructions

Figure 3 display the impact of acceleration factor R = 4 and sampling patterns for CG SENSE, dictionary learning, PI-CS TGV and our learned VN on coronal PD-weighted images. Additionally, we plot zero filling solutions to illustrate the amount and structure of undersampling artifacts. Difference images to the reference are visualized in Figure 4. The reconstruction results for acceleration factor R = 3 along with the difference images are illustrated in Supporting Figure S2 and Supporting Figure S3. Residual artifacts and noise amplification can be observed for CG SENSE, in particular for R = 4. In case of acceleration factor R = 3, the PI-CS image appears less noisy than CG SENSE; however, similar undersampling artifacts are present. For R = 4 the PI-CS TGV result contains fewer undersampling artifacts than CG SENSE but small details in the image are already lost. Dictionary learning leads to improved removal of undersampling artifacts, resulting in a lower NRMSE than PI-CS TGV for this particular case. The learned VN suppresses these artifacts while still providing sharper and slightly more homogeneous images. Interestingly, dictionary learning as well as the PI-CS TGV and learned VN reconstruction with R = 3 regular sampling perform slightly better than with variable-density random sampling in terms of intensity homogeneity and sharpness. For acceleration R = 4, randomness improves the reconstruction results. We depict the reconstruction videos of the whole imaged volume of a 29-year-old female patient for acceleration factor R = 4 in Supporting Video S1 for regular sampling and in Supporting Video S2 for variable-density random sampling.

Figure 3.

Coronal PD-weighted scan with acceleration R = 4 of a 32-year-old male. The green bracket indicates osteoarthritis. The first and third row depict reconstruction results for regular Cartesian sampling, the second and fourth row depict the same for variable-density random sampling. Zoomed views show that the learned VN reconstruction appears slightly sharper than the PI-CS TGV and dictionary learning reconstruction. The dictionary learning and VN reconstruction can significantly suppress artifacts unlike CG SENSE and PI-CS TGV. Results based on random sampling show reduced residual artifacts and slightly increased sharpness in comparison to regular sampling.

Figure 4.

Difference images to reference image for the reconstructed coronal PD-weighted scans with acceleration R = 4 presented in Figure 3. The undersampling artifacts can be clearly observed in the CG SENSE and zero filling results. While TGV has a remaining undersampling artifact for regular sampling, the dictionary learning method can suppress this artifact. However, we observe larger errors at object boundaries in the dictionary learning results. The VN result has the least error compared to the reference methods.

Similar observations can be made for coronal PD-weighted scans with fat saturation, as depicted in Figure 5. Again, the reconstruction results for acceleration factor R = 3 along with the difference images are illustrated in Supporting Figure S4 and Supporting Figure S5. The main difference is that this sequence has a lower SNR compared to the non-fat-saturated version. Since additional noise reduces sparsity, the PI-CS TGV reconstructions produce an even more unnatural blocky pattern and contain substantial residual artifacts. The dictionary learning results appear blurrier at image edges and the general reconstruction quality is lowered at this level of SNR, which can best be seen in the error maps in Figure 6 and is supported by the quantitative values for this particular slice. Our learned VN is able to suppress these undersampling artifacts and shows improved image quality at this SNR level as well.

Figure 5.

Coronal fat-saturated PD-weighted scan with acceleration R = 4 of a 57-year-old female. The green bracket indicates broad-based, full-thickness chondral loss and a subchondral cystic change. The green arrow depicts an extruded and torn medial meniscus. The first and second row depict reconstruction results for regular Cartesian sampling, the third and fourth row depict the same for variable-density random sampling. The zoomed views show that the learned VN reconstruction appears sharper than the PI-CS TGV and dictionary learning reconstruction. The VN reconstruction shows reduced artifacts compared to the other methods. Results based on random sampling show reduced residual artifacts and appear sharper than the results based on regular sampling.

Figure 6.

Difference images to reference image for the reconstructed coronal fat-saturated PD-weighted scans with acceleration R = 4 presented in Figure 5. The undersampling artifacts can be clearly observed in the CG SENSE and zero filling results. Both PI-CS TGV and dictionary learning have residual undersampling artifact for regular sampling. We observe larger errors at object boundaries in the dictionary learning results. The VN result has the least error compared to the reference methods and is able to suppress the undersampling artifacts.

All our observations are supported by the quantitative evaluation depicted in Table 1 for R = 4 and in Supporting Table S1 for R = 3. The wide range in quantitative values over the different sequences illustrates the effect of SNR on the reconstructions. The learned VN reconstructions show superior performance in terms of MSE, NRMSE and SSIM in all cases. Table 1 and Supporting Table S1 also supports the qualitative impression that there is no improvement using variable-density random sampling for R = 3 for PI-CS TGV and VN reconstruction. In contrast, random sampling outperforms regular sampling for R = 4 in all coronal cases.

Table 1.

Quantitative evaluation results in terms of MSE, NRMSE and SSIM as well as image quality reader scores for a clinical knee protocol and acceleration factor R = 4 for regular sampling and variable-density random sampling. For the reader scores, we depict the mean values and standard deviations averaged over both readers along with the p-value obtained by the one-sided Wilcoxon signed-rank test. Values that accept the alternative hypothesis with a significance level α = 0.05, that VN reconstructions have a better quality score, are marked as bold.

Data set	Method	Regular			Random			Criterion	Reader scores regular		p-value
		MSE	NRMSE	SSIM in %	MSE	NRMSE	SSIM in %		PI-CS TGV	Learning
Coronal PD	Zero Filling	19.41±4.43	0.17±0.02	79.00±2.36	15.83±3.68	0.16±0.02	80.64±2.41	Artifact	3.60±0.57	1.65±0.07	0.0010
	CG SENSE	5.20±0.97	0.16±0.03	84.01±2.21	4.26±0.98	0.15±0.03	85.57±2.29	Sharpness/Blur	2.90±0.14	2.15±0.07	0.0234
	PI-CS TGV	2.35±0.40	0.09±0.02	89.80±1.75	1.91±0.45	0.09±0.02	90.36±1.79	SNR	2.60±0.28	1.45±0.21	0.0078
	Learning	1.64±0.28	0.08±0.02	92.14±1.68	1.37±0.32	0.08±0.02	92.86±1.63	Overall image quality	3.30±0.14	2.05±0.21	0.0010
Coronal fat-sat. PD	Zero Filling	20.71±14.07	0.23±0.03	73.96±3.04	17.69±3.30	0.22±0.03	75.10±3.17	Artifact	3.95±0.07	2.90±0.42	0.0020
	CG SENSE	14.55±1.62	0.25±0.05	73.06±4.62	11.79±1.39	0.24±0.04	74.78±4.55	Sharpness/Blur	3.95±0.07	3.15±0.64	0.0020
	PI-CS TGV	7.73±1.14	0.19±0.04	79.19±4.14	7.07±1.07	0.18±0.03	79.69±4.09	SNR	3.75±0.21	2.90±0.71	0.0049
	Learning	6.49±0.80	0.17±0.03	81.97±3.60	5.81± 0.85	0.17±0.03	82.47±3.67	Overall image quality	3.95±0.07	3.20±0.57	0.0020
Sagittal fat-sat. T2	Zero Filling	16.66±3.14	0.19±0.03	85.71±2.62	17.35±3.21	0.19±0.03	84.91±2.59	Artifact	2.90±0.14	2.80±0.28	0.3750
	CG SENSE	6.27±1.62	0.15±0.04	87.86±3.08	9.55±2.11	0.18±0.04	85.06±3.11	Sharpness/Blur	3.40±0.14	2.75±0.21	0.0156
	PI-CS TGV	3.39±0.82	0.11±0.03	91.84±2.81	4.76±0.95	0.13±0.03	90.29±2.70	SNR	3.20±0.28	2.50±0.28	0.0234
	Learning	2.99±0.68	0.11±0.03	92.83±2.40	3.92±0.81	0.12±0.03	91.85±2.35	Overall image quality	3.30±0.28	2.75±0.07	0.0078
Sagittal PD	Zero Filling	5.17±0.75	0.11±0.01	87.53±1.95	3.32±0.51	0.09±0.01	89.49±1.80	Artifact	2.10±0.14	2.00±0.14	0.4063
	CG SENSE	0.86±0.15	0.06±0.02	92.74±1.46	1.03±0.16	0.07±0.02	92.37±1.48	Sharpness/Blur	2.10±0.14	2.10±0.14	0.6875
	PI-CS TGV	0.49±0.09	0.05±0.01	96.22±1.17	0.64±0.11	0.05±0.01	95.47±1.24	SNR	1.60±0.00	1.50±0.28	0.3828
	Learning	0.44±0.07	0.04±0.01	96.64±1.16	0.52±0.09	0.05±0.01	96.07±1.17	Overall image quality	2.20±0.14	2.05±0.07	0.2656
Axial fat-sat. T2	Zero Filling	44.57±9.95	0.27±0.02	78.52±1.92	48.03±11.13	0.28±0.02	77.80±1.98	Artifact	3.15±0.07	3.10±0.57	0.5000
	CG SENSE	23.75±4.56	0.24±0.03	80.30±3.20	31.98±4.88	0.27±0.02	78.87±2.43	Sharpness/Blur	3.05±0.07	2.95±0.49	0.3750
	PI-CS TGV	13.65±3.78	0.18±0.03	85.51±3.25	15.30±2.57	0.19±0.02	84.93±2.60	SNR	3.10±0.14	2.75±0.49	0.0313
	Learning	10.63±2.48	0.16±0.02	88.46±2.43	12.06±2.13	0.17±0.02	87.74±2.30	Overall image quality	3.20±0.14	3.05±0.49	0.2266

We illustrate results for individual scans with regular sampling of R = 4 for a complete knee protocol, which contains various pathologies, taken from subjects ranging in age from 15 to 57, and anatomical variants, including a pediatric case. In particular, the coronal PD-weighted scan (M32) depicted in Figure 3 shows osteoarthritis, most advanced within the lateral tibiofemoral compartment with associated marginal osteophyte formation, indicated by the green bracket. An extruded and torn medial meniscus, indicated by the green arrow, is visible in the coronal fat-saturated PD-weighted scan in Figure 5. Additionally, this patient (F57) has broad-based, full-thickness chondral loss within the medial compartment and a subchondral cystic change underlying the medial tibial plateau, as indicated by the green bracket. Further results for different orientations and contrasts are illustrated in Figure 7 for regular sampling with R = 4 along with the error maps in Supporting Figure S6. The sagittal PD-weighted scan illustrate a skeletally immature patient (F15) with almost completely fused tibial physes. A partial tear of the posterior cruciate ligament is visible in the sagittal fat-saturated T₂-weighted scan M34. A full-thickness chondral defect centered in the medial femoral trochlea (green arrow) is visible on the axial fat-saturated T₂-weighted scan (F45) on a background of patellofemoral osteoarthritis. A reconstruction video of all available image slices for the axial fat-saturated T₂-weighted case is shown in Supporting Video S3.

Figure 7.

Reconstruction results for sagittal fat-saturated T₂-weighted, sagittal PD-weighted and axial fat-saturated T₂-weighted sequences of a complete knee protocol for acceleration factor R = 4 with regular undersampling. Each sequence here is illustrated with results from a different patient, identified by gender and age (e.g., M50 indicates a 50-year-old male). Pathological cases and a pediatric case are shown for both male and female patients of various ages. Green arrows and brackets indicate pathologies. Yellow arrows show residual artifacts that are visible in the different reconstructions, but not in the learned VN reconstructions.

The presence of these particular variations, which were not included in the training data set, does not negatively affect the learned reconstruction. The reduction of residual aliasing artifacts, marked by yellow arrows, the reduced noise level, and the overall improved image quality lead to improved depiction of the pathologies when compared to the reference methods. Again, the quality improvement of the learned VN is supported by the quantitative analysis of similarity measures depicted in Table 1 and Supporting Table S1.

Prospective Variational Network Reconstructions

The reconstruction results of prospectively undersampled data for regular sampling and acceleration R = 4 are depicted in Figure 8. We observe a similar behaviour of the reconstruction methods as for the retrospectively undersampled data. While PI-CS TGV and dictionary learning perform reasonably well for non-fat-saturated scans, a noise pattern can be observed in certain regions for dictionary learning and blocky appearance for PI-CS TGV. Our VN reconstructions are more homogeneous and less prone to remaining artifacts.

Figure 8.

Reconstruction results of prospectively undersampled data for regular sampling R = 4. We show reconstruction results for dictionary learning, PI-CS TGV and our VN for a whole knee protocol of a 27-year old female volunteer. We observe a similar behavior as for the retrospective undersampled data. Dictionary learning and PI-CS TGV perform reasonably well for non-fat-saturated scans. While the fat-saturated scans appear artificial with a PI-CS TGV reconstruction, we observe a noise pattern in the dictionary learning results, most prominent in the sagittal fat-saturated T₂-weighted scan. Dictionary learning appears slightly blurrier, which is best seen in the axial slice. The VN reconstructions have less undersampling artifacts and an improved SNR.

Reader Study

The average scores of the readers together with the P-values of the Wilcoxon signed-rank test are listed in Table 1. The mean values of the reader scores indicate that all VN reconstructions have equal or better scores than the PI-CS TGV reconstructions. P-values indicate that the null hypothesis is rejected for most of the sequences for the given significance level α. Coronal as well as sagittal T₂ VN reconstructions have significantly better image quality than PI-CS TGV. The difference between the individual reconstruction methods for the sagittal PD case is not significant, which is already obvious in the negligible difference of the qualitative results and quantitative results for this sequence. No significant difference in image quality, except SNR, can be observed for the axial T₂-weighted scans.

Variational Network Parameters

Examples of learned filter kernel pairs for real and imaginary feature planes are plotted along with their corresponding activation and potential functions in Figure 9. The potential functions are computed by integrating the learned activation functions, and they can be linked directly to the norms that are used in the regularization terms of traditional CS algorithms. We observe that same are very close to the convex l₁ norm used in CS (e.g., the function in the 2nd column), but we can also observe substantial deviations. We can identify functions with student-t characteristics and concave functions. Some of the learned filter pairs have the same structure in both the real and imaginary plane while some of them seem to be inverted in the real and imaginary part.

Figure 9.

Examples of learned parameters of the VN. Filter kernels for the real k_RE and imaginary k_IM plane as well as their corresponding activation ϕ′ and potential function ϕ are shown. The potential function ϕ was obtained by integrating the activation function ϕ′ including an additional integration constant.

DISCUSSION

While deep learning has resulted in clear breakthroughs in Computer Vision, the application of deep learning to medical image reconstruction is just beginning (38). Initial results for our deep learning image reconstruction approach presented in detail here were first presented at the Annual Meeting of the International Society for Magnetic Resonance in Medicine in May of 2016 (39). Early attempts to use machine learning for MRI reconstruction were based on dictionary learning (36,40,41). The key difference to our VN approach is that they learn a reconstruction online as a combination of dictionary elements directly from undersampled data, hence, no reference data is required. Although the learned dictionary might be reused, a new optimization problem has to be performed for every new reconstruction, which is computationally demanding. While dictionary learning methods act on patches, which need to be properly combined, and do not involve non-linearities in the combination of dictionary elements, our proposed VN approach directly reconstructs the whole images and learns non-linearities, which are important to enhance or suppress certain filter responses. Wang et al. (42) showed first results using a convolutional neural network (CNN) architecture to define a relationship between zero filled solution and high-quality images based on pseudo-random sampling. The learned network can then be used as regularization in a non-linear reconstruction algorithm. Yang et al. (43) introduced a network architecture that is based on unrolling the Alternating Direction Method of Multipliers algorithm. They proposed to learn all parameters including image transforms and shrinkage functions for CS-based MRI. Han et al. (44) learned destreaking on CT images and then fine-tuned the learning on MR data to remove streaking from radially undersampled k-space data. All three approaches used single-coil data, and it remains unclear how they deal with the complex domain of MR images. Kwon et al. (45) introduced a neural network architecture to estimate the unfolding of multi-coil Cartesian undersampled data. Similar to a classic SENSE reconstruction (12), unfolding is performed line-by-line. This restricts the applicability to a fixed matrix size and a particular 1D undersampling pattern. Most recently, Lee et al. (46) used residual learning to train two CNNs to estimate the magnitude and phase images of Cartesian undersampled data.

In this work, we present the first learning-based MRI reconstruction approach for clinical multi-coil data. Our VN architecture combines useful properties of two successful fields: variational methods and deep learning. We formulate image reconstruction as a variational model and embed this model in a gradient descent scheme, which forms the specific VN structure. The VN was first introduced as a trainable reaction-diffusion model (6) with application to classic image processing tasks (6,29,30). All these tasks are similar in the sense that the data are corrupted by unstructured noise in the image domain. MR image reconstruction presents several substantial differences: complex-valued multi-coil data are acquired in the Fourier domain and transformed into the image domain. This involves the use of coil sensitivity maps and causes distinct artifacts related to the sampling pattern. For our MR image reconstruction task, the optimal design of the VN, such as the number of stages, the number of filters per stage and the kernel size, is currently an open question. Our particular design choice is based on preliminary experiments (39) and, in line with the experiments presented here, delivered consistent results for a wide range of experimental conditions. We also found that the performance of our VN was stable when varying the design of the architecture. In practice, the design of the network is essentially a trade-off between model complexity and training efficiency. For example, the number of RBFs that are used to model the activation functions in a smoothed function approximation, defines the flexibility to approximate arbitrary functions in an accurate way. In our experimental setup as well as in the latest studies on image processing tasks (32), we reduced the number of RBFs compared to the initial work (6) by a half without a loss in performance but with reduced training time.

Our VN structure allows us to visualize the learned parameters, which is non-trivial for classical CNNs (47). In general, the filters in both the real and imaginary part represent different (higher-order) derivative filters of various scales and orientations, similar to Gabor filters (48, 49). Handcrafted Gabor filters have been successfully used in image processing (50), and learning-based approaches (3) report similar filters. It has also been shown that these types of filters have a strong relation to the human perceptual system (51).

Some of the learned potential functions in Figure 9 are very close to the convex l₁ norm used in CS (e.g., the function in the 2nd column), but we can also observe substantial deviations. We can identify functions with student-t characteristics also used in (28). Indeed, non-convex functions of student-t type introduce more sparsity than, e.g., the convex l₁-norm and are reported to fit the statistics of natural images better than the l₁-norm (52). Potential functions like those in columns 1, 4 and 7 have been associated with image sharpening in the literature (53).

Designing filters and functions is not a trivial task. Using learning-based approaches provides a way to tune these parameters such that they are adapted to specific types of image features and artifact properties. The strength of our algorithm are the trainable activation functions which stands in contrast to other deep learning approaches that use fixed activation functions such as Rectified Linear Units or sigmoid functions. Hence, instead of adding more and more layers and creating deeper networks, we introduce more structure and flexibility in the individual layers, which might help to reduce the overall complexity of the network. As shown in (32) for image denoising and non-blind deblurring, fixing the activation functions to less flexible, e.g., convex, functions might also lead to a decrease in performance for our application.

Compared to convex L1 minimization where we can understand the characteristics and artifacts of hand-crafted filters and potential functions, learning-based methods are often considered to be black-boxes, which are difficult to interpret. While we cannot claim insight into the properties of the model and the resulting images to the same degree of a simpler model like TV, one of the key strengths of our proposed VN is the motivation by a generalized, trainable variational model. To gain an understanding of what the VN learns, we first inspect the intermediate outputs of the gradient descent steps of our VN (see Supporting Video S4). We observe successive low-pass and high-pass filtering, and note that the prevalence of undersampling artifacts decreases after each single iteration. A continuous improvement over the iterations does not occur because our training is designed such that the result after the last gradient step is optimal in terms of the error metric chosen for evaluation. Although it would be possible to train the VN for progressive improvement, this would reduce the flexibility of the algorithm for adjusting the learned parameters during the training procedure.

In any iterative CS approach, every reconstruction is handled as an individual optimization problem. This is a fundamental difference to our proposed data-driven VN. In our VN approach, we perform the computationally expensive optimization as an offline pre-computation step to learn a set of parameters for a small fixed number of iterations. In our experiments, one training took approximately four days on a single graphics card. Once the VN is trained, the application to new data is extremely efficient, because no new optimization problem has to be solved and no additional parameters have to be selected. In our experiments, the VN reconstruction took only 193 ms for one slice. In comparison, the reconstruction time for zero filling was 11 ms, for CG SENSE with 6 iterations 75 ms and for PI-CS TGV with 1000 primal-dual iterations (22) 11.73 s on average. Thus, the online VN reconstruction using the learned parameters for the fixed number of iterations does not affect the hard time constraints during a patient exam.

Our VN is individually trained for different sampling patterns, reflected in the forward and adjoint operators. We do not learn a global mapping between undersampled k-space and reconstruction, but how to enhance local structures, while ensuring consistency to the acquired k-space data. First results towards learning a general regularizer, that could be applied for any sampling pattern, were recently presented at the annual meeting of ISMRM in 2017 (54): We showed that a network trained for regular sampling patterns can be used for reconstruction of randomly sampled data, but a network trained for randomly sampled data is not capable of removing coherent undersampling artifacts, which indicates that the dependency of sampling patterns is required to train the regularizer. However, the systematic performance evaluation for a wide range of sampling patterns is beyond the scope of this particular manuscript, and will be the target of future work. We will not only explore joint training of various sampling patterns, acceleration factors and sequences, but also the application of VN reconstruction to non-Cartesian sampling, dynamic and multi-parametric data.

The reconstruction quality of all methods does not only rely on the sampling pattern, but also on other parameters. Larger filter sizes, such as the 11 × 11 filters used in our VN architecture, provide the possibility to capture more efficiently the characteristic backfolding artifacts of Cartesian undersampled data, which are spread over several pixels. This stands in contrast to models like TV or TGV that are based on gradient filters in a small neighborhood (e.g., only forward differences in the x and y direction are considered). To suppress artifacts with PI-CS TGV, the regularization parameters must be chosen in such a way that the remaining image appears over-smoothed, and fine details are lost. Even though the piecewise-affine prior model of TGV is more complex than the piecewise-constant prior model of TV, the images appear artificial, especially if MR images with low SNR are reconstructed. Dictionary learning involves also larger filter kernels and works reasonably well for data with high SNR, reconstructions of low SNR data contain lots of noisy regions and blurry edges.

The image quality reader study confirms our quantitative and qualitative observations for regular sampling of R = 4. In general, the image quality of the fat-saturated sequences was rated lower than for the non-fat-saturated sequences for both VN and PI-CS TGV. The difference between the two types of sequences is the baseline SNR, which is much lower for the fat-saturated sequences. It is well known that in all CS-based methods, the best performance can be achieved in the case of a high baseline SNR and incoherent artifacts. The presented experiments demonstrate that if the corruption of the reconstructed images is dominated by noise, performance of both CS and VN reconstruction drops. If the baseline SNR drops to a level where the noise has a higher impact than aliasing artifacts, the VN concentrates on denoising instead of undersampling artifact removal. In addition, some of our results show residual artifacts, most prominent in the axial sequences. The source of these artifacts is residual aliasing and Gibbs’ ringing. These residual artifacts are present in all our reconstructions and not unique for our VN.

While radiologists learn throughout their career to distinguish certain patterns in images such as artifacts, we have to reflect the quality of learning in our presented approach by not only choosing the right architecture but also a suitable similarity measure. As demonstrated by our evaluation, quantitative scores are not always on par with image quality readings by radiologists. The used MSE for training compares pixel-wise differences and is likely not optimal for representing similarity to artifact-free reference reconstructions. Future investigations will also involve the choice of different error metrics or the investigation of generative adversarial networks (55) for training.

CONCLUSION

Inspired by variational models and deep learning, we present a new approach, termed VN, for efficient reconstruction of complex multi-coil MR data. We learn the whole reconstruction procedure and all associated model parameters in an offline training step on clinical patient data sets. The VN-based reconstructions preserve important features not presented in the training data. Our proposed learning-based VN reconstruction approach outperforms traditional reconstructions for a wide range of pathologies and offers high reconstruction speed, which is substantial for integration into clinical workflow.

Supplementary Material

Figure S1-6-Table-S1

S. Figure S1. Proposed image reconstruction pipeline: A zero filled solution is computed from the undersampled k-space data by applying the adjoint operator A*. The adjoint operator A* involves application of coil sensitivity maps. We feed the undersampled k-space data, coil sensitivity maps and the zero filling solution to the VN to obtain a reconstruction. For simplicity, we show the magnitude images, but all the input and output data of the VN are complex-valued.

S. Figure S2. Coronal PD-weighted scan with acceleration R = 3 of a 32-year-old male. The green bracket indicates osteoarthritis. The first and second row depict reconstruction results for regular Cartesian sampling, the third and fourth row depict the same for variable-density random sampling. Zoomed views show that the learned VN reconstruction appears slightly sharper than the PI-CS TGV reconstruction. Although dictionary learning can handle artifacts better than PI-CS TGV and produce a visually more appealing results, the quantitative values are slightly worse. For regular sampling, the results illustrate that the VN reconstruction can suppress undersampling artifacts better than CG SENSE and PI-CS TGV, and works on similar lines with dictionary learning. Quantitative values for VN out. For this acceleration factor of R = 3, the results based on random sampling appear slightly blurrier than the results based on regular sampling.

S. Figure S3. Difference images to reference image for the reconstructed coronal PD-weighted scans with acceleration R = 3 presented in Supporting Figure S2. The VN reconstructions show the least error compared to the other methods.

S. Figure S4. Coronal fat-saturated PD-weighted scan with acceleration R = 3 of a 57-year-old female. The green bracket indicates broad-based, full-thickness chondral loss and a subchondral cystic change. The green arrow depicts an extruded and torn medial meniscus. The first and second row depict reconstruction results for regular Cartesian sampling, the third and fourth row depict the same for variable-density random sampling. The zoomed views show that the learned VN reconstruction appears sharper than the PI-CS TGV and dictionary learning reconstruction. For regular sampling, the results illustrate that the VN reconstruction can suppress undersampling artifacts better. Again, results based on random sampling appear slightly blurrier than the results based on regular sampling.

S. Figure S5. Difference images to reference image for the reconstructed coronal fat-saturated PD-weighted scans with acceleration R = 3 presented in Supporting Figure S4. We observe large errors at boundaries for dictionary learning. The VN reconstructions show the least error compared to the other methods.

S. Figure S6. Difference images for sagittal fat-saturated T₂-weighted, sagittal PD-weighted and axial fat-saturated T₂-weighted sequences of a complete knee protocol presented in Figure 7.

S. Table S1. Quantitative evaluation results in terms of MSE, NRMSE and SSIM for a clinical knee protocol and acceleration factor R = 3 for regular sampling and variable-density random sampling.

Video S1

S. Video S1. Reconstruction of a complete imaged volume for a coronal PD-weighted sequence in a 50-year-old male, for regular sampling with acceleration R = 4.

Video S2

S. Video S2. Reconstruction of a complete imaged volume for a coronal PD-weighted sequence in the same 50-year-old male patient as in Supporting Video S1, for variable-density random sampling with acceleration R = 4.

Video S3

S. Video S3. Reconstruction of a complete imaged volume for an axial fat-saturated T₂-weighted sequence in a 45-year-old female patient, for regular sampling with acceleration R = 4.

Video S4

S. Video S4. Intermediate gradient step outputs of the reconstruction algorithm for a coronal PD-weighted slice with acceleration R = 4. We observe alternating low-pass and high-pass filtering over the intermediate steps. The undersampling artifacts are continuously suppressed until we obtain an artifact-free image after the final step.

APPENDIX A

Inertial Incremental Proximal Gradient Algorithm (IIPG)

For network training, we consider following optimization problem:

$$\mathcal{L}(\mathit{\theta })=\underset{\mathit{\theta }}{min}\frac{1}{2S}{\displaystyle \sum _{s=1}^S{\Vert {|{\mathbf{u}}_s^T(\mathit{\theta })|}_{\epsilon }-{|{\mathbf{g}}_s|}_{\epsilon }\Vert }_2^2}\mathit{\theta }=\{{\mathit{\theta }}^0,\dots ,{\mathit{\theta }}^{T-1}\},{\mathit{\theta }}^t=\{w_{ij}^t,{\mathbf{k}}_i^t,{\lambda }^t\}$$

$${\mathbf{u}}_s^{t+1}={\mathbf{u}}_s^t-{\displaystyle \sum _{i=1}^{N_k}{({\mathbf{K}}_i^t)}^{\top }{\mathbf{\Phi }}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}_s^t)-{\lambda }^t{\mathbf{A}}^{\ast }(\mathbf{A}{\mathbf{u}}_s^t-{\mathbf{f}}_s),0\le t\le T-1}$$

$$\mathrm{s}.\mathrm{t}.\mathit{\theta }\in \mathcal{C}=\{{\lambda }^t\ge 0,{\mathbf{\xi }}_{\text{RE}}^{\top }{\mathbf{k}}_i^t=0,{\mathbf{\xi }}_{\text{IM}}^{\top }{\mathbf{k}}_i^t=0,{\Vert {\mathbf{k}}_i^t\Vert }_2=1\}.$$

To solve this highly non-convex training problem, we use the Inertial Incremental Proximal Gradient (IIPG) optimizer. This IIPG variant of projected gradient descent is related to the Inertial Proximal Alternating Linearized Minimization (IPALM) algorithm (31). The whole sequence generated by IPALM is guaranteed to converge to a stationary point in the non-convex non-stochastic case under certain constraints on the step size and inertial parameters. The analysis for the stochastic version is left to future research. In the IIPG Algorithm 1, the parameter updates are calculated in a stochastic way on a single mini batch. First, we perform over-relaxation where we set an over-relaxation constant β_e dependent on the current epoch e to achieve moderate acceleration. Second, we compute the gradient with respect to the parameters on the current mini batch which yields a new parameter update ${\stackrel{\mathbf{\sim }}{\mathit{\theta }}}^{m+1}$ for the current iteration m. To realize additional constraints on the parameters, we finally perform the projections

$$\left({\lambda }^{m+1},{\mathbf{k}}^{m+1}\right)={\text{proj}}_{\mathcal{C}}^{\eta }\left({\stackrel{\sim }{\lambda }}^{m+1},{\stackrel{\mathbf{\sim }}{\mathbf{k}}}^{m+1}\right).$$

As the constraints do not depend on each other, we can consider the projections independently. To realize the non-negativity constraint on the data term weights λ^m+1, the parameter update ${\stackrel{\sim }{\lambda }}^{m+1}$ is clamped at zero

$${\lambda }^{m+1}=max(0,{\stackrel{\sim }{\lambda }}^{m+1}).$$

For the projection onto the filter kernel constraints, we first subtract the means ${\mathbf{\xi }}_{\text{RE}}^{\top }{\stackrel{\mathbf{\sim }}{\mathbf{k}}}^{m+1}$, ${\mathbf{\xi }}_{\text{IM}}^{\top }{\stackrel{\mathbf{\sim }}{\mathbf{k}}}^{m+1}$ from the current kernel parameter estimates and then project the kernel onto the unit-sphere

$$\begin{gathered} {\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\mathbf{\xi }}^{m+1}=({\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\mathbf{\xi },\text{RE}}^{m+1},{\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\mathbf{\xi },\text{IM}}^{m+1})=({\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\text{RE}}^{m+1}-{\mathbf{\xi }}_{\text{RE}}^{\top }{\stackrel{\mathbf{\sim }}{\mathbf{k}}}^{m+1},{\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\text{IM}}^{m+1}-{\mathbf{\xi }}_{\text{IM}}^{\top }{\stackrel{\mathbf{\sim }}{\mathbf{k}}}^{m+1}) \\ {\mathbf{k}}^{m+1}=\frac{{\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\mathbf{\xi }}^{m+1}}{{\Vert {\stackrel{\mathbf{\sim }}{\mathbf{k}}}_{\mathbf{\xi }}^{m+1}\Vert }_2}. \end{gathered}$$

Algorithm 1.

Inertial Incremental Proximal Gradient (IIPG) Algorithm

APPENDIX B

Gradient Derivation of Network Parameters

In every gradient step t, we seek the derivatives with respect to the parameters ${\mathit{\theta }}^t=\{w_{ij},{\mathbf{k}}_i^t,{\lambda }^t\}$ of the loss function

$$\mathcal{L}(\mathit{\theta })=\underset{\mathit{\theta }}{min}\frac{1}{2S}{\displaystyle \sum _{s=1}^S{\Vert {|{\mathbf{u}}_s^T(\mathit{\theta })|}_{\epsilon }-{|{\mathbf{g}}_s|}_{\epsilon }\Vert }_2^2,{|\mathbf{x}|}_{\epsilon }=\sqrt{{\mathbf{x}}_{\text{RE}}^2+{\mathbf{x}}_{\text{IM}}^2+\epsilon }}$$

where |·|_ε is understood in a point-wise manner. For simplicity, we drop the dependency of u^T on the parameters θ and the subscript s and show the calculations only for a single training example. The gradient steps are given as

$${\mathbf{u}}^{t+1}={\mathbf{u}}^t-{\displaystyle \sum _{i=1}^{N_k}{({\mathbf{K}}_i^t)}^{\top }{\mathbf{\Phi }}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)-{\lambda }^t{\mathbf{A}}^{\ast }(\mathbf{A}{\mathbf{u}}^t-\mathbf{f}),0\le t\le T-1}.$$

The derivatives with respect to the parameters θ^t are obtained by back-propagation (33)

$$\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathit{\theta }}^t}=\frac{\partial {\mathbf{u}}^{t+1}}{\partial {\mathit{\theta }}^t}·\underset{{\mathbf{e}}^{t+1}}{\underbrace{\frac{\partial {\mathbf{u}}^{t+2}}{\partial {\mathbf{u}}^{t+1}}\dots \frac{\partial {\mathbf{u}}^T}{\partial {\mathbf{u}}^{T-1}}·\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{u}}^T}}}.$$

The reconstruction error of the t-th gradient step is given by $\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{u}}^{t+1}}={\mathbf{e}}^{t+1}$.

Derivative of the Loss Function

First, we require the gradient of the loss function $\mathcal{L}$ with respect to the reconstruction u^T defined as e^T. It is computed as

$$\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{u}}^T}={\mathbf{e}}^T\iff e_l^T=\frac{u_l^T}{{|u_l^T|}_{\epsilon }}\left({|u_l^T|}_{\epsilon }-{|g_l|}_{\epsilon }\right),l=1,\dots ,N.$$

Derivative of the Data Term Weights λ^t

The derivative of the reconstruction u^t wrt. to λ^t ∈ ℝ for the t-th gradient step is expressed as:

$$\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\lambda }^t}=\frac{\partial {\mathbf{u}}^{t+1}}{\partial {\lambda }^t}\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{u}}^{t+1}}=\langle -({\mathbf{A}}^{\ast }(\mathbf{A}{\mathbf{u}}^t-\mathbf{f})),{\mathbf{e}}^{t+1}\rangle .$$

Derivative of the Activation Functions ${\mathbf{\Phi }}_i^{tʹ}$

A single activation function ${\mathbf{\Phi }}_i^{tʹ}(\mathbf{z})=({\varphi }_i^{tʹ}(z_1),\dots ,{\varphi }_i^{tʹ}(z_N)):{\mathrm{ℝ}}^N\mapsto {\mathrm{ℝ}}^N$ is defined by a weighted combination of N_w Gaussian radial basis functions:

$${\varphi }_i^{tʹ}(z_l)={\displaystyle \sum _{j=1}^{N_w}{w}_{ij}^t\exp \left(-\frac{{\left(z_l-{\mu }_j\right)}^2}{2{\sigma }^2}\right)},l=1,\dots ,N,w_{ij}^t\in \mathrm{ℝ}.$$

This can be rewritten in a matrix-vector notation:

$${\mathbf{\Phi }}_i^{tʹ}(z)=\left(\begin{array}{c}{\varphi }_i^{tʹ}(z_1)\\ ⋮\\ {\varphi }_i^{tʹ}(z_N)\end{array}\right)=\left[\begin{array}{ccc}\exp \left(-\frac{{(z_1-{\mu }_1)}^2}{2{\sigma }^2}\right)& \cdots & \exp \left(-\frac{{(z_1-{\mu }_{N_w})}^2}{2{\sigma }^2}\right)\\ ⋮& \ddots & ⋮\\ \exp \left(-\frac{{(z_N-{\mu }_1)}^2}{2{\sigma }^2}\right)& \cdots & \exp \left(-\frac{{(z_N-{\mu }_{N_w})}^2}{2{\sigma }^2}\right)\end{array}\right]\left(\begin{array}{c}{w}_{i1}^t\\ ⋮\\ w_{i{N}_w}^t\end{array}\right)={\mathbf{M}}_i^t(\mathbf{z}){\mathbf{w}}_i^t.$$

During training, we learn the weights ${\mathbf{w}}_i^t\in {\mathrm{ℝ}}^{N_w}$ and express its gradient as:

$$\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{w}}_i^t}=\frac{\partial {\mathbf{u}}^{t+1}}{\partial {\mathbf{w}}_i^t}\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{u}}^{t+1}}=-\frac{\partial }{\partial {\mathbf{w}}_i^t}\{{({\mathbf{K}}_i^t)}^{\top }{\mathbf{M}}_i^t({\mathbf{K}}_i^t{\mathbf{u}}^t){\mathbf{w}}_i^t\}{\mathbf{e}}^{t+1}=-{({\mathbf{M}}_i^t({\mathbf{K}}_i^t{\mathbf{u}}^t))}^{\top }{\mathbf{K}}_i^t{\mathbf{e}}^{t+1}.$$

Derivative of the Intermediate Reconstructions u^t

Further gradients with respect to the reconstructions from intermediate steps are given as:

$$\frac{\partial {\mathbf{u}}^{t+1}}{\partial {\mathbf{u}}^t}=I-{\displaystyle \sum _{i=1}^{N_k}{({\mathbf{K}}_i^t)}^{\top }\text{diag}({\mathbf{\Phi }}_i^{t^{″}}({\mathbf{K}}_i^t{\mathbf{u}}^t))}{\mathbf{K}}_i^t-{\lambda }^t{\mathbf{A}}^{\ast }\mathbf{A}$$

where I denotes the identity matrix. This also requires the second derivative of the potential functions ${\mathbf{\Phi }}_i^{t^{″}}(\mathbf{z})$, which is expressed as:

$${\mathbf{\Phi }}_i^{t^{″}}(\mathbf{z})=\left[\begin{array}{ccc}-\frac{(z_1-{\mu }_1)}{{\sigma }^2}\exp \left(-\frac{{(z_1-{\mu }_1)}^2}{2{\sigma }^2}\right)& \cdots & -\frac{(z_1-{\mu }_{N_w})}{{\sigma }^2}\exp \left(-\frac{{(z_1-{\mu }_{N_w})}^2}{2{\sigma }^2}\right)\\ ⋮& \ddots & ⋮\\ -\frac{(z_N-{\mu }_1)}{{\sigma }^2}\exp \left(-\frac{{(z_N-{\mu }_1)}^2}{2{\sigma }^2}\right)& \cdots & -\frac{(z_N-{\mu }_{N_w})}{{\sigma }^2}\exp \left(-\frac{{(z_N-{\mu }_{N_w})}^2}{2{\sigma }^2}\right)\end{array}\right]{\mathbf{w}}_i^t$$

Derivative of the Filter Kernels $k_i^t$

To compute the derivative with respect to the filter kernels $k_i^t$ we have to introduce further relationships between our given parameters. The convolution can be defined as matrix-vector multiplication:

$$k_i^t\ast u^t\iff {\mathbf{K}}_i^t{\mathbf{u}}^t={\mathbf{U}}^t{\mathbf{k}}_i^t$$

where the matrix ${\mathbf{U}}^t:{\mathrm{ℝ}}^{2s^2}\mapsto {\mathrm{ℝ}}^N$ is a suitably shifted representation of the image u^t and ${\mathbf{k}}_i^t\in {\mathrm{ℝ}}^{2s^2}$ is the vectorized filter kernel. The gradient step also involves rotated filter kernels ${\overline{k}}_i^t$ due to the transpose operation of the kernel matrix ${({\mathbf{K}}_i^t)}^{\top }$. As we want to calculate the derivative with respect to ${\mathbf{k}}_i^t$ and not to their rotated version, we introduce a rotation matrix $\mathbf{R}:{\mathrm{ℝ}}^{2s^2}\mapsto {\mathrm{ℝ}}^{2s^2}$ that has the same effect as the transpose operation

$${\overline{\mathbf{k}}}_i^t=\mathbf{R}{\mathbf{k}}_i^t.$$

The convolution can be rewritten as

$${({\mathbf{K}}_i^t)}^{\top }{\mathbf{\Phi }}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)={\stackrel{\mathbf{\sim }}{\mathbf{\Phi }}}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t){\overline{\mathbf{k}}}_i^t={\stackrel{\mathbf{\sim }}{\mathbf{\Phi }}}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)\mathbf{R}{\mathbf{k}}_i^t$$

where ${\stackrel{\mathbf{\sim }}{\mathbf{\Phi }}}_i^t({\mathbf{K}}_i^t{\mathbf{u}}^t):{\mathrm{ℝ}}^N\mapsto {\mathrm{ℝ}}^{2s^2}$ is a suitable matrix representation of ${\mathbf{\Phi }}_i^t({\mathbf{K}}_i^t{\mathbf{u}}^t)$. Applying the product rule yields following expression for the kernel derivative

$$\frac{\partial {({\mathbf{K}}_i^t)}^{\top }{\mathbf{\Phi }}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)}{\partial {\mathbf{k}}_i^t}=\frac{\partial {\mathbf{\Phi }}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)}{\partial {\mathbf{k}}_i^t}{\mathbf{K}}_i^t+\frac{\partial {\mathbf{k}}_i^t}{\partial {\mathbf{k}}_i^t}{\left[{\stackrel{\mathbf{\sim }}{\mathbf{\Phi }}}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)\mathbf{R}\right]}^{\top }={({\mathbf{U}}^t)}^{\top }\text{diag}\left({\mathbf{\Phi }}_i^{t^{″}}({\mathbf{K}}_i^t{\mathbf{u}}^t)\right){\mathbf{K}}_i^t+{\mathbf{R}}^{\top }{\stackrel{\mathbf{\sim }}{\mathbf{\Phi }}}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t).$$

The full derivative may be expressed as

$$\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{k}}_i^t}=\frac{\partial {\mathbf{u}}^{t+1}}{\partial {\mathbf{k}}_i^t}\frac{\partial \mathcal{L}(\mathit{\theta })}{\partial {\mathbf{u}}^{t+1}}=-\left[{({\mathbf{U}}^t)}^{\top }\text{diag}\left({\mathbf{\Phi }}_i^{t^{″}}({\mathbf{K}}_i^t{\mathbf{u}}^t)\right){\mathbf{K}}_i^t+{\mathbf{R}}^{\top }{\stackrel{\mathbf{\sim }}{\mathbf{\Phi }}}_i^{tʹ}({\mathbf{K}}_i^t{\mathbf{u}}^t)\right]{\mathbf{e}}^{t+1}.$$