ACCELERATING MAGNETIC RESONANCE IMAGING VIA DEEP LEARNING

Abstract

This paper proposes a deep learning approach for accelerating magnetic resonance imaging (MRI) using a large number of existing high quality MR images as the training datasets. An off-line convolutional neural network is designed and trained to identify the mapping relationship between the MR images obtained from zero-filled and fully-sampled k-space data. The network is not only capable of restoring fine structures and details but is also compatible with online constrained reconstruction methods. Experimental results on real MR data have shown encouraging performance of the proposed method for efficient and effective imaging.

1. INTRODUCTION

Magnetic resonance imaging (MRI) is an indispensable tool for medical diagnosis, disease staging and clinical research due to its strong capability in providing rich anatomical and functional information and its non-radiation and non-ionizing nature. However, most of advanced applications such as cardiovascular imaging, functional MRI, magnetic resonance spectroscopy and parameter mapping are not yet widely used in clinic due to the long scanning time of MRI [1, 2].

To accelerate MR scans, efforts are mainly in three directions 1) physics based fast imaging sequences, 2) hardware based parallel imaging and 3) signal processing based MR image reconstruction from reduced samples. The combination of these techniques have also shown their appearance in a great number of publications [2, 3, 4, 5, 6]. The first two categories and a few techniques of the third category (e.g. partial Fourier) have already been applied in commercial scanners as a routine protocol for shortening the total scan time [2, 3, 4, 5, 6]. Signal processing based methods, explores prior information on MR images and utilize them to regularize the reconstruction from undersampled K-space measurements with the advantage of no physical, physiological and hardware restrictions. Sparsity is one commonly used prior information due to the emergence of Compressed sensing (C-S) and there are also other priors being considered, such as low-rank [7], statistics distribution regularization [8], manifold fitting [9], generalized series (GS) model [10] and so on. The prior information used can be roughly categorized into adaptive and non-adaptive ones. For example, total variation and Wavelet transform, singular value decomposition (SVD) are non-adaptive ones [3, 5, 4]; dictionary learning and data driven tight frames are adaptive [11, 12, 13, 14]. Generally, adaptive priors can capture more structures while non adaptive ones are more computationally efficient. Nevertheless, despite all the successes achieved by the aforementioned methods, it is easy to discover that they only explore the prior information either directly from the image to be reconstructed or with very few reference images involved. Considering the similarity on the anatomic information of the same organ/tissue between different people and the enormous images acquired every day, it is straightforward to collect many reference images and learn an off-line prior model to aid online fast imaging.

Deep learning, a technique attempting to model highlevel abstractions in data with multiple processing layers, has shown explosive popularity in recent years with the availability of powerful GPUs. Especially, convolutional neural network (CNN) has exhibited its significance in addressing large-scale vision tasks such as action recognition [15], image classification [16], super-resolution [17] and denoising [18]. CNNs have quite a few merits, such as the lack of dependence on prior-knowledge, no need to design hand-engineered features and strong ability to capture image structures, which motivated us to employ it for MR image reconstruction from undersampled k-space data. What is more important is that the relationship between the zero-filled MR image and the ground-truth image can be interpreted in a convolutional way, which will be explained in the theory parts.

In this paper we propose an off-line convolutional neural network to learn an end-to-end mapping between zero-filled and fully-sampled MR images. This network is not only capable of restoring the details and fine structures of the MR images, but is also compatible with any online reconstruction algorithm for more efficient and effective imaging. We have tested the proposed method on a set of in-vivo MR data and the results have shown promising.

2. THEORY

This section provides the concept and theoretical foundation for learning the convolutional neural network for undersampled MR image reconstruction. Fig. 1 presents the flowchart of the proposed method.

Fig. 1.

The flowchart of the proposed method

2.1. Overall Training Formulation

Consider the undersampled raw K-space data as

$$f=PFu$$ (1)

where P is a diagonal matrix representing the undersampling mask, F denotes the full Fourier encoding matrix normalized as F^HF = I, u is the original (ground truth) image and therefore Fu represents the full k-space data. H represents the Hermitian transpose operation. The zero-filled MR image z is generated as the direct inverse transform of the observed data like

$$z=F^{H}PFu$$ (2)

As stated in [19], in terms of linear algebra, the circular convolution of a signal u with a pulse p can be written as F^H PFu, where P is a diagonal matrix whose non-zero entries are the Fourier transform of p.

We try to learn a fully convolutional neural network to restore accurate MR images from undersampled Fourier data. Given a pre-acquired dataset of MR corrupted/ground truth images, we try to minimize the following objective

$$\underset{\Theta }{\text{argmin}}\left\{\frac{1}{2\text{T}}\sum _{t=1}^T{\Vert C\left(z_t;\Theta \right)-u_t\Vert }_2^2\right\}$$ (3)

where C means the end-to-end mapping function with its hidden parameters Θ ={(W₁, b₁), …, (W_l, b_l), …, (W_L, b_L)} to be estimated and T is the total number of training samples. In order to increase the robustness of the network, we generate more training samples by separating the whole image pairs into overlapping subimage pairs x_t,n and y_t,n and minimize

$$\underset{\Theta }{\text{argmin}}\left\{\frac{1}{2\text{TN}}\sum _{t=1}^T\sum _{n=1}^N{\Vert C\left(x_{t,n};\Theta \right)-y_{t,n}\Vert }_2^2\right\}$$ (4)

For the simplicity of explanation, we only consider one pair x and y in the following demonstration.

2.2. Forward-pass training subproblems

2.2.1. Feature generation

Unlike sparse representation, where each extracted image patch is approximated by a set of pre-trained bases, we use the equivalent convolution operation [17] and transfer the optimization of the bases into the network learning process. Therefore, the first layer of network can be described as follows

$$C_1=\sigma \left(W_1\ast x+b_1\right)$$ (5)

where W₁ denotes the convolution operator of size c × M₁ × M₁×n₁ and b₁ is the n₁ dimensional bias with its element associated with a filter. Here, c is the number of the image channels, M₁ means the filtered size and n₁ is the number of filters. We adopt the rectified linear unit (ReLu, max(0, x)) here for the nonlinear responses, which can be computed very efficiently [17].

2.2.2. Nonlinear mapping

We further perform non-linear mapping to project the n_l−1 dimensional vectors into an n_l one, which is conceptually the refined feature and structure to represent the full-data-reconstructed image

$$C_l=\sigma \left(W_l\ast C_{l-1}+b_l\right)$$ (6)

where W_l is of a size n_l−1 × M_l × M_l × n_l.

2.2.3. Last Layer convolution

To produce the final predicted image from CNN, we explore another layer of convolution and hope to learn a set of linear filters W_L which are capable of projecting the coefficients onto the image domain

$$C_L=\sigma \left(W_L\ast C_{L-1}+b_L\right)$$ (7)

where W_L is of a size n_L−1 ×M_L ×M_L ×c. To sum up, we have designed an L-layer convolutional neural network to learn the mapping relationship:

$$\{\begin{array}{l}{C}_0=x\\ C_l=\sigma \left(W_l\ast C_{l-1}+b_l\right),l\in 1,2,\mathrm{\dots },L-1\\ C_L=\sigma \left(W_L\ast C_{L-1}+b_L\right)\end{array}$$ (8)

2.3. Backward Propagation

Given the training pair (x,y), the forward pass Eqs. (5–8) computes the activations and output values. To update the network parameters, as [16] we perform backward propagation to calculate the related gradients. Staring with the single pair objective, Eq. (4) can be written as

$$J\left(\Theta \right)=\underset{\Theta }{\text{argmin}}\left\{\frac{1}{2}{\Vert C\left(x;\Theta \right)-y\Vert }_2^2\right\}$$ (9)

Let D_l = W_l∗C_l−1 +b_l and δ^l denote the” error term” propagated backwards. We first calculate the last layer gradient

$${\delta }^L=\frac{\partial J}{\partial b_L}=\frac{\partial J}{\partial D_L}\frac{\partial D_L}{\partial b_L}=C_L-y$$ (10)

Since $\frac{\partial D_l}{\partial b_l}=1$ and C_l = σ(D_l), δ^l of nonlinear mapping layer can be updated as follows

$${\delta }^l=\frac{\partial J}{\partial b_l}=\frac{\partial J}{\partial D_{l+1}}\frac{\partial D_{l+1}}{\partial C_l}\frac{\partial C_l}{\partial D_l}=\left({\delta }^{l+1}\star W^{l+1}\right)\circ \frac{\partial \left(D^l\right)}{\partial D^l}$$ (11)

where ★ means the cross-correlation operation which is different from the convolution in the feed-forward pass and ◦ denotes element-wise multiplication. Therefore, we can obtain the gradients for each layer

$$\{\begin{array}{l}\frac{\partial J}{\partial W_l}=\frac{\partial J}{\partial D^l}\frac{\partial D^l}{\partial W^l}={\delta }^l\star D^{l-1}\\ \frac{\partial J}{\partial b_l}=\frac{\partial J}{\partial D^l}\frac{\partial D^l}{\partial b^l}=\frac{\partial J}{\partial D^l}={\delta }^l\end{array}$$ (12)

which can be used to calculate the stochastic gradients $\frac{\partial J\left(\Theta \right)}{\partial \Theta }$ during the training stage.

2.4. MR reconstruction formulation

Once we learned the hidden parameters $\widehat{\Theta }$ from the pre-acquired datasets, we can reconstruct MR images by considering the following constrained optimization problem

$$\underset{u}{\text{argmin}}\left\{{\Vert C\left(F^{H}f;\widehat{\Theta }\right)-u\Vert }_2^2+\lambda {\Vert f-F_{M}u\Vert }_2^2\right\}$$ (13)

As can be seen, this is a simple least squares problem admitting an analytical solution. And the least square solution satisfies the normal equation

$$\left(1+\lambda F_M^H{F}_M\right)u=C\left(F^{H}f;\widehat{\Theta }\right)+\lambda F_M^{H}f$$ (14)

By transforming the equation from image space to Fourier space, we have

$$\left(1+\lambda F{F}_M^H{F}_M{F}^H\right)Fu=FC\left(F^{H}f;\widehat{\Theta }\right)+\lambda F{F}_M^{H}f$$ (15)

where $F{F}_M^H{F}_M{F}^H$ is a diagonal matrix consisting of ones and zeros. The ones are diagonal entries that correspond to the sampled locations in k-space. $F{F}_M^{H}f$ means zero-filled Fourier measurements. Therefore, we have

$$Fu\left(k_x,k_y\right)=\{\begin{array}{cc}S\left(k_x,k_y\right)& ,\hspace{0.17em}\text{if}\left(k_x,k_y\right)\notin \Omega \\ \frac{S\left(k_x,k_y\right)+\lambda S_0\left(k_x,k_y\right)}{1+\lambda }& ,\hspace{0.17em}\text{if}\left(k_x,k_y\right)\in \Omega \end{array}$$ (16)

where Ω is the sampled location set.

2.5. Combination with CS-MRI reconstruction methods.

Besides the simple reconstruction model, we also provide two options for the integration with CS-MRI methods. a) Sequential model: Two-phase CS-MRI reconstruction. At the first stage, generate $C\left(F^{H}f;\widehat{\Theta }\right)$ from the learned network. At the second stage, initialize CS-MRI with $C\left(F^{H}f;\widehat{\Theta }\right)$ and then reconstruct MR images with CS-MRI. b) Integration model: Use the image generated by the network as a reference image and use it as additional regularization term.

$$\underset{u}{\text{argmin}}\left\{{\Vert C\left(F^{H}f;\widehat{\Theta }\right)-u\Vert }_2^2+\lambda {\Vert f-F_{M}u\Vert }_2^2+\beta Reg\left(u\right)\right\}$$ (17)

where Reg(u) is the sparse regularization term.

3. EXPERIMENTS AND RESULTS

Datasets:

The training data consists of over 500 fully sampled MR brain images we collected from a 3T scanner (SIEMENS MAGNETOM TrioTim). The images are of a great diversity including axial, sagittal, horizontal ones, different contrast ones such as T1, T2 and PD-weighted images and of a variety of sizes. Informed consent was obtained from the imaging subject in compliance with the Institutional Review Board policy. Undersampled measurements were retrospectively obtained using the 1D low-frequency sampling mask and the 2D Poisson disk sampling mask. The large amount of corrupted/ground truth subimage pairs are then generated with the size of 33 × 33. Finally we use 90% of the subimage pairs as the training dataset and the rest 10% for validating the training process.

Implementation details:

We use three layers of convolution for the network. The parameters are respectively set as n₁ = 64, n₂ = 32, M₁ = 9, M₂ = 5 and M₃ = 5. The filter weights of each layers are initialized by random values from a Gaussian distribution with zero mean and standard deviation 0.001. The bias are all initialized as 0. The training takes about three days, on a workstation equipped with 128G memory and a processor of 16 cores (Intel Xeon (R) CPU E5–2680 V3 @2.5GHz).

Figure 2 shows a set of reconstruction results of a transversal brain image. The brain dataset was obtained fully-sampled with 12-channel head coil and T2-weighted turbo spin-echo (TSE) sequence (TE = 91.0ms, TR = 5000ms, FOV = 20 × 20cm, matrix = 256 × 270, slice thickness = 3mm) via 3T scanner. And the data was then undersampled retrospectively with 1D low-frequency sampling mask at an acceleration factor of 3 and the 2D Poisson disk at an acceleration factor of 5. We also tested the proposed method on a sagittal brain image which was acquired on a GE 3T scanner (GE Healthcare, Waukesha, WI) with a 32-channel head coil and 3D T1-weighted spoiled gradient echo sequence (TE=minimum full, TR= 7.5ms, FOV=24 × 24 cm, matrix = 256 × 256, slice thickness=1.7mm). We can observe from the images that there are quite a few details and structures captured by the network. Furthermore, the image generated by the simple reconstruction model is quite close to the original image. According to Fig. 3f, we can see the difference image is noise-like and consists only the contour information. There are no obvious details and structures lost. It demonstrates that the proposed network is capable of restoring the details and fine structures which are discarded in the zero-filled MR image. Furthermore, although the offline training takes roughly three days, under the same GPU configurations, it takes far less than 1 second for each online MR reconstruction case.

Fig. 2.

(a) Ground truth image; (b) 1 D central low-frequency sampling mask with acceleration factor of 3; (c) 2D poisson undersampling mask with acceleration factor of 5; (d)(e)(f) the zero-filled MR image, network output and reconstruction result from 1D undersampled data; (g)(h)(i) the zero-filled M-R image, network output and reconstruction result from 2D undersampled data.

Fig. 3.

The test results on another sagittal brain image at an acceleration factor of 3

4. CONCLUSIONS

An off-line convolutional neural network for accelerating M-R imaging is proposed in this paper, which includes the brief review and discussion of the concept, theoretical foundation, implementation and application of this network for undersampled MR image reconstruction. The experimental results on in-vivo MR images have shown the proposed network is capable of restoring the details and fine structures that are lost in the zero-filled MR image. We also provide two options for combing the proposed network with online CS-MRI methods for more efficient and effective imaging. More extensive experimental results will be provided in the future journal paper.