DEEP FACTOR MODEL: A NOVEL APPROACH FOR MOTION COMPENSATED MULTI-DIMENSIONAL MRI

Abstract

Recent quantitative parameter mapping methods including MR fingerprinting (MRF) collect a time series of images that capture the evolution of magnetization. The focus of this work is to introduce a novel approach termed as Deep Factor Model(DFM), which offers an efficient representation of the multi-contrast image time series. The higher efficiency of the representation enables the acquisition of the images in a highly undersampled fashion, which translates to reduced scan time in 3D high-resolution multi-contrast applications. The approach integrates motion estimation and compensation, making the approach robust to subject motion during the scan.

1. INTRODUCTION

The signal in MRI is sensitive to multiple physical properties of the tissue, which is exploited to observe organs with different contrast mechanisms. The classical MRI methods approach is to carefully choose the acquisition parameters to make the images sensitive to a specific physical property while being less sensitive to others. For example, the magnetization prepared rapid gradient echo (MPRAGE)[1] sequence uses inversion pulses and restricts the acquisition of the data at specific inversion times to obtain good contrast between the gray matter (GM) and white matter (WM) regions. However, a challenge with this approach is its low acquisition efficiency, resulting from the long waiting times and delay times to allow the magnetization to recover. The MPnRAGE[2] sequence was introduced recently, which relies on continuously acquiring the images using radial acquisitions after the inversion pulse; the radial data is binned depending on the inversion time and recovered using gridding. In addition to improving acquisition efficiency, this approach enables the acquisition of images at multiple inversion times.

The MPnRAGE approach has conceptual similarities to MRF [3] that continuously acquires the data using incoherent sampling patterns, while magnetization is continuously evolving. MRF attempts to estimate the physical parameters (e.g. $T_1,T_2$) directly from the undersampled k-t space data using pattern matching. These methods rely on large dictionaries of MRF derived using Bloch equation simulations, which correspond to different values of the physical parameters. Several constrained reconstruction algorithms were introduced to improve the reconstruction of the individual frames in MRF; the strategies include low-rank factor modeling of the reconstructed images [4].

The main focus of this paper is to introduce DFM for the joint recovery of inversion recovery data. The proposed DFM, illustrated in Fig. 1 (b), is a generalization of the low-rank model. It capitalizes on the benefits of convolutional neural networks (CNN) in representing images. While unrolled algorithms [5] were introduced to improve low-rank or subspace models, these methods need multiple copies of the subspace images at each unrolling step; the extremely high memory demand of these approaches makes them infeasible in our 3D setting.

Fig. 1.

Illustration of the low-rank model and DFM. (a) The low-rank method represents the image corresponding to the delay $\tau$ as the weighted linear combination of the spatial basis functions $\mathbf{\text{U}}\left(r\right)$ based on weights $\mathbf{\text{V}}\left(\tau \right)$; (b) DFM models the signal $\rho \left(\mathbf{\text{r}},\tau \right)$ as the output of a conditional network ${𝒩}_{\varphi ,\theta }$. The features of the CNN ${𝒰}_{\varphi }$ are modulated by $v\left(\tau \right)$ generated by a dense network ${𝒱}_{\theta }$. The input to ${𝒱}_{\theta }$ is $\tau$, while that of the ${𝒰}_{\varphi }$ is an approximate initialization $\gamma$ obtained from gridding.

The DFM represents each image in the time series as the output of a conditional CNN. The input of the CNN is a series of representative image volumes corresponding to a few inversion delays obtained from gridding. The inversion time-dependent network derives each image in the time series as a denoised non-linear function of the gridded images. The temporal factors derived from a dense network are used to modulate the features of the CNN. Unlike supervised deep learning models, DFM is an unsupervised learning approach that directly learns the subject-specific representation from the measured k-t space data. This approach of learning the network parameters from the undersampled measurements is motivated by [6], which learns an image or an ensemble of images from undersampled measurements. We hypothesize that non-linear deep representation is more efficient that the linear low-rank factor models, and offers implicit spatial regularization due to the implicit bias of CNN blocks towards images.

We also capitalize on the unsupervised learning strategy to compensate for subject motion. In particular, we model the subject motion during the scan as a rigid body. We use the relation between rigid body motions and corresponding transformations in the Fourier domain to absorb them into the forward model. These time-varying parameters are assumed to be unknowns and are solved during signal recovery.

The data is acquired using a radial ultrashort echo time (UTE) sequence with intermittent inversion pulses and a delay time for magnetization recovery. We acquire the data from 3T scanners, with and without motion.

2. PROPOSED APPROACH

2.1. Signal Acquisition and Modeling

The acquisition of data is illustrated in Fig. 2. We assume that the evolution of magnetization during each inversion block is identical, and one initial inversion and segmented UTE scheme is performed to reduce steady-state effects. We consider the image volume ${\rho }_{\tau }$ to be dependent on the delay $\tau \left(t\right)$ from the previous inversion pulse.

Fig. 2.

Illustration of the pulse sequence. The data is acquired using an inversion recovery with a segmented readout consisting of a train of UTE 3D radial acquisitions with $TR=4.4$ ms. Intermittent adiabatic inversion pulses are applied after 800 UTE radial lines, with a delay of 500 ms after each segmented acquisition block to allow for signal recovery to a steady state. The total acquisition time was 4.3 minutes to acquire 50K spokes with a matrix size of 256³. The radial spokes are ordered according to the tiny golden angle view order. The field of view was 24cm³.

We model the signal measurements at the time instant

$${\mathbf{\text{y}}}_t={\mathbf{\text{A}}}_t\left({\rho }_{\tau \left(t\right)}\right)+{\mathbf{\text{n}}}_t$$ (1)

Here, ${\mathbf{\text{A}}}_t$ denotes the forward model that accounts for multichannel Fourier measurements and the k-space trajectory at time $t$, which is measured from the start of the acquisition. In the absence of motion, the measurements $y_t$ corresponding to a specific $\tau$ value can be pooled together.

2.2. Deep Factor Model

We note that low-rank factor models represent the signal as

$$\rho \left(\mathbf{\text{r}},\tau \right)=\mathbf{\text{U}}\left(\mathbf{\text{r}}\right)\mathbf{\text{V}}\left(\tau \right),$$ (2)

where the columns of $\mathbf{\text{U}}\left(\mathbf{\text{r}}\right)$ are the spatial factors and the rows of $\mathbf{\text{V}}\left(\tau \right)$ are the temporal factors. The spatial basis functions are modulated by the temporal basis functions to generate the image at a specific time $\tau$. Note that low-rank factor models do not use any spatial regularization on the spatial basis functions $\mathbf{\text{U}}\left(\mathbf{\text{r}}\right)$.

We model the image ${\rho }_{\tau }$ as the output of a deep conditional network ${𝒩}_{\varphi ,\theta }$

$$\rho \left(\mathbf{\text{r}},\tau \right)={𝒩}_{\varphi ,\theta }\left(\gamma ,\tau \right)$$ (3)

$\varphi$ and $\theta$ denote the parameters of $𝒩$, while $\tau$ denotes the inversion delay. $\gamma$ is a coarse initial reconstruction.

We use a deep factor architecture to realize the conditional model as shown in Fig. 1 In particular, we feed delay time $\tau$ to a dense network ${𝒱}_{\theta }$, which provides the temporal factors

$$\mathbf{\text{v}}\left(\tau \right)={𝒱}_{\theta }\left(\tau \right).$$ (4)

We apply channel-wise multiplication between the feature maps and corresponding temporal factors to modulate the feature maps of ${𝒰}_{\varphi }$ and obtain

$$\rho \left(\mathbf{\text{r}},\tau \right)={𝒰}_{\varphi }\left(\gamma \left(\mathbf{\text{r}}\right),\mathbf{\text{v}}\left(\tau \right)\right)$$ (5)

One may view this approach as selectively activating and suppressing specific channels, based on the delay $\tau$. If the network ${𝒰}_{\varphi }$. only consists of a single hidden layer, this approach reduces to the traditional low-rank model. We expect the deep factorization at different layers to offer improved representation power. The unknowns of (3) are $\varphi$ and $\theta$. In the absence of motion, we pose the reconstction as

$$\left\{{\theta }^{*},{\varphi }^{*}\right\}=\text{arg\hspace{0.17em}}\underset{\theta ,\varphi }{\text{min}}\sum _t{\Vert {\mathbf{\text{y}}}_t-{\mathbf{\text{A}}}_t\left({𝒩}_{\theta ,\varphi }\left(\gamma ,\tau \right)\right)\Vert }^2$$ (6)

2.3. Deep Factor Model with Motion Compensation

We assume the brain to be a rigid body and model any subject motion using 3 rotations and 3 translations. The object at time instant $t$ is assumed to be a rotated and translated version of ${\rho }_{\tau \left(t\right)}$. We denote the transformation operator as $T_{{\nu }_t,{\delta }_t}$, where ${\nu }_t$ and ${\delta }_t$ are the rotation and translation parameters at time $t$. In particular,

$$T_{\nu ,\delta }\left(\rho \left(\mathbf{\text{r}}\right)\right)=\rho \left(R_{\nu }\mathbf{\text{r}}+\delta \right)$$ (7)

where $\mathbf{\text{r}}$ is the coordinate in the image domain. With this assumption, (1) changes as

$${\mathbf{\text{y}}}_t={\mathbf{\text{A}}}_t\left(T_{{\nu }_t,{\delta }_t}\left({\rho }_{\tau \left(t\right)}\right)\right)+{\mathbf{\text{n}}}_t$$ (8)

We note that a rotation in the image domain by an angle ${\nu }_t$ corresponds to a rotation by the same angle in the Fourier domain. This amounts to rotating the k-space trajectory by the same angle. Similarly, a translation in the image domain corresponds to a phase modulation in the Fourier domain. Using these Fourier properties, we rewrite the above relation as

$${\mathbf{\text{y}}}_t={\mathbf{\text{A}}}_{t,{\nu }_t,{\delta }_t}\left({\rho }_{\tau \left(t\right)}\right)+{\mathbf{\text{n}}}_t$$ (9)

This reformulation allows us to absorb the impact of motion into the forward model and further extend the proposed method to DFM with motion compensation(DFM-MC). In the presence of motion, we use (9) to minimize the cost function

$$C(\theta ,\varphi ,\nu ,\delta \}=\sum _t{\Vert {\mathbf{\text{y}}}_t-{\mathbf{\text{A}}}_{t,{\nu }_t,{\delta }_t}\left({𝒩}_{\theta ,\varphi }\left(\gamma ,\tau \right)\right)\Vert }^2+{\lambda }_1{\Vert \nabla {\nu }_t\Vert }^2+{\lambda }_2{\Vert \nabla {\delta }_t\Vert }^2$$ (10)

with respect to the network parameters $\theta ,\varphi$ as well as the motion parameters $\nu ,\delta$.

3. IMPLEMENTATION DETAILS

The data were acquired from two human volunteers on a 3.0T GE Premier scanner with 48 channel head coil. From each subject, we acquired two datasets. During the first acquisition, the subjects were instructed to stay still during the 4.3-minute acquisition. During the second acquisition, the subject was instructed to move the head multiple times during the scan. We implemented $\mathbf{\text{A}}$ using a multichannel NUFFT operator, which was differentiable with respect to the input image as well as the trajectory parameters. Post recovery, we fed the network $𝒩$ with different delays $\tau$ to generate the images. We used principal component analysis (PCA) to reduce the 48 coils to 10 virtual coils.

3.1. Deep Factor Model

We implemented ${𝒰}_{\varphi }$ using three 3D convolutional blocks, each with a 3D convolutional layer, a channel-wise multiplication layer, and a non-linear activation layer. Each block has 16 channels, except the last layer which has 2 channels representing the real and imaginary parts of the output image. ${𝒱}_{\theta }$ contains two 2D convolutional blocks with kernel size 1 and 32 channels. The first block of ${𝒰}_{\varphi }$ is activated by tanh function. Other blocks are activated by the Leaky ReLU function. $\varphi$ and $\theta$ are randomly initialized.

We binned the data into eight subsets according to $\tau \left(t\right)$ and gridding was applied to each subset to obtain 8 volumes, denoted by $\gamma$. While these approximate reconstructions are very noisy and exhibit significant alias artifacts, they capture the contrasts reasonably well. In this work, we considered the recovery of images with 32 different inversion times; $\tau \left(t\right)$ was linearly sampled from 0 to 1 with 32 steps. We jointly optimized $\varphi$ and $\theta$ based on Eq. 6 In this section, we assume the acquisition was collected without motion.

3.2. Deep Factor Model with Motion Compensation

We assume the motion after one inversion pulse remains to be the same and hence estimate the motion of the subject with temporal resolution=4s. The parameters ${\delta }_t$ and ${\nu }_t$ were initialized as zeros in (10) and solved with $\theta$ and $\varphi$.

4. RESULTS

4.1. Validation of Deep factor model

We first validate DFM in the absence of motion in Fig. 3. We compare DFM against the low-rank approach with two ranks. We observed that the lower rank translated to a poor fit to the magnetization recovery curve. In particular, we note that GM does not appear fully inverted in $r=4$ low-rank reconstructions at $\tau =550-660$ ms. By contrast, the DFM and $r=8$ low-rank reconstructions show nulled GM as expected.

Fig. 3.

Validation of DFM in the absence of motion. (a) The four columns correspond to four delay times $\tau$ after one inversion pulse; (b) Magnetization recovery curve of GM, WM, and CSF voxels indicated in the top image, estimated by different approaches.

The reconstructed images demonstrate that the DFM can offer less noisy reconstructions, especially the bins corresponding to $\tau =330-440$ ms and $\tau =550-660$ ms, where WM and GM are nulled. We note that low-rank reconstructions exhibit significant noise, while the DFM reconstructions are relatively less noisy. The plots of the voxel intensity profiles, shown in Fig. 3 (b), show that the DFM curves are closer to the expected magnetization recovery. While the accuracy of the low-rank curves would improve with rank, this will come with increased sensitivity to noise.

4.2. Validation of motion compensation

The results in Fig. 4 show the motion-compensated recovery using DFM-MC. The results demonstrate that the proposed DFM-MC can compensate for motion effects and offer reconstructions that are comparable to DFM reconstructions of the dataset without motion.

Fig. 4.

Validation of DFM-MC. (a)(c) DFM reconstructions of the acquisitions with or without motion, respectively; (b) DFM-MC reconstructions of the acquisition with motion; (d) Motion parameters estimated by DFM-MC including rotation and translation to correct (b) from (c).

5. CONCLUSION

We introduced DFM to jointly recover multi-contrast images from inversion recovery MRI data. The proposed approach further extended and generalized the low-rank method and recovered a series of images. Our experiments demonstrate that the improved representation translated to higher-quality reconstructions than low-rank models. In particular, the DFM reconstructions are less noisy and offer a more faithful representation of the magnetization recovery. The reconstructions also show the great potential of modeling and correcting for motion during the acquisition, which would be beneficial while imaging older subjects.

6. COMPLIANCE WITH ETHICAL STANDARDS

This research study was conducted using human subject data. The institutional review board at the local institution approved the acquisition of the data, and written consent was obtained from all participants.