Computationally Efficient IMplicit Training Strategy for UNrolled NEtworks (IMUNNE): A preliminary analysis using accelerated real-time cardiac cine MRI

Abstract

Objective:

Highly-undersampled, dynamic MRI reconstruction, particularly in multi-coil scenarios, is a challenging inverse problem. Unrolled networks achieve state-of-the-art performance in MRI reconstruction but suffer from long training times and extensive GPU memory cost.

Methods:

In this work, we propose a novel training strategy for IMplicit UNrolled NEtworks (IMUNNE) for highly-undersampled, multi-coil dynamic MRI reconstruction. It formulates the MRI reconstruction problem as an implicit fixed-point equation and leverages gradient approximation for backpropagation, enabling training of deep architectures with fixed memory cost. This study represents the first application of implicit network theory in the context of real-time cine MRI. The proposed method is evaluated using a prospectively undersampled, real-time cine dataset using radial k-space sampling, comprising balanced steady-state free precession (b-SSFP) readouts. Experiments include a hyperparameter search, head-to-head comparisons with a complex U-Net (CU-Net) and an alternating unrolled network (Alt-UN), and an analysis of robustness under noise perturbations; peak signal-to-noise ratio, structural similarity index, normalized root mean-square error, spatio-temporal entropic difference, and a blur metric were used.

Results:

IMUNNE produced significantly and slightly better image quality compared to CU-Net and Alt-UN, respectively. Compared with Alt-UN, IMUNNE significantly reduced training and inference times, making it a promising approach for highly-accelerated, multi-coil real-time cine MRI reconstruction.

Conclusion:

IMUNNE strategy successfully applies unrolled networks to image reconstruction of highly-accelerated, real-time radial cine MRI.

Significance:

Implicit training enables rapid, high-quality, and cost-effective CMR exams by reducing training and inference times and lowering memory cost associated with advanced reconstruction methods.

I. Introduction

Cine cardiovascular magnetic resonance (CMR) is the gold standard imaging test for evaluation of cardiac function and volumes [1], [2]. The standard pulse sequence is the retrospective, electrocardiogram gated, breath-held cine MRI with balanced steady state free precession (b-SSFP) readout [3]. In patients with arrhythmia and/or dyspnea, breath-hold, b-SSFP cine MRI may yield nondiagnostic image quality due to ghosting image artifacts [4]. In such scenarios, a real-time cine MRI may mitigate these artifacts. Standard real-time cine MRI using parallel imaging [5], [6], however, produces poor spatio-temporal resolution. One approach to achieve relatively high spatio-temporal resolution in real-time cine MRI is highly accelerating the scan using compressed sensing (CS) [7]. For data acceleration with CS, the incoherent k-space sampling pattern could be achieved with either Cartesian [8], [9] or non-Cartesian [10]–[12] approaches, with each having advantages and disadvantages. In this study, we focus on radial k-space sampling with tiny golden angles, which has the following advantages [13]: (a) incoherent k-space sampling; (b) retrospectively rebin to arbitrary acceleration rate to achieve patient-specific temporal resolution based on heart rate. A major hindrance to clinical translation with CS accelerated real-time cine with radial k-space sampling is the lengthy reconstruction time, owing to the iterative nature and also to the need to conduct nonuniform fast Fourier transform (NUFFT) [10], [14].

In recent years, deep learning (DL) reconstruction techniques have witnessed a remarkable surge of interest due to their promising ability to overcome the inherent challenges of iterative reconstruction methods. The most straightforward DL-based reconstruction techniques are the so-called post-processing or denoising networks [14]–[16]. These algorithms “learn” a function that maps a zero-filled MR image that is contaminated with aliasing artifacts to its de-aliased counterpart. The a-priori learning phase requires a large dataset consisting of pairs of aliased images and their corresponding “ground truth.” This approach, however, allows much faster inference than what is achievable with an iterative technique. Nevertheless, performance of purely data-driven denoisers depends highly on the size and the quality of the training dataset at hand. Furthermore, in general, there are no theoretical guarantees for performance or generalizability of the trained network. This a common problem in DL [17]. Closely related is the issue of adversarial attacks or instability with respect to small perturbations in the input [18], [19].

State-of-the-art reconstruction methods aim to combine the advantages of physics-based iterative and data-driven techniques [20], [21]. It has been shown that incorporating forward and adjoint operators into convolutional neural network (CNN)-based reconstruction increases stability and robustness against adversarial attacks [22]. Another study showed that including the forward model lowers the maximum error bound of the network [23]. An entire class of hybrid architectures has been derived from iterative algorithms by enhancing them with learned components. These networks are called variational, cascaded or unrolled networks as they often resemble an “unrolled” cascade of blocks, each consisting of a physics prior and a CNN-based regularizer. Therefore, each block successively refines the initial solution by simultaneously enforcing consistency with the measured k-space and regularizing the solution according to a data-driven term. Integrating prior knowledge into the architecture as a differentiable form of the imaging operator, enables significant reduction in the necessary training data [23], [24]. For several important iterative algorithms, such as ADMM [25] or ISTA [26] corresponding unrolled architectures have been successfully developed [27]–[29].

Although unrolled networks are effective with Cartesian acquisition schemes [30], [31], they require extensive computational capabilities in the case of non-trivial forward and adjoint operators (e.g., NUFFT [32]) to compute and propagate the error gradients “through” the model operators. Additionally, end-to-end training requires storing intermediate gradients of each block during backpropagation within the GPU VRAM. This computational bottleneck is a major hindrance to apply unrolled networks to multi-dimensional MRI with computationally demanding imaging operators [30], [32].

In this paper, we propose a novel strategy called IMplicit UNrolled NEtwork (IMUNNE) that leverages implicit network theory [33]–[35] to overcome the memory and training time limitations associated with unrolled networks, while retaining their advantages. Implicit networks have shown promising results in various machine learning tasks [36]–[38], and by incorporating implicit network theory into the framework of unrolled networks, we aim to reduce the computational burden and enhance the applicability of unrolled networks to multi-dimensional MRI reconstruction with radial k-space sampling.

Our research focuses on multi-coil 2D real-time cardiac cine MRI obtained using radial k-space sampling, where we evaluate the IMUNNE approach extensively. We compare the performance of our proposed method with two state-of-the-art methods: (a) a complex U-Net (CU-Net) [14], [30]; (b) end-to-end alternating unrolled network for non-Cartesian dynamic MRI reconstruction (Alt-UN) [32]. The main contributions of our work are as follows:

• We propose a novel strategy to train unrolled networks in the context of highly-undersampled, multi-coil dynamic MRI reconstruction. In particular, we reinterpret the end-to-end cascaded structure as an implicit network and apply Jacobian-free backpropagation [33] strategy to approximate the error gradient for weight updates. This strategy drastically reduces the computational demand and memory requirements during training, thereby enabling more complex operators and larger input sizes.

• We demonstrate this by applying IMUNNE as a reconstruction algorithm on a multi-coil real-time cine MRI dataset, using b-SSFP readouts. The method shows superior performance compared to state-of-the-art reconstruction techniques on the given dataset, which suggests versatility and adaptability of the method for real-time cine MRI using radial k-space sampling.

• We demonstrate that training IMUNNE does not require a large dataset. Moreover, IMUNNE consistently performs well when evaluated on noise-corrupted data. These benefits are typical of end-to-end unrolled networks and remain true for IMUNNE, even though the method avoids full end-to-end training and only approximates the correct error gradient.

The structure of the paper is organized as follows: In Section II, we provide a formal introduction to the reconstruction problem and present the details of the CNN-block used in our proposed method, along with the training strategy employed. Section III presents a comprehensive set of experiments conducted to validate the computational efficacy, accuracy and robustness of our approach. We compare our results with CU-Net – a purely data-driven approach – and Alt-UN – an end-to-end trainable unrolled network for real-time cine MRI using radial k-space sampling. Additionally, in Section IV, we discuss the main advantages and limitations of our work. There, we also provide a theoretical comparison of our method with related works. Finally, in Section V, we conclude our findings and discuss future research directions.

II. Methodology

A. Theoretical Background

The inverse problem of undersampled MRI reconstruction can be described as the following minimization problem:

$$\underset{\mathit{x}}{\mathit{min}}{\left||\mathit{A}\mathit{x}-\mathit{y}|\right|}_2^2+ℛ\left(\mathit{x}\right),$$ (1)

where $\mathit{A}$ is the NUFFT operator, $\mathit{x}$ the unknown image, $\mathit{y}$ the undersampled k-space data impacted by noise, and $ℛ$ a regularizer function restricting the solution space. Traditionally, this minimization problem is solved by iterative approximation algorithms, such as the conjugate gradient (CG) method. In our work, we are primarily concerned with multi-coil, real-time cine MRI using radial k-space sampling. In particular, $\mathit{x}\in {\mathbb{C}}^d$, with $d=d_x\times d_y\times d_t$ being the two spatial dimensions and the temporal dimension, respectively; $\mathit{y}\in {\mathbb{C}}^{d_x\times d_{sp}\times d_t\times n_c},n_c$ denoting the number of coils, $d_{sp}$ being the number of radial spokes per time frame; and $\mathit{A}$ given by the concatenation of the operator $\mathit{C}:{\mathbb{C}}^d\to {\mathbb{C}}^{d\times n_c}$, which multiplies the cine image with each of the coil sensitivity maps, and the NUFFT applied to each of the coil images. As a formula:

$$\mathit{A}≔NUFFT\circ \mathit{C}.$$ (2)

For highly accelerated sampling, i.e., $d_{sp}\ll d_y$, the Nyquist sampling criterion is not satisfied, and thus the problem is ill-conditioned. The higher the undersampling rate, the more the solution depends on the correct choice of regularization [39]. Traditionally, iterative minimization algorithms have employed different regularization strategies such as wavelet transform [40] or discrete cosine transform [41], [42]. In the context of real-time cine MRI, temporal total variation with L1 norm is widely used to enforce gradient sparsity in the time dimension [43]. However, for higher undersampling rates, the manual choice of the regularizer might prove insufficient to eliminate aliasing artifacts [44].

Unrolled networks [30], [32], [44], [45] can be seen as an iterative method which “learns” the optimal regularizer from a representative dataset. Assuming a differentiable regularizer $ℛ$, the gradient descent iterates for an optimal point of (1) are as follows:

$${\mathit{x}}_{k+1}≔{\mathit{x}}_k-s\cdot \left[{\mathit{A}}^{\perp }\left(\mathit{A}\left({\mathit{x}}_k\right)-\mathit{y}\right)+\nabla ℛ\left({\mathit{x}}_k\right)\right]$$ (3)

with a scalar step size $s>0$.

Next, we express the gradient of the regularizer $\nabla ℛ$ as a neural network $U_{\Theta }$ with a trainable parameter set $\Theta ≔\left({\theta }_1,\dots ,{\theta }_p\right)$. The update operator $R_{\Theta ,s}$ to produce the next iterate ${\mathit{x}}_{i+1}≔R_{\Theta ,s}\left({\mathit{x}}_i\right)$ is then given by:

$$R_{\Theta ,s}≔{\mathit{I}}_d\left(\cdot \right)-s\cdot \left[{\mathit{A}}^{\perp }\left(\mathit{A}\left(\cdot \right)-\mathit{y}\right)+U_{\Theta }(\cdot )\right]$$ (4)

where ${\mathit{I}}_d$ denotes the identity map.

Instead of iterating until convergence, “unrolled” networks apply the update operator $R_{\Theta ,s}$ for a fixed number of times $n$, whereby an initial input is successively refined by each of the unrolled blocks (Fig. 1). This setup defines a differentiable algorithm that is solely dependent on the parameter set $\Theta$, which can therefore be trained end-to-end using conventional error backpropagation [24], [46] to find the optimal parameters $\Theta$. However, conventional error backpropagation to train unrolled networks limits the unrolling to only very small numbers of blocks since the memory requirements grow linearly with the network depth [30], [32], [33]. This problem is particularly exacerbated by the presence of computationally demanding operators, such as NUFFT, and/or multi-dimensionality of real-time cine MRI data (i.e., multi-coil, 2D, time) [32].

Fig. 1.

Schematic illustration of the unrolled structure interpreted as an implicit network: Zero-filled input ${\mathit{x}}_0$, which was obtained by the inverse NUFFT operator, is refined by the update $R_{\Theta }$ until convergence. $R_{\Theta }$ consists of a data-consistency term and learned regularizer.

B. Implicit Networks

Another way to approach the minimization problem (1) is to consider the fact that the gradient of the (continuously differentiable) objective function attains zero at an optimal point $\widehat{\mathit{x}}$. This has the consequence that $\widehat{\mathit{x}}$ is a fixed-point of the update operator $R_{\Theta ,s}$:

$$R_{\Theta ,s}\left(\widehat{\mathit{x}}\right)=\widehat{\mathit{x}}-s\cdot \underset{=\nabla \left[{\left||\mathit{A}\widehat{\mathit{x}}-\mathit{y}|\right|}_2^2+𝓡\left(\widehat{\mathit{x}}\right)\right]=\mathbf{0}}{\underbrace{\left[{\mathit{A}}^{\perp }\left(\mathit{A}\left(\widehat{\mathit{x}}\right)-\mathit{y}\right)+U_{\Theta }\left(\widehat{\mathit{x}}\right)\right]}}=\widehat{\mathit{x}}$$ (5)

Under certain assumptions for $R_{\Theta ,s}$, we can formulate the MRI reconstruction problem in the following form of an implicit fixed-point equation:

$${Find\widehat{\mathit{x}}with\widehat{\mathit{x}}=R}_{\Theta ,s}\left(\widehat{\mathit{x}}\right).$$ (6)

The type of neural networks that find a solution of an implicit equation in the form of (6) are called implicit networks [33]–[35]. In contrast to classical feed-forward networks that produce an explicit deterministic output, implicit networks approximate the solution of an implicit equation via a fixed-point iteration similar to (3).

Given the inconsistent use of the term implicit in deep-learning literature, it is important to clarify that the implicit networks mentioned earlier are categorically different from implicit neural representation networks [47], [48]. This distinction is crucial, particularly because neural representation networks have already been extensively applied to dynamic MRI reconstruction [49]–[51]. Our research is entirely unrelated to implicit neural representation networks. Therefore, whenever we refer to an implicit network, we are specifically referencing the type of networks that solve a fixed-point equation [33].

The unrolled network structure defined by (4) can thus be viewed as an implicit network and the unrolled blocks correspond to the fixed-point iteration (Fig. 2), which solves the implicit problem stated in (6). Furthermore, if $R_{\Theta ,s}$ is a contraction, i.e., it is $\gamma$-Lipschitz with $\gamma \in \left[\mathrm{0,1}\right)$, a classical result in functional analysis guarantees linear convergence of the sequence defined in (3) [52].

Fig. 2.

Unrolled architecture defined by the update operator $R_{\Theta ,s}:n$ identical unrolled blocks consisting of data consistency (top), skip connection (center), and a learned regularizer term (bottom). Zero-filled initial estimate $x_0$ is refined consecutively by each block. For the sake of brevity, the figure does not show the measured k-space and the coil-sensitivity maps which are part of the input for the data consistency blocks. The last block produces the final image $x_n$. End-to-end training can be achieved by either conventional or Jacobian-based, i.e. analytical, backpropagation. In contrast to end-to-end training, IMUNNE interprets the unrolled scheme as an implicit network and applies Jacobian-free backpropagation [33], i.e., error is propagated only through the last block, thereby approximating the true gradient. Due to weight sharing, this is sufficient to update all regularizers. In our experiments, we use a complex U-Net to implement the learned regularizer $U_{\mathrm{\Theta }}$.

One substantial advantage of implicit networks is that they use analytical gradients for backpropagation, and thus enable training of very deep architectures with fixed memory cost. Furthermore, analytical gradients allow network training when the number of unrolls is a priori unknown. In contrast, conventional backpropagation computes gradients for each block sequentially in the reverse order and, therefore, requires a fixed number of unrolls. Previous works used the costly Jacobian operator for the analytical weight update [34], [35]. However, a recent study showed that the true gradient can be approximated by backpropagating through last iteration only using Jacobian-free backprop (Fig. 2) [33]. The authors of [33] provide a theoretical error bound of the approximation and experimentally show that this approach even alleviates some of the constraints on the operator $R_{\Theta ,s}$, e.g., the requirement for it to be a contraction.

In our work, we apply this theoretical result heuristically to train an unrolled network “implicitly”, i.e., performing backpropagation through the last unrolled iteration only using multi-coil, real-time cine MRI datasets. Given the fixed-point iteration, weight sharing is used between the “n” regularizers and, therefore, Jacobian-free backpropagation is only calculated for the last block. This setup would not be possible with conventional backpropagation due to memory constraints of currently available GPU cards. To the best of our knowledge, this study represents the first application of implicit network theory to enable unrolled training in the context of multi-dimensional MRI.

C. Proposed Training Method

In this work, we introduce IMUNNE, a simple scheme of gradient backpropagation through last iteration only. For architectures with identical unrolled blocks, we have explained in the last subsection how they can be reformulated as an implicit network and therefore benefit from Jacobian-free backpropagation. Fig. 2 depicts the architecture and the IMUNNE training scheme. In the following, we will use the abbreviation IMUNNE for both the architecture and the training strategy.

The structure is comprised of an unrolled CS scheme. The network receives zero-filled cine image ${\mathit{x}}_0$ as input and passes it through a computational block $n$ times. In accordance with (4), the block computes the sum of the following three components: Model operator term involving NUFFT; a skip connection; and a data-driven regularizer term incorporating a CU-Net [14], [30], [53] which acts as the regularizer $U_{\Theta }$. Each time the cine image set is processed by the block, it is being refined, until after $n$ steps it yields the network output ${\mathit{x}}_n$. In contrast to CS, where the gradient step size $s$ is determined by the line search method for each iteration, we treat $s$ as a hyper-parameter and keep it fixed for all unrolled blocks. More elaborate schemes to initialize (or even train) $s$ in combination with IMUNNE might be an interesting direction for future investigation, which is beyond the scope of this initial work.

The regularizer block was realized by a CU-Net from the CINENet architecture [30] which we adapted to process 2D+time input (Fig. 3). The authors showed the advantage of using a cascade of spatial convolutions followed by 1D temporal convolutions to better capture the spatial and dynamic features and to reduce the number of trainable parameters. In contrast to the original work, our version of the CU-Net consists of 3 encoding/decoding stages enhanced with skip connections. Each stage consists of 2D spatial followed by 1D temporal convolutional layers [30]. We used leaky ReLU as activation function and average pooling for dimensionality reduction. The final 3D convolution layer produces the network output.

Fig. 3.

Complex U-Net (CU-Net) architecture: encoding/decoding stages enhanced with skip connections (dashed line). Each stage consists of 2D spatial followed by 1D temporal convolutional layers (cf. [30]). We use leaky ReLU as activation function and average pooling. The final 3D convolution layer produces the final output. We employed CU-Net both as IMUNNE regularizer ($U_{\Theta }$) and as a standalone denoiser network with the same number of parameters.

IMUNNE was trained in a supervised manner: the $L_2$ difference between ${\mathit{x}}_n$ and the ground truth image $\mathit{x}$ was computed as the final loss function. We used the ADAM [54] optimizer with default parameters. Due to memory limitations, we set the batch size to be 1.

D. Experiments with Patient Data

To assess the computational efficacy and accuracy of IMUNNE, we conducted several preliminary experiments with in-vivo patient data described in the next section. First, we performed an extensive hyper-parameter grid search to find optimal values for the step size $s$ and the number of unrolled blocks $n$. Finally, we compared IMUNNE to CU-Net and Alt-UN. In the subsequent sections, we present comprehensive details pertaining to the utilized datasets, the evaluation methodologies employed, the implementation specifics of the proposed method, and the computational environment in which the experiments were conducted.

E. Datasets

We used a retrospective real-time cine dataset obtained using radial k-space sampling [10]. The dataset contained 49 real-time cine scans using b-SSFP readout acquired from a cohort of 40 patients (9 patients had follow-up exams on different days; 30 men; mean age = 68 ± 9.7 years) with atrial fibrillation, as previously described [14]. All patients provided informed consent in writing and agreed to future analysis of data. This study was performed in accordance with protocols approved by our institutional review board (Northwestern University; protocol number: STU00205545 [approval date: 10/08/2019]) and was Health Insurance Portability and Accountability Act (HIPAA) compliant.

Each scan consisted of 11–18 slices per patient and 40 cardiac frames per slice. We randomly split the dataset into 10 cases for training, 10 cases for validation and 29 cases for testing. This split was motivated by several reasons: First, allocating approximately 20% of the data for training enabled us to manage the long training times. This is reinforced by the fact that unrolled networks require less training data in general [23], [24]. Second, assigning 60% of the dataset to testing achieves a higher level of statistical fidelity during evaluation. Finally, this setup accurately simulates the practical constraints of working in a clinical context with limited data availability. The sample unit for training was a single cine series resulting in 140 training, 137 validation and 410 testing cine sets. Due to reasons listed in [14], we obtained the ground truth using Golden-angle radial sparse parallel MRI (GRASP) [55]. The coil sensitivity maps were self-calibrated from time-average data using the method described by Walsh et al. [56]. We applied coil compression to 8 virtual coils using PCA [57]. The reconstructed coil sensitivity maps were subsequently used within training and evaluation of the unrolled networks (IMUNNE and Alt-UN). For the initial reconstruction and throughout the experiments, we applied a simple ramp filter as density compensation for the radial trajectory.

Pertinent imaging parameters included: field of view (FOV) = 288 × 288 mm, matrix size = 160 × 160, spatial resolution = 1.8 mm × 1.8 mm, slice thickness = 8 mm, TE = 1.4 ms, TR = 2.7 ms, receiver bandwidth = 975 Hz/pixel, tiny golden angle sequence = 23.62814° [58], temporal resolution = 29.7 ms, and flip angle = 52°. We used 11 radial spokes per cardiac frame, corresponding to an effective acceleration factor of 15 (compared to fully sampled Cartesian counterpart).

F. Hyperparameter Search

To find an optimal network configuration, we first performed a hyper-parameter 2D grid search for different numbers of unrolled blocks ($n=2,3,4,5,6$) and fixed step sizes ($s=0.5,0.6,\dots ,1.3,1.4$). To faithfully compare the different configurations, we trained each of them exactly with 500 epochs.

G. Comparison with CU-Net and Alt-UN

We compared our IMUNNE strategy to a standalone CU-Net acting as a denoiser reconstruction network. For a fair comparison, we used the same configuration, particularly the same number of trained parameters, as in the CU-Net within the unrolled architecture. In addition to the dataset described in the previous section, we also performed cross-validation of CU-Net trained on the extended training set comprising 80% of the data and evaluated on the remaining 20%. Moreover, we compared IMUNNE to Alt-UN, an end-to-end UN specifically targeting multidimensional non-Cartesian MRI reconstruction [32]. In contrast to our architecture, this approach alternates data-consistency and CNN-based regularization blocks within the end-to-end reconstruction pipeline to reduce the GPU memory requirements.

Before training, the network weights were initialized using the default PyTorch initialization routine (e.g. uniform zero-mean initialization for convolutional layers; negative slope = 0.01 for leaky ReLU). For the sake of simplicity, we used the same learning rate of 5e-04 with a learning rate decay factor of 0.95 every 30 epochs for all training instances. Both CU-Net and IMUNNE were trained for 500 epochs, respectively. Alt-UN was trained according to the training scheme prescribed by the authors of the original publication [32]: the regularization block was pretrained followed by 150 epochs of fine-tuning of the end-to-end architecture.

Finally, we performed the same comparison experiments using noise-corrupted k-space measurements as input. Specifically, a given k-space measurement ${\mathrm{k}\mathrm{s}\mathrm{p}}_{\mathrm{m}}$ was corrupted by adding a scaled noise vector sampled from a complex normal distribution [59]. The scaling parameter for the noise was $0.1\times \max \left(\mathrm{a}\mathrm{b}\mathrm{s}\left(\mathrm{k}\mathrm{s}{\mathrm{p}}_{\mathrm{m}}\right)\right)$, where the right factor stands for the maximum absolute value of $\mathrm{k}\mathrm{s}{\mathrm{p}}_{\mathrm{m}}$. Subsequently, we used the distorted data to evaluate the stability of the networks which had been trained on the original uncorrupted data, as described above.

H. Evaluation Metrics

To evaluate and compare the performance of IMUNNE between different hyper-parameter settings and with other methods, we calculated the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM) [60] and the normalized root mean-square error (NRMSE) of network outputs with respect to the corresponding ground truth. A reference-free blur metric [61] (0: sharpest; 1: blurriest) was calculated to assess the preservation of fine details. To assess sharpness in spatial domain, we computed the metric for each frame of the $(x,y,t)$ cine series and report the mean value as “2D Blur”. To quantify temporal blurring, we follow the procedure in [62] and calculate the metric for the 1D-FFT-transformed series $(x,y,f)$ which we refer to as “FFT Blur”. Finally, we report the combined spatio-temporal reduced reference entropic difference (ST-RRED) [63], [64] to evaluate to overall video quality. All measures were calculated over a central region of 100×100 pixels, in order to focus on the heart region. Other relevant performance factors that we report are the training time and the evaluation time per instance.

I. Implementation and Computational Environment

To train the different network architectures, we used a Linux GPU workstation (NVIDIA A100-PCIE, 40GB VRAM GPU; AMD EPYC 7702P 64-Core CPU, 512GB RAM). All architectures were implemented using PyTorch with GPU acceleration. To incorporate backpropagatable NUFFT operators, we used the TorchKbNufft [65] library. The code implementation of the proposed method was made publicly available on GitHub¹.

For Alt-UN [32], we used the implementation which was provided by the authors as part of the original publication and only adapted it slightly to fit our datasets. However, we opted to eliminate the residual connection within the image domain since our datasets were reconstructed using an adaptable regularization weight, leading to different scaling magnitude between the zero-filled inputs and the reconstructed images. A visual comparison of the evaluation results, both with and without the skip connection, is shown in Fig. S3. In particular, the network version with the residual connection produced poor image quality. Otherwise, training was performed using the same hyperparameters as reported in the original publication with a trainable weight $\lambda$.

For both IMUNNE and Alt-UN, the network input consisted of four components listed in the following with the corresponding dimensions and their respective specific values ($x,y$: 2D spatial coordinates; $f$ number of time frames; $c$: number of coils; $nsp$: number of radial k-space spokes per time frame):

• Zero-filled input / Ground truth data: 320 × 320 × 40; $(x,y,t)$

• Measured k-space data: 320 × 8 × 11 × 40: $(x,c,nsp,t)$

• Coil sensitivity maps: 8 × 320 × 320: $(c,x,y)$.

Since CU-Net does not require k-space, the network input consisted only of the zero-filled image, spatially cropped to the central region of 160 × 160 × 40.

All subsequent evaluation experiments were conducted using the centred ROI of 100 × 100 × 40 pixels.

III. Results

In this section, we present the results obtained from our experiments with patient data, focusing on the training behavior and reconstruction performance of the proposed method.

A. Hyperparameter Choice

The first experiment involved finding the optimal values for the step size $s$ and the number of unrolled blocks $n$ for the IMUNNE architecture. Except for the blur metric, all other evaluation metrics indicate $n=5$ and $s=0.8$ to be an optimal pair (see also supplementary Fig. S4). In the case of 4 unrolls the blur metric shows improvement for some large $s$ values. However, since artificial sharpness can be introduced by an increased presense of noise [14], we continued to use the parameters $\left(n,s\right)=(5,0.8)$ for subsequent experiments.

A more extensive grid search would be required to thoroughly understand asymptotic behavior of the metrics for $n\to \infty$ and $s\to 0$. Heuristically, one might expect better results to the degree $R_{\mathrm{\Theta },\mathrm{s}}$ is a contraction while simultaneously increasing the number of iterations. It is straightforward to see from (4) that this corresponds to the case of very small values of $s$ and large values of $n$. While this is an iteresting theoretical consideration, it is practically infeasible to test this scenario due to the following reasons:

1. In order to have a practically relevant advantage to CS in terms of computational speed, $n$ must be significantly lower than the number of iterations in CS (30 in our case).

2. In order to compensate for a low $n$, we need to increase $s$ to allow the network to make “larger leaps” towards the optimum. This brings the operator $R_{\mathrm{\Theta },\mathrm{s}}$ further away from being a contraction.

3. Due to lengthy training times (~3–5 days) for just one hyperparameter pair $\left(n,s\right)$, it is infeasible to perform a extensive grid search for larger number of unrolls, since both training and the inference time increase proportionally with $n$.

Although these practical constraints (small $n$; large $s$) lead to a deviation from the theoretically optimal scenario (large $n$; small $s$), the optimal values found through the limited grid search, nevertheless, produced promising results, as we illustrate in the following sections.

B. Memory benchmarks

To demonstrate the memory-efficiency of IMUNNE, we performed an ablation study with different numbers of unrolled blocks (IMUNNE) and different numbers of alterations (Alt-UN; parameter $M$ in the original publication [32]), respectively. We measured the peak allocated GPU memory (in megabytes) for each of the networks during training. Our results clearly indicate the advantage of IMUNNE (Fig. 4): constant memory requirement despite growing number of iterations since training is performed only using the last unrolled block. On the other hand, Alt-UN, being an end-to-end trainable architecture, demonstrates a linear increase in memory with the growing number of alterations. Note that the case of “0 alterations” corresponds to CNN pretraining and is not a UN in the strict sense. Due to memory limits of our GPU, it was only possible to perform up to 3 alterations for Alt-UN. Despite the same trainable component of IMUNNE and CU-Net (21.7K trainable parameters), the former requires the additional storage of raw k-space data and the sensitivity maps for the data-consistency component, which leads a minor increase in memory. The initial gap between the both unrolled networks is cause by the difference in the number of trainable parameters (Alt-UN: 129K; IMUNNE: 21.7K).

Fig. 4.

Comparison of memory footprints of Alt-UN, IMUNNE and CU-Net during training, and GPU-accelerated CS reconstruction: The memory requirement for IMUNNE is constant regardless of the number of iterations, whereas Alt-UN shows a linear increase. Having the same trainable base, CU-Net and IMUNNE only differ due to additional storage of k-space and sensitivity maps within IMUNNE. The initial gap between Alt-UN and IMUNNE is caused by the difference in the number of trainable parameters (Alt-UN: 129K; IMUNNE 21.7K). Note that the number of unrolls/alterations is not applicable to CU-Net and CS which are, therefore, indicated as horizontal lines in the plot.

C. Performance Comparison

Here, we demostrate the reconstruction results of the fully-trained architectures. For IMUNNE, we used the hyperameters obtained through a grid search described above. Other relevant network and training parameters are described in Sections II.F and II.G.

Fig. 5 shows a representative example (see also supplementary Video S1 for dynamic display). Both unrolled networks clearly outperform the standalone CU-Net, even in case when CU-Net was trained with an extended training dataset (80% of all available data). In particular, CU-Net fails to remove the radial streaking artifacts. Compared to Alt-UN, IMUNNE showed slightly better image quality and significantly outperformed Alt-UN in all reported metrics except the spatial blur metric. Furthermore, IMUNNE was able to produce these results while reducing the training time from 124 hours to 80 hours (> 35% reduction); since our approach uses less unrolled iterations, the inference time was also reduced from 9.53 seconds to 2.04 seconds (> 75% reduction). The training and inference times and the image quality metrics are summarized in Table I.

Fig. 5.

Representative example from b-SSFP testing dataset. The upper row depicts the 15th frame along with the temporal profile along the vertical middle line (e.g., like M-mode in echocardiography; yellow line in the left image) of the cine series reconstructed with (from left to right): Compressed Sensing (ground truth image); inverse NUFFT of the undersampled k-space; CU-Net trained on 80% of the data; CU-Net; Alt-UN; IMUNNE. The bottom row shows the corresponding scaled difference images for each method. For dynamic display, see Video S1 in Supplemental Materials.

TABLE I.

QUALITATIVE EVALUATION METRICS, TRAINING TIMES, AND EVALUATION TIMES.

	CS Reference	Zero-filled NUFFT	CU-Net (80%)*	CU-Net	Alt-UN	IMUNNE
SSIM		0.223 ± 0.055	0.912 ± 0.033	0.880 ± 0.045	0.925 ± 0.029	0.950 ± 0.021
NRMSE		0.200 ± 0.042	0.021 ± 0.009	0.032 ± 0.012	0.026 ± 0.009	0.019 ± 0.007
PSNR (dB)		14.16 ± 1.84	34.10 ± 2.85	30.19 ± 2.94	32.34 ± 2.86	34.88 ± 3.20
2D Blur	0.362 ± 0.062	0.313 ± 0.048	0.398 ± 0.046	0.418 ± 0.061	0.392 ± 0.062+	0.385 ± 0.060 +
FFT Blur	0.223 ± 0.031	0.250 ± 0.031#	0.220 ± 0.027	0.277 ± 0.029	0.249 ± 0.029#	0.241 ± 0.030
ST-RRED		1.798 ± 0.837	0.286 ± 0.147	0.412 ± 0.221	0.311 ± 0.152	0.215 ± 0.125
Training time	-	-	1d 3h 500 epochs	7.5h 500 epochs	5d 4h 500+150 ep.**	3d 8h 500 epochs
Inference time	20 s ± 82.6 ms	607 ms ± 6.2 ms	653 ms ± 2.8 ms	653 ms ± 2.8 ms	9.53 s ± 40.4 ms	2.04 s ± 8.1 ms

^*CU-NET TRAINED ON THE EXTENDED TRAINING DATASET (80%) AND EVALUATED ON A LIMITED TESTING SET (20%).

^**TRAINING INVOLVES PRETRAINING THE CNN BLOCK AND FINE-TUNING THE END-TO-END NETWORK [32].

^+#P > 0.05 CORRESPONDS TO NON-SIGNIFICANT DIFFERENCE IN PAIR.

D. Robustness Analysis

In this subsection, we demonstrate the comparison results under the presense of complex Gaussian noise. The k-space measurements were noise-corrupted as described in Section II.G. Subsequently, the architectures were evaluated using the noisy k-space data as input.

Fig. 6 (see also supplementary Video S2) shows the same representative example as in Fig. 5. IMUNNE maintains its superior performance under presence of noise and successfully removes streaking artifacts. The evaluation metrics are summarized in the supplementary Table S.I which shows a similar trend as in the base case (uncorrupted k-space input).

Fig. 6.

Same representative example from b-SSFP testing dataset with noise-corrupted k-space as input. The upper row depicts the 15th frame along with the temporal profile along the vertical middle line (e.g., like M-mode in echocardiography; yellow line in the left image) of the cine series reconstructed with (from left to right): Compressed Sensing (ground truth image); inverse NUFFT of the undersampled k-space; CU-Net trained on 80% of the data; CU-Net; Alt-UN; IMUNNE. The bottom row shows the corresponding scaled difference images for each method. For dynamic display, see Video S2 in Supplemental Materials.

IV. Discussion

In this paper, we introduced a novel implicit training strategy for unrolled networks (IMUNNE) to address the challenges associated with memory-intensive end-to-end training of unrolled networks for multi-dimensional MRI reconstruction. Our approach leverages implicit network theory to approximate the gradient for weight updates during training, enabling us to overcome the computational bottleneck and efficiently apply unrolled networks to multi-dimensional MRI datasets.

Our experiments with 2D multi-coil real-time cardiac cine MRI dataset obtained using radial k-space sampling demonstrated the efficacy of IMUNNE. The results indicated that IMUNNE outperforms a purely data-driven approach (CU-Net reconstruction) and is competitive with a state-of-the-art end-to-end unrolled network (Alt-UN). While achieving similar image quality, IMUNNE is able to cut down training time significantly since – in contrast to end-to-end training – backpropagation is performed only for the last unrolled iteration. Given the free GPU memory, it is possible to further reduce the training time for IMUNNE by increasing the batch size (Fig. 4). However, for the sake of simplicity and a fair comparison with the baseline methods, we fixed the batch size to 1. Additionally, the 2D blur metric indicated that IMUNNE preserves fine details slightly better, enhancing the resulting spatial resolution. The significantly better FFT blur metric points to improved temporal resolution achieved by IMUNNE.

In the following, we present some theoretical considerations, discuss advantages and limitations of our proposed approach relative to other works.

A. Limitations

In order to calculate the fixed point of an implicit equation to a high degree of precision, implicit networks generally need to compute a large number of iterations [33], [34], [35], [52]. However, as we illustrated in Section III.A, the implicit network theory was only heuristically applied for IMUNNE; that is, the architecture could be trained for only a small number of unrolls due to practical constraints and lengthy training times. Despite this deviation from original implicit networks, we were able to successfully conduct a hyperparameter search to find the best values for the aforementioned parameters. As a result, IMUNNE showed promising results, outperforming CU-Net in terms of image quality and Alt-UN in terms of training and inference times.

Furthermore, we only evaluated IMUNNE in a supervised training setting [32], where ground truth data was obtained through a priori CS reconstruction. This is a limitation since, in this case, IMUNNE can never learn to outperform CS [66], [67]. However, there is nothing inherent in our proposed method that would preclude its application in an unsupervised training scenario. We leave this interesting research direction for future investigations.

B. Relation to Similar Works

Although extensive research has been conducted to accelerate MRI reconstruction using unrolled networks [30], [32], [44], [45], very few strategies apply unrolled networks to high-dimensional MRI with non-Cartesian k-space sampling patterns. Beside Alt-UN [32], a similar approach was proposed for 3D cardiac cine MRI reconstuction employing a cascade of backpropagatable data consistency layers interleaved with U-Net-based regularizers [30]. However, this work focuses on Cartesian k-space data which does not pose the additional challenge of NUFFT operators. Other works have addressed non-Cartesian k-space sampling patterns but mostly in the context of non-dynamic reconstruction [68], [69].

Since fully sampled imaging data for training DL networks is notoriously challenging to obtain, there has been a growing research interest to develop unsupervised training techniques for reconstruction networks [70], [71]. For example, the dual-domain self-supervised training [45] divides the k-space measurement into disjoint partitions and compares the reconstructions of each of the partitions in the image and the k-space domains, exploiting self-similarity in both domains in the loss function. This strategy achieves competitive performance relative to supervised training for low acceleration factors $R=\mathrm{2,4}$. However, for higher undersampling ratios, the given k-space coverage is much lower, which makes the partitioning strategy unsuitable because it would further reduce the available input information from the measurement.

Although a true end-to-end training of the unrolled network is impossible due to GPU memory limits, it would be conceivable to compare IMUNNE with a pseudo end-to-end training approach using gradient checkpointing [72]–[74]. This strategy reduces the memory cost by storing only a part of the intermediate values required for backpropagation. Then, after the error for the stored part has been evaluated, another forward pass is required to further calculate backpropagation throughout the architecture. In the most extreme case, this evaluation can be repeated for each unrolled block, thereby reducing the memory cost to $O\left(1\right)$. This, however, comes at the expense of additional computational time to recalculate the forward pass $n$ times, leading to a computational time increase from $O\left(n\right)$ to $O\left(n^2\right)$. Even though this approach calculates the gradient precisely, we decided not to perform this experiment since it would take several weeks to train the network in this manner for one single hyperparameter setting.

Finally, implicit networks have been applied to undersampled MRI reconstruction [36], [37]. In one of the most recent works [36] the authors developed a self-supervised implicit model that approaches the performance of supervised training. However, the method has only been tested on multi-coil 2D brain MRI using Cartesian k-space sampling with relatively moderate acceleration factors (≤ 8). Since the forward pass uses 100 iterations for the fixed-point estimation, it cannot be applied to dynamic non-Cartesian k-space data directly, especially when the model includes the computationally costly NUFFT operator.

In principle, it is expected that many of the available techniques can be combined with IMUNNE to further improve its performance. For example, we performed ablations with instance and layer normalization layers in the regularizer CU-Net but did not observe improvements in performance. We did not test batch normalization due to its poor performance for small batch sizes [75]. Furthermore, IMUNNE can be integrated with the self-supervised training approaches presented above. In contrast to these techniques, whose core objective is to develop a specialized loss term to achieve self-supervision, our strategy is primarily focused on a special backpropagation to update the network weights within the architecture itself. Nevertheless, a rigorous evaluation of a possible compatibility is necessary which we leave for future work.

V. Conclusion

In conclusion, our novel IMUNNE training strategy successfully applies unrolled networks to non-Cartesian, real-time cine MRI reconstruction, outperforming purely data-driven approaches. IMUNNE efficiently approximates the gradient for weight updates during training, significantly reducing memory usage and training time compared to fully unrolled networks. This work has the potential to facilitate a more widespread adoption of CMR by reducing training time, inference time and memory cost of state-of-the-art reconstruction methods, thereby lowering the clinical hardware requirements and energy consumption necessary for state-of-art reconstruction methods.