A network‐assisted joint image and motion estimation approach for robust 3D MRI motion correction across severity levels

Abstract

Purpose

The purpose of this work was to develop and evaluate a novel method that leverages neural networks and physical modeling for 3D motion correction at different levels of corruption.

Methods

The novel method (“UNet+JE”) combines an existing neural network (“UNet_mag”) with a physics‐informed algorithm for jointly estimating motion parameters and the motion‐compensated image (“JE”). UNet_mag and UNet+JE were trained on two training datasets separately with different distributions of motion corruption severity and compared to JE as a benchmark. All five resulting methods were tested on T₁w 3D MPRAGE scans of healthy participants with simulated (n = 40) and in vivo (n = 10) motion corruption ranging from mild to severe motion.

Results

UNet+JE provided better motion correction than UNet_mag ($p<{10}^{-2}$ for all metrics for both simulated and in vivo data), under both training datasets. UNet_mag exhibited residual image artifacts and blurring, as well as greater susceptibility to data distribution shifts than UNet+JE. UNet+JE and JE did not significantly differ in image correction quality ($p>0.05$ for all metrics), even under strong distribution shifts for UNet+JE. However, UNet+JE reduced runtimes by a median reduction factor of between 2.00 to 3.80 as well as 4.05 for the simulation and in vivo studies, respectively.

Conclusions

UNet+JE benefitted from the robustness of joint estimation and the fast image improvement provided by the neural network, enabling the method to provide high quality 3D image correction under a wide range of motion corruption within shorter runtimes.

1. INTRODUCTION

Patient motion is a common source of image quality degradation in MRI. ¹ Neuroanatomical MRI scans typically require several minutes of data acquisition, and changes to a patient's head position during that time can lead to ghosting artifacts, blurring, and signal loss, which reduce the diagnostic value of the image. Head motion is prevalent in pediatric and elderly cohorts, as well as patients with movement disorders and participants experiencing discomfort during scans. ² , ³ In standard clinical practice, heavily corrupted scans are often repeated, which extends the length of the MRI exam and further increases patient discomfort. Scan repeats because of motion occur in 20% of MRI exams ⁴ and amount to an estimated cost of $1.4 billion USD per year in the United States. ⁵ In research settings, motion artifacts can bias or confound findings. ⁶ , ⁷

Motion mitigation strategies include patient training, head restraints, ⁸ and administering sedatives, which are not always feasible or desired. ² Alternatively, prospective motion correction attempts to compensate for motion by adjusting MRI gradients in real time using data from external sensors ⁹ , ¹⁰ , ¹¹ , ¹² or navigator sequences. ¹³ , ¹⁴ , ¹⁵ , ¹⁶ Although prospective correction could theoretically account for most deleterious effects of motion, these approaches require dedicated hardware or specialized pulse sequences and are vulnerable to issues with calibration, accuracy, and latency.

Alternatively, data‐driven retrospective motion correction (RMC) methods estimate the uncorrupted image without requiring external motion tracking. Classical methods use a physical model of the impact of motion on the data ¹⁷ to jointly estimate the underlying image and the subject's head movement. The motion trajectory is estimated by minimizing some measure of the presence of motion artifacts in the data, such as image gradient entropy ¹⁸ , ¹⁹ , ²⁰ or data consistency. ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ These joint estimation (JE) approaches involve large‐scale non‐convex optimization, which is computationally challenging. Optimized sampling patterns ²⁵ and scout images ²⁶ , ²⁷ have been proposed to improve the convergence properties of JE.

More recently, various deep learning (DL) approaches have been proposed. Spieker et al. ²⁸ conducted a comprehensive review of DL methods, including image‐to‐image translation approaches. ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ Although DL methods are attractive because of their fast runtime and their ability to learn image priors that are difficult to mathematically formulate, DL‐based approaches suffer from “hallucinations” (i.e., spurious image features) and smoothing, particularly under data distribution shifts. ³⁰ , ⁴¹ , ⁴² Spieker et al. ²⁸ concluded that DL‐based methods should be grounded in the physics of image acquisition (i.e., enforcing data consistency) to increase their robustness.

Haskell et al. ⁴³ proposed the NAMER algorithm, which incorporated an artifact‐removal neural network within the JE algorithm, leading to better convergence properties. Neural networks have subsequently been shown to benefit image sparsity‐based JE ⁴⁴ and scout‐assisted JE. ⁴⁵ To provide greater robustness against different sampling patterns, Levac et al. ⁴⁶ trained an unsupervised image denoiser to assist with JE. To reduce JE to only its motion estimation subcomponent, Singh et al. ⁴⁷ trained a hypernetwork to estimate the motion‐compensated image. Conversely, Dabrowski et al. ⁴⁸ reduced the problem to only image estimation by training a regression network to directly estimate motion parameters from the corrupted k‐space data.

Although promising, hybrid DL‐JE works have only been applied to 2D image correction and have not been implemented for 3D volumetric correction, which is non‐trivial because of its greater computational complexity. Although Cordero‐Grande et al. ²⁴ , ²⁵ have worked extensively on JE for 3D motion correction, they do not leverage DL. Although Wang et al. ⁴⁹ have applied DL to improve 3D motion‐compensated image reconstruction, their study focused on the special case where motion parameters were known a priori.

Furthermore, limited research has been conducted on the performance of hybrid approaches under data distribution shifts with respect to motion corruption severity. It is important to characterize the impact of a method's training distribution on its performance. To this end, we developed a novel 3D hybrid DL‐JE algorithm and evaluated its performance on T₁‐weighted 3D MPRAGE scan data with a wide range of simulated head motion corruption severity relative to different training datasets. Finally, we validated our method on data with in vivo motion corruption. Our source code is publicly available on GitHub (https://github.com/BRAIN‐TO/PyMoCo_v2).

2. METHODS

2.1. Rigid body model of motion corruption

The impact of motion on the MRI signal can be approximated using basic properties of the Fourier transform under rotations and translations. ¹⁷ For rigid body motion, the transformed signal encoding operator is given by Equation 1, where $U$ is the sampling mask, $F$ is the Fourier transform, $C$ are the complex coil profiles, and $\theta$ consists of rotations followed by translations. The operators $U$ and $F$ are known a priori, whereas $C$ is typically measured independently. Eq. 1 does not account for the impact of head motion on physical MRI properties (e.g., B₀, B₁ ⁺, spin excitation and phase history ¹ ), which would require a more complicated physical model. Larger head motions lead to greater deviations from the assumption of a rigid body motion.

We can approximate continuous changes in head position as piecewise constant motion states, where the $n^{\mathit{th}}$ discrete motion state (${\theta }_n$) consists of 6 degrees of freedom and is prescribed over a segment of the sampling pattern ($U_n$). The motion‐corrupted signal encoding model is then described by Equation 2, where $s$ is the acquired k‐space signal, $m$ is the magnetization image, and $\eta$ is Gaussian noise.

$$E_{\theta }=Uℱ\mathit{C}\theta$$ (1)

$$s=\sum _{n=1}^N{U}_nℱC{\theta }_{n}m+\eta =E_{\theta }m+\eta$$ (2)

In the absence of explicit motion tracking (e.g., using external trackers ⁹ , ¹⁰ , ¹¹ , ¹² or navigators ¹³ , ¹⁴ , ¹⁵ , ¹⁶ ), we can jointly estimate both $m$ and $\theta$. ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ In this work, we focus on the data consistency‐based JE algorithm (Equation 3), which iteratively estimates $m$ and $\theta$ by minimizing discrepancies between the measured signal ($s$) and the projected data ($E_{\theta }m$).

$$(\widehat{m},\widehat{\theta })={\underset{m,\theta }{\text{argmin}}‖E_{\theta }m-s‖}_2^2$$ (3)

It is difficult for the JE algorithm to resolve motion states at every read‐out line because of the limited spatial frequency information contained per line. ²³ , ⁴⁶ Instead, discrete motion states are typically assumed to cover several phase‐encoding (PE) steps ²⁴ , ⁴⁷ (referred to as a “shot”) with negligible intra‐shot motion. The JE algorithm has been shown to benefit from re‐ordering the PE steps such that all shots sample a similarly wide range of low and high spatial frequencies. ²³ , ²⁴ , ²⁵ Although optimized patterns (i.e., DISORDER ²⁵ ) have been developed, simpler patterns (e.g., interleaved pattern ²⁴ ) have been proposed, which require fewer sequence modifications.

2.2. Implementation details of motion correction methods

Developing a hybrid DL‐JE method for 3D motion correction involved implementing a 3D JE algorithm, identifying a suitable 3D DL method to be combined with the JE algorithm, and finally evaluating the methods under different data distribution shifts with simulated and in vivo data. The DL approaches were trained on a NVIDIA V100 32 GB GPU (driver v470.42.01, CUDA v11.4). To distribute the computational workload, the methods were evaluated on a separate server equipped with a NVIDIA Quadro RTX 8000 48GB GPU (driver v470.239.06, CUDA v11.4).

2.2.1. Implementation of 3D joint estimation

Referencing the algorithms described by Haskell et al. ⁴³ and Cordero‐Grande et al., ²⁴ , ²⁵ we implemented a 3D JE algorithm (“JE”) (Figure 1A). JE solves Equation 3 by alternating between estimating the image ($\widehat{m}$) and the motion trajectory ($\widehat{\theta }$) (i.e., 6 parameters per shot) using the conjugate gradient algorithm ⁵⁰ and the quasi‐Newton Broyden‐Fletcher‐Goldfarb‐Shannon algorithm. ⁵¹ To simplify the algorithm, we did not implement the regularizers, multi‐resolution strategies, or data compression strategies described by Cordero‐Grande et al. ²⁴ , ²⁵

FIGURE 1.

Overview of the 3D retrospective motion correction methods compared in this work. (A) The first method is the data consistency‐based joint estimation (JE) algorithm. (B) The second method is the stacked UNet with self‐assisted prior (UNet_mag). (C) Our proposed method combines a complex‐valued version of the UNet (UNet_com) and the JE algorithm (UNet+JE). The JE algorithm consists of motion estimation (using 1 iteration of the Broyden‐Fletcher‐Goldfarb‐Shannon [BFGS] algorithm for each motion state separately) and image reconstruction (using 3 iterations of the conjugate gradient [CG] algorithm).

Similar to Haskell et al., ⁴³ the motion parameters for different shots (${\theta }_n$) are estimated independently (Equation 4a) because of the separable nature of motion corruption. ²⁴ , ⁴³ We assumed that each shot corresponded to one motion state. The convergence criterion was empirically selected such that JE terminates when the first‐order finite difference of the total data consistency loss (Equation 5) is less than 0.1 for a minimum of 10 consecutive iterations. The algorithm was implemented in Python v3.9.7 using JAX ⁵² v0.2.27 for accelerated computation. When evaluated on one test case with 16 motion states, the typical processing time for one iteration of JE was 61 s.

$$\widehat{{\theta }_n}={\underset{{\theta }_n}{\text{argmin}}‖E_{{\theta }_n}W\widehat{m}-s‖}_2^2$$ (4a)

$$\widehat{m}={\underset{m}{\text{argmin}}‖E_{\widehat{\theta }}m-s‖}_2^2$$ (4b)

$${ℒ}_{\mathit{\text{total}}}={‖E_{\theta }W\widehat{m}-s‖}_2^2$$ (5)

Our JE implementation includes a binary matrix, $W$, which masks out regions that violate the rigid‐body motion model (Equation 4a). $W$ is set equal to the identity matrix for the simulated dataset. For the in vivo dataset, $W$ is a 3D rectangular window that was manually designed to mask out axial slices below the base of each subject's cerebellum. This does not hinder motion estimation because it does not truncate ghosting artifacts, which manifest along the PE direction (i.e., within the axial plane).

2.2.2. Implementation of stacked UNets with self‐assisted priors

The stacked UNet with self‐assisted priors ³⁶ (“UNet_mag”) (Figure 1B) was selected as a representative 3D DL method because of its state‐of‐the‐art performance and the authors' thorough ablation study of their network architecture, which is available on GitHub. UNet_mag processes 3D volumes in stacks of three adjacent sagittal slices with a stride of 1, first extracting relevant through‐plane information using attention networks before applying in‐plane motion correction using a stacked UNet architecture. The authors showed that incorporating the local through‐plane information enabled them to correct 3D volumes while using only 2D convolution layers. We compared the UNet_mag model to another state‐of‐the‐art method ³⁹ that provides volumetric correction using 3D convolution layers. We found that UNet_mag provided better image correction when both models were trained with the same dataset; we present our methods and results in the Supporting Information (Figures S7–S9). UNet_mag was implemented using TensorFlow v2.4.1. ⁵³ We trained the network with structural similarity index measure (SSIM) loss using the Adam optimizer for 100 epochs, with a batch size of 10 and a learning rate of 0.001 (additional details are provided in Section 2.3). The typical time for running UNet_mag on one volume was 4 s.

2.2.3. Implementation of UNet‐Assisted JE

Our UNet‐assisted JE algorithm (“UNet+JE”) (Figure 1C) incorporates a modified version of UNet_mag with the complex‐valued JE algorithm. We modified the network architecture to process real and imaginary components of complex‐valued data in separate channels. This involved increasing the number of input channels in the first layer and output channels in the last layer of the UNet model, and the rest of the network architecture was kept the same. The resulting network (“UNet_com”) was trained independently from JE using a loss function comprised of a weighted sum of the SSIM of the magnitude, real, and imaginary components of the model's output (Equation 6). UNet_com was trained with the same optimizer, number of epochs, batch size, and learning rate as UNet_mag (Table S2).

$$\begin{array}{cc}{ℒ}_{\mathit{com}}& =0.70*\mathit{\text{SSIM}}(\left|\widehat{m}\right|)+0.15*\mathit{\text{SSIM}}(\mathit{Re}(\widehat{m}))\\ & +0.15*\mathit{\text{SSIM}}(\mathit{Im}(\widehat{m}))\end{array}$$ (6)

$$\widehat{\theta }={\underset{\theta }{\text{argmin}}‖E_{\theta }W\left(\mathit{UNe}{t}_{\mathit{com}}(\widehat{m})\right)-s‖}_2^2$$ (7a)

$$\widehat{m}={\underset{m}{\text{argmin}}‖E_{\widehat{\theta }}m-s‖}_2^2.$$ (7b)

The artifact‐removing UNet_com was applied to each intermediate image estimate before the motion estimation step (Equation 7a). The motion estimates were then used to update the image estimate (Equation 7b). All other details of JE remained the same when implementing and evaluating UNet+JE. When evaluated on one test case with 16 motion states, the typical processing time for one iteration of UNet+JE was 65 s.

2.3. Training and testing datasets with simulated motion corruption

We generated data with simulated motion corruption using publicly available T₁‐weighted 3D MPRAGE ⁵⁴ k‐space data ⁵⁵ (the scanning parameters are summarized in Table 1). This dataset contained fully sampled, 12‐channel data from 67 participants with negligible motion artifacts. The coil sensitivity profiles were estimated using the Berkeley Advanced Reconstruction Toolbox's (BART) ESPIRiT algorithm. ⁵⁶ , ⁵⁷ Data from 60 randomly selected participants were used to generate a training dataset, while the remaining seven participants were reserved for testing. Two of the testing data were subsequently excluded because of large wrap‐around artifacts.

TABLE 1.

The scanning parameters for the dataset with simulated motion corruption (generated from a public dataset) and our acquired dataset with in vivo motion corruption.

	Dataset with simulated motion corruption	Dataset with in vivo motion corruption
TR/TE/TI (ms)	1600/2.75/650
Phase encode directions	PE 1: A/P PE 2: L/R
Resolution (mm)	1 × 1 × 1
Acceleration	None
Partial Fourier	None
Bandwidth (Hz/px)	260
Matrix size (mm)	224 × 218 × 180	256 × 256 × 192
Acquisition time (min)	6:00	6:50

Note: The image contrast of the acquired dataset with in vivo motion corruption was matched to the public dataset that was used to generate the test dataset with simulated motion corruption.

Rigid‐body head motion was simulated using the MRI signal encoding model (Equation 2). The sampling pattern had no acceleration and was designed with an interleaved pattern along the first phase encoding (PE1) direction. The PE1 steps were subdivided into 14 segments (“shots”), each one resembling a Cartesian sampling pattern with a down‐sampling factor of R = 14 (Figure S1A). Each shot was assigned a discrete motion state, and the first shot corresponded to zero motion. Given that the TR of the acquisition was 1.6 s (Table 1), this corresponded to a duration of 22.4 s per motion state. We have investigated evaluating UNet+JE with simulated test cases with a temporal resolution of 3.2 s. This is documented in the Supporting Information (Table S3 and Figures S10–S11).

The motion trajectories were generated using a pseudo‐random walk algorithm with different probabilities of moving and displacement ranges for the 6 degrees of freedom. We established four levels of motion corruption, which are described in Table S1. The parameters for generating “mild” (e.g., maximum $˜1\mathrm{mm}$ and $˜1^{°}$) and “moderate” (e.g., maximum of $˜3\mathrm{mm}$ and $˜3^{°}$) motion were determined based on typical head motion characteristics reported in previous works. ³⁶ , ⁵⁸ , ⁵⁹ We implemented “large” and “extreme” simulation parameters to simulate clinical cases with severe patient motion.

2.3.1. Training datasets with simulated motion corruption

To evaluate the impact of data distribution shifts, we generated two training datasets (Table S2). Training dataset A consisted of 240 training samples, which were generated from 2 “mild” and 2 “moderate” motion simulations applied to each of the 60 images that were reserved for training. Training dataset B includes an additional 240 training samples with “large” and “extreme” levels of motion corruption, amounting to 480 training samples. We trained the UNet_mag and UNet_com models de novo using the two training datasets separately, resulting in UNet_mag,A and UNet_mag,B as well as UNet+JE_A and UNet+JE_B. The SSIM training loss curves for all models are plotted in Figure S2.

Although Al‐Masni et al. ³⁶ had trained UNet_mag on sagittal slices from 83 participants (21 248 sagittal slices), we decided to train on axial slices because the sagittal orientation distributes non‐brain head anatomy (e.g., neck) across all slices, which is not relevant for our brain‐specific application. To account for memory constraints and to reduce the number of slices containing non‐brain anatomy, we trained only on the central 128 slices out of 256 axial slices per volume, resulting in 30 720 and 61 440 axial slices for training datasets A and B.

2.3.2. Testing dataset with simulated motion corruption

To generate the testing dataset (n = 40), we imposed eight motion trajectories onto each of the five images reserved for testing (Table S2). For each test subject, we generated two motion trajectories from each of the four levels of motion corruption. We evaluated five different motion correction methods on this testing dataset: JE, UNet_mag,A, UNet_mag,B, UNet+JE_A, and UNet+JE_B.

2.4. Testing dataset with in vivo motion corruption

Ten healthy volunteers (age range: 24–47; 4 females) participated in this study with informed consent. The study conformed to the Declaration of Helsinki and was approved by the Research Ethics Board of the University Health Network. All scans were performed on a Siemens MAGNETOM Prisma 3 T scanner (Siemens Healthineers) at the Slaight Family Centre for Advanced MRI (Toronto Western Hospital). We acquired T₁‐weighted 3D MPRAGE data with scanning parameters that were matched to the dataset with simulated motion (Table 1), except for a larger FOV to minimize wrap‐around artifacts.

We acquired a pair of images from each subject: a reference scan where the subject was asked to be stationary and a motion corrupted scan. The participants were trained to identify their neutral, maximum (upward nod), and minimum (downward nod) head positions that they felt comfortable performing within the head coil. The participants were supervised while they practiced a sinusoidal trajectory that spanned these three positions across 16 timepoints (Figure S3). This corresponded to a duration of 25.6 s per motion state. During the scan, the participants wore MRI‐compatible headphones and were verbally cued to switch between the head positions at the onset of each shot. Participants were asked to maintain their pose for the duration of the shot to minimize intrashot motion.

The reference scans were acquired with a linear order of PE, whereas the motion‐corrupted scans were acquired with a custom interleaved pattern along the PE1 direction (Figure S1B). Data was acquired using a 20‐channel head and neck coil, and all participants were placed in a head‐first supine pose. The coil sensitivity profiles were estimated using BART's ESPIRiT algorithm. ⁵⁶ To focus on the data distribution shift of the in vivo testing dataset relative to training dataset A, we evaluated three different motion correction methods on this testing dataset: JE, UNet_mag,A, and UNet+JE_A.

2.5. Image quality and convergence rate assessment

We evaluated the performance of the different methods by computing the image percent error ($ϵ$, Equation 8) and $\text{SSIM}$, ⁶⁰ which are standard full‐reference image quality metrics (IQM). Additionally, gray‐matter (GM) segmentations were evaluated to assess the recovery of gray‐white matter boundaries. ³⁴ , ³⁸ , ⁶¹ This was done using FSL FAST ⁶² because its image‐based algorithm better reflects the impact of motion artifacts than other algorithms that use additional priors (e.g., SPM). ⁶³ The GM segmentation quality was quantified using Dice coefficients ($\mathit{\text{Dice}}$). Furthermore, we compared the convergence rates of JE and UNet+JE by evaluating their runtimes. Finally, we compared the motion estimation accuracy of JE and UNet+JE_A/UNet+JE_B for the simulated dataset.

To isolate the impact of motion artifacts within the brain, all images underwent rigid‐body registration ⁶⁴ to their corresponding reference images and underwent skull‐stripping ⁶⁵ before evaluating IQMs. All figures presenting image results (e.g., Figures 3, 4, 8) are plotted post‐registration. To account for the varying image quality across different scans, we reported the relative change of each IQM (Equation 9). We performed Friedman tests against the null hypothesis that all methods provided the same median relative change in each IQM. We then made pairwise comparisons by performing paired 2‐tailed sign tests with Bonferroni correction.

$$ϵ=\frac{{‖m_{\mathit{\text{final}}}-m_{\mathit{ref}}‖}_2}{{‖m_{\mathit{ref}}‖}_2}*100$$ (8)

$${\Delta }_{\mathit{IQM}}=\frac{\mathit{IQM}\left(m_{\mathit{\text{final}}}\right)-\mathit{IQM}\left(m_{\mathit{\text{initial}}}\right)}{\mathit{IQM}\left(m_{\mathit{\text{initial}}}\right)}$$ (9)

FIGURE 3.

Results of a test case with simulated “moderate” motion corruption, which falls within the distribution of both training datasets A and B (please see Table S2). The maximum translation and rotation magnitudes are provided on the left of the figure. The top row includes the motion‐free reference image and corrupted image, as well as the results of joint estimation (JE), UNet_mag,A, and UNet+JE_A. The bottom row shows the results of UNet_mag,B and UNet+JE_B. The global percent error ($ϵ$) and structural similarity index measure (SSIM) values are included. The zoomed‐in inset figures illustrate how well the different methods can recover intricate cortical folds, while the red arrows highlight residual artifacts.

FIGURE 4.

Results of a test case with simulated “large” motion corruption, which only falls within the distribution of training dataset B (please see Table S2). The maximum translation and rotation magnitudes are provided on the left of the figure. The top row includes the motion‐free reference image and corrupted image, as well as the results of joint estimation (JE), UNet_mag,A, and UNet+JE_A. The bottom row shows the results of UNet_mag,B and UNet+JE_B. The global percent error ($ϵ$) and structural similarity index measure (SSIM) values are included. The zoomed‐in inset figures illustrate how well the different methods can recover intricate cortical folds. The red arrows highlight subtle features that were well‐recovered by JE and UNet+JE compared to UNet_mag (for both training datasets A and B).

FIGURE 8.

Results of two test cases from the dataset with in vivo motion corruption, featuring moderate (A) and large (B) discrete head motion. For UNet_mag and UNet+JE, we only evaluated results for the models that were trained with training dataset A. The maximum motion magnitudes that were estimated by UNet+JE_A are included on the left of each row; the true motion trajectories were not measured when acquiring the dataset. In both (A,B), the top and bottom rows display axial and coronal slices of the motion‐free reference image, corrupted image, and the results of the three motion correction methods. The global percent error ($ϵ$), structural similarity index measure (SSIM) and Dice coefficients are included for each test case.

Figure 2 compares the $\text{SSIM}$ distribution of training datasets A and B with the simulated and in vivo test datasets. The simulated test data that fall within the distribution of training dataset A had a median SSIM ($\widetilde{\mathit{\text{SSIM}}}$) and interquartile range (IQR) of 0.85 and 0.12; the remaining simulated test data had a $\widetilde{\mathit{\text{SSIM}}}$ of 0.49 (IQR: 0.37). The in vivo testing dataset had a $\widetilde{\mathit{\text{SSIM}}}$ of 0.60 (IQR: 0.20). Please note that, for the in vivo data, motion induces effects on the MRI signal that are not modeled in Equation 1 (e.g., changes in B₀, B₁ ⁺, and spin history). Therefore, the $\mathit{\text{SSIM}}$ values could be lowered by these effects.

FIGURE 2.

Comparing the training and testing dataset's structural similarity index measure (SSIM) distributions. UNet_A and UNet_B correspond to the UNet models that were trained with training dataset A and B, respectively (please see Table S2). The SSIM values of training datasets A and B are displayed as orange and blue histograms, respectively. The scatterplot above the histogram displays the image quality of each simulated (n = 40) and in vivo (n = 10) test case. The test cases with simulated motion corruption are color‐coded with respect to their simulation parameters.

3. RESULTS

3.1. Dataset with simulated motion corruption

Figures 3 and 4 shows the results of two representative test cases generated with “moderate” (Figure 3) (maximum translation/rotation: $\left[1.37\mathrm{mm},{1.65}^{°}\right]$) and “large” (Figure 4) (maximum translation/rotation: $\left[4.19\mathrm{mm},{1.68}^{°}\right]$) simulation parameters. In each figure, the two left columns show the data before and after simulated motion corruption. The three right columns show the results of using JE, UNet_mag, and UNet+JE, while the top and bottom rows compare the methods when trained with training datasets A and B, respectively. A comparison of the corresponding GM segmentations for each result is included in the Supporting Information (Figures S4 and S5). Coronal and axial slices are presented to highlight the 3D motion correction performance of all methods. The inset zoomed‐in figures highlight intricate gray‐white matter boundaries.

In Figure 3, the corrupted image contained prominent ghosting artifacts $(\mathrm{ϵ}:11.60\%,\text{SSIM}:0.83)$, which led to false‐positive/negative labels in its GM segmentation (Figure S4) ($\text{Dice}:0.85$). Although UNet_mag,A $(\mathrm{ϵ}:8.73\%,\text{SSIM}:0.92,\text{Dice}:0.95)$ and UNet_mag,B $(\mathrm{ϵ}:13.55\%,\text{SSIM}:0.91,\text{Dice}:0.89)$ were able to remove ghosting artifacts, both methods produced residual artifacts and blurring as well as apparent changes to the brightness/contrast of the image (particularly for UNet_mag,B, which led to worsened $\mathrm{ϵ}$ values). In contrast, JE $(\mathrm{ϵ}:4.50\%,\text{SSIM}:0.98,\text{Dice}:0.97)$, UNet+JE_A $(\mathrm{ϵ}:3.81\%,\text{SSIM}:0.98,\text{Dice}:0.97)$, and UNet+JE_B $(\mathrm{ϵ}:2.02\%,\text{SSIM}:0.99,\text{Dice}:0.99)$ all accurately recovered the image.

In Figure 4 and its corresponding GM segmentations in Figure S5, the anatomical features in the corrupted image have largely been obscured $(ϵ:20.20\%,\mathit{\text{SSIM}}:0.78,\mathit{\text{Dice}}:0.84)$. UNet_mag,A $(ϵ:22.80\%,\mathit{\text{SSIM}}:0.66,\mathit{\text{Dice}}:0.78)$ smoothened the appearance of repeated edges, but significant residual artifacts and blurring remained. UNet_mag,B $(ϵ:25.06\%,\mathit{\text{SSIM}}:0.66,\mathit{\text{Dice}}:0.79)$ further smoothened the image while preserving the overall contour of the brain to a greater degree than UNet_mag,A. However, UNet_mag,B did not accurately recover the underlying anatomical features. The JE $(ϵ:9.87\%,\mathit{\text{SSIM}}:0.94,\mathit{\text{Dice}}:0.94)$, UNet+JE_A $(ϵ:9.84\%,\mathit{\text{SSIM}}:0.94,\mathit{\text{Dice}}:0.93)$ and UNet+JE_B $(ϵ:9.86\%,\mathit{\text{SSIM}}:0.94,\mathit{\text{Dice}}:0.93)$ algorithms recovered the image with high fidelity, although residual artifacts remained in contrast to the near‐perfect image recovery that these methods achieved in Figure 3.

Figures 5 and 6 summarize the performance of all methods on subsets of the simulated testing datasets that are within and outside of the distribution of training dataset A, respectively. In both figures, the boxplots in (A–C) display the relative change in percent error (${\mathrm{\Delta }}_rϵ$), SSIM (${\mathrm{\Delta }}_r\mathit{\text{SSIM}}$), and Dice coefficient (${\mathrm{\Delta }}_r\mathit{\text{Dice}}$). We evaluated the median relative change ($\widetilde{{\mathrm{\Delta }}_{\mathrm{r}}}$) for each IQM, which are reported below each corresponding boxplot. Additionally, the results of the 2‐tailed sign test are annotated.

FIGURE 5.

Comparing the performance of all five methods across the subset of the simulated testing dataset that falls within the distribution of training dataset A (n = 20) (refer to Table S2). (A–C) compare the distribution of relative change (${\Delta }_r$) in percent error ($ϵ$), structural similarity index measure (SSIM), and Dice, respectively. (D) compares the convergence rates of joint estimation (JE), UNet+JE_A and UNet+JE_B for the simulated test dataset. The boxplots are annotated with p‐values from evaluating paired 2‐tailed sign test samples with Bonferroni correction. The median of each boxplot is displayed below each subfigure.

FIGURE 6.

Similar to Figure 5, this figure compares the performance of all methods across the simulated test cases that fall outside of the distribution of training dataset A (n = 20) (refer to Table S2). (A–C) compare the distribution of relative change (${\Delta }_r$) in percent error ($ϵ$), structural similarity index measure (SSIM), and Dice, respectively. (D) compares the convergence rates of joint estimation (JE) and UNet+JE for the simulated test dataset. The boxplots are annotated with p‐values from evaluating paired 2‐tailed sign test samples with Bonferroni correction. The median of each boxplot is displayed below each subfigure.

First, considering only the methods trained on training dataset A, JE and UNet+JE_A consistently outperformed UNet_mag,A with strong statistical significance ($p<5.0·{10}^{-4}$ for all IQMs). In particular, UNet_mag,A provided minimal improvement in the accuracy of the GM segmentations (${\tilde{\mathrm{\Delta }}}_r\mathit{\text{Dice}}$: 0.05 and 0.03 in Figures 5C and 6C) compared to JE (${\tilde{\mathrm{\Delta }}}_r\mathit{\text{Dice}}$: 0.12 and 0.37) and UNet+JE_A (${\tilde{\mathrm{\Delta }}}_r\mathit{\text{Dice}}$: 0.12 and 0.33). Although JE and UNet+JE_A did not differ in image quality, UNet+JE_A converged significantly faster than JE ($p<5.0·{10}^{-4}$), leading to a speed up in runtime by a median factor of 2.35 (Figure 5D) and 3.20 (Figure 6D) for test cases with “small/moderate” and “large/extreme” motion corruption, respectively.

When comparing the impact of training distributions on UNet_mag,A/UNet_mag,B and UNet+JE_A/UNet+JE_B, there were generally no statistically significant differences in image quality for either subset of the testing dataset (Figures 5A–C and 6A–C). The only exceptions were for UNet+JE_A/UNet+JE_B for ${\mathrm{\Delta }}_rϵ$ with “mild/moderate” test cases (Figure 5A) ($W=5.00,p=2.07·{10}^{-1}$) and for ${\mathrm{\Delta }}_r\mathit{\text{Dice}}$ with “large/extreme” test cases (Figure 6C) ($W=4.00,p=5.77·{10}^{-1}$). There were no statistically significant differences in the convergence rates of UNet+JE_A and UNet+JE_B across the entire testing dataset (Figures 5D and 6D). Similar to UNet+JE_A, UNet+JE_B provided greater speed up in runtime for “large/extreme” motion cases (median factor: 3.8) than for “small/moderate” motion cases (median factor: 2.0).

To further investigate the impact of training distributions on UNet+JE_A/UNet+JE_B, Figure 7 compares the motion estimation error ($x_{\mathit{est}}-x_{\mathit{\text{true}}}$) of JE, UNet+JE_A, and UNet+JE_B. Although all methods generally estimated the motion parameters with high accuracy (median errors are on the order of ${10}^{-3}$ mm and degrees (deg) for all methods), only UNet+JE_B led to errors greater than 1.0 mm or 1.0 deg. These cases appear to be outliers and generally occurred for larger motion parameters (e.g., translations >5 mm; rotations >5 deg). Figure S6 presents one such test case where UNet+JE_B converged to a local minimum.

FIGURE 7.

The error (${\mathrm{x}}_{\mathrm{est}}-{\mathrm{x}}_{\text{true}}$) of the motion parameters estimated by joint estimation (JE), UNet+JE_A, and UNet+JE_B across all n = 40 simulated test cases. Rows (A,B) plot the distribution of the accuracy of the estimated translation and rotation parameters against the ground truth motion parameters, which have been grouped to the nearest 1 mm or 1 deg. along the x‐axis. All three methods generally estimated motion parameters with high accuracy (median accuracies of 0.0 for all methods). However, only UNet+JE_B exhibited outlier cases where motion parameters were mis‐estimated by >1.0 mm or >1.0 deg. Please refer to Figure S6 for an example of a test cases where UNet+JE_B converged to local minima.

3.2. Dataset with in vivo motion corruption

Figure 8 shows two test cases from the dataset with in vivo motion corruption. In Figure 8A, the image contained moderately strong ghosting artifacts $(ϵ:24.84\%,\mathit{\text{SSIM}}:0.72,\mathit{\text{Dice}}:0.80)$. Although UNet_mag,A reduced their appearance, residual artifacts and blurring remained $(ϵ:30.11\%,\mathit{\text{SSIM}}:0.75,\mathit{\text{Dice}}:0.79)$. Although JE $(ϵ:18.16\%,\mathit{\text{SSIM}}:0.91,\mathit{\text{Dice}}:0.95)$ and UNet+JE_A $(ϵ:18.69\%,\mathit{\text{SSIM}}:0.91,\mathit{\text{Dice}}:0.95)$ provided more accurate image recovery, they both contain mild residual artifacts. This led to slightly lower improvements in $ϵ$, $\mathit{\text{SSIM}}$, and $\mathit{\text{Dice}}$ compared to the near‐perfect correction of the simulated test dataset (e.g., Figure 3). JE and UNet+JE_A converged after 204 and 84 min, respectively.

In Figure 8B, the corrupted image contained strong artifacts $(ϵ:37.87\%,\mathit{\text{SSIM}}:0.49,\mathit{\text{Dice}}:0.66)$. Using UNet_mag,A on its own provided some improvement in sharpening the appearance of the ventricles and the cortical folds of the posterior lobes, however, anatomical features remained largely obscured $(ϵ:41.08\%,\mathit{\text{SSIM}}:0.54,\mathit{\text{Dice}}:0.70)$. Although the outputs of JE $(ϵ:22.91\%,\mathit{\text{SSIM}}:0.85,\mathit{\text{Dice}}:0.90)$ and UNet+JE_A $(ϵ:22.25\%,\mathit{\text{SSIM}}:0.85,\mathit{\text{Dice}}:0.90)$ contained obvious residual artifacts, they were able to recover more of the underlying anatomical features, leading to significant improvements in the accuracy of their GM segmentations. JE and UNet+JE_A converged after 533 and 119 min, respectively.

Figure 9 compares the three methods across the entire in vivo dataset. In Figure 9A–C, JE and UNet+JE_A outperformed UNet_mag,A with strong statistical significance ($p>1.0·{10}^{-3}$ in all 3 IQMs). Notably, UNet_mag,A consistently provided negligible improvement in SSIM (${\tilde{\mathrm{\Delta }}}_r\mathit{\text{Dice}}$: 0.05, IQR: 0.07) and GM segmentation quality (${\tilde{\mathrm{\Delta }}}_r\mathit{\text{Dice}}$: 0.03, IQR: 0.04). Although JE and UNet+JE_A did not statistically differ in any IQM, Figure 9D shows that UNet+JE_A converges significantly faster ($W=4.1,p=2.86·{10}^{-3}$), leading to a reduction in runtime by a median factor of 4.05.

FIGURE 9.

The performance of the three motion correction methods across the test dataset with in vivo motion corruption (n = 10). (A–C) compare the distribution of relative change (${\Delta }_r$) in percent error ($ϵ$), structural similarity index measure (SSIM), and Dice, respectively. (D) compares the convergence rates of joint estimation (JE) and UNet+JE for the in vivo test dataset. The boxplots are annotated with p‐values from evaluating paired 2‐tailed sign test samples with Bonferroni correction. The median of each boxplot is displayed below each subfigure.

4. DISCUSSION

Although recent RMC methods ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ have demonstrated the benefits of leveraging both DL and physical modeling, these methods are limited to 2D applications and their generalizability to different levels of motion corruption has yet to be evaluated. UNet+JE is the first hybrid DL‐JE algorithm that can provide 3D motion correction and that has been assessed under various extents of motion corruption relative to its training distribution.

For the simulated dataset, we found that UNet+JE_A/UNet+JE_B consistently outperformed UNet_mag,A/UNet_mag,B in all metrics, even under strong distribution shifts. This suggests that the JE subcomponent of UNet+JE conferred greater robustness to the overall algorithm. Furthermore, we found that UNet+JE_A/UNet+JE_B converged faster than JE, which corroborates findings reported by Haskell et al. ⁴³ In particular, we found that both UNet+JE_A and UNet+JE_B provided greater reduction in runtime for “large/extreme” motion cases than for “mild/moderate” cases, which indicates that UNet+JE offers greater performance gains for more heavily corrupted test cases, regardless of its training distribution. These results were echoed in the in vivo dataset, where JE and UNet+JE_A outperformed UNet_mag,A in all IQMs, and UNet+JE_A converged faster than JE.

Although UNet+JE_A and UNet+JE_B did not have any statistically differences in IQMs, we found that only UNet+JE_B yielded outlier cases where the algorithm converged to local minima (e.g., Figure S6). This suggests that training with the augmented dataset B led to more “hallucinated” image features and blurring that could hinder the joint estimation algorithm. It remains an open question to determine how to optimally train a neural network to generalize over a large distribution of test cases, particularly for clinical applications (e.g., more training samples; transfer learning from “specialized” models tailored to correcting large/extreme motion, etc.).

We found that UNet_mag,A/UNet_mag,B were able to reduce the appearance of ghosting artifacts (particularly in the background) while preserving the overall contour of the skull. However, both models struggled to reliably recover image features and frequently contained residual artifacts and image blurring, even in test cases that fell within the models' training distributions (e.g., Figure 3). On a case‐by‐case basis for “large” and “extreme” motion corruption, UNet_mag,B appeared to remove more ghosting artifacts than UNet_mag,A, albeit with a greater extent of image blurring and “hallucinated” image features, however, both models were still unable to recover the actual underlying image features.

Despite these discrepancies, the ϵ, SSIM and Dice values were similar between the image outputs of both UNet_mag models with no statistically significant differences. We believe that this is because of limitations of these full‐reference IQMs. Although these IQMs can be used to describe how dissimilar the image outputs are relative to their corresponding reference data, they do not directly describe how the image outputs differ from each other. In particular, while RMC outputs with “better” IQM scores (e.g. ε $\to 0$, SSIM and Dice $\to 1$) are close to their reference image and, therefore, necessarily more similar to each other, image outputs that are highly dissimilar between each other (e.g., in terms of smoothness) can share similarly poor IQM values (e.g., $ϵ>20\%$, SSIM and Dice <0.80) with respect to their reference image (e.g., UNet_mag,A and UNet_mag,B in Figure 4, both of which share an SSIM of 0.66). We believe that reference‐free IQMs that are tailored to quantifying salient image textures for our application (i.e., repeated edges, smoothing, and hypo‐intensities) would enable better comparisons between the different RMC methods. ⁶⁶

Another major limitations of our work is that UNet+JE requires data to be acquired with an interleaved PE1 ordering rather than the standard sequential ordering, which is known to lead to unfavorable convergence properties for JE. ²³ , ²⁴ Although previous works have proposed optimized sampling patterns for JE (i.e., DISORDER ²⁵ ), they require more sequence modifications and significantly amplify the appearance of motion artifacts. ²⁷ Although Levac et al. ⁴⁶ explored using unsupervised learning to increase robustness to different sampling patterns, their Cartesian data was still simulated with interleaved sampling. We are currently exploring using regression neural networks for motion estimation ⁴⁸ to circumvent the limitations of classical optimizers.

Additionally, UNet+JE currently assigns one motion state to each shot, similar to previous JE methods. ⁴⁷ We had instructed our participants to move accordingly to minimize intrashot motion. In our Supporting Information, we outlined and validated a multi‐resolution version of the UNet+JE algorithm, and we found that the modified algorithm was able to consistently improve image quality for simulated test cases at a temporal resolution of 3.2 s. To be able to estimate arbitrary motion trajectories at finer temporal resolutions, we will explore shot rejection ²⁴ and intra‐shot ⁴³ , ⁴⁶ motion estimation approaches. Although data consistency‐based JE methods require modified k‐space trajectories and assumptions regarding discrete head motion, the results of our in vivo study and preliminary simulation results with intrashot motion show that UNet+JE can still improve image quality despite violations to the method's assumptions.

Finally, in clinical settings, UNet+JE would need to preserve anatomical features (including pathology) with high fidelity, depending on the use case. Although UNet+JE has yet to be tested on clinical data, its performance on distribution shifts with respect to severe motion artifacts is promising. UNet+JE consistently yielded better image quality than the initial corrupted image for all test cases. Although residual artifacts remain visible for more severely corrupted cases (e.g., Figure 4, Figure 8B), UNet+JE recovered anatomical features in images that were otherwise diagnostically unusable. Previous studies ⁶¹ , ⁶⁷ have been conducted to assess the utility of RMC methods on clinical data. We will explore transfer learning ⁶⁸ approaches to ensure that UNet+JE generalizes well for clinical data.

5. CONCLUSIONS

We have presented a novel retrospective motion correction method (UNet+JE) that combines a neural network (UNet_mag) with a physics‐based algorithm (JE). UNet+JE is the first hybrid DL‐JE method that has been applied to 3D volumetric correction. When assessed on simulated and in vivo data with varying levels of motion corruption, we found that UNet+JE was more robust to data distribution shifts than UNet_mag, while converging to similar image outputs as JE in fewer iterations. Our findings demonstrate that it is worthwhile to combine DL and classical MRI signal modeling to leverage their respective benefits.

CONFLICT OF INTEREST

Z.W. now works for Siemens Healthineers, Oakville, ON, Canada.

Supporting information

Figure S1. Illustrations of the interleaved sampling patterns used to generate the simulated (A) and in vivo (B) datasets. In both subfigures, each column (delineated by dashed lines) plots the sampling pattern of the first phase encoding (PE1) and second phase encoding (PE2) steps for a single shot. In (A), the 218 PE1 steps are binned into 14 shots. The first 8 shots contain 16 PE1 steps, while the last 6 shots contain 15 PE1 steps; all shots have an effective undersampling factor of R = 14. In (B), the 256 PE1 steps are binned into 16 shots, each containing 16 equidistant PE1 lines.

Figure S2. Plots of the training loss (1‐SSIM) curves for all UNet models. The top row corresponds to the magnitude‐only UNet models (UNet_mag), while the bottom row displays the complex‐valued UNet models (UNet_com). The left and right columns display the results of training with Training Dataset A (UNet_mag,A, UNet_com,A) and Dataset B (UNet_mag,B, UNet_com,B), respectively. The blue and orange curves correspond to the loss values for the training and validation datasets. All UNet models were trained for 100 epochs and displayed stable convergence.

Figure S3. Illustrating the targeted head motion trajectory performed by participants for the in vivo data acquisitions. Participants were instructed to perform pitch rotations (i.e., rotations about the left–right axis). The gray dashed lines demarcate the minimum (downward nod), neutral, and maximum (upward nod) head positions which the participants were trained to identify before the scan. The blue solid lines indicate the relative head positions that the participants were asked to maintain during each shot, while the dotted blue lines indicate transitions between head positions.

Figure S4. The gray matter segmentations corresponding to each image displayed in Figure 3 (main text). For each result, the top image displays gray matter segmentations (bluish‐gray) overlayed on top of their corresponding anatomical images; the global Dice coefficients are included. The bottom image plots the difference image of each gray matter segmentation with respect to the reference segmentation; false‐positive and false‐negative labels are displayed as red and navy blue labels, respectively.

Figure S5. Similar to Figure S4, this figure shows the gray matter segmentations corresponding to each image displayed in Figure 4 (main text). For each result, the top image displays gray matter segmentations (bluish‐gray) overlayed on top of their corresponding anatomical images; the global Dice coefficients are included. The bottom image plots the difference image of each gray matter segmentation with respect to the reference segmentation; false‐positive and false‐negative labels are displayed as red and navy blue labels, respectively.

Figure S6. An example of a simulated test case where UNet+JE_B converged to a local minimum. (A) A test case with “extreme” motion corruption. The percent error $(ϵ)$ and structural similarity index $(\mathit{\text{SSIM}})$ are provided. (B) The relative data consistency (DC) loss is plotted for the JE, UNet+JE_A and UNet+JE_B algorithms. While JE and UNet+JE_A converge to similar DC values (0.06), UNet+JE_B resulted in a larger DC value (0.17). The dashed vertical lines denote the iteration at which each algorithm converged. (C) Comparing the image estimates provided by UNet_mag,A, UNet_mag,B, as well as the image and motion estimates provided by UNet+JE_A and UNet+JE_B. In the motion trajectory plots, the dashed lines correspond to the groundtruth parameters (“T^GT”, “R^GT”), while the solid lines are the estimated parameters (“T^est”, “R^est”). The red dashed rectangle highlights motion states that were mis‐estimated by UNet+JE_B.

Figure S7. Comparing the training loss curves of the UNet_{Al‐Masni et al} (Al‐Masni et al. NeuroImage 2021; left column) and the UNet_{Duffy et al} (Duffy et al. Nature 2021; right column) models. The former was trained with structural similarity (SSIM) loss, while the latter was trained with mean absolute error (MAE) loss. Both SSIM and MAE loss values were evaluated during all training sessions and displayed for completeness. Both models were trained with magnitude‐only data from Training Dataset A; the dataset was transformed into 3D image patches to be compatible with the 3D convolutions used by UNet_{Duffy et al}.

Figure S8. Comparing the performance of UNet_{Al‐Masni et al} (corresponds to “UNet_mag,A” in main text) and UNet_{Duffy et al} with the same simulated test cases from Figures 3 and 4. In (A), this test case falls within the training distribution of both models. Both methods improve the image quality of the corrupted scan, as reflected by the improved global SSIM values and the reduction of ghosting artifacts in the inset zoomed‐in ROIs. In (B), this test case falls outside of the training distribution of both UNet models. While both methods produced images with residual artifacts and strong image blurring, UNet_{Duffy et al} produces stronger image distortions, leading to worse $ϵ$ and $\mathit{\text{SSIM}}$ than the initial corrupted image.

Figure S9. Comparing UNet_{Al‐Masni et al} and UNet_{Duffy et al} across the simulated testing dataset (N = 40). Plots (A) – (C) compare the distribution of relative change (${\Delta }_r$) in percent error ($ϵ$), SSIM, and Dice, respectively. The boxplots are annotated with p‐values from evaluating paired two‐tailed sign test samples, and the median of each boxplot is displayed below each subfigure. Based on the median relative changes in each IQM, the UNet_{Al‐Masni et al} outperformed UNet_{Duffy et al} in all 3 IQMs with strong statistical significance.

Figure S10. (A) A test case with simulated intrashot motion corruption. Motion parameters were simulated at a temporal resolution of 3.2 seconds. This test cases corresponds to Subject 5 in Figure S11. (B) Comparing the image output and estimated motion parameters at Level 1 and Level 4 of the multiresolution UNet+JE algorithm. The top row displays the intermediate image and motion estimates at Level 1, in which the translation and rotation parameters are estimated at a temporal resolution of 25.6 seconds. The bottom row displays the final image and motion estimates at Level 4 of the algorithm, where motion parameters are estimated with a resolution of 3.2 seconds. The percent error $(ϵ)$ and structural similarity index $\mathit{\text{SSIM}}$ are included. (C) The change in total data consistency loss across all iterations of the multiresolution UNet+JE algorithm.

Figure S11. Displaying the change in image quality across the multi‐resolution UNet+JE algorithm for the 5 test cases with simulated intrashot motion. Plots (A) – (C) show the RMSE, SSIM, and Dice, respectively. While the RMSE trajectories display non‐monotonic behavior for certain test cases, the SSIM and Dice trajectories display more monotonic increase in image quality across each level.

Table S1. A summary of the motion corruption parameters for the training dataset. Motion trajectories were generated as a pseudo‐random walk from shot to shot, with different probabilities and step intervals assigned to each of the 3 translational and 3 rotational degrees of freedom.

Table S2. Details of the two datasets generated for training the UNet models. Training Dataset A consists of 240 samples, generated from data of 60 subjects that each underwent 2 simulations with “mild” motion corruption and 2 simulations with “moderate” motion corruption. Similarly, Training Dataset B consists of 480 samples generated from the same 60 subjects, now augmented to include “large” and “extreme” levels of motion corruption. During training, the number of epochs, the learning rate, and the batch size were kept the same between the different training sessions with Dataset A and B.

Table S3. Description of the multi‐resolution UNet+JE algorithm. The algorithm is divided into 4 levels, where each level carries out motion estimation and correction at increasingly higher temporal resolutions. Level 1 is equivalent to the standard UNet+JE_A presented in the main text.

Nghiem B., Wu Z., Kashyap S., Kasper L., and Uludağ K., “A network‐assisted joint image and motion estimation approach for robust 3D MRI motion correction across severity levels,” Magnetic Resonance in Medicine 95, no. 1 (2026): 363–381, 10.1002/mrm.70052.

DATA AVAILABILITY STATEMENT

The source code is open available on GitHub at https://github.com/BRAIN‐TO/PyMoCo_v2. The data that support the findings of this study are available from the corresponding author on reasonable request.