MCP-Net: Inter-frame Motion Correction with Patlak Regularization for Whole-body Dynamic PET

Abstract

Inter-frame patient motion introduces spatial misalignment and degrades parametric imaging in whole-body dynamic positron emission tomography (PET). Most current deep learning inter-frame motion correction works consider only the image registration problem, ignoring tracer kinetics. We propose an inter-frame Motion Correction framework with Patlak regularization (MCP-Net) to directly optimize the Patlak fitting error and further improve model performance. The MCP-Net contains three modules: a motion estimation module consisting of a multiple-frame 3-D U-Net with a convolutional long short-term memory layer combined at the bottleneck; an image warping module that performs spatial transformation; and an analytical Patlak module that estimates Patlak fitting with the motion-corrected frames and the individual input function. A Patlak loss penalization term using mean squared percentage fitting error is introduced to the loss function in addition to image similarity measurement and displacement gradient loss. Following motion correction, the parametric images were generated by standard Patlak analysis. Compared with both traditional and deep learning benchmarks, our network further corrected the residual spatial mismatch in the dynamic frames, improved the spatial alignment of Patlak K_i/V_b images, and reduced normalized fitting error. With the utilization of tracer dynamics and enhanced network performance, MCP-Net has the potential for further improving the quantitative accuracy of dynamic PET. Our code is released at https://github.com/gxq1998/MCP-Net.

1. Introduction

Whole-body dynamic positron emission tomography (PET) using 2-deoxy-2-[¹⁸F]fluoro-D-glucose (FDG) has emerged as a more accurate glycolytic metabolism measurement in clinical and research protocols [7] than static PET due to the time dependency of radiotracer uptake [21]. In continuous-bed-motion mode (CBM) [17], an image sequence is typically collected for 90 min starting at the tracer injection and then fitted with voxel-wise kinetic modeling for parametric imaging [22]. For FDG that typically follows irreversible 2-tissue compartmental model, the Patlak plot [18] is a simplified linear model for parametric estimation. The Patlak slope K_i [18], the net uptake rate constant, has been shown to improve oncological lesion identification with higher tumor-to-background and contrast-to-noise ratio [8].

However, patient motion is unpreventable during the long scanning period, which has a harmful impact on parametric imaging [14]. The inter-frame motion caused by body movement and the changes in the long-term cardiac and respiratory motion patterns lead to increased parameter estimation errors. In the whole-body scope, subject motion is non-rigid, complicated, and unpredictable, and the simultaneous presence of high uptake and low uptake organs makes motion correction substantially harder. The significant cross-frame tracer distribution variation also complicates the motion estimation and correction.

Recent breakthroughs in deep learning based image registration have achieved superior performance and computational efficiency over conventional non-rigid registration methods. Spatial-temporal network structures have been applied in motion-related dynamic image sequence registration [10,13,19,24], outperforming single image-pair networks [1,23]. Although multiple-frame analysis has the potential of improving dynamic PET motion correction, most methods still only consider image registration without utilizing tracer kinetics, which could lead to residual misalignment especially for images with low uptake [11]. While the incorporation of kinetic modeling into the registration has been proposed in dynamic contrast-enhanced magnetic resonance imaging [2,15], dynamic PET typically involves more complex tracer dynamics. Pharmacokinetics has been utilized in PET reconstruction [5] and brain registration [11], but such implementation has not been introduced in either deep learning or the whole-body scope.

In this work, we propose an inter-frame Motion Correction framework with Patlak regularization (MCP-Net) for whole-body dynamic PET. An analytical Patlak fitting module is integrated into the framework with Patlak loss penalization introduced to the loss function. We evaluated the proposed model using a simulated dataset and a 9-fold cross-validation on an internal real-patient dataset and compared with traditional and deep learning benchmarks.

2. Methods

2.1. Dataset and Pre-processing

Five healthy and 22 cancer subjects (27 in total) were enrolled at Yale PET Center, with obtained informed consent and Institutional Review Board approval. Each subject underwent a 90-min dynamic multi-frame whole-body PET scan using CBM protocol on a Siemens Biograph mCT with FDG bolus injection [16], obtaining 19 consecutive reconstructed whole-body frames (4 × 2 min and 15 × 5 min) as well as the individual input function. The details of the image acquisition are in Supplementary Fig. S1. In the 22 cancer patients, 57 hypermetabolic regions of interest (ROIs) were selected by a nuclear medicine physician for additional evaluation. Dynamic frames were calibrated to have units of standardized uptake values (SUV). Due to GPU RAM limitation, the input frames were downsampled by a factor of 4 and zero-padded to the same resolution of 128 × 128 × 256 for all the deep learning approaches. The displacements were upsampled back to the original resolution using a spline interpolation (order = 3) before warping the original frames.

2.2. Proposed Network

The MCP-Net consists of a spatial-temporal motion estimation network, a spatial transformation module, and an analytical Patlak fitting module (see Fig. 1).

Fig. 1.

The overall structure of the proposed motion correction framework MCP-Net.

Spatial-temporal Motion Estimation Network.

The dual-channel motion estimation input is a moving frame sequence, each concatenated with the reference frame. To decrease the driving force of high-uptake voxels while also minimize saturation in the local normalized cross-correlation (NCC) loss computation, an intensity cutoff layer is first applied with threshold of SUV = 2.5 and Gaussian noise (σ = 0.01) added to thresholded voxels [10]. To handle multiple-frame spatial-temporal analysis, the motion estimation module is a concatenated 3-D U-Net [6] with shared weights and a convolutional long short-term memory [20] layer integrated at the bottleneck [10]. This enables information extraction across both adjacent and non-adjacent frames, especially effective for long-duration motion and tracer changes. The number of both encoding and decoding U-Net levels is set to 4, with input sequence length N = 5. With the voxel-wise displacement field estimations, the spatial transformation layer warps the input frames and outputs motion compensated frames.

The Analytical Patlak Fitting Module.

The subsequent analytical Patlak fitting module estimates parametric imaging results using the Patlak plot [18] after a starting time t* = 20 min,

$$C_T(t)=K_i{\int }_0^t{C}_P(\tau )d\tau +V_b{C}_P(t),$$ (1)

where C_T is the tissue radiotracer concentration, C_P is the input function (plasma radiotracer concentration), K_i is the slope (the net uptake rate constant), and V_b is the y-axis intercept. This method incorporates image-based regression weights into the fitting process more easily [3]. The Patlak fitting module takes the subject input function and the whole CBM dynamic frame sequence as the input; in addition to the frames corrected by the current iteration, the remaining frames are corrected by the previous update. With the Patlak K_i and V_b estimations, the tracer kinetics information and parametric regularization are introduced to the framework.

The Patlak Penalization in Loss Function.

The Patlak penalization by mean squared percentage fitting error (MSPE) is incorporated to the loss function in addition to an image similarity measurement by local NCC and a displacement gradient loss by one-sided forward difference [1],

$$ℒ=\sum _{i=1}^N\left[-NCC\left(F_R,\widehat{F_{Mi}}\right)+\lambda {\left|\nabla {\varphi }_i\right|}^2\right]+\sum _{j=1}^T\alpha \mathit{\text{MSPE}}\left(\widehat{F_{Mj}},\widehat{F_{M{j}_P}}\right),$$ (2)

where N is the motion estimation input sequence length, F_R is the reference frame, $\widehat{F_{Mi}}$ is the i^th warped frame, λ is the gradient loss regularization factor, φ_i is the estimated displacement field for frame i, T is the total number of frames after t*, α is the Patlak penalization factor, $\widehat{F_{Mj}}$ is the j^th Patlak input dynamic frame, and $\widehat{F_{M{j}_P}}$ is the j^th Patlak fitted dynamic frame.

2.3. Training Details and Baseline Comparison

All 5-min frames were motion corrected with Frame 12 (in the middle of the 5-min scans) as the fixed frame. Five successive frames from the same subject are sent into the motion estimation network as the input sequence, with the remaining 10 frames from the last update and individual input function sent to the Patlak estimation module. Note that the motion estimation inputs are all the original frames without correction, while the remaining frames for Patlak fitting are updated every 50 epochs to provide the latest Patlak error estimation by the currently-trained network. The update cycle is determined from the trade-off between time consumption and network convergence while also avoiding the instability in the early stage of training. A preliminary ablation study of the updating strategy was conducted on a random fold of cross-validation and summarized in Supplementary Table S1.

We included a traditional entropy-based non-rigid registration method implemented by BioImage Suite [9,12] (BIS), a single-pair deep learning registration model Voxelmorph [1] (VXM), and a multiple-frame analysis model [10] (B-convLSTM) as the benchmarks for performance comparison under a 9-fold subject-wise cross-validation. Deep learning approaches were developed using Keras (TensorFlow backend) on an NVIDIA Quadro RTX 8000 GPU and trained using Adam optimizer (learning rate = 10⁻⁴). The stopping epoch was determined based on the observed minimal validation loss, which was 750 for VXM and 500 for B-convLSTM and MCP-Net. The regularization factor λ of both VXM and B-convLSTM was fixed as 1 [1,10]. The hyperparameters of MCP-Net were set at λ = 0.1 and α = 0.1. A preliminary sensitivity test of λ and α was implemented with results in Supplementary Tables S2 and S3.

2.4. Evaluation Metrics

To directly evaluate motion estimation, we ran a motion simulation test by applying patient-derived motion vector predictions to the selected “motion-free” frames of another subject, since ground-truth motion vectors are not available in real-patient datasets. Specifically, we selected 3 subjects with insignificant motion and treated the frames after motion correction by another well-trained deep learning model VXM-multiframe [10] as motion-free. To achieve realistic simulated motion, we selected another 3 subjects with visually significant motion and used the motion fields also estimated by VXM-multiframe as the ground-truth such that the model for motion generation is independent of the tested models. We calculated the average absolute prediction error in each frame,

$$\left|\Delta \varphi \right|=\frac{1}{V}\sum _{m=1}^V\left|{\varphi }_{x_m}-\widehat{{\varphi }_{x_m}}\right|+\left|{\varphi }_{y_m}-\widehat{{\varphi }_{y_m}}\right|+\left|{\varphi }_{z_m}-\widehat{{\varphi }_{z_m}}\right|\text{,}$$ (3)

where V is the total number of voxels per frame, (${\varphi }_{x_m}$, ${\varphi }_{y_m}$, ${\varphi }_{z_m}$) is the ground-truth motion and ($\widehat{{\varphi }_{x_m}}$, $\widehat{{\varphi }_{y_m}}$, $\widehat{{\varphi }_{z_m}}$) is the predicted motion. The voxel-wise motion prediction error maps were also visualized.

To assess dynamic frame similarity in real-patient results, we calculated the structural similarity index between the reference frame and the average of all warped moving frames (Avg-to-ref SSIM) in each subject. The K_i and V_b images were generated by the Patlak analysis at the original image resolution. The dynamic frames and K_i/V_b images were overlaid to visualize motion-related mismatch, as both K_i and V_b are motion-sensitive and higher alignment indicates better registration. Whole-body K_i/V_b normalized mutual information (NMI) was computed to measure alignment. To assess Patlak fitting, the normalized weighted mean fitting errors (NFE) were formulated as

$$NFE=\frac{\sum _{k=1}^T{w}_k{\left(\widehat{C_T}\left(t_k\right)-C_T\left(t_k\right)\right)}^2}{(T-2)\sum _{k=1}^T{\left(\frac{w_k{C}_T\left(t_k\right)}{T}\right)}^2},$$ (4)

where w_k is the fitting weight of time activity curve after decay correction [4], $\widehat{C_T}\left(t_k\right)$ is the fitted tissue tracer concentration, C_T (t_k) is the acquired tissue tracer concentration, t_k is the middle acquisition time of the k^th frame, and T is the total number of frames after t*. The NFE maps were visualized voxel-wise and the NFE statistics in whole-body, torso, and ROIs were computed. Paired two-tailed t-tests with significance level = 0.05 were used for statistical analysis.

3. Results

3.1. Motion Simulation Test

In Fig. 2, the voxel-wise motion prediction error map of MCP-Net is generally the darkest with reduced hotspots within the body outline and torso organs, indicating decreased inference error. All deep learning models achieved lower motion prediction errors than BIS. MCP-Net significantly decreased motion prediction error compared with other baselines (p < 0.05), suggesting improved robustness and prediction accuracy. Sample overlaid dynamic frames demonstrating motion simulation and estimation accuracy are shown in Supplementary Fig. S2.

Fig. 2.

Sample absolute error maps of predicted motion fields. The subject-wise mean absolute prediction errors (in mm, mean ± standard deviation) are annotated.

3.2. Qualitative Analysis

In Fig. 3, sample overlaid dynamic frame pairs showed inter-frame motion and correction effects for heart, liver, and kidney misalignment. Both single-pair baselines BIS and VXM significantly decreased the misalignment but residual under-correction was still present. While B-convLSTM with multiple-frame analysis further improved motion correction, our proposed MCP-Net achieved the lowest remaining spatial mismatch. Similarly, in Fig. 4, both BIS and VXM reduced the motion-related K_i/V_b misalignment but still had residual mismatch. While B-convLSTM showed improved alignment, the MCP-Net realized the best spatial match at the brain edge, liver dome and the lower edge, gastrointestinal (GI) tract, and bone. Thus, Patlak regularization improved both frame and K_i/V_b spatial alignment. In Fig. 5, using the same color scale, the voxel-wise NFE maps for MCP-Net were generally the darkest with the most reduced bright spots of high fitting error like the heart and liver regions. For the significant motion of the hand and bladder, which do not strictly follow the 2-tissue compartmental models, the proposed MCP-Net still has the capability to reduce error.

Fig. 3.

Overlaid dynamic frames showing inter-frame motion and the correction effects in heart (upper), liver (lower, white arrows), and kidney (lower, cyan arrows).

Fig. 4.

Overlaid Patlak K_i and V_b images showing motion correction impacts in brain (upper), liver (middle), GI tract (lower, white arrows), and bone (lower, cyan arrows).

Fig. 5.

Sample voxel-wise parametric NFE maps in liver (upper, white arrows), heart (upper, cyan arrows), bladder (lower, white arrows), and hand (lower, cyan arrows).

3.3. Quantitative Analysis

Table 1 summarized the quantitative analysis results. Both multiple-frame models achieved substantial improvements in each metric, while the proposed MCP-Net consistently achieved the lowest whole-body and torso NFEs as well as the highest Avg-to-ref SSIM and whole-body K_i/V_b NMI. Compared with BIS and VXM, both B-convLSTM and MCP-Net significantly reduced the NFEs and increased K_i/V_b NMI (p < 0.05). The MCP-Net achieved significantly lower torso NFE (p = 0.027), higher whole-body K_i/V_b NMI (p = 0.019), and improved Avg-to-ref SSIM (p = 5.01e − 5) than B-convLSTM. Thus, the MCP-Net with Patlak regularization could further improve parametric imaging with reduced fitting error and enhanced K_i/V_b alignment by directly utilizing tracer kinetics.

Table 1.

Quantitative analysis of the inter-frame motion correction approaches (mean ± standard deviation) with the best results marked in bold.

	Avg-to-ref SSIM	Whole-body NFE	Torso NFE	Whole-body Ki/Vb NMI
Original	0.9487 ± 0.0160	0.3634 ± 0.1044	0.7022 ± 0.1079	0.8923 ± 0.0376
BIS	0.9509 ± 0.0149	0.3370 ± 0.0943	0.6908 ± 0.1088	0.9105 ± 0.0362
VXM	0.9446 ± 0.0165	0.3422 ± 0.0977	0.6834 ± 0.1039	0.9067 ± 0.0446
B-convLSTM	0.9513 ± 0.0146	0.2857 ± 0.0836	0.6390 ± 0.1005	0.9281 ± 0.0357
MCP-Net	0.9523 ± 0.0141	0.2840 ± 0.0829	0.6197 ± 0.1032	0.9295 ± 0.0348

In Fig. 6, box plots show the distributions of the mean and maximum NFE of each ROI. The MCP-Net showed consistent improvement in reducing mean and maximum error as B-convLSTM, but MCP-Net demonstrated stronger reduction for the outliers with extremely high fitting error, suggesting increased robustness.

Fig. 6.

Mean (left) and maximum (right) ROI NFEs for each motion correction method.

4. Conclusion

We proposed MCP-Net, a whole-body dynamic PET inter-frame motion correction framework with Patlak penalization. With the integrated analytical Patlak fitting module and Patlak loss regularization, the MCP-Net demonstrated improved motion correction both qualitatively and quantitatively compared with conventional and deep learning benchmarks. This shows the advantage of directly optimizing parameter estimation and utilizing tracer kinetics in multiple-frame motion correction analysis. The proposed MCP-Net has the potential to be applied to other tracers and further developed into a joint end-to-end motion correction and parametric imaging framework. Future directions will include investigating intra-frame motion correction as well as the mismatch problem in attenuation and scatter correction.