On the Shape of Convolution Kernels in MRI Reconstruction: Rectangles versus Ellipsoids

Abstract

Purpose:

Many MRI reconstruction methods (including GRAPPA, SPIRiT, ESPIRiT, LORAKS, and convolutional neural network (CNN) methods) involve shift-invariant convolution models. Rectangular convolution kernel shapes are often chosen by default, although ellipsoidal kernel shapes have potentially appealing theoretical characteristics. In this work, we systematically investigate the differences between different kernel shape choices in several contexts.

Theory:

It is well-understood that a rectangular region of k-space is associated with anisotropic spatial resolution, while ellipsoidal regions can be associated with more isotropic resolution. Further, for a fixed spatial resolution, ellipsoidal kernels are associated with substantially fewer parameters than rectangular kernels. These characteristics suggest that ellipsoidal kernels may have certain advantages over rectangular kernels.

Methods:

We used real retrospectively-undersampled k-space data to empirically study the characteristics of rectangular and ellipsoidal kernels in the context of seven methods (GRAPPA, SPIRiT, ESPIRiT, SAKE, LORAKS, AC-LORAKS, and CNN-based reconstructions).

Results:

Empirical results suggest that both kernel shapes can produce reconstructed images with similar error metrics, although the ellipsoidal shape can often achieve this with reduced computation time and memory usage and/or fewer model parameters.

Conclusion:

Ellipsoidal kernel shapes may offer advantages over rectangular kernel shapes in various MRI applications.

INTRODUCTION

Constrained MRI reconstruction methods that rely on shift-invariant convolution models have existed for decades. One of the earliest approaches (which still remains quite popular) is to assume that a missing sample of k-space data can be predicted as a linear shift-invariant combination of neighboring samples (1, 2). In particular, it is common to assume that

$$\widehat{\rho }\left[\mathbf{k}\right]=\sum _{\mathbf{z}\in \mathrm{\Omega }}w\left[\mathbf{z}\right]\rho \left[\mathbf{k}-\mathbf{z}\right]$$ [1]

for an appropriately-chosen shift-invariant interpolation kernel w[k], where $\widehat{\rho }\left[\mathbf{k}\right]$ is the missing sample to be interpolated, the interpolation kernel w[k] has a shape dictated by the support set Ω, and the neighboring samples correspond to ρ[k − z] for z ∈ Ω. In many cases, the interpolation kernel w[z] is a quantity that must be learned from some kind of training data. Methods that rely on these kinds of concepts include popular approaches like SMASH (3), GRAPPA (4), SPIRiT (5), and structured low-rank matrix reconstruction methods (6–9), as well as related subspace methods like ESPIRiT (10). Another group of methods adopts the same basic structure, but moves from shift-invariant linear prediction to shift-invariant nonlinear prediction relationships to improve performance (11–14). And in recent years, due to the growing excitement about machine learning and deep learning in MRI, a huge number of different convolutional neural network (CNN) methods have been proposed (15–17).

Each of these methods can be implemented using shift-invariant filtering with appropriate kernels. Most of the existing methods use a rectangular kernel shape, in which the support set Ω of the kernel has a rectangular geometry. For example, the support set of an 11×11 rectangular kernel in 2D could be defined as Ω = {k = (n_xΔk_x, n_yΔk_y) : max(|n_x|, |n_y|) ≤ 5}, where Δk_x and Δk_y are the Nyquist sampling intervals along each dimension. In contrast, an ellipsoidal kernel shape would have a support set Ω with an ellipsoidal geometry, and could be defined for an 11×11 kernel in 2D as $\mathrm{\Omega }=\{\mathbf{k}=\left(n_x\mathrm{\Delta }{k}_x,n_y\mathrm{\Delta }{k}_y\right):\sqrt{{\left|n_x\right|}^2+{\left|n_y\right|}^2}\le 5\}$. Although we (7,14,18) and other researchers (19) have sometimes used ellipsoidally-shaped Ω, we are not aware of any previous systematic comparisons of rectangular and ellipsoidal kernel shapes.

In this work, we perform a detailed evaluation of rectangular versus ellipsoidal kernel shapes, and the results we obtained (to be described later) suggest that ellipsoidal kernels can indeed offer several advantages over more-common rectangular kernels. A preliminary account of portions of this work was previously presented at a recent conference (20).

THEORY

Many of the earliest convolutional reconstruction methods were designed for 1D reconstruction problems, in which case there is no distinction between a rectangular kernel and an ellipsoidal kernel. For more recent higher-dimensional methods, it is easy to access rectangle-shaped subcomponents of a higher-dimensional array in most programming languages (e.g., in Python, for a large 2D array denoted by B, the code B[1:3, 1:3] will yield easy access to a 3×3 rectangular subcomponent), while accessing an ellipsoid-shaped subcomponent of an array is generally more involved to code. This is likely a major contributing factor to the widespread modern use of rectangular kernels.

However, there are some theoretical reasons why an ellipsoidal kernel might be preferred over a rectangular kernel. One reason is that an ellipsoidal kernel can be viewed as a rectangular kernel with the corners removed, which implies that for ellipsoidal and rectangular kernels with “matched size” (i.e., the principal axes of the ellipse have the same lengths as the sides of the corresponding rectangle), an ellipsoidal kernel has smaller area/volume and fewer coefficients that need to be determined. In particular, in 2D, the area of an ellipse is more than 20% smaller than the area of the rectangle that inscribes it, while in 3D, the volume of an ellipsoid is nearly 50% smaller than the volume of the inscribing hyperrectangle (21). This difference is illustrated for the 2D case in Fig. 1. Practically, this means that ellipsoidal kernels are associated with fewer degrees-of-freedom (i.e., fewer coefficients w[z] that need to be learned in the linear case) than rectangular kernels of the same basic size.

Figure 1:

Rectangular kernels (first row) versus ellipsoidal kernels (second row) for different kernel sizes. The center of the kernel is marked in red, while other locations within the support are shown in white. For each configuration, the total number of kernel coefficients is indicated in yellow in the right-bottom corner. As the kernel size increases, the ellipsoidal kernels have substantially fewer coefficients than the corresponding rectangular kernels.

Choosing the number of parameters of a reconstruction model represents a classical trade-off. A reconstruction model with too few parameters may not have enough representational capabilities to accurately capture the important features of the desired image to be reconstructed. On the other hand, a model with too many parameters can be prone to overfitting, can be sensitive to noise, and can require more training data than models with fewer parameters. As a result, it may be important to choose a kernel shape that balances these factors appropriately.

Another factor to consider is that reconstruction methods that use convolution in k-space can always be interpreted as using multiplication in the image domain (2, 5, 7, 10, 22, 23). In that sense, the shape of the support set Ω implicitly determines the characteristics of the spatial-domain function corresponding to the convolution kernel w[k]. Interestingly, it is well-established that functions with rectangular k-space support are expected to have highly-anisotropic spatial resolution (with much higher spatial resolution along diagonal lines), while functions with ellipsoidal k-space support are expected to have more isotropic spatial resolution characteristics (21). This has led some authors to suggest that, in the context of data sampling to achieve a certain target resolution, it suffices to acquire an ellipsoidal region of k-space and acquiring the corners of k-space is often inefficient (21). By the same logic, it is reasonable to hypothesize that the corners of rectangular convolution kernels may not be very important in practice, and that ellipsoidal convolution kernels may be able to achieve similar capabilities with better efficiency.

METHODS

We carried out a systematic comparison between rectangular and the ellipsoidal kernel shapes, by assessing the performance and efficiency of seven representative methods: GRAPPA (4), SPIRiT (5), ESPIRiT (10), SAKE (6), LORAKS (7), AC-LORAKS (24), and a CNN-based approach (25).

In what follows we give a very brief explanation of the selected methods. GRAPPA (4) is a noniterative reconstruction method that interpolates a missing k-space point as a shift-invariant linear combination of neighboring samples (with the set of neighbors defined by the kernel shape) using kernel weights previously estimated from autocalibration (ACS) data. SPIRiT (5) is an iterative reconstruction method that uses the constraint that every point in k-space (regardless of whether that point was sampled or missing in the original acquisition) can be predicted as a shift-invariant linear combination of neighboring samples. As before, the set of neighbors is defined by the kernel shape, and the kernel weights are estimated using ACS data. The ESPIRiT (10), SAKE (6), LORAKS (7), and AC-LORAKS (24) methods are all based on convolution-structured (i.e., Hankel or Toeplitz) matrices that are formed from k-space data such that multiplying the structured matrix with a vector is equivalent to a convolution with the k-space data. In all of these methods, the kernel shape influences the way that the structured matrices are constructed. Each method uses structured matrices in slightly different ways. ESPIRiT (10) is a sensitivity-map estimation method that forms a structured matrix from ACS data, and obtains sensitivity maps from the eigenvectors of this matrix. SAKE (6) is a reconstruction method that recovers missing k-space samples by enforcing a constraint that a structured matrix formed from the reconstructed k-space data is expected to have low-rank characteristics due to parallel imaging and image support constraints. Similarly, LORAKS uses similar low-rank matrix constraints to recover missing k-space samples, but constructs the structured matrix and enforces the low-rank constraints differently than SAKE does. For the results shown in this work, we utilized the version of LORAKS based on the “S-matrix”, which simultaneously imposes parallel imaging, image support, and smooth-phase constraints (7). AC-LORAKS (24) is an autocalibrated version of LORAKS, in which the nullspace of the LORAKS matrix is estimated from ACS data. This allows image reconstruction to be reformulated as a simple least-squares problem instead of a more complicated low-rank matrix recovery problem, greatly improving computation speed. The CNN-based reconstruction method we considered is the U-Net approach that has been used as a benchmark in the FastMRI challenges (25). This approach uses a CNN to remove the artifacts from coil-combined images formed through simple zero-filled reconstructions, where the convolution filters are trained using thousands of reference datasets. It should be noted that the U-Net is based on image-domain convolution instead of k-space convolution (which all of the other methods are based on).

We assessed the performance of GRAPPA, SPIRiT, SAKE, LORAKS, and AC-LORAKS by reconstructing retrospectively-undersampled data using the ten different kernel supports shown in Fig. 1, comprising five rectangular (square) kernels with different side lengths and five ellipsoidal (circular) kernels with different radii. In all cases, we used the same optimization formulations and parameter selection procedures as recommended in the original papers. Two datasets (acquired with IRB approval) were used: 12-channel T2-weighted brain data acquired with a 256×187 matrix size; and 4-channel T1-weighted brain data acquired with a 258×256 matrix size. These datasets are shown in Fig. 2. Retrospective undersampling was performed for several different sampling patterns as shown in Fig. 2. The sampling patterns always include 24 fully-sampled lines at the center of k-space to be used as ACS data. To test the kernel shapes in different scenarios, we used different undersampling schemes for the different datasets. For the T1-weighted datataset, we used conventional uniform undersampling with acceleration factors of R = 2,3, and 4 outside of the ACS region. For the T2-weighted dataset, we used 6/8ths partial Fourier sampling with uniform undersampling and acceleration factors of R = 2,3, and 4 outside of the ACS region. Reconstruction results were evaluated using normalized root-mean-squared error (NRMSE), and we also evaluated the amount of computation time and the amount of RAM used in each case. These results were obtained on a computer with an Intel Xeon E5–1603 2.8 GHz quad core CPU and 32GB of RAM.

Figure 2:

The T2-weighted (first row) and the T1-weighted (second row) datasets with the corresponding sampling patterns used in our experiments. We show results after coil combination (root sum-of-squares).

ESPIRiT was evaluated using the same two datasets, using the same ten kernel supports and the same 24 lines of ACS data described previously. The quality of the estimated sensitivity maps was assessed by calculating the size of the residual error (quantified as NRMSE) after projecting the fully-sampled data onto the subspace spanned by the estimated sensitivity maps (10).

When considering computation time and RAM for the methods described above, it is important to keep in mind that reconstruction methods perform convolution-related computations in different ways. While we do not have space to provide comprehensive descriptions of the computations used by each of the different algorithms described above, we will discuss the convolution-related computational details of the SPIRiT approach (5), which we believe to be fairly representative. The first step of SPIRiT is to construct a convolution-structured matrix (i.e., a Hankel or Toeplitz matrix) out of the ACS data. Subsequently, convolution kernels are estimated by inverting that matrix. For these steps, the structured matrix will be much smaller when using ellipsoidal kernels than when rectangular kernels are used (the number of matrix columns is equal to the number of kernel parameters), potentially leading to substantial improvements in computation speed and memory usage. In a subsequent step, these estimated kernels will be repeatedly convolved with estimated k-space data in an iterative process. These convolution operations can be implemented efficiently using FFTs. Notably, there will generally not be a major difference between computation times for ellipsoidal and rectangular kernels for these convolution operations, because the FFT-based approach will use zero-padding to map the ellipsoidal kernels into a rectangular array prior to subsequent operations. As such, the use of ellipsoidal kernels is not expected to have an impact on FFT-based convolutions. Therefore, the ellipsoidal kernel shape is only expected to offer computational advantages for certain convolution-related algorithmic steps, but not for all of them.

The U-Net was evaluated with knee data from the IRB-approved 2019 FastMRI challenge (25). We compared the original U-Net (which used 3 × 3 rectangular kernels) against a modified U-Net (which used 3 × 3 ellipsoidal kernels, which was achieved using 3 × 3 rectangular kernels but forcing the values in the corners to be zero). Both the original and modified U-Nets were trained for 27 epochs from the same random initialization using 3,474 single-channel training examples. Training was performed on a single GPU following the procedure described in Ref. (25). For undersampling, we used the equispaced mask with 4-fold acceleration from Ref. (25). For validation, we used 7,135 single-channel datasets that were not part of the training set. Unlike the previous cases, we did not evaluate the U-Net with kernel sizes larger than 3 × 3. Due to the way that the U-Net is constructed, moving to a larger kernel size would dramatically increase the number of degrees-of-freedom of the network (e.g., even just moving from 3 × 3 to 5 × 5 would increase the number of network parameters by a factor of 3), which would likely require modifications to the network training approach and make it more difficult to obtain a fair and well-controlled comparison. Note also that, for the sake of simplicity, we used a naive implementation of the U-Net with ellipsoidal kernels in which the ellipsoidal kernels were simply treated as rectangular kernels with the corners constrained to zero throughout the entire training process. With such an approach, there is no expected difference in computational complexity compared to the U-Net with rectangular kernels, and we did not observe any obvious difference in convergence behavior. As such, we have not reported computation time or memory usage for this case.

RESULTS

Figure 3 shows the NRMSE results for the T2-weighted dataset obtained for GRAPPA, SPIRiT, ESPIRiT, SAKE, LORAKS, and AC-LORAKS, when using the kernel shapes and kernel sizes shown in Fig. 1, with the corresponding results for the T1-weighted dataset shown in Supporting Information Fig. S1. For both datasets it can be observed that, in most cases, the NRMSE achieved by both kernel shapes was quite similar, which can likely be attributed to the fact that both shapes have approximately the same spatial resolution. It should also be noted that for GRAPPA, the 3 × 3 rectangular kernel has significantly better performance than the 3 × 3 ellipsoidal kernel. This is expected since, for this small kernel size, the ellipsoidal kernel only uses two neighboring k-space samples (one on each side) to interpolate each missing sample, which is much smaller than the six neighboring k-space samples used by the rectangular kernel. In the case of the T2-weighted dataset with SPIRiT reconstruction, we observed a substantial increase in NRMSE for both kernel shapes when the kernel size was 11 × 11. The same phenomenon was observed for the T1-weighted dataset with SPIRiT reconstruction, but only for the rectangular kernel shape. This poor performance can be attributed to the fact that these large kernels have a large number of parameters that were difficult to estimate reliably based on the relatively small number of ACS lines.

Figure 3:

NRMSE for different methods for the T2-weighted dataset as a function of kernel size.

Figure 4 and Supporting Information Fig. S2 respectively show corresponding computation times for the T2-weighted and T1-weighted datasets. We observe that the ellipsoidal kernel shapes consistently yielded faster computation times than the rectangular kernel shapes, with generally bigger differences for larger kernel sizes. This matches the theoretical expectations described previously.

Figure 4:

Computation times for different methods for the T2-weighted dataset as a function of kernel size.

Supporting Information Figs. S3 and S4 respectively show the corresponding memory usage for the T2-weighted and T1-weighted datasets. In all cases, the ellipsoidal kernels used less memory than the rectangular kernels, as expected based on previous theoretical arguments.

Figure 5 shows the histogram of NRMSE values obtained from the validation set of images for CNN-based reconstruction. As can be seen, there is negligible difference in NRMSE performance for the two different kernel shapes, despite the fact that the network based on ellipsoidal kernels had substantially fewer parameters (≈ 4×10⁶) than the network based on rectangular kernels (≈ 7×10⁶). This difference could be potentially leveraged to reduce the complexity of the training process, although for simplicity, we utilized a naive implementation that did not (and would not be expected to) demonstrate computational benefits.

Figure 5:

Histograms of the NRMSE values obtained on the validation set by the CNN-based reconstruction for the different kernel shapes.

DISCUSSION AND CONCLUSIONS

To the best of our knowledge, this work represents the first systematic evaluation of kernel shapes for convolution-based MRI reconstruction methods. Although ellipsoidal kernels are not very popular in the modern literature, our results suggest that ellipsoidal kernels often offer advantages (in computation time, memory usage, and the number of model parameters) over more-common rectangular kernels, with largely similar NRMSE performance. We expect that these new insights may be valuable for improving the efficiency of MRI reconstruction in the future.

In our evaluations, we have focused only on square-shaped and circle-shaped kernels, without taking full advantage of the degrees of freedom offered by rectangles and ellipses (with different sizes along different axes). We anticipate that additional improvements might be obtained by leveraging this untapped degree of freedom. It should also be noted that the ellipsoidal kernel shape is not necessarily optimal, and that kernels with other shapes (perhaps learned directly from empirical data) or special structure (e.g., separable kernels (26–28)), might achieve even better results when considering performance, computational complexity, and/or memory usage. Exploration of such ideas is a potentially interesting direction for further research, though is beyond the scope of the present work.

Finally, it is also worth mentioning that the use of ellipsoidal kernels is a relatively simple change that is expected to be compatible with a wide range of reconstruction formulations and algorithms. As such, ellipsoidal kernels can likely be used synergistically with other techniques that can also improve the speed and quality of image reconstruction methods.