On Super-Resolution for Fetal Brain MRI

Abstract

Super-resolution techniques provide a route to studying fine scale anatomical detail using multiple lower resolution acquisitions. In particular, techniques that do not depend on regular sampling can be used in medical imaging situations where imaging time and resolution are limited by subject motion. We investigate in this work the use of super-resolution technique for anisotropic fetal brain MR data reconstruction without modifying the data acquisition protocol. The approach, which consists of iterative motion correction and high resolution image estimation, is compared with a previously used scattered data interpolation-based reconstruction method. To optimize acquisition time, an evaluation of the influence of the number of input images and image noise is also performed. Evaluation on simulated MR images and real data show significant improvements in performance provided by the super-resolution approach.

1 Introduction

Imaging moving subjects remains an open issue for Magnetic Resonance Imaging (MRI). Although the development of ultrafast 2D acquisition sequences has led to significant improvements for clinical studies (see for instance [1] for fetal studies), the slice acquisition time is still critical and has to be as short as possible to reduce the impact of the motion. As a result, sets of thick 2D slices are generally acquired in clinical studies and interpretation remains limited by visual inspection.

Several clinical imaging protocols make use of multiple orthogonal 2D multi-planar acquisitions with non-isotropic voxel size for brain studies. In the context of fetal imaging⁴, Rousseau et al. in [4] have proposed a registration-based method to compound multiple orthogonal sets of 2D fetal MRI slices into a single isotropic high resolution (HR) volume. This algorithm has been shown to be capable of reconstructing 3D geometrically consistent images from challenging clinically acquired data. The image reconstruction process is based on a local neighbourhood approach with a Gaussian kernel. Although this local interpolation approach is relatively computationally efficient for use during iterative refinement of slice to volume alignment, the final reconstruction can introduce additional blurring and does not make use of oversampling of the anatomy provided by the acquisition of multiple datasets. Jiang et al. in [5] have proposed a very similar approach for volume reconstruction based on a slice-to-volume registration and B-spline interpolation. They have applied their method on various MR brain images: awake neonates, deliberately moved adults and foetuses. To form 3D images of fetal brain, Kim et al. in [6] have recently proposed a new registration approach which considers the collective alignment of all slices directly, via shared structure in their intersections, rather than to an estimated 3D volume. After slice alignment a final volume is reconstructed using sparse data interpolation. All these approaches consist of two steps (motion estimation and image reconstruction) and make use of interpolation techniques to reconstruct a final 3D MR images.

In this paper we begin by assuming that fetal motion can be reliably estimated [4], and focus on the image reconstruction step and we investigate the use of a super resolution (SR) approach for this specific issue. SR in MRI has been mostly investigated using a specialized protocol to acquire shifted images, where the motion between slices is known (translation in the slice direction [7,8,9] or rotation around a common frequency encoding axis [10]). In this paper, we study reconstruction techniques with a commonly used fetal imaging protocol: multiple orthogonal stacks of thick slices where the slice thickness is usually around three times the dimension of the in-plane resolution. Since the limits of SR algorithms (maximum magnification factor, minimum number of low-resolution (LR) inputs) is still an open question [11,12], here we also experimentally investigate the number of LR images required to form a given image resolution in order to optimize the overall imaging time.

2 Problem Statement

2.1 Super-Resolution

The principle of super-resolution (SR) is to combine low resolution images to produce an image that has a higher spatial resolution than the original images [13]. This is a large research field encompassing many applications. However the majority of the work has focused on using lower resolution data acquired on a regular grid and often assuming simple translational motion between the lower resolution sample grids, unlike multislice brain MR data which is corrupted by full 3D rigid motion on a slice by slice basis (see Figure 1). Please note that this data specificity does not affect the way the SR problem is modeled. However, it introduces additional challenges because SR becomes a reconstruction problem with scattered anisotropic data.

Fig. 1.

Acquired fetal MR image data with anisotropic resolution. It can be noted that the LR images provide complementary views of the entire brain. From left to right: 1) axial LR image, 2) sagittal LR image, 3) coronal LR image.

As in most of common SR approaches, we model the physical problem and then compute a solution by inverting this observation model:

$${\mathbf{y}}_{\mathbf{r},\mathbf{s}}=S_r{B}_r{W}_s{W}_r\mathbf{x}+{\mathbf{n}}_{\mathbf{r}}\mathit{for}1\le r\le n,1\le s\le m_r$$ (1)

where n is the number of LR images and m_r is the number of slices of the LR image r, y_r,s denotes the slice s of the LR image r, x is the HR image, n_r represents the observation noise, S_r is the subsampling matrix, B_r a blur matrix, W_s and W_r are geometric transformations of sth slice of the y_r,s and of the rth low resolution image respectively. The purpose of super-resolution is to remove the effects of the blurring convolution and to increase the voxel grid density.

For simplicity, the four operators can be combined into a single matrix H_r,s: H_r,s = S_rB_rW_sW_r. The matrix H_r,s thus incorporates motion compensation, degradation effects, and sub-sampling for each slice y_r,s. In this paper, we assume that the operators D_r and B_r, and the noise characteristics are known and spatially invariant. The operators W_s and W_r are estimated using a similar method to the one proposed by Rousseau et al. in [4] using the pipeline detailed in Figure 2. Using this observation model, a straightforward definition of the data discrepancy functional is:

$$D(\mathbf{x})=\sum _r^n\sum _s^{m_r}\psi ({\mathbf{y}}_{\mathbf{r},\mathbf{s}}-H_{r,s}\mathbf{x}).$$ (2)

Fig. 2.

Pipeline used to estimate the matrix W and the HR image x. This pipeline is performed in an iterative fashion (this is represented by the gray arrow).

ψ is usually set as a quadratic functional by assuming that the image noise n_r follows a Gaussian distribution. However, x̂ cannot be uniquely determined by minimizing D(x) since the image reconstruction process is an ill-posed problem. For such inverse problems, some form of regularization plays a crucial role and must be included in the cost function to stabilize the problem or constrain the space of solutions.

2.2 Variational regularization

One of the most important classes of linear deterministic regularization methods is the one introduced by Tikhonov in [14] for which the regularization term can be defined as follows:

$$J(\mathbf{x})=\sum _{p=0}^P\int c_p(\mathbf{v}){\mid {\mathbf{x}}^{(p)}(\mathbf{v})\mid }^{2}d\mathbf{v}$$ (3)

where weights c_p are strictly positive and x^(p) is the p^th order derivative of x. Much work has been carried out on the derivation of regularization terms of the form φ(|∇x|) or φ(|x|) where φ is a convex even function, increasing on R⁺ (see for instance [15]). If φ(t) = t², one obtain L² norm which smooths strong variations in the image. If φ(t) = t, one obtain L¹ norm. One popular variational regularization functional relies on the total variation:

$$J_{TV}(\mathbf{x})=\int \mid \nabla \mathbf{x}(\mathbf{v})\mid d\mathbf{v}$$ (4)

Charbonnier et al. in [15]) have proposed several edge-preserving regularization functionals whose one we use in our experiments is:

$$J_{Ch}(\mathbf{x})=\int \phi (\nabla \mathbf{x}(\mathbf{v}))d\mathbf{v}$$ (5)

with $\phi (t)=2\sqrt{1+t^2}-2$.

3 Sufficient Number of LR Images

The principle of SR techniques is to make use of multiple LR images to estimate a single HR image. As it was pointed out in [11,12], two key points of SR algorithms are the magnification factor and the number of input images. Lin and Shum have shown in [12] that considering 2D images, the maximum magnification factor is 1.6 if the denoising and registration is not good enough and 5.7 under synthetic conditions. Moreover, when the magnification factor is an integer, the sufficient number of LR images is .

In the context of fetal MR data, we consider anisotropic data where the slice thickness is approximately three times bigger than the in-plane resolution. In this work, the sufficient number of LR fetal MR images is estimated by using simulated data. To explore the ability to reconstruct a high resolution volume for typical anatomical brain structures, we applied the SR reconstruction algorithm on a T1-weighted MR image from the Brainweb dataset [16]. Brainweb is a simulated brain database which is often used as a gold standard for the analysis of in vivo acquired data. In order to mimic the clinical fetal imaging protocol used in routine, orthogonal low resolution images (1 × 1 × 3 mm) have been created using the observation model described by Equation 1. When using more than 3 images, the LR orthogonal images are simulated using 1mm shift. In this noise free case, no noise except the rounding error is introduced to the LR images and the registration is exact. We also have evaluated the performance of the proposed method using noisy low resolution images (Gaussian noise with standard deviation set to 10). Reconstruction algorithms have been applied to these synthetic images in order to reconstruct an isotropic (1 × 1 × 1 mm) image.

For quantitative comparison, the peak signal to noise ratio (PSNR) is also reported in decibels (dB): $\mathit{PSNR}=10{log}_{10}\left(\frac{d^2}{{\mid \mathrm{\Omega }\mid }^{-1}{\sum }_{\mathbf{v}\in \mathrm{\Omega }}{(\mathbf{x}(\mathbf{v})-\widehat{\mathbf{x}}(\mathbf{v}))}^2}\right)$ where d is the reference image dynamic. PSNR performance with respect to the number of considered LR images are reported in Figure. Two SR regularization functionals (J_TV, J_Ch) are compared with the local scattered data interpolation approach proposed by Rousseau et al. in [4] (we found, in experiments not presented in this paper, that this interpolation method has similar performance to other scattered data interpolation techniques such as those using Radial Basis Functions).

Results are reported in Figure 3. When considering the noise free experiment, it clearly appears that using more than 3 LR images does not lead to significant improvement. This means that under optimal conditions, only 3 LR images are required to accurately estimate a HR image using SR techniques. When adding non negligible noise in LR images, the number of required LR images greatly increase (except for interpolation technique for which 3 remains a sufficient number of LR inputs) and is close to the estimated number obtained by Lin and Shum in [12]. Such experiment tends to show that efficient denoising method may have a substantial impact on SR results. In both cases, SR techniques lead to similar results and outperform the sparse interpolation approach.

Fig. 3.

PSNR in function of the number of input Brainweb images. Left: noise free experiment, Right: Gaussian noise experiment. □: SR technique with J_Ch regularisation, ○: SR technique with J_TV regularisation, ▽: sparse interpolation technique [4].

4 Experiments on Fetal Brain MR Data

We applied the algorithm to fetal MR scans: T2 weighted HASTE sequence (TE/TR = 147/3190 ms) on a 1.5 T Siemens Avanto MRI Scanner (SIEMENS, Erlangen, Germany), resolution: 0.74 × 0.74 × 3.45mm. Pregnant women were briefed before the exam and signed informed consent. To reduce motion artefacts, fetal sedation was obtained with 1 mg of flunitrazepam given orally to the mother 30 minutes before the exam. The study was approved by the local ethics committee.

Figure 4 shows one original low resolution image compared to the high resolution reconstructed images for axial, coronal, sagittal views. Results obtained with the SR approach with J_Ch regularisation compare favorably with the local sparse interpolation approach proposed by Rousseau et al. in [4]. It can be especially noticed that the boundaries of brain structures (like the cortex) are better recovered. Moreover, the noise present in the LR image has no major impact on the reconstructed images. However, reconstruction artefacts appear outside the fetal head (see for instance the surrounding structures). In this case, the assumption concerning the “rigidity” of the scene is violated and registration errors disturb the reconstruction process. The result shown in Figure 4 has been obtained using 3 LR images. Using up to 6 images gives only marginal qualitative improvements. This experiment tends to show that the pipeline described in Figure 2 may be efficient enough to reduce, in our context, fetal imaging time by avoiding unnecessary acquisitions.

Fig. 4.

Details of a reconstructed fetal brain MR image using 3 orthogonal LR images. From left to right: A) original low resolution image, B) reconstructed image using local sparse interpolation [4], C) SR reconstruction using J_Ch regularisation.

5 Discussion

The first contribution of this work concerns the use of a SR framework for HR 3D isotropic fetal brain MR image reconstruction. Contrary to previous works [4,5,6] where sparse interpolation frameworks were used, an observation model is introduced in this paper in order to truly take into account physics of MR image acquisition (point spread function, sub-sampling and noise). This model allows potentially to recover fine image details compared to interpolation based approaches. Visual analysis of the obtained results on real data are very encouraging. Such high resolution image reconstruction algorithm represents an important step towards analysis of fine scale anatomical details.

The second contribution of this paper is the study of the sufficient number of LR images based on simulations using the Brainweb images. Considering a set of orthogonal stack of 2D slices with an anisotropic resolution of 1 × 1 × 3mm, the sufficient number of LR images is three in the case of noise free images. When adding Gaussian noise, the sufficient number of LR inputs is less than 9. Including an efficient denoising step in the overall image processing pipeline should lead to a significant decrease of the number of required LR images. As a consequence, further work is to investigate the building of such pipeline in order to reduce the fetal imaging time.