Super-resolution Reconstruction for Tongue MR Images

Abstract

Magnetic resonance (MR) images of the tongue have been used in both clinical medicine and scientific research to reveal tongue structure and motion. In order to see different features of the tongue and its relation to the vocal tract it is beneficial to acquire three orthogonal image stacks—e.g., axial, sagittal and coronal volumes. In order to maintain both low noise and high visual detail, each set of images is typically acquired with in-plane resolution that is much better than the through-plane resolution. As a result, any one data set, by itself, is not ideal for automatic volumetric analyses such as segmentation and registration or even for visualization when oblique slices are required. This paper presents a method of super-resolution reconstruction of the tongue that generates an isotropic image volume using the three orthogonal image stacks. The method uses preprocessing steps that include intensity matching and registration and a data combination approach carried out by Markov random field optimization. The performance of the proposed method was demonstrated on five clinical datasets, yielding superior results when compared with conventional reconstruction methods.

1. INTRODUCTION

The mortality rate of oral cancer including tongue cancer is not considered to be high, but its morbidity in terms of speech, mastication, and swallowing problems is significant and seriously affects the quality of life. Characterizing the relationship between structure and function in the tongue is becoming a core requirement for both clinical diagnosis and scientific studies in the tongue/speech research community. In recent years, medical imaging, especially magnetic resonance imaging (MRI), has played an important role in this effort. MRI is a noninvasive technology that has been extensively used over last two decades to analyze tongue structure and function ranging from studies of the vocal tract^1–4 to studies on tongue muscle deformation.^5–8 For instance, high-resolution MRI provides exquisite depiction of muscle anatomy while cine MRI offers temporal information about its surface motion. With the growth in the number of images that can be acquired, research studies now involve 3D high-resolution images taken from different individuals, time points, and MRI modalities. Therefore, the requirement for automated methods that carry out image analysis of the acquired tongue image data is expected to grow rapidly.

Time limitations of current MRI acquisition protocols make it difficult to acquire a single high-resolution 3D structural image of the tongue and vocal tract. It is almost certain that tongue motion–particularly the gross motion of swallowing—will spoil every attempt to acquire such data. Therefore, in our current acquisition protocol, three orthogonal volumes with axial, sagittal, and coronal orientations are acquired one after the other. Motion may occur between these scans but the subject will return the tongue to a resting position for each scan. To speed up each acquisition, the fields of view (FOVs) of each acquired orientation are limited to encompass only the tongue itself, as shown in Figs. 1(a)–(c). Also, in order to rapidly acquire each stack of images with contiguous images (no gaps between the images), the through-plane resolution is worse than the in-plane resolution. In our case, the images have an in-plane resolution of 0.94 mm×0.94 mm but are 3 mm thick.

Figure 1.

Tongue images are acquired in three orthogonal volumes with field of view encompassing the tongue and surrounding structures. (a) Coronal, (b) sagittal, and (c) axial volumes are illustrated. The final super-resolution volume using the proposed method is shown in (d). The red arrows indicate the tongue region.

Because the slices are relatively thick, none of the acquired image stacks are ideal for 3D volumetric analyses such as segmentation, registration, and atlas building or even for visualization when oblique views are required. Therefore, reconstruction of a single high-resolution volumetric tongue MR image from the available orthogonal image stacks will improve our ability to visualize and analyze the tongue in living subjects. In this work, we develop a fully automated and accurate super-resolution volume reconstruction method from three orthogonal image stacks of the same subject. Because of the motion between scans and limited FOV, our problem is somewhat different than the conventional, well-studied super-resolution framework. We propose a number of preprocessing steps including motion correction, intensity normalization, etc, followed by a region-based maximum a posteriori (MAP) Markov random field (MRF) approach. In order to preserve important anatomical features such as the subtle boundaries of muscle, we use edge-preserving regularization. To our knowledge, This is the first attempt of super-resolution reconstruction applied to in vivo tongue high-resolution MR images. The resulting super-resolution volume improves both the signal-to-noise ratio (SNR) and resolution over the source images, thereby approximating an original high-resolution volume whose acquisition would have taken too long for the subject to refrain from swallowing (or making other motions). Our reconstruction allows full 3D volumetric data and improves further image/motion and visual analyses of the tongue.

2. RELATED WORK

To date there have been no reports describing super-resolution reconstruction of the tongue from low-resolution orthogonal MR images. However, some highly relevant work has been reported in other areas. For example, in brain imaging it is common to take multiple scans of the same subject from different orientations, and the acquired images typically have better in-plane than through-plane resolution. Bai et al.⁹ proposed an MAP super-resolution method to reconstruct a high-resolution volume from two orthogonal scans of the same subject. This strategy is at the heart of the method we propose here.

Several researchers have developed methods for fetal brain imaging where due to uncontrolled motion it is common to acquired multiple orthogonal 2D multiplanar acquisitions with anisotropic voxel sizes.^10–12 To yield a single high-resolution registered volume image of the fetal brain from this kind of data, Rousseau et al.¹⁰ incorporated a registration method to correct motion and final reconstruction process was based on a local neighborhood approach with a Gaussian kernel. Jiang et al.¹² proposed motion correction using registration and B-spline based scattered interpolation to reconstruct the final 3D fetal brain. Gholipour et al.¹³ proposed maximum likelihood (M-estimation) error norm to reconstruct the fetal MR images. In recent work, Rousseau¹¹ used an edge-preserving regularization method to reconstruct high-resolution images from the fetal MRI images and investigated the impact of the number of low-resolution images used. These approaches incorporate registration to align the observed data to a single anatomical model; this is a strategy we also use herein (although we use deformable registration due to the nature of the human tongue).

Example-based methods using non-local means have also been explored in the super-resolution of medical images. Rousseau,¹⁴ in particular, investigated so-called brain hallucination by incorporating a high-resolution brain image of a different subject (an atlas) in order to synthesize likely high-resolution texture patches from a low-resolution image of a subject. Although these methods are promising, we maintain that it is better to reconstruct higher-resolution images from actual imaging data whenever possible rather than relying on examples that may or may not be representative of the object actually being imaged. This is especially true in the imaging of abnormal anatomy, such as the tongue cancer patients who are the primary target of our scientific studies on the tongue.

3. METHOD

For our application, the imaging model that incorporates the overlap regions is then given by

$${\mathbf{g}}_k={\Lambda }_k\left(W_k{S}_k(h\ast \mathbf{f})\right)+{\mathbf{n}}_k,k=1,2,3$$ (1)

where g_k is one of the observed image volumes, Λ_k denotes a localized region (see Fig. 1), W_k is an intensity transformation caused by a coordinate transformation, S_k is a downsampling operator, h is a blurring operator, and n_k is a Gaussian noise with zero mean and variance ${\sigma }_k^2$. This model is combined as H_k = Λ_k(W_kS_k(h * f)). The goal of this work is to reconstruct a single high-resolution volume f from three orthogonal scans that approximates the original in-plane resolution in all three directions. The goal of super-resolution is to estimate f, which is a single high-resolution, isotropic image.

Our problem is somewhat different than the conventional super-resolution framework in that: (1) each volume is acquired with a different FOVs and (2) each observation has its resolution degraded in only one dimension. Our proposed method consists of multiple steps as follows: (1) preprocessing to address different FOVs and intensity differences, (2) registration to correct subject motion between volumes, and (3) a region-based MAP Markov random field (MRF) reconstruction. The flowchart of the proposed method is shown in Fig. 2.

Figure 2.

A flowchart of the proposed method.

3.1 Preprocessing

The three volumes to be combined into one super-resolved volume have different slice positions, orientations, and volume sizes. Therefore, several preprocessing steps are applied prior to the reconstruction of the super-resolved volume: (i) generation of isotropic volume, (ii) conversion of the orientation of each volume to the orientation of the target reference volume, (iii) padding of zero values to yield the same volume sizes, (iv) registration to correct subject motion between volumes, and (v) matching of intensities in the overlap region using piecewise spline regression, which normalizes the intensity values of a source volume based on the intensity values of a target volume. In addition, we generate masks from each volume to define overlap regions as shown in Fig. 3(b). We denote the regions of axial, coronal, and sagittal volumes by D₁, D₂, and D₃, respectively. The masks are given by

$$M_k\left(x\right)=\{\begin{array}{cc}1,\hfill & x\in D_k\hfill \\ 0,\hfill & x\notin D_k\hfill \end{array}\phantom{\}},k=1,2,3,$$ (2)

where M₁, M₂ and M₃ represent the characteristic functions of axial, coronal, and sagittal masks, respectively. In Fig. 3(b), the white region indicates the overlap region of all three volumes (i.e. ${\Omega }_1=D_1\cap D_2\cap D_3$), the gray regions indicate the overlap regions of two volumes (i.e. ${\Omega }_2=(D_1\cap D_2)\cup (D_2\cap D_3)\cup (D_1\cap D_3)$), and the black regions indicate both background as well as regions where only one volume is available (i.e., Ω3).

Figure 3.

One representative final super-resolved image is shown in (a), regions defined from the masks are shown in (b), a volume with regions defined from the masks is illustrated in (c), and a 3D volume rendering is illustrated in (d).

Because the precise positions of these acquired volumes relative to the anatomy might be slightly wrong due to patient motion, we use blurred versions of these masks—termed “softmasks”—which are given by

$$m_k=G_{\sigma }\ast M_k,k=1,2,3,$$ (3)

where * denotes the convolution operator and G_σ is a unit-height Gaussian kernel with standard deviation σ. In this work, we set σ=2 mm.

3.1.1 Registration

We use image registration to correct for subject motion between acquisitions. Accurate registration is of great importance in this application because small perturbations in alignment can lead to disturbing artifacts within the MAP-MRF reconstruction algorithm. To obtain accurate and robust registration results, we compute a global displacement estimate followed by a local deformation models. The global displacement estimate is characterized by an affine registration accounting for translation, rotation, and scaling (12 degrees of freedom in 3D). We use mutual information (MI)¹⁵ as the similarity measure for this step because orientation of acquisition can cause intensity differences even when the same pulse sequence on the same scanner is used. The algorithm we use is different than a conventional MI registration algorithm because of the FOV differences in the volumes. In this global registration step, we use the overlap regions of two volumes to weight the evaluation of MI in order to emphasize influence on parameter estimation to regions that are known to be overlapping.

In order to achieve sub-voxel accuracy, we estimate a local deformation using the Demons method.¹⁶ We also incorporate the overlap region to restrict the domain of registration. Prior to registration, we perform histogram matching to standardize intensity distribution to ensure that corresponding points in different volumes have similar intensity values for the local deformation model.

3.2 MAP-MRF Reconstruction

To find optimal solution, we use a Bayesian framework whereby we can estimate the high quality image given the three volumes. We follow the work of Villain et al.¹⁷ in setting up an MAP-MRF estimation framework. Using Bayes rule and MAP estimator, one can write

$$\widehat{\mathbf{f}}=\arg \underset{\mathbf{f}}{\max }P({\mathbf{g}}_1,{\mathbf{g}}_2,{\mathbf{g}}_3\mid \mathbf{f})P\left(\mathbf{f}\right),$$ (4)

where P(g₁, g₂, g₃|f) and P(f) denote likelihood and prior, respectively and f̂ is the estimated solution. In this work, we assume that the noise model is additive Gaussian noise and the likelihood P(g₁, g₂, g₃|f) which depends on image formation process defined in Eqn. (1) and noise model can be expressed as

$$P({\mathbf{g}}_1,{\mathbf{g}}_2,{\mathbf{g}}_3\mid \mathbf{f})=K\exp \left(\sum _{k=1}^3-\frac{{\Vert {\mathbf{g}}_k-H_k\mathbf{f}\Vert }^2}{2{\sigma }_k^2}\right),$$ (5)

where g_k is kth low resolution volume, f is the high quality volume to be reconstructed, and K denotes a normalization factor. For a prior model, we use an MRF model because it can restore sharp discontinuities in the volumes. The Gibbs formulation is used as our prior model, i.e.,

$$P\left(\mathbf{f}\right)=\frac{1}{G}\exp \left\{-{\sum }_{c\in C}{V}_c\right\},$$ (6)

where V_c denotes the Gibbs potential¹⁷ defined on each set c of voxels which are mutual neighbors (called a clique) and G is a normalization factor. Maximization of the Eqn. (4) is equivalent to minimizing negative of logarithm:

$$\widehat{\mathbf{f}}=\arg \underset{\mathbf{f}}{\min }[-\log P({\mathbf{g}}_1,{\mathbf{g}}_2,{\mathbf{g}}_3\mid \mathbf{f})-\log P\left(\mathbf{f}\right)],$$ (7)

which leads to

$$\widehat{\mathbf{f}}=\arg \underset{\mathbf{f}}{\min }E=\arg \underset{\mathbf{f}}{\min }\left\{\sum _{k=1}^3\frac{{\Vert {\mathbf{g}}_k-H_k\mathbf{f}\Vert }^2}{2{\sigma }_k^2}+\lambda {\sum }_{c\in C}\phi \left(u_c\right)\right\}.$$ (8)

Here, $\phi \left(u\right)=\sqrt{1+{(u∕\delta )}^2},u_c=\Delta x_c∕d_c$ and λ is a balancing parameter, where δ denotes a scaling factor and Δx_c and d_c denote the difference of the values and the distance of the two voxels in clique c, respectively. We use half-quadratic regularization technique to solve the minimization problem in Eqn. (8).¹⁷ In addition, we incorporate the region based approach as there are partial overlaps from the three volumes to reconstruct the final volume as follows:

$$\widehat{\mathbf{f}}=(\begin{array}{c}\arg \underset{\mathbf{f}}{\min }\sum _{k=1}^3{m}_1\cdot m_2\cdot m_3\frac{{\Vert {\mathbf{g}}_k-H_k\mathbf{f}\Vert }^2}{2{\sigma }_k^2}+\lambda {\sum }_{c\in C}\phi \left(u_c\right),\text{if}x\in {\Omega }_1\hfill \\ \arg \underset{\mathbf{f}}{\min }\sum _{k\in \{i,j\}}{m}_i\cdot m_j\frac{{\Vert {\mathbf{g}}_k-H_k\mathbf{f}\Vert }^2}{2{\sigma }_k^2}+\lambda {\sum }_{c\in C}\phi \left(u_c\right),\text{if}x\in {\Omega }_2(\text{determine}i,j\text{by thresholding})\hfill \\ \arg \underset{\mathbf{f}}{\min }\frac{{\Vert {\mathbf{g}}_k-H_k\mathbf{f}\Vert }^2}{2{\sigma }_k^2}+\lambda {\sum }_{c\in C}\phi \left(u_c\right),\text{if}x\in {\Omega }_3\left(\text{determine}k\text{by thresholding}\right).\hfill \end{array}\phantom{)}$$ (9)

Here, softmasks are used as a weighting function in order to reduce the effects of “seams” between different regions.

4. VALIDATION

4.1 Clinical Tongue Data

Five high-resolution MR datasets, consisting of three normal and two patients who had tongue cancer surgically resected (glossectomy) native American English speakers, were used in our experiments. The image size and resolution for high-resolution MRI were 256×256×z (z ranges from 10 to 24) and 0.94 mm×0.94 mm×0.94 mm, respectively. High-resolution MRI datasets were acquired at rest position using a head and neck coil. The subjects were required to remain still from 1.5 to 3 minutes for each plane.

4.2 Evaluation of the Algorithm

A challenge in testing the proposed method is the lack of the ground truth available in in vivo volumetric tongue MR data. The proposed method was first evaluated using five simulated datasets. In order to quantitatively evaluate the performance of the proposed method in terms of accuracy of the reconstruction, the final super-resolution volume using the proposed method was considered as our ground truth data. The ground truth was constructed with 256×256×256 voxels with resolution of 0.94 mm×0.94 mm×0.94 mm. Three low-resolution volumes (i.e., axial, sagittal, and coronal volumes) were formed in which the voxel dimensions was 0.94 mm×0.94 mm in-plane and the slice thickness was 3.46 mm (subsampling by a factor 4). Prior to subsample the volumes in slice-selection directions, Gaussian filtering with σ=0.5 (in-plane) and σ=2 (slice-selection direction) was applied in order to avoid anti-aliasing effect.

In what follows, volume reconstruction was carried out in four ways: First, 5th-order B-spline interpolation was performed in each plane independently, second, averaging of three up-sampled volumes was performed, third, reconstruction from three up-sampled volumes using Tikhonov regularization¹⁸ was performed, and finally, the proposed method using three up-sampled volumes were carried out. In this simulation study, preprocessing was not necessary except up-sampling process using 5th-order B-spline interpolation.

As a quantitative measure, the peak signal-to-noise ratio (PSNR) was used similar to the work,¹⁹ which is defined as

$$\mathrm{PSNR}=10{\log }_{10}\left(\frac{D\left(X\left(v\right)\right)}{{\mid {\Omega }_R\mid }^{-1}{\sum }_{v\in {\Omega }_R}{(X\left(v\right)-\widehat{X}\left(v\right))}^2}\right),$$ (10)

where Ω_R denotes a reference image domain, X denotes a reference volume, X̂ represents a reconstructed volume, and D(X(v)) represents the dynamic range of a reference volume.

Second, the proposed method was also visually compared to the different reconstruction schemes using original low-resolution datasets including averaging and Tikhonov regularization after same preprocessing steps were applied.

5. RESULTS

Once we obtained the super-resolution volume, a simulation study was carried out in which we subsampled the reconstructed volume by a factor of 4 in each axis independently, generating three volumes degraded in only one dimension. In order to quantitatively measure the performance of the reconstruction, PSNR was used after reconstruction using different methods, showing that the proposed method achieved better than other reconstruction methods. Table 1 summarizes the quantitative results after the reconstruction.

Table 1.

PSNR (dB) for differnt reconstruction methods on simulation data (Mean±SD)

Axial	Sagittal	Coronal	Averaging	Tikhonov	Proposed
28.9±3.8	27.1±3.7	26.5±4.6	29.1±4.0	30.2±4.4	34.1±3.2

In Figs. 4 and 5, two representative results using a normal subject (Fig. 4) and a glossectomy patient (Fig. 5) with different reconstruction methods are demonstrated, respectively. The rows show slices of three orthogonal views including axial, sagittal, and coronal, respectively. The first three columns ((a)-(c)) show three original scans after isotropic resampling (i.e., 0.94 mm×0.94 mm×0.94 mm) in the coronal, sagittal, and axial planes, respectively. (d) shows the reconstruction using averaging, (e) shows reconstruction using Tikhonov regularization, and (f) shows the reconstruction using the proposed method. (g) shows the original three orthogonal volumes. As shown in the Figures, the proposed method provided sharp muscle and fine anatomical detail as compared with the other methods. The target reference volume into which the other volumes were registered was the axial volume in both cases and therefore the reconstructed images and the original images shown in the figures may not be exactly same except the axial slice.

Figure 4.

Comparison of different reconstruction methods using normal subject. Original low-resolution coronal, sagittal, and axial volumes are shown in (a), (b), and (c), respectively. Red boxes represent original volumes after isotropic volume upsampling using B-spline. Three different reconstruction methods including simple averaging, Tikhonov regularization, and the proposed method are presented in (d), (e), and (f), respectively. Original three volumes are illustrated in (g). Please notice that the proposed method provides detailed anatomical information compared to averaging and Tikhonov regularization methods as visually assessed.

Figure 5.

Comparison of different reconstruction methods using a glossectomy patient. Original low-resolution coronal, sagittal, and axial volumes are shown in (a), (b), and (c), respectively. Red boxes represent original volumes after isotropic volume upsampling using B-spline. Three different reconstruction methods include (d) simple averaging, (e) Tikhonov regularization, and (f) the proposed method, respectively. Original three volumes are illustrated in (g). Please notice that the proposed method provides detailed anatomical information compared to averaging and Tikhonov regularization methods as visually assessed.

6. CONCLUSION

In this work, we investigated a super-resolution technique to generate isotropic and high resolution images for the tongue MR images. Due to the limited FOVs of three orthogonal volumes, we proposed to use the region based MAP-MRF reconstruction method to obtain super-resolved volume with fine detail of the anatomical structures. The experimental results were demonstrated with superior performance when compared with the conventional reconstruction methods.