Improved Guided Image Fusion for Magnetic Resonance and Computed Tomography Imaging

Abstract

Improved guided image fusion for magnetic resonance and computed tomography imaging is proposed. Existing guided filtering scheme uses Gaussian filter and two-level weight maps due to which the scheme has limited performance for images having noise. Different modifications in filter (based on linear minimum mean square error estimator) and weight maps (with different levels) are proposed to overcome these limitations. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.

1. Introduction

Medical images from different modalities reflect different levels of information (tissues, bones, etc.). A single modality cannot provide comprehensive and accurate information [1, 2]. For instance, structural images obtained from magnetic resonance (MR) imaging, computed tomography (CT), and ultrasonography, and so forth, provide high-resolution and anatomical information [1, 3]. On the other hand, functional images obtained from position emission tomography (PET), single-photon emission computed tomography (SPECT), and functional MR imaging, and so forth, provide low-spatial resolution and functional information [3, 4]. More precisely, CT imaging provides better information on denser tissue with less distortion. MR images have more distortion but can provide information on soft tissue [5, 6]. For blood flow and flood activity analysis, PET is used which provide low space resolution. Therefore, combining anatomical and functional medical images through image fusion to extract much more useful information is desirable [5, 6]. Fusion of CT/MR images combines anatomical and physiological characteristics of human body. Similarly fusion of PET/CT is helpful for tumor activity analysis [7].

Image fusion is performed on pixels, features, and decision levels [8–10]. Pixel-level methods fuse at each pixel and hence reserve most of the information [11]. Feature-level methods extract features from source images (such as edges or regions) and combine them into a single concatenated feature vector [12, 13]. Decision-level fusion [11, 14] comprises sensor information fusion, after the image has been processed by each sensor and some useful information has been extracted out of it.

Pixel-level methods include addition, subtraction, division, multiplication, minimum, maximum, median, and rank as well as more complicated operators like Markov random field and expectation-maximization algorithm [15]. Besides these, pixel level also includes statistical methods (principal component analysis (PCA), linear discriminant analysis, independent component analysis, canonical correlation analysis, and nonnegative matrix factorization). Multiscale transforms like pyramids and wavelets are also types of pixel-level fusion [11, 14]. Feature-level methods include feature based PCA [12, 13], segment fusion [13], edge fusion [13], and contour fusion [16]. They are usually robust to noise and misregistration. Weighted decision methods (voting techniques) [17], classical inference [17], Bayesian inference [17], and Dempster-Shafer method [17] are examples of decision-level fusion methods. These methods are application dependent; hence, they cannot be used generally [18].

Multiscale decomposition based medical image fusion decompose the input images into different levels. These include pyramid decomposition (Laplacian [19], morphological [20], and gradient [21]); discrete wavelet transform [22]; stationary wavelet transform [23]; redundant wavelet transform [24]; and dual-tree complex wavelet transform [25]. These schemes produce blocking effects because the decomposition process is not accompanied by any spatial orientation selectivity.

To overcome the limitations, multiscale geometric analysis methods were introduced for medical image fusion. Curvelet transform based fusion of CT and MR images [26] does not provide a proper multiresolution representation of the geometry (as curvelet transform is not built directly in the discrete domain) [27]. Contourlet transform based fusion improves the contrast, but shift-invariance is lost due to subsampling [27, 28]. Nonsubsampled contourlet transform with a variable weight for fusion of MR and SPECT images has large computational time and complexity [27, 29].

Recently, guided filter fusion (GFF) [30] is used to preserve edges and avoid blurring effects in the fused image. Guided filter is an edge-preserving filter and its computational time is also independent of filter size. However, the method provides limited performance for noisy images due to the use of Gaussian filter and two-level weight maps. An improved guided image fusion for MR and CT imaging is proposed to overcome these limitations. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.

2. Preliminaries

In this section, we briefly discuss the methodology of GFF [30]. The main steps of the GFF method are filtering (to obtain the two-scale representation), weight maps construction, and fusion of base and detail layers (using guided filtering and weighted average method).

Let F be the fused image obtained by combining input images A and B of same sizes (M × N). The base (I ₁₁ and I ₁₂) and detail (I ₂₁ and I ₂₂) layers of source images are

$$\begin{array}{c}\left[\begin{array}{cc}{I}_{\mathrm{11}}& I_{\mathrm{12}}\\ I_{\mathrm{21}}& I_{\mathrm{22}}\end{array}\right]=\left[\begin{array}{cc}A\ast f& B\ast f\\ A-A\ast f& B-B\ast f\end{array}\right]\text{,}\end{array}$$ (1)

where f is the average filter. The base and detail layers contain large- and small-scale variations, respectively. The saliency images are obtained by convolving A and B and with a Laplacian filter h followed by a Gaussian filter g; that is,

$$\begin{array}{c}\left[\frac{S_{\mathrm{1}}}{S_{\mathrm{2}}}\right]=\left[\frac{\left|A\ast h\right|\ast g}{\left|B\ast h\right|\ast g}\right].\end{array}$$ (2)

The weight maps P ₁ and P ₂ are

$$\begin{array}{c}\left[\frac{P_{\mathrm{1}}}{P_{\mathrm{2}}}\right]=\left[\frac{\xi \left(S_{\mathrm{1}},S_{\mathrm{2}}\right)}{\xi \left(S_{\mathrm{2}},S_{\mathrm{1}}\right)}\right],\end{array}$$ (3)

where ξ(S ₁, S ₂) is a function with value 1 for S ₁(m, n) ≥ S ₂(m, n) and value 0 for S ₁(m, n) < S ₂(m, n) (similarly for ξ(S ₂, S ₁)). S ₁(m, n) and S ₂(m, n) are the saliency values for (m, n) pixel in A and B, respectively.

Guided image filtering is performed to obtain the refined weights W ₁₁, W ₁₂, W ₂₁, and W ₂₂ as

$$\begin{array}{c}\left[\begin{array}{cc}{W}_{\mathrm{11}}& W_{\mathrm{12}}\\ W_{\mathrm{21}}& W_{\mathrm{22}}\end{array}\right]=\left[\begin{array}{cc}{G}_{r_{\mathrm{1}},{ϵ}_{\mathrm{1}}}\left(P_{\mathrm{1}},A\right)& G_{r_{\mathrm{2}},{ϵ}_{\mathrm{2}}}\left(P_{\mathrm{1}},A\right)\\ G_{r_{\mathrm{1}},{ϵ}_{\mathrm{1}}}\left(P_{\mathrm{2}},B\right)& G_{r_{\mathrm{2}},{ϵ}_{\mathrm{2}}}\left(P_{\mathrm{2}},B\right)\end{array}\right],\end{array}$$ (4)

where r ₁, ϵ ₁, r ₂, and ϵ ₂ are the parameters of the guided filter W ₁₁ and W ₂₁ and W ₁₂ and W ₂₂ are the base layer and the detail layer weight maps.

The fused image F is obtained by weighted averaging of the corresponding layers; that is,

$$\begin{array}{c}F={\displaystyle \sum }_{i_{\mathrm{1}}=\mathrm{1}}^{\mathrm{2}}\hspace{0.17em}‍{\displaystyle \sum }_{i_{\mathrm{2}}=\mathrm{1}}^{\mathrm{2}}{W}_{i_{\mathrm{1}}{i}_{\mathrm{2}}}{I}_{i_{\mathrm{2}}{i}_{\mathrm{1}}}.\end{array}$$ (5)

The major limitations of GFF [30] scheme are summarized as follows.

1. The Gaussian filter from (2) is not suitable for Rician noise removal. Thus, the algorithm has limited performance for noisy images. Hence filter of (2) needs to be modified to incorporate noise effects.

2. The weight maps P ₁ and P ₂ from (3) can be improved by defining more levels. The main issue with binary assignment (0 and 1) is that when the saliency values are approximately equal, the effect of one value is totally discarded, which results in degraded fused image.

3. Proposed Methodology

The proposed scheme follows the methodology of GFF [30] with necessary modifications to incorporate the above listed limitations. This section first discusses the modification proposed due to noise artifacts and then the improved weight maps are presented.

3.1. Improved Saliency Maps

The acquired medical images are usually of low quality (due to artifacts), which degrade the performance (both in terms of human visualization and quantitative analysis).

Beside other artifacts, MR images often contain Rician Noise (RN) which causes random fluctuations in the data and reduces image contrast [31]. RN is generated when real and imaginary parts of MR data are corrupted with zero-mean, equal variance uncorrelated Gaussian noise [32]. RN is a nonzero mean noise. Note that the noise distribution tends to Rayleigh distribution in low intensity regions and to a Gaussian distribution in regions of high intensity of the magnitude image [31, 32].

Let $\dot{A}=A+N_R$ be image obtained using MR imaging containing Rician noise N _R. The CT image B has higher spatial resolution and negligible noise level [33, 34].

The source images are first decomposed into base ${\dot{I}}_{\mathrm{11}}$ and I ₁₂ and detail ${\dot{I}}_{\mathrm{21}}$ and I ₂₂ layers following (1):

$$\begin{array}{c}\left[\begin{array}{cc}{\dot{I}}_{\mathrm{11}}& I_{\mathrm{12}}\\ {\dot{I}}_{\mathrm{21}}& I_{\mathrm{21}}\end{array}\right]=\left[\begin{array}{cc}\dot{A}\ast f& B\ast f\\ \dot{A}-\dot{A}\ast f& B-B\ast f\end{array}\right].\end{array}$$ (6)

${\dot{I}}_{\mathrm{11}}$ and ${\dot{I}}_{\mathrm{21}}$ have an added noise term compared to (1). Linear minimum mean square error estimator (LMMSE) is used instead of Gaussian filter for minimizing RN, consequently improving fused image quality.

The saliency maps ${\dot{S}}_{\mathrm{1}}$ and ${\dot{S}}_{\mathrm{2}}$ are thus computed by applying the LMMSE based filter q and following (2):

$$\begin{array}{c}\left[\frac{{\dot{S}}_{\mathrm{1}}}{{\dot{S}}_{\mathrm{2}}}\right]=\left[\frac{\left|A\ast h\right|\ast q+\left|N_R\ast h\right|\ast q}{\left|B\ast h\right|\ast q}\right].\end{array}$$ (7)

The main purpose of q is to make the extra term N _R∗h as in ${\dot{S}}_{\mathrm{1}}$ small as possible while enhancing the image details.

3.2. Improved Weight Maps

The saliency maps are linked with detail information in the image. The main issue with 0 and 1 weight assignments arises in GFF [30] when different images have approximately equal saliency values. In such cases, one value is totally discarded. For noisy MR images, the saliency value may be higher at a pixel due to noise; in that case it will assign value 1 (which is not desirable). An appropriate solution is to define a range of values for weight maps construction.

Let Δ_AB = S ₁(m, n) − S ₂(m, n), and S ₁(m, n) ≥ S ₂(m, n),

$$\begin{array}{c}{\dot{P}}_{\mathrm{1}}(m,n)=\{\begin{array}{cc}\mathrm{1}\hfill & \text{if}\mathrm{\hspace{0.17em}\hspace{0.17em}}{\mathrm{\Delta }}_{AB}\ge \mathrm{0.3}\hfill \\ \mathrm{0.8}\hfill & \text{if}\mathrm{\hspace{0.17em}\hspace{0.17em}}{\mathrm{\Delta }}_{AB}\ge \mathrm{0.25}\hfill \\ \mathrm{0.7}\hfill & \text{if}{\mathrm{\hspace{0.17em}\hspace{0.17em}}\mathrm{\Delta }}_{AB}\ge \mathrm{0.2}\hfill \\ \mathrm{0.6}\hfill & \text{if}\mathrm{\hspace{0.17em}\hspace{0.17em}}{\mathrm{\Delta }}_{AB}\ge \mathrm{0.15}\hfill \\ \mathrm{0}\hfill & \text{otherwise}.\hfill \end{array}\end{array}$$ (8)

Let Δ_BA = S ₂(m, n) − S ₁(m, n), and S ₂(m, n) ≥ S ₁(m, n),

$$\begin{array}{c}{\dot{P}}_{\mathrm{2}}(m,n)=\{\begin{array}{cc}\mathrm{1}\hfill & \text{if}\mathrm{\hspace{0.17em}\hspace{0.17em}}{\mathrm{\Delta }}_{BA}\ge \mathrm{0.3}\hfill \\ \mathrm{0.8}\hfill & \text{if}{\mathrm{\hspace{0.17em}\hspace{0.17em}}\mathrm{\Delta }}_{BA}\ge \mathrm{0.25}\hfill \\ \mathrm{0.7}\hfill & \text{if}\mathrm{\hspace{0.17em}\hspace{0.17em}}{\mathrm{\Delta }}_{BA}\ge \mathrm{0.2}\hfill \\ \mathrm{0.6}\hfill & \text{if}{\mathrm{\hspace{0.17em}\hspace{0.17em}}\mathrm{\Delta }}_{BA}\ge \mathrm{0.15}\hfill \\ \mathrm{0}\hfill & \text{otherwise}.\hfill \end{array}\end{array}$$ (9)

These values are selected empirically and may be further adjusted to improve results. Figures 1(a) and 1(b) show CT and noisy MR images, respectively. Figures 1(c)–1(f) show the results of applying different weights. The information in the upper portion of the fused image increases as more levels are added to the weight maps.

Figure 1.

Weight maps comparison: (a) CT image, (b) noisy MR image, (c) fused image with 3 weight maps, (d) fused image with 4 weight maps, (e) fused image with 5 weight maps, and (f) fused image with 6 weight maps.

The weight maps are passed through guided filter to obtain ${\dot{W}}_{\mathrm{11}}$, ${\dot{W}}_{\mathrm{21}}$, ${\dot{W}}_{\mathrm{12}}$, and ${\dot{W}}_{\mathrm{22}}$. Finally the fused image $\dot{F}$ is

$$\begin{array}{c}\dot{F}={\displaystyle \sum }_{i_{\mathrm{1}}=\mathrm{1}}^{\mathrm{2}}\hspace{0.17em}{\displaystyle \sum }_{i_{\mathrm{2}}=\mathrm{1}}^{\mathrm{2}}{\dot{W}}_{i_{\mathrm{1}}{i}_{\mathrm{2}}}{\dot{I}}_{i_{\mathrm{2}}{i}_{\mathrm{1}}}.\end{array}$$ (10)

LMMSE based filter reduces the Rician noise and the more levels of weight maps ensure that more information is transferred to the fused image. The incorporation of the LMMSE based filter and a range of weight map values makes the proposed method suitable for noisy images.

4. Results and Analysis

The proposed method is tested on several pairs of source (MR and CT) images. For quantitative evaluation, different measures including mutual information (MI) [35] measure ζ _MI, structural similarity (SSIM) [36] measure ζ _SSIM, Xydeas and Petrović's [37] measure ζ _XP, Zhao et al.'s [38] measure ζ _Z, Piella and Heijmans's [39] measures ζ _PH1 and ζ _PH2, and visual information fidelity fusion (VIFF) [40] metric ζ _VIFF are considered.

4.1. MI Measure

MI is a statistical measure which provides the degree of dependencies in different images. Large value of MI implies better quality and vice versa [11, 33, 35]:

$$\begin{array}{c}{\zeta }_{\text{MI}}=\mathrm{2}[\frac{\mathrm{1}}{\left(H_A+H_F\right)}{\displaystyle \sum }_{a,f}{P}_{AF}\left(a,f\right)\log \frac{P_{AF}\left(a,f\right)}{P_A\left(a\right)P_F\left(f\right)}\\ \hspace{1em}\hspace{1em}\hspace{0.5em}+\frac{\mathrm{1}}{\left(H_B+H_F\right)}{\displaystyle \sum }_{b,f}{P}_{BF}\left(b,f\right)\log \frac{P_{BF}\left(b,f\right)}{P_B\left(b\right)P_F\left(f\right)}],\end{array}$$ (11)

where H _A, H _B, and H _F are the entropies of A, B, and F images, respectively. P _AF is the jointly normalized histogram of A and F, P _BF is the jointly normalized histogram of B and F, and P _A, P _B, and P _F are the normalized histograms of A, B, and F, respectively.

4.2. SSIM [36] Measure

SSIM [36] measure is defined as

$$\begin{array}{c}{\zeta }_{\text{SSIM}}\left(A,B,F\right)\\ \hspace{1em}=\{\begin{array}{cc}{\lambda }_w{\zeta }_{\text{SSIM}}\left(A_w,F_w\right)+\left(\mathrm{1}-{\lambda }_w\right){\zeta }_{\text{SSIM}}\left(B_w,F_w\right)\hfill & \hfill \\ \hspace{1em}\text{if}\mathrm{\hspace{0.17em}\hspace{0.17em}}{\zeta }_{\text{SSIM}}\left(A_w,B_w\hspace{0.17em}\mid \hspace{0.17em}w\right)\ge \mathrm{0.75}\hfill & \hfill \\ \max ({\zeta }_{\text{SSIM}}(A_w,F_w),{\zeta }_{\text{SSIM}}(B_w,F_w))\hfill & \hfill \\ \hspace{1em}\text{if}{\mathrm{\hspace{0.17em}\hspace{0.17em}}\zeta }_{\text{SSIM}}\left(A_w,B_w\hspace{0.17em}\mid \hspace{0.17em}w\right)<\mathrm{0.75},\hfill & \hfill \end{array}\end{array}$$ (12)

where w is a sliding window and λ(w) is

$$\begin{array}{c}{\lambda }_w=\frac{{\sigma }_{A_w}}{{\sigma }_{A_w}+{\sigma }_{B_w}},\end{array}$$ (13)

where σ _Aw and σ _Bw are the variance of images A and B, respectively.

4.3. Xydeas and Petrović's [37] Measure

Xydeas and Petrović [37] proposed a metric to evaluate the amount of edge information, transferred from input images to fused image. It is calculated as

$$\begin{array}{c}{\zeta }_{\text{XP}}={\displaystyle \sum }_{m=\mathrm{1}}^N{\displaystyle \sum }_{n=\mathrm{1}}^M\left(Q^{AF}\left(m,n\right){\tau }^A\left(m,n\right)+Q^{BF}\left(m,n\right){\tau }^B\left(m,n\right)\right)\\ \hspace{1em}\times {\left({\displaystyle \sum }_{m=\mathrm{1}}^N{\displaystyle \sum }_{n=\mathrm{1}}^M\left({\tau }^A\left(m,n\right)+{\tau }^B\left(m,n\right)\right)\right)}^{-1},\end{array}$$ (14)

where Q ^AF and Q ^BF are the product of edge strength and orientation preservation values at location (m, n), respectively. The weights τ ^A(m, n) and τ ^B(m, n) reflect the importance of Q ^AF(m, n) and Q ^BF(m, n), respectively.

4.4. Zhao et al.'s [38] Metric

Zhao et al. [38] used the phase congruency (provides an absolute measure of image feature) to define an evaluation metric. The larger value of the metric describes a better fusion result. The metric ζ _Z is defined as the geometric product of phase congruency, maximum and minimum moments, respectively.

4.5. Piella and Heijmans's [39] Metric

Piella and Heijmans's [39] metrics ζ _P1 and ζ _P2 are defined as

$$\begin{array}{c}{\zeta }_{P_{\mathrm{1}}}=\frac{\mathrm{1}}{\left|W\right|}{\displaystyle \sum }_{w\in W}[\lambda \left(w\right)Q_o\left(A,F\hspace{0.17em}\mid \hspace{0.17em}w\right)\\ +\left(\mathrm{1}-\lambda \left(w\right)\right)Q_o\left(B,F\hspace{0.17em}\mid \hspace{0.17em}w\right)]\\ {\zeta }_{P_2}={\displaystyle \sum }_{w\in W}^{}c\left(w\right)[\lambda \left(w\right)Q_o\left(A,F\hspace{0.17em}\mid \hspace{0.17em}w\right)\\ +\left(\mathrm{1}-\lambda \left(w\right)\right)Q_o\left(B,F\hspace{0.17em}\mid \hspace{0.17em}w\right)],\end{array}$$ (15)

where Q _o(A, F | w) and Q _o(B, F | w) are the local quality indexes calculated in a sliding window w and λ(w) is defined as in (13). Consider

$$\begin{array}{c}{Q}_o\left(A,F\hspace{0.17em}\mid \hspace{0.17em}w\right)=\frac{\mathrm{4}{\sigma }_{AF}\stackrel{-}{A}\hspace{0.17em}\stackrel{-}{F}}{\left({\stackrel{-}{A}}^2+{\stackrel{-}{F}}^2\right)\left({\sigma }_A^{\mathrm{2}}+{\sigma }_F^{\mathrm{2}}\right)},\end{array}$$ (16)

where $\stackrel{-}{A}$ is the mean of A and σ _A ² and σ _AF are the variance of A and the covariance of A, B, respectively. Consider

$$\begin{array}{c}{c}_w=\frac{\max \left[{\sigma }_{A_w},{\sigma }_{B_w}\right]}{{{\displaystyle \sum }}_{w^{\prime }\in W}^{}\left[{\sigma }_{A_w^{\prime }},{\sigma }_{A_w^{\prime }}\right]},\end{array}$$ (17)

where σ _Aw and σ _Bw are the variance of images A and B within the window w, respectively.

4.6. VIFF [40] Metric

VIFF [40] is a multiresolution image fusion metric used to assess fusion performance objectively. It has four stages. (1) Source and fused images are filtered and divided into blocks. (2) Visual information is evaluated with and without distortion information in each block. (3) The VIFF of each subband is calculated and the overall quality measure is determined by weighing (of VIFF at different subbands).

Figure 2 shows a pair of CT and MR images. It can be seen that the CT image (Figure 2(a)) provides clear bones information but no soft tissues information, while in contrast to CT image the MR image in Figure 2(b) provides soft tissues information. The fused image must contain both the information of bones and soft tissues. The fused image obtained using proposed scheme in Figure 2(d) shows better results as compared to fused image obtained by GFF [30] in Figure 2(c).

Figure 2.

(a) CT image, (b) noisy MR image, (c) GFF [30] fused image, and (d) proposed fused image.

Figure 3 shows the images of a patient suffering from cerebral toxoplasmosis [41]. A more comprehensive information consisting of both the CT and MR images is the requirement in clinical diagnosis. The improvement in fused image using proposed scheme can be observed in Figure 3(d) compared to image obtained by GFF [30] in Figure 3(c).

Figure 3.

(a) CT image, (b) noisy MR image, (c) GFF [30] fused image, and (d) proposed fused image.

Figure 4 shows a pair of CT and MR images of a woman suffering from hypertensive encephalopathy [41]. The improvement in fused image using proposed scheme can be observed in Figure 4(d) compared to image obtained by GFF [30] in Figure 4(c).

Figure 4.

(a) CT image, (b) noisy MR image, (c) GFF [30] fused image, and (d) proposed fused image.

Figure 5 shows a pair of images containing acute stroke disease [41]. The improvement in quality of fused image obtained using proposed scheme can be observed in Figure 5(d) compared to Figure 5(c) (image obtained by GFF [30]).

Figure 5.

(a) CT image, (b) noisy MR image, (c) GFF [30] fused image, and (d) proposed fused image.

Table 1 shows that proposed scheme provides better quantitative results in terms of ζ _MI, ζ _SSIM, ζ _XP, ζ _Z, ζ _P1, ζ _P2, and ζ _VIFF as compared to GFF [30] scheme.

Table 1.

Quantitative analysis of GFF [30] and proposed schemes.

Quantitative measures	Example  1		Example  2		Example  3		Example  4
	GFF [30]	Proposed	GFF [30]	Proposed	GFF [30]	Proposed	GFF [30]	Proposed
ζ MI	0.2958	0.2965	0.4803	0.5198	0.4164	0.4759	0.4994	0.5526
ζ SSIM	0.3288	0.3540	0.3474	0.3519	0.3130	0.3139	0.2920	0.2940
ζ XP	0.4034	0.5055	0.4638	0.4678	0.4473	0.4901	0.4498	0.4653
ζ Z	0.1600	0.1617	0.3489	0.3091	0.2061	0.2193	0.3002	0.2855
ζ P1	0.4139	0.4864	0.2730	0.3431	0.2643	0.3247	0.2729	0.3339
ζ P2	0.4539	0.7469	0.5188	0.6387	0.6098	0.7453	0.5268	0.6717
ζ VIFF	0.2561	0.3985	0.1553	0.2968	0.1852	0.3009	0.1842	0.3487

5. Conclusions

An improved guided image fusion for MR and CT imaging is proposed. Different modifications in filter (LMMSE based) and weights maps (with different levels) are proposed to overcome the limitations of GFF scheme. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.