MRI and PET Image Fusion Using Fuzzy Logic and Image Local Features

Abstract

An image fusion technique for magnetic resonance imaging (MRI) and positron emission tomography (PET) using local features and fuzzy logic is presented. The aim of proposed technique is to maximally combine useful information present in MRI and PET images. Image local features are extracted and combined with fuzzy logic to compute weights for each pixel. Simulation results show that the proposed scheme produces significantly better results compared to state-of-art schemes.

1. Introduction

Fusion of images obtained from different imaging systems like computed tomography (CT), MRI, and PET plays an important role in medical diagnosis and other clinical applications. Each imaging technique provides a different level of information. For instance, CT (based on X-ray principle) is commonly used for visualizing dense structures and is not suitable for soft tissues and physiological analysis. MRI on the other hand provides better visualization of soft tissues and is commonly used for detection of tumors and other tissue abnormalities. Likewise, information of blood flow in the body is provided by PET (a nuclear imaging technique) but it suffers from low resolution as compared to CT and MRI. Hence, fusion of images obtained from different modalities is desirable to extract sufficient information for clinical diagnosis and treatment.

Image fusion integrates (complementary as well as redundant) information from multimodality images to create a fused image [1–6]. It not only provides accurate description of the same object but also helps in required memory reduction by storing fused images instead of multiple source images. Different techniques are developed for medical image fusion which can be generally grouped into pixel, feature, and decision level fusion [7]. Compared to feature and decision, pixel level methods [1, 2] are more suited for medical imaging as they can preserve spatial details in fused images [1, 8].

Conventional pixel level methods (including addition, subtraction, multiplication, and weighted average) are simpler but are less accurate. Intensity Hue saturation (IHS)-based methods fuse the images by replacing the intensity component [1, 5, 9]. These methods generally produce high-resolution fused images but cause spectral distortion (due to inaccurate estimation of spectral information) [10]. Similarly, principal components analysis based methods fuse images by replacing certain principle components [11].

Multiresolution techniques including pyramids, discrete wavelet transform (DWT), contourlet, curvelet, shearlet, and framelet transform image into different bands for fusion (a comprehensive comparison is presented in [12]). DWT-based schemes decompose the input images into horizontal, vertical, and diagonal subbands which are then fused using additive or substitutive methods. Earlier DWT-based fusion schemes cannot preserve the salient features of the source images efficiently, hence producing block artifacts and inconsistency in the fused results [2, 3]. Human visual system is combined with DWT to fuse the low frequency bands using visibility and variance features, respectively. Local window approach is used (to adjust coefficients adaptively) for noise reduction and maintaining homogeneity in fused image [4]. However, the method often produces block artifacts and reduced contrast [3, 5]. Consistency verification and activity measures combined with DWT can only capture limited directional information and hence are not suitable for sharp image transitions [13].

Texture features and visibility measure are used with framelet transform [5] to fuse high and low frequency components, respectively. Contourlet transform based methods use different and flexible directions to detect the intrinsic geometrical structures [13]. The common methods are variable weight using nonsubsampled contourlet transform [14]; and bio-inspired activity measurer using pulse-coded neural networks [15]. However, the down- and up-sampling in contourlet transform lack shift invariance and cause ringing artifacts [14]. Curvelet transform uses various directions and positions at length scales [16]; however, it does not provide a multiresolution representation of geometry [17]. Shearlet transform carries different features (like directionality, localization, and multiscale framework) and can decompose the image into any scale and direction to fuse the required information [17].

Prespecified transform matrix and learning techniques are used with kernel singular value decomposition to fuse images in sparse domain [18]. In [19], image fusion has been performed using redundancy DWT and contourlet transform. A pixel level neuro-fuzzy logic based fusion adjusts the membership functions (MFs) using backpropagation and least mean square algorithms [20]. A spiking cortical model is proposed to fuse different types of medical images [21]. However, these schemes are complex or work under certain assumptions/constraints.

A fusion technique for MRI and PET images using local features and fuzzy logic is presented. The proposed technique maximally combines the useful information present in MRI and PET images. Image local features are extracted and combined with fuzzy logic to compute weights for each pixel. Simulation results based on visual and quantitative analysis show the significance of the proposed scheme.

2. $\stackrel{`}{A}$-Trous-Based Image Fusion: An Overview

In contrast to conventional multiresolution schemes (where the output is downsampled after each level), à-trous or undecimated wavelet provides shift invariance, hence better suited for image fusion.

Let different approximations I _MRI,k of MRI image I _MRI (having dimensions M × N) be obtained by successive convolutions with a filter f, that is,

$$\begin{array}{c}{I}_{\text{MRI},k+\mathrm{1}}=I_{\text{MRI},k}\ast f,\end{array}$$ (1)

where I _MRI,0 = I _MRI and f is a bicubic B-spline filter. The kth wavelet plane W _MRI,k of I _MRI is,

$$\begin{array}{c}{W}_{\text{MRI},k}=I_{\text{MRI},k+\mathrm{1}}-I_{\text{MRI},k}.\end{array}$$ (2)

The image I _MRI is decomposed into low I _MRI,L and high I _MRI,H frequency components as

$$\begin{array}{c}{I}_{\text{MRI}}=I_{\text{MRI},L}+I_{\text{MRI},H}\\ =I_{\text{MRI},L}+{\displaystyle \sum }_{k=\mathrm{0}}^K{W}_{\text{MRI},k},\end{array}$$ (3)

where K is the total number of decomposition levels. Similarly PET image I _PET in terms of low I _PET,L and high I _PET,H frequency components is

$$\begin{array}{c}{I}_{\text{PET}}\left(\beta \right)=I_{\text{PET},L}\left(\beta \right)+{\displaystyle \sum }_{k=\mathrm{0}}^K{W}_{\text{PET},k}\left(\beta \right),\end{array}$$ (4)

where β ∈ {R, G, B}, as PET images are assumed to be in pseudocolor [9].

Different methods are present in literature to fuse low and high frequency components which are generally grouped into substitute wavelet (SW) and additive wavelet (AW). The fused image I _SW using SW is

$$\begin{array}{c}{I}_{\text{SW}}\left(\beta \right)=I_{\text{PET},L}\left(\beta \right)+{\displaystyle \sum }_{k=\mathrm{0}}^K{W}_{\text{MRI},k}.\end{array}$$ (5)

Note that SW method fuses image by completely replacing the high frequency components of PET by high frequency components of MRI image, which can cause geometric and spectral distortion. SW and IHS (SWI) are combined to overcome the limitation in fused image I _SWI; that is,

$$\begin{array}{c}{I}_{\text{SWI}}\left(\beta \right)=I_{\text{PET},L}\left(\beta \right)-{\displaystyle \sum }_{k=\mathrm{0}}^K{W}_{\text{INT},k}+{\displaystyle \sum }_{k=\mathrm{0}}^K{W}_{\text{MRI},k},\end{array}$$ (6)

where the intensity image I _INT is

$$\begin{array}{c}{I}_{\text{INT}}=\frac{\mathrm{1}}{B}{\displaystyle \sum }_{{\beta }^{\prime }}{I}_{\text{PET}}\left({\beta }^{\prime }\right).\end{array}$$ (7)

The substitution process in SWI method sometimes results in loss of information as the intensity component is obtained by simple averaging/weighting.

In AW method, the fused image I _AW is obtained by injecting high frequency components of I _MRI into I _PET:

$$\begin{array}{c}{I}_{\text{AW}}\left(\beta \right)=I_{\text{PET}}\left(\beta \right)+{\displaystyle \sum }_{k=\mathrm{0}}^K{W}_{\text{MRI},k}.\end{array}$$ (8)

AW method adds the same amount of high frequencies into low-resolution bands which causes redundancy of high frequency components (hence resulting in spectral distortion).

To cater the limitation, AW luminance proportional (AWLP) method injects the high frequencies in proportion to the intensity values [22]. Consider

$$\begin{array}{c}{I}_{\text{AWLP}}\left(\beta \right)=I_{\text{PET}}\left(\beta \right)+\frac{I_{\text{PET}}\left(\beta \right)}{\left(\mathrm{1}/B\right){{\displaystyle \sum }}_{{\beta }^{\prime }}{I}_{\text{PET}}\left({\beta }^{\prime }\right)}{\displaystyle \sum }_{k=\mathrm{1}}^K{W}_{\text{MRI},k},\end{array}$$ (9)

where B are total number of bands. The fused image I _AWLP of AWLP preserves the relative spectral information amongst different bands. The fused image using improved additive wavelet proportional (IAWP) [23] method is

$$\begin{array}{c}{I}_{\text{IAWP}}\left(\beta \right)=I_{\text{PET}}\left(\beta \right)+\frac{I_{\text{PET}}\left(\beta \right)}{\left(\mathrm{1}/B\right){{\displaystyle \sum }}_{{\beta }^{\prime }}{I}_{\text{PET}}\left({\beta }^{\prime }\right)},\\ \times \left[{\displaystyle \sum }_{k=\mathrm{1}}^K{W}_{\text{MRI},k}-{\displaystyle \sum }_{k=\mathrm{1}}^K{W}_{\text{MRIR},k}\right],\end{array}$$ (10)

where W _MRIR,k are wavelet planes of a low-resolution (a spatially degraded version of I _MRI) MRI image I _MRIR. The I _MRIR is obtained by filtering the high frequencies (by applying a smoothing filter). The major limitations of the above schemes includes induction of redundant high/low frequencies; and consequently spatial degradations.

3. Proposed Technique

The proposed scheme first decomposes the MRI and PET images into low and high frequencies using à-trous wavelet. High and low frequencies are then fused separately according to defined criterion. The overall fused image I _F in terms of high I _F,H and low I _F,L(β) frequencies is

$$\begin{array}{c}{I}_F\left(\beta \right)=I_{F,L}\left(\beta \right)+I_{F,H}.\end{array}$$ (11)

3.1. Fusion of Low Frequencies

Fusion of low frequencies I _MRI,L and I _PET,L is critical and challenging task. Various schemes utilize different criterions for fusion of low frequencies. For instance, one choice is to totally discard the low frequencies of one image, another choice is to take average or weighted average of both and so forth. However, the schemes provide limited performance as they do not cater the spatial properties of image. We have proposed fusion of low frequency using different weighting average for each pixel location. The weights are computed based on the amount of information contained in vicinity of each pixel.

3.1.1. Local Features

Local variance (LV) and local blur (LB) features are used with fuzzy inference engine to compute the desired weights for fusing low frequencies.

LV [24] is used to evaluate the regional characteristics of I _PET,L image and is defined as I _LV:

$$\begin{array}{c}{I}_{\text{LV}}\left(\beta ,m,n\right)=\frac{\mathrm{1}}{(\mathrm{2}{m}_{\mathrm{1}}+\mathrm{1})(\mathrm{2}{n}_{\mathrm{1}}+\mathrm{1})}\\ \times {\displaystyle \sum }_{m_{\mathrm{2}}=m-m_{\mathrm{1}}}^{m+m_{\mathrm{1}}}\hspace{0.17em}{\displaystyle \sum }_{n_{\mathrm{2}}=n-n_{\mathrm{1}}}^{n+n_{\mathrm{1}}}{\left(I_{\text{PET}}\left(\beta ,m_{\mathrm{2}},n_{\mathrm{2}}\right)-{\overline{I}}_{\text{PET}}\left(\beta \right)\right)}^{\mathrm{2}},\end{array}$$ (12)

where ${\overline{I}}_{\text{PET}}(\beta )$ is the mean value of m ₁ × n ₁ window centered at (m, n) pixel. Note that image containing sharp edges results in higher value (and vice versa).

LB I _LB is computed using local Rényi entropy [25] of I _PET,L image. Let P _βmn(k) be the probability (or normalized histogram) having intensity values k = 1,2,…, K within a local window (of size m ₁ × n ₁) centered at (β, m, n) pixel. I _LB is defined as [25]

$$\begin{array}{c}{I}_{\text{LB}}\left(\beta ,m,n\right)=-\frac{\mathrm{1}}{\mathrm{2}}\ln \left({\displaystyle \sum }_k^K{P}_{\beta mn}^{\mathrm{3}}\left(k\right)\right).\end{array}$$ (13)

High values of I _LV and I _LB show that I _PET,L contain more information and need to be assigned more weight as compared to I _MRI,L image.

3.1.2. Fuzzy Inference Engine

Let high ζ _LV,1(u) and low ζ _LV,2(u) Gaussian Membership functions (MFs) having means ${\overline{u}}^{(\mathrm{1})}$, ${\overline{u}}^{(\mathrm{2})}$ and variances σ _u ⁽¹⁾, σ _u ⁽²⁾ for LV be [26]

$$\begin{array}{c}{\zeta }_{\text{LV},\mathrm{1}}\left(u\right)=e^{-{\left(\left(u-{\overline{u}}^{\left(\mathrm{1}\right)}\right)/{\sigma }_u^{\left(\mathrm{1}\right)}\right)}^{\mathrm{2}}},\hspace{1em}\hspace{1em}{\zeta }_{\text{LV},\mathrm{2}}\left(u\right)=e^{-{\left((u-{\overline{u}}^{\left(\mathrm{2}\right)})/{\sigma }_u^{\left(\mathrm{2}\right)}\right)}^{\mathrm{2}}}.\end{array}$$ (14)

Similarly let high ζ _LB,1(v) and low μ _LB,2(v) Gaussian MFs having means ${\overline{v}}^{(\mathrm{1})}$, ${\overline{v}}^{(\mathrm{2})}$ and variances σ _v ⁽¹⁾, σ _v ⁽²⁾ for LB be

$$\begin{array}{c}{\zeta }_{\text{LB},L}\left(v\right)=e^{-{\left((v-{\overline{v}}^L)/{\sigma }_v^L\right)}^{\mathrm{2}}},\hspace{1em}\hspace{1em}{\zeta }_{\text{LB},H}\left(v\right)=e^{-{\left((v-{\overline{v}}^H)/{\sigma }_v^H\right)}^{\mathrm{2}}}.\end{array}$$ (15)

The inputs I _LV(β, m, n) and I _LV(β, m, n) are mapped into fuzzy set using Gaussian fuzzifier [27] as

$$\begin{array}{c}{\zeta }_{\text{LV},\text{LB}}\left(u,v\right)=e^{-{\left((u-I_{\text{LV}}(\beta ,m,n\left)\right)/{\varsigma }_{\mathrm{1}}\right)}^{\mathrm{2}}}\times e^{-{\left((v-I_{\text{LB}}(\beta ,m,n\left)\right)/{\varsigma }_{\mathrm{2}}\right)}^{\mathrm{2}}},\end{array}$$ (16)

where ς ₁ and ς ₂ are noise suppression parameters. The inputs are then processed by fuzzy inference engine using pre defined IF-THEN rules [26, 27] as follows.

•   Ru ⁽¹⁾: IF I _LV(β, m, n) is high and I _LB(β, m, n) is high THEN I _WT(β, m, n) is high.
•   Ru ⁽²⁾: IF I _LV(β, m, n) is low and I _LB(β, m, n) is high THEN I _WT(β, m, n) is medium.
•   Ru ⁽³⁾: IF I _LV(β, m, n) is high and I _LB(β, m, n) is low,THEN I _WT(β, m, n) is medium.
•   Ru ⁽⁴⁾: IF I _LV(β, m, n) is low and I _LB(β, m, n) is low THEN I _WT(β, m, n) is low.

The output MFs for high (having mean ${\overline{y}}^{(\mathrm{1})}$ and variance σ _y ⁽¹⁾), medium (having mean ${\overline{y}}^{(\mathrm{2})}$ and variance σ _y ⁽²⁾), and low (having mean ${\overline{y}}^{(\mathrm{3})}$ and variance σ _y ⁽³⁾) are defined as

$$\begin{array}{c}{\zeta }_{W,1}\left(y\right)=e^{-{\left((y-{\overline{y}}^{\left(\mathrm{1}\right)})/{\sigma }_y^{\left(\mathrm{1}\right)}\right)}^{\mathrm{2}}},\\ {\zeta }_{W,2}\left(y\right)=e^{-{\left((y-{\overline{y}}^{\left(\mathrm{2}\right)})/{\sigma }_y^{\left(\mathrm{2}\right)}\right)}^{\mathrm{2}}},\hspace{1em}\hspace{1em}{\zeta }_{W,3}\left(y\right)=e^{-{\left((y-{\overline{y}}^{\left(\mathrm{3}\right)})/{\sigma }_y^{\left(\mathrm{3}\right)}\right)}^{\mathrm{2}}}.\end{array}$$ (17)

The output of fuzzy inference engine is

$$\begin{array}{c}{\zeta }_{W,L}^{\prime }\left(y\right)=\underset{\left\{c,d,e\right\}}{\max }\left[\underset{\left\{u,v\right\}}{\sup }{\zeta }_{\text{LV},\text{LB}}\left(u,v\right){\zeta }_{\text{LV},c}\left(u\right){\zeta }_{\text{LV},d}\left(v\right){\zeta }_{W,e}\left(y\right)\right],\end{array}$$ (18)

where {c, d}∈{1,2} and e ∈ {1,2, 3}. The weights I _WT(β, m, n) are obtained by processing fuzzy outputs using center average defuzzifier [27].

The I _F,L(β) image is obtained by weighted sum of I _PET,L and I _MRI,L as

$$\begin{array}{c}{I}_{F,L}\left(\beta ,m,n\right)=I_{\text{WT}}(\beta ,m,n)I_{\text{PET},L}(\beta ,m,n)\\ +\left(\mathrm{1}-I_{\text{WT}}\left(\beta ,m,n\right)\right)I_{\text{MRI},L}\left(m,n\right).\end{array}$$ (19)

3.2. Fusion of High Frequencies

Let W _MRI-MRIR,k represent a wavelet plane of the resultant image I _MRI − I _MRIR. This ensures that only those high frequency components are used for image fusion, which are not already present in I _MRI. By the virtue of this, the proposed scheme not only avoids redundancy of information but also results in improved fusion results as compared to early techniques. The fused high frequency image I _F,H is

$$\begin{array}{c}{I}_{F,H}={\displaystyle \sum }_{k=\mathrm{1}}^K{W}_{\text{MRI-MRIR},k}.\end{array}$$ (20)

Note that I _F,H is not dependent on the bands β because I _MRI is gray-scale image.

4. Results and Discussion

The simulations of proposed and existing schemes are performed on PET and MRI images obtained from Harvard database [28]. The fusion database for brain images is classified into normal, grade II astrocytoma, and grade IV astrocytoma images. The MRI and PET images are coregistered with 256 × 256 spatial resolution. The proposed fusion scheme is compared visually and quantitatively (using entropy [29], mutual information (MI) [29], structural similarity (SSIM) [30], Xydeas and Petrovic [31] metric, and Piella [32] metric) with DWT [12], GIHS [6], IAWP [23], and GFF [33] schemes.

The original MRI images belonging to normal brain, grade II astrocytoma, and grade IV astrocytoma are shown in Figures 1(a)–1(c), respectively. Fluorodeoxyglucose (FDG) is a radiopharmaceutical commonly used for PET scans. The PET-FDG images of normal, grade II, and grade IV astrocytoma are shown in Figures 1(d)–1(f), respectively. It can be seen that different imaging modalities provide complementary information for the same region.

Figure 1.

Original MRI and PET images: (a)–(c) MRI; (d)–(f) PET.

Figure 2 shows fused images (of normal brain) obtained by using different techniques. It can be seen from Figure 2(e) that the proposed technique has preserved the complementary information of both modalities and the fuzzy based weight assessment has enabled offering less spectral information loss as compared to other state-of-art techniques.

Figure 2.

Image fusion results for normal images: (a) DWT [12]; (b) GIHS [6]; (c) GFF [33]; (d) IAWP [23]; (e) proposed technique.

Figure 3 shows fused images (of grade II astrocytoma class) obtained by using different techniques. From Figure 3(e), it can be observed that the proposed technique provides complementary information contained in both modalities and the fuzzy based weight assessment has enabled offering less spectral information loss as compared to other state of art techniques. The improvement in fused images is more visible in the tumorous region (bottom right corner).

Figure 3.

Image fusion results for grade II astrocytoma images: (a) DWT [12]; (b) GIHS [6]; (c) GFF [33]; (d) IAWP [23]; (e) proposed technique.

Figure 4 shows fused images (of Grade IV astrocytoma) obtained by using different techniques. Similar improvement (as that of Figures 2(e) and 3(e)) can be observed in Figure 4(e). It is easy to conclude that the proposed scheme provides better visual quality compared to the existing schemes.

Figure 4.

Image fusion results for grade IV astrocytoma images: (a) DWT [12]; (b) GIHS [6]; (c) GFF [33]; (d) IAWP [23]; (e) proposed technique.

Table 1 shows the quantitative comparison of different fusion techniques. Note that a higher value of the metric represents better quality. The fused images obtained using proposed technique provide better quantitative results in terms of entropy [29], MI [29], SSIM [30], Xydeas and Petrovic [31], and Piella [32] metrics.

Table 1.

Quantitative measures for fused PET-MRI images.

Scenario	Techniques	Entropy [29]	MI [29]	SSIM [30]	Xydeas and Petrovic [31]	Piella [32]
Normal brain	DWT [12]	5.403	1.6607	0.6083	0.4944	0.7558
	GIHS [6]	5.381	1.7017	0.7095	0.5362	0.8014
	GFF [33]	5.115	1.7479	0.6803	0.4825	0.6741
	IAWP [23]	5.152	1.7753	0.6735	0.3233	0.3331
	Proposed	5.738	1.7912	0.6788	0.5746	0.8469
Grade II astrocytoma	DWT [12]	3.4820	1.3817	0.7287	0.6495	0.8566
	GIHS [6]	3.4679	1.3848	0.8149	0.6227	0.8779
	GFF [33]	3.5558	1.3758	0.8120	0.6417	0.8561
	IAWP [23]	3.6351	1.3770	0.8018	0.3757	0.5405
	Proposed	3.5762	1.4292	0.8133	0.6674	0.9125
Grade IV astrocytoma	DWT [12]	5.4140	1.7487	0.6775	0.5727	0.8434
	GIHS [6]	5.7868	1.7084	0.6207	0.5697	0.8547
	GFF [33]	5.6628	1.7883	0.6819	0.5112	0.7917
	IAWP [23]	5.6831	1.8298	0.6718	0.3584	0.5642
	Proposed	5.8204	1.8683	0.6739	0.5885	0.8755

5. Conclusion

An image fusion technique for MRI and PET using local features and fuzzy logic is presented. The proposed scheme maximally combines the useful information present in MRI and PET images using image local features and fuzzy logic. Weights are assigned to different pixels for fusing low frequencies. Simulation results based on visual and quantitative analysis show that the proposed scheme produces significantly better results compared to state of art schemes.

6. Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.