Multimodal Medical Image Fusion Using Stacked Auto-encoder in NSCT Domain

Abstract

Medical image fusion is a process that aims to merge the important information from images with different modalities of the same organ of the human body to create a more informative fused image. In recent years, deep learning (DL) methods have achieved significant breakthroughs in the field of image fusion because of their great efficiency. The DL methods in image fusion have become an active topic due to their high feature extraction and data representation ability. In this work, stacked sparse auto-encoder (SSAE), a general category of deep neural networks, is exploited in medical image fusion. The SSAE is an efficient technique for unsupervised feature extraction. It has high capability of complex data representation. The proposed fusion method is carried as follows. Firstly, the source images are decomposed into low- and high-frequency coefficient sub-bands with the non-subsampled contourlet transform (NSCT). The NSCT is a flexible multi-scale decomposition technique, and  it is superior to traditional decomposition techniques in several aspects. After that, the SSAE is implemented for feature extraction to obtain a sparse and deep representation from high-frequency coefficients. Then, the spatial frequencies are computed for the obtained features to be used for high-frequency coefficient fusion. After that, a maximum-based fusion rule is applied to fuse the low-frequency sub-band coefficients. The final integrated image is acquired by applying the inverse NSCT. The proposed method has been applied and assessed on various groups of medical image modalities. Experimental results prove that the proposed method could effectively merge the multimodal medical images, while preserving the detail information, perfectly.

Introduction

In the medical imaging field, computer-aided diagnosis (CAD) is a computer-based system that aids specialists in making decisions, properly. Medical imaging offers the information in the image that allows specialists to diagnose and analyze the diseases, swiftly [1]. However, if medical images are taken with high energy, this will offer good-quality images, but it will damage the human body. On the other hand, images acquired with less energy will provide images with low contrast and bad quality. Therefore, the main purpose of CAD systems is to improve the quality of images, which helps to interpret the medical images correctly and process the images for highlighting the important regions [1]. Accordingly, CAD systems based on image processing have become an active topic in the medical image processing field. These systems help medical practitioners diagnose diseases from medical images such as X-ray, magnetic resonance (MR), computed tomography (CT), ultrasound, and microscopic images [2].

Medical imaging techniques generally create different forms of medical images to aid specialists in diagnosing diseases. Each modality has its specific properties. Therefore, several studies have aimed to merge complementary information from multiple medical images to obtain an integrated medical image. The obtained image provides more information and enhances the disease diagnosis capability for the medical examination compared to diagnosis from a single modality. Thus, multimodality medical image fusion has become one of the hottest medical imaging topics that can improve clinical diagnosis accuracy.

Medical imaging technologies are very salutary for specialists to determine clinical problems. Usually, medical images are acquired with several modalities, such as positron emission tomography (PET), single photon emission computed tomography (SPECT), CT, and MR. The medical image obtained with each modality provides different information for the organ/tissue of the body. For example, the MR images give richer soft tissue details and anatomical information with high resolution, but provide fewer boundary details. On the other hand, the CT images commonly offer detailed localization of dense structures such as bones.

In contrast, the functional information of the blood flow and metabolic variations can be provided by SPECT and PET modalities with high sensitivity, but with low spatial resolution. The limited information obtained by a single modality cannot achieve the requirements of clinical diagnosis. This makes the multimodality medical image fusion a key priority. Therefore, multimodality medical image fusion methods attempt to merge the complementary information included in different modalities and create an integrated image for obtaining better and easier decisions for different purposes [3]. The obtained final fused image should meet the following conditions. It should completely preserve the information in input images and should not reveal any synthetic information, such as artifacts, misregistration, and noise [4].

Recently, DL methods have achieved a breakthrough in different image processing and computer vision aspects. In the last three years, the studies based on DL in the area of image fusion have emerged, owing to the high capabilities of DL techniques in data representation and feature extraction [5]. The DL techniques, such as SSAE, convolutional sparse representation (CSR), and convolutional neural networks (CNNs), have achieved good results in various image fusion problems [6].

Image fusion of medical images significantly enhances the final fused image quality by retaining all the valuable and salient information from the input images. Image fusion approaches commonly merge appropriate information from two or more original images in a single integrated image. Traditionally, medical image fusion approaches are classified into spatial- and transform-domain approaches [7]. The fusion approaches in the spatial domain, such as principal component analysis (PCA), averaging, Brovey, and intensity hue and saturation (IHS) [8], offer high spatial resolution. However, spatial-domain approaches generate spectral and spatial distortion in the fusion results [5, 7]. For enhancing the final fused image quality and avoiding the limitations of the spatial-domain fusion, researchers directed their studies to transform-domain fusion, which reveals much details in the fusion results [9]. Commonly, transform-domain fusion depends on transforms such as Laplacian pyramid transform (LAP), discrete wavelet transform (DWT), curvelet transform, contourlet transform, shearlet transform, NSCT, non-subsampled shearlet transform (NSST), and sparse representation (SR) for medical image fusion [7, 10, 11]. The transform-domain fusion reveals much effectiveness in avoiding image distortion. In the last two years, from the literature, it is clear that almost no study performs spatial-domain fusion only [4]. Several researchers have addressed fusion approaches that merge spatial- and transform-domain fusion methods. Besides, some researchers addressed parameter improvement based on intelligent optimization procedures in transform domains coupled with DL techniques [5, 7].

The medical image fusion approaches are classified into three levels: pixel-level, feature-level, and decision-level fusion approaches. The fusion result is obtained directly from the set of pixels in the input images for pixel-level fusion approaches. Commonly, pixel-level fusion has been investigated in much research work. On the other hand, feature-level fusion requires searching for salient relative features of the input images, such as texture, edges, pixel intensities, and surfaces. Moreover, some presented techniques blend both pixel-level and feature-level fusion to merge medical images. For decision-level fusion, the initial object detection and classification results of the original images are fed as input to the fusion algorithms [12, 13].

Though the fusion methods at pixel level are easier to implement than the fusion methods at higher levels, these methods are mainly performed with more microscopic information. However, the fusion results often suffer from blurring effects, misregistration effects, and noise effects. Currently, most breakthroughs are achieved at this level, which is still the basis for higher-level fusion methods. Feature-level fusion methods have avoided problems of the pixel-level fusion represented in sensitivity to noise, contrast, and misregistration though they still have some drawbacks. Through the region extraction, some undesirable artifacts may be involved in the segmented regions leading to spatial distortions. In addition, it is necessary to select the segmentation technique, precisely. For decision-level fusion methods, the main challenge lies in the necessity of prior knowledge, which is difficult to obtain due to the variations in the characteristics of the environment and the target. The methods are usually complex to implement [14].

In this paper, the merits of multi-scale fusion methods, in addition to the effectiveness of DL methods, are exploited for blending of medical images. The proposed method combines both pixel-level and feature-level fusion to take advantage of both of them. The hybrid method avoids pixel-level fusion problems such as blurring artifacts and high sensitivity to noise. Moreover, it improves the feature-level fusion by incorporating the information content related to each pixel. This work commences by decomposing the two input images into low- and high-frequency components with the NSCT. Then, the low-frequency sub-bands are combined, adopting the maximum-based fusion rule, whereas the high-frequency coefficients are merged using the SSAE feature-based maximum fusion rule. Finally, the inverse NSCT is performed to reconstruct the final fused image.

The key contributions of this work include:

1. Establishment of a simple and efficient learning-based fusion method to fuse medical images.

2. Combining both pixel- and feature-level image fusion. The merits of both of them are incorporated.

3. Utilization of the NSCT as a multi-scale and multi-directional decomposition technique for medical image fusion. All sub-bands are further processed.

4. Utilization of the SSAE as an automatic feature extractor. The  extracted features are used in the fusion process.

5. Comparison of the obtained results with those of the existing methods. The results prove superiority of the proposed method compared to other existing methods.

The rest of this paper is organized as follows. “Related Work” debates related work. “Theoretical Backgrounds” presents and describes the theoretical backgrounds used in this paper. “Proposed Method” explains the proposed image fusion methodology. Experimental results and discussions are provided in “Results.” Finally, “Conclusion and Future Work” includes the conclusion and future work.

Related Work

In the medical imaging field, multimodality medical image fusion has become one of the emerging research topics. As a result, several research studies have been addressed in the area of medical image fusion. In this section, we review some of these works and discuss their contributions.

In [3], from the two input images to be fused, a weight map that combines the activity information of the pixels is generated using a Siamese convolutional network. Additionally, the source images are transformed with the Laplacian pyramid decomposition. The weight map obtained with the Siamese network is disintegrated into a Gaussian pyramid. A local energy map is estimated for each decomposition pyramid level. Then, the similarity measure utilized for fusion mode determination is calculated. The weighted-average fusion mode based on the weight map is implemented according to the selected threshold for similarity. Finally, the fusion result is obtained from the fused Laplacian pyramid coefficients.

The work in [15] introduced a medical image fusion scheme based on the features obtained with a multi-scale technique using a deep CNN. Initially, in the first layer of the CNN, the input images are divided into several sub-images by a Gauss-Laplace filter and a Gaussian filter. Then, the convolution kernel of the rest of layers is initialized using the HeK method. After that, for high-frequency coefficients, the local variance is calculated, and then the maximum local variance fusion rule is applied. On the other hand, the low-frequency coefficients are fused based on saliency metrics and  a matching index. Finally, the fused low- and high-frequency coefficients are fed back to the last layer of the CNN to acquire the final fusion result.

The authors of [16] adopted CNNs and a dual-channel spiking cortical model (DCSCM) for medical image fusion. First, the low-frequency components and a set of high-frequency coefficients of the two original images are acquired with the NSST. Then, the CNN is applied to fuse the low-frequency coefficients. Next, the weight map is created by an adaptive selection fusion rule with a set of feature maps. Furthermore, the high-frequency coefficient sub-bands are merged by applying  the DCSCM, which is motivated by the modified average gradient of coefficients. Finally, the final integrated image is obtained by the inverse NSST.

The work presented in [17] introduced a fusion system for CT and MR images that relies on a CNN and the shearlet transform. A pre-trained fully-Siamese CNN with natural data is utilized. The network is trained with medical images through a transfer learning technique. The input images are first decomposed by the NSST to obtain low- and high-frequency sub-bands. The training dataset is composed of negative and positive patches of the obtained shearlet coefficients. The patch pairs of the sub-bands are provided to the two branches of the network to get the feature maps. Then, a learning-based cross-correlation is applied as a similarity parameter to learn the mapping between features. Afterwards, based on the weighted normalized cross-correlation between feature maps, high-frequency coefficients are combined. In contrast, low-frequency sub-bands are fused by a local-energy-based rule. The final integrated image is obtained by performing the inverse NSST.

The researchers of [18] addressed a fusion approach that depends on a moving frame-based decomposition framework (MFDF) and the NSST. First, the original images are decomposed into texture and approximation components by applying the MFDF. Based on the maximum selection rule, texture components are fused to transfer salient gradient information to the final composite image. On the other hand, the approximation components are decomposed using the NSST. Then, the average fusion rule is applied to fuse the low-frequency sub-bands. The high-frequency sub-bands are merged based on the sum modified Laplacian fusion approach. The fused approximation component is obtained using the inverse NSST. Eventually, the fused image is acquired by synthesizing the fused texture and approximation components.

An algorithm for medical image fusion based on fast curvelet transform and genetic algorithms (GAs) was presented in [12]. At first, the input medical images are decomposed into source and detail images with the curvelet transform. Then, the ridgelet transform is performed using a wrapping algorithm for efficient computation and fast analysis. Super-position and probability concepts are adopted to assure accurate image fusion. Next, the dimensional feature vectors are extracted from the images by applying the fast curvelet transform. Subsequently, the extracted features are captured and enhanced using GA optimization. Next, the new curvelet coefficients for the merged images are selected from the source images according to some synthesis rules. In the last stage, the inverse curvelet transform is performed to reconstruct the final fusion result.

The authors of [19] introduced a multimodal medical image fusion scheme based on the NSST, a pulse-coupled neural network (PCNN), and the CSR. In this scheme, the low- and high-frequency sub-bands are acquired by applying the NSST. Then, the high-frequency sub-bands are merged by a parameter-adaptive PCNN, whereas the low-frequency coefficients are combined by the CSR. Finally, the inverse NSST is implemented to get the fusion result.

In the model presented in [20], multimodal fusion is implemented based on the PCNN optimized by the quantum-behavior particle swarm optimization (QPSO). The input images are initially processed by the QPSO-PCNN. Then, through the QPSO, the PCNN determines the optimal parameters for input image fusion. To increase the quality and efficiency of the QPSO, a hybrid fitness function based on entropy, spatial frequency (SF), and average gradient  evaluation metrics is chosen. The result of the fusion scheme is evaluated by an assessment factor related to the maps extracted from the two input images. The output  pixel values may be acquired from either image A or image B to be fused or may be obtained with a composite relation  between the two images.

The authors of [21] introduced a fusion scheme based on optimal thresholding using a DL model. First, an enhanced monarch butterfly optimization scheme is implemented to determine the optimal threshold for the fusion rules in the shearlet domain. Next, the discrete shearlet transform is applied on the input images. Then, low- and high-frequency coefficients are combined based on feature maps obtained from the DL model. Finally, inverse shearlet transform is applied to reconstruct the final fused image.

The authors of [22] presented a multimodality medical image fusion approach that relies on SR with two-scale image decomposition. In this approach, the input images are first pre-processed by a contrast enhancement procedure to improve their intensity distributions. Then, the edge information from the enhanced images is extracted using the spatial gradient edge detection technique. The enhanced input images are then decomposed into the base layer and the detail layer. The final fused detail layer is obtained by utilizing the spatial stimuli gradient sketch model. In contrast, the average-based fusion rule is used to fuse the base layers. Then, the extracted edge information is exploited for calculating the activity level maps. Finally, the fusion result is obtained by the linear integration of fused detail and base layers.

Rajalingam [23] adopted a Siamese convolutional network to create a weight map from the original images. This map combines the pixel movement information. The Gaussian pyramid decomposition is applied to decompose the obtained weight map. Additionally, the input images are decomposed using the Laplacian pyramid multi-scale decomposition. For every decomposition level, the local energy map is evaluated. For the fusion process, a local comparison-based method is implemented to correct the fusion mode for the decomposed coefficients, adaptively. Finally, the fused image is acquired from the Laplacian pyramid.

A multimodal medical image fusion scheme in the NSST domain was presented in [24]. In this scheme, the NSST decomposition technique is implemented to obtain the multi-scale and multi-direction representations of the two input images. The high-frequency sub-bands are merged using a parameter-adaptive PCNN model, while the low-frequency coefficients are merged by a strategy that aims to extract details and preserve energy, simultaneously. Finally, the fusion result is obtained by applying the inverse NSST.

The study introduced in [25] provided a method for merging medical images that relies on sparse representation (SR) and a decision map. Initially, the source images are divided using a sliding window. All obtained patches are set into vectors from left to right and from top to bottom. Then, these vectors are gathered into pairs of vectors related to the corresponding positions of original patches. Three decision maps are designed based on the extracted features: energy information map, structure information map, and structure and energy map to preserve more edge and energy information. The energy map includes features of the energy and energy distribution determined by the mean square deviation. The structure map contains the local structure features obtained by a Gaussian Laplacian. The decision maps are added to the normal SR method to enhance the speed of the fusion system. After that, the decision maps are utilized to select which vector of each group is used as the input to the SR system. Finally, the SR system is used to fuse the vector pairs.

A hybrid multimodality medical image fusion approach based on NSCT and NSST decomposition techniques was reported in [26]. First, the low-frequency and high-frequency coefficients of the original images are calculated using the NSCT. Then, for the obtained coefficients, the NSST is performed. The low-frequency coefficients are combined using the average fusion rule, while the maximum fusion rule is applied for fusing high-frequency sub-bands. The final integrated image is reconstructed by applying the inverse NSCT and inverse NSST.

Prakash et al. [27] addressed the medical image fusion problem by exploiting  the lifting-scheme-based biorthogonal wavelet transform (BWT) at different scales. First, the original images are decomposed using the lifting-scheme-based BWT at multiple scales into low-frequency and high-frequency coefficient sub-bands. Then, the obtained sub-bands are fused using either maximum or average fusion rule. In the final stage, the inverse BWT is implemented to reconstruct the final fusion result.

The authors of [28] proposed a hybrid pixel- and feature-level approach for multimodal medical image fusion. First, the original images are decomposed into low-frequency and high-frequency components in the pixel-level fusion stage by applying the DWT. Then, the low-frequency components are fused with the maximum fusion rule. For the high-frequency components, the curvelet transform is implemented, and the high-frequency sub-bands of the curvelet coefficients (fine-scale) are fused by using the PCA-based fusion rule. On the other hand, the feature-level fusion stage is implemented by obtaining various features from the curvelet coarse and detail sub-bands and exploiting them for the fusion process. These features comprise entropy, mean, variance, visibility, and standard deviation. Afterwards, the inverse curvelet transform is implemented on the fused high-frequency coefficients. The final merged image is obtained by implementing the inverse DWT on the fused low- and high-frequency coefficients.

The work of [29] addressed the multimodal medical image fusion problem using DWT and GAs. First, the two source images are decomposed into high- and low-frequency contents by performing the DWT. Next, the GA optimization is implemented to select the frequency coefficients suitable for the image fusion process. Then, the Bayesian model is applied to fuse the optimized coefficients. Finally, the fusion result is reconstructed by applying the inverse DWT.

The authors of [30] employed the coupled dictionary learning for medical image fusion. Firstly, a decomposition approach is used to divide the original images into their independent and correlated components. The correlated components are extracted by the SR technique, whereas the independent components are captured by the Pearson correlation algorithm. An alternating optimization procedure is implemented for source image decomposition and dictionary learning, simultaneously. The final fusion result is acquired by applying the max-absolute-value fusion rule on the correlated components only, while the whole independent components of the two images are preserved.

In [31], a multimodality medical image fusion method based on the NSST was presented. First, the NSST is applied on the source images to get a set of high-frequency sub-bands and a low-frequency sub-band. Then, the structure similarity between sub-bands is assessed. The fusion process in this method is accomplished in two stages. For the low-frequency sub-band fusion, the regional features are captured from the sub-bands with high similarity using region variance and energy. After that, the fusion step is implemented based on region feature weighting, while the sub-bands with low similarity are merged by the improved link synaptic computing network  (LSCN) model. The high-frequency sub-bands with low similarity are fused using improved guided filtering, while the sub-bands with high similarity are combined by the maximal regional absolute value. Finally, the fusion result is obtained by the inverse NSST.

In summary, the researchers resolve the drawbacks of spatial-domain fusion approaches by using transform-domain fusion. The fusion results of spatial-domain approaches have block artifacts on image boundaries. Furthermore, the final fusion results of these approaches suffer from spatial distortions and low signal-to-noise ratios (SNRs). On the other hand, each transform technique possesses its merits and disadvantages. For example, the DWT suffers from some shortcomings such as aliasing, directionality deficiency, and shift variance. In addition, the DWT has lack of the edge and curve representation of images. Therefore, to avoid these limitations, the researchers addressed some multi-scale methods in image fusion for their directionality, shift-invariance, and high ability to capture spatial structures of images. Examples of these transforms are curvelet transform, NSCT and NSST. However, these decomposition techniques also have their advantages as well as some limitations. Therefore, the authors were directed towards approaches that merge different transforms to incorporate their merits [7].

Recently, researchers have suggested fusion systems based on different optimization techniques to enhance the final fusion results. However, optimization-based fusion algorithms have a common problem; the fusion result quality is not high enough, since these algorithms require multiple iterations to select the global optimal solution. Furthermore, these methods have another drawback, which is over-smoothing of the output fused image.

Theoretical Background

In this section, we will address the related theories implemented in this work.

Non-subsampled Contourlet Transform

In 2006, Da Cunha et al. developed the NSCT, a multi-scale, shift-invariant, and multi-directional decomposition technique with fast implementation [32]. The NSCT provides much directional information, which is important in the reconstruction of images. Besides, it offers a set of sub-bands that are appropriate for image fusion. To construct the NSCT, two shift-invariant tasks are implemented with non-subsampled directional filter banks (NSDFBs) and non-subsampled pyramid (NSP). The NSP offers the multi-scale property, while the NSDFBs provide directional information. The NSCT decomposition technique is carried out as follows. Firstly, the image is disintegrated into a low-frequency sub-band and band-pass sub-bands with a multi-scale NSP. The band-pass sub-bands are decomposed into a series of band-pass sub-images with different directions using the NSDFBs. Figure 1 shows a summary of the NSCT decomposition. The NSCT is a very efficient decomposition technique in the field of image processing, especially for image fusion, due to its ability to extract essential geometrical information from the input images [15, 33].

Fig. 1.

The NSCT decomposition. a NSFB structure. b Obtained idealized frequency partitioning

Stacked Sparse Auto-encoder (SSAE)

The SSAE is a common class of deep neural networks employed in many image processing problems. The SSAE consists of several layers of auto-encoders, where the output of a layer is delivered as input to the next layer to generate more important features. The main objective of the auto-encoder is to determine the correlation between high-dimensional data to produce useful feature representations [5]. These feature representations are valuable for the image fusion process. Generally, the training stage of the SSAE includes two phases: unsupervised pre-training and then supervised fine-tuning. For the pre-training phase, an unsupervised method is implemented. The features are extracted from unlabeled data at the training step of the auto-encoder by learning the encoder and the decoder jointly [34, 35]. In the fine-tuning stage, one of the popular approaches is used to enhance the SSAE performance [36].

The structure of the auto-encoder is explained as follows. The input layer is followed by the hidden layers, and finally, the output layer, as displayed in Fig. 2. The procedure of an auto-encoder training stage includes two phases: encoder and decoder. First, the encoder is applied to map the source data into a hidden representation, whereas the decoder is used to recreate the input data from the hidden representation [34].

Fig. 2.

Structure of the stacked auto-encoder

Assume the unlabeled input data is ${\left\{{\mathbf{x}}_n\right\}}_{n=1}^N$, and ${\mathbf{h}}_n$ is the vector of the hidden encoder obtained from ${\mathbf{x}}_n$, whereas ${\widehat{\mathbf{x}}}_n$ is the vector obtained from the output layer of the decoder [37].

Therefore, the encoding stage is described as follows:

$${\mathbf{h}}_n=f({\mathbf{W}}_1{\mathbf{x}}_n+{\mathbf{b}}_1)$$ 1

where f  is the function of the encoding stage, W₁ is the encoder weight matrix, and b₁ is the bias vector.

The decoder process is defined as follows:

$${\widehat{\mathbf{x}}}_n=g({\mathbf{W}}_2{\mathbf{h}}_n+{\mathbf{b}}_2)$$ 2

where g is the function of the decoding stage, W₂ is the decoder weight matrix, and b₂ is the bias vector.

The parameters of the auto-encoder are adjusted to decrease the reconstruction error represented as:

$$\varnothing \left(\mathrm{\Theta }\right)={\arg }_{\theta ,{\theta }^{{}^{\prime }}}\min \frac{1}{N}{\sum }_{n=1}^{N}L({\mathbf{x}}^n,{\widehat{\mathbf{x}}}^n)$$ 3

where $L\left(\mathbf{x},,,\widehat{\mathbf{x}}\right)={\Vert \mathbf{x}-\widehat{\mathbf{x}}\Vert }^2$ denotes the loss function.

The SSAE-based approach can be summarized as follows:

1. Training of the first auto-encoder by the input data and acquiring the learned feature vector.

2. Delivering the obtained feature vector from the input layer as the input to the subsequent layer, and then repeating this step until the training process is finished.

3. After the training of all hidden layers is finished, the back-propagation algorithm is implemented to decrease the cost function and update the weights to allow fine tuning.

Proposed Method

The proposed method is illustrated in Fig. 3. The steps of the proposed method are described as follows:

1. The original images are disintegrated into low-frequency and high-frequency contents by implementing the NSCT decomposition technique.

2. The low-frequency components represent the image approximation. These components are merged by applying the maximum-based fusion rule.

3. The high-frequency components comprise the important detail information of the images, and these components are combined by the SSAE with the choose-max fusion approach.

4. The final fusion result is obtained by implementing the inverse NSCT on the fused low-frequency and high-frequency coefficients.

Fig. 3.

Framework of the proposed method

Fusion of NSCT Sub-band Coefficients

Fusion of the Low-Frequency Coefficients

The low-frequency coefficients represent the approximations of input images, which comprise most of the energy of the images. In the suggested method, the maximum-based fusion rule is adopted to combine the low-frequency coefficients as

$$C_L^F(i,j)=\mathrm{Max}(C_L^A(i,j),C_L^B(i,j))$$ 4

where $C_L^A\text{and}{C}_L^B$ represent the low-frequency coefficients of the source images A and B, $C_L^F$ represents the fused low-frequency coefficients, and i and j are the pixel positions of the sub-images. The maximum fusion rule assists in improving the visual quality of the final fusion results leading to better contrast.

Fusion of the High-Frequency Coefficients

The fusion strategy for the high-frequency components is the main contribution of the proposed method. The high-frequency components are significant for an image, since they include most of the details. Thus, the fusion of the high-frequency components greatly affects the quality of the final fusion result. Some existing methods are based on hand-crafted features as the activity-level measures of the original images. These features include variance, entropy, and gradient features, which are used by the fusion rule. However, these traditional methods, performed by humans, often cause inexact image representations, which lead to inferior fusion results due to the lost information [5]. Currently, most studies are directed to extracting features by machines automatically instead of humans. Thus, most researchers were directed to deep neural networks. These networks deliver the images and extract features, automatically. Compared to hand-crafted feature extraction methods, these methods achieved great success in image representation.

In the proposed method, the SSAE is exploited to obtain high-level features from  the high-frequency components, which are utilized later in the fusion process.

The fusion procedure is displayed in Fig. 4, and it is summarized as follows:

1. The high-frequency components of the two input images are divided into blocks by performing a sliding window technique.

2. All obtained blocks from images A and B are reshaped into vectors ${\mathbf{V}}_H^A,{\mathbf{V}}_H^B$.

3. The coefficient vectors are delivered as input to a two-layer SSAE for training, which produces the feature vectors.

4. The SF of the feature vectors is obtained to be adopted in the fusion process. The SF reflects the overall activity level in an image besides its sharpness. The higher the SF, the larger the image resolution is [38]. The SFs for the high-frequency sub-bands of the two source images are defined as follows:

   $${\mathrm{SF}}_H^A=\sqrt{{({\mathrm{RF}}_H^A)}^2+{({\mathrm{CF}}_H^A)}^2}$$ 5

   $${\mathrm{SF}}_H^B=\sqrt{{({\mathrm{RF}}_H^B)}^2+{({\mathrm{CF}}_H^B)}^2}$$ 6

   $${\mathrm{RF}}_H^A=\sqrt{\frac{1}{M(N-1)}{\sum }_{i=1}^M{\sum }_{j=2}^N{(C_H^A\left(i,,,j,-,1\right)-C_H^A(i,j))}^2}$$ 7

   $${\mathrm{CF}}_H^A=\sqrt{\frac{1}{(M-1)N}{\sum }_{i=2}^M{\sum }_{j=1}^N{(C_H^A\left(i,,,j\right)-C_H^A(i-1,j))}^2}$$ 8

   $${\mathrm{RF}}_H^B=\sqrt{\frac{1}{M(N-1)}{\sum }_{i=1}^M{\sum }_{j=2}^N{(C_H^B\left(i,,,j,-,1\right)-C_H^B(i,j))}^2}$$ 9

   $${\mathrm{CF}}_H^B=\sqrt{\frac{1}{(M-1)N}{\sum }_{i=2}^M{\sum }_{j=1}^N{(C_H^B\left(i,,,j\right)-C_H^B(i-1,j))}^2}$$ 10

   where ${\mathrm{SF}}_H^A$ and ${\mathrm{SF}}_H^B$ are the spatial frequencies of the high-frequency sub-bands of images A and B. ${\mathrm{RF}}_H^A$ and ${\mathrm{RF}}_H^B$ are the row frequencies; ${\mathrm{CF}}_H^A$ and ${\mathrm{CF}}_H^B$ are the column frequencies of the high frequency sub-bands $C_H^A$, and $C_H^B$ of images A and B. M is the number of rows, and N is the number of columns.
5. For each pair of coefficient vectors, we select the one with the maximum SF as the fusion result:

   $$C_H^F\left(i,,,j\right)=\left\{\begin{array}{c}{C}_H^A(i,j),if\left({\mathit{SF}}_H^A,\ge ,{\mathit{SF}}_H^B\right)\\ C_H^B(i,j),if({\mathit{SF}}_H^A<{\mathit{SF}}_H^B)\end{array}\right\}$$ 11

where $C_H^F$ is the fused high-frequency sub-band, $C_H^A$ and $C_H^B$ are the high-frequency sub-bands of the two source images, and ${\mathit{SF}}_H^A{\mathit{and}SF}_H^B$ are the SFs of the feature vectors.

Fig. 4.

Fusion of the high-frequency coefficients

Grayscale and Color Medical Image Fusion

In medical imaging techniques, functional modalities like SPECT and PET are usually presented in pseudocolor, and in the fusion process, these images are considered color images with RGB channels. This section clarifies the fusion of anatomical (MR) images and functional (SPECT, PET) images. In medical image fusion, the grayscale and color images can be merged by combining the grayscale image with each component of the RGB image. Then, the fusion result is reconstructed by integrating the three fused RGB components, but this method suffers from color distortion. To overcome this problem, separating the color image through some color space transform techniques can be implemented for fusion. In this paper, the hue, saturation, and value (HSV) space is exploited to perform the fusion of grayscale and color images. The HSV fusion converts the RGB image into HSV components, making it effective for separating color image channels [39, 40]. The suggested fusion method of grayscale and color images is carried out through the following stages. Firstly, the RGB image is transformed into HSV space to find the H, S, and V components. Then, the proposed fusion method is applied on the grayscale image and the V component. Finally, the final fused image is reconstructed by applying the inverse HSV conversion (HSV to RGB) on the fused V component and the original H and S components. Figure 5 displays the steps of the grayscale and color image fusion procedure.

Fig. 5.

Grayscale and color image fusion procedure

Results

Performance Analysis

To prove the effectiveness of the suggested fusion method, different types of medical images are utilized in the experiments. In the experimental results, several objective measures that are generally adopted in image fusion are used for quantitative evaluations. These objective indicators are entropy, standard deviation (std) [41], edge intensity [42], mutual information (MI) [43], feature mutual information (FMI) [41], and edge strength Q^AB/F [44].

Entropy

Entropy is defined as the amount of information contained in the image. The maximum value of entropy is based on the variability of the pixel values. For the images with 256 gray levels, the values of entropy range between 0 and 8, whereas the values of entropy for images with 128 gray levels range from 0 to 7. Entropy is defined as:

$$\mathrm{Entropy}=-{\sum }_{L=0}^{L-1}{p}_i\times {\log }_2{p}_i$$ 12

where L is the number of gray levels and p_i is a probability density function for each gray value i. The entropy value is related to the content of the information included in the fusion result.

Edge Intensity

The edge intensity of an image is assessed using the Sobel operator. A higher value of edge intensity yields an enhancement of the image quality [42]. The edge intensity is measured as:

$$\mathrm{Edge}\mathrm{intensity}=\sqrt{I_x^2+I_y^2}$$ 13

where $I_x=h_x\otimes f$, $I_y=h_y\otimes f$ for the image f.

and $h_x=\left[\begin{array}{c}101\\ -202\\ -101\end{array}\right]$, $h_y=\left[\begin{array}{c}-1-2-1\\ 000\\ 121\end{array}\right]$  

Standard Deviation

The standard deviation measures the amount of details included in the image. The large value of standard deviation indicates that the image has a high degree of contrast and clarity. Mathematically,

$$\mathrm{std}=\sqrt{\frac{{\sum }_{i=1}^M{\sum }_{j=1}^N{(F\left(i,,,j\right)-\mu )}^2}{M\times N}}$$ 14

where M and N are the dimensions of the image F(i, j), and µ is the mean value.

Mutual Information

The MI represents the dependence between the original images and the fusion result. The MI signifies how much amount of information included in the source images has been transferred to the fusion result. It is related to the information amount transferred to the fusion result. It is calculated as:

$$I\left(A,,,F\right)={\sum }_{A,F}p(A,F)\log \frac{p(A,F)}{p(A)p(F)}$$ 15

$$I\left(B,,,F\right)={\sum }_{B,F}p(B,F)\log \frac{p(B,F)}{p(B)p(F)}$$ 16

where p(A), p(B), and p(F) are the edge probability density functions of the two original images and the fusion result, respectively, whereas p(A, F) and p(B, F) are the joint probability density functions.

Subsequently, the image fusion evaluation can be defined as:

$$I_F^{\mathit{AB}}=I\left(A,,,F\right)+I\left(B,,,F\right)$$ 17

Feature Mutual Information

The FMI metric measures the amount of feature information transferred from the two source images into the final fused image. Mathematically,

$${\mathrm{FMI}}_F^{\mathit{AB}}={\mathrm{FI}}_{\mathit{AF}}+{\mathrm{FI}}_{\mathit{BF}}$$ 18

$${\mathrm{FI}}_{\mathit{AF}}={\sum }_{A,F}{p}_{\mathit{FA}}\left(i,,,j,,,k,,,l\right){\log }_2\frac{p_{\mathit{FA}}\left(i,,,j,,,k,,,l\right)}{p_F\left(i,,,j\right)p_A\left(k,,,l\right)}$$ 19

$${\mathrm{FI}}_{\mathit{BF}}={\sum }_{A,F}{p}_{\mathit{FB}}(i,j,k,l){\log }_2\frac{p_{\mathit{FB}}(i,j,k,l)}{p_F(i,j)p_B(k,l)}$$ 20

where ${\mathrm{FI}}_{\mathit{AF}}$ and ${\mathrm{FI}}_{\mathit{BF}}$ are the feature information transferred from the source images A and B into the fused image F, respectively, $p_{\mathit{FA}}$ is the joint distribution function between image A and fused image F, and $p_{\mathit{FB}}$ is the distribution function between the source image B and fused image F.

Edge Strength (${\mathit{Q}}^{\mathit{A}\mathit{B}/\mathit{F}}$ Metric)

The $Q^{AB/F}$ represents the content of edge information passed from the original image to the final fused image. The high value of Q^AB/F indicates the more ability of the proposed method to transfer the edge information to the fused image. The mathematical expression of $Q^{AB/F}$ is given by:

$$Q^{AB/F}=\frac{{\sum }_{i=1}^M{\sum }_{j=1}^N{Q}^{A/F}\left(i,,,j\right)h_A\left(i,.,j\right)+Q^{B/F}\left(i,,,j\right)h_B\left(i,,,j\right)}{{\sum }_{i=1}^M{\sum }_{j=1}^N{h}_A\left(i,,,j\right)+h_B\left(i,,,j\right)}$$ 21

$$Q^{A/F}\left(i,,,j\right)=Q_h^{A/F}(x,y)Q_{\propto }^{A/F}(x,y)$$ 22

$$Q^{B/F}\left(x,,,y\right)=Q_h^{B/F}(m,n)Q_{\propto }^{B/F}(x,y)$$ 23

where $Q^{A/F}$ and $Q^{B/F}$ are the values of edge information preservation for images A and B, $h_A$ and $h_B$ are the weight values of the input images, and $Q_{\propto }^{A/F}$ and $Q_{\propto }^{B/F}$ signify the edge strength and edge orientation values of a pixel.

Experimental Results

This section provides the performance analysis of the proposed fusion method. The multimodal medical images used in this work are MRI-CT, MRI: T1-T2, MRI-MRA, MRI-PET, and MRI-SPECT. The results demonstrate that the fusion of these multimodal images produces fusion results containing complementary information from different modalities. All source images can be obtained from http://www.metapix.de/fusion.htm, www.med.harvard.edu/aanlib/home.html, and www.imagefusion.org. The essential condition of the suggested method is that the source images should be preregistered. The proposed hybrid fusion method is evaluated analytically and visually. We implemented the proposed method with MATLAB 2017b, and experiments were done on a PC with 2.20-GHz CPU and 8-GB RAM.

The outcomes of the conducted experiments are demonstrated in Figs. 6–10. The objective evaluation is performed to measure the quality of the fusion results with different metrics. The quantitative results are presented in Tables 1, 2, 3, 4 and 5.

Fig. 6.

Fusion results of the MR and CT images

Fig. 10.

Fusion results of MR and SPECT images

Table 1.

Quantitative assessment results of different cases of MR and CT image fusion

Datasets	Size	Metric	Source image 1	Source image 2	Fused image
Group (1)	256 × 256	Entropy	6.633	2.032	6.767
		Std	57.121	37.469	68.572
		Edge intensity	0.229	0.124	0.278
		MI	6.359
		FMI	0.9059
		QAB/F	0.792
Group (2)	256 × 256	Entropy	3.446	2.677	4.433
		Std	55.384	83.369	90.406
		Edge intensity	0.208	0.281	0.317
		MI	3.277
		FMI	0.8605
		QAB/F	0.619
Group (3)	256 × 256	Entropy	7.5222	7.2754	7.7461
		Std	58.330	81.221	75.8535
		Edge intensity	0.5356	0.4527	0.5602
		MI	6.0176
		FMI	0.8226
		QAB/F	0.5540

Table 2.

Quantitative assessment results of different cases of MRI-T1 and MRI-T2 image fusion

Datasets	Size	Metric	Source image 1	Source image 2	Fused image
Group (4)	256 × 256	Entropy	3.332	3.081	4.443
		Std	51.542	50.367	62.493
		Edge intensity	0.235	0.191	0.253
		MI	2.918
		FMI	0.8499
		QAB/F	0.568
Group (5)	256 × 256	Entropy	3.585	3.263	4.598
		Std	60.542	54.857	71.000
		Edge intensity	0.283	0.152	0.286
		MI	7.742
		FMI	0.9099
		QAB/F	0.664
Group (6)	256 × 256	Entropy	2.975	3.209	4.081
		Std	58.983	63.582	74.181
		Edge intensity	0.197	0.172	0.202
		MI	3.414
		FMI	0.8944
		QAB/F	0.575

Table 3.

Quantitative assessment results of different cases of MRI and MRA image fusion

Datasets	Size	Metric	Source image 1	Source image 2	Fused image
Group (7)	256 × 256	Entropy	5.868	4.912	5.889
		Std	66.369	24.045	70.203
		Edge intensity	0.296	0.155	0.2967
		MI	5.636
		FMI	0.9144
		QAB/F	0.647

Table 4.

Quantitative assessment results of different cases of MR and PET image fusion

Datasets	Size	Metric	Source image 1	Source image 2	Fused image
Group (8)	128 × 128	Entropy	4.424	2.689	4.598
		Std	58.039	29.307	89.802
		Edge intensity	0.374	0.287	0.429
		MI	11.799
		FMI	0.8527
		QAB/F	0.655
Group (9)	256 × 256	Entropy	2.273	2.494	3.021
		Std	51.191	41.909	63.252
		Edge intensity	0.237	0.097	0.172
		MI	12.169
		FMI	0.8654
		QAB/F	0.459

Table 5.

Quantitative assessment results of different cases of MR and SPECT image fusion

Datasets	Size	Metric	Source image 1	Source image 2	Fused image
Group (10)	256 × 256	Entropy	4.653	4.405	6.204
		Std	51.737	45.004	64.925
		Edge intensity	0.251	0.125	0.207
		MI	6.665
		FMI	0.8466
		QAB/F	0.450
Group (11)	256 × 256	Entropy	4.485	2.499	5.641
		Std	68.668	38.049	69.361
		Edge intensity	0.356	0.085	0.279
		MI	3.212
		FMI	0.8218
		QAB/F	0.539

It is apparent from Figs. 6–10 that the suggested fusion method retains and improves the important image structures, morphological features, and color information of the source images, avoiding artifacts. Besides, important information, such as the shape and boundary of the input images, is perfectly conserved.

In Fig. 6, for groups (1), (2), and (3), the CT brain image illustrates the structures of bone, whereas the MR image shows the soft tissues of the brain. The tissue and bone information positioned at similar locations have to be determined to help in the pathological analysis. Therefore, the fused version of the CT and MR images provides the required information more than a single modality. Furthermore, as shown in groups (1), (2), and (3) in Fig. 6, the obtained fusion results can preserve the visual information at precise image locations with the same brightness and contrast in accordance with the source images.

In Figure 7, groups (4), (5), and (6) demonstrate the MR-T1 and MR-T2 image fusion. The MR-T1 images represent the contrast version of the MRI technique that is the most usually exploited for brain structure analysis. However, these images suffer from unclear boundaries of brain structures. This problem stimulates fusing the image with different weighted MR images to get images with improved quality and complementary information to be effective for clinical analysis. In the suggested method, MR-T1 and MR-T2 images are fused to improve the boundaries of the structure and promote image quality, as shown in groups (4), (5), and (6) in Fig. 7.

Fig. 7.

The fusion results of MRI-T1 and MRI-T2 images

The fusion of the MRA and MR-T1 images is presented in Fig. 8, group (7). The MRA image is acquired by an MRI technique that precisely illustrates vessels like arteries. The MR-T1 image reveals the soft tissue of the brain and provides superior anatomical structures. For the medical analysis and disease diagnosis, both image contents at the same position are necessary to be investigated and observed, which is seldom encountered in imaging techniques. From the visual results, the fused image accurately preserves the visual details of the input images at their exact locations, with the same contrast and brightness.

Fig. 8.

Fusion results of MRI and MRA images

For the fusion cases in Fig. 9, for groups (8) and (9), the PET image denotes the functional condition of the brain, while the MR image shows the soft tissue details with anatomical structures of the brain image. The obtained fusion results display the finest performance comprising the functional and morphological information. Besides, edges, color, and brightness are well preserved by the suggested method.

Fig. 9.

Fusion results of MRI and PET images

In Figure 10, groups (10) and (11) display the fusion of the MR and SPECT images. The SPECT image gives the blood flow and tissue of the brain. The suggested method achieves a good representation of the final integrated image, where the morphological and color information of both input images are precisely preserved.

In conclusion, the proposed fusion method accurately conserves and improves the morphological features, the significant image structures, and the color information from the source images without reconstructing any artifacts. This is due to the utilization of the NSCT, which is deemed a suitable tool for medical image edge representation as most of these images have curved shapes. Furthermore, the NSCT provides several decomposition sub-bands, which gives flexibility in image fusion. Over and above, the utilization of the DL achieves a breakthrough in feature extraction and data representation.

Owing to the difference in subjective evaluation between different people, the quantitative parameters generally utilized in the literature are chosen to evaluate the fusion results. The image fusion is common in various aspects of capturing important information from input images, preserving edges, and enhancing the sharpness and visual perception quality. However, there is no uniform standard to assess the quality of the obtained images.

In this section, we debate the quantitative evaluation of the fusion results in detail. The entropy displays the information of the fusion results. In contrast, the MI is the measure of information transferred from the original images into the obtained image. The higher value of MI points to more details and information acquired from the source images, which also affirms more clarity and activity levels in the fused images. Moreover, the FMI computes the MI of the image features. As the value of FMI is high, the quality of the final fused image is high. The high values of entropy, MI, and FMI of a fused image prove the high quality. The standard deviation displays the contrast level of the fused image. The higher value of standard deviation indicates that the obtained fused image has higher contrast than the original images. Over and above, we addressed other metrics, including edge intensity, and Q^AB/F that estimate the amount of edge information moved from the original images into the fusion result. These metrics are essential for medical image fusion. Larger values of these metrics denote much edge details as in preserving texture and bone structure, which is helpful for disease analysis. Generally, for all mentioned evaluation parameters, a higher score points to better performance. According to the quantitative analysis of the experimental results listed in the tables, the suggested method achieves the best performance in entropy, standard deviation, MI, FMI, edge intensity, and Q^AB/F. This denotes that the fused image has the best information preservation, less distortion, high contrast, and it comprises denser structures, texture, tissue information, and edge details.

Comparison with the Existing Methods

Comparison with the state-of-the-art methods in the area of medical image fusion is a challenging issue, because various studies adopted different datasets for experimentation. Moreover, several researchers utilized different metrics to evaluate their methods. However, the results attained by the proposed method are compared with those of several recent existing methods in a quantitative manner using the most common datasets and metrics. These approaches include CNNs with DCSCM, and NSST [16], CNNs with NSST [17], fast curvelet transform [12], NSCT-SR-PCNN [45], image decomposition and NSST [18], BWT [27], NSST with improved LSCN [31], hybrid pixel- and feature-level fusion [28], feature-level fusion [46], NSST and PCNN [47], and intuitionistic fuzzy sets [48]. Table 6 summarizes the performance of the proposed method and those of the existing methods. In addition, Figs. 11–15 display the subjective comparison between the proposed method and peer methods.

Table 6.

Comparison with existing methods

Modal	Methods	Metrics
		Entropy	MI	QAB/F
MR and CT fusion Group (1)	The CNNs, DCSCM, and NSST [16]	6.5997	2.6023	0.7276
	The CNNs with NSST [17]	6.7612	5.7545	-
	Fast curvelet transform [12]	-	3.3121	0.4262
	NSCT-SR-PCNN [45]	-	2.2426	0.6899
	Image decomposition framework and NSST [18]	6.4179	2.3761	-
	BWT [27]	5.9985	1.7880	0.6672
	NSST with improved LSCN [31]	7.021	3.796	0.729
	Hybrid pixel and feature level fusion [28]	6.7574	3.8827	0.7586
	Proposed method	6.767	6.359	0.792
MRI and CT fusion Group (2)	PCNN with NSST [49]	-	2.9259	0.5280
	PCNN with NSST [50]	-	2.9388	0.5329
	PCNN with NSST [24]	-	2.9265	0.5365
	Proposed method	4.433	3.277	0.619
MRI and PET fusion Group (8)	NSCT-SR-PCNN [45]	-	2.756	0.5681
	Fuzzy logic in NSCT domain [51]	-	3.0	0.65
	Proposed method	4.361	11.799	0.548
MRI and PET fusion Group (9)	Feature level fusion [46]	-	2.9074	0.8133
	NSST and PCNN [47]	3.4316	-	-
	Proposed method	3.021	12.169	0.459
MRI and MRA Group (7)	Intuitionistic fuzzy sets [48]	5.956	-	0.710
	Dictionary learning method [52]	4.9823	3.6796	0.6819
	Hybrid pixel and feature level fusion [28]	6.027	3.6747	0.669
	BWT [27]	6.6913	4.7852	0.6318
	Proposed method	5.889	5.636	0.645

Fig. 11.

Comparison with the existing methods for MR and CT fusion group (1)

Fig. 15.

Comparison with the existing methods for MRI and MRA fusion group (7)

The best score in each parameter is shown in boldface. By the previous definition of each metric, the best medical image fusion method should achieve higher values of entropy, MI, edge intensity, and Q^AB/F. Table 6 demonstrates that the recommended method is better in terms of these indicating scores.

In case of MR and CT image fusion, the results in Table 6 indicate that the performance of the proposed method is superior to those of the baseline methods. It is clear from the results that the suggested method outperforms the peer methods in terms of MI and Q^AB/F. The proposed method gives fused images with more details compared to the method of [31] (Figs. 11 and 12).

Fig. 12.

Comparison with the existing methods for MR and CT fusion group (2)

For MR and PET fusion group (8), our method outperforms the compared method with a large difference in terms of MI. This indicates that the suggested method produces images with more details. The work in [51] is the only method that outperforms the suggested method in Q^AB/F. For group (9), the method in [46] is the only method that outperforms the introduced method in Q^AB/F. However, the proposed method outperforms the compared methods in term of MI (Figs. 13 and 14).

Fig. 13.

Comparison with the existing methods for MR and PET fusion group (8)

Fig. 14.

Comparison with the existing methods for MR and PET fusion groups (8) and (9)

In the case of MRI and MRA fusion, the proposed method reports the best MI score. This signifies that more information is preserved in the fused image. Only the method in [27] outperforms the proposed method in entropy (Fig. 15).

Conclusion and Future Work

In this paper, a multimodal medical image fusion method was proposed based on SSAE and NSCT. The NSCT provides both the direction and multi-scale analysis of the original images, whereas the SSAE, a widespread tool of deep neural networks, is applied for feature extraction. The SSAE is an efficient unsupervised feature extraction technique, which has high ability of complex data representation. Motivated by the stated merits, an image fusion method that relies on SSAE and NSCT was proposed.

The NSCT is applied to disintegrate the source images into low- and high-frequency coefficients. In terms of high-frequency coefficients, the SSAE is used to obtain features from all image blocks. After training the SSAE using the obtained coefficient vector, the output of the SSAE signifies the high-frequency representation, which is utilized later in the fusion process. Based on the previous analyses, the high-frequency sub-bands are merged by using a fusion rule that relies on the SSAE. The low-frequency coefficients, on the other hand, are combined by applying the maximum fusion rule. The qualitative evaluation proves the appropriateness of the proposed fusion method for efficient and precise clinical analysis. The objective examination inferred that the suggested fusion method produces images with accurately reconstructed attributes and the complementary features of both input images. The comparison with the existing methods shows that the proposed fusion method provides better scores for entropy, MI, SSIM, and edge quality.

Future research works can be directed to the enhancement of fusion strategies based on decision level. Furthermore, the improvement of image fusion schemes based on different optimization techniques and the application of the DL in the field of multimodal medical image fusion have emerged as active topics in the last years, and they need more research work.

Author Contribution

The idea for the manuscript was conceived at a meeting attended by Nahed Tawfik, Heba A. Elnemr, Mahmoud Fakhr, Moawad I. Dessouky, and Fathi E. Abd El-Samie. Nahed Tawfik and Heba A. Elnemr participated in implementing all experiments and analyzing the experimental results. In addition, Nahed Tawfik and Heba A. Elnemr participated in the research plan, the study organization, and the manuscript writing. All authors reviewed the manuscript and were involved in its critical revision before submission.

Data Availability

The data that support the findings of this study are openly available. All the source images can be obtained from http://www.metapix.de/fusion.htm, www.med.harvard.edu/aanlib/home.html, and www.imagefusion.org.

Code Availability

The code that supports the findings of this study is available from the corresponding author, Nahed Tawfik, upon reasonable request.

Declarations

Ethics Approval

This article is original and contains unpublished material. The corresponding author confirms that no ethical issues involved.

Consent to Participate

I understand that even if I agree to participate now, I can withdraw at any time or refuse to answer any question without any consequences of any kind.

Consent for Publication

I understand that the text and any pictures or videos published in the article.

1. will be used only in educational publications intended for professionals or

2. if the publication or product is published on an open access basis. I understand that it will be freely available on the internet and may be seen by the general public.

The pictures, videos, and text may also appear on other websites or in print, may be translated into other languages, or used for commercial purposes.

Conflict of Interest

The authors declare no competing interests.