Medical image fusion by sparse-based modified fusion framework using block total least-square update dictionary learning algorithm

Abstract.

Purpose

The technique of fusing or integrating medical images collected from single and various modalities is known as medical image fusion. This is done to improve the quality of the images and combine information from several medical images. The whole procedure aids medical practitioners in gaining correct information from single images. Image fusion is one of the fastest-growing research topics in the medical imaging field. Sparse modeling is a popular signal representation technique used for image fusion with different dictionary learning approaches. We propose a medical image fusion with sparse representation (SR) and block total least-square (BLOTLESS) update dictionary learning.

Approach

The domain of dictionary learning is the most significant research domain related to SR. An efficient dictionary increases the effectiveness of sparse modeling. Due to SR being an ongoing interesting research area, the medical image fusion process is done with a modified image fusion framework with recently developed BLOTLESS update dictionary learning.

Results

The experimental results are compared for the image fusion process using other state-of-the-art dictionary learning algorithms, such as simultaneous codeword optimization, method of optimal directions, and K-singular value decomposition. The effectiveness of the algorithm is evaluated based on image fusion quantitative parameters. Results show that the BLOTLESS update dictionary algorithm is a promising modification for the sparse-based image fusion with its applicability in the fusion of images related to different diseases.

Conclusions

The experiments and results show that the dictionary learning algorithm plays an important role in the sparse-based image fusion general framework. The fusion results also show that the proposed improved image fusion framework for medical images is promising compared with frameworks with other dictionary learning algorithms. As an application, it is also used as a tool for the fusion of different modularities of images related to brain tumor and glioma.

1. Introduction

Image fusion is the sub-domain of information fusion that combines the information from several source images into a single image called the fused image.¹ The fused image is an integrated information source that allows users to acquire more details in less time and at a lower cost. The primary goal of the fusion process is to get superior results in terms of contrast, visual experience, and quality. Since the emergence of image fusion in 1985,² its application domain have spread continuously with various input source images. In military application cameras, visible and infrared image fusion³ is employed. Multi-focus image fusion^4,5 is a common feature in digital cameras. Satellite image processing, such as pan-sharpening,^6,7 uses remote sensing image fusion. One of the most important applications of digital image fusion is medical image fusion.^8,9

In disease diagnosis, an image plays an important role. An accurate diagnosis does not rely on a single type of clinical report. It needs a large amount of information from multi-model imaging reports.¹⁰ So the fused images generated by the image fusion process reduce the chance of error in disease diagnosis. Images from several modalities are used to extract enough information for clinical diagnosing for accurate treatment. Medical imaging is human anatomy screening for analyzing and disease diagnosis. There are many medical image techniques, such as x-ray imaging, which is considered to be the best method for identifying ruptures in bones. Cross-sectional layers are created by computed tomography (CT) and magnetic resonance imaging (MRI). Although CT scans are commonly used to examine dense structures, they are not suitable for soft tissue or physiological evaluation. On the other hand, MRI allows for a clearer view of soft tissues and is frequently used to identify tumors and other abnormalities. Both scans (MRI and CT) give the best results when a specific medical diagnosis requires precise images inside the body.¹¹

Positron emission tomography (PET) gives information on blood flow in the body to reveal various cancers and heart diseases. In a PET scan, a radiolabel is injected into a subject’s body with the bio-distribution of the chemical being measured as a function of time to identify physiologic quantities associated with the compound. However, as compared with CT and MRI, PET scans have a lower resolution. Single-photon emission computed tomography (SPECT) discovers clinical information related to human body structure characteristics.¹⁰ Different sensors are employed in different imaging techniques to find imaging information of the same region of the body. As the applicability of medical image fusion is concerned, it is used in the detection of different diseases, such as cancer, Alzheimer’s disease, glioma, etc.¹² Glioma and glioblastoma are two types of brain tumors.¹³ High-pixel-intensity cells and uneven border areas characterize glioma tumors. Glioblastoma tumors are low-pixel-intensity cells that may be identified with great accuracy using a variety of traditional techniques. By fusing different modularities of images, the accurate placement of the glioma tumor can be easily detected.¹³

Image fusion methods may be classified into three primary categories: pixel-level fusion,^14,15 feature-level image fusion,¹⁶ and decision-level fusion.¹⁷ As far as the process domain of fusion is concerned, researchers have developed different techniques to fulfill image fusion tasks. These are categorized into (a) spatial domain techniques and (b) transfer domain techniques.¹⁸ A few spatial domain methods are the averaging method, select maximum/minimum method, Brovey transform method, intensity hue saturation method, and principal component analysis method. The spatial domain method directly deals with the pixel values present in images. These methods have many disadvantages, such as artifacts, noise, etc. These disadvantages lead researchers to shift to transfer domain techniques, such as discrete cosine transform, discrete wavelet transform, gradient pyramid, non-sub-sampled contourlet transform (NSCT), etc.^1,2,18 Recently, researchers have developed some improved techniques for image fusion. In this direction, compressed sensing plays an important role. Sparse representation (SR)¹⁹ is one of the attractive signal processing techniques used for image fusion in which sparse coefficients are fused as a feature-level fusion. The fusion is based on SR having three stages: (a) SR of source images, (b) fusion encoding, and (c) fused image reconstruction.²⁰

Initially, all source images are split into patches of $n\times n$ size by the sliding window method. Then all extracted patches are arranged in lexicological order followed by SR coefficients, calculated with some pursuit algorithm²¹ and a predefined dictionary. A suitable fusion rule is applied to these sparse coefficients followed by reversing the process to get a single fused image. The whole process is collectively called SR-based general fusion.²⁰ In this paper, a sparse-based fusion framework is modified and tested on medical images. This paper is divided into eight sections. Section 2 describes the literature review on digital image fusion with SR and dictionary learning. Section 3 provides a theoretical and mathematical overview of SR. Section 4 provides a thorough review of several benchmark dictionary learning techniques. Section 5 describes the proposed system of image fusion with the new dictionary algorithm and its benefits. Section 6 explains the experimental results from all three stages of the experiment and the analysis.

2. Literature Review

The intensive research continues for many years on digital image fusion, SR, and dictionary learning. For different application areas, a collaborative study on the above topics is also ongoing. Yang and Li²² were the first to propose SR multi-focus image fusion. Yu et al.²³ introduced a new image fusion approach in which source images were represented using joint SRs. Yin and Li²⁴ proposed a method of fusion called the joint sparsity model (JSM) for medical images and infrared-visible (IR-VI) images. Yang and Li²⁵ conducted image fusion for various types of image pairs with the help of the simultaneous orthogonal matching pursuit algorithm. Yu et al.²⁶ used SR for medical image fusion with K-singular value decomposition (K-SVD) algorithm,²⁷ whereas Li and Yang²⁸ first performed pan-sharpening (a form of image fusion) based on SR. Yin et al.²⁹ conducted simultaneous image fusion and super-resolution using SR for medical and remote sensing images. Yang et al.³⁰ proposed a color fusion scheme for infrared and low-light-level images with SR used for the fused image of a particular channel. Liu et al.,³¹ proposed a technique for simultaneous fusion and denoising called adaptive SR. Liu et al.³² conducted fusion of medical images through NSCT and SR in which Laplacian pyramid (LP) and high-pass (HP) coefficients were calculated by NSCT for source images followed by LP bands fused with SR. Using the absolute values of coefficients as activity level measurements, the HP bands were fused. Finally, using inverse NSCT on the combined coefficients, the fused image was constructed. Liu et al.³³ proposed a multi-scale (MS) transform and SR framework. Also, Yin,³⁴ learned an MS dictionary that was used with SR for image fusion. Zong and Qiu,³⁵ proposed a medical image fusion method using classified image patches. Aishwarya and Thangammal³⁶ suggest a multimodal medical image fusion based on SR and modified spatial frequency. Huang and Chu³⁷ recently developed the medical image fusion method using guided filtering and SR. Liu et al.³⁸ combined the LP and convolutional SR for medical image fusion to achieve good performance. Maqsood and Javed,³⁹ proposed a fusion scheme based on two-scale image decomposition and SR. Hermessi et al.⁴⁰ provided a detailed overview of medical image fusion approaches including theoretical background and current advancements in 2021.

Dictionary learning is a crucial part of the medical image fusion domain using SR. Parallel research is continuously done in the area with equal importance. The sparse-based image fusion process’s effectiveness is directly proportional to the effectiveness of the dictionary learning algorithm. An overcomplete dictionary^19,27,41,42 generates sparse coefficients for a specific signal, such as an image. Overcomplete means dictionary equations are less than the total variables. Initially, some predefined dictionaries such as Fourier, wavelet, short-time Fourier transform (STFT), and Gabor transforms^20,43,44 are used. Randomization is used to create the initial dictionary before the dictionary is to be trained. Over the years, this research area has grown positively, resulting in several benchmark algorithms, such as the method of optimal directions (MOD)⁴⁵ and K-SVD,^27,43 being developed. Many K-SVD variations, such as label consistent-KSVD,⁴⁶ non-negative K-SVD,^27,43,47 deep K-SVD,⁴⁸ discriminative K-SVD,⁴⁹ JSM-DKSVD,⁵⁰ simultaneous codeword optimization (SimCo),⁵¹ and others, were designated as milestones in the previous decade. Some dictionary learning algorithms, such as clustering KSVD⁵² and block total least-squares (BLOTLESS) update dictionary learning,⁵³ have been recently developed. The BLOTLESS update algorithm has only been utilized in image denoising. All of the abovementioned algorithms are used in various signal and image processing tasks in which SR is the key representation technique.

3. Sparse Representation

The term sparse implies many or mostly zero in a vector or matrix. Researchers tried to retrieve an image’s features from the beginning of the image processing era to simplify the tasks performed on images for better representation. At the same time, storage reduction for images is treated as an essential sub-domain. SR has been a solution for both concerns in a practical manner over two decades. SR is the method or tool for representing the image as a linear combination of a few numbers of elementary signals (atoms).²⁷ Reconstruction and mapping of an image signal to the original one are done through the overcomplete dictionary.^19,27,41,42

3.1. Mathematical Representation

Let us assume that all mathematical details are for a one-dimensional signal. The original signal that needs to be recovered is represented as vector $x$. The whole system is termed in Eq. (1) as⁵⁴

$$T\alpha =x,$$ (1)

where T is a dictionary. In this case, $\alpha$ is a spare approximation for the original signal $x$, which is represented in Eq. (2) as

$$\Vert T\alpha -x\Vert <ϵ\mathrm{.}$$ (2)

As per Eq. (2), the $ϵ$ in RHS implies that the exact recovery of signal $x$ is not required if $\alpha$ is a sparse approximation of $x$, which is termed its solution.⁵⁴

The whole under-determined system is summarized, where original signal x is recovered by an overcomplete ($N>M$ means more unknowns than equations) dictionary $T\in R^{M*N}$, $\alpha \in R^N$, and $x\in R^K$. An SR of $x$ (which is “$\alpha$”) with very few non-zero entries is called sparse coefficients. The approximate solution of the above under-determined system can be represented as an optimization problem as defined in Eq. (3). The $l_0$ norm is used for the solution; it calculates the total non-zero entries present in vector $\alpha$. This particular system is meant to be an NP-hard problem and needs an approximated solution by approximation techniques.⁵⁵ Reseachers primarily use the OMP algorithm, which is a greedy pursuit algorithm. Its improved versions are stagewise OMP (StOMP), regularized orthogonal matching pursuit (ROMP), and compressed sensing matching pursuit (CoSaMP).^27,56

$$\underset{\alpha }{\min }{\Vert \alpha \Vert }_0\text{Subject}{\Vert x-T\alpha \Vert }_2.$$ (3)

4. Dictionary Learning

Dictionary learning²⁷ is one of the research topics from the era of signal processing. The purpose of dictionary learning is to seek out the most straightforward training dictionary to sparse the signal. Generally, a dictionary is a two-dimensional (2D) matrix of $M\times N$ where each column is called atoms or basis vectors. Initially, a dictionary is a predefined matrix based on signal transform functions, such as wavelet, STFT, contourlet, curvelet, etc. Later, a dictionary trained with a particular set of training signals is generally used. The input source images or a set of natural images are also used as a training set for dictionary training. Suppose that a set of some natural images is used as the training set $X=[x1,x2,x3,x4,x5,x6,\dots \dots \dots ..]\in R^{M*N}$ and a dictionary T exists to represent each signal sparsely as an X vector. The elementary method of dictionary learning is shown in Eq. (4) as⁵⁷

$$\underset{T,\alpha }{\min }{\Vert X-T\alpha \Vert }_F^2\text{Subject}{\Vert {\alpha }_j\Vert }_0{\forall }_j.$$ (4)

Here, $j=\mathrm{1,2},\dots ,N$, and ${\alpha }_j$ denotes a column with index $j$ of vector set $\alpha$.

Many image processing tasks using DL, such as denoising, super-resolution, segmentation, fusion, and so on, are influenced by the efficiency of the DL process. Typically when finding sparse coefficients, dictionary training has to be simultaneously estimated. These two processes run separately, and their solutions iteratively alternate until convergence.⁵⁷ Generally, dictionary learning algorithms run in two stages. The SR coefficient is computed in the first stage using approximation methods such as MP and OMP.⁵⁸ The dictionary, on the other hand, is fixed during this procedure. The dictionary must be updated in the second stage.

Many algorithms are based on the above two-stage principle of dictionary learning. As far as this method is concerned, some benchmark algorithms have been proposed in the last two decades, of which the MOD⁴⁵ algorithm and the K-SVD²⁷ algorithms play a vital role. Research results estimate that KSVD is far better than the MOD algorithm. The first stage is identical for both of them. However, for the second stage, the sparse coefficient matrix is fixed in MOD.

4.1. MOD Dictionary Learning

Engan, in 2006, proposed the state-of-the-art algorithm for dictionary learning termed the “MOD.”⁴⁵ As per the ideology described in the section above, MOD is also based on the same two-stage principle. The first stage is the sparse coding stage. The second stage is the dictionary update stage. The algorithm finds the sparse coefficient matrix $\alpha$ using an approximation algorithm with dictionary T fixed in the first stage. In the second stage, dictionary T is to be updated.⁵⁹ The MOD algorithms update all dictionary atoms at a time. Initially, the algorithm performance is better compared to other algorithms. But low convergence is a drawback of the algorithm.⁶⁰

4.2. K-SVD Dictionary Learning

The drawback of the MOD algorithm was recovered in K-SVD.²⁷ As the name implies, K-SVD consists of the joint property of SVD⁶¹ and the $K$-means algorithm.⁶² The only difference between the K-SVD and MOD algorithms is at the second stage. In the second stage of the dictionary update, one column at a time is processed with SVD computation sequentially. In K-SVD, each iteration consists of updating a single atom $t_n$ at a time for dictionary $T$ updating.

4.3. SimCo Dictionary Learning

As shown in previous sections, MOD and K-SVD are two benchmark algorithms for dictionary learning. Another milestone added to the dictionary learning research was the development of the SimCo algorithm.⁵¹ Later, two versions of the same algorithms, known as primitive and regularized SimCo,⁶³ were released. This algorithm assumed that there are $l_2$ norms present in the dictionary. The improved analysis SimCo was also presented by Dong et al.⁶⁴

The dictionary problem in the base SimCO algorithm is expressed in Eq. (5) as

$$\arg \underset{Tϵ\mathcal{T}}{\min }(T)=\arg \underset{Tϵ\mathcal{T}}{\min }\underset{\alpha ϵ\varrho }{\min }{\Vert X-T\alpha \Vert }_F^2,$$ (5)

where $(T)={\min }_{\alpha ϵ\varrho }{\Vert X-T\alpha \Vert }_F^2$.

Here, the sparse coefficient matrix is defined as $\alpha \in R^{N*d}$ shown in Eq. (5), and the dictionary is defined as $T\in R^{M*N}$. The SimCo algorithm assumes that the dictionary $T$ having $l_2$ norm columns and their all matrices are represented by $\mathcal{T}$ ($\mathrm{T}ϵ\mathcal{T}$). The non-zero element of sparse coefficient matrix $\alpha$ is fixed by constraint $\alpha ϵ\varrho$. SimCo is processed in two stages. The first stage is the same as in the conventional method. In contrast, optimization methods on manifolds⁶² are applied in the second stage for dictionary $T$ updating. Updating processed on columns of T takes place with unit $l_2$ norm constraints. The sparse coefficient matrix $\alpha$ is updated as a modified dictionary $T$. During the process, the positions of non-zero elements remain fixed. As the algorithm’s name implies SimCo, the whole “dictionary update” process is done such that multiple atoms of dictionary $T$ are updated with corresponding coefficients at a time.

4.4. Dictionary Learning with Block Total Least-Squares Update

The dictionary algorithm with the BLOTLESS update was proposed by Yu et al.⁵³ The process of simultaneously updating the dictionary blocks and corresponding sparse coefficients is done in this algorithm. In this update, The bilinear nonconvex blocks update problem transfer into a linear least-square problem, which is then efficiently solved in the algorithm.⁵³

4.4.1. Least-square solver

The training samples are generated from $X=T_0{\alpha }_0$ in $X$, where ${\mathrm{T}}_0$ is a square matrix or satisfies the condition of $(M>=N)$ with ${\alpha }_0$. The whole update issue of the dictionary is represented as a bilinear inverse problem for the sparsity pattern denoted by $\mho$. The goal is to find T and $\alpha$ with constraint $\mathrm{X}=\mathrm{T}\alpha$ being nonconvex, as shown in Eq. (6),

$$X=T\alpha \text{with}{Y}_{{\mho }^c}.$$ (6)

The issue described in Eq. (6) can be expressed as a convex problem for an invertible unknown dictionary matrix T.

Assume $\mathrm{H}={\mathrm{T}}^{-1}$; then $\mathrm{HX}=\alpha$. So to find H and X, the goal is set as

$$HX=\alpha \text{and}{Y}_{\mho }^c(\alpha )=0,$$ (7)

or

$$X^{\varphi }-I_{n*n}[H^{\varphi }{\alpha }^{\varphi }]=0_{n*m}\text{and}{Y}_{\mho }^c(\alpha )=0.$$ (8)

In this manner, according to Eq. (8), the bilinear problem in Eq. (6) is transformed into an equivalent problem of linear least squares.

The dictionary update problem shown in Eq. (6) has unique solutions if all solutions are in the form of Eq. (9) as

$$T=T_{0}D\text{and}\alpha =D^{-1}{\alpha }_0.$$ (9)

4.4.2. Block total least squares

The dictionary update process is essential in dictionary learning. Various methods are applied for non-overcomplete and overcomplete dictionaries. In non-overcomplete, three potential methods are structure total least squares, parallel total least squares, and iterative total least squares (IterTLS).⁵³ If the dictionary is overcomplete, the only single solution that exists comes with the situation arising when $T_0$, $T_0^{*}{T}_0\ne I$ where $T_0^{*}$ is the inverse of an overcomplete dictionary $T_0$.⁵³ This is a complex situation, and the abovementioned methods for non-overcomplete cannot be applied directly. Thus, as a solution, the entire dictionary $T$ is sub-divided into sets called sub-dictionaries ($T=T_0,T_1,\dots T_n$). The updating process is applied from $T_0$ to $T_n$ sub-dictionaries one by one with all other dictionaries and their relative coefficients being fixed. This update process takes place for all sub-dictionaries. The sub-dictionaries are counted as individual blocks. These blocks are updated one by one, as seen in the name “BLOTLESS.” Anyone can use the same approach with a non-overcomplete dictionary update. Dictionary learning based on BLOTLESS converges much faster and needs at least 1/3 less training samples than other benchmark dictionary update methods.⁵³ In this paper, BLOTLESS-IterTLS is used in all experiments.

5. Medical Image Fusion by Sparse-Based Modified Fusion Framework

The SR-based general fusion framework²⁰ has been used for image fusion for many years. It works for all types of input image pairs, so the word “general” is used with its name. A modified version of the SR-based general fusion framework is proposed in this paper. This modification is done using the BLOTLESS update dictionary learning algorithm as per Fig. 1. The basic algorithmic flow of the BLOTLESS update dictionary learning algorithm is also shown in Fig. 2.

Fig. 1.

Modification in the SR-based fusion framework by BLOTLESS update.

Fig. 2.

BLOTLESS update dictionary learning algorithm.

The whole process of image fusion with this framework is divided into five stages as described in the following sections.

5.1. Lexicological Arrangement of Extracted Image Patches

MRI and PET images are the input images for the experiments. Image patches are extracted in an overlapping fashion with a one-pixel step size. If a single patch size is $n\times n$, the total image patches generated are $t=(P-n+1)\times (R-n+1)$ for input image size $P\times R$.²⁰ Then, they are all arranged in lexicological ordering for further processing.

5.2. Dictionary Learning by BLOTLESS Update

The fusion method based on SR is very much affected by the quality of dictionary learning. So compromising with dictionary learning degrades the SR-based fusion method’s quality and exact image recovery condition. For this, many novel and benchmark algorithms have been proposed, some of which are shown in section 4. The framework that is presented in this paper is modified in this stage, with the BLOTLESS update algorithm being used for dictionary learning. First, it divides the dictionary into sub-dictionaries. Then, an algorithm updates the sub-dictionary and its corresponding sparse coefficients using the total least-squares approach.⁵³

5.3. Matrix Generation with Sparse Coefficients

This stage generates sparse coefficient matrices for all input images. It is accomplished using a learned dictionary and a suitable approximation algorithm, such as MP, OMP, StOMP, focal underdetermined system solver (FOCUSS), and others.⁶⁵

5.4. Sparse Coefficient Fusion

The image fusion rule⁶⁶ should be applied to the sparse coefficients at this stage. Including a specific fusion rule is critical because other aspects and the efficiency have an impact on the entire fusion process efficiency. The max rule, average method rule, max-abs fusion rule, mean of coefficients fusion rule, weighted average rule, and others are examples of general fusion rules. In the last two decades, various sophisticated fusion rules have been used in addition to the generic fusion rules to improve image fusion outcomes. They are the max weighted multi-norm fusion rule, max-pooling fusion rule, etc.^67,68

5.5. Final Image Reconstruction

This stage is termed the last stage of this fusion framework. The fused coefficient matrix is inversely transformed to the lexicological ordering matrix followed by recovery or reconstruction of the final single fused image. This is done through the trained dictionary generated by the BLOTLESS update algorithm.

6. Experiments

In this section, various fusion experiments with findings are shown. The source image pairs of 20 MRI-T1 and PET images are taken from the GitHub resource.⁶⁹ The dataset of medical image pairs resizes to $256\times 256$ according to the experiment requirements. The second dataset for Glioma disease is taken from the Whole Brain Atlas.⁷⁰

6.1. Experiment Setup and Description

The image fusion results with this modified framework by the BLOTLESS update algorithm are compared with results of fusion from other benchmark dictionary learning algorithms.⁵¹ The quality parameters for each algorithm are identical for all experiments. These parameters should be followed for fair comparison in the fusion method based on SR in this paper. About 16 various natural images are used for dictionary learning, as shown in Fig. 4. All dictionaries are of dimension $64\times 128$ with a sparsity level of five. Images are grayscale and of size $256\times 256$. The dictionary learning process is run up to 10 iterations in two stages. In the first stage, dictionaries are learned with 16 natural images. In the second stage, dictionaries are learned with the²⁰ pairs of PET-MRI input images, as shown in Fig. 3.

Fig. 4.

The fusion results obtained by different methods from source image pairs 1 to 10.

Fig. 3.

Input natural images used for training data for dictionary learning for BLOTLESS.

Additionally, third stage experiments are conducted for glioma disease using the proposed framework’s fusion of the MRI-T2 and SPECT-T1 image pairs and MRI-PET image pairs. Additionally, the framework is also tested on data of 38 patients for detecting a brain tumor. For all stages, dictionary learning, and sparse coefficient matrix generation, image patches of size $8\times 8$ are used. The sliding window technique is used with the step size of one pixel to extract the image patches. An Intel(R) Core™ i5-2450 CPU@2.50 GHZ with 8 GB RAM in ideal condition is used for all experiments. The investigation is accomplished on MATLAB version R2018a.

The whole experiment is conducted in three stages. All three stages differ in training images and testing images.

• Stage 1:

  For dictionary training by all four DL algorithms, 16 natural images are used, shown in Fig. 3, and 20 pairs of PET-MRI image pairs are used for testing.

• Stage 2:

  For training and testing purposes, 20 pairs of PET-MRI image pairs are used.

• Stage 3:

  For testing the proposed framework for scan images from different modularities of disease diagnosis, the data scan images as MRI, SPECT, and PET are taken from Harvard Whole Brain Atlas⁷⁰ and Radiopaedia.org⁷¹ for glioma and 38 patients of brain tumors data are taken from GitHub⁷² having image pairs of MRI-PET data.

6.2. Performance Evaluation Metrics

In evaluating image fusion performance, performance assessment methods are critical. Four quality evaluation metrics are used in the assessment of the proposed modified framework for image fusion. For all four quality evaluation metrics, the better value shows the better performance. The ground truth or reference image is generally unavailable in medical image fusion, so evaluation is conducted with no reference image.

6.2.1. Entropy

Entropy (EN) computes the average number of bits required to quantize the intensities in the image. It is defined in Eq. (10) as:

$$E=-\text{sum}(p{*\log }_2(p)).$$ (10)

Here, $p$ contains the histogram counts from the image histogram.

6.2.2. Mutual information

Mutual information (MI) is the image fusion parameter based on the intensity distribution of fused image and source image.⁷³ The measurement of MI is based on the Kullback–Leibler⁷⁴ calculation of the two random variable dependencies, which is calculated as a relationship between images plotted in a 2D histogram as

$$I_{A,B}=\sum _{a,b}{P}_{A,B}(a,b)\log \frac{P_{A,B}(a,b)}{P_A(a)P_B(b)},$$ (11)

where ${\mathrm{P}}_A$ is the marginal probability distribution of the A variable. ${\mathrm{P}}_B$ is the marginal probability distribution of the B variable. ${\mathrm{P}}_{A,B}$ is the normalized joint distributions.⁷⁵

The value of ${\mathrm{I}}_{A,B}$ is maximum when the value of $A$ variable is equals to value of variable $B$.

Generally, image pixel values are shown as random variables, with the image’s histogram values shown as a probability distribution. The joint information typically indicates the quantity of transferred information to the fused image from the source images.

$$MI=I_{A,F}+I_{B,F}.$$ (12)

6.2.3. Sum of the correlations of differences

As per the basic terminology of fusion, the fused image contains collective information of all source images. So the sum of the correlations of differences (SCD)⁷⁵ performance metric uses both source and their impact on the final output fused image rather than comparing the source images with a fused image.⁷³ Thus, according to the SCD metric, the information transfer from the second source image ${\mathrm{S}}_2$ is termed $D(S_1)=I_F-S_2$ (between fused source ${\mathrm{I}}_F$ and first source image ${\mathrm{S}}_1$). This is valid for all other source images, shown as Eqs. (13) and (14). The values of $D(S_1)$ and $D(S_2)$ show the quantity of information transferred to the fused image from source images ${\mathrm{S}}_2$ and ${\mathrm{S}}_1$, respectively. Then, the correlation values for both differences are calculated by the c(.) function in Eq. (15) as

$$D(S_1)=I_F-S_2\text{and}D(S_2)=I_F-S_1,$$ (13)

$$\mathrm{SCD}=c(D_1,S_1)+c(D,S_2),$$ (14)

where the c(.) function calculates the correlation between $S1$ and $D(S_1)$ and between $S2$ and $D(S_2)$ as

$$c(D_k,S_k)=\frac{\sum _{\mathit{i}}\sum _{\mathit{j}}(D_k(i,j)-{\overline{D}}_k)(S_k(i,j)-{\overline{S}}_k)}{\sqrt{(\sum _{\mathit{i}}\sum _{\mathit{j}}(D_k(i,j)-{\overline{D}}_k{)}^2\left)\right(\sum _i\sum _j{(S_k(i,j)-{\overline{S}}_k)}^2)}},$$ (15)

where $K=\mathrm{1,2}$ and ${\overline{D}}_k$ and ${\overline{S}}_k$ are average of pixels values of $D_k$ and $S_k$, respectively.

6.2.4. MS-SSIM

MS-SSIM^76,77 is the MS extension of the SSIM parameter that provides more flexibility compared with single-scale SSIM. It calculates the structural distortions between any two images. The single-scale SSIM compares two images concerning contrast, luminance, and structure. But with the advantages, MS-SSIM calculates similarities at each scale for contrast and structure, whereas luminance similarities are calculated at coarsest scale.

$$\mathrm{MS}-\mathrm{SSIM}=\prod _{i=1}^M{({\mathrm{SSIM}}_i)}^{{\gamma }_i}.$$ (16)

7. Experimental Results and Discussion

7.1. Results Stage 1: BLOTLESS Update DL by 16 Natural Images

This section explores the results of twenty medical image pairs, as shown in Figs. 4 and 5. Columns 4 to 7 show output results as fused images for fusion from MOD, KSVD, SimCo, and BLOTLESS, respectively. The quantitative parametric average results for different algorithms on all of the 20 MRI-PET image pairs are shown in Table 1. The results demonstrate that image fusion using a modified framework performs much better for all image pairs. The best outcome parameter is highlighted in bold and accompanied by values in Table 1. When the BLOTLESS update algorithm updates the framework, the outcome in terms of EN parameter is highest. This means that the image fusion was done through the proposed revised framework having a higher degree of detailed information. The MI and SCD parameters represent the quantity of information transferred from source images to the final fused image. These parameters seem good when the BLOTLESS update algorithm is used for dictionary learning. The MS-SSIM quality parameter also indicates the performance of image fusion by the proposed modified framework. This shows that the structural similarity is best in fused images compared with source images when the BLOTLESS update algorithm is used for dictionary learning. All results show the higher effectiveness of a modified sparse-based general fusion framework with the BLOTLESS update dictionary learning algorithm than with other benchmark algorithms.

Fig. 5.

The fusion results obtained by different methods from source image pairs 11 to 20.

Table 1.

Average fusion results of 20 MRI-PET image pairs with 16 natural images used for DL.

Algorithm	EN	MI	SCD	MS-SSIM
MOD	4.653497	9.306994	0.547864	0.853018
K-SVD	4.629932	9.259863	0.534047	0.849764
SimCo	4.632706	9.265411	0.537765	0.850245
BLOTLESS	4.805529	9.611057	0.681007	0.853615

7.2. Results Stage 2: BLOTLESS Update DL by 20 Pairs of Input PET-MRI

This section investigates the results of twenty medical image pairs when the same 20 pairs of PET-MRI input images are used as training images for all four dictionary learning algorithms. When input images are used as training images, the qualitative results for a suggested updated framework with BLOTLESS update DL are significantly superior for all PET-MRI pairs, as shown in Figs. 6 and 7 for image pairs 8 and 19, respectively. Additionally, the quantitative result improves for almost all parameters, as shown in Table 2 and Figs. 8–11. The blue dotted line indicates the parametric results when 16 natural images are used as training images. The green dotted line indicates improved parametric results when input PET-MRI images are used as training images.

Fig. 6.

Fusion results for the eighth image pair after dictionary training by 20 input PET-MRI image pairs. (a) Input MRI image; (b) input PET image; (c) fused image using the MOD algorithm; (d) fused image using the KSVD algorithm; (e) fused image by the SimCo algorithm; and (f) fused image using the BLOTLESS update algorithm.

Fig. 7.

Fusion results for 19th image pair after dictionary training by 20 input PET-MRI image pairs. (a) Input MRI image; (b) input PET image; (c) fused image using the MOD algorithm; (d) fused image using the KSVD algorithm; (e) fused image by the SimCo algorithm; and (f) fused image using the BLOTLESS update algorithm.

Table 2.

Average fusion results of 20 MRI-PET image pairs with input images used for DL by input images.

Algorithm	Entropy	MI	SCD	MS-SSIM
MOD	4.839575	9.67915	0.530137	0.855467
K-SVD	4.833474	9.666947	0.525598	0.854797
SimCO	4.851943	9.703886	0.53188	0.856449
BLOTLESS	4.903487	9.806974	0.577226	0.864089

Fig. 8.

EN variations for both training datasets and all four DL algorithms.

Fig. 9.

MI variations for both training datasets and all four DL algorithms.

Fig. 10.

SCD variations for both training datasets and all four DL algorithms.

Fig. 11.

MS-SSIM variations for both training datasets and all four DL algorithms.

7.3. Results Stage 3: Image Fusion by the Proposed Framework on Medical Image Pairs for Brain Tumor Disease

Stage 3 experiments are done for the disease brain tumor and glioma. For glioma, data of four patients containing MRI-T2 and SPECT-T1 and MRI-PET image pairs are used. For the fifth patient, image pairs are taken from Radiopaedia.org.^71,78 Two types of image pairs are taken from this source: a CT and MRI axial TI C+ image pair and a CT and axial T2 image pair. Additionally, for brain tumors, data of 38 patients containing CT-MRI image pairs are used to test the proposed fusion framework. All of the results are manually checked by a regular medical practitioner, and the yellow squares are drawn to indicate glioma or any unknown abnormality detect in fused images.

7.3.1. Details of experiments done on five glioma patients

Patient 1: For patient 1, the six pairs of MRI-T2 and SPECT-T1⁷⁰ are used. The results with the framework modified by the BLOTLESS update dictionary learning algorithm are shown in Fig. 12. The fusion results indicated that, for the glioma disease, the final image contains the information of both scans with tumor or abnormities shown as yellow squares.

Fig. 12.

Fusion results for the proposed framework in glioma disease for patient 1. (a)–(f) Input MRI-T2 images (six slices); (g)–(l) input SPECT-T1 images (six slices); and (m)–(r) fused images with yellow square indicating abnormality or tumor.

Patient 2: Three slice pairs of MRI-T2 and SPECT-T1 are taken for another patient. Fusion is performed on the input three pairs, and the results of fusion are shown in Fig. 13. The results indicated that, for the glioma disease, the final fused image shows scan information of both images with an indication of glioma tumor in the yellow square.

Fig. 13.

Fusion results for the proposed framework in glioma disease for patient 2. (a)–(c) Input MRI-T2 images (three slices); (d)–(f) input SPECT-T2 images (three slices); and (g)–(i) fused images with yellow square indicating glioma tumor.

Patients 3 and 4: For patient 3, the pairs of MRI and PET images from the Whole Brain Atlas are taken for glioma disease. The fusion is performed on input pairs, and the result is shown below in Figs. 14 and 15. As per the Figs. 14 and 15 results columns, the fused images contained input image information of both images with glioma tumor shown in the yellow square.

Fig. 14.

Fusion results for the proposed framework in glioma disease for patient 3. (a) Input MRI image; (b) input PET image; and (c) fused image with yellow square indicating glioma tumor.

Fig. 15.

Fusion results for the proposed framework in glioma disease for patient 4. (a) and (b) Input MRI-T2 images (two slices); (c) and (d) input PET image (two slices); and (e) and (f) fused images with yellow square indicating glioma tumor.

Patient 5: Patient 5 datasets are taken from the public dataset of radiographical images Radiopaedia.org. For this patient, two image scan pairs are taken for glioma disease. As a pair one , CT and MRI axial T1 C+ images are taken while as a pair two CT and MRI axial T2 images are taken shown in Fig. 16. The fusion is done with the proposed fusion framework with a BLOTLESS update dictionary learning algorithm. The results are shown in Fig. 16. As per the results, the fused image shows scan information of both images.

Fig. 16.

Fusion results for the proposed framework in glioma disease for patient 5. (a) and (b) Input CT image; (c) input MRI axial TI C+ image; (d) MRI axial T2 image; and (e) and (f) fused images.

7.3.2. Details of experiments done on 38 brain tumor patients

The proposed fusion framework is used to fuse the MRI-PET dataset of brain tumor patients, and the results state that the final fused images contain the information of both input medical images as shown in Fig. 19.

Fig. 19.

Eight random fusion results from 38 patients CT + MRI image pairs using the proposed framework.

All output images of experiment stage 3 are validated by several image experts, assistant professors, professional doctors, radiologists, and technologists from various medical colleges and image scan labs. This validation is processed in the form of questionnaires^79,80 on output images with two parameters (i) the quality of the output fused images and (ii) the amount of information transferred from the source images to fused images for disease detection. The opinions are taken as a rating from five to one with five being excellent results and one being poor results. The results of this process are shown in Figs. 17 and 18. Here, output set number means the patient number in stage 3 experiments.

Fig. 17.

Ratings of subjective evaluations on experiment stage 3 outputs for parameter quality of the output fused image.

Fig. 18.

Ratings of subjective evaluations on experiment stage 3 outputs for parameter “amount of information transfer from source images to fused images for disease detection.

Segmentation and classification of tumor are the main processes for glioma tumors detection^80,81 on individual image scans. For these processes, image specialists can use different techniques of image processings, such as edge detection filters, machine-learning (ML) models, and deep-learning (DL) techniques.^82,83 However, via medical image fusion, fused images will be more informative than the separate input scans, making them more suitable for tumor segmentation and classification challenges. The high-quality fused images will deliver better automated detection results than single-input scan medical images. On the other hand, if the detection of a tumor is done manually by doctors or radiologists, image fusion techniques will deliver a more informative single image rather than an individual one.

On the other hand, if we compared the proposed method with the state-of-art image fusion techniques, the basic spatial domain and frequency fusion methods are less complex but all have the problems of artifacts, blurring, long running time, enhancing the signal-to-noise ratio, and color distortion.^1,2,18 The latest techniques based on ML models and DL have better performances. In these methods, the workload is huge and the cost is expensive, which tends to address the lack of data with over-fitting. Thus, these methods need a rich dataset of training images. DL training is time-consuming with a complicated framework and necessitates a high level of computer system configuration.^79,84,85

Thus, sparse-based medical image fusion techniques¹⁹ have a better performance with low-cost resource requirements and a better running time. According to their nature, they also require low memory consumption for storage.⁸⁶

However, the approaches based on sparse-based fusion experienced issues with synchronization of sparsity level to loss of relevant information in the medical image recovery process and the problem of quality of the learned dictionary.^87–89

8. Conclusions and Future Work

In this paper, a modified sparsity-based image fusion framework was proposed. First, 16 natural grayscale images were used to train the dictionary of dimension $64\times 128$ with a five per atom sparsity level. This process was done for all four dictionary learning algorithms. An individual medical image was divided into $8\times 8$ overlapping patches arranged in lexicological order for the fusion process. Then, a sparse coefficients matrix was generated for fusion, followed by a fusion of coefficient matrices with a specific fusion rule. The reverse process generated final fused images for all 20 pairs of MRI-T1 and PET images. Experimental results on all image pairs demonstrated that the medical image fusion using the proposed sparse-based modified image fusion framework by the BLOTLESS update dictionary learning algorithm performed best compared with fusion using trained dictionaries by other benchmark algorithms. As per the quantitative results, the proposed framework with BLOTLESS update dictionary learning algorithm performed better than the framework with other dictionary learning algorithms.

Additionally, most of the parameters improved when the input images were used for dictionary learning. Also, improvements in qualitative results were seen in the proposed modified framework. In the third stage, the applicability of the proposed framework was tested for medical images in the case of brain tumor and glioma disease. Output images of stage 3 were also validated by multiple experts and professional doctors for glioma disease. Future research is required for comparing the proposed method with other image fusion standard algorithms as quantitative measures. Additionally, the effectiveness of the algorithm and proposed fusion framework may be compared with other medical image fusion methods, and for glioma detection, the outputs of experiments will be further segmented and classified.

Biographies

Lalit Kumar Saini is a research scholar at Manipal University Jaipur, Jaipur, India. He received his BE and MTech degrees in computer engineering from the University of Rajasthan and RTU, Kota, respectively. His current research interests include digital image fusion, SR, dictionary learning, and compressed sensing algorithms.

Pratistha Mathur is a professor in the Department of Information Technology at Manipal University Jaipur. She received her MTech degee and PhD in computer science from Banasthali Vidyapith in 1998 and 2012, respectively. Her research interests include digital image processing, soft computing, and ML. She has published more than 40 papers in international and national journals and conferences.

Disclosures

The authors have no relevant financial interests in the paper and no other potential conflicts of interest to disclose.