Tensor Ring Based Image Enhancement

Abstract

Background:

Image enhancement, including image de-noising, super-resolution, registration, reconstruction, in-painting, and so on, is an important issue in different research areas. Different methods which have been exploited for image analysis were mostly based on matrix or low order analysis. However, recent researches show the superior power of tensor-based methods for image enhancement.

Method:

In this article, a new method for image super-resolution using Tensor Ring decomposition has been proposed. The proposed image super-resolution technique has been derived for the super-resolution of low resolution and noisy images. The new approach is based on a modification and extension of previous tensor-based approaches used for super-resolution of datasets. In this method, a weighted combination of the original and the resulting image of the previous stage has been computed and used to provide a new input to the algorithm.

Result:

This enables the method to do the super-resolution and de-noising simultaneously.

Conclusion:

Simulation results show the effectiveness of the proposed approach, especially in highly noisy situations.

Introduction

Image enhancement is an important issue in different image processing areas.[1,2,3] Images achieved from different devices, usually do not have the desired quality and enhancement is an essential step for better exploiting of them. This becomes more important when working with biomedical images where accurate diagnosis should be done using images.[4,5,6,7]

Super-resolution of low resolution images for deriving high resolution ones is also an important issue in image processing area. This has been studied for different images such as natural, hyperspectral, or biomedical images. Among these, super-resolution of biomedical images achieves high attention due to its especial application and has been studied in different papers such as.[8,9,10,11,12] Deep learning methods have been widely used for image super-resolution.[12,13,14,15,16] However, these methods usually need large volume of training data for achieving desired performance which is not always available. Statistical modeling has also been exploited for image super-resolution.[17,18] Recently, tensor-based approaches have been also proposed for image super-resolution, including biomedical, hyperspectral, and natural images and show their high performance.[8,19,20,21,22,23,24]

In general, tensors are higher order arrays used for representing higher order datasets, such as RGB images, hyperspectral datasets, biomedical datasets, and so on.[25,26] Data recorded in a tensor can be analyzed in different ways. A common approach is to decompose a tensor into smaller matrices or lower order core-tensors, called tensor decomposition.[25,26] There are several tensor decomposition approaches. The well-known ones are CANDECOMP/PARAFAC (CP)[27,28,29,30] and Tucker decompositions,[31,32,33] an illustration for each has been shown in Figure 1.

Figure 1.

CANDECOMP/PARAFAC (first row) and Tucker (second row) decompositions of a third order tensor

CP and Tucker decompositions have been widely used for different applications, however, algorithms of CP decompositions are usually unstable and Tucker decomposition suffers from curse of dimensionality.[25] For overcoming these problems, other tensor decompositions, called tensor networks, have been proposed.[25] In tensor networks, the number of elements resulting from tensor decomposition increases linearly with the tensor order.[25] This property is highly important when working with higher order datasets.

Tensor Train (TT) and Tensor Ring (TR) decompositions are two members of tensor networks. In TT, an I₁ × I₂ × … × I_N tensor decomposes into a series of third order core tensors interconnected to each other. The core tensors are of size , where R_n is the n-th TT rank and R₀ = R_N = 1, so the first and the last core tensors are two matrices.[34] TR decomposition can be considered as a generalized version of TT decomposition in which an I₁ × I₂ × … × I_N tensor decomposes into a series of third order core tensors interconnected in a loop. The n-th core tensor, i.e., G⁽ⁿ⁾, is of size R_n-1 × I_n × R_n where R_n is the n-th TR rank, but with R₀ = R_n > 1.[35] TT and TR decompositions are denoted as

where X is an N-th order tensor to be decomposed. An example for TR (TT) decomposition has been illustrated in Figure 2.

Figure 2.

Tensor ring decomposition of a 4-th order tensor. For tensor train decomposition, R₀ = R₄ = 1

TT and TR decompositions are more effective for analyzing higher order datasets. Due to this reason and also for better exploiting the correlations of image pixels, the original low order image is usually reformatted into higher order tensors using different methods. A common approach is Hankelization in which a raw dataset reformatted into a matrix with Hankel structure. Different Hnakelization methods are Multi-way delay-embedding transform (MDT),[24] patch Hankelization[36] and overlapped patch Hankelization.[37] Considering this, several TT or TR based methods Hankelize the input datasets before applying TT or TR decompositions.[36,37] By applying Hankelization with overlapped patches, an I₁ × I₂ image transferred into a 6^th order tensor of size P × P × T₁ × D₁ × T₂ × D₂, where P is the patch size, T_i’s are window sizes, with and O is the overlap between patches.[37]

Determining proper ranks (i.e., R_i’s) is an important issue when working with TT or TR decompositions. The ranks indeed determine the size of the core tensors and can highly affect the performance of the decomposition. Several papers using predetermined fixed ranks for TT\TR decomposition. Some of other papers use incremental approaches in which the ranks are increased during iterations. However, selecting very low ranks, results in losing small details while too much increasing the ranks results in a degraded output, especially in noisy cases. Therefore, when the original image is noisy, the TR ranks usually limited to a maximum value to prevent appearing the noise in the final image.

In this paper, we have proposed a new method for super-resolution of noisy images which allows the ranks to be increased to higher values without highly degrading the final image. To allow the ranks to be increased more, in this new method for the available noisy pixels, a weighted combination of the original input and the output of the previous stage have been used as a new input for the next stage, while the ranks are also increased. The approach can be considered as a generalized version of[38] which has been previously proposed for super-resolution of noisy images. This results in a more accurate super-resolution method comparing to the other existing methods when the input is noisy (details will be discussed in Section III).

The remaining of this paper has been organized as: Notations and preliminaries are reviewed in Section II. The proposed super-resolution algorithm has been proposed in Section III. Simulation results have been presented in Section IV and finally Section V concludes the paper.

Notations and Preliminaries

Notations in this paper are basically the same as.[25] Tensors and matrices are denoted by underlined bold capital (X) and bold capital letters (X), respectively. Unfolding of a tensor of size I₁ × I₂ × … × I_N into a matrix of size I_n × I₁ I₂ × … I_n-1 I_n+1…I_N is called mod-n matricization and is denoted by X_(n). Mod-{N} canonical unfolding of an I₁ × I₂ × … × I_N tensor, denoted by X_[n], results in an I₁ I₂…I_n × I_n+1 … I_N matrix. Hadamard or elements wise product of two tensors or two matrices of the same size has been denoted by ⊛. Frobenius norm and trace of a matrix are denoted by ‖.‖_F and tr(.), respectively.

Proposed Super-resolution Approach

In this section, a new approach for super-resolution of noisy low resolution images using TR decomposition has been proposed.

Classic TR (Tucker)-based completion algorithms (super-resolution can be considered as a special case of tensor completion) can be summarized as follows:[24,38]

1. (Patch) Hankelize the incomplete low resolution input image which results in X

2. Compute the TR (Tucker) decomposition of input X with rank vector r = [R₀, R₁,…, R_N] which results in X̂

3. Update the input as

4. Increase the rank vector as r = r + inc, where inc can be any integer value

5. Repeat the procedure until desired accuracy or the maximum rank value (R_max) achieved

6. De-Hankel the output.

Note that Ω is a binary mask tensor with the same size as X whose entries are 1 for the observed and 0 for the missing elements of the input incomplete image.

In many cases, the input low resolution image is noisy. This can affect the quality of the final result, especially in highly noisy cases. Pre or post de-noising can increase the simulation time and also results in removing small details. In this paper we have proposed to modify the previous de-noising and super-resolution methods and derive a new algorithm for super-resolution of noisy images as

1. Patch Hankelize the incomplete low resolution noisy image with overlapped patches which results in a 6-th order tensor X

2. Compute the TR decomposition of input X with rank vector r which results in X

3. Update the input as

4. Increase the rank vector as r = r + inc

5. Repeat the procedure until desired accuracy or the maximum rank value (R_max) achieved

6. De-Hankel the output.

In the above procedure, α is a small nonnegative constant less or equal to 1. It is clear that putting α = 1 changes the update rule to the previous format.

The proposed algorithm allows the TR ranks to be increased to higher values without decreasing the quality of final result (see the simulations). This is a very important point, since in classic approaches, when the input image is noisy, the ranks (TR or Tucker ranks) cannot be highly increased. This is due to the fact that increasing the ranks allows the noise to be appeared in the final image. For preventing this issue, the maximum rank should be limited which from other side results in losing small details in the output. However, using the proposed approach enables us to increase the ranks to larger values without highly decreasing the output quality.

The value of α can affect the quality of output, since selecting smaller α allows the ranks to be increased to higher values but can also increase the simulation time. Larger values of α can result in a noisy output.

The performance of the algorithm will be evaluated in the next section.

Simulation Results

In this section, the validity of the proposed approach has been investigated. For testing the algorithm, fundus fluorescein angiogram photographs of diabetic patients (https://misp.mui.ac.ir/en/fundus-fluorescein-angiogram-photographs-diabetic-patients-0)[39] has been used. The dataset contains 70 images of size 576 × 720 with 30 normal and 40 abnormal cases.

For evaluating the quality of the proposed super-resolution algorithm, the proposed approach has been compared with several approaches such as MDT,[24] High Accuracy Low Rank Tensor Completion (HaLRTC),[40] Fast Super-Resolution Reconstruction Algorithm (FSRRA),[41] Tensor Train Weighted Optimization (TT-WOPT)[42] and Depth Super-Resolution (DSR).[43] For tensor-based approaches, i.e., HALRTC and TT-WOPT, the inputs were first patch Hankelized with overlapped patches similar to the proposed algorithm and then given to the algorithms (except for MDT which has its own Hankelization technique). Recall that super-resolution can be considered as a special case of tensor completion problem. For the proposed approach, patch size has been set to P = 2 with overlap O = 1 and α = 0.5. For MDT, the window size has been set to [2,2]. Low resolution noisy images have been derived by down-sampling the original images to the size 144 × 180 and adding Gaussian noise with variance σ. Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) of each image have also been reported. The results of super-resolution of the inputs with rate 2 and for different noise variances are illustrated in Tables 1-3. PSNR and SSIM results show that the proposed algorithm has the ability to preserve the details as much as possible, while decreasing the noise level of the final image.

Table 1.

Results of the algorithms for the super-resolution of low resolution noisy image with σ=0.02

Original image	Low resolution - noisy image	HaLRTC[40]	FSRRA[41]
		(16.5307, 0.1519)	(31.2794, 0.7767)
TT-WOPT[42]	DSR[43]	MDT[24]	Proposed
(29.0375, 0.7055)	(31.3445, 0.7280)	(31.1476, 0.7830)	(31.5719, 0.8305)

The super-resolution rate is 2

Table 3.

Results of the algorithms for super-resolution of low resolution noisy image with σ=0.06

Original image	Low resolution- noisy image	HaLRTC[40]	FSRRA[41]
		(16.2845, 0.1366)	(25.6078, 0.4010)
TT-WOPT[42]	DSR[43]	MDT[24]	Proposed
(25.7094, 0.4358)	(23.8312, 0.3023)	(25.9578, 0.4181)	(29.0146, 0.6905)

The super-resolution rate is 2

Table 2.

Results of the algorithms for the super-resolution of low resolution noisy image with σ=0.04

Original image	Low resolution - noisy image	HaLRTC[40]	FSRRA[41]
		(16.4356, 0.1463)	(28.3062, 0.5697)
TT-WOPT[42]	DSR[43]	MDT[24]	Proposed
(27.4350, 0.5691)	(27.0173, 0.4706)	(29.1530, 0.6522)	(30.2326, 0.7546)

The super-resolution rate is 2

The six algorithms have been applied for the super-resolution of 20 first cases of the dataset for σ = 0.06 and the averaged PSNR and SSIM for each algorithm are reported in Table 4. As the results show, the proposed approach has higher performance in comparison to the other algorithms. The computed P values were less that 0.05 (P < 0.05) which show there is a statistically significant difference between the proposed approach and other methods.

Table 4.

Averaged PSNRs and SSIMs of the algorithms for the super-resolution of 20 first cases of the dataset with σ=0.06 and rate 2

	HaLRTC[40]	FSRRA[41]	TT-WOPT[42]	DSR[43]	MDT[24]	Proposed
PSNR	16.3586±1.0410	25.1877±0.4279	25.1763±1.1106	23.6635±0.1801	25.5649±2.0908	27.7732±1.3619
SSIM	0.1643±0.0132	0.4376±0.0239	0.4181±0.0217	0.3624±0.0387	0.4733±0.0746	0.6175±0.044

The P value between the proposed approach and other methods is <0.05 (P<0.05) which shows a statistically significant difference between the proposed approach and other methods

For better understanding the effect of α on the performance of the algorithm, the results of previous simulations (with α = 0.5) have been compared with the situation when α = 1. By putting α = 1, the super-resolution procedure changes to the approach of.[37] The comparison has been done for two levels of noise variance and the results are shown in Table 5. For the second and fourth columns, the ranks and the results with the highest PSNR’s have been reported. The proposed approach has been compared with the situation when α = 1 and the maximum rank set to the value which has the best result for α = 1 and also compared with the situation when α = 1 and the maximum rank set the same as the rank for α = 0.5. The results show that the proposed algorithm achieved its best performance in higher ranks in comparison to the situation when α = 1. In addition, putting α = 0.5 has the ability of improving the performance of the output. This higher performance is more clear for higher level of noise. It can also be inferred that increasing the ranks to higher values for α = 1, results in decreasing the performance of the algorithm and the output image becomes noisy, while the proposed approach is more robust to the rank incremental.

Table 5.

Comparison of the proposed algorithm with α=0.5 and α=1

Low resolution noisy image with σ=0.04	Proposed approach with α=1 (Rmax=7)	Proposed approach with α=1 (Rmax=10)	Proposed approach with α=0.5 (Rmax=10)
	(30.1066, 0.7390)	(29.9515, 0.7110)	(30.2326, 0.7546)
Low resolution noisy image with σ=0.06	Proposed approach with α=1 (Rmax=5)	Proposed approach with α=1 (Rmax=8)	Proposed approach with α=0.5 (Rmax=8)
	(28.7621, 0.6789)	(28.3049, 0.6081)	(29.0146, 0.6905)

The PSNRs and SSIMs of the algorithm during rank incremental for α = 1 and α = 0.5 and σ = 0.06 are illustrated in Figures 3 and 4. As it is expected, for both cases, the PSNR starts to increase until some rank values and then decreases for higher ranks. This is due to the presence of noise which decreases the performance of the output for higher ranks. However, the maximum PSNR for α = 1 achieves in smaller ranks in comparison to α = 0.5. In addition the maximum value of PSNR for α = 1 is less than the maximum value of PSNR for α = 0.5. This shows the effectiveness of the proposed algorithm which allows the ranks to be increased to higher values without decreasing the performance of the output. The resulting PSNRs and SSIMs of the algorithm for the super-resolution of the noisy image with σ = 0.06 and for more different values of α have been shown in Table 6. The results show that, by decreasing α to some values, the performance of the algorithm increases. However, more reduction in α can result in a high increase in the computational burden without so much increasing the performance. This show that, depending on the situation, α cannot be selected too small or too large.

Figure 3.

PSNRs of the outputs during rank incremental for a low resolution noisy input with σ = 0.06 and for α = 1 and α = 0.5

Figure 4.

SSIMs of the outputs during rank incremental for a low resolution noisy input with σ = 0.06 and for α = 1 and α = 0.5

Table 6.

PSNRs and SSIMs of the super-resolution of the noisy image with σ=0.06 for different values of α

α=1 (Rmax=5)	α=0.75 (Rmax=6)	α=0.5 (Rmax=8)	α=0.25 (Rmax=13)
(28.7621, 0.6789)	(28.9470, 0.6875)	(29.0146, 0.6905)	(28.8881, 0.6880)

For a more accurate comparison among the algorithms, a vessel segmentation algorithm (https://github.com/farkoo/Retinal-Vessel-Segmentation/tree/master) has been applied on the output of each algorithm. For the initial low resolution noisy image, σ was set equal to 0.06 and the super-resolution rate was 2. The results are presented in Table 7 in which the images resulted from each algorithm in addition to its corresponding extracted vessels have been shown. As the results show, retina vessels can be extracted with high performance from the image resulted from the proposed approach. While the vessels extracted from other images do not provide sufficient quality.

Table 7.

Applying vessel segmentation algorithm to the resulting output of each algorithm

Original image		Low resolution noisy image
HaLRTC[40]		FSRRA[41]
TT-WOPT[42]		DSR[43]
MDT[24]		Proposed

Super-resolution rate is 2

Finally, the algorithm has been tested for super-resolution of images with rate 3. The results in addition to PSNRs and SSIMs are presented in Table 8. For the proposed approach, α has been set equal to 0.5 and patch size was 2 with overlap 1. For the MDT, the window size set to [4,4]. As the results show, the proposed approach is also effective for the super-resolution with higher rates.

Table 8.

Super-resolution of low resolution noisy images with rate 3 and σ=0.06

Original image	Low resolution noisy image	DSR[43]	MDT[24]	Proposed
		(22.6809, 0.3524)	(21.7222, 0.2877)	(24.8617, 0.4533)
		(23.1762, 0.3592)	(20.1438, 0.2988)	(26.0524, 0.5760)

Conclusion

In this article, a new approach for super-resolution of low resolution noisy images has been proposed. The proposed super-resolution approach has been derived by proposing a new update rule for the input image in each iteration by using a binary mask tensor and a combination weight. This allows the TR ranks to be increased to higher values without decreasing the quality of the final image. Simulation results and comparison with the existing algorithms confirmed the performance of the proposed approach.

Financial support and sponsorship

This work has been supported by Isfahan University of Medical Sciences (Grant number: 2400134).

Conflicts of interest

There are no conflicts of interest.