Multichannel Saliency Detection Based on Visual Bionics

Abstract

Inspired by the visual properties of the human eyes, the depth information of visual attention is integrated into the saliency detection to effectively solve problems such as low accuracy and poor stability under similar or complex background interference. Firstly, the improved SLIC algorithm was used to segment and cluster the RGBD image. Secondly, the depth saliency of the image region was obtained according to the anisotropic center-surround difference method. Then, the global feature saliency of RGB image was calculated according to the colour perception rule of human vision. The obtained multichannel saliency maps were weighted and fused based on information entropy to highlighting the target area and get the final detection results. The proposed method works within a complexity of O(N), and the experimental results show that our algorithm based on visual bionics effectively suppress the interference of similar or complex background and has high accuracy and stability.

1. Introduction

Saliency detection is an important research content in computer vision, which refers to the process of simulating human visual attention mechanism to accurately and quickly detect the most interesting regions in images. Borji et al. defined that saliency visually described the prominent target or area in the scene relative to its neighbouring area [1]. The human visual attention mechanism prioritizes a few significant areas or objectives, while ignoring or discarding others that are not, which can allocate computing resources selectively and greatly improve the efficiency of visual information processing. Therefore, the saliency computing model based on visual attention mechanism has been widely studied. When processing the input image or video, the computer can judge the importance of its visual information by detecting the saliency area. It has been widely applied in object detection and identification [2], image retrieval [3], video quality assessment [4], video compression [5], image cropping [6], and other fields.

The RGB image saliency detection model based on visual attention mechanism uses low-level feature contrast to calculate saliency [7, 8]. Typical of them are global feature comparison calculation model [9], local feature comparison calculation model [10], and combination of global and local feature comparison model [11]. In order to improve the accuracy of detection, the saliency detection model was proposed based on prior knowledge [12]. Typical of them are position prior [13], background prior [14, 15], colour prior [16], shape prior [17], and boundary prior [18, 19].

However, most 2D image saliency detection models based on human visual attention mechanism ignore the fact that human visual attention mechanism is based on 3D scene. It shows that depth provides extra important information of saliency detection for RGB image. Desingh et al. discussed 3D saliency detection methods based on depth appearance, depth-induced blur, and centre-bias [20]. Niu et al. conducted depth saliency detection based on parallax contrast and professional knowledge in vertical photography [21]. Further, Ju et al. proposed a depth saliency detection model based on depth image anisotropic center-surround difference [22]. Ren et al. [23], respectively, proposed the saliency detection of RGB-D images against a complex background by combining the prior knowledge of depth, indicating the validity of depth information in 3D saliency detection. However, there are two challenges in the process of saliency detection of RGB-D images. The first is how to calculate the saliency of depth images under similar or complex background interference, and the second is how to combine the saliency map of depth image and RGB image to obtain the final result with a good performance. In this paper, we proposed a multichannel saliency detection method based on RGBD images, which has the following contributions:

1. On the basis of SLIC algorithm, colour, texture, and depth information are used to measure the distance of superpixel segmentation

2. Based on the perception rule of human vision, we introduced the depth information and global information of RGB image as two feature channels for saliency computing

3. The weighted features of depth saliency and colour saliency were fused by information entropy, and experiment shows that the algorithm has a good performance in case of background interference

2. Saliency Detection

The algorithm framework of this paper is shown in Figure 1. Combining the depth map with the RGB map to carry out image preprocessing and colour, texture, and depth information are introduced as the basis of superpixel segmentation. Then, the colour and depth information were calculated as two feature channels of saliency map. As is shown in Figure 2, the depth saliency was obtained by the anisotropic center-surround difference (ACSD) method, and the global saliency of RGB image was calculated by global contrast method based on HSV space. Finally, information entropy is used to calculate the weights of two channels, respectively, and get the final fused saliency map.

Figure 1.

Our framework of saliency detection.

Figure 2.

Example of the ACSD operator in a region.

3. Image Preprocessing

The human visual observation system takes the image region as the basic unit, and the saliency detection based on the region conforms to the visual characteristics of the human eyes. As a construction method of pixel region, superpixel technology has been widely used in computer vision field. Superpixel can quickly segment the image into subregions with certain semantics, which is conducive to the extraction of local features and the expression of structural information [24]. SLIC algorithm has obtained a good balance in the two aspects of edge fitting degree and compactness, which has an excellent comprehensive performance. When the SLIC is used to segment the left image, the obtained boundary is not accurate because of ignoring the mutual constraint relationship between the 2D and depth information. Therefore, colour, texture, and depth information are used to measure the distance of superpixel segmentation in this paper.

Converting the left image to the CIE Lab colour space and dividing the image into k superpixels. Here, each pixel has a unique identifier  i. Extract the follow 7 d characteristics of each superpixel region as measurement property. It can be expressed:

$$\begin{array}{c}{S}_{p¯i}=\left\{l_i,a_i,b_i,C_{\text{con}i},C_{\text{cor}i},E_i,d_i\right\}i=1,2,3,\cdots ,k,\end{array}$$ (1)

where l_i, a_i, and b_i are the mean value of L, a, and b colour components of each superpixel region; C_coni, C_cori, and E_i  are the mean value of contrast, cross-correlation, and energy mean of gray level cooccurrence matrix of each superpixel region; and d_i is the depth value of each superpixel region. Then, we can describe the adjacent superpixel pair as S_p¯ij:

$$\begin{array}{c}{S}_{p¯ij}=\left\{\left(S_{p¯i},S_{p¯j}\right)\right\}i\in \left[1,k\right],j\in \left[1,k\right],i\ne j,\end{array}$$ (2)

where S_p¯ij superpixel pair with i and j as identifier, k is the number of superpixels of the image, and S_P¯i and S_p¯j are the 7 d characteristics of the adjacent superpixel pair. The number of adjacent superpixel pairs in each image is determined by SLIC superpixel segmentation.

Using colour, texture, and depth features to calculate the difference between all adjacent superpixel pairs S_p¯ij.  d_lab, d_glcm, and d_depth are defined to describe the measurement of colour, texture and depth characters:

$$\begin{array}{c}{d}_{lab}=\sqrt{{\left(l_i-l_j\right)}^2+{\left(a_i-a_j\right)}^2+{\left(b_i-b_j\right)}^2,}\\ d_{glcm}=\sqrt{{\left(C_{\text{con}i}-C_{\text{con}j}\right)}^2+{\left(C_{\text{cor}i}-C_{\text{cor}j}\right)}^2+{\left(E_i-E_j\right)}^2,}\\ d_{\text{depth}}=\sqrt{{\left(d_i-d_j\right)}^2.}\end{array}$$ (3)

Then, the distance measurement of superpixel segmentation D_ij is

$$\begin{array}{c}{D}_{ij}={\omega }_1\bullet \sqrt{\frac{\epsilon +d_{lab}^2}{3}}+{\omega }_2\bullet \sqrt{\frac{\epsilon +d_{glcm}^2}{3}}+{\omega }_3\bullet \sqrt{\epsilon +d_{\text{depth}}^2},\end{array}$$ (4)

where ε = 10⁻⁴. It is used to ensure the validity of the value. ω₁, ω₂, are ω₃ are the weight of colour, texture, and depth.

In the image, the greater the discreteness of a feature data set is, it means that the more influence this feature has on the image. Mean variance can effectively represent the degree of difference between data. Therefore, the global mean variance of colour, texture, and depth is used as the weight values of the three features ω₁, ω₂, and ω₃.

If the difference between adjacent superpixels is less than a certain threshold th₁, the adjacent superpixel pair will be merged.

$$\begin{array}{c}{\text{th}}_1=\sqrt{\frac{{\omega }_1\bullet \left(\overline{l}+\overline{a}+\overline{b}\right)}{3}}+\sqrt{\frac{{\omega }_2\bullet \left(\overline{C_{\text{con}}}+\overline{C_{\text{cor}}}+\overline{E}\right)}{3}}+\sqrt{{\omega }_3\bullet \overline{d}},\end{array}$$ (5)

where $\overline{l},\overline{a},$ and $\overline{b}$ are the mean value of L, a, and b colour components of the image; $\overline{C_{\text{con}}},\overline{C_{\text{cor}}},$ and $\overline{E}$ are the mean value of contrast, cross-correlation, and energy mean of gray level cooccurrence matrix of the image; and $\overline{d}$ is the depth value of the image.

Finding all similar adjacent superpixel pairs and taking the upper left superpixel of the image as the starting point of clustering. The output after clustering contains n regions R_i, 1 ≤ i ≤ n.

4. Depth Saliency Map

For each superpixel, the anisotropic center-surround difference (ACSD) value is calculated, and the value of center superpixel is assigned to each pixel within the region R_i. Performing an anisotropic scan along multiple directions, in each scanline, assuming the pixel with the minimum depth value as background and calculate the difference between the center pixel and background. L is the maximum scan length for each scanline. The typical value of L is a third of the diagonal length.

The anisotropic center-surround difference (ACSD) is summed over eight scanning directions 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. The mathematical description of anisotropic center-surround difference (ACSD) measure is

$$\begin{array}{c}{S}_d^m\left(x,y\right)=d\left(x,y\right)-\min \left(d^m\right),n\in \left[1,L\right],\\ S_d\left(x,y\right)=\sum _{m=1}^8{S}_d^m,\end{array}$$ (6)

where S_depth^m indicates the ACSD value of pixel (x, y) along the scanline m. d(x, y) is the depth value of pixel (x, y). n is the index of the pixels along the scan path m. min(d^m) is the minimum depth value along the scanline m. S_d(x, y) is the ACSD value of pixel (x, y) which sums the center-surround difference values in eight directions.

5. Global Saliency Map of RGB Image

Colour histogram is used to regularize the colour of the image to level 128 in order to reduce the computational complexity and save the storage space. On the other hand, the descending dimension algorithm for HSV colour space is proposed. With the decrease of saturation, any colour in HSV space can be described by the change of gray level. The intensity value determines the specific gray level of the conversion [25]. When the colour saturation is close to zero, all pixels look similar regardless of hue. With the increase of saturation, the pixels are distinguished by hue value.

Compared with colour saturation, human vision is more sensitive to hue and intensity. The pixels with lower colour saturation can be approximately represented by intensity level, while the pixels with higher colour saturation can be approximately represented by hue. Saturation value is used to determine whether each pixel can be represented by hue or intensity value, which is more consistent with the law of human visual perception. Saturation threshold th₂ is

$$\begin{array}{c}{\text{th}}_2=1-\frac{0.8I_v\left(x,y\right)}{255},\end{array}$$ (7)

where I_v(x, y) represents the V component value of a pixel. When the saturation value I_s(x, y) is greater than th₂, the pixel point is represented by the hue value I_v(x, y); when the saturation value is less than th₂, the pixel point is represented by the intensity value I_h(x, y). The saliency of each pixel is

$$\begin{array}{c}{S}_c\left(x,y\right)=\left\{\begin{array}{c}\left|\overline{I}-I_v\left(x,y\right)\right|I_s\left(x,y\right)\le {\text{th}}_2\hfill \\ \left|\overline{I}-I_h\left(x,y\right)\right|I_s\left(x,y\right)>{\text{th}}_2\hfill \end{array}\right.\forall \left(x,y\right)\in R_i,\end{array}$$ (8)

where I_s(x, y)  is the saturation value of the pixel, I_h(x, y)  is the hue value of the pixel, I_v(x, y) is the intensity value of the pixel, and $\overline{I}$ is the mean value of all pixels.

6. Fusion of Saliency Map

When synthesizing the colour saliency map and the depth saliency map, the information entropy is used to calculate the weights of the channels.

The information entropy of colour saliency is

$$\begin{array}{c}{H}_c\left(R\right)=-\sum _{i=1}^n{p}_c\left(R_i\right)\log \left(p_c\left(R_i\right)\right),\end{array}$$ (9)

where p_c(R_i) is the ratio of the sum of R_i colour saliency values to the whole image.

The information entropy of depth saliency is

$$\begin{array}{c}{H}_d\left(R\right)=-\sum _{i=1}^n{p}_d\left(R_i\right)\log \left(p_d\left(R_i\right)\right),\end{array}$$ (10)

where p_d(R_i) is the ratio of the sum of  R_i depth saliency values of to the whole image.

The saliency map S_fuse(x, y) was obtained by fusing the two channels:

$$\begin{array}{c}{S}_{\text{fuse}}\left(x,y\right)=\frac{H_c\left(R\right)}{H_c\left(R\right)+H_d\left(R\right)}{S}_c\left(x,y\right)+\frac{H_d\left(R\right)}{H_c\left(R\right)+H_d\left(R\right)}{S}_d\left(x,y\right)\forall \left(x,y\right)\in R_i.\end{array}$$ (11)

7. Experimental Comparison

We show a few saliency maps generated by different algorithms in Figure 3.

Figure 3.

Saliency comparisons of different methods in terms of NJU400 dataset: (a) RGB image; (b) depth image; (c) ground truth; (d) GS; (e) MC; (f) MR; (g) WCTR; (h) ACSD; (i) MSD.

The precision-recall curve is evaluated from two aspects: precision and recall. Precision refers to the ratio between the number of correct saliency pixels and the whole number of saliency pixels, which is used as the y-axis. Recall refers to the ratio of the number of correct saliency pixels to the number of true pixels, which is used as the x-axis.

The algorithms are tested on NJU400 datasets. Two test sets are divided from NJU400 according to the complexity and the similarity of the background. Four volunteers are invited to divide the raw datasets into the normal group (N group) and the similar/complex background group (S/C group). At last, 92 high quality and consistently labelled images are selected into the S/C group, and the rest are divided into the N group. The precision-recall curves of different algorithms tested on the N group, S/C group, and full datasets are given in Figure 4. The performance of different algorithms tested on three groups is given in Table 1.

Figure 4.

The precision-recall curves of different algorithms: (a) the precision-recall curves on the N group; (b) the precision-recall curves on the S/C group; (c) the precision-recall curves on full datasets.

Table 1.

The performance of different algorithms tested on three groups.

Group	Algorithms	Precision
N group	GS	0.68
	MC	0.79
	MR	0.85
	RBD	0.81
	ACSD	0.84
	MSD	0.82
S/C group	GS	0.49
	MC	0.50
	MR	0.63
	RBD	0.56
	ACSD	0.87
	MSD	0.95
Full datasets	GS	0.64
	MC	0.72
	MR	0.80
	RBD	0.76
	ACSD	0.81
	MSD	0.86

The proposed method works within a complexity of O(N), and the evaluation on the results of these saliency detection algorithms in the S/C group shows that our algorithm has a better performance than other algorithms. In full datasets, it also performs well. By selecting the salient subset for further processing, the complexity of higher visual analysis can be reduced significantly. Many applications benefit from saliency analysis such as object segmentation, image classification, and image/video retargeting.

8. Conclusions

A new framework based on visual bionics for saliency detection under similar or complex background interference is proposed in this paper: First, we combine the depth map with the RGB map, and colour, texture, and depth information are introduced as the basis of superpixel segmentation. Second, the colour and depth information were calculated as two feature channels of saliency map. Finally, information entropy is used to calculate the weights of two channels, respectively, and get the final fused saliency map. The proposed method works within a complexity of O(N), and the experimental results show that our saliency detection framework greatly reduces the error detection under similar and complex background and improves the overall saliency detection performance.

Data Availability

The NJU400 datasets used to support the findings of this study are included within the article.

Conflicts of Interest

The authors declare that there is no conflict of interests regarding the publication of this paper.