3D-local oriented zigzag ternary co-occurrence fused pattern for biomedical CT image retrieval

Abstract

In this letter, a new feature descriptor called three dimensional local oriented zigzag ternary co-occurrence fused pattern ($3D\text{-}LOZTCoFP$) is proposed for computed tomography (CT) image retrieval. Unlike the conventional local pattern based approaches, where the relationship between the reference and its neighbors in a circular shaped neighborhood are captured in a 2-D plane, the proposed descriptor encodes the relationship between the reference and it’s neighbors within a local 3D block drawn from multiscale Gaussian filtered images employing a new 3D zigzag sampling structure. The proposed 3D zigzag scan around a reference not only provides an effective texture representation by capturing non-uniform and uniform local texture patterns but the fine to coarse details are also captured via multiscale Gaussian filtered images. In this letter, we have introduced three unique 3D zigzag patterns in four diverse directions. In $3D\text{-}LOZTCoFP$, we first calculate the 3D local ternary pattern within a local 3D block around a reference using proposed 3D zigzag sampling structure at both radius 1 and 2. Then the co-occurrence of similar ternary edges within the local 3D cube is computed to further enhance the discriminative power of the descriptor. A quantization and fusion based scheme is introduced to reduce the feature dimension of the proposed descriptor. Experiments are conducted on popular NEMA and TCIA-CT image databases and the results demonstrate superior retrieval efficiency of the proposed $3D\text{-}LOZTCoFP$ descriptor over many local pattern based approaches in terms of average retrieval precision and average retrieval recall in CT image retrieval.

Introduction

Medical images are now a days given utmost importance in patient care and treatment. Notable applications include disease identification and diagnosis, medical research, clinical decision making and educational purposes etc. In recent times, advent of advanced imaging technologies has made image acquisition much easier and accessible for masses and thus paving the way for development of large image libraries. Management of such large medical image libraries in hospitals and medical centers is paramount for their pertinent utilization. Therefore, development of algorithms for proper indexing, classification and retrieval of medical images according to their genre has become a challenge and has gained significant attention among the researchers community. Earlier images were indexed, classified and retrieved using texts or meta data (text based image retrieval systems), which is now replaced by content based image retrieval systems (CBIR) because of the later’s better accuracy and ease of operation. A CBIR involves two major steps: feature extraction and measurement of similarity among the features. The feature extraction step is very crucial as the effectiveness of the features determine the effectiveness of the CBIR system. Therefore the objective of any feature extraction algorithm is to incorporate definitive and discriminative information about the image while maintaining a moderate feature dimension, which is essential for accurate as well as fast image matching. A huge number of feature descriptors are available in the literature for CBIR [8, 11, 19], which can be divided into several categories according to their working principle [10, 23].

Among various feature descriptors, local pattern based feature descriptors have been highly successful in describing medical image features. One appealing attribute of these descriptors is that they encode the local statistical information of the image by analyzing the differences between each pixel and its neighborhood which assures invariance with respect to intensity changes. Besides these methods can be tuned to make robust against various other constraints such as illumination, blur, noise, scale, rotation etc. These approaches have several advantages over deep features too. The local pattern based methods are computationally much simpler and cost effective than deep learning based methods. Besides their independency over training data makes them more effective than deep features in case of images where large training data is unavailable. More over local pattern based features are usually low dimensional and hence are time efficient [2].

Local Binary Pattern (LBP) is the first local pattern based descriptor which is inherently robust against monotonic gray intensity changes as well as illumination variation [18]. Over the years, to improve the performance of LBP in medical CBIR applications, several other local pattern based descriptors have been proposed in the literature. Notable mentions include local mesh pattern (LMeP) [12], local ternary co-occurrence pattern (LTCoP) [13], local diagonal extrema pattern (LDEP) [3] etc. These methods encodes the neighbor-center or neighbor-neighbor pixel relationship information of the raw spatial image to extract the features. On the other hand local bit plane decoded pattern (LBDP) [5] and local bit plane dissimilarity pattern (LBDisP) [6] were proposed to encode such relationships from binary bit plane slices of the image. Image feature extraction from multiscale Gaussian filtered images has also gained attention in recent years. Methods such as spherically symmetric 3D local ternary pattern (SS-3D-LTP) [14] and 3D-LTCoP [1] extracts the features from a local 3D cubical image block around the reference using multiscale Gaussian filtered images. These techniques can very well capture the fine to coarse image information via Gaussian filtered images. Image features are also extracted in terms of higher order statistical information in center symmetric local binary co-occurrence pattern (CSLBCoP) [24] and in co-occurrence of adjacent sparse local ternary pattern (CoALTP) [16]. In such approaches gray level co-occurrence matrix (GLCM) is employed for this task which extracts frequency as well as spatial co-occurrence information of the encoded pattern maps.

The 2D zigzag patterns proposed in [20] is a good alternative to traditional circular patterns. These patterns were shown to be robust against the order of sampling and the spatial inter relationship between the neighbors is also well preserved.

In [5], the authors demonstrated that the LBP cannot capture many fine details present in the input images. In local pattern based techniques, usually the relationship between the center pixel and its neighbors are analyzed using simple circular sampling structure which due to its shape cannot capture many non-uniform texture details present in the images. The techniques proposed in [1, 14] define the relationship between the center pixel and its neighbors using circular sampling structure across multiple scales captured through Gaussian filter banks only in five directions, which limits its discriminative power of features. The choice of an appropriate threshold is also a major limitation in these methods. Motivated from [1, 5, 14, 20], we propose a new local texture descriptor local oriented zigzag ternary co-occurrence fused pattern ($3D\text{-}LOZTCoFP$) for retrieval of CT images. In $3D\text{-}LOZTCoFP$, three unique 3D zigzag sampling structures oriented in four diverse directions i.e. a total of 12 no. of effective 3D zigzag sampling patterns are proposed to analyze the relationship between the center pixel and its neighbors across multiple scales, that are captured through multiscale Gaussian filtered images. The frequent variations in different angles of this novel 3D zigzag sampling structure enables in better capturing of more recurring changes in local textures present in the images. And, the use of multiscale Gaussian filtered images in $3D\text{-}LOZTCoFP$, further allows the effective capturing of more fine to smooth image details. Our main contributions in this paper are

1. We calculate the local ternary pattern using a local 3D block (around a reference), drawn from multiscale Gaussian filtered images at both radius 1 and 2, considering 3D zigzag sampling structure.

2. To comprehensively analyze both uniform and the non-uniform textures in a local 3D block, we propose three unique 3D zigzag patterns which are oriented to four different directions i.e. $0^{\circ }$, ${45}^{\circ }$, ${90}^{\circ }$ and ${135}^{\circ }$ to scan the neighbors around a reference, unlike a circularly scanned neighborhood. Therefore a total of 12 effective 3D-zigzag scan patterns are introduced in this paper.

3. The co-occurrence of local ternary edges with respect to radius 1 and 2 employing the 12 3D-zigzag sampling structures is computed.

4. A quantization and mean based fusion scheme is proposed to reduce the dimensions.

Proposed $3D\text{-}LOZTCoFP$

The proposed feature extraction method is schematically depicted in Fig. 1. The overall methodologies can be divided into following steps:

Fig. 1.

Block diagram of the proposed $3D\text{-}LOZTCoFP$ based feature extraction (s1: Formation of the 3D cube using Gaussian filter bank, s2: Computation of $3D-LOZTCo{P}_{\delta }^{U,\alpha }$ (upper) and $3D-LOZTCo{P}_{\delta }^{L,\alpha }$ (lower) patterns where $\alpha \in [1,2,3]$ are the pattern types and $\delta \in [0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }]$ are the pattern orientations, s3: Computation of $3D\text{-}LOZTCoF{P}^{U,\alpha }$ and $3D\text{-}LOZTCoF{P}^{L,\alpha }$ using quantization and fusion, s4: Formation of the final feature vector (FV))

Formation of the 3D cube

In this step the input image I is first filtered with 2D circular symmetric Gaussian filter bank to obtain five Gaussian filtered images $GI\_1,GI\_2$, $GI\_3,GI\_4$ and $GI\_5$ using (1) and (2).

$$\begin{array}{cc}\hfill G(i,j,\sigma )& =\frac{1}{2\pi {\sigma }^2}{e}^{-\frac{i^2+j^2}{2{\sigma }^2}}\hfill \end{array}$$ 1

$$\begin{array}{cc}\hfill GI\_k& =G(i,j,{\sigma }_k)\ast I\hfill \end{array}$$ 2

where ${\sigma }_k$ ($k\in [1,2,3,4,5]$) are the standard deviations of the filter G and signifies different scales while ($\ast$) denotes convolution operator. The values of ${\sigma }_1,{\sigma }_2,{\sigma }_3,{\sigma }_4$, and ${\sigma }_5$ are empirically chosen as 0.45, 0.5, 0.55, 0.6 and 0.65 respectively. The five Gaussian filtered images $GI\_1,GI\_2$, $GI\_3,GI\_4$ and $GI\_5$ are used to construct the five planes of the 3D cube. These five images contain multiscale information of the image. The use of these images together allows to capture information of inter scale variation in image texture which provide more comprehensive details of the image and thus enhances the discrimination capability of the descriptor.

3D-zigzag scanning

In this step, around each reference pixel position in the input image, we consider a $5\times 5\times 5$ local neighborhood in five Gaussian planes to form a local 3D cube. A demonstration of one such cube is depicted in Fig. 2.

Fig. 2.

Position of the neighboring pixels in all the individual planes of a 3D cube. ‘r’ denotes the neighbors at radius ($\mathrm{\Omega }$) = 1, ‘R’ denotes neighbors at radius ($\mathrm{\Omega }$) = 2

To scan this cube, three unique 3D zigzag patterns are proposed. A 2D view of these proposed 3D zigzag patterns are shown in Fig. 3 for ease of understanding. These patterns are oriented to $0^{\circ }$, ${45}^{\circ }$, ${90}^{\circ }$ and ${135}^{\circ }$ directions to scan the 3D cube. The scanning mechanism is demonstrated in Fig. 4, where three unique zigzag patterns in four diverse directions are shown to scan the 3D cube at both radius 1 and 2 resulting into a total of 12 oriented zigzag patterns.

Fig. 3.

The proposed zigzag patterns (2D view), a pattern 1 ($Z{P}^1$), b pattern 2 ($Z{P}^2$), c pattern 3 ($Z{P}^3$)

Fig. 4.

Consideration of 3D zigzag planes ($Z{P}_{\mathrm{\Omega },\delta }^{\alpha }$). a $Z{P}_{1,0^{\circ }}^1$ and $Z{P}_{2,0^{\circ }}^1$, b $Z{P}_{1,{45}^{\circ }}^1$ and $Z{P}_{2,{45}^{\circ }}^1$, c $Z{P}_{1,{90}^{\circ }}^1$ and $Z{P}_{2,{90}^{\circ }}^1$, d $Z{P}_{1,{135}^{\circ }}^1$ and $Z{P}_{2,{135}^{\circ }}^1$, e $Z{P}_{1,0^{\circ }}^2$ and $Z{P}_{2,0^{\circ }}^2$, f $Z{P}_{1,{45}^{\circ }}^2$ and $Z{P}_{2,{45}^{\circ }}^2$, g $Z{P}_{1,{90}^{\circ }}^2$ and $Z{P}_{2,{90}^{\circ }}^2$, h $Z{P}_{1,{135}^{\circ }}^2$ and $Z{P}_{2,{135}^{\circ }}^2$, i $Z{P}_{1,0^{\circ }}^3$ and $Z{P}_{2,0^{\circ }}^3$, j $Z{P}_{1,{45}^{\circ }}^3$ and $Z{P}_{2,{45}^{\circ }}^3$, k $Z{P}_{1,{90}^{\circ }}^3$ and $Z{P}_{2,{90}^{\circ }}^3$, l $Z{P}_{1,{135}^{\circ }}^3$ and $Z{P}_{2,{135}^{\circ }}^3$

We denote the proposed 3D zigzag patterns as $Z{P}_{\mathrm{\Omega },\delta }^{\alpha }$ (where $\alpha \in [1,2,3]$ is the pattern type, $\mathrm{\Omega }=[1,2]$ is the radius of the neighborhood and $\delta \in [0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }$] is the pattern orientation).

The zigzag patterns at $\mathrm{\Omega }=1$ are

$$\begin{array}{cc}\hfill Z{P}_{1,0^{\circ }}^1& =\left[\begin{array}{c}GI\_2(r4),GI\_2(C),GI\_3(r4),GI\_4(r4),\hfill \\ GI\_2(r5),GI\_3(r5),GI\_4(C),GI\_4(r5)\hfill \end{array}\right]\hfill \end{array}$$ 3

$$\begin{array}{cc}\hfill Z{P}_{1,0^{\circ }}^2& =\left[\begin{array}{c}GI\_3(r4),GI\_2(r4),GI\_2(C),GI\_2(r5),\hfill \\ GI\_4(r4),GI\_4(C),GI\_4(r5),GI\_3(r5)\hfill \end{array}\right]\hfill \end{array}$$ 4

$$\begin{array}{cc}\hfill Z{P}_{1,0^{\circ }}^3& =\left[\begin{array}{c}GI\_4(r4),GI\_3(r4),GI\_2(r4),GI\_2(C),\hfill \\ GI\_4(C),GI\_4(r5),GI\_3(r5),GI\_2(r5)\hfill \end{array}\right]\hfill \end{array}$$ 5

$$\begin{array}{cc}\hfill Z{P}_{1,{45}^{\circ }}^1& =\left[\begin{array}{c}GI\_2(r6),GI\_2(C),GI\_3(r6),GI\_4(r6),\hfill \\ GI\_2(r3),GI\_3(r3),GI\_4(C),GI\_4(r3)\hfill \end{array}\right]\hfill \end{array}$$ 6

$$\begin{array}{cc}\hfill Z{P}_{1,{45}^{\circ }}^2& =\left[\begin{array}{c}GI\_3(r6),GI\_2(r6),GI\_2(C),GI\_2(r3),\hfill \\ GI\_4(r6),GI\_4(C),GI\_4(r3),GI\_3(r3)\hfill \end{array}\right]\hfill \end{array}$$ 7

$$\begin{array}{cc}\hfill Z{P}_{1,{45}^{\circ }}^3& =\left[\begin{array}{c}GI\_4(r6),GI\_3(r6),GI\_2(r6),GI\_2(C),\hfill \\ GI\_4(C),GI\_4(r3),GI\_3(r3),GI\_2(r3)\hfill \end{array}\right]\hfill \end{array}$$ 8

$$\begin{array}{cc}\hfill Z{P}_{1,{90}^{\circ }}^1& =\left[\begin{array}{c}GI\_2(r7),GI\_2(C),GI\_3(r7),GI\_4(r7),\hfill \\ GI\_2(r2),GI\_3(r2),GI\_4(C),GI\_4(r2)\hfill \end{array}\right]\hfill \end{array}$$ 9

$$\begin{array}{cc}\hfill Z{P}_{1,{90}^{\circ }}^2& =\left[\begin{array}{c}GI\_3(r7),GI\_2(r7),GI\_2(C),GI\_2(r2),\hfill \\ GI\_4(r7),GI\_4(C),GI\_4(r2),GI\_3(r2)\hfill \end{array}\right]\hfill \end{array}$$ 10

$$\begin{array}{cc}\hfill Z{P}_{1,{90}^{\circ }}^3& =\left[\begin{array}{c}GI\_4(r7),GI\_3(r7),GI\_2(r7),GI\_2(C),\hfill \\ GI\_4(C),GI\_4(r2),GI\_3(r2),GI\_2(r2)\hfill \end{array}\right]\hfill \end{array}$$ 11

$$\begin{array}{cc}\hfill Z{P}_{1,{135}^{\circ }}^1& =\left[\begin{array}{c}GI\_2(r8),GI\_2(C),GI\_3(r8),GI\_4(r8),\hfill \\ GI\_2(r1),GI\_3(r1),GI\_4(C),GI\_4(r1)\hfill \end{array}\right]\hfill \end{array}$$ 12

$$\begin{array}{cc}\hfill Z{P}_{1,{135}^{\circ }}^2& =\left[\begin{array}{c}GI\_3(r8),GI\_2(r8),GI\_2(C),GI\_2(r1),\hfill \\ GI\_4(r8),GI\_4(C),GI\_4(r1),GI\_3(r1)\hfill \end{array}\right]\hfill \end{array}$$ 13

$$\begin{array}{cc}\hfill Z{P}_{1,{135}^{\circ }}^3& =\left[\begin{array}{c}GI\_4(r8),GI\_3(r8),GI\_2(r8),GI\_2(C),\hfill \\ GI\_4(C),GI\_4(r1),GI\_3(r1),GI\_2(r1)\hfill \end{array}\right]\hfill \end{array}$$ 14

similarly, zigzag patterns at $\mathrm{\Omega }=2$ are as follows

$$\begin{array}{cc}\hfill Z{P}_{2,0^{\circ }}^1& =\left[\begin{array}{c}GI\_1(R4),GI\_1(C),GI\_3(R4),GI\_5(R4),\hfill \\ GI\_1(R5),GI\_3(R5),GI\_5(C),GI\_5(R5)\hfill \end{array}\right]\hfill \end{array}$$ 15

$$\begin{array}{cc}\hfill Z{P}_{2,0^{\circ }}^2& =\left[\begin{array}{c}GI\_3(R4),GI\_1(R4),GI\_1(C),GI\_1(R5),\hfill \\ GI\_5(R4),GI\_5(C),GI\_5(R5),GI\_3(R5)\hfill \end{array}\right]\hfill \end{array}$$ 16

$$\begin{array}{cc}\hfill Z{P}_{2,0^{\circ }}^3& =\left[\begin{array}{c}GI\_5(R4),GI\_3(R4),GI\_1(R4),GI\_1(C),\hfill \\ GI\_5(C),GI\_5(R5),GI\_3(R5),GI\_1(R5)\hfill \end{array}\right]\hfill \end{array}$$ 17

$$\begin{array}{cc}\hfill Z{P}_{2,{45}^{\circ }}^1& =\left[\begin{array}{c}GI\_1(R6),GI\_1(C),GI\_3(R6),GI\_5(R6),\hfill \\ GI\_1(R3),GI\_3(R3),GI\_5(C),GI\_5(R3)\hfill \end{array}\right]\hfill \end{array}$$ 18

$$\begin{array}{cc}\hfill Z{P}_{2,{45}^{\circ }}^2& =\left[\begin{array}{c}GI\_3(R6),GI\_1(R6),GI\_1(C),GI\_1(R3),\hfill \\ GI\_5(R6),GI\_5(C),GI\_5(R3),GI\_3(R3)\hfill \end{array}\right]\hfill \end{array}$$ 19

$$\begin{array}{cc}\hfill Z{P}_{2,{45}^{\circ }}^3& =\left[\begin{array}{c}GI\_5(R6),GI\_3(R6),GI\_1(R6),GI\_1(C),\hfill \\ GI\_5(C),GI\_5(R3),GI\_3(R3),GI\_1(R3)\hfill \end{array}\right]\hfill \end{array}$$ 20

$$\begin{array}{cc}\hfill Z{P}_{2,{90}^{\circ }}^1& =\left[\begin{array}{c}GI\_1(R7),GI\_1(C),GI\_3(R7),GI\_5(R7),\hfill \\ GI\_1(R2),GI\_3(R2),GI\_5(C),GI\_5(R2)\hfill \end{array}\right]\hfill \end{array}$$ 21

$$\begin{array}{cc}\hfill Z{P}_{2,{90}^{\circ }}^2& =\left[\begin{array}{c}GI\_3(R7),GI\_1(R7),GI\_1(C),GI\_1(R2),\hfill \\ GI\_5(R7),GI\_5(C),GI\_5(R2),GI\_3(R2)\hfill \end{array}\right]\hfill \end{array}$$ 22

$$\begin{array}{cc}\hfill Z{P}_{2,{90}^{\circ }}^3& =\left[\begin{array}{c}GI\_5(R7),GI\_3(R7),GI\_1(R7),GI\_1(C),\hfill \\ GI\_5(C),GI\_5(R2),GI\_3(R2),GI\_1(R2)\hfill \end{array}\right]\hfill \end{array}$$ 23

$$\begin{array}{cc}\hfill Z{P}_{2,{135}^{\circ }}^1& =\left[\begin{array}{c}GI\_1(R8),GI\_1(C),GI\_3(R8),GI\_5(R8),\hfill \\ GI\_1(R1),GI\_3(R1),GI\_5(C),GI\_5(R1)\hfill \end{array}\right]\hfill \end{array}$$ 24

$$\begin{array}{cc}\hfill Z{P}_{2,{135}^{\circ }}^2& =\left[\begin{array}{c}GI\_3(R8),GI\_1(R8),GI\_1(C),GI\_1(R1),\hfill \\ GI\_5(R8),GI\_5(C),GI\_5(R1),GI\_3(R1)\hfill \end{array}\right]\hfill \end{array}$$ 25

$$\begin{array}{cc}\hfill Z{P}_{2,{135}^{\circ }}^3& =\left[\begin{array}{c}GI\_5(R8),GI\_3(R8),GI\_1(R8),GI\_1(C),\hfill \\ GI\_5(C),GI\_5(R1),GI\_3(R1),GI\_1(R1)\hfill \end{array}\right]\hfill \end{array}$$ 26

Threshold computation

Before computing $3D\text{-}LOZTCoFP$, the threshold of the 3D cube is computed. In many situations it is observed that the center pixels of all the five Gaussian filtered planes inside the local 3D cube may have significant variations within them, therefore we have calculated the threshold with mean of these values instead of only considering the center pixel of middle plane i.e. $GI\_3$ as proposed in [1]. The threshold ‘$\rho$’ is computed as

$$\begin{array}{c}\hfill \rho =\frac{1}{5}\sum _{i=1}^{5}GI\_i(C)\end{array}$$ 27

For instance, in NEMA database [17], using the $\rho$ obtained from (27) we get % average retrieval precision (ARP) [24] as 85.52%; while using the center pixel of the $GI\_3$ plane as $\rho$ we obtain % ARP as 85.05%. In TCIA database considering the proposed $\rho$, we obtain % ARP of 94.27 % and considering the center pixel of $GI\_3$ plane, we obtain % ARP of 92.96%. These results validate the effectiveness of the $\rho$ calculated using (27) over the technique used in [1].

3D-LOZTCoP computation

In this step the relationship between $Z{P}_{1,\delta }^{\alpha }$s and the reference ($\rho$) of the cube are first encoded into ternary values using following equations

$$\begin{array}{c}\hfill ZLT{P}_{1,\delta }^{\alpha }(I_C)=f(Z{P}_{1,\delta }^{\alpha },\rho ,\lambda )\end{array}$$ 28

where $\lambda$ is a user defined threshold and is empirically chosen here as 4 and

$$\begin{array}{c}\hfill f(a,b,c)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}a\ge b+c\\ -1,\hfill & \mathrm{if}a\le b-c\\ 0,\hfill & \mathrm{otherwise}\end{array}\right)\end{array}$$ 29

Then the relationship between $Z{P}_{2,\delta }^{\alpha }$ and $Z{P}_{1,\delta }^{\alpha }$ are encoded into ternary values using following equation

$$\begin{array}{c}\hfill ZLT{P}_{2,\delta }^{\alpha }(I_C)=f(Z{P}_{2,\delta }^{\alpha },Z{P}_{1,\delta }^{\alpha },\lambda )\end{array}$$ 30

Both $ZLT{P}_{1,\delta }^{\alpha }(I_C)$ and $ZLT{P}_{2,\delta }^{\alpha }(I_C)$ individually corresponds to 8 no. of zigzag ternary values obtained with respect to center pixel $I_C$ (Fig. 5d). As the local ternary based coding encodes the first order statistical information between the reference pixel and its neighbors, the further computation of co-occurrence between $ZLT{P}_{1,\delta }^{\alpha }$ and $ZLT{P}_{2,\delta }^{\alpha }$ provides second order statistics between them and hence increases the discrimination power of the descriptor. We calculate the co-occurrence between $ZLT{P}_{1,\delta }^{\alpha }$ and $ZLT{P}_{2,\delta }^{\alpha }$ and encode it into two binary codes upper and lower zigzag local ternary co-occurrence patterns ($ZLTCo{P}_{\delta }^{U,\alpha }$ and $ZLTCo{P}_{\delta }^{L,\alpha }$) using (31) and (32) respectively (Fig. 5e).

$$\begin{array}{cc}& ZLTCo{P}_{\delta }^{U,\alpha }(I_C)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}ZLT{P}_{2,\delta }^{\alpha }(I_C)=ZLT{P}_{1,\delta }^{\alpha }(I_C)=1\\ 0,\hfill & \mathrm{otherwise}\end{array}\right)\hfill \end{array}$$ 31

$$\begin{array}{cc}& ZLTCo{P}_{\delta }^{L,\alpha }(I_C)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}ZLT{P}_{2,\delta }^{\alpha }(I_C)=ZLT{P}_{1,\delta }^{\alpha }(I_C)=-1\\ 0,\hfill & \mathrm{otherwise}\end{array}\right)\hfill \end{array}$$ 32

where $ZLTCo{P}_{\delta }^{U,\alpha }$ are referred to as upper ZLTCoP bit patterns and $ZLTCo{P}_{\delta }^{L,\alpha }$ are the lower ZLTCoP bit patterns (Fig. 5e). From $ZLTCo{P}_{\delta }^{U,\alpha }$ and $ZLTCo{P}_{\delta }^{L,\alpha }$, we calculate the 3D local oriented zigzag ternary co-occurrence pattern maps $3D-LOZTCo{P}_{\delta }^{U,\alpha }$ and $3D-LOZTCo{P}_{\delta }^{L,\alpha }$ respectively by applying weights and then summing the weighted bits as follows (Fig. 5e).

$$\begin{array}{cc}& 3D-LOZTCo{P}_{\delta }^{U,\alpha }(I_C)=\sum _{i=1}^8{2}^{i-1}\times ZLTCo{P}_{\delta }^{U,\alpha }\hfill \end{array}$$ 33

$$\begin{array}{cc}& 3D-LOZTCo{P}_{\delta }^{L,\alpha }(I_C)=\sum _{i=1}^8{2}^{i-1}\times ZLTCo{P}_{\delta }^{L,\alpha }\hfill \end{array}$$ 34

Since there are three $\alpha$ values and four $\delta$ values, total of 12 ($3\times 4$) pattern maps each for $3D-LOZTCo{P}_{\delta }^{U,\alpha }$ and $3D-LOZTCo{P}_{\delta }^{L,\alpha }$ are obtained at the end of this step. An example computation of $3D-LOZTCoP$ is shown in Fig. 5, where $3D-LOZTCo{P}_{0^{\circ }}^{U,1}$ and $3D-LOZTCo{P}_{0^{\circ }}^{L,1}$ are computed from five Gaussian planes of a given image.

Fig. 5.

Example computation of $3D-LOZTCo{P}_{0^{\circ }}^{U,1}$ and $3D-LOZTCo{P}_{0^{\circ }}^{L,1}$ pattern values. a The five Gaussian planes, b consideration of the zigzag planes at $\mathrm{\Omega }=1$ and 2, c $Z{P}_{1,0^{\circ }}^1$ and $Z{P}_{2,0^{\circ }}^1$ (simplified 2D view), d $ZLT{P}_{1,0^{\circ }}^1$ and $ZLT{P}_{2,0^{\circ }}^1$ for $\rho$ = 115.4 (simplified 2D view), e final computation of $3D-LOZTCo{P}_{0^{\circ }}^{U,1}$ and $3D-LOZTCo{P}_{0^{\circ }}^{L,1}$ values

Quantization and fusion of $3D-LOZTCo{P}_{\delta }^{U,\alpha }$ and $3D-LOZTCo{P}_{\delta }^{L,\alpha }$

In this step, 12 number of $3D-LOZTCo{P}_{\delta }^{U,\alpha }$ and 12 number of $3D-LOZTCo{P}_{\delta }^{L,\alpha }$ pattern maps are first quantized in order to reduce its range of values. The values in these pattern maps are originally in the range [0, 255], therefore if histograms are computed from these maps and are concatenated directly it will result into a large feature dimension of $24\times 256=6144$ features. Therefore we propose to quantize the pattern values in the range [0, 15] using (35) and (36) to generate quantized pattern maps as $3D-LOZTCoq{P}_{\delta }^{U,\alpha }$ and $3D-LOZTCoq{P}_{\delta }^{L,\alpha }$ respectively. It should be noted that quantizing in the range of [0, 15] reduces the feature dimensions by 16 times. The quantization to a level of [0, 15] still exhibits better performance than many existing techniques including several state of the art. The effect of quantization with various levels on retrieval performance is described in Sect. 3.

$$\begin{array}{cc}& 3D-LOZTCoq{P}_{\delta }^{U,\alpha }\hfill \\ \hfill & =round\left(\frac{15}{255},\left(3,D,-,L,O,Z,T,C,o,P_{\delta }^{U,\alpha }\right)\right)\hfill \end{array}$$ 35

$$\begin{array}{cc}& 3D-LOZTCoq{P}_{\delta }^{L,\alpha }\hfill \\ \hfill & =round\left(\frac{15}{255},\left(3,D,-,L,O,Z,T,C,o,P_{\delta }^{L,\alpha }\right)\right)\hfill \end{array}$$ 36

After quantization, in order to reduce the feature dimension further, the quantized pattern maps for each unique 3D zigzag pattern ($\alpha =[1,2,3]$) is obtained by fusing the corresponding four orientations ($\delta =0^{\circ },{45}^{\circ },{90}^{\circ }$ and ${135}^{\circ }$) maps together. Mean based fusion is employed at this step. Therefore, corresponding to a given $\alpha$ the average of its pattern maps $\delta =0^{\circ },{45}^{\circ },{90}^{\circ }$ and ${135}^{\circ }$ are computed to construct the fused pattern maps $3D\text{-}LOZTCoF{P}^{U,\alpha }$ and $3D\text{-}LOZTCoF{P}^{L,\alpha }$ using (37) and (38) respectively. Since, there are three $\alpha$ values, at the end of these step, we will get a total of six $3D\text{-}LOZTCoF{P}^{U/L,\alpha }$ pattern maps (three pattern maps each for upper and lower), the values of which are in the range [0, 15].

$$\begin{array}{cc}& 3D{\text{-LOZTCoFP}}^{U,\alpha }=\frac{1}{4}\sum _{i=1}^{4}3D-LOZTCoq{P}_{\delta (i)}^{U,\alpha }\hfill \end{array}$$ 37

$$\begin{array}{cc}& 3D{\text{-LOZTCoFP}}^{L,\alpha }=\frac{1}{4}\sum _{i=1}^{4}3D-LOZTCoq{P}_{\delta (i)}^{L,\alpha }\hfill \end{array}$$ 38

where $\delta =[0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }]$ are the orientations of the patterns.

Feature vector formation

To extract the features, all the six maps are divided into a number of non overlapping sub-maps of equal size. Histograms are computed from each of these sub-maps and at last all the histograms are concatenated together to form the final feature vector. Here four sub-maps are considered for each pattern images. This step helps to incorporate more local spatial information into the features. The final feature vector is formed as

$$\begin{array}{c}\hfill FV=\left[\begin{array}{c}Hist(3D\text{-}LOZTCoF{P}_{S1}^{U,1}),Hist(3D\text{-}LOZTCoF{P}_{S2}^{U,1}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S3}^{U,1}),Hist(3D\text{-}LOZTCoF{P}_{S4}^{U,1}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S1}^{U,2}),Hist(3D\text{-}LOZTCoF{P}_{S2}^{U,2}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S3}^{U,2}),Hist(3D\text{-}LOZTCoF{P}_{S4}^{U,2}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S1}^{U,3}),Hist(3D\text{-}LOZTCoF{P}_{S2}^{U,3}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S3}^{U,3}),Hist(3D\text{-}LOZTCoF{P}_{S4}^{U,3}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S1}^{L,1}),Hist(3D\text{-}LOZTCoF{P}_{S2}^{L,1}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S3}^{L,1}),Hist(3D\text{-}LOZTCoF{P}_{S4}^{L,1}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S1}^{L,2}),Hist(3D\text{-}LOZTCoF{P}_{S2}^{L,2}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S3}^{L,2}),Hist(3D\text{-}LOZTCoF{P}_{S4}^{L,2}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S1}^{L,3}),Hist(3D\text{-}LOZTCoF{P}_{S2}^{L,3}),\hfill \\ Hist(3D\text{-}LOZTCoF{P}_{S3}^{L,3}),Hist(3D\text{-}LOZTCoF{P}_{S4}^{L,3})\hfill \end{array}\right]\end{array}$$ 39

where $3D\text{-}LOZTCoF{P}_{S1}^{U/L,\alpha }$, $3D\text{-}LOZTCoF{P}_{S2}^{U/L,\alpha }$, $3D\text{-}LOZTCoF{P}_{S3}^{U/L,\alpha }$ and $3D\text{-}LOZTCoF{P}_{S4}^{U/L,\alpha }$ are the non overlapping sub maps of corresponding $3D\text{-}LOZTCoF{P}^{U/L,\alpha }$ (where $\alpha =[1,2,3]$). The feature vector dimension of the proposed method is $6\times 4\times 16=384$ features.

Figure 6 shows the feature maps [12, 15, 21] of a sample image selected from NEMA-CT database, generated using different feature descriptors. Due to the use of proposed 3D zigzag scans in various directions inside a local 3D block, which is drawn from multiscale Gaussian filtered images, Fig. 6k clearly shows the capturing of both ‘uniform’ and ‘non-uniform’ textures as compared to the other feature maps shown in Fig. 6b–j. The capturing of very fine to coarse image information is also clearly visible in Fig. 6k.

Fig. 6.

Example of feature maps a original image, b using LBP, c using CSLBP, d using LWP, e using LBDP, f using LBDisP, g using CoCSLBP, h using LDEP, i using SS-3D-LTP, j using 3D-LTCoP, k using $3D\text{-}LOZTCoFP$

Experimental results

The retrieval efficiency of the proposed technique is examined on two benchmark medical CT image databases NEMA-CT [17] and TCIA-CT [22] and then compared with that of many recent local pattern based techniques. The details of these databases are given in Tables 1 and 2. The sample images of these databases are shown in Fig. 7. For performance comparison, LBP [18], CSLBCoP [24], LDEP [3], LWP [4], LBDP [5], LBDisP [6], CoCSLBP [9], SS-3D-LTP [14] and 3D-LTCoP [1] techniques are chosen.

Table 1.

Summary of National Electrical Manufacturers Association (NEMA) database

Image details (data set name: number of images)	Image size
CT0001: 36, CT0003: 18, CT0003: 36, CT0057: 37, CT0057: 41, CT0060: 30, CT0060: 23, CT0080: 70, CT0083: 24	$512\times 512$

Table 2.

Summary of The Cancer Imaging Archive (TCIA) database

Image details (subject ID: number of images)	image size
1.3.6.1.4.1.9328.50.5.0001: 59, TCGA-CS-6186: 25, TCGA-K7-A6G5: 32, PANCREAS_001: 40, PANCREAS_003: 31, C3N-02012: 42, AMC-027: 43, BreastDx-01-0075: 25, TCGA-ZF-AA5P: 20, TCGA-FI-A2F8: 36, TCGA-DE-A4MC: 44, C3N-02275: 73, 1.3.6.1.4.1.9328.50.4.0001: 57, 69, 63, PANCREAS_0003: 37	$512\times 512$

Fig. 7.

One image from each class of a NEMA database; b TCIA database

To evaluate the retrieval performance, every image in the database is considered as query once. Next, using a given technique, a set of images are retrieved from the database against a query image through similarity measurement of the respective features. With the proposed method, similarity measurement is done using ‘$d_1$’ distance measure. The number of images retrieved at a given instance is termed as ‘number of top matches’. Two standard retrieval efficiency determining parameters average retrieval precision (ARP) and average retrieval recall (ARR) [9] are then calculated for for each ‘number of top matches’ for all the techniques. It should be noted that higher the values of ARP and ARR, better the retrieval performance.

Experiment with NEMA-CT dataset

The first experiment is conducted on NEMA-CT database [17] which consists of 315 CT images corresponding to 9 different classes as shown in Table 1 and Fig. 7a. In each of these classes there are 36, 18, 36, 37, 41, 30, 23, 70 and 24 images respectively. To compute ARP and ARR, we retrieved the images for top match of 5, 10, 15, ...30 images. We plot the % ARP and % ARR values corresponding to different techniques against various number of top matches as shown in Fig. 8a, b. Table 3 presents the % ARP values of various techniques for 30 number of top matches. It can observed from Fig. 8a, b that the proposed $3D\text{-}LOZTCoFP$ consistently outperforms all the other techniques including relevant 3D plane based methods SS-3D-LTP and 3D-LTCoP for all the top matches. Table 3 depicts that in terms of % ARP, $3D\text{-}LOZTCoFP$ outperforms LBP, CSLBCoP, LDEP, LWP, LBDP, LBDisP, CoCSLBP, SS-3D-LTP and 3D-LTCoP techniques by 8.19%, 6.25%, 13.21%, 12.42%, 6.22%, 6.40%, 5.27%, 3.83% and 4.86% respectively. From these results we can validate the superior retrieval efficiency of the proposed method over the existing methods in NEMA-CT database.

Fig. 8.

Comparison of retrieval performance of the proposed approach with other approaches in terms of ARP and ARR respectively in a, b NEMA database, c, d TCIA database

Table 3.

Retrieval results comparison in terms of % ARP in NEMA (for 30 number of top matches) and TCIA (for 30 number of top matches) databases

Method name	LBP [18]	CSLBCoP [24]	LDEP [3]	LWP [4]	LBDP [5]	LBDisP [6]	CoCSLBP [9]	SS-3D-LTP [14]	3D-LTCoP [1]	$3D\text{-}LOZTCoFP$
NEMA database	79.12	80.57	75.62	76.15	80.59	80.46	81.32	82.44	81.64	85.52
TCIA database	75.93	75.24	73.49	75.36	81.96	89.73	87.67	91.64	92.00	94.27

Bold indicates the best values for each case

Experiment with TCIA-CT dataset

We conducted the next experiment on another CT image database built from ‘The Cancer Imaging Archive’ [22] and the details of this database are given in Table 2. TCIA-CT database contains 696 images of 16 different classes as shown in Fig. 7b. Each class contains 59, 25, 32, 40, 31, 42, 43, 25, 20, 36, 44, 73, 57, 69, 63 and 37 images respectively. Here using a given technique the images are retrieved for top match of 5, 10, 15, ...30 and ARP, ARR values are computed subsequently. In Fig. 8c, d the plots of % ARP and % ARR corresponding to different techniques as function of number of top matches considered are shown. The % ARP values of various techniques for 30 top matches are shown in Table 3. From Fig. 8c, d and Table 3, it can be observed that the proposed $3D\text{-}LOZTCoFP$ technique outperforms all the other techniques. In Table 3 for TCIA- CT database, $3D\text{-}LOZTCoFP$ again shows the highest % ARP for 30 i.e. the maximum number of top matches among all the techniques. It can be seen that $3D\text{-}LOZTCoFP$ improves upon the performance of LBP, CSLBCoP, LDEP, LWP, LBDP, LBDisP, CoCSLBP, SS-3D-LTP and 3D-LTCoP techniques by 24.15%, 25.29%, 28.28%, 25.08%, 15.02%, 5.06%, 7.52%, 2.86% and 2.46% respectively in terms of % ARP. From these results we can deduce that the proposed method shows superior retrieval performance than the existing methods in TCIA-CT database too.

The retrieval performance of proposed technique is investigated for various levels of quantization in $3D\text{-}LOZTCoF{P}_{\delta }^{U/L,\alpha }$ maps and the results are presented in Table 4. It is observed that for NEMA-CT (where the inter-class variation is high) with decreasing quantization levels, the retrieval performance slightly increases. However, for TCIA-CT(where the inter-class variation is low) with decreasing quantization levels, the retrieval performance slightly decreases. The effect of variation of quantization levels on retrieval performance is dataset dependent. One can select the optimum number of quantization levels based on the statistics of images in the dataset and/or the feature dimensions constraint (if any). From Table 4, it can be seen that the quantization level of 16 reduces the feature dimension significantly up to 384 features with relatively less drop in % ARP and % ARR and still outperforms the other techniques with a decent margin. Therefore in this paper, we set our quantization level at 16.

Table 4.

Retrieval results of $3D\text{-}LOZTCoFP$ in terms of % ARP and % ARR in NEMA (for 30 number of top matches) and TCIA (for 30 number of top matches) databases for different quantization levels

Number of quantization levels in the pattern image	NEMA		TCIA		Feature dimension
	% ARP	% ARR	% ARP	% ARR
256	83.48	68.90	95.56	64.43	6144
128	84.43	69.74	95.64	64.44	3072
64	84.94	70.04	95.39	64.25	1536
32	85.20	70.15	94.90	63.88	768
16	85.52	70.16	94.27	63.43	384

The effect of different fusion methods are also investigated with the proposed technique. We employed ‘max’, ‘min’, ‘median’ and ‘mean’ fusions with $3D\text{-}LOZTCoFP$. The retrieval results obtained for all the fusion methods in both the databases are presented in Table 5. We have chosen mean based fusion for our proposed technique as it provides better results for both NEMA-CT and TCIA-CT databases.

Table 5.

% ARP and % ARR comparison of $3D\text{-}LOZTCoFP$ for different fusion methods

	Max	Min	Median	Mean
NEMA
% ARP	85.82	82.92	86.20	85.52
% ARR	70.48	67.85	70.41	70.16
TCIA
% ARP	91.00	88.06	92.88	94.27
% ARR	61.18	59.19	62.57	63.43

The proposed approach is also experimented with four different distance measures ‘Manhattan’, ‘Euclidian’, ‘chi-square’ and ‘$d_1$’ [25], among which $d_1$ distance gives the best results in most of the cases (Table 6) and hence used for similarity measurement with $3D\text{-}LOZTCoFP$.

Table 6.

% ARP and % ARR comparison of $3D\text{-}LOZTCoFP$ for different distance measures

	Manhattan	Euclidian	Chi-sq	$d_1$
NEMA
% ARP	85.52	83.53	85.60	85.52
% ARR	69.73	68.32	69.87	70.16
TCIA
% ARP	91.31	88.90	93.84	94.27
% ARR	61.24	59.43	63.07	63.43

Bold indicates the best values for each case

Feature dimensions and computation time

It is observed that the feature extraction time of proposed $3D\text{-}LOZTCoFP$ is quite high as compared to all other techniques. It is mainly due to the generation of five multiscale Gaussian filtered images and also the use of three unique 3D zigzag patterns in four different directions to scan the 3D cube. It should be noted that in real time applications in CBIR systems, the extraction of features of all the images in the database are carried out only once which determines the total feature extraction time [5, 7, 24]. If a new input query image is given, the feature extraction is required to be carried out only for the query image, while for retrieval the similarity matching between the features of query and database images is needed. Therefore, once the extraction of features for the whole database is completed before the image retrieval, it is the total retrieval time which plays an important role [5, 7, 24].

The feature dimension of proposed $3D\text{-}LOZTCoFP$ descriptor is larger than LBP, LDEP, LWP, LBDP, LBDisP, CoCSLBP descriptors and much less than CSLBCoP, SS-3D-LTP and 3D-LTCoP descriptors.

From Table 7, it can be seen that the total feature extraction time of $3D\text{-}LOZTCoFP$ is {105.25,109.13}, {22.06,21.78}, {36.26,32.99}, {26.82,26.34}, {5.52,5.22}, {5.09,4.80}, {24.36,23.32}, {6.67,6.43}, {2.09,2.00} times slower than the total feature extraction times of LBP, CSLBCoP, LDEP, LWP, LBDP, LBDisP, CoCSLBP, SS-3D-LTP and 3D-LTCoP descriptors over {NEMA-CT,TCIA-CT} databases respectively. However, the total retrieval time of $3D\text{-}LOZTCoFP$ is {0.64,0.64}, {2.04,2.58}, {0.35,0.13}, {0.68,0.67}, {0.82,0.66}, {0.68,0.66}, {0.35,0.15}, {4.76,6.22}, {4.97,6.37} times faster than LBP, CSLBCoP, LDEP, LWP, LBDP, LBDisP, CoCSLBP, SS-3D-LTP, 3D-LTCoP techniques when tested over {NEMA-CT,TCIA-CT} databases respectively.

Table 7.

The feature dimension, total feature extraction time and total retrieval time (in seconds) for NEMA-CT and TCIA-CT databases

Method Name	LBP [18]	CSLBCoP [24]	LDEP [3]	LWP [4]	LBDP [5]	LBDisP [6]	CoCSLBP [9]	SS-3D-LTP [14]	3D-LTCoP [1]	3D{\text{-}}LOZTCoFP
Feature dimension	256	1024	24	256	256	256	64	2560	2560	384
Total feature extraction time (NEMA-CT database: 315 images)	17.52	83.57	50.85	68.75	333.84	362.35	75.71	276.39	878.78	1843.94
Total feature extraction time (TCIA-CT database: 696 images)	35.70	178.87	118.09	147.87	746.54	811.76	167.06	605.64	1947.93	3895.88
Total retrieval time (When 30 no. of images are retrieved in NEMA database)	0.79	2.50	0.43	0.84	1.01	0.84	0.43	5.84	6.10	1.228
Total retrieval time (When 30 no. of images are retrieved in TCIA database)	2.60	10.49	0.53	2.72	2.70	2.67	0.63	25.33	25.91	4.07

It is worth noting that the total retrieval time mainly depends on feature dimensions and low dimensional descriptors are highly time effective for huge image databases [5, 7, 24].

From this study, it is quite clear that the computations of $3D\text{-}LOZTCoFP$ is not much affecting the total retrieval time and is much faster than CSLBCoP, relevant SS-3D-LTP and 3D-LTCoP techniques in terms of retrieval time. Moreover, the retrieval performance of $3D\text{-}LOZTCoFP$ is also highly superior to all other techniques both in terms of %ARP and %ARR. However, $3D\text{-}LOZTCoFP$ is slower than all other methods only in terms of total feature extraction time which is carried out only once, before image retrieval.

Conclusion

In this letter, we propose a new and effective texture descriptor 3D local oriented zigzag ternary co-occurrence fused pattern ($3D\text{-}LOZTCoFP$) for CT image retrieval. We encode the relationship between reference pixel and its neighbors using 3D zigzag sampling structure within the local 3D block drawn from multiscale Gaussian filtered images. Three unique 3D zigzag patterns locally oriented to four different directions are proposed for effective representation of texture. The proposed descriptor not only captures the uniform and non-uniform texture information very well but the fine to coarse image information are also well captured via Gaussian filtered images. The computation of co-occurrence between local ternary edges within the local neighborhood of 3D block, when combined with proposed 3D zigzag sampling structure, increases the discriminativeness of the descriptor. The various angular variations of proposed 3D zigzag sampling structure catches more frequent variations in local texture patterns. A quantization and fusion based technique is proposed for reducing the feature dimensions. In order to evaluate the retrieval performance of the proposed descriptor, we conducted experiments using NEMA-CT and TCIA-CT image databases. The experimental results demonstrated that $3D\text{-}LOZTCoFP$ obtained the state of the art performance in comparison with existing techniques such as LBP, CSLBCoP, LDEP, LWP, LBDP, LBDisP, CoCSLBP, SS-3D-LTP and 3D-LTCoP.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical statement

This article does not contain any studies with human participants or animals performed by any of the authors.