X-ray Image Classification Using Random Forests with Local Wavelet-Based CS-Local Binary Patterns

Abstract

This paper presents a fast and efficient method for classifying X-ray images using random forests with proposed local wavelet-based local binary pattern (LBP) to improve image classification performance and reduce training and testing time. Most studies on local binary patterns and its modifications, including centre symmetric LBP (CS-LBP), focus on using image pixels as descriptors. To classify X-ray images, we first extract local wavelet-based CS-LBP (WCS-LBP) descriptors from local parts of the images to describe the wavelet-based texture characteristic. Then we apply the extracted feature vector to decision trees to construct random forests, which are an ensemble of random decision trees. Using the random forests with local WCS-LBP, we classified one test image into the category having the maximum posterior probability. Compared with other feature descriptors and classifiers, the proposed method shows both improved performance and faster processing time.

Introduction

There has been a considerable increase in the number and the size of digitalised medical images obtained from computerised medical devices, necessitating the use of medical image management systems. Systems such as the Picture Archiving Communication System and Digital Imaging and Communications in Medicine have attracted the attention of researchers in the fields of computer networking, image processing and database systems. However, these systems do not support image analysis, and as a consequence, a large number of medical images are required to be classified manually and then annotated by doctors and medical experts. Manual classification of a large number of images is labour intensive, repetitive and requires a highly trained expert. In addition, classification results are not always reliable because of experimental conditions, variable image quality and human subjectivity or tiredness, which can lead to misclassifications [1].

Thus, to overcome the limitations of manual classification, various content-based medical image classification and retrieval methods have been proposed over the last few decades. Unlike manual classification and text-based retrieval, content-based methods index images using colour, texture, shape and sound, which then form the basis for classification and retrieval [2]. However, content-based image retrieval in a massive-image database without image classification is a considerable computational burden because of the complexity of the operation. Therefore, image classification is important for reducing retrieval time and improving accuracy in image retrieval.

Several medical images such as X-ray, MRI, CT and PET have a dark background and bright foreground. Thus, it is necessary to develop a new classification algorithm different from that applied to natural images.

Toward developing advanced X-ray image classification, Bhattacharya et al. [3] presented a learning-based framework for medical image retrieval by linearly combining multi-class support vector machine (MSVM) and fuzzy c-mean clustering techniques. Here, a fusion-based similarity matching function uses the membership scores obtained from the learning algorithm to retrieve the images most similar to the query image. However, fuzzy c-mean clustering is sensitive to the initial value of clusters, and the weights for linear combinations depend on a heuristic method.

Avni et al. [4] proposed an X-ray image categorisation and retrieval system using local patch representation of the image content, a bag-of-features approach for defining image categories and a kernel-based support vector machine (SVM). However, because this method extracts features from all local patches, SVM is not suitable when a feature has high-dimensionality as a result of computational complexity.

Jeanne et al. [5] investigated the performance of five different types of visual features in a SVM-based learning framework for classifying X-ray images into classes corresponding to body parts. This method divides the images into small equal-sized non-overlapping regions and concatenates the local binary pattern (LBP) histograms extracted from each region into a single histogram. In their experiments, the LBP produced good class-specific accuracies and good global accuracy. However, because this method extracted 59 LBP patterns from all 4 × 4 regions, the dimension of the concatenated histogram was 944. Therefore, SVM classification is also computationally complex.

Pourghassem and Ghassemian [6] proposed a hierarchical medical image classification method, including two levels using a set of various shape and texture features. In each level of the hierarchical classifier, they created homogenous classes from overlapping classes in the database by using a merging scheme and multilayer perception classifiers. At each level, the merged class of the previous level is divided into several classes again using multilayer perception classifiers. However, the classification performance is variable, and the merging process can be carried out continuously, depending on the desirable value (T).

Shim et al. [7] proposed an algorithm for X-ray image classification and retrieval using MSVM with an ensemble feature vector by combining a colour structure descriptor (CSD) based on the Harris corner detector and an edge histogram descriptor of the image. Even though this method shows good classification performance, CSD is sensitive to the noise and weights for ensemble combination, depending on the heuristic method.

Kim et al. [8] proposed a novel algorithm for X-ray image classification to improve the classification computation time by using LBP with random forests. To classify an X-ray image, modified LBP descriptors are extracted from local grids. Next, these extracted feature descriptors are applied to random forests, yielding a probability histogram for each tree. Then, each probability histogram is combined into one histogram, and the class with the highest probability is chosen. However, this method has a somewhat lower performance than other methods that use the frequency domain [9, 10] since LBP descriptors were extracted from the original image intensity.

To improve the classification performance with respect to computation time and based on our previous research [8], this study proposes a novel X-ray image classification method combining new local wavelet-based centre symmetric LBP (WCS-LBP) with random forests. To classify X-ray images, we first extract WCS-LBP features from local parts of wavelet transformed sub-images to describe wavelet-based texture characteristics. Then we extract local WCS-LBP descriptors and apply them to decision trees to construct random forests, which are an ensemble of random decision trees. Using random forests with local WCS-LBP, a test image is classified into the category with the highest probability.

The remainder of this paper is organised as follows. “Feature Extraction Using Wavelet-Based CS-LBP” describes the feature extraction algorithm for classifying X-ray images using the proposed local WCS-LBP. “Image Classification Using Random Forests” introduces the image classification method using random forests. “Experimental Results” evaluates the accuracy and applicability of the proposed classification method based on experiments, and in “Conclusion”, we present some final conclusions and areas for future work.

Feature Extraction Using Local Wavelet-Based CS-LBP

Various parameters can be used to characterise images, such as colour, texture, luminance and shape. However, as X-ray images consist of two regions (an interesting bright foreground region and a meaningless dark background region) and have different shapes according to body parts, texture is the most appropriate feature for describing the contents of an X-ray image.

In this section, we introduce the recently proposed LBP texture operator, which has been used successfully in various computer vision applications. Then we introduce centre symmetric LBP (CS-LBP), which produces a rather short dimension with similar performance to LBP. After that, we present our proposed local wavelet-based CS-LBP.

Local Binary Pattern

Among the many existing texture operators, the LBP texture operator [11] has been successfully used in various computer vision applications, such as face recognition [12, 13], background modelling [14] and text detection [9], because it is robust against illumination changes, very fast to compute and does not require many parameters [15]. LBP describes grey-scale local texture of the image with low computational complexity by using a simple method. The original LBP descriptor forms different patterns based on the number of pixels by thresholding a specific range of neighbour sets with the centre grey-scale intensity value.

As shown in Fig. 1, the neighbour set is defined by the number of pixels P centred at g_c and with radius R. The distance between the centre pixel (g_c) and its neighbourhood () is simply calculated as a binary number and then the unique LBP code is allocated using Eqs. 1 and 2. Here the function s(x) produces 1 if the difference is above the threshold, and produces 0 otherwise. The binomial factor 2^p is assigned to the spatial structure of the local texture. When the neighbour set is defined as P = 8, R = 1.0, LBP can produce up to 256(2⁸) different patterns.

1

2

Fig. 1.

The neighbour sets of LBP [11]

However, since not all patterns provide good discrimination for texture classification, 59 uniform patterns are usually used to express the texture characteristics.

As a result, the LBP descriptor is considered to provide flexible code order modification, which varies the form of patterns such as centre symmetric local binary patterns (CS-LBP) [15] and multidirectional binary patterns [16].

Centre Symmetric Local Binary Pattern

As we mentioned earlier, even though LBP is widely used as a texture operator, it produces rather long histograms. To address the problem, Heikkilä et al. [15] proposed CS-LBP. CS-LBP uses a modified scheme of comparing neighbouring pixels of the original LBP to simplify the computation, while keeping the characteristics such as tolerance against illumination changes and robustness against monotonic grey-level changes. CS-LBP is different to LBP in that it compares centre symmetric pairs of pixels against a centre pixel, rather than comparing each pixel with the centre, as shown in Fig. 2. This halves the number of comparisons for the same number of neighbours and produces only 16(2⁴) different binary patterns. The concept is similar to a gradient operation, because it calculates the difference between pairs of opposite pixels in a neighbour. This relation leads to the methodology of applying CS-LBP to the wavelet domain.

Fig. 2.

LBP and CS-LBP features for a neighbourhood of 8 pixels

Local Wavelet-Based CS-LBP

CS-LBP in a grey image shows good category classification results with a shorter feature dimension than LBP. However, some researchers [9, 10] still prefer to use LBP in the wavelet domain. This is because wavelet transforms have a good spatial frequency localization property, and so they can preserve both spatial information and gradient information of an image. In addition, LBP pattern extraction on the wavelet domain can reduce the noise, because LBP and CS-LBP are suitable for modelling repetitive texture, which means they are sensitive to random noise in uniform image areas.

Ji et al. [9] extracted LBPs for text detection from the coefficients of three high-pass filtered sub-images (LH, HL and HH) of the pyramid Haar wavelet. In this method, they used a threshold-restricted 8-neighbourhood LBP to filter out gradual illumination variance and extract LBP from every scale pyramid. Even though this method used only high-pass filtered sub-images, looking for abrupt illumination changes, the low-pass filtered sub-image (LL) is also important, because it allows good energy compaction of the original image with relatively lower noise. Furthermore, since the X-ray image is generally captured in a restricted environment, it only has a dark background and bright foreground, without abrupt illumination changes.

Du et al. [10] decomposed the handwriting image into a series of wavelet sub-images and then calculated the LBP codes of these sub-images using the LBP operator. Using this method, they extracted the local LBP histograms from all of the multi-scale sub-images after S-level wavelet decomposition, and then concatenated the LBP codes as one LBP histogram for writer identification.

However, the dimension of the final LBP histogram can be larger depending on the wavelet level. For example, if the level of wavelet decomposition is 2, this generates six high-pass filtered sub-images and one low-pass filtered sub-image. When the neighbour set is defined as P = 8, LBP produces 256 different patterns for each sub-image, and the final dimension of the LBP histogram is 1,792 [256 × 7(6 + 1)]. Moreover, wavelet-based LBP is demanding in terms of computation time, needing eight subtractions, eight multiplications and eight summations for each pixel at each of the seven sub-images. Therefore, if the size of an image is 256 × 256, the number of operations is 33,554,432 [(8 × 8 × 8) × (low-pass filtered sub-image: 64 × 64 × 1 + 2-level high-pass filtered sub-images: 64 × 64 × 3 + 1-level high-pass filtered sub-images: 128 × 128 × 3)].

In this paper, we extract a CS-LBP rather than a LBP histogram from all multi-scale sub-images, including low-pass filtered sub-images, after two-level wavelet decomposition. For wavelet decomposition, we use Daubechies 4 filter because, while it has compaction support, it is continuous, yields better frequency resolution than the Haar wavelet and achieves better spatial resolution than other wavelets [17].

In general, since the X-ray image has strong edge distribution in the horizontal, vertical and diagonal directions, the three high-pass filtered sub-images (LH, HL and HH) have important properties when classifying image categories. However, since the original LBP and CS-LBP are designed to compare only centre symmetric pairs for pixel intensity, comparing the overall symmetry of high-pass filtered coefficients, regardless of their gradient directions, shows better performance than individual comparing, as shown in the experimental graph (Fig. 6).

Fig. 6.

Performance comparison between five different combinations: a precision, b recall

According to the results of the experiment, seven sub-images (, , , , , , ) are extracted after the two-level wavelet transform of an image, and all high-pass filtered sub-images of each level are combined as one wavelet energy, W¹ and W², using the following equation.

3

In Eq. 3, W¹ represents the wavelet energy of one-level and W² represents the wavelet energy of two-level energy.

The major problems when classifying medical X-ray images are high overlapping between image classes (i.e. hand is connected with carpal joint) in the very large database, and high intra-class variability within some of the classes [6]. Therefore, we divide the sub-images into 4 × 4 local grids, and extract local WCS-LBPs, incorporating spatial local information into WCS-LBP descriptors, to solve the problem of overlapping classes.

First, each wavelet energy and low-pass filtered sub-image is divided into 4 × 4 non-overlapping sub-regions. After we calculate local WCS-LBP codes with P = 8 from each wavelet energy W¹, W² and low-pass filtered sub-image , we can represent each local WCS-LBP distribution for one region of the sub-image on a histogram with 16 bins. The final histogram for each sub-image is generated by concatenating the local histograms. Since there are 16 sub-regions, this means we generate a total of 16 × 16 = 256 histogram bins for one low-pass filtered sub-image. Using the same method, we generate additional local WCS-LBP histograms for the other two wavelet energies. Finally, we concatenate all of the histograms to create the final local WCS-LBP histogram, as shown in Fig. 3.

Fig. 3.

Representation of final local WCS-LBP histogram generation. Local WCS-LBP histograms are generated from one low-pass filtered sub-image and the other two wavelet energies. Then, all histograms are concatenated to create the final local WCS-LBP histogram

The final dimension of the local WCS-LBP histogram is 768 [(16 × 3) × 16 sub-regions]. The concatenated final local WCS-LBP histogram is normalised to unit length using the Gaussian normalisation method [18].

The computational time for local WCS-LBP is demanding, including four subtractions, four multiplications and four summations for one pixel at one low-pass filtered sub-image and two wavelet energies. Therefore, if the size of an image is 256 × 256, the number of operations is 1,572,864 [(CS-LBP: 4 × 4 × 4) × (low-pass filtered sub-image: 64 × 64 + 2-level wavelet energy: 64 × 64 + 1-level wavelet energy: 128 × 128)], and it is as much as 21 times faster than that using the normal WLBP method proposed by Du et al. [10].

Image Classification Using Random Forests

After the feature extraction, the images are classified into predefined classes using pattern classifiers with local WCS-LBP descriptors. A multi-class support vector machine classifier is a reasonable choice for general classification due to its high performance and accuracy. However, MSVM is not suitable when the feature has high-dimensionality and the database contains over 1,000 images, due to computational complexity. Therefore, the high-dimensional local WCS-LBP feature vector that is extracted from wavelet sub-images with 768 dimensions might make training tasks very time consuming.

In this paper, we have chosen to classify images using random forests, as proposed by Breiman [19]. This classifier has been shown to be effective in a large variety of high-dimensional problems, with high computational performance and accuracy.

A random forest is a decision tree ensemble classifier, with each tree grown using some type of randomization. Random forests have a capacity for processing huge amounts of data with high training speeds, based on a decision tree. The structure of each tree in the random forest is binary and is created in a top–down manner, as shown in Fig. 4.

Fig. 4.

Classification process using local WCS-LBP with trained random forests. In this example, the test image is classified into the second class because it has a maximum posterior probability of 1.0

In the training procedure, the random forest starts by choosing a random subset I′ from the local WCS-LBP training data, I. At the node n, the training data I_n is iteratively split into left and right subsets I_l and I_r by using the threshold, t, and split function, f(v_i), for the feature vector, v, using Eq. 4. The threshold, t, is randomly chosen by the split function, f(v_i), in the range

4

Then, several candidates are randomly created by the split function and threshold at the split node. Among those, the candidate that maximises the information gain about the corresponding node is selected. The information gain, ∆E, is easily calculated by entropy estimation, according to Eq. 5.

5

In Eq. 5, E(I) is the Shannon entropy of the classes in the set of training images I.

There are two conditions that can end the iterative training. The first condition occurs if there is no more information gain possible. The second condition occurs if the training process reaches a leaf node that is at the maximum depth of the tree. Consequently, a leaf node has a posterior probability and the class distributions, p(c|n), are estimated empirically as a histogram of the class labels, c_i, of the training examples, i, that reached node n.

As shown in Fig. 4, when classifying the test image, the local WCS-LBP histogram of the test image is created over the whole wavelet transform. The test image is used as input to the trained random forest. The final class distribution is generated by ensemble (arithmetic averaging) of each distribution of all trees L = (l₁, l₂,…, l_r), using Eq. 6.

In Eq. 6, T is the number of trees, and we choose c_i as the final class of an input image if p(c_i|L) has the maximum value.

6

In Fig. 4, the test image is classified into the second class because it has the maximum posterior probability.

Using an ensemble of distributions of trees trained on only small random subsets of the data helps to speed up training and reduce overfitting. The random forests produce a limiting value of the generalisation error, but do not overfit as more trees are added [19]. In fact, according to use of the strong law of large numbers, they always converge so that overfitting is not a problem.

The important parameters of random forests are the depth of tree and the number trees, T. Bosch et al. [20] propose that increasing the depth of the tree increases performance, although this also increases the memory required to store the trees during experiments. In our experiments, our random forests showed the best classification performance in terms of accuracy and computational time with a maximum tree depth of 20, and number of trees set to 120. Figure 5 shows the results of the experiments.

Fig. 5.

Processing time and precision for an increasing number of trees: a processing time increases linearly as the number of trees increases and b precision became saturated when the number of trees reaches 120

Experimental Results

We performed experiments using a set of 2,400 (30 categories) X-ray images from IRMA 2007 (Image Retrieval in Medical Applications, http://ganymed.imib.rwth-aachen.de/irma), covering a wide variety of body parts, ranging from head to toes. We used 900 images for training, and 1,500 images for testing the 30 categories. Table 1 shows the image classes and the number of images per class when training and testing in our experiments.

Table 1.

Image classes and number of images per class for training and testing

Class	body part	No. of training data	No. of testing data	Class	Body part	No. of training data	No. of testing data
1	Cranium	30	50	16	Chest	30	50
2	Facial cranium	30	50	17	Left breast	30	50
3	Neuro-cranium	30	50	18	Right breast	30	50
4	Cervical spine	30	50	19	Abdomen	30	50
5	Thoracic spine	30	50	20	Pelvis	30	50
6	Lumbar spine	30	50	21	Toe	30	50
7	Finger	30	50	22	Left foot	30	50
8	Left hand	30	50	23	Right foot	30	50
9	Right hand	30	50	24	Left ankle joint	30	50
10	Left carpal joint	30	50	25	Right ankle joint	30	50
11	Right carpal joint	30	50	26	Patella	30	50
12	Left elbow	30	50	27	Left knee	30	50
13	Right elbow	30	50	28	Right knee	30	50
14	Left humero-scapular joint	30	50	29	Left hip	30	50
15	Right humero-scapular joint	30	50	30	Right hip	30	50

First, we compared the processing time and precision when training, as the number of trees increased. We varied the number of trees from 10 to 300. As shown in Fig. 5, precision became saturated when we used more than 120 trees. Moreover, because the processing time increases linearly as the number of trees increases, we fixed the number of trees at 120.

In this paper, we used local WCS-LBP with a low-pass filtered sub-image and two wavelet energies, rather than seven sub-images after the two-level wavelet transform of an image. Therefore, the final feature dimension is 768 [(16 × 3) × 16 sub-regions]. To verify our choice of sub-images, we compared the classification performance of the proposed local WCS-LBP with the following four other feature combinations:

• Combination A ([9]): LBP (256 patterns) combination with individual sub-images, ,, , , and , excluding two-level low-pass filtered sub-image . All sub-images are divided into 4 × 4 sub-regions. The feature dimension of this method is 24,576 [(256 × 3 + 256 × 3) × 16 sub-regions].

• Combination B([10]): LBP (256 patterns) combination with individual sub-images, ,, , , and , including two-level low-pass filtered sub-image . All sub-images are divided into 4 × 4 sub-regions. The feature dimension of this method is 28,672 [(256 × 4 + 256 × 3) × 16 sub-regions].

• Combination C: CS-LBP (16 patterns) combination with individual sub-images, ,, , , and , excluding two-level low-pass filtered sub-image . All sub-images are divided into 4 × 4 sub-regions. The feature dimension of this method is 1,536 [(16 × 3 + 16 × 3) × 16 sub-regions].

• Combination D: CS-LBP (16 patterns) combination with individual sub-images, ,, , , and , including two-level low-pass filtered sub-image . All sub-images are divided into 4 × 4 sub-regions. The feature dimension of this method is 1,792 [(16 × 4 + 16 × 3) × 16 sub-regions].

To validate the effectiveness of the proposed method (local WCS-LBP with random forests), we compared the classification precision and recall with that of the other four combinations. In all the experiments, we measured the performance using the average classification precision and recall on 30 classes. As shown in Fig. 6, the overall performance of our proposed approach outperformed that of the other four combinations based on precision percentages of 84.33%, 87.80%, 86.82%, 89.70% and 93.10%, and recall percentages of 81.20%, 85.74%, 85.00%, 88.94% and 89.43%.

In addition, we also compared the classification performance of our proposed method (local WCS-LBP with random forests) against the combination of a local WCS-LBP with MSVM, because the MSVM classifier is known to show reasonable results for general classification problems. As can be seen in Fig. 7, the classification performance of the local WCS-LBP and MSVM method shows 89.07% for precision and 86.40% for recall. In contrast, the local WCS-LBP with random forests method showed an average precision and recall performance of 93.10% and 89.43%, respectively, approximately 4% and 3% better than the MSVM method.

Fig. 7.

Performance comparison between local WCS-LBP with MSVM and local WCS-LBP with random forests: a precision, b recall

Apart from precision and recall, our experiments also showed that the WCS-LBP with random forests method performed better than the WCS-LBP with MSVM method when measuring the speeds of training and testing. The speed testing was conducted using an Intel® Core™ i7 PC with Windows® 7 operating system environment.

Figure 8 shows the training and testing times for the two methods. Note that we have omitted the time it took to load the data. As shown in Fig. 8, the processing speed for training and testing of the proposed method is approximately 36.8 and 66 times faster than the MSVM method using the same training and testing images. The reason for the fast training time is that random forests use non-iterative training, and training can be completed in a fixed number of operations. In particular, random forests reduced the computational time for testing regardless of increasing the number of test images. However, the computational time of the MSVM method increases linearly as the number of test images increases. As a result, even though the MSVM method showed good detection performance, it is not suitable for real-time applications due to its computational complexity when the database contains over 1,000 images.

Fig. 8.

Training and testing time comparison between local WCS-LBP with MSVM and local WCS-LBP with random forests

Conclusion

In this paper, we demonstrated that random forests with a proposed local WCS-LBP to improve medical image classification performance, especially X-ray images, and reduced training and testing time significantly when compared to a multi-class SVM using the same feature descriptor.

To classify medical images, we first extracted local WCS-LBP descriptors from local parts of the images to describe the wavelet-based texture characteristic. Then we applied the extracted feature vector to decision trees to construct random forests, which are an ensemble of random decision trees. Using the random forests with local WCS-LBP, we classified one test image into the category having the maximum posterior probability. The experimental results using CLEF-Med2007 images showed that our algorithm could indeed improve classification performance compared to other feature combinations or other classification methods.

In future works, we plan to apply our classification algorithm to other medical image classification, such as cell images, CT images and MRI images. Furthermore, we need to develop optimal standards for random forests, such as split and thresholds at each node to improve the classification performance.