Texture feature ranking with relevance learning to classify interstitial lung disease patterns

Abstract

Objective

The generalized matrix learning vector quantization (GMLVQ) is used to estimate the relevance of texture features in their ability to classify interstitial lung disease patterns in high-resolution computed tomography images.

Methodology

After a stochastic gradient descent, the GMLVQ algorithm provides a discriminative distance measure of relevance factors, which can account for pairwise correlations between different texture features and their importance for the classification of healthy and diseased patterns. 65 texture features were extracted from gray-level co-occurrence matrices (GLCMs). These features were ranked and selected according to their relevance obtained by GMLVQ and, for comparison, to a mutual information (MI) criteria. The classification performance for different feature subsets was calculated for a k-nearest-neighbor (kNN) and a random forests classifier (RanForest), and support vector machines with a linear and a radial basis function kernel (SVMlin, SVMrbf).

Results

For all classifiers, feature sets selected by the relevance ranking assessed by GMLVQ had a significantly better classification performance (p < 0.05) for many texture feature sets compared to the MI approach. For kNN, RanForest, and SVMrbf, some of these feature subsets had a significantly better classification performance when compared to the set consisting of all features (p < 0.05).

Conclusion

While this approach estimates the relevance of single features, future considerations of GMLVQ should include the pairwise correlation for the feature ranking, e.g. to reduce the redundancy of two equally relevant features.

1. Introduction

The ability to detect and classify pathological patterns in medical images is a key challenge in computer-aided diagnosis systems that consist of a feature extraction and machine learning task. In high-resolution computed tomography (HRCT) images of the chest, the classification of pathological patterns of interstitial lung diseases (ILDs) is a complicated classification task which requires substantial expertise. Different texture features and classifiers have been proposed to achieve the goal of distinguishing healthy lung patterns from disease patterns [1–3]. A reliable, consistent and reproducible computer-aided diagnosis could improve the radiologists efficiency and avoid surgical lung biopsies for some patients.

One approach to design an image-based, automated detection tool is to extract texture features from regions of interest (ROI) and then use supervised machine learning techniques to classify the ROI as healthy or pathological. Too many features deteriorate the classification performance of the machine learning algorithms summarized as ”curse of dimensionality’ [4]. In addition, irrelevant features degrade the classification performance. Because of these negative effects, it is common to precede the learning with a feature selection stage that strives to eliminate all but the most relevant features.

This study addresses the question how the texture feature can be identified that provide the best information from the underlying lung HRCT images for a specific (classification) problem. This objective can be addressed by estimating the global relevance of each individual texture feature relative to all other texture features in the set. The resulting relevance allows the ranking of texture features for a specific classification problem. The ranks in relevance can then be used (1) to compare the performance of texture features and (2) to select only a subset with the texture features most important for the classification task thereby reducing the dimensionality of the data set and avoiding any over-fitting [4]. With an independent feature selection step, many proposed texture feature approaches can be tested and the best set can then be used for the actual classification task.

The generalized matrix learning vector quantization (GMLVQ) [5] offers such a relevance estimation for features in a supervised pattern classification problem. The GMLVQ algorithm provides a discriminative distance measure of relevance factors, which can account for pairwise correlations between different texture features and their importance in classification using real-world data [6]. From lung HRCT images, texture features were extracted, ranked and selected according to the relevances obtained by GMLVQ. For comparison, features were also selected by a mutual information criteria. The selected features subsets were then evaluated in their ability to classify the lung patterns.

2. Related work

The learning vector quantization (LVQ) approach [7] and its variants (e.g. LVQ3) [8, 9] constitute a popular family of supervised classifiers based on prototypes in the feature space. Specifically, LVQ3 shifts the decision borders toward the Bayesian limits and applies an correction term that ensures that the prototypes approximate the corresponding class distribution. Relevance learning schemes quantify the importance of extracted features for a certain classification task in heterogenous data sets [9]. The GMLVQ is an extension of this concept that provides a matrix of relevances that accounts for correlations between different features. A similar approach is the distinction sensitive LVQ (DSLVQ) [10], which adapts LVQ3 for the weighting factors according to plausible heuristics. In contrast, the GMLVQ update constitutes a stochastic gradient descent on a cost function [8].

Several schemes for adaptive distance learning exist, e.g. the large margin nearest neighbor (LMNN)[11]. A comparison between the LMNN technique and the GMLVQ approach on the basis of a content based image retrieval application revealed that both methods can reach a similar performance; however the computational cost for GMLVQ training is typically lower [12, 13].

Several authors investigated the performance of certain groups of texture features and their combination with classifiers [1–3, 14–18] in lung HRCT images as well as other types of images [19, 20]. In addition, various feature selection schemes based on mutual information were proposed [21, 22].

In this work, the feature selection based on the GMVLQ relevances is compared with a mutual information criteria using a real-world data set.

3. Generalized matrix LVQ

LVQ [7] constitutes a particularly intuitive classification algorithm, which represents data by means of prototypes. LVQ itself constitutes a heuristic algorithm, hence extensions have been proposed for which convergence and ability to represent the data structure can be guaranteed [9, 23]. One particularly crucial aspect of LVQ schemes is the dependency on the underlying metric, usually the Euclidean metric, which may not suit the underlying data structure. Therefore, general metric adaptation has been introduced into LVQ schemes [9, 24]. Recent extensions parameterize the distance measure in terms of a relevance matrix, the rank of which may be controlled explicitly. The algorithm suggested in [5] can be employed for linear dimension reduction and visualization of labeled data.

We consider training data x⃗_i ∈ ℝ^N, i = 1…S with labels y_i corresponding to one of C classes respectively. The aim of LVQ is to find m prototypes w⃗_j ∈ ℝ^N with class labels c(w⃗_j) ∈ {1,…,C} such that they represent the classification as accurately as possible. A data point x⃗_i is assigned to the class of its closest prototype w⃗_j, where d(x⃗_i, w⃗_j) ≤ d(x⃗_i, w⃗_l) for all l ≠ j and d usually denotes the squared Euclidean distance

$$d({\overrightarrow{x}}_i,{\overrightarrow{w}}_j)={({\overrightarrow{x}}_i-{\overrightarrow{w}}_j)}^{\top }({\overrightarrow{x}}_i-{\overrightarrow{w}}_j).$$ (1)

Consider a function μ(x⃗_i),

$$\mu ({\overrightarrow{x}}_i)=\frac{d({\overrightarrow{w}}_J,{\overrightarrow{x}}_i)-d({\overrightarrow{w}}_K,{\overrightarrow{x}}_i)}{d({\overrightarrow{w}}_J,{\overrightarrow{x}}_i)+d({\overrightarrow{w}}_K,{\overrightarrow{x}}_i)},$$ (2)

where w⃗_J denotes the closest prototype with the same class label as x⃗_i (c(w⃗_J) = y_i), and w⃗_K is the closest prototype with a different class label (c(w⃗_K) ≠ y_i). Let Φ(·) denote a monotonically increasing function; then, the general scheme of generalized LVQ (GLVQ) [8] adapts prototype locations by minimizing the cost function

$$E_{\text{GLVQ}}=\sum _{i=1}^S\mathrm{\Phi }(\mu ({\overrightarrow{x}}_i)).$$ (3)

In this contribution, Φ was chosen to be the identity function, Φ(x) = x; other choices include e.g. the logistic function [9]. This cost function aims at an adaptation of the prototypes, such that a large hypothesis margin is obtained and a reliable and robust classification (see [25]) is achieved. A learning algorithm can be derived from the cost function EGLVQ by means of a stochastic gradient descent as shown in [9, 23].

Matrix learning in GLVQ (GMLVQ) [24, 25] substitutes the usual squared Euclidean distance d by a more advanced dissimilarity measure which contains an adaptive dissimilarity measure, thus resulting in a more complex and better adaptable classifier. In [5], it was proposed to choose the dissimilarity as

$$d_j^{\mathrm{\Lambda }}({\overrightarrow{w}}_j,{\overrightarrow{x}}_i)={({\overrightarrow{x}}_i-{\overrightarrow{w}}_j)}^{\top }\mathrm{\Lambda }({\overrightarrow{x}}_i-{\overrightarrow{w}}_j)$$ (4)

with an adaptive, symmetric and positive semi-definite matrix Λ ∈ ℝ^N×N. The dissimilarity measure Eq. (4) possesses the shape of a Mahalanobis distance. Note, however, that the precise matrix is determined in a discriminative way according to the given labeling, such that severe differences from the standard Mahalanobis distance based on correlations can be observed. By setting

$$\mathrm{\Lambda }={\mathrm{\Omega }}^{\top }\mathrm{\Omega },$$ (5)

with Ω ∈ ℝ^M×N with M ≤ Nsemi-definiteness and symmetry is guaranteed. Optimization takes place by a stochastic gradient descent of the cost function E_GLVQ in Eq. (3), with the distance measure d substituted by dΛ (see Eq. (4)).

The training takes place as an iterative learning scheme picking a sample x⃗_i in every step t from the randomized labeled training set and performing an update of both the prototypes w⃗_K,J and the matrix. The update of the nearest prototype w⃗_J with the same class label like x⃗_i and the nearest prototype w⃗_K with a different class label is given by

$${\overrightarrow{w}}_J^{\text{new}}={\overrightarrow{w}}_J+{ϵ}_1·{\mathrm{\gamma }}^{+}·2\mathrm{\Lambda }({\overrightarrow{x}}_i-{\overrightarrow{w}}_J)$$ (6)

$${\overrightarrow{w}}_K^{\text{new}}={\overrightarrow{w}}_K+{ϵ}_1·{\mathrm{\gamma }}^{-}·2\mathrm{\Lambda }({\overrightarrow{x}}_i-{\overrightarrow{w}}_K),$$ (7)

$$\text{with}{\mathrm{\gamma }}^{+}=\frac{2·d_K^{\mathrm{\Lambda }}}{{(d_J^{\mathrm{\Lambda }}+d_K^{\mathrm{\Lambda }})}^2}\text{and}{\mathrm{\gamma }}^{-}=\frac{-2·d_J^{\mathrm{\Lambda }}}{{(d_J^{\mathrm{\Lambda }}+d_K^{\mathrm{\Lambda }})}^2}.$$ (8)

The corresponding update of the matrix reads

$${\mathrm{\Omega }}_{mn}^{\text{new}}={\mathrm{\Omega }}_{nm}-{ϵ}_2·({\mathrm{\gamma }}^{+}·\frac{\partial d_J^{\mathrm{\Lambda }}}{\partial {\mathrm{\Omega }}_{mn}}+{\mathrm{\gamma }}^{-}·\frac{\partial d_K^{\mathrm{\Lambda }}}{\partial {\mathrm{\Omega }}_{mn}})$$ (9)

$$\frac{\partial d_L^{\mathrm{\Lambda }}}{\partial {\mathrm{\Omega }}_{mn}}=2{[\mathrm{\Omega }({\overrightarrow{x}}_i-{\overrightarrow{w}}_L)]}_m(x_{i,n}-w_{L,n})$$ (10)

with L ∈ {J, K}. After each training step t the matrix is normalized to ∑_i[Λ]_ii = 1 in order to prevent degeneration to 0. An additional regularization term in the cost function proportional to −ln(det(ΩΩ^⊤)) can be used to enforce full rank M of the relevance matrix and prevent over-simplification effects, see [26]. The learning stops after a maximal number of epochs (sweep through the shuffled training set) is reached. At the end of the learning process the algorithm provides a set of prototypes w⃗_j, their labels c(w⃗_j), and a task specific discriminative distance d^Λ.

The diagonal elements Λ_ii of the dissimilarity matrix can be interpreted as overall relevances of every feature i for the classification. The off-diagonal elements Λ_ij with j ≠ i weight the pairwise correlations between features i and j. High absolute values in the matrix denote highly relevant features, while values near zero can be seen as less important for the classification accuracy. So the GMLVQ approach offers typical representatives of the classes in form of prototypes, a discriminative distance measure and feature relevance information suitable for the specific classification task.

The cost function of GMLVQ is non convex and, in consequence, different local optima can occur, which lead to different results and subsequent data visualizations. The non-convexity of the cost function is mainly due to the discrete assignments of data points to one prototype, which may not be unique for real-world data sets with overlapping classes. In the experiments, different assignments and, in consequence, different data representations could be observed, where these representations focus on different relevant facets of the given data sets.

The choice Ω ∈ ℝ^M×N with M ≤ N transforms the data locally to an M-dimensional feature space. It can be shown that the adaptive distance d^Λ(w⃗_j, x⃗_i) in Eq. (4) equals the squared Euclidean distance in the transformed space under the transformation x⃗ ↦ Ωx⃗, because d^Λ(w⃗_j, x⃗_i) = (Ωx⃗_i − w⃗_j)². The target dimension M must be chosen in advance by intrinsic dimension estimation or according to available prior knowledge. For visualization purposes, usually a value of two or three is appropriate.

4. Experiment

4.1. Data

For each of the 14 patients with known occurrence of ILD patterns, a stack of 70 axial images reconstructed with a lung kernel from HRCT of the chest was assessed by an experienced radiologist. The resulting images are comparable to classical HRCT with the exception that not only reconstructions of a small number of slice positions are available. A morphological pattern related to a heterogenous group of ILDs is the so-called honeycombing. All images have a slice thickness of 1mm, an in-plane pixel size of 0.69×0.69 mm, and a slice distance of 5 mm. In this data set, 964 two-dimensional, squared ROIs with an edge length of 23 pixels were defined in 608 healthy and 356 pathological, honeycombing lung patterns (Fig. 1). In each of these ROIs, the set of the following vectors of texture features was calculated.

Figure 1.

ROIs with HRCT image patterns showing examples for healthy lung tissue (upper 2 rows) and pathological lung tissue (lower 2 rows).

4.2. Texture features

Gray level co-occurrence matrices (GLCM) [19] are second order histograms that estimate the joint probability P(g_i, g_j) for the occurrence of two pixels with the gray levels g_i and g_j for a given pixel spacing and direction. In each ROI, the gray levels were rescaled (min-max-scaling) to 128 gray level values and texture features were calculated for n_dir = 4 offset directions (0°, 45°, 90°, 135°) and five pixel spacings s_p = {1, 2, 3, 4, 5}. Pixels pairs were excluded from the GLCM if the neighbor pixel was outside the ROI. To achieve rotation invariant features, the GLCM of the four directions were summed to one matrix. From those isotropic GLCMs, 13 statistical features f_i(s_p) (i = 1,…,13) were calculated resulting in a set of 65 GLCM texture features. After the calculation, all features were standardized to zero mean and unit variance.

4.3. Mutual information

For a comparison with the estimated feature relevance by GMLVQ, an information theoretic approach to feature selection is used. Mutual information (MI) [4] is a measure of general independence between random variables. For two random variables X and Y, the MI is defined as

$$I(X,Y)=H(X)+H(Y)-H(X,Y),$$ (11)

where entropy H(·) measures the uncertainty associated with a random variable. The MI I(X, Y) estimates how the uncertainty of e.g. X is reduced if Y has been observed. If X and Y are independent, their MI is zero.

For ROI data set in this study, the MI between the single texture features f_i(s_p) and the corresponding class labels y_i was calculated by approximating the probability density function of each variable using histograms P(·):

$$I(f_i(s_p),y_i)=\sum ^{n_c}\sum ^{n_f}P(f_i(s_p),y_i){\text{log}}_2\frac{P(f_i(s_p),y_i)}{P(f_i(s_p))P(y_i)}.$$ (12)

Here, the number of classes n_c = 2 was used; the number of histogram bins for the texture features n_f was determined adaptively according to

$$n_f={\text{log}}_{2}N+1+{\text{log}}_2(1+\kappa \sqrt{N/6}),$$ (13)

where κ is the estimated kurtosis and N the number of ROIs in the data set[21]. Hence, more histogram bins were assigned to non-Gaussian distributed features.

4.4. Classification performance

After the ranking of the texture features, different classifiers were used to test the classification performance on ranked features acquired by the GMLVQ algorithm and the MI approach. Three commonly used classifiers were investigated in this study: a k-nearest neighbor (kNN), support vector machines with a linear and a radial basis function kernel (SVMlin, SVMrbf, respectively), and random forests (RanForest) [27, 28].

Free Parameters of all classifiers were subject to optimization via cross validation in the training phase. In one iteration, the data set is randomly divided into a training set (70%) and a test set (30%); all ROIs of one patient are either included in the training set or the test set. The training set is used to estimate the relevance with GMLVQ and MI ranking and to determine the best classifier parameter in a 10-fold cross-validation. The optimized classifier model was then applied on the independent test set to calculate the accuracy of one iteration. This scheme was repeated 50 times resulting in an accuracy distribution for each texture feature sets and classifier. In addition to a feature set including all features, sets consisting of the best {1,…,20} features according the the GMLVQ and MI ranking were evaluated for their classification performance. A Wilcoxon signed-rank test was used to compare two accuracy distributions.

5. Results

5.1. Feature ranking

In Fig. (2), the ranking of the used texture feature is shown for the relevance measure obtained by GMLVQ and the MI estimation, and the 20 best features in each ranking scheme are highlighted. Both feature selection approaches identify f₈(1) as the most important feature to distinguish the two classes of ROI patterns. However, the second best feature is already different (f₁₁(1) for GMLVQ, f₈(2) for MI) and the overall set of the best 20 texture features is not identical as well. The best ranked texture features f₈ and f₁₁ define the ’sum of entropy’ and the ’difference entropy’, respectively [19]. For the GMLVQ algorithm, the off-diagonal elements of Λ are shown in Fig. (3), which weight the pairwise correlations between two features.

Figure 2.

Ranking of the used texture features for the relevance measure obtained by GMLVQ (upper panel) and the mutual information MI (lower panel). For the 50 iteration, the mean ranking is shown for each feature; the error bars indicate the corresponding standard deviation. The GMLVQ relevance is calculated from the diagonal elements Λ_ii of the dissimilarity matrix (Eq. 5). The 20 best feature in each ranking approach are marked with black bars. The set of texture features consists of the statistical measures derived from the GLCMs, f_i(s_p), for i = {1,…,13} and s_p = {1,…,5}.

Figure 3.

The matrix Λ with pairwise-correlations between the GLCM features. Dark and bright entries correspond to high negative and positive correlations, respectively. The diagonal elements (see Fig. 2) are set to zero to emphasize the off-diagonal elements.

For the classification task, it is interesting to investigate distribution of the feature vectors in the high-dimensional feature space. In Fig. (4), the feature space of the sets including the best 5 features as defined by the two ranking methods are represented by a two-dimensional embedding, where each data point is replaced with the corresponding ROI. The embedding was obtained with a principal component analysis (PCA) leading to the first and second principal components, that include the highest variation found in the data set through a coordinate transformation with the covariance matrix. Using this linear PCA projection, features in the MI ranking seem to be more correlated compared to the GMLVQ ranking (Fig. 4). In addition, the two images classes, healthy and diseased, can be better distinguished in the latter ranking approach.

Figure 4.

Two dimensional projections of feature spaces with a subset of corresponding lung patterns: the first and second principal component are shown for the feature space of the best 5 features for the mutual information (upper panel) and the GMLVQ (lower panel) ranking. The subimages show the complete distributions of the healthy (gray circles) and diseased lung patterns (black dots). These representations reveal the shape of the feature space, especially that the features in the set selected by GMLVQ are less correlated. Note that not all lung pattern images are shown to avoid overlapping.

5.2. Classification performance

In Fig. (5), the test accuracies obtained by the four classifiers are shown for different feature sets including all features and the best n features selected by GMLVQ and MI rankings. For all classifiers, feature sets selected by the relevance ranking assessed by GMLVQ had a significantly better classification performance for the best {2,…,18} texture features compared to the MI approach (p < 0.05). The other subsets defined by the GMLVQ and MI ranking had a similar performance. For kNN, RanForest, and SVMrbf, some of these feature subsets had a significantly better classification performance when compared to the set consisting of all features, e.g. the best {3,…,10} texture features (p < 0.05). Fore these three classifiers, the best performance was found with sets between 4 to 6 features defined by the GMLVQ ranking; the performance decreased with increasing numbers of features. For SVMlin classifier, the best level of performance was reached with 6 features and remained similar with increasing number of features.

Figure 5.

Classification performance for different features sets and two classifiers. Median test accuracies (error bars indicate the 20th and 80th percentiles) are shown for a set consisting of all features (diamond), and sets with the best n features (n = {1,…,20}) according to the n highest relevances obtained from GMLVQ (squares) and n highest estimations of mutual information MI (triangles). Circles mark significant differences between the two ranking methods; crosses mark a significant better performance of GMLVQ ranking compared to the set including all features (p<0.05).

6. Discussion

In this novel application of the GMLVQ algorithm, the relevance of texture features was estimated in their ability to classify healthy and diseased lung patterns in chest HRCT images. This supervised learning scheme incorporates a stochastic gradient descent on an appropriate error surface and provides a feature selection approach to determine the relevance of several input dimensions for a specific classification task. To this end, GMLVQ weights the texture features corresponding to their ability in discriminating between the two lung tissue classes. In our experiment with real-world data, the feature sets selected by the GMLVQ approach had a similar or a significantly better classification performance compared with feature sets selected by a mutual information ranking. Some of the feature subsets selected by the GMLVQ ranking even improved the overall classification performance of three classifiers.

The common challenge in the field of feature selection is to find sets of features with a low pairwise correlation to avoid redundancy but with a high relevance of the complete set for the classification task. This means that even weak features can be included, if they provided complimentary information to strong features. In general, ranking single features based on their individual performance and then create feature sets according to this ranking may not accommodate these requirements and thus, may not identify the members nor the size of the optimal feature subset. The above ranking-based feature selection method considers only the individual performance of the features and does not consider pairwise correlations between features. However, for this application of interest, the GMLVQ relevance yields a satisfactory performance. While this approach estimates the relevance of single features, future considerations of GMLVQ should include the pairwise correlation (Fig. 3) for the feature ranking, e.g. to reduce the redundancy of two equally relevant features [21, 22].

In order to improve computed-aided diagnosis systems, the presented approach can be used to compare the performance of texture features and to optimize their free parameters in order to find the near optimal settings for subsequent supervised learning tasks. From the broad variety of approaches for texture classification [1], the best set of individual texture features can be identified that is best suited to distinguish between data classes. Most of these texture feature methods have free parameters and it is unclear, which set of parameters leads to the best discrimination between texture classes. For one of these methods, the parameter setting with the best classification performance can be found. For instance, in the presented experiment, the GLCM features with a pixel spacing s_p = 5 did not contribute to a significantly better classification performance (Fig. 2). More complex texture feature approaches may benefit from this reduction to the possible parameter space.

In terms of the detection of ILD patterns, a limiting factor of this study was that only one typical pathological lung pattern, i.e. honeycombing, was classified. This represents one of many interstitial lung disease patterns that are common in clinical chest HRCT [1] and should be investigated in future studies. For this endeavor, the GMLVQ approach can be extended for such a multi-class problem to provide feature rankings of individual features for each class. This leads to class specific feature selection and may reveal further insights into the texture quantification and the structure of the feature space.

7. Conclusions

Using the GMLVQ algorithm, the relevance of texture features was estimated in their ability to classify healthy and diseased lung patterns in chest CT images. In our experiment with real-world data, the feature sets selected by the GMLVQ approach had a significantly better classification performance compared with feature sets selected by a mutual information ranking. While this approach estimates the relevance of single features, future considerations of GMLVQ should include the pairwise correlation (Fig. 3) for the feature ranking, e.g. to reduce the redundancy of two equally relevant features [21, 22].