A Feature Selection Approach Based on Interclass and Intraclass Relative Contributions of Terms

Abstract

Feature selection plays a critical role in text categorization. During feature selecting, high-frequency terms and the interclass and intraclass relative contributions of terms all have significant effects on classification results. So we put forward a feature selection approach, IIRCT, based on interclass and intraclass relative contributions of terms in the paper. In our proposed algorithm, three critical factors, which are term frequency and the interclass relative contribution and the intraclass relative contribution of terms, are all considered synthetically. Finally, experiments are made with the help of kNN classifier. And the corresponding results on 20 NewsGroup and SougouCS corpora show that IIRCT algorithm achieves better performance than DF, t-Test, and CMFS algorithms.

1. Introduction

As the number of digital documents available on the Internet has been growing significantly in recent years, it is impossible to manipulate manually such enormous information [1]. More and more methods based on statistical theory and machine learning have been proposed, and they are applied successfully to information processing. An effective method for managing the vast amount of data is text categorization, which has been widely applied to many fields such as theme detection, spam filtering, identity recognition, web page classification, and semantic parsing.

The goal of text classification is to assign a new document automatically to a predefined category [2]. A typical text classification framework consists of preprocessing, document representation, feature selection, feature weighting, and classification stages [3]. In the preprocessing stage, it usually contains such tasks as tokenization, stop-word removal, lowercase conversion, and stemming. In the document representation stage, it generally utilizes the vector space model that makes use of the bag-of-words approach [4]. In the feature selection stage, it usually employs the filter methods such as document frequency (DF) [5], mutual information (MI) [6], information gain (IG) [7], and chi-square (CHI) [8]. In the feature weighting stage, it usually uses TF-IDF to calculate the weights of the selected features in each document. And in the classification stage, it always uses some popular classification algorithms, for example, decision trees [9], k-Nearest Neighbors (kNN) [10], and support vector machine (SVM) [11].

The major characteristic of text categorization is that the feature number in the feature space can easily reach up to tens or hundreds of thousands. It can not only increase computational time but also degrade classification accuracy [12]. As a consequence, feature selection plays a critical role in text classification.

The existing experimental results show that IG is one of the most effective feature selection methods, the performance of DF is similar to IG, and MI is the worst [13]. Through comparative analysis, it is easy to find that the performances of DF and IG are good, which means that high-frequency terms are really essential to text classification, while the performance of MI is bad as it is inclined to select low-frequency terms as features. Besides, t-Test method is also based on term frequency [14] and its performance is good. During feature selecting, Categorical Term Descriptor (CTD) method considers the document frequency of IDF and the category information of ICF particularly [15]. Similarly, Strong Class Information Words (SCIW) method selects the terms which have good abilities to distinguish categories [16] and it also considers the category information. Experimental results show that CTD and SCIW both have good accuracies. So we can easily know that feature selection methods based on category information always have good performances. As a result, we draw that high-frequency terms and category information are very important in improving the classification effectiveness. Comprehensively Measure Feature Selection (CMFS) method [1] considers high-frequency terms and category information simultaneously, and it also obtains good results. But it does not consider the interactions between categories. In view of these, we propose a new feature selection algorithm named as feature selection approach based on interclass and intraclass relative contributions of terms (IIRCT), in which term frequency and the interclass relative contribution and the intraclass relative contribution of terms are all considered synthetically.

2. Related Works

To deal with massive documents corpora, many feature selection approaches have been proposed. And their purpose is to select the terms whose classification capabilities are stronger comparatively in feature space. After feature selection, the dimensionality of feature space can be reduced, and the efficiency and accuracy of classifiers can be improved. Its main idea is as follows. Firstly, it uses the feature selection function to compute some important indicators of each word in feature space. And then, it sorts the words in descending order according to above values. Finally, it selects the top m words to construct the feature vector.

In this section, we introduce some symbols used in the following firstly.

tf_ij is the times that the term t _i appears in document d _j, namely, term frequency.

$\overline{\mathrm{t}{\mathrm{f}}_{ki}}$ is the average frequency of the term t _i within a single category C _k, and the calculation formula is as follows:

$$\begin{array}{c}\overline{\mathrm{t}{\mathrm{f}}_{ki}}={\displaystyle \sum }_{j=\mathrm{1}}^N\mathrm{t}{\mathrm{f}}_{ij}·\frac{I\left(d_j,C_k\right)}{N_k},\end{array}$$ (1)

where N is the document number in collection D, N _k is the document number in category C _k, and I(d _j, C _k) = {1,  d _j ∈ C _k; 0,  d _j ∉ C _k}, which is an indicator to discriminate whether document d _j belongs to category C _k.

$\overline{\mathrm{t}{\mathrm{f}}_i}$ is the average term frequency of the term t _i in collection D, and it is calculated according to

$$\begin{array}{c}\overline{\mathrm{t}{\mathrm{f}}_i}=\frac{\mathrm{1}}{N}{\displaystyle \sum }_{j=\mathrm{1}}^N\mathrm{t}{\mathrm{f}}_{ij}.\end{array}$$ (2)

Similarly, N is the document number in collection D.

Then we give the definition of three feature selection methods, which are DF, t-Test, and CMFS, respectively.

2.1. DF

DF method calculates the number of documents which contain the terms in the category to measure the relevance of the terms and the categories. And the terms can be reserved only when they appear in adequate documents. This measurement is based on such an assumption that the terms which have low values of DF have few effects on the classification performance [8]. So DF method always selects terms with high values of DF and removes terms with low values of DF.

DF method is a simple word reduction technology and has good performance. Due to its linear complexity, it can be easily scaled to be used in large-scale corpus.

2.2. t-Test

t-Test [14] is a feature selection approach based on term frequency, which is used to measure the diversity of the distributions of a term between the specific category and the entire corpus. And it is defined as follows:

$$\begin{array}{c}t\text{-}\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t}\left(t_i,C_k\right)=\frac{\left|\overline{\mathrm{t}{\mathrm{f}}_{ki}}-\overline{\mathrm{t}{\mathrm{f}}_i}\right|}{\sqrt{\mathrm{1}/N_k-\mathrm{1}/N}\ast s_i}.\end{array}$$ (3)

In (3), $\overline{\mathrm{t}{\mathrm{f}}_{ki}}$ is the average frequency of the term t _i within a single category C _k, $\overline{\mathrm{t}{\mathrm{f}}_i}$ is the average term frequency of the term t _i in collection D, N _k is the document number in category C _k, N is the document number in collection D, $s_i^{\mathrm{2}}=\left(\mathrm{1}/\left(N-\left|C\right|\right)\right){\sum }_{k=\mathrm{1}}^{\left|C\right|}{\sum }_{j\in C_k}^{}(\mathrm{t}{\mathrm{f}}_{ij}-\overline{\mathrm{t}{\mathrm{f}}_{ki}}{)}^{\mathrm{2}}$, and |C| is the category number in collection D.

The following two ways are used alternatively when the main features are finally selected:

$$\begin{array}{c}t\text{-}\mathrm{T}\mathrm{e}\mathrm{s}{\mathrm{t}}_{\mathrm{a}\mathrm{v}\mathrm{g}}\left(t_i\right)={\displaystyle \sum }_{k=\mathrm{1}}^{\left|C\right|}p\left(C_k\right)\ast t\text{-}\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t}\left(t_i,C_k\right),\\ \end{array}$$ (4)

$$\begin{array}{c}t\text{-}\mathrm{T}\mathrm{e}\mathrm{s}{\mathrm{t}}_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(t_i\right)=\underset{k=\mathrm{1}}{\overset{\left|C\right|}{\max }}\left\{t\text{-}\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t}\left(t_i,C_k\right)\right\},\\ \end{array}$$ (5)

where p(C _k) = N _k/N, N _k is the document number in category C _k, and N is the document number in collection D.

Generally, the method shown in (4) is always better than that shown in (5) for multiclass problem.

2.3. CMFS

When selecting features, DF method only computes the document frequency of each unique term in one category, and then the highest document frequency of a term in various categories is retained as the term's score. DIA association factor method [17] only calculates the distribution probability of a term in various categories, and then the highest probability of the term can be used as the term's score. Yang et al. [1] noticed that both DF and DIA methods only focus on one respect of the problems (row or column). Thus DF method concentrates on the column of the term-to-category matrix, while DIA focuses on the row of the term-to-category matrix. Based on such observation, a new feature selection algorithm, Comprehensively Measure Feature Selection (CMFS), is proposed by Yang et al. It comprehensively measures the significance of a term both in intercategory and intracategory. And it is defined as follows:

$$\begin{array}{c}\mathrm{C}\mathrm{M}\mathrm{F}\mathrm{S}\left(t_i,C_k\right)=p\left(t_i\hspace{0.17em}\mid \hspace{0.17em}{C}_k\right)\ast p\left(C_k\hspace{0.17em}\mid \hspace{0.17em}{t}_i\right).\end{array}$$ (6)

Here, p(t _i∣C _k) is the probability that the feature t _i appears in category C _k, and p(C _k∣t _i) can be considered as the conditional probability that the feature t _i belongs to category C _k when the feature t _i occurs.

To measure the goodness of a term globally, two alternate ways can be used to combine the category-specific scores of a term. And the formulae are as follows:

$$\begin{array}{c}\mathrm{C}\mathrm{M}\mathrm{F}{\mathrm{S}}_{\mathrm{a}\mathrm{v}\mathrm{g}}\left(t_i\right)={\displaystyle \sum }_{k=\mathrm{1}}^{\left|C\right|}p\left(C_k\right)\ast \mathrm{C}\mathrm{M}\mathrm{F}\mathrm{S}\left(t_i,C_k\right),\\ \\ \mathrm{C}\mathrm{M}\mathrm{F}{\mathrm{S}}_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(t_i\right)=\underset{k=\mathrm{1}}{\overset{\left|C\right|}{\max }}\left\{\mathrm{C}\mathrm{M}\mathrm{F}\mathrm{S}\left(t_i,C_k\right)\right\},\\ \end{array}$$ (7)

where p(C _k) = N _k/N, N _k is the document number in category C _k, and N is the document number in collection D.

3. IIRCT

In this section, we propose a feature selection approach based on interclass and intraclass relative contributions of terms. In the proposed algorithm, three critical factors, which are term frequency and the interclass relative contribution and the intraclass relative contribution of terms, are all considered synthetically.

3.1. Motivation

At present, a large number of feature selection algorithms emerge. Through studying and analysing them, we can easily find that DF, IG, and t-Test algorithms are inclined to select high-frequency terms as main features, and their performances are good. Among them, DF and IG algorithms are based on document frequency, and t-Test algorithm is based on term frequency. CTD and SCIW algorithms consider the category information, and they both have good accuracies.

Therefore, we conclude the following ones:

1. A term, which frequently occurs in a single class and does not occur in the other classes, is distinctive. Therefore, it should be given a high score.

2. A term, which rarely occurs in a single class and does not occur in the other classes, is irrelevant. Therefore, it should be given a low score.

3. A term, which frequently occurs in all classes, is irrelevant. Therefore, it should be given a low score.

4. A term, which occurs in some classes, is relatively distinctive. Therefore, it should be given a relatively high score.

From points (1) and (2), it can be seen that high-frequency terms have effects on the classification performance. From points (3) and (4), it can be seen that category information is also a very important factor which influences the classification effect. As a result, we have a conclusion that high-frequency terms and category information are both very important factors in improving the classification performance. In view of these, high-frequency terms and category information are considered synthetically when constructing feature selection function in this paper. When judging whether a word is a high-frequency term, term frequency method is used. While considering category information, we notice that ① if the probability that the feature t _i occurs in category C _k is higher than other features, t _i can represent C _k more effectively, ② if the probability that the feature t _i occurs in category C _k is higher than t _i occurs in other categories, t _i can represent C _k more effectively, ③ if the conditional probability that the feature t _i belongs to category C _k is higher than t _i belongs to other categories when the feature t _i occurs, t _i can represent C _k more effectively. So, the feature selection function constructed in this paper considers the interclass and intraclass relative contributions of terms to measure the category information.

Based on the above, we propose a new feature selection approach, IIRCT, in which term frequency and the interclass relative contribution and the intraclass relative contribution of terms are all considered synthetically.

3.2. Algorithm Implementation

In this section, we firstly introduce some symbols.

TF_ik is the term frequency of term t _i in category C _k, and it is calculated according to

$$\begin{array}{c}\mathrm{T}{\mathrm{F}}_{ik}={\displaystyle \sum }_{j=\mathrm{1}}^{N_k}\mathrm{t}{\mathrm{f}}_{ij},\end{array}$$ (8)

where N _k is the document number in category C _k and tf_ij is the times that the term t _i appears in document d _j.

df_ik is the document frequency of term t _i in category C _k.

TF_k is the total term frequency of all terms in category C _k, and the calculation formula is as follows:

$$\begin{array}{c}\mathrm{T}{\mathrm{F}}_k={\displaystyle \sum }_{i=\mathrm{1}}^{M_k}\mathrm{T}{\mathrm{F}}_{ik},\end{array}$$ (9)

where M _k is the term number in category C _k.

df_i is the total document frequency of term t _i in all categories, and it is calculated according to

$$\begin{array}{c}\mathrm{d}{\mathrm{f}}_i={\displaystyle \sum }_{k=\mathrm{1}}^{\left|C\right|}\mathrm{d}{\mathrm{f}}_{ik},\end{array}$$ (10)

where |C| is the category number.

IIRCT algorithm measures the significance of a term from three aspects comprehensively, which are term frequency and the interclass and intraclass relative contributions of terms. Thus, we define comprehensive measurement for each term t _i with respect to category C _k as follows:

$$\begin{array}{c}\mathrm{I}\mathrm{I}\mathrm{R}\mathrm{C}\mathrm{T}\left(t_i,C_k\right)={\displaystyle \sum }_{j=\mathrm{1},j\ne k}^{\left|C\right|}\left(\frac{\mathrm{T}{\mathrm{F}}_{ik}}{\mathrm{T}{\mathrm{F}}_k}\ast \frac{\mathrm{d}{\mathrm{f}}_{ik}}{\mathrm{d}{\mathrm{f}}_i}-\frac{\mathrm{T}{\mathrm{F}}_{ij}}{\mathrm{T}{\mathrm{F}}_j}\ast \frac{\mathrm{d}{\mathrm{f}}_{ij}}{\mathrm{d}{\mathrm{f}}_i}\right),\end{array}$$ (11)

where |C| is the category number, TF_ik is the term frequency of term t _i in category C _k, TF_k is the total term frequency of all terms in category C _k, df_ik is the document frequency of term t _i in category C _k, and df_i is the total document frequency of term t _i in all categories.

In view of the probability theory, we can regard TF_ik/TF_k in (11) as the probability that the feature t _i occurs in category C _k, that is, p(t _i∣C _k). df_ik/df_i in (11) can be considered as the conditional probability that the feature t _i belongs to category C _k when the feature t _i occurs, that is, p(C _k∣t _i). TF_ij/TF_j in (11) can be considered as the probability that the feature t _i occurs in category C _j, that is, p(t _i∣C _j). df_ij/df_i in (11) can be considered as the conditional probability that the feature t _i belongs to category C _j when the feature t _i occurs, that is, p(C _j∣t _i). So (11) can be further represented as follows:

$$\begin{array}{c}\mathrm{I}\mathrm{I}\mathrm{R}\mathrm{C}\mathrm{T}\left(t_i,C_k\right)={\displaystyle \sum }_{j=1,j\ne k}^{\left|C\right|}\left[p\left(t_i\hspace{0.17em}\mid \hspace{0.17em}{C}_k\right)\ast p\left(C_k\hspace{0.17em}\mid \hspace{0.17em}{t}_i\right)-p\left(t_i\hspace{0.17em}\mid \hspace{0.17em}{C}_j\right)\ast p\left(C_j\hspace{0.17em}\mid \hspace{0.17em}{t}_i\right)\right].\end{array}$$ (12)

Here, p(t _i∣C _k) is the probability that the feature t _i occurs in category C _k, and p(C _k∣t _i) can be considered as the conditional probability that the feature t _i belongs to category C _k when the feature t _i occurs.

To measure the goodness of a term globally, we construct the following function:

$$\begin{array}{c}\mathrm{I}\mathrm{I}\mathrm{R}\mathrm{C}\mathrm{T}\left(t_i\right)={\displaystyle \sum }_{k=\mathrm{1}}^{\left|C\right|}p\left(C_k\right)\ast \mathrm{I}\mathrm{I}\mathrm{R}\mathrm{C}\mathrm{T}\left(t_i,C_k\right),\end{array}$$ (13)

where p(C _k) = N _k/N which is the probability that category C _k occurs in the entire training set, N _k is the document number in category C _k, and N is the document number in collection D.

3.3. Algorithm Description

According to the above, we present a new feature selection algorithm, IIRCT, based on interclass and intraclass relative contributions of terms. Its pseudocode is as in Pseudocode 1.

Pseudocode 1.

4. Experiments Setup

4.1. Experimental Data

In this paper, we use two popular datasets, 20 NewsGroup and SougouCS.

The 20 NewsGroup corpus, which is collected by Ken Lang, has been widely used in text classification. This corpus contains 19997 newsgroup documents which are nearly evenly distributed among 20 discussion groups, and every group consists of 1,000 documents. All letters are converted into lowercase, and the word stemming is applied. In addition, we use the stop words list to filter words. The details of 20 NewsGroup corpus are as shown in Table 1.

Table 1.

20 NewsGroup corpus.

Category number	Category name
1	alt.atheism
2	comp.graphics
3	comp.os.ms-windows.misc
4	comp.sys.ibm.pc.hardware
5	comp.sys.mac.hardware
6	comp.windows.x
7	misc.forsale
8	rec.autos
9	rec.motorcycles
10	rec.sport.baseball
11	rec.sport.hockey
12	sci.crypt
13	sci.electronics
14	sci.med
15	sci.space
16	soc.religion.christian
17	talk.politics.guns
18	talk.politics.mideast
19	talk.politics.misc
20	talk.religion.misc

The SougouCS corpus is provided by Sogou Laboratory. The documents of the corpus are from Sohu news website which has a lot of classified information. As the number of web pages in some classes is too small, we only choose 12 classes. And the detail is as shown in Table 2.

Table 2.

SougouCS corpus.

Category number	Category name
1	Car
2	Finance
3	IT
4	Health
5	Sports
6	Tourism
7	Education
8	Culture
9	Military
10	Housing
11	Entertainment
12	Fashion

4.2. Document Representation

Documents are represented by vector space model [4]. That is, the content of a document is represented by a vector in the term space. It is illustrated in detail as the following. Consider V(d) = (t ₁, w ₁(d),…, t _i, w _i(d),…, t _m, w _m(d)), where m is the number of the features selected by feature selection algorithms and w _i(d) is the weight of feature t _i in document d. In experiments, Term Frequency-Inverse Document Frequency (TF-IDF) [18] is used to calculate the weights of the m selected features in each document.

4.3. Classifier Selection

In the experiments, k-Nearest Neighbors (kNN) is used to classify and test documents. And it is also a case-based or instance-based categorization algorithm. At present, kNN is widely used in text classification as it is simple and has low error rate.

The principle of kNN classification algorithm is very simple and intuitive. Giving a test document whose category is unknown, the classification system will find the k-nearest documents by computing the similarities between documents in training data. And then, we will get the category of the test documents according to the k-nearest documents. The similarity measure used for the classifier is the cosine function [19].

In the paper, we set k = 20. And we randomly select 65% instances from each category as training data and the rest as testing data.

4.4. Performance Measures

We measure the effectiveness of classifiers in terms of the combination of precision (p) and recall (r) widely used in text categorization. That is, we use the well-known F ₁ function [20] as follows:

$$\begin{array}{c}{F}_{\mathrm{1}}=\frac{\mathrm{2}\ast p\ast r}{p+r}.\end{array}$$ (14)

For multiclass text categorization, F ₁ is usually calculated in two ways. And they are the macroaveraged F ₁ (macro-F ₁) and the microaveraged F ₁ (micro-F ₁). Here, we only use macro-F ₁, as shown in

$$\begin{array}{c}\mathrm{m}\mathrm{a}\mathrm{c}\mathrm{r}\mathrm{o}\text{-}{F}_1=\frac{{\sum }_{k=1}^K{F}_1\left(k\right)}{K},\end{array}$$ (15)

where F ₁(k) is the F ₁ value of the predicted kth category.

5. Results and Discussions

5.1. Results and Discussions on 20 NewsGroup

Figure 1 shows the precision and recall of IIRCT, DF, t-Test, and CMFS on the 20 NewsGroup corpus when 1,500 features are selected in feature space. It can be seen from Figure 1(a) that the precision of IIRCT is higher than that of DF, t-Test, and CMFS. And in some categories, the precision of IIRCT almost reaches up to 95%. Similarly, Figure 1(b) also indicates that the performance of IIRCT is better than that of DF, t-Test, and CMFS, and the recall of most categories has some improvements.

Figure 1.

Precision and recall performance on the 20 NewsGroup corpus.

The numbers 1–20 in Figure 1 can be referred to in Table 1.

Figure 2 shows the macro-F ₁ performance of IIRCT, DF, t-Test, and CMFS on the 20 NewsGroup corpus with different feature dimensionalities. From Figure 2, we can conclude that the macro-F ₁ of IIRCT is close to that of CMFS when 100 features are selected. But if 200, 500, 1000, 1500, 2000, 2500, 3000, or 3500 terms are selected as features, the macro-F ₁ curve of IIRCT is higher than that of DF, t-Test, and CMFS. This means that the performance of IIRCT is better than the other three algorithms. Besides, it can be found that the value of macro-F ₁ decreases as the feature number increases. The reason for this is that the boundaries between categories are very clear in the 20 NewsGroup corpus. As a consequence, small amount of features can achieve good classification performance. But with the feature number increasing, many features have a negative impact on classification performance. And the classification effect gets poor.

Figure 2.

macro-F ₁ performance on the 20 NewsGroup corpus.

5.2. Results and Discussions on SougouCS

Figure 3 shows the precision and recall of IIRCT, DF, t-Test, and CMFS on the SougouCS corpus when 4,500 features are selected in feature space. It is clear that, in most categories, the precision and recall of IIRCT have some improvements compared to DF, t-Test, and CMFS. And this means that IIRCT achieves better performance than that of DF, t-Test, and CMFS.

Figure 3.

Precision and recall performance on the SougouCS corpus.

The numbers 1–12 in Figure 3 can be referred to in Table 2.

Figure 4 depicts the macro-F ₁ performance of the four algorithms on the SougouCS corpus. From Figure 4, we can know that the macro-F ₁ curve of IIRCT lies above the other three curves, which also means IIRCT has better performance than that of DF, t-Test, and CMFS. Besides, it can be found that the value of macro-F ₁ is the largest when 4500 features are selected. And when the selected feature number increases or decreases from 4500, the value of macro-F ₁ decreases. The reason for this is that, in the SougouCS corpus, some categories, such as fashion and entertainment, have many common words which make the boundaries between categories obscure. When small amount of features is selected, some documents cannot be classified correctly. And when the feature number increases to a certain value, these features make the boundaries between categories clear and improve the classification effect. When the feature number keeps increasing, many features have a negative impact on classification performance. And the classification effect gets poor.

Figure 4.

macro-F ₁ performance on the SougouCS corpus.

6. Conclusions

Feature selection plays a critical role in text classification and has an immediate impact on text categorization. So we put forward a feature selection approach, IIRCT, based on interclass and intraclass relative contributions of terms in the paper. In our proposed algorithm, term frequency and the interclass and intraclass relative contributions of terms are all considered synthetically. The experimental results on 20 NewsGroup and SougouCS corpora show that IIRCT achieves better performance than DF, t-Test, and CMFS. Therefore, the algorithm proposed in this paper is an effective feature selection method.

Competing Interests

The authors declare that they have no competing interests.