Effect of Data Combination on Predictive Modeling: A Study Using Gene Expression Data

Abstract

Background:

The quality of predictive modeling in biomedicine depends on the amount of data available for model building.

Objective:

To study the effect of combining microarray data sets on feature selection and predictive modeling performance.

Methods:

Empirical evaluation of stability of feature selection and discriminatory power of classifiers using three previously published gene expression data sets, analyzed both individually and in combination.

Results:

Feature selection was not robust for the individual as well as for the combined data sets. The classification performance of models built on individual and combined data sets was heavily dependent on the data set from which the features were extracted.

Conclusion:

We identified volatility of feature selection as contributing factor to some of the problems faced by predictive modeling using microarray data.

Introduction

Analysis of data from high-throughput experiments has become pervasive in biomedical research. The development of this technology, along with independent (but concurrent) advances in machine learning algorithms, have significantly changed our perception of how biomedical data should be analyzed and interpreted: In the future it is conceivable that a patient’s health status can be assessed more accurately by a combination of data items.

It is well-known in the field of machine learning that the quality of data analysis depends on two factors: the machine learning algorithm (mostly the choices of parameters for the algorithm), and the data that serves as input to the algorithm. If we restrict our attention to the task of predictive modeling, and consider that several machine learning algorithms have exhibited similar performance^1–3 (albeit on lower-dimensional data), then the data becomes the main determining factor for the quality (predictive power) of the model.

While it is hard to assess the quality of data at a level not involving intricate knowledge of the data generation process (for example, whether data acquisition was biased, or measurement devices malfunctioning), from a machine learning perspective it is clear that the quantity of data has a direct influence on model quality: The more data is available, the better the resulting model will be. This is because the goal of predictive modeling as an inference mechanism is to make predictions for cases from the data population as a whole, and not merely for cases from a particular data set. With increasing amounts of data, the sample data set becomes an increasingly accurate representation of the whole population.

Unfortunately, the size of the data space grows exponentially with the number of attributes in the data (the data dimensionality). This is especially true for high-throughput data generated by microarray or mass spectrometry experiments. So while more data will result in better models, the amount of data required to obtain substantially better models may need to grow exponentially as well. Thus, the ideal situation (from a predictive modeling perspective) is to build models using a large number of low-dimensional cases. Microarray experiments, however, exhibit reverse characteristics: Case numbers are limited to a few hundreds (at most), while the data dimensionality is several orders of magnitudes higher. The analysis of microarray experiments is therefore a major challenge for the field of machine learning.

Several publications have investigated how and to which extend predictive modeling of microarray data can benefit from the increased number of cases available by combining data sets.^4–7 Although the challenges posed by combining data from different microarray experiments are substantial,⁸ there are software packages available that help with this task.^{9, 10}

Our research question is to investigate the effect that data combination has on predictive modeling by focusing on the contribution of feature selection, an important first step in predictive model building. We evaluate data from three different publications that analyze gene expression profiles for breast cancer prognosis, both individually and after combining them. The machine learning task is binary classification: distinguishing patients with good outcome from those with poor outcome. Previous work has focused on which data sets to include in a study,⁷ on predictions based on pair-wise expression comparisons,⁵ on robust greedy feature selection,⁶ and on linear additive models based on Cox regression coefficients.⁴

In contrast to these studies, we evaluate how the performance of machine learning algorithms, in particular logistic regression models, depends on the number and choice of selected features (gene expressions). Using logistic regression over alternatives such as support vector machines and artificial neural networks is motivated by the fact that the performance of the latter models depends crucially on the choice of parameter settings. Since the goal of our work is to isolate and make comparable the influence of the choice of features on the predictive power of models, it is advantageous to use a parameter-free model for this comparison. We therefore apply logistic regression to data sets obtained from three different studies, analyzed both individually and as one merged data set.

Material and methods

Gene expression data

We downloaded three breast cancer data sets from NCBI Gene Expression Omnibus (GEO): The data sets used by Wang et al.¹¹(GSE2034), by Sotiriou et al.¹² (GSE2990), and by Miller et al.¹³ (GSE3494). We used raw CEL files and associated clinical information of the patients. For within-study normalization, we split GSE3494 samples into two groups, GSE3494-A and GSE3494-B, according to the sample’s Affymetrix platform.

To make the samples comparable, we selected patients based on the criteria used in Xu et al,⁵ meaning that we only used the patients who did not receive any adjuvant treatment and had negative lymph node status. Then we selected the patients with extreme outcomes: those with poor outcome (either recurrence or metastasis within five years) and those with good outcome (neither recurrence nor metastasis within eight years). After filtering, the number of samples in the data sets were as follows: 209 for GSE2034 (114 good/95 poor), 90 for GSE2990 (60 good/30 poor), and 242 for GSE3494 (224 good/18 poor).

We performed quantile-normalization preserving ranks of genes of the sample within a study.¹⁴ For probe-to-gene annotation, we first aligned a probe against genome, then matched its positions to those of exons of AceView, the most comprehensive transcript database.¹⁵ Then we summarized the mRNA abundance at the gene level using the median polish method from the multiple probe level measurement values.¹⁶ In their original study, Xu et al.⁵ used only the samples of the HGU133A platform from the publication of Miller et al.,¹³ discarding 201 samples. With a sequence-based annotation approach, we were able to include the samples of both the HGU133A and HGU133B platforms. The combined data set thus contained 541 samples, 398 with good prognosis, and 143 with poor prognosis. The number of genes in the combined data set (the intersection of genes in all data sets) was 4103.

Feature selection

Feature selection is usually the first step in predictive modeling, because machine learning algorithms are better able to generalize from training data to unseen cases after irrelevant features (genes, in our case) have been discarded. Ideally, feature selection is performed by experts in the field who can identify irrelevant features that may safely be omitted from the analysis. Because this knowledge is often not available (as is the case with microarray data), a number of automated algorithms for feature selection have been developed.

In our study, we performed feature ranking via Student’s t-test, which measures the spread between a gene’s mean expression values for patients with good and poor outcomes, respectively, relative to the dispersion of these values. Preliminary experiments (data not shown) using information gain and ReliefF as univariate and multivariate alternatives, respectively, to the t-test exhibited similar or worse performance, compared to the numbers reported below.

We used 10-fold cross-validation for the feature selection step to asses the variability of feature selection on microarray data sets. This means that a t-test is applied 10 times, every time to 9/10 of the data. We thus obtained 10 different rankings of feature relevance. A composite ranking can be computed by taking the mean of the 10 ranks of each feature, and sorting the means. Additionally, the standard deviation of a feature’s rank gives an indication of the volatility of this rank.

Model building and evaluation

Logistic regression models were built using WEKA, the free machine learning software suite^*. Data was normalized to zero mean and unit variance before model building. No feature selection scheme, such as forward selection or backwards elimination, was used, because relevant features had already been determined.

The performance of machine learning models is judged by how well they generalize, that is, perform on new cases. Therefore, a number of cases are set aside for later use as test set. A common measure for evaluating classification performance on this test set is the area under the ROC curve (AUC), which measures how well two classes can be separated by the model,¹⁷ regardless of where the optimal separating threshold lies. Perfect separation yields an AUC value of 1, no discrimination an AUC value of 0.5.

Model validation was done by five times randomly splitting the data into a training and test set. We chose this approach due to the low amount of cases with poor prognosis in two of the three data sets. 10-fold cross-validation would have resulted in only a few (one to three) cases in the test set. The performance numbers reported below are averages over the five test sets.

To guard against the possibility of the choice of LR models unduly influencing the results, we also investigated the use of decision tree models as classifiers. The results of these models, however, were worse than those of logistic regression models, regardless of whether the t-test or information gain was used as feature selection method.

Results

Using the methods described above, we investigated the following hypotheses:

1. Feature selection methods are robust across different data sets.

2. Machine learning performance increases for larger data sets.

Robustness of feature selection

We were able to assess the variability of a feature ranking by using 10-fold cross-validation in the feature selection step. Figure 1 illustrates this volatility for data set GSE2034. One can observe that the standard deviations are quite large, with an average of 9.46 over the 20 top-ranked features shown. The corresponding average standard deviation was 20.73 for GSE2990, 21.33 for GSE3494, and 12.94 for the combined data set.

Figure 1:

Average ranks (vertical axis) over ten cross-validation runs for the 20 top ranked features in GSE2034 (horizontal axis). Vertical bars indicate standard deviations of ranks.

Volatility was high not only within data sets, but also between data sets. To assess this volatility, we used the feature ranking of the combined data set against which we plotted the ranks of the corresponding features in each of the three individual data sets. This plot of inter-study ranking variability is shown in Figure 2. For example, the top-ranked gene for the combined data set (ZNF250, avg. rank = 5.1) is in positions 109.5, 517.1 and 10.5 in the three feature rankings for the individual data sets. In the ideal case, all features would have the same rank in each of the studies, and Figure 2 would show dots along the diagonal (due to the large scale difference in the axes, basically the same as the horizontal axis). The large deviation from this ideal case illustrates that feature rankings are heavily dependent on the data set from which they were generated.

Figure 2:

Ranks of genes obtained from individual data set rankings (vertical axis) plotted against rank from combined data set ranking (horizontal axis).

Benefit of data set combination

To assess the influence of the observed feature volatility on classifier performance, we tested how the features obtained from one data set performed on the other data sets. More precisely, we took the 200 top-ranked features from each data set and divided them into 40 feature sets containing the best 5, 10, 15, . . . features. We then trained 16×40 logistic regression models, one for each combination of feature set and data set.

Not surprisingly, the results showed that the feature sets exhibited the best performance on the data sets from which they were selected. Figure 3 depicts the classification performance of all four data sets trained on the best k features of the combined data set and the GSE3494 data set, respectively. Similar characteristics of classification performance can be observed for the features extracted from the other two data sets as well (not shown). It was thus not the case that the features from the combined data set are better than the others, in the sense that these features increase classifier performance on the other data sets.

Figure 3:

Discriminatory power of logistic regression models (measured by the AUC, vertical axis) trained on the k best features (horizontal axis) of the combined data set (a) and the GSE3494 data set (b) for the combined data set (thick black line), GSE2034 (thin red line), GSE2990 (thin green line), and GSE3494 (thin blue line).

Discussion

High-throughput experiments yield data on a large number of potential predictors, but the number of cases is usually several orders of magnitude smaller than the number of predictors (i.e., hundreds of thousands of predictors to hundreds of cases). Combining data sets to correct this imbalance is a tantalizing prospect. While some studies have shown that for predictive modeling, the data processing pipeline of combining high-throughput measurements is sufficiently advanced to yield improved results,^{7, 18} others report no benefits from combining data sets.⁴ Still others analyze combined data sets, but do not compare them to the individual parts from which they were formed.^{6, 19}

Of the above, the only publication to specifically address the role of feature selection in predictive modeling on combined data sets is the work of Yasrebi et al.⁴ While Wang et al.⁶ acknowledge that “predictors were inconsistent”, the other publications ignore this issue, instead using feature selection as a necessary black-box step in model building that does not warrant further scrutiny.

In this paper, we showed that on the contrary, feature selection in the context of high-throughput measurements is a highly volatile process that has great influence on predictive modeling performance. We demonstrated that there is high variability in the highest-ranked features within and across data sets, and the effect that this variability has on predictive performance. Although these results were obtained on only one combination of data sets, they highlight the need to specifically address feature selection when combining data sets for predictive modeling.