Skip to main content
PMC home page
EURASIP Journal on Bioinformatics and Systems Biology logoLink to EURASIP Journal on Bioinformatics and Systems Biology
. 2007 Oct 30;2007(1):38473. doi: 10.1155/2007/38473

Decorrelation of the True and Estimated Classifier Errors in High-Dimensional Settings

Blaise Hanczar 1,2,✉, Jianping Hua 3, Edward R Dougherty 1,3
PMCID: PMC3171336  PMID: 18288255

Abstract

The aim of many microarray experiments is to build discriminatory diagnosis and prognosis models. Given the huge number of features and the small number of examples, model validity which refers to the precision of error estimation is a critical issue. Previous studies have addressed this issue via the deviation distribution (estimated error minus true error), in particular, the deterioration of cross-validation precision in high-dimensional settings where feature selection is used to mitigate the peaking phenomenon (overfitting). Because classifier design is based upon random samples, both the true and estimated errors are sample-dependent random variables, and one would expect a loss of precision if the estimated and true errors are not well correlated, so that natural questions arise as to the degree of correlation and the manner in which lack of correlation impacts error estimation. We demonstrate the effect of correlation on error precision via a decomposition of the variance of the deviation distribution, observe that the correlation is often severely decreased in high-dimensional settings, and show that the effect of high dimensionality on error estimation tends to result more from its decorrelating effects than from its impact on the variance of the estimated error. We consider the correlation between the true and estimated errors under different experimental conditions using both synthetic and real data, several feature-selection methods, different classification rules, and three error estimators commonly used (leave-one-out cross-validation, Inline graphic-fold cross-validation, and .632 bootstrap). Moreover, three scenarios are considered: (1) feature selection, (2) known-feature set, and (3) all features. Only the first is of practical interest; however, the other two are needed for comparison purposes. We will observe that the true and estimated errors tend to be much more correlated in the case of a known feature set than with either feature selection or using all features, with the better correlation between the latter two showing no general trend, but differing for different models.

[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22]

Contributor Information

Blaise Hanczar, Email: bhanczar@ece.tamu.edu.

Jianping Hua, Email: jhua@tgen.org.

Edward R Dougherty, Email: edward@ece.tamu.edu.

References

  1. Jain A, Zongker D. Feature selection: evaluation, application, and small sample performance. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1997;19(2):153–158. doi: 10.1109/34.574797. [DOI] [Google Scholar]
  2. Hughes G. On the mean accuracy of statistical pattern recognizers. IEEE Transactions on Information Theory. 1968;14(1):55–63. doi: 10.1109/TIT.1968.1054102. [DOI] [Google Scholar]
  3. Hua J, Xiong Z, Lowey J, Suh E, Dougherty ER. Optimal number of features as a function of sample size for various classification rules. Bioinformatics. 2005;21(8):1509–1515. doi: 10.1093/bioinformatics/bti171. [DOI] [PubMed] [Google Scholar]
  4. Dougherty ER. Small sample issues for microarray-based classification. Comparative and Functional Genomics. 2001;2(1):28–34. doi: 10.1002/cfg.62. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Braga-Neto UM, Dougherty ER. Is cross-validation valid for small-sample microarray classification? Bioinformatics. 2004;20(3):374–380. doi: 10.1093/bioinformatics/btg419. [DOI] [PubMed] [Google Scholar]
  6. Molinaro AM, Simon R, Pfeiffer RM. Prediction error estimation: a comparison of resampling methods. Bioinformatics. 2005;21(15):3301–3307. doi: 10.1093/bioinformatics/bti499. [DOI] [PubMed] [Google Scholar]
  7. Xiao Y, Hua J, Dougherty ER. Quantification of the impact of feature selection on the variance of cross-validation error estimation. EURASIP Journal on Bioinformatics and Systems Biology. 2007;2007 doi: 10.1155/2007/16354. 11 pages. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Braga-Neto U, Hashimoto R, Dougherty ER, Nguyen DV, Carroll RJ. Is cross-validation better than resubstitution for ranking genes? Bioinformatics. 2004;20(2):253–258. doi: 10.1093/bioinformatics/btg399. [DOI] [PubMed] [Google Scholar]
  9. Sima C, Braga-Neto U, Dougherty ER. Superior feature-set ranking for small samples using bolstered error estimation. Bioinformatics. 2005;21(7):1046–1054. doi: 10.1093/bioinformatics/bti081. [DOI] [PubMed] [Google Scholar]
  10. Sima C, Attoor S, Braga-Neto U, Lowey J, Suh E, Dougherty ER. Impact of error estimation on feature-selection algorithms. Pattern Recognition. 2005;38(12):2472–2482. doi: 10.1016/j.patcog.2005.03.026. [DOI] [Google Scholar]
  11. Zhou X, Mao KZ. The ties problem resulting from counting-based error estimators and its impact on gene selection algorithms. Bioinformatics. 2006;22(20):2507–2515. doi: 10.1093/bioinformatics/btl438. [DOI] [PubMed] [Google Scholar]
  12. Sima C, Dougherty ER. What should be expected from feature selection in small-sample settings. Bioinformatics. 2006;22(19):2430–2436. doi: 10.1093/bioinformatics/btl407. [DOI] [PubMed] [Google Scholar]
  13. Mehta T, Tanik M, Allison DB. Towards sound epistemological foundations of statistical methods for high-dimensional biology. Nature Genetics. 2004;36(9):943–947. doi: 10.1038/ng1422. [DOI] [PubMed] [Google Scholar]
  14. Dougherty ER, Datta A, Sima C. Research issues in genomic signal processing. IEEE Signal Processing Magazine. 2005;22(6):46–68. [Google Scholar]
  15. Michiels S, Koscielny S, Hill C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. The Lancet. 2005;365(9458):488–492. doi: 10.1016/S0140-6736(05)17866-0. [DOI] [PubMed] [Google Scholar]
  16. Dougherty ER, Braga-Neto U. Epistemology of computational biology: mathematical models and experimental prediction as the basis of their validity. Journal of Biological Systems. 2006;14(1):65–90. doi: 10.1142/S0218339006001726. [DOI] [Google Scholar]
  17. Braga-Neto U. Fads and fallacies in the name of small-sample microarray classification—a highlight of misunderstanding and erroneous usage in the applications of genomic signal processing. IEEE Signal Processing Magazine. 2007;24(1):91–99. [Google Scholar]
  18. Dupuy A, Simon RM. Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting. Journal of the National Cancer Institute. 2007;99(2):147–157. doi: 10.1093/jnci/djk018. [DOI] [PubMed] [Google Scholar]
  19. Dougherty ER, Hua J, Bittner ML. Validation of computational methods in genomics. Current Genomics. 2007;8(1):1–19. doi: 10.2174/138920207780076956. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. van de Vijver MJ, He YD, van 't Veer LJ. et al. A gene-expression signature as a predictor of survival in breast cancer. New England Journal of Medicine. 2002;347(25):1999–2009. doi: 10.1056/NEJMoa021967. [DOI] [PubMed] [Google Scholar]
  21. Bhattacharjee A, Richards WG, Staunton J. et al. Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proceedings of the National Academy of Sciences of the United States of America. 2001;98(24):13790–13795. doi: 10.1073/pnas.191502998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Devroye L, Gyorfi L, Lugosi G. A Probabilistic Theory of Pattern Recognition. Springer, New York, NY, USA; 1996. [Google Scholar]

Articles from EURASIP Journal on Bioinformatics and Systems Biology are provided here courtesy of Springer

RESOURCES