Send to

Choose Destination
Expert Syst Appl. 2014 Nov 1;14(15):6945-6958.

Bias and Stability of Single Variable Classifiers for Feature Ranking and Selection.

Author information

Medical Image Analysis Laboratory, Department of Radiology, Henry Ford Health System, Detroit, MI 48202, USA ; Department of Computer Science, University of Maryland, College Park, MD 20740, USA.
Medical Image Analysis Laboratory, Department of Radiology, Henry Ford Health System, Detroit, MI 48202, USA ; Control and Intelligent Processing Center of Excellence (CIPCE), School of Electrical and Computer Engineering, University of Tehran, Tehran 14395, Iran.
College of Engineering, Wayne State University, Detroit, MI 48202, USA.


Feature rankings are often used for supervised dimension reduction especially when discriminating power of each feature is of interest, dimensionality of dataset is extremely high, or computational power is limited to perform more complicated methods. In practice, it is recommended to start dimension reduction via simple methods such as feature rankings before applying more complex approaches. Single Variable Classifier (SVC) ranking is a feature ranking based on the predictive performance of a classifier built using only a single feature. While benefiting from capabilities of classifiers, this ranking method is not as computationally intensive as wrappers. In this paper, we report the results of an extensive study on the bias and stability of such feature ranking method. We study whether the classifiers influence the SVC rankings or the discriminative power of features themselves has a dominant impact on the final rankings. We show the common intuition of using the same classifier for feature ranking and final classification does not always result in the best prediction performance. We then study if heterogeneous classifiers ensemble approaches provide more unbiased rankings and if they improve final classification performance. Furthermore, we calculate an empirical prediction performance loss for using the same classifier in SVC feature ranking and final classification from the optimal choices.


AdaBoost; Bias; Dimension Reduction; Feature Ranking; Feature Selection; K-Nearest Neighbors; Logistic Regression; Multilayer Perceptron; Naïve Bayes; Random Forests; Single Variable Classifier; Stability; Support Vector Machines

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center