Efficient feature selection based on independent component analysis

Abstract

Feature selection, often used as a pre-processing step to machine learning, is designed to reduce dimensionality, eliminate irrelevant data and improve accuracy. In this paper, we introduce a novel approach to reduce dimensionality of the feature space by employing independent component analysis. While ICA is primarily a feature extraction technique, we use it here as a feature selection technique in a generic way. Our technique, called FSS_ICA, is more efficient than many of its competitors without loss in accuracy. FSS_ICA determines a set of statistically independent features instead of merely reducing the number of the original features. In applications FSS_ICA results in a smaller number of effective features than the relief attribute estimator, and it usually outperforms both the relief attribute estimator and CFS, when used as a pre-processing step for naive Bayes, instance based learning and decision trees. In addition, by disregarding some features, we demonstrate that in some cases FSS_ICA is more accurate than classification based on all features. Also, decision trees built from the pre-processed data are often significantly smaller than those derived from the original feature space. In addition, we also report the performance of ICA on a "real world" application in medical image segmentation.