Combining feature ranking algorithms through rank aggregation Ronaldo C. Prati Centro de Matem´ atica, Computac ¸˜ ao e Cognic ¸˜ ao (CMCC) Universidade Federal do ABC (UFABC) Santo Andr´ e, SP, Brazil Email: ronaldo.prati@ufabc.edu.br Abstract—The problem of combining multiple feature rankings into a more robust ranking is investigated. A general framework for ensemble feature ranking is proposed, alongside four instan- tiations of this framework using different ranking aggregation methods. An empirical evaluation using 39 UCI datasets, three different learning algorithms and three different performance measures enable us to reach a compelling conclusion: ensemble feature ranking do improve the quality of feature rankings. Furthermore, one of the proposed methods was able to achieve results statistically significantly better than the others. I. I NTRODUCTION Feature selection has been an important research topic of pattern recognition, machine intelligence and data mining for a long time now [1], [2], [3]. There are plenty of studies pointing out that a subset of features may produce better predictive models (in terms of accuracy) than the entire feature set. This is because learning algorithms may be adversely af- fected by the presence of irrelevant and/or redundant features. Besides improving classification accuracy, feature selection significantly reduces the computational time necessary to induce the models, leading to simpler and faster classifiers for classifying new instances; facilitates data visualization and data understanding; and reduces the measurement and storage requirements. Feature selection can be broadly divided into three cat- egories. Filter methods are directly applied to datasets and generally provide an index for each feature quality by mea- suring some properties in the data. Wrapper and embedded methods, on the other hand, generally use a learning algorithm to indirectly assess the quality of each feature or feature sets. Wrapper uses the classification accuracy (or other performance measure) of a learning algorithm to guide a search process in the feature space and embedded methods use the internal parameters of some learning algorithms to evaluate features. A relaxed formalization of feature selection is feature rank- ing. In the feature ranking setting, a ranked list of features is produced and one can select the top ranked features, where the number of features to select can be analytically or experimentally determined or set by the user. Many feature selection algorithms use feature ranking as a principal or auxiliary step because of its simplicity, scalability, and good empirical success. Furthermore, a ranked list of feature might be interesting by itself, as for instance in the microarray analysis, where the ranked list of features is used by biologists to find correlations among top ranked features and some diseases [3]. Likewise as finding the best subset of features is impractical in most domains, we may expect that algorithms that construct the best ranking of features are unfeasible. Inspired from ensemble learning, where a combination of models is used to improve predictive performance, in this paper we investigate methods whereby the combination of independent feature rankings would result into a (hopefully) more robust feature ranking. These methods are based on different ranking aggre- gation approaches, and may take as input rankings produced by any combination of different feature ranking algorithms. The main contributions of this paper are: 1) A general and flexible framework for combining feature rankings. This framework is based into ranking aggre- gation mechanisms, and allows different instantiations depending on how the input base rankings are obtained as well as how the aggregation is carried out; 2) An empirical investigation of four concrete instantiations of this framework. These instantiations are based into four different aggregation approaches and are simple to implement, computationally cheap and in all but one case do not introduce new free parameters to set up; 3) An extensive empirical evaluation and analysis involving 39 datasets, three different learning algorithms and three different classification evaluation measures. Statistical analysis of the empirical evaluation enable us to reach a compelling conclusion: combining feature rankings do improve the quality of the ranked list of features. Furthermore, in our experiments, one of the proposed instantiations excels the others. This paper is organized as follows: Section II presents related work. Section III presents a general framework for ensemble feature ranking. Section IV describes the four in- stantiations of this framework using four different ranking aggregation approaches studied in this paper. Section V de- scribes the experimental setup used to evaluate the methods. Section VI presents and discusses the results, and Section VII concludes.