Under consideration for publication in Knowledge and Information Systems Class Imbalance Revisited: a New Experimental Setup to Assess the Performance of Treatment Methods Ronaldo C. Prati 1 , Gustavo E. A. P. A. Batista 2 and Diego F. Silva 2 1 Centro de Matem´atica, Computa¸ c˜aoeCogni¸c˜ao Universidade Federal do ABC, Santo Andr´ e, Brazil 2 Instituto de Ciˆ encias Matem´aticas e de Computa¸ c˜ao Universidade de S˜ao Paulo, S˜ao Carlos, Brazil Abstract. In the last decade, class imbalance has attracted a huge amount of attention from researchers and practitioners. Class imbalance is ubiquitous in Machine Learning, Data Mining and Pattern Recognition applications; therefore, these research commu- nities have responded to such interest with literally dozens of methods and techniques. Surprisingly, there are still many fundamental open-ended questions such as “Are all learning paradigms equally affected by class imbalance? ”, “What is the expected per- formance loss for different imbalance degrees? ” and “How much of the performance losses can be recovered by the treatment methods? ”. In this paper, we propose a simple experimental design to assess the performance of class imbalance treatment methods. This experimental setup uses real data sets with artificially modified class distributions to evaluate classifiers in a wide range of class imbalance. We apply such experimental design in a large-scale experimental evaluation with twenty- two data sets and seven learning algorithms from different paradigms. We also propose a statistical procedure aimed to evaluate the relative degradation and recoveries, based on confidence inter- vals. This procedure allows a simple yet insightful visualization of the results, as well as provide the basis for drawing statistical conclusions. Our results indicate that the expected performance loss, as a percentage of the performance obtained with the bal- anced distribution, is quite modest (below 5%) for the most balanced distributions up to 10% of minority examples. However, the loss tends to increase quickly for higher degrees of class imbalance, reaching 20% for 1% of minority class examples. Support Vector Machine is the classifier paradigm that is less affected by class imbalance, being almost insensitive to all but the most imbalanced distributions. Finally, we show that the treatment methods only partially recover the performance losses. On average, typ- Received Sep 26, 2013 Revised Apr 17, 2014 Accepted Sep 19, 2014