Is Independence Good For Combining Classifiers? L.I. Kuncheva, C.J. Whitaker, C.A. Shipp School of Informatics, University of Wales, Bangor Gwynedd LL57 1UT, UK E-mail: l.i.kuncheva,c.j.whitaker @bangor.ac.uk R.P.W. Duin Faculty of Applied Sciences Delft University of Technology P.O. Box 5046, 2600 GA Delft, The Netherlands E-mail: duin@ph.tn.tudelft.nl on Pattern Recognition, Barcelona Spain, (ICPR’2000) Abstract Independence between individual classifiers is typically viewed as an asset in classifier fusion. We study the lim- its on the majority vote accuracy when combining depen- dent classifiers. statistics are used to measure the depen- dence between classifiers. We show that dependent classi- fiers could offer a dramatic improvement over the individual accuracy. However, the relationship between dependency and accuracy of the pool is ambivalent. A synthetic experi- ment demonstrates the intuitive result that, in general, neg- ative dependence is preferable. 1. Introduction Let be a set (pool) of classifiers such that , where , assigns a class label . The majority vote method of combining classifier deci- sions, one of many methods in this important research area [4, 5, 6, 7, 8, 10, 11, 12], is to assign the class label to that is supported by the majority of the classifiers . Finding independent classifiers is one aim of classifier fusion methods for the following reason. Let L be odd, , and all classifiers have the same classifi- cation accuracy . The majority vote method with indepen- dent classifier decisions gives an overall correct classifica- tion accuracy calculated by the binomial formula (1) where denotes the largest integer smaller than . The probability of a correct classification for and is shown in Table 1. Then the majority vote method with independent classifiers is guaranteed to give a higher accuracy than individual clas- sifiers when . Table 1. Tabulated values of the majority vote accuracy of independent classifiers with in- dividual accuracy 0.6480 0.6826 0.7102 0.7334 0.7840 0.8369 0.8740 0.9012 0.8960 0.9421 0.9667 0.9804 0.9720 0.9914 0.9973 0.9991 In this study we are interested in combining dependent classifiers and establishing a relationship between the de- pendence and the accuracy of the pool. If all classifiers are totally positively dependent (i.e., they are identical) there will be no improvement over . However, if there are nega- tively dependent, i.e., commit mistakes on strongly different objects, we could expect improvement over the predicted value for independent classifiers. 2. Dependency between classifiers Let be a labeled data set, coming from the classification problem in question. For each classifier we design an -dimensional output vec- tor of correct classification, such that , if recognizes correctly , and 0, otherwise. There are various statistics to assess the similarity of and [1]. The statistic for two classifiers is (2) where is the number of elements of Z for which and (see Table 2). For statistically independent classifiers, . varies between -1 and 1. Classifiers that tend to recognize the same objects correctly will have positive values of ,