Selecting a multi-label classification method for an interactive system N.Y. Nair Benrekia 1,2 , P. Kuntz 2 , and F. Meyer 1 1 Orange Labs, AV. Pierre Marzin 22307 Lannion cedex France (yacinenoureddine.nairbenrekia, franck.meyer)@orange.com 2 LINA, la Chantrerie-BP 50609, 44360 NANTES cedex France pascale.kuntz@univ-nantes.fr Abstract Interactive classification-based systems engage users to coach learning algorithms to take into account their own individual preferences. However most of the recent interactive systems limit the users to a single-label classification, which may be not expressive enough in some organization tasks such as film classification, where a multi-label scheme is required. The objective of this paper is to compare the behaviors of 12 multi-label classification methods in an interactive framework where “good” predictions must be produced in a very short time from a very small set of multi-label training examples. Experimentations highlight important perfor- mance differences for 4 complementary evaluation measures (Log-Loss, Ranking- Loss, Learning and Prediction Times). The best results are obtained for Multi-label k Nearest Neighbours (ML-kNN), Ensemble of Classifier Chains (ECC) and Ensem- ble of Binary Relevance (EBR). 1 Introduction The usual classification systems do not allow users to directly interact with the learning models. Consequently, in practice, their results may deviate from their preferences. Modeling human preferences remains a difficult task, espe- cially for personalized systems where classical interviews are out of reach and large-scale behavioral logs are not available. An alternative is to embed the user into the learning process via an interactive visual support (Ware et al. (2000)). The user plays the role of a trainer for an automatic classification algorithm and steers it towards his/her desired concepts. More precisely, in a dynamical process, he/she can define a set of subjective labels L t on a set of training examples T t described by a set of features F and correct, if and/or when necessary, the labels predicted by an automatic classifier on a set of unlabeled examples S t . Such interactive machine learning process has recently received increas- ing attention and found applications in several domains. For instance, for document organization, iCluster (Drucker et al. (2011)) is an interactive,