Detection of Visual Concepts and Annotation of Images Using Ensembles of Trees for Hierarchical Multi-Label Classification Ivica Dimitrovski 1,2 , Dragi Kocev 1 , Suzana Loskovska 2 , and Saˇ so Dˇ zeroski 1 1 Department of Knowledge Technologies, Joˇ zef Stefan Institute Jamova cesta 39, 1000 Ljubljana, Slovenia 2 Department of Computer Science, Faculty of Electrical Engineering and Information Technology Karpoˇ s bb, 1000 Skopje, Macedonia ivicad@feit.ukim.edu.mk, Dragi.Kocev@ijs.si, suze@feit.ukim.edu.mk, Saso.Dzeroski@ijs.si Abstract. In this paper, we present a hierarchical multi-label classifi- cation system for visual concepts detection and image annotation. Hi- erarchical multi-label classification (HMLC) is a variant of classification where an instance may belong to multiple classes at the same time and these classes/labels are organized in a hierarchy. The system is com- posed of two parts: feature extraction and classification/annotation. The feature extraction part provides global and local descriptions of the images. These descriptions are then used to learn a classifier and to annotate an image with the corresponding concepts. To this end, we use predictive clustering trees (PCTs), which are able to classify tar- get concepts that are organized in a hierarchy. Our approach to HMLC exploits the annotation hierarchy by building a single predictive cluster- ing tree that can simultaneously predict all of the labels used to anno- tate an image. Moreover, we constructed ensembles (random forests) of PCTs, to improve the predictive performance. We tested our system on the image database from the ImageCLEF@ICPR 2010 photo annotation task. The extensive experiments conducted on the benchmark database show that our system has very high predictive performance and can be easily scaled to large number of visual concepts and large amounts of data. 1 Introduction An ever increasing amount of visual information is becoming available in digital form in various digital archives. The value of the information obtained from an image depends on how easily it can be found, retrieved, accessed, filtered and managed. Therefore, tools for efficient archiving, browsing, searching and annotation of images are a necessity. A straightforward approach, used in some existing information retrieval tools for visual materials, is to manually annotate the images by keywords and then D. ¨ Unay, Z. C ¸ ataltepe, and S. Aksoy (Eds.): ICPR 2010, LNCS 6388, pp. 152–161, 2010. c  Springer-Verlag Berlin Heidelberg 2010