published in the Proceedings of CIR2000, Brighton, UK, May 2000 available at <URL:http://www.cis.hut.fi/picsom/publications.html> The PicSOM Retrieval System: Description and Evaluations Markus Koskela, Jorma Laaksonen, Sami Laakso, and Erkki Oja Laboratory of Computer and Information Science, Helsinki University of Technology P.O.BOX 5400, Fin-02015 HUT, Finland Abstract We have developed an experimentalsystem called PicSOM for retrieving images similar to a given set of reference images in large unannotated image databases. The technique is based on a hierarchical variant of the Self-Organizing Map (SOM) called the Tree Structured Self-Organizing Map (TS-SOM). Given a set of reference images, PicSOM is able to retrieve another set of images which are most similar to the given ones. Each TS-SOM is formed using a different image feature representation like color, texture, or shape. A new technique introduced in PicSOM facilitates automatic combination of the responses from multiple TS-SOMs and their hierarchical levels. This mechanism adapts to the user’s preferences in selecting which images resemble each other. In this paper, a brief description of the system and a set of methods applicable to evaluating retrieval performance of image retrieval applications are presented. 1 Introduction Content-based image retrieval (CBIR) has been a subject for active research since the initial releases of the first notable CBIR systems such as QBIC [3] and Photobook [10] in the mid 90’s. The task of developing effective products based on CBIR has, however, proven to be extremely difficult. Due to the limitations of computer vision, the current CBIR systems have to rely only on rather low-level features extracted from the images. Therefore, images are typically described by rather simple features characterizing the color content, different textures, and primitive shapes detected in them. On the other hand, humans can easily detect distinct objects and use high-level semantic concepts in image recognition. As a result, humans routinely group together images which can be visually very different and, for a computer, mimicking this behavior is a very challenging task. For the development of effective image retrieval applications, one of the most urgent issues is to have widely- accepted performance assessment methods for different features and approaches. However, quantitative measures for the performance of an image retrieval system are problematic due to the subjectivity of human perception. As each user of a retrieval system has individual expectations, there does not exist a definite right answer to an image query. Also, there does not exist any widely accepted performance assessment methods. As a result, objective and quantitative comparisons between different algorithms or image retrieval systems based on different approaches are difficult to perform. Due to the lack of standard methods in this application area, the Moving Picture Experts Group (MPEG) has also started to work on a content representation standard for multimedia information search, filtering, management and processing called MPEG-7 [8]. 2 PicSOM The PicSOM image retrieval system is designed as a framework for generic research on algorithms and methods for content-based image retrieval. The system is based on querying by pictorial example (QBPE), which is a common retrieval paradigm in current CBIR applications. With QBPE, the queries are based on example images shown either from the database itself or some external location. The user identifies these example images as relevant or non- relevant to the current retrieval task and the system uses this information to select such images the user is most likely to be interested in. The accuracy of the queries is then improved by relevance feedback [11] which is a form of supervised learning adopted from traditional text-based information retrieval. In relevance feedback, the previous human-computer interaction is used to refine subsequent queries to better approximate the need of the user. The Challenge of Image Retrieval, Brighton, 2000 1