G. Qiu et al. (Eds.): PCM 2010, Part I, LNCS 6297, pp. 515–526, 2010. © Springer-Verlag Berlin Heidelberg 2010 Unifying Content and Context Similarities of the Textual and Visual Information in an Image Clustering Framework Bashar Tahayna 1 , Saadat M. Alashmi 1 , Mohammed Belkhatir 2 , Khaled Abbas 3 , and Yandan Wang 1 1 Monash University, Sunway Campus 2 Université Claude Bernard Lyon 1, France 3 University of Malaya, Malaysia {bashar.tahayna,saadat.m.alhashmi, yandan.wang}@infotech.monash.edu.my, mohammed.belkhatir@iut.univ-lyon1.fr, khaled@perdana.um.edu.my Abstract. Content-based image retrieval (CBIR) has been a challenging prob- lem and its performance relies on the efficiency in modeling the underlying content and the similarity measure between the query and the retrieved images. Most existing metrics evaluate pairwise image similarity based only on image content, which is denoted as content similarity. However, other schemes utilize the annotations and the surrounding text to improve the retrieval results. In this study we refer to content as the visual and the textual information belonging to an image. We propose a representation of an image surrounding text in terms of concepts by utilizing an online knowledge source, e.g., Wikipedia, and propose a similarity metric that takes into account the new conceptual representation of the text. Moreover, we combine the content information with the contexts of an image to improve the retrieval percentage. The context of an image is built by constructing a vector with each dimension representing the content (visual and textual/conceptual) similarity between the image and any image in the collec- tion. The context similarity between two images is obtained by computing the similarity between the corresponding context vectors using the vector similarity functions. Then, we fuse the similarity measures into a unified measure to evaluate the overall image similarity. Experimental results demonstrate that the new text representation and the use of the context similarity can significantly improve the retrieval performance. Keywords: Clustering, Classification, Content-based, Similarity measures, bipitrate graphs. 1 Introduction and Related Work Human’s perceptual abilities, analysis and reasoning tend to describe images with words, even with biased description, effectively and efficiently. Unfortunately, this is not the case with the current computer systems and algorithms. For example, in Yahoo