Pattern Recognition 35 (2002) 1661–1674 www.elsevier.com/locate/patcog Spatial arrangement of color in retrieval by visual similarity S. Berretti, A. Del Bimbo ∗ , E. Vicario Dipartimento Sistemi e Informatica, Universit a di Firenze, via S. Marta, 3, 50139 Firenze, Italy Received 5 July 2001 Abstract In image search based on chromatic similarity, the eectiveness of retrieval can be improved by taking into account the spatial arrangement of colors. This can serve both to distinguish images with the same colors in dierent arrangement, and to capture the similarity between images with dierent colors but similar arrangements. We propose a model of representation and comparison which attains this goal by partitioning the image in separate entities and by associating them with individual chromatic attributes and with mutual spatial relationships. The eectiveness of the proposed model is assessed in a user-based evaluation. Experimental results show the capability of the model to join and balance chromatic and spatial similarity, thus improving the eectiveness of retrieval with respect to representations based on a global histogram. ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Image content description; Spatial relationships; Retrieval eectiveness evaluation 1. Introduction With the recent advances in multimedia technology, on-line libraries of digital images are assuming an ever increasing relevance within a wide range of informa- tion systems. Eective access to such archives requires that conventional searching techniques based on exter- nal textual keywords be complemented by content-based queries addressing appearing visual features of searched data [1,2]. To this end, a number of models have been experimented which permit to represent and compare im- ages in terms of quantitative indexes of visual features [3]. In particular, representations based on chromatic in- dexes have been widely experimented and comprise the basic backbone of most commercial and research re- trieval engines such as QBIC, Virage, Visual Seek or * Corresponding author. Tel.: +39-055-479-6262; fax: +39-055-479-6363. E-mail addresses: delbimbo@dsi.uni.it (A. Del Bimbo), berretti@dsi.uni.it (S. Berretti), vicario@dsi.uni.it (E. Vicario). Picasso [4 –7]. This apparently depends on the capability of color-based models in combining robustness of auto- matic construction with a relative perceptual signicance of models. In the basic approach, the chromatic content of the overall image is represented by a global histogram: the (three-dimensional) space of colors is partitioned into a nite set of reference tessels, each associated with a bin representing the quantity of pixels whose color belongs to the tessel itself [4]. The similarity between two images is thus evaluated by comparing bins and their distribution [8]. In so doing, the evaluation of similarity does not ac- count for the spatial arrangement and coupling of colors over the image. This plays a two-fold role in the user’s perception, serving to distinguish images with common colors and to perceive similarities between images with dierent colors but similar arrangements. To account for both these aspects, chromatic information must be asso- ciated with individual spatial entities identied over the image. In Ref. [9], the image is partitioned into blocks along a xed grid and each block is associated with an indi- 0031-3203/02/$22.00 ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII:S0031-3203(01)00161-3