Aspect Coherence for Graph-Based Image Labelling Giuseppe Passino, Ioannis Patras, Ebroul Izquierdo * Queen Mary, University of London Mile End Road, London, E1 4NS United Kingdoms {giuseppe.passino,ioannis.patras,ebroul.izquierdo}@elec.qmul.ac.uk Abstract Semantic image labelling is the task of assigning each pixel of an image to a semantic category. To this end, in low-level image labelling, a labelled training set is available. In such a situation, structural information about the correlation between different image parts is particularly important. When a part- based inference algorithm is used to perform the association of semantic classes to pixels, however, a good choice on how to use structural information is crucial for learning an efficient and generalisable probabilistic model for the labelling task. In this paper we introduce an efficient way to take into account correlation between different image parts, embedding the parts relationships in a graph built according to aspect coherence of neighbouring image patches. 1 Introduction Low-level image analysis and inference is an important tool for semantic image classification, object detection and segmenta- tion. One important challenge related to these tasks is to asso- ciate high-level concepts with images or patches within them. These research areas are experiencing a period of particular excitement from the research community due to the number of practical applications involved. In particular, some exam- ples of systems that would benefit from advances in semantic image analysis are human-computer interaction systems, im- age database browsing and management, and Internet image search. The roles of a low-level image analysis system are in- deed broad as it can be used by itself or integrated in a more complex system for multimedia search, indexing and retrieval. In the field of content-based multimedia analysis research, a low-level image analysis system by definition does not take ad- vantages of preexistent, high level concepts information and structures (e.g., ontologies). Instead, it entirely relies on fea- tures extracted from the images being analysed. The associa- tion between concepts and image features is given in a set of annotated examples that can be used for training. Particularly appealing is the problem of part-based image analysis, in which features are extracted from specific areas rather than from the whole image. In this scenario the inference system can deal with the appearance associated to pictured objects instances instead of whole scenarios. Due to the ever-growing compu- tational capabilities of modern computers, taking into account * The research leading to this paper was supported by the European Com- mission under contract FP6-027026, Knowledge Space of semantic inference for automatic annotation and retrieval of multimedia content - K-Space, and under the COST Action 292. local features in a probabilistic framework is becoming gradu- ally more feasible. In this paper a system for automatic semantic image segmenta- tion based on patches obtained through oversegmentation and a discriminative probabilistic graphical model is presented. Ul- timately, we split the problem of grouping image areas having similar aspect models, and learning the association of aspect and high-level concepts. Oversegmentation can be then tackled considering aspect dissimilarity between neighbouring patches. The advantages of using such an approach are: • considering patches as basic elements of the probabilistic model eases the inference process, shifting the problem on a coarser domain; • an accurate oversegmentation is likely to place patches boundaries on the actual object boundaries, giving a good basis for obtaining accurate pixel-level labelling and co- herent feature extraction; • the use of a discriminative graphical model allows to di- rectly model and learn the a posteriori probability distri- bution for the semantic categories given the observation (extracted features); this simplifies the learning process and makse thus possible to consider richer information in the inference framework [5]. The paper is organised as follows: Section 2 presents a brief review of works related to the low-level semantic image la- belling problem. In Section 3 the segmentation algorithm used to obtain the image patches is discussed, and the features as- sociated to the so-obtained patches are discussed in Section 4. The learning process is treated in Section 5, and experimental results are presented in Section 6. Finally, Section 7 closes the paper giving final comments on the proposed approach. 2 Related Work Although image segmentation is an old problem in computer vision, semantic segmentation, that is, the association of a se- mantic category to each image pixel, is relatively recent. In this case the problem lies on two different levels: at a first level, there is the need to accurately identify objects boundaries (the classical segmentation problem); additionally, different areas of the images have to be associated coherently to high-level concepts that are often broad and not related with a single as- pect.