Incorporating Concept Ontology to Enable Probabilistic Concept Reasoning for Multi-Level Image Annotation Yuli Gao Dept of Computer Science UNC-Charlotte Charlotte, NC 28223, USA ygao@uncc.edu Jianping Fan * Dept of Computer Science UNC-Charlotte Charlotte, NC 28223, USA jfan@uncc.edu ABSTRACT To enable automatic multi-level image annotation, we have addressed two inter-related important issues: (1) more ef- fective framework for image content representation and fea- ture extraction to characterize the middle-level semantics of image contents; (2) new framework for hierarchical proba- bilistic image concept reasoning and detection. To address the first issue, salient objects are used as the semantic building blocks to characterize the middle-level semantics of image contents effectively while reducing the image analy- sis cost significantly. We have proposed three approaches to designing the detection functions for automatic salient object detection, and automatic function selection is also supported to find the “right” assumptions of the princi- pal visual properties for the corresponding salient object classes. To address the second issue, we have proposed a novel framework to incorporate the concept ontology to achieve hierarchical probabilistic image concept reasoning for multi-level image annotation. The concept ontology for a large-scale public image database called LabelMe is semi- automatically derived from the available image labels by us- ing WordNet. The image concepts at the first level of the concept ontology are used to characterize the most specific semantics of image contents with the smallest variations, and their correspondences with the semantic building blocks (i.e., salient objects) are well-defined and can be modeled ac- curately by using Bayesian networks. In addition, the pre- dictions of the appearances of the higher-level image con- cepts with large variations are adopted by the underlying concept ontology or by combining the available predictions of the appearances of their children concepts through hi- erarchical Bayesian networks. Our experiments on a large public dataset have shown that our framework for hierar- chical probabilistic image concept reasoning is scalable to diverse image contents (i.e., large amount of salient object classes) with large within-category variations. * Correspondence author, this project is supported by NSF IIS-0601542. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MIR ’06, October 22-28, 2006, Santa Barbara, CA, USA Copyright 2006 ACM 1-58113-737-0/03/0008 ...$5.00. Categories and Subject Descriptors I.4.8 [Image Processing and Computer Vision]: Scene Analysis-object recognition, H.2.8 [Database Management]: Database Applications - image databases. General Terms Algorithms, Measurement, Experimentation Keywords: Concept Ontology, Hierarchical Probabilistic image concept Reasoning, Bayesian Network, Multi-Level Image Annotation. 1. INTRODUCTION In the past decades, we have witnessed a remarkable growth of image archives on the Internet which have now become a major source of Internet content. When large-scale image collections come into view, there is an urgent need to support automatic image annotation based on their contents so that semantic image retrieval via keywords can be achieved [1-6]. The main obstacle for automatic image annotation is the semantic gap between the high-level human perceptions of image semantics (i.e., image concepts) and the low-level visual features due to the diversity of image contents with large within-category variations. Thus there is an urgent need to develop robust frameworks that are able to bridge the semantic gap effectively. Semantic image classification is one promising approach for automatic image annotation, but its performance largely depends on two inter-related issues: (1) suitable frameworks for image content representation and feature extraction; (2) effective algorithms for image concept reasoning or image classifier training. To address the first issue, the underly- ing visual patterns that are used for image content repre- sentation and feature extraction should be able to charac- terize the middle-level image semantics effectively and effi- ciently [1-6]. To address the second issue, robust techniques for image concept reasoning or image classifier training are needed to bridge the semantic gap successfully. Because of the uncertainty and diversity of the appearances of image concepts, the outputs for image concept reasoning or classifi- cation should be probabilistic. In addition, one single image may contain different meanings at multiple semantic levels [13-15]: (1) image semantics interpreted by the underlying object classes; (2) image concepts at multiple semantic lev- els which can be organized hierarchically by using concept ontology. Thus images may be similar at multiple semantic levels [13-15]: (a) same object class; (b) same category of image concepts at different semantic levels.