ShapeExplorer: Querying and Exploring Shapes using Visual Knowledge Tong Ge 1 , Yafang Wang 1* , Gerard de Melo 2 , Zengguang Hao 1 , Andrei Sharf 3 , Baoquan Chen 1 1 Shandong University, China; 2 Tsinghua University, China; 3 Ben-Gurion University, Israel ABSTRACT With unprecedented amounts of multimodal data on the Internet, there is an increasing demand for systems with a more fine-grained understanding of visual data. ShapeEx- plorer is an interactive software tool based on a detailed analysis of images in terms of object shapes and parts. For instance, given an image of a donkey, the system may rely on previously acquired knowledge about zebras and dogs to automatically locate and label the head, legs, tail, and so on. Based on such semantic models, ShapeExplorer can then generate morphing animations, synthesize new shape contours, and support object part-based queries as well as clipart-based image retrieval. Keywords Shape Knowledge Harvesting, Shape Matching, Shape Seg- mentation, Shape Synthesis 1. INTRODUCTION In recent years, we have seen an explosion in the availability of multimodal data on the Internet, driven mostly by the ubiquity of mobile devices and online sharing platforms. Despite great advances in tasks such as object detection and tracking and multimedia retrieval, we still lack systems that provide more fine-grained semantic analyses of visual data. In their widely noted work, Deng et al. [4] introduced ImageNet, a hierarchical organization of visual knowledge in raw images, according to semantic categories and relations. We take a further step in this direction and utilize the se- mantics of individual parts, subparts, and their shapes to facilitate their interpretation and manipulation. We present ShapeExplorer, an interactive software tool that analyzes images of objects and locates and labels specific object parts. For instance, given an image of a donkey, it can draw on previously analyzed images of related objects, e.g. of zebras or even just of dogs, to infer the location and labels of likely parts such as the head, legs, tail, and so on. An analysis in terms of parts is motivated by extant evidence from cognitive research on human vision showing that shape parts play an * Corresponding author: yafang.wang@sdu.edu.cn c  2016, Copyright is with the authors. Published in Proc. 19th Inter- national Conference on Extending Database Technology (EDBT), March 15-18, 2016 – Bordeaux, France: ISBN 978-3-89318-070-7, on OpenPro- ceedings.org. Distribution of this paper is permitted under the terms of the Creative Commons license CC-BY-NC-ND 4.0. important role in the lower stages of object recognition [9]. Seeing a small part of an object often suffices for a human to be able to recognize the object, provided that the part is sufficiently unique [3, 2]. Still, fine-grained shape under- standing remains a challenging problem in computer vision. It appears that richer data is necessary so that systems can be equipped with the required background knowledge. Independently from the developments in computer vision, there has been considerable progress on automatically con- structing knowledge bases (KB), utilizing textual information to extract relational facts and attributes. Examples include YAGO [10, 12], DBpedia [1], Freebase (www.freebase.com), ConceptNet [7], and WebChild [11]. Often, the backbone of such KBs is a taxonomy of entity types or of part-whole relationships (e.g., Head isPartOf Horse). In our work, we have constructed a visual knowledge base called PartNet, for object parts and their shapes. Part- Net semantically describes objects in terms of their classes, parts, and visual appearance. Unlike regular KBs, it gathers examples of the shape contours of objects and object parts. Based on this, ShapeExplorer provides several higher-level operations, including (partial) shape querying, semantic mor- phing, shape synthesis, and part-based image retrieval using cliparts. 2. FRAMEWORK Figure 1: Flow diagram Hierarchical Part Exploration. Figure 1 provides an overview of ShapeExplorer’s operational flow. The system is based on the PartNet knowledge repository, which users can explore hierarchically. This knowledge is also used in several applications such as morphing and querying. PartNet is organized according to taxonomic categories (animals, dinosaurs, home appliances, etc.), sub-categories Demonstration Series ISSN: 2367-2005 648 10.5441/002/edbt.2016.70