BUDA.ART: A Multimodal Content-Based Analysis and Retrieval System for Buddha Statues Benjamin Renoust, Matheus Oliveira Franca, Jacob Chan, Van Le, Ayaka Uesaka, Yuta Nakashima, Hajime Nagahara renoust@ids.osaka-u.ac.jp Institute for Datability Science, Osaka University Osaka, Japan Jueren Wang Yutaka Fujioka fujioka@let.osaka-u.ac.jp Graduate School of Letters, Osaka University Osaka, Japan Figure 1: An overview of the BUDA.ART system, from search, to exploration and 3D visualization. ABSTRACT We introduce BUDA.ART, a system designed to assist researchers in Art History, to explore and analyze an archive of pictures of Buddha statues. The system combines different CBIR and classical retrieval techniques to assemble 2D pictures, 3D statue scans and meta-data, that is focused on the Buddha facial characteristics. We build the system from an archive of 50,000 Buddhism pictures, identify unique Buddha statues, extract contextual information, and provide specific facial embedding to first index the archive. The system allows for mobile, on-site search, and to explore similarities of statues in the archive. In addition, we provide search visualization and 3D analysis of the statues. CCS CONCEPTS • Information systems → Search interfaces; Multimedia and multimodal retrieval; Data cleaning; • Human-centered com- puting → Visualization systems and tools; • Applied com- puting → Fine arts. KEYWORDS Art History, Multimedia Database, Search system, 2D, 3D Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). MM ’19, October 21–25, 2019, Nice, France © 2019 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-6889-6/19/10. https://doi.org/10.1145/3343031.3350591 1 INTRODUCTION The spread and evolution of Buddhism across Asia is the topic of many books [19, 27]. Multiple theories are confronting on which path(s) this spread took place across the Asian subcontinent, reach- ing the coasts of the Japanese archipelago along the Silk Road [15, 19, 21, 27]. Buddhism brought many works of art and their rules so that local people would craft new artworks by themselves. giving their identity to the resulting style [16]. Nowadays, only a few ex- perts can identify these works subjective to their own knowledge, sometimes disputing explanations [10]. In order to investigate Buddhism at a large scale, we analyze a large archive of Buddhism related documents through the produced art. To do so, we focus on the representation of Buddha, which is central to Buddhism art. Although their exist statues of many different types, their construction respects canons 1 which have been normalized over the centuries. Despite the rules, time and travels allowed for quite an evolution among the style of statues, and aligning many of these statues may allow us to capture the traces of this evolution [26]. Using modern face detection and recognition [24], we focus on the faces of Buddha. Our experts have accumulated a large amount of photos and pictures related to Buddhism, that they cannot bring with them every time they visit a temple or museum, even less hav- ing an overview of their collection. Statues are 3D objects but 2D pictures usually poorly convey their spatial structure. So we build BUDA.ART (for Buddha Archive Anaysis and Retrieval, Fig. 1) a web-based system that combines and deliver all three aspects: knowledge/metadata, 2D pictures, and 3D structure, such that ex- perts can query, search, and explore Buddha statues even on field. 1 A canon of art refers to a universal set of rules and principles establishing the funda- mentals and/or optimal. arXiv:1909.12932v1 [cs.CV] 17 Sep 2019