171 Copyright © 2013, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. Chapter 9 DOI: 10.4018/978-1-4666-2940-0.ch009 INTRODUCTION The last few years we have witnessed the phenom- enal success of Web 2.0, which has enabled users to create and exchange self-organized resources on the web, resulting in a huge amount of resources in “folksonomy” systems such as Flickr, YouTube, and De.li.ci.ous. As of October 2009, Flickr for example hosted more than four billion images with manual, user-annotated tags. Tagging functions are widely available in Web 2.0 applications. A tag is a non-hierarchical keyword or term assigned to Jyh-Ren Shieh National Taiwan University, Taiwan Ching-Yung Lin IBM T. J. Watson Research Center, USA Shun-Xuan Wang National Taiwan University, Taiwan Ja-Ling Wu National Taiwan University, Taiwan Building Multi-Modal Relational Graphs for Multimedia Retrieval ABSTRACT The abundance of Web 2.0 social media in various media formats calls for integration that takes into account tags associated with these resources. The authors present a new approach to multi-modal media search, based on novel related-tag graphs, in which a query is a resource in one modality, such as an image, and the results are semantically similar resources in various modalities, for instance text and video. Thus the use of resource tagging enables the use of multi-modal results and multi-modal queries, a marked departure from the traditional text-based search paradigm. Tag relation graphs are built based on multi-partite networks of existing Web 2.0 social media such as Flickr and Wikipedia. These multi-partite linkage networks (contributor-tag, tag-category, and tag-tag) are extracted from Wikipedia to construct relational tag graphs. In fusing these networks, the authors propose incorporating contributor-category networks to model contributor’s specialization; it is shown that this step signifcantly enhances the accu- racy of the inferred relatedness of the term-semantic graphs. Experiments based on 200 TREC-5 ad-hoc topics show that the algorithms outperform existing approaches. In addition, user studies demonstrate the superiority of this visualization system and its usefulness in the real world.