12 A Feature-Word-Topic Model for Image Annotation and Retrieval CAM-TU NGUYEN, National Key Laboratory for Novel Software Technology, Nanjing University, China NATSUDA KAOTHANTHONG and TAKESHI TOKUYAMA, Tohoku University, Japan XUAN-HIEU PHAN, University of Engineering and Technology, VNUH, Vietnam Image annotation is a process of finding appropriate semantic labels for images in order to obtain a more convenient way for indexing and searching images on the Web. This article proposes a novel method for image annotation based on combining feature-word distributions, which map from visual space to word space, and word-topic distributions, which form a structure to capture label relationships for annotation. We refer to this type of model as Feature-Word-Topic models. The introduction of topics allows us to efficiently take word associations, such as {ocean, fish, coral} or {desert, sand, cactus}, into account for image annotation. Unlike previous topic-based methods, we do not consider topics as joint distributions of words and visual features, but as distributions of words only. Feature-word distributions are utilized to define weights in computation of topic distributions for annotation. By doing so, topic models in text mining can be applied directly in our method. Our Feature-word-topic model, which exploits Gaussian Mixtures for feature-word distributions, and probabilistic Latent Semantic Analysis (pLSA) for word-topic distributions, shows that our method is able to obtain promising results in image annotation and retrieval. Categories and Subject Descriptors: H.3.1 [Information Storage and Retrieval]: Content Analysis and Indexing—Indexing Methods, Linguistic Processing; H.3.3 [Information Storage and Retrieval]: Infor- mation Search and Retrieval—Retrieval Modal General Terms: Algorithms, Design, Experimentation Additional Key Words and Phrases: Image retrieval, image annotation, topic models, multi-instance multil- abel learning, Gaussian mixtures, probabilistic Latent Semantic Analysis (pLSA) ACM Reference Format: Nguyen, C.-T., Kaothanthong, N., Tokuyama, T., and Phan, X.-H. 2013. A feature-word-topic model for image annotation and retrieval. ACM Trans. Web 7, 3, Article 12 (September 2013), 24 pages. DOI: http://dx.doi.org/10.1145/2516633.2516634 1. INTRODUCTION As high-resolution digital cameras become more affordable and widespread, the use of digital images is growing rapidly. At the same time, online photo-sharing Web sites and social networks (Flickr, Picasa, Facebook, etc.), hosting hundreds of millions of pictures, have quickly become an integral part of the Internet. On the other hand, traditional This article is an extension of a shorter version presented at CIKM’10 [Nguyen et al. 2010]. Authors’ addresses: C.-T. Nguyen, National Key Laboratory for Novel Software Technology, Nanjing Uni- versity, Nanjing 210046, China; University of Engineering and Technology, Vietnam National Univer- sity, Hanoi, Vietnam. Mailbox 603, 163 Xianlin Avenue, Qixia District, Nanjing 210046, China; email: nguyenct@lamda.nju.edu.cn; ncamtu@gmail.com. N. Kaothanthong, T. Tokuyama, Graduate School of Information Sciences, Tohoku University; Aobayama Campus, GSIS Building, Sendai, Japan; email: {natsuda,tokuyama}@dais.is.tohoku.ac.jp. X.-H. Phan, University of Engineering and Technology, Viet- nam National University, Ha Noi; 144 Xuan Thuy street, Cau Giay District, Hanoi, Vietnam; email: hieupx@vnu.edu.vn. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org. c  2013 ACM 1559-1131/2013/09-ART12 $15.00 DOI: http://dx.doi.org/10.1145/2516633.2516634 ACM Transactions on the Web, Vol. 7, No. 3, Article 12, Publication date: September 2013.