Understanding Affect in Images Jana Machajdik Institute of Computer Aided Automation, Vienna University of Technology Favoritenstraße 9/183, 1040 Vienna, Austria jana@caa.tuwien.ac.at Allan Hanbury Information Retrieval Facility Tech Gate Vienna, Donau City Straße 1, 1220 Vienna, Austria a.hanbury@ir-facility.org Julian Stöttinger Institute of Computer Aided Automation, Vienna University of Technology Favoritenstraße 9/183, 1040 Vienna, Austria julian@caa.tuwien.ac.at Categories and Subject Descriptors H.3.1 [Information storage and retrieval]: Content Anal- ysis and Indexing; I.4.7 [Image processing and com- puter vision]: Feature Measurement General Terms Algorithms Keywords image affect, image classification, image features, art theory, psychology, emotional semantic image retrieval 1. MOTIVATION Due to the exploding number of images on the Internet, current search methods based on text surrounding images or user-supplied tags are no longer sufficient. For popu- lar tags or words, the returned images can fill many result pages. Hence techniques are needed to refine search results, allowing the searcher to home in on a smaller group of im- ages using additional non-textual information. Colour is one such type of information that is slowly becoming searchable in image search engines. In many cases it would be useful to be able to search for images based on their emotional content. An example could be a photo editor searching for photos to illustrate an article about the impact of cars on the climate. The target photos should have a more sombre atmosphere instead of showing cars as objects of desire in marketing photos. The semantic content of an image has the greatest impact on the emotional influence of any picture. However, algo- rithms to extract semantic content from images are still in their infancy. Nevertheless, the combination and arrange- ment of colours in images can be used to evoke emotions in the observer. In art theory, Itten formulated concepts for combining colors to induce an emotional effect in the ob- server and to achieve a harmonious image. Psychological experiments measuring the emotional response of observers Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MM’10, October 25–29, 2010, Firenze, Italy. Copyright 2010 ACM 978-1-60558-933-6/10/10 ...$10.00. to colour combinations have also been done. Reformulating these findings as features allows basic emotion classification of images to be done [1]. 2. DEMO The demo application is based on the approach presented in [1]. For every image uploaded, it extracts the pertinent features, and uses them in a classifier. For each image, a histogram is produced showing the distribution of the in- tensities of the emotions evoked by the images. As in [1], eight emotions are used: Amusement, Awe, Contentment, and Excitement as positive emotions; and Anger, Disgust, Fear, and Sadness as negative emotions. 3. APPLICATIONS Storing the extracted emotion histograms as image meta- data opens the door to a number of applications. Some photo search engines currently allow search results to be re- fined by specifying colours that should appear in the images. Google image search allows one of 12 colours to be chosen, while Exalead Chromatik 1 allows multiple choices from at least 96 colours. This type of interface could also be used with the eight emotion categories, where the results of a key- word search could be refined by choosing one of the emotion categories. Such an emotion histogram representation also simplifies integration into multi-modal content analysis and search applications. In applications for automatically illustrating text, the affect extracted from the text [2] could be used to choose images not only pertinent to the content but also to the affect. Music can also be classified by emotion [3], and one can envisage illustrating a song based on the content of the lyrics refined by the emotion of the music. 4. REFERENCES [1] J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In Proc. ACM Multimedia, 2010. [2] P. Subasic and A. Huettner. Affect analysis of text using fuzzy semantic typing. Fuzzy Systems, IEEE Transactions on, 9(4):483 –496, 2001. [3] K. Trohidis, G. Tsoumakas, G. Kalliris, and I. Vlahavas. Multi-label classification of music into emotions. In Proc. ISMIR, pages 325–330, 2008. 1 http://chromatik.labs.exalead.com/