To Keep or not to Keep: An Expectation–oriented Photo Selection Method for Personal Photo Collections Andrea Ceroni L3S Research Center Leibniz Universität Hannover ceroni@L3S.de Vassilios Solachidis Information Technologies Institute / CERTH vsol@iti.gr Claudia Niederée L3S Research Center Leibniz Universität Hannover niederee@L3S.de Olga Papadopoulou Information Technologies Institute / CERTH olgapapa@iti.gr Nattiya Kanhabua L3S Research Center Leibniz Universität Hannover kanhabua@L3S.de Vasileios Mezaris Information Technologies Institute / CERTH bmezaris@iti.gr ABSTRACT When selecting important photos from a personal photo col- lection – e.g. for creating an enjoyable sub-collection for revisiting or preservation – photos are not considered in iso- lation. Therefore, collection-level criteria are also taken into account by automated photo selection methods. However, the typical two-step process of first clustering and subse- quently picking from the clusters seems to overstress cov- erage as a criterion when applied to the task of selecting the photos most important to a user. We, therefore, pro- pose a novel expectation-oriented photo selection method, which combines a variety of collection-level and image-level selection criteria in a flexible way. In our evaluation, which is based on large real-world personal photo collections with overall more than 18,000 images, we show that our method outperforms state-of-the-art photo selection methods. In ad- dition, the proposed method does not rely on any manual annotations, making it applicable in realistic settings of per- sonal photo collections. Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: Selection Process Keywords Photo Selection; User Expectations; Clustering; Coverage 1. INTRODUCTION With digital photography and the many possible devices, photo taking is effortless and tolerated nearly everywhere. This makes us easily ending up with hundreds of photos, for example, when returning from a holiday trip. Furthermore, photos are also taken of more mundane motives, such as food or everyday scenarios, further increasing the number Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICMR ’15, June 23-26, 2015, Shanghai, China. Copyright 2015 ACM 978-1-4503-3274-3/15/06 ...$15.00 http://dx.doi.org/10.1145/2671188.2749372. of photos to be dealt with. So, what to best do with all of these photos? With the decreased storage prices it is not a problem to store the photos somewhere. However, this often ends up as a kind of “dark archive” of photo collections, which are rarely accessed (and enjoyed) again. The mere size of the collection makes going through them as well as manual annotation and sorting of photos tedious tasks. Furthermore, there is the risk of losing photos by a ran- dom form of “digital forgetting” [8]: over decades storage devices break down, and formats and storage media become obsolete, making random parts of photo collections inacces- sible. Just consider, how difficult it would be today to access photos stored years ago in .mos format in a floppy disk. Both the risk of dark archives and of digital forgetting suggest to select, supported by automated methods, the most important photos and to invest some effort into keeping them enjoyable and accessible. However, to foster adoption, such automated selection methods have to keep the level of user investment low. We do not rely on any additional user investment such as photo annotation with text [14, 17, 18] or eye tracking information [21], because we believe it is exactly the reluctance of further investment that lets large photo collections unattended on our hard disks. When developing methods for semi-automatic photo se- lection, it is important to consider human expectations and practices. An important observation is that photo selection is a complex, partially subjective process, which does not consider images in isolation. Selection decisions also take the context of the other photos in the collection and of the photos already selected into account. Therefore, the aspect of coverage is used in a variety of photo selection methods [3, 10, 14]. In more detail, photo selection is modeled as a two- step process of first clustering the photo collection (for re- flecting sub-events in the collection) and subsequently pick- ing the most representative photos from the clusters. While coverage surely plays an important role for many photo se- lection tasks (see e.g. [21]), we believe that the complex deci- sion making in selecting important and personal photos can be better modeled by avoiding the strict splitting into a two step process, which overstresses the role of coverage. We suggest to model a multifaceted notion of image importance driven by user expectations, which represents what photos users perceive as important and would have selected. In this paper, we present an expectation-oriented method for photo selection, which relies on such a model of image Proc. ACM Int. Conf. on Multimedia Retrieval (ICMR 2015), Shanghai, China, June 2015.