PhotoStylist: Altering the Style of Photos Based on the Connotations of Texts Siamul Karim Khan (B ) , Daniel (Yue) Zhang, Ziyi Kou, Yang Zhang, and Dong Wang Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, USA {skhan22,yzhang40,zkou,yzhang42,dwang5}@nd.edu Abstract. The need to modify a photo to reflect the connotations of a text can arise due to multifarious reasons (e.g., a musician might modify a photo in the album cover to better reflect the connotations in her song lyrics). An interesting observation is that different styles of photos con- vey different feelings. In this paper, we propose the PhotoStylist scheme to effectively modify the style of an input photo to represent the conno- tations in an input text. Existing methods that aim to transfer emotions into photos rely on an emotion class being provided as input and modify the overall color of photos based on the input emotion class, generating unrealistic colors for many objects in the image. To address these limita- tions, we design PhotoStylist, a novel deep-learning-based approach, to alter the individual style of each object in the photo in a way that the connotations of the input text are naturally and effectively embedded into the modified photos. Evaluation results on the Amazon Mechanical Turk (MTurk) show that our scheme can achieve output photos signifi- cantly closer to the connotations of the input text than the output photos from the state-of-the-art baselines. 1 Introduction In many real world applications, an individual may need to modify a photo to match the connotations (i.e., the inherent sentiments and themes) of a given input text. For example, a writer may share an edited photo in her book to reflect the connotations of her writing. With the rise of massive data dissemination opportunities [17, 18], a frequent need for such photo modifications emerge in online social media posts that pair images with user input texts [23]. In this paper, we develop an AI system that can automatically alter the style of a photo to effectively match the connotation in the text (e.g., photo caption) given by the user (as shown in Fig. 1). Previous efforts on altering photos based on emotions, known as Emotional Color Transfer, primarily focus on changing the color characteristics of the photo based on a predefined set of emotion classes [8, 10, 11, 16]. In particular, these emotional color transfer techniques change the entire photo based on an emotion class provided by the user, which is a one- size-fits-all solution. The problem we focus on in this paper is more challenging because we are not provided a single emotion class a priori but rather we aim c  Springer Nature Switzerland AG 2021 K. Karlapalem et al. (Eds.): PAKDD 2021, LNAI 12712, pp. 642–654, 2021. https://doi.org/10.1007/978-3-030-75762-5_51