Color Navigation by Qualitative Attributes for Fashion Recommendation Yeongnam Chae, Jiu Xu, Bj¨ orn Stenger and Soh Masuko Rakuten Institute of Technology, Rakuten, Inc., 1-14-1 Tamagawa, Setagaya-ku, Tokyo, Japan {yeongnam.chae, jiu.xu, bjorn.stenger, so.masuko}@rakuten.com Abstract—This paper proposes a novel method to navigate a color palette using attributes recognized from speech input. Our target application is a fashion recommender system for mobile e-commerce. Starting with a selected color, a user can request to show items of a different color by qualitative attributes (e.g. ‘a little cuter’). These attributes are mapped to a query vector within the Lab color space in order to select the next color. The system distinguishes 85 attributes, each with three different possible magnitudes. This color navigation by speech was demonstrated in a mobile fashion recommender system. The proposed model is validated in a user study with 196 subjects. I. I NTRODUCTION The use of mobile devices for e-commerce has seen a strong increase in volume in recent years, as more and more prod- ucts can be conveniently placed at the consumers’ fingertips. More recently, e-commerce chatbots have been launched as a new shopping platform to enhance the shopping experience. However, challenges remains between real-world shopping experiences and digital shopping experiences. For example, one challenge is allowing the user to efficiently navigate the product space via emotional attributes. We consider the scenario in which users are presented with fashion items of a particular color and wish to explore items of related colors with a particular quality. For example, when a user is presented with a dress in burgundy red, she may wish to also look at other dresses in a ‘more cheerful’ color. In order to support this, we introduce attribute-based color navigation. An attribute, input via speech, is converted to a query vector in the Lab-color space by interpolation in polar coordinates. As a use case of the proposed method, we developed fashion concierge mobile application and implemented the attribute-based color navigation as shown in Fig. 1. II. COLOR NAVIGATION In this section, we describe the details of the color naviga- tion by using qualitative attributes. We use the Lab color space, which is designed to approximate the human visual perception of colors [1]. The Euclidean distance in Lab color space approximately corresponds to the perceptual color differences [2]. It makes that we can use linear regression to model the perceptual color differences in the human vision. In order to efficiently search data sets with millions of items, we use a discrete color palette (JIS Z8102 [3], Japanese Industrial Standard) containing 269 colors. This standard links color values with color names and attributes. In the JIS color Fig. 1. Mobile Fashion Concierge. The proposed fashion recommender system using speech input for color navigation. The displayed product selection is updated based on the input of qualitative attributes, which is mapped to a vector in the color space. description system, the hue dimension is discretized to 20 bins and colors within each bin are arranged in 2D on the brightness and saturation plane. Associated with each color is a qualitative attribute, such as “brighter” or “subdued”. A. Query Vector Generation To interpret the attribute-based query, we map it to a query vector in color space. In order to establish the relation between color values and attributes, we make use of a public dataset relating example color palettes and attributes [4]. Some relationships such as “vivid” are picked up from the JIS color system. To navigate along the ‘attribute axis’ we define a path from the current source color to the target color. The search query is required to lie on this path, the distance from the source color is dependent on the magnitude value in the search query (“a little”, “more”, “much more”). There are many ways to define a path between a source and target color pair. We consider two different options: (1) linear interpolation in the 3-dimensional Lab space, and (2) linear interpolation in the polar coordinate representation of