A Deep Transfer Learning Based Visual Complexity Evaluation Approach to Mobile User Interfaces Eren Akça, Ömer Özgür Tanrıöver * Department of Computer Engineering, Ankara University, Ankara 06830, Turkey Corresponding Author Email: tanriover@ankara.edu.tr https://doi.org/10.18280/ts.390511 ABSTRACT Received: 29 August 2022 Accepted: 20 October 2022 Visual complexity is an important factor affecting the efficiency and functionality of user interfaces. Its impact on the user's impression and the usability is significant, especially for mobile applications with constraints such as layout size, on screen keys and small input fields. Conventional approaches for visual complexity evaluation of user interfaces are either based on user evaluations with surveys or based on pre-specified formal metrics or on heuristics. Alternatively, in this study, we have explored the effectiveness of deep learning models for visual complexity evaluation, specifically, of mobile user interfaces. We have experimented with five state of the art pre-trained deep learning models known to be effective for computer vision tasks, namely, VGG16, DenseNet121, MobileNetv2, GoogleNet and ResNet152 were trained with 3635 different mobile user interface images as login, menu, search and settings. Furthermore, in order to validate the effectiveness of this approach, a new validation dataset and survey application was developed and an evaluation study was conducted with 98 participants where 7309 comparison result were obtained from the study. It was found that the agreement rate between the results of deep learning models and the user evaluations was up to 78% and 74% on the average. The high to moderate agreement rate between the results of deep learning models and the user evaluations reveals that this approach can be useful for designers in visual complexity evaluation of mobile user interfaces. Keywords: mobile user interface evaluation, transfer learning, visual complexity analysis 1. INTRODUCTION In recent years, people's access to mobile devices such as smartphones and tablets has become easier, and with regard to this technological change, the amount and use of mobile applications has grown dramatically. Considering that there are millions of mobile applications being developed in areas like online shopping, food delivery, online banking, social media, entertainment and online learning, it is obvious that mobile applications are the main gateway to the virtual world. Hence, for service/product providers or application developers, users interface complexity perception is one of the most important components to reach large audiences. Therefore, user interfaces that are perceived less complex, easy to use and in line with user`s mental representation are becoming more in demand. On the other hand, mobile applications with less intuitive, distractive or complex user interfaces (UI) may hinder user acceptance. From the user's point of view, almost every user screen of the application should be easily comprehensible but also respond to user expectations. From the application developer's point of view, it is a tedious process to develop low-complex graphical user interfaces especially for mobile applications taking into account constraints such as limited screen sizes, on screen keys and small input fields. Many different factors effect visual complexity, such as graphical design of the UI, the layout, the number and variety of components and the relationship between these components and so on. Complexity analysis of visual interface and its understandability has long been the focus of attention of researchers. While in earlier studies [1-3], perceptual visual complexity had been tried to be defined based on perceptual attributes, in later studies, visual complexity of UIs has been measured by user evaluation with surveys, developing formal metrics and models [4-6], heuristics [7] to machine learning based [8] methods. Although formal metric and model based methods are mostly used, due the need of a blend of different metrics for different attributes and their potential complex interaction this problem is still not fully resolved. One possible reason for this is that the success of this method is tightly dependent various assumptions such as number and combination of metrics and metric coefficients used in the conceived predictive model [4]. Another reason is that there may be other factors that affect the perception of visual complexity but still cannot be formulated precisely [5]. For the above stated reasons, as opposed to user evaluation, metric/model based methods; machine learning based approaches focus on visual complexity evaluation rely on machine learning models` understanding. Although, previously deep learning models for other type of general image visual complexity problems [9-11] and even for the quality assessment problem for 3D images [12] produced promising results, deep learning models for visual complexity of mobile UIs has not been studied thoroughly. The absence of latent factors that can be utilized in addition to existing metrics without directly involving human in the visual complexity analysis process encouraged us to explore pre-trained models that are known to be effective for computer vision tasks. Traitement du Signal Vol. 39, No. 5, October, 2022, pp. 1545-1556 Journal homepage: http://iieta.org/journals/ts 1545