BeHAVE: a Heatmap-based Audiovisual Representation of Personal Data Sihwa Park Media Arts and Technology, University of California, Santa Barbara sihwapark@mat.ucsb.edu ABSTRACT This paper presents BeHAVE, a web-based audiovisual piece that explores a way to represent personal behavioral data in a multimodal approach, by visualizing and sonify- ing this data based on the form of a heatmap visualization. As a way of typical visualization, it shows the location and time records of the author’s mobile phone use as clustered circles on an interactive map. In order to explore all of the data sequentially in a short period, it also transforms a year of data into sound and visuals based on a microsound timescale. By suggesting this multimodal data representa- tion as a means of revealing one’s personality or behav- ior in an audiovisual form, BeHAVE attempts not only to improve the perception and understanding of self-tracking data but also to arouse aesthetic enjoyment. 1. INTRODUCTION Sensing technology has been affecting our daily lives in various aspects. In particular, mobile and wearable de- vices have given rise to the Quantified Self (QS) 1 , a con- ceptual movement of self-knowledge through numbers by using self-tracking technology. In the era of QS, we can easily quantify ourselves and our activities with the help of self-tracking devices and applications. By looking at the numbers about quantified ourselves, we might be able to improve our mental or physical health and our performance for an activity. However, the purpose of quantifying one- self can be different in terms of making sense of data. In this regard, a multitude of artists and researchers has been trying to elicit sensations and evoke aesthetic curiosity by representing data in diverse ways [1, 2, 3]. Visualization can be a tool to help a person tell stories based on data and make data perceivable for others. Data sonification also can be used for the purpose of generating knowledge about data by rendering it into sound. When these two representations are combined, it can enhance the percep- tion and understanding of data with multisensory stimuli. From the perspective that aesthetics is about sensuous per- ception, the multimodal data representation can be not only pragmatic in terms of information design but also aesthetic as a form of data art [4]. It means that we can not only dis- cover the unexpected aspects of ourselves but also express 1 http://quantifiedself.com Copyright: ©2018 Sihwa Park et al. This is an open-access article dis- tributed under the terms of the Cre ative Com mons At tri bu tion Li cense 3.0 Un ported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ourselves aesthetically through multimodal data represen- tation or data art. In this regard, a web-based audiovisual piece BeHAVE (Behavioral data as Heatmap-based Audio-Visual Expres- sion) explores a way of revealing one’s personality or be- havior, which is potentially inherent in personal data, through multimodal data representation. Based on the data set about the author’s behavior of mobile phone use, which has been collected over a year by using a self-tracking app, BeHAVE visualizes and sonifies the geographic and temporal records of the phone use on an interactive online map. 2. RELATED WORK A variety of research has been conducted to visualize and sonify temporal and geo-related data in a various way. iSon- ic [5] sonifies georeferenced data to enable visually-impaired people to discover trends of data with nontextual sounds al- though it has only a practical purpose. COMPath [6] sug- gests the musical sonification of geo-related data on an in- teractive online map by generating sound via OSC or MIDI messages. Foo’s Two Trains 2 sonifies income data along with the New York City subway lines revealing income inequality with sound and visuals. Rosli [7] explores the possibilities of multimodal data representation through the visualization and sonification of NASA’s lightning data, based on the principles of Gestalt Psychology. Han and Tiwari [8] also use a sonification technique to enhance the perception of geographic and environmental data while vi- sualizing the data in multiple ways. Partially inspired by some of the works described above, BeHAVE has a unique approach to combining a conven- tional heatmap visualization with the form of data art by presenting a temporal data exploration mode generating abstract visuals and sound. It also focuses on personal data as an attempt to depict one’s personality or behavior in multimodal representation. 3. IMPLEMENTATION 3.1 Data Collection and Preprocessing For obtaining phone use data, an active screen time track- ing iPhone application ‘Moment’ 3 has been used since January 20, 2017. Basically, this app detects a phone pickup, which means a phone use location and time when a user turns his/her phone screen on, and calculates use duration until the screen is turned off. It also gives simple statisti- cal graphs of phone use data. The app exports all pickup 2 https://datadrivendj.com/tracks/subway 3 https://inthemoment.io