1530-437X (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JSEN.2014.2331970, IEEE Sensors Journal 1 Smart Diary: A Smartphone-based Framework for Sensing, Inferring and Logging Users’ Daily Life Jilong Liao, Zhibo Wang, Lipeng Wan, Qing Cao and Hairong Qi Abstract—In this paper, we present Smart Diary, a novel smartphone based framework that analyzes mobile sensing data to infer, predict, and summarize people’s daily activities, such as their behavioral patterns and life styles. Such activities are then used as the basis for knowledge representation, which generates personal digital diaries in an automatic manner. As users do not need to intentionally participate into this process, Smart Diary is able to make inferences and predictions based on a wide range of information sources, such as the phones’ sensor readings, locations, and interaction history with the users, by integrating such information into a sustainable mining model. This model is specifically developed to handle heterogeneous and noisy sensing data, and is made to be extensible in that users can define their own logic rules to express short-term, mid-term, and long-term event patterns and predictions. Our evaluation results are based on the Android platform, and they demonstrate that the Smart Diary framework can provide accurate and easy-to- read diaries for end users without their interventions. I. I NTRODUCTION A diary is a collection of records on what has happened over a period of time, usually sorted by dates. A personal diary usually records one’s experiences, thoughts, and feelings on events in the outside world. Some personal diaries, such as those by Anne Frank, have become widely read books and the basis of plays and films. Besides their value in terms of being a hobby and a literature source, diaries also serve important roles for researchers in social science. Some studies use diaries to study the correlation between certain activities (e.g., exercise) and health problems (e.g., diabetes) in certain populations [34]. However, recent studies have reported the insufficiency of this research method due to the quality on diaries collected from a group of volunteers [11]. The contents are usually incomplete, fragmented, or incorrect. Furthermore, long-term studies with consistent volunteer participation are particularly challenging, if possible at all. With the emergence of the Internet, the form of diaries has undergone significant changes. Online journals, blogs, micro- blogs, tweets, and Facebook status all contain continuous updates on a person’s life events. Moreover, the complexity of people’s lives have increased dramatically with the tight Jilong Liao, Lipeng Wang, Qing Cao and Hairong Qi are with the Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN, 37912 USA. Zhibo Wang is with the School of Computer, Wuhan University, China, 430072, and Department of Electrical Engineering and Com- puter Science, University of Tennessee, Knoxville, USA, 37909. (Emails: {jliao2, zwang32, lwan1, cao, hqi}@utk.edu). Copyright c 2013 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org. interweaving of both “online” and “offline” lives. On the other hand, however, with more and more activities attracting our attention, we have less and less time to write down the stories belonging to our unique life. Motivated by the need to not only keep user’s personal lives, but also to provide verified data sources for social studies, we propose a fully software-based approach to generate per- sonalized diaries. Our approach is enabled by the widespread adoption of smartphones in recent years. According to a report by IDC [21], the worldwide smartphone vendors will ship a total of 918.6 million smartphones in 2013, up 27.2% from the previous year. These smartphones are usually equipped with both a wide range of sensors and communication capabilities such as Wi-Fi and Bluetooth. Our key idea in this paper is that, as the user carries the smartphone, this device will be able to detect the environment context through its sensors and infer users’ behavior. For instance, Fig 1 presents a user’s life activities (e.g. motion, SMS, phone call, etc.) throughout a day, from which we can infer many different types of user- interested events and record them. The inference results are then summarized into human-readable forms (diaries). We name this application as “Smart Diary”, as it is purely based on software, and does not require user intervention. Although the central idea of Smart Diary is simple, there are numerous challenges that we have to address to bring it into reality. Because of the battery limitations on the smartphone, it is usually not acceptable to keep the application active for extended periods of time. Therefore, tradeoffs have to be made in terms of the amount of data that the application collects, and the accuracy that it achieves. Specifically, we summarize the following challenges that our design addresses: First, Smart Diary has to address the challenge of user privacy. Because the dairies generated contain unique daily events of users, such as their entertainments, social contacts, and health conditions, it is necessary to ensure that such information is not leaked to others. This problem is particularly urgent as recently exposed security holes have demonstrated that it is possible for these phones to be hacked through malware or NFC (near field communication) techniques [2]. Second, as each user has their own life styles and experi- ences, we need a mechanism to support the personalization of Smart Diary. In particular, different users may interpret the same sensing readings in their own manners, leading to user-specific output in diary contents. Indeed, the output from Smart Diary should reflect the user’s personality, and consist of the most valuable and interesting experiences, which can be different from user to user. In addition, the system’s framework should be adaptive to the dynamic requirements of the users.