Adaptive Activity Recognition with Dynamic Heterogeneous Sensor Fusion Ming Zeng, Xiao Wang, Le T. Nguyen, Pang Wu, Ole J. Mengshoel, Joy Zhang Department of Electrical and Computer Engineering Carnegie Mellon University Moffett Field, CA, USA Email: {ming.zeng, sean.wang, le.nguyen, pang.wu, ole.mengshoel, joy.zhang}@sv.cmu.edu Abstract—In spite of extensive research in the last decade, activity recognition still faces many challenges for real-world applications. On one hand, when attempting to recognize various activities, different sensors play different on different activity classes. This heterogeneity raises the necessity of learning the optimal combination of sensor modalities for each activity. On the other hand, users may consistently or occasionally annotate activities. To boost recognition accuracy, we need to incorporate the user input and incrementally adjust the model. To tackle these challenges, we propose an adaptive activity recognition with dynamic heterogeneous sensor fusion framework. We dynamically fuse various modalities to characterize different activities. The model is consistently updated upon arrival of newly labeled data. To evaluate the effectiveness of the proposed framework, we incorporate it into popular feature transformation algorithms, e.g., Linear Discriminant Analysis, Marginal Fisher’s Analysis, and Maximum Mutual Information in the proposed framework. Finally, we carry out experiments on a real-world dataset col- lected over two weeks. The result demonstrates the practical implication of our framework and its advantage over existing approaches. Keywords—Activity Recognition, Deep Learning, Convolutional Neural Network I. I NTRODUCTION The rapid spread of wearable devices with sensing capa- bilities offers the opportunity for human activity recognition. Knowing a user’s activity over a period of time enables applications such as continuous monitoring of user behavior, physical activity monitoring [18], abnormal activity detec- tion [3], elderly care [23] and physical activity recognition [2]. The activity recognition is usually formulated as a classi- fication problem [15]. Many classification methods have been leveraged in previous studies. The decision table, decision tree and naive Bayes classifier are experimented to recognize twenty predefined daily activities [2]. The support vector machine (SVM) and k-nearest neighbor (kNN) algorithm are used to perform fall detection [26]. The linear discriminant analysis and hidden Markov models are introduced to recog- nize predefined workshop activities [13]. However, most of the aforementioned activity recognition approaches frame activity recognition as a “static” machine learning problem, which assumes the types of activities to be recognized are predefined. This assumption does not hold for many real-life applications such as Lifelogger [6], social activity pattern detection, etc. In these systems, the number of activities is not constant. Moreover, different users have their own definition of a “meaningful activity”. It is infeasible to foresee activities that users may be interested in. So in the training phase, the systems are required to learn the most useful sensor modality combination according to different kinds of activity classes. We call these systems Adaptive Activity Recognition Systems In order to recognize personal, unseen activities, some incremental methods [22], [1] are proposed. However, their results are similar to those of non-personalized models [12], indicating that the feature selection is crucial for activity recog- nition [12]. The semantic attribute sequence based models are also used for recognizing unseen new activities [5], [4], but still fail to consider the influence of different features. We have developed a dynamic heterogeneous sensor fusion framework for adaptive activity recognition. The key idea is to find the most discrimnative combination of sensor modalities (motion, sound, location, time of the day, WiFi environment, etc.) for each activity. For example, if all sensor modalities are leveraged, the system will not be able to recognize that the user is walking unless he walks with the same motion, at the same location and at the same time as the training walking examples. On the other hand, when the user annotates new types of activities, the system needs to adjust the model to use additional sensor modalities in order to discriminate a new activity from existing activities. Specifically, when the user labels an activity as walking, the system learns that motion feature is sufficient to recognize this activity. Several days later if the user labels a new type of activity: grocery shopping, which has very similar motion as walking, the system will need to incorporate location information to distinguish these two types of activities. Then the “motion” and “location” sensors play important roles in this case. The sensor weight is a value representing the importance of a sensor. To examine the effectiveness of the proposed framework, we integrate several feature transformation meth- ods including Linear Discriminant analysis (LDA), Marginal Fisher’s Analysis (MFA) and Maximum Mutual Information (MMI) algorithm. To summarize, we develop a practical dynamic heteroge- neous sensor fusion framework, which addresses the challenge of dynamic sensor fusion in adaptive activity recognition. The key contributions of the paper are highlighted as follows: • We propose a sensor fusion framework to learn sensor weights for each activity class so that activities are eas- ier to be discriminated in the new distance space. We implement several feature transformation algorithms MobiCASE 2014, November 06-07, Austin, United States Copyright © 2014 ICST DOI 10.4108/icst.mobicase.2014.257787