Vol.:(0123456789) Wireless Personal Communications https://doi.org/10.1007/s11277-020-07903-0 1 3 Smart Handheld Based Human Activity Recognition Using Multiple Instance Multiple Label Learning Jayita Saha 1  · Dip Ghosh 2  · Chandreyee Chowdhury 3  · Sanghamitra Bandyopadhyay 2 Accepted: 29 October 2020 © Springer Science+Business Media, LLC, part of Springer Nature 2020 Abstract Human activity recognition (HAR) and monitoring is benefcial for many medical applica- tions, such as eldercare and post-trauma rehabilitation after surgery. HAR models based on smartphone’s accelerometer data could provide a convenient and ubiquitous solution to this problem. However, such models are mostly concerned with identifying basic activities such as ‘stand’/‘walk’ and thus the high-level context such as ‘walk in a queue’ for which a set of specifc activities is performed remain unnoticed. Consequently, in this paper, we design a HAR framework that can identify a group of activities (rather than a single basic activity) being performed in a time window, thus, enables us to extract more meaningful informa- tion about the subject’s overall context. An algorithm is designed to formulate HAR as a multi-instance multi-label (MIML) learning problem. The procedure of generating fea- ture bags of consecutive activity traces having multiple labels is formulated. In this work, the temporal relationship among activities is exploited to obtain a more comprehensive HAR model. Interestingly, the framework is found to completely/partially identify activity sequences that may not even be present in the training dataset. The framework is imple- mented and found to be working adequately when tested with real dataset collected from 8 users for 12 diferent activity combinations. MIML-kNN is found to provide maximum average precision (around 90%) even for an unseen test data-set. Keywords Activity recognition · Composite activity · Accelerometer · Semi supervised learning · Multi-instance and multi-label learning * Jayita Saha gjai.2000@gmail.com Dip Ghosh dipgi2005@gmail.com Chandreyee Chowdhury chandreyee.chowdhury@gmail.com Sanghamitra Bandyopadhyay sanghami@isical.ac.in 1 Department of Artifcial Intelligence and Data Science, Koneru Lakshmaiah Education Foundation, Deemed to be University, Hyderabad, India 2 Machine Intelligence Unit, Indian Statistical Institute, Kolkata, India 3 Department of CSE, Jadavpur University, Kolkata, India