e-Shoes: Smart Shoes for Unobtrusive Human Activity Recognition Cuong Pham, Nguyen Ngoc Diep, Tu Minh Phuong Computer Science Department and Machine Learning & Applications laboratory Posts and Telecommunications Institute of Technology, Hanoi, Vietnam Email: {cuongpv, diepnguyenngoc, phuongtm}@ptit.edu.vn Abstract—Many approaches to human activity recognition such as wearable based or computer vision based are obtrusive in the sense that they prevent the users from performing activities in a natural way, or they might raise privacy invasion concerns. This paper presents e-Shoes - smart shoes for unobtrusive human activity recognition. E-Shoes are shoes instrumented with tiny wireless accelerometers embedded inside the insole of the shoes. The sensors are seamless to the users making the system suitable for recognizing everyday activities. To analyze sensor signals, we propose a convolution neutral networks (CNN) model that automatically learns features from sensing data and makes predictions about performing activities. We verify the effectiveness of the approach with a real dataset that covers seven daily activities. The system achieved 93% accuracy in average, which is very promising, while being energy efficient and easy to use. I. INTRODUCTION Human activity recognition is a fundamental task in a wide range of practical applications such as assistive technologies for healthier cooking [1], situated services [2], prompting step by step people with dementia for meal preparation [3], energy expenditure estimation [4], etc. These applications are significantly promising for enhancing people’s quality lives, elderly’s independent livings, and remote health monitoring. Over past decade, human activity recognition has made a stride progression. However, many activity recognition works are obtrusive to the users such as computer vision and wearable sensors. The activity recognition based on computer vision analyses the stream of images captured from cameras installed in pre-setting areas and often raises privacy invasion concerns. While activity recognition based on wearable sensors requires the users to wear some sensors on different parts of user’s body such as wrist, hip, knee, etc. It can be understandable that wearable sensors can often be obtrusive to users and have considerable potentials to impact on the performance of many tasks. Unobtrusive approaches to activity recognition include sensors embedded inside the environment (therefore people do not need to wear the sensors), so-called pervasive sensing, or sensors instrumented inside the smart phone. Several pervasive sensing activity recognition systems detect user’s everyday activities at home [5], [6] or kitchen [2], [7], [8] toward situated services for the elderly or task-based foreign language learning assistance [9]. Although pervasive sensing can hide sensors inside object surroundings, it is often pricey as it requires embedding numerous sensors into the environment. Minority of research work in activity recognition exploits sensors such as accelerometers instrumented in the smart phone [10]–[12] in which few works can discriminate activities including unknown activities (characterized as a background model) that are relevant for real-time implementation. Several mobile phone-based activity recognition investigations indicated that it is relatively difficult for the tradeoff between battery life, sensor sampling frequency and accuracy then deploying the recognition system of human activities in real-time on smart phones [13]. A significant attempt made for unobtrusively detecting human activities is to embed and hide the sensors inside the fabrics that can be worn on people such as shoes or textile [14]. Few works such as SmartStep [15] utilizes pressure sensors (12.5 mm FSR402, Interlink Electronics) and accelerometers attached on shoe’s insole for analyzing personalized gait (a pattern of how a person walks). In another work [16], the sensor LIS3L02AS4 is attached on the back base of the shoes for detecting activities such as walking or running. The smart shoes presented in those studies look a bit bulky (not really unobtrusive), therefore somehow impact on human activities. Other work such as [17] make the sensors hidden to the people, however they are expensive. In this work, we propose e-Shoes, smart shoes embedded with tiny wireless accelerometer sensors developed by researchers at Open Lab, Newcastle University for the Ambient kitchen project [2], [3]. The sensor is completely integrated inside the insole of the shoes so that they are hidden and unobtrusive to the users. In addition, we propose deep convolutional neural networks for detecting a set of low-level activities (those can be performed within seconds to minutes) including running, walking, standing, jumping, kicking, cycling. A preliminary experiment on a dataset collected from 10 subjects who performed 7 different activities has demonstrated the feasibility of unobtrusive activity recognition with smart shoes. II. RELATED WORK Approaches to activity recognition can be divided into two categories: obtrusive and unobtrusive. The former might raises privacy invasion concerns or likely be uncomfortable in use for users, while the latter can make technologies hidden from the users, hence allowing the users to perform their activities in natural manners. Obtrusive activity recognition often includes computer vision [6], [8] or wearable sensors [18], [19], [29], [30] to detect human activities from the sequence of user’s motion. Computer vision analyzes images (i.e. frames) of the video