Fatigue Estimation using Facial Expression features and Remote-PPG Signal Masaki Hasegawa 1 , Kotaro Hayashi 2 and Jun Miura 3 Abstract—Currently, research and development of lifestyle support robots in daily life is being actively conducted. Health- case is one such function robots. In this research, we develop a fatigue estimation system using a camera that can easily be mounted on robots. Measurements taken in a real environment have to be consider noises caused by changes in light and the subject’s movement. This fatigue estimation system is based on a robust feature extraction method. As an indicator of fatigue, LF/HF-ratio was calculated from the power spectrum of RR interval in the electrocardiogram or the blood volume pulse (BVP). The BVP can be detected from the fingertip by using the photoplethysmography (PPG). In this study, we used a contactless PPG: remote-PPG (rPPG) detected by the luminance change of the face image. Some studies show facial expression features extracted from facial video are also useful for fatigue estimation. dimension reduction of past method using LLE spoiled the information in the large dimention of feature. We also developed a fatigue estimation method with such features using a camera for the healthcare robots. It used facial landmark points, line-of-sight vector, and size of the ellipse fitted with eyes and mouth landmark points. Therefore, proposed method simply use time-varying shape information of face like size of eyes, or gaze direction. We verified the performance of proposed features by the fatigue state classification using Support Vector Machine (SVM). I. INTRODUCTION The worldwide declining birthrate and society’s aging have become increasingly serious problems. Moreover, there are various problems in each country such as the self inflicted problem in the USA or the lifestyle diseases in Japan. The healthcare robot is one of the prevailing solutions to such problems. Robots can perform both physical and cognitive tasks. Among them, we focus on fatigue estimation. A fatigue state is important to get signs of crucial diseases or accidents. If a healthcare robot can estimate the fatigue state, it can advise a user to go to a hospital or report to an administrator in the workplace. It is important that a healthcare robot can say “Aren’t you tired?” or warn of signs of danger by fatigue detection. Facial expressions (FE) may reveal causes of the fatigue [1]. Recent studies have proposed detecting driver fatigue using a camera [2], [3], [4], because detection by a contract sensor has a high accuracy but is unsanitary and troublesome to use. Most robots have cameras for self-localization, robot mapping, 1 Masaki Hasegawa is with the Toyohashi University of Technology, Aichi 441-8580 Japan (phone: +08-0532-446826; e-mail: hasegawa@aisl.cs.tut.ac.jp). 2 K. Hayashi is with the Toyohashi University of Technology, Aichi 441-8580 Japan (phone: +08-0532-446826; e-mail: hayashik@cs.tut.ac.jp). 3 J. Miura is with the Toyohashi University of Technology, Aichi 441- 8580 Japan (phone: +08-0532-446773; e-mail: jun.miura@tut.jp). Person Classifier Negative / Positive Features Feature Extractor Sensor Data Fatigue Estimation System Robot Action Planner Estimation Result Why don’t you take a rest? I’m exhausted... Action (ex. suggestion) Robot Fig. 1: Image of a robot system for fatigue estimation object recognition and obstacle avoidance. Therefore, we propose a fatigue estimation method that measures heart rate variability from expression features measured by a camera. Heart rate variability, or the time fluctuation of electrocar- diogram RR variability, is well known as a fatigue estimation indicator. To measure it without using the cardiogram, the reflection of light from the skin can be used to calculate the BVP [5]. There are two ways of contactless measurement with BVP. One is using the waveform received from the Doppler radar, and the other is the light reflected from the skin. Current fatigue estimation studies show availability of the BVP-based method [6]. However, one should avoid having a tired user estimate fatigue. Therefore some fatigue estimation studies focus on the use of facial information obtained by a camera. Ji et al. used FE features to estimate fatigue by using an IR LED and a CCD camera [7]. The FE feature-based fatigue estimation methods must be compact which can be implemented into personal service robots. In this research, we use a simple webcam to get facial information because these robots can not use stereo camera due to the size. Additionally, the FE feature-based fatigue estimation method for healthcare robots need to deal with the various lighting environments and the target movement. The FE feature-based estimation must avoid failing to detect or incorrectly detect these changes. In this study, we aim to develop a robust method overcome these noises. II. RELATED WORKS A. Contactless BVP measurement BVP is a method of analyzing heart rate variability (HRV), because its peak appears in time with the beat. Generically, autonomic nervous activity can be able to estimete by HRV, so that HRV is used as an index of fatigue Photoplethys- mography (PPG) is a popular method to measure BVP [5], measuring reflected light and absolution quantity fluctuation as BVP. However, this method can only use optical sensors Proc. 2019 IEEE Int. Conf. on Robot and Human Interactive Communication (RO-MAN 2019), New Delhi, India, Oct. 2019.