Robust and Fast Heart Rate Monitoring Based on Video Analysis and Its Application Kouyou Otsu 1(&) , Qiang Zhong 1 , Das Keya 1 , Hisato Fukuda 1 , Antony Lam 2 , Yoshinori Kobayashi 1 , and Yoshinori Kuno 1 1 Saitama University, Saitama, Japan {otsu,keya0612,fukuda,kuno}@cv.ics.saitama-u.ac.jp, {kyochu,yosinori}@hci.ics.saitama-u.ac.jp 2 Mercari, Inc., Tokyo, Japan antonylam@mercari.com Abstract. Techniques to remotely measure heartbeat information are useful for many applications such as daily health management and emotion estimation. In recent years, some methods to measure heartbeat information using a consumer RGB camera have been proposed. However, it is still a dif ficult challenge to accurately and quickly measure heart rate from videos with significant body movements. In this study, we propose a video-based heart rate measurement method that enables robust measurement in real-time by improving over pre- vious slower methods that used local regions of the facial skin for measurement. From experiments using public datasets and self-collected videos, it was con- firmed that the proposed method enables fast measurements while maintaining the accuracy of conventional methods. Keywords: Heart rate sensing Á Remote photoplethysmography Á Image processing Á Healthcare 1 Introduction Heartbeat information such as heart rate (HR) and heart rate variability (HRV) is a useful index for daily health management. Generally, medical instruments such as an electrocardiogram (ECG) have been used for measuring heartbeat information. In recent years, several inexpensive measuring devices using photoplethysmography (PPG) have spread widely [1] making it easier to use heartbeat information for mea- suring cardiopulmonary function during activities such as playing sports or daily life. Heartbeat information is also known to reflect human internal psychological states such as tension or relaxation. In recent years, numerous human emotion estimation methods based on speaking analysis, and facial expression recognition have been proposed [2] but there are also many methods using heartbeat information obtained from ECG and PPG [3]. Such biological information reflects internal emotions that are not actively expressed by humans and cannot be felt by other people. Therefore, heartbeat information is a promising modality to achieve high accuracy emotion estimation. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 T. Ahram (Ed.): AHFE 2020, AISC 1213, pp. 250–257, 2021. https://doi.org/10.1007/978-3-030-51328-3_35