Capacitive Sensing Based On-board Hand Gesture Recognition with TinyML Sizhen Bian sizhen.bian@dfki.de German Research Center for Artifcial Intelligence Kaiserslautern, Germany Paul Lukowicz paul.lukowicz@dfki.de German Research Center for Artifcial Intelligence Kaiserslautern, Germany ABSTRACT Although hand gesture recognition has been widely explored with sensing modalities like IMU, electromyography and camera, it is still a challenge of those modalities to provide a compact, power- efcient on-board inferencing solution. In this work, we present a capacitive-sensing wristband surrounded by four single-end elec- trodes for on-board hand gesture recognition. By deploying a single convolutional hidden layer as the classifer at the sensing edge, the wristband can recognize seven hand gestures from a single user with an accuracy of 96.4%. CCS CONCEPTS · Human-centered computing → Human computer interac- tion (HCI); Gestural input. KEYWORDS human-computer interaction, capacitive sensing, hand gesture recog- nition, edge computing, TinyML ACM Reference Format: Sizhen Bian and Paul Lukowicz. 2021. Capacitive Sensing Based On-board Hand Gesture Recognition with TinyML. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers (UbiComp-ISWC ’21 Adjunct), September 21–26, 2021, Virtual, USA. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3460418.3479287 1 INTRODUCTION Hand gesture recognition is one of the most widely explored inter- action techniques for user intention interpreting. A power-efcient, on-board hand gesture recognition solution will enable a broad range of long-term applications, such as sign language recognition for deaf and speech-impaired community, device control for assis- tive input, games and virtual reality, etc. A typical example is that from 2021, Apple Watch supports a motion-controlled cursor for an assistive touch action, which relies on the built-in inertial sen- sors and accomplishes the primary "next" and "confrm" functional- ity. Besides inertial sensors, electromyography(EMG) and camera are the most widely explored sensing modalities for hand gesture recognition. EMG captures the electrical signals of skeletal muscles Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). UbiComp-ISWC ’21 Adjunct, September 21–26, 2021, Virtual, USA © 2021 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-8461-2/21/09. https://doi.org/10.1145/3460418.3479287 Figure 1: Capacitance variation around the wrist Figure 2: Wrist-worn prototype with only four capacitive sensing electrodes working in single-end mode during a particular action with multiple skin-touched surface elec- trodes. However, the limitations like contact-demand, computing delay remain a challenge for EMG-based real-life applications. The vision-based approach also faces limitations like heavy computing load, lighting variation, occlusion afection, privacy issues, etc. Although recent development on the far edge AI has enabled the deep neural network models to be deployed on weak devices by compressing the models with techniques like pruning and quantiza- tion, the on-board inference performance is still a challenge when balancing recognition accuracy, computation delay, and power con- sumption. To achieve the best in all those scores for hand gesture recognition, a fundamental approach is to fnd a more efective sensing modality delivering less data but with sufcient gesture information. Capacitive sensing [1, 2] has been verifed with great efectiveness for activity recognition beneftting from the straight- forward signal pattern of body motion. Here we explored the on- board hand gesture recognition with capacitive sensing. With a simple prototype, capacitive based design showed remarkable on- board recognition result with high accuracy, short inference delay, and very low power consumption. Capacitive based hand gesture recognition is not novel, a few previous works on capacitive-based hand gesture recognition [3, 4] have presented impressive of-line result. However, the previous supposed methods rely on luxury electrode pairs working in diferential mode to sense the muscle deformation, which enables more gestures to be recognized, but results in the deployment complexity, since more electrodes will bring the electrode position into consideration (otherwise the ac- curacy drops greatly). Besides that, the inference was carried out 4