Climbing Activity Recognition and Measurement with Sensor Data Analysis Iustina Ivanova Free University of Bozen-Bolzano Bozen-Bolzano, Italy iivanova@unibz.it Marina Andrić Free University of Bozen-Bolzano Bozen-Bolzano, Italy maandric@unibz.it Andrea Janes Free University of Bozen-Bolzano Bozen-Bolzano, Italy ajanes@unibz.it Francesco Ricci Free University of Bozen-Bolzano Bozen-Bolzano, Italy fricci@unibz.it Floriano Zini Free University of Bozen-Bolzano Bozen-Bolzano, Italy fzini@unibz.it ABSTRACT The automatic detection of climbers’ activities can be the basis of software systems able to support trainers to assess the climber performance and to defne more efective training programs. We propose an initial building block of such a system, for the unobtru- sive identifcation of the activity of someone pulling a rope after fnishing the ascent. We use a novel type of quickdraw, augmented with a tri-axial accelerometer sensor. The acceleration data gener- ated by the quickdraw during the climbs are used by a Machine Learning classifer for detecting the rope pulling activity. The ob- tained results show that this activity can be detected automatically with high accuracy, particularly by a Random Forest classifer. More- over, we show that data acquired by the quickdraw sensor, as well as the detected rope pulling, can also be used to benchmark climbers. CCS CONCEPTS · Human-centered computing → Ubiquitous computing; · Computing methodologies → Machine learning. KEYWORDS Sports analysis; Sensors; Climbing; Activity Recognition ACM Reference Format: Iustina Ivanova, Marina Andrić, Andrea Janes, Francesco Ricci, and Floriano Zini. 2020. Climbing Activity Recognition and Measurement with Sensor Data Analysis. In Companion Publication of the 2020 International Conference on Multimodal Interaction (ICMI ’20 Companion), October 25–29, 2020, Virtual event, Netherlands. ACM, New York, NY, USA, 5 pages. https://doi.org/10. 1145/3395035.3425303 1 INTRODUCTION In recent years, climbing has become a popular recreational and competitive sport worldwide [1, 2], producing a growing interest Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from permissions@acm.org. ICMI ’20 Companion, October 25–29, 2020, Virtual event, Netherlands © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-8002-7/20/10. . . $15.00 https://doi.org/10.1145/3395035.3425303 in the development of innovative software solutions that could support the training of both professional and amateur climbers. Professional climbers follow rigorous training programs designed by climbing coaches. A coach assesses the climber during a climbing session by observation and then provides feedback by pointing to the weaknesses in their technique and suggesting appropriate training routes. Coaching is also desirable at amateur level; however, due to the large number of climbing enthusiasts, this service cannot practically be provided to each climber in indoor climbing gyms. Automatic or semi-automatic climbing assessment systems have the potential to support professional coaches, and to make coaching more widely available. In order to implement such training software solutions, one should be able to automatically detect and classify climbing activi- ties and actions (e.g., a fall). We are developing practical and user acceptable solutions for that by exploiting the Internet of Things (IoT) and Artifcial Intelligence (AI). We aim at leveraging standard climbing equipment and novel sensing technologies for unobtrusive climbing activity recognition and performance assessment. We present here the results of some feld experiments that demon- strate the potential of our approach. We have used a standard piece of climbing equipment, namely a quickdraw, augmented with a 3-axial accelerometer sensor. This łSmart QuickDrawž is being de- veloped in collaboration with Vertical-Life Climbing 1 . The experi- ments were carried out at the Salewa Cube 2 and Vertikale 3 climbing gyms, and involved two of the authors of this paper. The specifc goal that we consider in this article is the detection of climbing episodes from a continuous stream of accelerometry data obtained from the quickdraw movements. While climbing episodes detection can be easily addressed by asking explicit climber feedback (e.g., by using a custom-designed app or by instrumenting the climbing wall with physical buttons), we are interested in developing solutions that are as unobtrusive as possible. Hence, we have developed a Machine Learning (ML) solution for detecting a particular type of climber’s activity, namely the rope pulling, which happens at the end of a climb. Our solution relies only on the analysis of the data generated by smart quickdraws. We have compared some ML tech- niques, and our fndings support the hypothesis that acceleration data collected by the sensors attached to quickdraws can be used 1 https://www.vertical-life.info/de/gym 2 http://www.salewa-cube.com/en/ 3 https://www.vertikale.it/ MAISTR'20 Workshop ICMI '20 Companion, October 25–29, 2020, Virtual Event, Netherlands 245