Proceedings of RoCHI 2021 - 134 - An Augmented Reality Piano Learning Tool Andra Simion, Adrian Iftene, Daniela Gîfu Faculty of Computer Science, “Alexandru Ioan Cuza” University of Iași, Romania {andra.simion, adiftene, daniela.gifu}@info.uaic.ro ABSTRACT Music learning (here, thinking of playing piano) without professor can be a real challenge. This can be difficult to achieve, given some traditional approaches to manage this process are still insufficiently satisfactory. Beyond the well- known computer-assisted music learning tools (1) screen- based and (2) hardware-based, a promising solution for piano learning is based on augmented reality (AR). This technology provides an ideal environment for the student, but of course, it is not enough to become a career musician. In this paper, we propose an application that helps all those who want to learn to play the piano in a friendly way, starting with understanding sheet music. By projecting information onto the instrument, our AR system can create more direct interactions for the user. Author Keywords ELearning, Augmented reality, Playing piano. ACM Classification Keywords H.5.2. Information interfaces and presentation (e.g., HCI): User Interfaces. H.3.2. Information Storage and Retrieval: Information Storage. General Terms Human Factors; Design. DOI: 10.37789/rochi.2021.1.1.21 INTRODUCTION Music learning represents a field that acquired importance to people for their health and well-being [1] due to the recent validation that attending music lessons can produce significant increases in psychological and cognitive areas [2]. However, learning a new instrument can be challenging as you cannot progress unless you know how to read sheet music [3]. Moreover, these days due the lack of time and financial constraints, people can’t afford too many lessons assisted by a professor, developing wrong habits in practice [4]. There are some well-known computer-assisted music learning approaches [5], namely: (1) screen-based (e.g., Piano [6] or Yousician [7]), that require users to visually transfer information from the device to apply on the specific instrument or (2) hardware-based (e.g., The ONE Smart Piano [8]), that provides direct guidance to the detriment of the user’s engagement – the problem to facilitate this process remains. Beyond these methods, a promising solution for computer-assisted piano learning is based on augmented reality (AR). AR is an interactive experience that can create a connection between the student and the system by defining more practical possibilities with the instrument. Azuma gathered existing information from several sources to conceive a survey of AR [9], considered a starting point for any AR system. The use of AR in education has become something natural lately, with applications to learn geography [10], chemistry [11], computer science [12], or how to work in a team [13]. One of the most well-known computer-assisted piano systems is called Synthesia where you can add MIDI files for certain songs you want to listen to or even practice playing them. The software processes the files and formats the notes visually so that it always displays the keys you should press along with its duration. Unfortunately, using Synthesia has its own disadvantages, namely, you need to have a computer on which the program runs, and this computer needs to be placed somewhere behind the keyboard so that you can follow the notes. The legitimate research question of this paper intends to answer: How efficient is AR-based way of learning the piano? This research intends to draw the basic knowledge level over which it intends to highlight the role of the AR- assisted music education. We aim to develop a rapid, friendly, and efficient system, assisted by AR, to help those who dream of playing the piano. The rest of paper is organized as follows: Section 2 presents a short overview of piano learning systems very useful in the musical practice of beginners, while Section 3 describes the prototype based on AR. Section 4 briefly discusses the evaluation of this new system before drawing some conclusions in the last section. OTHER PIANO LEARNING SYSTEMS Thanks to the popularization of new learning phenomenon recent findings indicate that AR-assisted music education seems to be a viable approach that can alleviate the shortage of qualified music teachers. Studies on this type of education have also started to gain momentum. In 2001 many studies were focused on personal contextual awareness [14]. We were invoking Cheng that develops two working systems with the focus on exploiting the user’s own body parts as triggers for the augmentation. HANDEL does not have a simplified display version for those who don’t know how to read sheet music, instead it makes use of the music notation itself augmented on the keyboard. Here, a see-through head mounted display is required so that the user’s hands are free to interact with the keyboard. The practice session begins at first, with the loaded music score into the program where it is handled by a custom music score language and secondly with the nod of the pianist when he’s ready to practice. No notes are rendered unless the user focuses on the hands