Interactive Musical Setting with Deep Learning and Object Recognition M´ ario Cardoso 1 a and Rui Pedro Lopes 2 b 1 Research Center for Basic Education, Instituto Polit´ ecnico de Braganc¸a, Portugal 2 Research Center for Digitalization and Industrial Robotics, Instituto Polit´ ecnico de Braganc¸a, Portugal Keywords: Deep Learning, Object Recognition, Musical Setting, Musical Textures, Musical Education. Abstract: The SeMI - Interactive Musical Setting, explores the possibilities of joining machine learning, the physical and the sound world. In this context, a machine learning algorithm and model was used to identify physical objects through image processing. Each physical object is associated with a student’s produced musical texture that starts playing when the object is recognized by the device. This allows defining use cases in which students have to develop diverse although interrelated sound textures and combine them with a physical world, in both a fake orchestra, that reacts to people and objects in front of it, and mood rooms, for example. The application was developed for iPad and iPhone, using Swift programming language and the iOS operating system and used in the classes of the masters on Teaching of Musical Education in the Basic School. 1 INTRODUCTION Over the last twentieth century, the nature of music itself has changed dramatically (Hargreaves and La- mont, 2017). Several musicologists document many revolutions and transformations: (i) the fresh and innovative music of Beethoven, Wagner, Debussy, Stravinsky, Schoenberg and Messiaen; (ii) the de- velopments of electronic sound production; (iii) the transformations and discussions of iconoclastic com- posers (Stockhausen, Boulez, Cage, Glass, Reich) and the impact in music performance and composi- tion; (iv) the influence and power of the different gen- res and styles; and (v) the digital revolution (to pro- duce, record and transmit) and the new sounds and effects. This last aspect had deep effects in the peo- ple’s lives and in particular in the way musicians (and non-musicians) perform and compose music (Car- doso et al., 2019; Ruthmann and Mantie, 2017). Nowadays, being a music teacher involves more skills: working knowledge of music software and hardware; arranging or improvising; arts technology (interactive art, computer programming, virtual and augmented reality). For Brown (2015), it is necessary to reinterpret the nature of musical experience. The digitisation of music means that we have to change and reinvent the paradigm of teaching and learning a https://orcid.org/0000-0003-3645-9641 b https://orcid.org/0000-0002-9170-5078 music. For Peppler (2017), this positive vision radi- cally shifts the lines between performer, listener and composer. In music education, technology can be a catalyst that contribute to expand the process of teach- ing and learning music into a more comprehensive, creative, innovative and imaginative experience (Car- doso et al., 2019; Brophy, 2001). In this framework, musical learning can benefit from the development and adoption of innovative devices and tools that fos- ter their autonomous work and help them overcome the difficult task of learning to play and compose mu- sic. It is important that the process puts great em- phasis on the student, encouraging higher education institutions and academic staff to place students at the center of their thinking and to help them man- age their expectations and be able to consciously and constructively design their learning paths through- out their higher education experience (Lopes et al., 2019; Tenorio et al., 2018). If the education is one of the most important aspects of human development, greatly influencing the path of professional develop- ment and success (Mesquita et al., 2015, 2014), it is important to increase the training at the higher- education level, that contributes to the scientific and technological qualification of youth and adults, to- wards the “creative, innovative and competitive de- velopment, with high productivity standards” (Cor- reia and Mesquita, 2006, 166). This paper describes the development of a ma- chine learning based application, that runs in an iOS Cardoso, M. and Lopes, R. Interactive Musical Setting with Deep Learning and Object Recognition. DOI: 10.5220/0009856406630667 In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020), pages 663-667 ISBN: 978-989-758-417-6 Copyright c  2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved 663