Proceedings of ISon 2016, 5 th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016 INTERACTIVE SONIFICATION OF MOVEMENT QUALITIES – A CASE STUDY ON FLUIDITY Paolo Alborno, Andrea Cera, Stefano Piana, Maurizio Mancini, Radoslaw Niewiadomski, Corrado Canepa, Gualtiero Volpe, Antonio Camurri University of Genova, Casa Paganini – InfoMus, DIBRIS paoloalborno@dibris.unige.it, andreawax@yahoo.it, steto84@infomus.org, Maurizio.mancini@unige.it, radoslaw.niewiadomski@dibris.unige.it, corrado@infomus.org, gualtiero.volpe@unige.it, antonio.camurri@unige.it ABSTRACT The EU H2020 ICT Project DANCE investigates how affective and social qualities of human full-body movements can be expressed, represented, and analysed by sound and music performance. In this paper we focus on one of the candidate movement qualities: Fluidity. An algorithm to detect Fluidity in full-body movement, and a model of interactive sonification to convey Fluidity through the auditory channel are presented. We developed a set of different sonifications: some follows the proposed sonification model, and others are based on different, in some cases opposite, rules. Our hypothesis is that our proposed sonification model is the most effective in communicating Fluidity. To confirm the hypothesis, we developed a serious game and performed an experiment with 22 participants at MOCO 2016 conference. Results suggest that the sonifications following our proposed model are the most effective in conveying Fluidity. 1. INTRODUCTION The EU H2020 ICT Project DANCE investigates how affective and social qualities of human full-body movements can be expressed, represented, and analysed by sound and music performance. DANCE addresses research challenges such as: is it possible to perceive movement expressive qualities in dance through the auditory channel? Can we imagine concrete ways to “listen to a choreography”, “feel a ballet”? If we can capture the inner and intimate expressive qualities conveyed by movement to an external observer, these qualities might be made manifest through other sensory modalities such as, for example, the auditory one. In such a way, by closing her eyes and by listening to the auditory representation of movement qualities, a user can be made aware of some information, which is hidden in the movement and may be difficult to be perceived otherwise. Interactive sonification is receiving a growing relevance in the scientific and artistic communities: it is used in rehabilitation, sensory substitution, perception enhancement, and human- computer interfaces (Dubus & Bresin, 2013). The importance of sonification in communicating movement qualities is observable since the silent movies era, where sonifications were improvised by a pianist playing while observing what was happening on the screen (Hermann, Hunt, & Neuhoff, 2011). With the development of more sophisticated techniques the role of sonification in movies became more and more important: sonifications are now used in a broad range of scenarios (e.g., to convey off-screen events, to cover cuts and scene transitions, signal flashbacks, and direct the watcher’s attention) (Hermann, Hunt, & Neuhoff, 2011). This paper focuses on designing and experimenting different interactive sonifications of a specific quality of movement: Fluidity (i.e., how to perceive movement Fluidity by the auditory channel). We start from our recent proposal of a computational model (and the corresponding EyesWeb software module) of Fluidity. In this work we adopt a conceptual framework recently proposed by Camurri and colleagues for the analysis of expressive movement qualities (Camurri, et al., 2016). 2. AUTOMATED ANALYSIS OF MOVEMENT QUALITIES: FLUIDITY Fluidity is often considered as a synonym of “good” movement (e.g., in certain dance styles), and is different from “smoothness”, which is referred to the movement of a single joint. Furthermore, Fluidity is one of the properties that seem to contribute significantly to perception of emotions (Camurri, Mazzarino, Ricchetti, Timmers, & Volpe, 2004). Caridakis and colleagues (Caridakis, et al., 2007) investigated fluidity of hands trajectories, and computed it as the sum of the variance of the norms of the hands’ motion vectors. Piana et al. (Piana, Stagliano', Camurri, & Odone, 2015) studied human motion trajectories and defined a Fluidity index based on the minimum jerk law. Starting from literature on biomechanics and psychology, and by conducting interviews and movement recordings with experts in human movement such as choreographers and dancers, we propose the following definition of Fluid movement [6] (performed by a part of the body, by the whole body, or by a group of dancers behaving as a single organism): - the movement of each joint is smooth, following the standard definitions in the literature of biomechanics (Viviani & Flash, 1995) (Morasso, 1981); - the energy is free to propagate along the kinematic chains (e.g., from head to trunk, from shoulders to arms; in a group from a dancer to another) according to a coordinated wave-like propagation. That is, there is an efficient propagation of movement along the kinematic chains, corresponding to a minimization of the dissipation of energy. 2.1. A Simplified Computational Model of Fluidity In this work, we measure movement Fluidity from IMUs (Inertial Measurement Units) data. Data can be extracted from sensors located on two different body joints. We consider the two IMUs ! ! and ! ! , placed, respectively, on the users’ right and left wrists 1 . 1 To extract the movement data we used the X-OSC sensors (X-IO Technology) that provide 9-axis inertial measurements of, respectively, the participant’s right and left hand. 28