The Effects of Pitch and Dynamics on the Emotional Characteristics of Piano Sounds Chuck-jee Chau Andrew Horner Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong chuckjee@cse.ust.hk, horner@cs.ust.hk ABSTRACT The piano is an instrument extensively used in classical, jazz, and pop music, since its broad pitch range and ample dy- namic levels allow the instrument to become self-contained and versatile for various kinds of music. Previous work has found the piano to be emotionally neutral among eight tested non-sustaining instruments. This paper further explores the emotional characteristics of piano sounds with a listening test comparing isolated one-second sounds of different pitches and dynamics on the piano over ten emotional categories: Happy, Sad, Heroic, Scary, Comic, Shy, Romantic, Mysteri- ous, Angry, and Calm. In the experiment, the loud bass was ranked the most Angry and Heroic, and the effect dropped with increasing pitch. The soft treble was ranked the most Calm and Shy, and the effect dropped with decreasing pitch. The trend was clear across the octaves. Both loud and soft sounds have distinguishing emotional characteristics. In con- trast, the emotional categories Mysterious and Happy were not much affected by dynamics. 1. INTRODUCTION Previous research has shown that both sustained [1, 2, 3] and non-sustained [4, 5] instrument sounds have strong emotional characteristics. For example, it has found that the trumpet, clarinet, and violin are relatively joyful compared to other sustained instruments, even in isolated sounds apart from mu- sical context, while the horn is relatively sad. The marimba and xylophone are relatively happier compared to other sus- tained instruments, while the harp and guitar are relatively depressed. In our recent study of non-sustained instrument sounds [4, 5], several musical instruments were tested: plucked violin, guitar, harp, marimba, xylophone, vibraphone, piano, and harpsichord. Among the eight instruments, the piano was found to be ranked neutral for eight emotional categories: Happy, Sad, Heroic, Scary, Comic, Shy, Joyful, and De- pressed. It was often ranked in the middle, indicating neutral emotional characteristics relative to the other tested instru- ments. Perhaps one might find this surprising since the piano has the widest repertoire among all instruments in classical, Copyright: c 2015 Chuck-jee Chau et al. This is an open-access article distributed under the terms of the Cre ative Com mons At tri bu tion Li cense 3.0 Un ported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. jazz, and pop music, and there are also abundant transcrip- tions written for the piano. What is the range of emotional characteristics for the piano? Previous work has only studied single mid-range pitches of the instrument, and the loudness was also equalized to allow consistent comparison. We were curious about the effects of pitch and dynamics in isolated sounds on the emotional char- acteristics of the piano. We then formulated the current study to carefully compare the emotional characteristics of piano pitches at different dy- namic levels. The study includes representative pitches rang- ing from the lowest to the highest octave. The dynamic levels included loud, medium, and soft (forte, mezzo, and piano). All sounds were isolated with a duration of one second. They were compared pairwise over ten emotional categories. This work provides a systematic overview of the emotional characteristics of the piano across the different octaves at dif- ferent dynamics. This research will help recording and au- dio engineers, composers, and pianists manipulate the emo- tional characteristics of the instrument in live performances and recordings. 2. BACKGROUND Much work has been done on emotion recognition in music, especially addressing melody [6], harmony [7], rhythm [8], etc. Researchers have gradually established connections be- tween music emotion and timbre. Scherer and Oshinsky [9] found that timbre is a salient factor in the rating of synthetic sounds. Peretz et al. [10] showed that timbre speeds up dis- crimination of emotion categories. It was also found that tim- bre is essential to musical genre recognition and discrimina- tion [11, 12]. Eerola et al. [1, 13] showed a direct connection between music emotion and timbre. Eerola carried out listening tests to investigate the correlation of emotion with temporal and spectral sound features. The study confirmed strong correla- tions between features such as attack time and brightness and the emotion dimensions valence and arousal for one-second isolated instrument sounds. We followed up Eerola’s work with our own studies of mu- sic emotion and timbre [2, 3, 4, 14, 15] to find out if some sounds were consistently perceived as being happier or sadder in pairwise comparisons. We designed listening tests to com- pare sounds from various string, wind, and percussion instru- ments. The results showed strong emotional characteristics for each instrument. The horn and flute were highly ranked ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas – 372 –