Temporal Interaction of Emotional Prosody and Emotional Semantics: Evidence from ERPs Silke Paulmann & Sonja A. Kotz Max Planck Institute for Human Cognitive and Brain Sciences Leipzig, Germany paulmann@cbs.mpg.de Abstract Emotional prosody carries information about the inner state of a speaker and therefore helps us to understand how other people feel. However, emotions are also transferred verbally. In or- der to further substantiate the underlying mechanisms of emo- tional prosodic processing we investigated the interaction of both emotional prosody and emotional semantics with event- related brain potentials (ERPs) utilizing a prosodic and inter- active (prosodic/semantic) violation paradigm. Results suggest that the time-course of emotional prosodic processing and emo- tional semantics differ. While a pure violation of a prosodic contour elicited a positivity between 450 ms and 600 ms, a vio- lation of both emotional prosody and semantics elicited a neg- ativity between 500 ms and 650 ms. These results suggest that emotional prosody and emotional semantics follow a different time-course. This holds true for all emotional prosodies (anger, disgust, fear, happy, pleasant surprise, sad) investigated. As the two conditions elicited two different electrophysiological com- ponents, the obtained results suggest that emotional prosody and semantics contribute differentially during the interaction of both information types. Furthermore, the data suggest that se- mantic information can override prosody when the two channels interact in time, that is, when the emotional prosodic contour agrees with the semantic content of a sentence. 1. Introduction Disorders of emotional prosody -be it at the level of perception or production- influence social interactions. However, the un- derlying mechanisms of emotional prosody perception are not well understood. The present work investigates the perception of emotional prosody, with a particular emphasis on the po- tential relatedness between emotional prosody and emotional- semantics. The main question pursued is in how far the tempo- ral integration of emotional semantics and emotional prosody can be specified, i.e. at which time do these two channels max- imally interact? So far, little is known on how and when emo- tional prosody and emotional-semantics interact at the sentence level. However, there is previous evidence [1, 2, 3] which sug- gests that the time course of emotional prosody and semantics differ. The aim of the current ERP-experiment was to further investigate the interaction of emotional prosody and emotional- semantics by isolating the respective contribution of each emo- tional channel to this interaction by using a cross-splicing pro- cedure. With the help of the cross-splicing method we violated an expectation specific to emotional prosody. Here we used pseudosentences as well as lexical sentences. Comparable to fil- tered speech, morphologically marked pseudosentences spoken with the varying emotional intonation patterns allow to elimi- nate lexical content while preserving emotional prosody. We know that linguistic information is transferred not only through speech sounds but also via properties of the speaker, i.e. speaker identity (e.g. female/male; young/old) as well as the speaker’s emotional state (e.g. happy/sad). Information about the speaker is typically thought to be encoded by pitch, intensity, and duration (or tempo) of the utterance. To explore the possible influence of speaker identity on the interaction of emotional prosody and emotional semantics, stimuli recorded from two speakers (male/female) were tested. In particular, we investigated voice identity to shed more light on the issue of gender voice specific emotional prosody processing. For in- stance, within the literature on emotional expression sex differ- ences have been observed, i.e. female and male differ in their emotional expressiveness. Interestingly, these differences were noticed to arise already during development [4]. It is thus also important to further specify if there are also gender voice spe- cific differences in emotional prosody perception [5]. 2. Methods 2.1. Participants Thirty native speakers of German (fifteen female) participated in the experiment. Female participants had a mean age of 24.13 (SD 1.96) and male participants had a mean age of 24.67 (SD 2.02). Participants received financial compensation for their participation. 2.2. Stimulus Material The material consisted of semantically and prosodically match- ing stimuli (lexical/pseudo) for each of the six basic emotions (anger, fear, disgust, happiness, pleasant surprise, sadness) and a neutral baseline. For each emotion and sentence type, 30 sentences were presented, adding up to 210 matching lexical sentences and 210 matching pseudosentences. In addition, the same sentences were cross-spliced in two ways: a) in the com- bined semantic/prosodic violaten condition, a semantically and prosodically neutral start of a sentence was cross-spliced to an emotional semantically and prosodically matching end of the sentence, and b) in the pure emotional prosodic violation condi- tion, a prosodically neutral start of a pseudosentence was cross- spliced to an emotional-prosodically end of the pseudosentence, resulting in 180 cross-spliced lexical sentences and 180 cross- spliced pseudosentences. As all sentences were spoken by a female and male speaker, a total of 1560 trials were presented in two sessions. Emotional prosodic valence was obtained in two earlier rating studies (one for the lexical sentences and one for the pseudosentences). All sentences were taped with Speech Prosody 2006 Dresden, Germany May 2-5, 2006 ISCA Archive http://www.isca-speech.org/archive Speech Prosody 2006, Dresden, Germany, May2-5, 2006