Intonation and hand gestures in narrations of healthy speakers and speakers suffering from right hemisphere damage: a pilot study Katharina Hogrefe 1 , Wolfram Ziegler 1 , Carina Tillmann 1 , Georg Goldenberg 2 1 Neuropsychological research group (EKN), Hospital Bogenhausen, Municipal Clinic München GmbH, Germany 2 Clinic for Neuropsychology, Hospital Bogenhausen, Municipal Clinic München GmbH, Germany Katharina.Hogrefe@extern.lrz-muenchen.de Abstract In this pilot study we investigated the relationship between intonational variability and gesture frequency in narrations of healthy speakers and speakers suffering from right hemisphere brain damage. We developed a new method for measuring intonational variability. In both groups, we found no relationship between intonational variability and gesture frequency. There were, however, significant differences between groups with respect to intonational variability and to gesture frequency. Index Terms: gesture, prosody, intonation, right hemisphere brain damage, variability 1. Introduction Prosody as well as gesture convey information beyond lexical and syntactic verbal systems; both have been shown to contribute to discourse organization (e.g. [1] for gesture, [2] for prosody). Both can be used for marking new or important information. Wennerstrom [3] claims that “intensified prosody can bring particular story events in the foreground. Certain key words, for example, may be uttered with a higher-than- usual pitch” (p. 200). Levy & McNeill [4] reported that information that was important for the course of a story was more often accompanied by a gesture. It might therefore be assumed that a speaker who has the communicative intention to highlight certain information uses prosodic and gestural means for making this information more salient. Therefore, we examined if speakers combine prosodic markers and gesture in narrations. One of these prosodic markers is intonation. A positive relationship between intonational variability and gesture use could indicate that speakers tend to combine both conversational means to realize their communicative intentions. This examination could also reveal different speaker types. We might be able to identify “vivid” storytellers who produce many gestures and speak with a high intonational variability. On the other hand, a pattern of monotonous speakers might be identifiable with low gesture rates and a flat intonation. In this regard, speakers with right hemisphere damage offer an interesting communication profile. These speakers often display deficits in pragmatic aspects of language. Their communication disorder can affect narrative-discourse processing, the processing of metaphors or idioms as well as processing and production of prosody. Disturbances of expressive prosody may consist in excessive or highly variable use of speech intonation or - which has been described more often for this patient group - in flat, monotonous intonation [5]. An overall reduction of communicative gesture production has been reported (e.g. [6], [7]). In an earlier description of two cases, the reduction of gesture and intonational variability has led to the assumption of an association of the two modes of expression [8]. However, a more recent study investigated the relationship of gesture and prosody in different discourse ranges [9]. The authors analyzed data from five speakers with right hemisphere damage and from healthy controls. They determined intonational phrases which were used as basic units for the identification of gestural measures. The number of total gestures was calculated per intonational phrase. Furthermore, a variation coefficient was determined for F 0 variation. There was no clear pattern of gesture use and intonation in the patient group: Three of the patients displayed an overall reduction in gesture frequency, whereas only two patients displayed reduced intonational variation, and in only one patient were both modes of expression reduced in parallel. This finding is surprising and the authors wondered if their variation of fundamental frequency was an insufficiently sensitive measure to capture prosodic variability. The aim of our pilot study was to investigate the relationship between intonation and gesture in healthy speakers and in speakers with right hemisphere damage. Further, we compared the patients with the healthy speakers with respect to gesture frequency and intonational variability. We applied three intonational measures that might be more sensitive to capture variability in speech. One was meant to identify variability on a text level, i.e., the variation of mean fundamental frequency between phrases, the second measure captured intonational variation on the phrase level, and the third measure was developed to identify changes within shorter time windows of the speech signal. 2. Research Objectives We analyzed healthy speakers’ behaviour in narrations and investigated the following questions: 1. Does gesture frequency correlate with intonational variability? We expected to observe a covariation of F 0 variability and gesture frequency across speakers. 2. Do phrases containing gestures display a higher intonational variability than phrases without gestures? To the extent that intonation and gesture are used to emphasize information, a relationship between the occurrence of gestures and high intonational variation within phrases can be expected. Furthermore, we examined the behaviour of speakers with lesions to the right hemisphere focusing on the following questions: 3. Is there a relationship between gesture frequency and intonational variability within the group? And do we find aprosodic and agestural speakers in the patient group? 4. Do we find between-group differences of gesture frequency and prosodic variability? Patients with right hemisphere damage have been described as producing fewer gestures than healthy speakers. With respect to prosody, the most frequent