Brushing up or aside? – Analyzing the modifying functions of gesture in different multimodal utterance contexts Farina Freigang and Stefan Kopp farina.freigang@uni-bielefeld.de, skopp@techfak.uni-bielefeld.de Faculty of Technology, Center of Excellence “Cognitive Interaction Technology” (CITEC) Bielefeld University, P.O. Box 100131, D-33501 Bielefeld, Germany Gestures can contribute to the meaning of an utterance not only by adding symbolic information, but also by modifying verbally or gesturally signified content. In most analyses, the propositional content is seen as the part that was intended to be communicated. From our perspective, how- ever, utterance givers also communicate their viewpoints, convictions, knowledge, attitudes, among others. One way to achieve this is to use one modality to modify - in an analogue way - the propo- sitional content of the other modalities. We define a modifying function to act upon and to carry meaning beyond mere propositional content. These functions are not as clearly signified, but are nevertheless communicatively efficient and significant for how recipients interpret the multimodal utterance as a whole. In some cases, the functions and meanings of different modalities can even appear incongruent, with the simplest but most pronounced case being a positive versus a negative element within an utterance. One interesting example are ‘brushing gestures’, which may appear in various utterance con- stellations. In Freigang and Kopp (2015) we present a cluster analysis of a rating study, in which participants judge the functions of gestures in different multimodal utterance contexts (drawn from our Bielefeld N atural I nteraction C orpus). First of all, we find that brushing gestures seems to change the utterance in a discounting or downtoning way (and sometimes are emotionally coloured), or as stated similarly by Payrat´ o and Teßendorf (2014, p. 1536) (see also M¨ uller and Speckmann (2002)), they express a ‘negative stance towards the object in question’. Secondly, depending on the tonal and facial expressions accompanying the brushing gesture, the combined meaning of an utterance may differ. If a normal video snippet (baseline condition) of a participant with a pos- itive tone of voice and positive facial expressions is given, the whole utterance is interpreted as emphasising and humorous. If the same video is played to raters, without showing the head of the participants in the video and muting the tone (gesture-only condition), the utterance is marked as rather discounting or downtoning, affirmative, critically and not humorous. If a participant shows a neutral facial expression, the utterance is interpreted almost the same in both conditions. Also, in the gesture-only condition, brushing gestures are rated more accurately, which may hint to the fact that in the baseline condition they are overwritten by other modalities. In our presentation, we will examine the functions of the brushing gesture and the contexts in which they appear in more detail by analysing the exact form features and inferring the different meanings of those body movements in interaction with the rest of the multimodal utterance. We are particularly interested in the contradictory elements of the negativity of a brushing gesture and the positivity that may be indicated through a positive tone of the voice or facial expressions and, further, how the negativity of a brushing gesture is related to the baseline meaning of the utterance. The results presented are based on five brushing gestures (in total the study consisted of 36 videos with various body movements) and neighbouring categories such as point-brushings and throwing gestures. A broader corpus analysis is needed in order to investigate this phenomenon further.