Neuropsychologia 48 (2010) 382–393 Contents lists available at ScienceDirect Neuropsychologia journal homepage: www.elsevier.com/locate/neuropsychologia Social cues, mentalizing and the neural processing of speech accompanied by gestures Benjamin Straube a,b,c,∗ , Antonia Green a,c , Andreas Jansen c , Anjan Chatterjee b , Tilo Kircher c a Department of Psychiatry and Psychotherapy, RWTH Aachen University, Aachen, Germany b Department of Neurology and the Center for Cognitive Neuroscience, The University of Pennsylvania, Philadelphia, PA 19104, USA c Department of Psychiatry und Psychotherapy, Philipps-University Marburg, Marburg, Germany article info Article history: Received 17 June 2009 Received in revised form 20 September 2009 Accepted 21 September 2009 Available online 25 September 2009 Keywords: Social relevance Body orientation Emblematic gestures Social cognition Speech-associated gestures abstract Body orientation and eye gaze influence how information is conveyed during face-to-face commu- nication. However, the neural pathways underpinning the comprehension of social cues in everyday interaction are not known. In this study we investigated the influence of addressing vs. non-addressing body orientation on the neural processing of speech accompanied by gestures. While in an fMRI scanner, participants viewed short video clips of an actor speaking sentences with object- (O; e.g., shape) or person-related content (P; e.g., saying goodbye) accompanied by iconic (e.g., circle) or emblematic gestures (e.g., waving), respectively. The actor’s body was oriented either toward the participant (frontal, F) or toward a third person (lateral, L) not visible. For frontal vs. lateral actor orientation (F > L), we observed activation of bilateral occipital, inferior frontal, medial frontal, right anterior temporal and left parietal brain regions. Additionally, we observed activity in the occipital and anterior temporal lobes due to an interaction effect between actor orientation and content of the communication (PF > PL) > (OF > OL). Our findings indicate that social cues influence the neural processing of speech–gesture utterances. Mentalizing (the process of inferring the mental state of another individual) could be responsible for these effects. In particular, socially relevant cues seem to activate regions of the anterior temporal lobes if abstract person-related content is communicated by speech and gesture. These new findings illustrate the complexity of interpersonal communication, as our data demonstrate that multisensory information pathways interact at both perceptual and semantic levels. © 2009 Elsevier Ltd. All rights reserved. 1. Introduction Human communication is composed of both linguistic informa- tion (words and sentences) and pragmatic information (non-verbal actions such as gestures, facial expressions, body orientation and eye gaze; e.g., Holler & Beattie, 2003). Of such pragmatic infor- mation, the direction in which a speaker’s body is oriented is particularly important during face-to-face interaction. For instance, in a group setting a speaker may position his or her body towards a particular person and compose his or her gestures expressly for that listener (see Özyürek, 2002). In this way, the speaker uses his or her body to guide communication. The impact of these non-verbal signals on the listener is poorly understood. In particular, little research has been done on ∗ Corresponding author at: Department of Psychiatry und Psychotherapy, Philipps-University Marburg, Marburg, Germany. Tel.: +49 6421 58 65076; fax: +49 3212 7586605. E-mail address: benjamin.straube@web.de (B. Straube). the neural substrates responsible for processing body orientation and gestures during verbal communication. Kilner, Marchant, and Frith (2006) used magnetic encephalography (MEG) techniques to investigate the effect of body orientation (forward vs. backward actor position) on the processing of arm and hand movements. They found that the observation of arm and hand movements interact with the social relevance of the body orientations (for- ward > backward presented actor) in parietal brain regions. This study showed that the brain processes underlying the observation of simple arm and hand movements are affected by social relevance (as operationalized by the speaker’s body’s orientation). However, the influence of social relevance at more complex levels of speech and gesture processing is unknown. Previous studies on the perception of social cues have focused on mentalizing (the process of inferring the mental state of another person) triggered, for example, by gaze direction (e.g., Kampe, Frith, & Frith, 2003). Mentalizing is thought to rely on a neural system comprised of the paracingulate cortex, the temporal poles, and the superior temporal sulcus at the temporo-parietal junction (Baron- Cohen et al., 1999; Brunet, Sarfati, Hardy-Bayle, & Decety, 2000; 0028-3932/$ – see front matter © 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.neuropsychologia.2009.09.025