www.elsevier.com/locate/brainres Available online at www.sciencedirect.com Research Report Semantic relation vs. surprise: The differential effects of related and unrelated co-verbal gestures on neural encoding and subsequent recognition Benjamin Straube n , Lea Meyer, Antonia Green, Tilo Kircher Department of Psychiatry und Psychotherapy, Philipps-University Marburg, Marburg, Germany article info Article history: Accepted 9 April 2014 Available online 16 April 2014 Keywords: Iconic gestures Multimodal processing Congruency Speech associated gestures Memory Attention abstract Speech-associated gesturing leads to memory advantages for spoken sentences. However, unexpected or surprising events are also likely to be remembered. With this study we test the hypothesis that different neural mechanisms (semantic elaboration and surprise) lead to memory advantages for iconic and unrelated gestures. During fMRI-data acquisition participants were presented with video clips of an actor verbalising concrete sentences accompanied by iconic gestures (IG; e.g., circular gesture; sentence: “The man is sitting at the round table”), unrelated free gestures (FG; e.g., unrelated up down movements; same sentence) and no gestures (NG; same sentence). After scanning, recognition performance for the three conditions was tested. Videos were evaluated regarding semantic relation and surprise by a different group of participants. The semantic relationship between speech and gesture was rated higher for IG (IG4FG), whereas surprise was rated higher for FG (FG4IG). Activation of the hippocampus correlated with subsequent memory performance of both gesture conditions (IGþFG4NG). For the IG condition we found activation in the left temporal pole and middle cingulate cortex (MCC; IG4FG). In contrast, for the FG condition posterior thalamic structures (FG4IG) as well as anterior and posterior cingulate cortices were activated (FG4NG). Our behavioral and fMRI-data suggest different mechanisms for processing related and unrelated co-verbal gestures, both of them leading to enhanced memory performance. Whereas activation in MCC and left temporal pole for iconic co- verbal gestures may reflect semantic memory processes, memory enhancement for unrelated gestures relies on the surprise response, mediated by anterior/posterior cingu- late cortex and thalamico-hippocampal structures. & 2014 Elsevier B.V. All rights reserved. 1. Introduction Human speech is typically accompanied by gestures. These gestures confer advantages in comprehension, learning and memory (e.g., Cook et al., 2010, 2013; Kelly and Church, 1998; Kelly et al., 1999; Straube et al., 2009, 2011a; Valenzeno et al., 2003). For example Kelly et al. (1999) demonstrated that certain nonverbal behaviors, such as deictic and iconic gestures, help people comprehend and remember pragmatic communication. Consistent with this finding Valenzeno et al. (2003) showed that http://dx.doi.org/10.1016/j.brainres.2014.04.012 0006-8993/& 2014 Elsevier B.V. All rights reserved. n Corresponding author. Fax: þ49 3212 7586605. E-mail addresses: straubeb@med.uni-marburg.de, benjamin.straube@web.de (B. Straube). brain research 1567 (2014) 42–56