Audiovisual integration of speech is disturbed in schizophrenia: An fMRI study G.R. Szycik a,b, ⁎, T.F. Münte b,c,d , W. Dillo a , B. Mohammadi b,c , A. Samii c , H.M. Emrich a , D.E. Dietrich a a Clinic for Psychiatry, Social Psychiatry and Psychotherapy, Medical School Hannover, Germany b Department of Neuropsychology, University of Magdeburg, Germany c International Neuroscience Institute, Hannover, Germany d Center for Behavioral Brain Science, Magdeburg, Germany article info abstract Article history: Received 22 June 2008 Received in revised form 22 February 2009 Accepted 3 March 2009 Available online 19 March 2009 Speech perception is an essential part of social interaction. Visual information (lip movements, facial expression) may supplement auditory information in particular under inadvertent listening situations. Schizophrenia patients have been shown to have a deficit in integrating articulatory motions with the auditory speech input. The goal of this study was to investigate the neural basis of this deficit in audiovisual speech processing in schizophrenia patients by using fMRI. Disyllabic nouns were presented in congruent (audio matches visual information) and incongruent conditions in a slow event related fMRI design. Schizophrenia patients (n =15) were compared to age and gender matched control participants. The statistical examination was conducted by analysis of variance with main factors: audiovisual congruency and group membership. The patients' brain activity differed from the control group as evidenced by congruency by group interaction effects. The pertinent brain sites were located predominantly in the right hemisphere and comprised the pars opercularis, middle frontal sulcus, and superior temporal gyrus. In addition, we observed interactions bilaterally in the fusiform gyrus and the nucleus accumbens. We suggest that schizophrenia patients' deficits in audiovisual integration during speech perception are due to a dysfunction of the speech motor system in the right hemisphere. Furthermore the results can be also seen as a reflection of reduced lateralization of language functions to the left hemisphere in schizophrenia. © 2009 Elsevier B.V. All rights reserved. Keywords: Speech Audiovisual integration fMRI Multimodal processing Schizophrenia Right hemisphere 1. Introduction To integrate information from different modalities into a coherent representation of the environment is an essential task which is not easy given the different processing times in the different modalities and the widely differing anatomical pathways and networks subserving the processing of unim- odal information. A special case of a multisensory integration process is the perception of speech. Unlike our introspective intuitions as normal hearing listeners we are constantly evaluating the characteristic lip movements associated with each phoneme. Indeed, seeing a speaker's articulatory move- ments facilitates the recognition of spoken words in noisy environments substantially (Erber, 1969; Grant and Seitz, 2000; Munhall et al., 2004; O'Neill, 1954; Sumby and Pollack, 1954). This gain in intelligibility from viewing visual articulations is maximal at intermediate signal-to-noise ratios (Ross et al., 2007). Furthermore, the presentation of audio- visual (AV) incongruent (auditory stream does not match the articulatory movements) speech may lead to new hybrid percepts occurring as a result of the fused information from both channels. The most prominent example is the McGurk effect (Dekle et al., 1992; McGurk and MacDonald, 1976). Thus, the comprehension of language results from the integration of auditory and visual modality information that is used and combined in a flexible, context dependent manner. Recently, a number of brain regions including the superior temporal sulcus (Calvert et al., 2000; Sekiyama et al., 2003; Szycik et al., 2008a; Wright et al., 2003), Broca's area Schizophrenia Research 110 (2009) 111–118 ⁎ Corresponding author. Clinic for Psychiatry, Social Psychiatry and Psychotherapy Medical School Hannover, Carl-Neuberg-Straße 1, 30625 Hannover, Germany. Tel.: +49 511 5325246; fax: +49 511 5323187. E-mail address: szycik.gregor@mh-hannover.de (G.R. Szycik). 0920-9964/$ – see front matter © 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.schres.2009.03.003 Contents lists available at ScienceDirect Schizophrenia Research journal homepage: www.elsevier.com/locate/schres