Integration of heard and seen speech: a factor in learning disabilities in children Erin A. Hayes a, * , Kaisa Tiippana b , Trent G. Nicol a , Mikko Sams b , Nina Kraus a,c,d a Department of Communication Sciences, Northwestern University, Evanston, IL, USA b Laboratory of Computational Engineering, Helsinki University of Technology, P.O. Box 9203, Helsinki, 02015 HUT, Finland c Departments of Neurobiology and Physiology, Northwestern University, Evanston, IL, USA d Department of Otolaryngology, Northwestern University, Evanston, IL, USA Received 2 June 2003; received in revised form 21 July 2003; accepted 25 July 2003 Abstract Normal-learning children (NL) and children with learning disabilities (LD) reported their perceptions of unisensory (auditory or visual), concordant audiovisual (e.g. visual /apa/ and auditory /apa/) and conflicting (e.g. visual /aka/ and auditory /apa/) speech stimuli in quiet and noise (0 dB and 2 12 dB signal-to-noise ratio, SNR). In normal populations, watching such conflicting combinations typically changes auditory percepts (‘McGurk effect’). NL and LD children identified unisensory auditory and congruent audiovisual stimuli similarly in all conditions. Despite being less accurate identifying unisensory visual stimuli, LD children were more likely than NL children to report hearing only the visual component of incongruent audiovisual stimuli at 2 12 dB SNR. Furthermore, LD children with brainstem timing deficits demonstrated a distinctive pattern of audiovisual perception. The results suggest that the perception of simultaneous auditory and visual speech differs between NL and LD children, perhaps reflecting variations in neural processing underlying multisensory integration. q 2003 Elsevier Ireland ltd. All rights reserved. Keywords: Audiovisual integration; McGurk effect; Learning disabilities; Dyslexia; Speech perception; Auditory brainstem response; Visual perception; Auditory perception Learning disabilities that are manifested as reading and spelling deficits have been attributed to deficits in the perception and neurophysiologic encoding of both auditory and visual stimuli [8,9]. More specifically, these studies have focused on auditory speech perception impairments [4], deficits in the magnocellular pathway of visual processing [3] and difficulties processing rapidly presented stimuli in any sensory modality [14]. Recent imaging studies have examined neural activity in multisensory cortical regions in normal and learning disabled readers and have discovered altered patterns of connectivity among primary sensory and multisensory processing areas in learning disabled subjects [12,13]. In these individuals, reading deficits may be attributable to deficits integrating multisensory information. One perceptual illusion that reflects multisensory inte- gration is the McGurk effect, which occurs when people see and hear a talker producing incongruent speech segments [11]. For example, a classic McGurk effect occurs when an auditory /pa/ and a visual /ka/ are presented together. People often hear /ta/, a fusion of the auditory and visual stimuli. When auditory /ka/ is dubbed onto visual /pa/, a typical auditory percept is /pka/ or /kpa/, combinations of the auditory and visual stimuli. Other possible percepts are /pa/, the visual component, or /ka/, the auditory component. Occasionally they may hear a phoneme that fails to incorporate the acoustic or visual aspects of either stimulus; these are categorized as other responses. In this study, auditory, visual and audiovisual speech perception were examined in normal learning and learning disabled children. Subjects were between the ages of 8 and 14, had normal binaural hearing thresholds (# 15 dB HL for octaves 500–8000 Hz), had normal binocular near vision (better than 20/40), were native speakers of American English and had normal mental ability (IQ $ 85 on the Brief Cognitive Scale, Woodcock-Johnson (WJ) [18]). Standar- dized measures of cognition and academic achievement were administered (see Table 1). Cross Out is a measure of visual processing, including visual scanning and attention [19]. Auditory Processing is a composite score derived from Incomplete Words and Sound Blending [19]. Listening Comprehension [19] and Memory for Words [19] measure Neuroscience Letters 351 (2003) 46–50 www.elsevier.com/locate/neulet 0304-3940/03/$ - see front matter q 2003 Elsevier Ireland ltd. All rights reserved. doi:10.1016/S0304-3940(03)00971-6 * Corresponding author. Northwestern University, 2240 N. Campus Drive, Evanston, IL 60208, USA. Tel.: þ 1-847-491-2465; fax: þ 1-847- 491-2523. E-mail address: e-hayes@northwestern.edu (E.A. Hayes).