Assistance for Autistic People by Segmenting and Highlighting Cross-Modal Perceptual Information Lars Schillingmann *1 , Matthias Rolf 1 , Shinichiro Kumagaya 2 , Satsuki Ayaya 2 , Yukie Nagai 1 1 Osaka University, 2 The University of Tokyo 1. Introduction People with Autistic Spectrum Disorder (ASD) have to deal with a wide range of perceptual problems. The problems in autism can be related to difficulties in temporal integration. This can affect perception of stimulus length, impaired temporal coherence [1], and problems in orienting attention as well as shift- ing attention from one stimulus to another [2]. The deficiencies can affect the processing of language [3], processing of visual information such as faces or visual signs of emotion [4], or acoustic signs of emotion [5]. However, multimodal input can still facilitate autistic persons’ communication. For example sign language and speech have been reported to mutually shed lights on relevant parts in the conversation [6]. Such additional multimodal cues can help to overcome integration and attention problems by (i) augmenting available information and (ii) highlighting parts of other modalities that are important. Likewise, aug- menting and highlighting interactions patterns have been discovered in parents’ child-directed communi- cation, which have gained recent interest also for the teaching of robots [7]. Yet, existing assistive systems act as a filter that irrevocably removes information from the perceptual stream. Such filters certainly prevent being over- whelmed by too much information, but do not allow for a mutual promotion of multimodal stimuli, and might also remove important information accidentally. We ar- gue that support systems for people with ASD should instead augment available information and highlight it, in order to guide their attention and to help extracting relevant information (see Figure 1). In this paper we approach the development of a support system that allows for such mutual promotion by capitalizing on previous effort on infant directed communication and robotic perceptual systems. We presented two robotic perceptual systems to people with ASD and analyzed their responses based on a questionnaire and a free discussion. Results agree with our hypothesis by iden- tifying the need for an immersive assistive device that integrates into the communicative situation instead of separating the user from it by removing information. 2. Method The general scope of this paper is to investigate the use of robotic perceptual systems for the integra- tion of auditory and visual stimuli as assistive systems. Fig.1 Conceptual diagram of a perceptual assisting system which highlights information instead of restricting the available information. Fig.2 Demonstrated systems: (top) Acoustic pack- aging; (bottom): Audio-visual synchrony.