RTO-MP-IST-043 19 - 1 Auditory and Visual Facilitation: Cross-Modal Fusion of Information in Multi-Modal Displays Captain Stephen Boyne and Nada Pavlovic Defence Research and Development Canada DRDC Toronto 1133 Sheppard Ave. W. Toronto, Ontario M3M 3B9 CANADA stephen.boyne@drdc-rddc.gc.ca nada.pavlovic@drdc-rddc.gc.ca Ryan Kilgore and Mark H. Chignell Department of Mechanical and Industrial Engineering, University of Toronto MIE, 5 King’s College Rd. Toronto, Ontario M5S 3G8 CANADA r.Kilgore@utoronto.ca chignell@mie.utoronto.ca ABSTRACT The modern battlefield is increasingly populated with vast amounts of electronic information. The plethora of data that can be delivered needs to be filtered, interpreted, and formatted in ways that are meaningful and useful for particular tasks and situations. Cross-modal fusion of information should be helpful in optimizing battlespace interfaces to provide the maximum amount of data to the commander and in enhancing their operational picture while avoiding increasing working memory load. In our research, carried out by researchers at the University of Toronto and Defence Research and Development Canada, we are looking at fundamental questions concerning the cross-modal fusion of information. In particular, we are focussing on how visual spatial awareness can be facilitated by presentation of auditory cues or information, and how auditory spatial awareness can be facilitated by visual information. We shall refer to instances of such facilitation as auditory facilitation, and visual facilitation, respectively. Auditory facilitation occurs when the information about entities and spatial relations among them is coded redundantly, using the auditory modality to augment the visual. For instance, designing for auditory facilitation can be used to transmit more information to the commander and aid integration and coordination of disparate pieces of information (Flanagan et al., 1998; Nelson et al., 1998; McKinley & Ericson, 1995; Begault, 1993). There are many potential advantages of using spatialized sound in visualization platforms to aid decision-making and to enhance situational awareness (SA) of the operational picture. The short-term auditory store supplements the visual store, providing an opportunity to store greater amounts of relevant information (Wickens and Hollands, 2000). In addition, the auditory sense is omni-directional and can be used to help in orienting, and in detecting speed and accuracy (Bronkhorst et al., 1996; Begault, 1993). Redundant information presented in a different modality must be compatible with the primary modality if improved task performance and enhanced SA are to be achieved (e.g., St. John et al., 2001; Hollands et al., 2003; Tlauka et al., 2000; Shelton and McNamara, 2004; Harwood and Wickens, 1991; Aretz, 1991). Visual facilitation occurs when visually displayed information enhances auditorily presented information. The type of visual facilitation considered in our research is where visual data representations are used to enhance awareness of target locations in spatial audio displays. This is an important application, because of Boyne, S.; Pavlovic, N.; Kilgore, R.; Chignell, M.H. (2005) Auditory and Visual Facilitation: Cross-Modal Fusion of Information in Multi- Modal Displays. In Visualisation and the Common Operational Picture (pp. 19-1 – 19-4). Meeting Proceedings RTO-MP-IST-043, Paper 19. Neuilly-sur-Seine, France: RTO. Available from: http://www.rto.nato.int/abstracts.asp. Click here to view PowerPoint presentation; Press Esc to exit