Designing a Model Human Cochlea: Issues and challenges in crossmodal audio-haptic displays Maria Karam Ryerson University Toronto, Ontario Canada maria.karam@ryerson.ca Deborah I. Fels Ryerson University Toronto, Ontario Canada dfels@ryerson.ca ABSTRACT In this paper, we describe a Model Human Cochlea (MHC), a sensory substitution technique aimed at translating some of the emotional content expressed in music onto a haptic ambient display. We present some of the issues and chal- lenges encountered in designing the model. This research is situated within the domain of crossmodal displays, with specific focus on enhancing the entertainment experience as- sociated with film audio for users who are deaf or hard of hearing. The interface, in its final form factor, will be in- tegrated into an EmotiChair, a multi-sensory entertainment interface that supports crossmodal audio-haptic display in- teractions. To assist with the design of the MHC, we have developed a flexible prototype to support research in cross- modal audio-haptic displays. Details of the multidisciplinary design process that has informed the development of the the MHC prototype, and the evaluation methodology adopted to explore the different configurations of the MHC are pre- sented. Categories and Subject Descriptors H.1.2 [Information Systems]: User/Machine Systems— Human factors, human information processing, software psy- chology ; H.5.2 [Information interfaces and presenta- tions]: User Interfaces—Haptic I/O ; General Terms Sensory substitution, model human cochlea, crossmodal dis- plays, haptic interfaces, music, emotion 1. INTRODUCTION Crossmodal displays represent a rapidly growing field in human-computer interaction research, where information that is designed to be presented to one sensory modality is dis- played using an alternative modality. Much of the research focus within this domain lies in the exchange of input and output stimuli between the audio, visual, and tactile modal- ities, although it is theoretically possible to create an al- ternative display for any sensory modality. These types of alternative display fall into the class of crossmodal displays, with the most prominent areas of research based in the pre- sentation of audio signals using visualizations [17], visual in- formation using haptic displays [21], and audio signals using haptic displays [3]. Crossmodal displays are used for a vari- ety of applications, including mobile computing [3], virtual reality [4], assistive and adaptive technologies [9], and enter- tainment applications [14]. Often these displays are used to augment a primary display with a source of redundant infor- mation that is presented using a different display modality. For example, a driving game that is presented on a primary visual display can also incorporate a secondary audio display to provide the corresponding driving sounds to the user. A a third display can also be used to further extend the mul- tisensory experience by including tactile sensations such as motions that reflect the physical events ocurring in the game environment. A different approach considers the substitution of one modal- ity using another, where the stimuli from the input modal- ity is replaced by the stimuli from the alternative display modality. This type of display is primarily aimed at provid- ing users who are blind or deaf with an alternative means of experiencing the stimuli from one sensory modality using the other, however it also offers an enhanced sensory experience to all users. Some examples include pin arrays, which pro- vide a tactile replacement for visual information, and music visualizations, which offers a visual interpretation of music audio. One of the main research challenges in creating such displays is in determining which of the characteristics of the input modality to present using the alternative display. This ex- change of stimuli from one modality to another is commonly referred to as sensory substitution [24] and involves the iden- tification, selection, and mapping of characteristics between the input and the display modalities. The characteristics or properties of the two modalities must be identified, drawing on perspectives such as the physical, perceptual, or compu- tational nature of the stimuli; visual stimuli can be described using angles, lines, or coordinates as physical characteristics, perceptual characteristics such as colour or type, or compu- tational characteristics such as the pixel activation of the rendered object. Once identified, the properties of the in- put modality must then be interpreted, translated, or oth- erwise mapped onto the properties of the alternative display to achieve the desired effect.