A. Pirhonen and S. Brewster (Eds.): HAID 2008, LNCS 5270, pp. 60–69, 2008. © Springer-Verlag Berlin Heidelberg 2008 Multimodal Interaction: Real Context Studies on Mobile Digital Artefacts Tiago Reis, Marco de Sá, and Luís Carriço LaSIGE, Faculdade de Ciências, Universidade de Lisboa treis@lasige.di.fc.ul.pt,{marcodesa,lmc}@di.fc.ul.pt Abstract. The way users interact with mobile applications varies according to the context where they are. We conducted a study where users had to manipu- late a multimodal questionnaire in 4 different contexts (home, park, subway and driving), considering different variables (lighting, noise, position, movement, type of content, number of people surrounding the user and time constraints) that affect interaction. This study aimed at understanding the effect of the con- text variables in users’ choices regarding the interaction modalities available (voice, gestures, etc). We describe the results of our study, eliciting situations where users adopted specific modalities and the reasons for that. Accordingly, we draw conclusions on users’ preferences regarding interaction modalities on real life contexts. Keywords: Multimodal Interaction, Mobile Devices, Studies in Real Contexts. 1 Introduction and Background Multimodal interaction is a characteristic of everyday human activities and communi- cations, in which we speak, listen, look, make gestures, write, draw, touch and point, alternatively or at the same time in order to achieve an objective. Considering the human perceptual channels [4] through the inclusion of elements of natural human behavior and communication on human-computer interfaces is the main goal of mul- timodal interaction. Multimodal interfaces can improve accessibility for different users and usage contexts, advance performance stability, robustness, expressive power, and efficiency of mobile activities [5, 6]. Recently, concerning the special needs of several groups of users, many researchers have focused on the design and development of universally accessible systems. These consider: impairments [1] and various usage context variables [2, 3]. Design approaches such as “Inclusive Design” or “Design for all” enhance the usability of the applications and consider the existence of multiple interaction modalities, providing accessibility and support to impaired users and ena- bling non-impaired users to interact with applications in suboptimal conditions [1]. This kind of interaction has been explored from different points of view [1, 7, 8, 9]. (e.g. support to impaired users, support to non-impaired users in suboptimal situa- tions, augmentation of unimodal activities, games, multimedia applications, etc.). The interaction modalities included on a multimodal system can be used either in a com- plementary way (to supplement the other modalities), in a redundant manner (to pro- vide the same information through more than one modality), or as an alternative to the