i*Chameleon: A Unified Web Service Framework for Integrating Multimodal Interaction Devices Kenneth W.K. Lo, Wai Wa Tang, Hong Va Leong Department of Computing The Hong Kong Polytechnic University Hong Kong e-mail: {cskenneth,cswwtang, cshleong}@comp.polyu.edu.hk Alvin Chan , Stephen Chan, Grace Ngai Department of Computing The Hong Kong Polytechnic University Hong Kong e-mail: {csschan,cstschan,csgngai}@comp.polyu.edu.hk Abstract— Multimodality inputs are becoming increasingly popular in supporting pervasive applications, due to the demand for highly responsive and intuitive human control interfaces beyond the traditional keyboard and mouse. However, the heterogeneous nature of novel multimodal input devices and the tight coupling between input devices and applications complicate their deployment, rendering their dynamic integration to the intended applications rather difficult. i*Chameleon exploits device abstraction in a web services-based framework to alleviate these problems. Developers can dynamically register new devices with the i*Chameleon framework. They can also map specific device inputs to keyboard and mouse events efficiently. A number of input modalities such as tangible devices, speech, and finger gestures have been implemented to validate the feasibility of the i*Chameleon framework in supporting multimodal input for pervasive applications. Keywords - component; multimodal; human computer interaction; middleware; web services; I. INTRODUCTION Mark Weiser coined the term pervasive computing, the third wave in computing describing the creation of environments with communication capability, yet gracefully integrated with human users [1]. Riding on the success of mature distributed systems and well-developed hardware technologies, nowadays users consider usability and functionality to be equally important. The proliferation of smart and human-natural devices, such as motion capture cameras and speech recognition engines, renders the integration of different modalities into one single application a concrete reality. Users prefer a combination of different modalities rather than just a single one when interacting with computer systems. With multimodality, reliability of human computer interaction was found to improve [2]. The term “multimodal” has been used in many contexts across several research areas [3, 4]. The purpose of multimodal human computer interaction is to study how computer technology can be made more user-friendly, through the interplay among three aspects: user, system and interaction [5]. Exploiting a diversity of networking technologies and multimodal interface techniques, cooperative interaction on pervasive computing system has become a new trend for human computer interaction. Many well-known multimodal systems like ICARE [6] and OpenInterface [7] were developed. OpenInterface is a component-based tool for rapidly deploying multimodal input for different kinds of applications. The framework includes a kernel, a builder and a design tool. After coding their own multimodal application and device drivers, developers will port and deploy them on the platform. However, it does not support plug-and-play and cannot readily support collaborative interaction over the Internet. Promising and intuitive interaction methods with heterogeneous input devices and interaction over the web browser have been developed by W3C. However, there remains much room for the development of multimodal interaction for pervasive systems. A unified multimodal framework is largely needed. This paper presents the i*Chameleon framework, which allows users to customize their interaction, capable of adapting to changes in the environment as well as user requirements according to the features of applications or the needs of different users, like a chameleon. Since this type of dynamic adaptation tremendously increases the complexity in the interaction design, a middleware which can mitigate the tight coupling of applications and specific input devices offers a practical solution. It enables the abstraction of hardware devices and communication technologies, and provides a well-defined interface to interact with other applications or widgets [8]. i*Chameleon focuses on the design considerations of multimodality for pervasive computing. It aims to develop a web-service framework with a separate analytic co-processor for collaborative multimodal interaction, providing a standard and semantic interface that facilitates the integration of new widgets with a wide variety of computer applications. This paper is organized as follows: Section II highlights the desirable system features and requirements. Section III presents the framework workflow and architecture, followed by system implementation and evaluation in Section IV. Brief concluding remarks are given in Section V. II. SYSTEM REQUIREMENTS The system design for supporting non-traditional interfaces such as a multimodal interaction framework is much more complex than those for traditional interfaces. Design efforts tend to be focused on implementing particular interface devices to fulfill specific requirements. Hence, the product interface is usually not flexible enough to be reused on new applications and new widgets or to 9th IEEE International Workshop on Managing Ubiquitous Communications and Services 2012, Lugano (19 March 2012) 978-1-4673-0907-3/12/$31.00 ©2012 IEEE 106