A Ubiquitous Context-sensitive Multimodal Multimedia Computing System and Its Machine Learning-based Reconfiguration at the Architectural Level * Manolo Dulva Hina 1,2 , Amar Ramdane-Cherif 2 , Chakib Tadj 1 1 Electrical Engineering Department, Université du Québec, École de technologie supérieure 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3 Canada 2 Prism Laboratory, CNRS, Université de Versailles – St. Quentin-en-Yvelines 45, avenue des États-Unis, 78035 Versailles Cedex, France Abstract—In this paper, we present our work on a ubiquitous context-sensitive multimodal multimedia computing system that progressively acquires machine knowledge. This ubiquitous computing system supports an automatic selection of devices and modalities deemed appropriate for the user’s context and user’s profile. The ability of the system to do so constitutes its acquired knowledge. The decision making for device/modality selection takes into account if the user has some special needs due to disability. The architecture of the system is designed to be pervasive and is conceived to resist failure. In case of one or more components being missing or found defective, the machine would resist failure by reconfiguring itself dynamically in the architectural level. It finds alternative replacement to the failed component using its acquired knowledge. * This work has been made possible by the funding awarded by the Natural Sciences and Engineering Research Council of Canada (NSERC). Index Terms— incremental learning, multimodal multimedia, ubiquitous computing, dynamic reconfiguration, software architecture. I. INTRODUCTION UBIQUITOUS computing (UC) is one in which a user’s task and data can be transported from one computing environment to another, making computing possible anytime, anywhere. The grand challenges to the realization of ubiquitous computing are (1) making wireless computing as effective and efficient as wired computing, (2) keeping the user’s task and data available and secured, and (3) keeping the system reliable and scalable in face of ever growing tasks and data. In a ubiquitous multimodal multimedia (MM) computing system [1], an additional element – the machine’s acquired knowledge concerning the selection of appropriate media devices and modalities based on user’s context – is also made transportable. Hence, conceptually, in a ubiquitous MM computing system, the user task, his profile, and machine knowledge are all following the user wherever he goes. A computing system is pervasive if it is able to keep itself functional and working at all times according to its specification and design. Generally, a pervasive system is persistent and fault-tolerant. Hence, in order to support pervasiveness, the software (as well as the system’s) architecture must be capable of reconfiguring itself dynamically when facing with a situation wherein (1) one or more of its components fail, or (2) variable system resources (e.g. bandwidth) are restricted and limited, and (3) user priorities change. The idea is obviously for the system to persist, resist error or malfunction, and adapt to the dynamic needs of users whose resources would vary over time. When reconfiguration takes place, a modified architecture comes out. Machine learning (ML) is incremental if its knowledge acquisition is progressive. In general, learning begins when a machine is given its initial knowledge or training set (called a priori knowledge). The machine is then subjected to further training via new scenarios. The machine continuously acquires new scenarios and stores them in its knowledge database. Over time, the machine would have acquired sufficient knowledge that it is, in general, capable of reacting “intelligently” to most scenarios. In contrast to the regular ML, the knowledge acquisition of incremental ML is continuous that it could possibly go on indefinitely. Applying ML to trigger dynamic reconfiguration at the architectural level means that the basis for the system’s modified architecture is taken from the knowledge acquired by the machine from previous scenarios. Hence, the dynamic reconfiguration of the system’s architecture would be efficient and correct only if the machine is sufficiently trained, that it has enough knowledge that the entire system could rely on. In this paper, we present our work on ubiquitous MM computing system, in particular that of the user context-sensitive MM computing system that selects the appropriate media and modalities based on user’s context and if needed, the user’s disability. The architectural structure will be further discussed in this paper, along with concepts of incremental ML. II. RELATED WORKS Some of the relevant works on UC include that of Garlan et al [4] who have worked on Project Aura; their Prism work is a good example. Our work has some similarities with theirs because of the nature of UC where user’s task must be omni-present. Our work is equally different because ours is a multi-agent system, and advocates the use of incremental learning. An element that is needed to realize UC is a network system that supports wired and wireless computing. Satyanarayanan’s works on distributed file access [5], and on Coda [6] are all noteworthy works on networks and distributed system to make UC possible. The challenge in MM has been in the fusion of different information from some human actions and I/O devices. Some important works on multimodal system are that of Oviatt et al work on combined speech and pen inputs [7], Oviatt and Cohen on combined speech and gestures inputs [8]. Situated computing [11] is an example of using the user’s context (in this case, business) in considering user applications’ setting. Ours is new because it combines the two – the configuration of the appropriate MM system based on user’s context. In [3], the authors described the conditions required for an incremental learning task. Okamoto [9] proposed methods for reducing the costs of re-computation of a PDFA (probabilistic deterministic finite state automata)-learning algorithm in a situation where the system collects data using the Wizard of Oz method for the conversational agent model, and in which the number of examples increases gradually. A multi-agent system such as that the work of Shi et al on MAGE, an agent-oriented software engineering environment [10] can be used for e-business and web spider application. In this paper, we present a