A Multimodal Assistive System to Operate a Smart Environment Alexandre Bissoli 1 , Flavio Ferrara 1,2 , Mariana Sime 1 , Teodiano Bastos-Filho 1 1 Laboratório de Robótica e Automação Inteligente, Departamento de Engenharia Elétrica, Universidade Federal do Espírito Santo, Av. Fernando Ferrari, 514, Vitória, Brasil. Tel. (27) 40092077. E-mail: alexandre-bissoli@hotmail.com, teodiano.bastos@ufes.br. 2 Politecnico di Milano: Piazza Leonardo da Vinci, 20133, Milano, Italia. Tel. +39 (02) 23992008. E-mail: femferrara@gmail.com. Abstract – In this work, we addressed the problem of integrate a wheelchair into an intelligent environment. Due to the variety of disabilities that benefit from assistive technologies, an optimal approach would permit the individual to choose the preferred control paradigm in according with the degree of his/her disability. The system allows the handling of various devices in the environment, e.g. a room, by means of biological signals read from the wheelchair. The physical communication between wheelchair and accessories employs radio-frequency (RF) and infrared (IR). We present this system with two kind of assistive control paradigms, using respectively muscle (EMG/EOG) and cerebral (EEG) signals. The muscle control has been implemented using Emotiv EPOC, a low-cost and easy to install headset, while the cerebral signals are recorded through a SSVEP BCI. The user can perform three different commands, through facial gestures or using the BCI, to navigate the menu and operate the desired device. 1 Introduction Within the field of assistive technology, the concept of Brain-Computer Interface (BCI) has emerged [1], with the aim of providing alternative communication and control for individuals with sever motor disabilities, in parallel with more traditional methods including voluntary muscles activation patterns. Examples of these works are: environment control [2], wheelchair navigation control [3]. More recently, Millan et al. discussed the present and future of BCI-based assistive technology including communication and control, motor substitution, entertainment, and motor recovery [4]. The functional model of a BCI system [5] allows identifying the main components involved in a generic BCI system and how they are functionally related. In particular, this architecture, in its simplified version, allows for multimodal configurations, where the control can be performed using different paradigms. This leads to the possibility of directly compare various BCI approaches (SSVEP, motor imagery), distinct biosignals-based controls (such as electromyographic or electrooculographic) or even a combination of different channels as in [6]. Persons with disabilities have struggles interacting with their environment, due to the limitations inherent in their disability. Simple activities like connecting lamp, air conditioning, television or any other equipment, independently, may be impossible for such individuals. With technological advances in the field of sensors and actuators, research has been conducted to evaluate the implementation of a Smart Home with features that can cover different aspects of users’ needs. Environments with multiple sensors and actuators installed are called Smart Environments [7]. 2 Assistive System In the context of smart environments applied to assistive technologies, this paper proposes an input interface allowing a disabled person to turn on and off appliances without help, from his/her wheelchair [7]. Part of this work was to design and build a smart acrylic box that allows up to four appliances in an environment, including television, radio, lamp, air conditioning, fan, among others. The user can issue commands from the wheelchair, and then the signal is transmitted through RF to the smart box, when the corresponding equipment is finally operated. The wheelchair is equipped with a small 10.1'' display that will exhibit the Control Interface (CI) and, in the case of SSVEP BCI, the visual stimuli required for the generation of evoked potentials. The CI will use the display to visualize a menu, through which the User can navigate to operate the desired device. It is worth noting that this menu is meant to be dynamic. Device options can change according to the current room, or be customized by the User before running the system, e.g. removing undesired options. Moreover, for some device we provide additional operations that can be performed using a sub-menu. For example, after turning on the television the system will present a sub-menu with options such as Channel Up, Channel Down, Volume Up, Volume Down. It will always be possible to go back to the main menu and turn off the system. The Control Interface was presented in [8]. This component is completely independent from the type of biological signals and how they are recognized. It offers an interface of procedures that can be accessed through Remote Procedure Call (RPC), that is, one for turning the interface on, one for turning it off, and one for transmitting logical commands encoded as integers. In