J. Gulliksen et al. (Eds.): EIS 2007, LNCS 4940, pp. 36–52, 2008. Formal Testing of Multimodal Interactive Systems Jullien Bouchet, Laya Madani, Laurence Nigay, Catherine Oriat, and Ioannis Parissis Laboratoire d’Informatique de Grenoble (LIG) BP 53 38041 Grenoble Cedex 9, France Forename.Name@imag.fr Abstract. This paper presents a method for automatically testing inter- active multimodal systems. The method is based on the Lutess testing environment, originally dedicated to synchronous software specified us- ing the Lustre language. The behaviour of synchronous systems, con- sisting of cycles starting by reading an external input and ending by issuing an output, is to a certain extent similar to the one of interactive systems. Under this hypothesis, the paper presents our method for automatically testing interactive multimodal systems using the Lutess environment. In particular, we show that automatic test data generation based on different strategies can be carried out. Furthermore, we show how multimodality-related properties can be specified in Lustre and in- tegrated in test oracles. 1 Introduction A multimodal system supports communication with the user through different modali- ties such as voice and gesture. Multimodal systems have been developed for a wide range of domains (medical, military, …) [5]. In such systems, modalities may be used sequentially or concurrently, and independently or combined synergistically. The seminal "Put that there" demonstrator [4] that combines speech and gesture illustrates a case of a synergistic usage of two modalities. The design space described in [25], based on the five Allen relationships, capture this variety of possible usages of several modalities. Moreover, the versatility of multimodal systems is further exacerbated by the huge variety of innovative input modalities, such as the phicons (physical icons) [14]. This versatility results in an increased complexity of the design, development and verification of multimodal systems. Approaches based on formal specifications automating the development and the validation activities can help in dealing with this complexity. Several approaches have been proposed. As a rule, they consist of adapting existing formalisms in the particu- lar context of interactive systems. Examples of such approaches are the Formal System Modelling (FSM) analysis [10], the Lotos Interactor Model (LIM) [23] or the Interactive Cooperative Objects (ICO), based on Petri Nets [21]. The synchronous approach has also been proposed as an alternative to modelling and verifying by model-checking of some properties of interactive systems [8]. Similarly to the previ- ous approaches, the latter requires formal description of the interactive systems such as Lustre [13] programs on which properties, also described as Lustre programs, are © Springer-Verlag Berlin Heidelberg 2008 The original version of this chapter was revised: The copyright line was incorrect. This has been corrected. The Erratum to this chapter is available at DOI: 10.1007/978-3-540-92698-6_37