Neural Comput & Applic (2000)9:297–305  2000 Springer-Verlag London Limited A Set of Neural Tools for Human-Computer Interactions: Application to the Handwritten Character Recognition, and Visual Speech Recognition Problems G. Vaucher, A. R. Baig and R. Se ´guier SUPE ´ LEC, Signal Processing and Neural Networks Team, Cesson Se ´vigne ´, France This paper presents a new technique of data coding and an associated set of homogenous processing tools for the development of Human Computer Inter- actions (HCI). The proposed technique facilitates the fusion of different sensorial modalities and sim- plifies the implementations. The coding takes into account the spatio-temporal nature of the signals to be processed in the framework of a sparse representation of data. Neural networks adapted to such a representation of data are proposed to per- form the recognition tasks. Their development is illustrated by two examples: one of on-line handwrit- ten character recognition; and the other of visual speech recognition. Keywords: Human-machine interaction; Lipreading; On-line handwritten character recognition; Spatio- temporal coding; Spatio-temporal neural networks; Spatio-temporal patterns; Spiking neurons; Visual speech recognition 1. Introduction The present means of communication between the humans and machines are more and more questioned and criticised. The pocket computer, which has a certain interest for the general public, has sizes such that the utilisation of a keyboard or mouse is unsuitable. Moreover, a broader expansion of the use of computers is harnessed by the fact that they Correspondence and offprint requests to: Correspondence and offprint requests to: G. Vaucher, Supe ´lec, Signal Processing and Neural Networks Team, B.P. 28, 35511 Cesson Se ´vigne ´ cedex, France. Email: gilles.vauchersupelec.fr impose the burden of understand and learning the functioning of the tool on the user. Ideally, it should be the task of the computer to adapt itself to the user. Realisation of HCI, which permits someone to command all the power of a machine while using means of communication which are more natural, is a challenge which occupies a large community of scientists and technicians. What we propose is to deal with the problem by adopting a unified approach at the level of the shaping of the data, as well as at the level of its processing. After having identified many characteristics com- mon to HCI, this paper presents a coding and a mode of homogenised processing, adapted to the signals thus characterised. To illustrate this approach, two examples of applications are presented. 2. User Interfaces 2.1. Characteristics of Signals in HCI Variety of signals to be processed. At present there exist many means of operational inputs for achieving an intuitive HCI. For example, the stylus for writing and drawing, the microphone for speech and sound, the touch screen and the TouchPad for the sense of touch, and video for the interpretation of gestures, recognition of facial expression, and analysis of the movements of the eyes, head and lips. Characteristics of the signals. The main character- istic which all these signals have in common is the