Designing Continuous Multisensory Interaction Davide Rocchesso Dept. of Art and Industrial Design IUAV University of Venice Pietro Polotti Dept. of Computer Science University of Verona ABSTRACT We claim that continuous interaction and multisensory feed- back are key ingredients for successful interactive artefacts of the future. However, the complex web of sensors, ac- tuators, and control logic that is necessary for exploiting such ingredients opens tremendous challenges to designers, who are used to visual thinking and discrete interactions. A method of research through pedagogical examples, called basic design and developed in some post-Bauhaus design schools, has been proposed as an effective mean to tackle the complexity of contemporary interaction design. Three such exercises, each prototypical for a class of interactions, are proposed here. The sonic feedback is realized through parametric control of sound synthesis algorithms. Author Keywords Basic Interaction Design; Sound; Continuous Interaction. ACM Classification Keywords H.5.2 User Interfaces: Interaction Styles; H.5.2 User In- terfaces: Auditory (nonspeech) feedback; H.5.5 Sound and Music Computing. EMBODIED INTERFACES, CONTINUOUS INTERACTION In Human-Computer Interaction, the virtues of direct manip- ulation [29] have been widely appreciated and exploited. Its kernel principles can be summarized as: (i) Persistent rep- resentation of the object of interest; (ii) Physical actions or labelled buttons instead of complex syntax; (iii) Rapid incre- mental reversible operations whose impact on the object of interest is immediately visible. In traditional WIMP/GUI interfaces, however, the principles of direct manipulation are applied to a world where the three key components of interaction (model, control, and view) are largely all in the digital domain, as there is physical sep- aration between actions (mouse movements) and feedback (displayed output). An attempt to move control and view mostly into the physical realm led to Tangible User Inter- faces [16], which rely on really-direct manipulation of to- kens having some representational capabilities. The fact that users are manipulating tokens eliminates a level of indirec- tion, since part of the feedback is indeed where the action is. However, a hermeneutic level is often introduced, due to the nature of tokens as representations. This is what embod- ied interfaces [12] tend to avoid by reducing mediations at a minimum, in the spirit of phenomenological thinking. A disembodied interface, as most of existing machine inter- faces are, gives a schizoid perception and action in the world. For the auditory world, this was well understood and exten- sively described by Murray Schafer [27], who also coined the term schizophonia to indicate the separation from sound sources induced by recording and broadcasting means. The danger of schizophonia was felt much earlier by the com- poser Bel´ a Bart´ ok, who wrote in 1937 [6] that “[...] the less foreign bodies are interposing themselves between the hu- man body and the vibrating body or, the longest the time during which the human body controls the vibration is, the more the created musical sound will be immediate and, so to speak, human.” It is easy to generalize such observation to non-musical, everyday situations of interaction with arte- facts. The tightness of the control–display loop give a stronger sense of power which, albeit being subjectively desirable, may be abused. Schafer has been reported to say that it should not be allowed to have sounds without knowing where they come from, so that you can destroy the source if you don’t like it. Indeed, distruction seems to be a compelling outcome of large-scale marketing of partially-embodied in- terfaces. For example, as soon as the Nintendo Wii con- sole entered the market, a web site 1 was launched to collect the experiences of people damaging the remote controller or hurting themselves because of excessive engagement in games. Embodied interfaces tend to exploit continuous manipula- tions in a large extent. Before the industrial revolution, most of human actions in the world were essentially continuous. Opening a door meant grabbing the handle, turning it, and pushing it so that the door swings on its hinges. These con- tinuous actions are far more prone to expressive manipula- tion than simple button triggering, as found for example in modern elevator doors. Indeed, as studies in musical acous- tics show [17], expressiveness can be induced through on- off switches only by temporal fluctuations of repeated trig- gering patterns. Being inherently expressive, continuous ac- tions and gestures are supposed to be more “natural” than triggers. Naturalness here means that control is left to the human manipulator rather than transferred to some machin- ery. In Bart´ ok’s words “less foreign bodies are interposing”. According to the tightness of sensory feedback to the han- dle, control can be more or less direct/physical. For exam- ple, sailing using the tiller is in a sense more engaging than 1 http://wiihaveaproblem.com/ 3