On Your Feet! Enhancing Vection in Leaning-Based Interfaces through Multisensory Stimuli ABSTRACT When navigating larger virtual environments and computer games, natural walking is often unfeasible. Here, we investigate how alternatives such as joystick- or leaning-based locomotion interfaces (“human joystick”) can be enhanced by adding walk- ing-related cues following a sensory substitution approach. Using a custom-designed foot haptics system and evaluating it in a mul- ti-part study, we show that adding walking related auditory cues (footstep sounds), visual cues (simulating bobbing head-motions from walking), and vibrotactile cues (via vibrotactile transducers and bass-shakers under participants’ feet) could all enhance par- ticipants’ sensation of self-motion (vection) and involve- ment/presence. These benefits occurred similarly for seated joy- stick and standing leaning locomotion. Footstep sounds and vi- brotactile cues also enhanced participants’ self-reported ability to judge self-motion velocities and distances traveled. Compared to seated joystick control, standing leaning enhanced self-motion sensations. Combining standing leaning with a minimal walking- in-place procedure showed no benefits and reduced usability, though. Together, results highlight the potential of incorporating walking-related auditory, visual, and vibrotactile cues for improv- ing user experience and self-motion perception in applications such as virtual reality, gaming, and tele-presence. CCS Concepts • Information interfaces and presentation - multimedia infor- mation systems, artificial, augmented, and virtual realities; Keywords Navigation interface; 3D user interface; leaning; VR; gaming; vibration; bass-shaker; whole-body interface; surface textures. 1. INTRODUCTION While joysticks and gamepads are widely used methods for navi- gating games and virtual reality (VR), they offer hardly any of the self-motion cues accompanying real-world locomotion. Allowing for free-space walking while wearing a head-mounted display provides appropriate physical motion cues, but is often unfeasible because of restrictions in the tracked space, concerns of safety, cost, or technical complexity, or fatigue for longer exposures. Leaning-based navigation interfaces using the Wii balance board [20, 21, 57, 61] and other approaches [19, 34, 62] have been pro- posed and used as an alternative that allows for long-range loco- motion without running into limitations of the tracked space. Compared to joystick and gamepad interfaces where the human body is mostly passive and vestibular/proprioceptive cues are largely lacking, leaning-based interfaces can improve navigation performance [21] and provide a more immersive and embodied experience as they allow for at least some full-body involvement and vestibular motion cueing, which can enhance self-motion perception (vection) [31, 40, 44]. Nevertheless, leaning-based interfaces still lack many of the self-motion cues experienced during real-world locomotion, such as full vestibular cues from translations and rotations, proprioceptive cues from walking, air moving by our ears, as well as haptic and auditory cues from our feet touching ground. We designed a multipart study to investigate if and how joystick- and leaning-based locomotion interfaces might be improved by adding different walking-related self- motion cues such as auditory cues (footstep sounds), visual cues (simulating bobbing head-motions from walking), vibrotactile cues (via vibrotactile transducers and shakers under participants’ feet) and minimal walking-in-place. While there is evidence that the visually-induced sensation of illusory self-motion (“vection”) can be enhanced by adding matching auditory cues (e.g., dynamic sound fields) and vibra- tions/subsonics [24, 32, 40], it is largely unknown how they affect active locomotion conditions using seated joystick versus standing leaning interfaces. As self-motion sensations are typically en- hanced by multisensory stimulation [32, 43–46], we do expect overall enhancement by providing additional self-motion related cues in the current studies. Similarly, we hypothesized additional benefits including improved speed and distance perception and overall performance and usability, especially in systems that re- quire more precise navigation or wayfinding, or a higher level of user engagement through increased realism. Some may consider the usage of walking-in-place (WIP) tech- niques [55] to overcome the caveat of lack of motion cues, as this technique does provide some proprioceptive cues by mimicking physical walking while not physically moving around. However, WIP can be tiring over time, which might lead to reduced usage. In addition, while WIP gestures are fairly natural for forward walking, they can be awkward for walking backwards or strafing, * The first and last author contributed equally to this article. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. SUI '16, October 15 - 16, 2016, Tokyo, Japan Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-4068-7/16/10…$15.00 DOI: http://dx.doi.org/10.1145/2983310.2985759 Ernst Kruijff 1* , Alexander Marquardt 1 , Christina Trepkowski 1 , Robert W. Lindeman 2 , Andre Hinkenjann 1 , Jens Maiero 1 , Bernhard E. Riecke 3* 1 Bonn-Rhein-Sieg University of Applied Sciences, Grantham-Allee 20, 53757 Sankt Augustin, Germany, 2 HIT Lab NZ, University of Canterbury, Christchurch 8140, New Zealand 3 Simon Fraser University, 250 –13450 102 Avenue Surrey, BC, V3T 0A3, Canada {ernst.kruijff, alexander.marquardt, christina.trepkowski, Andre.Hinkenjann, jens.maiero}@h-brs.de, gogo@hitlabnz.org, b_r@sfu.ca