Augmented Touch without Visual Obtrusion Francesco I. Cosco Universit ` a della Calabria Carlos Garre URJC Madrid Fabio Bruno Universit ` a della Calabria Maurizio Muzzupappa Universit ` a della Calabria Miguel A. Otaduy URJC Madrid Figure 1: From left to right: image of a visuo-haptic mixed reality scene where the haptic device produces visual obtrusion of the background; visual removal of the haptic device; background completion based on image-based rendering; and final composite scene. ABSTRACT Visuo-haptic mixed reality consists of adding to a real scene the ability to see and touch virtual objects. It requires the use of see- through display technology for visually mixing real and virtual ob- jects, and haptic devices for adding haptic interaction with the vir- tual objects. However, haptic devices tend to be bulky items that appear in the field of view of the user. In this work, we propose a novel mixed reality paradigm where it is possible to touch and see virtual objects in combination with a real scene, but without vi- sual obtrusion produced by the haptic device. This mixed reality paradigm relies on the following three technical steps: tracking of the haptic device, visual deletion of the device from the real scene, and background completion using image-based models. We have developed a successful proof-of-concept implementation, where a user can touch virtual objects in the context of a real scene. Index Terms: H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Artificial, Augmented, and Vir- tual Realities 1 I NTRODUCTION Mixed reality has typically dealt with the visual addition of virtual objects to a real scene. But, in order to achieve a full combina- tion of virtual and real objects, the rest of the sensory modalities must also be capable of perceiving the mixed environment. In this work, we address challenges related to visuo-haptic mixed reality, where a user can see and touch virtual objects in combination with real objects in the scene. Among others, visuo-haptic mixed reality has been introduced in medical applications [7], virtual prototyping, e.g., for the automotive industry [13], or digital entertainment [12]. In the typical desktop virtual-reality setup, the user looks at a screen, and visual and haptic stimuli are presented in a de-located manner. However, a mixed reality setup allows the user to perceive visual and haptic stimuli in a collocated manner, i.e., the user can see and touch virtual objects at the same spatial location. Colloca- tion improves the sensory integration of multimodal cues and makes the interaction more natural, but it also comes with technological challenges. The inclusion of haptic interaction in a mixed reality scene requires the use of a haptic actuator, but most haptic actuators are bulky devices that occupy a large space in the visual region of interest, i.e., in the location where the interaction is actually taking place. Therefore, in a collocated visuo-haptic mixed reality setup, the haptic device becomes an obtrusive visual element. Given this challenge, our main contribution is a novel visuo- haptic mixed reality paradigm. It provides augmented touch, as well as collocated visual and haptic interaction, but without vi- sual obtrusion produced by the haptic device. This mixed reality paradigm allows a user to touch virtual objects that are added to a real background or context scene, without suffering from obtrusive occlusion of this context scene. We propose an algorithm that makes this visuo-haptic mixed re- ality paradigm possible. It is based on visual removal of the hap- tic device from the context scene, together with image-based back- ground completion. Given view and haptic configurations, we first identify the region of the image plane occupied by the haptic de- vice, i.e., the region where the device is producing visual obtrusion. Then, given prerecorded views of the context scene, we substitute the haptic device with a view-dependent image of the background, using image-based rendering techniques. Last, we add the virtual objects, both visually and haptically, to produce the visuo-haptic mixed reality scene. The importance of unobtrusive haptic interaction has been ad- dressed in the past, and the proposed answers relied on mechanical solutions that placed the haptic actuators far from the region of in- terest using string-based haptic devices [16], or optical solutions based on retroreflective paint and a head-mounted projector [9]. Here, we propose instead a computational solution, which we be- lieve increases the versatility of visuo-haptic interaction setups. 2 RELATED WORK Several researchers have addressed the importance of collocating visual and haptic stimuli, as this allows virtual tasks to be car- ried out from a first-person point of view [5]. Visual and haptic de-location is however quite common, because the construction of a de-located setup is far simpler. Congedo et al. [4] emphasize that, in tasks where the contribution of touch is important, great effort should be undertaken to collocate vision and touch, so that the weight of the non-dominant modality, i.e., touch, is not penal- ized. Spence et al. [14] summarize crossmodal congruency effects involving vision and haptics. Visuo-haptic collocation can be achieved in several ways, and the most popular ones include workbenches with stereo projection systems [2, 16], mirror-based projection systems where the virtual image occludes the real scene [15], or head-mounted displays with head and device tracking [1]. Our approach uses see-through head- mounted display technology, and our mixed reality paradigm is in-