Niki and Julie: A Robot and Virtual Human for Studying Multimodal Social Interaction Ron Artstein, David Traum, Jill Boberg, Alesia Gainer, Jonathan Gratch, Emmanuel Johnson, Anton Leuski USC Institute for Creative Technologies 12015 Waterfront Dr., Playa Vista CA 90094 USA {lastname|ejohnson}@ict.usc.edu Mikio Nakano Honda Research Institute Japan 8-1 Honcho, Wako, Saitama 351-0188 Japan nakano@jp.honda-ri.com ABSTRACT We demonstrate two agents, a robot and a virtual human, which can be used for studying factors that impact social in- fluence. The agents engage in dialogue scenarios that build familiarity, share information, and attempt to influence a hu- man participant. The scenarios are variants of the classical “survival task,” where members of a team rank the impor- tance of a number of items (e.g., items that might help one survive a crash in the desert). These are ranked individu- ally and then re-ranked following a team discussion, and the difference in ranking provides an objective measure of so- cial influence. Survival tasks have been used in psychology, virtual human research, and human-robot interaction. Our agents are operated in a “Wizard-of-Oz” fashion, where a hidden human operator chooses the agents’ dialogue actions while interacting with an experiment participant. CCS Concepts •Human-centered computing → Human computer interaction (HCI); Collaborative and social comput- ing; •Computing methodologies → Intelligent agents; •Computer systems organization → Robotics; Keywords Human-robot/agent multimodal interaction, Multimodal in- teractive applications, Affective Computing and interaction 1. INTRODUCTION One factor that impacts social influence in general, and the survival task in particular, is embodiment. Psycholog- ical and communication studies suggest that embodiment increases social influence. For example, participants are less persuaded in a lunar survival task when communicat- ing via teleconference compared with face-to-face interac- tion [7]. Findings in human-machine interaction are mixed, Figure 1: Artificial Agents Julie and Niki showing increases in subjective engagement but failing to demonstrate objective persuasion [1]. This is the rationale for using two agents with radically different embodiments. Another factor that shapes social influence in general, and survival tasks in particular, is the familiarity team mem- bers have with each other. In general, people are less influ- enced by strangers or people they feel more distant from [3]. Telling jokes with machine teammates has been shown to increase persuasion in a lunar survival task [6], and various rapport-building techniques enhance human-machine team- work [4]. For this reason, the agents also engage in an ice- breaker to establish familiarity with the participant. 2. SYSTEM DETAILS We will demonstrate two embodied agents (Figure 1): a NAO robot, named Niki, and a virtual human, named Julie, whose animated body can appear on a monitor. Julie is presented in two modes, either multimodally with voice and virtual embodiment, or voice only accompanied by a static image, as if through a teleconference. Both agents are controlled by a “Wizard of Oz” system, with a human wizard using a push-button GUI (Figure 2). The interface runs in a web browser and sends messages using the VHMsg messaging protocol to trigger agent be- haviors. 1 The system architecture is shown in Figure 3. Behaviors of Julie, the virtual human, were created using the Virtual Human Toolkit [5], and her voice was synthe- sized using a voice from NEOspeech’s text to speech engine. Niki is a NAO Robot, a humanoid robot commonly used in human robot interaction studies [2]. Niki’s non-verbal be- haviors were authored using Choregraphe, a multi platform 1 https://sourceforge.net/projects/vhmsg/ Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). ICMI’16, November 12–16, 2016, Tokyo, Japan ACM. 978-1-4503-4556-9/16/11...$15.00 http://dx.doi.org/10.1145/2993148.2998532 402