Traces: Semi-Autonomous Avatars Phoebe Sengers and Simon Penny and Jeffrey Smith Abstract. Avatars are generally thought of as a direct representation of a user in a virtual environment. But as the complexity of virtual en- vironments increases and, with it, the scope and complexity of possi- ble avatar behavior, it becomes more and more difficult to maintain a direct correlation between the user’s wishes and the avatar’s actions. Instead of this now inadequate notion of ‘avatar-equals-user,’ we pro- pose that avatars can be fruitfully thought of as semi-autonomous agents, which have their own behaviors and intentionality, but are intimately tied to the user’s actions. We describe a system incorpo- rating avatars with varying levels of autonomy: Traces, a Virtual Re- ality system in which a user’s body movements spawn avatars which gradually become more autonomous. The formulation of avatars as semi-autonomous agents opens a new and rich conceptual space in the design of virtual environments. 1 Introduction Avatars are computational agents which represent users in virtual en- vironments (VE’s). Avatars range from simple programs that do little more than shuttle text between members of a text-based VE, to 3-D figures which move around graphical VE’s in response to motion commands by the user, to complex agents which search the Web for the user and have autonomous decision-making capability. Gener- ally, avatars, or ‘user embodiments,’ are thought of as soul-less bod- ies for which the user acts as mind. Correspondence between the user’s wishes and the avatar’s actions is maintained using low-level commands like “walk forward 3 steps” or “pick up the box,” or by using hardware sensors on the user’s body which are translated into the corresponding movements for the avatar. In this way of thinking; the avatar is what we might call a nonautonomous agent; the avatar, fundamentally, is the user. But as the complexity of virtual environments increases and, with it, the scope and complexity of possible avatar behavior, it becomes more difficult for the user to directly control all aspects of the avatar using simple low-level commands. In response, avatars have been built that allow the user to specify behavior at various levels, from “go north” to “find me an appropriate article” to “negotiate the re- lease of hostages,” while the avatar uses its own intelligence to fill in the details [1, 14, 7]. As these avatars become more independent, the idea that the avatar is a simple extension of the user becomes problematic [9, 17]. In essence, the avatar is no longer a thought- free ‘body,’ but develops its own ‘brain’ that takes over some of the mental tasks needed to function in complex environments. Media Arts Research Studies, GMD - German National Research Center for Information Technology, Schloss Birlinghoven, D-53754 Sankt Augustin, Germany, email: Phoebe.Sengers@gmd.de Carnegie Mellon University, School of Art and Robotics Institute, 5000 Forbes Ave., Pittsburgh PA 15213, USA, email:penny+@andrew.cmu.edu Carnegie Mellon University. Robotics Institute, 5000 Forbes Ave., Pitts- burgh PA 15213, USA, email:jeffrey@cs.cmu.edu In this context, the currently dominant metaphor of “user avatar” is no longer adequate to describe or innovatively solve prob- lems that come up in designing avatars. In this paper, we propose that in the context of current trends in avatar research, avatars can be more fruitfully thought of as semi-autonomous agents. Avatars are thought of as agents like any other with their own behavior and intentional- ity, but with a particularly intimate relationship with the human user. This notion of avatar not only describes current avatar work more accurately, but also widens the conceptual space of possible avatars. Avatars can now appear on the full range of autonomy levels, from a fully passive traditional avatar to a fully active traditional agent. In this paper, we will describe our experience with Traces, a Virtual Reality system we have built which allows the user to interact with a series of 3-dimensional avatars of increasing autonomy 2 Fellow Travelers Avatars may be talked about as though they are simple cyberspace extensions of the user; in the practice of building them, this idea quickly loses relevance. Avatars are often recalcitrant; they don’t do what I tell them, or at least not what I meant. It is not possible to actually construct avatars for any period of time, especially those with complex behaviors, without realizing that it takes a great deal of engineering effort to engender the illusion that the avatar is identi- cal to the user. As Bowers, O’Brien and Pycock argue, often a great deal of technical and social effort goes into having the avatar behave nonautonomously, i.e. as a direct and accurate representative of the user [2]. At the same time, several researchers have done innovative work that, rather than attempting to get rid of unwanted autonomy, uses that autonomy as a resource to create new, useful forms of the avatar-user relationship. Hannes Vilhj´ almsson’s and Justine Cassell’s pioneering system BodyChat, for example, uses autonomy in the form of body lan- guage to support interaction via avatars [16]. That is, while the user is chatting with other people, their avatars autonomously display the kinds of physical signals humans unconsciously use to support com- munication, like using glances to show whether or not one is open to communication, raising eyebrows on emphasis words, and using gaze exchange to support turn-taking. These are behaviors which are essential for supporting communication, but of which humans are generally unaware and therefore would find difficult to directly con- trol. While Vilhj´ almsson and Cassell’s avatar does have some semi- autonomous behavior, it is still a direct representative of the user — the avatar does what the user would do if s/he could. Interestingly, evaluation of their system suggested that users actually feel more in control of these avatars than ones where they had to directly control the avatar’s body movements [3]. Avatars do not just behave; they also sense the virtual environment for the user. Michael Mateas has developed ‘subjective avatars’ for