Perceptive Assistive Agents in Team Spaces Lisa D. Harper, Abigail S. Gertner, James A. Van Guilder The MITRE Corporation 7515 Colshire Drive McLean, VA 22102-7508 +1 703-883-5241 lisah@mitre.org Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: User Interfaces – natural language, evaluation/methodology, interaction styles. General Terms Design, Experimentation, Human Factors. Keywords perceptual computing, assistive agents, intelligent room. 1. INTRODUCTION The use of next generation interface technologies to facilitate human-human collaboration and human-computer interaction (HCI) has tremendous potential for enhancing team interaction and performance in collaborative environments. However, our experience has shown us that the insertion of new devices and technologies cannot be accomplished successfully without 1) adapting to the dynamics of group interaction and 2) facilitating process change in the context of technological change. The addition of next generation capabilities and devices does not necessarily solve problems of communication and team collaboration. For example, though interactive software “whiteboards” are developed and marketed as groupware tools that facilitate group brainstorming, how much do we actually use them in day-to-day meeting contexts? Despite the availability of all of these new tools, computational systems are still perceptually impoverished in terms of understanding natural capabilities of humans including communication, motor and perceptual skills. They do not take into account the ways in which humans naturally interact with each other and with the world. 2. PERCEPTIVE AGENTS Perceptual computing is concerned with using information from a variety of sensors such as video, audio and touch to make the computer aware of what a human is saying or doing via head, hand or body gestures. It also encompasses multimodal input and output. Though team spaces are supported by a variety of technologies from infrastructure to applications to devices, the combined use of these technologies is limited by physical context awareness. That is, the environment is not very adaptable to changes in group structure since elements of the environment (for example, noise level, location and number of participants, focus of activity, user technology savvy, and availability of resources) are not taken into account. We focus on the ability of assistive agents to monitor, access, and manipulate elements of the physical context, such as locating speakers and adjusting audio input to enhance speech recognition and remote listening quality or enriching the collaborative experience of remote participants via sound localization, panoramic video and shared views. Our hypothesis is that through the use of perceptive team space agents managing interactions between humans and the collaboration environment we will enhance team interaction and performance in both local and remote group interaction. Perceptual interfaces seek to leverage information about how humans interact in the physical world. Sensors and devices should be transparent and passive. Pentland [3] proposes that perceptual intelligence is key to interfacing with future generations of machines. These machines include both large-sized interfaces (e.g., large plasma displays) as well as wearable and other ubiquitous devices. Familiar WIMP (windows, icon, menus, pointer) interfaces are not ideally designed for these sorts of computing environments [9]. 3. THE EXPERIMENTAL TEAM ROOM Though we believe that perceptual environments and intelligent assistive agents may change the way we think about human computer interaction, HCI researchers to date have provided relatively little evidence to support this intuition. If we are to assess the potential of advanced HCI technologies to impact the collaborative process, we require a means to experiment on actual users in a working environment. The MITRE Experimental Team Room (ETR) provides a platform for experimentation. We have two objectives: 1) enhance the effectiveness of assistive agents by leveraging perceptual information accessible from the physical environment and 2) provide an experimental foundation for evaluating sophisticated HCI concepts, specifically, for assessing the potential benefits of perceptive assistive agents in the context of team room interactions. These interactions range from small informal working group meetings, discussion groups, instruction, and presentation. We are not only concerned with team activity when participants are co-present, but are also concerned with teams including remote participants – especially those connecting from the desktop or with mobile devices. We have implemented an initial ETR facility (see Figure 1) built on the existing state of the art MITRE team room facilities, which is envisioned as a future experimental facility for internal operational use of advanced collaboration technologies and human computer interfaces. Rather than being a static showcase Copyright is held by the author/owner(s). IUI’04, Jan. 13–16, 2004, Madeira, Funchal, Portugal. ACM 1-58113-815-6/04/0001. 253