Role-based Evaluation of Awareness Support: The SCAPE Method Jill L. Drury The MITRE Corporation 202 Burlington Road, Bedford, MA 01730 USA jldrury@mitre.org ABSTRACT This paper is describes the first methodology developed specifically for specifying and evaluating awareness- related features of synchronous collaborative computing applications. While previous work acknowledges that roles are important to understanding awareness needs, no method has been previously developed to provide a fine- grained, role-based approach to both specifying the awareness-related characteristics of collaborative computing applications and evaluating whether the application meets those specifications. We feel the framework we developed for this methodology can be used by other researchers, as well, to develop means of specifying and evaluating the ability of collaborative applications to support awareness. Keywords Evaluation, CSCW, awareness, privacy, complete awareness, partial awareness, symmetrical awareness, asymmetrical awareness, awareness relationship, SCAPE. INTRODUCTION Computing technologies that enable people to work together in real time are proliferating, and whether on the desktop or in the palm, these technologies share a key usability issue: Can users tell enough about each other's activities to collaborate effectively? Information that users have about each other's activities is commonly called awareness information. It helps them know who else is working in a shared workspace and what the other people are doing. Designed to emulate the kinds of non-verbal cues that people get when they collaborate face-to-face in the same physical location, awareness information is important for effective collabor- ation and coordination. There are many definitions of awareness, such as those listed in [Drury et al. 2003]. There is no standard definition of awareness, however, and we maintain that many of the existing definitions lack the precision needed to form the basis for a rigorous specification and evaluation technique. In this paper, we present a framework for fine-grained analysis of the awareness requirements of users of a synchronous collaborative application. The framework takes into account the role-related differences between users. We then present a novel method for evaluation of awareness support, based on the framework. It differs from previous work in that it allows usability engineers and evaluators to systematically create fine-grained, role- based awareness specifications, and then to evaluate applications for compliance with the specifications. RELATED WORK HCI evaluation methods can be divided into three broad categories: inspection methods (expert examination of user interfaces), empirical methods (experiments and user studies involving human test subjects), and formal methods (e.g., dialog and task modeling techniques). We focus on inspection methods because they are usually quick and inexpensive, compared to other methods and are known to be effective at uncovering usability problems in single-user applications. The most widely used inspection method is heuristic evaluation [Molich and Nielsen 1990]. None of the heuristics address collaboration issues such as awareness. Three previous research efforts have aimed at developing heuristics for collaborative applications: Benchmarks for workspace awareness [Villegas and Williams 1997], heuristics based on the Locales Framework [Greenberg, et al. 1999], and heuristics based on the “mechanics of collaboration” [Baker, et al. 2001]. These three research efforts have provided very general heuristics for use in evaluating collaborative applications. They assume that all participants in a collaborative session have the same awareness needs. None of them provides fine-grained analysis and evaluation of the awareness needs of individual participants using a collaborative application. AWARENESS FRAMEWORK In this section, we describe a framework for fine-grained analysis of awareness requirements. We start by defining the awareness that one participant has of another participant. Such a definition can be applied generally by viewing interactions between all participants as a series of pair-wise interactions. We define awareness as follows: Awareness: Given two participants p 1 and p 2 who are collaborating via a synchronous collaborative application, awareness is the understanding that p 1 has of the identity and activities of p 2 . To support p 1 ’s understanding of p 2 , an application provides p 1 with information about the identity and