1 Knowledge-Based Usability Evaluation For Reality-Based Interaction Georgios Christou European University Cyprus 6 Diogenes St., Nicosia, Cyprus gchristou@acm.org Frank E. Ritter College of IST Penn State University University Park, PA 16802, USA frank.ritter@psu.edu Robert J. K. Jacob Tufts University 161 College Ave., Medford, MA 02155, USA jacob@cs.tufts.edu ABSTRACT The great variety of new (Post-WIMP) interaction styles make them difficult to evaluate and compare. We propose a new evaluation method for them, Knowledge-Based Usability Evaluation (KBUE), that is based on similar ideas to those that drive cognitive architectures, such as ACT-R and Soar. We present KBUE as a way to formally specify the knowledge in the environment and in the user’s head, and how this specification can be used to examine whether the aforementioned set of knowledge covers the required knowledge for the performance of a task in a user interface. We believe that by using this specification, it becomes easier to evaluate and compare Reality-Based interfaces. Author Keywords GOMS; RBI; reality-based interaction; interface evaluation; evaluation method; cognitive architecture. ACM Classification Keywords H.5.2 [Information Interfaces and presentation]: User Interfaces—evaluation/methodology, theory and methods, GOMS; H.1.2 [Models and Principles]: User/Machine Systems — human factors, human information processing. INTRODUCTION Today there exist many different methods for the evaluation of Direct Manipulation Interfaces. However, it seems that these evaluation methods are not well suited for the evaluation of Reality Based Interaction (RBI) interfaces [4]. This is particularly true in cases where the RBI interfaces allow actions that cannot be modeled by the existing evaluation methods, or in cases where the nature of the interaction cannot be captured by these evaluation methods. An example of the first case is the reaching-and-grasping motion that is used in Virtual and Augmented Reality (VR and AR) interfaces, and in Tangible User Interfaces (TUI). This motion is not modeled in any of the existing evaluation methods, simply because it was not used in any of the existing interaction styles. An example of the second case is parallel performance of actions, although this example is being addressed currently by at least two versions of GOMS, GOMSL [15] and CPM-GOMS [10, 11], with different limitations in each version. Because of problems such as the ones already mentioned, it is imperative that we create a new method that allows the evaluation of RBIs. This should not only predict the completion time of tasks in RBIs, but should also allow the direct comparison between designs of an interface in different interaction styles. In this paper, we propose a step towards the solution of this problem in the guise of a new evaluation method we call Knowledge-Based Usability Evaluation (KBUE). BACKGROUND The usability evaluation process is many times coupled to GOMS, because GOMS is one of the most widely known evaluation methods in HCI. Based on the goal of the user to perform a task, the GOMS method suggests the analysis of the task using operators, methods and selection rules. Kieras [14] has written extensively on how GOMS can be used to evaluate how interfaces support tasks. GOMS though, presents some well known problems in the evaluation process. First, it only applies to expert, error-free performance [12], which excludes evaluation for occasional users who are the most frequent users of RBIs. Second, only one of its varieties, CPM-GOMS [7], allows the evaluation of parallel tasks, something that seems common in RBIs [9], but CPM-GOMS is very complex for most evaluation analyses [11]. While other evaluation methods exist, few provide prediction capabilities, and those that do are usually specifically created for direct manipulation interfaces, such as, for example, the User Action Framework [2, 8]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2009, April 4–9, 2009, Boston, MA, USA. Copyright 2009 ACM 978-1-60558-246-7/08/04…$5.00 In Proceedings of the workshop “Challenges in the Evaluation of Usability and User Experience in Reality-Based Interaction”. At CHI 2009 Conference on Human Factors in Computing Systems, Boston, MA, 2009.