Integrating Usability Design and Evaluation: Training Novice Evaluators in Usability Testing Mikael B. Skov and Jan Stage Department of Computer Science Aalborg University Aalborg Øst, Denmark +45 9635 8080 {dubois, jans}@cs.aau.dk ABSTRACT This paper reports from an empirical study of training of usability testing skills. 36 teams of novice evaluators with an interest but with no education in information technology were trained in a simple approach to web-site usability testing that can be taught in less than one week. The evaluators were all first-year university students. The paper describes how they applied this approach for planning, conducting, and interpreting a usability evaluation of the same web site. We discover that basic usability testing skills can be developed. The student teams gained competence in defining good task assignments and ability to express the problems they found. On the other hand, they were less successful when it came to interpretation and analytical skills. They found quite few problems, and they seemed to lack an understanding of the characteristics that makes a problem list applicable. Keywords Usability test, training novices, dissemination of usability skills INTRODUCTION Despite several years of research on usability testing and engineering, many computer-based information systems still suffer from low usability [4]. One problem arises from the fact that planning and conducting full-scale usability tests yields key challenges of e.g. user integration [7]. Considerable costs arise when a large group of users is involved in a series of tests. Furthermore for some applications it is difficult to recruit prospective test subjects [2]. The theoretical usability evaluation approach denoted as heuristic inspection evolved as a creative attempt to reduce such costs of usability evaluations [5, 6, 8]. The idea in heuristic inspection is that an interface design is evaluated by relating it to a set of guidelines, called heuristics [8]. The aim of the heuristics is to equip people who are not usability specialists to conduct heuristic inspections. Some of the empirical studies of the approach have been based on university students or readers of a computer magazine who act as evaluators [8].The idea behind heuristic inspection is to accomplish a simplified way of conducting usability tests. However, the empirical results indicate that we move the problem from finding users to finding user interface specialists. For a small organization developing web-based systems both of these problems may be equally hard to overcome. On a more general level the relevance of heuristic inspection can also be questioned. It has been argued that real users are an indispensable prerequisite for usability testing. If they are removed, it is at the expense of realism [10]. In this paper, we pursue a different idea of enhancing the knowledge of usability for software designers. One key problem in improving the usability of systems is the challenges involved in the interplay between the design and the evaluation of the system. Sometimes these activities are separated and detached making the interplay difficult and challenging e.g. one potential problem arises from the fact that designers and evaluators do not share a common language or set of tools in order to communicate. Our study explores how we can enhance usability testing competences for novice evaluators. Our aim is to train novice evaluators and compare their usability testing performances against the performances of professional usability testing labs. For our study, we use first-year university students as novice evaluators. First, we outline the taught usability testing approach and present the experiment behind the paper. Secondly, we compare the performances of the novice evaluators to the performances of professional labs on 17 different variables. Finally, we discuss and conclude our study. METHOD We have made an empirical study of the usability approach that was taught to the novice evaluators. Usability Testing Approach The approach to usability testing was developed through a course that was part of a curriculum for the first year at Aalborg University, Denmark. The overall purpose of the course was to teach and train students in fundamentals of