Development of the TUP Model - Evaluating Educational Software Roman Bednarik, Petri Gerdt, Ramin Miraftabi, Markku Tukiainen Department of Computer Science University of Joensuu, Finland {roman.bednarik, petri.gerdt, ramin.miraftabi, markku.tukiainen}@cs.joensuu.fi Abstract Usability evaluation is a crucial step in the iterative software development life-cycle, leading into successful usage of systems. However, most of the usability evaluation methods and tools do not consider all aspects of educational environments. These lacks materialize as unusable results that lead to failures in educational software. Often these results are not comparable and credible, or easily accessible to educators selecting the proper systems for use in their settings. The Technology, Usability, Pedagogy (TUP) evaluation model equally concentrates on the technological, usability, and instructional aspects of educational environments, leading to complete evaluations. In this paper we present the TUP model with the use scenario included and our current WWW prototype of evaluation system based upon it. 1. Introduction Undoubtedly, computing and the use of technology currently play key roles in supporting educational processes. Due to the increasing availability and diversity of educational environments, educators face the difficult task of finding and selecting the proper environment for their own courses and classes. The tasks of evaluating educational environments as well as finding and comparing the evaluations are difficult for teachers. We assert that the problem is rooted in the lack of a coherent approach to the evaluation of educational environments. We use ''educational environment'' to mean any software or electronic material that is used in an educational setting. The role of usability issues in educational software is to contribute towards the achievement of educational goals. Therefore, usability evaluation is not only a crucial step during the software development life- cycle, but it is also important for assessing the quality of released systems. Although a vast number of evaluation methods have been developed in the past, only a few of them can be applied to educational environments. This is caused by the fact that educational settings introduce a complex set of elements, processes, and relations between them. A method which allows for comprehensive analysis and evaluation of all the involved factors has yet to be developed. In this paper, we first present our proposal for an evaluation method for educational environments. We then proceed to describe the design of our web-based prototype evaluation system which uses the TUP model. After outlining some initial experiences with the prototype we end our paper with discussion about the future of the TUP. 2. The TUP model The TUP model is an acronym derived from the words technology, usability, and pedagogy [5], an evaluation scheme which equally concentrates on three aspects of educational environments: The technological aspects focus on issues such as the dependencies and interactions between the environment and surrounding software and hardware equipment, issues concerning security and privacy, material sharing and reuse. The usability aspects in educational environments concern into some extent the traditional usability issues as known from [6] and [7], or motor and perceptual factors [4]. The pedagogical aspects refer to the need for assessing the instructional qualities of educational environments. Thus, the issues such as the context of learning, the roles of participants, the motivation or progress tracking and assessment are concerned. The use scenario is an integral part of the TUP evaluation scheme. In addition to providing information related to technology, usability, and pedagogy, the evaluator-reviewer also submits the context in which the environment has been used. We argue that the integration of the context of use into the evaluation scheme satisfies the needs of the professional teachers when searching for appropriate software for their classes. The teachers know the settings of the courses they plan and can query the system in order to obtain the reviews of the materials or software whose use scenarios match their preference. Fig. 1 illustrates how the three aspects and the use scenario are interrelated in the TUP. Both usability and pedagogy build upon the foundation of the technological aspects. The use scenario intersects all of the other categories: the context determines whether a certain feature is applicable. Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT’04) 0-7695-2181-9/04 $20.00 © 2004 IEEE Authorized licensed use limited to: Joensuun Yliopisto. Downloaded on November 6, 2009 at 01:54 from IEEE Xplore. Restrictions apply.