Usability Evaluation of Learning Object Indexation: The ARIADNE Experience Jehad Najjar, Joris Klerkx, Stefaan Ternier, Katrien Verbert, Michael Meire and Erik Duval Computer Science Dept., Katholieke Universiteit Leuven, Belgium Jehad.Najjar@cs.kuleuven.ac.be Joris.Klerkx@cs.kuleuven.ac.be Stefaan.Ternier@cs.kuleuven.ac.be Katrien.Verbert@cs.kuleuven.ac.be Michael.Meire@cs.kuleuven.ac.be Erik.Duval@cs.kuleuven.ac.be Abstract: This paper investigates usability problems of indexation tools for Learning Objects. Such tools are used to describe and deploy learning objects with their metadata in Learning Object Repositories (LOR’s). Metadata is provided to facilitate the search and management of learning objects. The complexity of manually indexing learning objects results in a bottleneck for the introduction of such objects (Duval and Hodgins, 2003), (Kabel, et al., 2003). In this paper, we present the results of a usability evaluation aimed at determining some usability issues in two ARIADNE indexation tools. Results show that metadata semantics is significantly influenced by usability perspectives (such as the user interface, functionalities provided, domain knowledge, etc). The findings and recommendations of this study are generalised for other indexation tools. Keywords: Metadata, User Studies, Learning Object Indexation, Learning Objects 1. Introduction In recent years, a lot of attention has been given to the development of tools that facilitate indexation and retrieval of learning materials (learning objects) in Learning Object Repositories (LORs) (Duval and Hodgins, 2003), such as (Ariadne, 2004), (Merlot, 2004) and (SMETE, 2004). However, little attention has been given to the usability evaluation of these developed tools (Duval and Hodgins, 2003). This evaluation will help researchers to determine to what extent indexation tools enable users to reach their goals effectively and efficiently (Jones, et al., 1998), (Marchionini, et al., 2000). In addition to this, it will help us to determine users’ satisfaction on the overall use of the evaluated tools. Furthermore, usability evaluation for indexation tools allows us to understand how real users interact with such tools. For example, we can obtain the average time needed to perform a particular task, the level of satisfaction, and the ease of use of the indexation tool. Hereby, findings and recommendations obtained from such usability evaluations help to improve the indexation of learning objects. Several usability studies have been made in the context of Internet browsing (Cockburn and McKenzie, 2000) (Nielsen, 2004), digital libraries (Jones, et al., 1998) and other computer applications. On the other hand, this issue has not been studied intensively yet in the context of LORs (Duval and Hodgins, 2003). In a previous work (Najjar, et al., 2003) we investigated the behavior of users who index relevant learning objects in the KPS, by analysing the usage logs. In this paper, we show results of the usability evaluation applied to two Ariadne indexation tools (SILO and Toledo, see section 2). Both tools are deployed to the Ariadne Knowledge Pool System (KPS). Results of this study show that indexation of learning objects may be influenced by different usability perspectives. Firstly, interface of indexation tools. Secondly, functionalities provided to facilitate the indexation process, such as automatic indexation. Thirdly, indexer domain knowledge about the introduced learning objects. In order to collect high quality metadata for learning objects, we provide the following recommendations. First, the user interface of indexation tools should be simple and adapted to the user, not to the metadata standard used by the LOR. Second, functionalities such as automatic indexation are helpful and it should use intelligent techniques to extract metadata of high quality. 281