Evaluating a General-Purpose Adaptive Hypertext System Chris Staff University of Malta, Department of Computer Science and AI, Msida MSD 06, Malta Abstract. We describe the evaluation process of HyperContext, a frame- work for general-purpose adaptive and adaptable hypertext. In particu- lar, we are interested in users’ short-term, transient, interests. We cannot make any prior assumptions about a user’s interest or goal, as we do not have any prior knowledge of the user. We conducted evaluations on two aspects of HyperContext. One evaluation was completely automated, and the other involved participants. However, the availability of a test collection with value judgements would be a considerable asset for the independent and automated evaluation of adaptive hypertext systems in terms of cost, reliability of results, and repeatability of experiments. 1 Introduction HyperContext is a framework for adaptive and adaptable hypertext [8], [9]. We are currently using the HyperContext framework as part of the University of Malta’s contribution to the Reasoning on the Web with Rules and Semantics (REWERSE) FP6 Network of Excellence 1 . HyperContext focuses on building and maintaining a short-term user model to provide adaptive navigation support. We begin a user session with an empty user model and we add to the model as a user navigates through hyperspace and interacts with the system. A proof of concept HyperContext application has been evaluated. We had devised an evaluation strategy for HyperContext in 1999. However, due to a number of reasons, including hardware failure, the original evaluation strategy was abandoned. We eventually settled on a partially automated approach that did involve some participants, but which was less reliant on human participants. We are satisfied that the results of the automated evaluation show that the adaptive features of HyperContext can guide users to relevant information. We feel that our automated evaluation benefited from the fact that HyperContext assumes an initially empty user model that is then populated during short inter- actions with the system. Part of the evaluation involved showing users a series of documents (representing a short path through hyperspace) followed by two other documents in a random sequence. One of the two documents was recommended 1 staff.um.edu.mt/mmon1/research/REWERSE/