Evaluation of a Constraint-Based Error Diagnosis System for Logic Programming Nguyen-Thinh Le Department of Informatics, University of Hamburg, Germany le@informatik.uni-hamburg.de Abstract. We applied the constraint-based approach to develop a web-based diagnosis system for Prolog. In this paper, we introduce the results of our formative evaluation which reflects the current effectiveness of our system. We gathered 261 log files which are created by 99 users contained records of interactions with the tutoring system. In addition, we present the common problems of Prolog novice programmers, strengths and limitations of our system. Keywords: Formative evaluation, Constraint-based error diagnosis, Logic programming. Introduction Error diagnosis is one of the essential components of Intelligent Tutoring Systems because understanding the current difficulties of a student is indispensable for providing him with guiding help. This information is necessary irrespective of how this feedback is going to be presented. Thus, error diagnosis and feedback generation can and should be separated to a certain degree. Numerous approaches to error diagnosis in programming languages have been proposed. Many of them turned out difficult to apply, do not provide enough diagnostic information or are restricted to a particular pedagogic strategy. The constraint-based approach was introduced by Ohlsson in [1] and has been shown to be successful in building Intelligent Tutoring Systems (ITS) in the domains of SQL [2]. Since a logic programming language can be understood as the relational calculus enriched with recursion and function symbols, the question arises naturally, whether constraint- based diagnosis techniques can also be used in the more general case of a declarative programming language. To investigate this issue, we developed a diagnosis component and integrated it in a web-based tutoring system [3]. This system is intended to help first year university students to overcome difficulties while doing their homework assignments in programming with Prolog. Students are provided with a database of exercises. They can select from this database the exercises, for which they are in need of help. The system attempts to diagnose their solution and returns feedback which indicates possibilities for remedy. Thus, students can improve their solutions successively. A preliminary evaluation has been carried out at the University of Hamburg. System use was not mandatory, but recommended in case the student wanted additional help. Deliberately, we did not use an authentication mechanism. Hence, students could log in under different names. No reliable user identification was possible and even not desired to respect the privacy of the students and encourage group work. In the next section, we introduce briefly our diagnosis system. In the second section, we demonstrate the formative evaluation method which we applied to optimise our system. Thereafter, we illustrate our evaluation results obtained by analysing the log data. In the last section, we discuss the weakness and strengths of our evaluation method. Lastly, we resume our suggestions for improving the system.