Evaluating an Intelligent Diagnosis System of Historical Text Comprehension Grammatiki Tsaganou a , Maria Grigoriadou a , Theodora Cavoura b , Dimitra Koutra a a University of Athens, Department of Informatics and Telecommunications, GR-15784, Athens, Greece, e-mails: gram@di.uoa.gr, gregor@di.uoa.gr, mik@altec.gr b University of Thessaly, Dept. of Education, Argonafton & Filellinon strs, GR-38221, Volos, Greece, e-mail: theokav@pre.uth.gr Abstract This work aims to present and evaluate a Fuzzy-Case Based Reasoning Diagnosis system of Historical Text Comprehension (F-CBR-DHTC). The synergism of fuzzy logic and case based reasoning techniques handles the uncertainty in the acquisition of human expert’s knowledge regarding learner’s observable behavior and integrates the right balance between expert’s knowledge described in the form of fuzzy sets and previous experiences documented in the form of cases. The formative evaluation focused on the comparison of the system’s performance to the performance of human experts concerning the diagnosis accuracy. The system was also evaluated for its behavior when using two different historical texts. Empirical evaluation conducted with human experts and real students indicated the need for revision of the diagnosis model. The evaluation results are encouraging for the system’s educational impact on learners and for future work concerning an intelligent educational system for individualized learning. Keywords : Fuzzy Case based reasoning; Historical text comprehension; Diagnosis evaluation; Learner model. 1 Introduction In an Intelligent Tutoring System (ITS), learner diagnosis process imitates the human expert’s process of inferring the student’s internal characteristics from his observable behaviour (VanLehn, 1988). In the domain of comprehension of history, this computational diagnostic process imitates a human expert’s ability to estimate how the learners comprehend the historical text. An attempt towards this direction is our previous work concerning the Learner Model of learner’s cognitive profiles for Historical Text Comprehension (LMHTC) (Tsaganou et al, 2003). Such a learner model demands knowledge acquisition from a human expert, which means knowledge extraction regarding the student’s observable behaviour, its complete and accurate description and transfer to a knowledge base and sometimes to an inference engine. The main obstacle in this process is the uncertainty derived not only from the knowledge communication among the developer, the human expert and the system, but also from inaccuracy of the information captured and of approximation involved in all process steps (Richter, 2001). Case Based Reasoning (CBR) is claimed to be a paradigm that is more akin to the human way of solving complex diagnostic problems in domains like medicine or law. A human expert solves a diagnostic problem using rules derived from his previous experience-cases, whereas a novice requires complete and concrete rules. CBR integrates the right balance between hard to acquire expert knowledge and more easily acquired knowledge in the form of cases. So, for building of an ITS, CBR helps more easily than other methods to overcome problems of knowledge acquisition from the expert. CBR has been proposed for a variety of diagnostic applications (Kolodner, 1993), has been used in educational systems such as in CELIA, for modelling the memory and reasoning capabilities of a novice (Kolodner, 1993), in Engines for Education for case based coach (Schank et al, 1994), in Tutoring and Help Systems (Weber et al, 1998; Burke et al, 1994), in SYIM for distance education (Tsinakos et al, 2001), as part of the student modelling process in ITS systems (Shiri et al, 1998). For overcoming complex problems, like uncertainty in knowledge acquisition, developers recently build more hybrid case-based and knowledge-based systems than pure CBR systems. Fuzzy logic is designed to operate with linguistic expressions and express imprecision and subjectivity in human thinking. Fuzzy logic contributes to CBR in overcoming problems of managing the uncertainty and problems concerning case adaptation by improving performance of case retrieval (Jeng et al, 1995). In research community the interests of fuzzy logic and CBR for diagnosis recently intersect (Dubois et al, 1997; Richter et al, 1999; Hansen, 2000). Fuzzy logic is widely used in student modelling where variables are continuous, imprecise, or ambiguous. Fuzzy-based techniques have been used in educational systems for flexible case-based querying (Calmes et al, 2002). Often, we think of evaluation in terms of how well AI systems, perform. Evaluation of a student model tries to answer a question that is central to cognitive science, AI and education (Littman et al, 1988): What is the relationship between the architecture of the student model and its behavior? The evaluation methods can be used to construct an accurate picture of this relationship. The issue of noise, for example, must be addressed by any realistic system performing pedagogical diagnosis (Wenger, 1994). Three sources of noise are considered: First, noise in the data, which means non-consistency of student’s behavior over the time. Second, noise in the diagnostic process, which means inherent ambiguities in the diagnostic process. Third, noise in the model of communicative knowledge, which means that