Logging and Visualization of Learner Behaviour in Web-Based E- Testing Gennaro Costagliola, Vittorio Fuccella, Massimiliano Giordano, Giuseppe Polese Dipartimento di Matematica e Informatica – Università degli Studi di Salerno, Fisciano(SA),Italy {gcostagliola, vfuccella, mgiordano, gpolese}@unisa.it ABSTRACT In this paper we present a system for the logging and the visualization of the learners’ behaviour during the execution of structured tests based on Multiple Choice question type. The system is composed of two main components: a logging framework which, instantiated in e-testing systems, produces an XML formatted log of learners’ interactions with the system during tests and a stand-alone application which visualizes charts containing a chronological review of the tests. By analyzing the charts obtained through an experiment led in our department, we have defined several typical strategies used by the learners to execute tests. The effectiveness of these strategies has been inferred by correlating the strategies with the obtained scores. Further useful applications of our system allow us to detect correlations among questions and cheating attempts by the learners. Keywords e-testing, computer assisted assessment, CAA, on-line testing, test, learner interactions, multiple choice, learner behaviour, information visualization. 1. INTRODUCTION E-testing systems are more and more widely adopted in academic environments combined with other assessment means. Through these systems, tests composed of several question types can be presented to the learners in order to assess their knowledge. Multiple Choice question type is extremely popular, since, among other advantages, a large number of its outcomes can be easily corrected automatically. Among the disadvantages of structured tests, a low acceptance of the exam type by the learners is rather frequently noticed: many learners are afraid of not being able to best express their capacity, due to the characteristic of multiple choice questions of being closed. Even many examiners wonder if structured tests are effective in assessing the learners’ knowledge and if some learners are conditioned more by the question type than by the actual question difficulty. In order to teach to the learners how to better perform on structured tests, besides giving the usual advice of enhancing their knowledge by studying harder, and of devoting some time on practising with structured tests, several experiments aimed at tracking learners’ behaviour during structured exam tests based on multiple choice question items have been carried out in the past. Some of them [2, 3] have regarded the learner’s habit to verify the given answers and to change them with other, more plausible, options. In the experiments, right-to- wrong and wrong-to-right changes have been recorded and their number has been correlated to learners’ final mark on the test. Similar experiments have been carried out by correlating the time to complete the test with the final mark. [10, 15]. An interesting experiment [13] focuses on the strategies used by several learners with different abilities (“A”, “C” and “F” students) to complete the tests. In particular, the frequency of several habits has been recorded and then correlated to learners’ final mark on the test, such as: the habit of giving a rash answer to the question as soon as a plausible option is detected, often neglecting to consider further options; the habit of initially skipping the questions whose answer is more uncertain, in order to evaluate them subsequently; etc. The above experiments only consider some aspects of the execution of a test. We have not found in literature any study which considers the learner behaviour as a whole. In this paper we use information visualization in order to define several typical strategies used by the learners to execute tests. When exploring data, humans look for structures, patterns and relationships between data elements. Such analysis is easier if the data are presented in a graphical form in a visualization. Information visualization is defined as the use of interactive visual representation of abstract data to amplify cognition [17]. In the past, information visualization has successfully used in an e-learning application to measure the participation of the learners to on-line activities [11]. Our technique consists of logging all the interactions of the learners with the e-testing system interface. To elaborate, we capture the occurrence of question browsing and answering events by the learners. These data are used to visualize charts containing a chronological review of the tests. Besides identifying the most employed strategies, we try to determine their effectiveness by correlating their use with the scores obtained on the tests. Another useful application of our system allows us to