AUTOMATICALLY ASSESSING DIAGRAMS Pete Thomas, Neil Smith, Kevin Waugh Computing Department The Open University, Milton Keynes, UK, MK7 6AA ABSTRACT Diagrams are ubiquitous; they are used to communicate ideas and are often used to model aspects of the real world. We teach with them and expect students to develop modelling skills using them. In conventional, paper-based assessment, we do not think twice about asking students to draw free-hand diagrams to express both their knowledge and drawing skills. In e-Assessment, the situation is very different. There is very little support for assessing diagramming skills and where students can draw free-form diagrams (with a drawing tool) the assessment has to be human. In this paper we discuss an approach to the automatic marking of a certain class of diagrams (almost graph-based) illustrating how we have incorporated domain knowledge, pedagogy and marking schemes into the process. In particular, the paper examines the general strategy for automatic marking based on meaningful units and how marking schemes are constructed. The paper concludes with the results of applying our method to a corpus of student diagrams and illustrates how the consistency afforded by automatic marking can overcome some of the deficiencies in human marking. KEYWORDS Automatic marking, diagrams, e-assessment. 1. INTRODUCTION In this paper, we examine the area of e-assessment concerned with automatically grading student answers to questions posed by the system. Most, if not all, virtual learning environments (VLEs) or course management systems (CMSs) include quizzes for both formative and summative purposes. Such quizzes normally confine themselves to what is often referred to as objective testing in which the answer to a question is either unique or comes from a small set of answers; multiple choice questions are one type of question that falls into this category. Automatically marking such questions is straightforward and students can be supplied with answers and feedback almost instantaneously. However, while such quizzes are being used more and more, some see them as deficient because they do not assess higher order learning and are not representative of the more open-ended types of question typically found in traditional assessments. However, there have been noteworthy successes in using objective questions to test higher order learning [MacKenzie, 2008] but it is usually the case that the construction of such quizzes is very labour intensive. There have been successful attempts at the automatic assessment of essays [Burstein et al., 2003] but the approach is often based on statistical methods and relies on some human marking to start the process. Current e-assessment systems are deficient in the support they offer for free-form diagrams. By free-form we mean that students are relatively unconstrained in what they can draw. Contrast this with the limited use of diagrams in current online assessment systems where it might be possible to draw simple lines or curves on a preformed graph or drag and drop shapes on to ‘hot spots’ on a picture. Free-form diagrams do not have a pre-defined structure; students are able, with the aid of a tool, to draw diagrams using a palette of drawing primitives and append free-form text to elements on the diagram (note that we are not discussing hand-drawn diagrams that must be transformed into electronic format before being processed automatically; this is an entirely different field of investigation). Accurate automatic marking of such diagrams requires an interpretation of both the structure of the diagram and its textual components (though the latter tend to be quite brief in practice). Our initial aim in marking free-form diagrams was to see what could be achieved with a minimum of effort. If a relatively unsophisticated approach were found to be ‘reasonably’ effective we could then seek improvements by extending the approach. Just as importantly, we wanted to base our approach on a theory IADIS International Conference e-Learning 2009 263