Competency Assessment of Short Free Text Answers Adidah Lajis #1 , Normaziah Abdul Aziz *2 Malaysian Institute of Information Technology, Universiti Kuala Lumpur Kuala Lumpur, Malaysia 1 adidahl@unikl.edu.my * Kulliyah of Information and Communication Technology, International Islamic University Malaysia Gombak, Selangor, Malaysia 2 naa@iium.edu.my Abstract— The increased adoption of competency-based education has posed the need of an automated competency assessment. Most of the existing assisted assessment does not cater for competency assessment. The high percentage on the use of short free text answer as competency assessment shows that the need of the competency assisted assessment is urgent. This paper studies on the need and also review on existing assisted assessments focusing on short free text answer. A Node Link (NL) Scoring technique is proposed as an alternative automated solution to assess learners’ competency in short free text answers. Keyword: competency assessment; short free text answers; Node Link Scoring technique I. INTRODUCTION Assessment is an important part of learning. It is the process of making inferences about an individuals knowledge, skills, attitudes or other constructs using information from one or more methods such as tests, observations, interviews, projects or portfolios with reference to pre-defined criteria (learning expectations, measurement of learning outcomes) [1]. However, the increased numbers of students in Higher Education and the corresponding increase in time spent by staff on assessment has encouraged interest into how technology can assist in this area. Malaysia alone, the population of student in Higher Education has been increased gradually every year [2] For example, there is an increase of the intake of diploma student in public Higher Education Institute (HEI) form year 2009 for about 82,208 students and 94,026 students in year 2010 and also tremendous increase for the same level of study in private HEI that is from 198,760 students to 220,299 students. Assessment has always been an aspect of the use of information and communication technology in education. Thelwall (2000) considered computer based assessment as an educational tool in higher education [3]. Bull et al. (2004) and Mills et al. (2002) claimed that computer assisted assessment offers pedagogic and productivity benefit has raised the prospect of reductions in tutors’ assessment burden [4][5]. This advantages also has been agreed by Robinson et al. in their paper titled Mathematics Lecturers’ Practice and Perception of Computer Aided Assessment (2012) [6]. However, the usage of outcome based education in higher institution requires a well-blended of competencies and learning objectives to be achieved by the student. Assessment based on learning outcome has been broadly defined as what the learner should be able to achieve at the end of learning period and competencies will say how we can be certain they know it. Thus the assessment is not alone but the assessment of the competency also must be done. Yet, most current competence assessment methods are still exclusively based on paper-and-pencil formats [7]. This assessment medium puts both time and creativity constraints on the assessment process. With the rise of computer assisted education several attempts have been made to transfer existing competence assessment methods to the computer. By using new technology, assessments can be designed to be more authentic and challenging than paper-and-pencil-based assessments allowing for an active demonstration of knowledge, in contrast to talking or writing about it [8]. Apart from the fact that authentic and challenging assessments can provide a deeper insight into a learner’s competence, and may be able to measure higher order cognitive skills that cannot be easily tested with a standardized paper-and-pencil instrument [7]. Adidah and Normaziah (2012) have shown that Bloom’s competency test can be assessed by using NL Scoring technique [9]. The assessment method used in the study is on short free text answers with a length of 3 to 150 words. The competencies assessed were knowledge, understanding, analysis and evaluation. The experiment results showed that NL Scoring only can be used to assess knowledge, understanding and evaluation. The test data were collected from four different domains namely natural science, computer science, medical and engineering domain at school and tertiary level. This paper intends to further investigate the present landscape of competency assisted assessment through short free text answers and compare to the proposed NL Scoring technique. A. Assessment Methods Mansell (2002) stated that the type of knowledge and skills to be assessed can affect the choice of assessment methods needed [10]. And according to Bloom (1956), the length of the answer does not correlate with the levels of Bloom’s taxonomy [11]. Many academics examinations make heavy use of questions that require students to write one or two sentences.