Educational System Evaluation and Critical Analysis of Results Sarinder Kaur Kashmir Singh Raja Jamilah Raja Yusof University of Malaya University of Malaya sarinder@um.edu.my rjry@um.edu.my Abstract Most designers feel that educational systems are like any other computer systems, therefore should be treated equally. Due to this, educational systems suffer from great depression because they do not accommodate well for the target students. In this paper, the authors have come up with a checklist for evaluation, which considers not only the usability heuristics but also the socio-constructivist issues in learning as well as the cultural issues. The authors have used this method for testing six (6) educational systems and the results are analyzed critically. 1. Introduction The checklist recommended in this paper, which contains the usability heuristics as well as the socio- constructivist issues, has been used to evaluate six (6) educational systems. The evaluation checklist contains categories such as user characteristics, goal/task characteristics, social acceptability, practical acceptability, appropriateness of lesson structure, method of delivery, functionality, communication, display layout, program rigidity, navigational issues and lesson structure. 2. Analysis Results of Survey With the evaluation results, the authors have done some critical analysis to show the relationship between some of the issues in checklist with students’ social and practical acceptability of the systems. High level of user’s computer skills, experience and English proficiency results to a high level of social acceptability. Appropriateness and rigidity of educational systems also results to a higher level of user acceptance of the educational systems. The more the users prefer the educational systems to printed materials; the more the educational systems are socially acceptable to them. Users who feel that an educational system is fun to use also show a high level of social acceptance of the courseware. The results also prove that an educational system written in the Malaysian national language is more accepted in the Malaysian learning environment Educational systems with a high level of practical acceptability have fewer communication, functional and navigational problems. They have appropriate display layout as well. According to the trend in the analysis, an educational system, which is practical to use, should have less communication, functionality and navigational errors and it should contain a good display layout. The educational systems with an appropriate lesson structure seem to be less teacher- directed. This shows that, a properly structured lesson in an educational system will help the users to explore on their own rather than depending on the teacher to teach them. This way, the systems can be more learner control. 3. Conclusion The analysis in this paper shows that the acceptance of an educational system depends on all the issues presented in this paper. Thus, when designing an educational system for target users, it is important to look at all the issues discussed above carefully. 4. References [1] J. Nielsen, R.L. Mack (Eds.), Usability Inspection Methods, John Wiley, New York, 1994, p. 30. [2]D.C. Phillips, The good, the bad, the ugly: The many faces of constructivism, Educational Researcher 24 (7)(1995) 5–12. [3] E. Soloway, S.L. Jackson, J. Klein, C. Quintana, J. Reed, J. Spitulnik, S.J. Stratford, S. Studer, J. Eng, N. Scala, Learning theory in practice: case studies in learner-centred design, in: R. Mack, L. Marks, D. Collins (Eds.), CHI’96 Proceedings, ACM, 1996, pp. 189–196. [4]D.H. Jonassen, Thinking technology: Toward a constructivist design model, Educational Technology 34 (3)(1994) 34–37. [5]Cem Kaner et al. 1993. Testing Computer Software. New York, Van Nostrand Reinhold. [6] S.K. Card, T.P. Moran, A. Newell, The Psychology of Human Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, NJ, 1983. [7] Jonassen, D. April, 1994. Thinking technology. Educational Technology, 34(4), 34-37. [9] K., Sarinder, R. Jamilah. 2002. Guidelines for Designing Usable Educational System, Proceedings of the IEEE International Conference on Advanced Learning Technologies, Media and the Culture of Learning, Kazan, Russia, 9-12 August, pp 402-407. Proceedings of the The 3rd IEEE International Conference on Advanced Learning Technologies (ICALT’03) 0-7695-1967-9/03 $17.00 © 2003 IEEE