Validity and Reliability of Tests in Web-PSI Courses Jana Jacková University of Žilina, Faculty of Informatics and Management Science Bystrická cesta 21, 034 01 Ružomberok, Slovakia, Tel.: + 421 44 432 17 10, Fax: + 421 44 432 17 46 jackovaj@fria.fri.utc.sk Abstract “Test reliability and validity are the two most important features of a test. They indicate the quality and usefulness of the test”[1]. The article deals with problems of students’ knowledge evaluation in higher education e- courses. Stages of testing process needed for preparation a computer science course with mastery learning philosophy are described. The procedure is “content- independent”. This process of test construction is based on didactic principles. It is not very known among teachers - technicians (with master degree “Ing”) without pedagogy and psychology background. To clear the test assessment procedure the author focuses on steps needed before and after testing. Worldwide usage of PSI [2] (Personalized System of Instruction / Keller plan) courses shows higher efficiency than traditional ones. “Student feedback from the PSI courses was almost 100% positive -- almost everyone -- students, proctors, and I -- enjoyed the experience” [3]. Keywords: teacher qualification, test construction, specific aims, reliability, validity, PSI, mastery, criterion-referenced assessment 1. Introduction Secondary school teachers who are graduates from technical faculties have to pass specialized teacher training programme. On the other hand, university teachers are not required to have teacher qualification. Tests are common educational process element at almost all kind of schools and courses. The question is: Are the tests we do correct? And if, what do they really measure? Both secondary and university technical subjects teachers can obtain teacher qualification passing specialized teacher study at the Faculty of Material Sciences and Technology, Slovak Technical University where various teacher training subjects and also the test construction are taught. Another example is from Comenius University, Faculty of Mathematics, Physics and Informatics, branch of study Secondary and Post - Secondary Teaching, that offers a School Tests course [4]. Syllabus of these courses covers fundamentals of educational measurement theory, didactic principles of test construction, test usage in practice, current trends etc. Nowadays I have been preparing a basic skills programming web-PSI course as an alternative to a traditional one. PSI is based on mastery learning philosophy and criterion-referenced assessment. Mastery tests “assess whether or not the person can attain a pre- specified mastery level of performance”[5]. I have applied the school test construction theory from [6] [7] [8] [9] into the procedure for constructing PSI course tests. It does not matter if you do a traditional course or a web one. With or without technology, the procedure would have to be done step by step. Surely, technology, when used correctly, can multiply the effects both the testing procedure and a course. “For example, computerized quizzes that give students immediate feedback on their content knowledge and refer them to their textbook or other source of information to restudy amplify two important student characteristics. First, students learn the information better and come to appreciate the value of feedback and review. Second, students learn something about the effectiveness of their learning strategies and how to improve them” [10]. To receive reliable test results about real students’ knowledge is the aim of all teachers. To design reliable and valid tests is serious work, particularly if students’ results are important for very important decisions, e.g. passing/failing the final course examination, final course evaluation. 2. Validity and Reliability A good test is reliable and valid. Reliability is not enough; a test must also be valid for its use. If test scores are to be used to make accurate inferences about an examinee's ability, they must be both reliable and valid. Reliability is a prerequisite for validity and refers to the ability of a test to measure a particular trait or skill consistently. However, tests can be highly reliable and still not be valid for a particular purpose [11].