Karoulis, A., Stamelos, I., & Angelis, L. (2008). Experimental Evaluation of an Instructional Supporting Tool in Distance Learning. Educational Technology & Society, 11(3), 67–81. 67 ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. Experimental Evaluation of an Instructional Supporting Tool in Distance Learning Athanasis Karoulis, Ioannis Stamelos and Lefteris Angelis Dept. of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece // Tel. 0030/2310/99.19.10 // Fax. 0030/2310/99.84.19 // athanasis@karoulis.gr // stamelos@csd.auth.gr // lef@csd.auth.gr ABSTRACT This paper describes a controlled experiment concerning the use of a learning aid during an open and distance learning (ODL) course. The core issue of investigation is whether this instructional aid can support, guide, and scaffold the distant student in his/her study. For this purpose, a controlled experiment was conducted with the participation of 191 undergraduate students at the department of informatics at a university in Greece. The considered domain was two lessons concerning human–computer interaction (HCI), the first in usability engineering and the second in interface evaluation methodologies. A test session was also conducted to collect data on the assessment of the effectiveness of the proposed tool. Descriptive statistics as well as a variety of statistical methods have been applied to the collected data in order to test the research hypotheses. The results have shown a statistically significant difference in performance for the student group that used the tool. In order to validate these results, a validity evaluation was also considered and presented. Finally, concerns about the application of the tool in a broader context and further research on the area are also presented. Keywords Lesson sheets, Open and distance learning, Controlled experiment Introduction Theoretical framework of the study The constructivist model of learning assumes that knowledge is not transferred but is created by the learner, with the instructor as facilitator (Belanger & Jordan, 2000). Most recent approaches adopt the student-centered approach, which is clear and understandable (Lowman, 1981), is responsive to the ways in which students learn and communicate (Kolb, 1984), acknowledges students’ interests and motivations (Forsyth & McMillan, 1991), and focuses on the explicit needs of learners for meaningful and timely feedback (van Houten, 1980). All paradigms in this area suggest that ad-hoc prepared instructional material, either in open and distance learning (ODL) or in traditional approaches, has to follow specific guidelines in order to be characterized as sound (Georgiadou & Economides, 2000). Aims and targets, keywords, short chapters and paragraphs, simple language, explanation of difficult points, and exercises and activities are only some of the guidelines. In addition to these, new educational approaches and methodologies have evolved to integrate this material more smoothly into the instructional procedure. Furthermore, various learning aids have been proposed, such as study frameworks, time scheduling, or summaries, because learning issues are, after all, complex and highly case sensitive (Squires & Preece, 1996). So, there is a broad consensus that instructional tools aim to support the student in his/her work, to confront eventual learning difficulties, and to facilitate the students in personalizing the offered information. Existing research and motivation of this study In the described context, a new instructional tool, simply called the lesson sheet, has been proposed and thoroughly described in Karoulis, Demitriades, and Pombortsis (2004a). The investigation of the potency of this tool underwent three phases. The first one concerned a qualitative survey, which was applied in traditional classes and was published in the aforementioned study. The second evaluation was a controlled experiment concerning the application of the tool in both a traditional and a hybrid class, and is thoroughly described in Karoulis, Stamelos, Angelis, and Pombortsis (2004b). The work presented here concerns the third phase of the study, namely, to modify the proposed tool to make it suitable for ODL, to apply it in the framework of a pure distance-learning course, and to evaluate its potency and effectiveness in this context. So, a new controlled experiment has been organized and performed in order to elucidate this question.