Lessons Learned in Evaluating e-Learning Products: A Beginning Evaluator Checklist Krista Galyen, University of Missouri, kdgrz9@mizzou.edu Camille Dickson-Deane, University of Missouri, cdickson-deane@mizzou.edu Abstract The evaluation of e-learning products requires a level of preparation, forethought, and skill to which most beginning evaluators are not accustomed. Like any other project management process, planning for an evaluation takes more than just technique and available skills; it requires understanding the client, their intentions, their political environment, as well as learning how to modify techniques and still deliver the required report. This presentation discusses lessons learned from an effectiveness evaluation project in order to better inform and prepare beginning evaluators of these aspects. Aspects such as proximity of team members, communication, political environment, insufficient stakeholder management, and the availability of the specified participants will be described. In addition, a beginning evaluator checklist will be shown which assists the evaluator in thinking about potential variables and processes. INTRODUCTION The process of conducting an evaluation requires lots of planning, management and controlling of components. These components consist of human and technical resources as well as procedures which are used to provide a framework for conducting the evaluation. The different components need to be organized into some structure in order for the evaluation to be successful. New evaluators would normally follow some existing procedure in an effort to copy success, but the components that existed for the procedure may not be the same or even similar to the [new] evaluators project. E-learning evaluation is even more complicated in that the e-learning product may have different characteristics or even, be based on different definitions of e-learning. Allen & Seaman (2007) defines four types of courses where at one extreme traditional courses have no online content that is delivered and the other extreme has no face to face instruction. E-learning can fall into either category of face to face or no face to face instruction as per its definition of being learning provided through some technological media. The media in this instance can span anywhere from being a CD-Rom to being strictly provided via the Internet (Nicols, 2008). The medium of delivery is as important to the success of the evaluation as its evaluators; each piece of the puzzle must be carefully handled in order to produce a reasonable evaluation report. THE PROJECT The product OneTree Learning (OTL) supports students learning English as a second language (ESL). The company provides not only English learning services, but also acts as a guide to help students become more prepared for academic classrooms. Their product, Language Arts for ESOL, is a CALP- (Cognitive Academic Language Proficiency) based program that teaches a core curriculum of language arts to English language learners. OTL's Language Arts for ESOL can be used either as a dedicated ESL curriculum, or as a supplementary program to assist ESL students within existing mainstream classes. No specialized ESL training is