Methodology for evaluating a novel education technology: a case study of handheld video games in Chile q Jesse L. Margolis a , Miguel Nussbaum b, * , Patricio Rodriguez b , Ricardo Rosas c a Educational Consultant, 118 Mason Terrace, Brookline, MA, 02446, USA b School of Engineering, Pontificia Universidad Cato ´ lica de Chile, Santiago, Chile c School of Psychology, Pontificia Universidad Cato ´ lica de Chile, Santiago, Chile Received 14 April 2004; accepted 1 July 2004 Abstract Many school systems, in both the developed and developing world, are implementing educational tech- nology to assist in student learning. However, there is no clear consensus on how to evaluate these new technologies. This paper proposes a comprehensive methodology for estimating the value of a new educa- tional technology in three steps: benefit analysis, through the administration of a well-designed experiment; cost analysis, which incorporates costs to weigh against the benefits; and feasibility analysis, which intro- duces real-world concerns that may affect the ability to actually implement the technology. To illustrate the methodology, a case study from Chile is used where portable educational video games were introduced into first and second grade classrooms with the aim of improving learning in mathematics and language. This paper demonstrates the importance of all three steps in the evaluation process and provides a framework for future analyses. Ó 2004 Elsevier Ltd. All rights reserved. Keywords: Evaluation methodologies; Evaluation of CAL systems; Elementary education; Cost–effectiveness analysis; Feasibility analysis 0360-1315/$ - see front matter Ó 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.compedu.2004.07.007 q This work was partially funded by FONDEF of CONICYT. * Corresponding author. Present address: Dpto. de Ciencias de la Computacion, Escuela de Ingenieria, Casilla 306, Santigo 22, Chile. Fax: +56 2 552 4054. E-mail address: mn@ing.puc.cl (M. Nussbaum). www.elsevier.com/locate/compedu Computers & Education 46 (2006) 174–191