Designing an Educational Game Evaluation Framework Based on Game Mechanics Satrio Adi Rukmono School of Electrical Engineering and Informatics, Institut Teknologi Bandung Bandung, Indonesia Email: sar@itb.ac.id Fais Zharfan Azif School of Electrical Engineering and Informatics, Institut Teknologi Bandung Bandung, Indonesia Email: 23518037@std.stei.itb.ac.id Muhammad Zuhri Catur Candra School of Electrical Engineering and Informatics, Institut Teknologi Bandung Bandung, Indonesia Email: m.candra@itb.ac.id Abstract—Children in everyday life are increasingly using educational games. However, the quality of each of the many educational games available varies. Some evaluation frameworks exist, but most are prone to the evaluator’s subjectivity, which cannot be compared objectively. This study aims to formulate a framework that evaluates the quality of educational games objectively based on the game mechanics used. The framework is built upon Bloom’s taxonomy as the basis to ascertain the academic side and MDA (Mechanics-Dynamics-Aesthetics) Framework to distinguish the game side. Then, it assesses each educational mechanic based on a standard in the evaluation framework to obtain an accurate, quantifiable score as a measure. Validation of the framework involves using the framework to evaluate existing educational games and comparing the results with expert reviews. With this framework, an educational game quality can be measured objectively and quantitatively based on the technical and fundamental elements that exist in each game. Index Terms—educational game, Bloom taxonomy, game eval- uation framework, game mechanics I. I NTRODUCTION In recent years, technological developments have experi- enced rapid growth in all fields. Of the many areas, one in high demand is the entertainment sector, especially games. Because more and more games are being played, people start using games as a means for education. However, while tons of educational games have emerged, only a few are of good quality [1]. To find effective games, they need to be evaluated before being used. Fortunately, several frameworks are available for evaluating educational games. The Game-Based Learning Evaluation Framework [2] and Triadic Game Evaluation [3] offer a way to assess an educational game from a subjective perspective. Therefore, the frameworks are still subjected heavily to the evaluator’s opinion, making it difficult to compare with another evaluator’s evaluation objectively. For that, a more objective evaluation standard is needed to assess the various kinds of educational games in circulation so that learners and teachers can determine which games are effective. This study aims to build a game evaluation framework that objectively assesses an educational game to be comparable even among different evaluators. There are four sections in this paper. Related works are discussed in Section II. Section III presents the framework, while Section IV describes the evaluation process. Section V concludes the discussion and offers suggestions for future works. II. RELATED WORKS A. Game-Based Learning Evaluation Framework Games-Based Learning (GBL) Evaluation Framework [2] aims to provide a general framework for evaluating game- based learning, including educational games. This framework is based on a literature study containing empirical evaluation and the methods and metrics used. This framework provides a way to evaluate specific aspects of an educational game, including the game concept. Connolly et al. formulated an effective Game-Based Learning with six components plus one optional component, Collaboration. The six components are Learner Performance, Motivation, Perceptions, Attitudes, Preferences, and Environment. This framework relies on an evaluator’s expert opinion that may differ from person to person. The framework we propose in this paper aims to cover the subjectivity problem by making objective judgments so that assessments between games can be compared even when performed by different evaluators. B. Triadic Game Evaluation Triadic Game Evaluation (TGE) is a framework for evalu- ating serious games based on the Triadic Game Design (TGD) framework [3]. TGE adopts the philosophy from TGD, where a game must be designed by considering three important aspects; Reality, Meaning, and Play. Reality considers the real-world aspects modelled in the game. Meaning is the game’s goal or purpose, while Play is the experience the player feels when playing the game. These three components are considered essential to make an excellent serious game. TGE intends to evaluate the game based on these three aspects by examining the criteria that make up those aspects. These criteria are flexibility, validity, and fidelity for aspects of Reality; motivation, relevance, and transfer for aspects of Meaning; and engagement, immersion, and fun for Play. This framework has shortcomings, mainly in the interpretation of its aspects by evaluators. Different evaluators may have a di- verse understanding of an aspect, underlining the framework’s reliance on subjective analysis. The proposed framework in 2021 9th International Conference on Information and Communication Technology (ICoICT) 978-1-6654-0446-4/2021/$31.00 ©2021 97