Engaging Museum Visitors with Gamification of Body and Facial Expressions Maria Giovanna Donadio * maria.donadio@stud.unif.it Università di Firenze Italy Filippo Principi * flippo.principi@unif.it Università di Firenze - MICC Italy Andrea Ferracani andrea.ferracani@unif.it Università di Firenze - MICC Italy Marco Bertini marco.bertini@unif.it Università di Firenze - MICC Italy Alberto Del Bimbo alberto.delbimbo@unif.it Università di Firenze - MICC Italy ABSTRACT In this demo we present two applications designed for the cultural heritage domain that exploit gamifcation techniques in order to improve fruition and learning of museum artworks. The two appli- cations encourage users to replicate the poses and facial expressions of characters from paintings or statues, to help museum visitors make connections with works of art. Both applications challenge the user to fulfll a task in a funny way and provide the user with a visual report of the his/her experience that can be shared on social media, improving the engagement of the museums, and providing information on the artworks replicated in the challenge. CCS CONCEPTS • Software and its engineering → Interactive games; • Com- puting methodologies → Computer vision. KEYWORDS Cultural heritage, gamifcation, face pose, body pose ACM Reference Format: Maria Giovanna Donadio * , Filippo Principi, Andrea Ferracani, Marco Bertini, and Alberto Del Bimbo. 2022. Engaging Museum Visitors with Gamifcation of Body and Facial Expressions. In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3503161.3547744 1 INTRODUCTION Gamifcation is the process of exploiting strategies and game dy- namics into scenarios that are not a game [10]. It has already been proved to be useful to enhance skills and competences in a variety of domains such as marketing, industry training and entertainment. Certainly also cultural heritage can beneft from a gamifcation approach which represents an opportunity to engage visitors to * Main and equal contributors of this work. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). MM ’22, October 10–14, 2022, Lisboa, Portugal © 2022 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9203-7/22/10. https://doi.org/10.1145/3503161.3547744 museums content through the design of more entertaining, social and challenging digital learning scenarios [2, 4, 6, 8], to help muse- ums to move from the traditional łlook and do not touch" toward a łplay and interact" approach. In fact, it has been observed that the availability of tools like gamifed e-guides to visitors contributes to the sustainability of museums [1]. 2 THE DEMO SYSTEMS The goal of the two applications is to challenge the user to analyze and replicate artworks with their own body and face, obtaining i) information on the artworks that are replicated and ii) personalized artwork representations that can be shared on social networks. 2.1 Strike a Pose Strike-a-pose (Fig. 1) is a web application which performs analysis and evaluation of human poses compared to poses present in famous paintings or statues. Strike-a-Pose can be made available on the visitors’ phone, following the łBring Your Own Device" (BYOD) approach that has become more common in museums since the COVID-19 pandemic [1]; the system can be used also in a dedicated environment that uses a fxed station equipped with a large screen and a camera. The application exploits a gamifcation paradigm with the didac- tic purpose of getting users interested in works of art using fun. Once registered, the user is challenged to reproduce in sequence the poses of some artworks from the museum’s collections. The skeleton of both the artwork and the visitor can be displayed on the screen in order to facilitate the user in matching the various points and segments. Matching the poses provides the descriptions of each artwork. The poses to be matched are organized in sets of challenges, e.g. challenges to replicate poses using the whole body, using only the torso (e.g. to allow also wheelchair users to interact), or any other type of challenge that is considered interesting by the museum curators (e.g. based on thematic collections). Once all the poses have been matched, the application allows the user to generate a video that can be saved for any social sharing. The video shows the user matching process and the overall interactive experience lived at the museum. The application has been developed in JavaScript on the client side and in Python on the server side. Pose detection on the human bodies is achieved using TensorfowJS 1 detection API exploiting 1 https://www.tensorfow.org/js 7000