Impact of window size on the generalizability of collaboration quality estimation models developed using Multimodal Learning Analytics Pankaj Chejara pankajch@tlu.ee Tallinn University Tallinn, Estonia Luis P. Prieto luisp@tlu.ee Tallinn University Tallinn, Estonia María Jesús Rodríguez-Triana mjrt@tlu.ee Tallinn University Tallinn, Estonia Adolfo Ruiz-Calleja adolfo@tlu.ee Tallinn University Tallinn, Estonia Mohammad Khalil mohammad.khalil@uib.no University of Bergen Bergen, Norway ABSTRACT Multimodal Learning Analytics (MMLA) has been applied to col- laborative learning, often to estimate collaboration quality with the use of multimodal data, which often have uneven time scales. The diference in time scales is usually handled by dividing and aggregating data using a fxed-size time window. So far, the cur- rent MMLA research lacks a systematic exploration of whether and how much window size afects the generalizability of collaboration quality estimation models. In this paper, we investigate the impact of diferent window sizes (e.g., 30 seconds, 60s, 90s, 120s, 180s, 240s) on the generalizability of classifcation models for collaboration quality and its underlying dimensions (e.g., argumentation). Our results from an MMLA study involving the use of audio and log data showed that a 60 seconds window size enabled the development of more generalizable models for collaboration quality (AUC 61%) and argumentation (AUC 64%). In contrast, for modeling dimensions focusing on coordination, interpersonal relationship, and joint in- formation processing, a window size of 180 seconds led to better performance in terms of across-context generalizability (on average from 56% AUC to 63% AUC). These fndings have implications for the eventual application of MMLA in authentic practice. CCS CONCEPTS · Human-centered computing → Empirical studies in collab- orative and social computing;· Computing methodologies → Machine learning algorithms. KEYWORDS MultiModal Learning Analytics, Machine Learning, Collaboration Quality, Generalizability, Temporal Window Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from permissions@acm.org. LAK 2023, March 13ś17, 2023, Arlington, TX, USA © 2023 Association for Computing Machinery. ACM ISBN 978-1-4503-9865-7/23/03. . . $15.00 https://doi.org/10.1145/3576050.3576143 ACM Reference Format: Pankaj Chejara, Luis P. Prieto, María Jesús Rodríguez-Triana, Adolfo Ruiz- Calleja, and Mohammad Khalil. 2023. Impact of window size on the gener- alizability of collaboration quality estimation models developed using Mul- timodal Learning Analytics. In LAK23: 13th International Learning Analytics and Knowledge Conference (LAK 2023), March 13ś17, 2023, Arlington, TX, USA. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3576050.3576143 1 INTRODUCTION Collaboration is an essential skill in the 21st Century [6]. To de- velop this skill among students, collaborative activities are often combined with other pedagogical approaches (e.g., project-based learning [23]) in teaching practices. In such practices, teachers are by default expected to orchestrate and monitor group activities which are extremely difcult [4]. In this direction, the automa- tion of collaboration estimation holds the potential for supporting teachers with the development of monitoring tools [3ś5, 9, 17]. There has been a growing interest in automated estimation of collaboration [3, 17]. For example, a tool that identifes low collab- oration quality can help the teacher identify the group that needs support in the classroom. The development of such tools demands capturing data from the physical space in addition to the digital space which traditional (log-based) Learning Analytics (LA) ful- flls only partially. To address this limitation of capturing physical space, researchers have started employing other data sources (e.g., audio [25], video [23]) in addition to logs, to capture collabora- tion more holistically. This research feld that involves the use of multiple data sources is known as MultiModal Learning Analytics (MMLA) [2]. Earlier MMLA research works have provided preliminary evi- dence on the feasibility of automating the estimation of collabora- tion quality (or other aspects of collaboration) using multimodal data (audio and logs) in face-to-face (FtoF) settings [11, 14]. This research has been advanced by MMLA researchers, exploring a variety of modeling techniques (e.g., Random forest [25], Adaboost [22]) with diferent types of data (e.g., audio [25], eye-gaze [18], video [23]). Furthermore, MMLA work from Olsen et al.[15] on collaboration detection reported performance gains with the use of multimodal data models over models built with unimodal data. These research works’ fndings suggest the use of MMLA in au- tomating collaboration detection for FtoF settings. 559