Comparing “in the Wild” Studies With Laboratory Experiments: A Case of Educational Interactive Tabletops Bertrand Schneider, Stanford University, schneibe@stanford.edu Consuelo Valdes, Wellesley College, cvaldes@wellesley.edu Kelsey Temple, Wellesley College, ktempel@firstclass.wellesley.edu Chia Shen, Harvard University, chia_shen@harvard.edu Orit Shaer, Wellesley College, oshaer@wellesley.edu Abstract: In this paper, we describe our attempt at implementing an interactive learning activity on a multi-touch tabletop in an actual college classroom. In our previous in-lab experimental study, we found that this tabletop learning activity outperformed the same learning task on paper. Students had a higher quality of collaboration and learnt more when interacting with an interactive multi-touch tabletop display. Here we compare those results with an “in the wild” implementation of our system. We found that learning gains were significantly lower in the classroom compared to a laboratory setting. Our paper investigates this difference and suggests reasons for such a discrepancy. Introduction The introduction and increasing availability of multi-touch, high-resolution displays in the form of handhelds, tabletops, and whiteboards, open the opportunity to consider these technologies as a prominent alternative to current learning technologies. Given the potential of these novel technologies, numerous research prototypes have been developed by Human-Computer Interaction researchers, exploring how these emerging technologies will impact education. However, to date, only a small fraction of prototypes have materialized outside of research lab settings and studied in actual classroom settings. A foundational work in this domain was presented by Brown (1992), who introduced Design Experiments as complex interventions implemented in educational settings. This approach generated controversy due to the large number of confounding variables present in classrooms. However, Brown was able to identify the fundamental tensions between a natural and artificial setting. Many researchers followed her example and recognized the importance of testing educational interventions in an ecological way. However, to date, little research has been devoted to investigating the strengths and limitations of utilizing interactive surfaces in college-level learning. Comparing Laboratory and Classroom Experiments Phylo-Genie, a learning activity to help college-level students grasp basic concepts in phylogenetics, is implemented on an interactive multitouch tabletop display (Microsoft Surface TM ). This work has been previously presented at the ACM SIGCHI conference (Schneider et al., 2012). In particular, Phylo-Genie provides students opportunities to build their own phylogenetic trees through a learning scenario. In this section we summarize the two studies that we conducted to evaluate Phylo-Genie. The first study, a controlled experiment, was completed in a laboratory setting with college-level students; the second, in- class study, was conducted in a college classroom. Participants: controlled experiment: 28 undergraduate and graduate students (28 female, 28 male, average age = 21.28, SD = 3.70) volunteered to participate in the study, composing 14 dyads. Classroom setting: 32 students, enrolled in an introduction to biology course in a women college (all females), participated in our study. None of the participants had received college-level instruction in evolutionary biology before the study. Materials: The materials used in both studies were similar. A post-test was provided by a university professor who teaches introductory phylo-genetics to college-level students. We rated the quality of collaboration among dyads by using Meier’s rating scheme (Meier, Spada & Rummel, 2007). Procedure: We conducted the experiment in two private laboratory spaces for the controlled experiment, and in the classroom for the second study. Subjects received a 5-minutes introduction to phylogenetics before the Phylo-Genie activity. They then went through all the steps of the learning scenario and completed a post-test. Each session was approximately 60 minutes long. Due to limitations in hardware availability, some students had to work in groups of three in the classroom condition. Results: we found that students in the laboratory setting outperformed students in the classroom on the learning test: F(1,58) = 4.32, p < 0.05 (mean=0.9, SD=0.11 for the laboratory, mean=0.83, SD=0.13 for the classroom). CSCL 2013 Proceedings Volume 2: Short Papers, Panels, Posters, Demos, & Community Events © ISLS 351