A Novel Tool to Explore and Analyze Large Networks in Mixed Reality Alberto Betella * SPECS, Universitat Pompeu Fabra Enrique Mart´ ınez Bueno SPECS, Universitat Pompeu Fabra Wipawee Kongsantad SPECS, Universitat Pompeu Fabra Riccardo Zucca SPECS, Universitat Pompeu Fabra Xerxes D. Arsiwalla SPECS, Universitat Pompeu Fabra Pedro Omedas SPECS, Universitat Pompeu Fabra Paul F. M. J. Verschure † SPECS, Universitat Pompeu Fabra ICREA ABSTRACT The quantity of information we are producing is soaring: this gener- ates large amounts of data that are frequently left unexplored. Novel tools are needed to stem this “data deluge”. We developed a system that enhances the understanding of large datasets through embodied navigation and natural gestures using the immersive mixed reality space called “eXperience Induction Machine” (XIM). One of the applications of our system is in the exploration of the human brain connectome: the network of nodes and connections that defines the information flow in the brain. We exposed participants to a connec- tome dataset using either our system or a state of the art software for visualization and analysis of connectomic data. We measured their understanding and visual memory of the connectome structure. Our results showed that participants retained more information about the structure of the network when using our system. Overall, our system constitutes a novel approach in the exploration and under- standing of large network datasets. Index Terms: I.3.7 [Three-Dimensional Graphics and Realism]: Virtual reality—; E.1 [Data Structures]: Graphs and networks—; 1 I NTRODUCTION The quantity of information we are producing is soaring. This gen- erates the, so called, “data deluge” [2]. Large chunks of these data are left unexplored due to their heterogeneity and to the lack of tools to effectively visualize and analyze them [10]. These data are frequently organized semantically and stored hi- erarchically using standard formats, such as XML [22]. One of the most effective approaches to represent large amounts of data is the use of network (or “graph”) structures. Networks allow to symbolize the relationships between objects at different scales by visually displaying datasets as a series of nodes connected through edges that express different properties and can reveal the behavior and characteristics of complex systems shaped by the interactions among its components [16]. In context of large network visualization and understanding, im- mersive environments offer unique benefits when compared to stan- dard desktop environments. Previous studies have shown that large screens promote the use of more efficient cognitive strategies [24]; * Laboratory of Synthetic Perceptive, Emotive and Cognitive Systems (SPECS). Center for Autonomous Systems and Neurorobotics (N-RAS), Department of Technology, Universitat Pompeu Fabra. Roc Boronat, 138. 08018 Barcelona, Spain. e-mail: alberto.betella@upf.edu – http://specs.upf.edu † Paul F.M.J. Verschure is the director of the SPECS Laboratory and is an ICREA research professor. http://www.icrea.cat – e-mail: paul.verschure@upf.edu surrounding displays, in particular, offer kinetic depth cues (e.g. 3D rotation) thus allowing the user to amplify the understanding of graphs [1]. Moreover, immersive environments increase the perfor- mance in data analysis tasks that involve spatial relationships (e.g. volume, geometry, common features) thus enhancing spatial under- standing [18]. For these reasons we built an immersive system that uses multi- modal input and output and permits the embodied interaction with large network datasets. To do so, we used the eXperience Induction Machine (XIM), a mixed reality space equipped with a number of sensors and effectors that we constructed to conduct experiments in mixed reality [3]. Using the XIM infrastructure we have previously shown the im- pact of different navigation modes on the understanding of complex neuronal data designed through a neuronal network simulator [6]. Here we present a new mixed reality application capable of han- dling large and complex network structures in real time. As a test scenario we used the human brain connectome, “a com- prehensive structural description of the network of elements and connections forming the human brain” [21]. With our system the user can be fully immersed in this complex data seeking to understand its dynamics and to discover new pat- terns. We provide an ecological form of interaction since the user can literally grab data clusters and manipulate them. In addition, physiological measures (electrodermal activity, heart rate and respiration) are collected through wearable and unobtrusive sensors. These implicit responses are analyzed in real time to detect the user’s interest and suggest to the user new relevant areas in the dataset. To validate empirically our system, we compared it to the Con- nectome Viewer, a state of the art software for visualization and analysis of connectome data [9]. 2 METHODS 2.1 The eXperience Induction Machine The eXperience Induction Machine (Fig. 1) is an immersive space constructed to conduct empirical studies on human behavior in complex, ecologically valid situations that involve embodied inter- action in mixed reality [4]. The XIM covers an area of about 25 m 2 and is equipped with a number of sensors and effectors. XIM effectors include 4 pro- jection screens, a luminous interactive floor [8] and a sonification system [13]. The sensors include a marker-free multi-modal track- ing system [14], floor-based pressure sensors, microphones as well as wearable and unobtrusive sensors that measure the user’s phys- iological state. A glove prototype is used to measure electroder- mal activity (EDA), finger gestures and hand position in the space [12], whereas the Smartex TM WWS shirt measures electrocardio- gram, respiration and body movements [17]. In XIM the game engine Unity 3D [25] is used to render 360 degrees 3D content.