This work was carried out in the context of the Virtual Laboratory for e-Science project. This project is supported by a BSIK grant from the Dutch Ministry of Education, Culture and Science (OC&W) and is part of the I CT innovation program of the Ministry of Economic Affairs (EZ). This work was carried out in the context of the MultimediaN project. This project is supported by a BSIK grant from the Dutch Ministry of Education, Culture and Science (OC&W) and is part of the ICT innovation program of the Ministry of Economic Affairs (EZ). S. Ghebreab, P. Adriaans HCS Lab, University of Amsterdam The Netherlands Email: ghebreab@science.uva.nl URL: http:/ / www.vl-e.nl/ I ncremental Cluster-Wise Regression Analysis of Functional Brain Stimulation-Activation Data Sennay Ghebreab, Pieter Adriaans, Arnold Smeulders We propose a brain activation-stimulation data analysis method that can handle a variety of stimulation types. In case of well-defined stimulation, the method reduces to a standard model-and-fit approach. For ill-defined stimulation, the method searches for systematic components in the brain activation related to the stimulation. If the stimulation is undefined the method separates systematic components from noise. The method can be used to identify brain activity given stimulation data, or to predict stimulation data given brain activation data. Conclusion Conclusion Well-established brain activity analysis methods such as SPM [1] and FSL [4] are broadly used for brain function studies involving well-defined stimulations. Other widely-known methods such as PCA and ICA are useful for brain function studies with undefined stimulations. Our brain activity analysis method also handles ill-defined stimulations, occurring for example when passively viewing natural movies. The method differs from existing ones in that it (i) represents both stimulation and activation signals as continuous multivariate data, allowing the use of advanced nonparametric functional data analysis [3] and (ii) adopts a “search-model-fit” strategy [2], allowing to discover local recurring similarities between activation and stimulation signals The method also accounts for variation in Haemodynamics across brain areas and among subjects. In the 2006 PBAIC, we applied our method to predict feature ratings in Base and Actor categories for Movie3 (see illustration for a schematic explanation). We represented both voxel and feature signals as functional data. In the learning phase, a population-based incremental learning algorithm was used to explore systematic components in each subject fMRI scan, using feature signals to steer the exploration. This resulted in a set of potentially relevant voxel signals for each feature and each fMRI scan. In the subsequent group analysis, each set was explored once again for systematic components to identify optimal functional subspaces across subjects. In the predicting phase, these subspaces, represented by a spatiotemporal regression model per feature, were used to predict feature ratings of the 3 subjects for Movie3. Method Method What do humans visually sense in natural environments and how do they arrive at this sensation? To answer this question, we subject humans to visual stimulation in the form of natural video scenes while their brain activity is measured via whole-brain modalities such as M/EEG and fMRI. Our aim is to identify visual scene features at varying levels of detail (e.g. color, depth, texture, motion, objects) that can be reliably mapped to human brain function. To this end, we develop and apply advanced computer vision, neuro-image analysis, machine learning and data mining methods. Here, we describe the application of an incremental cluster-wise regression analyses method within the 2006 Pittsburgh Brain Activity Interpretation Competition [5]. I ntroduction I ntroduction For the Base category, highest cross-correlation scores ranged between -0.13 (“Tools”) and 0.40 (“BodyParts”), with an average of 0.11. In the category Actor, we obtained correlations of up to 0.65 (“Jill”), with an average of 0.46 by far the best predictions among the 40 participants in the 2006 PBAIC. Likely explanations for the discrepancy between the results in Base and Actor category are: (i) parameters were set for superficial exploration of functional spaces for Base category versus intensive exploration for Actor category (ii) blanks (‘movie off’ parts) were included in the analysis, adversely impacting continuously present (Base) features such as “Faces” and “Motion”, (ii) low agreement in subject feature ratings for Base versus high agreement in Actor category. Results Results S. Ghebreab, A.W.M. Smeulders ISLA Lab, University of Amsterdam The Netherlands Email: ghebreab@science.uva.nl URL: http:/ / www.multimedian.nl/ The 2006 PBAIC involves analysis of fMRI data of 3 subjects who watched three 20-minute movie segments. The subjects completed extensive ratings of multiple features (e.g., attention, faces, music, kitchen) in three categories, representing what they perceived during the fMRI scan. The fMRI data are provided to contestants along with the feature ratings for Movie1 and Movie2. For Movie3, fMRI data are provided but no feature ratings or other content information. The goal of the competition is to predict the experience (feature ratings) of the 3 subjects watching Movie3 purely based their fMRI data. Data Data - 1 0 1 2 3 4 5 6 0 5 0 0 1 0 0 0 1 5 0 Use method to predict feature signal Learning Learning Predicting Predicting image courtesy: Pittsburgh Brain Activity Interpretation Competition image courtesy: Pittsburgh Brain Activity Interpretation Competition Activation (voxel signal from subject fMRI scan) Stimulation (feature signal from subject rating) [1] Friston KJ, Penny W. Posterior probability maps and SPMs. Neuroimage. 2003 Jul;19(3):1240-9. [2] Ghebreab S, Jaffe CC, and Smeulders AWM. Population-based incremental interactive concept learning for image retrieval by stochastic string segmentations. IEEE transactions Medical Imaging, 23-6:676 - 689, 2004. [3] Ramsay JO, Silverman BW. Functional Data Analysis. New York: Springer-Verlag. 1997. [4] Smith SM et al.. Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage, 23(S1):208-219, 2004 [5] Schneider W, Bartels A, Formisano E, Haxby J, Goebel R, Mitchell T, Nichols T, Siegle T. Competition: Inferring Experience Based Cognition from fMRI. Proceedings Organization of Human Brain Mapping, Florence Italy, june, 2006. References References Functional representation of 37634 brain voxel signals from single fMRI Functional representation of feature “attention” signal Registration of “attention” signal to all brain voxel signals Subset of voxel signals with highest cross correlation to “attention” signal Functional PC’s from the subset spanning initial functional subspace Functional PC regression of subset voxel signals against registered ones Functional subspace mapping of all (red) , current (blue) and yet to be explored (black) voxel signal subsets in initial (left) and final (right) steps Selected set of 368 brain voxel signals spanning optimal subspace Selected voxel signals from multiple fMRI scans for group analysis Functional PCA regressors for “attention” signal Predicted feature “attention” signal for 3 subjects watching Movie3