Dependencies between stimuli and spatially independent fMRI sources: Towards brain correlates of natural stimuli Jarkko Ylipaavalniemi a, ⁎ , 1 , Eerika Savia a,b, 1 , Sanna Malinen c,d , Riitta Hari c,d , Ricardo Vigário a , Samuel Kaski a,b a Adaptive Informatics Research Centre, Department of Information and Computer Science, Helsinki University of Technology, P.O. Box 5400, FI-02015 TKK, Finland b Helsinki Institute for InformationTechnology HIIT, Department of Information and Computer Science, Helsinki University of Technology, P.O. Box 5400, FI-02015 TKK, Finland c Brain Research Unit, Low Temperature Laboratory, Helsinki University of Technology, P.O. Box 5100, FI-02015 TKK, Finland d Advanced Magnetic Imaging Centre, Helsinki University of Technology, P.O. Box 3000, FI-02015 TKK, Finland abstract article info Article history: Received 28 February 2008 Revised 22 December 2008 Accepted 18 March 2009 Available online 1 April 2009 Keywords: Canonical correlation analysis (CCA) Functional magnetic resonance imaging Human Independent component analysis Natural stimuli Natural stimuli are increasingly used in functional magnetic resonance imaging (fMRI) studies to imitate real-life situations. Consequently, challenges are created for novel analysis methods, including new machine- learning tools. With natural stimuli it is no longer feasible to assume single features of the experimental design alone to account for the brain activity. Instead, relevant combinations of rich enough stimulus features could explain the more complex activation patterns. We propose a novel two-step approach, where independent component analysis is first used to identify spatially independent brain processes, which we refer to as functional patterns. As the second step, temporal dependencies between stimuli and functional patterns are detected using canonical correlation analysis. Our proposed method looks for combinations of stimulus features and the corresponding combinations of functional patterns. This two-step approach was used to analyze measurements from an fMRI study during multi-modal stimulation. The detected complex activation patterns were explained as resulting from interactions of multiple brain processes. Our approach seems promising for analysis of data from studies with natural stimuli. © 2009 Elsevier Inc. All rights reserved. Introduction The focus of functional magnetic resonance imaging (fMRI) studies has been in rather simple block designs aimed to optimize stimulus control and signal-to-noise ratio. When genuinely natural stimuli are used, block designs are no longer appropriate. The aim of this paper is to take the first step forward by inferring brain correlates of natural stimuli with possibly overlapping stimuli. In such uncontrolled setups, it is extremely difficult to differentiate the stimulus-related processes from ongoing brain activity and, analogously, brain activity-related stimulus properties from all other aspects of the natural environment. The statistical hypotheses are no longer self-evidently derived from the experimental setup, which has conventionally constrained the set of possible hypotheses. Instead, identifying the correct statistical hypotheses is a goal of the analysis in itself. Therefore, we propose a data-driven analysis to define testable research questions. Purely hypothesis-driven methods have been used extensively in neuroimaging studies. Earlier, the most typical setup of an fMRI experiment has consisted of alternatingly repeating blocks of rest and controlled stimulation, often using only one type of stimuli (see, e.g., Worsley and Friston, 1995). The hypothesis-driven methods are well suited for these setups. However, the situation gets quickly more complicated when one infers correlates of more complex brain processes. Currently, the focus is starting to shift from simple unimodal stimuli towards integration of multiple sensory stimuli to study cognitive processes and, generally, brain activation related to natural stimuli. The recent interest in more real-life-like experimental setups has triggered the first experiments with natural stimuli, such as movies (see, e.g., Hasson et al., 2004; Bartels and Zeki, 2005a; Damoiseaux et al., 2006). Purely data-driven independent component analysis (ICA, Hyvä- rinen et al., 2001) is able to separate underlying sources of brain activity, or functional patterns (see, e.g., McKeown et al., 1998; Bartels and Zeki, 2005b). ICA looks for spatially independent patterns of activity without any prior knowledge on the location or temporal dynamics of the activity. It has quickly become a common tool to analyze fMRI data. The majority of the found independent patterns are typically left without an explanation in terms of the stimulus features (see, e.g., Bartels and Zeki, 2005a; Damoiseaux et al., 2006; Malinen et al., 2007). NeuroImage 48 (2009) 176–185 ⁎ Corresponding author. E-mail addresses: jarkko.ylipaavalniemi@tkk.fi (J. Ylipaavalniemi), eerika.savia@tkk.fi (E. Savia), sanna@neuro.hut.fi (S. Malinen), riitta.hari@tkk.fi (R. Hari), ricardo.vigario@tkk.fi (R. Vigário), samuel.kaski@tkk.fi (S. Kaski). 1 These authors contributed equally to this work. 1053-8119/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2009.03.056 Contents lists available at ScienceDirect NeuroImage journal homepage: www.elsevier.com/locate/ynimg