Brain Imaging and Behavior https://doi.org/10.1007/s11682-018-9901-5 ORIGINAL RESEARCH Encoding the local connectivity patterns of fMRI for cognitive task and state classification Itir Onal Ertugrul 1 · Mete Ozay 2 · Fatos T. Yarman Vural 3 © Springer Science+Business Media, LLC, part of Springer Nature 2018 Abstract In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary. Keywords fMRI · Brain decoding · Fisher vector encoding · Mesh arc descriptors Introduction Brain decoding methods employ brain activity records to predict information about external stimuli (Chen et al. 2014; Daliri 2014). Functional Magnetic Resonance Imaging (fMRI) is a powerful tool used for capturing neural activations observed in a wide range of cognitive tasks including object detection (Mitchell et al. 2008; Behroozi  Itir Onal Ertugrul iertugru@andrew.cmu.edu Mete Ozay mozay@vision.is.tohoku.ac.jp Fatos T. Yarman Vural vural@ceng.metu.edu.tr 1 Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA 2 Graduate School of Information Sciences, Tohoku University, Sendai, Miyagi, Japan 3 Department of Computer Engineering, Middle East Technical University, Ankara, Turkey and Daliri 2014), human emotion categorization (Saarim¨ aki et al. 2015), autobiographical memory retrieval (Rissman et al. 2016) and auditory categorization (Lee et al. 2015). Traditional approaches employ fMRI Blood Oxygenation Level Dependent (BOLD) response of voxels or anatomical regions for cognitive state classification (Haxby et al. 2001; Mitchell et al. 2004; Behroozi and Daliri 2015). Additionally, a number of feature selection methods are applied on voxel activations to select informative voxels (Daliri 2012). Yet, recent findings show that connectivity patterns observed between voxels or anatomical regions provide more information about activities performed in brain compared to the individual voxel BOLD responses. In addition, connectivity between BOLD responses, which are frequently used in resting-state fMRI analysis (Khazaee et al. 2016; Cai et al. 2017), has been shown to provide better classification performance compared to traditional approaches (Richiardi et al. 2011; Shirer et al. 2011). Brain connectivity is also represented by a set of local meshes (Onal et al. 2015a, b), where relationships among multiple voxels are estimated within a predefined neighborhood. Estimated relationships, called mesh arc descriptors (Ozay et al. 2012), are reported to give the best performance