Classifying neural responses to familiar and unfamiliar people over viewing distances in face and body selective areas Carina A. Hahn 1 , Alice J. O’Toole 1 , P. Jonathon Phillips 2 1 The University of Texas at Dallas, Richardson, TX, 2 The National Institute of Standards and Technology, Gaithersburg, MD Background Person recognition in the real world: • people are seen, in motion, from a distance Information from the face, body, and gait contributes to person recognition • differential contribution as a function of distance (Hahn et al., 2015) Neural responses to faces and bodies: • Faces • OFA (Halgren et al., 1999; Puce et al., 1996) • FFA (Kanwisher et al., 1997) • Bodies • FBA (Peelen & Downing, 2005) • EBA (Downing et al., 2001) • Biological motion • pSTS (c.f., Allison et al., 2000) Neural responses to familiarity in the brain leading up to a recognition decision • Previous work - still images and most compared neural response magnitude • Faces (Gobbini & Haxby, 2006, 2007; Rossion et al., 2003; cf. review, Natu & O’Toole, 2011a) • Bodies (Hodzic et al., 2009) • Natu & O’Toole (2015) – Decoded familiarity using static face images • Accurate classification: OFA + FFA; FG + precuneus; VTC + precuneus conjunctions Goal Investigate the neural time course of familiarity processing as person approaches -Which brain regions code the familiarity of the person? -When, in the time-course of the approach, do areas discriminate familiar vs. unfamiliar people? Approach Before scanning: familiarized participants with identities In fMRI scanner: test with videos of unfamiliar & familiar people approaching Analysis: determine discriminability of neural response to familiar vs. unfamiliar people using pattern-classification • examine specified ROIs • dissect discrimination across the timeline of the approach Method ROIs Learning phase (before scanning) Walking Talking Smiling 180° Head Rotation Familiarized with 30 identities  4 motion-based actions  120 videos total Results: Discrimination of Neural Responses to Familiar and Unfamiliar People § Functionally localized face & body selective VT cortex areas § Face selective (faces > objects + scrambled, p < .00001) § Body selective (bodies > objects + scrambled, p < .00001) §Anatomically localized pSTS Localizer session (for voxel selection) Recognition test with 8 second approach video (in fMRI scanner) TR1 TR2 TR3 TR4 0s 2s 4s 6s 8s Localizer OFA FFA EBA FBA pSTS Face selective Body selective Dorsal Ventral • 12 participants • 3T Philips Scanner • TR = 2000ms; TE = 30ms • whole brain coverage (38 slices) • voxel size: 3.44mm x 3.44mm x 4mm § 60 Identities (30 familiar & 30 unfamiliar) § when familiar: different clothes, appearance, hair, etc. § video shows person walking toward camera from 13.6 meters away Conclusions References • First study examining neural correlates of familiarity using naturalistic videos of whole people in motion • Accurate classification of familiar and unfamiliar people in both dorsal and ventral stream areas • Highest classification using distributed body selective voxels in VT cortex • distributed face-selective voxels in VT cortex did not yield accurate classification Classically defined ROIs: • At a distance: Accurate classification in both ventral and dorsal regions • Close-up views: Accurate classification in ventral regions • Correspondence between timing of highest neural decoding accuracy and timing of behavioral responses • Familiarity decoding with people in motion: • extends previous work that used static images (for review cf., Natu & O’Toole, 2011a; Natu & O’Toole, 2015) • when viewing people in motion: multiple time-points where discrimination is possible • classification accuracy possible in multiple, individual ROIs using naturalistic stimuli • previous study in our lab used static face images (Natu & O’Toole, 2015) • Future directions: • combinations of ROIs to examine network of regions involved in person familiarity processing • incorporating parietal regions (e.g., precuneus) and anterior temporal lobe to examine core and extended network (Haxby et al., 2000; Gobbini & Haxby, 2007) See Natu et al., (2010, 2011b, 2015) & O’Toole et al. (2014) for complete methods. Localizer Recognition test PCA of training scans. Project individual training scans into PCA space Pre-selection of “best” eigenvectors for classification Train multiple single dimension linear discriminant networks using coordinates of scans on individual PCs. Classify train data and output prediction scores (d’) Classification of Test Data (Leave-one-out procedure) Select PCs with best prediction scores. Combine best dimensions to create optimal linear discriminant network. Evaluate classification with test data. Test Predictions d’ = z(hit rate) - z(false alarm) Reaction time for accurate trials from Hahn et al. (2015) • Distributed body-selective, not distributed face-selective, voxels achieved accurate classification • Peak accuracy in TR3 at moderately close view Neural decoding timing corresponds to behavioral results in scanner: •Average RT: 4.86 s (within TR3) • d’ = 1.97 (+/- 0.18) Allison, T., Puce, A., & McCarthy, G. (2000). Social perception from visual cues: role of the STS region. TRENDS in Cognitive Sciences, 4(7), 267–278. Downing, P. E., Jiang, Y., Shuman, M., & Kanwisher, N. (2001). A cortical area selective for visual processing of the human body. Science, 293(5539), 2470–3. doi:10.1126/science.1063414 Gobbini, M. I., & Haxby, J. V. (2006). Neural response to the visual familiarity of faces. Brain Research Bulletin, 71(1-3), 76–82. doi:10.1016/j.brainresbull.2006.08.003 Gobbini, M. I., & Haxby, J. V. (2007). Neural systems for recognition of familiar faces. Neuropsychologia, 45(1), 32–41. doi:10.1016/j.neuropsychologia.2006.04.015 Hahn, C. A., O’Toole, A. J., & Phillips, P. J. (2015). Dissecting the time course of person recognition in natural viewing environments. The British Journal of Psychology, 1–18. doi:10.1111/bjop.12125 Halgren, E., Dale, A. M., Sereno, M. I., Tootell, R. B., Marinkovic, K., & Rosen, B. R. (1999). Location of human face-selective cortex with respect to retinotopic areas. Human Brain Mapping, 7(1), 29–37. Haxby, J. V., Hoffman, E. a., & Gobbini, M. I. (2000). The distributed human neural system for face perception. TRENDS in Cognitive Sciences, 4(6), 223–233. doi:10.1016/S1364-6613(00)01482-0 Hodzic, A., Kaas, A., Muckli, L., Stirn, A., & Singer, W. (2009). Distinct cortical networks for the detection and identification of human body. NeuroImage, 45(4), 1264–1271. doi:10.1016/j.neuroimage.2009.01.027 Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience, 17(11), 4302–11. Natu, V. S., Jiang, F., Narvekar, A., Keshvari, S., Blanz, V., & O’Toole, A. J. (2010). Dissociable neural patterns of facial identity across changes in viewpoint. Journal of Cognitive Neuroscience, 22(7), 1570–82. doi:10.1162/ jocn.2009.21312 Natu, V., & O’Toole, A. J. (2011a). The neural processing of familiar and unfamiliar faces : A review and synopsis. British Journal of Psychology, 102, 726–747. doi:10.1111/j.2044-8295.2011.02053.x Natu, V. S., & O’Toole, A. J. (2015). Spatiotemporal changes in neural response patterns to faces varying in visual familiarity. NeuroImage, 108, 151–159. doi:10.1016/j.neuroimage.2014.12.027 Natu, V., Raboy, D., & O’Toole, A. J. (2011b). Neural correlates of own- and other-race face perception: spatial and temporal response differences. NeuroImage, 54(3), 2547–55. doi:10.1016/j.neuroimage.2010.10.006 O’Toole, A. J., Natu, V., An, X., Rice, A., Ryland, J., & Phillips, P. J. (2014). The neural representation of faces and bodies in motion and at rest. NeuroImage, 91, 1–11. doi:10.1016/j.neuroimage.2014.01.038 Peelen, M. V, & Downing, P. E. (2005). Selectivity for the human body in the fusiform gyrus. Journal of Neurophysiology, 93, 603–608. doi:10.1152/jn.00513.2004. Puce, A., Allison, T., Asgari, M., Gore, J. C., & McCarthy, G. (1996). Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study. The Journal of Neuroscience, 16(16), 5205–15. Rossion, B., Schiltz, C., & Crommelinck, M. (2003). The functionally defined right occipital and fusiform “face areas” discriminate novel from visually familiar faces. NeuroImage, 19(3), 877–883. doi:10.1016/ S1053-8119(03)00105-8 Average number of voxels ROI Right Left Bilateral OFA 13.60 (10/12) 7.20 (10/12) 23.00 (8/12) FFA 11.00 (11/12) 7.92 (12/12) 19.27 (10/12) EBA 14.42 (12/12) 9.78 (9/12) 24.67 (9/12) FBA 7.27 (11/12) 5.82 (11/12) 13.70 (10/12) pSTS 87.50 (12/12) 88.33 (12/12) 175.83 (12/12) Average number of voxels ROI # voxels Face selective 392.08 Body selective 343.17 Ventral Classically defined Distributed VT cortex TR1 TR2 TR3 TR4 * * * ROIs with peak classifier accuracy at a moderately close view * * * * * * * Distributed VT Cortex Classically defined ROIs ROIs with peak classifier accuracy at a distant view *p < .01 *p = .011 *p = .003 *p = .001 *p = .002 *p < .0001 *p = .009 *p = .005 Carina.a.hahn@gmail.com