Partially Distributed Representations of Objects and Faces in Ventral Temporal Cortex Alice J. O’Toole 1 , Fang Jiang 1 , Herve ´ Abdi 1 , and James V. Haxby 2 Abstract & Object and face representations in ventral temporal (VT) cortex were investigated by combining object confusability data from a computational model of object classification with neural response confusability data from a functional neuroimaging experiment. A pattern-based classification algorithm learned to categorize individual brain maps according to the object category being viewed by the subject. An identical algorithm learned to classify an image-based, view-dependent represen- tation of the stimuli. High correlations were found between the confusability of object categories and the confusability of brain activity maps. This occurred even with the inclusion of multiple views of objects, and when the object classification model was tested with high spatial frequency ‘‘line drawings’’ of the stimuli. Consistent with a distributed representation of ob- jects in VT cortex, the data indicate that object categories with shared image-based attributes have shared neural structure. & INTRODUCTION Neuroimaging studies of ventral temporal (VT) cortex responses to objects have concluded variously in favor of modular (Spiridon & Kanwisher, 2002; Kanwisher, McDermott, & Chun, 1997), distributed (Carlson, Schra- ter, & He, 2003; Cox & Savoy, 2003; Haxby et al., 2001), and process-based (Gauthier, Skudlarski, Gore, & An- derson, 2000; Gauthier, Tarr, Anderson, Skudlarski, & Gore, 1999) accounts of visually based object recogni- tion. Evidence for both the modular and distributed hypotheses comes from the application of novel analyses of the patterns of neural activity that result when viewing objects. Although there is general agreement about the locations of maximal responses to certain classes of ob- jects in VT cortex, the analysis of brain activity patterns has led to an active debate on the nature of the neural representations that underlie these responses. Diverging conclusions about the distributed versus modular nature of object representations in VT cortex have been reached by researchers using reasonable, but nonconvergent, quantifications of modular/distributed codes. Resolving a debate about the nature of object rep- resentations in the cortex requires a quantitatively pre- cise description of the pattern parameters that define the different hypotheses. A more precise definition of ‘‘dis- tributed’’ versus ‘‘modular’’ patterns of activation will give us a standard for determining the degree to which individual patterns vary on this dimension. It will not, however, help us understand why certain areas are more or less modular/distributed or how object category rep- resentations are organized. We argue here that this requires a computational analysis of the structure of object categories that can account for the structure of the brain activity patterns. The representational pa- rameters of this analysis can constrain hypotheses about the kinds of representations that may underlie the neu- ral response patterns. One difficulty in interpreting pattern-based data in the current debate is that ‘‘modular’’ and ‘‘distributed’’ have been characterized often as qualitatively discrete kinds of representations rather than as the endpoints of a continuous variable—with partially distributed codes lying between these extremes. The variable connecting modular with distributed is ‘‘voxel information con- tent.’’ At the modular extreme, each voxel carries infor- mation relevant for only one category of objects. At the distributed extreme, all voxels carry information for all categories. In between, voxels vary in the quality of information they carry about different object categories (e.g., a voxel might contribute to classification accuracy for most, some, or no other object categories). We propose that understanding partially distributed codes in the context of stimulus parameters is the key to linking neural responses with the physical world they represent. This is because distributed and modular patterns of neural responses provide clues for the more interesting question of how we represent and recognize objects neurally. The theoretical viewpoints suggested by modular versus distributed codes provide predictions about the how distributed individual categories of ob- jects should be and about which categories should share voxels. What does a distributed activity pattern indicate about the representation of objects? According to the object- 1 The University of Texas, 2 Princeton University D 2005 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 17:4, pp. 580–590