Category representations in the brain are both discretely localized and widely distributed

Abstract

Whether category information is discretely localized or represented widely in the brain remains a contentious issue. Initial functional MRI studies supported the localizationist perspective that category information is represented in discrete brain regions. More recent fMRI studies using machine learning pattern classification techniques provide evidence for widespread distributed representations. However, these latter studies have not typically accounted for shared information. Here, we find strong support for distributed representations when brain regions are considered separately. However, localized representations are revealed by using analytical methods that separate unique from shared information among brain regions. The distributed nature of shared information and the localized nature of unique information suggest that brain connectivity may encourage spreading of information but category-specific computations are carried out in distinct domain-specific regions.

NEW & NOTEWORTHY Whether visual category information is localized in unique domain-specific brain regions or distributed in many domain-general brain regions is hotly contested. We resolve this debate by using multivariate analyses to parse functional MRI signals from different brain regions into unique and shared variance. Our findings support elements of both models and show information is initially localized and then shared among other regions leading to distributed representations being observed.

INTRODUCTION

Two opposing perspectives have dominated thinking about higher brain organization. The localizationist view associates mental functions with specific, discrete brain regions. The distributed view associates mental functions with combinatorial brain activity across broad brain regions. These two perspectives have a long and often combative history in neuroscience as represented by disputes between such localizationist as Gall and Ferrier, and those espousing equipotentialist and/or distributionist views such as Flourens, Goltz, and Lashley (Finger 2001). These disputes are not only of historical interest, but continue to divide the field, even with the advent of sophisticated neuroimaging and analytical tools (Haxby et al. 2001; Kanwisher 2010).

The modern localizationist perspective emerged from the intersection of univariate analyses of brain activity with psychological component models, in which psychologically defined functions are associated with activation of discrete brain regions (Kanwisher 2010). Using this approach, discrete domain-specific regions for faces, bodies, words, objects, and scenes have been identified within ventral occipito-temporal cortex and lateral occipital cortex (Cohen et al. 2000; Downing et al. 2001; Epstein and Kanwisher 1998; Gainotti 2000; Malach et al. 1995; Puce et al. 1996). Indeed, the localized activations have been so consistently observed that some brain regions have been named after their function; e.g., a region of the fusiform gyrus activated in fMRI studies by the perception of faces is known as the fusiform face area (FFA) (Kanwisher 2010).

The modern argument for distributed brain representations is often driven from computer science models that model mental function as the result of massively parallel distributed processing. Brain activity is assessed with multivariate pattern analysis (MVPA), which measures concurrent brain activity over a spatially distributed brain region(s) to predict a particular mental function or decode its held representation (Norman et al. 2006). In striking contrast to the localizationist view, MVPA studies have found that visual category information can be measured using patterns of activity, even in brain regions that do not show a change in mean activation level to that category (Davis and Poldrack 2014; Haxby et al. 2001) and thus would not be identified by univariate analysis. Indeed, these patterns of activity can classify object categories even when discrete regions identified in univariate localizationist analyses are removed from the data (Haxby et al. 2001). [Note, by distributed representations here, we do not mean hierarchical feed-forward and/or modular networks consisting of a small set of discretely localized domain-specific brain regions, as has been described for face processing (Haxby et al. 2000; Tsao et al. 2003).]

The differing methodological approaches described above may reveal different aspects of the same neural phenomena. Here, we reconcile the localized and distributed perspectives by considering the sources of information in a given region. In our view, information can be redundant, or shared, from other regions, or unique to a region due to local processing. Our hypothesis is that computations occur initially in localized regions and then are shared widely to other brain areas, resulting in the distributed representations observed in multivariate analyses. That is, localized computations give rise to distributed representations.

We tested this hypothesis by using fMRI to examine visual category representations for four visual categories: faces, letters, fruits/vegetables, and vehicles in seven brain regions of interest (ROIs). We first used MVPA to discriminate among category representations within each ROI. Our expectation, confirmed by the present analysis, was that the pattern of brain activity in each ROI would discriminate among these categories. We then removed shared information from among the ROIs to measure the distribution pattern of the remaining unique information, which we predicted would be localized in domain-specific regions.

Put another way, our goal was to estimate the classification accuracy of a target region for each visual category while removing the effects of other regions. To do this, we first calculated the classification accuracy for all regions to measure the combined contribution of all regions toward predicting each category (full model). We then compared this full model to the classification accuracy of all regions except the target region (full – left out model). The difference in classification accuracies represents the amount of additional information found in the target region over and above any other brain area (relative accuracy). A relative accuracy that is significantly greater than zero would mean that the target brain area has unique information predictive of a given visual category. In contrast, a relative accuracy that is nonsignificant would mean that this brain area may contain information predictive of a given category but this information is not unique and is shared with other brain areas.

We considered three possible outcomes. First, if information is initially localized to a region and not shared with other regions, then there should be no effect of removing covariation with other regions. Second, if unique information is widely distributed, then removing covariation will not change the distribution (as there is no covariation to remove). Finally, if information is initially localized to a region, but then widely shared among regions, removing covariation among regions will remove the shared information and reveal the initially localized representation. Based upon our prior intracranial electrophysiological studies (Engell and McCarthy 2010, 2011; Puce et al. 1999) of the temporal sequence of face processing across discrete brain regions, we predicted that our results would be consistent with this third outcome. Alternative temporal sequences of localized and distributed representations are considered in discussion.

MATERIALS AND METHODS

Participants.

Twenty right-handed, healthy adults participated in the study. All had normal or corrected-to-normal vision and no history of neurological or psychiatric illnesses. Institutional Review Board approval was obtained for this research protocol, and written informed consent was obtained from all participants.

Experimental design and stimuli.

Stimuli were high-quality color images of faces, letters (nonpronounceable nonsense words), fruits/vegetables, vehicles, and objects. In addition, phase-scrambled images were generated by computing a 2-D Fourier transform of the images of faces, letters, fruits/vegetables, vehicles, and objects, permuting the phase spectrum of each transform, and then inverse transforming the images. The resulting images contained the same spatial frequencies of the original images, but were otherwise unrecognizable.

On each trial, participants saw one of the stimuli for 0.5 s followed by phase-scrambled images every 0.5 s for a jittered intertrial interval of 6–12 s. Participants were instructed to press a button when they saw a circle, a predesignated target. Circles were shown a total of 30 times while each of the other categories was shown 60 times. We did not examine results for the circles, which were included only to ensure participants attended to the stimuli. We also did not examine the results for the object category, which was composed of several distinct subcategories that were included for a different purpose. Participants completed a total of 330 trials that were evenly split into 10 runs.

MRI acquisition.

Data were acquired using a 4T GE scanner (General Electric, Milwaukee, WI). Functional images were acquired using a gradient-recalled inverse spiral pulse acquisition (TR = 1,500 ms, TE = 35 ms, flip angle = 62°, FOV = 24 cm, matrix = 64 × 64, slice thickness = 3.8 mm, 34 slices, voxel size = 3.75 mm³). A high-resolution structural image was acquired for registration using a 3D fast spoiled gradient-recalled (SPGR) sequence (TR = 12.2 ms, TE = 5.3 ms, flip angle = 20°, FOV = 256 × 256 mm, matrix = 256 × 256, slice thickness = 1.2 mm, 124 slices).

Preprocessing.

Image preprocessing was performed using custom scripts (available on https://github.com/HumanNeuroscienceLab/facemem) that incorporate functions from AFNI (v 2014–10–23 OpenMP, https://afni.nimh.nih.gov/afni), FSL (v 5.0.7, https://www.fmrib.ox.ac.uk/fslwiki), and Freesurfer (v 5.3.0, https://surfer.nmr.mgh.harvard.edu). The analysis pipeline was as follows: 1) the first 4 volumes (6 s) were discarded to allow for MR equilibration, 2) motion correction was performed on the functional images using AFNI’s 3dvolreg, 3) skull stripping of the mean functional image was performed using FSL’s BET, 4) spatial smoothing with a 5-mm FWHM Gaussian mask was performed using FSL’s SUSAN, 5) high-pass filtering with a 0.01 Hz cut-off to remove low-frequency drift was conducted using FSL, as was mean-based global intensity normalization. Structural images were skull-stripped with AFNI’s 3dSkullStrip. The functional images were registered to the high-resolution structural images with a linear registration using FSL’s Flirt. The resulting structural images were then nonlinearly registered to the Montreal Neurological Institute’s MNI152 template (2 mm isotropic) using FSL’s FNIRT (Andersson et al. 2007), and this transform was then applied to the functional images in anatomical space.

Regions of interest.

Patterns of brain activity to each photo were measured in seven regions of interest (ROIs) constructed from meta-analytic brain maps from the Neurosynth database (http://neurosynth.org). Four ROIs were selected by prior research to be category-selective: the fusiform face area (FFA), parahippocampal place area (PPA), lateral occipital complex (LOC), and visual word form area (VWFA). Three sensory or motor control regions not thought to be category specific were also included: primary visual cortex (V1), primary/secondary motor areas (M1/2), and primary/secondary auditory areas (A1/2).

Table 1 illustrates the steps involved in constructing each ROI. First, a query was made to the Neurosynth database (frequency threshold of 0.05), producing meta-analytic activation maps specific to each category. For instance, for faces and the FFA, we searched for any word beginning with face or FFA. Then, we created contrasts of the meta-analytic maps to ensure category specificity. For instance, for faces, we created the following contrast: Face > (Scene + Object + Letter), which results in voxels that have more activity for faces than the average of the other three categories. We kept only significant voxels, setting a voxel-level threshold of Z > 2. Subsequently, we applied anatomical masks to further constrain each category map (Table 1); masks were taken from the Harvard-Oxford atlas and Freesurfer parcellations. We kept any clusters larger than 25 voxels to ensure a minimal level of clustering and removed any voxels that overlapped between ROIs to ensure greater independence between ROIs. Finally, we split ROIs into the right and left hemisphere.

Table 1.

Steps in generating ROIs

Category	Neurosynth Query	Contrast	Anat Mask
Face	(face* | ffa*)	Face < (Scene + Object + Letter)	TOFC
Scene	(scene* | place*)	Scene < (Face + Object + Letter)	PPAp, TOFC
Object	object* &~(face* | scene* | letter*)	Object < (Face + Scene + Letter)	LOCi
Letter	(letter* | word form) &~face*	Letter < (Face + Scene + Object)	ITGto
Visual	early and visual	Visual < (Motor + Auditory)	V1, Pericalc
Motor	motor and response	Motor < (Visual + Auditory)	Precentral
Auditory	Sound	Auditory < (Visual + Motor)	TransTemp

The first column indicates the different categories. The second column indicates the Neurosynth search query to generate the meta-analytic map. The third column indicates the contrast used to constrain the category-selective map. Finally, the fourth column indicates the anatomical mask from either the Harvard-Oxford atlas or Freesurfer parcellations. TOFC, temporo-occipito-fusiform cortex; PPAp, parahippocampal area posterior; LOCi, lateral occipital cortex inferior; ITGto, inferior temporal gyrus temporoparietal part; Pericalc, pericalcarine cortex; TransTemp, transverse temporal cortex.

Classification analyses.

Patterns of brain activity to each photo were measured in seven regions of interest (ROIs) constructed from meta-analytic brain maps from the Neurosynth database (http://neurosynth.org) masked by anatomical parcellations from Freesurfer. For each voxel in the ROIs, we measured trial-level activity by obtaining parameter estimates (or betas) for each trial at the subject-level using AFNI’s 3dDeconvolve and 3dREMLfit functions. We applied a linear model with one regressor for each trial and additional covariates of noninterest, including baseline effects for each run and 6 motion parameters. Each trial regressor was modeled using a double-gamma function with a duration of 0.5 s and produced, for each trial, a volume of voxelwise betas (i.e., patterns of brain activity).

Figure 1 provides an overview of our analytic approach. To classify the category of activity patterns, we applied a multinomial logistic regression with a lasso penalty (Least Absolute Shrinkage and Selection Operator; see Eq. 1) on the voxelwise betas in each ROI (Friedman et al. 2010). Each voxels beta series was Z-scored before entry in the lasso model to allow for the model to converge and to account for differences in activity levels between voxels. The classification analysis was implemented in the R packages caret (Kuhn 2008) and glmnet (Friedman et al. 2010). For each region, a linear model consisting of trial-level brain activity for all voxels was used to predict the four categories shown on each trial. Beta or feature weights were fit to each voxel and were calculated for each category. Each feature weight measured the contrast of a voxel’s activity for a given category vs. the other categories [e.g., faces > (letters + fruits/vegetables + vehicles)/3]. We used the lasso to select the most important features and to account for collinearity in the data by penalizing the feature weights. The optimal penalty level for the lasso (lambda) was determined by maximizing classification accuracy using a leave-one run out cross-validation; the optimal lambda was then applied to the full data to estimate the feature weights.

Fig. 1.

Our methods and predictions. A: our measures of classification accuracy for one category (faces) in regions A, B, and C. On the left, separate MVPA analyses were performed on each region, and activity patterns for each region were used to predict each category. On the right, we examined regions A, B, and C together and assessed the improvement in classification accuracy for region A over and above all other regions. To calculate the unique information in region A (A′), we took the difference in classification accuracy when using all regions (full model) from using all regions except region A (left-out model). B: three predictions of our analysis are shown when examining regions separately (left) or together (right). Darker red colors indicate more information for faces, whereas gray indicates no face information. In the first case, unique information is assumed to be localized in region A and not shared. In the second case, information is localized first to region A and then shared with other regions. Finally, in the third case, unique information for faces is present in every region.

We applied this classification approach for each subject to each individual region, to all regions combined (full-model), and to the full-model with each region left-out. We took the difference from the full model and left-out model (full minus left-out) to calculate the unique information in a region for a given category. If only one region contributes information to a category, then removing this region would lead to an accuracy of zero and the difference would be equal to the full model accuracy. In contrast, if information for a category is completely shared between regions, then removing a region would not lead to a change in accuracy, as this information is redundantly represented in other regions, and the difference would be zero. Thus difference values for a given region range from zero to the full model accuracy with higher values indicating more unique information for a category in a region and less shared information with other regions.¹ Group-level classification accuracies were obtained by averaging the accuracies from all participants for each category and region. Figure 2 illustrates this analytical process in terms of a graphical model.

Fig. 2.

A: a sample graphical model is shown with four nodes. This particular model is agnostic to the direction of the connections; therefore information is shown flowing into and out of each node. B: the information content of two nodes, B and C, is shown. Each node contains information unique to that node as well as information shared from other connected nodes. C: as an example, the information content of a node (A) is shown in regards to predicting whether a trial was a face or letter. The first row shows the prediction (strongly face-selective) using only node A as a regressor. The next row shows the prediction using node A but covarying for nodes B, C, and D (not face-selective). Removing the shared information reveals that node A did not have any unique information predictive of seeing a face.

$$\begin{array}{cc}-\left\{\frac{1}{N}{\displaystyle \sum _{i=1}^N\left[{\displaystyle \sum _{k=1}^K{y}_{il}\left({\beta }_{0k}+x_i^T{\beta }_k\right)-\log \left({\displaystyle \sum _{k=1}^K{e}^{{\beta }_{0k}+x_i^T{\beta }_k}}\right)}\right]}\right\}& +\lambda \left(\alpha {\displaystyle \sum _{j=1}^p{\Vert {\beta }_j\Vert }_q}\right)\\ \text{Logistic regression}& \text{Penalty}\end{array}$$ (1)

We determined the significance of the classification accuracies using permutation testing. The category label for each trial was randomly permuted or shuffled 200 times. On each permutation, the different classification analyses were rerun and the accuracies were stored. We used the same resampling between trials in each subject. Group-level permutation distributions were taken by averaging the accuracy values across subjects for each permutation. To obtain more precise P values, we fit a generalized Pareto distribution to the tail of each permutation distribution (Winkler et al. 2016). The false discovery rate (FDR) was used to correct for multiple comparisons.

RESULTS

We predicted the category of each stimulus on each trial by applying multinomial logistic regression with a lasso penalty to the activity patterns in the voxels within seven ROIs. Four ROIs were selected by prior research to be category-selective: the fusiform face area (FFA), parahippocampal place area (PPA), lateral occipital complex (LOC), and visual word form area (VWFA). Three sensory or motor control regions not thought to be category specific were also included: primary visual cortex (V1), primary/secondary motor areas (M1/2), and primary/secondary auditory areas (A1/2). We first applied the classification analyses to each ROI separately and, thus, in this first analysis, did not account for shared information. We then combined the seven ROIs together into a single full model and compared the performance to each region left-out, which allows us to remove information shared among regions (see Fig. 1; note all P values are based on permutation testing and FDR-corrected for multiple comparisons). To allow direct comparison between the separate and combined approaches, we obtained a relative accuracy measure by taking the difference between the observed classification accuracy and the average accuracy from the null distribution. For example, the classification accuracy for Faces in the FFA was 0.55; the relative accuracy was calculated as 0.30 (the classification accuracy 0.55 minus 0.25, the average chance accuracy).

Average classification accuracy across categories.

First, we investigated the distribution of category information among our ROIs by calculating the average prediction accuracy for the four categories. If category information is localized, we would expect significant classification accuracy in one (and possibly more) of our four category-selective regions, but not in the three sensory/motor control regions where we did not expect category-level information. However, when each ROI was separately analyzed, we found significant classification accuracies in all regions [mean relative accuracy = 0.084 ± 0.009 (SE), all P values < 0.05; Fig. 3A, Tables 2 and 3], including the sensory/motor control regions. Unique category information was more limited when ROIs were analyzed together. Two ROIs had greater category information compared with the other areas (FFA = 0.051 ± 0.011 and PPA = 0.012 ± 0.005; P values < 0.05) and two other areas had marginally significant category information (LOC = 0.008 ± 0.007 and VWFA = 0.007 ± 0.004; P values < 0.1). Notably, adding any single sensory/motor region to the full model reduced the accuracy of category predictions (V1 = −0.006 ± 0.003, M1/2 = −0.002 ± 0.0–5, A1/2 = 0.005 ± 0.003). When considering that sensory/motor areas could significantly categorize when tested in isolation, this result strongly indicates that this isolated classification accuracy was based upon redundant, shared information from category-selective regions.

Fig. 3.

The classification accuracy or information found on average for all four categories (A) and for each individual category (B) is shown when examining regions separately and together. Each cell indicates the classification accuracy relative to the average null distribution with greater accuracies as dark red and lower accuracies as yellow. Cells in gray indicate nonsignificant classification (P > 0.05, FDR-corrected).

Table 2.

Group-average classification accuracies relative to the average null distribution (± SEs) found on average for all 4 categories and for each category (columns) in each region (rows)

	Average	Faces	Fruits	Letters	Vehicles
Each region analyzed separately
R FFA	0.141 ± 0.032*	0.304 ± 0.068*	0.059 ± 0.013*	0.181 ± 0.041*	0.022 ± 0.005
PPA	0.115 ± 0.026*	0.16 ± 0.036*	0.105 ± 0.023*	0.104 ± 0.023*	0.092 ± 0.021*
LOC	0.098 ± 0.022*	0.114 ± 0.025*	0.095 ± 0.021*	0.134 ± 0.03*	0.049 ± 0.011*
L VWFA	0.103 ± 0.023*	0.143 ± 0.032*	0.074 ± 0.017*	0.126 ± 0.028*	0.068 ± 0.015*
V1	0.05 ± 0.011*	0.064 ± 0.014*	0.018 ± 0.004	0.079 ± 0.018*	0.038 ± 0.009*
M1/2	0.046 ± 0.01*	0.048 ± 0.011*	0.015 ± 0.003	0.092 ± 0.02*	0.031 ± 0.007*
A1/2	0.035 ± 0.008*	0.026 ± 0.006	0.035 ± 0.008*	0.044 ± 0.01*	0.037 ± 0.008*
Each region analyzed together
R FFA	0.051 ± 0.011*	0.124 ± 0.027*	0.025 ± 0.011*	0.045 ± 0.015*	0.011 ± 0.019
PPA	0.012 ± 0.005*	0.02 ± 0.01	0.004 ± 0.014	−0.004 ± 0.011	0.028 ± 0.017*
LOC	0.008 ± 0.007	−0.002 ± 0.013	0.032 ± 0.016*	−0.008 ± 0.02	0.011 ± 0.013
L VWFA	0.007 ± 0.004	0.014 ± 0.015	0.007 ± 0.013	0.002 ± 0.011	0.006 ± 0.01
V1	−0.006 ± 0.003	0.001 ± 0.011	−0.01 ± 0.013	−0.01 ± 0.011	−0.005 ± 0.015
M1/2	−0.002 ± 0.005	−0.001 ± 0.009	0.002 ± 0.015	−0.003 ± 0.013	−0.007 ± 0.017
A1/2	−0.005 ± 0.003	−0.012 ± 0.01	0.016 ± 0.013	−0.028 ± 0.011	0.004 ± 0.012

Results are arranged based on the type of analysis (separate or together). The average (permuted) null distribution was ~0.25 for each region analyzed separately and ~0 for each region analyzed together. R, right; L, left; FFA, fusiform face area; PPA, parahippocampal place area; LOC, lateral occipital complex; VWFA; visual word form area; V1, visual cortex; M1/2, primary/secondary motor areas; A1/2, primary/secondary auditory areas.

^*Significance at P < 0.05, false discovery rate (FDR)-corrected.

Table 3.

Shown are the Z-scores of the classification accuracies (see Table 2) found on average for all four categories and for each category (columns) in each region (rows)

	Average	Faces	Fruits	Letters	Vehicles
Each region analyzed separately
R FFA	10.85*	11.22*	3.52*	8.77*	1.37
PPA	10.32*	8.28*	5.9*	6.28*	4.73*
LOC	9.75*	6.23*	5.14*	7.86*	3.2*
L VWFA	9.85*		4.27*	6.18*	3.9*
V1	5.14*	3.57*	1.14	4.07*	2.13*
M1/2	5.85*	3.2*	1.03	6.23*	1.94*
A1/2	4.51*	1.63	2.51*	2.82*	2.46*
Each region analyzed together
R FFA	7.42*	4.58*	1.51*	2.16*	0.69
PPA	2.14*	1.03	0.24	−0.27	1.43*
LOC	1.4	−0.12	1.71*	−0.46	0.71
L VWFA	1.24	0.71	0.38	0.09	0.35
V1	−1.04	0.04	−0.64	−0.53	−0.29
M1/2	−0.4	−0.09	0.15	−0.22	−0.46
A1/2	−0.92	−0.76	1.15	−1.79	0.29

Results are arranged based on the type of analysis (separate or together). See Table 2 for abbreviation definition.

^*Significance at P < 0.05, FDR-corrected.

Classification accuracy for each category.

We next investigated the distribution of information for each individual category. If information for each category is localized, then we would expect specific regions to be associated with specific categories. However, when examining each region separately, we found distributed representations for each category. Significant category information was found for each category in almost every region (mean relative accuracy = 0.084 ± 0.021; P values < 0.05; Fig. 3B, Tables 2 and 3). Considering only these results, we would conclude that many brain regions contribute to representing each visual category. However, when we combined the regions to remove shared information, we found localized representations of unique information for each category (see Table 4 for full minus left-out model accuracies). Consistent with our predictions, we found unique information for faces localized in the FFA (relative accuracy = 0.124 ± 0.027; P < 0.05), fruits/vegetables partially localized in the LOC (relative accuracy = 0.032 ± 0.016; P < 0.05), and vehicles localized in the PPA (relative accuracy = 0.028 ± 0.017; P < 0.05). Although accuracy for letter information in the VWFA was nonsignificant (relative accuracy = 0.002 ± 0.011; P = 0.55), the amount of information for letters in the VWFA was the second highest after faces. These results, then, support our hypothesis that category information is initially localized and then shared with other regions (Fig. 1B).

Table 4.

Differences in the group-average classification accuracies between the full-model minus the left-out model (together analysis)

	Average	Faces	Fruits	Letters	Vehicles
R FFA	0.45–0.39 = 0.052	0.58–0.46 = 0.124	0.34–0.31 = 0.027	0.48–0.44 = 0.045	0.38–0.37 = 0.011
PPA	0.45–0.43 = 0.012	0.58–0.56 = 0.019	0.34–0.33 = 0.006	0.48–0.49 = −0.004	0.38–0.35 = 0.028
LOC	0.45–0.44 = 0.009	0.58–0.58 = −0.003	0.34–0.31 = 0.033	0.48–0.49 = −0.008	0.38–0.36 = 0.012
L VWFA	0.45–0.44 = 0.007	0.58–0.57 = 0.014	0.34–0.33 = 0.008	0.48–0.48 = 0.002	0.38–0.37 = 0.006
V1	0.45–0.45 = −0.006	0.58–0.58 = 0	0.34–0.35 = −0.009	0.48–0.49 = −0.009	0.38–0.38 = −0.005
M1/2	0.45–0.45 = −0.002	0.58–0.58 = −0.001	0.34–0.34 = 0.003	0.48–0.49 = −0.004	0.38–0.38 = −0.008
A1/2	0.45–0.45 = −0.005	0.58–0.59 = −0.013	0.34–0.32 = 0.017	0.48–0.51 = −0.028	0.38–0.37 = 0.004

Values are indicated for each region (rows) and category (columns). See Table 2 for abbreviation definitions.

Feature weights in the FFA for each category.

Inconsistent with our predictions for category information, we found significant representations (P values < 0.05) in the FFA for letters (relative accuracy = 0.045 ± 0.015) and fruits/vegetables (relative accuracy = 0.025 ± 0.011). Classification accuracy is like an F-statistic; it indicates the presence but not the direction of an effect. Consequently, to better understand the nature of the nonface representations in the FFA based on classification accuracy, we examined the feature weights for predicting each category. For a given category, voxels can have positive feature weights (i.e., higher activity is predictive of seeing a given category) or negative feature weights (i.e., lower activity is predictive of that category). We interpreted voxels with positive weights as representing that category, and voxels with negative weights as not representing that category. This is consistent with the univariate task-activation literature where higher activity for a face relative to baseline in FFA voxels (positive weights) is indicative of a face, while lower activity for letters relative to baseline in FFA voxels (negative weights) indicates it is not a face. We summarized the results by calculating the percentage of voxels in the FFA with positive or negative weights for each category (Fig. 4). Consistent with our prediction that the FFA has localized face representations, we found that only faces had a significant number of voxels with positive weights predictive of seeing a face (10.7 ± 1.1%; P = 0.009, FDR-corrected). The two categories (letters and fruits/vegetables) that had significant classification accuracy did not have significant positive weights (both P values = 1, FDR-corrected). Instead, for letters there was a significant number of voxels with negative weights predictive of not seeing a face (6.6 ± 1.1%; P = 0.021). Similarly, for fruits/vegetables there was a large proportion of voxels with negative weights (albeit nonsignificant, 5.7 ± 0.96%; P = 0.37). Since we were concerned with the instability of feature weights in multivariate models, we reestimated the weights for each voxel in a univariate model and found similar results (Fig. 5). The feature weights suggest that information for nonface categories is present in the FFA, but that information reflects not seeing a face.

Fig. 4.

Using a multivariate regression, the percent of positive (red) and negative (blue) feature weights in the FFA (y-axis) for each category (x-axis). Error bars indicate the standard error of the null distribution. Feature weights were calculated using a regularized multinomial regression with a lasso penalty that included voxels in all regions. The number of voxels in each region with positive or negative feature weights were counted.

Fig. 5.

Using a univariate regression, the percent of positive (red) and negative (blue) feature weights in the FFA (y-axis) for each category (x-axis). Error bars indicate the standard error of the null distribution. Feature weights were calculated using a binomial logistic regression. For each voxel, brain activity (betas) for each trial were fit to the category seen on that trial. Then, in each region, the number of voxels with positive or negative feature weights were counted. Permutation tests (N = 200) were performed for estimating the null distribution and significance testing.

DISCUSSION

We found support for both widely distributed and discretely localized category representations, but with important qualifications. Distributed representations were found within regions tested in isolation; i.e., information from several categories was represented within each of several regions. However, further analysis indicated that some category information was shared or redundantly represented among regions. When we combined the regions into one classification model, we found that unique information for each category was localized to previously identified category-selective regions. Unique information for nonface categories was found in the FFA but reflected decreased activity and a process of not seeing a face. Thus category information is initially localized in discrete brain regions, but then is shared resulting in widely distributed representations.

Temporally constrained modularity.

As predicted, unique information for faces was found in FFA (Allison et al. 1994), letters in VWFA albeit nonsignificant (Cohen et al. 2000), fruits/vegetables in the LOC (Coutanche and Thompson-Schill 2015), and vehicles in PPA. Our findings for the four categories are consistent with the concept of temporally constrained modularity previously proposed by our group for face processing on the basis of subdural recordings from implanted electrodes in human patients (Engell and McCarthy 2010, 2011; Puce et al. 1999). In that model, face information is initially computed in domain-specific regions, but then that information becomes distributed via brain connectivity (Bullmore and Sporns 2012; Van Essen and Maunsell 1983; Zeki and Shipp 1988). At a given moment in time, neural signal reflects both the unique information actively computed in an area and shared information distributed from other regions.

Hierarchical modular networks are an example of temporally constrained modularity (Felleman and Van Essen 1991; Haxby et al. 2000; Tsao et al. 2003; Zeki and Shipp 1988), but not in the sense explored here. Using the face network as an example, all regions of this network are face selective, but computations for faces (e.g., face parts vs. whole) are distributed in discrete domain-specific regions (Engell and McCarthy 2013; Haxby et al. 2000; Tsao et al. 2003). Here, we were interested in the degree to which regions clearly outside of these face-selective networks (e.g., auditory A1) contain representations for faces (and other stimulus categories). Indeed, we found that such remote regions contained information for faces, a result that is not predicted by hierarchical or other modular models. However, this information was not unique to these regions, but rather was shared with other well studied, face-selective regions.

Different classifiers extract different information.

In the present study, we examined category representations only. However, different classifiers could be used on the same data to parse other visual computations (e.g., edge detection and salience) or different levels of a hierarchical modular network (e.g., face parts vs. whole). For instance, Coutanche and Thompson-Schill (2015) used classifiers trained on different stimulus features to predict brain activity following a cue for a fruit or vegetable. Significant decoding for identity was found in the left anterior temporal lobe, shape in the LOC, and color in the right V4. This and related findings (Rotshtein et al. 2005) suggest that brain regions carry out distinct computations, and the unique information associated with a particular computation can be isolated by different classifiers.

Parallels with connectivity work.

Distinguishing between unique and shared information in category perception draws conceptually from work in functional connectivity and multivoxel pattern analysis. MVPA typically measures the amount of information for a given function in a particular region, whereas functional connectivity measures the flow or sharing of that information between regions. In a connectivity model, a region cannot be considered in isolation as its activity represents both incoming or shared information from other regions and unique information processed in that region (Mumford and Ramsey 2014; Ramsey et al. 2014). Therefore, connectivity analyses remove shared information to ensure that a connection between regions A and B is not dependent on another region C or set of regions such as with a lasso (Hyvarinen and Smith 2013; Mumford and Ramsey 2014). Here we applied that same logic to MVPA and category selectivity, but instead of removing shared information in connections, we removed shared information across voxels on the basis of how well each voxel’s activity predicted a given category. Combining voxels from other regions allowed us to remove the shared information in predicting category membership present across voxels in multiple regions.

Need for post hoc tests to indicate direction of effect.

Separating shared information from unique information dramatically changed the interpretation of our MVPA results. Care needs to be taken in interpreting a distributed organization based on MVPA as those results may not reflect unique computations occurring across many regions. Furthermore, any unique computations identified require further clarification. The ambiguity of classifier results is analogous to an F-test, which indicates the presence of an effect but not its direction (Shehzad et al. 2014). To determine the direction, a post hoc analysis is required such as a t-test or by examining the feature weights of the model. The direction of the feature weights helps clarify if a region is involved in a process (increase in activity) or not involved in another process (decrease in activity). With this approach, we can specify if a region has unique information for a particular process.

Defining information.

Our findings suggest that greater precision is needed when defining “information” within a region. Although we initially found distributed information for each category, when we probed the nature of that information, we found that unique information for each category was localized. We found that information in a given region (A) is composed of unique computations occurring in that region (A′) and shared representations from other regions (e.g., B and C). The inputs/outputs of a node then must be considered to understand the shared information represented in a region. In other words, a brain region cannot be considered in isolation especially within a connected network (Bullmore and Sporns 2012; Craddock et al. 2013; Van Essen and Maunsell 1983). This may require a shift in thinking for task-based MVPA studies.

We observed that the information content of a region will also depend on the feature weights. In the task-based literature, when a contrast such as “face > scene” yields voxels with positive activity, we consider those to be face selective, whereas voxels with negative activity, we consider to be scene selective. We took this viewpoint when interpreting the feature weights in our analyses, which held for both multivariate and univariate analyses. When activity was on average positive, this meant the region was category selective, whereas when activity was on average negative, this meant the region was not selective for that category. A region then has information for a category if the region significantly predicts that category and has positive feature weights.

Nature of shared information.

Our approach can also be used to specify the amount of shared information between regions. We found that the major proportion of information was shared among regions. But what is the nature of the shared information identified here, and its role in neural function? Information spread may be obligatory and epiphenomenal and simply reflect network connectivity (Deco and Jirsa 2012). The sharing of information, however, might be more functional. Shared information in a region could reflect inputs from other regions that are then used for local computations, such as V1 feeding forward to FFA or the multistage segregation and integration of information in visual cortex (Zeki and Shipp 1988). Shared information could also reflect preparation or predictive coding (for example, auditory activity from a person's voice could prime visual activity of a person's face) (Clark 2013). If each region has specialized computations, then communication between regions may allow for re-representation of the same information in a different space, allowing for different perspectives to be taken on the same input.

Considering distributed models.

High-resolution fMRI studies have found distributed and interdigitated representations for several categories throughout the ventral visual cortex, in contrast to the more localized findings of the present study (Grill-Spector et al. 2006; Weiner and Grill-Spector 2010). Hanson and Schmidt (2011) suggest that due to an averaging artifact the FFA appears domain specific at standard resolutions (e.g., 3 mm isotropic), whereas at higher resolutions (e.g., 1 mm isotropic) domain-general category representations can be observed. The prior work using high-resolution fMRI, however, has not disentangled unique from shared representations. Consequently, we cannot know if the nonface representations in the FFA reflect active computations occurring in the FFA or representations shared by other regions. Future high-resolution fMRI studies should consider incorporating our approach to resolve this issue.

Our approach of finding unique vs. shared information is agnostic to the underlying computations occurring in a region. Our results suggest that unique information in each region is category selective (e.g., FFA for faces). This localized perspective can still be consistent with more distributed models such as the object form topography model, which states that each voxel encodes a different attribute of an object with each attribute being shared across objects (O’Toole et al. 2005). In terms of our working model, category-selective regions encode attributes specific to that category. For instance, the FFA, we hypothesize, processes attributes specific to face (no matter the image), and even its response to nonface stimuli will reflect face processing (Mur et al. 2012). It is possible that with more stimuli and more objects with face-like attributes, we may have unique information for such nonface stimuli within the FFA.

Directed connectivity.

Our analysis is undirected and does not consider causal relationships. When examining shared information, we do not know if that information is input from another region or output to another region. For instance, consider the LOC and FFA. Object features of a face might be processed first in the LOC and then shared with the FFA, which additionally processes face features. Since object processing is common to LOC and FFA, that information would be removed from our analysis, which prioritizes unique information.

We have argued for a sequence of information processing whereby representations are initially localized and then widely distributed via sharing of information. This sequence was motivated by our prior electrophysiological studies (Engell and McCarthy 2010, 2011; Puce et al. 1999), However, it is not proven by the undirected analyses presented here. Indeed, it is possible that distributed representations occurred first, and then contracted to localized representations as processing by discrete face-selective brain regions created unique information in those discrete regions. Based on our definition of distributed representations (Fig. 1), it is unlikely that such representations occurred first (or last) since that would imply unique contributions in each region, which were not found. However, it is possible that information was first widely shared followed by localized computations. This viewpoint would be consistent with predictive coding models where a rapid feed-forward sharing of information initializes a percept and is followed by sustained processing of a stimulus in localized regions to fine-tune the percept (Clark 2013; Yuille and Kersten 2006). This alternate viewpoint is also consistent with TMS work showing two stages of information processing: an initial domain-general process followed by a more detailed domain-specific process (Pitcher et al. 2011). It may take an accrual of computations throughout the system before domain-specific processing can occur in localized regions. Future work may apply directed connectivity approaches to better elucidate information propagating throughout the brain.

ROI choice.

The choice of ROIs can have an impact on results. We chose ROIs based on a meta-analytic database in hopes of having an independent set of ROIs that would be highly reproducible at the group level. As an alternative, we could have chosen subject-specific ROIs. Although subject-specific ROIs have been shown to be reproducible (Peelen and Downing 2005), subject-specific ROIs may not be an improvement over group-average ROIs. Previous work using a resting-state fMRI data set found that a subject’s connectivity map in one session is more related to a group-average connectivity map from a second session than that subject’s own connectivity map from a second session (Shou et al. 2014). We have found similar results with face-scene localizers, wherein the group-average task activation map (e.g., face > scene) was more predictive of a subject’s task activation map (face > scene) than that subject’s own task activation map from a different session (unpublished results). Thus group-average ROIs might in some cases be more suitable as localizers when compared with individual subject ROIs.

In summary, we find that localized representations give rise to distributed representations via sharing of information between domain-specific regions, resolving a long-standing debate in neuroscience. We speculate that domain specificity may arise from distinct computations on incoming stimuli. In this case, activity patterns to nonface stimuli in the FFA may reflect face-specific computations in addition to shared information from other regions. Future work using encoding models (Naselaris et al. 2011) and representational similarity analyses (Kriegeskorte et al. 2008) will be required to separate representations from computations. One limitation of our model is that it did not include any interactions between regions or voxels. For instance, although activity in V1 did not contribute unique category information, it is possible that activity found in conjunction with FFA and V1 does contribute category information. Although including interactions between every pair of voxels can generate models of enormous size, recent methods exist to handle the large dimensionality resulting from many interaction terms (Bien et al. 2013).

GRANTS

This work was supported by National Institute of Mental Health Grant MH-005286 to G. McCarthy.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

Z.S. and G.M. conceived and designed research; Z.S. analyzed data; Z.S. and G.M. interpreted results of experiments; Z.S. prepared figures; Z.S. and G.M. drafted manuscript; Z.S. and G.M. edited and revised manuscript; Z.S. and G.M. approved final version of manuscript.