Connectivity-based constraints on category-specificity in the ventral object processing pathway

Abstract

Recent efforts to characterize visual object representations in the ventral object processing pathway in the human brain have led to contrasting proposals about the causes of neural specificity for different categories. Here we use multivariate techniques in a novel way to relate patterns of functional connectivity to patterns of stimulus preferences. Stimulus preferences were measured throughout the ventral stream to tools, animals, faces and places; separately, we measured the strength of functional connectivity of each voxel in the ventral stream to category-preferring regions outside the ventral stream. Multivariate analyses were then performed over ventral stream voxels, relating ‘category-preferences’ to ‘functional connectivity preferences’. We show that the relation of those two measures doubly dissociates ‘tools’ and ‘places’, within what is ostensibly ‘place’ selective cortex (parahippocampal gyrus). Specifically, in the parahippocampal gyrus, functional connectivity to the left inferior parietal lobule is selectively related to stimulus preferences for tools (and not places), while functional connectivity to retrosplenial cortex is selectively related to place preferences (and not tools preferences). These findings indicate that functional connectivity can be used to index representational content rather than just provide an understanding of ‘which regions are talking to which regions’. We suggest that the connectivity of the brain is what drives category-specificity in the ventral stream, and that if this is correct, then understanding the connectivity of the ventral stream will be key to understanding the causes and function of category-specific neural organization.

Introduction

Occipital and temporal regions within the ventral object processing pathway subserve object and scene recognition (Livingstone & Hubel 1988; Ungerleider & Mishkin 1982; Goodale & Milner 1992; Peelen & Kastner 2009) and express a macroscopic organization by semantic domain (Martin, 2007; Op de Beeck et al., 2008). On the ventral surface of occipital-temporal cortex, inanimate entities and those that are ‘rooted in place’ (e.g., tools, houses, places) yield differential BOLD responses in the medial fusiform gyrus and parahippocampal gyrus, while animate entities (animals, faces) yield differential BOLD responses in the lateral fusiform gyrus (e.g., Kanwisher et al. 1997; Chao et al. 1999; Bar & Aminoff 2003; Downing 2005; Epstein & Kanwisher 1998; Konkle and Oliva, 2012). Research on neural specificity for semantic categories in the ventral stream has led to conflicting accounts of how object representations are coded. One approach, which has emphasized univariate analysis methods and the amplitude of neural activity, determines the category-specificity of a region by the types of stimuli that maximally activate that region (e.g., Kanwisher et al. 1997; Chao et al. 1999; Grill-Spector et al. 2004; Downing et al. 2005; Martin 2007). On this view, anatomically segregated subregions of the ventral stream are specialized for distinct categories (e.g., faces, places), and there are sharp categorical boundaries among object representations from some categories. Another approach, using multivariate techniques to decode stimulus preferences from distributed patterns of activity, has found that patterns of neural activity can be used to decode stimulus types even in regions that are ostensibly specialized for other categories of stimuli (Haxby et al., 2001; Rogers et al., 2005). Those findings have led to the proposal that object representations in the ventral stream are ‘graded’ and ‘widely distributed’, which is to say that there are not sharp representational boundaries in ventral stream category representations.

A separate line of work indicates a relation between the category-preferences of ventral stream regions and the connectivity of those areas to regions outside the ventral stream that exhibit congruent category-preferences. For instance, both task-based (Pessoa et al. 2006; Almeida et al. 2013; Mahon et al. 2013; Garcea & Mahon 2014) and resting functional connectivity (Zhang et al., 2009; Turk-Browne et al., 2010; Zhu et al., 2011; Simmons & Martin, 2012; Baldassano et al., 2013; He et al., 2013; Hutchison et al., 2014; O’Neil et al., 2014; Stevens, et al., 2015; Wang et al., 2015), as well as structural connectivity (Saygin et al., 2011; 2016; Mars et al., 2012; Bouhali et al., 2014; Osher et al., 2015), suggest alignment between stimulus preferences and connectivity to regions outside the ventral stream (for discussion of a connectivity-constrained account of category specificity, see Riesenhuber 2007; Martin, 2006; 2009; Mahon & Caramazza 2011; Chen & Rogers 2014; Behrmann & Plaut 2013; Mahon et al., 2007; 2009; Mahon, 2015).

As noted above, tools and places preferentially activate overlapping regions within the medial ventral stream relative to baseline categories such as animals or faces (e.g., Epstein and Kanwisher, 1998; Chao et al., 1999). Place stimuli and large and highly contextualized nonmanipulable objects consistently lead to differentially stronger BOLD responses in medial aspects of the ventral stream compared to small manipulable objects such as tools and utensils (e.g., Bar and Aminoff, 2003; Downing et al., 2005; Konkle and Oliva, 2012; Mahon et al., 2007). For that reason, the area has been dubbed the ‘parahippocampal place area.’ However, it has been observed (Hutchinson et al., 2014; Mahon et al., 2007; 2013; Nopponey et al., 2006; Almeida et al., 2013; Garcea and Mahon, 2014; Stevens et al., 2015) that regions of ventral temporal cortex that exhibit stronger BOLD responses to place stimuli than tool stimuli also show i) differential stimulus-specific repetition suppression for tools compared to non-tool manipulable objects (e.g. book) and large nonmanipulable objects (e.g. car), and ii) strong functional connectivity to action representations in the left inferior parietal lobule (Mahon et al., 2007). Those observations led us to speculate that i) there is specificity for small manipulable objects in the medial ventral stream, despite the larger amplitude responses to places in the same regions (see Chao et al., 1999), and ii) that specialization for manipulable objects is driven by inputs from parietal action representations (Mahon et al., 2007). Extrapolating that type of an account to the other categories for which there is specificity in the ventral stream has been referred to as a ‘connectivity constrained account’ of the organization of the ventral stream (e.g., Mahon & Caramazza, 2011; Reisenhuber, 2007).

Here we use a multivariate metric that relates the distributed pattern of stimulus preferences within category-preferring regions of the ventral stream to the distributed pattern of functional connectivity to category-preferring regions outside of the ventral stream. This is possible because as noted, there are well-delineated regions outside of the ventral stream that exhibit neural specificity for tools (e.g., the left inferior parietal lobule, Chao and Martin, 2000), and places (retrosplenial cortex, e.g., Bar and Aminoff, 2003, Vann et al., 2009). We test the hypothesis that the voxel-wise distribution of place preferences in the medial ventral stream will be related to the voxel-wise distribution of functional connectivity with retrosplenial cortex, while the voxel-wise distribution of tool preferences in the same region of the ventral stream will be related to the voxel-wise distribution of functional connectivity to the inferior parietal lobule. Thus, within the same region of the ventral stream (i.e., over the same pool of voxels), we evaluate whether the (voxel-wise) distribution of functional connectivity is related to stimulus preferences. A double dissociation between places and tools in the multivariate relation between functional connectivity and stimulus preferences over the same set of voxels would suggest a sharp non-spatial yet categorical boundary between object category representations in the ventral stream. It would also suggest that the paradigm of deciding the granularity of category-specificity in the ventral stream by measuring which types of stimuli elicit maximal responses misses critical information that is coded in patterns of connectivity.

Methods

Experimental Design and Data Acquisition

Participants

Twenty-four right-handed participants were tested (mean age = 20.3 yrs., standard deviation, ± 2.3 yrs; 16 female). All participants were right hand dominant (as assessed with the Edinburgh Handedness Questionnaire), had normal or corrected to normal eyesight, and had no history of neurological disorders. All guidelines and requirements of the University of Rochester Research Subjects Review Board were followed for participant recruitment and experimental procedures.

Procedure

‘A Simple Framework’ (Schwarzbach, 2011) was used to control stimulus presentation in Psych toolbox in MATLAB on a MacPro (Brainard, 1997; Pelli, 1997). Stimuli were back projected (temporal resolution = 120Hz) on a screen that participants viewed with a mirror attached to the head coil.

Each participant completed a ~60 minute scanning session. The session began with a T1 anatomical scan. The session then proceeded with (in the following order), a) a 6 minute resting functional MRI scan in which participants were instructed to keep their eyes open while fixating on a central cross against a black screen, b) 8 three minute functional runs of the category localizer experiment (see below for design and stimuli), c) another 6 minute resting functional MRI, and d) diffusion tensor imaging (15 minutes; data not analyzed herein).

Category Localizer Design and Materials

Participants were presented with intact and phase-shifted images of tools, animals, famous faces, and famous places (e.g., for details on scrambling, see Fintzi & Mahon 2013; for discussion of this functional localizer, see (Q. Chen, Garcea, & Mahon, 2015). Twelve items per category (8 exemplars per item; 384 total stimuli) were presented in mini-blocks of 6 seconds (500 ms duration, 0 ms ISI) interspersed by 6-second fixation periods. For example, for tool stimuli, twelve different tools were selected (e.g., knife, corkscrew, screwdriver, etc) and all 12 items were presented in a 6 second miniblock. For each subsequent run of the localizer, different exemplars of the same items were used. The tool and animal stimuli were matched on lexical frequency (CELEX database; tools, M = 17.54, SD = 29.34; animals, M = 29.34, SD = 37.66; t < 1), and concept familiarity (MRC psycholinguistic database; tools, M = 525.37, SD = 46; animals, M = 492.73, SD = 71.17; t(28) = 1.27, p = .21). Famous face and place stimuli were balanced for number of characters (famous faces, M = 11.92, SD = 2.72; famous places, M = 12, SD = 4.26; t < 1). In addition, the root mean square contrast was equated (normalized) across all intact and scrambled images.

Resting State fMRI

Each participant took part in two six-minute runs of a resting state experiment, in which they were instructed to keep their eyes open while fixating on a black screen (180 volumes per run).

MRI Acquisition Parameters

Whole brain BOLD imaging was conducted on a 3-Tesla Siemens MAGNETOM Trio scanner with a 32-channel head coil at the Rochester Center for Brain Imaging. High-resolution structural T1 contrast images were acquired using a magnetization prepared rapid gradient echo (MPRAGE) pulse sequence at the start of each session (TR = 2530 ms, TE = 3.44 ms flip angle = 7 degrees, FOV = 256 mm, matrix = 256 × 256, 1×1×1mm sagittal left-to-right slices). An echo-planar imaging pulse sequence was used for T2* contrast (TR = 2000 ms, TE = 30 ms, flip angle = 90 degrees, FOV = 256 mm, matrix 64 × 64, 30 sagittal left-to-right slices, voxel size = 4×4×4mm). The first 6 volumes of each run were discarded to allow for signal equilibration (4 volumes at acquisition and 2 volumes during preprocessing).

MRI Preprocessing

fMRI data were analyzed with the Brain Voyager software package (Version 2.8) and in-house scripts drawing on the BVQX toolbox for MATLAB. Preprocessing of the functional data included, in the following order, slice scan time correction (sinc interpolation), motion correction with respect to the first volume of the first functional run, and linear trend removal in the temporal domain (cutoff: 2 cycles within the run). Functional data were registered (after contrast inversion of the first volume) to high-resolution de-skulled anatomy on a participant-by-participant basis in native space. For each participant, echo-planar and anatomical volumes were transformed into standardized space (Talairach & Tournoux, 1988). No spatial smoothing was applied.

For the category-localizer the general linear model was used to fit beta estimates to the events of interest. Experimental events were convolved with a standard 2-gamma hemodynamic response function. The first derivatives of 3D motion correction from each run were added to all models as regressors of no interest to attract variance attributable to head movement.

All functional connectivity analyses were time course based and used the time series from the entire resting fMRI run. Time courses were extracted from preprocessed functional data, which had also been regressed with the outputs from motion correction (change in head position across volumes) and the global mean time-course from the whole brain. Global signal regression is known to reduce the strength of interregional correlations and can even be the source of what seem to be ‘negative’ correlations among regions (e.g., Fox et al., 2009; Murphy et al., 2009; Saad et al., 2012); however, in this study we are not interested in the interregional correlation structure per se, but rather the voxel-wise relation between the interregional correlation and task-based activation. Thus, while the principal analyses are presented with global signal regression, we repeated the key analyses without global signal regression and obtained the same pattern (See Supplemental Figure S1).

All functional connectivity was computed over the residual resting state time series (having regressed the sources of unwanted variance as described above). Whole brain functional connectivity maps were computed with a mask fit to the deskulled Talairached anatomy. Whole brain maps were computed on a run-by-run basis (two runs per subject). The run-specific r maps were then averaged (within voxel) for each subject because no significant differences (i.e., interactions with key findings) were found in any of the subsequent analyses when modeling resting-run (first, second) as a factor (see text for details). The subject-level average connectivity maps were Fisher transformed.

ROI Definition

ROI Definition: Maintaining independence of voxel selection and test

In order to maintain strict independence of voxel selection (definition of category-preferring regions) and test (measurements of category preferences) (Kriegeskorte et al., 2009), the data for each subject were split into two sets, defined by even and odd runs. ROIs expressing category-preferences, both within and outside the ventral stream, were determined for each participant using half of their data (e.g., odd runs) from the category-localizer study. Then, having defined the relevant category-preferring regions in each participant, we used the other half of the data (e.g., even runs) to quantify voxel-wise stimulus preferences. All presented results are the average of the two independent splits on the data.

ROI Definition: Seed Regions for functional connectivity

Three seed regions outside of the ventral stream were selected separately for each participant: the place-preferring left retrosplenial cortex, the tool- preferring left inferior parietal lobule and the animal-preferring right superior temporal sulcus. Place-preferring voxels within left retrosplenial cortex were identified with the contrast of (Places) > (Animals + Faces + Tools, weighted equally). Tool-preferring voxels within the left inferior parietal lobule were defined with the contrast of (Tools) > (Animals + Faces + Places, weighted equally). Animal-selective voxels within the right posterior superior sulcus were defined by the (Animals) > (Places + Tools, weighted equally). We created three spherical seed regions (5-mm radius) around the peaks in left retrosplenial cortex (average peak Talairach coordinates: −18±3.9, −58±5.8, 9±4.3), the left inferior parietal lobule (−34±7.8, −46±9.0, 47±7.5) and the right posterior superior sulcus (47±6.9, −54±9.5, 6.5±5.1).

ROI Definition: Whole Ventral Stream ROI

First we defined all ventral stream voxels separately for each participant by the contrast (intact) > (scrambled). Because the activation extended into the dorsal stream, we applied an anatomical mask to select activated voxels within Brodmann areas BA-20 (inferior temporal gyrus) and BA-37 (fusiform gyrus). All ROIs were thresholded at a FDR q < 0.05 except that for one subject a more lenient threshold was used (p < 0.05, uncorrected) and one subject was excluded because there were no activated voxels (see Table 1 for details).

Table 1.

Talairach coordinates of subject-specific peaks used to define ventral stream ROIs

ROIs	Contrast		Talairach Coordinates			Peak t-value	Size (mm2)
			x	y	z
Object-selective ventral area	Intact > Scrambled (Restricted to the mask BA20 & BA37)	L	−42 ± 4.0	−55 ± 8.4	−13 ± 5.8	9.67 ± 3.24	4276 ± 1997
		R	41 ± 6.0	−56 ± 8.8	−12 ± 6.1	10.21 ± 2.91	4672 ± 2329
Medial ventral area	Tool + Place > Animal + Face	L	−29 ± 3.5	−47 ± 10.4	−13 ± 5.9	9.09 ± 2.54	8273 ± 3489
		R	28 ± 3.6	−47 ± 10.2	−11 ± 4.3	9.67 ± 2.46	7934 ± 2517
Medial place preferring area	Place > Animal + Face	L	−27 ± 3.6	−44 ± 7.0	−10 ± 4.5	10.34 ± 3.36	7849 ± 2877
		R	28 ± 3.3	−43 ± 7.4	−10 ± 4.2	11.25 ± 3.28	8418 ± 2795
Medial tool preferring area	Tool > Animal + Face	L	−28 ± 5.1	−52 ± 8.2	−14 ± 3.5	7.04 ± 1.49	5180 ± 1880
		R	26 ± 7.6	−50 ± 9.0	−13 ± 4.1	6.29 ± 1.59	3273 ± 1577

Note: All ROIs were defined separately for each participant. L, left hemisphere; R, right hemisphere.

ROI Definition: Medial Ventral Stream ROIs (Nonliving vs. Living)

To examine the double dissociation between places and tools in closer detail, the contrast [Places + Tools] > [Animals + Faces] was used to identify the bilateral medial ventral areas exhibiting preferences for nonliving compared to living things. All ROIs were thresholded at FDR q < 0.05, or stricter for each subject (see Table 1 for details; see also Figure 3D).

Figure 3. Double dissociation between places and tools within the medial ventral ROIs.

A. The bilateral medial ventral areas that show preferences for non-living things were identified by the contrast [Tools + Places] > [Animals + Faces]. The graph demonstrates that the double dissociation between tools and places is present regardless of the baseline used to compute stimulus preferences (scrambled images, animals, or faces), and that it is present in both the left and right hemispheres. The p values for t-tests against zero were Bonferroni adjusted based on 4 observations (2 categories X 2 seeds). B. Mean functional connectivity of medial ROIs. There is stronger functional connectivity between the medial ventral stream ROI and retrosplenial cortex than between the medial ventral stream ROI and the inferior parietal lobule. C. Mean BOLD amplitude within medial ROIs. In line with prior studies, place stimuli elicit stronger BOLD responses than do other categories in the medial ventral stream ROI (medial fusiform gyrus, parahippocampal cortex, and collateral sulcus). Tools show stronger BOLD responses than animals, faces and scrambled images in the same areas. All error bars reflect one standard error of the mean across participants. D. Overlap of individual subject medial ventral ROIs.

ROI Definition: Medial Ventral Stream ROIs (Tools and Places)

In addition, we separately identified place and tool preferring areas separately. We defined place preferring areas of the ventral stream by the contrast of [Places] > [Animals + Faces] (weighted equally). The tool preferring area was defined by [Tools] > [Animals + Faces] (weighted equally). All ROIs were thresholded at FDR q < 0.05, or stricter for each subject (see Table 1 for details).

Measuring Category-Preferences and ensuring effects are robust to baseline variation

We also ensured that our measurements of category-preferences that were entered into the key analyses were not dependent on the baseline used. For the analyses in Figure 2, scrambled images were used as a baseline condition against which to compute category preferences (contrast-weighted t-values). For the analyses in Figure 3, scrambled images, animals, and faces were each used separately as baselines against which to compute the stimulus preferences for tools and places. For the analyses in Figure 4, scrambled images were used as a baseline condition to compute the stimulus preferences for tools and places. The bottom line is that core findings were not altered (at all) by the choice of baseline against which category-preferences were computed.

Figure 2. ROI analyses of voxel-wise correlation between stimulus preferences and functional connectivity within the whole ventral stream.

Triple order dissociation between tools, places, animals and faces, and connectivity to the inferior parietal lobule, retrosplenial cortex, and the superior temporal sulcus. The p values for t-tests against zero were Bonferroni adjusted based on 12 observations (4 categories X 3 seeds). All error bars reflect one standard error of the mean across participants. (* = adjusted p value < 0.05; ** = adjusted p value < 0.01; *** = adjusted p value < 0.001)

Figure 4. Double dissociation between places and tools does not depend on how subregions of ventral stream are defined.

The bilateral medial ventral areas were identified by two different contrasts: [Places] > [Animals + Faces] and [Tools] > [Animals + Faces]. The double dissociation is present regardless of how the medial ventral stream ROIs are defined. The p values for t-tests against zero were Bonferroni adjusted based on 4 observations (2 categories X 2 seeds).

MVPA Analyses

ROI Analyses

Functional connectivity was computed over resting fMRI data acquired in the same session as the category-localizer data (see above). The linear correlation between the voxelwise distribution of functional connectivity (with each seed region) and stimulus preferences was calculated for each subject-specific ventral stream ROI. The correlation coefficients (r-values) were then Fisher transformed prior to group-level statistics. Figure 1 gives an example of how we calculated the multivariate relation of functional connectivity to category preferences for 1 category (tools) and 1 seed region (inferior parietal lobule).

Figure 1. Schematic of multivariate analysis relating functional connectivity to category preferences, illustrated for 1 category (tools) and 1 seed region (inferior parietal lobule).

A. Voxel-patterns for tools, places, animals and faces are computed over ventral stream voxels. B. Time series from individual voxels in the ventral stream are correlated with mean time series from ROIs outside ventral stream (left inferior parietal ROI shown). This produces voxel patterns of functional connectivity over ventral stream voxels. Those voxel patterns are correlated with voxels patterns (Panel A) from viewing pictures of tools, places, animals & faces to test whether connectivity patterns map onto voxel-level category preferences.

Searchlight Analyses

Whole-brain searchlight analyses (see Kriegeskorte et al. 2006) were conducted relating functional connectivity to category-preferences. Searchlight analyses were conducted as follows: each subject contributed 3 whole-brain functional connectivity maps (using retrosplenial cortex, inferior parietal, and superior temporal sulcus as seeds) and four category preference maps (places, tools, animals and faces). Scrambled images were used as baselines to compute the category preference maps (e.g., [Places > Scrambled Images; Faces > Scrambled Images]. There were thus 12 possible searchlight maps that could be generated for each subject (3 connectivity maps * 4 category-preference maps). At every voxel, the contrast-weighted t-values for a given category preference for the cube of surrounding voxels (n = 27) were extracted, as well as the Fisher transformed correlation coefficients for functional connectivity to a given seed region. Over those 27 voxels the correlation between category preferences and functional connectivity was computed, and the resulting correlation coefficient (r value) was Fisher transformed and written to the central voxel. That process was carried out 12 times (i.e. 3 connectivity maps * 4 category-preference maps) for each voxel of the brain, for each subject, for all subjects. At the group level, we tested (two-tailed one sample t-test over fisher transformed r values) whether each voxel was significantly different from zero (across 24 subjects); all whole-brain results were corrected at FDR q < 0.05.

Correction for multiple comparisons

Unless otherwise noted, all contrast maps are thresholded to maintain false discovery rate at 5% (FDR q < .05). Also, unless otherwise noted, all reported p values for t-tests in ROI analyses in the main text were Bonferroni adjusted for multiple comparisons within each ROI (i.e., the p value reported in the text is equal to the original p value for a given test, multiplied by the number of tests made for that ROI). Thus, Bonferroni adjusted p-values at p ≤ .05 were considered statistically significant.

Results

We first identified regions outside of the ventral stream that exhibit category preferences for places, tools, and biological agents (faces and animals), in each participant. Those regions were retrosplenial cortex (place preferences; e.g., Bar & Aminoff 2003; Epstein et al. 1999), the left inferior parietal lobule (tool preferences; e.g., Chao & Martin 2000; Mahon et al., 2007), and the right superior temporal sulcus (preferences for biological agents, faces and animals; e.g., Grossman et al. 2000; Beauchamp et al. 2002). Three whole-brain functional connectivity maps were generated for each participant over resting fMRI data, using the three regions outside of the ventral stream as seeds. This yielded, for each voxel in the ventral stream, measures of functional connectivity to category-preferring regions outside of the ventral stream. Separately, and always using independent data (Kriegeskorte et al., 2009), stimulus preferences for faces, places, tools and animals were measured for voxels in the ventral stream. We then related the multivoxel pattern of functional connectivity to the multivoxel pattern of category preferences. This approach was carried out using several ROI-based approaches to constrain the voxels in the ventral stream for which multivariate analyses were performed, and using multiple baselines against which to measure the voxel-wise strength of category-preferences, and was also carried out for the whole brain using the searchlight approach (Kriegeskorte et al., 2006).

ROI Analyses

Relation of functional connectivity to category preferences for the whole ventral stream

As noted, prior work indicates that category-preferring regions in the ventral stream exhibit privileged functional connectivity with regions outside of the ventral stream that show congruent category-preferences (Hutchison et al., 2014; O’Neil et al., 2014; Simmons & Martin, 2012; Stevens et al., 2015; Turk-Browne et al., 2010; Zhang et al., 2009; Zhu et al., 2011). It follows that there will be a positive relationship between connectivity and stimulus preferences when considering all voxels in the ventral stream. Thus, in a first step toward validating our approach, we replicated the previously reported alignment between functional connectivity and category-preferences. To that end, the ventral stream ROI was defined as the intersection between Brodmann areas BA-20 (inferior temporal gyrus) and BA-37 (fusiform gyrus), and all voxels that showed differential activity when viewing intact objects compared to scrambled objects (see Methods for details). The voxel vectors representing stimulus preferences (places, tools, animals and faces, each compared to scrambled images) were correlated with the voxel vectors representing functional connectivity (to retrosplenial cortex, inferior parietal lobule, and posterior superior temporal sulcus), yielding 12 correlation values per subject (4 stimulus types by 3 seeds for the functional connectivity analysis; see Methods for full details).

The correlation values were analyzed in repeated-measures ANOVAs with category (place, tool, animal, face) and seed (left retrosplenial cortex, left inferior parietal, right superior temporal sulcus) as factors. In the bilateral ‘whole ventral stream’ ROIs, there were significant interactions between category and seed (Left: F(6, 132) = 24.49, p < .001; Right: F(6, 132) = 23.92, p < .001). As shown in Figure 2, there was a triple-order dissociation between category preferences in the ventral stream and connectivity to category-preferring areas outside of the ventral stream. Post-hoc tests (all Bonferroni adjusted) indicated that place preferences were selectively related to connectivity with retrosplenial cortex (Left: t(22) = 3.93, adjusted p = .009; Right: t(22) = 4.43, adjusted p = .003), tool preferences were selectively related to connectivity with the inferior parietal lobule (Left: t(22) = 9.10, adjusted p < .001; Right: t(22) = 7.56, adjusted p < .001), and animal and face preferences were selectively related to connectivity with the superior temporal sulcus (Animals: Left: t(22) = 4.44, adjusted p = .003; Right: t(22) = 7.16, adjusted p < .001; Faces: Left: t(22) = 3.59, adjusted p = .02; Right: t(22) = 3.55, adjusted p = .02). All other comparisons were either negative or not significantly different from 0. The presence of this triple order dissociation, in conjunction with selective effects, indicates that this multivariate approach is sensitive to the alignment between category-preferences in the ventral stream and functional connectivity to regions outside of the ventral stream; this is essentially a replication within a multivariate framework what past research has done with an ROI-based and univariate framework (e.g., Stevens et al. 2015).

Double dissociation between places and tools within parahippocampal cortex

As noted above, medial aspects of the ventral stream exhibit their strongest neural responses to place stimuli, but also more neural activity to small manipulable objects than to living things (e.g., Chao et al. 1999; Noppeney 2005; Mahon et al. 2007). The key question is therefore whether the multivariate relation between functional connectivity and stimulus preferences doubly dissociates places and tools within the medial subregion of the ventral stream that has been argued to exhibit place specificity (the ‘parahippocampal place area’). We tested this across a series of ROI analyses in order to show that basic finding was not dependent on either the criteria used to define the ROI, or the baseline against which stimuli were compared to measure category-preferences.

We first defined the ROI for the medial ventral stream without introducing a bias as to whether those voxels tended to exhibit stronger responses for places or tools. The medial ventral ROI was defined by comparing all nonliving stimuli to all living stimuli (i.e., [Places + Tool] > [Animals + Faces]). Half of the data for each participant were used to define category-preferring cortex, and the other half of the data were used to measure category preferences voxel-wise within the ROI. Scrambled images were used as a baseline condition to compute stimulus preferences. A repeated-measures ANOVA with category (place, tool) and seed (left retrosplenial cortex, left inferior parietal lobule) as factors revealed a significant interaction between category and seed (Left: F(1, 23) = 85.26, p < .001; Right: F(1, 23) = 28.95, p < .001). Post-hoc tests using Bonferroni adjusted p-values were carried out (based on 4 comparisons, 2 categories x 2 seeds). There was a positive relation between place preferences and functional connectivity to retrosplenial cortex (Left: t(23) = 3.94, adjusted p = .002; Right: t(23) = 2.82, adjusted p = .04; see Figure 3A, left), but no relation between connectivity to retrosplenial cortex and preferences for tools. In contrast, in the same medial ROI, there was a positive voxel-wise correlation between tool preferences and functional connectivity to the inferior parietal lobule (Left: t(23) = 5.96, adjusted p < .001; Right: t(23) = 5.30, adjusted p < .001; see Figure 3A, left), and no relation between connectivity to the inferior parietal lobule and place preferences.

We also carried out paired t-tests to directly contrast place versus tool effects. The correlation (fisher transformed) between connectivity to retrosplenial cortex and place preferences was greater than the correlation between connectivity to retrosplenial cortex and tool preferences (Left: t(23) = 7.17, p < .001; Right: t(23) = 4.39, p < .001; see Figure 3A, left). On the other hand, the correlation between connectivity to the inferior parietal lobule and tool preferences was greater than the correlation between connectivity to the inferior parietal lobule and place preferences (Left: t(23) = 7.01, p < .001; Right: t(23) = 4.26, p < .001; see Figure 3A, left). There was thus a double dissociation within the medial ventral stream ROI whereby place preferences were selectively related to connectivity to retrosplenial cortex, and tool preferences were selectively related to connectivity with the inferior parietal lobule.

In a second series of analyses for this same ROI, we ensured that the double dissociation in the medial ROI did not depend on the type of baseline used to compute stimulus preferences; thus, rather than measuring tool and place preferences by contrasting with scrambled images, we used animals and faces as baselines. The results, shown in Figure 3A, indicated that the double dissociation was present in both the left and right hemispheres when tool- and place-preferences were computed relative to animals or faces. A repeated-measures ANOVA with category (place, tool) and seed (left retrosplenial cortex, left inferior parietal lobule) revealed a significant interaction between category and seed (animals as baseline: Left: F(1, 23) = 96.82, p < .001; Right: F(1, 23) = 39.58, p < .001; faces as baseline: Left: F(1, 23) = 105.50, p < .001; Right: F(1, 23) = 31.35, p < .001). Post-hoc tests (Bonferroni adjusted) indicated there was a positive voxel-wise correlation between place preferences and functional connectivity to retrosplenial cortex (animals as baseline, Figure 3A, middle panel: Left: t(23) = 6.55, adjusted p < .001; Right: t(23) = 3.17, adjusted p = .015; faces as baseline, Figure 3A, right panel: Left: t(23) = 5.73, adjusted p < .001; Right t(23) = 3.32, adjusted p = .01). In contrast, there was a positive voxel-wise correlation between tool preferences and functional connectivity to the inferior parietal lobule (animals as baseline, Figure 3A, middle panel: Left: t(23) = 3.65, adjusted p = .005; Right: t(23) = 2.87, adjusted p = .04; faces as baseline, Figure 3A, right panel: Left: t(23) = 6.30, adjusted p < .001; Right: t(23) = 2.92, adjusted p = .03). All other comparisons are either negative or non-significantly different from 0.

Paired t-tests showed that the correlation between connectivity to retrosplenial cortex and place preferences was greater than the correlation between connectivity to retrosplenial cortex and tool preferences (animals as baseline, Figure 3A, middle panel: Left: t(23) = 8.20, p < .001; Right: t(23) = 5.31, p < .001; faces as baseline, Figure 3A, right panel: Left: t(23) = 7.95, p < .001; Right t(23) = 4.78, p < .001). Paired t-tests also showed that the correlation between connectivity to the left inferior parietal lobule and tool preferences was greater than the correlation between connectivity to the left inferior parietal lobule and place preferences (animals as baseline, Figure 3A, middle panel: Left: t(23) = 6.80, p < .001; Right: t(23) = 4.50, p < .001; faces as baseline, Figure 3A, right panel: Left: t(23) = 7.59, p < .001; Right t(23) = 4.13, p < .001).

The double dissociation between tools and places in the multivoxel code relating functional connectivity to stimulus preferences cannot be explained by univariate analyses of either functional connectivity or stimulus preferences. As shown in Figure 3, the same medial ROIs exhibited overall (i.e., averaging over all voxels) stronger BOLD responses to place than to tool stimuli (Figure 3C), and overall (averaging over all voxels) stronger functional connectivity to retrosplenial cortex than to the inferior parietal lobule (Figure 3B). Those data underline the key finding: there is information contained in the voxel-wise relationship between functional connectivity and stimulus-evoked preferences that is not present in the aggregate data for stimulus preferences or functional connectivity, when either is considered alone.

Double-dissociation between places and tools does not depend on ROI Definition

As a final ROI-based test, and to further ensure that the double dissociation between places and tools was not due to anatomically segregated subregions within the medial ROI that were biased toward either places or tools, we identified the medial ventral ROI by place and tool preferences separately. The results confirmed the double dissociation (see Figure 4).

• When medial ROIs were defined by place preferences (places > animals + faces), a repeated-measures ANOVA with category (place, tool) and seed (left retrosplenial cortex, left inferior parietal lobule) revealed a significant interaction between category and seed (Left: F(1, 23) = 51.95, p < .001; Right: F(1, 23) = 31.75, p < .001). Post-hoc tests (Bonferroni adjusted) indicated there was a positive multivariate relation between place preferences and functional connectivity to retrosplenial cortex (Left: t(23) = 2.77, adjusted p = .04; Right: t(23) = 2.74, adjusted p = .05), and a positive relation between tool preferences and functional connectivity to the inferior parietal lobule (Left: t(23) = 4.80, adjusted p < .001; Right: t(23) = 3.91, adjusted p = .002; all other comparisons were either negative or non-significantly different from 0). Paired t-tests revealed a stronger correlation between connectivity to retrosplenial cortex and place preferences than the correlation between connectivity to retrosplenial cortex and tool preferences (Left: t(23) = 6.65, p < .001; Right: t(23) = 5.36, p < .001; see Figure 4, left panel). On the other hand, the correlation between connectivity to the inferior parietal lobule and tool preferences was significantly greater than the correlation between connectivity to the inferior parietal lobule and place preferences (Left: t(23) = 5.31, p < .001; Right: t(23) = 3.47, p < .001; see Figure 4, left).

• When medial ROIs are defined by tool preferences (tools > animals + faces), a repeated-measures ANOVA with category (place, tool) and seed (left retrosplenial cortex, left inferior parietal lobule) revealed a significant interaction between category and seed (Left: F(1, 23) = 45.16, p < .001; Right: F(1, 23) = 18.82, p < .001). Post-hoc tests (Bonferroni adjusted) showed there was a relation between place preferences and functional connectivity to retrosplenial cortex (Left: t(23) = 4.71, adjusted p < .001; Right: t(23) = 3.08, adjusted p = .02), and a relation between tool preferences and functional connectivity to the inferior parietal lobule (Left: t(23) = 4.79, adjusted p < .001; Right: t(23) = 4.83, adjusted p < .001; all other comparisons were either negative or non-significantly different from 0). Paired t-tests revealed a stronger correlation between connectivity to retrosplenial cortex and place preferences than the correlation between connectivity to retrosplenial cortex and tool preferences (Left: t(23) = 5.75, p < .001; Right: t(23) = 2.77, p < .01; see Figure 4, right panel). On the other hand, the correlation between connectivity to the inferior parietal lobule and tool preferences was greater than the correlation between connectivity to the inferior parietal lobule and place preferences (Left: t(23) = 5.32, p < .001; Right: t(23) = 4.47, p < .001; see Figure 4, right panel).

Summary of ROI Analyses

These ROI analyses collectively show that there is a double dissociation between places and tools over the same pool of voxels in the medial ventral stream. That double dissociation does not depend on how stimulus preferences are computed (relative to a scrambled image baseline, faces or animals), nor does it depend on how the subset of ventral stream voxels is selected (i.e., based on differential responses to tools and places together relative to biological agents, or either tools or places considered separately). There were also negligible differences between the left and right hemispheres. To explicitly test for hemispheric asymmetries, we conducted repeated measures ANOVAs with the factors Hemisphere (i.e, levels, left or right hemisphere), seeds (three levels, Retrosplenial Cortex, Inferior Parietal Lobule, Superior Temporal Sulcus), and category (2 levels, Tools and Places), for each ROI described above. There was no main effect of Hemisphere, and no interaction between Hemispheres and other factors, for all ROIs (all p values > 0.1).

Finally, we sought to ensure that the selective multivariate relation between functional connectivity and stimulus preferences was present when using resting data to compute connectivity that were acquired prior to participants having viewed any stimuli in the category localizer (see Methods for details). This was possible because two resting runs were collected in each participant—one before the category-localizer experiment and one after the category-localizer experiment. Repeated measures ANOVAs with the factors Run (i.e, two levels, resting scan before or after category-localizer), seeds (three levels), and category (2 levels) tested, for each ROI, whether any observed effects were modulated according to whether the resting run came before or after the category-localizer. The main effect of Run, and the interactions between Run and Category were non-significant, for all ROIs (all p values > 0.2). This indicates that the double-dissociation between tools and places in the medial ventral stream is not influenced by whether functional connectivity is computed on resting data acquired prior to, or after, when participants viewed the stimuli that were used to measure category-preferences.

Whole-brain searchlight analyses

In an independent set of analyses, the multivoxel relations between functional connectivity and stimulus preferences were computed throughout the brain using a searchlight approach (searchlight volume = 27 voxels, 729 mm³; Kriegeskorte et al. 2006). Each searchlight analysis mapped the correlation between functional connectivity and category preferences. There were a total of 12 searchlight analyses conducted for each subject, representing the factorial combination of connectivity maps from three seed regions (left retrosplenial cortex, left inferior parietal, and right posterior superior temporal sulcus) and four stimulus preference maps (places, tools, animals and faces, each compared to the scrambled image baseline). 12 group-level maps were computed by performing one-sample t-tests, within each voxel, over the fisher-transformed subject-level searchlight correlation maps. The results for the ventral surface of occipital-temporal cortex are shown in Figure 5 (see Supplemental Figures S2–S4 for whole brain images). As can be seen in Figure 5, the searchlight analyses confirmed the regional specificity observed with the ROI analyses: a) the searchlight analysis relating place preferences to connectivity with retrosplenial cortex identified a large swath of the medial aspect of the ventral stream (Figure 5, top row, left panel); b) the searchlight analysis relating tool preferences to connectivity with the left inferior parietal lobule identified the medial aspect of the ventral stream, as well as lateral temporal cortex in the vicinity of the posterior middle temporal gyrus (Figure 5, the second row, middle panel), while c) the searchlight analysis relating animal and face preferences to connectivity with the superior temporal sulcus identified the bilateral lateral fusiform gyrus (Figure 5, the third and last row, right panel).

Figure 5. Whole-brain searchlight correlation between stimulus preferences and functional connectivity.

Top: voxels (blue) showing positive voxel-wise correlations between place preferences and functional connectivity to the seed regions. Second row: voxels (green) showing positive voxel-wise correlations between tool preferences and functional connectivity to the seed regions. Third row: voxels (red) showing positive voxel-wise correlations between animal preferences and functional connectivity to the seed regions. Bottom: voxels (pink) showing positive voxel-wise correlations between face preferences and functional connectivity to the seed regions. Scrambled images were used as baseline to compute stimulus preferences. These whole brain analyses confirm the triple order dissociation, whereby tool preferences are selectively related to connectivity with the inferior parietal lobule, place preferences are selectively related to connectivity with retrosplenial cortex, and animal and face preferences are related to connectivity with the superior temporal sulcus. Voxels preferring inanimate entities defined by the contrast [places + tools] > [animals + faces] are highlighted by the blue outline. Voxels preferring animate entities identified by the contrast [animals + faces] > [places + tools] are outlined in yellow. All maps thresholded at FDR q < 0.05.

The key finding that emerges from the searchlight analyses is that the double dissociation between places and tools is present for a common set of voxels within the medial ventral stream—and this emerges in an unconstrained (i.e., whole brain) analysis. This is represented by the red and orange areas in Figure 6, which represents voxels for which there were relations, both between place preferences and functional connectivity to retrosplenial cortex, as well as a relation between tool preferences and functional connectivity to the left inferior parietal lobule. As shown in Figure 6, in the group map the size of the overlapping area is 2307 mm³ for left hemisphere and no overlap was found for right hemisphere when the searchlight maps were thresholded at FDR q < .05. At a more lenient threshold (p < .05, uncorrected), the size is 9597 mm³ for left hemisphere and 3665 mm³ for right hemisphere. It’s important to note that the sizes of the overlapping regions were much greater than the searchlight volume we used (729 mm³), so they could not simply be artifacts that arise due to adjacent but non-overlapping regions being blurred by the inherent spatial smoothing imposed by a searchlight analysis. Representative single subject searchlight maps (see Figure 6) are also shown to confirm that the presence of the double dissociation over the same set of voxels is not an artifact of group averaging. These whole-brain analyses thus provide independent confirmation of a sharp, and doubly dissociable, categorical boundary over the same set of voxels between tools and places.

Figure 6. Double dissociation between places and tools is present for overlapping subregions.

Overlap of whole-brain searchlight maps for places and tools. Voxels (blue) show positive voxel-wise correlations between place preferences and functional connectivity to the left retrosplenial (L-RSC) seed regions. Voxels (green) show positive voxel-wise correlations between tool preferences and functional connectivity to the left inferior parietal lobule (L-IPL) seed regions. Voxels (red and orange) are the overlap between the blue and green regions. Place preferring voxels defined by the contrast [places] > [animals + faces] are highlighted by a thick black outline. Tool preferring voxels identified by the contrast [tools] > [animals + faces] are outlined in yellow. All results are thresholded at p < 0.05, except for the orange voxels (thresholded at FDR q < 0.05).

It’s interesting to note that the searchlight analysis relating tool preferences to connectivity with the left inferior parietal lobule not only identified the medial aspect of the ventral stream, but also lateral temporal cortex (mainly the posterior middle temporal gyrus) which is thought to play an important role in naming and representing lexical semantic knowledge pertaining to tools (for reviews, see Lewis 2006; Martin, 2007). In addition, since we used an FDR approach to manage false positives, different minimum t values corresponded to FDR q < .05 across different maps. This was particularly the case for the maps relating place preferences to connectivity to retrosplenial cortex, and tool preferences to connectivity to the inferior parietal lobule; for those two maps, thresholds were lower than in the other two “off diagonal” maps (i.e., place references related to connectivity with the inferior parietal lobule, and tool preferences related to connectivity with the inferior parietal lobule). One may ask what the patterns look like when the minimum t value is held constant. In Supplemental Figure S5, we show that the double dissociation between places and tools is still present when the minimum t value is held constant for the searchlight analyses.

Discussion

We used a novel application of multivoxel pattern analysis to relate stimulus preferences to resting functional connectivity in the ventral object-processing stream. The most striking finding is that within a subregion of the ventral that is ostensibly ‘category-specific’, the so-called ‘parahippocampal place area’, there is equivalent evidence for tool specificity as there is for place specificity when specificity is measured as the multivariate relation of functional connectivity and stimulus preferences. Specifically, functional connectivity to the inferior parietal lobule is related to tool (but not place preferences) while functional connectivity to retrosplenial cortex is related to place (but not tool) preferences. That double dissociation cannot be explained by univariate analyses of stimulus preferences or functional connectivity considered alone: places elicit larger neural responses than do tools (hence the moniker ‘parahippocampal place area’), and there is greater connectivity between retrosplenial cortex and the medial ventral ROI than between the inferior parietal lobule and the medial ventral ROI. These findings suggest that relying on overall amplitude of response in order to adjudicate the ‘category’ for which a region exhibits specificity may be misleading.

It is important to emphasize that the double dissociation between ‘tools’ and ‘places’ is local to a subregion of the ventral stream defined on independent grounds as exhibiting category-specificity. Prior work has demonstrated large-scale alignments between functional connectivity and stimulus preferences (e.g., Stevens et al. 2015; Hutchison et al. 2014), whereby regions that exhibit category-specificity in the ventral stream also express privileged functional connectivity to regions outside the ventral stream that exhibit corresponding category preferences. That basic finding (Stevens et al., 2015; Hutchinson et al, 2014) was replicated in our first analysis in which we examined the alignment of category preferences and functional connectivity over all ventral stream voxels. What is novel to the current report is a double dissociation in the voxel-wise relation of category-preferences and functional connectivity between ‘tools’ and ‘places’ over the same pool of voxels.

It is also important to emphasize that, because functional connectivity and stimulus preferences were computed over entirely independent datasets, these findings cannot be an artifact of voxel selection (Kriegeskorte et al., 2009), nor can they be a result of activation spreading between ventral stream regions and seed regions outside the ventral stream during stimulus processing. These aspects of our analytic approach mean that the dissociation between places and tools is a reflection of the intrinsic organization of high-level visual cortex, and is not caused by how the brain responds in a given experimental paradigm.

Clues about the constraints that shape the organization of the ventral stream

Broadly speaking, there are two dimensions along which one might separate extant accounts of the nature and causes of neural specificity for different semantic categories in the ventral stream. On the one hand, theories can be distinguished between those that emphasize the specificity of object category representations and argue for sharp topographical boundaries among categories (Chao et al., 1999; Downing et al., 2005; Grill-Spector et al., 2004; Kanwisher et al., 1997; Martin, 2007) and those that emphasize the graded nature of object-representations (e.g., Haxby et al. 2001; Rogers et al. 2005). Our findings are clearly in line with the former set of proposals, in that we observe a ‘sharp’ boundary between tools and places, as expressed in the multivariate relation of stimulus preferences and functional connectivity. What is novel is that we have used a multivoxel approach to demonstrate this sharp boundary, while most prior studies using a multivoxel approach have argued in the tradition of ‘distributed and overlapping’ representations rather than sharp boundaries. Stated differently, we have found a sharp boundary in representational space between tools and places that does not correspond to a sharp topographical boundary.

A second dimension along which theories of the causes of category-specific in the ventral stream can be organized is between those that assume category-specificity arises due to statistical regularities in the visual input interacting with various (non-categorical) biases in the ventral stream (e.g., Levy et al. 2001; Hasson et al., 2002; 2003; Konkle and Oliva, 2012) and those that assume that category-specificity arises due to intrinsic factors of brain organization that are largely independent of experience (e.g., Riesenhuber 2007; Martin 2009; Mahon & Caramazza 2011; Chen & Rogers 2014; Behrmann & Plaut 2013). The most developed account within the former argues that biases for foveal or eccentric visual processing are inherited from early visual cortex by high level object recognition areas, and that categories ‘come to be represented are placed’ in those regions of high level cortex that have the appropriate inputs. Thus, faces (the so-called fusiform face area) are located in a subregion of the ventral stream that differentially receives foveal input, while places (the so-called parahippocampal place area) are located in a subregion of the ventral stream that differentially receives input from the visual periphery (e.g., Levy et al., 2001; Hasson et al., 2002). The idea that dimensions of organization that have nothing to do with category per se interact with statistical regularities in experience has recently gained momentum (e.g., Baker et al., 2007; Srihasam et al., 2014). One difficulty faced by proposals that assume a critical role for visual experience in driving category-specificity is that the broad organization by semantic domain in the ventral stream is present even in individuals with no visual experience (Striem-Amit et al., 2012; Mahon et al., 2009; Buchel et al., 1998; He et al., 2013).

In contrast to theories that emphasize the role of statistical regularities in visual experience in shaping category-specificity are so-called ‘connectivity constrained accounts.’ For instance, by hypothesis, connectivity between the medial fusiform gyrus and parietal somatosensory and action systems could bias that subregion of the ventral stream to represent the surface properties and texture of objects (see Cant et al., 2007; 2009). Tools and other manipulable objects are an ‘eccentric’ class of things in the world for which both surface texture information (relevant for grasping) and action information (relevant for functional use) must be brought into register with visual form. For that reason, the underlying neural code in that region relates tool-specificity to connectivity to parietal action systems (Mahon et al., 2007; for evidence and discussion, see Almeida et al., 2013; Chen and Rogers, 2014; Gallivan et al., 2013; Hutchison et al., 2014; Mahon et al., 2009; 2013; Osher et al., 2015; Stevens et al., 2015).

Largely on the basis of neuropsychological data indicating there can be a widespread collapse of knowledge pertaining to a given category of items (for review see Mahon and Caramazza, 2009; 2011), as well as the fact that category-specificity does not depend on visual experience, we and others proposed that the basic scaffolding by semantic category derives from intrinsic (i.e., innate) patterns of structural connectivity between the ventral stream and regions outside of the ventral stream (Mahon et al., 2007; 2009; Mahon and Caramazza, 2011; Martin, 2006). More recently, Kanwisher and colleagues (Saygin et al., 2016; Osher et al., 2015) have presented compelling evidence that there is a high degree of topographical alignment between patterns of structural connectivity and patterns of category preferences. Another line of work by Peelen and colleagues (Bracci et al., 2012) has shown a tight coupling between neural specificity for hands and tools in lateral posterior temporal cortex and functional connectivity with somatosensory cortex. Taken together, we would suggest that a connectivity constrained account of the origin of category-specificity in the ventral stream is able to explain a set of facts that would otherwise seem disconnected on an account that prioritized visual experience as the principal source of organizational constraints that lead to biases by semantic domain:

1. The basic organization by semantic category in the ventral stream is observed for the same categories of stimuli and in the same anatomical locations across sighted individuals, and in the same locations in congenitally blind individuals (Striem-Amit et al., 2012; Mahon et al., 2009; Buchel et al., 1998; He et al., 2013).

2. There is neural specificity for printed words, a category with no evolutionary history, in an anatomically consistent location in the ventral stream. The fact that printed words have no evolutionary history reduces the likelihood that there could be visual features (i.e., visual content) that is innately coded in the visual system that matches printed words. It would seem more likely that there is a subregion of high-level visual cortex that is innately connected to the language system, and that in turn drives the anatomical location of the visual word form area (Martin, 2006; Dehaene et al., 2005; Saygin et al., 2016).

3. Structural (Saygin et al., 2011; 2016; Osher et al., 2015) and functional connectivity (Stevens et al., 2015; Hutchinson et al., 2014; Garcea and Mahon, 2014; Mahon et al., 2007; 2013) is related to regional specificity for high-level visual categories.

4. Regions of the ventral stream that exhibit neural specificity for small manipulable objects also exhibit privileged functional connectivity to the left inferior parietal lobule, which represents complex object-directed actions (Almeida et al., 2013; Garcea and Mahon, 2014; Mahon et al., 2007; 2013; Stevens et al., 2015; Hutchinson et al., 2014).

5. There is independent evidence that variance in both behavioral and anatomical location of category-specific processing has a genetic basis (Wilmer et al., 2010; Polk et al., 2007; Zhu et al., 2010).

6. Subsequent to brain damage, some patients can exhibit a widespread collapse of knowledge for the impaired category (category-specific semantic deficits), suggesting that category-specific knowledge impairments are not due only to the lesion itself, but to the lesion ‘bringing down’ a whole network of regions (e.g., for discussion see Mahon, 2015).

The findings that we have reported herein are not direct evidence for a theory of the causes of category-specificity in the ventral stream; nonetheless, when considered in the context of the wider array of findings summarized above, our findings are strongly aligned with the idea that neural specificity for categories in the ventral stream is driven by patterns of connectivity between ventral stream regions and category-preferring regions outside the ventral stream. It will be important to investigate with future work whether other measures of neural specificity, such as repetition suppression, show the same pattern of alignment between connectivity and amplitude (or reduction in amplitude) of response that we have reported. For instance, based on prior work (Mahon et al., 2007) we know that stimulus specific repetition suppression in the ventral stream tracks action-relevant properties of manipulable objects. It would be interesting to therefore test whether voxels in the ventral stream exhibiting differential or selective repetition suppression for tool stimuli are just those voxels that express the strongest functional connectivity to the left inferior parietal lobule. Another important direction is to evaluate the alignment between structural connectivity of ventral stream to regions outside of the ventral stream and stimulus preferences (Saygin et al., 2016). Finally, critical insights on the functional role of such connections would be provided by careful study of neural responses in the ventral stream as a function of brain lesions outside the ventral stream (e.g., Price et al., 2001).

In summary, we suggest that the pressures that shape the organization of the ventral visual pathway go beyond those that are exerted by the computational demands of vision, and include the need for ventral stream regions to interact with regions outside of the ventral stream. On this view, the macroscopic organization of the ventral stream by semantic domain (Martin, 2007; Op de Beeck et al., 2008) and the regional alignment of category-preferences with functional connectivity (Hutchison et al., 2014; Stevens et al., 2015), may be two reflections of a common underlying neural code. That neural code is the common currency that, by hypothesis, relates category-specificity to the intrinsic connectivity of the ventral stream with other brain regions. This motivates moving beyond modeling representational similarity in terms of the properties or categories of the visual objects themselves, and toward an approach where the connectivity of a neural region is considered a constitutive part of that region’s representational content.

Supplementary Material

supplement

Supplemental Figure S1. Double dissociation between places and tools within the medial ventral ROIs without Global signal regression.

The bilateral medial ventral areas that show preferences for non-living things were identified by the contrast [Tools + Places] > [Animals + Faces]. The graph demonstrates that the double dissociation between tools and places is present without Global signal regression. The p values for t-tests against zero were Bonferroni adjusted based on 4 observations (2 categories X 2 seeds). All error bars reflect one standard error of the mean across participants.

Supplemental Figure S2. Searchlight analyses of voxel-wise correlation between stimulus preferences and functional connectivity to the left left inferior parietal lobe (L-IPL) seed (see also Figure 5).

Supplemental Figure S3. Searchlight analyses of voxel-wise correlation between stimulus preferences and functional connectivity to the left retrosplenial cortex (L-RSC) seed (see also Figure 5).

Supplemental Figure S4. Searchlight analyses of voxel-wise correlation between stimulus preferences and functional connectivity to the right superior temporal sulcus (R-STS) seed (see also Figure 5).

Supplemental Figure S5. Double dissociation between places and tools is present when the minimum t value is held constant for the searchlight analyses.

Top: voxels (blue) showing positive voxel-wise correlations between place preferences and functional connectivity to the seed regions. Bottom: voxels (green) showing positive voxel-wise correlations between tool preferences and functional connectivity to the seed regions. All maps thresholded at p < 0.005, uncorrected. When the minimum t value is held constant, the whole brain analyses confirm the double dissociation, whereby place preferences are selectively related to connectivity with retrosplenial cortex, tool preferences are selectively related to connectivity with the inferior parietal lobule.

Highlights.

1. There is a multivariate relation between stimulus preferences and resting functional connectivity in the ventral object-processing stream

2. Place preferences are selectively related to connectivity to retrosplenial cortex while mtool preferences are selectively related to connectivity with the inferior parietal lobule

3. Connectivity shapes the organization of the ventral stream

4. Non-spatial representational boundaries within a subregion of the ventral stream between semantic categories