Higher visual areas act like domain-general filters with strong selectivity and functional specialization

Meenakshi Khosla; Leila Wehbe

doi:10.1038/s41467-026-73938-9

. 2026 Jun 12;17:7484. doi: 10.1038/s41467-026-73938-9

Higher visual areas act like domain-general filters with strong selectivity and functional specialization

Meenakshi Khosla ^1,², Leila Wehbe ^3,^4,^✉

PMCID: PMC13408694 PMID: 42285966

Abstract

Neuroscientific studies rely heavily on a-priori hypotheses, which can bias results toward existing theories. Here, we use a hypothesis-neutral approach to study category selectivity in higher visual cortex. Using only stimulus images and their associated fMRI activity, we constrain randomly initialized neural networks to predict voxel activity. Despite no category-level supervision, units in the trained networks act as detectors for semantic concepts like ‘faces’ or ‘words’, providing solid empirical support for categorical selectivity. Importantly, this selectivity is mostly maintained when training the networks without images that contain the preferred category, strongly suggesting that selectivity is not domain-specific machinery, but sensitivity to generic patterns that characterize preferred categories. The ability of the models’ representations to transfer to perceptual tasks further reveals the functional role of their selective responses. Finally, our models show selectivity only for a limited number of categories, all previously identified, suggesting that the essential categories are already known.

Subject terms: Neural encoding, Perception

Using a hypothesis-neutral modeling approach, this study shows that brain-optimized neural networks spontaneously develop selectivity for high-level visual categories, revealing the computational and functional roles of higher visual areas.

Introduction

Primary visual cortex studies have exploited an approach that presents model organisms with abstract stimuli like oriented edges or sine-wave gratings and examines the stimuli-evoked neuronal responses^1,2. This approach has revealed that neurons in the visual cortex are selective to particular aspects of our visual world, thereby informing our understanding of the selectivities in the early visual system. However, detailed characterization of neuronal responses in higher-order visual areas has been more elusive due to a circular difficulty. Understanding what an area encodes requires presenting the optimal stimulus, but the selectivity of the visual area remains hidden until the optimal pattern is presented³.

Typically, past studies used an a priori hypothesis-driven approach. This deductive approach hypothesizes the stimulus attributes represented in a region and then designs experiments with carefully selected stimuli. It has identified several category-selective regions within the human ventral temporal cortex and macaque inferotemporal cortex, including regions responding selectively to faces^4–6, places^7–9, bodies^6,10–12, tools¹³, words^14,15, and other categories¹⁶. However, this approach is not exhaustive or scalable; response properties within large swathes of the sensory cortex remain elusive.

An alternative approach uses naturalistic images and videos to drive activity in the visual cortex and then encoding models to test hypotheses about voxel-level tuning^17–20. This inductive approach decouples data collection from hypothesis testing, as multiple candidate models are tested post hoc on the same dataset. Each model is constructed to test a given hypothesis and each hypothesis is formulated in terms of explicit quantitative functional forms, e.g., low-level gabor wavelet pyramid, motion-energy pyramid, semantic encoding models etc.

Recently, representations from deep neural networks have been used to construct encoding models for neural responses along the ventral visual pathway in humans and non-human primates^21–28. Optimizing for tasks like object recognition can lead to representations that accurately predict activity in the ventral visual pathway, and different tasks can align with specific brain regions²⁹. This suggests that these artificial and biological networks could share computational goals, offering a new way to test computational hypotheses about the brain. Under this framework, the task that a network is trained on determines the hypothesis.

All the approaches above rely on specific underlying hypotheses. In the deductive approach, the hypothesis governs the selection of the narrow range of stimuli. In the inductive system identification approach, hypotheses are specified via an encoding model. When an encoding model uses representations from task-optimized deep neural networks, the hypothesis is specified by the network task. In contrast, a hypothesis-neutral modeling approach utilizes fewer a priori assumptions, so the resulting models are more impartial, flexible and effective in revealing tuning properties throughout the visual system.

One such approach is response optimization, i.e., fitting model parameters to reproduce the brain responses related to stimuli directly. When successful in predicting new responses and generalizing to unseen contexts, the response-optimization approach can facilitate the discovery of unknown neuronal tuning properties or provide strong tests for existing theories. To date, the amount of data available to fit response-optimized models was often insufficient. However, recent advances in large-scale data collection, where the stimuli are not tethered to any hypothesis, present an opportunity to revisit this approach. Indeed, several recent studies have successfully built response-optimized models of early visual cortex^30–33.

However, we still do not know how such an approach generalizes to higher-order visual areas and what it might reveal about them. Thus, we adopt a hypothesis-neutral approach and systematically characterize selectivity and functional specialization in four human higher-level visual regions of interest (ROIs) via response-optimized models. We focus on the fusiform face area (FFA)⁴, the extrastriate body area (EBA)¹⁰, the visual word form area (VWFA)¹⁴, and the retrosplenial cortex (RSC, a visual place selective area)³⁴. We emphasize that our approach is hypothesis-neutral in the sense of model construction; i.e., the models are not constructed to test any particular hypothesis, rather they are largely informed by the data. First, we leverage the unprecedented scale and quality of the massive 7T fMRI Natural Scenes Dataset (NSD)³² to train a deep neural network model to predict activity of voxels in each ROI. Each network is randomly initialized (not pre-trained on any task) and is optimized to predict the activity of all voxels in the ROI from the stimulus image. Our models achieve high prediction performance, on par or outperforming state-of-the-art task optimized models. Then, we ask what functional properties emerge spontaneously in our response-optimized models. We examine the trained networks through structural analysis (feature visualizations), as well as functional analysis (feature verbalization) by running high-throughput experiments with these models on large-scale probe datasets and dissecting the evoked network activations^35,36. Strikingly, despite no category-level supervision, the units in the optimized networks act as detectors for high-level visual concepts like ‘faces’ (in the FFA model) or ‘words’ (in the VWFA model), providing one of the strongest evidences for categorical selectivity in these ROIs.

The observed strong semantic selectivity raises another important question: are the ROIs simply functioning as processing units for their preferred category (e.g., simply deciding whether the stimulus contains a face or a word) or are they a by-product of a non-category-specific visual processing mechanism? To probe this, we create selective deprivations in the visual diet of these response-optimized networks and study the selectivity of model neurons in the resulting ‘deprived’ networks. We find that the resulting models still mostly demonstrate selectivity for the preferred category, even in the absence of any experience with the preferred category. This suggests that the same “filters” used to encode the preferred category are used to encode other non-specific natural images as well. The results presented here further indicate that category-selective voxels do not respond to their preferred category by the unique structure of that category but perhaps by some other general structural characteristics that other stimuli may share with the preferred category.

We further demonstrate that the proposed models generalize remarkably and selectively to different perceptual tasks. Representations from the FFA model can predict facial identity while those from the RSC, VWFA, and EBA models can discriminate between spatial layouts of different indoor scenes, letters and body poses respectively, revealing important functional distinctions between different ROIs.

Finally, in a retrospective analysis, we asked if this approach would work even when we do not start a priori with the ROIs, i.e., would the category-selective regions reveal themselves when we train response-optimized models on responses from a sufficiently broad set of voxels across the entire ventral visual stream, and apply single unit selectivity analysis techniques on the optimized model? Our experiments yielded an affirmative answer to this question, illustrating the success of this unbiased computational approach in revealing the representational structure of the human visual cortex. Further, this experiment also suggests that faces, bodies, text and food^37–39 are the categories that selectively activate the highest number of voxels in the ventral visual cortex and provides evidence against the existence of single voxels or regions selective for thousands of other visual concepts covered in our broad network interpretability dictionary beyond these categories.

Together with these data-driven models for higher order visual areas and model interpretability techniques, our study illustrates that response-driven deep neural network models of visual cortex can serve as powerful and unbiased tools for probing the nature of representations and computations in the brain.

Results

We trained deep neural network models directly to predict the brain activity related to viewing natural images. We diverged from the current computational neuroscience approach of extracting image representations from networks previously optimized on large image databases for tasks such as object classification. Instead, we optimized a naive network starting from scratch to directly predict the recorded activity in the voxels of a given higher-order ROI, thereby bringing neuroscience data directly to bear on the model development process. Thus, the network learned to represent images in a way that is optimal for predicting voxel activity in that ROI, capturing only the relevant dimensions of variance and the tuning of the voxels. We trained a separate model for each of four ROIs: FFA, EBA, VWFA, and RSC. We capitalized on the natural variation in the rich Natural Scenes Dataset (NSD)³² to train these models directly on stimulus-response pairs from a wide range of naturalistic scenarios. Notably, the dataset contains complex and sometimes crowded images of various everyday objects in their natural contexts at varied viewpoints. The stimulus set is thus more typical of real-world vision, which permits the characterization of neural representations and computations in ethological conditions.

Accurate predictions on complex, cluttered scenes; rapid generalization to new subjects

Our models learned deep convolutional feature spaces that are shared across thousands of voxels over multiple subjects. We used a rotation-equivariant Convolutional Neural Network (CNN) architecture to learn these feature spaces directly from fMRI data. This architectural design choice enables the model to learn identical features at multiple orientations and spatial locations, mimicking the response properties of neurons in early and intermediate visual areas, which are known to capture similar features like edges and curves but at different orientations and locations in the visual field^40–43. This core “representational” network comprises four convolutional blocks, each with two layers and followed by an equivariant Average-Pooling operation. Each convolutional layer contains 48 feature sets where each feature set is extracted at 8 orientations and is followed by a ReLU non-linearity. We employed a linear readout model on top of this feature space to predict the responses of individual voxels in the ROIs. A linear readout on a shared feature space is further reminiscent of the often made assumption that the activity in one region in one individual can be related to the activity in the homologous region in another individual by a linear transform⁴⁴. The linear readout was factorized into spatial and feature dimensions following popular methods for neural system identification in mouse visual cortex⁴⁵. This factorization separates receptive field location (i.e., what portion of the visual space is the voxel most sensitive to?) from feature tuning (i.e., what features of the visual input is the voxel sensitive to?). Sharing the entire representational network across subjects, sharing convolutional filters weights across visual field locations (translation equivariance) and orientations (rotation equivariance) and a factorized readout jointly enable the sample-efficient training of the response-optimized models (see Appendix D).

Our baseline is a task-optimized model with a similar number of convolutional layers trained to perform object classification on the large-scale ImageNet dataset (see Fig. 1 and the Methods section for details). To assess the prediction performance of this model, we used the standard methodology of modeling the voxel response as a linear weighting of the task-optimized network’s output units (from the best-performing layer that is determined using a separate validation set). Further, just as in our response-optimized models, this linear mapping is factorized into spatial and feature dimensions as this was found to improve performance over the traditional non-sparse readout method.

We compared the performance of response-optimized and task-optimized models (see Fig. 1). The models are trained jointly on four subjects and their performance is estimated on a held-out set of 1000 images seen by these same subjects. For these subjects, the response-optimized models attained approximately 65% of the noise ceiling (median correlation), with noise-normalized correlations lying between 52–77% for 50% of the voxels, yielding one of the most computationally precise voxel-level models of these higher-order regions. Further, response-optimized networks achieve parity with task-optimized networks on the same subjects (FFA: p = 0.274, VWFA: p = 0.014, RSC: p = 0.709, EBA: p = 0.510. p values calculated by a two-sided permutation test with N = 1000). Consistent with results previously reported using recordings from non-human primates²¹, our models are far more predictive than the category ideal observer model which employed the category membership of labeled objects in the image (another, simpler baseline than task-optimized models).

Next, we assessed how these predictive models generalize to the remaining set of four subjects that were not used to train the model. For this analysis, we train each network to predict activity for the remaining subjects by only optimizing the weights of the final linear readout while keeping the rest of the network fixed. We vary the amount of stimulus/responses pairs from the new subjects to train the readouts, from only 100 samples to a large set of 4500 stimulus-response pairs. The influence of neural dataset size on predictive accuracy can complement prediction performance when evaluating the quality of two competing models. We consider the best representation to be the one that enables the most sample-efficient learning of the readout model for new subjects. Importantly, the difference in performance between response-optimized and task-optimized networks becomes even more striking as we limit the stimulus-response pairs (see Fig. 1.E). In FFA, EBA, and RSC, the average performance for the response-optimized networks is already at more than 78% of its final value after just 100 training samples, compared with 60 − 67% and 50 − 65% for the task-optimized and categorical models respectively. In FFA and EBA, we need 500 samples for the task-optimized network to achieve a comparable performance to the response-optimized network with 200 samples. This remarkable generalization of response-optimized networks suggests that they are able to sufficiently constrain the space of possible solutions in the right manner so that the readouts for new subjects can be learned with few samples.

Network dissection reveals selectivity, tolerance and clutter-invariance

Here we demonstrate that the high prediction accuracy and generalization of our hypothesis-neutral response-optimized networks do not come at the cost of model intelligibility. Instead, we show that the response-optimized models possess both empirical and esthetic virtues, being computationally precise and elegant at the same time. Notably, features that emerge in the trained networks result from optimization to match ROI responses. After training the networks, we can probe their learned features, understand the computations they perform and, consequently, understand the characteristics of the ROI responses they model.

We adapted the recently proposed technique of network dissection^35,36 to generate “verbal” explanations for the responses of different voxels. The technique measures the degree of alignment between a voxel’s response properties and an extensive visual concept dictionary, spanning objects and fine-grained visual concepts like parts of objects, colors, materials, and textures. This alignment is estimated on an independent, large-scale natural image database called the Places365 dataset comprising a diverse set of real-world environments (indoor, urban, and natural). These images were not seen by the subjects during the experiment. They are passed through a response-optimized network, resulting in activations at every level of the network. The weights for a given voxel, along with the activations of the last convolutional layer, allows the construction of an activation map that indicates how much each part of the original image contributes to the activity of that voxel. This procedure is repeated, yielding an image-specific activation map for each voxel, which is thresholded to yield a binary map that estimates regions that activate the voxels, and regions that don’t (Fig. 2). Every image in this separate database has been labeled at the pixel level with different concepts (indicating if that pixel is part of an objects, a part of an object, a texture, etc.). The activation maps for an image, and the pixel-level labels are then used to estimate the alignment between each voxel and each concept in the database.

Fig. 2 — A Schematic of the model conversion procedure used to obtain the dissection results. The spatial mask was discarded and the learned feature tuning of every voxel was employed to create an additional 1 × 1 convolutional layer, so that every voxel is represented by an independent unit in this convolutional layer. B demonstrates the median Intersection over Union (IoU) metric across all voxels belonging to a Region of Interest (ROI) for the top 20 visual concepts rank ordered by median IoU. The top concepts for Fusiform Face Area (FFA), Visual Word Form Area (VWFA) and Extrastriate Body Area (EBA) discovered using this hypothesis-neutral approach (`heads', `signboards' and `person' respectively) align remarkably well with the widely held selectivity of voxels in these regions. C shows the matched concepts for every response-optimized model, i.e., the number of units in the `voxel' layer that showed high alignment (stringent threshold of IoU > 0.04) with a human-interpretable visual concept. Multiple units (i.e., multiple voxels) are associated with the same high-level concept. Contrary to object recognition networks which are trained with explicit label supervision and which show a broad diversity of detectors^35,36, the matched visual concepts in these response-optimized networks are highly specific and aligned with the previously hypothesized functional role of these ROIs. D shows the matched concepts for 3 networks with the same architecture as response-optimized models but random weights. The detection of these concepts (e.g. sky, grass, road) is likely driven by low-level cues, like color, instead of complex features. Source data are provided as a source data file. The example image is by Myotus and sourced from https://commons.wikimedia.org/wiki/File:Welcome_sign_and_city_sign,_Atwater,_Minnesota.jpg under a CC BY 4.0 license.

Alignment is estimated using a metric called the Intersection over Union (IoU)³⁵. For each concept c/voxel v pair, the IoU is measured for each image in the database, then averaged over all images. For a given image, the binary activation map specific to voxel v is compared to a binary map that indicates if concept c is present at each pixel. The number of pixels for which both binary maps have a value of 1 (intersection) is normalized by the number of pixels for which at least one of the binary maps has a value of 1 (union) (see Methods section for further details). After computing the alignment across images, the result is an average alignment value for every voxel-concept pair.

It is essential to note this methodological framework’s subtle yet profound implications. Previously, response profiles have been primarily defined using image-level category labels. Our approach enables us to model voxels as convolutional filters and systematically identify the image properties that they respond to without an a priori hypothesis specification. Our approach characterizes not just ‘which’ images activate a particular brain voxel but also ‘what’ in those images drives the response, providing a rich characterization of neural responses to crowded natural scenes.

For the top 20 concepts in each ROI, the median IoU is shown across all voxels in that ROI (Fig. 2B). Following³⁵, we used a stringent IoU threshold of 0.04 to detect ‘matching’, i.e., if a voxel’s corresponding filter exhibits a high agreement with a concept map (exceeding 0.04 IoU threshold), we classified that particular voxel as detecting or encoding that concept. Figure 2C shows the concepts for which a high agreement is found and the count of the voxels that achieve a high agreement with that concept (each voxel is only counted once, against the concept with the top IoU). As a point of comparison, Fig. 2D shows the same measures for untrained networks having the same architecture as response-optimized models, but with random weights. Finally, Fig. 3 shows a visualization of the part of highly activating images that leads to the high activation. For each ROI and its preferred concept (according to the median IoU metric), the five top voxels were chosen, and for those, the top 100 images activating images were picked from the large scale dataset. For randomly selected images from this set, the area that leads to maximum activation of the voxel is shown in Fig. 3.

Fig. 3 — Panels **A–D** show segmentation maps for the top concept-selective voxels in the Fusiform Face Area (FFA), Visual Word Form Area (VWFA), Extrastriate Body Area (EBA), and Retrosplenial Cortex (RSC), respectively. Activation of a single voxel (converted to a 1 × 1 convolutional filter) in response to an input image is visualized as the region in the input space that elicits the highest (top 1% quantile level) activation in the corresponding filter output. Top voxels as ranked by the IoU with the preferred concept for every ROI (`heads', `signboards', `person' and `windows' for Fusiform Face Area (FFA), Visual Word Form Area (VWFA), Extrastriate Body Area (EBA) and Retrosplenial Cortex (RSC), respectively). For each voxel, we first identified the top 100 most activating images from the Places365 dataset (used for network dissection), then sourced visually similar images from Wikimedia Commons (CC BY 4.0) and re-ran network dissection on those images for visualization. For FFA and EBA, because the top activating images contained identifiable faces or individuals, we instead used simplified digital sketches created by Vani Srivastava using Procreate that mimicked the content and composition of the top Places365 images. These sketches were shared with us under a CC BY 4.0 license and were used for network dissection visualization only. The other images were by Myotus, sourced from https://commons.wikimedia.org/wiki/File:Welcome_sign_and_city_sign,_Atwater,_Minnesota.jpg, by Leimanbhradain, sourced from https://commons.wikimedia.org/wiki/File:Sign-1160756,_Galway,_Ireland.jpg, by Kerry Raymond, sourced from https://commons.wikimedia.org/wiki/File:Bethlehem_Lutheran_Church_signage,_Steiglitz,_2005.jpg, by Mucklagh, sourced from https://commons.wikimedia.org/wiki/File:Signs-1160963,_Killarney,_Co._Kerry,_Ireland.jpg, by OnwardToThePast, sourced from https://commons.wikimedia.org/wiki/File:The_reception_room_of_the_Schloss_Drachenburg_(2022).jpg, by Vyacheslav Argenbergm sourced from https://commons.wikimedia.org/wiki/File:University_Square,_Window_view,_Moscow_State_University,_Moscow,_Russia.jpg, by Bjankuloski06, sourced from https://commons.wikimedia.org/wiki/File:Piano_in_bay_window,_Drawing_Room,_Highwic.jpgand by Anasskoko, sourced from https://commons.wikimedia.org/wiki/File:Windows_Kaduna_02.jpg, all under a CC BY 4.0 license.

Figure 2 shows that the FFA network’s favorite concept is “head” (the large scale dataset did not have a face label, and faces are labeled as parts of heads). The head concept has the highest median IoU of 0.125, more than other top concepts for FFA and the other ROIs. Some FFA voxels predictors have IoU as high as 0.20 with the ‘head’ concept. These predictors can be contrasted with the “head” detectors (model units with IoU > 0.04 against the ‘head’ concept) that emerge spontaneously in the last convolutional layer of a standard AlexNet trained on image categorization (ImageNet). In this task-optimized model, the best detector yields an IoU of 0.15 against the ‘head’ concept maps with the median IoU among all head detectors being 0.08. Thus, the face selectivity of units in the FFA model even exceeds the face selectivity of units in models trained explicitly with semantic labels on a million images. Figure 3A illustrates how the chosen voxels in FFA act as head (face) detector, with the parts of images driving the predicted response being almost exclusively faces.

The VWFA network has high median IoU with signboards (containing letters) even though the number of images with writing is small in the NSD dataset. Figure 3B shows that signboards with very different lettering and signs (different backgrounds, fonts, styles, scales, colors, orientations etc) drive the predicted response in the top voxels, even though the scenes are often cluttered with a myriad objects simultaneously. This result replicates the highly invariant orthographic processing in the VWFA⁴⁶. The VWFA also has high alignment with the concepts of head and person, which is due to the anatomical overlap of localized VWFA with FFA, discussed in a following section. EBA has high IoU with people, head, and skin. The IoU with the people concept is highest, highlighting that EBA is tuned for body parts. Interestingly, for RSC, we observe a large number of ‘window’ detectors after applying the dissection procedure. Navigational affordances are important for scene perception, and windows might be particularly indicative of such affordances in indoor scenes and critical to functional scene understanding (for e.g., windows indicate an obstructed path where movements are blocked)⁴⁷. Other concepts evoking high response within the RSC (e.g., ‘wall’, ‘glass’, ‘building’) relate to scene perception and navigation, in contrast to concepts discovered within the models optimized for the other three ROIs. These results highlight the differences in selectivity between the visual areas. Despite no access to category labels during training, our networks gained a strong semantic selectivity for high-level visual concepts and this selectivity remained tolerant to changes in category-orthogonal nuisance variables (e.g., size, viewpoint, clutter etc.), as shown in Fig. 3.

Maximally exciting images reveal structured high-level features consistent with previously hypothesized ROI role

In classical neuroscience, identifying the optimal visual stimuli (the peak of the tuning curve) for different neurons has been instrumental in understanding neuronal selectivity and its contribution to perception. Our models permit following this approach for higher-order visual areas where the optimal stimulus can be arbitrarily complex. We performed an unconstrained optimization over input noise to discover input images that result in maximal (predicted) excitation of individual voxels. We refer to these images as the maximally exciting inputs. By performing unconstrained optimization, we are not restricting the space of maximally activating inputs to the naturalistic domain. While optimization without naturalistic constraints may impose its own set of challenges including generation of hard-to-recognize visual features, visualization with regularization or naturalistic constraints may not be truly faithful to the model. We favored the former approach and did not restrict the space of maximally exciting inputs to the naturalistic domain. Instead, we let the optimization process evolve complex visual inputs without constraints.

Figure 4 shows that maximally exciting images for different voxels in the same ROI capture very similar visual properties. Out of all the features that could emerge from unconstrained optimization in a network trained on cluttered natural scenes, from simple features such as rounded shapes or eyes to possibly more complex high-level features, face-like images with overlapping small and large circles almost exclusively pop up for all FFA voxels, providing a strong support for the hypothesis that full ‘face’ features lead to increased activation in FFA. Similarly, for EBA, we observe elongated curved shapes, loosely similar in form to body parts such as arms or legs. Maximally exciting inputs for the VWFA resemble orthographic units comprising curves and lines of different stroke widths like the visual form of letters. Finally, the maximally exciting inputs for voxels within RSC are reminiscent of windows (in accordance with the Network dissection results) in different reference frames, which may be linked to RSC’s role in spatial cognition⁴⁸. We also observe rectilinear features in the maximally activating images for RSC, consistent with the previously found rectilinear preference of scene-selective areas⁴⁹, while the EBA and the FFA only captured curvilinear features. Therefore, our results provide further evidence for this hypothesis by learning the model that best predicts the data instead of starting apriori with the hypothesis.

Fig. 4 — Panels **A–D** show maximally activating images for voxels in the Fusiform Face Area (FFA), Visual Word Form Area (VWFA), Extrastriate Body Area (EBA), and Retrosplenial Cortex (RSC) respectively as discovered by the optimal stimulus identification procedure, wherein an unconstrained optimization is performed over random input noise to discover the input that would result in maximal excitation of individual voxels. Images are shown for 5 randomly selected voxels per ROI starting from 3 random initial points per voxel. Each column is a randomly chosen voxel and each row is a different initialization. This structural analysis reveals that the features that drive individual voxels (the *visualizations*) are consistent with the *verbalizations* for those voxels as expressed by the network dissection procedure. More specifically, face-like visual forms, curved features akin to orthographic symbols, skin-colored shapes reminiscent of some body parts and window-like rectilinear features emerge spontaneously for FFA, VWFA, EBA and RSC respectively.

End-to-end models capture tuning differences between voxels in the same ROI

To investigate if the proposed models indeed capture meaningful differences between voxels, we computed spatial generalizability matrices by correlating the predicted response of each voxel against the measured response of every other voxel to obtain an N × N correlation matrix for shared models, where N is the total number of voxels across all participants^50–52. These matrices reveal the similarity of the tuning of each pair of voxels. To account for higher variability in measured versus predicted response, we normalize the rows and columns of this correlation matrix following⁵³. The diagonal dominance in these identifiability matrices (Fig. 5A) suggests that predicted responses are most similar to the same voxel’s measured responses, which indicates that all models are successfully able to capture meaningful voxel-level idiosyncracies to some extent. The generalization matrices reveal the presence of several distinct clusters (at least two) for all ROIs, such that the models of voxels in one cluster are highly predictive of responses of voxels in the same cluster (both within, and across participants), but do not generalize to other clusters. Importantly, this clustering structure is prevalent across participants (specifically for FFA, EBA and VWFA, with more variability for RSC), indicating a shared organization. We leave a thorough characterization of the differences between these clusters for future work.

Fig. 5 — A shows *spatial generalization matrices* for every ROI computed by correlating the predicted response of every voxel against the measured response of every other voxel belonging to the same ROI (within and across subjects). Black lines mark subject boundaries. B Scatter plot depicting the difference between Intersection over Union (IoU) of the preferred category (loosely, `signboard' and `person' for Visual Word Form Area (VWFA) and Extrastriate Body Area (EBA), respectively) and faces against the difference between the corresponding t-statistic values estimated by the functional localizer experiment. Each point is an individual voxel belonging to the same ROI. The inset indicates the correlation coefficient and whether they are significantly larger than chance after correcting for multiple comparisons. Asterisks indicate statistical significance of Pearson correlation coefficients computed per subject. Specifically, ** indicates p < 10⁻⁵. The pvalues correspond to two-sided tests of the null hypothesis that the true correlation is zero, based on the probability of observing a correlation at least as extreme as the one computed, under the assumption of no correlation. For the top panel, pvalues for subjects S1–S4 were 1.42 × 10⁻³⁹, 9.23 × 10⁻³⁵, 7.98 × 10⁻⁶¹, and 2.32 × 10⁻²². For the bottom panel, pvalues for subjects S1–S4 were 1.86 × 10⁻¹⁶⁹, 1.13 × 10⁻⁸⁴, 3.17 × 10⁻²⁰⁴, and 4.71 × 10⁻¹⁰⁵, respectively. C Cortical surface plot of face selectivity as measured by the localizer experiment against the IoU with the `head' concept as quantified by the dissection procedure for one subject (Similar plots for remaining subjects are shown in Supplementary Fig. S1). Source data are provided as a source data file.

Next, we examined voxels that were strongly selective for a different semantic concept than the conjectured preferred category for their visual area. We specifically examined the ‘head’ selective model neurons identified by the network dissection procedure within VWFA and EBA. We assessed the correlation between the quantitative agreement of these voxels with the “head” concept over the “signboard” or “person” concept (IoU, as evaluated by our dissection procedure) and the degree of their face-selectivity over selectivity for words or body parts (as quantified with the independent functional localizer experiment). Our results suggest a very strong correspondence between the two (Pearson’s R ~ 0.34–0.66, p < 0.001, Bonferroni corrected, for all 4 subjects and both comparisons, Fig. 5B), despite the very different experimental paradigms involved in the two quantifications. Further, in addition to comparing this relative selectivity, we also compared the absolute ‘head’ selectivity of voxels in EBA and VWFA against the face-selectivity (t-value) measured with the localizer experiment. Again, we see a striking pattern of similarity in these estimated and measured values (Pearson’s R ~ 0.6–0.7, p < 0.001, Bonferroni corrected); (see Fig. 5 and Supplementary Fig. S1). Thus, our computational models successfully capture the spatially overlapping representations of semantic categories and graded functional organization within human extrastriate cortex.

High selectivity persists in “face-deprived” and “body-deprived” networks

The strong semantic selectivity in our response-optimized models leads to an important question: are the ROIs they model simply functioning as detectors for their preferred category? In this case, then training for example a neural network to predict FFA activity would be equivalent to training this network with images associated with a label that indicates the presence of a face. An alternative hypothesis is that category-selective ROIs are sensitive to visual properties that are typical of their preferred category, even in the absence of that category. In this case, training a neural network to predict FFA would provide it with a more complex signal that allows it to learn the sensitivity of the FFA to those visual properties.

To differentiate between these alternatives, we focused on FFA and EBA and trained response-optimized models with the same architecture above but with a visual training diet entirely deprived of images containing the “person” category. Surprisingly, we find that, despite not seeing any image with human faces or bodies during training, the networks optimized to predict FFA and EBA retain their categorical selectivity (Fig. 6). Supplementary Fig. S6 further shows that these results hold even when further removing any images with animals from the training set. The training diet deprivation did decrease the agreement of model voxels with their respective preferred category (e.g., in FFA, the median IoU with ‘head’ dropped from ~ 0.13 in the non-deprived network to ~ 0.08 in the deprived network), indicating a slightly reduced selectivity (Fig. 6). However, the preferred concept was still overwhelmingly ‘head’ (with 1935 selective for ‘head’ versus 2192 in the non-deprived network) and the resulting segmentation pictures look qualitatively similar to those in Fig. 3. In other words, we found that networks trained with a visual diet deprived of their preferred category could still extrapolate the responses to the preferred category.

Further, we compare in Fig. 6C, D the average real and predicted activities in FFA and EBA in the presence and absence of persons. Note that the presence of persons doesn’t automatically shift the average responses over all images to a higher level than the images with no persons. This is because, while there is a strong dependence between the presence of persons and the voxel activity, this dependence is complex in natural images, and depends on the location and context in which these persons (and their faces and bodies) are, as well as on their size. Figure 6C, D also show that models do not incur a severe loss in prediction performance when data from a new domain (i.e. “faces” and “bodies” category) is presented at test time. Actually, the FFA model achieves even slightly better prediction performance on held-out images with faces. This example of systematic generalization in the developed response-optimized models is also interesting from the perspective of modern deep learning, which is often criticized for its failure to generalize in this systematic, out-of-distribution way. The ability of these models to generalize to new domains further validates their proposed usage as virtual stand-ins for fMRI experiments, supporting their ability to act as model organisms for large-scale fMRI experiments and revealing selectivity for stimuli beyond those encountered in the training set.

Models reveal key functional distinctions between different regions

Our results indicate that response-optimized models capture complex tuning properties relevant to their preferred category but not uniquely exhibited by it. Next, we hypothesized that our models meaningfully discriminate between stimuli belonging to their preferred set. More specifically, we tested existing accounts of functional specialization which implicate FFA in face perception, particularly face identity discrimination^54,55, RSC in spatial cognition⁴⁸, EBA in perception of the shape and posture of bodies⁵⁶, and VWFA in visual perception of symbols, like letters and digits⁵⁷.

We first performed a face discrimination task using the response-optimized models of all four ROIs. For each model, we extracted the predicted voxel-wise response for a small subset of facial images from the CelebA dataset, comprising 20 identities, each with 100 train, 30 validation and 30 test images^58,59. In this and every subsequent task, we include the performance of a general-purpose representation from an ImageNet-trained network (denoted as blue triangles in Fig. 7) as a benchmark. We also compared the face recognition accuracy of these models with that of a representation from a network trained explicitly to do facial recognition on a large set of VGG face identities⁶⁰. For each of the representations above (the four ROIs’ predictions and the general-purpose and face-specialized representations), we trained a linear function to predict the CelebA identities (see Fig. 7A). The resulting FFA predictions outperform the predictions from all other ROI models (discrimination score of 85% compared to 79–80%), even outperforming the highly transferable representation of ImageNet trained networks⁶¹ at 78% (see Fig. 7B). We note this performance still falling short against features from state-of-the-art networks trained to discriminate a large number of facial identities at 96%. This result provides additional evidence for the role of FFA in face identification and shows the ability of our model to pick up these functional capacities.

Fig. 7 — A Shows a diagram of the transfer procedure. The response optimized models are frozen, and a new linear layer is added and optimized for different tasks. B–E Show mean transfer performance of representations from response-optimized networks of each individual ROI across cross-validation folds on the four visual classification tasks considered in this study: face recognition, letter and body pose classification and room layout estimation. Error bars indicate the standard deviation across cross-validation folds. The blue triangle indicates the mean performance of representations extracted from an ImageNet optimized network across the folds. Remarkably, representations learned from Fusiform Face Area (FFA) achieve the best face recognition accuracy (beating even the ImageNet baseline, which is trained on a million images), representations from Visual Word Form Area (VWFA) achieve the best letter classification (also beating the ImageNet baseline), and representations from Extrastriate Body Area (EBA) and Retrosplenial Cortex (RSC) also achieve the best accuracy on body pose classification and room layout estimation, respectively. This result is remarkably consistent with (and providing strong evidence for) the previously hypothesized functional roles of these areas. Source data are provided as a source data file. The example image is by Bjankuloski06, sourced from https://commons.wikimedia.org/wiki/File:Piano_in_bay_window,_Drawing_Room,_Highwic.jpg under a CC BY 4.0 license.

Next, we followed the same procedure as above and trained a linear classifier on top of different model representations to perform a letter classification task. Here, the goal is to predict letter identities (A-Z) from images of printed letters created using different fonts, styles, sizes and position effects⁶² (examples shown in Fig. 7C). The VWFA response-optimized model consistently outperforms the remaining response-optimized models and the benchmark set by the ImageNet-trained network (Fig. 7C), providing computational support for the ability of VWFA responses to enable letter perception. We note that differences between models are small, possibly because of the substantial overlap between ROIs due to the liberal thresholds (t > 0) applied on functional localizer data.

Following a similar procedure, we quantified the ability of different model representations to support body pose classification on a computer graphics (CG)-generated body pose dataset containing exemplars from 5 well-known yoga poses⁶³. On this pose-discrimination task, we find that the EBA model consistently performs on par with or exceeds the performance of the remaining ROI models (see Fig. 7D). Here, however, we observe a gap between the performance of the best response-optimized model and the ImageNet-trained task-optimized model, possibly due to the richer variation of different body postures in the ImageNet dataset as compared to NSD stimuli, which may allow efficient transfer to this task, or to the particular challenges that the body pose estimation task presents. All response-optimized models, however, perform above chance (20%) on this challenging dataset.

Finally, we performed a room layout estimation task. The objective of this task is to predict the correct layout type from the 11 categories described by Zhang et al.⁶⁴ defined using a keypoint-based parameterization (Fig. 7E). The dataset comprises 4000 natural scenes across diverse indoor scene categories from the SUN database⁶⁵, that are split into sets of 3200 training, 400 validation and 400 test images. We followed the same procedure for quantifying the layout estimation accuracy as before, i.e., training a linear classifier on top of each representation (Fig. 7E). The RSC predictions outperform the other ROI models. They perform just as well as the ImageNet trained representations in solving this task, even though the RSC network was trained with ~ 35,000 images and their associated brain responses in contrast to the ImageNet network training size of a million images. This result strongly supports the role of RSC in scene understanding, particularly when relevant to spatial navigation, and again highlights our model’s ability to transfer to these tasks. It is important to note that classification accuracy is inherently task-specific, so direct comparisons across tasks is not meaningful; however, within each task, the ImageNet baseline provides a meaningful benchmark for contextualizing the performance of our ROI-specific models.

We note that while each of these regions has been stipulated to fulfill a specialized function before, thus far, no study has shown evidence of this task specialization from the pattern of neural responses in these regions directly (see⁵ for an exception) or computational models of these regions. In fact, evidence from task-optimized models suggests the contrary. Deep neural networks trained on face individuation perform worse than or on par with networks trained on object categorization in capturing face representations in the macaque⁶⁶ and human brain⁶⁷, posing a challenge to domain-specific functional accounts of face-selective regions. How do we reconcile our results with these findings? In modern DNNs, task-optimization and training datasets are often intertwined due to the limited availability of datasets that contain multiple types of learning signals (i.e., labels). This coupling confounds interpretations and models trained on ImageNet often surpass other models due to the rich coverage of the naturalistic stimulus space in this dataset over others, which can compensate for lack of sufficiently powerful priors; these ImageNet-trained object categorization models continue to yield best response predictions for all ventral visual regions alike²⁴. In contrast, the response-optimization approach offers an intriguing opportunity to probe functional distinctions among ROIs since these models can be trained on an identical image set and only the training signal (i.e., the brain response) can be varied between these models. Any differences in emergent function within the trained models can thus be more confidently attributed to the differences in response patterns between ROIs. Further, we note that category selective responses and functional specialization (fine-grained discrimination between exemplars of the preferred category) do not necessarily follow from each other. Selectivity is contingent on the extraction of features shared by all exemplars from the preferred category while fine-grained discrimination relies on features that distinguish each exemplar (each facial identity or each room configuration) from others. While selectivity is measured with respect to responses of single neurons or voxels, fine-grained discriminability is a measure of information contained within the population response. Selectivity and functional significance analyses thus do not equate from a computational perspective and offer a complementary purchase on our understanding of these regions.

Hypothesis-agnosticity reveals a limited set of categories for which selective responses are observed in the ventral visual cortex

We note that our approach, thus far, was hypothesis-neutral in the sense of model construction; i.e., the models were unconstrained by any particular hypothesis, rather they were largely informed by the data. While some of our architecture choices were informed by a priori knowledge of what could be important to model the primate visual system in general (convolutional structure, equivariant filters), these choices remain neutral with respect to the selectivities of the high-level ROIs we study. Nonetheless, the selection of regions itself was biased by a priori knowledge and their localization was guided by independent functional data measuring the brain responses to different categories of stimuli. We presented an approach to determine the selectivity of these regions and characterize their function in an impartial computational manner. The emergence of category-selective units in response-optimized models unbidden provided emphatic evidence for category-selectivity of these regions and suggested that these regions not only respond to low-level features most correlated with the preferred category but to the high-level category per se. Here, we take a step further and ask: can we discover category selectivities with an impartial approach that does not require ROI selection or localization? To answer this question, we performed a retrospective analysis: we trained a response-optimized model on responses from the entire ventral visual cortex (28,910 voxels) across all subjects. The resulting model is able capture the structure of ventral visual stream responses, as evidenced by a reasonably high response prediction accuracy (median noise-normalized R ~ 59%). We then applied the network dissection procedure on units within the optimized model. The results, shown in Fig. 8, highlight the categories previously believed to elicit selective responses, namely heads (or, faces), signboards (or text) and skin, which reflects selectivity for the ‘bodies’ category. A small number of units appear selective for elements such as sky and window, which can be attributed to processing scenes. Moreover, we also observe single-unit selectivities for the ‘food’ category, for which neural selectivity was also reported in recent works^37–39, validating our data-driven approach. The non-existence of units aligned with any of the thousands of other semantic concepts beyond these categories in the dissection concept dictionary (e.g., car, door, chair, plastic etc.) also serves as evidence against the existence of selectivity for those categories of stimuli in the ventral visual cortex.

Fig. 8 — Bottom left plot shows the un-normalized prediction accuracy (R) of every ventral visual stream voxel against the corresponding noise ceiling. Noise-normalized prediction accuracy is reported in the inset. Source data are provided as a source data file.

Discussion

Here, we exploited data-driven, hypothesis-neutral deep neural networks to model the responses of higher-order visual ROIs, specifically FFA, EBA, VWFA, and RSC. Through a large, rich stimulus set afforded by the NSD, we offered new evidence that generalizes decades worth of hypothesis-driven results to ethologically valid empirical settings, and further refines these hypotheses.

Brain response predictivity

We found that response-optimized deep neural network models—trained solely with supervision from fMRI activity—accurately predict activity related to new images in multiple visual category-selective ROIs. The performance of these models rivals the predictive performance of state-of-the-art task-optimized models used to predict brain activity²¹. Many model networks can be consistent with brain activity data as demonstrated by the number of recent papers in this area⁶⁸, and prediction performance is often limited however in what it reveals about brain representations⁶⁹. Thus, in this paper, prediction performance, measured through the correlation of the predicted and real activities, is only used to validate the response-optimized model. In the rest of the analyses, we use more expressive methods to understand the selectivities of the voxels and the functional properties of the response-optimized networks.

Further, it can be argued that a good representation would allow for efficient learning (in terms of the number of data points required) of a linear predictor of brain responses. We ran an analysis in which we evaluated the trained models on the ability of their representations to predict data for new subjects while restricting the training set size. We found that the response-optimized models were better able to generalize to novel subjects at small sample sizes than task-optimized models, suggesting that end-to-end optimization solely driven by response measurements can yield better correspondence to brain data than networks optimized on behaviorally relevant tasks with millions of images. Furthermore, these observations show that, even though we use a relatively small number of subjects (four for training and four for generalization) which might not allow us to model very fine individual differences in specialization (e.g., face recognition performance), our results showcase strong generalization of both the predictive ability of the response-optimized models and the patterns of selectivity they reveal.

Verbalization of voxel selectivity with network dissection

The current trend in visual and computational neuroscience uses naturalistic stimuli not amenable to parametrization and neural networks to model brain responses. This approach requires concomitant methodological advances to derive trustable conceptual understanding from brain response models. This view assumes that accurate and generalizable computational models of brain function can serve as a reasonable proxy to biological visual systems. Probing and interrogating these models may provide novel insights into the properties of their associated biological system. Here, we developed a systematic methodological framework to understand the tuning properties learned by our response-optimized networks. To reveal unidentified neuronal tuning properties, our framework relies on the recent progress in understanding neural networks using large-scale annotated datasets and network dissection procedures³⁵. This alignment procedure capitalizes on the use of a factorized readout that disentangles the spatial (“where”) and feature (“what”) dimensions of the voxel’s response properties and fully characterizes the “what” dimension using dissection procedures applied on the ‘model neuron’ corresponding to the voxel.

Using complex stimuli like cluttered natural scenes containing multiple objects in context to probe neuronal response properties poses multiple challenges. If an image set that highly activates a voxel is identified, several competing interpretations can be ascribed to that voxel’s tuning function. There can be multiple low-level (e.g., color or shape) or high-level (e.g., semantic) features consistent with the identified images. When stimuli become more complex, recognizing the consistent pattern implicit among any given image set becomes more challenging as visual features become increasingly tangled (textures, shapes, edges, etc.). The dissection procedure in this study helps simplify this problem by not only recognizing ‘which’ natural images elicit higher responses, but also systematically and quantitatively characterizing ‘what’ part of each image drives increased responses. With this approach, we can effectively verbalize the functional role of voxels in each candidate region. We demonstrated that emergent concepts in response-optimized networks are highly specific and aligned with each region’s previously hypothesized functional role. For example, we reported an exclusive presence of face detectors in the model for FFA. If the learned category-selectivity were due to low- or mid-level features correlated with the category of interest, those features should have become apparent. The dataset used in the network dissection analysis is one of the largest densely-annotated natural image datasets in computer vision research. If the selectivity of FFA to faces could be explained by the unique structure of eyes or skin-colored textures, response-optimized models would have highlighted these simpler features. The segmentation-based dissection procedure would have revealed this pattern of selectivity as it scores units against category-level labels and thousands of other concepts, including different textures and object parts (including eyes, nose, etc.). With our dissection procedure, we see an almost exclusive selectivity for full faces in the FFA voxels. Our results also indicate that computational models can account for perceptual invariance, even though they are only optimized for brain response prediction. The models respond selectively to their preferred category despite tremendous variation in the precise physical characteristics of the preferred objects.

Visualization of voxel selectivity with image synthesis

We perform an unconstrained optimization over input noise to discover the input that would result in maximal excitation of individual voxels, which we refer to as the maximally exciting inputs. It is important to note that by performing unconstrained optimization, we are not restricting the space of maximally activating inputs to the naturalistic domain; instead, we can let the optimization process evolve arbitrarily complex visual inputs. This is in contrast to prior approaches of modeling higher-order cortical activity, which rely on GAN-based priors for synthesizing maximally activating images^24,70 or more recently on diffusion models prios⁷¹. Such approaches use class-conditional GANs or diffusion models trained on large-scale datasets for image synthesis. They either optimize the image class first and then fine-tune the noise vector space⁷⁰, optimize both jointly to maximally activate neural responses²⁴, or use the gradient of the brain signal to directly guide the generation of new images⁷¹. Additionally, these studies synthesize maximally activating images at the region-level and not at a finer-grained voxel level.

For completeness, we perform a thorough comparison with some of these existing approaches for visualizing maximally activating inputs, particularly those of Ratan et al.²⁴ and Gu et al.⁷⁰ which employ GAN-based priors. However, instead of performing optimization with task-optimized models of brain activity, here we apply similar techniques to our response-optimized models of higher-order sensory regions. It is important to clarify that our use of GAN-based priors is solely a methodological strategy for enhancing the visualization of voxel selectivity and does not imply that the brain employs similar generative mechanisms. Our optimized images, depicted in Supplementary Fig. S10, again highlight the strong and distinct category-selective tuning of these regions, aligning with their previously hypothesized selectivity. Notably, using GAN-based priors has made the optimized images more comprehensible, contrasting with outcomes from methods lacking naturalistic priors (as in Fig. 4). However, we argue that both types of approaches are useful. The prior-free approach which we adopt in this manuscript can reveal the core visual forms that drive high-level category-selectivity in response-optimized models, even if the resulting images are harder to interpret. Consider that visualization with a natural image prior may not always be truly faithful to the model or to the brain, as the natural image prior is learned from a distribution of images in a large dataset, and does not correspond to a process that the brain is engaged in. In other words, the learned generative distribution doesn’t map to a generative process that the brain is doing, and instead is simply used to facilitate interpretation. For instance, generative model priors might inherently favor complex visual features (such as the presence of natural object categories) and favor categories that are over-represented in the dataset used to train the prior, irrespective of the tuning properties of voxels; the content of their visualizations, thus, has a greater risk of being linked correlationally, and not causally, to brain activation⁷². As an example, Gu et al.⁷⁰ have reported the prevalence of dogs in the maximally activating images for FFA. These seemingly counter-intuitive findings could potentially be clarified by considering the characteristics of ImageNet, the dataset upon which most task-optimized encoding models are trained. ImageNet comprises 120 different dog breeds but lacks explicit categories for humans. In contrast, the prior-free MEI generation method we use here avoids such biases as it does not depend on pre-trained datasets.

Specific versus non-specific mechanisms in shaping categorical selectivity

Next, we analyzed the role of visual experience in shaping response selectivity by training response optimized models with a visual diet completely deprived of faces or bodies. Despite this selective deprivation, units in models of FFA and EBA retained selectivity for their preferred semantic content (the top category was still the same as in the non-deprived models, and while the median IoU measure decreased for that preferred category, an overwhelming majority of voxels still had selectivity for that category). We posit that the models effectively became face and body selectors, which suggests that they could generalize their predictions to stimuli that were not included in training and learn which stimuli will induce maximal firing. The models could infer the preferred category of their corresponding ROI through training with a dataset devoid of this preferred category. The models could enable this generalization because of the properties of the representations of FFA and EBA voxels. These voxels appear to respond to visual configurations characteristic to faces or body parts, respectively. These configurations could be present in parts of images that do not correspond to a person but happen to have face-like (e.g., circles) or body-like (e.g., elongated, curved shapes) features. One potential mechanistic explanation is that attributes of these specific classes of visual stimuli (like faces and bodies) do not vary independently with the rest of the visual world and visual category-selective ROIs act like filters that constantly look for matches to their visual properties. While faces or bodies can activate these filters maximally, other visual patterns that resemble these categories can also activate the filters. For the FFA, these other visual patterns could act like pareidolia, or perhaps subtle pareidolia that is hard to detect by humans but that still activates the FFA. These results build on the work of Vinken et al.⁷³. Both studies converge on the idea that face selectivity does not rely on features exclusive to faces but instead arises from domain-general visual representations (with our findings extending to bodies as well). However, our approaches differ substantially. Vinken et al. rely on the observation that face-selective sites show reliable responses to non-face images and a common encoding axis in object-trained networks can predict face selectivity—networks that are, nonetheless, trained on datasets including faces, potentially embedding face-selective units. In contrast, our selective deprivation experiments remove face (or animate) stimuli entirely from the training set. This design eliminates the potential confound of prior face exposure and directly tests whether category-specific visual experience is necessary to account for responses in corresponding category-selective regions.

The systematic generalization and extrapolation ability in our computational models suggests they can, in principle, discover functional specializations in underexplored parts of the visual system. Most claims regarding functional specialization and domain-specificity in the brain are grounded in hypothesis-driven contrast-based experiments, which might not include the best-fit stimuli for these regions. Even large datasets like NSD might exclude certain categories that contain key features for a given brain region. After training our models on under-explored regions, their generalization ability can aide in characterizing selectivity post-hoc.

Our results can also inform studies on statistical learning in artificial and natural systems. In fact, out-of-distribution generalization remains a key issue in machine learning. However, our model could robustly generalize to unseen categories. This ability reflects the strong semantic selectivity of the ROIs and their responsiveness to features characteristic of their preferred category even in the absence of that category. Another key question focuses on whether developing selectivity for faces requires experience with the unique structure of faces. Existing behavioral evidence from some face deprivation studies⁷⁴ suggests that face-processing abilities can persist without any face-specific experience. While subsequent studies have provided counter-evidence, this critical dispute still has no consensus. Our experiments highlight that face and body selectivity can emerge spontaneously in computational models with no face and body experience, and this selectivity is maintained across diverse naturalistic variations (see segmentation maps in Fig. 6). We do note the caveat, however, that our models were trained using supervision from brain regions that have experienced faces and bodies.

Link between emergent representations and their functional roles

Goal-driven deep learning models of the visual cortex have paved a new way forward for understanding the representations and computations carried out by the visual system. They not just predict neural activity to novel stimuli, but also link it to complex visual behaviors in naturalistic domains due to their functional constraints. In fact, it is striking that biologically plausible representations emerge spontaneously in neural networks trained only with a handful of computational principles (gradient descent, convolutions etc.). While response-optimized models do not provide such a direct link to perception and behavior, here we asked if it may be possible to read out ‘behavior’ ensuingly from these models that are constrained solely with neural data, thereby investigating the functional role of different visual areas. To probe the functional capabilities of these representations and test existing specialization accounts implicating FFA in face identification⁵, RSC in spatial cognition⁴⁸ and VWFA and EBA in letter and body perception, respectively, we simulated these fine-grained discrimination tasks using the representations from response-optimized models of all ROIs. We found that the representations from the FFA network outperformed the other representations and a representation from a task-optimized network at the face identification task. We also found that the RSA network beat the other representations and a representation from a task-optimized network at the spatial task. Similarly, among response-optimized models, VWFA and EBA representations surpassed others at the letter classification and body pose estimation tasks, respectively. This selective transfer to different visual discrimination tasks provides strong clues into the functional roles subserved by different representations. Moreover, the observation that these complex visual capacities are realized spontaneously in neural networks optimized solely to predict brain activity suggests that the utility of these networks may not be restricted to modeling cortical activity, rather they may serve as useful models of biological function as well.

A possible complete characterization of selectivity in the ventral visual cortex?

When considering a mask of the ventral visual regions and training a new model to predict the entire region, we recovered the categories detected when considering the ROIs overlapping with this mask: heads (faces), signboards (words) and skin (bodies). We also obtained selectivity for food, which was also reported in recent works^37–39. However, we did not find detectors for the thousands of other semantic concepts present in the large stimulus set and the dissection concept dictionary. This result presents an intriguing hypothesis that we have already identified all the major categories for which the ventral visual cortex is selective. It should be specified that this result might be constrained by the diversity and coverage of images in the NSD dataset, the precision of the pixel-level labels in the datasets used for network dissection, the limited spatial resolution of fMRI, by our modeling approach, and by the fact that we perform the dissection at a single voxel level. Since the NSD dataset offers high-quality response measurements afforded by the use of ultra-high magnetic field strength (7T), is already quite expansive and diverse in the stimulus set, and since the datasets used for dissection have been meticulously labeled down to the pixel-level against a broad range of concepts, it is likely that this hypothesis holds merit. This hypothesis relies on the definition of selectivity as the most activating category for a voxel or a region. As we saw in this paper, regions in the high level visual system also respond to characteristic patterns in the absence of the preferred categories. It is possible that these graded responses to non-preferred categories might still be used by upstream regions when processing non-preferred categories.

Related studies

Our findings extend prior work (e.g., Murty et al., 2021) by showing that purely response-optimized models—trained exclusively on fMRI signals at the voxel level—can learn high-level category detectors without any explicit category labels. Moreover, by dissecting trained networks with a large-scale concept dictionary, we pinpoint the exact objects and object parts driving each voxel’s response. This methodology not only reveals that category-selective responses can emerge from the demands of predicting neural activity alone (even in the absence of direct training on those specific categories), but also highlights the existence of unique functional capacities in different ventral regions—illustrated by their specialized performance on tasks such as face recognition, body posture classification, or spatial layout estimation. In conjunction with our current research, recent studies^31,33,75 have also demonstrated the competitive performance of the response-optimization approach when trained with extensive datasets, often comparable to state-of-the-art task-optimized models. Khosla et al.⁷⁵ adopt similar neural network architectures for modeling a series of areas along the ventral visual hierarchy, from the earliest stages of processing at V1 to intermediate stages V2-V4 and higher-order ventral-occipital (VO) and lateral-occipital (LO) regions and provide a strong quantitative validation of the response-optimization approach. Seeliger et al.³¹ optimize parameterized models directly using responses from early visual areas during prolonged naturalistic stimulation and demonstrate the ability of trained models to predict the population receptive fields of these areas. St. Yves et al.³³ studied the early to intermediate visual cortex (V1–V4) and explored the brain hierarchy’s role in training brain-optimized models and presented counter evidence against the hierarchical nature of neural representations. Our study differs from these works in several aspects:

We extend beyond population receptive field estimation and employ a network dissection method to analyze the feature selectivity of units in trained networks, generating verbal descriptions of neural response properties.
We showcase how response-optimized models can be used to tackle critical scientific questions about selective responses. Through visual deprivation experiments, we reveal how category-selectivity can be influenced by domain-general mechanisms.
Our research uncovers a crucial link between emergent representations in response-optimized models and their functional roles. We demonstrate how the response-optimization approach yields models that excel at fine-grained perceptual tasks, providing empirical evidence for task specialization in distinct category-selective regions of interest (ROIs). Notably, this work assesses the functional capabilities of response-optimized models, i.e., their generalization to behavioral tasks. Interestingly, our models’ strong generalization ability sometimes surpasses the generalizability of even highly transferable representations from ImageNet-trained networks, yielding intriguing implications for computer vision models.

Collectively, these studies, along with our research, advocate for an alternative and complementary strategy to the prevailing task-optimized modeling framework for understanding neural representations.

Extensions

Probing computational models of the brain imposes several challenges that may confound our conceptions about neural representations and computations. One key issue is that any conceptual insights gained from a computational model are helpful insofar as the model is a good approximation of the biological system. The prediction accuracy of response-optimized DNN is not yet perfect. This may be due to limitations in the model architecture, insufficient stimulus-response data for fitting complex neural network models, supervision from noisy fMRI signals, or a combination of these and other factors. The work presented in this study, nevertheless, was able to (1) replicate decades of a significant body of hypothesis-driven work, (2) show the robustness of the learned representations even in the absence of the category of interest, (3) show that these representations could achieve important functional roles hypothesized to be characteristic of their respective ROIs and (4) that the category selectivities can be recovered even when the analysis approach is applied on the entire ventral visual pathway. This work also revealed a new empirical space to improve predictions of brain activity in future work, by considering different model architectures and focusing on high-order cortical regions (particularly along the dorsal visual pathway where the current task-optimized models have not yielded the same level of predictive success), or other imaging techniques (MEG, EEG, etc.).

An important aspect of the network dissection used here is that it studies units in isolation. In several cases, semantic concepts may be encoded by a combination of multiple units (voxels). Future studies could extend network dissection techniques to understand the properties of simulated population responses in underexplored regions of the visual cortex, whose precise functional characterization remains elusive. Even though single neurons or single voxels may not exhibit high selectivity for object categories in those areas of the brain, we can ask whether populations of neurons or voxels in these regions encode and represent human-understandable concepts using novel network interpretability techniques⁷⁶. Other than identifying new categories that might be represented in a distributed manner, analyzing selectivity at a multi-voxel pattern might also reveal a refinement of the voxel-level selectivity (e.g, one sub-group can be selective for groups of people and not just the presence of a face). Another way in which our interpretation analyses can be extended is to not only study high-level category selectivity, but instead use response-optimized models to study low-level feature representations and perceptual processes that are known to activate the ventral visual stream^77–79.

Our work demonstrates that a less hypothesis-committed approach can complement hypothesis-driven studies of the visual cortex in meaningful ways. This empirical approach, enabled by advances in data science, and large-scale compilation and dissemination of neural data, can offer a complementary basis for building broader theories about neural computations, that generalize to a range of ethologically relevant scenarios.

Methods

Natural scenes dataset

A detailed description of the Natural Scenes Dataset (NSD; http://naturalscenesdataset.org) is provided elsewhere³². Here, we briefly summarize the data acquisition and preprocessing steps. The NSD dataset contains measurements of fMRI responses from 8 participants (two males and six females) who each viewed 9000–10,000 distinct color natural scenes (22,000–30,000 trials) over the course of 30–40 scan sessions. Scanning was conducted at 7T using whole-brain gradient-echo EPI at 1.8-mm resolution and 1.6-s repetition time. Images were taken from the Microsoft Common Objects in Context (COCO) database⁸⁰, square cropped, and presented at a size of 8.4^∘ x 8.4^∘. A special set of 1000 images were shared across subjects; the remaining images were mutually exclusive across subjects. Images were presented for 3 s with 1-s gaps in between images. Subjects fixated centrally and performed a long-term continuous recognition task on the images. Informed consent was obtained from the participants and the study was approved by the University of Minnesota Institutional Review Board.

The fMRI data were pre-processed by performing one temporal interpolation (to correct for slice time differences) and one spatial interpolation (to correct for head motion). A general linear model in which the HRF is estimated for each voxel and the GLMdenoise technique is used for denoising was then used to estimate single-trial beta weights. Betas provided in the subject-native space (func1pt8mm) were used in all of our experiments. Every stimulus considered in this study had 3 repetitions. We averaged single-trial betas after z-scoring every voxel within each scan session to create our voxel responses. For each of four frequently studied categories (faces, places, bodies and word), we chose one previously reported ROI. The 4 ROIs considered in this study were thus the Fusiform face area (FFA, includes FFA1 and FFA2), Extrastriate body area (EBA), Visual word form area (VWFA) and Retrosplenial cortex (RSC). In the NSD dataset, the masks for those ROIs are provided, and result from running the floc localizer⁸¹ and collecting 7T fMRI recordings. This localizer consists of presenting grayscale images from ten categories in 4s mini-blocks, where each image is shown for 0.5 seconds, and each of the ten categories is presented six times, along with blank trials. The experiment included six runs of 300 seconds each. The ten categories were grouped into five domains (characters, bodies, faces, places and objects). Allen et al.³² pre-processed this data and applied a GLM analysis to obtain t-test maps for each of the five domains. Then, a liberal threshold was used to obtain masks for the ROIs. Note that when selecting the voxels for our analyses for a given area, we pool from the voxels in the masks from both hemispheres. If there is any asymmetry in the size of a given ROI, it will be expressed in the number of voxels that belong to the mask in either hemisphere. For instance, across the four NSD participants, 68% of the localized VWFA voxels belonged to the left hemisphere. The right lateralization of FFA was also noticeable: 58% of the localized voxels were located within the right hemisphere.

Response-optimized encoding model

We trained separate voxel-level predictive models for each of the above category-selective regions with the same backbone architecture. The predictive model comprises a shared convolutional neural network core common across all subjects that represents the feature space unique for specific visual areas. We employ a linear readout model on top of the feature space to predict the responses of individual voxels in a specific region of interest under the assumption that the feature space likely represents the input received by these areas and these regions perform close-to-linear transformations on this input. A linear readout on a shared feature space is further based upon the often made assumption that the activity across a set of neurons or voxels in one individual can be related to the activity of the second individual in the homologous functional region by a linear transform⁴⁴. Further, the linear readout is also factorized into spatial and feature dimensions following popular methods for neural system identification. This allows us to separate spatial tuning or receptive field locations (i.e., what portion of the sensory space is the voxel most sensitive to?) from feature tuning (i.e. what features of the visual input is the voxel sensitive to?).

The base feature extraction network or the core thus performs all nonlinear transformations to convert the raw sensory stimuli (i.e., pixels) into a representation characteristic of a particular visual area, whereas the readout linearly maps this extracted representation into voxel responses. The core consists of four sequential convolutional blocks, with each block comprising the following feedforward computations: two convolutional layers each followed by an inner batch norm and nonlinear activation (ReLU) operations and an anti-aliased AvgPool operation at the end. Instead of regular convolutions, we employ E(2)-steerable convolutions in the core of all our models to compute orientation dependent activations for many different orientations, thereby achieving joint equivariance under translations and rotations by design^43,82,83. This enables us to apply filters not just in every spatial location, as in a standard convolutional layer, but also in every orientation, increasing parameter sharing and improving the statistical efficiency of deep learning. This modeling choice is also inspired by neural computations in early visual areas where it is hypothesized that groups of neurons perform similar computations at different orientations, e.g., edge or curve detection at different orientations. We further evaluate the usefulness of our design choices in Appendix D and Supplementary Fig. S5.

Architectural details

Our core consists of four convolutional blocks, each comprising two convolutional layers. Each convolutional layer contains 48 feature sets where each feature is extracted at 8 orientations, resulting in 384 (48 × 8) feature maps. The core is thus equivariant under rotations by 45 degrees. Naive filter rotations can result in aliasing artifacts. From an implementation perspective, the filters in these equivariant convolutional operations are constructed as a linear combination of a fixed system of atomic filters, which helps avoid artifacts and enables arbitrary angular resolution with respect to sampled filter rotation⁸². The first convolutional block contains layers with a filter size of 5 while all the remaining layers have a filter size of 3. Each convolutional layer is followed by a BatchNorm and ReLU operation. Each block is followed by an Average Pooling operation with a stride of 2 (except for the last Average Pooling where we use a stride of 1). For Average Pooling and BatchNorm, we use the implementations provided for steerable convolutional neural networks so that the feature fields are processed in a way that ensures equivariance⁸³.

The readout contains all voxel-specific parameters and maps the representation extracted from the core (with dimensions 384 × 28 × 28) to individual voxel responses. Weights of the readout are a sum of outer products between a spatial filter (of dimensions 28 × 28 per voxel) and a feature vector (a 384-D vector per voxel). The spatial filter further had a positivity constraint (enforced using rectification) and was normalized independently for each voxel by dividing each spatial weight by the square-root of the sum of squared spatial weights across all locations.

Training and testing models

Input preprocessing

All input images were scaled to 224 × 224 pixels for computational efficiency. Further, all images were normalized according to the ImageNet mean ([0.485, 0.456, 0.406]) and standard deviation ([0.229, 0.224, 0.225]) for both response-optimized and task-optimized models.

Model optimization

To develop the model, we use the first 4 of 8 subjects. Among those 4 subjects, the dataset comprises 37,000 natural scene images, among which 1000 images are shared across all subjects and the rest are exclusive to each subject. We used the 1000 shared images for testing our models and split the remaining stimulus set into 35,000 training and 2000 validation images.

All parameters of the response-optimized model were optimized jointly to minimize the mean squared error between the predicted and measured response. Since for every image in the training set, the response is measured from only a single subject and not all subjects, we use a masked mean squared loss to train the model across multiple subjects. Let $r_{s, v}^{p r e d .}$ and $r_{s, v}^{m e a s .}$ denote the predicted and measured response of voxel v in subject s to image i, respectively. Then, the loss function during training is given, as

L = \sum_{i \in Batch} \sum_{s = 1}^{S} \sum_{v = 1}^{n_{s}} 1_{i \in I_{s}} {(r_{s, v}^{p r e d} - r_{s, v}^{m e a s .})}^{2}

where $1_{i \in I_{s}}$ is the indicator variable specifying if image i was shown to subject s. The proposed method allows us to propagate errors through the shared network even if the subjects are not exposed to common stimuli since we can always exclude the subjects/voxels for which the response is not present from mean error calculation within each batch. The shared network thus benefits from diverse, varying stimuli across subjects with less extensive constraints on data collection from single subjects. Models were trained for a maximum of 100 epochs using Adam with a learning rate of 1e–4, a batch size of 16 and early stopping (patience = 20) based on the Pearson’s correlation coefficient between the predicted and measures responses on the validation set; validation curves were monitored to ensure convergence.

We measure performance (‘predictive accuracy’) on 1,000 test images by computing the Pearson’s correlation coefficient between the predicted and measured fMRI response at each voxel.

Baseline models

Task-optimized models

We compared response-optimized models against standard task-optimized models which have shown state-of-the-art performance in predicting neural responses in the primate visual cortex. In all comparisons, we employed an AlexNet architecture⁸⁴ optimized for object recognition on the large-scale ImageNet dataset⁸⁵. The pre-trained model was downloaded from: https://pytorch.org/hub/pytorch_vision_alexnet/. We extracted features from intermediate layers of this network and employed the same spatial x feature factorized readout as used in the response-optimized networks to linearly map layer activations to brain voxel responses in each region. We selected the model layer that maximally predicted the brain responses in each region on a validation set(Conv-5 for all considered high-level visual areas). The readout parameters for task-optimized models were optimized independently for each visual region using the same training protocol as the response-optimized models. Thus, the readout models were trained for a maximum of 100 epochs using Adam with a learning rate of 1e–4 and a batch size of 16. We further applied an early stopping criterion (patience = 20) based on the Pearson’s correlation coefficient between the predicted and measures responses on the validation set.

Categorical models

Category ideal observer models employ the category membership of labeled objects in the image to predict the responses evoked by the image. Unlike task-optimized and the proposed response-optimized models, categorical models are not image-computable and rely on annotations generate by human observers. These oracle models have absolute access to the categories present in an image and have previously been shown to explain substantial variance in image representations in both macaque and human IT^21,86. One might expect their performance to be even higher for explaining image representations in category-selective visual clusters in high-level cortex. We obtained object category labels for every NSD image from the MS COCO database⁸⁰. The input to the categorical model is thus an 80-D binary vector corresponding to the 80 object categories annotated in the database, where each element indicates whether the corresponding category was present in the image or absent (note that NSD images contain multiple objects per image). We fitted l₂ regularized linear regression models (known as ridge regression) on this representational space to find weights corresponding to different categories for every voxel. The regularization parameter was optimized independently for each subject and for voxels in each visual area by testing among 8 log-spaced values in [1e–4, 1e4].

Generalization to new subjects

We tested the generalization of different competing models to a set of 4 new subjects in the Natural Scenes Dataset by freezing the weights of these models and only training linear readouts on top of their fixed representations. The readouts had the same architecture as the original models, i.e., they were factorized into spatial and feature dimensions. The spatial map had a positivity constraint and was again normalized independently for each voxel by dividing each spatial weight by the square-root of the sum of squared spatial weights across all locations. The linear readouts were trained with a gradient descent procedure (Adam optimizer, learning rate 1–4) that minimizes the mean squared error between the voxel-wise predictions and true measured responses. For each new subject, the dataset comprised brain responses for 5,445 stimuli with 3 repetitions (remaining stimuli had either 1 or 2 repetitions and were discarded from analysis). We varied the size of the training set from a mere 100 stimulus-response pairs to a large set of 4,500 pairs. The readouts were trained independently for each subject, each training set size and each visual ROI for 100 epochs with an early stopping criterion (patience of 20). We monitor the loss on a validation set comprising 430 samples. The performance is evaluated on an independent set of 515 stimulus-response pairs shared across the new subjects using Pearson’s correlation coefficient (R). These stimuli were not encountered during the training of response-optimized models on the original set of 4 subjects. An identical readout training procedure was followed for both response-optimized and task-optimized models. In the case of categorical models, we follow the same procedure that we applied on the original set of subjects and fit l₂ regularized linear regression models on the categorical representational space to find weights corresponding to different categories for every voxel. The regularization parameter was optimized independently for each subject, each training set size and for voxels in each visual area by testing among 8 log-spaced values in [1e–4, 1e4].

Quantifying the semantic selectivity of voxels

High-level visual concepts are generally verbalizable, and throughout this paper, we refer to voxels that encode and represent these concepts as ‘semantically selective’. To quantify the selectivity of voxels for different human-interpretable concept categories, we adapt the previously proposed framework of ‘Network dissection’^35,36 to our brain response-optimized models. We see this as a fine-grained approach to characterize voxels that looks at not just the image-level category labels but rather dense pixel-level segmentations across thousands of cluttered natural scenes to characterize a voxel. The probe dataset used for quantifying the semantic selectivity of voxels comprises 36,500 held-out images from the validation set of the large-scale Places365 dataset. The reference segmentation for these probe images comes from the Unified Perceptual Parsing image segmentation network⁸⁷ previously trained on 20,000 scene-centric images from the ADE20k dataset⁸⁸. The ADE20k dataset is exhaustively and densely annotated with objects, parts of objects and in some cases, even parts of parts. This reference segmentation assigns every pixel a semantic label from a large vocabulary of human-interpretable concepts, comprising 335 object classes, 1,452 object parts and 25 materials.

A unique advantage of the factorized readout employed in this study is that it allows us to disentangle spatial selectivity from feature selectivity. To enable the network dissection procedure to be applicable to our models, we first discard the learned spatial selectivity of every voxel and use the learned feature tuning of every voxel to create an additional 1 × 1 convolutional layer, so that every voxel is represented by an independent unit in this convolutional layer. These units are used to characterize the semantic selectivities of voxels irrespective of the position of the respective semantic categories in the visual field. This yields a model that is entirely convolutional. Dissecting the last layer of this model (which has as many units as the number of voxels in the ROI) reveals the semantic selectivities of all voxels. We only performed the dissection procedure on voxels that had a held-out raw prediction accuracy (R) > 0.1 and the total number of such voxels in each visual ROI across 4 subjects are listed in Table 1. As proposed in ref. ³⁵, the selectivity of a particular unit (or voxel in our case) is quantified by computing the Intersection over Union (IoU) of the corresponding thresholded and binarized activations of that unit for a large number of images from the probe dataset against the reference segmentation. This metric computes the ratio of the number of pixels common between both spatial maps and the total number of pixels activated in either the reference segmentation map or the binarized activation map. A voxel is termed as semantically selectivity for a concept if its IoU with the reference segmentation of that concept is greater than 0.04. Further details about the dissection procedure employed in this study are described elsewhere³⁶. Our main models are trained on the voxels of one ROI at a time. We evaluate the effect of using a shared model across ROIs on network dissection in Appendix B and the effect of fitting the single ROI models to new subjects in Appendix C.

Table 1.

Number of dissected units in each model, i.e. the number of voxels that had raw prediction accuracy (R) greater than 0.1

ROI	Number of voxels dissected
FFA	2721
VWFA	1840
EBA	12,732
RSC	2336

Open in a new tab

Synthesizing maximally exciting inputs

We performed a qualitative feature visualization analysis to find the visual pattern that would maximally activate individual model neurons emulating brain voxels. Neural networks are differentiable with respect to their inputs. Starting from a random noise input, we use these gradients to iteratively move the input towards the goal of maximizing activation in individual model neurons. This visualization technique is commonly employed in neural network interpretability research to find the features that drive model neurons⁸⁹. Most visualization techniques further employ an image prior in the form of a regulariser to restrict the maximally exciting input to a suitable subset of the image space^90,91. Formally, the goal of finding the maximally exciting input (MEI) x* is then expressed as the following optimization problem.

x^{*} = {\arg \max}_{x \in R^{H \times W \times C}} A_{i j} (θ, x) + R (x)

where A_(i, j)(θ, x) denotes the activation of unit i from layer j in the neural network to input x (H: Height, W: Width, C: Channels), and θ denotes the parameters of the network. The latter are fixed during the above optimization procedure. $R (x)$ denotes the regulariser. In order to generate MEI for the jth voxel, we set i to the network output layer and j to be the index of the model neuron in the output layer that emulates voxel j. The above optimization problem is, in general, a non-convex optimization problem but we can find (at the very least) a local minimum by performing gradient ascent in the input space and updating x iteratively in the direction of the gradient of $A_{i j} (θ, x) + R (x)$ . We employed a general weak form of regularization, called ‘transformation robustness’⁹², where we stochastically jitter (up to 8 px), rotate (up to 10 degrees) and scale (by a factor between 0.8 to 1.2) the image before each optimization step so that the optimized images don’t contain high frequency noise. The images are optimized starting from random noise with Adam optimizer for 512 steps using a learning rate of 1e–3.

Visual diet experiments

In the first set of visual deprivation experiments, we removed all stimuli belonging to the ‘person’ category (containing faces as well as bodies). This leaves much fewer 17,325 stimulus-response pairs for training, as compared to the original 35,000 pairs that were used for training response-optimized models in the main experiment. We trained response-optimized models for two visual ROIs, namely FFA and EBA, with the same architecture and training procedure as employed for the original models, but only changed the visual diet to the aforementioned deprived set. In another set of experimental results, shown in Supplementary Fig. S6, we removed all stimuli containing animate things, finally resulting in 12,559 stimulus-response pairs and trained another set of models for FFA and EBA on this inanimate subset. These selective reductions in training set size and characteristics resulted in a small drop in the overall prediction accuracy on the complete test set, as reported in Supplementary Fig. S9, but did not result in selective disruptions for the out-of-domain category versus in-domain stimuli.

Transfer learning on fine-grained visual discrimination tasks

To formally test existing functional specialization accounts which implicate FFA in face perception, RSC in spatial cognition, VWFA and EBA in letter and body perception respectively, we simulated several fine-grained discrimination tasks with independent stimuli in response-optimized models of all brain regions, as mentioned below:

Face identity discrimination task: This includes stimuli from the MiniCelebA dataset which comprises facial images of 20 identities, each having 100/30/30 train/validation/test images⁵⁸.
Spatial layout estimation task: Here, the stimuli include 4000 diverse indoor scenes from the SUN database⁶⁵. These stimuli were split into sets of 3200 training, 400 validation and 400 test images. Each stimulus image has a corresponding label for the room layout type, where the layout categories were defined using a keypoint-based parameterization (as illustrated in Fig. 7). This helps us frame the room layout estimation task as a classification problem.
Letter classification: The printed letters dataset, generated in prior work⁶², contains 65,520 images of the 26 uppercase Latin letters. Here, the same letter appears in different fonts (out of fourteen), different sizes (out of five), different weights (bold or not bold), different styles (italicized or not italicized) and different pixel locations (out of nine), creating diverse visual appearances. To make the classification task more difficult, we used a maximum of 1000 images across all letters (split into 80%training and 20% validation) for training and reserved the remaining 64,520 images for testing.
Body pose classification: Here, the dataset includes images of multiple computer-graphics (CG)-generated models doing 5 different yoga poses⁶³, divided into 1006 (split into 80%training and 20% validation) and 501 test images. The images are rendered on multiple different backgrounds, and contain both male and female models with a variety of skin and hair tones, captured from different viewpoints and lighting.

We use response-optimized models to extract the predicted responses of voxels in every region to stimuli from the fine-grained visual categorization tasks. To ensure that the differences in performance of response-optimized models on fine-grained visual categorization are not driven by the differences in the number of voxels in every region, we selected the top 512 voxels in every region based on test correlations (i.e., correlation between the predicted and measured responses on 1000 test images). This number was chosen to match the dimensionality of representations from the pre-trained VGG16 architecture⁶⁰ optimized for face-recognition on the large-scale VGGFace2 dataset⁹³. We consider the performance of the VGG16 representation on the MiniCelebA dataset as an estimate of the upper bound on performance expected by our models on face recognition (we did not have access to human face recognition performance on this dataset). We also consider the 512-D dimensional representation from a VGG16 architecture trained on image categorization using the large-scale ImageNet database as an additional baseline for all visual discrimination tasks. This baseline was chosen because ImageNet-trained networks yield highly transferable representations that perform well in a range of vision-based tasks⁶¹.

We fitted l₂ regularized linear classification models (known as ridge classifiers) on these different representational spaces to predict the class label (e.g. facial identity or room layout type) of held-out stimuli. We vary the size of each transfer dataset from 60-100% of the maximum training set size and report the corresponding classification accuracy on the held-out set as a function of the transfer dataset size. For each of the above representational models (response-optimized, ImageNet-optimized or VGGFace2-optimized), the regularization parameter was optimized independently for each task (face recognition, spatial layout estimation, letter classification and body pose classification) and each training set size (60–100%) by testing among 10 log-spaced values in [1e–5, 1e5]. We selected the regularization parameter value that yielded best classification accuracy on the validation dataset.

Training a model on the entire ventral visual cortex

To test whether category selectivities may be uncovered with an impartial approach that does not require ROI selection or localization, we trained a single response-optimized model on responses from the entire ventral visual stream across all subjects. The model employed the same architecture as the ROI-specific models and was trained following the same procedure as employed for ROI-specific models and described above. The ventral visual stream voxels in every subject were extracted using the streams atlas released with the Natural Scenes Dataset³². Briefly, this ROI collection reflects large-scale divisions of the visual cortex into primary visual cortex and intermediate and high-level ventral, lateral and dorsal visual areas. These were manually drawn by NSD curators for each subject after accounting for the voxel-level reliability metrics. We extracted the ROI mask corresponding to the higher-level ventral stream label. This ROI was drawn to follow the anterior lingual sulcus (ALS), including the anterior lingual gyrus (ALG) on its inferior border and to follow the inferior lip of the inferior temporal sulcus (ITS) on its superior border. The anterior border was drawn based on the midpoint of the occipital temporal sulcus (OTS). As shown in Fig. 8, it is very broad (7000-9000 voxels per subject), yielding a total of 28,910 voxels across all 4 NSD subjects.

Noise ceiling estimation

Imperfect predictions of models are not solely due to model limitations, but may arise due to the inherent noise in the fMRI signal, which biases the prediction accuracy downward. Noise ceiling for every voxel represents the performance of the “true” model underlying the generation of the responses (the best achievable accuracy) given the noise in the measurements. They were computed using the standard procedure followed in ref. ³² by considering the variability in voxel responses across repeat scans. The dataset contains 3 different responses to each stimulus image for every voxel. In the estimation framework, the variance of the responses, $σ_{response}^{2}$ , are split into two components, the measurement noise $σ_{noise}^{2}$ and the variability between images of the noise free responses $σ_{signal}^{2}$ .

{\hat{σ}}_{response}^{2} = {\hat{σ}}_{signal}^{2} + {\hat{σ}}_{noise}^{2}

An estimate of the variability of the noise is given as ${\hat{σ}}_{noise}^{2} = \frac{1}{n} \sum_{i = 1}^{n} Var (β_{i})$ , where i denotes the image (among n images) and Var(β_i) denotes the variance of the response across repetitions of the same image. An estimate of the variability of the noise free signal is then given as,

{\hat{σ}}_{signal}^{2} = {\hat{σ}}_{response}^{2} - {\hat{σ}}_{noise}^{2}

Since the measured responses were z-scored, ${\hat{σ}}_{response}^{2} = 1$ and ${\hat{σ}}_{signal}^{2} = 1 - {\hat{σ}}_{noise}^{2}$ . The noise ceiling (n.c.) expressed in correlation units is thus given as $n . c . = \sqrt{\frac{{\hat{σ}}_{signal}^{2}}{{\hat{σ}}_{signal}^{2} + {\hat{σ}}_{noise}^{2}}}$ . The models were evaluated in terms of their ability to explain the average response across 3 trials (i.e., repetitions) of the stimulus. To account for this trial averaging, the noise ceiling is expressed as $n . c . = \sqrt{\frac{{\hat{σ}}_{signal}^{2}}{{\hat{σ}}_{signal}^{2} + {\hat{σ}}_{noise}^{2} / 3}}$ . We computed noise ceiling using this formulation for every voxel in each subject and expressed the noise-normalized prediction accuracy (R) as a percentage of this noise ceiling.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Supplementary information

Supplementary Information^{(16.5MB, pdf)}

Reporting Summary^{(974.3KB, pdf)}

Transparent Peer Review file^{(2.3MB, pdf)}

Acknowledgements

The authors thank the Natural Scenes Dataset (NSD) team for collecting and sharing the dataset. This effort was led by PIs Kendrick Kay and Thomas Naselaris, and includes contribution by Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest and J Benjamin Hutchinson. Note that the images in Figs. 4, 6e and Supplementary Fig. 10 were created using a machine learning algorithm that generates images that maximize the predicted activity in different brain regions, based on the brain response optimized models we introduce in this paper. Thus, even though these images are generated, they are directly about the response optimized models that form the core of this paper and they are used to understand these models. L.W. was funded by startup funds from the Machine Learning Department at Carnegie Mellon University and by a Google Faculty Research Award. M.K. was funded by startup funds from the Cognitive Science Department at UC San Diego.

Author contributions

M.K. and L.W. conceptualized the research. M.K. built the model and ran the analyses. M.K. and L.W. wrote the paper.

Peer review

Peer review information

Nature Communications thanks the anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

We use the open 7T fMRI Natural Scenes Dataset (NSD)³². The preprocessed dataset used in our study is available at 10.6084/m9.figshare.31849762. Source data are provided with this paper.

Code availability

The code used in this paper can be found at https://github.com/mkhosla-ucsd/response-optimized-modeling/.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-73938-9.

References

1.Hubel, D. H. & Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol.160, 106–154 (1962). [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Gallant, J. L., Braun, J. & Van Essen, D. C. Selectivity for polar, hyperbolic, and cartesian gratings in macaque visual cortex. Science259, 100–103 (1993). [DOI] [PubMed] [Google Scholar]
3.Ziemba, C. M. & Freeman, J. Representing “stuff” in visual cortex. Proc. Natl. Acad. Sci.112, 942–943 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Kanwisher, N., McDermott, J. & Chun, M. M. The fusiform face area: A module in human extrastriate cortex specialized for face perception. J. Neurosci.17, 4302–4311 (1997). [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H. & Livingstone, M. S. A cortical region consisting entirely of face-selective cells. Science311, 670–674 (2006). [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Pinsk, M. A., DeSimone, K., Moore, T., Gross, C. G. & Kastner, S. Representations of faces and body parts in macaque temporal cortex: A functional mri study. Proc. Natl. Acad. Sci.102, 6996–7001 (2005). [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment. Nature392, 598–601 (1998). [DOI] [PubMed] [Google Scholar]
8.Nasr, S. et al. Scene-selective cortical regions in human and nonhuman primates. J. Neurosci.31, 13771–13785 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Bell, A. H. et al. Relationship between functional magnetic resonance imaging-identified regions and neuronal category selectivity. J. Neurosci.31, 12229–12240 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Downing, P. E., Jiang, Y., Shuman, M. & Kanwisher, N. A cortical area selective for visual processing of the human body. Science293, 2470–2473 (2001). [DOI] [PubMed] [Google Scholar]
11.Peelen, M. V. & Downing, P. E. Selectivity for the human body in the fusiform gyrus. J. Neurophysiol.93, 603–608 (2005). [DOI] [PubMed] [Google Scholar]
12.Weiner, K. S. & Grill-Spector, K. Sparsely-distributed organization of face and limb activations in human ventral temporal cortex. Neuroimage52, 1559–1573 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Chao, L. L., Haxby, J. V. & Martin, A. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat. Neurosci.2, 913–919 (1999). [DOI] [PubMed] [Google Scholar]
14.Cohen, L. et al. The visual word form area: Spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain123, 291–307 (2000). [DOI] [PubMed] [Google Scholar]
15.Srihasam, K., Mandeville, J. B., Morocz, I. A., Sullivan, K. J. & Livingstone, M. S. Behavioral and anatomical consequences of early versus late symbol training in macaques. Neuron73, 608–619 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Grill-Spector, K. & Weiner, K. S. The functional architecture of the ventral temporal cortex and its role in categorization. Nat. Rev. Neurosci.15, 536–548 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Wu, M. C.-K., David, S. V. & Gallant, J. L. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci.29, 477–505 (2006). [DOI] [PubMed] [Google Scholar]
18.Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fmri. Neuroimage56, 400–410 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron76, 1210–1224 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Lescroart, M. D. & Gallant, J. L. Human scene-selective areas represent 3d configurations of surfaces. Neuron101, 178–192 (2019). [DOI] [PubMed] [Google Scholar]
21.Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl. Acad. Sci.111, 8619–8624 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Schrimpf, M. et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv 407007 (2020).
23.Schrimpf, M. et al. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron108, 413–423 (2020). [DOI] [PubMed] [Google Scholar]
24.Murty, N. A. R., Bashivan, P., Abate, A., DiCarlo, J. J. & Kanwisher, N. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nat. Commun.12, 5540 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Storrs, K. R., Kietzmann, T. C., Walther, A., Mehrer, J. & Kriegeskorte, N. Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting. J. Cogn. Neurosci.33, 2044–2064 (2021). [DOI] [PubMed] [Google Scholar]
26.Eickenberg, M., Gramfort, A., Varoquaux, G. & Thirion, B. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage152, 184–194 (2017). [DOI] [PubMed] [Google Scholar]
27.Wen, H. et al. Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex28, 4136–4160 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Khosla, M., Ngo, G. H., Jamison, K., Kuceyeski, A. & Sabuncu, M. R. Cortical response to naturalistic stimuli is largely predictable with deep neural networks. Sci. Adv.7, eabe7547 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Wang, A., Tarr, M. & Wehbe, L. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. Advances in Neural Information Processing Systems32 (2019).
30.Oliver, M. & Gallant, J. A deep convolutional energy model of v4 responses to natural movies. J. Vis.16, 876–876 (2016). [Google Scholar]
31.Seeliger, K. et al. End-to-end neural system identification with neural information flow. PLOS Comput. Biol.17, e1008558 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Allen, E. J. et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci.25, 116–126 (2022). [DOI] [PubMed] [Google Scholar]
33.St-Yves, G., Allen, E. J., Wu, Y., Kay, K. & Naselaris, T. Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex. bioRxiv (2022). [DOI] [PMC free article] [PubMed]
34.Brodmann, K. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues. Barth (1909).
35.Bau, D., Zhou, B., Khosla, A., Oliva, A. & Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition 6541–6549 (2017).
36.Bau, D. et al. Understanding the role of individual units in a deep neural network. Proc. Natl. Acad. Sci.117, 30071–30078 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Khosla, M., Murty, N. A. R. & Kanwisher, N. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology (2022). [DOI] [PMC free article] [PubMed]
38.Jain, N. et al. Food for thought: selectivity for food in human ventral visual cortex. bioRxiv (2022).
39.Pennock, I. M. L. et al. Color-biased regions in the ventral visual pathway are food-selective. bioRxiv (2022). [DOI] [PMC free article] [PubMed]
40.Hubel, D. H., Wiesel, T. N. & Stryker, M. P. Anatomical demonstration of orientation columns in macaque monkey. J. Comp. Neurol.177, 361–379 (1978). [DOI] [PubMed] [Google Scholar]
41.Tootell, R. B. & Hamilton, S. L. Functional anatomy of the second visual area (v2) in the macaque. J. Neurosci.9, 2620–2644 (1989). [DOI] [PMC free article] [PubMed] [Google Scholar]
42.Yue, X., Pourladian, I. S., Tootell, R. B. H. & Ungerleider, L. G. Curvature-processing network in macaque visual cortex. Proc. Natl. Acad. Sci.111, E3467–E3475 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
43.Ecker, A. S. et al. A rotation-equivariant convolutional neural network model of primary visual cortex. International Conference on Learning Representations (2019).
44.Guntupalli, J. S. et al. A model of representational spaces in human cortex. Cereb. Cortex26, 2919–2934 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
45.Klindt, D. A., Ecker, A. S., Euler, T. & Bethge, M. Neural system identification for large populations separating what and where. In Proceedings of the 31st International Conference on Neural Information Processing Systems 3509–3519 (2017).
46.Dehaene, S. & Cohen, L. The unique role of the visual word form area in reading. Trends Cogn. Sci.15, 254–262 (2011). [DOI] [PubMed] [Google Scholar]
47.Bonner, M. F. & Epstein, R. A. Coding of navigational affordances in the human visual system. Proc. Natl. Acad. Sci.114, 4793–4798 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
48.Mitchell, A. S., Czajkowski, R., Zhang, N., Jeffery, K. & Nelson, A. J. D. Retrosplenial cortex and its role in spatial cognition. Brain Neurosci. Adv.2, 2398212818757098 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
49.Nasr, S., Echavarria, C. E. & Tootell, R. B. H. Thinking outside the box: rectilinear shapes selectively activate scene-selective cortex. J. Neurosci.34, 6721–6735 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
50.King, J.-R. & Dehaene, S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci.18, 203–210 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
51.Toneva, M., Mitchell, T. M. & Wehbe, L. Combining computational controls with natural text reveals new aspects of meaning composition. bioRxiv (2020). [DOI] [PMC free article] [PubMed]
52.Toneva, M., Williams, J., Bollu, A., Dann, C. & Wehbe, L. Same cause; different effects in the brain. Proceeedings of the conference on Causal Learning and Reasoning (2022). [PMC free article] [PubMed]
53.Tavor, I. et al. Task-free mri predicts individual differences in brain activity during task performance. Science352, 216–220 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
54.Grill-Spector, K., Knouf, N. & Kanwisher, N. The fusiform face area subserves face perception, not generic within-category identification. Nat. Neurosci.7, 555–562 (2004). [DOI] [PubMed] [Google Scholar]
55.Parvizi, J. et al. Electrical stimulation of human fusiform face-selective regions distorts face perception. J. Neurosci.32, 14915–14920 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
56.Downing, P. E. & Peelen, M. V. The role of occipitotemporal body-selective regions in person perception. Cogn. Neurosci.2-4, 186–203 (2011). [DOI] [PubMed] [Google Scholar]
57.McCandliss, B. D., Cohen, L. & Dehaene, S. The visual word form area: expertise for reading in the fusiform gyrus. Trends Cogn. Sci.7, 293–299 (2003). [DOI] [PubMed] [Google Scholar]
58.Tran, D. T., Kiranyaz, S., Gabbouj, M. & Iosifidis, A. Pygop: A python library for generalized operational perceptron algorithms. Knowl. -Based Syst.182, 104801 (2019). [Google Scholar]
59.Liu, Z., Luo, P., Wang, X. & Tang, X. Large-scale celebfaces attributes (celeba) dataset. Retrieved August15, 11 (2018). [Google Scholar]
60.Parkhi, O. M., Vedaldi, A. & Zisserman, A. Deep face recognition. (2015).
61.Huh, M., Agrawal, P. & Efros, A. A. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, (2016).
62.Testolin, A., Stoianov, I. & Zorzi, M. Letter perception emerges from unsupervised deep learning and recycling of natural image features. Nat. Hum. Behav.1, 657–664 (2017). [DOI] [PubMed] [Google Scholar]
63.Body pose classification dataset. https://www.tensorflow.org/lite/tutorials/pose_classification#download_the_yoga_dataset. Accessed: 2022-10-17.
64.Zhang, Y. et al. Large-scale scene understanding challenge: Room layout estimation. In CVPR Workshop, (2015).
65.Xiao, J., Hays, J., Ehinger, K. A., Oliva, A. & Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition 3485–3492 (IEEE, 2010).
66.Chang, L., Egger, B., Vetter, T. & Tsao, D. Y. Explaining face representation in the primate brain using different computational models. Curr. Biol.31, 2785–2795 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
67.Grossman, S. et al. Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks. Nat. Commun.10, 1–13 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
68.Lindsay, G. W. Convolutional neural networks as a model of the visual system: Past, present, and future. J. Cogn. Neurosci.33, 2017–2031 (2021). [DOI] [PubMed] [Google Scholar]
69.Bowers, J. S. et al. On the importance of severely testing deep learning models of cognition. Cogn. Syst. Res.82, 101158 (2023). [Google Scholar]
70.Gu, Z. et al. Neurogen: Activation optimized image synthesis for discovery neuroscience. NeuroImage247, 118812 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
71.Luo, A., Henderson, M., Wehbe, L. & Tarr, M. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36 (2023).
72.Olah, C. et al. The building blocks of interpretability. Distill3, e10 (2018). [Google Scholar]
73.Vinken, K., Prince, J. S., Konkle, T. & Livingstone, M. S. The neural code for “face cells” is not face-specific. Sci. Adv.9, eadg1736 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
74.Sugita, Y. Face perception in monkeys reared with no exposure to faces. Proc. Natl. Acad. Sci.105, 394–398 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]
75.Khosla, M., Jamison, K., Kuceyeski, A. & Sabuncu, M. Characterizing the ventral visual stream with response-optimized neural encoding models. Adv. Neural Inf. Process. Syst.35, 9389–9402 (2022). [Google Scholar]
76.Fong, R. & Vedaldi, A. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 8730–8738 (2018).
77.Cox, D., Meyers, E. & Sinha, P. Contextually evoked object-specific responses in human visual cortex. Science304, 115–117 (2004). [DOI] [PubMed] [Google Scholar]
78.Henderson, M. M., Tarr, M. J. & Wehbe, L. Low-level tuning biases in higher visual cortex reflect the semantic informativeness of visual features. J. Vis.23, 8–8 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
79.Sarch, G., Tarr, M., Fragkiadaki, K. & Wehbe, L. Brain dissection: fmri-trained networks reveal spatial selectivity in the processing of natural images. Advances in Neural Information Processing Systems, 36, (2023).
80.Lin, T.-Y. et al. Microsoft coco: Common objects in context. In European conference on computer vision 740–755 (Springer, 2014).
81.Stigliani, A., Weiner, K. S. & Grill-Spector, K. Temporal processing capacity in high-level visual cortex is domain specific. J. Neurosci.35, 12412–12424 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
82.Weiler, M., Hamprecht, F. A. & Storath, M. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 849–858, (2018).
83.Weiler, M. & Cesa, G. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32 (2019).
84.Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25 (2012).
85.Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition 248–255 (Ieee, 2009).
86.Jozwik, K., Kriegeskorte, N., Cichy, R. M. & Mur, M. Deep convolutional neural networks, features, and categories perform similarly at explaining primate high-level visual representations. (2019).
87.Xiao, T., Liu, Y., Zhou, B., Jiang, Y. & Sun, J. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV) 418–434 (2018).
88.Zhou, B. et al. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition 633–641 (2017).
89.Erhan, D., Bengio, Y., Courville, A. & Vincent, P. Visualizing higher-layer features of a deep network. Univ. Montr.1341, 1 (2009). [Google Scholar]
90.Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, (2015).
91.Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition 5188–5196 (2015).
92.Olah, C., Mordvintsev, A. & Schubert, L. Feature visualization. Distill, https://distill.pub/2017/feature-visualization (2017).
93.Cao, Q., Shen, L., Xie, W., Parkhi, O. M. & Zisserman, A. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) 67–74 (IEEE, 2018).
94.Wilkinson, F. et al. An fmri study of the selective activation of human extrastriate form vision areas by radial and concentric gratings. Curr. Biol.10, 1455–1458 (2000). [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Information^{(16.5MB, pdf)}

Reporting Summary^{(974.3KB, pdf)}

Transparent Peer Review file^{(2.3MB, pdf)}

Data Availability Statement

We use the open 7T fMRI Natural Scenes Dataset (NSD)³². The preprocessed dataset used in our study is available at 10.6084/m9.figshare.31849762. Source data are provided with this paper.

The code used in this paper can be found at https://github.com/mkhosla-ucsd/response-optimized-modeling/.

[CR1] 1.Hubel, D. H. & Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol.160, 106–154 (1962). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR2] 2.Gallant, J. L., Braun, J. & Van Essen, D. C. Selectivity for polar, hyperbolic, and cartesian gratings in macaque visual cortex. Science259, 100–103 (1993). [DOI] [PubMed] [Google Scholar]

[CR3] 3.Ziemba, C. M. & Freeman, J. Representing “stuff” in visual cortex. Proc. Natl. Acad. Sci.112, 942–943 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR4] 4.Kanwisher, N., McDermott, J. & Chun, M. M. The fusiform face area: A module in human extrastriate cortex specialized for face perception. J. Neurosci.17, 4302–4311 (1997). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR5] 5.Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H. & Livingstone, M. S. A cortical region consisting entirely of face-selective cells. Science311, 670–674 (2006). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR6] 6.Pinsk, M. A., DeSimone, K., Moore, T., Gross, C. G. & Kastner, S. Representations of faces and body parts in macaque temporal cortex: A functional mri study. Proc. Natl. Acad. Sci.102, 6996–7001 (2005). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR7] 7.Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment. Nature392, 598–601 (1998). [DOI] [PubMed] [Google Scholar]

[CR8] 8.Nasr, S. et al. Scene-selective cortical regions in human and nonhuman primates. J. Neurosci.31, 13771–13785 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR9] 9.Bell, A. H. et al. Relationship between functional magnetic resonance imaging-identified regions and neuronal category selectivity. J. Neurosci.31, 12229–12240 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR10] 10.Downing, P. E., Jiang, Y., Shuman, M. & Kanwisher, N. A cortical area selective for visual processing of the human body. Science293, 2470–2473 (2001). [DOI] [PubMed] [Google Scholar]

[CR11] 11.Peelen, M. V. & Downing, P. E. Selectivity for the human body in the fusiform gyrus. J. Neurophysiol.93, 603–608 (2005). [DOI] [PubMed] [Google Scholar]

[CR12] 12.Weiner, K. S. & Grill-Spector, K. Sparsely-distributed organization of face and limb activations in human ventral temporal cortex. Neuroimage52, 1559–1573 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR13] 13.Chao, L. L., Haxby, J. V. & Martin, A. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat. Neurosci.2, 913–919 (1999). [DOI] [PubMed] [Google Scholar]

[CR14] 14.Cohen, L. et al. The visual word form area: Spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain123, 291–307 (2000). [DOI] [PubMed] [Google Scholar]

[CR15] 15.Srihasam, K., Mandeville, J. B., Morocz, I. A., Sullivan, K. J. & Livingstone, M. S. Behavioral and anatomical consequences of early versus late symbol training in macaques. Neuron73, 608–619 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR16] 16.Grill-Spector, K. & Weiner, K. S. The functional architecture of the ventral temporal cortex and its role in categorization. Nat. Rev. Neurosci.15, 536–548 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR17] 17.Wu, M. C.-K., David, S. V. & Gallant, J. L. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci.29, 477–505 (2006). [DOI] [PubMed] [Google Scholar]

[CR18] 18.Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fmri. Neuroimage56, 400–410 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR19] 19.Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron76, 1210–1224 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR20] 20.Lescroart, M. D. & Gallant, J. L. Human scene-selective areas represent 3d configurations of surfaces. Neuron101, 178–192 (2019). [DOI] [PubMed] [Google Scholar]

[CR21] 21.Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl. Acad. Sci.111, 8619–8624 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR22] 22.Schrimpf, M. et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv 407007 (2020).

[CR23] 23.Schrimpf, M. et al. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron108, 413–423 (2020). [DOI] [PubMed] [Google Scholar]

[CR24] 24.Murty, N. A. R., Bashivan, P., Abate, A., DiCarlo, J. J. & Kanwisher, N. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nat. Commun.12, 5540 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR25] 25.Storrs, K. R., Kietzmann, T. C., Walther, A., Mehrer, J. & Kriegeskorte, N. Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting. J. Cogn. Neurosci.33, 2044–2064 (2021). [DOI] [PubMed] [Google Scholar]

[CR26] 26.Eickenberg, M., Gramfort, A., Varoquaux, G. & Thirion, B. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage152, 184–194 (2017). [DOI] [PubMed] [Google Scholar]

[CR27] 27.Wen, H. et al. Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex28, 4136–4160 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR28] 28.Khosla, M., Ngo, G. H., Jamison, K., Kuceyeski, A. & Sabuncu, M. R. Cortical response to naturalistic stimuli is largely predictable with deep neural networks. Sci. Adv.7, eabe7547 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR29] 29.Wang, A., Tarr, M. & Wehbe, L. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. Advances in Neural Information Processing Systems32 (2019).

[CR30] 30.Oliver, M. & Gallant, J. A deep convolutional energy model of v4 responses to natural movies. J. Vis.16, 876–876 (2016). [Google Scholar]

[CR31] 31.Seeliger, K. et al. End-to-end neural system identification with neural information flow. PLOS Comput. Biol.17, e1008558 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR32] 32.Allen, E. J. et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci.25, 116–126 (2022). [DOI] [PubMed] [Google Scholar]

[CR33] 33.St-Yves, G., Allen, E. J., Wu, Y., Kay, K. & Naselaris, T. Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex. bioRxiv (2022). [DOI] [PMC free article] [PubMed]

[CR34] 34.Brodmann, K. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues. Barth (1909).

[CR35] 35.Bau, D., Zhou, B., Khosla, A., Oliva, A. & Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition 6541–6549 (2017).

[CR36] 36.Bau, D. et al. Understanding the role of individual units in a deep neural network. Proc. Natl. Acad. Sci.117, 30071–30078 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR37] 37.Khosla, M., Murty, N. A. R. & Kanwisher, N. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology (2022). [DOI] [PMC free article] [PubMed]

[CR38] 38.Jain, N. et al. Food for thought: selectivity for food in human ventral visual cortex. bioRxiv (2022).

[CR39] 39.Pennock, I. M. L. et al. Color-biased regions in the ventral visual pathway are food-selective. bioRxiv (2022). [DOI] [PMC free article] [PubMed]

[CR40] 40.Hubel, D. H., Wiesel, T. N. & Stryker, M. P. Anatomical demonstration of orientation columns in macaque monkey. J. Comp. Neurol.177, 361–379 (1978). [DOI] [PubMed] [Google Scholar]

[CR41] 41.Tootell, R. B. & Hamilton, S. L. Functional anatomy of the second visual area (v2) in the macaque. J. Neurosci.9, 2620–2644 (1989). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR42] 42.Yue, X., Pourladian, I. S., Tootell, R. B. H. & Ungerleider, L. G. Curvature-processing network in macaque visual cortex. Proc. Natl. Acad. Sci.111, E3467–E3475 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR43] 43.Ecker, A. S. et al. A rotation-equivariant convolutional neural network model of primary visual cortex. International Conference on Learning Representations (2019).

[CR44] 44.Guntupalli, J. S. et al. A model of representational spaces in human cortex. Cereb. Cortex26, 2919–2934 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR45] 45.Klindt, D. A., Ecker, A. S., Euler, T. & Bethge, M. Neural system identification for large populations separating what and where. In Proceedings of the 31st International Conference on Neural Information Processing Systems 3509–3519 (2017).

[CR46] 46.Dehaene, S. & Cohen, L. The unique role of the visual word form area in reading. Trends Cogn. Sci.15, 254–262 (2011). [DOI] [PubMed] [Google Scholar]

[CR47] 47.Bonner, M. F. & Epstein, R. A. Coding of navigational affordances in the human visual system. Proc. Natl. Acad. Sci.114, 4793–4798 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR48] 48.Mitchell, A. S., Czajkowski, R., Zhang, N., Jeffery, K. & Nelson, A. J. D. Retrosplenial cortex and its role in spatial cognition. Brain Neurosci. Adv.2, 2398212818757098 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR49] 49.Nasr, S., Echavarria, C. E. & Tootell, R. B. H. Thinking outside the box: rectilinear shapes selectively activate scene-selective cortex. J. Neurosci.34, 6721–6735 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR50] 50.King, J.-R. & Dehaene, S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci.18, 203–210 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR51] 51.Toneva, M., Mitchell, T. M. & Wehbe, L. Combining computational controls with natural text reveals new aspects of meaning composition. bioRxiv (2020). [DOI] [PMC free article] [PubMed]

[CR52] 52.Toneva, M., Williams, J., Bollu, A., Dann, C. & Wehbe, L. Same cause; different effects in the brain. Proceeedings of the conference on Causal Learning and Reasoning (2022). [PMC free article] [PubMed]

[CR53] 53.Tavor, I. et al. Task-free mri predicts individual differences in brain activity during task performance. Science352, 216–220 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR54] 54.Grill-Spector, K., Knouf, N. & Kanwisher, N. The fusiform face area subserves face perception, not generic within-category identification. Nat. Neurosci.7, 555–562 (2004). [DOI] [PubMed] [Google Scholar]

[CR55] 55.Parvizi, J. et al. Electrical stimulation of human fusiform face-selective regions distorts face perception. J. Neurosci.32, 14915–14920 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR56] 56.Downing, P. E. & Peelen, M. V. The role of occipitotemporal body-selective regions in person perception. Cogn. Neurosci.2-4, 186–203 (2011). [DOI] [PubMed] [Google Scholar]

[CR57] 57.McCandliss, B. D., Cohen, L. & Dehaene, S. The visual word form area: expertise for reading in the fusiform gyrus. Trends Cogn. Sci.7, 293–299 (2003). [DOI] [PubMed] [Google Scholar]

[CR58] 58.Tran, D. T., Kiranyaz, S., Gabbouj, M. & Iosifidis, A. Pygop: A python library for generalized operational perceptron algorithms. Knowl. -Based Syst.182, 104801 (2019). [Google Scholar]

[CR59] 59.Liu, Z., Luo, P., Wang, X. & Tang, X. Large-scale celebfaces attributes (celeba) dataset. Retrieved August15, 11 (2018). [Google Scholar]

[CR60] 60.Parkhi, O. M., Vedaldi, A. & Zisserman, A. Deep face recognition. (2015).

[CR61] 61.Huh, M., Agrawal, P. & Efros, A. A. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, (2016).

[CR62] 62.Testolin, A., Stoianov, I. & Zorzi, M. Letter perception emerges from unsupervised deep learning and recycling of natural image features. Nat. Hum. Behav.1, 657–664 (2017). [DOI] [PubMed] [Google Scholar]

[CR63] 63.Body pose classification dataset. https://www.tensorflow.org/lite/tutorials/pose_classification#download_the_yoga_dataset. Accessed: 2022-10-17.

[CR64] 64.Zhang, Y. et al. Large-scale scene understanding challenge: Room layout estimation. In CVPR Workshop, (2015).

[CR65] 65.Xiao, J., Hays, J., Ehinger, K. A., Oliva, A. & Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition 3485–3492 (IEEE, 2010).

[CR66] 66.Chang, L., Egger, B., Vetter, T. & Tsao, D. Y. Explaining face representation in the primate brain using different computational models. Curr. Biol.31, 2785–2795 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR67] 67.Grossman, S. et al. Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks. Nat. Commun.10, 1–13 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR68] 68.Lindsay, G. W. Convolutional neural networks as a model of the visual system: Past, present, and future. J. Cogn. Neurosci.33, 2017–2031 (2021). [DOI] [PubMed] [Google Scholar]

[CR69] 69.Bowers, J. S. et al. On the importance of severely testing deep learning models of cognition. Cogn. Syst. Res.82, 101158 (2023). [Google Scholar]

[CR70] 70.Gu, Z. et al. Neurogen: Activation optimized image synthesis for discovery neuroscience. NeuroImage247, 118812 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR71] 71.Luo, A., Henderson, M., Wehbe, L. & Tarr, M. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36 (2023).

[CR72] 72.Olah, C. et al. The building blocks of interpretability. Distill3, e10 (2018). [Google Scholar]

[CR73] 73.Vinken, K., Prince, J. S., Konkle, T. & Livingstone, M. S. The neural code for “face cells” is not face-specific. Sci. Adv.9, eadg1736 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR74] 74.Sugita, Y. Face perception in monkeys reared with no exposure to faces. Proc. Natl. Acad. Sci.105, 394–398 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR75] 75.Khosla, M., Jamison, K., Kuceyeski, A. & Sabuncu, M. Characterizing the ventral visual stream with response-optimized neural encoding models. Adv. Neural Inf. Process. Syst.35, 9389–9402 (2022). [Google Scholar]

[CR76] 76.Fong, R. & Vedaldi, A. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 8730–8738 (2018).

[CR77] 77.Cox, D., Meyers, E. & Sinha, P. Contextually evoked object-specific responses in human visual cortex. Science304, 115–117 (2004). [DOI] [PubMed] [Google Scholar]

[CR78] 78.Henderson, M. M., Tarr, M. J. & Wehbe, L. Low-level tuning biases in higher visual cortex reflect the semantic informativeness of visual features. J. Vis.23, 8–8 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR79] 79.Sarch, G., Tarr, M., Fragkiadaki, K. & Wehbe, L. Brain dissection: fmri-trained networks reveal spatial selectivity in the processing of natural images. Advances in Neural Information Processing Systems, 36, (2023).

[CR80] 80.Lin, T.-Y. et al. Microsoft coco: Common objects in context. In European conference on computer vision 740–755 (Springer, 2014).

[CR81] 81.Stigliani, A., Weiner, K. S. & Grill-Spector, K. Temporal processing capacity in high-level visual cortex is domain specific. J. Neurosci.35, 12412–12424 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR82] 82.Weiler, M., Hamprecht, F. A. & Storath, M. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 849–858, (2018).

[CR83] 83.Weiler, M. & Cesa, G. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32 (2019).

[CR84] 84.Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25 (2012).

[CR85] 85.Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition 248–255 (Ieee, 2009).

[CR86] 86.Jozwik, K., Kriegeskorte, N., Cichy, R. M. & Mur, M. Deep convolutional neural networks, features, and categories perform similarly at explaining primate high-level visual representations. (2019).

[CR87] 87.Xiao, T., Liu, Y., Zhou, B., Jiang, Y. & Sun, J. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV) 418–434 (2018).

[CR88] 88.Zhou, B. et al. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition 633–641 (2017).

[CR89] 89.Erhan, D., Bengio, Y., Courville, A. & Vincent, P. Visualizing higher-layer features of a deep network. Univ. Montr.1341, 1 (2009). [Google Scholar]

[CR90] 90.Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, (2015).

[CR91] 91.Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition 5188–5196 (2015).

[CR92] 92.Olah, C., Mordvintsev, A. & Schubert, L. Feature visualization. Distill, https://distill.pub/2017/feature-visualization (2017).

[CR93] 93.Cao, Q., Shen, L., Xie, W., Parkhi, O. M. & Zisserman, A. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) 67–74 (IEEE, 2018).

[CR94] 94.Wilkinson, F. et al. An fmri study of the selective activation of human extrastriate form vision areas by radial and concentric gratings. Curr. Biol.10, 1455–1458 (2000). [DOI] [PubMed] [Google Scholar]

PERMALINK

Higher visual areas act like domain-general filters with strong selectivity and functional specialization

Meenakshi Khosla

Leila Wehbe

Abstract

Introduction

Results

Accurate predictions on complex, cluttered scenes; rapid generalization to new subjects

Fig. 1. Performance of response-optimized and baseline models.

Network dissection reveals selectivity, tolerance and clutter-invariance

Fig. 2. Selectivity revealed by network dissection.

Fig. 3. Segmentation maps resulting from network dissection.

Maximally exciting images reveal structured high-level features consistent with previously hypothesized ROI role

Fig. 4. Maximally exciting inputs obtained through optimization.

End-to-end models capture tuning differences between voxels in the same ROI

Fig. 5. Voxel-level tuning characterization.

High selectivity persists in “face-deprived” and “body-deprived” networks

Fig. 6. Selective deprivation in visual diet.

Models reveal key functional distinctions between different regions

Fig. 7. Task performance of brain-response derived representations.

Hypothesis-agnosticity reveals a limited set of categories for which selective responses are observed in the ventral visual cortex

Fig. 8. Dissection (right) of a response-optimized model trained to predict responses across the entire ventral visual stream (left, shown for one participant).

Discussion

Brain response predictivity

Verbalization of voxel selectivity with network dissection

Visualization of voxel selectivity with image synthesis

Specific versus non-specific mechanisms in shaping categorical selectivity

Link between emergent representations and their functional roles

A possible complete characterization of selectivity in the ventral visual cortex?

Related studies

Extensions

Methods

Natural scenes dataset

Response-optimized encoding model

Architectural details

Training and testing models

Input preprocessing

Model optimization

Baseline models

Task-optimized models

Categorical models

Generalization to new subjects

Quantifying the semantic selectivity of voxels

Table 1.

Synthesizing maximally exciting inputs

Visual diet experiments

Transfer learning on fine-grained visual discrimination tasks

Training a model on the entire ventral visual cortex

Noise ceiling estimation

Reporting summary

Supplementary information

Acknowledgements

Author contributions

Peer review

Peer review information

Data availability

Code availability

Competing interests

Footnotes

Supplementary information

References

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases