On the relationship between maps and domains in inferotemporal cortex

Abstract

How does the brain encode information about the environment? Decades of research have led to the pervasive notion that the object-processing pathway in primate cortex consists of multiple areas that are each specialized to process different object categories (such as faces, bodies, hands, non-face objects and scenes). The anatomical consistency and modularity of these regions have been interpreted as evidence that these regions are innately specialized. Here, we propose that ventral-stream modules do not represent clusters of circuits that each evolved to process some specific object category particularly important for survival, but instead reflect the effects of experience on a domain-general architecture that evolved to be able to adapt, within a lifetime, to its particular environment. Furthermore, we propose that the mechanisms underlying the development of domains are both evolutionarily old and universal across cortex. Topographic maps are fundamental, governing the development of specializations across systems, providing a framework for brain organization.

The primate ventral visual stream is responsible for our ability to recognize objects. Lesions to this part of the brain can impair object recognition, without affecting the ability to locate objects^1,2. Lesions in human ventral visual cortex can result in surprisingly specific deficits in object recognition, such as impairments in the ability to recognize faces, body parts, tools, text or places, without affecting the ability to recognize other object classes^3,4. Such specific deficits have led to the idea that different object categories must be represented by anatomically distinct parts of the ventral stream; such clustering of category selectivity has been extensively confirmed using functional MRI (fMRI) in both humans and monkeys^5–7. Electrophysiological recordings from monkeys and from humans have revealed that neurons in these regions are indeed selectively responsive to particular object categories^8,9.

The processing done by this part of our brain is remarkable: we can immediately recognize a particular familiar face, even though faces differ from each other only in subtle ways and despite the tremendous variation in the activity that any individual face may produce on our retina. The fluent reading you are currently performing is also an astonishing feat. Despite decades of research, fundamental questions remain regarding how we process faces and more generally recognize objects. Is the clustering into category-specific domains key to these abilities? Does having specific domains for particular categories indicate that our brains evolved specialized circuitry to recognize biologically important object categories such as conspecifics and suitable environments? Or do these specialized domains reflect how we learn to recognize the things we encounter? If these domains are innate, how could such remarkably specific selectivities get ‘wired up’? If their connections are determined by experience, why do virtually all humans have domains for different object categories in stereotyped neuroanatomical locations^10,11?

We previously addressed these questions by reviewing the development of face selectivity in inferotemporal cortex (IT)¹²; here, we back farther out and look for still more general principles to explain the localizations of these specializations, their correlations with other topographies and some recently reported cross-modalities. Asking how such selective circuits get wired up eschews the tempting teleological trap of accepting such circuitry as being ‘for’ what it does. That is, instead of assuming that domains in the ventral stream are ‘for’ what they do, in this Perspective we ask what mechanisms during development could cause different parts to become selective for some highly specific object categories and not others.

Ventral stream development

Innatist versus bottom-up models

There is a long history of attempts to determine whether particular behaviours are innate or learned, starting at least as early as the seventh century BC with the pharaoh Psammetichus I, who had two children raised by a shepherd who was forbidden to speak to them. The goal was to find out what language the children would speak spontaneously. There is an interesting assumption in this experiment: that, even though babies do not speak, language is nevertheless innate and a child needs only to mature for this innate ability to manifest itself. Although most people would now agree that any particular language must be learned, Chomsky influentially proposed that humans do possess innately wired brain circuits that underlie our language ability¹³. It has been similarly influentially argued that the category-selective domains, especially the face domains, are innate, and have evolved to support the uniquely social behaviour of humans and other primates¹⁴; we refer to this as the ‘innatist model’ or ‘top-down model’. We argue here that the burden of proof that lies with this high-level, anthropocentric view is not met. Instead, we aim to show that universal mechanisms of development, common to many species, can account for domain-specific brain regions; we refer to this as the ‘bottom-up model’.

The strongest arguments for the innatist model for ventral-stream domains are that they are found in stereotyped locations in both humans and monkeys and that domains exist for biologically important object categories, such as faces, bodies and places. But humans also have domains for tools and for text, and it is implausible that we evolved a domain for text, given how recently in our evolutionary history literacy has been prevalent. It has been argued that the text domain (the ‘visual word form area’ (VWFA)) represents recycling of a domain that was previously ‘for’ something else¹⁵. But does it necessarily follow that some high-level visual area is ‘for’ some function, just because in most humans it serves that function? If these high-level visual areas were produced by evolutionary selective pressures to recognize specific object categories, then they should not be co-opted efficiently by some other kind of stimulus, such as text, without retaining selectivity for the stimulus category they evolved to process.

Domain development.

Here, we summarize what little is known about the early development of visual-category domains. First, category-selective domains develop in consistent parts of cortex across individuals, but in different regions across species. The object-recognition pathway in monkeys, IT¹⁶, comprises cortex within and ventral to the superior temporal sulcus, areas TEO and TE. In humans, the object-recognition pathway, often referred to as ‘ventral occipital temporal cortex’ (VOT)¹⁷, comprises cortex within and around the fusiform, collateral and lateral occipital sulci. The localization of the ventral visual stream to different sulcal folds across species already indicates that the development of object-selective cortex is not under such rigid constraints as those that do result in localization of functional areas to consistent anatomical markers across primates, such as primary visual cortex to within and along the calcarine sulcus. Second, there is some direct evidence on the development of domains: adult-like face selectivity as measured using fMRI is not present at birth in either monkeys or humans^18,19 but develops over the first few months after birth, a time rich in face experience for the neonate (FIG. 1).

Fig. 1. Development of face selectivity in macaque and human infants.

Lack of face versus non-face object selectivity in macaque¹⁸ and human¹⁹ infants. a | Before approximately 200 days old, macaques do not show face > non-face-object selective regions, and after this age, face selectivity appears and is stable¹⁴⁸, as measured by functional MRI. b | Cerebral blood volume signal responses reveal that, before face and non-face-object domains become detectable, monkey inferotemporal cortex is responsive to visual stimuli, but not selective to image category¹⁴⁸. c | Regions that are face selective in adult human ventral occipital temporal cortex are not selective for faces over non-face objects in 4–6-month-old human infants, as reflected by percentage differences in blood oxygen level-dependent responses¹⁴ Some selectivity for scenes versus faces was observed in both these studies, but could reflect differences in visual-field stimulation (for example, centre versus periphery) by the stimuli used. FDR, false discovery rate; ROI, region of interest. Parts a and b adapted from REF.¹⁴⁸, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Part c adapted from REF.¹⁹, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).

In humans, the VWFA does not develop until school age and does not develop at all unless an individual has learned to read^20,21. Thus, development of a text domain requires text experience. It has been proposed that the stereotyped localization of the VWFA to the left occipitotemporal sulcus is due to its selective connectivity with language areas ^22,23. But monkeys, which do not have a language area, can develop domains that respond selectively to human symbols if the monkeys are intensively trained as juveniles to recognize symbols^22–25; furthermore, symbol use allows these monkeys increased accuracy in judging quantity²⁶. This result suggests that intensive experience alone is sufficient to produce a category-selective domain that may facilitate category expertise. Conversely, monkeys who are raised for the first year of life without seeing any kind of faces do not develop domains selective for faces over non-face objects, bodies or scenes (although their other domains are normal), indicating that experience is necessary for developing domains¹⁸ (FIG. 2a). The regions in IT of face-deprived monkeys that would be face selective in a control monkey respond more strongly to hands than to faces¹⁸. Thus, monkey studies indicate that extensive early experience of an object category is both necessary and sufficient for developing that category-selective domain, and that, at birth, IT lacks the specialized domain organization typically found in adults.

Fig. 2. Intrinsic and experience-dependent organization in macaque visual cortex.

a | Lack of face domains (left) but normal hand domains (right) in face-deprived monkeys (bottom row) as compared with controls (top row)¹⁸. Images show contrasts as determined with functional MRI. b | At birth, most of the cortex is made up of maps of the sensory periphery¹². This is a composite image illustrating the coverage of visual²⁷, somatosensory–motor²⁸ and auditory³⁴ maps. Maps of eccentricity in visual space cover occipital, temporal, posterior parietal and frontal eye fields. Maps of the body (face, hands and feet) cover areas within and around the central sulcus. Alternating representations of high (yellow, with white outline) and low (cyan, with black outline) tonotopic frequencies cover parts of the superior temporal gyrus. Part a adapted from REF.¹⁸, Springer Nature Limited. Part b adapted with permission from REF.²⁸, PNAS, and from REF.³⁴, Springer Nature Limited, using data from REFS^27,31.

Proto-architecture

What is present at birth in both human and monkey cortex is topographic maps^27,28. Virtually the entire visual system in newborn macaques consists of a series of retinotopic maps, with neighbouring neurons receiving input from neighbouring parts of the retina and therefore adjacent parts of visual space (FIG. 2b). Even higher visual areas in both monkeys and humans that were once considered non-retinotopic, such as human lateral occipital and ventral occipital cortex, and ventral temporal cortex in monkeys, have since been shown to be retinotopically organized in adults^29–33 as well as in newborn monkeys²⁷. More anteriorly, both somatosensory and motor areas are also already organized at birth into multiple body maps²⁸. It is likely that in between the multiple visual-field maps and body maps are, as in adults, multiple auditory maps of frequency^34,35. Thus, at birth, most of the neocortex consists of orderly spatial maps of the sensory and motor periphery, and what is innate is this machinery to systematically sample and respond to the environment.

In adult humans and macaques, face domains are retinotopically biased to central visual field (that is, face-selective neurons tend to have receptive fields that are located near fixation), and neuronal receptive fields in building and scene domains tend to be located in peripheral parts of the visual map^36,37. Furthermore, there is a centre-to-peripheral progression of processing text, faces, tools and scenes³⁸, as well as a gradient in processing objects according to real-world size³⁹, indicating a general retinotopic organizing principle for the localization of different category domains. Both top-down and bottom-up explanations have been offered for this retinotopy-category association^36,40. The top-down account posits that this association arises from the specific requirements for the recognition of different object categories; for example, face processing requires central scrutiny, and scene processing requires integration across the visual field^36,37,40. Thus, (innately determined) face domains would selectively recruit or retain central-visual-field inputs from antecedent areas, and (innately predetermined) scene domains recruit or retain peripheral inputs. By contrast, the bottom-up model proposes that, during development, category domain formation and localization are governed by pre-existing retinotopic organization³⁶.

In support of the bottom-up explanation for the retinotopy–category association, we found that in newborn macaques the parts of IT that are destined to become face selective are already centrally biased, and that the parts that are destined to become place domains are already peripherally biased^27,41. Thus, retinotopic maps precede category selectivity during development. The innatist interpretation of this fact is that proto-face domains start out with a central-field bias in order to facilitate face processing⁴¹. However, the central field is part of a larger-scale topographic map of visual space that spans object-selective cortex (FIG. 2b). Therefore, the development of the central-visual-field representation must be intrinsically tied to the development of the entire map in IT. It therefore seems implausible that face-domain localization could be predetermined independently of retinotopy as, if this were the case, the entire retinotopic map would have to conform to face-domain localization. More generally, finer-scale architecture must develop after, or at least not before, larger-scale architecture; it cannot be the reverse. That is, the details of a map can fill in only after the layout is established. The bottom-up interpretation of the fact that retinotopy precedes category selectivity would be that the maps, being present at birth, are primary, and there must be something about central vision at that level of the visual cortical hierarchy that facilitates the emergence of face selectivity there.

Retinotopy carries with it an organization for low-level and mid-level shape features, because acuity varies dramatically with eccentricity, as does receptive-field size. Central-visual-field neurons have tiny receptive fields, whereas peripheral-field neurons have huge fields⁴². Many cells in both cat and monkey visual cortex respond to short contours more strongly than to long contours^43,44, and such ‘end-stopped’ cells are prevalent in monkey V2 (REF.⁴⁵) and respond more strongly to curved contours and corners than to extended contours^43,45. Therefore, the optimum degree of curvature will scale with eccentricity, with central-visual-field regions preferring high curvature, and peripheral-visual-field regions preferring straighter contours.

Mistaking an elephant for a face

Face domains respond better to round curvy things, such as balls and clocks, and scene domains to rectilinear things⁴⁶. The innatist explanation for an association between category and curvature would be that the selectivity of face domains extends to curvy things because faces have a lot of curvy contours, and so curvy things appear similar to faces, and anything straight looks more like it could be part of a scene than a face. The bottom-up explanation would be the reverse: that, rather than category domains driving similar-shape localization, a systematic variation in shape preference may guide the stereotyped localization of different category domains. Furthermore, when fMRI is used to map responsiveness to curvy things versus straight things, not only does a ‘curvy > straight’ bias characterize face domains, it also distinguishes central-visual-field regions from peripheral-visual-field regions. This central–peripheral, curvy–straight correlation is present not just in category-selective cortex but also throughout the visual system, including primary visual cortex (V1), in monkeys²⁵ (FIG. 3a) and humans⁴⁷ (FIG. 3b). The fact that both these studies found such a strong correlation between curvature and eccentricity means that whatever causes neurons to prefer curved contours (which we propose is end-stopping) must be prevalent. The association between curvature and eccentricity extends beyond just category-selective regions into early visual areas. By the logic that a large-scale architecture must dominate a finer-scale architecture, category selectivity cannot be the driving force for curvature tuning and retinotopy being associated with particular parts of IT; instead, the causality must be the reverse.

Fig. 3. Topographic receptive-field tuning.

a | Curvature correlates with eccentricity throughout the macaque visual system²⁵. Maps of responses to peripheral minus central-visual-field stimuli (left), straight minus curvy stimuli (centre) and non-face objects minus faces (right), with examples of each stimulus type above each map²⁵. A, C, and P indicate anterior, central and posterior subdivisions of inferotemporal cortex, respectively. b | Curvature also correlates with eccentricity in the human visual system⁴⁷ The dashed white line represents the border between central and peripheral visual field in early visual cortex. The cyan–yellow scale correlates with curvilinear values of visual stimuli, such that the areas in red–yellow process curvy features and those in blue–cyan process rectilinear features. The locations of the fusiform face area (FFA; green outline), occipital face area (OFA; blue outline), occipital curvature preference patch (OCP; black outline) and fusiform curvature preference patch (FCP; white outline) are within areas that preferentially respond to curved features. By contrast, the location of the parahippocampal place area (PPA) is encompassed by the region responding preferentially to rectilinear features. The probabilistic locations of face domains (the OFA and FFA) and the scene domain (the PPA)¹⁴⁹ are superimposed on a human eccentricity map¹⁵⁰ (right). c | Gradient of shape selectivity in macaque inferotemporal cortex⁴⁸ that could reflect developmental origins in eccentricity-based low-level shape selectivity gradients. Colours show the correspondence between clustering in image shape space (top) and anatomical space (bottom). A, anterior; D, dorsal; P, posterior; RSC, retrosplenial cortex; STS, superior temporal sulcus; V, ventral; V1, primary visual cortex. Part a adapted from REF.²⁵, Springer Nature Limited. Part b adapted with permission from REF.⁴⁷, Elsevier (left) and generated using data from REFS^149,150. Part c adapted from REF.⁴⁸, Springer Nature Limited.

But we probably should not think of this eccentricity-derived shape selectivity as just end-stopping, curvature or retinotopy, because at every stage in the visual hierarchy inputs are combined to generate more complex and more abstract receptive-field properties. Such a combinatorial mechanism might result in something like the recently described gradient of shape selectivity in monkey IT⁴⁸ (FIG. 3c), with selectivity for ‘stubby’ things on the lip of the superior temporal sulcus (which maps to central visual field) grading to increasingly spiky things going more peripherally along the gyrus. The stubby-shape regions prefer images with a lot of curved contours (including faces), and spiky domains prefer images containing a lot of straight lines, sticking out. Thus, face and body domains, often postulated as spatially discrete regions, may reflect parts of a larger-scale map. The proportion of face-selective neurons is highest at the centre of a face patch and falls off at the patch edges^49,50, consistent with the notion that activity-dependent sorting mechanisms cause neural selectivity to change smoothly along the cortical surface^51,52. This parallels the smooth transitions in orientation and retinotopy in early visual cortex⁵³. Indeed, the apparent spatial discreteness of category-selective domains may be a manifestation of the contrasts and thresholds typically used to visualize category-selective regions (see, for example, Fig. 16 in REF.¹²), akin to the limited perspectives of the allegorical blind men interpreting different parts of an elephant. In anatomical tracer studies in IT⁵⁴ and early visual cortex⁵⁵, labelling appears ‘patchy’ (Supplementary Fig. 1): that is, labelled neurons in IT or VOT connected to regions into which a tracer is injected form discrete clusters. Knowing the underlying map organization of early visual cortex makes it clear that these patchy connections reflect connectivity between the same parts of the visual field across maps. We suggest that, rather than demonstrating connectivity between discrete domains, the connections between IT face patches may similarly reflect connections between corresponding parts of maps. The historically minded reader may recall a similar instance when the presence of colour-selective blobs in V1 was originally missed by Hubel and Wiesel and later discovered by Livingstone and Hubel once there were anatomical markers to distinguish them⁵⁶.

We mentioned earlier that one fact that favours the innatist model is that face, body and scene domains are always found in stereotyped anatomical locations in both humans and monkeys. However, these category domains are distributed along a dorsoventral–mediolateral anatomical trajectory that correlates with an eccentricity gradient, with faces represented more centrally than bodies and scenes represented more peripherally, in both humans and monkeys^{25,36–38,57}. Categories distinguished by animacy and real-world size are also distributed along this same dorsoventral–mediolateral gradient in temporal cortex, with small, animate things represented in central-visual-field parts of the map and large, inanimate things represented peripherally^39,58. Recently, these gradients were shown to be accounted for by low-level shape selectivity, rather than lexical category⁵⁹. Together, these studies illustrate an intrinsic link between visual categories and a map of visual space that carries with it biases for scale and shape features such as curvature.

However, although some studies find that category selectivity can be explained by low-level features^48,59,60, many have reported that the degree of category selectivity in the ventral visual stream, especially in more anterior areas, cannot be explained by low-level or mid-level features alone^61–66. We suggest that this is because postnatal activity-dependent plasticity further sculpts selectivity depending on what is experienced — by shifting and/or narrowing the tuning of neurons towards things that are regularly experienced; that is, experience sculpts a low-level eccentricity-based shape map into a higher-level organization of behaviourally meaningful categories. A gradient from shape-biased selectivity posteriorly to category-biased selectivity more anteriorly would be consistent with later and stronger experience-dependent plasticity going up the visual hierarchy, as has been suggested by the stronger effects of early strabismus (misalignment of the eyes) on areas beyond V1 (REFS^67,68). A retinotopic map automatically carries a low-level shape map, which could bias the central-visual-field part of monkey IT or human VOT to become selective to faces, which are made up of many curvy contours. For example, neurons in regions where face patches eventually develop may start out broadly responsive to concentric features, which are characteristic of faces, but also clocks, cookies and doughnuts. Over development, their tuning may refine to respond preferentially to faces, even particular faces, and less to other less frequently experienced or behaviourally important objects with similar features. Notably, even in adult monkeys, neurons in face patches respond to round things such as balls, cookies and clocks⁶⁹. The presence of topographic maps selective for image scale and curvature in newborn monkeys²⁷ provides further evidence that low-level and mid-level shape biases in IT provide the building blocks for domain development. The refinement of tuning to heavily experienced features (such as faces) throughout development would not break the underlying topographic shape map⁴⁸, but could result in expansion of the parts of the map selective for these particular features, resulting in an apparent growth of selectivity for some visual categories compared with others^70,71.

If the visual hierarchy starts out as a series of maps, this means that the brain is wired up at birth to systematically sample the visual environment. To what extent does experience of the environment modify this topography? Certainly postnatal activity is known to affect the circuitry of primary visual, auditory and somatosensory cortices: if one eye is inactive or less active during postnatal development, inputs from the seeing eye come to dominate visual cortex, and inputs from the inactive eye are permanently lost^72,73. Similarly, if visual input is filtered such that only a narrow range of orientations is experienced, cortex comes to over-represent the seen orientation^74–77. If sounds of a particular frequency range are excessively experienced during postnatal development, that range of frequencies comes to dominate auditory cortex^78,79. If a subset of whiskers on a rodent’s muzzle are cut during postnatal development, characteristic cytoarchitectonically distinct ‘barrel fields’ of somatosensory cortex representing those whiskers shrink and are taken over by adjacent whisker barrels⁸⁰. Thus, there is ample evidence that the early postnatal brain does modify its circuitry, at least in primary sensory cortices, to respond preferentially to experienced stimuli, constrained and guided by map topography. Given merely the fact that higher-level sensory areas receive their input from primary sensory areas, it follows that higher sensory areas should also come to be dominated by experienced stimuli. Indeed, there is some evidence that the same kinds of activity-dependent plasticity also hold in cortical regions beyond primary sensory areas^67,68. Therefore, IT or VOT is expected to become responsive to the most commonly encountered things in its environment.

Given the importance of activity-dependent plasticity, how young animals look at and experience their visual world could also bias where on this map category domains develop. From birth, the visual experience of human infants is heavily biased towards seeing faces⁸¹. Both human and monkey infants preferentially look at faces more than other shapes soon after birth^82–86. This is often given as evidence that face domains must be innate, but there is no evidence that cortical face domains are more involved in early face-looking behaviour than other cortical or even subcortical structures^12,83, and infant monkeys and humans look preferentially at faces before face domains are detectable.

In most infant face-looking studies, schematized faces are compared with scrambled faces, and seldom compared with objects of comparable spatial frequency composition. Indeed, faces turn out to have spatial frequency distributions that are most visible to neonates^87,88. Furthermore, small dark features in the upper half of the visual field may also be optimal stimuli for driving infant looking behaviour⁸⁹. However, identifying the intrinsic biases that promote face-looking behaviour is complicated by early reinforcement and learning by the infant. Preferential looking to the parental face increases over the first day after birth, indicating that even a few hours of experience has a behaviourally measurable effect on an infant’s face-looking behaviour⁹⁰. Thus, it is far from proven that infants look at faces because they have an innate predisposition to look at faces, rather than because the salient and dynamic features of faces fit the parameters of infant vision better than most other object categories.

One evolutionarily old structure that could support early looking behaviour towards top-heavy, high-contrast moving features is the superior colliculus. Given that the superior colliculus is directly involved in looking behaviour⁹¹, has an upper-field bias⁹² and is relatively mature at birth⁹³, this subcortical structure may be the reason why infants look at faces. Further support for the idea that face looking is not innate is the observation that normal monkeys look preferentially at faces, whereas face-deprived monkeys do not¹⁸. Note that an earlier study by Sugita reported that face-deprived monkeys look at faces more than at non-face objects, but the difference was reported to be not statistically significant⁹⁴. By contrast, Sugita did find a significant effect of postdeprivation experience on behaviour. Altogether, these data indicate that early face-looking behaviour is unlikely to be driven by cortical domains and therefore should not be taken as evidence for innate cortical face networks. However, how infants look at different things in their environment probably does constrain where on the cortical map those categories become localized, and so looking at faces would bias face-domain formation to central-visual-field locations, and scenes would be biased towards more peripheral parts of the visual field.

Structures without a function?

Although the stereotyped locations of domains in IT or VOT can be explained by postnatal experience acting on a domain-general architecture, several recent studies have reported distinct laminar and connectivity profiles of different domains. In humans, areas containing face and place domains were recently reported to show distinct cytoarchitectonics (that is, different patterns of cell density across the layers of cortex)¹¹, and face domains show selective interconnectivity with other face areas in both humans and monkeys^95–97. Furthermore, large-scale connectivity patterns in humans are predictive of the location of face domains⁹⁸, as well as of where the VWFA will appear in children⁹⁹. Are domains thus distinct cortical areas, with characteristic cytoarchitectures, cell-type distributions and intrinsic and extrinsic connectivities?

The anatomical differences between individual domains in IT or VOT might seem to support the idea that they are genetically predetermined. Indeed, face patches in macaques correspond to particular folds along the superior temporal sulcus that form in utero and have distinct laminar organization¹⁰⁰. However, these anatomical features are also present in face-deprived monkeys that lack face patches¹⁰⁰, demonstrating that anatomical features are insufficient to prove the presence of functional specializations and, instead, that the anatomy may correspond to features of the intrinsic topographic architecture, which also correlate with later-emerging domains. Furthermore, although differences in connectivity and circuitry may determine how ventral stream domains perform their highly selective computations, they may not be the reason why a particular domain performs a specific computation; indeed, causality may be the reverse. That is, the reason face patches become face patches may not be because they have predetermined face-specific cytoarchitecture and connectivity; rather, the distinct cytoarchitecture and connectivity of face patches may arise, or refine, because of how these regions are stimulated during development. To unwrap this argument, we need to look at cortical development in general.

Beyond just IT or VOT, different regions of cortex have long been distinguished on the basis of cytoarchitectonics¹⁰¹. These cytoarchitectonic divisions more often than not are also functionally distinct and have distinct intrinsic and extrinsic connectivities and different distributions of various cell types, molecular profiles and neurotransmitters. This has led to the idea that cells in different parts of the cortex are genetically programmed to express these unique features that allow them to perform computations characteristic of each area. However, individual functional maps do not align cleanly with cytoarchitectonic borders^11,102, and different cortical areas are more similar to each other than not: neocortical areas all share the same basic six-layered structure, with the same basic scheme of inputs to layer 4, thalamic outputs from layer 6, other subcortical projections from layer 5 and projections to other cortical areas from layers 2/3 (REFS^103,104). Different cortical areas all share the same basic cell types and neurotransmitters, although in different proportions, and they perform similar basic computations of input integration and gain control¹⁰⁴. There are multiple lines of evidence that cortex is multipotential (BOX 1) in the kinds of information it can process, rather than having specific circuitry for different functions.

At birth, inputs to V1 are segregated into ocular dominance domains, thought to form by activity-dependent learning rules originally proposed by Hebb¹⁰⁵ that reinforce correlated inputs and weaken uncorrelated inputs; prenatal waves of activity in the two retinas are independent, and therefore uncorrelated. At birth, V1 is also organized into orientation pinwheels, again thought to be generated by activity-dependent sorting of on and off inputs and lateral connections that encourage neurons to retain inputs similar to those of their neighbours.

Researchers used to argue about the function of ocular dominance columns and orientation columns; what they were ‘for’. In a seminal study, Law and Constantine-Paton implanted a third eye on a tadpole, which grew into a three-eyed frog¹⁰⁶. When they injected a tracer into the third eye, they saw ocular dominance columns in an animal that normally never has ocular dominance columns because, in normal frogs, inputs from the two eyes project entirely contralaterally (Supplementary Fig. 2a). This implies that ocular dominance columns (and probably orientation columns, too) are not ‘for’ anything; rather, they emerge merely as a consequence of a Hebbian clustering rule. This study reminds us that just because a brain region is correlated with a function, it does not necessarily exist because of that function. We propose that the same holds true for the stereotypical clustered architecture of IT or VOT. That is not to say that neurons in these domains are not required for these functions or do not promote a particular behaviour. Rather, the activity-dependent rules that promote clustering may also lead to experience-dependent refinement of neural tuning that supports specific perceptual processes, such as our ability to quickly and accurately recognize objects in our environment. Such simple underlying rules may be evolutionarily old optimizations for information coding in an unspecified environment. Given that neurons are more likely to correlate with, and therefore connect to, nearby neurons, clustering by category facilitates interconnectivity between neurons that code for similar things and therefore facilitates within-category comparisons, and so clustering probably optimizes processing of these categories. This may be especially true for categories such as faces that have a high degree of homogeneity across examples and would thus be naturally predisposed to recruiting similarly tuned neurons within a map of visual features. It remains an open question whether a brain could acquire similar abilities using completely distributed neural ensembles and what behavioural benefits, if any, clustering into domains might provide^9,107.

Notably, the spatial scale of category domains in IT or VOT is larger than the ocular dominance and orientation columns in V1 (Supplementary Fig. 2b). Can we attribute the large domains in IT to the same mechanisms that form smaller domains in V1? Nasr and Tootell mapped ocular dominance domains in human V1, and colour and disparity domains in human extrastriate cortex¹⁰⁸. The domains in V2 are coarser than in V1, and coarser in V3 than in V2, and possibly coarser still in V4 (Supplementary Fig. 2c). So, it is likely that domains just get larger going up the visual hierarchy (possibly owing to input convergence at each stage and therefore broader correlation patterns). If so, it would not be too surprising that the huge domains in the ventral stream emerge from the same activity-dependent rules that produce the mesoscale domain organization of V1.

All roads to Rome are topography

We propose that the category domains in IT arise because postnatal visual activity acts on a retinotopic proto-organization by the same kinds of activity-dependent sorting mechanisms that lead to ocular dominance and orientation domains in V1; indeed, these sorting mechanisms may be key to forming the protomaps themselves. Where on this map different categories arise is biased by a low-level shape map based on retinotopic eccentricity and scale and by how different things in the environment are most frequently viewed.

However, in the past few years, there have been several human fMRI studies on blind individuals presented with tactile or auditory stimuli that produce category-selective activations localized to similar parts of ventral temporal cortex as in sighted individuals (Supplementary Table 1). The interpretation of virtually all these studies is that there must be some innate predisposition for these regions to process particular high-level categories, regardless of the modality of the input, and that this bias is driven by top-down influences from still higher areas in the cortex. For example, in studies finding face selectivity in the fusiform sulcus in blind individuals presented with tactile faces¹⁰⁹, or sounds associated with faces¹¹⁰, a social area is proposed to provide top-down input that causes the face area to become selective to faces. Studies show that blind individuals reading Braille (tactile) or hearing auditory input that is associated with text activates a region that in sighted individuals is responsive to reading text^111,112. These studies propose that the VWFA is selectively connected to language areas early in development, and that it is this connectivity that predisposes this area to become selective for reading. However, the idea of an innate reading area seems inconsistent with how recently in human history literacy has become prevalent. There is also a report that the mediolateral gradation of the animate–inanimate category found in sighted individuals is found in blind individuals in response to auditory inputs¹¹³ Furthermore, there is a report that that small manipulatable things¹¹⁴ and large immovable things¹¹⁵ presented auditorily to blind individuals map to similar brain locations as they do for images of the same objects in sighted individuals.

It seems like a heavy burden to require top-down influences from higher cortical areas to guide the organization of ventral temporal cortex into domains selective for faces, scenes, body parts, text, big versus small things and animate versus inanimate things. This idea of top-down influences guiding IT or VOT domain formation is especially difficult to accept given that lower areas in the hierarchy preceded higher areas in evolution¹¹⁶, lower areas mature earlier than higher areas^117,118, feedforward projections precede feedback projections^119,120 and most inputs to IT or VOT arise predominantly from other visual areas, not higher areas^97,121.

Can a bottom-up explanation account for the localization of these category-selective responses in blind individuals? It has been proposed that cross-modal mapping (after loss of one modality) is generated by a reorganization of either thalamocortical pathways¹²² or corticocortical connectivity between sensory areas¹²³, with connections propagating throughout cortex via intrinsic map organization^27,28, or via multimodal intermediate regions such as parietal areas or parts of frontal cortex¹²⁴. Regardless of the specific pathway, we propose that the explanation for commonalities between original and substituted modalities lies in the global congruence and connectivity of maps that are conserved across evolution. In mice, monkeys and humans, there is shared orientation of the topographic representation of the sensory periphery across modalities (FIG. 4a): in both somatosensory and motor cortex, the face and upper body are mapped ventral in cortex, and the feet and lower body are mapped dorsal in cortex. This is also true in visual cortex: the lower-visual-field maps to dorsal occipital cortex, and the upper- field maps to ventral cortex. Indeed, the orientations of these different sensory maps are established by the same trophic factors, ephrins, early in development¹²⁵. We propose that topography-preserving long-range projections link up sensory maps, and that the congruency of the topographic maps enforces a regularity by which sensory modalities are interlinked.

Fig. 4. Congruence between sensory maps.

a | Global congruency of sensory map orientation in mice (left), macaques (middle) and humans (right). b | Congruence of visual and somatosensory maps in parietal cortex¹²⁶ Neurons responding to stimulation of central visual space also respond to touch on central parts of the face, whereas neurons responding to peripheral visual space respond to touch on more peripheral body parts. c | Connectivity between early visual cortex and inferotemporal cortex is predominantly along isoeccentricities¹³⁴. Neuronal tracer injection sites in central-visual-field parts of posterior inferotemporal cortex (dark red) are selectively connected to central-visual-field parts of early visual areas and. higher visual areas (lighter red). Injection sites in peripheral-visual-field parts of intermediate visual areas (dark green) are selectively connected to peripheral-visual-field parts of both lower and higher visual areas (light green). S1, primary somatosensory cortex; V1, primary visual cortex. Part b adapted with permission from REF.¹²⁶, The American Physiological Society. Part c adapted with permission from REF.¹³⁴, OUP.

Such congruency of sensory maps is apparent in multimodal association cortex, even in the absence of neurological damage. In parietal cortex, there are neurons that receive both somatosensory and visual input and have congruent visual and somatosensory fields¹²⁶ (FIG. 4b). Some visually responsive parietal cells also receive auditory input, with auditory-localization fields aligned with their visual receptive fields^127,128. Map congruency is also apparent in the alignment of visual, somatosensory and auditory representations across layers of the superior colliculus^129,130. Such linking of sensory maps provides an infrastructure for the correspondence of information about the environment across modalities. These connections may provide a pathway for inputs from other modalities when the normally dominant modality is damaged or missing.

To explain the localization of cross-modal face selectivity in blind individuals, we invoke the fact that connectivity within IT is along isoeccentricities¹³¹ (FIG. 4c), even in congenitally blind individuals^132–134. Isoeccentricity connectivity is broadly aligned anteroposteriorly across IT. We speculate that, projecting that connectivity, the central visual field aligns with low frequencies in the auditory map, and eventually with the face part of the somatosensory and motor body maps. It would follow that central-visual-field representations, which are biased by scale and by looking behaviour to become specialized during development for processing faces, would be, early in development, congruent with and inter connected with the low-frequency part of the auditory map (which itself becomes specialized for processing language¹³⁵) and with the face parts of the somatosensory map.

We further postulate that social areas in frontal lobes, rather than preceding and guiding the development of earlier sensory areas, selectively receive inputs from face IT, face body-map areas and vocalization areas owing to topography-preserving constraints, and it is those convergent inputs that define those areas as ‘social’. Connectivity with frontal cortex, and indeed across the entire cortex, is topographic^136,137. Furthermore, visual domains for faces, scenes, colours and depth may be organized topographically in macaque prefrontal cortex¹³⁸, probably reflecting topographic input from IT cortex^54,137. These prefrontal domains are probably responsive to multiple modalities¹³⁹ and suggest a convergence of topographic inputs across sensory modalities (for example, see REFS^136,140). Thus, the convergence of inputs from retinotopic maps of early visual areas could lead to topographic organization of experienced categories in mid-level areas such as IT or VOT, and the same principles can lead to still more abstract and multimodal topographies in higher areas such as prefrontal cortex (Supplementary Fig. 3).

The innatist argument that input from social areas drives face domains to respond selectively to faces requires that the social areas have a template for what is social and, in particular, what qualifies as a face. By contrast, the bottom-up model requires only that whatever the infant sees frequently in central visual field will sculpt the selectivity of central IT, and, for most monkey and human infants, that stimulus is faces¹⁴¹. Indeed, newborn humans see faces fully 25% of the time during waking, and mostly very close up⁸¹.

By extension, the activation of the text domain by Braille reading, or the activation of the fusiform gyrus by haptic or auditory face cues, is a small subset of the kinds of cross-modal activations observed in individuals deprived of one sensory modality, or even sometimes in controls^109,142. We predict that further studies that incorporate global topographic principles will find that such cross-modal activity shows anatomical specificities based on shared map axes. Indeed, somatosensory, motor and visual maps are congruent with respect to the environment and are already congruent at birth (FIG. 3) throughout the class of mammals. The fact that newborn precocial mammals, such as lambs, show visual and auditory orienting behaviour at birth¹⁴³ means that visual and motor maps, and visual and auditory maps, at least, are congruent and interconnected at birth. We propose that the apparent category selectivity of cross-modal activations in blind individuals is evidence for map-based congruities, not innate domain organization. Further observations of high-level congruities may reveal how high-level functions such as language develop from low-level topographic functional specializations present in evolutionarily older species.

Conclusions

We have described how the development of category selectivity in temporal cortex begins with the establishment of maps across the entire visual hierarchy. The axes of the primary maps are defined by molecular gradients and refined by activity-dependent synaptic reinforcement and pruning. It is unknown how the multiple higher maps are formed prenatally, but the same activity-dependent sorting mechanisms and waves of activity in peripheral organs and cortical areas^144–146 would promote smoothly organized topographies. These map axes are congruent across the entire brain, but become less anchored to peripheral inputs, more abstract and more multimodal going up the hierarchy, beyond early sensory areas. The maps within the visual system provide an eccentricity-based curvature organization that provides a proto-shape organization for the topographic organization of categories. During postnatal development, the proto-architecture is modified by daily experience to become selectively responsive to frequently encountered things, biased by low-level features and how these things are typically viewed. Differential cytoarchitectonics may also be acquired as a consequence of patterns of neuronal activity.

Marr¹⁴⁷ said, somewhat teleologically, that we need to understand the goal of a computation before asking how it is neuronally implemented; we suggest that asking how a circuit gets wired up may be even more informative. We believe that topographic maps and self-organizing plasticity rules acting both prenatally and postnatally may provide explanations for seemingly complex circuitries, without the necessity to argue that these circuits evolved to do exactly what they do.

Box 1 |. Cortex is cortex.

The organization of cortical areas revealed by electrophysiological recordings initially led to the hypothesis that the cortex is tiled by repeating computational units arranged in hierarchies^151,152. This idea of a repeating computational unit is consistent with how brains differ in size: across land mammals, brain size scales with body size, over five orders of magnitude, and this is almost entirely accounted for by changes in cortical surface area, not cortical thickness, suggesting that larger brains evolved by increasing the number of these units^104,153.

That cortex is pluripotent has been demonstrated directly in several ways. First, experiments in which the optic nerve was routed into somatosensory thalamus in hamsters or auditory thalamus in ferrets revealed visually driven cells in the host cortex with visual properties such as orientation and direction tuning as well as a 2D map of visual space^154–156, along with alterations in intrinsic connectivity¹⁵⁷ and corticocortical connectivity¹⁵⁸. Furthermore, the animals could use this misrouted pathway for visually guided behaviour¹⁵⁹. The second line of evidence that developing cortex is pluripotent and acquires its distinguishing features and its connectivity from exogenous cues comes from a series of heterologous transplants of fetal neocortex. The transplanted tissues acquire the cytoarchitectonic features and patterns of connectivity characteristic of the host location^160,161. Thus, the inputs, and in particular the pattern of activity in those inputs, to a cortical area determine its cytoarchitectonics, intrinsic connectivity, pattern of gene expression¹⁶² and projections to other areas.