INTEGRATION OF OBJECTS AND SPACE IN PERCEPTION AND MEMORY

Abstract

Distinct processing of objects and space has been an organizing principle for studying higher-level vision and medial temporal lobe memory. Here, however, we discuss how object and spatial information are in fact closely integrated in vision and memory. The ventral, object-processing visual pathway carries precise spatial information, transformed from retinotopic coordinates into relative dimensions. At the final stages of the ventral pathway (including area TEd), object-sensitive neurons are intermixed with neurons that process large-scale environmental space. TEd projects primarily to perirhinal cortex (PRC), which in turn projects to lateral entorhinal cortex (LEC). PRC and LEC also combine object and spatial information. For example, PRC and LEC neurons exhibit place fields that are evoked by landmark objects or the remembered locations of objects. Thus, spatial information, on both local and global scales, is deeply integrated into the ventral/temporal object-processing pathway in vision and memory.

Introduction

The fundamental insight that the visual hierarchy is divided into two pathways, ventral and dorsal¹, has guided research on visual cortex for decades, and has also influenced ideas about organization in prefrontal², auditory^3,4, and medial temporal lobe cortex⁵. The ventral (“what”) pathway is usually described as processing objects, whereas the dorsal (“where”) pathway is described as processing space (although the two pathways have also been described as processing perception [“what”] vs. processing action [“how”]^6,7). Recent research has refined and extended understanding of anatomy and function in the two visual pathways^7,8.

Here, we reexamine the object/space distinction for the ventral visual pathway and the medial temporal lobe processing stream it feeds. We discuss how spatial information, rather than being entirely segregated into a different pathway, is closely integrated with object processing throughout, in two senses. First, precise retinotopic spatial information about objects is not lost but instead transformed into relational dimensions. On the finest scales, neurons encode the regular, smooth relationships between points on boundaries and surfaces in the natural world. On a somewhat larger scale, neurons encode the positions of object fragments relative to each other and to the object as a whole. Second, information about large-scale, environmental space is closely intermixed with object information in the ventral visual pathway. This seems to support representation of object position within environments.

The two visual pathways continue into the medial temporal lobe memory system, in which the lateral entorhinal cortex (LEC) conveys ventral-pathway input to the hippocampus and the medial entorhinal cortex (MEC) conveys dorsal-pathway input^7–10 (Box 1). Episodic memory, defined as explicit memory of specific items or events tied to a specific spatiotemporal context, is fundamentally and inextricably tied to spatial processing¹¹. Many have proposed that the hippocampus is the site of binding of the “what” and “where” information to create and store conjunctive representations of experience that can be later retrieved and re-experienced as a conscious recollection of the original event^12–16. However, much evidence indicates that the ventral stream encodes spatial information at processing stages well before the hippocampus.

Transformation of retinotopic space into relational space

One of the defining features of the visual hierarchy is that receptive field size increases progressively at successive stages¹⁷. Concomitantly, retinotopic organization becomes gradually less clear. In the final stages of the ventral pathway in anterior temporal lobe (TE), receptive fields cover substantial bilateral portions of the visual field, making retinotopy coarse or absent^18,19. These strong trends naturally suggest that spatial information is discarded in the ventral pathway. Loss of spatial information could be regarded as a virtue, since a major goal of ventral pathway processing is to produce invariant representations of objects that do not depend on retinotopic position or size. Through either geometric transformations²⁰ and/or associative learning^21–23, TE could evolve stable signals for object identity completely independent of space.

Spatial information could be considered dispensable in this way for purely conceptual information like categorical identity. But object vision comprises much more than conceptual knowledge. In particular, we appreciate the detailed structure of objects and surfaces, on scales ranging down to millimeters. We do not just see a “dog”; we see a dog in glorious Technicolor, with all the subtle conformational characteristics that define its breed, all the variations and quirks that betray its individual identity, all the postural cues that reveal its emotional state and behavioral intentions, and all the incidental details that characterize a perceptual moment. We have immediate cognitive access to such information, allowing us to understand, manipulate, and verbally report on the precise structure of physical reality. We can explain, for example, how to differentiate dog breeds and read canine behavioral cues in terms of precise proportions, positions, and configurations of eyes, nose, lips, teeth, ears, neck, torso, limbs, toes, and tail. Thus, detailed spatial information about objects must be carried forward in explicit form to the final stages of the ventral pathway, the pathway that processes the finest-scale, foveal information and then communicates it to the rest of the brain^1,8,17–19. How can this be reconciled with the disappearance of retinotopic detail?

The perhaps obvious answer is that loss of retinotopy does not mean that spatial information is discarded or becomes cognitively inaccessible. Instead, it is transformed, into more useful, relational dimensions. While our cognitive access to absolute retinotopic image position is vague and coarse, we are acutely aware of relative positional relationships in the world. We don’t describe dogs in Cartesian image coordinates; we describe lengths, widths, diameters, aspect ratios, orientations, curvatures, attachments, relative distances and angles, and other measures of how one or more points or anatomical features relate to each other. As discussed below, the transformation of retinotopic space into relational dimensions is observable at the neural level throughout the ventral pathway.

Local spatial relationships: Neural coding of natural smoothness

Transformation into relative dimensions is represented in V1 by orientation tuning²⁴. Orientation is a spatial relationship between the points along an extended contour, such that the distances in the retinotopic x and y dimensions between any two points have the same ratio. It is a useful re-description for our natural world, in which physical boundaries have a high degree of smoothness, and thus constant orientation, on the scale of V1 receptive fields. A contour originally represented by many retinal photoreceptors can be re-described with a single orientation value. Complex cells, which generalize orientation tuning across a small span of visual space²⁵, implement an early trade-off of retinotopic accuracy for precise relational information.

On slightly larger scales, natural surfaces do not maintain a consistent orientation. But change in orientation, whether abrupt (corners) or gradual (curves), is itself a local spatial relationship that can be divorced from retinotopy. Thus, in V4, where receptive fields cover several degrees of visual angle (depending on eccentricity), tuning for change in orientation (curvature, the derivative of orientation) is prominent^26–34. V4 neurons are simultaneously tuned for both orientation and curvature, so that a given V4 neuron might respond to sharp convex angles pointing upwards or shallow concave curves opening to the left (Fig. 1A). These tuning characteristics are maintained across the larger V4 receptive fields (Fig. 1B), reflecting a further trade-off of retinotopic accuracy for relative spatial information about points along contours. (There is also evidence that V4 neurons can be tuned for spirality³⁵, a higher-order derivative that describes point relationships along some contours, for example the tails of dog breeds like Basenjis.)

Fig. 1. Transformation of retinotopic information into contour coding in area V4.

This neuron exemplifies how precise retinotopic information is recoded in terms of contour orientation, curvature, and object-relative position. The stimuli shown here (white shapes) were derived from a more wide-ranging test of shape sensitivity that revealed tuning for sharp convex curvature pointing (oriented) toward the upper right and positioned to the upper right of object center. (a) This fine-grained test shows gradual tuning for curvature (horizontal axis), orientation (vertical axis), and object-relative position (recursively plotted within axes) of the convex curvature. Response rate for each shape is indicated by background shade (see scale bar at right). (b) This test demonstrates how a shape with the critical convex curvature at its top right drives responses across a broad range of retinotopic positions (top), while a similar shape without this feature elicits little response (bottom). Adapted from Pasupathy and Connor (2001)²⁷.

These 2D orientations and curvatures in flat visual images typically reflect the orientations and curvatures of 3D structures in the real world. By the final stages of the ventral visual pathway neurons represent 3D surface orientation and curvature^36–40, and this representation is causally related to perception⁴¹. While 2D contour orientation occupies a polar domain, 3D surface orientation occupies a spherical domain: A surface can face towards you, away from you, to your right, to your left, upwards, or downwards, and anywhere in between. Neurons in TE are tuned for 3D surface orientations, with a predictable large bias toward orientations visible to the viewer³⁸, which span half the spherical space (the half represented by the near side of the moon in relationship to viewers on Earth; surface orientations on the moon’s far side cannot be seen). TE neurons are simultaneously tuned for 3D surface curvature, which is mathematically describable in terms of two “principal” cross-sectional curvatures, one maximum (most convex) and one minimum (most concave) (Fig. 2). A bump has convex max and min curvatures; a cylinder has convex and 0 (flat) curvatures; a dimple has concave curvatures, etc. TE neurons are tuned for a wide range of surface curvatures, with a strong bias toward convexity^38,40, which dominates the visible external surfaces of natural objects. By virtue of tuning for 3D orientation and curvature, TE neurons represent the regular spatial relationships between points across the smooth surface fragments that make up real world objects. These representations support detailed spatial perception of the infinite variety of bumps, dimples, ridges, creases and other features that can occur on natural surfaces⁴¹.

Fig. 2. Transformation of retinotopic information into surface coding in TE.

This neuron exemplifies how retinotopic spatial information is recoded in terms of 3D orientation, 3D curvature, and 3D object-relative position of surface fragments. (a) The response rates for this neuron were best fit by a model based on two multi-dimensional Gaussian tuning components that describe surface structure at a given point. Cyan and magenta circles mark the 1.0 s.d. boundaries for these Gaussians in minimum and maximum cross-sectional curvature (the most and least convex cross-sections through a point on the surface), 3D orientation (of a surface normal vector pointing away from the interior), x/y position, and z/y position (of a surface point, relative to object center of mass) (left to right). The response equation shows that only the product term had substantial weight, meaning that this neuron only responded to shapes with surfaces in both the cyan and magenta tuning ranges. The two tuning ranges were found with iterative fitting of the non-linear model, which required both tuning ranges even though they did not individually drive responses. (b) A high-response stimulus, generated by an adaptive algorithm responding to spike rates, shown in front view (left) and top view (right), with surfaces tinted to show regions within the two Gaussian tuning ranges. This neuron responded to objects with ridges (convex/flat curvature) facing the viewer and positioned in front of object center (magenta) and shallow flat/concave surfaces facing upwards and positioned near object center (cyan). (c–h) Responses were highly consistent across a wide range of transformations, as long as depth cues were present, demonstrating consistent coding of 3D surface shape. (c) Responses to highly effective (top), moderately effective (center) and minimally effective (bottom) stimuli with varying cues for shape in depth. Responses were strongest when stereo (binocular disparity) cues were present (black, dark green, dark blue). Responses remained substantial when only shading cues were present (gray). Responses collapsed when both stereo and shading were removed (light green, light blue). (d) Responses remained consistent across a 180° range of lighting directions in the horizontal plane (black line) and the sagittal plane (green line), which produced extremely different images. (e) Responses were consistent across a wide range of stereoscopic depths. (f) Tuning was consistent across a wide range of positions in the image plane. (g) Responses were consistent across a wide range of rotations in the image plane (around the z axis, blue line). There was less tolerance for rotations outside the image plane, around the x axis (black) or y axis (green). (h) Responses were consistent across two octaves of scale. Adapted from Yamane et al. (2008)³⁸.

Another prominent regularity in the natural world is medial axis structure—the cross-sectional symmetry of elongated structural elements often formed by biological growth processes or constructed according to engineering or aesthetic principles. Such structures can be efficiently described in terms of their extended axis of symmetry and the cross-sectional shape propagated along it^42–48. Many TE neurons encode these quantities simultaneously and thus represent the spatial structure of torsos, limbs, columns and beams in terms of smooth surface continuity along the paths defined by their medial axes (Fig. 3). (Late signals in V1 for 2D medial axes may reflect feedback from these TE representations⁴⁹.) These signals would support perception of spatial details like the lengths, diameters, curvatures, and musculature of a dog’s neck, chest, belly, thigh, etc.

Fig. 3. Transformation of retinotopic information into medial axis coding in TE.

This neuron exemplifies how retinotopic information is recoded in terms of medial symmetry axes (skeletal shape) of elongated, branching structures. (a) Results of an adaptive algorithm driven by spike rates. The response rate for each 3D shape is represented by background color (see scale bar at right), and stimuli are ordered by response rate from top left to bottom right. (b) Results from a simultaneous, independent stimulus lineage driven by the same adaptive algorithm. (c) Best-fit shape model applied to three high-response stimuli from the first lineage. Medial axis fit is represented by the red lines and surface fragment fits by the cyan- and green-tinted regions. (d) Best-fit shape model applied to three high response stimuli from the second lineage. Adapted from Hung et al. (2012)³⁹.

TE tuning for surface fragments and medial axis components is strikingly consistent across different image cues (shading, disparity; Fig. 2c), across different lighting directions that produce entirely different 2D images (Fig. 2d), across stereoscopic position in depth (Fig. 2e), across 2D position (Fig. 2f), across scale (Fig. 2g), across out-of-plane rotations of objects on the order of 60° (Fig. 2g), and across scale. In addition, responses of most neurons collapse when 3D cues (disparity and shading) are removed^38,39,50 (Fig. 2c). Thus, neurons in TE are no longer operating in retinotopic image space but rather in the 3D space of real physical structures.

Object-level relationships: Neural coding of spatial configurations

All of the spatial coding strategies discussed so far leverage some local smoothness or regularity in the natural world to transform retinotopic image space into relational descriptions of points along boundaries, surfaces, and symmetry axes. On larger scales, however, objects comprise entirely different parts with arbitrary spatial relationships and no surface continuity. Even on this larger scale, however, retinotopic space is transformed into relational signals. This is apparent by at least V4, where larger-scale retinotopic coding begins to give way to object-relative coordinates. V4 neurons that encode boundary orientation and curvature (see above), thus capturing detailed local spatial relationships, are also remarkably sensitive to object-relative position, thus capturing spatial relationships on the whole-object scale²⁷. The V4 example neuron (Fig. 1) tuned for convex curvature pointing to the upper right is also tuned for object-relative positions near the top right (Fig. 1a). In a cluttered environment, this relative spatial tuning is organized around the attended object^51,52. Together, V4 neurons span curvature, orientation, and object-relative position. As a result, V4 population response patterns represent boundary parts and where they occur, and thus the overall spatial configuration of an object⁵³.

Further along the ventral pathway, sensitivity to object-relative position remains acute as receptive field sizes increase and retinotopy fades. In addition, by at least TEO, neurons synthesize spatial configurations of multiple, disjoint parts^54,55. As a result, their response functions can only be described with equations that combine two or more tuning components, each defined in part by tuning for object-relative position (Fig. 2a). By the final stages in TE, neurons are tuned for 3D spatial configurations of surface fragments and/or medial axis elements^38,39 (Figs. 2, 3). Thus, the ventral pathway carries explicit signals for part-part and part-object spatial relationships.

Such signals must underlie our detailed understanding of 3D object structure, e.g. our ability to say that an antique silver teapot has a conical lid on top of a long, narrow neck that flows down into a round bottom, from which protrude an S-shaped spout on one side, a C-shaped handle in the same plane on exactly the opposite side, and four short legs oriented 45° degrees from vertical attached in a square configuration aligned with the spout and handle. Our perception of these structural relationships is precise, not coarse—if one part is even slightly misaligned, we recognize that the teapot is a cheap knockoff or a repaired original. And the percept is relational, not retinotopic—we fully appreciate the configuration of the teapot even as we turn it over to examine it from every angle, producing a confusing stream of retinotopic signals.

As noted above, explicit neural representation of and cognitive access to precise 3D structure is not necessary for recognition and discrimination. This point is beautifully illustrated by face discrimination. Humans and other primates are remarkably expert at discriminating and remembering thousands of faces based on extremely subtle, composite differences in the appearance and configuration of facial features (eyes, brows, nose, mouth, jaws, chin). Neurons in face-processing patches in anterior TE (ML/MF, AM) represent facial appearance so accurately that face photographs can be convincingly reconstructed from their population activity patterns⁵⁶. ML/MF neurons represent more information about larger-scale spatial configuration (e.g. face width and eye height), while AM neurons represent more information about finer details within features. However, this massive amount of spatial information is not represented with an explicit, easy to read code. Instead, neurons exhibit ramp-like tuning along specific directions in a high-dimensional space (on the order of 50D) in which each dimension represents composite changes in many features and configural relationships (Fig. 4)^56–58. This is a powerful strategy for discriminating thousands of faces that normally differ only in subtle, highly composite ways. It explains our seemingly unlimited capacity to distinguish thousands of essentially similar faces.

Fig. 4. Neural coding of face structure in highly composite dimensions.

This typical example neuron from face patch AM exhibits ramp-like tuning on a composite dimension along which forehead height, eyebrow shape, eye depth/surround contrast, nose length, mouth height, mouth shape, chin indentation, and face width all change gradually. This coding scheme is highly efficient for discriminating faces, but does not provide explicit, easily read signals for the underlying structure. Adapted with permission from the authors from Chang & Tsao (2017)⁵⁶.

The price, however, is that the underlying spatial structure of faces is largely buried in the complexity of the coding dimensionality. There are no explicit signals for things that determine facial appearance like the spatial relationship between the eyes and brows. Presumably, as a result, while I can instantly distinguish Scarlett Johansson, Jennifer Lawrence, Amy Adams, Emilia Clarke, and hundreds of other actresses (regardless of hair color and style), I cannot tell you what makes each woman’s face unique without deliberate, laborious measurement. Thus, identification can be superb without explicit neural representation of or cognitive access to the underlying spatial information. This argues that the explicit spatial coding observed for most objects exists to support not just recognition but also cognitive appreciation of structure.

Integration of objects with large-scale space

Large-scale spatial information about objects

The preceding section dealt entirely with spatial information about objects themselves. But objects exist within environments, and their relationships to and interactions with environments are inextricable aspects of object experience. This brings up another conundrum: If the ventral pathway achieves translation and scale invariance of object representations, mustn’t that entail loss of information about object/environment relationships? The simple answer to this one is that the ventral pathway does not throw away information about large-scale environmental space or object position. In fact, information about object position, scale, and orientation appears to increase along the ventral pathway, in parallel with information about categorical identity⁵⁹.

Moreover, the longstanding notion that the ventral pathway processes objects exclusively appears to be incorrect. Object coding is predominant in the uppermost channel through the ventral pathway, which in monkeys occupies the ventral bank of the superior temporal sulcus (STS). Below this channel, however, in TEd (dorsal TE), a majority of neurons respond strongly to large-scale environmental stimuli—landscapes and interiors—and only weakly to object-sized stimuli⁵⁰ (Fig. 5). These neurons are especially responsive to 3D planes, corners and edges, specifically within the orientation ranges that characterize natural landscapes and floors—the surfaces that most objects occupy in the real world⁴⁰. The upper/lower distinction in processing scale, surface curvature, and object/place organization is consistent across multiple studies^60–68. This organization may be inherited from retinotopic organization of early visual cortex^7,8,69.

Fig. 5. Large-scale environmental shape information in TE.

(a) Five representative neurons from channel TEd, on the lateral surface of the inferotemporal gyrus. Recording locations are shown in the coronal plane as colored dots superimposed on an MRI section. In each row, two high response stimuli are shown for a neuron. Responses are indicated by border colors (see scales at right). In each case, the left hand stimulus was generated in lineage 1 of a spike-adaptive shape algorithm and the right hand stimulus was generated in separate lineage 2. The arrows connecting each pair of stimuli color-code neuron identity. The majority of TEd neurons (66%), including these examples, were significantly more responsive to large-scale stimuli that exceeded the boundaries of the projection screen, which subtended 77° and 61° in the horizontal and vertical directions, respectively. (b) Five representative neurons from channel STS, in the ventral bank of the superior temporal sulcus. Conventions as in (a). The majority of STS neurons (75%) were more responsive to object-sized stimuli subtending on the order of 10° or less. (c) Average responses across all stimuli in lineage 1 (left) and 2 (right) as a function of stimulus size (maximum angular subtense), color-coded for the five TEd neurons. The correlations between these functions across lineages is given by the r values in (a). (d) Average responses as a function of stimulus size for the five STS neurons. Conventions as in (c). Adapted from Vaziri et al. (2014)⁵⁰.

The close juxtaposition of object and environment information in TEd is a natural basis for processing object-environment relationships and interactions. Significantly, the major cortical target for TEd is perirhinal cortex (PRC)⁸, the link between ventral pathway vision and medial temporal lobe memory^70–72. (In contrast, STS projects primarily to ventrolateral prefrontal cortex and orbitofrontal cortex, which are associated with short-term object memory and object value.) In some views ^73–75 (but see Ref. 76), PRC occupies the highest level of a hierarchy of object perception, binding together configurations of multiple attributes that define objects into a single neural representation. This binding includes the spatial arrangement of the components of an object⁷⁷. As discussed below, PRC and its distal targets carry forward the association between objects and environmental space inherited in part from TEd.

Object-based spatial processing: marking the locations of objects on a cognitive map

In order to serve as a useful guide to behavior and a framework for episodic memory, a cognitive map needs to incorporate representations of the locations of the objects and landmarks that are embedded within it. Current theories propose that the binding of objects to locations occurs in the hippocampus^12–16. However, increasing evidence shows that this binding may occur earlier in the processing stream. PRC, in conjunction with the hippocampus, is required for object-place association tasks, in which rats have to associate reward with a particular object in a particular location, but it is not required for simple discrimination of very different objects^78–79. Similarly, LEC and PRC processing is required to associate reward with specific objects within specific spatial contexts, even though simple object recognition and context recognition are intact^80–86. In spontaneous exploration studies, LEC lesions cause impairments in the ability to detect a spatial change of the configuration of objects when one of many objects (n > 3) is moved to a novel location^87–90. These studies show a clear role for the PRC/LEC pathway in object-space associations, but by themselves they do not reveal whether the PRC and/or LEC explicitly represent the spatial component of this association or whether they merely provide the object component to a downstream region (such as the hippocampus).

Early single-unit recording studies of PRC and LEC suggested that the contributions of these regions might be limited to providing object information. Neurons in PRC and IT of monkeys^91–93 and in PRC and LEC of rats^94,95 were responsive to 3-D objects or 2D images of objects. In both species, the neural responses to objects tended to decrease with repeated exposures. This “response suppression” was proposed to be a neural correlate of recognition memory^91,93,96. Other neurons encoded recency and familiarity of items, in line with the putative mnemonic functions of PRC and other MTL regions⁹².

More recent studies addressed the responsiveness of PRC and LEC neurons to 3-D objects in freely moving rats^97–99 (Fig. 6). Although many PRC and LEC neurons were active when the rats explored the objects, most did not discriminate strongly among the different objects. One possible interpretation is that these neurons do not encode objects per se, but rather the spatial locations of any salient objects that the rat encounters (i.e., they act as spatial pointers or drop pins on a map). There is a host of other interpretations, however (e.g., the cells may be encoding aspects of the exploratory behavior of the rat). A clue comes from studies in which objects are spatially displaced, similar to the spontaneous exploration lesion studies described above. In these experiments, a standard configuration of familiar objects is altered by moving one of the objects to a new location. Deshmukh and Knierim⁹⁷ reported that a small number of LEC cells fired not only at the new location of the object but also at the remembered location that the object had previously occupied (Fig. 7a). Tsao and colleagues¹⁰⁰ studied these rare “object trace” cells in detail and discovered remarkably that this object-place memory trace in LEC could last for at least 17 days. Similar findings of object trace activity were reported from neurons in anterior cingulate cortex¹⁰¹ and hippocampus¹⁰² (Fig. 7b,c). Because the objects were no longer present at these locations, the most likely interpretation is that these cells encoded the remembered locations that the objects had previously occupied.

Fig. 6. LEC responses to objects in freely moving rats.

(a) Recording apparatus containing 4 objects. (b) Each row is the firing rate map of a different LEC unit in 6 consecutive recording sessions. Sessions 1, 2, 4, and 6 were sessions with the familiar objects (white circles) in their standard spatial configuration. In Session 3, a novel object (white star) was placed into the arena. In Session 5, one or more objects were moved to novel locations. Unit 1 fired when the rat was in the proximity of each object. Unit 2 had a strong firing field at the location of one object, but when the object was moved in Session 5, the cell continued to fire at the original location. Units 3 and 4 were cells that had specific spatial firing in locations that were never occupied by an object. Unit 1 thus exhibited object-related firing, whereas units 2–4 showed spatial firing. PRC neurons similar to Unit 1, but not Units 2–4, were reported by Deshmukh et al (2012)⁹⁸. Reproduced from Deshmukh & Knierim (2011)⁹⁷.

Fig. 7. Object-space responses in LEC and hippocampus.

(a–c) Responses to remembered prior locations of objects. (a) A LEC neuron fired when the rat was in the proximity of one object in Session 1, and then fired at multiple objects in Session 2 when one of the objects was moved (magenta line connects old position [circle] and new position [star]). Note that the cell continued to fire weakly at the prior location of the moved object (white circle attached to magenta line). When the object was returned to its initial location in Session 3, the cell fired robustly at the location that the object had occupied in the previous session. From Deshmukh & Knierim (2011)⁹⁷. (b,c) Two units from the hippocampus that displayed object-location memory traces. Unit b had a place field along the left wall for sessions 1–4. In session 5, the cell fired when the rat was near a novel object, and maintained this firing in Session 6 after the novel object was removed. Unit c had a standard place field in Session 1. When a novel object was placed in the arena (Session 2), the cell did not respond, but it developed place fields at the previously occupied locations when the object was moved to new locations in Sessions 3 and 4. When the object was removed entirely (Session 5), the cell continued to fire at the 3 previously occupied locations of the object. From Deshmukh and Knierim (2013)¹⁰². (d) Four examples of “landmark vector cells” in the hippocampus. Each cell fires at a specific distance and allocentric bearing (denoted by magenta lines) relative to two or more objects in an environment. From Deshmukh & Knierim (2013)¹⁰².

Object-based spatial processing: defining locations relative to local objects

Spatial locations can be defined in multiple ways. Prominent, current models of the spatial firing of MEC grid cells and hippocampal place cells emphasize path integration computations and the calculation of distances and directions to extended, environmental boundaries¹⁰³. Spatial locations can also be defined relative to local object landmarks¹¹, which may be a function of the PRC and LEC given their role in object processing.

However, there is a conflicting literature about whether PRC/LEC lesions in rodents cause deficits in large-scale spatial tasks and whether cells in these regions show spatial firing properties (see Refs. 104 and 105 for comprehensive reviews of the lesion literature on this issue). Some studies showed little evidence of spatial functions of PRC and LEC. On quintessential, hippocampus-dependent spatial memory tasks, such as the Morris water maze and the Barnes circular platform, PRC and LEC lesions tend to have modest or no effects^{83,85,87,106–109} (but see Ref. 110). These results are consistent with early reports that PRC and LEC neurons do not display strong spatial firing when rats forage in an open field^{98,99,111–114}. Furthermore, LEC cells are weakly modulated by the theta rhythm compared to MEC, although some individual LEC cells show a modest phase-locking to theta¹¹⁵. Because the theta rhythm in rodents is strongly associated with movement through space, the lack of a strong theta signal in LEC reinforces its fundamentally different coding principles relative to the extremely specific and robust spatial coding of MEC^116,117.

Nonetheless, the responsiveness of upstream TEd neurons to landscape-scale scenes⁵⁰ (as described above) suggests that PRC and LEC might be involved in spatial processing at navigationally relevant spatial scales, at least in some tasks. Consistent with this prediction, PRC lesions cause a robust deficit in delayed nonmatch to position tasks^118–121 (but see Ref. 122), the radial 8-arm maze^{85,120,121,123,124}, and contextual fear conditioning^108,125,126 (but see Ref. 84). In a particularly compelling demonstration of the contribution of PRC to spatial memory in a plus-maze¹²⁷, control rats used an allocentric spatial strategy to solve the task (i.e., they chose an arm based on its spatial location in the room) whereas rats with PRC lesions used a response strategy (i.e., they chose an arm based on a left- or right-turn response or based on an intramaze floor cue). This change in strategies indicated that PRC was involved in allocentric spatial processing in the control rats¹²⁷. Finally, PRC neurons show broad, spatial selectivity in a visual-cue discrimination, spatial response task on a Figure-8 maze¹²⁸. These results support the idea that PRC neurons can display a degree of spatial representation, and that they do not only respond to discrete items or objects.

Animals can navigate to goals relative to local landmarks in an environment^129,130, and this type of navigation may depend on LEC processing^90,131. Consistent with this idea, some LEC neurons show place-field like responses in environments containing objects. These spatial firing fields can exist at locations distant from the objects, showing that they are spatial in nature and not simply responding to attributes of an object⁹⁷ (Fig. 6b, units 3 and 4). Such strong place fields have not been reported in environments that lack the local objects^97,113,114. They have also not been observed in PRC (although this negative result must be taken with caution given that they are rare [estimated < 10%⁹⁷] even in LEC and thus may have been missed in the PRC recordings⁹⁸). Despite this absence of spatial firing in PRC at the single-unit level of analysis, a hierarchical clustering analyses of PRC (and LEC) ensembles, recorded as rats performed a context-dependent, object-association task, revealed a significant signal related to spatial location (in addition to stronger signals related to context and objects)¹³². (Interestingly, a weak object-related signal was revealed in the space-dominated MEC ensemble. Thus, MEC and LEC/PRC showed evidence of both spatial and object-related activity, but the relative weights of each type of information differed between the two processing streams.)

There is strong evidence of object-relative spatial coding in the hippocampus. Many studies show responsiveness to the present (or remembered) locations of objects^{11,102,133–135}. Hippocampal cells can also encode locations defined by objects at a distance. Some hippocampal cells, called landmark vector cells, fire when the rat occupies multiple locations in an environment. Each location is defined by an identical distance and bearing from an object in the environment¹⁰² (Fig. 7d). Similarly, some hippocampal cells in bats fire when the bat is at a specific distance and bearing to a goal location¹³⁶. These results unequivocally demonstrate object-relative, spatial position coding. Conceivably, the hippocampus derives object-relative positions by combining LEC object location inputs with MEC distance and direction inputs.

Rethinking the roles of LEC and MEC

The hippocampus is thought to combine item/object information from the PRC/LEC pathway with spatial/temporal information from the parahippocampal (PHC)/MEC pathway, in order to represent the individual components of an experience within a spatiotemporal context^{10–16,137–140}. This conjunctive representation allows the components to be stored and later retrieved together, to be reconstructed as a coherent, episodic memory. However, as discussed here and argued elsewhere^{132,141–144}, spatial and object processing are already intertwined throughout the ventral stream, and it no longer seems accurate to characterize PRC/LEC as strictly object-related and PHC/MEC as strictly spatial. How then should the functions of the two pathways be described, and what is the precise role of the hippocampus in integrating these pathways?

Since PRC/LEC as well as PHC/MEC carry both what and where information (albeit to different degrees), the more accurate distinction might be how that information is used. The MEC, with its dense connectivity with retrosplenial cortex and presubiculum⁹, appears to be part of a path-integration-based navigation system that reports the moment-by-moment, allocentric position of the animal (or, under certain conditions when the animal is not moving, the passage of time¹⁴⁵). This system requires external sensory input to keep the position signal aligned to the external world (primarily via representations of the environmental borders and head direction, although distal landmarks are also influential). In contrast, the LEC system appears to represent primarily information about the external world, including (but not limited to) spatial information about objects in the environment and the animal’s location relative to these objects. In this view, the MEC might be part of what O’Keefe and Nadel¹¹ called the “internal navigation” system (path integration) and the LEC part of the “external navigation” system, relying on representations of local landmarks and their spatial relationships (see also ^141,142). Via anatomical crosstalk, these two systems interact to stay calibrated with each other. The hippocampus may be necessary for the rapid, one-trial binding of these representations to each other when they are novel or altered, on a fast time scale that is relevant for episodic encoding. In addition, through the process of hippocampal place cell remapping, the hippocampus may be crucial for creating context-specific, spatiotemporal representations of environments, and the events experienced in them, that are necessary for flexible, context-dependent learning and episodic memory.

Conclusions

Like many great ideas, the object/space distinction between ventral and dorsal pathways¹ has initiated a dialectical process leading to a more complex picture. Thus, while the dorsal pathway clearly emphasizes spatial information, it is now known to carry information about object shape^146–148 and object categories¹⁴⁹. Likewise, the spatial nature of the dorsal pathway is now viewed in part as a reflection of its role in guiding targeted actions in space^6,7.

Here, we have taken the ventral pathway identification with objects as a basis for examining the extent to which spatial processing is also involved. We have discussed how object shape processing is fundamentally a re-coding of local retinotopic spatial information in terms of common spatial configurations in the real world. This re-coding transforms the redundant, unreadable spatial information in 2D photoreceptor maps into compressed, explicit representations of 3D spatial structure. We have also discussed recent evidence that even large-scale space information is carried forward in parallel through the ventral visual pathway, providing a potential basis for perceiving object–scene relationships.

We next discussed how PRC and LEC, considered to be the continuation of the ventral object pathway into medial temporal lobe memory systems, also process spatial information, as demonstrated by lesion effects on spatial tasks. We examined how LEC place fields can be defined by object locations or even remembered object locations. We noted that the representation of space itself in the hippocampus can be organized with respect to landmark objects.

These observations reflect the fundamental nature of the world in which the brain must operate. It is a world in which all things are spatial and most important things are objects. Even at the highest levels of perception and cognition, objects do not become disembodied abstractions characterized only by semantic labels. They remain real things whose detailed meanings are defined by their precise spatial structures and their spatiotemporal interactions with the rest of the world. Likewise, space itself is not experienced as an independent abstraction, but as a dimensionality that organizes and is organized by the ecologically important objects it contains. The ventral and dorsal pathways treat objects and space differently, but they cannot treat them separately.