Do we have an internal model of the outside world?

Abstract

Our phenomenal world remains stationary in spite of movements of the eyes, head and body. In addition, we can point or turn to objects in the surroundings whether or not they are in the field of view. In this review, I argue that these two features of experience and behaviour are related. The ability to interact with objects we cannot see implies an internal memory model of the surroundings, available to the motor system. And, because we maintain this ability when we move around, the model must be updated, so that the locations of object memories change continuously to provide accurate directional information. The model thus contains an internal representation of both the surroundings and the motions of the head and body: in other words, a stable representation of space. Recent functional MRI studies have provided strong evidence that this egocentric representation has a location in the precuneus, on the medial surface of the superior parietal cortex. This is a region previously identified with ‘self-centred mental imagery’, so it seems likely that the stable egocentric representation, required by the motor system, is also the source of our conscious percept of a stable world.

1. Introduction

My interest in representations of outside space in the brain began with an observation that people doing ordinary tasks in the real world often made quite accurate turns to objects in the blind hemisphere behind them. These turns involved eye, head and body movements, and were almost always accompanied by a blink that prevented vision for all but the last few milliseconds before gaze movement ceased [1]. These giant saccadic turns (up to 180°) were usually followed by a small correction saccade of about 10° that finally brought gaze onto the intended target (figure 1). The implication of this observation is that there must exist in the brain a mapping of the identities and locations of recently viewed objects that the motor system, including the eye movement system, has access to. The main purpose of this review is to explore some of the properties of this mapping, but before that, I will briefly review some current ideas about representations of seen and unseen regions of space.

Figure 1.

Landing points of large saccades involving eyes, head and body, made by one subject while making a cup of tea on three occasions. All saccades were made to target objects that were initially out of sight (>90° from current gaze direction), and were accompanied by long blinks. Black dots show landing points when gaze stopped moving, and the vestibulo-ocular reflex recommenced. Open circles are final landing points after a correction saccade made under visual control. (Adapted from [1].)

The visible world that we see is three-dimensional, and appears to be in high definition and full colour. It is easy to show that while this may be true in central vision—the region around the fovea—it is certainly not true of the whole visual field: resolution in the peripheral retina is surprisingly bad. In spite of more than 50 years' work on the physiology of the primary visual cortex, and the many extra-striate regions around it, it has become clear that, in Bridgeman's words [2, p. 58], ‘… no brain area contained a panoramic, high-acuity representation that corresponds to our perceptual experience’. If this is true, then how is our phenomenal world constructed? In a deservedly famous article in 1992, O'Regan [3] proposed that we do not actually need a representation of the surroundings that survives our frequent fixation changes. The world itself acts as an ‘outside memory’ and if we require detailed information, then we simply direct our central gaze to the region in space from which this can be obtained, and then subject it to the analytic power of the cortical visual pathway. This view is supported by the ‘change blindness’ studies of the 1990s [4], which showed that very little is preserved from one fixation to the next, and these findings seem to do away with the need for an elaborate mechanism to join up the retinal images that precede and follow each saccade. Each fixation thus represents a new enquiry. This account certainly fits with the probing way the eyes are used when we are engaged in visually demanding tasks [5]. However, in its strong form—that there is no global representation of the outside world at all—the outside memory idea fails to provide a means of deciding where in space we should look for pertinent information, nor a basis for explaining the relationship between vision and action.

We can look round a room, then close our eyes, and point with reasonable accuracy to the doors, windows and a few other features. Clearly, a minimal mapping of objects and locations is stored [6–8]. The motor system needs to have information about the location of objects relative to the head and ultimately the trunk, and this must be accurate enough to support a range of actions. That such a working representation does exist has been suspected for some time, and it has been referred to as the egocentric representation [6], the spatial image [7] and the parietal window [8]: as in an earlier account [9], I will use the term egocentric memory here.

2. Visual stability and saccades

Another venerable problem is that the retinal image is shifted by saccadic eye movements up to three times a second. These not only smear the image on the retina but also displace it, and yet we do not note these dislocations. Some argue that this is the visual stability problem: if we can understand how saccades are dealt with, then the problem of why the world appears stationary will be largely solved [10].

Duhamel et al. [11] found that cells in the lateral intraparietal (LIP) area, which have a retinotopic mapping, shift the contents of their receptive fields at the time of each saccade, so that they anticipate the features they will encounter on the completion of the saccade. This ‘remapping’ in area LIP, and other cortical areas [12], is brought about in part by reafferent signals from the superior colliculi signalling the size of the impending saccade [13]. It is easy to see that these receptive field shifts might make the saccadic image change a less abrupt event, but the retinotopic image shift that results from the saccade remains unchanged. It is thus not very clear how this transient remapping might affect either the perception of a scene or the relation between the line of sight and any intended motor action on objects in the world. It seems likely that the receptive field shifts that occur at the time of saccades, along with saccadic suppression and change blindness itself, are more concerned with avoiding saccadic disruption than they are with seeing a stationary world.

This form of remapping, which takes place in a retinotopic framework, can have little bearing on the larger question of how different motor systems are able to access the coordinates of objects in the surroundings, whether or not they are in the field of view, and despite combined movements of eyes, head and body.

3. The relationships of vision, egocentric memory and motor action

The proposal here is that there exists, in the brain, a minimal representation of the outside world that supports memory traces of a small number of key features in the immediate vicinity. Crucially, when someone changes position by rotating or translating, the locations of these memory traces also change; this updating ensures that the objects they represent maintain a constant directional relationship with the external objects themselves.

The motor cortex appears to be organized in such a way that the neurons responsible for movements are tuned to particular directions. Georgopoulos et al. [14] showed that single neurons in the monkey motor cortex are tuned to hand direction, independent of the force required to achieve these movements. The direction of the limb action appears to be a product of the neuron population, with individual neurons contributing to, but not uniquely specifying the direction of the arm's motion, and other factors such as distance of travel may also be involved. Nevertheless, this shows that the motor cortex, which has a fixed location in the head, contains a representation of direction that maps onto actions. I assume that this mapping holds not only for seen objects, which were the subject of the Georgopoulos study, but also for the directions of objects outside the field of view that are specified from memory. I also assume that it holds for locomotory movements driven by the motor cortex, and for eye–head movements mediated by the frontal eye fields. The following example illustrates the key problem, which is to translate the remembered location of objects into a directional command that can be relayed to the motor system.

Consider the situation of a person in a kitchen. They decide to get some tea bags from a cupboard 120° to the left, which they turn to and fixate. Having picked up the tea bags, they then turn to another cupboard, which is 120° left of the first cupboard, to obtain some sugar. Both cupboards will be outside the visual field at the time of turning, although they have both been seen before during a cursory inspection of the room. This is a very ordinary task. In terms of the motor activity involved, the two turns are identical, and, presumably, so too is the directional information supplied to the motor cortex. However, this directional information is generated from the memory representations of different objects in different spatial locations. What kind of neural organization makes this possible?

4. A rotating model: the representation of seen and unseen objects

It will be helpful, at this stage, to contrast the ways that the motor system deals with seen and unseen objects. For objects in the field of view, direction relative to the trunk is specified by the approximate sum of the location of the object on the retina, the direction of the eye axis in the head, the direction of the head relative to the trunk and finally of the limb relative to the trunk. Information about all four variables is readily available from vision, proprioception or efference copy, and the parietal cortex and pre-motor cortex contain well-documented neural machinery for interconversion between different representations (eye-centred, head-centred, trunk-centred etc.) [15,16]. In figure 2, I have used a head-centred (craniotopic) representation of visual space (V); this is essentially a visual, eye-centred (retinotopic) representation with the eye movements removed by addition of eye-in-head position information. This has been chosen largely for convenience, but it does mean that this is a view of the world in which objects are represented in their true positions relative to the head, rather than in the moving retina. In such a scheme, if the head rotates, then the image of objects in craniotopic space moves across the visual representation (V) in the opposite direction (figure 2a,b). This, in turn, means that the origin of the direction information passed to the motor cortex (M) also moves, and so the motor system is continuously provided with new coordinates of objects in outside space on which to base actions. I assume that the final transformation, to trunk-based coordinates, occurs in the motor cortex by addition of proprioceptive information from the neck.

Figure 2.

Representations of a target in relation to a motor action. In (a,b), the target is within the head-centred field of vision (V). This supplies direction information to the appropriate motor structures (M). When the head rotates, the target direction moves across V and M. In (c,d), the target is outside the field of view and is held in egocentric memory (E). If, when the head rotates, the memory trace does not move relative to the head (open circles) the directional information supplied to M will be incorrect. Updating the location of the memory trace in E (filled circles) ensures that the motor system continues to receive correct information. (Adapted from [9].)

Consider now what happens to the memory trace of an object, outside the field of view, that is being held in egocentric memory (E in figure 2c). Representing memory here as a circle is a convenient way of indicating that each memory trace has a locus that is different from the loci of other memory traces; it does not imply that the mapping is actually circular or even map-like (see [17]). As with the visual representation V in figure 2a, the structure that holds the memory (E) supplies directional information to the motor cortex M. Now, suppose that the head rotates clockwise through 60°. Unlike the situation in figure 2b, there is no visual information about the target that can change the location of the memory trace in E. Nevertheless, if the memory trace stayed at the same location in E, then the directional information supplied to the motor system M would be wrong (open circles in figure 2d). To correct this, memory traces in E must rotate by the amount the head has rotated, but in the opposite direction (closed circles). In this way, the directional information that E supplies to M will remain accurate.

If this formulation is correct, then it also means that there exists in the brain a representation of external space that rotates in such a way that the memory traces it contains remain aligned with the corresponding objects in external space. Apart from the advantages this has for motor control, this could also provide the stable basis for the phenomenal world that we see. I will return to this later.

5. Updating egocentric memory

In the example just given, the locations of the memory traces of objects have to rotate within the structure that holds egocentric memory. For translational movements of the body, they must move in one or more other dimensions. In general, these changes that accompany body motion are referred to as ‘updating’. There have been numerous accounts (reviewed in [6–8]) that demonstrate that updating of egocentric memory does indeed occur, and the requirements of an updating system have been modelled in some detail [8].

What inputs are available to implement updating? In the simple example of figure 2, which involves only rotation of the head, the obvious input is from the semicircular canals of the vestibular system, and their involvement is easy to demonstrate from the effects of prolonged rotation on the accuracy of pointing [18]. If you rotate on an office chair for three or four revolutions, then this will corrupt the output of the semicircular canals and induce mild dizziness (figure 3). Take note of the location of a landmark such as a light switch and close your eyes, you will experience yourself rotating for a few seconds in the direction opposite to the original rotation. When this stops, still with the eyes closed, point to the assumed location of the landmark and open your eyes. The direction you point to will be wrong by an angle corresponding to the amount you feel you have rotated when your eyes were shut. What this demonstrates is that the incorrect vestibular signal has resulted in incorrect updating of egocentric memory, and the locations of memory traces of objects held in that representation have moved accordingly, just as they would if updating were normal and accurate. The fact that you also experience the rotation brought about by the inaccurate vestibular signal suggests that the updating signal not only provides information for the motor system, but also plays a role in keeping the phenomenal world stable during ordinary rotation.

Figure 3.

Experiment demonstrating the inaccurate updating of egocentric memory. (a) Subject sits on a revolving chair with feet off the ground, and is rotated through about four revolutions clockwise until mildly giddy. (b) Subject fixates a landmark and closes eyes. (c) The subject feels that he or she is rotating anticlockwise, for 10 or more seconds. Head direction (arrow) and target image (filled circles) have now diverged, in this case by 135°, on the egocentric memory (represented by the grey disc). (d) When asked to point to the landmark, the subject now points 135° to the right of its true position. Diagrams below show the positions of the cupula which senses rotation in one of the horizontal semicircular canals. When forced in the ‘wrong’ direction by the continued flow of fluid in the canal, it produces an inappropriate updating signal. (Adapted from [18].)

For translational movements, the otoliths of the inertial system can play a part, as can dead reckoning based on a subject's own locomotory movements [7]. Visual information also provides updating information, either from simply noting the locations of objects or from locomotion-induced flowfields. Auditory cues and verbal instruction are also effective [19]. The egocentric memory can also be updated from long-term allocentric memory. This other form of spatial memory is maintained in the hippocampus and other regions of the medial temporal cortex. These representations are map-like and non-directional, and have to be converted into an egocentric form that is aligned with current head direction (discussed in [6,8]). This makes possible a restructuring of egocentric memory, so that when one moves between locations at least some of the relevant features of the new environment are already in place.

A scheme not very different from that outlined here was devised by Feldman in 1985 [20], mainly to provide the basis for a computational model of vision. As here, his scheme contained a transient retinotopic representation, which is converted to a ‘stable-feature’ (head-based) frame by removing eye movements. He then proposes an ‘environmental frame’ corresponding to the space around the animal, which is similar to the allocentric memory just mentioned. Where this differs from the present scheme is that objects outside the visual field are held in the environmental frame, and there is no equivalent of the active egocentric memory outlined here. Feldman also proposes a fourth frame (a world knowledge formulary) containing information about objects and relations between them which is conceptual rather than geometrical. Arguably, this is the function of the ‘ventral stream’ in the formulation of Milner & Goodale [21] although this information may be much more generally distributed in the cortex and elsewhere.

6. Properties of the egocentric memory

Egocentric memory is not a high-fidelity representation. Pointing experiments under conditions where the target is not visible and the subject is not deliberately disoriented tend to come up with accuracy estimates between 5° and 25°. Similarly, large saccadic (greater than 90°) turns to objects needed in the execution of a task, which involve rotations of eye, head and body, tend to miss the target by about 10°, and are followed by secondary saccades when vision becomes available (figure 1). The number of object traces that can be held in egocentric memory is small, probably in the 5–10 range. There is some decrease in pointing accuracy with increasing object numbers (see [22]). This is what one would expect, because egocentric memory is necessarily a form of working memory. When egocentric memories are evoked from long-term allocentric memory, there appear to be similar limitations on the numbers of object traces that can be instantiated [19]. Because allocentric and egocentric memories are interconvertible, the question of the capacity of egocentric memory on its own is somewhat problematic.

7. Overlap with vision

Visual information is available, with varying resolution, over most of the forward pointing hemisphere. Does this mean that information from egocentric memory is excluded? That this is not the case is shown by several studies which demonstrate that visual and memory information are both available over much of the peripheral visual field. Brouwer & Knill [23] devised a reaching task in virtual reality in which vision and memory were pitted against each other. Two objects had to be picked up sequentially and placed in a trash can, situated about 30° away. In some trials, the second object was moved a short distance while the other was being moved to the trash. Participants did not note this, but it did affect their behaviour. When the second object was to be moved, in some cases the arm moved in the direction of the current location of the object, but in others it went to the object's previous location. Thus, the arm was sometimes guided by vision and sometimes by memory. It turned out that the relative weighting of vision and memory depended on the object's visibility, with high contrast in the scene favouring vision, and low contrast producing more memory guidance. Thus, the brain could choose between sources of information depending on their reliability. In a related study by Aivar et al. [24], the eye movements of subjects were examined while they were engaged in a virtual task in which blocks were picked up and moved to duplicate a pattern. Changes were made to the layout of the blocks used to make the copy while the subjects were looking elsewhere, but with the blocks still visible in peripheral vision. When gaze returned to the blocks, saccades were normally launched in the directions of their old locations, rather than in the new, visible locations. These two studies show that visual and memory representations overlap in peripheral vision, and probably throughout the visual field, and both can be used to guide movements of either the limbs or the eyes. They also imply that visually attended objects can be temporarily attached to and detached from egocentric memory.

8. Location of the egocentric representation: the precuneus?

Most of the regions of the parietal cortex that have been studied electrophysiologically in connection with reaching, grasping and saccadic fixation have spatial representations based on information derived from the visual field. Usually, this representation is in an eye-centred (retinotopic) frame, although head-centred (craniotopic) representations are also present [15]. None of these is likely to be the site of a representation that includes the unseen half of the panorama outside the fields of view of the eyes, nor of one whose contents move as the head and body rotate and translate in space. It is not surprising that such a representation has not been detected with the electrophysiological procedures used in primate research. The subject has to be able to move, and the ‘stimuli’ are essentially memory traces which move around, rather than defined visual stimuli.

There is, nevertheless, evidence that there is a region of the parietal cortex that maintains such a representation. This is the precuneus, a continuation of the superior parietal lobule on the medial surface of the cortex (figure 4). Based on earlier imaging studies, Byrne et al. [8] assumed in their model that the precuneus was the site containing the machinery of egocentric memory, including updating during self-motion. Since then, a number of papers have appeared reinforcing this assumption, and I will discuss two of them.

Figure 4.

Outline of the left cerebral cortex showing the location of the superior parietal lobule (Brodmann area 7) and its extension, the precuneus, on the medial face of the parietal cortex.

Wolbers et al. [22] used functional MRI (fMRI) to determine which parts of the cortex were involved in updating directional information in a virtual environment that simulated forward self-motion. The task consisted of identifying the locations of one to four different peg-like targets in a textured ground plane. These then disappeared, and for one subject group the plane moved so as to simulate forward motion, whereas it remained stationary for the other group. When the motion stopped, one of the targets was shown for identification, and the subjects had to use a joystick to point to its new (or unchanged) location. Accuracy varied with the number of targets, indicating the involvement of spatial working memory with its capacity limitations. Only two cortical regions responded differently to the moving (i.e. updating) and stationary conditions. These were the precuneus and the dorsal pre-motor cortex. Both regions thus appeared to be involved in the updating of spatial direction. However, when the subjects were asked to give a verbal report of direction the pre-motor area was no longer activated, presumably because it was no longer involved in organizing the pointing behaviour. Thus, the precuneus remained the only region in which working memory was involved in directional updating.

In Wolbers et al.'s paper, the targets were all within the frontal field of view. In a recent paper, Schindler & Bartels [17] addressed the situation where the targets were outside the field of view. They used a combination of fMRI and virtual reality to investigate responses to imagined locations in eight directions in surrounding space. They found that directions were discriminated in two distinct parietal regions: in the intraparietal sulcus (IPS) in the superior parietal cortex, and the temporo-parietal junction region (TPJ) in the inferior parietal cortex. In both regions, there was no overall increase in the fMRI signal, but particular directions were discriminated in distributed patterns of active voxels. In contrast to parts of the parietal cortex coding visual space, there was no indication of a ‘map’ of egocentric space.

In the IPS region, discrimination of unseen directions was strongest in regions IPS 1, 2 and 3 in what is considered to be the ‘reach’ area. The egocentric region of IPS 1 and 2 extends over into the precuneus on the medial wall of the parietal lobe. Schindler and Bartels comment ‘The present evidence suggests that the clusters in IPS 1/2 and precuneus hold the egocentric (most likely body-centred) spatial representation, whose impairment leads to optic ataxia’ [17, p. 179]. This area does not have a visual representation, but is bordered by regions IPS 0 and IPS 4 that do. Schindler and Bartels suggest that ‘Their distributed overlay with visual maps may facilitate flexible remapping between body-centred and visual coordinate systems depending on gaze position’ [17, p. 179]. This is consistent with the overlap between vision and memory discussed in §7. The representation of egocentric space in the inferior parietal region TPJ is probably not associated with action, but it is the region which, when damaged, is typically associated with spatial neglect, a disorder that results in objects in contralateral space being ignored. This could result from faulty interconversion between allo- and egocentric representations.

These two papers strongly implicate the precuneus and superior parts of the intraparietal sulcus as regions that are (i) involved in spatial updating of direction; (ii) concerned with regions of space both within and outside the field of view; (iii) addressable by imagining particular spatial locations. This is important because to point to something behind you requires first that you imagine where it is, i.e. direct attention to a locus that will provide the appropriate direction vector; and (iv) not map-like, in contrast to other vision-based parietal regions. This is significant because a representation in which the memory traces move as the body moves cannot have a fixed layout in the brain. Schindler and Bartels conclude ‘… egocentric space was represented in a distributed neural code… Our findings indicate a fundamentally distinct coding scheme compared to that of visual space’ [17, p. 181].

The idea of a representation that rotates within a fixed structure may seem at first outlandish, but there is a precedent for it in the study by Duhamel et al. [11] in LIP. There, the contents of the fields of ganglion cells change location in anticipation of a saccade, based on non-visual information. It seems that something very similar must be happening here, although in a different context and a different region of the parietal lobe.

9. Egocentric memory and the stability of the visual world

In their review of the functional anatomy of the precuneus, Cavanna & Trimble [25] list a wide variety of functions suggested by scanning and lesion studies. These include visuospatial imagery, episodic memory retrieval, first-person perspective taking and experience of agency. They conclude that ‘activation patterns appear to converge with anatomical and connectivity data in providing preliminary evidence for a functional subdivision within the precuneus into an anterior region, involved in self-centred mental imagery strategies, and a posterior region, subserving successful episodic memory retrieval’ [25, p. 564]. An egocentric representation of one's direction in relation to the surroundings would certainly fit a description of a ‘self-centred mental imagery strategy’.

If it is the case, as advocated here, that the motor system requires a representation of space that maintains a consistent relationship with objects in the outside world as the body moves within it, then this could also serve as a model of a stable outside world of which we can be conscious. As we have seen, this is not a high-definition representation, but it does not have to be. All that is required is that it provides a stable framework to which detailed information, provided by the visual pathways through the occipital and temporal lobes, can be temporarily attached. It is perhaps worth making a distinction between the contents of egocentric memory and the framework itself. Space appears as continuum, independent of the objects that from time to time populate it. It is the continuum rather than the particular contents that appears to remain still when, for example, we look around a room. On this view, our consciously perceived phenomenal world is a hybrid: the machinery of the precuneus provides a temporarily stable and sparsely populated world model, which we can use as an index for finding the sources information we need for action, before examining them in the way that O'Regan [3] suggested. This certainly fits with what we feel is going on: that we look around a stable world, parts of which move seamlessly in and out of vision, and that we retain usable information about the identity and direction of some of the objects behind us, as well as in front.