The ‘when’ parietal pathway explored by lesion studies

Summary

The perception of events in space and time is at the root of our interactions with the environment. The precision with which we perceive visual events in time enables us to act upon objects with great accuracy and the loss of such functions, due to brain lesions can be catastrophic. We outline a visual timing mechanism that deals with the trajectory of an object’s existence across time, a critical function when keeping track of multiple objects that temporally overlap or occur sequentially. Recent evidence suggests these functions are served by an extended network of areas which we call the ‘when’ pathway. Here we show that the when pathway is distinct from and interacts with, the well established ‘where’ and ‘what’ pathways.

Introduction

Time is a central, often overlooked, component of vision, touch and hearing and each of these sensory systems has neuronal substrates specialized for the sequential order of incoming stimuli and for the intrinsic frequency and/or duration components of discrete stimuli. Understanding the temporal component of perception, then, is as central to a description of perception as understanding spatial components, semantic components or primary features such as wavelength, orientation, pitch, pressure or movement. The temporal nature of sensory processing is likely to be underpinned by general rules common to all sensory systems but vision serves as a good model system for several reasons: good deal is already known about spatial vision; visual events are embedded in dimensions of space and time and although much is know about how the visual system registers the three dimensions of space, little is known yet about the encoding of time; vision is our dominant sense serving action; and the systems with which our temporal systems interact are intricately described in the visual domain. Registration of time is clearly essential for determining the sequence of events, judging their duration and identity, sensing the interval between events, and coordinating actions. To understand sensory and motor processing in the brain it is necessary to understand how time is processed over short timescales, often in milliseconds, in which most visual events occur [1,2]. Classically, this has been addressed by studying how the brain analyzes time intervals between events but recently several new approaches and ideas about the processing of time have been proposed [3–5]. Another way of understanding temporal brain systems is by analyzing changes in temporal perception and behavior as a function of cortical damage that produces visual deficits or as a function of temporary disruptions of temporal integration and perception by altering activity in different neural areas by application of transcranial magnetic stimulation (TMS).

In this review we address the question of time by focusing on the loss of visuo-temporal abilities. Evidence from TMS studies has helped to determine the direct involvement of selective brain areas in timing functions and experimental evidence suggests that the visual system might use algorithms that are more generalized across the brain to perform discrimination of transient stimuli and might generalize to other sensory modalities such as in audition for speech discrimination [6–8] and also to multisenosry events [9]. We define the ‘when’ visual pathway in the brain and characterize it as a module formed from several anatomically and functionally distinct areas analyzing visual timing at time scales longer than that resolved by hard-wired analysis in lower-level sensory areas such V3, V4 and V5/MT (e.g. the 50 msec upper limit for low-level motion detectors in vision) [10–13] and shorter than the cognitive judgments of elapsed time (e.g. how long have I been sitting here?). This mid range (50 msec to one second) underlies the choreography of immediate, ongoing events – the sequence of object appearances and disappearances, object displacements and transformations. The role of time at this scale is not so much to underpin the experience of time but to establish the ordering and nature of the flow of events.

Psychophysical and neurobiological evidence

A pair of transients marks the arrival and the disappearance of an object in the visual field and this pair can be considered the opening and closing punctuation defining the object’s existence, opening and closing the “object file”, to use the concept proposed by Kahneman et al. [14]. The interval between these two temporal markers establishes the duration of the object’s presence and if, for any reason one or the other markers is absent, the “object file” may not be opened (we don’t see the object) or may not be closed. Or, if the closing transient of an object at one location is followed closely by another transient, the two locations may be linked together as one object in motion, keeping the temporary token for the object open but updating its location. The analysis of the timings of these transients, when and where they are, and to which objects they belong is, then, the essential role of the ‘when’ pathway. We review manipulations that degrade or mask these transients and so affect judgments of order, duration and motion.

Transient visual events grab our attention and this exogenous orienting has been anatomically and functionally distinguished from an endogenous orienting mechanism [15,16]. It is also known that a transient change of status (e.g. the size) of the same event repeatedly presented across time on the same retinal location can be perceived as lasting longer, despite the fact that the presentation time is the same as the other events in the stream [17]. Therefore there is evidence that a sudden onset is enough to capture attention and must play a special role in object discrimination. Neurophysiological studies have shown that activity in the lateral intraparietal cortex (LIP) of non-human primates can precede activity in earlier visual areas when sudden transient events are attended and correctly perceived [18].

Recent data from elegant psychophysical studies have addressed the issue of how the visual system uses visual transients to individuate a salient change as a new object during tasks requiring temporal segmentation [19], and it has also been shown that visual events are timed by neural mechanisms that are spatially selective in spatiotopic coordinates [5,20]. This may seem to point to higher-level neural mechanisms responsible for timing visual events at the millisecond level and potential correlates of this in the form of temporal response preferences in spatially defined neurons have been observed in neurophysiological studies of LIP [21]. However, when the salience of visual transients are manipulated the visual system can fail to perceive as efficiently [22,23]. Finally, recent psychophysical studies have shown that exogenous attention is finely tuned to the perception of transients which can affect the timing of independent visual features such as color and motion [24].

In the visual system while events repeating in time can be detected as separate events even when presented in rapid succession, it is much harder to assign identity to sequential objects in the same events. For instance, when two target letters are presented on the same spatial location displaced in time using a rapid serial visual presentation paradigm (RSVP), the second letter is not recognized unless separated by at least 400 ms from the first target letter. This failure to report the second letter is called the attentional blink (AB) phenomenon. However, if subjects are asked to detect the presence of one letter within a stream of letters they can easily perform the task at a much higher rate [25,26]. Recent psychophysical and neurophysiological findings have shown that the limit of the AB is likely to be set at a high level in the visual system such as the parietal and frontal cortical areas [27–29]. Similarly, in motion perception, when discrete snapshots of a visual stimulus must be combined across long spatial and temporal intervals to determine for instance the direction of a dot moving in apparent motion, the visual system must integrate the stimulus presentations across temporal frequency (the rate of alternation of two discrete stimuli) and spatial displacement [30]. When the frequency of alternation is too high, the visual system can no longer perform the task and the perception of motion is lost. Thus in the case of one dot moving in apparent motion, two blinking dots will be perceived (or four blinking dots as in figure 2a [2]). Experimental evidence has shown that there must be an optimal frequency at which the visual system can solve the problem of temporal segregation and integration by sampling across discrete snapshots, however above such frequency the visual system might fail, leading to an incorrect perception of the stimulus [31,32]. Interestingly, in an EEG experiment subjects were presented with a discrete stimulus generating an illusory percept of reversed motion (also called the wagon wheel illusion, WWI [31]). The power spectrum of the EEG components obtained during real (the actual direction of rotation of a wheel as in figure 1a) and reversed illusory motion were compared and, only the component centered around 13 Hz (figure 1b) differed significantly between the two conditions and was lateralized over right parietal regions.

Figure 2. TMS disrupts apparent motion perception bilaterally.

a) The rate of alternation of the two frames on the left side of the panel was varied across trials and observers perceived either motion (bottom right) or flickering (top right, see [2] for a detailed description of the task). A comparison stimulus with four flickering dots was used and subjects were asked to report whether they saw motion or flickering. b) After 10 minutes of 1 Hz TMS delivered over the right IPL subjects’ reaction times were significantly slower on correctly reported motion trials both in the left (LVF) and right (RVF) visual field (average of 7 subjects). While TMS had no effect on correctly reported flickering trials (data not reported, Battelli L., Cavanagh P., Walsh V. and Pascual-Leone A., unpublished observation).

Figure 1. EEG correlates of the wagon wheel illusion (WWI).

a) Observers were asked to report the perceived direction of a continuous rotating wheel moving at different rates. b) 32-channel EEG recordings were made and the power spectra between the illusory and real motion were compared. A significant difference was found only at 13 Hz over right parietal electrode, independently of the rate of stimulus rotation. The difference must depend on internal processing mechanisms as the stimulus on the retina is the same during both real and illusory motion. (Reproduced with permission from [31]).

We used the same stimulus in a TMS experiment and we showed that the illusion was significantly reduced after TMS over the right IPL, while TMS had no effect when delivered over the left IPL [33]. This effect was bilateral, similar to the timing deficit we found in patients with a right parietal lesion [34]. Neuropsychological evidence reported in the next section has confirmed the notion of a ‘when’ pathway in the right parietal lobe. The cortical site of the lesion, as well as its behavioral manifestation, clearly distinguishes it from other visual and spatial cortical functions.

Deficits in transient visual perception

Critchley in his seminal book on the parietal lobes wrote: “…most interesting and complicated of all are those spatial disorders which also involve the conception of time …... one must distinguish between a primitive time sense, and a gnosic time-conception (by which is meant an understanding of chronological order)….” [35]. Although the concept of a deficit in event discrimination across time was quite clearly stated in his book, it has rarely been addressed as a deficit of object discrimination, since much of the subsequent work on time has concentrated on the normality and pathology of interval timing [3,36]. However, data from neuropsychological patients have clearly shown that the right parietal lobe, and in particular the IPL might play an important role in discriminating events that are displaced in time [2,37]. Patients with IPL lesions demonstrate deficits in visual event discrimination in both visual fields and that often coexist with other visuo-spatial deficits only in the field contralateral to the lesion [2]. Therefore the bilateral origin of the deficit might distinguish purely visual timing deficits from spatial ones. Other studies have found deficits similar in nature to the visual timing of events [38] and bilateral timing deficits have been reported for visual search tasks requiring spatiotemporal segmentation [39]. In future studies it would be interesting to study neurological patients with other high level timing tasks, as has been done in recent TMS studies [40].

TMS studies

In one study we provided evidence of a direct involvement of the right IPL in event discrimination across time by using TMS. We used the same task in which right parietal patients showed impaired performance in both visual fields [2] and we replicated the bilateral impairment (figure 2).

In a subsequent study we further assessed the direct involvement of right IPL in visual timing of discrete events. TMS significantly reduced the perception of a modified version of the WWI illusion (figure 1a) in both left and right visual field immediately after stimulation over right IPL [33]. Our result was similar to the performance of right parietal patients when presented with binocular rivalrous stimuli [41].

So far we have concentrated on the role of the ‘when’ pathway in individuating or integrating events in time and much of the processing required in such tasks requires the properties of neurons involved in motion perception and characteristic of the pathways originating in the magnocellular laminae of the LGN – transient responses, fast conduction time, large receptive fields. We can therefore expect there to be overlap between the neural systems that compute ‘when’ and those that compute ‘how long’. The latter has been studied in the context of temporal mechanisms in the motor system (in particular the basal ganglia, cerebellum and pre motor and pre supplementary motor areas [42,43]).

In this section we will concentrate on studies that have investigated sensory and association areas and explain the “extended when pathway”. The ‘when’ pathway as conceived so far in this article includes the posterior parietal cortex but as we anticipate when an event will occur, we estimate the duration of a period from one event in time to a predicted time in the future. This requires a computation of ‘when-past’, ‘how long’ and ‘when-future’. Several cortical areas are candidates for inclusion in this extended ‘when’ pathway. In addition to the posterior parietal cortex, we can expect sensory areas such as V5/MT (and indeed V4 – [44]) and also areas involved in predictive behaviors such as the prefrontal cortex, to be involved in event timing.

TMS studies on interval timing reveal results consistent with the existence of a ‘when’ pathway. For example, Alexander et al. [45] stimulated right and left parietal cortex in interval discriminations studies and observed an effect of online TMS only over the right posterior parietal cortex (PPC) during auditory discrimination tasks. This suggests that the parietal cortex may have a multimodal role in timing functions, although this role is not unique since other areas are also important for auditory timing discriminations [46]. Bueti et al., [47] applied TMS over V5/MT during temporal estimation tasks with visual and auditory stimuli and found disruptive effects of TMS only for visual stimuli. However, they found these effects for both moving and stationary stimuli suggesting that the transient response properties of V5/MT are useful for “time stamping” the onset and offset of visual events. The sustained properties of parvo-derived visual areas would not be suited to this purpose. A prediction that follows from this is that patients with when pathway damage including V5/MT will have deficits in temporal duration estimation of visual stimuli such as shapes or colors which they are otherwise able to perceive normally. This remains to be tested.

This ability to time stamp is useful for marking the beginning or actual end of a visual event but what of the present and future? TMS studies have examined the prime candidates for these functions and as we have seen the PPC [45,47,48] and V5/MT [47] clearly play a role. One area likely to play a role in temporal prediction is the prefrontal cortex and so far two TMS studies have examined this area in temporal functions. Koch and colleagues applied offline 1Hz TMS over the prefrontal cortex (PFC) and observed that subjects underestimated intervals in the range of a few seconds [49]. Another study [40] using theta burst TMS over PFC reduced the foreperiod effect. The foreperiod effect refers to the reduction in simple RTs when the foreperiod between presentation of a stimulus and imperative signal for the time of response increases (the longer you wait the faster you are). This is thought to index a preparatory timing mechanism of the kind required to connect future estimations with past event time stamps.

The point of knowing or predicting when something will occur is to allow us to interact with the environment and there are now many cases of overlap between systems for temporal and spatial processes. Following the proposal that temporal and spatial mechanisms adopt the same metric and share resources with mechanisms underlying numerical processing [4], Oliveri et al. [50] have shown that perceiving numerical information interferes with time perception. In the spatial domain, Mevorach and colleagues [51] have shown that TMS to parietal regions overlapping those stimulated in experiments that produce temporal processing deficits cause changes in spatial bias. Similar evidence of spatiotemporal convergence has been seen in neuropsychological patients who display a temporal neglect akin to the spatial neglect seen following parietal cortex damage [52–54]. Indeed, within the context of the frontoparietal cortex it has been suggested that temporal deficits may be the cause of apparent spatial deficits [55].

The extended pathway also includes other sensory modalities. So far we have intentionally limited our discussion vision but of course we perceive events and objects with our senses of hearing and touch. There are three possibilities for the interaction of the senses in the extended pathway: they may act in perfect temporal concert; they may proceed independently; the responses in one sensory domain may bias the responses in another. We can exclude the first possibility - that the senses act in perfect temporal concert – because visual, auditory and tactile neurons have different temporal response profiles and, as we shall see, these differences have consequences. The second possibility does occur. There are indeed cases in which the sensory modalities do compute time independently, for example in experiments when only one set of cues (auditory, visual, tactile) is available, or in tasks dominated by one sense (e.g. vision in catching ball). But in many everyday situations the brain is faced with two or more sources of temporal information and, the interesting question, then, is how do the senses interact in time? The answer is task dependent and several examples already exist in the literature. Temporal convergence is one such example. Fendrich and Corballis [56] showed that when auditory and visual stimuli were shown in close temporal sequence, the perception of one stimulus in time was “captured” by the preceding or succeeding stimulus. The timing system interpreted two events as one. Vroomen et al, [57] have shown similar effects of the timing system in the presence of two modalities. Dramatic effects of interpretative or, in Vroomen’s terminology recalibration of temporal mechanisms in the presence of multisensory asynchronies are also seen in the many demonstrations of the ventriloquist effect [58]. Such rapid integration and interpretation mechanisms are difficult to probe in patients but evidence from neglect patients shows that integration of different modalities is spatially dependent [59,60] and relies on an intact frontoparietal circuitry.

Conclusions and the future of ‘when’

The relationship between perceiving and acting in time is central to descriptions of behavior and the when pathway therefore has to take account of it. What we have shown so far is that the when pathway in vision is largely restricted to regions also involved in spatial processing and visuomotor transformations. These areas are also part of sensorimotor networks and it is therefore reasonable to expect such anatomical proximity.

We have described evidence supporting a ‘when’ pathway and have done so mainly in the context of vision. However, temporal processing and estimations will also be made in the auditory, somatosensory and motor systems – and often in conjunction. We propose that each sensory system will have its own ‘when’ pathway originating in sensory cortex and taking a route through the parietal and motor related cortices because the temporal information is most important for programming responses to the world. The need for separate initial analyses of time is consistent with evidence that the senses run on different clocks [61]. The need for a later common pathway is consistent with the fact that sensory motor associations can of course be made between any modality and action and action can be based on taking the initiative in time and space [62].

Many questions remain about the when pathway: areas such as the frontal eye field remain to be explored and the range of functions dependent on modality-specific timing also requires further exploration. We need to clarify the relationships between different temporal spaces and actions – is the analysis on short versus long time scales completely independent or intertwined? In pursuing these questions we would emphasize that the output of the ‘when’ pathway is the sequencing of perceptual events that anchors the coordination of action. Further experiments should keep this functional goal in mind when addressing the possible mechanisms of time analysis.