The ‘when’ pathway of the right parietal lobe

Abstract

The order of events, whether two events are seen as simultaneous or successive, sets the stage for the moment-to-moment interpretation of the visual world. Evidence from patients who have lesions to the parietal lobes and transcranial magnetic stimulation studies in normal subjects suggest that the right inferior parietal lobe underlies this analysis of event timing. Judgment of temporal order, simultaneity and high-level motion are all compromised following right parietal lesions and degraded after transcranial magnetic stimulation over the right parietal but not elsewhere. The results suggest that the right parietal lobe serves as part of a when pathway for both visual fields. We propose that the disruption of this mechanism is the underlying cause of a wide range of seemingly unrelated tasks being impaired in right parietal patients.

Introduction

A central concept in contemporary research is how we identify objects in our visual environment (what) and how we locate those objects (where, or more recently called ‘vision-for-action’) [1,2]. But an equally important ability is how we compute when visual events occur. In the past few years, there has been a growing interest in understanding the psychological and neuronal bases of the temporal dimension in the normally functioning brain and in the neurological population. In addition to behavioral measures and functional imaging studies, several research groups have begun to investigate the mechanisms that register time at the neuronal level [3–5].

The relative timing of events underlies an enormous range of neural functions, from the microsecond delays of auditory processing to the measure of the seasons. Many theories have been proposed to explain how the brain incorporates time into its computations. In particular, when the timing of two events are separated in space and time beyond the range of classic receptive fields and temporal integration periods, other neural mechanisms must be considered. Some suggest a common neuronal mechanism for all timing operations, from visual to speech perception to timing a wide range of motor tasks (the internal clock model [6]). Others suggest that timing is distributed among different neural structures [7,8].

Although new insights have been gained into how the temporal processing is performed over short intervals spanning microseconds (e.g. how the sound is localized by the auditory system) to milliseconds (low-level visual motion), the temporal processing at the intermediate level, across intervals up to one second in duration, is probably the most sophisticated form of temporal processing and it is still little studied and poorly understood [9,10]. There are two broad classes of temporal analysis at these longer scales: one that is metric – the judgment of duration or interval between events – and one that is ordinal – the judgment of order of events in a series, a judgment that for some conditions also supports the perception of motion. It is this temporal ordering of events at the intermediate scale, a core element in many cognitive functions, that we address here. We propose that event order at this scale is computed centrally and we report experimental evidence to indicate that this computation is performed in the right parietal lobe in humans.

Much of the work we review investigates judgments of whether two events are simultaneous or whether two events are seen as independent or integrated into the motion of a single object. Patient and transcranial magnetic stimulation (TMS) studies provide evidence for the lateralization of these functions in the brain. In particular, we find that the right parietal lobe has a dominant role in visual time processing of event order at intermediate scales in both visual fields, suggesting that it forms a core structure of a when pathway. The bilateral nature of the control of temporal attention by the right parietal cortex enables us to distinguish spatial and temporal components of attention. For example, the effects of parietal lesions are too variable to fit into a unitary syndrome: controversies continue over localizations of lesions that lead to neglect as well as over lateralization of deficits [11,12]. Here we propose that parietal control over spatial attention is strongly contralateral, whereas the control of the right parietal cortex over temporal attention is bilateral. The difference between bilateral and contralateral effects is then diagnostic of the contributions of spatial and temporal attention in tasks where both are involved. Bilateral deficits in right unilateral parietal lesions ought not to be spatial in nature and, in almost every case where a spatial deficit has been used to explain a bilateral impairment, it can be seen to be confounded by a temporal component.

The processing of the temporal dimension — the when pathway: a review of current research

Studies with normal subjects and cerebrally lesioned patients have shown that brain regions in the parietal lobe are involved in the analysis of time as well as space, for both visual [12–14] and auditory stimuli [15] (reviewed in Ref. [16]) (Box 1). Studies with non-human primates are consistent with this view [5] and, together with data from human subjects [17], shed more light on the mechanisms that underlie the orienting of attention to sudden change of events in time. In particular, the posterior portion of the inferior parietal lobe (IPL) has a major role in detecting visual events at unexpected locations [18] and studies on patients who have lesions in the right IPL suggest a specific role for this area of the brain in perceptual abilities that require the analysis of time [12,13].

Auditory and visual temporal processing: a hint of common mechanisms.

Although we have concentrated on visual temporal processing, there is now ample evidence that similar processes might be required when we interpret rapid events in the auditory modality. An interesting phenomenon is called auditory stream segregation [47]. Two rapidly alternating tones are heard as a trill when close in frequency but as two separate streams at large frequency differences. A parallel could be drawn between apparent motion and stream segregation. We suggest that the same neural mechanism underlies visual and auditory timing processes because there is evidence from both the neuropsychological [48] and the functional imaging literature [15,49] that the same cortical areas might subserve tasks of event timing in auditory and visual modality. The literature on normal subjects [50] strongly suggests that the inferior parietal lobe (IPL) is involved in multisensory integration processing, such as the detection of synchrony between auditory and visual stimuli. Alternatively, one can conceptualize the right IPL as an operator of temporal order independent of the sensory modality of the stimuli involved [51]. This notion might suggest a metamodal computation imposed by the right IPL on any two consecutive events, regardless of their sensory modalities.

Some elegant psychophysical experiments in vision have also led to the conclusion that different brain mechanisms are specifically dedicated to visual event timing [9,10], and the involvement of a particular brain area might depend on the nature of the timing task. In particular, it has been shown that different neural mechanisms are involved in analyzing duration and temporal frequency using a local time adaptation paradigm [10]. Johnston et al. have demonstrated that temporal duration perception can be altered at the location where subjects had previously adapted to oscillatory motion, implying spatially localized temporal mechanisms [10]. A more recent study has confirmed that such mechanisms are implicated in event timing [19].

Finally, strong evidence of a dedicated neural circuit for timing saccade-related visual stimuli comes from Morrone et al. [20]. This study showed that perception of duration (a delay between two briefly presented lines) is distorted at the time of saccades and the perceived order is often reversed just before saccades. The authors suggested that this distortion of perceived time might be related to activity of the lateral intraparietal (LIP) neurons [5] implicated in encoding brief temporal durations. Overall, neuropsychological, neurophysiological and psychophysical data suggest that the IPL (in the right hemisphere in humans) is the most likely site for the computation of the order of events (Box 1).

Event order and high-level motion

Motion perception is critically dependent on the discrimination of event order in time and space, and many psychophysical studies on motion perception have convincingly demonstrated that there are at least two motion systems [21]. A low-level system computes motion based on the direction selectivity of neurons in the primary visual cortex [22]. These neurons are directionally selective units and they are triggered by subsequent stimuli falling within their receptive field. This is the mechanism by which they signal motion. This system can no longer contribute to motion perception when motion must be perceived between a set of discrete stimuli that are flashed in sequence, separated by long space and time intervals, or when motion is revealed by attentive tracking in the absence of a low-level signal [23]. In this latter case, a high-level or ‘cognitive’ attention-based motion mechanism has been postulated.

Neuropsychological [13], neurophysiological [24] and fMRI [25] studies have demonstrated that the low- and high-level motion systems are anatomically and functionally distinct. However, an important question concerns the underlying mechanism that subserves high-level motion processing. We propose that this mechanism is the core of a when pathway, the disruption of which can cause dramatic, bilateral deficits in motion perception and, more generally, event-order discrimination [14].

A recent fMRI study on humans [25] corroborated our findings [13] and has demonstrated that the right IPL is significantly active while subjects perceive a stimulus moving in apparent motion, compared with the same dots flashing at an identical frequency. Furthermore, another fMRI study has shown significant activity in the right IPL when subjects were asked to covertly orient visual attention towards sudden visual stimuli [26]. A similar result was obtained in a magnetoencephalography study [17] in which subjects were asked to report a transient change in speed in a moving random dot pattern in the left or in the right visual field. A recent EEG study [27] showed a correlation between EEG signal in the right parietal lobe and the illusory perception of a rotating wheel that required discrete motion processing.

Finally, TMS studies have also shown that the right parietal lobe must be specialized in performing tasks that require the discrimination of discrete events in visual and auditory timing [28,29]. Altogether these data show evidence of a special role of the right IPL in visual event timing using both non-spatially lateralized dynamic and stationary stimuli. However, right parietal patients show a severe bilateral deficit when stimuli moving in apparent motion are presented in the two visual fields separately. Traditional neuropsychological studies of visual neglect might call this an object-based (as opposed to spatial-based) attentional deficit [30]. However, our psychophysical results described in the next sections identify the deficit as strictly temporal.

The right parietal lobe and bilateral control of transient attention

Our perception of the objects around us is not merely an accurate registration of their physical attributes. Instead, we shape the world into meaningful groupings. A typical example is when we watch the phenomenon of apparent motion: what is presented on the retina is not what we actually perceive (Figure 1a). Two simple spots (or two pairs of spots in Figure 1a) of light presented at different spatial locations and time intervals will be perceived as one single spot moving from one position to another [31].

Figure 1.

Bilateral impairment in apparent motion. (a) A modified version of the apparent motion task (‘Ternus display’ [55]), using a quartet of dots. Two frames with two dots each are alternated at a variable frequency (left, arrows indicate alternation in time). When the interval between the frames is very short (at high frequency, more than 10 Hz), subjects can perceive only flickering dots and no motion is reported (top right). At appropriate time intervals (around a frequency of alternation of 7 or 8 Hz), subjects report motion (bottom right). The motion can be perceived either horizontally (as depicted in the panel; dashed arrows indicate motion and were not present in the actual display) or vertically. This apparent motion experiment examined whether motion was seen irrespective of its direction. (b) Patients who have right parietal lesions (average data from three patients) show a severe deficit in perceiving apparent motion both in the left visual field (LVF) and in the right visual field (RVF). Their psychophysical threshold is about half that of normal controls (average of six age-matched controls are reported). The patients cannot distinguish between four flashing and two moving dots unless the alternation rate is as low as 4 or 5 Hz, which corresponds to an average of ~200 ms interval between frames [13].

Recent psychophysical [32] and neuropsychological [13,14,33] studies have demonstrated that attentional mechanisms are involved in the perception of apparent motion. Both spatial and temporal properties determine the perception of apparent motion and this enables us to ask whether visual spatial and timing functions are **sub-served by the same high-level neuronal substrates or whether they are distinct functions in the brain. Spatial deficits following parietal damage are typically lateralized but other deficits, specifically temporal ones, have been found to be non-lateralized (reviewed in Ref. [16]). We will argue that these results converge to the idea that the right IPL is specialized in transient attention in both visual fields, whereas the right and left superior portions of the parietal lobes, including the intraparietal sulcus (IPS), are specialized in spatial attention only for the contralateral visual field. This difference enables us to differentiate the contributions of spatial and temporal attention (where and when) to task deficits following disruptions from lesions or TMS.

Discriminating the order of visual events

Temporal attention in vision refers to subjects’ ability to perceive the order and structure of events. We are very good at detecting that a change has occurred, even at very high rates. However, we are not very good at determining what changed or which item changed first, unless the rate of change is greatly reduced. For example, subjects can detect the presence of rapid flicker at rates up to 60 Hz or more. However, if subjects are asked to discriminate between the light and dark phase of the flicker as individual events (a percept referred to as Gestalt flicker), the maximum rate is an order of magnitude lower, 6 to 8 Hz [34].

Similarly, in our apparent motion task (Figure 1a), we ask subjects to report if they see lights moving or flashing, and the highest rate at which this can be done is again 7 or 8 Hz for normal subjects [13]. This limit on apparent motion has previously been reported by Verstraten et al. [32], who claimed that the successive flashes of the stimulus had to be individuated by visual attention for the sequence to be integrated as a motion percept. The low rate at which this could be achieved was attributed to the slow response times of attention in selecting and individuating each successive flash. Our apparent motion and flicker synchrony tasks (Figure 2) reveal the slower processes that are required when two spatially separate streams of changing stimuli must be compared, either to integrate them as motion (Figure 1) or to determine whether they are in synchrony [14]. We argue that these operations are performed within the when pathway.

Figure 2.

Phase discrimination task: bilateral deficit of identification of a visual event. (a) Subjects were asked to identify the odd target among six squares that alternated black and white at the same frequency as the target (middle item in the column left of fixation, top panel; arrows indicate alternation in time). The phase of the target square was shifted in time relative to the other five squares (when the target was black, the distractors were white). The target pops out easily at low alternation rates, but the task becomes more difficult at high alternation rates (for demonstrations of the task and patients’ performance, see http://visionlab.harvard.edu/Members/Lorella/movies/ movies.htm). (b) Normal controls and patients who had left parietal lesions could perform the task up to a speed of ~7 or 8 Hz, whereas patients who had right parietal lesions had a threshold speed of ~3 Hz for both the left visual field (LVF) and the right visual field (RVF). Asynchrony (msec) between the target and the distractors is reported on the right axis. [14].

Patients who are affected by right parietal lesions present a dramatic deficit in all these timing tasks and, most strikingly, this deficit has been seen in both left and right visual fields (Figure 1b).

From our study, we concluded that the deficit involves the accuracy in discriminating offsets and onsets of the dots. If offsets and onsets could not be discriminated at higher rates, it would not be possible to link the offset of one dot with the onset of the other as a single object in motion. Without the percept of motion, it would be difficult to distinguish dots flickering in synchrony from dots alternating, unless the frequency was low enough to enable identification of each dot. Note that the patients perceive smooth motion normally and show deficits only in the contralesional visual field in other attention-related motion tasks [13]. Similarly, traditional neuropsychological studies with neglect patients have demonstrated a variety of deficits confined to the side that is contralateral to the lesion (the right IPL is often part of the cortical lesion) [35], whereas our studies demonstrate that bilateral timing deficits can be present and severe. However, a deficit in high-level motion is in apparent contrast with the fact that patients who have a parietal lesion generally do not report having problems with motion. Our control experiments that primarily tested timing showed that the core deficit is at the level of processing that specifies identity and order of objects in time.

Determining the identity of visual events

To determine if the apparent motion deficit was due to an onset versus offset confusion (thus, an inability to identify appearing and disappearing objects), we tested patients who had right parietal lesions and compared them with patients who had left parietal lesions on a simple judgment of synchrony of flicker in which the timing was very similar to the apparent motion experiment, except that no motion was involved. Six flickering squares alternated (onset– offset) at the same frequency; the target was out of phase relative to the distractors (Figure 2a). This task requires that the light and dark phases of the flickering stimuli are individuated and the highest rate at which this task can be performed is 7 or 8 Hz for normal subjects (a very similar threshold to apparent motion). The parietal patients failed dramatically at performing this task unless the target– distractor asynchrony was as long as ~150 ms, whereas 70 ms was the average threshold for the age-matched controls. It is important to point out that this task requires no motion perception. However, it does require the ability to detect the transients that are generated when each light or dark square reverses contrast to determine their polarity (light to dark or dark to light – see Figure 2 for a description of the task design).

Surprisingly, when the task was set such that onsets and offsets of targets and distractors were not overlapping in time, patients who had right parietal lesions performed like normals. The targets could now be treated as individual transients, independent of their polarity (black or white) [14], and the single transient was easy to detect. This suggests that the patients’ difficulty is not in registering the transients but in identifying whether it is an onset or an offset transient. Note that onset and offset are not defined in terms of stimulus intensity. The onset of a white square can have the same increase in luminance as the offset of a black square. The raw sensory transient at a given location needs to be interpreted in the context of the object found there. A luminance increment represents the appearance of a white object but the disappearance of a black object. We claim that it is the ability to link objects and sensory transients into object events – appearances and disappearances – that is the role of the temporal attention that is lost in patients who have right parietal lesions.

This phase discrimination task involves uniquely temporal attention, and an interesting and striking result came from a single case study of a patient who had medication-resistant epilepsy. Her entire right angular gyrus within the IPL was removed to treat her seizures. We tested her four days after surgery and again at nine, 69 and 103 days post surgery. Her only impairment was a severe and bilateral deficit in the phase discrimination task at four and nine days after surgery, where she performed like the patients who had right parietal lesions reported in Figure 2b [36]. Therefore, her deficit was selective for temporal attention, whereas her spatial attention (measured with other attentional visual tasks) remained intact.

Timing deficits could be less important for conditions where the stimuli are continuously present, like in multiple-object tracking [37] where a combination of spatial and temporal information contributes to event perception.

Interpreting spatiotemporal information of visual events

A helpful hint to the lateralization of the spatiotemporal component of attention came from TMS studies. In one experiment, we used multiple-object tracking, a task that involves both spatial and temporal attention [37]. The results showed that TMS over the right or left IPS induced a severe contralateral impairment in visual tracking [38]. Interestingly, in a successive experiment, we delivered TMS over the right IPL and subjects showed a bilateral impairment at performing the apparent motion task (L. Battelli, P. Cavanagh and A. Pascual-Leone, unpublished).

Finally, the most powerful example of visual stimuli requiring integration of spatiotemporal information is biological motion [39].A handful of dots representing the major joints of the body move to create a configuration that gives a compelling impression of human actions in the absence of shape or identity information. If the timing of the dots is not precisely computed, the action represented is lost.

We have shown that right parietal lobe patients are impaired at performing tasks of biological-motion discrimination using a visual search paradigm [33] and we have also shown impairment in biological motion using TMS as a ‘virtual lesion’ technique on normal subjects [40]. A temporary disruption of the right posterior superior temporal region caused a deficit in biological motion discrimination, whereas TMS over the motion area MT+ did not cause a deficit. This biological motion deficit caused by TMS can be attributed to spatiotemporal attention disruption; however, the stimuli were presented foveally, and a further study using lateralized stimuli should tell us whether the deficit is due to spatial or temporal information disruption.

Interestingly, the left parietal patient we tested in our study [33] had a more severe deficit and could not recognize biological motion even using a single point-light human walker. This indicates that biological motion might be limited by attention to spatial grouping, as in multiple-object tracking, and so it would be likely to show contralateral deficits when appropriately tested. Moreover, the left parietal lobe (and other areas, such as the premotor cortex) might be more specialized in discriminating visual events that are important for action observation [41] and movement control [42].

Concluding remarks

As it has recently been pointed out [43], the latest neuropsychological and imaging studies have challenged the traditional view that the right IPL (Figure 3) is responsible solely for visual spatial processing. Indeed, experimental evidence [44], including from our own studies, suggests that this area of the cortex has a crucial role in tasks that require the control of attention over time. Here we propose a when pathway to accommodate these new findings. This pathway is lateralized in the right hemisphere and is specialized at performing attention-mediated temporal processing at intermediate time scales. This is a core function that enables, for instance, the discrimination of two identical events that occur at the same point in space but at different time intervals. This function might underlie many seemingly heterogeneous deficits in visual neglect patients. The key feature of this when pathway is its control over both visual fields. This enables us to distinguish its role from that of parietal control over spatial attention, where only the contralateral field is controlled by each hemisphere. This has implications for the rehabilitation of neglect patients: the temporal properties of the visual field that is ipsilateral to the lesion, typically thought of as the ‘good’ field, are not normal [45]. Box 2 outlines questions for future research.

Figure 3.

The when pathway. The when pathway is represented in the brain. This pathway is lateralized in the right hemisphere. Information from the primary visual cortex (V1) travels along the dorsal pathway (spatial perception, determining where objects are) or the ventral pathway (object recognition, determining what objects are), according to the classical subdivision that has been proposed based on animal models [1]. A third pathway coming from V1 is dedicated to using time information to identify objects (e.g. determining when objects appeared or disappeared). Here, the temporoparietal junction (TPJ; considered the most common substrate of neglect [16]) is identified as a core anatomical locus, within the inferior parietal lobe (IPL); however, the when pathway is likely to include a bigger network of areas, including the right angular gyrus (Ang), the supramarginal gyrus (Smg) and the posterior superior temporal sulcus (included in the superior temporal gyrus, STG). All these areas are often involved in the cortical lesion of right parietal patients. The intraparietal sulcus (IPS) separates the IPL from the superior parietal lobe (not labeled). The middle temporal area MT+ is reported in yellow (also called the motion area, highly specialized in detecting and discriminating moving stimuli).

Questions for future research.

• The amount of information about the deficit in visual neglect patients could be greatly improved by systematically measuring psychophysical thresholds in detection or discrimination tasks. Such measurements could prove valuable in planning future rehabilitation. Can a timing deficit be improved and does this improvement facilitate rehabilitation of other cognitive functions?

  Recent functional connectivity MRI studies on neglect patients have shown that even areas that are not involved in the lesion might show disrupted functional connections with the damaged areas [52] in the acute phase after a stroke. Connectivity studies in patients, coupled with physchophysical measurements, might help to elucidate the effective contribution of each area to a specific task.

  Right IPL includes structures such as the angular gyrus and the supramarginal gyrus. Moreover, frontal sites such as the frontal eye field and the inferior frontal gyrus are often involved in the lesion. These cortical sites could be studied separately with carefully designed TMS experiments of visual timing. This would enable us to study not only the necessity of an area in a given task but also the chronometry of its involvement.

  TMS studies using tasks of phase discrimination and temporal order judgment should also be informative about the selective involvement of portions of the right IPL in timing tasks.

  Does the left IPL contribute to timing tasks at all? Experimental evidence suggests that the left IPL is involved in motor attention [42] as well as temporal processing for longer intervals [53]. More left parietal patients and TMS studies are needed, in particular, targeting those areas that are homologous to the right IPL.

  Why should visual timing be situated in the right parietal lobe? The left parietal lobe might have similar timing functions but specific for the language domain. For example, recent theories on dyslexia suggest a timing deficit (also called ‘dyschronia’) as an inability to perceive the rapid acoustic elements of human speech [54]: a fascinating theory that might explain other deficits shown by people who have dyslexia that are not limited to reading. People who have dyslexia should also be tested on timing tasks such as apparent motion and phase discrimination, as well as biological motion.

In conclusion, both spatial and temporal factors underlie our perceptual experience [46] and, in many cases, they might be inseparable. However, we propose that we can disentangle the spatial and temporal components of a task by identifying whether deficits (due to parietal lesions or TMS of parietal areas) are bilateral or contralateral.