Integration of goal- and stimulus-related visual signals revealed by damage to human parietal cortex

Abstract

Where we look is determined both by our current intentions and by the tendency of visually-salient items to ‘catch our eye’. Following damage to parietal cortex, the normal process of directing attention is often profoundly impaired. Here, we tracked parietal patients’ eye movements during visual search to separately map impairments in goal-directed orienting to targets, versus stimulus-driven gaze shifts to salient but task-irrelevant probes. Deficits in these two distinct types of attentional selection are shown to be identical in both magnitude and spatial distribution, consistent with damage to a ‘priority map’ that integrates goal- and stimulus-related signals to select visual targets. When goal-relevant and visually-salient items compete for attention, the outcome depends on a biased competition in which the priority of contralesional targets is undervalued. On the basis of these findings, we further demonstrate that parietal patients’ spatial bias (neglect) in goal-directed visual exploration can be corrected and even reversed by systematically manipulating the spatial distribution of stimulus salience in the visual array.

To extract useful information from the visual world we must direct our attention serially to different regions of space. In general, this involves making overt shifts of gaze to fixate objects of interest within the visual field. Attentional selection is thought to be determined by both an object’s relevance to our current goals and its inherent visual salience (Posner, 1980; Bundesen, 1990; Wolfe, 1994; Desimone and Duncan, 1995; Kastner and Ungerleider, 2000). Experimentally, these two components can be distinguished in visual search. Goal-directed factors are demonstrated by our ability to detect and orient to a target object on the basis of a pre-specified feature (such as color or location) distinguishing it from distractors (Treisman and Gelade, 1980; Egeth et al., 1984; Wolfe, 1998). Stimulus-driven orienting is observed in the tendency for distractors with high visual salience to automatically capture attention even though task-irrelevant (Jonides and Yantis, 1988; Theeuwes et al., 1998; Lamy and Zoaris, 2009).

Neuroimaging and neurophysiological studies have uncovered a network of frontal and parietal cortical areas associated with control of attention and eye movements (Andersen, 1995; Schall and Thompson, 1999; Colby and Goldberg, 1999; Kanwisher and Wojciulik, 2000; Corbetta and Shulman, 2002; Serences and Yantis, 2006; Monosov et al., 2008). However, the extent to which different areas within this attentional network contribute to goal- or stimulus-driven orienting remains controversial (Kincade et al., 2005; Peers et al., 2005; Natale et al., 2008). Differences in timing of goal- and stimulus-related activity between frontal and parietal cortex have recently been reported in macaque (Buschman and Miller, 2007), suggesting that goal-directed signals perhaps originate in frontal regions, whereas stimulus-driven orienting is the responsibility of parietal cortex. However, since activity in both regions ultimately correlates with attention shifts, interpretation of this temporal dissociation remains open.

To determine whether particular brain regions have a causal role in deciding the final target of attention, we must examine situations in which normal brain activity is disrupted. In humans, damage to the right parietal cortex commonly results in visual neglect, a syndrome characterized by failure to orient to the left of space (Vallar and Perani, 1986; Mort et al., 2003). Here we record eye movements of parietal patients during visual search. We show that their neglect deficits comprise equal impairments in goal-directed and stimulus-driven orienting of gaze, and that the spatial distributions of the two types of impairment are indistinguishable.

These results imply that posterior parietal cortex is critical for both goal-directed and stimulus-driven control of attention. Thus neglect may be a consequence of damage to a ‘priority map’ within posterior parietal cortex, where goal- and stimulus-related signals are combined to produce a unified representation of attentional priority (Bisley and Goldberg, 2003; Balan and Gottlieb, 2006). Consistent with this hypothesis, we further show that when goal-relevant and visually-salient items compete for attention, the outcome depends on a biased competition in which the priority of contralesional items is under-represented. Finally, we show that the lateralized exploration bias shown by neglect patients can be corrected and even reversed by systematically manipulating the distribution of bottom-up salience within the visual scene, providing further evidence for a distortion of attentional priority, and a new mechanism for improving visual inattention.

Materials and Methods

Subjects and apparatus

In total, 9 patients and 18 control subjects participated in the study after giving informed consent. All procedures were approved by the National Hospital for Neurology and Neurosurgery Ethics Committee. The patients had all received a diagnosis of left visual neglect following a stroke involving the right parietal lobe (with the exception of patient JK, whose data is examined separately in Experiment 2 as a case of frontal neglect without parietal involvement). All demonstrated neglect clinically in everyday life. In addition they showed a rightward bias on at least one of two standard pen-and-paper tests of neglect conducted on the day of testing: line bisection (Schenkenberg et al., 1980) and bells cancellation (Gauthier et al., 1989). See Supplementary Table 1 for further clinical details. Control subjects were neurologically-healthy individuals, with normal or corrected-to-normal vision, matched in age (median 70 years) to the patient group.

Stimuli were displayed on a 24” widescreen TFT monitor, viewed at a distance of 60 cm. Eye position was monitored online at 1000 Hz using a frame-mounted infra-red eye tracker (SR Research, Ontario, Canada). Subjects were positioned within the apparatus such that head and body mid-lines were aligned with the horizontal centre of the display; this alignment was monitored visually throughout the experiment and corrected as necessary – no systematic shifts were observed.

Experiment 1

We presented subjects (5 neglect, 10 control) with a series of search arrays (Fig 1a). Each array contained 40 green squares (2.5° × 2.5°; 5.3 cd m⁻²) uniformly arranged in a grid pattern on a black background (0.29 cd m⁻²), covering an area of the visual field 24.5° high and 41° wide. Each trial began with a blank screen. Once a stable fixation was detected within the display area, a search array was presented for 1250 ms. The display was then blanked for 250 ms before beginning the next trial. Subjects were instructed to search the array for red letters and respond to the letter X by pressing a response button as quickly as possible. The green squares acted as distractors. Note that these were different from probes (see below) which we used to index stimulus-driven orienting.

Figure 1. Visual search for target letters with salient non-targets.

(a) Structure of the task: subjects were presented with a sequence of randomly-interleaved target and probe arrays.

(b) A typical target display, consisting of distractors (green squares) and a single target (red letter). Arrows indicate one healthy subject’s sequence of fixations on the array: here the red target A (isoluminant with the green squares) is located on the second eye movement.

(c) A typical probe display, consisting of distractors and a single high-salience orientation probe (in this case a rotated square; note that probes could differ from distractors either in luminance, onset or, as here, orientation). Arrows indicate one healthy subject’s scanpath: despite its irrelevance to the search task, an immediate eye movement is provoked by the visual salience of the probe.

Each subject completed 720 trials, comprising 360 target trials and 360 probe trials presented in a random order. On target trials, a single red letter (2° height) was present at a random location in the search array, a minimum of 5.5° (one grid separation) from the subject’s initial fixation position (example shown in Fig 1b). The letter was either an X (50% of trials) or a different randomly-selected letter (50% of trials). A button press in response to the an X was rewarded with a pleasant tone, while a press following any other letter was punished with an unpleasant beep.

The red letter targets were perceptually isoluminant with the green distractors (mean luminance: 6.6 cd m⁻²). Perceptual isoluminance was determined separately for each subject prior to the experiment (subjects adjusted the luminance of red squares in a red-green checkerboard to match perceived brightness; the average of five repetitions determined target luminance for the subsequent experiment).

On probe trials, the visual salience of a single, randomly-chosen array item (minimum 5.5° from fixation) was increased in one of three ways: by altering its luminance, orientation or onset (orientation example shown in Fig 1c). Probes were completely irrelevant to the task and were designed to capture attention and gaze in a stimulus-driven fashion. The salience of probes was parameterized into three levels (low, medium, high) for each type of probe. Luminance probes were brighter than the green distractors (by 33%, 300% or 1500%); orientation probes were rotated relative to distractors (by 2°, 15° or 45° clockwise); onset probes appeared before the rest of the array (which was delayed by 33, 166, or 500 ms). Probes of each type and level were presented with equal frequency in a randomized order.

As our measure of performance on the search task, we calculated for each subject the proportion of target trials on which the target was successfully fixated (criteria: eye velocity < 50° s⁻¹, duration > 100 ms, distance < 2.75° from centre of target). As our measure of reflexive orienting, the proportion of probe trials on which the probe was fixated was calculated according to the same criteria, for each combination of salience type and level.

To compare a neglect patient’s deficit for targets with his or her deficit for probes, we calculated for each type of item a ratio (R) of the proportion of items fixated by the patient (p) and the mean proportion fixated by controls (c):

$$R=\frac{p-c}{p+c}$$

Whereas a simple ratio can take any value from zero to infinity, this normalized ratio is confined to the range −1 ≤ R ≤ 1. R = 0 indicates that a patient fixated the same proportion of items as controls (i.e. showed no impairment). Negative R values reflect the magnitude of deficit in patient performance compared to controls, with R = −1 indicating that the patient fixated none of the items (i.e. complete impairment). Positive values of R occur in cases where the patient fixated a greater proportion of items than controls.

To obtain a single value reflecting a patient’s horizontal bias in orienting to a particular type of item, we used the formula:

$$\mathit{bias}=\frac{\sum R_i^{\prime }{x}_i}{\sum R_i^{\prime }},$$

where x_i indicates the horizontal position of the ith column of distractors, and $R_i^{\prime }$ is the deficit ratio (R) for items appearing within that column, calculated as above and normalized to the range 0 ≤ R’ ≤ 1, i.e. R’ = ½ (R + 1).

Experiment 2

As in the first experiment, subjects (now 1 parietal neglect, 1 frontal neglect, and 8 controls) were presented with a series of search arrays and instructed to respond with a button press to the appearance of a letter X. Unlike in Experiment 1, each subject was randomly assigned a target color (red or blue, counterbalanced within each group) and instructed to respond only to a letter X of that color, and to ignore letters of any other color. Each search array consisted of 20 isoluminant letters (height 1.5°, 7.5 cd m⁻²) uniformly arranged in a grid pattern on a black background, creating a total search area 24° high and 30° wide. The majority of letters in each array were green distractors (example shown in Fig 5).

Figure 5. Visual search with simultaneously presented targets and probes.

On each trial, subjects were presented with a search array consisting of a grid of green distractor letters and either a single letter of the target color (target-only trial), a single letter of the non-target color (probe-only trial), or one letter of each color, as shown here (target+probe trial). In the example shown, the subject has been instructed to search for blue letters and ignore other colors, making the red letter the probe; 50% of subjects received opposite instructions. Subjects’ initial gaze locations at array onset were controlled by presentation of a dynamic fixation cross, which initially filled the display and then rapidly zoomed-in to a randomly-selected grid location.

Three different types of trial were tested in equal numbers and in a random order: target-only trials, probe-only trials, and target+probe trials (where both target and probe were presented simultaneously). On target-only trials, a single letter of the target color was present at a random location in the array; on probe-only trials, a single letter of the non-target color was shown; and on target+probe trials, one letter of the target color and one letter of the non-target color were presented. Each array was displayed for 2000 ms, followed by a 1000 ms blank period before the start of the next trial. Control subjects each completed 600 trials in total, as did the parietal neglect patient (DC); the frontal neglect patient (JK) completed 360 trials.

Subjects’ initial gaze position was controlled by presenting a fixation cross at the start of each trial. The cross was presented at a random array location on each trial, sampling equally from all horizontal positions during the course of the experimental session. As static fixation crosses often fail to attract gaze in neglect, each cross was initially presented at a size of 55° across and rapidly ‘zoomed in’ to its final position, appearing at a final size of 1.8° across. The array appeared as soon as a fixation was recorded within 3° of the final position of the cross.

In this experiment, unlike the previous one, we took as our performance measure the direction of the first saccade recorded following array onset (criterion: eye velocity > 50° s⁻¹). This eye movement was considered to be directed towards a target (or probe) if its mean direction fell within 30° of the direction from the fixation cross to the centre of the target (or probe) letter. Targets and probes were presented a minimum distance of 7.5° from the fixation cross, and on target+probe trials they were separated from each other by a minimum angle of 90° relative to initial gaze position.

Experiment 3

Two neglect patients, who had not participated in the previous experiments, were tested on a modified version of the search task used in Experiment 1. 50% of trials were target trials, comprising an array of green square distractors and a single red letter target, as described previously. Unlike in Experiment 1, the remaining 50% were now null trials, in which the distractor array (green squares) was presented without targets or probes. As a measure of visual exploration, we calculated the mean horizontal position of the fixations recorded on each of these null trials. Each patient completed 640 trials in total. For the first 160 trials, distractors were of uniform luminance (31 cd m⁻²). Subsequently, an adaptive algorithm varied distractor luminance horizontally across the screen trial-by-trial to find the luminance distribution that minimized the mean deviation of fixations from the centre of the display.

The luminance distribution on each trial was set by a parameter γ, which determined the relationship between the horizontal position of each distractor (x, taking values in the range 0, for the leftmost distractor, to 1, for the rightmost) and its RGB brightness (b, range 0 to 1, proportional to the square-root of luminance). When γ = 0, distractors were of uniform brightness. When γ was positive, the brightness of distractors increased smoothly from left to right: linearly for γ ≤ 1, b = ½ + (x − ½) γ, then exponentially for higher values, b = x ^γ. Negative values of γ corresponded to horizontal reflections of these distributions, such that brightness increased from right to left (see Supplementary Figure 5 for examples). On target trials, the brightness of the red target letters was perceptually matched to the brightness of distractors at the same horizontal position.

The relationship between γ and the mean horizontal fixation position on null trials was approximated by a cumulative gaussian function. Maximum likelihood estimation of the parameters of this function yielded an estimate of γ₀, the value of γ that would minimize the mean fixation deviation from the centre of the display. After the initial 20 null trials (where the value of γ was chosen at random in the range −5 to 5), the value of γ tested on each subsequent trial was chosen randomly from a uniform distribution bounded by the 95% confidence limits on the current estimate of γ₀.

Following the estimation session, one patient (KB) completed an extra experimental test, comprising a further 160 trials, in which distractor luminance was determined by the final estimate of γ₀, but target luminance was uniform across the display (as for γ = 0).

Results

Goal-directed visual orienting

To distinguish goal-directed from stimulus-driven control of visual orienting we recorded eye movements while participants performed a novel visual search task (Experiment 1). Subjects were shown a sequence of distractor arrays (green squares) in which were embedded isoluminant task-relevant targets (red letters; Fig 1a & b). A target was present on 50% of trials (maximum one target per display). Participants were instructed to search the arrays for the red letters, and respond with a button press as quickly as possible only to the letter X. Targets were sufficiently small that foveation was required to make this letter discrimination: the frequency with which subjects successfully fixated the targets was used as a measure of success for the goal-directed control of attention.

The large search area and low stimulus salience of the small, equiluminant target letters required active search, with 39% of targets requiring two or more fixations to locate in healthy controls. Nonetheless, control subjects ultimately fixated 97% of the red letter targets within the presentation time of the array (Fig 2a, blue bar) and responded correctly to the identity of all fixated letters. In comparison, patients with left neglect following lesions of the right parietal lobe (lesion overlap shown in Fig 2d) demonstrated severe performance deficits on the search task, indicating an impairment in goal-directed orienting to the letter targets. They fixated a significantly smaller proportion of targets than controls (43%; Fig 2a, red bar; t₍₁₃₎ =14.0, p < 0.001), and made a correct button response on only 32% of trials.

Figure 2. Goal-directed orienting to targets.

(a) Proportion of targets fixated by healthy controls (blue) and neglect patients (red). Error bars indicate s.e.m.

(b) Proportion of targets fixated as a function of the horizontal position of the target, showing a gradient of impairment in patients increasing from right to left.

(c) Heat maps show the proportion of targets fixated by controls (top) and patients (bottom) as a function of 2d target position (bicubic interpolation between target locations). Note the greater impairment in patients for inferior as well as leftward targets.

(d) Overlap map shows the location of lesions in the neglect group. The only area affected in all patients (shown in red) involves the posterior parietal cortex and temporoparietal junction.

The effect of spatial location on the probability of fixating a target is shown in Figs 2b & c. The poor performance of neglect patients on the task was due to a failure to fixate leftward targets, as might be anticipated from previous search studies (Johnston and Diller, 1986; Behrmann et al., 1997). This impairment consisted of a gradient of increasing deficit from right to left, with near-normal performance at the far right of the tested range, and an almost complete failure to fixate targets on the extreme left (Fig 2b, red line). In the vertical dimension, neglect patients also showed a small but significant deficit in orienting to targets in the lower half of the display (Rapcsak et al., 1988; Pitzalis et al., 1997) (Fig 2c, bottom; t₍₄₎ = 6.5, p = 0.003). These findings reveal the spatial distribution of each patient’s deficit in goal-directed orienting, i.e. in locating the red target embedded among isoluminant green distractor squares.

In order to assess deficits in orienting attention independent of deficits in object identification, we have taken as our principal performance measure the frequency with which targets were successfully fixated, rather than the frequency of correct button presses. However, parietal patients also demonstrated impairments in the manual responses that mirrored the spatial pattern of deficits in goal-directed orienting (see Supplementary Figure 1). Response errors were more common for contralesional targets (t₍₄₎ = 3.5, p = 0.02) and, to a lesser extent, those in inferior space (t₍₄₎ = 3.7, p = 0.02), even when the target was successfully fixated.

Stimulus-driven visual orienting

To obtain a measure of stimulus-driven control of attention by task-irrelevant stimuli, on the remaining 50% of trials – which contained no red letter – a single salient ‘probe’ stimulus was presented at a random location in the distractor array (Fig 1a & c). Probes resembled the distractor items (green squares) but with a single adjustment that increased their visual salience: either an increase in luminance, a change in orientation, or an earlier onset. Probes of each type could be presented at one of three salience levels (low, medium, high; see Methods) reflecting the amplitude of the difference between probe and distractor.

Despite their irrelevance to the search task, and despite sharing no visual features with the search targets (red letters), these salient probes attracted even control subjects’ gaze on a substantial proportion of trials (Fig 3a, blue bars). Medium- and high-salience probes of all types were fixated significantly more frequently than standard green square distractors (t₍₉₎ > 6.5, p < 0.001; low-salience probes did not consistently attract attention and so were excluded from further analysis). The location of the probes had a substantial effect on their probability of fixation even in control subjects, with centrally-located probes fixated more often than those in more peripheral locations (Fig 3b & c; t₍₉₎ = 11.0, p < 0.001).

Figure 3. Stimulus-driven orienting to probes.

(a) Proportion of luminance (top), orientation (middle), and prior onset (bottom) probes fixated by control subjects (blue) and neglect patients (red), as a function of the probe’s salience level. Error bars indicate s.e.m.

(b) Proportion of probes fixated as a function of the horizontal position of the probe, from left to right of the display.

(c) Heat maps show the proportion of probes of each type fixated by controls (top) and patients (bottom) as a function of 2d probe position. Note how more centrally-located probes attract a greater proportion of control subject’s fixations.

Importantly, however, even the most salient probes were fixated significantly less frequently than letter targets (68% v 97%; t₍₉₎ = 7.1, p < 0.001), revealing the influence of goal-relevance on orienting in healthy individuals, with eye movements towards targets much more frequent than towards task-irrelevant probes. When probes were fixated, it was on average earlier in a trial than for targets (t₍₉₎ = 3.1, p = 0.013), consistent with rapid involuntary orienting to probes driven by their high stimulus salience.

Neglect patients showed substantially reduced stimulus-driven orienting to the salient probes, fixating probes of all types (luminance, orientation and prior onset) significantly less frequently than controls (Fig 3a, red bars; t₍₁₃₎ > 3.5, p < 0.004). For each probe type, the spatial distribution of fixations showed a smooth decline from right to left, like that observed for the letter targets (Fig 3b & c). However, the frequency with which control subjects’ fixations were captured differed between probe types, and between probes and targets, across space. Therefore, in order to make a systematic comparison of patients’ impairment in the different conditions, we first normalized the fixation frequencies observed in patients by those obtained from controls (see Methods for details).

The results of this analysis are plotted in Fig 4. Once differences in normal performance for different probe types were taken into account, the patients’ deficit in orienting to probes was found to be equivalent for the different types of salience (Fig 4a, colored bars; t₍₄₎ < 1.6, p > 0.20). Crucially, the magnitude of the patients’ deficit was also equivalent for task-irrelevant salient probes and task-relevant targets (black bar; t₍₄₎ < 1.6, p > 0.20), indicating that the patient deficits for goal- and stimulus-driven orienting were statistically indistinguishable.

Figure 4. Comparison of neglect deficits for goal- and stimulus-driven orienting.

(a) Normalized frequency of patient fixations on targets (black) and luminance (red), orientation (green) and onset (blue) probes. A normalized ratio of 0 indicates no difference between patient and control performance; -1 indicates complete patient impairment. Error bars indicate s.e.m.

(b) Normalized fixation frequency as a function of horizontal position for targets (black lines) and probes (colored lines).

(c) Normalized frequency as a function of 2d position, for targets (top) and probes (bottom).

(d) Rightward bias in orienting to targets (black squares) and probes (colored circles) for each patient. Note the strong correlation between a patient’s bias in orienting to probes and to targets.

Similarly, the normalized spatial distribution of the neglect deficit along the horizontal axis was the same for each type of task-irrelevant probe (Fig 4b, colored lines; no significant difference at any horizontal location, t₍₄₎ < 1.5, p > 0.22). An identical distribution was obtained for task-relevant targets (Fig 4b, black line; t₍₄₎ < 2.1, p > 0.10). This correspondence was also observed in the vertical dimension, where the greater impairment seen for targets in inferior space was also observed for the salient probes (Fig 4c; t₍₄₎ = 3.0, p = 0.04). Although small compared to the left-right asymmetry, this advantage for items in superior over inferior space was consistently observed in all five patients (mean 18% greater frequency of fixation; range 6%–36%). Target and probe deficits also did not differ significantly in any individual quadrant of the display (t₍₄₎ < 1.6, p > 0.20).

An analysis of results from individual patients further confirmed the correspondence between goal-directed and stimulus-driven deficits. While every patient showed some degree of rightward bias, the magnitude of this spatial bias varied substantially between patients (Fig 4d; see also Supplementary Figure 2), as in other studies of neglect. But, in further support of a common basis for goal- and stimulus-related deficits, we found a strong correlation between an individual patient’s horizontal bias in orienting to task-relevant targets and their (independently measured) horizontal bias in orienting to task-irrelevant probes, with r² = 0.98 (p = 0.001). This very high correlation is consistent with disruption to an integrated representation of both goal-relevance and stimulus salience in human parietal cortex.

Competition for attention

In the first experiment, we separately examined the parietal neglect deficit in orienting to visual items based on their goal-relevance or visual salience, and found identical deficits in each case. This supports the hypothesis that goal- and stimulus-related signals are combined in posterior parietal cortex to produce a single representation of priority, damage to which results in spatial neglect. Next, in a second experiment, we examined how conflicts within this priority map are resolved.

Subjects in Experiment 2 were again presented with a sequence of simple search arrays. Each array consisted of an isoluminant grid of differently-colored letters (Fig 5). Each subject was given a target color (red or blue) at the start of the experiment and instructed to search for and respond to letter Xs only of that color, ignoring all other colors. Green letters acted as distractors, and now single letters of the non-target color (blue or red) acted as probes. Whereas, in the previous experiment, targets (embedded within the isoluminant green distractor squares) were low in stimulus salience compared to probes, in this experiment targets and probes were matched for stimulus salience. This design was chosen to facilitate a more detailed assessment of goal-driven orienting, as targets and probes were effectively identical except in their relevance to the task. Targets and probes could be presented on separate trials, as in the first experiment, or simultaneously within a single array, so there was direct competition between them.

In the first experiment, we did not constrain eye movements at any stage of the task. One consequence was that neglect patients’ initial gaze positions at the time a search array was presented tended to be to the right of mid-line. As a result, items on the left of the search array would have initially appeared further in the periphery of the visual field for patients than controls. To rule out any explanation for our results based on bias in resting gaze position, in the second experiment each array was preceded by a ‘dynamic’ fixation cross (which initially filled the display but rapidly zoomed-in to a randomly-selected location). This technique proved effective in guiding even a neglect patient’s gaze to leftward starting locations. The search array was presented only once the subject was fixating the cross, and we analysed the direction of the first eye movement from this initial location, thus ensuring that both patient and control performance was based on observing the same retinal locations of targets and probes.

Fig 6a shows performance of the control group on trials in which either a probe or a target was presented among green distractor letters. Despite explicit instructions to ignore items not of the target color, when a probe was presented, healthy control subjects’ initial eye movements were directed towards it more frequently than expected by chance (55% of trials; t₍₇₎ = 7.5, p < 0.001; Fig 6a, dashed line), indicating involuntary orienting to the singleton probe. However, this frequency of orienting was significantly less than that observed when the lone item was of the target color (71%; t₍₇₎ = 4.1, p = 0.004; Fig 6a, solid line).

Figure 6. Orienting to separately and simultaneously-presented targets and probes.

(a) Proportion of first saccades directed towards the target letter on target-only trials (solid line) and towards the probe letter on probe-only trials (dashed line) by healthy controls, as a function of the horizontal position of the target/probe.

(b) Proportion of first saccades directed towards the target letter (solid line) and probe letter (dashed line) by controls on target+probe trials, as a function of the horizontal locations of target (T) and probe (P) to left or right of subject mid-line. Error bars indicate s.e.m.

(c & d) Corresponding results for patient DC, a neglect patient with a posterior parietal lesion, plotted as in a & b. Note that both target (solid line) and probe responses (dashed line) display a gradient of impairment from normal performance at far right [compare (c) and (a)] falling to chance performance at far left. Error bars indicate s.e. of the binomial probability estimate.

(e & f) Corresponding results for patient JK, a frontal neglect patient without parietal damage. Note that both target and probe responses display a horizontal gradient of impairment as in parietal neglect. Additionally, orienting to ipsilesional (rightward) probes in the probe-only condition is enhanced in comparison to controls [compare dashed lines in (e) and (a)].

As target and probe items were matched for visual salience (no significant effect of color, red v blue: t₍₇₎ = 0.84, p = 0.41) this additional probability of orienting to targets must correspond to a purely goal-driven component of attention, resulting from the relevance of the target color to the task goal.

Analysis of performance of a parietal neglect patient (DC) on this task confirmed the presence of impairments in both goal- and stimulus-driven components of orienting. The frequency of stimulus-driven eye movements to the task-irrelevant probes exhibited a gradient of deficit from approximately normal performance at the far right of the display (36% of trials, compared to 41% for controls; Fig 6c, dashed line) diminishing to chance levels at the far left (20%; χ²₍₁₎ = 0.43, p = 0.49). At the far right of the display, eye movements to targets were significantly more frequent than to probes (79% v 36%; χ²₍₁₎ = 16.2, p < 0.001; Fig 6c, solid line), as observed for healthy individuals. However, this goal-driven advantage for targets over probes also exhibited a horizontal gradient of deficit, falling to non-significance at the far left of the display (26% v 20%, χ²₍₁₎ = 0.52, p = 0.47).

These results are fully consistent with our findings in the first experiment and, because the initial fixation on each array was controlled, cannot be accounted for by biases in initial gaze position. To further examine the relationship between goal- and stimulus-driven components of attention, a proportion of search arrays contained both a target letter and a probe letter (Supplementary Figure 3, top). As both items attracted attention when presented separately, we hypothesised that presenting them simultaneously would create a conflict that would need to be resolved by competition within the priority map. As targets had the same visual salience as probes but were also goal-relevant, their representation in a combined priority map should be stronger, so the competition for attention should be resolved in their favour (Supplementary Figure 3, middle). This was indeed the behaviour demonstrated by control subjects on these trials, who oriented gaze towards the target with a similar frequency as when no probe was present (66%; Fig 6b, solid line), while the frequency of eye movements towards the probe fell to chance (11%; Fig 6b, dashed line).

In our neglect patient with parietal damage, the outcome of the competition depended on the relative horizontal positions of target and probe. The results matched the predictions of a damaged priority map in which items in contralesional space are under-represented (Supplementary Figure 3, bottom). When the target was presented in the intact right hemispace (Fig 6d, right two columns), behaviour was similar to controls: eye movements were directed towards the target with similar frequencies as in the target-only condition (76%) and towards the probe no more often than chance (7%). These results are consistent with a preserved representation of the priority of the ipsilesional target (Supplementary Figure 3a & b).

However, when both target and probe were in the neglected left hemispace (Fig 6d, second column), orienting to targets decreased significantly (54%; χ²₍₁₎ = 4.4, p = 0.036), while nonetheless remaining more frequent than orienting to probes (25%; χ²₍₁₎ = 4.3, p = 0.039). This result is consistent with a simultaneous devaluing of the priority of both target and probe relative to distractors in the intact hemispace (Supplementary Figure 3c).

Finally, when the target was in the neglected left hemispace and the probe in the intact right hemispace, the normal pattern was reversed and initial eye movements were more frequent towards the probe than the target (46% v 24%; χ²₍₁₎ = 4.8, p = 0.029; Fig 6d, first column). In this situation, the priority of the target may be devalued sufficiently for the competition for attention to be resolved in favour of the probe (Supplementary Figure 3d).

These findings go beyond the predictions of a conventional extinction account (e.g. Mattingley et al., 1997; Driver and Mattingley, 1998; Wardak et al., 2002), in which the simultaneous presentation of a rightward stimulus extinguishes awareness of a stimulus on the left. Horizontal locations alone were insufficient to determine which item would be fixated; instead, the direction of the initial eye movement depended on an interaction between stimulus location and attentional priority. Hence, a high priority item (target) on the left demonstrated a weaker claim on attention than a lower priority item (probe) on the right (Fig 6d, first column). But the advantage for the rightward item was far smaller than in the reverse case, when the high priority item was on the right and low priority on the left (Fig 6d, compare first column: 46% v 24%, with last column: 76% v 7%).

Neglect following a frontal lesion

While the neglect syndrome is typically associated with parietal damage, it is on rare occasions observed as a consequence of a purely frontal lesion (Heilman and Valenstein, 1972; Vallar and Perani, 1986; Husain and Kennard, 1996). The results presented so far suggest that parietal neglect reflects equal deficits in representing both goal- and stimulus-based priority. To investigate whether these two types of attentional selection are differentially affected by frontal lesions we examined performance of a further patient (JK), who had developed left neglect as the result of a stroke affecting the right frontal lobe, with no parietal involvement (lesion plotted in Supplementary Figure 4).

Figs 6e & f show performance of this frontal neglect patient on Experiment 2. Like the patient with parietal damage, JK demonstrated a gradient of increasing deficit from right to left in orienting to task-relevant targets (Fig 6e, solid line; χ²₍₁₎ = 7.8, p = 0.005), falling from normal performance at the far right (70%) to chance at the far left (26%). A similar gradient of deficit was also observed for orienting to the task-irrelevant probes (Fig 6e, dashed line; χ²₍₁₎ = 5.5, p = 0.019). These results indicate that frontal damage, like parietal, causes deficits in both goal- and stimulus-driven orienting. However, unlike both healthy individuals and parietal patient DC, the frontal patient fixated probes with the same frequency as targets, even when presented in the ipsilesional (non-neglected) hemispace (χ²₍₁₎ < 0.9, p > 0.36).

Fig 6f shows performance of the frontal patient on ‘competition’ trials in which both a target and a probe were presented. As for the parietal patient (Fig 6d), the outcome of these trials depended on the relative horizontal positions of target and probe, again consistent with the predictions of a damaged representation of contralesional priority. Crucially, when both items were presented in the intact hemispace (third column), this patient consistently oriented to the target (80%) rather than the probe (which was fixated no more often than chance, χ²₍₁₎ < 0.1, p = 0.84). So the absence for this patient of the usual advantage for targets over probes when presented individually (Fig 6e) is not due to a failure to distinguish the two colors or understand the demands of the task. Rather it implies a separate deficit in suppressing reflexive eye movements to salient but task-irrelevant visual stimuli that may be a specific consequence of this patient’s frontal lesion. The patient found it difficult to ignore probes when presented alone, but could successfully select the target over the probe when both were presented together.

Manipulating salience to counteract the lateral bias in neglect

The results of the first two experiments strongly support the hypothesis that visual neglect is caused by damage to a unified representation of attentional priority. Right parietal damage may leave items in left space under-represented within this ‘priority map’, with the result that leftward items are relatively unlikely to be selected as targets for a saccade. If this view is correct, it should be possible to normalize a patient’s exploratory behaviour by biasing the distribution of visual salience to the left, i.e. by enhancing the physical salience of leftward items in order to counteract their weakened representation in the priority map.

In contrast, if neglect is due to damage at the output stage, i.e. difficulty in executing gaze shifts to leftward items following target selection, then manipulations affecting the conspicuity of leftward items will be unable to correct the exploration deficit. To distinguish between these two hypotheses and to investigate whether neglect might be ameliorated by using these principles, we performed a further study.

Two neglect patients with right parietal damage (KB & LE) took part in Experiment 3. As in the first experiment, the patients searched an array of green squares for letter targets, but now the distribution of luminance in the display was skewed by varying degrees to left or right (examples in Supplementary Figure 5a–c), using an on-line, gaze-contingent adaptive algorithm (Supplementary Figure 6; see Methods for full details). A target letter was present on 50% of trials. On the remaining trials only distractors (green squares) were presented, so the distribution of fixations on these trials could be taken as a pure measure of visual exploration.

As expected, when (baseline) arrays of uniform luminance were presented, both patients showed a significant rightward bias in their exploration (Fig 7c; mean horizontal fixation position: patient KB, 11.0° right of mid-line; patient LE, 5.3° right of mid-line; t₍₁₅₉₎ > 14.3, p < 0.001). However, by biasing the spatial distribution of distractor luminance to right or left we were able to alter the pattern of exploratory eye movements (Fig 7a & b). Our adaptive algorithm was programmed to search for a luminance distribution that would cancel out each patient’s rightward exploratory bias, such that fixations were distributed symmetrically about the midline. The algorithm converged successfully for each patient (KB, γ₀ = −2.6, s.e. 0.2; LE, γ₀ = −1.3, s.e. 0.3; see Methods for details). The obtained luminance distributions are shown in the insets of Fig 7a & b, and the corresponding spatial distributions of exploratory fixations are shown in Fig 7d.

Figure 7. Counteracting the ipsilesional bias in neglect.

(a & b) Mean horizontal location of exploratory fixations on null trials as a function of bias of the background luminance (γ), for (a) patient KB and (b) patient LE. Red lines represent the cumulative gaussian functions that best fit each patient’s data; the arrows indicate the luminance bias (γ₀) that minimizes the mean deviation of each patient’s exploratory fixations from the centre of the display. The corresponding search array is shown in the inset.

(c, d & e) Distribution of exploratory fixations on null trials for patient KB (left) and patient LE (right), when (c) display luminance is uniform (γ = 0), (d) luminance is adjusted to compensate for each patient’s rightward bias (γ = γ₀), and (e) luminance is biased further to the left (γ < γ₀).

When the luminance distribution was skewed even further to the left, we found that significant leftward exploratory biases were now induced in both patients, such that they preferentially explored the contralesional (previously neglected) hemispace rather than the ipsilesional ‘intact’ space (Fig 7e; KB: t₍₁₄₉₎ = 3.5, p < 0.001; LE: t₍₈₈₎ = 8.9, p < 0.001).

Because of the way we parameterized the space of possible luminance distributions (see Methods), the ‘optimal’ display identified for each patient by the adaptive algorithm fell in a range (γ < −1) where luminance was exponentially related to horizontal position, and some of the rightmost distractors may not have been visible. However, significant corrective effects on exploratory bias were observed with much less skewed luminance distributions, e.g. displays defined by γ = −0.75 corrected 89% of the rightward bias for patient LE (t₍₈₉₎ = 2.43, p = 0.017) and 45% for patient KD (t₍₈₀₎ = 2.85, p = 0.006). All distractors were visible in these displays: the minimum distractor luminance was 2.2 cd m⁻² (6.6 times brighter than the background); for comparison, patients in this experiment successfully fixated and responded to (ipsilesional) targets with luminance as low as 0.5 cd m⁻².

We wanted to know whether the manipulation of stimulus salience could compensate for neglect patients’ impairment in goal-driven orienting to left hemispace targets, and so improve their ability to perform the search task. An analysis of the target-present trials showed that, compared to the uniform array, skewing the display luminance to the left resulted in a significant increase in successful fixation of leftward targets: from 20% to 65% for patient KB (χ²₍₁₎ = 23.6, p < 0.001), and from 15% to 32% for patient LE (χ²₍₁₎ = 4.1, p = 0.042). However, because the luminance of right hemispace targets was correspondingly decreased, right targets were more difficult to detect leading to no overall improvement in search performance.

We therefore tested one patient (KB) in a further condition in which target luminance was held constant, and only the luminance of distractors was manipulated (example in Supplementary Figure 5c). The luminance distribution that normalized exploration now also improved overall performance on the search task by 30%, significantly increasing the number of fixations on left hemispace targets (from 20% to 55%; χ²₍₁₎ = 10.5, p = 0.001) while leaving rightward target fixations unaffected (82% v 78%; χ²₍₁₎ = 0.3; p = 0.58). Thus, this simple manipulation was sufficient to produce overall improvements in visual search performance.

Discussion

In this study we investigated the visual orienting deficits associated with spatial neglect following parietal lesions. We observed impairments in both goal-directed and stimulus-driven components of attention, and mapped out the two-dimensional spatial distribution of the deficit in each case (Experiment 1). Patients’ impairments in stimulus-driven orienting manifested as a reduction in the involuntary capture of gaze by visually-salient items in contralesional space. This deficit proved indistinguishable, in terms of both magnitude and spatial distribution, for three very different kinds of stimulus salience (defined by contrasts in luminance, onset or orientation). This finding is consistent with lateralized damage to a single representation of visual salience, that integrates information about the conspicuousness of a stimulus across multiple visual dimensions (Koch and Ullman, 1985; Pouget and Driver, 2000; Balan and Gottlieb, 2006; Töllner et al., 2010).

Healthy orienting of attention, as well as being driven by the bottom-up salience in a scene, can independently be guided top-down by our current goals and intentions. Thus a healthy individual can rapidly direct attention to a target object based on a feature that uniquely identifies it (e.g. a particular color) even when other objects in the scene have equal or greater visual salience (Duncan, 1989; Smallman and Boynton, 1990; Wolfe et al., 1990). Correspondingly, in monkeys, neural activity in multiple regions of the attentional network is independently correlated with both the visual salience and goal-relevance of a stimulus (Gottlieb et al., 1998; Balan and Gottlieb, 2006; Buschman and Miller, 2007). However, the causal role played by different brain areas in selecting targets for attention has not been established. To make such inferences, lesion or inactivation studies are necessary. One proposal that has recently come to the fore (Buschman and Miller, 2007) suggests separate roles for frontal areas in signalling the goal-relevance of visual objects, and posterior parietal areas in representing their stimulus salience. However this hypothesis is also supported only by correlations between neural activity and visual input.

If it were correct, we would expect parietal patients’ ability to detect and orient to task-relevant targets – which in healthy individuals can be guided top-down by knowledge of target features – to be less impaired than their tendency to be distracted by goal-irrelevant but visually-salient probes. In fact, we observed identical deficits, and indistinguishable spatial gradients, for orienting to targets as to the salient probes, implying that posterior parietal cortex is critical for both goal-directed and stimulus-driven control of attention.

The degree of ipsilesional bias varied substantially from patient to patient, reflecting heterogeneity in the severity of their lesions and in the time of testing post-stroke. Importantly, however, there was a remarkably strong correlation (r² = 0.98) between the goal-directed and stimulus-driven deficits observed in each patient, making it unlikely that the impairments arise from damage to separate neural representations. Instead, this result strongly suggests that the posterior neglect syndrome involves damage to a unified representation, combining both goal- and stimulus-related signals.

Such a representation may take the form of a topographically-organized ‘priority map’, in which spatially-selective neurons integrate top-down and bottom-up inputs to signal the overall attentional priority of items within their receptive field, with attention shifting to the location with the highest activation (Wolfe, 1994; Serences and Yantis, 2006; Bisley and Goldberg, 2003; Koch and Ullman, 1985; Itti and Koch, 2001; Fecteau and Munoz, 2006). If this hypothesis is correct, left neglect following right-hemisphere damage might be associated with under-representation of contralesional space in the priority map, resulting in a pathological bias of attention towards right hemispace.

This proposal is supported by the results of Experiment 2, in which goal-relevant and visually-salient items were presented simultaneously. Healthy controls preferentially directed attention to the stimulus with the highest combined priority (the target). Neglect performance was consistent with under-representation of the priority of contralesional items. Specifically, when salient probes were presented in the neglected hemispace, the normal priority relationship was preserved and saccades were preferentially directed to the item with the highest combined priority. However, when the lower priority probe was presented in the intact hemispace and the higher priority target to the neglected side, the low priority item prevailed in the competition for attention (see Fig. 6 and Supplementary Figure 3).

Of course, this proposal does not mean that other cognitive deficits might not also contribute to the neglect syndrome, which is considered to be composed of different component deficits (Duncan et al., 1999; Karnath et al., 2002; Husain and Rorden, 2003; Milner and McIntosh, 2005; Soto et al., 2009). The precise anatomical locus of the posterior neglect syndrome is also vigorously debated (Vallar and Perani, 1986; Karnath et al., 2001; Corbetta and Shulman, 2002; Mort et al., 2003; Bartolomeo et al., 2007). It is not the goal of the current study to resolve this issue, and while lesions in the posterior neglect group were centred on the inferior parietal lobe (IPL), the relatively small number of patients and absence of highly focal lesions make this an unsuitable study to address this question directly.

Stimulus-driven orienting was investigated in the present study using task-irrelevant probe items that derived their salience ‘bottom-up’ from visual contrast with other items in the same display. This type of stimulus-driven salience has been most closely associated with monkey lateral intraparietal area (LIP), which may have its human homologue in the intraparietal sulcus (Culham and Kanwisher, 2001). Although the centre of lesion overlap was in the IPL, all our parietal patients’ lesions extended to this area.

LIP has been proposed to play a role in covert visual attention, memory of spatial targets and overt shifts of gaze (Snyder et al., 1997; Gottlieb et al., 1998; Bisley and Goldberg, 2003; Kusunoki et al., 2000; Balan and Gottlieb, 2006; Gnadt and Andersen, 1988; Mazzoni et al., 1996). A priority map might be important for all three functions. Reversible inactivation of LIP has also been shown to slow goal-directed responses to contralateral targets and impair contralateral visual search (Wardak et al., 2002, 2004; Balan and Gottlieb, 2009).

In monkeys, LIP is one of several regions of the oculomotor network where activity has been identified that may correspond to a salience or priority map. In particular, activity of neurons in the frontal eye fields (FEF) has also been shown to reflect both the salience and relevance of visual stimuli (Schall and Thompson, 1999; Thompson and Bichot, 2005; Wardak et al., 2006; Buschman and Miller, 2007). Perhaps analogously, while visual neglect in humans is most commonly a consequence of parietal lesions, it is also more rarely observed following damage to the frontal lobe (Heilman and Valenstein, 1972; Vallar and Perani, 1986; Husain and Kennard, 1996). The proposal examined in this study – that parietal and frontal areas have separate roles in representing visual salience and goal-relevance, respectively – leads to the prediction that reflexive capture of gaze by visually salient stimuli should be relatively spared in frontal neglect.

To test this, we examined performance of a patient with neglect following a purely anterior lesion (Fig 6e & f). Contrary to the prediction, this patient showed a gradient of impairment in reflexive eye movements to salient but task-irrelevant probes that was identical to her impairment in orienting to goal-relevant targets, and which – as in parietal neglect – culminated in a complete deficit in orienting to salient items in far contralesional space. This result provides further evidence against the hypothesis that frontal and parietal components of the attentional network can be straightforwardly dissociated on the basis of their contributions to contralateral goal- and stimulus-driven control. It is notable, however, that this frontal neglect patient was abnormally responsive to ipsilesional probes when competition for attention was low. This could reflect an additional, spatially non-specific, involvement of frontal areas in suppressing reflexive responses to task-irrelevant stimuli (see also Husain and Kennard, 1997).

The priority map hypothesis proposes that targets of attention are selected by competition. Therefore, shifts of attention to left hemispace should still be possible in neglect as long as the leftward input is strong enough to overcome the lesion-induced advantage for rightward locations. This can take the form of a powerful top-down input, as when highly-predictive pre-cues direct a patient’s attention to left space (Ladavas et al., 1994; Smania et al., 1998; Bartolomeo et al., 2001). Here, we have shown that bottom-up visual inputs can also direct attention into ‘neglected’ space.

Biasing the distribution of luminance to the left of the visual field allowed us to overcome, and even reverse, the usual rightward bias in neglect (Experiment 3). This result provides further confirmation that neglect results from a deficit in representation rather than output: if neglect reflected an impairment in executing gaze shifts to leftward items following target selection, then manipulations affecting the conspicuousness of leftward items would be unable to correct the exploration deficit. The luminance manipulation also significantly improved patients’ ability to respond to targets in the contralesional hemispace, with potential implications for the treatment of neglect. Attempts to ameliorate neglect by training patients to exert top-down control have met with only limited success (Parton et al., 2004); future investigation could be directed towards alleviating symptoms by filtering or otherwise manipulating the visual input across space.