Integration of Goal- and Stimulus-Related Visual Signals Revealed by Damage to Human Parietal Cortex

Abstract

Where we look is determined both by our current intentions and by the tendency of visually salient items to “catch our eye.” After damage to parietal cortex, the normal process of directing attention is often profoundly impaired. Here, we tracked parietal patients' eye movements during visual search to separately map impairments in goal-directed orienting to targets versus stimulus-driven gaze shifts to salient but task-irrelevant probes. Deficits in these two distinct types of attentional selection are shown to be identical in both magnitude and spatial distribution, consistent with damage to a “priority map” that integrates goal- and stimulus-related signals to select visual targets. When goal-relevant and visually salient items compete for attention, the outcome depends on a biased competition in which the priority of contralesional targets is undervalued. On the basis of these findings, we further demonstrate that parietal patients' spatial bias (neglect) in goal-directed visual exploration can be corrected and even reversed by systematically manipulating the spatial distribution of stimulus salience in the visual array.

Introduction

To extract useful information from the visual world, we must direct our attention serially to different regions of space. In general, this involves making overt shifts of gaze to fixate objects of interest within the visual field. Attentional selection is thought to be determined by both an object's relevance to our current goals and its inherent visual salience (Posner, 1980; Bundesen, 1990; Wolfe, 1994; Desimone and Duncan, 1995; Kastner and Ungerleider, 2000). Experimentally, these two components can be distinguished in visual search. Goal-directed factors are demonstrated by our ability to detect and orient to a target object on the basis of a prespecified feature (such as color or location) distinguishing it from distractors (Treisman and Gelade, 1980; Egeth et al., 1984; Wolfe, 1998). Stimulus-driven orienting is observed in the tendency for distractors with high visual salience to automatically capture attention even though task irrelevant (Jonides and Yantis, 1988; Theeuwes et al., 1998; Lamy and Zoaris, 2009).

Neuroimaging and neurophysiological studies have uncovered a network of frontal and parietal cortical areas associated with control of attention and eye movements (Andersen, 1995; Colby and Goldberg, 1999; Schall and Thompson, 1999; Kanwisher and Wojciulik, 2000; Corbetta and Shulman, 2002; Serences and Yantis, 2006; Monosov et al., 2008). However, the extent to which different areas within this attentional network contribute to goal- or stimulus-driven orienting remains controversial (Kincade et al., 2005; Peers et al., 2005; Natale et al., 2008). Differences in timing of goal- and stimulus-related activity between frontal and parietal cortex have been reported recently in macaque (Buschman and Miller, 2007), suggesting that goal-directed signals perhaps originate in frontal regions, whereas stimulus-driven orienting is the responsibility of parietal cortex. However, since activity in both regions ultimately correlates with attention shifts, interpretation of this temporal dissociation remains open.

To determine whether particular brain regions have a causal role in deciding the final target of attention, we must examine situations in which normal brain activity is disrupted. In humans, damage to the right parietal cortex commonly results in visual neglect, a syndrome characterized by failure to orient to the left of space (Vallar and Perani, 1986; Mort et al., 2003). Here we record eye movements of parietal patients during visual search. We show that their neglect deficits comprise equal impairments in goal-directed and stimulus-driven orienting of gaze, and that the spatial distributions of the two types of impairment are indistinguishable.

These results imply that posterior parietal cortex is critical for both goal-directed and stimulus-driven control of attention. Thus, neglect may be a consequence of damage to a “priority map” within posterior parietal cortex, where goal- and stimulus-related signals are combined to produce a unified representation of attentional priority (Bisley and Goldberg, 2003; Balan and Gottlieb, 2006). Consistent with this hypothesis, we further show that when goal-relevant and visually salient items compete for attention, the outcome depends on a biased competition in which the priority of contralesional items is underrepresented. Finally, we show that the lateralized exploration bias shown by neglect patients can be corrected and even reversed by systematically manipulating the distribution of bottom-up salience within the visual scene, providing additional evidence for a distortion of attentional priority, and a new mechanism for improving visual inattention.

Materials and Methods

Subjects and apparatus.

In total, 9 patients and 18 control subjects participated in the study after giving informed consent. All procedures were approved by the National Hospital for Neurology and Neurosurgery Ethics Committee. The patients had all received a diagnosis of left visual neglect after a stroke involving the right parietal lobe (with the exception of patient J.K., whose data are examined separately in experiment 2 as a case of frontal neglect without parietal involvement). All demonstrated neglect clinically in everyday life. In addition, they showed a rightward bias on at least one of two standard pen-and-paper tests of neglect conducted on the day of testing: line bisection (Schenkenberg et al., 1980) and bells cancellation (Gauthier et al., 1989). See supplemental Table 1 (available at www.jneurosci.org as supplemental material) for additional clinical details. Control subjects were neurologically healthy individuals with normal or corrected-to-normal vision, matched in age (median 70 years) to the patient group.

Stimuli were displayed on a 24 inch widescreen flat panel monitor, viewed at a distance of 60 cm. Eye position was monitored online at 1000 Hz using a frame-mounted infrared eye tracker (SR Research). Subjects were positioned within the apparatus such that head and body midlines were aligned with the horizontal center of the display; this alignment was monitored visually throughout the experiment and corrected as necessary; no systematic shifts were observed.

Experiment 1.

We presented subjects (5 neglect, 10 control) with a series of search arrays (see Fig. 1a). Each array contained 40 green squares (2.5° × 2.5°; 5.3 cd m⁻²) uniformly arranged in a grid pattern on a black background (0.29 cd m⁻²), covering an area of the visual field 24.5° high and 41° wide. Each trial began with a blank screen. Once a stable fixation was detected within the display area, a search array was presented for 1250 ms. The display was then blanked for 250 ms before beginning the next trial. Subjects were instructed to search the array for red letters and respond to the letter X by pressing a response button as quickly as possible. The green squares acted as distractors. Note that these were different from probes (see below), which we used to index stimulus-driven orienting.

Figure 1.

Visual search for target letters with salient nontargets. a, Structure of the task: subjects were presented with a sequence of randomly interleaved target and probe arrays. b, A typical target display, consisting of distractors (green squares) and a single target (red letter). Arrows indicate one healthy subject's sequence of fixations on the array: here the red target A (isoluminant with the green squares) is located on the second eye movement. c, A typical probe display, consisting of distractors and a single high-salience orientation probe (in this case a rotated square; note that probes could differ from distractors either in luminance, onset, or, as here, orientation). Arrows indicate one healthy subject's scan path: despite its irrelevance to the search task, an immediate eye movement is provoked by the visual salience of the probe.

Each subject completed 720 trials, comprising 360 target trials and 360 probe trials presented in a random order. On target trials, a single red letter (2° height) was present at a random location in the search array, a minimum of 5.5° (one grid separation) from the subject's initial fixation position (see Fig. 1b). The letter was either an X (50% of trials) or a different randomly selected letter (50% of trials). A button press in response to the an X was rewarded with a pleasant tone, whereas a press after any other letter was punished with an unpleasant beep.

The red letter targets were perceptually isoluminant with the green distractors (mean luminance, 6.6 cd m⁻²). Perceptual isoluminance was determined separately for each subject before the experiment (subjects adjusted the luminance of red squares in a red/green checkerboard to match perceived brightness; the average of five repetitions determined target luminance for the subsequent experiment).

On probe trials, the visual salience of a single, randomly chosen array item (minimum 5.5° from fixation) was increased in one of three ways: by altering its luminance, orientation, or onset (for orientation example, see Fig. 1c). Probes were completely irrelevant to the task and were designed to capture attention and gaze in a stimulus-driven fashion. The salience of probes was parameterized into three levels (low, medium, and high) for each type of probe. Luminance probes were brighter than the green distractors (by 33, 300, or 1500%), orientation probes were rotated relative to distractors (by 2, 15, or 45° clockwise), and onset probes appeared before the rest of the array (which was delayed by 33, 166, or 500 ms). Probes of each type and level were presented with equal frequency in a randomized order.

As our measure of performance on the search task, we calculated for each subject the proportion of target trials on which the target was successfully fixated (criteria: eye velocity of <50° s⁻¹; duration of >100 ms; distance of <2.75° from center of target). As our measure of reflexive orienting, the proportion of probe trials on which the probe was fixated was calculated according to the same criteria, for each combination of salience type and level.

To compare a neglect patient's deficit for targets with his or her deficit for probes, we calculated for each type of item a ratio (R) of the proportion of items fixated by the patient (p) and the mean proportion fixated by controls (c):

Whereas a simple ratio can take any value from zero to infinity, this normalized ratio is confined to the range −1 ≤ R ≤ 1. A value of R = 0 indicates that a patient fixated the same proportion of items as controls (i.e., showed no impairment). Negative R values reflect the magnitude of deficit in patient performance compared to controls, with R = −1 indicating that the patient fixated none of the items (i.e., complete impairment). Positive values of R occur in cases where the patient fixated a greater proportion of items than controls.

To obtain a single value reflecting a patient's horizontal bias in orienting to a particular type of item, we used the following formula:

where x_i indicates the horizontal position of the ith column of distractors, and _R′_i is the deficit ratio (R) for items appearing within that column, calculated as above and normalized to the range 0 ≤ R′ ≤ 1, i.e., R′ = ½(R + 1).

Experiment 2.

As in the first experiment, subjects (now one parietal neglect, one frontal neglect, and eight controls) were presented with a series of search arrays and instructed to respond with a button press to the appearance of a letter X. Unlike in experiment 1, each subject was randomly assigned a target color (red or blue, counterbalanced within each group) and instructed to respond only to a letter X of that color, and to ignore letters of any other color. Each search array consisted of 20 isoluminant letters (height 1.5°, 7.5 cd m⁻²) uniformly arranged in a grid pattern on a black background, creating a total search area 24° high and 30° wide. The majority of letters in each array were green distractors (see Fig. 5).

Figure 5.

Visual search with simultaneously presented targets and probes. On each trial, subjects were presented with a search array consisting of a grid of green distractor letters and either a single letter of the target color (target-only trial), a single letter of the nontarget color (probe-only trial), or one letter of each color, as shown here (target-plus-probe trial). In the example shown, the subject has been instructed to search for blue letters and ignore other colors, making the red letter the probe; 50% of subjects received opposite instructions. Subjects' initial gaze locations at array onset were controlled by presentation of a dynamic fixation cross, which initially filled the display and then rapidly zoomed in to a randomly selected grid location.

Three different types of trial were tested in equal numbers and in a random order: target-only trials, probe-only trials, and target-plus-probe trials (where both target and probe were presented simultaneously). On target-only trials, a single letter of the target color was present at a random location in the array; on probe-only trials, a single letter of the nontarget color was shown; and on target-plus-probe trials, one letter of the target color and one letter of the nontarget color were presented. Each array was displayed for 2000 ms, followed by a 1000 ms blank period before the start of the next trial. Control subjects each completed 600 trials in total, as did the parietal neglect patient (D.C.); the frontal neglect patient (J.K.) completed 360 trials.

Subjects' initial gaze position was controlled by presenting a fixation cross at the start of each trial. The cross was presented at a random array location on each trial, sampling equally from all horizontal positions during the course of the experimental session. As static fixation crosses often fail to attract gaze in neglect, each cross was initially presented at a size of 55° across and rapidly “zoomed in” to its final position, appearing at a final size of 1.8° across. The array appeared as soon as a fixation was recorded within 3° of the final position of the cross.

In this experiment, unlike the previous one, we took as our performance measure the direction of the first saccade recorded after array onset (criterion: eye velocity of >50° s⁻¹). This eye movement was considered to be directed toward a target (or probe) if its mean direction fell within 30° of the direction from the fixation cross to the center of the target (or probe) letter. Targets and probes were presented a minimum distance of 7.5° from the fixation cross, and on target-plus-probe trials they were separated from each other by a minimum angle of 90° relative to initial gaze position.

Experiment 3.

Two neglect patients, who had not participated in the previous experiments, were tested on a modified version of the search task used in experiment 1. Fifty percent of trials were target trials, comprising an array of green square distractors and a single red letter target, as described previously. Unlike in experiment 1, the remaining 50% were now null trials, in which the distractor array (green squares) was presented without targets or probes. As a measure of visual exploration, we calculated the mean horizontal position of the fixations recorded on each of these null trials. Each patient completed 640 trials in total. For the first 160 trials, distractors were of uniform luminance (31 cd m⁻²). Subsequently, an adaptive algorithm varied distractor luminance horizontally across the screen, trial by trial, to find the luminance distribution that minimized the mean deviation of fixations from the center of the display.

The luminance distribution on each trial was set by a parameter γ, which determined the relationship between the horizontal position of each distractor (x taking values in the range 0, for the leftmost distractor, to 1, for the rightmost) and its RGB brightness (b, range 0 to 1, proportional to the square root of luminance). When γ = 0, distractors were of uniform brightness. When γ was positive, the brightness of distractors increased smoothly from left to right: linearly for γ ≤ 1, b = ½ + (x − ½)γ, then exponentially for higher values, b = x^γ. Negative values of γ corresponded to horizontal reflections of these distributions, such that brightness increased from right to left (see supplemental Fig. 5, available at www.jneurosci.org as supplemental material). On target trials, the brightness of the red target letters was perceptually matched to the brightness of distractors at the same horizontal position.

The relationship between γ and the mean horizontal fixation position on null trials was approximated by a cumulative Gaussian function. Maximum likelihood estimation of the parameters of this function yielded an estimate of γ₀, the value of γ that would minimize the mean fixation deviation from the center of the display. After the initial 20 null trials (where the value of γ was chosen at random in the range −5 to 5), the value of γ tested on each subsequent trial was chosen randomly from a uniform distribution bounded by the 95% confidence limits on the current estimate of γ₀.

After the estimation session, one patient (K.B.) completed an extra experimental test, comprising an additional 160 trials, in which distractor luminance was determined by the final estimate of γ₀, but target luminance was uniform across the display (as for γ = 0).

Results

Goal-directed visual orienting

To distinguish goal-directed from stimulus-driven control of visual orienting, we recorded eye movements while participants performed a novel visual search task (experiment 1). Subjects were shown a sequence of distractor arrays (green squares) in which were embedded isoluminant task-relevant targets (red letters) (Fig. 1a,b). A target was present on 50% of trials (maximum one target per display). Participants were instructed to search the arrays for the red letters, and respond with a button press as quickly as possible only to the letter X. Targets were sufficiently small that foveation was required to make this letter discrimination: the frequency with which subjects successfully fixated the targets was used as a measure of success for the goal-directed control of attention.

The large search area and low stimulus salience of the small, equiluminant target letters required active search, with 39% of targets requiring two or more fixations to locate in healthy controls. Nonetheless, control subjects ultimately fixated 97% of the red letter targets within the presentation time of the array (Fig. 2a, blue bar) and responded correctly to the identity of all fixated letters. In comparison, patients with left neglect after lesions of the right parietal lobe (lesion overlap shown in Fig. 2d) demonstrated severe performance deficits on the search task, indicating an impairment in goal-directed orienting to the letter targets. They fixated a significantly smaller proportion of targets than controls (43%; t₍₁₃₎ = 14.0; p < 0.001) (Fig. 2a, red bar), and made a correct button response on only 32% of trials.

Figure 2.

Goal-directed orienting to targets. a, Proportion of targets fixated by healthy controls (blue) and neglect patients (red). Error bars indicate SEM. b, Proportion of targets fixated as a function of the horizontal position of the target, showing a gradient of impairment in patients increasing from right to left. c, Heat maps show the proportion of targets fixated by controls (top) and patients (bottom) as a function of two-dimensional target position (bicubic interpolation between target locations). Note the greater impairment in patients for inferior as well as leftward targets. d, Overlap map shows the location of lesions in the neglect group. The only area affected in all patients (red) involves the posterior parietal cortex and temporoparietal junction.

The effect of spatial location on the probability of fixating a target is shown in Figure 2, b and c. The poor performance of neglect patients on the task was attributable to a failure to fixate leftward targets, as might be anticipated from previous search studies (Johnston and Diller, 1986; Behrmann et al., 1997). This impairment consisted of a gradient of increasing deficit from right to left, with near-normal performance at the far right of the tested range, and an almost complete failure to fixate targets on the extreme left (Fig. 2b, red line). In the vertical dimension, neglect patients also showed a small but significant deficit in orienting to targets in the lower half of the display (Rapcsak et al., 1988; Pitzalis et al., 1997) (Fig. 2c, bottom) (t₍₄₎ = 6.5; p = 0.003). These findings reveal the spatial distribution of each patient's deficit in goal-directed orienting, i.e., in locating the red target embedded among isoluminant green distractor squares.

To assess deficits in orienting attention independent of deficits in object identification, we took as our principal performance measure the frequency with which targets were successfully fixated, rather than the frequency of correct button presses. However, parietal patients also demonstrated impairments in the manual responses that mirrored the spatial pattern of deficits in goal-directed orienting (supplemental Fig. 1, available at www.jneurosci.org as supplemental material). Response errors were more common for contralesional targets (t₍₄₎ = 3.5; p = 0.02) and, to a lesser extent, those in inferior space (t₍₄₎ = 3.7; p = 0.02), even when the target was successfully fixated.

Stimulus-driven visual orienting

To obtain a measure of stimulus-driven control of attention by task-irrelevant stimuli, on the remaining 50% of trials, which contained no red letter, a single salient “probe” stimulus was presented at a random location in the distractor array (Fig. 1a,c). Probes resembled the distractor items (green squares) but with a single adjustment that increased their visual salience: either an increase in luminance, a change in orientation, or an earlier onset. Probes of each type could be presented at one of three salience levels (low, medium, and high; see Materials and Methods) reflecting the amplitude of the difference between probe and distractor.

Despite their irrelevance to the search task, and despite sharing no visual features with the search targets (red letters), these salient probes attracted even control subjects' gaze on a substantial proportion of trials (Fig. 3a, blue bars). Medium- and high-salience probes of all types were fixated significantly more frequently than standard green square distractors (t₍₉₎ > 6.5; p < 0.001; low-salience probes did not consistently attract attention and so were excluded from additional analysis). The location of the probes had a substantial effect on their probability of fixation even in control subjects, with centrally located probes fixated more often than those in more peripheral locations (Fig. 3b,c) (t₍₉₎ = 11.0; p < 0.001).

Figure 3.

Stimulus-driven orienting to probes. a, Proportion of luminance (top), orientation (middle), and prior onset (bottom) probes fixated by control subjects (blue) and neglect patients (red) as a function of the probe's salience level. Error bars indicate SEM. b, Proportion of probes fixated as a function of the horizontal position of the probe, from left to right of the display. c, Heat maps show the proportion of probes of each type fixated by controls (top) and patients (bottom) as a function of two-dimensional probe position. Note how more centrally located probes attract a greater proportion of control subject's fixations.

Importantly, however, even the most salient probes were fixated significantly less frequently than letter targets (68% vs 97%; t₍₉₎ = 7.1; p < 0.001), revealing the influence of goal-relevance on orienting in healthy individuals, with eye movements toward targets much more frequent than toward task-irrelevant probes. When probes were fixated, it was, on average, earlier in a trial than for targets (t₍₉₎ = 3.1; p = 0.013), consistent with rapid involuntary orienting to probes driven by their high stimulus salience.

Neglect patients showed substantially reduced stimulus-driven orienting to the salient probes, fixating probes of all types (luminance, orientation, and prior onset) significantly less frequently than controls (Fig. 3a, red bars) (t₍₁₃₎ > 3.5; p < 0.004). For each probe type, the spatial distribution of fixations showed a smooth decline from right to left, like that observed for the letter targets (Fig. 3b,c). However, the frequency with which control subjects' fixations were captured differed between probe types, and between probes and targets, across space. Therefore, to make a systematic comparison of patients' impairment in the different conditions, we first normalized the fixation frequencies observed in patients by those obtained from controls (see Materials and Methods for details).

The results of this analysis are plotted in Figure 4. Once differences in normal performance for different probe types were taken into account, the patients' deficit in orienting to probes was found to be equivalent for the different types of salience (Fig. 4a, colored bars) (t₍₄₎ < 1.6; p > 0.20). Crucially, the magnitude of the patients' deficit was also equivalent for task-irrelevant salient probes and task-relevant targets (black bar) (t₍₄₎ < 1.6; p > 0.20), indicating that the patient deficits for goal- and stimulus-driven orienting were statistically indistinguishable.

Figure 4.

Comparison of neglect deficits for goal- and stimulus-driven orienting. a, Normalized frequency of patient fixations on targets (black) and luminance (red), orientation (green), and onset (blue) probes. A normalized ratio of 0 indicates no difference between patient and control performance; −1 indicates complete patient impairment. Error bars indicate SEM. b, Normalized fixation frequency as a function of horizontal position for targets (black lines) and probes (colored lines). c, Normalized frequency as a function of two-dimensional position for targets (top) and probes (bottom). d, Rightward bias in orienting to targets (black squares) and probes (colored circles) for each patient. Note the strong correlation between a patient's bias in orienting to probes and to targets.

Similarly, the normalized spatial distribution of the neglect deficit along the horizontal axis was the same for each type of task-irrelevant probe (Fig. 4b, colored lines) (no significant difference at any horizontal location; t₍₄₎ < 1.5; p > 0.22). An identical distribution was obtained for task-relevant targets (Fig. 4b, black line) (t₍₄₎ < 2.1; p > 0.10). This correspondence was also observed in the vertical dimension, where the greater impairment seen for targets in inferior space was also observed for the salient probes (Fig. 4c; t₍₄₎ = 3.0; p = 0.04). Although small compared to the left–right asymmetry, this advantage for items in superior over inferior space was consistently observed in all five patients (mean 18% greater frequency of fixation; range 6–36%). Target and probe deficits also did not differ significantly in any individual quadrant of the display (t₍₄₎ < 1.6; p > 0.20).

An analysis of results from individual patients further confirmed the correspondence between goal-directed and stimulus-driven deficits. Although every patient showed some degree of rightward bias, the magnitude of this spatial bias varied substantially between patients (Fig. 4d; supplemental Fig. 2, available at www.jneurosci.org as supplemental material), as in other studies of neglect. But, in additional support of a common basis for goal- and stimulus-related deficits, we found a strong correlation between an individual patient's horizontal bias in orienting to task-relevant targets and their (independently measured) horizontal bias in orienting to task-irrelevant probes, with r² = 0.98 (p = 0.001). This very high correlation is consistent with disruption to an integrated representation of both goal-relevance and stimulus salience in human parietal cortex.

Competition for attention

In the first experiment, we separately examined the parietal neglect deficit in orienting to visual items based on their goal-relevance or visual salience, and found identical deficits in each case. This supports the hypothesis that goal- and stimulus-related signals are combined in posterior parietal cortex to produce a single representation of priority, damage to which results in spatial neglect. Next, in a second experiment, we examined how conflicts within this priority map are resolved.

Subjects in experiment 2 were again presented with a sequence of simple search arrays. Each array consisted of an isoluminant grid of differently colored letters (Fig. 5). Each subject was given a target color (red or blue) at the start of the experiment and instructed to search for and respond to letter Xs only of that color, ignoring all other colors. Green letters acted as distractors, and now single letters of the nontarget color (blue or red) acted as probes. Whereas, in the previous experiment, targets (embedded within the isoluminant green distractor squares) were low in stimulus salience compared to probes, in this experiment targets and probes were matched for stimulus salience. This design was chosen to facilitate a more detailed assessment of goal-driven orienting, as targets and probes were effectively identical except in their relevance to the task. Targets and probes could be presented on separate trials, as in the first experiment, or simultaneously within a single array, so there was direct competition between them.

In the first experiment, we did not constrain eye movements at any stage of the task. One consequence was that neglect patients' initial gaze positions at the time a search array was presented tended to be to the right of midline. As a result, items on the left of the search array would have initially appeared further in the periphery of the visual field for patients than controls. To rule out any explanation for our results based on bias in resting gaze position, in the second experiment each array was preceded by a “dynamic” fixation cross (which initially filled the display but rapidly zoomed-in to a randomly selected location). This technique proved effective in guiding even a neglect patient's gaze to leftward starting locations. The search array was presented only once the subject was fixating the cross, and we analyzed the direction of the first eye movement from this initial location, thus ensuring that both patient and control performance was based on observing the same retinal locations of targets and probes.

Figure 6a shows performance of the control group on trials in which either a probe or a target was presented among green distractor letters. Despite explicit instructions to ignore items not of the target color, when a probe was presented, healthy control subjects' initial eye movements were directed toward it more frequently than expected by chance (55% of trials; t₍₇₎ = 7.5; p < 0.001) (Fig. 6a, dashed line), indicating involuntary orienting to the singleton probe. However, this frequency of orienting was significantly less than that observed when the lone item was of the target color (71%; t₍₇₎ = 4.1; p = 0.004) (Fig. 6a, solid line).

Figure 6.

Orienting to separately and simultaneously presented targets and probes. a, Proportion of first saccades directed toward the target letter on target-only trials (solid line) and toward the probe letter on probe-only trials (dashed line) by healthy controls, as a function of the horizontal position of the target/probe. b, Proportion of first saccades directed toward the target letter (solid line) and probe letter (dashed line) by controls on target-plus-probe trials, as a function of the horizontal locations of target (T) and probe (P) to left or right of subject midline. Error bars indicate SEM. c, d, Corresponding results for patient D.C., a neglect patient with a posterior parietal lesion, plotted as in a and b. Note that both target (solid line) and probe responses (dashed line) display a gradient of impairment from normal performance at far right (compare c, a), falling to chance performance at far left. Error bars indicate SE of the binomial probability estimate. e, f, Corresponding results for patient J.K., a frontal neglect patient without parietal damage. Note that both target and probe responses display a horizontal gradient of impairment as in parietal neglect. Additionally, orienting to ipsilesional (rightward) probes in the probe-only condition is enhanced in comparison to controls (compare dashed lines in e, a).

As target and probe items were matched for visual salience (no significant effect of color; red vs blue, t₍₇₎ = 0.84, p = 0.41) this additional probability of orienting to targets must correspond to a purely goal-driven component of attention, resulting from the relevance of the target color to the task goal.

Analysis of performance of a parietal neglect patient (D.C.) on this task confirmed the presence of impairments in both goal- and stimulus-driven components of orienting. The frequency of stimulus-driven eye movements to the task-irrelevant probes exhibited a gradient of deficit from approximately normal performance at the far right of the display (36% of trials, compared to 41% for controls) (Fig. 6c, dashed line) diminishing to chance levels at the far left (20%; χ²₍₁₎ = 0.43; p = 0.49). At the far right of the display, eye movements to targets were significantly more frequent than to probes (79% vs 36%; χ²₍₁₎ = 16.2; p < 0.001) (Fig. 6c, solid line), as observed for healthy individuals. However, this goal-driven advantage for targets over probes also exhibited a horizontal gradient of deficit, falling to nonsignificance at the far left of the display (26% vs 20%; χ²₍₁₎ = 0.52; p = 0.47).

These results are fully consistent with our findings in the first experiment and, because the initial fixation on each array was controlled, cannot be accounted for by biases in initial gaze position. To further examine the relationship between goal- and stimulus-driven components of attention, a proportion of search arrays contained both a target letter and a probe letter (supplemental Fig. 3, top, available at www.jneurosci.org as supplemental material). As both items attracted attention when presented separately, we hypothesized that presenting them simultaneously would create a conflict that would need to be resolved by competition within the priority map. As targets had the same visual salience as probes but were also goal relevant, their representation in a combined priority map should be stronger, so the competition for attention should be resolved in their favor (supplemental Fig. 3, middle, available at www.jneurosci.org as supplemental material). This was indeed the behavior demonstrated by control subjects on these trials, who oriented gaze toward the target with a similar frequency as when no probe was present (66%) (Fig. 6b, solid line), while the frequency of eye movements toward the probe fell to chance (11%) (Fig. 6b, dashed line).

In our neglect patient with parietal damage, the outcome of the competition depended on the relative horizontal positions of target and probe. The results matched the predictions of a damaged priority map in which items in contralesional space are underrepresented (supplemental Fig. 3, bottom, available at www.jneurosci.org as supplemental material). When the target was presented in the intact right hemispace (Fig. 6d, right two columns), behavior was similar to controls: eye movements were directed toward the target with similar frequencies as in the target-only condition (76%), and toward the probe no more often than chance (7%). These results are consistent with a preserved representation of the priority of the ipsilesional target (supplemental Fig. 3a,b, available at www.jneurosci.org as supplemental material).

However, when both target and probe were in the neglected left hemispace (Fig. 6d, second column), orienting to targets decreased significantly (54%; χ²₍₁₎ = 4.4; p = 0.036), although nonetheless remaining more frequent than orienting to probes (25%; χ²₍₁₎ = 4.3; p = 0.039). This result is consistent with a simultaneous devaluing of the priority of both target and probe relative to distractors in the intact hemispace (supplemental Fig. 3c, available at www.jneurosci.org as supplemental material).

Finally, when the target was in the neglected left hemispace and the probe in the intact right hemispace, the normal pattern was reversed and initial eye movements were more frequent toward the probe than the target (46% vs 24%; χ²₍₁₎ = 4.8; p = 0.029) (Fig. 6d, first column). In this situation, the priority of the target may be devalued sufficiently for the competition for attention to be resolved in favor of the probe (supplemental Fig. 3d, available at www.jneurosci.org as supplemental material).

These findings go beyond the predictions of a conventional extinction account (e.g., Mattingley et al., 1997; Driver and Mattingley, 1998; Wardak et al., 2002), in which the simultaneous presentation of a rightward stimulus extinguishes awareness of a stimulus on the left. Horizontal locations alone were insufficient to determine which item would be fixated; instead, the direction of the initial eye movement depended on an interaction between stimulus location and attentional priority. Hence, a high-priority item (target) on the left demonstrated a weaker claim on attention than a lower-priority item (probe) on the right (Fig. 6d, first column). But the advantage for the rightward item was far smaller than in the reverse case, when the high-priority item was on the right and low priority on the left [Fig. 6d, compare first column (46% vs 24%), last column (76% vs 7%)].

Neglect after a frontal lesion

Although the neglect syndrome is typically associated with parietal damage, it is on rare occasions observed as a consequence of a purely frontal lesion (Heilman and Valenstein, 1972; Vallar and Perani, 1986; Husain and Kennard, 1996). The results presented so far suggest that parietal neglect reflects equal deficits in representing both goal- and stimulus-based priority. To investigate whether these two types of attentional selection are differentially affected by frontal lesions, we examined performance of an additional patient (J.K.), who developed left neglect as the result of a stroke affecting the right frontal lobe, with no parietal involvement (lesion plotted in supplemental Fig. 4, available at www.jneurosci.org as supplemental material).

Figure 6, e and f, shows performance of this frontal neglect patient on experiment 2. Like the patient with parietal damage, J.K. demonstrated a gradient of increasing deficit from right to left in orienting to task-relevant targets (Fig. 6e, solid line) (χ²₍₁₎ = 7.8; p = 0.005), falling from normal performance at the far right (70%) to chance at the far left (26%). A similar gradient of deficit was also observed for orienting to the task-irrelevant probes (Fig. 6e, dashed line) (χ²₍₁₎ = 5.5; p = 0.019). These results indicate that frontal damage, like parietal, causes deficits in both goal- and stimulus-driven orienting. However, unlike both healthy individuals and parietal patient D.C., the frontal patient fixated probes with the same frequency as targets, even when presented in the ipsilesional (nonneglected) hemispace (χ²₍₁₎ < 0.9; p > 0.36).

Figure 6f shows performance of the frontal patient on “competition” trials in which both a target and a probe were presented. As for the parietal patient (Fig. 6d), the outcome of these trials depended on the relative horizontal positions of target and probe, again consistent with the predictions of a damaged representation of contralesional priority. Crucially, when both items were presented in the intact hemispace (third column), this patient consistently oriented to the target (80%) rather than the probe (which was fixated no more often than chance; χ²₍₁₎ < 0.1; p = 0.84). So the absence for this patient of the usual advantage for targets over probes when presented individually (Fig. 6e) is not attributable to a failure to distinguish the two colors or understand the demands of the task. Rather, it implies a separate deficit in suppressing reflexive eye movements to salient but task-irrelevant visual stimuli that may be a specific consequence of this patient's frontal lesion. The patient found it difficult to ignore probes when presented alone, but could successfully select the target over the probe when both were presented together.

Manipulating salience to counteract the lateral bias in neglect

The results of the first two experiments strongly support the hypothesis that visual neglect is caused by damage to a unified representation of attentional priority. Right parietal damage may leave items in left space underrepresented within this “priority map,” with the result that leftward items are relatively unlikely to be selected as targets for a saccade. If this view is correct, it should be possible to normalize a patient's exploratory behavior by biasing the distribution of visual salience to the left, i.e., by enhancing the physical salience of leftward items to counteract their weakened representation in the priority map.

In contrast, if neglect is attributable to damage at the output stage, i.e., difficulty in executing gaze shifts to leftward items after target selection, then manipulations affecting the conspicuity of leftward items will be unable to correct the exploration deficit. To distinguish between these two hypotheses and to investigate whether neglect might be ameliorated by using these principles, we performed an additional study.

Two neglect patients with right parietal damage (K.B. and L.E.) took part in experiment 3. As in the first experiment, the patients searched an array of green squares for letter targets, but now the distribution of luminance in the display was skewed by varying degrees to left or right (see supplemental Fig. 5a–c, available at www.jneurosci.org as supplemental material) using an on-line, gaze-contingent adaptive algorithm (supplemental Fig. 6, available at www.jneurosci.org as supplemental material) (for full details, see Materials and Methods). A target letter was present on 50% of trials. On the remaining trials only distractors (green squares) were presented, so the distribution of fixations on these trials could be taken as a pure measure of visual exploration.

As expected, when (baseline) arrays of uniform luminance were presented, both patients showed a significant rightward bias in their exploration (Fig. 7c) (mean horizontal fixation position, patient K.B., 11.0° right of midline; patient L.E., 5.3° right of midline; t₍₁₅₉₎ > 14.3; p < 0.001). However, by biasing the spatial distribution of distractor luminance to right or left, we were able to alter the pattern of exploratory eye movements (Fig. 7a,b). Our adaptive algorithm was programmed to search for a luminance distribution that would cancel out each patient's rightward exploratory bias, such that fixations were distributed symmetrically about the midline. The algorithm converged successfully for each patient (K.B., γ₀ = −2.6; SE, 0.2; L.E., γ₀ = −1.3; SE, 0.3; for details, see Materials and Methods). The obtained luminance distributions are shown in the insets of Figure 7, a and b, and the corresponding spatial distributions of exploratory fixations are shown in Figure 7d.

Figure 7.

Counteracting the ipsilesional bias in neglect. a, b, Mean horizontal location of exploratory fixations on null trials as a function of bias of the background luminance (γ), for patient K.B. (a) and patient L.E. (b). Red lines represent the cumulative Gaussian functions that best fit each patient's data; the arrows indicate the luminance bias (γ₀) that minimizes the mean deviation of each patient's exploratory fixations from the center of the display. The corresponding search array is shown in the inset. c–e, Distribution of exploratory fixations on null trials for patient K.B. (left) and patient L.E. (right), when display luminance is uniform (γ = 0; c), luminance is adjusted to compensate for each patient's rightward bias (γ = γ₀; d), and luminance is biased further to the left (γ < γ₀; e).

When the luminance distribution was skewed even further to the left, we found that significant leftward exploratory biases were now induced in both patients, such that they preferentially explored the contralesional (previously neglected) hemispace rather than the ipsilesional “intact” space (Fig. 7e) (K.B., t₍₁₄₉₎ = 3.5, p < 0.001; L.E., t₍₈₈₎ = 8.9, p < 0.001).

Because of the way we parameterized the space of possible luminance distributions (see Materials and Methods), the “optimal” display identified for each patient by the adaptive algorithm fell in a range (γ < −1) where luminance was exponentially related to horizontal position, and some of the rightmost distractors may not have been visible. However, significant corrective effects on exploratory bias were observed with much less skewed luminance distributions, e.g., displays defined by γ = −0.75 corrected 89% of the rightward bias for patient L.E. (t₍₈₉₎ = 2.43; p = 0.017) and 45% for patient K.D. (t₍₈₀₎ = 2.85; p = 0.006). All distractors were visible in these displays: the minimum distractor luminance was 2.2 cd m⁻² (6.6× brighter than the background); for comparison, patients in this experiment successfully fixated and responded to (ipsilesional) targets with luminance as low as 0.5 cd m⁻².

We wanted to know whether the manipulation of stimulus salience could compensate for neglect patients' impairment in goal-driven orienting to left hemispace targets, and so improve their ability to perform the search task. An analysis of the target-present trials showed that, compared to the uniform array, skewing the display luminance to the left resulted in a significant increase in successful fixation of leftward targets: from 20 to 65% for patient K.B. (χ²₍₁₎ = 23.6; p < 0.001), and from 15 to 32% for patient L.E. (χ²₍₁₎ = 4.1; p = 0.042). However, because the luminance of right hemispace targets was correspondingly decreased, right targets were more difficult to detect leading to no overall improvement in search performance.

We therefore tested one patient (K.B.) in an additional condition in which target luminance was held constant, and only the luminance of distractors was manipulated (supplemental Fig. 5c, available at www.jneurosci.org as supplemental material). The luminance distribution that normalized exploration now also improved overall performance on the search task by 30%, significantly increasing the number of fixations on left hemispace targets (from 20 to 55%; χ²₍₁₎ = 10.5; p = 0.001) while leaving rightward target fixations unaffected (82 vs 78%; χ²₍₁₎ = 0.3; p = 0.58). Thus, this simple manipulation was sufficient to produce overall improvements in visual search performance.

Discussion

In this study we investigated the visual orienting deficits associated with spatial neglect after parietal lesions. We observed impairments in both goal-directed and stimulus-driven components of attention, and mapped out the two-dimensional spatial distribution of the deficit in each case (experiment 1). Patients' impairments in stimulus-driven orienting manifested as a reduction in the involuntary capture of gaze by visually salient items in contralesional space. This deficit proved indistinguishable, in terms of both magnitude and spatial distribution, for three very different kinds of stimulus salience (defined by contrasts in luminance, onset, or orientation). This finding is consistent with lateralized damage to a single representation of visual salience, that integrates information about the conspicuousness of a stimulus across multiple visual dimensions (Koch and Ullman, 1985; Pouget and Driver, 2000; Balan and Gottlieb, 2006; Töllner et al., 2010).

Healthy orienting of attention, as well as being driven by the bottom-up salience in a scene, can independently be guided top-down by our current goals and intentions. Thus, a healthy individual can rapidly direct attention to a target object based on a feature that uniquely identifies it (e.g., a particular color) even when other objects in the scene have equal or greater visual salience (Duncan, 1989; Smallman and Boynton, 1990; Wolfe et al., 1990). Correspondingly, in monkeys, neural activity in multiple regions of the attentional network is independently correlated with both the visual salience and goal-relevance of a stimulus (Gottlieb et al., 1998; Balan and Gottlieb, 2006; Buschman and Miller, 2007). However, the causal role played by different brain areas in selecting targets for attention has not been established. To make such inferences, lesion or inactivation studies are necessary. One proposal that has recently come to the fore (Buschman and Miller, 2007) suggests separate roles for frontal areas in signaling the goal-relevance of visual objects, and posterior parietal areas in representing their stimulus salience. However, this hypothesis is also supported only by correlations between neural activity and visual input.

If it were correct, we would expect parietal patients' ability to detect and orient to task-relevant targets—which in healthy individuals can be guided top-down by knowledge of target features—to be less impaired than their tendency to be distracted by goal-irrelevant but visually salient probes. In fact, we observed identical deficits, and indistinguishable spatial gradients, for orienting to targets as to the salient probes, implying that posterior parietal cortex is critical for both goal-directed and stimulus-driven control of attention.

The degree of ipsilesional bias varied substantially from patient to patient, reflecting heterogeneity in the severity of their lesions and in the time of testing after stroke. Importantly, however, there was a remarkably strong correlation (r² = 0.98) between the goal-directed and stimulus-driven deficits observed in each patient, making it unlikely that the impairments arise from damage to separate neural representations. Instead, this result strongly suggests that the posterior neglect syndrome involves damage to a unified representation, combining both goal- and stimulus-related signals.

Such a representation may take the form of a topographically organized “priority map,” in which spatially selective neurons integrate top-down and bottom-up inputs to signal the overall attentional priority of items within their receptive field, with attention shifting to the location with the highest activation (Koch and Ullman, 1985; Wolfe, 1994; Itti and Koch, 2001; Bisley and Goldberg, 2003; Fecteau and Munoz, 2006; Serences and Yantis, 2006). If this hypothesis is correct, left neglect after right-hemisphere damage might be associated with underrepresentation of contralesional space in the priority map, resulting in a pathological bias of attention toward right hemispace.

This proposal is supported by the results of experiment 2, in which goal-relevant and visually salient items were presented simultaneously. Healthy controls preferentially directed attention to the stimulus with the highest combined priority (the target). Neglect performance was consistent with underrepresentation of the priority of contralesional items. Specifically, when salient probes were presented in the neglected hemispace, the normal priority relationship was preserved and saccades were preferentially directed to the item with the highest combined priority. However, when the lower priority probe was presented in the intact hemispace and the higher priority target to the neglected side, the low-priority item prevailed in the competition for attention (Fig. 6a; supplemental Fig. 3, available at www.jneurosci.org as supplemental material).

Of course, this proposal does not mean that other cognitive deficits might not also contribute to the neglect syndrome, which is considered to be composed of different component deficits (Duncan et al., 1999; Karnath et al., 2002; Husain and Rorden, 2003; Milner and McIntosh, 2005; Soto et al., 2009). The precise anatomical locus of the posterior neglect syndrome is also vigorously debated (Vallar and Perani, 1986; Karnath et al., 2001; Corbetta and Shulman, 2002; Mort et al., 2003; Bartolomeo et al., 2007). It is not the goal of the current study to resolve this issue, and although lesions in the posterior neglect group were centered on the inferior parietal lobe (IPL), the relatively small number of patients and absence of highly focal lesions make this an unsuitable study to address this question directly.

Stimulus-driven orienting was investigated in the present study using task-irrelevant probe items that derived their salience “bottom up” from visual contrast with other items in the same display. This type of stimulus-driven salience has been most closely associated with monkey lateral intraparietal area (LIP), which may have its human homolog in the intraparietal sulcus (Culham and Kanwisher, 2001). Although the center of lesion overlap was in the IPL, all our parietal patients' lesions extended to this area.

LIP has been proposed to play a role in covert visual attention, memory of spatial targets, and overt shifts of gaze (Gnadt and Andersen, 1988; Mazzoni et al., 1996; Snyder et al., 1997; Gottlieb et al., 1998; Kusunoki et al., 2000; Bisley and Goldberg, 2003; Balan and Gottlieb, 2006). A priority map might be important for all three functions. Reversible inactivation of LIP has also been shown to slow goal-directed responses to contralateral targets and impair contralateral visual search (Wardak et al., 2002, 2004; Balan and Gottlieb, 2009).

In monkeys, LIP is one of several regions of the oculomotor network where activity has been identified that may correspond to a salience or priority map. In particular, activity of neurons in the frontal eye fields has also been shown to reflect both the salience and relevance of visual stimuli (Schall and Thompson, 1999; Thompson and Bichot, 2005; Wardak et al., 2006; Buschman and Miller, 2007). Perhaps analogously, whereas visual neglect in humans is most commonly a consequence of parietal lesions, it is also more rarely observed after damage to the frontal lobe (Heilman and Valenstein, 1972; Vallar and Perani, 1986; Husain and Kennard, 1996). The proposal examined in this study—that parietal and frontal areas have separate roles in representing visual salience and goal-relevance, respectively—leads to the prediction that reflexive capture of gaze by visually salient stimuli should be relatively spared in frontal neglect.

To test this, we examined performance of a patient with neglect following a purely anterior lesion (Fig. 6e,f). Contrary to the prediction, this patient showed a gradient of impairment in reflexive eye movements to salient but task-irrelevant probes that was identical to her impairment in orienting to goal-relevant targets, and which, as in parietal neglect, culminated in a complete deficit in orienting to salient items in far contralesional space. This result provides additional evidence against the hypothesis that frontal and parietal components of the attentional network can be straightforwardly dissociated on the basis of their contributions to contralateral goal- and stimulus-driven control. It is notable, however, that this frontal neglect patient was abnormally responsive to ipsilesional probes when competition for attention was low. This could reflect an additional, spatially nonspecific involvement of frontal areas in suppressing reflexive responses to task-irrelevant stimuli (Husain and Kennard, 1997).

The priority map hypothesis proposes that targets of attention are selected by competition. Therefore, shifts of attention to left hemispace should still be possible in neglect as long as the leftward input is strong enough to overcome the lesion-induced advantage for rightward locations. This can take the form of a powerful top-down input, as when highly predictive precues direct a patient's attention to left space (Ladavas et al., 1994; Smania et al., 1998; Bartolomeo et al., 2001). Here, we have shown that bottom-up visual inputs can also direct attention into “neglected” space.

Biasing the distribution of luminance to the left of the visual field allowed us to overcome, and even reverse, the usual rightward bias in neglect (experiment 3). This result provides further confirmation that neglect results from a deficit in representation rather than output: if neglect reflected an impairment in executing gaze shifts to leftward items after target selection, then manipulations affecting the conspicuousness of leftward items would be unable to correct the exploration deficit. The luminance manipulation also significantly improved patients' ability to respond to targets in the contralesional hemispace, with potential implications for the treatment of neglect. Attempts to ameliorate neglect by training patients to exert top-down control have met with only limited success (Parton et al., 2004); future investigation could be directed toward alleviating symptoms by filtering or otherwise manipulating the visual input across space.