Cortical systems mediating visual attention to both objects and spatial locations

Abstract

Natural visual scenes consist of many objects occupying a variety of spatial locations. Given that the plethora of information cannot be processed simultaneously, the multiplicity of inputs compete for representation. Using event-related functional MRI, we show that attention, the mechanism by which a subset of the input is selected, is mediated by the posterior parietal cortex (PPC). Of particular interest is that PPC activity is differentially sensitive to the object-based properties of the input, with enhanced activation for those locations bound by an attended object. Of great interest too is the ensuing modulation of activation in early cortical regions, reflected as differences in the temporal profile of the blood oxygenation level-dependent (BOLD) response for within-object versus between-object locations. These findings indicate that object-based selection results from an object-sensitive reorienting signal issued by the PPC. The dynamic circuit between the PPC and earlier sensory regions then enables observers to attend preferentially to objects of interest in complex scenes.

Efficiently representing visual information requires selecting only a fraction of the multitude of information that is available to the visual system at any one instant in time. Attentional selection is the mechanism by which the subset of incoming information is extracted from the complex sensory environment. Early models of attentional selection suggested that attention is directed to regions in space (1–4), whereas more recent models have also included object-based representations as possible candidates of attentional selection (5–10). The focus of this study is on elucidating the neural mechanisms subserving attentional selection, but with more specific emphasis on differentiating between the neural substrate of space-based and object-based attention. In particular, we trace the neural signal that gives rise to an advantage for selected space-based versus object-based information from the source of the attentional selection to its effects (modulations) in primary sensory cortex.

Although much progress has been made in understanding the behavioral mechanisms mediating attentional selection, most neurophysiological studies have focused solely on space-based attentional selection (4, 11–15), with few exceptions (10, 15–18). To examine and distinguish between the neural bases of space-based and object-based attentional selection, we used functional MRI to address two main goals. First, we investigated the source of the attentional control signal by examining the sensitivity of frontoparietal regions to space-based versus object-based shifts of attentional selection (12, 19, 20). Second, we examined the effects of both space-based and object-based attentional modulations of activation in extrastriate regions of the occipital cortex (Fig. 1). Specifically, we asked whether (i) parietal cortex issues a brief transient signal after an instruction to shift one’s spatial attention (12); (ii) this reorienting signal is differentially sensitive to object-based properties of the attended items (i.e., giving rise to the object-based advantage); and (iii) the effects of reorienting attention to a within-object versus a between-object location results in distinctive cortical activation patterns in earlier regions of visual cortex.

Fig. 1.

Behavioral task. Participants were asked to fixate on the central cross throughout each run in the scanner. Before the start of the functional run, a cue was presented instructing the subject which color patch to attend to first. After the subjects pressed the button to indicate readiness, the scan was started. A 10-s rectangle fixation display was followed by the onset of color patches. Color patches changed color synchronously every 250 ms. The colors were chosen at random from four different colors (gray, olive, purple, and silver) with the restriction that no identical color was present in two locations simultaneously and that there were no immediate repeats of the same color. Subjects detected three targets as red, blue, and green (by pressing corresponding buttons on the custom-made response glove) embedded among multicolored distractors. Each target color signaled to the subject where to orient next: red, hold attention within the same spatial location; blue, shift to a location within the same object; green, shift to a location in the neighboring object. (The blue circle denotes attentional locus.) This assignment of cognitive operation to color was chosen randomly for each subject for each session. Depending on the orientation of the rectangles, the between-object and within-object shifts were either horizontal or vertical shifts of attention. For example, as depicted within the figure, the first target was blue in the upper left location; therefore, subjects had to shift their attention within the same object to a bottom left location. After 4 s, on average, a green target appeared in the bottom left location; thus, subjects shifted their attention to a location in the neighboring object, lower right location. Then a red target appeared in the lower right location, indicating to the subject that attention has to be maintained within this location; thus, the next target will appear within the lower right patch. Participants practiced the task before scanning (outside the scanner). During practice, each subject performed a minimum of five experimental runs and continued until their target detection accuracy reached 90% or better.

We observed that blood oxygenation level-dependent (BOLD) activity in the posterior parietal cortex was enhanced after instructions to shift versus to hold attention (i.e., spatial attention), replicating existing findings (12, 16, 21, 22) and verifying that our paradigm successfully engages neural mechanisms of attentional shifts. However, the first important finding was that the space-based versus object-based attentional shifts are mediated by separable neural substrates: specifically, enhanced activity for within-object over between-object shifts of attention was observed in the left posterior parietal cortex, suggesting that this cortical region expressly subserves object-based attentional selection. These data are the first to suggest that the attentional control signal issued by the parietal cortex is object-sensitive and differentiates between attention redirected to a within-object versus a between-object location. The second major result was that the object-sensitive shifts of attention were accompanied by modulations within the extrastriate regions of the occipital cortex. These neuroimaging results elucidate the neural mechanism underlying object-based attentional selection described in many behavioral studies and reveal a network in which the consequences of attentional shifts triggered in parietal cortex are manifest in earlier, sensory regions of the visual system. It is this dynamic circuit between the parietal and earlier visual regions that enables observers to focus preferentially on objects of interest that appear in complex visual scenes.

Results

Behavioral Results.

No behavioral differences were observed among the three experimental conditions (see Table 1). This finding is not surprising given that participants were trained to asymptotic levels before the scan and that exposure duration and interval for response were long. The average accuracy of performance during the scan was 98%.

Table 1.

Behavioral results (in scanner)

Event type	Reaction time, ms	Accuracy, %
Hold	581	97.8
Shift within-object	603	98.1
Shift between-object	598	98.3

Data show reaction time and accuracy for identification of each event type collapsed over rectangle orientation and location of the target. Subjects were trained to perform the task and achieved target detection accuracy of 98%, with no significant differences between conditions (red, green, or blue target) or locations (all F values < 1). This equivalence across conditions is unsurprising given the temporal parameters of the target events; targets occurred, on average, every 4 s; therefore, any behavioral differences for detecting targets appearing within the same object as opposed to the different object are likely obscured by the long intertarget interval.

Shift-Related Activity.

To isolate the neural mechanism subserving attentional reorienting signals and to demarcate regions of interest (ROIs) for further investigation of object-based attention, we first compared the BOLD signal for shift versus hold events by contrasting the magnitude of their regression weights. Whole-brain random-effects analysis yielded two candidate cortical regions exhibiting greater BOLD response after shift than hold events: the right and the left precuneus/superior parietal lobule (SPL) (Table 2 and Fig. 2a). β-Coefficients were extracted from these two ROIs (Fig. 2b) for each subject and subjected to an ANOVA with the following factors: condition (averaged shift versus hold events), time (time points 3–7), and rectangle orientation (horizontal and vertical). Unsurprisingly, given that this is the designated contrast, significant increases of activity after shift events (average of shift-within and shift-between) as compared to hold events were observed in both ROIs [left lateral precuneus/SPL: F(1,2) = 15.38, P < 0.01; right precuneus/SPL: F(1,2) = 13.67, P < 0.01]. In addition, we observed a main effect of time [left lateral precuneus/SPL: F(4,8) = 22.34, P < 0.01; right precuneus/SPL: F(4,8) = 19.68, P < 0.01] but no significant effect of rectangle orientation or interactions (all F values < 1). Two other regions also revealed greater BOLD signal for shifts versus hold events (Table 2). Neither region, however, shows a clear pattern of activation as a function of time and may either play a role in the frontoparietal attentional network (23) or be epiphenomenally activated (24).

Table 2.

Cortical regions exhibiting shift-related activity

Anatomical region	Brodmann’s area (x, y, z)	No. of voxels	t(5)
Shift > Hold
Right precuneus/SPL	7 (8, −65, 55)	192	3.49
Right precentral gyrus	6 (27, −12, 53)	236	4.37
Right middle frontal gyrus	6 (15, 5, 58)	65	4.27
Left precuneus/SPL	7 (−13, −65, 56)	1,231	3.87
Hold > Shift
Left inferior parietal lobule	40 (−39, −34, 46)	30	4.76
Left middle frontal gyrus	10 (−37, 42, 13)	303	4.52

Coordinates (x, y, z) are reported in Talairach space (33). t values are computed across all voxels within the activated cluster. Two regions showed greater BOLD activation for sustained attention (i.e., hold events; see Table 1). Investigating sustained attentional effects was not the goal of this work, and thus we refrain from further discussion of these activations.

Fig. 2.

Selective activation of posterior parietal cortex following shifts of attention. (a) Group random-effects analysis: regions of parietal cortex exhibiting selective increase in BOLD response after shifts of attention. (b) The mean β-coefficients extracted from the cluster of activated voxels in left and right parietal cortices. (c) Extracted β-coefficients from two parietal ROIs (defined by shift versus hold contrast) when objects were oriented horizontally and vertically. Each panel represents averaged β-coefficients for all subjects extracted from the two foci within the parietal lobe. Planned comparisons for within-object versus between-object for each orientation reveal the same pattern within each parietal ROI, irrespective of orientation. (d) β-Coefficient difference calculated by subtracting between-object shift from the within-object shift for each subject from within the left parietal ROI (averaging over the peak, time points 4–7, for between-object and within-object shifts).

Object-Based Shift-Related Activity.

To evaluate more specifically whether this parietally mediated reorienting signal is sensitive to object-based properties of the attended items we compared the BOLD activity in the two selected ROIs for responses to within-object versus between-object shifts. The activation profiles for these two different shift events, extracted from the ROIs, were submitted to repeated-measures ANOVA with event type (shift within object and shift between objects), time (time points 3–7), rectangle orientation (vertical and horizontal), and region (left SPL and right SPL). Significantly enhanced BOLD signal for the within-object versus between-object events was observed in the left parietal ROI [F(1,5) = 18.78, P < 0.01] but not in the right parietal ROI (F < 1). A main effect of time was seen in both ROIs: left precuneus/SPL [F(4,20) = 46.35, P < 0.01] and right precuneus/SPL [F(4,20) = 18.45, P < 0.01]. Importantly, we observed a significant interaction of event type with time for left precuneus/SPL region only [F(4,20) = 4.31, P < 0.05], with an earlier rise of the BOLD signal for the within-object versus between-object shift. Furthermore, we observed a three-way interaction of region (right and left SPL) with event type (within and between) with time (time points 3–7) [F(4,20) = 2.97, P < 0.05]. No other main effects or interactions were significant (all F values ≤ 1). β-Coefficients for each object orientation are plotted in Fig. 2c. It is important to note that the differential enhancement for the within-object shift in the left parietal region was remarkably robust and is evident even in the individual subject data (with the exception of one subject in one rectangle orientation), irrespective of rectangle orientation (see Fig. 2d).

Location-Specific Attentional Modulation.

Our last investigation examined the differentiable cortical effects of reorienting one’s attention to a within-object location versus a between-object location. In particular, we explored whether shifting attention to a new location within the already attended (i.e., the same) object results in differential enhancement of early visual cortex compared to shifting to a spatially equidistant new location in a different object and whether both of these shifts differ from instances when attention is maintained on the current location. We first localized in the retinotopic cortex the four possible positions (using checkerboard patches; see Methods) to which attention could be shifted. Event-related time courses were then extracted from each patch location within retinotopic regions V1–V4, when it was the currently attended position (hold trials), the position to which attention is about to be shifted from within the same object (“shift-to from within”), and the position to which attention is about to be shifted from a different object (“shift-to from between”). This analysis enabled us to examine the activation pattern for the identical region when it serves as the target of the three different attentional conditions. We then collapsed over orientation of the rectangles (no significant effect of orientation was observed; F < 1) and over the four patch locations (Fig. 3b). Note that, because only ventral V4 was mapped, only top right and top left locations were averaged for this retinotopic region.

Fig. 3.

Event-related time courses averaged over all subjects, extracted from the retinotopic regions V1–V4 as defined by an external localizer and retinotopic mapping. (a) Activation within the right hemisphere after a contrast of “top left greater than all other locations” for one subject. Retinotopic borders are represented by the blue lines. (b) A schematic representation of the procedure for assigning events (i.e., hold, shift-to from within the object, and shift-to from between the object) for one of the patch locations. (c) Mean event-related BOLD time courses from V1, V2, V3, and ventral V4. Error bars reflect SEM.

The patterns of hold-related and shift-related activity are roughly similar for each of the four retinotopic regions V1–V4, revealing an early rise after a hold-attention event and roughly the same later, somewhat extended peak for between-object and within-object shifts. Interestingly, the degree of attentional modulation for hold, shift-to from within object, and shift-to from between object events increased gradually from V1 to V4 (Fig. 3c), suggesting that these modulations are scaled by the increasing size of the receptive fields of neurons in these extrastriate areas (25, 26).

In all regions, the BOLD signal is increased early after a hold event, reaching its peak at ≈4 s after target onset with sustained activation for several seconds thereafter. It is important to note that activity is already somewhat elevated at time point 0 (the onset of the target event) because attention is already engaged at that location by the previous target event (it is a hold trial). In contrast with this early peak for hold events, the shift-to from within object and shift-to from between object time courses reached their peak at about the same time (7 s on average) after the instruction to shift attention to that location. This delayed time-of-peak for shift events, relative to hold events, is expected for both shift types because the region of cortex from which the event-related time courses were extracted is yet to-be-attended for shift events, whereas it is already attended for the hold events. Thus, activity is delayed by the time that is needed to move one’s attention to the to-be-attended location. Although the BOLD activity for the two shift events peaks at the same time, of note is the steeper rise, as well as a shift in baseline for the shift-to from within object (Fig. 3c, blue curve) than shift-to from between object BOLD activation function (Fig. 3c, green curve).

To quantify the above effects, we derived an index that reflects the time of peak as well as the rise of the activation function to the peak for each of the three types of events. The index is computed by summing the values of the percent signal change starting at time 0 (onset of the target event) and ending at the time of peak and then dividing it by the time-of-peak point (Fig. 4a). We term this index the “velocity of the activation function” (VAF). The VAF index for each subject in each region was subjected to a four-way ANOVA with region (V1–V4), event type (hold, within-location, and between-location), patch location (top left, top right, bottom right, and bottom left), and rectangle orientation (horizontal and vertical) as within-subject factors, performed separately for V1 to ventral V4. Significant main effects for region [F(3,15) = 25.49, P < 0.01] and event type [F(2,10) = 55.16, P < 0.01], as well an interaction between them [F(6,30) = 18.45, P < 0.001], were observed. No other main effects or interactions reached significance (all F values < 1). In all four regions, planned comparisons revealed significant differences in the VAF index for hold compared with shift events {V1[F(1,5) = 65.85, P = 0.01]; V2[F(1,5) = 48.54, P < 0.01]; V3[F(1,5) = 73.25, P < 0.01]; and V4[F(1,5) = 65.51, P < 0.01]}. Of perhaps even greater interest for our current purposes, reflecting the object-based modulation, is the significantly enhanced BOLD signal for shift-to from within object compared with shift-to from between object events in all four regions of cortex {V1[F(1,5) = 10.15, P < 0.02]; V2[F(1,5) = 84.52, P < 0.01]; V3[F(1,5) = 28.65, P < 0.01]; and V4[F(1,5) = 44.52, P < 0.01]}.

Fig. 4.

Index of attentional modulation in retinotopic regions. (a) Effects, or consequences, of holding or shifting attention on retinotopic areas V1–V4. The computed VAF for each target event within each retinotopic region is shown. (b Left) The validity index is computed by averaging the VAF for both types of shift events and subtracting the value from the hold event. This manipulation allows for comparison between events that require shifts of attention and those that do not. (b Right) Object-based index is computed by subtracting VAF of the between-object shift from that of the within-object shift.

To compare the four regions directly, we computed the VAF index for the validity effect, which is well described in the behavioral literature, by comparing the VAF index for hold-related activity minus the VAF index for the average of the shift-related activity. In our case we compared targets that required a spatial shift of attention (within and between object shifts, equivalent to invalid trials) with targets that occurred within an already attended location (hold event, equivalent to valid trials). An ANOVA with region (V1–V4) as a within-subject factor (data were collapsed over location and object orientation) revealed significant differences in the validity effect [F(3,15) = 15.85, P < 0.01] (Fig. 4b, on the left). Planned comparisons indicated that differences between the VAF index of the hold activity and shift-related activity increased as one moves anterior within the visual cortex {V1&V2[F(1,5) = 95.21, P = 0.01]; V1&V3[F(1,5) = 82.96, P < 0.01]; V1&V4[F(1,5) = 84.56, P < 0.01]; V2&V3[not significant, F < 1]; V2&V4[F(1,5) = 46.25, P < 0.02]; and V3&V4 [F(1,5) = 28.56, P < 0.02]}. In addition, we computed an object-shift index, which reflects the differences between within-location and between-location event types (Fig. 4b, on the right) by subtracting the VAF index for the between-location from the within-location activations. ANOVA with region (V1–V4) as a factor (data were collapsed over location and object orientation) uncovered a significant difference in the object-shift index [F(3,15) = 78.56, P < 0.01]. Planned comparisons between each region indicated that differences between the within-object and between-object shifts increased as one moves more anterior within the visual cortex, such that activity for the within-object shift rises to the peak earlier than the between-object shift {V1&V2[F(1,5) = 65.45, P < 0.01]; V1&V3[F(1,5) = 71.25, P < 0.01]; V1&V4[F(1,5) = 67.84, P < 0.01]; V2&V3[F(1,5) = 10.95, P < 0.07]; V2&V[F(1,5) = 13.27, P < 0.04]; and V3&V4[F(1,5) = 34.56, P = 0.02]}.

Discussion

To investigate the source of attentional selection in visual scenes and specifically to differentiate the substrate for space-based and object-based attention, we compared events that required a shift of attention (within-object versus between-object) with events in which attention was held at the current locus. Shifts (versus hold) of attention selectively activated two loci within the posterior parietal cortex: left SPL/precuneus and right SPL/precuneus. We then extracted the time course of the BOLD activity from these loci to reveal two important properties of the shift function. The first property is that, in both loci, shift events elicited greater BOLD activation than hold events, suggesting that posterior parietal cortex, bilaterally, issues a transient switch signal that initiates shifts of attention to a new spatial location. The presence of a signal that triggers attentional shifts is compatible with behavioral theories of attentional orienting that cite differences between validly (targets appearing within the same spatial location as the cue) and invalidly cued targets (targets appearing in a location other than the cue) as evidence for space-based attentional selection (8, 12, 16, 21, 22, 27, 28). These findings suggest that the posterior parietal cortex issues a transient signal to reorient one’s current attentional locus.

The second and more major finding reveals that the pattern of activity within the left parietal region is differentially sensitive to the exact nature of the attentional shift. Specifically, we see that there is a selective increase in BOLD signal for a within-object compared to between-object shift. This increased rise in BOLD activity after a signal to reorient within-object might be due to the mechanism of attentional prioritization (i.e., locations within an already attended object are assigned higher priority) or from sensory enhancement (i.e., locations within an already attended automatically share in perceptual benefit) (10). Here we demonstrate that the reorienting signal issued by the posterior parietal lobe is sensitive to the object-based properties of the to-be-attended location. This enhanced within-object shift signal consequently results in greater modulation of sensory activity in cortical regions corresponding to the to-be-attended location, as discussed in the next two paragraphs. Interestingly, the within-object shift advantage is not present in the right posterior parietal cortex, suggesting that only the left parietal region is distinctly sensitive to object-based properties of the to-be-attended locations. This result is compatible with the finding that patients with damage to the left, but not right, parietal cortex do not exhibit the typical costs associated with detecting a target within the previously unattended object (7, 18). This pattern is easily accommodated by our results: in individuals with damage to the left parietal cortex, attentional reorienting is solely mediated by the signal that is elicited by the right posterior parietal cortex. Given that no object-sensitive BOLD differentiation is observed in this latter region, the within-object benefit is not accrued by these patients.

Having identified and uncovered the neural mechanism associated with space-based and object-based attentional shift signals, we then investigated the consequences of reorienting one’s attention on the early parts of the visual system (V1–V4). Based on the well documented behavioral object-based advantage and extant neuroimaging studies (10, 16), we expected to observe object-based attentional modulation effects in the extrastriate visual cortex.

Activity in retinotopic areas V1–V4 was modulated in a sustained fashion after a hold event target (validity index in Fig. 4b), consistent with the top-down effects of parietal attentional signals (12, 22, 29). Of greater interest is that the VAF index in these retinotopic regions, corresponding to previously unattended locations, was enhanced when the to-be-attended locations were bound to the currently attended object as compared with locations occupied by a different object. This object-based activation was observed in locations that were uncontaminated by target presentation, i.e., activity was measured after a target in another spatial location and before the target actually occurred at the to-be-attended spatial location (object index in Fig. 4b). This result is the first to suggest that the object-based attentional advantage, observed in behavioral studies, might emerge from a baseline shift in the BOLD response to within-object items. This result corroborates and extends previous demonstrations of attentional modulation within early sensory cortex after an object-based shift of attention (10, 16, 17) and confirms that our task effectively recruited object-selective attention.

The important contribution of our findings is that the posterior parietal cortex is sensitive to the object-based properties of the to-be-attended locations, as evidenced by a stronger attentional reorienting signal issued for shifts of attention within an already attended object relative to an unattended object. Moreover, this object-based signal results in distinctive attentional modulation of activity in the within-object compared to between-object locations in earlier regions of visual cortex. Given that activity in early visual cortex is known to correlate with behavioral performance in detection tasks (4, 10, 30) and that the VAF index for hold events (or valid trials) is much stronger than for shift events (i.e., invalid trials), these findings uncover the possible neural basis for the faster response time to stimuli appearing at cued (i.e., valid trials) than at the uncued (i.e., invalid trials) locations. Similarly, the larger object index for within-location than for between-location events potentially provides a neural basis for the performance advantage for stimuli appearing in the invalidly cued same-object locations compared to invalidly cued different-object locations (7–9, 31). The difference in magnitude between the validity index and the object index is consistent with the reaction time differences of the validity (100–200 ms) and of the object-based effect (10–30 ms).

The findings from this study uncover the neural mechanism underlying object-specific shifts of attention and implicate the left parietal cortex in this process. These results provide a clear depiction of a mechanism that might aid in parsing and organizing complex visual information based on spatial locations and the objects that occupy those spatial locations. It is by virtue of such a mechanism that information that is of preferential interest to the observer can be rapidly extracted and represented.

Methods

Participants.

Six neurologically healthy adults participated in two functional MRI sessions in which extensive data collection occurred (akin to the psychophysical approach where a few observers are tested intensively). Participants gave written consent to a protocol that was approved by the Institutional Review Board of Carnegie Mellon University.

Task.

The behavioral task, depicted in Fig. 1, is a variant of a two-rectangle method (7). Four color patches (1.20° × 1.4°) appeared at the ends of two black rectangles (8° × 2°), oriented either vertically (session 1) or horizontally (session 2) on a white background and centered 4.1° to the left and right, or above and below fixation. Participants fixated on a foveal plus sign subtending 0.74° × 0.74° and were instructed to maintain fixation throughout the length of each experimental run (see Fig. 1 for details).

The order of targets was random with two constraints: (i) no more than two hold events occurred in succession, and (ii) targets appeared only in the attended location. Targets within the attended stream were separated by a temporal interval that was pseudorandomly jittered between 3 and 5 s, with an average intertarget interval of 4 s. Such temporal jittering allows for the extraction of individual event-related BOLD signal time courses after the target events (32). Each subject performed 10 runs (per session), each of which was 3 min 48 s in duration and included 16 occurrences of each of the three target types: hold attention with the same color patch, shift to a within-object location, and shift to a between-object location.

Each subject participated in two scanning sessions (one for horizontal object and the other for vertical) on two separate days. Participants were instructed to hold attention on the currently attended stream even if they thought they had missed a target. Only target events that were detected were analyzed. Eye movement data were collected for three of six participants in the scanner to ensure that fixation was maintained. In cases when fixation was broken (and coincided with target onset), the run was repeated.

Functional MRI Data Analysis.

MRI scanning was carried out with an Allegra 3-T scanner (Siemens). High-resolution anatomical images (1-mm³ resolution) were acquired by using an MP-RAGE T1-weighted sequence [echo time (TE) = 3.49, flip angle = 8°, matrix 256 × 256, field of view = 256 × 256 mm², slice thickness = 1 mm, number of slices = 192]. Whole-brain echoplanar functional images were acquired in 29 transverse slices [repetition time (TR) = 1,500 ms, TE = 25 ms, flip angle = 60°, matrix 64 × 64, field of view = 200 mm, slice thickness = 3 mm, 1-mm gap].

brainvoyager software (Brain Innovation, Maastricht, The Netherlands) was used for the analyses. Images from each functional run were slice–time- and motion-corrected (within as well as between the runs) and then filtered (3 and 50 cycles per run, respectively) to remove low-frequency and high-frequency noise in the functional time series. Images for each subject were coregistered with the 3D structural data set and transformed into Talairach space (33).

The hemodynamic response function for each event type (hold, within-object shift, and between-object shift) was estimated by using a general linear model with separate regressors to estimate the BOLD response at the time of event onset and at each of the next 13 time points, 0–18 s after stimulus (34).

Statistical maps were computed by comparing the mean fit coefficients across 4.5–10.5 s after stimulus for each condition. The single-voxel threshold for all statistical maps was set at t(5) = 6.98 and P < 0.02. A minimum cluster size of four voxels was adopted to correct for multiple comparisons, yielding a mapwise false-positive probability of P < 0.01 (35).

To investigate the source of the signals underlying the different attentional processes, we initially defined ROIs by contrasting the main effect of attentional shifts (i.e., within-object shift and between-object shift) versus hold events using a random-effects analysis. This contrast (whole-brain analysis) uncovers areas of cortex that are selectively activated when attention is shifted from its current locus. β-Coefficients were then extracted from each ROI and plotted against time after the onset of the event (time 0). Each β weight time course is the mean of ≈320 events and is time-locked to the target event. A two-way repeated-measures ANOVA was then conducted with event type (within, between, and hold events) and time (time points 3–7). Note that the definition of these ROIs is agnostic to potential differences between responses to within-object versus between-object shift. We then contrasted two types of shift events in an ANOVA with event type (shift within-object and shift between-objects) and time (time points 3–7) with data derived from these ROIs to explore differences between the two types of attentional shifts.

To compare the effects or consequences of hold versus shift attentional operations, we examined the pattern of activity within the visual sensory areas. First, retinotopic mapping was carried out for each subject in a separate scan by using established procedures (36) to define borders between early visual areas (V1 to ventral V4). Then ROIs within these four areas were mapped by an external localizer consisting of flickering checkerboards (4 Hz) superimposed on the locations occupied by each of the four color patches (Fig. 3a). The cortical surface of each subject was reconstructed from the high-quality 3D data set by first segmenting the hemispheres, segmenting white from gray matter, inflating the cortical sheet, and then cutting the inflated brain along five segments including the calcarine sulcus. After projecting each of these ROIs for each subject onto their corresponding inflated cortex sheet, the percent signal change was extracted from ROIs within each retinotopic boundary, coded for an attention shift to this region of space after a within-object or a between-object shift, and after a hold event. For example, if the “shift within-object” target appeared in the top-left location, this event was coded as a “bottom-left within” event (Fig. 3b). After this procedure, event-related averages were extracted for each subject from each patch location in each retinotopic region (V1–V4) for the two rectangle orientations. The baseline for each of the event types was the mean BOLD signal for the 6 s preceding that event. Given the fact that, in hold conditions, the baseline activity within that region of cortex is already elevated because of attentional modulation from the previous target, whereas in the within-location and between-location conditions, attentional modulation was present at some other location (i.e., location of the previous target), we expected that this difference in baseline might obscure any differences between these conditions within the extrastriate cortex. Therefore, we used event-related averaging extraction, rather than β weight extraction, for this analysis because of the sensitivity of β weights to differences in baseline activity within a region.

Abbreviations

SPL

superior parietal lobule

ROI

region of interest

BOLD

blood oxygenation level-dependent

VAF

velocity of the activation function.