Attentional modulation of perceptual grouping in human visual cortex: ERP studies

Abstract

We sought to identify the neural substrates underlying perceptual grouping, and examined whether the grouping‐related neural activities are modulated by task relevance and attention by recording high‐density event‐related potentials (ERPs). Participants were presented with stimulus arrays, in which local elements were either evenly distributed or grouped into rows or columns by proximity or similarity, and had to discriminate orientations of the perceptual groups, or to identify the colors of dots around stimulus displays or of the fixation cross. We found that proximity grouping was indexed by a positive activity over the medial occipital cortex with a peak latency of about 100 ms after stimulus onset (Pd100), whereas grouping by similarity of shape was reflected in a negative activity with longer latency over the occipito‐temporal areas. Dipole modeling based on a realistic boundary element head model localized the Pd100 to the right calcarine cortex. Moreover, we showed that the grouping‐related activities were weakened when stimulus arrays were of low task relevance and fell outside an attended area of field. The results suggest that human calcarine cortex is engaged in early grouping operations defined by proximity, reinforcing the previous fMRI findings. Moreover, our ERP results indicate that the neural bases underlying perceptual grouping in human visual cortex can be modulated by task relevance and attention as early as 100 ms after sensory stimulation. Hum Brain Mapp, 2005. © 2005 Wiley‐Liss, Inc.

INTRODUCTION

An important function of the human perceptual system is to organize discrete entities in the visual field into perceptual units. This grouping process can be conducted on the basis of Gestalt principles such as proximity and similarity [Wertheimer, 1923] and has been assumed to take place at an early stage of visual perception [Lamy and Tsal, 2001; Vecera and Behrmann, 2001]. Previous behavioral research has shown evidence that grouping by proximity can be perceived faster or earlier than grouping by similarity [Ben‐Av and Sagi, 1995; Han and Humphreys, 1999; Han et al., 1999a, b]. This proposition is further reinforced by event‐related potential (ERP) studies [Han et al., 2001, 2002], which have demonstrated that proximity grouping is indexed by a positive activity over the medial occipital cortex with a peak latency of 100 ms after stimulus onset (Pd100), whereas grouping by similarity of shape or color is indexed by a long‐latency occipito‐temporal negativity.

Recent neuroimaging studies have tried to localize neural activities underlying the process of perceptual grouping in human brains by recording hemodynamic responses using functional magnetic resonance imaging (fMRI) when participants make judgments to perceptual groups. For example, Altmann et al. [2003] found that, relative to patterns of randomly oriented local elements, global shapes formed through grouping by collinearity or through binocular disparity generate activations in V1, V2, and the lateral occipital complex (LOC). Perceptual objects formed by grouping by motion also activated the LOC [Ferber et al., 2003]. In a recent fMRI study, we presented subjects with stimulus arrays in which local elements were either evenly spaced (a uniform stimulus) or grouped into columns or rows by proximity or by similarity of shape [Han et al., 2005]. We found that, relative to the uniform stimulus, proximity‐grouped stimuli generated stronger activation in the right calcarine cortex. In addition, we showed that the calcarine activity was weakened when the participants adopted a narrow attentional window so that the grouped elements fell outside the attentional window. This suggests that the neural activity underlying proximity grouping in the visual cortex is modulated by attention.

Although the previous ERP and fMRI studies have shown evidence that the activity of the medial occipital cortex is modulated by perceptual grouping, it is unclear whether the calcarine activation observed in the fMRI study [Han et al., 2005] is the neural generator of the grouping‐related ERP component (e.g., Pd100) recorded over the scalp [Han et al., 2001, 2002]. The first objective of the current work was to identify the neural generator of the grouping‐related ERP wave observed in our pervious work [Han et al., 2001, 2002]. We recorded high‐density ERPs to stimulus arrays consisting of local elements that were either evenly spaced (“a uniform stimulus”) or grouped into either columns or rows by proximity or similarity of shape (“grouped stimuli”). Grouping‐related neural activity was obtained by subtracting ERPs to the grouped stimuli from those to the uniform stimulus. We also obtained anatomical magnetic resonance (MR) images with high spatial resolution from the subjects who were involved in the ERP experiments. Real head models were then constructed from the MR images and used in dipole modeling to localize the sources for grouping‐related difference waves in the real head models. We were particularly interested in whether the Pd100 can be localized to the calcarine cortex.

The second objective of the current work was to investigate if the grouping‐related ERP components are modulated by task relevance and spatial attention by comparing the amplitudes of the grouping‐related activity in different tasks. Similar to our prior fMRI studies, subjects were asked to discriminate the orientations of perceptual groups in Task 1, so that features of perceptual groups in nontargets (i.e., orientations of perceptual groups in nontarget stimuli) were of high task relevance. In Task 2, subjects discriminated the colors of dots around the stimulus arrays, so that a large attentional window should be adopted. In Task 3, the task was to discriminate the color of the fixation cross. In this case, a small attentional window has to be set, so that the stimulus arrays in nontargets should fall outside the attentional window. In both Tasks 2 and 3 the features of perceptual groups in nontargets (i.e., orientations) were of low relevance to the feature (i.e., color) required for discrimination. The prior fMRI studies have shown that the grouping‐related calcarine activity can be modulated by variation of the size of an attentional window but not by task relevance. Here we examined whether the electrical neural activities show a similar pattern of attentional modulation of the grouping process and, particularly, when the attentional modulation of grouping operations takes place. The ERP results reported here reinforce our conclusions obtained from the fMRI findings and provide information about the temporal course of attentional modulation of grouping in human visual cortex.

SUBJECTS AND METHODS

Subjects

Eighteen graduate and undergraduate students from Peking University (14 males, 4 females; age range 19–25 years) participated in this study as paid volunteers. All subjects were right‐handed, had normal or corrected‐to‐normal vision, and were not colorblind. All subjects were free of any history of neurological or psychiatric problems. Informed consent was obtained according to the guidelines of Department of Psychology (Peking University).

Stimuli

White stimulus elements were presented with a personal computer running Presentation software (online at http://www.neurobehavioralsystems.com) on a black background on a 21‐inch color monitor at a viewing distance of 120 cm. A white fixation cross of 0.19 × 0.19° (width and height) was continuously visible in the center of the monitor. In Task 1, the nontarget stimuli consisted of a square lattice of gray elements (either filled circles or squares) in an 8 × 8 array, as shown in Figure 1. The uniform stimulus consisted of alternate circles and squares distributed evenly across the lattice. This arrangement prevented the local elements from grouping into rows or columns. The proximity‐grouped stimuli consisted of alternate circles and squares arranged in arrays to form separate perceptual groups (i.e., rows or columns) by adjusting the distances between two adjacent rows or columns of local elements so that the distances between two near or remote rows (or columns) were 0.20° and 0.75°, respectively. The similarity‐grouped stimuli were made by moving the circles and squares in the uniform stimulus to form rows or columns of elements with the same shape. The distance between two adjacent columns or rows was 0.42° for the uniform and similarity‐grouped stimuli. Each local shape subtended an angle of 0.38° × 0.38° and the global stimulus pattern subtended an angle of 6.0° × 6.0°. The target stimuli were the same as the nontargets except that they were only 62% of the size of nontarget stimuli. In Task 2 the stimuli were the same as those in Task 1 except for the following. The targets and nontargets were of the same size. Four dots (0.19° × 0.19°) at the corners of an imaginary square appeared along with the stimuli (illustrated in Fig. 2a). Each of the dots was 5.3° from fixation. The dots were white when accompanying nontargets but were red or green when accompanying targets. In Task 3 the stimuli were the same as in Task 1 except that the targets and nontargets were the same size and the fixation cross was either red or green in target displays (illustrated in Fig. 2b). In all tasks both the target and the nontarget displays were presented for 150 ms. The interstimulus intervals were randomized between 300–500 ms.

Figure 1.

Illustration of the stimulus arrays and procedures in Task 1. Local circles and squares were white on a black background and grouped into columns or rows based on proximity or shape similarity in the grouped stimuli, whereas the elements were evenly spaced in the uniform stimulus. Subjects identified the orientations (vertical vs. horizontal) of the perceptual groups in targets by a button press while ignoring nontargets.

Figure 2.

a: A nontarget stimulus used in Task 2, which is surrounded by four dots. The stimulus arrays in nontarget and target stimuli were of the same size. However, the dots around stimulus arrays were white for nontargets, but were red or green for target stimuli. b: A nontarget stimulus used in Task 3. The stimulus arrays in nontarget and target stimuli were of the same size. However, the fixation was white for nontargets, but was red or green for target stimuli.

Procedure

In Task 1 subjects were asked to indicate the presence of row‐ or column‐grouped target stimuli (regardless of whether proximity or similarity cues produced grouping) by pressing one of two keys with either the left or the right index finger while ignoring the nontarget stimuli. In Tasks 2 and 3 subjects discriminated the color change of the four dots around the target stimuli or the color change of the fixation cross accompanying the target stimuli by pressing one of two keys. In each task each subject completed 100 trials for practice, followed by 960 trials in four 240‐trial blocks. Targets appeared randomly on 20% of trials. The uniform stimulus, proximity‐grouped stimuli, and similarity‐grouped stimuli were each presented randomly on one‐third of the nontarget trials. The procedure of randomization guaranteed that the sequence of the stimuli was different between any two blocks of trials for every subject. Participants were instructed to maintain fixation on the central cross throughout the task while responding to targets as quickly and accurately as possible. The order of the tasks and the assignment of responding hand with the two types of targets (column vs. row in Task 1; red vs. green in Tasks 2 and 3) were counterbalanced across participants.

ERP Data Recording and Analysis

The electroencephalogram (EEG) was recorded using a 128‐channel NeuroScan system from 120 scalp electrodes labeled with numbers 1–120. Electrodes 59–71 were arranged along the midline of the skull and the others were located approximately symmetrically at the two sides of the skull. The skin resistance of each electrode was made less than 5 k°. The recording from an electrode at the right mastoid was used as reference. Eye blinks and vertical eye movement were monitored with electrodes located below the right eye. The horizontal electro‐oculogram was recorded from electrodes placed 1.5 cm lateral to the left and right external canthi. The EEG was amplified (bandpass 0.15–70 Hz) and digitized at a sampling rate of 250 Hz. The ERPs were averaged separately offline with averaging epochs beginning 200 ms before stimulus onset and continuing for 1,000 ms. Trials contaminated by eye blinks, eye movements, or muscle potentials exceeding ±50 μmv at any electrode were excluded from the average.

Grouping‐related difference waves were obtained by subtracting ERPs to uniform stimuli from ERPs to grouped (proximity or similarity) stimuli. Mean voltage of ERPs and grouping‐related difference waves were obtained (1) at 20‐ms intervals starting at 60 ms after stimulus onset and continuing until 200 ms poststimulus, and (2) at 40‐ms intervals from 200–800 ms poststimulus. Because preliminary analyses showed that most of the effects were evident over the posterior areas, we selected more electrodes over the temporal‐parietal‐occipital regions than the frontal areas for statistical analysis. Statistical analyses were conducted at each pair of electrodes over frontal (32–87, 34–89), parietal (37–92, 38–93), and occipito‐temporal (18–118, 19–99, 41–96, 42–97, 43–98, 44–72, 45–73, 6–120) regions, and at electrodes located along the midline of the skull (68, 69, 70, 71).

Reaction times and responses accuracies were subjected to repeated measures analyses of variance (ANOVAs) with Task (Tasks 1, 2, and 3) as independent variables. To assess the effect of attention on sensory processing, ERPs to nontarget stimuli were subjected to ANOVAs with the factors being Task (Tasks 1, 2, and 3), Grouping (uniform, proximity‐, or similarity‐stimuli), and Hemisphere (electrodes over the left or right hemisphere). To validate the effect of perceptual grouping in the visual cortex, in each task ERPs to uniform and grouping (proximity or similarity) nontargets were first subjected to ANOVAs with the factors being Grouping (proximity or similarity vs. uniform) and Hemisphere. Grouping‐related difference waves were then subjected to ANOVAs with Task (Tasks 1, 2, and 3) and Hemisphere as independent variables to investigate the effect of attention on perceptual grouping.

Dipole modeling was used to localize the source of the effects of attentional modulations of sensory and perceptual processing. The position of each electrode was measured using a 3Space Fastrak digitizer for each individual subject with a probe for sensing the 3‐D position of the probe tip with respect to a magnetic field source in the head support. The mean scalp location of each electrode site was estimated by averaging each electrode location across subjects and this was used for the dipole modeling and voltage topographic mapping. MR images were obtained from five subjects and were used for constructing real boundary element head models. The digitized fiducial landmarks corresponding to the electrode coordinates were coregistered with fiducial landmarks identified on whole‐head MR scan so that the locations of estimated dipoles could be related to individual brain–skull anatomy. Principle component analyses were conducted first to estimate the possible number of dipoles in a specific time window. Dipole modeling was then carried out to localize the sources for specific components of the grand averaged ERPs and grouping‐related difference waves in real head models. The dipoles were mapped onto the MR images and 3‐D brain models of individual subjects to estimate locations of sources with respect to brain anatomy. The 3‐D coordinates of each dipole in an individual subject's realistic‐head boundary‐element model were transformed to the coordinates of the Talairach and Tournoux [1998] atlas by marking the anterior and posterior commissures on each subject's MR scan. The Talairach coordinates with best dipole solutions were reported. The CURRY 4.5 program (Neurosoft) was used for analyses of localization.

MR Image Acquisition

Brain imaging was performed using a 1.5 T GE Signa MR scanner with a custom head coil. Anatomical images were obtained with a standard 3‐D T₁‐weighted sequence (resulting in a 256 × 256 × 66 matrix with 0.938 × 0.938 × 2.0‐mm spatial resolution, TR = 585 ms, TE = minimum).

RESULTS

In Task 1 subjects responded faster and with higher accuracies to proximity‐ (635 ms, 87.7%) than to similarity‐defined target stimuli (684 ms, 77.2%, F(1,17) = 28.6, 5.30, P < 0.001, 0.03, respectively). Behavioral responses were both faster and more accurate in Task 3 (central target) (553 ms, 95.1%) than in Task 2 (peripheral target) (569 ms, 91.1%) (F(1,17) = 5.32, 6.66, respectively, P < 0.03). Performance in Task 2 remained both faster and more accurate than the performance in Task 1 (660 ms, 82.5%, averaged across proximity and similarity conditions) (F(1,17) = 10.5, 12.0, respectively, P < 0.01).

ERPs to uniform and grouped nontarget stimuli in Task 1 and the voltage topographies of the specific components are illustrated in Figure 3a. ERPs to uniform and grouped nontarget stimuli were characterized with a negative component (N110) over the medial occipital cortex and a positive wave (P1) over bilateral occipital sites. Both components peaked between 80–120 ms after stimulus onset. The N110 was followed by a longer‐latency positivity (P2) over the medial occipital cortex peaking between 200 and 240 ms.

Figure 3.

a: Grand average ERPs at posterior electrodes elicited by uniform and grouped nontargets in Task 1. The voltage topographies show maximum amplitudes over the medial occipital area for the P2 but over bilateral occipital areas for the P1. b: Grouping related difference waves in Task 1 obtained by subtracting ERPs to the uniform nontargets from those to proximity‐ or similarity‐grouped nontargets. The voltage topography at 92–116 ms (Pd100) shows maximum amplitude over the medial occipital area. The long‐latency component (Nd240) shows maximum amplitudes over occipito‐parietal area for proximity grouping whereas over the occipito‐temporal area for similarity grouping.

To identify the electrophysiological activities related to perceptual grouping, difference waves were obtained by subtracting ERPs to the uniform nontarget stimuli from ERPs to grouped nontarget stimuli. As can be seen in Figure 3b, in Task 1 the difference wave in association with proximity grouping was characterized by a positivity peaking between 80 and 120 ms after stimulus onset at electrodes over the medial occipital site (Pd100) (F(1,17) = 19.8, P < 0.001). To localize the generators of the Pd100, voltage topographies of the grouping‐related difference waves were calculated and plotted on a real subject head model. The Pd100 showed maximum amplitude over the medial occipital sites (Fig. 3b). The neural generators of the Pd100 were then estimated by fitting one dipole to its grand average voltage distribution between 90 and 110 ms based in a realistic‐head boundary‐element model. The best‐fit dipole was situated in the calcarine cortex and slightly lateralized to the right hemisphere (Fig. 4a) with Talairach coordinates of x, y, z = 11.3, −72.3, 10.5. The dipole solution accounted for 92% of the variance of the topography of the Pd100 at 90–110 ms. The Pd100 was followed by a long latency negativity peaking at about 240 ms (Nd240) (F(1,17) = 9.91, P < 0.01) with maximum amplitudes over bilateral occipito‐parietal areas (see the voltage topography in Fig. 3b). Similarity grouping was indexed only by a long latency negativity (Nd240) (F(1,17) = 5.06, P < 0.04), which showed maximum amplitude in the occipito‐temporal areas (see the voltage topography in Fig. 3b).

Figure 4.

a: The best‐fit dipolar source for the Pd100 at 90–110 ms. The dipole was located in the calcarine cortex and is shown in the MR image (top row) and the realistic‐head boundary‐element model (bottom row) of a representative subject. b: The best‐fit dipolar sources for the P1 at 90–110 ms. Two dipoles were located to bilateral extrastriate cortex and shown in the MR image (top row) and the realistic‐head boundary‐element model (bottom row) of a representative subject.

As the peak latency of the sensory component P1 overlapped with that of the proximity‐grouping‐related activity (Pd100), dipole modeling was also performed on the P1 component to confirm the locations of its distinguished neural generators. The voltage topography showed that P1 had maximum amplitudes over bilateral occipital areas (see Fig. 3a). Dipole modeling suggested that the P1 had two generators in the left and right extrastriate cortex (Fig. 4b). The Talairach coordinates of the dipoles corresponding to the uniform stimulus in Task 1 are x, y, z = −23.5, −86.2, 16.0, and 28.9, −76.5, 7.7, respectively. The two dipoles accounted for 94% of the variance of the topography of the P1 at 80–112 ms.

Figure 5 shows ERPs to nontarget stimuli and grouping‐related difference waves in Tasks 2 and 3, respectively. Task relevance modulated the sensory components of the ERPs as early as 60 ms after stimulus onset. The P1 component had a larger amplitude at posterior occipital sites in Task 1 than in Task 2 (F(1,17) = 13.1, P < 0.002) and in Task 3 (F(1,17) = 32.2, P < 0.001). However, P1 amplitudes did not differ between Tasks 2 and 3 (P > 0.1). The large attentional window in Task 2 induced an enhanced N110 between 100 and 120 ms over the medial occipital sites relative to that in Task 3 (F(1,17) = 11.8, P < 0.003). In contrast, the long‐latency positivity (P2) was of greater amplitude between 200 and 240 ms at the posterior occipital sites in Task 3 relative to both Task 1 (F(1,17) = 5.02, P < 0.04) and Task 2 (F(1,17) = 11.8, P < 0.001). However, the P2 amplitudes did not differ between Tasks 1 and 2 (P > 0.2).

Figure 5.

a: Grand average ERPs (top row) and grouping related difference waves at posterior electrodes in Task 2. The voltage topographies show maximum distribution of the N110 at 100–120 ms and the early grouping related positivity (Pd100) at 92–112 ms over the medial occipital area. b: Grand average ERPs (top row) and grouping related difference waves at posterior electrodes in Task 3. The grand average voltage topography shows maximum distribution of the long latency grouping related activities (Nd220) over the medial occipital area at 212–240 ms.

As can be seen in the bottom row of Figure 5a, proximity grouping continued to influence early ERP components in Task 2, indicated by an early positivity peaking at 100 ms after stimulus onset (Pd100) in the grouping‐related difference waves (F(1,17) = 6.81, P < 0.02). Nevertheless, the Pd100 in Task 2 was much smaller than that in Task 1 (F(1,17) = 8.33, P < 0.01), as illustrated in Figure 6. Dipole modeling of the Pd100 in Task 2 did not come to a good solution because of a low signal‐to‐noise ratio. However, the voltage topography showed that the Pd100 had maximum amplitudes over the medial occipital cortex, similar to that in Task 1. Both proximity and similarity grouping were also indexed by long latency negativities between 180 and 280 ms in the grouping‐related difference waves (Nd200 for proximity, F(1,17) = 13.1, P < 0.002; Nd240 for similarity, F(1,17) = 4.91, P < 0.04). Once again, the long latency negativities related to proximity and similarity grouping in Task 2 were of smaller amplitudes than those in Task 1 (F(1,17) = 5.89, P < 0.025). When central color discriminations were required (in Task 3), proximity grouping alone was indicated only through a negativity peaking at 220 ms (Nd220, F(1,17) = 10.0, P < 0.006). The voltage topography showed that the Nd220 had maximum amplitude over the medial occipital cortex. However, no significant difference was found between similarity and uniform stimuli for any time window (P > 0.1).

Figure 6.

Grouping‐related difference waves at middle occipital electrode in Tasks 1, 2, and 3. The Pd100 is labeled to illustrate the difference of the Pd100 amplitude across tasks.

DISCUSSION

The current work identified neural correlates of early perceptual grouping in human brain based on dipole modeling of grouping‐related activity in a realistic head model made from MR images. The electrophysiological grouping effect was defined by the contrast between responses to displays in which local elements were either grouped together or evenly spaced. The ERP results showed that proximity grouping was indexed by an early positive wave between 80 and 120 ms (Pd100) after stimulus onset with maximum amplitudes over the medial occipital region. Although subjects were not required to respond to the nontarget stimuli in which local elements were either grouped (grouped stimuli) or evenly spaced (uniform stimulus), the time course and morphology of the Pd100 were similar to those obtained in the previous studies [Han et al., 2001, 2002], in which subjects responded to each stimulus array. It appears that the proximity‐grouping‐related Pd100 is independent of the behavioral response requirement.

The neural generator of the Pd100 was localized to the calcarine cortex slightly lateralized to the right hemisphere using dipole modeling in a realistic head model. Importantly, the location of the dipole for the Pd100 was close to (although a little bit posterior to) the locus where our previous fMRI studies identified activation associated with grouping by proximity [Han et al., 2005]. The current ERP results are consistent with our previous fMRI findings and provide temporal information about the grouping process in human brain. The calcarine activity could not simply reflect processing of low spatial frequencies in the proximity‐grouped stimuli because the previous ERP [Han et al., 2001] and fMRI [Han et al., 2005] studies observed similar grouping effects over the posterior area or in the calcarine cortex even when low spatial frequencies were not available in the proximity‐grouped stimuli. It is also unlikely that the calcarine activity simply resulted from refractoriness differences arising from the fact that uniform and similarity‐grouped stimuli were more frequent than proximity‐grouped stimuli in each block of trials of Task 1 and the local elements of the uniform and similarity‐grouped stimuli occupied locations different from those occupied by the elements of proximity‐grouped stimuli (see the Appendix for the presence of the Pd100 when the refractory effect was controlled for).1 Taken together, the ERP and fMRI results suggest that the proximity‐defined larger‐scale organization of discrete visual elements in the visual field may take place as early as 100 ms after sensory stimulation and has a neural generator in human calcarine cortex. In addition, both the ERP and fMRI results showed that the calcarne activity is specific to the proximity grouping. There is no evidence for grouping by similarity being associated with such early neural activity in the calcarine cortex, even though subjects performed the same discrimination task in the two stimulus conditions. One may notice that the contextual effects in the primary visual cortex observed in single‐unit recording studies [Kapadia et al., 1995; Marcus and Van Essen, 2002], which reflect integration between stimuli inside and outside receptive fields of neurons, occur much earlier (50–60 ms poststimulus) than the early grouping effect observed here (about 100 ms poststimulus). It is possible that the grouping‐related Pd100 reported here arose from the synchronous activities of neuronal populations engaged in a large‐scale grouping process and thus requires longer time to reach maximum amplitudes.

Our ERP results showed that the sensory responses indexed by the P1 component had generators in bilateral extrastriate cortex. This is consistent with previous reports [Clark and Hillyard, 1996; Heinze et al., 1994; Mangun et al., 1997; Martinez et al., 2001; Noesselt et al., 2002]. However, the proximity‐grouping‐related activity was located in the medial visual cortex and close to the calcarine fissure. Thus, it appears that, although the Pd100 partially overlapped the P1 component in time, they had neural sources in different locations in the visual cortex, suggesting that perceptual grouping operations are dissociated from at least some aspects of sensory processing in the visual cortex even though they may overlap in time.

Proximity grouping was also reflected in long latency activities (Nd240) with an occipito‐parietal distribution. This is in line with the prior fMRI observations that proximity grouping also induced activations over bilateral inferior parietal cortices [Han et al., 2005]. Unlike proximity grouping, grouping by shape similarity was indexed only by a long‐latency negativity (Nd240) with an occipito‐temporal distribution. The electrophysiological correlates of similarity grouping may have generators in the right medial occipital cortex, given that our fMRI results showed stronger activation for similarity‐grouped stimuli than for uniform stimuli in this area [Han et al., 2005]. Thus, our combined ERP and fMRI findings provide evidence that grouping operations defined by proximity and similarity of shape are dissociated in both the temporal and the spatial domain. The delayed neural activities for similarity grouping are possible neural substrates for the slower behavioral responses to similarity‐ rather than to proximity‐defined stimuli [Han and Humphreys, 1999; Han et al., 1999a, b].

Moreover, our ERP results show that the early grouping operations are modulated by task relevance and by whether stimulus arrays of grouped stimuli fall within an attended spatial region. The proximity‐grouping‐related Pd100 occurred most strongly within the calcarine cortex when the task required participants to judge the orientations of spatial groups. This early proximity‐grouping‐related neural activity was weakened when the nontarget stimuli were of low (Task 2) relative to high (Task 1) task relevance (see Fig. 6). In addition, the Pd100 was eliminated when the nontargets were of low task relevance and fell outside the attended area (Task 3). The contrast between the results of Tasks 1 and 3 is in agreement with our previous fMRI observations which showed fMRI activations in the calcarine cortex associated with proximity grouping in Task 1 but not in Task 3 [Han et al., 2005]. The combined ERP and fMRI results indicate that the early grouping operations in the visual cortex are influenced by the allocation of attentional resources, i.e., being salient only when stimulus arrays of grouped stimuli fall in an attended area of space. The early grouping process is reduced when visual elements fall outside an attentional window. However, the contrast between the results of Tasks 1 and 2 is different from our previous fMRI observations, where the calcarine activation associated with proximity grouping did not differ between in Tasks 1 and 2 [Han et al., 2005]. One possible account of the discrepancy between the ERP and fMRI results is that, relative to the ERPs, the fMRI signals recorded from the calcarine cortex included stronger effects of feedback from high brain structures because of the fMRI signal's longer response latency. The electrophysiological signals are thus more sensitive than the hemodynamic responses in disclosing the differential neural activities in the visual cortex in association with the early grouping operation. The ERP results indicate that attentional resources influence grouping in nontarget stimuli, but this occurs most strongly when the grouping cues present are used for the behavioral task with other items (i.e., the target stimuli).

The lateralized long‐latency ERP correlates of proximity grouping, occurring between 180 and 280 ms (i.e., Nd240), were also decreased when nontargets were of low task relevance (Task 2 and 3 relative to Task 1). However, a long‐latency activity was evident over the medial occipital cortex when the stimuli fell outside the attentional window (i.e., the Nd220 in Task 3). This indicates a further distinction between the late and early grouping processes. It is possible that feedback from higher visual areas facilitates the late component of perceptual grouping even when early grouping is impaired by a lack of attentional resources. This effect was specific to proximity‐grouping here, since the long‐latency ERP correlates of similarity grouping (Nd240) were decreased by low task relevance and eliminated by the shrinking of the attentional window. This may reflect the generally weaker effects of similarity grouping on performance, which remain more dependent on spatial attention than proximity‐grouping.

Our ERP findings conflict with the core idea of contemporary theories of visual perception that grouping operations at early stages of visual perception are independent of allocation of visual attention [Humphreys, 1998; Mattingly et al., 1997; Ward et al., 1984] and attention only performs on perceptual objects formed at a preattentive stage by grouping operations [Duncan, 1984; Duncan and Humphreys, 1989; Kahneman and Henik, 1982; Marr, 1982; Treisman, 1986]. The modulation of early grouping‐related activity by task relevance and size of attended area (e.g., the Pd100) cannot be explained by the fading of grouping information in memory [Moore and Egeth, 1997]. Together with our prior fMRI findings, the current ERP results support the proposal that attention is engaged even in the early process of grouping operations [Ben‐Av et al., 1992; Mack et al., 1992]. The process of grouping operations that form perceptual units is facilitated by spatial attention. Therefore, it may be proposed that attention may be involved in both the early and late stages of visual perception to facilitate the formation of perceptual units and selection of perceptual objects, respectively.

The current ERP results also showed neural modulations linked to sensory processing. For instance, the P1 component, which was localized to the bilateral extrastriate cortex, was of larger amplitude in Task 1 than in Tasks 2 and 3. The pattern of the P1 modulation by task relevance was similar to that of the early grouping‐related activity (Pd100). Previous studies have shown modulations of the P1 component by spatial attention in a similar time window and at similar spatial locations in the visual cortex [Clark and Hillyard, 1996; Heinze et al., 1994; Mangun et al., 1997; Martinez et al., 2001; Noesselt et al., 2002]. Our results reinforce the literature by showing that task relevance also enhanced neural activities associated with both sensory and perceptual processing of the nontarget stimuli. Relative to task irrelevant stimuli, task relevant stimuli might capture attention, even when they were not required to respond, and thus the corresponding sensory and perceptual processing were facilitated. The P1 appeared to be larger in Task 3 than in Task 2. However, the P1 difference between Tasks 2 and 3 did not reach significance. We also found that the N110 was enlarged in Task 2 relative to Task 3. Because Task 2 required subjects to establish a large attentional window to encompass the entire stimulus array, whereas Task 3 required focusing attention on the fixation and placed the main stimulus outside the attended region, the N110 effect suggests that the enlarged attentional window enhanced neural activities over the medial occipital area when stimuli were displayed at fixation. In contrast, the long‐latency component (P2) showed larger amplitude in Task 3 than in Task 2, suggesting that the P2 was enhanced by shrinking the attentional window. Given the long latency of the P2 component, it may be proposed that the P2 effect reflects the feedback from high‐level brain structures. It should be noted that modulations of the P1, N110, and P2 by task relevance and variation of attentional windows were different from the modulation of the early grouping‐related activity (Pd100) in location (P1), polarity (N110), and time course (P2), respectively. Therefore, it may be proposed that attentional effects on sensory processing are possibly distinct from those modulating perceptual grouping defined by proximity.

In conclusion, our ERP results provide electrophysiological evidence that early grouping processes defined by proximity may occur in the human calcarine cortex. The data converge with findings from animal studies, where responses of single neurons in monkey primary visual cortex have been shown to be modulated by grouping with stimuli both inside and outside the receptive field [Kapadia et al., 1995; Sugita, 1999]. Although the lateral P1 component was not modulated by grouping process, the current ERP data cannot rule out the possibility that the lateral extrastriate cortices are also involved in grouping operations [e.g., Altmann et al., 2003]. However, our ERP data suggest that the initial grouping operation defined by proximity occurs mainly in the calcarine cortex, similar to other early processes of visual perception such as figure‐ground segmentation [Skiera et al., 2000]. Representation of perceptual objects formed at such an early stage is then transferred to other brain areas in the visual pathway for further analysis and recognition. In addition, we show that task relevance and the spread of spatial attention modulated the grouping‐related activities even in early cortical regions. The data provide ERP evidence for the interactions between perceptual grouping and top‐down influences early on in the visual cortex.

One reviewer suggested that the refractory effect might contribute to the Pd100 observed in Task 1 because uniform and similarity‐grouped stimuli were more frequent than proximity‐grouped stimuli in each block of trials and the local elements of the uniform and similarity‐grouped stimuli occupied locations different from those occupied by the elements of proximity‐grouped stimuli. Moreover, the row and column proximity stimuli also stimulated different locations. To test if the proximity‐grouped related Pd100 simply arose from refractoriness differences, we conducted a new control experiment. In each block of trials in the control experiment, there were equal numbers of uniform and proximity‐grouped stimuli that were presented in a random order. Moreover, there was only one type of proximity‐grouped stimuli (i.e., either row or column stimuli) in each block of trials. Thus, half of the proximity stimuli were preceded by uniform stimuli (two successive stimuli stimulated different locations) and half of the proximity stimuli were preceded by proximity stimuli (two successive stimuli stimulated the same locations). However, the refractory effect was the same for uniform stimuli because half of the uniform stimuli were preceded by proximity stimuli (two successive stimuli stimulated different locations) and half of the uniform stimuli were preceded by uniform stimuli (two successive stimuli stimulated the same locations). Consequently, the effect of refractoriness on ERPs was the same for the uniform and proximity‐grouped stimuli. In addition, a long interstimulus interval was used in the control experiment to reduce the effect of overlap between ERPs elicited by two successive trials. We examined if the Pd100 can be observed when the refractory effect is comparable for proximity and uniform stimuli.

Method

Twelve graduate and undergraduate students (7 men, ages 19–26) from Peking University participated in this experiment as paid volunteers. All were right‐handed and had normal or corrected‐to‐normal vision. The stimuli and procedure were the same as those in Task 1 except the following. Both column and row proximity‐grouped target stimuli were used in each block of trials. However, there were only uniform and column proximity‐grouped nontarget stimuli in two blocks of trials, whereas only uniform and row proximity‐grouped nontarget stimuli in the other two blocks. Each subject completed 160 trials for practice, followed by 640 trials in four 160‐trial blocks. The interstimulus intervals were randomized between 800 and 1200 ms. EEG was recorded from 29 scalp electrodes extended from the 10/20 electrode system. ERP data analysis was the same as those in Task 1 and focused on the Pd110 over the medial occipital electrode (Oz).

Results and Discussion

Subjects correctly responded to 78.3% of proximity‐grouped targets with mean RTs of 712 ms. Figure A1 shows the grand average ERPs to nontarget stimuli recorded at the medial occipital electrode (Oz). The ERPs were characterized with a negative component (N110) over the medial occipital cortex, which was followed by a longer‐latency positivity (P2) over the medial occipital area. The grouping‐related difference wave was obtained by subtracting ERPs to the uniform stimuli from those of the proximity‐grouped stimuli (collapsed both column and row grouped stimuli). As can be seen in Figure A1, the grouping‐related difference wave was characterized by a positivity peaking between 84 and 124 ms after stimulus onset at electrodes over the medial occipital site (Pd100, t(11) = 2.45, P < 0.03), which was followed by a negative wave peaking between 200 and 260 ms (Nd240, t(11) = 2.57, P < 0.03). The Pd100 had maximum amplitude over the medial occipital area, as shown in the voltage topography in Figure A1 (b).

Figure A1.

a: Grand average ERPs elicited by uniform and proximity‐grouped nontargets at the medial occipital electrode (Oz). b: A grouping related difference wave at Oz obtained by subtracting ERPs to the uniform nontargets from those to proximity‐grouped nontargets. The voltage topography at 92–116 ms (Pd100) shows maximum amplitude over the medial occipital area.

The ERPs recorded at the middle‐occipital electrode in the new control experiment were characterized with two negative waves during the first 200 ms. These appear different from the ERPs in Task 1, which were completely positive in polarity. However, the ERPs in Task 1 also showed two negative‐going waves in the first 200 ms, similar to those in the control experiment. Given that the only difference between the two experiments was that the interstimulus intervals in Task 1 (300–500 ms) were shorter relative to those in the control experiment (800–1,200 ms), it may be suggested that the positive polarity in ERPs in the first 200 ms in Task 1 might reflect the refractoriness effects induced by the overlap of ERPs generated by two subsequent stimuli. When the interstimulus intervals were increased as those in the control experiment, the refractoriness effects were reduced, and thus the positive shift of the ERPs in the first 200 ms was eliminated.

Despite the difference in ERP waves between Task 1 and the control experiment, the control experiment observed an early grouping‐related difference wave, i.e., the Pd100, with maximum amplitude over the medial occipital area. Both the latency and the scalp distribution of the Pd100 observed in the control experiment are similar to those of the Pd100 observed in Task 1. However, the Pd100 observed in the control experiment was obtained under the condition that the refractory effect was comparable for proximity and uniform stimuli and the stimulus intervals were longer than those used in Task 1. The results indicate that the proximity‐grouping‐related Pd100 was evident even when the refractory effect was controlled for. Therefore, the Pd100 observed in Task 1 could not simply arise from the refractoriness differences.