Stimulus-specific competitive selection in macaque extrastriate visual area V4

Abstract

Macaque visual area V4 has been implicated in the selective processing of stimuli. Prior studies of selection in area V4 have used spatially separate stimuli, thus confounding selection of retinotopic location with selection of the stimulus at that location. We asked whether V4 neurons can selectively respond to one of two differently colored stimuli even when they are spatially superimposed. We find that delaying one of the two stimuli leads to selective processing of the delayed stimulus by area V4 neurons. This selective processing persists when the stimuli move together across the visual field, thereby successively activating different populations of neurons. We also find that this effect is not a spatially global form of feature-based selection. We conclude that selective processing in area V4 is neither exclusively spatial nor feature-based and may thus be surface- or object-based.

Single-unit recording studies in monkeys show that extrastriate cortex can preferentially process one of multiple stimuli. This selective processing is driven by endogenous attention (1) and by exogenous factors such as stimulus salience (2). In area V4, it has been found that when a preferred stimulus and a nonpreferred stimulus appear within a neuron's classical receptive field (CRF), attending to the preferred stimulus typically increases firing rate, whereas attending to the nonpreferred stimulus typically decreases firing rate (2–6). Similarly, when two stimuli appear in the CRF, increasing the preferred stimulus' contrast typically increases the firing rate of V4 neurons, whereas increasing the nonpreferred stimulus' contrast typically decreases firing rate (2). These results suggest that endogenously and exogenously driven stimulus selection engage a common mechanism (7). One proposal is that the CRF shifts toward or shrinks around the selected stimulus (1, 8–11). This model, however, cannot account for the obligatory spread of attention across surfaces (12) or for selective processing of one of two spatially superimposed stimuli (13–18). Such findings suggest that selection can be surface- or object-based. Our goal was to determine whether selective processing within area V4 is inherently spatial (in a retinotopic sense) or alternatively accommodates surface- or object-based selection.

Stimuli were spatially separated in previous studies of stimulus selection in area V4. These studies thus could not distinguish between selection of a retinotopic location and selection of the stimulus at that location. To overcome this limitation, we adapted a paradigm used in human psychophysical studies (15, 17–19), which used virtual surfaces defined by rigidly rotating patterns of dots. Retinotopically based selection was ruled out by superimposing two dot fields, rotating in opposite directions. This results in perception of two superimposed transparent surfaces with ambiguous depth ordering. To bias selection in favor of one of the surfaces, we took advantage of the fact that abrupt stimulus onset automatically captures attention (19–21). Delaying the onset of one of two superimposed surfaces leads human observers to be better at discriminating features of that surface than features of the other nondelayed surface (17).

The processing advantage conferred to the delayed surface has been found to last for at least 500 ms. We reasoned that if area V4 mediates this form of surface-based selection, V4 neurons should be driven preferentially by the delayed surface over a similar time course. Color-selective neurons should respond more when a surface of the preferred color is delayed than when the other surface is delayed. We tested this prediction by recording neuronal responses in area V4.

Results

Experiment 1: Does Selection Occur Under Conditions That Rule Out Spatial Selection?

In experiment 1, we asked whether selection mechanisms in area V4 operate when stimuli are spatially superimposed. Specifically, when one of two stimuli appearing within the CRF is delayed in onset, are V4 responses driven preferentially by the “new” surface? We recorded V4 responses from two fixating monkeys (99 neurons: 78, monkey A; 21, monkey B). One surface was of the neuron's preferred color (“preferred surface”) and the other was of an equiluminant nonpreferred color (“nonpreferred surface”). Either the preferred or nonpreferred surface was delayed in onset. Illustrative responses for one neuron are shown in Fig. 1. When the nonpreferred surface was added after the preferred surface (single-surface response; Fig. 1A, left), activity was suppressed (pair response; Fig. 1A, right). Conversely, when the preferred surface was added after the nonpreferred surface (single-surface response; Fig. 1B, left), responses increased (pair response; Fig. 1B, right). Although stimuli were identical after second surface onset, the pair response was greater when the preferred surface was delayed (Fig. 1B, right) as compared with when the nonpreferred surface was delayed (Fig. 1A, right).

Fig. 1.

Example neuron. (A) Delayed onset of nonpreferred surface. Neuron responded robustly to the appearance of preferred surface (left). Responses were suppressed by addition of the nonpreferred surface (right), whose onset is indicated by the vertical dotted line. (B) Delayed onset of preferred surface. The nonpreferred surface evoked a weak excitatory response (left). Response increased when the preferred surface was added (right). Note that sensory conditions in the latter part of the trial were identical. The difference in neuronal response during this period depended only on which surface was delayed.

This pattern held across the population (Fig. 2). The red line shows the average response on trials in which the preferred surface appeared first, and the nonpreferred surface was then added. The blue line shows the average response on trials in which the nonpreferred surface appeared first, and the preferred surface was then added. The vertical dotted line indicates when the second surface was added. Unsurprisingly, addition of the preferred surface increased the response. In contrast, adding the nonpreferred surface led to a significant dip in the response [see Materials and Methods in supporting information (SI) Text]. The pair response was thus higher when the delayed surface was preferred, rather than nonpreferred. The difference in the population average response across the two conditions is indicated by the shaded region between the population responses. The dip in the response caused by addition of the nonpreferred surface was not predicted by single-surface responses: nonpreferred surfaces, when presented alone, were excitatory for every neuron in our population and hence for the population average (Fig. 2, blue line).

Fig. 2.

Population response (experiment 1). The red and blue lines indicate the population average responses over the duration of the trial. The red line indicates the response when the preferred surface appeared first, followed by the addition of the nonpreferred surface. The blue line indicates the response when the nonpreferred surface appeared first, followed by the preferred surface. The vertical dotted line indicates when first surface appeared. Solid vertical lines indicate the time period over which modulation indices (MIs) were computed. Modulation resulting from delayed onset is indicated by shading.

These results are similar in two ways to previously observed interactions between spatially separated stimuli. First, earlier studies have found that responses to a preferred stimulus can be reduced by addition of an excitatory nonpreferred stimulus within the CRF (22, 23). Second, the magnitude of this reduction can be increased or decreased by factors that, respectively, favor the nonpreferred or the preferred stimulus. These factors include elevating the relative contrast of one of the two stimuli or directing attention to one of the stimuli (2–5, 7). In the present experiment, the response evoked by the pair was biased by delayed onset, a manipulation that puts the delayed stimulus at an advantage, as measured in human observers (17, 19, 21) and in monkeys (20).

Response latencies.

To compare single-surface and pair responses, we superimposed those two response periods with the first and second surface onsets aligned (Fig. 3A). Preferred and nonpreferred single-surface response latencies were 60 and 58 ms, respectively (see Materials and Methods in SI Text). We also measured the latency of significant suppression after addition of the nonpreferred surface (68 ms) and the latency of response increase after addition of the preferred surface (77 ms). Thus, the influence of the added surface was delayed 10–17 ms relative to single-surface responses.

Fig. 3.

Comparison of time course and computation of indices. (A) Responses to single surfaces and pairs, superimposed to allow comparison of the relative magnitude of selectivity to the delayed onset effect. Time 0 corresponds to the onset of the first surface (with single-surface responses indicated by dashed lines) and also to the onset of the second surface (with pair responses indicated by solid lines). Solid vertical lines indicate the time window over which response indices were computed. (B) Distribution of MIs is shifted to the right, indicating that neurons were driven preferentially by the delayed surface. (C) Modulation indices normalized by each neuron's selectivity. The distribution is again significantly shifted to the right.

Quantifying the effect of delayed onset.

To quantify the delayed onset effect, we computed a modulation index (MI) for each neuron:

This index is zero when neuronal responses to the pair do not depend on which surface was delayed. Positive MIs result if pair responses are larger when the preferred surface, rather than the nonpreferred, is delayed. Positive MIs thus indicate that the delayed surface is preferentially processed. Negative MIs indicate the reverse. Responses were computed over a 500-ms window (Figs. 2 and 3A, between the vertical solid bars), beginning 100 ms after second surface onset, which is when the pair responses, having crossed, deviated significantly from one another. The MI distribution was shifted rightward (Wilcoxon signed rank test, P < 0.001), indicating that pair responses were typically larger when the preferred surface was delayed (Fig. 3B). Thus, consistent with the population average responses illustrated in Figs. 2 and 3A, cell-by-cell pair responses across the population were biased in favor of the delayed surface.

Although positive MIs indicate that the delayed surface is preferentially processed, MI magnitude does not reveal the degree of dominance. This is because the MI does not take a neuron's selectivity into account. If, for example, a neuron's response to a preferred stimulus presented alone were 50% greater than to the nonpreferred stimulus presented alone, then even if pair responses were driven completely by the delayed surface, this would only yield an MI of 20%. To relate pair response selectivity to single-surface selectivity, we computed a normalized MI (NMI), which expresses pair response modulation as a percentage of each neuron's single-surface response selectivity: NMI = 100 × (MI/selectivity index), where selectivity index = 100 × (preferred − nonpreferred)/(preferred + nonpreferred).

An NMI of 100% would occur if the pair response were identical to the response evoked by the delayed surface appearing alone. That is, the NMI would be 100% if the pair response exclusively reflected the delayed surface. The distribution of NMIs (Fig. 3C) was shifted significantly to the right (Wilcoxon signed rank test, P < 0.001). The average modulation of the pair response was 29% of the neuron's single-surface selectivity.

Spike-dependent adaptation.

In experiment 1, the initial presentation of the preferred surface elicited more spikes than did the initial presentation of the nonpreferred surface. If this difference in the responses to the single surfaces led to differential adaptation, this could have contributed to the differential response to the pair. Our analyses show that any effect of adaptation was small (see Analysis of Spike-Dependent Adaptation's Contribution to Results of Experiment 1 in SI Text). Adaptation could not, in any event, account for the significant dip in mean response upon addition of the nonpreferred surface, which was excitatory when presented alone. Adaptation can reduce responsiveness but cannot, by itself, transform excitation into suppression. Experiment 2, below, showed that the delayed onset effect does not depend on spike-dependent adaptation.

Experiment 2: Is Selection Maintained as the Superimposed Surfaces Move Across the Visual Field?

In experiment 2, we asked whether the advantage conferred to the new surface was limited to the CRF or specific to the delayed surface itself. We tested this by adding the delayed surface at a location well outside the CRF and then moving the two superimposed surfaces into the CRF (Fig. 4A). The first surface appeared with its leading edge 8.7–12.6 degrees of visual arc (dva) (mean 9.6 dva) away from the edge of the CRF and moved toward it at 10 dva/s. After 750 ms, the second surface was superimposed on the first surface. The two surfaces continued moving for 750 ms. We reasoned that if the response modulation reflects preferential processing of the delayed surface, it should continue even as the surfaces move through space, successively activating neurons within area V4's retinotopic map. This manipulation also equated the neuron's response history over the period leading up to and including addition of the delayed surface, thereby excluding spike-dependent adaptation and other mechanisms dependent on response history.

Fig. 4.

Selection of one of two superimposed moving stimuli (experiment 2). (A) Experimental design. Responses were computed for each neuron during the period between III and IV. (B) Distribution of MIs is shifted to the right, indicating that the delayed surface exerted preferential control, even though it first appeared outside the CRF. (C) NMIs are normalized to each neuron's color selectivity. Again, the significant rightward shift indicates selection of the delayed surface.

We recorded from an additional 57 neurons. Twelve of these neurons were excluded because they elicited a significant response to the addition of one of the surfaces (one-tailed unpaired t test, P < 0.05), resulting in 45 neurons (24, monkey A; 21 monkey B). Even though the delayed onset evoked no response and there was no evidence of surround modulation before the appearance of the second surface (see Analysis of Surround Modulation's Contribution to Experiment 2 in SI Text), neuronal responses after the pair entered the receptive field were driven preferentially by the delayed surface (Fig. 4 B and C; mean NMI = 18%, Wilcoxon signed rank test, P = 0.005). Thus, the advantage conferred by sudden appearance of the delayed surface was maintained as the two surfaces moved across the visual field successively activating different neuronal populations. Whereas experiment 1 demonstrated selection under conditions that ruled out selection of a particular retinotopic location, this experiment demonstrates that selection is not restricted to the location of the delayed onset or to the neurons that responded to that onset. Further, because response history before entry of the pair was identical, the observed response bias cannot be attributed to mechanisms that depend on the neuron's response history, including spike-dependent adaptation.

Experiment 3: Is Selection Spatially Global and Feature-Based, or Specific to the Delayed Surface?

The results of experiments 1 and 2 cannot be accommodated by shifts or shrinkage of the CRF. However, we cannot yet conclude that they reflect surface-specific processing, because those results are also consistent with a spatially global bias favoring the color of the delayed surface. Both neurophysiological experiments in monkeys (24) and functional imaging studies in humans (25) have found evidence of this type of mechanism: endogenously directing attention to the color or motion of a stimulus enhances processing of that feature throughout the visual field. Although there was no endogenous cue in the present experiments, there was an exogenous cue (i.e., the abrupt onset of the delayed surface), which could have engaged global color-based mechanisms. If the delayed onset effect truly reflects surface-based selection, it should only apply to the delayed surface and not to surfaces of the same color at other locations.

To test this, we recorded from 22 additional neurons (17, monkey A; 5, monkey B) in two conditions (Fig. 5). In both conditions, two surfaces appeared inside the CRF and two other surfaces appeared outside the CRF. Each pair of surfaces included one that was of the neuron's preferred color and one that was of a nonpreferred color. In each condition, one pair of surfaces appeared simultaneously, along with the first surface of the other pair. After 750 ms, the delayed surface appeared. In the “delayed-onset-inside” condition (Fig. 5A), the delayed surface appeared in the CRF. In the “delayed-onset-outside” condition (Fig. 5B), the delayed surface appeared outside the CRF. If delayed onset engaged global color-based selection, we should have found that the color of the delayed surface was preferentially processed for both delayed-onset-inside and delayed-onset-outside conditions. Instead, consistent with a surface-specific mechanism, preferential processing was limited to the delayed-onset-inside condition. For this condition, as found in experiment 1, neurons were driven preferentially by the delayed surface (Fig. 6A), indicated by the rightward shift in the NMI distribution (Fig. 6B; mean NMI = 20%; Wilcoxon signed rank test, P = 0.019). No preferential processing (Fig. 6C) or corresponding shift in the NMI distribution (Fig. 6D) was seen for the delayed-onset-outside condition (mean NMI = −0.5%, n.s.; Wilcoxon signed rank test, P = 0.506). Thus spatially global feature-based attention does not account for the preferential processing elicited by delayed onset.

Fig. 5.

Stimulus conditions to test for spatially global feature-based selection (experiment 3). (A) In the delayed-onset-inside condition, one surface is delayed inside the RF. (B) In the delayed-onset-outside condition, the pair of surfaces in the RF appeared together and then the delayed surface appeared contralateral to the RF.

Fig. 6.

Results of test of spatially global feature-based attention. (A) The population average responses for the delayed-onset-inside condition shows preferential processing of the delayed surface, consistent with experiments 1 and 2. (B) Distribution of NMIs in the delayed-onset-inside condition shows a rightward shift, indicating that, consistent with experiments 1 and 2, these neurons were driven preferentially by the delayed surface. (C) Delaying the onset of the delayed surface outside the CRF (delayed-onset-outside condition) did not modulate population average response evoked by the pair of stimuli in the CRF and did not bias the NMI. (D) Modulation by color outside CRF, indicating that delayed onset does not lead to a spatially global feature-based bias.

Discussion

Over the past two decades there has been mounting evidence from psychophysical (12, 13, 15, 17, 18, 21, 26–29), neuropsychological (30), event-related potential (14, 19, 31, 32) and functional magnetic resonance imaging (16) studies that the visual system can select objects or surfaces for processing under conditions that preclude their selection on the basis of features or spatial location. In the present study, we find evidence for surface- or object-based selection in area V4. When one of two superimposed virtual surfaces is delayed in onset, this causes neuronal responses in V4 to be driven preferentially by the delayed surface. Responses to the pair are higher when the preferred-color surface is delayed, relative to when the nonpreferred-colored stimulus is delayed. This difference results from both an increase in response upon addition of the preferred surface and a decrease in response upon addition of the nonpreferred surface. The decrease in response was observed even though the nonpreferred stimulus was excitatory when presented alone. Because the stimuli always occupied the same retinotopic location, this form of selection cannot be explained by spatial mechanisms such as shrinkage of neuronal receptive fields. These effects are not due to spike-dependent adaptation or surround modulation (see Ruling Out Low-Level Mechanisms in SI Text), nor are they due to a spatially global color bias because delaying onset at one location did not result in selection of a feature-matched stimulus at another location. Remarkably, this difference in firing rate persists even when the delayed stimulus is added outside a neuron's CRF and the two superimposed stimuli move across the visual field into the CRF, a finding that is similar to findings in the lateral intraparietal area (a cortical area implicated in selective attention) (see Relationship to Previous Studies of Attention in LIP in SI Text). Neither spike-dependant adaptation nor center-surround antagonism appears to account for the effects we have observed. We conclude that delayed onset of a stimulus elicits selective processing in V4 that is specific to the conjunction of location and color that define the selected stimulus. Our findings are consistent with previous demonstrations of depth-based selection (see Depth-Based Selection in SI Text). They do not imply that V4 cannot be modulated by spatial or featural attention under other conditions (see Relationship to Previous Studies of Selection in Area V4 in SI Text).

Without a behavioral measure, the link between these findings and psychophysical surface-based effects should not be overstated. Elucidating this relationship requires comparison of neuronal and behavioral data, an important goal for future research. We note, however, that the time course of the neuronal effects observed here are comparable to that observed in psychophysical studies of exogenous attention. These studies have found that the effect of an exogenous cue is strongest shortly after the cue and falls off over the next several hundred milliseconds, both for spatially separate stimuli (29, 33–35) and for the type of superimposed stimuli used in the current study (15, 17). We thus suggest that the type of surface-based selection mechanism documented here may contribute to the surface-based attention effects that have been observed in humans.

Materials and Methods

All experimental and surgical procedures were approved by the Salk Institute Institutional Animal Care and Use Committee and conformed to National Institutes of Health guidelines for the care and use of laboratory animals. Recording chambers were positioned over V4 in two adult male macaques. Action potentials were recorded by using tungsten microelectrodes. Eye position was monitored by using an ISCAN Model ETL-400 eye monitor. Monkeys received juice reward for maintaining gaze within 0.75 dva of a 0.25-dva fixation point throughout each trial. Each dot field was 2.75 dva in radius, the density was 5 dots per dva², the dot size was 0.05 dva, and the speed of rotation was 50° of rotation per s. Dot colors were chosen according to the neuron's color preference and were equiluminant red, green, or blue. The experimental paradigms are as described in the relevant sections of Results. See Materials and Methods in SI Text for additional details.

Abbreviation

CRF

classical receptive field

dva

degrees of visual arc

MI

modulation index

NMI

normalized MI.