Short-Term Memory for Figure-Ground Organization in the Visual Cortex

Summary

Whether the visual system uses a buffer to store image information and the duration of that storage have been debated intensely in recent psychophysical studies. The long phases of stable perception of reversible figures suggest a memory that persists for seconds. But persistence of similar duration has not been found in signals of the visual cortex. Here we show that figure-ground signals in the visual cortex can persist for a second or more after the removal of the figure-ground cues. When new figure-ground information is presented, the signals adjust rapidly, but when a figure display is changed to an ambiguous edge display, the signals decay slowly – a behavior that is characteristic of memory devices. Figure-ground signals represent the layout of objects in a scene, and we propose that a short-term memory for object layout is important in providing continuity of perception in the rapid stream of images flooding our eyes.

Introduction

Responses of neurons in the visual cortex generally reflect changes in the visual stimulus within tens of milliseconds (Bair et al., 2002; DeAngelis et al., 1993) and, under natural viewing conditions, neurons signal new information several times per second as a result of eye movements (Vinje and Gallant, 2000; Yen et al., 2007). This continuous fluctuation of neural activity is hard to reconcile with the apparent stability of our visual world. One observation that suggests that there may be persistence of neural signals is the inertia of perception in ambiguous displays. The foreground-background assignment in Rubin’s vase-face figure, for example, persists for seconds between reversals. In the visual cortex foreground-background relationships are encoded with the representation of contours (Qiu and von der Heydt, 2005; Qiu and von der Heydt, 2007; Zhou et al., 2000): A piece of contour is represented by two groups of neurons with opposite preference for figure-ground direction. The left-hand contour of a square, for example, would produce high activity in neurons with figure-right preference, and low activity in neurons with figure-left preference. Neurons with such selectivity for border ownership are frequent in area V2 of the visual cortex of macaques (Qiu and von der Heydt, 2005; Zhou et al., 2000), and likely exist also in humans (von der Heydt et al., 2005). These neurons are part of mechanisms that integrate the image context and presumably play a role in organizing the information for subsequent selective processing (Qiu et al., 2007). We have now found that border ownership signals persist for about a second when a figure display is followed by an ambiguous display such as an edge forming the diameter of a circular field. Thus, the neural representation of figure-ground organization is more stable than expected based on traditional measurements of cortical response dynamics.

Results

We recorded single-cell activity from neurons of the visual cortex in two monkeys performing a fixation task. Edges of squares were presented in the receptive field of each neuron. Border-ownership selective neurons respond more strongly to the edge of a figure located on the preferred side than to the edge of a figure on the other side, although the edge in the receptive field is identical in both cases (Figure 1A). Despite the small size of their classical receptive fields, the responses of these neurons depend on the global image context.

Figure 1. Persistence of border-ownership specific activity in a neuron of monkey area V2.

(A) Modulation of firing rate by border ownership. The diagrams are schematic representations of the stimulus displays; the dashed ovals indicate the receptive field. The raster plots below show the spike activity. In the sequence shown at the top, the right edge of a light square was first presented in the receptive field (time -500) and then replaced by the left edge of a dark square (time 0). The sequence below shows presentation of the same stimuli in reversed order. The neuron responded with a high firing rate when the edge in the receptive field was owned by a square on the left of the receptive field (preferred side), but with a low firing rate when the edge was owned by a square on the right. The edge in the receptive field was identical in all four displays. The curves at the bottom show the smoothed averaged firing rates for the two sequences (solid and dashed lines). The firing rates reversed with a short delay after the change of border ownership.

(B) When a display with a square on the preferred side was switched to an edge with ambiguous ownership, the neuron continued to fire at a high rate, but when the display started with a figure on the non-preferred side, the neuron responded to the same ambiguous edge with a low firing rate. The responses only slowly approached a common level (see curves at bottom). The shaded gray background highlights the interval in which the displays were identical.

Persistence of figure-ground signals

The neuron illustrated in Figure 1 responded with a sharp increase in firing rate when a figure on the preferred side was presented, and with a drop when the figure was flipped to the non-preferred side (Figure 1A top). Similarly, when the order of presentation was reversed, the firing rate was initially low and then increased sharply (Figure 1A, lower). Thus, the responses quickly followed the change in border ownership. However, the selective activity persisted much longer when the figure was replaced by an edge that was ambiguous with regard to border ownership. After figure presentation on the preferred side the firing rate remained high, and after figure presentation on the non-preferred side the firing rate remained low (Figure 1B). Note that during the interval shaded gray the responses differed despite the displays being identical.

We calculated for each neuron the difference in instantaneous firing rate between the condition starting with figure on the preferred side and the condition starting with figure on the non-preferred side (that is, the difference between solid and dashed curves in Figure 1). We call this difference the border ownership signal (Zhou et al., 2000). The population border ownership signal was obtained by averaging the signals of all neurons in which border ownership produced a significant difference in the spike counts (p<0.05, ANOVA). We also fitted combinations of exponential functions to the data (see Methods). The results were similar for the two monkeys (Figure 2A, TH and JA; see also Table S1 in Supplemental Material). During the figure presentation, the border ownership signals showed a steep rise with a subsequent slight decline, and when the figure was replaced by the ambiguous edge, the signal remained positive, decaying slowly over the next second (persistence, red traces). But when the figure was switched to the opposite side, the signal reversed quickly (blue traces). To see how fast the signals changed under the two conditions we fit functions to the pooled data of 113 neurons from both monkeys and calculated the slopes after the transition. When the figure was flipped, the signal changed at a rate of 366 Hz per second, compared to only 19 Hz per second during the ambiguous edge phase (time constant 406 ms; for details see Table S1 in Supplemental Material). The rapid reversal in response to the change of border ownership contrasts with the slow decay in the ambiguous condition. We also looked at how the response decays when the figure is simply turned off (see Supplemental Material).

Figure 2. Persistence of the border ownership signal in the population response.

(A) The time course of the average border ownership signal for a 500 ms figure presentation followed by 1000 ms presentation of an ambiguous edge (red), and for the same figure presentation followed by a figure on the opposite side (blue). Data from two monkeys (TH, 65 cells, and JA, 48 cells; 7 cells were from V1 and 106 from V2). Smooth curves are combinations of exponential functions fit to the data (see Methods). The yellow background highlights the interval in which the displays were identical.

(B) Scatterplot showing the correlation between the border ownership modulation during the figure phase (average −400 to 0 ms) and the modulation during the persistence phase (average 200 to 1000 ms) for all 139 cells run with the experiment illustrated in Figure 1. Each symbol represents a cell; filled dots indicate cells with significant border ownership modulation (p < 0.05). The arrow marks the example neuron of Figure 1.

(C) Distribution of the ratio between persisting modulation and figure-induced modulation for neurons in which the figure-induced modulation was significant.

Finding persistence of responses is so unusual in the visual cortex that we first suspected it might be due to an afterimage. The 0.5s presentation of a bright or dark square might leave an afterimage on one side of the ambiguous edge which might then act like a real figure, producing a border-ownership signal. To control for this possibility, we included a flickering figure test that was identical to the test just described except that the gray values (or colors) of the square and its surround were rapidly alternated. This prevented or greatly reduced the build up of an afterimage. The flicker caused fluctuations of the border ownership signal in the figure phase due to the bursts of activity that followed each color reversal (Figure 3A purple traces). Nevertheless, during the ambiguous edge phase we found the same slow decay as for stationary figures (red traces). Indeed, the decay rates were virtually identical (Figure 3A, see Table S1 in Supplemental Material). We concluded that persistence of the border ownership signal is not due to an afterimage.

Figure 3. Varying the mode of presentation of the priming stimulus.

(A) The results of a control experiment in which figure and background colors were exchanged at a rate of 4 Hz to minimize the formation of an after image (flicker condition, purple curves; average of 34 cells). The border ownership signal fluctuated during the flicker phase, but persistence and decay were the same as after steady figure presentation (red curves).

(B) Persistence of the border ownership signal after various durations of figure presentation (black, 500 ms; blue, 250 ms; red, 50 ms; 62 cells). Amplitude and decay were similar for all durations.

The duration of the figure presentation was not critical for the persistence of the border ownership signal. We measured the decay of the signal for three figure durations, 500 ms (same as in Figure 2A), 250 ms, and 50 ms (Figure 3B). A flash of 50 ms produced almost the same amplitude of border ownership signal as a 500 ms presentation, and the persistence of the signal in the ambiguous edge phase was similar for all three durations. Note that the plots show the difference in activity between the two border ownership conditions; thus, a positive signal during the ambiguous phase indicates persistence.

So far we have described signals averaged over all border ownership selective neurons, but there were clear differences between neurons with respect to persistence. To show this we calculated, for each neuron, a border ownership modulation index for the persistence phase and the corresponding index for the figure phase. The modulation index is the difference between the spike counts for the two border ownership conditions divided by their sum. The scatter plot of the two indices (Figure 2B) shows all cells tested, and filled symbols indicate cells with significant border ownership modulation in the figure phase (p<0.05). Some cells showed virtually complete persistence (dots scattered around the diagonal line), while others with equally strong figure-induced border ownership signals showed no persistence (dots around the horizontal line). The heterogeneity of the population can be seen more clearly in the distribution of the ratio of persistent to figure-induced border ownership modulation (Figure 2C). The time constants of the decay also varied between neurons (Figure S3). Persistence seems to be a property of a subpopulation of border ownership neurons of V2. We found no obvious differences in other response properties, except that the onset of the averaged border ownership signal was slightly later in neurons with persistence than in the other neurons (114 versus 106 ms at half amplitude), while the onset of the responses was nearly the same in both groups (70 versus 69 ms at half amplitude).

Do border ownership effects accumulate?

We wondered if border ownership signals decay passively, like the voltage across a capacitor that is charged and discharged through a resistor. If so, the effects of successive stimulations should accumulate, like the charge on the capacitor. We investigated this by presenting figures twice within a fixation period, each time followed by a 1-second ambiguous edge presentation. The second figure could appear either on the same side as the first figure, or on the opposite side. We then determined whether the side of the first figure presentation influenced the border ownership signal during and after the second figure presentation (Figure 4, blue and red traces). Despite a clear difference at the onset of the second figure (time 0), the signals converged in the second figure response (arrow) and were virtually identical during the subsequent decay phase. In a third condition we flipped the figure back and forth twice (green trace). In this case, the signal was strongly negative at the beginning of the second cycle (time 0), but during the subsequent figure presentation it reached the same amplitude as in the other conditions. A capacitor-resistor circuit (or any linear low-pass filter in temporal frequency) would produce different signal levels during the second figure presentation because the charge accumulates: The signal in response to the second figure presentation would be higher if the first presentation was on the same side (blue) than it would if the first presentation was on the opposite side (red, green). The border ownership signal did not accumulate over repeated figure presentations, but each new presentation seemed to reset the signal. Thus, the persistence of the signal is not a passive decay, but resembles a trace in a storage device that can be set and reset by the incoming signals.

Figure 4. Test for summation of border ownership effects across repeated figure presentations.

The border ownership signal (average of 113 cells) was measured under three conditions. In two of them, figures were presented twice within a fixation period, either on the same side (blue) or on opposite sides (red), and each presentation was followed by one second of ambiguous edge display. In the third condition, the figure was switched back and forth without an intervening ambiguous display (green). In the second cycle the figure display was the same for all three conditions (0—500 ms). Note that all three border ownership signals converge during this interval (arrow). Thus, the border ownership effects did not accumulate.

Persistence of stereoscopically induced border ownership

So far we have described experiments with two-dimensional (2D) displays. We defined border ownership selectivity as selectivity for the location of a figure relative to the receptive field (comparing conditions in which the edge in the receptive field was identical). Finding this selectivity does not necessarily mean that a neuron is involved in figure-ground organization, which implies segregation into foreground and background. However, we have previously found that side-of-figure selectivity is correlated with a specific stereoscopic selectivity (Qiu and von der Heydt, 2005): Most of the side-of-figure selective neurons responded also to stereoscopic edges of proper orientation (e.g., edges produced with the method of random-dot stereograms) and were selective for the depth order of the edge. The vast majority of these neurons responded best to stereograms in which the surface on their preferred side of border ownership was the nearer, occluding surface, and less, or not at all, if the other side was the occluding surface. For example, if a neuron preferred figure-left in 2D, it would typically be selective for left-near/right-far edges in stereograms. This shows that the border ownership selectivity observed with 2D figures is intimately related to the process of creating a 3D representation.

To find out if persistence of border ownership signals is a property of a 3D representation, we performed the following test (Figure 5): Instead of presenting a square, we defined border ownership stereoscopically. We used three dots, the minimum stereoscopic stimulus that defines a plane in depth. The dots were presented either in the left or in the right half field of an ambiguous display, and with a stereoscopic disparity in the far range. In stereoscopic view (see stereograms in Fig. S1 and in Takeichi et al., 1992), the dots create the perception of a surface in the background: The half circle with the dots is perceived as a window, and the dots appear as lying on a plane surface that extends behind the window boundaries and behind the other half circle which appears as an occluding opaque surface, as illustrated in Figure 5A. Thus, the edge is owned on the side opposite to the dots. A trial started with the presentation of the ambiguous edge. After a brief interval, the three dots appeared, either on the left or on the right side. In Figure 5B top center, the light region would be perceived as in the back and the edge as owned on the left. The neuronal recordings showed a positive deflection of the border ownership signal upon appearance of dots (Figure 5B), corresponding to a left assignment, in agreement with perception. During the subsequent ambiguous phase, the signal persisted, decaying with an estimated time constant of 898 ms. The border ownership signal in this experiment was smaller than that found in the previous experiments (Figure 2–Figure 3). This is to be expected, because not all side-of-figure selective neurons also show stereoscopic edge selectivity. The result of this experiment shows that persistence is in the representation of foreground-background structure.

Figure 5. Persistence of stereoscopically induced border ownership signals.

(A) Illustration of the perceived depth ordering induced by a stereoscopic detail. Three dots with ‘far’ disparity were added to one side of an ambiguous edge display. The region with the dots appears to form a solid surface in the background, as if seen through a half-moon shaped window, whereas the region on the other side appears as an occluding surface.

(B) Test in neurons of area V2. After a 500 ms ambiguous edge presentation, the dots with far disparity were added on one side or the other for one second. The average border ownership signal (41 cells) showed a positive deviation, indicating that border ownership was assigned to the side opposite to the dots. The signal persisted after the dots had disappeared. The smooth curve is a piecewise fit of functions as in Figure 2A; dot indicates transition point.

Possible role of attention

Persistence and the ability to retain new information immediately are characteristics of memory circuits. The results described so far indicate a short-term memory for figure-ground organization. We suggest that this memory is set automatically by the stimulus, and not under the control of attention, because border ownership signals emerge independently of attention (Qiu et al., 2007), and we find persistence in monkeys that were simply fixating and not performing a memory task. Thus, the persistence is unlike a working memory that serves a specific task and requires attention. A correlate of “working memory” has been demonstrated by Super et al. (2001) who found enhanced firing in striate cortex when monkeys had to remember the location of a briefly presented figure for a future saccade. The elevation of activity at the figure location persisted until the execution of the saccade. Experiments of this kind require extensive training because the monkeys have to learn to divert their attention from the fixation point, but suppress eye movements to the location they are attending to. It would be surprising if the same diversion of attention occurred in monkeys that were simply fixating. Even if one assumes that attention was captured by the figure in a bottom-up fashion in our experiments and then remained on the same location after the switch to the ambiguous edge, one would have to explain why the influence of attention is different for a figure on one side of the receptive field compared to the other.

Whether or not an attention explanation would be plausible, we decided to examine the possible effect of attention capture experimentally. Our experiment is the exact bottom-up analogue of Super et al.’s top-down attention experiment. Super et al. showed that when, after presentation of the cue figure, a second object appeared in a new location and the monkey shifted attention to the new object, the response enhancement disappeared quickly (their Figure 4D). We presented a second figure 300 ms after the onset of the first figure and then switched both figures to ambiguous edges by presenting a mask with two circular holes (Figure 6). If attention capture was at the root of persistence, then the border ownership signals of the first figure should drop upon the appearance of the second figure, as in Super et al.’s experiment, and only the border ownership signals for the second figure should persist. However, the results did not show the expected drop of the signal at the first figure (Figure 6, continuous line). Rather, the signals persisted at both figures. If anything, the signal at the first-presented figure showed a stronger persistence, which is the opposite of the predicted result. Thus, the persistence of border ownership signals is most likely not a result of attention capture.

Figure 6. Control for an effect of attention capture: The onset of a second figure does not abolish border ownership signals at the first figure.

The presentation of one figure was followed after 300 ms by presentation of a second figure in a different location. Both figures were then replaced by ambiguous edges. The curves show the time course of the border ownership signal at the edge in the receptive field under two conditions: (1) first figure presented at the receptive field and second figure at a distance (continuous line), (2) presentation in reversed order (dotted line). Average of 23 cells. Arrow indicates the signal decay that would be produced by attention capture under the assumption that the signal reflects selective attention. See Results for further explanation.

Is persistence a network property of V2?

We have shown that V2 neurons respond to the same visual stimulus differently, depending on the stimulus history (Figure 4, compare red and blue curves), a phenomenon called hysteresis. One possible explanation is that border ownership information is stored in the state of activity of a network of neurons. The simplest circuit to achieve this would be a pair of neurons that mutually inhibit each other. Then, the neuron that initially achieved the higher firing rate will continue to inhibit its partner even after the signal that caused the initial differential activation has ceased. This is analogous to a flip-flop circuit in electronics which can be set by an external signal to one state or the other, and then holds this state after the external signal has been removed. The flip-flop hypothesis does not require the assumption of a process with a long time constant. The mutual inhibitory connections constitute a positive feedback loop which can maintain a firing rate difference over a long time, provided there is a source of continuous activation (e.g., input neurons that are activated by the edge stimulus, or recurrent self-activation). However, if the activity of both partners is interrupted, the stored information is lost.

This hypothesis can be tested by introducing a condition that effectively interrupts the activity of border ownership selective neurons in the cortex. We achieved this by inserting a blank field for 500 ms in the ambiguous edge phase (Figure 7). When the display went blank, the average firing rate returned to the resting level for several hundred milliseconds (thin trace, compare the blank interval around 500 with the blank period at the beginning). Consequently, the border ownership signal (thick trace) was zero during this interval. But when the ambiguous edge reappeared after the blank, the border ownership signal recovered. Indeed, it seemed to follow the same decay function as in Figure 2A, as if the edge stimulus had not been interrupted (dashed curve). This result rules out the hypothesis that the memory resides in the state of activity of V2 border ownership neurons. It might involve persisting activity in neurons elsewhere that influence the V2 neurons. Or, the information could be stored by a molecular mechanism, e.g., a temporary modification of membrane channels.

Figure 7. The border ownership signal survives the interruption of the edge responses.

A blank screen was displayed for 500 ms during the ambiguous edge phase, reducing the neural activity to the resting level (thin trace). The border ownership signal (thick trace) vanished during this period, but re-appeared and resumed its decay after the blank. Average of 47 cells. Dashed line shows an exponential with the same onset and time constant as the fit for the data of Figure 2A; amplitude and asymptote adjusted by eye.

Discussion

Our results demonstrate that neuronal responses related to border ownership persist for about a second. This is in contrast to the general observation that neuronal responses in the visual cortex rapidly follow changes in the stimulus (Bair et al., 2002; DeAngelis et al., 1993; Vinje and Gallant, 2000; Yen et al., 2007). The border ownership signals emerge rapidly in response to the visual stimulus, but persist after the figure-ground cues are removed (Figure 2). In the persistence phase the signals decay slowly, with a time constant of about 400 ms, but can be reset or reversed immediately by another figure presentation, leaving no trace of the previous signal value (Figure 4). We found similar persistence for displays in which border ownership was defined by stereoscopic depth (Figure 5). This result, and the experiment with rapidly alternating figure contrast (Figure 3A), rule out afterimages of the figure as a cause of the persistence. We also ruled out deviations in eye position as the cause (see Supplemental Material).

Persistence and the ability to retain new information immediately are characteristics of memory circuits. A short-term memory of this kind has not been reported before in signals of the visual cortex. The retention of border ownership is driven by the stimulus rather than by attentive processes, because we see it in monkeys performing a passive fixation task, and because previous experiments with controlled attention have shown that border ownership signals emerge independently of attention (Qiu et al., 2007). We have also ruled out automatic attention shifts (exogenous attention, Yantis and Jonides, 1984) as the cause of the persistence by showing that border ownership signals on one figure persist when a second figure is flashed on, an event that is known to capture attention (Figure 6). This experiment clearly shows that the persistence of border ownership signals is not a correlate of working memory (Super et al., 2001). Working memory critically depends on attention, whereas the memory for border ownership apparently does not. This is not to say that it cannot be influenced by attention. Volitional attention can modulate border ownerships signals (Qiu et al., 2007), and similar experiments with controlled attention are necessary to clarify if attention can influence the persistence of these signals.

Border ownership coding has been modeled as an emerging feature of networks of V2 neurons (Zhaoping, 2005). However, the hysteresis of border-ownership signals in our data cannot be the product of feedback loops in such networks because the signals survived the interruption of activity (Figure 7). But it is possible that information is held in reverberating circuits (Wang, 2001; Zipser et al., 1993) involving other cortical areas. Alternatively, the memory might be located in V2, but have a molecular basis. Time constants on the order of a second have been observed in Ca²⁺-induced K⁺ currents (McCormick and Williamson, 1989). However, such currents generally produce hyperpolarization, leading to a negative aftereffect. To explain the present results, a mechanism that leaves a positive aftereffect is needed.

We measured the persistence of border ownership signals when the border ownership cues were removed, but a contrast edge was left in the receptive field. When this edge was also removed, responses and border ownership signal vanished altogether within a tenth of a second (Figure 7 and also see Table S1 in Supplemental Material). Thus, we find persistence in border ownership signals, not in the edge responses per se. This is important, because persisting edge responses would interfere with the incoming signals and impair the ability of the system to represent rapidly changing visual information.

Psychophysical measurements in humans have indicated that visual iconic memory is of short duration (200 ms or less, Averbach and Coriell, 1961; Becker et al., 2000; Di Lollo, 1977; Sperling, 1960), but there is also evidence for a visual short-term memory of longer persistence, specifically for figure-ground organization and perception of 3D structure (Hulleman et al., 2005; Landman et al., 2004; Leopold et al., 2002; Sligte et al., 2008). Using a partial-recall paradigm in which subjects were asked to report the bar orientations in an array of eight texture-defined bars, Landman et al. found almost complete recall when subjects were cued 600 ms after the array offset. It has also been shown that border ownership assignment of an ambiguous edge can be influenced by priming with a brief presentation of an unambiguous display (Hulleman et al., 2005). We use the term “short-term memory” to describe the observation that a stimulus leaves a trace in the neural activity that lasts over a second, but can be reset within a small fraction of a second. The use of this term may be questionable, as we are not sure how our findings in the visual cortex relate to specific psychological memory concepts, and if the results from macaques can be generalized to humans. However, it seems important to stress the general similarity of our results with psychological findings on visual short-term memory.

Figure-ground organization is related to the process of object identification. Objects are perceived as continuous despite rapid fluctuations of the retinal images caused by eye movements, changes of viewpoint, and object movements. We propose that a short-term memory for figure-ground organization is important in providing this continuity of perception. Most commonly such a memory would have to bridge saccadic eye movements. Since the layout of objects in space is usually the same after a saccade, it would be useful to be able to retrieve the assignments from the previous fixation. More generally, border ownership signals may reflect the formation of tentative object representations, and what we see is the persistence of such representations. We are currently investigating if border ownership signals persist across eye movements and object movements.

Materials and Methods

We studied neurons in two male adult rhesus monkeys (Macaca mulatta). The details of our general methods have been described (Qiu and von der Heydt, 2005; Zhou et al., 2000).

Preparation

The animals were prepared by implanting, under general anesthesia, first three small posts for head fixation, and later two recording chambers (one over each hemisphere). Fixation training was achieved by controlling fluid intake and using small amounts of juice or water to reward steady fixation. All animal procedures conformed to National Institutes of Health and USDA guidelines as verified by the Animal Care and Use Committee of the Johns Hopkins University.

Stimuli and experimental design

Stimuli were generated with Open Inventor on a Pentium 4 Linux workstation with NVIDIA GeForce 6800 graphics card using the anti-aliasing feature of the software, and were presented on a 21-inch EIZO FlexScan T965 color monitor with 1600×1200 resolution at 72 Hz refresh rate. Stereoscopic pairs were presented side-by-side and superimposed optically at 40 cm viewing distance. The field of view subtended 17 by 26 deg visual angle. A neutral gray background of 28 cd/m² luminance was used, except for conditions in border ownership tests in which figure and background colors were flipped. A white (93 cd/m²) cross inside a 20 arcmin diameter disc of 9 cd/m² served as fixation point. The color tuning of each neuron was determined with stationary flashing bars, and the minimum response field was mapped with bars and drifting gratings. Orientation and disparity tunings were determined with moving bars. The test figures in the persistence tests were squares measuring typically 4 deg on a side. Occasionally, larger figures were used, so that the figure was at least twice the size of the receptive field. The figures were presented in a circular window with a diameter 3 times the size of the test squares, centered about the receptive field. Thus, for the 4-deg squares that were mostly used, the diameter of the window was 12 deg. The exception to this is the two figure, attention control experiment illustrated in Figure 6. Here the figures were presented without the surrounding circular window and the ambiguous configuration was created by introducing a surface with two circular holes that had diameters of 90% of the length of the edge of the squares. The color of the surrounding region was intermediate between the background and figure colors. The test squares and edges were generally presented with zero disparity (i.e., in the fixation plane), and the window with a near disparity of 37 arc min, so that it appeared about 58 mm in front of fixation point and test stimuli. In the 3-dot experiment the window disparity was set to zero. The dots were placed at a far disparity of 24 arc min, the equivalent of 35 mm of depth beyond the edge.

The direction of gaze was monitored for one eye with an infrared video-based system (Iscan ETL-200) at 60Hz with a spatial resolution of 5120 (H) and 2560 (V). The eyes were imaged through a hot mirror placing the camera on the axis of fixation. The optical magnification in our system resulted in a resolution of the corneal position signal of 0.08 deg visual angle in the horizontal and 0.16 deg in the vertical. Noise and drifts of the signal of course reduced the accuracy. Behavioral trials began with the presentation of the fixation mark on a blank screen. A test sequence was initiated when gaze was in a predetermined fixation window (1 deg radius) and the first stimulus appeared 300 ms after fixation was detected. The monkey was rewarded for keeping its gaze in the fixation window for a fixed duration of 2.3 or 3.3 s, depending on the experiment. After successful termination of a trial the display was blanked for an interval of 0.8 – 1.2 s (monkey TH) or 0.5 – 0.8 s (monkey JA). When fixation was broken, the trial was terminated and the following inter-trial interval was increased by 1 s. The recordings showed that fixation was generally more accurate than the size of the fixation window would suggest. The root mean squared values (SD) of the fixational eye movements in horizontal and vertical direction, calculated from all fixation periods of the main experiment, were (0.20, 0.26) deg in monkey TH and (0.16, 0.22) deg in monkey JA. This includes the errors of the recording method.

Each of the experiments described involved variation of several stimulus parameters. For example, the main experiment represented in Figure 2A and Figure 4 involved 4 binary variables: the local contrast polarity, the side on which the figure was presented initially, whether it was followed by a figure reversal or an ambiguous edge, and (in the case of ambiguous edge presentation) whether the second figure presentation was on the same side or opposite. Factorial or nested factorial designs were used and all conditions were presented in pseudo-random order in which each condition was presented once before moving on to the next repetition.

Recording procedures

Single-neuron activity was recorded extracellularly with epoxy-insulated tungsten microelectrodes inserted through the dura mater. A spike detection system (Alpha Omega MSD 3.22) was used. Spike times, stimulus events, and behavioral events were digitized and recorded by computer. The spike times were corrected for the spike detection delay. The stimulus events refer to the time when the vertical scan of the display monitor reached the average position of the receptive fields.

Cells in area V2 were recorded either in the lunate sulcus after passing through V1 and the white matter, or in the lip of the post-lunate gyrus. The assignment of cells to areas is based on location of tracks, depth of recording, and physiological criteria (topography and size of receptive fields). The eccentricities of the receptive fields ranged from 0.25 to 7.3 deg (median 2.2 deg). After isolating a cell we first characterized its selectivity for color, bar size, and orientation, and mapped its receptive field (Zhou et al., 2000). Next, border ownership selectivity was determined by a standard test using the edge of a square, and square sizes of 3 and 8 deg and both contrast polarities (Qiu and von der Heydt, 2005). If a cell was color selective, the preferred color and a 28 cd/m² gray were used for figure and background colors, otherwise white (93 cd/m²) and gray (28 cd/m²). The color of the blank screen shown between trials was intermediate between figure and background colors. The same color was used for the frame of the circular window in the following tests. Subsequently the various persistence tests were carried out as time permitted. Cells that showed no border ownership selectivity in the preliminary standard test (ANOVA, p ≥ 0.05) were generally not tested further.

Data analysis

A total of 190 cells were tested in our experiments (116 from monkey TH and 74 from monkey JA – see Supplemental Material for discussion of visual areas). Border ownership selectivity of each neuron was assessed by performing a 2-factor ANOVA on the square root–transformed spike counts, the factors being side of figure and contrast polarity. In the main experiment (Figure 2) spikes from 100 – 500 ms after figure onset were counted for the figure phase, and spikes from 200 – 1000 ms after the onset of the ambiguous edge were counted for the memory phase. In the 3-dot experiment (Figure 5) in which no figures were presented, the spike counts from the preliminary standard border ownership test were used.

A border ownership modulation index was defined as M = (a−b)/(a+b), where a and b are the squares of the mean square root–transformed spike counts for preferred and non-preferred conditions. For each cell, two indices, corresponding to figure phase and memory phase, were calculated (Figure 2B).

For the time course plots (Fig. 2–Fig 7) we computed a weighted average of the peristimulus time histograms (2 ms bin width) of the single neurons, weighting each neuron with the inverse of the standard deviation of residuals obtained from the ANOVA for the figure phase. Thus, neurons that showed small variation between the responses to repeated presentations of the same stimulus conditions were given a greater weight than neurons that showed large variation. (Computing the average with uniform weighting produced very similar results.) Only border ownership selective cells (p ≤ 0.05) were included in the average. The resulting averaged firing rates were smoothed with a Gaussian kernel of σ = 20 ms, except for Figure 1, where σ = 100 ms, and Fig. 3B and Fig 5, where σ = 40 ms.

To describe the time course of the border ownership signal we calculated fits to the population average (2 ms bin width) using multiphase least-squares approximation. Figure phases were fit with a sum of two exponentials with independent time constants, amplitudes, and asymptotes. Ambiguous edge phases were fit with a single exponential with zero asymptote. The fit for the ambiguous edge condition shown in Figure 2A, for example, consisted of a concatenation of three phases: (1) a zero line; (2) a sum of two exponentials with independent time constants, amplitudes, and asymptotes; and (3) an exponential with zero asymptote. For the figure-opposite condition (Figure 2A), phase (3) was again a sum of two exponentials with independent time constants, amplitudes, and asymptotes. The time points of transition were additional free parameters. Data from the two conditions were fit simultaneously and the fit was constrained by requiring the function to be the same for both conditions in phases (1) and (2), which represent the initial responses to identical stimuli. Thus, the transition point from phase 2 to phase 3 was also the same for both conditions. The fit for the flicker experiment (Figure 2B) was the same as described above for the ambiguous edge condition, except that the second phase of the flickering condition was a single exponential (most of which appears like a horizontal line). For both the flicker and non-flickering condition the time point of the transition between phases (2) and (3) was set to be the same as in the main experiment (Figure 2A). The border ownership signals resulting from two cycles of figure presentation (Figure 4) were fit similarly as described above, but with five phases. Again, sums of two exponentials were used for figure phases, and single exponentials for ambiguous edge phases.

For the stereo experiment, the fit was similar (sum of 2 exponentials for the stereo dot phase and a single exponential for the ambiguous edge phase) except that the initial phase in which an ambiguous edge was presented had a constant as a free parameter. We noticed that there was a small negative border-ownership signal during this initial phase and it is likely due to the randomization method we used for presenting stimuli (see Supplemental Material for further discussion).