The role of spatiotemporal edges in visibility and visual masking

Abstract

What parts of a visual stimulus produce the greatest neural signal? Previous studies have explored this question and found that the onset of a stimulus's edge is what excites early visual neurons most strongly. The role of inhibition at the edges of stimuli has remained less clear, however, and the importance of neural responses associated with the termination of stimuli has only recently been examined. Understanding all of these spatiotemporal parameters (the excitation and inhibition evoked by the stimulus's onset and termination, as well as its spatial edges) is crucial if we are to develop a general principle concerning the relationship between neural signals and the parts of the stimulus that generate them. Here, we use visual masking illusions to explore this issue, in combination with human psychophysics, awake behaving primate neurophysiology in the lateral geniculate nucleus of the thalamus, and optical recording in the primary visual cortex of anesthetized monkeys. The edges of the stimulus, rather than its interior, generate the strongest excitatory and inhibitory responses both perceptually and physiologically. These edges can be imaged directly by using optical recording techniques. Excitation and inhibition are moreover most powerful when the stimulus turns both on and off (what might be thought of as the stimulus's temporal edges). We thus conclude that there is a general principle that relates the generation of neural signals (excitatory and inhibitory) to the spatiotemporal edges of stimuli in the early visual system.

This is a study of the neural and perceptual responses to visual spatiotemporal edges. It is well known that in a lateral inhibitory network such as the retina, the lateral geniculate nucleus of the thalamus (LGN), or the primary visual cortex (area V-1), the spatial edges of stimuli excite neurons strongly (1–6), whereas the interiors of stimuli evoke relatively little response. This fact introduces a paradox, however: despite the fact that early visual neurons respond most strongly to the edges of a solid object (such as a rectangle), the interior of the object appears perceptually to be filled-in. This suggests the presence of a mechanism that produces the perception of the filled-in interior of an object by using responses generated at spatial edges (7). If this is true, then it may be possible to create a stimulus that can excite early visual neurons in a fashion that does not sustain the filling-in computation. One way to do this would be to display the stimulus for a short duration; long enough for early visual neurons to respond, but too short to provide meaningful information for the subsequent filling-in computation. From this concept, we predicted the following visual illusion: a stimulus that flickers slowly (i.e., with long duration) appears to be filled in, but if it flickers fast enough, the filling-in effect is defeated to some extent, and the edges of the object are more salient than its interior. This “Unfilled Flicker” illusion can be viewed on the worldwide web at http://cortex.med.harvard.edu/∼macknik.

We previously observed that both the onset-discharge and the termination-discharge of a stimulus are important for the stimulus's visibility (8). This is made evident when we consider that an image that is stabilized on the retina disappears after a short period (9–13). Edges must therefore be renewed in normal vision by eye movements, stimulus motion across visual space, or stimulus temporal modulation (i.e., flicker). It is not yet clear, however, which parts of the stimulus's lifetime are most effective in generating inhibitory signals. Here, we probe the role of inhibition at spatiotemporal edges using a combination of psychophysics and electrophysiology. We moreover used an optical imaging technique (14, 15) to observe activity-correlated signals derived by spatial edges on the surface of the primary visual cortex.

Visual masking is a phenomenon in which an otherwise visible stimulus, called a target, is rendered invisible (or less visible) by the suppressive action of a different stimulus, called the mask. Targets and masks can come in any number of shapes and colors, and here we have used oriented bars for both targets and masks. As we will discuss in detail, the temporal sequence of targets and masks is critically important in determining the target's resultant level of visibility. By manipulating the distance between targets and masks, as well as their relative size, we measured the excitatory and suppressive strengths of the stimuli's edges as opposed to their interiors. By manipulating the duration and temporal overlap of targets and masks, we measured the strength of excitation and suppression as a function of stimulus lifetime.

Materials and Methods

Psychophysical Stimuli.

Psychophysical stimuli and procedures (approved by the Harvard Medical School Committee on Human Studies, docket numbers D-082994-1 and X-62896-2) have been described previously (8).

Electrophysiological Stimuli.

Physiological stimuli were presented on a NEC 5FG monitor at a refresh rate of 100 Hz. The monitor subtended 59° by 40° at a viewing distance of 28 cm. The dimensions of the target (a single oriented black or white bar on a background with opposite contrast) were optimized for each cell. Masks were given the same characteristics as the targets and flanked, but never spatially overlapped, the targets.

Electrophysiological Procedure.

Standard electrophysiological techniques for recording from awake behaving primates and for minimizing the monkeys' discomfort were used (6, 8, 16). The Harvard Medical Area Standing Committee On Animals (protocol no. 02078) approved all electrophysiological experimentation.

Optical Recording Stimuli.

Stimuli were displayed at 100% contrast on a Mitsubishi Diamond Scan monitor. White stimuli on a black background were used in the images shown, and black on white stimuli generated similar images.

Optical Recording Procedures.

Optical imaging and electrophysiology techniques for recording from anesthetized paralyzed animals were conducted by using standard methods described previously (15, 17). Our images were generated in three anesthetized rhesus monkeys by viewing the reflectance of 720 nm light (the light source that was powered by a DC-regulated power supply) from the cortex with a charge-coupled device camera, which collected images at a rate of 32 Hz. Images were sampled using a custom 8-bit analog video system that had a focal length of 50 mm. We stimulated the visual field with each stimulus and recorded 20 s of cortical intrinsic signal (changes in reflectance from the cortex). We then averaged the 640 images that were collected (20 s × 32 Hz) and subtracted the average of 640 other frames that were recorded while the brain was unstimulated. Each image had a resolution of 512 × 480 pixels and was approximately 1.2 cm across on the cortical surface. The Duke University Institutional Animal Care and Use Committee approved all optical recording experiments, protocol no. A471-97-10R2.

Optical Image Processing.

Images shown here have been processed with matlab using standard techniques (14, 15, 17–20). Specifically, they have been cropped to a size of about 1 cm², smoothed with a Gaussian filter having a standard deviation of 4 pixels, and their look-up tables have been normalized and equalized.

Results

The Effect of Distance on Visual Masking.

Werner (21) showed psychophysically that a mask's inhibitory effect on a visual target depends critically on the spatial separation between the mask and the target. This fall off in suppression with distance has not (as far as we know) been measured physiologically. To address this, we recorded from 26 neurons in the LGN of the awake rhesus monkey (Macaca mulatta), while presenting an illusion in which a flickering target (a white or black bar of 60 ms duration) is rendered invisible (to human observers) by a mask (two bars of 110 ms duration that flank the target to either side). The mask flickered in counterphase alternation to the target (8). (This illusion, “Standing Wave of Invisibility” can be viewed on the worldwide web at http://cortex.med.harvard.edu/∼macknik.) By moving the mask away from the target, the target becomes more visible, and responses in the LGN show accordingly that physiological inhibition from the mask decreases as its distance from the target increases (Fig. 1).

Figure 1.

Responses from an on-center type-3 LGN neuron (41, 42) to the Standing Wave of Invisibility, the timing of which is depicted in the colored bars at the top of the figure. Pictures on the right represent the stimulus configuration on the monkey's display (cross near the top of each screen represents the fixation point); the mask distance from the target varied from a distance of zero in the top histogram to 8.57° of visual angle in the bottom. Stimulus size, position, and sign of stimulus varied depending on the optimal parameters of each cell's receptive field. Each histogram represents separately the results of increasing the mask's distance (as drawn in the Inset). Black traces represent the target-only condition for this cell (all black traces are identical). Pink traces represent the response to the mask alone at each distance, and the blue traces in each histogram represent the firing of the cell to both the target and mask presented cyclically. As is evident in the blue traces, increasing the mask distance decreases its inhibitory effect on the target. The distance of the mask from the target for each cell was recalculated depending on its eccentricity. Mask distances thus tested for a foveal cell would have been at 0°, 0.15°, 0.25°, and 0.35° of visual angle, and for all other receptive-field eccentricities, mask distances were computed based an approximate phase-one exponential decay of photoreceptor density with retinal eccentricity (43, 44). Notice that this neuron shows some response to the mask although the distance between the mask and target was 8.57° (much larger than the extent of the receptive-field). This long-range effect occurred in 16 cells (62%). For all but 1 of these 16 cells, the cell type was on-center, indicating that the long-range effect was probably due to light scatter within the eye.

Psychophysical studies of spatial edges.

The spatial edges of stimuli produce the strongest excitatory neural signals (2, 4–6). Ratliff (22) moreover suggested that physiological inhibition should also be strongest at the edge of a stimulus, and several perceptual studies have suggested that inhibition is stronger near the edge of a mask rather than within the mask's interior, at least when the mask and the target are presented simultaneously (3, 23–25). Here, we expand on these experiments by testing the suppressive effect of a mask stimulus that varies in size and onset time relative to the onset of the target, while using high-contrast suprathreshold stimuli in a two-alternative forced choice discrimination task. Subjects were required to determine the taller of two target bars of 30 ms duration, presented at various stimulus onset asynchronies (SOAs) with respect to spatially overlapping bars (masks) of 50 ms duration and various sizes (Fig. 2). Because the masks completely overlapped the target in every condition, any change in the magnitude of visual masking as a function of mask size was because of the distance between the mask's edge and the target (Fig. 2, Inset). As the mask increases in size, so does the subject's ability to discriminate the length of the two targets. That is, as the edges of the mask become more distant from the target, visual masking decreases, and counterintuitively, thinner masks have the larger effect, in congruence with earlier reports (3). Thus, the spatial edges of the mask convey greater inhibition than the mask's interior. Masking was furthermore strongest when the mask turned off just before the target turned on (forward masking) and when the mask turned off about 110 ms after the target turned off (backward masking), in agreement with masking studies using spatially nonoverlapping stimuli (8). [Lateral inhibition radiating from the mask can be seen directly by viewing the Stoper– Mansfield (26) illusion on the worldwide web at http://cortex.med.harvard.edu/∼macknik.]

Figure 2.

Human psychophysical length-discrimination measurements of visual masking effects from 23 human subjects using overlapping opaque masks of varied size (the mask's edge distance from the target's edge was 0°, 0.5°, 1°, 2°, or 4° as indicated in the Inset on the Right). The subject's task was to fixate on the central black dot and choose the longer target (Right or Left). Targets were black bars presented for 30 ms in duration, and masks were also black and presented for 50 ms: the subject's task was to fixate on the central black dot and choose the longer target (Right or Left). Targets turned on at time 0 ms, and masks were presented at various onset asynchronies so that they came on before, simultaneous to, or after the target in 20-ms steps. Stimulus onset asynchronies (SOAs) to the left of zero indicate forward masking conditions, and SOAs that are greater than zero indicate backward masking. Miniature gray markers with dotted connecting lines represent conditions during which the target and mask overlapped in time, and so the target was partially or completely hidden by the mask. The targets were 0.5° wide and had varied heights (5.5°, 5.0°, or 4.5°) and were placed 3° from the 0.2°-wide circular fixation dot in the center of the screen. The mask was a bar 6° tall with varied widths, spatially overlapped, and centered over each target. There were 540 various types of trials (2 possible choices × 2 differently sized target sets to foil local cue discrimination strategies × 5 various overlapping mask sizes × 27 stimulus onset asynchronies). Each condition was presented in random order five times to each subject, over a period of 2 days, for a total of 62,100 trials (summed over all 23 subjects).

Optical Images of Edges in the Primary Visual Cortex.

If the edges of stimuli generate the strongest signals, then an optical image of the cortex (14, 15, 27), generated with a flickering bar stimulus, should reveal stronger activity at the bar's edge than within its interior. When we flickered a thin bar (with 50 ms duration and 100 ms interstimulus interval), we obtained a single stripe of activation in area V-1 of anesthetized rhesus monkeys (Fig. 3a). When we widened the bar, the optical signal split into two separate stripes of activity corresponding to the edges of the stimulus (Fig. 3b). This confirms that the neural signals that represent a stimulus in the early visual system are strongest at the spatial edges of the stimulus.

Figure 3.

The optical image of a flickering bar. (a) Image of the area V-1 cortical intrinsic signal generated by a flickering bar 50 ms on and 100 ms off, that was 0.13° wide with an orientation of 132°. The patch is 1 cm² and was approximately 10–12° below and to the left of the foveal presentation and subtended about 4° of visual angle (as measured with microelectrode penetrations at each edge of the image), at the anterior-medial border of the operculum. The vertical meridian is parallel to the lower edge of this image; the fovea is to the right. (b) Image of the intrinsic signal generated by a flickering bar 0.64° wide in the same piece of cortex (the center of the bar was also shifted here approximately 0.29° away from the fovea). Notice that the widened bar has shifted in position and split into two edges.

The Perception of Visual Masking at the Temporal Edges of Stimuli.

Responses to a stimulus in primate area V-1 are strongest when the stimulus turns both on and off (8). But it remains unclear how the strength of inhibition varies with stimulus lifetime. Is inhibition produced most strongly at the onset and termination of a mask stimulus, is inhibition instead constant over the lifetime of the mask, or does inhibition build up over time? Earlier studies examined this question by testing the brightness detection threshold of a target while it was suppressed visually with a spatially overlapping conditioning field (or mask), that sometimes had a different brightness or color (3, 23–25). Because the mask overlapped spatially the target, adaptation effects, rather than lateral inhibition signals, could have played a role in these results. To better isolate the effects of lateral inhibition from intrinsic adaptation effects, we conducted a psychophysical length-discrimination experiment in humans, similar to that shown in Fig. 2. Here, two 10 ms duration targets were presented at various temporal onset asynchronies with respect to nonoverlapping masks of varied duration (Fig. 4). The results show that target length is more difficult to discriminate when the targets are presented simultaneously to either the onset or the termination of the mask, rather than to the midlife of the mask, suggesting that the suppression generated by the mask occurs at the temporal edges of the mask.

Figure 4.

Human psychophysical length-discrimination measurements of visual masking effects from 11 human subjects using nonoverlapping masks of varied duration (100, 300, or 500 ms). (a) The appearance of the stimuli during the course of a psychophysical trial. (b) SOA here represents the period between the onset of the mask and the onset of the target (and so has the opposite meaning than in Fig. 2). Masks (two 6° tall bars with a width of 0.5° flanking each side of each target) appear at time 0, and targets can appear earlier (backward masking), simultaneously, or later (forward masking), in 50-ms steps. Targets were black and presented for 10 ms duration, and masks were flanking black bars that abutted the target. Notice that target visibility is most greatly affected when the masks turn on and off.

Discussion

The question we have addressed here is: “What parts of a visual stimulus produce the greatest neural signal?” Previous studies have shown that excitatory signals for a stimulus are localized at its edges (1, 2, 4–6, 28). It has also been suggested that inhibitory signals are strongest at the spatial edges (21–25, 29). We expanded on the previous psychophysical studies concerning inhibition at the edge by carrying out visual masking experiments, perceptually and physiologically, in which we increased the distance of the mask's edge from the target, using both overlapping and nonoverlapping masks (Figs. 1 and 2). In the psychophysical experiments, the masks varied in size and timing with respect to the target. We also directly measured the strength of neural signals (presumably both excitatory and inhibitory) generated at a stimulus's edges by optically recording the responses to flickering targets in primate area V-1 (Fig. 3). Most previous optical imaging studies used sweeping stimuli to generate cortical activity, and so they could not view directly the cortical activation from simple stationary edges. Because the optical images generated here were made with stationary stimuli, they moreover represent the finest resolution pictures of the minimal spread of activity across the surface of the cortex (the cortical point-spread function; refs. 30–32). These images correlate well with similar images we published previously (15).

Having confirmed that the strongest excitatory and inhibitory neural signals occur at the spatial edges of a stimulus, we examined the variation in signal strength over the entire time course of the stimulus. Our previous research with visual masking illusions revealed that excitatory neural signals for target visibility were strongest at the temporal edges, or onset and termination times, of the target (8), but it remained unclear whether inhibitory signals followed the same temporal dynamics as excitatory signals. Crawford (33) found that the visibility of a stimulus was inhibited most completely when a full-field mask turned on, and he moreover showed a slight increase in masking magnitude when the mask turned off. The target inhibition he saw with mask termination was small, perhaps, because the spatial edges of his mask were far (11.5°) from the target. Furthermore, he used only one duration for his mask stimulus, so one cannot be sure that the inhibition at the mask's termination was truly caused by the termination of the mask, or whether it instead built up subsequent to the onset of the mask. Battersby and Wagman (3) controlled for some of these concerns by extending Crawford's experiment and adding a second condition that had a mask stimulus of longer duration, and found that, for each masking stimulus duration, the termination point of the stimulus generated more inhibition of the target than did mask midlife. Both of these studies, however, used spatially overlapping target and masks, and so the suppression caused by the mask could have been mediated by either extracellular lateral inhibition circuits (a neural network) or intracellular adaptation mechanisms (an intrinsic cell property). To address this issue, we performed the perceptual experiment shown in Fig. 4, in which we inhibited a target with masks of varied durations that did not spatially overlap the targets. The results show that masks suppress targets most strongly at the mask's onset and termination, possibly through a lateral inhibition circuit.

Our interests have also been to determine, within a spike train, those elements most significant to representing a stimulus. Many studies have suggested that the transient bursty firing generated by the onset of a stimulus is most important for neural information processing (6, 34–36). We previously showed that transient responses derived by both the onset and termination of a stimulus in area V-1 cells are most responsible for that stimulus's visibility; when these transient events were suppressed, the stimulus became invisible (8). We have also shown that V-1 neurons produce bursty firing after microsaccades, thereby preventing adaptation and regenerating visibility (13). The perceptual experiments in Figs. 2 and 4 show that the timing of visual masking correlates with transient responses generated by the mask's onset and termination. This supports the concept that transient bursty firing produces strong neural signaling.

These results bring us to the conclusion that a stimulus's spatiotemporal edges evoke the strongest neural signals (both excitatory and inhibitory). This is accomplished by generating transient activity in response to both the stimulus's onset and termination (its temporal edges) in neurons with receptive fields lying at the stimulus's spatial edges. The excitatory and inhibitory interplay that occurs between early visual neurons during visual masking also helps us to understand the seemingly mysterious spatiotemporal dynamics of visual masking.

Although these results seem to go far in answering what parts of the stimulus produce the strongest signals, there may be cognitive factors that also play a role in stimulus visibility. Several recently discovered contextual and shading effects may or may not be explained by the spatiotemporal nature of edges (37, 38). Attention has also been shown to play a role in illusions of invisibility (39, 40).

Abbreviations

LGN

lateral geniculate nucleus of the thalamus

SOAs

stimulus onset asynchronies