Remapped visual masking

Abstract

Cells in saccade control areas respond if a saccade is about to bring a target into their receptive fields (J. R. Duhamel, C. L. Colby, & M. R. Goldberg, 1992). This remapping process should shift the retinal location from which attention selects target information (P. Cavanagh, A. R. Hunt, S. R. Afraz, & M. Rolfs, 2010). We examined this attention shift in a masking experiment where target and mask were presented just before an eye movement. In a control condition with no eye movement, masks interfered with target identification only when they spatially overlapped. Just before a saccade, however, a mask overlapping the target had less effect, whereas a mask placed in the target’s remapped location was quite effective. The remapped location is the retinal position the target will have after the upcoming saccade, which corresponds to neither the retinotopic nor spatiotopic location of the target before the saccade. Both effects are consistent with a pre-saccadic shift in the location from which attention selects target information. In the case of retinally aligned target and mask, the shift of attention away from the target location reduces masking, but when the mask appears at the target’s remapped location, attention’s shift to that location brings in mask information that interferes with the target identification.

Introduction

The visual world is displaced on the retina several times each second as a consequence of saccadic eye movements. In spite of these frequent, abrupt shifts in the retinotopic locations of all the objects in our field of view, the flow of information coming in from the visual world is not noticeably perturbed, nor do we seem to have any trouble keeping track of objects from one fixation to the next. A plausible explanation of how the visual system compensates for eye movements has emerged from studies of the behavior of individual cells and populations of cells in saccade and attention control areas around the time of an eye movement. These results suggest that the visual system is able to use information about an upcoming eye movement to prepare for the expected consequences of that movement (e.g., Wurtz, 2008). In this paper, we show, using visual masking, that the location from which target information is accrued also shifts prior to the eye movement, suggesting that attention predicts where the target will be following the eye movement and starts picking up information from that location even before the eyes move.

Corollary discharge (or efference copy) refers to the idea that a copy of each oculomotor command is sent to the visual system to help it to predict and counteract the consequences of eye movements (e.g., Sperry, 1950; von Holst & Mittelstaedt, 1950). In support of this idea, Duhamel, Colby, and Goldberg (1992) found that almost all the neurons in lateral intraparietal cortex (LIP) begin to fire in response to a stimulus that will be brought into their respective fields by an eye movement, even if the stimulus is extinguished before the eyes arrive. The predictive response of these neurons can begin as early as 100 ms before the saccade and tends to peak at the onset of the saccade, which is still much earlier than the cell would be able to respond if the stimulus simply appeared in the cell’s receptive field following the eye movement. This pattern of predictive firing, known as saccadic remapping, also occurs in other areas of visual cortex (Nakamura & Colby, 2002) and in FEF (Sommer & Wurtz, 2002, 2006). A pathway that seems to mediate saccadic remapping connects the SC to the FEF via the medial dorsal nucleus (MD) of the thalamus. When MD relay neurons are deactivated, information about spatial locations appears to be disrupted by eye movements (Sommer & Wurtz, 2002) and remapping responses in FEF are attenuated (Sommer & Wurtz, 2006). Taken together, these results point to a process of advanced preparation for the retinal consequences of a saccade brought on by the motor command to move the eyes.

Remapping is predominantly associated with areas LIP and FEF, both of which have been linked with saccade and attention control (Gottlieb, Kusunoki, & Goldberg, 1998; Schall & Hanes, 1993). The activity peaks in these areas seem to specify not only the destination location if a saccade were to be executed to a given target but also the location where attentional benefits will be seen for that target at earlier levels of the visual cortex (e.g., Keller, Gandhi, & Weir, 1996; Moore & Armstrong, 2003; Moore & Fallah, 2004; see Awh, Armstrong, & Moore, 2006 for a review). In other words, the activity peak specifies the location, in retinal coordinates, from which visual features need to be picked up to identify the target, thereby conferring attentional benefits to that location. As objects in the visual world shift on the retina and remapping processes shift the corresponding activity peak to track its predicted new location, the retinotopic location from which the object’s information is being accrued should also shift to the location the object will have after an eye movement (Cavanagh, Hunt, Afraz, & Rolfs, 2010). This shift in the location of uptake should begin to occur shortly before the eye movement. Supporting this idea, attention can be maintained in spatial coordinates across eye movements (e.g., Golomb, Chun, & Mazer, 2009; Posner & Cohen, 1984), and attentional benefits are observed at the predicted retinotopic location of a future saccade goal (Rolfs, Jonikaitis, Deubel, & Cavanagh, in press). It is this potential pre-saccadic shift of attention that we will examine with masking effects prior to the eye movement.

We used backward visual masking to explore remapping’s effects on visual processing. A brief target is followed by the mask and then a saccade. Typically, backward masking is robust when the mask falls on the same retinotopic location as the target (Irwin, Brown, & Sun, 1988; Sun & Irwin, 1987), that is, even if a saccade occurs between the target and the mask, the masking is still seen at the retinotopic location, although there are also reports of spatiotopic masking (Deubel, Bridgeman, & Schneider, 1996; Irwin 1992; MacRae, Butler, & Popiel, 1987; White, 1976; but see Jonides, Irwin, & Yantis, 1983 for a cautionary note about monitor persistence). In a recent study, De Pisapia, Kaunitz, and Melcher (2010) showed what they refer to as target “unmasking” when a saccade intervened between a target and a mask shown at the same spatial location. A reduction in masking, unmasking, is expected if masking is retinotopic, because after the saccade, the mask would fall on a different part of the retina than the target. So their evidence of unmasking is again evidence for retinotopic masking. They also find that a post-saccadic mask that is spatially (but not retinally) aligned with a pre-saccadic target reduces accuracy more than when it is spatially (and retinally) misaligned. They present this result as evidence for spatiotopic masking. Based on this and the previous results, both retinotopic and spatiotopic spatial coding may be involved in backward masking when a saccade intervenes between the target and mask. In our experiment, however, both target and mask precede the saccade. De Pisapia et al. (2010, see Figure 3D) found that saccade-related unmasking begins to emerge even when the mask appears before the eye movement, suggesting that a simple offset in retinal alignment may not be the whole story for unmasking. They appeal to a perceptual misalignment between the target and mask that they measure independently. In our display, perceptual misalignments are at a minimum and we will look instead to the movements of attention prior to the saccade to understand the unmasking effect.

Figure 3.

Accuracy to discriminate the target orientation in the (left) upper and (right) lower locations, when no saccade was executed and when a saccade followed the mask within 100 ms. There is a significant interaction of masking with the saccade in both the target-up and target-down conditions. The sources of these two interactions are highlighted. Error bars represent the standard error of the mean.

In our experiment, we show both target and mask before the saccade to probe for the effect of remapping on masking that occurs even before the saccade begins. We anticipate that when activity in saccade and attention centers remaps, new downward projections that carry attentional benefits will be activated (Awh et al., 2006), thereby enabling information selection from the locations in early retinotopic cortices that correspond to the expected new location of the target. As a result, starting about 100 ms before the eye movement, the location from which attention selects target information should start to shift to the target’s expected post-saccadic retinal location. We are interested in two specific effects. First, a mask presented prior to the saccade but in the same spatial location as the target may interfere less with target processing, because the locus of information accrual will have shifted away from the actual location of the mask. Shifting attention away from a mask reduces its effectiveness (Ramachandran & Cobb, 1995). By the same token, a mask presented before the saccade, but at the post-saccadic retinal coordinates of the target (that is, its remapped location), should cause interference because the location of information pickup for the target will shift to a position where, prior to the saccade, there is a mask. This process is illustrated in Figure 1. Note that prior to the saccade, the remapped location that we will test corresponds neither to the retinal location of the target, nor to its spatiotopic location (since the eye movement has yet to occur), but to the retinal location the target will have following the saccade.

Figure 1.

Remapping and masking. (a) The observer fixates the green dot and attention begins to accumulate information from the target location. (b) The observer prepares a downward saccade and attention begins shifting the target’s post-saccadic retinotopic location in order to continue accumulating information. In this example, there is nothing at that location prior to the saccade. (c) The target is replaced by the mask, but attention has moved or started to move away. (d) The eyes move to the lower location, bringing the remapped location into line with the target’s expected post-saccadic location in space.

Methods

Four subjects (1 author) were seated individually in a dimly lit room with the head stabilized by a chin rest positioned 37 cm from a 20-inch, 85-Hz monitor (display area is 49.3° × 40.9°). Eye movements of the right eye only were monitored using a 120-Hz Eyelink I monitor (SR Research, Canada). The experiment was programmed using the Psychtoolbox and Eyelink Toolbox for MATLAB (Brainard, 1997; Cornelissen, Peters, & Palmer, 2002; Pelli, 1997). Subjects completed 6 blocks of 128 trials. Each block began with a 9-point calibration sequence. Each trial began with a fixation point 10° above the center of the screen (10.5° from the upper edge of the monitor) and two 2.3° boxes, one 10° to the right of the upper fixation and the other 20° below the first box (see Figure 2). The center of each box was 14.7° from the right edge of the screen. The color of the fixation point indicated whether the subjects should make a saccade on the upcoming trial (green) or stay fixated (red). The green and red trials were presented in runs of 16 trials (such that each block consisted of 16 green trials, followed by 16 red trials, and so on). To begin each trial, subjects fixated the fixation point and pressed the space bar. If a stable fixation was detected by the eye monitor, the trial would commence. After a random interval of 0–500 ms, a beep sounded at the same time as a second fixation point appeared 10° below the center of the screen. Subjects were instructed to shift their eyes to the lower fixation point as quickly as possible when the fixation point was green, but when it was red to remain fixated on the upper fixation point. After an interval determined using the method described below, the target, a 1.3° square with one of the four randomly selected sides missing to form a “C”, was presented inside either the upper or the lower square. The target remained for 35 ms. On mask trials, the mask appeared when the target was removed. To create the mask, the peripheral box in which the mask was to appear was divided into a 5 × 5 grid, and each segment of the grid was randomly selected to be filled either black or white. A different mask was generated on each trial. The mask remained until the subject indicated the orientation of the target C, using the 2, 4, 6, and 8 keys on the number pad to indicate down, left, right, and up, respectively. The target appeared in the upper box on half the trials and in the lower box on the other half of trials. A mask was presented on half the trials, and it was in the same location as the target on half of those trials, and on the other half, it appeared in the box not occupied by the target.

Figure 2.

The methods used in the experiment. (a) The events in a trial. The onset of the lower fixation (panel 2) is the signal to execute a saccade. The target (panel 3) was presented for 35 ms and the mask immediately followed (panel 4), usually appearing prior to the saccade (see Methods section for more details on how this was accomplished). (b) The timing of the events on a typical trial. Mean duration from the signal to saccade onset was 192 ms. If the target onset and offset occurred between 0 and 100 ms before the onset of the saccade (the gray area here), it was classified as a “remapping” trial. The target and mask were both equally likely to appear inside the upper and lower boxes.

The interval between the lower fixation point onset and the target onset was manipulated to maximize the number of trials where both the target and the mask appeared in the interval before the eye movement began. One extra trial was run at the beginning of each block, in which the fixation point-to-target onset interval was 176 ms (this first trial was excluded from analysis). After this trial, this interval was based on the mean saccadic latency for all the preceding trials (where saccadic latency is the time from the onset of the lower fixation point to the onset of the eye movement); the interval between the fixation point shift and the target onset was the current mean saccadic latency minus 118 ms. On the no-saccade trials, the onset time of the target was based on the mean saccade latency at the end of the last saccade trial. For an illustration of the timing of events on a typical trial, see Figure 2b.

Trials were excluded from analysis if the saccade landed further than 5° from the saccade target in either the vertical or horizontal direction (4.5% of trials) or if a saccade was executed on no-saccade trials (0.2% of trials). A target–mask pair was classified as “pre-saccadic” if both the target and the mask came on within the 100-ms interval before the onset of the saccade. Of the included saccade trials, 41% met these criteria. The target came on earlier than 100 ms before the onset of the saccade on 51% of trials. On the remaining 8% of trials, the target and/or mask followed the onset of the eye movement, and these trials were not analyzed because there were too few data points to produce a reliable pattern.

Results

Figure 3 shows the accuracy results for trials with no saccade and those when the saccade followed the mask onset within 100 ms. These were analyzed with a repeated-measures ANOVA. Figure 4 shows a summary graph of the data transformed into percent masking effect. Both these analyses show two important effects of the upcoming saccade on masking: reduction for retinotopically aligned target and mask and emergence of masking for a mask at the target’s remapped location (its future retinotopic location).

Figure 4.

Masking effect in the no-saccade condition and in three time periods around saccade onset for the four possible configurations of target and mask. Error bars represent the standard error of the mean.

In the ANOVA of the accuracy data, all three of the factors were significant: saccade (none vs. pre-saccade, [F(1, 3) = 11.28, p < 0.05]), target location (up or down, [F(1, 3) = 21.06, p < 0.05]), and mask (up, down, or none, [F(2, 6) = 74.81, p < 0.001]). There were significant interactions of saccade and target location [F(1, 3) = 13.13, p < 0.05] and of target location and mask [F(2, 6) = 48.17, p < 0.001] and a three-way interaction [F(2, 6) = 20.64, p < 0.01]. To understand the source of the three-way interaction, we did separate two-way ANOVAs on the target-up and target-down conditions. These revealed significant interactions of mask and saccade effects for both target positions [target up F(2, 6) = 15.12, p < 0.01; target down F(2, 6) = 11.40, p < 0.01], although the nature of these two interactions was quite different. When the target was in the upper location, there are significant (Tukey’s p < 0.05) differences in accuracy in both the no-saccade [F(2, 6) = 168.40, p < 0.001] and pre-saccade [F(2, 6) = 20.94, p < 0.01] conditions, even though the effect of the mask is much larger in the no-saccade condition, which is the source of this interaction. This result demonstrates that the typical retinotopic masking effect is reduced in the pre-saccade condition, and this is highlighted on the left side of Figure 3. When the target is in the lower location, in contrast, the interaction of masking with the saccade occurs because of a qualitatively different pattern of masking in two saccade conditions. With no saccade, there are significant (Tukey’s p < 0.05) differences between mask down and no mask, and no difference between mask up and no mask, demonstrating that the mask is only effective when it overlaps with the target. When a saccade is about to be executed, there is now a significant difference between mask down and no mask and between mask up and no mask (Tukey’s p < 0.05), but no difference between mask up and mask down. This is highlighted on the right side of Figure 3 by an arrow pointing out the drop in accuracy in the target-down mask-up condition when a saccade is about to be executed.

In the target-down condition in particular, accuracy is reduced when an eye movement is about to be executed, even without a mask present, probably because the dual-task cost of moving the eyes has a bigger impact when the target is slightly further from the fovea. To look at changes in masking effects over and above these changes in the baseline, we subtracted the percent correct with masks from the baseline percent correct without masks for each target location and timing condition. These results are shown in Figure 4, which also includes trials where the target and mask are presented more than 100 ms before the saccade. From this figure, the two interesting results that emerged from the above ANOVA are more apparent. First, the masking effect is reduced before the eye movement (the mean masking effect for overlapping masks and targets is 53.8% in the no-saccade condition and 35.6% in the pre-saccade condition [t(3) = 7.88, p < 0.01]). Second, in the no-saccade condition where the target and mask were not aligned (in any coordinates), the masking effect was, as expected, negligible (4.5% upper target, lower mask; 0.8% lower target, upper mask). However, in the pre-saccade condition (trials with the target onset within 100 ms of the saccade), masking did increase substantially (to 33% [t(3) = 5.39, p < 0.05]) when the target appeared in the lower location and the mask appeared in the upper location; it did not increase (4.3%, t(3) = 1.29) for the other configuration where the target was in the upper location and the mask in the lower location. In other words, the increased masking for non-overlapped target and mask occurred only when the mask was in the remapped location of the target (see Figure 1). This confirms the result that emerged from the three-way interaction in the overall ANOVA. From Figure 4, it can also be seen that the general pattern observed in the pre-saccadic condition begins to emerge earlier than 100 ms before the eye movement.

Discussion

Two striking changes in masking emerged when the target and mask both appeared just before the eye movement. First, there was a reduction in the masking effect relative to the no-saccade condition for overlapping masks and targets, replicating a result recently reported by De Pisapia et al. (2010). Second, when the mask was presented in the location corresponding to the future retinotopic location of the target (i.e., the target in the lower location and mask in the upper location), masking emerged where there had been none. Masking in this configuration is especially interesting because a mask in this location would not correspond to either the spatial or retinal location of the target. It is not possible, therefore, that the masking effect emerged from interactions at a physical level (for example, from properties of the CRT display), or that it emerged from interactions at the level of the retina, or at levels where spatiotopic masking might be mediated (De Pisapia et al., 2010; Deubel et al., 1996).

In the first case with overlapping target and mask, masking is reduced before the saccade, and one interpretation is that remapping moves attention away from the target location, reducing the effect of the mask that followed the target. In other words, shortly after the target is presented, attention begins to select information from its retinotopic location. With the impending eye movement, the location of this information uptake is shifted away from the target’s initial location to the target’s expected postsaccadic location. So when the mask appears, attention has moved or will soon move away, reducing the masking (see Figure 1). However, simple alternative explanations of the reduced masking before an eye movement are plausible. In particular, the sooner the saccade follows the mask presentation, the shorter the mask lasts at the target’s retinotopic location. The effectiveness of a mask is clearly reduced when it occurs during and after a saccade (a spatiotopic mask, e.g., Irwin et al., 1988). In addition, even if the most effective portion of the masking is its onset, it is possible that the effectiveness of the mask onset may have been reduced by the eye movement that immediately followed its presentation, because saccades are associated with a decrease in perceptual sensitivity to a pre-saccadic stimulus (e.g., Volkman, 1962). Another possibility is that pre-saccadic unmasking of targets is related to distortions in the perceived locations of stimuli that appear just before and during an eye movement (e.g., Matin & Pearce, 1965). This interpretation was proposed by De Pisapia et al. (2010), on the basis that the strength of masking in their results was inversely related to the frequency of perceived displacement of the target and mask.

On the other hand, for our second result, the emergence of masking for the non-overlapping stimuli, remapping is the only viable explanation. Masking was seen when the mask was located at the retinotopic position the target would have after the saccade. In this case, remapping shifts the location from which the visual system is accruing target information to a location where a mask is now present (see Figure 5 for an illustration). Note that there is no masking in this configuration without a saccade, and the masking only emerges for a target in the lower position where remapping brings its expected location on top of the mask. In the other non-overlapping configuration, little or no masking emerges when saccades are made because the remapped location above the target is now in blank space. Could our masking and unmasking be related to saccade-induced mislocalization the way De Pisapia et al. (2010) suggest for their stimuli? We do not think so. First, our mask should not be mislocalized, because it remains present after the saccade and up until the response is made. Stimuli with continued presence do not show mislocalization; otherwise, we would experience spatial distortion every time we moved our eyes (Matin & Pearce, 1965). The brief Landolt C target is also unlikely to be mislocalized. Previous research has shown that visible landmarks reduce mislocalization (e.g., Honda, 1999), and the two boxes that contain the targets would provide a stable landmark. The observers who participated in the experiment (including one of the authors) did not report experiencing the target appearing outside of the boxes. The perceived location of peri-saccadic events seem to be a strategic combination of retinal information with reliable information about the features of the environment (O’Regan, 1984), such as the fact that the target always appears inside the boxes. Saccade trials were interleaved with runs of no-saccade trials, and on these trials, it was quite clear that the target only ever appeared inside the two boxes. When the target appeared in the upper box, a downward saccade could conceivably shift its perceived location to the lower box on some trials, given that mislocalization here would be expected to follow the direction of the saccade. This seems unlikely, however, based on the pattern of results we obtained. If the saccade caused mislocalization of the target and brought the perceived location of the target from the upper box to the lower box, masking would be expected to emerge in the target-up mask-down configuration. Instead, we observed masking in the target-down mask-up configuration, making remapping (see Figure 5) the more likely explanation for our results. It is worth noting that our paper differed from De Pisapia et al. in a number of other respects, including the type of masking (they used metacontrast masks while we used noise masks) as well as the timing of target, mask, and saccade events.

Figure 5.

Illustration of the four configurations of the target and mask in the experiment. The green dotted circle indicates the current focus of attention, and the red dotted circle indicates the location to which attention should shift to continue accruing information from the target location after the eye movement. Note that when the target is in the lower location and the mask is in the upper location (b) attention will shift to a location occupied by a mask. This is the condition in which we observed masking just before the eye movement.

Several recent studies have demonstrated the pre-saccadic effects of remapping on measures of visual processing in human observers (e.g., Binda, Cicchina, Burr, & Morrone,2010; Rolfs et al., in press). As one example, we asked subjects to report when they thought their eyes arrived on a target (Hunt & Cavanagh, 2009). Subjects made a saccade to a clock with a rapidly spinning hand and reported what time was on the clock when the eyes arrived. The reported time was 40–60 ms earlier than the actual time the eyes arrived, which is consistent with the pattern reported in studies recording from visual cortex neurons (e.g., Duhamel et al., 1992; Nakamura & Colby, 2002). This suggests our perception of where we are looking is not as accurate as it seems and may be influenced by predictive processing. The present results take this further and demonstrate that the processing of a single target can be disrupted by a mask placed in its anticipated retinotopic location just before an eye movement. These results are consistent with the general conception of remapping as a process that helps shift attention between two retinotopic locations that correspond to the same target object. This predictive process helps maintain a continuous stream of information from a single target object when an eye movement shifts it from one place to another on the retina.