Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Oct 1.
Published in final edited form as: Atten Percept Psychophys. 2015 Oct;77(7):2260–2269. doi: 10.3758/s13414-015-0944-z

Saccadic remapping of object-selective information

Benjamin A Wolfe 1,*, David Whitney 1
PMCID: PMC4607590  NIHMSID: NIHMS700011  PMID: 26069198

Abstract

Saccadic remapping, a presaccadic increase in neural activity when a saccade is about to bring an object into a neuron’s receptive field, may be crucial for our perception of a stable world. Studies of perception and saccadic remapping, like ours, focus on the presaccadic acquisition of information from the saccade target, with no direct reference to underlying physiology. While information is known to be acquired prior to a saccade, it is unclear whether object-selective or feature-specific information is remapped. To test this, we performed a series of psychophysical experiments in which we presented a peripheral, non-foveated face as a presaccadic target. The target face disappeared at saccade onset. After making a saccade to the location of the peripheral target face (which was no longer visible), subjects misperceived the expression of a subsequent, foveally-presented neutral face as being repelled away from the peripheral presaccadic face target. This effect was similar to a sequential shape contrast or negative aftereffect, but required a saccade, as covert attention was not sufficient to generate the illusion. Additional experiments further revealed that inverting the faces disrupted the illusion, suggesting that presaccadic remapping is object-selective and not based on low-level features. Our results demonstrate that saccadic remapping can be an object-selective process, spatially tuned to the target of the saccade and distinct from covert attention in the absence of a saccade.

Keywords: Saccadic remapping, perisaccadic perception, presaccadic attention, object perception, visual stability

Introduction

Visual input across eye movements can be thought of as brief islands of stability in a storm of motion, yet we perceive the world as essentially stable. How we are able to do so has been a longstanding question for vision science (Helmholtz, 1898, trans. 1962). In recent years, saccadic remapping, where some visual neurons respond just before a saccade is about to bring a stimulus into their receptive field, has been suggested as a mechanism by which visual stability across saccades could be achieved. Early neurophysiological descriptions of remapping suggested that cells in lateral interparietal cortex shifted the locations of their receptive fields prior to a saccade (Duhamel, Colby, & Goldberg, 1992), and further work has suggested the corollary discharge (from the saccade planning process) may be key to this receptive field shift (Nakamura & Colby, 2002; Parks & Corballis, 2008; Sommer & Wurtz, 2008; Wurtz, 2008). Recently, it has been suggested that there may be neurophysiological evidence for remapping of visual features (Subramanian & Colby, 2014).

These neurophysiological findings have spurred psychophysical studies of saccadic remapping in the complete absence of direct recordings from neurons (Burr & Morrone, 2011; Cavanagh, Hunt, Afraz, & Rolfs, 2010; Melcher, 2005; 2007; Melcher & Morrone, 2003; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2010). In these studies of perception, the term saccadic remapping usually refers to the presaccadic acquisition of information from the saccade target, with no necessary reference to a particular underlying physiological mechanism. We adopt a similar functional definition; we use the term saccadic remapping in this paper to refer to the perceptual consequences of presaccadic information, completely separate from its neurophysiological foundations.

Psychophysical studies of what is commonly called saccadic remapping can be thought of as suggesting one of two accounts of how remapping might interact with representations of the visual world in the brain. A retinotopic account holds that saccadic remapping is, in fact, the remapping of attention pointers for objects in the world immediately prior to a saccade within a retinotopic map, rather than the remapping of the representations themselves (Born, Ansorge, & Kerzel, 2012; Cavanagh et al., 2010; Collins, Rolfs, Deubel, & Cavanagh, 2009; Rolfs et al., 2010). Another view is that there are spatiotopic representations in visual cortex, representing objects in world centered coordinates independent of their current retinal position, and the representation of the object itself, rather than a pointer to it, is remapped (Burr & Morrone, 2011; Hall & Colby, 2011; Melcher, 2005; 2007; 2009; Melcher & Morrone, 2003).

However, these are not mutually exclusive views; it has been suggested that both maps may exist in visual cortex, with different timeframes and resolution (Burr & Morrone, 2012). However, previous research has focused more on how remapping might occur, relative to these representations of visual space in the brain, rather than what information might be presaccadically acquired. Saccadic remapping has been suggested to acquire only individual features prior to the saccade, including orientation (Melcher, 2007; Zimmermann, Morrone, Fink, & Burr, 2013), motion direction (Fracasso, Caramazza, & Melcher, 2010; Turi & Burr, 2012) and shape (Demeyer, De Graef, Wagemans, & Verfaillie, 2009; Harrison & Bex, 2014; Hayhoe, Lachter, & Feldman, 1991). Coherent object representations might also be remapped, which would facilitate postsaccadic comparisons and continuity of perception across saccades. For example, Melcher (2005), in an experiment on transsaccadic integration, had subjects foveate a face for 5 sec, make a saccade away from the face and, after an 800 ms delay, view a test face for 250 ms, which they perceived differently as a result of the presaccadic foveation (Melcher, 2005). This result suggests that object-specific information can be integrated across multiple viewpoints and that this information can influence postsaccadic perception. However, Melcher’s 2005 work does not speak to what information might be acquired in saccadic remapping, since saccadic remapping only occurs very close to the time of the saccade (Harrison, Retell, Remington, & Mattingley, 2013b; Nakamura & Colby, 2002; Parks & Corballis, 2008; Rolfs et al., 2010; Sommer & Wurtz, 2008; Wurtz, 2008), and the adaptor was presented foveally prior to the saccade. More recent work suggests that combinations of features, as coherent objects, may be remapped (Melcher, 2007), but critically, these experiments relied on an initial foveal adaptor as well as a peripheral test after the saccade. Simply put, there has been no direct test of what information is remapped prior to a saccade using complex stimuli in the periphery as the saccade target. Related research has shown that saccades can reduce crowding with a variety of stimuli, including letters (Harrison, Mattingley, & Remington, 2013a) and even stimuli as complex as faces (Wolfe & Whitney, 2014), although this may be accounted for with attention rather than remapping (van Koningsbruggen & Buonocore, 2013). While previous research suggests that detailed information is captured around the time of a saccade, it does not indicate if saccadic remapping is an object-selective process.

Based on these studies, saccadic remapping might be an object-selective process; if so, there should be perceptual consequences of object-selective remapping with sufficiently complex stimuli. To approach this question, we used faces, which are natural, commonly encountered, and involve high-level object processing. We tested for a saccade-contingent sequential aftereffect in face perception, such as a negative face aftereffect or a sequential shape contrast effect. In addition, we tested whether this effect required holistic processing of faces, or if the sequential effect was based on a feature or features within our face stimuli. If saccadic remapping is an object-selective process, we might expect that perception of objects viewed postsaccadically would be modulated by information acquired prior to the saccade in an object-specific manner.

Methods

Display Setup

All experiments were performed using Matlab 2010a (Mathworks; Natick, MA), the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) and the Eyelink Toolbox (Cornelissen, Peters, & Palmer, 2002) on a Mac Mini (Apple; Cupertino, CA). Stimuli were displayed on a 47 cm Samsung cathode ray tube at a distance of 57 cm from the subject. For all experiments, monitor refresh was 100 Hz at a resolution of 1024 × 768.

Subjects

A total of 6 subjects participated in the experiments; the same 5 subjects participated in Experiments 1b-d and 2a-b. All subjects in all experiments had normal or corrected to normal vision. All subjects with the exception of one of the authors [BW] were naïve to the purposes of the study and provided written informed consent as required by the IRB at the University of California, Berkeley in accordance with the Declaration of Helsinki. All subjects were trained psychophysical observers and were accustomed to eyetracking experiments prior to data collection.

Stimuli

Stimuli were morphed emotional faces between the emotional states of happy, sad and angry, as originally used by Yamanashi Leib and colleagues (Yamanashi Leib et al., 2012). The morphs were generated by starting with two images of the same individual expressing either happy or sad emotional expressions, selected from the Ekman gallery (Ekman & Friesen, 1976). We then linearly morphed the faces to produce 48 morphs between each pair of basic emotions (i.e., 48 morphs between happy and sad). Morphs were created using Morph 2.5 (Gryphon Software, San Diego, CA), morphed between happy (morph 0) and sad (morph 49) with 48 intervening computer-generated morphs between them. Morph number 25 should be thought of as a neutral center-point within the morph space. In all experiments, the face stimuli, regardless of where they were presented on the display, subtended 4.26° high by 3.36° wide.

Trial Sequence

Subjects’ eye movements were recorded during all experiments, and they were not permitted to move their eyes from the fixation point (0.25 deg visual angle) except when instructed. All stimuli were presented gaze-contingently with a 0.5° threshold; if the point of gaze deviated by more than 0.5°, the stimuli were removed from the screen until gaze returned to the fixation point for a minimum of 500ms. Trials in all experiments began with an initial 1400 ms fixation period. In all experiments, the fixation dot was presented at the center of the screen and its Y-position varied randomly on a trial-by-trial basis within a 10 deg range from the center of the display.

The design of Experiment 1a (Figure 1a) was as follows: each trial started with a fixation period (1400 ms), followed by a peripheral adaptor (either a happy or sad face— morph 0 or 49, 15° to the left or right of fixation; Figure 1a) for 250 ms, and then a saccade cue (a 0.25° red dot 15° to the left or right of fixation, centered on the adaptor face). The adaptor face remained on the display while the saccade cue was presented. The adaptor and cue were both removed when the subject initiated the saccade, defined by the point of gaze deviating from fixation by more than 0.5°. Once the saccade was initiated, there was a 100 ms inter-stimulus interval, measured from the time of initial deviation from fixation (63 ms mean saccade duration for a 15° saccade), after which the morphed test face was presented. The test face, centered at the saccade cue location, was one of six possible faces, ranging from 15 to 35, centered at Morph 25 (between the happy and sad extremes); it was presented foveally after the saccade was complete. The test face duration was 100 ms, after which subjects were asked to judge whether they had perceived the foveal test face as happy or sad (2-alternative forced-choice task [2AFC]). Five subjects [four female; one author (BW); mean age 26.2] each performed one run of 360 trials.

Figure 1.

Figure 1

Open in a new tab

Stimulus sequence, Experiment 1. a: Experiment 1a, peripheral presentation of a happy or sad adaptor, followed by a saccade to the peripheral adaptor; judgment of morphed face after saccade. b: Experiment 1b, Foveal presentation of an emotional adaptor; identical in all respects to Experiment 1a. c: Experiment 1c, identical to Experiment 1a, with inverted stimuli throughout. d: Experiment 1d, identical to Experiment 1a, except that no saccade was permitted, and the test face was presented at the point of fixation for foveal assessment.

In Experiment 1b (Figure 1b), a test of foveal adaptation, the adaptor was placed at the initial fixation location, rather than in the periphery; all timing was identical to Experiment 1a. Subjects were instructed to make a saccade to the 0.25° saccade cue in the periphery. Subjects judged a test face identical to that used in the first experiment after making the saccade. Five subjects [four female; one author (BW), mean age 27.4] each performed one run of 360 trials.

Experiment 1c (Figure 1c) was a replication of Experiment 1a, except that all face stimuli (adaptors and test faces) were inverted—rotated 180° from their orientations in Experiment 1a. Five subjects [four female; one author (BW); mean age 27.4] each performed one run of 360 trials.

Experiment 1d (Figure 1d) was a test of covert attention, visually identical to Experiment 1a, except that subjects were not permitted to make a saccade to the adaptor, and the test face was presented at fixation to allow for foveal examination of the test stimulus, as in the saccade experiments. Five subjects [four female; one author (BW), mean age 27.4] each performed one run of 360 trials.

Experiment 2a (Figure 3a) was an extension of the paradigm from Experiment 1; the only change was that two adaptor faces (one happy, one sad) were now presented on each trial (each was randomly assigned 15° to the left or right of fixation; Figure 3a). Subjects were presented with two synchronously presented saccade cues: a 0.25° green dot placed 1° to the left or right of fixation as a hemifield cue, and a 0.25° saccade cue (red), identical to that used in Experiment 1a, centered on the cued adaptor. The cues were always consistent with each other. Five subjects [four female; one author (BW), mean age 27.4] each performed one run of 360 trials.

Figure 3.

Figure 3

Open in a new tab

Stimulus sequence, Experiments 2a, 2b. a: Experiments 2a and 2b were identical to Experiment 1, save for the addition of the second adaptor on each trial, as shown. There was always one happy and one sad adaptor presented during each trial and the hemifield they were respectively presented in varied randomly on a trial by trial basis. b: Experiment 2b varied the location of the saccade cue (red dot, on or near the cued target face), rather than locking it to the center of the adaptor, as in Experiment 1a–c and 2a.

Experiment 2b (Figure 3b) was a modified version of Experiment 2a; rather than having the saccade cue centered on one of the two adaptor faces (as described in Experiment 1a and 2a), the saccade cue could be placed anywhere within a 4° box centered on the adaptor face (Figure 3b). The test face was subsequently presented at the same onscreen location as the adaptor face following the saccade. This manipulation was to increase variance in saccade landing positions to better examine the spatial specificity of saccadic remapping. Five subjects [four female; one author (BW), mean age 27.4] each performed one run of 360 trials.

Eyetracking

Eyetracking was performed using an Eyelink 1000 (SR Research; Mississauga, Ontario, Canada) with a level desktop camera; data was recorded monocularly (right eye for all subjects) at 1000 Hz and saccade analysis was performed offline using the Eyelink parser. A saccade was defined as the first time point at which the velocity exceeded 30°/s and the acceleration exceeded 8000°/s2. In addition, a spatial displacement threshold was used to delay the start of each registered saccade until the eye had moved at least 0.15°. All subjects were stabilized on a chinrest during all experiments. Subjects were calibrated using a standard 9-point grid.

Eye movements were recorded at all times during all experiments; all stimuli were presented gaze-contingently. In experiments where a saccade was required (Experiments 1a–c, 2a–b), subjects were only permitted to make a saccade once the saccade cue was presented. Raw eye position was monitored continually throughout all experiments. Any deviation from fixation (greater than 0.5°) when the saccade cue was not present restarted the trial. If gaze deviated during an adaptation period, the adaptor(s) were removed from the screen until the subject foveated the fixation dot for a minimum of 500 ms. As a result, in experiments where a saccade was required, the subjects were unable to foveate the adaptor, since adaptor presentation was yoked to subjects’ eye position. In Experiment 1d, where no saccades were made during the trial, any deviation greater than 0.5° resulted in the trial being discarded during analysis. Note that gaze-contingent stimulus control did not require saccades to be parsed; online monitoring of eye position was performed exclusively on the raw position samples from the eyetracker.

Analysis

All eyetracking and behavioral data were analyzed offline using custom Matlab scripts and S-R Research’s EDFMEX data importation tool. The landing location of the first large saccade was calculated and used to filter the behavioral trials. In Experiments 1a–c and 2a, trials with saccades that landed greater than 2° from the center of the face were discarded (on average, 9% of trials in experiments requiring a saccade). Saccade errors greater than 2° were analyzed in Experiment 2b. Saccade landings in Experiment 2b were converted into a single linear vector of deviation from the center of the target face (face-centered), rather than screen-centered landing location, and were filtered using a overlapping progressive exclusion procedure with 0.5° radius steps from the center of the face to better assess the spatial tuning of saccadic remapping. Accordingly, trials in Experiment 2b where the deviation vector was 0–1° were analyzed separately than trials where the deviation vector was 0.5–1.5° (in steps of 0.5°) to a maximum deviation bin of 2–3°, incorporating 96% of all trials in the experiment. This served to smooth the data given the greater imprecision in saccade landing resulting from the randomly-shifted saccade cue.

The 2AFC psychophysical data were sorted by adaptor condition and the data were fit with the logistic function (x)=11+e−α(x−β), where parameter α is the slope and parameter β is the point of subjective equality (PSE). This generated two logistic functions per experiment per subject; the difference between the PSEs of these two functions is the size of the face aftereffect (see inset, Figure 2). Error bars in Experiments 1a–d and 2a are bootstrapped 95% confidence intervals, and significance was tested (2- tailed p-values) using a bootstrapping procedure on individual subjects (Efron & Tibshirani, 1993); data were averaged after individual bootstrapping to assess group performance. Error bars for Experiment 2b are one standard deviation, using the same procedure.

Figure 2.

Figure 2

Open in a new tab

Results, group effect size for Experiment 1a-d; a: Experiment 1a, remapping of a peripheral target, shows a significant face aftereffect (p<.001); Experiment 1b, with a foveal adaptor, shows significant adaptation when a face is briefly foveated (p<.001). Experiment 1c, with inverted stimuli, shows a significant (p=.008) but reduced aftereffect compared to Experiment 1a (p=0.035). Experiment 1d, similar to Experiment 1a but without eye movements and instead with only covert attention to the peripheral adaptors, shows no significant aftereffect (p>.05). Error bars are bootstrapped 95% confidence intervals. b: Psychometric functions from an exemplar subject in Experiments 1a–d. The difference between the functions is the size of the effect.

Experiment 1: Saccadic Remapping of Faces

To test if saccadic remapping is an object-selective process, which is to say, if more than an isolated feature was remapped prior to the saccade, we performed four initial experiments with a single adaptor. In Experiment 1a, subjects were presented with a peripheral adaptor that was never foveated, which was gaze-contingently removed from the screen on saccade onset. If we observed any face aftereffect in this experiment, which would require the complete representation of the face to be remapped, it would indicate that object-selective information is acquired prior to the saccade, potentially via saccadic remapping. Experiment 1b, with a foveal adaptor prior to a saccade to the peripheral saccade cue, determined the maximum aftereffect possible with foveal adaptation rather than saccadic remapping of a nonfoveated adaptor. Experiment 1c was a version of the saccadic remapping experiment (1a) with inverted stimuli to test for holistic, therefore object-selective, (Fischer & Whitney, 2011; Maurer, Grand, & Mondloch, 2002) versus featural (Valentine, 1988; Yin, 1969) processing of remapped faces. Experiment 1d,where no saccade was permitted to the peripheral adaptor, was a test of the capabilities of covert attention, for comparison with our results in Experiment 1a. In all experiments, subjects judged a foveally-presented test face at the end of the trial.

Results

Experiment 1a, with a never-foveated adaptor, reveals an average aftereffect of 2.21 morph units (2-tailed bootstrap t-test; p<.001; Figure 2). With the foveal adaptor in Experiment 1b, the average aftereffect was 5.32 morph units (2-tailed bootstrap t-test; p<.001). With the inverted stimuli in Experiment 1c, the average aftereffect was 0.853 morph units (2-tailed bootstrap t-test; p=.008). Experiment 1d, our covert attention experiment, showed a nonsignificant average aftereffect of −0.50 morph units (p>.05).

Experiment 1 Discussion

In Experiment 1a, the adaptor face was presented in the periphery, but it was gaze-contingently removed from the screen on saccade onset. Thus, the adaptor was never foveated. Only the test face was ever foveated. Despite presenting the adaptor and test faces to different retinal locations, there was a strong negative face aftereffect. Thus, the aftereffect we observe in this experiment must have been induced by information acquired during the saccade planning process. Our result in this experiment demonstrates that perception of a face can be changed by remapped information, suggesting that remapping is object-selective.

Experiments 1b–d serve to clarify our results in Experiment 1a, and allow us to rule out alternative explanations of the effect. Our results in Experiment 1b are in accord with previous findings demonstrating induction of a face aftereffect with brief foveal adaptation (Leopold, Rhodes, Müller, & Jeffery, 2005; Rhodes, Jeffery, Clifford, & Leopold, 2007). Experiment 1c, with inverted stimuli, addressed whether saccadic remapping is an object-selective or feature-selective process. It was possible that subjects in Experiment 1a acquired a salient feature from the stimuli that characterized the emotion of the face (e.g., the visibility of teeth in a smiling face versus their absence in a sad face). If this had been the case, we would have expected to find a significant aftereffect in Experiment 1c similar to what we found in Experiment 1a. We find a significantly reduced but significant aftereffect with inverted stimuli as expected from the preexisting literature (McKelvie, 1995; Rhodes, Evangelista, & Jeffery, 2009; Rutherford, Chattha, & Krysko, 2008; Yin, 1969). Experiment 1d, our covert attention experiment, showed that a saccade is necessary and covert attention alone is insufficient to generate the negative face aftereffect in Experiment 1a. The lack of an effect in Experiment 1d also suggests that our results in Experiment 1a cannot be accounted for by a global face aftereffect, as would be suggested by Afraz and Cavanagh (2008). Our results in Experiment 1 demonstrate that saccadic remapping is an object-selective process that cannot be accounted for by covert attention or feature remapping alone.

Experiment 2: Saccadic Remapping with Multiple Targets

Experiment 2a: Target Specificity of Saccadic Remapping

Some negative aftereffects spread globally and are therefore spatially non-specific (Afraz & Cavanagh, 2008; McKone, Jeffery, Boeing, & Clifford, 2014). Conversely, remapping is believed to be a spatially specific process (Duhamel et al., 1992; Hall & Colby, 2011; Rolfs et al., 2010; Wurtz, Joiner, & Berman, 2011). If the effect we find in Experiment 1a is due to remapping, rather than a global effect, we should expect spatial specificity of the aftereffect from the remapped location. To address the spatial specificity of the face aftereffect, we presented two adaptor faces, one in each visual field with opposite emotions (see Methods). If, in fact, our results could be accounted for by a globally-induced negative aftereffect, as suggested by Afraz and Cavanagh, we would expect to see no effect at all in this experiment, as the effects of each adaptor would cancel.

Results

We find an aftereffect in Experiment 2a of 2.16 morph units with a (2-tailed bootstrap t-test ; p<.001) (Figure 3). Additionally, we found no significant difference between the size of the effect in this experiment and in Experiment 1a (2-tailed bootstrap t-test, p>.05).

Experiment 2b: Spatial Tuning of Saccadic Remapping

Experiment 2a suggests some degree of spatial tuning, as the effect was undiminished by the presence of the adaptor with the opposite expression in the other hemifield. To more precisely characterize the spatial tuning of object-selective remapping, we varied the location of the saccade cue in Experiment 2b, rather than simply centering it on the adaptor face, as we had done in previous experiments (as illustrated in Figure 1a–d and Figure 3a). The saccade cue, to which subjects targeted their saccade, was presented at a random location in a ±2° box (4° on a side) centered on the adaptor (illustrated in Figure 3b). As a result, saccade landings in Experiment 2b were more widely distributed, relative to the adaptor face, than in Experiments 1a-c and 2a. This allowed us to analyze the spatial tuning of the effect. Note that the test face was presented at exactly the same location as the adaptor face; subjects’ saccade landing locations varied as to their proximity to the test face.

For each trial, we calculated the vector between the subject’s saccade landing location and the center of the adaptor face. This allowed us to perform a progressive, overlapping exclusion analysis (shown in Figure 4b and described in Methods); trials were binned by spatial accuracy relative to the location of the adaptor face in overlapping one degree bins (i.e., trials where the saccade landed 0–1° from the center of the face, 0.5–1.5° from the center of the face, etc.). Thus, trials fell into multiple bins, effectively smoothing the data. While such an analysis is possible in our previous experiments, the high precision of saccade landings in those experiments limited the number of high-error trials available for this type of analysis.

Figure 4.

Figure 4

Open in a new tab

Experiment 2 Results. a: Group mean results for Experiment 2a, with two adaptors on screen in each trial, and for comparison, Experiment 1a (from Figure 2). Both results are respectively significant, p<.001, and there is no significant difference between them (p>.05). b: Results of the progressive overlapping exclusion analysis for Experiment 2b. We find a strong presaccadically-induced face aftereffect only when the deviation between saccade landing and the center of the target face is between within 0.5° and 1.5° of the center of the face (asterisk indicates p<.001; Bonferroni-corrected α = .01; note that the 0–1° error bin is trending at p = .015). Given the size of the stimulus (3.6° wide by 4.3° high), this indicates that the saccade must land where the adaptor face had been. When saccade error is higher, we observe no significant aftereffect, p>.05 (in the 1°–2°, 1.5°–2.5° and 2°–3° saccade error bins). Error bars for this analysis only are bootstrapped 1SD.

Results

On trials where the saccade landed within 1° of the center of the adaptor location (Figure 4), the aftereffect was 2.06 morph units (p=.015; trending but not significant at a Bonferroni-corrected α = .01), consistent with the previous experiments. Trials where the saccade landed between 0.5–1.5° from the center of the adaptor location showed a significant aftereffect of 2.16 morph units (p<.001; significant at a Bonferroni-corrected α = .01) When saccade landings were further from the face, there was a decrease in the strength of the aftereffect; landing deviations greater than approximately 1.5 deg resulted in little measurable aftereffect. All individual subjects in this experiment displayed a decrease in aftereffect magnitude with greater saccadic imprecision. In addition, we performed a similar analysis in the absence of overlapping bins (i.e., bins from 0–1°, 1–2°) and found a similar pattern, with no significant aftereffect when saccades landed more than 1° from the center of the face.

Experiment 2 Discussion

Experiments 2a and 2b revealed a saccade-contingent negative aftereffect with the presence of two adaptors on each trial, which rules out the possibility that the effect we observed in Experiment 1a was the result of global spatially-nonspecific adaptation or other spatially non-specific decision biases. If our effect was caused by a spatially non366 specific process (something other than remapping), we would expect the two opposing adaptor faces to reduce or eliminate the negative aftereffect. Experiment 2b shows that not only are saccades required for the remapped negative face aftereffect, but the accuracy of the saccade targeting modulates the aftereffect (Fig. 4).

General Discussion

In this series of experiments, we have demonstrated that, prior to a saccade, a complete representation of a target object is acquired by the visual system, sufficient to induce a negative face aftereffect, rather than individual features of our stimuli, which would not have resulted in the effect we observed. Our findings reinforce the idea that saccadic remapping, in the sense we have used the term, may operate in a spatiotopic framework by remapping object representations (Burr & Morrone, 2011), rather than the remapping of attention pointers in a purely retinotopic framework (Rolfs et al., 2010), although both accounts could be generalized to accommodate our results. It is also possible that our results could be due to presaccadic compression of space (Pola, 2011; Ross, Morrone, & Burr, 1997; Zirnsak & Moore, 2014; Zirnsak, Gerhards, & Kiani, 2011). However, our primary interest concerned what information was acquired from the saccade target prior to the saccade itself, and our results suggest that more information is acquired than was previously suspected. This remapped information from a saccade target, acquired in an object-selective manner, may facilitate the postsaccadic updating of our perceptual representation of the visual world.

Several additional experiments rule out alternative explanations for our results. Experiment 1c showed that it is configural or holistic (i.e., object-level) information about faces that is remapped and produces a negative face aftereffect. Thus, it is unlikely that features (e.g., teeth) or a figural aftereffect (Rhodes et al., 2004) could explain our results. Experiment 1d demonstrated the necessity of saccades; simply attending to a peripherally presented face (as in Experiment 1d) does not allow the visual system to acquire sufficient information to change perception of a subsequent face. Further, the remapped face aftereffect was spatially specific (Experiments 2a and 2b), ruling out decision biases, non-spatially specific processes, global aftereffects (Afraz & Cavanagh, 2008), or other non-remapping selective processes (Deubel & Schneider, 1996; Harrison et al., 2013a; Kowler, Anderson, Dosher, & Blaser, 1995).

Our method for determining whether saccadic remapping is object-selective is based on how the perception of a foveated neutral face changes, depending on what, if any, information is acquired before it is presented. We have chosen to describe this as a negative aftereffect, building on the literature surrounding the face aftereffect (Webster, Kaping, Mizokami, & Duhamel, 2004), but it could also be characterized as an upright face-specific figural aftereffect (Asch & Witkin, 1948) or shape contrast illusion (Suzuki & Cavanagh, 1998), particularly given the brevity of our presentation. However, there is little reason to expect an inversion effect for a figural aftereffect, as we observed in Experiment 1c. While there may be debate over what our effect may best be called, the fact remains that subjects’ perception of the test face only changes when they have made a saccade to an emotional face that they never foveate, suggesting that saccadic remapping, as assessed through psychophysical methods, is an object-selective process. Our use of faces as stimuli allowed us to measure remapping with stimuli that were both familiar and sufficiently complex to be useful as precise tools for probing what information was acquired prior to the saccade. Future work could include further investigations of faces in the context of saccadic remapping, building on preexisting work in the face aftereffect literature on emotion, identity, gender and other complex, multi feature elements of faces.

Related to this, recent work on the serial dependence in face perception (Liberman, Fischer, & Whitney, 2014) has shown a positive shift in how subsequent faces are perceived over a longer timescale. This suggests that the continual process of constructing a stable perception of the world can be influenced in multiple ways. Our negative aftereffect from object-selective remapping operates on a brief timescale and reflects information available to the visual system immediately prior to the time of the saccade, whereas serial dependence of faces is a longer-term process, on the order of seconds rather than milliseconds (Fischer & Whitney, 2014). It may be that transsaccadic comparisons exaggerate differences between remapped and foveated information, maximizing sensitivity to changes, whereas serial dependence subserves a stable representation of objects in the world.

Saccadic remapping of object-selective information, which is a spatially tuned process, may facilitate object perception across eye movements by acquiring sufficient information prior to a saccade to permit immediate comparison after the saccade target is fixated. Acquiring information at the level of objects around the time of a saccade may help explain object stability across saccades (Collins, Heed, & Röder, 2010; Rensink, 2002; Wurtz et al., 2011) and may also speak to the speed with which objects can be recognized (Greene & Oliva, 2009; Oliva & Torralba, 2007; Pajak & Nuthmann, 2013). Immediately prior to a saccade, the visual system acquires object-specific information, and may use this information after the saccade to update representations, simultaneously maximizing sensitivity to any potential object changes while facilitating stable perception of unchanging objects.

Acknowledgements

This material is based upon work supported by a NSF-GRFP DGE1106400 to BW and NIH EY018216 and NSF 0748689 to DW. The authors also wish to thank Anna Kosovicheva for many helpful discussions.

Footnotes

Author Contributions

BW and DW designed the experiments; BW programmed the experiments, collected and analyzed the data, BW and DW wrote the manuscript.

References

  1. S-R, Cavanagh P. Retinotopy of the face aftereffect. Vision Research. 2008;48(1):42–54. doi: 10.1016/j.visres.2007.10.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Asch SE, Witkin HA. Studies in space orientation: I. Perception of the upright with displaced visual fields. Journal of Experimental Psychology. 1948;38(3):325. doi: 10.1037/h0057855. [DOI] [PubMed] [Google Scholar]
  3. Born S, Ansorge U, Kerzel D. Feature-based effects in the coupling between attention and saccades. Journal of Vision. 2012;12(11) doi: 10.1167/12.11.27. [DOI] [PubMed] [Google Scholar]
  4. Brainard DH. The psychophysics toolbox. Spatial Vision. 1997;10(4):433–436. [PubMed] [Google Scholar]
  5. Burr DC, Morrone MC. Spatiotopic coding and remapping in humans. Philosophical Transactions of the Royal Society B: Biological Sciences. 2011;366(1564):504–515. doi: 10.1098/rstb.2010.0244. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Burr DC, Morrone MC. Constructing stable spatial maps of the world. Perception. 2012;41(11):1355–1372. doi: 10.1068/p7392. [DOI] [PubMed] [Google Scholar]
  7. Cavanagh P, Hunt AR, Afraz A, Rolfs M. Visual stability based on remapping of attention pointers. Trends in Cognitive Sciences. 2010;14(4):147–153. doi: 10.1016/j.tics.2010.01.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Collins T, Heed T, Röder B. Eye-movement-driven changes in the perception of auditory space. Attention Perception and Psychophysics. 2010;72(3):736–746. doi: 10.3758/APP.72.3.736. [DOI] [PubMed] [Google Scholar]
  9. Collins T, Rolfs M, Deubel H, Cavanagh P. Post-saccadic location judgments reveal remapping of saccade targets to non-foveal locations. Journal of Vision. 2009;9(5):29–29. doi: 10.1167/9.5.29. [DOI] [PubMed] [Google Scholar]
  10. Cornelissen FW, Peters EM, Palmer J. The Eyelink Toolbox: eye tracking with MATLAB and the Psychophysics Toolbox. Behavior Research Methods, Instruments, & Computers : a Journal of the Psychonomic Society, Inc. 2002;34(4):613–617. doi: 10.3758/bf03195489. [DOI] [PubMed] [Google Scholar]
  11. Demeyer M, De Graef P, Wagemans J, Verfaillie K. Transsaccadic identification of highly similar artificial shapes. Journal of Vision. 2009;9(4):28, 1–14. doi: 10.1167/9.4.28. [DOI] [PubMed] [Google Scholar]
  12. Deubel H, Schneider WX. Saccade target selection and object recognition: evidence for a common attentional mechanism. Vision Research. 1996;36(12):1827–1837. doi: 10.1016/0042-6989(95)00294-4. [DOI] [PubMed] [Google Scholar]
  13. Duhamel J-R, Colby CL, Goldberg ME. The Updating of the Representation of Visual Space in Parietal Cortex by Intended Eye Movements. Science. 1992;116(8):1511–1518. doi: 10.1126/science.1553535. [DOI] [PubMed] [Google Scholar]
  14. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Boca Raton, FL: CRC Press; 1993. [Google Scholar]
  15. Ekman P, Friesen WV. Measuring facial movement. Environmental Psychology and Nonverbal Behavior. 1976;1(1):56–75. [Google Scholar]
  16. Fischer J, Whitney D. Object-level visual information gets through the bottleneck of crowding. Journal of Neurophysiology. 2011;106(3):1389–1398. doi: 10.1152/jn.00904.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Fischer J, Whitney D. Serial dependence in visual perception. Nature Neuroscience. 2014 doi: 10.1038/nn.3689. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Fracasso A, Caramazza A, Melcher D. Continuous perception of motion and shape across saccadic eye movements. Journal of Vision. 2010;10(13):14. doi: 10.1167/10.13.14. [DOI] [PubMed] [Google Scholar]
  19. Greene MR, Oliva A. The Briefest of Glances: The Time Course of Natural Scene Understanding. Psychological Science. 2009;20(4):464–472. doi: 10.1111/j.1467-9280.2009.02316.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hall NJ, Colby CL. Remapping for visual stability. Philosophical Transactions of the Royal Society B: Biological Sciences. 2011;366(1564):528–539. doi: 10.1098/rstb.2010.0248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Harrison WJ, Bex PJ. Integrating retinotopic features in spatiotopic coordinates. Journal of Neuroscience. 2014;34(21):7351–7360. doi: 10.1523/JNEUROSCI.5252-13.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Harrison WJ, Mattingley JB, Remington RW. Eye movement targets are released from visual crowding. Journal of Neuroscience. 2013a;33(7):2927–2933. doi: 10.1523/JNEUROSCI.4172-12.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Harrison WJ, Retell JD, Remington RW, Mattingley JB. Visual crowding at a distance during predictive remapping. Current Biology : CB. 2013b;23(9):793–798. doi: 10.1016/j.cub.2013.03.050. [DOI] [PubMed] [Google Scholar]
  24. Hayhoe M, Lachter J, Feldman J. Integration of form across saccadic eye movements. Perception. 1991;20(3):393–402. doi: 10.1068/p200393. [DOI] [PubMed] [Google Scholar]
  25. Helmholtz H. Treatise on Physiological Optics. Vol. 3. Treatise on Physiological Optics; 1962. [Google Scholar]
  26. Kowler E, Anderson E, Dosher B, Blaser E. The role of attention in the programming of saccades. Vision Research. 1995;35(13):1897–1916. doi: 10.1016/0042-6989(94)00279-u. [DOI] [PubMed] [Google Scholar]
  27. Leopold DA, Rhodes G, Müller K-M, Jeffery L. The dynamics of visual adaptation to faces. Proceedings. Biological Sciences / the Royal Society. 2005;272(1566):897–904. doi: 10.1098/rspb.2004.3022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Liberman A, Fischer J, Whitney D. Serial dependence in the perception of faces. Current Biology. 2014:1–19. doi: 10.1016/j.cub.2014.09.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Maurer D, Grand RL, Mondloch CJ. The many faces of configural processing. Trends in Cognitive Sciences. 2002;6(6):255–260. doi: 10.1016/s1364-6613(02)01903-4. [DOI] [PubMed] [Google Scholar]
  30. McKelvie SJ. Emotional expression in upside-down faces: evidence for configurational and componential processing. The British Journal of Social Psychology / the British Psychological Society. 1995;34(Pt 3):325–334. doi: 10.1111/j.2044-8309.1995.tb01067.x. [DOI] [PubMed] [Google Scholar]
  31. McKone E, Jeffery L, Boeing A, Clifford C. Face identity aftereffects increase monotonically with adaptor extremity over, but not beyond, the range of natural faces. Vision Research. 2014 doi: 10.1016/j.visres.2014.01.007. [DOI] [PubMed] [Google Scholar]
  32. Melcher D. Spatiotopic transfer of visual-form adaptation across saccadic eye movements. Current Biology. 2005;15(19):1745–1748. doi: 10.1016/j.cub.2005.08.044. [DOI] [PubMed] [Google Scholar]
  33. Melcher D. Predictive remapping of visual features precedes saccadic eye movements. Nature Neuroscience. 2007;10(7):903–907. doi: 10.1038/nn1917. [DOI] [PubMed] [Google Scholar]
  34. Melcher D. Selective attention and the active remapping of object features in trans-saccadic perception. Vision Research. 2009;49(10):1249–1255. doi: 10.1016/j.visres.2008.03.014. [DOI] [PubMed] [Google Scholar]
  35. Melcher D, Morrone MC. Spatiotopic temporal integration of visual motion across saccadic eye movements. Nature Neuroscience. 2003;6(8):877–881. doi: 10.1038/nn1098. [DOI] [PubMed] [Google Scholar]
  36. Nakamura K, Colby CL. Updating of the visual representation in monkey striate and extrastriate cortex during saccades. Proceedings of the National Academy of Sciences of the United States of America. 2002;99(6):4026–4031. doi: 10.1073/pnas.052379899. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Oliva A, Torralba A. The role of context in object recognition. Trends in Cognitive Sciences. 2007;11(12):520–527. doi: 10.1016/j.tics.2007.09.009. [DOI] [PubMed] [Google Scholar]
  38. Pajak M, Nuthmann A. Object-based saccadic selection during scene perception: Evidence from viewing position effects. Journal of Vision. 2013;13(5) doi: 10.1167/13.5.2. [DOI] [PubMed] [Google Scholar]
  39. Parks NA, Corballis PM. Electrophysiological correlates of presaccadic remapping in humans. Psychophysiology. 2008;45(5):776–783. doi: 10.1111/j.1469-8986.2008.00669.x. [DOI] [PubMed] [Google Scholar]
  40. Pelli DG. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision. 1997;10(4):437–442. [PubMed] [Google Scholar]
  41. Pola J. An explanation of perisaccadic compression of visual space. Vision Research. 2011;51(4):424–434. doi: 10.1016/j.visres.2010.12.010. [DOI] [PubMed] [Google Scholar]
  42. Rensink RA. Change detection. Annual Review of Psychology. 2002;53(1):245–277. doi: 10.1146/annurev.psych.53.100901.135125. [DOI] [PubMed] [Google Scholar]
  43. Rhodes G, Evangelista E, Jeffery L. Orientation-sensitivity of face identity aftereffects. Vision Research. 2009;49(19):2379–2385. doi: 10.1016/j.visres.2009.07.010. [DOI] [PubMed] [Google Scholar]
  44. Rhodes G, Jeffery L, Clifford CWG, Leopold DA. The timecourse of higher-level face aftereffects. Vision Research. 2007;47(17):2291–2296. doi: 10.1016/j.visres.2007.05.012. [DOI] [PubMed] [Google Scholar]
  45. Rhodes G, Jeffery L, Watson TL, Jaquet E, Winkler C, Clifford CWG. Orientation-contingent face aftereffects and implications for face-coding mechanisms. Current Biology. 2004;14(23):2119–2123. doi: 10.1016/j.cub.2004.11.053. [DOI] [PubMed] [Google Scholar]
  46. Rolfs M, Jonikaitis D, Deubel H, Cavanagh P. Predictive remapping of attention across eye movements. Nature Neuroscience. 2010;14(2):252–256. doi: 10.1038/nn.2711. [DOI] [PubMed] [Google Scholar]
  47. Ross J, Morrone M, Burr D. Compression of visual space before saccades. Nature. 1997 doi: 10.1038/386598a0. [DOI] [PubMed] [Google Scholar]
  48. Rutherford MD, Chattha HM, Krysko KM. The use of aftereffects in the study of relationships among emotion categories. Journal of Experimental Psychology: Human Perception and Performance. 2008;34(1):27–40. doi: 10.1037/0096-1523.34.1.27. [DOI] [PubMed] [Google Scholar]
  49. Sommer MA, Wurtz RH. Brain circuits for the internal monitoring of movements. Annual Review of Neuroscience. 2008;31:317–338. doi: 10.1146/annurev.neuro.31.060407.125627. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Subramanian J, Colby CL. Shape selectivity and remapping in dorsal stream visual area LIP. Journal of Neurophysiology. 2014;111(3):613–627. doi: 10.1152/jn.00841.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Suzuki S, Cavanagh P. A shape-contrast effect for briefly presented stimuli. Journal of Experimental Psychology: Human Perception and Performance. 1998;24(5):1315–1341. doi: 10.1037//0096-1523.24.5.1315. [DOI] [PubMed] [Google Scholar]
  52. Turi M, Burr D. Spatiotopic perceptual maps in humans: evidence from motion adaptation. Proceedings: Biological Sciences. 2012;279(1740):3091–3097. doi: 10.1098/rspb.2012.0637. doi:10.2307/41622646?ref=no-x-route:923dfbb32b55ee4e53397fa7384672e0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Valentine T. Upside-down faces: a review of the effect of inversion upon face recognition. British Journal of Psychology (London, England : 1953) 1988;79(Pt 4):471–491. doi: 10.1111/j.2044-8295.1988.tb02747.x. [DOI] [PubMed] [Google Scholar]
  54. van Koningsbruggen MG, Buonocore A. Mechanisms behind perisaccadic increase of perception. Journal of Neuroscience. 2013;33(28):11327–11328. doi: 10.1523/JNEUROSCI.1567-13.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Webster MA, Kaping D, Mizokami Y, Duhamel P. Adaptation to natural facial categories. Nature. 2004;428(6982):557–561. doi: 10.1038/nature02420. [DOI] [PubMed] [Google Scholar]
  56. Wolfe BA, Whitney D. Facilitating recognition of crowded faces with presaccadic attention. Frontiers in Human Neuroscience. 2014;8:103. doi: 10.3389/fnhum.2014.00103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Wurtz RH. Neuronal mechanisms of visual stability. Vision Research. 2008;48(20):2070–2089. doi: 10.1016/j.visres.2008.03.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Wurtz RH, Joiner WM, Berman RA. Neuronal mechanisms for visual stability: progress and problems. Philosophical Transactions of the Royal Society B: Biological Sciences. 2011;366(1564):492–503. doi: 10.1098/rstb.2010.0186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Yamanashi Leib A, Puri AM, Fischer J, Bentin S, Whitney D, Robertson L. Crowd perception in prosopagnosia. Neuropsychologia. 2012;50(7):1698–1707. doi: 10.1016/j.neuropsychologia.2012.03.026. [DOI] [PubMed] [Google Scholar]
  60. Yin RK. Looking at upside-down faces. Journal of Experimental Psychology. 1969;81(1):141. [Google Scholar]
  61. Zimmermann E, Morrone MC, Fink GR, Burr D. Spatiotopic neural representations develop slowly across saccades. Current Biology : CB. 2013;23(5):R193–R194. doi: 10.1016/j.cub.2013.01.065. [DOI] [PubMed] [Google Scholar]
  62. Zirnsak M, Moore T. Saccades and shifting receptive fields: anticipating consequences or selecting targets? Trends in Cognitive Sciences. 2014 doi: 10.1016/j.tics.2014.10.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Zirnsak M, Gerhards R, Kiani R. Anticipatory Saccade Target Processing and the Presaccadic Transfer of Visual Features. The Journal of …. 2011 doi: 10.1523/JNEUROSCI.2465-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES