Eye movements shape visual learning

Significance

In domains as diverse as radiology and airport baggage screening, it has been observed that experts make fewer eye movements than novices. We therefore hypothesized that eye movements influence learning in the context of noisy or complicated stimulus patterns. Observers practiced a task in which noisy visual signals were presented during a saccade. Under these conditions, most observers integrated these signals in a manner that was dictated by the eye movement direction. Surprisingly, observers who exhibited this tendency to integrate during saccades learned, through extensive training, to perform the same integration in the absence of eye movements. These results suggest that eye movements can shape the way in which people learn to extract meaningful information from challenging visual stimuli.

Abstract

Most people easily learn to recognize new faces and places, and with more extensive practice they can become experts at visual tasks as complex as radiological diagnosis and action video games. Such perceptual plasticity has been thoroughly studied in the context of training paradigms that require constant fixation. In contrast, when observers learn under more natural conditions, they make frequent saccadic eye movements. Here we show that such eye movements can play an important role in visual learning. Observers performed a task in which they executed a saccade while discriminating the motion of a cued visual stimulus. Additional stimuli, presented simultaneously with the cued one, permitted an assessment of the perceptual integration of information across visual space. Consistent with previous results on perisaccadic remapping [M. Szinte, D. Jonikaitis, M. Rolfs, P. Cavanagh, H. Deubel, J. Neurophysiol. 116, 1592–1602 (2016)], most observers preferentially integrated information from locations representing the presaccadic and postsaccadic retinal positions of the cue. With extensive training on the saccade task, these observers gradually acquired the ability to perform similar motion integration without making eye movements. Importantly, the newly acquired pattern of spatial integration was determined by the metrics of the saccades made during training. These results suggest that oculomotor influences on visual processing, long thought to subserve the function of perceptual stability, also play a role in visual plasticity.

A hallmark of visual expertise is the ability to extract, at a glance, information that novices can obtain only with careful scrutiny. For example, experienced radiologists can identify pathological features in X-rays more quickly and with fewer eye movements than medical students, and analogous findings have been reported in domains as diverse as chess, airport baggage screening, music reading, and sports (1).

In many of these domains, the development of expertise entails a shift toward “holistic” processing (2): Experts perceive larger patterns in the visual stimulus, while novices seem to inspect smaller elements individually. In other words, experts learn through experience to integrate relevant visual information that is distributed across the retinal image. The neural mechanisms by which observers learn to perform this integration are unknown, but the data suggest that eye movements play an important role (3).

In the visual cortex, many neurons respond to saccadic eye movements by shifting their receptive fields transiently across retinal space (4–9). Such receptive field remapping has been posited as a mechanism by which the brain can integrate attended visual information across saccades, and this idea has been confirmed in a number of psychophysical studies (10–19). We hypothesized that remapping might shape the way in which observers learn to integrate information across visual space.

To test this idea, we used a psychophysical paradigm in which observers made saccades while evaluating a motion stimulus presented at a specific location (10). By adding stimuli at different retinal positions, we were able to examine the ways in which observers integrate visual information across space during eye movements. Consistent with previous reports (10, 15, 16), we found that most observers naturally integrated visual information across saccades in a manner that was consistent with remapping. More importantly, we found that, with training, most observers gradually acquired the ability to integrate such visual information without making eye movements. Because the changes in visual integration were specific to the retinal locations involved in remapping, we suggest that remapping could serve as a mechanism by which observers develop holistic perception of familiar visual scenes.

Results

We made use of a paradigm (Fig. 1) in which visual stimuli could be integrated during eye movements (10). Observers began each trial by fixating a central target, which was flanked by four noisy random dot kinematograms (0% coherence). Subsequently, a saccade target appeared simultaneously with a cue indicating which of the four kinematograms would contain the stimulus to be discriminated. We refer to this stimulus as S1. At a random time before saccade onset, the motion in S1 became coherent, and on most trials, the motion of a second kinematogram (called S2) also became coherent. In 80% of these trials, the motion direction and coherence of S2 were the same as those of S1. Thus, all of the S2 locations were equally likely to be informative, but only one was consistent with remapping (Fig. 1).

Fig. 1.

Stimulus timing and experimental procedure. (A) Temporal sequence of visual stimuli in relation to onset of a saccadic eye movement (10). A fixation target (ft) appeared, and after fixation was acquired for 500 to 750 ms, four random dot kinematograms (RDKs) appeared at locations positioned symmetrically about the ft. Subsequently a saccade target (st) appeared along with a cue to indicate the location of the RDK (S1) to be discriminated. After a variable interval, the motion of S1 became coherent, and on most trials the motion of another RDK (S2) became coherent as well. The saccade was then executed, and the perceived motion direction was reported. A saccade could be horizontal (as shown here) or vertical. (B) Step-by-step evolution of visual stimulus presentation. The fixation period (1) was followed by the appearance of the st (2) and then a flashed green Gaussian blob to indicate the location of S1 (3). Trials with only S1 were called the Just-S1 condition (3a). In trials with two simultaneous motion signals (S1 + S2), depending on the relative locations of the cue and the saccade direction, the condition was called Remapping (3b), Diagonal-Control (3c), or Orthogonal-Control (3d). (4) After the 100 ms presentation of the motion signals, the RDKs became incoherent again for 400 ms. Observers were instructed to report the direction of S1 at the end of each trial. Note that, in this example, S1 is on the top right, but could occupy any of the other three locations on different trials.

Observers had to complete the saccade and then indicate the perceived motion direction in S1 with a button click (see Materials and Methods for details). Based on previous work (10), we expected that observers would preferentially integrate motion information when S1 and S2 were separated by the saccade vector, so as to engage remapping (Fig. 1); the question here was whether training in this saccade paradigm would lead to a lasting change in visual perception during fixation.

We trained 10 observers extensively on this task, with most (7/10) participating for a period of 10 wk. During the training period, all observers were exposed to stimuli that varied in motion direction, position, and coherence (Materials and Methods). These stimuli were configured in such a way that all combinations were experienced with equal frequency. However, the distribution of saccades was biased so that 80% were horizontal. On this basis, we predicted that, if remapping is important for learning, a horizontal bias for spatial integration should develop through the course of training.

Motion Integration in the Saccade Task.

To illustrate the main pattern of results, we first consider trials involving horizontal saccades in which S1 and S2 were congruent: Their motion direction and coherence levels were the same. Other saccade and motion directions were included in the protocol to prevent overtraining and to approximate a more naturalistic learning process, but for the most part, the resulting data are not presented here [see Szinte et al. (10) for a thorough analysis]. As described in Materials and Methods, we used mean d′ as a measure of the influence of S2 on the ability to discriminate the direction of the cued motion signal at S1. We calculated this metric for the Remapping condition, as well as the Orthogonal-Control and Diagonal-Control conditions (Fig. 1), wherein S2 was present and informative, but not placed so as to engage remapping.

Fig. 2A shows, for example observer 1, the evolution of mean d′ as the performance of saccade trials progressed. As expected from previous work (10), the data showed a strong effect of remapping: When a motion signal S2 was presented at the remapping location simultaneously with S1 (Fig. 2A, black line), performance was significantly better (P < 0.05, permutation test) than in the Just-S1 condition, when S1 was presented alone (Fig. 2A, green line). In contrast, performance in the Orthogonal-Control and Diagonal-Control conditions was not significantly different from in the Just-S1 condition (P > 0.05, permutation test). The superiority of performance in the Remapping condition was apparent in the first group of 200 trials (sessions 1 to 4) and persisted throughout the subsequent training period, indicating a consistent, preferential integration of motion signals in the Remapping condition.

Fig. 2.

Evolution of performance in the saccade task. (A) Motion sensitivity in the saccade condition for example observer 1. Each point on the x axis represents one group of 800 trials, which were subdivided into 200 trials of each of the four conditions listed in the Upper Right Inset. In total, about 4,000 trials were analyzed (SI Appendix, Supplementary Information Text). A mean d′ value (± SEM) for each of the five analysis periods was computed according to the observer’s performance at each coherence level, expressed as a mean of the fraction of correct trials within the period. In all five periods, the d′ values were significantly higher for the remapping condition (black) than for the Just-S1 condition (green). By comparison, performance in the other control conditions was not significantly different from performance in the Just-S1 condition. (B) Motion sensitivity in the saccade task for observer 2. This observer’s performance in the Remapping condition (black) was similar to that in the Just-S1 condition (green): Training significantly improved performance in each condition, but there was no advantage for remapping over any other condition (P > 0.05, permutation test). Asterisks: statistically significant differences between Remapping and Just-S1 conditions (P < 0.05, permutation test). Error bars show SEM.

We found a similar remapping effect in 7 of 10 observers (SI Appendix, Fig. S1A). For this group, performance was better in the Remapping condition than in the Just-S1 or in any of the control conditions (P < 0.05 for all observers, Wilcoxon signed-rank test [WSR] test, false discovery rate [FDR] corrected). On average, for these observers motion sensitivity in the Remapping condition was 27% greater than in the Just-S1 condition (mean d′ difference ± SEM = 0.27 ± 0.018; P < 0.001, WSR test), while the Diagonal-Control and the Orthogonal-Control conditions were not significantly different from the Just-S1 condition (mean d′ difference ± SEM = −0.008 ± 0.018 and 0.018 ± 0.02, respectively [P > 0.05, WSR test]) (SI Appendix, Fig. S1B). These results were not unique to horizontal saccades, as we observed a comparable effect during the minority (20%) of trials involving vertical saccades (SI Appendix, Fig. S2A).

Sample data from one of the observers (observer 2) who did not exhibit a detectable influence of remapping are shown in Fig. 2B. For this observer, the addition of coherent motion at any S2 position did not improve performance, which was never significantly greater than in the Just-S1 condition (P > 0.05, permutation test).

Remapping during Saccades Does Not Require Learning.

We next sought to determine whether the superior performance in the Remapping condition was learned through training. From Fig. 2A, it is evident that the example observer’s performance improved across all conditions at a rate of ∼0.01 d′ units per session. However, analyzing the data across all sessions did not reveal a significant difference in the slopes of the learning curves across stimulus conditions for the example observer (F [1, 3] = 0.17, P = 0.919, analysis of covariance [ANCOVA]).

To examine the effects of training on psychophysical remapping, we focused on the subset of observers who showed statistically significant remapping and who completed the full training protocol. For this group, overall performance in the last eight sessions of trials was significantly better than in the first eight sessions (P < 0.05, permutation test), but there was no evidence that the learning rate (slope) was different in the Remapping condition (P > 0.05, ANCOVA). Indeed, the difference in performance between the Remapping and Just-S1 did not change significantly over the course of the 10-wk training period (P > 0.05, permutation test) (Fig. 3A).

Fig. 3.

Evolution of motion integration through training on saccade trials. (A) Comparison of the remapping effect in the first (green) and the last (blue) eight sessions for the individual observers who showed a remapping effect. Error bars indicate SEM. (B) Effects of S2 on perception of S1, combined across observers. To describe quantitatively the timewise influence of training in the saccade task, for each observer, trials were divided into 27 steps: the first step contained the data for the first four sessions (∼200 saccade trials), and the step was moved forward 30 trials at a time. Each point indicates the mean improvement observed with presentation of S2 in the Remapping (black), Diagonal-Control (blue), and Orthogonal-Control (orange) conditions, combined across observers. Solid lines show the best-fitting linear regressions, indicating no learning effect specific to the presentation of S2 (Remapping condition–Just-S1 condition: slope = 0.0016, P = 0.11, SE of the slope = 0.001). Dashed lines around the regression lines show the confidence interval for the regression estimate.

To track this evolution in greater detail, we calculated d′ in a window of 200 trials, advancing the window by 30 trials at each step. Within each window we subtracted the d′ values in the Just-S1 condition from those in the Remapping condition. The results did not reveal a significant linear trend over time for any of the observers who showed remapping (linear regression; P > 0.05 for all observers). These data, pooled across observers, are shown in Fig. 3B. Thus, while these observers improved generally on the motion task with training, the relative advantage seen in the Remapping condition did not appear to change through time, suggesting that remapping during saccades occurs without training in this group of observers.

Motion Integration in the Fixation Task.

In addition to the saccade trials, we interleaved trials in which the same motion task was performed during steady fixation. These included a Like-Remapping condition, in which S1 and S2 were separated horizontally, as in the Remapping condition during most saccade trials. The analogous control conditions (Orthogonal-Control and Diagonal-Control) and the Just-S1 condition (Fig. 1) were also tested during fixation; all stimuli were shown with equal frequency. These conditions allowed us to examine the effect of training on visual integration.

Fig. 4A shows the evolution of performance in the fixation task across training sessions in observer 1, whose average performance is shown in Fig. 2A. For observer 1, early in the training procedure performance was independent on the position or availability of S2: The values of d′ were overlapping for the Like-Remapping condition, the control S2 positions, and the Just-S1 condition (P > 0.05, permutation test). However, after about 1,000 trials in total, a relative advantage emerged for the Like-Remapping condition, as performance outpaced that found in the Just-S1 condition (black line in Fig. 4A; P < 0.05, permutation test). No comparable effect was evident for the control S1 + S2 conditions (P > 0.05, permutation test). An ANCOVA test of the session-by-session data revealed that the slope of the learning curve in the Like-Remapping condition was significantly different from that obtained in the other three conditions [F (1, 3) = 3.0, P = 0.0361]. Similar results were found in all of the other observers who showed significant remapping in the saccade task (P < 0.05, ANCOVA). Thus, through training, observers learned to integrate visual information during fixation in a manner consistent with remapping during saccades.

Fig. 4.

Evolution of performance in the fixation task. (A) Performance of the example observer who showed a remapping effect during saccades. Same conventions as in Fig. 2. In the Like-Remapping condition (black), the mean d′ values are initially similar to those of the Just-S1 (green) and control (blue, orange) conditions. A significant improvement for Like-Remapping relative to Just-S1 trials emerged in the fourth group of sessions (P < 0.05, permutation test). In the other control conditions, no significant increase of performance occurred in any period in comparison with the Just-S1 condition. (B) Performance of observer 2 during fixation trials. Although performance improved through time, no location-specific improvement was found for the Like-Remapping (black) or any other condition. Error bars show SEM. Asterisks: statistically significant differences between Remapping and Just-S1 conditions (P < 0.05, permutation test).

In contrast, the example observer (observer 2) in Fig. 2B, who did not show evidence of remapping in the saccade condition, did not show a spatially specific pattern of learning. Performance in the fixation condition for the Just-S1 and Like-Remapping conditions (Fig. 4B) was indistinguishable at all time points throughout the training procedure (P > 0.05, permutation test). Data from all observers including another example observer (observer 5) who did not show an effect of remapping during saccades are shown in SI Appendix, Fig. S3.

These results are consistent with the reasoning outlined in the Introduction, namely that the learning of spatially specific patterns of integration depends on two factors: 1) The strength of remapping during saccades and 2) The extent of training on trials that involve remapping. To examine this idea more quantitatively, we again quantified remapping as the d′ difference between the Remapping and Just-S1 conditions during saccade trials. The second factor was simply the number of trials completed. We then asked whether performance during fixation, on the Like-Remapping condition, was a function of remapping during saccades (factor 1), practice (factor 2), or both. If changes in visual perception arise through remapping and training, this performance should be better predicted by the two factors than by either one alone.

We tested this possibility with multiple regression, using data from all 10 observers (Materials and Methods). The results (Fig. 5) show that the value of model with both factors was much greater than either model with a single factor ( = 0.89, = −0.12, P = 0.001, = 0.39, P = 0.0002, = 1.31×10⁻⁵, P = 0.001). This was true despite the increased complexity of the two-factor model, as revealed by the Bayesian Information Criterion (BIC), which decreased for the two-factor model by 13.91 relative to the model that used only remapping strength as a predictor and by 17.51 relative to the model that used only training duration as a predictor. When we considered a more sophisticated model that took into account individual observers’ data at different stages of training (SI Appendix, Fig. S4), the importance of the two factors became even clearer (BIC difference of −28.7 relative to the model that used only remapping strength as a predictor and −55.89 relative to the model that used only training duration as a predictor; SI Appendix, Fig. S4). Thus, the two factors in the experiment (remapping and training) are both necessary to account for the learning effects observed in fixation.

Fig. 5.

Prediction of visual learning with linear model based on number of trials and remapping. A statistical model attempted to capture the performance of observers on the Like-Remapping trials relative to the Just-S1 trials during fixation. Each dot corresponds to a single observer’s results at the end of training. (A) The solid line along with its confidence bounds (dashed lines) corresponds to a model that considered only the number of trials completed as a predictor ( = 0.32, BIC = −21.47). (B) A model that considered only the strength of remapping as a predictor ( = 0.52, BIC = −25.07). (C) A model that used both factors as predictors ( = 0.89, BIC = −38.98).

Emergence of Learned Visual Integration.

Thus far, the results suggest that the improved ability to integrate visual information across spatial locations during fixation was shaped by experience and specifically by remapping during saccades. To visualize this evolution, we focused on the subset of observers who showed a statistically significant effect of remapping during saccades and who completed the full training protocol. As before, we subtracted performance on the Just-S1 condition from that obtained in the Like-Remapping condition. Fig. 6A compares the relative advantage for the Like-Remapping condition during the first eight sessions and the last eight sessions for each observer. In contrast to the saccade task (Fig. 3A), the remapping effect during fixation was not present for any observer during the early sessions (permutation test, P > 0.05), but emerged in all observers by the end of the training period (permutation test, P < 0.05). Indeed, by the last eight sessions of the experiment, the mean performance in the Like-Remapping condition had nearly reached that observed in the actual Remapping condition during saccade trials (Fig. 6B). In the two other control conditions during fixation (Orthogonal-control and Diagonal-control), there were no significant increases in performance in any bin, in comparison with the Just-S1 condition (Fig. 6B; P > 0.05, permutation test). For these observers, a linear regression (Fig. 6C) showed a clear trend for the Like-Remapping condition (slope = 0.013, P < 0.001, SE of the slope = 0.0012) but not for the others (Diagonal-Control: slope = 0.0001, P = 0.9, SE of the slope = 0.0009 and Orthogonal-Control: slope = −0.0013, P = 0.19, SE of the slope = 0.0009). The lack of an effect of learning in the Orthogonal-Control condition indicates that the exposure to remapping during vertical saccades (SI Appendix, Fig. S2A) was insufficient to induce perceptual changes (recall that across the entire testing period, vertical saccades comprised 20% of all trials.) Indeed, when observers were tested in the fixation condition after a comparable number of horizontal saccades, the advantage for the Like-Remapping condition had not yet emerged (SI Appendix, Fig. S2B).

Fig. 6.

Evolution of motion integration through training on fixation trials. (A) Comparison of motion integration in Like-Remapping in the fixation task. The d′ differences were computed by subtracting the d′ values in the Just-S1 condition from the Like-Remapping condition during the first eight sessions (green) and last eight sessions (blue) for the individual observer who showed a remapping effect during the saccade task. (B) In the fixation task, d′ differences were computed for each observer by subtracting the d′ values in the Just-S1 condition from those obtained in the conditions with two motion signals (S1 + S2) for the last eight sessions. Differences larger than zero indicate an increase in motion sensitivity in the S1 + S2 conditions (motion integration). Motion integration of 25% (d′ difference = 0.25) was found in the Like-Remapping condition for observers with remapping during the saccade task. The d′ difference between Like-Remapping and Just-S1 conditions was significantly different from that between the other S1 + S2 control conditions and Just-S1 (P < 0.05, WSR test, FDR corrected). The dashed line indicates the actual remapping effect during the saccade trials (SI Appendix, Fig. S1B) (C). Effect of training during the fixation task. For each observer, trials were divided into 27 steps: the first step contained the data for the first four sessions (∼150 fixation trials), and the step was moved forward 25 trials at a time. Each point indicates the mean difference between the S1 + S2 and the Just-S1 conditions across all observers who showed a remapping effect during saccades. These points are fitted with linear regression (solid lines). The black dashed lines around the regression lines show the confidence interval for the regression estimate. The red dashed line indicates the remapping regression during the saccade task (Fig. 3). Error bars show SEM. Asterisks: statistical significance (P < 0.05).

We note that these effects cannot be attributed to a learned change in fixation patterns across stimulus conditions. Although microsaccades would be expected to vary with shifts of attention (19–21), it was not possible for observers to predict the location of S2 in any trial and, given the brief nature of the stimulus, it was not possible to react to it with a microsaccade. Indeed, we found empirically that, at the end of the training protocol, the pattern of microsaccades did not differ across stimulus conditions (P = 0.57, Kruskal–Wallis test; SI Appendix, Fig. S5). We consider other possible observer strategies in the next section and in Discussion.

Remapping Strength Depends on the Time of Stimulus Presentation Relative to Saccade Onset.

The results thus far show that remapping can influence the integration of visual information: Training on a task that entails remapping during saccades leads to a consistent pattern of spatial integration during fixation. However, the example observers who did not show an appreciable effect of remapping during saccades did not appear to suffer worse performance overall: Their performance was in line with that of the other observers (compare Figs. 2A and 4A and Figs. 2B and 4B; absolute performance levels are shown in SI Appendix, Fig. S6).

We suspected that these differences reflected different strategies on the part of different observers. Specifically, because the range of stimulus presentation times was fixed (Fig. 1), observers with longer saccade latencies had more time to inspect the stimulus before initiating an eye movement. For these observers, the timing of the stimulus presentation relative to the saccade might have been inadequate for engaging remapping (22). Indeed, inspection of the pooled saccade latency data suggests that the observers who did not show remapping had far longer saccade latencies than the other observers (Fig. 7B).

Fig. 7.

Relationship between saccade latency and remapping. (A) Time course of motion integration in remapping. Motion sensitivity in the Remapping condition relative to that in the Just-S1 condition is pooled for all observers and plotted through time in 25-ms windows, according to the timing of motion signal offset relative to saccade onset. Negative time values indicate that motion signals in S1 and S2 appeared and ended before saccade onset. Filled and open triangles indicate, respectively, mean timing difference for observers with and without remapping during saccade trials. When the stimulus was presented longer than ∼175 ms before a saccade (vertical bold dashed line), no remapping was observed. Error bars show SEM. (B) Distribution of saccade latencies. Observers who showed a remapping effect during saccades (Left; median 235.66 ms) had shorter latencies than those who did not show remapping (Right; median 398.76 ms). (C) Remapping is linked to stimulus timing relative to saccade onset. We separated the saccade trials for all observers into those in which the time between the stimulus and the saccade was greater than and less than 175 ms. For observers who showed remapping in the full dataset, there was no increase in motion sensitivity in the Remapping conditions for the longer-latency saccades (mean d′ difference ± SEM = −0.09 ± 0.12), while for trials with shorter latencies we observed motion integration in the Remapping condition (mean d′ difference ± SEM = 0.28 ± 0.06). For observers who did not show remapping (right), no motion integration was observed in trials with long latencies (mean d′ difference ± SEM = −0.12 ± 0.11), but clear remapping was observed for the short latency trials (mean d′ difference ± SEM = 0.18 ± 0.065).

To characterize this relationship in more detail, we binned the saccade trials into 25-ms windows according to the time at which the motion stimuli appeared relative to saccade onset and plotted the difference in d′ values in the Remapping and Just-S1 conditions, pooling data across observers. As shown in Fig. 7A, the largest difference was observed when the motion signal(s) ended less than 125 ms before the saccade. This finding is consistent with the time course of presaccadic remapping effects observed in the parietal cortex (22, 23) and in previous psychophysical work (12). For stimuli that ended more than 125 ms before a saccade, no reliable difference between the Just-S1 and Remapping conditions was observed.

The triangle marks on the abscissa of Fig. 7A show the mean time difference between motion signal offset and saccade onset for individual observers. Clearly, the observers who did not show evidence of remapping during saccades (open symbols) had latencies that placed the motion stimuli outside the temporal range for which remapping is likely to have occurred in single neurons as well (22). Consistent with this idea, observers whose mean performance was strongest in the Remapping condition did not show the remapping effect for the minority of trials in which the time between the motion signal offset and the saccade onset was greater than 175 ms (mean d′ difference ± SEM = −0.037 ± 0.155; Fig. 7C). Furthermore, observers who did not, on average, show remapping, performed better on the minority of Remapping trials when they executed saccades less than 175 ms after the visual stimulus (mean d′ difference ± SEM = 0.185 ± 0.095; Fig. 7C). Thus, the differences among observers appear to be primarily due to the use of different strategies, which might reflect a trade-off between rapid saccade execution and visual stimulus integration (24).

Discussion

Our results confirm previous psychophysical reports that saccades facilitate the integration of visual signals across discrete regions of retinal space (Figs. 2 and 3) (10, 12, 13, 17). These results are usually interpreted in terms of the physiological phenomenon of receptive field remapping, for which many functional roles have been suggested: The integration of visual information across saccades (10); the preservation of stable visual perception (14); and the updating of oculomotor plans (5). Here we show that remapping can serve a very different purpose: It shapes the learning of new visual capabilities over much longer timescales. In our study, most observers who trained extensively on a task that required eye movements and visual motion discrimination acquired the ability to integrate visual signals across space without making eye movements. This ability emerged over the course of several weeks of training (Fig. 5), and it was evident only in those observers who showed signs of remapping during saccades (Fig. 6). Thus, our results suggest that remapping could underlie learning in tasks that require specialized visual integration.

Previous Work on the Influence of Eye Movements in Visual Learning.

Hebb (1949) (25) proposed that infants learn to see by making sequences of saccades to salient features in an object. Specifically, he suggested that, through repeated exposure to the same object, the brain learns to anticipate the change in stimulation caused by each saccade and to synchronize the corresponding cellular activity into assemblies or small functional networks. Later work on “scanpath” theory extended this idea to adult observers (26). This line of thinking in some ways anticipated the discovery of receptive field remapping, which renders neurons sensitive to the visual consequences of a saccade (4) and is associated with a synchronization of neural activity across the corresponding retinal locations (27).

At the neural level, it has been argued that remapping serves to bring neural responses to the presaccadic and postsaccadic stimuli into better temporal alignment (11, 27). Thus, during remapping, two neurons with receptive fields separated by the saccade vector often fire at nearly the same time. Such temporal alignment sets up the conditions for Hebbian rewiring or other spike-timing–dependent mechanisms (28), which are sufficient in the adult animal for the development of new horizontal connections in the visual cortex (29). Whether such connections are sufficient to account for the improved performance during fixation reported here remains to be seen.

The evidence that eye movements facilitate visual learning over long timescales is abundant but rather indirect. As mentioned above, it has frequently been reported that experts are able to extract visual information more quickly and with fewer eye movements than novices (1). This is well-documented in domains like radiology and chess, where it is often linked to “holistic” processing: Experts are able to see larger patterns in the image, and therefore they do not need to inspect individual elements. Even in laboratory experiments in which observers are exposed to a new visual shape, recognition performance is better when eye movements are allowed during the initial learning phase (3). Similar results have been found with visual search (30). Additional evidence for a role for eye movements in the acquisition of visual competence comes from studies in which visual space is distorted with a prism (31). In this case, straight objects appear curved, and observers learn to perceive them correctly with time, but only if saccades are allowed (32). These results suggest a role for effector movements in establishing the layout of external space, as has been found in other contexts (33) and other sensory modalities (34).

More generally, the field of perceptual learning has documented visual changes that occur at the perceptual (35, 36) and neural (37–39) levels following training in psychophysical tasks. These studies for the most part have involved training and testing during fixation in the absence of eye movements. One exception is the learning of new visual associations between stimuli seen across saccades (40–43), but in most cases it does not appear that saccades are necessary to elicit this type of learning (39–41). Other studies have shown that perceptual learning is sensitive to spatiotopic locations defined in part by eye position (44) and that perceptual learning can improve visual sensitivity during saccades (45).

To our knowledge, the only study to date that has looked for a specific role for remapping in visual learning was carried out by Rolfs and colleagues (46). They trained observers in conditions that should facilitate receptive field remapping and found no influence of saccades on perceptual performance for isolated stimuli at the remapped location. In contrast to our paradigm, there was no visual stimulus at the remapped location during training in the Rolfs et al. (46) study. Thus, our results and theirs might be reconciled under the idea that remapping serves the perceptual function of integrating stimuli across retinal locations (10), rather than transferring information from one location to another.

Nonretinal Influences on Visual Learning.

Our finding that saccades influence visual perceptual learning is compatible with other nonretinal influences on learning, such as reward (47) and attention (48). Although we did not reward observers for correct performance on the task, we did provide feedback after each trial. Thus it is possible that the improved performance on the remapping trials during saccades served to reinforce the learning of stimulus associations during fixation. In this case, the effect of remapping on visual integration would have been mediated by a kind of reward signal, as has been found in subliminal learning in other contexts (47).

In addition to reward influences, attention is particularly relevant because, as pointed out previously (10), remapping in our experiments must have been determined in part by attentional selection of the target stimulus S1 (Fig. 1). Specifically, attending to the cued location while planning a saccade likely caused an additional allocation of attention to the stimulus S2 at the remapping location (14). Remapping in single neurons is similarly sensitive to the distribution of spatial attention (49–51). From a functional standpoint, attentional selection of relevant image regions would presumably be necessary for the development of useful holistic processing (2) to prevent uninformative visual features from being learned during the acquisition of expertise.

Another role for attention is suggested by the premotor theory (52), in which attention shifts are implemented by the oculomotor system. Indeed, if observers were planning but not executing a horizontal saccade during the fixation trials, they would likely be able to attend to S1 and S2 stimuli simultaneously in the Like-Remapping conditions, thereby improving performance. The premotor theory might also explain the association between remapping effects, visual latency, and overall performance: Observers who showed longer latencies also had reasonable performance (Fig. 4B), despite little or no remapping (Fig. 2B). In the premotor theory, this would be expected if these observers were more effective at deploying covert attention to the S1 target; this would improve task performance, but because attention shares neural resources with saccade programming, it would lead to increased latencies (53). Regardless of the underlying mechanism, the data suggest a trade-off between the speed at which saccades are executed and the amount of visual information that is integrated within a fixation. Our task required only a single saccade per trial, but extending this logic to a more naturalistic setting, one could imagine that there is an optimal saccade frequency for a given task: Observers might make more saccades when exploring an image and fewer saccades when integration of visual information is crucial (24). There is some evidence that visual integration can be quite slow for tasks requiring the extraction of information from noisy stimuli (54), whereas for other tasks, visual integration is quite rapid (55). For the former case (noisy stimuli), the development of visual expertise seems to entail a shift from the exploratory regime to the integration regime.

Our results suggest that eye movements, and remapping in particular, might facilitate this shift. Previous studies have highlighted different roles for remapping, many of which entail a transfer of information across a retinotopic map of visual space (4–7). However, an alternative idea is that remapping facilitates the integration of visual information across space, with receptive field shifts being a side effect (10). In the existing psychophysical literature (including the current study), such integration seems to occur only when low-level stimulus features such as orientation or motion direction are matched across retinal locations (10, 13). However, an intriguing idea is that different low-level features might be integrated during remapping, provided they are consistent with the same global object. This would provide strong evidence for the idea that remapping shapes the learning of higher-order patterns in visual stimuli, as would be required to facilitate the acquisition of visual expertise.

Materials and Methods

Participants.

The experiment was designed to test the effects of prolonged visual experience on perception. This posed unusual difficulties in terms of observer recruitment and retention. We initially recruited 10 observers, and despite some attrition, 7 observers completed the full experiment (ages 21 to 28 y, all right-eye dominant, 4 women). Each participated in both the fixation and saccade tasks, and received $15 Canadian per session as compensation. None had undergone visuomotor psychophysical testing before, and all were naive to the objectives and background of the present study. All observers had normal or corrected-to-normal vision and provided informed written consent. Experiments were approved by the McGill University Research Ethics Board.

Setup and Procedures.

Participants sat in a quiet, dimly illuminated room with their heads stabilized with a chin and forehead rest. Before the main experiment began, participants were introduced to the instructions, experiments, and environment via a familiarization session consisting of about 200 saccade and 200 fixation trials. To establish each observer’s overall motion sensitivity, we tested them in a yes–no task (see below). After that, they began the main experiment, which consisted of 20 experimental sessions, each composed of 500 trials; two sessions per week were run on different days. Each session lasted 60 to 90 min, including a break. A session contained five blocks of trials; each block consisted of 100 trials. Saccade and fixation trials were interleaved randomly, but there were twice as many saccade as fixation trials. In total, participants completed 10,000 trials in the saccade and fixation tasks combined. Further details are given in SI Appendix. Fixation and saccade data, linear model data of all observers, and code used to conduct the reported analyses can be accessed via the Open Science Framework through the following link: https://osf.io/ytdxn/.

Tasks.

The structure of the saccade task, inspired by Szinte et al. (10), is shown in Fig. 1A. The fixation task (interleaved with the saccade trials) was identical to the saccade task, except that no saccade target was shown and no saccade was required.

Because individual observers differ in their ability to perform the motion discrimination task, we sought to maintain performance across observers within a reasonable range. We therefore ran a yes–no task (56–58) before the start of the main experiment to establish each observer’s ability to detect coherent motion. The structure of this task was similar to that of the fixation task, except that only one coherent motion stimulus (S1) was presented, and observers were asked to detect the existence, not the direction, of motion at the cued location. Details of all tasks are given in SI Appendix.

Data Analysis.

Saccades were detected by a velocity-based algorithm (10, 59). To detect microsaccades during fixation trials (SI Appendix, Fig. S5), we smoothed the raw-eye position traces with a median filter (15 samples) (60) and then differentiated the resulting traces to estimate velocity. Details of the eye movement analyses are given in SI Appendix.

To analyze psychophysical performance in the saccade and fixation tasks, we divided the dataset for each observer into five bins, each containing four sessions (2 wk) of data (Figs. 2 and 6). For each bin, we computed the motion sensitivity (d′) of observers according to their individual performance for each coherence in each of the four saccade and four fixation conditions. The performance was expressed as the fraction of correct trials (percentage of correct discrimination of the cued motion signal) and then transformed into a standard measure of sensitivity, d′ (10). Next, to calculate the SEM, we drew 1,000 bootstrap mean samples with replacement from the original behavioral data and calculated the d′ sample values for each condition. We then calculated the SE of the bootstrapped data (61). To assess the significance level between conditions, we conducted a permutation test (62).

Statistical comparisons of performance differences across S1 + S2 conditions were based on the one-tailed WSR. The Benjamini–Hochberg procedure (63) was used to correct for multiple comparisons by controlling the FDR at 5% across all tests (Figs. 3 A and B and 5C and SI Appendix, Fig. S1).

To examine the relationship between remapping, practice, and fixation effects, we used a linear model. We also considered a more sophisticated model that took into account data from multiple time points. To compare the results across the one- and two-factor models, we used the BIC (64). Details of the data analyses are given in SI Appendix.