The extrafoveal preview paradigm as a measure of predictive, active sampling in visual perception

Abstract

A key feature of visual processing in humans is the use of saccadic eye movements to look around the environment. Saccades are typically used to bring relevant information, which is glimpsed with extrafoveal vision, into the high-resolution fovea for further processing. With the exception of some unusual circumstances, such as the first fixation when walking into a room, our saccades are mainly guided based on this extrafoveal preview. In contrast, the majority of experimental studies in vision science have investigated “passive” behavioral and neural responses to suddenly appearing and often temporally or spatially unpredictable stimuli. As reviewed here, a growing number of studies have investigated visual processing of objects under more natural viewing conditions in which observers move their eyes to a stationary stimulus, visible previously in extrafoveal vision, during each trial. These studies demonstrate that the extrafoveal preview has a profound influence on visual processing of objects, both for behavior and neural activity. Starting from the preview effect in reading research we follow subsequent developments in vision research more generally and finally argue that taking such evidence seriously leads to a reconceptualization of the nature of human visual perception that incorporates the strong influence of prediction and action on sensory processing. We review theoretical perspectives on visual perception under naturalistic viewing conditions, including theories of active vision, active sensing, and sampling. Although the extrafoveal preview paradigm has already provided useful information about the timing of, and potential mechanisms for, the close interaction of the oculomotor and visual systems while reading and in natural scenes, the findings thus far also raise many new questions for future research.

Introduction

In everyday life, we use our visual system to explore the visual world and to guide our actions. In some cases, such as reading, using the internet, driving, or making a cup of tea, our fixations are highly controlled by the task (for review, see Land 2006). In other cases, we also use our eyes to look around and see our environment in a more free manner (Ito, Yamane, Suzuki, Maldonado, Fujita, Tamura, & Grn, 2017; Le Meur & Liu, 2015; Leopold & Park, 2020). The choice of where to look is typically determined by information present outside the fovea. Objects do not randomly appear on the fovea, but are carefully and intentionally placed there based on a preview of that object in extrafoveal vision by means of eye movements (see Figure 1, for illustration).

Figure 1.

Illustration of a change in fixation position within a natural scene using a saccadic eye movement (indicated by the purple arrow). The saccade target (the plate of green vegetables) is visible in extrafoveal vision prior to fixating that item for further scrutiny and to guide further action such as a grasping movement. The images are blurred except from the central area to imitate in a simplified way the difference in acuity between the fovea and extrafoveal vision.

Naturalistic vision contrasts strongly with the way in which visual perception is studied in the laboratory. The vast majority of studies constrain fixation to be still while a stimulus is flashed for a brief time on the screen. In such experiments, each stimulus appearance is a single event, involves a sudden onset, and is largely unpredictable in terms of what, when, and often where the stimulus will appear. The dominant theories in visual perception mirror the passive stimulation paradigms used in the vast majority of experiments. As perhaps best expressed in Marr's computational theory of vision, such theories begin with visual input (transduction of light energy to form an image), which triggers a feedforward cascade, an “evoked” response in visual processing areas that detect increasingly complex features. The output of such a system is the representation of a three-dimensional object that can then be recognized by matching it to a memorized representation. This sequential, context-free, feedforward system might be, perhaps, an ideal processing system if our experience mainly consists of 2D images of objects displayed in random order on a computer screen while we consistently focus our gaze on a fixation cross. Such a model does not, however, take advantage of the predictability inherent in natural viewing. Many models of visual perception do include some role of other aspects of more naturalistic viewing conditions, including some role for context, attention, or learning, but for the most part do not make oculomotor behavior a key component of the system.

Under this conception of visual perception, saccadic eye movements are often viewed as a “problem,” which must be solved (for review, see Melcher, 2011). The fundamental difference between naturalistic vision and laboratory experiments is well illustrated in demonstrations that mimic the actual retinal input that results from active gaze behavior while the eye is in fact stationary. Such demonstrations illustrate how saccades create a spatial transformation (a new image is presented at the center of the retina) and a temporal discontinuity. This raises the question of why visual perception seems so spatially stable and temporally continuous (for reviews, see Melcher 2011; Melcher & Colby, 2008). Alternatively, a different question could be why our understanding of visual processing does not already include eye movements as an essential feature of visual processing (see An active-vision interpretation of preview effects on theories incorporating eye movements in vision, below). Perhaps the simple answer is that we are usually not aware of making saccades in the first place and hence many vision researchers ignore them and focus instead on the level of experimental control offered by maintained fixation.

The parafoveal preview paradigm in reading

Experimental protocols to study the influence of extrafoveally or, more specifically, parafoveally¹ available visual information have been used for decades in the field of reading research (Wagner, 1918; for reviews see Rayner, 1998; Schotter, Angele, & Rayner, 2012; for a meta-analysis see Vasilev & Angele, 2017). In the 1970s and 1980s, Keith Rayner and others introduced gaze-contingent experimental protocols that allowed researchers to change letters and words of a text during active reading (Balota, Pollatsek, & Rayner, 1985; McConkie & Rayner, 1975; Rayner, 1975; Rayner, 1998; Rayner, McConkie, & Ehrlich, 1978). Other research groups around the globe worked on very similar ideas at the same time (Buurman, Roersema, & Gerrissen, 1981; Ikeda & Saida, 1978). Using eye-tracking methodology, it was possible to change a word in a text presented at a computer screen at exactly the moment in time when the saccadic eye movement to this word was executed (Figure 2, top panel). Applying an imaginary line between the preview and target word that, once crossed by the eyes, triggered a change on screen, this paradigm was subsequently aptly termed the “boundary paradigm” (cf. Schotter et al., 2012). The original purpose of this method was to determine the perceptual span, that is the extent of the area of the visual field from which text can be read (Buurman et al., 1981; Ikeda & Saida, 1978; McConkie & Rayner, 1975; Rayner, 1975). Although still debated to some extent, for in-depth semantic processing of a word, this perceptual span turned out to be mostly limited to the area of foveal vision (Hohenstein & Kliegl, 2014; Hohenstein, Laubrock, & Kliegl, 2010) with the extent of the area thought to be somewhat larger for processing of other types of information such as lexical, morphological, phonological, and orthographical (Schotter et al., 2012).

Figure 2.

Illustration of a version of the extrafoveal preview paradigm. In the top panel, the influence of an extrafoveal preview, during reading of a word list, is tested by comparing a condition in which there is a valid preview (the same word is presented both before and after the saccade) and an invalid preview in which the word is changed during the saccade. Further details of this type of study are given in The parafoveal preview paradigm in reading of the manuscript. The bottom panel shows a variant using face stimuli, as described in The parafoveal preview paradigm in vision research of the manuscript. In each of the two conditions, participants make a cued saccade to a face. Two conditions are illustrated, in which the face orientation remains the same (valid) or changes (invalid) across the saccade. Note that both upright and inverted faces are usually presented as both targets and previews (Huber-Huber et al., 2019). The colored circles represent gaze position before (purple) and after (green/red) the eye movement. The purple arrow represents the saccade. Face images are used with permission from the Stirling PICS database http://pics.stir.ac.uk.

Reading research provides one crucial finding for theories of active vision: the preview benefit effect (Rayner et al., 1978; also see Hutzler, Fuchs, Gagl, Schuster, Richlan, Braun, & Hawelka, 2013; Hutzler, Schuster, Marx, & Hawelka, 2019; Kliegl, Hohenstein, Yan, & McDonald, 2013; Marx, Hawelka, Schuster, & Hutzler, 2015; for a debate on preview benefits versus preview costs). Changing a word exactly at the time that a saccade is made to that word, that is, manipulating the extrafoveal preview of a word, has a number of behavioral consequences once the eyes land on the word and processes it foveally. Compared to a condition without any trans-saccadic change (valid preview), a change (invalid preview) leads to longer fixation durations, longer latency of naming the word, longer responses in a lexical decision task, and other variables that are considered to measure levels of processing visual, linguistic, and semantic information (Schotter et al., 2012). The exact characteristics of this preview effect vary with the degree of consistent visual similarity between the pre-saccadic preview and the post-saccadic foveal word, that is, with the specifics of the parafoveal mask (Gagl, Hawelka, Richlan, Schuster, & Hutzler, 2014; Vasilev & Angele, 2017). The effect also depends on the eccentricity of the previewed word, in the sense that a word in the far visual periphery will no longer have a measurable effect. At parafoveal eccentricities, however, the effect has been found in many contexts, tasks, and languages (Rayner, 1998; Schotter et al., 2012; Vasilev & Angele, 2017). In addition to the eccentricity at which a stimulus is presented, the preview effect varies with the reader's visual acuity, their perceptual span, and appears to be affected by crowding to some extent (Frömer, Dimigen, Niefind, Krause, Kliegl, & Sommer, 2015; Risse, 2014). Besides these visual factors, a number of linguistic aspects have also been shown to influence the preview effect. For instance, there is now growing evidence that, in addition to preview validity effects, there are reliable preview difficulty effects (Andrews & Veldre, 2019; Schotter, 2018) associated not only with the preview word's frequency (Risse & Kliegl, 2014; Schotter, Leinenger, & von der Malsburg, 2018; Schotter & Fennell, 2019; Schotter, von der Malsburg, & Leinenger, 2019) but also with its semantic plausibility (Schotter, 2013; Schotter, Lee, Reiderman, & Rayner, 2015; Schotter & Jia, 2016; Veldre & Andrews, 2016; Veldre & Andrews, 2017; Veldre & Andrews, 2018; Yang, Li, Wang, Slattery, & Rayner, 2014; Yang, Wang, Tong, & Rayner, 2012). In particular experimental configurations, for instance, when additionally manipulating high versus low lexical frequency words, preview effects can even tend to reverse in certain experimental contrasts (Schotter & Leinenger, 2016).

Despite the historical convention of calling it a preview “benefit” effect (Schotter et al., 2012), one line of research sheds doubts on whether the term benefit is eventually a good description of the phenomenon. Taking a closer look and searching for a third neutral no preview baseline condition besides valid and invalid preview conditions (Hutzler et al., 2013) as well as using an incremental boundary paradigm, in which the visual saliency of the preview is incrementally varied (Hutzler et al., 2019), Hutzler and colleagues arrived at the conclusion that the term benefit is misleading because the underlying effect is actually a mixture of benefits and costs associated with the invalid parafoveal preview (Marx et al., 2015). This notion is also supported by Kliegl and colleagues (2013) and has been elaborated in the context of a meta-analysis (Vasilev & Angele, 2017). For this reason, we use the more neutral term “preview effect” to refer to an experimental contrast between a valid preview condition in which the same stimulus that is seen extrafoveally is also seen foveally (i.e. no trans-saccadic change), and an invalid preview condition in which an extrafoveally seen stimulus changes during the saccade that brings it into foveal vision.

In many cases, the preview effect results in better performance for valid compared to invalid preview conditions. A target word that remains the same across the eye movement that brings it into foveal vision is processed more efficiently than a target word that changes during the eye movement. This effect demonstrates that input from the extrafoveal visual field before a saccade matters for subsequent foveal processing, whether this is shown in a classic preview benefit or in terms of a cost. Overall, research is consistent with the idea that foveal perception is affected by imminent extrafoveally available visual input. As we will see later on in An active-vision interpretation of preview effects, this finding can nicely be accommodated in notions of active vision in which extrafoveal and foveal processing are linked by the occurrence of saccadic eye movements.

More recently, the neural correlates of the parafoveal preview effect in reading have been investigated by co-registering eye-tracking data with electroencephalography (EEG; Degno, Loberg, Zang, Zhang, Donnelly, & Liversedge, 2019; Dimigen, Kliegl, & Sommer, 2012; Dimigen, Sommer, Hohlfeld, Jacobs, & Kliegl, 2011; Sereno & Rayner, 2003; for a recent review see Himmelstoss, Schuster, Hutzler, Moran, & Hawelka, 2020). This technique provides a highly time-resolved magnifying glass on what happens around the time of saccadic eye movements. After the onset of fixation on the target word, event-related potentials time-locked to that fixation onset, aka fixation-related potentials (FRPs), showed an influence of the parafoveally previewed word on subsequent foveal processing in two stages: first at around 200 to 280 ms at occipitotemporal sites and second at around 350 to 400 ms at central sites resembling the N400 component. Both effects consisted in a stronger positivity for a valid preview, that is, no change in the word during the saccade than for an invalid preview, that is, a change during the saccade, which could be from a semantically related or unrelated word to the actual target word (Dimigen et al., 2012). Interestingly, the preview effect in FRPs was statistically the same for semantically related and unrelated preview words, which suggests that semantic information cannot be extracted from the parafovea and providing further evidence to the long-standing debate about semantic parafoveal preview effects (Schotter et al., 2012; Vasilev & Angele, 2017). Importantly, the occipitotemporal preview positivity around 200 ms post fixation onset did not depend on the participants’ awareness of the trans-saccadic display changes in the invalid condition.

Dimigen and colleagues’ preview effect in FRPs (2012) has subsequently been replicated across a range of experimental settings. Originally found in a protocol with lists of nouns presented within one line of text (Dimigen et al., 2012), which allows for rather natural eye movements but is still not a completely natural reading setting because real world texts rarely consist of only nouns, it was later replicated with meaningful sentences presented one per trial in a single line of text (Degno et al., 2019). In addition, the preview positivity was also replicated in an even more restricted reading setup with a Chinese language (Li, Niefind, Wang, Sommer, & Dimigen, 2015). In this study, Chinese sentences were presented in character triplets step-by-step at screen center while participants maintained constant fixation at the screen center. Despite showing slightly different ERPs, the preview effects obtained with this design was otherwise very similar to the original findings. In the following years, the preview positivity was further replicated with various setups including passive reading conditions without active eye movements (Kornrumpf, Niefind, Sommer, & Dimigen, 2016). In sum, this growing body of research shows that the preview effect is independent of a particular language, writing system, and the exact reading setup which already alludes to the idea that the preview effect could reflect a more general mechanism of the visual system.

The preview effect in natural viewing depends, by definition, on saccadic eye movements. Under laboratory conditions, however, it is possible to simulate to some extent the change in visual input due to a saccade without actually moving the eyes. Such a control condition is important for any theory of active vision because it is possible that the preview effect resulted simply from passively experiencing the repetition of visual input first in the extrafoveal and then again in the foveal visual field, which would mean that the preview effect would be essentially a type of priming effect that is independent of saccade execution and strictly speaking unrelated to active vision. Alternatively, the preview effect can depend critically on actual saccade execution shifting the stimulus from the extrafoveal to the foveal visual field, which would mean that the preview effect is a sign of active vision (see An active-vision interpretation of preview effects).

Up until now, several studies have contrasted active with passive reading conditions and, with some exceptions, their results speak overall rather for an explanation of the preview effect in terms of active vision instead of priming. An exception is one of the early reading studies, which reported very similar preview effects on target naming times in an active eye-movement and a passive control condition (Rayner et al., 1978). In contrast, the preview positivity in FRPs was found to be larger in an active than in a passive reading condition in which the text was presented stepwise from right to left in word-triples moving across central fixation (Kornrumpf et al., 2016; see also N. Li et al., 2015). Niefind and Dimigen (2016) even reported no preview effect for a passive reading condition. The discrepancies in the passive reading conditions across studies might be attributed to differences in how exactly passive visual input was simulated. In theory, different degrees and proportions of valid and invalid trials could also account for the variation in the results. We have recently shown that the behavioral benefits of a valid preview are strongly reduced when valid preview conditions are rare (Huber-Huber & Melcher, 2021). In sum, preview effects are generally found also in passive reading conditions, however, they tend to be less pronounced than in active reading conditions.

Apart from studies on the preview effect, similar research using concurrent EEG and eye-tracking supports the idea that active and passive reading are largely comparable, yet active reading tends to enhance some experimental effects. For instance, seeing a new compared to an old (repeated) word within a list of words has a very similar impact on FRPs in an active and a passive reading condition (Hutzler, Braun, Võ, Engl, Hofmann, Dambacher, Leder, & Jacobs, 2007). Certain manipulations, like letter spacing, elicit less pronounced effects in passive compared with active reading (Weiss, Knakker, & Vidnyánszky, 2016). In a related set of studies in which participants read complete sentences, the fixation-locked EEG signal showed largely similar event-related responses for active versus passive reading setups but neural oscillations differed in certain frequency bands (Hagoort, Hald, Bastiaansen, & Petersson, 2004; Metzner, von der Malsburg, Vasishth, & Rösler, 2015).

The obvious explanation for more pronounced effects under active reading conditions is that active vision involves different processes and mechanisms than just passively viewing the same visual input, such as the efference copy and saccadic remapping (Cavanaugh, Berman, Joiner, & Wurtz, 2016; Melcher & Colby, 2008; Sun & Goldberg, 2016) as well as related spatially and temporally specific changes (a combination of enhancement and suppression) of visual processing around the time of saccades (e.g. Buonocore, Fracasso, & Melcher, 2017; Deubel & Schneider, 1996; Huber-Huber, Steininger, Grüner, & Ansorge, 2021; Kowler, Anderson, Dosher, & Blaser, 1995; Kroell & Rolfs, 2021; Li, Barbot, & Carrasco, 2016; Moore, Armstrong, & Fallah, 2003). Therefore, preview effects in reading are probably best explained by a combination of the two hypotheses mentioned above. The part of the preview effect which can be observed under passive viewing condition can be explained by priming, but for a complete explanation it requires a theory of active vision taking into account that the motor act of making an eye movement fundamentally affects perception. As described further in An active-vision interpretation of preview effects, the extrafoveal preview effect can be used to distinguish between different factors and mechanisms in active perception and help to test specific predictions from these different theories of active vision.

The effect of imminent, parafoveally available input on foveal perception during reading has also been addressed without employing the boundary paradigm, that is without any artificial changes in visual input apart from the changes introduced by the participants’ eye movements. For instance, parafoveal-on-foveal effects describe the impact of the upcoming parafoveally visible word on processing the currently fixated word. The logic of parafoveal-on-foveal effects corresponds to what Stewart and colleagues (Stewart, Valsecchi, & Schütz, 2020) in their review of the relation between foveal and peripheral vision call foveal-peripheral interactions during fixation. The guiding question is similar to the question behind the preview effect and the boundary paradigm, in that it is asking about the size of the reading span and about what kind of information can be extracted from words in the parafovea. In particular, the question whether semantic information from word n+1 affects processing of word n has fueled many studies using concurrent EEG and eye-tracking (Baccino & Manunta, 2005; Barber, van der Meij, & Kutas, 2013; Dimigen et al., 2012; Kretzschmar, Bronkessel-Schlesewsky, & Schlesewsky, 2009; Li et al., 2015; Mirault, Yeaton, Broqua, Dufau, Holcomb, & Grainger, 2020; Simola, Holmqvist, & Lindgren, 2009; Stites, Payne, & Federmeier, 2017; for a review of parafoveal-on-foveal effects in the eye-tracking literature see Schotter et al., 2012; for an example of parafoveal-on-foveal effects in functional magnetic resonance imaging (fMRI) and eye-tracking data see Vignali, Hawelka, Hutzler, & Richlan, 2019). With respect to theories about active vision, however, it remains yet to be answered whether and, in case, how parafoveal-on-foveal effects are modulated by active compared to passive viewing.

Up until now, most studies on reading have restricted gaze behavior in one way or the other by presenting at most a single sentence on a screen. Only a handful of studies have allowed completely unconstrained gaze behavior during reading of more than a single sentence during recording of EEG and/or eye-tracking data (Henderson, Luke, Schmidt, & Richards, 2013; Takeda, Sugai, & Yagi, 2001). However, further advancements in eye-movement artifact correction that almost completely remove even the residual saccadic spike potential present in ICA-corrected EEG and eye-tracking data (Dimigen, 2020; Keren, Yuval-Greenberg, & Deouell, 2010) and in using deconvolution methods to deal with the inherently overlapping neural activity across subsequent fixations (Cornelissen, Sassenhagen, & Võ, 2019; Dimigen & Ehinger, 2019; Ehinger & Dimigen, 2019; Kristensen & Guérin-Dugué, 2017; Smith & Kutas, 2015) will render naturalistic reading studies much more feasible in the near future (Pfeiffer, Hollenstein, Zhang, & Langer, 2020).

Similar to using EEG and eye-tracking to study neural activity in natural active-reading setups, researchers have combined fMRI with concurrent eye-tracking to investigate various aspects of reading in naturalistic protocols (Henderson, Choi, Lowder, & Ferreira, 2016; Richlan, Gagl, Gawelka, Braun, Schurz, Kronbichler, & Hutzler, 2014; Schuster, Hawelka, Hutzler, Kronbichler, & Richlan, 2016; Schuster, Hawelka, Richlan, Ludersdorfer, & Hutzler, 2015). Schuster and colleagues (Schuster, Himmelstoss, Hutzler, Richland, Kronbichler, & Hawelka, 2021), for instance, investigated effects of predictability and semantic congruency of the final word in sentences to determine brain networks associated with predictive processing during reading. Similar brain networks might be involved in processing the predictability and semantic congruency of extrafoveal and foveal input during reading. However, to our knowledge there is no study yet using a gaze-contingent preview paradigm with concurrent fMRI and eye-tracking. The reason for that gap in the literature may be related to technical aspects of recording eye position in the MRI machine, but is also likely due to the comparatively high temporal resolution required to measure neural preview effects time-locked to fixation onsets in contrast to the relatively low temporal resolution of fMRI.

The parafoveal preview paradigm in vision research

At the time of writing the current manuscript, a quick literature search for “parafoveal preview” identified 185 published papers (PubMed, October 2020). Almost all of these papers study some aspect of reading. Whereas reading is undoubtedly an important field of study and indeed a critical part of modern life, it is a special, constrained behavior that has to be learned. Reading is not shared with other animals and, even in humans, was developed in its current form only within roughly the past five thousand years accompanying the development of written scripts. Compared to looking around a scene or using the eyes to guide locomotion or gaze, reading differs in many ways from how we use our eyes in naturalistic viewing. The logic of the preview effect, however, is the same for both reading and vision in general. For every single saccadic eye movement that we make, visual input is available extrafoveally before the saccade. Therefore, the question arises whether, for visual perception in general, imminent extrafoveal information affects foveal processing in principle in the same way as it does during reading (cf. Stewart et al., 2020).

Interestingly, the preview effect in reading has been linked to individual development of reading skills, which suggests that people learn to use parafoveal information when they learn to read. Marx and colleagues (2016) examined school children's reading skills, such as reading fluency and the efficiency of phonological decoding, and found a positive relation to the size of the preview effect. Moreover, these individual reading skills better predicted the preview effect than reading experience in terms of school grade (Marx, Hutzler, Schuster, & Hawelka, 2016). Their findings might suggest that the preview effect in general only develops in the course of learning to read. However, considering the evidence for the influence of parafoveal perception in vision in general, although with adult participants, it seems still likely that children (or illiterate adults) do not start with zero preview effect and learn to use upcoming parafoveal information only when they learn to read. A growing number of studies show that when there are regularities in the world that license prediction, our brains tend to pick up on, and incorporate, these predictions (de Lange, Heilbron, & Kok, 2018), which would seem to include extrafoveal previews. Instead, it seems more plausible that, while people learn to read foveally, they also develop reading skills in the parafovea, which provides the basis for a preview effect involving words. A preview effect for other type of visual input would seem likely to be present beforehand, learned through the predictable link between the saccade target prior to the saccade and the foveal stimulus after the saccade (Stewart et al., 2020). This hypothesis has yet to be tested by, for instance, demonstrating that children show visual preview effects with non-word stimuli even before they learn to read.

Examining visual perception outside the specific context of reading, a handful of studies drawing upon the invisible boundary paradigm do provide evidence for preview effects with non-word stimuli (e.g. Henderson, Pollatsek, & Rayner, 1987; Henderson, Pollatsek, & Rayner, 1989; Pollatsek, Rayner, & Collins, 1984). Interestingly, many of the more recent studies co-registered EEG with eye-tracking and used human faces as experimental stimuli (Buonocore, Dimigen, & Melcher, 2020; de Lissa, McArthur, Hawelka, Palermo, Mahajan, Degno, & Hutzler, 2019; Edwards, VanRullen, & Cavanagh, 2018; Huber-Huber, Buonocore, Dimigen, Hickey, & Melcher, 2019; Huber-Huber & Melcher, 2021), with one other study using gratings or Gabor patches (Ehinger, König, & Ossandón, 2015). Figure 2 (lower panel) illustrates an example. These studies manipulated the extrafoveally presented stimulus during the saccade that was directed toward it. In general, in trans-saccadic change conditions (invalid preview), the extrafoveally presented stimulus changed, for instance, from an inverted face to an upright face, during the saccade directed toward it. In trans-saccadic no-change conditions (valid preview), the extrafoveal stimulus remained the same. These studies consistently found generally better performance in behavioral tasks on the target stimulus and reduced FRP components in valid compared to invalid preview conditions.

For instance, in the study of Huber-Huber and colleagues (2019), participants showed more efficient performance in a face tilt discrimination task on a post-saccadic face stimulus with same-orientation compared to a different-orientation extrafoveal preview face. In addition, the fixation-locked face-sensitive N170 component was reduced. Buonocore and others (2020) also found a similar effect comparing intact and phase-scrambled preview faces. In contrast, changing the gender of the face during the saccade had no effect on the post-saccadic FRP response. The reduction in the FRP was similar for both face previews (same and different gender) compared to the phase-scrambled preview, which suggests that the preview effect depends on the information that participants can actually extract from the extrafoveal preview (Buonocore et al., 2020). Corroborating the preview effect, De Lissa and colleagues (de Lissa et al., 2019) found in their experiment 2, that a phase-scrambled preview face enhanced the fixation-related N170 compared to an intact preview face. However, some of their control conditions with images of watches showed rather unexpected results, consisting in N170-like effects opposite to what one would expect. With a phase-scrambled preview, watches elicited a more negative N170-like response upon fixation than faces, which suggests that the watches appeared more face-like. This finding could have multiple, perhaps even methodological reasons and clearly deserves further investigation. In addition, their experiment 1 showed that the face-inversion and face-versus-watch effects in the N170 component were present in response to extrafoveal presentation only but abolished upon subsequent fixation. In contrast, the other studies mentioned before still showed N170 effects upon fixation (Buonocore et al., 2020; Huber-Huber et al., 2019) as well as N170 effects upon re-fixation (Auerbach-Asch, Bein, & Deouell, 2020), which could suggest that the details of the experimental protocol, among other things, probably related to visual eccentricity and size of the stimuli, modulates face-processing effects in FRPs. Considering the recent evidence for the influence of statistical regularities on trans-saccadic preview effects (Huber-Huber & Melcher, 2021), albeit in behavior, it is possible that the absence of invalid trials in De Lissa's experiment 1 rendered the recognition of the extrafoveal face as face upon fixation obsolete and therefore did not trigger an N170 after fixation. In other words, foveal recognition might be less important if the stimulus could be seen clearly enough by extrafoveal preview, and if there are no invalid trials (e.g. change from a scrambled to an intact face during the saccade) that reduce predictability and validity of the extrafoveal stimulus. Additional evidence for this idea comes from a study looking at prediction of the spatial location of saccadic targets (Notaro, van Zoest, Altman, Melcher, & Hasson, 2019). When the proportion of trials with a predictable saccade target location was high, participants tended to already anticipate the target location and their gaze position was shifted toward the likely target location. This effect disappeared when location was less predictable. Moreover, the comparatively large stimulus size could have contributed to abolishing the preview effect by exciting largely the same receptive fields by the initial preview presentation and the subsequent fixation and therefore leading to strong neural adaptation and the lack of a second face-selective response upon fixation (cf. Grill-Spector, Henson, & Martin, 2006).

There is one interesting discrepancy between the preview effect in reading and the preview effect in vision investigated with face stimuli. Whereas the preview effect in reading is not only observed in FRPs but traditionally and more prominently in fixation durations on the target word, with first fixations on the target being longer for invalid than for valid preview conditions (Schotter et al., 2012), such an effect appears to be absent for faces, at least with a face-tilt discrimination task (Huber-Huber et al., 2019). This discrepancy might highlight that, although there are many analogies for preview effects in reading compared to non-reading tasks, the preview effects in both areas of research are probably not driven by exactly the same mechanisms. Alternatively, this discrepancy might also reflect the difference in task requirements between many reading tasks and those studies measuring preview effects with objects. It is possible that the amount and extent of task-relevant information that can be acquired from the preview and from the first target fixation determines the duration of the first target fixation in the sense that, if the task was simple enough, the duration of the first fixation might even be unaffected.

The use of EEG to study preview effects also goes beyond comparing the magnitude and timing of specific components, providing valuable information about the nature of preview effects in natural vision. In two experiments, we found evidence in the EEG signal for an interaction between the pre-saccadic preview stimulus (upright or inverted face) and the post-saccadic target (Huber-Huber et al., 2019). This effect occurred by 50 ms (experiment 2) or 90 ms (experiment 1) into the new fixation. In terms of characterizing the nature of preview effects, the entire pattern of results from that study suggests that we should consider at least three temporal stages: (1) prediction about the saccadic target, (2) integration of pre-saccadic and post-saccadic information starting at around 50 to 90 ms post fixation onset, and (3) post-saccadic facilitation of rapid categorization (see also Edwards et al., 2018, discussed below).

Interestingly, the rather early post-saccadic influence of an extrafoveal preview with face stimuli, described above, contrasts with a trans-saccadic change effect reported with Gabor patches (Ehinger et al., 2015). In a gaze-contingent design, Ehinger and colleagues (Ehinger et al., 2015) manipulated a small central area of a Gabor patch during the saccade. This trans-saccadic change affected the fixation-locked ERP around 200 ms post fixation on the Gabor patch. The onset of this effect is clearly later than the face-related preview effects found with the N170 component. Thus, this suggests that a supposedly low-level visual feature affects post-saccadic perception at a later point than a comparatively higher-level visual feature, such as facial configuration. This discrepancy highlights that the time point at which pre-saccadic extrafoveal input impacts on post-saccadic processing does not simply depend on the state of the extrafoveal input within a low-to-high-level visual processing hierarchy but may be adapted to the specific experimental context, including the salience of the feature in question, magnitude of the change, and the degree of predictability, among other factors. Similarly, visual eccentricity and the concomitant variation in sensitivity to spatial frequency, color, and visual acuity in general is also expected to strongly impact on the strength, timing, and shape of preview effects.

The preview effect in the fixation-locked EEG is further supported by a decoding study involving faces and houses (Edwards et al., 2018). Edwards and colleagues (Edwards et al., 2018) were able to decode the post-saccadic stimulus earlier in a condition where the stimulus had remained the same across the saccade compared to a change condition in which it changed from a face to a house or vice versa. The onset of decodability was roughly around the time of the preview effect mentioned above (Huber-Huber et al., 2019). Importantly, the earlier decoding in the valid condition reported by Edwards and colleagues is consistent with the post-saccadic facilitation of perceptual processing with valid compared to invalid previews.

As for the preview effect in reading, which obviously depends on eccentricity, there is also some indirect evidence from a non-human primate study that the preview effect in vision depends on eccentricity as well (Krishna, Ipata, Bisley, Gottlieb, & Goldberg, 2014). Krishna and colleagues (2014) had two monkeys perform a visual search task without gaze restrictions for a target T among distractor crosses. The time difference from fixation on target to button press for correct response was shorter if the preceding saccade in the free viewing gaze sequence that led to target fixation had ended closer to the target. This result suggests that the distance of the previewed stimulus from current fixation matters for processing that stimulus upon fixation. If the stimulus is closer to the current fixation, then subsequent foveal processing of that stimulus is facilitated. A systematic investigation into the precise relationship between visual eccentricity, stimulus material, and the presence, absence, and size of the preview effect is, however, still missing. This relationship is expected to depend on a great variety of interacting factors, including the systematic change in visual acuity across the visual field and the concomitant changes in sensitivity to spatial frequency, color, motion, and other visual features (J. Freeman & Simoncelli, 2011; Koenderink, Valsecchi, van Doorn, Wagemans, & Gegenfurtner, 2017; Wallis, Bethge, & Wichmann, 2016 for further details about how perception changes across visual eccentricity) as well as crowding, attention, predictability, and potentially even semantic aspects of the stimulus and task requirements.

The issue of retinal eccentricity remains an open question for parafoveal preview effects, also due to the fact that words and objects have different visual properties and are thought to involve somewhat different processing streams. At present, there are suggestions that object categorization may take place even quite far outside of the fovea. For example, Thorpe and colleagues (Thorpe, Gegenfurtner, Fabre-Thorpe, & Bülthoff, 2001) measured animal detection at eccentricities ranging from 0 degrees (central/foveal) to 75 degrees eccentricity. Critically, detection of an animal in a natural scene was still above 70% at 60 degrees eccentricity. This suggests that, in principle, some information would be available to support extrafoveal preview effects even at such far eccentricities. It is important to note, however, that with such eccentric extrafoveal presentations participants reported guessing, suggesting that implicit information played a role in their performance. A similar “perception without awareness” effect at large eccentricities was reported by Boucart and colleagues (2010) with pictures. They utilized both implicit (priming) and explicit (old/new recognition) measures in their study. They found priming effects and good old/new recognition performance for colored photographs of objects at 30 degrees eccentricity.

The pattern of results with stimuli at far eccentricities (Boucart et al., 2010; Thorpe et al., 2001) raises the question of whether the extrafoveal preview effects for objects, such as faces or pictures of animals, follows the pattern of implicit or explicit measures. If these effects rely only on the existence of information in the visual system (following the line of reasoning in studies using decoding of object category, as in Edwards et al., 2018), then preview effects might be found for stimuli far outside the fovea. In contrast, if extrafoveal preview effects are based on explicit categorization of the peripheral stimulus, then these preview effects might disappear when the preview is relatively far outside the fovea. Alternatively, the more relevant variable might be visibility, which would tend to be reduced when stimuli are shown further outside the fovea, but could be at least somewhat matched at different retinal eccentricities by varying the size, contrast or spatial frequency content. Similarly, the presence or absence of other nearby stimuli, and crowding, could also play more of a role for eccentric stimuli in natural viewing conditions, compared to laboratory studies in which single images are presented in isolation on an otherwise blank display screen.

The existence of this preview effects outside the reading domain demonstrates that foveal vision in general, not only during reading, is affected by imminent extrafoveal vision. Moreover, the spatiotemporally different profiles with different stimuli and in different contexts precludes the idea of a single trans-saccadic mechanism yielding these effects. At present, the pattern of results is consistent with the notion that trans-saccadic preview effects result from activity within the regions of the visual system where the feature that changes across the saccade is processed in the first place (Melcher & Colby, 2008). Moreover, preview effects apparently depend on what participants see extrafoveally and how this input is different from the subsequent foveal input.

The impact of imminent extrafoveal information on post-saccadic visual processing leads to one of the core questions of trans-saccadic research: To what extent, how, and when exactly is the extrafoveal input from the future target location available in the visual system before the saccade (or at the beginning of the new fixation)? Research on predictive remapping provides a wealth of evidence for the hypothesis that extrafoveal information is transferred across the visual system in sync with saccadic eye-movements (Melcher, 2007; Melcher & Morrone, 2015; Sun & Goldberg, 2016). However, the complex pattern of results has yet to be combined into a theory that can account for all of the findings. In a recent study, for example, Fabius and colleagues measured the information available about the spatial frequency of a target before and after a saccade (Fabius, Fracasso, Acunzo, Van der Stigchel, & Melcher, 2020). In this study, there was no evidence for pre-saccadic decoding of the spatial frequency of target gratings in post-saccadic retinotopic coordinates. Instead of tracing a neural signature of pre-saccadic remapping, the data was consistent with a gradual switch from pre- to post-saccadic representations of the target, with the post-saccadic representation commencing only after the saccade. To our knowledge, there is only a single case study which reports successful decoding of extrafoveal stimuli before saccades (Boring, Richardson, & Ghuman, 2020). In that case study, the authors reported above chance decoding of face versus no-face stimuli in co-registered electrocorticography and eye-tracking data obtained from a single participant in a free-viewing paradigm already at around 200 ms before saccade onset. It would be interesting to see whether this particular finding can be replicated in a larger sample. In addition, experiment 2 of the study by Huber-Huber and colleagues (2019) provides some evidence that the pre-saccadic signal is modulated by upcoming events. In this experiment, the proportion of valid and invalid trials was manipulated to render either valid or invalid trials more frequent in separate parts of the experiment. In line with this manipulation, neural activity shortly before the saccade associated with the current extrafoveal face orientation started to decline earlier if the post-saccadic face was more likely to be of a different orientation. Such a pattern of results is consistent with pre-saccadic prediction and the active maintenance of expected post-saccadic information from before to after the saccade.

In addition to preview effect designs, free-viewing experiments can provide evidence for the presence of stimulus-related information about an extrafoveal target before the saccade to that stimulus. Manipulating the low-level saliency of a set of Gabor patches revealed a relationship of post-saccadic stimulus information with pre-saccadic EEG activity in free-viewing visual search (Van Humbeeck, Meghanathan, Wagemans, van Leeuwen, & Nikolaev, 2018). Interestingly, object-scene congruency was reflected in FRPs already one fixation before foveating the actually congruent of incongruent object in a natural scene (Coco, Nuthmann, & Dimigen, 2019). These two examples demonstrate the influence of extrafoveal visual input on the current fixation in free-viewing experiments.

Future work may take advantage of the gaze-contingent paradigms used previously to study gaze control, visual search and memory (Cajar, Engbert, & Laubrock, 2016; Cimminella, Sala, & Coco, 2020; Hillstrom, Scholey, Liversedge, & Benson, 2012; Klein, Reichertz, Christie, Wong, & Maycock, 2019; Nuthmann, 2014; Nuthmann, De Groot, Huettig, & Olivers, 2019; Võ & Henderson, 2011; Yang, Lengyel, & Wolpert, 2016). In such paradigms, the display screen is dynamically modulated based on viewing position in order to make either foveal or extrafoveal information more or less available. In the extreme case, a sort of “tunnel vision” can be induced, which eliminates useful information from extrafoveal vision. Such studies have shown that reducing the extrafoveal preview information changes gaze patterns and influences memory under many conditions, as would be expected from studies showing extrafoveal processing in “single-shot” saccade tasks (Hillstrom et al., 2012; Klein et al., 2019). However, it remains controversial to which extent semantic information gleaned from extrafoveal vision guides search and impacts behavior (Cimminella et al., 2020; Nuthmann et al., 2019; Võ & Henderson, 2011).

An active-vision interpretation of preview effects

In natural viewing conditions, stimuli typically “appear” on the retina as the result of a saccadic eye movement, leading many researchers to suggest that it is useful to conceive of visual perception as “designed” to work in concert with eye movements. The extrafoveal preview effect is one paradigm that provides some evidence for this point of view. In fact, there is a long tradition, which can be traced back to George Berkeley's “Towards a New Theory of Vision” (1709), in arguing for a central role of action in visual perception. Indeed, perception was often considered as an active process, historically, in parallel to the development of psychophysics and the use of controlled fixation. The study by von Helmholtz, among others, argued that oculomotor signals were incorporated into visual perception (von Helmholtz, 1866; see also von Holst & Mittelstaedt, 1950). Merleau-Ponty (1974), for example, emphasized both an active and passive aspect of perception, with the observer playing a sort of question-and-answer game using movement and the corresponding sensory response to answer questions about objects that arise from previous sensory input. In the past 30 years, there have been a number of different theoretical approaches to how vision and action might interact (for review, see Bajcsy, Aloimonos, & Tsotsos, 2018). O'Regan (1992), for example, used the analogy of recognizing an object by touch, moving the fingers along the surface of an object. Touch can be viewed as a cyclical activity in which the movement of the hand creates sensation, which then is used to update cognition and to plan a subsequent action. In a similar way, seeing involves “verifying the sensations caused by possible actions” (p. 472; see also O'Regan & Noë, 2001). Reference can also be made to the theories of James Gibson, who argued that “we must perceive in order to move, but we must also move in order to perceive” (1979, p. 223). In the section below, we will consider some of these theoretical approaches to active perception and how they might relate to the preview paradigm.

In addition to the active perception theories described below, it is also worth pointing out briefly how the preview effect paradigm is related to notions of prediction (Aggelopoulos, 2015; Bar, 2011; de Lange et al., 2018; Henderson, 2017; Herwig & Schneider, 2014; Huettig, 2015; Valsecchi, Koenderink, van Doorn, & Gegenfurtner, 2018), predictive processing (Clark, 2013; Hohwy, 2020; Ransom, Fazelpour, & Mole, 2017; Walsh, McGovern, Clark, & O'Connell, 2020), and predictive coding (Bastos, Usrey, Adams, Mangun, Fries, & Friston, 2012; Friston, 2012; Friston, Adams, Perrinet, & Breakspear, 2012; Friston & Kiebel, 2009; Heilbron & Chait, 2018; Rao & Ballard, 1999; Spratling, 2017; Srinivasan, Laughlin, & Dubs, 1982). Across the past decade, research on prediction has flourished across all disciplines of the cognitive sciences and a complete review of the literature on prediction is certainly outside of the scope of this manuscript. Taking, however, a neutral stance on prediction, irrespective of underlying computational mechanisms or particular inferential principles, it is clear that in natural viewing and in preview paradigms, saccades license several types of predictions. In particular, it is possible to make specific (and potentially useful) predictions about “what” will appear on the fovea after the saccade (to the extent that this can be gleaned from the extrafoveal preview), about “where” objects will be placed on the retina with respect to the fovea (to the extent that the eye movement is accurate with respect to the planned movement) and “when” new foveal input will impinge on the retina and subsequently arrive in visual processing areas (for what and when in the auditory domain, see Auksztulewicz, Schwiedrzik, Thesen, Doyle, Devinsky, Nobre, Schroeder, Friston, & Melloni, 2018). Research using the preview paradigm has so far only examined the “what” aspect. In case the “what” was different from what was to be expected (i.e. invalid preview condition), the post-saccadic neural response was increased compared to a valid preview condition (Buonocore et al., 2020; de Lissa et al., 2019; Dimigen et al., 2012; Huber-Huber et al., 2019). This result is in line with the idea of a prediction error in case of a violated expectation. Moreover, under a predictive coding framework, such a prediction error signal results from the fact that saccades test perceptual hypotheses (Mirza, Adams, Mathys, & Friston, 2016; Donnarumma, Costantini, Ambrosini, Friston, & Pezzulo, 2017; Friston et al., 2012). Certainly, like any other reduction in neural response to repeated stimulation, the neural preview effect could be achieved by a predictive coding algorithm (Friston, 2005; Spratling, 2017). Still, in terms of predicting “what” will appear on the retina, a key question for preview research in reading as well as object perception regard how specific the prediction can be in terms of different levels of features or identity. Peripheral vision and foveal vision differ in many ways, so in order for a prediction to be useful (depending on the feature involved) the brain must take into account the relative reliability of the two sources of information (preview and post-saccadic foveal view) in order to optimally use the preview information (Ganmor, Landy, & Simoncelli, 2015; Stewart & Schütz, 2018; Wolf & Schütz, 2015). On the other hand, saccadic contingencies, in the sense of predictable changes in a stimulus across the saccade, even lead to a re-calibration of the perceptual appearance of the extrafoveal preview (Bompas & O'Regan, 2006; Bosco, Lappe, & Fattori, 2015; Bosco, Rifai, Wahl, Fattori, & Lappe, 2020; Herwig & Schneider, 2014; Paeye, Collins, Cavanagh, & Herwig, 2018; Valsecchi, Cassanello, Herwig, Rolfs, & Gegenfurtner, 2020; Valsecchi & Gegenfurtner, 2016; for a review see Stewart et al., 2020) which suggests that “what” predictions across a saccade can be relatively specific. Whether the preview effect can also be related to predicting the “when” and “where” of post-saccadic stimulation is yet to be explored.

Active vision is not limited to saccades. One of the first papers to use the term “active vision” considered how, in contrast to a static sensing system, a moving observer would access information that could help to disambiguate ill-posed visual problems such as depth and 3-D shape (Aloimonos, Weiss, & Bahdyopadhyay, 1988). More recently, Bajcsy, Aloimonos, and Tsotsos revisited these ideas in concluding that “the full task of perception requires an active agent” that makes inferences which it then tests by acquiring new sense data (2018, p. 4). Within computer vision, Ballard (1991) and others have emphasized the way in which an active system can not only solve some ill-posed problems, but could also act more quickly and efficiently than a passive system. Eye, head, and body movements would all be important for visual perception in natural viewing.

In their book “Active Vision” (Findlay & Gilchrist, 2003), Findlay and Gilchrist contrasted naturalistic vision with the “passive” methods and theories prevalent in studying perception. They argued that saccadic eye movements should be central to theories of attention and visual perception. The authors emphasized the important difference between foveal and extrafoveal physiology and the way that perception involves “fixation-move-fixation” dynamics allowing gaze shifts to “take the next sample” (p. 5). The authors argued that the major role of extrafoveal vision is to provide information for orienting movements. In other words, the extrafoveal retina is critical for deciding where to look next. This raises the question, however, of whether visual processing during the subsequent fixation takes into account any of the information gleaned from the preview, requiring some form of trans-saccadic integration. In their last chapter, the authors argue that (at least as of 2003) there was little evidence for the integration of detailed visual information across saccades. This particular conclusion has probably now to be revised considering the research on trans-saccadic integration and studies employing the preview paradigm in more recent years. The notion of active vision, however, still provides a useful theoretical framework for the preview effect.

Some active vision theories invert the traditional view of perception as “leading to action” (first sense, then act) by emphasizing how information about the motor signal is fed back into the processing stages, such that action influences perception (Merriam & Colby, 2005). This could provide a potential mechanism for trans-saccadic preview effects. For example, area V3A, which has been considered a visual processing area, is modulated by saccadic eye movements, attention and anticipation (Nakamura & Colby, 2000). Similar modulations by saccades have been reported in other visual processing areas, including V1 and the lateral geniculate nucleus (Barczak, Haegens, Ross, McGinnis, Lakatos, & Schroeder, 2019; Kogan, Gur, & Snodderly, 2008; Purpura, Kalik, & Schiff, 2003; Reppas, Usrey, & Reid, 2002; Ruiz & Paradiso, 2012; Tolias, Moore, Smirnakis, Tehovnik, Siapas, & Schiller, 2001). Such modulation is particularly evident in the parietal cortex, where spatial maps shift to take into account saccadic eye movements, with some of these shifts occurring predictively before saccade execution (Merriam & Colby, 2005). With the discovery of “remapping” of the spatial selectivity of neurons in parietal cortex, it became possible to directly demonstrate the influence of action, or intended action, on visual processing (Duhamel, Colby, & Goldberg, 1992). One potential explanation of at least some of the extrafoveal preview effects demonstrated thus far is that information about the preview is integrated with that of the post-saccadic foveal information, with remapping being one potential mechanism for this integration process.

Like “active vision,” the notion of “active sensing” can be traced to a solution to a technical problem of how an artificial system can gather and interpret useful data about the location and identity of objects in the world. Active sensing involves transmission of a signal and then measuring the resulting input to a sensor. Examples include radar, sonar, and (in the natural world) echolocation. More recently, the terminology of “active sensing” has been adapted by neuroscientists to the question of how biological organisms actively interrogate the physical world with the motor system (Leszczynski & Schroeder, 2019; Schroeder, Wilson, Radman, Scharfman, & Lakatos, 2010). An example is whisking in rodents (and other species), in which the rodent uses particular frequencies of whisker movements to sense the features of its immediate environment (e.g. Oladazimi, Brendel, & Schwarz, 2018). Another interesting example is olfaction. Numerous studies have shown that humans and other animals change the rate and force of inhaling based on the presence of odors. Many animals turn their heads, or even alter the position or shape of their ears, in order to use that alteration in order to gain specific information about the incoming sound for localization and recognition (for an example, see Yin & Müller, 2019). A particular example is the star-nosed mole that haptically senses potential prey by a specialized nose organ (Catania, 2011). This organ consists of minuscule finger-like appendages with varying tactile sensitivity and a high-resolution part similar to the layout of the retina with a foveal. The foraging behavior of the mole exhibits a sequence which repeatedly touches objects of interest with its “tactile fovea,” resembling in many ways eye gaze behavior (Catania, 2011). In the visual domain, active sensing has not only been used as a descriptive term (Parr & Friston, 2017) but computational models of the visual system have been developed in line with the ideas of active sensing (Friston et al., 2012; Parr & Friston, 2018). These examples illustrate the way in which sensory input is actively acquired by a motor and/or attentional sampling routine (Schroeder et al., 2010). Active sensing theories also emphasize the strategic and predictive nature of the motor/attention sampling (Schroeder et al., 2010). Thus, it might be fruitful to conceptualize the preview effects found so far in terms of active sensing theories, which go beyond the visual modality to encompass other sensorimotor interactions.

While in many ways “active vision” and “active sensing” seem to capture a common set of ideas, researchers using the latter approach have often given greater emphasis to the regular and rhythmic nature of these motor or attention routines. In the case of “whisking” or SONAR, for example, sensation is always processed in the context of a regular, rhythmic motor action (or sending out of a signal). In the case of whisking, the animal may vary the rate of action depending on the context. These rhythms have been linked to dominant frequencies found in brain oscillations (Leszczynski & Schroeder, 2019; Schroeder et al., 2010).

In writings about both of the above theories, authors often refer to “samples” or “sampling” when describing the sensorimotor process (Findlay & Gilchrist, 2003; Schroeder et al., 2010). In the case of extrafoveal preview effects, each fixation can be viewed as a sample, including both foveal and extrafoveal information. In order to further define extrafoveal preview effects, it might be useful to more specifically characterize (and eventually model) the nature of “sampling” in order to generate more specific hypotheses. “Looking” involves a series of discrete fixations, during which time the eye is relatively still (although see: Hafed, Goffart, & Krauzlis, 2009; Malevich, Buonocore, & Hafed, 2020; Poletti, Listorti, & Rucci, 2010; Rucci, Iovin, Poletti, & Santini, 2007), separated in space-time by saccades. Each fixation can be viewed as a “sample” and the process of dividing time into different fixations can be viewed as “sampling,” leading to some specific claims.

First, the term “sample” is used in statistics to describe a smaller quantity that is used to infer something about the larger whole. It is a subset of the larger population that contains the characteristics of that population. The extrafoveal preview effect could be used to test whether the influence of the preview depends on the degree to which the saccade target is characteristic of the rest of the scene. Second, a sample describes a set of values at a specific time and/or space. In the case of time, it is a discrete, temporal snapshot of a longer, ongoing event. This leads to the idea of sampling, which is a process done in space or time (or any other dimension) that turns a continuous (or extended) signal into a discrete sample. In this case, the sample is tied to a fixation, which breaks the visual input into discrete elements. “Sampling rate” refers to the number of samples per second (or per other unit) taken from a continuous signal (which converts analogue input to a discrete or digital signal). In the case of naturalistic viewing, we can describe oculomotor behavior in terms of the number of saccades (or, conversely, fixations) per second. A key idea is that each sample yields a limited amount of information and so a new sample must be made to accumulate further knowledge. Moreover, our model of the world, and predictions about it, must be regularly updated. Both the intake of new information after a saccade and the updating of our memories and predictions, break continuous sensory input into discrete units. Some evidence for the idea of “fixations as samples” comes from studies of counting, showing that accurate and precise counting when the stimuli are irregular and number above four items is only possible when participants are allowed to move their eyes (Kowler & Steinman, 1977; Kowler & Steinman, 1979; Landolt, 1891). This suggests that the amount of information that can be processed in a single sample is limited, requiring shifts of attention and/or eye position. This raises the question of whether the extrafoveal preview effect is influenced by temporal aspects and previous views in more naturalistic scanning sequences. Whereas previous studies measured the influence of saccades on perception within a small visual area, in natural viewing the importance of saccades to sample the environment is even more evident due to the inhomogeneity of the retina in terms of the density of rods and cones. Recently, this common notion that the inhomogeneity of the retina is the main reason for saccadic behavior has been challenged. Evidence based on the gaze behavior of mammals without a typical primate retina, that is, without a fovea, supports an alternative view according to which some aspects of saccadic behavior can be explained by maximizing the information gained with each saccade in terms of neuronal population activity in V1 (Samonds, Geisler, & Priebe, 2018; see also Baden, Euler, & Berens, 2020).

Further evidence that fixation periods may form a sort of natural boundary in the temporal structure of visual processing comes from studies showing how neural sensitivity to new input varies dramatically around the time of saccades. Whereas sensitivity to retinal input is reduced shortly before saccade onset in different visual and visuo-motor areas (Chen & Hafed, 2017; Hafed & Krauzlis, 2010; Kogan et al., 2008; Leopold & Logothetis, 1998; Noda & Adey, 1974; Royal, Sáry, Schall, & Casagrande, 2006; Sylvester, Haynes, & Rees, 2005; Vallines & Greenlee, 2006), early visual areas are actually highly sensitive to incoming signals at the beginning of new fixations (Ito, Maldonado, Singer, & Grün, 2011; Schroeder et al., 2010). Such studies suggest that visual processing has a temporal structure, defined by periods of high or low sensitivity to retinal input. Visual sensitivity is not uniform, but rather variable and perhaps almost discrete. This temporal structure is defined by, or aligned with, the execution of saccades (and micro-saccades: Lowet, Gips, Roberts, De Weerd, Jensen, & van der Eerden, 2018). This raises the question of whether preview effects follow such a temporal structure. Extrafoveal preview effects have been found within a rather extensive time window after the onset of a new fixation, as indexed by EEG, from as early as around 100 ms (Huber-Huber et al., 2019) up until around 400 ms (Buonocore et al., 2020; Dimigen et al., 2012; Huber-Huber et al., 2019). Fixation durations can vary but are usually shorter than 400 ms. Is the interaction (or integration) of extrafoveal and foveal information indexed by the preview effect strictly synchronized with the rhythm of the saccade-fixation cycle? This would suggest either that each new sample could only be acquired after processing the current sample was complete, or that there is a parallel process. The temporal relation between active sampling and the actual integration of new samples might be less regular and straightforward than what one would expect at first glance. In the case of reading, for example, integration effects can occur at different temporal levels as the meaning of an entire sentence or paragraph builds over time. Scene understanding can also increase over time as we look around, but the time course of extrafoveal preview effects in such naturalistic situations has yet to be explored. A first starting point could be a preview paradigm that allows for saccade-fixation sequences instead of the rather restrictive setting of only one saccade (see for example Auerbach-Asch et al., 2020).

The temporal dynamics of visual processing have led to the idea that visual perception may be described as a “discrete” process (Freeman, 2006; Herzog, Drissi-Daoudi, & Doerig, 2020; VanRullen, 2016; VanRullen & Koch, 2003). Evidence for this idea comes, for example, from the wagon wheel illusion – a wheel in movies or TV seemingly rotating backwards – which is supposed to occur when the “sampling rate” of visual processing and the temporal frequency of the stimulus are misaligned (VanRullen, Reddy, & Koch, 2005). However, studies have shown that visual temporal integration windows vary in size, ranging from a few tens of milliseconds, which is much briefer than a fixation, to several seconds for event perception, encompassing multiple unique fixations (for a review, see Pöppel 1997; Pöppel 2009). Notably, the saccade rate is slower than the proposed 7 to 10 Hz rates of discrete perception that has most often been suggested. Moreover, studies of trans-saccadic transformational apparent motion, for example, show that event perception boundaries can start before, and continue after, a saccade (Fracasso, Caramazza, & Melcher, 2010). Specifically, when an object changes shape and position, with the first frame shown prior to saccade onset and the second frame (change in shape and position) is shown after the saccade, a smooth and continuous motion event is perceived in external space, not a series of discrete snapshots (Fracasso et al., 2010). If temporal integration windows can be longer or shorter than a fixation duration, this raises the question of why saccades are not made more often. Making saccades 10 times per second, for example, would seem to better maximize the potential intake of visual information. One possibility could be that time is required to not only process the information at the fovea but also to form a prediction about the future saccade target, as found in the extrafoveal preview effect. The saccade rate, then, would be limited by the temporal dynamics of object perception (and word perception in reading), with a time frame of around 3 to 5 cycles per second (Drewes, Zhu, Wutz, & Melcher, 2015; Wutz, Muschter, van Koningsbruggen, Weisz, & Melcher, 2016). Visual perception would then reflect a hierarchy of different temporal scales, ranging from tens of milliseconds to seconds, with the fixation duration reflecting a complete “sample” of visual processing (Barczak et al., 2019) in retinotopic coordinates, in order to then provide information about the foveated target and the future saccade target, supporting stable perception across saccades beyond retinotopic coordinates (Melcher & Morrone, 2015). Previous studies of the time frame of object categorization (Drewes et al., 2015; Lamme & Roelfsema, 2000; Thorpe, Fize, & Marlot, 1996), as well as studies on the time point at which visual input transforms from retinotopically-defined to non-retinotopic and consciously available percepts (Fabius et al., 2020) tend to converge on a time scale of around 150 to 200 ms for the duration of the initial, retinotopic integration window. Interestingly, as the preview paradigm shows, this temporal scale resembles the time period in which pre- and post-saccadic visual input is integrated (Huber-Huber et al., 2019) and the period during which several types of visual predictions based on preceding events take effect (Johnston, Robinson, Kokkinakis, Ridgeway, Simpson, Johnson, Kaufman, & Young, 2017). Thus, further investigations of the timing of the extrafoveal preview effect may provide further insight into how temporal dynamics of vision and of the oculomotor system may interact in the timing of gaze movements in natural tasks.

Conclusions and future directions

The preview paradigm provides a potential way forward in the goal of characterizing the way in which perception works during natural viewing. Although much has been learned about visual perception, and its neural correlates, from the traditional laboratory paradigms using stable fixation and suddenly appearing stimuli, preview paradigms allow for the investigation of many of the other factors that may come into play during real world perception. Drawing upon reading research, the preview paradigm with non-word stimuli has made clear that not only word recognition, but foveal perception in general, is affected by the imminent extrafoveal input available at the saccade target location. This link across research fields is a step toward an even more general notion of trans-saccadic perception that highlights the active nature of perception, an element central to philosophical considerations on the mutually constitutive relationship between perception and action.

In particular, the preview paradigm can be used to further investigate the way in which saccades license several types of predictions, including “what,” “when,” and “where” information. As such, the preview paradigm can contribute to a better understanding of visual perception as a predictive process in cognitive neuroscience. There is currently evidence for prediction to some degree, but further work is needed to clarify what kind of information is processed extrafoveally and how this depends on bottom-up factors (retinal eccentricity, contrast, crowding, etc.) and top-down factors (attention, task, etc.). There is neurophysiological evidence that neural sensitivity to retinal input changes around the time of saccades, which suggests a prediction about “when” the stimulus will appear at the fovea under natural viewing conditions, but this has yet to be systematically studied in humans and the preview paradigm would seem a useful paradigm to investigate this question. Moreover, during natural viewing, both bottom-up and top-down mechanisms are involved, requiring coordination over time. One possibility, for example, is that when a prediction about object identity is licensed by the extrafoveal preview, there is a predominance of top-down signals prior to the saccade, compared to trials in which the prediction cannot be licensed. Similarly, there is more need for retinal input at the beginning of the new fixation when there is no prediction, compared to when there is a strong prediction or expectation of the upcoming stimulus. Previous neurophysiological studies suggest that sensitivity to incoming sensory input, versus top-down processes, varies over time with respect to the fixation onset (see above). This can be further tested by varying the type of information available in the extrafoveal preview, the match between the preview and the post-saccadic stimulus, and the frequency of trans-saccadic change versus no-change conditions across time.

The preview paradigm provides the opportunity to characterize the differences between active vision (moving the fovea to the stimulus) and the passive paradigms typically used in laboratory studies in which the stimulus suddenly appears while the eye maintains fixation. One recent report suggested that the classic N170 component for faces was present only for the first extrafoveal glimpse of the face but eliminated in a subsequent fixation on the face (de Lissa et al., 2019). In contrast, we found an N170 component for both extrafoveal preview and subsequent foveal inspection of a face (Buonocore et al., 2020; Huber-Huber et al., 2019). Apart from this discrepancy, however, both studies found fixation-locked preview effects when contrasting valid with invalid trials, which hints at the context-dependence of trans-saccadic perception. Besides this context-dependence, active and passive viewing conditions imply different scalp distribution of ERP components as illustrated for instance for the N170 (Auerbach-Asch et al., 2020). These findings suggest that other classic visual ERP components and, thus, the processes indexed by these components take a different form in active vision. In particular surprise- or expectancy-related processes could appear quite differently given the tight theoretical link of saccadic eye movements to expectation and prediction. In principle, it might be necessary to undertake a vast replication project in which classic visual ERP studies are re-run using the preview paradigm to see whether the same evoked components are still found under more natural viewing conditions.

The initial studies of visual object perception using the preview paradigm demonstrate that the extrafoveal preview alters the timing and magnitude of the evoked response to a stimulus as well as the behavior in response to the stimulus. At present, however, it is not clear what exactly changes about the neural mechanisms involved and whether these effects are described more accurately in terms of valid preview benefits or invalid preview costs, or a combination of both (Hutzler et al., 2013). Moreover, a set of highly intertwined concepts and mechanisms provide possible explanations for the reduction in neural response with a valid extrafoveal preview: prediction, perceptual re-calibration, statistical learning, repetition suppression, adaptation, and/or priming from the fact that the stimulus has been processed recently (e.g. Richter, Elkman, & de Lange, 2018; Rostalski, Amado, Kovács, 2019; Tang, Smout, Arabzadeh, & Mattingley, 2018; Todorovic & de Lange, 2012). In addition, visual attention and saccadic remapping likely play a role, which has not yet been the focus of any study. Instead of finally arriving at one mechanism as the best explanation for all kinds of preview effects, probably a certain combination will turn out to be optimal as we already suggested for the neural preview effect in reading (The parafoveal preview paradigm in reading). To determine the mechanisms for preview effects with non-reading tasks further studies are required to investigate important differences between active and passive paradigms. Moreover, it will probably be relevant to consider the influence of time on task (see for example Huber-Huber & Melcher, 2021). Eventually, effects in natural viewing conditions might be more automatic or occur more quickly already in the first few trials, whereas in passive viewing conditions effects might tend to build up across time as participants implicitly learn the statistical regularities they experience without moving their eyes.

In addition, this paradigm provides a tool for better defining the nature of trans-saccadic integration. There is a longstanding debate regarding the mechanisms underlying our subjective experience of stable and continuous perception (for reviews: Melcher & Colby 2008; Melcher & Morrone, 2015). A key piece of information in distinguishing between the different proposed mechanisms is whether or not feature information is predicted and whether it influences post-saccadic processing (trans-saccadic integration; for recent work see e.g. Grzeczkowski, van Leeuwen, Belopolsky, & Deubel, 2020). The preview paradigm provides a way to study both the pre-saccadic and the post-saccadic effects of a preview. In particular, neuroimaging tools with high temporal resolution, such as EEG or magnetoencephalography (MEG), can move beyond the question of whether or not there is trans-saccadic integration to the definition of the time courses and the neural mechanisms involved (e.g. Buonocore et al., 2020; Fabius et al., 2020; Huber-Huber et al., 2019).

Finally, the extrafoveal preview paradigm holds promise for the goal of characterizing the nature of active vision, in terms of concepts, such as “sensorimotor integration,” “active sensing,” “sampling,” or other theoretical constructs. The development of ideas about “active vision” was an important step in the study of visual perception. Future work can help clarify to what extent the integration between the (oculo)motor system and sensory processing is unique to vision, as opposed to reflecting more general principles of perception and action that are found with other senses and across different species (for review, see Friston, 2012; Schroeder et al., 2010). Perhaps active vision is best understood as a specific case of a more fundamental solution to the problem of how organisms can interact with environments in which there is a degree of both stability/predictability and dynamic change, due to the laws of physics that guide how objects move in space and time. Eventually, the primacy of action before perception postulated by some notions of active vision might have to be revised again. The preview paradigm shows that action fundamentally affects perception. Going one step further, however, it becomes apparent that the preview effect depends on being able to perceive at least something extrafoveally before any action (i.e. saccadic eye movement), is made. This fact goes against some active vision ideas as it shows that, again, perception precedes action. Perhaps an optimal theoretical framework will capture the relation of action and perception in an infinite action-perception/perception-action cycle, without a beginning, and without an end.