Circuits for presaccadic visual remapping

Abstract

Saccadic eye movements rapidly displace the image of the world that is projected onto the retinas. In anticipation of each saccade, many neurons in the visual system shift their receptive fields. This presaccadic change in visual sensitivity, known as remapping, was first documented in the parietal cortex and has been studied in many other brain regions. Remapping requires information about upcoming saccades via corollary discharge. Analyses of neurons in a corollary discharge pathway that targets the frontal eye field (FEF) suggest that remapping may be assembled in the FEF's local microcircuitry. Complementary data from reversible inactivation, neural recording, and modeling studies provide evidence that remapping contributes to transsaccadic continuity of action and perception. Multiple forms of remapping have been reported in the FEF and other brain areas, however, and questions remain about the reasons for these differences. In this review of recent progress, we identify three hypotheses that may help to guide further investigations into the structure and function of circuits for remapping.

we perceive the environment as continuous, but neurons in the brain take samples that are constrained in space and time. Specific ranges of stimuli drive a neuron to produce action potentials. From the 1960s through the 1970s, it was essentially dogma that the stimulus-spike relationship of a sensory neuron revealed its “receptive field,” the portion of the world to which a sensory neuron responds. Response gain could be modulated by internal factors such as attention or external factors such as contrast, but the structure of receptive fields seemed immutable and their function, to report what is out there, seemed passive. In the case of the visual system, neurons were assumed to have static receptive fields relative to the fovea.

Evidence for exceptions to this rule emerged in the 1980s, and the dogma was overturned in the 1990s, when behaving animal preparations allowed for studies in which eye movements were incorporated as factors in experimental design. It became clear that around the time of saccades the spatial structure of receptive fields can change. Here we summarize the advances in our understanding of labile receptive field location with a focus on circuit-level studies from the past 20 years. Progress has been made in elucidating the mechanisms of receptive field remapping both at the macrocircuit level (pathways between brain areas) and the microcircuit level (processing within brain areas). Recent physiological and modeling experiments have leveraged this circuit information to explore the visuomotor and perceptual functions of remapping. Taken together, a new picture of the genesis and role of presaccadic remapping emerges, yielding testable hypotheses for future work.

Presaccadic Visual Remapping

The first hint that visual receptive fields can be spatially dynamic was reported by Mays and Sparks (1980) in a study of neurons located in the intermediate layers of the macaque superior colliculus (SC). A subset of neurons, termed “quasi-visual cells,” had strong, sustained visual responses and modest motor responses for saccades made to single targets. In a task requiring sequential saccades to two flashed targets, however, the neurons fired for the second stimulus even while it was outside of the receptive field. In the mid-1980s, when Bruce, Goldberg, and colleagues (Bruce et al. 1985; Bruce and Goldberg 1985) characterized the frontal eye field (FEF), they noticed a similar effect: some FEF neurons fired for visual stimuli used as targets for second saccades in a sequence, even though those stimuli were never in the receptive field (Goldberg and Bruce 1990). The effect seemed purely visual, as it occurred for visually responsive neurons even if they lacked saccade-related activity.

Duhamel et al. (1992) made sense of the effect with a landmark study of neurons in the lateral intraparietal area (LIP), which is reciprocally connected with the FEF anatomically and functionally (Blatt et al. 1990; Chafee and Goldman-Rakic 1998, 2000; Stanton et al. 1995). They demonstrated that LIP neurons predictively report visual stimuli that will be in their receptive field after a saccade is completed. The visual sensitivity of the neurons shifts from the classical receptive field to what became known as the “future field.” Duhamel et al. (1992) postulated that the effect could be related to the percept of visual stability across saccades. Since that work, presaccadic remapping has been found in a number of visual and oculomotor areas (V1, V2, V3, and V3A: Nakamura and Colby 2002; V4: Neupane et al. 2016a, 2016b; SC: Churan et al. 2012; Walker et al. 1995; FEF: Mayo et al. 2016; Sommer and Wurtz 2006; Umeno and Goldberg 1997, 2001), although not all of them (MT: Hartmann et al. 2011; Inaba and Kawano 2014; Ong and Bisley 2011; Yao et al. 2016). Some neurons in V1, LGN, the SC, area MST, and area VIP exhibit a transient, presaccadic decrease in activity that may be related to the perceptual effect of saccadic suppression (reviewed by Ibbotson and Krekelberg 2011), but this phenomenon seems to occur in the absence of presaccadic remapping to a location outside the classical receptive field.

Presaccadic remapping is a form of visual response because it requires visual stimulation in the future field. But a remapped response differs from a classical visual response in two major ways. First, the spatial location of a remapped response depends on both the location of the classical receptive field and the vector of the upcoming saccade (Duhamel et al. 1992; Goldberg and Bruce 1990; Sommer and Wurtz 2006). Second, the timing of a remapped response is unrelated to the visual latency of a neuron (Umeno and Goldberg 1997); it aligns well, however, with saccade initiation (reviewed by Sommer and Wurtz 2008a). For many neurons (∼30–40% in LIP and FEF), the remapping is considered predictive because it precedes saccade initiation (Duhamel et al. 1992; Kusonoki et al. 2003; Sommer and Wurtz 2006) and the arrival of postsaccadic (reafferent) visual responses (Umeno and Goldberg 1997). Neurons that remap, therefore, must be receiving information about when the next saccade will start and where it will go. Such information about imminent movement is called corollary discharge (or efference copy; Sperry 1950; Von Helmholtz 1962; Von Holst and Mittelstaedt 1950). The source of saccadic corollary discharge, and how it may be combined with visual input to create remapping, is discussed in the next two sections.

Macrocircuits for Remapping

Research into the neural circuits that support remapping began with attempts to identify its corollary discharge input. Guided by anatomical data from Lynch et al. (1994), Wurtz and Sommer (2004) proposed four criteria for identifying corollary discharge and applied them to a pathway they studied. First, a corollary discharge signal must originate in a region known to control movement. In primates, Lynch et al. (1994) used retrograde viral tracer injections in FEF to identify disynaptic input pathways from three subcortical structures. One pathway originated in the intermediate layers of the SC. They could not distinguish the thalamic relay for the pathway because several nuclei showed first-order labeling, but other anatomical studies implicated the lateral edge of the mediodorsal nucleus (MD; Benevento and Fallon 1975; Goldman-Rakic and Porrino 1985). Electrophysiologically, this connectivity was confirmed by using antidromic and orthodromic stimulation techniques to identify the pathway's source neurons in SC, relay neurons in MD, and recipient neurons in FEF (Fig. 1; Sommer and Wurtz 1998, 2002, 2004a). Second, a corollary discharge pathway should convey movement-related signals that start prior to the movement and represent its temporal and spatial parameters. In the SC-MD-FEF pathway, most MD relay neurons conveyed presaccadic bursts of activity that met this criterion, as did the SC source neurons and FEF recipient neurons (Sommer and Wurtz 2004a). Third, elimination of a corollary discharge pathway should not affect movements that do not require corollary discharge. Consistent with this criterion, inactivation of lateral MD had no effect on single saccades except for a slight, omnidirectional increase in reaction times (Sommer and Wurtz 2004b; see also Tanaka 2006). In contrast, inactivation of the SC itself causes marked deficits in contralateral saccade trajectory, latency, and speed (Aizawa and Wurtz 1998; Quaia et al. 1998a), and inactivation of the FEF causes similar contralateral deficits in the context of making saccades to remembered stimuli (Chafee and Goldman-Rakic 2000; Dias et al. 1995; Dias and Segraves 1999; Sommer and Tehovnik 1997). Fourth, elimination of a corollary discharge pathway should disrupt performance on tasks that require corollary discharge. Inactivation of lateral MD caused significant, contralateral deficits in compensating for the first saccade in a two-saccade sequence (Sommer and Wurtz 2002, 2004b) and drastically reduced remapping associated with contralateral saccades in the FEF (Sommer and Wurtz 2006). More recently, MD inactivation was shown to impair perceptual localization of visual stimuli after contralateral saccades (Cavanaugh et al. 2016). The SC-MD-FEF pathway therefore meets all four criteria for a “macrocircuit” for corollary discharge.

Fig. 1.

Pathway for corollary discharge of saccades. A: electrophysiological identification of the pathway's relay neurons in mediodorsal thalamus (MD). Individual neurons were double-identified as orthodromically activated from the superior colliculus (SC) and antidromically activated from the frontal eye field (FEF). Bottom: traces show repeated, superimposed recording traces of an example MD neuron. Just after stimulation of the SC (left), the neuron fired with a variable latency. The same neuron showed a consistent latency of activation from the FEF (right). B: schematics depicting a close-up of the target of the pathway adapted from Lynch et al. (1994) with permission of Springer: the FEF (top); the relay zone at the lateral edge of MD (middle); and the source zone in the intermediate SC (bottom). PS, principal sulcus; AS, arcuate sulcus; VL, ventrolateral nucleus; VLm, ventrolateral nucleus pars medialis; SO, stratum opticum; SGI, stratum griseum intermedium; PAG, periaqueductal gray; Aq, cerebral aqueduct.

Presaccadic remapping in the FEF depends on activity provided by the SC-MD-FEF pathway (Sommer and Wurtz 2006), and that pathway provides corollary discharge of saccades (reviewed by Sommer and Wurtz 2008a). The implication is that the corollary discharge combines with “passive” visual inputs to create the remapping. But can we rule out a simpler hypothesis, that remapping is transferred from the SC up to the FEF? The SC itself has visual responses and remapping (Churan et al. 2011, 2012; Mays and Sparks 1980; Walker et al. 1995), and the SC-MD-FEF pathway conveys visual signals in addition to presaccadic bursts of activity (Sommer and Wurtz 2004a). Visual signals in MD relay neurons of the pathway have not yet been tested for presaccadic remapping. Two lines of evidence, however, suggest that remapping in the FEF is not simply inherited from the SC. First, there are striking differences between remapping in the SC and FEF, as discussed next. Second, FEF recipient neurons of the pathway exhibit unique presaccadic modulations of their visual responses that suggest de novo construction of remapping, as described in Microcircuits for Remapping.

Walker et al. (1995) and Churan and colleagues (2011, 2012) examined remapping in the SC intermediate layers. Both groups found that a hallmark of remapping in SC is a bimodal distribution of activity: a burst before the saccade, a pause, and then a burst after the saccade. That is, remapped visual responses are transiently quenched during the saccade, similar to saccade-related suppression of visual responses found in the superficial SC and FEF (Mayo and Sommer 2008; Richmond and Wurtz 1980; Robinson and Wurtz 1976). If remapping in the SC were simply conveyed to the FEF, one might expect to see the same saccade-related quenching of remapping in the FEF. Instead, remapped visual signals in FEF are relatively unperturbed (Sommer and Wurtz 2006; Umeno and Goldberg 1997). Some saccade-related suppression can be detected in FEF with modeling analyses (Joiner et al. 2013b), but its magnitude is small and alignment with saccade initiation poor, yielding at most “dips” in remapped responses rather than stark bimodal distributions as in the SC.

A commonality between remapping in the SC and the FEF is weaker future field responses compared with receptive field responses. But this is more pronounced for the SC, in which future field responses are ∼50–60% weaker than receptive field responses (Fig. 3F of Churan et al. 2012; see also Fig. 8 of Walker et al. 1995), compared with only around a 30% difference in the FEF (Fig. 9 of Umeno and Goldberg 1997). The final notable difference between remapping in the FEF and SC concerns visual sensitivity between the receptive field and the future field. Neurons in the SC exhibit significant responses to visual stimuli presented at the midpoint between the fields (Churan et al. 2012), but neurons in the FEF do not. Rather, FEF neurons exhibit a “jump” of visual sensitivity from receptive field to future field with no activation in the middle (Mayo et al. 2016; Sommer and Wurtz 2006).

Fig. 3.

Proposed microcircuitry for the assembly of visual remapping within FEF. Cortical visual and subcortical corollary discharge (CD) inputs target excitatory and inhibitory neurons in layer IV. These long-range projections are excitatory. Remapping is piecewise in the recipient neurons (↑, presaccadic increase in visual sensitivity; ↓, presaccadic decrease; RF, receptive field; FF, future field). Only putative excitatory neurons (Exc) and presumed low-threshold spiking (LTS) inhibitory interneurons show presaccadic changes to their visual sensitivity. Putative parvalbumin-positive neurons (not shown) have visual responses, but they do not change before saccades. Layer IV projects predominantly to layers II/III via excitatory interneurons (reviewed in Shin and Sommer 2012). The changes in visual sensitivity at the RF and FF must be inverted and combined to create the full remapping that is found in layer V corticotectal neurons. Whether the resultant, full remapping is sent to extrastriate visual areas is unknown, but it is depicted here as a hypothesis. White triangles, excitatory synapses; black triangles, inhibitory synapses.

Neurons in LIP are like those in the SC in that they respond at the midpoint. Fine-scaled spatial and temporal testing of this effect in LIP suggests, moreover, that visual sensitivity spreads through the midpoint and onward to the future field (Wang et al. 2016). This spread might reflect local connections that propel remapping (Quaia et al. 1998b). If that is how remapping is created, and if LIP sends only the outcome of the process to the FEF, it would explain why FEF neurons show a jump of visual sensitivity to the future field. But the reverse scenario is also plausible. Remapping could originate in the FEF through mechanisms that yield a clean jump of visual sensitivity from receptive field to future field. Through divergence of FEF outputs and convergence onto LIP, the jump in FEF might be “smeared” into an apparent spread in LIP.

The hypothesis that remapping originates in the FEF is consistent with many lines of evidence. The FEF almost certainly receives more corollary discharge input than extrastriate areas (Fig. 2A). The pathway from SC to FEF arises from the SC intermediate layers, with little contribution from its superficial layers (Lynch et al. 1994; Lyon et al. 2010). The pathways from the SC to extrastriate areas are just the opposite (Adams et al. 2000; Benevento and Rezak 1976; Clower et al. 2001; Leichnetz 2001). The FEF distributes its signals widely. It has a dense monosynaptic connection to LIP (Blatt et al. 1990; Bullier et al. 1996; Petrides and Pandya 1984; Stanton et al. 1995) and sends a diversity of signals, including remapped visual responses, to the intermediate SC (Shin and Sommer 2012; Sommer and Wurtz 2000, 2001, 2006). LIP sends a profusion of signals to SC as well, but it is unknown whether they include remapping (Ferraina et al. 2002; Paré and Wurtz 1997, 2001; Wurtz et al. 2001).

Fig. 2.

Overview of macrocircuits for remapping. A: the intermediate SC layers (SCi) and, to a much lesser extent, the superficial SC (SCs) project to MD thalamus, where signals are relayed to the FEF. The converse is true for SC projections to pulvinar (Pul.) that are relayed to extrastriate areas and V1. B: the FEF may serve as the source for remapped visual signals to extrastriate areas such as LIP and V4 (Fries 1984), with its influence diminishing toward V1 via sequential relays, as shown, and via increasingly sparser direct projections (not depicted; Stanton et al. 1995). The FEF and extrastriate regions project to the SCi and could transmit remapping to there, although this has been demonstrated only for FEF (Shin and Sommer 2012; Sommer and Wurtz 2006).

If the FEF creates remapping through a combination of corollary discharge and visual input, it could send its signals to LIP directly and other extrastriate areas directly or polysynaptically (Fig. 2B; Merriam et al. 2007; Nakamura and Colby 2002). The extrastriate areas and FEF all could send remapping to the SC. The extrastriate areas would exclude area MT, where there is little evidence of remapping as noted above (Hartmann et al. 2011; Inaba and Kawano 2014; Ong and Bisley 2011; Yao et al. 2016). The broad convergence of remapped visual signals at the SC could contribute to the complex context-dependent nature of remapping in those layers (Churan et al. 2011).

The hypothesis that FEF creates remapping makes three clear experimental predictions. First, it predicts that remapping is conveyed out of the FEF. This has been confirmed for FEF output to the SC with antidromic activation (Shin and Sommer 2012; Sommer and Wurtz 2006). It could be tested in the same way for FEF output to extrastriate cortex. Second, the hypothesis predicts that remapping happens in FEF first. This could be tested by comparing the timing of remapping in FEF and other areas with constant experimental parameters (ideally, simultaneous recording from different areas of the same monkey). Third, the hypothesis predicts that FEF is necessary for remapping elsewhere. This can be tested by reversibly inactivating the FEF while recording from neurons at another site such as LIP. Remapping in the neurons should be eliminated.

Microcircuits for Remapping

If remapping is created in the FEF, what are the mechanisms? Remapped visual information needs to be modulated by corollary discharge, but how are those signals combined to enable transient shifts in visual sensitivity? What little we understand about this comes from a comparison of remapping across layers in the FEF. Corollary discharge from the SC is relayed by MD thalamus and arrives in layer IV and deep layer III of the FEF (Giguere and Goldman-Rakic 1988; for brevity, we call this layer IV). FEF layer IV neurons innervated by this pathway can be identified by disynaptic activation from electrical stimulation in the SC (Sommer and Wurtz 1998, 2004a). One would expect them to exhibit presaccadic activity, since that is what corollary discharge is, and they do. But the most remarkable characteristic of these FEF layer IV neurons is their near-ubiquity of visual responses: 94% of the sample in Sommer and Wurtz (2004a). This is a much higher incidence of visual responses than found in the MD neurons that relay signals from SC to FEF (Sommer and Wurtz 2004a), suggesting that the FEF layer IV neurons receive considerable visual input from extrastriate cortex. Projections and visual signals from extrastriate cortex to FEF do, in fact, target its layer IV (Ferraina et al. 2002; Schall et al. 1995). Visual responses are less common in the FEF as a whole (43% of neurons as reported by Bruce and Goldberg 1985). From these properties alone—prolific visual responses and known corollary discharge input—FEF layer IV neurons are promising candidates for contributing to the generation of remapping.

Shin and Sommer (2012) studied the contribution of FEF layer IV neurons to remapping by identifying them through orthodromic activation from the SC and testing their presaccadic visual sensitivity at the receptive field and future field. As a point of comparison, they ran the same tests on output neurons of layer V in FEF as identified by antidromic activation from the SC. An important analysis was to distinguish between the putative cell types in layer IV. Unlike layer V corticotectal neurons that are entirely excitatory (pyramidal), neurons in layer IV can be either excitatory or inhibitory; both cell classes reside in layer IV and receive thalamic input (Benevento and Fallon 1975; Giguere and Goldman-Rakic 1988; Goldman-Rakic and Porrino 1985). Using analyses of action potential width (Chen et al. 2008; Cohen et al. 2009; Constantinidis and Goldman-Rakic 2002; Song and McPeek 2010), with the known layer V pyramidals serving as a reference set of known excitatory neurons, Shin and Sommer (2012) segregated their layer IV neurons into putative excitatory and inhibitory neurons, with a smaller class of neurons that were indeterminate (“ambiguous” neurons).

In FEF layer V, many individual neurons and the population response exhibited full remapping, defined as a presaccadic decrease in visual sensitivity at the receptive field coupled with an increase at the future field (Shin and Sommer 2012). In contrast, none of the layer IV neurons did. The layer IV putative excitatory neurons increased their sensitivity at the receptive field just before a saccade, but they did not remap to the future field. The layer IV ambiguous neurons did remap—they increased their sensitivity at the future field just before a saccade—but they showed no decrease in sensitivity at the receptive field. The putative inhibitory neurons showed no modulation in visual sensitivity at all. The conclusion was that “pieces” of remapping are distributed across cell types in FEF layer IV, as if full remapping were in an early stage of assembly.

What were the ambiguous neurons? Not only were they the only layer IV neurons that remapped, they were also the neurons with the strongest delay activity in a memory-guided saccade task, suggesting that they have unusual abilities to extend signals in both space and time. This capacity to sustain long-lasting firing rates and their intermediate action potential widths suggested that the ambiguous neurons may be low-threshold spiking, somatostatin-positive inhibitory interneurons (Shin and Sommer 2012). New techniques in which opsins are expressed in a cell type-specific manner include the use of promoters to make somatostatin-positive interneurons selectively responsive to light (e.g., Wilson et al. 2012; reviewed by Hangya et al. 2014). With more development, these advances in the mouse model may be translated to primate models to genetically confirm the identity of the FEF layer IV ambiguous neurons.

How the pieces of remapping in FEF layer IV may be combined to create full remapping in layer V is unknown, but it could involve local microcircuits extending into layer II/III (Fig. 3; modified from Shin and Sommer 2012; based on interlaminar connectivity reviewed by Douglas and Martin 2004). Just before a saccade, layer IV neurons that project to layer II/III would increase their sensitivity to visual stimulation at the receptive field and decrease their sensitivity to visual stimulation at the future field (in Fig. 3 represented as ↑RF and ↓FF). This is the opposite of full remapping, so in layer II/III the signals would need to be inverted (↓RF and ↑FF). Regarding that process and subsequent steps, there are no data yet to indicate mechanisms. Layer II/III-specific recordings with laminar probes, or a computational model version of the Fig. 3 conceptual diagram, could help to advance our understanding of the microcircuitry.

Microcircuits that create remapping need input from all of visual and oculomotor space, because remapping is a full-field phenomenon: it occurs in association with saccades made into both the contralateral and ipsilateral hemifield (e.g., Heiser and Colby 2006; Sommer and Wurtz 2006). This seems paradoxical for the FEF given that it is a highly lateralized structure, as is its source of corollary discharge, the SC. Both the FEF and SC represent contralateral space almost exclusively. For FEF neurons to remap in both directions, they would need to receive corollary discharge from both SCs, left and right. Do they? Crapse and Sommer (2009) examined this question, using orthodromic activation to identify recipient neurons in FEF, using stimulation in both the homolateral SC (as done previously by Sommer and Wurtz 1998, 2004a) and the opposite SC. They showed that some neurons in the FEF do receive input from both SCs and that the relative strengths of those inputs predict the lateralization of the FEF neurons' receptive and movement fields. Therefore, the FEF is the target of bilateral inputs that effect an elegant structure-function relationship and could provide information about all saccades.

A related question about full-field remapping is how FEF neurons are able to respond to visual stimuli far outside their classical receptive field, even in ipsilateral space. One possible answer is that they receive visual information from large swaths of the visual field, beyond what their classical receptive fields imply. That is, they have a “covert” receptive field at their synaptic inputs that is larger than the “overt” receptive field at their spiking output. Studies of primary visual cortex demonstrate that this is plausible anatomically (e.g., Angelucci et al. 2002) and physiologically (macaque: Bair et al. 2003; cat: Bringuier et al. 1999; reviewed by Gilbert et al. 1996). Direct study of such input-output relationships can be accomplished in vivo in mouse visual cortex with two-photon imaging of dendrites combined with whole cell recordings (Jia et al. 2010), methods that are at early stages of application in behaving monkeys (Heider et al. 2010; Tan et al. 2014). An alternative model could involve all-to-all connections and a lateral transfer of visual signals in the FEF akin to that proposed for LIP (Quaia et al. 1998b). As discussed above, evidence for this idea in the form of a spread of visual activity has not been found in the FEF (Mayo et al. 2016; Sommer and Wurtz 2006).

Behavioral Implications of Remapping

Presaccadic remapping has long been postulated to maintain visual continuity across saccades (Duhamel et al. 1992). But despite decades of careful psychophysical work on the behavioral implications of remapping (Bansal et al. 2015; Binda et al. 2009; Bridgeman et al. 1975; Collins et al. 2009; Deubel et al. 1998; Jayet Bray et al. 2016; Melcher 2007; Rao et al. 2016a; for reviews see Hall and Colby 2011; Melcher and Colby 2008; Sommer and Wurtz 2008a, 2008b), a link between remapping and perception has only recently gained neurophysiological traction (Crapse and Sommer 2012; Mirpour and Bisley 2012, 2016). Particularly compelling was a recent, causal experiment that affirmed a key role for MD thalamus. Cavanaugh et al. (2016) trained monkeys to report their perceived saccade vector during MD thalamus inactivation and no-inactivation control trials. Building on recent efforts to establish monkey-compatible paradigms for perceptual reporting (Joiner et al. 2013a), they showed that inactivation of the lateral edge of MD thalamus led to marked changes in the reported perception of where the eyes moved, unrelated to where the eyes actually moved. The results provide the strongest evidence yet that MD-mediated corollary discharge, and by extension the FEF remapping that depends on it (Sommer and Wurtz 2006), has an influence that extends to perception.

How might remapping affect perception at the neural level? One possibility is that single neurons may report transsaccadic visual change, that is, whether a visual stimulus remains stable or not across saccades. Crapse and Sommer (2012) demonstrated that some FEF neurons exhibit such a report. When the receptive fields of the neurons were brought onto a visual stimulus by a saccade, the neurons responded differently to the stimulus depending on whether it had remained stable the whole time or moved during the saccade, and the reafferent responses were tuned to the amount of intrasaccadic movement. Surprisingly, the neurons also reported featural changes (e.g., color), suggesting that they play multiple roles in transsaccadic change detection (Crapse and Sommer 2012). The transsaccadic tuning occurred in neurons that remapped, as predicted, but also in other visual neurons, suggesting a propagation of the effect throughout FEF. The results point to a high-level change detection program that emerges from microcircuits of the FEF and, perhaps, interconnected cortical areas.

The final link from neural activity in FEF to perceptual continuity across saccades still needs to be understood. The following is one hypothetical framework, presented as a sequence of operations that could link neurophysiology and perception. Corollary discharge and visual inputs combine in FEF to generate presaccadic remapping, which provides a mechanism for sampling the postsaccadic receptive field before the eyes move (Fig. 4A). Sent to extrastriate cortex, the remapped response provides a prediction of what will be in the receptive field after the saccade that is compared with reafferent responses from V1 (Fig. 4B; Crapse and Sommer 2008). The resulting prediction error modulates reafferent responses in the FEF and possibly other areas as a function of how much the visual stimulus changed during the saccade (Fig. 4C; Crapse and Sommer 2012). At the level of the whole visual scene and populations of neurons, the transsaccadic visual change signal may provide a top-down input to saliency maps (Fig. 4D; Arcizet et al. 2011; Treue 2003) or, more precisely, priority maps (Bisley and Goldberg 2010; Fecteau and Munoz 2006). A prediction of this hypothetical framework is that modulations in FEF reafferent responses correlate with perception of transsaccadic visual changes. Preliminary data support this hypothesis (Crapse and Sommer 2010a, 2010b). A broader question is whether neural correlates of transsaccadic visual change are found only in FEF or throughout the “where” pathway for spatial visual perception (Mishkin et al. 1983).

Fig. 4.

Neural-perceptual link for transsaccadic visual continuity. A: simple visual scene with 1 object, a star. Left: just before a saccade, a neuron with a receptive field (RF) as shown remaps its visual sensitivity to the future field (FF). In this example, the FF is at the star. Right: after the saccade, the presaccadic FF is replaced by the postsaccadic RF (New RF) at the same location. The response at the FF, in effect, provides a prediction of what the neuron will “see” in its New RF after the saccade. B: circuit schematic of how the FF prediction might be used. The FF prediction is sent from the FEF to extrastriate cortex (dashed arrow). It is compared with New RF input that arrives from V1 (solid arrow). Any discrepancy, or “prediction error,” is fed back to FEF and possibly used locally to modulate reafferent visual responses (dotted arrow). Modified from Crapse and Sommer (2008) with permission from Elsevier. C: the end result is that reafferent responses are tuned for transsaccadic change, as found in FEF by Crapse and Sommer (2012). The cartoon depicts reafferent responses to a star that either changed during the saccade (Δ⭐, dashed line) or stayed the same (☆, solid line). D: at the population level, the transsaccadic change-induced modulation may provide an input to saliency maps. First panel: example scene that includes the star along with other objects. For simplicity, all objects are presumed to have equal salience. Second panel: during a saccade, the star moves slightly. Third panel: after the saccade, the reafferent response across the population of neurons is enhanced for the star. Fourth panel: this increased reafferent response is read out as increased salience (or priority) of the star relative to the rest of the scene. Other influences to saliency/priority maps (e.g., attention) could be active as well but are not shown.

Another prediction of the hypothesis is that remapped responses convey information about visual images. Moreover, if remapped responses are compared with reafferent responses, it would seem advantageous for the future field and receptive field to respond identically to visual stimuli. The predictions seem to be confirmed for LIP, where some neurons have remapped responses with visual tuning similar to their receptive field tuning (Mirpour and Bisley 2012; Subramanian and Colby 2014). However, the degree of similarity is difficult to quantify because the retinal eccentricity corresponding to the future field and receptive field will generally be different, so that the two fields may “see” images at different visual acuity. In the FEF, receptive fields have long been thought to have little feature tuning (Mohler et al. 1973), suggesting that remapped responses may contribute to visual processing only indirectly, as pointers in the service of spatial attention (see Cavanagh et al. 2010; Mayo and Sommer 2010). However, the visual capabilities of FEF neurons may be underappreciated. Reports have found feature selectivity in FEF (e.g., Bichot et al. 1996; Peng et al. 2008) and support a role for it in top-down control of feature attention (Zhou and Desimone 2011).

One subtlety of remapping in the FEF is that remapped responses can occur long after the disappearance of the visual stimulus that elicited them (Umeno and Goldberg 2001). The hypothetical process of Fig. 4 can accommodate such neurons despite their unusual “memory remapping.” If an image disappears, the neurons would continue to produce remapped responses despite the absence of reafferent input, yielding a large prediction error. But it does not matter, because the prediction error has nothing to modulate. There are no reafferent visual responses. The memory remapping signals may contribute to other processes of spatial memory maintenance as discussed by Umeno and Goldberg (2001).

Circuit-Based Modeling

Many computational models have examined the origin and role of presaccadic visual remapping. They have incorporated varying degrees of biologically based model design, with some meant to be abstract and others more neuromorphic. An example of the latter was the all-to-all network of Quaia et al. (1998b), one of the first models to provide a hypothesis for the neural instantiation of remapping based on physiological data and the circuitry of the primate brain. Related models used topographically organized layers that produced visual updating with velocity commands (Bozis and Moschovakis 1998; Droulez and Berthoz 1991), a functional architecture likened to that in the primate oculomotor system. More recent work in neural networks abstract away from known circuits and brain regions to focus on the properties of trained networks (White and Snyder 2004; Xing and Andersen 2000), linear and nonlinear computations in spatial updating (Deneve et al. 2007; Pouget et al. 2002; Salinas and Sejnowski 2001), and geometric aspects of remapping (Keith et al. 2010; Keith and Crawford 2008).

In the tradition of Quaia et al. (1998b), Rao et al. (2016b) designed a circuit-based model for investigating remapping (Fig. 5A). The model served as the visuosaccadic system of a robot. Rao et al. (2016b) used the system to test useful visual stability as a proxy for perceptual visual stability. Could the robot be trained to make accurate visually guided reaches despite saccades of its camera “eye”? Would this be achieved through the emergence of presaccadic remapping? A simulated FEF received retinotopic visual input and corollary discharge from a simulated SC that controlled the eye. Importantly, the FEF was not trained directly; rather, the model as a whole was trained to achieve accurate reaches during saccades. Rao et al. (2016b) found that in the trained model the neural locus of activity shifted in the simulated FEF just before each saccade (Fig. 5A, FEF sheet). The effect corresponded to presaccadic remapping in the visual field as observed in neurophysiology experiments (Fig. 5B; see Rao et al. 2016b for details). Simulated inactivations of the corollary discharge pathway from SC to FEF replicated and helped to explain inactivations of MD thalamus performed in vivo (Rao et al. 2016b; Sommer and Wurtz 2006).

Fig. 5.

Circuit-based computational model. A: the neural network of Rao et al. (2016b), shown overlaid onto macrocircuits for remapping to illustrate concepts. Brain areas were modeled as sheets of neurons. The FEF sheet received visual and corollary discharge input mediated by a recurrent hidden layer. FEF output was combined with internal eye position (EP) signals to achieve craniotopic coordinates. The network was trained to optimize accuracy of a robot that points to a red ball (depicted as the red circle in the sheets) while making saccades. Once the model was trained, each saccade command in the SC sheet (yellow arrow) induced an equal and opposite presaccadic shift in the locus of neural activity in the FEF sheet (white arrow). B: the presaccadic shift in the FEF sheet implied a shift of sensitivity in the visual field. Consider a neuron located up and to the left in the FEF sheet (at the location of the red circle). Its classical receptive field (RF) is up and to the left. Long before the saccade, it is inactive because the red ball is not in its RF. Just before the saccade, it becomes active. It is responding to the red ball, even though the ball has not moved. The neuron's visual sensitivity has remapped to the FF. In the trained model, this remapping required corollary discharge (CD) but was time-locked to a different signal, the update in eye position from its initial location (EP) to its final location (EP′). C: evidence for synchronization of remapping and eye position updating from physiological and psychophysical studies. Shown superimposed are cumulative distributions of remapping onset times in FEF (red curve), eye position update times in thalamus (orange dashed curve), and the time course of the internal representation of eye position in humans (green curve). Based on data from Sommer and Wurtz (2006), Tanaka (2007), and Honda (1991), respectively. Modified with permission from Rao et al. (2016b). CC-BY 4.0, modified (http://journal.frontiersin.org/article/10.3389/fncom.2016.00052/full).

Intriguingly, the model suggested the importance of eye position signals to remapping. An eye position sheet was included in the model, and its instantaneous representation of eye position was summed with FEF output to achieve a head-centered reference frame that the robot needed for reaching. During training, the timing of presaccadic remapping in FEF became synchronized to the updating of eye position (from its presaccadic to its postsaccadic location). Time-locking of the signals was necessary for continuously accurate reaching. In the real brain, eye position updates are often predictive. Many neurons in the thalamus provide eye position signals that update before the eyes move (Schlag-Rey and Schlag 1984; Tanaka 2007; Wyder et al. 2003), and such signals seem to influence cerebral cortical visual processing (Graf and Andersen 2014; Morris et al. 2012, 2016; Sereno et al. 2014). Including predictive eye position signals in the model yielded time-locked, predictive remapping in FEF.

The implication is that, in the real brain, the timing of remapping may be synchronized to the timing of eye position updates in order to maintain a continuously useful coordinate transformation. This is different from the prior assumption that remapping is synchronized to saccade initiation (e.g., Sommer and Wurtz 2008a). The new hypothesis maintains that the remapping-saccade correlation is indirect, while the true functional relationship is between remapping and eye position updating. The hypothesis is supported by a comparison between papers that reported the timing of remapping in FEF and the timing of eye position update signals as recorded in thalamus and inferred psychophysically (Fig. 5C). Reciprocal, monosynaptic projections between the FEF and thalamic eye position areas are profuse (Barbas and Mesulam 1981; Huerta et al. 1986; Kievit and Kuypers 1977; Stanton et al. 1988), providing a structural basis for the putative synchronization. There are several neurophysiological approaches for testing the hypothesis in vivo, as described in Rao et al. (2016b).

Unresolved Issues

The study of presaccadic remapping has become a mature area of research. Little disagreement remains about the core finding that visual sensitivity moves just before a saccade. Yet some discrepancies between studies persist. The most puzzling issue relates to the spatial attributes of remapping. Using the nomenclature of Marino and Mazer (2016), who recently summarized the discrepancies, some experiments find “forward remapping,” in which visual sensitivity moves parallel to the saccade, while others find “convergent remapping,” in which visual sensitivity moves toward the target of the saccade. Both forms of remapping have gained theoretical support from models having different suppositions and architectures (e.g., forward mapping: Quaia et al. 1998b; Rao et al. 2016b; convergent remapping: Hamker and Zirnsak 2006; Zirnsak et al. 2010). We evaluate the physiological evidence for forward and convergent remapping with a focus on technical differences between studies that hint at reasons for the discrepancies in their results.

Direction of saccade.

In most of the brain areas where remapping occurs, neurons fire for saccades made into contralateral space. The SC and the FEF contain significant motor-related presaccadic activity, often found in the same neurons that have visual responses (“visuomovement neurons”) (e.g., Bruce and Goldberg 1985; Wurtz et al. 2001). Extrastriate visual areas such as LIP exhibit similar presaccadic surges of activity with saccades (Barash et al. 1991a, 1991b; Wurtz et al. 2001). The region of the visual field into which saccades cause elevated firing (the “movement field”) generally overlaps with the visual receptive field. Hence it is critical to evaluate the extent to which apparent remapping represents a response to the saccade rather than to the visual stimulus. One can make this evaluation using “saccade-only” control trials in which a visual stimulus is not presented. Confounds from saccade-related activity can be reduced or eliminated by directing saccades outside the movement field.

Churan et al. (2011, 2012), Duhamel et al. (1992), Kusunoki and Goldberg (2003), Nakamura and Colby (2002), and Walker et al. (1995) dealt with this issue in LIP and the SC by directing saccades outside of the receptive/movement field. In the FEF, Sommer and Wurtz (2006) and Umeno and Goldberg (1997, 2001) used a sequence of saccades. The first saccade was made to a location outside the receptive/movement field, and then a second saccade was made away from the location of the visual stimulus used to probe the future field. Knowing that the FEF contributes to planning the generation of sequences of movements (Schall 2002, 2013), the inclusion of the second saccade reduced the possibility that the observed neural activity was a motor plan toward the visual probe. The confounding effects of such motor plans were shown by Walker et al. (1995; their Fig. 15B). In all of these studies, forward remapping was found. Even the use of trials in which visual probes were placed near the saccadic target (Sommer and Wurtz 2006) revealed no evidence of convergent remapping.

In contrast, convergent remapping was reported by Tolias et al. (2001) in V4 and by Zirnsak at al. (2014) in FEF. Neupane et al. (2016b) followed up on the V4 work with a study that tested the influence of saccade direction. They found that convergent remapping occurred only for contralateral saccades and began well after saccade initiation. Early after saccade initiation, forward remapping occurred for all directions of saccades. To clarify the dynamics of remapping for contralateral saccades, they designed a new task in which saccades went partway toward the receptive field (Neupane et al. 2016a), a geometry that disambiguates forward and convergent remapping by forcing them to go in opposite directions. Forward remapping dominated at all time points (Neupane et al. 2016a). The presence of convergent remapping in some experiments remains curious. In V4 and FEF, it seems to occur only for contralateral saccades (Neupane et al. 2016b; Zirnsak et al. 2014). In that configuration, the saccade target—a major attractor of attention (Deubel and Schneider 1996; Kowler et al. 1995)—is located in the same hemifield as the receptive field. This nearby attentional focus likely shifts a neuron's visual sensitivity toward the target (Connor et al. 1996, 1997; Hamker and Zirnsak 2006; Zirnsak et al. 2010) and distorts forward remapping.

Data alignment and temporal averaging.

Nearly all studies of presaccadic remapping have analyzed the remapped activity aligned to saccade initiation, which has been shown repeatedly to provide a tighter response distribution than alignment to stimulus onset (Joiner et al. 2013b; Nakamura and Colby 2002; Sommer and Wurtz 2006; Umeno and Goldberg 1997; Walker et al. 1995; reviewed by Sommer and Wurtz 2008a). This makes sense because neurons should remap only when the saccadic system has committed to moving the eyes, as supported by psychophysical studies (Atsma et al. 2014) and modeling (Rao et al. 2016b). Neupane et al. (2016b) demonstrated that the temporal window for quantifying the remapping activity is important as well. In V4, forward remapping occurs only during the first 200 ms after a contralateral saccade. After that, convergent remapping takes over. How data are aligned and averaged, therefore, can impact conclusions about forward vs. convergent remapping. The one study that found convergent remapping in FEF (Zirnsak et al. 2014) aligned the data differently from most other studies—to stimulus onset rather than saccade initiation—and used an averaging window of 50 to 350 ms after stimulus onset, or roughly 20 ms before to 280 ms after saccade initiation (given that their stimuli were presented ∼70 ms before saccade initiation). The extent to which the findings of that study may have hinged on these analysis choices is an open question.

Coarseness of spatial mapping.

The density of spatial testing varies appreciably between studies and may contribute to different interpretations of the spatial nature of remapping. For example, in LIP Kusunoki and Goldberg (2003) tested the temporal properties of remapping in detail but therefore could not test more than a couple of spatial locations, at the receptive field and at the future field. A recent study from the same lab, studying LIP with higher spatial resolution, found spatial dynamics that were unrecognized previously (Wang et al. 2016). Though experimentally challenging, an exhaustive assessment—in any brain area—of the full spatiotemporal dynamics of remapping still needs to be accomplished. One step toward that goal is to test a large region of the visual field with a grid of stimuli, an approach pioneered by Tolias et al. (2001), followed up by Zirnsak et al. (2014), and refined by Neupane et al. (2016b). A new, different approach uses probabilistic modeling (Mayo et al. 2015, 2016). As monkeys fixate, FEF receptive fields are mapped in spatial and temporal detail using rapid, sparse sampling of the visual field and a generalized linear model. Applied during saccades, the method could settle the controversy of forward vs. convergent remapping in FEF. Preliminary results appear to confirm forward remapping (Mayo et al. 2016).

Differences across cortical layers.

As described above in Microcircuits for Remapping, remapping is known to differ across cortical layers, at least in the FEF (Shin and Sommer 2012). Physiologically identified layer V neurons exhibit full remapping in the forward direction (Shin and Sommer 2012; Sommer and Wurtz 2006). A caveat to this conclusion is that a large grid of probe locations was not used, although if convergent remapping occurred it should have been detected by probes near the saccade target (Sommer and Wurtz 2006). Zirnsak et al. (2014) provided a much denser sampling of the visual field and concluded that FEF neurons engage in convergent remapping. Their neurons were recorded throughout the FEF, with laminar locations unknown. Putting these conclusions together and taking both at face value yields the possibility that the direction of remapping (forward vs. convergent) may vary between layers and output channels of the FEF. Convergent remapping in layer II/III neurons that project to other cerebral cortical regions could contribute to focusing attention around the time of the saccade, while forward remapping in the layer V neurons that project to the SC could be more involved in the maintenance of a stable visual representation. A better understanding of laminar-specific output signals of the FEF could play a key role in untangling its multitude of interacting signals (e.g., attention, target selection, and movement generation; Mayo and Verhoef 2014; Schall 2002, 2013).

Conclusions

Just before saccades, neurons in many parts of the visual system remap—they transiently sample regions of the visual field outside of their classical receptive fields. We have come a long way in understanding the circuits that mediate these central effects. In this review, we highlighted three main hypotheses:

H1.

The FEF is the source of presaccadic remapping that is sent to the rest of the brain (based on analyses of macrocircuits and microcircuits).

H2.

This remapping compares pre- vs. postsaccadic visual space, which, in turn, modulates reafferent visual responses in FEF and extrastriate cortex to signal violations of visual continuity (based on consideration of the behavioral implications of remapping).

H3.

The timing of remapping is synchronized to eye position updates in order to maintain coordinate transformations for seamless visually guided behaviors (based on circuit-inspired computational modeling).

Note that H3 implies that the ultimate reason why predictive remapping occurs is to optimize visually guided movements. The ability to perceive visual continuity across saccades (H2) would be a secondary benefit (for more on this idea, see Rao et al. 2016b). This conclusion is consistent with the broader idea that motor system evolution forms the basis for higher cognitive and perceptual functions (for a recent, comprehensive review, see Mendoza and Merchant 2014).

All of hypotheses H1–H3, though speculative, are based on a body of interrelated literature. All of them are experimentally testable. Evaluating them may provide a systematic, promising pathway forward to revealing the functions of presaccadic visual remapping. More broadly, the progress made in this field of research serves as a prompt, we hope, for investigators of other sensory systems and animal models to examine the receptive fields of their neurons for evidence of spatial lability—a remarkable form of active sensing—during behavior.

GRANTS

This work was supported by the National Science Foundation (NSF) through a Graduate Research Fellowship (GRFP) and an IGERT (DGE-1068871) fellowship to H. M. Rao and by the National Institutes of Health through Grant R01 EY-017592 to M. A. Sommer.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

H.M.R. and M.A.S. conceived and designed research; H.M.R. and M.A.S. performed experiments; H.M.R. and M.A.S. analyzed data; H.M.R., J.P.M., and M.A.S. interpreted results of experiments; H.M.R., J.P.M., and M.A.S. prepared figures; H.M.R., J.P.M., and M.A.S. drafted manuscript; H.M.R., J.P.M., and M.A.S. edited and revised manuscript; H.M.R., J.P.M., and M.A.S. approved final version of manuscript.