The frontal eye field as a prediction map

Abstract

Predictive processes are widespread in the motor and sensory areas of the primate brain. They enable rapid computations despite processing delays and assist in resolving noisy, ambiguous input. Here we propose that the frontal eye field, a cortical area devoted to sensorimotor aspects of eye movement control, implements a prediction map of the postsaccadic visual scene for the purpose of constructing a stable percept despite saccadic eye movements.

Introduction

The brain is an inferential machine. Both its motor areas and sensory networks engage in predictive computations (Miall and Wolpert, 1996; Bullier, 2001; Friston, 2005). Hierarchical models suggest that visual cortical neurons fire predominately to signal deviations from predicted inputs (Rao and Ballard, 1999; Lee and Mumford, 2003). Studies in the motor realm, too, suggest that predictions are used for adaptive control (Miall and Wolpert, 1996; Hwang and Shadmehr, 2005). Here we propose that predictive operations for both the sensory and motor domains find unification in the primate visuosaccadic system for the purpose of constructing a stable transaccadic percept.

The primate visuosaccadic system faces a serious problem (Fig. 1). It must generate a stable percept despite the frequent disruptions in gaze induced by saccadic eye movements. To appreciate this, consider the saccade as an event comprised of three constituent parts: presaccadically, a target is selected; intrasaccadically, the world as sensed actually shifts; and postsaccadically, the target is foveated. The intrasaccadic component is the problem. Were the animal to perceive the world exactly as sensed during the intrasaccadic epoch, the visual scene would seem to leap from place-to-place dozens of times per minute. Yet it is not perceived as such, and the brain can even distinguish what aspects of the jumpy visual inflow are artefactual (due to saccades) as opposed to real (due to changes in the world). How does it do it?

Fig. 1.

The saccadic eye movement as a visual problem. Each saccade is composed of three elements: a presaccadic component when a target is selected and the saccade initiated, an intrasaccadic component during which time the world as sensed actually shifts, and a postsaccadic component when the target is reached. The intrasaccadic component is the most disruptive element for the visual system. The fact that the world shifts with each saccade, and yet is not perceived to do so, implies a corrective mechanism within the visual system. This mechanism is rooted in predictive operations. Fov, fovea; m, movement vector; v, retinal vector emphasizing direction and magnitude of the shift.

An important cortical gaze control component is the frontal eye field (FEF) (Fig. 2; Sommer and Wurtz, 2008). Located within the anterior bank of the arcuate sulcus, the FEF contains topographically arranged neurons with response properties that span the continuum from purely visual to purely movement. The FEF receives input from many cortical and subcortical areas including the superior colliculus (SC) via thalamic relay neurons. We recently demonstrated that convergent inputs from both SCs provide each FEF with a full-field representation of all saccades and all of visual space (Crapse and Sommer, 2007).

Fig. 2.

Full-field representation in the frontal eye field (FEF). The FEF has information about all of visual space and all eye movements via pathways from ipsilateral (Ips) and contralateral (Con) superior colliculus (SC).

Like several other primate areas, the FEF contains neurons that shift their response fields (RFs) before eye movements (a “shifting RF”) (Umeno and Goldberg, 1997). While a typical RF is firmly retinotopic and samples a new part of the visual field (the new RF) only after the eye moves, a shifting RF is dynamic and starts sampling the new RF location even before a saccade. Such neurons depend on corollary discharge (CD) from the midbrain to trigger the shift and are thought to contribute to a percept of visual stability (Sommer and Wurtz, 2006, 2008). How might these neurons influence the rest of the brain?

A prediction map in primate frontal eye field

We propose that shifting neurons of the FEF are components of a larger FEF inferential architecture that engages in predictive coding. That is, the FEF is a prediction map. This scheme assigns a causal role to the FEF in the construction of a stable visual percept despite saccadic interruptions. According to this conception, predictions of the future scene (postsaccadic) are generated based on extrapolations from the current scene (presaccadic). Neurons with shifting RFs, informed spatiotemporally of the imminent change in gaze by CD, collect data about what will fall in their RF after the saccade. These data are convolved with estimates of an internal model of the visual world. What follows is a prediction of the visual structure of the postsaccadic scene.

How would the prediction be used? We hypothesize that around the time of the saccade, the predictive signals are sent from the FEF to the posterior lobes. The visual lobes are primed with activity-constraining expectations. Deviations from the predicted inputs (prediction errors) are reported back to the FEF. The prediction errors therefore are ultimately manifest in the visual responses of FEF neurons. This residual alerts the FEF to unpredicted events. This may entail calibration/updating if the error is related to a miscalculation/noise or it may provide useful information about the environment that was unexpected, i.e., perhaps something moved or suddenly appeared. Either way, the final result of the iterative signalling would be the perception of a stable world despite the change in gaze. In the rest of this article we will formalize these ideas.

Computational basis of the prediction map

The prediction map may be grounded in an inferential process based on empirical Bayesian principles. Neurons throughout the brain seem to engage in probabilistic and inferential computations (Gold and Shadlen, 2007). The probabilistic aspect seems necessary because of the noise and ambiguity intrinsic to neural computation (Knill and Pouget, 2004). Bayesian inference would allow prior and conditional probability distributions to be utilized to generate a posterior distribution, i.e., the outcome expected on the balance of known probabilities and other inputs (in other words, a prediction; Fig. 3). For the case of a single saccade, the proposed Bayesian computation would express the conditional expectation or probability of observing the postsaccadic scene (p) given the relative probability of two independent pieces of evidence: the visual structure of the current scene (c), and various saccadic parameters (s), written:

$$\mathbf{P}(\mathrm{p}\mid \mathrm{c},\mathrm{s})=\frac{\mathbf{P}(\mathrm{c}\mid \mathrm{p})\mathbf{P}(\mathrm{s}\mid \mathrm{p})\mathbf{P}(\mathrm{p})}{\mathbf{P}(\mathrm{c})\mathbf{P}(\mathrm{s})}={\lambda }^{\rho }$$

Fig. 3.

Computational basis of the prediction map. A prediction map could be based on Bayesian computations. See text for details.

Bayesian priors about the structure of the visual scene P(c) and how the world behaves during a saccade P(s) could be learned and updated through experience and stored in FEF networks. The priors would permit calculation of local conditional probabilities relating the postsaccadic scene to the current scene P(c|p) and the postsaccadic scene to the saccade P(s|p). Ultimately, these computations would result in a global conditional expectation λ^ρ that would be convolved with a generative model Ψ to generate a prediction of the postsaccadic scene Ψ(λ^ρ). The prediction would be compared with the actual input λ^α to yield a prediction error ε=λ^α−Ψ(λ^ρ).

Physiological mechanism of the prediction map

Mechanistically, frontal modulatory control of the posterior lobes could be implemented through imposed patterns of synchronization mediated by cortico-cortical connections (Womelsdorf et al., 2007). Visually evoked activity of single neurons is surprisingly quite deterministic (Arieli et al., 1996). The oft-encountered variability in single neuron responses seems to emerge from the dynamics of ongoing network activity. This fact could be exploited by the prediction map for purposes of ensuring transaccadic perception. The imposed predictions could alter the spatiotemporal properties of visual cortical network activity to influence the specific visually evoked responses that are ultimately integrated into the network and expressed as spikes. The FEF would initiate a specific synchronization pattern among the relevant visual neurons (those activated by features of the current scene) and this would serve to amplify behaviourally relevant signals in the cortex.

The sequence of events

A detailed account of what we propose occurs during each voluntary saccade is as follows (Fig. 4). The imminent saccade would initiate an iteration of recurrent processing between the FEF and an assortment of visual cortical areas, beginning with receipt in the FEF of CD (leftmost line) from the SC. This information represents the when and where (a vector quantity) of the imminent change in gaze and induces a transient alteration in local functional topology of the FEF network. At the centre of the alteration would be FEF network hubs (h) (Sporns et al., 2007). FEF network hubs have access to information about the entire visual field (I_hD(h)(t)) and are considered to be neurons with shifting RFs. The CD serves a gating function (C_hD(h)(t)) and routes the appropriate visual information through the hub (f_hD(h)(t)) to downstream neurons D(h). [If CD is not present (C_hD(h)(t)=0), the hub simply passes on the visual information provided to it from node x.] These relationships may be written as:

$$\begin{array}{l}{f}_{\text{hD}(\mathrm{h})}(t)={\mathrm{I}}_{\text{hD}(\mathrm{h})}(t){\mathrm{C}}_{\text{hD}(\mathrm{h})}(t),\text{if}{\mathrm{C}}_{\text{hD}(\mathrm{h})}(t)>0\\ \text{Otherwise}:f_{\text{hD}(\mathrm{h})}(t)=x,\text{if}{\mathrm{C}}_{\text{hD}(\mathrm{h})}(t)=0\end{array}$$

Fig. 4.

Mechanics of the prediction map. (A) A centrepiece of the prediction map would be neurons with shifting receptive fields. These would serve as network hubs (h) that selectively route visual information fhD(h)(t) through the network based on the information IhD(h)(t) provided by upstream neurons P(h) and CD from the midbrain ChD(h)(t). This information would then be forwarded to neurons downstream D(h) that perform additional computations for the sake of prediction. Equations are explained in text. (B) Predictions Ψ(λ^ρ) would exit the FEF as feedback that targets various areas of the visual brain. Errors ε=λ^α−Ψ(λ^ρ) between the predicted visual input Ψ(λ^ρ) and actual visual input λ^α would be then routed back to the FEF.

The RF is said to shift and corresponds to a collection of visual information about the portions of space that the RF will encompass after the saccade. This information then would be combined with the causal estimates of the generative model embedded within the FEF network. A prediction would be generated and exported, manifested as a round of perisaccadic synchronized activity between the frontal and posterior lobes. The synchronized activity would alter the visual cortical network dynamics and effect how the visual stimulus-driven activity is integrated into the network. The synchronized activity is equivalent to a prediction, the output of the generative model. If the actual visual input does not match this prediction, then residuals (prediction errors) are routed downstream back to the FEF and iterative processes are continued.

Site of prediction error calculation

The prediction errors could be calculated at any number of visual cortical depots. Virtually every portion of the cortical mantle exhibits some degree of saccadic modulation (Baker et al., 2006). Dorsal stream components seem most likely for two reasons. First, the temporal structure of information flow through the primate visual system points to a dorsal stream speed advantage over the ventral stream (Bullier, 2001; Bar, 2007). Dorsal stream components exhibit activation latencies that often trail V1 responses by a meagre 10 ms. Some even activate before V1 (Schmolesky et al., 1998; Bullier et al., 2001). This speed advantage would be optimal for the rapid detection and routing of prediction errors back to the FEF. Second, the dorsal stream is known as the where pathway (Ungerleider and Mishkin, 1982). Components are concerned with motion and the physical location of objects in the visual scene. Since the primary concern of the visual system during a saccade is to ensure a stable world, the dorsal stream would seem to be particularly important in controlling the illusory percept of motion concomitant to saccade generation. Any errors related to the prediction could be a consequence of an object that actually moved on its own accord and thus was not predicted. The object would be with high probability worthy of further inspection. The FEF would then direct the allocation of attentional and targeting mechanisms to these objects via retrojections of activity to stations of the ventral stream such as V4 and IT (Moore and Armstrong, 2003; Hamker, 2005).

Relation to previous FEF studies

Previous studies have uncovered a number of FEF response properties consistent with the notion of a prediction map. One implication of prediction error in general is that single neurons should not respond if the stimulus falling in its RF is predicted. Burman and Segraves (1994) found that when monkeys rescanned a previously scanned image, visual activity was virtually unaffected by the contents of the image that fell within the RF. In contrast, these same neurons fired vigorously during the initial scan and when a target light suddenly appeared in the RF. After the initial scanning, a memory trace of the image was likely formed, and only deviations from the predicted image components were signalled as visual bursts. This suggests that the visual response of FEF neurons signals prediction error, the unexpected.

An additional implication of a prediction map applies to the movement-related aspects of the FEF, generally. If a function of the FEF is to generate predictions, then movement responses, too, may be viewed probabilistically. In fact, some pure movement cells have been found that generate vigorous bursts even when the monkey errs and does not perform the saccade (Bruce and Goldberg, 1985). Other movement neurons fail to fire even when a saccade is launched into its RF. This is evidence that the movement signal in FEF is closely akin to a prediction or probability estimate of a saccade being generated into its movement field. It bears little relationship to saccade dynamics (Segraves and Park, 1993). We suggest that the forced manner in which FEF movement signals have been studied classically, counting spikes while monkeys make required eye movements into a neuron’s RF, may mask this probabilistic aspect of FEF function and redirect focus on alleged deterministic aspects of movement generation.

Other frontal lobe functions

The prediction map is consistent with a number of other phenomena involving the frontal lobes and visual function. Among other things, the frontal lobes are thought to play a role in resolving visual ambiguity. Visual scenes are often ambiguous; the sensory data are consistent with multiple interpretations. This often results in illusions and multistable percepts, i.e., depth reversals, binocular rivalry, ambiguous figures, etc. A host of studies point to a role of the frontal lobes, FEF included, in resolving visual ambiguity (Lumer et al., 1998). Some contend that the frontal lobes bias or control the posterior lobes in an attempt to resolve the ambiguity and arrive at a stable percept (Miller and Cohen, 2001). We submit that the same frontal circuits that are responsible for resolving visual ambiguity and arriving at a stable percept of the present scene are part of the prediction map and are utilized by the oculomotor system for transsaccadic stability.

In a more speculative sense, the frontal lobes are implicated in processes related to mental imagery (Goebel et al., 1998). When a subject is asked to “imagine” a particular image the frontal lobes invariably come on line. Feedback connections from the frontal lobes to the posterior lobes are thought to activate the same circuits that would be activated if the actual image had been presented to the retinas (Kosslyn et al., 1995). Furthermore, the eye movements of subjects asked to imagine a particular scene reflect the content and spatial arrangement of objects populating the scene (Mast and Kosslyn, 2002). Again this may involve the same circuits, as ancestral primates were likely making saccades and requiring mechanisms of transsaccadic stability long before they were engaging in mental imagery and imaginative musings.