The neural instantiation of a priority map

Abstract

The term priority map is commonly used to describe a map of the visual scene in which objects and locations are represented by their attentional priority, which itself is a combination of low-level salience and top-down control. The aim of this review is to examine how such a map may be represented at the neuronal level. We propose that there is not a single, common map in the brain, but that a number of cortical areas work together to generate the resultant behavior. Specifically, we suggest that the lateral intraparietal area (LIP) of posterior parietal cortex provides a simple representation of attentional priority, which remaps across saccades, so that there is an apparent allocentric map in a region with retinocentric encoding scheme. We propose that the frontal eye field (FEF) of prefrontal cortex receives the responses from LIP, but can suppress them to control the flow of eye movement behavior, and that the intermediate layers of the superior colliculus (SCi) reflect the final saccade goal. Together, these areas function to guide eye movements and may play a similar role in allocating covert visual attention.

Introduction

The concept of a priority map is derived from the saliency map models of Itti, Koch and colleagues [1], which were aimed at modeling shifts of visual attention. These models were primarily driven by low-level salience, with a modicum of top-down inputs. Attention was allocated, in a winner-take-all manner, to the peak of the map, which was then inhibited so that attention could move on to the next highest point. However, a whole host of factors influence the allocation of attention, so we [2] and others [3,4] prefer to use the term priority map to describe the map that ultimately is used to guide eye movements and covert visual attention. We define a priority map as a map of the visual scene, in which activity is driven by low-level salience and by a range top-down influences, such as task rules and goals, experience, expectations and saccade plans (Fig. 1). Below, we will describe regions of the non-human primate brain thought to act as priority maps and propose a mechanism for how these areas work together to guide saccades and the allocation of covert attention.

Figure 1.

A hypothetical priority map (right panel) in response to an array of stimuli (left panel) in which the subject is asked to find a circle. Each stimulus is represented by a response. The popout (yellow triangle) is represented by a slightly higher response due to its low level salience and the circle is represented by a much higher response as this represents the goal of the task, i.e. a top down input. Note that top down influences are not limited to task rules and goals; in natural behavior they can be driven by influences such as experience, saccade plans, memories and expectations.

Brain areas involved in priority map processing

A number of cortical and subcortical areas of the brain have been proposed as priority maps. These include the lateral intraparietal area (LIP) of posterior parietal cortex, the frontal eye field (FEF) of prefrontal cortex, the superior colliculus (SC) and several visual cortical areas [5–8, 9*, 10]. Given that our definition of a priority map is one in which the activity is primarily driven by attentional priority and which is involved in the allocation of covert attention and the guidance of eye movements, we think it unlikely that visual areas can act as true priority maps because their responses are primarily driven by stimulus features. However, it is clear that many visual areas include enhanced responses to salience within the feature space that they encode [5,6,10], so it may not be inappropriate to call them saliency maps. Indeed, the term saliency map is still used in several contexts, so it is worth differentiating them here. Some use the term to describe what we have called a priority map, that is a map that integrates both top-down and bottom-up inputs to guide attention and eye movements [8]. However, the term has also been used to illustrate the way visual area V1 [5] and the superficial layers of the superior colliculus [9*,11] preferentially respond to salient stimuli. Recent work has noted that the salience response emerges in the superficial layers of the superior colliculus before it emerges in V1 [11], which may be a result of the evolution of attentional behaviors in animals without a neocortex [12*, but see 13].

LIP, FEF and the intermediate layers of the superior colliculus (which we shall abbreviate as SCi) seem to fit the profile for a priority map. Each has responses that are modulated by low-level salience [14–16] and a variety of top-down factors [17*, 18–23, 24**]. In addition to being driven by the right sort of factors, LIP, FEF and SCi are involved in both the guidance of eye movements and covert attention [17*, 25–30].

The priority map for saccade goal selection

Behaviorally, we tend to think of a priority map as the final common map for guiding attention [31,32]. This is because behavioral studies can only manipulate external factors and then interpret the resulting behavior. As noted above, however, at least three brain areas in the non-human primate have been proposed as priority maps. This has led to questions about the underlying hypothesis [33*], such as: if there are multiple maps, which map is used to allocate attention or guide saccades? Here, we propose a way for these areas to work in concert to act, effectively, as a single priority map.

To elucidate the roles played by each area in more natural behavior, we refer to data from a series of studies using free viewing visual search tasks. In our work, we have used a visual foraging task, in which multiple target stimuli are presented among distractors, but only one target stimulus will give the animal a reward. This leads each animal to visually forage among the stimuli, until it finds the reward-bearing target. By aligning the stimuli so that one stimulus is in the recorded neuron’s receptive field when the animal is fixating another stimulus, data can be collected across multiple eye movements within a single trial and a representation of all aspects of the map can be built up within and across trials. Using this task, we have identified a number of differences between LIP and FEF, which allow us to identify the different roles they play in behavior.

Responses in LIP seem to make up a simple priority map. In the visual foraging task, the majority of neurons respond more to a target than to a distractor and the remaining neurons tend to not have task-related responses [34]. When a saccade is to be made into a neurons’ receptive field, many neurons show a burst of activity leading up to the saccade, suggesting that they get top-down feedback indicating the goal of the upcoming saccade. The task-responsive neurons also tend to show a reduced response to a target, if that target had been looked at earlier in the trial and did not give a reward [34]. This is similar to the inhibition of return incorporated into saliency map models [1], but is not as effective: neuronal responses are only slightly reduced. Most importantly, once the array of stimuli has appeared, responses stay fairly consistent during each fixation and are remapped after each saccade. We use the term remapped to describe the shifting of responses within LIP, analogous to the shifting of the image on the retina after an eye movement, but that occurs before the information could reach LIP from the retina. In the visual foraging task, LIP robustly reflects the identity of the stimulus in neurons’ receptive fields 20 ms after a saccade [35]. This means that, except for a brief period immediately after an eye movement, the representation of the entire visual field is present in LIP at all times during the task. Thus, LIP activity seems to be a simple priority map: most neurons have responses that are driven by the task demands and they respond the same way throughout the trial and across saccades.

FEF activity in this task is subtlety different. The majority of neurons in FEF respond like priority map neurons, i.e. they preferentially respond to targets more than to distractors and have reduced responses to previously fixated targets. However, unlike LIP neurons, the FEF responses get suppressed while animals make purposeful fixations [36]. Approximately 150 ms after the eyes stop moving, the responses of these neurons drop down to below baseline levels. If one thinks of FEF as creating a map to identify saccade goals, then the suppression of the map makes sense: when the animal wants to keep its eyes stable, it removes any activity that could lead to the generation of a saccade. The neuronal responses reactivate about 150 ms before the next saccade, which could allow for the sort of rise-to-threshold decision-making process that has been hypothesized to occur in FEF [8], and it would do so on a saccade by saccade basis. For short fixations, the activity remains elevated throughout the fixation, just as it does in LIP. We hypothesize that while LIP provides a priority map at all times, the modulations in FEF allow the animal to control the timing and flow of saccades. When FEF inhibition is released, it uses the input from LIP to make the decision about where to look. This is consistent with other studies that have examined FEF activity in free viewing behavior of natural stimuli and which have found responses that primarily represent the goal of the next saccade [37–39].

Based on these results, we suggest that during natural visual behavior, in which eye movements are made 2–3 times per second, LIP acts as the default priority map, combining top-down and bottom-up inputs. When things change, whether externally or based on internal goals, the activity in LIP will change [40]. The priority map in FEF, on the other hand, can choose to use the activity from LIP to generate an eye movement or can remain silent to keep the eyes stable. We hypothesize that SCi would reflect the output of FEF and is unlikely to be active during stable fixation. Indeed, we expect that SCi is likely to represent the winner-take-all aspect of the priority map process, with activity that primarily identifies where and when the next saccade will go. Only two studies have examined responses of neurons in SCi during free viewing. In both, activity was almost silent, unless an eye movement was about to go into the neuron’s receptive field [9*, 41].

The priority map for covert spatial attention

Most previous studies have examined activity in LIP, FEF and SCi using tasks in which animals were allowed to make only a single saccade. Unlike the free viewing responses described above, activity in these sorts of tasks is usually elevated throughout the duration of fixation in FEF and SCi [15, 42]. These tasks rely on explicit covert visual attention – if the animals make the wrong eye movement, then they will forfeit their reward – so they must be sure, covertly, that their saccade goal is the correct one before generating the eye movement. Such behavior is relatively uncommon in natural conditions (with the caveat that it can occur in non-human primates with social hierarchy rules that limit eye contact with dominant males). Instead, we tend to use eye movements to probe the visual scene. As such, it is unclear whether the activity in these conditions is different because the system is in a different state, in which covert attention is being explicitly used while the eyes don’t move, or because different neurons become active during this behavior. Whatever the genesis, data from these experiments gives important insight into how the activity in these areas might work as priority maps to guide covert attention.

We suggest that covert attention follows the same process as overt attention, particularly given the automatic allocation of attention at the goal of a saccade [28, 43–45] and the interactions between microsaccades and covert attention [46, 47]. The only exception would be that covert attention is allocated to the peak of a priority map on a moment-by-moment basis. However, understanding this process is hindered because the neural correlates of covert attention [48–50] have recently been dissociated from behaviors that would classically be described as covert attention [51**, 52]. This can lead to different interpretations of what roles areas play, depending on how one defines covert attention. So, for example, pharmacological activation of FEF modifies the attentional modulation of activity in visual areas [53], but the inactivation of SC does not [51**], yet the inactivation of SC has a much greater effect on behavior [30] than the inactivation of FEF [54].

Our hypothesis is that the priority map system is similar for overt and covert attention: LIP activity creates a first level priority map, influenced by both top-down and bottom up factors. FEF, which appears to remain active during periods of explicit covert attention, receives inputs from LIP and feeds back to visual areas to modulate their responses. SC, which likely represents similar activity to FEF and LIP via feedforward networks, seems to be critical in controlling the behavioral use of the activity represented in FEF and LIP. Under this hypothesis, during ongoing visual behavior, when FEF and SC are mostly suppressed, the corollary discharge sent to FEF from SC before an eye movement [55] is likely to drive the pre-saccadic benefit of attention seen at the goal of the saccade [28,43,44].

There are a number of experimental predictions that are born from this hypothesis, but the most obvious is that if FEF and SC are mostly silent during ongoing search, then one might predict that we should not see traditional attentional modulation in visual cortical areas during ongoing visual search. We know that V4 neurons show feature based attention during ongoing search [6,56] and that they have clear modulation when saccades are made to neurons’ receptive fields [57], but whether they show the sort of responses seen in covert attentional tasks [48–50] is not known. It may seem counterintuitive to suggest that these effects are not present in natural viewing behaviors, but we have recently shown that covert attentional modulation is absent while animals perform a change detection task with multiple items [52], yet as soon as they utilize explicit covert attention, the attentional modulation is seen. The presence of feature based attentional modulation during ongoing search could mean that feature based attention is driven by LIP [58], which is active during each fixation, or FEF [18], which is active during short fixations, but it is also possible that it is driven by pre-frontal areas other than FEF [59]. While it is commonly thought that feature based attention is not spatial, it is implemented in a spatial based system, so it is possible that priority maps could play a role in this process.

This review has focused on the process occurring in the non-human primate, given our ability to probe the activity of individual neurons. Functional imaging and various electrophysiological methods have shown somewhat similar activity in putative priority maps in the human [60]. However, it is clear that the human has a more complex network. As such, it is also possible that the control of covert attention is even more complex than in the non-human primate. This possibility is supported by the subtle perceptual differences in the effects of covert attention in the human when driven by different mechanisms [61].