Neural Mechanisms of Saccade Target Selection: Gated Accumulator Model of Visual-Motor Cascade

Abstract

We review a new computational model developed to understand how evidence about stimulus salience in visual search is translated into a saccade command. The model uses the activity of visually responsive neurons in the frontal eye field as evidence for stimulus salience that is accumulated in a network of stochastic accumulators to produce accurate and timely saccades. We discovered that only when the input to the accumulation process is gated could the model account for the variability in search performance and predict the dynamics of movement neuron discharge rates. This union of cognitive modeling and neurophysiology indicates how the visual-motor transformation can occur and provides a concrete mapping between neuron function and specific cognitive processes.

Introduction

Overt visual search requires at least two stages of processing: a relevant target must be discriminated from distracting items and selection of that target must be used to guide a response. Traditionally, these processes have been studied independently. Models of visual selection propose various mechanisms by which irrelevant information is filtered or suppressed and relevant information is retained or enhanced (e.g., Bundesen, 1990; Bundesen et al., 2005; Desimone & Duncan, 1995; Cave, 1999;Itti & Koch, 2001; Wolfe, 2007). Models of response selection explain how alternative potential responses compete for selection and execution (e.g., Ratcliff & Smith, 2004). How selected visual information guides responses and determines their speed is an open question; no current model of visual search can account for distributions of response times (Wolfe et al., 2010). This article will review a new model of visual search that explains how evidence about the salience of objects in the visual field is used to guide a saccade response (Purcell et al., 2010a). We identify the activity of one population of neurons with the representation of object salience and another with the accumulation of that evidence to a threshold. We show that the model can account for the distributions of response times in visual search as well as the temporal dynamics of neurons that control saccade initiation. The success of this model indicates certain constraints on the nature of the interactions between distinct classes of neurons mediating the visual-to-motor transformation, embodies specific linking propositions mapping particular cognitive and neurophysiological processes, and provides a highly constrained and neurally plausible framework to begin bridging models of visual search and decision making.

Neural circuits for visually guided saccades

The last ten years have witnessed a very fruitful body of research on how the brain guides where and when to move the eyes (Glimcher, 2003; Gold & Shadlen, 2007; Schall, 2001, 2003; Smith & Ratcliff, 2004). Eye movements are guided by a distributed network of structures (e.g., Munoz and Schall 2003; Wurtz et al, 2001). Three major structures have been studied most extensively – the frontal eye field (FEF), the superior colliculus (SC), and the lateral intraparietal area (LIP). These structures are densely interconnected. LIP neurons project to FEF and SC (Ferraina et al., 2002; Paré & Wurtz, 1997). FEF neurons project to SC (Segraves & Goldberg, 1987; Sommer & Wurtz, 2000) as well as to LIP (Schall, Morel, King, & Bullier, 1995). Furthermore, FEF, LIP, and SC receive converging projections from numerous visual cortical areas (Morel & Bullier 1990; Schall et al. 1995; Lock et al. 2003). FEF and SC movement neurons project to brainstem nuclei to execute saccadic eye movements (Scudder et al., 2002; Sparks, 2002). FEF and LIP can also influence SC responses via projections to the basal ganglia (Stanton et al., 1988). FEF and SC are also connected with brain regions implicated in cognitive control including medial frontal and dorsolateral prefrontal cortex (e.g., Goldman-Rakic & Porrino, 1985; Schall et al., 1993; Stanton et al., 1995; see also Johnston & Everling 2006). Thus, these areas lie at the junction between perceptual and motor processing and are anatomically situated to influence the guidance and control of movements of the eyes (Munoz & Schall, 2003; Schall, 2003).

FEF, LIP, and SC are comprised of a diversity of neurons. Visual neurons in FEF, LIP, and SC respond briskly when a visual stimulus is flashed in their receptive field; some of these exhibit sustained elevation of firing rate when a stimulus must be remembered, but respond weakly or not at all immediately prior to a saccade (e.g., Bruce & Goldberg, 1985; Gnadt & Andersen, 1988; Paré & Wurtz, 2001). In monkeys performing visual search, nearly all visually responsive neurons in FEF, LIP and SC initially respond to a stimulus regardless of its significance. Over time, the firing rates of most visual neurons evolve to select the target either by maintaining an increased firing rate or producing a second burst if the target is in the neuron’s receptive field and reducing firing rate or failing to produce the second burst if a distractor is in the neuron’s receptive field (Figure 1). This target selection process has been observed in a diversity of visually responsive neurons in FEF (e.g., Schall & Hanes, 1993), SC (e.g., McPeek & Keller, 2002; Shen & Paré 2007), and LIP (Ipata et al., 2006; Ogawa & Komatsu, 2009; Thomas & Paré, 2007). This target selection process in visually-responsive neurons is independent of movement production. Target selection takes place if the animal is instructed to withhold a saccade (Thompson, et al. 1997) or if the animal responds manually (Thompson et al., 2005). When the location of the target unexpectedly changes, visually responsive neurons select the final location of the new target even if monkeys incorrectly direct gaze to the original target location (Murthy et al., 2001; Murthy et al., 2009). During antisaccades, some visual neurons initially select the location of an informative stimulus although the saccade is ultimately executed to a different location (Sato & Schall, 2003). Reversible inactivation leads to deficits in search performance even when no eye movements are required; this has been demonstrated in FEF (Wardak et al., 2006), LIP (Wardak et al., 2002), and SC (Lovejoy & Krauzlis, 2010). Thus, the visually selective response can be dissociated from saccade preparation, although gaze is typically directed to the selected location.

FIGURE 1.

Representative visual and movement neurons recorded in the frontal eye field of a macaque monkey performing visual search for a target that was easy (red) or hard (green) to distinguish from distractors. The activity when the target (solid) or distractor (dashed) fell in the receptive field or movement field is compared. Note the delay in the time when the visual neuron distinguishes the target from the distractor in hard as compared to easy search. Note the corresponding delay in the time when the movement neuron activity begins to accumulate to a threshold.

Visually responsive neurons in FEF, SC, and LIP are hypothesized to represent the salience of an object in their receptive field (Fecteau & Munoz, 2006; Findlay & Gilchrist, 1998; Goldberg & Bisley, 2010; Gottlieb, 2007; Gottlieb & Balan, 2010; Thompson & Bichot, 2004). By salience we refer to both physical conspicuousness, also referred to as bottom-up salience, and behavioral relevance, also referred to as top-down salience or priority (Bisley & Goldberg, 2010; Fecteau & Munoz, 2006). The great majority of neurons in these higher-order structures are not feature selective (e.g., Wurtz & Mohler, 1976); rather, the firing rate scales with the relevance of the object to visually guided behavior. When the target and distractor are easily discriminated, the target is selected earlier and the post-selection difference in discharge rate is larger (Bichot & Schall, 1999; Sato et al., 2001). If no target is presented, then firing rates are greater if the distractors are more similar to the stored target representation (Sato et al., 2003). Selection is also delayed and firing rates are reduced when the number of distracting items in the array is increased (Basso & Wurtz, 1997; Balan & Gottlieb, 2008; Cohen et al., 2009a), but only if the target is difficult to discriminate (Purcell et al., 2010b). When the monkey incorrectly makes a saccade to a distractor location, visually responsive FEF neurons select the location of the distractor, indicating a failure to properly encode the object salience (Heitz et al., 2010; Thompson et al., 2005). Under some circumstances, though, the visual neurons in FEF can locate targets accurately but errant gaze (Murthy et al. 2009) or manual movements (Trageser, et al., 2008) can be produced through failures of response selection. Altogether, these observations suggest that visually responsive FEF, SC and LIP neurons encode the salience of objects in the visual field. However, it is poorly understood how this representation of salience is read out to guide an eye movement response.

A distinct class of movement neurons in FEF and SC responds very little or not at all when a stimulus is flashed anywhere in the visual field, but show a robust increase in firing rate immediately prior to a saccade (Bruce & Goldberg, 1985; Munoz & Wurtz, 1995; Schiller & Koerner, 1971; Schall 1991; Sparks 1978). These neurons can produce saccades when no stimuli are visible (e.g., Bruce & Goldberg 1985) and when the saccade is directed away from the stimulus (e.g., Everling et al. 1999; Everling & Munoz, 2000). During visual search, these neurons respond only if a movement is executed to their movement field (e.g., Schall et al. 1995; McPeek and Keller 2002) (Figure 1). If the task is made more difficult, then response times are longer, but movement neurons still reach the same firing rate immediately prior to a saccade (Woodman et al., 2008); however, this may not be the case for production of antisaccades (Everling & Munoz, 2000; Everling et al., 1999). The time when firing rates reach a fixed level also accounts for random variability in response time (Hanes & Schall, 1996). These observations have led to the hypothesis that FEF and SC movement neurons initiate a saccade when activity reaches a threshold.

A number of studies provide converging support for the fixed threshold hypothesis. When a warning signal predicts the onset of an upcoming saccade target, SC movement neurons increase firing at a faster rate, but reach the same level of pre-saccadic activity (Fecteau & Munoz, 2007). When fixation point offset precedes the onset of a saccade target, the firing rate of FEF and SC movement neurons increases with decreasing saccade response time (Dorris et al., 1997; Everling & Munoz, 2000). With repeated saccades to the same location, saccade latencies decrease as the level of preparatory activity in SC neurons increases (Dorris et al., 2000). On a trial-by-trial basis, movement neurons, but not visually-responsive FEF neurons, can explain the probability the animal makes a saccade by assuming a fixed spike rate threshold (Brown et al., 2008). When a monkey is instructed to withhold a pre-cued response, movement neurons, but not visual neurons, modulate their firing rate before reaching threshold (Hanes et al., 1998; Murthy et al. 2009). During two-alternative forced choice decision tasks, SC neurons reach a fixed firing rate immediately prior to a saccade (Ratcliff et al., 2003, 2007). These observations suggest that SC and FEF movement neurons initiate a saccade when firing rates reach a constant threshold of activity. In other words, variability in the time when these neurons reach a particular firing rate can explain variability in behavior. However, the source of inputs driving movement neurons to threshold is unknown.

In formulating this model, we emphasize the distinction between neurons with nearly exclusive visual responses and neurons with nearly exclusive movement-related activity. It is well know, though, that many neurons in FEF and SC have both visual and movement-related modulation. Such visuomovement neurons project from FEF to SC (Sommer & Wurtz 2000) and from SC to the brainstem reticular formation (Rodgers et al. 2006), so they certainly influence saccade production. However, physiological (Ray et al. 2009) and biophysical (Cohen et al. 2009b) evidence indicates that they are functionally distinct from visual and movement neurons. Nevertheless, when we include visuomovement neuron activity in the population input to the accumulator, the key model predictions are unchanged.

Stochastic accumulator models of perceptual decision making

The observation that FEF and SC movement neurons initiate a saccade when a firing rate threshold is reached corresponds to the form of sequential sampling, stochastic accumulator models (e.g., Carpenter, 1999; Smith & Ratcliff, 2004). This framework explains variability in decision-making behavior by assuming that perceptual information accumulates over time and a choice is made when it reaches a response threshold. In cognitive psychology, stochastic accumulators are the dominant models of perceptual decision making performance. The success of accumulator models can be attributed to their excellent account of full correct and error response time distributions as well as response probabilities – the basic dependent variables of cognitive psychology (Ratcliff and McKoon, 2008). Within the stochastic accumulator framework, different models make different assumptions about the nature of the accumulation mechanism and the sources of within-trial and across-trial variability. For example, models may assume that the accumulation of perceptual evidence is noisy (e.g., Ratcliff & Rouder, 1998) or that the accumulation is ballistic and all behavioral variability is due to trial-by-trial fluctuations in the average rate of accumulation (Brown & Heathcote, 2005, 2008; Carpenter, 1999; Carpenter & Williams, 1995). For example, one recent model is based on the idea that individual sensory units are linked via a non-linearity in the form of a latched binary threshold to motor units (Carpenter et al., 2009). According to this model, individual sensory units assert their binary signals when their individual thresholds are exceeded by accumulation of their sensory signals, and these signals remain asserted until the final motor response is achieved. The random variability in the motor units arises from a source that is independent of the noise in the afferent signals.

Models also can include different kinds of inhibitory interactions among competing responses. For example, some models propose that accumulators representing alternative potential responses are competitive (e.g., Usher & McClelland, 2001), while others assume that model inputs simultaneously inhibit accumulators representing alternative actions (e.g., Ditterich, 2006; Mazurek et al., 2003). However, a serious challenge for this framework is that models that assume very different decision-making architectures can often account for many of the same behavioral phenomena. This is referred to as model mimicry (Van Zandt & Ratcliff, 1995). In some cases, under certain conditions, models that make very different assumptions about underlying mechanisms may be formally identical (Bogacz et al., 2006).

The observation that the pattern of activity of certain neurons resembles an accumulation to threshold is important because it suggests that the accumulation process embodied in these models can be identified with the activity of individual neurons. If the mapping between model process and neural activity is correct, then neurophysiological data can be used as a model selection tool. If two models cannot be distinguished by behavioral data alone, but one model provides a better account of neurophysiological observations, then we propose that the model inconsistent with neurophysiology can be rejected. It has been shown that the dynamics predicted by some accumulator models is compatible with observed patterns of FEF and SC movement neurons, while other model architectures can be ruled out (Boucher et al., 2007; Ditterich, 2006; Ditterich, 2010; Ratcliff et al., 2003; Ratcliff et al., 2007).

We turn now to the question of what drives the accumulation process. In many stochastic accumulator models the drift rate is a free parameter. Some cognitive models include a perceptual processing stage that precedes information accumulation and defines the mechanisms that drive the accumulation (e.g., Nosofsky & Palmeri, 1997; Palmeri, 1997). The neural source of inputs driving movement neurons to threshold is not known. Likewise, how the firing rate of visually-responsive neurons is read out to guide a saccade response is not known. We developed a model that answers both of these questions based on the simple working hypothesis that the activity of tonic visual neurons in FEF is the sole source of the drive on movement neurons during saccadic visual search (Purcell et al., 2010a). To test this hypothesis, the model uses observed discharge rates of visual neurons in FEF to predict variability in response time during search through a network of stochastic accumulators. If we can predict search behavior using FEF visual neuron signals, then we will have support for the hypothesis that variability in behavior derives directly and entirely from variability in the representation of object salience in visually selective neurons and variability in the accumulation of those representations over time. If we can predict both observed behavior and the dynamics of FEF movement neurons also recorded during search, then we will have support for the hypothesis that movement neurons are implementing the accumulation operation defined by our model. We distinguished models by the quality of their fits to distributions of response times (RT) and their predictions of neuronal dynamics that accumulate to a threshold to produce a response. A model in which the flow of information to a leaky integrator is gated between perceptual processing and evidence accumulation provided the best account of both behavioral and neural data, while feed-forward inhibition and lateral inhibition were less important parameters.

Neurally constrained modeling approach

Our model tests the hypothesis that visually responsive neurons drive an accumulation process that initiates a response when a fixed threshold is crossed. An innovation of our approach was to use the actual spike rates recorded from neurons as the input to a network of stochastic accumulators. Another innovation was to explore through nested model testing the contributions of leak, feedforward inhibition and lateral inhibition. Figure 2 illustrates our general model architecture. Two model movement units accumulate incoming perceptual evidence and initiate a saccade to the target or a distractor when evidence reaches a fixed threshold, which was a free parameter of all models. Other parameters determined the inhibitory interactions between units and the flow of information from perceptual evidence to accumulator units. Competitive interactions could either take place between model inputs and accumulator units (feed-forward inhibition) or between the accumulator units themselves (lateral inhibition). Perceptual evidence, however, was not a free parameter, but was defined by the time-evolution of FEF visual neuron discharge rates.

FIGURE 2.

Model architecture. Perceptual evidence supporting a saccade to the target, v_T, and supporting a saccade to a distractor, v_D, were derived directly from observed discharge rates of FEF visually selective neurons (see text for details). These inputs were integrated by accumulator units representing a saccade to the target, m_T, or a saccade to a distractor, m_D. A response was initiated when accumulated support for a particular saccade reached a fixed threshold, ⊖, plus ~15 ms ballistic time for the eye movement to be executed. The parameter k determined the strength of leakage and g determined the level of the gate. The parameter u determined the strength of feed-forward inhibition and was fixed to one for the diffusion model. Beta determined the strength of lateral inhibition and was free to vary for the competitive model.

To determine the input to the accumulator units, visual neuron activity recorded during individual trials of the visual search tasks was divided into two subsets. The first subset consisted of trials when the target fell in the neuron’s receptive field. The second subset consisted of trials when a distractor fell in the neuron’s receptive field. For each simulated trial, we randomly sampled, with replacement, spike trains from the population of trials in which the target fell in a neuron’s receptive field – the input to the accumulator for a decision to saccade to the target location – and spike trains from the population of trials in which a distractor fell in a neuron’s receptive field – the input to the accumulator for a decision to saccade to the distractor location. We generated a normalized activation function (spike density function) from the collection of randomly sampled spike train trials. This visual neuron activation function was the input to each accumulator on a simulated trial. Accumulator units integrate their inputs and produce a response at the time when the accumulation of one of the units reaches its threshold. Thousands of trials were simulated this way to produce a predicted RT distribution. The threshold value was optimized to fit observed RT distributions.

Each simulated input used visual neuron activity that began several hundred milliseconds before the presentation of the visual search array while the animal fixated the center of the screen. In that way, model simulations spanned a time from before search array presentation until a simulated saccade was initiated. An important advantage of beginning our simulations at a time prior to the appearance of the search array is that, unlike most computational models, we could eliminate entirely the need for any parameters that might determine the initial value of each accumulator (the starting point, or baseline), the duration of perceptual processing (pre-decision time, or the time before the accumulation begins), and any parameters that would govern how those values vary across trials and conditions. This also allowed us to explore predicted model dynamics throughout the entire time-course of a trial, which had important implications for selecting between alternative model architectures.

This modeling approach allowed us to address several basic questions: First, how does visual salience influence saccade choice? In other words, what is the nature of the mechanism that governs the flow of information from perceptual inputs to accumulated response preparation? To answer this question, we evaluated models that assumed perfect, leaky, or gated accumulation of evidence. Second, what, if any, inhibitory interactions must exist among proposed model inputs or competition between the accumulator units themselves? To answer this question, we evaluated models that assumed feed-forward or lateral inhibitory competition between model accumulator units. Third, can we identify the accumulation process with FEF movement neurons? To answer this question, we directly compared the predicted dynamics of the model accumulation units with the observed dynamics of movement neurons during search.

How does visual salience influence saccade choice?

We first tested an independent race model that assumed perfect integration of visual neuron inputs until a response threshold is reached. Later, we evaluated other models that make different assumptions about the interactions among accumulating units and their inputs. The overall fit of the perfect integrator model was very poor (Figure 3). Why does this model fail when similar stochastic accumulator models have been successful? Here, the model fails because we assume that visual neuron activity is input to accumulator units continuously over time in a cascaded fashion (McClelland, 1979). There is no mechanism to limit the rate of accumulation prior to the onset of the stimulus array. Visual neurons do not discriminate the target from distractors until relatively late in the trial (Figure 1), which means that units accumulate noise for the much of the simulated trial. As a result, important stimulus-dependent differences in the model inputs have too little impact on the accumulation. Some mechanism is necessary to limit the rate of accumulation until an informative signal is available to guide the decision.

FIGURE 3.

Behavioral fits and neural predictions of the perfect, leaky, and gated accumulator models. Left, observed saccade response time quantiles (open circles) and predicted cumulative response time distribution (solid lines) for the easy (red) and difficult (green) search task. Right, model trajectories for trials which resulted in a fast (black) or slow (grey) saccade response time.

One means of limiting the rate of integration is to introduce leakage. Leakage is implemented as a self-inhibition that scales with the activation of the unit at a given point in time (e.g., Usher & McClelland, 2001). Functionally, this means that there may be a point during the accumulation when the input is equal to the size of the leakage, which causes the accumulation to asymptote. Biologically, some form of leakage is expected due to biophysical properties of individual neurons or properties of the local circuitry that cause firing rates to asymptote. Effectively, noisy inputs combined with leakage leads to accumulators with a baseline firing rate.

We evaluated a leaky accumulator model in which the accumulator units were self-inhibited by their current activation state weighted by a leakage parameter. Compared to the perfect integrator models, all leaky integrator models provided a significantly better account of the data (Figure 3). Here, leakage is advantageous because it asymptotically limits accumulation of perceptual evidence prior to the onset of the search array. Visual neuron activity is approximately constant in the absence of any stimulus, so accumulator activity levels out at a baseline level of firing. Following the presentation of the search array, visual neuron inputs increase, so the accumulators begin to increase their activity until threshold is reached. Thus, leakage provides one way to limit the rate of accumulation in the presence of dynamic neural inputs that flow continuously.

Leakage limits the rate at which evidence is accumulated, but evidence still flows continuously into the accumulator units. As a consequence, any change in the firing rate of visually-selective neurons necessarily propagates into the accumulation process. One potential problem with such a network is that pre-trial fluctuations due to noise may influence the accumulation process in maladaptive ways. For example, at appearance of the visual array, the accumulator representing a distractor may, by chance, begin at a higher firing rate than the target accumulator, and this elevated initial firing rate can ultimately lead to an erroneous saccade. Furthermore, visual selection does not necessarily lead to saccade preparation (Juan et al. 2004; Murthy et al., 2009; Thompson et al., 2005; Zhou & Thompson, 2009), so some mechanism must be capable of dissociating visual inputs from the accumulation. Most importantly, as discussed later, models with leakage and a continuous flow of information could not predict the observed dynamics of movement neurons. Therefore, we proposed a new mechanism that gates the onset of the accumulation and temporally dissociates the activation of visual and movement related neurons.

We evaluated a new gated accumulator model (Purcell et al., 2010a) of perceptual decision making that represents a neurally-plausible implementation of discrete processing stages. This particular architecture assumes the same dynamic input from visual neurons as the perfect integrator models discussed above, but now a gate parameter controls the minimum level of visual neuron input that can be passed on for accumulation by the response units. Mathematically, the gate parameter is a constant inhibition applied to the visual neuron input but rectified to be greater than zero. Functionally, the gate can be considered to be like a threshold that the model inputs must exceed in order for input to be accumulated by the simulated movement units. The key point is that for an optimally tuned gate parameter, the start of the accumulation is delayed until approximately the time when the target location is selected by visual neurons. Gated architectures provided excellent fits to the performance data (Figure 3). Quantitatively, the gated and leaky models fit the behavioral data equally well.

The gating inhibition mechanism is unique to our stochastic accumulator model. When optimized to fit behavior, the gate parameter prevented accumulation until a signal is present in the visual neuron activity. This essentially decomposes RT into two stages: an initial stage in which the representation of object salience is still emerging and no accumulation toward a decision takes place, and a later stage in which the accumulation begins and a decision is ultimately made. When the gate is exceeded, a potential saccade target enters into the competition for a response. In this way, our gated accumulator model provides a neurally plausible implementation of stages of information flow (Sternberg, 2001), which can be interpreted as a controlled visual-to-motor cascade.

Gating inhibition could also provide an alternative explanation for tradeoffs between the speed and accuracy of decisions. Accumulator models traditionally assume that strategic adjustments in performance are due to changes in response threshold (Bogacz et al., 2006; Nakahara et al., 2006; Simen et al., 2006; Ratcliff and McKoon, 2008). While this has been investigated through functional brain imaging (e.g., Domenech & Dreher, 2010; Forstmann et al., 2008; Ivanoff et al. 2008), there is currently no evidence for threshold changes at the level of individual neurons when speed versus accuracy is stressed. In our model, if a task emphasizes accuracy, the gate parameter can be raised, causing the onset of movement neuron activity to be delayed until well after the target is selected and the distractor representations are suppressed in visual neuron activity. The response will be slow and more accurate. If the task emphasizes speed, then the gate parameter can be lowered and accumulators associated with all responses will begin increasing to threshold earlier. The response will be fast, but because target selection is still emerging within visual neurons, it will be more likely that a distractor will erroneously reach threshold first. Therefore, our model predicts that adjustments of speed and accuracy could be evident in the baseline and onset of activity instead of, or in addition to, changes in the threshold. This prediction can be directly tested by recording from FEF neurons during search tasks in which animals are trained to emphasize speed or accuracy.

What is the role of inhibition?

The perfect, leaky, and gated accumulator models discussed above all share a common simplifying assumption that each unit independently accumulates the activity of a visual neuron representing the object in its receptive field. Thus, they can be considered part of the family of independent race or counter models (e.g., Smith & Van Zandt, 2000; Vickers, 1970). Alternative stochastic accumulator models make various assumptions about inhibitory interactions among accumulator units (e.g., Boucher et al., 2007). So in addition to mechanisms that govern the flow of information from perceptual processing to evidence accumulation (leak or gate), we also evaluated models that make different assumptions about the inhibitory interactions between model units. We evaluated perfect, leaky, and gated diffusion models in which evidence for one response was simultaneously counted as evidence against the competing response; mathematically, the accumulator units integrated the difference in salience represented by the visual neurons with the target or a distractor in their receptive field (e.g., Ditterich, 2006a; Mazurek et al., 2003). We also evaluated race, leaky, and gated competitive models in which the activation of each accumulator unit was inhibited by the current activation of the other unit weighted by the connection strength; this corresponds to models that implement winner-take-all dynamics through mutually inhibitory units (e.g., Usher & McClelland, 2001). Regardless of the form of inhibitory interactions, the perfect integrator models failed to account for behavior and both while the leaky and gated models accounted well for the behavior. Adding either form of inhibition, though, did not significantly improve fit to behavior the fits compared to the independent race versions. This is primarily because the fits were already quite good even without any additional model parameters; thus, there was little room for improvement.

We can draw several conclusions from these behavioral fits. Models that assume perfect integration of activity could not account for distributions of saccade response times and therefore can be rejected, while models that assume leaky or gated integration provide an excellent account. Models that assume different forms of inhibitory interactions accounted for saccade response time equally well and could not be differentiated from one another, at least for the dataset used by Purcell et al (2010).

It should be appreciated as rather surprising that any of the models successfully accounted for observed RT distributions in the first place. The use of raw neural inputs dramatically reduces the number of free parameters that would typically be optimized to fit behavior. Yet, by assuming a simple feedforward cascade between visual and movement neurons these models capture the essential characteristics of visual search behavior. Thus, at least under these testing conditions, the variability in the neural representation of object salience by visual neurons is sufficient to account for the variability in search behavior.

Identification of accumulator units with movement neurons

The leaky and gated models make indistinguishable predictions about behavioral data despite although making very different assumptions about the flow of information to the accumulators. We used neural characteristics to resolve this apparent model mimicry. This requires establishing the linking proposition that the stochastic accumulator units correspond to presaccadic movement neurons in FEF (and SC). If this mapping is correct, then the simulated dynamics of the stochastic accumulator units should correspond to the observed dynamics of the movement neurons.

We went beyond qualitative characterization of model and neuron activity by quantifying how specific measures of movement neuron activity change with response time. We then assessed whether particular models could predict those quantitative measures in their own accumulator unit activity. The parameters that optimized fits to the behavior were used to generate predicted activity trajectories by model accumulator units. Those activity trajectories were then measured in exactly the same way that we measured movement neuron activity trajectories. In this way, the dynamics of movement neuron activity is a prediction of the model, not a fit to data. Model architectures that fit behavioral data and predict neural activity should be selected in favor of models that only fit behavior.

The starting point for our comparison is a study by Woodman et al. (2008) that described how FEF movement neuron activity varied with RT in monkeys performing visual search with stimuli supporting more or less efficient search. The study tested four ways in which variability in movement neuron firing rates could lead to variability in RT. Variability in RT could be due to variability in the time when the firing rate left baseline (the onset), variability in the growth rate of firing rate leading up to a saccade, variability in the baseline activity prior to the start of a trial, or variability in the pre-saccadic firing rate (the threshold). Across conditions when search was more or less efficient, they observed that variability in RT was accompanied only by changes in the onset of the accumulation of firing rate. In addition, within a condition, they observed that variability in RT across trials was associated only with changes in the onset of firing rate accumulation. Thus, the question for us is not only whether the models predict the observed variability in RT, but whether the same parameters used to fit the observed RTs can also predict that the onset of accumulation, but not the growth rate, baseline, or threshold will related to variability in RT.

We analyzed model trajectories using the same measurement procedures applied to the FEF movement neurons (Figure 3). Although models with leaky integration fit behavior well, they systematically fail to predict the time course of movement neuron activity. Specifically, those models predict strong negative correlations between baseline activity and RT as well as a level of accumulation of salience evidence when the distractor is in the movement field that were not observed. These failures are largely a result of the continuous flow of evidence into the accumulators.

In contrast, the gated accumulator architecture predicted very well the neural dynamics. The gate prevented integration of evidence until the visual evidence was of sufficiently quality to merit accumulation. Indeed, for nearly all gated models, the value of the gate parameter that optimized behavioral fits was sufficiently high that the start of the accumulation was delayed until visual neuron inputs elevated in response to the stimulus. Thus, although the gated and leaky models produce indistinguishable accounts of the behavioral data, the gated models provided a superior account of the observed pattern of movement neuron dynamics.

Surprisingly, the dynamics of the simulated accumulation process did not vary much with the different forms of inhibition, whether none at all (race), feed-forward (diffusion), or lateral (competition; Figure 4). None of the leaky models (without gate) produced a pattern of accumulation corresponding to the observed movement neuron dynamics. However, all of the models with gating inhibition did predict the movement neuron dynamics. From this we can conclude that gating inhibition seems to be a critical mechanism, with other inhibitory interactions less important. The models that included inhibitory interactions simply failed to fit the observed data significantly better than the independent race models. However, it is possible that model testing with behavioral and neurophysiological data obtained under a wider range of conditions, such as variations of search array set size, would reveal an important role for inhibition.

FIGURE 4.

Behavioral fits and neural predictions of the gated race, gated diffusion, and gated competitive accumulator models. Conventions as in Figure 3.

We found that a gated accumulator model with input from visually selective neurons could account for both observed search behavior and observed FEF movement neuron dynamics. Stochastic accumulator models assume two basic processes: (a) a stimulus must be encoded with respect to the current task to represent perceptual evidence, and (b) some mechanism must accumulate that evidence to enact a decision. In order to guide a potential response, sensory information must have already been encoded and weighted according to its relevance to the task; that is, the salience of a particular object or location. This is exactly what FEF visually responsive neurons encode (e.g., Bichot & Schall, 1999). Thus, we conclude that FEF visually responsive neurons can be identified with the perceptual evidence driving the accumulation to threshold. According to the accumulator model framework, crossing a response threshold is a critical event producing a choice.signaling the end of a decision. Prior to reaching threshold, an action may still be canceled and an alternative action selected, but crossing the threshold reflects an irreversible commitment to action. This is exactly what FEF and SC movement neurons do (Hanes & Schall, 1996; Murthy et al., 2009; Paré & Hanes, 2003). Thus, we conclude that FEF movement neurons can be identified with the accumulation of evidence to a response threshold. Here, we have reviewed a computational model that synthesized empirical observations into an accumulator model framework that accounts remarkably well for both behavioral and neurophysiological search data.

Another line of research has identified the quantity being accumulated as the activity of feature-selective neurons in extrastriate visual areas (Ditterich et al., 2003) and the evidence accumulation process with the growth of activity in tonic visually responsive LIP neurons (reviewed by Gold & Shadlen, 2007). These linking propositions are supported by the observation that the growth of activity in LIP neurons is stimulus-dependent (Churchland et al., 2008; Roitman & Shadlen, 2002), the effects of MT and LIP microstimulation on performance (Ditterich et al., 2003; Hanks et al., 2006; Salzman et al., 1990; Salzman et al., 1992), and the effects of motion pulse stimuli on behavior and LIP activity (Huk & Shadlen, 2005). However, this framework cannot explain LIP or FEF visual neuron responses during search. The observation that the initial visual response often exceeds the level of presaccadic activity is difficult to reconcile with the prediction that a response is initiated when a fixed threshold is reached (Gottlieb et al., 2005; Ipata et al., 2006; Thomas & Paré, 2007). The presaccadic firing rate of LIP neurons is also shown to depend strongly on whether or not a stimulus is present in the receptive field of the cell (Paré & Wurtz, 2001; Wurtz et al., 2001). When an animal is instructed to withhold a saccade, visually-responsive LIP neurons show little, if any, modulation prior to the saccade (Brunamonti et al., 2008). Thus, while it is possible that a portion of the LIP response is determined by integrating weighted converging inputs from extrastraite areas, this mechanism alone cannot explain search data and these cells do not seem to initiate a response when a fixed threshold is reached. Some additionally processing is necessary, and FEF and SC movement neurons are the most likely candidates. Thus, we propose a different set of linking propositions is necessary to explain the full duration of the decision process.

Neurophysiological implementation

Our model predicts that visually responsive FEF, LIP, and SC neurons provide the source of drive to FEF and SC movement neurons. Most directly, this could be interpreted as local topographic projections from FEF and SC visual neurons to movement neurons, but empirical evidence for intrinsic connections between functionally defined neuronal populations is difficult to obtain. Neurophysiological studies provide some support for this interpretation. In SC, visual and motor receptive fields are closely aligned (e.g., Marino, et al. 2008; Schiller and Koerner, 1972; Wurtz & Goldberg, 1972). Microstimulation of FEF that evoke saccades of a given vector also activate SC neurons that encode the same vector (Schlag-Rey et al., 1992), and suppress neurons in SC and FEF neurons that encode different vectors (Schlag et al., 1998). Reversible inactivation of SC neurons eliminates saccades evoked by microstimulation of FEF corresponding to the same saccade vector (Hanes & Wurtz, 2001). Anatomically, FEF movement neurons are thought to correspond to layer V pyramidal neurons that project to SC and brainstem saccade generating nuclei, while visually responsive neurons are found in both supragranular and infragranular layers (Segraves, 1992; Segraves & Goldberg, 1987; Sommer & Wurtz, 2000; Thompson et al., 1996). Across cortex, layer V pyramidal neurons have been shown to have large ascending and horizontal bundles of dendrites (Connors & Amitai, 1995), which would support synaptic integration across converging visual inputs as well as competitive interactions. In rodent SC, slice work has revealed monosynaptic connections between visually responsive superficial SC neurons and intermediate layer SC neurons that project to saccade generating brainstem nuclei (Isa & Hall, 2009). These observations suggest that anatomical connectivity is appropriate to carry out the hypothesized connections. Behavioral data demonstrate that these connections must be flexible (e.g., Hallett & Lightstone, 1976; Mays & Sparks, 1980). Nevertheless, when saccades are produced directly to the location of a visual object, the mapping of visual activity to a saccade command of the same vector is entirely adaptive, and other models of visually guided saccades make use of this connection (e.g., Hamker, 2005; Heinzle et al., 2007).

Our model predicts that some gating mechanism intervenes between visual and movement neurons. One potential implementation of the gating inhibition is via inhibitory interneurons within FEF. Layer V FEF movement neurons are colocalized with parvalbumin inhibitory interneurons (Pouget et al., 2009). Gating could be implemented via intermediary interneurons or by axoaxonal shunting inhibition applied to visual neuron efferents. A second potential implementation of the gating inhibition could include FEF fixation neurons which fire tonically in the absence of movement, but cease immediately prior to a saccade. This pattern of activity could reflect the release of tonic inhibition when the gate criterion is exceeded. A third potential neurophysiological implementation of the gate is through basal ganglia circuitry. FEF neurons project to the caudate nucleus, which sends inhibitory projections to the substantia nigra pars reticulate. The substantia nigra, in turn, tonically inhibits SC (Hikosaka & Wurtz, 1983) and FEF via the mediodorsal thalamus (Goldman-Rakic & Porrino, 1985) in the absence of saccades. A number of models have proposed that the substantia nigra gates the preparation of motor responses (e.g., Brown et al., 2004; Frank, 2006). During search, sugstantia nigra neurons cease firing prior to a saccade when a search target is in their preferred location, which may also reflect a release of tonic gating inhibition (Basso & Wurtz, 2002). A recent spiking model has proposed that these projections serve a gating function that detects a threshold crossing in cortical neurons (Lo & Wang, 2006). In that model, the strength of synaptic weights between cortical neurons and caudate neurons determines a firing rate threshold that must be exceeded to remove tonic inhibition of SC burst neurons that move the eyes. A key difference in our model is that, the crossing of the gate value does not trigger an all or none burst; rather, it initiates the start of evidence accumulation in downstream neurons. Thus, the form of input after the gate is crossed plays a critical role in determining the dynamics in a subsequent accumulation stage.

Our results show that the simplest gated model, the independent gated race, is sufficient to account for the observed pattern of neural and behavioral data. However, there are neurophysiological data suggesting that some form of inhibitory interaction may be operating. There appears to be a local center-surround inhibition between ~20% of neurons in FEF and SC (Schall et al., 1995; Schall et al., 2004). When two FEF neurons in opposite hemifields both select the target, then spike timing correlations are negative when the target is in one neurons RF, but not the other (Cohen et al., 2010). Microstimulation of FEF in one hemisphere can reduce firing rates in the opposite hemisphere (Schlag et al., 1998). In SC, when a visual distractor was flashed in the opposite hemifield while animals prepared a saccade to the location encoded by an SC neuron, the response of that neuron was inhibited by the onset of the distractor stimulus (Dorris et al., 2007). There are also theoretical arguments for the importance of competitive interactions. Competition is necessary for accumulator models to optimize tradeoffs between speed and accuracy in order to maximize rate of reward (Bogacz et al., 2006). When compared head-to-head, models assuming competitive interactions are shown to provide a somewhat better account of behavioral data than those that do not (Ratcliff & Smith, 2004). It is likely that we failed to find evidence of inhibitory interactions because our behavioral data were relatively limited. It will be important to evaluate the model in more difficult search tasks in which varying the number of potential saccade targets would be expected to induce differences in the strength of competitive interactions.

Visual search as a perceptual decision

Many visual search models include a salience map that represents explicitly the behavioral relevance of objects in the visual field (e.g., Bundesen et al., 2005; Cave 1999; Itti & Koch, 2001; Treisman & Gormican, 1988; Wolfe, 2007). FEF, SC, and LIP visual neurons have been associated with a neural instantiation of the hypothetical saliency map (Findlay & Gilchrist, 1998; Goldberg et al., 2006; Gottlieb, 2007; Fecteau & Munoz, 2006; Thompson & Bichot, 2004). For saccade decisions, the task for the brain is to correctly discriminate the distribution of activity representing the search target location from the distributions representing distractor locations (e.g., Palmer et al., 2000; Navalpaakam & Itti, 2007). This presents two challenges. First, firing rates are noisy. Even if multiple signals are pooled prior to read out, the resolution may be limited by correlated noise (Cohen et al., 2010). Repeated sampling and integration over time as predicted by accumulator models is an ideal mechanism to deal with noisy samples. Second, as we have shown, the neurophysiological representation of visual salience evolves gradually over time. Thus, it is not enough to specify how the map is read-out, but also how it is read out at an appropriate time. Our model assumes that the saliency map is the representation of perceptual evidence supporting a saccade to that location in visual space. The accumulation begins when evidence exceeds the gate. In other words, the function of the elevated firing rates of visually selective neurons is to essentially enter an object location into the competition for a saccade response.

The probabilistic population coding hypothesis proposes that neurons in sensorimotor areas like FEF and LIP encode complete probability distributions for certain motor or sensory parameters (Beck et al., 2008; Kim & Basso, 2010; Ma et al., 2006;). During search, this is the probability that an object matching the stored features of the target stimulus is within the receptive field of the neuron given the available sensory information (e.g., Najemnik & Geisler, 2005). Studies using explicit cuing have shown that the firing rates of some LIP neurons are modulated by the probability that a saccade to their receptive field will result in reward (Kiani & Shadlen, 2009; Platt & Glimcheer, 1999; Yang & Shadlen, 2006), but these results can be explained through differential attentional allocation (Maunsell, 2004). Less is known about whether the firing rates of FEF visual neurons can be directly identified with a representation of saccade target probability. Decreases in mean firing rate with increasing distractors could be interpreted as a decrease in the probability of a saccade target with increasing potential responses (Cohen et al., 2009a), but these results can also be explained by flanking suppression (Schall et al., 1995; Schall et al., 2004). Regardless, our framework does not depend on a probabilistic interpretation of FEF visual neurons. However, it is compatible with a probabilistic interpretation because the location with the highest probability would be most likely to win the race to threshold. Note that this interpretation does not apply FEF movement neurons that reach a fixed level of firing prior to a saccade.

Conclusions and future directions

We reviewed a new gated accumulator model (Purcell et al., 2010a), that seeks to explain the guidance of saccades during visual search by translating the activity of neurons encoding the salience of objects through a network of stochastic units that accumulate evidence. There are several key results from our work. First, and perhaps most importantly, we showed that the framework actually works. Integration over time of dynamically changing firing rates of visually selective neurons is sufficient to explain search behavior. This suggests a simple and neurally plausible algorithm whereby visual target selection can guide saccade commands. Importantly, no parameters varied within or across trials. This means that variability in behavior can be directly derived from variability in the firing rates of visually selective neurons. An obvious limitation of the gated accumulator model is that the accumulation of activation of the movement units is derived entirely from the visual units, but it is well known that saccadic eye movements can be produced in the absence of stimuli (memory-guided saccade in darkness) or even in opposition to the location of stimuli (antisaccades). Therefore, movement units must derive activation from an extraretinal source. The source of this activation is uncertain but is likely to include medial frontal areas with the basal ganglia. This other source of activation may also be another source of variation of response time, but at least under the testing conditions evaluated for this model it does not contribute variation beyond what can be explained by the visual salience inputs.

Second, we showed that the same models that account for observed behavior also make precise quantitative predictions about the form of the accumulation observed in FEF movement neurons during search. This suggests that we can directly identify the accumulation process predicted by accumulator models with the increase in firing rate of movement neurons to threshold. These results raise questions about the identification of the stochastic accumulator model framework with neurons in other areas. Accumulation to threshold is a very general operation that is likely to operate in different brain areas to perform different functions. Decades of work in cognitive psychology have identified a role for stochastic accumulation in perceptual decision making. Here, we reviewed the reasons why we believe that the linking propositions associated with the gated accumulator model framework most closely fulfill that role.

The work reviewed here establishes a foundation to begin bridging accumulator models of perceptual decisions with models of search and eye movements in a manner compatible with neurophysiological observations (Wolfe et al. 2010). However, many open questions remain. For example, how does the model explain differences in behavior when the number of items in the array is varied? To address this question the framework will need to be extended to a multiple-alternative decision process. A second question of significant interest is how the saliency map is formed. Incoming sensory signals must somehow be enhanced or suppressed in accordance with their relevance to the task (i.e., similarity to a stored target representation). Detailed theories have been developed to explain the formation of the saliency map (e.g., Bundesen et al., 2005; Cave, 1999; Itti & Koch, 2001; Treisman & Gormican, 1988; Wolfe, 2007), but it is not known if these theories can be extended to explain the evolution of the salience representation over time in a manner that is consistent with neurophysiological observations. A third question concerns the hypothesized role of the saliency map in guiding perceptual enhancements (Moore & Armstrong, 2003). Sophisticated models have been developed that propose a role of FEF movement neurons in guiding perceptual enhancements in visual cortex (Hamker, 2004; Hamker, 2005). However, these models are inconsistent with recent anatomical data that show FEF feedback projections originate in supragranular layers, not layer V pyramidal neurons (Pouget et al., 2009). Our methodology will allow for a rigorous test of the mechanisms that are sufficient or necessary to trigger perceptual enhancements without saccadic eye movements, but this will require tasks which explicitly test for perceptual improvements.