Selective attention without a neocortex

Abstract

Selective attention refers to the ability to restrict neural processing and behavioral responses to a relevant subset of available stimuli, while simultaneously excluding other valid stimuli from consideration. In primates and other mammals, descriptions of this ability typically emphasize the neural processing that takes place in the cerebral neocortex. However, non-mammals such as birds, reptiles, amphibians and fish, which completely lack a neocortex, also have the ability to selectively attend. In this article, we survey the behavioral evidence for selective attention in non-mammals, and review the midbrain and forebrain structures that are responsible. The ancestral forms of selective attention are presumably selective orienting behaviors, such as prey-catching and predator avoidance. These behaviors depend critically on a set of subcortical structures, including the optic tectum, thalamus and striatum, that are highly conserved across vertebrate evolution. In contrast, the contributions of different pallial regions in the forebrain to selective attention have been subject to more substantial changes and reorganization. This evolutionary perspective makes plain that selective attention is not a function achieved de novo with the emergence of the neocortex, but instead is implemented by circuits accrued and modified over hundreds of millions of years, beginning well before the forebrain contained a neocortex. Determining how older subcortical circuits interact with the more recently evolved components in the neocortex will likely be crucial for understanding the complex properties of selective attention in primates and other mammals, and for identifying the etiology of attention disorders.

1. Introduction

Selective attention is the ability to limit neural processing to a subset of signals, while actively excluding other eligible signals from consideration. In humans, this ability takes an especially complex and sophisticated form. We are able to selectively attend across an enormously diverse range of elements and features, including all of the physical objects in our surroundings and their individual parts, but also items from memory and thoughts about things that might not even exist.

Most studies of selective attention in human and non-human primates emphasize the importance of areas in the cerebral neocortex, highlighting attention-related changes in signal processing that take place in sensory cortical areas and the roles of frontal and parietal neocortex in regulating these changes (Kastner & Ungerleider, 2000; Petersen & Posner, 2012; Reynolds & Chelazzi, 2004). Subcortical areas are also implicated in attention (Krauzlis, Lovejoy, & Zenon, 2013; Peck & Salzman, 2014; Saalmann & Kastner, 2011), but are typically presented as playing a secondary role. However, as we will describe in this review, signs of selective attention can be found in vertebrate species that lack a neocortex, including birds, amphibians, reptiles, and fish; all of these animals respond selectively for specific stimuli, and there is evidence that other eligible items are considered and actively excluded. If the neocortex contains the primary mechanisms for implementing selective attention, how can selective attention be present in these species with brain plans that do not feature a neocortex?

The possible answer we explore is influenced by an evolutionary perspective on brain function (Figure 1A). We posit that selective attention in primates does not arise out of whole cloth, but is stitched together from several identifiable brain circuits that have evolved over hundreds of millions of years. The most extensively studied circuit components are contained within the cerebral neocortex (Figure 1B), which first emerged about 200 million years ago as one of the defining features of mammals, along with hair, milk, and those unusual middle-ear bones. Other circuit components are located in subcortical structures that appeared earlier in the evolution of vertebrates, such as the thalamus, superior colliculus, and striatum (Karnath, Himmelbach, & Rorden, 2002; Krauzlis et al., 2013; Murakami et al., 2014; Saalmann & Kastner, 2011). It is an open question whether the neural organization of selective attention as identified in primates holds true across all mammalian species; recent work in rodents and other non-primates is supportive but the picture is still developing, and there are likely to be interesting differences (Carandini & Churchland, 2013). In this review, we will lay aside the issue of selective attention in mammals, and instead focus on non-mammals – vertebrates that completely lack a neocortex.

Figure 1. Evolution and comparison of the brain plans in mammals and non-mammals.

A. Simplified cladogram illustrating the divergence of vertebrate lines during the course of evolution. Mammals diverged about 200 million years ago (mya) from a line of animals (therapsids) that are now extinct, while birds are believed to have diverged more recently (150 mya) from a line that gave rise to modern reptiles (E. D. Jarvis et al., 2005). Vertebrates highlighted in bold are discussed in detail in the article. B. Diagram illustrating a lateral view of a simplified and generic mammalian brain, based loosely on the monkey brain. Regions implicated in selective attention are highlighted in color. C. Lateral view of a simplified and generic non-mammalian brain, based loosely on the bird brain. Brain regions implicated in selective attention are again highlighted, using colors matching those in (B) to indicate homologies.

An introductory comparison between mammalian and non-mammalian brains helps to illustrate some of the questions brought into focus by this approach. The homologies for the subcortical brain regions implicated in attention are fairly clear, including the optic tectum, thalamus, and striatum (Figure 1C). The primate versions of these structures are often viewed as carrying out functions on behalf of the neocortex; in animals that do not have a neocortex, do they still play a subsidiary role but under different management? For other forebrain regions, the homologies are less clear. The territories that initially make up the embryonic pallium are similar across vertebrates, but then take very different developmental paths in mammals and non-mammals (Dugas-Ford & Ragsdale, 2015; Jarvis et al., 2005; Karten, 2015; Montiel & Aboitiz, 2015). In mammals, the dorsal pallium contains progenitor cells that contribute to the 6-layered, columnar neocortex, but in non-mammals these assume a more segregated and clustered organization, containing many of the same cell types. The ventral and lateral pallium give rise to the distinctive dorsal ventricular ridge (DVR) in birds and reptiles, but the fate of these cells in mammals is unsettled, perhaps including parts of the amygdala and claustrum, or specific layers of the neocortex. How important are these different pallial subdivisions for selective attention? By examining the behavior and brains of several non-mammalian species – birds, reptiles, amphibians and fish – we can identify how the properties of selective attention are generated by circuits distributed across these different subcortical and pallial regions, and hopefully gain new insight into how selective attention has evolved into the particular variety that we enjoy as primates.

2.1 Birds

There are numerous examples of bird behavior that appear to involve selective attention. In a classic study in pigeons (Reynolds, 1961), birds were reinforced for pecking at keys illuminated by particular combinations of shapes and colors (e.g., white triangle on red background). When tested later with keys showing just shape or color, one bird might peck mostly for triangles while another pecked mostly for red, suggesting that individual birds had attended to different features during the training. However, the results of a later study using the same method indicated that other factors were also involved – pigeons appeared to attend to color during initial training, but when trained again with a new set of stimuli, their learning was faster for the previously paired shape (Wilkie & Masson, 1976). This suggests that birds probably learned about both color and shape during training, rather than attending to just one visual feature.

The pecking behavior of pigeons is influenced by spatial cues and temporal sequences. When a cue is presented at the same location (valid) or the opposite location (invalid) as a subsequent color-defined target key, the reaction time of pigeons to peck the target is shorter after valid than invalid cues (Shimp & Friedrich, 1993), similar to the classic attention effects described by Posner (Posner, 1980). When keys are illuminated in sequence, the reaction times of pigeon pecks are shorter when the sequence follows a predictable pattern, compared to random, showing that pigeons can learn to anticipate the sequence and timing of recurring stimulus events (Froehlich, Herbranson, Loper, Wood, & Shimp, 2004).

Orienting movements of the head and eyes of birds have also been used to draw inferences about selective attention. When perched, barn owls orient to auditory events by turning their heads in the direction of the sound source. If the auditory stimulus is preceded by a visual cue, the latencies of head turns are shorter when the visual cue correctly indicates the direction of the upcoming auditory stimulus, compared to invalid visual cues that indicate the wrong direction (Johnen, Wagner, & Gaese, 2001). In freely moving owls, the use of head-mounted wireless cameras reveals that owls tend to look in the direction of salient visual features, but usually not at the most salient features, which could be interpreted as an interaction between goal-driven and stimulus-driven mechanisms of control (Ohayon, Harmening, Wagner, & Rivlin, 2008). The value of a visual object’s meaning, rather than its low-level image salience, is also illustrated by telemetric eye-tracking in peacocks, which shows that peacocks, quite reasonably, spend more time looking at a taxidermy fox than expected by chance (Yorzinski & Platt, 2013).

The clearest demonstration of selective attention in birds comes from a recent study in chickens (Sridharan, Ramamurthy, Schwarz, & Knudsen, 2014). Using a touch-sensitive video screen, chickens pecked their way through trials structured very similarly to those used in studies of selective attention in primates (Figure 2A). After pecking a central cross to start a trial, chickens were provided with either a spatial cue or no cue, before target stimuli were presented. One key feature of the experimental design was that the spatial cue varied along an axis orthogonal to the behavioral choice: the cue indicated the horizontal location of the upcoming target (i.e., left or right), but the target itself was offset vertically (i.e., up or down). Thus, the cue pointed out the location that should be attended, but carried no information about the correct response. A second key feature was that the target was accompanied by a distracting stimulus on the other side. Thus, when the chicken completed each trial by pecking either the upper or lower response box to report the location of the target, they needed to actively exclude information about the vertical position of the distracter.

Figure 2. Behavioral evidence of selective attention in chickens.

A. Schematic of the behavioral task. Chickens performed a target localization task that required them to report the vertical location of a target stimulus while ignoring a task-irrelevant distracter. On trials with no cue, the chicken could not know which of the two stimuli was the target until the response boxes were presented at the end of the trial. On trials with a cue, the chicken had prior information about which stimulus was behaviorally relevant and could ignore the other one. B. Psychometric functions showing performance (percent correct) as a function of relative target strength (target to distracter contrast ratio). Performance on trials with a cue (red) was better than performance on trials with no cue (gray). C. Response times with and without cues, plotted as a function of relative target strength. Reaction times on trials with a cue (red) were faster than on trials with no cue (gray). Adapted with permission from (Sridharan et al., 2014).

The performance of chickens in this task clearly illustrates the behavioral signatures of selective attention. On trials with no cue, chickens usually correctly reported the vertical position of the target, but could be misled into reporting the position of the distracter if it was physically more salient (i.e., higher contrast) than the target, showing that there was a competition between the two possible sources of information. On trials with a cue, the disruptive effects of the distracter were substantially reduced, and the chickens correctly localized the targets more often (Figure 2B) and with shorter reaction times (Figure 2C). Because of the experimental design, it was possible to demonstrate that performance improvements with spatial cues were increases in perceptual sensitivity, d’ in signal detection terms (Macmillan & Creelman, 2005), rather than changes in response bias (i.e., decision criterion).

What are the brain circuits that implement these changes in performance? One of the central players is the optic tectum, a multi-layered structure that sits on the roof of the midbrain (Figure 3). The superficial layers of the optic tectum receive direct inputs from retinal ganglion cells and are strictly visual, whereas the intermediate and deeper layers have wide-ranging connections and process a variety of signals, including visual inputs, but also combined with other sensory modalities and activity related to orienting behaviors. Benchmark studies in the pigeon demonstrated that lesions of the optic tectum produce large deficits in the ability to discriminate and localize visual targets (Hodos & Karten, 1974; Jarvis, 1974). As shown more recently in the barn owl, the presence of multiple possible targets activates a mechanism for global inhibition in the intermediate layers of the optic tectum (Mysore, Asadollahi, & Knudsen, 2010). This inhibition results in a rapid and flexible identification of the strongest stimulus, with a subset of tectal neurons exhibiting “switch-like” behavior that denotes the outcome of this winner-take-all mechanism (Mysore & Knudsen, 2011; Mysore, Asadollahi, & Knudsen, 2011).

Figure 3. Schematic diagram of the bird brain.

The lateral view illustrates the prominent position of the optic tectum (OT) in the midbrain, and outlines the locations of the visual Wulst and entopallium in the forebrain. The midbrain slice shows the spatial relationship between the optic tectum, and the adjacent nucleus isthmi pars parvocellularis (Ipc) and nucleus isthmi pars magnocellularis (Imc). The sagittal section provides a cut-away view that makes other attention-related structures visible: the isthmo-optic nucleus (ION) in the midbrain, the nucleus rotundus (Rt) and lateral geniculate nucleus pars dorsalis (GLd) in the thalamus, and the striatum and arcopallial gaze field (AGF) in the forebrain.

This competition between stimuli in the optic tectum is accomplished in tandem with other structures in the brainstem, especially the isthmic nuclei, lesions of which produce deficits in visual detection and discrimination similar to those seen after tectal lesions (Hodos & Karten, 1974; Jarvis, 1974). The nucleus isthmi pars magnocellularis (Imc) receives topographically organized inputs from the optic tectum and returns an inhibitory (i.e., GABAergic) projection back to the tectum (Wang, Major, & Karten, 2004); the homologous structure in mammals is the periparabigeminal lateral tegmental nucleus. The feedback from Imc has a distinct “anti-topographic” organization – unlike typical surround inhibition, it does not fall off with distance but covers the entire visual field except the tectal location that provided the input (Lai, Brandt, Luksch, & Wessel, 2011; Mysore et al., 2010). Imc neurons also provide reciprocal inhibition to other neurons within the Imc, allowing the selection criterion achieved with the tectum to shift flexibly depending on the strength and number of competitors (Mysore & Knudsen, 2012; Sharpee, 2012).

The isthmi nuclei also contribute to selection through other circuit mechanisms. The Imc sends another inhibitory projection to its sister nucleus, the nucleus isthmi pars parvocellularis (Ipc); in mammals, the homologous structure is the parabigeminal nucleus. This inhibitory input contributes to the “switch-like” properties of some Ipc neurons, similar to that found in the tectum (Asadollahi, Mysore, & Knudsen, 2011). The Ipc receives a topographic projection from the tectum, and the Ipc, in turn, provides topographically organized cholinergic feedback to the tectum (Wang, Luksch, Brecha, & Karten, 2006), which has another distinctive effect on stimulus selection. The cholinergic terminals from Ipc modulate the efficacy of transmission of the retinal inputs onto tectal ganglion cells (Marín et al., 2005), a class of tectal neurons that respond to visual motion across large regions of the visual field and project to the thalamic nucleus rotundus, homologous to the mammalian pulvinar. When neuronal activity in the Ipc is locally blocked, this eliminates the visually evoked cholinergic feedback to the tectum, and neurons in the rotundus getting input from the tectum no longer respond to motion in that subregion of the visual field (Marín et al., 2007).

As in mammals, birds have two distinct ascending visual pathways to the forebrain. The tectofugal pathway conveys visual signals from the tectum through the nucleus rotundus (Rt) of the thalamus, mentioned above, and terminates in the entopallium of the dorsal ventricular ridge (Engelage & Bischof, 1993); the mammalian analogue is the extrageniculate system. The thalamofugal pathway conveys signals from the retina through the lateral geniculate nucleus pars dorsalis (GLd) of the thalamus, and terminates in the visual Wulst (Güntürkün, Miceli, & Watanabe, 1993); the mammalian analogue is the geniculostriate system. Lesions of the tectofugal pathway have profound effects on visual discrimination, whereas lesions of the thalamofugal pathway have little effect except when combined with tectofugal damage (Hodos & Bonbright, 1974; Hodos, Macko, & Bessette, 1984; Macko & Hodos, 1984); in contrast, in primates lesions of the geniculostriate system cause blindness (Glickstein, 1988; Horton & Hoyt, 1991; Mohler & Wurtz, 1977). The organization of the tectofugal pathway suggests that different visual features are processed in parallel channels, and that these channels remain largely segregated into higher-order forebrain areas and in the striatum of the basal ganglia (Shimizu, Patton, & Husband, 2010). The pathway into the striatum is important for linking visual selection to the behavioral context – lesions of the avian striatum cause deficits in learning visual discriminations, especially when the rules of the task change (Watanabe, 2001).

The forebrain of birds is also the source of feedback signals that may regulate visual selection. The optic tectum receives inputs from the arcopallial gaze field (AGF) in the forebrain, known for its role in controlling orienting movements in barn owls (Knudsen, Cohen, & Masino, 1995). Electrical microstimulation of the AGF that is below the threshold to elicit gaze shifts nonetheless has pronounced effects on signal processing in the tectum, increasing responsiveness and sharpening the tuning of tectal neurons that represent the corresponding locations in auditory space (Winkowski & Knudsen, 2007; 2008), analogous to the attention-related effects found for the frontal eye field cortical area in primates (Squire, Noudoost, Schafer, & Moore, 2013).

Birds also possess a mechanism for selective attention that has no evident parallel in mammals – feedback control of the retina. As originally detected by Cajal, the bird retina receives a substantial input from the brain, primarily from the isthmo-optic nucleus (ION), another midbrain nucleus that gets most of its inputs from the optic tectum (Cowan, 1970). The ION input to the retina locally facilitates the responses of retinal ganglion cells without changing their tuning properties (Miles, 1970; Uchiyama & Barlow, 1994), and conversely, lesions of the ION reduce the accuracy of pecking for food without disrupting visual acuity (Hahmann & Güntürkün, 1992; Knipling, 1978). Deficits in visual selection after ION lesions are not observed with single targets, but only when there are competing alternatives, showing that disruption of retinal feedback in birds causes visual extinction (Uchiyama, Ohno, & Kodama, 2012), akin to the deficits seen in human neglect patients (Karnath et al., 2002; Karnath & Rorden, 2012).

2.2 Reptiles

Compared to pigeons and chickens, reptiles have been a less popular choice for studies of selective attention, although the natural behavior of lizards provides some relevant examples. One of the selection problems faced by the Anolis lizard, a type of iguana, is distinguishing between the movement of prey items, which represent a potential meal, and the movement of wind-blown plants, which are just part of the landscape. Lizards solve this problem with a hard-wired preference for some movement patterns over others – their visual grasp reflex is better triggered by jerky movements than by smooth periodic motions, and short-term habituation filters out the repetitive movement of irrelevant background elements, although unfortunately for lizards this property also provides a loophole that can allow snakes to escape detection (Fleishman, 1986; Pallus, Fleishman, & Castonguay, 2010). This preference for movements with high accelerations and peak velocities is also exploited by lizards wishing to attract attention and assert their social dominance when communicating with other lizards by bobbing their heads up and down (Fleishman, 1988).

Most invasive studies of selection behavior in reptiles have used turtles, and consistent with the data from birds, circuits involving the optic tectum play a central role. Lesions of the optic tectum cause deficits in the ability of Amazonian river turtles to locate food objects (Bass, 1977). Normally turtles swim directly toward the target, but after bilateral tectal lesions they take less direct routes and appear to be guided by tactile or olfactory cues rather than vision, although optokinetic testing shows they are not blind. After unilateral tectal lesions, turtles appear to show a form of neglect, showing deficits only when the food is placed in the affected visual field.

Similar to birds, turtles possess both tectofugal and thalamofugal visual pathways, but only the tectofugal pathway is critical for visual task performance. Lesions of the nucleus rotundus or its target in the forebrain, the core nucleus of the dorsal ventricular ridge, cause large deficits in visual discrimination tasks; lesions of the dorsal pallium, the termination of the thalamofugal pathway, have no evident effect on performance (Reiner & Powers, 1978; 1980; 1983). The degree of behavioral impairment is correlated with the extent of damage to the core nucleus and, at least for visual pattern discrimination, with the amount of damage to the lateral striatum (Reiner & Powers, 1983).

2.3 Amphibians

The behavior of frogs, toads and other amphibians fall into patterns that seem stereotyped, but these animals also illustrate how properties of selective attention can be built up from more basic components. To a first approximation, the visual world of a frog or toad can be divided into two important categories – prey and predators. Small horizontal bars, generally between 2 and 12 degrees of visual angle in length, act as triggers to release prey-catching movements that consist of turning towards the prey item and snapping, especially if the bars move parallel to their long axis (Ewert, 1970; 1974; 1968). Larger moving patterns, bigger than 30 degrees across, act as triggers for avoidance movements that involve turning away, crouching, and jumping. The visual triggers for prey-catching in salamanders are somewhat different, perhaps reflecting differences in their diet: both horizontal and vertical bars can be effective, provided the vertical bars move quickly (Luthardt & Roth, 1979). The links between these visual triggers and action patterns are not fixed, but depend on novelty and motivation. The orienting response habituates after several presentations of a potential prey item; conversely, if sufficiently enticed by the odor of mealworms, frogs will snap at large objects that would otherwise have been avoided (Ewert, 1970). If presented with two possible prey items, frogs will pick one, usually the nearer, and the reaction time is delayed if both items fall within the binocular field in front of the animal (Ingle, 1973b).

The key amphibian brain structure responsible for orienting toward prey items is the optic tectum (Figure 4A). The optic tectum receives visual signals from retinal ganglion cells that report behaviorally relevant features such as the jerky motion of small bug-like convex shapes, as well as moving edges and changes in brightness (Lettvin, Maturana, McCulloch, & Pitts, 1959). Tectal neurons elaborate on these signals and exhibit properties that match the features of prey-catching behavior. Many tectal neurons show a preference for “newness”, rapidly habituating during the repeated presentation of bug-like stimuli (Gaillard, 1990; Lettvin, Maturana, Pitts, & McCulloch, 1961). So-called “attention units” exhibit activity that continues for 3–4 seconds after their initial response to the brief movement of small prey-sized objects; this aligns with the delayed snapping responses to stationary prey, which are most likely to occur a few seconds after the prey has stopped moving (Ingle, 1975). In salamanders as well, tectal neurons exhibit activity related to the probability that the object in their receptive field is a prey item, based on the object’s shape and motion (Finkenstadt & Ewert, 1983).

Figure 4. Brain and prey-catching in the frog.

A. Lateral view of the frog brain, showing the location of the optic tectum (OT) and pretectal nucleli (pT) in the midbrain, and the striatum and dorsal pallium in the forebrain. B. Dorsal view of the frog brain (left) and schematic illustration (right) of the range of visual field locations that elicit prey-catching responses in the intact frog. C. Dorsal view of the frog brain after removal of the left optic tectum (left), and illustration (right) of the loss of prey-catching behavior in the affected portion of the right visual field. Prey-catching diagrams in panels (B) and (C) are a schematic reconstruction of the data presented in (Kostyk & Grobstein, 1982; 1987).

The tectum appears to be both sufficient and necessary to initiate prey-catching responses. Electrical stimulation of sites in the tectum evokes prey-catching behavior directed to locations in the front and contralateral side of the visual field, consistent with the eye-centered map in the tectum (Ewert, 1970). Conversely, after removal of the tectal lobe on one side (Figure 4B–C), frogs are completely unresponsive to prey items (e.g., mealworms) that are placed in the contralateral monocular visual field (Kostyk & Grobstein, 1982; 1987), and have longer reaction times for stimuli in the binocular visual field (Patton & Grobstein, 1998a). When the tectum is disconnected from downstream motor circuits by unilateral hemisection, frogs can detect prey items at all locations in the visual field but they no longer orient accurately for stimuli located on one side (Kostyk & Grobstein, 1982; 1987).

For predator avoidance, the pretectal region (pT) plays an important role (Figure 4A). Neurons in the pretectum respond well to features that indicate danger: tall moving objects, looming objects, and large objects moving or stationary (Ewert, 1974). Electrical stimulation of this region evokes defensive postures, such as ducking, jumping or turning away (Ewert, 1970). Lesions of the pretectum not only eliminate these predator avoidance behaviors, but also unleash prey-catching responses, creating a frog that indiscriminately snaps at almost anything that moves, as though they can no longer tell the difference between safe and dangerous objects (Ewert, 1970). These changes in prey-catching behavior are mediated through the tectum. The pretectum provides an inhibitory input to the tectum (Kang & Li, 2008; Wilczyniski & Northcutt, 1977), and the loss of this inhibition in pretectum-lesioned animals elevates the activity of tectal neurons to pathological levels, increases tectal receptive field size, and eliminates the response habituation normally found with repeated stimulus presentations (Ingle, 1973a).

As in birds, the optic tectum is also tightly associated with the midbrain isthmi nuclei (Gruberg & Udin, 1978). Localized lesions to the isthmi nuclei eliminate both prey catching and avoidance responses for visual items presented in the affected portion of the visual field (Caine & Gruberg, 1985; Gruberg, Wallace, Caine, & Mote, 1991); such lesions also increase the size of receptive fields for tectal neurons in the affected region (Gruberg et al., 1991).

The forebrain regulates the operation of these midbrain circuits (Figure 4A). Unilateral removal of the forebrain causes a contralateral deficit in the ability of frogs to orient toward visual prey (Patton & Grobstein, 1998a). The crucial structure appears to be the caudal striatum, because lesions restricted to this area produce neglect-like deficits as severe as removal of the entire telencephalic lobe, whereas removal of the entire dorsal pallium has little effect on visual orienting behavior (Patton & Grobstein, 1998b). Similarly, in aquatic frogs relying on their lateral line system to localize prey rather than vision, unilateral lesions of the caudal striatum cause contralateral tactile neglect; frogs with large forebrain lesions that spared the caudal striatum are unimpaired (Traub & Elepfandt, 1990). When presented with two competing prey items, intact salamanders readily choose one as the target for orienting, but after lesions of the striatum this behavior is nearly abolished and salamanders fail to orient, except if the prey item is especially salient (Ruhl, Hanslian, & Dicke, 2016).

The striatum appears to exert its influence on visual orienting through two major pathways to the optic tectum. One route is through the pretectum, noted above for its strong inhibitory influence on the tectum. Prey catching is eliminated by forebrain lesions but returns if a subsequent lesion is made to the pretectum; this suggests that the striatum may facilitate prey catching by reducing pretectal inhibition of the tectum (Ewert, 1970). This disynaptic route through pretectum is present in reptiles and birds as well as frogs, but has no evident homolog in mammals (Wilczynski & Northcutt, 1983). A second route is through the nucleus profundus mesencephalic, which receives a large striatal input and projects heavily to the optic tectum; this nucleus may be the homolog of the substantia nigra pars reticulata in mammals (Wilczynski & Northcutt, 1983).

Of course, frogs do more than snap at food and duck from danger – they also woo mates with a distinctive sonorous display. As male frogs engage in chorusing to attract a female, they appear to selectively attend to their loudest competitors and adjust the timing of their own call so that they get to lead the chorus (Greenfield & Rand, 2000). Conversely, as female frogs are serenaded, they favor calls that appear to occur first in each round, uncluttered by other croaks, providing a solution to the frog version of the cocktail party problem (Tarano, 2015). Acoustically guided behaviors like these are regulated by the internal state and motivation of the animal, and depend on forebrain structures including the striatum; after lesions of the striatum, female frogs ignore male callers, even though their ability to localize sounds remains intact (Walkowiak, Berlinger, Schul, & Gerhardt, 1999).

2.4 Fish

Over the past several years, there has been a flood of studies using larval zebrafish (Figure 5), which offer several advantages for investigating how brain circuits relate to behavior (Feierstein, Portugues, & Orger, 2015). Zebrafish are amenable to genetic modifications that make it possible to target specific populations of neurons, their heads are small and translucent so the entire brain can be observed at once at cellular resolution using a variety of imaging techniques (Figure 5C), and experiments can be done in intact and behaving animals either swimming freely or tethered in agarose gel while they move their eyes and tail fins in response to visual stimuli.

Figure 5. Brain and behavior in the zebrafish.

A. Dorsal view of a zebrafish illustrating the J-turn tail movement and convergent eye movements elicited by the presentation of a visual prey stimulus. B. Dorsal schematic view of the zebrafish brain showing the locations of the optic tectum and pallium. C. Image of larval zebrafish brain obtained using light-sheet microscopy while animal was presented with moving visual stimuli. Because the brain is small and the tissues are transparent, it is possible to image individual neurons throughout the tectum and pallium. Individual neurons are color-coded based on their preferred direction of motion, with magenta indicating preference for rightward motion and yellow indicating preference for leftward motion. Image reproduced with permission from (Freeman et al., 2014). D. Schematic diagram of the aquarium tank used in a 2-alternative choice task. After being introduced in the holding area, a trial was started when the opaque barriers were removed and the fish gained access to the choice zone and food delivery areas. Fish expressed their choice by swimming into one of the two food delivery areas and approaching the colored visual stimulus. E. The number of trials required to reach criterion performance (6 consecutive correct trials) during the four phases of the experiment. Panels (D) and (E) adapted with permission from (Parker et al., 2012).

Zebrafish have their own version of prey catching and predator avoidance behaviors (Figure 5A). Prey catching is prompted by small paramecium-sized (1 deg) moving spots, and begins with convergence eye movements that increase the binocular field of view and presumably enable stereoscopic targeting, accompanied by a series of “J-turn” tail movements to orient toward the prey item (Bianco, Kampff, & Engert, 2011; Trivedi & Bollmann, 2013). Predator avoidance is triggered by large or looming visual stimuli; as in frogs, the critical feature for determining when and if an escape should be initiated is size (20 degrees or bigger) and shadow-like dark forms on a bright background are especially effective (Dunn et al., 2016; Temizer, Donovan, Baier, & Semmelhack, 2015).

Visual signals supporting these behaviors are present on the outputs from the retina and elaborated in the midbrain (Figure 5B). Different classes of retinal ganglion cells show activity related to small prey-like stimuli or to larger looming predator-like stimuli, and terminate primarily in the optic tectum and the pretectum (Dunn et al., 2016; Preuss, Trivedi, Berg-Maurer, Ryu, & Bollmann, 2014; Semmelhack et al., 2014; Temizer et al., 2015). In the tectum, retinal inputs responsive to small stimuli terminate in the superficial layers, whereas inputs responsive to larger stimuli tend to terminate in deeper layers (Preuss et al., 2014). Distinct subsets of tectal neurons exhibit additional tuning, shaped by the activity of local inhibitory interneurons, for ethologically relevant combinations of features (Del Bene et al., 2010; Muto, Ohkura, Abe, Nakai, & Kawakami, 2013). For example, one population of tectal neurons shows selectivity for the particular mixture of size, contrast polarity, and movement speed needed to recognize an archetypal “prey” stimulus (Bianco & Engert, 2015).

Lesions of the tectum impair these approach and avoidance movements without affecting the optomotor responses evoked by large-field movements of the visual surround (Barker & Baier, 2015; Gahtan, Tanger, & Baier, 2005; Roeser & Baier, 2003; Temizer et al., 2015). Unilateral laser-induced lesions of retinal axons entering the tectum impairs the escape response evoked by looming visual stimuli (Temizer et al., 2015). Similarly, laser ablation of retinal axons in the pretectum reduces prey responses (Semmelhack et al., 2014). Genetic targeting of specific populations of neurons in the optic tectum reveals the specificity of the circuit elements. Targeted ablation of one population of tectal neurons reduces the avoidance behaviors evoked by large stimuli. Ablation of a different population of tectal neurons shifts the behavioral responses evoked by small stimuli from approach to avoidance, and conversely, optogenetic activation of these neurons increases the probability of approach behavior (Barker & Baier, 2015).

The behavioral choice to approach or avoid an object depends on the feeding state (Filosa, Barker, Dal Maschio, & Baier, 2016). When zebrafish are hungry, induced either by fasting or by manipulations of the neuroendocrine pathways, the frequency of approach behavior increases and the frequency of avoidance behavior decreases. Concomitant with these behavioral changes, there is an increase in the number of neurons in the optic tectum tuned for small visual stimuli, and this increase is apparently achieved by activating additional neurons tuned for small stimuli rather than by changing the tuning of individual neurons.

The flexibility of the association between visual stimuli and approach behavior has been demonstrated in experiments that require zebrafish to make an explicit choice in order to obtain food (Bilotta, Risner, Davis, & Haggbloom, 2005; Parker et al., 2012). In these studies, zebrafish start each trial in the anteroom of an aquarium and make their choice by swimming into one of two (or three) possible food delivery compartments, each marked on the back wall with a distinctive visual stimulus, such as a colored patch (Figure 5D). Over repeated trials and training sessions, zebrafish learn that one visual feature (e.g., a particular color) is associated with reward, and achieve reliable discrimination performance within a few hundred trials. After learning which color stimulus is associated with food, if the rule is then flipped so that the previously unrewarded color is now associated with reward, and vice versa, zebrafish not only learn the new rule but take significantly fewer trials to do so (Figure 5E), and this process of learning and flipping can be repeated with an entirely new set of color stimuli, leading to the interpretation that zebrafish can form and maintain some form of “attentional set” (Parker et al., 2012).

One of the most remarkable examples of selective orienting in fish is the prey catching behavior of the archer fish. Archer fish hunt in groups and use a precisely aimed squirt of water to knock down insect prey from overlying vegetation or out of the air (Figure 6A). They aim their squirt by orienting their bodies at the visual target (Timmermans & Souren, 2004) and can compensate for the refractive errors and optical distortions caused by viewing the target through the water-air interface at various angles (Schuster, Rossel, Schmidtmann, Jäger, & Poralla, 2004; Timmermans, 2001). They can hit moving targets, taking into account the extra distance that the target will travel until impact, as well as how the gravitational braking of their projectile depends on the height of the target, and can improve their accuracy not only through direct practice but also by observing other fish (Schuster, Wöhl, Griebsch, & Klostermeier, 2006). Once the prey is hit, the archer fish can predict within about 100 milliseconds where the prey will land on the water and quickly swim straight to that spot with kinematics as brisk as those used during escape movements (Rossel, Corlija, & Schuster, 2002; Wöhl & Schuster, 2007), presumably to beat out the other members of their hunting group. These impressive abilities presumably depend on the extraction of visual motion signals and other features in the retina and optic tectum (Ben-Tov et al., 2013; Tsvilling, Donchin, Shamir, & Segev, 2012; Vasserman, Shamir, Ben-Simon, & Segev, 2010).

Figure 6. Behavioral evidence of selective attention in archer fish.

A. Typical experimental arrangement with archer fish. Visual stimuli were presented on a visual display suspended over an aquarium. Reaction time was determined by contact of the water jet with a glass sheet protecting the visual display, and correct shots were rewarded with a small pellet of food. B. Schematic of a behavioral task using valid and invalid cues. On most trials (80%), the valid cue was presented that correctly indicated the location of the target stimulus. On a minority of trials (20%), an invalid cue was presented instead. Reaction time of the shot depended on the validity of the cue and the time delay between the cue and target appearance. C. Behavioral task using symbolic cues. The cue was presented at a central location on the display and its color (red or green) indicated the likely location of the upcoming target.

The prey catching behavior of archer fish also has properties that are analogous to well-known features of visual search and selective attention in mammals. When archer fish are presented with multiple visual stimuli, but the target “pops out” because it differs in speed or direction from the other distracter stimuli, the reaction time of their water squirt does not change as the number of distracters is varied (Ben-Tov, Donchin, Ben-Shahar, & Segev, 2015); this suggests that archer fish can exhibit pop-out effects during visual search like those seen in humans (Treisman & Gelade, 1980). The reaction times of archer fish are also influenced by spatial pre-cues like those used in classic attention paradigms with humans (Posner, Snyder, & Davidson, 1980). When archer fish are shown partially valid cues that indicate the likely location of an upcoming visual target, they tend to have faster reaction times when the cue is valid than when it is invalid (Figure 6B). But this effect depends on the timing – if the target appears long after the cue, then the effect is reversed, and they have longer reaction times for the validly cued location (Gabay, Leibovich, Ben-Simon, Henik, & Segev, 2013). Thus, archer fish show both a validity effect for spatial cues, and also inhibition of return, which is believed to encourage orienting toward novel locations (Klein, 2000). In addition, these spatial cueing effects can be found not only with spatial cues presented at the actual stimulus locations, but also with symbolic cues (e.g., color squares) presented at the center of the display (Figure 6C), suggesting that archer fish are also capable of some form of voluntary, endogenous control (Saban, Sekely, Klein, & Gabay, 2017).

3. The evolution of selective attention

As this brief survey illustrates, selective attention in at least an elemental form is ubiquitous across vertebrate species. The core function appears to be selective orienting – the ability to identify and choose an object in the environment as the impetus for an appropriate motor response, typically approaching a prey item or avoiding a threat. Selective orienting is strongly influenced by the physical properties of the stimulus, but it also depends on the motivation and internal state of the animal, and the novelty and relevance of the stimulus. It is the flexibility demonstrated during these seemingly simple behaviors that satisfies the definition of selective attention stated at the outset of this article: the ability to limit neural processing to a subset of signals, while actively excluding other eligible signals. Even in primates, selective orienting in the form of voluntary saccades has been taken as evidence of selective visual attention, dating back to the earliest physiological studies of attention in monkeys (Goldberg & Wurtz, 1972). Whether it is fish squirting at targets, birds pecking at screens, or monkeys making saccades to spots of light, vertebrates share an ability to guide their motor actions based on selective processing of signals in their environment. As Posner (1980) wrote: “While orienting to stimuli in visual space is a restricted sense of attention, I believe its study is capable of providing us both with important tests of the adequacy of general models of human cognition and with insights into the role of attention in more complex human activity” (page 4).

We can now offer some provisional answers to the questions raised in the introduction of the article. In animals that do not have a neocortex, do the subcortical brain regions implicated in attention also play a subsidiary role under the control of other forebrain areas? Apparently not. The optic tectum plays a primary role in stimulus selection in all of the non-mammalian vertebrates discussed in this article. The striatum and related circuits through the basal ganglia and thalamus are important for linking stimulus selection to the motivational state of the animal, and for action selection, and also appear to be present in at least rudimentary form in all vertebrate species (Grillner & Robertson, 2016; Redgrave, Prescott, & Gurney, 1999).

How important are different pallial subdivisions of the forebrain for selective attention? Here, the answers are intriguing but less certain because the homologies between non-mammals and mammals remain a topic of vigorous debate. In non-mammals, there is evidence that the dorsal pallium may not be crucial for selective attention (Patton & Grobstein, 1998b; Traub & Elepfandt, 1990), even though the same developmental territory in mammals is the source of progenitors for the neocortex (Dugas-Ford & Ragsdale, 2015; Jarvis et al., 2005; Karten, 2015; Montiel & Aboitiz, 2015), including the well-known sensory sites and control centers for selective attention (Kastner & Ungerleider, 2000; Petersen & Posner, 2012; Reynolds & Chelazzi, 2004). In non-mammals, the ventral and lateral pallium provide the terminus for the tectofugal pathway, including the dorsal ventricular ridge, all of which are crucial for visual discrimination in non-mammals; in mammals, the homologous circuits are not yet firmly established and serve less prominent functions, perhaps related to blindsight (Leopold, 2012) or to high-valence stimuli like faces (Johnson, 2005), although other recent work shows a direct link to selective attention (Peck & Salzman, 2014; Peck, Lau, & Salzman, 2013). The fate of these different pallial subdivisions in the primate, and their functional relationship to the tectofugal pathway, are important unsolved problems.

These answers prompt a further question: How did selective attention change with the emergence of the neocortex? We speculate that there may have been at least three important consequences. First, the expansion of sensory cortex made it possible to represent many more stimulus feature dimensions, so that conjunctions of values across these extra feature dimensions make it possible to identify many more distinct objects than would otherwise be possible. The visual world of a frog is painted with a very broad brush (prey or predator?), but mammals can recognize a seemingly limitless number of objects, and the individual meaning and value of each object can be learned and exploited. Second, unlike subcortical sensory areas, the neocortex has a general-purpose processing architecture, with the feature maps and tuning properties determined by experience-dependent mechanisms rather than being hardwired (Srihasam, Mandeville, Morocz, Sullivan, & Livingstone, 2012). The plasticity of the neocortex makes it possible to have detectors tuned to the statistics of the environment as experienced in the lifetime of an individual, rather than detectors improved incrementally through natural selection over multiple generations. Third, shifting the task of object recognition to the neocortex may have increased the value of making predictions. For some non-mammals, with rapid selection mechanisms operating subcortically at the earliest stages of sensory processing, it may be sufficient to rely on sensory inputs to discover the objects present in the environment and then react. But for mammals, if recognition is delayed until neocortical processing, selection might be dangerously slow on the draw, unless they could also make predictions that bias their sensory processing and motor actions in favor of the likely or planned outcomes (Miller & Cohen, 2001), or if some aspects of rapid innate recognition were preserved (Crouzet, 2010; Kirchner & Thorpe, 2006). This type of predictive control is not necessarily unique to mammals, and may be related to similar forebrain mechanisms present in vertebrates that lack a neocortex (Winkowski & Knudsen, 2007; 2008).

4. Conclusions

Birds, amphibians, reptiles, and fish do not have a neocortex, but they all respond selectively to stimuli, and there is emerging evidence that other eligible items are considered and actively excluded depending on the goals and motivational state of the animal. This capacity for selective attention depends on a set of highly conserved brain structures that are part of the brain plan for all vertebrates. In mammals, this brain plan is edited but not replaced, and amended and reorganized with the emergence of the neocortex. This combination of old and new brain circuits suggests several novel sites to investigate as potential loci for attentional control, and will hopefully lead to circuit-level explanations for how attention operates as a composite of several different cognitive processes. One of the most important unresolved issues is how the newer cortical circuits interact with, and are constrained by, the older evolutionarily conserved subcortical mechanisms. Understanding these interactions is likely to provide important insights into the etiology of attention disorders, neglect syndromes, and related brain dysfunctions.

Abbreviations

AGF

arcopallial gaze field

DVR

dorsal ventricular ridge

GLd

lateral geniculate pars dorsalis

Imc

nucleus isthmi pars magnocellularis

ION

isthmo-optic nucleus

Ipc

nucleus isthmi pars parvocellularis

OT

optic tectum

pT

pretectal region

Rt

nucleus rotundus of thalamus