Evolution of neural processing for visual perception in vertebrates

Abstract

Visual perception requires both visual information and attention. This review compares, across classes of vertebrates, the functional and anatomical characteristics of (a) the neural pathways that process visual information about objects, and (b) stimulus selection pathways that determine the objects to which an animal attends. Early in the evolution of vertebrate species, visual perception was dominated by information transmitted via the midbrain (retinotectal) visual pathway, and attention was probably controlled primarily by a selection network in the midbrain. In contrast, in primates, visual perception is dominated by information transmitted via the forebrain (retinogeniculate) visual pathway, and attention is mediated largely by networks in the forebrain. In birds and nonprimate mammals, both the retinotectal and retinogeniculate pathways contribute critically to visual information processing, and both midbrain and forebrain networks play important roles in controlling attention. The computations and processing strategies in birds and mammals share some strikingly similar characteristics despite over 300 million years of independent evolution and being implemented by distinct brain architectures. The similarity of these functional characteristics suggests that they provide valuable advantages to visual perception in advanced visual systems. A schema is proposed that describes the evolution of the pathways and computations that enable visual perception in vertebrate species.

Visual perception requires both visual information and attention. This review compares, across classes of vertebrates, the functional and anatomical characteristics of (a) the neural pathways that process visual information about objects, and (b) stimulus selection pathways that determine the objects to which an animal attends.

1. INTRODUCTION

Visual perception involves both the evaluation of stimulus features and attention, defined here as the differential processing of selected information that is relevant to behavior. The networks responsible for both of these functions have evolved together over the course of vertebrate evolution. Interestingly, the brain pathways and mechanisms that support these functions in mammals and birds have evolved independently for over 300 million years (Butler, Reiner, & Karten, 2011; Puelles, 2017). During this time, they have been shaped by selection pressures that are quite different due, in part, to the distinct visual challenges associated with walking on land versus flying through the air. It is not surprising, therefore, that they exhibit substantial differences between mammals and birds, including different anatomical routes of information transmission and the cytoarchitectures of the areas that process the information. This review emphasizes, however, remarkable similarities in the functional organization of these networks in mammals and birds, the two classes of vertebrates with the most sophisticated visual capabilities in the animal kingdom (Shimizu, Patton, & Husband, 2010; Wylie, Gutierrez‐Ibanez, Pakan, & Iwaniuk, 2009). These functional similarities either have been preserved over prolonged periods of independent evolution or have been arrived at independently through convergent evolution. In either case, such similarities indicate that these particular strategies of information processing are fundamentally important to visual perception in advanced visual systems.

2. PARALLEL PROCESSING IN THE RETINA

A basic strategy of information processing that is common to all vertebrate species is the extensive analysis of visual information by the retina (Baden et al., 2016; Ewert et al., 2001; Seabrook, Burbridge, Crair, & Huberman, 2017; Vanegas & Ito, 1983). For a given location, different retinal circuits extract different kinds of information, including luminance, local contrast, spatial and temporal frequency, motion, motion direction, loom, and wavelength. This information is transmitted to the brain via different classes of retinal ganglion cell (RGC) axons (Figure 1a). The different classes of RGCs exhibit distinct morphologies, input connections within the retina, and output connections to the brain. These characteristics indicate that they represent parallel channels for processing different kinds of information. The retina in all vertebrate species contains ~10 to 20 such functional classes of RGC filters. This parallel processing of independent visual properties by the retina enables the brain to analyze the visual scene rapidly and efficiently based on these specific properties.

Figure 1.

Schema for visual perception and the relative proportions of retinal axons contained in the retinogeniculate and retinotectal pathways in different classes of vertebrates. (a) Schema for information processing. Information from the eyes is processed in parallel in the forebrain (upper broken rectangle) and the midbrain (lower broken rectangle). Black rectangles: Stages in information processing. Blue arrows: Visual information. Purple arrows: Attentional signals that modulate or gate visual information. (b–d) The relative number of retinal ganglion cell axons that travel in the retinogeniculate (upper arrow) and retinotectal (lower arrow) pathways is represented by the relative widths of the arrows for monkeys (b), cats (c), pigeons (d), and turtles (e)

3. TWO PATHWAYS TO THE FOREBRAIN

In all vertebrate animals, the forebrain is the site where networks responsible for making cognitive decisions, generating emotional responses, and orchestrating complex behaviors reside (Figure 1a). Therefore, in the forebrain, stimuli must be identified. Visual information encoded by the RGCs reaches these forebrain networks via two afferent pathways. One pathway travels from the retina to the thalamus and, from there, to the forebrain; this pathway is referred to here as the “retinogeniculate pathway.” The second pathway travels from the retina to the optic tectum (OT; called the superior colliculus, SC, in mammals), from there to the thalamus, and, from there, to the forebrain; this pathway is referred to here as the “retinotectal pathway.”

The relative roles of these two pathways in conveying visual information to the forebrain has changed dramatically with the evolution of vertebrate classes (Figure 1b–e). In fish, amphibians and reptiles, the vast majority of visual information to the forebrain is transmitted via the retinotectal pathway. Conversely, the retinogeniculate pathway conveys information from only a small subset of RGC filters (Ewert et al., 2001; Fournier, Muller, Schneider, & Laurent, 2018; Vanegas & Ito, 1983). In birds and nonprimate mammals, the retinotectal pathway still comprises the majority of RGC axons, however the retinogeniculate pathway has become larger and its associated structures more differentiated (Kelly & Gilbert, 1975; Seabrook et al., 2017; Shimizu & Bowers, 1999). In primates, the relative importance of these pathways is reversed: The retinogeniculate pathway transmits the vast majority of visual information, and the associated structures are greatly hypertrophied and architecturally elaborated (Livingstone & Hubel, 1988). Conversely, the retinotectal pathway has become far less important in transmitting visual information to the forebrain, although it continues to play a critical role in directing spatial attention (Krauzlis, Lovejoy, & Zenon, 2013).

4. VISUAL PROCESSING IN MAMMALS

4.1. Retinogeniculate pathway in primates

Most of our current knowledge of how the vertebrate visual system processes information comes from studies of primates, particularly of old world monkeys (Livingstone & Hubel, 1988; Nassi & Callaway, 2009). In primates, visual perception depends almost exclusively on information conveyed by the retinogeniculate pathway. Correspondingly, the architecture of the primate retinogeniculate pathway is highly differentiated. More than 10 different classes of RGC filters project to a large, extensively laminated, dorsal lateral geniculate nucleus (LGN) (Figure 2a). Classes of RGCs, conveying specific kinds of information, project to specific laminae in the LGN, and LGN neurons largely maintain the response properties of their RGC inputs. The LGN represents topographically the foveal and contralateral visual field from both eyes, with spatially corresponding information from the ipsilateral and contralateral retinae projecting to spatially aligned, interleaved layers.

Figure 2.

Histological sections through the dorsal lateral geniculate nucleus (LGN) in different classes of vertebrates. These are transverse, Nissl‐stained sections of the LGN. Scale bar: 1 mm. Arrowhead: Dorsolateral border of the nucleus. (a) Monkey (Macaca mulatta); downloaded from BrainMaps.org. (b) Mouse (Mus musclulus); downloaded from BrainMaps.org. (c) Owl (Tyto alba). (d) Turtle (Trachemys scripta elegans); provided by Dr. Catherine Carr

Lesions restricted to specific LGN layers result in perceptual deficits for particular visual properties (Merigan, 1989; Merigan, Katz, & Maunsell, 1991). For example, lesions in the lower LGN layers impair contrast sensitivity for high temporal and low spatial frequencies, but have little effect on color sensitivity, speed or motion discrimination. Conversely, lesions in the upper LGN layers cause severe impairments in color discrimination, a decrease in contrast sensitivity for low temporal and high spatial frequencies, but have little effect on shape discrimination.

The major target of LGN neurons is the primary visual cortex (V1) (Livingstone & Hubel, 1988; Van Essen, 1985). Major transformations in the representation of visual information take place in V1. The various functional LGN channels, created originally by the RGC filters, project to different V1 layers or sublayers. Specialized circuits combine inputs, sometimes across information channels, to yield newly computed neuronal tuning for ocular dominance, retinal disparity, orientation, direction of motion, and wavelength contrast (Garg, Li, Rashid, & Callaway, 2019). These V1 circuits are dominated by feed forward activity (Figure 1a) and, therefore, they process information rapidly. The computed properties are represented systematically in overlapping columns in a single, topographic map of space. The high degree of functional organization in V1 is reflected in its crystalline anatomical organization, which gave rise to its commonly used name: “striate cortex.”

The functional channels created in V1 project systematically to segregated anatomical areas in a second visual area (V2) (Nassi & Callaway, 2009). The outputs of V1 and V2, in turn, provide the major source of visual information to many extra‐striate areas in the primate forebrain. These extrastriate areas group into two functional streams: the dorsal and ventral streams (Maunsell & Newsome, 1987). The dorsal stream (including MT, MST, and numerous areas in the parietal cortex) is specialized for processing motion, depth and spatial relationships, whereas the ventral stream (including V4 and numerous areas in the infrotemporal cortex) is specialized for processing object color, form and identity.

Lesion studies confirm the functional implications of the parallel computations performed by the dorsal and ventral streams (Nassi & Callaway, 2009). Lesions in dorsal stream areas impair the discrimination of object motion, direction and speed, the control of smooth pursuit eye movements, and spatial attention (Newsome & Pare, 1988; Quintana & Fuster, 1993). In contrast, lesions in ventral stream areas impair the discrimination of object shape and orientation, perceptual invariance, and attention to features (De Weerd, Peralta 3rd, Desimone, & Ungerleider, 1999; Merigan, 1996).

The forebrain pathways are organized hierarchically and operate as a layered control architecture (Prescott, Redgrave, & Gurney, 1999). At high levels in the hierarchy, networks act as dynamical systems, transforming the information from the dorsal and ventral streams into distributed population codes (Chaisangmongkon, Swaminathan, Freedman, & Wang, 2017; Mante, Sussillo, Shenoy, & Newsome, 2013; Rigotti et al., 2013; Shanahan, Bingman, Shimizu, Wild, & Gunturkun, 2013; X. J. Wang, 2008). These networks are distinguished by their extensively interconnected and recurrent architectures. They have access to a wide range of information, including memories, emotional state, context and priorities, and they are capable of computing decisions about stimulus identity and choosing and planning movements in response to those stimuli (Figure 1a). Neurons in these networks have large receptive fields, and responses correlate with multiple stimuli and task conditions as the population response unfolds over time. This distributed, dynamical code can communicate readily with memory and motor planning networks (Shenoy, Sahani, & Churchland, 2013; Sugar & Moser, 2019).

In addition, the primate forebrain also contains networks that select the information to which the animal attends (Buschman & Kastner, 2015). Selection is mediated by a variety of circuits, including feedback circuits from higher to lower levels in the processing hierarchy, thalamic circuits (particularly the thalamic reticular nucleus), which gate information to the cortex, and neuromodulatory circuits in the basal forebrain that enhance the representation of selected information (Figure 1a) (Knudsen, 2018). These circuits, which are controlled by attention networks primarily in the prefrontal and parietal cortex, gate and enhance the visual information that gains access to decision making networks. Lesions in these networks result in visual agnosia, that is, an inability to interpret visual information (Lynch & McLaren, 1989), rather than a disruption of visual information per se.

4.2. Retinotectal pathway in primates

The retinotectal pathway in primates is not essential for visual perception. The SC receives input from only about 10% of RGCs, particularly from RGCs that are highly sensitive to local motion and that also project to the lower LGN layers, which are known to process stimulus motion (Dacey, 2000; Perry & Cowey, 1984). The RGC projection to the SC originates from both eyes and represents topographically the fovea and the visual field contralateral (but not ipsilateral) to the fovea (Kruger, 1970).

Like the SC in all mammals, the primate SC contains 6 anatomical layers, grouped functionally into the superficial visual layers and the deeper multimodal and motor related layers (Figure 3a) (Wurtz & Albano, 1980). The superficial visual layers, in addition to receiving input directly from the retina, also receive descending input from striate and extrastriate cortices.

Figure 3.

Histological sections through the optic tectum/superior colliculus (OT/SC) in different classes of vertebrates. These are transverse, Nissl‐stained sections of the OT/SC. Scale bar: 1 mm. Arrowhead: Dorsolateral border of the structure. (a) Monkey (Macaca mulatta); downloaded from BrainMaps.org. (b) Mouse (Mus musclulus); downloaded from BrainMaps.org. (c) Owl (Tyto alba). (d) Turtle (Trachemys scripta elegans); provided by Dr. Catherine Carr

Spatial features of the visual scene are not analyzed parametrically within the SC, as they are in V1. Neurons in the superficial SC layers have small receptive fields, those in the deeper layers have large receptive fields and are often multimodal, and all neurons exhibit a strong preference for small stimuli (Stein & Meredith, 1993). The responses of SC neurons largely follow those of their RGC input, with response rates increasing with increasing stimulus strength. Thus, the SC maintains the functional channels that were established in the retina, and it transmits them (particularly motion) to its target structures in the thalamus (Lyon, Nassi, & Callaway, 2010).

Although feature values are not analyzed in the SC, the responses that pass through the SC can be modulated powerfully by the priority of a stimulus (Fecteau & Munoz, 2006; Knudsen & Schwarz, 2017). The SC receives descending input from the frontoparietal attention networks (Figure 1a). In addition, it interacts with a selection network in the midbrain tegmentum that amplifies the strength of responses to stimuli at the highest priority location (based on the physical salience and behavioral relevance of stimuli) and globally suppresses responses to inputs from all other locations (Knudsen, 2012).

Lesions in the primate SC do not impair visual discriminations. Instead, they cause deficits in spatial attention (Krauzlis et al., 2013). The attention deficit is profound when a monkey is confronted with competing stimuli: in the presence of a competing stimulus, they no longer discriminate the feature properties of cued target stimuli at the lesioned location. The evidence indicates that the SC directs spatial attention to the highest priority location by gating visual information from the selected location into decision‐making networks in the forebrain (Sridharan, Steinmetz, Moore, & Knudsen, 2017).

Ascending SC outputs, which convey visual and attentional information, project to the thalamus: to the LGN and to the pulvinar nucleus. The pulvinar projects to extrastriate areas in both the dorsal and ventral streams and to frontoparietal areas that control attention (Saalmann & Kastner, 2011; H. H. Zhou, Schafer, & Desimone, 2016).

4.3. Visual processing in nonprimate mammals

The visual system in primates is not typical of mammalian species for a number of reasons. (a) Primates have a single deep fovea, they depend heavily on foveal vision for high spatial resolution, and the vast majority of forebrain visual areas are devoted to processing information from the foveae (Van Essen, 1985). These traits are not true for nonprimate mammals, which have shallow or no foveas (Rapaport & Stone, 1984). (b) Unlike in primates, the SC in nonprimate mammals receives a greater proportion of retinal projections than the LGN (Figure 1c) (Ellis, Gauvain, Sivyer, & Murphy, 2016; Ito & Feldheim, 2018; Kelly & Gilbert, 1975), and it represents the entire visual field of the contralateral retina (in primates, the SC representation stops at the vertical meridian) (Kruger, 1970). (c) The architecture of the LGN and V1 is less differentiated than in primates (Figure 2b), and the LGN can project strongly to multiple forebrain visual areas (Glickstein, King, Miller, & Berkley, 1967). Despite these differences, however, visual processing in nonprimate and primate mammals is remarkably similar.

4.4. Retinogeniculate pathway in nonprimate mammals

As in primates, the LGN receives inputs from both eyes and it preserves the functional channels established by the RGC filters (Lennie, 1980; Seabrook et al., 2017; Yeh, Stoelzel, & Alonso, 2003). The LGN projects directly or indirectly to multiple cortical areas, and these areas divide into those primarily involved with motion and location processing (like the primate dorsal stream) and those primarily involved with identity processing (like the primate ventral stream) (Marshel, Garrett, Nauhaus, & Callaway, 2011). In V1, specialized circuits that are dominated by feed‐forward activity transform LGN input into neuronal tuning for ocular dominance, retinal disparity, orientation, direction of motion and spatial frequency. In cats, these properties are represented systematically in overlapping columns in a single, retinotopic map of space (Hubel & Wiesel, 1962). In mice, V1 neurons with these properties are distributed more randomly (Niell & Stryker, 2008; Samonds, Choi, & Priebe, 2019; Scholl, Burge, & Priebe, 2013; Seabrook et al., 2017).

The information conveyed by the retinogeniculate pathway contributes critically to perception. Lesions of V1 result in behavioral deficits in visual feature discrimination, although some pattern and form discrimination remains when higher visual areas are left intact (Orban, Vandenbussche, Sprague, & De Weerd, 1990; Tohmi, Meguro, Tsukano, Hishida, & Shibuki, 2014). Moreover, direct optogenetic activation of V1 neurons tuned for a particular contour orientation leads to behavioral reports of orientation perception (Marshel et al., 2019).

In addition, attention networks in the forebrain (Figure 1a), analogous to those in primates, control the visual information that nonprimate mammals perceive (Knudsen, 2018). These networks have been studied primarily in mice. They include feedback circuits from high‐order cortical areas to V1 (Hu et al., 2019; Zhang et al., 2014), thalamic circuits (particularly in the thalamic reticular nucleus) that gate selected information and inhibit other information on its way to V1 (Wimmer et al., 2015), and cholinergic circuits formed by neurons in the basal forebrain that enhance the representation of visual information in the cortex (Pinto et al., 2013; Sarter, Hasselmo, Bruno, & Givens, 2005). These circuits act cooperatively to gate and enhance selected information as it ascends the visual processing hierarchy.

4.5. Retinotectal pathway in nonprimate mammals

Unlike in primates, the SC is a major target of retinal afferents in nonprimate mammals (Figure 1c) (Ito & Feldheim, 2018; Kelly & Gilbert, 1975; Wassle & Illing, 1980). As in primates, the SC consists of six layers (Figure 3b), and different RGC types terminate in different layers. Numerous RGC channels confer on SC neurons functional selectivities for visual features that correlate strongly with behaviorally relevant stimuli, such as stimuli that loom, move, or are unusually dark or bright (Gale & Murphy, 2014). Most RGCs that project to the SC have poor spatial, but high temporal resolution.

The SC projects to the thalamus: to the LGN and to a number of zones in the pulvinar nucleus (Bickford, Zhou, Krahe, Govindaiah, & Guido, 2015; N. A. Zhou, Maire, Masterson, & Bickford, 2017). Different functional classes of SC neurons project to different pulvinar zones. Some zones receive topographic SC input, while others receive decidedly nontopographic input, as in birds (Wylie et al., 2009). The thalamic nuclei project, in turn, to both striate and extrastriate visual areas, providing a substrate for the SC to contribute to the processing of object features in the forebrain (Hu et al., 2019).

As in all vertebrates, there is no evidence that the SC performs parametric analysis of visual features. A report of orientation columns in the mouse SC (Feinberg & Meister, 2015) likely represents, instead, systematic changes in receptive field anisotropy across the space map, as neurons representing a particular location are all tuned to the same orientation, rather than to various orientations, as in V1.

Results from lesion‐behavior studies demonstrate that the SC contributes to the analysis of visual objects by the forebrain. SC lesions reduce neural responses to visual stimuli in extrastriate visual cortical areas (Ogino & Ohtsuka, 2000; Tohmi et al., 2014). In several of these areas, responses to high velocity stimuli are selectively lost following SC lesions. In mice, a cortical area (postrhinal cortex) encodes object motion, and its responses depend on input from the SC, but not on input from V1 (Beltramo & Scanziani, 2019). Lesions of the SC also lead to profound neglect of visual stimuli at the lesioned locations (Dean & Redgrave, 1984), and this neglect does not require the presence of competing stimuli to be expressed as agnosia, as it does in primates (Lovejoy & Krauzlis, 2010).

In summary, in nonprimate mammals as in primates, the retinogeniculate pathway is primarily responsible for providing the information that the forebrain uses for the analysis of visual objects, and lesions in V1 impair the animal's capacity to discriminate feature values. In addition, forebrain circuits for mediating attention (Figure 1a) are well‐developed. However, information conveyed via the retinotectal pathway plays a far greater role in visual perception in nonprimates than in primates: it contributes critically both to the analysis of object motion and to the control of spatial attention.

5. VISUAL PROCESSING IN BIRDS

In contrast with mammals, the retinotectal pathway in birds plays the dominant role in visual perception. The anatomical differentiation of the retinogeniculate and retinotectal pathways reflects this difference in their relative contributions to perception.

5.1. Retinogeniculate pathway in birds

The architecture of the retinogeniculate pathway is not as elaborated in birds as it is in mammals (Shimizu & Bowers, 1999). All of the RGC axons that project to the LGN cross in the chiasm to the contralateral LGN (Figure 2c), where they create a monocular representation of the visual field of the contralateral eye (Bravo & Pettigrew, 1981; Remy & Gunturkun, 1991). The LGN, in turn, projects to the visual Wulst, a three‐layered pallial structure analogous to the mammalian V1 but organized very differently (Karten, Hodos, Nauta, & Revzin, 1973).

Another interesting architectural difference between birds and mammals is the forebrain pathway that supports binocular integration (Karten et al., 1973; Pettigrew & Konishi, 1976b). In mammals, binocular integration in the forebrain is enabled by the partial decussation of RGC axons in the chiasm: some RGC axons representing the binocular visual field remain ipsilateral and project to the ipsilateral LGN while the rest project to the contralateral LGN. The resulting binocular representation in the mammalian LGN is transmitted to the ipsilateral V1 where corresponding inputs from both eyes are integrated. In birds, the projection from each retina is entirely crossed and, instead, the portion of the LGN representing the binocular visual field projects bilaterally to the Wulst (Karten et al., 1973), where corresponding binocular inputs are integrated.

Given these architectural differences and the over 300 million years of independent evolution of birds and mammals, the discovery made by Jack Pettigrew and Mark Konishi that the major computations performed in the bird Wulst and the mammalian V1 are the same, is surprising (Pettigrew & Konishi, 1976b). Pettigrew and Konishi found that, just like neurons in the mammalian V1, neurons in the Wulst of owls transform the responses of monocular, center‐surround LGN inputs into binocular tuning for orientation, direction of motion and retinal disparity of local straight‐line contours, and that these neurons are organized topographically in a map of the frontal and contralateral visual field. They found no systematic organization of neurons based on orientation or binocular tuning, hence the representation in the Wulst of owls is most similar to that in the V1 of mice (Seabrook et al., 2017). In addition, they showed that eye alignment as well as binocularity and retinal disparity tuning in the Wulst, depend on early binocular experience, as they do in mammals (Pettigrew & Konishi, 1976a). Subsequent studies show that the Wulst contains several retinotopic areas, each with distinct thalamo‐Wulst inputs and functional properties (Bischof et al., 2016). In one area, >90% of neurons respond to illusory contours (contours inferred from boundaries of phase‐shifted gratings), as do a subset of neurons in the mammalian V1 (Nieder & Wagner, 1999). The degree of elaboration of the Wulst varies dramatically across bird species (Iwaniuk, Heesy, Hall, & Wylie, 2008), suggesting that its functional capacities vary across bird lineages.

The derivation of feature values in the Wulst argues strongly for a role of the retinogeniculate pathway in object identification. One would expect, therefore, that lesions of the LGN or Wulst would lead to severe impairments in object discrimination in birds, comparable to those observed following lesions of the LGN or V1 in mammals. Surprisingly, this is not the case. Visual deficits following lesions of the Wulst are not readily apparent, except for deficits in the memorization of object locations (Watanabe, Mayer, & Bischof, 2011). Wulst lesions do not cause deficits in the discrimination of light intensities, large patterns of dots from stripes, categories of foods or conspecifics, or stripe intervals (Hodos, Karten, & Bonbright Jr., 1973; Watanabe, 1996).

These results demonstrate that information conveyed in the retinogeniculate pathway (Figure 1a) is not required for basic visual perception in birds. However, none of the lesion‐behavior studies have tested for the ability of birds to discriminate among values of specific spatial features (i.e., among different orientations, distances, or directions of motion). Behavioral paradigms and stimuli that enable such measurements have now been developed (Knudsen, Schwarz, Knudsen, & Sridharan, 2017; Nguyen et al., 2004; Nieder & Wagner, 2001). Future experiments should apply these techniques to test threshold discrimination performance in birds with lesions in the retinogeniculate pathway. The functional properties of Wulst neurons predict perceptual deficits in such discriminations.

Circuits in the bird forebrain that are likely to mediate attention have been identified anatomically (Knudsen, 2018). However, their functions have not been studied in behaving animals. The circuits include feedback circuits among forebrain pallial areas (Shanahan et al., 2013), top‐down circuits from high‐order forebrain areas to the midbrain (Winkowski & Knudsen, 2008), a well‐developed thalamic reticular nucleus (Butler, 2008; Reiner, Yamamoto, & Karten, 2005), and cholinergic circuits formed by neurons in the basal forebrain (Medina & Reiner, 1994; Sarter et al., 2005).

5.2. Retinotectal pathway in birds

The substantial capacity for visual perception that birds retain following lesions in the retinogeniculate pathway can be accounted for by the processing of information provided by their well‐developed retinotectal pathway (Figure 1a,d). The basic anatomical and functional organization of this pathway is similar to that in mammals (Shimizu & Bowers, 1999). The most conspicuous difference is that the architecture of the OT is far more differentiated in birds (Figure 3c), perhaps reflecting the exceptional challenges associated with responding efficiently to visual information while flying.

The bird OT contains 15 distinct layers. Like in the six‐layered mammalian SC, the layers are divided into superficial visual layers and deeper multimodal and motor‐related layers (Knudsen, 2011). The visual layers receive RGC inputs directly from the contralateral retina from all functional classes of RGCs, with specific classes terminating in specific superficial layers (Hellmann, Gunturkun, & Manns, 2004; Marin et al., 2003; Yamagata, Weiner, Dulac, Roth, & Sanes, 2006). In addition, they receive descending visual input from the Wulst, and they send ascending projections to the LGN, as well as to other vision‐related structures.

As in mammals, the deeper, multimodal and motor‐related layers contain mutually aligned retinotopic maps of space for all sensory modalities that provide spatial information and for the spatial goal of impending movements (Knudsen & Schwarz, 2017). These layers receive visual inputs directly from the retina, descending visual input from the Wulst, ascending and descending sensory input, as well as movement related input from the forebrain. Output pathways from the deep layers include (a) descending pathways that connect with premotor structures in the brainstem and spinal cord to coordinate immediate, goal‐oriented movements and (b) ascending pathways that project to various thalamic nuclei that transmit visual and attention‐related information to the forebrain (Figure 1a).

Ascending pathways from the OT project mainly to the nucleus rotundus, analogous to the pulvinar nucleus in mammals (Karten & Shimizu, 1989; Wylie et al., 2009). Like the mammalian pulvinar (N. A. Zhou et al., 2017), the nucleus rotundus contains a number of functionally specialized zones, with different zones receiving input from different functional classes of visual neurons in the deeper OT layers (Hellmann et al., 2004; Marin et al., 2003). Neurons in an anterior zone of the nucleus rotundus respond strongly to changes in color or luminance, whereas neurons in ventral and posterior zones respond strongly to motion.

In addition to visual information, the OT provides the nucleus rotundus with information about the location of the highest priority visual stimulus (Figure 1a) (Marin et al., 2007). Stimulus priority is computed in the OT based on the relative salience and relevance of stimuli, and the highest priority stimulus is represented categorically as the location in the OT space map with the highest level of activity (Mysore, Asadollahi, & Knudsen, 2011). The computation results from the interaction of the OT with a selection network in the midbrain tegmentum that compares the relative strength of activation across the OT space map, amplifies responses at the selected location, and suppresses responses at all other locations (Asadollahi & Knudsen, 2016; Garrido‐Charad et al., 2018; Mysore & Knudsen, 2012). This network exists in all vertebrate classes, but is highly elaborated in birds, suggesting that it contributes critically to the control of attention (Knudsen, 2011).

The evaluation of stimulus salience is based on multimodal stimulus characteristics, and the nucleus rotundus contains neurons that are strongly multimodal (Reches & Gutfreund, 2009). Moving visual stimuli, particularly looming stimuli, are extremely effective in driving the responses of these multimodal neurons (Knudsen & Schwarz, 2017). Many functional studies of the nucleus rotundus have not tested for multimodal responses. Hence, it is possible that studies that described the sensitivity of neurons in the nucleus rotundus to visual motion in the posterior or ventral regions were actually describing the responses of multimodal neurons to stimulus salience, neurons that contribute to attention control.

A striking transformation in the representation of visual information takes place in the nucleus rotundus: Visual receptive fields become extremely large and retinotopy is lost or at least severely degraded (Y. C. Wang, Jiang, & Frost, 1993; Wylie et al., 2009). Thus, unlike the retinotopic representation that is maintained in the Wulst, retinotopic information in the retinotectal pathway is transformed at the level of the nucleus rotundus into a distributed code. This distributed representation is reminiscent of the representations of object identity information in high‐level visual areas in the pulvinar and extra‐striate cortex of primates (Petersen, Robinson, & Keys, 1985; Saalmann & Kastner, 2011).

Much like the mammalian pulvinar nucleus, subdivisions of the nucleus rotundus project systematically to different regions of the entopallium, the equivalent of mammalian cortex (Laverghetta & Shimizu, 2003). Some entopallial regions process certain kinds of visual information: Lesions in the anterior area result in behavioral impairments in color and pattern discrimination tasks, whereas lesions of the posterior area result in impairments in motion discrimination (Cook, Patton, & Shimizu, 2013; Nguyen et al., 2004). In addition, the entopallium contains a region that encodes multimodal stimulus salience rather than motion per se and, as discussed above for the nucleus rotundus, this region may be involved in the control of attention (Marin et al., 2007; Reches & Gutfreund, 2009).

The behavioral deficits that result from OT inactivation confirm that the retinotectal pathway conveys attention control signals as well as visual object information to the forebrain. OT lesions result in complete neglect of visual stimuli presented in the lesioned portion of the visual field. Most lesion studies could not distinguish deficits caused by agnosia (attention deficit) from those caused by the loss of the visual information per se (Cook et al., 2013; Nguyen et al., 2004; Watanabe et al., 2011). However, one OT lesion study has demonstrated agnosia for line orientations, a feature value that is processed in the Wulst (Knudsen et al., 2017). The inability of birds to discriminate line orientations following OT lesions indicates that OT attention signals gate not only visual information that is transmitted to the forebrain via the retinotectal pathway, but also gate information that is transmitted to the forebrain via the retinogeniculate pathway, as is true also for the SC in mammals (Krauzlis et al., 2013; Sridharan et al., 2017).

Because the functional properties of neurons in the nucleus rotundus and entopallium are likely to be dramatically modulated by attention, and because they have not been studied previously in behaving birds, the full extent of their coding capabilities remains unknown. In the future, it will be essential to study the properties of these neurons in behaving birds using paradigms that control attention (Knudsen et al., 2017).

6. OVERVIEW OF VISUAL INFORMATION PROCESSING IN MAMMALS AND BIRDS

The pathways responsible for visual perception share many similarities in mammals and birds (Figure 1). The retina sends visual information in numerous functional RGC channels to both the LGN and to the SC/OT in the retinogeniculate and retinotectal pathways, respectively. Both the LGN and the SC/OT maintain these functional channels in high‐resolution retinotopic representations.

The LGN transmits information to the V1/Wulst, where certain features of the visual scene are analyzed parametrically in retinotopic representations of space. The features that are analyzed in the V1 and the Wulst (i.e., binocularity, retinal disparity, local line orientation and direction of motion) are strikingly similar, given that these structures evolved independently and have different architectures. This functional similarity suggests that these particular visual features are valuable to the subsequent computations performed in the forebrain visual hierarchy and that this level in the forebrain hierarchy (and not in the OT) is the preferred level for performing these computations.

In the parallel, retinotecal pathway, visual information is conveyed to the SC/OT, where the visual information is maintained and the priorities (relative salience and relevance) of stimuli across the visual field are compared. The absence of parametric feature analysis in the SC/OT indicates that feature values are not essential to the performance of midbrain functions. The SC/OT transmits visual information and attentional signals to the forebrain via the pulvinar/rotundus (Figure 1a).

Visual information from the retinogeniculate and retinotectal pathways converges in the forebrain to support visual perception. In mammals, high‐order forebrain areas transform the information into distributed, dynamical codes that synthesize visual and other information and make decisions about stimulus identity and the spatial relationships of stimuli (Chaisangmongkon et al., 2017; Gold & Shadlen, 2007; Mante et al., 2013; Meyers, Freedman, Kreiman, Miller, & Poggio, 2008). A similar transformation probably occurs in the bird forebrain (Shanahan et al., 2013), as evidenced by the transformation of retinotopic information into a distributed code in the nucleus rotundus and entopallium (Wylie et al., 2009).

A conspicuous difference between mammals and birds is the relative importance of the retinogeniculate versus the retinotectal pathway in supporting visual perception (Figure 1b–d). In mammals and particularly in primates, the retinogeniculate pathway is the dominant pathway, whereas in birds, the retinotectal pathway predominates. To gain insight into the origins of this difference, we turn to the organization of visual information processing in fish, amphibians, and reptiles.

7. VISUAL PROCESSING IN FISH, AMPHIBIANS AND REPTILES

Visual information processing in fish, amphibians and reptiles is most similar to that in birds. In these classes of vertebrates, the retinotectal pathway is the dominant pathway supporting visual perception (Figure 4). In addition to providing visual and attention‐related information to the forebrain, the OT communicates directly with descending pathways that coordinate immediate orienting, feeding and defensive behaviors essential to survival (Ewert et al., 2001; Foster & Hall, 1975; D. P. M. Northmore, 2011; Vanegas & Ito, 1983).

Figure 4.

Schema for visual perception in reptiles. Conventions are the same as in Figure 1. The relative widths of the blue arrows represent the relative number of retinal ganglion cell axons that travel in the retinogeniculate (upper arrow) and retinotectal (lower arrow) pathways in turtles

Essentially all RGCs in the contralateral retina send input to the OT, which receives and sorts their input in a retinotopic map of space (Ben‐Tov et al., 2013; Ewert et al., 2001; B. E. Stein & Gaither, 1983; Vanegas & Ito, 1983). Some RGC filters are tuned for stimulus properties that are consistently useful in prey capture or predator avoidance, such as sudden motion or loom (Ben‐Tov et al., 2013; Ewert et al., 2001; Lettvin, Maturana, McCulloch, & Pitts, 1959). As in mammals and birds, the OT comprises superficial visual layers in which the RGCs terminate, and deeper multimodal and motor‐related layers (Figure 3d). OT neurons respond best to small, physically salient stimuli, particularly to moving stimuli. A network in the midbrain tegmentum interconnects with the OT (Gruberg et al., 2006; Luiten, 1981) and, working together, they could compute the relative salience of stimuli across the OT map of space (although this capacity has not been demonstrated physiologically).

The major ascending pathway from the OT to the forebrain originates in the deeper layers (Belekhova et al., 2003; Ewert et al., 2001; Vanegas & Ito, 1983). These layers project to the thalamic nucleus rotundus, which projects to visual areas in the dorsal ventricular ridge of the forebrain. Lesions made in the OT result in complete neglect of highly salient stimuli (Ingle, 1973; D. P. Northmore & Masino, 1984; Reiner & Powers, 1983). This neglect probably reflects both the loss of visual information per se and the loss of a spatial attention signal. Nevertheless, animals with OT lesions are still capable of navigating through obstacles.

The retinogeniculate pathway is not well differentiated anatomically (Figure 2d) (Ewert et al., 2001; Fournier et al., 2018; Vanegas & Ito, 1983). Only a small subset of RGC filters send axonal branches to the LGN. The LGN projects to a region of the forebrain pallium that is separate from the dorsal ventricular ridge that receives input from the nucleus rotundus. Lesions of the LGN or visual pallium have no apparent effects on visually guided behaviors, including the discrimination of luminance or large patterns (Reiner & Powers, 1983), although deficits in feature value discrimination at threshold have not been tested. Thus, the information conveyed by the retinotectal pathway can support most visual perception.

Ancestral attention circuits in the forebrain, analogous to those in mammals and birds, are likely to influence the visual information that these classes of vertebrates perceive (Knudsen, 2018). These circuits have been identified anatomically, but have not been studied functionally. They include the thalamic reticular nucleus (Butler, 2008), thought to gate selected information in the forebrain (Wimmer et al., 2015), and cholinergic circuits formed by neurons in the basal forebrain that could amplify the representation of selected information (Sarter et al., 2005; Wullimann & Rink, 2002).

7.1. The dorsal cortex in turtles

Jack Pettigrew and Mark Konishi discovered that the V1 in mammals and the Wulst in birds compute similar kinds of information (Pettigrew & Konishi, 1976b). Does this similarity originate from a common ancestor or from convergent evolution? The retinogeniculate pathway in turtles provides insights to the likely precursor of the V1/Wulst in mammals and birds.

Turtles evolved from reptilian stem species about 250 million years ago, much like mammals (Butler et al., 2011; Puelles, 2017). In turtles, the equivalent of the V1/Wulst is called the dorsal cortex, a three‐layered cortex that covers the turtle forebrain and receives visual input from the LGN (Figure 2d). Unlike the V1/Wulst, the dorsal cortex does not perform parallel parametric analyses of various visual features across the visual scene (Fournier et al., 2018). Instead, the retinotopic organization in the LGN is severely degraded, reflecting an absence of point‐to‐point projections from the LGN to the dorsal cortex (Mulligan & Ulinski, 1990). Dorsal cortex neurons have extremely large receptive fields that encompass most or all of the contralateral visual field (Fournier et al., 2018). Their responses to stimuli at different locations within the receptive field vary, providing spatial information at the level of the population response. Pronounced, sustained oscillatory activity develops over time, and this activity represents the spatial and temporal structure of the visual scene in a distributed code (Figure 4). These properties are reminiscent of the distributed, dynamical representations described for high‐order forebrain areas in mammals (Chaisangmongkon et al., 2017; Meyers et al., 2008). The recurrent neural architecture that supports dynamical coding in the forebrain presumably first evolved to analyze and identify olfactory input from the first cranial nerve (Aboitiz & Montiel, 2015; Niessing & Friedrich, 2010).

Networks that depend on population dynamics to derive information are slower to settle on solutions than are hierarchies of feedforward filters. On the other hand, for small populations of neurons, a distributed recurrent architecture is far more flexible and versatile in its ability to convert high‐dimensional visual information into low‐dimensional information about stimulus identity and spatial relationships (X. J. Wang, 2008). Therefore, a factor that may explain the immediate transformation of information from the LGN into a distributed code in the dorsal cortex (Figure 4) is the paucity of neuronal resources in the forebrain: The reptilian dorsal cortex (and the equivalent visual forebrain areas in fish and amphibians) consists of a relatively small population of neurons (Butler et al., 2011; Karten, 2015; Wicht & Northcutt, 1992). Moreover, there is, so far, no evidence of multiple hierarchical levels of visual processing in the forebrains of these species.

The vastly different cytoarchitectures in V1, Wulst, and dorsal cortex, and the absence of parallel parametric analysis of visual features across the visual field in the turtle dorsal cortex indicate that the computations performed by both the mammalian V1 and the bird Wulst probably evolved independently. This convergent evolution implies that these particular feature values, computed in parallel across the visual scene, are highly beneficial as input to the distributed dynamical networks that operate in decision making at higher hierarchical levels in mammals and birds (Figure 1a).

8. HYPOTHETICAL SCHEMA FOR THE EVOLUTION OF VISUAL PROCESSING

The data presented in this review are consistent with the following schema for the evolution of information processing for visual perception in vertebrates.

Early in vertebrate evolution, the OT was the primary target of the optic nerve (Figures 1e and 4). Essentially all RGC axons projected topographically to the contralateral OT, providing the OT with a high‐resolution representation of all information from the visual field of the contralateral eye. Visual information was processed in the retina by different classes of RGC filters for basic stimulus properties that aided general scene analysis (e.g., local contrast and luminance) as well as for stimulus properties that correlated strongly with properties of prey or predators (e.g., motion, loom, or particular shape). Information from the retina was sorted to different OT layers, which maintained the information channels established by the RGCs. This information ascended through the thalamus to distributed recurrent networks in the forebrain for scene analysis and object identification.

The highest priority information in the visual scene, to which the animal attended, was selected and enhanced in the OT (Figure 4). The OT interacted with a network of neurons in the midbrain tegmentum that compared the relative levels of activity across the OT space map. In addition to visual input, the network received descending information from the forebrain, indicating the behavioral relevance of locations or stimuli. The network amplified retinal input at the site with the strongest activity, and decreased responses at all other sites. Powerful response adaptation of OT neurons assured that the site of maximal activity changed over time.

The effect of the midbrain network was to enhance the representation of the selected visual information that the OT transmitted to the thalamus (Figure 4). (When appropriate, this activity could also trigger immediate orienting, feeding or defensive behavior.) In the thalamus, the enhanced activity caused the reticular thalamic nucleus to spatially gate the visual information that gained access to decision and memory networks in the forebrain. Thus, both visual information for object identification and attention were dominated by activity ascending from the midbrain.

Nevertheless, axonal branches from a small proportion of RGC axons also provided nonadapting retinotopic input to neurons in the LGN. The LGN relayed this input directly to forebrain neurons that were embedded in a network with distributed, recurrent architecture. This network analyzed the global spatial and temporal statistics of the visual scene.

Information from the retinogeniculate and retinotectal pathways was transformed in the forebrain into distributed dynamical representations, and the information from both pathways was combined to derive object identity. Object identity was used for generating emotional responses, for planning complex behaviors, and for storage in memory, operations that an animal could afford to carry out slowly and deliberately.

8.1. Birds

Visual information processing in birds included these ancestral characteristics and added a couple of important improvements. First, perhaps enabled by the availability of larger populations of forebrain neurons, the retinogeniculate pathway added an analytic stage in the Wulst before the information entered the distributed forebrain networks for synthesis and decision making (Figure 1a). The Wulst computed line orientations, directions of motion, and retinal disparities parametrically and retinotopically across the visual scene. Although this additional processing step cost substantial neural resources, the benefits were that the values of these features were computed quickly and with high resolution. This additional step greatly increased the speed and resolution of the forebrain networks in identifying stimuli, a critically important capacity when flying.

Second, the anatomical architectures of both the OT and the midbrain selection network were elaborated. The increased layering of the OT increased the number of functional channels for transmitting information to the forebrain. The increased size and differentiation of the midbrain selection network improved the speed and precision with which the OT computed the highest priority stimulus in the visual scene for attention. The hypertrophy of the midbrain network reflected its continued dominance as a major source of stimulus‐driven attention signals for gating visual information in the forebrain (Figure 1a).

8.2. Mammals

Visual information processing in early mammals also reflected the characteristics of their ancestors. Unlike in birds, however, the improvements that appeared in mammals were almost entirely in the retinogeniculate pathway. The LGN acquired input from both eyes and sorted RGC axons into eye‐specific layers. The V1 enlarged and began computing feature values parametrically and retinotopically across the visual field. Numerous cortical areas were driven by visual information transmitted through the LGN. These areas became organized in multilayered hierarchies that analyzed spatial relationships in the scene (dorsal stream) and the identity of stimuli (ventral stream). Recurrent networks synthesized visual information with context and task‐related information, and decisions were represented dynamically.

Along with the elaboration of visual processing architecture, there was also an elaboration of high‐order areas that controlled attention. Forebrain networks exploited thalamic and neuromodulatory circuits, as well as greatly increased feedback connections, to differentially enhance sensory responses to selected stimuli. Forebrain attentional signals competed and coordinated with signals from the midbrain to determine the visual information that gained access to decision‐making networks in the forebrain.

In contrast, the retinotectal pathway underwent little change from that in ancestral vertebrates.

8.3. Primates

The evolutionary trends that began in nonprimate mammals, greatly accelerated in primates. In primates, the retinogeniculate pathway became, by far, the dominant pathway for vision. The vast majority of RGC axons terminated in the LGN rather than in the SC (Figure 1b). The LGN and V1 became hypertrophied and exquisitely differentiated (Figure 2a), and the functional hierarchies of information processing increased in both number of hierarchical layers and number of areas in each layer. Ultimately, visual information about stimulus identity was synthesized by forebrain neurons with large receptive fields and mixed selectivities that were embedded in recurrent networks with distributed architectures, acting as dynamical systems. Also, forebrain networks that mediated attention greatly expanded in the prefrontal and parietal cortex and largely controlled the visual information that was differentially processed in the forebrain visual hierarchy. These changes enabled the exceptionally sophisticated visual perceptual capabilities that are characteristic of primates.

In contrast to the retinogeniculate pathway, the retinotectal pathway decreased in importance in primates (Figure 1b). It no longer was required to provide information used by the forebrain for stimulus identification, even though visual information (particularly motion) propagated through it. However, a contribution of the SC to spatial attention persisted. The SC continued to signal to the forebrain the highest priority location for attention. Indeed, when there were multiple competing stimuli in the environment and the forebrain and midbrain signals differed in their selection of the highest priority location, it was the signal from the SC that controlled spatial attention (Lovejoy & Krauzlis, 2010). The capability of the SC to direct visual spatial attention in primates reflects the ancestral roles of the OT in directing spatial attention and immediate ballistic responses to high priority stimuli, roles that have been essential to an animal's survival since the beginning of vertebrate evolution.

9. CONCLUDING REMARKS

The brain is a miracle of evolution. Its capacities are superior in many ways to those of our most powerful computers. The processing strategies and mechanisms that it employs to enable visual perception serve as a window to understanding how the brain implements these capacities. Although the visual system of humans is among the most sophisticated in the animal kingdom, its capacities reflect the selective pressures that have shaped our species. Other lineages have had to respond to different pressures and have evolved certain visual capacities that are superior to our own. Exploring similarities as well as differences across species will lead to a deep understanding of mechanisms that are critical for perception.

The seminal work of Jack Pettigrew demonstrates the unique insights that can be gained only through comparative studies. Among his many contributions, he discovered that similar, dramatic transformations in visual information coding occur in both the visual cortex of cats and the visual Wulst of owls, species separated by over 300 million years of independent evolution. In both lineages, these transformations take place at the same stage in the forebrain pathway, and not in the midbrain, and they precede the transformation of information into distributed, dynamical codes for decision making. Hence, theses specific transformations must provide distinct advantages to the computations that occur in the forebrains of birds and mammals to support visual perception.

As emphasized in this review, visual perception involves both sensory information processing and attention. Much comparative work, particularly in nonprimate species, has been conducted on animals that were anesthetized and, therefore, unable to control eye positions or to attend to stimuli. In primates, attention has been shown to alter or modulate the representation of information even at early stages of visual processing, and all classes of vertebrates possess circuits that mediate attention. Therefore, when studying any species, in order to reveal the full capacity of the brain to process visual information and the full range of mechanisms involved, the functional properties of neurons must be studied in animals that can control their eyes and that are engaged in visual tasks. This is especially true when exploring the properties of neurons at higher levels in visual pathways.

Knudsen EI. Evolution of neural processing for visual perception in vertebrates. J Comp Neurol. 2020;528:2888–2901. 10.1002/cne.24871

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analyzed in this study.